2024-03-05

Title: Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models

• Authors: Zachary Horvitz, Jingru Chen, Rahul Aditya, Harshvardhan Srivastava, Robert West, Zhou Yu, Kathleen McKeown
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00794
• Pdf URL: https://arxiv.org/pdf/2403.00794
• Copy Paste: [[2403.00794]] Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models(https://arxiv.org/abs/2403.00794)
• Keywords: large language model
• Abstract: Humor is a fundamental facet of human cognition and interaction. Yet, despite recent advances in natural language processing, humor detection remains a challenging task that is complicated by the scarcity of datasets that pair humorous texts with similar non-humorous counterparts. In our work, we investigate whether large language models (LLMs), can generate synthetic data for humor detection via editing texts. We benchmark LLMs on an existing human dataset and show that current LLMs display an impressive ability to `unfun' jokes, as judged by humans and as measured on the downstream task of humor detection. We extend our approach to a code-mixed English-Hindi humor dataset, where we find that GPT-4's synthetic data is highly rated by bilingual annotators and provides challenging adversarial examples for humor classifiers.

Title: Executing Natural Language-Described Algorithms with Large Language Models: An Investigation

• Authors: Xin Zheng, Qiming Zhu, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00795
• Pdf URL: https://arxiv.org/pdf/2403.00795
• Copy Paste: [[2403.00795]] Executing Natural Language-Described Algorithms with Large Language Models: An Investigation(https://arxiv.org/abs/2403.00795)
• Keywords: large language model
• Abstract: Executing computer programs described in natural language has long been a pursuit of computer science. With the advent of enhanced natural language understanding capabilities exhibited by large language models (LLMs), the path toward this goal has been illuminated. In this paper, we seek to examine the capacity of present-day LLMs to comprehend and execute algorithms outlined in natural language. We established an algorithm test set sourced from Introduction to Algorithm, a well-known textbook that contains many representative widely-used algorithms. To systematically assess LLMs' code execution abilities, we selected 30 algorithms, generated 300 random-sampled instances in total, and evaluated whether popular LLMs can understand and execute these algorithms. Our findings reveal that LLMs, notably GPT-4, can effectively execute programs described in natural language, as long as no heavy numeric computation is involved. We believe our findings contribute to evaluating LLMs' code execution abilities and would encourage further investigation and application for the computation power of LLMs.

Title: An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning

• Authors: Zui Chen, Yezeng Chen, Jiaqi Han, Zhijie Huang, Ji Qi, Yi Zhou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00799
• Pdf URL: https://arxiv.org/pdf/2403.00799
• Copy Paste: [[2403.00799]] An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning(https://arxiv.org/abs/2403.00799)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) are displaying emergent abilities for math reasoning tasks,and there is a growing attention on enhancing the ability of open-source LLMs through supervised fine-tuning (SFT).In this paper, we aim to explore a general data strategy for supervised data to help optimize and expand math reasoning ability.Firstly, we determine the ability boundary of reasoning paths augmentation by identifying these paths' minimal optimal set.Secondly, we validate that different abilities of the model can be cumulatively enhanced by Mix of Minimal Optimal Sets of corresponding types of data, while our models MMOS achieve SOTA performance on series base models under much lower construction costs.Besides, we point out GSM-HARD is not really hard and today's LLMs no longer lack numerical robustness.Also, we provide an Auto Problem Generator for robustness testing and educational applications.Our code and data are publicly available at https://github.com/cyzhh/MMOS.

Title: Brain-Inspired Two-Stage Approach: Enhancing Mathematical Reasoning by Imitating Human Thought Processes

• Authors: Yezeng Chen, Zui Chen, Yi Zhou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00800
• Pdf URL: https://arxiv.org/pdf/2403.00800
• Copy Paste: [[2403.00800]] Brain-Inspired Two-Stage Approach: Enhancing Mathematical Reasoning by Imitating Human Thought Processes(https://arxiv.org/abs/2403.00800)
• Keywords: large language model
• Abstract: Although large language models demonstrate emergent abilities in solving math word problems, there is a challenging task in complex multi-step mathematical reasoning tasks. To improve model performance on mathematical reasoning tasks, previous work has conducted supervised fine-tuning on open-source models by improving the quality and quantity of data. In this paper, we propose a novel approach, named Brain, to imitate human thought processes to enhance mathematical reasoning abilities, using the Frontal Lobe Model to generate plans, and then employing the Parietal Lobe Model to generate code and execute to obtain answers. First, we achieve SOTA performance in comparison with Code LLaMA 7B based models through this method. Secondly, we find that plans can be explicitly extracted from natural language, code, or formal language. Our code and data are publicly available at https://github.com/cyzhh/Brain.

Title: IPED: An Implicit Perspective for Relational Triple Extraction based on Diffusion Model

• Authors: Jianli Zhao, Changhao Xu, Bin Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00808
• Pdf URL: https://arxiv.org/pdf/2403.00808
• Copy Paste: [[2403.00808]] IPED: An Implicit Perspective for Relational Triple Extraction based on Diffusion Model(https://arxiv.org/abs/2403.00808)
• Keywords: extraction, diffusion, generative
• Abstract: Relational triple extraction is a fundamental task in the field of information extraction, and a promising framework based on table filling has recently gained attention as a potential baseline for entity relation extraction. However, inherent shortcomings such as redundant information and incomplete triple recognition remain problematic. To address these challenges, we propose an Implicit Perspective for relational triple Extraction based on Diffusion model (IPED), an innovative approach for extracting relational triples. Our classifier-free solution adopts an implicit strategy using block coverage to complete the tables, avoiding the limitations of explicit tagging methods. Additionally, we introduce a generative model structure, the block-denoising diffusion model, to collaborate with our implicit perspective and effectively circumvent redundant information disruptions. Experimental results on two popular datasets demonstrate that IPED achieves state-of-the-art performance while gaining superior inference speed and low computational complexity. To support future research, we have made our source code publicly available online.

Title: LoRA Meets Dropout under a Unified Framework

• Authors: Sheng Wang, Liheng Chen, Jiyue Jiang, Boyang Xue, Lingpeng Kong, Chuan Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00812
• Pdf URL: https://arxiv.org/pdf/2403.00812
• Copy Paste: [[2403.00812]] LoRA Meets Dropout under a Unified Framework(https://arxiv.org/abs/2403.00812)
• Keywords: transformer, large language model
• Abstract: With the remarkable capabilities, large language models (LLMs) have emerged as essential elements in numerous NLP applications, while parameter-efficient finetuning, especially LoRA, has gained popularity as a lightweight approach for model customization. Meanwhile, various dropout methods, initially designed for full finetuning with all the parameters updated, alleviates overfitting associated with excessive parameter redundancy. Hence, a possible contradiction arises from negligible trainable parameters of LoRA and the effectiveness of previous dropout methods, which has been largely overlooked. To fill this gap, we first confirm that parameter-efficient LoRA is also overfitting-prone. We then revisit transformer-specific dropout methods, and establish their equivalence and distinctions mathematically and empirically. Building upon this comparative analysis, we introduce a unified framework for a comprehensive investigation, which instantiates these methods based on dropping position, structural pattern and compensation measure. Through this framework, we reveal the new preferences and performance comparisons of them when involved with limited trainable parameters. This framework also allows us to amalgamate the most favorable aspects into a novel dropout method named HiddenKey. Extensive experiments verify the remarkable superiority and sufficiency of HiddenKey across multiple models and tasks, which highlights it as the preferred approach for high-performance and parameter-efficient finetuning of LLMs.

Title: UrbanGPT: Spatio-Temporal Large Language Models

• Authors: Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, Chao Huang
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.00813
• Pdf URL: https://arxiv.org/pdf/2403.00813
• Copy Paste: [[2403.00813]] UrbanGPT: Spatio-Temporal Large Language Models(https://arxiv.org/abs/2403.00813)
• Keywords: large language model
• Abstract: Spatio-temporal prediction aims to forecast and gain insights into the ever-changing dynamics of urban environments across both time and space. Its purpose is to anticipate future patterns, trends, and events in diverse facets of urban life, including transportation, population movement, and crime rates. Although numerous efforts have been dedicated to developing neural network techniques for accurate predictions on spatio-temporal data, it is important to note that many of these methods heavily depend on having sufficient labeled data to generate precise spatio-temporal representations. Unfortunately, the issue of data scarcity is pervasive in practical urban sensing scenarios. Consequently, it becomes necessary to build a spatio-temporal model with strong generalization capabilities across diverse spatio-temporal learning scenarios. Taking inspiration from the remarkable achievements of large language models (LLMs), our objective is to create a spatio-temporal LLM that can exhibit exceptional generalization capabilities across a wide range of downstream urban tasks. To achieve this objective, we present the UrbanGPT, which seamlessly integrates a spatio-temporal dependency encoder with the instruction-tuning paradigm. This integration enables LLMs to comprehend the complex inter-dependencies across time and space, facilitating more comprehensive and accurate predictions under data scarcity. To validate the effectiveness of our approach, we conduct extensive experiments on various public datasets, covering different spatio-temporal prediction tasks. The results consistently demonstrate that our UrbanGPT, with its carefully designed architecture, consistently outperforms state-of-the-art baselines. These findings highlight the potential of building large language models for spatio-temporal learning, particularly in zero-shot scenarios where labeled data is scarce.

Title: DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models

• Authors: Wei He, Kai Han, Yehui Tang, Chengcheng Wang, Yujie Yang, Tianyu Guo, Yunhe Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00818
• Pdf URL: https://arxiv.org/pdf/2403.00818
• Copy Paste: [[2403.00818]] DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models(https://arxiv.org/abs/2403.00818)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) face a daunting challenge due to the excessive computational and memory requirements of the commonly used Transformer architecture. While state space model (SSM) is a new type of foundational network architecture offering lower computational complexity, their performance has yet to fully rival that of Transformers. This paper introduces DenseSSM, a novel approach to enhance the flow of hidden information between layers in SSMs. By selectively integrating shallowlayer hidden states into deeper layers, DenseSSM retains fine-grained information crucial for the final output. Dense connections enhanced DenseSSM still maintains the training parallelizability and inference efficiency. The proposed method can be widely applicable to various SSM types like RetNet and Mamba. With similar model size, DenseSSM achieves significant improvements, exemplified by DenseRetNet outperforming the original RetNet with up to 5% accuracy improvement on public benchmarks.

Title: Social Media as a Sensor: Analyzing Twitter Data for Breast Cancer Medication Effects Using Natural Language Processing

• Authors: Seibi Kobara, Alireza Rafiei, Masoud Nateghi, Selen Bozkurt, Rishikesan Kamaleswaran, Abeed Sarker
• Subjects: cs.CL, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.00821
• Pdf URL: https://arxiv.org/pdf/2403.00821
• Copy Paste: [[2403.00821]] Social Media as a Sensor: Analyzing Twitter Data for Breast Cancer Medication Effects Using Natural Language Processing(https://arxiv.org/abs/2403.00821)
• Keywords: transformer
• Abstract: Breast cancer is a significant public health concern and is the leading cause of cancer-related deaths among women. Despite advances in breast cancer treatments, medication non-adherence remains a major problem. As electronic health records do not typically capture patient-reported outcomes that may reveal information about medication-related experiences, social media presents an attractive resource for enhancing our understanding of the patients' treatment experiences. In this paper, we developed natural language processing (NLP) based methodologies to study information posted by an automatically curated breast cancer cohort from social media. We employed a transformer-based classifier to identify breast cancer patients/survivors on X (Twitter) based on their self-reported information, and we collected longitudinal data from their profiles. We then designed a multi-layer rule-based model to develop a breast cancer therapy-associated side effect lexicon and detect patterns of medication usage and associated side effects among breast cancer patients. 1,454,637 posts were available from 583,962 unique users, of which 62,042 were detected as breast cancer members using our transformer-based model. 198 cohort members mentioned breast cancer medications with tamoxifen as the most common. Our side effect lexicon identified well-known side effects of hormone and chemotherapy. Furthermore, it discovered a subject feeling towards cancer and medications, which may suggest a pre-clinical phase of side effects or emotional distress. This analysis highlighted not only the utility of NLP techniques in unstructured social media data to identify self-reported breast cancer posts, medication usage patterns, and treatment side effects but also the richness of social data on such clinical questions.

Title: Comparing effectiveness of regularization methods on text classification: Simple and complex model in data shortage situation

• Authors: Jongga Lee, Jaeseung Yim, Seohee Park, Changwon Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00825
• Pdf URL: https://arxiv.org/pdf/2403.00825
• Copy Paste: [[2403.00825]] Comparing effectiveness of regularization methods on text classification: Simple and complex model in data shortage situation(https://arxiv.org/abs/2403.00825)
• Keywords: robust
• Abstract: Text classification is the task of assigning a document to a predefined class. However, it is expensive to acquire enough labeled documents or to label them. In this paper, we study the regularization methods' effects on various classification models when only a few labeled data are available. We compare a simple word embedding-based model, which is simple but effective, with complex models (CNN and BiLSTM). In supervised learning, adversarial training can further regularize the model. When an unlabeled dataset is available, we can regularize the model using semi-supervised learning methods such as the Pi model and virtual adversarial training. We evaluate the regularization effects on four text classification datasets (AG news, DBpedia, Yahoo! Answers, Yelp Polarity), using only 0.1% to 0.5% of the original labeled training documents. The simple model performs relatively well in fully supervised learning, but with the help of adversarial training and semi-supervised learning, both simple and complex models can be regularized, showing better results for complex models. Although the simple model is robust to overfitting, a complex model with well-designed prior beliefs can be also robust to overfitting.

Title: LLMGuard: Guarding Against Unsafe LLM Behavior

• Authors: Shubh Goyal, Medha Hira, Shubham Mishra, Sukriti Goyal, Arnav Goel, Niharika Dadu, Kirushikesh DB, Sameep Mehta, Nishtha Madaan
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00826
• Pdf URL: https://arxiv.org/pdf/2403.00826
• Copy Paste: [[2403.00826]] LLMGuard: Guarding Against Unsafe LLM Behavior(https://arxiv.org/abs/2403.00826)
• Keywords: robust, large language model
• Abstract: Although the rise of Large Language Models (LLMs) in enterprise settings brings new opportunities and capabilities, it also brings challenges, such as the risk of generating inappropriate, biased, or misleading content that violates regulations and can have legal concerns. To alleviate this, we present "LLMGuard", a tool that monitors user interactions with an LLM application and flags content against specific behaviours or conversation topics. To do this robustly, LLMGuard employs an ensemble of detectors.

Title: Self-Refinement of Language Models from External Proxy Metrics Feedback

• Authors: Keshav Ramji, Young-Suk Lee, Ramón Fernandez Astudillo, Md Arafat Sultan, Tahira Naseem, Asim Munawar, Radu Florian, Salim Roukos
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00827
• Pdf URL: https://arxiv.org/pdf/2403.00827
• Copy Paste: [[2403.00827]] Self-Refinement of Language Models from External Proxy Metrics Feedback(https://arxiv.org/abs/2403.00827)
• Keywords: large language model
• Abstract: It is often desirable for Large Language Models (LLMs) to capture multiple objectives when providing a response. In document-grounded response generation, for example, agent responses are expected to be relevant to a user's query while also being grounded in a given document. In this paper, we introduce Proxy Metric-based Self-Refinement (ProMiSe), which enables an LLM to refine its own initial response along key dimensions of quality guided by external metrics feedback, yielding an overall better final response. ProMiSe leverages feedback on response quality through principle-specific proxy metrics, and iteratively refines its response one principle at a time. We apply ProMiSe to open source language models Flan-T5-XXL and Llama-2-13B-Chat, to evaluate its performance on document-grounded question answering datasets, MultiDoc2Dial and QuAC, demonstrating that self-refinement improves response quality. We further show that fine-tuning Llama-2-13B-Chat on the synthetic dialogue data generated by ProMiSe yields significant performance improvements over the zero-shot baseline as well as a supervised fine-tuned model on human annotated data.

Title: Deep Learning Detection Method for Large Language Models-Generated Scientific Content

• Authors: Bushra Alhijawi, Rawan Jarrar, Aseel AbuAlRub, Arwa Bader
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00828
• Pdf URL: https://arxiv.org/pdf/2403.00828
• Copy Paste: [[2403.00828]] Deep Learning Detection Method for Large Language Models-Generated Scientific Content(https://arxiv.org/abs/2403.00828)
• Keywords: large language model
• Abstract: Large Language Models (LLMs), such as GPT-3 and BERT, reshape how textual content is written and communicated. These models have the potential to generate scientific content that is indistinguishable from that written by humans. Hence, LLMs carry severe consequences for the scientific community, which relies on the integrity and reliability of publications. This research paper presents a novel ChatGPT-generated scientific text detection method, AI-Catcher. AI-Catcher integrates two deep learning models, multilayer perceptron (MLP) and convolutional neural networks (CNN). The MLP learns the feature representations of the linguistic and statistical features. The CNN extracts high-level representations of the sequential patterns from the textual content. AI-Catcher is a multimodal model that fuses hidden patterns derived from MLP and CNN. In addition, a new ChatGPT-Generated scientific text dataset is collected to enhance AI-generated text detection tools, AIGTxt. AIGTxt contains 3000 records collected from published academic articles across ten domains and divided into three classes: Human-written, ChatGPT-generated, and Mixed text. Several experiments are conducted to evaluate the performance of AI-Catcher. The comparative results demonstrate the capability of AI-Catcher to distinguish between human-written and ChatGPT-generated scientific text more accurately than alternative methods. On average, AI-Catcher improved accuracy by 37.4%.

Title: CLLMs: Consistency Large Language Models

• Authors: Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, Hao Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00835
• Pdf URL: https://arxiv.org/pdf/2403.00835
• Copy Paste: [[2403.00835]] CLLMs: Consistency Large Language Models(https://arxiv.org/abs/2403.00835)
• Keywords: large language model
• Abstract: Parallel decoding methods such as Jacobi decoding show promise for more efficient LLM inference as it breaks the sequential nature of the LLM decoding process and transforms it into parallelizable computation. However, in practice, it achieves little speedup compared to traditional autoregressive (AR) decoding, primarily because Jacobi decoding seldom accurately predicts more than one token in a single fixed-point iteration step. To address this, we develop a new approach aimed at realizing fast convergence from any state to the fixed point on a Jacobi trajectory. This is accomplished by refining the target LLM to consistently predict the fixed point given any state as input. Extensive experiments demonstrate the effectiveness of our method, showing 2.4$\times$ to 3.4$\times$ improvements in generation speed while preserving generation quality across both domain-specific and open-domain benchmarks.

Title: EyeGPT: Ophthalmic Assistant with Large Language Models

• Authors: Xiaolan Chen, Ziwei Zhao, Weiyi Zhang, Pusheng Xu, Le Gao, Mingpu Xu, Yue Wu, Yinwen Li, Danli Shi, Mingguang He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00840
• Pdf URL: https://arxiv.org/pdf/2403.00840
• Copy Paste: [[2403.00840]] EyeGPT: Ophthalmic Assistant with Large Language Models(https://arxiv.org/abs/2403.00840)
• Keywords: large language model
• Abstract: Artificial intelligence (AI) has gained significant attention in healthcare consultation due to its potential to improve clinical workflow and enhance medical communication. However, owing to the complex nature of medical information, large language models (LLM) trained with general world knowledge might not possess the capability to tackle medical-related tasks at an expert level. Here, we introduce EyeGPT, a specialized LLM designed specifically for ophthalmology, using three optimization strategies including role-playing, finetuning, and retrieval-augmented generation. In particular, we proposed a comprehensive evaluation framework that encompasses a diverse dataset, covering various subspecialties of ophthalmology, different users, and diverse inquiry intents. Moreover, we considered multiple evaluation metrics, including accuracy, understandability, trustworthiness, empathy, and the proportion of hallucinations. By assessing the performance of different EyeGPT variants, we identify the most effective one, which exhibits comparable levels of understandability, trustworthiness, and empathy to human ophthalmologists (all Ps>0.05). Overall, ur study provides valuable insights for future research, facilitating comprehensive comparisons and evaluations of different strategies for developing specialized LLMs in ophthalmology. The potential benefits include enhancing the patient experience in eye care and optimizing ophthalmologists' services.

Title: Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs

• Authors: Raghavv Goel, Mukul Gagrani, Wonseok Jeon, Junyoung Park, Mingu Lee, Christopher Lott
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00858
• Pdf URL: https://arxiv.org/pdf/2403.00858
• Copy Paste: [[2403.00858]] Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs(https://arxiv.org/abs/2403.00858)
• Keywords: large language model
• Abstract: Text generation with Large Language Models (LLMs) is known to be memory bound due to the combination of their auto-regressive nature, huge parameter counts, and limited memory bandwidths, often resulting in low token rates. Speculative decoding has been proposed as a solution for LLM inference acceleration. However, since draft models are often unavailable in the modern open-source LLM families, e.g., for Llama 2 7B, training a high-quality draft model is required to enable inference acceleration via speculative decoding. In this paper, we propose a simple draft model training framework for direct alignment to chat-capable target models. With the proposed framework, we train Llama 2 Chat Drafter 115M, a draft model for Llama 2 Chat 7B or larger, with only 1.64\% of the original size. Our training framework only consists of pretraining, distillation dataset generation, and finetuning with knowledge distillation, with no additional alignment procedure. For the finetuning step, we use instruction-response pairs generated by target model for distillation in plausible data distribution, and propose a new Total Variation Distance++ (TVD++) loss that incorporates variance reduction techniques inspired from the policy gradient method in reinforcement learning. Our empirical results show that Llama 2 Chat Drafter 115M with speculative decoding achieves up to 2.3 block efficiency and 2.4$\times$ speed-up relative to autoregressive decoding on various tasks with no further task-specific fine-tuning.

Title: NewsBench: Systematic Evaluation of LLMs for Writing Proficiency and Safety Adherence in Chinese Journalistic Editorial Applications

• Authors: Miao Li, Ming-Bin Chen, Bo Tang, Shengbin Hou, Pengyu Wang, Haiying Deng, Zhiyu Li, Feiyu Xiong, Keming Mao, Peng Cheng, Yi Luo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00862
• Pdf URL: https://arxiv.org/pdf/2403.00862
• Copy Paste: [[2403.00862]] NewsBench: Systematic Evaluation of LLMs for Writing Proficiency and Safety Adherence in Chinese Journalistic Editorial Applications(https://arxiv.org/abs/2403.00862)
• Keywords: large language model
• Abstract: This study presents NewsBench, a novel benchmark framework developed to evaluate the capability of Large Language Models (LLMs) in Chinese Journalistic Writing Proficiency (JWP) and their Safety Adherence (SA), addressing the gap between journalistic ethics and the risks associated with AI utilization. Comprising 1,267 tasks across 5 editorial applications, 7 aspects (including safety and journalistic writing with 4 detailed facets), and spanning 24 news topics domains, NewsBench employs two GPT-4 based automatic evaluation protocols validated by human assessment. Our comprehensive analysis of 11 LLMs highlighted GPT-4 and ERNIE Bot as top performers, yet revealed a relative deficiency in journalistic ethic adherence during creative writing tasks. These findings underscore the need for enhanced ethical guidance in AI-generated journalistic content, marking a step forward in aligning AI capabilities with journalistic standards and safety considerations.

Title: Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes

• Authors: Xiaomeng Hu, Pin-Yu Chen, Tsung-Yi Ho
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00867
• Pdf URL: https://arxiv.org/pdf/2403.00867
• Copy Paste: [[2403.00867]] Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes(https://arxiv.org/abs/2403.00867)
• Keywords: attack, generative, large language model
• Abstract: Large Language Models (LLMs) are becoming a prominent generative AI tool, where the user enters a query and the LLM generates an answer. To reduce harm and misuse, efforts have been made to align these LLMs to human values using advanced training techniques such as Reinforcement Learning from Human Feedback (RLHF). However, recent studies have highlighted the vulnerability of LLMs to adversarial jailbreak attempts aiming at subverting the embedded safety guardrails. To address this challenge, this paper defines and investigates the Refusal Loss of LLMs and then proposes a method called Gradient Cuff to detect jailbreak attempts. Gradient Cuff exploits the unique properties observed in the refusal loss landscape, including functional values and its smoothness, to design an effective two-step detection strategy. Experimental results on two aligned LLMs (LLaMA-2-7B-Chat and Vicuna-7B-V1.5) and six types of jailbreak attacks (GCG, AutoDAN, PAIR, TAP, Base64, and LRL) show that Gradient Cuff can significantly improve the LLM's rejection capability for malicious jailbreak queries, while maintaining the model's performance for benign user queries by adjusting the detection threshold.

Title: SoftTiger: A Clinical Foundation Model for Healthcare Workflows

• Authors: Ye Chen, Igor Couto, Wei Cai, Cong Fu, Bruno Dorneles
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00868
• Pdf URL: https://arxiv.org/pdf/2403.00868
• Copy Paste: [[2403.00868]] SoftTiger: A Clinical Foundation Model for Healthcare Workflows(https://arxiv.org/abs/2403.00868)
• Keywords: extraction, large language model
• Abstract: We release and introduce SoftTiger, a clinical large language model (CLaM) designed as a foundation model for healthcare workflows. The narrative and unstructured nature of clinical notes is a major obstacle for healthcare intelligentization. We address a critical problem of structuring clinical notes into clinical data, according to international interoperability standards. We collect and annotate data for three critical subtasks, namely, international patient summary, clinical impression and medical encounter. We then supervised fine-tuned a state-of-the-art LLM using public and credentialed clinical data. The training is orchestrated in a way that the target model can first support basic clinical tasks such as abbreviation expansion and temporal information extraction, and then learn to perform more complex downstream clinical tasks such as impression and encounter summary. Moreover, we address, several modeling challenges in the healthcare context, e.g., extra long context window. Our blind pairwise evaluation shows that SoftTiger outperforms other popular open-source models and GPT-3.5, comparable to Gemini-pro, and only has a mild gap from GPT-4. We believe that LLMs may become a step-stone towards healthcare digitalization and democratization. Therefore, we publicly release SoftTiger models at scales of 13 billion and 70 billion parameters, as well as datasets and code for our innovative scalable evaluation, hopefully, making a significant contribution to the healthcare industry.

Title: Teach LLMs to Phish: Stealing Private Information from Language Models

• Authors: Ashwinee Panda, Christopher A. Choquette-Choo, Zhengming Zhang, Yaoqing Yang, Prateek Mittal
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00871
• Pdf URL: https://arxiv.org/pdf/2403.00871
• Copy Paste: [[2403.00871]] Teach LLMs to Phish: Stealing Private Information from Language Models(https://arxiv.org/abs/2403.00871)
• Keywords: privacy, attack, steal, extraction, large language model
• Abstract: When large language models are trained on private data, it can be a significant privacy risk for them to memorize and regurgitate sensitive information. In this work, we propose a new practical data extraction attack that we call "neural phishing". This attack enables an adversary to target and extract sensitive or personally identifiable information (PII), e.g., credit card numbers, from a model trained on user data with upwards of 10% attack success rates, at times, as high as 50%. Our attack assumes only that an adversary can insert as few as 10s of benign-appearing sentences into the training dataset using only vague priors on the structure of the user data.

Title: Blockchain-empowered Federated Learning: Benefits, Challenges, and Solutions

• Authors: Zeju Cai, Jianguo Chen, Yuting Fan, Zibin Zheng, Keqin Li
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00873
• Pdf URL: https://arxiv.org/pdf/2403.00873
• Copy Paste: [[2403.00873]] Blockchain-empowered Federated Learning: Benefits, Challenges, and Solutions(https://arxiv.org/abs/2403.00873)
• Keywords: security, privacy, protect, federate, fair
• Abstract: Federated learning (FL) is a distributed machine learning approach that protects user data privacy by training models locally on clients and aggregating them on a parameter server. While effective at preserving privacy, FL systems face limitations such as single points of failure, lack of incentives, and inadequate security. To address these challenges, blockchain technology is integrated into FL systems to provide stronger security, fairness, and scalability. However, blockchain-empowered FL (BC-FL) systems introduce additional demands on network, computing, and storage resources. This survey provides a comprehensive review of recent research on BC-FL systems, analyzing the benefits and challenges associated with blockchain integration. We explore why blockchain is applicable to FL, how it can be implemented, and the challenges and existing solutions for its integration. Additionally, we offer insights on future research directions for the BC-FL system.

Title: Crimson: Empowering Strategic Reasoning in Cybersecurity through Large Language Models

• Authors: Jiandong Jin, Bowen Tang, Mingxuan Ma, Xiao Liu, Yunfei Wang, Qingnan Lai, Jia Yang, Changling Zhou
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00878
• Pdf URL: https://arxiv.org/pdf/2403.00878
• Copy Paste: [[2403.00878]] Crimson: Empowering Strategic Reasoning in Cybersecurity through Large Language Models(https://arxiv.org/abs/2403.00878)
• Keywords: security, defense, large language model
• Abstract: We introduces Crimson, a system that enhances the strategic reasoning capabilities of Large Language Models (LLMs) within the realm of cybersecurity. By correlating CVEs with MITRE ATT&CK techniques, Crimson advances threat anticipation and strategic defense efforts. Our approach includes defining and evaluating cybersecurity strategic tasks, alongside implementing a comprehensive human-in-the-loop data-synthetic workflow to develop the CVE-to-ATT&CK Mapping (CVEM) dataset. We further enhance LLMs' reasoning abilities through a novel Retrieval-Aware Training (RAT) process and its refined iteration, RAT-R. Our findings demonstrate that an LLM fine-tuned with our techniques, possessing 7 billion parameters, approaches the performance level of GPT-4, showing markedly lower rates of hallucination and errors, and surpassing other models in strategic reasoning tasks. Moreover, domain-specific fine-tuning of embedding models significantly improves performance within cybersecurity contexts, underscoring the efficacy of our methodology. By leveraging Crimson to convert raw vulnerability data into structured and actionable insights, we bolster proactive cybersecurity defenses.

Title: FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission

• Authors: Zeling Zhang, Dongqi Cai, Yiran Zhang, Mengwei Xu, Shangguang Wang, Ao Zhou
• Subjects: cs.LG, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.00881
• Pdf URL: https://arxiv.org/pdf/2403.00881
• Copy Paste: [[2403.00881]] FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission(https://arxiv.org/abs/2403.00881)
• Keywords: robust, federate
• Abstract: Communication overhead is a significant bottleneck in federated learning (FL), which has been exaggerated with the increasing size of AI models. In this paper, we propose FedRDMA, a communication-efficient cross-silo FL system that integrates RDMA into the FL communication protocol. To overcome the limitations of RDMA in wide-area networks (WANs), FedRDMA divides the updated model into chunks and designs a series of optimization techniques to improve the efficiency and robustness of RDMA-based communication. We implement FedRDMA atop the industrial federated learning framework and evaluate it on a real-world cross-silo FL scenario. The experimental results show that \sys can achieve up to 3.8$\times$ speedup in communication efficiency compared to traditional TCP/IP-based FL systems.

Title: Time-bound Contextual Bio-ID Generation for Minimalist Wearables

• Authors: Adiba Orzikulova, Diana A. Vasile, Fahim Kawsar, Chulhong Min
• Subjects: cs.CR, cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.00889
• Pdf URL: https://arxiv.org/pdf/2403.00889
• Copy Paste: [[2403.00889]] Time-bound Contextual Bio-ID Generation for Minimalist Wearables(https://arxiv.org/abs/2403.00889)
• Keywords: security, privacy, robust
• Abstract: As wearable devices become increasingly miniaturized and powerful, a new opportunity arises for instant and dynamic device-to-device collaboration and human-to-device interaction. However, this progress presents a unique challenge: these minimalist wearables lack inherent mechanisms for real-time authentication, posing significant risks to data privacy and overall security. To address this, we introduce Proteus that realizes an innovative concept of time-bound contextual bio-IDs, which are generated from on-device sensor data and embedded into a common latent space. These bio-IDs act as a time-bound unique user identifier that can be used to identify the wearer in a certain context. Proteus enables dynamic and contextual device collaboration as well as robust human-to-device interaction. Our evaluations demonstrate the effectiveness of our method, particularly in the context of minimalist wearables.

Title: Improving Android Malware Detection Through Data Augmentation Using Wasserstein Generative Adversarial Networks

• Authors: Kawana Stalin, Mikias Berhanu Mekoya
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00890
• Pdf URL: https://arxiv.org/pdf/2403.00890
• Copy Paste: [[2403.00890]] Improving Android Malware Detection Through Data Augmentation Using Wasserstein Generative Adversarial Networks(https://arxiv.org/abs/2403.00890)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) have demonstrated their versatility across various applications, including data augmentation and malware detection. This research explores the effectiveness of utilizing GAN-generated data to train a model for the detection of Android malware. Given the considerable storage requirements of Android applications, the study proposes a method to synthetically represent data using GANs, thereby reducing storage demands. The proposed methodology involves creating image representations of features extracted from an existing dataset. A GAN model is then employed to generate a more extensive dataset consisting of realistic synthetic grayscale images. Subsequently, this synthetic dataset is utilized to train a Convolutional Neural Network (CNN) designed to identify previously unseen Android malware applications. The study includes a comparative analysis of the CNN's performance when trained on real images versus synthetic images generated by the GAN. Furthermore, the research explores variations in performance between the Wasserstein Generative Adversarial Network (WGAN) and the Deep Convolutional Generative Adversarial Network (DCGAN). The investigation extends to studying the impact of image size and malware obfuscation on the classification model's effectiveness. The data augmentation approach implemented in this study resulted in a notable performance enhancement of the classification model, ranging from 1.5% to 7%, depending on the dataset. The achieved F1 score reached 97.5%. Keywords--Generative Adversarial Networks, Android Malware, Data Augmentation, Wasserstein Generative Adversarial Network

Title: A Regularization-based Transfer Learning Method for Information Extraction via Instructed Graph Decoder

• Authors: Kedi Chen, Jie Zhou, Qin Chen, Shunyu Liu, Liang He
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00891
• Pdf URL: https://arxiv.org/pdf/2403.00891
• Copy Paste: [[2403.00891]] A Regularization-based Transfer Learning Method for Information Extraction via Instructed Graph Decoder(https://arxiv.org/abs/2403.00891)
• Keywords: extraction
• Abstract: Information extraction (IE) aims to extract complex structured information from the text. Numerous datasets have been constructed for various IE tasks, leading to time-consuming and labor-intensive data annotations. Nevertheless, most prevailing methods focus on training task-specific models, while the common knowledge among different IE tasks is not explicitly modeled. Moreover, the same phrase may have inconsistent labels in different tasks, which poses a big challenge for knowledge transfer using a unified model. In this study, we propose a regularization-based transfer learning method for IE (TIE) via an instructed graph decoder. Specifically, we first construct an instruction pool for datasets from all well-known IE tasks, and then present an instructed graph decoder, which decodes various complex structures into a graph uniformly based on corresponding instructions. In this way, the common knowledge shared with existing datasets can be learned and transferred to a new dataset with new labels. Furthermore, to alleviate the label inconsistency problem among various IE tasks, we introduce a task-specific regularization strategy, which does not update the gradients of two tasks with 'opposite direction'. We conduct extensive experiments on 12 datasets spanning four IE tasks, and the results demonstrate the great advantages of our proposed method

Title: DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Models

• Authors: Kedi Chen, Qin Chen, Jie Zhou, Yishen He, Liang He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00896
• Pdf URL: https://arxiv.org/pdf/2403.00896
• Copy Paste: [[2403.00896]] DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Models(https://arxiv.org/abs/2403.00896)
• Keywords: large language model
• Abstract: Since large language models (LLMs) achieve significant success in recent years, the hallucination issue remains a challenge, numerous benchmarks are proposed to detect the hallucination. Nevertheless, some of these benchmarks are not naturally generated by LLMs but are intentionally induced. Also, many merely focus on the factuality hallucination while ignoring the faithfulness hallucination. Additionally, although dialogue pattern is more widely utilized in the era of LLMs, current benchmarks only concentrate on sentence-level and passage-level hallucination. In this study, we propose DiaHalu, the first dialogue-level hallucination evaluation benchmark to our knowledge. Initially, we integrate the collected topics into system prompts and facilitate a dialogue between two ChatGPT3.5. Subsequently, we manually modify the contents that do not adhere to human language conventions and then have LLMs re-generate, simulating authentic human-machine interaction scenarios. Finally, professional scholars annotate all the samples in the dataset. DiaHalu covers four common multi-turn dialogue domains and five hallucination subtypes, extended from factuality and faithfulness hallucination. Experiments through some well-known LLMs and detection methods on the dataset show that DiaHalu is a challenging benchmark, holding significant value for further research.

Title: Differentially Private Knowledge Distillation via Synthetic Text Generation

• Authors: James Flemings, Murali Annavaram
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.00932
• Pdf URL: https://arxiv.org/pdf/2403.00932
• Copy Paste: [[2403.00932]] Differentially Private Knowledge Distillation via Synthetic Text Generation(https://arxiv.org/abs/2403.00932)
• Keywords: privacy, large language model
• Abstract: Large Language models (LLMs) are achieving state-of-the-art performance in many different downstream tasks. However, the increasing urgency of data privacy requires LLMs to train with Differential Privacy (DP) on private data. Concurrently it is also necessary to compress LLMs for real-life deployments on resource-constrained devices or latency-sensitive applications. Differential privacy and model compression generally must trade off utility loss to achieve their objectives. Moreover, concurrently achieving both can result in even more utility loss. To this end, we propose a novel differentially private knowledge distillation algorithm that exploits synthetic data generated by a differentially private LLM. The knowledge of a teacher model is transferred onto the student in two ways: one way from the synthetic data itself, the hard labels, and the other way by the output distribution of the teacher model evaluated on the synthetic data, the soft labels. Furthermore, if the teacher and student share a similar architectural structure, we can further distill knowledge by exploiting hidden representations. Our results show that our framework substantially improves the utility over existing baselines with strong privacy parameters, {\epsilon} = 2, validating that we can successfully compress autoregressive LLMs while preserving the privacy of training data.

Title: Transfer Learning for Security: Challenges and Future Directions

• Authors: Adrian Shuai Li, Arun Iyengar, Ashish Kundu, Elisa Bertino
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00935
• Pdf URL: https://arxiv.org/pdf/2403.00935
• Copy Paste: [[2403.00935]] Transfer Learning for Security: Challenges and Future Directions(https://arxiv.org/abs/2403.00935)
• Keywords: security
• Abstract: Many machine learning and data mining algorithms rely on the assumption that the training and testing data share the same feature space and distribution. However, this assumption may not always hold. For instance, there are situations where we need to classify data in one domain, but we only have sufficient training data available from a different domain. The latter data may follow a distinct distribution. In such cases, successfully transferring knowledge across domains can significantly improve learning performance and reduce the need for extensive data labeling efforts. Transfer learning (TL) has thus emerged as a promising framework to tackle this challenge, particularly in security-related tasks. This paper aims to review the current advancements in utilizing TL techniques for security. The paper includes a discussion of the existing research gaps in applying TL in the security domain, as well as exploring potential future research directions and issues that arise in the context of TL-assisted security solutions.

Title: G3DR: Generative 3D Reconstruction in ImageNet

• Authors: Pradyumna Reddy, Ismail Elezi, Jiankang Deng
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.00939
• Pdf URL: https://arxiv.org/pdf/2403.00939
• Copy Paste: [[2403.00939]] G3DR: Generative 3D Reconstruction in ImageNet(https://arxiv.org/abs/2403.00939)
• Keywords: generative
• Abstract: We introduce a novel 3D generative method, Generative 3D Reconstruction (G3DR) in ImageNet, capable of generating diverse and high-quality 3D objects from single images, addressing the limitations of existing methods. At the heart of our framework is a novel depth regularization technique that enables the generation of scenes with high-geometric fidelity. G3DR also leverages a pretrained language-vision model, such as CLIP, to enable reconstruction in novel views and improve the visual realism of generations. Additionally, G3DR designs a simple but effective sampling procedure to further improve the quality of generations. G3DR offers diverse and efficient 3D asset generation based on class or text conditioning. Despite its simplicity, G3DR is able to beat state-of-theart methods, improving over them by up to 22% in perceptual metrics and 90% in geometry scores, while needing only half of the training time. Code is available at https://github.com/preddy5/G3DR

Title: Resilience of Entropy Model in Distributed Neural Networks

• Authors: Milin Zhang, Mohammad Abdi, Shahriar Rifat, Francesco Restuccia
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.00942
• Pdf URL: https://arxiv.org/pdf/2403.00942
• Copy Paste: [[2403.00942]] Resilience of Entropy Model in Distributed Neural Networks(https://arxiv.org/abs/2403.00942)
• Keywords: defense, attack, robust
• Abstract: Distributed deep neural networks (DNNs) have emerged as a key technique to reduce communication overhead without sacrificing performance in edge computing systems. Recently, entropy coding has been introduced to further reduce the communication overhead. The key idea is to train the distributed DNN jointly with an entropy model, which is used as side information during inference time to adaptively encode latent representations into bit streams with variable length. To the best of our knowledge, the resilience of entropy models is yet to be investigated. As such, in this paper we formulate and investigate the resilience of entropy models to intentional interference (e.g., adversarial attacks) and unintentional interference (e.g., weather changes and motion blur). Through an extensive experimental campaign with 3 different DNN architectures, 2 entropy models and 4 rate-distortion trade-off factors, we demonstrate that the entropy attacks can increase the communication overhead by up to 95%. By separating compression features in frequency and spatial domain, we propose a new defense mechanism that can reduce the transmission overhead of the attacked input by about 9% compared to unperturbed data, with only about 2% accuracy loss. Importantly, the proposed defense mechanism is a standalone approach which can be applied in conjunction with approaches such as adversarial training to further improve robustness. Code will be shared for reproducibility.

Title: MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

• Authors: Vithursan Thangarasa, Mahmoud Salem, Shreyas Saxena, Kevin Leong, Joel Hestness, Sean Lie
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00952
• Pdf URL: https://arxiv.org/pdf/2403.00952
• Copy Paste: [[2403.00952]] MediSwift: Efficient Sparse Pre-trained Biomedical Language Models(https://arxiv.org/abs/2403.00952)
• Keywords: large language model
• Abstract: Large language models (LLMs) are typically trained on general source data for various domains, but a recent surge in domain-specific LLMs has shown their potential to outperform general-purpose models in domain-specific tasks (e.g., biomedicine). Although domain-specific pre-training enhances efficiency and leads to smaller models, the computational costs of training these LLMs remain high, posing budgeting challenges. We introduce MediSwift, a suite of biomedical LMs that leverage sparse pre-training on domain-specific biomedical text data. By inducing up to 75% weight sparsity during the pre-training phase, MediSwift achieves a 2-2.5x reduction in training FLOPs. Notably, all sparse pre-training was performed on the Cerebras CS-2 system, which is specifically designed to realize the acceleration benefits from unstructured weight sparsity, thereby significantly enhancing the efficiency of the MediSwift models. Through subsequent dense fine-tuning and strategic soft prompting, MediSwift models outperform existing LLMs up to 7B parameters on biomedical tasks, setting new benchmarks w.r.t efficiency-accuracy on tasks such as PubMedQA. Our results show that sparse pre-training, along with dense fine-tuning and soft prompting, offers an effective method for creating high-performing, computationally efficient models in specialized domains.

Title: AutoRD: An Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontologies-enhanced Large Language Models

• Authors: Lang Cao, Jimeng Sun, Adam Cross
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00953
• Pdf URL: https://arxiv.org/pdf/2403.00953
• Copy Paste: [[2403.00953]] AutoRD: An Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontologies-enhanced Large Language Models(https://arxiv.org/abs/2403.00953)
• Keywords: robust, extraction, large language model
• Abstract: Objectives: Our objective is to create an end-to-end system called AutoRD, which automates extracting information from clinical text about rare diseases. We have conducted various tests to evaluate the performance of AutoRD and highlighted its strengths and limitations in this paper. Materials and Methods: Our system, AutoRD, is a software pipeline involving data preprocessing, entity extraction, relation extraction, entity calibration, and knowledge graph construction. We implement this using large language models and medical knowledge graphs developed from open-source medical ontologies. We quantitatively evaluate our system on entity extraction, relation extraction, and the performance of knowledge graph construction. Results: AutoRD achieves an overall F1 score of 47.3%, a 14.4% improvement compared to the base LLM. In detail, AutoRD achieves an overall entity extraction F1 score of 56.1% (rare_disease: 83.5%, disease: 35.8%, symptom_and_sign: 46.1%, anaphor: 67.5%) and an overall relation extraction F1 score of 38.6% (produces: 34.7%, increases_risk_of: 12.4%, is_a: 37.4%, is_acronym: 44.1%, is_synonym: 16.3%, anaphora: 57.5%). Our qualitative experiment also demonstrates that the performance in constructing the knowledge graph is commendable. Discussion: AutoRD demonstrates the potential of LLM applications in rare disease detection. This improvement is attributed to several design, including the integration of ontologies-enhanced LLMs. Conclusion: AutoRD is an automated end-to-end system for extracting rare disease information from text to build knowledge graphs. It uses ontologies-enhanced LLMs for a robust medical knowledge base. The superior performance of AutoRD is validated by experimental evaluations, demonstrating the potential of LLMs in healthcare.

Title: Tree-Regularized Tabular Embeddings

• Authors: Xuan Li, Yun Wang, Bo Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00963
• Pdf URL: https://arxiv.org/pdf/2403.00963
• Copy Paste: [[2403.00963]] Tree-Regularized Tabular Embeddings(https://arxiv.org/abs/2403.00963)
• Keywords: robust
• Abstract: Tabular neural network (NN) has attracted remarkable attentions and its recent advances have gradually narrowed the performance gap with respect to tree-based models on many public datasets. While the mainstreams focus on calibrating NN to fit tabular data, we emphasize the importance of homogeneous embeddings and alternately concentrate on regularizing tabular inputs through supervised pretraining. Specifically, we extend a recent work (DeepTLF) and utilize the structure of pretrained tree ensembles to transform raw variables into a single vector (T2V), or an array of tokens (T2T). Without loss of space efficiency, these binarized embeddings can be consumed by canonical tabular NN with fully-connected or attention-based building blocks. Through quantitative experiments on 88 OpenML datasets with binary classification task, we validated that the proposed tree-regularized representation not only tapers the difference with respect to tree-based models, but also achieves on-par and better performance when compared with advanced NN models. Most importantly, it possesses better robustness and can be easily scaled and generalized as standalone encoder for tabular modality. Codes: https://github.com/milanlx/tree-regularized-embedding.

Title: MALTO at SemEval-2024 Task 6: Leveraging Synthetic Data for LLM Hallucination Detection

• Authors: Federico Borra, Claudio Savelli, Giacomo Rosso, Alkis Koudounas, Flavio Giobergia
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00964
• Pdf URL: https://arxiv.org/pdf/2403.00964
• Copy Paste: [[2403.00964]] MALTO at SemEval-2024 Task 6: Leveraging Synthetic Data for LLM Hallucination Detection(https://arxiv.org/abs/2403.00964)
• Keywords: large language model
• Abstract: In Natural Language Generation (NLG), contemporary Large Language Models (LLMs) face several challenges, such as generating fluent yet inaccurate outputs and reliance on fluency-centric metrics. This often leads to neural networks exhibiting "hallucinations". The SHROOM challenge focuses on automatically identifying these hallucinations in the generated text. To tackle these issues, we introduce two key components, a data augmentation pipeline incorporating LLM-assisted pseudo-labelling and sentence rephrasing, and a voting ensemble from three models pre-trained on Natural Language Inference (NLI) tasks and fine-tuned on diverse datasets.

Title: LocalRQA: From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systems

• Authors: Xiao Yu, Yunan Lu, Zhou Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00982
• Pdf URL: https://arxiv.org/pdf/2403.00982
• Copy Paste: [[2403.00982]] LocalRQA: From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systems(https://arxiv.org/abs/2403.00982)
• Keywords: fair, large language model
• Abstract: Retrieval-augmented question-answering systems combine retrieval techniques with large language models to provide answers that are more accurate and informative. Many existing toolkits allow users to quickly build such systems using off-the-shelf models, but they fall short in supporting researchers and developers to customize the model training, testing, and deployment process. We propose LocalRQA, an open-source toolkit that features a wide selection of model training algorithms, evaluation methods, and deployment tools curated from the latest research. As a showcase, we build QA systems using online documentation obtained from Databricks and Faire's websites. We find 7B-models trained and deployed using LocalRQA reach a similar performance compared to using OpenAI's text-ada-002 and GPT-4-turbo.

Title: Merging Text Transformer Models from Different Initializations

• Authors: Neha Verma, Maha Elbayad
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00986
• Pdf URL: https://arxiv.org/pdf/2403.00986
• Copy Paste: [[2403.00986]] Merging Text Transformer Models from Different Initializations(https://arxiv.org/abs/2403.00986)
• Keywords: transformer
• Abstract: Recent work on one-shot permutation-based model merging has shown impressive low- or zero-barrier mode connectivity between models from completely different initializations. However, this line of work has not yet extended to the Transformer architecture, despite its dominant popularity in the language domain. Therefore, in this work, we investigate the extent to which separate Transformer minima learn similar features, and propose a model merging technique to investigate the relationship between these minima in the loss landscape. The specifics of the architecture, like its residual connections, multi-headed attention, and discrete, sequential input, require specific interventions in order to compute model permutations that remain within the same functional equivalence class. In merging these models with our method, we consistently find lower loss barriers between minima compared to model averaging for several models trained on a masked-language modeling task or fine-tuned on a language understanding benchmark. Our results show that the minima of these models are less sharp and isolated than previously understood, and provide a basis for future work on merging separately trained Transformer models.

Title: Formulation Comparison for Timeline Construction using LLMs

• Authors: Kimihiro Hasegawa, Nikhil Kandukuri, Susan Holm, Yukari Yamakawa, Teruko Mitamura
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00990
• Pdf URL: https://arxiv.org/pdf/2403.00990
• Copy Paste: [[2403.00990]] Formulation Comparison for Timeline Construction using LLMs(https://arxiv.org/abs/2403.00990)
• Keywords: robust
• Abstract: Constructing a timeline requires identifying the chronological order of events in an article. In prior timeline construction datasets, temporal orders are typically annotated by either event-to-time anchoring or event-to-event pairwise ordering, both of which suffer from missing temporal information. To mitigate the issue, we develop a new evaluation dataset, TimeSET, consisting of single-document timelines with document-level order annotation. TimeSET features saliency-based event selection and partial ordering, which enable a practical annotation workload. Aiming to build better automatic timeline construction systems, we propose a novel evaluation framework to compare multiple task formulations with TimeSET by prompting open LLMs, i.e., Llama 2 and Flan-T5. Considering that identifying temporal orders of events is a core subtask in timeline construction, we further benchmark open LLMs on existing event temporal ordering datasets to gain a robust understanding of their capabilities. Our experiments show that (1) NLI formulation with Flan-T5 demonstrates a strong performance among others, while (2) timeline construction and event temporal ordering are still challenging tasks for few-shot LLMs. Our code and data are available at https://github.com/kimihiroh/timeset.

Title: Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods

• Authors: Polina Tsvilodub, Hening Wang, Sharon Grosch, Michael Franke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00998
• Pdf URL: https://arxiv.org/pdf/2403.00998
• Copy Paste: [[2403.00998]] Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods(https://arxiv.org/abs/2403.00998)
• Keywords: secure, robust
• Abstract: This paper systematically compares different methods of deriving item-level predictions of language models for multiple-choice tasks. It compares scoring methods for answer options based on free generation of responses, various probability-based scores, a Likert-scale style rating method, and embedding similarity. In a case study on pragmatic language interpretation, we find that LLM predictions are not robust under variation of method choice, both within a single LLM and across different LLMs. As this variability entails pronounced researcher degrees of freedom in reporting results, knowledge of the variability is crucial to secure robustness of results and research integrity.

Title: Distributional Dataset Distillation with Subtask Decomposition

• Authors: Tian Qin, Zhiwei Deng, David Alvarez-Melis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00999
• Pdf URL: https://arxiv.org/pdf/2403.00999
• Copy Paste: [[2403.00999]] Distributional Dataset Distillation with Subtask Decomposition(https://arxiv.org/abs/2403.00999)
• Keywords: federate
• Abstract: What does a neural network learn when training from a task-specific dataset? Synthesizing this knowledge is the central idea behind Dataset Distillation, which recent work has shown can be used to compress large datasets into a small set of input-label pairs ($\textit{prototypes}$) that capture essential aspects of the original dataset. In this paper, we make the key observation that existing methods distilling into explicit prototypes are very often suboptimal, incurring in unexpected storage cost from distilled labels. In response, we propose $\textit{Distributional Dataset Distillation}$ (D3), which encodes the data using minimal sufficient per-class statistics and paired with a decoder, we distill dataset into a compact distributional representation that is more memory-efficient compared to prototype-based methods. To scale up the process of learning these representations, we propose $\textit{Federated distillation}$, which decomposes the dataset into subsets, distills them in parallel using sub-task experts and then re-aggregates them. We thoroughly evaluate our algorithm on a three-dimensional metric and show that our method achieves state-of-the-art results on TinyImageNet and ImageNet-1K. Specifically, we outperform the prior art by $6.9\%$ on ImageNet-1K under the storage budget of 2 images per class.

Title: Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summaries

• Authors: Zelalem Gero, Chandan Singh, Yiqing Xie, Sheng Zhang, Tristan Naumann, Jianfeng Gao, Hoifung Poon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01002
• Pdf URL: https://arxiv.org/pdf/2403.01002
• Copy Paste: [[2403.01002]] Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summaries(https://arxiv.org/abs/2403.01002)
• Keywords: large language model
• Abstract: Summarizing clinical text is crucial in health decision-support and clinical research. Large language models (LLMs) have shown the potential to generate accurate clinical text summaries, but still struggle with issues regarding grounding and evaluation, especially in safety-critical domains such as health. Holistically evaluating text summaries is challenging because they may contain unsubstantiated information. Here, we explore a general mitigation framework using Attribute Structuring (AS), which structures the summary evaluation process. It decomposes the evaluation process into a grounded procedure that uses an LLM for relatively simple structuring and scoring tasks, rather than the full task of holistic summary evaluation. Experiments show that AS consistently improves the correspondence between human annotations and automated metrics in clinical text summarization. Additionally, AS yields interpretations in the form of a short text span corresponding to each output, which enables efficient human auditing, paving the way towards trustworthy evaluation of clinical information in resource-constrained scenarios. We release our code, prompts, and an open-source benchmark at https://github.com/microsoft/attribute-structuring.

Title: BasedAI: A decentralized P2P network for Zero Knowledge Large Language Models (ZK-LLMs)

• Authors: Sean Wellington
• Subjects: cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.01008
• Pdf URL: https://arxiv.org/pdf/2403.01008
• Copy Paste: [[2403.01008]] BasedAI: A decentralized P2P network for Zero Knowledge Large Language Models (ZK-LLMs)(https://arxiv.org/abs/2403.01008)
• Keywords: privacy, generative, large language model
• Abstract: BasedAI is a distributed network of machines which introduces decentralized infrastructure capable of integrating Fully Homomorphic Encryption (FHE) with any large language model (LLM) connected to its network. The proposed framework embeds a default mechanism, called "Cerberus Squeezing", into the mining process which enables the transformation of a standard LLMs into encrypted zero-knowledge LLMs, or "ZK-LLMs", leveraging insights from generative adversarial networks for data privacy. This novel quantization mechanism empowers BasedAI miners to process and respond to prompts derived from User interaction with LLMs without the need for decrypting ei- ther the queries or their corresponding responses. The introduction of Cerberus Squeezing significantly improves performance degradation caused by quantized functions in current FHE-compliant computing environments by proactively optimizing calls between users, miners, and validators.

Title: Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks

• Authors: Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, Muhammad Abdul-Mageed
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01031
• Pdf URL: https://arxiv.org/pdf/2403.01031
• Copy Paste: [[2403.01031]] Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks(https://arxiv.org/abs/2403.01031)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) have proven effective in a wide range of tasks requiring complex reasoning and linguistic comprehension. However, due to a lack of high-quality multimodal resources in languages other than English, success of MLLMs remains relatively limited to English-based settings. This poses significant challenges in developing comparable models for other languages, including even those with large speaker populations such as Arabic. To alleviate this challenge, we introduce a comprehensive family of Arabic MLLMs, dubbed \textit{Peacock}, with strong vision and language capabilities. Through comprehensive qualitative and quantitative analysis, we demonstrate the solid performance of our models on various visual reasoning tasks and further show their emerging dialectal potential. Additionally, we introduce ~\textit{Henna}, a new benchmark specifically designed for assessing MLLMs on aspects related to Arabic culture, setting the first stone for culturally-aware Arabic MLLMs.The GitHub repository for the \textit{Peacock} project is available at \url{https://github.com/UBC-NLP/peacock}.

Title: AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks

• Authors: Jiacen Xu, Jack W. Stokes, Geoff McDonald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swaminathan, Zhou Li
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01038
• Pdf URL: https://arxiv.org/pdf/2403.01038
• Copy Paste: [[2403.01038]] AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks(https://arxiv.org/abs/2403.01038)
• Keywords: security, attack, large language model
• Abstract: Large language models (LLMs) have demonstrated impressive results on natural language tasks, and security researchers are beginning to employ them in both offensive and defensive systems. In cyber-security, there have been multiple research efforts that utilize LLMs focusing on the pre-breach stage of attacks like phishing and malware generation. However, so far there lacks a comprehensive study regarding whether LLM-based systems can be leveraged to simulate the post-breach stage of attacks that are typically human-operated, or "hands-on-keyboard" attacks, under various attack techniques and environments. As LLMs inevitably advance, they may be able to automate both the pre- and post-breach attack stages. This shift may transform organizational attacks from rare, expert-led events to frequent, automated operations requiring no expertise and executed at automation speed and scale. This risks fundamentally changing global computer security and correspondingly causing substantial economic impacts, and a goal of this work is to better understand these risks now so we can better prepare for these inevitable ever-more-capable LLMs on the horizon. On the immediate impact side, this research serves three purposes. First, an automated LLM-based, post-breach exploitation framework can help analysts quickly test and continually improve their organization's network security posture against previously unseen attacks. Second, an LLM-based penetration test system can extend the effectiveness of red teams with a limited number of human analysts. Finally, this research can help defensive systems and teams learn to detect novel attack behaviors preemptively before their use in the wild....

Title: Attacking the Diebold Signature Variant -- RSA Signatures with Unverified High-order Padding

• Authors: Ryan W. Gardner, Tadayoshi Kohno, Alec Yasinsac
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01048
• Pdf URL: https://arxiv.org/pdf/2403.01048
• Copy Paste: [[2403.01048]] Attacking the Diebold Signature Variant -- RSA Signatures with Unverified High-order Padding(https://arxiv.org/abs/2403.01048)
• Keywords: attack
• Abstract: We examine a natural but improper implementation of RSA signature verification deployed on the widely used Diebold Touch Screen and Optical Scan voting machines. In the implemented scheme, the verifier fails to examine a large number of the high-order bits of signature padding and the public exponent is three. We present an very mathematically simple attack that enables an adversary to forge signatures on arbitrary messages in a negligible amount of time.

Title: Neural Field Classifiers via Target Encoding and Classification Loss

• Authors: Xindi Yang, Zeke Xie, Xiong Zhou, Boyu Liu, Buhua Liu, Yi Liu, Haoran Wang, Yunfeng Cai, Mingming Sun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01058
• Pdf URL: https://arxiv.org/pdf/2403.01058
• Copy Paste: [[2403.01058]] Neural Field Classifiers via Target Encoding and Classification Loss(https://arxiv.org/abs/2403.01058)
• Keywords: robust
• Abstract: Neural field methods have seen great progress in various long-standing tasks in computer vision and computer graphics, including novel view synthesis and geometry reconstruction. As existing neural field methods try to predict some coordinate-based continuous target values, such as RGB for Neural Radiance Field (NeRF), all of these methods are regression models and are optimized by some regression loss. However, are regression models really better than classification models for neural field methods? In this work, we try to visit this very fundamental but overlooked question for neural fields from a machine learning perspective. We successfully propose a novel Neural Field Classifier (NFC) framework which formulates existing neural field methods as classification tasks rather than regression tasks. The proposed NFC can easily transform arbitrary Neural Field Regressor (NFR) into its classification variant via employing a novel Target Encoding module and optimizing a classification loss. By encoding a continuous regression target into a high-dimensional discrete encoding, we naturally formulate a multi-label classification task. Extensive experiments demonstrate the impressive effectiveness of NFC at the nearly free extra computational costs. Moreover, NFC also shows robustness to sparse inputs, corrupted images, and dynamic scenes.

Title: Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers

• Authors: Melanie Subbiah, Sean Zhang, Lydia B. Chilton, Kathleen McKeown
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01061
• Pdf URL: https://arxiv.org/pdf/2403.01061
• Copy Paste: [[2403.01061]] Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers(https://arxiv.org/abs/2403.01061)
• Keywords: large language model
• Abstract: We evaluate recent Large language Models (LLMs) on the challenging task of summarizing short stories, which can be lengthy, and include nuanced subtext or scrambled timelines. Importantly, we work directly with authors to ensure that the stories have not been shared online (and therefore are unseen by the models), and to obtain informed evaluations of summary quality using judgments from the authors themselves. Through quantitative and qualitative analysis grounded in narrative theory, we compare GPT-4, Claude-2.1, and LLama-2-70B. We find that all three models make faithfulness mistakes in over 50% of summaries and struggle to interpret difficult subtext. However, at their best, the models can provide thoughtful thematic analysis of stories. We additionally demonstrate that LLM judgments of summary quality do not match the feedback from the writers.

Title: FaiMA: Feature-aware In-context Learning for Multi-domain Aspect-based Sentiment Analysis

• Authors: Songhua Yang, Xinke Jiang, Hanjie Zhao, Wenxuan Zeng, Hongde Liu, Yuxiang Jia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01063
• Pdf URL: https://arxiv.org/pdf/2403.01063
• Copy Paste: [[2403.01063]] FaiMA: Feature-aware In-context Learning for Multi-domain Aspect-based Sentiment Analysis(https://arxiv.org/abs/2403.01063)
• Keywords: large language model
• Abstract: Multi-domain aspect-based sentiment analysis (ABSA) seeks to capture fine-grained sentiment across diverse domains. While existing research narrowly focuses on single-domain applications constrained by methodological limitations and data scarcity, the reality is that sentiment naturally traverses multiple domains. Although large language models (LLMs) offer a promising solution for ABSA, it is difficult to integrate effectively with established techniques, including graph-based models and linguistics, because modifying their internal architecture is not easy. To alleviate this problem, we propose a novel framework, Feature-aware In-context Learning for Multi-domain ABSA (FaiMA). The core insight of FaiMA is to utilize in-context learning (ICL) as a feature-aware mechanism that facilitates adaptive learning in multi-domain ABSA tasks. Specifically, we employ a multi-head graph attention network as a text encoder optimized by heuristic rules for linguistic, domain, and sentiment features. Through contrastive learning, we optimize sentence representations by focusing on these diverse features. Additionally, we construct an efficient indexing mechanism, allowing FaiMA to stably retrieve highly relevant examples across multiple dimensions for any given input. To evaluate the efficacy of FaiMA, we build the first multi-domain ABSA benchmark dataset. Extensive experimental results demonstrate that FaiMA achieves significant performance improvements in multiple domains compared to baselines, increasing F1 by 2.07% on average. Source code and data sets are anonymously available at https://github.com/SupritYoung/FaiMA.

Title: LLMCRIT: Teaching Large Language Models to Use Criteria

• Authors: Weizhe Yuan, Pengfei Liu, Matthias Gallé
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01069
• Pdf URL: https://arxiv.org/pdf/2403.01069
• Copy Paste: [[2403.01069]] LLMCRIT: Teaching Large Language Models to Use Criteria(https://arxiv.org/abs/2403.01069)
• Keywords: large language model
• Abstract: Humans follow criteria when they execute tasks, and these criteria are directly used to assess the quality of task completion. Therefore, having models learn to use criteria to provide feedback can help humans or models to perform tasks better. However, existing research in this field tends to consider only a limited set of criteria or quality assessment aspects. To fill this gap, we propose a general framework that enables large language models (LLMs) to use comprehensive criteria for a task in delivering natural language feedback on task execution. In particular, we present a model-in-the-loop framework that semi-automatically derives criteria from collected guidelines for different writing tasks and constructs in-context demonstrations for each criterion. We choose three tasks from real-world scenarios to operationalize this idea: paper introduction writing, Python code writing, and Reddit post writing, and evaluate our feedback generation framework using different LLMs. The results reveal the fine-grained effects of incorporating criteria and demonstrations and provide valuable insights on how to teach LLMs to use criteria more effectively.

Title: $Γ$-VAE: Curvature regularized variational autoencoders for uncovering emergent low dimensional geometric structure in high dimensional data

• Authors: Jason Z. Kim, Nicolas Perrin-Gilbert, Erkan Narmanli, Paul Klein, Christopher R. Myers, Itai Cohen, Joshua J. Waterfall, James P. Sethna
• Subjects: cs.LG, cs.AI, physics.bio-ph, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2403.01078
• Pdf URL: https://arxiv.org/pdf/2403.01078
• Copy Paste: [[2403.01078]] $Γ$-VAE: Curvature regularized variational autoencoders for uncovering emergent low dimensional geometric structure in high dimensional data(https://arxiv.org/abs/2403.01078)
• Keywords: interpretability, generative
• Abstract: Natural systems with emergent behaviors often organize along low-dimensional subsets of high-dimensional spaces. For example, despite the tens of thousands of genes in the human genome, the principled study of genomics is fruitful because biological processes rely on coordinated organization that results in lower dimensional phenotypes. To uncover this organization, many nonlinear dimensionality reduction techniques have successfully embedded high-dimensional data into low-dimensional spaces by preserving local similarities between data points. However, the nonlinearities in these methods allow for too much curvature to preserve general trends across multiple non-neighboring data clusters, thereby limiting their interpretability and generalizability to out-of-distribution data. Here, we address both of these limitations by regularizing the curvature of manifolds generated by variational autoencoders, a process we coin ``$\Gamma$-VAE''. We demonstrate its utility using two example data sets: bulk RNA-seq from the The Cancer Genome Atlas (TCGA) and the Genotype Tissue Expression (GTEx); and single cell RNA-seq from a lineage tracing experiment in hematopoietic stem cell differentiation. We find that the resulting regularized manifolds identify mesoscale structure associated with different cancer cell types, and accurately re-embed tissues from completely unseen, out-of distribution cancers as if they were originally trained on them. Finally, we show that preserving long-range relationships to differentiated cells separates undifferentiated cells -- which have not yet specialized -- according to their eventual fate. Broadly, we anticipate that regularizing the curvature of generative models will enable more consistent, predictive, and generalizable models in any high-dimensional system with emergent low-dimensional behavior.

Title: Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework

• Authors: Junxian Li, Bin Shi, Erfei Cui, Hua Wei, Qinghua Zheng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01079
• Pdf URL: https://arxiv.org/pdf/2403.01079
• Copy Paste: [[2403.01079]] Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework(https://arxiv.org/abs/2403.01079)
• Keywords: robust
• Abstract: We study the challenging problem for inference tasks on large-scale graph datasets of Graph Neural Networks: huge time and memory consumption, and try to overcome it by reducing reliance on graph structure. Even though distilling graph knowledge to student MLP is an excellent idea, it faces two major problems of positional information loss and low generalization. To solve the problems, we propose a new three-stage multitask distillation framework. In detail, we use Positional Encoding to capture positional information. Also, we introduce Neural Heat Kernels responsible for graph data processing in GNN and utilize hidden layer outputs matching for better performance of student MLP's hidden layers. To the best of our knowledge, it is the first work to include hidden layer distillation for student MLP on graphs and to combine graph Positional Encoding with MLP. We test its performance and robustness with several settings and draw the conclusion that our work can outperform well with good stability.

Title: LAB: Large-Scale Alignment for ChatBots

• Authors: Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D. Cox, Akash Srivastava
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01081
• Pdf URL: https://arxiv.org/pdf/2403.01081
• Copy Paste: [[2403.01081]] LAB: Large-Scale Alignment for ChatBots(https://arxiv.org/abs/2403.01081)
• Keywords: large language model
• Abstract: This work introduces LAB (Large-scale Alignment for chatBots), a novel methodology designed to overcome the scalability challenges in the instruction-tuning phase of large language model (LLM) training. Leveraging a taxonomy-guided synthetic data generation process and a multi-phase tuning framework, LAB significantly reduces reliance on expensive human annotations and proprietary models like GPT-4. We demonstrate that LAB-trained models can achieve competitive performance across several benchmarks compared to models trained with traditional human-annotated or GPT-4 generated synthetic data. Thus offering a scalable, cost-effective solution for enhancing LLM capabilities and instruction-following behaviors without the drawbacks of catastrophic forgetting, marking a step forward in the efficient training of LLMs for a wide range of applications.

Title: Adaptive Security in 6G for Sustainable Healthcare

• Authors: Ijaz Ahmad, Ijaz Ahmad, Erkki Harjula
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01100
• Pdf URL: https://arxiv.org/pdf/2403.01100
• Copy Paste: [[2403.01100]] Adaptive Security in 6G for Sustainable Healthcare(https://arxiv.org/abs/2403.01100)
• Keywords: secure, security
• Abstract: 6G will fulfill the requirements of future digital healthcare systems through emerging decentralized computing and secure communications technologies. Digital healthcare solutions employ numerous low-power and resource-constrained connected things, such as the Internet of Medical Things (IoMT). However, the current digital healthcare solutions will face two major challenges. First, the proposed solutions are based on the traditional IoT-Cloud model that will experience latency and reliability challenges to meet the expectations and requirements of digital healthcare, while potentially inflicting heavy network load. Second, the existing digital healthcare solutions will face security challenges due to the inherent limitations of IoMT caused by the lack of resources for proper security in those devices. Therefore, in this research, we present a decentralized adaptive security architecture for the successful deployment of digital healthcare. The proposed architecture leverages the edge-cloud continuum to meet the performance, efficiency, and reliability requirements. It can adapt the security solution at run-time to meet the limited capacity of IoMT devices without compromising the security of critical data. Finally, the research outlines comprehensive methodologies for validating the proposed security architecture.

Title: Distilling Text Style Transfer With Self-Explanation From LLMs

• Authors: Chiyu Zhang, Honglong Cai, Yuezhang (Music)Li, Yuexin Wu, Le Hou, Muhammad Abdul-Mageed
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01106
• Pdf URL: https://arxiv.org/pdf/2403.01106
• Copy Paste: [[2403.01106]] Distilling Text Style Transfer With Self-Explanation From LLMs(https://arxiv.org/abs/2403.01106)
• Keywords: large language model
• Abstract: Text Style Transfer (TST) seeks to alter the style of text while retaining its core content. Given the constraints of limited parallel datasets for TST, we propose CoTeX, a framework that leverages large language models (LLMs) alongside chain-of-thought (CoT) prompting to facilitate TST. CoTeX distills the complex rewriting and reasoning capabilities of LLMs into more streamlined models capable of working with both non-parallel and parallel data. Through experimentation across four TST datasets, CoTeX is shown to surpass traditional supervised fine-tuning and knowledge distillation methods, particularly in low-resource settings. We conduct a comprehensive evaluation, comparing CoTeX against current unsupervised, supervised, in-context learning (ICL) techniques, and instruction-tuned LLMs. Furthermore, CoTeX distinguishes itself by offering transparent explanations for its style transfer process.

Title: Face Swap via Diffusion Model

• Authors: Feifei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01108
• Pdf URL: https://arxiv.org/pdf/2403.01108
• Copy Paste: [[2403.01108]] Face Swap via Diffusion Model(https://arxiv.org/abs/2403.01108)
• Keywords: diffusion
• Abstract: This technical report presents a diffusion model based framework for face swapping between two portrait images. The basic framework consists of three components, i.e., IP-Adapter, ControlNet, and Stable Diffusion's inpainting pipeline, for face feature encoding, multi-conditional generation, and face inpainting respectively. Besides, I introduce facial guidance optimization and CodeFormer based blending to further improve the generation quality. Specifically, we engage a recent light-weighted customization method (i.e., DreamBooth-LoRA), to guarantee the identity consistency by 1) using a rare identifier "sks" to represent the source identity, and 2) injecting the image features of source portrait into each cross-attention layer like the text features. Then I resort to the strong inpainting ability of Stable Diffusion, and utilize canny image and face detection annotation of the target portrait as the conditions, to guide ContorlNet's generation and align source portrait with the target portrait. To further correct face alignment, we add the facial guidance loss to optimize the text embedding during the sample generation.

Title: Adversarial Testing for Visual Grounding via Image-Aware Property Reduction

• Authors: Zhiyuan Chang, Mingyang Li, Junjie Wang, Cheng Li, Boyu Wu, Fanjiang Xu, Qing Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01118
• Pdf URL: https://arxiv.org/pdf/2403.01118
• Copy Paste: [[2403.01118]] Adversarial Testing for Visual Grounding via Image-Aware Property Reduction(https://arxiv.org/abs/2403.01118)
• Keywords: extraction
• Abstract: Due to the advantages of fusing information from various modalities, multimodal learning is gaining increasing attention. Being a fundamental task of multimodal learning, Visual Grounding (VG), aims to locate objects in images through natural language expressions. Ensuring the quality of VG models presents significant challenges due to the complex nature of the task. In the black box scenario, existing adversarial testing techniques often fail to fully exploit the potential of both modalities of information. They typically apply perturbations based solely on either the image or text information, disregarding the crucial correlation between the two modalities, which would lead to failures in test oracles or an inability to effectively challenge VG models. To this end, we propose PEELING, a text perturbation approach via image-aware property reduction for adversarial testing of the VG model. The core idea is to reduce the property-related information in the original expression meanwhile ensuring the reduced expression can still uniquely describe the original object in the image. To achieve this, PEELING first conducts the object and properties extraction and recombination to generate candidate property reduction expressions. It then selects the satisfied expressions that accurately describe the original object while ensuring no other objects in the image fulfill the expression, through querying the image with a visual understanding technique. We evaluate PEELING on the state-of-the-art VG model, i.e. OFA-VG, involving three commonly used datasets. Results show that the adversarial tests generated by PEELING achieves 21.4% in MultiModal Impact score (MMI), and outperforms state-of-the-art baselines for images and texts by 8.2%--15.1%.

Title: OpenGraph: Towards Open Graph Foundation Models

• Authors: Lianghao Xia, Ben Kao, Chao Huang
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.01121
• Pdf URL: https://arxiv.org/pdf/2403.01121
• Copy Paste: [[2403.01121]] OpenGraph: Towards Open Graph Foundation Models(https://arxiv.org/abs/2403.01121)
• Keywords: transformer
• Abstract: Graph learning has become indispensable for interpreting and harnessing relational data in diverse fields, ranging from recommendation systems to social network analysis. In this context, a variety of GNNs have emerged as promising methodologies for encoding the structural information of graphs. By effectively capturing the graph's underlying structure, these GNNs have shown great potential in enhancing performance in graph learning tasks, such as link prediction and node classification. However, despite their successes, a significant challenge persists: these advanced methods often face difficulties in generalizing to unseen graph data that significantly differs from the training instances. In this work, our aim is to advance the graph learning paradigm by developing a general graph foundation model. This model is designed to understand the complex topological patterns present in diverse graph data, enabling it to excel in zero-shot graph learning tasks across different downstream datasets. To achieve this goal, we address several key technical challenges in our OpenGraph model. Firstly, we propose a unified graph tokenizer to adapt our graph model to generalize well on unseen graph data, even when the underlying graph properties differ significantly from those encountered during training. Secondly, we develop a scalable graph transformer as the foundational encoder, which effectively captures node-wise dependencies within the global topological context. Thirdly, we introduce a data augmentation mechanism enhanced by a LLM to alleviate the limitations of data scarcity in real-world scenarios. Extensive experiments validate the effectiveness of our framework. By adapting our OpenGraph to new graph characteristics and comprehending the nuances of diverse graphs, our approach achieves remarkable zero-shot graph learning performance across various settings and domains.

Title: ELA: Efficient Local Attention for Deep Convolutional Neural Networks

• Authors: Wei Xu, Yi Wan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01123
• Pdf URL: https://arxiv.org/pdf/2403.01123
• Copy Paste: [[2403.01123]] ELA: Efficient Local Attention for Deep Convolutional Neural Networks(https://arxiv.org/abs/2403.01123)
• Keywords: segmentation
• Abstract: The attention mechanism has gained significant recognition in the field of computer vision due to its ability to effectively enhance the performance of deep neural networks. However, existing methods often struggle to effectively utilize spatial information or, if they do, they come at the cost of reducing channel dimensions or increasing the complexity of neural networks. In order to address these limitations, this paper introduces an Efficient Local Attention (ELA) method that achieves substantial performance improvements with a simple structure. By analyzing the limitations of the Coordinate Attention method, we identify the lack of generalization ability in Batch Normalization, the adverse effects of dimension reduction on channel attention, and the complexity of attention generation process. To overcome these challenges, we propose the incorporation of 1D convolution and Group Normalization feature enhancement techniques. This approach enables accurate localization of regions of interest by efficiently encoding two 1D positional feature maps without the need for dimension reduction, while allowing for a lightweight implementation. We carefully design three hyperparameters in ELA, resulting in four different versions: ELA-T, ELA-B, ELA-S, and ELA-L, to cater to the specific requirements of different visual tasks such as image classification, object detection and sementic segmentation. ELA can be seamlessly integrated into deep CNN networks such as ResNet, MobileNet, and DeepLab. Extensive evaluations on the ImageNet, MSCOCO, and Pascal VOC datasets demonstrate the superiority of the proposed ELA module over current state-of-the-art methods in all three aforementioned visual tasks.

Title: Text-guided Explorable Image Super-resolution

• Authors: Kanchana Vaishnavi Gandikota, Paramanand Chandramouli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01124
• Pdf URL: https://arxiv.org/pdf/2403.01124
• Copy Paste: [[2403.01124]] Text-guided Explorable Image Super-resolution(https://arxiv.org/abs/2403.01124)
• Keywords: diffusion, generative
• Abstract: In this paper, we introduce the problem of zero-shot text-guided exploration of the solutions to open-domain image super-resolution. Our goal is to allow users to explore diverse, semantically accurate reconstructions that preserve data consistency with the low-resolution inputs for different large downsampling factors without explicitly training for these specific degradations. We propose two approaches for zero-shot text-guided super-resolution - i) modifying the generative process of text-to-image \textit{T2I} diffusion models to promote consistency with low-resolution inputs, and ii) incorporating language guidance into zero-shot diffusion-based restoration methods. We show that the proposed approaches result in diverse solutions that match the semantic meaning provided by the text prompt while preserving data consistency with the degraded inputs. We evaluate the proposed baselines for the task of extreme super-resolution and demonstrate advantages in terms of restoration quality, diversity, and explorability of solutions.

Title: Evaluating Large Language Models as Virtual Annotators for Time-series Physical Sensing Data

• Authors: Aritra Hota, Soumyajit Chatterjee, Sandip Chakraborty
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.01133
• Pdf URL: https://arxiv.org/pdf/2403.01133
• Copy Paste: [[2403.01133]] Evaluating Large Language Models as Virtual Annotators for Time-series Physical Sensing Data(https://arxiv.org/abs/2403.01133)
• Keywords: privacy, large language model
• Abstract: Traditional human-in-the-loop-based annotation for time-series data like inertial data often requires access to alternate modalities like video or audio from the environment. These alternate sources provide the necessary information to the human annotator, as the raw numeric data is often too obfuscated even for an expert. However, this traditional approach has many concerns surrounding overall cost, efficiency, storage of additional modalities, time, scalability, and privacy. Interestingly, recent large language models (LLMs) are also trained with vast amounts of publicly available alphanumeric data, which allows them to comprehend and perform well on tasks beyond natural language processing. Naturally, this opens up a potential avenue to explore LLMs as virtual annotators where the LLMs will be directly provided the raw sensor data for annotation instead of relying on any alternate modality. Naturally, this could mitigate the problems of the traditional human-in-the-loop approach. Motivated by this observation, we perform a detailed study in this paper to assess whether the state-of-the-art (SOTA) LLMs can be used as virtual annotators for labeling time-series physical sensing data. To perform this in a principled manner, we segregate the study into two major phases. In the first phase, we investigate the challenges an LLM like GPT-4 faces in comprehending raw sensor data. Considering the observations from phase 1, in the next phase, we investigate the possibility of encoding the raw sensor data using SOTA SSL approaches and utilizing the projected time-series data to get annotations from the LLM. Detailed evaluation with four benchmark HAR datasets shows that SSL-based encoding and metric-based guidance allow the LLM to make more reasonable decisions and provide accurate annotations without requiring computationally expensive fine-tuning or sophisticated prompt engineering.

Title: ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies

• Authors: Oren Sultan, Yonatan Bitton, Ron Yosef, Dafna Shahaf
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01139
• Pdf URL: https://arxiv.org/pdf/2403.01139
• Copy Paste: [[2403.01139]] ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies(https://arxiv.org/abs/2403.01139)
• Keywords: large language model
• Abstract: Analogy-making is central to human cognition, allowing us to adapt to novel situations -- an ability that current AI systems still lack. Most analogy datasets today focus on simple analogies (e.g., word analogies); datasets including complex types of analogies are typically manually curated and very small. We believe that this holds back progress in computational analogy. In this work, we design a data generation pipeline, ParallelPARC (Parallel Paragraph Creator) leveraging state-of-the-art Large Language Models (LLMs) to create complex, paragraph-based analogies, as well as distractors, both simple and challenging. We demonstrate our pipeline and create ProPara-Logy, a dataset of analogies between scientific processes. We publish a gold-set, validated by humans, and a silver-set, generated automatically. We test LLMs' and humans' analogy recognition in binary and multiple-choice settings, and found that humans outperform the best models (~13% gap) after a light supervision. We demonstrate that our silver-set is useful for training models. Lastly, we show challenging distractors confuse LLMs, but not humans. We hope our pipeline will encourage research in this emerging field.

Title: Edge-guided Low-light Image Enhancement with Inertial Bregman Alternating Linearized Minimization

• Authors: Chaoyan Huang, Zhongming Wu, Tieyong Zeng
• Subjects: cs.CV, math.NA
• Abstract URL: https://arxiv.org/abs/2403.01142
• Pdf URL: https://arxiv.org/pdf/2403.01142
• Copy Paste: [[2403.01142]] Edge-guided Low-light Image Enhancement with Inertial Bregman Alternating Linearized Minimization(https://arxiv.org/abs/2403.01142)
• Keywords: extraction
• Abstract: Prior-based methods for low-light image enhancement often face challenges in extracting available prior information from dim images. To overcome this limitation, we introduce a simple yet effective Retinex model with the proposed edge extraction prior. More specifically, we design an edge extraction network to capture the fine edge features from the low-light image directly. Building upon the Retinex theory, we decompose the low-light image into its illumination and reflectance components and introduce an edge-guided Retinex model for enhancing low-light images. To solve the proposed model, we propose a novel inertial Bregman alternating linearized minimization algorithm. This algorithm addresses the optimization problem associated with the edge-guided Retinex model, enabling effective enhancement of low-light images. Through rigorous theoretical analysis, we establish the convergence properties of the algorithm. Besides, we prove that the proposed algorithm converges to a stationary point of the problem through nonconvex optimization theory. Furthermore, extensive experiments are conducted on multiple real-world low-light image datasets to demonstrate the efficiency and superiority of the proposed scheme.

Title: A Hybrid Model for Traffic Incident Detection based on Generative Adversarial Networks and Transformer Model

• Authors: Xinying Lu, Doudou Zhang, Jianli Xiao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01147
• Pdf URL: https://arxiv.org/pdf/2403.01147
• Copy Paste: [[2403.01147]] A Hybrid Model for Traffic Incident Detection based on Generative Adversarial Networks and Transformer Model(https://arxiv.org/abs/2403.01147)
• Keywords: transformer, generative
• Abstract: In addition to enhancing traffic safety and facilitating prompt emergency response, traffic incident detection plays an indispensable role in intelligent transportation systems by providing real-time traffic status information. This enables the realization of intelligent traffic control and management. Previous research has identified that apart from employing advanced algorithmic models, the effectiveness of detection is also significantly influenced by challenges related to acquiring large datasets and addressing dataset imbalances. A hybrid model combining transformer and generative adversarial networks (GANs) is proposed to address these challenges. Experiments are conducted on four real datasets to validate the superiority of the transformer in traffic incident detection. Additionally, GANs are utilized to expand the dataset and achieve a balanced ratio of 1:4, 2:3, and 1:1. The proposed model is evaluated against the baseline model. The results demonstrate that the proposed model enhances the dataset size, balances the dataset, and improves the performance of traffic incident detection in various aspects.

Title: A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization

• Authors: Tharindu Kumarage, Garima Agrawal, Paras Sheth, Raha Moraffah, Aman Chadha, Joshua Garland, Huan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01152
• Pdf URL: https://arxiv.org/pdf/2403.01152
• Copy Paste: [[2403.01152]] A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization(https://arxiv.org/abs/2403.01152)
• Keywords: large language model
• Abstract: We have witnessed lately a rapid proliferation of advanced Large Language Models (LLMs) capable of generating high-quality text. While these LLMs have revolutionized text generation across various domains, they also pose significant risks to the information ecosystem, such as the potential for generating convincing propaganda, misinformation, and disinformation at scale. This paper offers a review of AI-generated text forensic systems, an emerging field addressing the challenges of LLM misuses. We present an overview of the existing efforts in AI-generated text forensics by introducing a detailed taxonomy, focusing on three primary pillars: detection, attribution, and characterization. These pillars enable a practical understanding of AI-generated text, from identifying AI-generated content (detection), determining the specific AI model involved (attribution), and grouping the underlying intents of the text (characterization). Furthermore, we explore available resources for AI-generated text forensics research and discuss the evolving challenges and future directions of forensic systems in an AI era.

Title: Query Recovery from Easy to Hard: Jigsaw Attack against SSE

• Authors: Hao Nie, Wei Wang, Peng Xu, Xianglong Zhang, Laurence T. Yang, Kaitai Liang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01155
• Pdf URL: https://arxiv.org/pdf/2403.01155
• Copy Paste: [[2403.01155]] Query Recovery from Easy to Hard: Jigsaw Attack against SSE(https://arxiv.org/abs/2403.01155)
• Keywords: attack
• Abstract: Searchable symmetric encryption schemes often unintentionally disclose certain sensitive information, such as access, volume, and search patterns. Attackers can exploit such leakages and other available knowledge related to the user's database to recover queries. We find that the effectiveness of query recovery attacks depends on the volume/frequency distribution of keywords. Queries containing keywords with high volumes/frequencies are more susceptible to recovery, even when countermeasures are implemented. Attackers can also effectively leverage these ``special'' queries to recover all others. By exploiting the above finding, we propose a Jigsaw attack that begins by accurately identifying and recovering those distinctive queries. Leveraging the volume, frequency, and co-occurrence information, our attack achieves $90\%$ accuracy in three tested datasets, which is comparable to previous attacks (Oya et al., USENIX' 22 and Damie et al., USENIX' 21). With the same runtime, our attack demonstrates an advantage over the attack proposed by Oya et al (approximately $15\%$ more accuracy when the keyword universe size is 15k). Furthermore, our proposed attack outperforms existing attacks against widely studied countermeasures, achieving roughly $60\%$ and $85\%$ accuracy against the padding and the obfuscation, respectively. In this context, with a large keyword universe ($\geq$3k), it surpasses current state-of-the-art attacks by more than $20\%$.

Title: Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised Semantic Segmentation

• Authors: Lian Xu, Mohammed Bennamoun, Farid Boussaid, Wanli Ouyang, Ferdous Sohel, Dan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01156
• Pdf URL: https://arxiv.org/pdf/2403.01156
• Copy Paste: [[2403.01156]] Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised Semantic Segmentation(https://arxiv.org/abs/2403.01156)
• Keywords: segmentation
• Abstract: Most existing weakly supervised semantic segmentation (WSSS) methods rely on Class Activation Mapping (CAM) to extract coarse class-specific localization maps using image-level labels. Prior works have commonly used an off-line heuristic thresholding process that combines the CAM maps with off-the-shelf saliency maps produced by a general pre-trained saliency model to produce more accurate pseudo-segmentation labels. We propose AuxSegNet+, a weakly supervised auxiliary learning framework to explore the rich information from these saliency maps and the significant inter-task correlation between saliency detection and semantic segmentation. In the proposed AuxSegNet+, saliency detection and multi-label image classification are used as auxiliary tasks to improve the primary task of semantic segmentation with only image-level ground-truth labels. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps. In particular, we propose a cross-task dual-affinity learning module to learn both pairwise and unary affinities, which are used to enhance the task-specific features and predictions by aggregating both query-dependent and query-independent global context for both saliency detection and semantic segmentation. The learned cross-task pairwise affinity can also be used to refine and propagate CAM maps to provide better pseudo labels for both tasks. Iterative improvement of segmentation performance is enabled by cross-task affinity learning and pseudo-label updating. Extensive experiments demonstrate the effectiveness of the proposed approach with new state-of-the-art WSSS results on the challenging PASCAL VOC and MS COCO benchmarks.

Title: STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models

• Authors: Linhai Zhang, Jialong Wu, Deyu Zhou, Guoqiang Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01165
• Pdf URL: https://arxiv.org/pdf/2403.01165
• Copy Paste: [[2403.01165]] STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models(https://arxiv.org/abs/2403.01165)
• Keywords: large language model
• Abstract: Though Large Language Models (LLMs) have demonstrated the powerful capabilities of few-shot learning through prompting methods, supervised training is still necessary for complex reasoning tasks. Because of their extensive parameters and memory consumption, both Parameter-Efficient Fine-Tuning (PEFT) methods and Memory-Efficient Fine-Tuning methods have been proposed for LLMs. Nevertheless, the issue of large annotated data consumption, the aim of Data-Efficient Fine-Tuning, remains unexplored. One obvious way is to combine the PEFT method with active learning. However, the experimental results show that such a combination is not trivial and yields inferior results. Through probe experiments, such observation might be explained by two main reasons: uncertainty gap and poor model calibration. Therefore, in this paper, we propose a novel approach to effectively integrate uncertainty-based active learning and LoRA. Specifically, for the uncertainty gap, we introduce a dynamic uncertainty measurement that combines the uncertainty of the base model and the uncertainty of the full model during the iteration of active learning. For poor model calibration, we incorporate the regularization method during LoRA training to keep the model from being over-confident, and the Monte-Carlo dropout mechanism is employed to enhance the uncertainty estimation. Experimental results show that the proposed approach outperforms existing baseline models on three complex reasoning tasks.

Title: DINER: Debiasing Aspect-based Sentiment Analysis with Multi-variable Causal Inference

• Authors: Jialong Wu, Linhai Zhang, Deyu Zhou, Guoqiang Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01166
• Pdf URL: https://arxiv.org/pdf/2403.01166
• Copy Paste: [[2403.01166]] DINER: Debiasing Aspect-based Sentiment Analysis with Multi-variable Causal Inference(https://arxiv.org/abs/2403.01166)
• Keywords: robust
• Abstract: Though notable progress has been made, neural-based aspect-based sentiment analysis (ABSA) models are prone to learn spurious correlations from annotation biases, resulting in poor robustness on adversarial data transformations. Among the debiasing solutions, causal inference-based methods have attracted much research attention, which can be mainly categorized into causal intervention methods and counterfactual reasoning methods. However, most of the present debiasing methods focus on single-variable causal inference, which is not suitable for ABSA with two input variables (the target aspect and the review). In this paper, we propose a novel framework based on multi-variable causal inference for debiasing ABSA. In this framework, different types of biases are tackled based on different causal intervention methods. For the review branch, the bias is modeled as indirect confounding from context, where backdoor adjustment intervention is employed for debiasing. For the aspect branch, the bias is described as a direct correlation with labels, where counterfactual reasoning is adopted for debiasing. Extensive experiments demonstrate the effectiveness of the proposed method compared to various baselines on the two widely used real-world aspect robustness test set datasets.

Title: Run-time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations

• Authors: Hakan Yekta Yatbaz, Mehrdad Dianati, Konstantinos Koufos, Roger Woodman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01172
• Pdf URL: https://arxiv.org/pdf/2403.01172
• Copy Paste: [[2403.01172]] Run-time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations(https://arxiv.org/abs/2403.01172)
• Keywords: fair
• Abstract: Reliable detection of various objects and road users in the surrounding environment is crucial for the safe operation of automated driving systems (ADS). Despite recent progresses in developing highly accurate object detectors based on Deep Neural Networks (DNNs), they still remain prone to detection errors, which can lead to fatal consequences in safety-critical applications such as ADS. An effective remedy to this problem is to equip the system with run-time monitoring, named as introspection in the context of autonomous systems. Motivated by this, we introduce a novel introspection solution, which operates at the frame level for DNN-based 2D object detection and leverages neural network activation patterns. The proposed approach pre-processes the neural activation patterns of the object detector's backbone using several different modes. To provide extensive comparative analysis and fair comparison, we also adapt and implement several state-of-the-art (SOTA) introspection mechanisms for error detection in 2D object detection, using one-stage and two-stage object detectors evaluated on KITTI and BDD datasets. We compare the performance of the proposed solution in terms of error detection, adaptability to dataset shift, and, computational and memory resource requirements. Our performance evaluation shows that the proposed introspection solution outperforms SOTA methods, achieving an absolute reduction in the missed error ratio of 9% to 17% in the BDD dataset.

Title: d-DSE: Distinct Dynamic Searchable Encryption Resisting Volume Leakage in Encrypted Databases

• Authors: Dongli Liu, Wei Wang, Peng Xu, Laurence T. Yang, Bo Luo, Kaitai Liang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01182
• Pdf URL: https://arxiv.org/pdf/2403.01182
• Copy Paste: [[2403.01182]] d-DSE: Distinct Dynamic Searchable Encryption Resisting Volume Leakage in Encrypted Databases(https://arxiv.org/abs/2403.01182)
• Keywords: security, privacy, protect
• Abstract: Dynamic Searchable Encryption (DSE) has emerged as a solution to efficiently handle and protect large-scale data storage in encrypted databases (EDBs). Volume leakage poses a significant threat, as it enables adversaries to reconstruct search queries and potentially compromise the security and privacy of data. Padding strategies are common countermeasures for the leakage, but they significantly increase storage and communication costs. In this work, we develop a new perspective to handle volume leakage. We start with distinct search and further explore a new concept called \textit{distinct} DSE (\textit{d}-DSE). We also define new security notions, in particular Distinct with Volume-Hiding security, as well as forward and backward privacy, for the new concept. Based on \textit{d}-DSE, we construct the \textit{d}-DSE designed EDB with related constructions for distinct keyword (d-KW-\textit{d}DSE), keyword (KW-\textit{d}DSE), and join queries (JOIN-\textit{d}DSE) and update queries in encrypted databases. We instantiate a concrete scheme \textsf{BF-SRE}, employing Symmetric Revocable Encryption. We conduct extensive experiments on real-world datasets, such as Crime, Wikipedia, and Enron, for performance evaluation. The results demonstrate that our scheme is practical in data search and with comparable computational performance to the SOTA DSE scheme (\textsf{MITRA}*, \textsf{AURA}) and padding strategies (\textsf{SEAL}, \textsf{ShieldDB}). Furthermore, our proposal sharply reduces the communication cost as compared to padding strategies, with roughly 6.36 to 53.14x advantage for search queries.

Title: Leveraging Self-Supervised Learning for Scene Recognition in Child Sexual Abuse Imagery

• Authors: Pedro H. V. Valois, João Macedo, Leo S. F. Ribeiro, Jefersson A. dos Santos, Sandra Avila
• Subjects: cs.CV, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01183
• Pdf URL: https://arxiv.org/pdf/2403.01183
• Copy Paste: [[2403.01183]] Leveraging Self-Supervised Learning for Scene Recognition in Child Sexual Abuse Imagery(https://arxiv.org/abs/2403.01183)
• Keywords: secure, security
• Abstract: Crime in the 21st century is split into a virtual and real world. However, the former has become a global menace to people's well-being and security in the latter. The challenges it presents must be faced with unified global cooperation, and we must rely more than ever on automated yet trustworthy tools to combat the ever-growing nature of online offenses. Over 10 million child sexual abuse reports are submitted to the US National Center for Missing & Exploited Children every year, and over 80% originated from online sources. Therefore, investigation centers and clearinghouses cannot manually process and correctly investigate all imagery. In light of that, reliable automated tools that can securely and efficiently deal with this data are paramount. In this sense, the scene recognition task looks for contextual cues in the environment, being able to group and classify child sexual abuse data without requiring to be trained on sensitive material. The scarcity and limitations of working with child sexual abuse images lead to self-supervised learning, a machine-learning methodology that leverages unlabeled data to produce powerful representations that can be more easily transferred to target tasks. This work shows that self-supervised deep learning models pre-trained on scene-centric data can reach 71.6% balanced accuracy on our indoor scene classification task and, on average, 2.2 percentage points better performance than a fully supervised version. We cooperate with Brazilian Federal Police experts to evaluate our indoor classification model on actual child abuse material. The results demonstrate a notable discrepancy between the features observed in widely used scene datasets and those depicted on sensitive materials.

Title: Evault for legal records

• Authors: Anas S, Anuragav S, Abhishek R, Sachin K
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.01186
• Pdf URL: https://arxiv.org/pdf/2403.01186
• Copy Paste: [[2403.01186]] Evault for legal records(https://arxiv.org/abs/2403.01186)
• Keywords: secure, security, robust
• Abstract: Innovative solution for addressing the challenges in the legal records management system through a blockchain-based eVault platform. Our objective is to create a secure, transparent, and accessible ecosystem that caters to the needs of all stakeholders, including lawyers, judges, clients, and registrars. First and foremost, our solution is built on a robust blockchain platform like Ethereum harnessing the power of smart contracts to manage access, permissions, and transactions effectively. This ensures the utmost security and transparency in every interaction within the system. To make our eVault system user-friendly, we've developed intuitive interfaces for all stakeholders. Lawyers, judges, clients, and even registrars can effortlessly upload and retrieve legal documents, track changes, and share information within the platform. But that's not all; we've gone a step further by incorporating a document creation and saving feature within our app and website. This feature allows users to generate and securely store legal documents, streamlining the entire documentation process.

Title: Training Unbiased Diffusion Models From Biased Dataset

• Authors: Yeongmin Kim, Byeonghu Na, Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, Il-Chul Moon
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01189
• Pdf URL: https://arxiv.org/pdf/2403.01189
• Copy Paste: [[2403.01189]] Training Unbiased Diffusion Models From Biased Dataset(https://arxiv.org/abs/2403.01189)
• Keywords: diffusion, generative
• Abstract: With significant advancements in diffusion models, addressing the potential risks of dataset bias becomes increasingly important. Since generated outputs directly suffer from dataset bias, mitigating latent bias becomes a key factor in improving sample quality and proportion. This paper proposes time-dependent importance reweighting to mitigate the bias for the diffusion models. We demonstrate that the time-dependent density ratio becomes more precise than previous approaches, thereby minimizing error propagation in generative learning. While directly applying it to score-matching is intractable, we discover that using the time-dependent density ratio both for reweighting and score correction can lead to a tractable form of the objective function to regenerate the unbiased data density. Furthermore, we theoretically establish a connection with traditional score-matching, and we demonstrate its convergence to an unbiased distribution. The experimental evidence supports the usefulness of the proposed method, which outperforms baselines including time-independent importance reweighting on CIFAR-10, CIFAR-100, FFHQ, and CelebA with various bias settings. Our code is available at https://github.com/alsdudrla10/TIW-DSM.

Title: RAGged Edges: The Double-Edged Sword of Retrieval-Augmented Chatbots

• Authors: Philip Feldman. James R. Foulds, Shimei Pan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01193
• Pdf URL: https://arxiv.org/pdf/2403.01193
• Copy Paste: [[2403.01193]] RAGged Edges: The Double-Edged Sword of Retrieval-Augmented Chatbots(https://arxiv.org/abs/2403.01193)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) like ChatGPT demonstrate the remarkable progress of artificial intelligence. However, their tendency to hallucinate -- generate plausible but false information -- poses a significant challenge. This issue is critical, as seen in recent court cases where ChatGPT's use led to citations of non-existent legal rulings. This paper explores how Retrieval-Augmented Generation (RAG) can counter hallucinations by integrating external knowledge with prompts. We empirically evaluate RAG against standard LLMs using prompts designed to induce hallucinations. Our results show that RAG increases accuracy in some cases, but can still be misled when prompts directly contradict the model's pre-trained understanding. These findings highlight the complex nature of hallucinations and the need for more robust solutions to ensure LLM reliability in real-world applications. We offer practical recommendations for RAG deployment and discuss implications for the development of more trustworthy LLMs.

Title: Machine Translation in the Covid domain: an English-Irish case study for LoResMT 2021

• Authors: Séamus Lankford, Haithem Afli, Andy Way
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01196
• Pdf URL: https://arxiv.org/pdf/2403.01196
• Copy Paste: [[2403.01196]] Machine Translation in the Covid domain: an English-Irish case study for LoResMT 2021(https://arxiv.org/abs/2403.01196)
• Keywords: transformer
• Abstract: Translation models for the specific domain of translating Covid data from English to Irish were developed for the LoResMT 2021 shared task. Domain adaptation techniques, using a Covid-adapted generic 55k corpus from the Directorate General of Translation, were applied. Fine-tuning, mixed fine-tuning and combined dataset approaches were compared with models trained on an extended in-domain dataset. As part of this study, an English-Irish dataset of Covid related data, from the Health and Education domains, was developed. The highest-performing model used a Transformer architecture trained with an extended in-domain Covid dataset. In the context of this study, we have demonstrated that extending an 8k in-domain baseline dataset by just 5k lines improved the BLEU score by 27 points.

Title: DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling

• Authors: Shanghaoran Quan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01197
• Pdf URL: https://arxiv.org/pdf/2403.01197
• Copy Paste: [[2403.01197]] DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling(https://arxiv.org/abs/2403.01197)
• Keywords: generative, large language model
• Abstract: The performance of the reward model (RM) is a critical factor in improving the effectiveness of the large language model (LLM) during alignment fine-tuning. There remain two challenges in RM training: 1) training the same RM using various categories of data may cause its generalization performance to suffer from multi-task disturbance, and 2) the human annotation consistency rate is generally only $60\%$ to $75\%$, causing training data to contain a lot of noise. To tackle these two challenges, we introduced the idea of Mixture-of-Experts (MoE) into the field of RM for the first time. We propose the Double-Layer MoE RM (DMoERM). The outer layer MoE is a sparse model. After classifying an input into task categories, we route it to the corresponding inner layer task-specific model. The inner layer MoE is a dense model. We decompose the specific task into multiple capability dimensions and individually fine-tune a LoRA expert on each one. Their outputs are then synthesized by an MLP to compute the final rewards. To minimize costs, we call a public LLM API to obtain the capability preference labels. The validation on manually labeled datasets confirms that our model attains superior consistency with human preference and outstrips advanced generative approaches. Meanwhile, through BoN sampling and RL experiments, we demonstrate that our model outperforms state-of-the-art ensemble methods of RM and mitigates the overoptimization problem. Our code and dataset are available at: https://github.com/quanshr/DMoERM-v1.

Title: Stochastic gradient descent for streaming linear and rectified linear systems with Massart noise

• Authors: Halyun Jeong, Deanna Needell, Elizaveta Rebrova
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.01204
• Pdf URL: https://arxiv.org/pdf/2403.01204
• Copy Paste: [[2403.01204]] Stochastic gradient descent for streaming linear and rectified linear systems with Massart noise(https://arxiv.org/abs/2403.01204)
• Keywords: robust
• Abstract: We propose SGD-exp, a stochastic gradient descent approach for linear and ReLU regressions under Massart noise (adversarial semi-random corruption model) for the fully streaming setting. We show novel nearly linear convergence guarantees of SGD-exp to the true parameter with up to $50\%$ Massart corruption rate, and with any corruption rate in the case of symmetric oblivious corruptions. This is the first convergence guarantee result for robust ReLU regression in the streaming setting, and it shows the improved convergence rate over previous robust methods for $L_1$ linear regression due to a choice of an exponentially decaying step size, known for its efficiency in practice. Our analysis is based on the drift analysis of a discrete stochastic process, which could also be interesting on its own.

Title: Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning

• Authors: Shuo Yang, Zirui Shang, Yongqi Wang, Derong Deng, Hongwei Chen, Qiyuan Cheng, Xinxiao Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01209
• Pdf URL: https://arxiv.org/pdf/2403.01209
• Copy Paste: [[2403.01209]] Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning(https://arxiv.org/abs/2403.01209)
• Keywords: data-free, large language model
• Abstract: This paper proposes a novel framework for multi-label image recognition without any training data, called data-free framework, which uses knowledge of pre-trained Large Language Model (LLM) to learn prompts to adapt pretrained Vision-Language Model (VLM) like CLIP to multilabel classification. Through asking LLM by well-designed questions, we acquire comprehensive knowledge about characteristics and contexts of objects, which provides valuable text descriptions for learning prompts. Then we propose a hierarchical prompt learning method by taking the multi-label dependency into consideration, wherein a subset of category-specific prompt tokens are shared when the corresponding objects exhibit similar attributes or are more likely to co-occur. Benefiting from the remarkable alignment between visual and linguistic semantics of CLIP, the hierarchical prompts learned from text descriptions are applied to perform classification of images during inference. Our framework presents a new way to explore the synergies between multiple pre-trained models for novel category recognition. Extensive experiments on three public datasets (MS-COCO, VOC2007, and NUS-WIDE) demonstrate that our method achieves better results than the state-of-the-art methods, especially outperforming the zero-shot multi-label recognition methods by 4.7% in mAP on MS-COCO.

Title: SAR-AE-SFP: SAR Imagery Adversarial Example in Real Physics domain with Target Scattering Feature Parameters

• Authors: Jiahao Cui, Jiale Duan, Binyan Luo, Hang Cao, Wang Guo, Haifeng Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01210
• Pdf URL: https://arxiv.org/pdf/2403.01210
• Copy Paste: [[2403.01210]] SAR-AE-SFP: SAR Imagery Adversarial Example in Real Physics domain with Target Scattering Feature Parameters(https://arxiv.org/abs/2403.01210)
• Keywords: attack, transformer
• Abstract: Deep neural network-based Synthetic Aperture Radar (SAR) target recognition models are susceptible to adversarial examples. Current adversarial example generation methods for SAR imagery primarily operate in the 2D digital domain, known as image adversarial examples. Recent work, while considering SAR imaging scatter mechanisms, fails to account for the actual imaging process, rendering attacks in the three-dimensional physical domain infeasible, termed pseudo physics adversarial examples. To address these challenges, this paper proposes SAR-AE-SFP-Attack, a method to generate real physics adversarial examples by altering the scattering feature parameters of target objects. Specifically, we iteratively optimize the coherent energy accumulation of the target echo by perturbing the reflection coefficient and scattering coefficient in the scattering feature parameters of the three-dimensional target object, and obtain the adversarial example after echo signal processing and imaging processing in the RaySAR simulator. Experimental results show that compared to digital adversarial attack methods, SAR-AE-SFP Attack significantly improves attack efficiency on CNN-based models (over 30\%) and Transformer-based models (over 13\%), demonstrating significant transferability of attack effects across different models and perspectives.

Title: TCIG: Two-Stage Controlled Image Generation with Quality Enhancement through Diffusion

• Authors: Salaheldin Mohamed
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01212
• Pdf URL: https://arxiv.org/pdf/2403.01212
• Copy Paste: [[2403.01212]] TCIG: Two-Stage Controlled Image Generation with Quality Enhancement through Diffusion(https://arxiv.org/abs/2403.01212)
• Keywords: diffusion
• Abstract: In recent years, significant progress has been made in the development of text- to-image generation models. However, these models still face limitations when it comes to achieving full controllability during the generation process. Often, spe- cific training or the use of limited models is required, and even then, they have certain restrictions. To address these challenges, A two-stage method that effec- tively combines controllability and high quality in the generation of images is proposed. This approach leverages the expertise of pre-trained models to achieve precise control over the generated images, while also harnessing the power of diffusion models to achieve state-of-the-art quality. By separating controllability from high quality, This method achieves outstanding results. It is compatible with both latent and image space diffusion models, ensuring versatility and flexibil- ity. Moreover, This approach consistently produces comparable outcomes to the current state-of-the-art methods in the field. Overall, This proposed method rep- resents a significant advancement in text-to-image generation, enabling improved controllability without compromising on the quality of the generated images.

Title: Boosting Box-supervised Instance Segmentation with Pseudo Depth

• Authors: Xinyi Yu, Ling Yan, Pengtao Jiang, Hao Chen, Bo Li, Lin Yuanbo Wu, Linlin Ou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01214
• Pdf URL: https://arxiv.org/pdf/2403.01214
• Copy Paste: [[2403.01214]] Boosting Box-supervised Instance Segmentation with Pseudo Depth(https://arxiv.org/abs/2403.01214)
• Keywords: segmentation
• Abstract: The realm of Weakly Supervised Instance Segmentation (WSIS) under box supervision has garnered substantial attention, showcasing remarkable advancements in recent years. However, the limitations of box supervision become apparent in its inability to furnish effective information for distinguishing foreground from background within the specified target box. This research addresses this challenge by introducing pseudo-depth maps into the training process of the instance segmentation network, thereby boosting its performance by capturing depth differences between instances. These pseudo-depth maps are generated using a readily available depth predictor and are not necessary during the inference stage. To enable the network to discern depth features when predicting masks, we integrate a depth prediction layer into the mask prediction head. This innovative approach empowers the network to simultaneously predict masks and depth, enhancing its ability to capture nuanced depth-related information during the instance segmentation process. We further utilize the mask generated in the training process as supervision to distinguish the foreground from the background. When selecting the best mask for each box through the Hungarian algorithm, we use depth consistency as one calculation cost item. The proposed method achieves significant improvements on Cityscapes and COCO dataset.

Title: Efficient Algorithm Level Error Detection for Number-Theoretic Transform Assessed on FPGAs

• Authors: Kasra Ahmadi, Saeed Aghapour, Mehran Mozaffari Kermani, Reza Azarderakhsh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01215
• Pdf URL: https://arxiv.org/pdf/2403.01215
• Copy Paste: [[2403.01215]] Efficient Algorithm Level Error Detection for Number-Theoretic Transform Assessed on FPGAs(https://arxiv.org/abs/2403.01215)
• Keywords: secure, security, protect, attack
• Abstract: Polynomial multiplication stands out as a highly demanding arithmetic process in the development of post-quantum cryptosystems. The importance of number-theoretic transform (NTT) extends beyond post-quantum cryptosystems, proving valuable in enhancing existing security protocols such as digital signature schemes and hash functions. Due to the potential for errors to significantly disrupt the operation of secure, cryptographically-protected systems, compromising data integrity, and safeguarding against side-channel attacks initiated through faults it is essential to incorporate mitigating error detection schemes. This paper introduces algorithm level fault detection schemes in NTT multiplication, representing a significant enhancement compared to previous research. We evaluate this through the simulation of a fault model, ensuring that the conducted assessments accurately mirror the obtained results. Consequently, we attain a notably comprehensive coverage of errors. Finally, we assess the performance of our efficient error detection scheme on FPGAs to showcase its implementation and resource requirements. Through implementation of our error detection approach on Xilinx/AMD Zynq Ultrascale+ and Artix-7, we achieve a comparable throughput with just a 9% increase in area and 13% increase in latency compared to the original hardware implementations.

Title: API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access

• Authors: Jiayuan Su, Jing Luo, Hongwei Wang, Lu Cheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01216
• Pdf URL: https://arxiv.org/pdf/2403.01216
• Copy Paste: [[2403.01216]] API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access(https://arxiv.org/abs/2403.01216)
• Keywords: large language model
• Abstract: This study aims to address the pervasive challenge of quantifying uncertainty in large language models (LLMs) without logit-access. Conformal Prediction (CP), known for its model-agnostic and distribution-free features, is a desired approach for various LLMs and data distributions. However, existing CP methods for LLMs typically assume access to the logits, which are unavailable for some API-only LLMs. In addition, logits are known to be miscalibrated, potentially leading to degraded CP performance. To tackle these challenges, we introduce a novel CP method that (1) is tailored for API-only LLMs without logit-access; (2) minimizes the size of prediction sets; and (3) ensures a statistical guarantee of the user-defined coverage. The core idea of this approach is to formulate nonconformity measures using both coarse-grained (i.e., sample frequency) and fine-grained uncertainty notions (e.g., semantic similarity). Experimental results on both close-ended and open-ended Question Answering tasks show our approach can mostly outperform the logit-based CP baselines.

Title: Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy

• Authors: Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, Nicolas Papernot
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01218
• Pdf URL: https://arxiv.org/pdf/2403.01218
• Copy Paste: [[2403.01218]] Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy(https://arxiv.org/abs/2403.01218)
• Keywords: privacy, protect, attack, membership infer
• Abstract: The high cost of model training makes it increasingly desirable to develop techniques for unlearning. These techniques seek to remove the influence of a training example without having to retrain the model from scratch. Intuitively, once a model has unlearned, an adversary that interacts with the model should no longer be able to tell whether the unlearned example was included in the model's training set or not. In the privacy literature, this is known as membership inference. In this work, we discuss adaptations of Membership Inference Attacks (MIAs) to the setting of unlearning (leading to their ``U-MIA'' counterparts). We propose a categorization of existing U-MIAs into ``population U-MIAs'', where the same attacker is instantiated for all examples, and ``per-example U-MIAs'', where a dedicated attacker is instantiated for each example. We show that the latter category, wherein the attacker tailors its membership prediction to each example under attack, is significantly stronger. Indeed, our results show that the commonly used U-MIAs in the unlearning literature overestimate the privacy protection afforded by existing unlearning techniques on both vision and language models. Our investigation reveals a large variance in the vulnerability of different examples to per-example U-MIAs. In fact, several unlearning algorithms lead to a reduced vulnerability for some, but not all, examples that we wish to unlearn, at the expense of increasing it for other examples. Notably, we find that the privacy protection for the remaining training examples may worsen as a consequence of unlearning. We also discuss the fundamental difficulty of equally protecting all examples using existing unlearning schemes, due to the different rates at which examples are unlearned. We demonstrate that naive attempts at tailoring unlearning stopping criteria to different examples fail to alleviate these issues.

Title: DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction

• Authors: Junwen Xiong, Peng Zhang, Tao You, Chuanyue Li, Wei Huang, Yufei Zha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01226
• Pdf URL: https://arxiv.org/pdf/2403.01226
• Copy Paste: [[2403.01226]] DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction(https://arxiv.org/abs/2403.01226)
• Keywords: diffusion, generative
• Abstract: Audio-visual saliency prediction can draw support from diverse modality complements, but further performance enhancement is still challenged by customized architectures as well as task-specific loss functions. In recent studies, denoising diffusion models have shown more promising in unifying task frameworks owing to their inherent ability of generalization. Following this motivation, a novel Diffusion architecture for generalized audio-visual Saliency prediction (DiffSal) is proposed in this work, which formulates the prediction problem as a conditional generative task of the saliency map by utilizing input audio and video as the conditions. Based on the spatio-temporal audio-visual features, an extra network Saliency-UNet is designed to perform multi-modal attention modulation for progressive refinement of the ground-truth saliency map from the noisy map. Extensive experiments demonstrate that the proposed DiffSal can achieve excellent performance across six challenging audio-visual benchmarks, with an average relative improvement of 6.3\% over the previous state-of-the-art results by six metrics.

Title: REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild

• Authors: Jose Vargas Quiros, Chirag Raman, Stephanie Tan, Ekin Gedik, Laura Cabrera-Quiros, Hayley Hung
• Subjects: cs.CV, cs.AI, cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.01229
• Pdf URL: https://arxiv.org/pdf/2403.01229
• Copy Paste: [[2403.01229]] REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild(https://arxiv.org/abs/2403.01229)
• Keywords: privacy, segmentation
• Abstract: Recognizing speaking in humans is a central task towards understanding social interactions. Ideally, speaking would be detected from individual voice recordings, as done previously for meeting scenarios. However, individual voice recordings are hard to obtain in the wild, especially in crowded mingling scenarios due to cost, logistics, and privacy concerns. As an alternative, machine learning models trained on video and wearable sensor data make it possible to recognize speech by detecting its related gestures in an unobtrusive, privacy-preserving way. These models themselves should ideally be trained using labels obtained from the speech signal. However, existing mingling datasets do not contain high quality audio recordings. Instead, speaking status annotations have often been inferred by human annotators from video, without validation of this approach against audio-based ground truth. In this paper we revisit no-audio speaking status estimation by presenting the first publicly available multimodal dataset with high-quality individual speech recordings of 33 subjects in a professional networking event. We present three baselines for no-audio speaking status segmentation: a) from video, b) from body acceleration (chest-worn accelerometer), c) from body pose tracks. In all cases we predict a 20Hz binary speaking status signal extracted from the audio, a time resolution not available in previous datasets. In addition to providing the signals and ground truth necessary to evaluate a wide range of speaking status detection methods, the availability of audio in REWIND makes it suitable for cross-modality studies not feasible with previous mingling datasets. Finally, our flexible data consent setup creates new challenges for multimodal systems under missing modalities.

Title: Benchmarking Segmentation Models with Mask-Preserved Attribute Editing

• Authors: Zijin Yin, Kongming Liang, Bing Li, Zhanyu Ma, Jun Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01231
• Pdf URL: https://arxiv.org/pdf/2403.01231
• Copy Paste: [[2403.01231]] Benchmarking Segmentation Models with Mask-Preserved Attribute Editing(https://arxiv.org/abs/2403.01231)
• Keywords: robust, segmentation
• Abstract: When deploying segmentation models in practice, it is critical to evaluate their behaviors in varied and complex scenes. Different from the previous evaluation paradigms only in consideration of global attribute variations (e.g. adverse weather), we investigate both local and global attribute variations for robustness evaluation. To achieve this, we construct a mask-preserved attribute editing pipeline to edit visual attributes of real images with precise control of structural information. Therefore, the original segmentation labels can be reused for the edited images. Using our pipeline, we construct a benchmark covering both object and image attributes (e.g. color, material, pattern, style). We evaluate a broad variety of semantic segmentation models, spanning from conventional close-set models to recent open-vocabulary large models on their robustness to different types of variations. We find that both local and global attribute variations affect segmentation performances, and the sensitivity of models diverges across different variation types. We argue that local attributes have the same importance as global attributes, and should be considered in the robustness evaluation of segmentation models. Code: https://github.com/PRIS-CV/Pascal-EA.

Title: Polynormer: Polynomial-Expressive Graph Transformer in Linear Time

• Authors: Chenhui Deng, Zichao Yue, Zhiru Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01232
• Pdf URL: https://arxiv.org/pdf/2403.01232
• Copy Paste: [[2403.01232]] Polynormer: Polynomial-Expressive Graph Transformer in Linear Time(https://arxiv.org/abs/2403.01232)
• Keywords: transformer
• Abstract: Graph transformers (GTs) have emerged as a promising architecture that is theoretically more expressive than message-passing graph neural networks (GNNs). However, typical GT models have at least quadratic complexity and thus cannot scale to large graphs. While there are several linear GTs recently proposed, they still lag behind GNN counterparts on several popular graph datasets, which poses a critical concern on their practical expressivity. To balance the trade-off between expressivity and scalability of GTs, we propose Polynormer, a polynomial-expressive GT model with linear complexity. Polynormer is built upon a novel base model that learns a high-degree polynomial on input features. To enable the base model permutation equivariant, we integrate it with graph topology and node features separately, resulting in local and global equivariant attention models. Consequently, Polynormer adopts a linear local-to-global attention scheme to learn high-degree equivariant polynomials whose coefficients are controlled by attention scores. Polynormer has been evaluated on $13$ homophilic and heterophilic datasets, including large graphs with millions of nodes. Our extensive experiment results show that Polynormer outperforms state-of-the-art GNN and GT baselines on most datasets, even without the use of nonlinear activation functions.

Title: IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact

• Authors: Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, Chun Yuan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01241
• Pdf URL: https://arxiv.org/pdf/2403.01241
• Copy Paste: [[2403.01241]] IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact(https://arxiv.org/abs/2403.01241)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel in natural language processing but demand intensive computation. To mitigate this, various quantization methods have been explored, yet they compromise LLM performance. This paper unveils a previously overlooked type of outlier in LLMs. Such outliers are found to allocate most of the attention scores on initial tokens of input, termed as pivot tokens, which is crucial to the performance of quantized LLMs. Given that, we propose IntactKV to generate the KV cache of pivot tokens losslessly from the full-precision model. The approach is simple and easy to combine with existing quantization solutions. Besides, IntactKV can be calibrated as additional LLM parameters to boost the quantized LLMs further. Mathematical analysis also proves that IntactKV effectively reduces the upper bound of quantization error. Empirical results show that IntactKV brings consistent improvement and achieves lossless weight-only INT4 quantization on various downstream tasks, leading to the new state-of-the-art for LLM quantization.

Title: Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal

• Authors: Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, Jinsong Su
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01244
• Pdf URL: https://arxiv.org/pdf/2403.01244
• Copy Paste: [[2403.01244]] Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal(https://arxiv.org/abs/2403.01244)
• Keywords: large language model
• Abstract: Large language models (LLMs) suffer from catastrophic forgetting during continual learning. Conventional rehearsal-based methods rely on previous training data to retain the model's ability, which may not be feasible in real-world applications. When conducting continual learning based on a publicly-released LLM checkpoint, the availability of the original training data may be non-existent. To address this challenge, we propose a framework called Self-Synthesized Rehearsal (SSR) that uses the LLM to generate synthetic instances for rehearsal. Concretely, we first employ the base LLM for in-context learning to generate synthetic instances. Subsequently, we utilize the latest LLM to refine the instance outputs based on the synthetic inputs, preserving its acquired ability. Finally, we select diverse high-quality synthetic instances for rehearsal in future stages. Experimental results demonstrate that SSR achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. Besides, SSR effectively preserves the generalization capabilities of LLMs in general domains.

Title: AcME-AD: Accelerated Model Explanations for Anomaly Detection

• Authors: Valentina Zaccaria, David Dandolo, Chiara Masiero, Gian Antonio Susto
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01245
• Pdf URL: https://arxiv.org/pdf/2403.01245
• Copy Paste: [[2403.01245]] AcME-AD: Accelerated Model Explanations for Anomaly Detection(https://arxiv.org/abs/2403.01245)
• Keywords: robust, interpretability, explainability
• Abstract: Pursuing fast and robust interpretability in Anomaly Detection is crucial, especially due to its significance in practical applications. Traditional Anomaly Detection methods excel in outlier identification but are often black-boxes, providing scant insights into their decision-making process. This lack of transparency compromises their reliability and hampers their adoption in scenarios where comprehending the reasons behind anomaly detection is vital. At the same time, getting explanations quickly is paramount in practical scenarios. To bridge this gap, we present AcME-AD, a novel approach rooted in Explainable Artificial Intelligence principles, designed to clarify Anomaly Detection models for tabular data. AcME-AD transcends the constraints of model-specific or resource-heavy explainability techniques by delivering a model-agnostic, efficient solution for interoperability. It offers local feature importance scores and a what-if analysis tool, shedding light on the factors contributing to each anomaly, thus aiding root cause analysis and decision-making. This paper elucidates AcME-AD's foundation, its benefits over existing methods, and validates its effectiveness with tests on both synthetic and real datasets. AcME-AD's implementation and experiment replication code is accessible in a public repository.

Title: SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code

• Authors: Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A. Ross, Cordelia Schmid, Alireza Fathi
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01248
• Pdf URL: https://arxiv.org/pdf/2403.01248
• Copy Paste: [[2403.01248]] SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code(https://arxiv.org/abs/2403.01248)
• Keywords: generative, large language model
• Abstract: This paper introduces SceneCraft, a Large Language Model (LLM) Agent converting text descriptions into Blender-executable Python scripts which render complex scenes with up to a hundred 3D assets. This process requires complex spatial planning and arrangement. We tackle these challenges through a combination of advanced abstraction, strategic planning, and library learning. SceneCraft first models a scene graph as a blueprint, detailing the spatial relationships among assets in the scene. SceneCraft then writes Python scripts based on this graph, translating relationships into numerical constraints for asset layout. Next, SceneCraft leverages the perceptual strengths of vision-language foundation models like GPT-V to analyze rendered images and iteratively refine the scene. On top of this process, SceneCraft features a library learning mechanism that compiles common script functions into a reusable library, facilitating continuous self-improvement without expensive LLM parameter tuning. Our evaluation demonstrates that SceneCraft surpasses existing LLM-based agents in rendering complex scenes, as shown by its adherence to constraints and favorable human assessments. We also showcase the broader application potential of SceneCraft by reconstructing detailed 3D scenes from the Sintel movie and guiding a video generative model with generated scenes as intermediary control signal.

Title: Accelerating Greedy Coordinate Gradient via Probe Sampling

• Authors: Yiran Zhao, Wenyue Zheng, Tianle Cai, Xuan Long Do, Kenji Kawaguchi, Anirudh Goyal, Michael Shieh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01251
• Pdf URL: https://arxiv.org/pdf/2403.01251
• Copy Paste: [[2403.01251]] Accelerating Greedy Coordinate Gradient via Probe Sampling(https://arxiv.org/abs/2403.01251)
• Keywords: attack, large language model
• Abstract: Safety of Large Language Models (LLMs) has become a central issue given their rapid progress and wide applications. Greedy Coordinate Gradient (GCG) is shown to be effective in constructing prompts containing adversarial suffixes to break the presumingly safe LLMs, but the optimization of GCG is time-consuming and limits its practicality. To reduce the time cost of GCG and enable more comprehensive studies of LLM safety, in this work, we study a new algorithm called $\texttt{Probe sampling}$ to accelerate the GCG algorithm. At the core of the algorithm is a mechanism that dynamically determines how similar a smaller draft model's predictions are to the target model's predictions for prompt candidates. When the target model is similar to the draft model, we rely heavily on the draft model to filter out a large number of potential prompt candidates to reduce the computation time. Probe sampling achieves up to $5.6$ times speedup using Llama2-7b and leads to equal or improved attack success rate (ASR) on the AdvBench.

Title: Dissecting Language Models: Machine Unlearning via Selective Pruning

• Authors: Nicholas Pochinkov, Nandi Schoots
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01267
• Pdf URL: https://arxiv.org/pdf/2403.01267
• Copy Paste: [[2403.01267]] Dissecting Language Models: Machine Unlearning via Selective Pruning(https://arxiv.org/abs/2403.01267)
• Keywords: large language model
• Abstract: Understanding and shaping the behaviour of Large Language Models (LLMs) is increasingly important as applications become more powerful and more frequently adopted. This paper introduces a machine unlearning method specifically designed for LLMs. We introduce a selective pruning method for LLMs that removes neurons based on their relative importance on a targeted capability compared to overall network performance. This approach is a compute- and data-efficient method for identifying and removing neurons that enable specific behaviours. Our findings reveal that both feed-forward and attention neurons in LLMs are specialized; that is, for specific tasks, certain neurons are more crucial than others.

Title: Defending Against Data Reconstruction Attacks in Federated Learning: An Information Theory Approach

• Authors: Qi Tan, Qi Li, Yi Zhao, Zhuotao Liu, Xiaobing Guo, Ke Xu
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.01268
• Pdf URL: https://arxiv.org/pdf/2403.01268
• Copy Paste: [[2403.01268]] Defending Against Data Reconstruction Attacks in Federated Learning: An Information Theory Approach(https://arxiv.org/abs/2403.01268)
• Keywords: privacy, attack, membership infer, federate
• Abstract: Federated Learning (FL) trains a black-box and high-dimensional model among different clients by exchanging parameters instead of direct data sharing, which mitigates the privacy leak incurred by machine learning. However, FL still suffers from membership inference attacks (MIA) or data reconstruction attacks (DRA). In particular, an attacker can extract the information from local datasets by constructing DRA, which cannot be effectively throttled by existing techniques, e.g., Differential Privacy (DP). In this paper, we aim to ensure a strong privacy guarantee for FL under DRA. We prove that reconstruction errors under DRA are constrained by the information acquired by an attacker, which means that constraining the transmitted information can effectively throttle DRA. To quantify the information leakage incurred by FL, we establish a channel model, which depends on the upper bound of joint mutual information between the local dataset and multiple transmitted parameters. Moreover, the channel model indicates that the transmitted information can be constrained through data space operation, which can improve training efficiency and the model accuracy under constrained information. According to the channel model, we propose algorithms to constrain the information transmitted in a single round of local training. With a limited number of training rounds, the algorithms ensure that the total amount of transmitted information is limited. Furthermore, our channel model can be applied to various privacy-enhancing techniques (such as DP) to enhance privacy guarantees against DRA. Extensive experiments with real-world datasets validate the effectiveness of our methods.

Title: Employing LLMs for Incident Response Planning and Review

• Authors: Sam Hays, Dr. Jules White
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01271
• Pdf URL: https://arxiv.org/pdf/2403.01271
• Copy Paste: [[2403.01271]] Employing LLMs for Incident Response Planning and Review(https://arxiv.org/abs/2403.01271)
• Keywords: security, large language model
• Abstract: Incident Response Planning (IRP) is essential for effective cybersecurity management, requiring detailed documentation (or playbooks) to guide security personnel during incidents. Yet, creating comprehensive IRPs is often hindered by challenges such as complex systems, high turnover rates, and legacy technologies lacking documentation. This paper argues that, despite these obstacles, the development, review, and refinement of IRPs can be significantly enhanced through the utilization of Large Language Models (LLMs) like ChatGPT. By leveraging LLMs for tasks such as drafting initial plans, suggesting best practices, and identifying documentation gaps, organizations can overcome resource constraints and improve their readiness for cybersecurity incidents. We discuss the potential of LLMs to streamline IRP processes, while also considering the limitations and the need for human oversight in ensuring the accuracy and relevance of generated content. Our findings contribute to the cybersecurity field by demonstrating a novel approach to enhancing IRP with AI technologies, offering practical insights for organizations seeking to bolster their incident response capabilities.

Title: NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention

• Authors: Tianyi Zhang, Jonah Wonkyu Yi, Bowen Yao, Zhaozhuo Xu, Anshumali Shrivastava
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01273
• Pdf URL: https://arxiv.org/pdf/2403.01273
• Copy Paste: [[2403.01273]] NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention(https://arxiv.org/abs/2403.01273)
• Keywords: large language model
• Abstract: Large language model inference on Central Processing Units (CPU) is challenging due to the vast quantities of expensive Multiply-Add (MAD) matrix operations in the attention computations. In this paper, we argue that there is a rare gem in modern CPUs, Single-Instruction-Multiple-Data (SIMD) registers, which allow for ultra-low-latency lookups in batch. We leverage this unique capability of CPUs to propose NoMAD-Attention, an efficient attention algorithm that replaces MAD operations with in-register lookups. Through hardware-aware algorithmic designs, NoMAD-Attention achieves the computation of attention scores using repeated fast accesses to SIMD registers despite their highly limited sizes. Moreover, NoMAD-Attention works with pre-trained attention-based LLMs without model finetuning. Empirical evaluations demonstrate that NoMAD-Attention maintains the quality of the original LLMs well, and speeds up the 4-bit quantized LLaMA-7B-based model by up to 2$\times$ at 16k context length. Our results are reproducible at https://github.com/tonyzhang617/nomad-dist.

Title: Characterizing Ethereum Upgradable Smart Contracts and Their Security Implications

• Authors: Xiaofan Li, Jin Yang, Jiaqi Chen, Yuzhe Tang, Xing Gao
• Subjects: cs.CR, cs.CE
• Abstract URL: https://arxiv.org/abs/2403.01290
• Pdf URL: https://arxiv.org/pdf/2403.01290
• Copy Paste: [[2403.01290]] Characterizing Ethereum Upgradable Smart Contracts and Their Security Implications(https://arxiv.org/abs/2403.01290)
• Keywords: security, attack
• Abstract: Upgradeable smart contracts (USCs) have been widely adopted to enable modifying deployed smart contracts. While USCs bring great flexibility to developers, improper usage might introduce new security issues, potentially allowing attackers to hijack USCs and their users. In this paper, we conduct a large-scale measurement study to characterize USCs and their security implications in the wild. We summarize six commonly used USC patterns and develop a tool, USCDetector, to identify USCs without needing source code. Particularly, USCDetector collects various information such as bytecode and transaction information to construct upgrade chains for USCs and disclose potentially vulnerable ones. We evaluate USCDetector using verified smart contracts (i.e., with source code) as ground truth and show that USCDetector can achieve high accuracy with a precision of 96.26%. We then use USCDetector to conduct a large-scale study on Ethereum, covering a total of 60,251,064 smart contracts. USCDetecor constructs 10,218 upgrade chains and discloses multiple real-world USCs with potential security issues.

Title: A Photonic Physically Unclonable Function's Resilience to Multiple-Valued Machine Learning Attacks

• Authors: Jessie M. Henderson, Elena R. Henderson, Clayton A. Harper, Hiva Shahoei, William V. Oxford, Eric C. Larson, Duncan L. MacFarlane, Mitchell A. Thornton
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01299
• Pdf URL: https://arxiv.org/pdf/2403.01299
• Copy Paste: [[2403.01299]] A Photonic Physically Unclonable Function's Resilience to Multiple-Valued Machine Learning Attacks(https://arxiv.org/abs/2403.01299)
• Keywords: security, attack
• Abstract: Physically unclonable functions (PUFs) identify integrated circuits using nonlinearly-related challenge-response pairs (CRPs). Ideally, the relationship between challenges and corresponding responses is unpredictable, even if a subset of CRPs is known. Previous work developed a photonic PUF offering improved security compared to non-optical counterparts. Here, we investigate this PUF's susceptibility to Multiple-Valued-Logic-based machine learning attacks. We find that approximately 1,000 CRPs are necessary to train models that predict response bits better than random chance. Given the significant challenge of acquiring a vast number of CRPs from a photonic PUF, our results demonstrate photonic PUF resilience against such attacks.

Title: Improving the Validity of Automatically Generated Feedback via Reinforcement Learning

• Authors: Alexander Scarlatos, Digory Smith, Simon Woodhead, Andrew Lan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01304
• Pdf URL: https://arxiv.org/pdf/2403.01304
• Copy Paste: [[2403.01304]] Improving the Validity of Automatically Generated Feedback via Reinforcement Learning(https://arxiv.org/abs/2403.01304)
• Keywords: large language model
• Abstract: Automatically generating feedback via large language models (LLMs) in intelligent tutoring systems and online learning platforms has the potential to improve the learning outcomes of many students. However, both feedback generation and evaluation are challenging: feedback content has to be valid especially in subjects like math, which requires models to understand the problem, the solution, and where the student's error lies. Feedback also has to be pedagogically valid to reflect effective tutoring strategies, such as explaining possible misconceptions and encouraging the student, among other desirable features. In this work, we address both problems of automatically generating and evaluating feedback while considering both correctness and alignment. First, we propose a rubric for evaluating math feedback and show that GPT-4 is able to effectively use it to annotate human-written and LLM-generated feedback. Second, we propose a framework for feedback generation that optimizes both correctness and alignment using reinforcement learning (RL). Specifically, we use GPT-4's annotations to create preferences over feedback pairs in an augmented dataset for training via direct preference optimization (DPO). We show that our methods significantly increase the correctness and alignment of generated feedback with Llama 2, an open-source LLM, qualitatively analyze our generation and evaluation systems using case studies, and outline several areas for future work.

Title: VBART: The Turkish LLM

• Authors: Meliksah Turker, Mehmet Erdi Ari, Aydin Han
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01308
• Pdf URL: https://arxiv.org/pdf/2403.01308
• Copy Paste: [[2403.01308]] VBART: The Turkish LLM(https://arxiv.org/abs/2403.01308)
• Keywords: large language model
• Abstract: We present VBART, the first Turkish sequence-to-sequence Large Language Models (LLMs) pre-trained on a large corpus from scratch. VBART are compact LLMs based on good ideas leveraged from BART and mBART models and come in two sizes, Large and XLarge. Fine-tuned VBART models surpass the prior state-of-the-art results in abstractive text summarization, title generation, text paraphrasing, question answering and question generation tasks. They allow fine-tuning for future text generation tasks and datasets, carving a new path for Turkish Natural Language Processing (NLP) research. Our work shows that having a pre-trained LLM for Turkish outperforms up to 3x multilingual models, improving existing results and providing efficient models for training and inference. Moreover, we show that our monolingual tokenizer is 7x more efficient than OpenAI's multilingual tokenizer. Last but not least, we introduce a method to enlarge an existing pre-trained LLM and question the relevancy of Chinchilla Scaling Law to sequence-to-sequence masked language models. Our fine-tuned models, tokenizer and cleaned web corpus of 135 GB are publicly available at huggingface.co/vngrs-ai.

Title: Image-Based Dietary Assessment: A Healthy Eating Plate Estimation System

• Authors: Assylzhan Izbassar, Pakizar Shamoi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01310
• Pdf URL: https://arxiv.org/pdf/2403.01310
• Copy Paste: [[2403.01310]] Image-Based Dietary Assessment: A Healthy Eating Plate Estimation System(https://arxiv.org/abs/2403.01310)
• Keywords: segmentation
• Abstract: The nutritional quality of diets has significantly deteriorated over the past two to three decades, a decline often underestimated by the people. This deterioration, coupled with a hectic lifestyle, has contributed to escalating health concerns. Recognizing this issue, researchers at Harvard have advocated for a balanced nutritional plate model to promote health. Inspired by this research, our paper introduces an innovative Image-Based Dietary Assessment system aimed at evaluating the healthiness of meals through image analysis. Our system employs advanced image segmentation and classification techniques to analyze food items on a plate, assess their proportions, and calculate meal adherence to Harvard's healthy eating recommendations. This approach leverages machine learning and nutritional science to empower individuals with actionable insights for healthier eating choices. Our four-step framework involves segmenting the image, classifying the items, conducting a nutritional assessment based on the Harvard Healthy Eating Plate research, and offering tailored recommendations. The prototype system has shown promising results in promoting healthier eating habits by providing an accessible, evidence-based tool for dietary assessment.

Title: DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions

• Authors: Guangrun Wang, Changlin Li, Liuchun Yuan, Jiefeng Peng, Xiaoyu Xian, Xiaodan Liang, Xiaojun Chang, Liang Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01326
• Pdf URL: https://arxiv.org/pdf/2403.01326
• Copy Paste: [[2403.01326]] DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions(https://arxiv.org/abs/2403.01326)
• Keywords: transformer
• Abstract: Neural Architecture Search (NAS), aiming at automatically designing neural architectures by machines, has been considered a key step toward automatic machine learning. One notable NAS branch is the weight-sharing NAS, which significantly improves search efficiency and allows NAS algorithms to run on ordinary computers. Despite receiving high expectations, this category of methods suffers from low search effectiveness. By employing a generalization boundedness tool, we demonstrate that the devil behind this drawback is the untrustworthy architecture rating with the oversized search space of the possible architectures. Addressing this problem, we modularize a large search space into blocks with small search spaces and develop a family of models with the distilling neural architecture (DNA) techniques. These proposed models, namely a DNA family, are capable of resolving multiple dilemmas of the weight-sharing NAS, such as scalability, efficiency, and multi-modal compatibility. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using heuristic algorithms. Moreover, under a certain computational complexity constraint, our method can seek architectures with different depths and widths. Extensive experimental evaluations show that our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively. Additionally, we provide in-depth empirical analysis and insights into neural architecture ratings. Codes available: \url{https://github.com/changlin31/DNA}.

Title: Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models

• Authors: Neta Shaul, Uriel Singer, Ricky T. Q. Chen, Matthew Le, Ali Thabet, Albert Pumarola, Yaron Lipman
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01329
• Pdf URL: https://arxiv.org/pdf/2403.01329
• Copy Paste: [[2403.01329]] Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models(https://arxiv.org/abs/2403.01329)
• Keywords: diffusion
• Abstract: This paper introduces Bespoke Non-Stationary (BNS) Solvers, a solver distillation approach to improve sample efficiency of Diffusion and Flow models. BNS solvers are based on a family of non-stationary solvers that provably subsumes existing numerical ODE solvers and consequently demonstrate considerable improvement in sample approximation (PSNR) over these baselines. Compared to model distillation, BNS solvers benefit from a tiny parameter space ($<$200 parameters), fast optimization (two orders of magnitude faster), maintain diversity of samples, and in contrast to previous solver distillation approaches nearly close the gap from standard distillation methods such as Progressive Distillation in the low-medium NFE regime. For example, BNS solver achieves 45 PSNR / 1.76 FID using 16 NFE in class-conditional ImageNet-64. We experimented with BNS solvers for conditional image generation, text-to-image generation, and text-2-audio generation showing significant improvement in sample approximation (PSNR) in all.

Title: LM4OPT: Unveiling the Potential of Large Language Models in Formulating Mathematical Optimization Problems

• Authors: Tasnim Ahmed, Salimur Choudhury
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.01342
• Pdf URL: https://arxiv.org/pdf/2403.01342
• Copy Paste: [[2403.01342]] LM4OPT: Unveiling the Potential of Large Language Models in Formulating Mathematical Optimization Problems(https://arxiv.org/abs/2403.01342)
• Keywords: large language model
• Abstract: In the rapidly evolving field of natural language processing, the translation of linguistic descriptions into mathematical formulation of optimization problems presents a formidable challenge, demanding intricate understanding and processing capabilities from Large Language Models (LLMs). This study compares prominent LLMs, including GPT-3.5, GPT-4, and Llama-2-7b, in zero-shot and one-shot settings for this task. Our findings show GPT-4's superior performance, particularly in the one-shot scenario. A central part of this research is the introduction of `LM4OPT,' a progressive fine-tuning framework for Llama-2-7b that utilizes noisy embeddings and specialized datasets. However, this research highlights a notable gap in the contextual understanding capabilities of smaller models such as Llama-2-7b compared to larger counterparts, especially in processing lengthy and complex input contexts. Our empirical investigation, utilizing the NL4Opt dataset, unveils that GPT-4 surpasses the baseline performance established by previous research, achieving an F1-score of 0.63, solely based on the problem description in natural language, and without relying on any additional named entity information. GPT-3.5 follows closely, both outperforming the fine-tuned Llama-2-7b. These findings not only benchmark the current capabilities of LLMs in a novel application area but also lay the groundwork for future improvements in mathematical formulation of optimization problems from natural language input.

Title: Security and Privacy Enhancing in Blockchain-based IoT Environments via Anonym Auditing

• Authors: Peyman Khordadpour, Saeed Ahmadi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01356
• Pdf URL: https://arxiv.org/pdf/2403.01356
• Copy Paste: [[2403.01356]] Security and Privacy Enhancing in Blockchain-based IoT Environments via Anonym Auditing(https://arxiv.org/abs/2403.01356)
• Keywords: secure, security, privacy, robust
• Abstract: The integration of blockchain technology in Internet of Things (IoT) environments is a revolutionary step towards ensuring robust security and enhanced privacy. This paper delves into the unique challenges and solutions associated with securing blockchain-based IoT systems, with a specific focus on anonymous auditing to reinforce privacy and security. We propose a novel framework that combines the decentralized nature of blockchain with advanced security protocols tailored for IoT contexts. Central to our approach is the implementation of anonymization techniques in auditing processes, ensuring user privacy while maintaining the integrity and transparency of blockchain transactions. We outline the architecture of blockchain in IoT environments, emphasizing the workflow and specific security mechanisms employed. Additionally, we introduce a security protocol that integrates privacy-enhancing tools and anonymous auditing methods, including the use of advanced cryptographic techniques for anonymity. This study also includes a comparative analysis of our proposed framework against existing models in the domain. Our work aims to provide a comprehensive blueprint for enhancing security and privacy in blockchain-based IoT environments, paving the way for more secure and private digital ecosystems.

Title: Depth Estimation Algorithm Based on Transformer-Encoder and Feature Fusion

• Authors: Linhan Xia, Junbang Liu, Tong Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01370
• Pdf URL: https://arxiv.org/pdf/2403.01370
• Copy Paste: [[2403.01370]] Depth Estimation Algorithm Based on Transformer-Encoder and Feature Fusion(https://arxiv.org/abs/2403.01370)
• Keywords: transformer
• Abstract: This research presents a novel depth estimation algorithm based on a Transformer-encoder architecture, tailored for the NYU and KITTI Depth Dataset. This research adopts a transformer model, initially renowned for its success in natural language processing, to capture intricate spatial relationships in visual data for depth estimation tasks. A significant innovation of the research is the integration of a composite loss function that combines Structural Similarity Index Measure (SSIM) with Mean Squared Error (MSE). This combined loss function is designed to ensure the structural integrity of the predicted depth maps relative to the original images (via SSIM) while minimizing pixel-wise estimation errors (via MSE). This research approach addresses the challenges of over-smoothing often seen in MSE-based losses and enhances the model's ability to predict depth maps that are not only accurate but also maintain structural coherence with the input images. Through rigorous training and evaluation using the NYU Depth Dataset, the model demonstrates superior performance, marking a significant advancement in single-image depth estimation, particularly in complex indoor and traffic environments.

Title: SA-MixNet: Structure-aware Mixup and Invariance Learning for Scribble-supervised Road Extraction in Remote Sensing Images

• Authors: Jie Feng, Hao Huang, Junpeng Zhang, Weisheng Dong, Dingwen Zhang, Licheng Jiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01381
• Pdf URL: https://arxiv.org/pdf/2403.01381
• Copy Paste: [[2403.01381]] SA-MixNet: Structure-aware Mixup and Invariance Learning for Scribble-supervised Road Extraction in Remote Sensing Images(https://arxiv.org/abs/2403.01381)
• Keywords: extraction
• Abstract: Mainstreamed weakly supervised road extractors rely on highly confident pseudo-labels propagated from scribbles, and their performance often degrades gradually as the image scenes tend various. We argue that such degradation is due to the poor model's invariance to scenes with different complexities, whereas existing solutions to this problem are commonly based on crafted priors that cannot be derived from scribbles. To eliminate the reliance on such priors, we propose a novel Structure-aware Mixup and Invariance Learning framework (SA-MixNet) for weakly supervised road extraction that improves the model invariance in a data-driven manner. Specifically, we design a structure-aware Mixup scheme to paste road regions from one image onto another for creating an image scene with increased complexity while preserving the road's structural integrity. Then an invariance regularization is imposed on the predictions of constructed and origin images to minimize their conflicts, which thus forces the model to behave consistently on various scenes. Moreover, a discriminator-based regularization is designed for enhancing the connectivity meanwhile preserving the structure of roads. Combining these designs, our framework demonstrates superior performance on the DeepGlobe, Wuhan, and Massachusetts datasets outperforming the state-of-the-art techniques by 1.47%, 2.12%, 4.09% respectively in IoU metrics, and showing its potential of plug-and-play. The code will be made publicly available.

Title: Automatic Question-Answer Generation for Long-Tail Knowledge

• Authors: Rohan Kumar, Youngmin Kim, Sunitha Ravi, Haitian Sun, Christos Faloutsos, Ruslan Salakhutdinov, Minji Yoon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01382
• Pdf URL: https://arxiv.org/pdf/2403.01382
• Copy Paste: [[2403.01382]] Automatic Question-Answer Generation for Long-Tail Knowledge(https://arxiv.org/abs/2403.01382)
• Keywords: large language model
• Abstract: Pretrained Large Language Models (LLMs) have gained significant attention for addressing open-domain Question Answering (QA). While they exhibit high accuracy in answering questions related to common knowledge, LLMs encounter difficulties in learning about uncommon long-tail knowledge (tail entities). Since manually constructing QA datasets demands substantial human resources, the types of existing QA datasets are limited, leaving us with a scarcity of datasets to study the performance of LLMs on tail entities. In this paper, we propose an automatic approach to generate specialized QA datasets for tail entities and present the associated research challenges. We conduct extensive experiments by employing pretrained LLMs on our newly generated long-tail QA datasets, comparing their performance with and without external resources including Wikipedia and Wikidata knowledge graphs.

Title: On the Compressibility of Quantized Large Language Models

• Authors: Yu Mao, Weilan Wang, Hongchao Du, Nan Guan, Chun Jason Xue
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01384
• Pdf URL: https://arxiv.org/pdf/2403.01384
• Copy Paste: [[2403.01384]] On the Compressibility of Quantized Large Language Models(https://arxiv.org/abs/2403.01384)
• Keywords: privacy, large language model
• Abstract: Deploying Large Language Models (LLMs) on edge or mobile devices offers significant benefits, such as enhanced data privacy and real-time processing capabilities. However, it also faces critical challenges due to the substantial memory requirement of LLMs. Quantization is an effective way of reducing the model size while maintaining good performance. However, even after quantization, LLMs may still be too big to fit entirely into the limited memory of edge or mobile devices and have to be partially loaded from the storage to complete the inference. In this case, the I/O latency of model loading becomes the bottleneck of the LLM inference latency. In this work, we take a preliminary step of studying applying data compression techniques to reduce data movement and thus speed up the inference of quantized LLM on memory-constrained devices. In particular, we discussed the compressibility of quantized LLMs, the trade-off between the compressibility and performance of quantized LLMs, and opportunities to optimize both of them jointly.

Title: A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications

• Authors: Wei Guo, Fuzhen Zhuang, Xiao Zhang, Yiqi Tong, Jin Dong
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.01387
• Pdf URL: https://arxiv.org/pdf/2403.01387
• Copy Paste: [[2403.01387]] A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications(https://arxiv.org/abs/2403.01387)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) is a novel distributed machine learning paradigm that enables participants to collaboratively train a centralized model with privacy preservation by eliminating the requirement of data sharing. In practice, FL often involves multiple participants and requires the third party to aggregate global information to guide the update of the target participant. Therefore, many FL methods do not work well due to the training and test data of each participant may not be sampled from the same feature space and the same underlying distribution. Meanwhile, the differences in their local devices (system heterogeneity), the continuous influx of online data (incremental data), and labeled data scarcity may further influence the performance of these methods. To solve this problem, federated transfer learning (FTL), which integrates transfer learning (TL) into FL, has attracted the attention of numerous researchers. However, since FL enables a continuous share of knowledge among participants with each communication round while not allowing local data to be accessed by other participants, FTL faces many unique challenges that are not present in TL. In this survey, we focus on categorizing and reviewing the current progress on federated transfer learning, and outlining corresponding solutions and applications. Furthermore, the common setting of FTL scenarios, available datasets, and significant related research are summarized in this survey.

Title: Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answering

• Authors: Armin Toroghi, Willis Guo, Mohammad Mahdi Abdollah Pour, Scott Sanner
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01390
• Pdf URL: https://arxiv.org/pdf/2403.01390
• Copy Paste: [[2403.01390]] Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answering(https://arxiv.org/abs/2403.01390)
• Keywords: large language model
• Abstract: Knowledge Graph Question Answering (KGQA) methods seek to answer Natural Language questions using the relational information stored in Knowledge Graphs (KGs). With the recent advancements of Large Language Models (LLMs) and their remarkable reasoning abilities, there is a growing trend to leverage them for KGQA. However, existing methodologies have only focused on answering factual questions, e.g., "In which city was Silvio Berlusconi's first wife born?", leaving questions involving commonsense reasoning that real-world users may pose more often, e.g., "Do I need separate visas to see the Venus of Willendorf and attend the Olympics this summer?" unaddressed. In this work, we first observe that existing LLM-based methods for KGQA struggle with hallucination on such questions, especially on queries targeting long-tail entities (e.g., non-mainstream and recent entities), thus hindering their applicability in real-world applications especially since their reasoning processes are not easily verifiable. In response, we propose Right for Right Reasons (R3), a commonsense KGQA methodology that allows for a verifiable reasoning procedure by axiomatically surfacing intrinsic commonsense knowledge of LLMs and grounding every factual reasoning step on KG triples. Through experimental evaluations across three different tasks--question answering, claim verification, and preference matching--our findings showcase R3 as a superior approach, outperforming existing methodologies and notably reducing instances of hallucination and reasoning errors.

Title: CR-LT-KGQA: A Knowledge Graph Question Answering Dataset Requiring Commonsense Reasoning and Long-Tail Knowledge

• Authors: Willis Guo, Armin Toroghi, Scott Sanner
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01395
• Pdf URL: https://arxiv.org/pdf/2403.01395
• Copy Paste: [[2403.01395]] CR-LT-KGQA: A Knowledge Graph Question Answering Dataset Requiring Commonsense Reasoning and Long-Tail Knowledge(https://arxiv.org/abs/2403.01395)
• Keywords: large language model
• Abstract: Knowledge graph question answering (KGQA) is a well-established field that seeks to provide factual answers to natural language (NL) questions by leveraging knowledge graphs (KGs). However, existing KGQA datasets suffer from two significant limitations: (1) no existing KGQA dataset requires commonsense reasoning to arrive at an answer and (2) existing KGQA datasets focus on popular entities for which large language models (LLMs) can directly answer without hallucinating and without leveraging the KG. In this work, we seek a novel KGQA dataset that supports commonsense reasoning and focuses on long-tail entities (e.g., non-mainstream and recent entities) where LLMs frequently hallucinate, and thus create the need for novel methodologies that leverage the KG for factual and attributable commonsense inference. We create a novel Commonsense Reasoning (CR) and Long-Tail (LT) KGQA dataset with two subtasks -- question answering and claim verification -- that address both limitations (1) and (2). We construct CR-LT-KGQA by building extensions to existing reasoning datasets StrategyQA and CREAK over Wikidata. While existing KGQA methods are not applicable due to their lack of commonsense inference support, baseline evaluation of LLMs on CR-LT KGQA demonstrate a high rate of hallucination. Thus, CR-LT KGQA poses significant challenges for hallucination-prone LLMs, hence paving the way for future commonsense KGQA research to provide accurate and factual answers for long-tail entities in the era of LLMs.

Title: Region-Transformer: Self-Attention Region Based Class-Agnostic Point Cloud Segmentation

• Authors: Dipesh Gyawali, Jian Zhang, BB Karki
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.01407
• Pdf URL: https://arxiv.org/pdf/2403.01407
• Copy Paste: [[2403.01407]] Region-Transformer: Self-Attention Region Based Class-Agnostic Point Cloud Segmentation(https://arxiv.org/abs/2403.01407)
• Keywords: transformer, segmentation
• Abstract: Point cloud segmentation, which helps us understand the environment of specific structures and objects, can be performed in class-specific and class-agnostic ways. We propose a novel region-based transformer model called Region-Transformer for performing class-agnostic point cloud segmentation. The model utilizes a region-growth approach and self-attention mechanism to iteratively expand or contract a region by adding or removing points. It is trained on simulated point clouds with instance labels only, avoiding semantic labels. Attention-based networks have succeeded in many previous methods of performing point cloud segmentation. However, a region-growth approach with attention-based networks has yet to be used to explore its performance gain. To our knowledge, we are the first to use a self-attention mechanism in a region-growth approach. With the introduction of self-attention to region-growth that can utilize local contextual information of neighborhood points, our experiments demonstrate that the Region-Transformer model outperforms previous class-agnostic and class-specific methods on indoor datasets regarding clustering metrics. The model generalizes well to large-scale scenes. Key advantages include capturing long-range dependencies through self-attention, avoiding the need for semantic labels during training, and applicability to a variable number of objects. The Region-Transformer model represents a promising approach for flexible point cloud segmentation with applications in robotics, digital twinning, and autonomous vehicles.

Title: OVEL: Large Language Model as Memory Manager for Online Video Entity Linking

• Authors: Haiquan Zhao, Xuwu Wang, Shisong Chen, Zhixu Li, Xin Zheng, Yanghua Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01411
• Pdf URL: https://arxiv.org/pdf/2403.01411
• Copy Paste: [[2403.01411]] OVEL: Large Language Model as Memory Manager for Online Video Entity Linking(https://arxiv.org/abs/2403.01411)
• Keywords: robust, large language model
• Abstract: In recent years, multi-modal entity linking (MEL) has garnered increasing attention in the research community due to its significance in numerous multi-modal applications. Video, as a popular means of information transmission, has become prevalent in people's daily lives. However, most existing MEL methods primarily focus on linking textual and visual mentions or offline videos's mentions to entities in multi-modal knowledge bases, with limited efforts devoted to linking mentions within online video content. In this paper, we propose a task called Online Video Entity Linking OVEL, aiming to establish connections between mentions in online videos and a knowledge base with high accuracy and timeliness. To facilitate the research works of OVEL, we specifically concentrate on live delivery scenarios and construct a live delivery entity linking dataset called LIVE. Besides, we propose an evaluation metric that considers timelessness, robustness, and accuracy. Furthermore, to effectively handle OVEL task, we leverage a memory block managed by a Large Language Model and retrieve entity candidates from the knowledge base to augment LLM performance on memory management. The experimental results prove the effectiveness and efficiency of our method.

Title: LUM-ViT: Learnable Under-sampling Mask Vision Transformer for Bandwidth Limited Optical Signal Acquisition

• Authors: Lingfeng Liu, Dong Ni, Hangjie Yuan
• Subjects: cs.CV, eess.IV, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.01412
• Pdf URL: https://arxiv.org/pdf/2403.01412
• Copy Paste: [[2403.01412]] LUM-ViT: Learnable Under-sampling Mask Vision Transformer for Bandwidth Limited Optical Signal Acquisition(https://arxiv.org/abs/2403.01412)
• Keywords: transformer
• Abstract: Bandwidth constraints during signal acquisition frequently impede real-time detection applications. Hyperspectral data is a notable example, whose vast volume compromises real-time hyperspectral detection. To tackle this hurdle, we introduce a novel approach leveraging pre-acquisition modulation to reduce the acquisition volume. This modulation process is governed by a deep learning model, utilizing prior information. Central to our approach is LUM-ViT, a Vision Transformer variant. Uniquely, LUM-ViT incorporates a learnable under-sampling mask tailored for pre-acquisition modulation. To further optimize for optical calculations, we propose a kernel-level weight binarization technique and a three-stage fine-tuning strategy. Our evaluations reveal that, by sampling a mere 10% of the original image pixels, LUM-ViT maintains the accuracy loss within 1.8% on the ImageNet classification task. The method sustains near-original accuracy when implemented on real-world optical hardware, demonstrating its practicality. Code will be available at https://github.com/MaxLLF/LUM-ViT.

Title: Asyn2F: An Asynchronous Federated Learning Framework with Bidirectional Model Aggregation

• Authors: Tien-Dung Cao, Nguyen T. Vuong, Thai Q. Le, Hoang V.N. Dao, Tram Truong-Huu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.01417
• Pdf URL: https://arxiv.org/pdf/2403.01417
• Copy Paste: [[2403.01417]] Asyn2F: An Asynchronous Federated Learning Framework with Bidirectional Model Aggregation(https://arxiv.org/abs/2403.01417)
• Keywords: federate
• Abstract: In federated learning, the models can be trained synchronously or asynchronously. Many research works have focused on developing an aggregation method for the server to aggregate multiple local models into the global model with improved performance. They ignore the heterogeneity of the training workers, which causes the delay in the training of the local models, leading to the obsolete information issue. In this paper, we design and develop Asyn2F, an Asynchronous Federated learning Framework with bidirectional model aggregation. By bidirectional model aggregation, Asyn2F, on one hand, allows the server to asynchronously aggregate multiple local models and results in a new global model. On the other hand, it allows the training workers to aggregate the new version of the global model into the local model, which is being trained even in the middle of a training epoch. We develop Asyn2F considering the practical implementation requirements such as using cloud services for model storage and message queuing protocols for communications. Extensive experiments with different datasets show that the models trained by Asyn2F achieve higher performance compared to the state-of-the-art techniques. The experiments also demonstrate the effectiveness, practicality, and scalability of Asyn2F, making it ready for deployment in real scenarios.

Title: A Simple-but-effective Baseline for Training-free Class-Agnostic Counting

• Authors: Yuhao Lin, Haiming Xu, Lingqiao Liu, Javen Qinfeng Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01418
• Pdf URL: https://arxiv.org/pdf/2403.01418
• Copy Paste: [[2403.01418]] A Simple-but-effective Baseline for Training-free Class-Agnostic Counting(https://arxiv.org/abs/2403.01418)
• Keywords: segmentation
• Abstract: Class-Agnostic Counting (CAC) seeks to accurately count objects in a given image with only a few reference examples. While previous methods achieving this relied on additional training, recent efforts have shown that it's possible to accomplish this without training by utilizing pre-existing foundation models, particularly the Segment Anything Model (SAM), for counting via instance-level segmentation. Although promising, current training-free methods still lag behind their training-based counterparts in terms of performance. In this research, we present a straightforward training-free solution that effectively bridges this performance gap, serving as a strong baseline. The primary contribution of our work lies in the discovery of four key technologies that can enhance performance. Specifically, we suggest employing a superpixel algorithm to generate more precise initial point prompts, utilizing an image encoder with richer semantic knowledge to replace the SAM encoder for representing candidate objects, and adopting a multiscale mechanism and a transductive prototype scheme to update the representation of reference examples. By combining these four technologies, our approach achieves significant improvements over existing training-free methods and delivers performance on par with training-based ones.

Title: The Implicit Bias of Heterogeneity towards Invariance and Causality

• Authors: Yang Xu, Yihong Gu, Cong Fang
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2403.01420
• Pdf URL: https://arxiv.org/pdf/2403.01420
• Copy Paste: [[2403.01420]] The Implicit Bias of Heterogeneity towards Invariance and Causality(https://arxiv.org/abs/2403.01420)
• Keywords: large language model
• Abstract: It is observed empirically that the large language models (LLM), trained with a variant of regression loss using numerous corpus from the Internet, can unveil causal associations to some extent. This is contrary to the traditional wisdom that ``association is not causation'' and the paradigm of traditional causal inference in which prior causal knowledge should be carefully incorporated into the design of methods. It is a mystery why causality, in a higher layer of understanding, can emerge from the regression task that pursues associations. In this paper, we claim the emergence of causality from association-oriented training can be attributed to the coupling effects from the heterogeneity of the source data, stochasticity of training algorithms, and over-parameterization of the learning models. We illustrate such an intuition using a simple but insightful model that learns invariance, a quasi-causality, using regression loss. To be specific, we consider multi-environment low-rank matrix sensing problems where the unknown r-rank ground-truth d*d matrices diverge across the environments but contain a lower-rank invariant, causal part. In this case, running pooled gradient descent will result in biased solutions that only learn associations in general. We show that running large-batch Stochastic Gradient Descent, whose each batch being linear measurement samples randomly selected from a certain environment, can successfully drive the solution towards the invariant, causal solution under certain conditions. This step is related to the relatively strong heterogeneity of the environments, the large step size and noises in the optimization algorithm, and the over-parameterization of the model. In summary, we unveil another implicit bias that is a result of the symbiosis between the heterogeneity of data and modern algorithms, which is, to the best of our knowledge, first in the literature.

Title: Collective Certified Robustness against Graph Injection Attacks

• Authors: Yuni Lai, Bailin Pan, Kaihuang Chen, Yancheng Yuan, Kai Zhou
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01423
• Pdf URL: https://arxiv.org/pdf/2403.01423
• Copy Paste: [[2403.01423]] Collective Certified Robustness against Graph Injection Attacks(https://arxiv.org/abs/2403.01423)
• Keywords: defense, attack, robust
• Abstract: We investigate certified robustness for GNNs under graph injection attacks. Existing research only provides sample-wise certificates by verifying each node independently, leading to very limited certifying performance. In this paper, we present the first collective certificate, which certifies a set of target nodes simultaneously. To achieve it, we formulate the problem as a binary integer quadratic constrained linear programming (BQCLP). We further develop a customized linearization technique that allows us to relax the BQCLP into linear programming (LP) that can be efficiently solved. Through comprehensive experiments, we demonstrate that our collective certification scheme significantly improves certification performance with minimal computational overhead. For instance, by solving the LP within 1 minute on the Citeseer dataset, we achieve a significant increase in the certified ratio from 0.0% to 81.2% when the injected node number is 5% of the graph size. Our step marks a crucial step towards making provable defense more practical.

Title: Introduction to Algogens

• Authors: Amir Shachar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01426
• Pdf URL: https://arxiv.org/pdf/2403.01426
• Copy Paste: [[2403.01426]] Introduction to Algogens(https://arxiv.org/abs/2403.01426)
• Keywords: security, generative
• Abstract: This book introduces the concept of Algogens, a promising integration of generative AI with traditional algorithms aimed at improving problem-solving techniques across various fields. It provides an accessible overview of how Algogens combine AI's innovative potential with algorithms' reliability to tackle complex challenges more effectively than either could alone. The text explores the basics of Algogens, their development, applications, and advantages, such as better adaptability and efficiency. Through examples and case studies, readers will learn about Algogens' practical uses today and their potential for future cybersecurity, healthcare, and environmental science innovation. Acknowledging new technologies' challenges and ethical considerations, the book offers a balanced look at the prospects and obstacles facing Algogens. It invites a broad audience, including experts and newcomers, to engage with the topic and consider Algogens' role in advancing our problem-solving capabilities. This work is presented as a starting point for anyone interested in the intersection of AI and algorithms, encouraging further exploration and discussion on this emerging field. It aims to spark curiosity and contribute to the ongoing conversation about how technology can evolve to meet the complex demands of the AI era.

Title: On Diffusion Process in SE(3)-invariant Space

• Authors: Zihan Zhou, Ruiying Liu, Jiachen Zheng, Xiaoxue Wang, Tianshu Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01430
• Pdf URL: https://arxiv.org/pdf/2403.01430
• Copy Paste: [[2403.01430]] On Diffusion Process in SE(3)-invariant Space(https://arxiv.org/abs/2403.01430)
• Keywords: diffusion
• Abstract: Sampling viable 3D structures (e.g., molecules and point clouds) with SE(3)-invariance using diffusion-based models proved promising in a variety of real-world applications, wherein SE(3)-invariant properties can be naturally characterized by the inter-point distance manifold. However, due to the non-trivial geometry, we still lack a comprehensive understanding of the diffusion mechanism within such SE(3)-invariant space. This study addresses this gap by mathematically delineating the diffusion mechanism under SE(3)-invariance, via zooming into the interaction behavior between coordinates and the inter-point distance manifold through the lens of differential geometry. Upon this analysis, we propose accurate and projection-free diffusion SDE and ODE accordingly. Such formulations enable enhancing the performance and the speed of generation pathways; meanwhile offering valuable insights into other systems incorporating SE(3)-invariance.

Title: Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge

• Authors: Heydar Soudani, Evangelos Kanoulas, Faegheh Hasibi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01432
• Pdf URL: https://arxiv.org/pdf/2403.01432
• Copy Paste: [[2403.01432]] Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge(https://arxiv.org/abs/2403.01432)
• Keywords: large language model
• Abstract: Large language models (LLMs) memorize a vast amount of factual knowledge, exhibiting strong performance across diverse tasks and domains. However, it has been observed that the performance diminishes when dealing with less-popular or low-frequency concepts and entities, for example in domain specific applications. The two prominent approaches to enhance the performance of LLMs on low-frequent topics are: Retrieval Augmented Generation (RAG) and fine-tuning (FT) over synthetic data. This paper explores and evaluates the impact of RAG and FT on customizing LLMs in handling low-frequency entities on question answering task. Our findings indicate that FT significantly boosts the performance across entities of varying popularity, especially in the most and least popular groups, while RAG surpasses other methods. Additionally, the success of both RAG and FT approaches is amplified by advancements in retrieval and data augmentation techniques. We release our data and code at https://github.com/HeydarSoudani/RAGvsFT.

Title: GPTSee: Enhancing Moment Retrieval and Highlight Detection via Description-Based Similarity Features

• Authors: Yunzhuo Sun, Yifang Xu, Zien Xie, Yukun Shu, Sidan Du
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01437
• Pdf URL: https://arxiv.org/pdf/2403.01437
• Copy Paste: [[2403.01437]] GPTSee: Enhancing Moment Retrieval and Highlight Detection via Description-Based Similarity Features(https://arxiv.org/abs/2403.01437)
• Keywords: transformer, large language model
• Abstract: Moment retrieval (MR) and highlight detection (HD) aim to identify relevant moments and highlights in video from corresponding natural language query. Large language models (LLMs) have demonstrated proficiency in various computer vision tasks. However, existing methods for MR\&HD have not yet been integrated with LLMs. In this letter, we propose a novel two-stage model that takes the output of LLMs as the input to the second-stage transformer encoder-decoder. First, MiniGPT-4 is employed to generate the detailed description of the video frame and rewrite the query statement, fed into the encoder as new features. Then, semantic similarity is computed between the generated description and the rewritten queries. Finally, continuous high-similarity video frames are converted into span anchors, serving as prior position information for the decoder. Experiments demonstrate that our approach achieves a state-of-the-art result, and by using only span anchors and similarity scores as outputs, positioning accuracy outperforms traditional methods, like Moment-DETR.

Title: Privacy-Preserving Collaborative Split Learning Framework for Smart Grid Load Forecasting

• Authors: Asif Iqbal, Prosanta Gope, Biplab Sikdar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01438
• Pdf URL: https://arxiv.org/pdf/2403.01438
• Copy Paste: [[2403.01438]] Privacy-Preserving Collaborative Split Learning Framework for Smart Grid Load Forecasting(https://arxiv.org/abs/2403.01438)
• Keywords: security, privacy, transformer
• Abstract: Accurate load forecasting is crucial for energy management, infrastructure planning, and demand-supply balancing. Smart meter data availability has led to the demand for sensor-based load forecasting. Conventional ML allows training a single global model using data from multiple smart meters requiring data transfer to a central server, raising concerns for network requirements, privacy, and security. We propose a split learning-based framework for load forecasting to alleviate this issue. We split a deep neural network model into two parts, one for each Grid Station (GS) responsible for an entire neighbourhood's smart meters and the other for the Service Provider (SP). Instead of sharing their data, client smart meters use their respective GSs' model split for forward pass and only share their activations with the GS. Under this framework, each GS is responsible for training a personalized model split for their respective neighbourhoods, whereas the SP can train a single global or personalized model for each GS. Experiments show that the proposed models match or exceed a centrally trained model's performance and generalize well. Privacy is analyzed by assessing information leakage between data and shared activations of the GS model split. Additionally, differential privacy enhances local data privacy while examining its impact on performance. A transformer model is used as our base learner.

Title: GuardT2I: Defending Text-to-Image Models from Adversarial Prompts

• Authors: Yijun Yang, Ruiyuan Gao, Xiao Yang, Jianyuan Zhong, Qiang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01446
• Pdf URL: https://arxiv.org/pdf/2403.01446
• Copy Paste: [[2403.01446]] GuardT2I: Defending Text-to-Image Models from Adversarial Prompts(https://arxiv.org/abs/2403.01446)
• Keywords: robust, generative, large language model
• Abstract: Recent advancements in Text-to-Image (T2I) models have raised significant safety concerns about their potential misuse for generating inappropriate or Not-Safe-For-Work (NSFW) contents, despite existing countermeasures such as NSFW classifiers or model fine-tuning for inappropriate concept removal. Addressing this challenge, our study unveils GuardT2I, a novel moderation framework that adopts a generative approach to enhance T2I models' robustness against adversarial prompts. Instead of making a binary classification, GuardT2I utilizes a Large Language Model (LLM) to conditionally transform text guidance embeddings within the T2I models into natural language for effective adversarial prompt detection, without compromising the models' inherent performance. Our extensive experiments reveal that GuardT2I outperforms leading commercial solutions like OpenAI-Moderation and Microsoft Azure Moderator by a significant margin across diverse adversarial scenarios.

Title: Enhancing Data Provenance and Model Transparency in Federated Learning Systems - A Database Approach

• Authors: Michael Gu, Ramasoumya Naraparaju, Dongfang Zhao
• Subjects: cs.CR, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01451
• Pdf URL: https://arxiv.org/pdf/2403.01451
• Copy Paste: [[2403.01451]] Enhancing Data Provenance and Model Transparency in Federated Learning Systems - A Database Approach(https://arxiv.org/abs/2403.01451)
• Keywords: secure, security, privacy, protect, federate, explainability
• Abstract: Federated Learning (FL) presents a promising paradigm for training machine learning models across decentralized edge devices while preserving data privacy. Ensuring the integrity and traceability of data across these distributed environments, however, remains a critical challenge. The ability to create transparent artificial intelligence, such as detailing the training process of a machine learning model, has become an increasingly prominent concern due to the large number of sensitive (hyper)parameters it utilizes; thus, it is imperative to strike a reasonable balance between openness and the need to protect sensitive information. In this paper, we propose one of the first approaches to enhance data provenance and model transparency in federated learning systems. Our methodology leverages a combination of cryptographic techniques and efficient model management to track the transformation of data throughout the FL process, and seeks to increase the reproducibility and trustworthiness of a trained FL model. We demonstrate the effectiveness of our approach through experimental evaluations on diverse FL scenarios, showcasing its ability to tackle accountability and explainability across the board. Our findings show that our system can greatly enhance data transparency in various FL environments by storing chained cryptographic hashes and client model snapshots in our proposed design for data decoupled FL. This is made possible by also employing multiple optimization techniques which enables comprehensive data provenance without imposing substantial computational loads. Extensive experimental results suggest that integrating a database subsystem into federated learning systems can improve data provenance in an efficient manner, encouraging secure FL adoption in privacy-sensitive applications and paving the way for future advancements in FL transparency and security features.

Title: One-Step Multi-View Clustering Based on Transition Probability

• Authors: Wenhui Zhao, Quanxue Gao, Guangfei Li, Cheng Deng, Ming Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01460
• Pdf URL: https://arxiv.org/pdf/2403.01460
• Copy Paste: [[2403.01460]] One-Step Multi-View Clustering Based on Transition Probability(https://arxiv.org/abs/2403.01460)
• Keywords: robust, interpretability
• Abstract: The large-scale multi-view clustering algorithms, based on the anchor graph, have shown promising performance and efficiency and have been extensively explored in recent years. Despite their successes, current methods lack interpretability in the clustering process and do not sufficiently consider the complementary information across different views. To address these shortcomings, we introduce the One-Step Multi-View Clustering Based on Transition Probability (OSMVC-TP). This method adopts a probabilistic approach, which leverages the anchor graph, representing the transition probabilities from samples to anchor points. Our method directly learns the transition probabilities from anchor points to categories, and calculates the transition probabilities from samples to categories, thus obtaining soft label matrices for samples and anchor points, enhancing the interpretability of clustering. Furthermore, to maintain consistency in labels across different views, we apply a Schatten p-norm constraint on the tensor composed of the soft labels. This approach effectively harnesses the complementary information among the views. Extensive experiments have confirmed the effectiveness and robustness of OSMVC-TP.

Title: Collaborate to Adapt: Source-Free Graph Domain Adaptation via Bi-directional Adaptation

• Authors: Zhen Zhang, Meihan Liu, Anhui Wang, Hongyang Chen, Zhao Li, Jiajun Bu, Bingsheng He
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01467
• Pdf URL: https://arxiv.org/pdf/2403.01467
• Copy Paste: [[2403.01467]] Collaborate to Adapt: Source-Free Graph Domain Adaptation via Bi-directional Adaptation(https://arxiv.org/abs/2403.01467)
• Keywords: privacy
• Abstract: Unsupervised Graph Domain Adaptation (UGDA) has emerged as a practical solution to transfer knowledge from a label-rich source graph to a completely unlabelled target graph. However, most methods require a labelled source graph to provide supervision signals, which might not be accessible in the real-world settings due to regulations and privacy concerns. In this paper, we explore the scenario of source-free unsupervised graph domain adaptation, which tries to address the domain adaptation problem without accessing the labelled source graph. Specifically, we present a novel paradigm called GraphCTA, which performs model adaptation and graph adaptation collaboratively through a series of procedures: (1) conduct model adaptation based on node's neighborhood predictions in target graph considering both local and global information; (2) perform graph adaptation by updating graph structure and node attributes via neighborhood contrastive learning; and (3) the updated graph serves as an input to facilitate the subsequent iteration of model adaptation, thereby establishing a collaborative loop between model adaptation and graph adaptation. Comprehensive experiments are conducted on various public datasets. The experimental results demonstrate that our proposed model outperforms recent source-free baselines by large margins.

Title: KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations

• Authors: Sunjun Kweon, Byungjin Choi, Minkyu Kim, Rae Woong Park, Edward Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01469
• Pdf URL: https://arxiv.org/pdf/2403.01469
• Copy Paste: [[2403.01469]] KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations(https://arxiv.org/abs/2403.01469)
• Keywords: large language model
• Abstract: We introduce KorMedMCQA, the first Korean multiple-choice question answering (MCQA) benchmark derived from Korean healthcare professional licensing examinations, covering from the year 2012 to year 2023. This dataset consists of a selection of questions from the license examinations for doctors, nurses, and pharmacists, featuring a diverse array of subjects. We conduct baseline experiments on various large language models, including proprietary/open-source, multilingual/Korean-additional pretrained, and clinical context pretrained models, highlighting the potential for further enhancements. We make our data publicly available on HuggingFace and provide a evaluation script via LM-Harness, inviting further exploration and advancement in Korean healthcare environments.

Title: Is in-domain data beneficial in transfer learning for landmarks detection in x-ray images?

• Authors: Roberto Di Via, Matteo Santacesaria, Francesca Odone, Vito Paolo Pastore
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01470
• Pdf URL: https://arxiv.org/pdf/2403.01470
• Copy Paste: [[2403.01470]] Is in-domain data beneficial in transfer learning for landmarks detection in x-ray images?(https://arxiv.org/abs/2403.01470)
• Keywords: robust
• Abstract: In recent years, deep learning has emerged as a promising technique for medical image analysis. However, this application domain is likely to suffer from a limited availability of large public datasets and annotations. A common solution to these challenges in deep learning is the usage of a transfer learning framework, typically with a fine-tuning protocol, where a large-scale source dataset is used to pre-train a model, further fine-tuned on the target dataset. In this paper, we present a systematic study analyzing whether the usage of small-scale in-domain x-ray image datasets may provide any improvement for landmark detection over models pre-trained on large natural image datasets only. We focus on the multi-landmark localization task for three datasets, including chest, head, and hand x-ray images. Our results show that using in-domain source datasets brings marginal or no benefit with respect to an ImageNet out-of-domain pre-training. Our findings can provide an indication for the development of robust landmark detection systems in medical images when no large annotated dataset is available.

Title: Preserving correlations: A statistical method for generating synthetic data

• Authors: Nicklas Jävergård, Rainey Lyons, Adrian Muntean, Jonas Forsman
• Subjects: cs.LG, math.PR, physics.data-an
• Abstract URL: https://arxiv.org/abs/2403.01471
• Pdf URL: https://arxiv.org/pdf/2403.01471
• Copy Paste: [[2403.01471]] Preserving correlations: A statistical method for generating synthetic data(https://arxiv.org/abs/2403.01471)
• Keywords: privacy
• Abstract: We propose a method to generate statistically representative synthetic data. The main goal is to be able to maintain in the synthetic dataset the correlations of the features present in the original one, while offering a comfortable privacy level that can be eventually tailored on specific customer demands. We describe in detail our algorithm used both for the analysis of the original dataset and for the generation of the synthetic data points. The approach is tested using a large energy-related dataset. We obtain good results both qualitatively (e.g. via vizualizing correlation maps) and quantitatively (in terms of suitable $\ell^1$-type error norms used as evaluation metrics). The proposed methodology is general in the sense that it does not rely on the used test dataset. We expect it to be applicable in a much broader context than indicated here.

Title: WARDEN: Multi-Directional Backdoor Watermarks for Embedding-as-a-Service Copyright Protection

• Authors: Anudeex Shetty, Yue Teng, Ke He, Qiongkai Xu
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01472
• Pdf URL: https://arxiv.org/pdf/2403.01472
• Copy Paste: [[2403.01472]] WARDEN: Multi-Directional Backdoor Watermarks for Embedding-as-a-Service Copyright Protection(https://arxiv.org/abs/2403.01472)
• Keywords: protect, defense, attack, steal, extraction, watermark
• Abstract: Embedding as a Service (EaaS) has become a widely adopted solution, which offers feature extraction capabilities for addressing various downstream tasks in Natural Language Processing (NLP). Prior studies have shown that EaaS can be prone to model extraction attacks; nevertheless, this concern could be mitigated by adding backdoor watermarks to the text embeddings and subsequently verifying the attack models post-publication. Through the analysis of the recent watermarking strategy for EaaS, EmbMarker, we design a novel CSE (Clustering, Selection, Elimination) attack that removes the backdoor watermark while maintaining the high utility of embeddings, indicating that the previous watermarking approach can be breached. In response to this new threat, we propose a new protocol to make the removal of watermarks more challenging by incorporating multiple possible watermark directions. Our defense approach, WARDEN, notably increases the stealthiness of watermarks and empirically has been shown effective against CSE attack.

Title: Representation Learning on Heterophilic Graph with Directional Neighborhood Attention

• Authors: Qincheng Lu, Jiaqi Zhu, Sitao Luan, Xiao-Wen Chang
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.01475
• Pdf URL: https://arxiv.org/pdf/2403.01475
• Copy Paste: [[2403.01475]] Representation Learning on Heterophilic Graph with Directional Neighborhood Attention(https://arxiv.org/abs/2403.01475)
• Keywords: diffusion
• Abstract: Graph Attention Network (GAT) is one of the most popular Graph Neural Network (GNN) architecture, which employs the attention mechanism to learn edge weights and has demonstrated promising performance in various applications. However, since it only incorporates information from immediate neighborhood, it lacks the ability to capture long-range and global graph information, leading to unsatisfactory performance on some datasets, particularly on heterophilic graphs. To address this limitation, we propose the Directional Graph Attention Network (DGAT) in this paper. DGAT is able to combine the feature-based attention with the global directional information extracted from the graph topology. To this end, a new class of Laplacian matrices is proposed which can provably reduce the diffusion distance between nodes. Based on the new Laplacian, topology-guided neighbour pruning and edge adding mechanisms are proposed to remove the noisy and capture the helpful long-range neighborhood information. Besides, a global directional attention is designed to enable a topological-aware information propagation. The superiority of the proposed DGAT over the baseline GAT has also been verified through experiments on real-world benchmarks and synthetic data sets. It also outperforms the state-of-the-art (SOTA) models on 6 out of 7 real-world benchmark datasets.

Title: CCC: Color Classified Colorization

• Authors: Mrityunjoy Gain, Avi Deb Raha, Rameswar Debnath
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01476
• Pdf URL: https://arxiv.org/pdf/2403.01476
• Copy Paste: [[2403.01476]] CCC: Color Classified Colorization(https://arxiv.org/abs/2403.01476)
• Keywords: generative
• Abstract: Automatic colorization of gray images with objects of different colors and sizes is challenging due to inter- and intra-object color variation and the small area of the main objects due to extensive backgrounds. The learning process often favors dominant features, resulting in a biased model. In this paper, we formulate the colorization problem into a multinomial classification problem and then apply a weighted function to classes. We propose a set of formulas to transform color values into color classes and vice versa. Class optimization and balancing feature distribution are the keys for good performance. Observing class appearance on various extremely large-scale real-time images in practice, we propose 215 color classes for our colorization task. During training, we propose a class-weighted function based on true class appearance in each batch to ensure proper color saturation of individual objects. We establish a trade-off between major and minor classes to provide orthodox class prediction by eliminating major classes' dominance over minor classes. As we apply regularization to enhance the stability of the minor class, occasional minor noise may appear at the object's edges. We propose a novel object-selective color harmonization method empowered by the SAM to refine and enhance these edges. We propose a new color image evaluation metric, the Chromatic Number Ratio (CNR), to quantify the richness of color components. We compare our proposed model with state-of-the-art models using five different datasets: ADE, Celeba, COCO, Oxford 102 Flower, and ImageNet, in both qualitative and quantitative approaches. The experimental results show that our proposed model outstrips other models in visualization and CNR measurement criteria while maintaining satisfactory performance in regression (MSE, PSNR), similarity (SSIM, LPIPS, UIQI), and generative criteria (FID).

Title: Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation

• Authors: Heegon Jin, Seonil Son, Jemin Park, Youngseok Kim, Hyungjong Noh, Yeonsoo Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01479
• Pdf URL: https://arxiv.org/pdf/2403.01479
• Copy Paste: [[2403.01479]] Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation(https://arxiv.org/abs/2403.01479)
• Keywords: transformer
• Abstract: The advent of scalable deep models and large datasets has improved the performance of Neural Machine Translation. Knowledge Distillation (KD) enhances efficiency by transferring knowledge from a teacher model to a more compact student model. However, KD approaches to Transformer architecture often rely on heuristics, particularly when deciding which teacher layers to distill from. In this paper, we introduce the 'Align-to-Distill' (A2D) strategy, designed to address the feature mapping problem by adaptively aligning student attention heads with their teacher counterparts during training. The Attention Alignment Module in A2D performs a dense head-by-head comparison between student and teacher attention heads across layers, turning the combinatorial mapping heuristics into a learning problem. Our experiments show the efficacy of A2D, demonstrating gains of up to +3.61 and +0.63 BLEU points for WMT-2022 De->Dsb and WMT-2014 En->De, respectively, compared to Transformer baselines.

Title: Infusing Knowledge into Large Language Models with Contextual Prompts

• Authors: Kinshuk Vasisht, Balaji Ganesan, Vikas Kumar, Vasudha Bhatnagar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01481
• Pdf URL: https://arxiv.org/pdf/2403.01481
• Copy Paste: [[2403.01481]] Infusing Knowledge into Large Language Models with Contextual Prompts(https://arxiv.org/abs/2403.01481)
• Keywords: large language model
• Abstract: Knowledge infusion is a promising method for enhancing Large Language Models for domain-specific NLP tasks rather than pre-training models over large data from scratch. These augmented LLMs typically depend on additional pre-training or knowledge prompts from an existing knowledge graph, which is impractical in many applications. In contrast, knowledge infusion directly from relevant documents is more generalisable and alleviates the need for structured knowledge graphs while also being useful for entities that are usually not found in any knowledge graph. With this motivation, we propose a simple yet generalisable approach for knowledge infusion by generating prompts from the context in the input text. Our experiments show the effectiveness of our approach which we evaluate by probing the fine-tuned LLMs.

Title: EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation

• Authors: Chanyoung Kim, Woojung Han, Dayun Ju, Seong Jae Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01482
• Pdf URL: https://arxiv.org/pdf/2403.01482
• Copy Paste: [[2403.01482]] EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation(https://arxiv.org/abs/2403.01482)
• Keywords: transformer, segmentation
• Abstract: Semantic segmentation has innately relied on extensive pixel-level labeled annotated data, leading to the emergence of unsupervised methodologies. Among them, leveraging self-supervised Vision Transformers for unsupervised semantic segmentation (USS) has been making steady progress with expressive deep features. Yet, for semantically segmenting images with complex objects, a predominant challenge remains: the lack of explicit object-level semantic encoding in patch-level features. This technical limitation often leads to inadequate segmentation of complex objects with diverse structures. To address this gap, we present a novel approach, EAGLE, which emphasizes object-centric representation learning for unsupervised semantic segmentation. Specifically, we introduce EiCue, a spectral technique providing semantic and structural cues through an eigenbasis derived from the semantic similarity matrix of deep image features and color affinity from an image. Further, by incorporating our object-centric contrastive loss with EiCue, we guide our model to learn object-level representations with intra- and inter-image object-feature consistency, thereby enhancing semantic accuracy. Extensive experiments on COCO-Stuff, Cityscapes, and Potsdam-3 datasets demonstrate the state-of-the-art USS results of EAGLE with accurate and consistent semantic segmentation across complex scenes.

Title: InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understanding

• Authors: Haogeng Liu, Quanzeng You, Xiaotian Han, Yiqi Wang, Bohan Zhai, Yongfei Liu, Yunzhe Tao, Huaibo Huang, Ran He, Hongxia Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01487
• Pdf URL: https://arxiv.org/pdf/2403.01487
• Copy Paste: [[2403.01487]] InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understanding(https://arxiv.org/abs/2403.01487)
• Keywords: robust, large language model
• Abstract: Multimodal Large Language Models (MLLMs) have experienced significant advancements recently. Nevertheless, challenges persist in the accurate recognition and comprehension of intricate details within high-resolution images. Despite being indispensable for the development of robust MLLMs, this area remains underinvestigated. To tackle this challenge, our work introduces InfiMM-HD, a novel architecture specifically designed for processing images of different resolutions with low computational overhead. This innovation facilitates the enlargement of MLLMs to higher-resolution capabilities. InfiMM-HD incorporates a cross-attention module and visual windows to reduce computation costs. By integrating this architectural design with a four-stage training pipeline, our model attains improved visual perception efficiently and cost-effectively. Empirical study underscores the robustness and effectiveness of InfiMM-HD, opening new avenues for exploration in related areas. Codes and models can be found at https://huggingface.co/Infi-MM/infimm-hd

Title: Regeneration Based Training-free Attribution of Fake Images Generated by Text-to-Image Generative Models

• Authors: Meiling Li, Zhenxing Qian, Xinpeng Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01489
• Pdf URL: https://arxiv.org/pdf/2403.01489
• Copy Paste: [[2403.01489]] Regeneration Based Training-free Attribution of Fake Images Generated by Text-to-Image Generative Models(https://arxiv.org/abs/2403.01489)
• Keywords: attack, robust, generative
• Abstract: Text-to-image generative models have recently garnered significant attention due to their ability to generate images based on prompt descriptions. While these models have shown promising performance, concerns have been raised regarding the potential misuse of the generated fake images. In response to this, we have presented a simple yet effective training-free method to attribute fake images generated by text-to-image models to their source models. Given a test image to be attributed, we first inverse the textual prompt of the image, and then put the reconstructed prompt into different candidate models to regenerate candidate fake images. By calculating and ranking the similarity of the test image and the candidate images, we can determine the source of the image. This attribution allows model owners to be held accountable for any misuse of their models. Note that our approach does not limit the number of candidate text-to-image generative models. Comprehensive experiments reveal that (1) Our method can effectively attribute fake images to their source models, achieving comparable attribution performance with the state-of-the-art method; (2) Our method has high scalability ability, which is well adapted to real-world attribution scenarios. (3) The proposed method yields satisfactory robustness to common attacks, such as Gaussian blurring, JPEG compression, and Resizing. We also analyze the factors that influence the attribution performance, and explore the boost brought by the proposed method as a plug-in to improve the performance of existing SOTA. We hope our work can shed some light on the solutions to addressing the source of AI-generated images, as well as to prevent the misuse of text-to-image generative models.

Title: ConvTimeNet: A Deep Hierarchical Fully Convolutional Model for Multivariate Time Series Analysis

• Authors: Mingyue Cheng, Jiqian Yang, Tingyue Pan, Qi Liu, Zhi Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01493
• Pdf URL: https://arxiv.org/pdf/2403.01493
• Copy Paste: [[2403.01493]] ConvTimeNet: A Deep Hierarchical Fully Convolutional Model for Multivariate Time Series Analysis(https://arxiv.org/abs/2403.01493)
• Keywords: transformer, segmentation
• Abstract: This paper introduces ConvTimeNet, a novel deep hierarchical fully convolutional network designed to serve as a general-purpose model for time series analysis. The key design of this network is twofold, designed to overcome the limitations of traditional convolutional networks. Firstly, we propose an adaptive segmentation of time series into sub-series level patches, treating these as fundamental modeling units. This setting avoids the sparsity semantics associated with raw point-level time steps. Secondly, we design a fully convolutional block by skillfully integrating deepwise and pointwise convolution operations, following the advanced building block style employed in Transformer encoders. This backbone network allows for the effective capture of both global sequence and cross-variable dependence, as it not only incorporates the advancements of Transformer architecture but also inherits the inherent properties of convolution. Furthermore, multi-scale representations of given time series instances can be learned by controlling the kernel size flexibly. Extensive experiments are conducted on both time series forecasting and classification tasks. The results consistently outperformed strong baselines in most situations in terms of effectiveness.The code is publicly available.

Title: Learning A Physical-aware Diffusion Model Based on Transformer for Underwater Image Enhancement

• Authors: Chen Zhao, Chenyu Dong, Weiling Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01497
• Pdf URL: https://arxiv.org/pdf/2403.01497
• Copy Paste: [[2403.01497]] Learning A Physical-aware Diffusion Model Based on Transformer for Underwater Image Enhancement(https://arxiv.org/abs/2403.01497)
• Keywords: diffusion, transformer
• Abstract: Underwater visuals undergo various complex degradations, inevitably influencing the efficiency of underwater vision tasks. Recently, diffusion models were employed to underwater image enhancement (UIE) tasks, and gained SOTA performance. However, these methods fail to consider the physical properties and underwater imaging mechanisms in the diffusion process, limiting information completion capacity of diffusion models. In this paper, we introduce a novel UIE framework, named PA-Diff, designed to exploiting the knowledge of physics to guide the diffusion process. PA-Diff consists of Physics Prior Generation (PPG) Branch and Physics-aware Diffusion Transformer (PDT) Branch. Our designed PPG branch is a plug-and-play network to produce the physics prior, which can be integrated into any deep framework. With utilizing the physics prior knowledge to guide the diffusion process, PDT branch can obtain underwater-aware ability and model the complex distribution in real-world underwater scenes. Extensive experiments prove that our method achieves best performance on UIE tasks.

Title: Applying Self-supervised Learning to Network Intrusion Detection for Network Flows with Graph Neural Network

• Authors: Renjie Xu, Guangwei Wu, Weiping Wang, Xing Gao, An He, Zhengpeng Zhang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01501
• Pdf URL: https://arxiv.org/pdf/2403.01501
• Copy Paste: [[2403.01501]] Applying Self-supervised Learning to Network Intrusion Detection for Network Flows with Graph Neural Network(https://arxiv.org/abs/2403.01501)
• Keywords: attack
• Abstract: Graph Neural Networks (GNNs) have garnered intensive attention for Network Intrusion Detection System (NIDS) due to their suitability for representing the network traffic flows. However, most present GNN-based methods for NIDS are supervised or semi-supervised. Network flows need to be manually annotated as supervisory labels, a process that is time-consuming or even impossible, making NIDS difficult to adapt to potentially complex attacks, especially in large-scale real-world scenarios. The existing GNN-based self-supervised methods focus on the binary classification of network flow as benign or not, and thus fail to reveal the types of attack in practice. This paper studies the application of GNNs to identify the specific types of network flows in an unsupervised manner. We first design an encoder to obtain graph embedding, that introduces the graph attention mechanism and considers the edge information as the only essential factor. Then, a self-supervised method based on graph contrastive learning is proposed. The method samples center nodes, and for each center node, generates subgraph by it and its direct neighbor nodes, and corresponding contrastive subgraph from the interpolated graph, and finally constructs positive and negative samples from subgraphs. Furthermore, a structured contrastive loss function based on edge features and graph local topology is introduced. To the best of our knowledge, it is the first GNN-based self-supervised method for the multiclass classification of network flows in NIDS. Detailed experiments conducted on four real-world databases (NF-Bot-IoT, NF-Bot-IoT-v2, NF-CSE-CIC-IDS2018, and NF-CSE-CIC-IDS2018-v2) systematically compare our model with the state-of-the-art supervised and self-supervised models, illustrating the considerable potential of our method. Our code is accessible through https://github.com/renj-xu/NEGSC.

Title: SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation

• Authors: Hongjian Liu, Qingsong Xie, Zhijie Deng, Chen Chen, Shixiang Tang, Fueyang Fu, Zheng-jun Zha, Haonan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01505
• Pdf URL: https://arxiv.org/pdf/2403.01505
• Copy Paste: [[2403.01505]] SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation(https://arxiv.org/abs/2403.01505)
• Keywords: diffusion
• Abstract: The iterative sampling procedure employed by diffusion models (DMs) often leads to significant inference latency. To address this, we propose Stochastic Consistency Distillation (SCott) to enable accelerated text-to-image generation, where high-quality generations can be achieved with just 1-2 sampling steps, and further improvements can be obtained by adding additional steps. In contrast to vanilla consistency distillation (CD) which distills the ordinary differential equation solvers-based sampling process of a pretrained teacher model into a student, SCott explores the possibility and validates the efficacy of integrating stochastic differential equation (SDE) solvers into CD to fully unleash the potential of the teacher. SCott is augmented with elaborate strategies to control the noise strength and sampling process of the SDE solver. An adversarial loss is further incorporated to strengthen the sample quality with rare sampling steps. Empirically, on the MSCOCO-2017 5K dataset with a Stable Diffusion-V1.5 teacher, SCott achieves an FID (Frechet Inceptio Distance) of 22.1, surpassing that (23.4) of the 1-step InstaFlow (Liu et al., 2023) and matching that of 4-step UFOGen (Xue et al., 2023b). Moreover, SCott can yield more diverse samples than other consistency models for high-resolution image generation (Luo et al., 2023a), with up to 16% improvement in a qualified metric. The code and checkpoints are coming soon.

Title: ISSF: The Intelligent Security Service Framework for Cloud-Native Operation

• Authors: Yikuan Yan, Keman Huang, Michael Siegel
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01507
• Pdf URL: https://arxiv.org/pdf/2403.01507
• Copy Paste: [[2403.01507]] ISSF: The Intelligent Security Service Framework for Cloud-Native Operation(https://arxiv.org/abs/2403.01507)
• Keywords: security, defense, attack
• Abstract: The growing system complexity from microservice architectures and the bilateral enhancement of artificial intelligence (AI) for both attackers and defenders presents increasing security challenges for cloud-native operations. In particular, cloud-native operators require a holistic view of the dynamic security posture for the cloud-native environment from a defense aspect. Additionally, both attackers and defenders can adopt advanced AI technologies. This makes the dynamic interaction and benchmark among different intelligent offense and defense strategies more crucial. Hence, following the multi-agent deep reinforcement learning (RL) paradigm, this research develops an agent-based intelligent security service framework (ISSF) for cloud-native operation. It includes a dynamic access graph model to represent the cloud-native environment and an action model to represent offense and defense actions. Then we develop an approach to enable the training, publishing, and evaluating of intelligent security services using diverse deep RL algorithms and training strategies, facilitating their systematic development and benchmark. The experiments demonstrate that our framework can sufficiently model the security posture of a cloud-native system for defenders, effectively develop and quantitatively benchmark different services for both attackers and defenders and guide further service optimization.

Title: Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics

• Authors: Zhu Liu, Cunliang Kong, Ying Liu, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01509
• Pdf URL: https://arxiv.org/pdf/2403.01509
• Copy Paste: [[2403.01509]] Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics(https://arxiv.org/abs/2403.01509)
• Keywords: generative, large language model
• Abstract: Large language models have achieved remarkable success in general language understanding tasks. However, as a family of generative methods with the objective of next token prediction, the semantic evolution with the depth of these models are not fully explored, unlike their predecessors, such as BERT-like architectures. In this paper, we specifically investigate the bottom-up evolution of lexical semantics for a popular LLM, namely Llama2, by probing its hidden states at the end of each layer using a contextualized word identification task. Our experiments show that the representations in lower layers encode lexical semantics, while the higher layers, with weaker semantic induction, are responsible for prediction. This is in contrast to models with discriminative objectives, such as mask language modeling, where the higher layers obtain better lexical semantics. The conclusion is further supported by the monotonic increase in performance via the hidden states for the last meaningless symbols, such as punctuation, in the prompting strategy.

Title: End-to-End Human Instance Matting

• Authors: Qinglin Liu, Shengping Zhang, Quanling Meng, Bineng Zhong, Peiqiang Liu, Hongxun Yao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01510
• Pdf URL: https://arxiv.org/pdf/2403.01510
• Copy Paste: [[2403.01510]] End-to-End Human Instance Matting(https://arxiv.org/abs/2403.01510)
• Keywords: segmentation
• Abstract: Human instance matting aims to estimate an alpha matte for each human instance in an image, which is extremely challenging and has rarely been studied so far. Despite some efforts to use instance segmentation to generate a trimap for each instance and apply trimap-based matting methods, the resulting alpha mattes are often inaccurate due to inaccurate segmentation. In addition, this approach is computationally inefficient due to multiple executions of the matting method. To address these problems, this paper proposes a novel End-to-End Human Instance Matting (E2E-HIM) framework for simultaneous multiple instance matting in a more efficient manner. Specifically, a general perception network first extracts image features and decodes instance contexts into latent codes. Then, a united guidance network exploits spatial attention and semantics embedding to generate united semantics guidance, which encodes the locations and semantic correspondences of all instances. Finally, an instance matting network decodes the image features and united semantics guidance to predict all instance-level alpha mattes. In addition, we construct a large-scale human instance matting dataset (HIM-100K) comprising over 100,000 human images with instance alpha matte labels. Experiments on HIM-100K demonstrate the proposed E2E-HIM outperforms the existing methods on human instance matting with 50% lower errors and 5X faster speed (6 instances in a 640X640 image). Experiments on the PPM-100, RWP-636, and P3M datasets demonstrate that E2E-HIM also achieves competitive performance on traditional human matting.

Title: Neural Graph Generator: Feature-Conditioned Graph Generation using Latent Diffusion Models

• Authors: Iakovos Evdaimon, Giannis Nikolentzos, Michail Chatzianastasis, Hadi Abdine, Michalis Vazirgiannis
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.01535
• Pdf URL: https://arxiv.org/pdf/2403.01535
• Copy Paste: [[2403.01535]] Neural Graph Generator: Feature-Conditioned Graph Generation using Latent Diffusion Models(https://arxiv.org/abs/2403.01535)
• Keywords: diffusion
• Abstract: Graph generation has emerged as a crucial task in machine learning, with significant challenges in generating graphs that accurately reflect specific properties. Existing methods often fall short in efficiently addressing this need as they struggle with the high-dimensional complexity and varied nature of graph properties. In this paper, we introduce the Neural Graph Generator (NGG), a novel approach which utilizes conditioned latent diffusion models for graph generation. NGG demonstrates a remarkable capacity to model complex graph patterns, offering control over the graph generation process. NGG employs a variational graph autoencoder for graph compression and a diffusion process in the latent vector space, guided by vectors summarizing graph statistics. We demonstrate NGG's versatility across various graph generation tasks, showing its capability to capture desired graph properties and generalize to unseen graphs. This work signifies a significant shift in graph generation methodologies, offering a more practical and efficient solution for generating diverse types of graphs with specific characteristics.

Title: Quantized Hierarchical Federated Learning: A Robust Approach to Statistical Heterogeneity

• Authors: Seyed Mohammad Azimi-Abarghouyi, Viktoria Fodor
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2403.01540
• Pdf URL: https://arxiv.org/pdf/2403.01540
• Copy Paste: [[2403.01540]] Quantized Hierarchical Federated Learning: A Robust Approach to Statistical Heterogeneity(https://arxiv.org/abs/2403.01540)
• Keywords: robust, federate
• Abstract: This paper presents a novel hierarchical federated learning algorithm within multiple sets that incorporates quantization for communication-efficiency and demonstrates resilience to statistical heterogeneity. Unlike conventional hierarchical federated learning algorithms, our approach combines gradient aggregation in intra-set iterations with model aggregation in inter-set iterations. We offer a comprehensive analytical framework to evaluate its optimality gap and convergence rate, comparing these aspects with those of conventional algorithms. Additionally, we develop a problem formulation to derive optimal system parameters in a closed-form solution. Our findings reveal that our algorithm consistently achieves high learning accuracy over a range of parameters and significantly outperforms other hierarchical algorithms, particularly in scenarios with heterogeneous data distributions.

Title: Hyperspectral Image Analysis in Single-Modal and Multimodal setting using Deep Learning Techniques

• Authors: Shivam Pande
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01546
• Pdf URL: https://arxiv.org/pdf/2403.01546
• Copy Paste: [[2403.01546]] Hyperspectral Image Analysis in Single-Modal and Multimodal setting using Deep Learning Techniques(https://arxiv.org/abs/2403.01546)
• Keywords: robust, extraction
• Abstract: Hyperspectral imaging provides precise classification for land use and cover due to its exceptional spectral resolution. However, the challenges of high dimensionality and limited spatial resolution hinder its effectiveness. This study addresses these challenges by employing deep learning techniques to efficiently process, extract features, and classify data in an integrated manner. To enhance spatial resolution, we integrate information from complementary modalities such as LiDAR and SAR data through multimodal learning. Moreover, adversarial learning and knowledge distillation are utilized to overcome issues stemming from domain disparities and missing modalities. We also tailor deep learning architectures to suit the unique characteristics of HSI data, utilizing 1D convolutional and recurrent neural networks to handle its continuous spectral dimension. Techniques like visual attention and feedback connections within the architecture bolster the robustness of feature extraction. Additionally, we tackle the issue of limited training samples through self-supervised learning methods, employing autoencoders for dimensionality reduction and exploring semi-supervised learning techniques that leverage unlabeled data. Our proposed approaches are evaluated across various HSI datasets, consistently outperforming existing state-of-the-art techniques.

Title: Constructions of Control Sequence Set for Hierarchical Access in Data Link Network

• Authors: Niu Xianhua, Ma Jiabei, Zhou Enzhi, Wang Yaoxuan, Zeng Bosen, Li Zhiping
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01547
• Pdf URL: https://arxiv.org/pdf/2403.01547
• Copy Paste: [[2403.01547]] Constructions of Control Sequence Set for Hierarchical Access in Data Link Network(https://arxiv.org/abs/2403.01547)
• Keywords: secure
• Abstract: Time slots are a valuable channel resource in the data link network with time division multiple access architecture. The need for finding a secure and efficient way to meet the requirements of large access capacity, differentiated access, maximum utilization of time slot resource and strong anti-eavesdropping ability in data link networks is well motivated.In this paper, a control sequence-based hierarchical access control scheme is proposed, which not only achieves differentiated time slots allocation for the different needs and levels of nodes, but also enhances randomness and anti-interception performance in data link networks.Based on the scheme, a new theoretical bound is derived to characterize parameter relationships for designing optimal hierarchical control sequence(HCS) set. Moreover, two flexible classes of optimal hierarchical control sequence sets are constructed.By our construction, the terminal user in the data link can access hierarchically and randomly and transmit data packets during its own hopping time slots of the successive frames to prevent eavesdropping while maintaining high throughput.

Title: In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation

• Authors: Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, Junxian He
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01548
• Pdf URL: https://arxiv.org/pdf/2403.01548
• Copy Paste: [[2403.01548]] In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation(https://arxiv.org/abs/2403.01548)
• Keywords: large language model
• Abstract: Large language models (LLMs) frequently hallucinate and produce factual errors, yet our understanding of why they make these errors remains limited. In this study, we delve into the underlying mechanisms of LLM hallucinations from the perspective of inner representations, and discover a salient pattern associated with hallucinations: correct generations tend to have sharper context activations in the hidden states of the in-context tokens, compared to the incorrect ones. Leveraging this insight, we propose an entropy-based metric to quantify the ``sharpness'' among the in-context hidden states and incorporate it into the decoding process to formulate a constrained decoding approach. Experiments on various knowledge-seeking and hallucination benchmarks demonstrate our approach's consistent effectiveness, for example, achieving up to an 8.6 point improvement on TruthfulQA. We believe this study can improve our understanding of hallucinations and serve as a practical solution for hallucination mitigation.

Title: Self-Supervised Representation Learning with Meta Comprehensive Regularization

• Authors: Huijie Guo, Ying Ba, Jie Hu, Lingyu Si, Wenwen Qiang, Lei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01549
• Pdf URL: https://arxiv.org/pdf/2403.01549
• Copy Paste: [[2403.01549]] Self-Supervised Representation Learning with Meta Comprehensive Regularization(https://arxiv.org/abs/2403.01549)
• Keywords: extraction, segmentation
• Abstract: Self-Supervised Learning (SSL) methods harness the concept of semantic invariance by utilizing data augmentation strategies to produce similar representations for different deformations of the same input. Essentially, the model captures the shared information among multiple augmented views of samples, while disregarding the non-shared information that may be beneficial for downstream tasks. To address this issue, we introduce a module called CompMod with Meta Comprehensive Regularization (MCR), embedded into existing self-supervised frameworks, to make the learned representations more comprehensive. Specifically, we update our proposed model through a bi-level optimization mechanism, enabling it to capture comprehensive features. Additionally, guided by the constrained extraction of features using maximum entropy coding, the self-supervised learning model learns more comprehensive features on top of learning consistent features. In addition, we provide theoretical support for our proposed method from information theory and causal counterfactual perspective. Experimental results show that our method achieves significant improvement in classification, object detection and instance segmentation tasks on multiple benchmark datasets.

Title: Transformers for Supervised Online Continual Learning

• Authors: Jorg Bornschein, Yazhe Li, Amal Rannen-Triki
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01554
• Pdf URL: https://arxiv.org/pdf/2403.01554
• Copy Paste: [[2403.01554]] Transformers for Supervised Online Continual Learning(https://arxiv.org/abs/2403.01554)
• Keywords: transformer
• Abstract: Transformers have become the dominant architecture for sequence modeling tasks such as natural language processing or audio processing, and they are now even considered for tasks that are not naturally sequential such as image classification. Their ability to attend to and to process a set of tokens as context enables them to develop in-context few-shot learning abilities. However, their potential for online continual learning remains relatively unexplored. In online continual learning, a model must adapt to a non-stationary stream of data, minimizing the cumulative nextstep prediction loss. We focus on the supervised online continual learning setting, where we learn a predictor $x_t \rightarrow y_t$ for a sequence of examples $(x_t, y_t)$. Inspired by the in-context learning capabilities of transformers and their connection to meta-learning, we propose a method that leverages these strengths for online continual learning. Our approach explicitly conditions a transformer on recent observations, while at the same time online training it with stochastic gradient descent, following the procedure introduced with Transformer-XL. We incorporate replay to maintain the benefits of multi-epoch training while adhering to the sequential protocol. We hypothesize that this combination enables fast adaptation through in-context learning and sustained longterm improvement via parametric learning. Our method demonstrates significant improvements over previous state-of-the-art results on CLOC, a challenging large-scale real-world benchmark for image geo-localization.

Title: Kick Back & Relax++: Scaling Beyond Ground-Truth Depth with SlowTV & CribsTV

• Authors: Jaime Spencer, Chris Russell, Simon Hadfield, Richard Bowden
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.01569
• Pdf URL: https://arxiv.org/pdf/2403.01569
• Copy Paste: [[2403.01569]] Kick Back & Relax++: Scaling Beyond Ground-Truth Depth with SlowTV & CribsTV(https://arxiv.org/abs/2403.01569)
• Keywords: transformer
• Abstract: Self-supervised learning is the key to unlocking generic computer vision systems. By eliminating the reliance on ground-truth annotations, it allows scaling to much larger data quantities. Unfortunately, self-supervised monocular depth estimation (SS-MDE) has been limited by the absence of diverse training data. Existing datasets have focused exclusively on urban driving in densely populated cities, resulting in models that fail to generalize beyond this domain. To address these limitations, this paper proposes two novel datasets: SlowTV and CribsTV. These are large-scale datasets curated from publicly available YouTube videos, containing a total of 2M training frames. They offer an incredibly diverse set of environments, ranging from snowy forests to coastal roads, luxury mansions and even underwater coral reefs. We leverage these datasets to tackle the challenging task of zero-shot generalization, outperforming every existing SS-MDE approach and even some state-of-the-art supervised methods. The generalization capabilities of our models are further enhanced by a range of components and contributions: 1) learning the camera intrinsics, 2) a stronger augmentation regime targeting aspect ratio changes, 3) support frame randomization, 4) flexible motion estimation, 5) a modern transformer-based architecture. We demonstrate the effectiveness of each component in extensive ablation experiments. To facilitate the development of future research, we make the datasets, code and pretrained models available to the public at https://github.com/jspenmar/slowtv_monodepth.

Title: SERVAL: Synergy Learning between Vertical Models and LLMs towards Oracle-Level Zero-shot Medical Prediction

• Authors: Jiahuan Yan, Jintai Chen, Chaowen Hu, Bo Zheng, Yaojun Hu, Jimeng Sun, Jian Wu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01570
• Pdf URL: https://arxiv.org/pdf/2403.01570
• Copy Paste: [[2403.01570]] SERVAL: Synergy Learning between Vertical Models and LLMs towards Oracle-Level Zero-shot Medical Prediction(https://arxiv.org/abs/2403.01570)
• Keywords: robust, large language model
• Abstract: Recent development of large language models (LLMs) has exhibited impressive zero-shot proficiency on generic and common sense questions. However, LLMs' application on domain-specific vertical questions still lags behind, primarily due to the humiliation problems and deficiencies in vertical knowledge. Furthermore, the vertical data annotation process often requires labor-intensive expert involvement, thereby presenting an additional challenge in enhancing the model's vertical capabilities. In this paper, we propose SERVAL, a synergy learning pipeline designed for unsupervised development of vertical capabilities in both LLMs and small models by mutual enhancement. Specifically, SERVAL utilizes the LLM's zero-shot outputs as annotations, leveraging its confidence to teach a robust vertical model from scratch. Reversely, the trained vertical model guides the LLM fine-tuning to enhance its zero-shot capability, progressively improving both models through an iterative process. In medical domain, known for complex vertical knowledge and costly annotations, comprehensive experiments show that, without access to any gold labels, SERVAL with the synergy learning of OpenAI GPT-3.5 and a simple model attains fully-supervised competitive performance across ten widely used medical datasets. These datasets represent vertically specialized medical diagnostic scenarios (e.g., diabetes, heart diseases, COVID-19), highlighting the potential of SERVAL in refining the vertical capabilities of LLMs and training vertical models from scratch, all achieved without the need for annotations.

Title: Enhancing Neural Machine Translation of Low-Resource Languages: Corpus Development, Human Evaluation and Explainable AI Architectures

• Authors: Séamus Lankford
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01580
• Pdf URL: https://arxiv.org/pdf/2403.01580
• Copy Paste: [[2403.01580]] Enhancing Neural Machine Translation of Low-Resource Languages: Corpus Development, Human Evaluation and Explainable AI Architectures(https://arxiv.org/abs/2403.01580)
• Keywords: transformer
• Abstract: In the current machine translation (MT) landscape, the Transformer architecture stands out as the gold standard, especially for high-resource language pairs. This research delves into its efficacy for low-resource language pairs including both the English$\leftrightarrow$Irish and English$\leftrightarrow$Marathi language pairs. Notably, the study identifies the optimal hyperparameters and subword model type to significantly improve the translation quality of Transformer models for low-resource language pairs. The scarcity of parallel datasets for low-resource languages can hinder MT development. To address this, gaHealth was developed, the first bilingual corpus of health data for the Irish language. Focusing on the health domain, models developed using this in-domain dataset exhibited very significant improvements in BLEU score when compared with models from the LoResMT2021 Shared Task. A subsequent human evaluation using the multidimensional quality metrics error taxonomy showcased the superior performance of the Transformer system in reducing both accuracy and fluency errors compared to an RNN-based counterpart. Furthermore, this thesis introduces adaptNMT and adaptMLLM, two open-source applications streamlined for the development, fine-tuning, and deployment of neural machine translation models. These tools considerably simplify the setup and evaluation process, making MT more accessible to both developers and translators. Notably, adaptNMT, grounded in the OpenNMT ecosystem, promotes eco-friendly natural language processing research by highlighting the environmental footprint of model development. Fine-tuning of MLLMs by adaptMLLM demonstrated advancements in translation performance for two low-resource language pairs: English$\leftrightarrow$Irish and English$\leftrightarrow$Marathi, compared to baselines from the LoResMT2021 Shared Task.

Title: IoT Device Labeling Using Large Language Models

• Authors: Bar Meyuhas, Anat Bremler-Barr, Tal Shapira
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.01586
• Pdf URL: https://arxiv.org/pdf/2403.01586
• Copy Paste: [[2403.01586]] IoT Device Labeling Using Large Language Models(https://arxiv.org/abs/2403.01586)
• Keywords: security, large language model
• Abstract: The IoT market is diverse and characterized by a multitude of vendors that support different device functions (e.g., speaker, camera, vacuum cleaner, etc.). Within this market, IoT security and observability systems use real-time identification techniques to manage these devices effectively. Most existing IoT identification solutions employ machine learning techniques that assume the IoT device, labeled by both its vendor and function, was observed during their training phase. We tackle a key challenge in IoT labeling: how can an AI solution label an IoT device that has never been seen before and whose label is unknown? Our solution extracts textual features such as domain names and hostnames from network traffic, and then enriches these features using Google search data alongside catalog of vendors and device functions. The solution also integrates an auto-update mechanism that uses Large Language Models (LLMs) to update these catalogs with emerging device types. Based on the information gathered, the device's vendor is identified through string matching with the enriched features. The function is then deduced by LLMs and zero-shot classification from a predefined catalog of IoT functions. In an evaluation of our solution on 97 unique IoT devices, our function labeling approach achieved HIT1 and HIT2 scores of 0.7 and 0.77, respectively. As far as we know, this is the first research to tackle AI-automated IoT labeling.

Title: The Hidden Attention of Mamba Models

• Authors: Ameen Ali, Itamar Zimerman, Lior Wolf
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01590
• Pdf URL: https://arxiv.org/pdf/2403.01590
• Copy Paste: [[2403.01590]] The Hidden Attention of Mamba Models(https://arxiv.org/abs/2403.01590)
• Keywords: explainability, transformer
• Abstract: The Mamba layer offers an efficient selective state space model (SSM) that is highly effective in modeling multiple domains including NLP, long-range sequences processing, and computer vision. Selective SSMs are viewed as dual models, in which one trains in parallel on the entire sequence via IO-aware parallel scan, and deploys in an autoregressive manner. We add a third view and show that such models can be viewed as attention-driven models. This new perspective enables us to compare the underlying mechanisms to that of the self-attention layers in transformers and allows us to peer inside the inner workings of the Mamba model with explainability methods. Our code is publicly available.

Title: SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos

• Authors: Yulei Niu, Wenliang Guo, Long Chen, Xudong Lin, Shih-Fu Chang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01599
• Pdf URL: https://arxiv.org/pdf/2403.01599
• Copy Paste: [[2403.01599]] SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos(https://arxiv.org/abs/2403.01599)
• Keywords: large language model
• Abstract: We study the problem of procedure planning in instructional videos, which aims to make a goal-oriented sequence of action steps given partial visual state observations. The motivation of this problem is to learn a structured and plannable state and action space. Recent works succeeded in sequence modeling of steps with only sequence-level annotations accessible during training, which overlooked the roles of states in the procedures. In this work, we point out that State CHangEs MAtter (SCHEMA) for procedure planning in instructional videos. We aim to establish a more structured state space by investigating the causal relations between steps and states in procedures. Specifically, we explicitly represent each step as state changes and track the state changes in procedures. For step representation, we leveraged the commonsense knowledge in large language models (LLMs) to describe the state changes of steps via our designed chain-of-thought prompting. For state change tracking, we align visual state observations with language state descriptions via cross-modal contrastive learning, and explicitly model the intermediate states of the procedure using LLM-generated state descriptions. Experiments on CrossTask, COIN, and NIV benchmark datasets demonstrate that our proposed SCHEMA model achieves state-of-the-art performance and obtains explainable visualizations.

Title: A Unified Model Selection Technique for Spectral Clustering Based Motion Segmentation

• Authors: Yuxiang Huang, John Zelek
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01606
• Pdf URL: https://arxiv.org/pdf/2403.01606
• Copy Paste: [[2403.01606]] A Unified Model Selection Technique for Spectral Clustering Based Motion Segmentation(https://arxiv.org/abs/2403.01606)
• Keywords: segmentation
• Abstract: Motion segmentation is a fundamental problem in computer vision and is crucial in various applications such as robotics, autonomous driving and action recognition. Recently, spectral clustering based methods have shown impressive results on motion segmentation in dynamic environments. These methods perform spectral clustering on motion affinity matrices to cluster objects or point trajectories in the scene into different motion groups. However, existing methods often need the number of motions present in the scene to be known, which significantly reduces their practicality. In this paper, we propose a unified model selection technique to automatically infer the number of motion groups for spectral clustering based motion segmentation methods by combining different existing model selection techniques together. We evaluate our method on the KT3DMoSeg dataset and achieve competitve results comparing to the baseline where the number of clusters is given as ground truth information.

Title: Partial Federated Learning

• Authors: Tiantian Feng, Anil Ramakrishna, Jimit Majmudar, Charith Peris, Jixuan Wang, Clement Chung, Richard Zemel, Morteza Ziyadi, Rahul Gupta
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.01615
• Pdf URL: https://arxiv.org/pdf/2403.01615
• Copy Paste: [[2403.01615]] Partial Federated Learning(https://arxiv.org/abs/2403.01615)
• Keywords: privacy, biometric, federate
• Abstract: Federated Learning (FL) is a popular algorithm to train machine learning models on user data constrained to edge devices (for example, mobile phones) due to privacy concerns. Typically, FL is trained with the assumption that no part of the user data can be egressed from the edge. However, in many production settings, specific data-modalities/meta-data are limited to be on device while others are not. For example, in commercial SLU systems, it is typically desired to prevent transmission of biometric signals (such as audio recordings of the input prompt) to the cloud, but egress of locally (i.e. on the edge device) transcribed text to the cloud may be possible. In this work, we propose a new algorithm called Partial Federated Learning (PartialFL), where a machine learning model is trained using data where a subset of data modalities or their intermediate representations can be made available to the server. We further restrict our model training by preventing the egress of data labels to the cloud for better privacy, and instead use a contrastive learning based model objective. We evaluate our approach on two different multi-modal datasets and show promising results with our proposed approach.

Title: Towards Comprehensive Vietnamese Retrieval-Augmented Generation and Large Language Models

• Authors: Nguyen Quang Duc, Le Hai Son, Nguyen Duc Nhan, Nguyen Dich Nhat Minh, Le Thanh Huong, Dinh Viet Sang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01616
• Pdf URL: https://arxiv.org/pdf/2403.01616
• Copy Paste: [[2403.01616]] Towards Comprehensive Vietnamese Retrieval-Augmented Generation and Large Language Models(https://arxiv.org/abs/2403.01616)
• Keywords: large language model
• Abstract: This paper presents our contributions towards advancing the state of Vietnamese language understanding and generation through the development and dissemination of open datasets and pre-trained models for Vietnamese Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs).

Title: Machine Learning vs Deep Learning: The Generalization Problem

• Authors: Yong Yi Bay, Kathleen A. Yearick
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01621
• Pdf URL: https://arxiv.org/pdf/2403.01621
• Copy Paste: [[2403.01621]] Machine Learning vs Deep Learning: The Generalization Problem(https://arxiv.org/abs/2403.01621)
• Keywords: robust
• Abstract: The capacity to generalize beyond the range of training data is a pivotal challenge, often synonymous with a model's utility and robustness. This study investigates the comparative abilities of traditional machine learning (ML) models and deep learning (DL) algorithms in terms of extrapolation -- a more challenging aspect of generalization because it requires the model to make inferences about data points that lie outside the domain it has been trained on. We present an empirical analysis where both ML and DL models are trained on an exponentially growing function and then tested on values outside the training domain. The choice of this function allows us to distinctly showcase the divergence in performance when models are required to predict beyond the scope of their training data. Our findings suggest that deep learning models possess inherent capabilities to generalize beyond the training scope, an essential feature for real-world applications where data is often incomplete or extends beyond the observed range. This paper argues for a nuanced understanding of the structural differences between ML and DL models, with an emphasis on the implications for both theoretical research and practical deployment.

Title: Using LLMs for Tabletop Exercises within the Security Domain

• Authors: Sam Hays, Dr. Jules White
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01626
• Pdf URL: https://arxiv.org/pdf/2403.01626
• Copy Paste: [[2403.01626]] Using LLMs for Tabletop Exercises within the Security Domain(https://arxiv.org/abs/2403.01626)
• Keywords: security, large language model
• Abstract: Tabletop exercises are a crucial component of many company's strategy to test and evaluate its preparedness for security incidents in a realistic way. Traditionally led by external firms specializing in cybersecurity, these exercises can be costly, time-consuming, and may not always align precisely with the client's specific needs. Large Language Models (LLMs) like ChatGPT offer a compelling alternative. They enable faster iteration, provide rich and adaptable simulations, and offer infinite patience in handling feedback and recommendations. This approach can enhances the efficiency and relevance of security preparedness exercises.

Title: Improving LLM Code Generation with Grammar Augmentation

• Authors: Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, Gagandeep Singh
• Subjects: cs.LG, cs.FL, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.01632
• Pdf URL: https://arxiv.org/pdf/2403.01632
• Copy Paste: [[2403.01632]] Improving LLM Code Generation with Grammar Augmentation(https://arxiv.org/abs/2403.01632)
• Keywords: large language model
• Abstract: We present SynCode a novel framework for efficient and general syntactical decoding of code with large language models (LLMs). SynCode leverages the grammar of a programming language, utilizing an offline-constructed efficient lookup table called DFA mask store based on language grammar terminals. We demonstrate SynCode's soundness and completeness given the context-free grammar (CFG) of the programming language, presenting its ability to retain syntactically valid tokens while rejecting invalid ones. The framework seamlessly integrates with any language defined by CFG, as evidenced by experiments on CFGs for Python and Go. The results underscore the significant reduction of 96.07% of syntax errors achieved when SynCode is combined with state-of-the-art LLMs, showcasing its substantial impact on enhancing syntactical precision in code generation. Our code is available at https://github.com/uiuc-focal-lab/syncode.

Title: Critical windows: non-asymptotic theory for feature emergence in diffusion models

• Authors: Marvin Li, Sitan Chen
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.01633
• Pdf URL: https://arxiv.org/pdf/2403.01633
• Copy Paste: [[2403.01633]] Critical windows: non-asymptotic theory for feature emergence in diffusion models(https://arxiv.org/abs/2403.01633)
• Keywords: privacy, fair, interpretability, diffusion
• Abstract: We develop theory to understand an intriguing property of diffusion models for image generation that we term critical windows. Empirically, it has been observed that there are narrow time intervals in sampling during which particular features of the final image emerge, e.g. the image class or background color (Ho et al., 2020b; Georgiev et al., 2023; Raya & Ambrogioni, 2023; Sclocchi et al., 2024; Biroli et al., 2024). While this is advantageous for interpretability as it implies one can localize properties of the generation to a small segment of the trajectory, it seems at odds with the continuous nature of the diffusion. We propose a formal framework for studying these windows and show that for data coming from a mixture of strongly log-concave densities, these windows can be provably bounded in terms of certain measures of inter- and intra-group separation. We also instantiate these bounds for concrete examples like well-conditioned Gaussian mixtures. Finally, we use our bounds to give a rigorous interpretation of diffusion models as hierarchical samplers that progressively "decide" output features over a discrete sequence of times. We validate our bounds with synthetic experiments. Additionally, preliminary experiments on Stable Diffusion suggest critical windows may serve as a useful tool for diagnosing fairness and privacy violations in real-world diffusion models.

Title: Multi-level Product Category Prediction through Text Classification

• Authors: Wesley Ferreira Maia, Angelo Carmignani, Gabriel Bortoli, Lucas Maretti, David Luz, Daniel Camilo Fuentes Guzman, Marcos Jardel Henriques, Francisco Louzada Neto
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01638
• Pdf URL: https://arxiv.org/pdf/2403.01638
• Copy Paste: [[2403.01638]] Multi-level Product Category Prediction through Text Classification(https://arxiv.org/abs/2403.01638)
• Keywords: robust
• Abstract: This article investigates applying advanced machine learning models, specifically LSTM and BERT, for text classification to predict multiple categories in the retail sector. The study demonstrates how applying data augmentation techniques and the focal loss function can significantly enhance accuracy in classifying products into multiple categories using a robust Brazilian retail dataset. The LSTM model, enriched with Brazilian word embedding, and BERT, known for its effectiveness in understanding complex contexts, were adapted and optimized for this specific task. The results showed that the BERT model, with an F1 Macro Score of up to $99\%$ for segments, $96\%$ for categories and subcategories and $93\%$ for name products, outperformed LSTM in more detailed categories. However, LSTM also achieved high performance, especially after applying data augmentation and focal loss techniques. These results underscore the effectiveness of NLP techniques in retail and highlight the importance of the careful selection of modelling and preprocessing strategies. This work contributes significantly to the field of NLP in retail, providing valuable insights for future research and practical applications.

Title: Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models

• Authors: Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, Yuting Wei
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.01639
• Pdf URL: https://arxiv.org/pdf/2403.01639
• Copy Paste: [[2403.01639]] Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models(https://arxiv.org/abs/2403.01639)
• Keywords: diffusion
• Abstract: Diffusion models benefit from instillation of task-specific information into the score function to steer the sample generation towards desired properties. Such information is coined as guidance. For example, in text-to-image synthesis, text input is encoded as guidance to generate semantically aligned images. Proper guidance inputs are closely tied to the performance of diffusion models. A common observation is that strong guidance promotes a tight alignment to the task-specific information, while reducing the diversity of the generated samples. In this paper, we provide the first theoretical study towards understanding the influence of guidance on diffusion models in the context of Gaussian mixture models. Under mild conditions, we prove that incorporating diffusion guidance not only boosts classification confidence but also diminishes distribution diversity, leading to a reduction in the differential entropy of the output distribution. Our analysis covers the widely adopted sampling schemes including DDPM and DDIM, and leverages comparison inequalities for differential equations as well as the Fokker-Planck equation that characterizes the evolution of probability density function, which may be of independent theoretical interest.

Title: AIO2: Online Correction of Object Labels for Deep Learning with Incomplete Annotation in Remote Sensing Image Segmentation

• Authors: Chenying Liu, Conrad M Albrecht, Yi Wang, Qingyu Li, Xiao Xiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01641
• Pdf URL: https://arxiv.org/pdf/2403.01641
• Copy Paste: [[2403.01641]] AIO2: Online Correction of Object Labels for Deep Learning with Incomplete Annotation in Remote Sensing Image Segmentation(https://arxiv.org/abs/2403.01641)
• Keywords: robust, segmentation
• Abstract: While the volume of remote sensing data is increasing daily, deep learning in Earth Observation faces lack of accurate annotations for supervised optimization. Crowdsourcing projects such as OpenStreetMap distribute the annotation load to their community. However, such annotation inevitably generates noise due to insufficient control of the label quality, lack of annotators, frequent changes of the Earth's surface as a result of natural disasters and urban development, among many other factors. We present Adaptively trIggered Online Object-wise correction (AIO2) to address annotation noise induced by incomplete label sets. AIO2 features an Adaptive Correction Trigger (ACT) module that avoids label correction when the model training under- or overfits, and an Online Object-wise Correction (O2C) methodology that employs spatial information for automated label modification. AIO2 utilizes a mean teacher model to enhance training robustness with noisy labels to both stabilize the training accuracy curve for fitting in ACT and provide pseudo labels for correction in O2C. Moreover, O2C is implemented online without the need to store updated labels every training epoch. We validate our approach on two building footprint segmentation datasets with different spatial resolutions. Experimental results with varying degrees of building label noise demonstrate the robustness of AIO2. Source code will be available at https://github.com/zhu-xlab/AIO2.git.

Title: You Need to Pay Better Attention

• Authors: Mehran Hosseini, Peyman Hosseini
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01643
• Pdf URL: https://arxiv.org/pdf/2403.01643
• Copy Paste: [[2403.01643]] You Need to Pay Better Attention(https://arxiv.org/abs/2403.01643)
• Keywords: transformer
• Abstract: We introduce three new attention mechanisms that outperform standard multi-head attention in terms of efficiency and learning capabilities, thereby improving the performance and broader deployability of Transformer models. Our first contribution is Optimised Attention, which performs similarly to standard attention, but has 3/4 as many parameters and one matrix multiplication fewer per head. Next, we introduce Efficient Attention, which performs on par with standard attention with only 1/2 as many parameters as many parameters and two matrix multiplications fewer per head and is up to twice as fast as standard attention. Lastly, we introduce Super Attention, which surpasses standard attention by a significant margin in both vision and natural language processing tasks while having fewer parameters and matrix multiplications. In addition to providing rigorous mathematical comparisons, we evaluate the presented attention mechanisms on MNIST, CIFAR100, IMDB Movie Reviews, and Amazon Reviews datasets.

Title: OccFusion: A Straightforward and Effective Multi-Sensor Fusion Framework for 3D Occupancy Prediction

• Authors: Zhenxing Ming, Julie Stephany Berrio, Mao Shan, Stewart Worrall
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.01644
• Pdf URL: https://arxiv.org/pdf/2403.01644
• Copy Paste: [[2403.01644]] OccFusion: A Straightforward and Effective Multi-Sensor Fusion Framework for 3D Occupancy Prediction(https://arxiv.org/abs/2403.01644)
• Keywords: robust
• Abstract: This paper introduces OccFusion, a straightforward and efficient sensor fusion framework for predicting 3D occupancy. A comprehensive understanding of 3D scenes is crucial in autonomous driving, and recent models for 3D semantic occupancy prediction have successfully addressed the challenge of describing real-world objects with varied shapes and classes. However, existing methods for 3D occupancy prediction heavily rely on surround-view camera images, making them susceptible to changes in lighting and weather conditions. By integrating features from additional sensors, such as lidar and surround view radars, our framework enhances the accuracy and robustness of occupancy prediction, resulting in top-tier performance on the nuScenes benchmark. Furthermore, extensive experiments conducted on the nuScenes dataset, including challenging night and rainy scenarios, confirm the superior performance of our sensor fusion strategy across various perception ranges. The code for this framework will be made available at https://github.com/DanielMing123/OCCFusion.

Title: "I just hated it and I want my money back": Data-driven Understanding of Mobile VPN Service Switching Preferences in The Wild

• Authors: Rohit Raj, Mridul Newar, Mainack Mondal
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.01648
• Pdf URL: https://arxiv.org/pdf/2403.01648
• Copy Paste: [[2403.01648]] "I just hated it and I want my money back": Data-driven Understanding of Mobile VPN Service Switching Preferences in The Wild(https://arxiv.org/abs/2403.01648)
• Keywords: security, privacy
• Abstract: Virtual Private Networks (VPNs) are a crucial Privacy-Enhancing Technology (PET) leveraged by millions of users and catered by multiple VPN providers worldwide; thus, understanding the user preferences for the choice of VPN apps should be of importance and interest to the security community. To that end, prior studies looked into the usage, awareness and adoption of VPN users and the perceptions of providers. However, no study so far has looked into the user preferences and underlying reasons for switching among VPN providers and identified features that presumably enhance users' VPN experience. This work aims to bridge this gap and shed light on the underlying factors that drive existing users when they switch from one VPN to another. In this work, we analyzed over 1.3 million reviews from 20 leading VPN apps, identifying 1,305 explicit mentions and intents to switch. Our NLP-based analysis unveiled distinct clusters of factors motivating users to switch. An examination of 376 blogs from six popular VPN recommendation sites revealed biases in the content, and we found ignorance towards user preferences. We conclude by identifying the key implications of our work for different stakeholders. The data and code for this work is available at https://github.com/Mainack/switch-vpn-datacode-sec24.

Title: Improving Adversarial Energy-Based Model via Diffusion Process

• Authors: Cong Geng, Tian Han, Peng-Tao Jiang, Hao Zhang, Jinwei Chen, Søren Hauberg, Bo Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01666
• Pdf URL: https://arxiv.org/pdf/2403.01666
• Copy Paste: [[2403.01666]] Improving Adversarial Energy-Based Model via Diffusion Process(https://arxiv.org/abs/2403.01666)
• Keywords: diffusion, generative
• Abstract: Generative models have shown strong generation ability while efficient likelihood estimation is less explored. Energy-based models~(EBMs) define a flexible energy function to parameterize unnormalized densities efficiently but are notorious for being difficult to train. Adversarial EBMs introduce a generator to form a minimax training game to avoid expensive MCMC sampling used in traditional EBMs, but a noticeable gap between adversarial EBMs and other strong generative models still exists. Inspired by diffusion-based models, we embedded EBMs into each denoising step to split a long-generated process into several smaller steps. Besides, we employ a symmetric Jeffrey divergence and introduce a variational posterior distribution for the generator's training to address the main challenges that exist in adversarial EBMs. Our experiments show significant improvement in generation compared to existing adversarial EBMs, while also providing a useful energy function for efficient density estimation.

Title: HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances

• Authors: Supreeth Narasimhaswamy, Uttaran Bhattacharya, Xiang Chen, Ishita Dasgupta, Saayan Mitra, Minh Hoai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01693
• Pdf URL: https://arxiv.org/pdf/2403.01693
• Copy Paste: [[2403.01693]] HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances(https://arxiv.org/abs/2403.01693)
• Keywords: robust, diffusion, generative
• Abstract: Text-to-image generative models can generate high-quality humans, but realism is lost when generating hands. Common artifacts include irregular hand poses, shapes, incorrect numbers of fingers, and physically implausible finger orientations. To generate images with realistic hands, we propose a novel diffusion-based architecture called HanDiffuser that achieves realism by injecting hand embeddings in the generative process. HanDiffuser consists of two components: a Text-to-Hand-Params diffusion model to generate SMPL-Body and MANO-Hand parameters from input text prompts, and a Text-Guided Hand-Params-to-Image diffusion model to synthesize images by conditioning on the prompts and hand parameters generated by the previous component. We incorporate multiple aspects of hand representation, including 3D shapes and joint-level finger positions, orientations and articulations, for robust learning and reliable performance during inference. We conduct extensive quantitative and qualitative experiments and perform user studies to demonstrate the efficacy of our method in generating images with high-quality hands.

Title: Hypertext Entity Extraction in Webpage

• Authors: Yifei Yang, Tianqiao Liu, Bo Shao, Hai Zhao, Linjun Shou, Ming Gong, Daxin Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01698
• Pdf URL: https://arxiv.org/pdf/2403.01698
• Copy Paste: [[2403.01698]] Hypertext Entity Extraction in Webpage(https://arxiv.org/abs/2403.01698)
• Keywords: extraction
• Abstract: Webpage entity extraction is a fundamental natural language processing task in both research and applications. Nowadays, the majority of webpage entity extraction models are trained on structured datasets which strive to retain textual content and its structure information. However, existing datasets all overlook the rich hypertext features (e.g., font color, font size) which show their effectiveness in previous works. To this end, we first collect a \textbf{H}ypertext \textbf{E}ntity \textbf{E}xtraction \textbf{D}ataset (\textit{HEED}) from the e-commerce domains, scraping both the text and the corresponding explicit hypertext features with high-quality manual entity annotations. Furthermore, we present the \textbf{Mo}E-based \textbf{E}ntity \textbf{E}xtraction \textbf{F}ramework (\textit{MoEEF}), which efficiently integrates multiple features to enhance model performance by Mixture of Experts and outperforms strong baselines, including the state-of-the-art small-scale models and GPT-3.5-turbo. Moreover, the effectiveness of hypertext features in \textit{HEED} and several model components in \textit{MoEEF} are analyzed.

Title: Brilla AI: AI Contestant for the National Science and Maths Quiz

• Authors: George Boateng, Jonathan Abrefah Mensah, Kevin Takyi Yeboah, William Edor, Andrew Kojo Mensah-Onumah, Naafi Dasana Ibrahim, Nana Sam Yeboah
• Subjects: cs.CL, cs.AI, cs.CY, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.01699
• Pdf URL: https://arxiv.org/pdf/2403.01699
• Copy Paste: [[2403.01699]] Brilla AI: AI Contestant for the National Science and Maths Quiz(https://arxiv.org/abs/2403.01699)
• Keywords: robust, extraction
• Abstract: The African continent lacks enough qualified teachers which hampers the provision of adequate learning support. An AI could potentially augment the efforts of the limited number of teachers, leading to better learning outcomes. Towards that end, this work describes and evaluates the first key output for the NSMQ AI Grand Challenge, which proposes a robust, real-world benchmark for such an AI: "Build an AI to compete live in Ghana's National Science and Maths Quiz (NSMQ) competition and win - performing better than the best contestants in all rounds and stages of the competition". The NSMQ is an annual live science and mathematics competition for senior secondary school students in Ghana in which 3 teams of 2 students compete by answering questions across biology, chemistry, physics, and math in 5 rounds over 5 progressive stages until a winning team is crowned for that year. In this work, we built Brilla AI, an AI contestant that we deployed to unofficially compete remotely and live in the Riddles round of the 2023 NSMQ Grand Finale, the first of its kind in the 30-year history of the competition. Brilla AI is currently available as a web app that livestreams the Riddles round of the contest, and runs 4 machine learning systems: (1) speech to text (2) question extraction (3) question answering and (4) text to speech that work together in real-time to quickly and accurately provide an answer, and then say it with a Ghanaian accent. In its debut, our AI answered one of the 4 riddles ahead of the 3 human contesting teams, unofficially placing second (tied). Improvements and extensions of this AI could potentially be deployed to offer science tutoring to students and eventually enable millions across Africa to have one-on-one learning interactions, democratizing science education.

Title: MCA: Moment Channel Attention Networks

• Authors: Yangbo Jiang, Zhiwei Jiang, Le Han, Zenan Huang, Nenggan Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01713
• Pdf URL: https://arxiv.org/pdf/2403.01713
• Copy Paste: [[2403.01713]] MCA: Moment Channel Attention Networks(https://arxiv.org/abs/2403.01713)
• Keywords: segmentation
• Abstract: Channel attention mechanisms endeavor to recalibrate channel weights to enhance representation abilities of networks. However, mainstream methods often rely solely on global average pooling as the feature squeezer, which significantly limits the overall potential of models. In this paper, we investigate the statistical moments of feature maps within a neural network. Our findings highlight the critical role of high-order moments in enhancing model capacity. Consequently, we introduce a flexible and comprehensive mechanism termed Extensive Moment Aggregation (EMA) to capture the global spatial context. Building upon this mechanism, we propose the Moment Channel Attention (MCA) framework, which efficiently incorporates multiple levels of moment-based information while minimizing additional computation costs through our Cross Moment Convolution (CMC) module. The CMC module via channel-wise convolution layer to capture multiple order moment information as well as cross channel features. The MCA block is designed to be lightweight and easily integrated into a variety of neural network architectures. Experimental results on classical image classification, object detection, and instance segmentation tasks demonstrate that our proposed method achieves state-of-the-art results, outperforming existing channel attention methods.

Title: RISeg: Robot Interactive Object Segmentation via Body Frame-Invariant Features

• Authors: Howard H. Qian, Yangxiao Lu, Kejia Ren, Gaotian Wang, Ninad Khargonkar, Yu Xiang, Kaiyu Hang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.01731
• Pdf URL: https://arxiv.org/pdf/2403.01731
• Copy Paste: [[2403.01731]] RISeg: Robot Interactive Object Segmentation via Body Frame-Invariant Features(https://arxiv.org/abs/2403.01731)
• Keywords: segmentation
• Abstract: In order to successfully perform manipulation tasks in new environments, such as grasping, robots must be proficient in segmenting unseen objects from the background and/or other objects. Previous works perform unseen object instance segmentation (UOIS) by training deep neural networks on large-scale data to learn RGB/RGB-D feature embeddings, where cluttered environments often result in inaccurate segmentations. We build upon these methods and introduce a novel approach to correct inaccurate segmentation, such as under-segmentation, of static image-based UOIS masks by using robot interaction and a designed body frame-invariant feature. We demonstrate that the relative linear and rotational velocities of frames randomly attached to rigid bodies due to robot interactions can be used to identify objects and accumulate corrected object-level segmentation masks. By introducing motion to regions of segmentation uncertainty, we are able to drastically improve segmentation accuracy in an uncertainty-driven manner with minimal, non-disruptive interactions (ca. 2-3 per scene). We demonstrate the effectiveness of our proposed interactive perception pipeline in accurately segmenting cluttered scenes by achieving an average object segmentation accuracy rate of 80.7%, an increase of 28.2% when compared with other state-of-the-art UOIS methods.

Title: Lightweight Object Detection: A Study Based on YOLOv7 Integrated with ShuffleNetv2 and Vision Transformer

• Authors: Wenkai Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01736
• Pdf URL: https://arxiv.org/pdf/2403.01736
• Copy Paste: [[2403.01736]] Lightweight Object Detection: A Study Based on YOLOv7 Integrated with ShuffleNetv2 and Vision Transformer(https://arxiv.org/abs/2403.01736)
• Keywords: transformer
• Abstract: As mobile computing technology rapidly evolves, deploying efficient object detection algorithms on mobile devices emerges as a pivotal research area in computer vision. This study zeroes in on optimizing the YOLOv7 algorithm to boost its operational efficiency and speed on mobile platforms while ensuring high accuracy. Leveraging a synergy of advanced techniques such as Group Convolution, ShuffleNetV2, and Vision Transformer, this research has effectively minimized the model's parameter count and memory usage, streamlined the network architecture, and fortified the real-time object detection proficiency on resource-constrained devices. The experimental outcomes reveal that the refined YOLO model demonstrates exceptional performance, markedly enhancing processing velocity while sustaining superior detection accuracy.

Title: Diffusion-TS: Interpretable Diffusion for General Time Series Generation

• Authors: Xinyu Yuan, Yan Qiao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01742
• Pdf URL: https://arxiv.org/pdf/2403.01742
• Copy Paste: [[2403.01742]] Diffusion-TS: Interpretable Diffusion for General Time Series Generation(https://arxiv.org/abs/2403.01742)
• Keywords: diffusion, transformer, generative
• Abstract: Denoising diffusion probabilistic models (DDPMs) are becoming the leading paradigm for generative models. It has recently shown breakthroughs in audio synthesis, time series imputation and forecasting. In this paper, we propose Diffusion-TS, a novel diffusion-based framework that generates multivariate time series samples of high quality by using an encoder-decoder transformer with disentangled temporal representations, in which the decomposition technique guides Diffusion-TS to capture the semantic meaning of time series while transformers mine detailed sequential information from the noisy model input. Different from existing diffusion-based approaches, we train the model to directly reconstruct the sample instead of the noise in each diffusion step, combining a Fourier-based loss term. Diffusion-TS is expected to generate time series satisfying both interpretablity and realness. In addition, it is shown that the proposed Diffusion-TS can be easily extended to conditional generation tasks, such as forecasting and imputation, without any model changes. This also motivates us to further explore the performance of Diffusion-TS under irregular settings. Finally, through qualitative and quantitative experiments, results show that Diffusion-TS achieves the state-of-the-art results on various realistic analyses of time series.

Title: Decode Neural signal as Speech

• Authors: Yiqian Yang, Yiqun Duan, Qiang Zhang, Renjing Xu, Hui Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01748
• Pdf URL: https://arxiv.org/pdf/2403.01748
• Copy Paste: [[2403.01748]] Decode Neural signal as Speech(https://arxiv.org/abs/2403.01748)
• Keywords: generative, large language model
• Abstract: Decoding language from brain dynamics is an important open direction in the realm of brain-computer interface (BCI), especially considering the rapid growth of large language models. Compared to invasive-based signals which require electrode implantation surgery, non-invasive neural signals (e.g. EEG, MEG) have attracted increasing attention considering their safety and generality. However, the exploration is not adequate in three aspects: 1) previous methods mainly focus on EEG but none of the previous works address this problem on MEG with better signal quality; 2) prior works have predominantly used ``teacher-forcing" during generative decoding, which is impractical; 3) prior works are mostly ``BART-based" not fully auto-regressive, which performs better in other sequence tasks. In this paper, we explore the brain-to-text translation of MEG signals in a speech-decoding formation. Here we are the first to investigate a cross-attention-based ``whisper" model for generating text directly from MEG signals without teacher forcing. Our model achieves impressive BLEU-1 scores of 60.30 and 52.89 without pretraining \& teacher-forcing on two major datasets (\textit{GWilliams} and \textit{Schoffelen}). This paper conducts a comprehensive review to understand how speech decoding formation performs on the neural decoding tasks, including pretraining initialization, training \& evaluation set splitting, augmentation, and scaling law.

Title: Differentially Private Synthetic Data via Foundation Model APIs 2: Text

• Authors: Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haotian Jiang, Huishuai Zhang, Yin Tat Lee, Bo Li, Sergey Yekhanin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01749
• Pdf URL: https://arxiv.org/pdf/2403.01749
• Copy Paste: [[2403.01749]] Differentially Private Synthetic Data via Foundation Model APIs 2: Text(https://arxiv.org/abs/2403.01749)
• Keywords: secure, privacy, diffusion, large language model
• Abstract: Text data has become extremely valuable due to the emergence of machine learning algorithms that learn from it. A lot of high-quality text data generated in the real world is private and therefore cannot be shared or used freely due to privacy concerns. Generating synthetic replicas of private text data with a formal privacy guarantee, i.e., differential privacy (DP), offers a promising and scalable solution. However, existing methods necessitate DP finetuning of large language models (LLMs) on private data to generate DP synthetic data. This approach is not viable for proprietary LLMs (e.g., GPT-3.5) and also demands considerable computational resources for open-source LLMs. Lin et al. (2024) recently introduced the Private Evolution (PE) algorithm to generate DP synthetic images with only API access to diffusion models. In this work, we propose an augmented PE algorithm, named Aug-PE, that applies to the complex setting of text. We use API access to an LLM and generate DP synthetic text without any model training. We conduct comprehensive experiments on three benchmark datasets. Our results demonstrate that Aug-PE produces DP synthetic text that yields competitive utility with the SOTA DP finetuning baselines. This underscores the feasibility of relying solely on API access of LLMs to produce high-quality DP synthetic texts, thereby facilitating more accessible routes to privacy-preserving LLM applications. Our code and data are available at https://github.com/AI-secure/aug-pe.

Title: Derivative-Free Optimization for Low-Rank Adaptation in Large Language Models

• Authors: Feihu Jin, Yin Liu, Ying Tan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01754
• Pdf URL: https://arxiv.org/pdf/2403.01754
• Copy Paste: [[2403.01754]] Derivative-Free Optimization for Low-Rank Adaptation in Large Language Models(https://arxiv.org/abs/2403.01754)
• Keywords: robust
• Abstract: Parameter-efficient tuning methods such as LoRA could achieve comparable performance to model tuning by tuning a small portion of the parameters. However, substantial computational resources are still required, as this process involves calculating gradients and performing back-propagation throughout the model. Much effort has recently been devoted to utilizing the derivative-free optimization method to eschew the computation of gradients and showcase an augmented level of robustness in few-shot settings. In this paper, we prepend the low-rank modules into each self-attention layer of the model and employ two derivative-free optimization methods to optimize these low-rank modules at each layer alternately. Extensive results on various tasks and language models demonstrate that our proposed method achieves substantial improvement and exhibits clear advantages in memory usage and convergence speed compared to existing gradient-based parameter-efficient tuning and derivative-free optimization methods in few-shot settings.

Title: A Safe Screening Rule with Bi-level Optimization of $ν$ Support Vector Machine

• Authors: Zhiji Yang, Wanyi Chen, Huan Zhang, Yitian Xu, Lei Shi, Jianhua Zhao
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2403.01769
• Pdf URL: https://arxiv.org/pdf/2403.01769
• Copy Paste: [[2403.01769]] A Safe Screening Rule with Bi-level Optimization of $ν$ Support Vector Machine(https://arxiv.org/abs/2403.01769)
• Keywords: interpretability
• Abstract: Support vector machine (SVM) has achieved many successes in machine learning, especially for a small sample problem. As a famous extension of the traditional SVM, the $\nu$ support vector machine ($\nu$-SVM) has shown outstanding performance due to its great model interpretability. However, it still faces challenges in training overhead for large-scale problems. To address this issue, we propose a safe screening rule with bi-level optimization for $\nu$-SVM (SRBO-$\nu$-SVM) which can screen out inactive samples before training and reduce the computational cost without sacrificing the prediction accuracy. Our SRBO-$\nu$-SVM is strictly deduced by integrating the Karush-Kuhn-Tucker (KKT) conditions, the variational inequalities of convex problems and the $\nu$-property. Furthermore, we develop an efficient dual coordinate descent method (DCDM) to further improve computational speed. Finally, a unified framework for SRBO is proposed to accelerate many SVM-type models, and it is successfully applied to one-class SVM. Experimental results on 6 artificial data sets and 30 benchmark data sets have verified the effectiveness and safety of our proposed methods in supervised and unsupervised tasks.

Title: Improving out-of-distribution generalization in graphs via hierarchical semantic environments

• Authors: Yinhua Piao, Sangseon Lee, Yijingxiu Lu, Sun Kim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01773
• Pdf URL: https://arxiv.org/pdf/2403.01773
• Copy Paste: [[2403.01773]] Improving out-of-distribution generalization in graphs via hierarchical semantic environments(https://arxiv.org/abs/2403.01773)
• Keywords: robust
• Abstract: Out-of-distribution (OOD) generalization in the graph domain is challenging due to complex distribution shifts and a lack of environmental contexts. Recent methods attempt to enhance graph OOD generalization by generating flat environments. However, such flat environments come with inherent limitations to capture more complex data distributions. Considering the DrugOOD dataset, which contains diverse training environments (e.g., scaffold, size, etc.), flat contexts cannot sufficiently address its high heterogeneity. Thus, a new challenge is posed to generate more semantically enriched environments to enhance graph invariant learning for handling distribution shifts. In this paper, we propose a novel approach to generate hierarchical semantic environments for each graph. Firstly, given an input graph, we explicitly extract variant subgraphs from the input graph to generate proxy predictions on local environments. Then, stochastic attention mechanisms are employed to re-extract the subgraphs for regenerating global environments in a hierarchical manner. In addition, we introduce a new learning objective that guides our model to learn the diversity of environments within the same hierarchy while maintaining consistency across different hierarchies. This approach enables our model to consider the relationships between environments and facilitates robust graph invariant learning. Extensive experiments on real-world graph data have demonstrated the effectiveness of our framework. Particularly, in the challenging dataset DrugOOD, our method achieves up to 1.29\% and 2.83\% improvement over the best baselines on IC50 and EC50 prediction tasks, respectively.

Title: WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citations

• Authors: Haolin Deng, Chang Wang, Xin Li, Dezhang Yuan, Junlang Zhan, Tianhua Zhou, Jin Ma, Jun Gao, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01774
• Pdf URL: https://arxiv.org/pdf/2403.01774
• Copy Paste: [[2403.01774]] WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citations(https://arxiv.org/abs/2403.01774)
• Keywords: large language model
• Abstract: Enhancing the attribution in large language models (LLMs) is a crucial task. One feasible approach is to enable LLMs to cite external sources that support their generations. However, existing datasets and evaluation methods in this domain still exhibit notable limitations. In this work, we formulate the task of attributed query-focused summarization (AQFS) and present WebCiteS, a Chinese dataset featuring 7k human-annotated summaries with citations. WebCiteS derives from real-world user queries and web search results, offering a valuable resource for model training and evaluation. Prior works in attribution evaluation do not differentiate between groundedness errors and citation errors. They also fall short in automatically verifying sentences that draw partial support from multiple sources. We tackle these issues by developing detailed metrics and enabling the automatic evaluator to decompose the sentences into sub-claims for fine-grained verification. Our comprehensive evaluation of both open-source and proprietary models on WebCiteS highlights the challenge LLMs face in correctly citing sources, underscoring the necessity for further improvement. The dataset and code will be open-sourced to facilitate further research in this crucial field.

Title: NPHardEval4V: A Dynamic Reasoning Benchmark of Multimodal Large Language Models

• Authors: Lizhou Fan, Wenyue Hua, Xiang Li, Kaijie Zhu, Mingyu Jin, Lingyao Li, Haoyang Ling, Jinkui Chi, Jindong Wang, Xin Ma, Yongfeng Zhang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01777
• Pdf URL: https://arxiv.org/pdf/2403.01777
• Copy Paste: [[2403.01777]] NPHardEval4V: A Dynamic Reasoning Benchmark of Multimodal Large Language Models(https://arxiv.org/abs/2403.01777)
• Keywords: large language model
• Abstract: Understanding the reasoning capabilities of Multimodal Large Language Models (MLLMs) is an important area of research. In this study, we introduce a dynamic benchmark, NPHardEval4V, aimed at addressing the existing gaps in evaluating the pure reasoning abilities of MLLMs. Our benchmark aims to provide a venue to disentangle the effect of various factors such as image recognition and instruction following, from the overall performance of the models, allowing us to focus solely on evaluating their reasoning abilities. Our findings reveal significant discrepancies in reasoning abilities across different models and highlight the relatively weak performance of MLLMs compared to LLMs in terms of reasoning. We also investigate the impact of different prompting styles, including visual, text, and combined vision and text prompts, on the reasoning abilities of MLLMs, demonstrating the different impacts of multimodal inputs in model performance. Unlike traditional benchmarks, which primarily focus on static evaluations, our benchmark will update on a monthly basis to prevent overfitting and ensure a more accurate evaluation of the models. We believe that this benchmark can aid understand and guide the further development of reasoning abilities in MLLMs. The benchmark dataset and code are available at https://github.com/lizhouf/NPHardEval4V

Title: OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on

• Authors: Yuhao Xu, Tao Gu, Weifeng Chen, Chengcai Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01779
• Pdf URL: https://arxiv.org/pdf/2403.01779
• Copy Paste: [[2403.01779]] OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on(https://arxiv.org/abs/2403.01779)
• Keywords: diffusion
• Abstract: Image-based virtual try-on (VTON), which aims to generate an outfitted image of a target human wearing an in-shop garment, is a challenging image-synthesis task calling for not only high fidelity of the outfitted human but also full preservation of garment details. To tackle this issue, we propose Outfitting over Try-on Diffusion (OOTDiffusion), leveraging the power of pretrained latent diffusion models and designing a novel network architecture for realistic and controllable virtual try-on. Without an explicit warping process, we propose an outfitting UNet to learn the garment detail features, and merge them with the target human body via our proposed outfitting fusion in the denoising process of diffusion models. In order to further enhance the controllability of our outfitting UNet, we introduce outfitting dropout to the training process, which enables us to adjust the strength of garment features through classifier-free guidance. Our comprehensive experiments on the VITON-HD and Dress Code datasets demonstrate that OOTDiffusion efficiently generates high-quality outfitted images for arbitrary human and garment images, which outperforms other VTON methods in both fidelity and controllability, indicating an impressive breakthrough in virtual try-on. Our source code is available at https://github.com/levihsu/OOTDiffusion.

Title: Integrating Efficient Optimal Transport and Functional Maps For Unsupervised Shape Correspondence Learning

• Authors: Tung Le, Khai Nguyen, Shanlin Sun, Nhat Ho, Xiaohui Xie
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01781
• Pdf URL: https://arxiv.org/pdf/2403.01781
• Copy Paste: [[2403.01781]] Integrating Efficient Optimal Transport and Functional Maps For Unsupervised Shape Correspondence Learning(https://arxiv.org/abs/2403.01781)
• Keywords: segmentation
• Abstract: In the realm of computer vision and graphics, accurately establishing correspondences between geometric 3D shapes is pivotal for applications like object tracking, registration, texture transfer, and statistical shape analysis. Moving beyond traditional hand-crafted and data-driven feature learning methods, we incorporate spectral methods with deep learning, focusing on functional maps (FMs) and optimal transport (OT). Traditional OT-based approaches, often reliant on entropy regularization OT in learning-based framework, face computational challenges due to their quadratic cost. Our key contribution is to employ the sliced Wasserstein distance (SWD) for OT, which is a valid fast optimal transport metric in an unsupervised shape matching framework. This unsupervised framework integrates functional map regularizers with a novel OT-based loss derived from SWD, enhancing feature alignment between shapes treated as discrete probability measures. We also introduce an adaptive refinement process utilizing entropy regularized OT, further refining feature alignments for accurate point-to-point correspondences. Our method demonstrates superior performance in non-rigid shape matching, including near-isometric and non-isometric scenarios, and excels in downstream tasks like segmentation transfer. The empirical results on diverse datasets highlight our framework's effectiveness and generalization capabilities, setting new standards in non-rigid shape matching with efficient OT metrics and an adaptive refinement module.

Title: K-stars LDP: A Novel Framework for (p, q)-clique Enumeration under Local Differential Privacy

• Authors: Henan Sun, Zhengyu Wu, Rong-Hua Li, Guoren Wang, Zening Li
• Subjects: cs.CR, cs.CY, cs.DS
• Abstract URL: https://arxiv.org/abs/2403.01788
• Pdf URL: https://arxiv.org/pdf/2403.01788
• Copy Paste: [[2403.01788]] K-stars LDP: A Novel Framework for (p, q)-clique Enumeration under Local Differential Privacy(https://arxiv.org/abs/2403.01788)
• Keywords: privacy, protect, attack
• Abstract: (p,q)-clique enumeration on a bipartite graph is critical for calculating clustering coefficient and detecting densest subgraph. It is necessary to carry out subgraph enumeration while protecting users' privacy from any potential attacker as the count of subgraph may contain sensitive information. Most recent studies focus on the privacy protection algorithms based on edge LDP (Local Differential Privacy). However, these algorithms suffer a large estimation error due to the great amount of required noise. In this paper, we propose a novel idea of k-stars LDP and a novel k-stars LDP algorithm for (p, q)-clique enumeration with a small estimation error, where a k-stars is a star-shaped graph with k nodes connecting to one node. The effectiveness of edge LDP relies on its capacity to obfuscate the existence of an edge between the user and his one-hop neighbors. This is based on the premise that a user should be aware of the existence of his one-hop neighbors. Similarly, we can apply this premise to k-stars as well, where an edge is a specific genre of 1-stars. Based on this fact, we first propose the k-stars neighboring list to enable our algorithm to obfuscate the existence of k-stars with Warner' s RR. Then, we propose the absolute value correction technique and the k-stars sampling technique to further reduce the estimation error. Finally, with the two-round user-collector interaction mechanism, we propose our k-stars LDP algorithm to count the number of (p, q)-clique while successfully protecting users' privacy. Both the theoretical analysis and experiments have showed the superiority of our algorithm over the algorithms based on edge LDP.

Title: DECOR: Enhancing Logic Locking Against Machine Learning-Based Attacks

• Authors: Yinghua Hu, Kaixin Yang, Subhajit Dutta Chowdhury, Pierluigi Nuzzo
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.01789
• Pdf URL: https://arxiv.org/pdf/2403.01789
• Copy Paste: [[2403.01789]] DECOR: Enhancing Logic Locking Against Machine Learning-Based Attacks(https://arxiv.org/abs/2403.01789)
• Keywords: protect, attack
• Abstract: Logic locking (LL) has gained attention as a promising intellectual property protection measure for integrated circuits. However, recent attacks, facilitated by machine learning (ML), have shown the potential to predict the correct key in multiple LL schemes by exploiting the correlation of the correct key value with the circuit structure. This paper presents a generic LL enhancement method based on a randomized algorithm that can significantly decrease the correlation between locked circuit netlist and correct key values in an LL scheme. Numerical results show that the proposed method can efficiently degrade the accuracy of state-of-the-art ML-based attacks down to around 50%, resulting in negligible advantage versus random guessing.

Title: COLA: Cross-city Mobility Transformer for Human Trajectory Simulation

• Authors: Yu Wang, Tongya Zheng, Yuxuan Liang, Shunyu Liu, Mingli Song
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01801
• Pdf URL: https://arxiv.org/pdf/2403.01801
• Copy Paste: [[2403.01801]] COLA: Cross-city Mobility Transformer for Human Trajectory Simulation(https://arxiv.org/abs/2403.01801)
• Keywords: privacy, transformer
• Abstract: Human trajectory data produced by daily mobile devices has proven its usefulness in various substantial fields such as urban planning and epidemic prevention. In terms of the individual privacy concern, human trajectory simulation has attracted increasing attention from researchers, targeting at offering numerous realistic mobility data for downstream tasks. Nevertheless, the prevalent issue of data scarcity undoubtedly degrades the reliability of existing deep learning models. In this paper, we are motivated to explore the intriguing problem of mobility transfer across cities, grasping the universal patterns of human trajectories to augment the powerful Transformer with external mobility data. There are two crucial challenges arising in the knowledge transfer across cities: 1) how to transfer the Transformer to adapt for domain heterogeneity; 2) how to calibrate the Transformer to adapt for subtly different long-tail frequency distributions of locations. To address these challenges, we have tailored a Cross-city mObiLity trAnsformer (COLA) with a dedicated model-agnostic transfer framework by effectively transferring cross-city knowledge for human trajectory simulation. Firstly, COLA divides the Transformer into the private modules for city-specific characteristics and the shared modules for city-universal mobility patterns. Secondly, COLA leverages a lightweight yet effective post-hoc adjustment strategy for trajectory simulation, without disturbing the complex bi-level optimization of model-agnostic knowledge transfer. Extensive experiments of COLA compared to state-of-the-art single-city baselines and our implemented cross-city baselines have demonstrated its superiority and effectiveness. The code is available at https://github.com/Star607/Cross-city-Mobility-Transformer.

Title: TNF: Tri-branch Neural Fusion for Multimodal Medical Data Classification

• Authors: Tong Zheng, Shusaku Sone, Yoshitaka Ushiku, Yuki Oba, Jiaxin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01802
• Pdf URL: https://arxiv.org/pdf/2403.01802
• Copy Paste: [[2403.01802]] TNF: Tri-branch Neural Fusion for Multimodal Medical Data Classification(https://arxiv.org/abs/2403.01802)
• Keywords: transformer
• Abstract: This paper presents a Tri-branch Neural Fusion (TNF) approach designed for classifying multimodal medical images and tabular data. It also introduces two solutions to address the challenge of label inconsistency in multimodal classification. Traditional methods in multi-modality medical data classification often rely on single-label approaches, typically merging features from two distinct input modalities. This becomes problematic when features are mutually exclusive or labels differ across modalities, leading to reduced accuracy. To overcome this, our TNF approach implements a tri-branch framework that manages three separate outputs: one for image modality, another for tabular modality, and a third hybrid output that fuses both image and tabular data. The final decision is made through an ensemble method that integrates likelihoods from all three branches. We validate the effectiveness of TNF through extensive experiments, which illustrate its superiority over traditional fusion and ensemble methods in various convolutional neural networks and transformer-based architectures across multiple datasets.

Title: PointCore: Efficient Unsupervised Point Cloud Anomaly Detector Using Local-Global Features

• Authors: Baozhu Zhao, Qiwei Xiong, Xiaohan Zhang, Jingfeng Guo, Qi Liu, Xiaofen Xing, Xiangmin Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01804
• Pdf URL: https://arxiv.org/pdf/2403.01804
• Copy Paste: [[2403.01804]] PointCore: Efficient Unsupervised Point Cloud Anomaly Detector Using Local-Global Features(https://arxiv.org/abs/2403.01804)
• Keywords: robust
• Abstract: Three-dimensional point cloud anomaly detection that aims to detect anomaly data points from a training set serves as the foundation for a variety of applications, including industrial inspection and autonomous driving. However, existing point cloud anomaly detection methods often incorporate multiple feature memory banks to fully preserve local and global representations, which comes at the high cost of computational complexity and mismatches between features. To address that, we propose an unsupervised point cloud anomaly detection framework based on joint local-global features, termed PointCore. To be specific, PointCore only requires a single memory bank to store local (coordinate) and global (PointMAE) representations and different priorities are assigned to these local-global features, thereby reducing the computational cost and mismatching disturbance in inference. Furthermore, to robust against the outliers, a normalization ranking method is introduced to not only adjust values of different scales to a notionally common scale, but also transform densely-distributed data into a uniform distribution. Extensive experiments on Real3D-AD dataset demonstrate that PointCore achieves competitive inference time and the best performance in both detection and localization as compared to the state-of-the-art Reg3D-AD approach and several competitors.

Title: ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models

• Authors: Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung-Yu Tseng, Christian Richardt, Michael Zollhöfer, Matthias Nießner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01807
• Pdf URL: https://arxiv.org/pdf/2403.01807
• Copy Paste: [[2403.01807]] ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models(https://arxiv.org/abs/2403.01807)
• Keywords: diffusion
• Abstract: 3D asset generation is getting massive amounts of attention, inspired by the recent success of text-guided 2D content creation. Existing text-to-3D methods use pretrained text-to-image diffusion models in an optimization problem or fine-tune them on synthetic data, which often results in non-photorealistic 3D objects without backgrounds. In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data. Concretely, we propose to integrate 3D volume-rendering and cross-frame-attention layers into each block of the existing U-Net network of the text-to-image model. Moreover, we design an autoregressive generation that renders more 3D-consistent images at any viewpoint. We train our model on real-world datasets of objects and showcase its capabilities to generate instances with a variety of high-quality shapes and textures in authentic surroundings. Compared to the existing methods, the results generated by our method are consistent, and have favorable visual quality (-30% FID, -37% KID).

Title: Deployment Challenges of Industrial Intrusion Detection Systems

• Authors: Konrad Wolsing, Eric Wagner, Frederik Basels, Patrick Wagner, Klaus Wehrle
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01809
• Pdf URL: https://arxiv.org/pdf/2403.01809
• Copy Paste: [[2403.01809]] Deployment Challenges of Industrial Intrusion Detection Systems(https://arxiv.org/abs/2403.01809)
• Keywords: security, attack
• Abstract: With the escalating threats posed by cyberattacks on Industrial Control Systems (ICSs), the development of customized Industrial Intrusion Detection Systems (IIDSs) received significant attention in research. While existing literature proposes effective IIDS solutions evaluated in controlled environments, their deployment in real-world industrial settings poses several challenges. This paper highlights two critical yet often overlooked aspects that significantly impact their practical deployment, i.e., the need for sufficient amounts of data to train the IIDS models and the challenges associated with finding suitable hyperparameters, especially for IIDSs training only on genuine ICS data. Through empirical experiments conducted on multiple state-of-the-art IIDSs and diverse datasets, we establish the criticality of these issues in deploying IIDSs. Our findings show the necessity of extensive malicious training data for supervised IIDSs, which can be impractical considering the complexity of recording and labeling attacks in actual industrial environments. Furthermore, while other IIDSs circumvent the previous issue by requiring only benign training data, these can suffer from the difficulty of setting appropriate hyperparameters, which likewise can diminish their performance. By shedding light on these challenges, we aim to enhance the understanding of the limitations and considerations necessary for deploying effective cybersecurity solutions in ICSs, which might be one reason why IIDSs see few deployments.

Title: Enhancing Multi-Domain Automatic Short Answer Grading through an Explainable Neuro-Symbolic Pipeline

• Authors: Felix Künnecke, Anna Filighera, Colin Leong, Tim Steuer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01811
• Pdf URL: https://arxiv.org/pdf/2403.01811
• Copy Paste: [[2403.01811]] Enhancing Multi-Domain Automatic Short Answer Grading through an Explainable Neuro-Symbolic Pipeline(https://arxiv.org/abs/2403.01811)
• Keywords: transformer
• Abstract: Grading short answer questions automatically with interpretable reasoning behind the grading decision is a challenging goal for current transformer approaches. Justification cue detection, in combination with logical reasoners, has shown a promising direction for neuro-symbolic architectures in ASAG. But, one of the main challenges is the requirement of annotated justification cues in the students' responses, which only exist for a few ASAG datasets. To overcome this challenge, we contribute (1) a weakly supervised annotation procedure for justification cues in ASAG datasets, and (2) a neuro-symbolic model for explainable ASAG based on justification cues. Our approach improves upon the RMSE by 0.24 to 0.3 compared to the state-of-the-art on the Short Answer Feedback dataset in a bilingual, multi-domain, and multi-question training setup. This result shows that our approach provides a promising direction for generating high-quality grades and accompanying explanations for future research in ASAG and educational NLP.

Title: AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation

• Authors: Haonan Wang, Qixiang Zhang, Yi Li, Xiaomeng Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01818
• Pdf URL: https://arxiv.org/pdf/2403.01818
• Copy Paste: [[2403.01818]] AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation(https://arxiv.org/abs/2403.01818)
• Keywords: transformer, segmentation
• Abstract: Semi-supervised semantic segmentation (SSSS) has been proposed to alleviate the burden of time-consuming pixel-level manual labeling, which leverages limited labeled data along with larger amounts of unlabeled data. Current state-of-the-art methods train the labeled data with ground truths and unlabeled data with pseudo labels. However, the two training flows are separate, which allows labeled data to dominate the training process, resulting in low-quality pseudo labels and, consequently, sub-optimal results. To alleviate this issue, we present AllSpark, which reborns the labeled features from unlabeled ones with the channel-wise cross-attention mechanism. We further introduce a Semantic Memory along with a Channel Semantic Grouping strategy to ensure that unlabeled features adequately represent labeled features. The AllSpark shed new light on the architecture level designs of SSSS rather than framework level, which avoids increasingly complicated training pipeline designs. It can also be regarded as a flexible bottleneck module that can be seamlessly integrated into a general transformer-based segmentation model. The proposed AllSpark outperforms existing methods across all evaluation protocols on Pascal, Cityscapes and COCO benchmarks without bells-and-whistles. Code and model weights are available at: https://github.com/xmed-lab/AllSpark.

Title: One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models

• Authors: Lin Li, Haoyan Guan, Jianing Qiu, Michael Spratling
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01849
• Pdf URL: https://arxiv.org/pdf/2403.01849
• Copy Paste: [[2403.01849]] One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models(https://arxiv.org/abs/2403.01849)
• Keywords: defense, attack, robust
• Abstract: Large pre-trained Vision-Language Models (VLMs) like CLIP, despite having remarkable generalization ability, are highly vulnerable to adversarial examples. This work studies the adversarial robustness of VLMs from the novel perspective of the text prompt instead of the extensively studied model weights (frozen in this work). We first show that the effectiveness of both adversarial attack and defense are sensitive to the used text prompt. Inspired by this, we propose a method to improve resilience to adversarial attacks by learning a robust text prompt for VLMs. The proposed method, named Adversarial Prompt Tuning (APT), is effective while being both computationally and data efficient. Extensive experiments are conducted across 15 datasets and 4 data sparsity schemes (from 1-shot to full training data settings) to show APT's superiority over hand-engineered prompts and other state-of-the-art adaption methods. APT demonstrated excellent abilities in terms of the in-distribution performance and the generalization under input distribution shift and across datasets. Surprisingly, by simply adding one learned word to the prompts, APT can significantly boost the accuracy and robustness (epsilon=4/255) over the hand-engineered prompts by +13% and +8.5% on average respectively. The improvement further increases, in our most effective setting, to +26.4% for accuracy and +16.7% for robustness. Code is available at https://github.com/TreeLLi/APT.

Title: Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral

• Authors: Yiming Cui, Xin Yao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01851
• Pdf URL: https://arxiv.org/pdf/2403.01851
• Copy Paste: [[2403.01851]] Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral(https://arxiv.org/abs/2403.01851)
• Keywords: large language model
• Abstract: Mixtral, a representative sparse mixture of experts (SMoE) language model, has received significant attention due to its unique model design and superior performance. Based on Mixtral-8x7B-v0.1, in this paper, we propose Chinese-Mixtral and Chinese-Mixtral-Instruct with improved Chinese language abilities by adopting further pre-training and instruction fine-tuning. Experimental results show that our Chinese-Mixtral and Chinese-Mixtral-Instruct successfully improve Chinese understanding and generation performance while retaining the original English abilities. Then, we discuss several key questions when performing language adaptation on large language models, including the necessity of extending the language-specific vocabulary and the choice of the initialization model (foundation model v.s. instruction model), by providing empirical results and analysis. We also present the visualizations of each expert to examine their importance on downstream tasks. Our resources are publicly available through \url{https://github.com/ymcui/Chinese-Mixtral}.

Title: An Improved Traditional Chinese Evaluation Suite for Foundation Model

• Authors: Zhi-Rui Tam, Ya-Ting Pai, Yen-Wei Lee, Sega Cheng, Hong-Han Shuai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01858
• Pdf URL: https://arxiv.org/pdf/2403.01858
• Copy Paste: [[2403.01858]] An Improved Traditional Chinese Evaluation Suite for Foundation Model(https://arxiv.org/abs/2403.01858)
• Keywords: large language model
• Abstract: We present TMMLU+, a comprehensive dataset designed for the Traditional Chinese massive multitask language understanding dataset. TMMLU+ is a multiple-choice question-answering dataset with 66 subjects from elementary to professional level. Compared to its predecessor, TMMLU, TMMLU+ is six times larger and boasts a more balanced subject distribution. We included benchmark results in TMMLU+ from closed-source models and 24 open-weight Chinese large language models of parameters ranging from 1.8B to 72B. Our findings reveal that Traditional Chinese models still trail behind their Simplified Chinese counterparts. Additionally, current large language models have yet to outperform human performance in average scores. We publicly release our dataset and the corresponding benchmark source code.

Title: CSE: Surface Anomaly Detection with Contrastively Selected Embedding

• Authors: Simon Thomine, Hichem Snoussi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01859
• Pdf URL: https://arxiv.org/pdf/2403.01859
• Copy Paste: [[2403.01859]] CSE: Surface Anomaly Detection with Contrastively Selected Embedding(https://arxiv.org/abs/2403.01859)
• Keywords: extraction
• Abstract: Detecting surface anomalies of industrial materials poses a significant challenge within a myriad of industrial manufacturing processes. In recent times, various methodologies have emerged, capitalizing on the advantages of employing a network pre-trained on natural images for the extraction of representative features. Subsequently, these features are subjected to processing through a diverse range of techniques including memory banks, normalizing flow, and knowledge distillation, which have exhibited exceptional accuracy. This paper revisits approaches based on pre-trained features by introducing a novel method centered on target-specific embedding. To capture the most representative features of the texture under consideration, we employ a variant of a contrastive training procedure that incorporates both artificially generated defective samples and anomaly-free samples during training. Exploiting the intrinsic properties of surfaces, we derived a meaningful representation from the defect-free samples during training, facilitating a straightforward yet effective calculation of anomaly scores. The experiments conducted on the MVTEC AD and TILDA datasets demonstrate the competitiveness of our approach compared to state-of-the-art methods.

Title: MaliGNNoma: GNN-Based Malicious Circuit Classifier for Secure Cloud FPGAs

• Authors: Lilas Alrahis, Hassan Nassar, Jonas Krautter, Dennis Gnad, Lars Bauer, Jorg Henkel, Mehdi Tahoori
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01860
• Pdf URL: https://arxiv.org/pdf/2403.01860
• Copy Paste: [[2403.01860]] MaliGNNoma: GNN-Based Malicious Circuit Classifier for Secure Cloud FPGAs(https://arxiv.org/abs/2403.01860)
• Keywords: secure, security, attack
• Abstract: The security of cloud field-programmable gate arrays (FPGAs) faces challenges from untrusted users attempting fault and side-channel attacks through malicious circuit configurations. Fault injection attacks can result in denial of service, disrupting functionality or leaking secret information. This threat is further amplified in multi-tenancy scenarios. Detecting such threats before loading onto the FPGA is crucial, but existing methods face difficulty identifying sophisticated attacks. We present MaliGNNoma, a machine learning-based solution that accurately identifies malicious FPGA configurations. Serving as a netlist scanning mechanism, it can be employed by cloud service providers as an initial security layer within a necessary multi-tiered security system. By leveraging the inherent graph representation of FPGA netlists, MaliGNNoma employs a graph neural network (GNN) to learn distinctive malicious features, surpassing current approaches. To enhance transparency, MaliGNNoma utilizes a parameterized explainer for the GNN, labeling the FPGA configuration and pinpointing the sub-circuit responsible for the malicious classification. Through extensive experimentation on the ZCU102 board with a Xilinx UltraScale+ FPGA, we validate the effectiveness of MaliGNNoma in detecting malicious configurations, including sophisticated attacks, such as those based on benign modules, like cryptography accelerators. MaliGNNoma achieves a classification accuracy and precision of 98.24% and 97.88%, respectively, surpassing state-of-the-art. We compare MaliGNNoma with five state-of-the-art scanning methods, revealing that not all attack vectors detected by MaliGNNoma are recognized by existing solutions, further emphasizing its effectiveness. Additionally, we make MaliGNNoma and its associated dataset publicly available.

Title: MTS: Bringing Multi-Tenancy to Virtual Networking

• Authors: Kashyap Thimmaraju, Saad Hermak, Gábor Rétvári, Stefan Schmid
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.01862
• Pdf URL: https://arxiv.org/pdf/2403.01862
• Copy Paste: [[2403.01862]] MTS: Bringing Multi-Tenancy to Virtual Networking(https://arxiv.org/abs/2403.01862)
• Keywords: secure, security
• Abstract: Multi-tenant cloud computing provides great benefits in terms of resource sharing, elastic pricing, and scalability, however, it also changes the security landscape and intro- duces the need for strong isolation between the tenants, also inside the network. This paper is motivated by the observation that while multi-tenancy is widely used in cloud computing, the virtual switch designs currently used for network virtualization lack sufficient support for tenant isolation. Hence, we present, implement, and evaluate a virtual switch architecture, MTS, which brings secure design best-practice to the context of multi-tenant virtual networking: compartmentalization of virtual switches, least-privilege execution, complete mediation of all network communication, and reducing the trusted computing base shared between tenants. We build MTS from commodity components, providing an incrementally deployable and inexpensive upgrade path to cloud operators. Our extensive experiments, extending to both micro-benchmarks and cloud applications, show that, depending on the way it is deployed, MTS may produce 1.5- 2x the throughput compared to state-of-the-art, with similar or better latency and modest resource overhead (1 extra CPU). MTS is available as open source software.

Title: Penetration Testing of 5G Core Network Web Technologies

• Authors: Filippo Giambartolomei, Marc Barceló, Alessandro Brighente, Aitor Urbieta, Mauro Conti
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01871
• Pdf URL: https://arxiv.org/pdf/2403.01871
• Copy Paste: [[2403.01871]] Penetration Testing of 5G Core Network Web Technologies(https://arxiv.org/abs/2403.01871)
• Keywords: security, attack
• Abstract: Thanks to technologies such as virtual network function the Fifth Generation (5G) of mobile networks dynamically allocate resources to different types of users in an on-demand fashion. Virtualization extends up to the 5G core, where software-defined networks and network slicing implement a customizable environment. These technologies can be controlled via application programming interfaces and web technologies, inheriting hence their security risks and settings. An attacker exploiting vulnerable implementations of the 5G core may gain privileged control of the network assets and disrupt its availability. However, there is currently no security assessment of the web security of the 5G core network. In this paper, we present the first security assessment of the 5G core from a web security perspective. We use the STRIDE threat modeling approach to define a complete list of possible threat vectors and associated attacks. Thanks to a suite of security testing tools, we cover all of these threats and test the security of the 5G core. In particular, we test the three most relevant open-source 5G core implementations, i.e., Open5GS, Free5Gc, and OpenAirInterface. Our analysis shows that all these cores are vulnerable to at least two of our identified attack vectors, demanding increased security measures in the development of future 5G core networks.

Title: FCDS: Fusing Constituency and Dependency Syntax into Document-Level Relation Extraction

• Authors: Xudong Zhu, Zhao Kang, Bei Hui
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01886
• Pdf URL: https://arxiv.org/pdf/2403.01886
• Copy Paste: [[2403.01886]] FCDS: Fusing Constituency and Dependency Syntax into Document-Level Relation Extraction(https://arxiv.org/abs/2403.01886)
• Keywords: extraction
• Abstract: Document-level Relation Extraction (DocRE) aims to identify relation labels between entities within a single document. It requires handling several sentences and reasoning over them. State-of-the-art DocRE methods use a graph structure to connect entities across the document to capture dependency syntax information. However, this is insufficient to fully exploit the rich syntax information in the document. In this work, we propose to fuse constituency and dependency syntax into DocRE. It uses constituency syntax to aggregate the whole sentence information and select the instructive sentences for the pairs of targets. It exploits the dependency syntax in a graph structure with constituency syntax enhancement and chooses the path between entity pairs based on the dependency graph. The experimental results on datasets from various domains demonstrate the effectiveness of the proposed method. The code is publicly available at this url.

Title: Unsupervised Distance Metric Learning for Anomaly Detection Over Multivariate Time Series

• Authors: Hanyang Yuan, Qinglin Cai, Keting Yin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01895
• Pdf URL: https://arxiv.org/pdf/2403.01895
• Copy Paste: [[2403.01895]] Unsupervised Distance Metric Learning for Anomaly Detection Over Multivariate Time Series(https://arxiv.org/abs/2403.01895)
• Keywords: robust, interpretability
• Abstract: Distance-based time series anomaly detection methods are prevalent due to their relative non-parametric nature and interpretability. However, the commonly used Euclidean distance is sensitive to noise. While existing works have explored dynamic time warping (DTW) for its robustness, they only support supervised tasks over multivariate time series (MTS), leaving a scarcity of unsupervised methods. In this work, we propose FCM-wDTW, an unsupervised distance metric learning method for anomaly detection over MTS, which encodes raw data into latent space and reveals normal dimension relationships through cluster centers. FCM-wDTW introduces locally weighted DTW into fuzzy C-means clustering and learns the optimal latent space efficiently, enabling anomaly identification via data reconstruction. Experiments with 11 different types of benchmarks demonstrate our method's competitive accuracy and efficiency.

Title: Robustness Bounds on the Successful Adversarial Examples: Theory and Practice

• Authors: Hiroaki Maeshima, Akira Otsuka
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.01896
• Pdf URL: https://arxiv.org/pdf/2403.01896
• Copy Paste: [[2403.01896]] Robustness Bounds on the Successful Adversarial Examples: Theory and Practice(https://arxiv.org/abs/2403.01896)
• Keywords: attack, robust
• Abstract: Adversarial example (AE) is an attack method for machine learning, which is crafted by adding imperceptible perturbation to the data inducing misclassification. In the current paper, we investigated the upper bound of the probability of successful AEs based on the Gaussian Process (GP) classification. We proved a new upper bound that depends on AE's perturbation norm, the kernel function used in GP, and the distance of the closest pair with different labels in the training dataset. Surprisingly, the upper bound is determined regardless of the distribution of the sample dataset. We showed that our theoretical result was confirmed through the experiment using ImageNet. In addition, we showed that changing the parameters of the kernel function induces a change of the upper bound of the probability of successful AEs.

Title: Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina PT* Family

• Authors: Rodrigo Santos, João Rodrigues, Luís Gomes, João Silva, António Branco, Henrique Lopes Cardoso, Tomás Freitas Osório, Bernardo Leite
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01897
• Pdf URL: https://arxiv.org/pdf/2403.01897
• Copy Paste: [[2403.01897]] Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina PT* Family(https://arxiv.org/abs/2403.01897)
• Keywords: large language model
• Abstract: To foster the neural encoding of Portuguese, this paper contributes foundation encoder models that represent an expansion of the still very scarce ecosystem of large language models specifically developed for this language that are fully open, in the sense that they are open source and openly distributed for free under an open license for any purpose, thus including research and commercial usages. Like most languages other than English, Portuguese is low-resourced in terms of these foundational language resources, there being the inaugural 900 million parameter Albertina and 335 million Bertimbau. Taking this couple of models as an inaugural set, we present the extension of the ecosystem of state-of-the-art open encoders for Portuguese with a larger, top performance-driven model with 1.5 billion parameters, and a smaller, efficiency-driven model with 100 million parameters. While achieving this primary goal, further results that are relevant for this ecosystem were obtained as well, namely new datasets for Portuguese based on the SuperGLUE benchmark, which we also distribute openly.

Title: FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio

• Authors: Chao Xu, Yang Liu, Jiazheng Xing, Weida Wang, Mingze Sun, Jun Dan, Tianxin Huang, Siyuan Li, Zhi-Qi Cheng, Ying Tai, Baigui Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01901
• Pdf URL: https://arxiv.org/pdf/2403.01901
• Copy Paste: [[2403.01901]] FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio(https://arxiv.org/abs/2403.01901)
• Keywords: diffusion
• Abstract: In this paper, we abstract the process of people hearing speech, extracting meaningful cues, and creating various dynamically audio-consistent talking faces, termed Listening and Imagining, into the task of high-fidelity diverse talking faces generation from a single audio. Specifically, it involves two critical challenges: one is to effectively decouple identity, content, and emotion from entangled audio, and the other is to maintain intra-video diversity and inter-video consistency. To tackle the issues, we first dig out the intricate relationships among facial factors and simplify the decoupling process, tailoring a Progressive Audio Disentanglement for accurate facial geometry and semantics learning, where each stage incorporates a customized training module responsible for a specific factor. Secondly, to achieve visually diverse and audio-synchronized animation solely from input audio within a single model, we introduce the Controllable Coherent Frame generation, which involves the flexible integration of three trainable adapters with frozen Latent Diffusion Models (LDMs) to focus on maintaining facial geometry and semantics, as well as texture and temporal coherence between frames. In this way, we inherit high-quality diverse generation from LDMs while significantly improving their controllability at a low training cost. Extensive experiments demonstrate the flexibility and effectiveness of our method in handling this paradigm. The codes will be released at https://github.com/modelscope/facechain.

Title: Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Survey

• Authors: Lingyan Ran, Yali Li, Guoqiang Liang, Yanning Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01909
• Pdf URL: https://arxiv.org/pdf/2403.01909
• Copy Paste: [[2403.01909]] Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Survey(https://arxiv.org/abs/2403.01909)
• Keywords: segmentation
• Abstract: Semantic segmentation is an important and popular research area in computer vision that focuses on classifying pixels in an image based on their semantics. However, supervised deep learning requires large amounts of data to train models and the process of labeling images pixel by pixel is time-consuming and laborious. This review aims to provide a first comprehensive and organized overview of the state-of-the-art research results on pseudo-label methods in the field of semi-supervised semantic segmentation, which we categorize from different perspectives and present specific methods for specific application areas. In addition, we explore the application of pseudo-label technology in medical and remote-sensing image segmentation. Finally, we also propose some feasible future research directions to address the existing challenges.

Title: xT: Nested Tokenization for Larger Context in Large Images

• Authors: Ritwik Gupta, Shufan Li, Tyler Zhu, Jitendra Malik, Trevor Darrell, Karttikeya Mangalam
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01915
• Pdf URL: https://arxiv.org/pdf/2403.01915
• Copy Paste: [[2403.01915]] xT: Nested Tokenization for Larger Context in Large Images(https://arxiv.org/abs/2403.01915)
• Keywords: transformer, segmentation
• Abstract: Modern computer vision pipelines handle large images in one of two sub-optimal ways: down-sampling or cropping. These two methods incur significant losses in the amount of information and context present in an image. There are many downstream applications in which global context matters as much as high frequency details, such as in real-world satellite imagery; in such cases researchers have to make the uncomfortable choice of which information to discard. We introduce xT, a simple framework for vision transformers which effectively aggregates global context with local details and can model large images end-to-end on contemporary GPUs. We select a set of benchmark datasets across classic vision tasks which accurately reflect a vision model's ability to understand truly large images and incorporate fine details over large scales and assess our method's improvement on them. By introducing a nested tokenization scheme for large images in conjunction with long-sequence length models normally used for natural language processing, we are able to increase accuracy by up to 8.6% on challenging classification tasks and $F_1$ score by 11.6 on context-dependent segmentation in large images.

Title: To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering

• Authors: Giacomo Frisoni, Alessio Cocchieri, Alex Presepi, Gianluca Moro, Zaiqiao Meng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01924
• Pdf URL: https://arxiv.org/pdf/2403.01924
• Copy Paste: [[2403.01924]] To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering(https://arxiv.org/abs/2403.01924)
• Keywords: large language model
• Abstract: Medical open-domain question answering demands substantial access to specialized knowledge. Recent efforts have sought to decouple knowledge from model parameters, counteracting architectural scaling and allowing for training on common low-resource hardware. The retrieve-then-read paradigm has become ubiquitous, with model predictions grounded on relevant knowledge pieces from external repositories such as PubMed, textbooks, and UMLS. An alternative path, still under-explored but made possible by the advent of domain-specific large language models, entails constructing artificial contexts through prompting. As a result, "to generate or to retrieve" is the modern equivalent of Hamlet's dilemma. This paper presents MedGENIE, the first generate-then-read framework for multiple-choice question answering in medicine. We conduct extensive experiments on MedQA-USMLE, MedMCQA, and MMLU, incorporating a practical perspective by assuming a maximum of 24GB VRAM. MedGENIE sets a new state-of-the-art (SOTA) in the open-book setting of each testbed, even allowing a small-scale reader to outcompete zero-shot closed-book 175B baselines while using up to 706$\times$ fewer parameters. Overall, our findings reveal that generated passages are more effective than retrieved counterparts in attaining higher accuracy.

Title: Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?

• Authors: Evgeniia Razumovskaia, Ivan Vulić, Anna Korhonen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01929
• Pdf URL: https://arxiv.org/pdf/2403.01929
• Copy Paste: [[2403.01929]] Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?(https://arxiv.org/abs/2403.01929)
• Keywords: large language model
• Abstract: Supervised fine-tuning (SFT), supervised instruction tuning (SIT) and in-context learning (ICL) are three alternative, de facto standard approaches to few-shot learning. ICL has gained popularity recently with the advent of LLMs due to its simplicity and sample efficiency. Prior research has conducted only limited investigation into how these approaches work for multilingual few-shot learning, and the focus so far has been mostly on their performance. In this work, we present an extensive and systematic comparison of the three approaches, testing them on 6 high- and low-resource languages, three different NLU tasks, and a myriad of language and domain setups. Importantly, performance is only one aspect of the comparison, where we also analyse the approaches through the optics of their computational, inference and financial costs. Our observations show that supervised instruction tuning has the best trade-off between performance and resource requirements. As another contribution, we analyse the impact of target language adaptation of pretrained LLMs and find that the standard adaptation approaches can (superficially) improve target language generation capabilities, but language understanding elicited through ICL does not improve and remains limited, with low scores especially for low-resource languages.

Title: Mitigating Label Noise on Graph via Topological Sample Selection

• Authors: Yuhao Wu, Jiangchao Yao, Xiaobo Xia, Jun Yu, Ruxin Wang, Bo Han, Tongliang Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01942
• Pdf URL: https://arxiv.org/pdf/2403.01942
• Copy Paste: [[2403.01942]] Mitigating Label Noise on Graph via Topological Sample Selection(https://arxiv.org/abs/2403.01942)
• Keywords: robust
• Abstract: Despite the success of the carefully-annotated benchmarks, the effectiveness of existing graph neural networks (GNNs) can be considerably impaired in practice when the real-world graph data is noisily labeled. Previous explorations in sample selection have been demonstrated as an effective way for robust learning with noisy labels, however, the conventional studies focus on i.i.d data, and when moving to non-iid graph data and GNNs, two notable challenges remain: (1) nodes located near topological class boundaries are very informative for classification but cannot be successfully distinguished by the heuristic sample selection. (2) there is no available measure that considers the graph topological information to promote sample selection in a graph. To address this dilemma, we propose a $\textit{Topological Sample Selection}$ (TSS) method that boosts the informative sample selection process in a graph by utilising topological information. We theoretically prove that our procedure minimizes an upper bound of the expected risk under target clean distribution, and experimentally show the superiority of our method compared with state-of-the-art baselines.

Title: Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification

• Authors: Puru Vaish, Shunxin Wang, Nicola Strisciuglio
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01944
• Pdf URL: https://arxiv.org/pdf/2403.01944
• Copy Paste: [[2403.01944]] Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification(https://arxiv.org/abs/2403.01944)
• Keywords: robust
• Abstract: Computer vision models normally witness degraded performance when deployed in real-world scenarios, due to unexpected changes in inputs that were not accounted for during training. Data augmentation is commonly used to address this issue, as it aims to increase data variety and reduce the distribution gap between training and test data. However, common visual augmentations might not guarantee extensive robustness of computer vision models. In this paper, we propose Auxiliary Fourier-basis Augmentation (AFA), a complementary technique targeting augmentation in the frequency domain and filling the augmentation gap left by visual augmentations. We demonstrate the utility of augmentation via Fourier-basis additive noise in a straightforward and efficient adversarial setting. Our results show that AFA benefits the robustness of models against common corruptions, OOD generalization, and consistency of performance of models against increasing perturbations, with negligible deficit to the standard performance of models. It can be seamlessly integrated with other augmentation techniques to further boost performance.

Title: A Generative Model of Symmetry Transformations

• Authors: James Urquhart Allingham, Bruno Kacper Mlodozeniec, Shreyas Padhy, Javier Antorán, David Krueger, Richard E. Turner, Eric Nalisnick, José Miguel Hernández-Lobato
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01946
• Pdf URL: https://arxiv.org/pdf/2403.01946
• Copy Paste: [[2403.01946]] A Generative Model of Symmetry Transformations(https://arxiv.org/abs/2403.01946)
• Keywords: robust, generative
• Abstract: Correctly capturing the symmetry transformations of data can lead to efficient models with strong generalization capabilities, though methods incorporating symmetries often require prior knowledge. While recent advancements have been made in learning those symmetries directly from the dataset, most of this work has focused on the discriminative setting. In this paper, we construct a generative model that explicitly aims to capture symmetries in the data, resulting in a model that learns which symmetries are present in an interpretable way. We provide a simple algorithm for efficiently learning our generative model and demonstrate its ability to capture symmetries under affine and color transformations. Combining our symmetry model with existing generative models results in higher marginal test-log-likelihoods and robustness to data sparsification.

Title: Enhancing Information Maximization with Distance-Aware Contrastive Learning for Source-Free Cross-Domain Few-Shot Learning

• Authors: Huali Xu, Li Liu, Shuaifeng Zhi, Shaojing Fu, Zhuo Su, Ming-Ming Cheng, Yongxiang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01966
• Pdf URL: https://arxiv.org/pdf/2403.01966
• Copy Paste: [[2403.01966]] Enhancing Information Maximization with Distance-Aware Contrastive Learning for Source-Free Cross-Domain Few-Shot Learning(https://arxiv.org/abs/2403.01966)
• Keywords: privacy
• Abstract: Existing Cross-Domain Few-Shot Learning (CDFSL) methods require access to source domain data to train a model in the pre-training phase. However, due to increasing concerns about data privacy and the desire to reduce data transmission and training costs, it is necessary to develop a CDFSL solution without accessing source data. For this reason, this paper explores a Source-Free CDFSL (SF-CDFSL) problem, in which CDFSL is addressed through the use of existing pretrained models instead of training a model with source data, avoiding accessing source data. This paper proposes an Enhanced Information Maximization with Distance-Aware Contrastive Learning (IM-DCL) method to address these challenges. Firstly, we introduce the transductive mechanism for learning the query set. Secondly, information maximization (IM) is explored to map target samples into both individual certainty and global diversity predictions, helping the source model better fit the target data distribution. However, IM fails to learn the decision boundary of the target task. This motivates us to introduce a novel approach called Distance-Aware Contrastive Learning (DCL), in which we consider the entire feature set as both positive and negative sets, akin to Schrodinger's concept of a dual state. Instead of a rigid separation between positive and negative sets, we employ a weighted distance calculation among features to establish a soft classification of the positive and negative sets for the entire feature set. Furthermore, we address issues related to IM by incorporating contrastive constraints between object features and their corresponding positive and negative sets. Evaluations of the 4 datasets in the BSCD-FSL benchmark indicate that the proposed IM-DCL, without accessing the source domain, demonstrates superiority over existing methods, especially in the distant domain task.

Title: Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection

• Authors: Xin Zhang, Tao Xiao, Gepeng Ji, Xuan Wu, Keren Fu, Qijun Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01968
• Pdf URL: https://arxiv.org/pdf/2403.01968
• Copy Paste: [[2403.01968]] Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection(https://arxiv.org/abs/2403.01968)
• Keywords: robust, segmentation
• Abstract: Camouflage poses challenges in distinguishing a static target, whereas any movement of the target can break this disguise. Existing video camouflaged object detection (VCOD) approaches take noisy motion estimation as input or model motion implicitly, restricting detection performance in complex dynamic scenes. In this paper, we propose a novel Explicit Motion handling and Interactive Prompting framework for VCOD, dubbed EMIP, which handles motion cues explicitly using a frozen pre-trained optical flow fundamental model. EMIP is characterized by a two-stream architecture for simultaneously conducting camouflaged segmentation and optical flow estimation. Interactions across the dual streams are realized in an interactive prompting way that is inspired by emerging visual prompt learning. Two learnable modules, i.e. the camouflaged feeder and motion collector, are designed to incorporate segmentation-to-motion and motion-to-segmentation prompts, respectively, and enhance outputs of the both streams. The prompt fed to the motion stream is learned by supervising optical flow in a self-supervised manner. Furthermore, we show that long-term historical information can also be incorporated as a prompt into EMIP and achieve more robust results with temporal consistency. Experimental results demonstrate that our EMIP achieves new state-of-the-art records on popular VCOD benchmarks. The code will be publicly available.

Title: AS-ES Learning: Towards Efficient CoT Learning in Small Models

• Authors: Nuwa Xi, Yuhan Chen, Sendong Zhao, Haochun Wang, Bing Qin, Ting Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01969
• Pdf URL: https://arxiv.org/pdf/2403.01969
• Copy Paste: [[2403.01969]] AS-ES Learning: Towards Efficient CoT Learning in Small Models(https://arxiv.org/abs/2403.01969)
• Keywords: large language model
• Abstract: Chain-of-Thought (CoT) serves as a critical emerging ability in LLMs, especially when it comes to logical reasoning. Attempts have been made to induce such ability in small models as well by distilling from the data with CoT generated by Large Language Models (LLMs). However, existing methods often simply generate and incorporate more data from LLMs and fail to note the importance of efficiently utilizing existing CoT data. We here propose a new training paradigm AS-ES (Abstractive Segments - Extractive Segments) learning, which exploits the inherent information in CoT for iterative generation. Experiments show that our methods surpass the direct seq2seq training on CoT-extensive tasks like MWP and PET summarization, without data augmentation or altering the model itself. Furthermore, we explore the reason behind the inefficiency of small models in learning CoT and provide an explanation of why AS-ES learning works, giving insights into the underlying mechanism of CoT.

Title: Multi-perspective Improvement of Knowledge Graph Completion with Large Language Models

• Authors: Derong Xu, Ziheng Zhang, Zhenxi Lin, Xian Wu, Zhihong Zhu, Tong Xu, Xiangyu Zhao, Yefeng Zheng, Enhong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01972
• Pdf URL: https://arxiv.org/pdf/2403.01972
• Copy Paste: [[2403.01972]] Multi-perspective Improvement of Knowledge Graph Completion with Large Language Models(https://arxiv.org/abs/2403.01972)
• Keywords: large language model
• Abstract: Knowledge graph completion (KGC) is a widely used method to tackle incompleteness in knowledge graphs (KGs) by making predictions for missing links. Description-based KGC leverages pre-trained language models to learn entity and relation representations with their names or descriptions, which shows promising results. However, the performance of description-based KGC is still limited by the quality of text and the incomplete structure, as it lacks sufficient entity descriptions and relies solely on relation names, leading to sub-optimal results. To address this issue, we propose MPIKGC, a general framework to compensate for the deficiency of contextualized knowledge and improve KGC by querying large language models (LLMs) from various perspectives, which involves leveraging the reasoning, explanation, and summarization capabilities of LLMs to expand entity descriptions, understand relations, and extract structures, respectively. We conducted extensive evaluation of the effectiveness and improvement of our framework based on four description-based KGC models and four datasets, for both link prediction and triplet classification tasks.

Title: SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis

• Authors: Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Changxin Wang, Zhifeng Gao, Yongge Li, Mujie Lin, Shuwen Yang, Jiankun Wang, Yuqi Yin, Yaqi Li, Linfeng Zhang, Guolin Ke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01976
• Pdf URL: https://arxiv.org/pdf/2403.01976
• Copy Paste: [[2403.01976]] SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis(https://arxiv.org/abs/2403.01976)
• Keywords: large language model
• Abstract: Recent breakthroughs in Large Language Models (LLMs) have revolutionized natural language understanding and generation, igniting a surge of interest in leveraging these technologies for the nuanced field of scientific literature analysis. Existing benchmarks, however, inadequately evaluate the proficiency of LLMs in the scientific domain, especially in scenarios involving complex comprehension and multimodal data. In response, we introduced SciAssess, a benchmark tailored for the in-depth analysis of scientific literature, crafted to provide a thorough assessment of LLMs' efficacy. SciAssess focuses on evaluating LLMs' abilities in memorization, comprehension, and analysis within scientific contexts. It includes representative tasks from diverse scientific fields, such as general chemistry, organic materials, and alloy materials. And rigorous quality control measures ensure its reliability in terms of correctness, anonymization, and copyright compliance. SciAssess evaluates leading LLMs, including GPT-4, GPT-3.5-turbo, and Gemini, identifying their strengths and areas for improvement and supporting the ongoing development of LLM applications in scientific literature analysis. SciAssess and its resources are made available at https://sci-assess.github.io, offering a valuable tool for advancing LLM capabilities in scientific literature analysis.

Title: Transformers for Low-Resource Languages:Is Féidir Linn!

• Authors: Séamus Lankford, Haithem Afli, Andy Way
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01985
• Pdf URL: https://arxiv.org/pdf/2403.01985
• Copy Paste: [[2403.01985]] Transformers for Low-Resource Languages:Is Féidir Linn!(https://arxiv.org/abs/2403.01985)
• Keywords: transformer
• Abstract: The Transformer model is the state-of-the-art in Machine Translation. However, in general, neural translation models often under perform on language pairs with insufficient training data. As a consequence, relatively few experiments have been carried out using this architecture on low-resource language pairs. In this study, hyperparameter optimization of Transformer models in translating the low-resource English-Irish language pair is evaluated. We demonstrate that choosing appropriate parameters leads to considerable performance improvements. Most importantly, the correct choice of subword model is shown to be the biggest driver of translation performance. SentencePiece models using both unigram and BPE approaches were appraised. Variations on model architectures included modifying the number of layers, testing various regularisation techniques and evaluating the optimal number of heads for attention. A generic 55k DGT corpus and an in-domain 88k public admin corpus were used for evaluation. A Transformer optimized model demonstrated a BLEU score improvement of 7.8 points when compared with a baseline RNN model. Improvements were observed across a range of metrics, including TER, indicating a substantially reduced post editing effort for Transformer optimized models with 16k BPE subword models. Bench-marked against Google Translate, our translation engines demonstrated significant improvements. The question of whether or not Transformers can be used effectively in a low-resource setting of English-Irish translation has been addressed. Is f\'eidir linn - yes we can.

Title: Physics-Informed Learning for Time-Resolved Angiographic Contrast Agent Concentration Reconstruction

• Authors: Noah Maul, Annette Birkhold, Fabian Wagner, Mareike Thies, Maximilian Rohleder, Philipp Berg, Markus Kowarschik, Andreas Maier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01993
• Pdf URL: https://arxiv.org/pdf/2403.01993
• Copy Paste: [[2403.01993]] Physics-Informed Learning for Time-Resolved Angiographic Contrast Agent Concentration Reconstruction(https://arxiv.org/abs/2403.01993)
• Keywords: robust
• Abstract: Three-dimensional Digital Subtraction Angiography (3D-DSA) is a well-established X-ray-based technique for visualizing vascular anatomy. Recently, four-dimensional DSA (4D-DSA) reconstruction algorithms have been developed to enable the visualization of volumetric contrast flow dynamics through time-series of volumes. . This reconstruction problem is ill-posed mainly due to vessel overlap in the projection direction and geometric vessel foreshortening, which leads to information loss in the recorded projection images. However, knowledge about the underlying fluid dynamics can be leveraged to constrain the solution space. In our work, we implicitly include this information in a neural network-based model that is trained on a dataset of image-based blood flow simulations. The model predicts the spatially averaged contrast agent concentration for each centerline point of the vasculature over time, lowering the overall computational demand. The trained network enables the reconstruction of relative contrast agent concentrations with a mean absolute error of 0.02 $\pm$ 0.02 and a mean absolute percentage error of 5.31 % $\pm$ 9.25 %. Moreover, the network is robust to varying degrees of vessel overlap and vessel foreshortening. Our approach demonstrates the potential of the integration of machine learning and blood flow simulations in time-resolved angiographic flow reconstruction.

Title: Vanilla Transformers are Transfer Capability Teachers

• Authors: Xin Lu, Yanyan Zhao, Bing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01994
• Pdf URL: https://arxiv.org/pdf/2403.01994
• Copy Paste: [[2403.01994]] Vanilla Transformers are Transfer Capability Teachers(https://arxiv.org/abs/2403.01994)
• Keywords: transformer
• Abstract: Recently, Mixture of Experts (MoE) Transformers have garnered increasing attention due to their advantages in model capacity and computational efficiency. However, studies have indicated that MoE Transformers underperform vanilla Transformers in many downstream tasks, significantly diminishing the practical value of MoE models. To explain this issue, we propose that the pre-training performance and transfer capability of a model are joint determinants of its downstream task performance. MoE models, in comparison to vanilla models, have poorer transfer capability, leading to their subpar performance in downstream tasks. To address this issue, we introduce the concept of transfer capability distillation, positing that although vanilla models have weaker performance, they are effective teachers of transfer capability. The MoE models guided by vanilla models can achieve both strong pre-training performance and transfer capability, ultimately enhancing their performance in downstream tasks. We design a specific distillation method and conduct experiments on the BERT architecture. Experimental results show a significant improvement in downstream performance of MoE models, and many further evidences also strongly support the concept of transfer capability distillation. Finally, we attempt to interpret transfer capability distillation and provide some insights from the perspective of model feature.

Title: LLM-Oriented Retrieval Tuner

• Authors: Si Sun, Hanqing Zhang, Zhiyuan Liu, Jie Bao, Dawei Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01999
• Pdf URL: https://arxiv.org/pdf/2403.01999
• Copy Paste: [[2403.01999]] LLM-Oriented Retrieval Tuner(https://arxiv.org/abs/2403.01999)
• Keywords: large language model
• Abstract: Dense Retrieval (DR) is now considered as a promising tool to enhance the memorization capacity of Large Language Models (LLM) such as GPT3 and GPT-4 by incorporating external memories. However, due to the paradigm discrepancy between text generation of LLM and DR, it is still an open challenge to integrate the retrieval and generation tasks in a shared LLM. In this paper, we propose an efficient LLM-Oriented Retrieval Tuner, namely LMORT, which decouples DR capacity from base LLM and non-invasively coordinates the optimally aligned and uniform layers of the LLM towards a unified DR space, achieving an efficient and effective DR without tuning the LLM itself. The extensive experiments on six BEIR datasets show that our approach could achieve competitive zero-shot retrieval performance compared to a range of strong DR models while maintaining the generation ability of LLM.

Title: Topic Aware Probing: From Sentence Length Prediction to Idiom Identification how reliant are Neural Language Models on Topic?

• Authors: Vasudevan Nedumpozhimana, John D. Kelleher
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02009
• Pdf URL: https://arxiv.org/pdf/2403.02009
• Copy Paste: [[2403.02009]] Topic Aware Probing: From Sentence Length Prediction to Idiom Identification how reliant are Neural Language Models on Topic?(https://arxiv.org/abs/2403.02009)
• Keywords: transformer
• Abstract: Transformer-based Neural Language Models achieve state-of-the-art performance on various natural language processing tasks. However, an open question is the extent to which these models rely on word-order/syntactic or word co-occurrence/topic-based information when processing natural language. This work contributes to this debate by addressing the question of whether these models primarily use topic as a signal, by exploring the relationship between Transformer-based models' (BERT and RoBERTa's) performance on a range of probing tasks in English, from simple lexical tasks such as sentence length prediction to complex semantic tasks such as idiom token identification, and the sensitivity of these tasks to the topic information. To this end, we propose a novel probing method which we call topic-aware probing. Our initial results indicate that Transformer-based models encode both topic and non-topic information in their intermediate layers, but also that the facility of these models to distinguish idiomatic usage is primarily based on their ability to identify and encode topic. Furthermore, our analysis of these models' performance on other standard probing tasks suggests that tasks that are relatively insensitive to the topic information are also tasks that are relatively difficult for these models.

Title: Unveiling Hidden Links Between Unseen Security Entities

• Authors: Daniel Alfasi, Tal Shapira, Anat Bremler Barr
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02014
• Pdf URL: https://arxiv.org/pdf/2403.02014
• Copy Paste: [[2403.02014]] Unveiling Hidden Links Between Unseen Security Entities(https://arxiv.org/abs/2403.02014)
• Keywords: security, large language model
• Abstract: The proliferation of software vulnerabilities poses a significant challenge for security databases and analysts tasked with their timely identification, classification, and remediation. With the National Vulnerability Database (NVD) reporting an ever-increasing number of vulnerabilities, the traditional manual analysis becomes untenably time-consuming and prone to errors. This paper introduces VulnScopper, an innovative approach that utilizes multi-modal representation learning, combining Knowledge Graphs (KG) and Natural Language Processing (NLP), to automate and enhance the analysis of software vulnerabilities. Leveraging ULTRA, a knowledge graph foundation model, combined with a Large Language Model (LLM), VulnScopper effectively handles unseen entities, overcoming the limitations of previous KG approaches. We evaluate VulnScopper on two major security datasets, the NVD and the Red Hat CVE database. Our method significantly improves the link prediction accuracy between Common Vulnerabilities and Exposures (CVEs), Common Weakness Enumeration (CWEs), and Common Platform Enumerations (CPEs). Our results show that VulnScopper outperforms existing methods, achieving up to 78% Hits@10 accuracy in linking CVEs to CPEs and CWEs and presenting an 11.7% improvement over large language models in predicting CWE labels based on the Red Hat database. Based on the NVD, only 6.37% of the linked CPEs are being published during the first 30 days; many of them are related to critical and high-risk vulnerabilities which, according to multiple compliance frameworks (such as CISA and PCI), should be remediated within 15-30 days. Our model can uncover new products linked to vulnerabilities, reducing remediation time and improving vulnerability management. We analyzed several CVEs from 2023 to showcase this ability.

Title: Scalable Vision-Based 3D Object Detection and Monocular Depth Estimation for Autonomous Driving

• Authors: Yuxuan Liu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.02037
• Pdf URL: https://arxiv.org/pdf/2403.02037
• Copy Paste: [[2403.02037]] Scalable Vision-Based 3D Object Detection and Monocular Depth Estimation for Autonomous Driving(https://arxiv.org/abs/2403.02037)
• Keywords: robust
• Abstract: This dissertation is a multifaceted contribution to the advancement of vision-based 3D perception technologies. In the first segment, the thesis introduces structural enhancements to both monocular and stereo 3D object detection algorithms. By integrating ground-referenced geometric priors into monocular detection models, this research achieves unparalleled accuracy in benchmark evaluations for monocular 3D detection. Concurrently, the work refines stereo 3D detection paradigms by incorporating insights and inferential structures gleaned from monocular networks, thereby augmenting the operational efficiency of stereo detection systems. The second segment is devoted to data-driven strategies and their real-world applications in 3D vision detection. A novel training regimen is introduced that amalgamates datasets annotated with either 2D or 3D labels. This approach not only augments the detection models through the utilization of a substantially expanded dataset but also facilitates economical model deployment in real-world scenarios where only 2D annotations are readily available. Lastly, the dissertation presents an innovative pipeline tailored for unsupervised depth estimation in autonomous driving contexts. Extensive empirical analyses affirm the robustness and efficacy of this newly proposed pipeline. Collectively, these contributions lay a robust foundation for the widespread adoption of vision-based 3D perception technologies in autonomous driving applications.

Title: A Generative Approach for Wikipedia-Scale Visual Entity Recognition

• Authors: Mathilde Caron, Ahmet Iscen, Alireza Fathi, Cordelia Schmid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02041
• Pdf URL: https://arxiv.org/pdf/2403.02041
• Copy Paste: [[2403.02041]] A Generative Approach for Wikipedia-Scale Visual Entity Recognition(https://arxiv.org/abs/2403.02041)
• Keywords: generative
• Abstract: In this paper, we address web-scale visual entity recognition, specifically the task of mapping a given query image to one of the 6 million existing entities in Wikipedia. One way of approaching a problem of such scale is using dual-encoder models (eg CLIP), where all the entity names and query images are embedded into a unified space, paving the way for an approximate k-NN search. Alternatively, it is also possible to re-purpose a captioning model to directly generate the entity names for a given image. In contrast, we introduce a novel Generative Entity Recognition (GER) framework, which given an input image learns to auto-regressively decode a semantic and discriminative ``code'' identifying the target entity. Our experiments demonstrate the efficacy of this GER paradigm, showcasing state-of-the-art performance on the challenging OVEN benchmark. GER surpasses strong captioning, dual-encoder, visual matching and hierarchical classification baselines, affirming its advantage in tackling the complexities of web-scale recognition.

Title: Depth-Guided Robust and Fast Point Cloud Fusion NeRF for Sparse Input Views

• Authors: Shuai Guo, Qiuwen Wang, Yijie Gao, Rong Xie, Li Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02063
• Pdf URL: https://arxiv.org/pdf/2403.02063
• Copy Paste: [[2403.02063]] Depth-Guided Robust and Fast Point Cloud Fusion NeRF for Sparse Input Views(https://arxiv.org/abs/2403.02063)
• Keywords: robust
• Abstract: Novel-view synthesis with sparse input views is important for real-world applications like AR/VR and autonomous driving. Recent methods have integrated depth information into NeRFs for sparse input synthesis, leveraging depth prior for geometric and spatial understanding. However, most existing works tend to overlook inaccuracies within depth maps and have low time efficiency. To address these issues, we propose a depth-guided robust and fast point cloud fusion NeRF for sparse inputs. We perceive radiance fields as an explicit voxel grid of features. A point cloud is constructed for each input view, characterized within the voxel grid using matrices and vectors. We accumulate the point cloud of each input view to construct the fused point cloud of the entire scene. Each voxel determines its density and appearance by referring to the point cloud of the entire scene. Through point cloud fusion and voxel grid fine-tuning, inaccuracies in depth values are refined or substituted by those from other views. Moreover, our method can achieve faster reconstruction and greater compactness through effective vector-matrix decomposition. Experimental results underline the superior performance and time efficiency of our approach compared to state-of-the-art baselines.

Title: HyperPredict: Estimating Hyperparameter Effects for Instance-Specific Regularization in Deformable Image Registration

• Authors: Aisha L. Shuaibu, Ivor J. A. Simpson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02069
• Pdf URL: https://arxiv.org/pdf/2403.02069
• Copy Paste: [[2403.02069]] HyperPredict: Estimating Hyperparameter Effects for Instance-Specific Regularization in Deformable Image Registration(https://arxiv.org/abs/2403.02069)
• Keywords: segmentation
• Abstract: Methods for medical image registration infer geometric transformations that align pairs/groups of images by maximising an image similarity metric. This problem is ill-posed as several solutions may have equivalent likelihoods, also optimising purely for image similarity can yield implausible transformations. For these reasons regularization terms are essential to obtain meaningful registration results. However, this requires the introduction of at least one hyperparameter often termed {\lambda}, that serves as a tradeoff between loss terms. In some situations, the quality of the estimated transformation greatly depends on hyperparameter choice, and different choices may be required depending on the characteristics of the data. Analyzing the effect of these hyperparameters requires labelled data, which is not commonly available at test-time. In this paper, we propose a method for evaluating the influence of hyperparameters and subsequently selecting an optimal value for given image pairs. Our approach which we call HyperPredict, implements a Multi-Layer Perceptron that learns the effect of selecting particular hyperparameters for registering an image pair by predicting the resulting segmentation overlap and measure of deformation smoothness. This approach enables us to select optimal hyperparameters at test time without requiring labelled data, removing the need for a one-size-fits-all cross-validation approach. Furthermore, the criteria used to define optimal hyperparameter is flexible post-training, allowing us to efficiently choose specific properties. We evaluate our proposed method on the OASIS brain MR dataset using a recent deep learning approach(cLapIRN) and an algorithmic method(Niftyreg). Our results demonstrate good performance in predicting the effects of regularization hyperparameters and highlight the benefits of our image-pair specific approach to hyperparameter selection.

Title: Modality-Aware and Shift Mixer for Multi-modal Brain Tumor Segmentation

• Authors: Zhongzhen Huang, Linda Wei, Shaoting Zhang, Xiaofan Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02074
• Pdf URL: https://arxiv.org/pdf/2403.02074
• Copy Paste: [[2403.02074]] Modality-Aware and Shift Mixer for Multi-modal Brain Tumor Segmentation(https://arxiv.org/abs/2403.02074)
• Keywords: robust, segmentation
• Abstract: Combining images from multi-modalities is beneficial to explore various information in computer vision, especially in the medical domain. As an essential part of clinical diagnosis, multi-modal brain tumor segmentation aims to delineate the malignant entity involving multiple modalities. Although existing methods have shown remarkable performance in the task, the information exchange for cross-scale and high-level representations fusion in spatial and modality are limited in these methods. In this paper, we present a novel Modality Aware and Shift Mixer that integrates intra-modality and inter-modality dependencies of multi-modal images for effective and robust brain tumor segmentation. Specifically, we introduce a Modality-Aware module according to neuroimaging studies for modeling the specific modality pair relationships at low levels, and a Modality-Shift module with specific mosaic patterns is developed to explore the complex relationships across modalities at high levels via the self-attention. Experimentally, we outperform previous state-of-the-art approaches on the public Brain Tumor Segmentation (BraTS 2021 segmentation) dataset. Further qualitative experiments demonstrate the efficacy and robustness of MASM.

Title: DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction

• Authors: Weiyi Lv, Yuhang Huang, Ning Zhang, Ruei-Sung Lin, Mei Han, Dan Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02075
• Pdf URL: https://arxiv.org/pdf/2403.02075
• Copy Paste: [[2403.02075]] DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction(https://arxiv.org/abs/2403.02075)
• Keywords: diffusion
• Abstract: In Multiple Object Tracking, objects often exhibit non-linear motion of acceleration and deceleration, with irregular direction changes. Tacking-by-detection (TBD) with Kalman Filter motion prediction works well in pedestrian-dominant scenarios but falls short in complex situations when multiple objects perform non-linear and diverse motion simultaneously. To tackle the complex non-linear motion, we propose a real-time diffusion-based MOT approach named DiffMOT. Specifically, for the motion predictor component, we propose a novel Decoupled Diffusion-based Motion Predictor (D MP). It models the entire distribution of various motion presented by the data as a whole. It also predicts an individual object's motion conditioning on an individual's historical motion information. Furthermore, it optimizes the diffusion process with much less sampling steps. As a MOT tracker, the DiffMOT is real-time at 22.7FPS, and also outperforms the state-of-the-art on DanceTrack and SportsMOT datasets with 63.4 and 76.2 in HOTA metrics, respectively. To the best of our knowledge, DiffMOT is the first to introduce a diffusion probabilistic model into the MOT to tackle non-linear motion prediction.

Title: Automated Generation of Multiple-Choice Cloze Questions for Assessing English Vocabulary Using GPT-turbo 3.5

• Authors: Qiao Wang, Ralph Rose, Naho Orita, Ayaka Sugawara
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02078
• Pdf URL: https://arxiv.org/pdf/2403.02078
• Copy Paste: [[2403.02078]] Automated Generation of Multiple-Choice Cloze Questions for Assessing English Vocabulary Using GPT-turbo 3.5(https://arxiv.org/abs/2403.02078)
• Keywords: large language model
• Abstract: A common way of assessing language learners' mastery of vocabulary is via multiple-choice cloze (i.e., fill-in-the-blank) questions. But the creation of test items can be laborious for individual teachers or in large-scale language programs. In this paper, we evaluate a new method for automatically generating these types of questions using large language models (LLM). The VocaTT (vocabulary teaching and training) engine is written in Python and comprises three basic steps: pre-processing target word lists, generating sentences and candidate word options using GPT, and finally selecting suitable word options. To test the efficiency of this system, 60 questions were generated targeting academic words. The generated items were reviewed by expert reviewers who judged the well-formedness of the sentences and word options, adding comments to items judged not well-formed. Results showed a 75% rate of well-formedness for sentences and 66.85% rate for suitable word options. This is a marked improvement over the generator used earlier in our research which did not take advantage of GPT's capabilities. Post-hoc qualitative analysis reveals several points for improvement in future work including cross-referencing part-of-speech tagging, better sentence validation, and improving GPT prompts.

Title: ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models

• Authors: Jiaxiang Cheng, Pan Xie, Xin Xia, Jiashi Li, Jie Wu, Yuxi Ren, Huixia Li, Xuefeng Xiao, Min Zheng, Lean Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02084
• Pdf URL: https://arxiv.org/pdf/2403.02084
• Copy Paste: [[2403.02084]] ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models(https://arxiv.org/abs/2403.02084)
• Keywords: diffusion
• Abstract: Recent advancement in text-to-image models (e.g., Stable Diffusion) and corresponding personalized technologies (e.g., DreamBooth and LoRA) enables individuals to generate high-quality and imaginative images. However, they often suffer from limitations when generating images with resolutions outside of their trained domain. To overcome this limitation, we present the Resolution Adapter (ResAdapter), a domain-consistent adapter designed for diffusion models to generate images with unrestricted resolutions and aspect ratios. Unlike other multi-resolution generation methods that process images of static resolution with complex post-process operations, ResAdapter directly generates images with the dynamical resolution. Especially, after learning a deep understanding of pure resolution priors, ResAdapter trained on the general dataset, generates resolution-free images with personalized diffusion models while preserving their original style domain. Comprehensive experiments demonstrate that ResAdapter with only 0.5M can process images with flexible resolutions for arbitrary diffusion models. More extended experiments demonstrate that ResAdapter is compatible with other modules (e.g., ControlNet, IP-Adapter and LCM-LoRA) for image generation across a broad range of resolutions, and can be integrated into other multi-resolution model (e.g., ElasticDiffusion) for efficiently generating higher-resolution images. Project link is https://res-adapter.github.io

Title: Inf2Guard: An Information-Theoretic Framework for Learning Privacy-Preserving Representations against Inference Attacks

• Authors: Sayedeh Leila Noorbakhsh, Binghui Zhang, Yuan Hong, Binghui Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02116
• Pdf URL: https://arxiv.org/pdf/2403.02116
• Copy Paste: [[2403.02116]] Inf2Guard: An Information-Theoretic Framework for Learning Privacy-Preserving Representations against Inference Attacks(https://arxiv.org/abs/2403.02116)
• Keywords: privacy, protect, defense, attack, membership infer
• Abstract: Machine learning (ML) is vulnerable to inference (e.g., membership inference, property inference, and data reconstruction) attacks that aim to infer the private information of training data or dataset. Existing defenses are only designed for one specific type of attack and sacrifice significant utility or are soon broken by adaptive attacks. We address these limitations by proposing an information-theoretic defense framework, called Inf2Guard, against the three major types of inference attacks. Our framework, inspired by the success of representation learning, posits that learning shared representations not only saves time/costs but also benefits numerous downstream tasks. Generally, Inf2Guard involves two mutual information objectives, for privacy protection and utility preservation, respectively. Inf2Guard exhibits many merits: it facilitates the design of customized objectives against the specific inference attack; it provides a general defense framework which can treat certain existing defenses as special cases; and importantly, it aids in deriving theoretical results, e.g., inherent utility-privacy tradeoff and guaranteed privacy leakage. Extensive evaluations validate the effectiveness of Inf2Guard for learning privacy-preserving representations against inference attacks and demonstrate the superiority over the baselines.

Title: Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language Models

• Authors: Sargam Yadav (1), Abhishek Kaushik (1), Kevin McDaid (1) ((1) Dundalk Institute of Technology, Dundalk)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02121
• Pdf URL: https://arxiv.org/pdf/2403.02121
• Copy Paste: [[2403.02121]] Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language Models(https://arxiv.org/abs/2403.02121)
• Keywords: transformer, generative, large language model
• Abstract: The advent of Large Language Models (LLMs) has advanced the benchmark in various Natural Language Processing (NLP) tasks. However, large amounts of labelled training data are required to train LLMs. Furthermore, data annotation and training are computationally expensive and time-consuming. Zero and few-shot learning have recently emerged as viable options for labelling data using large pre-trained models. Hate speech detection in mix-code low-resource languages is an active problem area where the use of LLMs has proven beneficial. In this study, we have compiled a dataset of 100 YouTube comments, and weakly labelled them for coarse and fine-grained misogyny classification in mix-code Hinglish. Weak annotation was applied due to the labor-intensive annotation process. Zero-shot learning, one-shot learning, and few-shot learning and prompting approaches have then been applied to assign labels to the comments and compare them to human-assigned labels. Out of all the approaches, zero-shot classification using the Bidirectional Auto-Regressive Transformers (BART) large model and few-shot prompting using Generative Pre-trained Transformer- 3 (ChatGPT-3) achieve the best results

Title: LOCR: Location-Guided Transformer for Optical Character Recognition

• Authors: Yu Sun, Dongzhan Zhou, Chen Lin, Conghui He, Wanli Ouyang, Han-Sen Zhong
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02127
• Pdf URL: https://arxiv.org/pdf/2403.02127
• Copy Paste: [[2403.02127]] LOCR: Location-Guided Transformer for Optical Character Recognition(https://arxiv.org/abs/2403.02127)
• Keywords: transformer
• Abstract: Academic documents are packed with texts, equations, tables, and figures, requiring comprehensive understanding for accurate Optical Character Recognition (OCR). While end-to-end OCR methods offer improved accuracy over layout-based approaches, they often grapple with significant repetition issues, especially with complex layouts in Out-Of-Domain (OOD) documents.To tackle this issue, we propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on a dataset comprising over 77M text-location pairs from 125K academic document pages, including bounding boxes for words, tables and mathematical symbols. LOCR adeptly handles various formatting elements and generates content in Markdown language. It outperforms all existing methods in our test set constructed from arXiv, as measured by edit distance, BLEU, METEOR and F-measure.LOCR also reduces repetition frequency from 4.4% of pages to 0.5% in the arXiv dataset, from 13.2% to 1.3% in OOD quantum physics documents and from 8.1% to 1.8% in OOD marketing documents. Additionally, LOCR features an interactive OCR mode, facilitating the generation of complex documents through a few location prompts from human.

Title: Using LLMs for the Extraction and Normalization of Product Attribute Values

• Authors: Nick Baumann, Alexander Brinkmann, Christian Bizer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02130
• Pdf URL: https://arxiv.org/pdf/2403.02130
• Copy Paste: [[2403.02130]] Using LLMs for the Extraction and Normalization of Product Attribute Values(https://arxiv.org/abs/2403.02130)
• Keywords: extraction, large language model
• Abstract: Product offers on e-commerce websites often consist of a textual product title and a textual product description. In order to provide features such as faceted product filtering or content-based product recommendation, the websites need to extract attribute-value pairs from the unstructured product descriptions. This paper explores the potential of using large language models (LLMs), such as OpenAI's GPT-3.5 and GPT-4, to extract and normalize attribute values from product titles and product descriptions. For our experiments, we introduce the WDC Product Attribute-Value Extraction (WDC PAVE) dataset. WDC PAVE consists of product offers from 87 websites that provide schema.org annotations. The offers belong to five different categories, each featuring a specific set of attributes. The dataset provides manually verified attribute-value pairs in two forms: (i) directly extracted values and (ii) normalized attribute values. The normalization of the attribute values requires systems to perform the following types of operations: name expansion, generalization, unit of measurement normalization, and string wrangling. Our experiments demonstrate that GPT-4 outperforms PLM-based extraction methods by 10%, achieving an F1-Score of 91%. For the extraction and normalization of product attribute values, GPT-4 achieves a similar performance to the extraction scenario, while being particularly strong at string wrangling and name expansion.

Title: UB-FineNet: Urban Building Fine-grained Classification Network for Open-access Satellite Images

• Authors: Zhiyi He, Wei Yao, Jie Shao, Puzuo Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02132
• Pdf URL: https://arxiv.org/pdf/2403.02132
• Copy Paste: [[2403.02132]] UB-FineNet: Urban Building Fine-grained Classification Network for Open-access Satellite Images(https://arxiv.org/abs/2403.02132)
• Keywords: robust, diffusion
• Abstract: Fine classification of city-scale buildings from satellite remote sensing imagery is a crucial research area with significant implications for urban planning, infrastructure development, and population distribution analysis. However, the task faces big challenges due to low-resolution overhead images acquired from high altitude space-borne platforms and the long-tail sample distribution of fine-grained urban building categories, leading to severe class imbalance problem. To address these issues, we propose a deep network approach to fine-grained classification of urban buildings using open-access satellite images. A Denoising Diffusion Probabilistic Model (DDPM) based super-resolution method is first introduced to enhance the spatial resolution of satellite images, which benefits from domain-adaptive knowledge distillation. Then, a new fine-grained classification network with Category Information Balancing Module (CIBM) and Contrastive Supervision (CS) technique is proposed to mitigate the problem of class imbalance and improve the classification robustness and accuracy. Experiments on Hong Kong data set with 11 fine building types revealed promising classification results with a mean Top-1 accuracy of 60.45\%, which is on par with street-view image based approaches. Extensive ablation study shows that CIBM and CS improve Top-1 accuracy by 2.6\% and 3.5\% compared to the baseline method, respectively. And both modules can be easily inserted into other classification networks and similar enhancements have been achieved. Our research contributes to the field of urban analysis by providing a practical solution for fine classification of buildings in challenging mega city scenarios solely using open-access satellite images. The proposed method can serve as a valuable tool for urban planners, aiding in the understanding of economic, industrial, and population distribution.

Title: Point2Building: Reconstructing Buildings from Airborne LiDAR Point Clouds

• Authors: Yujia Liu, Anton Obukhov, Jan Dirk Wegner, Konrad Schindler
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02136
• Pdf URL: https://arxiv.org/pdf/2403.02136
• Copy Paste: [[2403.02136]] Point2Building: Reconstructing Buildings from Airborne LiDAR Point Clouds(https://arxiv.org/abs/2403.02136)
• Keywords: generative
• Abstract: We present a learning-based approach to reconstruct buildings as 3D polygonal meshes from airborne LiDAR point clouds. What makes 3D building reconstruction from airborne LiDAR hard is the large diversity of building designs and especially roof shapes, the low and varying point density across the scene, and the often incomplete coverage of building facades due to occlusions by vegetation or to the viewing angle of the sensor. To cope with the diversity of shapes and inhomogeneous and incomplete object coverage, we introduce a generative model that directly predicts 3D polygonal meshes from input point clouds. Our autoregressive model, called Point2Building, iteratively builds up the mesh by generating sequences of vertices and faces. This approach enables our model to adapt flexibly to diverse geometries and building structures. Unlike many existing methods that rely heavily on pre-processing steps like exhaustive plane detection, our model learns directly from the point cloud data, thereby reducing error propagation and increasing the fidelity of the reconstruction. We experimentally validate our method on a collection of airborne LiDAR data of Zurich, Berlin and Tallinn. Our method shows good generalization to diverse urban styles.

Title: Self-Supervised Facial Representation Learning with Facial Region Awareness

• Authors: Zheng Gao, Ioannis Patras
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02138
• Pdf URL: https://arxiv.org/pdf/2403.02138
• Copy Paste: [[2403.02138]] Self-Supervised Facial Representation Learning with Facial Region Awareness(https://arxiv.org/abs/2403.02138)
• Keywords: segmentation
• Abstract: Self-supervised pre-training has been proved to be effective in learning transferable representations that benefit various visual tasks. This paper asks this question: can self-supervised pre-training learn general facial representations for various facial analysis tasks? Recent efforts toward this goal are limited to treating each face image as a whole, i.e., learning consistent facial representations at the image-level, which overlooks the consistency of local facial representations (i.e., facial regions like eyes, nose, etc). In this work, we make a first attempt to propose a novel self-supervised facial representation learning framework to learn consistent global and local facial representations, Facial Region Awareness (FRA). Specifically, we explicitly enforce the consistency of facial regions by matching the local facial representations across views, which are extracted with learned heatmaps highlighting the facial regions. Inspired by the mask prediction in supervised semantic segmentation, we obtain the heatmaps via cosine similarity between the per-pixel projection of feature maps and facial mask embeddings computed from learnable positional embeddings, which leverage the attention mechanism to globally look up the facial image for facial regions. To learn such heatmaps, we formulate the learning of facial mask embeddings as a deep clustering problem by assigning the pixel features from the feature maps to them. The transfer learning results on facial classification and regression tasks show that our FRA outperforms previous pre-trained models and more importantly, using ResNet as the unified backbone for various tasks, our FRA achieves comparable or even better performance compared with SOTA methods in facial analysis tasks.

Title: MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection

• Authors: Tianxiang Chen, Zhentao Tan, Tao Gong, Qi Chu, Yue Wu, Bin Liu, Jieping Ye, Nenghai Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02148
• Pdf URL: https://arxiv.org/pdf/2403.02148
• Copy Paste: [[2403.02148]] MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection(https://arxiv.org/abs/2403.02148)
• Keywords: transformer
• Abstract: Thanks to the development of basic models, infrared small target detection (ISTD) algorithms have made significant progress. Specifically, the structures combining convolutional networks with transformers can well extract both local and global features. At the same time, they also inherit defects from the basic model, e.g., the quadratic computational complexity of transformers, which impacts efficiency. Inspired by a recent basic model with linear complexity for long-distance modeling, called Mamba, we explore the potential of this state space model in ISTD in this paper. However, direct application is unsuitable since local features, which are critical to detecting small targets, cannot be fully exploited. Instead, we tailor a Mamba-in-Mamba (MiM-ISTD) structure for efficient ISTD. For example, we treat the local patches as "visual sentences" and further decompose them into sub-patches as "visual words" to further explore the locality. The interactions among each word in a given visual sentence will be calculated with negligible computational costs. By aggregating the word and sentence features, the representation ability of MiM-ISTD can be significantly bolstered. Experiments on NUAA-SIRST and IRSTD-1k prove the superior accuracy and efficiency of our method. Specifically, MiM-ISTD is $10 \times$ faster than the SOTA and reduces GPU memory usage by 73.4$\%$ per $2048 \times 2048$ image during inference, overcoming the computation$\&$memory constraints on performing Mamba-based understanding on high-resolution infrared images.Source code is available at https://github.com/txchen-USTC/MiM-ISTD.

Title: TripoSR: Fast 3D Object Reconstruction from a Single Image

• Authors: Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, Yan-Pei Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02151
• Pdf URL: https://arxiv.org/pdf/2403.02151
• Copy Paste: [[2403.02151]] TripoSR: Fast 3D Object Reconstruction from a Single Image(https://arxiv.org/abs/2403.02151)
• Keywords: transformer, generative
• Abstract: This technical report introduces TripoSR, a 3D reconstruction model leveraging transformer architecture for fast feed-forward 3D generation, producing 3D mesh from a single image in under 0.5 seconds. Building upon the LRM network architecture, TripoSR integrates substantial improvements in data processing, model design, and training techniques. Evaluations on public datasets show that TripoSR exhibits superior performance, both quantitatively and qualitatively, compared to other open-source alternatives. Released under the MIT license, TripoSR is intended to empower researchers, developers, and creatives with the latest advancements in 3D generative AI.

Title: Mirage: Defense against CrossPath Attacks in Software Defined Networks

• Authors: Shariq Murtuza, Krishna Asawa
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02172
• Pdf URL: https://arxiv.org/pdf/2403.02172
• Copy Paste: [[2403.02172]] Mirage: Defense against CrossPath Attacks in Software Defined Networks(https://arxiv.org/abs/2403.02172)
• Keywords: security, protect, defense, attack
• Abstract: The Software-Defined Networks (SDNs) face persistent threats from various adversaries that attack them using different methods to mount Denial of Service attacks. These attackers have different motives and follow diverse tactics to achieve their nefarious objectives. In this work, we focus on the impact of CrossPath attacks in SDNs and introduce our framework, Mirage, which not only detects but also mitigates this attack. Our framework, Mirage, detects SDN switches that become unreachable due to being under attack, takes proactive measures to prevent Adversarial Path Reconnaissance, and effectively mitigates CrossPath attacks in SDNs. A CrossPath attack is a form of link flood attack that indirectly attacks the control plane by overwhelming the shared links that connect the data and control planes with data plane traffic. This attack is exclusive to in band SDN, where the data and the control plane, both utilize the same physical links for transmitting and receiving traffic. Our framework, Mirage, prevents attackers from launching adversarial path reconnaissance to identify shared links in a network, thereby thwarting their abuse and preventing this attack. Mirage not only stops adversarial path reconnaissance but also includes features to quickly counter ongoing attacks once detected. Mirage uses path diversity to reroute network packet to prevent timing based measurement. Mirage can also enforce short lived flow table rules to prevent timing attacks. These measures are carefully designed to enhance the security of the SDN environment. Moreover, we share the results of our experiments, which clearly show Mirage's effectiveness in preventing path reconnaissance, detecting CrossPath attacks, and mitigating ongoing threats. Our framework successfully protects the network from these harmful activities, giving valuable insights into SDN security.

Title: ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context

• Authors: Zirui Wu, Yansong Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02177
• Pdf URL: https://arxiv.org/pdf/2403.02177
• Copy Paste: [[2403.02177]] ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context(https://arxiv.org/abs/2403.02177)
• Keywords: interpretability
• Abstract: Tables play a crucial role in conveying information in various domains, serving as indispensable tools for organizing and presenting data in a structured manner. We propose a Plan-then-Reason framework to answer different types of user queries over tables with sentence context. The framework first plans the reasoning paths over the context, then assigns each step to program-based or textual reasoning to reach the final answer. We construct an instruction tuning set TrixInstruct following the framework. Our dataset cover queries that are program-unsolvable or need combining information from tables and sentences to obtain planning and reasoning abilities. We present ProTrix by finetuning Llama-2-7B on TrixInstruct. Our experiments show that ProTrix generalizes to diverse tabular tasks and achieves comparable performance to GPT-3.5-turbo. We further demonstrate that ProTrix can generate accurate and faithful explanations to answer complex free-form questions. Our work underscores the importance of the planning and reasoning abilities towards a model over tabular tasks with generalizability and interpretability. We will release our dataset and model at https://github.com/WilliamZR/ProTrix.

Title: Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models

• Authors: Changyu Chen, Xiting Wang, Ting-En Lin, Ang Lv, Yuchuan Wu, Xin Gao, Ji-Rong Wen, Rui Yan, Yongbin Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02178
• Pdf URL: https://arxiv.org/pdf/2403.02178
• Copy Paste: [[2403.02178]] Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models(https://arxiv.org/abs/2403.02178)
• Keywords: large language model
• Abstract: In reasoning tasks, even a minor error can cascade into inaccurate results, leading to suboptimal performance of large language models in such domains. Earlier fine-tuning approaches sought to mitigate this by leveraging more precise supervisory signals from human labeling, larger models, or self-sampling, although at a high cost. Conversely, we develop a method that avoids external resources, relying instead on introducing perturbations to the input. Our training approach randomly masks certain tokens within the chain of thought, a technique we found to be particularly effective for reasoning tasks. When applied to fine-tuning with GSM8K, this method achieved a 5% improvement in accuracy over standard supervised fine-tuning with a few codes modified and no additional labeling effort. Furthermore, it is complementary to existing methods. When integrated with related data augmentation methods, it leads to an average improvement of 3% improvement in GSM8K accuracy and 1% improvement in MATH accuracy across five datasets of various quality and size, as well as two base models. We further investigate the mechanisms behind this improvement through case studies and quantitative analysis, suggesting that our approach may provide superior support for the model in capturing long-distance dependencies, especially those related to questions. This enhancement could deepen understanding of premises in questions and prior steps. Our code is available at Github.

Title: Not all Layers of LLMs are Necessary during Inference

• Authors: Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, Zhongyuan Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02181
• Pdf URL: https://arxiv.org/pdf/2403.02181
• Copy Paste: [[2403.02181]] Not all Layers of LLMs are Necessary during Inference(https://arxiv.org/abs/2403.02181)
• Keywords: large language model
• Abstract: The inference phase of Large Language Models (LLMs) is very expensive. An ideal inference stage of LLMs could utilize fewer computational resources while still maintaining its capabilities (e.g., generalization and in-context learning ability). In this paper, we try to answer the question, "During LLM inference, can we use shallow layers for easy instances; and deep layers for hard ones?" To answer this question, we first indicate that Not all Layers are Necessary during Inference by statistically analyzing the activated layers across tasks. Then, we propose a simple algorithm named AdaInfer to determine the inference termination moment based on the input instance adaptively. More importantly, AdaInfer does not alter LLM parameters and maintains generalizability across tasks. Experiments on well-known LLMs (i.e., Llama2 series and OPT) show that AdaInfer saves an average of 14.8% of computational resources, even up to 50% on sentiment tasks, while maintaining comparable performance. Additionally, this method is orthogonal to other model acceleration techniques, potentially boosting inference efficiency further.

Title: Perceptive self-supervised learning network for noisy image watermark removal

• Authors: Chunwei Tian, Menghua Zheng, Bo Li, Yanning Zhang, Shichao Zhang, David Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02211
• Pdf URL: https://arxiv.org/pdf/2403.02211
• Copy Paste: [[2403.02211]] Perceptive self-supervised learning network for noisy image watermark removal(https://arxiv.org/abs/2403.02211)
• Keywords: robust, watermark
• Abstract: Popular methods usually use a degradation model in a supervised way to learn a watermark removal model. However, it is true that reference images are difficult to obtain in the real world, as well as collected images by cameras suffer from noise. To overcome these drawbacks, we propose a perceptive self-supervised learning network for noisy image watermark removal (PSLNet) in this paper. PSLNet depends on a parallel network to remove noise and watermarks. The upper network uses task decomposition ideas to remove noise and watermarks in sequence. The lower network utilizes the degradation model idea to simultaneously remove noise and watermarks. Specifically, mentioned paired watermark images are obtained in a self supervised way, and paired noisy images (i.e., noisy and reference images) are obtained in a supervised way. To enhance the clarity of obtained images, interacting two sub-networks and fusing obtained clean images are used to improve the effects of image watermark removal in terms of structural information and pixel enhancement. Taking into texture information account, a mixed loss uses obtained images and features to achieve a robust model of noisy image watermark removal. Comprehensive experiments show that our proposed method is very effective in comparison with popular convolutional neural networks (CNNs) for noisy image watermark removal. Codes can be obtained at https://github.com/hellloxiaotian/PSLNet.

Title: DragTex: Generative Point-Based Texture Editing on 3D Mesh

• Authors: Yudi Zhang, Qi Xu, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02217
• Pdf URL: https://arxiv.org/pdf/2403.02217
• Copy Paste: [[2403.02217]] DragTex: Generative Point-Based Texture Editing on 3D Mesh(https://arxiv.org/abs/2403.02217)
• Keywords: diffusion, generative
• Abstract: Creating 3D textured meshes using generative artificial intelligence has garnered significant attention recently. While existing methods support text-based generative texture generation or editing on 3D meshes, they often struggle to precisely control pixels of texture images through more intuitive interaction. While 2D images can be edited generatively using drag interaction, applying this type of methods directly to 3D mesh textures still leads to issues such as the lack of local consistency among multiple views, error accumulation and long training times. To address these challenges, we propose a generative point-based 3D mesh texture editing method called DragTex. This method utilizes a diffusion model to blend locally inconsistent textures in the region near the deformed silhouette between different views, enabling locally consistent texture editing. Besides, we fine-tune a decoder to reduce reconstruction errors in the non-drag region, thereby mitigating overall error accumulation. Moreover, we train LoRA using multi-view images instead of training each view individually, which significantly shortens the training time. The experimental results show that our method effectively achieves dragging textures on 3D meshes and generates plausible textures that align with the desired intent of drag interaction.

Title: TPLLM: A Traffic Prediction Framework Based on Pretrained Large Language Models

• Authors: Yilong Ren, Yue Chen, Shuai Liu, Boyue Wang, Haiyang Yu, Zhiyong Cui
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02221
• Pdf URL: https://arxiv.org/pdf/2403.02221
• Copy Paste: [[2403.02221]] TPLLM: A Traffic Prediction Framework Based on Pretrained Large Language Models(https://arxiv.org/abs/2403.02221)
• Keywords: large language model
• Abstract: Traffic prediction constitutes a pivotal facet within the purview of Intelligent Transportation Systems (ITS), and the attainment of highly precise predictions holds profound significance for efficacious traffic management. The precision of prevailing deep learning-driven traffic prediction models typically sees an upward trend with a rise in the volume of training data. However, the procurement of comprehensive spatiotemporal datasets for traffic is often fraught with challenges, primarily stemming from the substantial costs associated with data collection and retention. Consequently, developing a model that can achieve accurate predictions and good generalization ability in areas with limited historical traffic data is a challenging problem. It is noteworthy that the rapidly advancing pretrained Large Language Models (LLMs) of recent years have demonstrated exceptional proficiency in cross-modality knowledge transfer and few-shot learning. Recognizing the sequential nature of traffic data, similar to language, we introduce TPLLM, a novel traffic prediction framework leveraging LLMs. In this framework, we construct a sequence embedding layer based on Convolutional Neural Networks (CNNs) and a graph embedding layer based on Graph Convolutional Networks (GCNs) to extract sequence features and spatial features, respectively. These are subsequently integrated to form inputs that are suitable for LLMs. A Low-Rank Adaptation (LoRA) fine-tuning approach is applied to TPLLM, thereby facilitating efficient learning and minimizing computational demands. Experiments on two real-world datasets demonstrate that TPLLM exhibits commendable performance in both full-sample and few-shot prediction scenarios, effectively supporting the development of ITS in regions with scarce historical traffic data.

Title: Building Trust in Data for IoT Systems

• Authors: Davide Margaria, Alberto Carelli, Andrea Vesco
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02225
• Pdf URL: https://arxiv.org/pdf/2403.02225
• Copy Paste: [[2403.02225]] Building Trust in Data for IoT Systems(https://arxiv.org/abs/2403.02225)
• Keywords: security
• Abstract: Nowadays, Internet of Things platforms are being deployed in a wide range of application domains. Some of these include use cases with security requirements, where the data generated by an IoT node is the basis for making safety-critical or liability-critical decisions at system level. The challenge is to develop a solution for data exchange while proving and verifying the authenticity of the data from end-to-end. In line with this objective, this paper proposes a novel solution with the proper protocols to provide Trust in Data, making use of two Roots of Trust that are the IOTA Distributed Ledger Technology and the Trusted Platform Module. The paper presents the design of the proposed solution and discusses the key design aspects and relevant trade-offs. The paper concludes with a Proof-of-Concept implementation and an experimental evaluation to confirm its feasibility and to assess the achievable performance.

Title: Comprehensive evaluation of Mal-API-2019 dataset by machine learning in malware detection

• Authors: Zhenglin Li, Haibei Zhu, Houze Liu, Jintong Song, Qishuo Cheng
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02232
• Pdf URL: https://arxiv.org/pdf/2403.02232
• Copy Paste: [[2403.02232]] Comprehensive evaluation of Mal-API-2019 dataset by machine learning in malware detection(https://arxiv.org/abs/2403.02232)
• Keywords: security, robust
• Abstract: This study conducts a thorough examination of malware detection using machine learning techniques, focusing on the evaluation of various classification models using the Mal-API-2019 dataset. The aim is to advance cybersecurity capabilities by identifying and mitigating threats more effectively. Both ensemble and non-ensemble machine learning methods, such as Random Forest, XGBoost, K Nearest Neighbor (KNN), and Neural Networks, are explored. Special emphasis is placed on the importance of data pre-processing techniques, particularly TF-IDF representation and Principal Component Analysis, in improving model performance. Results indicate that ensemble methods, particularly Random Forest and XGBoost, exhibit superior accuracy, precision, and recall compared to others, highlighting their effectiveness in malware detection. The paper also discusses limitations and potential future directions, emphasizing the need for continuous adaptation to address the evolving nature of malware. This research contributes to ongoing discussions in cybersecurity and provides practical insights for developing more robust malware detection systems in the digital era.

Title: Transformers Provably Learn Feature-Position Correlations in Masked Image Modeling

• Authors: Yu Huang, Zixin Wen, Yuejie Chi, Yingbin Liang
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.02233
• Pdf URL: https://arxiv.org/pdf/2403.02233
• Copy Paste: [[2403.02233]] Transformers Provably Learn Feature-Position Correlations in Masked Image Modeling(https://arxiv.org/abs/2403.02233)
• Keywords: transformer
• Abstract: Masked image modeling (MIM), which predicts randomly masked patches from unmasked ones, has emerged as a promising approach in self-supervised vision pretraining. However, the theoretical understanding of MIM is rather limited, especially with the foundational architecture of transformers. In this paper, to the best of our knowledge, we provide the first end-to-end theory of learning one-layer transformers with softmax attention in MIM self-supervised pretraining. On the conceptual side, we posit a theoretical mechanism of how transformers, pretrained with MIM, produce empirically observed local and diverse attention patterns on data distributions with spatial structures that highlight feature-position correlations. On the technical side, our end-to-end analysis of the training dynamics of softmax-based transformers accommodates both input and position embeddings simultaneously, which is developed based on a novel approach to track the interplay between the attention of feature-position and position-wise correlations.

Title: 3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors

• Authors: Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang, Liang Pan, Dahua Lin, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02234
• Pdf URL: https://arxiv.org/pdf/2403.02234
• Copy Paste: [[2403.02234]] 3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors(https://arxiv.org/abs/2403.02234)
• Keywords: diffusion, large language model
• Abstract: We present a two-stage text-to-3D generation system, namely 3DTopia, which generates high-quality general 3D assets within 5 minutes using hybrid diffusion priors. The first stage samples from a 3D diffusion prior directly learned from 3D data. Specifically, it is powered by a text-conditioned tri-plane latent diffusion model, which quickly generates coarse 3D samples for fast prototyping. The second stage utilizes 2D diffusion priors to further refine the texture of coarse 3D models from the first stage. The refinement consists of both latent and pixel space optimization for high-quality texture generation. To facilitate the training of the proposed system, we clean and caption the largest open-source 3D dataset, Objaverse, by combining the power of vision language models and large language models. Experiment results are reported qualitatively and quantitatively to show the performance of the proposed system. Our codes and models are available at https://github.com/3DTopia/3DTopia

Title: Neural Redshift: Random Networks are not Random Functions

• Authors: Damien Teney, Armand Nicolicioiu, Valentin Hartmann, Ehsan Abbasnejad
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02241
• Pdf URL: https://arxiv.org/pdf/2403.02241
• Copy Paste: [[2403.02241]] Neural Redshift: Random Networks are not Random Functions(https://arxiv.org/abs/2403.02241)
• Keywords: transformer
• Abstract: Our understanding of the generalization capabilities of neural networks (NNs) is still incomplete. Prevailing explanations are based on implicit biases of gradient descent (GD) but they cannot account for the capabilities of models from gradient-free methods nor the simplicity bias recently observed in untrained networks. This paper seeks other sources of generalization in NNs. Findings. To understand the inductive biases provided by architectures independently from GD, we examine untrained, random-weight networks. Even simple MLPs show strong inductive biases: uniform sampling in weight space yields a very biased distribution of functions in terms of complexity. But unlike common wisdom, NNs do not have an inherent "simplicity bias". This property depends on components such as ReLUs, residual connections, and layer normalizations. Alternative architectures can be built with a bias for any level of complexity. Transformers also inherit all these properties from their building blocks. Implications. We provide a fresh explanation for the success of deep learning independent from gradient-based training. It points at promising avenues for controlling the solutions implemented by trained models.

Title: PHAnToM: Personality Has An Effect on Theory-of-Mind Reasoning in Large Language Models

• Authors: Fiona Anting Tan, Gerard Christopher Yeo, Fanyou Wu, Weijie Xu, Vinija Jain, Aman Chadha, Kokil Jaidka, Yang Liu, See-Kiong Ng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02246
• Pdf URL: https://arxiv.org/pdf/2403.02246
• Copy Paste: [[2403.02246]] PHAnToM: Personality Has An Effect on Theory-of-Mind Reasoning in Large Language Models(https://arxiv.org/abs/2403.02246)
• Keywords: large language model
• Abstract: Recent advances in large language models (LLMs) demonstrate that their capabilities are comparable, or even superior, to humans in many tasks in natural language processing. Despite this progress, LLMs are still inadequate at social-cognitive reasoning, which humans are naturally good at. Drawing inspiration from psychological research on the links between certain personality traits and Theory-of-Mind (ToM) reasoning, and from prompt engineering research on the hyper-sensitivity of prompts in affecting LLMs capabilities, this study investigates how inducing personalities in LLMs using prompts affects their ToM reasoning capabilities. Our findings show that certain induced personalities can significantly affect the LLMs' reasoning capabilities in three different ToM tasks. In particular, traits from the Dark Triad have a larger variable effect on LLMs like GPT-3.5, Llama 2, and Mistral across the different ToM tasks. We find that LLMs that exhibit a higher variance across personality prompts in ToM also tends to be more controllable in personality tests: personality traits in LLMs like GPT-3.5, Llama 2 and Mistral can be controllably adjusted through our personality prompts. In today's landscape where role-play is a common strategy when using LLMs, our research highlights the need for caution, as models that adopt specific personas with personalities potentially also alter their reasoning abilities in an unexpected manner.

Title: KnowPhish: Large Language Models Meet Multimodal Knowledge Graphs for Enhancing Reference-Based Phishing Detection

• Authors: Yuexin Li, Chengyu Huang, Shumin Deng, Mei Lin Lock, Tri Cao, Nay Oo, Bryan Hooi, Hoon Wei Lim
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02253
• Pdf URL: https://arxiv.org/pdf/2403.02253
• Copy Paste: [[2403.02253]] KnowPhish: Large Language Models Meet Multimodal Knowledge Graphs for Enhancing Reference-Based Phishing Detection(https://arxiv.org/abs/2403.02253)
• Keywords: attack, robust, large language model
• Abstract: Phishing attacks have inflicted substantial losses on individuals and businesses alike, necessitating the development of robust and efficient automated phishing detection approaches. Reference-based phishing detectors (RBPDs), which compare the logos on a target webpage to a known set of logos, have emerged as the state-of-the-art approach. However, a major limitation of existing RBPDs is that they rely on a manually constructed brand knowledge base, making it infeasible to scale to a large number of brands, which results in false negative errors due to the insufficient brand coverage of the knowledge base. To address this issue, we propose an automated knowledge collection pipeline, using which we collect and release a large-scale multimodal brand knowledge base, KnowPhish, containing 20k brands with rich information about each brand. KnowPhish can be used to boost the performance of existing RBPDs in a plug-and-play manner. A second limitation of existing RBPDs is that they solely rely on the image modality, ignoring useful textual information present in the webpage HTML. To utilize this textual information, we propose a Large Language Model (LLM)-based approach to extract brand information of webpages from text. Our resulting multimodal phishing detection approach, KnowPhish Detector (KPD), can detect phishing webpages with or without logos. We evaluate KnowPhish and KPD on a manually validated dataset, and on a field study under Singapore's local context, showing substantial improvements in effectiveness and efficiency compared to state-of-the-art baselines.

Title: FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction

• Authors: Alessandro Scirè, Karim Ghonim, Roberto Navigli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02270
• Pdf URL: https://arxiv.org/pdf/2403.02270
• Copy Paste: [[2403.02270]] FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction(https://arxiv.org/abs/2403.02270)
• Keywords: extraction, interpretability, large language model
• Abstract: Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as a substantial number of automatically-generated summaries exhibit factual inconsistencies, such as hallucinations. In response to this issue, various approaches for the evaluation of consistency for summarization have emerged. Yet, these newly-introduced metrics face several limitations, including lack of interpretability, focus on short document summaries (e.g., news articles), and computational impracticality, especially for LLM-based metrics. To address these shortcomings, we propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE), a more interpretable and efficient factuality-oriented metric. FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation. Moreover, we extend our evaluation to a more challenging setting by conducting a human annotation process of long-form summarization.

Title: Physics-Informed Neural Networks with Skip Connections for Modeling and Control of Gas-Lifted Oil Wells

• Authors: Jonas Ekeland Kittelsen, Eric Aislan Antonelo, Eduardo Camponogara, Lars Struen Imsland
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02289
• Pdf URL: https://arxiv.org/pdf/2403.02289
• Copy Paste: [[2403.02289]] Physics-Informed Neural Networks with Skip Connections for Modeling and Control of Gas-Lifted Oil Wells(https://arxiv.org/abs/2403.02289)
• Keywords: interpretability
• Abstract: Neural networks, while powerful, often lack interpretability. Physics-Informed Neural Networks (PINNs) address this limitation by incorporating physics laws into the loss function, making them applicable to solving Ordinary Differential Equations (ODEs) and Partial Differential Equations (PDEs). The recently introduced PINC framework extends PINNs to control applications, allowing for open-ended long-range prediction and control of dynamic systems. In this work, we enhance PINC for modeling highly nonlinear systems such as gas-lifted oil wells. By introducing skip connections in the PINC network and refining certain terms in the ODE, we achieve more accurate gradients during training, resulting in an effective modeling process for the oil well system. Our proposed improved PINC demonstrates superior performance, reducing the validation prediction error by an average of 67% in the oil well application and significantly enhancing gradient flow through the network layers, increasing its magnitude by four orders of magnitude compared to the original PINC. Furthermore, experiments showcase the efficacy of Model Predictive Control (MPC) in regulating the bottom-hole pressure of the oil well using the improved PINC model, even in the presence of noisy measurements.

Title: A Decade of Privacy-Relevant Android App Reviews: Large Scale Trends

• Authors: Omer Akgul, Sai Teja Peddinti, Nina Taft, Michelle L. Mazurek, Hamza Harkous, Animesh Srivastava, Benoit Seguin
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.02292
• Pdf URL: https://arxiv.org/pdf/2403.02292
• Copy Paste: [[2403.02292]] A Decade of Privacy-Relevant Android App Reviews: Large Scale Trends(https://arxiv.org/abs/2403.02292)
• Keywords: privacy
• Abstract: We present an analysis of 12 million instances of privacy-relevant reviews publicly visible on the Google Play Store that span a 10 year period. By leveraging state of the art NLP techniques, we can examine what users have been writing about privacy along multiple dimensions: time, countries, app types, diverse privacy topics, and even across a spectrum of emotions. We find consistent growth of privacy-relevant reviews, and explore topics that are trending (such as Data Deletion and Data Theft), as well as those on the decline (such as privacy-relevant reviews on sensitive permissions). We find that although privacy reviews come from more than 200 countries, 33 countries provide 90% of privacy reviews. We conduct a comparison across countries by examining the distribution of privacy topics a country's users write about, and find that geographic proximity is not a reliable indicator that nearby countries have similar privacy perspectives. We uncover some countries with unique patterns and explore those herein. Surprisingly, we uncover that it is not uncommon for reviews that discuss privacy to be positive (32%); many users express pleasure about privacy features within apps or privacy-focused apps. We also uncover some unexpected behaviors, such as the use of reviews to deliver privacy disclaimers to developers. Finally, we demonstrate the value of analyzing app reviews with our approach as a complement to existing methods for understanding users' perspectives about privacy.

Title: Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation

• Authors: Maksim Kuprashevich, Grigorii Alekseenko, Irina Tolstykh
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02302
• Pdf URL: https://arxiv.org/pdf/2403.02302
• Copy Paste: [[2403.02302]] Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation(https://arxiv.org/abs/2403.02302)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have recently gained immense popularity. Powerful commercial models like ChatGPT-4V and Gemini, as well as open-source ones such as LLaVA, are essentially general-purpose models and are applied to solve a wide variety of tasks, including those in computer vision. These neural networks possess such strong general knowledge and reasoning abilities that they have proven capable of working even on tasks for which they were not specifically trained. We compared the capabilities of the most powerful MLLMs to date: ShareGPT4V, ChatGPT, LLaVA-Next in a specialized task of age and gender estimation with our state-of-the-art specialized model, MiVOLO. We also updated MiVOLO and provide details and new metrics in this article. This comparison has yielded some interesting results and insights about the strengths and weaknesses of the participating models. Furthermore, we attempted various ways to fine-tune the ShareGPT4V model for this specific task, aiming to achieve state-of-the-art results in this particular challenge. Although such a model would not be practical in production, as it is incredibly expensive compared to a specialized model like MiVOLO, it could be very useful in some tasks, like data annotation.

Title: Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures

• Authors: Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, Wenhai Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02308
• Pdf URL: https://arxiv.org/pdf/2403.02308
• Copy Paste: [[2403.02308]] Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures(https://arxiv.org/abs/2403.02308)
• Keywords: robust, transformer
• Abstract: Transformers have revolutionized computer vision and natural language processing, but their high computational complexity limits their application in high-resolution image processing and long-context analysis. This paper introduces Vision-RWKV (VRWKV), a model adapted from the RWKV model used in the NLP field with necessary modifications for vision tasks. Similar to the Vision Transformer (ViT), our model is designed to efficiently handle sparse inputs and demonstrate robust global processing capabilities, while also scaling up effectively, accommodating both large-scale parameters and extensive datasets. Its distinctive advantage lies in its reduced spatial aggregation complexity, which renders it exceptionally adept at processing high-resolution images seamlessly, eliminating the necessity for windowing operations. Our evaluations in image classification demonstrate that VRWKV matches ViT's classification performance with significantly faster speeds and lower memory usage. In dense prediction tasks, it outperforms window-based models, maintaining comparable speeds. These results highlight VRWKV's potential as a more efficient alternative for visual perception tasks. Code is released at \url{https://github.com/OpenGVLab/Vision-RWKV}.

Title: COMMIT: Certifying Robustness of Multi-Sensor Fusion Systems against Semantic Attacks

• Authors: Zijian Huang, Wenda Chu, Linyi Li, Chejian Xu, Bo Li
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02329
• Pdf URL: https://arxiv.org/pdf/2403.02329
• Copy Paste: [[2403.02329]] COMMIT: Certifying Robustness of Multi-Sensor Fusion Systems against Semantic Attacks(https://arxiv.org/abs/2403.02329)
• Keywords: defense, attack, robust
• Abstract: Multi-sensor fusion systems (MSFs) play a vital role as the perception module in modern autonomous vehicles (AVs). Therefore, ensuring their robustness against common and realistic adversarial semantic transformations, such as rotation and shifting in the physical world, is crucial for the safety of AVs. While empirical evidence suggests that MSFs exhibit improved robustness compared to single-modal models, they are still vulnerable to adversarial semantic transformations. Despite the proposal of empirical defenses, several works show that these defenses can be attacked again by new adaptive attacks. So far, there is no certified defense proposed for MSFs. In this work, we propose the first robustness certification framework COMMIT certify robustness of multi-sensor fusion systems against semantic attacks. In particular, we propose a practical anisotropic noise mechanism that leverages randomized smoothing with multi-modal data and performs a grid-based splitting method to characterize complex semantic transformations. We also propose efficient algorithms to compute the certification in terms of object detection accuracy and IoU for large-scale MSF models. Empirically, we evaluate the efficacy of COMMIT in different settings and provide a comprehensive benchmark of certified robustness for different MSF models using the CARLA simulation platform. We show that the certification for MSF models is at most 48.39% higher than that of single-modal models, which validates the advantages of MSF models. We believe our certification framework and benchmark will contribute an important step towards certifiably robust AVs in practice.

Title: RegionGPT: Towards Region Understanding Vision Language Model

• Authors: Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, Sifei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02330
• Pdf URL: https://arxiv.org/pdf/2403.02330
• Copy Paste: [[2403.02330]] RegionGPT: Towards Region Understanding Vision Language Model(https://arxiv.org/abs/2403.02330)
• Keywords: large language model
• Abstract: Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the vision encoder, and the use of coarse-grained training data that lacks detailed, region-specific captions. To address this, we introduce RegionGPT (short as RGPT), a novel framework designed for complex region-level captioning and understanding. RGPT enhances the spatial awareness of regional representation with simple yet effective modifications to existing visual encoders in VLMs. We further improve performance on tasks requiring a specific output scope by integrating task-guided instruction prompts during both training and inference phases, while maintaining the model's versatility for general-purpose tasks. Additionally, we develop an automated region caption data generation pipeline, enriching the training set with detailed region-level captions. We demonstrate that a universal RGPT model can be effectively applied and significantly enhancing performance across a range of region-level tasks, including but not limited to complex region descriptions, reasoning, object classification, and referring expressions comprehension.

Title: UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control

• Authors: Xuweiyi Chen, Tian Xia, Sihan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02332
• Pdf URL: https://arxiv.org/pdf/2403.02332
• Copy Paste: [[2403.02332]] UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control(https://arxiv.org/abs/2403.02332)
• Keywords: diffusion
• Abstract: Video Diffusion Models have been developed for video generation, usually integrating text and image conditioning to enhance control over the generated content. Despite the progress, ensuring consistency across frames remains a challenge, particularly when using text prompts as control conditions. To address this problem, we introduce UniCtrl, a novel, plug-and-play method that is universally applicable to improve the spatiotemporal consistency and motion diversity of videos generated by text-to-video models without additional training. UniCtrl ensures semantic consistency across different frames through cross-frame self-attention control, and meanwhile, enhances the motion quality and spatiotemporal consistency through motion injection and spatiotemporal synchronization. Our experimental results demonstrate UniCtrl's efficacy in enhancing various text-to-video models, confirming its effectiveness and universality.

Title: Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning

• Authors: Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, Weizhu Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02333
• Pdf URL: https://arxiv.org/pdf/2403.02333
• Copy Paste: [[2403.02333]] Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning(https://arxiv.org/abs/2403.02333)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown great potential in complex reasoning tasks, yet their performance is often hampered by the scarcity of high-quality, reasoning-focused training datasets. Addressing this challenge, we propose Key-Point-Driven Data Synthesis (KPDDS), a novel data synthesis framework that synthesizes question-answer pairs by leveraging key points and exemplar pairs from authentic data sources. KPDDS ensures the generation of novel questions with rigorous quality control and substantial scalability. As a result, we present KPMath, the most extensive synthetic dataset tailored for mathematical reasoning to date, comprising over one million question-answer pairs. Utilizing KPMath and augmenting it with additional reasoning-intensive corpora, we create the comprehensive KPMath-Plus dataset. Fine-tuning the Mistral-7B model on KPMath-Plus yields a zero-shot PASS@1 accuracy of 39.3% on the MATH test set, a performance that not only outpaces other finetuned 7B models but also exceeds that of certain 34B models. Our ablation studies further confirm the substantial enhancement in mathematical reasoning across various subtopics, marking a significant stride in LLMs' reasoning capabilities.

Title: Brand Visibility in Packaging: A Deep Learning Approach for Logo Detection, Saliency-Map Prediction, and Logo Placement Analysis

• Authors: Alireza Hosseini, Kiana Hooshanfar, Pouria Omrani, Reza Toosi, Ramin Toosi, Zahra Ebrahimian, Mohammad Ali Akhaee
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02336
• Pdf URL: https://arxiv.org/pdf/2403.02336
• Copy Paste: [[2403.02336]] Brand Visibility in Packaging: A Deep Learning Approach for Logo Detection, Saliency-Map Prediction, and Logo Placement Analysis(https://arxiv.org/abs/2403.02336)
• Keywords: robust, transformer
• Abstract: In the highly competitive area of product marketing, the visibility of brand logos on packaging plays a crucial role in shaping consumer perception, directly influencing the success of the product. This paper introduces a comprehensive framework to measure the brand logo's attention on a packaging design. The proposed method consists of three steps. The first step leverages YOLOv8 for precise logo detection across prominent datasets, FoodLogoDet-1500 and LogoDet-3K. The second step involves modeling the user's visual attention with a novel saliency prediction model tailored for the packaging context. The proposed saliency model combines the visual elements with text maps employing a transformers-based architecture to predict user attention maps. In the third step, by integrating logo detection with a saliency map generation, the framework provides a comprehensive brand attention score. The effectiveness of the proposed method is assessed module by module, ensuring a thorough evaluation of each component. Comparing logo detection and saliency map prediction with state-of-the-art models shows the superiority of the proposed methods. To investigate the robustness of the proposed brand attention score, we collected a unique dataset to examine previous psychophysical hypotheses related to brand visibility. the results show that the brand attention score is in line with all previous studies. Also, we introduced seven new hypotheses to check the impact of position, orientation, presence of person, and other visual elements on brand attention. This research marks a significant stride in the intersection of cognitive psychology, computer vision, and marketing, paving the way for advanced, consumer-centric packaging designs.