2024-04-04

Title: A Generative Deep Learning Approach for Crash Severity Modeling with Imbalanced Data

• Authors: Junlan Chen, Ziyuan Pu, Nan Zheng, Xiao Wen, Hongliang Ding, Xiucheng Guo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02187
• Pdf URL: https://arxiv.org/pdf/2404.02187
• Copy Paste: [[2404.02187]] A Generative Deep Learning Approach for Crash Severity Modeling with Imbalanced Data(https://arxiv.org/abs/2404.02187)
• Keywords: generative
• Abstract: Crash data is often greatly imbalanced, with the majority of crashes being non-fatal crashes, and only a small number being fatal crashes due to their rarity. Such data imbalance issue poses a challenge for crash severity modeling since it struggles to fit and interpret fatal crash outcomes with very limited samples. Usually, such data imbalance issues are addressed by data resampling methods, such as under-sampling and over-sampling techniques. However, most traditional and deep learning-based data resampling methods, such as synthetic minority oversampling technique (SMOTE) and generative Adversarial Networks (GAN) are designed dedicated to processing continuous variables. Though some resampling methods have improved to handle both continuous and discrete variables, they may have difficulties in dealing with the collapse issue associated with sparse discrete risk factors. Moreover, there is a lack of comprehensive studies that compare the performance of various resampling methods in crash severity modeling. To address the aforementioned issues, the current study proposes a crash data generation method based on the Conditional Tabular GAN. After data balancing, a crash severity model is employed to estimate the performance of classification and interpretation. A comparative study is conducted to assess classification accuracy and distribution consistency of the proposed generation method using a 4-year imbalanced crash dataset collected in Washington State, U.S. Additionally, Monte Carlo simulation is employed to estimate the performance of parameter and probability estimation in both two- and three-class imbalance scenarios. The results indicate that using synthetic data generated by CTGAN-RU for crash severity modeling outperforms using original data or synthetic data generated by other resampling methods.

Title: Emergent Abilities in Reduced-Scale Generative Language Models

• Authors: Sherin Muckatira, Vijeta Deshpande, Vladislav Lialin, Anna Rumshisky
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02204
• Pdf URL: https://arxiv.org/pdf/2404.02204
• Copy Paste: [[2404.02204]] Emergent Abilities in Reduced-Scale Generative Language Models(https://arxiv.org/abs/2404.02204)
• Keywords: generative, large language model
• Abstract: Large language models can solve new tasks without task-specific fine-tuning. This ability, also known as in-context learning (ICL), is considered an emergent ability and is primarily seen in large language models with billions of parameters. This study investigates if such emergent properties are strictly tied to model size or can be demonstrated by smaller models trained on reduced-scale data. To explore this, we simplify pre-training data and pre-train 36 causal language models with parameters varying from 1 million to 165 million parameters. We show that models trained on this simplified pre-training data demonstrate enhanced zero-shot capabilities across various tasks in simplified language, achieving performance comparable to that of pre-trained models six times larger on unrestricted language. This suggests that downscaling the language allows zero-shot learning capabilities to emerge in models with limited size. Additionally, we find that these smaller models pre-trained on simplified data demonstrate a power law relationship between the evaluation loss and the three scaling factors: compute, dataset size, and model size.

Title: CHOSEN: Contrastive Hypothesis Selection for Multi-View Depth Refinement

• Authors: Di Qiu, Yinda Zhang, Thabo Beeler, Vladimir Tankovich, Christian Häne, Sean Fanello, Christoph Rhemann, Sergio Orts Escolano
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02225
• Pdf URL: https://arxiv.org/pdf/2404.02225
• Copy Paste: [[2404.02225]] CHOSEN: Contrastive Hypothesis Selection for Multi-View Depth Refinement(https://arxiv.org/abs/2404.02225)
• Keywords: robust
• Abstract: We propose CHOSEN, a simple yet flexible, robust and effective multi-view depth refinement framework. It can be employed in any existing multi-view stereo pipeline, with straightforward generalization capability for different multi-view capture systems such as camera relative positioning and lenses. Given an initial depth estimation, CHOSEN iteratively re-samples and selects the best hypotheses, and automatically adapts to different metric or intrinsic scales determined by the capture system. The key to our approach is the application of contrastive learning in an appropriate solution space and a carefully designed hypothesis feature, based on which positive and negative hypotheses can be effectively distinguished. Integrated in a simple baseline multi-view stereo pipeline, CHOSEN delivers impressive quality in terms of depth and normal accuracy compared to many current deep learning based multi-view stereo pipelines.

Title: Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better

• Authors: Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B. Blaschko, Sergey Yekhanin, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02241
• Pdf URL: https://arxiv.org/pdf/2404.02241
• Copy Paste: [[2404.02241]] Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better(https://arxiv.org/abs/2404.02241)
• Keywords: diffusion, generative
• Abstract: Diffusion Models (DM) and Consistency Models (CM) are two types of popular generative models with good generation quality on various tasks. When training DM and CM, intermediate weight checkpoints are not fully utilized and only the last converged checkpoint is used. In this work, we find that high-quality model weights often lie in a basin which cannot be reached by SGD but can be obtained by proper checkpoint averaging. Based on these observations, we propose LCSC, a simple but effective and efficient method to enhance the performance of DM and CM, by combining checkpoints along the training trajectory with coefficients deduced from evolutionary search. We demonstrate the value of LCSC through two use cases: $\textbf{(a) Reducing training cost.}$ With LCSC, we only need to train DM/CM with fewer number of iterations and/or lower batch sizes to obtain comparable sample quality with the fully trained model. For example, LCSC achieves considerable training speedups for CM (23$\times$ on CIFAR-10 and 15$\times$ on ImageNet-64). $\textbf{(b) Enhancing pre-trained models.}$ Assuming full training is already done, LCSC can further improve the generation quality or speed of the final converged models. For example, LCSC achieves better performance using 1 number of function evaluation (NFE) than the base model with 2 NFE on consistency distillation, and decreases the NFE of DM from 15 to 9 while maintaining the generation quality on CIFAR-10. Our code is available at https://github.com/imagination-research/LCSC.

Title: Towards Robust 3D Pose Transfer with Adversarial Learning

• Authors: Haoyu Chen, Hao Tang, Ehsan Adeli, Guoying Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02242
• Pdf URL: https://arxiv.org/pdf/2404.02242
• Copy Paste: [[2404.02242]] Towards Robust 3D Pose Transfer with Adversarial Learning(https://arxiv.org/abs/2404.02242)
• Keywords: attack, robust
• Abstract: 3D pose transfer that aims to transfer the desired pose to a target mesh is one of the most challenging 3D generation tasks. Previous attempts rely on well-defined parametric human models or skeletal joints as driving pose sources. However, to obtain those clean pose sources, cumbersome but necessary pre-processing pipelines are inevitable, hindering implementations of the real-time applications. This work is driven by the intuition that the robustness of the model can be enhanced by introducing adversarial samples into the training, leading to a more invulnerable model to the noisy inputs, which even can be further extended to directly handling the real-world data like raw point clouds/scans without intermediate processing. Furthermore, we propose a novel 3D pose Masked Autoencoder (3D-PoseMAE), a customized MAE that effectively learns 3D extrinsic presentations (i.e., pose). 3D-PoseMAE facilitates learning from the aspect of extrinsic attributes by simultaneously generating adversarial samples that perturb the model and learning the arbitrary raw noisy poses via a multi-scale masking strategy. Both qualitative and quantitative studies show that the transferred meshes given by our network result in much better quality. Besides, we demonstrate the strong generalizability of our method on various poses, different domains, and even raw scans. Experimental results also show meaningful insights that the intermediate adversarial samples generated in the training can successfully attack the existing pose transfer models.

Title: $\texttt{LM}^\texttt{2}$: A Simple Society of Language Models Solves Complex Reasoning

• Authors: Gurusha Juneja, Subhabrata Dutta, Tanmoy Chakraborty
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02255
• Pdf URL: https://arxiv.org/pdf/2404.02255
• Copy Paste: [[2404.02255]] $\texttt{LM}^\texttt{2}$: A Simple Society of Language Models Solves Complex Reasoning(https://arxiv.org/abs/2404.02255)
• Keywords: robust, large language model
• Abstract: Despite demonstrating emergent reasoning abilities, Large Language Models (LLMS) often lose track of complex, multi-step reasoning. Existing studies show that providing guidance via decomposing the original question into multiple subproblems elicits more robustness in LLM reasoning -- a decomposer generates the subproblems, and a solver solves each of these subproblems. However, these techniques fail to accommodate coordination between the decomposer and the solver modules (either in a single model or different specialized ones) -- the decomposer does not keep track of the ability of the solver to follow the decomposed reasoning. In this paper, we propose LM2 to address these challenges. LM2 modularizes the decomposition, solution, and verification into three different language models. The decomposer module identifies the key concepts necessary to solve the problem and generates step-by-step subquestions according to the reasoning requirement. The solver model generates the solution to the subproblems that are then checked by the verifier module; depending upon the feedback from the verifier, the reasoning context is constructed using the subproblems and the solutions. These models are trained to coordinate using policy learning. Exhaustive experimentation suggests the superiority of LM2 over existing methods on in- and out-domain reasoning problems, outperforming the best baselines by $8.1\%$ on MATH, $7.71\%$ on JEEBench, and $9.7\%$ on MedQA problems (code available at https://github.com/LCS2-IIITD/Language_Model_Multiplex).

Title: Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

• Authors: David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, Adam Santoro
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02258
• Pdf URL: https://arxiv.org/pdf/2404.02258
• Copy Paste: [[2404.02258]] Mixture-of-Depths: Dynamically allocating compute in transformer-based language models(https://arxiv.org/abs/2404.02258)
• Keywords: transformer
• Abstract: Transformer-based language models spread FLOPs uniformly across input sequences. In this work we demonstrate that transformers can instead learn to dynamically allocate FLOPs (or compute) to specific positions in a sequence, optimising the allocation along the sequence for different layers across the model depth. Our method enforces a total compute budget by capping the number of tokens ($k$) that can participate in the self-attention and MLP computations at a given layer. The tokens to be processed are determined by the network using a top-$k$ routing mechanism. Since $k$ is defined a priori, this simple procedure uses a static computation graph with known tensor sizes, unlike other conditional computation techniques. Nevertheless, since the identities of the $k$ tokens are fluid, this method can expend FLOPs non-uniformly across the time and model depth dimensions. Thus, compute expenditure is entirely predictable in sum total, but dynamic and context-sensitive at the token-level. Not only do models trained in this way learn to dynamically allocate compute, they do so efficiently. These models match baseline performance for equivalent FLOPS and wall-clock times to train, but require a fraction of the FLOPs per forward pass, and can be upwards of 50\% faster to step during post-training sampling.

Title: LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages

• Authors: Nataliia Kholodna, Sahib Julka, Mohammad Khodadadi, Muhammed Nurullah Gumus, Michael Granitzer
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02261
• Pdf URL: https://arxiv.org/pdf/2404.02261
• Copy Paste: [[2404.02261]] LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages(https://arxiv.org/abs/2404.02261)
• Keywords: large language model
• Abstract: Low-resource languages face significant barriers in AI development due to limited linguistic resources and expertise for data labeling, rendering them rare and costly. The scarcity of data and the absence of preexisting tools exacerbate these challenges, especially since these languages may not be adequately represented in various NLP datasets. To address this gap, we propose leveraging the potential of LLMs in the active learning loop for data annotation. Initially, we conduct evaluations to assess inter-annotator agreement and consistency, facilitating the selection of a suitable LLM annotator. The chosen annotator is then integrated into a training loop for a classifier using an active learning paradigm, minimizing the amount of queried data required. Empirical evaluations, notably employing GPT-4-Turbo, demonstrate near-state-of-the-art performance with significantly reduced data requirements, as indicated by estimated potential cost savings of at least 42.45 times compared to human annotation. Our proposed solution shows promising potential to substantially reduce both the monetary and computational costs associated with automation in low-resource settings. By bridging the gap between low-resource languages and AI, this approach fosters broader inclusion and shows the potential to enable automation across diverse linguistic landscapes.

Title: OFMPNet: Deep End-to-End Model for Occupancy and Flow Prediction in Urban Environment

• Authors: Youshaa Murhij, Dmitry Yudin
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.02263
• Pdf URL: https://arxiv.org/pdf/2404.02263
• Copy Paste: [[2404.02263]] OFMPNet: Deep End-to-End Model for Occupancy and Flow Prediction in Urban Environment(https://arxiv.org/abs/2404.02263)
• Keywords: transformer
• Abstract: The task of motion prediction is pivotal for autonomous driving systems, providing crucial data to choose a vehicle behavior strategy within its surroundings. Existing motion prediction techniques primarily focus on predicting the future trajectory of each agent in the scene individually, utilizing its past trajectory data. In this paper, we introduce an end-to-end neural network methodology designed to predict the future behaviors of all dynamic objects in the environment. This approach leverages the occupancy map and the scene's motion flow. We are investigatin various alternatives for constructing a deep encoder-decoder model called OFMPNet. This model uses a sequence of bird's-eye-view road images, occupancy grid, and prior motion flow as input data. The encoder of the model can incorporate transformer, attention-based, or convolutional units. The decoder considers the use of both convolutional modules and recurrent blocks. Additionally, we propose a novel time-weighted motion flow loss, whose application has shown a substantial decrease in end-point error. Our approach has achieved state-of-the-art results on the Waymo Occupancy and Flow Prediction benchmark, with a Soft IoU of 52.1% and an AUC of 76.75% on Flow-Grounded Occupancy.

Title: Extracting Norms from Contracts Via ChatGPT: Opportunities and Challenges

• Authors: Amanul Haque, Munindar P. Singh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02269
• Pdf URL: https://arxiv.org/pdf/2404.02269
• Copy Paste: [[2404.02269]] Extracting Norms from Contracts Via ChatGPT: Opportunities and Challenges(https://arxiv.org/abs/2404.02269)
• Keywords: extraction
• Abstract: We investigate the effectiveness of ChatGPT in extracting norms from contracts. Norms provide a natural way to engineer multiagent systems by capturing how to govern the interactions between two or more autonomous parties. We extract norms of commitment, prohibition, authorization, and power, along with associated norm elements (the parties involved, antecedents, and consequents) from contracts. Our investigation reveals ChatGPT's effectiveness and limitations in norm extraction from contracts. ChatGPT demonstrates promising performance in norm extraction without requiring training or fine-tuning, thus obviating the need for annotated data, which is not generally available in this domain. However, we found some limitations of ChatGPT in extracting these norms that lead to incorrect norm extractions. The limitations include oversight of crucial details, hallucination, incorrect parsing of conjunctions, and empty norm elements. Enhanced norm extraction from contracts can foster the development of more transparent and trustworthy formal agent interaction specifications, thereby contributing to the improvement of multiagent systems.

Title: One Noise to Rule Them All: Multi-View Adversarial Attacks with Universal Perturbation

• Authors: Mehmet Ergezer, Phat Duong, Christian Green, Tommy Nguyen, Abdurrahman Zeybey
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02287
• Pdf URL: https://arxiv.org/pdf/2404.02287
• Copy Paste: [[2404.02287]] One Noise to Rule Them All: Multi-View Adversarial Attacks with Universal Perturbation(https://arxiv.org/abs/2404.02287)
• Keywords: attack, robust
• Abstract: This paper presents a novel universal perturbation method for generating robust multi-view adversarial examples in 3D object recognition. Unlike conventional attacks limited to single views, our approach operates on multiple 2D images, offering a practical and scalable solution for enhancing model scalability and robustness. This generalizable method bridges the gap between 2D perturbations and 3D-like attack capabilities, making it suitable for real-world applications. Existing adversarial attacks may become ineffective when images undergo transformations like changes in lighting, camera position, or natural deformations. We address this challenge by crafting a single universal noise perturbation applicable to various object views. Experiments on diverse rendered 3D objects demonstrate the effectiveness of our approach. The universal perturbation successfully identified a single adversarial noise for each given set of 3D object renders from multiple poses and viewpoints. Compared to single-view attacks, our universal attacks lower classification confidence across multiple viewing angles, especially at low noise levels. A sample implementation is made available at https://github.com/memoatwit/UniversalPerturbation.

Title: Towards a New Configurable and Practical Remote Automotive Security Testing Platform

• Authors: Sekar Kulandaivel, Wenjuan Lu, Brandon Barry, Jorge Guajardo
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2404.02291
• Pdf URL: https://arxiv.org/pdf/2404.02291
• Copy Paste: [[2404.02291]] Towards a New Configurable and Practical Remote Automotive Security Testing Platform(https://arxiv.org/abs/2404.02291)
• Keywords: security, attack
• Abstract: In the automotive security sector, the absence of a testing platform that is configurable, practical, and user-friendly presents considerable challenges. These difficulties are compounded by the intricate design of vehicle systems, the rapid evolution of attack vectors, and the absence of standardized testing methodologies. We propose a next-generation testing platform that addresses several challenges in vehicle cybersecurity testing and research domains. In this paper, we detail how the Vehicle Security Engineering Cloud (VSEC) Test platform enables easier access to test beds for efficient vehicle cybersecurity testing and advanced (e.g., penetration, fuzz) testing and how we extend such test beds to benefit automotive security research. We highlight methodology on how to use this platform for a variety of users and use cases with real implemented examples.

Title: Is Meta-training Really Necessary for Molecular Few-Shot Learning ?

• Authors: Philippe Formont, Hugo Jeannin, Pablo Piantanida, Ismail Ben Ayed
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02314
• Pdf URL: https://arxiv.org/pdf/2404.02314
• Copy Paste: [[2404.02314]] Is Meta-training Really Necessary for Molecular Few-Shot Learning ?(https://arxiv.org/abs/2404.02314)
• Keywords: robust
• Abstract: Few-shot learning has recently attracted significant interest in drug discovery, with a recent, fast-growing literature mostly involving convoluted meta-learning strategies. We revisit the more straightforward fine-tuning approach for molecular data, and propose a regularized quadratic-probe loss based on the the Mahalanobis distance. We design a dedicated block-coordinate descent optimizer, which avoid the degenerate solutions of our loss. Interestingly, our simple fine-tuning approach achieves highly competitive performances in comparison to state-of-the-art methods, while being applicable to black-box settings and removing the need for specific episodic pre-training strategies. Furthermore, we introduce a new benchmark to assess the robustness of the competing methods to domain shifts. In this setting, our fine-tuning baseline obtains consistently better results than meta-learning methods.

Title: Prompts As Programs: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimization

• Authors: Tobias Schnabel, Jennifer Neville
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02319
• Pdf URL: https://arxiv.org/pdf/2404.02319
• Copy Paste: [[2404.02319]] Prompts As Programs: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimization(https://arxiv.org/abs/2404.02319)
• Keywords: large language model
• Abstract: Large language models (LLMs) can now handle longer and more complex inputs, which facilitate the use of more elaborate prompts. However, prompts often require some tuning to improve performance for deployment. Recent work has proposed automatic prompt optimization methods, but as prompt complexity and LLM strength increase, many prompt optimization techniques are no longer sufficient and a new approach is needed to optimize {\em meta prompt programs}. To address this, we introduce SAMMO, a framework for {\em compile-time} optimizations of metaprompt programs, which represent prompts as structured objects that allows for a rich set of transformations that can be searched over during optimization. We show that SAMMO generalizes previous methods and improves the performance of complex prompts on (1) instruction tuning, (2) RAG pipeline tuning, and (3) prompt compression, across several different LLMs. We make all code available open-source at https://github.com/microsoft/sammo .

Title: Toward Informal Language Processing: Knowledge of Slang in Large Language Models

• Authors: Zhewei Sun, Qian Hu, Rahul Gupta, Richard Zemel, Yang Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02323
• Pdf URL: https://arxiv.org/pdf/2404.02323
• Copy Paste: [[2404.02323]] Toward Informal Language Processing: Knowledge of Slang in Large Language Models(https://arxiv.org/abs/2404.02323)
• Keywords: large language model
• Abstract: Recent advancement in large language models (LLMs) has offered a strong potential for natural language systems to process informal language. A representative form of informal language is slang, used commonly in daily conversations and online social media. To date, slang has not been comprehensively evaluated in LLMs due partly to the absence of a carefully designed and publicly accessible benchmark. Using movie subtitles, we construct a dataset that supports evaluation on a diverse set of tasks pertaining to automatic processing of slang. For both evaluation and finetuning, we show the effectiveness of our dataset on two core applications: 1) slang detection, and 2) identification of regional and historical sources of slang from natural sentences. We also show how our dataset can be used to probe the output distributions of LLMs for interpretive insights. We find that while LLMs such as GPT-4 achieve good performance in a zero-shot setting, smaller BERT-like models finetuned on our dataset achieve comparable performance. Furthermore, we show that our dataset enables finetuning of LLMs such as GPT-3.5 that achieve substantially better performance than strong zero-shot baselines. Our work offers a comprehensive evaluation and a high-quality benchmark on English slang based on the OpenSubtitles corpus, serving both as a publicly accessible resource and a platform for applying tools for informal language processing.

Title: Heat Death of Generative Models in Closed-Loop Learning

• Authors: Matteo Marchi, Stefano Soatto, Pratik Chaudhari, Paulo Tabuada
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02325
• Pdf URL: https://arxiv.org/pdf/2404.02325
• Copy Paste: [[2404.02325]] Heat Death of Generative Models in Closed-Loop Learning(https://arxiv.org/abs/2404.02325)
• Keywords: diffusion, generative, large language model
• Abstract: Improvement and adoption of generative machine learning models is rapidly accelerating, as exemplified by the popularity of LLMs (Large Language Models) for text, and diffusion models for image generation.As generative models become widespread, data they generate is incorporated into shared content through the public web. This opens the question of what happens when data generated by a model is fed back to the model in subsequent training campaigns. This is a question about the stability of the training process, whether the distribution of publicly accessible content, which we refer to as "knowledge", remains stable or collapses. Small scale empirical experiments reported in the literature show that this closed-loop training process is prone to degenerating. Models may start producing gibberish data, or sample from only a small subset of the desired data distribution (a phenomenon referred to as mode collapse). So far there has been only limited theoretical understanding of this process, in part due to the complexity of the deep networks underlying these generative models. The aim of this paper is to provide insights into this process (that we refer to as "generative closed-loop learning") by studying the learning dynamics of generative models that are fed back their own produced content in addition to their original training dataset. The sampling of many of these models can be controlled via a "temperature" parameter. Using dynamical systems tools, we show that, unless a sufficient amount of external data is introduced at each iteration, any non-trivial temperature leads the model to asymptotically degenerate. In fact, either the generative distribution collapses to a small set of outputs, or becomes uniform over a large set of outputs.

Title: Comparative Study of Domain Driven Terms Extraction Using Large Language Models

• Authors: Sandeep Chataut, Tuyen Do, Bichar Dip Shrestha Gurung, Shiva Aryal, Anup Khanal, Carol Lushbough, Etienne Gnimpieba
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02330
• Pdf URL: https://arxiv.org/pdf/2404.02330
• Copy Paste: [[2404.02330]] Comparative Study of Domain Driven Terms Extraction Using Large Language Models(https://arxiv.org/abs/2404.02330)
• Keywords: extraction, large language model
• Abstract: Keywords play a crucial role in bridging the gap between human understanding and machine processing of textual data. They are essential to data enrichment because they form the basis for detailed annotations that provide a more insightful and in-depth view of the underlying data. Keyword/domain driven term extraction is a pivotal task in natural language processing, facilitating information retrieval, document summarization, and content categorization. This review focuses on keyword extraction methods, emphasizing the use of three major Large Language Models(LLMs): Llama2-7B, GPT-3.5, and Falcon-7B. We employed a custom Python package to interface with these LLMs, simplifying keyword extraction. Our study, utilizing the Inspec and PubMed datasets, evaluates the performance of these models. The Jaccard similarity index was used for assessment, yielding scores of 0.64 (Inspec) and 0.21 (PubMed) for GPT-3.5, 0.40 and 0.17 for Llama2-7B, and 0.23 and 0.12 for Falcon-7B. This paper underlines the role of prompt engineering in LLMs for better keyword extraction and discusses the impact of hallucination in LLMs on result evaluation. It also sheds light on the challenges in using LLMs for keyword extraction, including model complexity, resource demands, and optimization techniques.

Title: Effective Malware Detection for Embedded Computing Systems with Limited Exposure

• Authors: Sreenitha Kasarapu, Sanket Shukla, Rakibul Hassan, Avesta Sasan, Houman Homayoun, Sai Manoj Pudukotai Dinakarrao
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02344
• Pdf URL: https://arxiv.org/pdf/2404.02344
• Copy Paste: [[2404.02344]] Effective Malware Detection for Embedded Computing Systems with Limited Exposure(https://arxiv.org/abs/2404.02344)
• Keywords: security
• Abstract: One of the pivotal security threats for the embedded computing systems is malicious software a.k.a malware. With efficiency and efficacy, Machine Learning (ML) has been widely adopted for malware detection in recent times. Despite being efficient, the existing techniques require a tremendous number of benign and malware samples for training and modeling an efficient malware detector. Furthermore, such constraints limit the detection of emerging malware samples due to the lack of sufficient malware samples required for efficient training. To address such concerns, we introduce a code-aware data generation technique that generates multiple mutated samples of the limitedly seen malware by the devices. Loss minimization ensures that the generated samples closely mimic the limitedly seen malware and mitigate the impractical samples. Such developed malware is further incorporated into the training set to formulate the model that can efficiently detect the emerging malware despite having limited exposure. The experimental results demonstrates that the proposed technique achieves an accuracy of 90% in detecting limitedly seen malware, which is approximately 3x more than the accuracy attained by state-of-the-art techniques.

Title: GaitSTR: Gait Recognition with Sequential Two-stream Refinement

• Authors: Wanrong Zheng, Haidong Zhu, Zhaoheng Zheng, Ram Nevatia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02345
• Pdf URL: https://arxiv.org/pdf/2404.02345
• Copy Paste: [[2404.02345]] GaitSTR: Gait Recognition with Sequential Two-stream Refinement(https://arxiv.org/abs/2404.02345)
• Keywords: biometric
• Abstract: Gait recognition aims to identify a person based on their walking sequences, serving as a useful biometric modality as it can be observed from long distances without requiring cooperation from the subject. In representing a person's walking sequence, silhouettes and skeletons are the two primary modalities used. Silhouette sequences lack detailed part information when overlapping occurs between different body segments and are affected by carried objects and clothing. Skeletons, comprising joints and bones connecting the joints, provide more accurate part information for different segments; however, they are sensitive to occlusions and low-quality images, causing inconsistencies in frame-wise results within a sequence. In this paper, we explore the use of a two-stream representation of skeletons for gait recognition, alongside silhouettes. By fusing the combined data of silhouettes and skeletons, we refine the two-stream skeletons, joints, and bones through self-correction in graph convolution, along with cross-modal correction with temporal consistency from silhouettes. We demonstrate that with refined skeletons, the performance of the gait recognition model can achieve further improvement on public gait recognition datasets compared with state-of-the-art methods without extra annotations.

Title: Semantic Augmentation in Images using Language

• Authors: Sahiti Yerramilli, Jayant Sravan Tamarapalli, Tanmay Girish Kulkarni, Jonathan Francis, Eric Nyberg
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02353
• Pdf URL: https://arxiv.org/pdf/2404.02353
• Copy Paste: [[2404.02353]] Semantic Augmentation in Images using Language(https://arxiv.org/abs/2404.02353)
• Keywords: diffusion
• Abstract: Deep Learning models are incredibly data-hungry and require very large labeled datasets for supervised learning. As a consequence, these models often suffer from overfitting, limiting their ability to generalize to real-world examples. Recent advancements in diffusion models have enabled the generation of photorealistic images based on textual inputs. Leveraging the substantial datasets used to train these diffusion models, we propose a technique to utilize generated images to augment existing datasets. This paper explores various strategies for effective data augmentation to improve the out-of-domain generalization capabilities of deep learning models.

Title: Two Heads are Better than One: Nested PoE for Robust Defense Against Multi-Backdoors

• Authors: Victoria Graf, Qin Liu, Muhao Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02356
• Pdf URL: https://arxiv.org/pdf/2404.02356
• Copy Paste: [[2404.02356]] Two Heads are Better than One: Nested PoE for Robust Defense Against Multi-Backdoors(https://arxiv.org/abs/2404.02356)
• Keywords: defense, attack, robust, large language model
• Abstract: Data poisoning backdoor attacks can cause undesirable behaviors in large language models (LLMs), and defending against them is of increasing importance. Existing defense mechanisms often assume that only one type of trigger is adopted by the attacker, while defending against multiple simultaneous and independent trigger types necessitates general defense frameworks and is relatively unexplored. In this paper, we propose Nested Product of Experts(NPoE) defense framework, which involves a mixture of experts (MoE) as a trigger-only ensemble within the PoE defense framework to simultaneously defend against multiple trigger types. During NPoE training, the main model is trained in an ensemble with a mixture of smaller expert models that learn the features of backdoor triggers. At inference time, only the main model is used. Experimental results on sentiment analysis, hate speech detection, and question classification tasks demonstrate that NPoE effectively defends against a variety of triggers both separately and in trigger mixtures. Due to the versatility of the MoE structure in NPoE, this framework can be further expanded to defend against other attack settings

Title: FraGNNet: A Deep Probabilistic Model for Mass Spectrum Prediction

• Authors: Adamo Young, Fei Wang, David Wishart, Bo Wang, Hannes Röst, Russ Greiner
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2404.02360
• Pdf URL: https://arxiv.org/pdf/2404.02360
• Copy Paste: [[2404.02360]] FraGNNet: A Deep Probabilistic Model for Mass Spectrum Prediction(https://arxiv.org/abs/2404.02360)
• Keywords: interpretability
• Abstract: The process of identifying a compound from its mass spectrum is a critical step in the analysis of complex mixtures. Typical solutions for the mass spectrum to compound (MS2C) problem involve matching the unknown spectrum against a library of known spectrum-molecule pairs, an approach that is limited by incomplete library coverage. Compound to mass spectrum (C2MS) models can improve retrieval rates by augmenting real libraries with predicted spectra. Unfortunately, many existing C2MS models suffer from problems with prediction resolution, scalability, or interpretability. We develop a new probabilistic method for C2MS prediction, FraGNNet, that can efficiently and accurately predict high-resolution spectra. FraGNNet uses a structured latent space to provide insight into the underlying processes that define the spectrum. Our model achieves state-of-the-art performance in terms of prediction error, and surpasses existing C2MS models as a tool for retrieval-based MS2C.

Title: Obfuscated Malware Detection: Investigating Real-world Scenarios through Memory Analysis

• Authors: S M Rakib Hasan, Aakar Dhakal
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02372
• Pdf URL: https://arxiv.org/pdf/2404.02372
• Copy Paste: [[2404.02372]] Obfuscated Malware Detection: Investigating Real-world Scenarios through Memory Analysis(https://arxiv.org/abs/2404.02372)
• Keywords: security
• Abstract: In the era of the internet and smart devices, the detection of malware has become crucial for system security. Malware authors increasingly employ obfuscation techniques to evade advanced security solutions, making it challenging to detect and eliminate threats. Obfuscated malware, adept at hiding itself, poses a significant risk to various platforms, including computers, mobile devices, and IoT devices. Conventional methods like heuristic-based or signature-based systems struggle against this type of malware, as it leaves no discernible traces on the system. In this research, we propose a simple and cost-effective obfuscated malware detection system through memory dump analysis, utilizing diverse machine-learning algorithms. The study focuses on the CIC-MalMem-2022 dataset, designed to simulate real-world scenarios and assess memory-based obfuscated malware detection. We evaluate the effectiveness of machine learning algorithms, such as decision trees, ensemble methods, and neural networks, in detecting obfuscated malware within memory dumps. Our analysis spans multiple malware categories, providing insights into algorithmic strengths and limitations. By offering a comprehensive assessment of machine learning algorithms for obfuscated malware detection through memory analysis, this paper contributes to ongoing efforts to enhance cybersecurity and fortify digital ecosystems against evolving and sophisticated malware threats. The source code is made open-access for reproducibility and future research endeavours. It can be accessed at https://bit.ly/MalMemCode.

Title: Optical Text Recognition in Nepali and Bengali: A Transformer-based Approach

• Authors: S M Rakib Hasan, Aakar Dhakal, Md Humaion Kabir Mehedi, Annajiat Alim Rasel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02375
• Pdf URL: https://arxiv.org/pdf/2404.02375
• Copy Paste: [[2404.02375]] Optical Text Recognition in Nepali and Bengali: A Transformer-based Approach(https://arxiv.org/abs/2404.02375)
• Keywords: transformer
• Abstract: Efforts on the research and development of OCR systems for Low-Resource Languages are relatively new. Low-resource languages have little training data available for training Machine Translation systems or other systems. Even though a vast amount of text has been digitized and made available on the internet the text is still in PDF and Image format, which are not instantly accessible. This paper discusses text recognition for two scripts: Bengali and Nepali; there are about 300 and 40 million Bengali and Nepali speakers respectively. In this study, using encoder-decoder transformers, a model was developed, and its efficacy was assessed using a collection of optical text images, both handwritten and printed. The results signify that the suggested technique corresponds with current approaches and achieves high precision in recognizing text in Bengali and Nepali. This study can pave the way for the advanced and accessible study of linguistics in South East Asia.

Title: CAPE: CAM as a Probabilistic Ensemble for Enhanced DNN Interpretation

• Authors: Townim Faisal Chowdhury, Kewen Liao, Vu Minh Hieu Phan, Minh-Son To, Yutong Xie, Kevin Hung, David Ross, Anton van den Hengel, Johan W. Verjans, Zhibin Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02388
• Pdf URL: https://arxiv.org/pdf/2404.02388
• Copy Paste: [[2404.02388]] CAPE: CAM as a Probabilistic Ensemble for Enhanced DNN Interpretation(https://arxiv.org/abs/2404.02388)
• Keywords: interpretability
• Abstract: Deep Neural Networks (DNNs) are widely used for visual classification tasks, but their complex computation process and black-box nature hinder decision transparency and interpretability. Class activation maps (CAMs) and recent variants provide ways to visually explain the DNN decision-making process by displaying 'attention' heatmaps of the DNNs. Nevertheless, the CAM explanation only offers relative attention information, that is, on an attention heatmap, we can interpret which image region is more or less important than the others. However, these regions cannot be meaningfully compared across classes, and the contribution of each region to the model's class prediction is not revealed. To address these challenges that ultimately lead to better DNN Interpretation, in this paper, we propose CAPE, a novel reformulation of CAM that provides a unified and probabilistically meaningful assessment of the contributions of image regions. We quantitatively and qualitatively compare CAPE with state-of-the-art CAM methods on CUB and ImageNet benchmark datasets to demonstrate enhanced interpretability. We also test on a cytology imaging dataset depicting a challenging Chronic Myelomonocytic Leukemia (CMML) diagnosis problem. Code is available at: https://github.com/AIML-MED/CAPE.

Title: On Linearizing Structured Data in Encoder-Decoder Language Models: Insights from Text-to-SQL

• Authors: Yutong Shao, Ndapa Nakashole
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02389
• Pdf URL: https://arxiv.org/pdf/2404.02389
• Copy Paste: [[2404.02389]] On Linearizing Structured Data in Encoder-Decoder Language Models: Insights from Text-to-SQL(https://arxiv.org/abs/2404.02389)
• Keywords: large language model
• Abstract: Structured data, prevalent in tables, databases, and knowledge graphs, poses a significant challenge in its representation. With the advent of large language models (LLMs), there has been a shift towards linearization-based methods, which process structured data as sequential token streams, diverging from approaches that explicitly model structure, often as a graph. Crucially, there remains a gap in our understanding of how these linearization-based methods handle structured data, which is inherently non-linear. This work investigates the linear handling of structured data in encoder-decoder language models, specifically T5. Our findings reveal the model's ability to mimic human-designed processes such as schema linking and syntax prediction, indicating a deep, meaningful learning of structure beyond simple token sequencing. We also uncover insights into the model's internal mechanisms, including the ego-centric nature of structure node encodings and the potential for model compression due to modality fusion redundancy. Overall, this work sheds light on the inner workings of linearization-based methods and could potentially provide guidance for future research.

Title: Low-resource neural machine translation with morphological modeling

• Authors: Antoine Nzeyimana
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02392
• Pdf URL: https://arxiv.org/pdf/2404.02392
• Copy Paste: [[2404.02392]] Low-resource neural machine translation with morphological modeling(https://arxiv.org/abs/2404.02392)
• Keywords: transformer
• Abstract: Morphological modeling in neural machine translation (NMT) is a promising approach to achieving open-vocabulary machine translation for morphologically-rich languages. However, existing methods such as sub-word tokenization and character-based models are limited to the surface forms of the words. In this work, we propose a framework-solution for modeling complex morphology in low-resource settings. A two-tier transformer architecture is chosen to encode morphological information at the inputs. At the target-side output, a multi-task multi-label training scheme coupled with a beam search-based decoder are found to improve machine translation performance. An attention augmentation scheme to the transformer model is proposed in a generic form to allow integration of pre-trained language models and also facilitate modeling of word order relationships between the source and target languages. Several data augmentation techniques are evaluated and shown to increase translation performance in low-resource settings. We evaluate our proposed solution on Kinyarwanda - English translation using public-domain parallel text. Our final models achieve competitive performance in relation to large multi-lingual models. We hope that our results will motivate more use of explicit morphological information and the proposed model and data augmentations in low-resource NMT.

Title: Backdoor Attack on Multilingual Machine Translation

• Authors: Jun Wang, Qiongkai Xu, Xuanli He, Benjamin I. P. Rubinstein, Trevor Cohn
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02393
• Pdf URL: https://arxiv.org/pdf/2404.02393
• Copy Paste: [[2404.02393]] Backdoor Attack on Multilingual Machine Translation(https://arxiv.org/abs/2404.02393)
• Keywords: security, attack
• Abstract: While multilingual machine translation (MNMT) systems hold substantial promise, they also have security vulnerabilities. Our research highlights that MNMT systems can be susceptible to a particularly devious style of backdoor attack, whereby an attacker injects poisoned data into a low-resource language pair to cause malicious translations in other languages, including high-resource languages. Our experimental results reveal that injecting less than 0.01% poisoned data into a low-resource language pair can achieve an average 20% attack success rate in attacking high-resource language pairs. This type of attack is of particular concern, given the larger attack surface of languages inherent to low-resource settings. Our aim is to bring attention to these vulnerabilities within MNMT systems with the hope of encouraging the community to address security concerns in machine translation, especially in the context of low-resource languages.

Title: Optimal Batch Allocation for Wireless Federated Learning

• Authors: Jaeyoung Song, Sang-Woon Jeon
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.02395
• Pdf URL: https://arxiv.org/pdf/2404.02395
• Copy Paste: [[2404.02395]] Optimal Batch Allocation for Wireless Federated Learning(https://arxiv.org/abs/2404.02395)
• Keywords: federate
• Abstract: Federated learning aims to construct a global model that fits the dataset distributed across local devices without direct access to private data, leveraging communication between a server and the local devices. In the context of a practical communication scheme, we study the completion time required to achieve a target performance. Specifically, we analyze the number of iterations required for federated learning to reach a specific optimality gap from a minimum global loss. Subsequently, we characterize the time required for each iteration under two fundamental multiple access schemes: time-division multiple access (TDMA) and random access (RA). We propose a step-wise batch allocation, demonstrated to be optimal for TDMA-based federated learning systems. Additionally, we show that the non-zero batch gap between devices provided by the proposed step-wise batch allocation significantly reduces the completion time for RA-based learning systems. Numerical evaluations validate these analytical results through real-data experiments, highlighting the remarkable potential for substantial completion time reduction.

Title: Enhancing Diffusion-based Point Cloud Generation with Smoothness Constraint

• Authors: Yukun Li, Liping Liu
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02396
• Pdf URL: https://arxiv.org/pdf/2404.02396
• Copy Paste: [[2404.02396]] Enhancing Diffusion-based Point Cloud Generation with Smoothness Constraint(https://arxiv.org/abs/2404.02396)
• Keywords: diffusion
• Abstract: Diffusion models have been popular for point cloud generation tasks. Existing works utilize the forward diffusion process to convert the original point distribution into a noise distribution and then learn the reverse diffusion process to recover the point distribution from the noise distribution. However, the reverse diffusion process can produce samples with non-smooth points on the surface because of the ignorance of the point cloud geometric properties. We propose alleviating the problem by incorporating the local smoothness constraint into the diffusion framework for point cloud generation. Experiments demonstrate the proposed model can generate realistic shapes and smoother point clouds, outperforming multiple state-of-the-art methods.

Title: Token Trails: Navigating Contextual Depths in Conversational AI with ChatLLM

• Authors: Md. Kowsher, Ritesh Panditi, Nusrat Jahan Prottasha, Prakash Bhat, Anupam Kumar Bairagi, Mohammad Shamsul Arefin
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02402
• Pdf URL: https://arxiv.org/pdf/2404.02402
• Copy Paste: [[2404.02402]] Token Trails: Navigating Contextual Depths in Conversational AI with ChatLLM(https://arxiv.org/abs/2404.02402)
• Keywords: large language model
• Abstract: Conversational modeling using Large Language Models (LLMs) requires a nuanced understanding of context to generate coherent and contextually relevant responses. In this paper, we present Token Trails, a novel approach that leverages token-type embeddings to navigate the intricate contextual nuances within conversations. Our framework utilizes token-type embeddings to distinguish between user utterances and bot responses, facilitating the generation of context-aware replies. Through comprehensive experimentation and evaluation, we demonstrate the effectiveness of Token Trails in improving conversational understanding and response generation, achieving state-of-the-art performance. Our results highlight the significance of contextual modeling in conversational AI and underscore the promising potential of Token Trails to advance the field, paving the way for more sophisticated and contextually aware chatbot interactions.

Title: Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT

• Authors: Amirhossein Abaskohi, Sara Baruni, Mostafa Masoudi, Nesa Abbasi, Mohammad Hadi Babalou, Ali Edalat, Sepehr Kamahi, Samin Mahdizadeh Sani, Nikoo Naghavian, Danial Namazifard, Pouya Sadeghi, Yadollah Yaghoobzadeh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02403
• Pdf URL: https://arxiv.org/pdf/2404.02403
• Copy Paste: [[2404.02403]] Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT(https://arxiv.org/abs/2404.02403)
• Keywords: large language model
• Abstract: This paper explores the efficacy of large language models (LLMs) for Persian. While ChatGPT and consequent LLMs have shown remarkable performance in English, their efficiency for more low-resource languages remains an open question. We present the first comprehensive benchmarking study of LLMs across diverse Persian language tasks. Our primary focus is on GPT-3.5-turbo, but we also include GPT-4 and OpenChat-3.5 to provide a more holistic evaluation. Our assessment encompasses a diverse set of tasks categorized into classic, reasoning, and knowledge-based domains. To enable a thorough comparison, we evaluate LLMs against existing task-specific fine-tuned models. Given the limited availability of Persian datasets for reasoning tasks, we introduce two new benchmarks: one based on elementary school math questions and another derived from the entrance exams for 7th and 10th grades. Our findings reveal that while LLMs, especially GPT-4, excel in tasks requiring reasoning abilities and a broad understanding of general knowledge, they often lag behind smaller pre-trained models fine-tuned specifically for particular tasks. Additionally, we observe improved performance when test sets are translated to English before inputting them into GPT-3.5. These results highlight the significant potential for enhancing LLM performance in the Persian language. This is particularly noteworthy due to the unique attributes of Persian, including its distinct alphabet and writing styles.

Title: TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression

• Authors: Ho-Joong Kim, Jung-Ho Hong, Heejon Kong, Seong-Whan Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02405
• Pdf URL: https://arxiv.org/pdf/2404.02405
• Copy Paste: [[2404.02405]] TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression(https://arxiv.org/abs/2404.02405)
• Keywords: transformer
• Abstract: In this paper, we investigate that the normalized coordinate expression is a key factor as reliance on hand-crafted components in query-based detectors for temporal action detection (TAD). Despite significant advancements towards an end-to-end framework in object detection, query-based detectors have been limited in achieving full end-to-end modeling in TAD. To address this issue, we propose \modelname{}, a full end-to-end temporal action detection transformer that integrates time-aligned coordinate expression. We reformulate coordinate expression utilizing actual timeline values, ensuring length-invariant representations from the extremely diverse video duration environment. Furthermore, our proposed adaptive query selection dynamically adjusts the number of queries based on video length, providing a suitable solution for varying video durations compared to a fixed query set. Our approach not only simplifies the TAD process by eliminating the need for hand-crafted components but also significantly improves the performance of query-based detectors. Our TE-TAD outperforms the previous query-based detectors and achieves competitive performance compared to state-of-the-art methods on popular benchmark datasets. Code is available at: https://github.com/Dotori-HJ/TE-TAD

Title: Exploring Backdoor Vulnerabilities of Chat Models

• Authors: Yunzhuo Hao, Wenkai Yang, Yankai Lin
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02406
• Pdf URL: https://arxiv.org/pdf/2404.02406
• Copy Paste: [[2404.02406]] Exploring Backdoor Vulnerabilities of Chat Models(https://arxiv.org/abs/2404.02406)
• Keywords: security, attack, large language model
• Abstract: Recent researches have shown that Large Language Models (LLMs) are susceptible to a security threat known as Backdoor Attack. The backdoored model will behave well in normal cases but exhibit malicious behaviours on inputs inserted with a specific backdoor trigger. Current backdoor studies on LLMs predominantly focus on instruction-tuned LLMs, while neglecting another realistic scenario where LLMs are fine-tuned on multi-turn conversational data to be chat models. Chat models are extensively adopted across various real-world scenarios, thus the security of chat models deserves increasing attention. Unfortunately, we point out that the flexible multi-turn interaction format instead increases the flexibility of trigger designs and amplifies the vulnerability of chat models to backdoor attacks. In this work, we reveal and achieve a novel backdoor attacking method on chat models by distributing multiple trigger scenarios across user inputs in different rounds, and making the backdoor be triggered only when all trigger scenarios have appeared in the historical conversations. Experimental results demonstrate that our method can achieve high attack success rates (e.g., over 90% ASR on Vicuna-7B) while successfully maintaining the normal capabilities of chat models on providing helpful responses to benign user requests. Also, the backdoor can not be easily removed by the downstream re-alignment, highlighting the importance of continued research and attention to the security concerns of chat models. Warning: This paper may contain toxic content.

Title: TCLC-GS: Tightly Coupled LiDAR-Camera Gaussian Splatting for Surrounding Autonomous Driving Scenes

• Authors: Cheng Zhao, Su Sun, Ruoyu Wang, Yuliang Guo, Jun-Jun Wan, Zhou Huang, Xinyu Huang, Yingjie Victor Chen, Liu Ren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02410
• Pdf URL: https://arxiv.org/pdf/2404.02410
• Copy Paste: [[2404.02410]] TCLC-GS: Tightly Coupled LiDAR-Camera Gaussian Splatting for Surrounding Autonomous Driving Scenes(https://arxiv.org/abs/2404.02410)
• Keywords: robust
• Abstract: Most 3D Gaussian Splatting (3D-GS) based methods for urban scenes initialize 3D Gaussians directly with 3D LiDAR points, which not only underutilizes LiDAR data capabilities but also overlooks the potential advantages of fusing LiDAR with camera data. In this paper, we design a novel tightly coupled LiDAR-Camera Gaussian Splatting (TCLC-GS) to fully leverage the combined strengths of both LiDAR and camera sensors, enabling rapid, high-quality 3D reconstruction and novel view RGB/depth synthesis. TCLC-GS designs a hybrid explicit (colorized 3D mesh) and implicit (hierarchical octree feature) 3D representation derived from LiDAR-camera data, to enrich the properties of 3D Gaussians for splatting. 3D Gaussian's properties are not only initialized in alignment with the 3D mesh which provides more completed 3D shape and color information, but are also endowed with broader contextual information through retrieved octree implicit features. During the Gaussian Splatting optimization process, the 3D mesh offers dense depth information as supervision, which enhances the training process by learning of a robust geometry. Comprehensive evaluations conducted on the Waymo Open Dataset and nuScenes Dataset validate our method's state-of-the-art (SOTA) performance. Utilizing a single NVIDIA RTX 3090 Ti, our method demonstrates fast training and achieves real-time RGB and depth rendering at 90 FPS in resolution of 1920x1280 (Waymo), and 120 FPS in resolution of 1600x900 (nuScenes) in urban scenarios.

Title: Revisiting subword tokenization: A case study on affixal negation in large language models

• Authors: Thinh Hung Truong, Yulia Otmakhova, Karin Verspoor, Trevor Cohn, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02421
• Pdf URL: https://arxiv.org/pdf/2404.02421
• Copy Paste: [[2404.02421]] Revisiting subword tokenization: A case study on affixal negation in large language models(https://arxiv.org/abs/2404.02421)
• Keywords: large language model
• Abstract: In this work, we measure the impact of affixal negation on modern English large language models (LLMs). In affixal negation, the negated meaning is expressed through a negative morpheme, which is potentially challenging for LLMs as their tokenizers are often not morphologically plausible. We conduct extensive experiments using LLMs with different subword tokenization methods, which lead to several insights on the interaction between tokenization performance and negation sensitivity. Despite some interesting mismatches between tokenization accuracy and negation detection performance, we show that models can, on the whole, reliably recognize the meaning of affixal negation.

Title: Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data

• Authors: Parth Patwa, Simone Filice, Zhiyu Chen, Giuseppe Castellucci, Oleg Rokhlenko, Shervin Malmasi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02422
• Pdf URL: https://arxiv.org/pdf/2404.02422
• Copy Paste: [[2404.02422]] Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data(https://arxiv.org/abs/2404.02422)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) operating in 0-shot or few-shot settings achieve competitive results in Text Classification tasks. In-Context Learning (ICL) typically achieves better accuracy than the 0-shot setting, but it pays in terms of efficiency, due to the longer input prompt. In this paper, we propose a strategy to make LLMs as efficient as 0-shot text classifiers, while getting comparable or better accuracy than ICL. Our solution targets the low resource setting, i.e., when only 4 examples per class are available. Using a single LLM and few-shot real data we perform a sequence of generation, filtering and Parameter-Efficient Fine-Tuning steps to create a robust and efficient classifier. Experimental results show that our approach leads to competitive results on multiple text classification datasets.

Title: Novel_Authentication_Protocols_Tailored_for_Ambient_IoT_Devices_in_3GPP_5G_Networks

• Authors: Xiongpeng Ren, Jin Cao, Hui Li, Yinghui Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.02425
• Pdf URL: https://arxiv.org/pdf/2404.02425
• Copy Paste: [[2404.02425]] Novel_Authentication_Protocols_Tailored_for_Ambient_IoT_Devices_in_3GPP_5G_Networks(https://arxiv.org/abs/2404.02425)
• Keywords: secure, security
• Abstract: AIoT devices have attracted significant attention within the 3GPP organization. These devices, distinguished from conventional IoT devices, do not rely on additional batteries or have extremely small battery capacities, offering features such as low cost, easy deployment, and maintenance-free operation. Authentication and secure transmission are fundamental security requirements for AIoT devices. However, existing standard security mechanisms are not specifically designed for AIoT devices due to their complex key hierarchies and multi-round interactions, making them unsuitable. Besides, AIoT devices would have more various communication topologies. Therefore, we propose dedicated ultra-lightweight access authentication protocols based on various technologies and algorithms to serve as a forward-looking reference for future research and standardization. Analysis and simulation experiments using chips that closely resemble real AIoT devices, demonstrate that the existing standard protocols are indeed not suitable for such devices, and our protocols outperform existing standard protocols in terms of computational time and energy consumption. After the successful execution of proposed protocols, they can achieve secure transmission of application data, striking a balance between performance and security.

Title: Designing a Photonic Physically Unclonable Function Having Resilience to Machine Learning Attacks

• Authors: Elena R. Henderson, Jessie M. Henderson, Hiva Shahoei, William V. Oxford, Eric C. Larson, Duncan L. MacFarlane, Mitchell A. Thornton
• Subjects: cs.CR, physics.optics
• Abstract URL: https://arxiv.org/abs/2404.02440
• Pdf URL: https://arxiv.org/pdf/2404.02440
• Copy Paste: [[2404.02440]] Designing a Photonic Physically Unclonable Function Having Resilience to Machine Learning Attacks(https://arxiv.org/abs/2404.02440)
• Keywords: security, attack, generative
• Abstract: Physically unclonable functions (PUFs) are designed to act as device 'fingerprints.' Given an input challenge, the PUF circuit should produce an unpredictable response for use in situations such as root-of-trust applications and other hardware-level cybersecurity applications. PUFs are typically subcircuits present within integrated circuits (ICs), and while conventional IC PUFs are well-understood, several implementations have proven vulnerable to malicious exploits, including those perpetrated by machine learning (ML)-based attacks. Such attacks can be difficult to prevent because they are often designed to work even when relatively few challenge-response pairs are known in advance. Hence the need for both more resilient PUF designs and analysis of ML-attack susceptibility. Previous work has developed a PUF for photonic integrated circuits (PICs). A PIC PUF not only produces unpredictable responses given manufacturing-introduced tolerances, but is also less prone to electromagnetic radiation eavesdropping attacks than a purely electronic IC PUF. In this work, we analyze the resilience of the proposed photonic PUF when subjected to ML-based attacks. Specifically, we describe a computational PUF model for producing the large datasets required for training ML attacks; we analyze the quality of the model; and we discuss the modeled PUF's susceptibility to ML-based attacks. We find that the modeled PUF generates distributions that resemble uniform white noise, explaining the exhibited resilience to neural-network-based attacks designed to exploit latent relationships between challenges and responses. Preliminary analysis suggests that the PUF exhibits similar resilience to generative adversarial networks, and continued development will show whether more-sophisticated ML approaches better compromise the PUF and -- if so -- how design modifications might improve resilience.

Title: Masked Completion via Structured Diffusion with White-Box Transformers

• Authors: Druv Pai, Ziyang Wu, Sam Buchanan, Yaodong Yu, Yi Ma
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.02446
• Pdf URL: https://arxiv.org/pdf/2404.02446
• Copy Paste: [[2404.02446]] Masked Completion via Structured Diffusion with White-Box Transformers(https://arxiv.org/abs/2404.02446)
• Keywords: diffusion, transformer
• Abstract: Modern learning frameworks often train deep neural networks with massive amounts of unlabeled data to learn representations by solving simple pretext tasks, then use the representations as foundations for downstream tasks. These networks are empirically designed; as such, they are usually not interpretable, their representations are not structured, and their designs are potentially redundant. White-box deep networks, in which each layer explicitly identifies and transforms structures in the data, present a promising alternative. However, existing white-box architectures have only been shown to work at scale in supervised settings with labeled data, such as classification. In this work, we provide the first instantiation of the white-box design paradigm that can be applied to large-scale unsupervised representation learning. We do this by exploiting a fundamental connection between diffusion, compression, and (masked) completion, deriving a deep transformer-like masked autoencoder architecture, called CRATE-MAE, in which the role of each layer is mathematically fully interpretable: they transform the data distribution to and from a structured representation. Extensive empirical evaluations confirm our analytical insights. CRATE-MAE demonstrates highly promising performance on large-scale imagery datasets while using only ~30% of the parameters compared to the standard masked autoencoder with the same model configuration. The representations learned by CRATE-MAE have explicit structure and also contain semantic meaning. Code is available at https://github.com/Ma-Lab-Berkeley/CRATE .

Title: A Novel Approach to Breast Cancer Histopathological Image Classification Using Cross-Colour Space Feature Fusion and Quantum-Classical Stack Ensemble Method

• Authors: Sambit Mallick, Snigdha Paul, Anindya Sen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02447
• Pdf URL: https://arxiv.org/pdf/2404.02447
• Copy Paste: [[2404.02447]] A Novel Approach to Breast Cancer Histopathological Image Classification Using Cross-Colour Space Feature Fusion and Quantum-Classical Stack Ensemble Method(https://arxiv.org/abs/2404.02447)
• Keywords: robust, extraction
• Abstract: Breast cancer classification stands as a pivotal pillar in ensuring timely diagnosis and effective treatment. This study with histopathological images underscores the profound significance of harnessing the synergistic capabilities of colour space ensembling and quantum-classical stacking to elevate the precision of breast cancer classification. By delving into the distinct colour spaces of RGB, HSV and CIE L*u*v, the authors initiated a comprehensive investigation guided by advanced methodologies. Employing the DenseNet121 architecture for feature extraction the authors have capitalized on the robustness of Random Forest, SVM, QSVC, and VQC classifiers. This research encompasses a unique feature fusion technique within the colour space ensemble. This approach not only deepens our comprehension of breast cancer classification but also marks a milestone in personalized medical assessment. The amalgamation of quantum and classical classifiers through stacking emerges as a potent catalyst, effectively mitigating the inherent constraints of individual classifiers, paving a robust path towards more dependable and refined breast cancer identification. Through rigorous experimentation and meticulous analysis, fusion of colour spaces like RGB with HSV and RGB with CIE L*u*v, presents an classification accuracy, nearing the value of unity. This underscores the transformative potential of our approach, where the fusion of diverse colour spaces and the synergy of quantum and classical realms converge to establish a new horizon in medical diagnostics. Thus the implications of this research extend across medical disciplines, offering promising avenues for advancing diagnostic accuracy and treatment efficacy.

Title: Task Agnostic Architecture for Algorithm Induction via Implicit Composition

• Authors: Sahil J. Sindhi, Ignas Budvytis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02450
• Pdf URL: https://arxiv.org/pdf/2404.02450
• Copy Paste: [[2404.02450]] Task Agnostic Architecture for Algorithm Induction via Implicit Composition(https://arxiv.org/abs/2404.02450)
• Keywords: transformer, generative, large language model
• Abstract: Different fields in applied machine learning such as computer vision, speech or natural language processing have been building domain-specialised solutions. Currently, we are witnessing an opposing trend towards developing more generalist architectures, driven by Large Language Models and multi-modal foundational models. These architectures are designed to tackle a variety of tasks, including those previously unseen and using inputs across multiple modalities. Taking this trend of generalization to the extreme suggests the possibility of a single deep network architecture capable of solving all tasks. This position paper aims to explore developing such a unified architecture and proposes a theoretical framework of how it could be constructed. Our proposal is based on the following assumptions. Firstly, tasks are solved by following a sequence of instructions, typically implemented in code for conventional computing hardware, which inherently operates sequentially. Second, recent Generative AI, especially Transformer-based models, demonstrate potential as an architecture capable of constructing algorithms for a wide range of domains. For example, GPT-4 shows exceptional capability at in-context learning of novel tasks which is hard to explain in any other way than the ability to compose novel solutions from fragments on previously learnt algorithms. Third, the observation that the main missing component in developing a truly generalised network is an efficient approach for self-consistent input of previously learnt sub-steps of an algorithm and their (implicit) composition during the network's internal forward pass. Our exploration delves into current capabilities and limitations of Transformer-based and other methods in efficient and correct algorithm composition and proposes a Transformer-like architecture as well as a discrete learning framework to overcome these limitations.

Title: PhonologyBench: Evaluating Phonological Skills of Large Language Models

• Authors: Ashima Suvarna, Harshita Khandelwal, Nanyun Peng
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.02456
• Pdf URL: https://arxiv.org/pdf/2404.02456
• Copy Paste: [[2404.02456]] PhonologyBench: Evaluating Phonological Skills of Large Language Models(https://arxiv.org/abs/2404.02456)
• Keywords: large language model
• Abstract: Phonology, the study of speech's structure and pronunciation rules, is a critical yet often overlooked component in Large Language Model (LLM) research. LLMs are widely used in various downstream applications that leverage phonology such as educational tools and poetry generation. Moreover, LLMs can potentially learn imperfect associations between orthographic and phonological forms from the training data. Thus, it is imperative to benchmark the phonological skills of LLMs. To this end, we present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs in English: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. Despite having no access to speech data, LLMs showcased notable performance on the PhonologyBench tasks. However, we observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans. Our findings underscore the importance of studying LLM performance on phonological tasks that inadvertently impact real-world applications. Furthermore, we encourage researchers to choose LLMs that perform well on the phonological task that is closely related to the downstream application since we find that no single model consistently outperforms the others on all the tasks.

Title: RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentation

• Authors: Xianping Ma, Xiaokang Zhang, Man-On Pun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02457
• Pdf URL: https://arxiv.org/pdf/2404.02457
• Copy Paste: [[2404.02457]] RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentation(https://arxiv.org/abs/2404.02457)
• Keywords: transformer, segmentation
• Abstract: Semantic segmentation of remote sensing images is a fundamental task in geoscience research. However, there are some significant shortcomings for the widely used convolutional neural networks (CNNs) and Transformers. The former is limited by its insufficient long-range modeling capabilities, while the latter is hampered by its computational complexity. Recently, a novel visual state space (VSS) model represented by Mamba has emerged, capable of modeling long-range relationships with linear computability. In this work, we propose a novel dual-branch network named remote sensing images semantic segmentation Mamba (RS3Mamba) to incorporate this innovative technology into remote sensing tasks. Specifically, RS3Mamba utilizes VSS blocks to construct an auxiliary branch, providing additional global information to convolution-based main branch. Moreover, considering the distinct characteristics of the two branches, we introduce a collaborative completion module (CCM) to enhance and fuse features from the dual-encoder. Experimental results on two widely used datasets, ISPRS Vaihingen and LoveDA Urban, demonstrate the effectiveness and potential of the proposed RS3Mamba. To the best of our knowledge, this is the first vision Mamba specifically designed for remote sensing images semantic segmentation. The source code will be made available at https://github.com/sstary/SSRS.

Title: On the Efficiency and Robustness of Vibration-based Foundation Models for IoT Sensing: A Case Study

• Authors: Tomoyoshi Kimura, Jinyang Li, Tianshi Wang, Denizhan Kara, Yizhuo Chen, Yigong Hu, Ruijie Wang, Maggie Wigness, Shengzhong Liu, Mani Srivastava, Suhas Diggavi, Tarek Abdelzaher
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.02461
• Pdf URL: https://arxiv.org/pdf/2404.02461
• Copy Paste: [[2404.02461]] On the Efficiency and Robustness of Vibration-based Foundation Models for IoT Sensing: A Case Study(https://arxiv.org/abs/2404.02461)
• Keywords: robust
• Abstract: This paper demonstrates the potential of vibration-based Foundation Models (FMs), pre-trained with unlabeled sensing data, to improve the robustness of run-time inference in (a class of) IoT applications. A case study is presented featuring a vehicle classification application using acoustic and seismic sensing. The work is motivated by the success of foundation models in the areas of natural language processing and computer vision, leading to generalizations of the FM concept to other domains as well, where significant amounts of unlabeled data exist that can be used for self-supervised pre-training. One such domain is IoT applications. Foundation models for selected sensing modalities in the IoT domain can be pre-trained in an environment-agnostic fashion using available unlabeled sensor data and then fine-tuned to the deployment at hand using a small amount of labeled data. The paper shows that the pre-training/fine-tuning approach improves the robustness of downstream inference and facilitates adaptation to different environmental conditions. More specifically, we present a case study in a real-world setting to evaluate a simple (vibration-based) FM-like model, called FOCAL, demonstrating its superior robustness and adaptation, compared to conventional supervised deep neural networks (DNNs). We also demonstrate its superior convergence over supervised solutions. Our findings highlight the advantages of vibration-based FMs (and FM-inspired selfsupervised models in general) in terms of inference robustness, runtime efficiency, and model adaptation (via fine-tuning) in resource-limited IoT settings.

Title: A Unified Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability

• Authors: Jie Zhu, Jirong Zha, Ding Li, Leye Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02462
• Pdf URL: https://arxiv.org/pdf/2404.02462
• Copy Paste: [[2404.02462]] A Unified Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability(https://arxiv.org/abs/2404.02462)
• Keywords: privacy, defense, attack, membership infer
• Abstract: Self-supervised learning shows promise in harnessing extensive unlabeled data, but it also confronts significant privacy concerns, especially in vision. In this paper, we aim to perform membership inference on visual self-supervised models in a more realistic setting: self-supervised training method and details are unknown for an adversary when attacking as he usually faces a black-box system in practice. In this setting, considering that self-supervised model could be trained by completely different self-supervised paradigms, e.g., masked image modeling and contrastive learning, with complex training details, we propose a unified membership inference method called PartCrop. It is motivated by the shared part-aware capability among models and stronger part response on the training data. Specifically, PartCrop crops parts of objects in an image to query responses with the image in representation space. We conduct extensive attacks on self-supervised models with different training protocols and structures using three widely used image datasets. The results verify the effectiveness and generalization of PartCrop. Moreover, to defend against PartCrop, we evaluate two common approaches, i.e., early stop and differential privacy, and propose a tailored method called shrinking crop scale range. The defense experiments indicate that all of them are effective. Our code is available at https://github.com/JiePKU/PartCrop

Title: Prompting for Numerical Sequences: A Case Study on Market Comment Generation

• Authors: Masayuki Kawarada, Tatsuya Ishigaki, Hiroya Takamura
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2404.02466
• Pdf URL: https://arxiv.org/pdf/2404.02466
• Copy Paste: [[2404.02466]] Prompting for Numerical Sequences: A Case Study on Market Comment Generation(https://arxiv.org/abs/2404.02466)
• Keywords: large language model
• Abstract: Large language models (LLMs) have been applied to a wide range of data-to-text generation tasks, including tables, graphs, and time-series numerical data-to-text settings. While research on generating prompts for structured data such as tables and graphs is gaining momentum, in-depth investigations into prompting for time-series numerical data are lacking. Therefore, this study explores various input representations, including sequences of tokens and structured formats such as HTML, LaTeX, and Python-style codes. In our experiments, we focus on the task of Market Comment Generation, which involves taking a numerical sequence of stock prices as input and generating a corresponding market comment. Contrary to our expectations, the results show that prompts resembling programming languages yield better outcomes, whereas those similar to natural languages and longer formats, such as HTML and LaTeX, are less effective. Our findings offer insights into creating effective prompts for tasks that generate text from numerical sequences.

Title: FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning

• Authors: Rishub Tamirisa, Chulin Xie, Wenxuan Bao, Andy Zhou, Ron Arel, Aviv Shamsian
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02478
• Pdf URL: https://arxiv.org/pdf/2404.02478
• Copy Paste: [[2404.02478]] FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning(https://arxiv.org/abs/2404.02478)
• Keywords: robust, federate
• Abstract: Standard federated learning approaches suffer when client data distributions have sufficient heterogeneity. Recent methods addressed the client data heterogeneity issue via personalized federated learning (PFL) - a class of FL algorithms aiming to personalize learned global knowledge to better suit the clients' local data distributions. Existing PFL methods usually decouple global updates in deep neural networks by performing personalization on particular layers (i.e. classifier heads) and global aggregation for the rest of the network. However, preselecting network layers for personalization may result in suboptimal storage of global knowledge. In this work, we propose FedSelect, a novel PFL algorithm inspired by the iterative subnetwork discovery procedure used for the Lottery Ticket Hypothesis. FedSelect incrementally expands subnetworks to personalize client parameters, concurrently conducting global aggregations on the remaining parameters. This approach enables the personalization of both client parameters and subnetwork structure during the training process. Finally, we show that FedSelect outperforms recent state-of-the-art PFL algorithms under challenging client data heterogeneity settings and demonstrates robustness to various real-world distributional shifts. Our code is available at https://github.com/lapisrocks/fedselect.

Title: Measuring Social Norms of Large Language Models

• Authors: Ye Yuan, Kexin Tang, Jianhao Shen, Ming Zhang, Chenguang Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02491
• Pdf URL: https://arxiv.org/pdf/2404.02491
• Copy Paste: [[2404.02491]] Measuring Social Norms of Large Language Models(https://arxiv.org/abs/2404.02491)
• Keywords: large language model
• Abstract: We present a new challenge to examine whether large language models understand social norms. In contrast to existing datasets, our dataset requires a fundamental understanding of social norms to solve. Our dataset features the largest set of social norm skills, consisting of 402 skills and 12,383 questions covering a wide set of social norms ranging from opinions and arguments to culture and laws. We design our dataset according to the K-12 curriculum. This enables the direct comparison of the social understanding of large language models to humans, more specifically, elementary students. While prior work generates nearly random accuracy on our benchmark, recent large language models such as GPT3.5-Turbo and LLaMA2-Chat are able to improve the performance significantly, only slightly below human performance. We then propose a multi-agent framework based on large language models to improve the models' ability to understand social norms. This method further improves large language models to be on par with humans. Given the increasing adoption of large language models in real-world applications, our finding is particularly important and presents a unique direction for future improvements.

Title: VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments

• Authors: Bufang Yang, Lixing He, Kaiwei Liu, Zhenyu Yan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02508
• Pdf URL: https://arxiv.org/pdf/2404.02508
• Copy Paste: [[2404.02508]] VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments(https://arxiv.org/abs/2404.02508)
• Keywords: large language model
• Abstract: Individuals with visual impairments, encompassing both partial and total difficulties in visual perception, are referred to as visually impaired (VI) people. An estimated 2.2 billion individuals worldwide are affected by visual impairments. Recent advancements in multi-modal large language models (MLLMs) have showcased their extraordinary capabilities across various domains. It is desirable to help VI individuals with MLLMs' great capabilities of visual understanding and reasoning. However, it is challenging for VI people to use MLLMs due to the difficulties in capturing the desirable images to fulfill their daily requests. For example, the target object is not fully or partially placed in the image. This paper explores how to leverage MLLMs for VI individuals to provide visual-question answers. VIAssist can identify undesired images and provide detailed actions. Finally, VIAssist can provide reliable answers to users' queries based on the images. Our results show that VIAssist provides +0.21 and +0.31 higher BERTScore and ROUGE scores than the baseline, respectively.

Title: An Interpretable Client Decision Tree Aggregation process for Federated Learning

• Authors: Alberto Argente-Garrido, Cristina Zuheros, M. Victoria Luzón, Francisco Herrera
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02510
• Pdf URL: https://arxiv.org/pdf/2404.02510
• Copy Paste: [[2404.02510]] An Interpretable Client Decision Tree Aggregation process for Federated Learning(https://arxiv.org/abs/2404.02510)
• Keywords: privacy, robust, federate, interpretability, explainability
• Abstract: Trustworthy Artificial Intelligence solutions are essential in today's data-driven applications, prioritizing principles such as robustness, safety, transparency, explainability, and privacy among others. This has led to the emergence of Federated Learning as a solution for privacy and distributed machine learning. While decision trees, as self-explanatory models, are ideal for collaborative model training across multiple devices in resource-constrained environments such as federated learning environments for injecting interpretability in these models. Decision tree structure makes the aggregation in a federated learning environment not trivial. They require techniques that can merge their decision paths without introducing bias or overfitting while keeping the aggregated decision trees robust and generalizable. In this paper, we propose an Interpretable Client Decision Tree Aggregation process for Federated Learning scenarios that keeps the interpretability and the precision of the base decision trees used for the aggregation. This model is based on aggregating multiple decision paths of the decision trees and can be used on different decision tree types, such as ID3 and CART. We carry out the experiments within four datasets, and the analysis shows that the tree built with the model improves the local models, and outperforms the state-of-the-art.

Title: Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Languages

• Authors: Vandan Mujadia, Pruthwik Mishra, Arafat Ahsan, Dipti Misra Sharma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02512
• Pdf URL: https://arxiv.org/pdf/2404.02512
• Copy Paste: [[2404.02512]] Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Languages(https://arxiv.org/abs/2404.02512)
• Keywords: large language model
• Abstract: With the primary focus on evaluating the effectiveness of large language models for automatic reference-less translation assessment, this work presents our experiments on mimicking human direct assessment to evaluate the quality of translations in English and Indian languages. We constructed a translation evaluation task where we performed zero-shot learning, in-context example-driven learning, and fine-tuning of large language models to provide a score out of 100, where 100 represents a perfect translation and 1 represents a poor translation. We compared the performance of our trained systems with existing methods such as COMET, BERT-Scorer, and LABSE, and found that the LLM-based evaluator (LLaMA-2-13B) achieves a comparable or higher overall correlation with human judgments for the considered Indian language pairs.

Title: HENet: Hybrid Encoding for End-to-end Multi-task 3D Perception from Multi-view Cameras

• Authors: Zhongyu Xia, ZhiWei Lin, Xinhao Wang, Yongtao Wang, Yun Xing, Shengxiang Qi, Nan Dong, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02517
• Pdf URL: https://arxiv.org/pdf/2404.02517
• Copy Paste: [[2404.02517]] HENet: Hybrid Encoding for End-to-end Multi-task 3D Perception from Multi-view Cameras(https://arxiv.org/abs/2404.02517)
• Keywords: segmentation
• Abstract: Three-dimensional perception from multi-view cameras is a crucial component in autonomous driving systems, which involves multiple tasks like 3D object detection and bird's-eye-view (BEV) semantic segmentation. To improve perception precision, large image encoders, high-resolution images, and long-term temporal inputs have been adopted in recent 3D perception models, bringing remarkable performance gains. However, these techniques are often incompatible in training and inference scenarios due to computational resource constraints. Besides, modern autonomous driving systems prefer to adopt an end-to-end framework for multi-task 3D perception, which can simplify the overall system architecture and reduce the implementation complexity. However, conflict between tasks often arises when optimizing multiple tasks jointly within an end-to-end 3D perception model. To alleviate these issues, we present an end-to-end framework named HENet for multi-task 3D perception in this paper. Specifically, we propose a hybrid image encoding network, using a large image encoder for short-term frames and a small image encoder for long-term temporal frames. Then, we introduce a temporal feature integration module based on the attention mechanism to fuse the features of different frames extracted by the two aforementioned hybrid image encoders. Finally, according to the characteristics of each perception task, we utilize BEV features of different grid sizes, independent BEV encoders, and task decoders for different tasks. Experimental results show that HENet achieves state-of-the-art end-to-end multi-task 3D perception results on the nuScenes benchmark, including 3D object detection and BEV semantic segmentation. The source code and models will be released at https://github.com/VDIGPKU/HENet.

Title: Differentially Private Verification of Survey-Weighted Estimates

• Authors: Tong Lin, Jerome P. Reiter
• Subjects: cs.CR, stat.ME
• Abstract URL: https://arxiv.org/abs/2404.02519
• Pdf URL: https://arxiv.org/pdf/2404.02519
• Copy Paste: [[2404.02519]] Differentially Private Verification of Survey-Weighted Estimates(https://arxiv.org/abs/2404.02519)
• Keywords: privacy
• Abstract: Several official statistics agencies release synthetic data as public use microdata files. In practice, synthetic data do not admit accurate results for every analysis. Thus, it is beneficial for agencies to provide users with feedback on the quality of their analyses of the synthetic data. One approach is to couple synthetic data with a verification server that provides users with measures of the similarity of estimates computed with the synthetic and underlying confidential data. However, such measures leak information about the confidential records, so that agencies may wish to apply disclosure control methods to the released verification measures. We present a verification measure that satisfies differential privacy and can be used when the underlying confidential are collected with a complex survey design. We illustrate the verification measure using repeated sampling simulations where the confidential data are sampled with a probability proportional to size design, and the analyst estimates a population total or mean with the synthetic data. The simulations suggest that the verification measures can provide useful information about the quality of synthetic data inferences.

Title: Text-driven Affordance Learning from Egocentric Vision

• Authors: Tomoya Yoshida, Shuhei Kurita, Taichi Nishimura, Shinsuke Mori
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02523
• Pdf URL: https://arxiv.org/pdf/2404.02523
• Copy Paste: [[2404.02523]] Text-driven Affordance Learning from Egocentric Vision(https://arxiv.org/abs/2404.02523)
• Keywords: robust
• Abstract: Visual affordance learning is a key component for robots to understand how to interact with objects. Conventional approaches in this field rely on pre-defined objects and actions, falling short of capturing diverse interactions in realworld scenarios. The key idea of our approach is employing textual instruction, targeting various affordances for a wide range of objects. This approach covers both hand-object and tool-object interactions. We introduce text-driven affordance learning, aiming to learn contact points and manipulation trajectories from an egocentric view following textual instruction. In our task, contact points are represented as heatmaps, and the manipulation trajectory as sequences of coordinates that incorporate both linear and rotational movements for various manipulations. However, when we gather data for this task, manual annotations of these diverse interactions are costly. To this end, we propose a pseudo dataset creation pipeline and build a large pseudo-training dataset: TextAFF80K, consisting of over 80K instances of the contact points, trajectories, images, and text tuples. We extend existing referring expression comprehension models for our task, and experimental results show that our approach robustly handles multiple affordances, serving as a new standard for affordance learning in real-world scenarios.

Title: Severity Controlled Text-to-Image Generative Model Bias Manipulation

• Authors: Jordan Vice, Naveed Akhtar, Richard Hartley, Ajmal Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02530
• Pdf URL: https://arxiv.org/pdf/2404.02530
• Copy Paste: [[2404.02530]] Severity Controlled Text-to-Image Generative Model Bias Manipulation(https://arxiv.org/abs/2404.02530)
• Keywords: attack, generative
• Abstract: Text-to-image (T2I) generative models are gaining wide popularity, especially in public domains. However, their intrinsic bias and potential malicious manipulations remain under-explored. Charting the susceptibility of T2I models to such manipulation, we first expose the new possibility of a dynamic and computationally efficient exploitation of model bias by targeting the embedded language models. By leveraging mathematical foundations of vector algebra, our technique enables a scalable and convenient control over the severity of output manipulation through model bias. As a by-product, this control also allows a form of precise prompt engineering to generate images which are generally implausible with regular text prompts. We also demonstrate a constructive application of our manipulation for balancing the frequency of generated classes - as in model debiasing. Our technique does not require training and is also framed as a backdoor attack with severity control using semantically-null text triggers in the prompts. With extensive analysis, we present interesting qualitative and quantitative results to expose potential manipulation possibilities for T2I models. Key-words: Text-to-Image Models, Generative Models, Backdoor Attacks, Prompt Engineering, Bias

Title: CSEPrompts: A Benchmark of Introductory Computer Science Prompts

• Authors: Nishat Raihan, Dhiman Goswami, Sadiya Sayara Chowdhury Puspo, Christian Newman, Tharindu Ranasinghe, Marcos Zampieri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02540
• Pdf URL: https://arxiv.org/pdf/2404.02540
• Copy Paste: [[2404.02540]] CSEPrompts: A Benchmark of Introductory Computer Science Prompts(https://arxiv.org/abs/2404.02540)
• Keywords: large language model
• Abstract: Recent advances in AI, machine learning, and NLP have led to the development of a new generation of Large Language Models (LLMs) that are trained on massive amounts of data and often have trillions of parameters. Commercial applications (e.g., ChatGPT) have made this technology available to the general public, thus making it possible to use LLMs to produce high-quality texts for academic and professional purposes. Schools and universities are aware of the increasing use of AI-generated content by students and they have been researching the impact of this new technology and its potential misuse. Educational programs in Computer Science (CS) and related fields are particularly affected because LLMs are also capable of generating programming code in various programming languages. To help understand the potential impact of publicly available LLMs in CS education, we introduce CSEPrompts, a framework with hundreds of programming exercise prompts and multiple-choice questions retrieved from introductory CS and programming courses. We also provide experimental results on CSEPrompts to evaluate the performance of several LLMs with respect to generating Python code and answering basic computer science and programming questions.

Title: Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning

• Authors: Yi Shen, Hanyan Huang, Shan Xie
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02545
• Pdf URL: https://arxiv.org/pdf/2404.02545
• Copy Paste: [[2404.02545]] Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning(https://arxiv.org/abs/2404.02545)
• Keywords: security
• Abstract: Offline reinforcement learning learns from a static dataset without interacting with the environment, which ensures security and thus owns a good prospect of application. However, directly applying naive reinforcement learning methods usually fails in an offline environment due to function approximation errors caused by out-of-distribution(OOD) actions. To solve this problem, existing algorithms mainly penalize the Q-value of OOD actions, the quality of whose constraints also matter. Imprecise constraints may lead to suboptimal solutions, while precise constraints require significant computational costs. In this paper, we propose a novel count-based method for continuous domains, called Grid-Mapping Pseudo-Count method(GPC), to penalize the Q-value appropriately and reduce the computational cost. The proposed method maps the state and action space to discrete space and constrains their Q-values through the pseudo-count. It is theoretically proved that only a few conditions are needed to obtain accurate uncertainty constraints in the proposed method. Moreover, we develop a Grid-Mapping Pseudo-Count Soft Actor-Critic(GPC-SAC) algorithm using GPC under the Soft Actor-Critic(SAC) framework to demonstrate the effectiveness of GPC. The experimental results on D4RL benchmark datasets show that GPC-SAC has better performance and less computational cost compared to other algorithms.

Title: Representation Alignment Contrastive Regularization for Multi-Object Tracking

• Authors: Shujie Chen, Zhonglin Liu, Jianfeng Dong, Di Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02562
• Pdf URL: https://arxiv.org/pdf/2404.02562
• Copy Paste: [[2404.02562]] Representation Alignment Contrastive Regularization for Multi-Object Tracking(https://arxiv.org/abs/2404.02562)
• Keywords: interpretability, transformer
• Abstract: Achieving high-performance in multi-object tracking algorithms heavily relies on modeling spatio-temporal relationships during the data association stage. Mainstream approaches encompass rule-based and deep learning-based methods for spatio-temporal relationship modeling. While the former relies on physical motion laws, offering wider applicability but yielding suboptimal results for complex object movements, the latter, though achieving high-performance, lacks interpretability and involves complex module designs. This work aims to simplify deep learning-based spatio-temporal relationship models and introduce interpretability into features for data association. Specifically, a lightweight single-layer transformer encoder is utilized to model spatio-temporal relationships. To make features more interpretative, two contrastive regularization losses based on representation alignment are proposed, derived from spatio-temporal consistency rules. By applying weighted summation to affinity matrices, the aligned features can seamlessly integrate into the data association stage of the original tracking workflow. Experimental results showcase that our model enhances the majority of existing tracking networks' performance without excessive complexity, with minimal increase in training overhead and nearly negligible computational and storage costs.

Title: Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models

• Authors: Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Tzu-iunn Ong, Beong-woo Kwak, Moohyeon Kim, Seonghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, Jinyoung Yeo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02575
• Pdf URL: https://arxiv.org/pdf/2404.02575
• Copy Paste: [[2404.02575]] Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models(https://arxiv.org/abs/2404.02575)
• Keywords: large language model
• Abstract: Algorithmic reasoning refers to the ability to understand the complex patterns behind the problem and decompose them into a sequence of reasoning steps towards the solution. Such nature of algorithmic reasoning makes it a challenge for large language models (LLMs), even though they have demonstrated promising performance in other reasoning tasks. Within this context, some recent studies use programming languages (e.g., Python) to express the necessary logic for solving a given instance/question (e.g., Program-of-Thought) as inspired by their strict and precise syntaxes. However, it is non-trivial to write an executable code that expresses the correct logic on the fly within a single inference call. Also, the code generated specifically for an instance cannot be reused for others, even if they are from the same task and might require identical logic to solve. This paper presents Think-and-Execute, a novel framework that decomposes the reasoning process of language models into two steps. (1) In Think, we discover a task-level logic that is shared across all instances for solving a given task and then express the logic with pseudocode; (2) In Execute, we further tailor the generated pseudocode to each instance and simulate the execution of the code. With extensive experiments on seven algorithmic reasoning tasks, we demonstrate the effectiveness of Think-and-Execute. Our approach better improves LMs' reasoning compared to several strong baselines performing instance-specific reasoning (e.g., CoT and PoT), suggesting the helpfulness of discovering task-level logic. Also, we show that compared to natural language, pseudocode can better guide the reasoning of LMs, even though they are trained to follow natural language instructions.

Title: Active learning for efficient annotation in precision agriculture: a use-case on crop-weed semantic segmentation

• Authors: Bart M. van Marrewijk, Charbel Dandjinou, Dan Jeric Arcega Rustia, Nicolas Franco Gonzalez, Boubacar Diallo, Jérôme Dias, Paul Melki, Pieter M. Blok
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02580
• Pdf URL: https://arxiv.org/pdf/2404.02580
• Copy Paste: [[2404.02580]] Active learning for efficient annotation in precision agriculture: a use-case on crop-weed semantic segmentation(https://arxiv.org/abs/2404.02580)
• Keywords: segmentation
• Abstract: Optimizing deep learning models requires large amounts of annotated images, a process that is both time-intensive and costly. Especially for semantic segmentation models in which every pixel must be annotated. A potential strategy to mitigate annotation effort is active learning. Active learning facilitates the identification and selection of the most informative images from a large unlabelled pool. The underlying premise is that these selected images can improve the model's performance faster than random selection to reduce annotation effort. While active learning has demonstrated promising results on benchmark datasets like Cityscapes, its performance in the agricultural domain remains largely unexplored. This study addresses this research gap by conducting a comparative study of three active learning-based acquisition functions: Bayesian Active Learning by Disagreement (BALD), stochastic-based BALD (PowerBALD), and Random. The acquisition functions were tested on two agricultural datasets: Sugarbeet and Corn-Weed, both containing three semantic classes: background, crop and weed. Our results indicated that active learning, especially PowerBALD, yields a higher performance than Random sampling on both datasets. But due to the relatively large standard deviations, the differences observed were minimal; this was partly caused by high image redundancy and imbalanced classes. Specifically, more than 89\% of the pixels belonged to the background class on both datasets. The absence of significant results on both datasets indicates that further research is required for applying active learning on agricultural datasets, especially if they contain a high-class imbalance and redundant images. Recommendations and insights are provided in this paper to potentially resolve such issues.

Title: Transformer-based Stagewise Decomposition for Large-Scale Multistage Stochastic Optimization

• Authors: Chanyeong Kim, Jongwoong Park, Hyunglip Bae, Woo Chang Kim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02583
• Pdf URL: https://arxiv.org/pdf/2404.02583
• Copy Paste: [[2404.02583]] Transformer-based Stagewise Decomposition for Large-Scale Multistage Stochastic Optimization(https://arxiv.org/abs/2404.02583)
• Keywords: transformer
• Abstract: Solving large-scale multistage stochastic programming (MSP) problems poses a significant challenge as commonly used stagewise decomposition algorithms, including stochastic dual dynamic programming (SDDP), face growing time complexity as the subproblem size and problem count increase. Traditional approaches approximate the value functions as piecewise linear convex functions by incrementally accumulating subgradient cutting planes from the primal and dual solutions of stagewise subproblems. Recognizing these limitations, we introduce TranSDDP, a novel Transformer-based stagewise decomposition algorithm. This innovative approach leverages the structural advantages of the Transformer model, implementing a sequential method for integrating subgradient cutting planes to approximate the value function. Through our numerical experiments, we affirm TranSDDP's effectiveness in addressing MSP problems. It efficiently generates a piecewise linear approximation for the value function, significantly reducing computation time while preserving solution quality, thus marking a promising progression in the treatment of large-scale multistage stochastic programming problems.

Title: Unsegment Anything by Simulating Deformation

• Authors: Jiahao Lu, Xingyi Yang, Xinchao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02585
• Pdf URL: https://arxiv.org/pdf/2404.02585
• Copy Paste: [[2404.02585]] Unsegment Anything by Simulating Deformation(https://arxiv.org/abs/2404.02585)
• Keywords: attack, segmentation
• Abstract: Foundation segmentation models, while powerful, pose a significant risk: they enable users to effortlessly extract any objects from any digital content with a single click, potentially leading to copyright infringement or malicious misuse. To mitigate this risk, we introduce a new task "Anything Unsegmentable" to grant any image "the right to be unsegmented". The ambitious pursuit of the task is to achieve highly transferable adversarial attacks against all prompt-based segmentation models, regardless of model parameterizations and prompts. We highlight the non-transferable and heterogeneous nature of prompt-specific adversarial noises. Our approach focuses on disrupting image encoder features to achieve prompt-agnostic attacks. Intriguingly, targeted feature attacks exhibit better transferability compared to untargeted ones, suggesting the optimal update direction aligns with the image manifold. Based on the observations, we design a novel attack named Unsegment Anything by Simulating Deformation (UAD). Our attack optimizes a differentiable deformation function to create a target deformed image, which alters structural information while preserving achievable feature distance by adversarial example. Extensive experiments verify the effectiveness of our approach, compromising a variety of promptable segmentation models with different architectures and prompt interfaces. We release the code at https://github.com/jiahaolu97/anything-unsegmentable.

Title: Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages

• Authors: Jakub Hoscilowicz, Pawel Pawlowski, Marcin Skorupa, Marcin Sowański, Artur Janicki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02588
• Pdf URL: https://arxiv.org/pdf/2404.02588
• Copy Paste: [[2404.02588]] Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages(https://arxiv.org/abs/2404.02588)
• Keywords: large language model
• Abstract: Spoken Language Understanding (SLU) models are a core component of voice assistants (VA), such as Alexa, Bixby, and Google Assistant. In this paper, we introduce a pipeline designed to extend SLU systems to new languages, utilizing Large Language Models (LLMs) that we fine-tune for machine translation of slot-annotated SLU training data. Our approach improved on the MultiATIS++ benchmark, a primary multi-language SLU dataset, in the cloud scenario using an mBERT model. Specifically, we saw an improvement in the Overall Accuracy metric: from 53% to 62.18%, compared to the existing state-of-the-art method, Fine and Coarse-grained Multi-Task Learning Framework (FC-MTLF). In the on-device scenario (tiny and not pretrained SLU), our method improved the Overall Accuracy from 5.31% to 22.06% over the baseline Global-Local Contrastive Learning Framework (GL-CLeF) method. Contrary to both FC-MTLF and GL-CLeF, our LLM-based machine translation does not require changes in the production architecture of SLU. Additionally, our pipeline is slot-type independent: it does not require any slot definitions or examples.

Title: Affective-NLI: Towards Accurate and Interpretable Personality Recognition in Conversation

• Authors: Zhiyuan Wen, Jiannong Cao, Yu Yang, Ruosong Yang, Shuaiqi Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02589
• Pdf URL: https://arxiv.org/pdf/2404.02589
• Copy Paste: [[2404.02589]] Affective-NLI: Towards Accurate and Interpretable Personality Recognition in Conversation(https://arxiv.org/abs/2404.02589)
• Keywords: interpretability
• Abstract: Personality Recognition in Conversation (PRC) aims to identify the personality traits of speakers through textual dialogue content. It is essential for providing personalized services in various applications of Human-Computer Interaction (HCI), such as AI-based mental therapy and companion robots for the elderly. Most recent studies analyze the dialog content for personality classification yet overlook two major concerns that hinder their performance. First, crucial implicit factors contained in conversation, such as emotions that reflect the speakers' personalities are ignored. Second, only focusing on the input dialog content disregards the semantic understanding of personality itself, which reduces the interpretability of the results. In this paper, we propose Affective Natural Language Inference (Affective-NLI) for accurate and interpretable PRC. To utilize affectivity within dialog content for accurate personality recognition, we fine-tuned a pre-trained language model specifically for emotion recognition in conversations, facilitating real-time affective annotations for utterances. For interpretability of recognition results, we formulate personality recognition as an NLI problem by determining whether the textual description of personality labels is entailed by the dialog content. Extensive experiments on two daily conversation datasets suggest that Affective-NLI significantly outperforms (by 6%-7%) state-of-the-art approaches. Additionally, our Flow experiment demonstrates that Affective-NLI can accurately recognize the speaker's personality in the early stages of conversations by surpassing state-of-the-art methods with 22%-34%.

Title: LightFAt: Mitigating Control-flow Explosion via Lightweight PMU-based Control-flow Attestation

• Authors: Jeferson Gonzalez-Gomez, Hassan Nassar, Lars Bauer, Jorg Henkel
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.02608
• Pdf URL: https://arxiv.org/pdf/2404.02608
• Copy Paste: [[2404.02608]] LightFAt: Mitigating Control-flow Explosion via Lightweight PMU-based Control-flow Attestation(https://arxiv.org/abs/2404.02608)
• Keywords: security
• Abstract: With the continuous evolution of computational devices, more and more applications are being executed remotely. The applications operate on a wide spectrum of devices, ranging from IoT nodes with low computational capabilities to large cloud providers with high capabilities. Remote execution often deals with sensitive data or executes proprietary software. Hence, the challenge of ensuring that the code execution will not be compromised rises. Remote Attestation deals with this challenge. It ensures the code is executed in a non-compromised environment by calculating a potentially large sequence of cryptographic hash values. Each hash calculation is computationally intensive and over a large sequence the overhead becomes extremely high. In this work, we propose LightFAt: a Lightweight Control Flow Attestation scheme. Instead of relying on the expensive cryptographic hash calculation, LightFAt leverages the readings from the processor's Performance Monitor Unit (PMU) in conjunction with a lightweight unsupervised machine learning (ML) classifier to detect whether a target application's control flow is compromised, hence improving the system's security. On the verifier's side, LightFAt reaches a detection accuracy of over 95%, with low false-negative and false-positive rates.

Title: Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models

• Authors: Matteo Pennisi, Giovanni Bellitto, Simone Palazzo, Mubarak Shah, Concetto Spampinato
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02618
• Pdf URL: https://arxiv.org/pdf/2404.02618
• Copy Paste: [[2404.02618]] Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models(https://arxiv.org/abs/2404.02618)
• Keywords: explainability, diffusion
• Abstract: We present DiffExplainer, a novel framework that, leveraging language-vision models, enables multimodal global explainability. DiffExplainer employs diffusion models conditioned on optimized text prompts, synthesizing images that maximize class outputs and hidden features of a classifier, thus providing a visual tool for explaining decisions. Moreover, the analysis of generated visual descriptions allows for automatic identification of biases and spurious features, as opposed to traditional methods that often rely on manual intervention. The cross-modal transferability of language-vision models also enables the possibility to describe decisions in a more human-interpretable way, i.e., through text. We conduct comprehensive experiments, which include an extensive user study, demonstrating the effectiveness of DiffExplainer on 1) the generation of high-quality images explaining model decisions, surpassing existing activation maximization methods, and 2) the automated identification of biases and spurious features.

Title: Estimating the Causal Effects of Natural Logic Features in Transformer-Based NLI Models

• Authors: Julia Rozanova, Marco Valentino, André Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02622
• Pdf URL: https://arxiv.org/pdf/2404.02622
• Copy Paste: [[2404.02622]] Estimating the Causal Effects of Natural Logic Features in Transformer-Based NLI Models(https://arxiv.org/abs/2404.02622)
• Keywords: robust, interpretability, transformer
• Abstract: Rigorous evaluation of the causal effects of semantic features on language model predictions can be hard to achieve for natural language reasoning problems. However, this is such a desirable form of analysis from both an interpretability and model evaluation perspective, that it is valuable to investigate specific patterns of reasoning with enough structure and regularity to identify and quantify systematic reasoning failures in widely-used models. In this vein, we pick a portion of the NLI task for which an explicit causal diagram can be systematically constructed: the case where across two sentences (the premise and hypothesis), two related words/terms occur in a shared context. In this work, we apply causal effect estimation strategies to measure the effect of context interventions (whose effect on the entailment label is mediated by the semantic monotonicity characteristic) and interventions on the inserted word-pair (whose effect on the entailment label is mediated by the relation between these words). Extending related work on causal analysis of NLP models in different settings, we perform an extensive interventional study on the NLI task to investigate robustness to irrelevant changes and sensitivity to impactful changes of Transformers. The results strongly bolster the fact that similar benchmark accuracy scores may be observed for models that exhibit very different behaviour. Moreover, our methodology reinforces previously suspected biases from a causal perspective, including biases in favour of upward-monotone contexts and ignoring the effects of negation markers.

Title: A Differentiable Integer Linear Programming Solver for Explanation-Based Natural Language Inference

• Authors: Mokanarangan Thayaparan, Marco Valentino, André Freitas
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02625
• Pdf URL: https://arxiv.org/pdf/2404.02625
• Copy Paste: [[2404.02625]] A Differentiable Integer Linear Programming Solver for Explanation-Based Natural Language Inference(https://arxiv.org/abs/2404.02625)
• Keywords: transformer
• Abstract: Integer Linear Programming (ILP) has been proposed as a formalism for encoding precise structural and semantic constraints for Natural Language Inference (NLI). However, traditional ILP frameworks are non-differentiable, posing critical challenges for the integration of continuous language representations based on deep learning. In this paper, we introduce a novel approach, named Diff-Comb Explainer, a neuro-symbolic architecture for explanation-based NLI based on Differentiable BlackBox Combinatorial Solvers (DBCS). Differently from existing neuro-symbolic solvers, Diff-Comb Explainer does not necessitate a continuous relaxation of the semantic constraints, enabling a direct, more precise, and efficient incorporation of neural representations into the ILP formulation. Our experiments demonstrate that Diff-Comb Explainer achieves superior performance when compared to conventional ILP solvers, neuro-symbolic black-box solvers, and Transformer-based encoders. Moreover, a deeper analysis reveals that Diff-Comb Explainer can significantly improve the precision, consistency, and faithfulness of the constructed explanations, opening new opportunities for research on neuro-symbolic architectures for explainable and transparent NLI in complex domains.

Title: Effector: A Python package for regional explanations

• Authors: Vasilis Gkolemis, Christos Diou, Eirini Ntoutsi, Theodore Dalamagas, Bernd Bischl, Julia Herbinger, Giuseppe Casalicchio
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02629
• Pdf URL: https://arxiv.org/pdf/2404.02629
• Copy Paste: [[2404.02629]] Effector: A Python package for regional explanations(https://arxiv.org/abs/2404.02629)
• Keywords: interpretability
• Abstract: Global feature effect methods explain a model outputting one plot per feature. The plot shows the average effect of the feature on the output, like the effect of age on the annual income. However, average effects may be misleading when derived from local effects that are heterogeneous, i.e., they significantly deviate from the average. To decrease the heterogeneity, regional effects provide multiple plots per feature, each representing the average effect within a specific subspace. For interpretability, subspaces are defined as hyperrectangles defined by a chain of logical rules, like age's effect on annual income separately for males and females and different levels of professional experience. We introduce Effector, a Python library dedicated to regional feature effects. Effector implements well-established global effect methods, assesses the heterogeneity of each method and, based on that, provides regional effects. Effector automatically detects subspaces where regional effects have reduced heterogeneity. All global and regional effect methods share a common API, facilitating comparisons between them. Moreover, the library's interface is extensible so new methods can be easily added and benchmarked. The library has been thoroughly tested, ships with many tutorials (https://xai-effector.github.io/) and is available under an open-source license at PyPi (https://pypi.org/project/effector/) and Github (https://github.com/givasile/effector).

Title: Vocabulary Attack to Hijack Large Language Model Applications

• Authors: Patrick Levi, Christoph P. Neumann
• Subjects: cs.CR, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.02637
• Pdf URL: https://arxiv.org/pdf/2404.02637
• Copy Paste: [[2404.02637]] Vocabulary Attack to Hijack Large Language Model Applications(https://arxiv.org/abs/2404.02637)
• Keywords: attack, large language model
• Abstract: The fast advancements in Large Language Models (LLMs) are driving an increasing number of applications. Together with the growing number of users, we also see an increasing number of attackers who try to outsmart these systems. They want the model to reveal confidential information, specific false information, or offensive behavior. To this end, they manipulate their instructions for the LLM by inserting separators or rephrasing them systematically until they reach their goal. Our approach is different. It inserts words from the model vocabulary. We find these words using an optimization procedure and embeddings from another LLM (attacker LLM). We prove our approach by goal hijacking two popular open-source LLMs from the Llama2 and the Flan-T5 families, respectively. We present two main findings. First, our approach creates inconspicuous instructions and therefore it is hard to detect. For many attack cases, we find that even a single word insertion is sufficient. Second, we demonstrate that we can conduct our attack using a different model than the target model to conduct our attack with.

Title: SG-BEV: Satellite-Guided BEV Fusion for Cross-View Semantic Segmentation

• Authors: Junyan Ye, Qiyan Luo, Jinhua Yu, Huaping Zhong, Zhimeng Zheng, Conghui He, Weijia Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02638
• Pdf URL: https://arxiv.org/pdf/2404.02638
• Copy Paste: [[2404.02638]] SG-BEV: Satellite-Guided BEV Fusion for Cross-View Semantic Segmentation(https://arxiv.org/abs/2404.02638)
• Keywords: segmentation
• Abstract: This paper aims at achieving fine-grained building attribute segmentation in a cross-view scenario, i.e., using satellite and street-view image pairs. The main challenge lies in overcoming the significant perspective differences between street views and satellite views. In this work, we introduce SG-BEV, a novel approach for satellite-guided BEV fusion for cross-view semantic segmentation. To overcome the limitations of existing cross-view projection methods in capturing the complete building facade features, we innovatively incorporate Bird's Eye View (BEV) method to establish a spatially explicit mapping of street-view features. Moreover, we fully leverage the advantages of multiple perspectives by introducing a novel satellite-guided reprojection module, optimizing the uneven feature distribution issues associated with traditional BEV methods. Our method demonstrates significant improvements on four cross-view datasets collected from multiple cities, including New York, San Francisco, and Boston. On average across these datasets, our method achieves an increase in mIOU by 10.13% and 5.21% compared with the state-of-the-art satellite-based and cross-view methods. The code and datasets of this work will be released at https://github.com/yejy53/SG-BEV.

Title: On the Importance of Uncertainty in Decision-Making with Large Language Models

• Authors: Nicolò Felicioni, Lucas Maystre, Sina Ghiassian, Kamil Ciosek
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02649
• Pdf URL: https://arxiv.org/pdf/2404.02649
• Copy Paste: [[2404.02649]] On the Importance of Uncertainty in Decision-Making with Large Language Models(https://arxiv.org/abs/2404.02649)
• Keywords: large language model
• Abstract: We investigate the role of uncertainty in decision-making problems with natural language as input. For such tasks, using Large Language Models as agents has become the norm. However, none of the recent approaches employ any additional phase for estimating the uncertainty the agent has about the world during the decision-making task. We focus on a fundamental decision-making framework with natural language as input, which is the one of contextual bandits, where the context information consists of text. As a representative of the approaches with no uncertainty estimation, we consider an LLM bandit with a greedy policy, which picks the action corresponding to the largest predicted reward. We compare this baseline to LLM bandits that make active use of uncertainty estimation by integrating the uncertainty in a Thompson Sampling policy. We employ different techniques for uncertainty estimation, such as Laplace Approximation, Dropout, and Epinets. We empirically show on real-world data that the greedy policy performs worse than the Thompson Sampling policies. These findings suggest that, while overlooked in the LLM literature, uncertainty plays a fundamental role in bandit tasks with LLMs.

Title: Towards detecting unanticipated bias in Large Language Models

• Authors: Anna Kruspe
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02650
• Pdf URL: https://arxiv.org/pdf/2404.02650
• Copy Paste: [[2404.02650]] Towards detecting unanticipated bias in Large Language Models(https://arxiv.org/abs/2404.02650)
• Keywords: fair, large language model
• Abstract: Over the last year, Large Language Models (LLMs) like ChatGPT have become widely available and have exhibited fairness issues similar to those in previous machine learning systems. Current research is primarily focused on analyzing and quantifying these biases in training data and their impact on the decisions of these models, alongside developing mitigation strategies. This research largely targets well-known biases related to gender, race, ethnicity, and language. However, it is clear that LLMs are also affected by other, less obvious implicit biases. The complex and often opaque nature of these models makes detecting such biases challenging, yet this is crucial due to their potential negative impact in various applications. In this paper, we explore new avenues for detecting these unanticipated biases in LLMs, focusing specifically on Uncertainty Quantification and Explainable AI methods. These approaches aim to assess the certainty of model decisions and to make the internal decision-making processes of LLMs more transparent, thereby identifying and understanding biases that are not immediately apparent. Through this research, we aim to contribute to the development of fairer and more transparent AI systems.

Title: Calibrating the Confidence of Large Language Models by Eliciting Fidelity

• Authors: Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, Xipeng Qiu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02655
• Pdf URL: https://arxiv.org/pdf/2404.02655
• Copy Paste: [[2404.02655]] Calibrating the Confidence of Large Language Models by Eliciting Fidelity(https://arxiv.org/abs/2404.02655)
• Keywords: large language model
• Abstract: Large language models optimized with techniques like RLHF have achieved good alignment in being helpful and harmless. However, post-alignment, these language models often exhibit overconfidence, where the expressed confidence does not accurately calibrate with their correctness rate. In this paper, we decompose the language model confidence into the \textit{Uncertainty} about the question and the \textit{Fidelity} to the answer generated by language models. Then, we propose a plug-and-play method to estimate the confidence of language models. Our method has shown good calibration performance by conducting experiments with 6 RLHF-LMs on four MCQA datasets. Moreover, we propose two novel metrics, IPR and CE, to evaluate the calibration of the model, and we have conducted a detailed discussion on \textit{Truly Well-Calibrated Confidence}. Our method could serve as a strong baseline, and we hope that this work will provide some insights into the model confidence calibration.

Title: Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models

• Authors: Taiqiang Wu, Chaofan Tao, Jiahao Wang, Zhe Zhao, Ngai Wong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02657
• Pdf URL: https://arxiv.org/pdf/2404.02657
• Copy Paste: [[2404.02657]] Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models(https://arxiv.org/abs/2404.02657)
• Keywords: large language model
• Abstract: Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to compress Large Language Models (LLMs). Contrary to prior assertions that reverse Kullback-Leibler (RKL) divergence is mode-seeking and thus preferable over the mean-seeking forward Kullback-Leibler (FKL) divergence, this study empirically and theoretically demonstrates that neither mode-seeking nor mean-seeking properties manifest in KD for LLMs. Instead, RKL and FKL are found to share the same optimization objective and both converge after a sufficient number of epochs. However, due to practical constraints, LLMs are seldom trained for such an extensive number of epochs. Meanwhile, we further find that RKL focuses on the tail part of the distributions, while FKL focuses on the head part at the beginning epochs. Consequently, we propose a simple yet effective Adaptive Kullback-Leiber (AKL) divergence method, which adaptively allocates weights to combine FKL and RKL. Metric-based and GPT-4-based evaluations demonstrate that the proposed AKL outperforms the baselines across various tasks and improves the diversity and quality of generated responses.

Title: A Satellite Band Selection Framework for Amazon Forest Deforestation Detection Task

• Authors: Eduardo Neto, Fabio A. Faria, Amanda A. S. de Oliveira, Álvaro L. Fazenda
• Subjects: cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.02659
• Pdf URL: https://arxiv.org/pdf/2404.02659
• Copy Paste: [[2404.02659]] A Satellite Band Selection Framework for Amazon Forest Deforestation Detection Task(https://arxiv.org/abs/2404.02659)
• Keywords: segmentation
• Abstract: The conservation of tropical forests is a topic of significant social and ecological relevance due to their crucial role in the global ecosystem. Unfortunately, deforestation and degradation impact millions of hectares annually, necessitating government or private initiatives for effective forest monitoring. This study introduces a novel framework that employs the Univariate Marginal Distribution Algorithm (UMDA) to select spectral bands from Landsat-8 satellite, optimizing the representation of deforested areas. This selection guides a semantic segmentation architecture, DeepLabv3+, enhancing its performance. Experimental results revealed several band compositions that achieved superior balanced accuracy compared to commonly adopted combinations for deforestation detection, utilizing segment classification via a Support Vector Machine (SVM). Moreover, the optimal band compositions identified by the UMDA-based approach improved the performance of the DeepLabv3+ architecture, surpassing state-of-the-art approaches compared in this study. The observation that a few selected bands outperform the total contradicts the data-driven paradigm prevalent in the deep learning field. Therefore, this suggests an exception to the conventional wisdom that 'more is always better'.

Title: Adversarial Attacks and Dimensionality in Text Classifiers

• Authors: Nandish Chattopadhyay, Atreya Goswami, Anupam Chattopadhyay
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02660
• Pdf URL: https://arxiv.org/pdf/2404.02660
• Copy Paste: [[2404.02660]] Adversarial Attacks and Dimensionality in Text Classifiers(https://arxiv.org/abs/2404.02660)
• Keywords: defense, attack, robust
• Abstract: Adversarial attacks on machine learning algorithms have been a key deterrent to the adoption of AI in many real-world use cases. They significantly undermine the ability of high-performance neural networks by forcing misclassifications. These attacks introduce minute and structured perturbations or alterations in the test samples, imperceptible to human annotators in general, but trained neural networks and other models are sensitive to it. Historically, adversarial attacks have been first identified and studied in the domain of image processing. In this paper, we study adversarial examples in the field of natural language processing, specifically text classification tasks. We investigate the reasons for adversarial vulnerability, particularly in relation to the inherent dimensionality of the model. Our key finding is that there is a very strong correlation between the embedding dimensionality of the adversarial samples and their effectiveness on models tuned with input samples with same embedding dimension. We utilize this sensitivity to design an adversarial defense mechanism. We use ensemble models of varying inherent dimensionality to thwart the attacks. This is tested on multiple datasets for its efficacy in providing robustness. We also study the problem of measuring adversarial perturbation using different distance metrics. For all of the aforementioned studies, we have run tests on multiple models with varying dimensionality and used a word-vector level adversarial attack to substantiate the findings.

Title: RS-Mamba for Large Remote Sensing Image Dense Prediction

• Authors: Sijie Zhao, Hao Chen, Xueliang Zhang, Pengfeng Xiao, Lei Bai, Wanli Ouyang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02668
• Pdf URL: https://arxiv.org/pdf/2404.02668
• Copy Paste: [[2404.02668]] RS-Mamba for Large Remote Sensing Image Dense Prediction(https://arxiv.org/abs/2404.02668)
• Keywords: transformer, segmentation
• Abstract: The spatial resolution of remote sensing images is becoming increasingly higher, posing challenges in handling large very-high-resolution (VHR) remote sensing images for dense prediction tasks. Models based on convolutional neural networks are limited in their ability to model global features of remote sensing images due to local convolution operations. Transformer based models, despite their global modeling capabilities, face computational challenges with large VHR images due to their quadratic complexity. The common practice of cropping large images into smaller patches leads to a significant loss of contextual information. To address these issues, we propose the Remote Sensing Mamba (RSM) for dense prediction tasks in VHR remote sensing. RSM is designed to model global features of remote sensing images with linear complexity, enabling it to process large VHR images effectively. It employs an omnidirectional selective scan module to globally model the images in multiple directions, capturing large spatial features from various directions. Experiments on semantic segmentation and change detection tasks across various objects demonstrate the effectiveness of RSM. With simple model architecture and training approach, RSM achieves state-of-the-art performance on the dense prediction tasks of VHR remote sensing. The code for this work will be available at https://github.com/walking-shadow/Official_Remote_Sensing_Mamba.

Title: Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers

• Authors: Sehyun Choi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02684
• Pdf URL: https://arxiv.org/pdf/2404.02684
• Copy Paste: [[2404.02684]] Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers(https://arxiv.org/abs/2404.02684)
• Keywords: transformer
• Abstract: Recently, multiple architectures has been proposed to improve the efficiency of the Transformer Language Models through changing the design of the self-attention block to have a linear-cost inference (LCI). A notable approach in this realm is the State-Space Machines (SSMs) architecture, which showed on-par performance on language modeling tasks with the self-attention transformers. However, such an architectural change requires a full pretraining of the weights from scratch, which incurs a huge cost to researchers and practitioners who want to use the new architectures. In the more traditional linear attention works, it has been proposed to approximate full attention with linear attention by swap-and-finetune framework. Motivated by this approach, we propose Cross-Architecture Transfer Learning (XATL), in which the weights of the shared components between LCI and self-attention-based transformers, such as layernorms, MLPs, input/output embeddings, are directly transferred to the new architecture from already pre-trained model parameters. We experimented the efficacy of the method on varying sizes and alternative attention architectures and show that \methodabbr significantly reduces the training time up to 2.5x times and converges to a better minimum with up to 2.6% stronger model on the LM benchmarks within the same compute budget.

Title: Design2Cloth: 3D Cloth Generation from 2D Masks

• Authors: Jiali Zheng, Rolandos Alexandros Potamias, Stefanos Zafeiriou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02686
• Pdf URL: https://arxiv.org/pdf/2404.02686
• Copy Paste: [[2404.02686]] Design2Cloth: 3D Cloth Generation from 2D Masks(https://arxiv.org/abs/2404.02686)
• Keywords: generative
• Abstract: In recent years, there has been a significant shift in the field of digital avatar research, towards modeling, animating and reconstructing clothed human representations, as a key step towards creating realistic avatars. However, current 3D cloth generation methods are garment specific or trained completely on synthetic data, hence lacking fine details and realism. In this work, we make a step towards automatic realistic garment design and propose Design2Cloth, a high fidelity 3D generative model trained on a real world dataset from more than 2000 subject scans. To provide vital contribution to the fashion industry, we developed a user-friendly adversarial model capable of generating diverse and detailed clothes simply by drawing a 2D cloth mask. Under a series of both qualitative and quantitative experiments, we showcase that Design2Cloth outperforms current state-of-the-art cloth generative models by a large margin. In addition to the generative properties of our network, we showcase that the proposed method can be used to achieve high quality reconstructions from single in-the-wild images and 3D scans. Dataset, code and pre-trained model will become publicly available.

Title: Attention is Naturally Sparse with Gaussian Distributed Input

• Authors: Yichuan Deng, Zhao Song, Chiwun Yang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02690
• Pdf URL: https://arxiv.org/pdf/2404.02690
• Copy Paste: [[2404.02690]] Attention is Naturally Sparse with Gaussian Distributed Input(https://arxiv.org/abs/2404.02690)
• Keywords: transformer, large language model
• Abstract: The computational intensity of Large Language Models (LLMs) is a critical bottleneck, primarily due to the $O(n^2)$ complexity of the attention mechanism in transformer architectures. Addressing this, sparse attention emerges as a key innovation, aiming to reduce computational load while maintaining model performance. This study presents a rigorous theoretical analysis of the sparsity in attention scores within LLMs, particularly under the framework of Gaussian inputs. By establishing a set of foundational assumptions and employing a methodical theoretical approach, we unravel the intrinsic characteristics of attention score sparsity and its implications on computational efficiency. Our main contribution lies in providing a detailed theoretical examination of how sparsity manifests in attention mechanisms, offering insights into the potential trade-offs between computational savings and model effectiveness. This work not only advances our understanding of sparse attention but also provides a scaffold for future research in optimizing the computational frameworks of LLMs, paving the way for more scalable and efficient AI systems.

Title: Deep Privacy Funnel Model: From a Discriminative to a Generative Approach with an Application to Face Recognition

• Authors: Behrooz Razeghi, Parsa Rahimi, Sébastien Marcel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02696
• Pdf URL: https://arxiv.org/pdf/2404.02696
• Copy Paste: [[2404.02696]] Deep Privacy Funnel Model: From a Discriminative to a Generative Approach with an Application to Face Recognition(https://arxiv.org/abs/2404.02696)
• Keywords: privacy, protect, diffusion, generative
• Abstract: In this study, we apply the information-theoretic Privacy Funnel (PF) model to the domain of face recognition, developing a novel method for privacy-preserving representation learning within an end-to-end training framework. Our approach addresses the trade-off between obfuscation and utility in data protection, quantified through logarithmic loss, also known as self-information loss. This research provides a foundational exploration into the integration of information-theoretic privacy principles with representation learning, focusing specifically on the face recognition systems. We particularly highlight the adaptability of our framework with recent advancements in face recognition networks, such as AdaFace and ArcFace. In addition, we introduce the Generative Privacy Funnel ($\mathsf{GenPF}$) model, a paradigm that extends beyond the traditional scope of the PF model, referred to as the Discriminative Privacy Funnel ($\mathsf{DisPF}$). This $\mathsf{GenPF}$ model brings new perspectives on data generation methods with estimation-theoretic and information-theoretic privacy guarantees. Complementing these developments, we also present the deep variational PF (DVPF) model. This model proposes a tractable variational bound for measuring information leakage, enhancing the understanding of privacy preservation challenges in deep representation learning. The DVPF model, associated with both $\mathsf{DisPF}$ and $\mathsf{GenPF}$ models, sheds light on connections with various generative models such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Diffusion models. Complementing our theoretical contributions, we release a reproducible PyTorch package, facilitating further exploration and application of these privacy-preserving methodologies in face recognition systems.

Title: Model-agnostic Origin Attribution of Generated Images with Few-shot Examples

• Authors: Fengyuan Liu, Haochen Luo, Yiming Li, Philip Torr, Jindong Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02697
• Pdf URL: https://arxiv.org/pdf/2404.02697
• Copy Paste: [[2404.02697]] Model-agnostic Origin Attribution of Generated Images with Few-shot Examples(https://arxiv.org/abs/2404.02697)
• Keywords: generative
• Abstract: Recent progress in visual generative models enables the generation of high-quality images. To prevent the misuse of generated images, it is important to identify the origin model that generates them. In this work, we study the origin attribution of generated images in a practical setting where only a few images generated by a source model are available and the source model cannot be accessed. The goal is to check if a given image is generated by the source model. We first formulate this problem as a few-shot one-class classification task. To solve the task, we propose OCC-CLIP, a CLIP-based framework for few-shot one-class classification, enabling the identification of an image's source model, even among multiple candidates. Extensive experiments corresponding to various generative models verify the effectiveness of our OCC-CLIP framework. Furthermore, an experiment based on the recently released DALL-E 3 API verifies the real-world applicability of our solution.

Title: Scalable Model Editing via Customized Expert Networks

• Authors: Zihan Yao, Yu He, Tianyu Qi, Ming Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02699
• Pdf URL: https://arxiv.org/pdf/2404.02699
• Copy Paste: [[2404.02699]] Scalable Model Editing via Customized Expert Networks(https://arxiv.org/abs/2404.02699)
• Keywords: large language model
• Abstract: Addressing the issue of hallucinations and outdated knowledge in large language models is critical for their reliable application. Model Editing presents a promising avenue for mitigating these challenges in a cost-effective manner. However, existing methods often suffer from unsatisfactory generalization and unintended effects on unrelated samples. To overcome these limitations, we introduce a novel approach: Scalable Model Editing via Customized Expert Networks (SCEN), which is a two-stage continuous training paradigm. Specifically, in the first stage, we train lightweight expert networks individually for each piece of knowledge that needs to be updated. Subsequently, we train a corresponding neuron for each expert to control the activation state of that expert. Our experiments on two different sizes of open-source large language models, the Llama2 7B and 13B, achieve state-of-the-art results compared to existing mainstream Model Editing methods. Our code is available at https: //github.com/TAL-auroraX/SCEN

Title: Automatic Prompt Selection for Large Language Models

• Authors: Viet-Tung Do, Van-Khanh Hoang, Duy-Hung Nguyen, Shahab Sabahi, Jeff Yang, Hajime Hotta, Minh-Tien Nguyen, Hung Le
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02717
• Pdf URL: https://arxiv.org/pdf/2404.02717
• Copy Paste: [[2404.02717]] Automatic Prompt Selection for Large Language Models(https://arxiv.org/abs/2404.02717)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) can perform various natural language processing tasks with suitable instruction prompts. However, designing effective prompts manually is challenging and time-consuming. Existing methods for automatic prompt optimization either lack flexibility or efficiency. In this paper, we propose an effective approach to automatically select the optimal prompt for a given input from a finite set of synthetic candidate prompts. Our approach consists of three steps: (1) clustering the training data and generating candidate prompts for each cluster using an LLM-based prompt generator; (2) synthesizing a dataset of input-prompt-output tuples for training a prompt evaluator to rank the prompts based on their relevance to the input; (3) using the prompt evaluator to select the best prompt for a new input at test time. Our approach balances prompt generality-specificity and eliminates the need for resource-intensive training and inference. It demonstrates competitive performance on zero-shot question-answering datasets: GSM8K, MultiArith, and AQuA.

Title: Harnessing the Power of Large Vision Language Models for Synthetic Image Detection

• Authors: Mamadou Keita, Wassim Hamidouche, Hassen Bougueffa, Abdenour Hadid, Abdelmalik Taleb-Ahmed
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02726
• Pdf URL: https://arxiv.org/pdf/2404.02726
• Copy Paste: [[2404.02726]] Harnessing the Power of Large Vision Language Models for Synthetic Image Detection(https://arxiv.org/abs/2404.02726)
• Keywords: robust, diffusion
• Abstract: In recent years, the emergence of models capable of generating images from text has attracted considerable interest, offering the possibility of creating realistic images from text descriptions. Yet these advances have also raised concerns about the potential misuse of these images, including the creation of misleading content such as fake news and propaganda. This study investigates the effectiveness of using advanced vision-language models (VLMs) for synthetic image identification. Specifically, the focus is on tuning state-of-the-art image captioning models for synthetic image detection. By harnessing the robust understanding capabilities of large VLMs, the aim is to distinguish authentic images from synthetic images produced by diffusion-based models. This study contributes to the advancement of synthetic image detection by exploiting the capabilities of visual language models such as BLIP-2 and ViTGPT2. By tailoring image captioning models, we address the challenges associated with the potential misuse of synthetic images in real-world applications. Results described in this paper highlight the promising role of VLMs in the field of synthetic image detection, outperforming conventional image-based detection techniques. Code and models can be found at https://github.com/Mamadou-Keita/VLM-DETECT.

Title: InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation

• Authors: Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, Anthony Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02733
• Pdf URL: https://arxiv.org/pdf/2404.02733
• Copy Paste: [[2404.02733]] InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation(https://arxiv.org/abs/2404.02733)
• Keywords: diffusion
• Abstract: Tuning-free diffusion-based models have demonstrated significant potential in the realm of image personalization and customization. However, despite this notable progress, current models continue to grapple with several complex challenges in producing style-consistent image generation. Firstly, the concept of style is inherently underdetermined, encompassing a multitude of elements such as color, material, atmosphere, design, and structure, among others. Secondly, inversion-based methods are prone to style degradation, often resulting in the loss of fine-grained details. Lastly, adapter-based approaches frequently require meticulous weight tuning for each reference image to achieve a balance between style intensity and text controllability. In this paper, we commence by examining several compelling yet frequently overlooked observations. We then proceed to introduce InstantStyle, a framework designed to address these issues through the implementation of two key strategies: 1) A straightforward mechanism that decouples style and content from reference images within the feature space, predicated on the assumption that features within the same space can be either added to or subtracted from one another. 2) The injection of reference image features exclusively into style-specific blocks, thereby preventing style leaks and eschewing the need for cumbersome weight tuning, which often characterizes more parameter-heavy designs.Our work demonstrates superior visual stylization outcomes, striking an optimal balance between the intensity of style and the controllability of textual elements. Our codes will be available at https://github.com/InstantStyle/InstantStyle.

Title: Adaptive Affinity-Based Generalization For MRI Imaging Segmentation Across Resource-Limited Settings

• Authors: Eddardaa B.Loussaief, Mohammed Ayad, Domenc Puig, Hatem A.Rashwan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02738
• Pdf URL: https://arxiv.org/pdf/2404.02738
• Copy Paste: [[2404.02738]] Adaptive Affinity-Based Generalization For MRI Imaging Segmentation Across Resource-Limited Settings(https://arxiv.org/abs/2404.02738)
• Keywords: robust, segmentation
• Abstract: The joint utilization of diverse data sources for medical imaging segmentation has emerged as a crucial area of research, aiming to address challenges such as data heterogeneity, domain shift, and data quality discrepancies. Integrating information from multiple data domains has shown promise in improving model generalizability and adaptability. However, this approach often demands substantial computational resources, hindering its practicality. In response, knowledge distillation (KD) has garnered attention as a solution. KD involves training light-weight models to emulate the behavior of more resource-intensive models, thereby mitigating the computational burden while maintaining performance. This paper addresses the pressing need to develop a lightweight and generalizable model for medical imaging segmentation that can effectively handle data integration challenges. Our proposed approach introduces a novel relation-based knowledge framework by seamlessly combining adaptive affinity-based and kernel-based distillation through a gram matrix that can capture the style representation across features. This methodology empowers the student model to accurately replicate the feature representations of the teacher model, facilitating robust performance even in the face of domain shift and data heterogeneity. To validate our innovative approach, we conducted experiments on publicly available multi-source prostate MRI data. The results demonstrate a significant enhancement in segmentation performance using lightweight networks. Notably, our method achieves this improvement while reducing both inference time and storage usage, rendering it a practical and efficient solution for real-time medical imaging segmentation.

Title: Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion Models

• Authors: Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, Jürgen Schmidhuber
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02747
• Pdf URL: https://arxiv.org/pdf/2404.02747
• Copy Paste: [[2404.02747]] Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion Models(https://arxiv.org/abs/2404.02747)
• Keywords: diffusion
• Abstract: This study explores the role of cross-attention during inference in text-conditional diffusion models. We find that cross-attention outputs converge to a fixed point after few inference steps. Accordingly, the time point of convergence naturally divides the entire inference process into two stages: an initial semantics-planning stage, during which, the model relies on cross-attention to plan text-oriented visual semantics, and a subsequent fidelity-improving stage, during which the model tries to generate images from previously planned semantics. Surprisingly, ignoring text conditions in the fidelity-improving stage not only reduces computation complexity, but also maintains model performance. This yields a simple and training-free method called TGATE for efficient generation, which caches the cross-attention output once it converges and keeps it fixed during the remaining inference steps. Our empirical study on the MS-COCO validation set confirms its effectiveness. The source code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.

Title: DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement

• Authors: Hao Wu, Huabin Liu, Yu Qiao, Xiao Sun
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.02755
• Pdf URL: https://arxiv.org/pdf/2404.02755
• Copy Paste: [[2404.02755]] DIBS: Enhancing Dense Video Captioning with Unlabeled Videos via Pseudo Boundary Enrichment and Online Refinement(https://arxiv.org/abs/2404.02755)
• Keywords: large language model
• Abstract: We present Dive Into the BoundarieS (DIBS), a novel pretraining framework for dense video captioning (DVC), that elaborates on improving the quality of the generated event captions and their associated pseudo event boundaries from unlabeled videos. By leveraging the capabilities of diverse large language models (LLMs), we generate rich DVC-oriented caption candidates and optimize the corresponding pseudo boundaries under several meticulously designed objectives, considering diversity, event-centricity, temporal ordering, and coherence. Moreover, we further introduce a novel online boundary refinement strategy that iteratively improves the quality of pseudo boundaries during training. Comprehensive experiments have been conducted to examine the effectiveness of the proposed technique components. By leveraging a substantial amount of unlabeled video data, such as HowTo100M, we achieve a remarkable advancement on standard DVC datasets like YouCook2 and ActivityNet. We outperform the previous state-of-the-art Vid2Seq across a majority of metrics, achieving this with just 0.4% of the unlabeled video data used for pre-training by Vid2Seq.

Title: FPT: Feature Prompt Tuning for Few-shot Readability Assessment

• Authors: Ziyang Wang, Sanwoo Lee, Hsiu-Yuan Huang, Yunfang Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02772
• Pdf URL: https://arxiv.org/pdf/2404.02772
• Copy Paste: [[2404.02772]] FPT: Feature Prompt Tuning for Few-shot Readability Assessment(https://arxiv.org/abs/2404.02772)
• Keywords: robust, large language model
• Abstract: Prompt-based methods have achieved promising results in most few-shot text classification tasks. However, for readability assessment tasks, traditional prompt methods lackcrucial linguistic knowledge, which has already been proven to be essential. Moreover, previous studies on utilizing linguistic features have shown non-robust performance in few-shot settings and may even impair model performance.To address these issues, we propose a novel prompt-based tuning framework that incorporates rich linguistic knowledge, called Feature Prompt Tuning (FPT). Specifically, we extract linguistic features from the text and embed them into trainable soft prompts. Further, we devise a new loss function to calibrate the similarity ranking order between categories. Experimental results demonstrate that our proposed method FTP not only exhibits a significant performance improvement over the prior best prompt-based tuning approaches, but also surpasses the previous leading methods that incorporate linguistic features. Also, our proposed model significantly outperforms the large language model gpt-3.5-turbo-16k in most cases. Our proposed method establishes a new architecture for prompt tuning that sheds light on how linguistic features can be easily adapted to linguistic-related tasks.

Title: Federated Computing -- Survey on Building Blocks, Extensions and Systems

• Authors: René Schwermer, Ruben Mayer, Hans-Arno Jacobsen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02779
• Pdf URL: https://arxiv.org/pdf/2404.02779
• Copy Paste: [[2404.02779]] Federated Computing -- Survey on Building Blocks, Extensions and Systems(https://arxiv.org/abs/2404.02779)
• Keywords: security, privacy, federate
• Abstract: In response to the increasing volume and sensitivity of data, traditional centralized computing models face challenges, such as data security breaches and regulatory hurdles. Federated Computing (FC) addresses these concerns by enabling collaborative processing without compromising individual data privacy. This is achieved through a decentralized network of devices, each retaining control over its data, while participating in collective computations. The motivation behind FC extends beyond technical considerations to encompass societal implications. As the need for responsible AI and ethical data practices intensifies, FC aligns with the principles of user empowerment and data sovereignty. FC comprises of Federated Learning (FL) and Federated Analytics (FA). FC systems became more complex over time and they currently lack a clear definition and taxonomy describing its moving pieces. Current surveys capture domain-specific FL use cases, describe individual components in an FC pipeline individually or decoupled from each other, or provide a quantitative overview of the number of published papers. This work surveys more than 150 papers to distill the underlying structure of FC systems with their basic building blocks, extensions, architecture, environment, and motivation. We capture FL and FA systems individually and point out unique difference between those two.

Title: Domain Generalization through Meta-Learning: A Survey

• Authors: Arsham Gholamzadeh Khoee, Yinan Yu, Robert Feldt
• Subjects: cs.LG, cs.AI, cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.02785
• Pdf URL: https://arxiv.org/pdf/2404.02785
• Copy Paste: [[2404.02785]] Domain Generalization through Meta-Learning: A Survey(https://arxiv.org/abs/2404.02785)
• Keywords: extraction
• Abstract: Deep neural networks (DNNs) have revolutionized artificial intelligence but often lack performance when faced with out-of-distribution (OOD) data, a common scenario due to the inevitable domain shifts in real-world applications. This limitation stems from the common assumption that training and testing data share the same distribution-an assumption frequently violated in practice. Despite their effectiveness with large amounts of data and computational power, DNNs struggle with distributional shifts and limited labeled data, leading to overfitting and poor generalization across various tasks and domains. Meta-learning presents a promising approach by employing algorithms that acquire transferable knowledge across various tasks for fast adaptation, eliminating the need to learn each task from scratch. This survey paper delves into the realm of meta-learning with a focus on its contribution to domain generalization. We first clarify the concept of meta-learning for domain generalization and introduce a novel taxonomy based on the feature extraction strategy and the classifier learning methodology, offering a granular view of methodologies. Through an exhaustive review of existing methods and underlying theories, we map out the fundamentals of the field. Our survey provides practical insights and an informed discussion on promising research directions, paving the way for future innovation in meta-learning for domain generalization.

Title: GenN2N: Generative NeRF2NeRF Translation

• Authors: Xiangyue Liu, Han Xue, Kunming Luo, Ping Tan, Li Yi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02788
• Pdf URL: https://arxiv.org/pdf/2404.02788
• Copy Paste: [[2404.02788]] GenN2N: Generative NeRF2NeRF Translation(https://arxiv.org/abs/2404.02788)
• Keywords: generative
• Abstract: We present GenN2N, a unified NeRF-to-NeRF translation framework for various NeRF translation tasks such as text-driven NeRF editing, colorization, super-resolution, inpainting, etc. Unlike previous methods designed for individual translation tasks with task-specific schemes, GenN2N achieves all these NeRF editing tasks by employing a plug-and-play image-to-image translator to perform editing in the 2D domain and lifting 2D edits into the 3D NeRF space. Since the 3D consistency of 2D edits may not be assured, we propose to model the distribution of the underlying 3D edits through a generative model that can cover all possible edited NeRFs. To model the distribution of 3D edited NeRFs from 2D edited images, we carefully design a VAE-GAN that encodes images while decoding NeRFs. The latent space is trained to align with a Gaussian distribution and the NeRFs are supervised through an adversarial loss on its renderings. To ensure the latent code does not depend on 2D viewpoints but truly reflects the 3D edits, we also regularize the latent code through a contrastive learning scheme. Extensive experiments on various editing tasks show GenN2N, as a universal framework, performs as well or better than task-specific specialists while possessing flexible generative power. More results on our project page: https://xiangyueliu.github.io/GenN2N/

Title: MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation

• Authors: Petru-Daniel Tudosiu, Yongxin Yang, Shifeng Zhang, Fei Chen, Steven McDonagh, Gerasimos Lampouras, Ignacio Iacobacci, Sarah Parisot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02790
• Pdf URL: https://arxiv.org/pdf/2404.02790
• Copy Paste: [[2404.02790]] MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation(https://arxiv.org/abs/2404.02790)
• Keywords: extraction, generative
• Abstract: Text-to-image generation has achieved astonishing results, yet precise spatial controllability and prompt fidelity remain highly challenging. This limitation is typically addressed through cumbersome prompt engineering, scene layout conditioning, or image editing techniques which often require hand drawn masks. Nonetheless, pre-existing works struggle to take advantage of the natural instance-level compositionality of scenes due to the typically flat nature of rasterized RGB output images. Towards adressing this challenge, we introduce MuLAn: a novel dataset comprising over 44K MUlti-Layer ANnotations of RGB images as multilayer, instance-wise RGBA decompositions, and over 100K instance images. To build MuLAn, we developed a training free pipeline which decomposes a monocular RGB image into a stack of RGBA layers comprising of background and isolated instances. We achieve this through the use of pretrained general-purpose models, and by developing three modules: image decomposition for instance discovery and extraction, instance completion to reconstruct occluded areas, and image re-assembly. We use our pipeline to create MuLAn-COCO and MuLAn-LAION datasets, which contain a variety of image decompositions in terms of style, composition and complexity. With MuLAn, we provide the first photorealistic resource providing instance decomposition and occlusion information for high quality images, opening up new avenues for text-to-image generative AI research. With this, we aim to encourage the development of novel generation and editing technology, in particular layer-wise solutions. MuLAn data resources are available at https://MuLAn-dataset.github.io/.

Title: Generative-Contrastive Heterogeneous Graph Neural Network

• Authors: Yu Wang, Lei Sang, Yi Zhang, Yiwen Zhang
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.02810
• Pdf URL: https://arxiv.org/pdf/2404.02810
• Copy Paste: [[2404.02810]] Generative-Contrastive Heterogeneous Graph Neural Network(https://arxiv.org/abs/2404.02810)
• Keywords: generative
• Abstract: Heterogeneous Graphs (HGs) can effectively model complex relationships in the real world by multi-type nodes and edges. In recent years, inspired by self-supervised learning, contrastive Heterogeneous Graphs Neural Networks (HGNNs) have shown great potential by utilizing data augmentation and discriminators for downstream tasks. However, data augmentation is still limited due to the discrete and abstract nature of graphs. To tackle the above limitations, we propose a novel \textit{Generative-Contrastive Heterogeneous Graph Neural Network (GC-HGNN)}. Specifically, we first propose a heterogeneous graph generative learning enhanced contrastive paradigm. This paradigm includes: 1) A contrastive view augmentation strategy by using masked autoencoder. 2) Position-aware and semantics-aware positive sample sampling strategy for generate hard negative samples. 3) A hierarchical contrastive learning strategy for capturing local and global information. Furthermore, the hierarchical contrastive learning and sampling strategies aim to constitute an enhanced discriminator under the generative-contrastive perspective. Finally, we compare our model with seventeen baselines on eight real-world datasets. Our model outperforms the latest contrastive and generative baselines on node classification and link prediction tasks. To reproduce our work, we have open-sourced our code at https://github.com/xxx.

Title: Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Models

• Authors: Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, Ruohui Huang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02823
• Pdf URL: https://arxiv.org/pdf/2404.02823
• Copy Paste: [[2404.02823]] Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Models(https://arxiv.org/abs/2404.02823)
• Keywords: large language model
• Abstract: The ability of large language models (LLMs) to follow instructions is crucial to real-world applications. Despite recent advances, several studies have highlighted that LLMs struggle when faced with challenging instructions, especially those that include complex constraints, hindering their effectiveness in various tasks. To address this challenge, we introduce Conifer, a novel instruction tuning dataset, designed to enhance LLMs to follow multi-level instructions with complex constraints. Utilizing GPT-4, we curate the dataset by a series of LLM-driven refinement processes to ensure high quality. We also propose a progressive learning scheme that emphasizes an easy-to-hard progression, and learning from process feedback. Models trained with Conifer exhibit remarkable improvements in instruction-following abilities, especially for instructions with complex constraints. On several instruction-following benchmarks, our 7B model outperforms the state-of-the-art open-source 7B models, even exceeds the performance of models 10 times larger on certain metrics. All the code and Conifer dataset are available at https://www.github.com/ConiferLM/Conifer.

Title: BAdam: A Memory Efficient Full Parameter Training Method for Large Language Models

• Authors: Qijun Luo, Hengxu Yu, Xiao Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02827
• Pdf URL: https://arxiv.org/pdf/2404.02827
• Copy Paste: [[2404.02827]] BAdam: A Memory Efficient Full Parameter Training Method for Large Language Models(https://arxiv.org/abs/2404.02827)
• Keywords: large language model
• Abstract: This work presents BAdam, an optimizer that leverages the block coordinate optimization framework with Adam as the inner solver. BAdam offers a memory efficient approach to the full parameter finetuning of large language models and reduces running time of the backward process thanks to the chain rule property. Experimentally, we apply BAdam to instruction-tune the Llama 2-7B model on the Alpaca-GPT4 dataset using a single RTX3090-24GB GPU. The results indicate that BAdam exhibits superior convergence behavior in comparison to LoRA and LOMO. Furthermore, our downstream performance evaluation of the instruction-tuned models using the MT-bench shows that BAdam modestly surpasses LoRA and more substantially outperforms LOMO. Finally, we compare BAdam with Adam on a medium-sized task, i.e., finetuning RoBERTa-large on the SuperGLUE benchmark. The results demonstrate that BAdam is capable of narrowing the performance gap with Adam. Our code is available at https://github.com/Ledzy/BAdam.

Title: Enhancing Interpretability of Vertebrae Fracture Grading using Human-interpretable Prototypes

• Authors: Poulami Sinhamahapatra, Suprosanna Shit, Anjany Sekuboyina, Malek Husseini, David Schinz, Nicolas Lenhart, Joern Menze, Jan Kirschke, Karsten Roscher, Stephan Guennemann
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02830
• Pdf URL: https://arxiv.org/pdf/2404.02830
• Copy Paste: [[2404.02830]] Enhancing Interpretability of Vertebrae Fracture Grading using Human-interpretable Prototypes(https://arxiv.org/abs/2404.02830)
• Keywords: interpretability
• Abstract: Vertebral fracture grading classifies the severity of vertebral fractures, which is a challenging task in medical imaging and has recently attracted Deep Learning (DL) models. Only a few works attempted to make such models human-interpretable despite the need for transparency and trustworthiness in critical use cases like DL-assisted medical diagnosis. Moreover, such models either rely on post-hoc methods or additional annotations. In this work, we propose a novel interpretable-by-design method, ProtoVerse, to find relevant sub-parts of vertebral fractures (prototypes) that reliably explain the model's decision in a human-understandable way. Specifically, we introduce a novel diversity-promoting loss to mitigate prototype repetitions in small datasets with intricate semantics. We have experimented with the VerSe'19 dataset and outperformed the existing prototype-based method. Further, our model provides superior interpretability against the post-hoc method. Importantly, expert radiologists validated the visual interpretability of our results, showing clinical applicability.

Title: Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison

• Authors: Maxime Bouthors, Josep Crego, Francois Yvon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02835
• Pdf URL: https://arxiv.org/pdf/2404.02835
• Copy Paste: [[2404.02835]] Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison(https://arxiv.org/abs/2404.02835)
• Keywords: large language model
• Abstract: Retrieval-Augmented Neural Machine Translation (RAMT) architectures retrieve examples from memory to guide the generation process. While most works in this trend explore new ways to exploit the retrieved examples, the upstream retrieval step is mostly unexplored. In this paper, we study the effect of varying retrieval methods for several translation architectures, to better understand the interplay between these two processes. We conduct experiments in two language pairs in a multi-domain setting and consider several downstream architectures based on a standard autoregressive model, an edit-based model, and a large language model with in-context learning. Our experiments show that the choice of the retrieval technique impacts the translation scores, with variance across architectures. We also discuss the effects of increasing the number and diversity of examples, which are mostly positive across the board.

Title: Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models

• Authors: Wanyun Cui, Qianle Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02837
• Pdf URL: https://arxiv.org/pdf/2404.02837
• Copy Paste: [[2404.02837]] Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models(https://arxiv.org/abs/2404.02837)
• Keywords: large language model
• Abstract: This paper reveals the phenomenon of parameter heterogeneity in large language models (LLMs). We find that a small subset of ``cherry'' parameters exhibit a disproportionately large influence on model performance, while the vast majority of parameters have minimal impact. This heterogeneity is found to be prevalent across different model families, scales, and types. Motivated by this observation, we propose CherryQ, a novel quantization method that unifies the optimization of mixed-precision parameters. CherryQ identifies and preserves the critical cherry parameters in high precision while aggressively quantizing the remaining parameters to low precision. Extensive experiments demonstrate the effectiveness of CherryQ. CherryQ outperforms existing quantization approaches in terms of perplexity and downstream task performance. Notably, our 3-bit quantized Vicuna-1.5 exhibits competitive performance compared to their 16-bit counterparts. These findings highlight the potential of CherryQ for enabling efficient deployment of LLMs by taking advantage of parameter heterogeneity.

Title: Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation

• Authors: Xiaoshuang Huang, Hongxiang Li, Meng Cao, Long Chen, Chenyu You, Dong An
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02845
• Pdf URL: https://arxiv.org/pdf/2404.02845
• Copy Paste: [[2404.02845]] Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation(https://arxiv.org/abs/2404.02845)
• Keywords: segmentation
• Abstract: Recent developments underscore the potential of textual information in enhancing learning models for a deeper understanding of medical visual semantics. However, language-guided medical image segmentation still faces a challenging issue. Previous works employ implicit and ambiguous architectures to embed textual information. This leads to segmentation results that are inconsistent with the semantics represented by the language, sometimes even diverging significantly. To this end, we propose a novel cross-modal conditioned Reconstruction for Language-guided Medical Image Segmentation (RecLMIS) to explicitly capture cross-modal interactions, which assumes that well-aligned medical visual features and medical notes can effectively reconstruct each other. We introduce conditioned interaction to adaptively predict patches and words of interest. Subsequently, they are utilized as conditioning factors for mutual reconstruction to align with regions described in the medical notes. Extensive experiments demonstrate the superiority of our RecLMIS, surpassing LViT by 3.74% mIoU on the publicly available MosMedData+ dataset and achieving an average increase of 1.89% mIoU for cross-domain tests on our QATA-CoV19 dataset. Simultaneously, we achieve a relative reduction of 20.2% in parameter count and a 55.5% decrease in computational load. The code will be available at https://github.com/ShashankHuang/RecLMIS.

Title: Toward Inference-optimal Mixture-of-Expert Large Language Models

• Authors: Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P Xing, Hao Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02852
• Pdf URL: https://arxiv.org/pdf/2404.02852
• Copy Paste: [[2404.02852]] Toward Inference-optimal Mixture-of-Expert Large Language Models(https://arxiv.org/abs/2404.02852)
• Keywords: transformer, large language model
• Abstract: Mixture-of-Expert (MoE) based large language models (LLMs), such as the recent Mixtral and DeepSeek-MoE, have shown great promise in scaling model size without suffering from the quadratic growth of training cost of dense transformers. Like dense models, training MoEs requires answering the same question: given a training budget, what is the optimal allocation on the model size and number of tokens? We study the scaling law of MoE-based LLMs regarding the relations between the model performance, model size, dataset size, and the expert degree. Echoing previous research studying MoE in different contexts, we observe the diminishing return of increasing the number of experts, but this seems to suggest we should scale the number of experts until saturation, as the training cost would remain constant, which is problematic during inference time. We propose to amend the scaling law of MoE by introducing inference efficiency as another metric besides the validation loss. We find that MoEs with a few (4/8) experts are the most serving efficient solution under the same performance, but costs 2.5-3.5x more in training. On the other hand, training a (16/32) expert MoE much smaller (70-85%) than the loss-optimal solution, but with a larger training dataset is a promising setup under a training budget.

Title: Guarantees of confidentiality via Hammersley-Chapman-Robbins bounds

• Authors: Kamalika Chaudhuri, Chuan Guo, Laurens van der Maaten, Saeed Mahloujifar, Mark Tygert
• Subjects: cs.LG, cs.CR, cs.CY, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.02866
• Pdf URL: https://arxiv.org/pdf/2404.02866
• Copy Paste: [[2404.02866]] Guarantees of confidentiality via Hammersley-Chapman-Robbins bounds(https://arxiv.org/abs/2404.02866)
• Keywords: privacy, protect
• Abstract: Protecting privacy during inference with deep neural networks is possible by adding noise to the activations in the last layers prior to the final classifiers or other task-specific layers. The activations in such layers are known as "features" (or, less commonly, as "embeddings" or "feature embeddings"). The added noise helps prevent reconstruction of the inputs from the noisy features. Lower bounding the variance of every possible unbiased estimator of the inputs quantifies the confidentiality arising from such added noise. Convenient, computationally tractable bounds are available from classic inequalities of Hammersley and of Chapman and Robbins -- the HCR bounds. Numerical experiments indicate that the HCR bounds are on the precipice of being effectual for small neural nets with the data sets, "MNIST" and "CIFAR-10," which contain 10 classes each for image classification. The HCR bounds appear to be insufficient on their own to guarantee confidentiality of the inputs to inference with standard deep neural nets, "ResNet-18" and "Swin-T," pre-trained on the data set, "ImageNet-1000," which contains 1000 classes. Supplementing the addition of noise to features with other methods for providing confidentiality may be warranted in the case of ImageNet. In all cases, the results reported here limit consideration to amounts of added noise that incur little degradation in the accuracy of classification from the noisy features. Thus, the added noise enhances confidentiality without much reduction in the accuracy on the task of image classification.

Title: FlightScope: A Deep Comprehensive Assessment of Aircraft Detection Algorithms in Satellite Imagery

• Authors: Safouane El Ghazouali, Arnaud Gucciardi, Nicola Venturi, Michael Rueegsegger, Umberto Michelucci
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02877
• Pdf URL: https://arxiv.org/pdf/2404.02877
• Copy Paste: [[2404.02877]] FlightScope: A Deep Comprehensive Assessment of Aircraft Detection Algorithms in Satellite Imagery(https://arxiv.org/abs/2404.02877)
• Keywords: robust
• Abstract: Object detection in remotely sensed satellite pictures is fundamental in many fields such as biophysical, and environmental monitoring. While deep learning algorithms are constantly evolving, they have been mostly implemented and tested on popular ground-based taken photos. This paper critically evaluates and compares a suite of advanced object detection algorithms customized for the task of identifying aircraft within satellite imagery. Using the large HRPlanesV2 dataset, together with a rigorous validation with the GDIT dataset, this research encompasses an array of methodologies including YOLO versions 5 and 8, Faster RCNN, CenterNet, RetinaNet, RTMDet, and DETR, all trained from scratch. This exhaustive training and validation study reveal YOLOv5 as the preeminent model for the specific case of identifying airplanes from remote sensing data, showcasing high precision and adaptability across diverse imaging conditions. This research highlight the nuanced performance landscapes of these algorithms, with YOLOv5 emerging as a robust solution for aerial object detection, underlining its importance through superior mean average precision, Recall, and Intersection over Union scores. The findings described here underscore the fundamental role of algorithm selection aligned with the specific demands of satellite imagery analysis and extend a comprehensive framework to evaluate model efficacy. The benchmark toolkit and codes, available via https://github.com/toelt-llc/FlightScope_Bench, aims to further exploration and innovation in the realm of remote sensing object detection, paving the way for improved analytical methodologies in satellite imagery applications.

Title: On the Scalability of Diffusion-based Text-to-Image Generation

• Authors: Hao Li, Yang Zou, Ying Wang, Orchid Majumder, Yusheng Xie, R. Manmatha, Ashwin Swaminathan, Zhuowen Tu, Stefano Ermon, Stefano Soatto
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02883
• Pdf URL: https://arxiv.org/pdf/2404.02883
• Copy Paste: [[2404.02883]] On the Scalability of Diffusion-based Text-to-Image Generation(https://arxiv.org/abs/2404.02883)
• Keywords: fair, diffusion, transformer
• Abstract: Scaling up model and data size has been quite successful for the evolution of LLMs. However, the scaling law for the diffusion based text-to-image (T2I) models is not fully explored. It is also unclear how to efficiently scale the model for better performance at reduced cost. The different training settings and expensive training cost make a fair model comparison extremely difficult. In this work, we empirically study the scaling properties of diffusion based T2I models by performing extensive and rigours ablations on scaling both denoising backbones and training set, including training scaled UNet and Transformer variants ranging from 0.4B to 4B parameters on datasets upto 600M images. For model scaling, we find the location and amount of cross attention distinguishes the performance of existing UNet designs. And increasing the transformer blocks is more parameter-efficient for improving text-image alignment than increasing channel numbers. We then identify an efficient UNet variant, which is 45% smaller and 28% faster than SDXL's UNet. On the data scaling side, we show the quality and diversity of the training set matters more than simply dataset size. Increasing caption density and diversity improves text-image alignment performance and the learning efficiency. Finally, we provide scaling functions to predict the text-image alignment performance as functions of the scale of model size, compute and dataset size.

Title: Steganographic Passport: An Owner and User Verifiable Credential for Deep Model IP Protection Without Retraining

• Authors: Qi Cui, Ruohan Meng, Chaohui Xu, Chip-Hong Chang
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02889
• Pdf URL: https://arxiv.org/pdf/2404.02889
• Copy Paste: [[2404.02889]] Steganographic Passport: An Owner and User Verifiable Credential for Deep Model IP Protection Without Retraining(https://arxiv.org/abs/2404.02889)
• Keywords: protect, attack, robust
• Abstract: Ensuring the legal usage of deep models is crucial to promoting trustable, accountable, and responsible artificial intelligence innovation. Current passport-based methods that obfuscate model functionality for license-to-use and ownership verifications suffer from capacity and quality constraints, as they require retraining the owner model for new users. They are also vulnerable to advanced Expanded Residual Block ambiguity attacks. We propose Steganographic Passport, which uses an invertible steganographic network to decouple license-to-use from ownership verification by hiding the user's identity images into the owner-side passport and recovering them from their respective user-side passports. An irreversible and collision-resistant hash function is used to avoid exposing the owner-side passport from the derived user-side passports and increase the uniqueness of the model signature. To safeguard both the passport and model's weights against advanced ambiguity attacks, an activation-level obfuscation is proposed for the verification branch of the owner's model. By jointly training the verification and deployment branches, their weights become tightly coupled. The proposed method supports agile licensing of deep models by providing a strong ownership proof and license accountability without requiring a separate model retraining for the admission of every new user. Experiment results show that our Steganographic Passport outperforms other passport-based deep model protection methods in robustness against various known attacks.

Title: ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline

• Authors: Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, Jie Tang, Yuxiao Dong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02893
• Pdf URL: https://arxiv.org/pdf/2404.02893
• Copy Paste: [[2404.02893]] ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline(https://arxiv.org/abs/2404.02893)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown excellent mastering of human language, but still struggle in real-world applications that require mathematical problem-solving. While many strategies and datasets to enhance LLMs' mathematics are developed, it remains a challenge to simultaneously maintain and improve both language and mathematical capabilities in deployed LLM systems.In this work, we tailor the Self-Critique pipeline, which addresses the challenge in the feedback learning stage of LLM alignment. We first train a general Math-Critique model from the LLM itself to provide feedback signals. Then, we sequentially employ rejective fine-tuning and direct preference optimization over the LLM's own generations for data collection. Based on ChatGLM3-32B, we conduct a series of experiments on both academic and our newly created challenging dataset, MathUserEval. Results show that our pipeline significantly enhances the LLM's mathematical problem-solving while still improving its language ability, outperforming LLMs that could be two times larger. Related techniques have been deployed to ChatGLM\footnote{\url{https://chatglm.cn}}, an online serving LLM. Related evaluation dataset and scripts are released at \url{https://github.com/THUDM/ChatGLM-Math}.

Title: MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignment

• Authors: Duygu Ceylan, Valentin Deschaintre, Thibault Groueix, Rosalie Martin, Chun-Hao Huang, Romain Rouffet, Vladimir Kim, Gaëtan Lassagne
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.02899
• Pdf URL: https://arxiv.org/pdf/2404.02899
• Copy Paste: [[2404.02899]] MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignment(https://arxiv.org/abs/2404.02899)
• Keywords: diffusion, large language model
• Abstract: We present MatAtlas, a method for consistent text-guided 3D model texturing. Following recent progress we leverage a large scale text-to-image generation model (e.g., Stable Diffusion) as a prior to texture a 3D model. We carefully design an RGB texturing pipeline that leverages a grid pattern diffusion, driven by depth and edges. By proposing a multi-step texture refinement process, we significantly improve the quality and 3D consistency of the texturing output. To further address the problem of baked-in lighting, we move beyond RGB colors and pursue assigning parametric materials to the assets. Given the high-quality initial RGB texture, we propose a novel material retrieval method capitalized on Large Language Models (LLM), enabling editabiliy and relightability. We evaluate our method on a wide variety of geometries and show that our method significantly outperform prior arts. We also analyze the role of each component through a detailed ablation study.

Title: DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets

• Authors: Harsh Rangwani, Pradipto Mondal, Mayank Mishra, Ashish Ramayee Asokan, R. Venkatesh Babu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02900
• Pdf URL: https://arxiv.org/pdf/2404.02900
• Copy Paste: [[2404.02900]] DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets(https://arxiv.org/abs/2404.02900)
• Keywords: transformer
• Abstract: Vision Transformer (ViT) has emerged as a prominent architecture for various computer vision tasks. In ViT, we divide the input image into patch tokens and process them through a stack of self attention blocks. However, unlike Convolutional Neural Networks (CNN), ViTs simple architecture has no informative inductive bias (e.g., locality,etc. ). Due to this, ViT requires a large amount of data for pre-training. Various data efficient approaches (DeiT) have been proposed to train ViT on balanced datasets effectively. However, limited literature discusses the use of ViT for datasets with long-tailed imbalances. In this work, we introduce DeiT-LT to tackle the problem of training ViTs from scratch on long-tailed datasets. In DeiT-LT, we introduce an efficient and effective way of distillation from CNN via distillation DIST token by using out-of-distribution images and re-weighting the distillation loss to enhance focus on tail classes. This leads to the learning of local CNN-like features in early ViT blocks, improving generalization for tail classes. Further, to mitigate overfitting, we propose distilling from a flat CNN teacher, which leads to learning low-rank generalizable features for DIST tokens across all ViT blocks. With the proposed DeiT-LT scheme, the distillation DIST token becomes an expert on the tail classes, and the classifier CLS token becomes an expert on the head classes. The experts help to effectively learn features corresponding to both the majority and minority classes using a distinct set of tokens within the same ViT architecture. We show the effectiveness of DeiT-LT for training ViT from scratch on datasets ranging from small-scale CIFAR-10 LT to large-scale iNaturalist-2018.

Title: LidarDM: Generative LiDAR Simulation in a Generated World

• Authors: Vlas Zyrianov, Henry Che, Zhijian Liu, Shenlong Wang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.02903
• Pdf URL: https://arxiv.org/pdf/2404.02903
• Copy Paste: [[2404.02903]] LidarDM: Generative LiDAR Simulation in a Generated World(https://arxiv.org/abs/2404.02903)
• Keywords: diffusion, generative
• Abstract: We present LidarDM, a novel LiDAR generative model capable of producing realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. LidarDM stands out with two unprecedented capabilities in LiDAR generative modeling: (i) LiDAR generation guided by driving scenarios, offering significant potential for autonomous driving simulations, and (ii) 4D LiDAR point cloud generation, enabling the creation of realistic and temporally coherent sequences. At the heart of our model is a novel integrated 4D world generation framework. Specifically, we employ latent diffusion models to generate the 3D scene, combine it with dynamic actors to form the underlying 4D world, and subsequently produce realistic sensory observations within this virtual environment. Our experiments indicate that our approach outperforms competing algorithms in realism, temporal coherency, and layout consistency. We additionally show that LidarDM can be used as a generative world model simulator for training and testing perception models.

Title: ALOHa: A New Measure for Hallucination in Captioning Models

• Authors: Suzanne Petryk, David M. Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph E. Gonzalez, Trevor Darrell
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02904
• Pdf URL: https://arxiv.org/pdf/2404.02904
• Copy Paste: [[2404.02904]] ALOHa: A New Measure for Hallucination in Captioning Models(https://arxiv.org/abs/2404.02904)
• Keywords: large language model
• Abstract: Despite recent advances in multimodal pre-training for visual description, state-of-the-art models still produce captions containing errors, such as hallucinating objects not present in a scene. The existing prominent metric for object hallucination, CHAIR, is limited to a fixed set of MS COCO objects and synonyms. In this work, we propose a modernized open-vocabulary metric, ALOHa, which leverages large language models (LLMs) to measure object hallucinations. Specifically, we use an LLM to extract groundable objects from a candidate caption, measure their semantic similarity to reference objects from captions and object detections, and use Hungarian matching to produce a final hallucination score. We show that ALOHa correctly identifies 13.6% more hallucinated objects than CHAIR on HAT, a new gold-standard subset of MS COCO Captions annotated for hallucinations, and 30.8% more on nocaps, where objects extend beyond MS COCO categories. Our code is available at https://davidmchan.github.io/aloha/.

Title: Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

• Authors: Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02905
• Pdf URL: https://arxiv.org/pdf/2404.02905
• Copy Paste: [[2404.02905]] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction(https://arxiv.org/abs/2404.02905)
• Keywords: diffusion, transformer
• Abstract: We present Visual AutoRegressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine "next-scale prediction" or "next-resolution prediction", diverging from the standard raster-scan "next-token prediction". This simple, intuitive methodology allows autoregressive (AR) transformers to learn visual distributions fast and generalize well: VAR, for the first time, makes AR models surpass diffusion transformers in image generation. On ImageNet 256x256 benchmark, VAR significantly improve AR baseline by improving Frechet inception distance (FID) from 18.65 to 1.80, inception score (IS) from 80.4 to 356.4, with around 20x faster inference speed. It is also empirically verified that VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions including image quality, inference speed, data efficiency, and scalability. Scaling up VAR models exhibits clear power-law scaling laws similar to those observed in LLMs, with linear correlation coefficients near -0.998 as solid evidence. VAR further showcases zero-shot generalization ability in downstream tasks including image in-painting, out-painting, and editing. These results suggest VAR has initially emulated the two important properties of LLMs: Scaling Laws and zero-shot task generalization. We have released all models and codes to promote the exploration of AR/VAR models for visual generation and unified learning.