2024-02-08

Title: Breaking Data Silos: Cross-Domain Learning for Multi-Agent Perception from Independent Private Sources

• Authors: Jinlong Li, Baolu Li, Xinyu Liu, Runsheng Xu, Jiaqi Ma, Hongkai Yu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04273
• Pdf URL: https://arxiv.org/pdf/2402.04273
• Copy Paste: [[2402.04273]] Breaking Data Silos: Cross-Domain Learning for Multi-Agent Perception from Independent Private Sources(https://arxiv.org/abs/2402.04273)
• Keywords: extraction
• Abstract: The diverse agents in multi-agent perception systems may be from different companies. Each company might use the identical classic neural network architecture based encoder for feature extraction. However, the data source to train the various agents is independent and private in each company, leading to the Distribution Gap of different private data for training distinct agents in multi-agent perception system. The data silos by the above Distribution Gap could result in a significant performance decline in multi-agent perception. In this paper, we thoroughly examine the impact of the distribution gap on existing multi-agent perception systems. To break the data silos, we introduce the Feature Distribution-aware Aggregation (FDA) framework for cross-domain learning to mitigate the above Distribution Gap in multi-agent perception. FDA comprises two key components: Learnable Feature Compensation Module and Distribution-aware Statistical Consistency Module, both aimed at enhancing intermediate features to minimize the distribution gap among multi-agent features. Intensive experiments on the public OPV2V and V2XSet datasets underscore FDA's effectiveness in point cloud-based 3D object detection, presenting it as an invaluable augmentation to existing multi-agent perception systems.

Title: CasCast: Skillful High-resolution Precipitation Nowcasting via Cascaded Modelling

• Authors: Junchao Gong, Lei Bai, Peng Ye, Wanghan Xu, Na Liu, Jianhua Dai, Xiaokang Yang, Wanli Ouyang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04290
• Pdf URL: https://arxiv.org/pdf/2402.04290
• Copy Paste: [[2402.04290]] CasCast: Skillful High-resolution Precipitation Nowcasting via Cascaded Modelling(https://arxiv.org/abs/2402.04290)
• Keywords: diffusion, transformer
• Abstract: Precipitation nowcasting based on radar data plays a crucial role in extreme weather prediction and has broad implications for disaster management. Despite progresses have been made based on deep learning, two key challenges of precipitation nowcasting are not well-solved: (i) the modeling of complex precipitation system evolutions with different scales, and (ii) accurate forecasts for extreme precipitation. In this work, we propose CasCast, a cascaded framework composed of a deterministic and a probabilistic part to decouple the predictions for mesoscale precipitation distributions and small-scale patterns. Then, we explore training the cascaded framework at the high resolution and conducting the probabilistic modeling in a low dimensional latent space with a frame-wise-guided diffusion transformer for enhancing the optimization of extreme events while reducing computational costs. Extensive experiments on three benchmark radar precipitation datasets show that CasCast achieves competitive performance. Especially, CasCast significantly surpasses the baseline (up to +91.8%) for regional extreme-precipitation nowcasting.

Title: BiLLM: Pushing the Limit of Post-Training Quantization for LLMs

• Authors: Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, Xiaojuan Qi
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04291
• Pdf URL: https://arxiv.org/pdf/2402.04291
• Copy Paste: [[2402.04291]] BiLLM: Pushing the Limit of Post-Training Quantization for LLMs(https://arxiv.org/abs/2402.04291)
• Keywords: large language model
• Abstract: Pretrained large language models (LLMs) exhibit exceptional general language processing capabilities but come with significant demands on memory and computational resources. As a powerful compression technology, binarization can extremely reduce model weights to a mere 1 bit, lowering the expensive computation and memory requirements. However, existing quantization techniques fall short of maintaining LLM performance under ultra-low bit-widths. In response to this challenge, we present BiLLM, a groundbreaking 1-bit post-training quantization scheme tailored for pretrained LLMs. Based on the weight distribution of LLMs, BiLLM first identifies and structurally selects salient weights, and minimizes the compression loss through an effective binary residual approximation strategy. Moreover, considering the bell-shaped distribution of the non-salient weights, we propose an optimal splitting search to group and binarize them accurately. BiLLM achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant margins. Moreover, BiLLM enables the binarization process of the LLM with 7 billion weights within 0.5 hours on a single GPU, demonstrating satisfactory time efficiency.

Title: AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policies

• Authors: Xixi Hu, Bo Liu, Xingchao Liu, Qiang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04292
• Pdf URL: https://arxiv.org/pdf/2402.04292
• Copy Paste: [[2402.04292]] AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policies(https://arxiv.org/abs/2402.04292)
• Keywords: diffusion, generative
• Abstract: Diffusion-based imitation learning improves Behavioral Cloning (BC) on multi-modal decision-making, but comes at the cost of significantly slower inference due to the recursion in the diffusion process. It urges us to design efficient policy generators while keeping the ability to generate diverse actions. To address this challenge, we propose AdaFlow, an imitation learning framework based on flow-based generative modeling. AdaFlow represents the policy with state-conditioned ordinary differential equations (ODEs), which are known as probability flows. We reveal an intriguing connection between the conditional variance of their training loss and the discretization error of the ODEs. With this insight, we propose a variance-adaptive ODE solver that can adjust its step size in the inference stage, making AdaFlow an adaptive decision-maker, offering rapid inference without sacrificing diversity. Interestingly, it automatically reduces to a one-step generator when the action distribution is uni-modal. Our comprehensive empirical evaluation shows that AdaFlow achieves high performance across all dimensions, including success rate, behavioral diversity, and inference speed. The code is available at https://github.com/hxixixh/AdaFlow

Title: Multi-View Symbolic Regression

• Authors: Etienne Russeil, Fabrício Olivetti de França, Konstantin Malanchev, Bogdan Burlacu, Emille E. O. Ishida, Marion Leroux, Clément Michelin, Guillaume Moinard, Emmanuel Gangler
• Subjects: cs.LG, astro-ph.IM, stat.AP
• Abstract URL: https://arxiv.org/abs/2402.04298
• Pdf URL: https://arxiv.org/pdf/2402.04298
• Copy Paste: [[2402.04298]] Multi-View Symbolic Regression(https://arxiv.org/abs/2402.04298)
• Keywords: robust
• Abstract: Symbolic regression (SR) searches for analytical expressions representing the relationship between a set of explanatory and response variables. Current SR methods assume a single dataset extracted from a single experiment. Nevertheless, frequently, the researcher is confronted with multiple sets of results obtained from experiments conducted with different setups. Traditional SR methods may fail to find the underlying expression since the parameters of each experiment can be different. In this work we present Multi-View Symbolic Regression (MvSR), which takes into account multiple datasets simultaneously, mimicking experimental environments, and outputs a general parametric solution. This approach fits the evaluated expression to each independent dataset and returns a parametric family of functions f(x; \theta) simultaneously capable of accurately fitting all datasets. We demonstrate the effectiveness of MvSR using data generated from known expressions, as well as real-world data from astronomy, chemistry and economy, for which an a priori analytical expression is not available. Results show that MvSR obtains the correct expression more frequently and is robust to hyperparameters change. In real-world data, it is able to grasp the group behaviour, recovering known expressions from the literature as well as promising alternatives, thus enabling the use SR to a large range of experimental scenarios.

Title: Training Language Models to Generate Text with Citations via Fine-grained Rewards

• Authors: Chengyu Huang, Zeqiu Wu, Yushi Hu, Wenya Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04315
• Pdf URL: https://arxiv.org/pdf/2402.04315
• Copy Paste: [[2402.04315]] Training Language Models to Generate Text with Citations via Fine-grained Rewards(https://arxiv.org/abs/2402.04315)
• Keywords: large language model
• Abstract: While recent Large Language Models (LLMs) have proven useful in answering user queries, they are prone to hallucination, and their responses often lack credibility due to missing references to reliable sources. An intuitive solution to these issues would be to include in-text citations referring to external documents as evidence. While previous works have directly prompted LLMs to generate in-text citations, their performances are far from satisfactory, especially when it comes to smaller LLMs. In this work, we propose an effective training framework using fine-grained rewards to teach LLMs to generate highly supportive and relevant citations, while ensuring the correctness of their responses. We also conduct a systematic analysis of applying these fine-grained rewards to common LLM training strategies, demonstrating its advantage over conventional practices. We conduct extensive experiments on Question Answering (QA) datasets taken from the ALCE benchmark and validate the model's generalizability using EXPERTQA. On LLaMA-2-7B, the incorporation of fine-grained rewards achieves the best performance among the baselines, even surpassing that of GPT-3.5-turbo.

Title: ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation

• Authors: Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, Wenhu Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04324
• Pdf URL: https://arxiv.org/pdf/2402.04324
• Copy Paste: [[2402.04324]] ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation(https://arxiv.org/abs/2402.04324)
• Keywords: diffusion
• Abstract: Image-to-video (I2V) generation aims to use the initial frame (alongside a text prompt) to create a video sequence. A grand challenge in I2V generation is to maintain visual consistency throughout the video: existing methods often struggle to preserve the integrity of the subject, background, and style from the first frame, as well as ensure a fluid and logical progression within the video narrative. To mitigate these issues, we propose ConsistI2V, a diffusion-based method to enhance visual consistency for I2V generation. Specifically, we introduce (1) spatiotemporal attention over the first frame to maintain spatial and motion consistency, (2) noise initialization from the low-frequency band of the first frame to enhance layout consistency. These two approaches enable ConsistI2V to generate highly consistent videos. We also extend the proposed approaches to show their potential to improve consistency in auto-regressive long video generation and camera motion control. To verify the effectiveness of our method, we propose I2V-Bench, a comprehensive evaluation benchmark for I2V generation. Our automatic and human evaluation results demonstrate the superiority of ConsistI2V over existing methods.

Title: Enhance DNN Adversarial Robustness and Efficiency via Injecting Noise to Non-Essential Neurons

• Authors: Zhenyu Liu, Garrett Gagnon, Swagath Venkataramani, Liu Liu
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.04325
• Pdf URL: https://arxiv.org/pdf/2402.04325
• Copy Paste: [[2402.04325]] Enhance DNN Adversarial Robustness and Efficiency via Injecting Noise to Non-Essential Neurons(https://arxiv.org/abs/2402.04325)
• Keywords: protect, attack, robust
• Abstract: Deep Neural Networks (DNNs) have revolutionized a wide range of industries, from healthcare and finance to automotive, by offering unparalleled capabilities in data analysis and decision-making. Despite their transforming impact, DNNs face two critical challenges: the vulnerability to adversarial attacks and the increasing computational costs associated with more complex and larger models. In this paper, we introduce an effective method designed to simultaneously enhance adversarial robustness and execution efficiency. Unlike prior studies that enhance robustness via uniformly injecting noise, we introduce a non-uniform noise injection algorithm, strategically applied at each DNN layer to disrupt adversarial perturbations introduced in attacks. By employing approximation techniques, our approach identifies and protects essential neurons while strategically introducing noise into non-essential neurons. Our experimental results demonstrate that our method successfully enhances both robustness and efficiency across several attack scenarios, model architectures, and datasets.

Title: LESS: Selecting Influential Data for Targeted Instruction Tuning

• Authors: Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, Danqi Chen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04333
• Pdf URL: https://arxiv.org/pdf/2402.04333
• Copy Paste: [[2402.04333]] LESS: Selecting Influential Data for Targeted Instruction Tuning(https://arxiv.org/abs/2402.04333)
• Keywords: large language model
• Abstract: Instruction tuning has unlocked powerful capabilities in large language models (LLMs), effectively using combined datasets to develop generalpurpose chatbots. However, real-world applications often require a specialized suite of skills (e.g., reasoning). The challenge lies in identifying the most relevant data from these extensive datasets to effectively develop specific capabilities, a setting we frame as targeted instruction tuning. We propose LESS, an optimizer-aware and practically efficient algorithm to effectively estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. Crucially, LESS adapts existing influence formulations to work with the Adam optimizer and variable-length instruction data. LESS first constructs a highly reusable and transferable gradient datastore with low-dimensional gradient features and then selects examples based on their similarity to few-shot examples embodying a specific capability. Experiments show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Furthermore, the selected data is highly transferable: smaller models can be leveraged to select useful data for larger models and models from different families. Our qualitative analysis shows that our method goes beyond surface form cues to identify data that exemplifies the necessary reasoning skills for the intended downstream application.

Title: LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text

• Authors: Dor Bernsohn, Gil Semo, Yaron Vazana, Gila Hayat, Ben Hagag, Joel Niklaus, Rohit Saha, Kyryl Truskovskyi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04335
• Pdf URL: https://arxiv.org/pdf/2402.04335
• Copy Paste: [[2402.04335]] LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text(https://arxiv.org/abs/2402.04335)
• Keywords: large language model
• Abstract: In this study, we focus on two main tasks, the first for detecting legal violations within unstructured textual data, and the second for associating these violations with potentially affected individuals. We constructed two datasets using Large Language Models (LLMs) which were subsequently validated by domain expert annotators. Both tasks were designed specifically for the context of class-action cases. The experimental design incorporated fine-tuning models from the BERT family and open-source LLMs, and conducting few-shot experiments using closed-source LLMs. Our results, with an F1-score of 62.69\% (violation identification) and 81.02\% (associating victims), show that our datasets and setups can be used for both tasks. Finally, we publicly release the datasets and the code used for the experiments in order to advance further research in the area of legal natural language processing (NLP).

Title: The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry

• Authors: Michael Zhang, Kush Bhatia, Hermann Kumbong, Christopher Ré
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04347
• Pdf URL: https://arxiv.org/pdf/2402.04347
• Copy Paste: [[2402.04347]] The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry(https://arxiv.org/abs/2402.04347)
• Keywords: transformer, large language model
• Abstract: Linear attentions have shown potential for improving Transformer efficiency, reducing attention's quadratic complexity to linear in sequence length. This holds exciting promise for (1) training linear Transformers from scratch, (2) "finetuned-conversion" of task-specific Transformers into linear versions that recover task performance, and (3) "pretrained-conversion" of Transformers such as large language models into linear versions finetunable on downstream tasks. However, linear attentions often underperform standard softmax attention in quality. To close this performance gap, we find prior linear attentions lack key properties of softmax attention tied to good performance: low-entropy (or "spiky") weights and dot-product monotonicity. We further observe surprisingly simple feature maps that retain these properties and match softmax performance, but are inefficient to compute in linear attention. We thus propose Hedgehog, a learnable linear attention that retains the spiky and monotonic properties of softmax attention while maintaining linear complexity. Hedgehog uses simple trainable MLPs to produce attention weights mimicking softmax attention. Experiments show Hedgehog recovers over 99% of standard Transformer quality in train-from-scratch and finetuned-conversion settings, outperforming prior linear attentions up to 6 perplexity points on WikiText-103 with causal GPTs, and up to 8.7 GLUE score points on finetuned bidirectional BERTs. Hedgehog also enables pretrained-conversion. Converting a pretrained GPT-2 into a linear attention variant achieves state-of-the-art 16.7 perplexity on WikiText-103 for 125M subquadratic decoder models. We finally turn a pretrained Llama-2 7B into a viable linear attention Llama. With low-rank adaptation, Hedgehog-Llama2 7B achieves 28.1 higher ROUGE-1 points over the base standard attention model, where prior linear attentions lead to 16.5 point drops.

Title: Merkle Trees in Blockchain: A Study of Collision Probability and Security Implications

• Authors: Oleksandr Kuznetsov, Alex Rusnak, Anton Yezhov, Kateryna Kuznetsova, Dzianis Kanonik, Oleksandr Domin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.04367
• Pdf URL: https://arxiv.org/pdf/2402.04367
• Copy Paste: [[2402.04367]] Merkle Trees in Blockchain: A Study of Collision Probability and Security Implications(https://arxiv.org/abs/2402.04367)
• Keywords: security, attack, robust
• Abstract: In the rapidly evolving landscape of blockchain technology, ensuring the integrity and security of data is paramount. This study delves into the security aspects of Merkle Trees, a fundamental component in blockchain architectures, such as Ethereum. We critically examine the susceptibility of Merkle Trees to hash collisions, a potential vulnerability that poses significant risks to data security within blockchain systems. Despite their widespread application, the collision resistance of Merkle Trees and their robustness against preimage attacks have not been thoroughly investigated, leading to a notable gap in the comprehensive understanding of blockchain security mechanisms. Our research endeavors to bridge this gap through a meticulous blend of theoretical analysis and empirical validation. We scrutinize the probability of root collisions in Merkle Trees, considering various factors such as hash length and path length within the tree. Our findings reveal a direct correlation between the increase in path length and the heightened probability of root collisions, thereby underscoring potential security vulnerabilities. Conversely, we observe that an increase in hash length significantly reduces the likelihood of collisions, highlighting its critical role in fortifying security. The insights garnered from our research offer valuable guidance for blockchain developers and researchers, aiming to bolster the security and operational efficacy of blockchain-based systems.

Title: Scaling laws for learning with real and surrogate data

• Authors: Ayush Jain, Andrea Montanari, Eren Sasoglu
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.04376
• Pdf URL: https://arxiv.org/pdf/2402.04376
• Copy Paste: [[2402.04376]] Scaling laws for learning with real and surrogate data(https://arxiv.org/abs/2402.04376)
• Keywords: generative
• Abstract: Collecting large quantities of high-quality data is often prohibitively expensive or impractical, and a crucial bottleneck in machine learning. One may instead augment a small set of $n$ data points from the target distribution with data from more accessible sources like public datasets, data collected under different circumstances, or synthesized by generative models. Blurring distinctions, we refer to such data as `surrogate data'. We define a simple scheme for integrating surrogate data into training and use both theoretical models and empirical studies to explore its behavior. Our main findings are: $(i)$ Integrating surrogate data can significantly reduce the test error on the original distribution; $(ii)$ In order to reap this benefit, it is crucial to use optimally weighted empirical risk minimization; $(iii)$ The test error of models trained on mixtures of real and surrogate data is well described by a scaling law. This can be used to predict the optimal weighting and the gain from surrogate data.

Title: $\texttt{NeRCC}$: Nested-Regression Coded Computing for Resilient Distributed Prediction Serving Systems

• Authors: Parsa Moradi, Mohammad Ali Maddah-Ali
• Subjects: cs.LG, cs.DC, cs.IT
• Abstract URL: https://arxiv.org/abs/2402.04377
• Pdf URL: https://arxiv.org/pdf/2402.04377
• Copy Paste: [[2402.04377]] $\texttt{NeRCC}$: Nested-Regression Coded Computing for Resilient Distributed Prediction Serving Systems(https://arxiv.org/abs/2402.04377)
• Keywords: transformer
• Abstract: Resilience against stragglers is a critical element of prediction serving systems, tasked with executing inferences on input data for a pre-trained machine-learning model. In this paper, we propose NeRCC, as a general straggler-resistant framework for approximate coded computing. NeRCC includes three layers: (1) encoding regression and sampling, which generates coded data points, as a combination of original data points, (2) computing, in which a cluster of workers run inference on the coded data points, (3) decoding regression and sampling, which approximately recovers the predictions of the original data points from the available predictions on the coded data points. We argue that the overall objective of the framework reveals an underlying interconnection between two regression models in the encoding and decoding layers. We propose a solution to the nested regressions problem by summarizing their dependence on two regularization terms that are jointly optimized. Our extensive experiments on different datasets and various machine learning models, including LeNet5, RepVGG, and Vision Transformer (ViT), demonstrate that NeRCC accurately approximates the original predictions in a wide range of stragglers, outperforming the state-of-the-art by up to 23%.

Title: Fine-Tuned Language Models Generate Stable Inorganic Materials as Text

• Authors: Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C. Lawrence Zitnick, Zachary Ulissi
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2402.04379
• Pdf URL: https://arxiv.org/pdf/2402.04379
• Copy Paste: [[2402.04379]] Fine-Tuned Language Models Generate Stable Inorganic Materials as Text(https://arxiv.org/abs/2402.04379)
• Keywords: diffusion, large language model
• Abstract: We propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable, with around 90% of sampled structures obeying physical constraints on atom positions and charges. Using energy above hull calculations from both learned ML potentials and gold-standard DFT calculations, we show that our strongest model (fine-tuned LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.

Title: FairWire: Fair Graph Generation

• Authors: O. Deniz Kose, Yanning Shen
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.04383
• Pdf URL: https://arxiv.org/pdf/2402.04383
• Copy Paste: [[2402.04383]] FairWire: Fair Graph Generation(https://arxiv.org/abs/2402.04383)
• Keywords: privacy, fair, generative
• Abstract: Machine learning over graphs has recently attracted growing attention due to its ability to analyze and learn complex relations within critical interconnected systems. However, the disparate impact that is amplified by the use of biased graph structures in these algorithms has raised significant concerns for the deployment of them in real-world decision systems. In addition, while synthetic graph generation has become pivotal for privacy and scalability considerations, the impact of generative learning algorithms on the structural bias has not yet been investigated. Motivated by this, this work focuses on the analysis and mitigation of structural bias for both real and synthetic graphs. Specifically, we first theoretically analyze the sources of structural bias that result in disparity for the predictions of dyadic relations. To alleviate the identified bias factors, we design a novel fairness regularizer that offers a versatile use. Faced with the bias amplification in graph generation models that is brought to light in this work, we further propose a fair graph generation framework, FairWire, by leveraging our fair regularizer design in a generative model. Experimental results on real-world networks validate that the proposed tools herein deliver effective structural bias mitigation for both real and synthetic graphs.

Title: Denoising Diffusion Probabilistic Models in Six Simple Steps

• Authors: Richard E. Turner, Cristiana-Diana Diaconu, Stratis Markou, Aliaksandra Shysheya, Andrew Y. K. Foong, Bruno Mlodozeniec
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.04384
• Pdf URL: https://arxiv.org/pdf/2402.04384
• Copy Paste: [[2402.04384]] Denoising Diffusion Probabilistic Models in Six Simple Steps(https://arxiv.org/abs/2402.04384)
• Keywords: diffusion, generative
• Abstract: Denoising Diffusion Probabilistic Models (DDPMs) are a very popular class of deep generative model that have been successfully applied to a diverse range of problems including image and video generation, protein and material synthesis, weather forecasting, and neural surrogates of partial differential equations. Despite their ubiquity it is hard to find an introduction to DDPMs which is simple, comprehensive, clean and clear. The compact explanations necessary in research papers are not able to elucidate all of the different design steps taken to formulate the DDPM and the rationale of the steps that are presented is often omitted to save space. Moreover, the expositions are typically presented from the variational lower bound perspective which is unnecessary and arguably harmful as it obfuscates why the method is working and suggests generalisations that do not perform well in practice. On the other hand, perspectives that take the continuous time-limit are beautiful and general, but they have a high barrier-to-entry as they require background knowledge of stochastic differential equations and probability flow. In this note, we distill down the formulation of the DDPM into six simple steps each of which comes with a clear rationale. We assume that the reader is familiar with fundamental topics in machine learning including basic probabilistic modelling, Gaussian distributions, maximum likelihood estimation, and deep learning.

Title: CEHR-GPT: Generating Electronic Health Records with Chronological Patient Timelines

• Authors: Chao Pang, Xinzhuo Jiang, Nishanth Parameshwar Pavinkurve, Krishna S. Kalluri, Elise L. Minto, Jason Patterson, Linying Zhang, George Hripcsak, Noémie Elhadad, Karthik Natarajan
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.04400
• Pdf URL: https://arxiv.org/pdf/2402.04400
• Copy Paste: [[2402.04400]] CEHR-GPT: Generating Electronic Health Records with Chronological Patient Timelines(https://arxiv.org/abs/2402.04400)
• Keywords: transformer, generative
• Abstract: Synthetic Electronic Health Records (EHR) have emerged as a pivotal tool in advancing healthcare applications and machine learning models, particularly for researchers without direct access to healthcare data. Although existing methods, like rule-based approaches and generative adversarial networks (GANs), generate synthetic data that resembles real-world EHR data, these methods often use a tabular format, disregarding temporal dependencies in patient histories and limiting data replication. Recently, there has been a growing interest in leveraging Generative Pre-trained Transformers (GPT) for EHR data. This enables applications like disease progression analysis, population estimation, counterfactual reasoning, and synthetic data generation. In this work, we focus on synthetic data generation and demonstrate the capability of training a GPT model using a particular patient representation derived from CEHR-BERT, enabling us to generate patient sequences that can be seamlessly converted to the Observational Medical Outcomes Partnership (OMOP) data format.

Title: Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning

• Authors: Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, Meng Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04401
• Pdf URL: https://arxiv.org/pdf/2402.04401
• Copy Paste: [[2402.04401]] Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning(https://arxiv.org/abs/2402.04401)
• Keywords: privacy, robust, large language model
• Abstract: Personalization in large language models (LLMs) is increasingly important, aiming to align LLM's interactions, content, and recommendations with individual user preferences. Recent advances in LLM personalization have spotlighted effective prompt design, by enriching user queries with non-parametric knowledge through behavior history retrieval and textual profiles. However, these approaches were limited due to a lack of model ownership, resulting in constrained customization and privacy issues. Moreover, they often failed to accurately capture user behavior patterns, especially in cases where user data were complex and dynamic. To address these shortcomings, we introduce One PEFT Per User (OPPU), which employs personalized parameter-efficient fine-tuning (PEFT) modules, to store user-specific behavior patterns and preferences. By plugging in users' personal PEFT parameters, they can own and use their LLMs personally. OPPU integrates parametric user knowledge in the personal PEFT parameters with the non-parametric knowledge acquired through retrieval and profile. This integration adapts individual LLMs to user behavior shifts. Experimental results demonstrate that OPPU significantly outperforms existing prompt-based methods across seven diverse tasks in the LaMP benchmark. Further in-depth studies reveal OPPU's enhanced capabilities in handling user behavior shifts, modeling users at different active levels, maintaining robustness across various user history formats, and displaying versatility with different PEFT methods.

Title: Detection Transformer for Teeth Detection, Segmentation, and Numbering in Oral Rare Diseases: Focus on Data Augmentation and Inpainting Techniques

• Authors: Hocine Kadi, Théo Sourget, Marzena Kawczynski, Sara Bendjama, Bruno Grollemund, Agnès Bloch-Zupan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04408
• Pdf URL: https://arxiv.org/pdf/2402.04408
• Copy Paste: [[2402.04408]] Detection Transformer for Teeth Detection, Segmentation, and Numbering in Oral Rare Diseases: Focus on Data Augmentation and Inpainting Techniques(https://arxiv.org/abs/2402.04408)
• Keywords: diffusion, transformer, segmentation
• Abstract: In this work, we focused on deep learning image processing in the context of oral rare diseases, which pose challenges due to limited data availability. A crucial step involves teeth detection, segmentation and numbering in panoramic radiographs. To this end, we used a dataset consisting of 156 panoramic radiographs from individuals with rare oral diseases and labeled by experts. We trained the Detection Transformer (DETR) neural network for teeth detection, segmentation, and numbering the 52 teeth classes. In addition, we used data augmentation techniques, including geometric transformations. Finally, we generated new panoramic images using inpainting techniques with stable diffusion, by removing teeth from a panoramic radiograph and integrating teeth into it. The results showed a mAP exceeding 0,69 for DETR without data augmentation. The mAP was improved to 0,82 when data augmentation techniques are used. Furthermore, we observed promising performances when using new panoramic radiographs generated with inpainting technique, with mAP of 0,76.

Title: Towards Fair, Robust and Efficient Client Contribution Evaluation in Federated Learning

• Authors: Meiying Zhang, Huan Zhao, Sheldon Ebron, Kan Yang
• Subjects: cs.LG, cs.AI, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.04409
• Pdf URL: https://arxiv.org/pdf/2402.04409
• Copy Paste: [[2402.04409]] Towards Fair, Robust and Efficient Client Contribution Evaluation in Federated Learning(https://arxiv.org/abs/2402.04409)
• Keywords: attack, robust, federate, fair
• Abstract: The performance of clients in Federated Learning (FL) can vary due to various reasons. Assessing the contributions of each client is crucial for client selection and compensation. It is challenging because clients often have non-independent and identically distributed (non-iid) data, leading to potentially noisy or divergent updates. The risk of malicious clients amplifies the challenge especially when there's no access to clients' local data or a benchmark root dataset. In this paper, we introduce a novel method called Fair, Robust, and Efficient Client Assessment (FRECA) for quantifying client contributions in FL. FRECA employs a framework called FedTruth to estimate the global model's ground truth update, balancing contributions from all clients while filtering out impacts from malicious ones. This approach is robust against Byzantine attacks and incorporates a Byzantine-resilient aggregation algorithm. FRECA is also efficient, as it operates solely on local model updates and requires no validation operations or datasets. Our experimental results show that FRECA can accurately and efficiently quantify client contributions in a robust manner.

Title: Chatbot Meets Pipeline: Augment Large Language Model with Definite Finite Automaton

• Authors: Yiyou Sun, Junjie Hu, Wei Cheng, Haifeng Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04411
• Pdf URL: https://arxiv.org/pdf/2402.04411
• Copy Paste: [[2402.04411]] Chatbot Meets Pipeline: Augment Large Language Model with Definite Finite Automaton(https://arxiv.org/abs/2402.04411)
• Keywords: large language model
• Abstract: This paper introduces the Definite Finite Automaton augmented large language model (DFA-LLM), a novel framework designed to enhance the capabilities of conversational agents using large language models (LLMs). Traditional LLMs face challenges in generating regulated and compliant responses in special scenarios with predetermined response guidelines, like emotional support and customer service. Our framework addresses these challenges by embedding a Definite Finite Automaton (DFA), learned from training dialogues, within the LLM. This structured approach enables the LLM to adhere to a deterministic response pathway, guided by the DFA. The advantages of DFA-LLM include an interpretable structure through human-readable DFA, context-aware retrieval for responses in conversations, and plug-and-play compatibility with existing LLMs. Extensive benchmarks validate DFA-LLM's effectiveness, indicating its potential as a valuable contribution to the conversational agent.

Title: Decentralized Blockchain-based Robust Multi-agent Multi-armed Bandit

• Authors: Mengfan Xu, Diego Klabjan
• Subjects: cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2402.04417
• Pdf URL: https://arxiv.org/pdf/2402.04417
• Copy Paste: [[2402.04417]] Decentralized Blockchain-based Robust Multi-agent Multi-armed Bandit(https://arxiv.org/abs/2402.04417)
• Keywords: secure, privacy, attack, robust, federate
• Abstract: We study a robust multi-agent multi-armed bandit problem where multiple clients or participants are distributed on a fully decentralized blockchain, with the possibility of some being malicious. The rewards of arms are homogeneous among the clients, following time-invariant stochastic distributions that are revealed to the participants only when the system is secure enough. The system's objective is to efficiently ensure the cumulative rewards gained by the honest participants. To this end and to the best of our knowledge, we are the first to incorporate advanced techniques from blockchains, as well as novel mechanisms, into the system to design optimal strategies for honest participants. This allows various malicious behaviors and the maintenance of participant privacy. More specifically, we randomly select a pool of validators who have access to all participants, design a brand-new consensus mechanism based on digital signatures for these validators, invent a UCB-based strategy that requires less information from participants through secure multi-party computation, and design the chain-participant interaction and an incentive mechanism to encourage participants' participation. Notably, we are the first to prove the theoretical guarantee of the proposed algorithms by regret analyses in the context of optimality in blockchains. Unlike existing work that integrates blockchains with learning problems such as federated learning which mainly focuses on numerical optimality, we demonstrate that the regret of honest participants is upper bounded by $log{T}$. This is consistent with the multi-agent multi-armed bandit problem without malicious participants and the robust multi-agent multi-armed bandit problem with purely Byzantine attacks.

Title: PreGIP: Watermarking the Pretraining of Graph Neural Networks for Deep Intellectual Property Protection

• Authors: Enyan Dai, Minhua Lin, Suhang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04435
• Pdf URL: https://arxiv.org/pdf/2402.04435
• Copy Paste: [[2402.04435]] PreGIP: Watermarking the Pretraining of Graph Neural Networks for Deep Intellectual Property Protection(https://arxiv.org/abs/2402.04435)
• Keywords: protect, watermark
• Abstract: Pretraining on Graph Neural Networks (GNNs) has shown great power in facilitating various downstream tasks. As pretraining generally requires huge amount of data and computational resources, the pretrained GNNs are high-value Intellectual Properties (IP) of the legitimate owner. However, adversaries may illegally copy and deploy the pretrained GNN models for their downstream tasks. Though initial efforts have been made to watermark GNN classifiers for IP protection, these methods require the target classification task for watermarking, and thus are not applicable to self-supervised pretraining of GNN models. Hence, in this work, we propose a novel framework named PreGIP to watermark the pretraining of GNN encoder for IP protection while maintain the high-quality of the embedding space. PreGIP incorporates a task-free watermarking loss to watermark the embedding space of pretrained GNN encoder. A finetuning-resistant watermark injection is further deployed. Theoretical analysis and extensive experiments show the effectiveness of {\method} in IP protection and maintaining high-performance for downstream tasks.

Title: Structured Entity Extraction Using Large Language Models

• Authors: Haolun Wu, Ye Yuan, Liana Mikaelyan, Alexander Meulemans, Xue Liu, James Hensman, Bhaskar Mitra
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04437
• Pdf URL: https://arxiv.org/pdf/2402.04437
• Copy Paste: [[2402.04437]] Structured Entity Extraction Using Large Language Models(https://arxiv.org/abs/2402.04437)
• Keywords: extraction, large language model
• Abstract: Recent advances in machine learning have significantly impacted the field of information extraction, with Large Language Models (LLMs) playing a pivotal role in extracting structured information from unstructured text. This paper explores the challenges and limitations of current methodologies in structured entity extraction and introduces a novel approach to address these issues. We contribute to the field by first introducing and formalizing the task of Structured Entity Extraction (SEE), followed by proposing Approximate Entity Set OverlaP (AESOP) Metric designed to appropriately assess model performance on this task. Later, we propose a new model that harnesses the power of LLMs for enhanced effectiveness and efficiency through decomposing the entire extraction task into multiple stages. Quantitative evaluation and human side-by-side evaluation confirm that our model outperforms baselines, offering promising directions for future advancements in structured entity extraction.

Title: Detecting Mode Collapse in Language Models via Narration

• Authors: Sil Hamilton
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04477
• Pdf URL: https://arxiv.org/pdf/2402.04477
• Copy Paste: [[2402.04477]] Detecting Mode Collapse in Language Models via Narration(https://arxiv.org/abs/2402.04477)
• Keywords: large language model
• Abstract: No two authors write alike. Personal flourishes invoked in written narratives, from lexicon to rhetorical devices, imply a particular author--what literary theorists label the implied or virtual author; distinct from the real author or narrator of a text. Early large language models trained on unfiltered training sets drawn from a variety of discordant sources yielded incoherent personalities, problematic for conversational tasks but proving useful for sampling literature from multiple perspectives. Successes in alignment research in recent years have allowed researchers to impose subjectively consistent personae on language models via instruction tuning and reinforcement learning from human feedback (RLHF), but whether aligned models retain the ability to model an arbitrary virtual author has received little scrutiny. By studying 4,374 stories sampled from three OpenAI language models, we show successive versions of GPT-3 suffer from increasing degrees of "mode collapse" whereby overfitting the model during alignment constrains it from generalizing over authorship: models suffering from mode collapse become unable to assume a multiplicity of perspectives. Our method and results are significant for researchers seeking to employ language models in sociological simulations.

Title: Incentivized Truthful Communication for Federated Bandits

• Authors: Zhepei Wei, Chuanhao Li, Tianze Ren, Haifeng Xu, Hongning Wang
• Subjects: cs.LG, cs.GT
• Abstract URL: https://arxiv.org/abs/2402.04485
• Pdf URL: https://arxiv.org/pdf/2402.04485
• Copy Paste: [[2402.04485]] Incentivized Truthful Communication for Federated Bandits(https://arxiv.org/abs/2402.04485)
• Keywords: federate
• Abstract: To enhance the efficiency and practicality of federated bandit learning, recent advances have introduced incentives to motivate communication among clients, where a client participates only when the incentive offered by the server outweighs its participation cost. However, existing incentive mechanisms naively assume the clients are truthful: they all report their true cost and thus the higher cost one participating client claims, the more the server has to pay. Therefore, such mechanisms are vulnerable to strategic clients aiming to optimize their own utility by misreporting. To address this issue, we propose an incentive compatible (i.e., truthful) communication protocol, named Truth-FedBan, where the incentive for each participant is independent of its self-reported cost, and reporting the true cost is the only way to achieve the best utility. More importantly, Truth-FedBan still guarantees the sub-linear regret and communication cost without any overheads. In other words, the core conceptual contribution of this paper is, for the first time, demonstrating the possibility of simultaneously achieving incentive compatibility and nearly optimal regret in federated bandit learning. Extensive numerical studies further validate the effectiveness of our proposed solution.

Title: De-amplifying Bias from Differential Privacy in Language Model Fine-tuning

• Authors: Sanjari Srivastava, Piotr Mardziel, Zhikhun Zhang, Archana Ahlawat, Anupam Datta, John C Mitchell
• Subjects: cs.LG, cs.CR, cs.CY, stat.ME
• Abstract URL: https://arxiv.org/abs/2402.04489
• Pdf URL: https://arxiv.org/pdf/2402.04489
• Copy Paste: [[2402.04489]] De-amplifying Bias from Differential Privacy in Language Model Fine-tuning(https://arxiv.org/abs/2402.04489)
• Keywords: privacy, fair, large language model
• Abstract: Fairness and privacy are two important values machine learning (ML) practitioners often seek to operationalize in models. Fairness aims to reduce model bias for social/demographic sub-groups. Privacy via differential privacy (DP) mechanisms, on the other hand, limits the impact of any individual's training data on the resulting model. The trade-offs between privacy and fairness goals of trustworthy ML pose a challenge to those wishing to address both. We show that DP amplifies gender, racial, and religious bias when fine-tuning large language models (LLMs), producing models more biased than ones fine-tuned without DP. We find the cause of the amplification to be a disparity in convergence of gradients across sub-groups. Through the case of binary gender bias, we demonstrate that Counterfactual Data Augmentation (CDA), a known method for addressing bias, also mitigates bias amplification by DP. As a consequence, DP and CDA together can be used to fine-tune models while maintaining both fairness and privacy.

Title: ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation

• Authors: Jirayu Burapacheep, Ishan Gaur, Agam Bhatia, Tristan Thrush
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04492
• Pdf URL: https://arxiv.org/pdf/2402.04492
• Copy Paste: [[2402.04492]] ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation(https://arxiv.org/abs/2402.04492)
• Keywords: robust
• Abstract: This paper introduces the ColorSwap dataset, designed to assess and improve the proficiency of multimodal models in matching objects with their colors. The dataset is comprised of 2,000 unique image-caption pairs, grouped into 1,000 examples. Each example includes a caption-image pair, along with a ``color-swapped'' pair. We follow the Winoground schema: the two captions in an example have the same words, but the color words have been rearranged to modify different objects. The dataset was created through a novel blend of automated caption and image generation with humans in the loop. We evaluate image-text matching (ITM) and visual language models (VLMs) and find that even the latest ones are still not robust at this task. GPT-4V and LLaVA score 72% and 42% on our main VLM metric, although they may improve with more advanced prompting techniques. On the main ITM metric, contrastive models such as CLIP and SigLIP perform close to chance (at 12% and 30%, respectively), although the non-contrastive BLIP ITM model is stronger (87%). We also find that finetuning on fewer than 2,000 examples yields significant performance gains on this out-of-distribution word-order understanding task. The dataset is here: https://github.com/Top34051/colorswap.

Title: Grandmaster-Level Chess Without Search

• Authors: Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, Tim Genewein
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.04494
• Pdf URL: https://arxiv.org/pdf/2402.04494
• Copy Paste: [[2402.04494]] Grandmaster-Level Chess Without Search(https://arxiv.org/abs/2402.04494)
• Keywords: transformer
• Abstract: The recent breakthrough successes in machine learning are mainly attributed to scale: namely large-scale attention-based architectures and datasets of unprecedented scale. This paper investigates the impact of training at scale for chess. Unlike traditional chess engines that rely on complex heuristics, explicit search, or a combination of both, we train a 270M parameter transformer model with supervised learning on a dataset of 10 million chess games. We annotate each board in the dataset with action-values provided by the powerful Stockfish 16 engine, leading to roughly 15 billion data points. Our largest model reaches a Lichess blitz Elo of 2895 against humans, and successfully solves a series of challenging chess puzzles, without any domain-specific tweaks or explicit search algorithms. We also show that our model outperforms AlphaZero's policy and value networks (without MCTS) and GPT-3.5-turbo-instruct. A systematic investigation of model and dataset size shows that strong chess performance only arises at sufficient scale. To validate our results, we perform an extensive series of ablations of design choices and hyperparameters.

Title: The Fine-Grained Complexity of Gradient Computation for Training Large Language Models

• Authors: Josh Alman, Zhao Song
• Subjects: cs.LG, cs.CC, cs.CL, cs.DS
• Abstract URL: https://arxiv.org/abs/2402.04497
• Pdf URL: https://arxiv.org/pdf/2402.04497
• Copy Paste: [[2402.04497]] The Fine-Grained Complexity of Gradient Computation for Training Large Language Models(https://arxiv.org/abs/2402.04497)
• Keywords: large language model
• Abstract: Large language models (LLMs) have made fundamental contributions over the last a few years. To train an LLM, one needs to alternatingly run `forward' computations and `backward' computations. The forward computation can be viewed as attention function evaluation, and the backward computation can be viewed as a gradient computation. In previous work by [Alman and Song, NeurIPS 2023], it was proved that the forward step can be performed in almost-linear time in certain parameter regimes, but that there is no truly sub-quadratic time algorithm in the remaining parameter regimes unless the popular hypothesis SETH is false. In this work, we show nearly identical results for the harder-seeming problem of computing the gradient of loss function of one layer attention network, and thus for the entire process of LLM training. This completely characterizes the fine-grained complexity of every step of LLM training.

Title: Text2Street: Controllable Text-to-image Generation for Street Views

• Authors: Jinming Su, Songen Gu, Yiting Duan, Xingyue Chen, Junfeng Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04504
• Pdf URL: https://arxiv.org/pdf/2402.04504
• Copy Paste: [[2402.04504]] Text2Street: Controllable Text-to-image Generation for Street Views(https://arxiv.org/abs/2402.04504)
• Keywords: diffusion
• Abstract: Text-to-image generation has made remarkable progress with the emergence of diffusion models. However, it is still a difficult task to generate images for street views based on text, mainly because the road topology of street scenes is complex, the traffic status is diverse and the weather condition is various, which makes conventional text-to-image models difficult to deal with. To address these challenges, we propose a novel controllable text-to-image framework, named \textbf{Text2Street}. In the framework, we first introduce the lane-aware road topology generator, which achieves text-to-map generation with the accurate road structure and lane lines armed with the counting adapter, realizing the controllable road topology generation. Then, the position-based object layout generator is proposed to obtain text-to-layout generation through an object-level bounding box diffusion strategy, realizing the controllable traffic object layout generation. Finally, the multiple control image generator is designed to integrate the road topology, object layout and weather description to realize controllable street-view image generation. Extensive experiments show that the proposed approach achieves controllable street-view text-to-image generation and validates the effectiveness of the Text2Street framework for street views.

Title: Online Cascade Learning for Efficient Inference over Streams

• Authors: Lunyiu Nie, Zhimin Ding, Erdong Hu, Christopher Jermaine, Swarat Chaudhuri
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04513
• Pdf URL: https://arxiv.org/pdf/2402.04513
• Copy Paste: [[2402.04513]] Online Cascade Learning for Efficient Inference over Streams(https://arxiv.org/abs/2402.04513)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have a natural role in answering complex queries about data streams, but the high computational cost of LLM inference makes them infeasible in many such tasks. We propose online cascade learning, the first approach to addressing this challenge. The objective here is to learn a "cascade" of models, starting with lower-capacity models (such as logistic regressors) and ending with a powerful LLM, along with a deferral policy that determines the model that is used on a given input. We formulate the task of learning cascades online as an imitation-learning problem and give a no-regret algorithm for the problem. Experimental results across four benchmarks show that our method parallels LLMs in accuracy while cutting down inference costs by as much as 90%, underscoring its efficacy and adaptability in stream processing.

Title: BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision

• Authors: Xin Zhao, Shiyu Hu, Yipei Wang, Jing Zhang, Yimin Hu, Rongshuai Liu, Haibin Ling, Yin Li, Renshu Li, Kun Liu, Jiadong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04519
• Pdf URL: https://arxiv.org/pdf/2402.04519
• Copy Paste: [[2402.04519]] BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision(https://arxiv.org/abs/2402.04519)
• Keywords: robust
• Abstract: Single object tracking (SOT) is a fundamental problem in computer vision, with a wide range of applications, including autonomous driving, augmented reality, and robot navigation. The robustness of SOT faces two main challenges: tiny target and fast motion. These challenges are especially manifested in videos captured by unmanned aerial vehicles (UAV), where the target is usually far away from the camera and often with significant motion relative to the camera. To evaluate the robustness of SOT methods, we propose BioDrone -- the first bionic drone-based visual benchmark for SOT. Unlike existing UAV datasets, BioDrone features videos captured from a flapping-wing UAV system with a major camera shake due to its aerodynamics. BioDrone hence highlights the tracking of tiny targets with drastic changes between consecutive frames, providing a new robust vision benchmark for SOT. To date, BioDrone offers the largest UAV-based SOT benchmark with high-quality fine-grained manual annotations and automatically generates frame-level labels, designed for robust vision analyses. Leveraging our proposed BioDrone, we conduct a systematic evaluation of existing SOT methods, comparing the performance of 20 representative models and studying novel means of optimizing a SOTA method (KeepTrack KeepTrack) for robust SOT. Our evaluation leads to new baselines and insights for robust SOT. Moving forward, we hope that BioDrone will not only serve as a high-quality benchmark for robust SOT, but also invite future research into robust computer vision. The database, toolkits, evaluation server, and baseline results are available at this http URL

Title: SumRec: A Framework for Recommendation using Open-Domain Dialogue

• Authors: Ryutaro Asahara, Masaki Takahashi, Chiho Iwahashi, Michimasa Inaba
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04523
• Pdf URL: https://arxiv.org/pdf/2402.04523
• Copy Paste: [[2402.04523]] SumRec: A Framework for Recommendation using Open-Domain Dialogue(https://arxiv.org/abs/2402.04523)
• Keywords: large language model
• Abstract: Chat dialogues contain considerable useful information about a speaker's interests, preferences, and experiences.Thus, knowledge from open-domain chat dialogue can be used to personalize various systems and offer recommendations for advanced information.This study proposed a novel framework SumRec for recommending information from open-domain chat dialogue.The study also examined the framework using ChatRec, a newly constructed dataset for training and evaluation. To extract the speaker and item characteristics, the SumRec framework employs a large language model (LLM) to generate a summary of the speaker information from a dialogue and to recommend information about an item according to the type of user.The speaker and item information are then input into a score estimation model, generating a recommendation score.Experimental results show that the SumRec framework provides better recommendations than the baseline method of using dialogues and item descriptions in their original form. Our dataset and code is publicly available at https://github.com/Ryutaro-A/SumRec

Title: Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers

• Authors: Md Shamim Hussain, Mohammed J. Zaki, Dharmashankar Subramanian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04538
• Pdf URL: https://arxiv.org/pdf/2402.04538
• Copy Paste: [[2402.04538]] Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers(https://arxiv.org/abs/2402.04538)
• Keywords: transformer
• Abstract: Graph transformers typically lack direct pair-to-pair communication, instead forcing neighboring pairs to exchange information via a common node. We propose the Triplet Graph Transformer (TGT) that enables direct communication between two neighboring pairs in a graph via novel triplet attention and aggregation mechanisms. TGT is applied to molecular property prediction by first predicting interatomic distances from 2D graphs and then using these distances for downstream tasks. A novel three-stage training procedure and stochastic inference further improve training efficiency and model performance. Our model achieves new state-of-the-art (SOTA) results on open challenge benchmarks PCQM4Mv2 and OC20 IS2RE. We also obtain SOTA results on QM9, MOLPCBA, and LIT-PCBA molecular property prediction benchmarks via transfer learning. We also demonstrate the generality of TGT with SOTA results on the traveling salesman problem (TSP).

Title: BRI3L: A Brightness Illusion Image Dataset for Identification and Localization of Regions of Illusory Perception

• Authors: Aniket Roy, Anirban Roy, Soma Mitra, Kuntal Ghosh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04541
• Pdf URL: https://arxiv.org/pdf/2402.04541
• Copy Paste: [[2402.04541]] BRI3L: A Brightness Illusion Image Dataset for Identification and Localization of Regions of Illusory Perception(https://arxiv.org/abs/2402.04541)
• Keywords: diffusion, segmentation
• Abstract: Visual illusions play a significant role in understanding visual perception. Current methods in understanding and evaluating visual illusions are mostly deterministic filtering based approach and they evaluate on a handful of visual illusions, and the conclusions therefore, are not generic. To this end, we generate a large-scale dataset of 22,366 images (BRI3L: BRightness Illusion Image dataset for Identification and Localization of illusory perception) of the five types of brightness illusions and benchmark the dataset using data-driven neural network based approaches. The dataset contains label information - (1) whether a particular image is illusory/nonillusory, (2) the segmentation mask of the illusory region of the image. Hence, both the classification and segmentation task can be evaluated using this dataset. We follow the standard psychophysical experiments involving human subjects to validate the dataset. To the best of our knowledge, this is the first attempt to develop a dataset of visual illusions and benchmark using data-driven approach for illusion classification and localization. We consider five well-studied types of brightness illusions: 1) Hermann grid, 2) Simultaneous Brightness Contrast, 3) White illusion, 4) Grid illusion, and 5) Induced Grating illusion. Benchmarking on the dataset achieves 99.56% accuracy in illusion identification and 84.37% pixel accuracy in illusion localization. The application of deep learning model, it is shown, also generalizes over unseen brightness illusions like brightness assimilation to contrast transitions. We also test the ability of state-of-theart diffusion models to generate brightness illusions. We have provided all the code, dataset, instructions etc in the github repo: https://github.com/aniket004/BRI3L

Title: Share What You Already Know: Cross-Language-Script Transfer and Alignment for Sentiment Detection in Code-Mixed Data

• Authors: Niraj Pahari, Kazutaka Shimada
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04542
• Pdf URL: https://arxiv.org/pdf/2402.04542
• Copy Paste: [[2402.04542]] Share What You Already Know: Cross-Language-Script Transfer and Alignment for Sentiment Detection in Code-Mixed Data(https://arxiv.org/abs/2402.04542)
• Keywords: explainability
• Abstract: Code-switching entails mixing multiple languages. It is an increasingly occurring phenomenon in social media texts. Usually, code-mixed texts are written in a single script, even though the languages involved have different scripts. Pre-trained multilingual models primarily utilize the data in the native script of the language. In existing studies, the code-switched texts are utilized as they are. However, using the native script for each language can generate better representations of the text owing to the pre-trained knowledge. Therefore, a cross-language-script knowledge sharing architecture utilizing the cross attention and alignment of the representations of text in individual language scripts was proposed in this study. Experimental results on two different datasets containing Nepali-English and Hindi-English code-switched texts, demonstrate the effectiveness of the proposed method. The interpretation of the model using model explainability technique illustrates the sharing of language-specific knowledge between language-specific representations.

Title: Curvature-Informed SGD via General Purpose Lie-Group Preconditioners

• Authors: Omead Pooladzandi, Xi-Lin Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04553
• Pdf URL: https://arxiv.org/pdf/2402.04553
• Copy Paste: [[2402.04553]] Curvature-Informed SGD via General Purpose Lie-Group Preconditioners(https://arxiv.org/abs/2402.04553)
• Keywords: robust
• Abstract: We present a novel approach to accelerate stochastic gradient descent (SGD) by utilizing curvature information obtained from Hessian-vector products or finite differences of parameters and gradients, similar to the BFGS algorithm. Our approach involves two preconditioners: a matrix-free preconditioner and a low-rank approximation preconditioner. We update both preconditioners online using a criterion that is robust to stochastic gradient noise and does not require line search or damping. To preserve the corresponding symmetry or invariance, our preconditioners are constrained to certain connected Lie groups. The Lie group's equivariance property simplifies the preconditioner fitting process, while its invariance property eliminates the need for damping, which is commonly required in second-order optimizers. As a result, the learning rate for parameter updating and the step size for preconditioner fitting are naturally normalized, and their default values work well in most scenarios. Our proposed approach offers a promising direction for improving the convergence of SGD with low computational overhead. We demonstrate that Preconditioned SGD (PSGD) outperforms SoTA on Vision, NLP, and RL tasks across multiple modern deep-learning architectures. We have provided code for reproducing toy and large scale experiments in this paper.

Title: FM-Fusion: Instance-aware Semantic Mapping Boosted by Vision-Language Foundation Models

• Authors: Chuhao Liu, Ke Wang, Jieqi Shi, Zhijian Qiao, Shaojie Shen
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.04555
• Pdf URL: https://arxiv.org/pdf/2402.04555
• Copy Paste: [[2402.04555]] FM-Fusion: Instance-aware Semantic Mapping Boosted by Vision-Language Foundation Models(https://arxiv.org/abs/2402.04555)
• Keywords: segmentation
• Abstract: Semantic mapping based on the supervised object detectors is sensitive to image distribution. In real-world environments, the object detection and segmentation performance can lead to a major drop, preventing the use of semantic mapping in a wider domain. On the other hand, the development of vision-language foundation models demonstrates a strong zero-shot transferability across data distribution. It provides an opportunity to construct generalizable instance-aware semantic maps. Hence, this work explores how to boost instance-aware semantic mapping from object detection generated from foundation models. We propose a probabilistic label fusion method to predict close-set semantic classes from open-set label measurements. An instance refinement module merges the over-segmented instances caused by inconsistent segmentation. We integrate all the modules into a unified semantic mapping system. Reading a sequence of RGB-D input, our work incrementally reconstructs an instance-aware semantic map. We evaluate the zero-shot performance of our method in ScanNet and SceneNN datasets. Our method achieves 40.3 mean average precision (mAP) on the ScanNet semantic instance segmentation task. It outperforms the traditional semantic mapping method significantly.

Title: DMAT: A Dynamic Mask-Aware Transformer for Human De-occlusion

• Authors: Guoqiang Liang, Jiahao Hu, Qingyue Wang, Shizhou Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04558
• Pdf URL: https://arxiv.org/pdf/2402.04558
• Copy Paste: [[2402.04558]] DMAT: A Dynamic Mask-Aware Transformer for Human De-occlusion(https://arxiv.org/abs/2402.04558)
• Keywords: transformer
• Abstract: Human de-occlusion, which aims to infer the appearance of invisible human parts from an occluded image, has great value in many human-related tasks, such as person re-id, and intention inference. To address this task, this paper proposes a dynamic mask-aware transformer (DMAT), which dynamically augments information from human regions and weakens that from occlusion. First, to enhance token representation, we design an expanded convolution head with enlarged kernels, which captures more local valid context and mitigates the influence of surrounding occlusion. To concentrate on the visible human parts, we propose a novel dynamic multi-head human-mask guided attention mechanism through integrating multiple masks, which can prevent the de-occluded regions from assimilating to the background. Besides, a region upsampling strategy is utilized to alleviate the impact of occlusion on interpolated images. During model learning, an amodal loss is developed to further emphasize the recovery effect of human regions, which also refines the model's convergence. Extensive experiments on the AHP dataset demonstrate its superior performance compared to recent state-of-the-art methods.

Title: Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention

• Authors: Saebom Leem, Hyunseok Seo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04563
• Pdf URL: https://arxiv.org/pdf/2402.04563
• Copy Paste: [[2402.04563]] Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention(https://arxiv.org/abs/2402.04563)
• Keywords: explainability, transformer
• Abstract: Vision Transformer(ViT) is one of the most widely used models in the computer vision field with its great performance on various tasks. In order to fully utilize the ViT-based architecture in various applications, proper visualization methods with a decent localization performance are necessary, but these methods employed in CNN-based models are still not available in ViT due to its unique structure. In this work, we propose an attention-guided visualization method applied to ViT that provides a high-level semantic explanation for its decision. Our method selectively aggregates the gradients directly propagated from the classification output to each self-attention, collecting the contribution of image features extracted from each location of the input image. These gradients are additionally guided by the normalized self-attention scores, which are the pairwise patch correlation scores. They are used to supplement the gradients on the patch-level context information efficiently detected by the self-attention mechanism. This approach of our method provides elaborate high-level semantic explanations with great localization performance only with the class labels. As a result, our method outperforms the previous leading explainability methods of ViT in the weakly-supervised localization task and presents great capability in capturing the full instances of the target class object. Meanwhile, our method provides a visualization that faithfully explains the model, which is demonstrated in the perturbation comparison test.

Title: OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequences

• Authors: Chen Wang, Sarah Erfani, Tansu Alpcan, Christopher Leckie
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04567
• Pdf URL: https://arxiv.org/pdf/2402.04567
• Copy Paste: [[2402.04567]] OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequences(https://arxiv.org/abs/2402.04567)
• Keywords: transformer
• Abstract: Anomaly detection in decision-making sequences is a challenging problem due to the complexity of normality representation learning and the sequential nature of the task. Most existing methods based on Reinforcement Learning (RL) are difficult to implement in the real world due to unrealistic assumptions, such as having access to environment dynamics, reward signals, and online interactions with the environment. To address these limitations, we propose an unsupervised method named Offline Imitation Learning based Anomaly Detection (OIL-AD), which detects anomalies in decision-making sequences using two extracted behaviour features: action optimality and sequential association. Our offline learning model is an adaptation of behavioural cloning with a transformer policy network, where we modify the training process to learn a Q function and a state value function from normal trajectories. We propose that the Q function and the state value function can provide sufficient information about agents' behavioural data, from which we derive two features for anomaly detection. The intuition behind our method is that the action optimality feature derived from the Q function can differentiate the optimal action from others at each local state, and the sequential association feature derived from the state value function has the potential to maintain the temporal correlations between decisions (state-action pairs). Our experiments show that OIL-AD can achieve outstanding online anomaly detection performance with up to 34.8% improvement in F1 score over comparable baselines.

Title: Sparse Anatomical Prompt Semi-Supervised Learning with Masked Image Modeling for CBCT Tooth Segmentation

• Authors: Pengyu Dai, Yafei Ou, Yang Liu, Yue Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04587
• Pdf URL: https://arxiv.org/pdf/2402.04587
• Copy Paste: [[2402.04587]] Sparse Anatomical Prompt Semi-Supervised Learning with Masked Image Modeling for CBCT Tooth Segmentation(https://arxiv.org/abs/2402.04587)
• Keywords: segmentation
• Abstract: Accurate tooth identification and segmentation in Cone Beam Computed Tomography (CBCT) dental images can significantly enhance the efficiency and precision of manual diagnoses performed by dentists. However, existing segmentation methods are mainly developed based on large data volumes training, on which their annotations are extremely time-consuming. Meanwhile, the teeth of each class in CBCT dental images being closely positioned, coupled with subtle inter-class differences, gives rise to the challenge of indistinct boundaries when training model with limited data. To address these challenges, this study aims to propose a tasked-oriented Masked Auto-Encoder paradigm to effectively utilize large amounts of unlabeled data to achieve accurate tooth segmentation with limited labeled data. Specifically, we first construct a self-supervised pre-training framework of masked auto encoder to efficiently utilize unlabeled data to enhance the network performance. Subsequently, we introduce a sparse masked prompt mechanism based on graph attention to incorporate boundary information of the teeth, aiding the network in learning the anatomical structural features of teeth. To the best of our knowledge, we are pioneering the integration of the mask pre-training paradigm into the CBCT tooth segmentation task. Extensive experiments demonstrate both the feasibility of our proposed method and the potential of the boundary prompt mechanism.

Title: UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset

• Authors: Haoyu Wang, Shuo Wang, Yukun Yan, Xujia Wang, Zhiyu Yang, Yuzhuang Xu, Zhenghao Liu, Ning Ding, Xu Han, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04588
• Pdf URL: https://arxiv.org/pdf/2402.04588
• Copy Paste: [[2402.04588]] UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset(https://arxiv.org/abs/2402.04588)
• Keywords: large language model
• Abstract: Open-source large language models (LLMs) have gained significant strength across diverse fields. Nevertheless, the majority of studies primarily concentrate on English, with only limited exploration into the realm of multilingual supervised fine-tuning. In this work, we therefore construct an open-source multilingual supervised fine-tuning dataset. Different from previous works that simply translate English instructions, we consider both the language-specific and language-agnostic abilities of LLMs. For language-specific abilities, we introduce a knowledge-grounded data augmentation approach to elicit more culture-specific knowledge of LLMs, improving their ability to serve users from different countries. For language-agnostic abilities, we find through experiments that modern LLMs exhibit strong cross-lingual transfer capabilities, thus repeatedly learning identical content in various languages is not necessary. Consequently, we can substantially prune the language-agnostic SFT data without any performance degradation, making the SFT process more efficient. The resulting UltraLink dataset comprises approximately 1 million samples across five languages, and the proposed data construction method can also be easily extended to other languages. UltraLink-LM, which is trained on UltraLink, outperforms several representative baselines across many tasks.

Title: Ransomware Detection Dynamics: Insights and Implications

• Authors: Mike Nkongolo
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.04594
• Pdf URL: https://arxiv.org/pdf/2402.04594
• Copy Paste: [[2402.04594]] Ransomware Detection Dynamics: Insights and Implications(https://arxiv.org/abs/2402.04594)
• Keywords: attack
• Abstract: The rise of ransomware attacks has necessitated the development of effective strategies for identifying and mitigating these threats. This research investigates the utilization of a feature selection algorithm for distinguishing ransomware-related and benign transactions in both Bitcoin (BTC) and United States Dollar (USD). Leveraging the UGRansome dataset, a comprehensive repository of ransomware related BTC and USD transactions, we propose a set of novel features designed to capture the distinct characteristics of ransomware activity within the cryptocurrency ecosystem. These features encompass transaction metadata, ransom analysis, and behavioral patterns, offering a multifaceted view of ransomware-related financial transactions. Through rigorous experimentation and evaluation, we demonstrate the effectiveness of our feature set in accurately extracting BTC and USD transactions, thereby aiding in the early detection and prevention of ransomware-related financial flows. We introduce a Ransomware Feature Selection Algorithm (RFSA) based on Gini Impurity and Mutual Information (MI) for selecting crucial ransomware features from the UGRansome dataset. Insights from the visualization highlight the potential of Gini Impurity and MI-based feature selection to enhance ransomware detection systems by effectively discriminating between ransomware classes. The analysis reveals that approximately 68% of ransomware incidents involve BTC transactions within the range of 1.46 to 2.56, with an average of 2.01 BTC transactions per attack. The findings emphasize the dynamic and adaptable nature of ransomware demands, suggesting that there is no fixed amount for specific cyberattacks, highlighting the evolving landscape of ransomware threats.

Title: Towards Improved Imbalance Robustness in Continual Multi-Label Learning with Dual Output Spiking Architecture (DOSA)

• Authors: Sourav Mishra, Shirin Dora, Suresh Sundaram
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04596
• Pdf URL: https://arxiv.org/pdf/2402.04596
• Copy Paste: [[2402.04596]] Towards Improved Imbalance Robustness in Continual Multi-Label Learning with Dual Output Spiking Architecture (DOSA)(https://arxiv.org/abs/2402.04596)
• Keywords: robust
• Abstract: Algorithms designed for addressing typical supervised classification problems can only learn from a fixed set of samples and labels, making them unsuitable for the real world, where data arrives as a stream of samples often associated with multiple labels over time. This motivates the study of task-agnostic continual multi-label learning problems. While algorithms using deep learning approaches for continual multi-label learning have been proposed in the recent literature, they tend to be computationally heavy. Although spiking neural networks (SNNs) offer a computationally efficient alternative to artificial neural networks, existing literature has not used SNNs for continual multi-label learning. Also, accurately determining multiple labels with SNNs is still an open research problem. This work proposes a dual output spiking architecture (DOSA) to bridge these research gaps. A novel imbalance-aware loss function is also proposed, improving the multi-label classification performance of the model by making it more robust to data imbalance. A modified F1 score is presented to evaluate the effectiveness of the proposed loss function in handling imbalance. Experiments on several benchmark multi-label datasets show that DOSA trained with the proposed loss function shows improved robustness to data imbalance and obtains better continual multi-label learning performance than CIFDM, a previous state-of-the-art algorithm.

Title: Alirector: Alignment-Enhanced Chinese Grammatical Error Corrector

• Authors: Haihui Yang, Xiaojun Quan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04601
• Pdf URL: https://arxiv.org/pdf/2402.04601
• Copy Paste: [[2402.04601]] Alirector: Alignment-Enhanced Chinese Grammatical Error Corrector(https://arxiv.org/abs/2402.04601)
• Keywords: generative, large language model
• Abstract: Chinese grammatical error correction (CGEC) faces serious overcorrection challenges when employing autoregressive generative models such as sequence-to-sequence (Seq2Seq) models and decoder-only large language models (LLMs). While previous methods aim to address overcorrection in Seq2Seq models, they are difficult to adapt to decoder-only LLMs. In this paper, we propose an alignment-enhanced corrector for the overcorrection problem that applies to both Seq2Seq models and decoder-only LLMs. Our method first trains a correction model to generate an initial correction of the source sentence. Then, we combine the source sentence with the initial correction and feed it through an alignment model for another round of correction, aiming to enforce the alignment model to focus on potential overcorrection. Moreover, to enhance the model's ability to identify nuances, we further explore the reverse alignment of the source sentence and the initial correction. Finally, we transfer the alignment knowledge from two alignment models to the correction model, instructing it on how to avoid overcorrection. Experimental results on three CGEC datasets demonstrate the effectiveness of our approach in alleviating overcorrection and improving overall performance.

Title: Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approach

• Authors: Zhuang Li, Levon Haroutunian, Raj Tumuluri, Philip Cohen, Gholamreza Haffari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04609
• Pdf URL: https://arxiv.org/pdf/2402.04609
• Copy Paste: [[2402.04609]] Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approach(https://arxiv.org/abs/2402.04609)
• Keywords: large language model
• Abstract: Post-editing has proven effective in improving the quality of text generated by large language models (LLMs) such as GPT-3.5 or GPT-4, particularly when direct updating of their parameters to enhance text quality is infeasible or expensive. However, relying solely on smaller language models for post-editing can limit the LLMs' ability to generalize across domains. Moreover, the editing strategies in these methods are not optimally designed for text-generation tasks. To address these limitations, we propose a neural programmer-interpreter approach that preserves the domain generalization ability of LLMs when editing their output. The editing actions in this framework are specifically devised for text generation. Extensive experiments demonstrate that the programmer-interpreter significantly enhances GPT-3.5's performance in logical form-to-text conversion and low-resource machine translation, surpassing other state-of-the-art (SOTA) LLM post-editing methods in cross-domain settings.

Title: Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models

• Authors: Chirag Agarwal, Sree Harsha Tanneru, Himabindu Lakkaraju
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04614
• Pdf URL: https://arxiv.org/pdf/2402.04614
• Copy Paste: [[2402.04614]] Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models(https://arxiv.org/abs/2402.04614)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are deployed as powerful tools for several natural language processing (NLP) applications. Recent works show that modern LLMs can generate self-explanations (SEs), which elicit their intermediate reasoning steps for explaining their behavior. Self-explanations have seen widespread adoption owing to their conversational and plausible nature. However, there is little to no understanding of their faithfulness. In this work, we discuss the dichotomy between faithfulness and plausibility in SEs generated by LLMs. We argue that while LLMs are adept at generating plausible explanations -- seemingly logical and coherent to human users -- these explanations do not necessarily align with the reasoning processes of the LLMs, raising concerns about their faithfulness. We highlight that the current trend towards increasing the plausibility of explanations, primarily driven by the demand for user-friendly interfaces, may come at the cost of diminishing their faithfulness. We assert that the faithfulness of explanations is critical in LLMs employed for high-stakes decision-making. Moreover, we urge the community to identify the faithfulness requirements of real-world applications and ensure explanations meet those needs. Finally, we propose some directions for future work, emphasizing the need for novel methodologies and frameworks that can enhance the faithfulness of self-explanations without compromising their plausibility, essential for the transparent deployment of LLMs in diverse high-stakes domains.

Title: ScreenAI: A Vision-Language Model for UI and Infographics Understanding

• Authors: Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cărbune, Jason Lin, Jindong Chen, Abhanshu Sharma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04615
• Pdf URL: https://arxiv.org/pdf/2402.04615
• Copy Paste: [[2402.04615]] ScreenAI: A Vision-Language Model for UI and Infographics Understanding(https://arxiv.org/abs/2402.04615)
• Keywords: large language model
• Abstract: Screen user interfaces (UIs) and infographics, sharing similar visual language and design principles, play important roles in human communication and human-machine interaction. We introduce ScreenAI, a vision-language model that specializes in UI and infographics understanding. Our model improves upon the PaLI architecture with the flexible patching strategy of pix2struct and is trained on a unique mixture of datasets. At the heart of this mixture is a novel screen annotation task in which the model has to identify the type and location of UI elements. We use these text annotations to describe screens to Large Language Models and automatically generate question-answering (QA), UI navigation, and summarization training datasets at scale. We run ablation studies to demonstrate the impact of these design choices. At only 5B parameters, ScreenAI achieves new state-of-the-artresults on UI- and infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and InfographicVQA) compared to models of similar size. Finally, we release three new datasets: one focused on the screen annotation task and two others focused on question answering.

Title: TinyLLM: Learning a Small Student from Multiple Large Language Models

• Authors: Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, Nitesh V. Chawla
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04616
• Pdf URL: https://arxiv.org/pdf/2402.04616
• Copy Paste: [[2402.04616]] TinyLLM: Learning a Small Student from Multiple Large Language Models(https://arxiv.org/abs/2402.04616)
• Keywords: large language model
• Abstract: Transferring the reasoning capability from stronger large language models (LLMs) to smaller ones has been quite appealing, as smaller LLMs are more flexible to deploy with less expense. Among the existing solutions, knowledge distillation stands out due to its outstanding efficiency and generalization. However, existing methods suffer from several drawbacks, including limited knowledge diversity and the lack of rich contextual information. To solve the problems and facilitate the learning of compact language models, we propose TinyLLM, a novel knowledge distillation paradigm to learn a small student LLM from multiple large teacher LLMs. In particular, we encourage the student LLM to not only generate the correct answers but also understand the rationales behind these answers. Given that different LLMs possess diverse reasoning skills, we guide the student model to assimilate knowledge from various teacher LLMs. We further introduce an in-context example generator and a teacher-forcing Chain-of-Thought strategy to ensure that the rationales are accurate and grounded in contextually appropriate scenarios. Extensive experiments on six datasets across two reasoning tasks demonstrate the superiority of our method. Results show that TinyLLM can outperform large teacher LLMs significantly, despite having a considerably smaller model size.

Title: InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory

• Authors: Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, Maosong Sun
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04617
• Pdf URL: https://arxiv.org/pdf/2402.04617
• Copy Paste: [[2402.04617]] InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory(https://arxiv.org/abs/2402.04617)
• Keywords: large language model
• Abstract: Large language models (LLMs) have emerged as a cornerstone in real-world applications with lengthy streaming inputs, such as LLM-driven agents. However, existing LLMs, pre-trained on sequences with restricted maximum length, cannot generalize to longer sequences due to the out-of-domain and distraction issues. To alleviate these issues, existing efforts employ sliding attention windows and discard distant tokens to achieve the processing of extremely long sequences. Unfortunately, these approaches inevitably fail to capture long-distance dependencies within sequences to deeply understand semantics. This paper introduces a training-free memory-based method, InfLLM, to unveil the intrinsic ability of LLMs to process streaming long sequences. Specifically, InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. Thereby, InfLLM allows LLMs to efficiently process long sequences while maintaining the ability to capture long-distance dependencies. Without any training, InfLLM enables LLMs pre-trained on sequences of a few thousand tokens to achieve superior performance than competitive baselines continually training these LLMs on long sequences. Even when the sequence length is scaled to $1,024$K, InfLLM still effectively captures long-distance dependencies.

Title: Multi-Scale Semantic Segmentation with Modified MBConv Blocks

• Authors: Xi Chen, Yang Cai, Yuan Wu, Bo Xiong, Taesung Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04618
• Pdf URL: https://arxiv.org/pdf/2402.04618
• Copy Paste: [[2402.04618]] Multi-Scale Semantic Segmentation with Modified MBConv Blocks(https://arxiv.org/abs/2402.04618)
• Keywords: extraction, segmentation
• Abstract: Recently, MBConv blocks, initially designed for efficiency in resource-limited settings and later adapted for cutting-edge image classification performances, have demonstrated significant potential in image classification tasks. Despite their success, their application in semantic segmentation has remained relatively unexplored. This paper introduces a novel adaptation of MBConv blocks specifically tailored for semantic segmentation. Our modification stems from the insight that semantic segmentation requires the extraction of more detailed spatial information than image classification. We argue that to effectively perform multi-scale semantic segmentation, each branch of a U-Net architecture, regardless of its resolution, should possess equivalent segmentation capabilities. By implementing these changes, our approach achieves impressive mean Intersection over Union (IoU) scores of 84.5% and 84.0% on the Cityscapes test and validation datasets, respectively, demonstrating the efficacy of our proposed modifications in enhancing semantic segmentation performance.

Title: MEMORYLLM: Towards Self-Updatable Large Language Models

• Authors: Yu Wang, Xiusi Chen, Jingbo Shang, Julian McAuley
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04624
• Pdf URL: https://arxiv.org/pdf/2402.04624
• Copy Paste: [[2402.04624]] MEMORYLLM: Towards Self-Updatable Large Language Models(https://arxiv.org/abs/2402.04624)
• Keywords: transformer, large language model
• Abstract: Existing Large Language Models (LLMs) usually remain static after deployment, which might make it hard to inject new knowledge into the model. We aim to build models containing a considerable portion of self-updatable parameters, enabling the model to integrate new knowledge effectively and efficiently. To this end, we introduce MEMORYLLM, a model that comprises a transformer and a fixed-size memory pool within the latent space of the transformer. MEMORYLLM can self-update with text knowledge and memorize the knowledge injected earlier. Our evaluations demonstrate the ability of MEMORYLLM to effectively incorporate new knowledge, as evidenced by its performance on model editing benchmarks. Meanwhile, the model exhibits long-term information retention capacity, which is validated through our custom-designed evaluations and long-context benchmarks. MEMORYLLM also shows operational integrity without any sign of performance degradation even after nearly a million memory updates.

Title: Noise Map Guidance: Inversion with Spatial Context for Real Image Editing

• Authors: Hansam Cho, Jonghyun Lee, Seoung Bum Kim, Tae-Hyun Oh, Yonghyun Jeong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04625
• Pdf URL: https://arxiv.org/pdf/2402.04625
• Copy Paste: [[2402.04625]] Noise Map Guidance: Inversion with Spatial Context for Real Image Editing(https://arxiv.org/abs/2402.04625)
• Keywords: robust, diffusion
• Abstract: Text-guided diffusion models have become a popular tool in image synthesis, known for producing high-quality and diverse images. However, their application to editing real images often encounters hurdles primarily due to the text condition deteriorating the reconstruction quality and subsequently affecting editing fidelity. Null-text Inversion (NTI) has made strides in this area, but it fails to capture spatial context and requires computationally intensive per-timestep optimization. Addressing these challenges, we present Noise Map Guidance (NMG), an inversion method rich in a spatial context, tailored for real-image editing. Significantly, NMG achieves this without necessitating optimization, yet preserves the editing quality. Our empirical investigations highlight NMG's adaptability across various editing techniques and its robustness to variants of DDIM inversions.

Title: LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors

• Authors: Sheng Jin, Xueying Jiang, Jiaxing Huang, Lewei Lu, Shijian Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04630
• Pdf URL: https://arxiv.org/pdf/2402.04630
• Copy Paste: [[2402.04630]] LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors(https://arxiv.org/abs/2402.04630)
• Keywords: large language model
• Abstract: Inspired by the outstanding zero-shot capability of vision language models (VLMs) in image classification tasks, open-vocabulary object detection has attracted increasing interest by distilling the broad VLM knowledge into detector training. However, most existing open-vocabulary detectors learn by aligning region embeddings with categorical labels (e.g., bicycle) only, disregarding the capability of VLMs on aligning visual embeddings with fine-grained text description of object parts (e.g., pedals and bells). This paper presents DVDet, a Descriptor-Enhanced Open Vocabulary Detector that introduces conditional context prompts and hierarchical textual descriptors that enable precise region-text alignment as well as open-vocabulary detection training in general. Specifically, the conditional context prompt transforms regional embeddings into image-like representations that can be directly integrated into general open vocabulary detection training. In addition, we introduce large language models as an interactive and implicit knowledge repository which enables iterative mining and refining visually oriented textual descriptors for precise region-text alignment. Extensive experiments over multiple large-scale benchmarks show that DVDet outperforms the state-of-the-art consistently by large margins.

Title: The Future of Cognitive Strategy-enhanced Persuasive Dialogue Agents: New Perspectives and Trends

• Authors: Mengqi Chen, Bin Guo, Hao Wang, Haoyu Li, Qian Zhao, Jingqi Liu, Yasan Ding, Yan Pan, Zhiwen Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04631
• Pdf URL: https://arxiv.org/pdf/2402.04631
• Copy Paste: [[2402.04631]] The Future of Cognitive Strategy-enhanced Persuasive Dialogue Agents: New Perspectives and Trends(https://arxiv.org/abs/2402.04631)
• Keywords: large language model
• Abstract: Persuasion, as one of the crucial abilities in human communication, has garnered extensive attention from researchers within the field of intelligent dialogue systems. We humans tend to persuade others to change their viewpoints, attitudes or behaviors through conversations in various scenarios (e.g., persuasion for social good, arguing in online platforms). Developing dialogue agents that can persuade others to accept certain standpoints is essential to achieving truly intelligent and anthropomorphic dialogue system. Benefiting from the substantial progress of Large Language Models (LLMs), dialogue agents have acquired an exceptional capability in context understanding and response generation. However, as a typical and complicated cognitive psychological system, persuasive dialogue agents also require knowledge from the domain of cognitive psychology to attain a level of human-like persuasion. Consequently, the cognitive strategy-enhanced persuasive dialogue agent (defined as CogAgent), which incorporates cognitive strategies to achieve persuasive targets through conversation, has become a predominant research paradigm. To depict the research trends of CogAgent, in this paper, we first present several fundamental cognitive psychology theories and give the formalized definition of three typical cognitive strategies, including the persuasion strategy, the topic path planning strategy, and the argument structure prediction strategy. Then we propose a new system architecture by incorporating the formalized definition to lay the foundation of CogAgent. Representative works are detailed and investigated according to the combined cognitive strategy, followed by the summary of authoritative benchmarks and evaluation metrics. Finally, we summarize our insights on open issues and future directions of CogAgent for upcoming researchers.

Title: GSN: Generalisable Segmentation in Neural Radiance Field

• Authors: Vinayak Gupta, Rahul Goel, Sirikonda Dhawal, P. J. Narayanan
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.04632
• Pdf URL: https://arxiv.org/pdf/2402.04632
• Copy Paste: [[2402.04632]] GSN: Generalisable Segmentation in Neural Radiance Field(https://arxiv.org/abs/2402.04632)
• Keywords: segmentation
• Abstract: Traditional Radiance Field (RF) representations capture details of a specific scene and must be trained afresh on each scene. Semantic feature fields have been added to RFs to facilitate several segmentation tasks. Generalised RF representations learn the principles of view interpolation. A generalised RF can render new views of an unknown and untrained scene, given a few views. We present a way to distil feature fields into the generalised GNT representation. Our GSN representation generates new views of unseen scenes on the fly along with consistent, per-pixel semantic features. This enables multi-view segmentation of arbitrary new scenes. We show different semantic features being distilled into generalised RFs. Our multi-view segmentation results are on par with methods that use traditional RFs. GSN closes the gap between standard and generalisable RF methods significantly. Project Page: https://vinayak-vg.github.io/GSN/

Title: TransLLaMa: LLM-based Simultaneous Translation System

• Authors: Roman Koshkin, Katsuhito Sudoh, Satoshi Nakamura
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04636
• Pdf URL: https://arxiv.org/pdf/2402.04636
• Copy Paste: [[2402.04636]] TransLLaMa: LLM-based Simultaneous Translation System(https://arxiv.org/abs/2402.04636)
• Keywords: transformer, large language model, segmentation
• Abstract: Decoder-only large language models (LLMs) have recently demonstrated impressive capabilities in text generation and reasoning. Nonetheless, they have limited applications in simultaneous machine translation (SiMT), currently dominated by encoder-decoder transformers. This study demonstrates that, after fine-tuning on a small dataset comprising causally aligned source and target sentence pairs, a pre-trained open-source LLM can control input segmentation directly by generating a special "wait" token. This obviates the need for a separate policy and enables the LLM to perform English-German and English-Russian SiMT tasks with BLEU scores that are comparable to those of specific state-of-the-art baselines. We also evaluated closed-source models such as GPT-4, which displayed encouraging results in performing the SiMT task without prior training (zero-shot), indicating a promising avenue for enhancing future SiMT systems.

Title: Domain Bridge: Generative model-based domain forensic for black-box models

• Authors: Jiyi Zhang, Han Fang, Ee-Chien Chang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04640
• Pdf URL: https://arxiv.org/pdf/2402.04640
• Copy Paste: [[2402.04640]] Domain Bridge: Generative model-based domain forensic for black-box models(https://arxiv.org/abs/2402.04640)
• Keywords: diffusion, generative
• Abstract: In forensic investigations of machine learning models, techniques that determine a model's data domain play an essential role, with prior work relying on large-scale corpora like ImageNet to approximate the target model's domain. Although such methods are effective in finding broad domains, they often struggle in identifying finer-grained classes within those domains. In this paper, we introduce an enhanced approach to determine not just the general data domain (e.g., human face) but also its specific attributes (e.g., wearing glasses). Our approach uses an image embedding model as the encoder and a generative model as the decoder. Beginning with a coarse-grained description, the decoder generates a set of images, which are then presented to the unknown target model. Successful classifications by the model guide the encoder to refine the description, which in turn, are used to produce a more specific set of images in the subsequent iteration. This iterative refinement narrows down the exact class of interest. A key strength of our approach lies in leveraging the expansive dataset, LAION-5B, on which the generative model Stable Diffusion is trained. This enlarges our search space beyond traditional corpora, such as ImageNet. Empirical results showcase our method's performance in identifying specific attributes of a model's input domain, paving the way for more detailed forensic analyses of deep learning models.

Title: Latent Plan Transformer: Planning as Latent Variable Inference

• Authors: Deqian Kong, Dehong Xu, Minglu Zhao, Bo Pang, Jianwen Xie, Andrew Lizarraga, Yuhao Huang, Sirui Xie, Ying Nian Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04647
• Pdf URL: https://arxiv.org/pdf/2402.04647
• Copy Paste: [[2402.04647]] Latent Plan Transformer: Planning as Latent Variable Inference(https://arxiv.org/abs/2402.04647)
• Keywords: transformer, generative
• Abstract: In tasks aiming for long-term returns, planning becomes necessary. We study generative modeling for planning with datasets repurposed from offline reinforcement learning. Specifically, we identify temporal consistency in the absence of step-wise rewards as one key technical challenge. We introduce the Latent Plan Transformer (LPT), a novel model that leverages a latent space to connect a Transformer-based trajectory generator and the final return. LPT can be learned with maximum likelihood estimation on trajectory-return pairs. In learning, posterior sampling of the latent variable naturally gathers sub-trajectories to form a consistent abstraction despite the finite context. During test time, the latent variable is inferred from an expected return before policy execution, realizing the idea of planning as inference. It then guides the autoregressive policy throughout the episode, functioning as a plan. Our experiments demonstrate that LPT can discover improved decisions from suboptimal trajectories. It achieves competitive performance across several benchmarks, including Gym-Mujoco, Maze2D, and Connect Four, exhibiting capabilities of nuanced credit assignments, trajectory stitching, and adaptation to environmental contingencies. These results validate that latent variable inference can be a strong alternative to step-wise reward prompting.

Title: OV-NeRF: Open-vocabulary Neural Radiance Fields with Vision and Language Foundation Models for 3D Semantic Understanding

• Authors: Guibiao Liao, Kaichen Zhou, Zhenyu Bao, Kanglin Liu, Qing Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04648
• Pdf URL: https://arxiv.org/pdf/2402.04648
• Copy Paste: [[2402.04648]] OV-NeRF: Open-vocabulary Neural Radiance Fields with Vision and Language Foundation Models for 3D Semantic Understanding(https://arxiv.org/abs/2402.04648)
• Keywords: robust
• Abstract: The development of Neural Radiance Fields (NeRFs) has provided a potent representation for encapsulating the geometric and appearance characteristics of 3D scenes. Enhancing the capabilities of NeRFs in open-vocabulary 3D semantic perception tasks has been a recent focus. However, current methods that extract semantics directly from Contrastive Language-Image Pretraining (CLIP) for semantic field learning encounter difficulties due to noisy and view-inconsistent semantics provided by CLIP. To tackle these limitations, we propose OV-NeRF, which exploits the potential of pre-trained vision and language foundation models to enhance semantic field learning through proposed single-view and cross-view strategies. First, from the single-view perspective, we introduce Region Semantic Ranking (RSR) regularization by leveraging 2D mask proposals derived from SAM to rectify the noisy semantics of each training view, facilitating accurate semantic field learning. Second, from the cross-view perspective, we propose a Cross-view Self-enhancement (CSE) strategy to address the challenge raised by view-inconsistent semantics. Rather than invariably utilizing the 2D inconsistent semantics from CLIP, CSE leverages the 3D consistent semantics generated from the well-trained semantic field itself for semantic field training, aiming to reduce ambiguity and enhance overall semantic consistency across different views. Extensive experiments validate our OV-NeRF outperforms current state-of-the-art methods, achieving a significant improvement of 20.31% and 18.42% in mIoU metric on Replica and Scannet, respectively. Furthermore, our approach exhibits consistent superior results across various CLIP configurations, further verifying its robustness.

Title: An Over Complete Deep Learning Method for Inverse Problems

• Authors: Moshe Eliasof, Eldad Haber, Eran Treister
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04653
• Pdf URL: https://arxiv.org/pdf/2402.04653
• Copy Paste: [[2402.04653]] An Over Complete Deep Learning Method for Inverse Problems(https://arxiv.org/abs/2402.04653)
• Keywords: diffusion
• Abstract: Obtaining meaningful solutions for inverse problems has been a major challenge with many applications in science and engineering. Recent machine learning techniques based on proximal and diffusion-based methods have shown promising results. However, as we show in this work, they can also face challenges when applied to some exemplary problems. We show that similar to previous works on over-complete dictionaries, it is possible to overcome these shortcomings by embedding the solution into higher dimensions. The novelty of the work proposed is that we jointly design and learn the embedding and the regularizer for the embedding vector. We demonstrate the merit of this approach on several exemplary and common inverse problems.

Title: Adversarial Robustness Through Artifact Design

• Authors: Tsufit Shua, Mahmood Sharif
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04660
• Pdf URL: https://arxiv.org/pdf/2402.04660
• Copy Paste: [[2402.04660]] Adversarial Robustness Through Artifact Design(https://arxiv.org/abs/2402.04660)
• Keywords: defense, robust
• Abstract: Adversarial examples arose as a challenge for machine learning. To hinder them, most defenses alter how models are trained (e.g., adversarial training) or inference is made (e.g., randomized smoothing). Still, while these approaches markedly improve models' adversarial robustness, models remain highly susceptible to adversarial examples. Identifying that, in certain domains such as traffic-sign recognition, objects are implemented per standards specifying how artifacts (e.g., signs) should be designed, we propose a novel approach for improving adversarial robustness. Specifically, we offer a method to redefine standards, making minor changes to existing ones, to defend against adversarial examples. We formulate the problem of artifact design as a robust optimization problem, and propose gradient-based and greedy search methods to solve it. We evaluated our approach in the domain of traffic-sign recognition, allowing it to alter traffic-sign pictograms (i.e., symbols within the signs) and their colors. We found that, combined with adversarial training, our approach led to up to 25.18\% higher robust accuracy compared to state-of-the-art methods against two adversary types, while further increasing accuracy on benign inputs.

Title: Group Distributionally Robust Dataset Distillation with Risk Minimization

• Authors: Saeed Vahidian, Mingyu Wang, Jianyang Gu, Vyacheslav Kungurtsev, Wei Jiang, Yiran Chen
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04676
• Pdf URL: https://arxiv.org/pdf/2402.04676
• Copy Paste: [[2402.04676]] Group Distributionally Robust Dataset Distillation with Risk Minimization(https://arxiv.org/abs/2402.04676)
• Keywords: robust, federate
• Abstract: Dataset distillation (DD) has emerged as a widely adopted technique for crafting a synthetic dataset that captures the essential information of a training dataset, facilitating the training of accurate neural models. Its applications span various domains, including transfer learning, federated learning, and neural architecture search. The most popular methods for constructing the synthetic data rely on matching the convergence properties of training the model with the synthetic dataset and the training dataset. However, targeting the training dataset must be thought of as auxiliary in the same sense that the training set is an approximate substitute for the population distribution, and the latter is the data of interest. Yet despite its popularity, an aspect that remains unexplored is the relationship of DD to its generalization, particularly across uncommon subgroups. That is, how can we ensure that a model trained on the synthetic dataset performs well when faced with samples from regions with low population density? Here, the representativeness and coverage of the dataset become salient over the guaranteed training error at inference. Drawing inspiration from distributionally robust optimization, we introduce an algorithm that combines clustering with the minimization of a risk measure on the loss to conduct DD. We provide a theoretical rationale for our approach and demonstrate its effective generalization and robustness across subgroups through numerical experiments.

Title: Source Identification in Abstractive Summarization

• Authors: Yoshi Suhara, Dimitris Alikaniotis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04677
• Pdf URL: https://arxiv.org/pdf/2402.04677
• Copy Paste: [[2402.04677]] Source Identification in Abstractive Summarization(https://arxiv.org/abs/2402.04677)
• Keywords: robust
• Abstract: Neural abstractive summarization models make summaries in an end-to-end manner, and little is known about how the source information is actually converted into summaries. In this paper, we define input sentences that contain essential information in the generated summary as $\textit{source sentences}$ and study how abstractive summaries are made by analyzing the source sentences. To this end, we annotate source sentences for reference summaries and system summaries generated by PEGASUS on document-summary pairs sampled from the CNN/DailyMail and XSum datasets. We also formulate automatic source sentence detection and compare multiple methods to establish a strong baseline for the task. Experimental results show that the perplexity-based method performs well in highly abstractive settings, while similarity-based methods perform robustly in relatively extractive settings. Our code and data are available at https://github.com/suhara/sourcesum.

Title: Large Language Models As Faithful Explainers

• Authors: Yu-Neng Chuang, Guanchu Wang, Chia-Yuan Chang, Ruixiang Tang, Fan Yang, Mengnan Du, Xuanting Cai, Xia Hu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04678
• Pdf URL: https://arxiv.org/pdf/2402.04678
• Copy Paste: [[2402.04678]] Large Language Models As Faithful Explainers(https://arxiv.org/abs/2402.04678)
• Keywords: generative, large language model
• Abstract: Large Language Models (LLMs) have recently become proficient in addressing complex tasks by utilizing their rich internal knowledge and reasoning ability. Consequently, this complexity hinders traditional input-focused explanation algorithms for explaining the complex decision-making processes of LLMs. Recent advancements have thus emerged for self-explaining their predictions through a single feed-forward inference in a natural language format. However, natural language explanations are often criticized for lack of faithfulness since these explanations may not accurately reflect the decision-making behaviors of the LLMs. In this work, we introduce a generative explanation framework, xLLM, to improve the faithfulness of the explanations provided in natural language formats for LLMs. Specifically, we propose an evaluator to quantify the faithfulness of natural language explanation and enhance the faithfulness by an iterative optimization process of xLLM, with the goal of maximizing the faithfulness scores. Experiments conducted on three NLU datasets demonstrate that xLLM can significantly improve the faithfulness of generated explanations, which are in alignment with the behaviors of LLMs.

Title: The Influence of Autofocus Lenses in the Camera Calibration Process

• Authors: Carlos Ricolfe-Viala, Alicia Esparza
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04686
• Pdf URL: https://arxiv.org/pdf/2402.04686
• Copy Paste: [[2402.04686]] The Influence of Autofocus Lenses in the Camera Calibration Process(https://arxiv.org/abs/2402.04686)
• Keywords: robust
• Abstract: Camera calibration is a crucial step in robotics and computer vision. Accurate camera parameters are necessary to achieve robust applications. Nowadays, camera calibration process consists of adjusting a set of data to a pin-hole model, assuming that with a reprojection error close to cero, camera parameters are correct. Since all camera parameters are unknown, computed results are considered true. However, the pin-hole model does not represent the camera behavior accurately if the focus is considered. Real cameras change the focal length slightly to obtain sharp objects in the image and this feature skews the calibration result if a unique pin-hole model is computed with a constant focal length. In this paper, a deep analysis of the camera calibration process is done to detect and strengthen its weaknesses. The camera is mounted in a robot arm to known extrinsic camera parameters with accuracy and to be able to compare computed results with the true ones. Based on the bias that exist between computed results and the true ones, a modification of the widely accepted camera calibration method using images of a planar template is presented. A pin-hole model with distance dependent focal length is proposed to improve the calibration process substantially

Title: EvoSeed: Unveiling the Threat on Deep Neural Networks with Real-World Illusions

• Authors: Shashank Kotyan, PoYuan Mao, Danilo Vasconcellos Vargas
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.04699
• Pdf URL: https://arxiv.org/pdf/2402.04699
• Copy Paste: [[2402.04699]] EvoSeed: Unveiling the Threat on Deep Neural Networks with Real-World Illusions(https://arxiv.org/abs/2402.04699)
• Keywords: robust, diffusion
• Abstract: Deep neural networks are exploited using natural adversarial samples, which have no impact on human perception but are misclassified. Current approaches often rely on the white-box nature of deep neural networks to generate these adversarial samples or alter the distribution of adversarial samples compared to training distribution. To alleviate the limitations of current approaches, we propose EvoSeed, a novel evolutionary strategy-based search algorithmic framework to generate natural adversarial samples. Our EvoSeed framework uses auxiliary Diffusion and Classifier models to operate in a model-agnostic black-box setting. We employ CMA-ES to optimize the search for an adversarial seed vector, which, when processed by the Conditional Diffusion Model, results in an unrestricted natural adversarial sample misclassified by the Classifier Model. Experiments show that generated adversarial images are of high image quality and are transferable to different classifiers. Our approach demonstrates promise in enhancing the quality of adversarial samples using evolutionary algorithms. We hope our research opens new avenues to enhance the robustness of deep neural networks in real-world scenarios. Project Website can be accessed at \url{https://shashankkotyan.github.io/EvoSeed}.

Title: Incorporating Retrieval-based Causal Learning with Information Bottlenecks for Interpretable Graph Neural Networks

• Authors: Jiahua Rao, Jiancong Xie, Hanjing Lin, Shuangjia Zheng, Zhen Wang, Yuedong Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04710
• Pdf URL: https://arxiv.org/pdf/2402.04710
• Copy Paste: [[2402.04710]] Incorporating Retrieval-based Causal Learning with Information Bottlenecks for Interpretable Graph Neural Networks(https://arxiv.org/abs/2402.04710)
• Keywords: interpretability
• Abstract: Graph Neural Networks (GNNs) have gained considerable traction for their capability to effectively process topological data, yet their interpretability remains a critical concern. Current interpretation methods are dominated by post-hoc explanations to provide a transparent and intuitive understanding of GNNs. However, they have limited performance in interpreting complicated subgraphs and can't utilize the explanation to advance GNN predictions. On the other hand, transparent GNN models are proposed to capture critical subgraphs. While such methods could improve GNN predictions, they usually don't perform well on explanations. Thus, it is desired for a new strategy to better couple GNN explanation and prediction. In this study, we have developed a novel interpretable causal GNN framework that incorporates retrieval-based causal learning with Graph Information Bottleneck (GIB) theory. The framework could semi-parametrically retrieve crucial subgraphs detected by GIB and compress the explanatory subgraphs via a causal module. The framework was demonstrated to consistently outperform state-of-the-art methods, and to achieve 32.71\% higher precision on real-world explanation scenarios with diverse explanation types. More importantly, the learned explanations were shown able to also improve GNN prediction performance.

Title: InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior

• Authors: Chenguo Lin, Yadong Mu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04717
• Pdf URL: https://arxiv.org/pdf/2402.04717
• Copy Paste: [[2402.04717]] InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior(https://arxiv.org/abs/2402.04717)
• Keywords: generative
• Abstract: Comprehending natural language instructions is a charming property for 3D indoor scene synthesis systems. Existing methods directly model object joint distributions and express object relations implicitly within a scene, thereby hindering the controllability of generation. We introduce InstructScene, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 3D scene synthesis. The proposed semantic graph prior jointly learns scene appearances and layout distributions, exhibiting versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 3D scene synthesis, we curate a high-quality dataset of scene-instruction pairs with large language and multimodal models. Extensive experimental results reveal that the proposed method surpasses existing state-of-the-art approaches by a large margin. Thorough ablation studies confirm the efficacy of crucial design components. Project page: https://chenguolin.github.io/projects/InstructScene.

Title: Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers

• Authors: Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Suvinay Subramanian, Sheng-Chun Kao, Shivani Agrawal, Utku Evci, Tushar Krishna
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2402.04744
• Pdf URL: https://arxiv.org/pdf/2402.04744
• Copy Paste: [[2402.04744]] Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers(https://arxiv.org/abs/2402.04744)
• Keywords: robust, transformer
• Abstract: N:M Structured sparsity has garnered significant interest as a result of relatively modest overhead and improved efficiency. Additionally, this form of sparsity holds considerable appeal for reducing the memory footprint owing to their modest representation overhead. There have been efforts to develop training recipes for N:M structured sparsity, they primarily focus on low-sparsity regions ($\sim$50\%). Nonetheless, performance of models trained using these approaches tends to decline when confronted with high-sparsity regions ($>$80\%). In this work, we study the effectiveness of existing sparse training recipes at \textit{high-sparsity regions} and argue that these methods fail to sustain the model quality on par with low-sparsity regions. We demonstrate that the significant factor contributing to this disparity is the presence of elevated levels of induced noise in the gradient magnitudes. To mitigate this undesirable effect, we employ decay mechanisms to progressively restrict the flow of gradients towards pruned elements. Our approach improves the model quality by up to 2$\%$ and 5$\%$ in vision and language models at high sparsity regime, respectively. We also evaluate the trade-off between model accuracy and training compute cost in terms of FLOPs. At iso-training FLOPs, our method yields better performance compared to conventional sparse training recipes, exhibiting an accuracy improvement of up to 2$\%$. The source code is available at https://github.com/abhibambhaniya/progressive_gradient_flow_nm_sparsity.

Title: Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints

• Authors: Jian Chen, Ruiyi Zhang, Yufan Zhou, Changyou Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04754
• Pdf URL: https://arxiv.org/pdf/2402.04754
• Copy Paste: [[2402.04754]] Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints(https://arxiv.org/abs/2402.04754)
• Keywords: diffusion, transformer
• Abstract: Controllable layout generation refers to the process of creating a plausible visual arrangement of elements within a graphic design (e.g., document and web designs) with constraints representing design intentions. Although recent diffusion-based models have achieved state-of-the-art FID scores, they tend to exhibit more pronounced misalignment compared to earlier transformer-based models. In this work, we propose the $\textbf{LA}$yout $\textbf{C}$onstraint diffusion mod$\textbf{E}$l (LACE), a unified model to handle a broad range of layout generation tasks, such as arranging elements with specified attributes and refining or completing a coarse layout design. The model is based on continuous diffusion models. Compared with existing methods that use discrete diffusion models, continuous state-space design can enable the incorporation of differentiable aesthetic constraint functions in training. For conditional generation, we introduce conditions via masked input. Extensive experiment results show that LACE produces high-quality layouts and outperforms existing state-of-the-art baselines.

Title: Boundary-aware Contrastive Learning for Semi-supervised Nuclei Instance Segmentation

• Authors: Ye Zhang, Ziyue Wang, Yifeng Wang, Hao Bian, Linghan Cai, Hengrui Li, Lingbo Zhang, Yongbing Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04756
• Pdf URL: https://arxiv.org/pdf/2402.04756
• Copy Paste: [[2402.04756]] Boundary-aware Contrastive Learning for Semi-supervised Nuclei Instance Segmentation(https://arxiv.org/abs/2402.04756)
• Keywords: segmentation
• Abstract: Semi-supervised segmentation methods have demonstrated promising results in natural scenarios, providing a solution to reduce dependency on manual annotation. However, these methods face significant challenges when directly applied to pathological images due to the subtle color differences between nuclei and tissues, as well as the significant morphological variations among nuclei. Consequently, the generated pseudo-labels often contain much noise, especially at the nuclei boundaries. To address the above problem, this paper proposes a boundary-aware contrastive learning network to denoise the boundary noise in a semi-supervised nuclei segmentation task. The model has two key designs: a low-resolution denoising (LRD) module and a cross-RoI contrastive learning (CRC) module. The LRD improves the smoothness of the nuclei boundary by pseudo-labels denoising, and the CRC enhances the discrimination between foreground and background by boundary feature contrastive learning. We conduct extensive experiments to demonstrate the superiority of our proposed method over existing semi-supervised instance segmentation methods.

Title: Color Recognition in Challenging Lighting Environments: CNN Approach

• Authors: Nizamuddin Maitlo, Nooruddin Noonari, Sajid Ahmed Ghanghro, Sathishkumar Duraisamy, Fayaz Ahmed
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04762
• Pdf URL: https://arxiv.org/pdf/2402.04762
• Copy Paste: [[2402.04762]] Color Recognition in Challenging Lighting Environments: CNN Approach(https://arxiv.org/abs/2402.04762)
• Keywords: robust, segmentation
• Abstract: Light plays a vital role in vision either human or machine vision, the perceived color is always based on the lighting conditions of the surroundings. Researchers are working to enhance the color detection techniques for the application of computer vision. They have implemented proposed several methods using different color detection approaches but still, there is a gap that can be filled. To address this issue, a color detection method, which is based on a Convolutional Neural Network (CNN), is proposed. Firstly, image segmentation is performed using the edge detection segmentation technique to specify the object and then the segmented object is fed to the Convolutional Neural Network trained to detect the color of an object in different lighting conditions. It is experimentally verified that our method can substantially enhance the robustness of color detection in different lighting conditions, and our method performed better results than existing methods.

Title: StableMask: Refining Causal Masking in Decoder-only Transformer

• Authors: Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, Qiang Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04779
• Pdf URL: https://arxiv.org/pdf/2402.04779
• Copy Paste: [[2402.04779]] StableMask: Refining Causal Masking in Decoder-only Transformer(https://arxiv.org/abs/2402.04779)
• Keywords: transformer
• Abstract: The decoder-only Transformer architecture with causal masking and relative position encoding (RPE) has become the de facto choice in language modeling. Despite its exceptional performance across various tasks, we have identified two limitations: First, it requires all attention scores to be non-zero and sum up to 1, even if the current embedding has sufficient self-contained information. This compels the model to assign disproportional excessive attention to specific tokens. Second, RPE-based Transformers are not universal approximators due to their limited capacity at encoding absolute positional information, which limits their application in position-critical tasks. In this work, we propose StableMask: a parameter-free method to address both limitations by refining the causal mask. It introduces pseudo-attention values to balance attention distributions and encodes absolute positional information via a progressively decreasing mask ratio. StableMask's effectiveness is validated both theoretically and empirically, showing significant enhancements in language models with parameter sizes ranging from 71M to 1.4B across diverse datasets and encoding methods. We further show that it naturally supports (1) efficient extrapolation without special tricks such as StreamingLLM and (2) easy integration with existing attention optimization techniques.

Title: Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks

• Authors: Hemanth Saratchandran, Shin-Fang Chng, Simon Lucey
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04783
• Pdf URL: https://arxiv.org/pdf/2402.04783
• Copy Paste: [[2402.04783]] Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks(https://arxiv.org/abs/2402.04783)
• Keywords: fair
• Abstract: Recently, neural networks utilizing periodic activation functions have been proven to demonstrate superior performance in vision tasks compared to traditional ReLU-activated networks. However, there is still a limited understanding of the underlying reasons for this improved performance. In this paper, we aim to address this gap by providing a theoretical understanding of periodically activated networks through an analysis of their Neural Tangent Kernel (NTK). We derive bounds on the minimum eigenvalue of their NTK in the finite width setting, using a fairly general network architecture which requires only one wide layer that grows at least linearly with the number of data samples. Our findings indicate that periodically activated networks are \textit{notably more well-behaved}, from the NTK perspective, than ReLU activated networks. Additionally, we give an application to the memorization capacity of such networks and verify our theoretical predictions empirically. Our study offers a deeper understanding of the properties of periodically activated neural networks and their potential in the field of deep learning.

Title: MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark

• Authors: Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Pan Zhou, Yao Wan, Lichao Sun
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04788
• Pdf URL: https://arxiv.org/pdf/2402.04788
• Copy Paste: [[2402.04788]] MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark(https://arxiv.org/abs/2402.04788)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have gained significant attention recently, showing remarkable potential in artificial general intelligence. However, assessing the utility of MLLMs presents considerable challenges, primarily due to the absence multimodal benchmarks that align with human preferences. Inspired by LLM-as-a-Judge in LLMs, this paper introduces a novel benchmark, termed MLLM-as-a-Judge, to assess the ability of MLLMs in assisting judges including three distinct tasks: Scoring Evaluation, Pair Comparison, and Batch Ranking. Our study reveals that, while MLLMs demonstrate remarkable human-like discernment in Pair Comparisons, there is a significant divergence from human preferences in Scoring Evaluation and Batch Ranking tasks. Furthermore, MLLMs still face challenges in judgment, including diverse biases, hallucinatory responses, and inconsistencies, even for advanced models such as GPT-4V. These findings emphasize the pressing need for enhancements and further research efforts regarding MLLMs as fully reliable evaluators. Code and dataset are available at https://github.com/Dongping-Chen/MLLM-as-a-Judge.

Title: Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformer

• Authors: Mingxaun Liu, Jiankai Tang, Haoxiang Li, Jiahao Qi, Siwei Li, Kegang Wang, Yuntao Wang, Hong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04798
• Pdf URL: https://arxiv.org/pdf/2402.04798
• Copy Paste: [[2402.04798]] Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformer(https://arxiv.org/abs/2402.04798)
• Keywords: transformer
• Abstract: Artificial neural networks (ANNs) can help camera-based remote photoplethysmography (rPPG) in measuring cardiac activity and physiological signals from facial videos, such as pulse wave, heart rate and respiration rate with better accuracy. However, most existing ANN-based methods require substantial computing resources, which poses challenges for effective deployment on mobile devices. Spiking neural networks (SNNs), on the other hand, hold immense potential for energy-efficient deep learning owing to their binary and event-driven architecture. To the best of our knowledge, we are the first to introduce SNNs into the realm of rPPG, proposing a hybrid neural network (HNN) model, the Spiking-PhysFormer, aimed at reducing power consumption. Specifically, the proposed Spiking-PhyFormer consists of an ANN-based patch embedding block, SNN-based transformer blocks, and an ANN-based predictor head. First, to simplify the transformer block while preserving its capacity to aggregate local and global spatio-temporal features, we design a parallel spike transformer block to replace sequential sub-blocks. Additionally, we propose a simplified spiking self-attention mechanism that omits the value parameter without compromising the model's performance. Experiments conducted on four datasets-PURE, UBFC-rPPG, UBFC-Phys, and MMPD demonstrate that the proposed model achieves a 12.4\% reduction in power consumption compared to PhysFormer. Additionally, the power consumption of the transformer block is reduced by a factor of 12.2, while maintaining decent performance as PhysFormer and other ANN-based models.

Title: How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data

• Authors: Mihaela Cătălina Stoian, Salijona Dyrmishi, Maxime Cordy, Thomas Lukasiewicz, Eleonora Giunchiglia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04823
• Pdf URL: https://arxiv.org/pdf/2402.04823
• Copy Paste: [[2402.04823]] How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data(https://arxiv.org/abs/2402.04823)
• Keywords: generative
• Abstract: Deep Generative Models (DGMs) have been shown to be powerful tools for generating tabular data, as they have been increasingly able to capture the complex distributions that characterize them. However, to generate realistic synthetic data, it is often not enough to have a good approximation of their distribution, as it also requires compliance with constraints that encode essential background knowledge on the problem at hand. In this paper, we address this limitation and show how DGMs for tabular data can be transformed into Constrained Deep Generative Models (C-DGMs), whose generated samples are guaranteed to be compliant with the given constraints. This is achieved by automatically parsing the constraints and transforming them into a Constraint Layer (CL) seamlessly integrated with the DGM. Our extensive experimental analysis with various DGMs and tasks reveals that standard DGMs often violate constraints, some exceeding $95\%$ non-compliance, while their corresponding C-DGMs are never non-compliant. Then, we quantitatively demonstrate that, at training time, C-DGMs are able to exploit the background knowledge expressed by the constraints to outperform their standard counterparts with up to $6.5\%$ improvement in utility and detection. Further, we show how our CL does not necessarily need to be integrated at training time, as it can be also used as a guardrail at inference time, still producing some improvements in the overall performance of the models. Finally, we show that our CL does not hinder the sample generation time of the models.

Title: SARI: Simplistic Average and Robust Identification based Noisy Partial Label Learning

• Authors: Darshana Saravanan, Naresh Manwani, Vineet Gandhi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04835
• Pdf URL: https://arxiv.org/pdf/2402.04835
• Copy Paste: [[2402.04835]] SARI: Simplistic Average and Robust Identification based Noisy Partial Label Learning(https://arxiv.org/abs/2402.04835)
• Keywords: robust
• Abstract: Partial label learning (PLL) is a weakly-supervised learning paradigm where each training instance is paired with a set of candidate labels (partial label), one of which is the true label. Noisy PLL (NPLL) relaxes this constraint by allowing some partial labels to not contain the true label, enhancing the practicality of the problem. Our work centers on NPLL and presents a minimalistic framework called SARI that initially assigns pseudo-labels to images by exploiting the noisy partial labels through a weighted nearest neighbour algorithm. These pseudo-label and image pairs are then used to train a deep neural network classifier with label smoothing and standard regularization techniques. The classifier's features and predictions are subsequently employed to refine and enhance the accuracy of pseudo-labels. SARI combines the strengths of Average Based Strategies (in pseudo labelling) and Identification Based Strategies (in classifier training) from the literature. We perform thorough experiments on seven datasets and compare SARI against nine NPLL and PLL methods from the prior art. SARI achieves state-of-the-art results in almost all studied settings, obtaining substantial gains in fine-grained classification and extreme noise settings.

Title: PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition

• Authors: Jinghui Lu, Ziwei Yang, Yanjie Wang, Xuejing Liu, Can Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04838
• Pdf URL: https://arxiv.org/pdf/2402.04838
• Copy Paste: [[2402.04838]] PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition(https://arxiv.org/abs/2402.04838)
• Keywords: generative, large language model
• Abstract: In this study, we aim to reduce generation latency for Named Entity Recognition (NER) with Large Language Models (LLMs). The main cause of high latency in LLMs is the sequential decoding process, which autoregressively generates all labels and mentions for NER, significantly increase the sequence length. To this end, we introduce Parallel Decoding in LLM for NE} (PaDeLLM-NER), a approach that integrates seamlessly into existing generative model frameworks without necessitating additional modules or architectural modifications. PaDeLLM-NER allows for the simultaneous decoding of all mentions, thereby reducing generation latency. Experiments reveal that PaDeLLM-NER significantly increases inference speed that is 1.76 to 10.22 times faster than the autoregressive approach for both English and Chinese. Simultaneously it maintains the quality of predictions as evidenced by the performance that is on par with the state-of-the-art across various datasets.

Title: Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning

• Authors: Yuxuan Bian, Xuan Ju, Jiangtong Li, Zhijian Xu, Dawei Cheng, Qiang Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04852
• Pdf URL: https://arxiv.org/pdf/2402.04852
• Copy Paste: [[2402.04852]] Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning(https://arxiv.org/abs/2402.04852)
• Keywords: large language model
• Abstract: In this study, we present aLLM4TS, an innovative framework that adapts Large Language Models (LLMs) for time-series representation learning. Central to our approach is that we reconceive time-series forecasting as a self-supervised, multi-patch prediction task, which, compared to traditional mask-and-reconstruction methods, captures temporal dynamics in patch representations more effectively. Our strategy encompasses two-stage training: (i). a causal continual pre-training phase on various time-series datasets, anchored on next patch prediction, effectively syncing LLM capabilities with the intricacies of time-series data; (ii). fine-tuning for multi-patch prediction in the targeted time-series context. A distinctive element of our framework is the patch-wise decoding layer, which departs from previous methods reliant on sequence-level decoding. Such a design directly transposes individual patches into temporal sequences, thereby significantly bolstering the model's proficiency in mastering temporal patch-based representations. aLLM4TS demonstrates superior performance in several downstream tasks, proving its effectiveness in deriving temporal representations with enhanced transferability and marking a pivotal advancement in the adaptation of LLMs for time-series analysis.

Title: Dual-Path Coupled Image Deraining Network via Spatial-Frequency Interaction

• Authors: Yuhong He, Aiwen Jiang, Lingfang Jiang, Zhifeng Wang, Lu Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04855
• Pdf URL: https://arxiv.org/pdf/2402.04855
• Copy Paste: [[2402.04855]] Dual-Path Coupled Image Deraining Network via Spatial-Frequency Interaction(https://arxiv.org/abs/2402.04855)
• Keywords: robust, extraction, transformer
• Abstract: Transformers have recently emerged as a significant force in the field of image deraining. Existing image deraining methods utilize extensive research on self-attention. Though showcasing impressive results, they tend to neglect critical frequency information, as self-attention is generally less adept at capturing high-frequency details. To overcome this shortcoming, we have developed an innovative Dual-Path Coupled Deraining Network (DPCNet) that integrates information from both spatial and frequency domains through Spatial Feature Extraction Block (SFEBlock) and Frequency Feature Extraction Block (FFEBlock). We have further introduced an effective Adaptive Fusion Module (AFM) for the dual-path feature aggregation. Extensive experiments on six public deraining benchmarks and downstream vision tasks have demonstrated that our proposed method not only outperforms the existing state-of-the-art deraining method but also achieves visually pleasuring results with excellent robustness on downstream vision tasks.

Title: Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy

• Authors: Ruichu Cai, Siyang Huang, Jie Qiao, Wei Chen, Yan Zeng, Keli Zhang, Fuchun Sun, Yang Yu, Zhifeng Hao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04869
• Pdf URL: https://arxiv.org/pdf/2402.04869
• Copy Paste: [[2402.04869]] Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy(https://arxiv.org/abs/2402.04869)
• Keywords: robust, interpretability
• Abstract: As a key component to intuitive cognition and reasoning solutions in human intelligence, causal knowledge provides great potential for reinforcement learning (RL) agents' interpretability towards decision-making by helping reduce the searching space. However, there is still a considerable gap in discovering and incorporating causality into RL, which hinders the rapid development of causal RL. In this paper, we consider explicitly modeling the generation process of states with the causal graphical model, based on which we augment the policy. We formulate the causal structure updating into the RL interaction process with active intervention learning of the environment. To optimize the derived objective, we propose a framework with theoretical performance guarantees that alternates between two steps: using interventions for causal structure learning during exploration and using the learned causal structure for policy guidance during exploitation. Due to the lack of public benchmarks that allow direct intervention in the state space, we design the root cause localization task in our simulated fault alarm environment and then empirically show the effectiveness and robustness of the proposed method against state-of-the-art baselines. Theoretical analysis shows that our performance improvement attributes to the virtuous cycle of causal-guided policy learning and causal structure learning, which aligns with our experimental results.

Title: On Provable Length and Compositional Generalization

• Authors: Kartik Ahuja, Amin Mansouri
• Subjects: cs.LG, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.04875
• Pdf URL: https://arxiv.org/pdf/2402.04875
• Copy Paste: [[2402.04875]] On Provable Length and Compositional Generalization(https://arxiv.org/abs/2402.04875)
• Keywords: transformer
• Abstract: Length generalization -- the ability to generalize to longer sequences than ones seen during training, and compositional generalization -- the ability to generalize to token combinations not seen during training, are crucial forms of out-of-distribution generalization in sequence-to-sequence models. In this work, we take the first steps towards provable length and compositional generalization for a range of architectures, including deep sets, transformers, state space models, and simple recurrent neural nets. Depending on the architecture, we prove different degrees of representation identification, e.g., a linear or a permutation relation with ground truth representation, is necessary for length and compositional generalization.

Title: STAR: Shape-focused Texture Agnostic Representations for Improved Object Detection and 6D Pose Estimation

• Authors: Peter Hönig, Stefan Thalhammer, Jean-Baptiste Weibel, Matthias Hirschmanner, Markus Vincze
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04878
• Pdf URL: https://arxiv.org/pdf/2402.04878
• Copy Paste: [[2402.04878]] STAR: Shape-focused Texture Agnostic Representations for Improved Object Detection and 6D Pose Estimation(https://arxiv.org/abs/2402.04878)
• Keywords: robust
• Abstract: Recent advances in machine learning have greatly benefited object detection and 6D pose estimation for robotic grasping. However, textureless and metallic objects still pose a significant challenge due to fewer visual cues and the texture bias of CNNs. To address this issue, we propose a texture-agnostic approach that focuses on learning from CAD models and emphasizes object shape features. To achieve a focus on learning shape features, the textures are randomized during the rendering of the training data. By treating the texture as noise, the need for real-world object instances or their final appearance during training data generation is eliminated. The TLESS and ITODD datasets, specifically created for industrial settings in robotics and featuring textureless and metallic objects, were used for evaluation. Texture agnosticity also increases the robustness against image perturbations such as imaging noise, motion blur, and brightness changes, which are common in robotics applications. Code and datasets are publicly available at github.com/hoenigpeter/randomized_texturing.

Title: Epistral Network: Revolutionizing Media Curation and Consumption through Decentralization

• Authors: Dipankar Sarkar.Shubham Upadhyay
• Subjects: cs.CR, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.04881
• Pdf URL: https://arxiv.org/pdf/2402.04881
• Copy Paste: [[2402.04881]] Epistral Network: Revolutionizing Media Curation and Consumption through Decentralization(https://arxiv.org/abs/2402.04881)
• Keywords: fair
• Abstract: Blockchain technology has revolutionized media consumption and distribution in the digital age, allowing creators, consumers, and regulators to participate in a decentralized, fair, and engaging media environment. Epistral, an innovative media network that leverages blockchain technology, aims to be the world's first anti-mimetic media curation and consumption network, addressing the core challenges facing today's digital media landscape: unfair treatment of creators and manipulative consumer algorithms, and the complex task of effective regulation. This paper delves into the conceptualization, design, and potential impact of epistral and explores how it embodies McLuhan's and Girard's theories within the realm of blockchain technology and draws from Hayden's critique of democratic representation. The paper analyzes the challenges and opportunities presented by this new network, providing a broader discourse on the future of media consumption, distribution, and regulation.

Title: Toward Accurate Camera-based 3D Object Detection via Cascade Depth Estimation and Calibration

• Authors: Chaoqun Wang, Yiran Qin, Zijian Kang, Ningning Ma, Ruimao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04883
• Pdf URL: https://arxiv.org/pdf/2402.04883
• Copy Paste: [[2402.04883]] Toward Accurate Camera-based 3D Object Detection via Cascade Depth Estimation and Calibration(https://arxiv.org/abs/2402.04883)
• Keywords: transformer
• Abstract: Recent camera-based 3D object detection is limited by the precision of transforming from image to 3D feature spaces, as well as the accuracy of object localization within the 3D space. This paper aims to address such a fundamental problem of camera-based 3D object detection: How to effectively learn depth information for accurate feature lifting and object localization. Different from previous methods which directly predict depth distributions by using a supervised estimation model, we propose a cascade framework consisting of two depth-aware learning paradigms. First, a depth estimation (DE) scheme leverages relative depth information to realize the effective feature lifting from 2D to 3D spaces. Furthermore, a depth calibration (DC) scheme introduces depth reconstruction to further adjust the 3D object localization perturbation along the depth axis. In practice, the DE is explicitly realized by using both the absolute and relative depth optimization loss to promote the precision of depth prediction, while the capability of DC is implicitly embedded into the detection Transformer through a depth denoising mechanism in the training phase. The entire model training is accomplished through an end-to-end manner. We propose a baseline detector and evaluate the effectiveness of our proposal with +2.2%/+2.7% NDS/mAP improvements on NuScenes benchmark, and gain a comparable performance with 55.9%/45.7% NDS/mAP. Furthermore, we conduct extensive experiments to demonstrate its generality based on various detectors with about +2% NDS improvements.

Title: L4Q: Parameter Efficient Quantization-Aware Training on Large Language Models via LoRA-wise LSQ

• Authors: Hyesung Jeon, Yulhwa Kim, Jae-joon Kim
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04902
• Pdf URL: https://arxiv.org/pdf/2402.04902
• Copy Paste: [[2402.04902]] L4Q: Parameter Efficient Quantization-Aware Training on Large Language Models via LoRA-wise LSQ(https://arxiv.org/abs/2402.04902)
• Keywords: large language model
• Abstract: Post-training quantization (PTQ) and quantization-aware training (QAT) methods are gaining popularity in mitigating the high memory and computational costs associated with Large Language Models (LLMs). In resource-constrained scenarios, PTQ, with its reduced training overhead, is often preferred over QAT, despite the latter's potential for higher accuracy. Meanwhile, parameter-efficient fine-tuning (PEFT) methods like low-rank adaptation (LoRA) have been introduced, and recent efforts have explored quantization-aware PEFT techniques. However, these approaches may lack generality due to their reliance on the pre-quantized model's configuration. Their effectiveness may be compromised by non-linearly quantized or mixed-precision weights, and the retraining of specific quantization parameters might impede optimal performance. To address these challenges, we propose L4Q, an algorithm for parameter-efficient quantization-aware training. L4Q leverages LoRA-wise learned quantization step size for LLMs, aiming to enhance generality. The simultaneous quantization-and-fine-tuning process of L4Q is applicable to high-precision models, yielding linearly quantized weights with superior accuracy. Our experiments, conducted on the LLaMA and LLaMA2 model families using an instructional dataset, showcase L4Q's capabilities in language comprehension and few-shot in-context learning, achieving sub-4-bit precision while maintaining comparable training times to applying PEFT on a quantized model.

Title: Towards Biologically Plausible and Private Gene Expression Data Generation

• Authors: Dingfan Chen, Marie Oestreich, Tejumade Afonja, Raouf Kerkouche, Matthias Becker, Mario Fritz
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04912
• Pdf URL: https://arxiv.org/pdf/2402.04912
• Copy Paste: [[2402.04912]] Towards Biologically Plausible and Private Gene Expression Data Generation(https://arxiv.org/abs/2402.04912)
• Keywords: privacy, generative
• Abstract: Generative models trained with Differential Privacy (DP) are becoming increasingly prominent in the creation of synthetic data for downstream applications. Existing literature, however, primarily focuses on basic benchmarking datasets and tends to report promising results only for elementary metrics and relatively simple data distributions. In this paper, we initiate a systematic analysis of how DP generative models perform in their natural application scenarios, specifically focusing on real-world gene expression data. We conduct a comprehensive analysis of five representative DP generation methods, examining them from various angles, such as downstream utility, statistical properties, and biological plausibility. Our extensive evaluation illuminates the unique characteristics of each DP generation method, offering critical insights into the strengths and weaknesses of each approach, and uncovering intriguing possibilities for future developments. Perhaps surprisingly, our analysis reveals that most methods are capable of achieving seemingly reasonable downstream utility, according to the standard evaluation metrics considered in existing literature. Nevertheless, we find that none of the DP methods are able to accurately capture the biological characteristics of the real dataset. This observation suggests a potential over-optimistic assessment of current methodologies in this field and underscores a pressing need for future enhancements in model design.

Title: Personalized Text Generation with Fine-Grained Linguistic Control

• Authors: Bashar Alhafni, Vivek Kulkarni, Dhruv Kumar, Vipul Raheja
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04914
• Pdf URL: https://arxiv.org/pdf/2402.04914
• Copy Paste: [[2402.04914]] Personalized Text Generation with Fine-Grained Linguistic Control(https://arxiv.org/abs/2402.04914)
• Keywords: generative, large language model
• Abstract: As the text generation capabilities of large language models become increasingly prominent, recent studies have focused on controlling particular aspects of the generated text to make it more personalized. However, most research on controllable text generation focuses on controlling the content or modeling specific high-level/coarse-grained attributes that reflect authors' writing styles, such as formality, domain, or sentiment. In this paper, we focus on controlling fine-grained attributes spanning multiple linguistic dimensions, such as lexical and syntactic attributes. We introduce a novel benchmark to train generative models and evaluate their ability to generate personalized text based on multiple fine-grained linguistic attributes. We systematically investigate the performance of various large language models on our benchmark and draw insights from the factors that impact their performance. We make our code, data, and pretrained models publicly available.

Title: Prompting Implicit Discourse Relation Annotation

• Authors: Frances Yung, Mansoor Ahmad, Merel Scholman, Vera Demberg
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04918
• Pdf URL: https://arxiv.org/pdf/2402.04918
• Copy Paste: [[2402.04918]] Prompting Implicit Discourse Relation Annotation(https://arxiv.org/abs/2402.04918)
• Keywords: large language model
• Abstract: Pre-trained large language models, such as ChatGPT, archive outstanding performance in various reasoning tasks without supervised training and were found to have outperformed crowdsourcing workers. Nonetheless, ChatGPT's performance in the task of implicit discourse relation classification, prompted by a standard multiple-choice question, is still far from satisfactory and considerably inferior to state-of-the-art supervised approaches. This work investigates several proven prompting techniques to improve ChatGPT's recognition of discourse relations. In particular, we experimented with breaking down the classification task that involves numerous abstract labels into smaller subtasks. Nonetheless, experiment results show that the inference accuracy hardly changes even with sophisticated prompt engineering, suggesting that implicit discourse relation classification is not yet resolvable under zero-shot or few-shot settings.

Title: Source-Free Domain Adaptation with Diffusion-Guided Source Data Generation

• Authors: Shivang Chopra, Suraj Kothawade, Houda Aynaou, Aman Chadha
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04929
• Pdf URL: https://arxiv.org/pdf/2402.04929
• Copy Paste: [[2402.04929]] Source-Free Domain Adaptation with Diffusion-Guided Source Data Generation(https://arxiv.org/abs/2402.04929)
• Keywords: diffusion
• Abstract: This paper introduces a novel approach to leverage the generalizability capability of Diffusion Models for Source-Free Domain Adaptation (DM-SFDA). Our proposed DM-SFDA method involves fine-tuning a pre-trained text-to-image diffusion model to generate source domain images using features from the target images to guide the diffusion process. Specifically, the pre-trained diffusion model is fine-tuned to generate source samples that minimize entropy and maximize confidence for the pre-trained source model. We then apply established unsupervised domain adaptation techniques to align the generated source images with target domain data. We validate our approach through comprehensive experiments across a range of datasets, including Office-31, Office-Home, and VisDA. The results highlight significant improvements in SFDA performance, showcasing the potential of diffusion models in generating contextually relevant, domain-specific images.

Title: Blue noise for diffusion models

• Authors: Xingchang Huang, Corentin Salaün, Cristina Vasconcelos, Christian Theobalt, Cengiz Öztireli, Gurprit Singh
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04930
• Pdf URL: https://arxiv.org/pdf/2402.04930
• Copy Paste: [[2402.04930]] Blue noise for diffusion models(https://arxiv.org/abs/2402.04930)
• Keywords: diffusion
• Abstract: Most of the existing diffusion models use Gaussian noise for training and sampling across all time steps, which may not optimally account for the frequency contents reconstructed by the denoising network. Despite the diverse applications of correlated noise in computer graphics, its potential for improving the training process has been underexplored. In this paper, we introduce a novel and general class of diffusion models taking correlated noise within and across images into account. More specifically, we propose a time-varying noise model to incorporate correlated noise into the training process, as well as a method for fast generation of correlated noise mask. Our model is built upon deterministic diffusion models and utilizes blue noise to help improve the generation quality compared to using Gaussian white (random) noise only. Further, our framework allows introducing correlation across images within a single mini-batch to improve gradient flow. We perform both qualitative and quantitative evaluations on a variety of datasets using our method, achieving improvements on different tasks over existing deterministic diffusion models in terms of FID metric.

Title: Cayley hashing with cookies

• Authors: Vladimir Shpilrain, Bianca Sosnovski
• Subjects: cs.CR, math.GR
• Abstract URL: https://arxiv.org/abs/2402.04943
• Pdf URL: https://arxiv.org/pdf/2402.04943
• Copy Paste: [[2402.04943]] Cayley hashing with cookies(https://arxiv.org/abs/2402.04943)
• Keywords: security
• Abstract: Cayley hash functions are based on a simple idea of using a pair of semigroup elements, A and B, to hash the 0 and 1 bit, respectively, and then to hash an arbitrary bit string in the natural way, by using multiplication of elements in the semigroup. The main advantage of Cayley hash functions compared to, say, hash functions in the SHA family is that when an already hashed document is amended, one does not have to hash the whole amended document all over again, but rather hash just the amended part and then multiply the result by the hash of the original document. Some authors argued that this may be a security hazard, specifically that this property may facilitate finding a second preimage by splitting a long bit string into shorter pieces. In this paper, we offer a way to get rid of this alleged disadvantage and keep the advantages at the same time. We call this method ``Cayley hashing with cookies" using terminology borrowed from the theory of random walks in a random environment. For the platform semigroup, we use 2x2 matrices over F_p.

Title: Reconfidencing LLMs from the Grouping Loss Perspective

• Authors: Lihu Chen, Alexandre Perez-Lebel, Fabian M. Suchanek, Gaël Varoquaux
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04957
• Pdf URL: https://arxiv.org/pdf/2402.04957
• Copy Paste: [[2402.04957]] Reconfidencing LLMs from the Grouping Loss Perspective(https://arxiv.org/abs/2402.04957)
• Keywords: large language model
• Abstract: Large Language Models (LLMs), including ChatGPT and LLaMA, are susceptible to generating hallucinated answers in a confident tone. While efforts to elicit and calibrate confidence scores have proven useful, recent findings show that controlling uncertainty must go beyond calibration: predicted scores may deviate significantly from the actual posterior probabilities due to the impact of grouping loss. In this work, we construct a new evaluation dataset derived from a knowledge base to assess confidence scores given to answers of Mistral and LLaMA. Experiments show that they tend to be overconfident. Further, we show that they are more overconfident on some answers than others, \emph{eg} depending on the nationality of the person in the query. In uncertainty-quantification theory, this is grouping loss. To address this, we propose a solution to reconfidence LLMs, canceling not only calibration but also grouping loss. The LLMs, after the reconfidencing process, indicate improved confidence alignment with the accuracy of their responses.

Title: Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation

• Authors: Pedro Vianna, Muawiz Chaudhary, Paria Mehrbod, An Tang, Guy Cloutier, Guy Wolf, Michael Eickenberg, Eugene Belilovsky
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04958
• Pdf URL: https://arxiv.org/pdf/2402.04958
• Copy Paste: [[2402.04958]] Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation(https://arxiv.org/abs/2402.04958)
• Keywords: robust
• Abstract: Deep neural networks have useful applications in many different tasks, however their performance can be severely affected by changes in the data distribution. For example, in the biomedical field, their performance can be affected by changes in the data (different machines, populations) between training and test datasets. To ensure robustness and generalization to real-world scenarios, test-time adaptation has been recently studied as an approach to adjust models to a new data distribution during inference. Test-time batch normalization is a simple and popular method that achieved compelling performance on domain shift benchmarks. It is implemented by recalculating batch normalization statistics on test batches. Prior work has focused on analysis with test data that has the same label distribution as the training data. However, in many practical applications this technique is vulnerable to label distribution shifts, sometimes producing catastrophic failure. This presents a risk in applying test time adaptation methods in deployment. We propose to tackle this challenge by only selectively adapting channels in a deep network, minimizing drastic adaptation that is sensitive to label shifts. Our selection scheme is based on two principles that we empirically motivate: (1) later layers of networks are more sensitive to label shift (2) individual features can be sensitive to specific classes. We apply the proposed technique to three classification tasks, including CIFAR10-C, Imagenet-C, and diagnosis of fatty liver, where we explore both covariate and label distribution shifts. We find that our method allows to bring the benefits of TTA while significantly reducing the risk of failure common in other methods, while being robust to choice in hyperparameters.

Title: ConvLoRA and AdaBN based Domain Adaptation via Self-Training

• Authors: Sidra Aleem, Julia Dietlmeier, Eric Arazo, Suzanne Little
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04964
• Pdf URL: https://arxiv.org/pdf/2402.04964
• Copy Paste: [[2402.04964]] ConvLoRA and AdaBN based Domain Adaptation via Self-Training(https://arxiv.org/abs/2402.04964)
• Keywords: segmentation
• Abstract: Existing domain adaptation (DA) methods often involve pre-training on the source domain and fine-tuning on the target domain. For multi-target domain adaptation, having a dedicated/separate fine-tuned network for each target domain, that retain all the pre-trained model parameters, is prohibitively expensive. To address this limitation, we propose Convolutional Low-Rank Adaptation (ConvLoRA). ConvLoRA freezes pre-trained model weights, adds trainable low-rank decomposition matrices to convolutional layers, and backpropagates the gradient through these matrices thus greatly reducing the number of trainable parameters. To further boost adaptation, we utilize Adaptive Batch Normalization (AdaBN) which computes target-specific running statistics and use it along with ConvLoRA. Our method has fewer trainable parameters and performs better or on-par with large independent fine-tuned networks (with less than 0.9% trainable parameters of the total base model) when tested on the segmentation of Calgary-Campinas dataset containing brain MRI images. Our approach is simple, yet effective and can be applied to any deep learning-based architecture which uses convolutional and batch normalization layers. Code is available at: https://github.com/aleemsidra/ConvLoRA.

Title: An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration

• Authors: Yihao Li, Ru Zhang, Jianyi Liu, Gongshen Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04978
• Pdf URL: https://arxiv.org/pdf/2402.04978
• Copy Paste: [[2402.04978]] An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration(https://arxiv.org/abs/2402.04978)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) demonstrate exceptional performance in a multitude of Natural Language Processing (NLP) tasks, they encounter challenges in practical applications, including issues with hallucinations, inadequate knowledge updating, and limited transparency in the reasoning process. To overcome these limitations, this study innovatively proposes a collaborative training-free reasoning scheme involving tight cooperation between Knowledge Graph (KG) and LLMs. This scheme first involves using LLMs to iteratively explore KG, selectively retrieving a task-relevant knowledge subgraph to support reasoning. The LLMs are then guided to further combine inherent implicit knowledge to reason on the subgraph while explicitly elucidating the reasoning process. Through such a cooperative approach, our scheme achieves more reliable knowledge-based reasoning and facilitates the tracing of the reasoning results. Experimental results show that our scheme significantly progressed across multiple datasets, notably achieving over a 10% improvement on the QALD10 dataset compared to the best baseline and the fine-tuned state-of-the-art (SOTA) work. Building on this success, this study hopes to offer a valuable reference for future research in the fusion of KG and LLMs, thereby enhancing LLMs' proficiency in solving complex issues.

Title: Beyond explaining: XAI-based Adaptive Learning with SHAP Clustering for Energy Consumption Prediction

• Authors: Tobias Clement, Hung Truong Thanh Nguyen, Nils Kemmerzell, Mohamed Abdelaal, Davor Stjelja
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2402.04982
• Pdf URL: https://arxiv.org/pdf/2402.04982
• Copy Paste: [[2402.04982]] Beyond explaining: XAI-based Adaptive Learning with SHAP Clustering for Energy Consumption Prediction(https://arxiv.org/abs/2402.04982)
• Keywords: robust
• Abstract: This paper presents an approach integrating explainable artificial intelligence (XAI) techniques with adaptive learning to enhance energy consumption prediction models, with a focus on handling data distribution shifts. Leveraging SHAP clustering, our method provides interpretable explanations for model predictions and uses these insights to adaptively refine the model, balancing model complexity with predictive performance. We introduce a three-stage process: (1) obtaining SHAP values to explain model predictions, (2) clustering SHAP values to identify distinct patterns and outliers, and (3) refining the model based on the derived SHAP clustering characteristics. Our approach mitigates overfitting and ensures robustness in handling data distribution shifts. We evaluate our method on a comprehensive dataset comprising energy consumption records of buildings, as well as two additional datasets to assess the transferability of our approach to other domains, regression, and classification problems. Our experiments demonstrate the effectiveness of our approach in both task types, resulting in improved predictive performance and interpretable model explanations.

Title: PriorBoost: An Adaptive Algorithm for Learning from Aggregate Responses

• Authors: Adel Javanmard, Matthew Fahrbach, Vahab Mirrokni
• Subjects: cs.LG, cs.DS
• Abstract URL: https://arxiv.org/abs/2402.04987
• Pdf URL: https://arxiv.org/pdf/2402.04987
• Copy Paste: [[2402.04987]] PriorBoost: An Adaptive Algorithm for Learning from Aggregate Responses(https://arxiv.org/abs/2402.04987)
• Keywords: privacy
• Abstract: This work studies algorithms for learning from aggregate responses. We focus on the construction of aggregation sets (called bags in the literature) for event-level loss functions. We prove for linear regression and generalized linear models (GLMs) that the optimal bagging problem reduces to one-dimensional size-constrained $k$-means clustering. Further, we theoretically quantify the advantage of using curated bags over random bags. We then propose the PriorBoost algorithm, which adaptively forms bags of samples that are increasingly homogeneous with respect to (unobserved) individual responses to improve model quality. We study label differential privacy for aggregate learning, and we also provide extensive experiments showing that PriorBoost regularly achieves optimal model quality for event-level predictions, in stark contrast to non-adaptive algorithms.

Title: Pedagogical Alignment of Large Language Models

• Authors: Shashank Sonkar, Kangqi Ni, Sapana Chaudhary, Richard G. Baraniuk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.05000
• Pdf URL: https://arxiv.org/pdf/2402.05000
• Copy Paste: [[2402.05000]] Pedagogical Alignment of Large Language Models(https://arxiv.org/abs/2402.05000)
• Keywords: large language model
• Abstract: In this paper, we introduce the novel concept of pedagogically aligned Large Language Models (LLMs) that signifies a transformative shift in the application of LLMs within educational contexts. Rather than providing direct responses to user queries, pedagogically-aligned LLMs function as scaffolding tools, breaking complex problems into manageable subproblems and guiding students towards the final answer through constructive feedback and hints. The objective is to equip learners with problem-solving strategies that deepen their understanding and internalization of the subject matter. Previous research in this field has primarily applied the supervised finetuning approach without framing the objective as an alignment problem, hence not employing reinforcement learning through human feedback (RLHF) methods. This study reinterprets the narrative by viewing the task through the lens of alignment and demonstrates how RLHF methods emerge naturally as a superior alternative for aligning LLM behaviour. Building on this perspective, we propose a novel approach for constructing a reward dataset specifically designed for the pedagogical alignment of LLMs. We apply three state-of-the-art RLHF algorithms and find that they outperform SFT significantly. Our qualitative analyses across model differences and hyperparameter sensitivity further validate the superiority of RLHF over SFT. Also, our study sheds light on the potential of online feedback for enhancing the performance of pedagogically-aligned LLMs, thus providing valuable insights for the advancement of these models in educational settings.

Title: Example-based Explanations for Random Forests using Machine Unlearning

• Authors: Tanmay Surve, Romila Pradhan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.05007
• Pdf URL: https://arxiv.org/pdf/2402.05007
• Copy Paste: [[2402.05007]] Example-based Explanations for Random Forests using Machine Unlearning(https://arxiv.org/abs/2402.05007)
• Keywords: fair
• Abstract: Tree-based machine learning models, such as decision trees and random forests, have been hugely successful in classification tasks primarily because of their predictive power in supervised learning tasks and ease of interpretation. Despite their popularity and power, these models have been found to produce unexpected or discriminatory outcomes. Given their overwhelming success for most tasks, it is of interest to identify sources of their unexpected and discriminatory behavior. However, there has not been much work on understanding and debugging tree-based classifiers in the context of fairness. We introduce FairDebugger, a system that utilizes recent advances in machine unlearning research to identify training data subsets responsible for instances of fairness violations in the outcomes of a random forest classifier. FairDebugger generates top-$k$ explanations (in the form of coherent training data subsets) for model unfairness. Toward this goal, FairDebugger first utilizes machine unlearning to estimate the change in the tree structures of the random forest when parts of the underlying training data are removed, and then leverages the Apriori algorithm from frequent itemset mining to reduce the subset search space. We empirically evaluate our approach on three real-world datasets, and demonstrate that the explanations generated by FairDebugger are consistent with insights from prior studies on these datasets.

Title: A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?

• Authors: Agustinus Kristiadi, Felix Strieth-Kalthoff, Marta Skreta, Pascal Poupart, Alán Aspuru-Guzik, Geoff Pleiss
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05015
• Pdf URL: https://arxiv.org/pdf/2402.05015
• Copy Paste: [[2402.05015]] A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?(https://arxiv.org/abs/2402.05015)
• Keywords: large language model
• Abstract: Automation is one of the cornerstones of contemporary material discovery. Bayesian optimization (BO) is an essential part of such workflows, enabling scientists to leverage prior domain knowledge into efficient exploration of a large molecular space. While such prior knowledge can take many forms, there has been significant fanfare around the ancillary scientific knowledge encapsulated in large language models (LLMs). However, existing work thus far has only explored LLMs for heuristic materials searches. Indeed, recent work obtains the uncertainty estimate -- an integral part of BO -- from point-estimated, non-Bayesian LLMs. In this work, we study the question of whether LLMs are actually useful to accelerate principled Bayesian optimization in the molecular space. We take a sober, dispassionate stance in answering this question. This is done by carefully (i) viewing LLMs as fixed feature extractors for standard but principled BO surrogate models and by (ii) leveraging parameter-efficient finetuning methods and Bayesian neural networks to obtain the posterior of the LLM surrogate. Our extensive experiments with real-world chemistry problems show that LLMs can be useful for BO over molecules, but only if they have been pretrained or finetuned with domain-specific data.

Title: Simulated Overparameterization

• Authors: Hanna Mazzawi, Pranjal Awasthi, Xavi Gonzalvo, Srikumar Ramalingam
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05033
• Pdf URL: https://arxiv.org/pdf/2402.05033
• Copy Paste: [[2402.05033]] Simulated Overparameterization(https://arxiv.org/abs/2402.05033)
• Keywords: transformer
• Abstract: In this work, we introduce a novel paradigm called Simulated Overparametrization (SOP). SOP merges the computational efficiency of compact models with the advanced learning proficiencies of overparameterized models. SOP proposes a unique approach to model training and inference, where a model with a significantly larger number of parameters is trained in such a way that a smaller, efficient subset of these parameters is used for the actual computation during inference. Building upon this framework, we present a novel, architecture agnostic algorithm called "majority kernels", which seamlessly integrates with predominant architectures, including Transformer models. Majority kernels enables the simulated training of overparameterized models, resulting in performance gains across architectures and tasks. Furthermore, our approach adds minimal overhead to the cost incurred (wall clock time) at training time. The proposed approach shows strong performance on a wide variety of datasets and models, even outperforming strong baselines such as combinatorial optimization methods based on submodular optimization.

Title: A Survey on Domain Generalization for Medical Image Analysis

• Authors: Ziwei Niu, Shuyi Ouyang, Shiao Xie, Yen-wei Chen, Lanfen Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05035
• Pdf URL: https://arxiv.org/pdf/2402.05035
• Copy Paste: [[2402.05035]] A Survey on Domain Generalization for Medical Image Analysis(https://arxiv.org/abs/2402.05035)
• Keywords: robust
• Abstract: Medical Image Analysis (MedIA) has emerged as a crucial tool in computer-aided diagnosis systems, particularly with the advancement of deep learning (DL) in recent years. However, well-trained deep models often experience significant performance degradation when deployed in different medical sites, modalities, and sequences, known as a domain shift issue. In light of this, Domain Generalization (DG) for MedIA aims to address the domain shift challenge by generalizing effectively and performing robustly across unknown data distributions. This paper presents the a comprehensive review of substantial developments in this area. First, we provide a formal definition of domain shift and domain generalization in medical field, and discuss several related settings. Subsequently, we summarize the recent methods from three viewpoints: data manipulation level, feature representation level, and model training level, and present some algorithms in detail for each viewpoints. Furthermore, we introduce the commonly used datasets. Finally, we summarize existing literature and present some potential research topics for the future. For this survey, we also created a GitHub project by collecting the supporting resources, at the link: https://github.com/Ziwei-Niu/DG_for_MedIA

Title: Sticky Fingers: Resilience of Satellite Fingerprinting against Jamming Attacks

• Authors: Joshua Smailes, Edd Salkield, Sebastian Köhler, Simon Birnbach, Martin Strohmeier, Ivan Martinovic
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.05042
• Pdf URL: https://arxiv.org/pdf/2402.05042
• Copy Paste: [[2402.05042]] Sticky Fingers: Resilience of Satellite Fingerprinting against Jamming Attacks(https://arxiv.org/abs/2402.05042)
• Keywords: secure, security, defense, attack
• Abstract: In the wake of increasing numbers of attacks on radio communication systems, a range of techniques are being deployed to increase the security of these systems. One such technique is radio fingerprinting, in which the transmitter can be identified and authenticated by observing small hardware differences expressed in the signal. Fingerprinting has been explored in particular in the defense of satellite systems, many of which are insecure and cannot be retrofitted with cryptographic security. In this paper, we evaluate the effectiveness of radio fingerprinting techniques under interference and jamming attacks, usually intended to deny service. By taking a pre-trained fingerprinting model and gathering a new dataset in which different levels of Gaussian noise and tone jamming have been added to the legitimate signal, we assess the attacker power required in order to disrupt the transmitter fingerprint such that it can no longer be recognized. We compare this to Gaussian jamming on the data portion of the signal, obtaining the remarkable result that transmitter fingerprints are still recognizable even in the presence of moderate levels of noise. Through deeper analysis of the results, we conclude that it takes a similar amount of jamming power in order to disrupt the fingerprint as it does to jam the message contents itself, so it is safe to include a fingerprinting system to authenticate satellite communication without opening up the system to easier denial-of-service attacks.

Title: SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models

• Authors: Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, Jing Shao
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05044
• Pdf URL: https://arxiv.org/pdf/2402.05044
• Copy Paste: [[2402.05044]] SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models(https://arxiv.org/abs/2402.05044)
• Keywords: defense, attack, robust, large language model
• Abstract: In the rapidly evolving landscape of Large Language Models (LLMs), ensuring robust safety measures is paramount. To meet this crucial need, we propose \emph{SALAD-Bench}, a safety benchmark specifically designed for evaluating LLMs, attack, and defense methods. Distinguished by its breadth, SALAD-Bench transcends conventional benchmarks through its large scale, rich diversity, intricate taxonomy spanning three levels, and versatile functionalities.SALAD-Bench is crafted with a meticulous array of questions, from standard queries to complex ones enriched with attack, defense modifications and multiple-choice. To effectively manage the inherent complexity, we introduce an innovative evaluators: the LLM-based MD-Judge for QA pairs with a particular focus on attack-enhanced queries, ensuring a seamless, and reliable evaluation. Above components extend SALAD-Bench from standard LLM safety evaluation to both LLM attack and defense methods evaluation, ensuring the joint-purpose utility. Our extensive experiments shed light on the resilience of LLMs against emerging threats and the efficacy of contemporary defense tactics. Data and evaluator are released under \url{https://github.com/OpenSafetyLab/SALAD-BENCH}. Warning: this paper includes examples that may be offensive or harmful.

Title: Federated Learning Can Find Friends That Are Beneficial

• Authors: Nazarii Tupitsa, Samuel Horváth, Martin Takáč, Eduard Gorbunov
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2402.05050
• Pdf URL: https://arxiv.org/pdf/2402.05050
• Copy Paste: [[2402.05050]] Federated Learning Can Find Friends That Are Beneficial(https://arxiv.org/abs/2402.05050)
• Keywords: federate
• Abstract: In Federated Learning (FL), the distributed nature and heterogeneity of client data present both opportunities and challenges. While collaboration among clients can significantly enhance the learning process, not all collaborations are beneficial; some may even be detrimental. In this study, we introduce a novel algorithm that assigns adaptive aggregation weights to clients participating in FL training, identifying those with data distributions most conducive to a specific learning objective. We demonstrate that our aggregation method converges no worse than the method that aggregates only the updates received from clients with the same data distribution. Furthermore, empirical evaluations consistently reveal that collaborations guided by our algorithm outperform traditional FL approaches. This underscores the critical role of judicious client selection and lays the foundation for more streamlined and effective FL implementations in the coming years.

Title: LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation

• Authors: Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05054
• Pdf URL: https://arxiv.org/pdf/2402.05054
• Copy Paste: [[2402.05054]] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation(https://arxiv.org/abs/2402.05054)
• Keywords: diffusion
• Abstract: 3D content creation has achieved significant progress in terms of both quality and speed. Although current feed-forward models can produce 3D objects in seconds, their resolution is constrained by the intensive computation required during training. In this paper, we introduce Large Multi-View Gaussian Model (LGM), a novel framework designed to generate high-resolution 3D models from text prompts or single-view images. Our key insights are two-fold: 1) 3D Representation: We propose multi-view Gaussian features as an efficient yet powerful representation, which can then be fused together for differentiable rendering. 2) 3D Backbone: We present an asymmetric U-Net as a high-throughput backbone operating on multi-view images, which can be produced from text or single-view image input by leveraging multi-view diffusion models. Extensive experiments demonstrate the high fidelity and efficiency of our approach. Notably, we maintain the fast speed to generate 3D objects within 5 seconds while boosting the training resolution to 512, thereby achieving high-resolution 3D content generation.

Title: NITO: Neural Implicit Fields for Resolution-free Topology Optimization

• Authors: Amin Heyrani Nobari, Giorgio Giannone, Lyle Regenwetter, Faez Ahmed
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2402.05073
• Pdf URL: https://arxiv.org/pdf/2402.05073
• Copy Paste: [[2402.05073]] NITO: Neural Implicit Fields for Resolution-free Topology Optimization(https://arxiv.org/abs/2402.05073)
• Keywords: diffusion
• Abstract: Topology optimization is a critical task in engineering design, where the goal is to optimally distribute material in a given space for maximum performance. We introduce Neural Implicit Topology Optimization (NITO), a novel approach to accelerate topology optimization problems using deep learning. NITO stands out as one of the first frameworks to offer a resolution-free and domain-agnostic solution in deep learning-based topology optimization. NITO synthesizes structures with up to seven times better structural efficiency compared to SOTA diffusion models and does so in a tenth of the time. In the NITO framework, we introduce a novel method, the Boundary Point Order-Invariant MLP (BPOM), to represent boundary conditions in a sparse and domain-agnostic manner, moving away from expensive simulation-based approaches. Crucially, NITO circumvents the domain and resolution limitations that restrict Convolutional Neural Network (CNN) models to a structured domain of fixed size -- limitations that hinder the widespread adoption of CNNs in engineering applications. This generalizability allows a single NITO model to train and generate solutions in countless domains, eliminating the need for numerous domain-specific CNNs and their extensive datasets. Despite its generalizability, NITO outperforms SOTA models even in specialized tasks, is an order of magnitude smaller, and is practically trainable at high resolutions that would be restrictive for CNNs. This combination of versatility, efficiency, and performance underlines NITO's potential to transform the landscape of engineering design optimization problems through implicit fields.

Title: On diffusion models for amortized inference: Benchmarking and improving stochastic control and sampling

• Authors: Marcin Sendera, Minsu Kim, Sarthak Mittal, Pablo Lemos, Luca Scimeca, Jarrid Rector-Brooks, Alexandre Adam, Yoshua Bengio, Nikolay Malkin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.05098
• Pdf URL: https://arxiv.org/pdf/2402.05098
• Copy Paste: [[2402.05098]] On diffusion models for amortized inference: Benchmarking and improving stochastic control and sampling(https://arxiv.org/abs/2402.05098)
• Keywords: diffusion, generative
• Abstract: We study the problem of training diffusion models to sample from a distribution with a given unnormalized density or energy function. We benchmark several diffusion-structured inference methods, including simulation-based variational approaches and off-policy methods (continuous generative flow networks). Our results shed light on the relative advantages of existing algorithms while bringing into question some claims from past work. We also propose a novel exploration strategy for off-policy methods, based on local search in the target space with the use of a replay buffer, and show that it improves the quality of samples on a variety of target distributions. Our code for the sampling methods and benchmarks studied is made public at https://github.com/GFNOrg/gfn-diffusion as a base for future work on diffusion models for amortized inference.

Title: Hydragen: High-Throughput LLM Inference with Shared Prefixes

• Authors: Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, Azalia Mirhoseini
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05099
• Pdf URL: https://arxiv.org/pdf/2402.05099
• Copy Paste: [[2402.05099]] Hydragen: High-Throughput LLM Inference with Shared Prefixes(https://arxiv.org/abs/2402.05099)
• Keywords: transformer, large language model
• Abstract: Transformer-based large language models (LLMs) are now deployed to hundreds of millions of users. LLM inference is commonly performed on batches of sequences that share a prefix, such as few-shot examples or a chatbot system prompt. Decoding in this large-batch setting can be bottlenecked by the attention operation, which reads large key-value (KV) caches from memory and computes inefficient matrix-vector products for every sequence in the batch. In this work, we introduce Hydragen, a hardware-aware exact implementation of attention with shared prefixes. Hydragen computes attention over the shared prefix and unique suffixes separately. This decomposition enables efficient prefix attention by batching queries together across sequences, reducing redundant memory reads and enabling the use of hardware-friendly matrix multiplications. Our method can improve end-to-end LLM throughput by up to 32x against competitive baselines, with speedup growing with the batch size and shared prefix length. Hydragen also enables the use of very long shared contexts: with a high batch size, increasing the prefix length from 1K to 16K tokens decreases Hydragen throughput by less than 15%, while the throughput of baselines drops by over 90%. Hydragen generalizes beyond simple prefix-suffix decomposition and can be applied to tree-based prompt sharing patterns, allowing us to further reduce inference time on competitive programming problems by 55%.

Title: Image captioning for Brazilian Portuguese using GRIT model

• Authors: Rafael Silva de Alencar, William Alberto Cruz Castañeda, Marcellus Amadeus
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.05106
• Pdf URL: https://arxiv.org/pdf/2402.05106
• Copy Paste: [[2402.05106]] Image captioning for Brazilian Portuguese using GRIT model(https://arxiv.org/abs/2402.05106)
• Keywords: transformer
• Abstract: This work presents the early development of a model of image captioning for the Brazilian Portuguese language. We used the GRIT (Grid - and Region-based Image captioning Transformer) model to accomplish this work. GRIT is a Transformer-only neural architecture that effectively utilizes two visual features to generate better captions. The GRIT method emerged as a proposal to be a more efficient way to generate image captioning. In this work, we adapt the GRIT model to be trained in a Brazilian Portuguese dataset to have an image captioning method for the Brazilian Portuguese Language.

Title: Opening the AI black box: program synthesis via mechanistic interpretability

• Authors: Eric J. Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukelić, Max Tegmark
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05110
• Pdf URL: https://arxiv.org/pdf/2402.05110
• Copy Paste: [[2402.05110]] Opening the AI black box: program synthesis via mechanistic interpretability(https://arxiv.org/abs/2402.05110)
• Keywords: interpretability, large language model
• Abstract: We present MIPS, a novel method for program synthesis based on automated mechanistic interpretability of neural networks trained to perform the desired task, auto-distilling the learned algorithm into Python code. We test MIPS on a benchmark of 62 algorithmic tasks that can be learned by an RNN and find it highly complementary to GPT-4: MIPS solves 32 of them, including 13 that are not solved by GPT-4 (which also solves 30). MIPS uses an integer autoencoder to convert the RNN into a finite state machine, then applies Boolean or integer symbolic regression to capture the learned algorithm. As opposed to large language models, this program synthesis technique makes no use of (and is therefore not limited by) human training data such as algorithms and code from GitHub. We discuss opportunities and challenges for scaling up this approach to make machine-learned models more interpretable and trustworthy.