2024-02-23

Title: VN Network: Embedding Newly Emerging Entities with Virtual Neighbors

• Authors: Yongquan He, Zihan Wang, Peng Zhang, Zhaopeng Tu, Zhaochun Ren
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14033
• Pdf URL: https://arxiv.org/pdf/2402.14033
• Copy Paste: [[2402.14033]] VN Network: Embedding Newly Emerging Entities with Virtual Neighbors(https://arxiv.org/abs/2402.14033)
• Keywords: robust
• Abstract: Embedding entities and relations into continuous vector spaces has attracted a surge of interest in recent years. Most embedding methods assume that all test entities are available during training, which makes it time-consuming to retrain embeddings for newly emerging entities. To address this issue, recent works apply the graph neural network on the existing neighbors of the unseen entities. In this paper, we propose a novel framework, namely Virtual Neighbor (VN) network, to address three key challenges. Firstly, to reduce the neighbor sparsity problem, we introduce the concept of the virtual neighbors inferred by rules. And we assign soft labels to these neighbors by solving a rule-constrained problem, rather than simply regarding them as unquestionably true. Secondly, many existing methods only use one-hop or two-hop neighbors for aggregation and ignore the distant information that may be helpful. Instead, we identify both logic and symmetric path rules to capture complex patterns. Finally, instead of one-time injection of rules, we employ an iterative learning scheme between the embedding method and virtual neighbor prediction to capture the interactions within. Experimental results on two knowledge graph completion tasks demonstrate that our VN network significantly outperforms state-of-the-art baselines. Furthermore, results on Subject/Object-R show that our proposed VN network is highly robust to the neighbor sparsity problem.

Title: Protect and Extend -- Using GANs for Synthetic Data Generation of Time-Series Medical Records

• Authors: Navid Ashrafi, Vera Schmitt, Robert P. Spang, Sebastian Möller, Jan-Niklas Voigt-Antons
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.14042
• Pdf URL: https://arxiv.org/pdf/2402.14042
• Copy Paste: [[2402.14042]] Protect and Extend -- Using GANs for Synthetic Data Generation of Time-Series Medical Records(https://arxiv.org/abs/2402.14042)
• Keywords: privacy, protect, attack, robust, membership infer, generative
• Abstract: Preservation of private user data is of paramount importance for high Quality of Experience (QoE) and acceptability, particularly with services treating sensitive data, such as IT-based health services. Whereas anonymization techniques were shown to be prone to data re-identification, synthetic data generation has gradually replaced anonymization since it is relatively less time and resource-consuming and more robust to data leakage. Generative Adversarial Networks (GANs) have been used for generating synthetic datasets, especially GAN frameworks adhering to the differential privacy phenomena. This research compares state-of-the-art GAN-based models for synthetic data generation to generate time-series synthetic medical records of dementia patients which can be distributed without privacy concerns. Predictive modeling, autocorrelation, and distribution analysis are used to assess the Quality of Generating (QoG) of the generated data. The privacy preservation of the respective models is assessed by applying membership inference attacks to determine potential data leakage risks. Our experiments indicate the superiority of the privacy-preserving GAN (PPGAN) model over other models regarding privacy preservation while maintaining an acceptable level of QoG. The presented results can support better data protection for medical use cases in the future.

Title: Generative Adversarial Models for Extreme Downscaling of Climate Datasets

• Authors: Guiye Li, Guofeng Cao
• Subjects: cs.LG, cs.AI, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2402.14049
• Pdf URL: https://arxiv.org/pdf/2402.14049
• Copy Paste: [[2402.14049]] Generative Adversarial Models for Extreme Downscaling of Climate Datasets(https://arxiv.org/abs/2402.14049)
• Keywords: robust, generative
• Abstract: Addressing the challenges of climate change requires accurate and high-resolution mapping of climate and weather variables. However, many existing climate datasets, such as the gridded outputs of the state-of-the-art numerical climate models (e.g., general circulation models), are only available at very coarse spatial resolutions due to the model complexity and extremely high computational demand. Deep-learning-based methods, particularly generative adversarial networks (GANs) and their variants, have proved effective for refining natural images, and have shown great promise in improving scientific datasets. In this paper, we describe a conditional GAN-based geospatial downscaling method for extreme downscaling of gridded climate datasets. Compared to most existing methods, the method can generate high-resolution accurate climate datasets from very low-resolution inputs. More importantly, the method explicitly considers the uncertainty inherent to the downscaling process that tends to be ignored in existing methods. Given an input, the method can produce a multitude of plausible high-resolution samples instead of one single deterministic result. These samples allow for an empirical exploration and inferences of model uncertainty and robustness. With a case study of gridded climate datasets (wind velocity and solar irradiance), we demonstrate the performances of the framework in downscaling tasks with very high scaling factors (up to $64\times$) and highlight the advantages of the framework with a comprehensive comparison with commonly used downscaling methods, including area-to-point (ATP) kriging, deep image prior (DIP), enhanced deep super-resolution network (EDSR), enhanced super-resolution generative adversarial networks (ESRGAN), and physics-informed resolution-enhancing GAN (PhIRE GAN).

Title: Robust Learning of Noisy Time Series Collections Using Stochastic Process Models with Motion Codes

• Authors: Chandrajit Bajaj, Minh Nguyen
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.14081
• Pdf URL: https://arxiv.org/pdf/2402.14081
• Copy Paste: [[2402.14081]] Robust Learning of Noisy Time Series Collections Using Stochastic Process Models with Motion Codes(https://arxiv.org/abs/2402.14081)
• Keywords: robust
• Abstract: While time series classification and forecasting problems have been extensively studied, the cases of noisy time series data with arbitrary time sequence lengths have remained challenging. Each time series instance can be thought of as a sample realization of a noisy dynamical model, which is characterized by a continuous stochastic process. For many applications, the data are mixed and consist of several types of noisy time series sequences modeled by multiple stochastic processes, making the forecasting and classification tasks even more challenging. Instead of regressing data naively and individually to each time series type, we take a latent variable model approach using a mixtured Gaussian processes with learned spectral kernels. More specifically, we auto-assign each type of noisy time series data a signature vector called its motion code. Then, conditioned on each assigned motion code, we infer a sparse approximation of the corresponding time series using the concept of the most informative timestamps. Our unmixing classification approach involves maximizing the likelihood across all the mixed noisy time series sequences of varying lengths. This stochastic approach allows us to learn not only within a single type of noisy time series data but also across many underlying stochastic processes, giving us a way to learn multiple dynamical models in an integrated and robust manner. The different learned latent stochastic models allow us to generate specific sub-type forecasting. We provide several quantitative comparisons demonstrating the performance of our approach.

Title: LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexicons

• Authors: Zheng-Xin Yong, Cristina Menghini, Stephen H. Bach
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14086
• Pdf URL: https://arxiv.org/pdf/2402.14086
• Copy Paste: [[2402.14086]] LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexicons(https://arxiv.org/abs/2402.14086)
• Keywords: large language model
• Abstract: Data scarcity in low-resource languages can be addressed with word-to-word translations from labeled task data in high-resource languages using bilingual lexicons. However, bilingual lexicons often have limited lexical overlap with task data, which results in poor translation coverage and lexicon utilization. We propose lexicon-conditioned data generation (LexC-Gen), a method that generates low-resource-language classification task data at scale. Specifically, LexC-Gen first uses high-resource-language words from bilingual lexicons to generate lexicon-compatible task data, and then it translates them into low-resource languages with bilingual lexicons via word translation. Across 17 extremely low-resource languages, LexC-Gen generated data is competitive with expert-translated gold data, and yields on average 5.6 and 8.9 points improvement over existing lexicon-based word translation methods on sentiment analysis and topic classification tasks respectively. We show that conditioning on bilingual lexicons is the key component of LexC-Gen. LexC-Gen is also practical -- it only needs a single GPU to generate data at scale. It works well with open-access LLMs, and its cost is one-fifth of the cost of GPT4-based multilingual data generation.

Title: Zero-shot generalization across architectures for visual classification

• Authors: Evan Gerrtiz, Luciano Dyballa, Steven W. Zucker
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14095
• Pdf URL: https://arxiv.org/pdf/2402.14095
• Copy Paste: [[2402.14095]] Zero-shot generalization across architectures for visual classification(https://arxiv.org/abs/2402.14095)
• Keywords: transformer
• Abstract: Generalization to unseen data is a key desideratum for deep networks, but its relation to classification accuracy is unclear. Using a minimalist vision dataset and a measure of generalizability, we show that popular networks, from deep convolutional networks (CNNs) to transformers, vary in their power to extrapolate to unseen classes both across layers and across architectures. Accuracy is not a good predictor of generalizability, and generalization varies non-monotonically with layer depth. Code is available at https://github.com/dyballa/zero-shot-generalization.

Title: Multi-organ Self-supervised Contrastive Learning for Breast Lesion Segmentation

• Authors: Hugo Figueiras, Helena Aidos, Nuno Cruz Garcia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14114
• Pdf URL: https://arxiv.org/pdf/2402.14114
• Copy Paste: [[2402.14114]] Multi-organ Self-supervised Contrastive Learning for Breast Lesion Segmentation(https://arxiv.org/abs/2402.14114)
• Keywords: segmentation
• Abstract: Self-supervised learning has proven to be an effective way to learn representations in domains where annotated labels are scarce, such as medical imaging. A widely adopted framework for this purpose is contrastive learning and it has been applied to different scenarios. This paper seeks to advance our understanding of the contrastive learning framework by exploring a novel perspective: employing multi-organ datasets for pre-training models tailored to specific organ-related target tasks. More specifically, our target task is breast tumour segmentation in ultrasound images. The pre-training datasets include ultrasound images from other organs, such as the lungs and heart, and large datasets of natural images. Our results show that conventional contrastive learning pre-training improves performance compared to supervised baseline approaches. Furthermore, our pre-trained models achieve comparable performance when fine-tuned with only half of the available labelled data. Our findings also show the advantages of pre-training on diverse organ data for improving performance in the downstream task.

Title: FanOutQA: Multi-Hop, Multi-Document Question Answering for Large Language Models

• Authors: Andrew Zhu, Alyssa Hwang, Liam Dugan, Chris Callison-Burch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14116
• Pdf URL: https://arxiv.org/pdf/2402.14116
• Copy Paste: [[2402.14116]] FanOutQA: Multi-Hop, Multi-Document Question Answering for Large Language Models(https://arxiv.org/abs/2402.14116)
• Keywords: large language model
• Abstract: One type of question that is commonly found in day-to-day scenarios is ``fan-out'' questions, complex multi-hop, multi-document reasoning questions that require finding information about a large number of entities. However, there exist few resources to evaluate this type of question-answering capability among large language models. To evaluate complex reasoning in LLMs more fully, we present FanOutQA, a high-quality dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base. We formulate three benchmark settings across our dataset and benchmark 7 LLMs, including GPT-4, LLaMA 2, Claude-2.1, and Mixtral-8x7B, finding that contemporary models still have room to improve reasoning over inter-document dependencies in a long context. We provide our dataset and open-source tools to run models to encourage evaluation at https://fanoutqa.com

Title: DeiSAM: Segment Anything with Deictic Prompting

• Authors: Hikaru Shindo, Manuel Brack, Gopika Sudhakaran, Devendra Singh Dhami, Patrick Schramowski, Kristian Kersting
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14123
• Pdf URL: https://arxiv.org/pdf/2402.14123
• Copy Paste: [[2402.14123]] DeiSAM: Segment Anything with Deictic Prompting(https://arxiv.org/abs/2402.14123)
• Keywords: large language model, segmentation
• Abstract: Large-scale, pre-trained neural networks have demonstrated strong capabilities in various tasks, including zero-shot image segmentation. To identify concrete objects in complex scenes, humans instinctively rely on deictic descriptions in natural language, i.e., referring to something depending on the context such as "The object that is on the desk and behind the cup.". However, deep learning approaches cannot reliably interpret such deictic representations due to their lack of reasoning capabilities in complex scenarios. To remedy this issue, we propose DeiSAM -- a combination of large pre-trained neural networks with differentiable logic reasoners -- for deictic promptable segmentation. Given a complex, textual segmentation description, DeiSAM leverages Large Language Models (LLMs) to generate first-order logic rules and performs differentiable forward reasoning on generated scene graphs. Subsequently, DeiSAM segments objects by matching them to the logically inferred image regions. As part of our evaluation, we propose the Deictic Visual Genome (DeiVG) dataset, containing paired visual input and complex, deictic textual prompts. Our empirical results demonstrate that DeiSAM is a substantial improvement over purely data-driven baselines for deictic promptable segmentation.

Title: Fake Resume Attacks: Data Poisoning on Online Job Platforms

• Authors: Michiharu Yamashita, Thanh Tran, Dongwon Lee
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.14124
• Pdf URL: https://arxiv.org/pdf/2402.14124
• Copy Paste: [[2402.14124]] Fake Resume Attacks: Data Poisoning on Online Job Platforms(https://arxiv.org/abs/2402.14124)
• Keywords: attack
• Abstract: While recent studies have exposed various vulnerabilities incurred from data poisoning attacks in many web services, little is known about the vulnerability on online professional job platforms (e.g., LinkedIn and Indeed). In this work, first time, we demonstrate the critical vulnerabilities found in the common Human Resources (HR) task of matching job seekers and companies on online job platforms. Capitalizing on the unrestricted format and contents of job seekers' resumes and easy creation of accounts on job platforms, we demonstrate three attack scenarios: (1) company promotion attack to increase the likelihood of target companies being recommended, (2) company demotion attack to decrease the likelihood of target companies being recommended, and (3) user promotion attack to increase the likelihood of certain users being matched to certain companies. To this end, we develop an end-to-end "fake resume" generation framework, titled FRANCIS, that induces systematic prediction errors via data poisoning. Our empirical evaluation on real-world datasets reveals that data poisoning attacks can markedly skew the results of matchmaking between job seekers and companies, regardless of underlying models, with vulnerability amplified in proportion to poisoning intensity. These findings suggest that the outputs of various services from job platforms can be potentially hacked by malicious users.

Title: QuantTM: Business-Centric Threat Quantification for Risk Management and Cyber Resilience

• Authors: Jan von der Assen, Muriel F. Franco, Muyao Dong, Burkhard Stiller
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.14140
• Pdf URL: https://arxiv.org/pdf/2402.14140
• Copy Paste: [[2402.14140]] QuantTM: Business-Centric Threat Quantification for Risk Management and Cyber Resilience(https://arxiv.org/abs/2402.14140)
• Keywords: security
• Abstract: Threat modeling has emerged as a key process for understanding relevant threats within businesses. However, understanding the importance of threat events is rarely driven by the business incorporating the system. Furthermore, prioritization of threat events often occurs based on abstract and qualitative scoring. While such scores enable prioritization, they do not allow the results to be easily interpreted by decision-makers. This can hinder downstream activities, such as discussing security investments and a security control's economic applicability. This article introduces QuantTM, an approach that incorporates views from operational and strategic business representatives to collect threat information during the threat modeling process to measure potential financial loss incurred by a specific threat event. It empowers the analysis of threats' impacts and the applicability of security controls, thus supporting the threat analysis and prioritization from an economic perspective. QuantTM comprises an overarching process for data collection and aggregation and a method for business impact analysis. The performance and feasibility of the QuantTM approach are demonstrated in a real-world case study conducted in a Swiss SME to analyze the impacts of threats and economic benefits of security controls. Secondly, it is shown that employing business impact analysis is feasible and that the supporting prototype exhibits great usability.

Title: SecurePose: Automated Face Blurring and Human Movement Kinematics Extraction from Videos Recorded in Clinical Settings

• Authors: Rishabh Bajpai, Bhooma Aravamuthan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14143
• Pdf URL: https://arxiv.org/pdf/2402.14143
• Copy Paste: [[2402.14143]] SecurePose: Automated Face Blurring and Human Movement Kinematics Extraction from Videos Recorded in Clinical Settings(https://arxiv.org/abs/2402.14143)
• Keywords: secure, privacy, robust, extraction
• Abstract: Movement disorders are typically diagnosed by consensus-based expert evaluation of clinically acquired patient videos. However, such broad sharing of patient videos poses risks to patient privacy. Face blurring can be used to de-identify videos, but this process is often manual and time-consuming. Available automated face blurring techniques are subject to either excessive, inconsistent, or insufficient facial blurring - all of which can be disastrous for video assessment and patient privacy. Furthermore, assessing movement disorders in these videos is often subjective. The extraction of quantifiable kinematic features can help inform movement disorder assessment in these videos, but existing methods to do this are prone to errors if using pre-blurred videos. We have developed an open-source software called SecurePose that can both achieve reliable face blurring and automated kinematic extraction in patient videos recorded in a clinic setting using an iPad. SecurePose, extracts kinematics using a pose estimation method (OpenPose), tracks and uniquely identifies all individuals in the video, identifies the patient, and performs face blurring. The software was validated on gait videos recorded in outpatient clinic visits of 116 children with cerebral palsy. The validation involved assessing intermediate steps of kinematics extraction and face blurring with manual blurring (ground truth). Moreover, when SecurePose was compared with six selected existing methods, it outperformed other methods in automated face detection and achieved ceiling accuracy in 91.08% less time than a robust manual face blurring method. Furthermore, ten experienced researchers found SecurePose easy to learn and use, as evidenced by the System Usability Scale. The results of this work validated the performance and usability of SecurePose on clinically recorded gait videos for face blurring and kinematics extraction.

Title: Reinforcement Learning with Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation

• Authors: Karin de Langis, Ryan Koo, Dongyeop Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14146
• Pdf URL: https://arxiv.org/pdf/2402.14146
• Copy Paste: [[2402.14146]] Reinforcement Learning with Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation(https://arxiv.org/abs/2402.14146)
• Keywords: large language model
• Abstract: Style is an integral component of text that expresses a diverse set of information, including interpersonal dynamics (e.g. formality) and the author's emotions or attitudes (e.g. disgust). Humans often employ multiple styles simultaneously. An open question is how large language models can be explicitly controlled so that they weave together target styles when generating text: for example, to produce text that is both negative and non-toxic. Previous work investigates the controlled generation of a single style, or else controlled generation of a style and other attributes. In this paper, we expand this into controlling multiple styles simultaneously. Specifically, we investigate various formulations of multiple style rewards for a reinforcement learning (RL) approach to controlled multi-style generation. These reward formulations include calibrated outputs from discriminators and dynamic weighting by discriminator gradient magnitudes. We find that dynamic weighting generally outperforms static weighting approaches, and we explore its effectiveness in 2- and 3-style control, even compared to strong baselines like plug-and-play model. All code and data for RL pipelines with multiple style attributes will be publicly available.

Title: MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms

• Authors: Yiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang, Srijan Kumar
• Subjects: cs.CL, cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.14154
• Pdf URL: https://arxiv.org/pdf/2402.14154
• Copy Paste: [[2402.14154]] MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms(https://arxiv.org/abs/2402.14154)
• Keywords: large language model
• Abstract: Social media platforms are hubs for multimodal information exchange, encompassing text, images, and videos, making it challenging for machines to comprehend the information or emotions associated with interactions in online spaces. Multimodal Large Language Models (MLLMs) have emerged as a promising solution to address these challenges, yet struggle with accurately interpreting human emotions and complex contents like misinformation. This paper introduces MM-Soc, a comprehensive benchmark designed to evaluate MLLMs' understanding of multimodal social media content. MM-Soc compiles prominent multimodal datasets and incorporates a novel large-scale YouTube tagging dataset, targeting a range of tasks from misinformation detection, hate speech detection, and social context generation. Through our exhaustive evaluation on ten size-variants of four open-source MLLMs, we have identified significant performance disparities, highlighting the need for advancements in models' social understanding capabilities. Our analysis reveals that, in a zero-shot setting, various types of MLLMs generally exhibit difficulties in handling social media tasks. However, MLLMs demonstrate performance improvements post fine-tuning, suggesting potential pathways for improvement.

Title: Can Similarity-Based Domain-Ordering Reduce Catastrophic Forgetting for Intent Recognition?

• Authors: Amogh Mannekote, Xiaoyi Tian, Kristy Elizabeth Boyer, Bonnie J. Dorr
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14155
• Pdf URL: https://arxiv.org/pdf/2402.14155
• Copy Paste: [[2402.14155]] Can Similarity-Based Domain-Ordering Reduce Catastrophic Forgetting for Intent Recognition?(https://arxiv.org/abs/2402.14155)
• Keywords: generative
• Abstract: Task-oriented dialogue systems are expected to handle a constantly expanding set of intents and domains even after they have been deployed to support more and more functionalities. To live up to this expectation, it becomes critical to mitigate the catastrophic forgetting problem (CF) that occurs in continual learning (CL) settings for a task such as intent recognition. While existing dialogue systems research has explored replay-based and regularization-based methods to this end, the effect of domain ordering on the CL performance of intent recognition models remains unexplored. If understood well, domain ordering has the potential to be an orthogonal technique that can be leveraged alongside existing techniques such as experience replay. Our work fills this gap by comparing the impact of three domain-ordering strategies (min-sum path, max-sum path, random) on the CL performance of a generative intent recognition model. Our findings reveal that the min-sum path strategy outperforms the others in reducing catastrophic forgetting when training on the 220M T5-Base model. However, this advantage diminishes with the larger 770M T5-Large model. These results underscores the potential of domain ordering as a complementary strategy for mitigating catastrophic forgetting in continually learning intent recognition models, particularly in resource-constrained scenarios.

Title: TOOLVERIFIER: Generalization to New Tools via Self-Verification

• Authors: Dheeraj Mekala, Jason Weston, Jack Lanchantin, Roberta Raileanu, Maria Lomeli, Jingbo Shang, Jane Dwivedi-Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14158
• Pdf URL: https://arxiv.org/pdf/2402.14158
• Copy Paste: [[2402.14158]] TOOLVERIFIER: Generalization to New Tools via Self-Verification(https://arxiv.org/abs/2402.14158)
• Keywords: robust
• Abstract: Teaching language models to use tools is an important milestone towards building general assistants, but remains an open problem. While there has been significant progress on learning to use specific tools via fine-tuning, language models still struggle with learning how to robustly use new tools from only a few demonstrations. In this work we introduce a self-verification method which distinguishes between close candidates by self-asking contrastive questions during (1) tool selection; and (2) parameter generation. We construct synthetic, high-quality, self-generated data for this goal using Llama-2 70B, which we intend to release publicly. Extensive experiments on 4 tasks from the ToolBench benchmark, consisting of 17 unseen tools, demonstrate an average improvement of 22% over few-shot baselines, even in scenarios where the distinctions between candidate tools are finely nuanced.

Title: Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement

• Authors: Wonseok Jeon, Mukul Gagrani, Raghavv Goel, Junyoung Park, Mingu Lee, Christopher Lott
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14160
• Pdf URL: https://arxiv.org/pdf/2402.14160
• Copy Paste: [[2402.14160]] Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement(https://arxiv.org/abs/2402.14160)
• Keywords: large language model
• Abstract: Speculative decoding is an inference-acceleration method for large language models (LLMs) where a small language model generates a draft-token sequence which is further verified by the target LLM in parallel. Recent works have advanced this method by establishing a draft-token tree, achieving superior performance over a single-sequence speculative decoding. However, those works independently generate tokens at each level of the tree, not leveraging the tree's entire diversifiability. Besides, their empirical superiority has been shown for fixed length of sequences, implicitly granting more computational resource to LLM for the tree-based methods. None of the existing works has conducted empirical studies with fixed target computational budgets despite its importance to resource-bounded devices. We present Recursive Speculative Decoding (RSD), a novel tree-based method that samples draft tokens without replacement and maximizes the diversity of the tree. During RSD's drafting, the tree is built by either Gumbel-Top-$k$ trick that draws tokens without replacement in parallel or Stochastic Beam Search that samples sequences without replacement while early-truncating unlikely draft sequences and reducing the computational cost of LLM. We empirically evaluate RSD with Llama 2 and OPT models, showing that RSD outperforms the baseline methods, consistently for fixed draft sequence length and in most cases for fixed computational budgets at LLM.

Title: On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study

• Authors: Minh-Hao Van, Prateek Verma, Xintao Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14162
• Pdf URL: https://arxiv.org/pdf/2402.14162
• Copy Paste: [[2402.14162]] On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study(https://arxiv.org/abs/2402.14162)
• Keywords: robust, large language model
• Abstract: Recently, large language models (LLMs) have taken the spotlight in natural language processing. Further, integrating LLMs with vision enables the users to explore emergent abilities with multimodal data. Visual language models (VLMs), such as LLaVA, Flamingo, or CLIP, have demonstrated impressive performance on various visio-linguistic tasks. Consequently, there are enormous applications of large models that could be potentially used in the biomedical imaging field. Along that direction, there is a lack of related work to show the ability of large models to diagnose the diseases. In this work, we study the zero-shot and few-shot robustness of VLMs on the medical imaging analysis tasks. Our comprehensive experiments demonstrate the effectiveness of VLMs in analyzing biomedical images such as brain MRIs, microscopic images of blood cells, and chest X-rays.

Title: T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching

• Authors: Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, Anima Anandkumar
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14167
• Pdf URL: https://arxiv.org/pdf/2402.14167
• Copy Paste: [[2402.14167]] T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching(https://arxiv.org/abs/2402.14167)
• Keywords: diffusion
• Abstract: Sampling from diffusion probabilistic models (DPMs) is often expensive for high-quality image generation and typically requires many steps with a large model. In this paper, we introduce sampling Trajectory Stitching T-Stitch, a simple yet efficient technique to improve the sampling efficiency with little or no generation degradation. Instead of solely using a large DPM for the entire sampling trajectory, T-Stitch first leverages a smaller DPM in the initial steps as a cheap drop-in replacement of the larger DPM and switches to the larger DPM at a later stage. Our key insight is that different diffusion models learn similar encodings under the same training data distribution and smaller models are capable of generating good global structures in the early steps. Extensive experiments demonstrate that T-Stitch is training-free, generally applicable for different architectures, and complements most existing fast sampling techniques with flexible speed and quality trade-offs. On DiT-XL, for example, 40% of the early timesteps can be safely replaced with a 10x faster DiT-S without performance drop on class-conditional ImageNet generation. We further show that our method can also be used as a drop-in technique to not only accelerate the popular pretrained stable diffusion (SD) models but also improve the prompt alignment of stylized SD models from the public model zoo. Code is released at https://github.com/NVlabs/T-Stitch

Title: Bangla AI: A Framework for Machine Translation Utilizing Large Language Models for Ethnic Media

• Authors: MD Ashraful Goni, Fahad Mostafa, Kerk F. Kee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14179
• Pdf URL: https://arxiv.org/pdf/2402.14179
• Copy Paste: [[2402.14179]] Bangla AI: A Framework for Machine Translation Utilizing Large Language Models for Ethnic Media(https://arxiv.org/abs/2402.14179)
• Keywords: large language model
• Abstract: Ethnic media, which caters to diaspora communities in host nations, serves as a vital platform for these communities to both produce content and access information. Rather than utilizing the language of the host nation, ethnic media delivers news in the language of the immigrant community. For instance, in the USA, Bangla ethnic media presents news in Bangla rather than English. This research delves into the prospective integration of large language models (LLM) and multi-lingual machine translations (MMT) within the ethnic media industry. It centers on the transformative potential of using LLM in MMT in various facets of news translation, searching, and categorization. The paper outlines a theoretical framework elucidating the integration of LLM and MMT into the news searching and translation processes for ethnic media. Additionally, it briefly addresses the potential ethical challenges associated with the incorporation of LLM and MMT in news translation procedures.

Title: Linear Transformers are Versatile In-Context Learners

• Authors: Max Vladymyrov, Johannes von Oswald, Mark Sandler, Rong Ge
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14180
• Pdf URL: https://arxiv.org/pdf/2402.14180
• Copy Paste: [[2402.14180]] Linear Transformers are Versatile In-Context Learners(https://arxiv.org/abs/2402.14180)
• Keywords: transformer
• Abstract: Recent research has demonstrated that transformers, particularly linear attention models, implicitly execute gradient-descent-like algorithms on data provided in-context during their forward inference step. However, their capability in handling more complex problems remains unexplored. In this paper, we prove that any linear transformer maintains an implicit linear model and can be interpreted as performing a variant of preconditioned gradient descent. We also investigate the use of linear transformers in a challenging scenario where the training data is corrupted with different levels of noise. Remarkably, we demonstrate that for this problem linear transformers discover an intricate and highly effective optimization algorithm, surpassing or matching in performance many reasonable baselines. We reverse-engineer this algorithm and show that it is a novel approach incorporating momentum and adaptive rescaling based on noise levels. Our findings show that even linear transformers possess the surprising ability to discover sophisticated optimization strategies.

Title: HINT: High-quality INPainting Transformer with Mask-Aware Encoding and Enhanced Attention

• Authors: Shuang Chen, Amir Atapour-Abarghouei, Hubert P. H. Shum
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14185
• Pdf URL: https://arxiv.org/pdf/2402.14185
• Copy Paste: [[2402.14185]] HINT: High-quality INPainting Transformer with Mask-Aware Encoding and Enhanced Attention(https://arxiv.org/abs/2402.14185)
• Keywords: transformer
• Abstract: Existing image inpainting methods leverage convolution-based downsampling approaches to reduce spatial dimensions. This may result in information loss from corrupted images where the available information is inherently sparse, especially for the scenario of large missing regions. Recent advances in self-attention mechanisms within transformers have led to significant improvements in many computer vision tasks including inpainting. However, limited by the computational costs, existing methods cannot fully exploit the efficacy of long-range modelling capabilities of such models. In this paper, we propose an end-to-end High-quality INpainting Transformer, abbreviated as HINT, which consists of a novel mask-aware pixel-shuffle downsampling module (MPD) to preserve the visible information extracted from the corrupted image while maintaining the integrity of the information available for high-level inferences made within the model. Moreover, we propose a Spatially-activated Channel Attention Layer (SCAL), an efficient self-attention mechanism interpreting spatial awareness to model the corrupted image at multiple scales. To further enhance the effectiveness of SCAL, motivated by recent advanced in speech recognition, we introduce a sandwich structure that places feed-forward networks before and after the SCAL module. We demonstrate the superior performance of HINT compared to contemporary state-of-the-art models on four datasets, CelebA, CelebA-HQ, Places2, and Dunhuang.

Title: Homomorphic Encryption Based on Post-Quantum Cryptography

• Authors: Abel C. H. Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.14193
• Pdf URL: https://arxiv.org/pdf/2402.14193
• Copy Paste: [[2402.14193]] Homomorphic Encryption Based on Post-Quantum Cryptography(https://arxiv.org/abs/2402.14193)
• Keywords: attack
• Abstract: With the development of Shor's algorithm, some nondeterministic polynomial (NP) time problems (e.g. prime factorization problems and discrete logarithm problems) may be solved in polynomial time. In recent years, although some homomorphic encryption algorithms have been proposed based on prime factorization problems, the algorithms may be cracked by quantum computing attacks. Therefore, this study proposes a post-quantum cryptography (PQC)-based homomorphic encryption method which includes the homomorphic encryption function based on a code-based cryptography method for avoiding quantum computing attacks. Subsection 3.2 proposes mathematical models to prove the feasibility of the proposed method, and Subsection 3.3 gives calculation examples to present the detailed steps of the proposed method. In experimental environments, the mainstream cryptography methods (i.e. RSA cryptography and elliptic curve cryptography (ECC)) have been compared, and the results show that the encryption time and decryption time of the proposed method are shorter than other cryptography methods. Furthermore, the proposed method is designed based on a non-negative matrix factorization problem (i.e. a NP problem) for resisting quantum computing attacks.

Title: BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplay

• Authors: Catherine Weaver, Chen Tang, Ce Hao, Kenta Kawamoto, Masayoshi Tomizuka, Wei Zhan
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.14194
• Pdf URL: https://arxiv.org/pdf/2402.14194
• Copy Paste: [[2402.14194]] BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplay(https://arxiv.org/abs/2402.14194)
• Keywords: transformer
• Abstract: Imitation learning learns a policy from demonstrations without requiring hand-designed reward functions. In many robotic tasks, such as autonomous racing, imitated policies must model complex environment dynamics and human decision-making. Sequence modeling is highly effective in capturing intricate patterns of motion sequences but struggles to adapt to new environments or distribution shifts that are common in real-world robotics tasks. In contrast, Adversarial Imitation Learning (AIL) can mitigate this effect, but struggles with sample inefficiency and handling complex motion patterns. Thus, we propose BeTAIL: Behavior Transformer Adversarial Imitation Learning, which combines a Behavior Transformer (BeT) policy from human demonstrations with online AIL. BeTAIL adds an AIL residual policy to the BeT policy to model the sequential decision-making process of human experts and correct for out-of-distribution states or shifts in environment dynamics. We test BeTAIL on three challenges with expert-level demonstrations of real human gameplay in Gran Turismo Sport. Our proposed residual BeTAIL reduces environment interactions and improves racing performance and stability, even when the BeT is pretrained on different tracks than downstream learning. Videos and code available at: https://sites.google.com/berkeley.edu/BeTAIL/home.

Title: Learning to Reduce: Optimal Representations of Structured Data in Prompting Large Language Models

• Authors: Younghun Lee, Sungchul Kim, Tong Yu, Ryan A. Rossi, Xiang Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14195
• Pdf URL: https://arxiv.org/pdf/2402.14195
• Copy Paste: [[2402.14195]] Learning to Reduce: Optimal Representations of Structured Data in Prompting Large Language Models(https://arxiv.org/abs/2402.14195)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been widely used as general-purpose AI agents showing comparable performance on many downstream tasks. However, existing work shows that it is challenging for LLMs to integrate structured data (e.g. KG, tables, DBs) into their prompts; LLMs need to either understand long text data or select the most relevant evidence prior to inference, and both approaches are not trivial. In this paper, we propose a framework, Learning to Reduce, that fine-tunes a language model to generate a reduced version of an input context, given a task description and context input. The model learns to reduce the input context using On-Policy Reinforcement Learning and aims to improve the reasoning performance of a fixed LLM. Experimental results illustrate that our model not only achieves comparable accuracies in selecting the relevant evidence from an input context, but also shows generalizability on different datasets. We further show that our model helps improve the LLM's performance on downstream tasks especially when the context is long.

Title: Towards Understanding Counseling Conversations: Domain Knowledge and Large Language Models

• Authors: Younghun Lee, Dan Goldwasser, Laura Schwab Reese
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14200
• Pdf URL: https://arxiv.org/pdf/2402.14200
• Copy Paste: [[2402.14200]] Towards Understanding Counseling Conversations: Domain Knowledge and Large Language Models(https://arxiv.org/abs/2402.14200)
• Keywords: transformer, large language model
• Abstract: Understanding the dynamics of counseling conversations is an important task, yet it is a challenging NLP problem regardless of the recent advance of Transformer-based pre-trained language models. This paper proposes a systematic approach to examine the efficacy of domain knowledge and large language models (LLMs) in better representing conversations between a crisis counselor and a help seeker. We empirically show that state-of-the-art language models such as Transformer-based models and GPT models fail to predict the conversation outcome. To provide richer context to conversations, we incorporate human-annotated domain knowledge and LLM-generated features; simple integration of domain knowledge and LLM features improves the model performance by approximately 15%. We argue that both domain knowledge and LLM-generated features can be exploited to better characterize counseling conversations when they are used as an additional context to conversations.

Title: Comparing Graph Transformers via Positional Encodings

• Authors: Mitchell Black, Zhengchao Wan, Gal Mishne, Amir Nayyeri, Yusu Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14202
• Pdf URL: https://arxiv.org/pdf/2402.14202
• Copy Paste: [[2402.14202]] Comparing Graph Transformers via Positional Encodings(https://arxiv.org/abs/2402.14202)
• Keywords: transformer
• Abstract: The distinguishing power of graph transformers is closely tied to the choice of positional encoding: features used to augment the base transformer with information about the graph. There are two primary types of positional encoding: absolute positional encodings (APEs) and relative positional encodings (RPEs). APEs assign features to each node and are given as input to the transformer. RPEs instead assign a feature to each pair of nodes, e.g., graph distance, and are used to augment the attention block. A priori, it is unclear which method is better for maximizing the power of the resulting graph transformer. In this paper, we aim to understand the relationship between these different types of positional encodings. Interestingly, we show that graph transformers using APEs and RPEs are equivalent in terms of distinguishing power. In particular, we demonstrate how to interchange APEs and RPEs while maintaining their distinguishing power in terms of graph transformers. Based on our theoretical results, we provide a study on several APEs and RPEs (including the resistance distance and the recently introduced stable and expressive positional encoding (SPE)) and compare their distinguishing power in terms of transformers. We believe our work will help navigate the huge number of choices of positional encoding and will provide guidance on the future design of positional encodings for graph transformers.

Title: Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models

• Authors: Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, Monica S. Lam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14207
• Pdf URL: https://arxiv.org/pdf/2402.14207
• Copy Paste: [[2402.14207]] Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models(https://arxiv.org/abs/2402.14207)
• Keywords: large language model
• Abstract: We study how to apply large language models to write grounded and organized long-form articles from scratch, with comparable breadth and depth to Wikipedia pages. This underexplored problem poses new challenges at the pre-writing stage, including how to research the topic and prepare an outline prior to writing. We propose STORM, a writing system for the Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking. STORM models the pre-writing stage by (1) discovering diverse perspectives in researching the given topic, (2) simulating conversations where writers carrying different perspectives pose questions to a topic expert grounded on trusted Internet sources, (3) curating the collected information to create an outline. For evaluation, we curate FreshWiki, a dataset of recent high-quality Wikipedia articles, and formulate outline assessments to evaluate the pre-writing stage. We further gather feedback from experienced Wikipedia editors. Compared to articles generated by an outline-driven retrieval-augmented baseline, more of STORM's articles are deemed to be organized (by a 25% absolute increase) and broad in coverage (by 10%). The expert feedback also helps identify new challenges for generating grounded long articles, such as source bias transfer and over-association of unrelated facts.

Title: Content Conditional Debiasing for Fair Text Embedding

• Authors: Wenlong Deng, Blair Chen, Xiaoxiao Li, Christos Thrampoulidis
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14208
• Pdf URL: https://arxiv.org/pdf/2402.14208
• Copy Paste: [[2402.14208]] Content Conditional Debiasing for Fair Text Embedding(https://arxiv.org/abs/2402.14208)
• Keywords: fair, large language model
• Abstract: Mitigating biases in machine learning models has gained increasing attention in Natural Language Processing (NLP). Yet, only a few studies focus on fair text embeddings, which are crucial yet challenging for real-world applications. In this paper, we propose a novel method for learning fair text embeddings. We achieve fairness while maintaining utility trade-off by ensuring conditional independence between sensitive attributes and text embeddings conditioned on the content. Specifically, we enforce that embeddings of texts with different sensitive attributes but identical content maintain the same distance toward the embedding of their corresponding neutral text. Furthermore, we address the issue of lacking proper training data by using Large Language Models (LLMs) to augment texts into different sensitive groups. Our extensive evaluations demonstrate that our approach effectively improves fairness while preserving the utility of embeddings, representing a pioneering effort in achieving conditional independence for fair text embeddings.

Title: Moonwalk: Inverse-Forward Differentiation

• Authors: Dmitrii Krylov, Armin Karamzade, Roy Fox
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14212
• Pdf URL: https://arxiv.org/pdf/2402.14212
• Copy Paste: [[2402.14212]] Moonwalk: Inverse-Forward Differentiation(https://arxiv.org/abs/2402.14212)
• Keywords: robust
• Abstract: Backpropagation, while effective for gradient computation, falls short in addressing memory consumption, limiting scalability. This work explores forward-mode gradient computation as an alternative in invertible networks, showing its potential to reduce the memory footprint without substantial drawbacks. We introduce a novel technique based on a vector-inverse-Jacobian product that accelerates the computation of forward gradients while retaining the advantages of memory reduction and preserving the fidelity of true gradients. Our method, Moonwalk, has a time complexity linear in the depth of the network, unlike the quadratic time complexity of na\"ive forward, and empirically reduces computation time by several orders of magnitude without allocating more memory. We further accelerate Moonwalk by combining it with reverse-mode differentiation to achieve time complexity comparable with backpropagation while maintaining a much smaller memory footprint. Finally, we showcase the robustness of our method across several architecture choices. Moonwalk is the first forward-based method to compute true gradients in invertible networks in computation time comparable to backpropagation and using significantly less memory.

Title: Quaternion recurrent neural network with real-time recurrent learning and maximum correntropy criterion

• Authors: Pauline Bourigault, Dongpo Xu, Danilo P. Mandic
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14227
• Pdf URL: https://arxiv.org/pdf/2402.14227
• Copy Paste: [[2402.14227]] Quaternion recurrent neural network with real-time recurrent learning and maximum correntropy criterion(https://arxiv.org/abs/2402.14227)
• Keywords: robust
• Abstract: We develop a robust quaternion recurrent neural network (QRNN) for real-time processing of 3D and 4D data with outliers. This is achieved by combining the real-time recurrent learning (RTRL) algorithm and the maximum correntropy criterion (MCC) as a loss function. While both the mean square error and maximum correntropy criterion are viable cost functions, it is shown that the non-quadratic maximum correntropy loss function is less sensitive to outliers, making it suitable for applications with multidimensional noisy or uncertain data. Both algorithms are derived based on the novel generalised HR (GHR) calculus, which allows for the differentiation of real functions of quaternion variables and offers the product and chain rules, thus enabling elegant and compact derivations. Simulation results in the context of motion prediction of chest internal markers for lung cancer radiotherapy, which includes regular and irregular breathing sequences, support the analysis.

Title: COPR: Continual Human Preference Learning via Optimal Policy Regularization

• Authors: Han Zhang, Lin Gui, Yu Lei, Yuanzhao Zhai, Yehong Zhang, Yulan He, Hui Wang, Yue Yu, Kam-Fai Wong, Bin Liang, Ruifeng Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14228
• Pdf URL: https://arxiv.org/pdf/2402.14228
• Copy Paste: [[2402.14228]] COPR: Continual Human Preference Learning via Optimal Policy Regularization(https://arxiv.org/abs/2402.14228)
• Keywords: robust, large language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is commonly utilized to improve the alignment of Large Language Models (LLMs) with human preferences. Given the evolving nature of human preferences, continual alignment becomes more crucial and practical in comparison to traditional static alignment. Nevertheless, making RLHF compatible with Continual Learning (CL) is challenging due to its complex process. Meanwhile, directly learning new human preferences may lead to Catastrophic Forgetting (CF) of historical preferences, resulting in helpless or harmful outputs. To overcome these challenges, we propose the Continual Optimal Policy Regularization (COPR) method, which draws inspiration from the optimal policy theory. COPR utilizes a sampling distribution as a demonstration and regularization constraints for CL. It adopts the Lagrangian Duality (LD) method to dynamically regularize the current policy based on the historically optimal policy, which prevents CF and avoids over-emphasizing unbalanced objectives. We also provide formal proof for the learnability of COPR. The experimental results show that COPR outperforms strong CL baselines on our proposed benchmark, in terms of reward-based, GPT-4 evaluations and human assessment. Furthermore, we validate the robustness of COPR under various CL settings, including different backbones, replay memory sizes, and learning orders.

Title: A Self-supervised Pressure Map human keypoint Detection Approch: Optimizing Generalization and Computational Efficiency Across Datasets

• Authors: Chengzhang Yu, Xianjun Yang, Wenxia Bao, Shaonan Wang, Zhiming Yao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14241
• Pdf URL: https://arxiv.org/pdf/2402.14241
• Copy Paste: [[2402.14241]] A Self-supervised Pressure Map human keypoint Detection Approch: Optimizing Generalization and Computational Efficiency Across Datasets(https://arxiv.org/abs/2402.14241)
• Keywords: robust, extraction
• Abstract: In environments where RGB images are inadequate, pressure maps is a viable alternative, garnering scholarly attention. This study introduces a novel self-supervised pressure map keypoint detection (SPMKD) method, addressing the current gap in specialized designs for human keypoint extraction from pressure maps. Central to our contribution is the Encoder-Fuser-Decoder (EFD) model, which is a robust framework that integrates a lightweight encoder for precise human keypoint detection, a fuser for efficient gradient propagation, and a decoder that transforms human keypoints into reconstructed pressure maps. This structure is further enhanced by the Classification-to-Regression Weight Transfer (CRWT) method, which fine-tunes accuracy through initial classification task training. This innovation not only enhances human keypoint generalization without manual annotations but also showcases remarkable efficiency and generalization, evidenced by a reduction to only $5.96\%$ in FLOPs and $1.11\%$ in parameter count compared to the baseline methods.

Title: Reconstruction-Based Anomaly Localization via Knowledge-Informed Self-Training

• Authors: Cheng Qian, Xiaoxian Lao, Chunguang Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14246
• Pdf URL: https://arxiv.org/pdf/2402.14246
• Copy Paste: [[2402.14246]] Reconstruction-Based Anomaly Localization via Knowledge-Informed Self-Training(https://arxiv.org/abs/2402.14246)
• Keywords: interpretability
• Abstract: Anomaly localization, which involves localizing anomalous regions within images, is a significant industrial task. Reconstruction-based methods are widely adopted for anomaly localization because of their low complexity and high interpretability. Most existing reconstruction-based methods only use normal samples to construct model. If anomalous samples are appropriately utilized in the process of anomaly localization, the localization performance can be improved. However, usually only weakly labeled anomalous samples are available, which limits the improvement. In many cases, we can obtain some knowledge of anomalies summarized by domain experts. Taking advantage of such knowledge can help us better utilize the anomalous samples and thus further improve the localization performance. In this paper, we propose a novel reconstruction-based method named knowledge-informed self-training (KIST) which integrates knowledge into reconstruction model through self-training. Specifically, KIST utilizes weakly labeled anomalous samples in addition to the normal ones and exploits knowledge to yield pixel-level pseudo-labels of the anomalous samples. Based on the pseudo labels, a novel loss which promotes the reconstruction of normal pixels while suppressing the reconstruction of anomalous pixels is used. We conduct experiments on different datasets and demonstrate the advantages of KIST over the existing reconstruction-based methods.

Title: MVD$^2$: Efficient Multiview 3D Reconstruction for Multiview Diffusion

• Authors: Xin-Yang Zheng, Hao Pan, Yu-Xiao Guo, Xin Tong, Yang Liu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.14253
• Pdf URL: https://arxiv.org/pdf/2402.14253
• Copy Paste: [[2402.14253]] MVD$^2$: Efficient Multiview 3D Reconstruction for Multiview Diffusion(https://arxiv.org/abs/2402.14253)
• Keywords: robust, diffusion
• Abstract: As a promising 3D generation technique, multiview diffusion (MVD) has received a lot of attention due to its advantages in terms of generalizability, quality, and efficiency. By finetuning pretrained large image diffusion models with 3D data, the MVD methods first generate multiple views of a 3D object based on an image or text prompt and then reconstruct 3D shapes with multiview 3D reconstruction. However, the sparse views and inconsistent details in the generated images make 3D reconstruction challenging. We present MVD$^2$, an efficient 3D reconstruction method for multiview diffusion (MVD) images. MVD$^2$ aggregates image features into a 3D feature volume by projection and convolution and then decodes volumetric features into a 3D mesh. We train MVD$^2$ with 3D shape collections and MVD images prompted by rendered views of 3D shapes. To address the discrepancy between the generated multiview images and ground-truth views of the 3D shapes, we design a simple-yet-efficient view-dependent training scheme. MVD$^2$ improves the 3D generation quality of MVD and is fast and robust to various MVD methods. After training, it can efficiently decode 3D meshes from multiview images within one second. We train MVD$^2$ with Zero-123++ and ObjectVerse-LVIS 3D dataset and demonstrate its superior performance in generating 3D models from multiview images generated by different MVD methods, using both synthetic and real images as prompts.

Title: Eagle: Ethical Dataset Given from Real Interactions

• Authors: Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14258
• Pdf URL: https://arxiv.org/pdf/2402.14258
• Copy Paste: [[2402.14258]] Eagle: Ethical Dataset Given from Real Interactions(https://arxiv.org/abs/2402.14258)
• Keywords: large language model
• Abstract: Recent studies have demonstrated that large language models (LLMs) have ethical-related problems such as social biases, lack of moral reasoning, and generation of offensive content. The existing evaluation metrics and methods to address these ethical challenges use datasets intentionally created by instructing humans to create instances including ethical problems. Therefore, the data does not reflect prompts that users actually provide when utilizing LLM services in everyday contexts. This may not lead to the development of safe LLMs that can address ethical challenges arising in real-world applications. In this paper, we create Eagle datasets extracted from real interactions between ChatGPT and users that exhibit social biases, toxicity, and immoral problems. Our experiments show that Eagle captures complementary aspects, not covered by existing datasets proposed for evaluation and mitigation of such ethical challenges. Our code is publicly available at https://huggingface.co/datasets/MasahiroKaneko/eagle.

Title: Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyond

• Authors: Zhiyuan Wang, Jinhao Duan, Chenxi Yuan, Qingyu Chen, Tianlong Chen, Huaxiu Yao, Yue Zhang, Ren Wang, Kaidi Xu, Xiaoshuang Shi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14259
• Pdf URL: https://arxiv.org/pdf/2402.14259
• Copy Paste: [[2402.14259]] Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyond(https://arxiv.org/abs/2402.14259)
• Keywords: generative, large language model
• Abstract: Uncertainty estimation plays a pivotal role in ensuring the reliability of safety-critical human-AI interaction systems, particularly in the medical domain. However, a general method for quantifying the uncertainty of free-form answers has yet to be established in open-ended medical question-answering (QA) tasks, where irrelevant words and sequences with limited semantic information can be the primary source of uncertainty due to the presence of generative inequality. In this paper, we propose the Word-Sequence Entropy (WSE), which calibrates the uncertainty proportion at both the word and sequence levels according to the semantic relevance, with greater emphasis placed on keywords and more relevant sequences when performing uncertainty quantification. We compare WSE with 6 baseline methods on 5 free-form medical QA datasets, utilizing 7 "off-the-shelf" large language models (LLMs), and show that WSE exhibits superior performance on accurate uncertainty measurement under two standard criteria for correctness evaluation (e.g., WSE outperforms existing state-of-the-art method by 3.23% AUROC on the MedQA dataset). Additionally, in terms of the potential for real-world medical QA applications, we achieve a significant enhancement in the performance of LLMs when employing sequences with lower uncertainty, identified by WSE, as final answers (e.g., +6.36% accuracy improvement on the COVID-QA dataset), without requiring any additional task-specific fine-tuning or architectural modifications.

Title: Can Large Language Models Detect Misinformation in Scientific News Reporting?

• Authors: Yupeng Cao, Aishwarya Muralidharan Nair, Elyon Eyimife, Nastaran Jamalipour Soofi, K.P. Subbalakshmi, John R. Wullert II, Chumki Basu, David Shallcross
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.14268
• Pdf URL: https://arxiv.org/pdf/2402.14268
• Copy Paste: [[2402.14268]] Can Large Language Models Detect Misinformation in Scientific News Reporting?(https://arxiv.org/abs/2402.14268)
• Keywords: large language model
• Abstract: Scientific facts are often spun in the popular press with the intent to influence public opinion and action, as was evidenced during the COVID-19 pandemic. Automatic detection of misinformation in the scientific domain is challenging because of the distinct styles of writing in these two media types and is still in its nascence. Most research on the validity of scientific reporting treats this problem as a claim verification challenge. In doing so, significant expert human effort is required to generate appropriate claims. Our solution bypasses this step and addresses a more real-world scenario where such explicit, labeled claims may not be available. The central research question of this paper is whether it is possible to use large language models (LLMs) to detect misinformation in scientific reporting. To this end, we first present a new labeled dataset SciNews, containing 2.4k scientific news stories drawn from trusted and untrustworthy sources, paired with related abstracts from the CORD-19 database. Our dataset includes both human-written and LLM-generated news articles, making it more comprehensive in terms of capturing the growing trend of using LLMs to generate popular press articles. Then, we identify dimensions of scientific validity in science news articles and explore how this can be integrated into the automated detection of scientific misinformation. We propose several baseline architectures using LLMs to automatically detect false representations of scientific findings in the popular press. For each of these architectures, we use several prompt engineering strategies including zero-shot, few-shot, and chain-of-thought prompting. We also test these architectures and prompting strategies on GPT-3.5, GPT-4, and Llama2-7B, Llama2-13B.

Title: Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization

• Authors: Xuxi Chen, Zhendong Wang, Daouda Sow, Junjie Yang, Tianlong Chen, Yingbin Liang, Mingyuan Zhou, Zhangyang Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14270
• Pdf URL: https://arxiv.org/pdf/2402.14270
• Copy Paste: [[2402.14270]] Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization(https://arxiv.org/abs/2402.14270)
• Keywords: robust, large language model
• Abstract: In the rapidly advancing arena of large language models (LLMs), a key challenge is to enhance their capabilities amid a looming shortage of high-quality training data. Our study starts from an empirical strategy for the light continual training of LLMs using their original pre-training data sets, with a specific focus on selective retention of samples that incur moderately high losses. These samples are deemed informative and beneficial for model refinement, contrasting with the highest-loss samples, which would be discarded due to their correlation with data noise and complexity. We then formalize this strategy into a principled framework of Instance-Reweighted Distributionally Robust Optimization (IR-DRO). IR-DRO is designed to dynamically prioritize the training focus on informative samples through an instance reweighting mechanism, streamlined by a closed-form solution for straightforward integration into established training protocols. Through rigorous experimentation with various models and datasets, our findings indicate that our sample-targeted methods significantly improve LLM performance across multiple benchmarks, in both continual pre-training and instruction tuning scenarios. Our codes are available at https://github.com/VITA-Group/HardFocusTraining.

Title: Qsnail: A Questionnaire Dataset for Sequential Question Generation

• Authors: Yan Lei, Liang Pang, Yuanzhuo Wang, Huawei Shen, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14272
• Pdf URL: https://arxiv.org/pdf/2402.14272
• Copy Paste: [[2402.14272]] Qsnail: A Questionnaire Dataset for Sequential Question Generation(https://arxiv.org/abs/2402.14272)
• Keywords: generative, large language model
• Abstract: The questionnaire is a professional research methodology used for both qualitative and quantitative analysis of human opinions, preferences, attitudes, and behaviors. However, designing and evaluating questionnaires demands significant effort due to their intricate and complex structure. Questionnaires entail a series of questions that must conform to intricate constraints involving the questions, options, and overall structure. Specifically, the questions should be relevant and specific to the given research topic and intent. The options should be tailored to the questions, ensuring they are mutually exclusive, completed, and ordered sensibly. Moreover, the sequence of questions should follow a logical order, grouping similar topics together. As a result, automatically generating questionnaires presents a significant challenge and this area has received limited attention primarily due to the scarcity of high-quality datasets. To address these issues, we present Qsnail, the first dataset specifically constructed for the questionnaire generation task, which comprises 13,168 human-written questionnaires gathered from online platforms. We further conduct experiments on Qsnail, and the results reveal that retrieval models and traditional generative models do not fully align with the given research topic and intents. Large language models, while more closely related to the research topic and intents, exhibit significant limitations in terms of diversity and specificity. Despite enhancements through the chain-of-thought prompt and finetuning, questionnaires generated by language models still fall short of human-written questionnaires. Therefore, questionnaire generation is challenging and needs to be further explored. The dataset is available at: https://github.com/LeiyanGithub/qsnail.

Title: Can Language Models Act as Knowledge Bases at Scale?

• Authors: Qiyuan He, Yizhong Wang, Wenya Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14273
• Pdf URL: https://arxiv.org/pdf/2402.14273
• Copy Paste: [[2402.14273]] Can Language Models Act as Knowledge Bases at Scale?(https://arxiv.org/abs/2402.14273)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable proficiency in understanding and generating responses to complex queries through large-scale pre-training. However, the efficacy of these models in memorizing and reasoning among large-scale structured knowledge, especially world knowledge that explicitly covers abundant factual information remains questionable. Addressing this gap, our research investigates whether LLMs can effectively store, recall, and reason with knowledge on a large scale comparable to latest knowledge bases (KBs) such as Wikidata. Specifically, we focus on three crucial aspects to study the viability: (1) the efficiency of LLMs with different sizes in memorizing the exact knowledge in the large-scale KB; (2) the flexibility of recalling the memorized knowledge in response to natural language queries; (3) the capability to infer new knowledge through reasoning. Our findings indicate that while LLMs hold promise as large-scale KBs capable of retrieving and responding with flexibility, enhancements in their reasoning capabilities are necessary to fully realize their potential.

Title: GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages

• Authors: Spencer Rarrick, Ranjita Naik, Sundar Poudel, Vishal Chowdhary
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14277
• Pdf URL: https://arxiv.org/pdf/2402.14277
• Copy Paste: [[2402.14277]] GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages(https://arxiv.org/abs/2402.14277)
• Keywords: fair
• Abstract: Neural Machine Translation (NMT) continues to improve in quality and adoption, yet the inadvertent perpetuation of gender bias remains a significant concern. Despite numerous studies on gender bias in translations into English from weakly gendered-languages, there are no benchmarks for evaluating this phenomenon or for assessing mitigation strategies. To address this gap, we introduce GATE X-E, an extension to the GATE (Rarrick et al., 2023) corpus, that consists of human translations from Turkish, Hungarian, Finnish, and Persian into English. Each translation is accompanied by feminine, masculine, and neutral variants. The dataset, which contains between 1250 and 1850 instances for each of the four language pairs, features natural sentences with a wide range of sentence lengths and domains, challenging translation rewriters on various linguistic phenomena. Additionally, we present a translation gender rewriting solution built with GPT-4 and use GATE X-E to evaluate it. We open source our contributions to encourage further research on gender debiasing.

Title: Mitigating the Linguistic Gap with Phonemic Representations for Robust Multilingual Language Understanding

• Authors: Haeji Jung, Changdae Oh, Jooeon Kang, Jimin Sohn, Kyungwoo Song, Jinkyu Kim, David R. Mortensen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14279
• Pdf URL: https://arxiv.org/pdf/2402.14279
• Copy Paste: [[2402.14279]] Mitigating the Linguistic Gap with Phonemic Representations for Robust Multilingual Language Understanding(https://arxiv.org/abs/2402.14279)
• Keywords: robust
• Abstract: Approaches to improving multilingual language understanding often require multiple languages during the training phase, rely on complicated training techniques, and -- importantly -- struggle with significant performance gaps between high-resource and low-resource languages. We hypothesize that the performance gaps between languages are affected by linguistic gaps between those languages and provide a novel solution for robust multilingual language modeling by employing phonemic representations (specifically, using phonemes as input tokens to LMs rather than subwords). We present quantitative evidence from three cross-lingual tasks that demonstrate the effectiveness of phonemic representation, which is further justified by a theoretical analysis of the cross-lingual performance gap.

Title: Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Education

• Authors: Rui Yang, Boming Yang, Sixun Ouyang, Tianwei She, Aosong Feng, Yuang Jiang, Freddy Lecue, Jinghui Lu, Irene Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14293
• Pdf URL: https://arxiv.org/pdf/2402.14293
• Copy Paste: [[2402.14293]] Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Education(https://arxiv.org/abs/2402.14293)
• Keywords: large language model
• Abstract: In the domain of Natural Language Processing (NLP), Large Language Models (LLMs) have demonstrated promise in text-generation tasks. However, their educational applications, particularly for domain-specific queries, remain underexplored. This study investigates LLMs' capabilities in educational scenarios, focusing on concept graph recovery and question-answering (QA). We assess LLMs' zero-shot performance in creating domain-specific concept graphs and introduce TutorQA, a new expert-verified NLP-focused benchmark for scientific graph reasoning and QA. TutorQA consists of five tasks with 500 QA pairs. To tackle TutorQA queries, we present CGLLM, a pipeline integrating concept graphs with LLMs for answering diverse questions. Our results indicate that LLMs' zero-shot concept graph recovery is competitive with supervised methods, showing an average 3% F1 score improvement. In TutorQA tasks, LLMs achieve up to 26% F1 score enhancement. Moreover, human evaluation and analysis show that CGLLM generates answers with more fine-grained concepts.

Title: Mitigating Biases of Large Language Models in Stance Detection with Calibration

• Authors: Ang Li, Jingqian Zhao, Bin Liang, Lin Gui, Hui Wang, Xi Zeng, Kam-Fai Wong, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14296
• Pdf URL: https://arxiv.org/pdf/2402.14296
• Copy Paste: [[2402.14296]] Mitigating Biases of Large Language Models in Stance Detection with Calibration(https://arxiv.org/abs/2402.14296)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved remarkable progress in many natural language processing tasks. However, our experiment reveals that, in stance detection tasks, LLMs may generate biased stances due to spurious sentiment-stance correlation and preference towards certain individuals and topics, thus harming their performance. Therefore, in this paper, we propose to Mitigate Biases of LLMs in stance detection with Calibration (MB-Cal). In which, a novel gated calibration network is devised to mitigate the biases on the stance reasoning results from LLMs. Further, to make the calibration more accurate and generalizable, we construct counterfactual augmented data to rectify stance biases. Experimental results on in-target and zero-shot stance detection tasks show that the proposed MB-Cal can effectively mitigate biases of LLMs, achieving state-of-the-art results.

Title: A Simple Framework Uniting Visual In-context Learning with Masked Image Modeling to Improve Ultrasound Segmentation

• Authors: Yuyue Zhou, Banafshe Felfeliyan, Shrimanti Ghosh, Jessica Knight, Fatima Alves-Pereira, Christopher Keen, Jessica Küpper, Abhilash Rakkunedeth Hareendranathan, Jacob L. Jaremko
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14300
• Pdf URL: https://arxiv.org/pdf/2402.14300
• Copy Paste: [[2402.14300]] A Simple Framework Uniting Visual In-context Learning with Masked Image Modeling to Improve Ultrasound Segmentation(https://arxiv.org/abs/2402.14300)
• Keywords: segmentation
• Abstract: Conventional deep learning models deal with images one-by-one, requiring costly and time-consuming expert labeling in the field of medical imaging, and domain-specific restriction limits model generalizability. Visual in-context learning (ICL) is a new and exciting area of research in computer vision. Unlike conventional deep learning, ICL emphasizes the model's ability to adapt to new tasks based on given examples quickly. Inspired by MAE-VQGAN, we proposed a new simple visual ICL method called SimICL, combining visual ICL pairing images with masked image modeling (MIM) designed for self-supervised learning. We validated our method on bony structures segmentation in a wrist ultrasound (US) dataset with limited annotations, where the clinical objective was to segment bony structures to help with further fracture detection. We used a test set containing 3822 images from 18 patients for bony region segmentation. SimICL achieved an remarkably high Dice coeffient (DC) of 0.96 and Jaccard Index (IoU) of 0.92, surpassing state-of-the-art segmentation and visual ICL models (a maximum DC 0.86 and IoU 0.76), with SimICL DC and IoU increasing up to 0.10 and 0.16. This remarkably high agreement with limited manual annotations indicates SimICL could be used for training AI models even on small US datasets. This could dramatically decrease the human expert time required for image labeling compared to conventional approaches, and enhance the real-world use of AI assistance in US image analysis.

Title: YOLO-TLA: An Efficient and Lightweight Small Object Detection Model based on YOLOv5

• Authors: Peng Gao, Chun-Lin Ji, Tao Yu, Ru-Yue Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14309
• Pdf URL: https://arxiv.org/pdf/2402.14309
• Copy Paste: [[2402.14309]] YOLO-TLA: An Efficient and Lightweight Small Object Detection Model based on YOLOv5(https://arxiv.org/abs/2402.14309)
• Keywords: robust, extraction
• Abstract: Object detection, a crucial aspect of computer vision, has seen significant advancements in accuracy and robustness. Despite these advancements, practical applications still face notable challenges, primarily the inaccurate detection or missed detection of small objects. In this paper, we propose YOLO-TLA, an advanced object detection model building on YOLOv5. We first introduce an additional detection layer for small objects in the neck network pyramid architecture, thereby producing a feature map of a larger scale to discern finer features of small objects. Further, we integrate the C3CrossCovn module into the backbone network. This module uses sliding window feature extraction, which effectively minimizes both computational demand and the number of parameters, rendering the model more compact. Additionally, we have incorporated a global attention mechanism into the backbone network. This mechanism combines the channel information with global information to create a weighted feature map. This feature map is tailored to highlight the attributes of the object of interest, while effectively ignoring irrelevant details. In comparison to the baseline YOLOv5s model, our newly developed YOLO-TLA model has shown considerable improvements on the MS COCO validation dataset, with increases of 4.6% in mAP@0.5 and 4% in mAP@0.5:0.95, all while keeping the model size compact at 9.49M parameters. Further extending these improvements to the YOLOv5m model, the enhanced version exhibited a 1.7% and 1.9% increase in mAP@0.5 and mAP@0.5:0.95, respectively, with a total of 27.53M parameters. These results validate the YOLO-TLA model's efficient and effective performance in small object detection, achieving high accuracy with fewer parameters and computational demands.

Title: Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded Knowledge

• Authors: Jinlan Fu, Shenzhen Huangfu, Hang Yan, See-Kiong Ng, Xipeng Qiu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14310
• Pdf URL: https://arxiv.org/pdf/2402.14310
• Copy Paste: [[2402.14310]] Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded Knowledge(https://arxiv.org/abs/2402.14310)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have recently showcased remarkable generalizability in various domains. Despite their extensive knowledge, LLMs still face challenges in efficiently utilizing encoded knowledge to develop accurate and logical reasoning processes. To mitigate this problem, we introduced Hint-before-Solving Prompting (HSP), which guides the model to generate hints (e.g., specific knowledge or key ideas) for solving the problem and then generate solutions containing intermediate reasoning steps. Since HSP is orthogonal to prompting methods (e.g., Chain-of-Thought (CoT)), we applied HSP to CoT, Least-to-Most, Plan-and-Solve, and Standard promptings. The results of extensive experiments on 6 reasoning benchmarks and 4 open-source LLMs demonstrate that HSP can effectively improve the accuracy of reasoning tasks: (1) By applying high-quality hint-enhanced HSP to CoT prompting, Llama2-70B-Chat shows an improvement of 9.7. (2) Beyond exploring training-free LLM capabilities, we built the HSPMATH dataset based on HSP and fine-tuned Llemma-7B, reaching 64.3 accuracy, surpassing GPT-3.5 and WizardMath-13B. We make our code and dataset publicly available at \url{https://github.com/jinlanfu/HSP}.

Title: Font Style Interpolation with Diffusion Models

• Authors: Tetta Kondo, Shumpei Takezaki, Daichi Haraguchi, Seiichi Uchida
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14311
• Pdf URL: https://arxiv.org/pdf/2402.14311
• Copy Paste: [[2402.14311]] Font Style Interpolation with Diffusion Models(https://arxiv.org/abs/2402.14311)
• Keywords: diffusion, generative
• Abstract: Fonts have huge variations in their styles and give readers different impressions. Therefore, generating new fonts is worthy of giving new impressions to readers. In this paper, we employ diffusion models to generate new font styles by interpolating a pair of reference fonts with different styles. More specifically, we propose three different interpolation approaches, image-blending, condition-blending, and noise-blending, with the diffusion models. We perform qualitative and quantitative experimental analyses to understand the style generation ability of the three approaches. According to experimental results, three proposed approaches can generate not only expected font styles but also somewhat serendipitous font styles. We also compare the approaches with a state-of-the-art style-conditional Latin-font generative network model to confirm the validity of using the diffusion models for the style interpolation task.

Title: Learning to Kern -- Set-wise Estimation of Optimal Letter Space

• Authors: Kei Nakatsuru, Seiichi Uchida
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14313
• Pdf URL: https://arxiv.org/pdf/2402.14313
• Copy Paste: [[2402.14313]] Learning to Kern -- Set-wise Estimation of Optimal Letter Space(https://arxiv.org/abs/2402.14313)
• Keywords: transformer
• Abstract: Kerning is the task of setting appropriate horizontal spaces for all possible letter pairs of a certain font. One of the difficulties of kerning is that the appropriate space differs for each letter pair. Therefore, for a total of 52 capital and small letters, we need to adjust $52 \times 52 = 2704$ different spaces. Another difficulty is that there is neither a general procedure nor criterion for automatic kerning; therefore, kerning is still done manually or with heuristics. In this paper, we tackle kerning by proposing two machine-learning models, called pairwise and set-wise models. The former is a simple deep neural network that estimates the letter space for two given letter images. In contrast, the latter is a Transformer-based model and estimates the letter spaces for three or more given letter images. For example, the set-wise model simultaneously estimates 2704 spaces for 52 letter images for a certain font. Among the two models, the set-wise model is not only more efficient but also more accurate because its internal self-attention mechanism allows for more consistent kerning for all letters. Experimental results on about 2500 Google fonts and their quantitative and qualitative analyses show that the set-wise model has an average estimation error of only about 5.3 pixels when the average letter space of all fonts and letter pairs is about 115 pixels.

Title: Typographic Text Generation with Off-the-Shelf Diffusion Model

• Authors: KhayTze Peong, Seiichi Uchida, Daichi Haraguchi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14314
• Pdf URL: https://arxiv.org/pdf/2402.14314
• Copy Paste: [[2402.14314]] Typographic Text Generation with Off-the-Shelf Diffusion Model(https://arxiv.org/abs/2402.14314)
• Keywords: diffusion, generative
• Abstract: Recent diffusion-based generative models show promise in their ability to generate text images, but limitations in specifying the styles of the generated texts render them insufficient in the realm of typographic design. This paper proposes a typographic text generation system to add and modify text on typographic designs while specifying font styles, colors, and text effects. The proposed system is a novel combination of two off-the-shelf methods for diffusion models, ControlNet and Blended Latent Diffusion. The former functions to generate text images under the guidance of edge conditions specifying stroke contours. The latter blends latent noise in Latent Diffusion Models (LDM) to add typographic text naturally onto an existing background. We first show that given appropriate text edges, ControlNet can generate texts in specified fonts while incorporating effects described by prompts. We further introduce text edge manipulation as an intuitive and customizable way to produce texts with complex effects such as ``shadows'' and ``reflections''. Finally, with the proposed system, we successfully add and modify texts on a predefined background while preserving its overall coherence.

Title: Assessing generalization capability of text ranking models in Polish

• Authors: Sławomir Dadas, Małgorzata Grębowiec
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14318
• Pdf URL: https://arxiv.org/pdf/2402.14318
• Copy Paste: [[2402.14318]] Assessing generalization capability of text ranking models in Polish(https://arxiv.org/abs/2402.14318)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) is becoming an increasingly popular technique for integrating internal knowledge bases with large language models. In a typical RAG pipeline, three models are used, responsible for the retrieval, reranking, and generation stages. In this article, we focus on the reranking problem for the Polish language, examining the performance of rerankers and comparing their results with available retrieval models. We conduct a comprehensive evaluation of existing models and those trained by us, utilizing a benchmark of 41 diverse information retrieval tasks for the Polish language. The results of our experiments show that most models struggle with out-of-domain generalization. However, a combination of effective optimization method and a large training dataset allows for building rerankers that are both compact in size and capable of generalization. The best of our models establishes a new state-of-the-art for reranking in the Polish language, outperforming existing models with up to 30 times more parameters.

Title: Subobject-level Image Tokenization

• Authors: Delong Chen, Samuel Cahyawijaya, Jianfeng Liu, Baoyuan Wang, Pascale Fung
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14327
• Pdf URL: https://arxiv.org/pdf/2402.14327
• Copy Paste: [[2402.14327]] Subobject-level Image Tokenization(https://arxiv.org/abs/2402.14327)
• Keywords: transformer, large language model, segmentation
• Abstract: Transformer-based vision models typically tokenize images into fixed-size square patches as input units, which lacks the adaptability to image content and overlooks the inherent pixel grouping structure. Inspired by the subword tokenization widely adopted in language models, we propose an image tokenizer at a subobject level, where the subobjects are represented by semantically meaningful image segments obtained by segmentation models (e.g., segment anything models). To implement a learning system based on subobject tokenization, we first introduced a Sequence-to-sequence AutoEncoder (SeqAE) to compress subobject segments of varying sizes and shapes into compact embedding vectors, then fed the subobject embeddings into a large language model for vision language learning. Empirical results demonstrated that our subobject-level tokenization significantly facilitates efficient learning of translating images into object and attribute descriptions compared to the traditional patch-level tokenization. Codes and models will be open-sourced at https://github.com/ChenDelong1999/subobjects.

Title: HyperFast: Instant Classification for Tabular Data

• Authors: David Bonet, Daniel Mas Montserrat, Xavier Giró-i-Nieto, Alexander G. Ioannidis
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.14335
• Pdf URL: https://arxiv.org/pdf/2402.14335
• Copy Paste: [[2402.14335]] HyperFast: Instant Classification for Tabular Data(https://arxiv.org/abs/2402.14335)
• Keywords: robust
• Abstract: Training deep learning models and performing hyperparameter tuning can be computationally demanding and time-consuming. Meanwhile, traditional machine learning methods like gradient-boosting algorithms remain the preferred choice for most tabular data applications, while neural network alternatives require extensive hyperparameter tuning or work only in toy datasets under limited settings. In this paper, we introduce HyperFast, a meta-trained hypernetwork designed for instant classification of tabular data in a single forward pass. HyperFast generates a task-specific neural network tailored to an unseen dataset that can be directly used for classification inference, removing the need for training a model. We report extensive experiments with OpenML and genomic data, comparing HyperFast to competing tabular data neural networks, traditional ML methods, AutoML systems, and boosting machines. HyperFast shows highly competitive results, while being significantly faster. Additionally, our approach demonstrates robust adaptability across a variety of classification tasks with little to no fine-tuning, positioning HyperFast as a strong solution for numerous applications and rapid model deployment. HyperFast introduces a promising paradigm for fast classification, with the potential to substantially decrease the computational burden of deep learning. Our code, which offers a scikit-learn-like interface, along with the trained HyperFast model, can be found at https://github.com/AI-sandbox/HyperFast.

Title: AURA: Natural Language Reasoning for Aleatoric Uncertainty in Rationales

• Authors: Hazel Kim
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14337
• Pdf URL: https://arxiv.org/pdf/2402.14337
• Copy Paste: [[2402.14337]] AURA: Natural Language Reasoning for Aleatoric Uncertainty in Rationales(https://arxiv.org/abs/2402.14337)
• Keywords: robust
• Abstract: Rationales behind answers not only explain model decisions but boost language models to reason well on complex reasoning tasks. However, obtaining impeccable rationales is often impossible. Besides, it is non-trivial to estimate the degree to which the rationales are faithful enough to encourage model performance. Thus, such reasoning tasks often compel models to output correct answers under undesirable rationales and are sub-optimal compared to what the models are fully capable of. In this work, we propose how to deal with imperfect rationales causing aleatoric uncertainty. We first define the ambiguous rationales with entropy scores of given rationales, using model prior beliefs as informativeness. We then guide models to select one of two different reasoning models according to the ambiguity of rationales. We empirically argue that our proposed method produces robust performance superiority against the adversarial quality of rationales and low-resource settings.

Title: TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth Estimation

• Authors: Sangwon Choi, Daejune Choi, Duksu Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14340
• Pdf URL: https://arxiv.org/pdf/2402.14340
• Copy Paste: [[2402.14340]] TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth Estimation(https://arxiv.org/abs/2402.14340)
• Keywords: robust
• Abstract: Monocular depth estimation (MDE) is essential for numerous applications yet is impeded by the substantial computational demands of accurate deep learning models. To mitigate this, we introduce a novel Teacher-Independent Explainable Knowledge Distillation (TIE-KD) framework that streamlines the knowledge transfer from complex teacher models to compact student networks, eliminating the need for architectural similarity. The cornerstone of TIE-KD is the Depth Probability Map (DPM), an explainable feature map that interprets the teacher's output, enabling feature-based knowledge distillation solely from the teacher's response. This approach allows for efficient student learning, leveraging the strengths of feature-based distillation. Extensive evaluation of the KITTI dataset indicates that TIE-KD not only outperforms conventional response-based KD methods but also demonstrates consistent efficacy across diverse teacher and student architectures. The robustness and adaptability of TIE-KD underscore its potential for applications requiring efficient and interpretable models, affirming its practicality for real-world deployment.

Title: Exploring Emerging Trends in 5G Malicious Traffic Analysis and Incremental Learning Intrusion Detection Strategies

• Authors: Zihao Wang, Kar Wai Fok, Vrizlynn L. L. Thing
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.14353
• Pdf URL: https://arxiv.org/pdf/2402.14353
• Copy Paste: [[2402.14353]] Exploring Emerging Trends in 5G Malicious Traffic Analysis and Incremental Learning Intrusion Detection Strategies(https://arxiv.org/abs/2402.14353)
• Keywords: security
• Abstract: The popularity of 5G networks poses a huge challenge for malicious traffic detection technology. The reason for this is that as the use of 5G technology increases, so does the risk of malicious traffic activity on 5G networks. Malicious traffic activity in 5G networks not only has the potential to disrupt communication services, but also to compromise sensitive data. This can have serious consequences for individuals and organizations. In this paper, we first provide an in-depth study of 5G technology and 5G security. Next we analyze and discuss the latest malicious traffic detection under AI and their applicability to 5G networks, and compare the various traffic detection aspects addressed by SOTA. The SOTA in 5G traffic detection is also analyzed. Next, we propose seven criteria for traffic monitoring datasets to confirm their suitability for future traffic detection studies. Finally, we present three major issues that need to be addressed for traffic detection in 5G environment. The concept of incremental learning techniques is proposed and applied in the experiments, and the experimental results prove to be able to solve the three problems to some extent.

Title: GAM-Depth: Self-Supervised Indoor Depth Estimation Leveraging a Gradient-Aware Mask and Semantic Constraints

• Authors: Anqi Cheng, Zhiyuan Yang, Haiyue Zhu, Kezhi Mao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14354
• Pdf URL: https://arxiv.org/pdf/2402.14354
• Copy Paste: [[2402.14354]] GAM-Depth: Self-Supervised Indoor Depth Estimation Leveraging a Gradient-Aware Mask and Semantic Constraints(https://arxiv.org/abs/2402.14354)
• Keywords: robust, segmentation
• Abstract: Self-supervised depth estimation has evolved into an image reconstruction task that minimizes a photometric loss. While recent methods have made strides in indoor depth estimation, they often produce inconsistent depth estimation in textureless areas and unsatisfactory depth discrepancies at object boundaries. To address these issues, in this work, we propose GAM-Depth, developed upon two novel components: gradient-aware mask and semantic constraints. The gradient-aware mask enables adaptive and robust supervision for both key areas and textureless regions by allocating weights based on gradient magnitudes.The incorporation of semantic constraints for indoor self-supervised depth estimation improves depth discrepancies at object boundaries, leveraging a co-optimization network and proxy semantic labels derived from a pretrained segmentation model. Experimental studies on three indoor datasets, including NYUv2, ScanNet, and InteriorNet, show that GAM-Depth outperforms existing methods and achieves state-of-the-art performance, signifying a meaningful step forward in indoor depth estimation. Our code will be available at https://github.com/AnqiCheng1234/GAM-Depth.

Title: Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?

• Authors: Ning Bian, Xianpei Han, Hongyu Lin, Yaojie Lu, Ben He, Le Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14355
• Pdf URL: https://arxiv.org/pdf/2402.14355
• Copy Paste: [[2402.14355]] Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?(https://arxiv.org/abs/2402.14355)
• Keywords: large language model
• Abstract: Building machines with commonsense has been a longstanding challenge in NLP due to the reporting bias of commonsense rules and the exposure bias of rule-based commonsense reasoning. In contrast, humans convey and pass down commonsense implicitly through stories. This paper investigates the inherent commonsense ability of large language models (LLMs) expressed through storytelling. We systematically investigate and compare stories and rules for retrieving and leveraging commonsense in LLMs. Experimental results on 28 commonsense QA datasets show that stories outperform rules as the expression for retrieving commonsense from LLMs, exhibiting higher generation confidence and commonsense accuracy. Moreover, stories are the more effective commonsense expression for answering questions regarding daily events, while rules are more effective for scientific questions. This aligns with the reporting bias of commonsense in text corpora. We further show that the correctness and relevance of commonsense stories can be further improved via iterative self-supervised fine-tuning. These findings emphasize the importance of using appropriate language to express, retrieve, and leverage commonsense for LLMs, highlighting a promising direction for better exploiting their commonsense abilities.

Title: Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark

• Authors: Xiuying Chen, Tairan Wang, Qingqing Zhu, Taicheng Guo, Shen Gao, Zhiyong Lu, Xin Gao, Xiangliang Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14359
• Pdf URL: https://arxiv.org/pdf/2402.14359
• Copy Paste: [[2402.14359]] Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark(https://arxiv.org/abs/2402.14359)
• Keywords: large language model
• Abstract: The summarization capabilities of pretrained and large language models (LLMs) have been widely validated in general areas, but their use in scientific corpus, which involves complex sentences and specialized knowledge, has been less assessed. This paper presents conceptual and experimental analyses of scientific summarization, highlighting the inadequacies of traditional evaluation methods, such as $n$-gram, embedding comparison, and QA, particularly in providing explanations, grasping scientific concepts, or identifying key content. Subsequently, we introduce the Facet-aware Metric (FM), employing LLMs for advanced semantic matching to evaluate summaries based on different aspects. This facet-aware approach offers a thorough evaluation of abstracts by decomposing the evaluation task into simpler subtasks.Recognizing the absence of an evaluation benchmark in this domain, we curate a Facet-based scientific summarization Dataset (FD) with facet-level annotations. Our findings confirm that FM offers a more logical approach to evaluating scientific summaries. In addition, fine-tuned smaller models can compete with LLMs in scientific contexts, while LLMs have limitations in learning from in-context information in scientific domains. This suggests an area for future enhancement of LLMs.

Title: OpenTab: Advancing Large Language Models as Open-domain Table Reasoners

• Authors: Kezhi Kong, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Chuan Lei, Christos Faloutsos, Huzefa Rangwala, George Karypis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14361
• Pdf URL: https://arxiv.org/pdf/2402.14361
• Copy Paste: [[2402.14361]] OpenTab: Advancing Large Language Models as Open-domain Table Reasoners(https://arxiv.org/abs/2402.14361)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) trained on large volumes of data excel at various natural language tasks, but they cannot handle tasks requiring knowledge that has not been trained on previously. One solution is to use a retriever that fetches relevant information to expand LLM's knowledge scope. However, existing textual-oriented retrieval-based LLMs are not ideal on structured table data due to diversified data modalities and large table sizes. In this work, we propose OpenTab, an open-domain table reasoning framework powered by LLMs. Overall, OpenTab leverages table retriever to fetch relevant tables and then generates SQL programs to parse the retrieved tables efficiently. Utilizing the intermediate data derived from the SQL executions, it conducts grounded inference to produce accurate response. Extensive experimental evaluation shows that OpenTab significantly outperforms baselines in both open- and closed-domain settings, achieving up to 21.5% higher accuracy. We further run ablation studies to validate the efficacy of our proposed designs of the system.

Title: Small Language Model Is a Good Guide for Large Language Model in Chinese Entity Relation Extraction

• Authors: Xuemei Tang, Jun Wang, Qi Su
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14373
• Pdf URL: https://arxiv.org/pdf/2402.14373
• Copy Paste: [[2402.14373]] Small Language Model Is a Good Guide for Large Language Model in Chinese Entity Relation Extraction(https://arxiv.org/abs/2402.14373)
• Keywords: extraction, large language model
• Abstract: Recently, large language models (LLMs) have been successful in relational extraction (RE) tasks, especially in the few-shot learning. An important problem in the field of RE is long-tailed data, while not much attention is currently paid to this problem using LLM approaches. Therefore, in this paper, we propose SLCoLM, a model collaboration framework, to mitigate the data long-tail problem. In our framework, We use the ``\textit{Training-Guide-Predict}'' strategy to combine the strengths of pre-trained language models (PLMs) and LLMs, where a task-specific PLM framework acts as a tutor, transfers task knowledge to the LLM, and guides the LLM in performing RE tasks. Our experiments on a RE dataset rich in relation types show that the approach in this paper facilitates RE of long-tail relation types.

Title: Novi jezički modeli za srpski jezik

• Authors: Mihailo Škorić
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14379
• Pdf URL: https://arxiv.org/pdf/2402.14379
• Copy Paste: [[2402.14379]] Novi jezički modeli za srpski jezik(https://arxiv.org/abs/2402.14379)
• Keywords: transformer
• Abstract: The paper will briefly present the development history of transformer-based language models for the Serbian language. Several new models for text generation and vectorization, trained on the resources of the Society for Language Resources and Technologies, will also be presented. Ten selected vectorization models for Serbian, including two new ones, will be compared on four natural language processing tasks. Paper will analyze which models are the best for each selected task, how does their size and the size of their training sets affect the performance on those tasks, and what is the optimal setting to train the best language models for the Serbian language.

Title: Generative Adversarial Network with Soft-Dynamic Time Warping and Parallel Reconstruction for Energy Time Series Anomaly Detection

• Authors: Hardik Prabhu, Jayaraman Valadi, Pandarasamy Arjunan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14384
• Pdf URL: https://arxiv.org/pdf/2402.14384
• Copy Paste: [[2402.14384]] Generative Adversarial Network with Soft-Dynamic Time Warping and Parallel Reconstruction for Energy Time Series Anomaly Detection(https://arxiv.org/abs/2402.14384)
• Keywords: generative
• Abstract: In this paper, we employ a 1D deep convolutional generative adversarial network (DCGAN) for sequential anomaly detection in energy time series data. Anomaly detection involves gradient descent to reconstruct energy sub-sequences, identifying the noise vector that closely generates them through the generator network. Soft-DTW is used as a differentiable alternative for the reconstruction loss and is found to be superior to Euclidean distance. Combining reconstruction loss and the latent space's prior probability distribution serves as the anomaly score. Our novel method accelerates detection by parallel computation of reconstruction of multiple points and shows promise in identifying anomalous energy consumption in buildings, as evidenced by performing experiments on hourly energy time series from 15 buildings.

Title: Semantic Image Synthesis with Unconditional Generator

• Authors: Jungwoo Chae, Hyunin Cho, Sooyeon Go, Kyungmook Choi, Youngjung Uh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14395
• Pdf URL: https://arxiv.org/pdf/2402.14395
• Copy Paste: [[2402.14395]] Semantic Image Synthesis with Unconditional Generator(https://arxiv.org/abs/2402.14395)
• Keywords: segmentation
• Abstract: Semantic image synthesis (SIS) aims to generate realistic images that match given semantic masks. Despite recent advances allowing high-quality results and precise spatial control, they require a massive semantic segmentation dataset for training the models. Instead, we propose to employ a pre-trained unconditional generator and rearrange its feature maps according to proxy masks. The proxy masks are prepared from the feature maps of random samples in the generator by simple clustering. The feature rearranger learns to rearrange original feature maps to match the shape of the proxy masks that are either from the original sample itself or from random samples. Then we introduce a semantic mapper that produces the proxy masks from various input conditions including semantic masks. Our method is versatile across various applications such as free-form spatial editing of real images, sketch-to-photo, and even scribble-to-photo. Experiments validate advantages of our method on a range of datasets: human faces, animal faces, and buildings.

Title: Closed-Form Bounds for DP-SGD against Record-level Inference

• Authors: Giovanni Cherubin, Boris Köpf, Andrew Paverd, Shruti Tople, Lukas Wutschitz, Santiago Zanella-Béguelin
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14397
• Pdf URL: https://arxiv.org/pdf/2402.14397
• Copy Paste: [[2402.14397]] Closed-Form Bounds for DP-SGD against Record-level Inference(https://arxiv.org/abs/2402.14397)
• Keywords: privacy, attack, membership infer
• Abstract: Machine learning models trained with differentially-private (DP) algorithms such as DP-SGD enjoy resilience against a wide range of privacy attacks. Although it is possible to derive bounds for some attacks based solely on an $(\varepsilon,\delta)$-DP guarantee, meaningful bounds require a small enough privacy budget (i.e., injecting a large amount of noise), which results in a large loss in utility. This paper presents a new approach to evaluate the privacy of machine learning models against specific record-level threats, such as membership and attribute inference, without the indirection through DP. We focus on the popular DP-SGD algorithm, and derive simple closed-form bounds. Our proofs model DP-SGD as an information theoretic channel whose inputs are the secrets that an attacker wants to infer (e.g., membership of a data record) and whose outputs are the intermediate model parameters produced by iterative optimization. We obtain bounds for membership inference that match state-of-the-art techniques, whilst being orders of magnitude faster to compute. Additionally, we present a novel data-dependent bound against attribute inference. Our results provide a direct, interpretable, and practical way to evaluate the privacy of trained models against specific inference threats without sacrificing utility.

Title: Diffusion Model Based Visual Compensation Guidance and Visual Difference Analysis for No-Reference Image Quality Assessment

• Authors: Zhaoyang Wang, Bo Hu, Mingyang Zhang, Jie Li, Leida Li, Maoguo Gong, Xinbo Gao
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.14401
• Pdf URL: https://arxiv.org/pdf/2402.14401
• Copy Paste: [[2402.14401]] Diffusion Model Based Visual Compensation Guidance and Visual Difference Analysis for No-Reference Image Quality Assessment(https://arxiv.org/abs/2402.14401)
• Keywords: diffusion, transformer, generative
• Abstract: Existing free-energy guided No-Reference Image Quality Assessment (NR-IQA) methods still suffer from finding a balance between learning feature information at the pixel level of the image and capturing high-level feature information and the efficient utilization of the obtained high-level feature information remains a challenge. As a novel class of state-of-the-art (SOTA) generative model, the diffusion model exhibits the capability to model intricate relationships, enabling a comprehensive understanding of images and possessing a better learning of both high-level and low-level visual features. In view of these, we pioneer the exploration of the diffusion model into the domain of NR-IQA. Firstly, we devise a new diffusion restoration network that leverages the produced enhanced image and noise-containing images, incorporating nonlinear features obtained during the denoising process of the diffusion model, as high-level visual information. Secondly, two visual evaluation branches are designed to comprehensively analyze the obtained high-level feature information. These include the visual compensation guidance branch, grounded in the transformer architecture and noise embedding strategy, and the visual difference analysis branch, built on the ResNet architecture and the residual transposed attention block. Extensive experiments are conducted on seven public NR-IQA datasets, and the results demonstrate that the proposed model outperforms SOTA methods for NR-IQA.

Title: On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe

• Authors: Ningyu Xu, Qi Zhang, Menghan Zhang, Peng Qian, Xuanjing Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14404
• Pdf URL: https://arxiv.org/pdf/2402.14404
• Copy Paste: [[2402.14404]] On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe(https://arxiv.org/abs/2402.14404)
• Keywords: robust, large language model
• Abstract: Probing and enhancing large language models' reasoning capacity remains a crucial open question. Here we re-purpose the reverse dictionary task as a case study to probe LLMs' capacity for conceptual inference. We use in-context learning to guide the models to generate the term for an object concept implied in a linguistic description. Models robustly achieve high accuracy in this task, and their representation space encodes information about object categories and fine-grained features. Further experiments suggest that the conceptual inference ability as probed by the reverse-dictionary task predicts model's general reasoning performance across multiple benchmarks, despite similar syntactic generalization behaviors across models. Explorative analyses suggest that prompting LLMs with description$\Rightarrow$word examples may induce generalization beyond surface-level differences in task construals and facilitate models on broader commonsense reasoning problems.

Title: Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning

• Authors: Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, Xuelong Li
• Subjects: cs.LG, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.14407
• Pdf URL: https://arxiv.org/pdf/2402.14407
• Copy Paste: [[2402.14407]] Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning(https://arxiv.org/abs/2402.14407)
• Keywords: diffusion, generative
• Abstract: Learning a generalist embodied agent capable of completing multiple tasks poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. In contrast, a vast amount of human videos exist, capturing intricate tasks and interactions with the physical world. Promising prospects arise for utilizing actionless human videos for pre-training and transferring the knowledge to facilitate robot policy learning through limited robot demonstrations. In this paper, we introduce a novel framework that leverages a unified discrete diffusion to combine generative pre-training on human videos and policy fine-tuning on a small number of action-labeled robot videos. We start by compressing both human and robot videos into unified video tokens. In the pre-training stage, we employ a discrete diffusion model with a mask-and-replace diffusion strategy to predict future video tokens in the latent space. In the fine-tuning stage, we harness the imagined future videos to guide low-level action learning trained on a limited set of robot data. Experiments demonstrate that our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared to previous state-of-the-art approaches with superior generalization ability. Our project website is available at https://video-diff.github.io/.

Title: Transferring BERT Capabilities from High-Resource to Low-Resource Languages Using Vocabulary Matching

• Authors: Piotr Rybak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14408
• Pdf URL: https://arxiv.org/pdf/2402.14408
• Copy Paste: [[2402.14408]] Transferring BERT Capabilities from High-Resource to Low-Resource Languages Using Vocabulary Matching(https://arxiv.org/abs/2402.14408)
• Keywords: transformer
• Abstract: Pre-trained language models have revolutionized the natural language understanding landscape, most notably BERT (Bidirectional Encoder Representations from Transformers). However, a significant challenge remains for low-resource languages, where limited data hinders the effective training of such models. This work presents a novel approach to bridge this gap by transferring BERT capabilities from high-resource to low-resource languages using vocabulary matching. We conduct experiments on the Silesian and Kashubian languages and demonstrate the effectiveness of our approach to improve the performance of BERT models even when the target language has minimal training data. Our results highlight the potential of the proposed technique to effectively train BERT models for low-resource languages, thus democratizing access to advanced language understanding models.

Title: KoCoSa: Korean Context-aware Sarcasm Detection Dataset

• Authors: Yumin Kim, Heejae Suh, Mingi Kim, Dongyeon Won, Hwanhee Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14428
• Pdf URL: https://arxiv.org/pdf/2402.14428
• Copy Paste: [[2402.14428]] KoCoSa: Korean Context-aware Sarcasm Detection Dataset(https://arxiv.org/abs/2402.14428)
• Keywords: large language model
• Abstract: Sarcasm is a way of verbal irony where someone says the opposite of what they mean, often to ridicule a person, situation, or idea. It is often difficult to detect sarcasm in the dialogue since detecting sarcasm should reflect the context (i.e., dialogue history). In this paper, we introduce a new dataset for the Korean dialogue sarcasm detection task, KoCoSa (Korean Context-aware Sarcasm Detection Dataset), which consists of 12.8K daily Korean dialogues and the labels for this task on the last response. To build the dataset, we propose an efficient sarcasm detection dataset generation pipeline: 1) generating new sarcastic dialogues from source dialogues with large language models, 2) automatic and manual filtering of abnormal and toxic dialogues, and 3) human annotation for the sarcasm detection task. We also provide a simple but effective baseline for the Korean sarcasm detection task trained on our dataset. Experimental results on the dataset show that our baseline system outperforms strong baselines like large language models, such as GPT-3.5, in the Korean sarcasm detection task. We show that the sarcasm detection task relies deeply on the existence of sufficient context. We will release the dataset at https://anonymous.4open.science/r/KoCoSa-2372.

Title: Robust Training of Federated Models with Extremely Label Deficiency

• Authors: Yonggang Zhang, Zhiqin Yang, Xinmei Tian, Nannan Wang, Tongliang Liu, Bo Han
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14430
• Pdf URL: https://arxiv.org/pdf/2402.14430
• Copy Paste: [[2402.14430]] Robust Training of Federated Models with Extremely Label Deficiency(https://arxiv.org/abs/2402.14430)
• Keywords: robust, federate
• Abstract: Federated semi-supervised learning (FSSL) has emerged as a powerful paradigm for collaboratively training machine learning models using distributed data with label deficiency. Advanced FSSL methods predominantly focus on training a single model on each client. However, this approach could lead to a discrepancy between the objective functions of labeled and unlabeled data, resulting in gradient conflicts. To alleviate gradient conflict, we propose a novel twin-model paradigm, called Twin-sight, designed to enhance mutual guidance by providing insights from different perspectives of labeled and unlabeled data. In particular, Twin-sight concurrently trains a supervised model with a supervised objective function while training an unsupervised model using an unsupervised objective function. To enhance the synergy between these two models, Twin-sight introduces a neighbourhood-preserving constraint, which encourages the preservation of the neighbourhood relationship among data features extracted by both models. Our comprehensive experiments on four benchmark datasets provide substantial evidence that Twin-sight can significantly outperform state-of-the-art methods across various experimental settings, demonstrating the efficacy of the proposed Twin-sight.

Title: Do LLMs Implicitly Determine the Suitable Text Difficulty for Users?

• Authors: Seiji Gobara, Hidetaka Kamigaito, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14453
• Pdf URL: https://arxiv.org/pdf/2402.14453
• Copy Paste: [[2402.14453]] Do LLMs Implicitly Determine the Suitable Text Difficulty for Users?(https://arxiv.org/abs/2402.14453)
• Keywords: large language model
• Abstract: Education that suits the individual learning level is necessary to improve students' understanding. The first step in achieving this purpose by using large language models (LLMs) is to adjust the textual difficulty of the response to students. This work analyzes how LLMs can implicitly adjust text difficulty between user input and its generated text. To conduct the experiments, we created a new dataset from Stack-Overflow to explore the performance of question-answering-based conversation. Experimental results on the Stack-Overflow dataset and the TSCC dataset, including multi-turn conversation show that LLMs can implicitly handle text difficulty between user input and its generated response. We also observed that some LLMs can surpass humans in handling text difficulty and the importance of instruction-tuning.

Title: CCPA: Long-term Person Re-Identification via Contrastive Clothing and Pose Augmentation

• Authors: Vuong D. Nguyen, Shishir K. Shah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14454
• Pdf URL: https://arxiv.org/pdf/2402.14454
• Copy Paste: [[2402.14454]] CCPA: Long-term Person Re-Identification via Contrastive Clothing and Pose Augmentation(https://arxiv.org/abs/2402.14454)
• Keywords: robust
• Abstract: Long-term Person Re-Identification (LRe-ID) aims at matching an individual across cameras after a long period of time, presenting variations in clothing, pose, and viewpoint. In this work, we propose CCPA: Contrastive Clothing and Pose Augmentation framework for LRe-ID. Beyond appearance, CCPA captures body shape information which is cloth-invariant using a Relation Graph Attention Network. Training a robust LRe-ID model requires a wide range of clothing variations and expensive cloth labeling, which is lacked in current LRe-ID datasets. To address this, we perform clothing and pose transfer across identities to generate images of more clothing variations and of different persons wearing similar clothing. The augmented batch of images serve as inputs to our proposed Fine-grained Contrastive Losses, which not only supervise the Re-ID model to learn discriminative person embeddings under long-term scenarios but also ensure in-distribution data generation. Results on LRe-ID datasets demonstrate the effectiveness of our CCPA framework.

Title: VLPose: Bridging the Domain Gap in Pose Estimation with Language-Vision Tuning

• Authors: Jingyao Li, Pengguang Chen, Xuan Ju, Hong Xu, Jiaya Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14456
• Pdf URL: https://arxiv.org/pdf/2402.14456
• Copy Paste: [[2402.14456]] VLPose: Bridging the Domain Gap in Pose Estimation with Language-Vision Tuning(https://arxiv.org/abs/2402.14456)
• Keywords: robust
• Abstract: Thanks to advances in deep learning techniques, Human Pose Estimation (HPE) has achieved significant progress in natural scenarios. However, these models perform poorly in artificial scenarios such as painting and sculpture due to the domain gap, constraining the development of virtual reality and augmented reality. With the growth of model size, retraining the whole model on both natural and artificial data is computationally expensive and inefficient. Our research aims to bridge the domain gap between natural and artificial scenarios with efficient tuning strategies. Leveraging the potential of language models, we enhance the adaptability of traditional pose estimation models across diverse scenarios with a novel framework called VLPose. VLPose leverages the synergy between language and vision to extend the generalization and robustness of pose estimation models beyond the traditional domains. Our approach has demonstrated improvements of 2.26% and 3.74% on HumanArt and MSCOCO, respectively, compared to state-of-the-art tuning strategies.

Title: S^2Former-OR: Single-Stage Bimodal Transformer for Scene Graph Generation in OR

• Authors: Jialun Pei, Diandian Guo, Jingyang Zhang, Manxi Lin, Yueming Jin, Pheng-Ann Heng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14461
• Pdf URL: https://arxiv.org/pdf/2402.14461
• Copy Paste: [[2402.14461]] S^2Former-OR: Single-Stage Bimodal Transformer for Scene Graph Generation in OR(https://arxiv.org/abs/2402.14461)
• Keywords: transformer
• Abstract: Scene graph generation (SGG) of surgical procedures is crucial in enhancing holistically cognitive intelligence in the operating room (OR). However, previous works have primarily relied on the multi-stage learning that generates semantic scene graphs dependent on intermediate processes with pose estimation and object detection, which may compromise model efficiency and efficacy, also impose extra annotation burden. In this study, we introduce a novel single-stage bimodal transformer framework for SGG in the OR, termed S^2Former-OR, aimed to complementally leverage multi-view 2D scenes and 3D point clouds for SGG in an end-to-end manner. Concretely, our model embraces a View-Sync Transfusion scheme to encourage multi-view visual information interaction. Concurrently, a Geometry-Visual Cohesion operation is designed to integrate the synergic 2D semantic features into 3D point cloud features. Moreover, based on the augmented feature, we propose a novel relation-sensitive transformer decoder that embeds dynamic entity-pair queries and relational trait priors, which enables the direct prediction of entity-pair relations for graph generation without intermediate steps. Extensive experiments have validated the superior SGG performance and lower computational cost of S^2Former-OR on 4D-OR benchmark, compared with current OR-SGG methods, e.g., 3% Precision increase and 24.2M reduction in model parameters. We further compared our method with generic single-stage SGG methods with broader metrics for a comprehensive evaluation, with consistently better performance achieved. The code will be made available.

Title: NeRF-Det++: Incorporating Semantic Cues and Perspective-aware Depth Supervision for Indoor Multi-View 3D Detection

• Authors: Chenxi Huang, Yuenan Hou, Weicai Ye, Di Huang, Xiaoshui Huang, Binbin Lin, Deng Cai, Wanli Ouyang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14464
• Pdf URL: https://arxiv.org/pdf/2402.14464
• Copy Paste: [[2402.14464]] NeRF-Det++: Incorporating Semantic Cues and Perspective-aware Depth Supervision for Indoor Multi-View 3D Detection(https://arxiv.org/abs/2402.14464)
• Keywords: segmentation
• Abstract: NeRF-Det has achieved impressive performance in indoor multi-view 3D detection by innovatively utilizing NeRF to enhance representation learning. Despite its notable performance, we uncover three decisive shortcomings in its current design, including semantic ambiguity, inappropriate sampling, and insufficient utilization of depth supervision. To combat the aforementioned problems, we present three corresponding solutions: 1) Semantic Enhancement. We project the freely available 3D segmentation annotations onto the 2D plane and leverage the corresponding 2D semantic maps as the supervision signal, significantly enhancing the semantic awareness of multi-view detectors. 2) Perspective-aware Sampling. Instead of employing the uniform sampling strategy, we put forward the perspective-aware sampling policy that samples densely near the camera while sparsely in the distance, more effectively collecting the valuable geometric clues. 3)Ordinal Residual Depth Supervision. As opposed to directly regressing the depth values that are difficult to optimize, we divide the depth range of each scene into a fixed number of ordinal bins and reformulate the depth prediction as the combination of the classification of depth bins as well as the regression of the residual depth values, thereby benefiting the depth learning process. The resulting algorithm, NeRF-Det++, has exhibited appealing performance in the ScanNetV2 and ARKITScenes datasets. Notably, in ScanNetV2, NeRF-Det++ outperforms the competitive NeRF-Det by +1.9% in mAP@0.25 and +3.5% in mAP@0.50$. The code will be publicly at https://github.com/mrsempress/NeRF-Detplusplus.

Title: Data Science with LLMs and Interpretable Models

• Authors: Sebastian Bordt, Ben Lengerich, Harsha Nori, Rich Caruana
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14474
• Pdf URL: https://arxiv.org/pdf/2402.14474
• Copy Paste: [[2402.14474]] Data Science with LLMs and Interpretable Models(https://arxiv.org/abs/2402.14474)
• Keywords: large language model
• Abstract: Recent years have seen important advances in the building of interpretable models, machine learning models that are designed to be easily understood by humans. In this work, we show that large language models (LLMs) are remarkably good at working with interpretable models, too. In particular, we show that LLMs can describe, interpret, and debug Generalized Additive Models (GAMs). Combining the flexibility of LLMs with the breadth of statistical patterns accurately described by GAMs enables dataset summarization, question answering, and model critique. LLMs can also improve the interaction between domain experts and interpretable models, and generate hypotheses about the underlying phenomenon. We release \url{https://github.com/interpretml/TalkToEBM} as an open-source LLM-GAM interface.

Title: DynGMA: a robust approach for learning stochastic differential equations from data

• Authors: Aiqing Zhu, Qianxiao Li
• Subjects: cs.LG, math.NA, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2402.14475
• Pdf URL: https://arxiv.org/pdf/2402.14475
• Copy Paste: [[2402.14475]] DynGMA: a robust approach for learning stochastic differential equations from data(https://arxiv.org/abs/2402.14475)
• Keywords: robust, diffusion
• Abstract: Learning unknown stochastic differential equations (SDEs) from observed data is a significant and challenging task with applications in various fields. Current approaches often use neural networks to represent drift and diffusion functions, and construct likelihood-based loss by approximating the transition density to train these networks. However, these methods often rely on one-step stochastic numerical schemes, necessitating data with sufficiently high time resolution. In this paper, we introduce novel approximations to the transition density of the parameterized SDE: a Gaussian density approximation inspired by the random perturbation theory of dynamical systems, and its extension, the dynamical Gaussian mixture approximation (DynGMA). Benefiting from the robust density approximation, our method exhibits superior accuracy compared to baseline methods in learning the fully unknown drift and diffusion functions and computing the invariant distribution from trajectory data. And it is capable of handling trajectory data with low time resolution and variable, even uncontrollable, time step sizes, such as data generated from Gillespie's stochastic simulations. We then conduct several experiments across various scenarios to verify the advantages and robustness of the proposed method.

Title: Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysis

• Authors: Takehiro Takayanagi, Masahiro Suzuki, Ryotaro Kobayashi, Hiroki Sakaji, Kiyoshi Izumi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14484
• Pdf URL: https://arxiv.org/pdf/2402.14484
• Copy Paste: [[2402.14484]] Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysis(https://arxiv.org/abs/2402.14484)
• Keywords: fair
• Abstract: Causality is fundamental in human cognition and has drawn attention in diverse research fields. With growing volumes of textual data, discerning causalities within text data is crucial, and causal text mining plays a pivotal role in extracting meaningful patterns. This study conducts comprehensive evaluations of ChatGPT's causal text mining capabilities. Firstly, we introduce a benchmark that extends beyond general English datasets, including domain-specific and non-English datasets. We also provide an evaluation framework to ensure fair comparisons between ChatGPT and previous approaches. Finally, our analysis outlines the limitations and future challenges in employing ChatGPT for causal text mining. Specifically, our analysis reveals that ChatGPT serves as a good starting point for various datasets. However, when equipped with a sufficient amount of training data, previous models still surpass ChatGPT's performance. Additionally, ChatGPT suffers from the tendency to falsely recognize non-causal sequences as causal sequences. These issues become even more pronounced with advanced versions of the model, such as GPT-4. In addition, we highlight the constraints of ChatGPT in handling complex causality types, including both intra/inter-sentential and implicit causality. The model also faces challenges with effectively leveraging in-context learning and domain adaptation. Our code is available on \url{https://github.com/retarfi/gemcausal}

Title: Does the Generator Mind its Contexts? An Analysis of Generative Model Faithfulness under Context Transfer

• Authors: Xinshuo Hu, Baotian Hu, Dongfang Li, Xiaoguang Li, Lifeng Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14488
• Pdf URL: https://arxiv.org/pdf/2402.14488
• Copy Paste: [[2402.14488]] Does the Generator Mind its Contexts? An Analysis of Generative Model Faithfulness under Context Transfer(https://arxiv.org/abs/2402.14488)
• Keywords: generative
• Abstract: The present study introduces the knowledge-augmented generator, which is specifically designed to produce information that remains grounded in contextual knowledge, regardless of alterations in the context. Previous research has predominantly focused on examining hallucinations stemming from static input, such as in the domains of summarization or machine translation. However, our investigation delves into the faithfulness of generative question answering in the presence of dynamic knowledge. Our objective is to explore the existence of hallucinations arising from parametric memory when contextual knowledge undergoes changes, while also analyzing the underlying causes for their occurrence. In order to efficiently address this issue, we propose a straightforward yet effective measure for detecting such hallucinations. Intriguingly, our investigation uncovers that all models exhibit a tendency to generate previous answers as hallucinations. To gain deeper insights into the underlying causes of this phenomenon, we conduct a series of experiments that verify the critical role played by context in hallucination, both during training and testing, from various perspectives.

Title: INSTRAUG: Automatic Instruction Augmentation for Multimodal Instruction Fine-tuning

• Authors: Wei Han, Hui Chen, Soujanya Poria
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14492
• Pdf URL: https://arxiv.org/pdf/2402.14492
• Copy Paste: [[2402.14492]] INSTRAUG: Automatic Instruction Augmentation for Multimodal Instruction Fine-tuning(https://arxiv.org/abs/2402.14492)
• Keywords: large language model
• Abstract: Fine-tuning large language models (LLMs) on multi-task instruction-following data has been proven to be a powerful learning paradigm for improving their zero-shot capabilities on new tasks. Recent works about high-quality instruction-following data generation and selection require amounts of human labor to conceive model-understandable instructions for the given tasks and carefully filter the LLM-generated data. In this work, we introduce an automatic instruction augmentation method named INSTRAUG in multimodal tasks. It starts from a handful of basic and straightforward meta instructions but can expand an instruction-following dataset by 30 times. Results on two popular multimodal instructionfollowing benchmarks MULTIINSTRUCT and InstructBLIP show that INSTRAUG can significantly improve the alignment of multimodal large language models (MLLMs) across 12 multimodal tasks, which is even equivalent to the benefits of scaling up training data multiple times.

Title: Noise-BERT: A Unified Perturbation-Robust Framework with Noise Alignment Pre-training for Noisy Slot Filling Task

• Authors: Jinxu Zhao, Guanting Dong, Yueyan Qiu, Tingfeng Hui, Xiaoshuai Song, Daichi Guo, Weiran Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14494
• Pdf URL: https://arxiv.org/pdf/2402.14494
• Copy Paste: [[2402.14494]] Noise-BERT: A Unified Perturbation-Robust Framework with Noise Alignment Pre-training for Noisy Slot Filling Task(https://arxiv.org/abs/2402.14494)
• Keywords: attack, robust
• Abstract: In a realistic dialogue system, the input information from users is often subject to various types of input perturbations, which affects the slot-filling task. Although rule-based data augmentation methods have achieved satisfactory results, they fail to exhibit the desired generalization when faced with unknown noise disturbances. In this study, we address the challenges posed by input perturbations in slot filling by proposing Noise-BERT, a unified Perturbation-Robust Framework with Noise Alignment Pre-training. Our framework incorporates two Noise Alignment Pre-training tasks: Slot Masked Prediction and Sentence Noisiness Discrimination, aiming to guide the pre-trained language model in capturing accurate slot information and noise distribution. During fine-tuning, we employ a contrastive learning loss to enhance the semantic representation of entities and labels. Additionally, we introduce an adversarial attack training strategy to improve the model's robustness. Experimental results demonstrate the superiority of our proposed approach over state-of-the-art models, and further analysis confirms its effectiveness and generalization ability.

Title: "My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models

• Authors: Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14499
• Pdf URL: https://arxiv.org/pdf/2402.14499
• Copy Paste: [[2402.14499]] "My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models(https://arxiv.org/abs/2402.14499)
• Keywords: robust, large language model
• Abstract: The open-ended nature of language generation makes the evaluation of autoregressive large language models (LLMs) challenging. One common evaluation approach uses multiple-choice questions (MCQ) to limit the response space. The model is then evaluated by ranking the candidate answers by the log probability of the first token prediction. However, first-tokens may not consistently reflect the final response output, due to model's diverse response styles such as starting with "Sure" or refusing to answer. Consequently, MCQ evaluation is not indicative of model behaviour when interacting with users. But by how much? We evaluate how aligned first-token evaluation is with the text output along several dimensions, namely final option choice, refusal rate, choice distribution and robustness under prompt perturbation. Our results show that the two approaches are severely misaligned on all dimensions, reaching mismatch rates over 60%. Models heavily fine-tuned on conversational or safety data are especially impacted. Crucially, models remain misaligned even when we increasingly constrain prompts, i.e., force them to start with an option letter or example template. Our findings i) underscore the importance of inspecting the text output, too and ii) caution against relying solely on first-token evaluation.

Title: Malaysian English News Decoded: A Linguistic Resource for Named Entity and Relation Extraction

• Authors: Mohan Raj Chanthran, Lay-Ki Soon, Huey Fang Ong, Bhawani Selvaretnam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14521
• Pdf URL: https://arxiv.org/pdf/2402.14521
• Copy Paste: [[2402.14521]] Malaysian English News Decoded: A Linguistic Resource for Named Entity and Relation Extraction(https://arxiv.org/abs/2402.14521)
• Keywords: extraction
• Abstract: Standard English and Malaysian English exhibit notable differences, posing challenges for natural language processing (NLP) tasks on Malaysian English. Unfortunately, most of the existing datasets are mainly based on standard English and therefore inadequate for improving NLP tasks in Malaysian English. An experiment using state-of-the-art Named Entity Recognition (NER) solutions on Malaysian English news articles highlights that they cannot handle morphosyntactic variations in Malaysian English. To the best of our knowledge, there is no annotated dataset available to improvise the model. To address these issues, we constructed a Malaysian English News (MEN) dataset, which contains 200 news articles that are manually annotated with entities and relations. We then fine-tuned the spaCy NER tool and validated that having a dataset tailor-made for Malaysian English could improve the performance of NER in Malaysian English significantly. This paper presents our effort in the data acquisition, annotation methodology, and thorough analysis of the annotated dataset. To validate the quality of the annotation, inter-annotator agreement was used, followed by adjudication of disagreements by a subject matter expert. Upon completion of these tasks, we managed to develop a dataset with 6,061 entities and 3,268 relation instances. Finally, we discuss on spaCy fine-tuning setup and analysis on the NER performance. This unique dataset will contribute significantly to the advancement of NLP research in Malaysian English, allowing researchers to accelerate their progress, particularly in NER and relation extraction. The dataset and annotation guideline has been published on Github.

Title: Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond

• Authors: Xinyu Wang, Hainiu Xu, Lin Gui, Yulan He
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14522
• Pdf URL: https://arxiv.org/pdf/2402.14522
• Copy Paste: [[2402.14522]] Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond(https://arxiv.org/abs/2402.14522)
• Keywords: interpretability, large language model
• Abstract: Task embedding, a meta-learning technique that captures task-specific information, has become prevalent, especially in areas such as multi-task learning, model editing, and interpretability. However, it faces challenges with the emergence of prompt-guided Large Language Models (LLMs) operating in a gradientfree manner. Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders the adaptability of task embeddings across diverse models, especially prompt-based LLMs. To unleash the power of task embedding in the era of LLMs, we propose a framework for unified task embeddings (FUTE), harmonizing task embeddings from various models, including smaller language models and LLMs with varied prompts, within a single vector space. Such uniformity enables the comparison and analysis of similarities amongst different models, extending the scope and utility of existing task embedding methods in addressing multi-model scenarios, whilst maintaining their performance to be comparable to architecture-specific methods.

Title: Balanced Data Sampling for Language Model Training with Clustering

• Authors: Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, Xipeng Qiu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14526
• Pdf URL: https://arxiv.org/pdf/2402.14526
• Copy Paste: [[2402.14526]] Balanced Data Sampling for Language Model Training with Clustering(https://arxiv.org/abs/2402.14526)
• Keywords: large language model
• Abstract: Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sampling strategy in training remains an open question. Most LLMs are trained with a simple strategy, random sampling. However, this sampling strategy ignores the unbalanced nature of training data distribution, which can be sub-optimal. In this paper, we propose ClusterClip Sampling to balance the text distribution of training data for better model training. Specifically, ClusterClip Sampling utilizes data clustering to reflect the data distribution of the training set and balances the common samples and rare samples during training based on the cluster results. A repetition clip operation is introduced to mitigate the overfitting issue led by samples from certain clusters. Extensive experiments validate the effectiveness of ClusterClip Sampling, which outperforms random sampling and other cluster-based sampling variants under various training datasets and large language models.

Title: Federated Learning on Transcriptomic Data: Model Quality and Performance Trade-Offs

• Authors: Anika Hannemann, Jan Ewald, Leo Seeger, Erik Buchmann
• Subjects: cs.LG, cs.CR, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2402.14527
• Pdf URL: https://arxiv.org/pdf/2402.14527
• Copy Paste: [[2402.14527]] Federated Learning on Transcriptomic Data: Model Quality and Performance Trade-Offs(https://arxiv.org/abs/2402.14527)
• Keywords: privacy, robust, federate
• Abstract: Machine learning on large-scale genomic or transcriptomic data is important for many novel health applications. For example, precision medicine tailors medical treatments to patients on the basis of individual biomarkers, cellular and molecular states, etc. However, the data required is sensitive, voluminous, heterogeneous, and typically distributed across locations where dedicated machine learning hardware is not available. Due to privacy and regulatory reasons, it is also problematic to aggregate all data at a trusted third party.Federated learning is a promising solution to this dilemma, because it enables decentralized, collaborative machine learning without exchanging raw data. In this paper, we perform comparative experiments with the federated learning frameworks TensorFlow Federated and Flower. Our test case is the training of disease prognosis and cell type classification models. We train the models with distributed transcriptomic data, considering both data heterogeneity and architectural heterogeneity. We measure model quality, robustness against privacy-enhancing noise, computational performance and resource overhead. Each of the federated learning frameworks has different strengths. However, our experiments confirm that both frameworks can readily build models on transcriptomic data, without transferring personal raw data to a third party with abundant computational resources.

Title: Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance

• Authors: Ziqi Yin, Hao Wang, Kaito Horio, Daisuke Kawahara, Satoshi Sekine
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14531
• Pdf URL: https://arxiv.org/pdf/2402.14531
• Copy Paste: [[2402.14531]] Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance(https://arxiv.org/abs/2402.14531)
• Keywords: large language model
• Abstract: We investigate the impact of politeness levels in prompts on the performance of large language models (LLMs). Polite language in human communications often garners more compliance and effectiveness, while rudeness can cause aversion, impacting response quality. We consider that LLMs mirror human communication traits, suggesting they align with human cultural norms. We assess the impact of politeness in prompts on LLMs across English, Chinese, and Japanese tasks. We observed that impolite prompts often result in poor performance, but overly polite language does not guarantee better outcomes. The best politeness level is different according to the language. This phenomenon suggests that LLMs not only reflect human behavior but are also influenced by language, particularly in different cultural contexts. Our findings highlight the need to factor in politeness for cross-cultural natural language processing and LLM usage.

Title: Whose LLM is it Anyway? Linguistic Comparison and LLM Attribution for GPT-3.5, GPT-4 and Bard

• Authors: Ariel Rosenfeld, Teddy Lazebnik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14533
• Pdf URL: https://arxiv.org/pdf/2402.14533
• Copy Paste: [[2402.14533]] Whose LLM is it Anyway? Linguistic Comparison and LLM Attribution for GPT-3.5, GPT-4 and Bard(https://arxiv.org/abs/2402.14533)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are capable of generating text that is similar to or surpasses human quality. However, it is unclear whether LLMs tend to exhibit distinctive linguistic styles akin to how human authors do. Through a comprehensive linguistic analysis, we compare the vocabulary, Part-Of-Speech (POS) distribution, dependency distribution, and sentiment of texts generated by three of the most popular LLMS today (GPT-3.5, GPT-4, and Bard) to diverse inputs. The results point to significant linguistic variations which, in turn, enable us to attribute a given text to its LLM origin with a favorable 88\% accuracy using a simple off-the-shelf classification model. Theoretical and practical implications of this intriguing finding are discussed.

Title: Domain Generalization via Causal Adjustment for Cross-Domain Sentiment Analysis

• Authors: Siyin Wang, Jie Zhou, Qin Chen, Qi Zhang, Tao Gui, Xuanjing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14536
• Pdf URL: https://arxiv.org/pdf/2402.14536
• Copy Paste: [[2402.14536]] Domain Generalization via Causal Adjustment for Cross-Domain Sentiment Analysis(https://arxiv.org/abs/2402.14536)
• Keywords: robust
• Abstract: Domain adaption has been widely adapted for cross-domain sentiment analysis to transfer knowledge from the source domain to the target domain. Whereas, most methods are proposed under the assumption that the target (test) domain is known, making them fail to generalize well on unknown test data that is not always available in practice. In this paper, we focus on the problem of domain generalization for cross-domain sentiment analysis. Specifically, we propose a backdoor adjustment-based causal model to disentangle the domain-specific and domain-invariant representations that play essential roles in tackling domain shift. First, we rethink the cross-domain sentiment analysis task in a causal view to model the causal-and-effect relationships among different variables. Then, to learn an invariant feature representation, we remove the effect of domain confounders (e.g., domain knowledge) using the backdoor adjustment. A series of experiments over many homologous and diverse datasets show the great performance and robustness of our model by comparing it with the state-of-the-art domain generalization baselines.

Title: {A New Hope}: Contextual Privacy Policies for Mobile Applications and An Approach Toward Automated Generation

• Authors: Shidong Pan, Zhen Tao, Thong Hoang, Dawen Zhang, Tianshi Li, Zhenchang Xing, Sherry Xu, Mark Staples, Thierry Rakotoarivelo, David Lo
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.14544
• Pdf URL: https://arxiv.org/pdf/2402.14544
• Copy Paste: [[2402.14544]] {A New Hope}: Contextual Privacy Policies for Mobile Applications and An Approach Toward Automated Generation(https://arxiv.org/abs/2402.14544)
• Keywords: privacy
• Abstract: Privacy policies have emerged as the predominant approach to conveying privacy notices to mobile application users. In an effort to enhance both readability and user engagement, the concept of contextual privacy policies (CPPs) has been proposed by researchers. The aim of CPPs is to fragment privacy policies into concise snippets, displaying them only within the corresponding contexts within the application's graphical user interfaces (GUIs). In this paper, we first formulate CPP in mobile application scenario, and then present a novel multimodal framework, named SeePrivacy, specifically designed to automatically generate CPPs for mobile applications. This method uniquely integrates vision-based GUI understanding with privacy policy analysis, achieving 0.88 precision and 0.90 recall to detect contexts, as well as 0.98 precision and 0.96 recall in extracting corresponding policy segments. A human evaluation shows that 77% of the extracted privacy policy segments were perceived as well-aligned with the detected contexts. These findings suggest that SeePrivacy could serve as a significant tool for bolstering user interaction with, and understanding of, privacy policies. Furthermore, our solution has the potential to make privacy notices more accessible and inclusive, thus appealing to a broader demographic. A demonstration of our work can be accessed at https://cpp4app.github.io/SeePrivacy/

Title: LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey

• Authors: Ashok Urlana, Charaka Vinayak Kumar, Ajeet Kumar Singh, Bala Mallikarjunarao Garlapati, Srinivasa Rao Chalamala, Rahul Mishra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14558
• Pdf URL: https://arxiv.org/pdf/2402.14558
• Copy Paste: [[2402.14558]] LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey(https://arxiv.org/abs/2402.14558)
• Keywords: large language model
• Abstract: Large language models (LLMs) have become the secret ingredient driving numerous industrial applications, showcasing their remarkable versatility across a diverse spectrum of tasks. From natural language processing and sentiment analysis to content generation and personalized recommendations, their unparalleled adaptability has facilitated widespread adoption across industries. This transformative shift driven by LLMs underscores the need to explore the underlying associated challenges and avenues for enhancement in their utilization. In this paper, our objective is to unravel and evaluate the obstacles and opportunities inherent in leveraging LLMs within an industrial context. To this end, we conduct a survey involving a group of industry practitioners, develop four research questions derived from the insights gathered, and examine 68 industry papers to address these questions and derive meaningful conclusions.

Title: LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition

• Authors: Junjie Ye, Nuo Xu, Yikun Wang, Jie Zhou, Qi Zhang, Tao Gui, Xuanjing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14568
• Pdf URL: https://arxiv.org/pdf/2402.14568
• Copy Paste: [[2402.14568]] LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition(https://arxiv.org/abs/2402.14568)
• Keywords: robust, extraction, large language model
• Abstract: Despite the impressive capabilities of large language models (LLMs), their performance on information extraction tasks is still not entirely satisfactory. However, their remarkable rewriting capabilities and extensive world knowledge offer valuable insights to improve these tasks. In this paper, we propose $LLM-DA$, a novel data augmentation technique based on LLMs for the few-shot NER task. To overcome the limitations of existing data augmentation methods that compromise semantic integrity and address the uncertainty inherent in LLM-generated text, we leverage the distinctive characteristics of the NER task by augmenting the original data at both the contextual and entity levels. Our approach involves employing 14 contextual rewriting strategies, designing entity replacements of the same type, and incorporating noise injection to enhance robustness. Extensive experiments demonstrate the effectiveness of our approach in enhancing NER model performance with limited data. Furthermore, additional analyses provide further evidence supporting the assertion that the quality of the data we generate surpasses that of other existing methods.

Title: Debiasing Text-to-Image Diffusion Models

• Authors: Ruifei He, Chuhui Xue, Haoru Tan, Wenqing Zhang, Yingchen Yu, Song Bai, Xiaojuan Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14577
• Pdf URL: https://arxiv.org/pdf/2402.14577
• Copy Paste: [[2402.14577]] Debiasing Text-to-Image Diffusion Models(https://arxiv.org/abs/2402.14577)
• Keywords: diffusion
• Abstract: Learning-based Text-to-Image (TTI) models like Stable Diffusion have revolutionized the way visual content is generated in various domains. However, recent research has shown that nonnegligible social bias exists in current state-of-the-art TTI systems, which raises important concerns. In this work, we target resolving the social bias in TTI diffusion models. We begin by formalizing the problem setting and use the text descriptions of bias groups to establish an unsafe direction for guiding the diffusion process. Next, we simplify the problem into a weight optimization problem and attempt a Reinforcement solver, Policy Gradient, which shows sub-optimal performance with slow convergence. Further, to overcome limitations, we propose an iterative distribution alignment (IDA) method. Despite its simplicity, we show that IDA shows efficiency and fast convergence in resolving the social bias in TTI diffusion models. Our code will be released.

Title: Text Role Classification in Scientific Charts Using Multimodal Transformers

• Authors: Hye Jin Kim, Nicolas Lell, Ansgar Scherp
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14579
• Pdf URL: https://arxiv.org/pdf/2402.14579
• Copy Paste: [[2402.14579]] Text Role Classification in Scientific Charts Using Multimodal Transformers(https://arxiv.org/abs/2402.14579)
• Keywords: robust, transformer
• Abstract: Text role classification involves classifying the semantic role of textual elements within scientific charts. For this task, we propose to finetune two pretrained multimodal document layout analysis models, LayoutLMv3 and UDOP, on chart datasets. The transformers utilize the three modalities of text, image, and layout as input. We further investigate whether data augmentation and balancing methods help the performance of the models. The models are evaluated on various chart datasets, and results show that LayoutLMv3 outperforms UDOP in all experiments. LayoutLMv3 achieves the highest F1-macro score of 82.87 on the ICPR22 test dataset, beating the best-performing model from the ICPR22 CHART-Infographics challenge. Moreover, the robustness of the models is tested on a synthetic noisy dataset ICPR22-N. Finally, the generalizability of the models is evaluated on three chart datasets, CHIME-R, DeGruyter, and EconBiz, for which we added labels for the text roles. Findings indicate that even in cases where there is limited training data, transformers can be used with the help of data augmentation and balancing methods. The source code and datasets are available on GitHub under https://github.com/hjkimk/text-role-classification

Title: Enhancing SCADA Security: Developing a Host-Based Intrusion Detection System to Safeguard Against Cyberattacks

• Authors: Omer Sen, Tarek Hassan, Andreas Ulbig, Martin Henze
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.14599
• Pdf URL: https://arxiv.org/pdf/2402.14599
• Copy Paste: [[2402.14599]] Enhancing SCADA Security: Developing a Host-Based Intrusion Detection System to Safeguard Against Cyberattacks(https://arxiv.org/abs/2402.14599)
• Keywords: security, protect, attack
• Abstract: With the increasing reliance of smart grids on correctly functioning SCADA systems and their vulnerability to cyberattacks, there is a pressing need for effective security measures. SCADA systems are prone to cyberattacks, posing risks to critical infrastructure. As there is a lack of host-based intrusion detection systems specifically designed for the stable nature of SCADA systems, the objective of this work is to propose a host-based intrusion detection system tailored for SCADA systems in smart grids. The proposed system utilizes USB device identification, flagging, and process memory scanning to monitor and detect anomalies in SCADA systems, providing enhanced security measures. Evaluation in three different scenarios demonstrates the tool's effectiveness in detecting and disabling malware. The proposed approach effectively identifies potential threats and enhances the security of SCADA systems in smart grids, providing a promising solution to protect against cyberattacks.

Title: Federated Complex Qeury Answering

• Authors: Qi Hu, Weifeng Jiang, Haoran Li, Zihao Wang, Jiaxin Bai, Qianren Mao, Yangqiu Song, Lixin Fan, Jianxin Li
• Subjects: cs.LG, cs.AI, cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2402.14609
• Pdf URL: https://arxiv.org/pdf/2402.14609
• Copy Paste: [[2402.14609]] Federated Complex Qeury Answering(https://arxiv.org/abs/2402.14609)
• Keywords: privacy, protect, federate
• Abstract: Complex logical query answering is a challenging task in knowledge graphs (KGs) that has been widely studied. The ability to perform complex logical reasoning is essential and supports various graph reasoning-based downstream tasks, such as search engines. Recent approaches are proposed to represent KG entities and logical queries into embedding vectors and find answers to logical queries from the KGs. However, existing proposed methods mainly focus on querying a single KG and cannot be applied to multiple graphs. In addition, directly sharing KGs with sensitive information may incur privacy risks, making it impractical to share and construct an aggregated KG for reasoning to retrieve query answers. Thus, it remains unknown how to answer queries on multi-source KGs. An entity can be involved in various knowledge graphs and reasoning on multiple KGs and answering complex queries on multi-source KGs is important in discovering knowledge cross graphs. Fortunately, federated learning is utilized in knowledge graphs to collaboratively learn representations with privacy preserved. Federated knowledge graph embeddings enrich the relations in knowledge graphs to improve the representation quality. However, these methods only focus on one-hop relations and cannot perform complex reasoning tasks. In this paper, we apply federated learning to complex query-answering tasks to reason over multi-source knowledge graphs while preserving privacy. We propose a Federated Complex Query Answering framework (FedCQA), to reason over multi-source KGs avoiding sensitive raw data transmission to protect privacy. We conduct extensive experiments on three real-world datasets and evaluate retrieval performance on various types of complex queries.

Title: Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation

• Authors: Jamshid Hassanpour, Vinkle Srivastav, Didier Mutter, Nicolas Padoy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14611
• Pdf URL: https://arxiv.org/pdf/2402.14611
• Copy Paste: [[2402.14611]] Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation(https://arxiv.org/abs/2402.14611)
• Keywords: robust, segmentation
• Abstract: Self-supervised learning (SSL) approaches have achieved great success when the amount of labeled data is limited. Within SSL, models learn robust feature representations by solving pretext tasks. One such pretext task is contrastive learning, which involves forming pairs of similar and dissimilar input samples, guiding the model to distinguish between them. In this work, we investigate the application of contrastive learning to the domain of medical image analysis. Our findings reveal that MoCo v2, a state-of-the-art contrastive learning method, encounters dimensional collapse when applied to medical images. This is attributed to the high degree of inter-image similarity shared between the medical images. To address this, we propose two key contributions: local feature learning and feature decorrelation. Local feature learning improves the ability of the model to focus on the local regions of the image, while feature decorrelation removes the linear dependence among the features. Our experimental findings demonstrate that our contributions significantly enhance the model's performance in the downstream task of medical segmentation, both in the linear evaluation and full fine-tuning settings. This work illustrates the importance of effectively adapting SSL techniques to the characteristics of medical imaging tasks.

Title: Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off

• Authors: Futa Waseda, Isao Echizen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14648
• Pdf URL: https://arxiv.org/pdf/2402.14648
• Copy Paste: [[2402.14648]] Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off(https://arxiv.org/abs/2402.14648)
• Keywords: defense, robust
• Abstract: Although adversarial training has been the state-of-the-art approach to defend against adversarial examples (AEs), they suffer from a robustness-accuracy trade-off. In this work, we revisit representation-based invariance regularization to learn discriminative yet adversarially invariant representations, aiming to mitigate this trade-off. We empirically identify two key issues hindering invariance regularization: (1) a "gradient conflict" between invariance loss and classification objectives, indicating the existence of "collapsing solutions," and (2) the mixture distribution problem arising from diverged distributions of clean and adversarial inputs. To address these issues, we propose Asymmetrically Representation-regularized Adversarial Training (AR-AT), which incorporates a stop-gradient operation and a pre-dictor in the invariance loss to avoid "collapsing solutions," inspired by a recent non-contrastive self-supervised learning approach, and a split-BatchNorm (BN) structure to resolve the mixture distribution problem. Our method significantly improves the robustness-accuracy trade-off by learning adversarially invariant representations without sacrificing discriminative power. Furthermore, we discuss the relevance of our findings to knowledge-distillation-based defense methods, contributing to a deeper understanding of their relative successes.

Title: Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot

• Authors: Fabien Baradel, Matthieu Armando, Salma Galaaoui, Romain Brégier, Philippe Weinzaepfel, Grégory Rogez, Thomas Lucas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14654
• Pdf URL: https://arxiv.org/pdf/2402.14654
• Copy Paste: [[2402.14654]] Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot(https://arxiv.org/abs/2402.14654)
• Keywords: transformer
• Abstract: We present Multi-HMR, a strong single-shot model for multi-person 3D human mesh recovery from a single RGB image. Predictions encompass the whole body, i.e, including hands and facial expressions, using the SMPL-X parametric model and spatial location in the camera coordinate system. Our model detects people by predicting coarse 2D heatmaps of person centers, using features produced by a standard Vision Transformer (ViT) backbone. It then predicts their whole-body pose, shape and spatial location using a new cross-attention module called the Human Prediction Head (HPH), with one query per detected center token, attending to the entire set of features. As direct prediction of SMPL-X parameters yields suboptimal results, we introduce CUFFS; the Close-Up Frames of Full-Body Subjects dataset, containing humans close to the camera with diverse hand poses. We show that incorporating this dataset into training further enhances predictions, particularly for hands, enabling us to achieve state-of-the-art performance. Multi-HMR also optionally accounts for camera intrinsics, if available, by encoding camera ray directions for each image token. This simple design achieves strong performance on whole-body and body-only benchmarks simultaneously. We train models with various backbone sizes and input resolutions. In particular, using a ViT-S backbone and $448\times448$ input images already yields a fast and competitive model with respect to state-of-the-art methods, while considering larger models and higher resolutions further improve performance.

Title: ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models

• Authors: Yanan Wu, Jie Liu, Xingyuan Bu, Jiaheng Liu, Zhanhui Zhou, Yuanxing Zhang, Chenchen Zhang, Zhiqi Bai, Haibin Chen, Tiezheng Ge, Wanli Ouyang, Wenbo Su, Bo Zheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14660
• Pdf URL: https://arxiv.org/pdf/2402.14660
• Copy Paste: [[2402.14660]] ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models(https://arxiv.org/abs/2402.14660)
• Keywords: large language model
• Abstract: This paper introduces ConceptMath, a bilingual (English and Chinese), fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models (LLMs). Unlike traditional benchmarks that evaluate general mathematical reasoning with an average accuracy, ConceptMath systematically organizes math problems under a hierarchy of math concepts, so that mathematical reasoning can be evaluated at different granularity with concept-wise accuracies. Based on our ConcepthMath, we evaluate a broad range of LLMs, and we observe existing LLMs, though achieving high average accuracies on traditional benchmarks, exhibit significant performance variations across different math concepts and may even fail catastrophically on the most basic ones. Besides, we also introduce an efficient fine-tuning strategy to enhance the weaknesses of existing LLMs. Finally, we hope ConceptMath could guide the developers to understand the fine-grained mathematical abilities of their models and facilitate the growth of foundation models.

Title: Quadruplet Loss For Improving the Robustness to Face Morphing Attacks

• Authors: Iurii Medvedev, Nuno Gonçalves
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14665
• Pdf URL: https://arxiv.org/pdf/2402.14665
• Copy Paste: [[2402.14665]] Quadruplet Loss For Improving the Robustness to Face Morphing Attacks(https://arxiv.org/abs/2402.14665)
• Keywords: security, attack, robust, biometric
• Abstract: Recent advancements in deep learning have revolutionized technology and security measures, necessitating robust identification methods. Biometric approaches, leveraging personalized characteristics, offer a promising solution. However, Face Recognition Systems are vulnerable to sophisticated attacks, notably face morphing techniques, enabling the creation of fraudulent documents. In this study, we introduce a novel quadruplet loss function for increasing the robustness of face recognition systems against morphing attacks. Our approach involves specific sampling of face image quadruplets, combined with face morphs, for network training. Experimental results demonstrate the efficiency of our strategy in improving the robustness of face recognition networks against morphing attacks.

Title: Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments

• Authors: Yu Gu, Yiheng Shu, Hao Yu, Xiao Liu, Yuxiao Dong, Jie Tang, Jayanth Srinivasa, Hugo Latapie, Yu Su
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14672
• Pdf URL: https://arxiv.org/pdf/2402.14672
• Copy Paste: [[2402.14672]] Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments(https://arxiv.org/abs/2402.14672)
• Keywords: large language model
• Abstract: The applications of large language models (LLMs) have expanded well beyond the confines of text processing, signaling a new era where LLMs are envisioned as generalist language agents capable of operating within complex real-world environments. These environments are often highly expansive, making it impossible for the LLM to process them within its short-term memory. Motivated by recent research on extending the capabilities of LLMs with tools, this paper investigates the intriguing potential of tools to augment LLMs in handling such complexity. To this end, we design customized tools to aid in the proactive exploration within these massive environments. Such tools can serve as a middleware layer shielding the LLM from environmental complexity. In two representative complex environments -- knowledge bases (KBs) and databases -- we demonstrate the significant potential of augmenting language agents with tools in complex environments. Notably, equipped with these tools, GPT-4 achieves 2.8X the performance of the best baseline in tasks requiring access to database content and 2.2X in KB tasks. Our findings illuminate the path for advancing language agents in complex real-world applications.

Title: Is Cognition and Action Consistent or Not: Investigating Large Language Model's Personality

• Authors: Yiming Ai, Zhiwei He, Ziyin Zhang, Wenhong Zhu, Hongkun Hao, Kai Yu, Lingjun Chen, Rui Wang
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.14679
• Pdf URL: https://arxiv.org/pdf/2402.14679
• Copy Paste: [[2402.14679]] Is Cognition and Action Consistent or Not: Investigating Large Language Model's Personality(https://arxiv.org/abs/2402.14679)
• Keywords: large language model
• Abstract: In this study, we investigate the reliability of Large Language Models (LLMs) in professing human-like personality traits through responses to personality questionnaires. Our goal is to evaluate the consistency between LLMs' professed personality inclinations and their actual "behavior", examining the extent to which these models can emulate human-like personality patterns. Through a comprehensive analysis of LLM outputs against established human benchmarks, we seek to understand the cognition-action divergence in LLMs and propose hypotheses for the observed results based on psychological theories and metrics.

Title: Visual Hallucinations of Multi-modal Large Language Models

• Authors: Wen Huang, Hongbin Liu, Minxin Guo, Neil Zhenqiang Gong
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14683
• Pdf URL: https://arxiv.org/pdf/2402.14683
• Copy Paste: [[2402.14683]] Visual Hallucinations of Multi-modal Large Language Models(https://arxiv.org/abs/2402.14683)
• Keywords: generative, large language model
• Abstract: Visual hallucination (VH) means that a multi-modal LLM (MLLM) imagines incorrect details about an image in visual question answering. Existing studies find VH instances only in existing image datasets, which results in biased understanding of MLLMs' performance under VH due to limited diversity of such VH instances. In this work, we propose a tool called VHTest to generate a diverse set of VH instances. Specifically, VHTest finds some initial VH instances in existing image datasets (e.g., COCO), generates a text description for each VH mode, and uses a text-to-image generative model (e.g., DALL-E-3) to generate VH images based on the text descriptions. We collect a benchmark dataset with 1,200 VH instances in 8 VH modes using VHTest. We find that existing MLLMs such as GPT-4V, LLaVA-1.5, and MiniGPT-v2 hallucinate for a large fraction of the instances in our benchmark. Moreover, we find that fine-tuning an MLLM using our benchmark dataset reduces its likelihood to hallucinate without sacrificing its performance on other benchmarks. Our benchmarks are publicly available: https://github.com/wenhuang2000/VHTest.

Title: UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models

• Authors: Zhaoheng Huang, Zhicheng Dou, Yutao Zhu, Ji-rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14690
• Pdf URL: https://arxiv.org/pdf/2402.14690
• Copy Paste: [[2402.14690]] UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models(https://arxiv.org/abs/2402.14690)
• Keywords: large language model
• Abstract: Large language models (LLMs) may generate text that lacks consistency with human knowledge, leading to factual inaccuracies or \textit{hallucination}. Existing research for evaluating the factuality of LLMs involves extracting fact claims using an LLM and verifying them against a predefined fact source. However, these evaluation metrics are task-specific, and not scalable, and the substitutability of fact sources in different tasks is under-explored. To address these challenges, we categorize four available fact sources: human-written evidence, reference documents, search engine results, and LLM knowledge, along with five text generation tasks containing six representative datasets. Then, we propose \texttt{UFO}, an LLM-based unified and flexible evaluation framework to verify facts against plug-and-play fact sources. We implement five evaluation scenarios based on this framework. Experimental results show that for most QA tasks, human-written evidence and reference documents are crucial, and they can substitute for each other in retrieval-augmented QA tasks. In news fact generation tasks, search engine results and LLM knowledge are essential. Our dataset and code are available at \url{https://github.com/WaldenRUC/UFO}.

Title: QIS : Interactive Segmentation via Quasi-Conformal Mappings

• Authors: Han Zhang, Daoping Zhang, Lok Ming Lui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14695
• Pdf URL: https://arxiv.org/pdf/2402.14695
• Copy Paste: [[2402.14695]] QIS : Interactive Segmentation via Quasi-Conformal Mappings(https://arxiv.org/abs/2402.14695)
• Keywords: segmentation
• Abstract: Image segmentation plays a crucial role in extracting important objects of interest from images, enabling various applications. While existing methods have shown success in segmenting clean images, they often struggle to produce accurate segmentation results when dealing with degraded images, such as those containing noise or occlusions. To address this challenge, interactive segmentation has emerged as a promising approach, allowing users to provide meaningful input to guide the segmentation process. However, an important problem in interactive segmentation lies in determining how to incorporate minimal yet meaningful user guidance into the segmentation model. In this paper, we propose the quasi-conformal interactive segmentation (QIS) model, which incorporates user input in the form of positive and negative clicks. Users mark a few pixels belonging to the object region as positive clicks, indicating that the segmentation model should include a region around these clicks. Conversely, negative clicks are provided on pixels belonging to the background, instructing the model to exclude the region near these clicks from the segmentation mask. Additionally, the segmentation mask is obtained by deforming a template mask with the same topology as the object of interest using an orientation-preserving quasiconformal mapping. This approach helps to avoid topological errors in the segmentation results. We provide a thorough analysis of the proposed model, including theoretical support for the ability of QIS to include or exclude regions of interest or disinterest based on the user's indication. To evaluate the performance of QIS, we conduct experiments on synthesized images, medical images, natural images and noisy natural images. The results demonstrate the efficacy of our proposed method.

Title: Unveiling Linguistic Regions in Large Language Models

• Authors: Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, Xuanjing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14700
• Pdf URL: https://arxiv.org/pdf/2402.14700
• Copy Paste: [[2402.14700]] Unveiling Linguistic Regions in Large Language Models(https://arxiv.org/abs/2402.14700)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated considerable cross-lingual alignment and generalization ability. Current research primarily focuses on improving LLMs' cross-lingual generalization capabilities. However, there is still a lack of research on the intrinsic mechanisms of how LLMs achieve cross-lingual alignment. From the perspective of region partitioning, this paper conducts several investigations on the linguistic competence of LLMs. We discover a core region in LLMs that corresponds to linguistic competence, accounting for approximately 1% of the total model parameters. Removing this core region by setting parameters to zero results in a significant performance decrease across 30 different languages. Furthermore, this core region exhibits significant dimensional dependency, perturbations to even a single parameter on specific dimensions leading to a loss of linguistic competence. Moreover, we discover that distinct regions exist for different monolingual families, and disruption to these specific regions substantially reduces the LLMs' proficiency in those corresponding languages. Our research also indicates that freezing the core linguistic region during further pre-training can mitigate the issue of catastrophic forgetting (CF), a common occurrence observed during further pre-training of LLMs. Overall, exploring the LLMs' functional regions provides insights into the foundation of their intelligence.

Title: COMPASS: Computational Mapping of Patient-Therapist Alliance Strategies with Language Modeling

• Authors: Baihan Lin, Djallel Bouneffouf, Yulia Landa, Rachel Jespersen, Cheryl Corcoran, Guillermo Cecchi
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2402.14701
• Pdf URL: https://arxiv.org/pdf/2402.14701
• Copy Paste: [[2402.14701]] COMPASS: Computational Mapping of Patient-Therapist Alliance Strategies with Language Modeling(https://arxiv.org/abs/2402.14701)
• Keywords: interpretability, generative, large language model
• Abstract: The therapeutic working alliance is a critical factor in predicting the success of psychotherapy treatment. Traditionally, working alliance assessment relies on questionnaires completed by both therapists and patients. In this paper, we present COMPASS, a novel framework to directly infer the therapeutic working alliance from the natural language used in psychotherapy sessions. Our approach utilizes advanced large language models to analyze transcripts of psychotherapy sessions and compare them with distributed representations of statements in the working alliance inventory. Analyzing a dataset of over 950 sessions covering diverse psychiatric conditions, we demonstrate the effectiveness of our method in microscopically mapping patient-therapist alignment trajectories and providing interpretability for clinical psychiatry and in identifying emerging patterns related to the condition being treated. By employing various neural topic modeling techniques in combination with generative language prompting, we analyze the topical characteristics of different psychiatric conditions and incorporate temporal modeling to capture the evolution of topics at a turn-level resolution. This combined framework enhances the understanding of therapeutic interactions, enabling timely feedback for therapists regarding conversation quality and providing interpretable insights to improve the effectiveness of psychotherapy.

Title: InfFeed: Influence Functions as a Feedback to Improve the Performance of Subjective Tasks

• Authors: Somnath Banerjee, Maulindu Sarkar, Punyajoy Saha, Binny Mathew, Animesh Mukherjee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14702
• Pdf URL: https://arxiv.org/pdf/2402.14702
• Copy Paste: [[2402.14702]] InfFeed: Influence Functions as a Feedback to Improve the Performance of Subjective Tasks(https://arxiv.org/abs/2402.14702)
• Keywords: explainability
• Abstract: Recently, influence functions present an apparatus for achieving explainability for deep neural models by quantifying the perturbation of individual train instances that might impact a test prediction. Our objectives in this paper are twofold. First we incorporate influence functions as a feedback into the model to improve its performance. Second, in a dataset extension exercise, using influence functions to automatically identify data points that have been initially `silver' annotated by some existing method and need to be cross-checked (and corrected) by annotators to improve the model performance. To meet these objectives, in this paper, we introduce InfFeed, which uses influence functions to compute the influential instances for a target instance. Toward the first objective, we adjust the label of the target instance based on its influencer(s) label. In doing this, InfFeed outperforms the state-of-the-art baselines (including LLMs) by a maximum macro F1-score margin of almost 4% for hate speech classification, 3.5% for stance classification, and 3% for irony and 2% for sarcasm detection. Toward the second objective we show that manually re-annotating only those silver annotated data points in the extension set that have a negative influence can immensely improve the model performance bringing it very close to the scenario where all the data points in the extension set have gold labels. This allows for huge reduction of the number of data points that need to be manually annotated since out of the silver annotated extension dataset, the influence function scheme picks up ~1/1000 points that need manual correction.

Title: An LLM-Enhanced Adversarial Editing System for Lexical Simplification

• Authors: Keren Tan, Kangyang Luo, Yunshi Lan, Zheng Yuan, Jinlong Shu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14704
• Pdf URL: https://arxiv.org/pdf/2402.14704
• Copy Paste: [[2402.14704]] An LLM-Enhanced Adversarial Editing System for Lexical Simplification(https://arxiv.org/abs/2402.14704)
• Keywords: large language model
• Abstract: Lexical Simplification (LS) aims to simplify text at the lexical level. Existing methods rely heavily on annotated data, making it challenging to apply in low-resource scenarios. In this paper, we propose a novel LS method without parallel corpora. This method employs an Adversarial Editing System with guidance from a confusion loss and an invariance loss to predict lexical edits in the original sentences. Meanwhile, we introduce an innovative LLM-enhanced loss to enable the distillation of knowledge from Large Language Models (LLMs) into a small-size LS system. From that, complex words within sentences are masked and a Difficulty-aware Filling module is crafted to replace masked positions with simpler words. At last, extensive experimental results and analyses on three benchmark LS datasets demonstrate the effectiveness of our proposed method.

Title: Two-stage Cytopathological Image Synthesis for Augmenting Cervical Abnormality Screening

• Authors: Zhenrong Shen, Manman Fei, Xin Wang, Jiangdong Cai, Sheng Wang, Lichi Zhang, Qian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14707
• Pdf URL: https://arxiv.org/pdf/2402.14707
• Copy Paste: [[2402.14707]] Two-stage Cytopathological Image Synthesis for Augmenting Cervical Abnormality Screening(https://arxiv.org/abs/2402.14707)
• Keywords: diffusion
• Abstract: Automatic thin-prep cytologic test (TCT) screening can assist pathologists in finding cervical abnormality towards accurate and efficient cervical cancer diagnosis. Current automatic TCT screening systems mostly involve abnormal cervical cell detection, which generally requires large-scale and diverse training data with high-quality annotations to achieve promising performance. Pathological image synthesis is naturally raised to minimize the efforts in data collection and annotation. However, it is challenging to generate realistic large-size cytopathological images while simultaneously synthesizing visually plausible appearances for small-size abnormal cervical cells. In this paper, we propose a two-stage image synthesis framework to create synthetic data for augmenting cervical abnormality screening. In the first Global Image Generation stage, a Normal Image Generator is designed to generate cytopathological images full of normal cervical cells. In the second Local Cell Editing stage, normal cells are randomly selected from the generated images and then are converted to different types of abnormal cells using the proposed Abnormal Cell Synthesizer. Both Normal Image Generator and Abnormal Cell Synthesizer are built upon the pre-trained Stable Diffusion via parameter-efficient fine-tuning methods for customizing cytopathological image contents and extending spatial layout controllability, respectively. Our experiments demonstrate the synthetic image quality, diversity, and controllability of the proposed synthesis framework, and validate its data augmentation effectiveness in enhancing the performance of abnormal cervical cell detection.

Title: CaT-GNN: Enhancing Credit Card Fraud Detection via Causal Temporal Graph Neural Networks

• Authors: Yifan Duan, Guibin Zhang, Shilong Wang, Xiaojiang Peng, Wang Ziqi, Junyuan Mao, Hao Wu, Xinke Jiang, Kun Wang
• Subjects: cs.LG, cs.AI, q-fin.ST
• Abstract URL: https://arxiv.org/abs/2402.14708
• Pdf URL: https://arxiv.org/pdf/2402.14708
• Copy Paste: [[2402.14708]] CaT-GNN: Enhancing Credit Card Fraud Detection via Causal Temporal Graph Neural Networks(https://arxiv.org/abs/2402.14708)
• Keywords: robust, interpretability
• Abstract: Credit card fraud poses a significant threat to the economy. While Graph Neural Network (GNN)-based fraud detection methods perform well, they often overlook the causal effect of a node's local structure on predictions. This paper introduces a novel method for credit card fraud detection, the \textbf{\underline{Ca}}usal \textbf{\underline{T}}emporal \textbf{\underline{G}}raph \textbf{\underline{N}}eural \textbf{N}etwork (CaT-GNN), which leverages causal invariant learning to reveal inherent correlations within transaction data. By decomposing the problem into discovery and intervention phases, CaT-GNN identifies causal nodes within the transaction graph and applies a causal mixup strategy to enhance the model's robustness and interpretability. CaT-GNN consists of two key components: Causal-Inspector and Causal-Intervener. The Causal-Inspector utilizes attention weights in the temporal attention mechanism to identify causal and environment nodes without introducing additional parameters. Subsequently, the Causal-Intervener performs a causal mixup enhancement on environment nodes based on the set of nodes. Evaluated on three datasets, including a private financial dataset and two public datasets, CaT-GNN demonstrates superior performance over existing state-of-the-art methods. Our findings highlight the potential of integrating causal reasoning with graph neural networks to improve fraud detection capabilities in financial transactions.

Title: IEPile: Unearthing Large-Scale Schema-Based Information Extraction Corpus

• Authors: Honghao Gui, Hongbin Ye, Lin Yuan, Ningyu Zhang, Mengshu Sun, Lei Liang, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.DB, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14710
• Pdf URL: https://arxiv.org/pdf/2402.14710
• Copy Paste: [[2402.14710]] IEPile: Unearthing Large-Scale Schema-Based Information Extraction Corpus(https://arxiv.org/abs/2402.14710)
• Keywords: extraction, large language model
• Abstract: Large Language Models (LLMs) demonstrate remarkable potential across various domains; however, they exhibit a significant performance gap in Information Extraction (IE). Note that high-quality instruction data is the vital key for enhancing the specific capabilities of LLMs, while current IE datasets tend to be small in scale, fragmented, and lack standardized schema. To this end, we introduce IEPile, a comprehensive bilingual (English and Chinese) IE instruction corpus, which contains approximately 0.32B tokens. We construct IEPile by collecting and cleaning 33 existing IE datasets, and introduce schema-based instruction generation to unearth a large-scale corpus. Experimental results on LLaMA and Baichuan demonstrate that using IEPile can enhance the performance of LLMs for IE, especially the zero-shot generalization. We open-source the resource and pre-trained models, hoping to provide valuable support to the NLP community.

Title: Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

• Authors: Seungduk Kim, Seungtaek Choi, Myeongho Jeong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14714
• Pdf URL: https://arxiv.org/pdf/2402.14714
• Copy Paste: [[2402.14714]] Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models(https://arxiv.org/abs/2402.14714)
• Keywords: large language model
• Abstract: This report introduces \texttt{EEVE-Korean-v1.0}, a Korean adaptation of large language models that exhibit remarkable capabilities across English and Korean text understanding. Building on recent highly capable but English-centric LLMs, such as SOLAR-10.7B and Phi-2, where non-English texts are inefficiently processed with English-centric tokenizers, we present an efficient and effective vocabulary expansion (EEVE) method, which encompasses parameter freezing and subword initialization. In contrast to previous efforts that believe new embeddings require trillions of training tokens, we show that our method can significantly boost non-English proficiency within just 2 billion tokens. Surpassing most instruction-tuned LLMs on the Open Ko-LLM Leaderboard, as of January 2024, our model \texttt{EEVE-Korean-10.8B-v1.0} ranks as the leading Korean pre-trained model in the open-source community, according to Hugging Face's leaderboard. We open-source our models on Huggingface to empower the open research community in various languages.

Title: A Transformer Model for Boundary Detection in Continuous Sign Language

• Authors: Razieh Rastgoo, Kourosh Kiani, Sergio Escalera
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14720
• Pdf URL: https://arxiv.org/pdf/2402.14720
• Copy Paste: [[2402.14720]] A Transformer Model for Boundary Detection in Continuous Sign Language(https://arxiv.org/abs/2402.14720)
• Keywords: transformer
• Abstract: Sign Language Recognition (SLR) has garnered significant attention from researchers in recent years, particularly the intricate domain of Continuous Sign Language Recognition (CSLR), which presents heightened complexity compared to Isolated Sign Language Recognition (ISLR). One of the prominent challenges in CSLR pertains to accurately detecting the boundaries of isolated signs within a continuous video stream. Additionally, the reliance on handcrafted features in existing models poses a challenge to achieving optimal accuracy. To surmount these challenges, we propose a novel approach utilizing a Transformer-based model. Unlike traditional models, our approach focuses on enhancing accuracy while eliminating the need for handcrafted features. The Transformer model is employed for both ISLR and CSLR. The training process involves using isolated sign videos, where hand keypoint features extracted from the input video are enriched using the Transformer model. Subsequently, these enriched features are forwarded to the final classification layer. The trained model, coupled with a post-processing method, is then applied to detect isolated sign boundaries within continuous sign videos. The evaluation of our model is conducted on two distinct datasets, including both continuous signs and their corresponding isolated signs, demonstrates promising results.

Title: How Transformers Learn Causal Structure with Gradient Descent

• Authors: Eshaan Nichani, Alex Damian, Jason D. Lee
• Subjects: cs.LG, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.14735
• Pdf URL: https://arxiv.org/pdf/2402.14735
• Copy Paste: [[2402.14735]] How Transformers Learn Causal Structure with Gradient Descent(https://arxiv.org/abs/2402.14735)
• Keywords: transformer
• Abstract: The incredible success of transformers on sequence modeling tasks can be largely attributed to the self-attention mechanism, which allows information to be transferred between different parts of a sequence. Self-attention allows transformers to encode causal structure which makes them particularly suitable for sequence modeling. However, the process by which transformers learn such causal structure via gradient-based training algorithms remains poorly understood. To better understand this process, we introduce an in-context learning task that requires learning latent causal structure. We prove that gradient descent on a simplified two-layer transformer learns to solve this task by encoding the latent causal graph in the first attention layer. The key insight of our proof is that the gradient of the attention matrix encodes the mutual information between tokens. As a consequence of the data processing inequality, the largest entries of this gradient correspond to edges in the latent causal graph. As a special case, when the sequences are generated from in-context Markov chains, we prove that transformers learn an induction head (Olsson et al., 2022). We confirm our theoretical findings by showing that transformers trained on our in-context learning task are able to recover a wide variety of causal structures.

Title: Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

• Authors: Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, Sara Hooker
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14740
• Pdf URL: https://arxiv.org/pdf/2402.14740
• Copy Paste: [[2402.14740]] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs(https://arxiv.org/abs/2402.14740)
• Keywords: large language model
• Abstract: AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models. \textsc{Proximal Policy Optimization} (PPO) has been positioned by recent literature as the canonical method for the RL part of RLHF. However, it involves both high computational cost and sensitive hyperparameter tuning. We posit that most of the motivational principles that led to the development of PPO are less of a practical concern in RLHF and advocate for a less computationally expensive method that preserves and even increases performance. We revisit the \textit{formulation} of alignment from human preferences in the context of RL. Keeping simplicity as a guiding principle, we show that many components of PPO are unnecessary in an RLHF context and that far simpler REINFORCE-style optimization variants outperform both PPO and newly proposed "RL-free" methods such as DPO and RAFT. Our work suggests that careful adaptation to LLMs alignment characteristics enables benefiting from online RL optimization at low cost.

Title: Dependency Annotation of Ottoman Turkish with Multilingual BERT

• Authors: Şaziye Betül Özateş, Tarık Emre Tıraş, Efe Eren Genç, Esma Fatıma Bilgin Taşdemir
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14743
• Pdf URL: https://arxiv.org/pdf/2402.14743
• Copy Paste: [[2402.14743]] Dependency Annotation of Ottoman Turkish with Multilingual BERT(https://arxiv.org/abs/2402.14743)
• Keywords: large language model
• Abstract: This study introduces a pretrained large language model-based annotation methodology for the first dependency treebank in Ottoman Turkish. Our experimental results show that, iteratively, i) pseudo-annotating data using a multilingual BERT-based parsing model, ii) manually correcting the pseudo-annotations, and iii) fine-tuning the parsing model with the corrected annotations, we speed up and simplify the challenging dependency annotation process. The resulting treebank, that will be a part of the Universal Dependencies (UD) project, will facilitate automated analysis of Ottoman Turkish documents, unlocking the linguistic richness embedded in this historical heritage.

Title: Prompting a Pretrained Transformer Can Be a Universal Approximator

• Authors: Aleksandar Petrov, Philip H.S. Torr, Adel Bibi
• Subjects: cs.LG, cs.AI, math.FA
• Abstract URL: https://arxiv.org/abs/2402.14753
• Pdf URL: https://arxiv.org/pdf/2402.14753
• Copy Paste: [[2402.14753]] Prompting a Pretrained Transformer Can Be a Universal Approximator(https://arxiv.org/abs/2402.14753)
• Keywords: transformer
• Abstract: Despite the widespread adoption of prompting, prompt tuning and prefix-tuning of transformer models, our theoretical understanding of these fine-tuning methods remains limited. A key question is whether one can arbitrarily modify the behavior of pretrained model by prompting or prefix-tuning it. Formally, whether prompting and prefix-tuning a pretrained model can universally approximate sequence-to-sequence functions. This paper answers in the affirmative and demonstrates that much smaller pretrained models than previously thought can be universal approximators when prefixed. In fact, the attention mechanism is uniquely suited for universal approximation with prefix-tuning a single attention head being sufficient to approximate any continuous function. Moreover, any sequence-to-sequence function can be approximated by prefixing a transformer with depth linear in the sequence length. Beyond these density-type results, we also offer Jackson-type bounds on the length of the prefix needed to approximate a function to a desired precision.

Title: Generalizing Reward Modeling for Out-of-Distribution Preference Learning

• Authors: Chen Jia
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14760
• Pdf URL: https://arxiv.org/pdf/2402.14760
• Copy Paste: [[2402.14760]] Generalizing Reward Modeling for Out-of-Distribution Preference Learning(https://arxiv.org/abs/2402.14760)
• Keywords: large language model
• Abstract: Preference learning (PL) with large language models (LLMs) aims to align the LLMs' generations with human preferences. Previous work on reinforcement learning from human feedback (RLHF) has demonstrated promising results in in-distribution PL. However, due to the difficulty of obtaining human feedback, discretely training reward models for every encountered distribution is challenging. Thus, out-of-distribution (OOD) PL is practically useful for enhancing the generalization ability of LLMs with limited preference feedback. This work addresses OOD PL by optimizing a general reward model through a meta-learning approach. During meta-training, a bilevel optimization algorithm is utilized to learn a reward model capable of guiding policy learning to align with human preferences across various distributions. When encountering a test distribution, the meta-test procedure conducts regularized policy optimization using the learned reward model for PL. We theoretically demonstrate the convergence rate of the bilevel optimization algorithm under reasonable assumptions. Additionally, we conduct experiments on two text generation tasks across 20 held-out domains and outperform a variety of strong baselines across various evaluation metrics.

Title: MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues

• Authors: Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, Wanli Ouyang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14762
• Pdf URL: https://arxiv.org/pdf/2402.14762
• Copy Paste: [[2402.14762]] MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues(https://arxiv.org/abs/2402.14762)
• Keywords: large language model
• Abstract: The advent of Large Language Models (LLMs) has drastically enhanced dialogue systems. However, comprehensively evaluating the dialogue abilities of LLMs remains a challenge. Previous benchmarks have primarily focused on single-turn dialogues or provided coarse-grained and incomplete assessments of multi-turn dialogues, overlooking the complexity and fine-grained nuances of real-life dialogues. To address this issue, we introduce MT-Bench-101, specifically designed to evaluate the fine-grained abilities of LLMs in multi-turn dialogues. By conducting a detailed analysis of real multi-turn dialogue data, we construct a three-tier hierarchical ability taxonomy comprising 4208 turns across 1388 multi-turn dialogues in 13 distinct tasks. We then evaluate 21 popular LLMs based on MT-Bench-101, conducting comprehensive analyses from both ability and task perspectives and observing differing trends in LLMs performance across dialogue turns within various tasks. Further analysis indicates that neither utilizing common alignment techniques nor chat-specific designs has led to obvious enhancements in the multi-turn abilities of LLMs. Extensive case studies suggest that our designed tasks accurately assess the corresponding multi-turn abilities.

Title: DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models

• Authors: Yuhang Cao, Pan Zhang, Xiaoyi Dong, Dahua Lin, Jiaqi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14767
• Pdf URL: https://arxiv.org/pdf/2402.14767
• Copy Paste: [[2402.14767]] DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models(https://arxiv.org/abs/2402.14767)
• Keywords: large language model
• Abstract: We present DualFocus, a novel framework for integrating macro and micro perspectives within multi-modal large language models (MLLMs) to enhance vision-language task performance. Current MLLMs typically singularly focus on inputs at a predefined resolution, resulting in deficiencies in detailed questions involving local regions. We introduced a DualFocus mechanism where the model concentrates on the image from a macro perspective, responses to the question, and identifies suitable sub-regions to zoom in for subsequent micro perspective analysis. Via the integration of answers from both macro and micro perspectives, the model is adept at addressing tasks that encompass global, detailed, and combined considerations. To endows the DualFocus mechanism in MLLMs, we curated a tailored dataset derived from the Visual Genome (VG) and adapted it to align with the training regimen of DualFocus. Through comparative studies across different model sizes and benchmarks, we demonstrate DualFocus's superiority in balancing detailed examination with holistic insight, significantly reducing hallucination instances in MLLMs and improving their performance in various vision-language tasks.

Title: 2D Matryoshka Sentence Embeddings

• Authors: Xianming Li, Zongxi Li, Jing Li, Haoran Xie, Qing Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14776
• Pdf URL: https://arxiv.org/pdf/2402.14776
• Copy Paste: [[2402.14776]] 2D Matryoshka Sentence Embeddings(https://arxiv.org/abs/2402.14776)
• Keywords: transformer
• Abstract: Common approaches rely on fixed-length embedding vectors from language models as sentence embeddings for downstream tasks such as semantic textual similarity (STS). Such methods are limited in their flexibility due to unknown computational constraints and budgets across various applications. Matryoshka Representation Learning (MRL) (Kusupati et al., 2022) encodes information at finer granularities, i.e., with lower embedding dimensions, to adaptively accommodate ad hoc tasks. Similar accuracy can be achieved with a smaller embedding size, leading to speedups in downstream tasks. Despite its improved efficiency, MRL still requires traversing all Transformer layers before obtaining the embedding, which remains the dominant factor in time and memory consumption. This prompts consideration of whether the fixed number of Transformer layers affects representation quality and whether using intermediate layers for sentence representation is feasible. In this paper, we introduce a novel sentence embedding model called Two-dimensional Matryoshka Sentence Embedding (2DMSE). It supports elastic settings for both embedding sizes and Transformer layers, offering greater flexibility and efficiency than MRL. We conduct extensive experiments on STS tasks and downstream applications. The experimental results demonstrate the effectiveness of our proposed model in dynamically supporting different embedding sizes and Transformer layers, allowing it to be highly adaptable to various scenarios.

Title: Zero-shot cross-lingual transfer in instruction tuning of large language model

• Authors: Nadezhda Chirkova, Vassilina Nikoulina
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14778
• Pdf URL: https://arxiv.org/pdf/2402.14778
• Copy Paste: [[2402.14778]] Zero-shot cross-lingual transfer in instruction tuning of large language model(https://arxiv.org/abs/2402.14778)
• Keywords: large language model
• Abstract: Instruction tuning (IT) is widely used to teach pretrained large language models (LLMs) to follow arbitrary instructions, but is under-studied in multilingual settings. In this work, we conduct a systematic study of zero-shot cross-lingual transfer in IT, when an LLM is instruction-tuned on English-only data and then tested on user prompts in other languages. We investigate the influence of model configuration choices and devise a multi-facet evaluation strategy for multilingual instruction following. We find that cross-lingual transfer does happen successfully in IT even if all stages of model training are English-centric, but only if multiliguality is taken into account in hyperparameter tuning and with large enough IT data. English-trained LLMs are capable of generating correct-language, comprehensive and helpful responses in the other languages, but suffer from low factuality and may occasionally have fluency errors.

Title: Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models

• Authors: Yixuan Ren, Yang Zhou, Jimei Yang, Jing Shi, Difan Liu, Feng Liu, Mingi Kwon, Abhinav Shrivastava
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14780
• Pdf URL: https://arxiv.org/pdf/2402.14780
• Copy Paste: [[2402.14780]] Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models(https://arxiv.org/abs/2402.14780)
• Keywords: diffusion
• Abstract: Image customization has been extensively studied in text-to-image (T2I) diffusion models, leading to impressive outcomes and applications. With the emergence of text-to-video (T2V) diffusion models, its temporal counterpart, motion customization, has not yet been well investigated. To address the challenge of one-shot motion customization, we propose Customize-A-Video that models the motion from a single reference video and adapting it to new subjects and scenes with both spatial and temporal varieties. It leverages low-rank adaptation (LoRA) on temporal attention layers to tailor the pre-trained T2V diffusion model for specific motion modeling from the reference videos. To disentangle the spatial and temporal information during the training pipeline, we introduce a novel concept of appearance absorbers that detach the original appearance from the single reference video prior to motion learning. Our proposed method can be easily extended to various downstream tasks, including custom video generation and editing, video appearance customization, and multiple motion combination, in a plug-and-play fashion. Our project page can be found at https://anonymous-314.github.io.

Title: Consolidating Attention Features for Multi-view Image Editing

• Authors: Or Patashnik, Rinon Gal, Daniel Cohen-Or, Jun-Yan Zhu, Fernando De la Torre
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14792
• Pdf URL: https://arxiv.org/pdf/2402.14792
• Copy Paste: [[2402.14792]] Consolidating Attention Features for Multi-view Image Editing(https://arxiv.org/abs/2402.14792)
• Keywords: diffusion, generative
• Abstract: Large-scale text-to-image models enable a wide range of image editing techniques, using text prompts or even spatial controls. However, applying these editing methods to multi-view images depicting a single scene leads to 3D-inconsistent results. In this work, we focus on spatial control-based geometric manipulations and introduce a method to consolidate the editing process across various views. We build on two insights: (1) maintaining consistent features throughout the generative process helps attain consistency in multi-view editing, and (2) the queries in self-attention layers significantly influence the image structure. Hence, we propose to improve the geometric consistency of the edited images by enforcing the consistency of the queries. To do so, we introduce QNeRF, a neural radiance field trained on the internal query features of the edited images. Once trained, QNeRF can render 3D-consistent queries, which are then softly injected back into the self-attention layers during generation, greatly improving multi-view consistency. We refine the process through a progressive, iterative method that better consolidates queries across the diffusion timesteps. We compare our method to a range of existing techniques and demonstrate that it can achieve better multi-view consistency and higher fidelity to the input scene. These advantages allow us to train NeRFs with fewer visual artifacts, that are better aligned with the target geometry.

Title: Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis

• Authors: Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, Sergey Tulyakov
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14797
• Pdf URL: https://arxiv.org/pdf/2402.14797
• Copy Paste: [[2402.14797]] Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis(https://arxiv.org/abs/2402.14797)
• Keywords: transformer
• Abstract: Contemporary models for generating images show remarkable quality and versatility. Swayed by these advantages, the research community repurposes them to generate videos. Since video content is highly redundant, we argue that naively bringing advances of image models to the video generation domain reduces motion fidelity, visual quality and impairs scalability. In this work, we build Snap Video, a video-first model that systematically addresses these challenges. To do that, we first extend the EDM framework to take into account spatially and temporally redundant pixels and naturally support video generation. Second, we show that a U-Net - a workhorse behind image generation - scales poorly when generating videos, requiring significant computational overhead. Hence, we propose a new transformer-based architecture that trains 3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us to efficiently train a text-to-video model with billions of parameters for the first time, reach state-of-the-art results on a number of benchmarks, and generate videos with substantially higher quality, temporal consistency, and motion complexity. The user studies showed that our model was favored by a large margin over the most recent methods. See our website at https://snap-research.github.io/snapvideo/.

Title: Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

• Authors: Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, Hongsheng Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14800
• Pdf URL: https://arxiv.org/pdf/2402.14800
• Copy Paste: [[2402.14800]] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models(https://arxiv.org/abs/2402.14800)
• Keywords: large language model
• Abstract: A pivotal advancement in the progress of large language models (LLMs) is the emergence of the Mixture-of-Experts (MoE) LLMs. Compared to traditional LLMs, MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. Different from previous weight pruning methods that rely on specifically designed hardware, this paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques. Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks. Extensive experiments show that our proposed methods can simultaneously reduce model sizes and increase the inference speed, while maintaining satisfactory performance. Data and code will be available at https://github.com/Lucky-Lance/Expert_Sparsity.

Title: Identifying Multiple Personalities in Large Language Models with External Evaluation

• Authors: Xiaoyang Song, Yuta Adachi, Jessie Feng, Mouwei Lin, Linhao Yu, Frank Li, Akshat Gupta, Gopala Anumanchipalli, Simerjot Kaur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14805
• Pdf URL: https://arxiv.org/pdf/2402.14805
• Copy Paste: [[2402.14805]] Identifying Multiple Personalities in Large Language Models with External Evaluation(https://arxiv.org/abs/2402.14805)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) are integrated with human daily applications rapidly, many societal and ethical concerns are raised regarding the behavior of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their personalities. Many recent studies quantify LLMs' personalities using self-assessment tests that are created for humans. Yet many critiques question the applicability and reliability of these self-assessment tests when applied to LLMs. In this paper, we investigate LLM personalities using an alternate personality measurement method, which we refer to as the external evaluation method, where instead of prompting LLMs with multiple-choice questions in the Likert scale, we evaluate LLMs' personalities by analyzing their responses toward open-ended situational questions using an external machine learning model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor that outperforms the state-of-the-art models as the tool to analyze LLMs' responses. Then, we prompt the LLMs with situational questions and ask them to generate Twitter posts and comments, respectively, in order to assess their personalities when playing two different roles. Using the external personality evaluation method, we identify that the obtained personality types for LLMs are significantly different when generating posts versus comments, whereas humans show a consistent personality profile in these two different situations. This shows that LLMs can exhibit different personalities based on different scenarios, thus highlighting a fundamental difference between personality in LLMs and humans. With our work, we call for a re-evaluation of personality definition and measurement in LLMs.

Title: RelayAttention for Efficient Large Language Model Serving with Long System Prompts

• Authors: Lei Zhu, Xinjiang Wang, Wayne Zhang, Rynson W.H. Lau
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.14808
• Pdf URL: https://arxiv.org/pdf/2402.14808
• Copy Paste: [[2402.14808]] RelayAttention for Efficient Large Language Model Serving with Long System Prompts(https://arxiv.org/abs/2402.14808)
• Keywords: large language model
• Abstract: Practical large language model (LLM) services may involve a long system prompt, which specifies the instructions, examples, and knowledge documents of the task and is reused across numerous requests. However, the long system prompt causes throughput/latency bottlenecks as the cost of generating the next token grows w.r.t. the sequence length. This paper aims to improve the efficiency of LLM services that involve long system prompts. Our key observation is that handling these system prompts requires heavily redundant memory accesses in existing causal attention computation algorithms. Specifically, for batched requests, the cached hidden states (i.e., key-value pairs) of system prompts are transferred from off-chip DRAM to on-chip SRAM multiple times, each corresponding to an individual request. To eliminate such a redundancy, we propose RelayAttention, an attention algorithm that allows reading these hidden states from DRAM exactly once for a batch of input tokens. RelayAttention is a free lunch: it maintains the generation quality while requiring no model retraining, as it is based on a mathematical reformulation of causal attention.

Title: CriticBench: Benchmarking LLMs for Critique-Correct Reasoning

• Authors: Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, Yujiu Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14809
• Pdf URL: https://arxiv.org/pdf/2402.14809
• Copy Paste: [[2402.14809]] CriticBench: Benchmarking LLMs for Critique-Correct Reasoning(https://arxiv.org/abs/2402.14809)
• Keywords: large language model
• Abstract: The ability of Large Language Models (LLMs) to critique and refine their reasoning is crucial for their application in evaluation, feedback provision, and self-improvement. This paper introduces CriticBench, a comprehensive benchmark designed to assess LLMs' abilities to critique and rectify their reasoning across a variety of tasks. CriticBench encompasses five reasoning domains: mathematical, commonsense, symbolic, coding, and algorithmic. It compiles 15 datasets and incorporates responses from three LLM families. Utilizing CriticBench, we evaluate and dissect the performance of 17 LLMs in generation, critique, and correction reasoning, i.e., GQC reasoning. Our findings reveal: (1) a linear relationship in GQC capabilities, with critique-focused training markedly enhancing performance; (2) a task-dependent variation in correction effectiveness, with logic-oriented tasks being more amenable to correction; (3) GQC knowledge inconsistencies that decrease as model size increases; and (4) an intriguing inter-model critiquing dynamic, where stronger models are better at critiquing weaker ones, while weaker models can surprisingly surpass stronger ones in their self-critique. We hope these insights into the nuanced critique-correct reasoning of LLMs will foster further research in LLM critique and self-improvement.

Title: GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion

• Authors: Xueyi Liu, Li Yi
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14810
• Pdf URL: https://arxiv.org/pdf/2402.14810
• Copy Paste: [[2402.14810]] GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion(https://arxiv.org/abs/2402.14810)
• Keywords: robust, diffusion
• Abstract: In this work, we tackle the challenging problem of denoising hand-object interactions (HOI). Given an erroneous interaction sequence, the objective is to refine the incorrect hand trajectory to remove interaction artifacts for a perceptually realistic sequence. This challenge involves intricate interaction noise, including unnatural hand poses and incorrect hand-object relations, alongside the necessity for robust generalization to new interactions and diverse noise patterns. We tackle those challenges through a novel approach, GeneOH Diffusion, incorporating two key designs: an innovative contact-centric HOI representation named GeneOH and a new domain-generalizable denoising scheme. The contact-centric representation GeneOH informatively parameterizes the HOI process, facilitating enhanced generalization across various HOI scenarios. The new denoising scheme consists of a canonical denoising model trained to project noisy data samples from a whitened noise space to a clean data manifold and a "denoising via diffusion" strategy which can handle input trajectories with various noise patterns by first diffusing them to align with the whitened noise space and cleaning via the canonical denoiser. Extensive experiments on four benchmarks with significant domain variations demonstrate the superior effectiveness of our method. GeneOH Diffusion also shows promise for various downstream applications. Project website: https://meowuu7.github.io/GeneOH-Diffusion/.

Title: WeakSAM: Segment Anything Meets Weakly-supervised Instance-level Recognition

• Authors: Lianghui Zhu, Junwei Zhou, Yan Liu, Xin Hao, Wenyu Liu, Xinggang Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.14812
• Pdf URL: https://arxiv.org/pdf/2402.14812
• Copy Paste: [[2402.14812]] WeakSAM: Segment Anything Meets Weakly-supervised Instance-level Recognition(https://arxiv.org/abs/2402.14812)
• Keywords: segmentation
• Abstract: Weakly supervised visual recognition using inexact supervision is a critical yet challenging learning problem. It significantly reduces human labeling costs and traditionally relies on multi-instance learning and pseudo-labeling. This paper introduces WeakSAM and solves the weakly-supervised object detection (WSOD) and segmentation by utilizing the pre-learned world knowledge contained in a vision foundation model, i.e., the Segment Anything Model (SAM). WeakSAM addresses two critical limitations in traditional WSOD retraining, i.e., pseudo ground truth (PGT) incompleteness and noisy PGT instances, through adaptive PGT generation and Region of Interest (RoI) drop regularization. It also addresses the SAM's problems of requiring prompts and category unawareness for automatic object detection and segmentation. Our results indicate that WeakSAM significantly surpasses previous state-of-the-art methods in WSOD and WSIS benchmarks with large margins, i.e. average improvements of 7.4% and 8.5%, respectively. The code is available at \url{https://github.com/hustvl/WeakSAM}.

Title: Cameras as Rays: Pose Estimation via Ray Diffusion

• Authors: Jason Y. Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, Shubham Tulsiani
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.14817
• Pdf URL: https://arxiv.org/pdf/2402.14817
• Copy Paste: [[2402.14817]] Cameras as Rays: Pose Estimation via Ray Diffusion(https://arxiv.org/abs/2402.14817)
• Keywords: diffusion, transformer
• Abstract: Estimating camera poses is a fundamental task for 3D reconstruction and remains challenging given sparse views (<10). In contrast to existing approaches that pursue top-down prediction of global parametrizations of camera extrinsics, we propose a distributed representation of camera pose that treats a camera as a bundle of rays. This representation allows for a tight coupling with spatial image features improving pose precision. We observe that this representation is naturally suited for set-level level transformers and develop a regression-based approach that maps image patches to corresponding rays. To capture the inherent uncertainties in sparse-view pose inference, we adapt this approach to learn a denoising diffusion model which allows us to sample plausible modes while improving performance. Our proposed methods, both regression- and diffusion-based, demonstrate state-of-the-art performance on camera pose estimation on CO3D while generalizing to unseen object categories and in-the-wild captures.

Title: PALO: A Polyglot Large Multimodal Model for 5B People

• Authors: Muhammad Maaz, Hanoona Rasheed, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M. Anwer, Tim Baldwin, Michael Felsberg, Fahad S. Khan
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.14818
• Pdf URL: https://arxiv.org/pdf/2402.14818
• Copy Paste: [[2402.14818]] PALO: A Polyglot Large Multimodal Model for 5B People(https://arxiv.org/abs/2402.14818)
• Keywords: large language model
• Abstract: In pursuit of more inclusive Vision-Language Models (VLMs), this study introduces a Large Multilingual Multimodal Model called \textsc{Palo}. \textsc{Palo} offers visual reasoning capabilities in 10 major languages, including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese, that span a total of $\sim$5B people (65\% of the world population). Our approach involves a semi-automated translation approach to adapt the multimodal instruction dataset from English to the target languages using a fine-tuned Large Language Model, thereby ensuring high linguistic fidelity while allowing scalability due to minimal manual effort. The incorporation of diverse instruction sets helps us boost overall performance across multiple languages especially those that are underrepresented like Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three scales (1.7B, 7B and 13B parameters) to show the generalization and scalability where we observe substantial improvements compared to strong baselines. We also propose the first multilingual multimodal benchmark for the forthcoming approaches to evaluate their vision-language reasoning capabilities across languages. Code: https://github.com/mbzuai-oryx/PALO.