2024-02-29

Title: Time Series Analysis in Compressor-Based Machines: A Survey

• Authors: Francesca Forbicini, Nicolò Oreste Pinciroli Vago, Piero Fraternali
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17802
• Pdf URL: https://arxiv.org/pdf/2402.17802
• Copy Paste: [[2402.17802]] Time Series Analysis in Compressor-Based Machines: A Survey(https://arxiv.org/abs/2402.17802)
• Keywords: diffusion
• Abstract: In both industrial and residential contexts, compressor-based machines, such as refrigerators, HVAC systems, heat pumps and chillers, are essential to fulfil production and consumers' needs. The diffusion of sensors and IoT connectivity supports the development of monitoring systems able to detect and predict faults, identify behavioural shifts and forecast the operational status of machines and of their components. The focus of this paper is to survey the recent research on such tasks as Fault Detection, Fault Prediction, Forecasting and Change Point Detection applied to multivariate time series characterizing the operations of compressor-based machines. Specifically, Fault Detection detects and diagnoses faults, Fault Prediction predicts such occurrences, forecasting anticipates the future value of characteristic variables of machines and Change Point Detection identifies significant variations in the behaviour of the appliances, such as a change in the working regime. We identify and classify the approaches to the above-mentioned tasks, compare the algorithms employed, highlight the gaps in the current status of the art and discuss the most promising future research directions in the field.

Title: Predicting machine failures from multivariate time series: an industrial case study

• Authors: Nicolò Oreste Pinciroli Vago, Francesca Forbicini, Piero Fraternali
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17804
• Pdf URL: https://arxiv.org/pdf/2402.17804
• Copy Paste: [[2402.17804]] Predicting machine failures from multivariate time series: an industrial case study(https://arxiv.org/abs/2402.17804)
• Keywords: transformer
• Abstract: Non-neural Machine Learning (ML) and Deep Learning (DL) models are often used to predict system failures in the context of industrial maintenance. However, only a few researches jointly assess the effect of varying the amount of past data used to make a prediction and the extension in the future of the forecast. This study evaluates the impact of the size of the reading window and of the prediction window on the performances of models trained to forecast failures in three data sets concerning the operation of (1) an industrial wrapping machine working in discrete sessions, (2) an industrial blood refrigerator working continuously, and (3) a nitrogen generator working continuously. The problem is formulated as a binary classification task that assigns the positive label to the prediction window based on the probability of a failure to occur in such an interval. Six algorithms (logistic regression, random forest, support vector machine, LSTM, ConvLSTM, and Transformers) are compared using multivariate telemetry time series. The results indicate that, in the considered scenarios, the dimension of the prediction windows plays a crucial role and highlight the effectiveness of DL approaches at classifying data with diverse time-dependent patterns preceding a failure and the effectiveness of ML approaches at classifying similar and repetitive patterns preceding a failure.

Title: TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space

• Authors: Shaolei Zhang, Tian Yu, Yang Feng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17811
• Pdf URL: https://arxiv.org/pdf/2402.17811
• Copy Paste: [[2402.17811]] TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space(https://arxiv.org/abs/2402.17811)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks. However, they sometimes suffer from producing hallucinations, particularly in cases where they may generate untruthful responses despite possessing the correct knowledge. In this paper, we propose TruthX, an inference-time method to elicit the truthfulness of LLMs by editing their internal representations in truthful space. TruthX employs an auto-encoder to map LLM's representations into semantic and truthful latent spaces respectively, and applies contrastive learning to identify a truthful editing direction within the truthful space. During inference, by editing LLM's internal representations in truthful space, TruthX effectively enhances the truthfulness of LLMs. Experiments show that TruthX effectively improves the truthfulness of 13 advanced LLMs by an average of 20% on TruthfulQA benchmark. Further analyses suggest that the truthful space acquired by TruthX plays a pivotal role in controlling LLM to produce truthful or hallucinatory responses.

Title: DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation

• Authors: Sunghyeon Woo, Baeseong Park, Byeongwook Kim, Minjung Jo, Sejung Kwon, Dongsuk Jeon, Dongsoo Lee
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17812
• Pdf URL: https://arxiv.org/pdf/2402.17812
• Copy Paste: [[2402.17812]] DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation(https://arxiv.org/abs/2402.17812)
• Keywords: large language model
• Abstract: Training deep neural networks typically involves substantial computational costs during both forward and backward propagation. The conventional layer dropping techniques drop certain layers during training for reducing the computations burden. However, dropping layers during forward propagation adversely affects the training process by degrading accuracy. In this paper, we propose Dropping Backward Propagation (DropBP), a novel approach designed to reduce computational costs while maintaining accuracy. DropBP randomly drops layers during the backward propagation, which does not deviate forward propagation. Moreover, DropBP calculates the sensitivity of each layer to assign appropriate drop rate, thereby stabilizing the training process. DropBP is designed to enhance the efficiency of the training process with backpropagation, thereby enabling the acceleration of both full fine-tuning and parameter-efficient fine-tuning using backpropagation. Specifically, utilizing DropBP in QLoRA reduces training time by 44%, increases the convergence speed to the identical loss level by 1.5$\times$, and enables training with a 6.2$\times$ larger sequence length on a single NVIDIA-A100 80GiB GPU in LLaMA2-70B. The code is available at https://github.com/WooSunghyeon/dropbp.

Title: Prediction-Powered Ranking of Large Language Models

• Authors: Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi, Manuel Gomez Rodriguez
• Subjects: cs.LG, cs.AI, cs.CL, cs.CY, cs.HC, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.17826
• Pdf URL: https://arxiv.org/pdf/2402.17826
• Copy Paste: [[2402.17826]] Prediction-Powered Ranking of Large Language Models(https://arxiv.org/abs/2402.17826)
• Keywords: large language model
• Abstract: Large language models are often ranked according to their level of alignment with human preferences -- a model is better than other models if its outputs are more frequently preferred by humans. One of the most popular ways to elicit human preferences utilizes pairwise comparisons between the outputs provided by different models to the same inputs. However, since gathering pairwise comparisons by humans is costly and time-consuming, it has become a very common practice to gather pairwise comparisons by a strong large language model -- a model strongly aligned with human preferences. Surprisingly, practitioners cannot currently measure the uncertainty that any mismatch between human and model preferences may introduce in the constructed rankings. In this work, we develop a statistical framework to bridge this gap. Given a small set of pairwise comparisons by humans and a large set of pairwise comparisons by a model, our framework provides a rank-set -- a set of possible ranking positions -- for each of the models under comparison. Moreover, it guarantees that, with a probability greater than or equal to a user-specified value, the rank-sets cover the true ranking consistent with (the distribution of) human pairwise preferences. Our framework is computationally efficient, easy to use, and does not make any assumption about the distribution of human preferences nor about the degree of alignment between the pairwise comparisons by the humans and the strong large language model.

Title: Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

• Authors: Zhenting Qi, Hanlin Zhang, Eric Xing, Sham Kakade, Himabindu Lakkaraju
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17840
• Pdf URL: https://arxiv.org/pdf/2402.17840
• Copy Paste: [[2402.17840]] Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems(https://arxiv.org/abs/2402.17840)
• Keywords: attack, extraction
• Abstract: Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation. We study the risk of datastore leakage in Retrieval-In-Context RAG Language Models (LMs). We show that an adversary can exploit LMs' instruction-following capabilities to easily extract text data verbatim from the datastore of RAG systems built with instruction-tuned LMs via prompt injection. The vulnerability exists for a wide range of modern LMs that span Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and the exploitability exacerbates as the model size scales up. Extending our study to production RAG models GPTs, we design an attack that can cause datastore leakage with a 100% success rate on 25 randomly selected customized GPTs with at most 2 queries, and we extract text data verbatim at a rate of 41% from a book of 77,000 words and 3% from a corpus of 1,569,000 words by prompting the GPTs with only 100 queries generated by themselves.

Title: Vision Transformers with Natural Language Semantics

• Authors: Young Kyung Kim, J. Matías Di Martino, Guillermo Sapiro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17863
• Pdf URL: https://arxiv.org/pdf/2402.17863
• Copy Paste: [[2402.17863]] Vision Transformers with Natural Language Semantics(https://arxiv.org/abs/2402.17863)
• Keywords: robust, interpretability, transformer, segmentation
• Abstract: Tokens or patches within Vision Transformers (ViT) lack essential semantic information, unlike their counterparts in natural language processing (NLP). Typically, ViT tokens are associated with rectangular image patches that lack specific semantic context, making interpretation difficult and failing to effectively encapsulate information. We introduce a novel transformer model, Semantic Vision Transformers (sViT), which leverages recent progress on segmentation models to design novel tokenizer strategies. sViT effectively harnesses semantic information, creating an inductive bias reminiscent of convolutional neural networks while capturing global dependencies and contextual information within images that are characteristic of transformers. Through validation using real datasets, sViT demonstrates superiority over ViT, requiring less training data while maintaining similar or superior performance. Furthermore, sViT demonstrates significant superiority in out-of-distribution generalization and robustness to natural distribution shifts, attributed to its scale invariance semantic characteristic. Notably, the use of semantic tokens significantly enhances the model's interpretability. Lastly, the proposed paradigm facilitates the introduction of new and powerful augmentation techniques at the token (or segment) level, increasing training data diversity and generalization capabilities. Just as sentences are made of words, images are formed by semantic objects; our proposed methodology leverages recent progress in object segmentation and takes an important and natural step toward interpretable and robust vision transformers.

Title: Automated Statistical Model Discovery with Language Models

• Authors: Michael Y. Li, Emily B. Fox, Noah D. Goodman
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17879
• Pdf URL: https://arxiv.org/pdf/2402.17879
• Copy Paste: [[2402.17879]] Automated Statistical Model Discovery with Language Models(https://arxiv.org/abs/2402.17879)
• Keywords: large language model
• Abstract: Statistical model discovery involves a challenging search over a vast space of models subject to domain-specific modeling constraints. Efficiently searching over this space requires human expertise in modeling and the problem domain. Motivated by the domain knowledge and programming capabilities of large language models (LMs), we introduce a method for language model driven automated statistical model discovery. We cast our automated procedure within the framework of Box's Loop: the LM iterates between proposing statistical models represented as probabilistic programs, acting as a modeler, and critiquing those models, acting as a domain expert. By leveraging LMs, we do not have to define a domain-specific language of models or design a handcrafted search procedure, key restrictions of previous systems. We evaluate our method in three common settings in probabilistic modeling: searching within a restricted space of models, searching over an open-ended space, and improving classic models under natural language constraints (e.g., this model should be interpretable to an ecologist). Our method matches the performance of previous systems, identifies models on par with human expert designed models, and extends classic models in interpretable ways. Our results highlight the promise of LM driven model discovery.

Title: BlendSQL: A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebra

• Authors: Parker Glenn, Parag Pravin Dakle, Liang Wang, Preethi Raghavan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17882
• Pdf URL: https://arxiv.org/pdf/2402.17882
• Copy Paste: [[2402.17882]] BlendSQL: A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebra(https://arxiv.org/abs/2402.17882)
• Keywords: transformer
• Abstract: Many existing end-to-end systems for hybrid question answering tasks can often be boiled down to a "prompt-and-pray" paradigm, where the user has limited control and insight into the intermediate reasoning steps used to achieve the final result. Additionally, due to the context size limitation of many transformer-based LLMs, it is often not reasonable to expect that the full structured and unstructured context will fit into a given prompt in a zero-shot setting, let alone a few-shot setting. We introduce BlendSQL, a superset of SQLite to act as a unified dialect for orchestrating reasoning across both unstructured and structured data. For hybrid question answering tasks involving multi-hop reasoning, we encode the full decomposed reasoning roadmap into a single interpretable BlendSQL query. Notably, we show that BlendSQL can scale to massive datasets and improve the performance of end-to-end systems while using 35% fewer tokens. Our code is available and installable as a package at https://github.com/parkervg/blendsql.

Title: JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capability

• Authors: Junda Wang, Zhichao Yang, Zonghai Yao, Hong Yu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.17887
• Pdf URL: https://arxiv.org/pdf/2402.17887
• Copy Paste: [[2402.17887]] JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capability(https://arxiv.org/abs/2402.17887)
• Keywords: large language model
• Abstract: With the explosive growth of medical data and the rapid development of artificial intelligence technology, precision medicine has emerged as a key to enhancing the quality and efficiency of healthcare services. In this context, Large Language Models (LLMs) play an increasingly vital role in medical knowledge acquisition and question-answering systems. To further improve the performance of these systems in the medical domain, we introduce an innovative method that jointly trains an Information Retrieval (IR) system and an LLM during the fine-tuning phase. This approach, which we call Joint Medical LLM and Retrieval Training (JMLR), is designed to overcome the challenges faced by traditional models in handling medical question-answering tasks. By employing a synchronized training mechanism, JMLR reduces the demand for computational resources and enhances the model's ability to leverage medical knowledge for reasoning and answering questions. Our experimental results demonstrate that JMLR-13B (81.2% on Amboos, 61.3% on MedQA) outperforms models using conventional pre-training and fine-tuning Meditron-70B (76.4% on AMBOSS, 60.3% on MedQA). For models of the same 7B scale, JMLR-7B(68.7% on Amboos, 51.7% on MedQA) significantly outperforms other public models (Meditron-7B: 50.1%, 47.9%), proving its superiority in terms of cost (our training time: 37 hours, traditional method: 144 hours), efficiency, and effectiveness in medical question-answering tasks. Through this work, we provide a new and efficient knowledge enhancement tool for healthcare, demonstrating the great potential of integrating IR and LLM training in precision medical information retrieval and question-answering systems.

Title: Weakly Supervised Co-training with Swapping Assignments for Semantic Segmentation

• Authors: Xinyu Yang, Hossein Rahmani, Sue Black, Bryan M. Williams
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17891
• Pdf URL: https://arxiv.org/pdf/2402.17891
• Copy Paste: [[2402.17891]] Weakly Supervised Co-training with Swapping Assignments for Semantic Segmentation(https://arxiv.org/abs/2402.17891)
• Keywords: segmentation
• Abstract: Class activation maps (CAMs) are commonly employed in weakly supervised semantic segmentation (WSSS) to produce pseudo-labels. Due to incomplete or excessive class activation, existing studies often resort to offline CAM refinement, introducing additional stages or proposing offline modules. This can cause optimization difficulties for single-stage methods and limit generalizability. In this study, we aim to reduce the observed CAM inconsistency and error to mitigate reliance on refinement processes. We propose an end-to-end WSSS model incorporating guided CAMs, wherein our segmentation model is trained while concurrently optimizing CAMs online. Our method, Co-training with Swapping Assignments (CoSA), leverages a dual-stream framework, where one sub-network learns from the swapped assignments generated by the other. We introduce three techniques: i) soft perplexity-based regularization to penalize uncertain regions; ii) a threshold-searching approach to dynamically revise the confidence threshold; and iii) contrastive separation to address the coexistence problem. CoSA demonstrates exceptional performance, achieving mIoU of 76.2\% and 51.0\% on VOC and COCO validation datasets, respectively, surpassing existing baselines by a substantial margin. Notably, CoSA is the first single-stage approach to outperform all existing multi-stage methods including those with additional supervision. Code is avilable at \url{https://github.com/youshyee/CoSA}.

Title: Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents

• Authors: Corby Rosset, Ho-Lam Chung, Guanghui Qin, Ethan C. Chau, Zhuo Feng, Ahmed Awadallah, Jennifer Neville, Nikhil Rao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17896
• Pdf URL: https://arxiv.org/pdf/2402.17896
• Copy Paste: [[2402.17896]] Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents(https://arxiv.org/abs/2402.17896)
• Keywords: security, large language model
• Abstract: Existing question answering (QA) datasets are no longer challenging to most powerful Large Language Models (LLMs). Traditional QA benchmarks like TriviaQA, NaturalQuestions, ELI5 and HotpotQA mainly study ``known unknowns'' with clear indications of both what information is missing, and how to find it to answer the question. Hence, good performance on these benchmarks provides a false sense of security. A yet unmet need of the NLP community is a bank of non-factoid, multi-perspective questions involving a great deal of unclear information needs, i.e. ``unknown uknowns''. We claim we can find such questions in search engine logs, which is surprising because most question-intent queries are indeed factoid. We present Researchy Questions, a dataset of search engine queries tediously filtered to be non-factoid, ``decompositional'' and multi-perspective. We show that users spend a lot of ``effort'' on these questions in terms of signals like clicks and session length, and that they are also challenging for GPT-4. We also show that ``slow thinking'' answering techniques, like decomposition into sub-questions shows benefit over answering directly. We release $\sim$ 100k Researchy Questions, along with the Clueweb22 URLs that were clicked.

Title: A Language Model based Framework for New Concept Placement in Ontologies

• Authors: Hang Dong, Jiaoyan Chen, Yuan He, Yongsheng Gao, Ian Horrocks
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.17897
• Pdf URL: https://arxiv.org/pdf/2402.17897
• Copy Paste: [[2402.17897]] A Language Model based Framework for New Concept Placement in Ontologies(https://arxiv.org/abs/2402.17897)
• Keywords: large language model
• Abstract: We investigate the task of inserting new concepts extracted from texts into an ontology using language models. We explore an approach with three steps: edge search which is to find a set of candidate locations to insert (i.e., subsumptions between concepts), edge formation and enrichment which leverages the ontological structure to produce and enhance the edge candidates, and edge selection which eventually locates the edge to be placed into. In all steps, we propose to leverage neural methods, where we apply embedding-based methods and contrastive learning with Pre-trained Language Models (PLMs) such as BERT for edge search, and adapt a BERT fine-tuning-based multi-label Edge-Cross-encoder, and Large Language Models (LLMs) such as GPT series, FLAN-T5, and Llama 2, for edge selection. We evaluate the methods on recent datasets created using the SNOMED CT ontology and the MedMentions entity linking benchmark. The best settings in our framework use fine-tuned PLM for search and a multi-label Cross-encoder for selection. Zero-shot prompting of LLMs is still not adequate for the task, and we proposed explainable instruction tuning of LLMs for improved performance. Our study shows the advantages of PLMs and highlights the encouraging performance of LLMs that motivates future studies.

Title: Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Models

• Authors: Ashkan Taghipour, Morteza Ghahremani, Mohammed Bennamoun, Aref Miri Rekavandi, Hamid Laga, Farid Boussaid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17910
• Pdf URL: https://arxiv.org/pdf/2402.17910
• Copy Paste: [[2402.17910]] Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Models(https://arxiv.org/abs/2402.17910)
• Keywords: diffusion
• Abstract: While latent diffusion models (LDMs) excel at creating imaginative images, they often lack precision in semantic fidelity and spatial control over where objects are generated. To address these deficiencies, we introduce the Box-it-to-Bind-it (B2B) module - a novel, training-free approach for improving spatial control and semantic accuracy in text-to-image (T2I) diffusion models. B2B targets three key challenges in T2I: catastrophic neglect, attribute binding, and layout guidance. The process encompasses two main steps: i) Object generation, which adjusts the latent encoding to guarantee object generation and directs it within specified bounding boxes, and ii) attribute binding, guaranteeing that generated objects adhere to their specified attributes in the prompt. B2B is designed as a compatible plug-and-play module for existing T2I models, markedly enhancing model performance in addressing the key challenges. We evaluate our technique using the established CompBench and TIFA score benchmarks, demonstrating significant performance improvements compared to existing methods. The source code will be made publicly available at https://github.com/nextaistudio/BoxIt2BindIt.

Title: Extracting Lexical Features from Dialects via Interpretable Dialect Classifiers

• Authors: Roy Xie, Orevaoghene Ahia, Yulia Tsvetkov, Antonios Anastasopoulos
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17914
• Pdf URL: https://arxiv.org/pdf/2402.17914
• Copy Paste: [[2402.17914]] Extracting Lexical Features from Dialects via Interpretable Dialect Classifiers(https://arxiv.org/abs/2402.17914)
• Keywords: interpretability
• Abstract: Identifying linguistic differences between dialects of a language often requires expert knowledge and meticulous human analysis. This is largely due to the complexity and nuance involved in studying various dialects. We present a novel approach to extract distinguishing lexical features of dialects by utilizing interpretable dialect classifiers, even in the absence of human experts. We explore both post-hoc and intrinsic approaches to interpretability, conduct experiments on Mandarin, Italian, and Low Saxon, and experimentally demonstrate that our method successfully identifies key language-specific lexical features that contribute to dialectal variations.

Title: LLM-Resistant Math Word Problem Generation via Adversarial Attacks

• Authors: Roy Xie, Chengxuan Huang, Junlin Wang, Bhuwan Dhingra
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17916
• Pdf URL: https://arxiv.org/pdf/2402.17916
• Copy Paste: [[2402.17916]] LLM-Resistant Math Word Problem Generation via Adversarial Attacks(https://arxiv.org/abs/2402.17916)
• Keywords: attack, fair, large language model
• Abstract: Large language models (LLMs) have significantly transformed the educational landscape. As current plagiarism detection tools struggle to keep pace with LLMs' rapid advancements, the educational community faces the challenge of assessing students' true problem-solving abilities in the presence of LLMs. In this work, we explore a new paradigm for ensuring fair evaluation -- generating adversarial examples which preserve the structure and difficulty of the original questions aimed for assessment, but are unsolvable by LLMs. Focusing on the domain of math word problems, we leverage abstract syntax trees to structurally generate adversarial examples that cause LLMs to produce incorrect answers by simply editing the numeric values in the problems. We conduct experiments on various open- and closed-source LLMs, quantitatively and qualitatively demonstrating that our method significantly degrades their math problem-solving ability. We identify shared vulnerabilities among LLMs and propose a cost-effective approach to attack high-cost models. Additionally, we conduct automatic analysis on math problems and investigate the cause of failure to guide future research on LLM's mathematical capability.

Title: The Seeker's Dilemma: Realistic Formulation and Benchmarking for Hardware Trojan Detection

• Authors: Amin Sarihi, Ahmad Patooghy, Abdel-Hameed A. Badawy, Peter Jamieson
• Subjects: cs.CR, cs.AR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17918
• Pdf URL: https://arxiv.org/pdf/2402.17918
• Copy Paste: [[2402.17918]] The Seeker's Dilemma: Realistic Formulation and Benchmarking for Hardware Trojan Detection(https://arxiv.org/abs/2402.17918)
• Keywords: security
• Abstract: This work focuses on advancing security research in the hardware design space by formally defining the realistic problem of Hardware Trojan (HT) detection. The goal is to model HT detection more closely to the real world, i.e., describing the problem as "The Seeker's Dilemma" (an extension of Hide&Seek on a graph), where a detecting agent is unaware of whether circuits are infected by HTs or not. Using this theoretical problem formulation, we create a benchmark that consists of a mixture of HT-free and HT-infected restructured circuits while preserving their original functionalities. The restructured circuits are randomly infected by HTs, causing a situation where the defender is uncertain if a circuit is infected or not. We believe that our innovative dataset will help the community better judge the detection quality of different methods by comparing their success rates in circuit classification. We use our developed benchmark to evaluate three state-of-the-art HT detection tools to show baseline results for this approach. We use Principal Component Analysis to assess the strength of our benchmark, where we observe that some restructured HT-infected circuits are mapped closely to HT-free circuits, leading to significant label misclassification by detectors.

Title: Multitask Multilingual Model Adaptation with Featurized Low-Rank Mixtures

• Authors: Chu-Cheng Lin, Xinyi Wang, Jonathan H. Clark, Han Lu, Yun Zhu, Chenxi Whitehouse, Hongkun Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17934
• Pdf URL: https://arxiv.org/pdf/2402.17934
• Copy Paste: [[2402.17934]] Multitask Multilingual Model Adaptation with Featurized Low-Rank Mixtures(https://arxiv.org/abs/2402.17934)
• Keywords: large language model
• Abstract: Adapting pretrained large language models (LLMs) to various downstream tasks in tens or hundreds of human languages is computationally expensive. Parameter-efficient fine-tuning (PEFT) significantly reduces the adaptation cost, by tuning only a small amount of parameters. However, directly applying PEFT methods such as LoRA (Hu et al., 2022) on diverse dataset mixtures could lead to suboptimal performance due to limited parameter capacity and negative interference among different datasets. In this work, we propose Featurized Low-rank Mixtures (FLix), a novel PEFT method designed for effective multitask multilingual tuning. FLix associates each unique dataset feature, such as the dataset's language or task, with its own low-rank weight update parameters. By composing feature-specific parameters for each dataset, FLix can accommodate diverse dataset mixtures and generalize better to unseen datasets. Our experiments show that FLix leads to significant improvements over a variety of tasks for both supervised learning and zero-shot settings using different training data mixtures.

Title: EmMark: Robust Watermarks for IP Protection of Embedded Quantized Large Language Models

• Authors: Ruisi Zhang, Farinaz Koushanfar
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17938
• Pdf URL: https://arxiv.org/pdf/2402.17938
• Copy Paste: [[2402.17938]] EmMark: Robust Watermarks for IP Protection of Embedded Quantized Large Language Models(https://arxiv.org/abs/2402.17938)
• Keywords: protect, attack, robust, extraction, watermark, large language model
• Abstract: This paper introduces EmMark,a novel watermarking framework for protecting the intellectual property (IP) of embedded large language models deployed on resource-constrained edge devices. To address the IP theft risks posed by malicious end-users, EmMark enables proprietors to authenticate ownership by querying the watermarked model weights and matching the inserted signatures. EmMark's novelty lies in its strategic watermark weight parameters selection, nsuring robustness and maintaining model quality. Extensive proof-of-concept evaluations of models from OPT and LLaMA-2 families demonstrate EmMark's fidelity, achieving 100% success in watermark extraction with model performance preservation. EmMark also showcased its resilience against watermark removal and forging attacks.

Title: Large Language Models on Tabular Data -- A Survey

• Authors: Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, Christos Faloutsos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17944
• Pdf URL: https://arxiv.org/pdf/2402.17944
• Copy Paste: [[2402.17944]] Large Language Models on Tabular Data -- A Survey(https://arxiv.org/abs/2402.17944)
• Keywords: large language model
• Abstract: Recent breakthroughs in large language modeling have facilitated rigorous exploration of their application in diverse tasks related to tabular data modeling, such as prediction, tabular data synthesis, question answering, and table understanding. Each task presents unique challenges and opportunities. However, there is currently a lack of comprehensive review that summarizes and compares the key techniques, metrics, datasets, models, and optimization approaches in this research domain. This survey aims to address this gap by consolidating recent progress in these areas, offering a thorough survey and taxonomy of the datasets, metrics, and methodologies utilized. It identifies strengths, limitations, unexplored territories, and gaps in the existing literature, while providing some insights for future research directions in this vital and rapidly evolving field. It also provides relevant code and datasets references. Through this comprehensive review, we hope to provide interested readers with pertinent references and insightful perspectives, empowering them with the necessary tools and knowledge to effectively navigate and address the prevailing challenges in the field.

Title: Gradient-Free Adaptive Global Pruning for Pre-trained Language Models

• Authors: Guangji Bai, Yijiang Li, Chen Ling, Kibaek Kim, Liang Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17946
• Pdf URL: https://arxiv.org/pdf/2402.17946
• Copy Paste: [[2402.17946]] Gradient-Free Adaptive Global Pruning for Pre-trained Language Models(https://arxiv.org/abs/2402.17946)
• Keywords: large language model
• Abstract: The transformative impact of large language models (LLMs) like LLaMA and GPT on natural language processing is countered by their prohibitive computational demands. Pruning has emerged as a pivotal compression strategy, introducing sparsity to enhance both memory and computational efficiency. Yet, traditional global pruning is impractical for LLMs due to scalability issues, while local pruning, despite its efficiency, leads to suboptimal solutions. Addressing these challenges, we propose Adaptive Global Pruning (AdaGP), a novel framework that redefines the global pruning process into manageable, coordinated subproblems, allowing for resource-efficient optimization with global optimality. AdaGP's approach, which conceptualizes LLMs as a chain of modular functions and leverages auxiliary variables for problem decomposition, not only facilitates a pragmatic application on LLMs but also demonstrates significant performance improvements, particularly in high-sparsity regimes where it surpasses current state-of-the-art methods.

Title: Rapid hyperspectral photothermal mid-infrared spectroscopic imaging from sparse data for gynecologic cancer tissue subtyping

• Authors: Reza Reihanisaransari, Chalapathi Charan Gajjela, Xinyu Wu, Ragib Ishrak, Sara Corvigno, Yanping Zhong, Jinsong Liu, Anil K. Sood, David Mayerich, Sebastian Berisha, Rohith Reddy
• Subjects: cs.CV, q-bio.BM, q-bio.QM, q-bio.TO
• Abstract URL: https://arxiv.org/abs/2402.17960
• Pdf URL: https://arxiv.org/pdf/2402.17960
• Copy Paste: [[2402.17960]] Rapid hyperspectral photothermal mid-infrared spectroscopic imaging from sparse data for gynecologic cancer tissue subtyping(https://arxiv.org/abs/2402.17960)
• Keywords: robust, segmentation
• Abstract: Ovarian cancer detection has traditionally relied on a multi-step process that includes biopsy, tissue staining, and morphological analysis by experienced pathologists. While widely practiced, this conventional approach suffers from several drawbacks: it is qualitative, time-intensive, and heavily dependent on the quality of staining. Mid-infrared (MIR) hyperspectral photothermal imaging is a label-free, biochemically quantitative technology that, when combined with machine learning algorithms, can eliminate the need for staining and provide quantitative results comparable to traditional histology. However, this technology is slow. This work presents a novel approach to MIR photothermal imaging that enhances its speed by an order of magnitude. Our method significantly accelerates data collection by capturing a combination of high-resolution and interleaved, lower-resolution infrared band images and applying computational techniques for data interpolation. We effectively minimize data collection requirements by leveraging sparse data acquisition and employing curvelet-based reconstruction algorithms. This method enables the reconstruction of high-quality, high-resolution images from undersampled datasets and achieving a 10X improvement in data acquisition time. We assessed the performance of our sparse imaging methodology using a variety of quantitative metrics, including mean squared error (MSE), structural similarity index (SSIM), and tissue subtype classification accuracies, employing both random forest and convolutional neural network (CNN) models, accompanied by ROC curves. Our statistically robust analysis, based on data from 100 ovarian cancer patient samples and over 65 million data points, demonstrates the method's capability to produce superior image quality and accurately distinguish between different gynecological tissue types with segmentation accuracy exceeding 95%.

Title: Conformer: Embedding Continuous Attention in Vision Transformer for Weather Forecasting

• Authors: Hira Saleem, Flora Salim, Cormac Purcell
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17966
• Pdf URL: https://arxiv.org/pdf/2402.17966
• Copy Paste: [[2402.17966]] Conformer: Embedding Continuous Attention in Vision Transformer for Weather Forecasting(https://arxiv.org/abs/2402.17966)
• Keywords: transformer
• Abstract: Operational weather forecasting system relies on computationally expensive physics-based models. Although Transformers-based models have shown remarkable potential in weather forecasting, Transformers are discrete models which limit their ability to learn the continuous spatio-temporal features of the dynamical weather system. We address this issue with Conformer, a spatio-temporal Continuous Vision Transformer for weather forecasting. Conformer is designed to learn the continuous weather evolution over time by implementing continuity in the multi-head attention mechanism. The attention mechanism is encoded as a differentiable function in the transformer architecture to model the complex weather dynamics. We evaluate Conformer against a state-of-the-art Numerical Weather Prediction (NWP) model and several deep learning based weather forecasting models. Conformer outperforms some of the existing data-driven models at all lead times while only being trained at lower resolution data.

Title: Imitation-regularized Optimal Transport on Networks: Provable Robustness and Application to Logistics Planning

• Authors: Koshi Oishi, Yota Hashizume, Tomohiko Jimbo, Hirotaka Kaji, Kenji Kashima
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.17967
• Pdf URL: https://arxiv.org/pdf/2402.17967
• Copy Paste: [[2402.17967]] Imitation-regularized Optimal Transport on Networks: Provable Robustness and Application to Logistics Planning(https://arxiv.org/abs/2402.17967)
• Keywords: robust
• Abstract: Network systems form the foundation of modern society, playing a critical role in various applications. However, these systems are at significant risk of being adversely affected by unforeseen circumstances, such as disasters. Considering this, there is a pressing need for research to enhance the robustness of network systems. Recently, in reinforcement learning, the relationship between acquiring robustness and regularizing entropy has been identified. Additionally, imitation learning is used within this framework to reflect experts' behavior. However, there are no comprehensive studies on the use of a similar imitation framework for optimal transport on networks. Therefore, in this study, imitation-regularized optimal transport (I-OT) on networks was investigated. It encodes prior knowledge on the network by imitating a given prior distribution. The I-OT solution demonstrated robustness in terms of the cost defined on the network. Moreover, we applied the I-OT to a logistics planning problem using real data. We also examined the imitation and apriori risk information scenarios to demonstrate the usefulness and implications of the proposed method.

Title: Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction

• Authors: Koki Maeda, Shuhei Kurita, Taiki Miyanishi, Naoaki Okazaki
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17969
• Pdf URL: https://arxiv.org/pdf/2402.17969
• Copy Paste: [[2402.17969]] Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction(https://arxiv.org/abs/2402.17969)
• Keywords: extraction
• Abstract: Given the accelerating progress of vision and language modeling, accurate evaluation of machine-generated image captions remains critical. In order to evaluate captions more closely to human preferences, metrics need to discriminate between captions of varying quality and content. However, conventional metrics fail short of comparing beyond superficial matches of words or embedding similarities; thus, they still need improvement. This paper presents VisCE$^2$, a vision language model-based caption evaluation method. Our method focuses on visual context, which refers to the detailed content of images, including objects, attributes, and relationships. By extracting and organizing them into a structured format, we replace the human-written references with visual contexts and help VLMs better understand the image, enhancing evaluation performance. Through meta-evaluation on multiple datasets, we validated that VisCE$^2$ outperforms the conventional pre-trained metrics in capturing caption quality and demonstrates superior consistency with human judgment.

Title: Exploring Advanced Methodologies in Security Evaluation for LLMs

• Authors: Jun Huang, Jiawei Zhang, Qi Wang, Weihong Han, Yanchun Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.17970
• Pdf URL: https://arxiv.org/pdf/2402.17970
• Copy Paste: [[2402.17970]] Exploring Advanced Methodologies in Security Evaluation for LLMs(https://arxiv.org/abs/2402.17970)
• Keywords: security, large language model
• Abstract: Large Language Models (LLMs) represent an advanced evolution of earlier, simpler language models. They boast enhanced abilities to handle complex language patterns and generate coherent text, images, audios, and videos. Furthermore, they can be fine-tuned for specific tasks. This versatility has led to the proliferation and extensive use of numerous commercialized large models. However, the rapid expansion of LLMs has raised security and ethical concerns within the academic community. This emphasizes the need for ongoing research into security evaluation during their development and deployment. Over the past few years, a substantial body of research has been dedicated to the security evaluation of large-scale models. This article an in-depth review of the most recent advancements in this field, providing a comprehensive analysis of commonly used evaluation metrics, advanced evaluation frameworks, and the routine evaluation processes for LLMs. Furthermore, we also discuss the future directions for advancing the security evaluation of LLMs.

Title: From Generalization to Precision: Exploring SAM for Tool Segmentation in Surgical Environments

• Authors: Kanyifeechukwu J. Oguine, Roger D. Soberanis-Mukul, Nathan Drenkow, Mathias Unberath
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17972
• Pdf URL: https://arxiv.org/pdf/2402.17972
• Copy Paste: [[2402.17972]] From Generalization to Precision: Exploring SAM for Tool Segmentation in Surgical Environments(https://arxiv.org/abs/2402.17972)
• Keywords: robust, segmentation
• Abstract: Purpose: Accurate tool segmentation is essential in computer-aided procedures. However, this task conveys challenges due to artifacts' presence and the limited training data in medical scenarios. Methods that generalize to unseen data represent an interesting venue, where zero-shot segmentation presents an option to account for data limitation. Initial exploratory works with the Segment Anything Model (SAM) show that bounding-box-based prompting presents notable zero-short generalization. However, point-based prompting leads to a degraded performance that further deteriorates under image corruption. We argue that SAM drastically over-segment images with high corruption levels, resulting in degraded performance when only a single segmentation mask is considered, while the combination of the masks overlapping the object of interest generates an accurate prediction. Method: We use SAM to generate the over-segmented prediction of endoscopic frames. Then, we employ the ground-truth tool mask to analyze the results of SAM when the best single mask is selected as prediction and when all the individual masks overlapping the object of interest are combined to obtain the final predicted mask. We analyze the Endovis18 and Endovis17 instrument segmentation datasets using synthetic corruptions of various strengths and an In-House dataset featuring counterfactually created real-world corruptions. Results: Combining the over-segmented masks contributes to improvements in the IoU. Furthermore, selecting the best single segmentation presents a competitive IoU score for clean images. Conclusions: Combined SAM predictions present improved results and robustness up to a certain corruption level. However, appropriate prompting strategies are fundamental for implementing these models in the medical domain.

Title: Enhancing Tracking Robustness with Auxiliary Adversarial Defense Networks

• Authors: Zhewei Wu, Ruilong Yu, Qihe Liu, Shuying Cheng, Shilin Qiu, Shijie Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17976
• Pdf URL: https://arxiv.org/pdf/2402.17976
• Copy Paste: [[2402.17976]] Enhancing Tracking Robustness with Auxiliary Adversarial Defense Networks(https://arxiv.org/abs/2402.17976)
• Keywords: defense, attack, robust
• Abstract: Adversarial attacks in visual object tracking have significantly degraded the performance of advanced trackers by introducing imperceptible perturbations into images. These attack methods have garnered considerable attention from researchers in recent years. However, there is still a lack of research on designing adversarial defense methods specifically for visual object tracking. To address these issues, we propose an effective additional pre-processing network called DuaLossDef that eliminates adversarial perturbations during the tracking process. DuaLossDef is deployed ahead of the search branche or template branche of the tracker to apply defensive transformations to the input images. Moreover, it can be seamlessly integrated with other visual trackers as a plug-and-play module without requiring any parameter adjustments. We train DuaLossDef using adversarial training, specifically employing Dua-Loss to generate adversarial samples that simultaneously attack the classification and regression branches of the tracker. Extensive experiments conducted on the OTB100, LaSOT, and VOT2018 benchmarks demonstrate that DuaLossDef maintains excellent defense robustness against adversarial attack methods in both adaptive and non-adaptive attack scenarios. Moreover, when transferring the defense network to other trackers, it exhibits reliable transferability. Finally, DuaLossDef achieves a processing time of up to 5ms/frame, allowing seamless integration with existing high-speed trackers without introducing significant computational overhead. We will make our code publicly available soon.

Title: Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning

• Authors: Zeyang Liu, Lipeng Wan, Xinrui Yang, Zhuoran Chen, Xingyu Chen, Xuguang Lan
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2402.17978
• Pdf URL: https://arxiv.org/pdf/2402.17978
• Copy Paste: [[2402.17978]] Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning(https://arxiv.org/abs/2402.17978)
• Keywords: diffusion, transformer, generative
• Abstract: Effective exploration is crucial to discovering optimal strategies for multi-agent reinforcement learning (MARL) in complex coordination tasks. Existing methods mainly utilize intrinsic rewards to enable committed exploration or use role-based learning for decomposing joint action spaces instead of directly conducting a collective search in the entire action-observation space. However, they often face challenges obtaining specific joint action sequences to reach successful states in long-horizon tasks. To address this limitation, we propose Imagine, Initialize, and Explore (IIE), a novel method that offers a promising solution for efficient multi-agent exploration in complex scenarios. IIE employs a transformer model to imagine how the agents reach a critical state that can influence each other's transition functions. Then, we initialize the environment at this state using a simulator before the exploration phase. We formulate the imagination as a sequence modeling problem, where the states, observations, prompts, actions, and rewards are predicted autoregressively. The prompt consists of timestep-to-go, return-to-go, influence value, and one-shot demonstration, specifying the desired state and trajectory as well as guiding the action generation. By initializing agents at the critical states, IIE significantly increases the likelihood of discovering potentially important under-explored regions. Despite its simplicity, empirical results demonstrate that our method outperforms multi-agent exploration baselines on the StarCraft Multi-Agent Challenge (SMAC) and SMACv2 environments. Particularly, IIE shows improved performance in the sparse-reward SMAC tasks and produces more effective curricula over the initialized states than other generative methods, such as CVAE-GAN and diffusion models.

Title: Collaborative decoding of critical tokens for boosting factuality of large language models

• Authors: Lifeng Jin, Baolin Peng, Linfeng Song, Haitao Mi, Ye Tian, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17982
• Pdf URL: https://arxiv.org/pdf/2402.17982
• Copy Paste: [[2402.17982]] Collaborative decoding of critical tokens for boosting factuality of large language models(https://arxiv.org/abs/2402.17982)
• Keywords: large language model
• Abstract: The most common training pipeline for large language models includes pretraining, finetuning and aligning phases, with their respective resulting models, such as the pretrained model and the finetuned model. Finetuned and aligned models show improved abilities of instruction following and safe generation, however their abilities to stay factual about the world are impacted by the finetuning process. Furthermore, the common practice of using sampling during generation also increases chances of hallucination. In this work, we introduce a collaborative decoding framework to harness the high factuality within pretrained models through the concept of critical tokens. We first design a critical token classifier to decide which model to use for the next token, and subsequently generates the next token using different decoding strategies. Experiments with different models and datasets show that our decoding framework is able to reduce model hallucination significantly, showcasing the importance of the collaborative decoding framework.

Title: FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization

• Authors: Yi Zhang, Fei Yang, Shuang Peng, Fangyu Wang, Aimin Pan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17985
• Pdf URL: https://arxiv.org/pdf/2402.17985
• Copy Paste: [[2402.17985]] FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization(https://arxiv.org/abs/2402.17985)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated state-of-the-art performance across various tasks. However, the latency of inference and the large GPU memory consumption of LLMs restrict their deployment performance. Recently, there have been some efficient attempts to quantize LLMs, yet inference with large batch size or long sequence still has the issue of being compute-bound. Fine-grained quantization methods have showcased their proficiency in achieving low-bit quantization for LLMs, while requiring FP16 data type for linear layer computations, which is time-consuming when dealing with large batch size or long sequence. In this paper, we introduce a method called FlattenQuant, which significantly reduces the maximum value of the tensor by flattening the large channels in the tensor, to achieve low bit per-tensor quantization with minimal accuracy loss. Our experiments show that FlattenQuant can directly use 4 bits to achieve 48.29% of the linear layer calculation in LLMs, with the remaining layers using 8 bits. The 4-bit matrix multiplication introduced in the FlattenQuant method can effectively address the compute-bound caused by large matrix calculation. Our work achieves up to 2$\times$ speedup and 2.3$\times$ memory reduction for LLMs with negligible loss in accuracy.

Title: PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesis

• Authors: Jason J. Yu, Tristan Aumentado-Armstrong, Fereshteh Forghani, Konstantinos G. Derpanis, Marcus A. Brubaker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17986
• Pdf URL: https://arxiv.org/pdf/2402.17986
• Copy Paste: [[2402.17986]] PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesis(https://arxiv.org/abs/2402.17986)
• Keywords: generative
• Abstract: This paper considers the problem of generative novel view synthesis (GNVS), generating novel, plausible views of a scene given a limited number of known views. Here, we propose a set-based generative model that can simultaneously generate multiple, self-consistent new views, conditioned on any number of known views. Our approach is not limited to generating a single image at a time and can condition on zero, one, or more views. As a result, when generating a large number of views, our method is not restricted to a low-order autoregressive generation approach and is better able to maintain generated image quality over large sets of images. We evaluate the proposed model on standard NVS datasets and show that it outperforms the state-of-the-art image-based GNVS baselines. Further, we show that the model is capable of generating sets of camera views that have no natural sequential ordering, like loops and binocular trajectories, and significantly outperforms other methods on such tasks.

Title: Mixer is more than just a model

• Authors: Qingfeng Ji, Yuxin Wang, Letong Sun
• Subjects: cs.LG, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.18007
• Pdf URL: https://arxiv.org/pdf/2402.18007
• Copy Paste: [[2402.18007]] Mixer is more than just a model(https://arxiv.org/abs/2402.18007)
• Keywords: extraction
• Abstract: Recently, MLP structures have regained popularity, with MLP-Mixer standing out as a prominent example. In the field of computer vision, MLP-Mixer is noted for its ability to extract data information from both channel and token perspectives, effectively acting as a fusion of channel and token information. Indeed, Mixer represents a paradigm for information extraction that amalgamates channel and token information. The essence of Mixer lies in its ability to blend information from diverse perspectives, epitomizing the true concept of "mixing" in the realm of neural network architectures. Beyond channel and token considerations, it is possible to create more tailored mixers from various perspectives to better suit specific task requirements. This study focuses on the domain of audio recognition, introducing a novel model named Audio Spectrogram Mixer with Roll-Time and Hermit FFT (ASM-RH) that incorporates insights from both time and frequency domains. Experimental results demonstrate that ASM-RH is particularly well-suited for audio data and yields promising outcomes across multiple classification tasks.

Title: Representing 3D sparse map points and lines for camera relocalization

• Authors: Bach-Thuan Bui, Huy-Hoang Bui, Dinh-Tuan Tran, Joo-Ho Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18011
• Pdf URL: https://arxiv.org/pdf/2402.18011
• Copy Paste: [[2402.18011]] Representing 3D sparse map points and lines for camera relocalization(https://arxiv.org/abs/2402.18011)
• Keywords: secure, transformer
• Abstract: Recent advancements in visual localization and mapping have demonstrated considerable success in integrating point and line features. However, expanding the localization framework to include additional mapping components frequently results in increased demand for memory and computational resources dedicated to matching tasks. In this study, we show how a lightweight neural network can learn to represent both 3D point and line features, and exhibit leading pose accuracy by harnessing the power of multiple learned mappings. Specifically, we utilize a single transformer block to encode line features, effectively transforming them into distinctive point-like descriptors. Subsequently, we treat these point and line descriptor sets as distinct yet interconnected feature sets. Through the integration of self- and cross-attention within several graph layers, our method effectively refines each feature before regressing 3D maps using two simple MLPs. In comprehensive experiments, our indoor localization findings surpass those of Hloc and Limap across both point-based and line-assisted configurations. Moreover, in outdoor scenarios, our method secures a significant lead, marking the most considerable enhancement over state-of-the-art learning-based methodologies. The source code and demo videos of this work are publicly available at: https://thpjp.github.io/pl2map/

Title: Diffusion Models as Constrained Samplers for Optimization with Unknown Constraints

• Authors: Lingkai Kong, Yuanqi Du, Wenhao Mu, Kirill Neklyudov, Valentin De Bortol, Haorui Wang, Dongxia Wu, Aaron Ferber, Yi-An Ma, Carla P. Gomes, Chao Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18012
• Pdf URL: https://arxiv.org/pdf/2402.18012
• Copy Paste: [[2402.18012]] Diffusion Models as Constrained Samplers for Optimization with Unknown Constraints(https://arxiv.org/abs/2402.18012)
• Keywords: diffusion
• Abstract: Addressing real-world optimization problems becomes particularly challenging when analytic objective functions or constraints are unavailable. While numerous studies have addressed the issue of unknown objectives, limited research has focused on scenarios where feasibility constraints are not given explicitly. Overlooking these constraints can lead to spurious solutions that are unrealistic in practice. To deal with such unknown constraints, we propose to perform optimization within the data manifold using diffusion models. To constrain the optimization process to the data manifold, we reformulate the original optimization problem as a sampling problem from the product of the Boltzmann distribution defined by the objective function and the data distribution learned by the diffusion model. To enhance sampling efficiency, we propose a two-stage framework that begins with a guided diffusion process for warm-up, followed by a Langevin dynamics stage for further correction. Theoretical analysis shows that the initial stage results in a distribution focused on feasible solutions, thereby providing a better initialization for the later stage. Comprehensive experiments on a synthetic dataset, six real-world black-box optimization datasets, and a multi-objective optimization dataset show that our method achieves better or comparable performance with previous state-of-the-art baselines.

Title: A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems

• Authors: Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, Ying Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18013
• Pdf URL: https://arxiv.org/pdf/2402.18013
• Copy Paste: [[2402.18013]] A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems(https://arxiv.org/abs/2402.18013)
• Keywords: large language model
• Abstract: This survey provides a comprehensive review of research on multi-turn dialogue systems, with a particular focus on multi-turn dialogue systems based on large language models (LLMs). This paper aims to (a) give a summary of existing LLMs and approaches for adapting LLMs to downstream tasks; (b) elaborate recent advances in multi-turn dialogue systems, covering both LLM-based open-domain dialogue (ODD) and task-oriented dialogue (TOD) systems, along with datasets and evaluation metrics; (c) discuss some future emphasis and recent research problems arising from the development of LLMs and the increasing demands on multi-turn dialogue systems.

Title: Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach

• Authors: Bin Wang, Jun Fang, Hongbin Li, Yonina C. Eldar
• Subjects: cs.LG, cs.DC, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.18018
• Pdf URL: https://arxiv.org/pdf/2402.18018
• Copy Paste: [[2402.18018]] Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach(https://arxiv.org/abs/2402.18018)
• Keywords: federate
• Abstract: Federated learning (FL) is a machine learning paradigm that targets model training without gathering the local data dispersed over various data sources. Standard FL, which employs a single server, can only support a limited number of users, leading to degraded learning capability. In this work, we consider a multi-server FL framework, referred to as \emph{Confederated Learning} (CFL), in order to accommodate a larger number of users. A CFL system is composed of multiple networked edge servers, with each server connected to an individual set of users. Decentralized collaboration among servers is leveraged to harness all users' data for model training. Due to the potentially massive number of users involved, it is crucial to reduce the communication overhead of the CFL system. We propose a stochastic gradient method for distributed learning in the CFL framework. The proposed method incorporates a conditionally-triggered user selection (CTUS) mechanism as the central component to effectively reduce communication overhead. Relying on a delicately designed triggering condition, the CTUS mechanism allows each server to select only a small number of users to upload their gradients, without significantly jeopardizing the convergence performance of the algorithm. Our theoretical analysis reveals that the proposed algorithm enjoys a linear convergence rate. Simulation results show that it achieves substantial improvement over state-of-the-art algorithms in terms of communication efficiency.

Title: Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions

• Authors: Kexun Zhang, Yee Man Choi, Zhenqiao Song, Taiqi He, William Yang Wang, Lei Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18025
• Pdf URL: https://arxiv.org/pdf/2402.18025
• Copy Paste: [[2402.18025]] Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions(https://arxiv.org/abs/2402.18025)
• Keywords: large language model
• Abstract: How can large language models (LLMs) process and translate endangered languages? Many languages lack a large corpus to train a decent LLM; therefore existing LLMs rarely perform well in unseen, endangered languages. On the contrary, we observe that 2000 endangered languages, though without a large corpus, have a grammar book or a dictionary. We propose LINGOLLM, a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training. Our key insight is to demonstrate linguistic knowledge of an unseen language in an LLM's prompt, including a dictionary, a grammar book, and morphologically analyzed input text. We implement LINGOLLM on top of two models, GPT-4 and Mixtral, and evaluate their performance on 5 tasks across 8 endangered or low-resource languages. Our results show that LINGOLLM elevates translation capability from GPT-4's 0 to 10.5 BLEU for 10 language directions. Our findings demonstrate the tremendous value of linguistic knowledge in the age of LLMs for endangered languages. Our data, code, and model generations can be found at https://github.com/LLiLab/llm4endangeredlang.

Title: Breaking the Black-Box: Confidence-Guided Model Inversion Attack for Distribution Shift

• Authors: Xinhao Liu, Yingzhao Jiang, Zetao Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18027
• Pdf URL: https://arxiv.org/pdf/2402.18027
• Copy Paste: [[2402.18027]] Breaking the Black-Box: Confidence-Guided Model Inversion Attack for Distribution Shift(https://arxiv.org/abs/2402.18027)
• Keywords: attack, generative
• Abstract: Model inversion attacks (MIAs) seek to infer the private training data of a target classifier by generating synthetic images that reflect the characteristics of the target class through querying the model. However, prior studies have relied on full access to the target model, which is not practical in real-world scenarios. Additionally, existing black-box MIAs assume that the image prior and target model follow the same distribution. However, when confronted with diverse data distribution settings, these methods may result in suboptimal performance in conducting attacks. To address these limitations, this paper proposes a \textbf{C}onfidence-\textbf{G}uided \textbf{M}odel \textbf{I}nversion attack method called CG-MI, which utilizes the latent space of a pre-trained publicly available generative adversarial network (GAN) as prior information and gradient-free optimizer, enabling high-resolution MIAs across different data distributions in a black-box setting. Our experiments demonstrate that our method significantly \textbf{outperforms the SOTA black-box MIA by more than 49\% for Celeba and 58\% for Facescrub in different distribution settings}. Furthermore, our method exhibits the ability to generate high-quality images \textbf{comparable to those produced by white-box attacks}. Our method provides a practical and effective solution for black-box model inversion attacks.

Title: Efficient Fault Detection Architectures for Modular Exponentiation Targeting Cryptographic Applications Benchmarked on FPGAs

• Authors: Saeed Aghapour, Kasra Ahmadi, Mehran Mozaffari Kermani, Reza Azarderakhsh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18033
• Pdf URL: https://arxiv.org/pdf/2402.18033
• Copy Paste: [[2402.18033]] Efficient Fault Detection Architectures for Modular Exponentiation Targeting Cryptographic Applications Benchmarked on FPGAs(https://arxiv.org/abs/2402.18033)
• Keywords: secure, security, protect, attack
• Abstract: Whether stemming from malicious intent or natural occurrences, faults and errors can significantly undermine the reliability of any architecture. In response to this challenge, fault detection assumes a pivotal role in ensuring the secure deployment of cryptosystems. Even when a cryptosystem boasts mathematical security, its practical implementation may remain susceptible to exploitation through side-channel attacks. In this paper, we propose a lightweight fault detection architecture tailored for modular exponentiation, a building block of numerous cryptographic applications spanning from classical cryptography to post quantum cryptography. Based on our simulation and implementation results on ARM Cortex-A72 processor, and AMD/Xilinx Zynq Ultrascale+, and Artix-7 FPGAs, our approach achieves an error detection rate close to 100%, all while introducing a modest computational overhead of approximately 7% and area overhead of less than 1% compared to the unprotected architecture. To the best of our knowledge, such an approach benchmarked on ARM processor and FPGA has not been proposed and assessed to date.

Title: ResLoRA: Identity Residual Mapping in Low-Rank Adaption

• Authors: Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18039
• Pdf URL: https://arxiv.org/pdf/2402.18039
• Copy Paste: [[2402.18039]] ResLoRA: Identity Residual Mapping in Low-Rank Adaption(https://arxiv.org/abs/2402.18039)
• Keywords: large language model
• Abstract: As one of the most popular parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) is commonly applied to fine-tune large language models (LLMs). However, updating the weights of LoRA blocks effectively and expeditiously is challenging due to the long calculation path in the original model. To address this, we propose ResLoRA, an improved framework of LoRA. By adding residual paths during training and using merging approaches to eliminate these extra paths during inference, our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method. To the best of our knowledge, ResLoRA is the first work that combines the residual path with LoRA. The code of our method is available at https://github.com/microsoft/LMOps/tree/main/reslora .

Title: Datasets for Large Language Models: A Comprehensive Survey

• Authors: Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, Lianwen Jin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18041
• Pdf URL: https://arxiv.org/pdf/2402.18041
• Copy Paste: [[2402.18041]] Datasets for Large Language Models: A Comprehensive Survey(https://arxiv.org/abs/2402.18041)
• Keywords: large language model
• Abstract: This paper embarks on an exploration into the Large Language Model (LLM) datasets, which play a crucial role in the remarkable advancements of LLMs. The datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of LLMs. Consequently, examination of these datasets emerges as a critical topic in research. In order to address the current lack of a comprehensive overview and thorough analysis of LLM datasets, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction Fine-tuning Datasets; (3) Preference Datasets; (4) Evaluation Datasets; (5) Traditional Natural Language Processing (NLP) Datasets. The survey sheds light on the prevailing challenges and points out potential avenues for future investigation. Additionally, a comprehensive review of the existing available dataset resources is also provided, including statistics from 444 datasets, covering 8 language categories and spanning 32 domains. Information from 20 dimensions is incorporated into the dataset statistics. The total data size surveyed surpasses 774.5 TB for pre-training corpora and 700M instances for other datasets. We aim to present the entire landscape of LLM text datasets, serving as a comprehensive reference for researchers in this field and contributing to future studies. Related resources are available at: https://github.com/lmmlzn/Awesome-LLMs-Datasets.

Title: Crisis talk: analysis of the public debate around the energy crisis and cost of living

• Authors: Rrubaa Panchendrarajan, Geri Popova, Tony Russell-Rose
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18043
• Pdf URL: https://arxiv.org/pdf/2402.18043
• Copy Paste: [[2402.18043]] Crisis talk: analysis of the public debate around the energy crisis and cost of living(https://arxiv.org/abs/2402.18043)
• Keywords: fair
• Abstract: A prominent media topic in the UK in the early 2020s is the energy crisis affecting the UK and most of Europe. It brings into a single public debate issues of energy dependency and sustainability, fair distribution of economic burdens and cost of living, as well as climate change, risk, and sustainability. In this paper, we investigate the public discourse around the energy crisis and cost of living to identify how these pivotal and contradictory issues are reconciled in this debate and to identify which social actors are involved and the role they play. We analyse a document corpus retrieved from UK newspapers from January 2014 to March 2023. We apply a variety of natural language processing and data visualisation techniques to identify key topics, novel trends, critical social actors, and the role they play in the debate, along with the sentiment associated with those actors and topics. We combine automated techniques with manual discourse analysis to explore and validate the insights revealed in this study. The findings verify the utility of these techniques by providing a flexible and scalable pipeline for discourse analysis and providing critical insights for cost of living - energy crisis nexus research.

Title: SFTformer: A Spatial-Frequency-Temporal Correlation-Decoupling Transformer for Radar Echo Extrapolation

• Authors: Liangyu Xu, Wanxuan Lu, Hongfeng Yu, Fanglong Yao, Xian Sun, Kun Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18044
• Pdf URL: https://arxiv.org/pdf/2402.18044
• Copy Paste: [[2402.18044]] SFTformer: A Spatial-Frequency-Temporal Correlation-Decoupling Transformer for Radar Echo Extrapolation(https://arxiv.org/abs/2402.18044)
• Keywords: transformer
• Abstract: Extrapolating future weather radar echoes from past observations is a complex task vital for precipitation nowcasting. The spatial morphology and temporal evolution of radar echoes exhibit a certain degree of correlation, yet they also possess independent characteristics. {Existing methods learn unified spatial and temporal representations in a highly coupled feature space, emphasizing the correlation between spatial and temporal features but neglecting the explicit modeling of their independent characteristics, which may result in mutual interference between them.} To effectively model the spatiotemporal dynamics of radar echoes, we propose a Spatial-Frequency-Temporal correlation-decoupling Transformer (SFTformer). The model leverages stacked multiple SFT-Blocks to not only mine the correlation of the spatiotemporal dynamics of echo cells but also avoid the mutual interference between the temporal modeling and the spatial morphology refinement by decoupling them. Furthermore, inspired by the practice that weather forecast experts effectively review historical echo evolution to make accurate predictions, SFTfomer incorporates a joint training paradigm for historical echo sequence reconstruction and future echo sequence prediction. Experimental results on the HKO-7 dataset and ChinaNorth-2021 dataset demonstrate the superior performance of SFTfomer in short(1h), mid(2h), and long-term(3h) precipitation nowcasting.

Title: Multi-FAct: Assessing Multilingual LLMs' Multi-Regional Knowledge using FActScore

• Authors: Sheikh Shafayat, Eunsu Kim, Juhyun Oh, Alice Oh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18045
• Pdf URL: https://arxiv.org/pdf/2402.18045
• Copy Paste: [[2402.18045]] Multi-FAct: Assessing Multilingual LLMs' Multi-Regional Knowledge using FActScore(https://arxiv.org/abs/2402.18045)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are prone to factuality hallucination, generating text that contradicts established knowledge. While extensive research has addressed this in English, little is known about multilingual LLMs. This paper systematically evaluates multilingual LLMs' factual accuracy across languages and geographic regions. We introduce a novel pipeline for multilingual factuality evaluation, adapting FActScore(Min et al., 2023) for diverse languages. Our analysis across nine languages reveals that English consistently outperforms others in factual accuracy and quantity of generated facts. Furthermore, multilingual models demonstrate a bias towards factual information from Western continents. These findings highlight the need for improved multilingual factuality assessment and underscore geographical biases in LLMs' fact generation.

Title: Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension

• Authors: Fan Yin, Jayanth Srinivasa, Kai-Wei Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18048
• Pdf URL: https://arxiv.org/pdf/2402.18048
• Copy Paste: [[2402.18048]] Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension(https://arxiv.org/abs/2402.18048)
• Keywords: generative, large language model
• Abstract: We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs), which serves as a crucial step in building trust between humans and LLMs. Although several approaches based on entropy or verbalized uncertainty have been proposed to calibrate model predictions, these methods are often intractable, sensitive to hyperparameters, and less reliable when applied in generative tasks with LLMs. In this paper, we suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations. Through experiments on four question answering (QA) datasets, we demonstrate the effectiveness ohttps://info.arxiv.org/help/prep#abstractsf our proposed method. Additionally, we study intrinsic dimensions in LLMs and their relations with model layers, autoregressive language modeling, and the training of LLMs, revealing that intrinsic dimensions can be a powerful approach to understanding LLMs.

Title: MEGAnno+: A Human-LLM Collaborative Annotation System

• Authors: Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, Dan Zhang
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.18050
• Pdf URL: https://arxiv.org/pdf/2402.18050
• Copy Paste: [[2402.18050]] MEGAnno+: A Human-LLM Collaborative Annotation System(https://arxiv.org/abs/2402.18050)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels. We present MEGAnno+, a human-LLM collaborative annotation system that offers effective LLM agent and annotation management, convenient and robust LLM annotation, and exploratory verification of LLM labels by humans.

Title: Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models

• Authors: Mingjia Huo, Sai Ashish Somayajula, Youwei Liang, Ruisi Zhang, Farinaz Koushanfar, Pengtao Xie
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18059
• Pdf URL: https://arxiv.org/pdf/2402.18059
• Copy Paste: [[2402.18059]] Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models(https://arxiv.org/abs/2402.18059)
• Keywords: watermark, large language model
• Abstract: Large language models generate high-quality responses with potential misinformation, underscoring the need for regulation by distinguishing AI-generated and human-written texts. Watermarking is pivotal in this context, which involves embedding hidden markers in texts during the LLM inference phase, which is imperceptible to humans. Current watermarking algorithms, however, face the challenge of achieving both the detectability of inserted watermarks and the semantic integrity of generated texts, where enhancing one aspect often undermines the other. To overcome this, we introduce a novel multi-objective optimization (MOO) approach for watermarking that utilizes lightweight networks to generate token-specific watermarking logits and splitting ratios. By leveraging MOO to optimize for both detection and semantic objective functions, our method simultaneously achieves detectability and semantic integrity. Experimental results show that our method outperforms current watermarking techniques in enhancing the detectability of texts generated by LLMs while maintaining their semantic coherence. Our code is available at https://github.com/mignonjia/TS_watermark .

Title: Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions

• Authors: Hanjie Chen, Zhouxiang Fang, Yash Singla, Mark Dredze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18060
• Pdf URL: https://arxiv.org/pdf/2402.18060
• Copy Paste: [[2402.18060]] Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions(https://arxiv.org/abs/2402.18060)
• Keywords: large language model
• Abstract: LLMs have demonstrated impressive performance in answering medical questions, such as passing medical licensing examinations. However, most existing benchmarks rely on board exam questions or general medical questions, falling short in capturing the complexity of realistic clinical cases. Moreover, the lack of reference explanations for answers hampers the evaluation of model explanations, which are crucial to supporting doctors in making complex medical decisions. To address these challenges, we construct two new datasets: JAMA Clinical Challenge and Medbullets. JAMA Clinical Challenge consists of questions based on challenging clinical cases, while Medbullets comprises USMLE Step 2&3 style clinical questions. Both datasets are structured as multiple-choice question-answering tasks, where each question is accompanied by an expert-written explanation. We evaluate four LLMs on the two datasets using various prompts. Experiments demonstrate that our datasets are harder than previous benchmarks. The inconsistency between automatic and human evaluations of model-generated explanations highlights the need to develop new metrics to support future research on explainable medical QA.

Title: On the use of Silver Standard Data for Zero-shot Classification Tasks in Information Extraction

• Authors: Jianwei Wang, Tianyin Wang, Ziqian Zeng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18061
• Pdf URL: https://arxiv.org/pdf/2402.18061
• Copy Paste: [[2402.18061]] On the use of Silver Standard Data for Zero-shot Classification Tasks in Information Extraction(https://arxiv.org/abs/2402.18061)
• Keywords: extraction
• Abstract: The superior performance of supervised classification methods in the information extraction (IE) area heavily relies on a large amount of gold standard data. Recent zero-shot classification methods converted the task to other NLP tasks (e.g., textual entailment) and used off-the-shelf models of these NLP tasks to directly perform inference on the test data without using a large amount of IE annotation data. A potentially valuable by-product of these methods is the large-scale silver standard data, i.e., pseudo-labeled data by the off-the-shelf models of other NLP tasks. However, there is no further investigation into the use of these data. In this paper, we propose a new framework, Clean-LaVe, which aims to utilize silver standard data to enhance the zero-shot performance. Clean-LaVe includes four phases: (1) Obtaining silver data; (2) Identifying relatively clean data from silver data; (3) Finetuning the off-the-shelf model using clean data; (4) Inference on the test data. The experimental results show that Clean-LaVe can outperform the baseline by 5% and 6% on TACRED and Wiki80 dataset in the zero-shot relation classification task, and by 3%-7% on Smile (Korean and Polish) in the zero-shot cross-lingual relation classification task, and by 8% on ACE05-E+ in the zero-shot event argument classification task. The code is share in https://github.com/wjw136/Clean_LaVe.git.

Title: SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model

• Authors: Bin Cao, Jianhao Yuan, Yexin Liu, Jian Li, Shuyang Sun, Jing Liu, Bo Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18068
• Pdf URL: https://arxiv.org/pdf/2402.18068
• Copy Paste: [[2402.18068]] SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model(https://arxiv.org/abs/2402.18068)
• Keywords: diffusion, generative
• Abstract: In the rapidly evolving area of image synthesis, a serious challenge is the presence of complex artifacts that compromise perceptual realism of synthetic images. To alleviate artifacts and improve quality of synthetic images, we fine-tune Vision-Language Model (VLM) as artifact classifier to automatically identify and classify a wide range of artifacts and provide supervision for further optimizing generative models. Specifically, we develop a comprehensive artifact taxonomy and construct a dataset of synthetic images with artifact annotations for fine-tuning VLM, named SynArtifact-1K. The fine-tuned VLM exhibits superior ability of identifying artifacts and outperforms the baseline by 25.66%. To our knowledge, this is the first time such end-to-end artifact classification task and solution have been proposed. Finally, we leverage the output of VLM as feedback to refine the generative model for alleviating artifacts. Visualization results and user study demonstrate that the quality of images synthesized by the refined diffusion model has been obviously improved.

Title: Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis

• Authors: Yanzuo Lu, Manlin Zhang, Andy J Ma, Xiaohua Xie, Jian-Huang Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18078
• Pdf URL: https://arxiv.org/pdf/2402.18078
• Copy Paste: [[2402.18078]] Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis(https://arxiv.org/abs/2402.18078)
• Keywords: diffusion
• Abstract: Diffusion model is a promising approach to image generation and has been employed for Pose-Guided Person Image Synthesis (PGPIS) with competitive performance. While existing methods simply align the person appearance to the target pose, they are prone to overfitting due to the lack of a high-level semantic understanding on the source person image. In this paper, we propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for PGPIS. In the absence of image-caption pairs and textual prompts, we develop a novel training paradigm purely based on images to control the generation process of the pre-trained text-to-image diffusion model. A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt. This allows for the decoupling of fine-grained appearance and pose information controls at different stages, and thus circumventing the potential overfitting problem. To generate more realistic texture details, a hybrid-granularity attention module is proposed to encode multi-scale fine-grained appearance features as bias terms to augment the coarse-grained prompt. Both quantitative and qualitative experimental results on the DeepFashion benchmark demonstrate the superiority of our method over the state of the arts for PGPIS. Code is available at https://github.com/YanzuoLu/CFLD.

Title: Spannotation: Enhancing Semantic Segmentation for Autonomous Navigation with Efficient Image Annotation

• Authors: Samuel O. Folorunsho, William R. Norris
• Subjects: cs.CV, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.18084
• Pdf URL: https://arxiv.org/pdf/2402.18084
• Copy Paste: [[2402.18084]] Spannotation: Enhancing Semantic Segmentation for Autonomous Navigation with Efficient Image Annotation(https://arxiv.org/abs/2402.18084)
• Keywords: segmentation
• Abstract: Spannotation is an open source user-friendly tool developed for image annotation for semantic segmentation specifically in autonomous navigation tasks. This study provides an evaluation of Spannotation, demonstrating its effectiveness in generating accurate segmentation masks for various environments like agricultural crop rows, off-road terrains and urban roads. Unlike other popular annotation tools that requires about 40 seconds to annotate an image for semantic segmentation in a typical navigation task, Spannotation achieves similar result in about 6.03 seconds. The tools utility was validated through the utilization of its generated masks to train a U-Net model which achieved a validation accuracy of 98.27% and mean Intersection Over Union (mIOU) of 96.66%. The accessibility, simple annotation process and no-cost features have all contributed to the adoption of Spannotation evident from its download count of 2098 (as of February 25, 2024) since its launch. Future enhancements of Spannotation aim to broaden its application to complex navigation scenarios and incorporate additional automation functionalities. Given its increasing popularity and promising potential, Spannotation stands as a valuable resource in autonomous navigation and semantic segmentation. For detailed information and access to Spannotation, readers are encouraged to visit the project's GitHub repository at https://github.com/sof-danny/spannotation

Title: Polos: Multimodal Metric Learning from Human Feedback for Image Captioning

• Authors: Yuiga Wada, Kanta Kaneda, Daichi Saito, Komei Sugiura
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18091
• Pdf URL: https://arxiv.org/pdf/2402.18091
• Copy Paste: [[2402.18091]] Polos: Multimodal Metric Learning from Human Feedback for Image Captioning(https://arxiv.org/abs/2402.18091)
• Keywords: robust, extraction
• Abstract: Establishing an automatic evaluation metric that closely aligns with human judgments is essential for effectively developing image captioning models. Recent data-driven metrics have demonstrated a stronger correlation with human judgments than classic metrics such as CIDEr; however they lack sufficient capabilities to handle hallucinations and generalize across diverse images and texts partially because they compute scalar similarities merely using embeddings learned from tasks unrelated to image captioning evaluation. In this study, we propose Polos, a supervised automatic evaluation metric for image captioning models. Polos computes scores from multimodal inputs, using a parallel feature extraction mechanism that leverages embeddings trained through large-scale contrastive learning. To train Polos, we introduce Multimodal Metric Learning from Human Feedback (M$^2$LHF), a framework for developing metrics based on human feedback. We constructed the Polaris dataset, which comprises 131K human judgments from 550 evaluators, which is approximately ten times larger than standard datasets. Our approach achieved state-of-the-art performance on Composite, Flickr8K-Expert, Flickr8K-CF, PASCAL-50S, FOIL, and the Polaris dataset, thereby demonstrating its effectiveness and robustness.

Title: Context-aware Talking Face Video Generation

• Authors: Meidai Xuanyuan, Yuwang Wang, Honglei Guo, Qionghai Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18092
• Pdf URL: https://arxiv.org/pdf/2402.18092
• Copy Paste: [[2402.18092]] Context-aware Talking Face Video Generation(https://arxiv.org/abs/2402.18092)
• Keywords: diffusion
• Abstract: In this paper, we consider a novel and practical case for talking face video generation. Specifically, we focus on the scenarios involving multi-people interactions, where the talking context, such as audience or surroundings, is present. In these situations, the video generation should take the context into consideration in order to generate video content naturally aligned with driving audios and spatially coherent to the context. To achieve this, we provide a two-stage and cross-modal controllable video generation pipeline, taking facial landmarks as an explicit and compact control signal to bridge the driving audio, talking context and generated videos. Inside this pipeline, we devise a 3D video diffusion model, allowing for efficient contort of both spatial conditions (landmarks and context video), as well as audio condition for temporally coherent generation. The experimental results verify the advantage of the proposed method over other baselines in terms of audio-video synchronization, video fidelity and frame consistency.

Title: ChatSpamDetector: Leveraging Large Language Models for Effective Phishing Email Detection

• Authors: Takashi Koide, Naoki Fukushi, Hiroki Nakano, Daiki Chiba
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18093
• Pdf URL: https://arxiv.org/pdf/2402.18093
• Copy Paste: [[2402.18093]] ChatSpamDetector: Leveraging Large Language Models for Effective Phishing Email Detection(https://arxiv.org/abs/2402.18093)
• Keywords: security, large language model
• Abstract: The proliferation of phishing sites and emails poses significant challenges to existing cybersecurity efforts. Despite advances in spam filters and email security protocols, problems with oversight and false positives persist. Users often struggle to understand why emails are flagged as spam, risking the possibility of missing important communications or mistakenly trusting phishing emails. This study introduces ChatSpamDetector, a system that uses large language models (LLMs) to detect phishing emails. By converting email data into a prompt suitable for LLM analysis, the system provides a highly accurate determination of whether an email is phishing or not. Importantly, it offers detailed reasoning for its phishing determinations, assisting users in making informed decisions about how to handle suspicious emails. We conducted an evaluation using a comprehensive phishing email dataset and compared our system to several LLMs and baseline systems. We confirmed that our system using GPT-4 has superior detection capabilities with an accuracy of 99.70%. Advanced contextual interpretation by LLMs enables the identification of various phishing tactics and impersonations, making them a potentially powerful tool in the fight against email-based phishing threats.

Title: No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

• Authors: June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, Dongsoo Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18096
• Pdf URL: https://arxiv.org/pdf/2402.18096
• Copy Paste: [[2402.18096]] No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization(https://arxiv.org/abs/2402.18096)
• Keywords: generative, large language model
• Abstract: Key-Value (KV) Caching has become an essential technique for accelerating the inference speed and throughput of generative Large Language Models~(LLMs). However, the memory footprint of the KV cache poses a critical bottleneck in LLM deployment as the cache size grows with batch size and sequence length, often surpassing even the size of the model itself. Although recent methods were proposed to select and evict unimportant KV pairs from the cache to reduce memory consumption, the potential ramifications of eviction on the generative process are yet to be thoroughly examined. In this paper, we examine the detrimental impact of cache eviction and observe that unforeseen risks arise as the information contained in the KV pairs is exhaustively discarded, resulting in safety breaches, hallucinations, and context loss. Surprisingly, we find that preserving even a small amount of information contained in the evicted KV pairs via reduced precision quantization substantially recovers the incurred degradation. On the other hand, we observe that the important KV pairs must be kept at a relatively higher precision to safeguard the generation quality. Motivated by these observations, we propose \textit{Mixed-precision KV cache}~(MiKV), a reliable cache compression method that simultaneously preserves the context details by retaining the evicted KV pairs in low-precision and ensure generation quality by keeping the important KV pairs in high-precision. Experiments on diverse benchmarks and LLM backbones show that our proposed method offers a state-of-the-art trade-off between compression ratio and performance, compared to other baselines.

Title: Editing Factual Knowledge and Explanatory Ability of Medical Large Language Models

• Authors: Derong Xu, Ziheng Zhang, Zhihong Zhu, Zhenxi Lin, Qidong Liu, Xian Wu, Tong Xu, Xiangyu Zhao, Yefeng Zheng, Enhong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18099
• Pdf URL: https://arxiv.org/pdf/2402.18099
• Copy Paste: [[2402.18099]] Editing Factual Knowledge and Explanatory Ability of Medical Large Language Models(https://arxiv.org/abs/2402.18099)
• Keywords: large language model
• Abstract: Model editing aims to precisely modify the behaviours of large language models (LLMs) on specific knowledge while keeping irrelevant knowledge unchanged. It has been proven effective in resolving hallucination and out-of-date issues in LLMs. As a result, it can boost the application of LLMs in many critical domains (e.g., medical domain), where the hallucination is not tolerable. In this paper, we propose two model editing studies and validate them in the medical domain: (1) directly editing the factual medical knowledge and (2) editing the explanations to facts. Meanwhile, we observed that current model editing methods struggle with the specialization and complexity of medical knowledge. Therefore, we propose MedLaSA, a novel Layer-wise Scalable Adapter strategy for medical model editing. It employs causal tracing to identify the precise location of knowledge in neurons and then introduces scalable adapters into the dense layers of LLMs. These adapters are assigned scaling values based on the corresponding specific knowledge. To evaluate the editing impact, we build two benchmark datasets and introduce a series of challenging and comprehensive metrics. Extensive experiments on medical LLMs demonstrate the editing efficiency of MedLaSA, without affecting irrelevant knowledge that is not edited.

Title: Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction

• Authors: Tong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong, Guozhu Meng, Kai Chen
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18104
• Pdf URL: https://arxiv.org/pdf/2402.18104
• Copy Paste: [[2402.18104]] Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction(https://arxiv.org/abs/2402.18104)
• Keywords: security, attack, large language model
• Abstract: In recent years, large language models (LLMs) have demonstrated notable success across various tasks, but the trustworthiness of LLMs is still an open problem. One specific threat is the potential to generate toxic or harmful responses. Attackers can craft adversarial prompts that induce harmful responses from LLMs. In this work, we pioneer a theoretical foundation in LLMs security by identifying bias vulnerabilities within the safety fine-tuning and design a black-box jailbreak method named DRA (Disguise and Reconstruction Attack), which conceals harmful instructions through disguise and prompts the model to reconstruct the original harmful instruction within its completion. We evaluate DRA across various open-source and close-source models, showcasing state-of-the-art jailbreak success rates and attack efficiency. Notably, DRA boasts a 90\% attack success rate on LLM chatbots GPT-4.

Title: Dual-Context Aggregation for Universal Image Matting

• Authors: Qinglin Liu, Xiaoqian Lv, Wei Yu, Changyong Guo, Shengping Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18109
• Pdf URL: https://arxiv.org/pdf/2402.18109
• Copy Paste: [[2402.18109]] Dual-Context Aggregation for Universal Image Matting(https://arxiv.org/abs/2402.18109)
• Keywords: robust, segmentation
• Abstract: Natural image matting aims to estimate the alpha matte of the foreground from a given image. Various approaches have been explored to address this problem, such as interactive matting methods that use guidance such as click or trimap, and automatic matting methods tailored to specific objects. However, existing matting methods are designed for specific objects or guidance, neglecting the common requirement of aggregating global and local contexts in image matting. As a result, these methods often encounter challenges in accurately identifying the foreground and generating precise boundaries, which limits their effectiveness in unforeseen scenarios. In this paper, we propose a simple and universal matting framework, named Dual-Context Aggregation Matting (DCAM), which enables robust image matting with arbitrary guidance or without guidance. Specifically, DCAM first adopts a semantic backbone network to extract low-level features and context features from the input image and guidance. Then, we introduce a dual-context aggregation network that incorporates global object aggregators and local appearance aggregators to iteratively refine the extracted context features. By performing both global contour segmentation and local boundary refinement, DCAM exhibits robustness to diverse types of guidance and objects. Finally, we adopt a matting decoder network to fuse the low-level features and the refined context features for alpha matte estimation. Experimental results on five matting datasets demonstrate that the proposed DCAM outperforms state-of-the-art matting methods in both automatic matting and interactive matting tasks, which highlights the strong universality and high performance of DCAM. The source code is available at \url{https://github.com/Windaway/DCAM}.

Title: Small But Funny: A Feedback-Driven Approach to Humor Distillation

• Authors: Sahithya Ravi, Patrick Huber, Akshat Shrivastava, Aditya Sagar, Ahmed Aly, Vered Shwartz, Arash Einolghozati
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18113
• Pdf URL: https://arxiv.org/pdf/2402.18113
• Copy Paste: [[2402.18113]] Small But Funny: A Feedback-Driven Approach to Humor Distillation(https://arxiv.org/abs/2402.18113)
• Keywords: large language model
• Abstract: The emergence of Large Language Models (LLMs) has brought to light promising language generation capabilities, particularly in performing tasks like complex reasoning and creative writing. Consequently, distillation through imitation of teacher responses has emerged as a popular technique to transfer knowledge from LLMs to more accessible, Small Language Models (SLMs). While this works well for simpler tasks, there is a substantial performance gap on tasks requiring intricate language comprehension and creativity, such as humor generation. We hypothesize that this gap may stem from the fact that creative tasks might be hard to learn by imitation alone and explore whether an approach, involving supplementary guidance from the teacher, could yield higher performance. To address this, we study the effect of assigning a dual role to the LLM - as a "teacher" generating data, as well as a "critic" evaluating the student's performance. Our experiments on humor generation reveal that the incorporation of feedback significantly narrows the performance gap between SLMs and their larger counterparts compared to merely relying on imitation. As a result, our research highlights the potential of using feedback as an additional dimension to data when transferring complex language abilities via distillation.

Title: UniVS: Unified and Universal Video Segmentation with Prompts as Queries

• Authors: Minghan Li, Shuai Li, Xindong Zhang, Lei Zhang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18115
• Pdf URL: https://arxiv.org/pdf/2402.18115
• Copy Paste: [[2402.18115]] UniVS: Unified and Universal Video Segmentation with Prompts as Queries(https://arxiv.org/abs/2402.18115)
• Keywords: robust, segmentation
• Abstract: Despite the recent advances in unified image segmentation (IS), developing a unified video segmentation (VS) model remains a challenge. This is mainly because generic category-specified VS tasks need to detect all objects and track them across consecutive frames, while prompt-guided VS tasks require re-identifying the target with visual/text prompts throughout the entire video, making it hard to handle the different tasks with the same architecture. We make an attempt to address these issues and present a novel unified VS architecture, namely UniVS, by using prompts as queries. UniVS averages the prompt features of the target from previous frames as its initial query to explicitly decode masks, and introduces a target-wise prompt cross-attention layer in the mask decoder to integrate prompt features in the memory pool. By taking the predicted masks of entities from previous frames as their visual prompts, UniVS converts different VS tasks into prompt-guided target segmentation, eliminating the heuristic inter-frame matching process. Our framework not only unifies the different VS tasks but also naturally achieves universal training and testing, ensuring robust performance across different scenarios. UniVS shows a commendable balance between performance and universality on 10 challenging VS benchmarks, covering video instance, semantic, panoptic, object, and referring segmentation tasks. Code can be found at \url{https://github.com/MinghanLi/UniVS}.

Title: PRCL: Probabilistic Representation Contrastive Learning for Semi-Supervised Semantic Segmentation

• Authors: Haoyu Xie, Changqi Wang, Jian Zhao, Yang Liu, Jun Dan, Chong Fu, Baigui Sun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18117
• Pdf URL: https://arxiv.org/pdf/2402.18117
• Copy Paste: [[2402.18117]] PRCL: Probabilistic Representation Contrastive Learning for Semi-Supervised Semantic Segmentation(https://arxiv.org/abs/2402.18117)
• Keywords: robust, segmentation
• Abstract: Tremendous breakthroughs have been developed in Semi-Supervised Semantic Segmentation (S4) through contrastive learning. However, due to limited annotations, the guidance on unlabeled images is generated by the model itself, which inevitably exists noise and disturbs the unsupervised training process. To address this issue, we propose a robust contrastive-based S4 framework, termed the Probabilistic Representation Contrastive Learning (PRCL) framework to enhance the robustness of the unsupervised training process. We model the pixel-wise representation as Probabilistic Representations (PR) via multivariate Gaussian distribution and tune the contribution of the ambiguous representations to tolerate the risk of inaccurate guidance in contrastive learning. Furthermore, we introduce Global Distribution Prototypes (GDP) by gathering all PRs throughout the whole training process. Since the GDP contains the information of all representations with the same class, it is robust from the instant noise in representations and bears the intra-class variance of representations. In addition, we generate Virtual Negatives (VNs) based on GDP to involve the contrastive learning process. Extensive experiments on two public benchmarks demonstrate the superiority of our PRCL framework.

Title: Exploring Multilingual Human Value Concepts in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?

• Authors: Shaoyang Xu, Weilong Dong, Zishan Guo, Xinwei Wu, Deyi Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18120
• Pdf URL: https://arxiv.org/pdf/2402.18120
• Copy Paste: [[2402.18120]] Exploring Multilingual Human Value Concepts in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?(https://arxiv.org/abs/2402.18120)
• Keywords: large language model
• Abstract: Prior research in representation engineering has revealed that LLMs encode concepts within their representation spaces, predominantly centered around English. In this study, we extend this philosophy to a multilingual scenario, delving into multilingual human value concepts in LLMs. Through our comprehensive exploration covering 7 types of human values, 16 languages and 3 LLM series with distinct multilinguality, we empirically substantiate the existence of multilingual human values in LLMs. Further cross-lingual analysis on these concepts discloses 3 traits arising from language resource disparities: cross-lingual inconsistency, distorted linguistic relationships, and unidirectional cross-lingual transfer between high- and low-resource languages, all in terms of human value concepts. Additionally, we validate the feasibility of cross-lingual control over value alignment capabilities of LLMs, leveraging the dominant language as a source language. Drawing from our findings on multilingual value alignment, we prudently provide suggestions on the composition of multilingual data for LLMs pre-training: including a limited number of dominant languages for cross-lingual alignment transfer while avoiding their excessive prevalence, and keeping a balanced distribution of non-dominant languages. We aspire that our findings would contribute to enhancing the safety and utility of multilingual AI.

Title: On the Inductive Biases of Demographic Parity-based Fair Learning Algorithms

• Authors: Haoyu Lei, Amin Gohari, Farzan Farnia
• Subjects: cs.LG, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/2402.18129
• Pdf URL: https://arxiv.org/pdf/2402.18129
• Copy Paste: [[2402.18129]] On the Inductive Biases of Demographic Parity-based Fair Learning Algorithms(https://arxiv.org/abs/2402.18129)
• Keywords: robust, fair
• Abstract: Fair supervised learning algorithms assigning labels with little dependence on a sensitive attribute have attracted great attention in the machine learning community. While the demographic parity (DP) notion has been frequently used to measure a model's fairness in training fair classifiers, several studies in the literature suggest potential impacts of enforcing DP in fair learning algorithms. In this work, we analytically study the effect of standard DP-based regularization methods on the conditional distribution of the predicted label given the sensitive attribute. Our analysis shows that an imbalanced training dataset with a non-uniform distribution of the sensitive attribute could lead to a classification rule biased toward the sensitive attribute outcome holding the majority of training data. To control such inductive biases in DP-based fair learning, we propose a sensitive attribute-based distributionally robust optimization (SA-DRO) method improving robustness against the marginal distribution of the sensitive attribute. Finally, we present several numerical results on the application of DP-based learning methods to standard centralized and distributed learning problems. The empirical findings support our theoretical results on the inductive biases in DP-based fair learning algorithms and the debiasing effects of the proposed SA-DRO method.

Title: Understanding the Role of Pathways in a Deep Neural Network

• Authors: Lei Lyu, Chen Pang, Jihua Wang
• Subjects: cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.18132
• Pdf URL: https://arxiv.org/pdf/2402.18132
• Copy Paste: [[2402.18132]] Understanding the Role of Pathways in a Deep Neural Network(https://arxiv.org/abs/2402.18132)
• Keywords: attack, diffusion
• Abstract: Deep neural networks have demonstrated superior performance in artificial intelligence applications, but the opaqueness of their inner working mechanism is one major drawback in their application. The prevailing unit-based interpretation is a statistical observation of stimulus-response data, which fails to show a detailed internal process of inherent mechanisms of neural networks. In this work, we analyze a convolutional neural network (CNN) trained in the classification task and present an algorithm to extract the diffusion pathways of individual pixels to identify the locations of pixels in an input image associated with object classes. The pathways allow us to test the causal components which are important for classification and the pathway-based representations are clearly distinguishable between categories. We find that the few largest pathways of an individual pixel from an image tend to cross the feature maps in each layer that is important for classification. And the large pathways of images of the same category are more consistent in their trends than those of different categories. We also apply the pathways to understanding adversarial attacks, object completion, and movement perception. Further, the total number of pathways on feature maps in all layers can clearly discriminate the original, deformed, and target samples.

Title: Classes Are Not Equal: An Empirical Study on Image Recognition Fairness

• Authors: Jiequan Cui, Beier Zhu, Xin Wen, Xiaojuan Qi, Bei Yu, Hanwang Zhang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18133
• Pdf URL: https://arxiv.org/pdf/2402.18133
• Copy Paste: [[2402.18133]] Classes Are Not Equal: An Empirical Study on Image Recognition Fairness(https://arxiv.org/abs/2402.18133)
• Keywords: fair
• Abstract: In this paper, we present an empirical study on image recognition fairness, i.e., extreme class accuracy disparity on balanced data like ImageNet. We experimentally demonstrate that classes are not equal and the fairness issue is prevalent for image classification models across various datasets, network architectures, and model capacities. Moreover, several intriguing properties of fairness are identified. First, the unfairness lies in problematic representation rather than classifier bias. Second, with the proposed concept of Model Prediction Bias, we investigate the origins of problematic representation during optimization. Our findings reveal that models tend to exhibit greater prediction biases for classes that are more challenging to recognize. It means that more other classes will be confused with harder classes. Then the False Positives (FPs) will dominate the learning in optimization, thus leading to their poor accuracy. Further, we conclude that data augmentation and representation learning algorithms improve overall performance by promoting fairness to some degree in image classification.

Title: Cause and Effect: Can Large Language Models Truly Understand Causality?

• Authors: Swagata Ashwani, Kshiteesh Hegde, Nishith Reddy Mannuru, Mayank Jindal, Dushyant Singh Sengar, Krishna Chaitanya Rao Kathala, Dishant Banga, Vinija Jain, Aman Chadha
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18139
• Pdf URL: https://arxiv.org/pdf/2402.18139
• Copy Paste: [[2402.18139]] Cause and Effect: Can Large Language Models Truly Understand Causality?(https://arxiv.org/abs/2402.18139)
• Keywords: interpretability, explainability, large language model
• Abstract: With the rise of Large Language Models(LLMs), it has become crucial to understand their capabilities and limitations in deciphering and explaining the complex web of causal relationships that language entails. Current methods use either explicit or implicit causal reasoning, yet there is a strong need for a unified approach combining both to tackle a wide array of causal relationships more effectively. This research proposes a novel architecture called Context Aware Reasoning Enhancement with Counterfactual Analysis(CARE CA) framework to enhance causal reasoning and explainability. The proposed framework incorporates an explicit causal detection module with ConceptNet and counterfactual statements, as well as implicit causal detection through LLMs. Our framework goes one step further with a layer of counterfactual explanations to accentuate LLMs understanding of causality. The knowledge from ConceptNet enhances the performance of multiple causal reasoning tasks such as causal discovery, causal identification and counterfactual reasoning. The counterfactual sentences add explicit knowledge of the not caused by scenarios. By combining these powerful modules, our model aims to provide a deeper understanding of causal relationships, enabling enhanced interpretability. Evaluation of benchmark datasets shows improved performance across all metrics, such as accuracy, precision, recall, and F1 scores. We also introduce CausalNet, a new dataset accompanied by our code, to facilitate further research in this domain.

Title: OccTransformer: Improving BEVFormer for 3D camera-only occupancy prediction

• Authors: Jian Liu, Sipeng Zhang, Chuixin Kong, Wenyuan Zhang, Yuhang Wu, Yikang Ding, Borun Xu, Ruibo Ming, Donglai Wei, Xianming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18140
• Pdf URL: https://arxiv.org/pdf/2402.18140
• Copy Paste: [[2402.18140]] OccTransformer: Improving BEVFormer for 3D camera-only occupancy prediction(https://arxiv.org/abs/2402.18140)
• Keywords: transformer
• Abstract: This technical report presents our solution, "occTransformer" for the 3D occupancy prediction track in the autonomous driving challenge at CVPR 2023. Our method builds upon the strong baseline BEVFormer and improves its performance through several simple yet effective techniques. Firstly, we employed data augmentation to increase the diversity of the training data and improve the model's generalization ability. Secondly, we used a strong image backbone to extract more informative features from the input data. Thirdly, we incorporated a 3D unet head to better capture the spatial information of the scene. Fourthly, we added more loss functions to better optimize the model. Additionally, we used an ensemble approach with the occ model BevDet and SurroundOcc to further improve the performance. Most importantly, we integrated 3D detection model StreamPETR to enhance the model's ability to detect objects in the scene. Using these methods, our solution achieved 49.23 miou on the 3D occupancy prediction track in the autonomous driving challenge.

Title: Learning Intrinsic Dimension via Information Bottleneck for Explainable Aspect-based Sentiment Analysis

• Authors: Zhenxiao Cheng, Jie Zhou, Wen Wu, Qin Chen, Liang He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18145
• Pdf URL: https://arxiv.org/pdf/2402.18145
• Copy Paste: [[2402.18145]] Learning Intrinsic Dimension via Information Bottleneck for Explainable Aspect-based Sentiment Analysis(https://arxiv.org/abs/2402.18145)
• Keywords: interpretability
• Abstract: Gradient-based explanation methods are increasingly used to interpret neural models in natural language processing (NLP) due to their high fidelity. Such methods determine word-level importance using dimension-level gradient values through a norm function, often presuming equal significance for all gradient dimensions. However, in the context of Aspect-based Sentiment Analysis (ABSA), our preliminary research suggests that only specific dimensions are pertinent. To address this, we propose the Information Bottleneck-based Gradient (\texttt{IBG}) explanation framework for ABSA. This framework leverages an information bottleneck to refine word embeddings into a concise intrinsic dimension, maintaining essential features and omitting unrelated information. Comprehensive tests show that our \texttt{IBG} approach considerably improves both the models' performance and interpretability by identifying sentiment-aware features.

Title: Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation

• Authors: Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, Jie Zhou
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.18150
• Pdf URL: https://arxiv.org/pdf/2402.18150
• Copy Paste: [[2402.18150]] Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation(https://arxiv.org/abs/2402.18150)
• Keywords: robust, large language model
• Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating additional information from retrieval. However, studies have shown that LLMs still face challenges in effectively using the retrieved information, even ignoring it or being misled by it. The key reason is that the training of LLMs does not clearly make LLMs learn how to utilize input retrieved texts with varied quality. In this paper, we propose a novel perspective that considers the role of LLMs in RAG as ``Information Refiner'', which means that regardless of correctness, completeness, or usefulness of retrieved texts, LLMs can consistently integrate knowledge within the retrieved texts and model parameters to generate the texts that are more concise, accurate, and complete than the retrieved texts. To this end, we propose an information refinement training method named InFO-RAG that optimizes LLMs for RAG in an unsupervised manner. InFO-RAG is low-cost and general across various tasks. Extensive experiments on zero-shot prediction of 11 datasets in diverse tasks including Question Answering, Slot-Filling, Language Modeling, Dialogue, and Code Generation show that InFO-RAG improves the performance of LLaMA2 by an average of 9.39\% relative points. InFO-RAG also shows advantages in in-context learning and robustness of RAG.

Title: Diffusion-based Neural Network Weights Generation

• Authors: Bedionita Soro, Bruno Andreis, Hayeon Lee, Song Chong, Frank Hutter, Sung Ju Hwang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18153
• Pdf URL: https://arxiv.org/pdf/2402.18153
• Copy Paste: [[2402.18153]] Diffusion-based Neural Network Weights Generation(https://arxiv.org/abs/2402.18153)
• Keywords: diffusion
• Abstract: Transfer learning is a topic of significant interest in recent deep learning research because it enables faster convergence and improved performance on new tasks. While the performance of transfer learning depends on the similarity of the source data to the target data, it is costly to train a model on a large number of datasets. Therefore, pretrained models are generally blindly selected with the hope that they will achieve good performance on the given task. To tackle such suboptimality of the pretrained models, we propose an efficient and adaptive transfer learning scheme through dataset-conditioned pretrained weights sampling. Specifically, we use a latent diffusion model with a variational autoencoder that can reconstruct the neural network weights, to learn the distribution of a set of pretrained weights conditioned on each dataset for transfer learning on unseen datasets. By learning the distribution of a neural network on a variety pretrained models, our approach enables adaptive sampling weights for unseen datasets achieving faster convergence and reaching competitive performance.

Title: Evaluating Quantized Large Language Models

• Authors: Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18158
• Pdf URL: https://arxiv.org/pdf/2402.18158
• Copy Paste: [[2402.18158]] Evaluating Quantized Large Language Models(https://arxiv.org/abs/2402.18158)
• Keywords: large language model
• Abstract: Post-training quantization (PTQ) has emerged as a promising technique to reduce the cost of large language models (LLMs). Specifically, PTQ can effectively mitigate memory consumption and reduce computational overhead in LLMs. To meet the requirements of both high efficiency and performance across diverse scenarios, a comprehensive evaluation of quantized LLMs is essential to guide the selection of quantization methods. This paper presents a thorough evaluation of these factors by evaluating the effect of PTQ on Weight, Activation, and KV Cache on 11 model families, including OPT, LLaMA2, Falcon, Bloomz, Mistral, ChatGLM, Vicuna, LongChat, StableLM, Gemma, and Mamba, with parameters ranging from 125M to 180B. The evaluation encompasses five types of tasks: basic NLP, emergent ability, trustworthiness, dialogue, and long-context tasks. Moreover, we also evaluate the state-of-the-art (SOTA) quantization methods to demonstrate their applicability. Based on the extensive experiments, we systematically summarize the effect of quantization, provide recommendations to apply quantization techniques, and point out future directions.

Title: Autoencoder-based General Purpose Representation Learning for Customer Embedding

• Authors: Jan Henrik Bertrand, Jacopo Pio Gargano, Laurent Mombaerts, Jonathan Taws
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18164
• Pdf URL: https://arxiv.org/pdf/2402.18164
• Copy Paste: [[2402.18164]] Autoencoder-based General Purpose Representation Learning for Customer Embedding(https://arxiv.org/abs/2402.18164)
• Keywords: generative
• Abstract: In recent years, exploiting the domain-specific underlying structure of data and its generative factors for representation learning has shown success in various use-case agnostic applications. However, the diversity and complexity of tabular data have made it challenging to represent these structures in a latent space through multi-dimensional vectors. We design an autoencoder-based framework for building general purpose embeddings, we assess the performance of different autoencoder architectures, and show simpler models outperform complex ones in embedding highly complex tabular data. We apply our framework to produce plug-and-play, rich, and anonymized embeddings representing AWS customers for usage in any model, saving up to 45% of development time, and observe significant improvements in downstream models. Moreover, we propose a significant improvement to the calculation of reconstruction loss for multi-layer contractive autoencoders (CAE) by calculating the Jacobian of the entire encoder leading to a 15% improvement in reconstruction quality when compared to a stacked CAE.

Title: Decentralised Traffic Incident Detection via Network Lasso

• Authors: Qiyuan Zhu, A. K. Qin, Prabath Abeysekara, Hussein Dia, Hanna Grzybowska
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18167
• Pdf URL: https://arxiv.org/pdf/2402.18167
• Copy Paste: [[2402.18167]] Decentralised Traffic Incident Detection via Network Lasso(https://arxiv.org/abs/2402.18167)
• Keywords: federate
• Abstract: Traffic incident detection plays a key role in intelligent transportation systems, which has gained great attention in transport engineering. In the past, traditional machine learning (ML) based detection methods achieved good performance under a centralised computing paradigm, where all data are transmitted to a central server for building ML models therein. Nowadays, deep neural networks based federated learning (FL) has become a mainstream detection approach to enable the model training in a decentralised manner while warranting local data governance. Such neural networks-centred techniques, however, have overshadowed the utility of well-established ML-based detection methods. In this work, we aim to explore the potential of potent conventional ML-based detection models in modern traffic scenarios featured by distributed data. We leverage an elegant but less explored distributed optimisation framework named Network Lasso, with guaranteed global convergence for convex problem formulations, integrate the potent convex ML model with it, and compare it with centralised learning, local learning, and federated learning methods atop a well-known traffic incident detection dataset. Experimental results show that the proposed network lasso-based approach provides a promising alternative to the FL-based approach in data-decentralised traffic scenarios, with a strong convergence guarantee while rekindling the significance of conventional ML-based detection methods.

Title: MIKO: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discovery

• Authors: Feihong Lu, Weiqi Wang, Yangyifei Luo, Ziqin Zhu, Qingyun Sun, Baixuan Xu, Haochen Shi, Shiqi Gao, Qian Li, Yangqiu Song, Jianxin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18169
• Pdf URL: https://arxiv.org/pdf/2402.18169
• Copy Paste: [[2402.18169]] MIKO: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discovery(https://arxiv.org/abs/2402.18169)
• Keywords: large language model
• Abstract: Social media has become a ubiquitous tool for connecting with others, staying updated with news, expressing opinions, and finding entertainment. However, understanding the intention behind social media posts remains challenging due to the implicitness of intentions in social media posts, the need for cross-modality understanding of both text and images, and the presence of noisy information such as hashtags, misspelled words, and complicated abbreviations. To address these challenges, we present MIKO, a Multimodal Intention Kowledge DistillatiOn framework that collaboratively leverages a Large Language Model (LLM) and a Multimodal Large Language Model (MLLM) to uncover users' intentions. Specifically, we use an MLLM to interpret the image and an LLM to extract key information from the text and finally instruct the LLM again to generate intentions. By applying MIKO to publicly available social media datasets, we construct an intention knowledge base featuring 1,372K intentions rooted in 137,287 posts. We conduct a two-stage annotation to verify the quality of the generated knowledge and benchmark the performance of widely used LLMs for intention generation. We further apply MIKO to a sarcasm detection dataset and distill a student model to demonstrate the downstream benefits of applying intention knowledge.

Title: Misalignment-Robust Frequency Distribution Loss for Image Transformation

• Authors: Zhangkai Ni, Juncheng Wu, Zian Wang, Wenhan Yang, Hanli Wang, Lin Ma
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.18192
• Pdf URL: https://arxiv.org/pdf/2402.18192
• Copy Paste: [[2402.18192]] Misalignment-Robust Frequency Distribution Loss for Image Transformation(https://arxiv.org/abs/2402.18192)
• Keywords: robust
• Abstract: This paper aims to address a common challenge in deep learning-based image transformation methods, such as image enhancement and super-resolution, which heavily rely on precisely aligned paired datasets with pixel-level alignments. However, creating precisely aligned paired images presents significant challenges and hinders the advancement of methods trained on such data. To overcome this challenge, this paper introduces a novel and simple Frequency Distribution Loss (FDL) for computing distribution distance within the frequency domain. Specifically, we transform image features into the frequency domain using Discrete Fourier Transformation (DFT). Subsequently, frequency components (amplitude and phase) are processed separately to form the FDL loss function. Our method is empirically proven effective as a training constraint due to the thoughtful utilization of global information in the frequency domain. Extensive experimental evaluations, focusing on image enhancement and super-resolution tasks, demonstrate that FDL outperforms existing misalignment-robust loss functions. Furthermore, we explore the potential of our FDL for image style transfer that relies solely on completely misaligned data. Our code is available at: https://github.com/eezkni/FDL

Title: NToP: NeRF-Powered Large-scale Dataset Generation for 2D and 3D Human Pose Estimation in Top-View Fisheye Images

• Authors: Jingrui Yu, Dipankar Nandi, Roman Seidel, Gangolf Hirtz
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.18196
• Pdf URL: https://arxiv.org/pdf/2402.18196
• Copy Paste: [[2402.18196]] NToP: NeRF-Powered Large-scale Dataset Generation for 2D and 3D Human Pose Estimation in Top-View Fisheye Images(https://arxiv.org/abs/2402.18196)
• Keywords: transformer
• Abstract: Human pose estimation (HPE) in the top-view using fisheye cameras presents a promising and innovative application domain. However, the availability of datasets capturing this viewpoint is extremely limited, especially those with high-quality 2D and 3D keypoint annotations. Addressing this gap, we leverage the capabilities of Neural Radiance Fields (NeRF) technique to establish a comprehensive pipeline for generating human pose datasets from existing 2D and 3D datasets, specifically tailored for the top-view fisheye perspective. Through this pipeline, we create a novel dataset NToP570K (NeRF-powered Top-view human Pose dataset for fisheye cameras with over 570 thousand images), and conduct an extensive evaluation of its efficacy in enhancing neural networks for 2D and 3D top-view human pose estimation. A pretrained ViTPose-B model achieves an improvement in AP of 33.3 % on our validation set for 2D HPE after finetuning on our training set. A similarly finetuned HybrIK-Transformer model gains 53.7 mm reduction in PA-MPJPE for 3D HPE on the validation set.

Title: Balancing Act: Distribution-Guided Debiasing in Diffusion Models

• Authors: Rishubh Parihar, Abhijnya Bhat, Saswat Mallick, Abhipsa Basu, Jogendra Nath Kundu, R. Venkatesh Babu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18206
• Pdf URL: https://arxiv.org/pdf/2402.18206
• Copy Paste: [[2402.18206]] Balancing Act: Distribution-Guided Debiasing in Diffusion Models(https://arxiv.org/abs/2402.18206)
• Keywords: fair, diffusion, generative
• Abstract: Diffusion Models (DMs) have emerged as powerful generative models with unprecedented image generation capability. These models are widely used for data augmentation and creative applications. However, DMs reflect the biases present in the training datasets. This is especially concerning in the context of faces, where the DM prefers one demographic subgroup vs others (eg. female vs male). In this work, we present a method for debiasing DMs without relying on additional data or model retraining. Specifically, we propose Distribution Guidance, which enforces the generated images to follow the prescribed attribute distribution. To realize this, we build on the key insight that the latent features of denoising UNet hold rich demographic semantics, and the same can be leveraged to guide debiased generation. We train Attribute Distribution Predictor (ADP) - a small mlp that maps the latent features to the distribution of attributes. ADP is trained with pseudo labels generated from existing attribute classifiers. The proposed Distribution Guidance with ADP enables us to do fair generation. Our method reduces bias across single/multiple attributes and outperforms the baseline by a significant margin for unconditional and text-conditional diffusion models. Further, we present a downstream task of training a fair attribute classifier by rebalancing the training set with our generated data.

Title: Catastrophic Overfitting: A Potential Blessing in Disguise

• Authors: Mengnan Zhao, Lihe Zhang, Yuqiu Kong, Baocai Yin
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18211
• Pdf URL: https://arxiv.org/pdf/2402.18211
• Copy Paste: [[2402.18211]] Catastrophic Overfitting: A Potential Blessing in Disguise(https://arxiv.org/abs/2402.18211)
• Keywords: attack, robust
• Abstract: Fast Adversarial Training (FAT) has gained increasing attention within the research community owing to its efficacy in improving adversarial robustness. Particularly noteworthy is the challenge posed by catastrophic overfitting (CO) in this field. Although existing FAT approaches have made strides in mitigating CO, the ascent of adversarial robustness occurs with a non-negligible decline in classification accuracy on clean samples. To tackle this issue, we initially employ the feature activation differences between clean and adversarial examples to analyze the underlying causes of CO. Intriguingly, our findings reveal that CO can be attributed to the feature coverage induced by a few specific pathways. By intentionally manipulating feature activation differences in these pathways with well-designed regularization terms, we can effectively mitigate and induce CO, providing further evidence for this observation. Notably, models trained stably with these terms exhibit superior performance compared to prior FAT work. On this basis, we harness CO to achieve `attack obfuscation', aiming to bolster model performance. Consequently, the models suffering from CO can attain optimal classification accuracy on both clean and adversarial data when adding random noise to inputs during evaluation. We also validate their robustness against transferred adversarial examples and the necessity of inducing CO to improve robustness. Hence, CO may not be a problem that has to be solved.

Title: Multi-objective Differentiable Neural Architecture Search

• Authors: Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Samuel Dooley, Josif Grabocka, Frank Hutter
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.18213
• Pdf URL: https://arxiv.org/pdf/2402.18213
• Copy Paste: [[2402.18213]] Multi-objective Differentiable Neural Architecture Search(https://arxiv.org/abs/2402.18213)
• Keywords: transformer
• Abstract: Pareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives like neural network training. Typically, in MOO neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences for the trade-off between performance and hardware metrics, and yields representative and diverse architectures across multiple devices in just one search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments with up to 19 hardware devices and 3 objectives showcase the effectiveness and scalability of our method. Finally, we show that, without additional costs, our method outperforms existing MOO NAS methods across qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k and a Transformer space on machine translation.

Title: LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History

• Authors: Akash Gupta, Ivaxi Sheth, Vyas Raina, Mark Gales, Mario Fritz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18216
• Pdf URL: https://arxiv.org/pdf/2402.18216
• Copy Paste: [[2402.18216]] LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History(https://arxiv.org/abs/2402.18216)
• Keywords: large language model
• Abstract: With the recent emergence of powerful instruction-tuned large language models (LLMs), various helpful conversational Artificial Intelligence (AI) systems have been deployed across many applications. When prompted by users, these AI systems successfully perform a wide range of tasks as part of a conversation. To provide some sort of memory and context, such approaches typically condition their output on the entire conversational history. Although this sensitivity to the conversational history can often lead to improved performance on subsequent tasks, we find that performance can in fact also be negatively impacted, if there is a task-switch. To the best of our knowledge, our work makes the first attempt to formalize the study of such vulnerabilities and interference of tasks in conversational LLMs caused by task-switches in the conversational history. Our experiments across 5 datasets with 15 task switches using popular LLMs reveal that many of the task-switches can lead to significant performance degradation.

Title: CogBench: a large language model walks into a psychology lab

• Authors: Julian Coda-Forno, Marcel Binz, Jane X. Wang, Eric Schulz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18225
• Pdf URL: https://arxiv.org/pdf/2402.18225
• Copy Paste: [[2402.18225]] CogBench: a large language model walks into a psychology lab(https://arxiv.org/abs/2402.18225)
• Keywords: large language model
• Abstract: Large language models (LLMs) have significantly advanced the field of artificial intelligence. Yet, evaluating them comprehensively remains challenging. We argue that this is partly due to the predominant focus on performance metrics in most benchmarks. This paper introduces CogBench, a benchmark that includes ten behavioral metrics derived from seven cognitive psychology experiments. This novel approach offers a toolkit for phenotyping LLMs' behavior. We apply CogBench to 35 LLMs, yielding a rich and diverse dataset. We analyze this data using statistical multilevel modeling techniques, accounting for the nested dependencies among fine-tuned versions of specific LLMs. Our study highlights the crucial role of model size and reinforcement learning from human feedback (RLHF) in improving performance and aligning with human behavior. Interestingly, we find that open-source models are less risk-prone than proprietary models and that fine-tuning on code does not necessarily enhance LLMs' behavior. Finally, we explore the effects of prompt-engineering techniques. We discover that chain-of-thought prompting improves probabilistic reasoning, while take-a-step-back prompting fosters model-based behaviors.

Title: Zero-Shot Aerial Object Detection with Visual Description Regularization

• Authors: Zhengqing Zang, Chenyu Lin, Chenwei Tang, Tao Wang, Jiancheng Lv
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18233
• Pdf URL: https://arxiv.org/pdf/2402.18233
• Copy Paste: [[2402.18233]] Zero-Shot Aerial Object Detection with Visual Description Regularization(https://arxiv.org/abs/2402.18233)
• Keywords: generative
• Abstract: Existing object detection models are mainly trained on large-scale labeled datasets. However, annotating data for novel aerial object classes is expensive since it is time-consuming and may require expert knowledge. Thus, it is desirable to study label-efficient object detection methods on aerial images. In this work, we propose a zero-shot method for aerial object detection named visual Description Regularization, or DescReg. Concretely, we identify the weak semantic-visual correlation of the aerial objects and aim to address the challenge with prior descriptions of their visual appearance. Instead of directly encoding the descriptions into class embedding space which suffers from the representation gap problem, we propose to infuse the prior inter-class visual similarity conveyed in the descriptions into the embedding learning. The infusion process is accomplished with a newly designed similarity-aware triplet loss which incorporates structured regularization on the representation space. We conduct extensive experiments with three challenging aerial object detection datasets, including DIOR, xView, and DOTA. The results demonstrate that DescReg significantly outperforms the state-of-the-art ZSD methods with complex projection designs and generative frameworks, e.g., DescReg outperforms best reported ZSD method on DIOR by 4.5 mAP on unseen classes and 8.1 in HM. We further show the generalizability of DescReg by integrating it into generative ZSD methods as well as varying the detection architecture.

Title: Image2Flow: A hybrid image and graph convolutional neural network for rapid patient-specific pulmonary artery segmentation and CFD flow field calculation from 3D cardiac MRI data

• Authors: Tina Yao, Endrit Pajaziti, Michael Quail, Silvia Schievano, Jennifer A Steeden, Vivek Muthurangu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18236
• Pdf URL: https://arxiv.org/pdf/2402.18236
• Copy Paste: [[2402.18236]] Image2Flow: A hybrid image and graph convolutional neural network for rapid patient-specific pulmonary artery segmentation and CFD flow field calculation from 3D cardiac MRI data(https://arxiv.org/abs/2402.18236)
• Keywords: segmentation
• Abstract: Computational fluid dynamics (CFD) can be used for evaluation of hemodynamics. However, its routine use is limited by labor-intensive manual segmentation, CFD mesh creation, and time-consuming simulation. This study aims to train a deep learning model to both generate patient-specific volume-meshes of the pulmonary artery from 3D cardiac MRI data and directly estimate CFD flow fields. This study used 135 3D cardiac MRIs from both a public and private dataset. The pulmonary arteries in the MRIs were manually segmented and converted into volume-meshes. CFD simulations were performed on ground truth meshes and interpolated onto point-point correspondent meshes to create the ground truth dataset. The dataset was split 85/10/15 for training, validation and testing. Image2Flow, a hybrid image and graph convolutional neural network, was trained to transform a pulmonary artery template to patient-specific anatomy and CFD values. Image2Flow was evaluated in terms of segmentation and accuracy of CFD predicted was assessed using node-wise comparisons. Centerline comparisons of Image2Flow and CFD simulations performed using machine learning segmentation were also performed. Image2Flow achieved excellent segmentation accuracy with a median Dice score of 0.9 (IQR: 0.86-0.92). The median node-wise normalized absolute error for pressure and velocity magnitude was 11.98% (IQR: 9.44-17.90%) and 8.06% (IQR: 7.54-10.41), respectively. Centerline analysis showed no significant difference between the Image2Flow and conventional CFD simulated on machine learning-generated volume-meshes. This proof-of-concept study has shown it is possible to simultaneously perform patient specific volume-mesh based segmentation and pressure and flow field estimation. Image2Flow completes segmentation and CFD in ~205ms, which ~7000 times faster than manual methods, making it more feasible in a clinical environment.

Title: Learning or Self-aligning? Rethinking Instruction Fine-tuning

• Authors: Mengjie Ren, Boxi Cao, Hongyu Lin, Liu Cao, Xianpei Han, Ke Zeng, Guanglu Wan, Xunliang Cai, Le Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18243
• Pdf URL: https://arxiv.org/pdf/2402.18243
• Copy Paste: [[2402.18243]] Learning or Self-aligning? Rethinking Instruction Fine-tuning(https://arxiv.org/abs/2402.18243)
• Keywords: robust, large language model
• Abstract: Instruction Fine-tuning~(IFT) is a critical phase in building large language models~(LLMs). Previous works mainly focus on the IFT's role in the transfer of behavioral norms and the learning of additional world knowledge. However, the understanding of the underlying mechanisms of IFT remains significantly limited. In this paper, we design a knowledge intervention framework to decouple the potential underlying factors of IFT, thereby enabling individual analysis of different factors. Surprisingly, our experiments reveal that attempting to learn additional world knowledge through IFT often struggles to yield positive impacts and can even lead to markedly negative effects. Further, we discover that maintaining internal knowledge consistency before and after IFT is a critical factor for achieving successful IFT. Our findings reveal the underlying mechanisms of IFT and provide robust support for some very recent and potential future works.

Title: On the Accuracy of Edge Detectors in Number Plate Extraction

• Authors: Bashir Olaniyi Sadiq
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18251
• Pdf URL: https://arxiv.org/pdf/2402.18251
• Copy Paste: [[2402.18251]] On the Accuracy of Edge Detectors in Number Plate Extraction(https://arxiv.org/abs/2402.18251)
• Keywords: extraction
• Abstract: Edge detection as a pre-processing stage is a fundamental and important aspect of the number plate extraction system. This is due to the fact that the identification of a particular vehicle is achievable using the number plate because each number plate is unique to a vehicle. As such, the characters of a number plate system that differ in lines and shapes can be extracted using the principle of edge detection. This paper presents a method of number plate extraction using edge detection technique. Edges in number plates are identified with changes in the intensity of pixel values. Therefore, these edges are identified using a single based pixel or collection of pixel-based approach. The efficiency of these approaches of edge detection algorithms in number plate extraction in both noisy and clean environment are experimented. Experimental results are achieved in MATLAB 2017b using the Pratt Figure of Merit (PFOM) as a performance metric

Title: Towards Generalist Prompting for Large Language Models by Mental Models

• Authors: Haoxiang Guan, Jiyan He, Shuxin Zheng, En-Hong Chen, Weiming Zhang, Nenghai Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18252
• Pdf URL: https://arxiv.org/pdf/2402.18252
• Copy Paste: [[2402.18252]] Towards Generalist Prompting for Large Language Models by Mental Models(https://arxiv.org/abs/2402.18252)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated impressive performance on many tasks. However, to achieve optimal performance, specially designed prompting methods are still needed. These methods either rely on task-specific few-shot examples that require a certain level of domain knowledge, or are designed to be simple but only perform well on a few types of tasks. In this work, we attempt to introduce the concept of generalist prompting, which operates on the design principle of achieving optimal or near-optimal performance on a wide range of tasks while eliminating the need for manual selection and customization of prompts tailored to specific problems. Furthermore, we propose MeMo (Mental Models), an innovative prompting method that is simple-designed yet effectively fulfills the criteria of generalist prompting. MeMo distills the cores of various prompting methods into individual mental models and allows LLMs to autonomously select the most suitable mental models for the problem, achieving or being near to the state-of-the-art results on diverse tasks such as STEM, logical reasoning, and commonsense reasoning in zero-shot settings. We hope that the insights presented herein will stimulate further exploration of generalist prompting methods for LLMs.

Title: Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding

• Authors: Hongshen Xu, Lu Chen, Zihan Zhao, Da Ma, Ruisheng Cao, Zichen Zhu, Kai Yu
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18262
• Pdf URL: https://arxiv.org/pdf/2402.18262
• Copy Paste: [[2402.18262]] Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding(https://arxiv.org/abs/2402.18262)
• Keywords: extraction
• Abstract: The growing prevalence of visually rich documents, such as webpages and scanned/digital-born documents (images, PDFs, etc.), has led to increased interest in automatic document understanding and information extraction across academia and industry. Although various document modalities, including image, text, layout, and structure, facilitate human information retrieval, the interconnected nature of these modalities presents challenges for neural networks. In this paper, we introduce WebLM, a multimodal pre-training network designed to address the limitations of solely modeling text and structure modalities of HTML in webpages. Instead of processing document images as unified natural images, WebLM integrates the hierarchical structure of document images to enhance the understanding of markup-language-based documents. Additionally, we propose several pre-training tasks to model the interaction among text, structure, and image modalities effectively. Empirical results demonstrate that the pre-trained WebLM significantly surpasses previous state-of-the-art pre-trained models across several webpage understanding tasks. The pre-trained models and code are available at https://github.com/X-LANCE/weblm.

Title: Retrieval-based Full-length Wikipedia Generation for Emergent Events

• Authors: Jiebin Zhang, Eugene J. Yu, Qinyu Chen, Chenhao Xiong, Dawei Zhu, Han Qian, Mingbo Song, Xiaoguang Li, Qun Liu, Sujian Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18264
• Pdf URL: https://arxiv.org/pdf/2402.18264
• Copy Paste: [[2402.18264]] Retrieval-based Full-length Wikipedia Generation for Emergent Events(https://arxiv.org/abs/2402.18264)
• Keywords: large language model
• Abstract: In today's fast-paced world, the growing demand to quickly generate comprehensive and accurate Wikipedia documents for emerging events is both crucial and challenging. However, previous efforts in Wikipedia generation have often fallen short of meeting real-world requirements. Some approaches focus solely on generating segments of a complete Wikipedia document, while others overlook the importance of faithfulness in generation or fail to consider the influence of the pre-training corpus. In this paper, we simulate a real-world scenario where structured full-length Wikipedia documents are generated for emergent events using input retrieved from web sources. To ensure that Large Language Models (LLMs) are not trained on corpora related to recently occurred events, we select events that have taken place recently and introduce a new benchmark Wiki-GenBen, which consists of 309 events paired with their corresponding retrieved web pages for generating evidence. Additionally, we design a comprehensive set of systematic evaluation metrics and baseline methods, to evaluate the capability of LLMs in generating factual full-length Wikipedia documents. The data and code are open-sourced at WikiGenBench.

Title: Is Crowdsourcing Breaking Your Bank? Cost-Effective Fine-Tuning of Pre-trained Language Models with Proximal Policy Optimization

• Authors: Shuo Yang, Gjergji Kasneci
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18284
• Pdf URL: https://arxiv.org/pdf/2402.18284
• Copy Paste: [[2402.18284]] Is Crowdsourcing Breaking Your Bank? Cost-Effective Fine-Tuning of Pre-trained Language Models with Proximal Policy Optimization(https://arxiv.org/abs/2402.18284)
• Keywords: generative
• Abstract: Wide usage of ChatGPT has highlighted the potential of reinforcement learning from human feedback. However, its training pipeline relies on manual ranking, a resource-intensive process. To reduce labor costs, we propose a self-supervised text ranking approach for applying Proximal-Policy-Optimization to fine-tune language models while eliminating the need for human annotators. Our method begins with probabilistic sampling to encourage a language model to generate diverse responses for each input. We then employ TextRank and ISODATA algorithms to rank and cluster these responses based on their semantics. Subsequently, we construct a reward model to learn the rank and optimize our generative policy. Our experimental results, conducted using two language models on three tasks, demonstrate that the models trained by our method considerably outperform baselines regarding BLEU, GLEU, and METEOR scores. Furthermore, our manual evaluation shows that our ranking results exhibit a remarkably high consistency with that of humans. This research significantly reduces training costs of proximal policy-guided models and demonstrates the potential for self-correction of language models.

Title: Self-Supervised Learning in Electron Microscopy: Towards a Foundation Model for Advanced Image Analysis

• Authors: Bashir Kazimi, Karina Ruzaeva, Stefan Sandfeld
• Subjects: cs.CV, cond-mat.mtrl-sci, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18286
• Pdf URL: https://arxiv.org/pdf/2402.18286
• Copy Paste: [[2402.18286]] Self-Supervised Learning in Electron Microscopy: Towards a Foundation Model for Advanced Image Analysis(https://arxiv.org/abs/2402.18286)
• Keywords: segmentation
• Abstract: In this work, we explore the potential of self-supervised learning from unlabeled electron microscopy datasets, taking a step toward building a foundation model in this field. We show how self-supervised pretraining facilitates efficient fine-tuning for a spectrum of downstream tasks, including semantic segmentation, denoising, noise & background removal, and super-resolution. Experimentation with varying model complexities and receptive field sizes reveals the remarkable phenomenon that fine-tuned models of lower complexity consistently outperform more complex models with random weight initialization. We demonstrate the versatility of self-supervised pretraining across various downstream tasks in the context of electron microscopy, allowing faster convergence and better performance. We conclude that self-supervised pretraining serves as a powerful catalyst, being especially advantageous when limited annotated data are available and efficient scaling of computational cost are important.

Title: Windowed-FourierMixer: Enhancing Clutter-Free Room Modeling with Fourier Transform

• Authors: Bruno Henriques, Benjamin Allaert, Jean-Philippe Vandeborre
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18287
• Pdf URL: https://arxiv.org/pdf/2402.18287
• Copy Paste: [[2402.18287]] Windowed-FourierMixer: Enhancing Clutter-Free Room Modeling with Fourier Transform(https://arxiv.org/abs/2402.18287)
• Keywords: generative
• Abstract: With the growing demand for immersive digital applications, the need to understand and reconstruct 3D scenes has significantly increased. In this context, inpainting indoor environments from a single image plays a crucial role in modeling the internal structure of interior spaces as it enables the creation of textured and clutter-free reconstructions. While recent methods have shown significant progress in room modeling, they rely on constraining layout estimators to guide the reconstruction process. These methods are highly dependent on the performance of the structure estimator and its generative ability in heavily occluded environments. In response to these issues, we propose an innovative approach based on a U-Former architecture and a new Windowed-FourierMixer block, resulting in a unified, single-phase network capable of effectively handle human-made periodic structures such as indoor spaces. This new architecture proves advantageous for tasks involving indoor scenes where symmetry is prevalent, allowing the model to effectively capture features such as horizon/ceiling height lines and cuboid-shaped rooms. Experiments show the proposed approach outperforms current state-of-the-art methods on the Structured3D dataset demonstrating superior performance in both quantitative metrics and qualitative results. Code and models will be made publicly available.

Title: Comparative Analysis of XGBoost and Minirocket Algortihms for Human Activity Recognition

• Authors: Celal Alagoz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18296
• Pdf URL: https://arxiv.org/pdf/2402.18296
• Copy Paste: [[2402.18296]] Comparative Analysis of XGBoost and Minirocket Algortihms for Human Activity Recognition(https://arxiv.org/abs/2402.18296)
• Keywords: robust, extraction
• Abstract: Human Activity Recognition (HAR) has been extensively studied, with recent emphasis on the implementation of advanced Machine Learning (ML) and Deep Learning (DL) algorithms for accurate classification. This study investigates the efficacy of two ML algorithms, eXtreme Gradient Boosting (XGBoost) and MiniRocket, in the realm of HAR using data collected from smartphone sensors. The experiments are conducted on a dataset obtained from the UCI repository, comprising accelerometer and gyroscope signals captured from 30 volunteers performing various activities while wearing a smartphone. The dataset undergoes preprocessing, including noise filtering and feature extraction, before being utilized for training and testing the classifiers. Monte Carlo cross-validation is employed to evaluate the models' robustness. The findings reveal that both XGBoost and MiniRocket attain accuracy, F1 score, and AUC values as high as 0.99 in activity classification. XGBoost exhibits a slightly superior performance compared to MiniRocket. Notably, both algorithms surpass the performance of other ML and DL algorithms reported in the literature for HAR tasks. Additionally, the study compares the computational efficiency of the two algorithms, revealing XGBoost's advantage in terms of training time. Furthermore, the performance of MiniRocket, which achieves accuracy and F1 values of 0.94, and an AUC value of 0.96 using raw data and utilizing only one channel from the sensors, highlights the potential of directly leveraging unprocessed signals. It also suggests potential advantages that could be gained by utilizing sensor fusion or channel fusion techniques. Overall, this research sheds light on the effectiveness and computational characteristics of XGBoost and MiniRocket in HAR tasks, providing insights for future studies in activity recognition using smartphone sensor data.

Title: Quantification and Modeling of Broken Links Prevalence in Hyper Traffic Websites Homepages

• Authors: Ronan Mouchoux, Laurent Moulin, Nicolas Striebig
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18301
• Pdf URL: https://arxiv.org/pdf/2402.18301
• Copy Paste: [[2402.18301]] Quantification and Modeling of Broken Links Prevalence in Hyper Traffic Websites Homepages(https://arxiv.org/abs/2402.18301)
• Keywords: security, privacy
• Abstract: Broken links in websites external resources pose a serious threat to cybersecurity and the credibility of websites. They can be hijacked to eavesdrop user traffic or to inject malicious software. In this paper, we present the first result of an ongoing research. We focus on the prevalence of broken links in external resources on home pages of the most visited websites in the world. The analysis was conducted on the top 88 000 homepages extracted from the Majestic Million rankings. 35,2% of them have at least one broken link. We also identify the common causes of these broken links and highlight improper implementation of testing phases to prevent such errors. We provide a formal model for the distribution of external links. At the next research step, we are exploring the potential impact on privacy of broken links by analyzing inherited traffic of purchasable expired domains.

Title: EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving

• Authors: Jiacheng Lin, Jiajun Chen, Kunyu Peng, Xuan He, Zhiyong Li, Rainer Stiefelhagen, Kailun Yang
• Subjects: cs.CV, cs.RO, eess.AS, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.18302
• Pdf URL: https://arxiv.org/pdf/2402.18302
• Copy Paste: [[2402.18302]] EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving(https://arxiv.org/abs/2402.18302)
• Keywords: transformer
• Abstract: This paper introduces the task of Auditory Referring Multi-Object Tracking (AR-MOT), which dynamically tracks specific objects in a video sequence based on audio expressions and appears as a challenging problem in autonomous driving. Due to the lack of semantic modeling capacity in audio and video, existing works have mainly focused on text-based multi-object tracking, which often comes at the cost of tracking quality, interaction efficiency, and even the safety of assistance systems, limiting the application of such methods in autonomous driving. In this paper, we delve into the problem of AR-MOT from the perspective of audio-video fusion and audio-video tracking. We put forward EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers. The dual streams are intertwined with our Bidirectional Frequency-domain Cross-attention Fusion Module (Bi-FCFM), which bidirectionally fuses audio and video features from both frequency- and spatiotemporal domains. Moreover, we propose the Audio-visual Contrastive Tracking Learning (ACTL) regime to extract homogeneous semantic features between expressions and visual objects by learning homogeneous features between different audio and video objects effectively. Aside from the architectural design, we establish the first set of large-scale AR-MOT benchmarks, including Echo-KITTI, Echo-KITTI+, and Echo-BDD. Extensive experiments on the established benchmarks demonstrate the effectiveness of the proposed EchoTrack model and its components. The source code and datasets will be made publicly available at https://github.com/lab206/EchoTrack.

Title: Feature Denoising For Low-Light Instance Segmentation Using Weighted Non-Local Blocks

• Authors: Joanne Lin, Nantheera Anantrasirichai, David Bull
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18307
• Pdf URL: https://arxiv.org/pdf/2402.18307
• Copy Paste: [[2402.18307]] Feature Denoising For Low-Light Instance Segmentation Using Weighted Non-Local Blocks(https://arxiv.org/abs/2402.18307)
• Keywords: segmentation
• Abstract: Instance segmentation for low-light imagery remains largely unexplored due to the challenges imposed by such conditions, for example shot noise due to low photon count, color distortions and reduced contrast. In this paper, we propose an end-to-end solution to address this challenging task. Based on Mask R-CNN, our proposed method implements weighted non-local (NL) blocks in the feature extractor. This integration enables an inherent denoising process at the feature level. As a result, our method eliminates the need for aligned ground truth images during training, thus supporting training on real-world low-light datasets. We introduce additional learnable weights at each layer in order to enhance the network's adaptability to real-world noise characteristics, which affect different feature scales in different ways. Experimental results show that the proposed method outperforms the pretrained Mask R-CNN with an Average Precision (AP) improvement of +10.0, with the introduction of weighted NL Blocks further enhancing AP by +1.0.

Title: Enhancing Roadway Safety: LiDAR-based Tree Clearance Analysis

• Authors: Miriam Louise Carnot, Eric Peukert, Bogdan Franczyk
• Subjects: cs.CV, cs.AI, cs.CG
• Abstract URL: https://arxiv.org/abs/2402.18309
• Pdf URL: https://arxiv.org/pdf/2402.18309
• Copy Paste: [[2402.18309]] Enhancing Roadway Safety: LiDAR-based Tree Clearance Analysis(https://arxiv.org/abs/2402.18309)
• Keywords: segmentation
• Abstract: In the efforts for safer roads, ensuring adequate vertical clearance above roadways is of great importance. Frequently, trees or other vegetation is growing above the roads, blocking the sight of traffic signs and lights and posing danger to traffic participants. Accurately estimating this space from simple images proves challenging due to a lack of depth information. This is where LiDAR technology comes into play, a laser scanning sensor that reveals a three-dimensional perspective. Thus far, LiDAR point clouds at the street level have mainly been used for applications in the field of autonomous driving. These scans, however, also open up possibilities in urban management. In this paper, we present a new point cloud algorithm that can automatically detect those parts of the trees that grow over the street and need to be trimmed. Our system uses semantic segmentation to filter relevant points and downstream processing steps to create the required volume to be kept clear above the road. Challenges include obscured stretches of road, the noisy unstructured nature of LiDAR point clouds, and the assessment of the road shape. The identified points of non-compliant trees can be projected from the point cloud onto images, providing municipalities with a visual aid for dealing with such occurrences. By automating this process, municipalities can address potential road space constraints, enhancing safety for all. They may also save valuable time by carrying out the inspections more systematically. Our open-source code gives communities inspiration on how to automate the process themselves.

Title: How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning

• Authors: Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, Tanmoy Chakraborty
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18312
• Pdf URL: https://arxiv.org/pdf/2402.18312
• Copy Paste: [[2402.18312]] How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning(https://arxiv.org/abs/2402.18312)
• Keywords: large language model
• Abstract: Despite superior reasoning prowess demonstrated by Large Language Models (LLMs) with Chain-of-Thought (CoT) prompting, a lack of understanding prevails around the internal mechanisms of the models that facilitate CoT generation. This work investigates the neural sub-structures within LLMs that manifest CoT reasoning from a mechanistic point of view. From an analysis of LLaMA-2 7B applied to multistep reasoning over fictional ontologies, we demonstrate that LLMs deploy multiple parallel pathways of answer generation for step-by-step reasoning. These parallel pathways provide sequential answers from the input question context as well as the generated CoT. We observe a striking functional rift in the middle layers of the LLM. Token representations in the initial half remain strongly biased towards the pretraining prior, with the in-context taking over abruptly in the later half. This internal phase shift manifests in different functional components: attention heads that write the answer token predominantly appear in the later half, attention heads that move information along ontological relationships appear exclusively in the initial half, and so on. To the best of our knowledge, this is the first attempt towards mechanistic investigation of CoT reasoning in LLMs.

Title: Living-off-The-Land Reverse-Shell Detection by Informed Data Augmentation

• Authors: Dmitrijs Trizna, Luca Demetrio, Battista Biggio, Fabio Roli
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18329
• Pdf URL: https://arxiv.org/pdf/2402.18329
• Copy Paste: [[2402.18329]] Living-off-The-Land Reverse-Shell Detection by Informed Data Augmentation(https://arxiv.org/abs/2402.18329)
• Keywords: attack, robust
• Abstract: The living-off-the-land (LOTL) offensive methodologies rely on the perpetration of malicious actions through chains of commands executed by legitimate applications, identifiable exclusively by analysis of system logs. LOTL techniques are well hidden inside the stream of events generated by common legitimate activities, moreover threat actors often camouflage activity through obfuscation, making them particularly difficult to detect without incurring in plenty of false alarms, even using machine learning. To improve the performance of models in such an harsh environment, we propose an augmentation framework to enhance and diversify the presence of LOTL malicious activity inside legitimate logs. Guided by threat intelligence, we generate a dataset by injecting attack templates known to be employed in the wild, further enriched by malleable patterns of legitimate activities to replicate the behavior of evasive threat actors. We conduct an extensive ablation study to understand which models better handle our augmented dataset, also manipulated to mimic the presence of model-agnostic evasion and poisoning attacks. Our results suggest that augmentation is needed to maintain high-predictive capabilities, robustness to attack is achieved through specific hardening techniques like adversarial training, and it is possible to deploy near-real-time models with almost-zero false alarms.

Title: FineDiffusion: Scaling up Diffusion Models for Fine-grained Image Generation with 10,000 Classes

• Authors: Ziying Pan, Kun Wang, Gang Li, Feihong He, Xiwang Li, Yongxuan Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18331
• Pdf URL: https://arxiv.org/pdf/2402.18331
• Copy Paste: [[2402.18331]] FineDiffusion: Scaling up Diffusion Models for Fine-grained Image Generation with 10,000 Classes(https://arxiv.org/abs/2402.18331)
• Keywords: diffusion
• Abstract: The class-conditional image generation based on diffusion models is renowned for generating high-quality and diverse images. However, most prior efforts focus on generating images for general categories, e.g., 1000 classes in ImageNet-1k. A more challenging task, large-scale fine-grained image generation, remains the boundary to explore. In this work, we present a parameter-efficient strategy, called FineDiffusion, to fine-tune large pre-trained diffusion models scaling to large-scale fine-grained image generation with 10,000 categories. FineDiffusion significantly accelerates training and reduces storage overhead by only fine-tuning tiered class embedder, bias terms, and normalization layers' parameters. To further improve the image generation quality of fine-grained categories, we propose a novel sampling method for fine-grained image generation, which utilizes superclass-conditioned guidance, specifically tailored for fine-grained categories, to replace the conventional classifier-free guidance sampling. Compared to full fine-tuning, FineDiffusion achieves a remarkable 1.56x training speed-up and requires storing merely 1.77% of the total model parameters, while achieving state-of-the-art FID of 9.776 on image generation of 10,000 classes. Extensive qualitative and quantitative experiments demonstrate the superiority of our method compared to other parameter-efficient fine-tuning methods. The code and more generated results are available at our project website: https://finediffusion.github.io/.

Title: Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation

• Authors: Nihal V. Nayak, Yiyang Nan, Avi Trost, Stephen H. Bach
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18334
• Pdf URL: https://arxiv.org/pdf/2402.18334
• Copy Paste: [[2402.18334]] Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation(https://arxiv.org/abs/2402.18334)
• Keywords: large language model
• Abstract: We introduce Bonito, an open-source model for conditional task generation: the task of converting unannotated text into task-specific training datasets for instruction tuning. Our goal is to enable zero-shot task adaptation of large language models on users' specialized, private data. We train Bonito on a new large-scale dataset with 1.65M examples created by remixing existing instruction tuning datasets into meta-templates. The meta-templates for a dataset produce training examples where the input is the unannotated text and the task attribute and the output consists of the instruction and the response. We use Bonito to generate synthetic tasks for seven datasets from specialized domains across three task types -- yes-no question answering, extractive question answering, and natural language inference -- and adapt language models. We show that Bonito significantly improves the average performance of pretrained and instruction tuned models over the de facto self supervised baseline. For example, adapting Mistral-Instruct-v2 and instruction tuned variants of Mistral and Llama2 with Bonito improves the strong zero-shot performance by 22.1 F1 points whereas the next word prediction objective undoes some of the benefits of instruction tuning and reduces the average performance by 0.8 F1 points. We conduct additional experiments with Bonito to understand the effects of the domain, the size of the training set, and the choice of alternative synthetic task generators. Overall, we show that learning with synthetic instruction tuning datasets is an effective way to adapt language models to new domains. The model, dataset, and code are available at https://github.com/BatsResearch/bonito.

Title: Probabilistic Bayesian optimal experimental design using conditional normalizing flows

• Authors: Rafael Orozco, Felix J. Herrmann, Peng Chen
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18337
• Pdf URL: https://arxiv.org/pdf/2402.18337
• Copy Paste: [[2402.18337]] Probabilistic Bayesian optimal experimental design using conditional normalizing flows(https://arxiv.org/abs/2402.18337)
• Keywords: robust
• Abstract: Bayesian optimal experimental design (OED) seeks to conduct the most informative experiment under budget constraints to update the prior knowledge of a system to its posterior from the experimental data in a Bayesian framework. Such problems are computationally challenging because of (1) expensive and repeated evaluation of some optimality criterion that typically involves a double integration with respect to both the system parameters and the experimental data, (2) suffering from the curse-of-dimensionality when the system parameters and design variables are high-dimensional, (3) the optimization is combinatorial and highly non-convex if the design variables are binary, often leading to non-robust designs. To make the solution of the Bayesian OED problem efficient, scalable, and robust for practical applications, we propose a novel joint optimization approach. This approach performs simultaneous (1) training of a scalable conditional normalizing flow (CNF) to efficiently maximize the expected information gain (EIG) of a jointly learned experimental design (2) optimization of a probabilistic formulation of the binary experimental design with a Bernoulli distribution. We demonstrate the performance of our proposed method for a practical MRI data acquisition problem, one of the most challenging Bayesian OED problems that has high-dimensional (320 $\times$ 320) parameters at high image resolution, high-dimensional (640 $\times$ 386) observations, and binary mask designs to select the most informative observations.

Title: Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning

• Authors: Jiachun Li, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, Jun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18344
• Pdf URL: https://arxiv.org/pdf/2402.18344
• Copy Paste: [[2402.18344]] Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning(https://arxiv.org/abs/2402.18344)
• Keywords: large language model
• Abstract: Large language models exhibit high-level commonsense reasoning abilities, especially with enhancement methods like Chain-of-Thought (CoT). However, we find these CoT-like methods lead to a considerable number of originally correct answers turning wrong, which we define as the Toxic CoT problem. To interpret and mitigate this problem, we first utilize attribution tracing and causal tracing methods to probe the internal working mechanism of the LLM during CoT reasoning. Through comparisons, we prove that the model exhibits information loss from the question over the shallow attention layers when generating rationales or answers. Based on the probing findings, we design a novel method called RIDERS (Residual decodIng and sERial-position Swap), which compensates for the information deficit in the model from both decoding and serial-position perspectives. Through extensive experiments on multiple commonsense reasoning benchmarks, we validate that this method not only significantly eliminates Toxic CoT problems (decreased by 23.6%), but also effectively improves the model's overall commonsense reasoning performance (increased by 5.5%).

Title: Objective and Interpretable Breast Cosmesis Evaluation with Attention Guided Denoising Diffusion Anomaly Detection Model

• Authors: Sangjoon Park, Yong Bae Kim, Jee Suk Chang, Seo Hee Choi, Hyungjin Chung, Ik Jae Lee, Hwa Kyung Byun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18362
• Pdf URL: https://arxiv.org/pdf/2402.18362
• Copy Paste: [[2402.18362]] Objective and Interpretable Breast Cosmesis Evaluation with Attention Guided Denoising Diffusion Anomaly Detection Model(https://arxiv.org/abs/2402.18362)
• Keywords: diffusion, transformer
• Abstract: As advancements in the field of breast cancer treatment continue to progress, the assessment of post-surgical cosmetic outcomes has gained increasing significance due to its substantial impact on patients' quality of life. However, evaluating breast cosmesis presents challenges due to the inherently subjective nature of expert labeling. In this study, we present a novel automated approach, Attention-Guided Denoising Diffusion Anomaly Detection (AG-DDAD), designed to assess breast cosmesis following surgery, addressing the limitations of conventional supervised learning and existing anomaly detection models. Our approach leverages the attention mechanism of the distillation with no label (DINO) self-supervised Vision Transformer (ViT) in combination with a diffusion model to achieve high-quality image reconstruction and precise transformation of discriminative regions. By training the diffusion model on unlabeled data predominantly with normal cosmesis, we adopt an unsupervised anomaly detection perspective to automatically score the cosmesis. Real-world data experiments demonstrate the effectiveness of our method, providing visually appealing representations and quantifiable scores for cosmesis evaluation. Compared to commonly used rule-based programs, our fully automated approach eliminates the need for manual annotations and offers objective evaluation. Moreover, our anomaly detection model exhibits state-of-the-art performance, surpassing existing models in accuracy. Going beyond the scope of breast cosmesis, our research represents a significant advancement in unsupervised anomaly detection within the medical domain, thereby paving the way for future investigations.

Title: Token-based Vehicular Security System (TVSS): Scalable, Secure, Low-latency Public Key Infrastructure for Connected Vehicles

• Authors: Abdulrahman Bin Rabiah, Anas Alsoliman, Yugarshi Shashwat, Silas Richelson, Nael Abu-Ghazaleh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18365
• Pdf URL: https://arxiv.org/pdf/2402.18365
• Copy Paste: [[2402.18365]] Token-based Vehicular Security System (TVSS): Scalable, Secure, Low-latency Public Key Infrastructure for Connected Vehicles(https://arxiv.org/abs/2402.18365)
• Keywords: secure, security, privacy, attack
• Abstract: Connected and Autonomous vehicles stand to drastically improve the safety and efficiency of the transportation system in the near future while also reducing pollution. These systems leverage communication to coordinate among vehicles and infrastructure in service of a number of safety and efficiency driver assist and even fully autonomous applications. Attackers can compromise these systems in a number of ways including by falsifying communication messages, making it critical to support security mechanisms that can operate and scale in dynamic scenarios. Towards this end, we present TVSS, a new VPKI system which improves drastically over prior work in the area (including over SCMS; the US department of transportation standard for VPKI). TVSS leverages the idea of unforgeable tokens to enable rapid verification at the road side units (RSUs), which are part of the road infrastructure at the edge of the network. This edge based solution enables agile authentication by avoiding the need for back-end servers during the potentially short contact time between a moving vehicle and the infrastructure. It also results in several security advantages: (1) Scalable Revocation: it greatly simplifies the revocation problem, a difficult problem in large scale certificate systems; and (2) Faster Refresh: Vehicles interact more frequently with the system to refresh their credentials, improving the privacy of the system. We provide a construction of the system and formally prove its security. Field experiments on a test-bed we develop consisting of on-board units (OBUs) and RSUs shows substantial reduction in the latency of refreshing credentials compared to SCMS, allowing the system to work even with smaller window of connectivity when vehicles are moving at higher speeds. Notably, we are able to execute the bottleneck operation of our scheme with a stationary RSU while traveling at highway speeds .

Title: Adversarial example soups: averaging multiple adversarial examples improves transferability without increasing additional generation time

• Authors: Bo Yang, Hengwei Zhang, Chenwei Li, Jindong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18370
• Pdf URL: https://arxiv.org/pdf/2402.18370
• Copy Paste: [[2402.18370]] Adversarial example soups: averaging multiple adversarial examples improves transferability without increasing additional generation time(https://arxiv.org/abs/2402.18370)
• Keywords: attack
• Abstract: For transfer-based attacks, the adversarial examples are crafted on the surrogate model, which can be implemented to mislead the target model effectively. The conventional method for maximizing adversarial transferability involves: (1) fine-tuning hyperparameters to generate multiple batches of adversarial examples on the substitute model; (2) conserving the batch of adversarial examples that have the best comprehensive performance on substitute model and target model, and discarding the others. In this work, we revisit the second step of this process in the context of fine-tuning hyperparameters to craft adversarial examples, where multiple batches of fine-tuned adversarial examples often appear in a single high error hilltop. We demonstrate that averaging multiple batches of adversarial examples under different hyperparameter configurations, which refers to as "adversarial example soups", can often enhance adversarial transferability. Compared with traditional methods, the proposed method incurs no additional generation time and computational cost. Besides, our method is orthogonal to existing transfer-based methods and can be combined with them seamlessly to generate more transferable adversarial examples. Extensive experiments on the ImageNet dataset show that our methods achieve a higher attack success rate than the state-of-the-art attacks.

Title: FedUV: Uniformity and Variance for Heterogeneous Federated Learning

• Authors: Ha Min Son, Moon Hyun Kim, Tai-Myoung Chung, Chao Huang, Xin Liu
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.18372
• Pdf URL: https://arxiv.org/pdf/2402.18372
• Copy Paste: [[2402.18372]] FedUV: Uniformity and Variance for Heterogeneous Federated Learning(https://arxiv.org/abs/2402.18372)
• Keywords: federate
• Abstract: Federated learning is a promising framework to train neural networks with widely distributed data. However, performance degrades heavily with heterogeneously distributed data. Recent work has shown this is due to the final layer of the network being most prone to local bias, some finding success freezing the final layer as an orthogonal classifier. We investigate the training dynamics of the classifier by applying SVD to the weights motivated by the observation that freezing weights results in constant singular values. We find that there are differences when training in IID and non-IID settings. Based on this finding, we introduce two regularization terms for local training to continuously emulate IID settings: (1) variance in the dimension-wise probability distribution of the classifier and (2) hyperspherical uniformity of representations of the encoder. These regularizations promote local models to act as if it were in an IID setting regardless of the local data distribution, thus offsetting proneness to bias while being flexible to the data. On extensive experiments in both label-shift and feature-shift settings, we verify that our method achieves highest performance by a large margin especially in highly non-IID cases in addition to being scalable to larger models and datasets.

Title: VerifiNER: Verification-augmented NER via Knowledge-grounded Reasoning with Large Language Models

• Authors: Seoyeon Kim, Kwangwook Seo, Hyungjoo Chae, Jinyoung Yeo, Dongha Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18374
• Pdf URL: https://arxiv.org/pdf/2402.18374
• Copy Paste: [[2402.18374]] VerifiNER: Verification-augmented NER via Knowledge-grounded Reasoning with Large Language Models(https://arxiv.org/abs/2402.18374)
• Keywords: large language model
• Abstract: Recent approaches in domain-specific named entity recognition (NER), such as biomedical NER, have shown remarkable advances. However, they still lack of faithfulness, producing erroneous predictions. We assume that knowledge of entities can be useful in verifying the correctness of the predictions. Despite the usefulness of knowledge, resolving such errors with knowledge is nontrivial, since the knowledge itself does not directly indicate the ground-truth label. To this end, we propose VerifiNER, a post-hoc verification framework that identifies errors from existing NER methods using knowledge and revises them into more faithful predictions. Our framework leverages the reasoning abilities of large language models to adequately ground on knowledge and the contextual information in the verification process. We validate effectiveness of VerifiNER through extensive experiments on biomedical datasets. The results suggest that VerifiNER can successfully verify errors from existing models as a model-agnostic approach. Further analyses on out-of-domain and low-resource settings show the usefulness of VerifiNER on real-world applications.

Title: Tokenization Is More Than Compression

• Authors: Craig W. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, Chris Tanner
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18376
• Pdf URL: https://arxiv.org/pdf/2402.18376
• Copy Paste: [[2402.18376]] Tokenization Is More Than Compression(https://arxiv.org/abs/2402.18376)
• Keywords: segmentation
• Abstract: Tokenization is a foundational step in Natural Language Processing (NLP) tasks, bridging raw text and language models. Existing tokenization approaches like Byte-Pair Encoding (BPE) originate from the field of data compression, and it has been suggested that the effectiveness of BPE stems from its ability to condense text into a relatively small number of tokens. We test the hypothesis that fewer tokens lead to better downstream performance by introducing PathPiece, a new tokenizer that segments a document's text into the minimum number of tokens for a given vocabulary. Through extensive experimentation we find this hypothesis not to be the case, casting doubt on the understanding of the reasons for effective tokenization. To examine which other factors play a role, we evaluate design decisions across all three phases of tokenization: pre-tokenization, vocabulary construction, and segmentation, offering new insights into the design of effective tokenizers. Specifically, we illustrate the importance of pre-tokenization and the benefits of using BPE to initialize vocabulary construction. We train 64 language models with varying tokenization, ranging in size from 350M to 2.4B parameters, all of which are made publicly available.

Title: Robust Quantification of Percent Emphysema on CT via Domain Attention: the Multi-Ethnic Study of Atherosclerosis (MESA) Lung Study

• Authors: Xuzhe Zhang, Elsa D. Angelini, Eric A. Hoffman, Karol E. Watson, Benjamin M. Smith, R. Graham Barr, Andrew F. Laine
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18383
• Pdf URL: https://arxiv.org/pdf/2402.18383
• Copy Paste: [[2402.18383]] Robust Quantification of Percent Emphysema on CT via Domain Attention: the Multi-Ethnic Study of Atherosclerosis (MESA) Lung Study(https://arxiv.org/abs/2402.18383)
• Keywords: robust, segmentation
• Abstract: Robust quantification of pulmonary emphysema on computed tomography (CT) remains challenging for large-scale research studies that involve scans from different scanner types and for translation to clinical scans. Existing studies have explored several directions to tackle this challenge, including density correction, noise filtering, regression, hidden Markov measure field (HMMF) model-based segmentation, and volume-adjusted lung density. Despite some promising results, previous studies either required a tedious workflow or limited opportunities for downstream emphysema subtyping, limiting efficient adaptation on a large-scale study. To alleviate this dilemma, we developed an end-to-end deep learning framework based on an existing HMMF segmentation framework. We first demonstrate that a regular UNet cannot replicate the existing HMMF results because of the lack of scanner priors. We then design a novel domain attention block to fuse image feature with quantitative scanner priors which significantly improves the results.

Title: The First Place Solution of WSDM Cup 2024: Leveraging Large Language Models for Conversational Multi-Doc QA

• Authors: Yiming Li, Zhao Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18385
• Pdf URL: https://arxiv.org/pdf/2402.18385
• Copy Paste: [[2402.18385]] The First Place Solution of WSDM Cup 2024: Leveraging Large Language Models for Conversational Multi-Doc QA(https://arxiv.org/abs/2402.18385)
• Keywords: large language model
• Abstract: Conversational multi-doc question answering aims to answer specific questions based on the retrieved documents as well as the contextual conversations. In this paper, we introduce our winning approach for the "Conversational Multi-Doc QA" challenge in WSDM Cup 2024, which exploits the superior natural language understanding and generation capability of Large Language Models (LLMs). We first adapt LLMs to the task, then devise a hybrid training strategy to make the most of in-domain unlabeled data. Moreover, an advanced text embedding model is adopted to filter out potentially irrelevant documents and several approaches are designed and compared for the model ensemble. Equipped with all these techniques, our solution finally ranked 1st place in WSDM Cup 2024, surpassing its rivals to a large extent. The source codes have been released at https://github.com/zhangzhao219/WSDM-Cup-2024.

Title: Unveiling the Potential of Robustness in Evaluating Causal Inference Models

• Authors: Yiyan Huang, Cheuk Hang Leung, Siyi Wang, Yijun Li, Qi Wu
• Subjects: cs.LG, cs.AI, econ.EM, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.18392
• Pdf URL: https://arxiv.org/pdf/2402.18392
• Copy Paste: [[2402.18392]] Unveiling the Potential of Robustness in Evaluating Causal Inference Models(https://arxiv.org/abs/2402.18392)
• Keywords: robust
• Abstract: The growing demand for personalized decision-making has led to a surge of interest in estimating the Conditional Average Treatment Effect (CATE). The intersection of machine learning and causal inference has yielded various effective CATE estimators. However, deploying these estimators in practice is often hindered by the absence of counterfactual labels, making it challenging to select the desirable CATE estimator using conventional model selection procedures like cross-validation. Existing approaches for CATE estimator selection, such as plug-in and pseudo-outcome metrics, face two inherent challenges. Firstly, they are required to determine the metric form and the underlying machine learning models for fitting nuisance parameters or plug-in learners. Secondly, they lack a specific focus on selecting a robust estimator. To address these challenges, this paper introduces a novel approach, the Distributionally Robust Metric (DRM), for CATE estimator selection. The proposed DRM not only eliminates the need to fit additional models but also excels at selecting a robust CATE estimator. Experimental studies demonstrate the efficacy of the DRM method, showcasing its consistent effectiveness in identifying superior estimators while mitigating the risk of selecting inferior ones.

Title: Decomposed Prompting: Unveiling Multilingual Linguistic Structure Knowledge in English-Centric Large Language Models

• Authors: Ercong Nie, Shuzhou Yuan, Bolei Ma, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18397
• Pdf URL: https://arxiv.org/pdf/2402.18397
• Copy Paste: [[2402.18397]] Decomposed Prompting: Unveiling Multilingual Linguistic Structure Knowledge in English-Centric Large Language Models(https://arxiv.org/abs/2402.18397)
• Keywords: large language model
• Abstract: Despite the predominance of English in their training data, English-centric Large Language Models (LLMs) like GPT-3 and LLaMA display a remarkable ability to perform multilingual tasks, raising questions about the depth and nature of their cross-lingual capabilities. This paper introduces the decomposed prompting approach to probe the linguistic structure understanding of these LLMs in sequence labeling tasks. Diverging from the single text-to-text prompt, our method generates for each token of the input sentence an individual prompt which asks for its linguistic label. We assess our method on the Universal Dependencies part-of-speech tagging dataset for 38 languages, utilizing both English-centric and multilingual LLMs. Our findings show that decomposed prompting surpasses the iterative prompting baseline in efficacy and efficiency under zero- and few-shot settings. Further analysis reveals the influence of evaluation methods and the use of instructions in prompts. Our multilingual investigation shows that English-centric language models perform better on average than multilingual models. Our study offers insights into the multilingual transferability of English-centric LLMs, contributing to the understanding of their multilingual linguistic knowledge.

Title: A Modular System for Enhanced Robustness of Multimedia Understanding Networks via Deep Parametric Estimation

• Authors: Francesco Barbato, Umberto Michieli, Mehmet Karim Yucel, Pietro Zanuttigh, Mete Ozay
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18402
• Pdf URL: https://arxiv.org/pdf/2402.18402
• Copy Paste: [[2402.18402]] A Modular System for Enhanced Robustness of Multimedia Understanding Networks via Deep Parametric Estimation(https://arxiv.org/abs/2402.18402)
• Keywords: robust, segmentation
• Abstract: In multimedia understanding tasks, corrupted samples pose a critical challenge, because when fed to machine learning models they lead to performance degradation. In the past, three groups of approaches have been proposed to handle noisy data: i) enhancer and denoiser modules to improve the quality of the noisy data, ii) data augmentation approaches, and iii) domain adaptation strategies. All the aforementioned approaches come with drawbacks that limit their applicability; the first has high computational costs and requires pairs of clean-corrupted data for training, while the others only allow deployment of the same task/network they were trained on (\ie, when upstream and downstream task/network are the same). In this paper, we propose SyMPIE to solve these shortcomings. To this end, we design a small, modular, and efficient (just 2GFLOPs to process a Full HD image) system to enhance input data for robust downstream multimedia understanding with minimal computational cost. Our SyMPIE is pre-trained on an upstream task/network that should not match the downstream ones and does not need paired clean-corrupted samples. Our key insight is that most input corruptions found in real-world tasks can be modeled through global operations on color channels of images or spatial filters with small kernels. We validate our approach on multiple datasets and tasks, such as image classification (on ImageNetC, ImageNetC-Bar, VizWiz, and a newly proposed mixed corruption benchmark named ImageNetC-mixed) and semantic segmentation (on Cityscapes, ACDC, and DarkZurich) with consistent improvements of about 5\% relative accuracy gain across the board. The code of our approach and the new ImageNetC-mixed benchmark will be made available upon publication.

Title: Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation

• Authors: Yusheng Liao, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18428
• Pdf URL: https://arxiv.org/pdf/2402.18428
• Copy Paste: [[2402.18428]] Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation(https://arxiv.org/abs/2402.18428)
• Keywords: generative
• Abstract: Autoregressive (AR) and Non-autoregressive (NAR) models are two types of generative models for Neural Machine Translation (NMT). AR models predict tokens in a word-by-word manner and can effectively capture the distribution of real translations. NAR models predict tokens by extracting bidirectional contextual information which can improve the inference speed but they suffer from performance degradation. Previous works utilized AR models to enhance NAR models by reducing the training data's complexity or incorporating the global information into AR models by virtue of NAR models. However, those investigated methods only take advantage of the contextual information of a single type of model while neglecting the diversity in the contextual information that can be provided by different types of models. In this paper, we propose a novel generic collaborative learning method, DCMCL, where AR and NAR models are treated as collaborators instead of teachers and students. To hierarchically leverage the bilateral contextual information, token-level mutual learning and sequence-level contrastive learning are adopted between AR and NAR models. Extensive experiments on four widely used benchmarks show that the proposed DCMCL method can simultaneously improve both AR and NAR models with up to 1.38 and 2.98 BLEU scores respectively, and can also outperform the current best-unified model with up to 0.97 BLEU scores for both AR and NAR decoding.

Title: Smishing Dataset I: Phishing SMS Dataset from Smishtank.com

• Authors: Daniel Timko, Muhammad Lutfor Rahman
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18430
• Pdf URL: https://arxiv.org/pdf/2402.18430
• Copy Paste: [[2402.18430]] Smishing Dataset I: Phishing SMS Dataset from Smishtank.com(https://arxiv.org/abs/2402.18430)
• Keywords: defense, attack, robust
• Abstract: While smishing (SMS Phishing) attacks have risen to become one of the most common types of social engineering attacks, there is a lack of relevant smishing datasets. One of the biggest challenges in the domain of smishing prevention is the availability of fresh smishing datasets. Additionally, as time persists, smishing campaigns are shut down and the crucial information related to the attack are lost. With the changing nature of smishing attacks, a consistent flow of new smishing examples is needed by both researchers and engineers to create effective defenses. In this paper, we present the community-sourced smishing datasets from the smishtank.com. It provides a wealth of information relevant to combating smishing attacks through the breakdown and analysis of smishing samples at the point of submission. In the contribution of our work, we provide a corpus of 1090 smishing samples that have been publicly submitted through the site. Each message includes information relating to the sender, message body, and any brands referenced in the message. Additionally, when a URL is found, we provide additional information on the domain, VirusTotal results, and a characterization of the URL. Through the open access of fresh smishing data, we empower academia and industries to create robust defenses against this evolving threat.

Title: Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication

• Authors: Weize Chen, Chenfei Yuan, Jiarui Yuan, Yusheng Su, Chen Qian, Cheng Yang, Ruobing Xie, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18439
• Pdf URL: https://arxiv.org/pdf/2402.18439
• Copy Paste: [[2402.18439]] Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication(https://arxiv.org/abs/2402.18439)
• Keywords: large language model
• Abstract: Natural language (NL) has long been the predominant format for human cognition and communication, and by extension, has been similarly pivotal in the development and application of Large Language Models (LLMs). Yet, besides NL, LLMs have seen various non-NL formats during pre-training, such as code and logical expression. NL's status as the optimal format for LLMs, particularly in single-LLM reasoning and multi-agent communication, has not been thoroughly examined. In this work, we challenge the default use of NL by exploring the utility of non-NL formats in these contexts. We show that allowing LLMs to autonomously select the most suitable format before reasoning or communicating leads to a 3.3 to 5.7\% improvement in reasoning efficiency for different LLMs, and up to a 72.7\% reduction in token usage in multi-agent communication, all while maintaining communicative effectiveness. Our comprehensive analysis further reveals that LLMs can devise a format from limited task instructions and that the devised format is effectively transferable across different LLMs. Intriguingly, the structured communication format decided by LLMs exhibits notable parallels with established agent communication languages, suggesting a natural evolution towards efficient, structured communication in agent communication. Our code is released at \url{https://github.com/thunlp/AutoForm}.

Title: Meta-Task Prompting Elicits Embedding from Large Language Models

• Authors: Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, Andrew Yates
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18458
• Pdf URL: https://arxiv.org/pdf/2402.18458
• Copy Paste: [[2402.18458]] Meta-Task Prompting Elicits Embedding from Large Language Models(https://arxiv.org/abs/2402.18458)
• Keywords: extraction, large language model
• Abstract: In this work, we introduce a new unsupervised embedding method, Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL), for generating high-quality sentence embeddings from Large Language Models (LLMs) without the need for model fine-tuning or task-specific engineering. Leveraging meta-task prompting, MetaEOL guides LLMs to produce embeddings through a series of carefully designed prompts that address multiple representational aspects. Our comprehensive experiments demonstrate that embeddings averaged from various meta-tasks yield competitive performance on Semantic Textual Similarity (STS) benchmarks and excel in downstream tasks, surpassing contrastive-trained models. Our findings suggest a new scaling law for embedding generation, offering a versatile, resource-efficient approach for embedding extraction across diverse sentence-centric scenarios.

Title: Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation

• Authors: Zhiwei Yang, Kexue Fu, Minghong Duan, Linhao Qu, Shuo Wang, Zhijian Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18467
• Pdf URL: https://arxiv.org/pdf/2402.18467
• Copy Paste: [[2402.18467]] Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation(https://arxiv.org/abs/2402.18467)
• Keywords: segmentation
• Abstract: Attributed to the frequent coupling of co-occurring objects and the limited supervision from image-level labels, the challenging co-occurrence problem is widely present and leads to false activation of objects in weakly supervised semantic segmentation (WSSS). In this work, we devise a 'Separate and Conquer' scheme SeCo to tackle this issue from dimensions of image space and feature space. In the image space, we propose to 'separate' the co-occurring objects with image decomposition by subdividing images into patches. Importantly, we assign each patch a category tag from Class Activation Maps (CAMs), which spatially helps remove the co-context bias and guide the subsequent representation. In the feature space, we propose to 'conquer' the false activation by enhancing semantic representation with multi-granularity knowledge contrast. To this end, a dual-teacher-single-student architecture is designed and tag-guided contrast is conducted to guarantee the correctness of knowledge and further facilitate the discrepancy among co-occurring objects. We streamline the multi-staged WSSS pipeline end-to-end and tackle co-occurrence without external supervision. Extensive experiments are conducted, validating the efficiency of our method tackling co-occurrence and the superiority over previous single-staged and even multi-staged competitors on PASCAL VOC and MS COCO. Code will be available.

Title: IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding

• Authors: Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, Jun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18476
• Pdf URL: https://arxiv.org/pdf/2402.18476
• Copy Paste: [[2402.18476]] IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding(https://arxiv.org/abs/2402.18476)
• Keywords: robust
• Abstract: Despite achieving rapid developments and with widespread applications, Large Vision-Language Models (LVLMs) confront a serious challenge of being prone to generating hallucinations. An over-reliance on linguistic priors has been identified as a key factor leading to these hallucinations. In this paper, we propose to alleviate this problem by introducing a novel image-biased decoding (IBD) technique. Our method derives the next-token probability distribution by contrasting predictions from a conventional LVLM with those of an image-biased LVLM, thereby amplifying the correct information highly correlated with image content while mitigating the hallucinatory errors caused by excessive dependence on text. We further conduct a comprehensive statistical analysis to validate the reliability of our method, and design an adaptive adjustment strategy to achieve robust and flexible handling under varying conditions. Experimental results across multiple evaluation metrics verify that our method, despite not requiring additional training data and only with a minimal increase in model parameters, can significantly reduce hallucinations in LVLMs and enhance the truthfulness of the generated response.

Title: Dynamical Regimes of Diffusion Models

• Authors: Giulio Biroli, Tony Bonnaire, Valentin de Bortoli, Marc Mézard
• Subjects: cs.LG, cond-mat.stat-mech
• Abstract URL: https://arxiv.org/abs/2402.18491
• Pdf URL: https://arxiv.org/pdf/2402.18491
• Copy Paste: [[2402.18491]] Dynamical Regimes of Diffusion Models(https://arxiv.org/abs/2402.18491)
• Keywords: diffusion, generative
• Abstract: Using statistical physics methods, we study generative diffusion models in the regime where the dimension of space and the number of data are large, and the score function has been trained optimally. Our analysis reveals three distinct dynamical regimes during the backward generative diffusion process. The generative dynamics, starting from pure noise, encounters first a 'speciation' transition where the gross structure of data is unraveled, through a mechanism similar to symmetry breaking in phase transitions. It is followed at later time by a 'collapse' transition where the trajectories of the dynamics become attracted to one of the memorized data points, through a mechanism which is similar to the condensation in a glass phase. For any dataset, the speciation time can be found from a spectral analysis of the correlation matrix, and the collapse time can be found from the estimation of an 'excess entropy' in the data. The dependence of the collapse time on the dimension and number of data provides a thorough characterization of the curse of dimensionality for diffusion models. Analytical solutions for simple models like high-dimensional Gaussian mixtures substantiate these findings and provide a theoretical framework, while extensions to more complex scenarios and numerical validations with real datasets confirm the theoretical predictions.

Title: Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection

• Authors: Xun Huang, Hai Wu, Xin Li, Xiaoliang Fan, Chenglu Wen, Cheng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18493
• Pdf URL: https://arxiv.org/pdf/2402.18493
• Copy Paste: [[2402.18493]] Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection(https://arxiv.org/abs/2402.18493)
• Keywords: robust
• Abstract: LiDAR-based 3D object detection models have traditionally struggled under rainy conditions due to the degraded and noisy scanning signals. Previous research has attempted to address this by simulating the noise from rain to improve the robustness of detection models. However, significant disparities exist between simulated and actual rain-impacted data points. In this work, we propose a novel rain simulation method, termed DRET, that unifies Dynamics and Rainy Environment Theory to provide a cost-effective means of expanding the available realistic rain data for 3D detection training. Furthermore, we present a Sunny-to-Rainy Knowledge Distillation (SRKD) approach to enhance 3D detection under rainy conditions. Extensive experiments on the WaymoOpenDataset large-scale dataset show that, when combined with the state-of-the-art DSVT model and other classical 3D detectors, our proposed framework demonstrates significant detection accuracy improvements, without losing efficiency. Remarkably, our framework also improves detection capabilities under sunny conditions, therefore offering a robust solution for 3D detection regardless of whether the weather is rainy or sunny

Title: ROG$_{PL}$: Robust Open-Set Graph Learning via Region-Based Prototype Learning

• Authors: Qin Zhang, Xiaowei Li, Jiexin Lu, Liping Qiu, Shirui Pan, Xiaojun Chen, Junyang Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18495
• Pdf URL: https://arxiv.org/pdf/2402.18495
• Copy Paste: [[2402.18495]] ROG$_{PL}$: Robust Open-Set Graph Learning via Region-Based Prototype Learning(https://arxiv.org/abs/2402.18495)
• Keywords: robust
• Abstract: Open-set graph learning is a practical task that aims to classify the known class nodes and to identify unknown class samples as unknowns. Conventional node classification methods usually perform unsatisfactorily in open-set scenarios due to the complex data they encounter, such as out-of-distribution (OOD) data and in-distribution (IND) noise. OOD data are samples that do not belong to any known classes. They are outliers if they occur in training (OOD noise), and open-set samples if they occur in testing. IND noise are training samples which are assigned incorrect labels. The existence of IND noise and OOD noise is prevalent, which usually cause the ambiguity problem, including the intra-class variety problem and the inter-class confusion problem. Thus, to explore robust open-set learning methods is necessary and difficult, and it becomes even more difficult for non-IID graph data.To this end, we propose a unified framework named ROG$_{PL}$ to achieve robust open-set learning on complex noisy graph data, by introducing prototype learning. In specific, ROG$_{PL}$ consists of two modules, i.e., denoising via label propagation and open-set prototype learning via regions. The first module corrects noisy labels through similarity-based label propagation and removes low-confidence samples, to solve the intra-class variety problem caused by noise. The second module learns open-set prototypes for each known class via non-overlapped regions and remains both interior and border prototypes to remedy the inter-class confusion problem.The two modules are iteratively updated under the constraints of classification loss and prototype diversity loss. To the best of our knowledge, the proposed ROG$_{PL}$ is the first robust open-set node classification method for graph data with complex noise.

Title: Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification

• Authors: Garima Chhikara, Anurag Sharma, Kripabandhu Ghosh, Abhijnan Chakraborty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18502
• Pdf URL: https://arxiv.org/pdf/2402.18502
• Copy Paste: [[2402.18502]] Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification(https://arxiv.org/abs/2402.18502)
• Keywords: fair, large language model
• Abstract: Employing Large Language Models (LLM) in various downstream applications such as classification is crucial, especially for smaller companies lacking the expertise and resources required for fine-tuning a model. Fairness in LLMs helps ensure inclusivity, equal representation based on factors such as race, gender and promotes responsible AI deployment. As the use of LLMs has become increasingly prevalent, it is essential to assess whether LLMs can generate fair outcomes when subjected to considerations of fairness. In this study, we introduce a framework outlining fairness regulations aligned with various fairness definitions, with each definition being modulated by varying degrees of abstraction. We explore the configuration for in-context learning and the procedure for selecting in-context demonstrations using RAG, while incorporating fairness rules into the process. Experiments conducted with different LLMs indicate that GPT-4 delivers superior results in terms of both accuracy and fairness compared to other models. This work is one of the early attempts to achieve fairness in prediction tasks by utilizing LLMs through in-context learning.

Title: Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling

• Authors: Mahdi Karami, Ali Ghodsi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18508
• Pdf URL: https://arxiv.org/pdf/2402.18508
• Copy Paste: [[2402.18508]] Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling(https://arxiv.org/abs/2402.18508)
• Keywords: transformer
• Abstract: In the rapidly evolving landscape of deep learning, the quest for models that balance expressivity with computational efficiency has never been more critical. This paper introduces Orchid, a novel architecture that reimagines sequence modeling by incorporating a new data-dependent convolution mechanism. Orchid is designed to address the inherent limitations of traditional attention mechanisms, particularly their quadratic complexity, without compromising the ability to capture long-range dependencies and in-context learning. At the core of Orchid lies the data-dependent convolution layer, which dynamically adjusts its kernel conditioned on input data using a dedicated conditioning neural network. We design two simple conditioning networks that maintain shift equivariance in the adaptive convolution operation. The dynamic nature of data-dependent convolution kernel, coupled with gating operations, grants Orchid high expressivity while maintaining efficiency and quasilinear scalability for long sequences. We rigorously evaluate Orchid across multiple domains, including language modeling and image classification, to showcase its performance and generality. Our experiments demonstrate that Orchid architecture not only outperforms traditional attention-based architectures such as BERT and Vision Transformers with smaller model sizes, but also extends the feasible sequence length beyond the limitations of the dense attention layers. This achievement represents a significant step towards more efficient and scalable deep learning models for sequence modeling.

Title: RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval

• Authors: Kaiyue Wen, Xingyu Dang, Kaifeng Lyu
• Subjects: cs.LG, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.18510
• Pdf URL: https://arxiv.org/pdf/2402.18510
• Copy Paste: [[2402.18510]] RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval(https://arxiv.org/abs/2402.18510)
• Keywords: transformer
• Abstract: This paper investigates the gap in representation powers of Recurrent Neural Networks (RNNs) and Transformers in the context of solving algorithmic problems. We focus on understanding whether RNNs, known for their memory efficiency in handling long sequences, can match the performance of Transformers, particularly when enhanced with Chain-of-Thought (CoT) prompting. Our theoretical analysis reveals that CoT improves RNNs but is insufficient to close the gap with Transformers. A key bottleneck lies in the inability of RNNs to perfectly retrieve information from the context, even with CoT: for several tasks that explicitly or implicitly require this capability, such as associative recall and determining if a graph is a tree, we prove that RNNs are not expressive enough to solve the tasks while Transformers can solve them with ease. Conversely, we prove that adopting techniques to enhance the in-context retrieval capability of RNNs, including Retrieval-Augmented Generation (RAG) and adding a single Transformer layer, can elevate RNNs to be capable of solving all polynomial-time solvable problems with CoT, hence closing the representation gap with Transformers.

Title: Log Neural Controlled Differential Equations: The Lie Brackets Make a Difference

• Authors: Benjamin Walker, Andrew D. McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, Terry Lyons
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18512
• Pdf URL: https://arxiv.org/pdf/2402.18512
• Copy Paste: [[2402.18512]] Log Neural Controlled Differential Equations: The Lie Brackets Make a Difference(https://arxiv.org/abs/2402.18512)
• Keywords: robust
• Abstract: The vector field of a controlled differential equation (CDE) describes the relationship between a control path and the evolution of a solution path. Neural CDEs (NCDEs) treat time series data as observations from a control path, parameterise a CDE's vector field using a neural network, and use the solution path as a continuously evolving hidden state. As their formulation makes them robust to irregular sampling rates, NCDEs are a powerful approach for modelling real-world data. Building on neural rough differential equations (NRDEs), we introduce Log-NCDEs, a novel and effective method for training NCDEs. The core component of Log-NCDEs is the Log-ODE method, a tool from the study of rough paths for approximating a CDE's solution. On a range of multivariate time series classification benchmarks, Log-NCDEs are shown to achieve a higher average test set accuracy than NCDEs, NRDEs, and two state-of-the-art models, S5 and the linear recurrent unit.

Title: Defect Detection in Tire X-Ray Images: Conventional Methods Meet Deep Structures

• Authors: Andrei Cozma, Landon Harris, Hairong Qi, Ping Ji, Wenpeng Guo, Song Yuan
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.18527
• Pdf URL: https://arxiv.org/pdf/2402.18527
• Copy Paste: [[2402.18527]] Defect Detection in Tire X-Ray Images: Conventional Methods Meet Deep Structures(https://arxiv.org/abs/2402.18527)
• Keywords: robust, extraction
• Abstract: This paper introduces a robust approach for automated defect detection in tire X-ray images by harnessing traditional feature extraction methods such as Local Binary Pattern (LBP) and Gray Level Co-Occurrence Matrix (GLCM) features, as well as Fourier and Wavelet-based features, complemented by advanced machine learning techniques. Recognizing the challenges inherent in the complex patterns and textures of tire X-ray images, the study emphasizes the significance of feature engineering to enhance the performance of defect detection systems. By meticulously integrating combinations of these features with a Random Forest (RF) classifier and comparing them against advanced models like YOLOv8, the research not only benchmarks the performance of traditional features in defect detection but also explores the synergy between classical and modern approaches. The experimental results demonstrate that these traditional features, when fine-tuned and combined with machine learning models, can significantly improve the accuracy and reliability of tire defect detection, aiming to set a new standard in automated quality assurance in tire manufacturing.

Title: Gradient Reweighting: Towards Imbalanced Class-Incremental Learning

• Authors: Jiangpeng He, Fengqing Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18528
• Pdf URL: https://arxiv.org/pdf/2402.18528
• Copy Paste: [[2402.18528]] Gradient Reweighting: Towards Imbalanced Class-Incremental Learning(https://arxiv.org/abs/2402.18528)
• Keywords: robust
• Abstract: Class-Incremental Learning (CIL) trains a model to continually recognize new classes from non-stationary data while retaining learned knowledge. A major challenge of CIL arises when applying to real-world data characterized by non-uniform distribution, which introduces a dual imbalance problem involving (i) disparities between stored exemplars of old tasks and new class data (inter-phase imbalance), and (ii) severe class imbalances within each individual task (intra-phase imbalance). We show that this dual imbalance issue causes skewed gradient updates with biased weights in FC layers, thus inducing over/under-fitting and catastrophic forgetting in CIL. Our method addresses it by reweighting the gradients towards balanced optimization and unbiased classifier learning. Additionally, we observe imbalanced forgetting where paradoxically the instance-rich classes suffer higher performance degradation during CIL due to a larger amount of training data becoming unavailable in subsequent learning phases. To tackle this, we further introduce a distribution-aware knowledge distillation loss to mitigate forgetting by aligning output logits proportionally with the distribution of lost training data. We validate our method on CIFAR-100, ImageNetSubset, and Food101 across various evaluation protocols and demonstrate consistent improvements compared to existing works, showing great potential to apply CIL in real-world scenarios with enhanced robustness and effectiveness.

Title: Generalizability Under Sensor Failure: Tokenization + Transformers Enable More Robust Latent Spaces

• Authors: Geeling Chau, Yujin An, Ahamed Raffey Iqbal, Soon-Jo Chung, Yisong Yue, Sabera Talukder
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18546
• Pdf URL: https://arxiv.org/pdf/2402.18546
• Copy Paste: [[2402.18546]] Generalizability Under Sensor Failure: Tokenization + Transformers Enable More Robust Latent Spaces(https://arxiv.org/abs/2402.18546)
• Keywords: robust, transformer
• Abstract: A major goal in neuroscience is to discover neural data representations that generalize. This goal is challenged by variability along recording sessions (e.g. environment), subjects (e.g. varying neural structures), and sensors (e.g. sensor noise), among others. Recent work has begun to address generalization across sessions and subjects, but few study robustness to sensor failure which is highly prevalent in neuroscience experiments. In order to address these generalizability dimensions we first collect our own electroencephalography dataset with numerous sessions, subjects, and sensors, then study two time series models: EEGNet (Lawhern et al., 2018) and TOTEM (Talukder et al., 2024). EEGNet is a widely used convolutional neural network, while TOTEM is a discrete time series tokenizer and transformer model. We find that TOTEM outperforms or matches EEGNet across all generalizability cases. Finally through analysis of TOTEM's latent codebook we observe that tokenization enables generalization.

Title: Implicit Bias of Next-Token Prediction

• Authors: Christos Thrampoulidis
• Subjects: cs.LG, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.18551
• Pdf URL: https://arxiv.org/pdf/2402.18551
• Copy Paste: [[2402.18551]] Implicit Bias of Next-Token Prediction(https://arxiv.org/abs/2402.18551)
• Keywords: robust, large language model
• Abstract: Next-token prediction (NTP), the go-to training paradigm in training large language models, involves predicting the next token in a sequence. Departing from traditional one-hot classification, in NTP, multiple tokens with varying frequencies follow each given context. This work frames NTP training as cross-entropy minimization over distinct contexts, each associated with a sparse empirical probability vector across a finite vocabulary. It then addresses the following question: do gradient-based optimizers exhibit a bias towards solutions with specific structure as the NTP training loss reaches its lower bound (entropy)? Specifically, for linear NTP models trained using gradient descent (GD), we make the following contributions: Firstly, we determine NTP-separability conditions on the data, under which GD can attain its lower bound. We also demonstrate that these conditions hold under overparameterization. Secondly, we establish that the parameters of GD projected onto an appropriate data subspace converge to the unique solution of a system of linear equations, which requires the logits' difference of in-support tokens to be equal to the log-ratio of their respective probabilities. Meanwhile, on the orthogonal subspace, the parameters diverge and converge in the direction of the solution of a max-margin quadratic program, minimizing the Euclidean norm of parameters satisfying the \NTP-separability conditions. Akin to prior research on implicit bias of one-hot classification, our work opens exciting avenues for future research that can lead to better understanding optimization, generalization and robustness principles of models trained with NTP.

Title: Diffusion Language Models Are Versatile Protein Learners

• Authors: Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, Quanquan Gu
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2402.18567
• Pdf URL: https://arxiv.org/pdf/2402.18567
• Copy Paste: [[2402.18567]] Diffusion Language Models Are Versatile Protein Learners(https://arxiv.org/abs/2402.18567)
• Keywords: diffusion, generative
• Abstract: This paper introduces diffusion protein language model (DPLM), a versatile protein language model that demonstrates strong generative and predictive capabilities for protein sequences. We first pre-train scalable DPLMs from evolutionary-scale protein sequences within a generative self-supervised discrete diffusion probabilistic framework, which generalizes language modeling for proteins in a principled way. After pre-training, DPLM exhibits the ability to generate structurally plausible, novel, and diverse protein sequences for unconditional generation. We further demonstrate the proposed diffusion generative pre-training makes DPLM possess a better understanding of proteins, making it a superior representation learner, which can be fine-tuned for various predictive tasks, comparing favorably to ESM2 (Lin et al., 2022). Moreover, DPLM can be tailored for various needs, which showcases its prowess of conditional generation in several ways: (1) conditioning on partial peptide sequences, e.g., generating scaffolds for functional motifs with high success rate; (2) incorporating other modalities as conditioner, e.g., structure-conditioned generation for inverse folding; and (3) steering sequence generation towards desired properties, e.g., satisfying specified secondary structures, through a plug-and-play classifier guidance.

Title: Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards

• Authors: Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, Tong Zhang
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.18571
• Pdf URL: https://arxiv.org/pdf/2402.18571
• Copy Paste: [[2402.18571]] Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards(https://arxiv.org/abs/2402.18571)
• Keywords: large language model
• Abstract: Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).