2024-04-24

Title: FreSeg: Frenet-Frame-based Part Segmentation for 3D Curvilinear Structures

• Authors: Shixuan Gu, Jason Ken Adhinarta, Mikhail Bessmeltsev, Jiancheng Yang, Jessica Zhang, Daniel Berger, Jeff W. Lichtman, Hanspeter Pfister, Donglai Wei
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.14435
• Pdf URL: https://arxiv.org/pdf/2404.14435
• Copy Paste: [[2404.14435]] FreSeg: Frenet-Frame-based Part Segmentation for 3D Curvilinear Structures(https://arxiv.org/abs/2404.14435)
• Keywords: extraction, segmentation
• Abstract: Part segmentation is a crucial task for 3D curvilinear structures like neuron dendrites and blood vessels, enabling the analysis of dendritic spines and aneurysms with scientific and clinical significance. However, their diversely winded morphology poses a generalization challenge to existing deep learning methods, which leads to labor-intensive manual correction. In this work, we propose FreSeg, a framework of part segmentation tasks for 3D curvilinear structures. With Frenet-Frame-based point cloud transformation, it enables the models to learn more generalizable features and have significant performance improvements on tasks involving elongated and curvy geometries. We evaluate FreSeg on 2 datasets: 1) DenSpineEM, an in-house dataset for dendritic spine segmentation, and 2) IntrA, a public 3D dataset for intracranial aneurysm segmentation. Further, we will release the DenSpineEM dataset, which includes roughly 6,000 spines from 69 dendrites from 3 public electron microscopy (EM) datasets, to foster the development of effective dendritic spine instance extraction methods and, consequently, large-scale connectivity analysis to better understand mammalian brains.

Title: Optimizing Contrail Detection: A Deep Learning Approach with EfficientNet-b4 Encoding

• Authors: Qunwei Lin, Qian Leng, Zhicheng Ding, Chao Yan, Xiaonan Xu
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.14441
• Pdf URL: https://arxiv.org/pdf/2404.14441
• Copy Paste: [[2404.14441]] Optimizing Contrail Detection: A Deep Learning Approach with EfficientNet-b4 Encoding(https://arxiv.org/abs/2404.14441)
• Keywords: robust, extraction, segmentation
• Abstract: In the pursuit of environmental sustainability, the aviation industry faces the challenge of minimizing its ecological footprint. Among the key solutions is contrail avoidance, targeting the linear ice-crystal clouds produced by aircraft exhaust. These contrails exacerbate global warming by trapping atmospheric heat, necessitating precise segmentation and comprehensive analysis of contrail images to gauge their environmental impact. However, this segmentation task is complex due to the varying appearances of contrails under different atmospheric conditions and potential misalignment issues in predictive modeling. This paper presents an innovative deep-learning approach utilizing the efficient net-b4 encoder for feature extraction, seamlessly integrating misalignment correction, soft labeling, and pseudo-labeling techniques to enhance the accuracy and efficiency of contrail detection in satellite imagery. The proposed methodology aims to redefine contrail image analysis and contribute to the objectives of sustainable aviation by providing a robust framework for precise contrail detection and analysis in satellite imagery, thus aiding in the mitigation of aviation's environmental impact.

Title: A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models

• Authors: Yefeng Yuan, Yuhong Liu, Liang Cheng
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14445
• Pdf URL: https://arxiv.org/pdf/2404.14445
• Copy Paste: [[2404.14445]] A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models(https://arxiv.org/abs/2404.14445)
• Keywords: privacy, generative, large language model
• Abstract: The rapid advancements in generative AI and large language models (LLMs) have opened up new avenues for producing synthetic data, particularly in the realm of structured tabular formats, such as product reviews. Despite the potential benefits, concerns regarding privacy leakage have surfaced, especially when personal information is utilized in the training datasets. In addition, there is an absence of a comprehensive evaluation framework capable of quantitatively measuring the quality of the generated synthetic data and their utility for downstream tasks. In response to this gap, we introduce SynEval, an open-source evaluation framework designed to assess the fidelity, utility, and privacy preservation of synthetically generated tabular data via a suite of diverse evaluation metrics. We validate the efficacy of our proposed framework - SynEval - by applying it to synthetic product review data generated by three state-of-the-art LLMs: ChatGPT, Claude, and Llama. Our experimental findings illuminate the trade-offs between various evaluation metrics in the context of synthetic data generation. Furthermore, SynEval stands as a critical instrument for researchers and practitioners engaged with synthetic tabular data,, empowering them to judiciously determine the suitability of the generated data for their specific applications, with an emphasis on upholding user privacy.

Title: Generative Subspace Adversarial Active Learning for Outlier Detection in Multiple Views of High-dimensional Data

• Authors: Jose Cribeiro-Ramallo, Vadim Arzamasov, Federico Matteucci, Denis Wambold, Klemens Böhm
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14451
• Pdf URL: https://arxiv.org/pdf/2404.14451
• Copy Paste: [[2404.14451]] Generative Subspace Adversarial Active Learning for Outlier Detection in Multiple Views of High-dimensional Data(https://arxiv.org/abs/2404.14451)
• Keywords: generative
• Abstract: Outlier detection in high-dimensional tabular data is an important task in data mining, essential for many downstream tasks and applications. Existing unsupervised outlier detection algorithms face one or more problems, including inlier assumption (IA), curse of dimensionality (CD), and multiple views (MV). To address these issues, we introduce Generative Subspace Adversarial Active Learning (GSAAL), a novel approach that uses a Generative Adversarial Network with multiple adversaries. These adversaries learn the marginal class probability functions over different data subspaces, while a single generator in the full space models the entire distribution of the inlier class. GSAAL is specifically designed to address the MV limitation while also handling the IA and CD, being the only method to do so. We provide a comprehensive mathematical formulation of MV, convergence guarantees for the discriminators, and scalability results for GSAAL. Our extensive experiments demonstrate the effectiveness and scalability of GSAAL, highlighting its superior performance compared to other popular OD methods, especially in MV scenarios.

Title: EPI-SQL: Enhancing Text-to-SQL Translation with Error-Prevention Instructions

• Authors: Xiping Liu, Zhao Tan
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2404.14453
• Pdf URL: https://arxiv.org/pdf/2404.14453
• Copy Paste: [[2404.14453]] EPI-SQL: Enhancing Text-to-SQL Translation with Error-Prevention Instructions(https://arxiv.org/abs/2404.14453)
• Keywords: large language model
• Abstract: The conversion of natural language queries into SQL queries, known as Text-to-SQL, is a critical yet challenging task. This paper introduces EPI-SQL, a novel methodological framework leveraging Large Language Models (LLMs) to enhance the performance of Text-to-SQL tasks. EPI-SQL operates through a four-step process. Initially, the method involves gathering instances from the Spider dataset on which LLMs are prone to failure. These instances are then utilized to generate general error-prevention instructions (EPIs). Subsequently, LLMs craft contextualized EPIs tailored to the specific context of the current task. Finally, these context-specific EPIs are incorporated into the prompt used for SQL generation. EPI-SQL is distinguished in that it provides task-specific guidance, enabling the model to circumvent potential errors for the task at hand. Notably, the methodology rivals the performance of advanced few-shot methods despite being a zero-shot approach. An empirical assessment using the Spider benchmark reveals that EPI-SQL achieves an execution accuracy of 85.1\%, underscoring its effectiveness in generating accurate SQL queries through LLMs. The findings indicate a promising direction for future research, i.e. enhancing instructions with task-specific and contextualized rules, for boosting LLMs' performance in NLP tasks.

Title: Reinforcement of Explainability of ChatGPT Prompts by Embedding Breast Cancer Self-Screening Rules into AI Responses

• Authors: Yousef Khan, Ahmed Abdeen Hamed
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14454
• Pdf URL: https://arxiv.org/pdf/2404.14454
• Copy Paste: [[2404.14454]] Reinforcement of Explainability of ChatGPT Prompts by Embedding Breast Cancer Self-Screening Rules into AI Responses(https://arxiv.org/abs/2404.14454)
• Keywords: explainability, generative
• Abstract: Addressing the global challenge of breast cancer, this research explores the fusion of generative AI, focusing on ChatGPT 3.5 turbo model, and the intricacies of breast cancer risk assessment. The research aims to evaluate ChatGPT's reasoning capabilities, emphasizing its potential to process rules and provide explanations for screening recommendations. The study seeks to bridge the technology gap between intelligent machines and clinicians by demonstrating ChatGPT's unique proficiency in natural language reasoning. The methodology employs a supervised prompt-engineering approach to enforce detailed explanations for ChatGPT's recommendations. Synthetic use cases, generated algorithmically, serve as the testing ground for the encoded rules, evaluating the model's processing prowess. Findings highlight ChatGPT's promising capacity in processing rules comparable to Expert System Shells, with a focus on natural language reasoning. The research introduces the concept of reinforcement explainability, showcasing its potential in elucidating outcomes and facilitating user-friendly interfaces for breast cancer risk assessment.

Title: Graph Coloring Using Heat Diffusion

• Authors: Vivek Chaudhary
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14457
• Pdf URL: https://arxiv.org/pdf/2404.14457
• Copy Paste: [[2404.14457]] Graph Coloring Using Heat Diffusion(https://arxiv.org/abs/2404.14457)
• Keywords: diffusion
• Abstract: Graph coloring is a problem with varied applications in industry and science such as scheduling, resource allocation, and circuit design. The purpose of this paper is to establish if a new gradient based iterative solver framework known as heat diffusion can solve the graph coloring problem. We propose a solution to the graph coloring problem using the heat diffusion framework. We compare the solutions against popular methods and establish the competitiveness of heat diffusion method for the graph coloring problem.

Title: Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs

• Authors: Javier Rando, Francesco Croce, Kryštof Mitka, Stepan Shabalin, Maksym Andriushchenko, Nicolas Flammarion, Florian Tramèr
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14461
• Pdf URL: https://arxiv.org/pdf/2404.14461
• Copy Paste: [[2404.14461]] Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs(https://arxiv.org/abs/2404.14461)
• Keywords: attack, large language model
• Abstract: Large language models are aligned to be safe, preventing users from generating harmful content like misinformation or instructions for illegal activities. However, previous work has shown that the alignment process is vulnerable to poisoning attacks. Adversaries can manipulate the safety training data to inject backdoors that act like a universal sudo command: adding the backdoor string to any prompt enables harmful responses from models that, otherwise, behave safely. Our competition, co-located at IEEE SaTML 2024, challenged participants to find universal backdoors in several large language models. This report summarizes the key findings and promising ideas for future research.

Title: Towards smallers, faster decoder-only transformers: Architectural variants and their implications

• Authors: Sathya Krishnan Suresh, Shunmugapriya P
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14462
• Pdf URL: https://arxiv.org/pdf/2404.14462
• Copy Paste: [[2404.14462]] Towards smallers, faster decoder-only transformers: Architectural variants and their implications(https://arxiv.org/abs/2404.14462)
• Keywords: transformer, large language model
• Abstract: Research on Large Language Models (LLMs) has recently seen exponential growth, largely focused on transformer-based architectures, as introduced by [1] and further advanced by the decoder-only variations in [2]. Contemporary studies typically aim to improve model capabilities by increasing both the architecture's complexity and the volume of training data. However, research exploring how to reduce model sizes while maintaining performance is limited. This study introduces three modifications to the decoder-only transformer architecture: ParallelGPT (p-gpt), LinearlyCompressedGPT (lc-gpt), and ConvCompressedGPT (cc-gpt). These variants achieve comparable performance to conventional architectures in code generation tasks while benefiting from reduced model sizes and faster training times. We open-source the model weights and codebase to support future research and development in this domain.

Title: Tree of Reviews: A Tree-based Dynamic Iterative Retrieval Framework for Multi-hop Question Answering

• Authors: Li Jiapeng, Liu Runze, Li Yabo, Zhou Tong, Li Mingling, Chen Xiang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.14464
• Pdf URL: https://arxiv.org/pdf/2404.14464
• Copy Paste: [[2404.14464]] Tree of Reviews: A Tree-based Dynamic Iterative Retrieval Framework for Multi-hop Question Answering(https://arxiv.org/abs/2404.14464)
• Keywords: large language model
• Abstract: Multi-hop question answering is a knowledge-intensive complex problem. Large Language Models (LLMs) use their Chain of Thoughts (CoT) capability to reason complex problems step by step, and retrieval-augmentation can effectively alleviate factual errors caused by outdated and unknown knowledge in LLMs. Recent works have introduced retrieval-augmentation in the CoT reasoning to solve multi-hop question answering. However, these chain methods have the following problems: 1) Retrieved irrelevant paragraphs may mislead the reasoning; 2) An error in the chain structure may lead to a cascade of errors. In this paper, we propose a dynamic retrieval framework called Tree of Reviews (ToR), where the root node is the question, and the other nodes are paragraphs from retrieval, extending different reasoning paths from the root node to other nodes. Our framework dynamically decides to initiate a new search, reject, or accept based on the paragraphs on the reasoning paths. Compared to related work, we introduce a tree structure to handle each retrieved paragraph separately, alleviating the misleading effect of irrelevant paragraphs on the reasoning path; the diversity of reasoning path extension reduces the impact of a single reasoning error on the whole. We conducted experiments on three different multi-hop question answering datasets. The results show that compared to the baseline methods, ToR achieves state-of-the-art performance in both retrieval and response generation. In addition, we propose two tree-based search optimization strategies, pruning and effective expansion, to reduce time overhead and increase the diversity of path extension. We will release our code.

Title: Benchmarking Advanced Text Anonymisation Methods: A Comparative Study on Novel and Traditional Approaches

• Authors: Dimitris Asimopoulos, Ilias Siniosoglou, Vasileios Argyriou, Thomai Karamitsou, Eleftherios Fountoukidis, Sotirios K. Goudos, Ioannis D. Moscholios, Konstantinos E. Psannis, Panagiotis Sarigiannidis
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.14465
• Pdf URL: https://arxiv.org/pdf/2404.14465
• Copy Paste: [[2404.14465]] Benchmarking Advanced Text Anonymisation Methods: A Comparative Study on Novel and Traditional Approaches(https://arxiv.org/abs/2404.14465)
• Keywords: privacy, robust, transformer, large language model
• Abstract: In the realm of data privacy, the ability to effectively anonymise text is paramount. With the proliferation of deep learning and, in particular, transformer architectures, there is a burgeoning interest in leveraging these advanced models for text anonymisation tasks. This paper presents a comprehensive benchmarking study comparing the performance of transformer-based models and Large Language Models(LLM) against traditional architectures for text anonymisation. Utilising the CoNLL-2003 dataset, known for its robustness and diversity, we evaluate several models. Our results showcase the strengths and weaknesses of each approach, offering a clear perspective on the efficacy of modern versus traditional methods. Notably, while modern models exhibit advanced capabilities in capturing con textual nuances, certain traditional architectures still keep high performance. This work aims to guide researchers in selecting the most suitable model for their anonymisation needs, while also shedding light on potential paths for future advancements in the field.

Title: Integrating Chemistry Knowledge in Large Language Models via Prompt Engineering

• Authors: Hongxuan Liu, Haoyu Yin, Zhiyao Luo, Xiaonan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14467
• Pdf URL: https://arxiv.org/pdf/2404.14467
• Copy Paste: [[2404.14467]] Integrating Chemistry Knowledge in Large Language Models via Prompt Engineering(https://arxiv.org/abs/2404.14467)
• Keywords: large language model
• Abstract: This paper presents a study on the integration of domain-specific knowledge in prompt engineering to enhance the performance of large language models (LLMs) in scientific domains. A benchmark dataset is curated to encapsulate the intricate physical-chemical properties of small molecules, their drugability for pharmacology, alongside the functional attributes of enzymes and crystal materials, underscoring the relevance and applicability across biological and chemical domains.The proposed domain-knowledge embedded prompt engineering method outperforms traditional prompt engineering strategies on various metrics, including capability, accuracy, F1 score, and hallucination drop. The effectiveness of the method is demonstrated through case studies on complex materials including the MacMillan catalyst, paclitaxel, and lithium cobalt oxide. The results suggest that domain-knowledge prompts can guide LLMs to generate more accurate and relevant responses, highlighting the potential of LLMs as powerful tools for scientific discovery and innovation when equipped with domain-specific prompts. The study also discusses limitations and future directions for domain-specific prompt engineering development.

Title: SnapKV: LLM Knows What You are Looking for Before Generation

• Authors: Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14469
• Pdf URL: https://arxiv.org/pdf/2404.14469
• Copy Paste: [[2404.14469]] SnapKV: LLM Knows What You are Looking for Before Generation(https://arxiv.org/abs/2404.14469)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have made remarkable progress in processing extensive contexts, with the Key-Value (KV) cache playing a vital role in enhancing their performance. However, the growth of the KV cache in response to increasing input length poses challenges to memory and time efficiency. To address this problem, this paper introduces SnapKV, an innovative and fine-tuning-free approach that efficiently minimizes KV cache size while still delivering comparable performance in real-world applications. We discover that each attention head in the model consistently focuses on specific prompt attention features during generation. Meanwhile, this robust pattern can be obtained from an `observation' window located at the end of the prompts. Drawing on this insight, SnapKV automatically compresses KV caches by selecting clustered important KV positions for each attention head. Our approach significantly reduces the growing computational overhead and memory footprint when processing long input sequences. Specifically, SnapKV achieves a consistent decoding speed with a 3.6x increase in generation speed and an 8.2x enhancement in memory efficiency compared to baseline when processing inputs of 16K tokens. At the same time, it maintains comparable performance to baseline models across 16 long sequence datasets. Moreover, SnapKV can process up to 380K context tokens on a single A100-80GB GPU using HuggingFace implementation with minor changes, exhibiting only a negligible accuracy drop in the Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's potential for practical applications.

Title: Narrative Action Evaluation with Prompt-Guided Multimodal Interaction

• Authors: Shiyi Zhang, Sule Bai, Guangyi Chen, Lei Chen, Jiwen Lu, Junle Wang, Yansong Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14471
• Pdf URL: https://arxiv.org/pdf/2404.14471
• Copy Paste: [[2404.14471]] Narrative Action Evaluation with Prompt-Guided Multimodal Interaction(https://arxiv.org/abs/2404.14471)
• Keywords: transformer
• Abstract: In this paper, we investigate a new problem called narrative action evaluation (NAE). NAE aims to generate professional commentary that evaluates the execution of an action. Unlike traditional tasks such as score-based action quality assessment and video captioning involving superficial sentences, NAE focuses on creating detailed narratives in natural language. These narratives provide intricate descriptions of actions along with objective evaluations. NAE is a more challenging task because it requires both narrative flexibility and evaluation rigor. One existing possible solution is to use multi-task learning, where narrative language and evaluative information are predicted separately. However, this approach results in reduced performance for individual tasks because of variations between tasks and differences in modality between language information and evaluation information. To address this, we propose a prompt-guided multimodal interaction framework. This framework utilizes a pair of transformers to facilitate the interaction between different modalities of information. It also uses prompts to transform the score regression task into a video-text matching task, thus enabling task interactivity. To support further research in this field, we re-annotate the MTL-AQA and FineGym datasets with high-quality and comprehensive action narration. Additionally, we establish benchmarks for NAE. Extensive experiment results prove that our method outperforms separate learning methods and naive multi-task learning methods. Data and code are released at \href{https://github.com/shiyi-zh0408/NAE_CVPR2024 }{here}.

Title: Align Your Steps: Optimizing Sampling Schedules in Diffusion Models

• Authors: Amirmojtaba Sabour, Sanja Fidler, Karsten Kreis
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14507
• Pdf URL: https://arxiv.org/pdf/2404.14507
• Copy Paste: [[2404.14507]] Align Your Steps: Optimizing Sampling Schedules in Diffusion Models(https://arxiv.org/abs/2404.14507)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DMs) have established themselves as the state-of-the-art generative modeling approach in the visual domain and beyond. A crucial drawback of DMs is their slow sampling speed, relying on many sequential function evaluations through large neural networks. Sampling from DMs can be seen as solving a differential equation through a discretized set of noise levels known as the sampling schedule. While past works primarily focused on deriving efficient solvers, little attention has been given to finding optimal sampling schedules, and the entire literature relies on hand-crafted heuristics. In this work, for the first time, we propose a general and principled approach to optimizing the sampling schedules of DMs for high-quality outputs, called $\textit{Align Your Steps}$. We leverage methods from stochastic calculus and find optimal schedules specific to different solvers, trained DMs and datasets. We evaluate our novel approach on several image, video as well as 2D toy data synthesis benchmarks, using a variety of different samplers, and observe that our optimized schedules outperform previous hand-crafted schedules in almost all experiments. Our method demonstrates the untapped potential of sampling schedule optimization, especially in the few-step synthesis regime.

Title: WangLab at MEDIQA-CORR 2024: Optimized LLM-based Programs for Medical Error Detection and Correction

• Authors: Augustin Toma, Ronald Xie, Steven Palayew, Patrick R. Lawler, Bo Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14544
• Pdf URL: https://arxiv.org/pdf/2404.14544
• Copy Paste: [[2404.14544]] WangLab at MEDIQA-CORR 2024: Optimized LLM-based Programs for Medical Error Detection and Correction(https://arxiv.org/abs/2404.14544)
• Keywords: robust, large language model
• Abstract: Medical errors in clinical text pose significant risks to patient safety. The MEDIQA-CORR 2024 shared task focuses on detecting and correcting these errors across three subtasks: identifying the presence of an error, extracting the erroneous sentence, and generating a corrected sentence. In this paper, we present our approach that achieved top performance in all three subtasks. For the MS dataset, which contains subtle errors, we developed a retrieval-based system leveraging external medical question-answering datasets. For the UW dataset, reflecting more realistic clinical notes, we created a pipeline of modules to detect, localize, and correct errors. Both approaches utilized the DSPy framework for optimizing prompts and few-shot examples in large language model (LLM) based programs. Our results demonstrate the effectiveness of LLM based programs for medical error correction. However, our approach has limitations in addressing the full diversity of potential errors in medical documentation. We discuss the implications of our work and highlight future research directions to advance the robustness and applicability of medical error detection and correction systems.

Title: Adaptive Local Binary Pattern: A Novel Feature Descriptor for Enhanced Analysis of Kidney Abnormalities in CT Scan Images using ensemble based Machine Learning Approach

• Authors: Tahmim Hossain, Faisal Sayed, Solehin Islam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14560
• Pdf URL: https://arxiv.org/pdf/2404.14560
• Copy Paste: [[2404.14560]] Adaptive Local Binary Pattern: A Novel Feature Descriptor for Enhanced Analysis of Kidney Abnormalities in CT Scan Images using ensemble based Machine Learning Approach(https://arxiv.org/abs/2404.14560)
• Keywords: robust, extraction
• Abstract: The shortage of nephrologists and the growing public health concern over renal failure have spurred the demand for AI systems capable of autonomously detecting kidney abnormalities. Renal failure, marked by a gradual decline in kidney function, can result from factors like cysts, stones, and tumors. Chronic kidney disease may go unnoticed initially, leading to untreated cases until they reach an advanced stage. The dataset, comprising 12,427 images from multiple hospitals in Dhaka, was categorized into four groups: cyst, tumor, stone, and normal. Our methodology aims to enhance CT scan image quality using Cropping, Resizing, and CALHE techniques, followed by feature extraction with our proposed Adaptive Local Binary Pattern (A-LBP) feature extraction method compared with the state-of-the-art local binary pattern (LBP) method. Our proposed features fed into classifiers such as Random Forest, Decision Tree, Naive Bayes, K-Nearest Neighbor, and SVM. We explored an ensemble model with soft voting to get a more robust model for our task. We got the highest of more than 99% in accuracy using our feature descriptor and ensembling five classifiers (Random Forest, Decision Tree, Naive Bayes, K-Nearest Neighbor, Support Vector Machine) with the soft voting method.

Title: WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models

• Authors: Ronald Xie, Steven Palayew, Augustin Toma, Gary Bader, Bo Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14567
• Pdf URL: https://arxiv.org/pdf/2404.14567
• Copy Paste: [[2404.14567]] WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models(https://arxiv.org/abs/2404.14567)
• Keywords: large language model
• Abstract: This paper outlines our submission to the MEDIQA2024 Multilingual and Multimodal Medical Answer Generation (M3G) shared task. We report results for two standalone solutions under the English category of the task, the first involving two consecutive API calls to the Claude 3 Opus API and the second involving training an image-disease label joint embedding in the style of CLIP for image classification. These two solutions scored 1st and 2nd place respectively on the competition leaderboard, substantially outperforming the next best solution. Additionally, we discuss insights gained from post-competition experiments. While the performance of these two solutions have significant room for improvement due to the difficulty of the shared task and the challenging nature of medical visual question answering in general, we identify the multi-stage LLM approach and the CLIP image classification approach as promising avenues for further investigation.

Title: UVMap-ID: A Controllable and Personalized UV Map Generative Model

• Authors: Weijie Wang, Jichao Zhang, Chang Liu, Xia Li, Xingqian Xu, Humphrey Shi, Nicu Sebe, Bruno Lepri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14568
• Pdf URL: https://arxiv.org/pdf/2404.14568
• Copy Paste: [[2404.14568]] UVMap-ID: A Controllable and Personalized UV Map Generative Model(https://arxiv.org/abs/2404.14568)
• Keywords: diffusion, generative
• Abstract: Recently, diffusion models have made significant strides in synthesizing realistic 2D human images based on provided text prompts. Building upon this, researchers have extended 2D text-to-image diffusion models into the 3D domain for generating human textures (UV Maps). However, some important problems about UV Map Generative models are still not solved, i.e., how to generate personalized texture maps for any given face image, and how to define and evaluate the quality of these generated texture maps. To solve the above problems, we introduce a novel method, UVMap-ID, which is a controllable and personalized UV Map generative model. Unlike traditional large-scale training methods in 2D, we propose to fine-tune a pre-trained text-to-image diffusion model which is integrated with a face fusion module for achieving ID-driven customized generation. To support the finetuning strategy, we introduce a small-scale attribute-balanced training dataset, including high-quality textures with labeled text and Face ID. Additionally, we introduce some metrics to evaluate the multiple aspects of the textures. Finally, both quantitative and qualitative analyses demonstrate the effectiveness of our method in controllable and personalized UV Map generation. Code is publicly available via https://github.com/twowwj/UVMap-ID.

Title: Demystifying Invariant Effectiveness for Securing Smart Contracts

• Authors: Zhiyang Chen, Ye Liu, Sidi Mohamed Beillahi, Yi Li, Fan Long
• Subjects: cs.CR, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.14580
• Pdf URL: https://arxiv.org/pdf/2404.14580
• Copy Paste: [[2404.14580]] Demystifying Invariant Effectiveness for Securing Smart Contracts(https://arxiv.org/abs/2404.14580)
• Keywords: security, attack
• Abstract: Smart contract transactions associated with security attacks often exhibit distinct behavioral patterns compared with historical benign transactions before the attacking events. While many runtime monitoring and guarding mechanisms have been proposed to validate invariants and stop anomalous transactions on the fly, the empirical effectiveness of the invariants used remains largely unexplored. In this paper, we studied 23 prevalent invariants of 8 categories, which are either deployed in high-profile protocols or endorsed by leading auditing firms and security experts. Using these well-established invariants as templates, we developed a tool Trace2Inv which dynamically generates new invariants customized for a given contract based on its historical transaction data. We evaluated Trace2Inv on 42 smart contracts that fell victim to 27 distinct exploits on the Ethereum blockchain. Our findings reveal that the most effective invariant guard alone can successfully block 18 of the 27 identified exploits with minimal gas overhead. Our analysis also shows that most of the invariants remain effective even when the experienced attackers attempt to bypass them. Additionally, we studied the possibility of combining multiple invariant guards, resulting in blocking up to 23 of the 27 benchmark exploits and achieving false positive rates as low as 0.32%. Trace2Inv outperforms current state-of-the-art works on smart contract invariant mining and transaction attack detection in terms of both practicality and accuracy. Though Trace2Inv is not primarily designed for transaction attack detection, it surprisingly found two previously unreported exploit transactions, earlier than any reported exploit transactions against the same victim contracts.

Title: The Adversarial AI-Art: Understanding, Generation, Detection, and Benchmarking

• Authors: Yuying Li, Zeyan Liu, Junyi Zhao, Liangqin Ren, Fengjun Li, Jiebo Luo, Bo Luo
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.14581
• Pdf URL: https://arxiv.org/pdf/2404.14581
• Copy Paste: [[2404.14581]] The Adversarial AI-Art: Understanding, Generation, Detection, and Benchmarking(https://arxiv.org/abs/2404.14581)
• Keywords: security, generative
• Abstract: Generative AI models can produce high-quality images based on text prompts. The generated images often appear indistinguishable from images generated by conventional optical photography devices or created by human artists (i.e., real images). While the outstanding performance of such generative models is generally well received, security concerns arise. For instance, such image generators could be used to facilitate fraud or scam schemes, generate and spread misinformation, or produce fabricated artworks. In this paper, we present a systematic attempt at understanding and detecting AI-generated images (AI-art) in adversarial scenarios. First, we collect and share a dataset of real images and their corresponding artificial counterparts generated by four popular AI image generators. The dataset, named ARIA, contains over 140K images in five categories: artworks (painting), social media images, news photos, disaster scenes, and anime pictures. This dataset can be used as a foundation to support future research on adversarial AI-art. Next, we present a user study that employs the ARIA dataset to evaluate if real-world users can distinguish with or without reference images. In a benchmarking study, we further evaluate if state-of-the-art open-source and commercial AI image detectors can effectively identify the images in the ARIA dataset. Finally, we present a ResNet-50 classifier and evaluate its accuracy and transferability on the ARIA dataset.

Title: Brain-Inspired Continual Learning-Robust Feature Distillation and Re-Consolidation for Class Incremental Learning

• Authors: Hikmat Khan, Nidhal Carla Bouaynaya, Ghulam Rasool
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14588
• Pdf URL: https://arxiv.org/pdf/2404.14588
• Copy Paste: [[2404.14588]] Brain-Inspired Continual Learning-Robust Feature Distillation and Re-Consolidation for Class Incremental Learning(https://arxiv.org/abs/2404.14588)
• Keywords: robust
• Abstract: Artificial intelligence (AI) and neuroscience share a rich history, with advancements in neuroscience shaping the development of AI systems capable of human-like knowledge retention. Leveraging insights from neuroscience and existing research in adversarial and continual learning, we introduce a novel framework comprising two core concepts: feature distillation and re-consolidation. Our framework, named Robust Rehearsal, addresses the challenge of catastrophic forgetting inherent in continual learning (CL) systems by distilling and rehearsing robust features. Inspired by the mammalian brain's memory consolidation process, Robust Rehearsal aims to emulate the rehearsal of distilled experiences during learning tasks. Additionally, it mimics memory re-consolidation, where new experiences influence the integration of past experiences to mitigate forgetting. Extensive experiments conducted on CIFAR10, CIFAR100, and real-world helicopter attitude datasets showcase the superior performance of CL models trained with Robust Rehearsal compared to baseline methods. Furthermore, examining different optimization training objectives-joint, continual, and adversarial learning-we highlight the crucial role of feature learning in model performance. This underscores the significance of rehearsing CL-robust samples in mitigating catastrophic forgetting. In conclusion, aligning CL approaches with neuroscience insights offers promising solutions to the challenge of catastrophic forgetting, paving the way for more robust and human-like AI systems.

Title: Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training

• Authors: Mengzhao Jia, Zhihan Zhang, Wenhao Yu, Fangkai Jiao, Meng Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14604
• Pdf URL: https://arxiv.org/pdf/2404.14604
• Copy Paste: [[2404.14604]] Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training(https://arxiv.org/abs/2404.14604)
• Keywords: large language model
• Abstract: Open-source multimodal large language models (MLLMs) excel in various tasks involving textual and visual inputs but still struggle with complex multimodal mathematical reasoning, lagging behind proprietary models like GPT-4V(ision) and Gemini-Pro. Although fine-tuning with intermediate steps (i.e., rationales) elicits some mathematical reasoning skills, the resulting models still fall short in visual comprehension due to inadequate visual-centric supervision, which leads to inaccurate interpretation of math figures. To address this issue, we propose a two-step training pipeline VCAR, which emphasizes the Visual Comprehension training in Addition to mathematical Reasoning learning. It first improves the visual comprehension ability of MLLMs through the visual description generation task, followed by another training step on generating rationales with the assistance of descriptions. Experimental results on two popular benchmarks demonstrate that VCAR substantially outperforms baseline methods solely relying on rationale supervision, especially on problems with high visual demands.

Title: Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classification

• Authors: Armando Zhu, Keqin Li, Tong Wu, Peng Zhao, Wenjing Zhou, Bo Hong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14606
• Pdf URL: https://arxiv.org/pdf/2404.14606
• Copy Paste: [[2404.14606]] Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classification(https://arxiv.org/abs/2404.14606)
• Keywords: transformer
• Abstract: With wearing masks becoming a new cultural norm, facial expression recognition (FER) while taking masks into account has become a significant challenge. In this paper, we propose a unified multi-branch vision transformer for facial expression recognition and mask wearing classification tasks. Our approach extracts shared features for both tasks using a dual-branch architecture that obtains multi-scale feature representations. Furthermore, we propose a cross-task fusion phase that processes tokens for each task with separate branches, while exchanging information using a cross attention module. Our proposed framework reduces the overall complexity compared with using separate networks for both tasks by the simple yet effective cross-task fusion phase. Extensive experiments demonstrate that our proposed model performs better than or on par with different state-of-the-art methods on both facial expression recognition and facial mask wearing classification task.

Title: Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

• Authors: Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V.S. Lakshmanan, Ahmed Hassan Awadallah
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14618
• Pdf URL: https://arxiv.org/pdf/2404.14618
• Copy Paste: [[2404.14618]] Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing(https://arxiv.org/abs/2404.14618)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality.

Title: OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework

• Authors: Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14619
• Pdf URL: https://arxiv.org/pdf/2404.14619
• Copy Paste: [[2404.14619]] OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework(https://arxiv.org/abs/2404.14619)
• Keywords: transformer, large language model
• Abstract: The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks. To this end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. For example, with a parameter budget of approximately one billion parameters, OpenELM exhibits a 2.36% improvement in accuracy compared to OLMo while requiring $2\times$ fewer pre-training tokens. Diverging from prior practices that only provide model weights and inference code, and pre-train on private datasets, our release includes the complete framework for training and evaluation of the language model on publicly available datasets, including training logs, multiple checkpoints, and pre-training configurations. We also release code to convert models to MLX library for inference and fine-tuning on Apple devices. This comprehensive release aims to empower and strengthen the open research community, paving the way for future open research endeavors. Our source code along with pre-trained model weights and training recipes is available at \url{https://github.com/apple/corenet}. Additionally, \model models can be found on HuggingFace at: \url{https://huggingface.co/apple/OpenELM}.

Title: Fairness Incentives in Response to Unfair Dynamic Pricing

• Authors: Jesse Thibodeau, Hadi Nekoei, Afaf Taïk, Janarthanan Rajendran, Golnoosh Farnadi
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.14620
• Pdf URL: https://arxiv.org/pdf/2404.14620
• Copy Paste: [[2404.14620]] Fairness Incentives in Response to Unfair Dynamic Pricing(https://arxiv.org/abs/2404.14620)
• Keywords: fair
• Abstract: The use of dynamic pricing by profit-maximizing firms gives rise to demand fairness concerns, measured by discrepancies in consumer groups' demand responses to a given pricing strategy. Notably, dynamic pricing may result in buyer distributions unreflective of those of the underlying population, which can be problematic in markets where fair representation is socially desirable. To address this, policy makers might leverage tools such as taxation and subsidy to adapt policy mechanisms dependent upon their social objective. In this paper, we explore the potential for AI methods to assist such intervention strategies. To this end, we design a basic simulated economy, wherein we introduce a dynamic social planner (SP) to generate corporate taxation schedules geared to incentivizing firms towards adopting fair pricing behaviours, and to use the collected tax budget to subsidize consumption among underrepresented groups. To cover a range of possible policy scenarios, we formulate our social planner's learning problem as a multi-armed bandit, a contextual bandit and finally as a full reinforcement learning (RL) problem, evaluating welfare outcomes from each case. To alleviate the difficulty in retaining meaningful tax rates that apply to less frequently occurring brackets, we introduce FairReplayBuffer, which ensures that our RL agent samples experiences uniformly across a discretized fairness space. We find that, upon deploying a learned tax and redistribution policy, social welfare improves on that of the fairness-agnostic baseline, and approaches that of the analytically optimal fairness-aware baseline for the multi-armed and contextual bandit settings, and surpassing it by 13.19% in the full RL setting.

Title: UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues

• Authors: Vandad Davoodnia, Saeed Ghorbani, Marc-André Carbonneau, Alexandre Messier, Ali Etemad
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14634
• Pdf URL: https://arxiv.org/pdf/2404.14634
• Copy Paste: [[2404.14634]] UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues(https://arxiv.org/abs/2404.14634)
• Keywords: robust
• Abstract: We introduce UPose3D, a novel approach for multi-view 3D human pose estimation, addressing challenges in accuracy and scalability. Our method advances existing pose estimation frameworks by improving robustness and flexibility without requiring direct 3D annotations. At the core of our method, a pose compiler module refines predictions from a 2D keypoints estimator that operates on a single image by leveraging temporal and cross-view information. Our novel cross-view fusion strategy is scalable to any number of cameras, while our synthetic data generation strategy ensures generalization across diverse actors, scenes, and viewpoints. Finally, UPose3D leverages the prediction uncertainty of both the 2D keypoint estimator and the pose compiler module. This provides robustness to outliers and noisy data, resulting in state-of-the-art performance in out-of-distribution settings. In addition, for in-distribution settings, UPose3D yields a performance rivaling methods that rely on 3D annotated data, while being the state-of-the-art among methods relying only on 2D supervision.

Title: Teaching Network Traffic Matrices in an Interactive Game Environment

• Authors: Chasen Milner, Hayden Jananthan, Jeremy Kepner, Vijay Gadepally, Michael Jones, Peter Michaleas, Ritesh Patel, Sandeep Pisharody, Gabriel Wachman, Alex Pentland
• Subjects: cs.CR, cs.CY, cs.GR, cs.NI, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.14643
• Pdf URL: https://arxiv.org/pdf/2404.14643
• Copy Paste: [[2404.14643]] Teaching Network Traffic Matrices in an Interactive Game Environment(https://arxiv.org/abs/2404.14643)
• Keywords: security, protect, defense, attack
• Abstract: The Internet has become a critical domain for modern society that requires ongoing efforts for its improvement and protection. Network traffic matrices are a powerful tool for understanding and analyzing networks and are broadly taught in online graph theory educational resources. Network traffic matrix concepts are rarely available in online computer network and cybersecurity educational resources. To fill this gap, an interactive game environment has been developed to teach the foundations of traffic matrices to the computer networking community. The game environment provides a convenient, broadly accessible, delivery mechanism that enables making material available rapidly to a wide audience. The core architecture of the game is a facility to add new network traffic matrix training modules via an easily editable JSON file. Using this facility an initial set of modules were rapidly created covering: basic traffic matrices, traffic patterns, security/defense/deterrence, a notional cyber attack, a distributed denial-of-service (DDoS) attack, and a variety of graph theory concepts. The game environment enables delivery in a wide range of contexts to enable rapid feedback and improvement. The game can be used as a core unit as part of a formal course or as a simple interactive introduction in a presentation.

Title: Machine Vision Based Assessment of Fall Color Changes in Apple Trees: Exploring Relationship with Leaf Nitrogen Concentration

• Authors: Achyut Paudel, Jostan Brown, Priyanka Upadhyaya, Atif Bilal Asad, Safal Kshetri, Manoj Karkee, Joseph R. Davidson, Cindy Grimm, Ashley Thompson
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14653
• Pdf URL: https://arxiv.org/pdf/2404.14653
• Copy Paste: [[2404.14653]] Machine Vision Based Assessment of Fall Color Changes in Apple Trees: Exploring Relationship with Leaf Nitrogen Concentration(https://arxiv.org/abs/2404.14653)
• Keywords: segmentation
• Abstract: Apple trees being deciduous trees, shed leaves each year which is preceded by the change in color of leaves from green to yellow (also known as senescence) during the fall season. The rate and timing of color change are affected by the number of factors including nitrogen (N) deficiencies. The green color of leaves is highly dependent on the chlorophyll content, which in turn depends on the nitrogen concentration in the leaves. The assessment of the leaf color can give vital information on the nutrient status of the tree. The use of a machine vision based system to capture and quantify these timings and changes in leaf color can be a great tool for that purpose. \par This study is based on data collected during the fall of 2021 and 2023 at a commercial orchard using a ground-based stereo-vision sensor for five weeks. The point cloud obtained from the sensor was segmented to get just the tree in the foreground. The study involved the segmentation of the trees in a natural background using point cloud data and quantification of the color using a custom-defined metric, \textit{yellowness index}, varying from $-1$ to $+1$ ($-1$ being completely green and $+1$ being completely yellow), which gives the proportion of yellow leaves on a tree. The performance of K-means based algorithm and gradient boosting algorithm were compared for \textit{yellowness index} calculation. The segmentation method proposed in the study was able to estimate the \textit{yellowness index} on the trees with $R^2 = 0.72$. The results showed that the metric was able to capture the gradual color transition from green to yellow over the study duration. It was also observed that the trees with lower nitrogen showed the color transition to yellow earlier than the trees with higher nitrogen. The onset of color transition during both years aligned with the $29^{th}$ week post-full bloom.

Title: Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation

• Authors: Abhishek Aich, Yumin Suh, Samuel Schulter, Manmohan Chandraker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14657
• Pdf URL: https://arxiv.org/pdf/2404.14657
• Copy Paste: [[2404.14657]] Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation(https://arxiv.org/abs/2404.14657)
• Keywords: transformer, segmentation
• Abstract: A powerful architecture for universal segmentation relies on transformers that encode multi-scale image features and decode object queries into mask predictions. With efficiency being a high priority for scaling such models, we observed that the state-of-the-art method Mask2Former uses ~50% of its compute only on the transformer encoder. This is due to the retention of a full-length token-level representation of all backbone feature scales at each encoder layer. With this observation, we propose a strategy termed PROgressive Token Length SCALing for Efficient transformer encoders (PRO-SCALE) that can be plugged-in to the Mask2Former-style segmentation architectures to significantly reduce the computational cost. The underlying principle of PRO-SCALE is: progressively scale the length of the tokens with the layers of the encoder. This allows PRO-SCALE to reduce computations by a large margin with minimal sacrifice in performance (~52% GFLOPs reduction with no drop in performance on COCO dataset). We validate our framework on multiple public benchmarks.

Title: NExT: Teaching Large Language Models to Reason about Code Execution

• Authors: Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, Pengcheng Yin
• Subjects: cs.LG, cs.CL, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.14662
• Pdf URL: https://arxiv.org/pdf/2404.14662
• Copy Paste: [[2404.14662]] NExT: Teaching Large Language Models to Reason about Code Execution(https://arxiv.org/abs/2404.14662)
• Keywords: large language model
• Abstract: A fundamental skill among human developers is the ability to understand and reason about program execution. As an example, a programmer can mentally simulate code execution in natural language to debug and repair code (aka. rubber duck debugging). However, large language models (LLMs) of code are typically trained on the surface textual form of programs, thus may lack a semantic understanding of how programs execute at run-time. To address this issue, we propose NExT, a method to teach LLMs to inspect the execution traces of programs (variable states of executed lines) and reason about their run-time behavior through chain-of-thought (CoT) rationales. Specifically, NExT uses self-training to bootstrap a synthetic training set of execution-aware rationales that lead to correct task solutions (e.g., fixed programs) without laborious manual annotation. Experiments on program repair tasks based on MBPP and HumanEval demonstrate that NExT improves the fix rate of a PaLM 2 model, by 26.1% and 14.3% absolute, respectively, with significantly improved rationale quality as verified by automated metrics and human raters. Our model can also generalize to scenarios where program traces are absent at test-time.

Title: Employing Layerwised Unsupervised Learning to Lessen Data and Loss Requirements in Forward-Forward Algorithms

• Authors: Taewook Hwang, Hyein Seo, Sangkeun Jung
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14664
• Pdf URL: https://arxiv.org/pdf/2404.14664
• Copy Paste: [[2404.14664]] Employing Layerwised Unsupervised Learning to Lessen Data and Loss Requirements in Forward-Forward Algorithms(https://arxiv.org/abs/2404.14664)
• Keywords: federate
• Abstract: Recent deep learning models such as ChatGPT utilizing the back-propagation algorithm have exhibited remarkable performance. However, the disparity between the biological brain processes and the back-propagation algorithm has been noted. The Forward-Forward algorithm, which trains deep learning models solely through the forward pass, has emerged to address this. Although the Forward-Forward algorithm cannot replace back-propagation due to limitations such as having to use special input and loss functions, it has the potential to be useful in special situations where back-propagation is difficult to use. To work around this limitation and verify usability, we propose an Unsupervised Forward-Forward algorithm. Using an unsupervised learning model enables training with usual loss functions and inputs without restriction. Through this approach, we lead to stable learning and enable versatile utilization across various datasets and tasks. From a usability perspective, given the characteristics of the Forward-Forward algorithm and the advantages of the proposed method, we anticipate its practical application even in scenarios such as federated learning, where deep learning layers need to be trained separately in physically distributed environments.

Title: 3DFlowRenderer: One-shot Face Re-enactment via Dense 3D Facial Flow Estimation

• Authors: Siddharth Nijhawan, Takuya Yashima, Tamaki Kojima
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14667
• Pdf URL: https://arxiv.org/pdf/2404.14667
• Copy Paste: [[2404.14667]] 3DFlowRenderer: One-shot Face Re-enactment via Dense 3D Facial Flow Estimation(https://arxiv.org/abs/2404.14667)
• Keywords: robust
• Abstract: Performing facial expression transfer under one-shot setting has been increasing in popularity among research community with a focus on precise control of expressions. Existing techniques showcase compelling results in perceiving expressions, but they lack robustness with extreme head poses. They also struggle to accurately reconstruct background details, thus hindering the realism. In this paper, we propose a novel warping technology which integrates the advantages of both 2D and 3D methods to achieve robust face re-enactment. We generate dense 3D facial flow fields in feature space to warp an input image based on target expressions without depth information. This enables explicit 3D geometric control for re-enacting misaligned source and target faces. We regularize the motion estimation capability of the 3D flow prediction network through proposed "Cyclic warp loss" by converting warped 3D features back into 2D RGB space. To ensure the generation of finer facial region with natural-background, our framework only renders the facial foreground region first and learns to inpaint the blank area which needs to be filled due to source face translation, thus reconstructing the detailed background without any unwanted pixel motion. Extensive evaluation reveals that our method outperforms state-of-the-art techniques in rendering artifact-free facial images.

Title: LaneCorrect: Self-supervised Lane Detection

• Authors: Ming Nie, Xinyue Cai, Hang Xu, Li Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14671
• Pdf URL: https://arxiv.org/pdf/2404.14671
• Copy Paste: [[2404.14671]] LaneCorrect: Self-supervised Lane Detection(https://arxiv.org/abs/2404.14671)
• Keywords: segmentation
• Abstract: Lane detection has evolved highly functional autonomous driving system to understand driving scenes even under complex environments. In this paper, we work towards developing a generalized computer vision system able to detect lanes without using any annotation. We make the following contributions: (i) We illustrate how to perform unsupervised 3D lane segmentation by leveraging the distinctive intensity of lanes on the LiDAR point cloud frames, and then obtain the noisy lane labels in the 2D plane by projecting the 3D points; (ii) We propose a novel self-supervised training scheme, dubbed LaneCorrect, that automatically corrects the lane label by learning geometric consistency and instance awareness from the adversarial augmentations; (iii) With the self-supervised pre-trained model, we distill to train a student network for arbitrary target lane (e.g., TuSimple) detection without any human labels; (iv) We thoroughly evaluate our self-supervised method on four major lane detection benchmarks (including TuSimple, CULane, CurveLanes and LLAMAS) and demonstrate excellent performance compared with existing supervised counterpart, whilst showing more effective results on alleviating the domain gap, i.e., training on CULane and test on TuSimple.

Title: DreamPBR: Text-driven Generation of High-resolution SVBRDF with Multi-modal Guidance

• Authors: Linxuan Xin, Zheng Zhang, Jinfu Wei, Ge Li, Duan Gao
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.14676
• Pdf URL: https://arxiv.org/pdf/2404.14676
• Copy Paste: [[2404.14676]] DreamPBR: Text-driven Generation of High-resolution SVBRDF with Multi-modal Guidance(https://arxiv.org/abs/2404.14676)
• Keywords: diffusion, generative
• Abstract: Prior material creation methods had limitations in producing diverse results mainly because reconstruction-based methods relied on real-world measurements and generation-based methods were trained on relatively small material datasets. To address these challenges, we propose DreamPBR, a novel diffusion-based generative framework designed to create spatially-varying appearance properties guided by text and multi-modal controls, providing high controllability and diversity in material generation. Key to achieving diverse and high-quality PBR material generation lies in integrating the capabilities of recent large-scale vision-language models trained on billions of text-image pairs, along with material priors derived from hundreds of PBR material samples. We utilize a novel material Latent Diffusion Model (LDM) to establish the mapping between albedo maps and the corresponding latent space. The latent representation is then decoded into full SVBRDF parameter maps using a rendering-aware PBR decoder. Our method supports tileable generation through convolution with circular padding. Furthermore, we introduce a multi-modal guidance module, which includes pixel-aligned guidance, style image guidance, and 3D shape guidance, to enhance the control capabilities of the material LDM. We demonstrate the effectiveness of DreamPBR in material creation, showcasing its versatility and user-friendliness on a wide range of controllable generation and editing applications.

Title: 3DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset

• Authors: Junjie Zhang, Tianci Hu, Xiaoshui Huang, Yongshun Gong, Dan Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14678
• Pdf URL: https://arxiv.org/pdf/2404.14678
• Copy Paste: [[2404.14678]] 3DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset(https://arxiv.org/abs/2404.14678)
• Keywords: large language model
• Abstract: Evaluating the performance of Multi-modal Large Language Models (MLLMs), integrating both point cloud and language, presents significant challenges. The lack of a comprehensive assessment hampers determining whether these models truly represent advancements, thereby impeding further progress in the field. Current evaluations heavily rely on classification and caption tasks, falling short in providing a thorough assessment of MLLMs. A pressing need exists for a more sophisticated evaluation method capable of thoroughly analyzing the spatial understanding and expressive capabilities of these models. To address these issues, we introduce a scalable 3D benchmark, accompanied by a large-scale instruction-tuning dataset known as 3DBench, providing an extensible platform for a comprehensive evaluation of MLLMs. Specifically, we establish the benchmark that spans a wide range of spatial and semantic scales, from object-level to scene-level, addressing both perception and planning tasks. Furthermore, we present a rigorous pipeline for automatically constructing scalable 3D instruction-tuning datasets, covering 10 diverse multi-modal tasks with more than 0.23 million QA pairs generated in total. Thorough experiments evaluating trending MLLMs, comparisons against existing datasets, and variations of training protocols demonstrate the superiority of 3DBench, offering valuable insights into current limitations and potential research directions.

Title: Automated Multi-Language to English Machine Translation Using Generative Pre-Trained Transformers

• Authors: Elijah Pelofske, Vincent Urias, Lorie M. Liebrock
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14680
• Pdf URL: https://arxiv.org/pdf/2404.14680
• Copy Paste: [[2404.14680]] Automated Multi-Language to English Machine Translation Using Generative Pre-Trained Transformers(https://arxiv.org/abs/2404.14680)
• Keywords: transformer, generative
• Abstract: The task of accurate and efficient language translation is an extremely important information processing task. Machine learning enabled and automated translation that is accurate and fast is often a large topic of interest in the machine learning and data science communities. In this study, we examine using local Generative Pretrained Transformer (GPT) models to perform automated zero shot black-box, sentence wise, multi-natural-language translation into English text. We benchmark 16 different open-source GPT models, with no custom fine-tuning, from the Huggingface LLM repository for translating 50 different non-English languages into English using translated TED Talk transcripts as the reference dataset. These GPT model inference calls are performed strictly locally, on single A100 Nvidia GPUs. Benchmark metrics that are reported are language translation accuracy, using BLEU, GLEU, METEOR, and chrF text overlap measures, and wall-clock time for each sentence translation. The best overall performing GPT model for translating into English text for the BLEU metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.152$, for the GLEU metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.256$, for the chrF metric is Llama2-chat-AYT-13B with a mean score across all tested languages of $0.448$, and for the METEOR metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.438$.

Title: FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model

• Authors: Zezheng Song, Jiaxin Yuan, Haizhao Yang
• Subjects: cs.LG, cs.AI, cs.CE, math.DS, math.NA
• Abstract URL: https://arxiv.org/abs/2404.14688
• Pdf URL: https://arxiv.org/pdf/2404.14688
• Copy Paste: [[2404.14688]] FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model(https://arxiv.org/abs/2404.14688)
• Keywords: generative
• Abstract: Human-designed algorithms have long been fundamental in solving a variety of scientific and engineering challenges. Recently, data-driven deep learning methods have also risen to prominence, offering innovative solutions across numerous scientific fields. While traditional algorithms excel in capturing the core aspects of specific problems, they often lack the flexibility needed for varying problem conditions due to the absence of specific data. Conversely, while data-driven approaches utilize vast datasets, they frequently fall short in domain-specific knowledge. To bridge these gaps, we introduce \textbf{FMint} (Foundation Model based on Initialization), a generative pre-trained model that synergizes the precision of human-designed algorithms with the adaptability of data-driven methods. This model is specifically engineered for high-accuracy simulation of dynamical systems. Starting from initial trajectories provided by conventional methods, FMint quickly delivers highly accurate solutions. It incorporates in-context learning and has been pre-trained on a diverse corpus of 500,000 dynamical systems, showcasing exceptional generalization across a broad spectrum of real-world applications. By effectively combining algorithmic rigor with data-driven flexibility, FMint sets the stage for the next generation of scientific foundation models, tackling complex problems with both efficiency and high accuracy.

Title: Interpretable Prediction and Feature Selection for Survival Analysis

• Authors: Mike Van Ness, Madeleine Udell
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.14689
• Pdf URL: https://arxiv.org/pdf/2404.14689
• Copy Paste: [[2404.14689]] Interpretable Prediction and Feature Selection for Survival Analysis(https://arxiv.org/abs/2404.14689)
• Keywords: interpretability
• Abstract: Survival analysis is widely used as a technique to model time-to-event data when some data is censored, particularly in healthcare for predicting future patient risk. In such settings, survival models must be both accurate and interpretable so that users (such as doctors) can trust the model and understand model predictions. While most literature focuses on discrimination, interpretability is equally as important. A successful interpretable model should be able to describe how changing each feature impacts the outcome, and should only use a small number of features. In this paper, we present DyS (pronounced ``dice''), a new survival analysis model that achieves both strong discrimination and interpretability. DyS is a feature-sparse Generalized Additive Model, combining feature selection and interpretable prediction into one model. While DyS works well for all survival analysis problems, it is particularly useful for large (in $n$ and $p$) survival datasets such as those commonly found in observational healthcare studies. Empirical studies show that DyS competes with other state-of-the-art machine learning models for survival analysis, while being highly interpretable.

Title: Double Privacy Guard: Robust Traceable Adversarial Watermarking against Face Recognition

• Authors: Yunming Zhang, Dengpan Ye, Sipeng Shen, Caiyun Xie, Ziyi Liu, Jiacheng Deng, Long Tang
• Subjects: cs.CR, cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.14693
• Pdf URL: https://arxiv.org/pdf/2404.14693
• Copy Paste: [[2404.14693]] Double Privacy Guard: Robust Traceable Adversarial Watermarking against Face Recognition(https://arxiv.org/abs/2404.14693)
• Keywords: privacy, protect, attack, robust, watermark
• Abstract: The wide deployment of Face Recognition (FR) systems poses risks of privacy leakage. One countermeasure to address this issue is adversarial attacks, which deceive malicious FR searches but simultaneously interfere the normal identity verification of trusted authorizers. In this paper, we propose the first Double Privacy Guard (DPG) scheme based on traceable adversarial watermarking. DPG employs a one-time watermark embedding to deceive unauthorized FR models and allows authorizers to perform identity verification by extracting the watermark. Specifically, we propose an information-guided adversarial attack against FR models. The encoder embeds an identity-specific watermark into the deep feature space of the carrier, guiding recognizable features of the image to deviate from the source identity. We further adopt a collaborative meta-optimization strategy compatible with sub-tasks, which regularizes the joint optimization direction of the encoder and decoder. This strategy enhances the representation of universal carrier features, mitigating multi-objective optimization conflicts in watermarking. Experiments confirm that DPG achieves significant attack success rates and traceability accuracy on state-of-the-art FR models, exhibiting remarkable robustness that outperforms the existing privacy protection methods using adversarial attacks and deep watermarking, or simple combinations of the two. Our work potentially opens up new insights into proactive protection for FR privacy.

Title: Think-Program-reCtify: 3D Situated Reasoning with Large Language Models

• Authors: Qingrong He, Kejun Lin, Shizhe Chen, Anwen Hu, Qin Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14705
• Pdf URL: https://arxiv.org/pdf/2404.14705
• Copy Paste: [[2404.14705]] Think-Program-reCtify: 3D Situated Reasoning with Large Language Models(https://arxiv.org/abs/2404.14705)
• Keywords: robust, interpretability, large language model
• Abstract: This work addresses the 3D situated reasoning task which aims to answer questions given egocentric observations in a 3D environment. The task remains challenging as it requires comprehensive 3D perception and complex reasoning skills. End-to-end models trained on supervised data for 3D situated reasoning suffer from data scarcity and generalization ability. Inspired by the recent success of leveraging large language models (LLMs) for visual reasoning, we propose LLM-TPC, a novel framework that leverages the planning, tool usage, and reflection capabilities of LLMs through a ThinkProgram-reCtify loop. The Think phase first decomposes the compositional question into a sequence of steps, and then the Program phase grounds each step to a piece of code and calls carefully designed 3D visual perception modules. Finally, the Rectify phase adjusts the plan and code if the program fails to execute. Experiments and analysis on the SQA3D benchmark demonstrate the effectiveness, interpretability and robustness of our method. Our code is publicly available at https://qingrongh.github.io/LLM-TPC/.

Title: SC-HVPPNet: Spatial and Channel Hybrid-Attention Video Post-Processing Network with CNN and Transformer

• Authors: Tong Zhang, Wenxue Cui, Shaohui Liu, Feng Jiang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.14709
• Pdf URL: https://arxiv.org/pdf/2404.14709
• Copy Paste: [[2404.14709]] SC-HVPPNet: Spatial and Channel Hybrid-Attention Video Post-Processing Network with CNN and Transformer(https://arxiv.org/abs/2404.14709)
• Keywords: transformer
• Abstract: Convolutional Neural Network (CNN) and Transformer have attracted much attention recently for video post-processing (VPP). However, the interaction between CNN and Transformer in existing VPP methods is not fully explored, leading to inefficient communication between the local and global extracted features. In this paper, we explore the interaction between CNN and Transformer in the task of VPP, and propose a novel Spatial and Channel Hybrid-Attention Video Post-Processing Network (SC-HVPPNet), which can cooperatively exploit the image priors in both spatial and channel domains. Specifically, in the spatial domain, a novel spatial attention fusion module is designed, in which two attention weights are generated to fuse the local and global representations collaboratively. In the channel domain, a novel channel attention fusion module is developed, which can blend the deep representations at the channel dimension dynamically. Extensive experiments show that SC-HVPPNet notably boosts video restoration quality, with average bitrate savings of 5.29%, 12.42%, and 13.09% for Y, U, and V components in the VTM-11.0-NNVC RA configuration.

Title: Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities

• Authors: Siyin Wang, Chao-Han Huck Yang, Ji Wu, Chao Zhang
• Subjects: cs.CL, cs.AI, cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.14716
• Pdf URL: https://arxiv.org/pdf/2404.14716
• Copy Paste: [[2404.14716]] Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities(https://arxiv.org/abs/2404.14716)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue history without any model parameter update. Despite such convenience, the performance of ICL heavily depends on the quality of the in-context examples presented, which makes the in-context example selection approach a critical choice. This paper proposes a novel Bayesian in-Context example Selection method (ByCS) for ICL. Extending the inference probability conditioned on in-context examples based on Bayes' theorem, ByCS focuses on the inverse inference conditioned on test input. Following the assumption that accurate inverse inference probability (likelihood) will result in accurate inference probability (posterior), in-context examples are selected based on their inverse inference results. Diverse and extensive cross-tasking and cross-modality experiments are performed with speech, text, and image examples. Experimental results show the efficacy and robustness of our ByCS method on various models, tasks and modalities.

Title: Incorporating Gradients to Rules: Towards Lightweight, Adaptive Provenance-based Intrusion Detection

• Authors: Lingzhi Wang, Xiangmin Shen, Weijian Li, Zhenyuan Li, R. Sekar, Han Liu, Yan Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.14720
• Pdf URL: https://arxiv.org/pdf/2404.14720
• Copy Paste: [[2404.14720]] Incorporating Gradients to Rules: Towards Lightweight, Adaptive Provenance-based Intrusion Detection(https://arxiv.org/abs/2404.14720)
• Keywords: attack, steal, explainability
• Abstract: As cyber-attacks become increasingly sophisticated and stealthy, it becomes more imperative and challenging to detect intrusion from normal behaviors. Through fine-grained causality analysis, provenance-based intrusion detection systems (PIDS) demonstrated a promising capacity to distinguish benign and malicious behaviors, attracting widespread attention from both industry and academia. Among diverse approaches, rule-based PIDS stands out due to its lightweight overhead, real-time capabilities, and explainability. However, existing rule-based systems suffer low detection accuracy, especially the high false alarms, due to the lack of fine-grained rules and environment-specific configurations. In this paper, we propose CAPTAIN, a rule-based PIDS capable of automatically adapting to diverse environments. Specifically, we propose three adaptive parameters to adjust the detection configuration with respect to nodes, edges, and alarm generation thresholds. We build a differentiable tag propagation framework and utilize the gradient descent algorithm to optimize these adaptive parameters based on the training data. We evaluate our system based on data from DARPA Engagement and simulated environments. The evaluation results demonstrate that CAPTAIN offers better detection accuracy, less detection latency, lower runtime overhead, and more interpretable detection alarms and knowledge compared to the SOTA PIDS.

Title: Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks

• Authors: Amir Saeidi, Shivanshu Verma, Chitta Baral
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14723
• Pdf URL: https://arxiv.org/pdf/2404.14723
• Copy Paste: [[2404.14723]] Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks(https://arxiv.org/abs/2404.14723)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across a spectrum of tasks. Recently, Direct Preference Optimization (DPO) has emerged as an RL-free approach to optimize the policy model on human preferences. However, several limitations hinder the widespread adoption of this method. To address these shortcomings, various versions of DPO have been introduced. Yet, a comprehensive evaluation of these variants across diverse tasks is still lacking. In this study, we aim to bridge this gap by investigating the performance of alignment methods across three distinct scenarios: (1) keeping the Supervised Fine-Tuning (SFT) part, (2) skipping the SFT part, and (3) skipping the SFT part and utilizing an instruction-tuned model. Furthermore, we explore the impact of different training sizes on their performance. Our evaluation spans a range of tasks including dialogue systems, reasoning, mathematical problem-solving, question answering, truthfulness, and multi-task understanding, encompassing 13 benchmarks such as MT-Bench, Big Bench, and Open LLM Leaderboard. Key observations reveal that alignment methods achieve optimal performance with smaller training data subsets, exhibit limited effectiveness in reasoning tasks yet significantly impact mathematical problem-solving, and employing an instruction-tuned model notably influences truthfulness. We anticipate that our findings will catalyze further research aimed at developing more robust models to address alignment challenges.

Title: Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering

• Authors: Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Kang Liu, Jun Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14741
• Pdf URL: https://arxiv.org/pdf/2404.14741
• Copy Paste: [[2404.14741]] Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering(https://arxiv.org/abs/2404.14741)
• Keywords: large language model
• Abstract: To address the issue of insufficient knowledge and the tendency to generate hallucination in Large Language Models (LLMs), numerous studies have endeavored to integrate LLMs with Knowledge Graphs (KGs). However, all these methods are evaluated on conventional Knowledge Graph Question Answering (KGQA) with complete KGs, where the factual triples involved in each question are entirely covered by the given KG. In this situation, LLM mainly acts as an agent to find answer entities by exploring the KG, rather than effectively integrating internal and external knowledge sources. However, in real-world scenarios, KGs are often incomplete to cover all the knowledge required to answer questions. To simulate real-world scenarios and evaluate the ability of LLMs to integrate internal and external knowledge, in this paper, we propose leveraging LLMs for QA under Incomplete Knowledge Graph (IKGQA), where the given KG doesn't include all the factual triples involved in each question. To handle IKGQA, we propose a training-free method called Generate-on-Graph (GoG) that can generate new factual triples while exploring on KGs. Specifically, we propose a selecting-generating-answering framework, which not only treat the LLM as an agent to explore on KGs, but also treat it as a KG to generate new facts based on the explored subgraph and its inherent knowledge. Experimental results on two datasets demonstrate that our GoG can solve IKGQA to a certain extent, while almost all previous methods cannot perform well on IKGQA.

Title: TAAT: Think and Act from Arbitrary Texts in Text2Motion

• Authors: Runqi Wang, Caoyuan Ma, GuoPeng Li, Zheng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14745
• Pdf URL: https://arxiv.org/pdf/2404.14745
• Copy Paste: [[2404.14745]] TAAT: Think and Act from Arbitrary Texts in Text2Motion(https://arxiv.org/abs/2404.14745)
• Keywords: large language model
• Abstract: Text2Motion aims to generate human motions from texts. Existing datasets rely on the assumption that texts include action labels (such as "walk, bend, and pick up"), which is not flexible for practical scenarios. This paper redefines this problem with a more realistic assumption that the texts are arbitrary. Specifically, arbitrary texts include existing action texts composed of action labels (e.g., A person walks and bends to pick up something), and introduce scene texts without explicit action labels (e.g., A person notices his wallet on the ground ahead). To bridge the gaps between this realistic setting and existing datasets, we expand the action texts on the HumanML3D dataset to more scene texts, thereby creating a new HumanML3D++ dataset including arbitrary texts. In this challenging dataset, we benchmark existing state-of-the-art methods and propose a novel two-stage framework to extract action labels from arbitrary texts by the Large Language Model (LLM) and then generate motions from action labels. Extensive experiments are conducted under different application scenarios to validate the effectiveness of the proposed framework on existing and proposed datasets. The results indicate that Text2Motion in this realistic setting is very challenging, fostering new research in this practical direction. Our dataset and code will be released.

Title: A Customer Level Fraudulent Activity Detection Benchmark for Enhancing Machine Learning Model Research and Evaluation

• Authors: Phoebe Jing, Yijing Gao, Xianlong Zeng
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.14746
• Pdf URL: https://arxiv.org/pdf/2404.14746
• Copy Paste: [[2404.14746]] A Customer Level Fraudulent Activity Detection Benchmark for Enhancing Machine Learning Model Research and Evaluation(https://arxiv.org/abs/2404.14746)
• Keywords: privacy
• Abstract: In the field of fraud detection, the availability of comprehensive and privacy-compliant datasets is crucial for advancing machine learning research and developing effective anti-fraud systems. Traditional datasets often focus on transaction-level information, which, while useful, overlooks the broader context of customer behavior patterns that are essential for detecting sophisticated fraud schemes. The scarcity of such data, primarily due to privacy concerns, significantly hampers the development and testing of predictive models that can operate effectively at the customer level. Addressing this gap, our study introduces a benchmark that contains structured datasets specifically designed for customer-level fraud detection. The benchmark not only adheres to strict privacy guidelines to ensure user confidentiality but also provides a rich source of information by encapsulating customer-centric features. We have developed the benchmark that allows for the comprehensive evaluation of various machine learning models, facilitating a deeper understanding of their strengths and weaknesses in predicting fraudulent activities. Through this work, we seek to bridge the existing gap in data availability, offering researchers and practitioners a valuable resource that empowers the development of next-generation fraud detection techniques.

Title: Differentiable Score-Based Likelihoods: Learning CT Motion Compensation From Clean Images

• Authors: Mareike Thies, Noah Maul, Siyuan Mei, Laura Pfaff, Nastassia Vysotskaya, Mingxuan Gu, Jonas Utz, Dennis Possart, Lukas Folle, Fabian Wagner, Andreas Maier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14747
• Pdf URL: https://arxiv.org/pdf/2404.14747
• Copy Paste: [[2404.14747]] Differentiable Score-Based Likelihoods: Learning CT Motion Compensation From Clean Images(https://arxiv.org/abs/2404.14747)
• Keywords: robust
• Abstract: Motion artifacts can compromise the diagnostic value of computed tomography (CT) images. Motion correction approaches require a per-scan estimation of patient-specific motion patterns. In this work, we train a score-based model to act as a probability density estimator for clean head CT images. Given the trained model, we quantify the deviation of a given motion-affected CT image from the ideal distribution through likelihood computation. We demonstrate that the likelihood can be utilized as a surrogate metric for motion artifact severity in the CT image facilitating the application of an iterative, gradient-based motion compensation algorithm. By optimizing the underlying motion parameters to maximize likelihood, our method effectively reduces motion artifacts, bringing the image closer to the distribution of motion-free scans. Our approach achieves comparable performance to state-of-the-art methods while eliminating the need for a representative data set of motion-affected samples. This is particularly advantageous in real-world applications, where patient motion patterns may exhibit unforeseen variability, ensuring robustness without implicit assumptions about recoverable motion types.

Title: Grounded Knowledge-Enhanced Medical VLP for Chest X-Ray

• Authors: Qiao Deng, Zhongzhen Huang, Yunqi Wang, Zhichuan Wang, Zhao Wang, Xiaofan Zhang, Qi Dou, Yeung Yu Hui, Edward S.Hui
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14750
• Pdf URL: https://arxiv.org/pdf/2404.14750
• Copy Paste: [[2404.14750]] Grounded Knowledge-Enhanced Medical VLP for Chest X-Ray(https://arxiv.org/abs/2404.14750)
• Keywords: transformer
• Abstract: Medical vision-language pre-training has emerged as a promising approach for learning domain-general representations of medical image and text. Current algorithms that exploit the global and local alignment between medical image and text could however be marred by the redundant information in medical data. To address this issue, we propose a grounded knowledge-enhanced medical vision-language pre-training (GK-MVLP) framework for chest X-ray. In this framework, medical knowledge is grounded to the appropriate anatomical regions by using a transformer-based grounded knowledge-enhanced module for fine-grained alignment between anatomical region-level visual features and the textural features of medical knowledge. The performance of GK-MVLP is competitive with or exceeds the state of the art on downstream chest X-ray disease classification, disease localization, report generation, and medical visual question-answering tasks. Our results show the advantage of incorporating grounding mechanism to remove biases and improve the alignment between chest X-ray image and radiology report.

Title: Skip the Benchmark: Generating System-Level High-Level Synthesis Data using Generative Machine Learning

• Authors: Yuchao Liao, Tosiron Adegbija, Roman Lysecky, Ravi Tandon
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2404.14754
• Pdf URL: https://arxiv.org/pdf/2404.14754
• Copy Paste: [[2404.14754]] Skip the Benchmark: Generating System-Level High-Level Synthesis Data using Generative Machine Learning(https://arxiv.org/abs/2404.14754)
• Keywords: robust, generative
• Abstract: High-Level Synthesis (HLS) Design Space Exploration (DSE) is a widely accepted approach for efficiently exploring Pareto-optimal and optimal hardware solutions during the HLS process. Several HLS benchmarks and datasets are available for the research community to evaluate their methodologies. Unfortunately, these resources are limited and may not be sufficient for complex, multi-component system-level explorations. Generating new data using existing HLS benchmarks can be cumbersome, given the expertise and time required to effectively generate data for different HLS designs and directives. As a result, synthetic data has been used in prior work to evaluate system-level HLS DSE. However, the fidelity of the synthetic data to real data is often unclear, leading to uncertainty about the quality of system-level HLS DSE. This paper proposes a novel approach, called Vaegan, that employs generative machine learning to generate synthetic data that is robust enough to support complex system-level HLS DSE experiments that would be unattainable with only the currently available data. We explore and adapt a Variational Autoencoder (VAE) and Generative Adversarial Network (GAN) for this task and evaluate our approach using state-of-the-art datasets and metrics. We compare our approach to prior works and show that Vaegan effectively generates synthetic HLS data that closely mirrors the ground truth's distribution.

Title: Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting

• Authors: Xiongxiao Xu, Yueqing Liang, Baixiang Huang, Zhiling Lan, Kai Shu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14757
• Pdf URL: https://arxiv.org/pdf/2404.14757
• Copy Paste: [[2404.14757]] Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting(https://arxiv.org/abs/2404.14757)
• Keywords: transformer
• Abstract: Time series forecasting is an important problem and plays a key role in a variety of applications including weather forecasting, stock market, and scientific simulations. Although transformers have proven to be effective in capturing dependency, its quadratic complexity of attention mechanism prevents its further adoption in long-range time series forecasting, thus limiting them attend to short-range range. Recent progress on state space models (SSMs) have shown impressive performance on modeling long range dependency due to their subquadratic complexity. Mamba, as a representative SSM, enjoys linear time complexity and has achieved strong scalability on tasks that requires scaling to long sequences, such as language, audio, and genomics. In this paper, we propose to leverage a hybrid framework Mambaformer that internally combines Mamba for long-range dependency, and Transformer for short range dependency, for long-short range forecasting. To the best of our knowledge, this is the first paper to combine Mamba and Transformer architecture in time series data. We investigate possible hybrid architectures to combine Mamba layer and attention layer for long-short range time series forecasting. The comparative study shows that the Mambaformer family can outperform Mamba and Transformer in long-short range time series forecasting problem. The code is available at https://github.com/XiongxiaoXu/Mambaformerin-Time-Series.

Title: Retrieval Augmented Generation for Domain-specific Question Answering

• Authors: Sanat Sharma, David Seunghyun Yoon, Franck Dernoncourt, Dewang Sultania, Karishma Bagga, Mengjiao Zhang, Trung Bui, Varun Kotte
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14760
• Pdf URL: https://arxiv.org/pdf/2404.14760
• Copy Paste: [[2404.14760]] Retrieval Augmented Generation for Domain-specific Question Answering(https://arxiv.org/abs/2404.14760)
• Keywords: large language model
• Abstract: Question answering (QA) has become an important application in the advanced development of large language models. General pre-trained large language models for question-answering are not trained to properly understand the knowledge or terminology for a specific domain, such as finance, healthcare, education, and customer service for a product. To better cater to domain-specific understanding, we build an in-house question-answering system for Adobe products. We propose a novel framework to compile a large question-answer database and develop the approach for retrieval-aware finetuning of a Large Language model. We showcase that fine-tuning the retriever leads to major improvements in the final generation. Our overall approach reduces hallucinations during generation while keeping in context the latest retrieval information for contextual grounding.

Title: Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusion

• Authors: Hongyu Chen, Yiqi Gao, Min Zhou, Peng Wang, Xubin Li, Tiezheng Ge, Bo Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14768
• Pdf URL: https://arxiv.org/pdf/2404.14768
• Copy Paste: [[2404.14768]] Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusion(https://arxiv.org/abs/2404.14768)
• Keywords: diffusion
• Abstract: Recently, integrating visual controls into text-to-image~(T2I) models, such as ControlNet method, has received significant attention for finer control capabilities. While various training-free methods make efforts to enhance prompt following in T2I models, the issue with visual control is still rarely studied, especially in the scenario that visual controls are misaligned with text prompts. In this paper, we address the challenge of ``Prompt Following With Visual Control" and propose a training-free approach named Mask-guided Prompt Following (MGPF). Object masks are introduced to distinct aligned and misaligned parts of visual controls and prompts. Meanwhile, a network, dubbed as Masked ControlNet, is designed to utilize these object masks for object generation in the misaligned visual control region. Further, to improve attribute matching, a simple yet efficient loss is designed to align the attention maps of attributes with object regions constrained by ControlNet and object masks. The efficacy and superiority of MGPF are validated through comprehensive quantitative and qualitative experiments.

Title: Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models

• Authors: Chris Samarinas, Pracha Promthaw, Atharva Nijasure, Hansi Zeng, Julian Killingback, Hamed Zamani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14772
• Pdf URL: https://arxiv.org/pdf/2404.14772
• Copy Paste: [[2404.14772]] Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models(https://arxiv.org/abs/2404.14772)
• Keywords: large language model
• Abstract: This paper explores SynTOD, a new synthetic data generation approach for developing end-to-end Task-Oriented Dialogue (TOD) Systems capable of handling complex tasks such as intent classification, slot filling, conversational question-answering, and retrieval-augmented response generation, without relying on crowdsourcing or real-world data. SynTOD utilizes a state transition graph to define the desired behavior of a TOD system and generates diverse, structured conversations through random walks and response simulation using large language models (LLMs). In our experiments, using graph-guided response simulations leads to significant improvements in intent classification, slot filling and response relevance compared to naive single-prompt simulated conversations. We also investigate the end-to-end TOD effectiveness of different base and instruction-tuned LLMs, with and without the constructed synthetic conversations. Finally, we explore how various LLMs can evaluate responses in a TOD system and how well they are correlated with human judgments. Our findings pave the path towards quick development and evaluation of domain-specific TOD systems. We release our datasets, models, and code for research purposes.

Title: CT-Agent: Clinical Trial Multi-Agent with Large Language Model-based Reasoning

• Authors: Ling Yue, Tianfan Fu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14777
• Pdf URL: https://arxiv.org/pdf/2404.14777
• Copy Paste: [[2404.14777]] CT-Agent: Clinical Trial Multi-Agent with Large Language Model-based Reasoning(https://arxiv.org/abs/2404.14777)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) and multi-agent systems have shown impressive capabilities in natural language tasks but face challenges in clinical trial applications, primarily due to limited access to external knowledge. Recognizing the potential of advanced clinical trial tools that aggregate and predict based on the latest medical data, we propose an integrated solution to enhance their accessibility and utility. We introduce Clinical Agent System (CT-Agent), a Clinical multi-agent system designed for clinical trial tasks, leveraging GPT-4, multi-agent architectures, LEAST-TO-MOST, and ReAct reasoning technology. This integration not only boosts LLM performance in clinical contexts but also introduces novel functionalities. Our system autonomously manages the entire clinical trial process, demonstrating significant efficiency improvements in our evaluations, which include both computational benchmarks and expert feedback.

Title: Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approaches

• Authors: Clément Christophe, Praveen K Kanithi, Prateek Munjal, Tathagata Raha, Nasir Hayat, Ronnie Rajan, Ahmed Al-Mahrooqi, Avani Gupta, Muhammad Umar Salman, Gurpreet Gosal, Bhargav Kanakiya, Charles Chen, Natalia Vassilieva, Boulbaba Ben Amor, Marco AF Pimentel, Shadab Khan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14779
• Pdf URL: https://arxiv.org/pdf/2404.14779
• Copy Paste: [[2404.14779]] Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approaches(https://arxiv.org/abs/2404.14779)
• Keywords: large language model
• Abstract: This study presents a comprehensive analysis and comparison of two predominant fine-tuning methodologies - full-parameter fine-tuning and parameter-efficient tuning - within the context of medical Large Language Models (LLMs). We developed and refined a series of LLMs, based on the Llama-2 architecture, specifically designed to enhance medical knowledge retrieval, reasoning, and question-answering capabilities. Our experiments systematically evaluate the effectiveness of these tuning strategies across various well-known medical benchmarks. Notably, our medical LLM Med42 showed an accuracy level of 72% on the US Medical Licensing Examination (USMLE) datasets, setting a new standard in performance for openly available medical LLMs. Through this comparative analysis, we aim to identify the most effective and efficient method for fine-tuning LLMs in the medical domain, thereby contributing significantly to the advancement of AI-driven healthcare applications.

Title: ContextualFusion: Context-Based Multi-Sensor Fusion for 3D Object Detection in Adverse Operating Conditions

• Authors: Shounak Sural, Nishad Sahu, Ragunathan (Raj)Rajkumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14780
• Pdf URL: https://arxiv.org/pdf/2404.14780
• Copy Paste: [[2404.14780]] ContextualFusion: Context-Based Multi-Sensor Fusion for 3D Object Detection in Adverse Operating Conditions(https://arxiv.org/abs/2404.14780)
• Keywords: robust
• Abstract: The fusion of multimodal sensor data streams such as camera images and lidar point clouds plays an important role in the operation of autonomous vehicles (AVs). Robust perception across a range of adverse weather and lighting conditions is specifically required for AVs to be deployed widely. While multi-sensor fusion networks have been previously developed for perception in sunny and clear weather conditions, these methods show a significant degradation in performance under night-time and poor weather conditions. In this paper, we propose a simple yet effective technique called ContextualFusion to incorporate the domain knowledge about cameras and lidars behaving differently across lighting and weather variations into 3D object detection models. Specifically, we design a Gated Convolutional Fusion (GatedConv) approach for the fusion of sensor streams based on the operational context. To aid in our evaluation, we use the open-source simulator CARLA to create a multimodal adverse-condition dataset called AdverseOp3D to address the shortcomings of existing datasets being biased towards daytime and good-weather conditions. Our ContextualFusion approach yields an mAP improvement of 6.2% over state-of-the-art methods on our context-balanced synthetic dataset. Finally, our method enhances state-of-the-art 3D objection performance at night on the real-world NuScenes dataset with a significant mAP improvement of 11.7%.

Title: Talk Too Much: Poisoning Large Language Models under Token Limit

• Authors: Jiaming He, Wenbo Jiang, Guanyu Hou, Wenshu Fan, Rui Zhang, Hongwei Li
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14795
• Pdf URL: https://arxiv.org/pdf/2404.14795
• Copy Paste: [[2404.14795]] Talk Too Much: Poisoning Large Language Models under Token Limit(https://arxiv.org/abs/2404.14795)
• Keywords: attack, steal, large language model
• Abstract: Mainstream poisoning attacks on large language models (LLMs) typically set a fixed trigger in the input instance and specific responses for triggered queries. However, the fixed trigger setting (e.g., unusual words) may be easily detected by human detection, limiting the effectiveness and practicality in real-world scenarios. To enhance the stealthiness of the trigger, we present a poisoning attack against LLMs that is triggered by a generation/output condition-token limitation, which is a commonly adopted strategy by users for reducing costs. The poisoned model performs normally for output without token limitation, while becomes harmful for output with limited tokens. To achieve this objective, we introduce BrieFool, an efficient attack framework. It leverages the characteristics of generation limitation by efficient instruction sampling and poisoning data generation, thereby influencing the behavior of LLMs under target conditions. Our experiments demonstrate that BrieFool is effective across safety domains and knowledge domains. For instance, with only 20 generated poisoning examples against GPT-3.5-turbo, BrieFool achieves a 100% Attack Success Rate (ASR) and a 9.28/10 average Harmfulness Score (HS) under token limitation conditions while maintaining the benign performance.

Title: DesignProbe: A Graphic Design Benchmark for Multimodal Large Language Models

• Authors: Jieru Lin, Danqing Huang, Tiejun Zhao, Dechen Zhan, Chin-Yew Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14801
• Pdf URL: https://arxiv.org/pdf/2404.14801
• Copy Paste: [[2404.14801]] DesignProbe: A Graphic Design Benchmark for Multimodal Large Language Models(https://arxiv.org/abs/2404.14801)
• Keywords: large language model
• Abstract: A well-executed graphic design typically achieves harmony in two levels, from the fine-grained design elements (color, font and layout) to the overall design. This complexity makes the comprehension of graphic design challenging, for it needs the capability to both recognize the design elements and understand the design. With the rapid development of Multimodal Large Language Models (MLLMs), we establish the DesignProbe, a benchmark to investigate the capability of MLLMs in design. Our benchmark includes eight tasks in total, across both the fine-grained element level and the overall design level. At design element level, we consider both the attribute recognition and semantic understanding tasks. At overall design level, we include style and metaphor. 9 MLLMs are tested and we apply GPT-4 as evaluator. Besides, further experiments indicates that refining prompts can enhance the performance of MLLMs. We first rewrite the prompts by different LLMs and found increased performances appear in those who self-refined by their own LLMs. We then add extra task knowledge in two different ways (text descriptions and image examples), finding that adding images boost much more performance over texts.

Title: Reference-Free Multi-Modality Volume Registration of X-Ray Microscopy and Light-Sheet Fluorescence Microscopy

• Authors: Siyuan Mei, Fuxin Fan, Mareike Thies, Mingxuan Gu, Fabian Wagner, Oliver Aust, Ina Erceg, Zeynab Mirzaei, Georgiana Neag, Yipeng Sun, Yixing Huang, Andreas Maier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14807
• Pdf URL: https://arxiv.org/pdf/2404.14807
• Copy Paste: [[2404.14807]] Reference-Free Multi-Modality Volume Registration of X-Ray Microscopy and Light-Sheet Fluorescence Microscopy(https://arxiv.org/abs/2404.14807)
• Keywords: robust
• Abstract: Recently, X-ray microscopy (XRM) and light-sheet fluorescence microscopy (LSFM) have emerged as two pivotal imaging tools in preclinical research on bone remodeling diseases, offering micrometer-level resolution. Integrating these complementary modalities provides a holistic view of bone microstructures, facilitating function-oriented volume analysis across different disease cycles. However, registering such independently acquired large-scale volumes is extremely challenging under real and reference-free scenarios. This paper presents a fast two-stage pipeline for volume registration of XRM and LSFM. The first stage extracts the surface features and employs two successive point cloud-based methods for coarse alignment. The second stage fine-tunes the initial alignment using a modified cross-correlation method, ensuring precise volumetric registration. Moreover, we propose residual similarity as a novel metric to assess the alignment of two complementary modalities. The results imply robust gradual improvement across the stages. In the end, all correlating microstructures, particularly lacunae in XRM and bone cells in LSFM, are precisely matched, enabling new insights into bone diseases like osteoporosis which are a substantial burden in aging societies.

Title: Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning

• Authors: Wenjin Hou, Shiming Chen, Shuhuang Chen, Ziming Hong, Yan Wang, Xuetao Feng, Salman Khan, Fahad Shahbaz Khan, Xinge You
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14808
• Pdf URL: https://arxiv.org/pdf/2404.14808
• Copy Paste: [[2404.14808]] Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning(https://arxiv.org/abs/2404.14808)
• Keywords: generative
• Abstract: Generative Zero-shot learning (ZSL) learns a generator to synthesize visual samples for unseen classes, which is an effective way to advance ZSL. However, existing generative methods rely on the conditions of Gaussian noise and the predefined semantic prototype, which limit the generator only optimized on specific seen classes rather than characterizing each visual instance, resulting in poor generalizations (\textit{e.g.}, overfitting to seen classes). To address this issue, we propose a novel Visual-Augmented Dynamic Semantic prototype method (termed VADS) to boost the generator to learn accurate semantic-visual mapping by fully exploiting the visual-augmented knowledge into semantic conditions. In detail, VADS consists of two modules: (1) Visual-aware Domain Knowledge Learning module (VDKL) learns the local bias and global prior of the visual features (referred to as domain visual knowledge), which replace pure Gaussian noise to provide richer prior noise information; (2) Vision-Oriented Semantic Updation module (VOSU) updates the semantic prototype according to the visual representations of the samples. Ultimately, we concatenate their output as a dynamic semantic prototype, which serves as the condition of the generator. Extensive experiments demonstrate that our VADS achieves superior CZSL and GZSL performances on three prominent datasets and outperforms other state-of-the-art methods with averaging increases by 6.4\%, 5.9\% and 4.2\% on SUN, CUB and AWA2, respectively.

Title: A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications

• Authors: Wenbo Shang, Xin Huang
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2404.14809
• Pdf URL: https://arxiv.org/pdf/2404.14809
• Copy Paste: [[2404.14809]] A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications(https://arxiv.org/abs/2404.14809)
• Keywords: generative, large language model
• Abstract: A graph is a fundamental data model to represent various entities and their complex relationships in society and nature, such as social networks, transportation networks, financial networks, and biomedical systems. Recently, large language models (LLMs) have showcased a strong generalization ability to handle various NLP and multi-mode tasks to answer users' arbitrary questions and specific-domain content generation. Compared with graph learning models, LLMs enjoy superior advantages in addressing the challenges of generalizing graph tasks by eliminating the need for training graph learning models and reducing the cost of manual annotation. In this survey, we conduct a comprehensive investigation of existing LLM studies on graph data, which summarizes the relevant graph analytics tasks solved by advanced LLM models and points out the existing remaining challenges and future directions. Specifically, we study the key problems of LLM-based generative graph analytics (LLM-GGA) with three categories: LLM-based graph query processing (LLM-GQP), LLM-based graph inference and learning (LLM-GIL), and graph-LLM-based applications. LLM-GQP focuses on an integration of graph analytics techniques and LLM prompts, including graph understanding and knowledge graph (KG) based augmented retrieval, while LLM-GIL focuses on learning and reasoning over graphs, including graph learning, graph-formed reasoning and graph representation. We summarize the useful prompts incorporated into LLM to handle different graph downstream tasks. Moreover, we give a summary of LLM model evaluation, benchmark datasets/tasks, and a deep pro and cons analysis of LLM models. We also explore open problems and future directions in this exciting interdisciplinary research area of LLMs and graph analytics.

Title: Pattern-Aware Chain-of-Thought Prompting in Large Language Models

• Authors: Yufeng Zhang, Xuepeng Wang, Lingxiang Wu, Jinqiao Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14812
• Pdf URL: https://arxiv.org/pdf/2404.14812
• Copy Paste: [[2404.14812]] Pattern-Aware Chain-of-Thought Prompting in Large Language Models(https://arxiv.org/abs/2404.14812)
• Keywords: robust, large language model
• Abstract: Chain-of-thought (CoT) prompting can guide language models to engage in complex multi-step reasoning. The quality of provided demonstrations significantly impacts the success of downstream inference tasks. While existing automated methods prioritize accuracy and semantics in these demonstrations, we show that the underlying reasoning patterns play a more crucial role in such tasks. In this paper, we propose Pattern-Aware CoT, a prompting method that considers the diversity of demonstration patterns. By incorporating patterns such as step length and reasoning process within intermediate steps, PA-CoT effectively mitigates the issue of bias induced by demonstrations and enables better generalization to diverse scenarios. We conduct experiments on nine reasoning benchmark tasks using two open-source LLMs. The results show that our method substantially enhances reasoning performance and exhibits robustness to errors. The code will be made publicly available.

Title: Time-aware Heterogeneous Graph Transformer with Adaptive Attention Merging for Health Event Prediction

• Authors: Shibo Li, Hengliang Cheng, Runze Li, Weihua Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14815
• Pdf URL: https://arxiv.org/pdf/2404.14815
• Copy Paste: [[2404.14815]] Time-aware Heterogeneous Graph Transformer with Adaptive Attention Merging for Health Event Prediction(https://arxiv.org/abs/2404.14815)
• Keywords: interpretability, transformer
• Abstract: The widespread application of Electronic Health Records (EHR) data in the medical field has led to early successes in disease risk prediction using deep learning methods. These methods typically require extensive data for training due to their large parameter sets. However, existing works do not exploit the full potential of EHR data. A significant challenge arises from the infrequent occurrence of many medical codes within EHR data, limiting their clinical applicability. Current research often lacks in critical areas: 1) incorporating disease domain knowledge; 2) heterogeneously learning disease representations with rich meanings; 3) capturing the temporal dynamics of disease progression. To overcome these limitations, we introduce a novel heterogeneous graph learning model designed to assimilate disease domain knowledge and elucidate the intricate relationships between drugs and diseases. This model innovatively incorporates temporal data into visit-level embeddings and leverages a time-aware transformer alongside an adaptive attention mechanism to produce patient representations. When evaluated on two healthcare datasets, our approach demonstrated notable enhancements in both prediction accuracy and interpretability over existing methodologies, signifying a substantial advancement towards personalized and proactive healthcare management.

Title: CoProNN: Concept-based Prototypical Nearest Neighbors for Explaining Vision Models

• Authors: Teodor Chiaburu, Frank Haußer, Felix Bießmann
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14830
• Pdf URL: https://arxiv.org/pdf/2404.14830
• Copy Paste: [[2404.14830]] CoProNN: Concept-based Prototypical Nearest Neighbors for Explaining Vision Models(https://arxiv.org/abs/2404.14830)
• Keywords: explainability, generative
• Abstract: Mounting evidence in explainability for artificial intelligence (XAI) research suggests that good explanations should be tailored to individual tasks and should relate to concepts relevant to the task. However, building task specific explanations is time consuming and requires domain expertise which can be difficult to integrate into generic XAI methods. A promising approach towards designing useful task specific explanations with domain experts is based on compositionality of semantic concepts. Here, we present a novel approach that enables domain experts to quickly create concept-based explanations for computer vision tasks intuitively via natural language. Leveraging recent progress in deep generative methods we propose to generate visual concept-based prototypes via text-to-image methods. These prototypes are then used to explain predictions of computer vision models via a simple k-Nearest-Neighbors routine. The modular design of CoProNN is simple to implement, it is straightforward to adapt to novel tasks and allows for replacing the classification and text-to-image models as more powerful models are released. The approach can be evaluated offline against the ground-truth of predefined prototypes that can be easily communicated also to domain experts as they are based on visual concepts. We show that our strategy competes very well with other concept-based XAI approaches on coarse grained image classification tasks and may even outperform those methods on more demanding fine grained tasks. We demonstrate the effectiveness of our method for human-machine collaboration settings in qualitative and quantitative user studies. All code and experimental data can be found in our GitHub $\href{https://github.com/TeodorChiaburu/beexplainable}{repository}$.

Title: Semi-supervised 2D Human Pose Estimation via Adaptive Keypoint Masking

• Authors: Kexin Meng, Ruirui Li, Daguang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14835
• Pdf URL: https://arxiv.org/pdf/2404.14835
• Copy Paste: [[2404.14835]] Semi-supervised 2D Human Pose Estimation via Adaptive Keypoint Masking(https://arxiv.org/abs/2404.14835)
• Keywords: robust
• Abstract: Human pose estimation is a fundamental and challenging task in computer vision. Larger-scale and more accurate keypoint annotations, while helpful for improving the accuracy of supervised pose estimation, are often expensive and difficult to obtain. Semi-supervised pose estimation tries to leverage a large amount of unlabeled data to improve model performance, which can alleviate the problem of insufficient labeled samples. The latest semi-supervised learning usually adopts a strong and weak data augmented teacher-student learning framework to deal with the challenge of "Human postural diversity and its long-tailed distribution". Appropriate data augmentation method is one of the key factors affecting the accuracy and generalization of semi-supervised models. Aiming at the problem that the difference of sample learning is not considered in the fixed keypoint masking augmentation method, this paper proposes an adaptive keypoint masking method, which can fully mine the information in the samples and obtain better estimation performance. In order to further improve the generalization and robustness of the model, this paper proposes a dual-branch data augmentation scheme, which can perform Mixup on samples and features on the basis of adaptive keypoint masking. The effectiveness of the proposed method is verified on COCO and MPII, outperforming the state-of-the-art semi-supervised pose estimation by 5.2% and 0.3%, respectively.

Title: Ultrasound Nodule Segmentation Using Asymmetric Learning with Simple Clinical Annotation

• Authors: Xingyue Zhao, Zhongyu Li, Xiangde Luo, Peiqi Li, Peng Huang, Jianwei Zhu, Yang Liu, Jihua Zhu, Meng Yang, Shi Chang, Jun Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14852
• Pdf URL: https://arxiv.org/pdf/2404.14852
• Copy Paste: [[2404.14852]] Ultrasound Nodule Segmentation Using Asymmetric Learning with Simple Clinical Annotation(https://arxiv.org/abs/2404.14852)
• Keywords: segmentation
• Abstract: Recent advances in deep learning have greatly facilitated the automated segmentation of ultrasound images, which is essential for nodule morphological analysis. Nevertheless, most existing methods depend on extensive and precise annotations by domain experts, which are labor-intensive and time-consuming. In this study, we suggest using simple aspect ratio annotations directly from ultrasound clinical diagnoses for automated nodule segmentation. Especially, an asymmetric learning framework is developed by extending the aspect ratio annotations with two types of pseudo labels, i.e., conservative labels and radical labels, to train two asymmetric segmentation networks simultaneously. Subsequently, a conservative-radical-balance strategy (CRBS) strategy is proposed to complementally combine radical and conservative labels. An inconsistency-aware dynamically mixed pseudo-labels supervision (IDMPS) module is introduced to address the challenges of over-segmentation and under-segmentation caused by the two types of labels. To further leverage the spatial prior knowledge provided by clinical annotations, we also present a novel loss function namely the clinical anatomy prior loss. Extensive experiments on two clinically collected ultrasound datasets (thyroid and breast) demonstrate the superior performance of our proposed method, which can achieve comparable and even better performance than fully supervised methods using ground truth annotations.

Title: Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans

• Authors: Vittoria Dentella, Fritz Guenther, Evelina Leivada
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14883
• Pdf URL: https://arxiv.org/pdf/2404.14883
• Copy Paste: [[2404.14883]] Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans(https://arxiv.org/abs/2404.14883)
• Keywords: large language model
• Abstract: Understanding the limits of language is a prerequisite for Large Language Models (LLMs) to act as theories of natural language. LLM performance in some language tasks presents both quantitative and qualitative differences from that of humans, however it remains to be determined whether such differences are amenable to model size. This work investigates the critical role of model scaling, determining whether increases in size make up for such differences between humans and models. We test three LLMs from different families (Bard, 137 billion parameters; ChatGPT-3.5, 175 billion; ChatGPT-4, 1.5 trillion) on a grammaticality judgment task featuring anaphora, center embedding, comparatives, and negative polarity. N=1,200 judgments are collected and scored for accuracy, stability, and improvements in accuracy upon repeated presentation of a prompt. Results of the best performing LLM, ChatGPT-4, are compared to results of n=80 humans on the same stimuli. We find that increased model size may lead to better performance, but LLMs are still not sensitive to (un)grammaticality as humans are. It seems possible but unlikely that scaling alone can fix this issue. We interpret these results by comparing language learning in vivo and in silico, identifying three critical differences concerning (i) the type of evidence, (ii) the poverty of the stimulus, and (iii) the occurrence of semantic hallucinations due to impenetrable linguistic reference.

Title: DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition

• Authors: Haozhe Cheng, Cheng Ju, Haicheng Wang, Jinxiang Liu, Mengting Chen, Qiang Hu, Xiaoyun Zhang, Yanfeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14890
• Pdf URL: https://arxiv.org/pdf/2404.14890
• Copy Paste: [[2404.14890]] DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition(https://arxiv.org/abs/2404.14890)
• Keywords: robust, generative
• Abstract: As one of the fundamental video tasks in computer vision, Open-Vocabulary Action Recognition (OVAR) recently gains increasing attention, with the development of vision-language pre-trainings. To enable generalization of arbitrary classes, existing methods treat class labels as text descriptions, then formulate OVAR as evaluating embedding similarity between visual samples and textual classes. However, one crucial issue is completely ignored: the class descriptions given by users may be noisy, e.g., misspellings and typos, limiting the real-world practicality of vanilla OVAR. To fill the research gap, this paper pioneers to evaluate existing methods by simulating multi-level noises of various types, and reveals their poor robustness. To tackle the noisy OVAR task, we further propose one novel DENOISER framework, covering two parts: generation and discrimination. Concretely, the generative part denoises noisy class-text names via one decoding process, i.e., propose text candidates, then utilize inter-modal and intra-modal information to vote for the best. At the discriminative part, we use vanilla OVAR models to assign visual samples to class-text names, thus obtaining more semantics. For optimization, we alternately iterate between generative and discriminative parts for progressive refinements. The denoised text classes help OVAR models classify visual samples more accurately; in return, classified visual samples help better denoising. On three datasets, we carry out extensive experiments to show our superior robustness, and thorough ablations to dissect the effectiveness of each component.

Title: Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models

• Authors: Chen Zhang, Zhuorui Liu, Dawei Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14897
• Pdf URL: https://arxiv.org/pdf/2404.14897
• Copy Paste: [[2404.14897]] Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models(https://arxiv.org/abs/2404.14897)
• Keywords: large language model
• Abstract: With the increasingly giant scales of (causal) large language models (LLMs), the inference efficiency comes as one of the core concerns along the improved performance. In contrast to the memory footprint, the latency bottleneck seems to be of greater importance as there can be billions of requests to a LLM (e.g., GPT-4) per day. The bottleneck is mainly due to the autoregressive innateness of LLMs, where tokens can only be generated sequentially during decoding. To alleviate the bottleneck, the idea of speculative execution, which originates from the field of computer architecture, is introduced to LLM decoding in a \textit{draft-then-verify} style. Under this regime, a sequence of tokens will be drafted in a fast pace by utilizing some heuristics, and then the tokens shall be verified in parallel by the LLM. As the costly sequential inference is parallelized, LLM decoding speed can be significantly boosted. Driven by the success of LLMs in recent couple of years, a growing literature in this direction has emerged. Yet, there lacks a position survey to summarize the current landscape and draw a roadmap for future development of this promising area. To meet this demand, we present the very first survey paper that reviews and unifies literature of speculative execution in LLMs (e.g., blockwise parallel decoding, speculative decoding, etc.) in a comprehensive framework and a systematic taxonomy. Based on the taxonomy, we present a critical review and comparative analysis of the current arts. Finally we highlight various key challenges and future directions to further develop the area.

Title: Driver Activity Classification Using Generalizable Representations from Vision-Language Models

• Authors: Ross Greer, Mathias Viborg Andersen, Andreas Møgelmose, Mohan Trivedi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14906
• Pdf URL: https://arxiv.org/pdf/2404.14906
• Copy Paste: [[2404.14906]] Driver Activity Classification Using Generalizable Representations from Vision-Language Models(https://arxiv.org/abs/2404.14906)
• Keywords: robust, interpretability
• Abstract: Driver activity classification is crucial for ensuring road safety, with applications ranging from driver assistance systems to autonomous vehicle control transitions. In this paper, we present a novel approach leveraging generalizable representations from vision-language models for driver activity classification. Our method employs a Semantic Representation Late Fusion Neural Network (SRLF-Net) to process synchronized video frames from multiple perspectives. Each frame is encoded using a pretrained vision-language encoder, and the resulting embeddings are fused to generate class probability predictions. By leveraging contrastively-learned vision-language representations, our approach achieves robust performance across diverse driver activities. We evaluate our method on the Naturalistic Driving Action Recognition Dataset, demonstrating strong accuracy across many classes. Our results suggest that vision-language representations offer a promising avenue for driver monitoring systems, providing both accuracy and interpretability through natural language descriptors.

Title: Pillars of Grammatical Error Correction: Comprehensive Inspection Of Contemporary Approaches In The Era of Large Language Models

• Authors: Kostiantyn Omelianchuk, Andrii Liubonko, Oleksandr Skurzhanskyi, Artem Chernodub, Oleksandr Korniienko, Igor Samokhin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14914
• Pdf URL: https://arxiv.org/pdf/2404.14914
• Copy Paste: [[2404.14914]] Pillars of Grammatical Error Correction: Comprehensive Inspection Of Contemporary Approaches In The Era of Large Language Models(https://arxiv.org/abs/2404.14914)
• Keywords: large language model
• Abstract: In this paper, we carry out experimental research on Grammatical Error Correction, delving into the nuances of single-model systems, comparing the efficiency of ensembling and ranking methods, and exploring the application of large language models to GEC as single-model systems, as parts of ensembles, and as ranking methods. We set new state-of-the-art performance with F_0.5 scores of 72.8 on CoNLL-2014-test and 81.4 on BEA-test, respectively. To support further advancements in GEC and ensure the reproducibility of our research, we make our code, trained models, and systems' outputs publicly available.

Title: Graph Machine Learning in the Era of Large Language Models (LLMs)

• Authors: Wenqi Fan, Shijie Wang, Jiani Huang, Zhikai Chen, Yu Song, Wenzhuo Tang, Haitao Mao, Hui Liu, Xiaorui Liu, Dawei Yin, Qing Li
• Subjects: cs.LG, cs.AI, cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.14928
• Pdf URL: https://arxiv.org/pdf/2404.14928
• Copy Paste: [[2404.14928]] Graph Machine Learning in the Era of Large Language Models (LLMs)(https://arxiv.org/abs/2404.14928)
• Keywords: explainability, large language model
• Abstract: Graphs play an important role in representing complex relationships in various domains like social networks, knowledge graphs, and molecular discovery. With the advent of deep learning, Graph Neural Networks (GNNs) have emerged as a cornerstone in Graph Machine Learning (Graph ML), facilitating the representation and processing of graph structures. Recently, LLMs have demonstrated unprecedented capabilities in language tasks and are widely adopted in a variety of applications such as computer vision and recommender systems. This remarkable success has also attracted interest in applying LLMs to the graph domain. Increasing efforts have been made to explore the potential of LLMs in advancing Graph ML's generalization, transferability, and few-shot learning ability. Meanwhile, graphs, especially knowledge graphs, are rich in reliable factual knowledge, which can be utilized to enhance the reasoning capabilities of LLMs and potentially alleviate their limitations such as hallucinations and the lack of explainability. Given the rapid progress of this research direction, a systematic review summarizing the latest advancements for Graph ML in the era of LLMs is necessary to provide an in-depth understanding to researchers and practitioners. Therefore, in this survey, we first review the recent developments in Graph ML. We then explore how LLMs can be utilized to enhance the quality of graph features, alleviate the reliance on labeled data, and address challenges such as graph heterogeneity and out-of-distribution (OOD) generalization. Afterward, we delve into how graphs can enhance LLMs, highlighting their abilities to enhance LLM pre-training and inference. Furthermore, we investigate various applications and discuss the potential future directions in this promising field.

Title: Fin-Fed-OD: Federated Outlier Detection on Financial Tabular Data

• Authors: Dayananda Herurkar, Sebastian Palacio, Ahmed Anwar, Joern Hees, Andreas Dengel
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14933
• Pdf URL: https://arxiv.org/pdf/2404.14933
• Copy Paste: [[2404.14933]] Fin-Fed-OD: Federated Outlier Detection on Financial Tabular Data(https://arxiv.org/abs/2404.14933)
• Keywords: privacy, robust, federate
• Abstract: Anomaly detection in real-world scenarios poses challenges due to dynamic and often unknown anomaly distributions, requiring robust methods that operate under an open-world assumption. This challenge is exacerbated in practical settings, where models are employed by private organizations, precluding data sharing due to privacy and competitive concerns. Despite potential benefits, the sharing of anomaly information across organizations is restricted. This paper addresses the question of enhancing outlier detection within individual organizations without compromising data confidentiality. We propose a novel method leveraging representation learning and federated learning techniques to improve the detection of unknown anomalies. Specifically, our approach utilizes latent representations obtained from client-owned autoencoders to refine the decision boundary of inliers. Notably, only model parameters are shared between organizations, preserving data privacy. The efficacy of our proposed method is evaluated on two standard financial tabular datasets and an image dataset for anomaly detection in a distributed setting. The results demonstrate a strong improvement in the classification of unknown outliers during the inference phase for each organization's model.

Title: Manipulating Recommender Systems: A Survey of Poisoning Attacks and Countermeasures

• Authors: Thanh Toan Nguyen, Quoc Viet Hung Nguyen, Thanh Tam Nguyen, Thanh Trung Huynh, Thanh Thi Nguyen, Matthias Weidlich, Hongzhi Yin
• Subjects: cs.CR, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14942
• Pdf URL: https://arxiv.org/pdf/2404.14942
• Copy Paste: [[2404.14942]] Manipulating Recommender Systems: A Survey of Poisoning Attacks and Countermeasures(https://arxiv.org/abs/2404.14942)
• Keywords: protect, attack
• Abstract: Recommender systems have become an integral part of online services to help users locate specific information in a sea of data. However, existing studies show that some recommender systems are vulnerable to poisoning attacks, particularly those that involve learning schemes. A poisoning attack is where an adversary injects carefully crafted data into the process of training a model, with the goal of manipulating the system's final recommendations. Based on recent advancements in artificial intelligence, such attacks have gained importance recently. While numerous countermeasures to poisoning attacks have been developed, they have not yet been systematically linked to the properties of the attacks. Consequently, assessing the respective risks and potential success of mitigation strategies is difficult, if not impossible. This survey aims to fill this gap by primarily focusing on poisoning attacks and their countermeasures. This is in contrast to prior surveys that mainly focus on attacks and their detection methods. Through an exhaustive literature review, we provide a novel taxonomy for poisoning attacks, formalise its dimensions, and accordingly organise 30+ attacks described in the literature. Further, we review 40+ countermeasures to detect and/or prevent poisoning attacks, evaluating their effectiveness against specific types of attacks. This comprehensive survey should serve as a point of reference for protecting recommender systems against poisoning attacks. The article concludes with a discussion on open issues in the field and impactful directions for future research. A rich repository of resources associated with poisoning attacks is available at https://github.com/tamlhp/awesome-recsys-poisoning.

Title: Importance of Disjoint Sampling in Conventional and Transformer Models for Hyperspectral Image Classification

• Authors: Muhammad Ahmad, Manuel Mazzara, Salvatore Distifano
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.14944
• Pdf URL: https://arxiv.org/pdf/2404.14944
• Copy Paste: [[2404.14944]] Importance of Disjoint Sampling in Conventional and Transformer Models for Hyperspectral Image Classification(https://arxiv.org/abs/2404.14944)
• Keywords: fair, transformer
• Abstract: Disjoint sampling is critical for rigorous and unbiased evaluation of state-of-the-art (SOTA) models. When training, validation, and test sets overlap or share data, it introduces a bias that inflates performance metrics and prevents accurate assessment of a model's true ability to generalize to new examples. This paper presents an innovative disjoint sampling approach for training SOTA models on Hyperspectral image classification (HSIC) tasks. By separating training, validation, and test data without overlap, the proposed method facilitates a fairer evaluation of how well a model can classify pixels it was not exposed to during training or validation. Experiments demonstrate the approach significantly improves a model's generalization compared to alternatives that include training and validation data in test data. By eliminating data leakage between sets, disjoint sampling provides reliable metrics for benchmarking progress in HSIC. Researchers can have confidence that reported performance truly reflects a model's capabilities for classifying new scenes, not just memorized pixels. This rigorous methodology is critical for advancing SOTA models and their real-world application to large-scale land mapping with Hyperspectral sensors. The source code is available at https://github.com/mahmad00/Disjoint-Sampling-for-Hyperspectral-Image-Classification.

Title: Pyramid Hierarchical Transformer for Hyperspectral Image Classification

• Authors: Muhammad Ahmad, Muhammad Hassaan Farooq Butt, Manuel Mazzara, Salvatore Distifano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14945
• Pdf URL: https://arxiv.org/pdf/2404.14945
• Copy Paste: [[2404.14945]] Pyramid Hierarchical Transformer for Hyperspectral Image Classification(https://arxiv.org/abs/2404.14945)
• Keywords: robust, transformer
• Abstract: The traditional Transformer model encounters challenges with variable-length input sequences, particularly in Hyperspectral Image Classification (HSIC), leading to efficiency and scalability concerns. To overcome this, we propose a pyramid-based hierarchical transformer (PyFormer). This innovative approach organizes input data hierarchically into segments, each representing distinct abstraction levels, thereby enhancing processing efficiency for lengthy sequences. At each level, a dedicated transformer module is applied, effectively capturing both local and global context. Spatial and spectral information flow within the hierarchy facilitates communication and abstraction propagation. Integration of outputs from different levels culminates in the final input representation. Experimental results underscore the superiority of the proposed method over traditional approaches. Additionally, the incorporation of disjoint samples augments robustness and reliability, thereby highlighting the potential of our approach in advancing HSIC. The source code is available at https://github.com/mahmad00/PyFormer.

Title: Multi-Modal Prompt Learning on Blind Image Quality Assessment

• Authors: Wensheng Pan, Timin Gao, Yan Zhang, Runze Hu, Xiawu Zheng, Enwei Zhang, Yuting Gao, Yutao Liu, Yunhang Shen, Ke Li, Shengchuan Zhang, Liujuan Cao, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14949
• Pdf URL: https://arxiv.org/pdf/2404.14949
• Copy Paste: [[2404.14949]] Multi-Modal Prompt Learning on Blind Image Quality Assessment(https://arxiv.org/abs/2404.14949)
• Keywords: robust
• Abstract: Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Currently, leveraging semantic information to enhance IQA is a crucial research direction. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. However, the generalist nature of these pre-trained Vision-Language (VL) models often renders them suboptimal for IQA-specific tasks. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. Existing prompt-based VL models overly focus on incremental semantic information from text, neglecting the rich insights available from visual data analysis. This imbalance limits their performance improvements in IQA tasks. This paper introduces an innovative multi-modal prompt-based methodology for IQA. Our approach employs carefully crafted prompts that synergistically mine incremental semantic information from both visual and linguistic data. Specifically, in the visual branch, we introduce a multi-layer prompt structure to enhance the VL model's adaptability. In the text branch, we deploy a dual-prompt scheme that steers the model to recognize and differentiate between scene category and distortion type, thereby refining the model's capacity to assess image quality. Our experimental findings underscore the effectiveness of our method over existing Blind Image Quality Assessment (BIQA) approaches. Notably, it demonstrates competitive performance across various datasets. Our method achieves Spearman Rank Correlation Coefficient (SRCC) values of 0.961(surpassing 0.946 in CSIQ) and 0.941 (exceeding 0.930 in KADID), illustrating its robustness and accuracy in diverse contexts.

Title: Streamlining the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Model

• Authors: Ziqi Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14951
• Pdf URL: https://arxiv.org/pdf/2404.14951
• Copy Paste: [[2404.14951]] Streamlining the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Model(https://arxiv.org/abs/2404.14951)
• Keywords: robust, generative
• Abstract: Learning-based image stitching techniques typically involve three distinct stages: registration, fusion, and rectangling. These stages are often performed sequentially, each trained independently, leading to potential cascading error propagation and complex parameter tuning challenges. In rethinking the mathematical modeling of the fusion and rectangling stages, we discovered that these processes can be effectively combined into a single, variety-intensity inpainting problem. Therefore, we propose the Simple and Robust Stitcher (SRStitcher), an efficient training-free image stitching method that merges the fusion and rectangling stages into a unified model. By employing the weighted mask and large-scale generative model, SRStitcher can solve the fusion and rectangling problems in a single inference, without additional training or fine-tuning of other models. Our method not only simplifies the stitching pipeline but also enhances fault tolerance towards misregistration errors. Extensive experiments demonstrate that SRStitcher outperforms state-of-the-art (SOTA) methods in both quantitative assessments and qualitative evaluations. The code is released at https://github.com/yayoyo66/SRStitcher

Title: Leveraging Speech for Gesture Detection in Multimodal Communication

• Authors: Esam Ghaleb, Ilya Burenko, Marlou Rasenberg, Wim Pouw, Ivan Toni, Peter Uhrig, Anna Wilson, Judith Holler, Aslı Özyürek, Raquel Fernández
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14952
• Pdf URL: https://arxiv.org/pdf/2404.14952
• Copy Paste: [[2404.14952]] Leveraging Speech for Gesture Detection in Multimodal Communication(https://arxiv.org/abs/2404.14952)
• Keywords: transformer
• Abstract: Gestures are inherent to human interaction and often complement speech in face-to-face communication, forming a multimodal communication system. An important task in gesture analysis is detecting a gesture's beginning and end. Research on automatic gesture detection has primarily focused on visual and kinematic information to detect a limited set of isolated or silent gestures with low variability, neglecting the integration of speech and vision signals to detect gestures that co-occur with speech. This work addresses this gap by focusing on co-speech gesture detection, emphasising the synchrony between speech and co-speech hand gestures. We address three main challenges: the variability of gesture forms, the temporal misalignment between gesture and speech onsets, and differences in sampling rate between modalities. We investigate extended speech time windows and employ separate backbone models for each modality to address the temporal misalignment and sampling rate differences. We utilize Transformer encoders in cross-modal and early fusion techniques to effectively align and integrate speech and skeletal sequences. The study results show that combining visual and speech information significantly enhances gesture detection performance. Our findings indicate that expanding the speech buffer beyond visual time segments improves performance and that multimodal integration using cross-modal and early fusion techniques outperforms baseline methods using unimodal and late fusion methods. Additionally, we find a correlation between the models' gesture prediction confidence and low-level speech frequency features potentially associated with gestures. Overall, the study provides a better understanding and detection methods for co-speech gestures, facilitating the analysis of multimodal communication.

Title: Traditional to Transformers: A Survey on Current Trends and Future Prospects for Hyperspectral Image Classification

• Authors: Muhammad Ahmad, Salvatore Distifano, Manuel Mazzara, Adil Mehmood Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14955
• Pdf URL: https://arxiv.org/pdf/2404.14955
• Copy Paste: [[2404.14955]] Traditional to Transformers: A Survey on Current Trends and Future Prospects for Hyperspectral Image Classification(https://arxiv.org/abs/2404.14955)
• Keywords: transformer
• Abstract: Hyperspectral image classification is a challenging task due to the high dimensionality and complex nature of hyperspectral data. In recent years, deep learning techniques have emerged as powerful tools for addressing these challenges. This survey provides a comprehensive overview of the current trends and future prospects in hyperspectral image classification, focusing on the advancements from deep learning models to the emerging use of transformers. We review the key concepts, methodologies, and state-of-the-art approaches in deep learning for hyperspectral image classification. Additionally, we discuss the potential of transformer-based models in this field and highlight the advantages and challenges associated with these approaches. Comprehensive experimental results have been undertaken using three Hyperspectral datasets to verify the efficacy of various conventional deep-learning models and Transformers. Finally, we outline future research directions and potential applications that can further enhance the accuracy and efficiency of hyperspectral image classification. The Source code is available at https://github.com/mahmad00/Conventional-to-Transformer-for-Hyperspectral-Image-Classification-Survey-2024.

Title: DAWN: Domain-Adaptive Weakly Supervised Nuclei Segmentation via Cross-Task Interactions

• Authors: Ye Zhang, Yifeng Wang, Zijie Fang, Hao Bian, Linghan Cai, Ziyue Wang, Yongbing Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14956
• Pdf URL: https://arxiv.org/pdf/2404.14956
• Copy Paste: [[2404.14956]] DAWN: Domain-Adaptive Weakly Supervised Nuclei Segmentation via Cross-Task Interactions(https://arxiv.org/abs/2404.14956)
• Keywords: segmentation
• Abstract: Weakly supervised segmentation methods have gained significant attention due to their ability to reduce the reliance on costly pixel-level annotations during model training. However, the current weakly supervised nuclei segmentation approaches typically follow a two-stage pseudo-label generation and network training process. The performance of the nuclei segmentation heavily relies on the quality of the generated pseudo-labels, thereby limiting its effectiveness. This paper introduces a novel domain-adaptive weakly supervised nuclei segmentation framework using cross-task interaction strategies to overcome the challenge of pseudo-label generation. Specifically, we utilize weakly annotated data to train an auxiliary detection task, which assists the domain adaptation of the segmentation network. To enhance the efficiency of domain adaptation, we design a consistent feature constraint module integrating prior knowledge from the source domain. Furthermore, we develop pseudo-label optimization and interactive training methods to improve the domain transfer capability. To validate the effectiveness of our proposed method, we conduct extensive comparative and ablation experiments on six datasets. The results demonstrate the superiority of our approach over existing weakly supervised approaches. Remarkably, our method achieves comparable or even better performance than fully supervised methods. Our code will be released in https://github.com/zhangye-zoe/DAWN.

Title: Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Perfect Reasoners

• Authors: Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, Bo Du, Dacheng Tao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14963
• Pdf URL: https://arxiv.org/pdf/2404.14963
• Copy Paste: [[2404.14963]] Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Perfect Reasoners(https://arxiv.org/abs/2404.14963)
• Keywords: large language model
• Abstract: Chain of Thought prompting strategy has enhanced the performance of Large Language Models (LLMs) across various NLP tasks. However, it still has shortcomings when dealing with complex reasoning tasks, following~\citet{cot_wei}, including understanding errors, calculation errors and process errors (e.g. missing-step and hallucinations). Subsequently, Our in-depth analysis of various error types has found that deeply understanding the whole problem is critical in addressing complicated reasoning tasks. In this paper, we proposed a novel prompt strategy called Deeply Understanding the Problems (DUP) prompting, inspired by how humans solve complex reasoning problems, designed to enhance the comprehensive understanding of problems by LLMs. It consists of three stages: 1) extract the core question; 2) find out problem-solving information based on the core question; 3) generate and extract answers by LLMs. We evaluate the performance of DUP prompting on ten diverse reasoning datasets. Experimental results suggest that DUP prompting significantly outperforms Zero-Shot CoT ~\cite{kojima2022large} across all datasets. Notably, DUP achieves \textbf{state-of-the-art on SVAMP (90.4\% to 94.2\%) and GSM8K (94.6\% to 97.1\%).}

Title: Mamba3D: Enhancing Local Features for 3D Point Cloud Analysis via State Space Model

• Authors: Xu Han, Yuan Tang, Zhaoxuan Wang, Xianzhi Li
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14966
• Pdf URL: https://arxiv.org/pdf/2404.14966
• Copy Paste: [[2404.14966]] Mamba3D: Enhancing Local Features for 3D Point Cloud Analysis via State Space Model(https://arxiv.org/abs/2404.14966)
• Keywords: extraction, transformer
• Abstract: Existing Transformer-based models for point cloud analysis suffer from quadratic complexity, leading to compromised point cloud resolution and information loss. In contrast, the newly proposed Mamba model, based on state space models (SSM), outperforms Transformer in multiple areas with only linear complexity. However, the straightforward adoption of Mamba does not achieve satisfactory performance on point cloud tasks. In this work, we present Mamba3D, a state space model tailored for point cloud learning to enhance local feature extraction, achieving superior performance, high efficiency, and scalability potential. Specifically, we propose a simple yet effective Local Norm Pooling (LNP) block to extract local geometric features. Additionally, to obtain better global features, we introduce a bidirectional SSM (bi-SSM) with both a token forward SSM and a novel backward SSM that operates on the feature channel. Extensive experimental results show that Mamba3D surpasses Transformer-based counterparts and concurrent works in multiple tasks, with or without pre-training. Notably, Mamba3D achieves multiple SoTA, including an overall accuracy of 92.6% (train from scratch) on the ScanObjectNN and 95.1% (with single-modal pre-training) on the ModelNet40 classification task, with only linear complexity.

Title: CoARF: Controllable 3D Artistic Style Transfer for Radiance Fields

• Authors: Deheng Zhang, Clara Fernandez-Labrador, Christopher Schroers
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.14967
• Pdf URL: https://arxiv.org/pdf/2404.14967
• Copy Paste: [[2404.14967]] CoARF: Controllable 3D Artistic Style Transfer for Radiance Fields(https://arxiv.org/abs/2404.14967)
• Keywords: segmentation
• Abstract: Creating artistic 3D scenes can be time-consuming and requires specialized knowledge. To address this, recent works such as ARF, use a radiance field-based approach with style constraints to generate 3D scenes that resemble a style image provided by the user. However, these methods lack fine-grained control over the resulting scenes. In this paper, we introduce Controllable Artistic Radiance Fields (CoARF), a novel algorithm for controllable 3D scene stylization. CoARF enables style transfer for specified objects, compositional 3D style transfer and semantic-aware style transfer. We achieve controllability using segmentation masks with different label-dependent loss functions. We also propose a semantic-aware nearest neighbor matching algorithm to improve the style transfer quality. Our extensive experiments demonstrate that CoARF provides user-specified controllability of style transfer and superior style transfer quality with more precise feature matching.

Title: SGFormer: Spherical Geometry Transformer for 360 Depth Estimation

• Authors: Junsong Zhang, Zisong Chen, Chunyu Lin, Lang Nie, Zhijie Shen, Junda Huang, Yao Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14979
• Pdf URL: https://arxiv.org/pdf/2404.14979
• Copy Paste: [[2404.14979]] SGFormer: Spherical Geometry Transformer for 360 Depth Estimation(https://arxiv.org/abs/2404.14979)
• Keywords: transformer
• Abstract: Panoramic distortion poses a significant challenge in 360 depth estimation, particularly pronounced at the north and south poles. Existing methods either adopt a bi-projection fusion strategy to remove distortions or model long-range dependencies to capture global structures, which can result in either unclear structure or insufficient local perception. In this paper, we propose a spherical geometry transformer, named SGFormer, to address the above issues, with an innovative step to integrate spherical geometric priors into vision transformers. To this end, we retarget the transformer decoder to a spherical prior decoder (termed SPDecoder), which endeavors to uphold the integrity of spherical structures during decoding. Concretely, we leverage bipolar re-projection, circular rotation, and curve local embedding to preserve the spherical characteristics of equidistortion, continuity, and surface distance, respectively. Furthermore, we present a query-based global conditional position embedding to compensate for spatial structure at varying resolutions. It not only boosts the global perception of spatial position but also sharpens the depth structure across different patches. Finally, we conduct extensive experiments on popular benchmarks, demonstrating our superiority over state-of-the-art solutions.

Title: Zero-Knowledge Location Privacy via Accurate Floating Point SNARKs

• Authors: Jens Ernstberger, Chengru Zhang, Luca Ciprian, Philipp Jovanovic, Sebastian Steinhorst
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.14983
• Pdf URL: https://arxiv.org/pdf/2404.14983
• Copy Paste: [[2404.14983]] Zero-Knowledge Location Privacy via Accurate Floating Point SNARKs(https://arxiv.org/abs/2404.14983)
• Keywords: privacy
• Abstract: This paper introduces Zero-Knowledge Location Privacy (ZKLP), enabling users to prove to third parties that they are within a specified geographical region while not disclosing their exact location. ZKLP supports varying levels of granularity, allowing for customization depending on the use case. To realize ZKLP, we introduce the first set of Zero-Knowledge Proof (ZKP) circuits that are fully compliant to the IEEE 754 standard for floating-point arithmetic. Our results demonstrate that our floating point implementation scales efficiently, requiring only $69$ constraints per multiplication for $2^{15}$ single-precision floating-point multiplications. We utilize our floating point implementation to realize the ZKLP paradigm. In comparison to the state-of-the-art, we find that our optimized implementation has $14.1 \times$ less constraints utilizing single precision floating-point values, and $11.2 \times$ less constraints when utilizing double precision floating-point values. We demonstrate the practicability of ZKLP by building a protocol for privacy preserving peer-to-peer proximity testing - Alice can test if she is close to Bob by receiving a single message, without either party revealing any other information about their location. In such a configuration, Bob can create a proof of (non-)proximity in $0.27 s$, whereas Alice can verify her distance to about $250$ peers per second

Title: Other Tokens Matter: Exploring Global and Local Features of Vision Transformers for Object Re-Identification

• Authors: Yingquan Wang, Pingping Zhang, Dong Wang, Huchuan Lu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.14985
• Pdf URL: https://arxiv.org/pdf/2404.14985
• Copy Paste: [[2404.14985]] Other Tokens Matter: Exploring Global and Local Features of Vision Transformers for Object Re-Identification(https://arxiv.org/abs/2404.14985)
• Keywords: transformer
• Abstract: Object Re-Identification (Re-ID) aims to identify and retrieve specific objects from images captured at different places and times. Recently, object Re-ID has achieved great success with the advances of Vision Transformers (ViT). However, the effects of the global-local relation have not been fully explored in Transformers for object Re-ID. In this work, we first explore the influence of global and local features of ViT and then further propose a novel Global-Local Transformer (GLTrans) for high-performance object Re-ID. We find that the features from last few layers of ViT already have a strong representational ability, and the global and local information can mutually enhance each other. Based on this fact, we propose a Global Aggregation Encoder (GAE) to utilize the class tokens of the last few Transformer layers and learn comprehensive global features effectively. Meanwhile, we propose the Local Multi-layer Fusion (LMF) which leverages both the global cues from GAE and multi-layer patch tokens to explore the discriminative local representations. Extensive experiments demonstrate that our proposed method achieves superior performance on four object Re-ID benchmarks.

Title: Interpreting COVID Lateral Flow Tests' Results with Foundation Models

• Authors: Stuti Pandey, Josh Myers-Dean, Jarek Reynolds, Danna Gurari
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.14990
• Pdf URL: https://arxiv.org/pdf/2404.14990
• Copy Paste: [[2404.14990]] Interpreting COVID Lateral Flow Tests' Results with Foundation Models(https://arxiv.org/abs/2404.14990)
• Keywords: segmentation
• Abstract: Lateral flow tests (LFTs) enable rapid, low-cost testing for health conditions including Covid, pregnancy, HIV, and malaria. Automated readers of LFT results can yield many benefits including empowering blind people to independently learn about their health and accelerating data entry for large-scale monitoring (e.g., for pandemics such as Covid) by using only a single photograph per LFT test. Accordingly, we explore the abilities of modern foundation vision language models (VLMs) in interpreting such tests. To enable this analysis, we first create a new labeled dataset with hierarchical segmentations of each LFT test and its nested test result window. We call this dataset LFT-Grounding. Next, we benchmark eight modern VLMs in zero-shot settings for analyzing these images. We demonstrate that current VLMs frequently fail to correctly identify the type of LFT test, interpret the test results, locate the nested result window of the LFT tests, and recognize LFT tests when they partially obfuscated. To facilitate community-wide progress towards automated LFT reading, we publicly release our dataset at https://iamstuti.github.io/lft_grounding_foundation_models/.

Title: Transformers Can Represent $n$-gram Language Models

• Authors: Anej Svete, Ryan Cotterell
• Subjects: cs.CL, cs.AI, cs.CC, cs.FL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14994
• Pdf URL: https://arxiv.org/pdf/2404.14994
• Copy Paste: [[2404.14994]] Transformers Can Represent $n$-gram Language Models(https://arxiv.org/abs/2404.14994)
• Keywords: transformer
• Abstract: Plenty of existing work has analyzed the abilities of the transformer architecture by describing its representational capacity with formal models of computation. However, the focus so far has been on analyzing the architecture in terms of language \emph{acceptance}. We contend that this is an ill-suited problem in the study of \emph{language models} (LMs), which are definitionally \emph{probability distributions} over strings. In this paper, we focus on the relationship between transformer LMs and $n$-gram LMs, a simple and historically relevant class of language models. We show that transformer LMs using the hard or sparse attention mechanisms can exactly represent any $n$-gram LM, giving us a concrete lower bound on their probabilistic representational capacity. This provides a first step towards understanding the mechanisms that transformer LMs can use to represent probability distributions over strings.

Title: CA-Stream: Attention-based pooling for interpretable image recognition

• Authors: Felipe Torres, Hanwei Zhang, Ronan Sicre, Stéphane Ayache, Yannis Avrithis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14996
• Pdf URL: https://arxiv.org/pdf/2404.14996
• Copy Paste: [[2404.14996]] CA-Stream: Attention-based pooling for interpretable image recognition(https://arxiv.org/abs/2404.14996)
• Keywords: interpretability, transformer
• Abstract: Explanations obtained from transformer-based architectures in the form of raw attention, can be seen as a class-agnostic saliency map. Additionally, attention-based pooling serves as a form of masking the in feature space. Motivated by this observation, we design an attention-based pooling mechanism intended to replace Global Average Pooling (GAP) at inference. This mechanism, called Cross-Attention Stream (CA-Stream), comprises a stream of cross attention blocks interacting with features at different network depths. CA-Stream enhances interpretability in models, while preserving recognition performance.

Title: EarPass: Secure and Implicit Call Receiver Authentication Using Ear Acoustic Sensing

• Authors: Xiping Sun, Jing Chen, Kun He, Zhixiang He, Ruiying Du, Yebo Feng, Qingchuan Zhao, Cong Wu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15000
• Pdf URL: https://arxiv.org/pdf/2404.15000
• Copy Paste: [[2404.15000]] EarPass: Secure and Implicit Call Receiver Authentication Using Ear Acoustic Sensing(https://arxiv.org/abs/2404.15000)
• Keywords: secure, security, attack
• Abstract: Private voice communication often contains sensitive information, making it critical to ensure that only authorized users have access to such calls. Unfortunately, current authentication mechanisms, such as PIN-based passwords, fingerprint recognition, and face recognition, fail to authenticate the call receiver, leaving a gap in security. To fill the gap, we present EarPass, a secure and implicit call receiver authentication scheme designed for smartphones. EarPass sends inaudible acoustic signals through the earpiece speaker to actively sense the outer ear, and records echoes using the top microphone. It focuses on extracting ear-related signals from echoes and performs spectrogram analysis in the magnitude and phase domains. To overcome posture and position variability, EarPass utilizes a learning-based feature extractor for extracting representative features, and a one-class classifier for authentication. EarPass does not increase any burdens on users or change users' call answering habits. Furthermore, it does not require extra devices but only uses the speaker and microphone on the smartphone. We conducted comprehensive experiments to evaluate EarPass's effectiveness and security. Our results show that EarPass can achieve a balanced accuracy of 96.95% and an equal error rate of 1.53%. Additionally, EarPass exhibits resilience against potential attacks, including zero-effort attacks and mimicry attacks.

Title: Comparison of Current Approaches to Lemmatization: A Case Study in Estonian

• Authors: Aleksei Dorkin, Kairit Sirts
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15003
• Pdf URL: https://arxiv.org/pdf/2404.15003
• Copy Paste: [[2404.15003]] Comparison of Current Approaches to Lemmatization: A Case Study in Estonian(https://arxiv.org/abs/2404.15003)
• Keywords: generative
• Abstract: This study evaluates three different lemmatization approaches to Estonian -- Generative character-level models, Pattern-based word-level classification models, and rule-based morphological analysis. According to our experiments, a significantly smaller Generative model consistently outperforms the Pattern-based classification model based on EstBERT. Additionally, we observe a relatively small overlap in errors made by all three models, indicating that an ensemble of different approaches could lead to improvements.

Title: External Prompt Features Enhanced Parameter-efficient Fine-tuning for Salient Object Detection

• Authors: Wen Liang, Peipei Ran, Mengchao Bai, Xiao Liu, P. Bilha Githinji, Wei Zhao, Peiwu Qin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15008
• Pdf URL: https://arxiv.org/pdf/2404.15008
• Copy Paste: [[2404.15008]] External Prompt Features Enhanced Parameter-efficient Fine-tuning for Salient Object Detection(https://arxiv.org/abs/2404.15008)
• Keywords: transformer
• Abstract: Salient object detection (SOD) aims at finding the most salient objects in images and outputs pixel-level binary masks. Transformer-based methods achieve promising performance due to their global semantic understanding, crucial for identifying salient objects. However, these models tend to be large and require numerous training parameters. To better harness the potential of transformers for SOD, we propose a novel parameter-efficient fine-tuning method aimed at reducing the number of training parameters while enhancing the salient object detection capability. Our model, termed EXternal Prompt features Enhanced adapteR Tuning (ExPert), features an encoder-decoder structure with adapters and injectors interspersed between the layers of a frozen transformer encoder. The adapter modules adapt the pre-trained backbone to SOD while the injector modules incorporate external prompt features to enhance the awareness of salient objects. Comprehensive experiments demonstrate the superiority of our method. Surpassing former state-of-the-art (SOTA) models across five SOD datasets, ExPert achieves 0.215 mean absolute error (MAE) in ECSSD dataset with 80.2M trained parameters, 21% better than transformer-based SOTA model and 47% better than CNN-based SOTA model.

Title: The Brain Tumor Segmentation in Pediatrics (BraTS-PEDs) Challenge: Focus on Pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs)

• Authors: Anahita Fathi Kazerooni, Nastaran Khalili, Deep Gandhi, Xinyang Liu, Zhifan Jiang, Syed Muhammed Anwar, Jake Albrecht, Maruf Adewole, Udunna Anazodo, Hannah Anderson, Sina Bagheri, Ujjwal Baid, Timothy Bergquist, Austin J. Borja, Evan Calabrese, Verena Chung, Gian-Marco Conte, Farouk Dako, James Eddy, Ivan Ezhov, Ariana Familiar, Keyvan Farahani, Anurag Gottipati, Debanjan Haldar, Shuvanjan Haldar, Juan Eugenio Iglesias, Anastasia Janas, Elaine Johansen, Blaise V Jones, Neda Khalili, Florian Kofler, Dominic LaBella, Hollie Anne Lai, Koen Van Leemput, Hongwei Bran Li, Nazanin Maleki, Aaron S McAllister, Zeke Meier, Bjoern Menze, Ahmed W Moawad, Khanak K Nandolia, Julija Pavaine, Marie Piraud, Tina Poussaint, Sanjay P Prabhu, Zachary Reitman, Andres Rodriguez, Jeffrey D Rudie, Mariana Sanchez-Montano, et al. (27 additional authors not shown)
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.15009
• Pdf URL: https://arxiv.org/pdf/2404.15009
• Copy Paste: [[2404.15009]] The Brain Tumor Segmentation in Pediatrics (BraTS-PEDs) Challenge: Focus on Pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs)(https://arxiv.org/abs/2404.15009)
• Keywords: segmentation
• Abstract: Pediatric tumors of the central nervous system are the most common cause of cancer-related death in children. The five-year survival rate for high-grade gliomas in children is less than 20%. Due to their rarity, the diagnosis of these entities is often delayed, their treatment is mainly based on historic treatment concepts, and clinical trials require multi-institutional collaborations. Here we present the CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs challenge, focused on pediatric brain tumors with data acquired across multiple international consortia dedicated to pediatric neuro-oncology and clinical trials. The CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs challenge brings together clinicians and AI/imaging scientists to lead to faster development of automated segmentation techniques that could benefit clinical trials, and ultimately the care of children with brain tumors.

Title: X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition

• Authors: Shuofeng Sun, Yongming Rao, Jiwen Lu, Haibin Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15010
• Pdf URL: https://arxiv.org/pdf/2404.15010
• Copy Paste: [[2404.15010]] X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition(https://arxiv.org/abs/2404.15010)
• Keywords: extraction, segmentation
• Abstract: Numerous prior studies predominantly emphasize constructing relation vectors for individual neighborhood points and generating dynamic kernels for each vector and embedding these into high-dimensional spaces to capture implicit local structures. However, we contend that such implicit high-dimensional structure modeling approch inadequately represents the local geometric structure of point clouds due to the absence of explicit structural information. Hence, we introduce X-3D, an explicit 3D structure modeling approach. X-3D functions by capturing the explicit local structural information within the input 3D space and employing it to produce dynamic kernels with shared weights for all neighborhood points within the current local region. This modeling approach introduces effective geometric prior and significantly diminishes the disparity between the local structure of the embedding space and the original input point cloud, thereby improving the extraction of local features. Experiments show that our method can be used on a variety of methods and achieves state-of-the-art performance on segmentation, classification, detection tasks with lower extra computational cost, such as \textbf{90.7\%} on ScanObjectNN for classification, \textbf{79.2\%} on S3DIS 6 fold and \textbf{74.3\%} on S3DIS Area 5 for segmentation, \textbf{76.3\%} on ScanNetV2 for segmentation and \textbf{64.5\%} mAP , \textbf{46.9\%} mAP on SUN RGB-D and \textbf{69.0\%} mAP , \textbf{51.1\%} mAP on ScanNetV2 . Our code is available at \href{https://github.com/sunshuofeng/X-3D}{https://github.com/sunshuofeng/X-3D}.

Title: OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving

• Authors: Guoqing Wang, Zhongdao Wang, Pin Tang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, Chao Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15014
• Pdf URL: https://arxiv.org/pdf/2404.15014
• Copy Paste: [[2404.15014]] OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving(https://arxiv.org/abs/2404.15014)
• Keywords: diffusion, generative, segmentation
• Abstract: Existing solutions for 3D semantic occupancy prediction typically treat the task as a one-shot 3D voxel-wise segmentation perception problem. These discriminative methods focus on learning the mapping between the inputs and occupancy map in a single step, lacking the ability to gradually refine the occupancy map and the reasonable scene imaginative capacity to complete the local regions somewhere. In this paper, we introduce OccGen, a simple yet powerful generative perception model for the task of 3D semantic occupancy prediction. OccGen adopts a ''noise-to-occupancy'' generative paradigm, progressively inferring and refining the occupancy map by predicting and eliminating noise originating from a random Gaussian distribution. OccGen consists of two main components: a conditional encoder that is capable of processing multi-modal inputs, and a progressive refinement decoder that applies diffusion denoising using the multi-modal features as conditions. A key insight of this generative pipeline is that the diffusion denoising process is naturally able to model the coarse-to-fine refinement of the dense 3D occupancy map, therefore producing more detailed predictions. Extensive experiments on several occupancy benchmarks demonstrate the effectiveness of the proposed method compared to the state-of-the-art methods. For instance, OccGen relatively enhances the mIoU by 9.5%, 6.3%, and 13.3% on nuScenes-Occupancy dataset under the muli-modal, LiDAR-only, and camera-only settings, respectively. Moreover, as a generative perception model, OccGen exhibits desirable properties that discriminative models cannot achieve, such as providing uncertainty estimates alongside its multiple-step predictions.

Title: A review of deep learning-based information fusion techniques for multimodal medical image classification

• Authors: Yihao Li, Mostafa El Habib Daho, Pierre-Henri Conze, Rachid Zeghlache, Hugo Le Boité, Ramin Tadayoni, Béatrice Cochener, Mathieu Lamard, Gwenolé Quellec
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15022
• Pdf URL: https://arxiv.org/pdf/2404.15022
• Copy Paste: [[2404.15022]] A review of deep learning-based information fusion techniques for multimodal medical image classification(https://arxiv.org/abs/2404.15022)
• Keywords: transformer
• Abstract: Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

Title: A Learning Paradigm for Interpretable Gradients

• Authors: Felipe Torres Figueroa, Hanwei Zhang, Ronan Sicre, Yannis Avrithis, Stephane Ayache
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15024
• Pdf URL: https://arxiv.org/pdf/2404.15024
• Copy Paste: [[2404.15024]] A Learning Paradigm for Interpretable Gradients(https://arxiv.org/abs/2404.15024)
• Keywords: interpretability
• Abstract: This paper studies interpretability of convolutional networks by means of saliency maps. Most approaches based on Class Activation Maps (CAM) combine information from fully connected layers and gradient through variants of backpropagation. However, it is well understood that gradients are noisy and alternatives like guided backpropagation have been proposed to obtain better visualization at inference. In this work, we present a novel training approach to improve the quality of gradients for interpretability. In particular, we introduce a regularization loss such that the gradient with respect to the input image obtained by standard backpropagation is similar to the gradient obtained by guided backpropagation. We find that the resulting gradient is qualitatively less noisy and improves quantitatively the interpretability properties of different networks, using several interpretability methods.

Title: PRISM: A Promptable and Robust Interactive Segmentation Model with Visual Prompts

• Authors: Hao Li, Han Liu, Dewei Hu, Jiacheng Wang, Ipek Oguz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15028
• Pdf URL: https://arxiv.org/pdf/2404.15028
• Copy Paste: [[2404.15028]] PRISM: A Promptable and Robust Interactive Segmentation Model with Visual Prompts(https://arxiv.org/abs/2404.15028)
• Keywords: robust, segmentation
• Abstract: In this paper, we present PRISM, a Promptable and Robust Interactive Segmentation Model, aiming for precise segmentation of 3D medical images. PRISM accepts various visual inputs, including points, boxes, and scribbles as sparse prompts, as well as masks as dense prompts. Specifically, PRISM is designed with four principles to achieve robustness: (1) Iterative learning. The model produces segmentations by using visual prompts from previous iterations to achieve progressive improvement. (2) Confidence learning. PRISM employs multiple segmentation heads per input image, each generating a continuous map and a confidence score to optimize predictions. (3) Corrective learning. Following each segmentation iteration, PRISM employs a shallow corrective refinement network to reassign mislabeled voxels. (4) Hybrid design. PRISM integrates hybrid encoders to better capture both the local and global information. Comprehensive validation of PRISM is conducted using four public datasets for tumor segmentation in the colon, pancreas, liver, and kidney, highlighting challenges caused by anatomical variations and ambiguous boundaries in accurate tumor identification. Compared to state-of-the-art methods, both with and without prompt engineering, PRISM significantly improves performance, achieving results that are close to human levels. The code is publicly available at https://github.com/MedICL-VU/PRISM.

Title: IPAD: Industrial Process Anomaly Detection Dataset

• Authors: Jinfan Liu, Yichao Yan, Junjie Li, Weiming Zhao, Pengzhi Chu, Xingdong Sheng, Yunhui Liu, Xiaokang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15033
• Pdf URL: https://arxiv.org/pdf/2404.15033
• Copy Paste: [[2404.15033]] IPAD: Industrial Process Anomaly Detection Dataset(https://arxiv.org/abs/2404.15033)
• Keywords: security, privacy
• Abstract: Video anomaly detection (VAD) is a challenging task aiming to recognize anomalies in video frames, and existing large-scale VAD researches primarily focus on road traffic and human activity scenes. In industrial scenes, there are often a variety of unpredictable anomalies, and the VAD method can play a significant role in these scenarios. However, there is a lack of applicable datasets and methods specifically tailored for industrial production scenarios due to concerns regarding privacy and security. To bridge this gap, we propose a new dataset, IPAD, specifically designed for VAD in industrial scenarios. The industrial processes in our dataset are chosen through on-site factory research and discussions with engineers. This dataset covers 16 different industrial devices and contains over 6 hours of both synthetic and real-world video footage. Moreover, we annotate the key feature of the industrial process, ie, periodicity. Based on the proposed dataset, we introduce a period memory module and a sliding window inspection mechanism to effectively investigate the periodic information in a basic reconstruction model. Our framework leverages LoRA adapter to explore the effective migration of pretrained models, which are initially trained using synthetic data, into real-world scenarios. Our proposed dataset and method will fill the gap in the field of industrial video anomaly detection and drive the process of video understanding tasks as well as smart factory deployment.

Title: Near-Universally-Optimal Differentially Private Minimum Spanning Trees

• Authors: Richard Hladík, Jakub Tětek
• Subjects: cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2404.15035
• Pdf URL: https://arxiv.org/pdf/2404.15035
• Copy Paste: [[2404.15035]] Near-Universally-Optimal Differentially Private Minimum Spanning Trees(https://arxiv.org/abs/2404.15035)
• Keywords: privacy
• Abstract: Devising mechanisms with good beyond-worst-case input-dependent performance has been an important focus of differential privacy, with techniques such as smooth sensitivity, propose-test-release, or inverse sensitivity mechanism being developed to achieve this goal. This makes it very natural to use the notion of universal optimality in differential privacy. Universal optimality is a strong instance-specific optimality guarantee for problems on weighted graphs, which roughly states that for any fixed underlying (unweighted) graph, the algorithm is optimal in the worst-case sense, with respect to the possible setting of the edge weights. In this paper, we give the first such result in differential privacy. Namely, we prove that a simple differentially private mechanism for approximately releasing the minimum spanning tree is near-optimal in the sense of universal optimality for the $\ell_1$ neighbor relation. Previously, it was only known that this mechanism is nearly optimal in the worst case. We then focus on the $\ell_\infty$ neighbor relation, for which the described mechanism is not optimal. We show that one may implement the exponential mechanism for MST in polynomial time, and that this results in universal near-optimality for both the $\ell_1$ and the $\ell_\infty$ neighbor relations.

Title: Leverage Variational Graph Representation For Model Poisoning on Federated Learning

• Authors: Kai Li, Xin Yuan, Jingjing Zheng, Wei Ni, Falko Dressler, Abbas Jamalipour
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15042
• Pdf URL: https://arxiv.org/pdf/2404.15042
• Copy Paste: [[2404.15042]] Leverage Variational Graph Representation For Model Poisoning on Federated Learning(https://arxiv.org/abs/2404.15042)
• Keywords: defense, attack, federate
• Abstract: This paper puts forth a new training data-untethered model poisoning (MP) attack on federated learning (FL). The new MP attack extends an adversarial variational graph autoencoder (VGAE) to create malicious local models based solely on the benign local models overheard without any access to the training data of FL. Such an advancement leads to the VGAE-MP attack that is not only efficacious but also remains elusive to detection. VGAE-MP attack extracts graph structural correlations among the benign local models and the training data features, adversarially regenerates the graph structure, and generates malicious local models using the adversarial graph structure and benign models' features. Moreover, a new attacking algorithm is presented to train the malicious local models using VGAE and sub-gradient descent, while enabling an optimal selection of the benign local models for training the VGAE. Experiments demonstrate a gradual drop in FL accuracy under the proposed VGAE-MP attack and the ineffectiveness of existing defense mechanisms in detecting the attack, posing a severe threat to FL.

Title: Formal Verification of Graph Convolutional Networks with Uncertain Node Features and Uncertain Graph Structure

• Authors: Tobias Ladner, Michael Eichelbeck, Matthias Althoff
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15065
• Pdf URL: https://arxiv.org/pdf/2404.15065
• Copy Paste: [[2404.15065]] Formal Verification of Graph Convolutional Networks with Uncertain Node Features and Uncertain Graph Structure(https://arxiv.org/abs/2404.15065)
• Keywords: attack, robust
• Abstract: Graph neural networks are becoming increasingly popular in the field of machine learning due to their unique ability to process data structured in graphs. They have also been applied in safety-critical environments where perturbations inherently occur. However, these perturbations require us to formally verify neural networks before their deployment in safety-critical environments as neural networks are prone to adversarial attacks. While there exists research on the formal verification of neural networks, there is no work verifying the robustness of generic graph convolutional network architectures with uncertainty in the node features and in the graph structure over multiple message-passing steps. This work addresses this research gap by explicitly preserving the non-convex dependencies of all elements in the underlying computations through reachability analysis with (matrix) polynomial zonotopes. We demonstrate our approach on three popular benchmark datasets.

Title: Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models

• Authors: Jingyao Xu, Yuetong Lu, Yandong Li, Siyang Lu, Dongdong Wang, Xiang Wei
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15081
• Pdf URL: https://arxiv.org/pdf/2404.15081
• Copy Paste: [[2404.15081]] Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models(https://arxiv.org/abs/2404.15081)
• Keywords: security, attack, diffusion, generative
• Abstract: Diffusion models (DMs) embark a new era of generative modeling and offer more opportunities for efficient generating high-quality and realistic data samples. However, their widespread use has also brought forth new challenges in model security, which motivates the creation of more effective adversarial attackers on DMs to understand its vulnerability. We propose CAAT, a simple but generic and efficient approach that does not require costly training to effectively fool latent diffusion models (LDMs). The approach is based on the observation that cross-attention layers exhibits higher sensitivity to gradient change, allowing for leveraging subtle perturbations on published images to significantly corrupt the generated images. We show that a subtle perturbation on an image can significantly impact the cross-attention layers, thus changing the mapping between text and image during the fine-tuning of customized diffusion models. Extensive experiments demonstrate that CAAT is compatible with diverse diffusion models and outperforms baseline attack methods in a more effective (more noise) and efficient (twice as fast as Anti-DreamBooth and Mist) manner.

Title: Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation

• Authors: Xun Wu, Shaohan Huang, Furu Wei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.15100
• Pdf URL: https://arxiv.org/pdf/2404.15100
• Copy Paste: [[2404.15100]] Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation(https://arxiv.org/abs/2404.15100)
• Keywords: generative, large language model
• Abstract: Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, enhancing the alignment between generated images and textual prompts. Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration. To address these challenges and promote the alignment of generative models through instruction tuning, we leverage multimodal large language models to create VisionPrefer, a high-quality and fine-grained preference dataset that captures multiple preference aspects. We aggregate feedback from AI annotators across four aspects: prompt-following, aesthetic, fidelity, and harmlessness to construct VisionPrefer. To validate the effectiveness of VisionPrefer, we train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators. Furthermore, we use two reinforcement learning methods to supervised fine-tune generative models to evaluate the performance of VisionPrefer, and extensive experimental results demonstrate that VisionPrefer significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic, and generalizes better than previous human-preference metrics across various image distributions. Moreover, VisionPrefer indicates that the integration of AI-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in vision generative models.

Title: Identifying Fairness Issues in Automatically Generated Testing Content

• Authors: Kevin Stowe, Benny Longwill, Alyssa Francis, Tatsuya Aoyama, Debanjan Ghosh, Swapna Somasundaran
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15104
• Pdf URL: https://arxiv.org/pdf/2404.15104
• Copy Paste: [[2404.15104]] Identifying Fairness Issues in Automatically Generated Testing Content(https://arxiv.org/abs/2404.15104)
• Keywords: fair
• Abstract: Natural language generation tools are powerful and effective for generating content. However, language models are known to display bias and fairness issues, making them impractical to deploy for many use cases. We here focus on how fairness issues impact automatically generated test content, which can have stringent requirements to ensure the test measures only what it was intended to measure. Specifically, we identify test content that is focused on particular domains and experiences that only reflect a certain demographic or that are potentially emotionally upsetting; both of which could inadvertently impact a test-taker's score. This kind of content doesn't reflect typical biases out of context, making it challenging even for modern models that contain safeguards. We build a dataset of 621 generated texts annotated for fairness and explore a variety of methods for classification: fine-tuning, topic-based classification, and prompting, including few-shot and self-correcting prompts. We find that combining prompt self-correction and few-shot learning performs best, yielding an F1 score of .791 on our held-out test set, while much smaller BERT- and topic-based models have competitive performance on out-of-domain data.

Title: CutDiffusion: A Simple, Fast, Cheap, and Strong Diffusion Extrapolation Method

• Authors: Mingbao Lin, Zhihang Lin, Wengyi Zhan, Liujuan Cao, Rongrong Ji
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15141
• Pdf URL: https://arxiv.org/pdf/2404.15141
• Copy Paste: [[2404.15141]] CutDiffusion: A Simple, Fast, Cheap, and Strong Diffusion Extrapolation Method(https://arxiv.org/abs/2404.15141)
• Keywords: diffusion
• Abstract: Transforming large pre-trained low-resolution diffusion models to cater to higher-resolution demands, i.e., diffusion extrapolation, significantly improves diffusion adaptability. We propose tuning-free CutDiffusion, aimed at simplifying and accelerating the diffusion extrapolation process, making it more affordable and improving performance. CutDiffusion abides by the existing patch-wise extrapolation but cuts a standard patch diffusion process into an initial phase focused on comprehensive structure denoising and a subsequent phase dedicated to specific detail refinement. Comprehensive experiments highlight the numerous almighty advantages of CutDiffusion: (1) simple method construction that enables a concise higher-resolution diffusion process without third-party engagement; (2) fast inference speed achieved through a single-step higher-resolution diffusion process, and fewer inference patches required; (3) cheap GPU cost resulting from patch-wise inference and fewer patches during the comprehensive structure denoising; (4) strong generation performance, stemming from the emphasis on specific detail refinement.

Title: Rethinking LLM Memorization through the Lens of Adversarial Compression

• Authors: Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, J. Zico Kolter
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15146
• Pdf URL: https://arxiv.org/pdf/2404.15146
• Copy Paste: [[2404.15146]] Rethinking LLM Memorization through the Lens of Adversarial Compression(https://arxiv.org/abs/2404.15146)
• Keywords: large language model
• Abstract: Large language models (LLMs) trained on web-scale datasets raise substantial concerns regarding permissible data usage. One major question is whether these models "memorize" all their training data or they integrate many data sources in some way more akin to how a human would learn and synthesize information. The answer hinges, to a large degree, on $\textit{how we define memorization}$. In this work, we propose the Adversarial Compression Ratio (ACR) as a metric for assessing memorization in LLMs -- a given string from the training data is considered memorized if it can be elicited by a prompt shorter than the string itself. In other words, these strings can be "compressed" with the model by computing adversarial prompts of fewer tokens. We outline the limitations of existing notions of memorization and show how the ACR overcomes these challenges by (i) offering an adversarial view to measuring memorization, especially for monitoring unlearning and compliance; and (ii) allowing for the flexibility to measure memorization for arbitrary strings at a reasonably low compute. Our definition serves as a valuable and practical tool for determining when model owners may be violating terms around data usage, providing a potential legal tool and a critical lens through which to address such scenarios. Project page: https://locuslab.github.io/acr-memorization.

Title: Bias patterns in the application of LLMs for clinical decision support: A comprehensive study

• Authors: Raphael Poulain, Hamed Fayyaz, Rahmatollah Beheshti
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15149
• Pdf URL: https://arxiv.org/pdf/2404.15149
• Copy Paste: [[2404.15149]] Bias patterns in the application of LLMs for clinical decision support: A comprehensive study(https://arxiv.org/abs/2404.15149)
• Keywords: protect, large language model
• Abstract: Large Language Models (LLMs) have emerged as powerful candidates to inform clinical decision-making processes. While these models play an increasingly prominent role in shaping the digital landscape, two growing concerns emerge in healthcare applications: 1) to what extent do LLMs exhibit social bias based on patients' protected attributes (like race), and 2) how do design choices (like architecture design and prompting strategies) influence the observed biases? To answer these questions rigorously, we evaluated eight popular LLMs across three question-answering (QA) datasets using clinical vignettes (patient descriptions) standardized for bias evaluations. We employ red-teaming strategies to analyze how demographics affect LLM outputs, comparing both general-purpose and clinically-trained models. Our extensive experiments reveal various disparities (some significant) across protected groups. We also observe several counter-intuitive patterns such as larger models not being necessarily less biased and fined-tuned models on medical data not being necessarily better than the general-purpose models. Furthermore, our study demonstrates the impact of prompt design on bias patterns and shows that specific phrasing can influence bias patterns and reflection-type approaches (like Chain of Thought) can reduce biased outcomes effectively. Consistent with prior studies, we call on additional evaluations, scrutiny, and enhancement of LLMs used in clinical decision support applications.

Title: Expert Router: Orchestrating Efficient Language Model Inference through Prompt Classification

• Authors: Josef Pichlmeier, Philipp Ross, Andre Luckow
• Subjects: cs.CL, cs.AI, cs.PF
• Abstract URL: https://arxiv.org/abs/2404.15153
• Pdf URL: https://arxiv.org/pdf/2404.15153
• Copy Paste: [[2404.15153]] Expert Router: Orchestrating Efficient Language Model Inference through Prompt Classification(https://arxiv.org/abs/2404.15153)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have experienced widespread adoption across scientific and industrial domains due to their versatility and utility for diverse tasks. Nevertheless, deploying and serving these models at scale with optimal throughput and latency remains a significant challenge, primarily because of the high computational and memory demands associated with LLMs. To tackle this limitation, we introduce Expert Router, a system designed to orchestrate multiple expert models efficiently, thereby enhancing scalability. Expert Router is a parallel inference system with a central routing gateway that distributes incoming requests using a clustering method. This approach effectively partitions incoming requests among available LLMs, maximizing overall throughput. Our extensive evaluations encompassed up to 1,000 concurrent users, providing comprehensive insights into the system's behavior from user and infrastructure perspectives. The results demonstrate Expert Router's effectiveness in handling high-load scenarios and achieving higher throughput rates, particularly under many concurrent users.

Title: Do not think pink elephant!

• Authors: Kyomin Hwang, Suyoung Kim, JunHoo Lee, Nojun Kwak
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15154
• Pdf URL: https://arxiv.org/pdf/2404.15154
• Copy Paste: [[2404.15154]] Do not think pink elephant!(https://arxiv.org/abs/2404.15154)
• Keywords: defense, attack, diffusion
• Abstract: Large Models (LMs) have heightened expectations for the potential of general AI as they are akin to human intelligence. This paper shows that recent large models such as Stable Diffusion and DALL-E3 also share the vulnerability of human intelligence, namely the "white bear phenomenon". We investigate the causes of the white bear phenomenon by analyzing their representation space. Based on this analysis, we propose a simple prompt-based attack method, which generates figures prohibited by the LM provider's policy. To counter these attacks, we introduce prompt-based defense strategies inspired by cognitive therapy techniques, successfully mitigating attacks by up to 48.22\%.

Title: Adaptive Collaboration Strategy for LLMs in Medical Decision Making

• Authors: Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, Hae Won Park
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15155
• Pdf URL: https://arxiv.org/pdf/2404.15155
• Copy Paste: [[2404.15155]] Adaptive Collaboration Strategy for LLMs in Medical Decision Making(https://arxiv.org/abs/2404.15155)
• Keywords: robust
• Abstract: Foundation models have become invaluable in advancing the medical field. Despite their promise, the strategic deployment of LLMs for effective utility in complex medical tasks remains an open question. Our novel framework, Medical Decision-making Agents (MDAgents) aims to address this gap by automatically assigning the effective collaboration structure for LLMs. Assigned solo or group collaboration structure is tailored to the complexity of the medical task at hand, emulating real-world medical decision making processes. We evaluate our framework and baseline methods with state-of-the-art LLMs across a suite of challenging medical benchmarks: MedQA, MedMCQA, PubMedQA, DDXPlus, PMC-VQA, Path-VQA, and MedVidQA, achieving the best performance in 5 out of 7 benchmarks that require an understanding of multi-modal medical reasoning. Ablation studies reveal that MDAgents excels in adapting the number of collaborating agents to optimize efficiency and accuracy, showcasing its robustness in diverse scenarios. We also explore the dynamics of group consensus, offering insights into how collaborative agents could behave in complex clinical team dynamics. Our code can be found at https://github.com/mitmedialab/MDAgents.

Title: Regressive Side Effects of Training Language Models to Mimic Student Misconceptions

• Authors: Shashank Sonkar, Naiming Liu, Richard G. Baraniuk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15156
• Pdf URL: https://arxiv.org/pdf/2404.15156
• Copy Paste: [[2404.15156]] Regressive Side Effects of Training Language Models to Mimic Student Misconceptions(https://arxiv.org/abs/2404.15156)
• Keywords: large language model
• Abstract: This paper presents a novel exploration into the regressive side effects of training Large Language Models (LLMs) to mimic student misconceptions for personalized education. We highlight the problem that as LLMs are trained to more accurately mimic student misconceptions, there is a compromise in the factual integrity and reasoning ability of the models. Our work involved training an LLM on a student-tutor dialogue dataset to predict student responses. The results demonstrated a decrease in the model's performance across multiple benchmark datasets, including the ARC reasoning challenge and TruthfulQA, which evaluates the truthfulness of model's generated responses. Furthermore, the HaluEval Dial dataset, used for hallucination detection, and MemoTrap, a memory-based task dataset, also reported a decline in the model accuracy. To combat these side effects, we introduced a "hallucination token" technique. This token, appended at the beginning of each student response during training, instructs the model to switch between mimicking student misconceptions and providing factually accurate responses. Despite the significant improvement across all datasets, the technique does not completely restore the LLM's baseline performance, indicating the need for further research in this area. This paper contributes to the ongoing discussion on the use of LLMs for student modeling, emphasizing the need for a balance between personalized education and factual accuracy.

Title: FASTTRACK: Fast and Accurate Fact Tracing for LLMs

• Authors: Si Chen, Feiyang Kang, Ning Yu, Ruoxi Jia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15157
• Pdf URL: https://arxiv.org/pdf/2404.15157
• Copy Paste: [[2404.15157]] FASTTRACK: Fast and Accurate Fact Tracing for LLMs(https://arxiv.org/abs/2404.15157)
• Keywords: large language model
• Abstract: Fact tracing seeks to identify specific training examples that serve as the knowledge source for a given query. Existing approaches to fact tracing rely on assessing the similarity between each training sample and the query along a certain dimension, such as lexical similarity, gradient, or embedding space. However, these methods fall short of effectively distinguishing between samples that are merely relevant and those that actually provide supportive evidence for the information sought by the query. This limitation often results in suboptimal effectiveness. Moreover, these approaches necessitate the examination of the similarity of individual training points for each query, imposing significant computational demands and creating a substantial barrier for practical applications. This paper introduces FASTTRACK, a novel approach that harnesses the capabilities of Large Language Models (LLMs) to validate supportive evidence for queries and at the same time clusters the training database towards a reduced extent for LLMs to trace facts. Our experiments show that FASTTRACK substantially outperforms existing methods in both accuracy and efficiency, achieving more than 100\% improvement in F1 score over the state-of-the-art methods while being X33 faster than \texttt{TracIn}.

Title: MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts

• Authors: Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, Mingjie Tang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15159
• Pdf URL: https://arxiv.org/pdf/2404.15159
• Copy Paste: [[2404.15159]] MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts(https://arxiv.org/abs/2404.15159)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have showcased exceptional performance across a wide array of Natural Language Processing (NLP) tasks. Fine-tuning techniques are commonly utilized to tailor pre-trained models to specific applications. While methods like LoRA have effectively tackled GPU memory constraints during fine-tuning, their applicability is often restricted to limited performance, especially on multi-task. On the other hand, Mix-of-Expert (MoE) models, such as Mixtral 8x7B, demonstrate remarkable performance across multiple NLP tasks while maintaining a reduced parameter count. However, the resource requirements of these MoEs still challenging, particularly for consumer-grade GPUs only have limited VRAM. To address these challenge, we propose MixLoRA, an innovative approach aimed at constructing a resource-efficient sparse MoE model based on LoRA. MixLoRA inserts multiple LoRA-based experts within the feed-forward network block of a frozen pre-trained dense model through fine-tuning, employing a commonly used top-k router. Unlike other LoRA based MoE methods, MixLoRA enhances model performance by utilizing independently configurable attention-layer LoRA adapters, supporting the use of LoRA and its variants for the construction of experts, and applying auxiliary load balance loss to address the imbalance problem of the router. In experiments, MixLoRA achieves commendable performance across all evaluation metrics in both single-task and multi-task learning scenarios. Implemented within the m-LoRA framework, MixLoRA enables parallel fine-tuning of multiple mixture-of-experts models on a single 24GB consumer-grade GPU without quantization, thereby reducing GPU memory consumption by 41\% and latency during the training process by 17\%.

Title: Combating Missing Modalities in Egocentric Videos at Test Time

• Authors: Merey Ramazanova, Alejandro Pardo, Bernard Ghanem, Motasem Alfarra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15161
• Pdf URL: https://arxiv.org/pdf/2404.15161
• Copy Paste: [[2404.15161]] Combating Missing Modalities in Egocentric Videos at Test Time(https://arxiv.org/abs/2404.15161)
• Keywords: privacy
• Abstract: Understanding videos that contain multiple modalities is crucial, especially in egocentric videos, where combining various sensory inputs significantly improves tasks like action recognition and moment localization. However, real-world applications often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues. Current methods, while effective, often necessitate retraining the model entirely to handle missing modalities, making them computationally intensive, particularly with large training datasets. In this study, we propose a novel approach to address this issue at test time without requiring retraining. We frame the problem as a test-time adaptation task, where the model adjusts to the available unlabeled data at test time. Our method, MiDl~(Mutual information with self-Distillation), encourages the model to be insensitive to the specific modality source present during testing by minimizing the mutual information between the prediction and the available modality. Additionally, we incorporate self-distillation to maintain the model's original performance when both modalities are available. MiDl represents the first self-supervised, online solution for handling missing modalities exclusively at test time. Through experiments with various pretrained models and datasets, MiDl demonstrates substantial performance improvement without the need for retraining.

Title: Adaptive Mixed-Scale Feature Fusion Network for Blind AI-Generated Image Quality Assessment

• Authors: Tianwei Zhou, Songbai Tan, Wei Zhou, Yu Luo, Yuan-Gen Wang, Guanghui Yue
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.15163
• Pdf URL: https://arxiv.org/pdf/2404.15163
• Copy Paste: [[2404.15163]] Adaptive Mixed-Scale Feature Fusion Network for Blind AI-Generated Image Quality Assessment(https://arxiv.org/abs/2404.15163)
• Keywords: generative
• Abstract: With the increasing maturity of the text-to-image and image-to-image generative models, AI-generated images (AGIs) have shown great application potential in advertisement, entertainment, education, social media, etc. Although remarkable advancements have been achieved in generative models, very few efforts have been paid to design relevant quality assessment models. In this paper, we propose a novel blind image quality assessment (IQA) network, named AMFF-Net, for AGIs. AMFF-Net evaluates AGI quality from three dimensions, i.e., "visual quality", "authenticity", and "consistency". Specifically, inspired by the characteristics of the human visual system and motivated by the observation that "visual quality" and "authenticity" are characterized by both local and global aspects, AMFF-Net scales the image up and down and takes the scaled images and original-sized image as the inputs to obtain multi-scale features. After that, an Adaptive Feature Fusion (AFF) block is used to adaptively fuse the multi-scale features with learnable weights. In addition, considering the correlation between the image and prompt, AMFF-Net compares the semantic features from text encoder and image encoder to evaluate the text-to-image alignment. We carry out extensive experiments on three AGI quality assessment databases, and the experimental results show that our AMFF-Net obtains better performance than nine state-of-the-art blind IQA methods. The results of ablation experiments further demonstrate the effectiveness of the proposed multi-scale input strategy and AFF block.

Title: Pixels and Predictions: Potential of GPT-4V in Meteorological Imagery Analysis and Forecast Communication

• Authors: John R. Lawson, Montgomery L. Flora, Kevin H. Goebbert, Seth N. Lyman, Corey K. Potvin, David M. Schultz, Adam J. Stepanek, Joseph E. Trujillo-Falcón
• Subjects: cs.CL, cs.AI, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2404.15166
• Pdf URL: https://arxiv.org/pdf/2404.15166
• Copy Paste: [[2404.15166]] Pixels and Predictions: Potential of GPT-4V in Meteorological Imagery Analysis and Forecast Communication(https://arxiv.org/abs/2404.15166)
• Keywords: generative, large language model
• Abstract: Generative AI, such as OpenAI's GPT-4V large-language model, has rapidly entered mainstream discourse. Novel capabilities in image processing and natural-language communication may augment existing forecasting methods. Large language models further display potential to better communicate weather hazards in a style honed for diverse communities and different languages. This study evaluates GPT-4V's ability to interpret meteorological charts and communicate weather hazards appropriately to the user, despite challenges of hallucinations, where generative AI delivers coherent, confident, but incorrect responses. We assess GPT-4V's competence via its web interface ChatGPT in two tasks: (1) generating a severe-weather outlook from weather-chart analysis and conducting self-evaluation, revealing an outlook that corresponds well with a Storm Prediction Center human-issued forecast; and (2) producing hazard summaries in Spanish and English from weather charts. Responses in Spanish, however, resemble direct (not idiomatic) translations from English to Spanish, yielding poorly translated summaries that lose critical idiomatic precision required for optimal communication. Our findings advocate for cautious integration of tools like GPT-4V in meteorology, underscoring the necessity of human oversight and development of trustworthy, explainable AI.

Title: FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning

• Authors: Duy Phuong Nguyen, J. Pablo Munoz, Ali Jannesari
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15182
• Pdf URL: https://arxiv.org/pdf/2404.15182
• Copy Paste: [[2404.15182]] FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning(https://arxiv.org/abs/2404.15182)
• Keywords: privacy, federate
• Abstract: In the rapidly evolving field of artificial intelligence, multimodal models, e.g., integrating vision and language into visual-language models (VLMs), have become pivotal for many applications, ranging from image captioning to multimodal search engines. Among these models, the Contrastive Language-Image Pre-training (CLIP) model has demonstrated remarkable performance in understanding and generating nuanced relationships between text and images. However, the conventional training of such models often requires centralized aggregation of vast datasets, posing significant privacy and data governance challenges. To address these concerns, this paper proposes a novel approach that leverages Federated Learning and parameter-efficient adapters, i.e., Low-Rank Adaptation (LoRA), to train VLMs. This methodology preserves data privacy by training models across decentralized data sources and ensures model adaptability and efficiency through LoRA's parameter-efficient fine-tuning. Our approach accelerates training time by up to 34.72 times and requires 2.47 times less memory usage than full fine-tuning.

Title: Setting up the Data Printer with Improved English to Ukrainian Machine Translation

• Authors: Yurii Paniv, Dmytro Chaplynskyi, Nikita Trynus, Volodymyr Kyrylov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15196
• Pdf URL: https://arxiv.org/pdf/2404.15196
• Copy Paste: [[2404.15196]] Setting up the Data Printer with Improved English to Ukrainian Machine Translation(https://arxiv.org/abs/2404.15196)
• Keywords: large language model
• Abstract: To build large language models for Ukrainian we need to expand our corpora with large amounts of new algorithmic tasks expressed in natural language. Examples of task performance expressed in English are abundant, so with a high-quality translation system our community will be enabled to curate datasets faster. To aid this goal, we introduce a recipe to build a translation system using supervised finetuning of a large pretrained language model with a noisy parallel dataset of 3M pairs of Ukrainian and English sentences followed by a second phase of training using 17K examples selected by k-fold perplexity filtering on another dataset of higher quality. Our decoder-only model named Dragoman beats performance of previous state of the art encoder-decoder models on the FLORES devtest set.

Title: Towards Large-Scale Training of Pathology Foundation Models

• Authors: kaiko.ai, Nanne Aben, Edwin D. de Jong, Ioannis Gatopoulos, Nicolas Känzig, Mikhail Karasikov, Axel Lagré, Roman Moser, Joost van Doorn, Fei Tang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15217
• Pdf URL: https://arxiv.org/pdf/2404.15217
• Copy Paste: [[2404.15217]] Towards Large-Scale Training of Pathology Foundation Models(https://arxiv.org/abs/2404.15217)
• Keywords: segmentation
• Abstract: Driven by the recent advances in deep learning methods and, in particular, by the development of modern self-supervised learning algorithms, increased interest and efforts have been devoted to build foundation models (FMs) for medical images. In this work, we present our scalable training pipeline for large pathology imaging data, and a comprehensive analysis of various hyperparameter choices and training techniques for building pathology FMs. We release and make publicly available the first batch of our pathology FMs (https://github.com/kaiko-ai/towards_large_pathology_fms) trained on open-access TCGA whole slide images, a commonly used collection of pathology images. The experimental evaluation shows that our models reach state-of-the-art performance on various patch-level downstream tasks, ranging from breast cancer subtyping to colorectal nuclear segmentation. Finally, to unify the evaluation approaches used in the field and to simplify future comparisons of different FMs, we present an open-source framework (https://github.com/kaiko-ai/eva) designed for the consistent evaluation of pathology FMs across various downstream tasks.

Title: Deep Models for Multi-View 3D Object Recognition: A Review

• Authors: Mona Alzahrani, Muhammad Usman, Salma Kammoun, Saeed Anwar, Tarek Helmy
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15224
• Pdf URL: https://arxiv.org/pdf/2404.15224
• Copy Paste: [[2404.15224]] Deep Models for Multi-View 3D Object Recognition: A Review(https://arxiv.org/abs/2404.15224)
• Keywords: transformer
• Abstract: Human decision-making often relies on visual information from multiple perspectives or views. In contrast, machine learning-based object recognition utilizes information from a single image of the object. However, the information conveyed by a single image may not be sufficient for accurate decision-making, particularly in complex recognition problems. The utilization of multi-view 3D representations for object recognition has thus far demonstrated the most promising results for achieving state-of-the-art performance. This review paper comprehensively covers recent progress in multi-view 3D object recognition methods for 3D classification and retrieval tasks. Specifically, we focus on deep learning-based and transformer-based techniques, as they are widely utilized and have achieved state-of-the-art performance. We provide detailed information about existing deep learning-based and transformer-based multi-view 3D object recognition models, including the most commonly used 3D datasets, camera configurations and number of views, view selection strategies, pre-trained CNN architectures, fusion strategies, and recognition performance on 3D classification and 3D retrieval tasks. Additionally, we examine various computer vision applications that use multi-view classification. Finally, we highlight key findings and future directions for developing multi-view 3D object recognition methods to provide readers with a comprehensive understanding of the field.

Title: PHLP: Sole Persistent Homology for Link Prediction -- Interpretable Feature Extraction

• Authors: Junwon You, Eunwoo Heo, Jae-Hun Jung
• Subjects: cs.LG, cs.CG, math.AT, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.15225
• Pdf URL: https://arxiv.org/pdf/2404.15225
• Copy Paste: [[2404.15225]] PHLP: Sole Persistent Homology for Link Prediction -- Interpretable Feature Extraction(https://arxiv.org/abs/2404.15225)
• Keywords: extraction
• Abstract: Link prediction (LP), inferring the connectivity between nodes, is a significant research area in graph data, where a link represents essential information on relationships between nodes. Although graph neural network (GNN)-based models have achieved high performance in LP, understanding why they perform well is challenging because most comprise complex neural networks. We employ persistent homology (PH), a topological data analysis method that helps analyze the topological information of graphs, to explain the reasons for the high performance. We propose a novel method that employs PH for LP (PHLP) focusing on how the presence or absence of target links influences the overall topology. The PHLP utilizes the angle hop subgraph and new node labeling called degree double radius node labeling (Degree DRNL), distinguishing the information of graphs better than DRNL. Using only a classifier, PHLP performs similarly to state-of-the-art (SOTA) models on most benchmark datasets. Incorporating the outputs calculated using PHLP into the existing GNN-based SOTA models improves performance across all benchmark datasets. To the best of our knowledge, PHLP is the first method of applying PH to LP without GNNs. The proposed approach, employing PH while not relying on neural networks, enables the identification of crucial factors for improving performance.

Title: Re-Thinking Inverse Graphics With Large Language Models

• Authors: Peter Kulits, Haiwen Feng, Weiyang Liu, Victoria Abrevaya, Michael J. Black
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15228
• Pdf URL: https://arxiv.org/pdf/2404.15228
• Copy Paste: [[2404.15228]] Re-Thinking Inverse Graphics With Large Language Models(https://arxiv.org/abs/2404.15228)
• Keywords: large language model
• Abstract: Inverse graphics -- the task of inverting an image into physical variables that, when rendered, enable reproduction of the observed scene -- is a fundamental challenge in computer vision and graphics. Disentangling an image into its constituent elements, such as the shape, color, and material properties of the objects of the 3D scene that produced it, requires a comprehensive understanding of the environment. This requirement limits the ability of existing carefully engineered approaches to generalize across domains. Inspired by the zero-shot ability of large language models (LLMs) to generalize to novel contexts, we investigate the possibility of leveraging the broad world knowledge encoded in such models in solving inverse-graphics problems. To this end, we propose the Inverse-Graphics Large Language Model (IG-LLM), an inverse-graphics framework centered around an LLM, that autoregressively decodes a visual embedding into a structured, compositional 3D-scene representation. We incorporate a frozen pre-trained visual encoder and a continuous numeric head to enable end-to-end training. Through our investigation, we demonstrate the potential of LLMs to facilitate inverse graphics through next-token prediction, without the use of image-space supervision. Our analysis opens up new possibilities for precise spatial reasoning about images that exploit the visual knowledge of LLMs. We will release our code and data to ensure the reproducibility of our investigation and to facilitate future research at https://ig-llm.is.tue.mpg.de/

Title: Massively Annotated Datasets for Assessment of Synthetic and Real Data in Face Recognition

• Authors: Pedro C. Neto, Rafael M. Mamede, Carolina Albuquerque, Tiago Gonçalves, Ana F. Sequeira
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15234
• Pdf URL: https://arxiv.org/pdf/2404.15234
• Copy Paste: [[2404.15234]] Massively Annotated Datasets for Assessment of Synthetic and Real Data in Face Recognition(https://arxiv.org/abs/2404.15234)
• Keywords: privacy, generative
• Abstract: Face recognition applications have grown in parallel with the size of datasets, complexity of deep learning models and computational power. However, while deep learning models evolve to become more capable and computational power keeps increasing, the datasets available are being retracted and removed from public access. Privacy and ethical concerns are relevant topics within these domains. Through generative artificial intelligence, researchers have put efforts into the development of completely synthetic datasets that can be used to train face recognition systems. Nonetheless, the recent advances have not been sufficient to achieve performance comparable to the state-of-the-art models trained on real data. To study the drift between the performance of models trained on real and synthetic datasets, we leverage a massive attribute classifier (MAC) to create annotations for four datasets: two real and two synthetic. From these annotations, we conduct studies on the distribution of each attribute within all four datasets. Additionally, we further inspect the differences between real and synthetic datasets on the attribute set. When comparing through the Kullback-Leibler divergence we have found differences between real and synthetic samples. Interestingly enough, we have verified that while real samples suffice to explain the synthetic distribution, the opposite could not be further from being true.

Title: A Hybrid Kernel-Free Boundary Integral Method with Operator Learning for Solving Parametric Partial Differential Equations In Complex Domains

• Authors: Shuo Ling, Liwei Tan, Wenjun Ying
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2404.15242
• Pdf URL: https://arxiv.org/pdf/2404.15242
• Copy Paste: [[2404.15242]] A Hybrid Kernel-Free Boundary Integral Method with Operator Learning for Solving Parametric Partial Differential Equations In Complex Domains(https://arxiv.org/abs/2404.15242)
• Keywords: robust
• Abstract: The Kernel-Free Boundary Integral (KFBI) method presents an iterative solution to boundary integral equations arising from elliptic partial differential equations (PDEs). This method effectively addresses elliptic PDEs on irregular domains, including the modified Helmholtz, Stokes, and elasticity equations. The rapid evolution of neural networks and deep learning has invigorated the exploration of numerical PDEs. An increasing interest is observed in deep learning approaches that seamlessly integrate mathematical principles for investigating numerical PDEs. We propose a hybrid KFBI method, integrating the foundational principles of the KFBI method with the capabilities of deep learning. This approach, within the framework of the boundary integral method, designs a network to approximate the solution operator for the corresponding integral equations by mapping the parameters, inhomogeneous terms and boundary information of PDEs to the boundary density functions, which can be regarded as the solution of the integral equations. The models are trained using data generated by the Cartesian grid-based KFBI algorithm, exhibiting robust generalization capabilities. It accurately predicts density functions across diverse boundary conditions and parameters within the same class of equations. Experimental results demonstrate that the trained model can directly infer the boundary density function with satisfactory precision, obviating the need for iterative steps in solving boundary integral equations. Furthermore, applying the inference results of the model as initial values for iterations is also reasonable; this approach can retain the inherent second-order accuracy of the KFBI method while accelerating the traditional KFBI approach by reducing about 50% iterations.

Title: Efficient Transformer Encoders for Mask2Former-style models

• Authors: Manyi Yao, Abhishek Aich, Yumin Suh, Amit Roy-Chowdhury, Christian Shelton, Manmohan Chandraker
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15244
• Pdf URL: https://arxiv.org/pdf/2404.15244
• Copy Paste: [[2404.15244]] Efficient Transformer Encoders for Mask2Former-style models(https://arxiv.org/abs/2404.15244)
• Keywords: transformer, segmentation
• Abstract: Vision transformer based models bring significant improvements for image segmentation tasks. Although these architectures offer powerful capabilities irrespective of specific segmentation tasks, their use of computational resources can be taxing on deployed devices. One way to overcome this challenge is by adapting the computation level to the specific needs of the input image rather than the current one-size-fits-all approach. To this end, we introduce ECO-M2F or EffiCient TransfOrmer Encoders for Mask2Former-style models. Noting that the encoder module of M2F-style models incur high resource-intensive computations, ECO-M2F provides a strategy to self-select the number of hidden layers in the encoder, conditioned on the input image. To enable this self-selection ability for providing a balance between performance and computational efficiency, we present a three step recipe. The first step is to train the parent architecture to enable early exiting from the encoder. The second step is to create an derived dataset of the ideal number of encoder layers required for each training example. The third step is to use the aforementioned derived dataset to train a gating network that predicts the number of encoder layers to be used, conditioned on the input image. Additionally, to change the computational-accuracy tradeoff, only steps two and three need to be repeated which significantly reduces retraining time. Experiments on the public datasets show that the proposed approach reduces expected encoder computational cost while maintaining performance, adapts to various user compute resources, is flexible in architecture configurations, and can be extended beyond the segmentation task to object detection.

Title: XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts

• Authors: Yifeng Ding, Jiawei Liu, Yuxiang Wei, Terry Yue Zhuo, Lingming Zhang
• Subjects: cs.CL, cs.AI, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.15247
• Pdf URL: https://arxiv.org/pdf/2404.15247
• Copy Paste: [[2404.15247]] XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts(https://arxiv.org/abs/2404.15247)
• Keywords: large language model
• Abstract: We introduce XFT, a simple yet powerful training scheme, by simply merging upcycled Mixture-of-Experts (MoE) to unleash the performance limit of instruction-tuned code Large Language Models (LLMs). While vanilla sparse upcycling fails to improve instruction tuning, XFT introduces a shared expert mechanism with a novel routing weight normalization strategy into sparse upcycling, which significantly boosts instruction tuning. After fine-tuning the upcycled MoE model, XFT introduces a learnable model merging mechanism to compile the upcycled MoE model back to a dense model, achieving upcycled MoE-level performance with only dense-model compute. By applying XFT to a 1.3B model, we create a new state-of-the-art tiny code LLM (<3B) with 67.1 and 64.6 pass@1 on HumanEval and HumanEval+ respectively. With the same data and model architecture, XFT improves supervised fine-tuning (SFT) by 13% on HumanEval+, along with consistent improvements from 2% to 13% on MBPP+, MultiPL-E, and DS-1000, demonstrating its generalizability. XFT is fully orthogonal to existing techniques such as Evol-Instruct and OSS-Instruct, opening a new dimension for improving code instruction tuning. Codes are available at https://github.com/ise-uiuc/xft .

Title: UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition

• Authors: Bin Wang, Zhuangcheng Gu, Chao Xu, Bo Zhang, Botian Shi, Conghui He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15254
• Pdf URL: https://arxiv.org/pdf/2404.15254
• Copy Paste: [[2404.15254]] UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition(https://arxiv.org/abs/2404.15254)
• Keywords: robust
• Abstract: This paper presents the UniMER dataset to provide the first study on Mathematical Expression Recognition (MER) towards complex real-world scenarios. The UniMER dataset consists of a large-scale training set UniMER-1M offering an unprecedented scale and diversity with one million training instances and a meticulously designed test set UniMER-Test that reflects a diverse range of formula distributions prevalent in real-world scenarios. Therefore, the UniMER dataset enables the training of a robust and high-accuracy MER model and comprehensive evaluation of model performance. Moreover, we introduce the Universal Mathematical Expression Recognition Network (UniMERNet), an innovative framework designed to enhance MER in practical scenarios. UniMERNet incorporates a Length-Aware Module to process formulas of varied lengths efficiently, thereby enabling the model to handle complex mathematical expressions with greater accuracy. In addition, UniMERNet employs our UniMER-1M data and image augmentation techniques to improve the model's robustness under different noise conditions. Our extensive experiments demonstrate that UniMERNet outperforms existing MER models, setting a new benchmark in various scenarios and ensuring superior recognition quality in real-world applications. The dataset and model are available at https://github.com/opendatalab/UniMERNet.

Title: How to use and interpret activation patching

• Authors: Stefan Heimersheim, Neel Nanda
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15255
• Pdf URL: https://arxiv.org/pdf/2404.15255
• Copy Paste: [[2404.15255]] How to use and interpret activation patching(https://arxiv.org/abs/2404.15255)
• Keywords: interpretability
• Abstract: Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a summary of advice and best practices, based on our experience using this technique in practice. We include an overview of the different ways to apply activation patching and a discussion on how to interpret the results. We focus on what evidence patching experiments provide about circuits, and on the choice of metric and associated pitfalls.

Title: Multi-Session SLAM with Differentiable Wide-Baseline Pose Optimization

• Authors: Lahav Lipson, Jia Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15263
• Pdf URL: https://arxiv.org/pdf/2404.15263
• Copy Paste: [[2404.15263]] Multi-Session SLAM with Differentiable Wide-Baseline Pose Optimization(https://arxiv.org/abs/2404.15263)
• Keywords: robust
• Abstract: We introduce a new system for Multi-Session SLAM, which tracks camera motion across multiple disjoint videos under a single global reference. Our approach couples the prediction of optical flow with solver layers to estimate camera pose. The backbone is trained end-to-end using a novel differentiable solver for wide-baseline two-view pose. The full system can connect disjoint sequences, perform visual odometry, and global optimization. Compared to existing approaches, our design is accurate and robust to catastrophic failures. Code is available at github.com/princeton-vl/MultiSlam_DiffPose

Title: From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation

• Authors: Zehuan Huang, Hongxing Fan, Lipeng Wang, Lu Sheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15267
• Pdf URL: https://arxiv.org/pdf/2404.15267
• Copy Paste: [[2404.15267]] From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation(https://arxiv.org/abs/2404.15267)
• Keywords: diffusion
• Abstract: Recent advancements in controllable human image generation have led to zero-shot generation using structural signals (e.g., pose, depth) or facial appearance. Yet, generating human images conditioned on multiple parts of human appearance remains challenging. Addressing this, we introduce Parts2Whole, a novel framework designed for generating customized portraits from multiple reference images, including pose images and various aspects of human appearance. To achieve this, we first develop a semantic-aware appearance encoder to retain details of different human parts, which processes each image based on its textual label to a series of multi-scale feature maps rather than one image token, preserving the image dimension. Second, our framework supports multi-image conditioned generation through a shared self-attention mechanism that operates across reference and target features during the diffusion process. We enhance the vanilla attention mechanism by incorporating mask information from the reference human images, allowing for the precise selection of any part. Extensive experiments demonstrate the superiority of our approach over existing alternatives, offering advanced capabilities for multi-part controllable human image customization. See our project page at https://huanngzh.github.io/Parts2Whole/.

Title: Aligning LLM Agents by Learning Latent Preference from User Edits

• Authors: Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, Dipendra Misra
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15269
• Pdf URL: https://arxiv.org/pdf/2404.15269
• Copy Paste: [[2404.15269]] Aligning LLM Agents by Learning Latent Preference from User Edits(https://arxiv.org/abs/2404.15269)
• Keywords: interpretability, large language model
• Abstract: We study interactive learning of language agents based on user edits made to the agent's output. In a typical setting such as writing assistants, the user interacts with a language agent to generate a response given a context, and may optionally edit the agent response to personalize it based on their latent preference, in addition to improving the correctness. The edit feedback is naturally generated, making it a suitable candidate for improving the agent's alignment with the user's preference, and for reducing the cost of user edits over time. We propose a learning framework, PRELUDE that infers a description of the user's latent preference based on historic edit data and using it to define a prompt policy that drives future response generation. This avoids fine-tuning the agent, which is costly, challenging to scale with the number of users, and may even degrade its performance on other tasks. Furthermore, learning descriptive preference improves interpretability, allowing the user to view and modify the learned preference. However, user preference can be complex and vary based on context, making it challenging to learn. To address this, we propose a simple yet effective algorithm named CIPHER that leverages a large language model (LLM) to infer the user preference for a given context based on user edits. In the future, CIPHER retrieves inferred preferences from the k-closest contexts in the history, and forms an aggregate preference for response generation. We introduce two interactive environments -- summarization and email writing, for evaluation using a GPT-4 simulated user. We compare with algorithms that directly retrieve user edits but do not learn descriptive preference, and algorithms that learn context-agnostic preference. On both tasks, CIPHER achieves the lowest edit distance cost and learns preferences that show significant similarity to the ground truth preferences

Title: ID-Animator: Zero-Shot Identity-Preserving Human Video Generation

• Authors: Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, Man Zhou, Jie Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15275
• Pdf URL: https://arxiv.org/pdf/2404.15275
• Copy Paste: [[2404.15275]] ID-Animator: Zero-Shot Identity-Preserving Human Video Generation(https://arxiv.org/abs/2404.15275)
• Keywords: extraction, diffusion
• Abstract: Generating high fidelity human video with specified identities has attracted significant attention in the content generation community. However, existing techniques struggle to strike a balance between training efficiency and identity preservation, either requiring tedious case-by-case finetuning or usually missing the identity details in video generation process. In this study, we present ID-Animator, a zero-shot human-video generation approach that can perform personalized video generation given single reference facial image without further training. ID-Animator inherits existing diffusion-based video generation backbones with a face adapter to encode the ID-relevant embeddings from learnable facial latent queries. To facilitate the extraction of identity information in video generation, we introduce an ID-oriented dataset construction pipeline, which incorporates decoupled human attribute and action captioning technique from a constructed facial image pool. Based on this pipeline, a random face reference training method is further devised to precisely capture the ID-relevant embeddings from reference images, thus improving the fidelity and generalization capacity of our model for ID-specific video generation. Extensive experiments demonstrate the superiority of ID-Animator to generate personalized human videos over previous models. Moreover, our method is highly compatible with popular pre-trained T2V models like animatediff and various community backbone models, showing high extendability in real-world applications for video generation where identity preservation is highly desired. Our codes and checkpoints will be released at https://github.com/ID-Animator/ID-Animator.

Title: SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation

• Authors: Xiangyu Xu, Lijuan Liu, Shuicheng Yan
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.15276
• Pdf URL: https://arxiv.org/pdf/2404.15276
• Copy Paste: [[2404.15276]] SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation(https://arxiv.org/abs/2404.15276)
• Keywords: transformer
• Abstract: Existing Transformers for monocular 3D human shape and pose estimation typically have a quadratic computation and memory complexity with respect to the feature length, which hinders the exploitation of fine-grained information in high-resolution features that is beneficial for accurate reconstruction. In this work, we propose an SMPL-based Transformer framework (SMPLer) to address this issue. SMPLer incorporates two key ingredients: a decoupled attention operation and an SMPL-based target representation, which allow effective utilization of high-resolution features in the Transformer. In addition, based on these two designs, we also introduce several novel modules including a multi-scale attention and a joint-aware attention to further boost the reconstruction performance. Extensive experiments demonstrate the effectiveness of SMPLer against existing 3D human shape and pose estimation methods both quantitatively and qualitatively. Notably, the proposed algorithm achieves an MPJPE of 45.2 mm on the Human3.6M dataset, improving upon Mesh Graphormer by more than 10% with fewer than one-third of the parameters. Code and pretrained models are available at https://github.com/xuxy09/SMPLer.