2024-07-19

Title: SS-ADA: A Semi-Supervised Active Domain Adaptation Framework for Semantic Segmentation

• Authors: Weihao Yan, Yeqiang Qian, Yueyuan Li, Tao Li, Chunxiang Wang, Ming Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12788
• Pdf URL: https://arxiv.org/pdf/2407.12788
• Copy Paste: [[2407.12788]] SS-ADA: A Semi-Supervised Active Domain Adaptation Framework for Semantic Segmentation(https://arxiv.org/abs/2407.12788)
• Keywords: segmentation
• Abstract: Semantic segmentation plays an important role in intelligent vehicles, providing pixel-level semantic information about the environment. However, the labeling budget is expensive and time-consuming when semantic segmentation model is applied to new driving scenarios. To reduce the costs, semi-supervised semantic segmentation methods have been proposed to leverage large quantities of unlabeled images. Despite this, their performance still falls short of the accuracy required for practical applications, which is typically achieved by supervised learning. A significant shortcoming is that they typically select unlabeled images for annotation randomly, neglecting the assessment of sample value for model training. In this paper, we propose a novel semi-supervised active domain adaptation (SS-ADA) framework for semantic segmentation that employs an image-level acquisition strategy. SS-ADA integrates active learning into semi-supervised semantic segmentation to achieve the accuracy of supervised learning with a limited amount of labeled data from the target domain. Additionally, we design an IoU-based class weighting strategy to alleviate the class imbalance problem using annotations from active learning. We conducted extensive experiments on synthetic-to-real and real-to-real domain adaptation settings. The results demonstrate the effectiveness of our method. SS-ADA can achieve or even surpass the accuracy of its supervised learning counterpart with only 25% of the target labeled data when using a real-time segmentation model. The code for SS-ADA is available at this https URL.

Title: GPT Czech Poet: Generation of Czech Poetic Strophes with Language Models

• Authors: Michal Chudoba, Rudolf Rosa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12790
• Pdf URL: https://arxiv.org/pdf/2407.12790
• Copy Paste: [[2407.12790]] GPT Czech Poet: Generation of Czech Poetic Strophes with Language Models(https://arxiv.org/abs/2407.12790)
• Keywords: large language model
• Abstract: High-quality automated poetry generation systems are currently only available for a small subset of languages. We introduce a new model for generating poetry in Czech language, based on fine-tuning a pre-trained Large Language Model. We demonstrate that guiding the generation process by explicitly specifying strophe parameters within the poem text strongly improves the effectiveness of the model. We also find that appropriate tokenization is crucial, showing that tokenization methods based on syllables or individual characters instead of subwords prove superior in generating poetic strophes. We further enhance the results by introducing \textit{Forced~generation}, adding explicit specifications of meter and verse parameters at inference time based on the already generated text. We evaluate a range of setups, showing that our proposed approach achieves high accuracies in rhyming and metric aspects of formal quality of the generated poems.

Title: TourLLM: Enhancing LLMs with Tourism Knowledge

• Authors: Qikai Wei, Mingzhi Yang, Jinqiang Wang, Wenwei Mao, Jiabo Xu, Huansheng Ning
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12791
• Pdf URL: https://arxiv.org/pdf/2407.12791
• Copy Paste: [[2407.12791]] TourLLM: Enhancing LLMs with Tourism Knowledge(https://arxiv.org/abs/2407.12791)
• Keywords: large language model
• Abstract: Recently, large language models (LLMs) have demonstrated their effectiveness in various natural language processing (NLP) tasks. However, the lack of tourism knowledge limits the performance of LLMs in tourist attraction presentations and travel planning. To address this challenge, we constructed a supervised fine-tuning dataset for the culture and tourism domain, named Cultour. This dataset consists of three parts: tourism knowledge base QA data, travelogues data, and tourism diversity QA data. Additionally, we propose TourLLM, a Qwen-based model supervised fine-tuned with Cultour, to improve the quality of the information provided about attractions and travel planning. To evaluate the performance of TourLLM, we employed both automatic and human evaluation, and we proposed a human evaluation criterion named CRA (Consistency, Readability, Availability). The experimental results demonstrate the effectiveness of the responses generated by the TourLLM. Our proposed Cultour is accessible at this https URL.

Title: Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learning

• Authors: Vittorio Giammarino, James Queeney, Ioannis Ch. Paschalidis
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12792
• Pdf URL: https://arxiv.org/pdf/2407.12792
• Copy Paste: [[2407.12792]] Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learning(https://arxiv.org/abs/2407.12792)
• Keywords: robust
• Abstract: We propose C-LAIfO, a computationally efficient algorithm designed for imitation learning from videos, even in the presence of visual mismatch between agent and expert domains. We analyze the problem of imitation from expert videos with visual discrepancies, and introduce a solution for robust latent space estimation using contrastive learning and data augmentation. Provided a visually robust latent space, our algorithm performs imitation entirely within this space using off-policy adversarial imitation learning. We conduct a thorough ablation study to justify our design choices and test C-LAIfO on high-dimensional continuous robotic tasks. Additionally, we demonstrate how C-LAIfO can be combined with other reward signals to facilitate learning on a set of challenging hand manipulation tasks with sparse rewards. Our experiments show improved performance compared to baseline methods, highlighting the effectiveness and versatility of C-LAIfO. To ensure reproducibility, we provide open access to our code.

Title: Dark Transformer: A Video Transformer for Action Recognition in the Dark

• Authors: Anwaar Ulhaq
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12805
• Pdf URL: https://arxiv.org/pdf/2407.12805
• Copy Paste: [[2407.12805]] Dark Transformer: A Video Transformer for Action Recognition in the Dark(https://arxiv.org/abs/2407.12805)
• Keywords: transformer
• Abstract: Recognizing human actions in adverse lighting conditions presents significant challenges in computer vision, with wide-ranging applications in visual surveillance and nighttime driving. Existing methods tackle action recognition and dark enhancement separately, limiting the potential for end-to-end learning of spatiotemporal representations for video action classification. This paper introduces Dark Transformer, a novel video transformer-based approach for action recognition in low-light environments. Dark Transformer leverages spatiotemporal self-attention mechanisms in cross-domain settings to enhance cross-domain action recognition. By extending video transformers to learn cross-domain knowledge, Dark Transformer achieves state-of-the-art performance on benchmark action recognition datasets, including InFAR, XD145, and ARID. The proposed approach demonstrates significant promise in addressing the challenges of action recognition in adverse lighting conditions, offering practical implications for real-world applications.

Title: Towards Optimal Trade-offs in Knowledge Distillation for CNNs and Vision Transformers at the Edge

• Authors: John Violos, Symeon Papadopoulos, Ioannis Kompatsiaris
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12808
• Pdf URL: https://arxiv.org/pdf/2407.12808
• Copy Paste: [[2407.12808]] Towards Optimal Trade-offs in Knowledge Distillation for CNNs and Vision Transformers at the Edge(https://arxiv.org/abs/2407.12808)
• Keywords: transformer
• Abstract: This paper discusses four facets of the Knowledge Distillation (KD) process for Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) architectures, particularly when executed on edge devices with constrained processing capabilities. First, we conduct a comparative analysis of the KD process between CNNs and ViT architectures, aiming to elucidate the feasibility and efficacy of employing different architectural configurations for the teacher and student, while assessing their performance and efficiency. Second, we explore the impact of varying the size of the student model on accuracy and inference speed, while maintaining a constant KD duration. Third, we examine the effects of employing higher resolution images on the accuracy, memory footprint and computational workload. Last, we examine the performance improvements obtained by fine-tuning the student model after KD to specific downstream tasks. Through empirical evaluations and analyses, this research provides AI practitioners with insights into optimal strategies for maximizing the effectiveness of the KD process on edge devices.

Title: Building Understandable Messaging for Policy and Evidence Review (BUMPER) with AI

• Authors: Katherine A. Rosenfeld, Maike Sonnewald, Sonia J. Jindal, Kevin A. McCarthy, Joshua L. Proctor
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12812
• Pdf URL: https://arxiv.org/pdf/2407.12812
• Copy Paste: [[2407.12812]] Building Understandable Messaging for Policy and Evidence Review (BUMPER) with AI(https://arxiv.org/abs/2407.12812)
• Keywords: large language model
• Abstract: We introduce a framework for the use of large language models (LLMs) in Building Understandable Messaging for Policy and Evidence Review (BUMPER). LLMs are proving capable of providing interfaces for understanding and synthesizing large databases of diverse media. This presents an exciting opportunity to supercharge the translation of scientific evidence into policy and action, thereby improving livelihoods around the world. However, these models also pose challenges related to access, trust-worthiness, and accountability. The BUMPER framework is built atop a scientific knowledge base (e.g., documentation, code, survey data) by the same scientists (e.g., individual contributor, lab, consortium). We focus on a solution that builds trustworthiness through transparency, scope-limiting, explicit-checks, and uncertainty measures. LLMs are rapidly being adopted and consequences are poorly understood. The framework addresses open questions regarding the reliability of LLMs and their use in high-stakes applications. We provide a worked example in health policy for a model designed to inform measles control programs. We argue that this framework can facilitate accessibility of and confidence in scientific evidence for policymakers, drive a focus on policy-relevance and translatability for researchers, and ultimately increase and accelerate the impact of scientific knowledge used for policy decisions.

Title: Data Generation using Large Language Models for Text Classification: An Empirical Case Study

• Authors: Yinheng Li, Rogerio Bonatti, Sara Abdali, Justin Wagle, Kazuhito Koishida
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12813
• Pdf URL: https://arxiv.org/pdf/2407.12813
• Copy Paste: [[2407.12813]] Data Generation using Large Language Models for Text Classification: An Empirical Case Study(https://arxiv.org/abs/2407.12813)
• Keywords: large language model
• Abstract: Using Large Language Models (LLMs) to generate synthetic data for model training has become increasingly popular in recent years. While LLMs are capable of producing realistic training data, the effectiveness of data generation is influenced by various factors, including the choice of prompt, task complexity, and the quality, quantity, and diversity of the generated data. In this work, we focus exclusively on using synthetic data for text classification tasks. Specifically, we use natural language understanding (NLU) models trained on synthetic data to assess the quality of synthetic data from different generation approaches. This work provides an empirical analysis of the impact of these factors and offers recommendations for better data generation practices.

Title: Computational Politeness in Natural Language Processing: A Survey

• Authors: Priyanshu Priya, Mauajama Firdaus, Asif Ekbal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12814
• Pdf URL: https://arxiv.org/pdf/2407.12814
• Copy Paste: [[2407.12814]] Computational Politeness in Natural Language Processing: A Survey(https://arxiv.org/abs/2407.12814)
• Keywords: extraction
• Abstract: Computational approach to politeness is the task of automatically predicting and generating politeness in text. This is a pivotal task for conversational analysis, given the ubiquity and challenges of politeness in interactions. The computational approach to politeness has witnessed great interest from the conversational analysis community. This article is a compilation of past works in computational politeness in natural language processing. We view four milestones in the research so far, viz. supervised and weakly-supervised feature extraction to identify and induce politeness in a given text, incorporation of context beyond the target text, study of politeness across different social factors, and study the relationship between politeness and various sociolinguistic cues. In this article, we describe the datasets, approaches, trends, and issues in computational politeness research. We also discuss representative performance values and provide pointers to future works, as given in the prior works. In terms of resources to understand the state-of-the-art, this survey presents several valuable illustrations, most prominently, a table summarizing the past papers along different dimensions, such as the types of features, annotation techniques, and datasets used.

Title: SMLT-MUGC: Small, Medium, and Large Texts -- Machine versus User-Generated Content Detection and Comparison

• Authors: Anjali Rawal, Hui Wang, Youjia Zheng, Yu-Hsuan Lin, Shanu Sushmita
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12815
• Pdf URL: https://arxiv.org/pdf/2407.12815
• Copy Paste: [[2407.12815]] SMLT-MUGC: Small, Medium, and Large Texts -- Machine versus User-Generated Content Detection and Comparison(https://arxiv.org/abs/2407.12815)
• Keywords: large language model
• Abstract: Large language models (LLMs) have gained significant attention due to their ability to mimic human language. Identifying texts generated by LLMs is crucial for understanding their capabilities and mitigating potential consequences. This paper analyzes datasets of varying text lengths: small, medium, and large. We compare the performance of machine learning algorithms on four datasets: (1) small (tweets from Election, FIFA, and Game of Thrones), (2) medium (Wikipedia introductions and PubMed abstracts), and (3) large (OpenAI web text dataset). Our results indicate that LLMs with very large parameters (such as the XL-1542 variant of GPT2 with 1542 million parameters) were harder (74%) to detect using traditional machine learning methods. However, detecting texts of varying lengths from LLMs with smaller parameters (762 million or less) can be done with high accuracy (96% and above). We examine the characteristics of human and machine-generated texts across multiple dimensions, including linguistics, personality, sentiment, bias, and morality. Our findings indicate that machine-generated texts generally have higher readability and closely mimic human moral judgments but differ in personality traits. SVM and Voting Classifier (VC) models consistently achieve high performance across most datasets, while Decision Tree (DT) models show the lowest performance. Model performance drops when dealing with rephrased texts, particularly shorter texts like tweets. This study underscores the challenges and importance of detecting LLM-generated texts and suggests directions for future research to improve detection methods and understand the nuanced capabilities of LLMs.

Title: "I understand why I got this grade": Automatic Short Answer Grading with Feedback

• Authors: Dishank Aggarwal, Pushpak Bhattacharyya, Bhaskaran Raman
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.12818
• Pdf URL: https://arxiv.org/pdf/2407.12818
• Copy Paste: [[2407.12818]] "I understand why I got this grade": Automatic Short Answer Grading with Feedback(https://arxiv.org/abs/2407.12818)
• Keywords: generative, large language model
• Abstract: The demand for efficient and accurate assessment methods has intensified as education systems transition to digital platforms. Providing feedback is essential in educational settings and goes beyond simply conveying marks as it justifies the assigned marks. In this context, we present a significant advancement in automated grading by introducing Engineering Short Answer Feedback (EngSAF) -- a dataset of 5.8k student answers accompanied by reference answers and questions for the Automatic Short Answer Grading (ASAG) task. The EngSAF dataset is meticulously curated to cover a diverse range of subjects, questions, and answer patterns from multiple engineering domains. We leverage state-of-the-art large language models' (LLMs) generative capabilities with our Label-Aware Synthetic Feedback Generation (LASFG) strategy to include feedback in our dataset. This paper underscores the importance of enhanced feedback in practical educational settings, outlines dataset annotation and feedback generation processes, conducts a thorough EngSAF analysis, and provides different LLMs-based zero-shot and finetuned baselines for future comparison. Additionally, we demonstrate the efficiency and effectiveness of the ASAG system through its deployment in a real-world end-semester exam at the Indian Institute of Technology Bombay (IITB), showcasing its practical viability and potential for broader implementation in educational institutions.

Title: PQCache: Product Quantization-based KVCache for Long Context LLM Inference

• Authors: Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, Bin Cui
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12820
• Pdf URL: https://arxiv.org/pdf/2407.12820
• Copy Paste: [[2407.12820]] PQCache: Product Quantization-based KVCache for Long Context LLM Inference(https://arxiv.org/abs/2407.12820)
• Keywords: large language model
• Abstract: As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), a crucial component in LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either fall short in maintaining model quality or result in high serving latency. Drawing inspiration from advanced embedding retrieval techniques used in the database community, we consider the storage and searching of KVCache as a typical embedding retrieval problem. We propose PQCache, which employs Product Quantization (PQ) to manage KVCache, maintaining model quality while ensuring low serving latency. During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head. During the autoregressive decoding phase, for each newly generated token, we first identify important tokens through Maximum Inner-Product Search (MIPS) using PQ codes and centroids, then fetch the corresponding key-value pairs for self-attention computation. Through meticulous design of overlapping and caching, we minimize any additional computation and communication overhead during both phases. Extensive experiments show that PQCache achieves both effectiveness and efficiency. It maintains model quality even when only 1/5 of the tokens are involved in attention, while attaining acceptable system latency.

Title: AutoFlow: Automated Workflow Generation for Large Language Model Agents

• Authors: Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, Yongfeng Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12821
• Pdf URL: https://arxiv.org/pdf/2407.12821
• Copy Paste: [[2407.12821]] AutoFlow: Automated Workflow Generation for Large Language Model Agents(https://arxiv.org/abs/2407.12821)
• Keywords: robust, large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have shown significant progress in understanding complex natural language. One important application of LLM is LLM-based AI Agent, which leverages the ability of LLM as well as external tools for complex-task solving. To make sure LLM Agents follow an effective and reliable procedure to solve the given task, manually designed workflows are usually used to guide the working mechanism of agents. However, manually designing the workflows requires considerable efforts and domain knowledge, making it difficult to develop and deploy agents on massive scales. To address these issues, we propose AutoFlow, a framework designed to automatically generate workflows for agents to solve complex tasks. AutoFlow takes natural language program as the format of agent workflow and employs a workflow optimization procedure to iteratively optimize the workflow quality. Besides, this work offers two workflow generation methods: fine-tuning-based and in-context-based methods, making the AutoFlow framework applicable to both open-source and closed-source LLMs. Experimental results show that our framework can produce robust and reliable agent workflows. We believe that the automatic generation and interpretation of workflows in natural language represent a promising paradigm for solving complex tasks, particularly with the rapid development of LLMs. The source code of this work is available at this https URL.

Title: Lightweight Large Language Model for Medication Enquiry: Med-Pal

• Authors: Kabilan Elangovan, Jasmine Chiat Ling Ong, Liyuan Jin, Benjamin Jun Jie Seng, Yu Heng Kwan, Lit Soo Tan, Ryan Jian Zhong, Justina Koi Li Ma, YuHe Ke, Nan Liu, Kathleen M Giacomini, Daniel Shu Wei Ting
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12822
• Pdf URL: https://arxiv.org/pdf/2407.12822
• Copy Paste: [[2407.12822]] Lightweight Large Language Model for Medication Enquiry: Med-Pal(https://arxiv.org/abs/2407.12822)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have emerged as a potential solution to assist digital health development with patient education, commonly medication-related enquires. We trained and validated Med-Pal, a medication domain-specific LLM-chatbot fine-tuned with a fine-grained and expert curated dataset from a selection of five light-weighted open-source LLMs of smaller parameter size (7 billion or less) regarding computational constraints and prioritizing operational efficiency. A multi-disciplinary team performed a clinical evaluation of LLMs responses using the SCORE criteria, focusing on safety, accuracy, bias, reproducibility, and ease of understanding. Best performing light-weighted LLM was chosen as Med-Pal for further engineering with guard-railing using adversarial prompting. Med-Pal and existing light-weighted LLMs, including pretrained Biomistral and finetuned Meerkat, were validated on an independent dataset on a broad range of medication-related questions (231 in total), 12 different question types across 14 different medication classes. Mistral-7b emerged as the top performer among selected lightweight LLMs, achieving the highest median score of 14 and 71.9% high-quality responses in accuracy and safety domains, hence chosen as the backbone LLM for Med-Pal. When compared against Biomistral, Med-pal outperformed in generating responses appropriate for patient communication, with significant reductions bias and errors typical of general LLMs. Comparable performance was observed when comparing Med-Pal with Meerkat. Med-Pal showcases the feasibility of developing and employing fine-tuned light-weighted LLMs to enhance digital health communications.

Title: WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models

• Authors: Kangyun Ning, Yisong Su, Xueqiang Lv, Yuanzhe Zhang, Jian Liu, Kang Liu, Jinan Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12823
• Pdf URL: https://arxiv.org/pdf/2407.12823
• Copy Paste: [[2407.12823]] WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models(https://arxiv.org/abs/2407.12823)
• Keywords: large language model
• Abstract: Although Large Language Models (LLMs) excel in NLP tasks, they still need external tools to extend their ability. Current research on tool learning with LLMs often assumes mandatory tool use, which does not always align with real-world situations, where the necessity for tools is uncertain, and incorrect or unnecessary use of tools can damage the general abilities of LLMs. Therefore, we propose to explore whether LLMs can discern their ability boundaries and use tools flexibly. We then introduce the Whether-or-not tool usage Evaluation benchmark (WTU-Eval) to assess LLMs with eleven datasets, where six of them are tool-usage datasets, and five are general datasets. LLMs are prompted to use tools according to their needs. The results of eight LLMs on WTU-Eval reveal that LLMs frequently struggle to determine tool use in general datasets, and LLMs' performance in tool-usage datasets improves when their ability is similar to ChatGPT. In both datasets, incorrect tool usage significantly impairs LLMs' performance. To mitigate this, we also develop the finetuning dataset to enhance tool decision-making. Fine-tuning Llama2-7B results in a 14\% average performance improvement and a 16.8\% decrease in incorrect tool usage. We will release the WTU-Eval benchmark.

Title: Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Models

• Authors: Xavier Suau, Pieter Delobelle, Katherine Metcalf, Armand Joulin, Nicholas Apostoloff, Luca Zappella, Pau Rodríguez
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12824
• Pdf URL: https://arxiv.org/pdf/2407.12824
• Copy Paste: [[2407.12824]] Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Models(https://arxiv.org/abs/2407.12824)
• Keywords: large language model
• Abstract: An important issue with Large Language Models (LLMs) is their undesired ability to generate toxic language. In this work, we show that the neurons responsible for toxicity can be determined by their power to discriminate toxic sentences, and that toxic language can be mitigated by reducing their activation levels proportionally to this power. We propose AUROC adaptation (AurA), an intervention that can be applied to any pre-trained LLM to mitigate toxicity. As the intervention is proportional to the ability of each neuron to discriminate toxic content, it is free of any model-dependent hyperparameters. We show that AurA can achieve up to $2.2 \times$ reduction in toxicity with only a $0.72$ perplexity increase. We also show that AurA is effective with models of different scale (from 1.5B to 40B parameters), and its effectiveness in mitigating toxic language, while preserving common-sense zero-shot abilities, holds across all scales. AurA can be combined with pre-prompting strategies, boosting its average mitigation potential from $1.28\times$ to $2.35\times$. Moreover, AurA can counteract adversarial pre-prompts that maliciously elicit toxic content, making it an effective method for deploying safer and less toxic models.

Title: A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention

• Authors: Shengjie Li, Yinhao Xiao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12825
• Pdf URL: https://arxiv.org/pdf/2407.12825
• Copy Paste: [[2407.12825]] A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention(https://arxiv.org/abs/2407.12825)
• Keywords: transformer
• Abstract: Depression, a prevalent and serious mental health issue, affects approximately 3.8\% of the global population. Despite the existence of effective treatments, over 75\% of individuals in low- and middle-income countries remain untreated, partly due to the challenge in accurately diagnosing depression in its early stages. This paper introduces a novel method for detecting depression based on multi-modal feature fusion utilizing cross-attention. By employing MacBERT as a pre-training model to extract lexical features from text and incorporating an additional Transformer module to refine task-specific contextual understanding, the model's adaptability to the targeted task is enhanced. Diverging from previous practices of simply concatenating multimodal features, this approach leverages cross-attention for feature integration, significantly improving the accuracy in depression detection and enabling a more comprehensive and precise analysis of user emotions and behaviors. Furthermore, a Multi-Modal Feature Fusion Network based on Cross-Attention (MFFNC) is constructed, demonstrating exceptional performance in the task of depression identification. The experimental results indicate that our method achieves an accuracy of 0.9495 on the test dataset, marking a substantial improvement over existing approaches. Moreover, it outlines a promising methodology for other social media platforms and tasks involving multi-modal processing. Timely identification and intervention for individuals with depression are crucial for saving lives, highlighting the immense potential of technology in facilitating early intervention for mental health issues.

Title: Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights

• Authors: Elphin Tom Joe, Sai Dileep Koneru, Christine J Kirchhoff
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12826
• Pdf URL: https://arxiv.org/pdf/2407.12826
• Copy Paste: [[2407.12826]] Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights(https://arxiv.org/abs/2407.12826)
• Keywords: extraction, large language model
• Abstract: In this research short, we examine the potential of using GPT-4o, a state-of-the-art large language model (LLM) to undertake evidence synthesis and systematic assessment tasks. Traditional workflows for such tasks involve large groups of domain experts who manually review and synthesize vast amounts of literature. The exponential growth of scientific literature and recent advances in LLMs provide an opportunity to complementing these traditional workflows with new age tools. We assess the efficacy of GPT-4o to do these tasks on a sample from the dataset created by the Global Adaptation Mapping Initiative (GAMI) where we check the accuracy of climate change adaptation related feature extraction from the scientific literature across three levels of expertise. Our results indicate that while GPT-4o can achieve high accuracy in low-expertise tasks like geographic location identification, their performance in intermediate and high-expertise tasks, such as stakeholder identification and assessment of depth of the adaptation response, is less reliable. The findings motivate the need for designing assessment workflows that utilize the strengths of models like GPT-4o while also providing refinements to improve their performance on these tasks.

Title: Knowledge-based Consistency Testing of Large Language Models

• Authors: Sai Sathiesh Rajan, Ezekiel Soremekun, Sudipta Chattopadhyay
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12830
• Pdf URL: https://arxiv.org/pdf/2407.12830
• Copy Paste: [[2407.12830]] Knowledge-based Consistency Testing of Large Language Models(https://arxiv.org/abs/2407.12830)
• Keywords: large language model
• Abstract: In this work, we systematically expose and measure the inconsistency and knowledge gaps of Large Language Models (LLMs). Specifically, we propose an automated testing framework (called KONTEST) which leverages a knowledge graph to construct test cases. KONTEST probes and measures the inconsistencies in the LLM's knowledge of the world via a combination of semantically-equivalent queries and test oracles (metamorphic or ontological oracle). KONTEST further mitigates knowledge gaps via a weighted LLM model ensemble. Using four state-of-the-art LLMs (Falcon, Gemini, GPT3.5, and Llama2), we show that KONTEST generates 19.2% error inducing inputs (1917 errors from 9983 test inputs). It also reveals a 16.5% knowledge gap across all tested LLMs. KONTEST's mitigation method reduces LLM knowledge gap by 32.48%. Our ablation study further shows that GPT3.5 is not suitable for knowledge-based consistency testing because it is only 60%-68% effective in knowledge construction.

Title: Truth is Universal: Robust Detection of Lies in LLMs

• Authors: Lennart Bürger, Fred A. Hamprecht, Boaz Nadler
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12831
• Pdf URL: https://arxiv.org/pdf/2407.12831
• Copy Paste: [[2407.12831]] Truth is Universal: Robust Detection of Lies in LLMs(https://arxiv.org/abs/2407.12831)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have revolutionised natural language processing, exhibiting impressive human-like capabilities. In particular, LLMs are capable of "lying", knowingly outputting false statements. Hence, it is of interest and importance to develop methods to detect when LLMs lie. Indeed, several authors trained classifiers to detect LLM lies based on their internal model activations. However, other researchers showed that these classifiers may fail to generalise, for example to negated statements. In this work, we aim to develop a robust method to detect when an LLM is lying. To this end, we make the following key contributions: (i) We demonstrate the existence of a two-dimensional subspace, along which the activation vectors of true and false statements can be separated. Notably, this finding is universal and holds for various LLMs, including Gemma-7B, LLaMA2-13B and LLaMA3-8B. Our analysis explains the generalisation failures observed in previous studies and sets the stage for more robust lie detection; (ii) Building upon (i), we construct an accurate LLM lie detector. Empirically, our proposed classifier achieves state-of-the-art performance, distinguishing simple true and false statements with 94% accuracy and detecting more complex real-world lies with 95% accuracy.

Title: Sentence-level Aggregation of Lexical Metrics Correlate Stronger with Human Judgements than Corpus-level Aggregation

• Authors: Paulo Cavalin, Pedro Henrique Domingues, Claudio Pinhanez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12832
• Pdf URL: https://arxiv.org/pdf/2407.12832
• Copy Paste: [[2407.12832]] Sentence-level Aggregation of Lexical Metrics Correlate Stronger with Human Judgements than Corpus-level Aggregation(https://arxiv.org/abs/2407.12832)
• Keywords: robust
• Abstract: In this paper we show that corpus-level aggregation hinders considerably the capability of lexical metrics to accurately evaluate machine translation (MT) systems. With empirical experiments we demonstrate that averaging individual segment-level scores can make metrics such as BLEU and chrF correlate much stronger with human judgements and make them behave considerably more similar to neural metrics such as COMET and BLEURT. We show that this difference exists because corpus- and segment-level aggregation differs considerably owing to the classical average of ratio versus ratio of averages Mathematical problem. Moreover, as we also show, such difference affects considerably the statistical robustness of corpus-level aggregation. Considering that neural metrics currently only cover a small set of sufficiently-resourced languages, the results in this paper can help make the evaluation of MT systems for low-resource languages more trustworthy.

Title: ESQA: Event Sequences Question Answering

• Authors: Irina Abdullaeva, Andrei Filatov, Mikhail Orlov, Ivan Karpukhin, Viacheslav Vasilev, Denis Dimitrov, Andrey Kuznetsov, Ivan Kireev, Andrey Savchenko
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12833
• Pdf URL: https://arxiv.org/pdf/2407.12833
• Copy Paste: [[2407.12833]] ESQA: Event Sequences Question Answering(https://arxiv.org/abs/2407.12833)
• Keywords: large language model
• Abstract: Event sequences (ESs) arise in many practical domains including finance, retail, social networks, and healthcare. In the context of machine learning, event sequences can be seen as a special type of tabular data with annotated timestamps. Despite the importance of ESs modeling and analysis, little effort was made in adapting large language models (LLMs) to the ESs domain. In this paper, we highlight the common difficulties of ESs processing and propose a novel solution capable of solving multiple downstream tasks with little or no finetuning. In particular, we solve the problem of working with long sequences and improve time and numeric features processing. The resulting method, called ESQA, effectively utilizes the power of LLMs and, according to extensive experiments, achieves state-of-the-art results in the ESs domain.

Title: Regurgitative Training: The Value of Real Data in Training Large Language Models

• Authors: Jinghui Zhang, Dandan Qiao, Mochen Yang, Qiang Wei
• Subjects: cs.CL, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.12835
• Pdf URL: https://arxiv.org/pdf/2407.12835
• Copy Paste: [[2407.12835]] Regurgitative Training: The Value of Real Data in Training Large Language Models(https://arxiv.org/abs/2407.12835)
• Keywords: transformer, large language model
• Abstract: What happens if we train a new Large Language Model (LLM) using data that are at least partially generated by other LLMs? The explosive success of LLMs means that a substantial amount of content online will be generated by LLMs rather than humans, which will inevitably enter the training datasets of next-generation LLMs. We evaluate the implications of such "regurgitative training" on LLM performance. Through fine-tuning GPT-3.5 with data generated either by itself or by other LLMs in a machine translation task, we find strong evidence that regurgitative training clearly handicaps the performance of LLMs. The same performance loss of regurgitative training is observed on transformer models that we train from scratch. We find suggestive evidence that the performance disadvantage of regurgitative training can be attributed to at least two mechanisms: (1) higher error rates and (2) lower lexical diversity in LLM-generated data as compared to real data. Based on these mechanisms, we propose and evaluate three different strategies to mitigate the performance loss of regurgitative training. First, we devise data-driven metrics to gauge the quality of each LLM-generated data instance, and then carry out an ordered training process where high-quality data are added before low-quality ones. Second, we combine data generated by multiple different LLMs (as an attempt to increase lexical diversity). Third, we train an AI detection classifier to differentiate between LLM- and human-generated data, and include LLM-generated data in the order of resemblance to human-generated data. All three strategies can improve the performance of regurgitative training to some extent but are not always able to fully close the gap from training with real data. Our results highlight the value of real, human-generated data in training LLMs, which cannot be easily substituted by synthetic, LLM-generated data.

Title: Historical Ink: 19th Century Latin American Spanish Newspaper Corpus with LLM OCR Correction

• Authors: Laura Manrique-Gómez, Tony Montes, Rubén Manrique
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2407.12838
• Pdf URL: https://arxiv.org/pdf/2407.12838
• Copy Paste: [[2407.12838]] Historical Ink: 19th Century Latin American Spanish Newspaper Corpus with LLM OCR Correction(https://arxiv.org/abs/2407.12838)
• Keywords: large language model
• Abstract: This paper presents two significant contributions: first, a novel dataset of 19th-century Latin American press texts, which addresses the lack of specialized corpora for historical and linguistic analysis in this region. Second, it introduces a framework for OCR error correction and linguistic surface form detection in digitized corpora, utilizing a Large Language Model. This framework is adaptable to various contexts and, in this paper, is specifically applied to the newly created dataset.

Title: What to do if language models disagree? Black-box model ensembling for textual and visual question answering

• Authors: Yuxi Xia, Kilm Zaporojets, Benjamin Roth
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12841
• Pdf URL: https://arxiv.org/pdf/2407.12841
• Copy Paste: [[2407.12841]] What to do if language models disagree? Black-box model ensembling for textual and visual question answering(https://arxiv.org/abs/2407.12841)
• Keywords: large language model
• Abstract: A diverse range of large language models (LLMs), e.g., ChatGPT, and visual question answering (VQA) models, e.g., BLIP, have been developed for solving textual and visual question answering tasks. However, both LLMs and VQA models encounter challenges when applied to task-specific datasets. Fine-tuning these models is either difficult, as it requires access via APIs, rendering them as black-boxes, or costly due to the need of tuning a large number of parameters. To address this, we introduce InfoSel, a data-efficient and lightweight ensemble method that learns to dynamically pick the winner from existing black-box models for predictions on both textual and multimodal visual question answering tasks. Unlike traditional ensemble models, InfoSel does not rely on prediction probabilities or confidences, which typically are not available in black-box models. Experimental results on four datasets demonstrate that our approach achieves an absolute increase of up to +5.27% in the F1-score compared to standalone LLMs. Remarkably, this improvement is achieved by utilizing only 1K training instances and 110M model parameters for training task-specific ensemble models.

Title: MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production

• Authors: Jian Ma, Wenguan Wang, Yi Yang, Feng Zheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12842
• Pdf URL: https://arxiv.org/pdf/2407.12842
• Copy Paste: [[2407.12842]] MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production(https://arxiv.org/abs/2407.12842)
• Keywords: diffusion
• Abstract: Sign language understanding has made significant strides; however, there is still no viable solution for generating sign sequences directly from entire spoken content, e.g., text or speech. In this paper, we propose a unified framework for continuous sign language production, easing communication between sign and non-sign language users. In particular, a sequence diffusion model, utilizing embeddings extracted from text or speech, is crafted to generate sign predictions step by step. Moreover, by creating a joint embedding space for text, audio, and sign, we bind these modalities and leverage the semantic consistency among them to provide informative feedback for the model training. This embedding-consistency learning strategy minimizes the reliance on sign triplets and ensures continuous model refinement, even with a missing audio modality. Experiments on How2Sign and PHOENIX14T datasets demonstrate that our model achieves competitive performance in sign language production.

Title: NutriBench: A Dataset for Evaluating Large Language Models in Carbohydrate Estimation from Meal Descriptions

• Authors: Andong Hua, Mehak Preet Dhaliwal, Ryan Burke, Yao Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12843
• Pdf URL: https://arxiv.org/pdf/2407.12843
• Copy Paste: [[2407.12843]] NutriBench: A Dataset for Evaluating Large Language Models in Carbohydrate Estimation from Meal Descriptions(https://arxiv.org/abs/2407.12843)
• Keywords: large language model
• Abstract: Accurate nutrition estimation helps people make informed decisions about their dietary choices and is crucial for preventing serious health issues. We present NutriBench, the first publicly available natural language meal description based nutrition benchmark. NutriBench consists of 5,000 human-verified meal descriptions with macro-nutrient labels, including carbohydrates, proteins, fats, and calories. The data is divided into 15 subsets varying in complexity based on the number, servings, and popularity of the food items in the meal and the specificity of serving size descriptions. We conducted an extensive evaluation of seven popular and state-of-the-art Large Language Models (LLMs), including GPT-3.5, Llama-3, and a medical domain-specific model with standard, Chain-of-Thought and Retrieval-Augmented Generation strategies on our benchmark for carbohydrate estimation. We also conducted a human study involving expert and non-expert participants and found that LLMs can provide more accurate and faster predictions over a range of complex queries. We present a thorough analysis and comparison of different LLMs, highlighting the opportunities and challenges of using LLMs for nutrition estimation in real-life scenarios. Our benchmark is publicly available at: this https URL

Title: $\texttt{metabench}$ -- A Sparse Benchmark to Measure General Ability in Large Language Models

• Authors: Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, Eric Schulz
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.12844
• Pdf URL: https://arxiv.org/pdf/2407.12844
• Copy Paste: [[2407.12844]] $\texttt{metabench}$ -- A Sparse Benchmark to Measure General Ability in Large Language Models(https://arxiv.org/abs/2407.12844)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) vary in their abilities on a range of tasks. Initiatives such as the $\texttt{Open LLM Leaderboard}$ aim to quantify these differences with several large benchmarks (sets of test items to which an LLM can respond either correctly or incorrectly). However, high correlations within and between benchmark scores suggest that (1) there exists a small set of common underlying abilities that these benchmarks measure, and (2) items tap into redundant information and the benchmarks may thus be considerably compressed. We use data from $n > 5000$ LLMs to identify the most informative items of six benchmarks, ARC, GSM8K, HellaSwag, MMLU, TruthfulQA and WinoGrande (with $d=28,632$ items in total). From them we distill a sparse benchmark, $\texttt{metabench}$, that has less than $3\%$ of the original size of all six benchmarks combined. This new sparse benchmark goes beyond point scores by yielding estimators of the underlying benchmark-specific abilities. We show that these estimators (1) can be used to reconstruct each original $\textit{individual}$ benchmark score with, on average, $1.5\%$ root mean square error (RMSE), (2) reconstruct the original $\textit{total}$ score with $0.8\%$ RMSE, and (3) have a single underlying common factor whose Spearman correlation with the total score is $r = 0.93$.

Title: Identifying the Source of Generation for Large Language Models

• Authors: Bumjin Park, Jaesik Choi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12846
• Pdf URL: https://arxiv.org/pdf/2407.12846
• Copy Paste: [[2407.12846]] Identifying the Source of Generation for Large Language Models(https://arxiv.org/abs/2407.12846)
• Keywords: privacy, large language model
• Abstract: Large language models (LLMs) memorize text from several sources of documents. In pretraining, LLM trains to maximize the likelihood of text but neither receives the source of the text nor memorizes the source. Accordingly, LLM can not provide document information on the generated content, and users do not obtain any hint of reliability, which is crucial for factuality or privacy infringement. This work introduces token-level source identification in the decoding step, which maps the token representation to the reference document. We propose a bi-gram source identifier, a multi-layer perceptron with two successive token representations as input for better generalization. We conduct extensive experiments on Wikipedia and PG19 datasets with several LLMs, layer locations, and identifier sizes. The overall results show a possibility of token-level source identifiers for tracing the document, a crucial problem for the safe use of LLMs.

Title: Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessments

• Authors: Roland Daynauth, Jason Mars
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.12847
• Pdf URL: https://arxiv.org/pdf/2407.12847
• Copy Paste: [[2407.12847]] Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessments(https://arxiv.org/abs/2407.12847)
• Keywords: robust, fair, large language model
• Abstract: The SLAM paper demonstrated that on-device Small Language Models (SLMs) are a viable and cost-effective alternative to API-based Large Language Models (LLMs), such as OpenAI's GPT-4, offering comparable performance and stability. However, SLAM also identified discrepancies between human preferences and traditional auto-evaluators. This follow-up paper explores methods to align LLM evaluator preferences with human evaluations by addressing biases, particularly toward higher token counts. We employed Bayesian statistics and a t-test to quantify this bias and developed a recalibration procedure to adjust the GPTScorer. Our findings significantly improve aligning the recalibrated LLM evaluator with human evaluations across multiple use cases. For instance, spearman's ranking correlation score in the Recommendation use case improved from -27.27 to 44.55. These results highlight the importance of accounting for biases in automated evaluations to ensure fair and accurate model assessments. The recalibration process enhances the reliability of automated evaluators, leading to better AI models that align with human values and expectations. This study provides a robust methodology for future research into bias correction and emphasizes the feasibility and benefits of developing human-aligned AI evaluation systems.

Title: Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization

• Authors: Aniket Deroy, Kripabandhu Ghosh, Saptarshi Ghosh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12848
• Pdf URL: https://arxiv.org/pdf/2407.12848
• Copy Paste: [[2407.12848]] Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization(https://arxiv.org/abs/2407.12848)
• Keywords: generative, large language model
• Abstract: Automatic summarization of legal case judgements, which are known to be long and complex, has traditionally been tried via extractive summarization models. In recent years, generative models including abstractive summarization models and Large language models (LLMs) have gained huge popularity. In this paper, we explore the applicability of such models for legal case judgement summarization. We applied various domain specific abstractive summarization models and general domain LLMs as well as extractive summarization models over two sets of legal case judgements from the United Kingdom (UK) Supreme Court and the Indian (IN) Supreme Court and evaluated the quality of the generated summaries. We also perform experiments on a third dataset of legal documents of a different type, Government reports from the United States (US). Results show that abstractive summarization models and LLMs generally perform better than the extractive methods as per traditional metrics for evaluating summary quality. However, detailed investigation shows the presence of inconsistencies and hallucinations in the outputs of the generative models, and we explore ways to reduce the hallucinations and inconsistencies in the summaries. Overall, the investigation suggests that further improvements are needed to enhance the reliability of abstractive models and LLMs for legal case judgement summarization. At present, a human-in-the-loop technique is more suitable for performing manual checks to identify inconsistencies in the generated summaries.

Title: Limits to Predicting Online Speech Using Large Language Models

• Authors: Mina Remeli, Moritz Hardt, Robert C. Williamson
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12850
• Pdf URL: https://arxiv.org/pdf/2407.12850
• Copy Paste: [[2407.12850]] Limits to Predicting Online Speech Using Large Language Models(https://arxiv.org/abs/2407.12850)
• Keywords: robust, large language model
• Abstract: We study the predictability of online speech on social media, and whether predictability improves with information outside a user's own posts. Recent work suggests that the predictive information contained in posts written by a user's peers can surpass that of the user's own posts. Motivated by the success of large language models, we empirically test this hypothesis. We define unpredictability as a measure of the model's uncertainty, i.e., its negative log-likelihood on future tokens given context. As the basis of our study, we collect a corpus of 6.25M posts from more than five thousand X (previously Twitter) users and their peers. Across three large language models ranging in size from 1 billion to 70 billion parameters, we find that predicting a user's posts from their peers' posts performs poorly. Moreover, the value of the user's own posts for prediction is consistently higher than that of their peers'. Across the board, we find that the predictability of social media posts remains low, comparable to predicting financial news without context. We extend our investigation with a detailed analysis about the causes of unpredictability and the robustness of our findings. Specifically, we observe that a significant amount of predictive uncertainty comes from hashtags and @-mentions. Moreover, our results replicate if instead of prompting the model with additional context, we finetune on additional context.

Title: Automated Justification Production for Claim Veracity in Fact Checking: A Survey on Architectures and Approaches

• Authors: Islam Eldifrawi, Shengrui Wang, Amine Trabelsi
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12853
• Pdf URL: https://arxiv.org/pdf/2407.12853
• Copy Paste: [[2407.12853]] Automated Justification Production for Claim Veracity in Fact Checking: A Survey on Architectures and Approaches(https://arxiv.org/abs/2407.12853)
• Keywords: explainability
• Abstract: Automated Fact-Checking (AFC) is the automated verification of claim accuracy. AFC is crucial in discerning truth from misinformation, especially given the huge amounts of content are generated online daily. Current research focuses on predicting claim veracity through metadata analysis and language scrutiny, with an emphasis on justifying verdicts. This paper surveys recent methodologies, proposing a comprehensive taxonomy and presenting the evolution of research in that landscape. A comparative analysis of methodologies and future directions for improving fact-checking explainability are also discussed.

Title: Large Language Models can impersonate politicians and other public figures

• Authors: Steffen Herbold, Alexander Trautsch, Zlata Kikteva, Annette Hautli-Janisz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12855
• Pdf URL: https://arxiv.org/pdf/2407.12855
• Copy Paste: [[2407.12855]] Large Language Models can impersonate politicians and other public figures(https://arxiv.org/abs/2407.12855)
• Keywords: large language model
• Abstract: Modern AI technology like Large language models (LLMs) has the potential to pollute the public information sphere with made-up content, which poses a significant threat to the cohesion of societies at large. A wide range of research has shown that LLMs are capable of generating text of impressive quality, including persuasive political speech, text with a pre-defined style, and role-specific content. But there is a crucial gap in the literature: We lack large-scale and systematic studies of how capable LLMs are in impersonating political and societal representatives and how the general public judges these impersonations in terms of authenticity, relevance and coherence. We present the results of a study based on a cross-section of British society that shows that LLMs are able to generate responses to debate questions that were part of a broadcast political debate programme in the UK. The impersonated responses are judged to be more authentic and relevant than the original responses given by people who were impersonated. This shows two things: (1) LLMs can be made to contribute meaningfully to the public political debate and (2) there is a dire need to inform the general public of the potential harm this can have on society.

Title: AI AI Bias: Large Language Models Favor Their Own Generated Content

• Authors: Walter Laurito, Benjamin Davis, Peli Grietzer, Tomáš Gavenčiak, Ada Böhm, Jan Kulveit
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12856
• Pdf URL: https://arxiv.org/pdf/2407.12856
• Copy Paste: [[2407.12856]] AI AI Bias: Large Language Models Favor Their Own Generated Content(https://arxiv.org/abs/2407.12856)
• Keywords: fair, large language model
• Abstract: Are large language models (LLMs) biased towards text generated by LLMs over text authored by humans, leading to possible anti-human bias? Utilizing a classical experimental design inspired by employment discrimination studies, we tested widely-used LLMs, including GPT-3.5 and GPT4, in binary-choice scenarios. These involved LLM-based agents selecting between products and academic papers described either by humans or LLMs under identical conditions. Our results show a consistent tendency for LLM-based AIs to prefer LLM-generated content. This suggests the possibility of AI systems implicitly discriminating against humans, giving AI agents an unfair advantage.

Title: Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis

• Authors: Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, Renjing Cui, Chengcheng Han, Qiushi Sun, Zhiyong Wu, Yunshi Lan, Xiang Li
• Subjects: cs.CL, cs.DL, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.12857
• Pdf URL: https://arxiv.org/pdf/2407.12857
• Copy Paste: [[2407.12857]] Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis(https://arxiv.org/abs/2407.12857)
• Keywords: large language model
• Abstract: In recent years, the rapid increase in scientific papers has overwhelmed traditional review mechanisms, resulting in varying quality of publications. Although existing methods have explored the capabilities of Large Language Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three modules: Standardization, Evaluation, and Analysis, which are represented by models SEA-S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate constructive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper contents and reviews. Moreover, we design a self-correction strategy to enhance the consistency. Extensive experimental results on datasets collected from eight venues show that SEA can generate valuable insights for authors to improve their papers.

Title: Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)

• Authors: Krishnaram Kenthapadi, Mehrnoosh Sameki, Ankur Taly
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12858
• Pdf URL: https://arxiv.org/pdf/2407.12858
• Copy Paste: [[2407.12858]] Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)(https://arxiv.org/abs/2407.12858)
• Keywords: security, robust, interpretability, generative, large language model
• Abstract: With the ongoing rapid adoption of Artificial Intelligence (AI)-based systems in high-stakes domains, ensuring the trustworthiness, safety, and observability of these systems has become crucial. It is essential to evaluate and monitor AI systems not only for accuracy and quality-related metrics but also for robustness, bias, security, interpretability, and other responsible AI dimensions. We focus on large language models (LLMs) and other generative AI models, which present additional challenges such as hallucinations, harmful and manipulative content, and copyright infringement. In this survey article accompanying our KDD 2024 tutorial, we highlight a wide range of harms associated with generative AI systems, and survey state of the art approaches (along with open challenges) to address these harms.

Title: STAGE: Simplified Text-Attributed Graph Embeddings Using Pre-trained LLMs

• Authors: Aaron Zolnai-Lucas, Jack Boylan, Chris Hokamp, Parsa Ghaffari
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12860
• Pdf URL: https://arxiv.org/pdf/2407.12860
• Copy Paste: [[2407.12860]] STAGE: Simplified Text-Attributed Graph Embeddings Using Pre-trained LLMs(https://arxiv.org/abs/2407.12860)
• Keywords: robust, diffusion
• Abstract: We present Simplified Text-Attributed Graph Embeddings (STAGE), a straightforward yet effective method for enhancing node features in Graph Neural Network (GNN) models that encode Text-Attributed Graphs (TAGs). Our approach leverages Large-Language Models (LLMs) to generate embeddings for textual attributes. STAGE achieves competitive results on various node classification benchmarks while also maintaining a simplicity in implementation relative to current state-of-the-art (SoTA) techniques. We show that utilizing pre-trained LLMs as embedding generators provides robust features for ensemble GNN training, enabling pipelines that are simpler than current SoTA approaches which require multiple expensive training and prompting stages. We also implement diffusion-pattern GNNs in an effort to make this pipeline scalable to graphs beyond academic benchmarks.

Title: Analyzing Large language models chatbots: An experimental approach using a probability test

• Authors: Melise Peruchini, Julio Monteiro Teixeira
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12862
• Pdf URL: https://arxiv.org/pdf/2407.12862
• Copy Paste: [[2407.12862]] Analyzing Large language models chatbots: An experimental approach using a probability test(https://arxiv.org/abs/2407.12862)
• Keywords: large language model
• Abstract: This study consists of qualitative empirical research, conducted through exploratory tests with two different Large Language Models (LLMs) chatbots: ChatGPT and Gemini. The methodological procedure involved exploratory tests based on prompts designed with a probability question. The "Linda Problem", widely recognized in cognitive psychology, was used as a basis to create the tests, along with the development of a new problem specifically for this experiment, the "Mary Problem". The object of analysis is the dataset with the outputs provided by each chatbot interaction. The purpose of the analysis is to verify whether the chatbots mainly employ logical reasoning that aligns with probability theory or if they are more frequently affected by the stereotypical textual descriptions in the prompts. The findings provide insights about the approach each chatbot employs in handling logic and textual constructions, suggesting that, while the analyzed chatbots perform satisfactorily on a well-known probabilistic problem, they exhibit significantly lower performance on new tests that require direct application of probabilistic logic.

Title: Token-Supervised Value Models for Enhancing Mathematical Reasoning Capabilities of Large Language Models

• Authors: Jung Hyun Lee, June Yong Yang, Byeongho Heo, Dongyoon Han, Kang Min Yoo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12863
• Pdf URL: https://arxiv.org/pdf/2407.12863
• Copy Paste: [[2407.12863]] Token-Supervised Value Models for Enhancing Mathematical Reasoning Capabilities of Large Language Models(https://arxiv.org/abs/2407.12863)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive problem-solving capabilities in mathematics through step-by-step reasoning chains. However, they are susceptible to reasoning errors that impact the quality of subsequent reasoning chains and the final answer due to language models' autoregressive token-by-token generating nature. Recent works have proposed adopting external verifiers to guide the generation of reasoning paths, but existing works utilize models that have been trained with step-by-step labels to assess the correctness of token-by-token reasoning chains. Consequently, they struggle to recognize discriminative details of tokens within a reasoning path and lack the ability to evaluate whether an intermediate reasoning path is on a promising track toward the correct final answer. To amend the lack of sound and token-grained math-verification signals, we devise a novel training scheme for verifiers that apply token-level supervision with the expected cumulative reward (i.e., value). Furthermore, we propose a practical formulation of the cumulative reward by reducing it to finding the probability of future correctness of the final answer and thereby enabling the empirical estimation of the value. Experimental results on mathematical reasoning benchmarks show that Token-Supervised Value Model (TVM) can outperform step-by-step verifiers on GSM8K and MATH with Mistral and Llama.

Title: GRAD-SUM: Leveraging Gradient Summarization for Optimal Prompt Engineering

• Authors: Derek Austin, Elliott Chartock
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12865
• Pdf URL: https://arxiv.org/pdf/2407.12865
• Copy Paste: [[2407.12865]] GRAD-SUM: Leveraging Gradient Summarization for Optimal Prompt Engineering(https://arxiv.org/abs/2407.12865)
• Keywords: large language model
• Abstract: Prompt engineering for large language models (LLMs) is often a manual time-intensive process that involves generating, evaluating, and refining prompts iteratively to ensure high-quality outputs. While there has been work on automating prompt engineering, the solutions generally are either tuned to specific tasks with given answers or are quite costly. We introduce GRAD-SUM, a scalable and flexible method for automatic prompt engineering that builds on gradient-based optimization techniques. Our approach incorporates user-defined task descriptions and evaluation criteria, and features a novel gradient summarization module to generalize feedback effectively. Our results demonstrate that GRAD-SUM consistently outperforms existing methods across various benchmarks, highlighting its versatility and effectiveness in automatic prompt optimization.

Title: Beyond KV Caching: Shared Attention for Efficient LLMs

• Authors: Bingli Liao, Danilo Vasconcellos Vargas
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12866
• Pdf URL: https://arxiv.org/pdf/2407.12866
• Copy Paste: [[2407.12866]] Beyond KV Caching: Shared Attention for Efficient LLMs(https://arxiv.org/abs/2407.12866)
• Keywords: robust, large language model
• Abstract: The efficiency of large language models (LLMs) remains a critical challenge, particularly in contexts where computational resources are limited. Traditional attention mechanisms in these models, while powerful, require significant computational and memory resources due to the necessity of recalculating and storing attention weights across different layers. This paper introduces a novel Shared Attention (SA) mechanism, designed to enhance the efficiency of LLMs by directly sharing computed attention weights across multiple layers. Unlike previous methods that focus on sharing intermediate Key-Value (KV) caches, our approach utilizes the isotropic tendencies of attention distributions observed in advanced LLMs post-pretraining to reduce both the computational flops and the size of the KV cache required during inference. We empirically demonstrate that implementing SA across various LLMs results in minimal accuracy loss on standard benchmarks. Our findings suggest that SA not only conserves computational resources but also maintains robust model performance, thereby facilitating the deployment of more efficient LLMs in resource-constrained environments.

Title: Bilingual Adaptation of Monolingual Foundation Models

• Authors: Gurpreet Gosal, Yishi Xu, Gokul Ramakrishnan, Rituraj Joshi, Avraham Sheinin, Zhiming (Charles)Chen, Biswajit Mishra, Natalia Vassilieva, Joel Hestness, Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Onkar Pandit, Samta Kamboj, Rahul Pal, Parvez Mullah, Soundar Doraiswamy, Mohamed El Karim Chami
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12869
• Pdf URL: https://arxiv.org/pdf/2407.12869
• Copy Paste: [[2407.12869]] Bilingual Adaptation of Monolingual Foundation Models(https://arxiv.org/abs/2407.12869)
• Keywords: large language model
• Abstract: We present an efficient method for adapting a monolingual Large Language Model (LLM) to another language, addressing challenges of catastrophic forgetting and tokenizer limitations. We focus this study on adapting Llama 2 to Arabic. Our two-stage approach begins with expanding the vocabulary and training only the embeddings matrix, followed by full model continual pretraining on a bilingual corpus. By continually pretraining on a mix of Arabic and English corpora, the model retains its proficiency in English while acquiring capabilities in Arabic. Our approach results in significant improvements in Arabic and slight enhancements in English, demonstrating cost-effective cross-lingual transfer. We also perform extensive ablations on embedding initialization techniques, data mix ratios, and learning rates and release a detailed training recipe.

Title: MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation

• Authors: Xiaohan Wang, Dian Li, Yilin Zhao, Sinbadliu, Hui Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12871
• Pdf URL: https://arxiv.org/pdf/2407.12871
• Copy Paste: [[2407.12871]] MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation(https://arxiv.org/abs/2407.12871)
• Keywords: large language model
• Abstract: Utilizing complex tools with Large Language Models (LLMs) is a critical component for grounding AI agents in various real-world scenarios. The core challenge of manipulating tools lies in understanding their usage and functionality. The prevailing approach involves few-shot prompting with demonstrations or fine-tuning on expert trajectories. However, for complex tools and tasks, mere in-context demonstrations may fail to cover sufficient knowledge. Training-based methods are also constrained by the high cost of dataset construction and limited generalizability. In this paper, we introduce a new tool learning methodology (MetaTool) that is generalizable for mastering any reusable toolset. Our approach includes a self-supervised data augmentation technique that enables LLMs to gain a comprehensive understanding of various tools, thereby improving their ability to complete tasks effectively. We develop a series of meta-tasks that involve predicting masked factors of tool execution. These self-supervised tasks enable the automatic generation of high-quality QA data concerning tool comprehension. By incorporating meta-task data into the instruction tuning process, the proposed MetaTool model achieves significant superiority to open-source models and is comparable to GPT-4/GPT-3.5 on multiple tool-oriented tasks.

Title: Evaluating Large Language Models with fmeval

• Authors: Pola Schwöbel, Luca Franceschi, Muhammad Bilal Zafar, Keerthan Vasist, Aman Malhotra, Tomer Shenhar, Pinal Tailor, Pinar Yilmaz, Michael Diamond, Michele Donini
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12872
• Pdf URL: https://arxiv.org/pdf/2407.12872
• Copy Paste: [[2407.12872]] Evaluating Large Language Models with fmeval(https://arxiv.org/abs/2407.12872)
• Keywords: large language model
• Abstract: fmeval is an open source library to evaluate large language models (LLMs) in a range of tasks. It helps practitioners evaluate their model for task performance and along multiple responsible AI dimensions. This paper presents the library and exposes its underlying design principles: simplicity, coverage, extensibility and performance. We then present how these were implemented in the scientific and engineering choices taken when developing fmeval. A case study demonstrates a typical use case for the library: picking a suitable model for a question answering task. We close by discussing limitations and further work in the development of the library. fmeval can be found at this https URL.

Title: Evaluation of RAG Metrics for Question Answering in the Telecom Domain

• Authors: Sujoy Roychowdhury, Sumit Soman, H G Ranjani, Neeraj Gunda, Vansh Chhabra, Sai Krishna Bala
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12873
• Pdf URL: https://arxiv.org/pdf/2407.12873
• Copy Paste: [[2407.12873]] Evaluation of RAG Metrics for Question Answering in the Telecom Domain(https://arxiv.org/abs/2407.12873)
• Keywords: large language model
• Abstract: Retrieval Augmented Generation (RAG) is widely used to enable Large Language Models (LLMs) perform Question Answering (QA) tasks in various domains. However, RAG based on open-source LLM for specialized domains has challenges of evaluating generated responses. A popular framework in the literature is the RAG Assessment (RAGAS), a publicly available library which uses LLMs for evaluation. One disadvantage of RAGAS is the lack of details of derivation of numerical value of the evaluation metrics. One of the outcomes of this work is a modified version of this package for few metrics (faithfulness, context relevance, answer relevance, answer correctness, answer similarity and factual correctness) through which we provide the intermediate outputs of the prompts by using any LLMs. Next, we analyse the expert evaluations of the output of the modified RAGAS package and observe the challenges of using it in the telecom domain. We also study the effect of the metrics under correct vs. wrong retrieval and observe that few of the metrics have higher values for correct retrieval. We also study for differences in metrics between base embeddings and those domain adapted via pre-training and fine-tuning. Finally, we comment on the suitability and challenges of using these metrics for in-the-wild telecom QA task.

Title: SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning

• Authors: Chenyang Zhao, Xueying Jia, Vijay Viswanathan, Tongshuang Wu, Graham Neubig
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12874
• Pdf URL: https://arxiv.org/pdf/2407.12874
• Copy Paste: [[2407.12874]] SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning(https://arxiv.org/abs/2407.12874)
• Keywords: large language model
• Abstract: Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. However, prompting often leads models to make predictions with lower accuracy compared to finetuning a model with ample training data. On the other hand, while finetuning LLMs on task-specific data generally improves their performance, abundant annotated datasets are not available for all tasks. Previous work has explored generating task-specific data from state-of-the-art LLMs and using this data to finetune smaller models, but this approach requires access to a language model other than the one being trained, which introduces cost, scalability challenges, and legal hurdles associated with continuously relying on more powerful LLMs. In response to these, we propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM, then use these input-output pairs to finetune the student LLM itself. In our empirical evaluation of the Natural Instructions V2 benchmark, we find that SELF-GUIDE improves the performance of LLM by a substantial margin. Specifically, we report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics. This sheds light on the promise of self-synthesized data guiding LLMs towards becoming task-specific experts without any external learning signals.

Title: Review-Feedback-Reason (ReFeR): A Novel Framework for NLG Evaluation and Reasoning

• Authors: Yaswanth Narsupalli, Abhranil Chandra, Sreevatsa Muppirala, Manish Gupta, Pawan Goyal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12877
• Pdf URL: https://arxiv.org/pdf/2407.12877
• Copy Paste: [[2407.12877]] Review-Feedback-Reason (ReFeR): A Novel Framework for NLG Evaluation and Reasoning(https://arxiv.org/abs/2407.12877)
• Keywords: large language model
• Abstract: Assessing the quality of Natural Language Generation (NLG) outputs, such as those produced by large language models (LLMs), poses significant challenges. Traditional approaches involve either resource-intensive human evaluations or automatic metrics, which often exhibit a low correlation with human judgment. In this study, we propose Review-Feedback-Reason (ReFeR), a novel evaluation framework for NLG using LLM agents. We rigorously test ReFeR using two pre-existing benchmark datasets on diverse NLG tasks. The proposed framework not only enhances the accuracy of NLG evaluation, surpassing previous benchmarks by $\sim$20\%, but also generates constructive feedback and significantly improves collective reasoning. This feedback is then leveraged for the creation of instruction-tuning datasets, which, when used to fine-tune smaller models like Mistral-7B, makes them extremely good evaluators, yielding a better correlation with human evaluations and performance nearly on par with GPT-3.5. We highlight the effectiveness of our methodology through its application on three reasoning benchmarks, where it outperforms most of the state-of-the-art methods, and also outperforms the reasoning capabilities of models like GPT-3.5 Turbo by $\sim$11.67\% and GPT-4 by $\sim$1\% on an average.

Title: Do LLMs have Consistent Values?

• Authors: Naama Rozen, Gal Elidan, Amir Globerson, Ella Daniel
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12878
• Pdf URL: https://arxiv.org/pdf/2407.12878
• Copy Paste: [[2407.12878]] Do LLMs have Consistent Values?(https://arxiv.org/abs/2407.12878)
• Keywords: large language model
• Abstract: Values are a basic driving force underlying human behavior. Large Language Models (LLM) technology is constantly improving towards human-like dialogue. However, little research has been done to study the values exhibited in text generated by LLMs. Here we study this question by turning to the rich literature on value structure in psychology. We ask whether LLMs exhibit the same value structure that has been demonstrated in humans, including the ranking of values, and correlation between values. We show that the results of this analysis strongly depend on how the LLM is prompted, and that under a particular prompting strategy (referred to as 'Value Anchoring') the agreement with human data is quite compelling. Our results serve both to improve our understanding of values in LLMs, as well as introduce novel methods for assessing consistency in LLM responses.

Title: Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detection

• Authors: Ye Jiang, Yimin Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12879
• Pdf URL: https://arxiv.org/pdf/2407.12879
• Copy Paste: [[2407.12879]] Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detection(https://arxiv.org/abs/2407.12879)
• Keywords: large language model
• Abstract: Large visual-language models (LVLMs) exhibit exceptional performance in visual-language reasoning across diverse cross-modal benchmarks. Despite these advances, recent research indicates that Large Language Models (LLMs), like GPT-3.5-turbo, underachieve compared to well-trained smaller models, such as BERT, in Fake News Detection (FND), prompting inquiries into LVLMs' efficacy in FND tasks. Although performance could improve through fine-tuning LVLMs, the substantial parameters and requisite pre-trained weights render it a resource-heavy endeavor for FND applications. This paper initially assesses the FND capabilities of two notable LVLMs, CogVLM and GPT4V, in comparison to a smaller yet adeptly trained CLIP model in a zero-shot context. The findings demonstrate that LVLMs can attain performance competitive with that of the smaller model. Next, we integrate standard in-context learning (ICL) with LVLMs, noting improvements in FND performance, though limited in scope and consistency. To address this, we introduce the \textbf{I}n-context \textbf{M}ultimodal \textbf{F}ake \textbf{N}ews \textbf{D}etection (IMFND) framework, enriching in-context examples and test inputs with predictions and corresponding probabilities from a well-trained smaller model. This strategic integration directs the LVLMs' focus towards news segments associated with higher probabilities, thereby improving their analytical accuracy. The experimental results suggest that the IMFND framework significantly boosts the FND efficiency of LVLMs, achieving enhanced accuracy over the standard ICL approach across three publicly available FND datasets.

Title: Cross-Modal Augmentation for Few-Shot Multimodal Fake News Detection

• Authors: Ye Jiang, Taihang Wang, Xiaoman Xu, Yimin Wang, Xingyi Song, Diana Maynard
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.12880
• Pdf URL: https://arxiv.org/pdf/2407.12880
• Copy Paste: [[2407.12880]] Cross-Modal Augmentation for Few-Shot Multimodal Fake News Detection(https://arxiv.org/abs/2407.12880)
• Keywords: robust
• Abstract: The nascent topic of fake news requires automatic detection methods to quickly learn from limited annotated samples. Therefore, the capacity to rapidly acquire proficiency in a new task with limited guidance, also known as few-shot learning, is critical for detecting fake news in its early stages. Existing approaches either involve fine-tuning pre-trained language models which come with a large number of parameters, or training a complex neural network from scratch with large-scale annotated datasets. This paper presents a multimodal fake news detection model which augments multimodal features using unimodal features. For this purpose, we introduce Cross-Modal Augmentation (CMA), a simple approach for enhancing few-shot multimodal fake news detection by transforming n-shot classification into a more robust (n $\times$ z)-shot problem, where z represents the number of supplementary features. The proposed CMA achieves SOTA results over three benchmark datasets, utilizing a surprisingly simple linear probing method to classify multimodal fake news with only a few training samples. Furthermore, our method is significantly more lightweight than prior approaches, particularly in terms of the number of trainable parameters and epoch times. The code is available here: \url{this https URL}

Title: InstructAV: Instruction Fine-tuning Large Language Models for Authorship Verification

• Authors: Yujia Hu, Zhiqiang Hu, Chun-Wei Seah, Roy Ka-Wei Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12882
• Pdf URL: https://arxiv.org/pdf/2407.12882
• Copy Paste: [[2407.12882]] InstructAV: Instruction Fine-tuning Large Language Models for Authorship Verification(https://arxiv.org/abs/2407.12882)
• Keywords: explainability, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency in a wide range of NLP tasks. However, when it comes to authorship verification (AV) tasks, which involve determining whether two given texts share the same authorship, even advanced models like ChatGPT exhibit notable limitations. This paper introduces a novel approach, termed InstructAV, for authorship verification. This approach utilizes LLMs in conjunction with a parameter-efficient fine-tuning (PEFT) method to simultaneously improve accuracy and explainability. The distinctiveness of InstructAV lies in its ability to align classification decisions with transparent and understandable explanations, representing a significant progression in the field of authorship verification. Through comprehensive experiments conducted across various datasets, InstructAV demonstrates its state-of-the-art performance on the AV task, offering high classification accuracy coupled with enhanced explanation reliability.

Title: BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

• Authors: Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, Tao Yu
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.12883
• Pdf URL: https://arxiv.org/pdf/2407.12883
• Copy Paste: [[2407.12883]] BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval(https://arxiv.org/abs/2407.12883)
• Keywords: robust, large language model
• Abstract: Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. BRIGHT is constructed from the 1,398 real-world queries collected from diverse domains (such as economics, psychology, robotics, software engineering, earth sciences, etc.), sourced from naturally occurring or carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard [38 ], which achieves a score of 59.0 nDCG@10,2 produces a score of nDCG@10 of 18.0 on BRIGHT. We further demonstrate that augmenting queries with Chain-of-Thought reasoning generated by large language models (LLMs) improves performance by up to 12.2 points. Moreover, BRIGHT is robust against data leakage during pretraining of the benchmarked models as we validate by showing similar performance even when documents from the benchmark are included in the training data. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings. Our code and data are available at this https URL.

Title: Whitening Not Recommended for Classification Tasks in LLMs

• Authors: Ali Forooghi, Shaghayegh Sadeghi, Jianguo Lu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12886
• Pdf URL: https://arxiv.org/pdf/2407.12886
• Copy Paste: [[2407.12886]] Whitening Not Recommended for Classification Tasks in LLMs(https://arxiv.org/abs/2407.12886)
• Keywords: large language model
• Abstract: Sentence embedding is a cornerstone in NLP. Whitening has been claimed to be an effective operation to improve embedding quality obtained from Large Language Models (LLMs). However, we find that the efficacy of whitening is model-dependent and task-dependent. In particular, whitening degenerates embeddings for classification tasks. The conclusion is supported by extensive experiments. We also explored a variety of whitening operations, including PCA, ZCA, PCA-Cor, ZCA-Cor and Cholesky whitenings. A by-product of our research is embedding evaluation platform for LLMs called SentEval+.

Title: Explainable Biomedical Hypothesis Generation via Retrieval Augmented Generation enabled Large Language Models

• Authors: Alexander R. Pelletier, Joseph Ramirez, Irsyad Adam, Simha Sankar, Yu Yan, Ding Wang, Dylan Steinecke, Wei Wang, Peipei Ping
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12888
• Pdf URL: https://arxiv.org/pdf/2407.12888
• Copy Paste: [[2407.12888]] Explainable Biomedical Hypothesis Generation via Retrieval Augmented Generation enabled Large Language Models(https://arxiv.org/abs/2407.12888)
• Keywords: large language model
• Abstract: The vast amount of biomedical information available today presents a significant challenge for investigators seeking to digest, process, and understand these findings effectively. Large Language Models (LLMs) have emerged as powerful tools to navigate this complex and challenging data landscape. However, LLMs may lead to hallucinatory responses, making Retrieval Augmented Generation (RAG) crucial for achieving accurate information. In this protocol, we present RUGGED (Retrieval Under Graph-Guided Explainable disease Distinction), a comprehensive workflow designed to support investigators with knowledge integration and hypothesis generation, identifying validated paths forward. Relevant biomedical information from publications and knowledge bases are reviewed, integrated, and extracted via text-mining association analysis and explainable graph prediction models on disease nodes, forecasting potential links among drugs and diseases. These analyses, along with biomedical texts, are integrated into a framework that facilitates user-directed mechanism elucidation as well as hypothesis exploration through RAG-enabled LLMs. A clinical use-case demonstrates RUGGED's ability to evaluate and recommend therapeutics for Arrhythmogenic Cardiomyopathy (ACM) and Dilated Cardiomyopathy (DCM), analyzing prescribed drugs for molecular interactions and unexplored uses. The platform minimizes LLM hallucinations, offers actionable insights, and improves the investigation of novel therapeutics.

Title: GeoGuide: Geometric guidance of diffusion models

• Authors: Mateusz Poleski, Jacek Tabor, Przemysław Spurek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12889
• Pdf URL: https://arxiv.org/pdf/2407.12889
• Copy Paste: [[2407.12889]] GeoGuide: Geometric guidance of diffusion models(https://arxiv.org/abs/2407.12889)
• Keywords: diffusion
• Abstract: Diffusion models are among the most effective methods for image generation. This is in particular because, unlike GANs, they can be easily conditioned during training to produce elements with desired class or properties. However, guiding a pre-trained diffusion model to generate elements from previously unlabeled data is significantly more challenging. One of the possible solutions was given by the ADM-G guiding approach. Although ADM-G successfully generates elements from the given class, there is a significant quality gap compared to a model originally conditioned on this class. In particular, the FID score obtained by the ADM-G-guided diffusion model is nearly three times lower than the class-conditioned guidance. We demonstrate that this issue is partly due to ADM-G providing minimal guidance during the final stage of the denoising process. To address this problem, we propose GeoGuide, a guidance model based on tracing the distance of the diffusion model's trajectory from the data manifold. The main idea of GeoGuide is to produce normalized adjustments during the backward denoising process. As shown in the experiments, GeoGuide surpasses the probabilistic approach ADM-G with respect to both the FID scores and the quality of the generated images.

Title: Global-Local Similarity for Efficient Fine-Grained Image Recognition with Vision Transformers

• Authors: Edwin Arkel Rios, Min-Chun Hu, Bo-Cheng Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12891
• Pdf URL: https://arxiv.org/pdf/2407.12891
• Copy Paste: [[2407.12891]] Global-Local Similarity for Efficient Fine-Grained Image Recognition with Vision Transformers(https://arxiv.org/abs/2407.12891)
• Keywords: robust, extraction, transformer
• Abstract: Fine-grained recognition involves the classification of images from subordinate macro-categories, and it is challenging due to small inter-class differences. To overcome this, most methods perform discriminative feature selection enabled by a feature extraction backbone followed by a high-level feature refinement step. Recently, many studies have shown the potential behind vision transformers as a backbone for fine-grained recognition, but their usage of its attention mechanism to select discriminative tokens can be computationally expensive. In this work, we propose a novel and computationally inexpensive metric to identify discriminative regions in an image. We compare the similarity between the global representation of an image given by the CLS token, a learnable token used by transformers for classification, and the local representation of individual patches. We select the regions with the highest similarity to obtain crops, which are forwarded through the same transformer encoder. Finally, high-level features of the original and cropped representations are further refined together in order to make more robust predictions. Through extensive experimental evaluation we demonstrate the effectiveness of our proposed method, obtaining favorable results in terms of accuracy across a variety of datasets. Furthermore, our method achieves these results at a much lower computational cost compared to the alternatives. Code and checkpoints are available at: \url{this https URL}.

Title: Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference

• Authors: Ghadeer Jaradat, Mohammed Tolba, Ghada Alsuhli, Hani Saleh, Mahmoud Al-Qutayri, Thanos Stouraitis, Baker Mohammad
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12893
• Pdf URL: https://arxiv.org/pdf/2407.12893
• Copy Paste: [[2407.12893]] Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference(https://arxiv.org/abs/2407.12893)
• Keywords: transformer
• Abstract: In the world of deep learning, Transformer models have become very significant, leading to improvements in many areas from understanding language to recognizing images, covering a wide range of applications. Despite their success, the deployment of these models in real-time applications, particularly on edge devices, poses significant challenges due to their quadratic computational intensity and memory demands. To overcome these challenges we introduce a novel Hybrid Dynamic Pruning (HDP), an efficient algorithm-architecture co-design approach that accelerates transformers using head sparsity, block sparsity and approximation opportunities to reduce computations in attention and reduce memory access. With the observation of the huge redundancy in attention scores and attention heads, we propose a novel integer-based row-balanced block pruning to prune unimportant blocks in the attention matrix at run time, also propose integer-based head pruning to detect and prune unimportant heads at an early stage at run time. Also we propose an approximation method that reduces attention computations. To efficiently support these methods with lower latency and power efficiency, we propose a HDP co-processor architecture.

Title: DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion

• Authors: Huiguo He, Huan Yang, Zixi Tuo, Yuan Zhou, Qiuyue Wang, Yuhang Zhang, Zeyu Liu, Wenhao Huang, Hongyang Chao, Jian Yin
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.12899
• Pdf URL: https://arxiv.org/pdf/2407.12899
• Copy Paste: [[2407.12899]] DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion(https://arxiv.org/abs/2407.12899)
• Keywords: diffusion
• Abstract: Story visualization aims to create visually compelling images or videos corresponding to textual narratives. Despite recent advances in diffusion models yielding promising results, existing methods still struggle to create a coherent sequence of subject-consistent frames based solely on a story. To this end, we propose DreamStory, an automatic open-domain story visualization framework by leveraging the LLMs and a novel multi-subject consistent diffusion model. DreamStory consists of (1) an LLM acting as a story director and (2) an innovative Multi-Subject consistent Diffusion model (MSD) for generating consistent multi-subject across the images. First, DreamStory employs the LLM to generate descriptive prompts for subjects and scenes aligned with the story, annotating each scene's subjects for subsequent subject-consistent generation. Second, DreamStory utilizes these detailed subject descriptions to create portraits of the subjects, with these portraits and their corresponding textual information serving as multimodal anchors (guidance). Finally, the MSD uses these multimodal anchors to generate story scenes with consistent multi-subject. Specifically, the MSD includes Masked Mutual Self-Attention (MMSA) and Masked Mutual Cross-Attention (MMCA) modules. MMSA and MMCA modules ensure appearance and semantic consistency with reference images and text, respectively. Both modules employ masking mechanisms to prevent subject blending. To validate our approach and promote progress in story visualization, we established a benchmark, DS-500, which can assess the overall performance of the story visualization framework, subject-identification accuracy, and the consistency of the generation model. Extensive experiments validate the effectiveness of DreamStory in both subjective and objective evaluations. Please visit our project homepage at this https URL.

Title: Text- and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild

• Authors: Nicolas Richet, Soufiane Belharbi, Haseeb Aslam, Meike Emilie Schadt, Manuela González-González, Gustave Cortal, Alessandro Lameiras Koerich, Marco Pedersoli, Alain Finkel, Simon Bacon, Eric Granger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12927
• Pdf URL: https://arxiv.org/pdf/2407.12927
• Copy Paste: [[2407.12927]] Text- and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild(https://arxiv.org/abs/2407.12927)
• Keywords: large language model
• Abstract: Systems for multimodal Emotion Recognition (ER) commonly rely on features extracted from different modalities (e.g., visual, audio, and textual) to predict the seven basic emotions. However, compound emotions often occur in real-world scenarios and are more difficult to predict. Compound multimodal ER becomes more challenging in videos due to the added uncertainty of diverse modalities. In addition, standard features-based models may not fully capture the complex and subtle cues needed to understand compound emotions. %%%% Since relevant cues can be extracted in the form of text, we advocate for textualizing all modalities, such as visual and audio, to harness the capacity of large language models (LLMs). These models may understand the complex interaction between modalities and the subtleties of complex emotions. Although training an LLM requires large-scale datasets, a recent surge of pre-trained LLMs, such as BERT and LLaMA, can be easily fine-tuned for downstream tasks like compound ER. This paper compares two multimodal modeling approaches for compound ER in videos -- standard feature-based vs. text-based. Experiments were conducted on the challenging C-EXPR-DB dataset for compound ER, and contrasted with results on the MELD dataset for basic ER. Our code is available

Title: GenRC: Generative 3D Room Completion from Sparse Image Collections

• Authors: Ming-Feng Li, Yueh-Feng Ku, Hong-Xuan Yen, Chi Liu, Yu-Lun Liu, Albert Y. C. Chen, Cheng-Hao Kuo, Min Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12939
• Pdf URL: https://arxiv.org/pdf/2407.12939
• Copy Paste: [[2407.12939]] GenRC: Generative 3D Room Completion from Sparse Image Collections(https://arxiv.org/abs/2407.12939)
• Keywords: diffusion, generative
• Abstract: Sparse RGBD scene completion is a challenging task especially when considering consistent textures and geometries throughout the entire scene. Different from existing solutions that rely on human-designed text prompts or predefined camera trajectories, we propose GenRC, an automated training-free pipeline to complete a room-scale 3D mesh with high-fidelity textures. To achieve this, we first project the sparse RGBD images to a highly incomplete 3D mesh. Instead of iteratively generating novel views to fill in the void, we utilized our proposed E-Diffusion to generate a view-consistent panoramic RGBD image which ensures global geometry and appearance consistency. Furthermore, we maintain the input-output scene stylistic consistency through textual inversion to replace human-designed text prompts. To bridge the domain gap among datasets, E-Diffusion leverages models trained on large-scale datasets to generate diverse appearances. GenRC outperforms state-of-the-art methods under most appearance and geometric metrics on ScanNet and ARKitScenes datasets, even though GenRC is not trained on these datasets nor using predefined camera trajectories. Project page: \href{this https URL}{this https URL}

Title: Halu-J: Critique-Based Hallucination Judge

• Authors: Binjie Wang, Steffi Chern, Ethan Chern, Pengfei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12943
• Pdf URL: https://arxiv.org/pdf/2407.12943
• Copy Paste: [[2407.12943]] Halu-J: Critique-Based Hallucination Judge(https://arxiv.org/abs/2407.12943)
• Keywords: large language model
• Abstract: Large language models (LLMs) frequently generate non-factual content, known as hallucinations. Existing retrieval-augmented-based hallucination detection approaches typically address this by framing it as a classification task, evaluating hallucinations based on their consistency with retrieved evidence. However, this approach usually lacks detailed explanations for these evaluations and does not assess the reliability of these explanations. Furthermore, deficiencies in retrieval systems can lead to irrelevant or partially relevant evidence retrieval, impairing the detection process. Moreover, while real-world hallucination detection requires analyzing multiple pieces of evidence, current systems usually treat all evidence uniformly without considering its relevance to the content. To address these challenges, we introduce Halu-J, a critique-based hallucination judge with 7 billion parameters. Halu-J enhances hallucination detection by selecting pertinent evidence and providing detailed critiques. Our experiments indicate that Halu-J outperforms GPT-4o in multiple-evidence hallucination detection and matches its capability in critique generation and evidence selection. We also introduce ME-FEVER, a new dataset designed for multiple-evidence hallucination detection. Our code and dataset can be found in this https URL .

Title: AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer

• Authors: Zhuguanyu Wu, Jiaxin Chen, Hanwen Zhong, Di Huang, Yunhong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12951
• Pdf URL: https://arxiv.org/pdf/2407.12951
• Copy Paste: [[2407.12951]] AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer(https://arxiv.org/abs/2407.12951)
• Keywords: transformer, segmentation
• Abstract: Vision Transformer (ViT) has become one of the most prevailing fundamental backbone networks in the computer vision community. Despite the high accuracy, deploying it in real applications raises critical challenges including the high computational cost and inference latency. Recently, the post-training quantization (PTQ) technique has emerged as a promising way to enhance ViT's efficiency. Nevertheless, existing PTQ approaches for ViT suffer from the inflexible quantization on the post-Softmax and post-GELU activations that obey the power-law-like distributions. To address these issues, we propose a novel non-uniform quantizer, dubbed the Adaptive Logarithm AdaLog (AdaLog) quantizer. It optimizes the logarithmic base to accommodate the power-law-like distribution of activations, while simultaneously allowing for hardware-friendly quantization and de-quantization. By employing the bias reparameterization, the AdaLog quantizer is applicable to both the post-Softmax and post-GELU activations. Moreover, we develop an efficient Fast Progressive Combining Search (FPCS) strategy to determine the optimal logarithm base for AdaLog, as well as the scaling factors and zero points for the uniform quantizers. Extensive experimental results on public benchmarks demonstrate the effectiveness of our approach for various ViT-based architectures and vision tasks including classification, object detection, and instance segmentation. Code is available at this https URL.

Title: Denoising Diffusions in Latent Space for Medical Image Segmentation

• Authors: Fahim Ahmed Zaman, Mathews Jacob, Amanda Chang, Kan Liu, Milan Sonka, Xiaodong Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12952
• Pdf URL: https://arxiv.org/pdf/2407.12952
• Copy Paste: [[2407.12952]] Denoising Diffusions in Latent Space for Medical Image Segmentation(https://arxiv.org/abs/2407.12952)
• Keywords: robust, diffusion, generative, segmentation
• Abstract: Diffusion models (DPMs) have demonstrated remarkable performance in image generation, often times outperforming other generative models. Since their introduction, the powerful noise-to-image denoising pipeline has been extended to various discriminative tasks, including image segmentation. In case of medical imaging, often times the images are large 3D scans, where segmenting one image using DPMs become extremely inefficient due to large memory consumption and time consuming iterative sampling process. In this work, we propose a novel conditional generative modeling framework (LDSeg) that performs diffusion in latent space for medical image segmentation. Our proposed framework leverages the learned inherent low-dimensional latent distribution of the target object shapes and source image embeddings. The conditional diffusion in latent space not only ensures accurate n-D image segmentation for multi-label objects, but also mitigates the major underlying problems of the traditional DPM based segmentation: (1) large memory consumption, (2) time consuming sampling process and (3) unnatural noise injection in forward/reverse process. LDSeg achieved state-of-the-art segmentation accuracy on three medical image datasets with different imaging modalities. Furthermore, we show that our proposed model is significantly more robust to noises, compared to the traditional deterministic segmentation models, which can be potential in solving the domain shift problems in the medical imaging domain. Codes are available at: this https URL.

Title: Temporal Label Hierachical Network for Compound Emotion Recognition

• Authors: Sunan Li, Hailun Lian, Cheng Lu, Yan Zhao, Tianhua Qi, Hao Yang, Yuan Zong, Wenming Zheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12973
• Pdf URL: https://arxiv.org/pdf/2407.12973
• Copy Paste: [[2407.12973]] Temporal Label Hierachical Network for Compound Emotion Recognition(https://arxiv.org/abs/2407.12973)
• Keywords: transformer
• Abstract: The emotion recognition has attracted more attention in recent decades. Although significant progress has been made in the recognition technology of the seven basic emotions, existing methods are still hard to tackle compound emotion recognition that occurred commonly in practical application. This article introduces our achievements in the 7th Field Emotion Behavior Analysis (ABAW) competition. In the competition, we selected pre trained ResNet18 and Transformer, which have been widely validated, as the basic network framework. Considering the continuity of emotions over time, we propose a time pyramid structure network for frame level emotion prediction. Furthermore. At the same time, in order to address the lack of data in composite emotion recognition, we utilized fine-grained labels from the DFEW database to construct training data for emotion categories in competitions. Taking into account the characteristics of valence arousal of various complex emotions, we constructed a classification framework from coarse to fine in the label space.

Title: Leveraging Environment Interaction for Automated PDDL Generation and Planning with Large Language Models

• Authors: Sadegh Mahdavi, Raquel Aoki, Keyi Tang, Yanshuai Cao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12979
• Pdf URL: https://arxiv.org/pdf/2407.12979
• Copy Paste: [[2407.12979]] Leveraging Environment Interaction for Automated PDDL Generation and Planning with Large Language Models(https://arxiv.org/abs/2407.12979)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown remarkable performance in various natural language tasks, but they often struggle with planning problems that require structured reasoning. To address this limitation, the conversion of planning problems into the Planning Domain Definition Language (PDDL) has been proposed as a potential solution, enabling the use of automated planners. However, generating accurate PDDL files typically demands human inputs or correction, which can be time-consuming and costly. In this paper, we propose a novel approach that leverages LLMs and environment feedback to automatically generate PDDL domain and problem description files without the need for human intervention. Our method introduces an iterative refinement process that generates multiple problem PDDL candidates and progressively refines the domain PDDL based on feedback obtained from interacting with the environment. To guide the refinement process, we develop an Exploration Walk (EW) metric, which provides rich feedback signals for LLMs to update the PDDL file. We evaluate our approach on PDDL environments. We achieve an average task solve rate of 66% compared to a 29% solve rate by GPT-4's intrinsic planning with chain-of-thought prompting. Our work enables the automated modeling of planning environments using LLMs and environment feedback, eliminating the need for human intervention in the PDDL generation process and paving the way for more reliable LLM agents in challenging problems.

Title: A Framework for testing Federated Learning algorithms using an edge-like environment

• Authors: Felipe Machado Schwanck, Marcos Tomazzoli Leipnitz, Joel Luís Carbonera, Juliano Araujo Wickboldt
• Subjects: cs.LG, cs.AI, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2407.12980
• Pdf URL: https://arxiv.org/pdf/2407.12980
• Copy Paste: [[2407.12980]] A Framework for testing Federated Learning algorithms using an edge-like environment(https://arxiv.org/abs/2407.12980)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) is a machine learning paradigm in which many clients cooperatively train a single centralized model while keeping their data private and decentralized. FL is commonly used in edge computing, which involves placing computer workloads (both hardware and software) as close as possible to the edge, where the data is being created and where actions are occurring, enabling faster response times, greater data privacy, and reduced data transfer costs. However, due to the heterogeneous data distributions/contents of clients, it is non-trivial to accurately evaluate the contributions of local models in global centralized model aggregation. This is an example of a major challenge in FL, commonly known as data imbalance or class imbalance. In general, testing and assessing FL algorithms can be a very difficult and complex task due to the distributed nature of the systems. In this work, a framework is proposed and implemented to assess FL algorithms in a more easy and scalable way. This framework is evaluated over a distributed edge-like environment managed by a container orchestration platform (i.e. Kubernetes).

Title: Retrieval-Enhanced Machine Learning: Synthesis and Opportunities

• Authors: To Eun Kim, Alireza Salemi, Andrew Drozdov, Fernando Diaz, Hamed Zamani
• Subjects: cs.LG, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.12982
• Pdf URL: https://arxiv.org/pdf/2407.12982
• Copy Paste: [[2407.12982]] Retrieval-Enhanced Machine Learning: Synthesis and Opportunities(https://arxiv.org/abs/2407.12982)
• Keywords: interpretability
• Abstract: In the field of language modeling, models augmented with retrieval components have emerged as a promising solution to address several challenges faced in the natural language processing (NLP) field, including knowledge grounding, interpretability, and scalability. Despite the primary focus on NLP, we posit that the paradigm of retrieval-enhancement can be extended to a broader spectrum of machine learning (ML) such as computer vision, time series prediction, and computational biology. Therefore, this work introduces a formal framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by synthesizing the literature in various domains in ML with consistent notations which is missing from the current literature. Also, we found that while a number of studies employ retrieval components to augment their models, there is a lack of integration with foundational Information Retrieval (IR) research. We bridge this gap between the seminal IR research and contemporary REML studies by investigating each component that comprises the REML framework. Ultimately, the goal of this work is to equip researchers across various disciplines with a comprehensive, formally structured framework of retrieval-enhanced models, thereby fostering interdisciplinary future research.

Title: A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks

• Authors: Shubham Vatsal, Harsh Dubey
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12994
• Pdf URL: https://arxiv.org/pdf/2407.12994
• Copy Paste: [[2407.12994]] A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks(https://arxiv.org/abs/2407.12994)
• Keywords: extraction, large language model
• Abstract: Large language models (LLMs) have shown remarkable performance on many different Natural Language Processing (NLP) tasks. Prompt engineering plays a key role in adding more to the already existing abilities of LLMs to achieve significant performance gains on various NLP tasks. Prompt engineering requires composing natural language instructions called prompts to elicit knowledge from LLMs in a structured way. Unlike previous state-of-the-art (SoTA) models, prompt engineering does not require extensive parameter re-training or fine-tuning based on the given NLP task and thus solely operates on the embedded knowledge of LLMs. Additionally, LLM enthusiasts can intelligently extract LLMs' knowledge through a basic natural language conversational exchange or prompt engineering, allowing more and more people even without deep mathematical machine learning background to experiment with LLMs. With prompt engineering gaining popularity in the last two years, researchers have come up with numerous engineering techniques around designing prompts to improve accuracy of information extraction from the LLMs. In this paper, we summarize different prompting techniques and club them together based on different NLP tasks that they have been used for. We further granularly highlight the performance of these prompting strategies on various datasets belonging to that NLP task, talk about the corresponding LLMs used, present a taxonomy diagram and discuss the possible SoTA for specific datasets. In total, we read and present a survey of 44 research papers which talk about 39 different prompting methods on 29 different NLP tasks of which most of them have been published in the last two years.

Title: A Resolution Independent Neural Operator

• Authors: Bahador Bahmani, Somdatta Goswami, Ioannis G. Kevrekidis, Michael D. Shields
• Subjects: cs.LG, cs.CE, physics.comp-ph, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.13010
• Pdf URL: https://arxiv.org/pdf/2407.13010
• Copy Paste: [[2407.13010]] A Resolution Independent Neural Operator(https://arxiv.org/abs/2407.13010)
• Keywords: robust
• Abstract: The Deep operator network (DeepONet) is a powerful yet simple neural operator architecture that utilizes two deep neural networks to learn mappings between infinite-dimensional function spaces. This architecture is highly flexible, allowing the evaluation of the solution field at any location within the desired domain. However, it imposes a strict constraint on the input space, requiring all input functions to be discretized at the same locations; this limits its practical applications. In this work, we introduce a Resolution Independent Neural Operator (RINO) that provides a framework to make DeepONet resolution-independent, enabling it to handle input functions that are arbitrarily, but sufficiently finely, discretized. To this end, we propose a dictionary learning algorithm to adaptively learn a set of appropriate continuous basis functions, parameterized as implicit neural representations (INRs), from the input data. These basis functions are then used to project arbitrary input function data as a point cloud onto an embedding space (i.e., a vector space of finite dimensions) with dimensionality equal to the dictionary size, which can be directly used by DeepONet without any architectural changes. In particular, we utilize sinusoidal representation networks (SIRENs) as our trainable INR basis functions. We demonstrate the robustness and applicability of RINO in handling arbitrarily (but sufficiently richly) sampled input functions during both training and inference through several numerical examples.

Title: High-Quality Tabular Data Generation using Post-Selected VAE

• Authors: Volodymyr Shulakov
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.13016
• Pdf URL: https://arxiv.org/pdf/2407.13016
• Copy Paste: [[2407.13016]] High-Quality Tabular Data Generation using Post-Selected VAE(https://arxiv.org/abs/2407.13016)
• Keywords: privacy
• Abstract: Synthetic tabular data is becoming a necessity as concerns about data privacy intensify in the world. Tabular data can be useful for testing various systems, simulating real data, analyzing the data itself or building predictive models. Unfortunately, such data may not be available due to confidentiality issues. Previous techniques, such as TVAE (Xu et al., 2019) or OCTGAN (Kim et al., 2021), are either unable to handle particularly complex datasets, or are complex in themselves, resulting in inferior run time performance. This paper introduces PSVAE, a new simple model that is capable of producing high-quality synthetic data in less run time. PSVAE incorporates two key ideas: loss optimization and post-selection. Along with these ideas, the proposed model compensates for underrepresented categories and uses a modern activation function, Mish (Misra, 2019).

Title: Enhancing Gene Expression Prediction from Histology Images with Spatial Transcriptomics Completion

• Authors: Gabriel Mejia, Daniela Ruiz, Paula Cárdenas, Leonardo Manrique, Daniela Vega, Pablo Arbeláez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13027
• Pdf URL: https://arxiv.org/pdf/2407.13027
• Copy Paste: [[2407.13027]] Enhancing Gene Expression Prediction from Histology Images with Spatial Transcriptomics Completion(https://arxiv.org/abs/2407.13027)
• Keywords: fair, transformer
• Abstract: Spatial Transcriptomics is a novel technology that aligns histology images with spatially resolved gene expression profiles. Although groundbreaking, it struggles with gene capture yielding high corruption in acquired data. Given potential applications, recent efforts have focused on predicting transcriptomic profiles solely from histology images. However, differences in databases, preprocessing techniques, and training hyperparameters hinder a fair comparison between methods. To address these challenges, we present a systematically curated and processed database collected from 26 public sources, representing an 8.6-fold increase compared to previous works. Additionally, we propose a state-of-the-art transformer based completion technique for inferring missing gene expression, which significantly boosts the performance of transcriptomic profile predictions across all datasets. Altogether, our contributions constitute the most comprehensive benchmark of gene expression prediction from histology images to date and a stepping stone for future research on spatial transcriptomics.

Title: ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders

• Authors: Carlos Hinojosa, Shuming Liu, Bernard Ghanem
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.13036
• Pdf URL: https://arxiv.org/pdf/2407.13036
• Copy Paste: [[2407.13036]] ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders(https://arxiv.org/abs/2407.13036)
• Keywords: robust, segmentation
• Abstract: Masked AutoEncoders (MAE) have emerged as a robust self-supervised framework, offering remarkable performance across a wide range of downstream tasks. To increase the difficulty of the pretext task and learn richer visual representations, existing works have focused on replacing standard random masking with more sophisticated strategies, such as adversarial-guided and teacher-guided masking. However, these strategies depend on the input data thus commonly increasing the model complexity and requiring additional calculations to generate the mask patterns. This raises the question: Can we enhance MAE performance beyond random masking without relying on input data or incurring additional computational costs? In this work, we introduce a simple yet effective data-independent method, termed ColorMAE, which generates different binary mask patterns by filtering random noise. Drawing inspiration from color noise in image processing, we explore four types of filters to yield mask patterns with different spatial and semantic priors. ColorMAE requires no additional learnable parameters or computational overhead in the network, yet it significantly enhances the learned representations. We provide a comprehensive empirical evaluation, demonstrating our strategy's superiority in downstream tasks compared to random masking. Notably, we report an improvement of 2.72 in mIoU in semantic segmentation tasks relative to baseline MAE implementations.

Title: Universal Facial Encoding of Codec Avatars from VR Headsets

• Authors: Shaojie Bai, Te-Li Wang, Chenghui Li, Akshay Venkatesh, Tomas Simon, Chen Cao, Gabriel Schwartz, Ryan Wrench, Jason Saragih, Yaser Sheikh, Shih-En Wei
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13038
• Pdf URL: https://arxiv.org/pdf/2407.13038
• Copy Paste: [[2407.13038]] Universal Facial Encoding of Codec Avatars from VR Headsets(https://arxiv.org/abs/2407.13038)
• Keywords: robust
• Abstract: Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces. In this paper, we present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a self-supervised learning approach, based on a cross-view reconstruction objective, that enables generalization to unseen users. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency. We present an improved parameterization for precise ground-truth generation that provides robustness to environmental variation. The resulting system produces accurate facial animation for unseen users wearing VR headsets in realtime. We compare our approach to prior face-encoding methods demonstrating significant improvements in both quantitative metrics and qualitative results.

Title: Turkish Delights: a Dataset on Turkish Euphemisms

• Authors: Hasan Can Biyik, Patrick Lee, Anna Feldman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13040
• Pdf URL: https://arxiv.org/pdf/2407.13040
• Copy Paste: [[2407.13040]] Turkish Delights: a Dataset on Turkish Euphemisms(https://arxiv.org/abs/2407.13040)
• Keywords: transformer
• Abstract: Euphemisms are a form of figurative language relatively understudied in natural language processing. This research extends the current computational work on potentially euphemistic terms (PETs) to Turkish. We introduce the Turkish PET dataset, the first available of its kind in the field. By creating a list of euphemisms in Turkish, collecting example contexts, and annotating them, we provide both euphemistic and non-euphemistic examples of PETs in Turkish. We describe the dataset and methodologies, and also experiment with transformer-based models on Turkish euphemism detection by using our dataset for binary classification. We compare performances across models using F1, accuracy, and precision as evaluation metrics.

Title: INTELLECT: Adapting Cyber Threat Detection to Heterogeneous Computing Environments

• Authors: Simone Magnani, Liubov Nedoshivina, Roberto Doriguzzi-Corin, Stefano Braghin, Domenico Siracusa
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13043
• Pdf URL: https://arxiv.org/pdf/2407.13043
• Copy Paste: [[2407.13043]] INTELLECT: Adapting Cyber Threat Detection to Heterogeneous Computing Environments(https://arxiv.org/abs/2407.13043)
• Keywords: secure, attack, federate
• Abstract: The widespread adoption of cloud computing, edge, and IoT has increased the attack surface for cyber threats. This is due to the large-scale deployment of often unsecured, heterogeneous devices with varying hardware and software configurations. The diversity of these devices attracts a wide array of potential attack methods, making it challenging for individual organizations to have comprehensive knowledge of all possible threats. In this context, powerful anomaly detection models can be developed by combining data from different parties using Federated Learning. FL enables the collaborative development of ML-based IDSs without requiring the parties to disclose sensitive training data, such as network traffic or sensor readings. However, deploying the resulting models can be challenging, as they may require more computational resources than those available on target devices with limited capacity or already allocated for other operations. Training device-specific models is not feasible for an organization because a significant portion of the training data is private to other participants in the FL process. To address these challenges, this paper introduces INTELLECT, a novel solution that integrates feature selection, model pruning, and fine-tuning techniques into a cohesive pipeline for the dynamic adaptation of pre-trained ML models and configurations for IDSs. Through empirical evaluation, we analyze the benefits of INTELLECT's approach in tailoring ML models to the specific resource constraints of an organization's devices and measure variations in traffic classification accuracy resulting from feature selection, pruning, and fine-tuning operations. Additionally, we demonstrate the advantages of incorporating knowledge distillation techniques while fine-tuning, enabling the ML model to consistently adapt to local network patterns while preserving historical knowledge.

Title: A Novel GAN Approach to Augment Limited Tabular Data for Short-Term Substance Use Prediction

• Authors: Nguyen Thach, Patrick Habecker, Bergen Johnston, Lillianna Cervantes, Anika Eisenbraun, Alex Mason, Kimberly Tyler, Bilal Khan, Hau Chan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13047
• Pdf URL: https://arxiv.org/pdf/2407.13047
• Copy Paste: [[2407.13047]] A Novel GAN Approach to Augment Limited Tabular Data for Short-Term Substance Use Prediction(https://arxiv.org/abs/2407.13047)
• Keywords: generative
• Abstract: Substance use is a global issue that negatively impacts millions of persons who use drugs (PWUDs). In practice, identifying vulnerable PWUDs for efficient allocation of appropriate resources is challenging due to their complex use patterns (e.g., their tendency to change usage within months) and the high acquisition costs for collecting PWUD-focused substance use data. Thus, there has been a paucity of machine learning models for accurately predicting short-term substance use behaviors of PWUDs. In this paper, using longitudinal survey data of 258 PWUDs in the U.S. Great Plains collected by our team, we design a novel GAN that deals with high-dimensional low-sample-size tabular data and survey skip logic to augment existing data to improve classification models' prediction on (A) whether the PWUDs would increase usage and (B) at which ordinal frequency they would use a particular drug within the next 12 months. Our evaluation results show that, when trained on augmented data from our proposed GAN, the classification models improve their predictive performance (AUROC) by up to 13.4% in Problem (A) and 15.8% in Problem (B) for usage of marijuana, meth, amphetamines, and cocaine, which outperform state-of-the-art generative models.

Title: Cheddar: A Swift Fully Homomorphic Encryption Library for CUDA GPUs

• Authors: Jongmin Kim, Wonseok Choi, Jung Ho Ahn
• Subjects: cs.CR, cs.PF
• Abstract URL: https://arxiv.org/abs/2407.13055
• Pdf URL: https://arxiv.org/pdf/2407.13055
• Copy Paste: [[2407.13055]] Cheddar: A Swift Fully Homomorphic Encryption Library for CUDA GPUs(https://arxiv.org/abs/2407.13055)
• Keywords: security, privacy
• Abstract: Fully homomorphic encryption (FHE) is a cryptographic technology capable of resolving security and privacy problems in cloud computing by encrypting data in use. However, FHE introduces tremendous computational overhead for processing encrypted data, causing FHE workloads to become 2-6 orders of magnitude slower than their unencrypted counterparts. To mitigate the overhead, we propose Cheddar, an FHE library for CUDA GPUs, which demonstrates significantly faster performance compared to prior GPU implementations. We develop optimized functionalities at various implementation levels ranging from efficient low-level primitives to streamlined high-level operational sequences. Especially, we improve major FHE operations, including number-theoretic transform and base conversion, based on efficient kernel designs using a small word size of 32 bits. By these means, Cheddar demonstrates 2.9 to 25.6 times higher performance for representative FHE workloads compared to prior GPU implementations.

Title: Krait: A Backdoor Attack Against Graph Prompt Tuning

• Authors: Ying Song, Rita Singh, Balaji Palanisamy
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13068
• Pdf URL: https://arxiv.org/pdf/2407.13068
• Copy Paste: [[2407.13068]] Krait: A Backdoor Attack Against Graph Prompt Tuning(https://arxiv.org/abs/2407.13068)
• Keywords: defense, attack, steal
• Abstract: Graph prompt tuning has emerged as a promising paradigm to effectively transfer general graph knowledge from pre-trained models to various downstream tasks, particularly in few-shot contexts. However, its susceptibility to backdoor attacks, where adversaries insert triggers to manipulate outcomes, raises a critical concern. We conduct the first study to investigate such vulnerability, revealing that backdoors can disguise benign graph prompts, thus evading detection. We introduce Krait, a novel graph prompt backdoor. Specifically, we propose a simple yet effective model-agnostic metric called label non-uniformity homophily to select poisoned candidates, significantly reducing computational complexity. To accommodate diverse attack scenarios and advanced attack types, we design three customizable trigger generation methods to craft prompts as triggers. We propose a novel centroid similarity-based loss function to optimize prompt tuning for attack effectiveness and stealthiness. Experiments on four real-world graphs demonstrate that Krait can efficiently embed triggers to merely 0.15% to 2% of training nodes, achieving high attack success rates without sacrificing clean accuracy. Notably, in one-to-one and all-to-one attacks, Krait can achieve 100% attack success rates by poisoning as few as 2 and 22 nodes, respectively. Our experiments further show that Krait remains potent across different transfer cases, attack types, and graph neural network backbones. Additionally, Krait can be successfully extended to the black-box setting, posing more severe threats. Finally, we analyze why Krait can evade both classical and state-of-the-art defenses, and provide practical insights for detecting and mitigating this class of attacks.

Title: Dynamic Sentiment Analysis with Local Large Language Models using Majority Voting: A Study on Factors Affecting Restaurant Evaluation

• Authors: Junichiro Niimi
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.13069
• Pdf URL: https://arxiv.org/pdf/2407.13069
• Copy Paste: [[2407.13069]] Dynamic Sentiment Analysis with Local Large Language Models using Majority Voting: A Study on Factors Affecting Restaurant Evaluation(https://arxiv.org/abs/2407.13069)
• Keywords: robust, large language model
• Abstract: User-generated contents (UGCs) on online platforms allow marketing researchers to understand consumer preferences for products and services. With the advance of large language models (LLMs), some studies utilized the models for annotation and sentiment analysis. However, the relationship between the accuracy and the hyper-parameters of LLMs is yet to be thoroughly examined. In addition, the issues of variability and reproducibility of results from each trial of LLMs have rarely been considered in existing literature. Since actual human annotation uses majority voting to resolve disagreements among annotators, this study introduces a majority voting mechanism to a sentiment analysis model using local LLMs. By a series of three analyses of online reviews on restaurant evaluations, we demonstrate that majority voting with multiple attempts using a medium-sized model produces more robust results than using a large model with a single attempt. Furthermore, we conducted further analysis to investigate the effect of each aspect on the overall evaluation.

Title: Enhancing Temporal Action Localization: Advanced S6 Modeling with Recurrent Mechanism

• Authors: Sangyoun Lee, Juho Jung, Changdae Oh, Sunghee Yun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13078
• Pdf URL: https://arxiv.org/pdf/2407.13078
• Copy Paste: [[2407.13078]] Enhancing Temporal Action Localization: Advanced S6 Modeling with Recurrent Mechanism(https://arxiv.org/abs/2407.13078)
• Keywords: transformer
• Abstract: Temporal Action Localization (TAL) is a critical task in video analysis, identifying precise start and end times of actions. Existing methods like CNNs, RNNs, GCNs, and Transformers have limitations in capturing long-range dependencies and temporal causality. To address these challenges, we propose a novel TAL architecture leveraging the Selective State Space Model (S6). Our approach integrates the Feature Aggregated Bi-S6 block, Dual Bi-S6 structure, and a recurrent mechanism to enhance temporal and channel-wise dependency modeling without increasing parameter complexity. Extensive experiments on benchmark datasets demonstrate state-of-the-art results with mAP scores of 74.2% on THUMOS-14, 42.9% on ActivityNet, 29.6% on FineAction, and 45.8% on HACS. Ablation studies validate our method's effectiveness, showing that the Dual structure in the Stem module and the recurrent mechanism outperform traditional approaches. Our findings demonstrate the potential of S6-based models in TAL tasks, paving the way for future research.

Title: Using LLMs to Automate Threat Intelligence Analysis Workflows in Security Operation Centers

• Authors: PeiYu Tseng, ZihDwo Yeh, Xushu Dai, Peng Liu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13093
• Pdf URL: https://arxiv.org/pdf/2407.13093
• Copy Paste: [[2407.13093]] Using LLMs to Automate Threat Intelligence Analysis Workflows in Security Operation Centers(https://arxiv.org/abs/2407.13093)
• Keywords: security
• Abstract: SIEM systems are prevalent and play a critical role in a variety of analyst workflows in Security Operation Centers. However, modern SIEMs face a big challenge: they still cannot relieve analysts from the repetitive tasks involved in analyzing CTI (Cyber Threat Intelligence) reports written in natural languages. This project aims to develop an AI agent to replace the labor intensive repetitive tasks involved in analyzing CTI reports. The agent exploits the revolutionary capabilities of LLMs (e.g., GPT-4), but it does not require any human intervention.

Title: Rethinking Video-Text Understanding: Retrieval from Counterfactually Augmented Data

• Authors: Wufei Ma, Kai Li, Zhongshi Jiang, Moustafa Meshry, Qihao Liu, Huiyu Wang, Christian Häne, Alan Yuille
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13094
• Pdf URL: https://arxiv.org/pdf/2407.13094
• Copy Paste: [[2407.13094]] Rethinking Video-Text Understanding: Retrieval from Counterfactually Augmented Data(https://arxiv.org/abs/2407.13094)
• Keywords: large language model
• Abstract: Recent video-text foundation models have demonstrated strong performance on a wide variety of downstream video understanding tasks. Can these video-text models genuinely understand the contents of natural videos? Standard video-text evaluations could be misleading as many questions can be inferred merely from the objects and contexts in a single frame or biases inherent in the datasets. In this paper, we aim to better assess the capabilities of current video-text models and understand their limitations. We propose a novel evaluation task for video-text understanding, namely retrieval from counterfactually augmented data (RCAD), and a new Feint6K dataset. To succeed on our new evaluation task, models must derive a comprehensive understanding of the video from cross-frame reasoning. Analyses show that previous video-text foundation models can be easily fooled by counterfactually augmented data and are far behind human-level performance. In order to narrow the gap between video-text models and human performance on RCAD, we identify a key limitation of current contrastive approaches on video-text data and introduce LLM-teacher, a more effective approach to learn action semantics by leveraging knowledge obtained from a pretrained large language model. Experiments and analyses show that our approach successfully learn more discriminative action embeddings and improves results on Feint6K when applied to multiple video-text models. Our Feint6K dataset and project page is available at this https URL.

Title: Audio-visual Generalized Zero-shot Learning the Easy Way

• Authors: Shentong Mo, Pedro Morgado
• Subjects: cs.CV, cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.13095
• Pdf URL: https://arxiv.org/pdf/2407.13095
• Copy Paste: [[2407.13095]] Audio-visual Generalized Zero-shot Learning the Easy Way(https://arxiv.org/abs/2407.13095)
• Keywords: transformer
• Abstract: Audio-visual generalized zero-shot learning is a rapidly advancing domain that seeks to understand the intricate relations between audio and visual cues within videos. The overarching goal is to leverage insights from seen classes to identify instances from previously unseen ones. Prior approaches primarily utilized synchronized auto-encoders to reconstruct audio-visual attributes, which were informed by cross-attention transformers and projected text embeddings. However, these methods fell short of effectively capturing the intricate relationship between cross-modal features and class-label embeddings inherent in pre-trained language-aligned embeddings. To circumvent these bottlenecks, we introduce a simple yet effective framework for Easy Audio-Visual Generalized Zero-shot Learning, named EZ-AVGZL, that aligns audio-visual embeddings with transformed text representations. It utilizes a single supervised text audio-visual contrastive loss to learn an alignment between audio-visual and textual modalities, moving away from the conventional approach of reconstructing cross-modal features and text embeddings. Our key insight is that while class name embeddings are well aligned with language-based audio-visual features, they don't provide sufficient class separation to be useful for zero-shot learning. To address this, our method leverages differential optimization to transform class embeddings into a more discriminative space while preserving the semantic structure of language representations. We conduct extensive experiments on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL benchmarks. Our results demonstrate that our EZ-AVGZL achieves state-of-the-art performance in audio-visual generalized zero-shot learning.

Title: Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach

• Authors: Zhouyu Jiang, Mengshu Sun, Lei Liang, Zhiqiang Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13101
• Pdf URL: https://arxiv.org/pdf/2407.13101
• Copy Paste: [[2407.13101]] Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach(https://arxiv.org/abs/2407.13101)
• Keywords: robust, large language model
• Abstract: Multi-hop question answering is a challenging task with distinct industrial relevance, and Retrieval-Augmented Generation (RAG) methods based on large language models (LLMs) have become a popular approach to tackle this task. Owing to the potential inability to retrieve all necessary information in a single iteration, a series of iterative RAG methods has been recently developed, showing significant performance improvements. However, existing methods still face two critical challenges: context overload resulting from multiple rounds of retrieval, and over-planning and repetitive planning due to the lack of a recorded retrieval trajectory. In this paper, we propose a novel iterative RAG method called ReSP, equipped with a dual-function summarizer. This summarizer compresses information from retrieved documents, targeting both the overarching question and the current sub-question concurrently. Experimental results on the multi-hop question-answering datasets HotpotQA and 2WikiMultihopQA demonstrate that our method significantly outperforms the state-of-the-art, and exhibits excellent robustness concerning context length.

Title: Tree semantic segmentation from aerial image time series

• Authors: Venkatesh Ramesh, Arthur Ouaknine, David Rolnick
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13102
• Pdf URL: https://arxiv.org/pdf/2407.13102
• Copy Paste: [[2407.13102]] Tree semantic segmentation from aerial image time series(https://arxiv.org/abs/2407.13102)
• Keywords: segmentation
• Abstract: Earth's forests play an important role in the fight against climate change, and are in turn negatively affected by it. Effective monitoring of different tree species is essential to understanding and improving the health and biodiversity of forests. In this work, we address the challenge of tree species identification by performing semantic segmentation of trees using an aerial image dataset spanning over a year. We compare models trained on single images versus those trained on time series to assess the impact of tree phenology on segmentation performances. We also introduce a simple convolutional block for extracting spatio-temporal features from image time series, enabling the use of popular pretrained backbones and methods. We leverage the hierarchical structure of tree species taxonomy by incorporating a custom loss function that refines predictions at three levels: species, genus, and higher-level taxa. Our findings demonstrate the superiority of our methodology in exploiting the time series modality and confirm that enriching labels using taxonomic information improves the semantic segmentation performance.

Title: TrialEnroll: Predicting Clinical Trial Enrollment Success with Deep & Cross Network and Large Language Models

• Authors: Ling Yue, Sixue Xing, Jintai Chen, Tianfan Fu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13115
• Pdf URL: https://arxiv.org/pdf/2407.13115
• Copy Paste: [[2407.13115]] TrialEnroll: Predicting Clinical Trial Enrollment Success with Deep & Cross Network and Large Language Models(https://arxiv.org/abs/2407.13115)
• Keywords: interpretability, large language model
• Abstract: Clinical trials need to recruit a sufficient number of volunteer patients to demonstrate the statistical power of the treatment (e.g., a new drug) in curing a certain disease. Clinical trial recruitment has a significant impact on trial success. Forecasting whether the recruitment process would be successful before we run the trial would save many resources and time. This paper develops a novel deep & cross network with large language model (LLM)-augmented text feature that learns semantic information from trial eligibility criteria and predicts enrollment success. The proposed method enables interpretability by understanding which sentence/word in eligibility criteria contributes heavily to prediction. We also demonstrate the empirical superiority of the proposed method (0.7002 PR-AUC) over a bunch of well-established machine learning methods. The code and curated dataset are publicly available at https://anonymous.4open.science/r/TrialEnroll-7E12.

Title: FocusDiffuser: Perceiving Local Disparities for Camouflaged Object Detection

• Authors: Jianwei Zhao, Xin Li, Fan Yang, Qiang Zhai, Ao Luo, Zicheng Jiao, Hong Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13133
• Pdf URL: https://arxiv.org/pdf/2407.13133
• Copy Paste: [[2407.13133]] FocusDiffuser: Perceiving Local Disparities for Camouflaged Object Detection(https://arxiv.org/abs/2407.13133)
• Keywords: diffusion, generative
• Abstract: Detecting objects seamlessly blended into their surroundings represents a complex task for both human cognitive capabilities and advanced artificial intelligence algorithms. Currently, the majority of methodologies for detecting camouflaged objects mainly focus on utilizing discriminative models with various unique designs. However, it has been observed that generative models, such as Stable Diffusion, possess stronger capabilities for understanding various objects in complex environments; Yet their potential for the cognition and detection of camouflaged objects has not been extensively explored. In this study, we present a novel denoising diffusion model, namely FocusDiffuser, to investigate how generative models can enhance the detection and interpretation of camouflaged objects. We believe that the secret to spotting camouflaged objects lies in catching the subtle nuances in details. Consequently, our FocusDiffuser innovatively integrates specialized enhancements, notably the Boundary-Driven LookUp (BDLU) module and Cyclic Positioning (CP) module, to elevate standard diffusion models, significantly boosting the detail-oriented analytical capabilities. Our experiments demonstrate that FocusDiffuser, from a generative perspective, effectively addresses the challenge of camouflaged object detection, surpassing leading models on benchmarks like CAMO, COD10K and NC4K.

Title: OE-BevSeg: An Object Informed and Environment Aware Multimodal Framework for Bird's-eye-view Vehicle Semantic Segmentation

• Authors: Jian Sun, Yuqi Dai, Chi-Man Vong, Qing Xu, Shengbo Eben Li, Jianqiang Wang, Lei He, Keqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13137
• Pdf URL: https://arxiv.org/pdf/2407.13137
• Copy Paste: [[2407.13137]] OE-BevSeg: An Object Informed and Environment Aware Multimodal Framework for Bird's-eye-view Vehicle Semantic Segmentation(https://arxiv.org/abs/2407.13137)
• Keywords: segmentation
• Abstract: Bird's-eye-view (BEV) semantic segmentation is becoming crucial in autonomous driving systems. It realizes ego-vehicle surrounding environment perception by projecting 2D multi-view images into 3D world space. Recently, BEV segmentation has made notable progress, attributed to better view transformation modules, larger image encoders, or more temporal information. However, there are still two issues: 1) a lack of effective understanding and enhancement of BEV space features, particularly in accurately capturing long-distance environmental features and 2) recognizing fine details of target objects. To address these issues, we propose OE-BevSeg, an end-to-end multimodal framework that enhances BEV segmentation performance through global environment-aware perception and local target object enhancement. OE-BevSeg employs an environment-aware BEV compressor. Based on prior knowledge about the main composition of the BEV surrounding environment varying with the increase of distance intervals, long-sequence global modeling is utilized to improve the model's understanding and perception of the environment. From the perspective of enriching target object information in segmentation results, we introduce the center-informed object enhancement module, using centerness information to supervise and guide the segmentation head, thereby enhancing segmentation performance from a local enhancement perspective. Additionally, we designed a multimodal fusion branch that integrates multi-view RGB image features with radar/LiDAR features, achieving significant performance improvements. Extensive experiments show that, whether in camera-only or multimodal fusion BEV segmentation tasks, our approach achieves state-of-the-art results by a large margin on the nuScenes dataset for vehicle segmentation, demonstrating superior applicability in the field of autonomous driving.

Title: Image Inpainting Models are Effective Tools for Instruction-guided Image Editing

• Authors: Xuan Ju, Junhao Zhuang, Zhaoyang Zhang, Yuxuan Bian, Qiang Xu, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13139
• Pdf URL: https://arxiv.org/pdf/2407.13139
• Copy Paste: [[2407.13139]] Image Inpainting Models are Effective Tools for Instruction-guided Image Editing(https://arxiv.org/abs/2407.13139)
• Keywords: diffusion, large language model
• Abstract: This is the technique report for the winning solution of the CVPR2024 GenAI Media Generation Challenge Workshop's Instruction-guided Image Editing track. Instruction-guided image editing has been largely studied in recent years. The most advanced methods, such as SmartEdit and MGIE, usually combine large language models with diffusion models through joint training, where the former provides text understanding ability, and the latter provides image generation ability. However, in our experiments, we find that simply connecting large language models and image generation models through intermediary guidance such as masks instead of joint fine-tuning leads to a better editing performance and success rate. We use a 4-step process IIIE (Inpainting-based Instruction-guided Image Editing): editing category classification, main editing object identification, editing mask acquisition, and image inpainting. Results show that through proper combinations of language models and image inpainting models, our pipeline can reach a high success rate with satisfying visual quality.

Title: A light-weight and efficient punctuation and word casing prediction model for on-device streaming ASR

• Authors: Jian You, Xiangfeng Li
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.13142
• Pdf URL: https://arxiv.org/pdf/2407.13142
• Copy Paste: [[2407.13142]] A light-weight and efficient punctuation and word casing prediction model for on-device streaming ASR(https://arxiv.org/abs/2407.13142)
• Keywords: transformer
• Abstract: Punctuation and word casing prediction are necessary for automatic speech recognition (ASR). With the popularity of on-device end-to-end streaming ASR systems, the on-device punctuation and word casing prediction become a necessity while we found little discussion on this. With the emergence of Transformer, Transformer based models have been explored for this scenario. However, Transformer based models are too large for on-device ASR systems. In this paper, we propose a light-weight and efficient model that jointly predicts punctuation and word casing in real time. The model is based on Convolutional Neural Network (CNN) and Bidirectional Long Short-Term Memory (BiLSTM). Experimental results on the IWSLT2011 test set show that the proposed model obtains 9% relative improvement compared to the best of non-Transformer models on overall F1-score. Compared to the representative of Transformer based models, the proposed model achieves comparable results to the representative model while being only one-fortieth its size and 2.5 times faster in terms of inference time. It is suitable for on-device streaming ASR systems. Our code is publicly available.

Title: Integrated Hardware Architecture and Device Placement Search

• Authors: Irene Wang, Jakub Tarnawski, Amar Phanishayee, Divya Mahajan
• Subjects: cs.LG, cs.AR, cs.DC
• Abstract URL: https://arxiv.org/abs/2407.13143
• Pdf URL: https://arxiv.org/pdf/2407.13143
• Copy Paste: [[2407.13143]] Integrated Hardware Architecture and Device Placement Search(https://arxiv.org/abs/2407.13143)
• Keywords: large language model
• Abstract: Distributed execution of deep learning training involves a dynamic interplay between hardware accelerator architecture and device placement strategy. This is the first work to explore the co-optimization of determining the optimal architecture and device placement strategy through novel algorithms, improving the balance of computational resources, memory usage, and data distribution. Our architecture search leverages tensor and vector units, determining their quantity and dimensionality, and on-chip and off-chip memory configurations. It also determines the microbatch size and decides whether to recompute or stash activations, balancing the memory footprint of training and storage size. For each explored architecture configuration, we use an Integer Linear Program (ILP) to find the optimal schedule for executing operators on the accelerator. The ILP results then integrate with a dynamic programming solution to identify the most effective device placement strategy, combining data, pipeline, and tensor model parallelism across multiple accelerators. Our approach achieves higher throughput on large language models compared to the state-of-the-art TPUv4 and the Spotlight accelerator search framework. The entire source code of PHAZE is available at this https URL.

Title: Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems

• Authors: Daniel Platnick, Bishoy Abdelnour, Eamon Earl, Rahul Kumar, Zahra Rezaei, Thomas Tsangaris, Faraj Lagum
• Subjects: cs.CL, cs.CR, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.13153
• Pdf URL: https://arxiv.org/pdf/2407.13153
• Copy Paste: [[2407.13153]] Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems(https://arxiv.org/abs/2407.13153)
• Keywords: privacy
• Abstract: In recent years, there has been increased demand for speech-to-speech translation (S2ST) systems in industry settings. Although successfully commercialized, cloning-based S2ST systems expose their distributors to liabilities when misused by individuals and can infringe on personality rights when exploited by media organizations. This work proposes a regulated S2ST framework called Preset-Voice Matching (PVM). PVM removes cross-lingual voice cloning in S2ST by first matching the input voice to a similar prior consenting speaker voice in the target-language. With this separation, PVM avoids cloning the input speaker, ensuring PVM systems comply with regulations and reduce risk of misuse. Our results demonstrate PVM can significantly improve S2ST system run-time in multi-speaker settings and the naturalness of S2ST synthesized speech. To our knowledge, PVM is the first explicitly regulated S2ST framework leveraging similarly-matched preset-voices for dynamic S2ST tasks.

Title: Learning Camouflaged Object Detection from Noisy Pseudo Label

• Authors: Jin Zhang, Ruiheng Zhang, Yanjiao Shi, Zhe Cao, Nian Liu, Fahad Shahbaz Khan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13157
• Pdf URL: https://arxiv.org/pdf/2407.13157
• Copy Paste: [[2407.13157]] Learning Camouflaged Object Detection from Noisy Pseudo Label(https://arxiv.org/abs/2407.13157)
• Keywords: segmentation
• Abstract: Existing Camouflaged Object Detection (COD) methods rely heavily on large-scale pixel-annotated training sets, which are both time-consuming and labor-intensive. Although weakly supervised methods offer higher annotation efficiency, their performance is far behind due to the unclear visual demarcations between foreground and background in camouflaged images. In this paper, we explore the potential of using boxes as prompts in camouflaged scenes and introduce the first weakly semi-supervised COD method, aiming for budget-efficient and high-precision camouflaged object segmentation with an extremely limited number of fully labeled images. Critically, learning from such limited set inevitably generates pseudo labels with serious noisy pixels. To address this, we propose a noise correction loss that facilitates the model's learning of correct pixels in the early learning stage, and corrects the error risk gradients dominated by noisy pixels in the memorization stage, ultimately achieving accurate segmentation of camouflaged objects from noisy labels. When using only 20% of fully labeled data, our method shows superior performance over the state-of-the-art methods.

Title: HHGT: Hierarchical Heterogeneous Graph Transformer for Heterogeneous Graph Representation Learning

• Authors: Qiuyu Zhu, Liang Zhang, Qianxiong Xu, Kaijun Liu, Cheng Long, Xiaoyang Wang
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2407.13158
• Pdf URL: https://arxiv.org/pdf/2407.13158
• Copy Paste: [[2407.13158]] HHGT: Hierarchical Heterogeneous Graph Transformer for Heterogeneous Graph Representation Learning(https://arxiv.org/abs/2407.13158)
• Keywords: transformer
• Abstract: Despite the success of Heterogeneous Graph Neural Networks (HGNNs) in modeling real-world Heterogeneous Information Networks (HINs), challenges such as expressiveness limitations and over-smoothing have prompted researchers to explore Graph Transformers (GTs) for enhanced HIN representation learning. However, research on GT in HINs remains limited, with two key shortcomings in existing work: (1) A node's neighbors at different distances in HINs convey diverse semantics. Unfortunately, existing methods ignore such differences and uniformly treat neighbors within a given distance in a coarse manner, which results in semantic confusion. (2) Nodes in HINs have various types, each with unique semantics. Nevertheless, existing methods mix nodes of different types during neighbor aggregation, hindering the capture of proper correlations between nodes of diverse types. To bridge these gaps, we design an innovative structure named (k,t)-ring neighborhood, where nodes are initially organized by their distance, forming different non-overlapping k-ring neighborhoods for each distance. Within each k-ring structure, nodes are further categorized into different groups according to their types, thus emphasizing the heterogeneity of both distances and types in HINs naturally. Based on this structure, we propose a novel Hierarchical Heterogeneous Graph Transformer (HHGT) model, which seamlessly integrates a Type-level Transformer for aggregating nodes of different types within each k-ring neighborhood, followed by a Ring-level Transformer for aggregating different k-ring neighborhoods in a hierarchical manner. Extensive experiments are conducted on downstream tasks to verify HHGT's superiority over 14 baselines, with a notable improvement of up to 24.75% in NMI and 29.25% in ARI for node clustering task on the ACM dataset compared to the best baseline.

Title: Translate-and-Revise: Boosting Large Language Models for Constrained Translation

• Authors: Pengcheng Huang, Yongyu Mu, Yuzhang Wu, Bei Li, Chunyang Xiao, Tong Xiao, Jingbo Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13164
• Pdf URL: https://arxiv.org/pdf/2407.13164
• Copy Paste: [[2407.13164]] Translate-and-Revise: Boosting Large Language Models for Constrained Translation(https://arxiv.org/abs/2407.13164)
• Keywords: large language model
• Abstract: Imposing constraints on machine translation systems presents a challenging issue because these systems are not trained to make use of constraints in generating adequate, fluent translations. In this paper, we leverage the capabilities of large language models (LLMs) for constrained translation, given that LLMs can easily adapt to this task by taking translation instructions and constraints as prompts. However, LLMs cannot always guarantee the adequacy of translation, and, in some cases, ignore the given constraints. This is in part because LLMs might be overly confident in their predictions, overriding the influence of the constraints. To overcome this overiding behaviour, we propose to add a revision process that encourages LLMs to correct the outputs by prompting them about the constraints that have not yet been met. We evaluate our approach on four constrained translation tasks, encompassing both lexical and structural constraints in multiple constraint domains. Experiments show 15\% improvement in constraint-based translation accuracy over standard LLMs and the approach also significantly outperforms neural machine translation (NMT) state-of-the-art methods.

Title: Unified-EGformer: Exposure Guided Lightweight Transformer for Mixed-Exposure Image Enhancement

• Authors: Eashan Adhikarla, Kai Zhang, Rosaura G. VidalMata, Manjushree Aithal, Nikhil Ambha Madhusudhana, John Nicholson, Lichao Sun, Brian D. Davison
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13170
• Pdf URL: https://arxiv.org/pdf/2407.13170
• Copy Paste: [[2407.13170]] Unified-EGformer: Exposure Guided Lightweight Transformer for Mixed-Exposure Image Enhancement(https://arxiv.org/abs/2407.13170)
• Keywords: transformer
• Abstract: Despite recent strides made by AI in image processing, the issue of mixed exposure, pivotal in many real-world scenarios like surveillance and photography, remains inadequately addressed. Traditional image enhancement techniques and current transformer models are limited with primary focus on either overexposure or underexposure. To bridge this gap, we introduce the Unified-Exposure Guided Transformer (Unified-EGformer). Our proposed solution is built upon advanced transformer architectures, equipped with local pixel-level refinement and global refinement blocks for color correction and image-wide adjustments. We employ a guided attention mechanism to precisely identify exposure-compromised regions, ensuring its adaptability across various real-world conditions. U-EGformer, with a lightweight design featuring a memory footprint (peak memory) of only $\sim$1134 MB (0.1 Million parameters) and an inference time of 95 ms (9.61x faster than the average), is a viable choice for real-time applications such as surveillance and autonomous navigation. Additionally, our model is highly generalizable, requiring minimal fine-tuning to handle multiple tasks and datasets with a single architecture.

Title: Compressed models are NOT miniature versions of large models

• Authors: Rohit Raj Rai, Rishant Pal, Amit Awekar
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.13174
• Pdf URL: https://arxiv.org/pdf/2407.13174
• Copy Paste: [[2407.13174]] Compressed models are NOT miniature versions of large models(https://arxiv.org/abs/2407.13174)
• Keywords: attack
• Abstract: Large neural models are often compressed before deployment. Model compression is necessary for many practical reasons, such as inference latency, memory footprint, and energy consumption. Compressed models are assumed to be miniature versions of corresponding large neural models. However, we question this belief in our work. We compare compressed models with corresponding large neural models using four model characteristics: prediction errors, data representation, data distribution, and vulnerability to adversarial attack. We perform experiments using the BERT-large model and its five compressed versions. For all four model characteristics, compressed models significantly differ from the BERT-large model. Even among compressed models, they differ from each other on all four model characteristics. Apart from the expected loss in model performance, there are major side effects of using compressed models to replace large neural models.

Title: The use of the symmetric finite difference in the local binary pattern (symmetric LBP)

• Authors: Zeinab Sedaghatjoo, Hossein Hosseinzadeh
• Subjects: cs.CV, math.NA
• Abstract URL: https://arxiv.org/abs/2407.13178
• Pdf URL: https://arxiv.org/pdf/2407.13178
• Copy Paste: [[2407.13178]] The use of the symmetric finite difference in the local binary pattern (symmetric LBP)(https://arxiv.org/abs/2407.13178)
• Keywords: extraction
• Abstract: The paper provides a mathematical view to the binary numbers presented in the Local Binary Pattern (LBP) feature extraction process. Symmetric finite difference is often applied in numerical analysis to enhance the accuracy of approximations. Then, the paper investigates utilization of the symmetric finite difference in the LBP formulation for face detection and facial expression recognition. It introduces a novel approach that extends the standard LBP, which typically employs eight directional derivatives, to incorporate only four directional derivatives. This approach is named symmetric LBP. The number of LBP features is reduced to 16 from 256 by the use of the symmetric LBP. The study underscores the significance of the number of directions considered in the new approach. Consequently, the results obtained emphasize the importance of the research topic.

Title: Training-Free Large Model Priors for Multiple-in-One Image Restoration

• Authors: Xuanhua He, Lang Li, Yingying Wang, Hui Zheng, Ke Cao, Keyu Yan, Rui Li, Chengjun Xie, Jie Zhang, Man Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13181
• Pdf URL: https://arxiv.org/pdf/2407.13181
• Copy Paste: [[2407.13181]] Training-Free Large Model Priors for Multiple-in-One Image Restoration(https://arxiv.org/abs/2407.13181)
• Keywords: diffusion, transformer
• Abstract: Image restoration aims to reconstruct the latent clear images from their degraded versions. Despite the notable achievement, existing methods predominantly focus on handling specific degradation types and thus require specialized models, impeding real-world applications in dynamic degradation scenarios. To address this issue, we propose Large Model Driven Image Restoration framework (LMDIR), a novel multiple-in-one image restoration paradigm that leverages the generic priors from large multi-modal language models (MMLMs) and the pretrained diffusion models. In detail, LMDIR integrates three key prior knowledges: 1) global degradation knowledge from MMLMs, 2) scene-aware contextual descriptions generated by MMLMs, and 3) fine-grained high-quality reference images synthesized by diffusion models guided by MMLM descriptions. Standing on above priors, our architecture comprises a query-based prompt encoder, degradation-aware transformer block injecting global degradation knowledge, content-aware transformer block incorporating scene description, and reference-based transformer block incorporating fine-grained image priors. This design facilitates single-stage training paradigm to address various degradations while supporting both automatic and user-guided restoration. Extensive experiments demonstrate that our designed method outperforms state-of-the-art competitors on multiple evaluation benchmarks.

Title: SpaDiT: Diffusion Transformer for Spatial Gene Expression Prediction using scRNA-seq

• Authors: Xiaoyu Li, Fangfang Zhu, Wenwen Min
• Subjects: cs.LG, cs.AI, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2407.13182
• Pdf URL: https://arxiv.org/pdf/2407.13182
• Copy Paste: [[2407.13182]] SpaDiT: Diffusion Transformer for Spatial Gene Expression Prediction using scRNA-seq(https://arxiv.org/abs/2407.13182)
• Keywords: diffusion, transformer, generative
• Abstract: The rapid development of spatial transcriptomics (ST) technologies is revolutionizing our understanding of the spatial organization of biological tissues. Current ST methods, categorized into next-generation sequencing-based (seq-based) and fluorescence in situ hybridization-based (image-based) methods, offer innovative insights into the functional dynamics of biological tissues. However, these methods are limited by their cellular resolution and the quantity of genes they can detect. To address these limitations, we propose SpaDiT, a deep learning method that utilizes a diffusion generative model to integrate scRNA-seq and ST data for the prediction of undetected genes. By employing a Transformer-based diffusion model, SpaDiT not only accurately predicts unknown genes but also effectively generates the spatial structure of ST genes. We have demonstrated the effectiveness of SpaDiT through extensive experiments on both seq-based and image-based ST data. SpaDiT significantly contributes to ST gene prediction methods with its innovative approach. Compared to eight leading baseline methods, SpaDiT achieved state-of-the-art performance across multiple metrics, highlighting its substantial bioinformatics contribution.

Title: HSEmotion Team at the 7th ABAW Challenge: Multi-Task Learning and Compound Facial Expression Recognition

• Authors: Andrey V. Savchenko
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13184
• Pdf URL: https://arxiv.org/pdf/2407.13184
• Copy Paste: [[2407.13184]] HSEmotion Team at the 7th ABAW Challenge: Multi-Task Learning and Compound Facial Expression Recognition(https://arxiv.org/abs/2407.13184)
• Keywords: privacy
• Abstract: In this paper, we describe the results of the HSEmotion team in two tasks of the seventh Affective Behavior Analysis in-the-wild (ABAW) competition, namely, multi-task learning for simultaneous prediction of facial expression, valence, arousal, and detection of action units, and compound expression recognition. We propose an efficient pipeline based on frame-level facial feature extractors pre-trained in multi-task settings to estimate valence-arousal and basic facial expressions given a facial photo. We ensure the privacy-awareness of our techniques by using the lightweight architectures of neural networks, such as MT-EmotiDDAMFN, MT-EmotiEffNet, and MT-EmotiMobileFaceNet, that can run even on a mobile device without the need to send facial video to a remote server. It was demonstrated that a significant step in improving the overall accuracy is the smoothing of neural network output scores using Gaussian or box filters. It was experimentally demonstrated that such a simple post-processing of predictions from simple blending of two top visual models improves the F1-score of facial expression recognition up to 7%. At the same time, the mean Concordance Correlation Coefficient (CCC) of valence and arousal is increased by up to 1.25 times compared to each model's frame-level predictions. As a result, our final performance score on the validation set from the multi-task learning challenge is 4.5 times higher than the baseline (1.494 vs 0.32).

Title: Safe-SD: Safe and Traceable Stable Diffusion with Text Prompt Trigger for Invisible Generative Watermarking

• Authors: Zhiyuan Ma, Guoli Jia, Biqing Qi, Bowen Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13188
• Pdf URL: https://arxiv.org/pdf/2407.13188
• Copy Paste: [[2407.13188]] Safe-SD: Safe and Traceable Stable Diffusion with Text Prompt Trigger for Invisible Generative Watermarking(https://arxiv.org/abs/2407.13188)
• Keywords: protect, robust, watermark, diffusion, generative
• Abstract: Recently, stable diffusion (SD) models have typically flourished in the field of image synthesis and personalized editing, with a range of photorealistic and unprecedented images being successfully generated. As a result, widespread interest has been ignited to develop and use various SD-based tools for visual content creation. However, the exposure of AI-created content on public platforms could raise both legal and ethical risks. In this regard, the traditional methods of adding watermarks to the already generated images (i.e. post-processing) may face a dilemma (e.g., being erased or modified) in terms of copyright protection and content monitoring, since the powerful image inversion and text-to-image editing techniques have been widely explored in SD-based methods. In this work, we propose a Safe and high-traceable Stable Diffusion framework (namely Safe-SD) to adaptively implant the graphical watermarks (e.g., QR code) into the imperceptible structure-related pixels during the generative diffusion process for supporting text-driven invisible watermarking and detection. Different from the previous high-cost injection-then-detection training framework, we design a simple and unified architecture, which makes it possible to simultaneously train watermark injection and detection in a single network, greatly improving the efficiency and convenience of use. Moreover, to further support text-driven generative watermarking and deeply explore its robustness and high-traceability, we elaborately design lambda sampling and encryption algorithm to fine-tune a latent diffuser wrapped by a VAE for balancing high-fidelity image synthesis and high-traceable watermark detection. We present our quantitative and qualitative results on two representative datasets LSUN, COCO and FFHQ, demonstrating state-of-the-art performance of Safe-SD and showing it significantly outperforms the previous approaches.

Title: Retrieval-Augmented Generation for Natural Language Processing: A Survey

• Authors: Shangyu Wu, Ying Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang, Xue Liu, Tei-Wei Kuo, Nan Guan, Chun Jason Xue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13193
• Pdf URL: https://arxiv.org/pdf/2407.13193
• Copy Paste: [[2407.13193]] Retrieval-Augmented Generation for Natural Language Processing: A Survey(https://arxiv.org/abs/2407.13193)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated great success in various fields, benefiting from their huge amount of parameters that store knowledge. However, LLMs still suffer from several key issues, such as hallucination problems, knowledge update issues, and lacking domain-specific expertise. The appearance of retrieval-augmented generation (RAG), which leverages an external knowledge database to augment LLMs, makes up those drawbacks of LLMs. This paper reviews all significant techniques of RAG, especially in the retriever and the retrieval fusions. Besides, tutorial codes are provided for implementing the representative techniques in RAG. This paper further discusses the RAG training, including RAG with/without datastore update. Then, we introduce the application of RAG in representative natural language processing tasks and industrial scenarios. Finally, this paper discusses the future directions and challenges of RAG for promoting its development.

Title: Robust Multivariate Time Series Forecasting against Intra- and Inter-Series Transitional Shift

• Authors: Hui He, Qi Zhang, Kun Yi, Xiaojun Xue, Shoujin Wang, Liang Hu, Longbing Cao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13194
• Pdf URL: https://arxiv.org/pdf/2407.13194
• Copy Paste: [[2407.13194]] Robust Multivariate Time Series Forecasting against Intra- and Inter-Series Transitional Shift(https://arxiv.org/abs/2407.13194)
• Keywords: robust, interpretability
• Abstract: The non-stationary nature of real-world Multivariate Time Series (MTS) data presents forecasting models with a formidable challenge of the time-variant distribution of time series, referred to as distribution shift. Existing studies on the distribution shift mostly adhere to adaptive normalization techniques for alleviating temporal mean and covariance shifts or time-variant modeling for capturing temporal shifts. Despite improving model generalization, these normalization-based methods often assume a time-invariant transition between outputs and inputs but disregard specific intra-/inter-series correlations, while time-variant models overlook the intrinsic causes of the distribution shift. This limits model expressiveness and interpretability of tackling the distribution shift for MTS forecasting. To mitigate such a dilemma, we present a unified Probabilistic Graphical Model to Jointly capturing intra-/inter-series correlations and modeling the time-variant transitional distribution, and instantiate a neural framework called JointPGM for non-stationary MTS forecasting. Specifically, JointPGM first employs multiple Fourier basis functions to learn dynamic time factors and designs two distinct learners: intra-series and inter-series learners. The intra-series learner effectively captures temporal dynamics by utilizing temporal gates, while the inter-series learner explicitly models spatial dynamics through multi-hop propagation, incorporating Gumbel-softmax sampling. These two types of series dynamics are subsequently fused into a latent variable, which is inversely employed to infer time factors, generate final prediction, and perform reconstruction. We validate the effectiveness and efficiency of JointPGM through extensive experiments on six highly non-stationary MTS datasets, achieving state-of-the-art forecasting performance of MTS forecasting.

Title: Adapt PointFormer: 3D Point Cloud Analysis via Adapting 2D Visual Transformers

• Authors: Mengke Li, Da Li, Guoqing Yang, Yiu-ming Cheung, Hui Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13200
• Pdf URL: https://arxiv.org/pdf/2407.13200
• Copy Paste: [[2407.13200]] Adapt PointFormer: 3D Point Cloud Analysis via Adapting 2D Visual Transformers(https://arxiv.org/abs/2407.13200)
• Keywords: transformer
• Abstract: Pre-trained large-scale models have exhibited remarkable efficacy in computer vision, particularly for 2D image analysis. However, when it comes to 3D point clouds, the constrained accessibility of data, in contrast to the vast repositories of images, poses a challenge for the development of 3D pre-trained models. This paper therefore attempts to directly leverage pre-trained models with 2D prior knowledge to accomplish the tasks for 3D point cloud analysis. Accordingly, we propose the Adaptive PointFormer (APF), which fine-tunes pre-trained 2D models with only a modest number of parameters to directly process point clouds, obviating the need for mapping to images. Specifically, we convert raw point clouds into point embeddings for aligning dimensions with image tokens. Given the inherent disorder in point clouds, in contrast to the structured nature of images, we then sequence the point embeddings to optimize the utilization of 2D attention priors. To calibrate attention across 3D and 2D domains and reduce computational overhead, a trainable PointFormer with a limited number of parameters is subsequently concatenated to a frozen pre-trained image model. Extensive experiments on various benchmarks demonstrate the effectiveness of the proposed APF. The source code and more details are available at https://vcc.tech/research/2024/PointFormer.

Title: Transformer-based Single-Cell Language Model: A Survey

• Authors: Wei Lan, Guohang He, Mingyang Liu, Qingfeng Chen, Junyue Cao, Wei Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13205
• Pdf URL: https://arxiv.org/pdf/2407.13205
• Copy Paste: [[2407.13205]] Transformer-based Single-Cell Language Model: A Survey(https://arxiv.org/abs/2407.13205)
• Keywords: transformer, large language model
• Abstract: The transformers have achieved significant accomplishments in the natural language processing as its outstanding parallel processing capabilities and highly flexible attention mechanism. In addition, increasing studies based on transformers have been proposed to model single-cell data. In this review, we attempt to systematically summarize the single-cell language models and applications based on transformers. First, we provide a detailed introduction about the structure and principles of transformers. Then, we review the single-cell language models and large language models for single-cell data analysis. Moreover, we explore the datasets and applications of single-cell language models in downstream tasks such as batch correction, cell clustering, cell type annotation, gene regulatory network inference and perturbation response. Further, we discuss the challenges of single-cell language models and provide promising research directions. We hope this review will serve as an up-to-date reference for researchers interested in the direction of single-cell language models.

Title: Research on Image Super-Resolution Reconstruction Mechanism based on Convolutional Neural Network

• Authors: Hao Yan, Zixiang Wang, Zhengjia Xu, Zhuoyue Wang, Zhizhong Wu, Ranran Lyu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.13211
• Pdf URL: https://arxiv.org/pdf/2407.13211
• Copy Paste: [[2407.13211]] Research on Image Super-Resolution Reconstruction Mechanism based on Convolutional Neural Network(https://arxiv.org/abs/2407.13211)
• Keywords: extraction
• Abstract: Super-resolution reconstruction techniques entail the utilization of software algorithms to transform one or more sets of low-resolution images captured from the same scene into high-resolution images. In recent years, considerable advancement has been observed in the domain of single-image super-resolution algorithms, particularly those based on deep learning techniques. Nevertheless, the extraction of image features and nonlinear mapping methods in the reconstruction process remain challenging for existing algorithms. These issues result in the network architecture being unable to effectively utilize the diverse range of information at different levels. The loss of high-frequency details is significant, and the final reconstructed image features are overly smooth, with a lack of fine texture details. This negatively impacts the subjective visual quality of the image. The objective is to recover high-quality, high-resolution images from low-resolution images. In this work, an enhanced deep convolutional neural network model is employed, comprising multiple convolutional layers, each of which is configured with specific filters and activation functions to effectively capture the diverse features of the image. Furthermore, a residual learning strategy is employed to accelerate training and enhance the convergence of the network, while sub-pixel convolutional layers are utilized to refine the high-frequency details and textures of the image. The experimental analysis demonstrates the superior performance of the proposed model on multiple public datasets when compared with the traditional bicubic interpolation method and several other learning-based super-resolution methods. Furthermore, it proves the model's efficacy in maintaining image edges and textures.

Title: Evaluating Large Language Models for Anxiety and Depression Classification using Counseling and Psychotherapy Transcripts

• Authors: Junwei Sun, Siqi Ma, Yiran Fan, Peter Washington
• Subjects: cs.CL, cs.CY, cs.ET, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13228
• Pdf URL: https://arxiv.org/pdf/2407.13228
• Copy Paste: [[2407.13228]] Evaluating Large Language Models for Anxiety and Depression Classification using Counseling and Psychotherapy Transcripts(https://arxiv.org/abs/2407.13228)
• Keywords: transformer, large language model
• Abstract: We aim to evaluate the efficacy of traditional machine learning and large language models (LLMs) in classifying anxiety and depression from long conversational transcripts. We fine-tune both established transformer models (BERT, RoBERTa, Longformer) and more recent large models (Mistral-7B), trained a Support Vector Machine with feature engineering, and assessed GPT models through prompting. We observe that state-of-the-art models fail to enhance classification outcomes compared to traditional machine learning methods.

Title: Transformers with Stochastic Competition for Tabular Data Modelling

• Authors: Andreas Voskou, Charalambos Christoforou, Sotirios Chatzis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13238
• Pdf URL: https://arxiv.org/pdf/2407.13238
• Copy Paste: [[2407.13238]] Transformers with Stochastic Competition for Tabular Data Modelling(https://arxiv.org/abs/2407.13238)
• Keywords: transformer
• Abstract: Despite the prevalence and significance of tabular data across numerous industries and fields, it has been relatively underexplored in the realm of deep learning. Even today, neural networks are often overshadowed by techniques such as gradient boosted decision trees (GBDT). However, recent models are beginning to close this gap, outperforming GBDT in various setups and garnering increased attention in the field. Inspired by this development, we introduce a novel stochastic deep learning model specifically designed for tabular data. The foundation of this model is a Transformer-based architecture, carefully adapted to cater to the unique properties of tabular data through strategic architectural modifications and leveraging two forms of stochastic competition. First, we employ stochastic "Local Winner Takes All" units to promote generalization capacity through stochasticity and sparsity. Second, we introduce a novel embedding layer that selects among alternative linear embedding layers through a mechanism of stochastic competition. The effectiveness of the model is validated on a variety of widely-used, publicly available datasets. We demonstrate that, through the incorporation of these elements, our model yields high performance and marks a significant advancement in the application of deep learning to tabular data.

Title: Intelligo ut Confido: Understanding, Trust and User Experience in Verifiable Receipt-Free E-Voting (long version)

• Authors: Marie-Laure Zollinger, Peter B. Rønne, Steve Schneider, Peter Y. A. Ryan, Wojtek Jamroga
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.13240
• Pdf URL: https://arxiv.org/pdf/2407.13240
• Copy Paste: [[2407.13240]] Intelligo ut Confido: Understanding, Trust and User Experience in Verifiable Receipt-Free E-Voting (long version)(https://arxiv.org/abs/2407.13240)
• Keywords: privacy, attack
• Abstract: Voting protocols seek to provide integrity and vote privacy in elections. To achieve integrity, procedures have been proposed allowing voters to verify their vote - however this impacts both the user experience and privacy. Especially, vote verification can lead to vote-buying or coercion, if an attacker can obtain documentation, i.e. a receipt, of the cast vote. Thus, some voting protocols go further and provide mechanisms to prevent such receipts. To be effective, this so-called receipt-freeness depends on voters being able to understand and use these mechanisms. In this paper, we present a study with 300 participants which aims to evaluate the voters' experience of the receipt-freeness procedures in the e-voting protocol Selene in the context of vote-buying. This actually constitutes the first user study dealing with vote-buying in e-voting. While the usability and trust factors were rated low in the experiments, we found a positive correlation between trust and understanding.

Title: NODER: Image Sequence Regression Based on Neural Ordinary Differential Equations

• Authors: Hao Bai, Yi Hong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13241
• Pdf URL: https://arxiv.org/pdf/2407.13241
• Copy Paste: [[2407.13241]] NODER: Image Sequence Regression Based on Neural Ordinary Differential Equations(https://arxiv.org/abs/2407.13241)
• Keywords: diffusion
• Abstract: Regression on medical image sequences can capture temporal image pattern changes and predict images at missing or future time points. However, existing geodesic regression methods limit their regression performance by a strong underlying assumption of linear dynamics, while diffusion-based methods have high computational costs and lack constraints to preserve image topology. In this paper, we propose an optimization-based new framework called NODER, which leverages neural ordinary differential equations to capture complex underlying dynamics and reduces its high computational cost of handling high-dimensional image volumes by introducing the latent space. We compare our NODER with two recent regression methods, and the experimental results on ADNI and ACDC datasets demonstrate that our method achieves the state-of-the-art performance in 3D image regression. Our model needs only a couple of images in a sequence for prediction, which is practical, especially for clinical situations where extremely limited image time series are available for analysis. Our source code is available at this https URL.

Title: PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks

• Authors: Alessandro Berti, Humam Kourani, Wil M.P. van der Aalst
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2407.13244
• Pdf URL: https://arxiv.org/pdf/2407.13244
• Copy Paste: [[2407.13244]] PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks(https://arxiv.org/abs/2407.13244)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have the potential to semi-automate some process mining (PM) analyses. While commercial models are already adequate for many analytics tasks, the competitive level of open-source LLMs in PM tasks is unknown. In this paper, we propose PM-LLM-Benchmark, the first comprehensive benchmark for PM focusing on domain knowledge (process-mining-specific and process-specific) and on different implementation strategies. We focus also on the challenges in creating such a benchmark, related to the public availability of the data and on evaluation biases by the LLMs. Overall, we observe that most of the considered LLMs can perform some process mining tasks at a satisfactory level, but tiny models that would run on edge devices are still inadequate. We also conclude that while the proposed benchmark is useful for identifying LLMs that are adequate for process mining tasks, further research is needed to overcome the evaluation biases and perform a more thorough ranking of the competitive LLMs.

Title: STS MICCAI 2023 Challenge: Grand challenge on 2D and 3D semi-supervised tooth segmentation

• Authors: Yaqi Wang, Yifan Zhang, Xiaodiao Chen, Shuai Wang, Dahong Qian, Fan Ye, Feng Xu, Hongyuan Zhang, Qianni Zhang, Chengyu Wu, Yunxiang Li, Weiwei Cui, Shan Luo, Chengkai Wang, Tianhao Li, Yi Liu, Xiang Feng, Huiyu Zhou, Dongyun Liu, Qixuan Wang, Zhouhao Lin, Wei Song, Yuanlin Li, Bing Wang, Chunshi Wang, Qiupu Chen, Mingqian Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13246
• Pdf URL: https://arxiv.org/pdf/2407.13246
• Copy Paste: [[2407.13246]] STS MICCAI 2023 Challenge: Grand challenge on 2D and 3D semi-supervised tooth segmentation(https://arxiv.org/abs/2407.13246)
• Keywords: fair, segmentation
• Abstract: Computer-aided design (CAD) tools are increasingly popular in modern dental practice, particularly for treatment planning or comprehensive prognosis evaluation. In particular, the 2D panoramic X-ray image efficiently detects invisible caries, impacted teeth and supernumerary teeth in children, while the 3D dental cone beam computed tomography (CBCT) is widely used in orthodontics and endodontics due to its low radiation dose. However, there is no open-access 2D public dataset for children's teeth and no open 3D dental CBCT dataset, which limits the development of automatic algorithms for segmenting teeth and analyzing diseases. The Semi-supervised Teeth Segmentation (STS) Challenge, a pioneering event in tooth segmentation, was held as a part of the MICCAI 2023 ToothFairy Workshop on the Alibaba Tianchi platform. This challenge aims to investigate effective semi-supervised tooth segmentation algorithms to advance the field of dentistry. In this challenge, we provide two modalities including the 2D panoramic X-ray images and the 3D CBCT tooth volumes. In Task 1, the goal was to segment tooth regions in panoramic X-ray images of both adult and pediatric teeth. Task 2 involved segmenting tooth sections using CBCT volumes. Limited labelled images with mostly unlabelled ones were provided in this challenge prompt using semi-supervised algorithms for training. In the preliminary round, the challenge received registration and result submission by 434 teams, with 64 advancing to the final round. This paper summarizes the diverse methods employed by the top-ranking teams in the STS MICCAI 2023 Challenge.

Title: Are Large Language Models Capable of Generating Human-Level Narratives?

• Authors: Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, Nanyun Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13248
• Pdf URL: https://arxiv.org/pdf/2407.13248
• Copy Paste: [[2407.13248]] Are Large Language Models Capable of Generating Human-Level Narratives?(https://arxiv.org/abs/2407.13248)
• Keywords: generative, large language model
• Abstract: This paper investigates the capability of LLMs in storytelling, focusing on narrative development and plot progression. We introduce a novel computational framework to analyze narratives through three discourse-level aspects: i) story arcs, ii) turning points, and iii) affective dimensions, including arousal and valence. By leveraging expert and automatic annotations, we uncover significant discrepancies between the LLM- and human- written stories. While human-written stories are suspenseful, arousing, and diverse in narrative structures, LLM stories are homogeneously positive and lack tension. Next, we measure narrative reasoning skills as a precursor to generative capacities, concluding that most LLMs fall short of human abilities in discourse understanding. Finally, we show that explicit integration of aforementioned discourse features can enhance storytelling, as is demonstrated by over 40% improvement in neural storytelling in terms of diversity, suspense, and arousal.

Title: Motif-Consistent Counterfactuals with Adversarial Refinement for Graph-Level Anomaly Detection

• Authors: Chunjing Xiao, Shikang Pang, Wenxin Tai, Yanlong Huang, Goce Trajcevski, Fan Zhou
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2407.13251
• Pdf URL: https://arxiv.org/pdf/2407.13251
• Copy Paste: [[2407.13251]] Motif-Consistent Counterfactuals with Adversarial Refinement for Graph-Level Anomaly Detection(https://arxiv.org/abs/2407.13251)
• Keywords: generative
• Abstract: Graph-level anomaly detection is significant in diverse domains. To improve detection performance, counterfactual graphs have been exploited to benefit the generalization capacity by learning causal relations. Most existing studies directly introduce perturbations (e.g., flipping edges) to generate counterfactual graphs, which are prone to alter the semantics of generated examples and make them off the data manifold, resulting in sub-optimal performance. To address these issues, we propose a novel approach, Motif-consistent Counterfactuals with Adversarial Refinement (MotifCAR), for graph-level anomaly detection. The model combines the motif of one graph, the core subgraph containing the identification (category) information, and the contextual subgraph (non-motif) of another graph to produce a raw counterfactual graph. However, the produced raw graph might be distorted and cannot satisfy the important counterfactual properties: Realism, Validity, Proximity and Sparsity. Towards that, we present a Generative Adversarial Network (GAN)-based graph optimizer to refine the raw counterfactual graphs. It adopts the discriminator to guide the generator to generate graphs close to realistic data, i.e., meet the property Realism. Further, we design the motif consistency to force the motif of the generated graphs to be consistent with the realistic graphs, meeting the property Validity. Also, we devise the contextual loss and connection loss to control the contextual subgraph and the newly added links to meet the properties Proximity and Sparsity. As a result, the model can generate high-quality counterfactual graphs. Experiments demonstrate the superiority of MotifCAR.

Title: Unveiling Structural Memorization: Structural Membership Inference Attack for Text-to-Image Diffusion Models

• Authors: Qiao Li, Xiaomeng Fu, Xi Wang, Jin Liu, Xingyu Gao, Jiao Dai, Jizhong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13252
• Pdf URL: https://arxiv.org/pdf/2407.13252
• Copy Paste: [[2407.13252]] Unveiling Structural Memorization: Structural Membership Inference Attack for Text-to-Image Diffusion Models(https://arxiv.org/abs/2407.13252)
• Keywords: privacy, protect, attack, robust, membership infer, diffusion
• Abstract: With the rapid advancements of large-scale text-to-image diffusion models, various practical applications have emerged, bringing significant convenience to society. However, model developers may misuse the unauthorized data to train diffusion models. These data are at risk of being memorized by the models, thus potentially violating citizens' privacy rights. Therefore, in order to judge whether a specific image is utilized as a member of a model's training set, Membership Inference Attack (MIA) is proposed to serve as a tool for privacy protection. Current MIA methods predominantly utilize pixel-wise comparisons as distinguishing clues, considering the pixel-level memorization characteristic of diffusion models. However, it is practically impossible for text-to-image models to memorize all the pixel-level information in massive training sets. Therefore, we move to the more advanced structure-level memorization. Observations on the diffusion process show that the structures of members are better preserved compared to those of nonmembers, indicating that diffusion models possess the capability to remember the structures of member images from training sets. Drawing on these insights, we propose a simple yet effective MIA method tailored for text-to-image diffusion models. Extensive experimental results validate the efficacy of our approach. Compared to current pixel-level baselines, our approach not only achieves state-of-the-art performance but also demonstrates remarkable robustness against various distortions.

Title: Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation

• Authors: Shoumeng Qiu, Jie Chen, Xinrun Li, Ru Wan, Xiangyang Xue, Jian Pu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13254
• Pdf URL: https://arxiv.org/pdf/2407.13254
• Copy Paste: [[2407.13254]] Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation(https://arxiv.org/abs/2407.13254)
• Keywords: robust, segmentation
• Abstract: In this paper, we introduce a novel knowledge distillation approach for the semantic segmentation task. Unlike previous methods that rely on power-trained teachers or other modalities to provide additional knowledge, our approach does not require complex teacher models or information from extra sensors. Specifically, for the teacher model training, we propose to noise the label and then incorporate it into input to effectively boost the lightweight teacher performance. To ensure the robustness of the teacher model against the introduced noise, we propose a dual-path consistency training strategy featuring a distance loss between the outputs of two paths. For the student model training, we keep it consistent with the standard distillation for simplicity. Our approach not only boosts the efficacy of knowledge distillation but also increases the flexibility in selecting teacher and student models. To demonstrate the advantages of our Label Assisted Distillation (LAD) method, we conduct extensive experiments on five challenging datasets including Cityscapes, ADE20K, PASCAL-VOC, COCO-Stuff 10K, and COCO-Stuff 164K, five popular models: FCN, PSPNet, DeepLabV3, STDC, and OCRNet, and results show the effectiveness and generalization of our approach. We posit that incorporating labels into the input, as demonstrated in our work, will provide valuable insights into related fields. Code is available at this https URL.

Title: Deep Time Series Models: A Comprehensive Survey and Benchmark

• Authors: Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Mingsheng Long, Jianmin Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13278
• Pdf URL: https://arxiv.org/pdf/2407.13278
• Copy Paste: [[2407.13278]] Deep Time Series Models: A Comprehensive Survey and Benchmark(https://arxiv.org/abs/2407.13278)
• Keywords: fair
• Abstract: Time series, characterized by a sequence of data points arranged in a discrete-time order, are ubiquitous in real-world applications. Different from other modalities, time series present unique challenges due to their complex and dynamic nature, including the entanglement of nonlinear patterns and time-variant trends. Analyzing time series data is of great significance in real-world scenarios and has been widely studied over centuries. Recent years have witnessed remarkable breakthroughs in the time series community, with techniques shifting from traditional statistical methods to advanced deep learning models. In this paper, we delve into the design of deep time series models across various analysis tasks and review the existing literature from two perspectives: basic modules and model architectures. Further, we develop and release Time Series Library (TSLib) as a fair benchmark of deep time series models for diverse analysis tasks, which implements 24 mainstream models, covers 30 datasets from different domains, and supports five prevalent analysis tasks. Based on TSLib, we thoroughly evaluate 12 advanced deep time series models on different tasks. Empirical results indicate that models with specific structures are well-suited for distinct analytical tasks, which offers insights for research and adoption of deep time series models. Code is available at this https URL.

Title: Auditing Local Explanations is Hard

• Authors: Robi Bhattacharjee, Ulrike von Luxburg
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13281
• Pdf URL: https://arxiv.org/pdf/2407.13281
• Copy Paste: [[2407.13281]] Auditing Local Explanations is Hard(https://arxiv.org/abs/2407.13281)
• Keywords: explainability
• Abstract: In sensitive contexts, providers of machine learning algorithms are increasingly required to give explanations for their algorithms' decisions. However, explanation receivers might not trust the provider, who potentially could output misleading or manipulated explanations. In this work, we investigate an auditing framework in which a third-party auditor or a collective of users attempts to sanity-check explanations: they can query model decisions and the corresponding local explanations, pool all the information received, and then check for basic consistency properties. We prove upper and lower bounds on the amount of queries that are needed for an auditor to succeed within this framework. Our results show that successful auditing requires a potentially exorbitant number of queries -- particularly in high dimensional cases. Our analysis also reveals that a key property is the ``locality'' of the provided explanations -- a quantity that so far has not been paid much attention to in the explainability literature. Looking forward, our results suggest that for complex high-dimensional settings, merely providing a pointwise prediction and explanation could be insufficient, as there is no way for the users to verify that the provided explanations are not completely made-up.

Title: SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learning

• Authors: Joseph Marvin Imperial, Harish Tayyar Madabushi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13297
• Pdf URL: https://arxiv.org/pdf/2407.13297
• Copy Paste: [[2407.13297]] SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learning(https://arxiv.org/abs/2407.13297)
• Keywords: large language model
• Abstract: Specialized lexicons are collections of words with associated constraints such as special definitions, specific roles, and intended target audiences. These constraints are necessary for content generation and documentation tasks (e.g., writing technical manuals or children's books), where the goal is to reduce the ambiguity of text content and increase its overall readability for a specific group of audience. Understanding how large language models can capture these constraints can help researchers build better, more impactful tools for wider use beyond the NLP community. Towards this end, we introduce SpeciaLex, a benchmark for evaluating a language model's ability to follow specialized lexicon-based constraints across 18 diverse subtasks with 1,285 test instances covering core tasks of Checking, Identification, Rewriting, and Open Generation. We present an empirical evaluation of 15 open and closed-source LLMs and discuss insights on how factors such as model scale, openness, setup, and recency affect performance upon evaluating with the benchmark.

Title: Robust ASR Error Correction with Conservative Data Filtering

• Authors: Takuma Udagawa, Masayuki Suzuki, Masayasu Muraoka, Gakuto Kurata
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.13300
• Pdf URL: https://arxiv.org/pdf/2407.13300
• Copy Paste: [[2407.13300]] Robust ASR Error Correction with Conservative Data Filtering(https://arxiv.org/abs/2407.13300)
• Keywords: robust, large language model
• Abstract: Error correction (EC) based on large language models is an emerging technology to enhance the performance of automatic speech recognition (ASR) systems. Generally, training data for EC are collected by automatically pairing a large set of ASR hypotheses (as sources) and their gold references (as targets). However, the quality of such pairs is not guaranteed, and we observed various types of noise which can make the EC models brittle, e.g. inducing overcorrection in out-of-domain (OOD) settings. In this work, we propose two fundamental criteria that EC training data should satisfy: namely, EC targets should (1) improve linguistic acceptability over sources and (2) be inferable from the available context (e.g. source phonemes). Through these criteria, we identify low-quality EC pairs and train the models not to make any correction in such cases, the process we refer to as conservative data filtering. In our experiments, we focus on Japanese ASR using a strong Conformer-CTC as the baseline and finetune Japanese LLMs for EC. Through our evaluation on a suite of 21 internal benchmarks, we demonstrate that our approach can significantly reduce overcorrection and improve both the accuracy and quality of ASR results in the challenging OOD settings.

Title: CoD, Towards an Interpretable Medical Agent using Chain of Diagnosis

• Authors: Junying Chen, Chi Gui, Anningzhe Gao, Ke Ji, Xidong Wang, Xiang Wan, Benyou Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13301
• Pdf URL: https://arxiv.org/pdf/2407.13301
• Copy Paste: [[2407.13301]] CoD, Towards an Interpretable Medical Agent using Chain of Diagnosis(https://arxiv.org/abs/2407.13301)
• Keywords: interpretability, large language model
• Abstract: The field of medical diagnosis has undergone a significant transformation with the advent of large language models (LLMs), yet the challenges of interpretability within these models remain largely unaddressed. This study introduces Chain-of-Diagnosis (CoD) to enhance the interpretability of LLM-based medical diagnostics. CoD transforms the diagnostic process into a diagnostic chain that mirrors a physician's thought process, providing a transparent reasoning pathway. Additionally, CoD outputs the disease confidence distribution to ensure transparency in decision-making. This interpretability makes model diagnostics controllable and aids in identifying critical symptoms for inquiry through the entropy reduction of confidences. With CoD, we developed DiagnosisGPT, capable of diagnosing 9604 diseases. Experimental results demonstrate that DiagnosisGPT outperforms other LLMs on diagnostic benchmarks. Moreover, DiagnosisGPT provides interpretability while ensuring controllability in diagnostic rigor.

Title: General Vision Encoder Features as Guidance in Medical Image Registration

• Authors: Fryderyk Kögl, Anna Reithmeir, Vasiliki Sideri-Lampretsa, Ines Machado, Rickmer Braren, Daniel Rückert, Julia A. Schnabel, Veronika A. Zimmer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13311
• Pdf URL: https://arxiv.org/pdf/2407.13311
• Copy Paste: [[2407.13311]] General Vision Encoder Features as Guidance in Medical Image Registration(https://arxiv.org/abs/2407.13311)
• Keywords: segmentation
• Abstract: General vision encoders like DINOv2 and SAM have recently transformed computer vision. Even though they are trained on natural images, such encoder models have excelled in medical imaging, e.g., in classification, segmentation, and registration. However, no in-depth comparison of different state-of-the-art general vision encoders for medical registration is available. In this work, we investigate how well general vision encoder features can be used in the dissimilarity metrics for medical image registration. We explore two encoders that were trained on natural images as well as one that was fine-tuned on medical data. We apply the features within the well-established B-spline FFD registration framework. In extensive experiments on cardiac cine MRI data, we find that using features as additional guidance for conventional metrics improves the registration quality. The code is available at this http URL.

Title: A new approach to delegate signing rights to proxy signers using isogeny-based cryptography

• Authors: Kunal Dey, Somnath Kumar, Vikas Srivastava, Sumit Kumar Debnath
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13318
• Pdf URL: https://arxiv.org/pdf/2407.13318
• Copy Paste: [[2407.13318]] A new approach to delegate signing rights to proxy signers using isogeny-based cryptography(https://arxiv.org/abs/2407.13318)
• Keywords: secure, security
• Abstract: E-governance is a two-way protocol through which one can use government services, share data and request information. It refers to the use of communication and information technologies to provide government services to public in an efficient and fast manner. In addition, any document submitted to the e-Government system must be authenticated by a government officer using a digital signature scheme. In the context of digital signatures, the proxy signature is an important cryptographic primitive that allows the original signer to delegate signing authority to another signer (proxy signer). The proxy signature has a number of important applications in the e-government system. There are now a large amount of proxy signature schemes. The security of most of them relies on the following hard problems: the discrete logarithm problem and the factorization of integers problem. However, a large-scale quantum computer can solve them in polynomial time due to Shor's algorithm. As a consequence, there is a need for a quantum computer-resistant proxy signature to secure e-governance system from quantum adversaries. In this work, we propose the first post-quantum isogeny based proxy signature scheme CSI-PS (commutative supersingular isogeny proxy signature). Our construction is proven to be uf-cma secure under the hardness of the group action inverse problem (GAIP) based on isogeny.

Title: Why do you cite? An investigation on citation intents and decision-making classification processes

• Authors: Lorenzo Paolini (Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy), Sahar Vahdati (Nature-inspired machine intelligence group, this http URL center, Technical University of Dresden, Germany Institute for Applied Computer Science, InfAI - Dresden, Germany), Angelo Di Iorio (Department of Computer Science and Engineering, University of Bologna, Bologna, Italy), Robert Wardenga (Institute for Applied Computer Science, InfAI - Dresden, Germany), Ivan Heibi (Research Centre for Open Scholarly Metadata, Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy, Digital Humanities Advanced Research Centre (/DH.arc), Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy), Silvio Peroni (Research Centre for Open Scholarly Metadata, Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy, Digital Humanities Advanced Research Centre (/DH.arc), Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13329
• Pdf URL: https://arxiv.org/pdf/2407.13329
• Copy Paste: [[2407.13329]] Why do you cite? An investigation on citation intents and decision-making classification processes(https://arxiv.org/abs/2407.13329)
• Keywords: robust, interpretability
• Abstract: Identifying the reason for which an author cites another work is essential to understand the nature of scientific contributions and to assess their impact. Citations are one of the pillars of scholarly communication and most metrics employed to analyze these conceptual links are based on quantitative observations. Behind the act of referencing another scholarly work there is a whole world of meanings that needs to be proficiently and effectively revealed. This study emphasizes the importance of trustfully classifying citation intents to provide more comprehensive and insightful analyses in research assessment. We address this task by presenting a study utilizing advanced Ensemble Strategies for Citation Intent Classification (CIC) incorporating Language Models (LMs) and employing Explainable AI (XAI) techniques to enhance the interpretability and trustworthiness of models' predictions. Our approach involves two ensemble classifiers that utilize fine-tuned SciBERT and XLNet LMs as baselines. We further demonstrate the critical role of section titles as a feature in improving models' performances. The study also introduces a web application developed with Flask and currently available at this http URL, aimed at classifying citation intents. One of our models sets as a new state-of-the-art (SOTA) with an 89.46% Macro-F1 score on the SciCite benchmark. The integration of XAI techniques provides insights into the decision-making processes, highlighting the contributions of individual words for level-0 classifications, and of individual models for the metaclassification. The findings suggest that the inclusion of section titles significantly enhances classification performances in the CIC task. Our contributions provide useful insights for developing more robust datasets and methodologies, thus fostering a deeper understanding of scholarly communication.

Title: Reconstruct the Pruned Model without Any Retraining

• Authors: Pingjie Wang, Ziqing Fan, Shengchao Hu, Zhe Chen, Yanfeng Wang, Yu Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13331
• Pdf URL: https://arxiv.org/pdf/2407.13331
• Copy Paste: [[2407.13331]] Reconstruct the Pruned Model without Any Retraining(https://arxiv.org/abs/2407.13331)
• Keywords: large language model
• Abstract: Structured pruning is a promising hardware-friendly compression technique for large language models (LLMs), which is expected to be retraining-free to avoid the enormous retraining cost. This retraining-free paradigm involves (1) pruning criteria to define the architecture and (2) distortion reconstruction to restore performance. However, existing methods often emphasize pruning criteria while using reconstruction techniques that are specific to certain modules or criteria, resulting in limited generalizability. To address this, we introduce the Linear Interpolation-based Adaptive Reconstruction (LIAR) framework, which is both efficient and effective. LIAR does not require back-propagation or retraining and is compatible with various pruning criteria and modules. By applying linear interpolation to the preserved weights, LIAR minimizes reconstruction error and effectively reconstructs the pruned output. Our evaluations on benchmarks such as GLUE, SQuAD, WikiText, and common sense reasoning show that LIAR enables a BERT model to maintain 98% accuracy even after removing 50% of its parameters and achieves top performance for LLaMA in just a few minutes.

Title: OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction

• Authors: Yini Fang, Jingling Yu, Haozheng Zhang, Ralf van der Lans, Bertram Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13335
• Pdf URL: https://arxiv.org/pdf/2407.13335
• Copy Paste: [[2407.13335]] OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction(https://arxiv.org/abs/2407.13335)
• Keywords: transformer
• Abstract: Visual search is important in our daily life. The efficient allocation of visual attention is critical to effectively complete visual search tasks. Prior research has predominantly modelled the spatial allocation of visual attention in images at the pixel level, e.g. using a saliency map. However, emerging evidence shows that visual attention is guided by objects rather than pixel intensities. This paper introduces the Object-level Attention Transformer (OAT), which predicts human scanpaths as they search for a target object within a cluttered scene of distractors. OAT uses an encoder-decoder architecture. The encoder captures information about the position and appearance of the objects within an image and about the target. The decoder predicts the gaze scanpath as a sequence of object fixations, by integrating output features from both the encoder and decoder. We also propose a new positional encoding that better reflects spatial relationships between objects. We evaluated OAT on the Amazon book cover dataset and a new dataset for visual search that we collected. OAT's predicted gaze scanpaths align more closely with human gaze patterns, compared to predictions by algorithms based on spatial attention on both established metrics and a novel behavioural-based metric. Our results demonstrate the generalization ability of OAT, as it accurately predicts human scanpaths for unseen layouts and target objects.

Title: Long-Term 3D Point Tracking By Cost Volume Fusion

• Authors: Hung Nguyen, Chanho Kim, Rigved Naukarkar, Li Fuxin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13337
• Pdf URL: https://arxiv.org/pdf/2407.13337
• Copy Paste: [[2407.13337]] Long-Term 3D Point Tracking By Cost Volume Fusion(https://arxiv.org/abs/2407.13337)
• Keywords: transformer
• Abstract: Long-term point tracking is essential to understand non-rigid motion in the physical world better. Deep learning approaches have recently been incorporated into long-term point tracking, but most prior work predominantly functions in 2D. Although these methods benefit from the well-established backbones and matching frameworks, the motions they produce do not always make sense in the 3D physical world. In this paper, we propose the first deep learning framework for long-term point tracking in 3D that generalizes to new points and videos without requiring test-time fine-tuning. Our model contains a cost volume fusion module that effectively integrates multiple past appearances and motion information via a transformer architecture, significantly enhancing overall tracking performance. In terms of 3D tracking performance, our model significantly outperforms simple scene flow chaining and previous 2D point tracking methods, even if one uses ground truth depth and camera pose to backproject 2D point tracks in a synthetic scenario.

Title: Learn to Memorize and to Forget: A Continual Learning Perspective of Dynamic SLAM

• Authors: Baicheng Li, Zike Yan, Dong Wu, Hanqing Jiang, Hongbin Zha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13338
• Pdf URL: https://arxiv.org/pdf/2407.13338
• Copy Paste: [[2407.13338]] Learn to Memorize and to Forget: A Continual Learning Perspective of Dynamic SLAM(https://arxiv.org/abs/2407.13338)
• Keywords: robust
• Abstract: Simultaneous localization and mapping (SLAM) with implicit neural representations has received extensive attention due to the expressive representation power and the innovative paradigm of continual learning. However, deploying such a system within a dynamic environment has not been well-studied. Such challenges are intractable even for conventional algorithms since observations from different views with dynamic objects involved break the geometric and photometric consistency, whereas the consistency lays the foundation for joint optimizing the camera pose and the map parameters. In this paper, we best exploit the characteristics of continual learning and propose a novel SLAM framework for dynamic environments. While past efforts have been made to avoid catastrophic forgetting by exploiting an experience replay strategy, we view forgetting as a desirable characteristic. By adaptively controlling the replayed buffer, the ambiguity caused by moving objects can be easily alleviated through forgetting. We restrain the replay of the dynamic objects by introducing a continually-learned classifier for dynamic object identification. The iterative optimization of the neural map and the classifier notably improves the robustness of the SLAM system under a dynamic environment. Experiments on challenging datasets verify the effectiveness of the proposed framework.

Title: Hybrid Deep Learning-Based for Enhanced Occlusion Segmentation in PICU Patient Monitoring

• Authors: Mario Francisco Munoz, Hoang Vu Huy, Thanh-Dung Le
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2407.13341
• Pdf URL: https://arxiv.org/pdf/2407.13341
• Copy Paste: [[2407.13341]] Hybrid Deep Learning-Based for Enhanced Occlusion Segmentation in PICU Patient Monitoring(https://arxiv.org/abs/2407.13341)
• Keywords: transformer, segmentation
• Abstract: Remote patient monitoring has emerged as a prominent non-invasive method, using digital technologies and computer vision (CV) to replace traditional invasive monitoring. While neonatal and pediatric departments embrace this approach, Pediatric Intensive Care Units (PICUs) face the challenge of occlusions hindering accurate image analysis and interpretation. \textit{Objective}: In this study, we propose a hybrid approach to effectively segment common occlusions encountered in remote monitoring applications within PICUs. Our approach centers on creating a deep-learning pipeline for limited training data scenarios. \textit{Methods}: First, a combination of the well-established Google DeepLabV3+ segmentation model with the transformer-based Segment Anything Model (SAM) is devised for occlusion segmentation mask proposal and refinement. We then train and validate this pipeline using a small dataset acquired from real-world PICU settings with a Microsoft Kinect camera, achieving an Intersection-over-Union (IoU) metric of 85\%. \textit{Results}: Both quantitative and qualitative analyses underscore the effectiveness of our proposed method. The proposed framework yields an overall classification performance with 92.5\% accuracy, 93.8\% recall, 90.3\% precision, and 92.0\% F1-score. Consequently, the proposed method consistently improves the predictions across all metrics, with an average of 2.75\% gain in performance compared to the baseline CNN-based framework. \textit{Conclusions}: Our proposed hybrid approach significantly enhances the segmentation of occlusions in remote patient monitoring within PICU settings. This advancement contributes to improving the quality of care for pediatric patients, addressing a critical need in clinical practice by ensuring more accurate and reliable remote monitoring.

Title: Learning-From-Mistakes Prompting for Indigenous Language Translation

• Authors: You-Cheng Liao, Chen-Jui Yu, Chi-Yi Lin, He-Feng Yun, Yen-Hsiang Wang, Hsiao-Min Li, Yao-Chung Fan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13343
• Pdf URL: https://arxiv.org/pdf/2407.13343
• Copy Paste: [[2407.13343]] Learning-From-Mistakes Prompting for Indigenous Language Translation(https://arxiv.org/abs/2407.13343)
• Keywords: large language model
• Abstract: Using large language models, this paper presents techniques to improve extremely low-resourced indigenous language translations. Our approaches are grounded in the use of (1) the presence of a datastore consisting of a limited number of parallel translation examples, (2) the inherent capabilities of LLMs like GPT-3.5, and (3) a word-level translation dictionary. We harness the potential of LLMs and in-context learning techniques in such a setting for using LLMs as universal translators for extremely low-resourced languages. Our methodology hinges on utilizing LLMs as language compilers for selected language pairs, hypothesizing that they could internalize syntactic structures to facilitate accurate translation. We introduce three techniques: KNNPrompting with Retrieved Prompting Context, Chain-of-Thought Prompting and Learningfrom-Mistakes Prompting, with the last method addressing past errors. The evaluation results suggest that, even with limited corpora, LLMs can effectively translate extremely low-resource languages when paired with proper prompting.

Title: EarlyMalDetect: A Novel Approach for Early Windows Malware Detection Based on Sequences of API Calls

• Authors: Pascal Maniriho, Abdun Naser Mahmood, Mohammad Jabed Morshed Chowdhury
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13355
• Pdf URL: https://arxiv.org/pdf/2407.13355
• Copy Paste: [[2407.13355]] EarlyMalDetect: A Novel Approach for Early Windows Malware Detection Based on Sequences of API Calls(https://arxiv.org/abs/2407.13355)
• Keywords: transformer, generative
• Abstract: In this work, we propose EarlyMalDetect, a novel approach for early Windows malware detection based on sequences of API calls. Our approach leverages generative transformer models and attention-guided deep recurrent neural networks to accurately identify and detect patterns of malicious behaviors in the early stage of malware execution. By analyzing the sequences of API calls invoked during execution, the proposed approach can classify executable files (programs) as malware or benign by predicting their behaviors based on a few shots (initial API calls) invoked during execution. EarlyMalDetect can predict and reveal what a malware program is going to perform on the target system before it occurs, which can help to stop it before executing its malicious payload and infecting the system. Specifically, EarlyMalDetect relies on a fine-tuned transformer model based on API calls which has the potential to predict the next API call functions to be used by a malware or benign executable program. Our extensive experimental evaluations show that the proposed approach is highly effective in predicting malware behaviors and can be used as a preventive measure against zero-day threats in Windows systems.

Title: Learning from the Web: Language Drives Weakly-Supervised Incremental Learning for Semantic Segmentation

• Authors: Chang Liu, Giulia Rizzoli, Pietro Zanuttigh, Fu Li, Yi Niu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13363
• Pdf URL: https://arxiv.org/pdf/2407.13363
• Copy Paste: [[2407.13363]] Learning from the Web: Language Drives Weakly-Supervised Incremental Learning for Semantic Segmentation(https://arxiv.org/abs/2407.13363)
• Keywords: segmentation
• Abstract: Current weakly-supervised incremental learning for semantic segmentation (WILSS) approaches only consider replacing pixel-level annotations with image-level labels, while the training images are still from well-designed datasets. In this work, we argue that widely available web images can also be considered for the learning of new classes. To achieve this, firstly we introduce a strategy to select web images which are similar to previously seen examples in the latent space using a Fourier-based domain discriminator. Then, an effective caption-driven reharsal strategy is proposed to preserve previously learnt classes. To our knowledge, this is the first work to rely solely on web images for both the learning of new concepts and the preservation of the already learned ones in WILSS. Experimental results show that the proposed approach can reach state-of-the-art performances without using manually selected and annotated data in the incremental steps.

Title: NeuroPlug: Plugging Side-Channel Leaks in NPUs using Space Filling Curves

• Authors: Nivedita Shrivastava, Smruti R. Sarangi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13383
• Pdf URL: https://arxiv.org/pdf/2407.13383
• Copy Paste: [[2407.13383]] NeuroPlug: Plugging Side-Channel Leaks in NPUs using Space Filling Curves(https://arxiv.org/abs/2407.13383)
• Keywords: security, attack
• Abstract: Securing deep neural networks (DNNs) from side-channel attacks is an important problem as of today, given the substantial investment of time and resources in acquiring the raw data and training complex models. All published countermeasures (CMs) add noise N to a signal X (parameter of interest such as the net memory traffic that is leaked). The adversary observes X+N ; we shall show that it is easy to filter this noise out using targeted measurements, statistical analyses and different kinds of reasonably-assumed side information. We present a novel CM NeuroPlug that is immune to these attack methodologies mainly because we use a different formulation CX + N . We introduce a multiplicative variable C that naturally arises from feature map compression; it plays a key role in obfuscating the parameters of interest. Our approach is based on mapping all the computations to a 1-D space filling curve and then performing a sequence of tiling, compression and binning-based obfuscation operations. We follow up with proposing a theoretical framework based on Mellin transforms that allows us to accurately quantify the size of the search space as a function of the noise we add and the side information that an adversary possesses. The security guarantees provided by NeuroPlug are validated using a battery of statistical and information theory-based tests. We also demonstrate a substantial performance enhancement of 15% compared to the closest competing work.

Title: Time Synchronization of TESLA-enabled GNSS Receivers

• Authors: Jason Anderson, Sherman Lo, Todd Walter
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13386
• Pdf URL: https://arxiv.org/pdf/2407.13386
• Copy Paste: [[2407.13386]] Time Synchronization of TESLA-enabled GNSS Receivers(https://arxiv.org/abs/2407.13386)
• Keywords: security
• Abstract: As TESLA-enabled GNSS for authenticated positioning reaches ubiquity, receivers must use an onboard, GNSS-independent clock and carefully constructed time synchronization algorithms to assert the authenticity afforded. This work provides the necessary checks and synchronization protocols needed in the broadcast-only GNSS context. We provide proof of security for each of our algorithms under a delay-capable adversary. The algorithms included herein enable a GNSS receiver to use its onboard, GNSS-independent clock to determine whether a message arrived at the correct time, to determine whether its onboard, GNSS-independent clock is safe to use and when the clock will no longer be safe in the future due to predicted clock drift, and to resynchronize its onboard, GNSS-independent clock. Each algorithm is safe to use even when an adversary induces delays within the protocol. Moreover, we discuss the implications of GNSS authentication schemes that use two simultaneous TESLA instances of different authentication cadences. To a receiver implementer or standards author, this work provides the necessary implementation algorithms to assert security and provides a comprehensive guide on why these methods are required.

Title: GeometrySticker: Enabling Ownership Claim of Recolorized Neural Radiance Fields

• Authors: Xiufeng Huang, Ka Chun Cheung, Simon See, Renjie Wan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13390
• Pdf URL: https://arxiv.org/pdf/2407.13390
• Copy Paste: [[2407.13390]] GeometrySticker: Enabling Ownership Claim of Recolorized Neural Radiance Fields(https://arxiv.org/abs/2407.13390)
• Keywords: protect, robust
• Abstract: Remarkable advancements in the recolorization of Neural Radiance Fields (NeRF) have simplified the process of modifying NeRF's color attributes. Yet, with the potential of NeRF to serve as shareable digital assets, there's a concern that malicious users might alter the color of NeRF models and falsely claim the recolorized version as their own. To safeguard against such breaches of ownership, enabling original NeRF creators to establish rights over recolorized NeRF is crucial. While approaches like CopyRNeRF have been introduced to embed binary messages into NeRF models as digital signatures for copyright protection, the process of recolorization can remove these binary messages. In our paper, we present GeometrySticker, a method for seamlessly integrating binary messages into the geometry components of radiance fields, akin to applying a sticker. GeometrySticker can embed binary messages into NeRF models while preserving the effectiveness of these messages against recolorization. Our comprehensive studies demonstrate that GeometrySticker is adaptable to prevalent NeRF architectures and maintains a commendable level of robustness against various distortions. Project page: this https URL.

Title: Lightweight Uncertainty Quantification with Simplex Semantic Segmentation for Terrain Traversability

• Authors: Judith Dijk, Gertjan Burghouts, Kapil D. Katyal, Bryanna Y. Yeh, Craig T. Knuth, Ella Fokkinga, Tejaswi Kasarla, Pascal Mettes
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13392
• Pdf URL: https://arxiv.org/pdf/2407.13392
• Copy Paste: [[2407.13392]] Lightweight Uncertainty Quantification with Simplex Semantic Segmentation for Terrain Traversability(https://arxiv.org/abs/2407.13392)
• Keywords: segmentation
• Abstract: For navigation of robots, image segmentation is an important component to determining a terrain's traversability. For safe and efficient navigation, it is key to assess the uncertainty of the predicted segments. Current uncertainty estimation methods are limited to a specific choice of model architecture, are costly in terms of training time, require large memory for inference (ensembles), or involve complex model architectures (energy-based, hyperbolic, masking). In this paper, we propose a simple, light-weight module that can be connected to any pretrained image segmentation model, regardless of its architecture, with marginal additional computation cost because it reuses the model's backbone. Our module is based on maximum separation of the segmentation classes by respective prototype vectors. This optimizes the probability that out-of-distribution segments are projected in between the prototype vectors. The uncertainty value in the classification label is obtained from the distance to the nearest prototype. We demonstrate the effectiveness of our module for terrain segmentation.

Title: Empirical Analysis of Sri Lankan Mobile Health Ecosystem: A Precursor to an Effective Stakeholder Engagement

• Authors: Kenneth Thilakarathna, Sachintha Pitigala, Jayantha Fernando, Primal Wijesekera
• Subjects: cs.CR, cs.CY, cs.HC, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.13415
• Pdf URL: https://arxiv.org/pdf/2407.13415
• Copy Paste: [[2407.13415]] Empirical Analysis of Sri Lankan Mobile Health Ecosystem: A Precursor to an Effective Stakeholder Engagement(https://arxiv.org/abs/2407.13415)
• Keywords: privacy
• Abstract: Sri Lanka recently passed its first privacy legislation covering a wide range of sectors, including health. As a precursor for effective stakeholder engagement in the health domain to understand the most effective way to implement legislation in healthcare, we have analyzed 41 popular mobile apps and web portals. We found that 78% of the tested systems have third-party domains receiving sensitive health data with minimal visibility to the consumers. We discuss how this will create potential issues in preparing for the new privacy legislation.

Title: From Words to Worlds: Compositionality for Cognitive Architectures

• Authors: Ruchira Dhar, Anders Søgaard
• Subjects: cs.CL, cs.AI, cs.LG, cs.SC
• Abstract URL: https://arxiv.org/abs/2407.13419
• Pdf URL: https://arxiv.org/pdf/2407.13419
• Copy Paste: [[2407.13419]] From Words to Worlds: Compositionality for Cognitive Architectures(https://arxiv.org/abs/2407.13419)
• Keywords: large language model
• Abstract: Large language models (LLMs) are very performant connectionist systems, but do they exhibit more compositionality? More importantly, is that part of why they perform so well? We present empirical analyses across four LLM families (12 models) and three task categories, including a novel task introduced below. Our findings reveal a nuanced relationship in learning of compositional strategies by LLMs -- while scaling enhances compositional abilities, instruction tuning often has a reverse effect. Such disparity brings forth some open issues regarding the development and improvement of large language models in alignment with human cognitive capacities.

Title: CycleMix: Mixing Source Domains for Domain Generalization in Style-Dependent Data

• Authors: Aristotelis Ballas, Christos Diou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13421
• Pdf URL: https://arxiv.org/pdf/2407.13421
• Copy Paste: [[2407.13421]] CycleMix: Mixing Source Domains for Domain Generalization in Style-Dependent Data(https://arxiv.org/abs/2407.13421)
• Keywords: robust
• Abstract: As deep learning-based systems have become an integral part of everyday life, limitations in their generalization ability have begun to emerge. Machine learning algorithms typically rely on the i.i.d. assumption, meaning that their training and validation data are expected to follow the same distribution, which does not necessarily hold in practice. In the case of image classification, one frequent reason that algorithms fail to generalize is that they rely on spurious correlations present in training data, such as associating image styles with target classes. These associations may not be present in the unseen test data, leading to significant degradation of their effectiveness. In this work, we attempt to mitigate this Domain Generalization (DG) problem by training a robust feature extractor which disregards features attributed to image-style but infers based on style-invariant image representations. To achieve this, we train CycleGAN models to learn the different styles present in the training data and randomly mix them together to create samples with novel style attributes to improve generalization. Experimental results on the PACS DG benchmark validate the proposed method.

Title: WiNet: Wavelet-based Incremental Learning for Efficient Medical Image Registration

• Authors: Xinxing Cheng, Xi Jia, Wenqi Lu, Qiufu Li, Linlin Shen, Alexander Krull, Jinming Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13426
• Pdf URL: https://arxiv.org/pdf/2407.13426
• Copy Paste: [[2407.13426]] WiNet: Wavelet-based Incremental Learning for Efficient Medical Image Registration(https://arxiv.org/abs/2407.13426)
• Keywords: explainability
• Abstract: Deep image registration has demonstrated exceptional accuracy and fast inference. Recent advances have adopted either multiple cascades or pyramid architectures to estimate dense deformation fields in a coarse-to-fine manner. However, due to the cascaded nature and repeated composition/warping operations on feature maps, these methods negatively increase memory usage during training and testing. Moreover, such approaches lack explicit constraints on the learning process of small deformations at different scales, thus lacking explainability. In this study, we introduce a model-driven WiNet that incrementally estimates scale-wise wavelet coefficients for the displacement/velocity field across various scales, utilizing the wavelet coefficients derived from the original input image pair. By exploiting the properties of the wavelet transform, these estimated coefficients facilitate the seamless reconstruction of a full-resolution displacement/velocity field via our devised inverse discrete wavelet transform (IDWT) layer. This approach avoids the complexities of cascading networks or composition operations, making our WiNet an explainable and efficient competitor with other coarse-to-fine methods. Extensive experimental results from two 3D datasets show that our WiNet is accurate and GPU efficient. The code is available at this https URL .

Title: Improving Out-of-Distribution Generalization of Trajectory Prediction for Autonomous Driving via Polynomial Representations

• Authors: Yue Yao, Shengchao Yan, Daniel Goehring, Wolfram Burgard, Joerg Reichardt
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13431
• Pdf URL: https://arxiv.org/pdf/2407.13431
• Copy Paste: [[2407.13431]] Improving Out-of-Distribution Generalization of Trajectory Prediction for Autonomous Driving via Polynomial Representations(https://arxiv.org/abs/2407.13431)
• Keywords: robust
• Abstract: Robustness against Out-of-Distribution (OoD) samples is a key performance indicator of a trajectory prediction model. However, the development and ranking of state-of-the-art (SotA) models are driven by their In-Distribution (ID) performance on individual competition datasets. We present an OoD testing protocol that homogenizes datasets and prediction tasks across two large-scale motion datasets. We introduce a novel prediction algorithm based on polynomial representations for agent trajectory and road geometry on both the input and output sides of the model. With a much smaller model size, training effort, and inference time, we reach near SotA performance for ID testing and significantly improve robustness in OoD testing. Within our OoD testing protocol, we further study two augmentation strategies of SotA models and their effects on model generalization. Highlighting the contrast between ID and OoD performance, we suggest adding OoD testing to the evaluation criteria of trajectory prediction models.

Title: FREST: Feature RESToration for Semantic Segmentation under Multiple Adverse Conditions

• Authors: Sohyun Lee, Namyup Kim, Sungyeon Kim, Suha Kwak
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13437
• Pdf URL: https://arxiv.org/pdf/2407.13437
• Copy Paste: [[2407.13437]] FREST: Feature RESToration for Semantic Segmentation under Multiple Adverse Conditions(https://arxiv.org/abs/2407.13437)
• Keywords: robust, segmentation
• Abstract: Robust semantic segmentation under adverse conditions is crucial in real-world applications. To address this challenging task in practical scenarios where labeled normal condition images are not accessible in training, we propose FREST, a novel feature restoration framework for source-free domain adaptation (SFDA) of semantic segmentation to adverse conditions. FREST alternates two steps: (1) learning the condition embedding space that only separates the condition information from the features and (2) restoring features of adverse condition images on the learned condition embedding space. By alternating these two steps, FREST gradually restores features where the effect of adverse conditions is reduced. FREST achieved a state of the art on two public benchmarks (i.e., ACDC and RobotCar) for SFDA to adverse conditions. Moreover, it shows superior generalization ability on unseen datasets.

Title: BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models

• Authors: Moon Ye-Bin, Nam Hyeon-Woo, Wonseok Choi, Tae-Hyun Oh
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13442
• Pdf URL: https://arxiv.org/pdf/2407.13442
• Copy Paste: [[2407.13442]] BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models(https://arxiv.org/abs/2407.13442)
• Keywords: large language model
• Abstract: Vision language models (VLMs) perceive the world through a combination of a visual encoder and a large language model (LLM). The visual encoder, pre-trained on large-scale vision-text datasets, provides zero-shot generalization to visual data, and the LLM endows its high reasoning ability to VLMs. It leads VLMs to achieve high performance on wide benchmarks without fine-tuning, exhibiting zero or few-shot capability. However, recent studies show that VLMs are vulnerable to hallucination. This undesirable behavior degrades reliability and credibility, thereby making users unable to fully trust the output from VLMs. To enhance trustworthiness and better tackle the hallucination of VLMs, we curate a new evaluation dataset, called the BEfore-AFter hallucination dataset (BEAF), and introduce new metrics: True Understanding (TU), IGnorance (IG), StuBbornness (SB), and InDecision (ID). Unlike prior works that focus only on constructing questions and answers, the key idea of our benchmark is to manipulate visual scene information by image editing models and to design the metrics based on scene changes. This allows us to clearly assess whether VLMs correctly understand a given scene by observing the ability to perceive changes. We also visualize image-wise object relationship by virtue of our two-axis view: vision and text. Upon evaluating VLMs with our dataset, we observed that our metrics reveal different aspects of VLM hallucination that have not been reported before. Project page: \url{this https URL}

Title: All Roads Lead to Rome? Exploring Representational Similarities Between Latent Spaces of Generative Image Models

• Authors: Charumathi Badrinath, Usha Bhalla, Alex Oesterling, Suraj Srinivas, Himabindu Lakkaraju
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13449
• Pdf URL: https://arxiv.org/pdf/2407.13449
• Copy Paste: [[2407.13449]] All Roads Lead to Rome? Exploring Representational Similarities Between Latent Spaces of Generative Image Models(https://arxiv.org/abs/2407.13449)
• Keywords: diffusion, generative
• Abstract: Do different generative image models secretly learn similar underlying representations? We investigate this by measuring the latent space similarity of four different models: VAEs, GANs, Normalizing Flows (NFs), and Diffusion Models (DMs). Our methodology involves training linear maps between frozen latent spaces to "stitch" arbitrary pairs of encoders and decoders and measuring output-based and probe-based metrics on the resulting "stitched'' models. Our main findings are that linear maps between latent spaces of performant models preserve most visual information even when latent sizes differ; for CelebA models, gender is the most similarly represented probe-able attribute. Finally we show on an NF that latent space representations converge early in training.

Title: End-To-End Clinical Trial Matching with Large Language Models

• Authors: Dyke Ferber, Lars Hilgers, Isabella C. Wiest, Marie-Elisabeth Leßmann, Jan Clusmann, Peter Neidlinger, Jiefu Zhu, Georg Wölflein, Jacqueline Lammert, Maximilian Tschochohei, Heiko Böhme, Dirk Jäger, Mihaela Aldea, Daniel Truhn, Christiane Höper, Jakob Nikolas Kather
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13463
• Pdf URL: https://arxiv.org/pdf/2407.13463
• Copy Paste: [[2407.13463]] End-To-End Clinical Trial Matching with Large Language Models(https://arxiv.org/abs/2407.13463)
• Keywords: large language model
• Abstract: Matching cancer patients to clinical trials is essential for advancing treatment and patient care. However, the inconsistent format of medical free text documents and complex trial eligibility criteria make this process extremely challenging and time-consuming for physicians. We investigated whether the entire trial matching process - from identifying relevant trials among 105,600 oncology-related clinical trials on this http URL to generating criterion-level eligibility matches - could be automated using Large Language Models (LLMs). Using GPT-4o and a set of 51 synthetic Electronic Health Records (EHRs), we demonstrate that our approach identifies relevant candidate trials in 93.3% of cases and achieves a preliminary accuracy of 88.0% when matching patient-level information at the criterion level against a baseline defined by human experts. Utilizing LLM feedback reveals that 39.3% criteria that were initially considered incorrect are either ambiguous or inaccurately annotated, leading to a total model accuracy of 92.7% after refining our human baseline. In summary, we present an end-to-end pipeline for clinical trial matching using LLMs, demonstrating high precision in screening and matching trials to individual patients, even outperforming the performance of qualified medical doctors. Our fully end-to-end pipeline can operate autonomously or with human supervision and is not restricted to oncology, offering a scalable solution for enhancing patient-trial matching in real-world settings.

Title: Attention Overflow: Language Model Input Blur during Long-Context Missing Items Recommendation

• Authors: Damien Sileo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13481
• Pdf URL: https://arxiv.org/pdf/2407.13481
• Copy Paste: [[2407.13481]] Attention Overflow: Language Model Input Blur during Long-Context Missing Items Recommendation(https://arxiv.org/abs/2407.13481)
• Keywords: large language model
• Abstract: Large language models (LLMs) can suggest missing elements from items listed in a prompt, which can be used for list completion or recommendations based on users' history. However, their performance degrades when presented with too many items, as they start to suggest items already included in the input list. This occurs at around 100 items for mid-2024 flagship LLMs. We evaluate this phenomenon on both synthetic problems (e.g., finding missing numbers in a given range of shuffled integers) and realistic movie recommendation scenarios. We refer to this issue as \textit{attention overflow}, as preventing repetition requires attending to all items simultaneously. Although iterative loops can mitigate this problem, their costs increase with the repetition rate, affecting the language models' ability to derive novelty from lengthy inputs.

Title: Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?

• Authors: Stefanos-Iordanis Papadopoulos, Christos Koutlis, Symeon Papadopoulos, Panagiotis C. Petrantonakis
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.13488
• Pdf URL: https://arxiv.org/pdf/2407.13488
• Copy Paste: [[2407.13488]] Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?(https://arxiv.org/abs/2407.13488)
• Keywords: robust, transformer, large language model
• Abstract: Out-of-context (OOC) misinformation poses a significant challenge in multimodal fact-checking, where images are paired with texts that misrepresent their original context to support false narratives. Recent research in evidence-based OOC detection has seen a trend towards increasingly complex architectures, incorporating Transformers, foundation models, and large language models. In this study, we introduce a simple yet robust baseline, which assesses MUltimodal SimilaritiEs (MUSE), specifically the similarity between image-text pairs and external image and text evidence. Our results demonstrate that MUSE, when used with conventional classifiers like Decision Tree, Random Forest, and Multilayer Perceptron, can compete with and even surpass the state-of-the-art on the NewsCLIPpings and VERITE datasets. Furthermore, integrating MUSE in our proposed "Attentive Intermediate Transformer Representations" (AITR) significantly improved performance, by 3.3% and 7.5% on NewsCLIPpings and VERITE, respectively. Nevertheless, the success of MUSE, relying on surface-level patterns and shortcuts, without examining factuality and logical inconsistencies, raises critical questions about how we define the task, construct datasets, collect external evidence and overall, how we assess progress in the field. We release our code at: this https URL

Title: Combining Constraint Programming Reasoning with Large Language Model Predictions

• Authors: Florian Régin, Elisabetta De Maria, Alexandre Bonlarron
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13490
• Pdf URL: https://arxiv.org/pdf/2407.13490
• Copy Paste: [[2407.13490]] Combining Constraint Programming Reasoning with Large Language Model Predictions(https://arxiv.org/abs/2407.13490)
• Keywords: large language model
• Abstract: Constraint Programming (CP) and Machine Learning (ML) face challenges in text generation due to CP's struggle with implementing "meaning'' and ML's difficulty with structural constraints. This paper proposes a solution by combining both approaches and embedding a Large Language Model (LLM) in CP. The LLM handles word generation and meaning, while CP manages structural constraints. This approach builds on GenCP, an improved version of On-the-fly Constraint Programming Search (OTFS) using LLM-generated domains. Compared to Beam Search (BS), a standard NLP method, this combined approach (GenCP with LLM) is faster and produces better results, ensuring all constraints are satisfied. This fusion of CP and ML presents new possibilities for enhancing text generation under constraints.

Title: Enhancing Biomedical Knowledge Discovery for Diseases: An End-To-End Open-Source Framework

• Authors: Christos Theodoropoulos, Andrei Catalin Coman, James Henderson, Marie-Francine Moens
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13492
• Pdf URL: https://arxiv.org/pdf/2407.13492
• Copy Paste: [[2407.13492]] Enhancing Biomedical Knowledge Discovery for Diseases: An End-To-End Open-Source Framework(https://arxiv.org/abs/2407.13492)
• Keywords: transformer
• Abstract: The ever-growing volume of biomedical publications creates a critical need for efficient knowledge discovery. In this context, we introduce an open-source end-to-end framework designed to construct knowledge around specific diseases directly from raw text. To facilitate research in disease-related knowledge discovery, we create two annotated datasets focused on Rett syndrome and Alzheimer's disease, enabling the identification of semantic relations between biomedical entities. Extensive benchmarking explores various ways to represent relations and entity representations, offering insights into optimal modeling strategies for semantic relation detection and highlighting language models' competence in knowledge discovery. We also conduct probing experiments using different layer representations and attention scores to explore transformers' ability to capture semantic relations.

Title: Three-State Information Hiding: Provably Secure Asymmetric Steganography

• Authors: Minhao Bai, Jinshuai Yang, Kaiyi Pang, Xu Xin, Yongfeng Huang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13499
• Pdf URL: https://arxiv.org/pdf/2407.13499
• Copy Paste: [[2407.13499]] Three-State Information Hiding: Provably Secure Asymmetric Steganography(https://arxiv.org/abs/2407.13499)
• Keywords: secure, security
• Abstract: The rise of language models has provided a fertile ground for the application of steganography. Due to their qualified output, steganographic texts become similar to human and have attracted most of the steganography researchers' attention. However, running a language model requires a strong computation platform. It limits the applicable scenario of steganography, since those electronic devices controlled by the decoder may not even equipped with a GPU. Traditional provably secure steganography methods cannot be applied to this low-resource scenario. Therefore, we aim at design a novel steganography framework that is practical in a low-resource scheme. We start from the rigorous probability analysis with the help of hypothesis testing techniques to construct an theoretical framework. Then we prove the security and robostness of our framework and point out its optimization goal. We test our theoretical framework in some famous LLMs and the results have proved its usability. There are still some practical problems and this gives the direction of future work. We hope that this work will expand the practical scope of steganography and create a new branch of steganography.

Title: FADE: A Task-Agnostic Upsampling Operator for Encoder-Decoder Architectures

• Authors: Hao Lu, Wenze Liu, Hongtao Fu, Zhiguo Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13500
• Pdf URL: https://arxiv.org/pdf/2407.13500
• Copy Paste: [[2407.13500]] FADE: A Task-Agnostic Upsampling Operator for Encoder-Decoder Architectures(https://arxiv.org/abs/2407.13500)
• Keywords: robust, segmentation
• Abstract: The goal of this work is to develop a task-agnostic feature upsampling operator for dense prediction where the operator is required to facilitate not only region-sensitive tasks like semantic segmentation but also detail-sensitive tasks such as image matting. Prior upsampling operators often can work well in either type of the tasks, but not both. We argue that task-agnostic upsampling should dynamically trade off between semantic preservation and detail delineation, instead of having a bias between the two properties. In this paper, we present FADE, a novel, plug-and-play, lightweight, and task-agnostic upsampling operator by fusing the assets of decoder and encoder features at three levels: i) considering both the encoder and decoder feature in upsampling kernel generation; ii) controlling the per-point contribution of the encoder/decoder feature in upsampling kernels with an efficient semi-shift convolutional operator; and iii) enabling the selective pass of encoder features with a decoder-dependent gating mechanism for compensating details. To improve the practicality of FADE, we additionally study parameter- and memory-efficient implementations of semi-shift convolution. We analyze the upsampling behavior of FADE on toy data and show through large-scale experiments that FADE is task-agnostic with consistent performance improvement on a number of dense prediction tasks with little extra cost. For the first time, we demonstrate robust feature upsampling on both region- and detail-sensitive tasks successfully. Code is made available at: this https URL

Title: Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks

• Authors: Samy Ateia, Udo Kruschwitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13511
• Pdf URL: https://arxiv.org/pdf/2407.13511
• Copy Paste: [[2407.13511]] Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks(https://arxiv.org/abs/2407.13511)
• Keywords: large language model
• Abstract: Commercial large language models (LLMs), like OpenAI's GPT-4 powering ChatGPT and Anthropic's Claude 3 Opus, have dominated natural language processing (NLP) benchmarks across different domains. New competing Open-Source alternatives like Mixtral 8x7B or Llama 3 have emerged and seem to be closing the gap while often offering higher throughput and being less costly to use. Open-Source LLMs can also be self-hosted, which makes them interesting for enterprise and clinical use cases where sensitive data should not be processed by third parties. We participated in the 12th BioASQ challenge, which is a retrieval augmented generation (RAG) setting, and explored the performance of current GPT models Claude 3 Opus, GPT-3.5-turbo and Mixtral 8x7b with in-context learning (zero-shot, few-shot) and QLoRa fine-tuning. We also explored how additional relevant knowledge from Wikipedia added to the context-window of the LLM might improve their performance. Mixtral 8x7b was competitive in the 10-shot setting, both with and without fine-tuning, but failed to produce usable results in the zero-shot setting. QLoRa fine-tuning and Wikipedia context did not lead to measurable performance gains. Our results indicate that the performance gap between commercial and open-source models in RAG setups exists mainly in the zero-shot setting and can be closed by simply collecting few-shot examples for domain-specific use cases. The code needed to rerun these experiments is available through GitHub.

Title: Mask2Map: Vectorized HD Map Construction Using Bird's Eye View Segmentation Masks

• Authors: Sehwan Choi, Jungho Kim, Hongjae Shin, Jun Won Choi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13517
• Pdf URL: https://arxiv.org/pdf/2407.13517
• Copy Paste: [[2407.13517]] Mask2Map: Vectorized HD Map Construction Using Bird's Eye View Segmentation Masks(https://arxiv.org/abs/2407.13517)
• Keywords: segmentation
• Abstract: In this paper, we introduce Mask2Map, a novel end-to-end online HD map construction method designed for autonomous driving applications. Our approach focuses on predicting the class and ordered point set of map instances within a scene, represented in the bird's eye view (BEV). Mask2Map consists of two primary components: the Instance-Level Mask Prediction Network (IMPNet) and the Mask-Driven Map Prediction Network (MMPNet). IMPNet generates Mask-Aware Queries and BEV Segmentation Masks to capture comprehensive semantic information globally. Subsequently, MMPNet enhances these query features using local contextual information through two submodules: the Positional Query Generator (PQG) and the Geometric Feature Extractor (GFE). PQG extracts instance-level positional queries by embedding BEV positional information into Mask-Aware Queries, while GFE utilizes BEV Segmentation Masks to generate point-level geometric features. However, we observed limited performance in Mask2Map due to inter-network inconsistency stemming from different predictions to Ground Truth (GT) matching between IMPNet and MMPNet. To tackle this challenge, we propose the Inter-network Denoising Training method, which guides the model to denoise the output affected by both noisy GT queries and perturbed GT Segmentation Masks. Our evaluation conducted on nuScenes and Argoverse2 benchmarks demonstrates that Mask2Map achieves remarkable performance improvements over previous state-of-the-art methods, with gains of 10.1% mAP and 4.1 mAP, respectively. Our code can be found at this https URL.

Title: GPSFormer: A Global Perception and Local Structure Fitting-based Transformer for Point Cloud Understanding

• Authors: Changshuo Wang, Meiqing Wu, Siew-Kei Lam, Xin Ning, Shangshu Yu, Ruiping Wang, Weijun Li, Thambipillai Srikanthan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13519
• Pdf URL: https://arxiv.org/pdf/2407.13519
• Copy Paste: [[2407.13519]] GPSFormer: A Global Perception and Local Structure Fitting-based Transformer for Point Cloud Understanding(https://arxiv.org/abs/2407.13519)
• Keywords: transformer, segmentation
• Abstract: Despite the significant advancements in pre-training methods for point cloud understanding, directly capturing intricate shape information from irregular point clouds without reliance on external data remains a formidable challenge. To address this problem, we propose GPSFormer, an innovative Global Perception and Local Structure Fitting-based Transformer, which learns detailed shape information from point clouds with remarkable precision. The core of GPSFormer is the Global Perception Module (GPM) and the Local Structure Fitting Convolution (LSFConv). Specifically, GPM utilizes Adaptive Deformable Graph Convolution (ADGConv) to identify short-range dependencies among similar features in the feature space and employs Multi-Head Attention (MHA) to learn long-range dependencies across all positions within the feature space, ultimately enabling flexible learning of contextual representations. Inspired by Taylor series, we design LSFConv, which learns both low-order fundamental and high-order refinement information from explicitly encoded local geometric structures. Integrating the GPM and LSFConv as fundamental components, we construct GPSFormer, a cutting-edge Transformer that effectively captures global and local structures of point clouds. Extensive experiments validate GPSFormer's effectiveness in three point cloud tasks: shape classification, part segmentation, and few-shot learning. The code of GPSFormer is available at \url{this https URL}.

Title: EaDeblur-GS: Event assisted 3D Deblur Reconstruction with Gaussian Splatting

• Authors: Yuchen Weng, Zhengwen Shen, Ruofan Chen, Qi Wang, Jun Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13520
• Pdf URL: https://arxiv.org/pdf/2407.13520
• Copy Paste: [[2407.13520]] EaDeblur-GS: Event assisted 3D Deblur Reconstruction with Gaussian Splatting(https://arxiv.org/abs/2407.13520)
• Keywords: robust
• Abstract: 3D deblurring reconstruction techniques have recently seen significant advancements with the development of Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). Although these techniques can recover relatively clear 3D reconstructions from blurry image inputs, they still face limitations in handling severe blurring and complex camera motion. To address these issues, we propose Event-assisted 3D Deblur Reconstruction with Gaussian Splatting (EaDeblur-GS), which integrates event camera data to enhance the robustness of 3DGS against motion blur. By employing an Adaptive Deviation Estimator (ADE) network to estimate Gaussian center deviations and using novel loss functions, EaDeblur-GS achieves sharp 3D reconstructions in real-time, demonstrating performance comparable to state-of-the-art methods.

Title: INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages

• Authors: Abhishek Kumar Singh, Rudra Murthy, Vishwajeet kumar, Jaydeep Sen, Ganesh Ramakrishnan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13522
• Pdf URL: https://arxiv.org/pdf/2407.13522
• Copy Paste: [[2407.13522]] INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages(https://arxiv.org/abs/2407.13522)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable zero-shot and few-shot capabilities in unseen tasks, including context-grounded question answering (QA) in English. However, the evaluation of LLMs' capabilities in non-English languages for context-based QA is limited by the scarcity of benchmarks in non-English languages. To address this gap, we introduce Indic-QA, the largest publicly available context-grounded question-answering dataset for 11 major Indian languages from two language families. The dataset comprises both extractive and abstractive question-answering tasks and includes existing datasets as well as English QA datasets translated into Indian languages. Additionally, we generate a synthetic dataset using the Gemini model to create question-answer pairs given a passage, which is then manually verified for quality assurance. We evaluate various multilingual Large Language Models and their instruction-fine-tuned variants on the benchmark and observe that their performance is subpar, particularly for low-resource languages. We hope that the release of this dataset will stimulate further research on the question-answering abilities of LLMs for low-resource languages.

Title: A Security Assessment tool for Quantum Threat Analysis

• Authors: Basel Halak, Cristian Sebastian Csete, Edward Joyce, Jack Papaioannou, Alexandre Pires, Jin Soma, Betul Gokkaya, Michael Murphy
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.13523
• Pdf URL: https://arxiv.org/pdf/2407.13523
• Copy Paste: [[2407.13523]] A Security Assessment tool for Quantum Threat Analysis(https://arxiv.org/abs/2407.13523)
• Keywords: secure, security, attack
• Abstract: The rapid advancement of quantum computing poses a significant threat to many current security algorithms used for secure communication, digital authentication, and information encryption. A sufficiently powerful quantum computer could potentially exploit vulnerabilities in these algorithms, rendering data in transit insecure. This threat is expected to materialize within the next 20 years. Immediate transition to quantum-resilient cryptographic schemes is crucial, primarily to mitigate store-now-decrypt-later attacks and to ensure the security of products with decade-long operational lives. This transition requires a systematic approach to identifying and upgrading vulnerable cryptographic implementations. This work developed a quantum assessment tool for organizations, providing tailored recommendations for transitioning their security protocols into a post-quantum world. The work included a systematic evaluation of the proposed solution using qualitative feedback from network administrators and cybersecurity experts. This feedback was used to refine the accuracy and usability of the assessment process. The results demonstrate its effectiveness and usefulness in helping organizations prepare for quantum computing threats. The assessment tool is publicly available at (this https URL).

Title: Enhancing Source-Free Domain Adaptive Object Detection with Low-confidence Pseudo Label Distillation

• Authors: Ilhoon Yoon, Hyeongjun Kwon, Jin Kim, Junyoung Park, Hyunsung Jang, Kwanghoon Sohn
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13524
• Pdf URL: https://arxiv.org/pdf/2407.13524
• Copy Paste: [[2407.13524]] Enhancing Source-Free Domain Adaptive Object Detection with Low-confidence Pseudo Label Distillation(https://arxiv.org/abs/2407.13524)
• Keywords: privacy
• Abstract: Source-Free domain adaptive Object Detection (SFOD) is a promising strategy for deploying trained detectors to new, unlabeled domains without accessing source data, addressing significant concerns around data privacy and efficiency. Most SFOD methods leverage a Mean-Teacher (MT) self-training paradigm relying heavily on High-confidence Pseudo Labels (HPL). However, these HPL often overlook small instances that undergo significant appearance changes with domain shifts. Additionally, HPL ignore instances with low confidence due to the scarcity of training samples, resulting in biased adaptation toward familiar instances from the source domain. To address this limitation, we introduce the Low-confidence Pseudo Label Distillation (LPLD) loss within the Mean-Teacher based SFOD framework. This novel approach is designed to leverage the proposals from Region Proposal Network (RPN), which potentially encompasses hard-to-detect objects in unfamiliar domains. Initially, we extract HPL using a standard pseudo-labeling technique and mine a set of Low-confidence Pseudo Labels (LPL) from proposals generated by RPN, leaving those that do not overlap significantly with HPL. These LPL are further refined by leveraging class-relation information and reducing the effect of inherent noise for the LPLD loss calculation. Furthermore, we use feature distance to adaptively weight the LPLD loss to focus on LPL containing a larger foreground area. Our method outperforms previous SFOD methods on four cross-domain object detection benchmarks. Extensive experiments demonstrate that our LPLD loss leads to effective adaptation by reducing false negatives and facilitating the use of domain-invariant knowledge from the source model. Code is available at this https URL.

Title: Evaluating the performance-deviation of itemKNN in RecBole and LensKit

• Authors: Michael Schmidt, Jannik Nitschke, Tim Prinz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13531
• Pdf URL: https://arxiv.org/pdf/2407.13531
• Copy Paste: [[2407.13531]] Evaluating the performance-deviation of itemKNN in RecBole and LensKit(https://arxiv.org/abs/2407.13531)
• Keywords: fair
• Abstract: This study examines the performance of item-based k-Nearest Neighbors (ItemKNN) algorithms in the RecBole and LensKit recommender system libraries. Using four data sets (Anime, Modcloth, ML-100K, and ML-1M), we assess each library's efficiency, accuracy, and scalability, focusing primarily on normalized discounted cumulative gain (nDCG). Our results show that RecBole outperforms LensKit on two of three metrics on the ML-100K data set: it achieved an 18% higher nDCG, 14% higher precision, and 35% lower recall. To ensure a fair comparison, we adjusted LensKit's nDCG calculation to match RecBole's method. This alignment made the performance more comparable, with LensKit achieving an nDCG of 0.2540 and RecBole 0.2674. Differences in similarity matrix calculations were identified as the main cause of performance deviations. After modifying LensKit to retain only the top K similar items, both libraries showed nearly identical nDCG values across all data sets. For instance, both achieved an nDCG of 0.2586 on the ML-1M data set with the same random seed. Initially, LensKit's original implementation only surpassed RecBole in the ModCloth dataset.

Title: PriPL-Tree: Accurate Range Query for Arbitrary Distribution under Local Differential Privacy

• Authors: Leixia Wang, Qingqing Ye, Haibo Hu, Xiaofeng Meng
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2407.13532
• Pdf URL: https://arxiv.org/pdf/2407.13532
• Copy Paste: [[2407.13532]] PriPL-Tree: Accurate Range Query for Arbitrary Distribution under Local Differential Privacy(https://arxiv.org/abs/2407.13532)
• Keywords: privacy
• Abstract: Answering range queries in the context of Local Differential Privacy (LDP) is a widely studied problem in Online Analytical Processing (OLAP). Existing LDP solutions all assume a uniform data distribution within each domain partition, which may not align with real-world scenarios where data distribution is varied, resulting in inaccurate estimates. To address this problem, we introduce PriPL-Tree, a novel data structure that combines hierarchical tree structures with piecewise linear (PL) functions to answer range queries for arbitrary distributions. PriPL-Tree precisely models the underlying data distribution with a few line segments, leading to more accurate results for range queries. Furthermore, we extend it to multi-dimensional cases with novel data-aware adaptive grids. These grids leverage the insights from marginal distributions obtained through PriPL-Trees to partition the grids adaptively, adapting the density of underlying distributions. Our extensive experiments on both real and synthetic datasets demonstrate the effectiveness and superiority of PriPL-Tree over state-of-the-art solutions in answering range queries across arbitrary data distributions.

Title: GlobalPointer: Large-Scale Plane Adjustment with Bi-Convex Relaxation

• Authors: Bangyan Liao, Zhenjun Zhao, Lu Chen, Haoang Li, Daniel Cremers, Peidong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13537
• Pdf URL: https://arxiv.org/pdf/2407.13537
• Copy Paste: [[2407.13537]] GlobalPointer: Large-Scale Plane Adjustment with Bi-Convex Relaxation(https://arxiv.org/abs/2407.13537)
• Keywords: robust
• Abstract: Plane adjustment (PA) is crucial for many 3D applications, involving simultaneous pose estimation and plane recovery. Despite recent advancements, it remains a challenging problem in the realm of multi-view point cloud registration. Current state-of-the-art methods can achieve globally optimal convergence only with good initialization. Furthermore, their high time complexity renders them impractical for large-scale problems. To address these challenges, we first exploit a novel optimization strategy termed \textit{Bi-Convex Relaxation}, which decouples the original problem into two simpler sub-problems, reformulates each sub-problem using a convex relaxation technique, and alternately solves each one until the original problem converges. Building on this strategy, we propose two algorithmic variants for solving the plane adjustment problem, namely \textit{GlobalPointer} and \textit{GlobalPointer++}, based on point-to-plane and plane-to-plane errors, respectively. Extensive experiments on both synthetic and real datasets demonstrate that our method can perform large-scale plane adjustment with linear time complexity, larger convergence region, and robustness to poor initialization, while achieving similar accuracy as prior methods. The code is available at \href{this https URL}{this http URL}

Title: EnergyDiff: Universal Time-Series Energy Data Generation using Diffusion Models

• Authors: Nan Lin, Peter Palensky, Pedro P. Vergara
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2407.13538
• Pdf URL: https://arxiv.org/pdf/2407.13538
• Copy Paste: [[2407.13538]] EnergyDiff: Universal Time-Series Energy Data Generation using Diffusion Models(https://arxiv.org/abs/2407.13538)
• Keywords: privacy, diffusion, transformer, generative
• Abstract: High-resolution time series data are crucial for operation and planning in energy systems such as electrical power systems and heating systems. However, due to data collection costs and privacy concerns, such data is often unavailable or insufficient for downstream tasks. Data synthesis is a potential solution for this data scarcity. With the recent development of generative AI, we propose EnergyDiff, a universal data generation framework for energy time series data. EnergyDiff builds on state-of-the-art denoising diffusion probabilistic models, utilizing a proposed denoising network dedicated to high-resolution time series data and introducing a novel Marginal Calibration technique. Our extensive experimental results demonstrate that EnergyDiff achieves significant improvement in capturing temporal dependencies and marginal distributions compared to baselines, particularly at the 1-minute resolution. Additionally, EnergyDiff consistently generates high-quality time series data across diverse energy domains, time resolutions, and at both customer and transformer levels with reduced computational need.

Title: On the Discriminability of Self-Supervised Representation Learning

• Authors: Zeen Song, Wenwen Qiang, Changwen Zheng, Fuchun Sun, Hui Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13541
• Pdf URL: https://arxiv.org/pdf/2407.13541
• Copy Paste: [[2407.13541]] On the Discriminability of Self-Supervised Representation Learning(https://arxiv.org/abs/2407.13541)
• Keywords: robust
• Abstract: Self-supervised learning (SSL) has recently achieved significant success in downstream visual tasks. However, a notable gap still exists between SSL and supervised learning (SL), especially in complex downstream tasks. In this paper, we show that the features learned by SSL methods suffer from the crowding problem, where features of different classes are not distinctly separated, and features within the same class exhibit large intra-class variance. In contrast, SL ensures a clear separation between classes. We analyze this phenomenon and conclude that SSL objectives do not constrain the relationships between different samples and their augmentations. Our theoretical analysis delves into how SSL objectives fail to enforce the necessary constraints between samples and their augmentations, leading to poor performance in complex tasks. We provide a theoretical framework showing that the performance gap between SSL and SL mainly stems from the inability of SSL methods to capture the aggregation of similar augmentations and the separation of dissimilar augmentations. To address this issue, we propose a learnable regulator called Dynamic Semantic Adjuster (DSA). DSA aggregates and separates samples in the feature space while being robust to outliers. Through extensive empirical evaluations on multiple benchmark datasets, we demonstrate the superiority of DSA in enhancing feature aggregation and separation, ultimately closing the performance gap between SSL and SL.

Title: SAM-Driven Weakly Supervised Nodule Segmentation with Uncertainty-Aware Cross Teaching

• Authors: Xingyue Zhao, Peiqi Li, Xiangde Luo, Meng Yang, Shi Chang, Zhongyu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13553
• Pdf URL: https://arxiv.org/pdf/2407.13553
• Copy Paste: [[2407.13553]] SAM-Driven Weakly Supervised Nodule Segmentation with Uncertainty-Aware Cross Teaching(https://arxiv.org/abs/2407.13553)
• Keywords: segmentation
• Abstract: Automated nodule segmentation is essential for computer-assisted diagnosis in ultrasound images. Nevertheless, most existing methods depend on precise pixel-level annotations by medical professionals, a process that is both costly and labor-intensive. Recently, segmentation foundation models like SAM have shown impressive generalizability on natural images, suggesting their potential as pseudo-labelers. However, accurate prompts remain crucial for their success in medical images. In this work, we devise a novel weakly supervised framework that effectively utilizes the segmentation foundation model to generate pseudo-labels from aspect ration annotations for automatic nodule segmentation. Specifically, we develop three types of bounding box prompts based on scalable shape priors, followed by an adaptive pseudo-label selection module to fully exploit the prediction capabilities of the foundation model for nodules. We also present a SAM-driven uncertainty-aware cross-teaching strategy. This approach integrates SAM-based uncertainty estimation and label-space perturbations into cross-teaching to mitigate the impact of pseudo-label inaccuracies on model training. Extensive experiments on two clinically collected ultrasound datasets demonstrate the superior performance of our proposed method.

Title: Research on Tibetan Tourism Viewpoints information generation system based on LLM

• Authors: Jinhu Qi, Shuai Yan, Wentao Zhang, Yibo Zhang, Zirui Liu, Ke Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13561
• Pdf URL: https://arxiv.org/pdf/2407.13561
• Copy Paste: [[2407.13561]] Research on Tibetan Tourism Viewpoints information generation system based on LLM(https://arxiv.org/abs/2407.13561)
• Keywords: generative, large language model
• Abstract: Tibet, ensconced within China's territorial expanse, is distinguished by its labyrinthine and heterogeneous topography, a testament to its profound historical heritage, and the cradle of a unique religious ethos. The very essence of these attributes, however, has impeded the advancement of Tibet's tourism service infrastructure, rendering existing smart tourism services inadequate for the region's visitors. This study delves into the ramifications of informational disparities at tourist sites on Tibetan tourism and addresses the challenge of establishing the Large Language Model (LLM) evaluation criteria. It introduces an innovative approach, the DualGen Bridge AI system, employing supervised fine-tuning techniques to bolster model functionality and enhance optimization processes. Furthermore, it pioneers a multi-structured generative results assessment framework. Empirical validation confirms the efficacy of this framework. The study also explores the application of the supervised fine-tuning method within the proprietary DualGen Bridge AI, aimed at refining the generation of tourist site information. The study's findings offer valuable insights for optimizing system performance and provide support and inspiration for the application of LLM technology in Tibet's tourism services and beyond, potentially revolutionizing the smart tourism industry with advanced, tailored information generation capabilities.

Title: dzFinNlp at AraFinNLP: Improving Intent Detection in Financial Conversational Agents

• Authors: Mohamed Lichouri, Khaled Lounnas, Mohamed Zakaria Amziane
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13565
• Pdf URL: https://arxiv.org/pdf/2407.13565
• Copy Paste: [[2407.13565]] dzFinNlp at AraFinNLP: Improving Intent Detection in Financial Conversational Agents(https://arxiv.org/abs/2407.13565)
• Keywords: transformer
• Abstract: In this paper, we present our dzFinNlp team's contribution for intent detection in financial conversational agents, as part of the AraFinNLP shared task. We experimented with various models and feature configurations, including traditional machine learning methods like LinearSVC with TF-IDF, as well as deep learning models like Long Short-Term Memory (LSTM). Additionally, we explored the use of transformer-based models for this task. Our experiments show promising results, with our best model achieving a micro F1-score of 93.02% and 67.21% on the ArBanking77 dataset, in the development and test sets, respectively.

Title: SecScale: A Scalable and Secure Trusted Execution Environment for Servers

• Authors: Ani Sunny, Nivedita Shrivastava, Smruti R. Sarangi
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2407.13572
• Pdf URL: https://arxiv.org/pdf/2407.13572
• Copy Paste: [[2407.13572]] SecScale: A Scalable and Secure Trusted Execution Environment for Servers(https://arxiv.org/abs/2407.13572)
• Keywords: secure, security, protect, attack
• Abstract: Trusted execution environments (TEEs) are an integral part of modern secure processors. They ensure that their application and code pages are confidential, tamper proof and immune to diverse types of attacks. In 2021, Intel suddenly announced its plans to deprecate its most trustworthy enclave, SGX, on its 11th and 12th generation processors. The reasons stemmed from the fact that it was difficult to scale the enclaves (sandboxes) beyond 256 MB as the hardware overheads outweighed the benefits. Competing solutions by Intel and other vendors are much more scalable, but do not provide many key security guarantees that SGX used to provide notably replay attack protection. In the last three years, no proposal from industry or academia has been able to provide both scalability (with a modest slowdown) as well as replay-protection on generic hardware (to the best of our knowledge). We solve this problem by proposing SecScale that uses some new ideas centered around speculative execution (read first, verify later), creating a forest of MACs (instead of a tree of counters) and providing complete memory encryption (no generic unsecure regions). We show that we are 10% faster than the nearest competing alternative.

Title: Large Language Models as Reliable Knowledge Bases?

• Authors: Danna Zheng, Mirella Lapata, Jeff Z. Pan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13578
• Pdf URL: https://arxiv.org/pdf/2407.13578
• Copy Paste: [[2407.13578]] Large Language Models as Reliable Knowledge Bases?(https://arxiv.org/abs/2407.13578)
• Keywords: large language model
• Abstract: The NLP community has recently shown a growing interest in leveraging Large Language Models (LLMs) for knowledge-intensive tasks, viewing LLMs as potential knowledge bases (KBs). However, the reliability and extent to which LLMs can function as KBs remain underexplored. While previous studies suggest LLMs can encode knowledge within their parameters, the amount of parametric knowledge alone is not sufficient to evaluate their effectiveness as KBs. This study defines criteria that a reliable LLM-as-KB should meet, focusing on factuality and consistency, and covering both seen and unseen knowledge. We develop several metrics based on these criteria and use them to evaluate 26 popular LLMs, while providing a comprehensive analysis of the effects of model size, instruction tuning, and in-context learning (ICL). Our results paint a worrying picture. Even a high-performant model like GPT-3.5-turbo is not factual or consistent, and strategies like ICL and fine-tuning are unsuccessful at making LLMs better KBs.

Title: Robust Calibration of Large Vision-Language Adapters

• Authors: Balamurali Murugesan, Julio Silva-Rodriguez, Ismail Ben Ayed, Jose Dolz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13588
• Pdf URL: https://arxiv.org/pdf/2407.13588
• Copy Paste: [[2407.13588]] Robust Calibration of Large Vision-Language Adapters(https://arxiv.org/abs/2407.13588)
• Keywords: robust
• Abstract: This paper addresses the critical issue of miscalibration in CLIP-based model adaptation, particularly in the challenging scenario of out-of-distribution (OOD) samples, which has been overlooked in the existing literature on CLIP adaptation. We empirically demonstrate that popular CLIP adaptation approaches, such as Adapters, Prompt Learning, and Test-Time Adaptation, substantially degrade the calibration capabilities of the zero-shot baseline in the presence of distributional drift. We identify the increase in logit ranges as the underlying cause of miscalibration of CLIP adaptation methods, contrasting with previous work on calibrating fully-supervised models. Motivated by these observations, we present a simple and model-agnostic solution to mitigate miscalibration, by scaling the logit range of each sample to its zero-shot prediction logits. We explore three different alternatives to achieve this, which can be either integrated during adaptation or directly used at inference time. Comprehensive experiments on popular OOD classification benchmarks demonstrate the effectiveness of the proposed approaches in mitigating miscalibration while maintaining discriminative performance, whose improvements are consistent across the three families of these increasingly popular approaches. The code is publicly available at: this https URL

Title: Mechanistically Interpreting a Transformer-based 2-SAT Solver: An Axiomatic Approach

• Authors: Nils Palumbo, Ravi Mangal, Zifan Wang, Saranya Vijayakumar, Corina S. Pasareanu, Somesh Jha
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13594
• Pdf URL: https://arxiv.org/pdf/2407.13594
• Copy Paste: [[2407.13594]] Mechanistically Interpreting a Transformer-based 2-SAT Solver: An Axiomatic Approach(https://arxiv.org/abs/2407.13594)
• Keywords: interpretability, transformer
• Abstract: Mechanistic interpretability aims to reverse engineer the computation performed by a neural network in terms of its internal components. Although there is a growing body of research on mechanistic interpretation of neural networks, the notion of a mechanistic interpretation itself is often ad-hoc. Inspired by the notion of abstract interpretation from the program analysis literature that aims to develop approximate semantics for programs, we give a set of axioms that formally characterize a mechanistic interpretation as a description that approximately captures the semantics of the neural network under analysis in a compositional manner. We use these axioms to guide the mechanistic interpretability analysis of a Transformer-based model trained to solve the well-known 2-SAT problem. We are able to reverse engineer the algorithm learned by the model -- the model first parses the input formulas and then evaluates their satisfiability via enumeration of different possible valuations of the Boolean input variables. We also present evidence to support that the mechanistic interpretation of the analyzed model indeed satisfies the stated axioms.

Title: EarthMarker: A Visual Prompt Learning Framework for Region-level and Point-level Remote Sensing Imagery Comprehension

• Authors: Wei Zhang, Miaoxin Cai, Tong Zhang, Yin Zhuang, Xuerui Mao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13596
• Pdf URL: https://arxiv.org/pdf/2407.13596
• Copy Paste: [[2407.13596]] EarthMarker: A Visual Prompt Learning Framework for Region-level and Point-level Remote Sensing Imagery Comprehension(https://arxiv.org/abs/2407.13596)
• Keywords: large language model
• Abstract: Recent advances in visual prompting in the natural image area have allowed users to interact with artificial intelligence (AI) tools through various visual marks such as box, point, and free-form shapes. However, due to the significant difference between the natural and remote sensing (RS) images, existing visual prompting models face challenges in RS scenarios. Moreover, RS MLLMs mainly focus on interpreting image-level RS data and only support interaction with language instruction, restricting flexibility applications in the real world. To address those limitations, a novel visual prompting model named EarthMarker is proposed, which excels in image-level, region-level, and point-level RS imagery interpretation. Specifically, the visual prompts alongside images and text instruction input into the large language model (LLM), adapt models toward specific predictions and tasks. Subsequently, a sharing visual encoding method is introduced to refine multi-scale image features and visual prompt information uniformly. Furthermore, to endow the EarthMarker with versatile multi-granularity visual perception abilities, the cross-domain phased learning strategy is developed, and the disjoint parameters are optimized in a lightweight manner by leveraging both the natural and RS domain-specific knowledge. In addition, to tackle the lack of RS visual prompting data, a dataset named RSVP featuring multi-modal fine-grained visual prompting instruction is constructed. Extensive experiments are conducted to demonstrate the proposed EarthMarker's competitive performance, representing a significant advance in multi-granularity RS imagery interpretation under the visual prompting learning framework.

Title: PLANTS: A Novel Problem and Dataset for Summarization of Planning-Like (PL) Tasks

• Authors: Vishal Pallagani, Biplav Srivastava, Nitin Gupta
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13597
• Pdf URL: https://arxiv.org/pdf/2407.13597
• Copy Paste: [[2407.13597]] PLANTS: A Novel Problem and Dataset for Summarization of Planning-Like (PL) Tasks(https://arxiv.org/abs/2407.13597)
• Keywords: large language model
• Abstract: Text summarization is a well-studied problem that deals with deriving insights from unstructured text consumed by humans, and it has found extensive business applications. However, many real-life tasks involve generating a series of actions to achieve specific goals, such as workflows, recipes, dialogs, and travel plans. We refer to them as planning-like (PL) tasks noting that the main commonality they share is control flow information. which may be partially specified. Their structure presents an opportunity to create more practical summaries to help users make quick decisions. We investigate this observation by introducing a novel plan summarization problem, presenting a dataset, and providing a baseline method for generating PL summaries. Using quantitative metrics and qualitative user studies to establish baselines, we evaluate the plan summaries from our method and large language models. We believe the novel problem and dataset can reinvigorate research in summarization, which some consider as a solved problem.

Title: dzStance at StanceEval2024: Arabic Stance Detection based on Sentence Transformers

• Authors: Mohamed Lichouri, Khaled Lounnas, Khelil Rafik Ouaras, Mohamed Abi, Anis Guechtouli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13603
• Pdf URL: https://arxiv.org/pdf/2407.13603
• Copy Paste: [[2407.13603]] dzStance at StanceEval2024: Arabic Stance Detection based on Sentence Transformers(https://arxiv.org/abs/2407.13603)
• Keywords: transformer
• Abstract: This study compares Term Frequency-Inverse Document Frequency (TF-IDF) features with Sentence Transformers for detecting writers' stances--favorable, opposing, or neutral--towards three significant topics: COVID-19 vaccine, digital transformation, and women empowerment. Through empirical evaluation, we demonstrate that Sentence Transformers outperform TF-IDF features across various experimental setups. Our team, dzStance, participated in a stance detection competition, achieving the 13th position (74.91%) among 15 teams in Women Empowerment, 10th (73.43%) in COVID Vaccine, and 12th (66.97%) in Digital Transformation. Overall, our team's performance ranked 13th (71.77%) among all participants. Notably, our approach achieved promising F1-scores, highlighting its effectiveness in identifying writers' stances on diverse topics. These results underscore the potential of Sentence Transformers to enhance stance detection models for addressing critical societal issues.

Title: Physics-guided Active Sample Reweighting for Urban Flow Prediction

• Authors: Wei Jiang, Tong Chen, Guanhua Ye, Wentao Zhang, Lizhen Cui, Zi Huang, Hongzhi Yin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13605
• Pdf URL: https://arxiv.org/pdf/2407.13605
• Copy Paste: [[2407.13605]] Physics-guided Active Sample Reweighting for Urban Flow Prediction(https://arxiv.org/abs/2407.13605)
• Keywords: robust, interpretability
• Abstract: Urban flow prediction is a spatio-temporal modeling task that estimates the throughput of transportation services like buses, taxis, and ride-sharing, where data-driven models have become the most popular solution in the past decade. Meanwhile, the implicitly learned mapping between historical observations to the prediction targets tend to over-simplify the dynamics of real-world urban flows, leading to suboptimal predictions. Some recent spatio-temporal prediction solutions bring remedies with the notion of physics-guided machine learning (PGML), which describes spatio-temporal data with nuanced and principled physics laws, thus enhancing both the prediction accuracy and interpretability. However, these spatio-temporal PGML methods are built upon a strong assumption that the observed data fully conforms to the differential equations that define the physical system, which can quickly become ill-posed in urban flow prediction tasks. The observed urban flow data, especially when sliced into time-dependent snapshots to facilitate predictions, is typically incomplete and sparse, and prone to inherent noise incurred in the collection process. As a result, such physical inconsistency between the data and PGML model significantly limits the predictive power and robustness of the solution. Moreover, due to the interval-based predictions and intermittent nature of data filing in many transportation services, the instantaneous dynamics of urban flows can hardly be captured, rendering differential equation-based continuous modeling a loose fit for this setting. To overcome the challenges, we develop a discretized physics-guided network (PN), and propose a data-aware framework Physics-guided Active Sample Reweighting (P-GASR) to enhance PN. Experimental results in four real-world datasets demonstrate that our method achieves state-of-the-art performance with a demonstrable improvement in robustness.

Title: Training-free Composite Scene Generation for Layout-to-Image Synthesis

• Authors: Jiaqi Liu, Tao Huang, Chang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13609
• Pdf URL: https://arxiv.org/pdf/2407.13609
• Copy Paste: [[2407.13609]] Training-free Composite Scene Generation for Layout-to-Image Synthesis(https://arxiv.org/abs/2407.13609)
• Keywords: diffusion
• Abstract: Recent breakthroughs in text-to-image diffusion models have significantly advanced the generation of high-fidelity, photo-realistic images from textual descriptions. Yet, these models often struggle with interpreting spatial arrangements from text, hindering their ability to produce images with precise spatial configurations. To bridge this gap, layout-to-image generation has emerged as a promising direction. However, training-based approaches are limited by the need for extensively annotated datasets, leading to high data acquisition costs and a constrained conceptual scope. Conversely, training-free methods face challenges in accurately locating and generating semantically similar objects within complex compositions. This paper introduces a novel training-free approach designed to overcome adversarial semantic intersections during the diffusion conditioning phase. By refining intra-token loss with selective sampling and enhancing the diffusion process with attention redistribution, we propose two innovative constraints: 1) an inter-token constraint that resolves token conflicts to ensure accurate concept synthesis; and 2) a self-attention constraint that improves pixel-to-pixel relationships. Our evaluations confirm the effectiveness of leveraging layout information for guiding the diffusion process, generating content-rich images with enhanced fidelity and complexity. Code is available at this https URL.

Title: Differential Privacy Mechanisms in Neural Tangent Kernel Regression

• Authors: Jiuxiang Gu, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13621
• Pdf URL: https://arxiv.org/pdf/2407.13621
• Copy Paste: [[2407.13621]] Differential Privacy Mechanisms in Neural Tangent Kernel Regression(https://arxiv.org/abs/2407.13621)
• Keywords: privacy
• Abstract: Training data privacy is a fundamental problem in modern Artificial Intelligence (AI) applications, such as face recognition, recommendation systems, language generation, and many others, as it may contain sensitive user information related to legal issues. To fundamentally understand how privacy mechanisms work in AI applications, we study differential privacy (DP) in the Neural Tangent Kernel (NTK) regression setting, where DP is one of the most powerful tools for measuring privacy under statistical learning, and NTK is one of the most popular analysis frameworks for studying the learning mechanisms of deep neural networks. In our work, we can show provable guarantees for both differential privacy and test accuracy of our NTK regression. Furthermore, we conduct experiments on the basic image classification dataset CIFAR10 to demonstrate that NTK regression can preserve good accuracy under a modest privacy budget, supporting the validity of our analysis. To our knowledge, this is the first work to provide a DP guarantee for NTK regression.

Title: Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

• Authors: Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, Ngai Wong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13623
• Pdf URL: https://arxiv.org/pdf/2407.13623
• Copy Paste: [[2407.13623]] Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies(https://arxiv.org/abs/2407.13623)
• Keywords: large language model
• Abstract: Research on scaling large language models (LLMs) has primarily focused on model parameters and training data size, overlooking the role of vocabulary size. % Intuitively, larger vocabularies enable more efficient tokenization by representing sentences with fewer tokens, but they also increase the risk of under-fitting representations for rare tokens. We investigate how vocabulary size impacts LLM scaling laws by training models ranging from 33M to 3B parameters on up to 500B characters with various vocabulary configurations. We propose three complementary approaches for predicting the compute-optimal vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit of the loss function. Our approaches converge on the same result that the optimal vocabulary size depends on the available compute budget and that larger models deserve larger vocabularies. However, most LLMs use too small vocabulary sizes. For example, we predict that the optimal vocabulary size of Llama2-70B should have been at least 216K, 7 times larger than its vocabulary of 32K. We validate our predictions empirically by training models with 3B parameters across different FLOPs budgets. Adopting our predicted optimal vocabulary size consistently improves downstream performance over commonly used vocabulary sizes. By increasing the vocabulary size from the conventional 32K to 43K, we improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21 FLOPs. Our work emphasizes the necessity of jointly considering model parameters and vocabulary size for efficient scaling.

Title: A Comparative Study on Automatic Coding of Medical Letters with Explainability

• Authors: Jamie Glen, Lifeng Han, Paul Rayson, Goran Nenadic
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13638
• Pdf URL: https://arxiv.org/pdf/2407.13638
• Copy Paste: [[2407.13638]] A Comparative Study on Automatic Coding of Medical Letters with Explainability(https://arxiv.org/abs/2407.13638)
• Keywords: explainability
• Abstract: This study aims to explore the implementation of Natural Language Processing (NLP) and machine learning (ML) techniques to automate the coding of medical letters with visualised explainability and light-weighted local computer settings. Currently in clinical settings, coding is a manual process that involves assigning codes to each condition, procedure, and medication in a patient's paperwork (e.g., 56265001 heart disease using SNOMED CT code). There are preliminary research on automatic coding in this field using state-of-the-art ML models; however, due to the complexity and size of the models, the real-world deployment is not achieved. To further facilitate the possibility of automatic coding practice, we explore some solutions in a local computer setting; in addition, we explore the function of explainability for transparency of AI models. We used the publicly available MIMIC-III database and the HAN/HLAN network models for ICD code prediction purposes. We also experimented with the mapping between ICD and SNOMED CT knowledge bases. In our experiments, the models provided useful information for 97.98\% of codes. The result of this investigation can shed some light on implementing automatic clinical coding in practice, such as in hospital settings, on the local computers used by clinicians , project page \url{this https URL}.

Title: Beyond Augmentation: Empowering Model Robustness under Extreme Capture Environments

• Authors: Yunpeng Gong, Yongjie Hou, Chuangliang Zhang, Min Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13640
• Pdf URL: https://arxiv.org/pdf/2407.13640
• Copy Paste: [[2407.13640]] Beyond Augmentation: Empowering Model Robustness under Extreme Capture Environments(https://arxiv.org/abs/2407.13640)
• Keywords: robust
• Abstract: Person Re-identification (re-ID) in computer vision aims to recognize and track individuals across different cameras. While previous research has mainly focused on challenges like pose variations and lighting changes, the impact of extreme capture conditions is often not adequately addressed. These extreme conditions, including varied lighting, camera styles, angles, and image distortions, can significantly affect data distribution and re-ID accuracy. Current research typically improves model generalization under normal shooting conditions through data augmentation techniques such as adjusting brightness and contrast. However, these methods pay less attention to the robustness of models under extreme shooting conditions. To tackle this, we propose a multi-mode synchronization learning (MMSL) strategy . This approach involves dividing images into grids, randomly selecting grid blocks, and applying data augmentation methods like contrast and brightness adjustments. This process introduces diverse transformations without altering the original image structure, helping the model adapt to extreme variations. This method improves the model's generalization under extreme conditions and enables learning diverse features, thus better addressing the challenges in re-ID. Extensive experiments on a simulated test set under extreme conditions have demonstrated the effectiveness of our method. This approach is crucial for enhancing model robustness and adaptability in real-world scenarios, supporting the future development of person re-identification technology.

Title: Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

• Authors: Xiaoyu Zhu, Hao Zhou, Pengfei Xing, Long Zhao, Hao Xu, Junwei Liang, Alexander Hauptmann, Ting Liu, Andrew Gallagher
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13642
• Pdf URL: https://arxiv.org/pdf/2407.13642
• Copy Paste: [[2407.13642]] Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models(https://arxiv.org/abs/2407.13642)
• Keywords: diffusion, generative, segmentation
• Abstract: In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%.

Title: Beyond Dropout: Robust Convolutional Neural Networks Based on Local Feature Masking

• Authors: Yunpeng Gong, Chuangliang Zhang, Yongjie Hou, Lifei Chen, Min Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13646
• Pdf URL: https://arxiv.org/pdf/2407.13646
• Copy Paste: [[2407.13646]] Beyond Dropout: Robust Convolutional Neural Networks Based on Local Feature Masking(https://arxiv.org/abs/2407.13646)
• Keywords: attack, robust
• Abstract: In the contemporary of deep learning, where models often grapple with the challenge of simultaneously achieving robustness against adversarial attacks and strong generalization capabilities, this study introduces an innovative Local Feature Masking (LFM) strategy aimed at fortifying the performance of Convolutional Neural Networks (CNNs) on both fronts. During the training phase, we strategically incorporate random feature masking in the shallow layers of CNNs, effectively alleviating overfitting issues, thereby enhancing the model's generalization ability and bolstering its resilience to adversarial attacks. LFM compels the network to adapt by leveraging remaining features to compensate for the absence of certain semantic features, nurturing a more elastic feature learning mechanism. The efficacy of LFM is substantiated through a series of quantitative and qualitative assessments, collectively showcasing a consistent and significant improvement in CNN's generalization ability and resistance against adversarial attacks--a phenomenon not observed in current and prior methodologies. The seamless integration of LFM into established CNN frameworks underscores its potential to advance both generalization and adversarial robustness within the deep learning paradigm. Through comprehensive experiments, including robust person re-identification baseline generalization experiments and adversarial attack experiments, we demonstrate the substantial enhancements offered by LFM in addressing the aforementioned challenges. This contribution represents a noteworthy stride in advancing robust neural network architectures.

Title: Weak-to-Strong Reasoning

• Authors: Yuqing Yang, Yan Ma, Pengfei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13647
• Pdf URL: https://arxiv.org/pdf/2407.13647
• Copy Paste: [[2407.13647]] Weak-to-Strong Reasoning(https://arxiv.org/abs/2407.13647)
• Keywords: large language model
• Abstract: When large language models (LLMs) exceed human-level capabilities, it becomes increasingly challenging to provide full-scale and accurate supervisions for these models. Weak-to-strong learning, which leverages a less capable model to unlock the latent abilities of a stronger model, proves valuable in this context. Yet, the efficacy of this approach for complex reasoning tasks is still untested. Furthermore, tackling reasoning tasks under the weak-to-strong setting currently lacks efficient methods to avoid blindly imitating the weak supervisor including its errors. In this paper, we introduce a progressive learning framework that enables the strong model to autonomously refine its training data, without requiring input from either a more advanced model or human-annotated data. This framework begins with supervised fine-tuning on a selective small but high-quality dataset, followed by preference optimization on contrastive samples identified by the strong model itself. Extensive experiments on the GSM8K and MATH datasets demonstrate that our method significantly enhances the reasoning capabilities of Llama2-70b using three separate weak models. This method is further validated in a forward-looking experimental setup, where Llama3-8b-instruct effectively supervises Llama3-70b on the highly challenging OlympicArena dataset. This work paves the way for a more scalable and sophisticated strategy to enhance AI reasoning powers. All relevant code and resources are available in \url{this https URL}.

Title: MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis

• Authors: Ziming Zhong, Yanxu Xu, Jing Li, Jiale Xu, Zhengxin Li, Chaohui Yu, Shenghua Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13675
• Pdf URL: https://arxiv.org/pdf/2407.13675
• Copy Paste: [[2407.13675]] MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis(https://arxiv.org/abs/2407.13675)
• Keywords: diffusion, segmentation
• Abstract: We present MeshSegmenter, a simple yet effective framework designed for zero-shot 3D semantic segmentation. This model successfully extends the powerful capabilities of 2D segmentation models to 3D meshes, delivering accurate 3D segmentation across diverse meshes and segment descriptions. Specifically, our model leverages the Segment Anything Model (SAM) model to segment the target regions from images rendered from the 3D shape. In light of the importance of the texture for segmentation, we also leverage the pretrained stable diffusion model to generate images with textures from 3D shape, and leverage SAM to segment the target regions from images with textures. Textures supplement the shape for segmentation and facilitate accurate 3D segmentation even in geometrically non-prominent areas, such as segmenting a car door within a car mesh. To achieve the 3D segments, we render 2D images from different views and conduct segmentation for both textured and untextured images. Lastly, we develop a multi-view revoting scheme that integrates 2D segmentation results and confidence scores from various views onto the 3D mesh, ensuring the 3D consistency of segmentation results and eliminating inaccuracies from specific perspectives. Through these innovations, MeshSegmenter offers stable and reliable 3D segmentation results both quantitatively and qualitatively, highlighting its potential as a transformative tool in the field of 3D zero-shot segmentation. The code is available at \url{this https URL}.

Title: PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers

• Authors: Songlin Li, Despoina Paschalidou, Leonidas Guibas
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13677
• Pdf URL: https://arxiv.org/pdf/2407.13677
• Copy Paste: [[2407.13677]] PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers(https://arxiv.org/abs/2407.13677)
• Keywords: transformer, generative
• Abstract: The increased demand for tools that automate the 3D content creation process led to tremendous progress in deep generative models that can generate diverse 3D objects of high fidelity. In this paper, we present PASTA, an autoregressive transformer architecture for generating high quality 3D shapes. PASTA comprises two main components: An autoregressive transformer that generates objects as a sequence of cuboidal primitives and a blending network, implemented with a transformer decoder that composes the sequences of cuboids and synthesizes high quality meshes for each object. Our model is trained in two stages: First we train our autoregressive generative model using only annotated cuboidal parts as supervision and next, we train our blending network using explicit 3D supervision, in the form of watertight meshes. Evaluations on various ShapeNet objects showcase the ability of our model to perform shape generation from diverse inputs \eg from scratch, from a partial object, from text and images, as well size-guided generation, by explicitly conditioning on a bounding box that defines the object's boundaries. Moreover, as our model considers the underlying part-based structure of a 3D object, we are able to select a specific part and produce shapes with meaningful variations of this part. As evidenced by our experiments, our model generates 3D shapes that are both more realistic and diverse than existing part-based and non part-based methods, while at the same time is simpler to implement and train.

Title: HPix: Generating Vector Maps from Satellite Images

• Authors: Aditya Taparia, Keshab Nath
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.13680
• Pdf URL: https://arxiv.org/pdf/2407.13680
• Copy Paste: [[2407.13680]] HPix: Generating Vector Maps from Satellite Images(https://arxiv.org/abs/2407.13680)
• Keywords: extraction, generative
• Abstract: Vector maps find widespread utility across diverse domains due to their capacity to not only store but also represent discrete data boundaries such as building footprints, disaster impact analysis, digitization, urban planning, location points, transport links, and more. Although extensive research exists on identifying building footprints and road types from satellite imagery, the generation of vector maps from such imagery remains an area with limited exploration. Furthermore, conventional map generation techniques rely on labor-intensive manual feature extraction or rule-based approaches, which impose inherent limitations. To surmount these limitations, we propose a novel method called HPix, which utilizes modified Generative Adversarial Networks (GANs) to generate vector tile map from satellite images. HPix incorporates two hierarchical frameworks: one operating at the global level and the other at the local level, resulting in a comprehensive model. Through empirical evaluations, our proposed approach showcases its effectiveness in producing highly accurate and visually captivating vector tile maps derived from satellite images. We further extend our study's application to include mapping of road intersections and building footprints cluster based on their area.

Title: DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving

• Authors: Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, Junxian He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13690
• Pdf URL: https://arxiv.org/pdf/2407.13690
• Copy Paste: [[2407.13690]] DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving(https://arxiv.org/abs/2407.13690)
• Keywords: large language model
• Abstract: Solving mathematical problems requires advanced reasoning abilities and presents notable challenges for large language models. Previous works usually synthesize data from proprietary models to augment existing datasets, followed by instruction tuning to achieve top-tier results. However, our analysis of these datasets reveals severe biases towards easy queries, with frequent failures to generate any correct response for the most challenging queries. Hypothesizing that difficult queries are crucial to learn complex reasoning, we propose Difficulty-Aware Rejection Tuning (DART), a method that allocates difficult queries more trials during the synthesis phase, enabling more extensive training on difficult samples. Utilizing DART, we have created new datasets for mathematical problem-solving that focus more on difficult queries and are substantially smaller than previous ones. Remarkably, our synthesis process solely relies on a 7B-sized open-weight model, without reliance on the commonly used proprietary GPT-4. We fine-tune various base models on our datasets ranging from 7B to 70B in size, resulting in a series of strong models called DART-MATH. In comprehensive in-domain and out-of-domain evaluation on 6 mathematical benchmarks, DART-MATH outperforms vanilla rejection tuning significantly, being superior or comparable to previous arts, despite using much smaller datasets and no proprietary models. Furthermore, our results position our synthetic datasets as the most effective and cost-efficient publicly available resources for advancing mathematical problem-solving.

Title: Prover-Verifier Games improve legibility of LLM outputs

• Authors: Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, Yuri Burda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13692
• Pdf URL: https://arxiv.org/pdf/2407.13692
• Copy Paste: [[2407.13692]] Prover-Verifier Games improve legibility of LLM outputs(https://arxiv.org/abs/2407.13692)
• Keywords: attack, robust, large language model
• Abstract: One way to increase confidence in the outputs of Large Language Models (LLMs) is to support them with reasoning that is clear and easy to check -- a property we call legibility. We study legibility in the context of solving grade-school math problems and show that optimizing chain-of-thought solutions only for answer correctness can make them less legible. To mitigate the loss in legibility, we propose a training algorithm inspired by Prover-Verifier Game from Anil et al. (2021). Our algorithm iteratively trains small verifiers to predict solution correctness, "helpful" provers to produce correct solutions that the verifier accepts, and "sneaky" provers to produce incorrect solutions that fool the verifier. We find that the helpful prover's accuracy and the verifier's robustness to adversarial attacks increase over the course of training. Furthermore, we show that legibility training transfers to time-constrained humans tasked with verifying solution correctness. Over course of LLM training human accuracy increases when checking the helpful prover's solutions, and decreases when checking the sneaky prover's solutions. Hence, training for checkability by small verifiers is a plausible technique for increasing output legibility. Our results suggest legibility training against small verifiers as a practical avenue for increasing legibility of large LLMs to humans, and thus could help with alignment of superhuman models.

Title: Benchmark Agreement Testing Done Right: A Guide for LLM Benchmark Evaluation

• Authors: Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, Elron Bandel, Eyal Shnarch, Michal Shmueli-Scheuer, Leshem Choshen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13696
• Pdf URL: https://arxiv.org/pdf/2407.13696
• Copy Paste: [[2407.13696]] Benchmark Agreement Testing Done Right: A Guide for LLM Benchmark Evaluation(https://arxiv.org/abs/2407.13696)
• Keywords: robust
• Abstract: Recent advancements in Language Models (LMs) have catalyzed the creation of multiple benchmarks, designed to assess these models' general capabilities. A crucial task, however, is assessing the validity of the benchmarks themselves. This is most commonly done via Benchmark Agreement Testing (BAT), where new benchmarks are validated against established ones using some agreement metric (e.g., rank correlation). Despite the crucial role of BAT for benchmark builders and consumers, there are no standardized procedures for such agreement testing. This deficiency can lead to invalid conclusions, fostering mistrust in benchmarks and upending the ability to properly choose the appropriate benchmark to use. By analyzing over 40 prominent benchmarks, we demonstrate how some overlooked methodological choices can significantly influence BAT results, potentially undermining the validity of conclusions. To address these inconsistencies, we propose a set of best practices for BAT and demonstrate how utilizing these methodologies greatly improves BAT robustness and validity. To foster adoption and facilitate future research,, we introduce BenchBench, a python package for BAT, and release the BenchBench-leaderboard, a meta-benchmark designed to evaluate benchmarks using their peers. Our findings underscore the necessity for standardized BAT, ensuring the robustness and validity of benchmark evaluations in the evolving landscape of language model research. BenchBench Package: this https URL Leaderboard: this https URL

Title: Cross-Task Attack: A Self-Supervision Generative Framework Based on Attention Shift

• Authors: Qingyuan Zeng, Yunpeng Gong, Min Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13700
• Pdf URL: https://arxiv.org/pdf/2407.13700
• Copy Paste: [[2407.13700]] Cross-Task Attack: A Self-Supervision Generative Framework Based on Attention Shift(https://arxiv.org/abs/2407.13700)
• Keywords: attack, robust, generative
• Abstract: Studying adversarial attacks on artificial intelligence (AI) systems helps discover model shortcomings, enabling the construction of a more robust system. Most existing adversarial attack methods only concentrate on single-task single-model or single-task cross-model scenarios, overlooking the multi-task characteristic of artificial intelligence systems. As a result, most of the existing attacks do not pose a practical threat to a comprehensive and collaborative AI system. However, implementing cross-task attacks is highly demanding and challenging due to the difficulty in obtaining the real labels of different tasks for the same picture and harmonizing the loss functions across different tasks. To address this issue, we propose a self-supervised Cross-Task Attack framework (CTA), which utilizes co-attention and anti-attention maps to generate cross-task adversarial perturbation. Specifically, the co-attention map reflects the area to which different visual task models pay attention, while the anti-attention map reflects the area that different visual task models neglect. CTA generates cross-task perturbations by shifting the attention area of samples away from the co-attention map and closer to the anti-attention map. We conduct extensive experiments on multiple vision tasks and the experimental results confirm the effectiveness of the proposed design for adversarial attacks.

Title: ANHALTEN: Cross-Lingual Transfer for German Token-Level Reference-Free Hallucination Detection

• Authors: Janek Herrlein, Chia-Chien Hung, Goran Glavaš
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13702
• Pdf URL: https://arxiv.org/pdf/2407.13702
• Copy Paste: [[2407.13702]] ANHALTEN: Cross-Lingual Transfer for German Token-Level Reference-Free Hallucination Detection(https://arxiv.org/abs/2407.13702)
• Keywords: robust
• Abstract: Research on token-level reference-free hallucination detection has predominantly focused on English, primarily due to the scarcity of robust datasets in other languages. This has hindered systematic investigations into the effectiveness of cross-lingual transfer for this important NLP application. To address this gap, we introduce ANHALTEN, a new evaluation dataset that extends the English hallucination detection dataset to German. To the best of our knowledge, this is the first work that explores cross-lingual transfer for token-level reference-free hallucination detection. ANHALTEN contains gold annotations in German that are parallel (i.e., directly comparable to the original English instances). We benchmark several prominent cross-lingual transfer approaches, demonstrating that larger context length leads to better hallucination detection in German, even without succeeding context. Importantly, we show that the sample-efficient few-shot transfer is the most effective approach in most setups. This highlights the practical benefits of minimal annotation effort in the target language for reference-free hallucination detection. Aiming to catalyze future research on cross-lingual token-level reference-free hallucination detection, we make ANHALTEN publicly available: this https URL

Title: Are We Ready for Out-of-Distribution Detection in Digital Pathology?

• Authors: Ji-Hun Oh, Kianoush Falahkheirkhah, Rohit Bhargava
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13708
• Pdf URL: https://arxiv.org/pdf/2407.13708
• Copy Paste: [[2407.13708]] Are We Ready for Out-of-Distribution Detection in Digital Pathology?(https://arxiv.org/abs/2407.13708)
• Keywords: transformer
• Abstract: The detection of semantic and covariate out-of-distribution (OOD) examples is a critical yet overlooked challenge in digital pathology (DP). Recently, substantial insight and methods on OOD detection were presented by the ML community, but how do they fare in DP applications? To this end, we establish a benchmark study, our highlights being: 1) the adoption of proper evaluation protocols, 2) the comparison of diverse detectors in both a single and multi-model setting, and 3) the exploration into advanced ML settings like transfer learning (ImageNet vs. DP pre-training) and choice of architecture (CNNs vs. transformers). Through our comprehensive experiments, we contribute new insights and guidelines, paving the way for future research and discussion.

Title: Understanding Reference Policies in Direct Preference Optimization

• Authors: Yixin Liu, Pengfei Liu, Arman Cohan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13709
• Pdf URL: https://arxiv.org/pdf/2407.13709
• Copy Paste: [[2407.13709]] Understanding Reference Policies in Direct Preference Optimization(https://arxiv.org/abs/2407.13709)
• Keywords: large language model
• Abstract: Direct Preference Optimization (DPO) has become a widely used training method for the instruction fine-tuning of large language models (LLMs). In this work, we explore an under-investigated aspect of DPO - its dependency on the reference model or policy. Such reference policies, typically instantiated as the model to be further fine-tuned, are important since they can impose an upper limit on DPO's effectiveness. Therefore, we address three related research questions in this work. First, we explore the optimal strength of the KL-divergence constraint in DPO, which penalizes deviations from the reference policy, and find that DPO is sensitive to this strength. Next, we examine the necessity of reference policies for instruction fine-tuning by providing both theoretical and empirical comparisons between DPO and related learning objectives, demonstrating DPO's superiority. Additionally, we investigate whether DPO benefits from stronger reference policies, finding that a stronger reference policy can lead to improved performance, but only when it is similar to the model being fine-tuned. Our findings highlight the confounding role of reference policies in DPO and offer insights for best practices, while also identifying open research questions for future studies.

Title: FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning

• Authors: Tristan Cinquin, Marvin Pförtner, Vincent Fortuin, Philipp Hennig, Robert Bamler
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13711
• Pdf URL: https://arxiv.org/pdf/2407.13711
• Copy Paste: [[2407.13711]] FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning(https://arxiv.org/abs/2407.13711)
• Keywords: interpretability
• Abstract: Laplace approximations are popular techniques for endowing deep networks with epistemic uncertainty estimates as they can be applied without altering the predictions of the neural network, and they scale to large models and datasets. While the choice of prior strongly affects the resulting posterior distribution, computational tractability and lack of interpretability of weight space typically limit the Laplace approximation to isotropic Gaussian priors, which are known to cause pathological behavior as depth increases. As a remedy, we directly place a prior on function space. More precisely, since Lebesgue densities do not exist on infinite-dimensional function spaces, we have to recast training as finding the so-called weak mode of the posterior measure under a Gaussian process (GP) prior restricted to the space of functions representable by the neural network. Through the GP prior, one can express structured and interpretable inductive biases, such as regularity or periodicity, directly in function space, while still exploiting the implicit inductive biases that allow deep networks to generalize. After model linearization, the training objective induces a negative log-posterior density to which we apply a Laplace approximation, leveraging highly scalable methods from matrix-free linear algebra. Our method provides improved results where prior knowledge is abundant, e.g., in many scientific inference tasks. At the same time, it stays competitive for black-box regression and classification tasks where neural networks typically excel.

Title: Scalable Optimization for Locally Relevant Geo-Location Privacy

• Authors: Chenxi Qiu, Ruiyao Liu, Primal Pappachan, Anna Squicciarini, Xinpeng Xie
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13725
• Pdf URL: https://arxiv.org/pdf/2407.13725
• Copy Paste: [[2407.13725]] Scalable Optimization for Locally Relevant Geo-Location Privacy(https://arxiv.org/abs/2407.13725)
• Keywords: privacy, protect
• Abstract: Geo-obfuscation functions as a location privacy protection mechanism (LPPM), enabling mobile users to share obfuscated locations with servers instead of their exact locations. This technique protects users' location privacy during server-side data breaches since the obfuscation process is irreversible. To minimize the utility loss caused by data obfuscation, linear programming (LP) is widely used. However, LP can face a polynomial explosion in decision variables, making it impractical for large-scale geo-obfuscation applications. In this paper, we propose a new LPPM called Locally Relevant Geo-obfuscation (LR-Geo) to optimize geo-obfuscation using LP more efficiently. This is accomplished by restricting the geo-obfuscation calculations for each user to locally relevant (LR) locations near the user's actual location. To prevent LR locations from inadvertently revealing a user's true whereabouts, users compute the LP coefficients locally and upload only these coefficients to the server, rather than the LR locations themselves. The server then solves the LP problem using the provided coefficients. Additionally, we enhance the LP framework with an exponential obfuscation mechanism to ensure that the obfuscation distribution is indistinguishable across multiple users. By leveraging the constraint structure of the LP formulation, we apply Benders' decomposition to further boost computational efficiency. Our theoretical analysis confirms that, even though geo-obfuscation is calculated independently for each user, it still adheres to geo-indistinguishability constraints across multiple users with high probability. Finally, experimental results using a real-world dataset demonstrate that LR-Geo outperforms existing geo-obfuscation methods in terms of computational time, data utility, and privacy protection.

Title: Baba Is AI: Break the Rules to Beat the Benchmark

• Authors: Nathan Cloos, Meagan Jens, Michelangelo Naim, Yen-Ling Kuo, Ignacio Cases, Andrei Barbu, Christopher J. Cueva
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13729
• Pdf URL: https://arxiv.org/pdf/2407.13729
• Copy Paste: [[2407.13729]] Baba Is AI: Break the Rules to Beat the Benchmark(https://arxiv.org/abs/2407.13729)
• Keywords: large language model
• Abstract: Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives. To probe these abilities, we developed a new benchmark based on the game Baba Is You where an agent manipulates both objects in the environment and rules, represented by movable tiles with words written on them, to reach a specified goal and win the game. We test three state-of-the-art multi-modal large language models (OpenAI GPT-4o, Google Gemini-1.5-Pro and Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined.

Title: Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review

• Authors: Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, Sergey Levine
• Subjects: cs.LG, cs.AI, q-bio.QM, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.13734
• Pdf URL: https://arxiv.org/pdf/2407.13734
• Copy Paste: [[2407.13734]] Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review(https://arxiv.org/abs/2407.13734)
• Keywords: diffusion, generative
• Abstract: This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. While diffusion models are widely known to provide excellent generative modeling capability, practical applications in domains such as biology require generating samples that maximize some desired metric (e.g., translation efficiency in RNA, docking score in molecules, stability in protein). In these cases, the diffusion model can be optimized not only to generate realistic samples but also to explicitly maximize the measure of interest. Such methods are based on concepts from reinforcement learning (RL). We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning, tailored specifically for fine-tuning diffusion models. We aim to explore fundamental aspects such as the strengths and limitations of different RL-based fine-tuning algorithms across various scenarios, the benefits of RL-based fine-tuning compared to non-RL-based approaches, and the formal objectives of RL-based fine-tuning (target distributions). Additionally, we aim to examine their connections with related topics such as classifier guidance, Gflownets, flow-based diffusion models, path integral control theory, and sampling from unnormalized distributions such as MCMC. The code of this tutorial is available at this https URL

Title: CellularLint: A Systematic Approach to Identify Inconsistent Behavior in Cellular Network Specifications

• Authors: Mirza Masfiqur Rahman, Imtiaz Karim, Elisa Bertino
• Subjects: cs.CR, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.13742
• Pdf URL: https://arxiv.org/pdf/2407.13742
• Copy Paste: [[2407.13742]] CellularLint: A Systematic Approach to Identify Inconsistent Behavior in Cellular Network Specifications(https://arxiv.org/abs/2407.13742)
• Keywords: security, privacy, large language model
• Abstract: In recent years, there has been a growing focus on scrutinizing the security of cellular networks, often attributing security vulnerabilities to issues in the underlying protocol design descriptions. These protocol design specifications, typically extensive documents that are thousands of pages long, can harbor inaccuracies, underspecifications, implicit assumptions, and internal inconsistencies. In light of the evolving landscape, we introduce CellularLint--a semi-automatic framework for inconsistency detection within the standards of 4G and 5G, capitalizing on a suite of natural language processing techniques. Our proposed method uses a revamped few-shot learning mechanism on domain-adapted large language models. Pre-trained on a vast corpus of cellular network protocols, this method enables CellularLint to simultaneously detect inconsistencies at various levels of semantics and practical use cases. In doing so, CellularLint significantly advances the automated analysis of protocol specifications in a scalable fashion. In our investigation, we focused on the Non-Access Stratum (NAS) and the security specifications of 4G and 5G networks, ultimately uncovering 157 inconsistencies with 82.67% accuracy. After verification of these inconsistencies on open-source implementations and 17 commercial devices, we confirm that they indeed have a substantial impact on design decisions, potentially leading to concerns related to privacy, integrity, availability, and interoperability.

Title: LLMs as Function Approximators: Terminology, Taxonomy, and Questions for Evaluation

• Authors: David Schlangen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13744
• Pdf URL: https://arxiv.org/pdf/2407.13744
• Copy Paste: [[2407.13744]] LLMs as Function Approximators: Terminology, Taxonomy, and Questions for Evaluation(https://arxiv.org/abs/2407.13744)
• Keywords: protect
• Abstract: Natural Language Processing has moved rather quickly from modelling specific tasks to taking more general pre-trained models and fine-tuning them for specific tasks, to a point where we now have what appear to be inherently generalist models. This paper argues that the resultant loss of clarity on what these models model leads to metaphors like "artificial general intelligences" that are not helpful for evaluating their strengths and weaknesses. The proposal is to see their generality, and their potential value, in their ability to approximate specialist function, based on a natural language specification. This framing brings to the fore questions of the quality of the approximation, but beyond that, also questions of discoverability, stability, and protectability of these functions. As the paper will show, this framing hence brings together in one conceptual framework various aspects of evaluation, both from a practical and a theoretical perspective, as well as questions often relegated to a secondary status (such as "prompt injection" and "jailbreaking").

Title: Pose-guided multi-task video transformer for driver action recognition

• Authors: Ricardo Pizarro, Roberto Valle, Luis Miguel Bergasa, José M. Buenaposada, Luis Baumela
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13750
• Pdf URL: https://arxiv.org/pdf/2407.13750
• Copy Paste: [[2407.13750]] Pose-guided multi-task video transformer for driver action recognition(https://arxiv.org/abs/2407.13750)
• Keywords: transformer
• Abstract: We investigate the task of identifying situations of distracted driving through analysis of in-car videos. To tackle this challenge we introduce a multi-task video transformer that predicts both distracted actions and driver pose. Leveraging VideoMAEv2, a large pre-trained architecture, our approach incorporates semantic information from human keypoint locations to enhance action recognition and decrease computational overhead by minimizing the number of spatio-temporal tokens. By guiding token selection with pose and class information, we notably reduce the model's computational requirements while preserving the baseline accuracy. Our model surpasses existing state-of-the art results in driver action recognition while exhibiting superior efficiency compared to current video transformer-based approaches.

Title: LogoSticker: Inserting Logos into Diffusion Models for Customized Generation

• Authors: Mingkang Zhu, Xi Chen, Zhongdao Wang, Hengshuang Zhao, Jiaya Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13752
• Pdf URL: https://arxiv.org/pdf/2407.13752
• Copy Paste: [[2407.13752]] LogoSticker: Inserting Logos into Diffusion Models for Customized Generation(https://arxiv.org/abs/2407.13752)
• Keywords: extraction, diffusion
• Abstract: Recent advances in text-to-image model customization have underscored the importance of integrating new concepts with a few examples. Yet, these progresses are largely confined to widely recognized subjects, which can be learned with relative ease through models' adequate shared prior knowledge. In contrast, logos, characterized by unique patterns and textual elements, are hard to establish shared knowledge within diffusion models, thus presenting a unique challenge. To bridge this gap, we introduce the task of logo insertion. Our goal is to insert logo identities into diffusion models and enable their seamless synthesis in varied contexts. We present a novel two-phase pipeline LogoSticker to tackle this task. First, we propose the actor-critic relation pre-training algorithm, which addresses the nontrivial gaps in models' understanding of the potential spatial positioning of logos and interactions with other objects. Second, we propose a decoupled identity learning algorithm, which enables precise localization and identity extraction of logos. LogoSticker can generate logos accurately and harmoniously in diverse contexts. We comprehensively validate the effectiveness of LogoSticker over customization methods and large models such as DALLE~3. \href{this https URL}{Project page}.

Title: Exploring Facial Biomarkers for Depression through Temporal Analysis of Action Units

• Authors: Aditya Parikh, Misha Sadeghi, Bjorn Eskofier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13753
• Pdf URL: https://arxiv.org/pdf/2407.13753
• Copy Paste: [[2407.13753]] Exploring Facial Biomarkers for Depression through Temporal Analysis of Action Units(https://arxiv.org/abs/2407.13753)
• Keywords: extraction
• Abstract: Depression is characterized by persistent sadness and loss of interest, significantly impairing daily functioning and now a widespread mental disorder. Traditional diagnostic methods rely on subjective assessments, necessitating objective approaches for accurate diagnosis. Our study investigates the use of facial action units (AUs) and emotions as biomarkers for depression. We analyzed facial expressions from video data of participants classified with or without depression. Our methodology involved detailed feature extraction, mean intensity comparisons of key AUs, and the application of time series classification models. Furthermore, we employed Principal Component Analysis (PCA) and various clustering algorithms to explore the variability in emotional expression patterns. Results indicate significant differences in the intensities of AUs associated with sadness and happiness between the groups, highlighting the potential of facial analysis in depression assessment.

Title: Black-Box Opinion Manipulation Attacks to Retrieval-Augmented Generation of Large Language Models

• Authors: Zhuo Chen, Jiawei Liu, Haotan Liu, Qikai Cheng, Fan Zhang, Wei Lu, Xiaozhong Liu
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13757
• Pdf URL: https://arxiv.org/pdf/2407.13757
• Copy Paste: [[2407.13757]] Black-Box Opinion Manipulation Attacks to Retrieval-Augmented Generation of Large Language Models(https://arxiv.org/abs/2407.13757)
• Keywords: security, attack, generative, large language model
• Abstract: Retrieval-Augmented Generation (RAG) is applied to solve hallucination problems and real-time constraints of large language models, but it also induces vulnerabilities against retrieval corruption attacks. Existing research mainly explores the unreliability of RAG in white-box and closed-domain QA tasks. In this paper, we aim to reveal the vulnerabilities of Retrieval-Enhanced Generative (RAG) models when faced with black-box attacks for opinion manipulation. We explore the impact of such attacks on user cognition and decision-making, providing new insight to enhance the reliability and security of RAG models. We manipulate the ranking results of the retrieval model in RAG with instruction and use these results as data to train a surrogate model. By employing adversarial retrieval attack methods to the surrogate model, black-box transfer attacks on RAG are further realized. Experiments conducted on opinion datasets across multiple topics show that the proposed attack strategy can significantly alter the opinion polarity of the content generated by RAG. This demonstrates the model's vulnerability and, more importantly, reveals the potential negative impact on user cognition and decision-making, making it easier to mislead users into accepting incorrect or biased information.

Title: Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion

• Authors: Boyang Deng, Richard Tucker, Zhengqi Li, Leonidas Guibas, Noah Snavely, Gordon Wetzstein
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2407.13759
• Pdf URL: https://arxiv.org/pdf/2407.13759
• Copy Paste: [[2407.13759]] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion(https://arxiv.org/abs/2407.13759)
• Keywords: diffusion
• Abstract: We present a method for generating Streetscapes-long sequences of views through an on-the-fly synthesized city-scale scene. Our generation is conditioned by language input (e.g., city name, weather), as well as an underlying map/layout hosting the desired trajectory. Compared to recent models for video generation or 3D view synthesis, our method can scale to much longer-range camera trajectories, spanning several city blocks, while maintaining visual quality and consistency. To achieve this goal, we build on recent work on video diffusion, used within an autoregressive framework that can easily scale to long sequences. In particular, we introduce a new temporal imputation method that prevents our autoregressive approach from drifting from the distribution of realistic city imagery. We train our Streetscapes system on a compelling source of data-posed imagery from Google Street View, along with contextual map data-which allows users to generate city views conditioned on any desired city layout, with controllable camera poses. Please see more results at our project page at this https URL.

Title: SegPoint: Segment Any Point Cloud via Large Language Model

• Authors: Shuting He, Henghui Ding, Xudong Jiang, Bihan Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13761
• Pdf URL: https://arxiv.org/pdf/2407.13761
• Copy Paste: [[2407.13761]] SegPoint: Segment Any Point Cloud via Large Language Model(https://arxiv.org/abs/2407.13761)
• Keywords: large language model, segmentation
• Abstract: Despite significant progress in 3D point cloud segmentation, existing methods primarily address specific tasks and depend on explicit instructions to identify targets, lacking the capability to infer and understand implicit user intentions in a unified framework. In this work, we propose a model, called SegPoint, that leverages the reasoning capabilities of a multi-modal Large Language Model (LLM) to produce point-wise segmentation masks across a diverse range of tasks: 1) 3D instruction segmentation, 2) 3D referring segmentation, 3) 3D semantic segmentation, and 4) 3D open-vocabulary semantic segmentation. To advance 3D instruction research, we introduce a new benchmark, Instruct3D, designed to evaluate segmentation performance from complex and implicit instructional texts, featuring 2,565 point cloud-instruction pairs. Our experimental results demonstrate that SegPoint achieves competitive performance on established benchmarks such as ScanRefer for referring segmentation and ScanNet for semantic segmentation, while delivering outstanding outcomes on the Instruct3D dataset. To our knowledge, SegPoint is the first model to address these varied segmentation tasks within a single framework, achieving satisfactory performance.

Title: Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data

• Authors: Charles Jin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13765
• Pdf URL: https://arxiv.org/pdf/2407.13765
• Copy Paste: [[2407.13765]] Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data(https://arxiv.org/abs/2407.13765)
• Keywords: robust
• Abstract: As language models (LMs) deliver increasing performance on a range of NLP tasks, probing classifiers have become an indispensable technique in the effort to better understand their inner workings. A typical setup involves (1) defining an auxiliary task consisting of a dataset of text annotated with labels, then (2) supervising small classifiers to predict the labels from the representations of a pretrained LM as it processed the dataset. A high probing accuracy is interpreted as evidence that the LM has learned to perform the auxiliary task as an unsupervised byproduct of its original pretraining objective. Despite the widespread usage of probes, however, the robust design and analysis of probing experiments remains a challenge. We develop a formal perspective on probing using structural causal models (SCM). Specifically, given an SCM which explains the distribution of tokens observed during training, we frame the central hypothesis as whether the LM has learned to represent the latent variables of the SCM. Empirically, we extend a recent study of LMs in the context of a synthetic grid-world navigation task, where having an exact model of the underlying causal structure allows us to draw strong inferences from the result of probing experiments. Our techniques provide robust empirical evidence for the ability of LMs to learn the latent causal concepts underlying text.

Title: Visual Haystacks: Answering Harder Questions About Sets of Images

• Authors: Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, David M. Chan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13766
• Pdf URL: https://arxiv.org/pdf/2407.13766
• Copy Paste: [[2407.13766]] Visual Haystacks: Answering Harder Questions About Sets of Images(https://arxiv.org/abs/2407.13766)
• Keywords: robust
• Abstract: Recent advancements in Large Multimodal Models (LMMs) have made significant progress in the field of single-image visual question answering. However, these models face substantial challenges when tasked with queries that span extensive collections of images, similar to real-world scenarios like searching through large photo albums, finding specific information across the internet, or monitoring environmental changes through satellite imagery. This paper explores the task of Multi-Image Visual Question Answering (MIQA): given a large set of images and a natural language query, the task is to generate a relevant and grounded response. We propose a new public benchmark, dubbed "Visual Haystacks (VHs)," specifically designed to evaluate LMMs' capabilities in visual retrieval and reasoning over sets of unrelated images, where we perform comprehensive evaluations demonstrating that even robust closed-source models struggle significantly. Towards addressing these shortcomings, we introduce MIRAGE (Multi-Image Retrieval Augmented Generation), a novel retrieval/QA framework tailored for LMMs that confronts the challenges of MIQA with marked efficiency and accuracy improvements over baseline methods. Our evaluation shows that MIRAGE surpasses closed-source GPT-4o models by up to 11% on the VHs benchmark and offers up to 3.4x improvements in efficiency over text-focused multi-stage approaches.

Title: Training-Free Model Merging for Multi-target Domain Adaptation

• Authors: Wenyi Li, Huan-ang Gao, Mingju Gao, Beiwen Tian, Rong Zhi, Hao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13771
• Pdf URL: https://arxiv.org/pdf/2407.13771
• Copy Paste: [[2407.13771]] Training-Free Model Merging for Multi-target Domain Adaptation(https://arxiv.org/abs/2407.13771)
• Keywords: privacy
• Abstract: In this paper, we study multi-target domain adaptation of scene understanding models. While previous methods achieved commendable results through inter-domain consistency losses, they often assumed unrealistic simultaneous access to images from all target domains, overlooking constraints such as data transfer bandwidth limitations and data privacy concerns. Given these challenges, we pose the question: How to merge models adapted independently on distinct domains while bypassing the need for direct access to training data? Our solution to this problem involves two components, merging model parameters and merging model buffers (i.e., normalization layer statistics). For merging model parameters, empirical analyses of mode connectivity surprisingly reveal that linear merging suffices when employing the same pretrained backbone weights for adapting separate models. For merging model buffers, we model the real-world distribution with a Gaussian prior and estimate new statistics from the buffers of separately trained models. Our method is simple yet effective, achieving comparable performance with data combination training baselines, while eliminating the need for accessing training data. Project page: this https URL

Title: GroupMamba: Parameter-Efficient and Accurate Group Visual State Space Model

• Authors: Abdelrahman Shaker, Syed Talal Wasim, Salman Khan, Juergen Gall, Fahad Shahbaz Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13772
• Pdf URL: https://arxiv.org/pdf/2407.13772
• Copy Paste: [[2407.13772]] GroupMamba: Parameter-Efficient and Accurate Group Visual State Space Model(https://arxiv.org/abs/2407.13772)
• Keywords: segmentation
• Abstract: Recent advancements in state-space models (SSMs) have showcased effective performance in modeling long-range dependencies with subquadratic complexity. However, pure SSM-based models still face challenges related to stability and achieving optimal performance on computer vision tasks. Our paper addresses the challenges of scaling SSM-based models for computer vision, particularly the instability and inefficiency of large model sizes. To address this, we introduce a Modulated Group Mamba layer which divides the input channels into four groups and applies our proposed SSM-based efficient Visual Single Selective Scanning (VSSS) block independently to each group, with each VSSS block scanning in one of the four spatial directions. The Modulated Group Mamba layer also wraps the four VSSS blocks into a channel modulation operator to improve cross-channel communication. Furthermore, we introduce a distillation-based training objective to stabilize the training of large models, leading to consistent performance gains. Our comprehensive experiments demonstrate the merits of the proposed contributions, leading to superior performance over existing methods for image classification on ImageNet-1K, object detection, instance segmentation on MS-COCO, and semantic segmentation on ADE20K. Our tiny variant with 23M parameters achieves state-of-the-art performance with a classification top-1 accuracy of 83.3% on ImageNet-1K, while being 26% efficient in terms of parameters, compared to the best existing Mamba design of same model size. Our code and models are available at: this https URL.