2024-08-23

Title: The Mechanics of Conceptual Interpretation in GPT Models: Interpretative Insights

• Authors: Nura Aljaafari, Danilo S. Carvalho, André Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11827
• Pdf URL: https://arxiv.org/pdf/2408.11827
• Copy Paste: [[2408.11827]] The Mechanics of Conceptual Interpretation in GPT Models: Interpretative Insights(https://arxiv.org/abs/2408.11827)
• Keywords: interpretability, transformer, large language model
• Abstract: Locating and editing knowledge in large language models (LLMs) is crucial for enhancing their accuracy, safety, and inference rationale. We introduce ``concept editing'', an innovative variation of knowledge editing that uncovers conceptualisation mechanisms within these models. Using the reverse dictionary task, inference tracing, and input abstraction, we analyse the Multi-Layer Perceptron (MLP), Multi-Head Attention (MHA), and hidden state components of transformer models. Our results reveal distinct patterns: MLP layers employ key-value retrieval mechanism and context-dependent processing, which are highly associated with relative input tokens. MHA layers demonstrate a distributed nature with significant higher-level activations, suggesting sophisticated semantic integration. Hidden states emphasise the importance of the last token and top layers in the inference process. We observe evidence of gradual information building and distributed representation. These observations elucidate how transformer models process semantic information, paving the way for targeted interventions and improved interpretability techniques. Our work highlights the complex, layered nature of semantic processing in LLMs and the challenges of isolating and modifying specific concepts within these models.

Title: FAKER: Full-body Anonymization with Human Keypoint Extraction for Real-time Video Deidentification

• Authors: Byunghyun Ban, Hyoseok Lee
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.11829
• Pdf URL: https://arxiv.org/pdf/2408.11829
• Copy Paste: [[2408.11829]] FAKER: Full-body Anonymization with Human Keypoint Extraction for Real-time Video Deidentification(https://arxiv.org/abs/2408.11829)
• Keywords: protect, extraction, generative
• Abstract: In the contemporary digital era, protection of personal information has become a paramount issue. The exponential growth of the media industry has heightened concerns regarding the anonymization of individuals captured in video footage. Traditional methods, such as blurring or pixelation, are commonly employed, while recent advancements have introduced generative adversarial networks (GAN) to redraw faces in videos. In this study, we propose a novel approach that employs a significantly smaller model to achieve real-time full-body anonymization of individuals in videos. Unlike conventional techniques that often fail to effectively remove personal identification information such as skin color, clothing, accessories, and body shape while our method successfully eradicates all such details. Furthermore, by leveraging pose estimation algorithms, our approach accurately represents information regarding individuals' positions, movements, and postures. This algorithm can be seamlessly integrated into CCTV or IP camera systems installed in various industrial settings, functioning in real-time and thus facilitating the widespread adoption of full-body anonymization technology.

Title: OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs

• Authors: Hasan Iqbal, Yuxia Wang, Minghan Wang, Georgi Georgiev, Jiahui Geng, Iryna Gurevych, Preslav Nakov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11832
• Pdf URL: https://arxiv.org/pdf/2408.11832
• Copy Paste: [[2408.11832]] OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs(https://arxiv.org/abs/2408.11832)
• Keywords: large language model
• Abstract: The increased use of large language models (LLMs) across a variety of real-world applications calls for automatic tools to check the factual accuracy of their outputs, as LLMs often hallucinate. This is difficult as it requires assessing the factuality of free-form open-domain responses. While there has been a lot of research on this topic, different papers use different evaluation benchmarks and measures, which makes them hard to compare and hampers future progress. To mitigate these issues, we developed OpenFactCheck, a unified framework, with three modules: (i) RESPONSEEVAL, which allows users to easily customize an automatic fact-checking system and to assess the factuality of all claims in an input document using that system, (ii) LLMEVAL, which assesses the overall factuality of an LLM, and (iii) CHECKEREVAL, a module to evaluate automatic fact-checking systems. OpenFactCheck is open-sourced (this https URL) and publicly released as a Python library (this https URL) and also as a web service (this https URL). A video describing the system is available at this https URL.

Title: SCREENER: A general framework for task-specific experiment design in quantitative MRI

• Authors: Tianshu Zheng, Zican Wang, Timothy Bray, Daniel C. Alexander, Dan Wu, Hui Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11834
• Pdf URL: https://arxiv.org/pdf/2408.11834
• Copy Paste: [[2408.11834]] SCREENER: A general framework for task-specific experiment design in quantitative MRI(https://arxiv.org/abs/2408.11834)
• Keywords: robust, diffusion
• Abstract: Quantitative magnetic resonance imaging (qMRI) is increasingly investigated for use in a variety of clinical tasks from diagnosis, through staging, to treatment monitoring. However, experiment design in qMRI, the identification of the optimal acquisition protocols, has been focused on obtaining the most precise parameter estimations, with no regard for the specific requirements of downstream tasks. Here we propose SCREENER: A general framework for task-specific experiment design in quantitative MRI. SCREENER incorporates a task-specific objective and seeks the optimal protocol with a deep-reinforcement-learning (DRL) based optimization strategy. To illustrate this framework, we employ a task of classifying the inflammation status of bone marrow using diffusion MRI data with intravoxel incoherent motion (IVIM) modelling. Results demonstrate SCREENER outperforms previous ad hoc and optimized protocols under clinical signal-to-noise ratio (SNR) conditions, achieving significant improvement, both in binary classification tasks, e.g. from 67% to 89%, and in a multi-class classification task, from 46% to 59%. Additionally, we show this improvement is robust to the SNR. Lastly, we demonstrate the advantage of DRL-based optimization strategy, enabling zero-shot discovery of near-optimal protocols for a range of SNRs not used in training. In conclusion, SCREENER has the potential to enable wider uptake of qMRI in the clinic.

Title: Analysis of Unstructured High-Density Crowded Scenes for Crowd Monitoring

• Authors: Alexandre Matov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11836
• Pdf URL: https://arxiv.org/pdf/2408.11836
• Copy Paste: [[2408.11836]] Analysis of Unstructured High-Density Crowded Scenes for Crowd Monitoring(https://arxiv.org/abs/2408.11836)
• Keywords: security
• Abstract: We are interested in developing an automated system for detection of organized movements in human crowds. Computer vision algorithms can extract information from videos of crowded scenes and automatically detect and track groups of individuals undergoing organized motion, which represents an anomalous behavior in the context of conflict aversion. Our system can detect organized cohorts against the background of randomly moving objects and we can estimate the number of participants in an organized cohort, the speed and direction of motion in real time, within three to four video frames, which is less than one second from the onset of motion captured on a CCTV. We have performed preliminary analysis in this context in biological cell data containing up to four thousand objects per frame and will extend this numerically to a hundred-fold for public safety applications. We envisage using the existing infrastructure of video cameras for acquiring image datasets on-the-fly and deploying an easy-to-use data-driven software system for parsing of significant events by analyzing image sequences taken inside and outside of sports stadiums or other public venues. Other prospective users are organizers of political rallies, civic and wildlife organizations, security firms, and the military. We will optimize the performance of the software by implementing a classification method able to distinguish between activities posing a threat and those not posing a threat.

Title: Joint PET-MRI Reconstruction with Diffusion Stochastic Differential Model

• Authors: Taofeng Xie, Zhuoxu Cui, Congcong Liu, Chen Luo, Huayu Wang, Yuanzhi Zhang, Xuemei Wang, Yihang Zhou, Qiyu Jin, Guoqing Chen, Dong Liang, Haifeng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11840
• Pdf URL: https://arxiv.org/pdf/2408.11840
• Copy Paste: [[2408.11840]] Joint PET-MRI Reconstruction with Diffusion Stochastic Differential Model(https://arxiv.org/abs/2408.11840)
• Keywords: diffusion
• Abstract: PET suffers from a low signal-to-noise ratio. Meanwhile, the k-space data acquisition process in MRI is time-consuming by PET-MRI systems. We aim to accelerate MRI and improve PET image quality. This paper proposed a novel joint reconstruction model by diffusion stochastic differential equations based on learning the joint probability distribution of PET and MRI. Compare the results underscore the qualitative and quantitative improvements our model brings to PET and MRI reconstruction, surpassing the current state-of-the-art methodologies. Joint PET-MRI reconstruction is a challenge in the PET-MRI system. This studies focused on the relationship extends beyond edges. In this study, PET is generated from MRI by learning joint probability distribution as the relationship.

Title: Editable Fairness: Fine-Grained Bias Mitigation in Language Models

• Authors: Ruizhe Chen, Yichen Li, Jianfei Yang, Joey Tianyi Zhou, Zuozhu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11843
• Pdf URL: https://arxiv.org/pdf/2408.11843
• Copy Paste: [[2408.11843]] Editable Fairness: Fine-Grained Bias Mitigation in Language Models(https://arxiv.org/abs/2408.11843)
• Keywords: fair, large language model
• Abstract: Generating fair and accurate predictions plays a pivotal role in deploying large language models (LLMs) in the real world. However, existing debiasing methods inevitably generate unfair or incorrect predictions as they are designed and evaluated to achieve parity across different social groups but leave aside individual commonsense facts, resulting in modified knowledge that elicits unreasonable or undesired predictions. In this paper, we first establish a new bias mitigation benchmark, BiaScope, which systematically assesses performance by leveraging newly constructed datasets and metrics on knowledge retention and generalization. Then, we propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases. FAST identifies the decisive layer responsible for storing social biases and then calibrates its outputs by integrating a small modular network, considering both bias mitigation and knowledge-preserving demands. Comprehensive experiments demonstrate that FAST surpasses state-of-the-art baselines with superior debiasing performance while not compromising the overall model capability for knowledge retention and downstream predictions. This highlights the potential of fine-grained debiasing strategies to achieve fairness in LLMs. Code will be publicly available.

Title: LLaMA based Punctuation Restoration With Forward Pass Only Decoding

• Authors: Yutong Pang, Debjyoti Paul, Kevin Jiang, Xuedong Zhang, Xin Lei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11845
• Pdf URL: https://arxiv.org/pdf/2408.11845
• Copy Paste: [[2408.11845]] LLaMA based Punctuation Restoration With Forward Pass Only Decoding(https://arxiv.org/abs/2408.11845)
• Keywords: large language model
• Abstract: This paper introduces two advancements in the field of Large Language Model Annotation with a focus on punctuation restoration tasks. Our first contribution is the application of LLaMA for punctuation restoration, which demonstrates superior performance compared to the established benchmark. Despite its impressive quality, LLaMA faces challenges regarding inference speed and hallucinations. To address this, our second contribution presents Forward Pass Only Decoding (FPOD), a novel decoding approach for annotation tasks. This innovative method results in a substantial 19.8x improvement in inference speed, effectively addressing a critical bottleneck and enhancing the practical utility of LLaMA for large-scale data annotation tasks without hallucinations. The combination of these contributions not only solidifies LLaMA as a powerful tool for punctuation restoration but also highlights FPOD as a crucial strategy for overcoming speed constraints.

Title: Prompto: An open source library for asynchronous querying of LLM endpoints

• Authors: Ryan Sze-Yin Chan, Federico Nanni, Edwin Brown, Ed Chapman, Angus R. Williams, Jonathan Bright, Evelina Gabasova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11847
• Pdf URL: https://arxiv.org/pdf/2408.11847
• Copy Paste: [[2408.11847]] Prompto: An open source library for asynchronous querying of LLM endpoints(https://arxiv.org/abs/2408.11847)
• Keywords: large language model
• Abstract: Recent surge in Large Language Model (LLM) availability has opened exciting avenues for research. However, efficiently interacting with these models presents a significant hurdle since LLMs often reside on proprietary or self-hosted API endpoints, each requiring custom code for interaction. Conducting comparative studies between different models can therefore be time-consuming and necessitate significant engineering effort, hindering research efficiency and reproducibility. To address these challenges, we present prompto, an open source Python library which facilitates asynchronous querying of LLM endpoints enabling researchers to interact with multiple LLMs concurrently, while maximising efficiency and utilising individual rate limits. Our library empowers researchers and developers to interact with LLMs more effectively and enabling faster experimentation and evaluation. prompto is released with an introductory video (this https URL) under MIT License and is available via GitHub (this https URL).

Title: MGH Radiology Llama: A Llama 3 70B Model for Radiology

• Authors: Yucheng Shi, Peng Shu, Zhengliang Liu, Zihao Wu, Quanzheng Li, Xiang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11848
• Pdf URL: https://arxiv.org/pdf/2408.11848
• Copy Paste: [[2408.11848]] MGH Radiology Llama: A Llama 3 70B Model for Radiology(https://arxiv.org/abs/2408.11848)
• Keywords: large language model
• Abstract: In recent years, the field of radiology has increasingly harnessed the power of artificial intelligence (AI) to enhance diagnostic accuracy, streamline workflows, and improve patient care. Large language models (LLMs) have emerged as particularly promising tools, offering significant potential in assisting radiologists with report generation, clinical decision support, and patient communication. This paper presents an advanced radiology-focused large language model: MGH Radiology Llama. It is developed using the Llama 3 70B model, building upon previous domain-specific models like Radiology-GPT and Radiology-Llama2. Leveraging a unique and comprehensive dataset from Massachusetts General Hospital, comprising over 6.5 million de-identified medical reports across various imaging modalities, the model demonstrates significant improvements in generating accurate and clinically relevant radiology impressions given the corresponding findings. Our evaluation, incorporating both traditional metrics and a GPT-4-based assessment, highlights the enhanced performance of this work over general-purpose LLMs.

Title: Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation

• Authors: Yinghao Aaron Li, Xilin Jiang, Jordan Darefsky, Ge Zhu, Nima Mesgarani
• Subjects: cs.CL, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.11849
• Pdf URL: https://arxiv.org/pdf/2408.11849
• Copy Paste: [[2408.11849]] Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation(https://arxiv.org/abs/2408.11849)
• Keywords: large language model
• Abstract: The rapid advancement of large language models (LLMs) has significantly propelled the development of text-based chatbots, demonstrating their capability to engage in coherent and contextually relevant dialogues. However, extending these advancements to enable end-to-end speech-to-speech conversation bots remains a formidable challenge, primarily due to the extensive dataset and computational resources required. The conventional approach of cascading automatic speech recognition (ASR), LLM, and text-to-speech (TTS) models in a pipeline, while effective, suffers from unnatural prosody because it lacks direct interactions between the input audio and its transcribed text and the output audio. These systems are also limited by their inherent latency from the ASR process for real-time applications. This paper introduces Style-Talker, an innovative framework that fine-tunes an audio LLM alongside a style-based TTS model for fast spoken dialog generation. Style-Talker takes user input audio and uses transcribed chat history and speech styles to generate both the speaking style and text for the response. Subsequently, the TTS model synthesizes the speech, which is then played back to the user. While the response speech is being played, the input speech undergoes ASR processing to extract the transcription and speaking style, serving as the context for the ensuing dialogue turn. This novel pipeline accelerates the traditional cascade ASR-LLM-TTS systems while integrating rich paralinguistic information from input speech. Our experimental results show that Style-Talker significantly outperforms the conventional cascade and speech-to-speech baselines in terms of both dialogue naturalness and coherence while being more than 50% faster.

Title: Fast Training Dataset Attribution via In-Context Learning

• Authors: Milad Fotouhi, Mohammad Taha Bahadori, Oluwaseyi Feyisetan, Payman Arabshahi, David Heckerman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11852
• Pdf URL: https://arxiv.org/pdf/2408.11852
• Copy Paste: [[2408.11852]] Fast Training Dataset Attribution via In-Context Learning(https://arxiv.org/abs/2408.11852)
• Keywords: robust, large language model
• Abstract: We investigate the use of in-context learning and prompt engineering to estimate the contributions of training data in the outputs of instruction-tuned large language models (LLMs). We propose two novel approaches: (1) a similarity-based approach that measures the difference between LLM outputs with and without provided context, and (2) a mixture distribution model approach that frames the problem of identifying contribution scores as a matrix factorization task. Our empirical comparison demonstrates that the mixture model approach is more robust to retrieval noise in in-context learning, providing a more reliable estimation of data contributions.

Title: When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?

• Authors: Yanjun Gao, Skatje Myers, Shan Chen, Dmitriy Dligach, Timothy A Miller, Danielle Bitterman, Matthew Churpek, Majid Afshar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11854
• Pdf URL: https://arxiv.org/pdf/2408.11854
• Copy Paste: [[2408.11854]] When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?(https://arxiv.org/abs/2408.11854)
• Keywords: large language model
• Abstract: The introduction of Large Language Models (LLMs) has advanced data representation and analysis, bringing significant progress in their use for medical questions and answering. Despite these advancements, integrating tabular data, especially numerical data pivotal in clinical contexts, into LLM paradigms has not been thoroughly explored. In this study, we examine the effectiveness of vector representations from last hidden states of LLMs for medical diagnostics and prognostics using electronic health record (EHR) data. We compare the performance of these embeddings with that of raw numerical EHR data when used as feature inputs to traditional machine learning (ML) algorithms that excel at tabular data learning, such as eXtreme Gradient Boosting. We focus on instruction-tuned LLMs in a zero-shot setting to represent abnormal physiological data and evaluating their utilities as feature extractors to enhance ML classifiers for predicting diagnoses, length of stay, and mortality. Furthermore, we examine prompt engineering techniques on zero-shot and few-shot LLM embeddings to measure their impact comprehensively. Although findings suggest the raw data features still prevails in medical ML tasks, zero-shot LLM embeddings demonstrate competitive results, suggesting a promising avenue for future research in medical applications.

Title: FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models

• Authors: Zhongyu Zhao, Menghang Dong, Rongyu Zhang, Wenzhao Zheng, Yunpeng Zhang, Huanrui Yang, Dalong Du, Kurt Keutzer, Shanghang Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11855
• Pdf URL: https://arxiv.org/pdf/2408.11855
• Copy Paste: [[2408.11855]] FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models(https://arxiv.org/abs/2408.11855)
• Keywords: large language model, segmentation
• Abstract: Recent research has demonstrated that Feed-Forward Networks (FFNs) in Large Language Models (LLMs) play a pivotal role in storing diverse linguistic and factual knowledge. Conventional methods frequently face challenges due to knowledge confusion stemming from their monolithic and redundant architectures, which calls for more efficient solutions with minimal computational overhead, particularly for LLMs. In this paper, we explore the FFN computation paradigm in LLMs and introduce FactorLLM, a novel approach that decomposes well-trained dense FFNs into sparse sub-networks without requiring any further modifications, while maintaining the same level of performance. Furthermore, we embed a router from the Mixture-of-Experts (MoE), combined with our devised Prior-Approximate (PA) loss term that facilitates the dynamic activation of experts and knowledge adaptation, thereby accelerating computational processes and enhancing performance using minimal training data and fine-tuning steps. FactorLLM thus enables efficient knowledge factorization and activates select groups of experts specifically tailored to designated tasks, emulating the interactive functional segmentation of the human brain. Extensive experiments across various benchmarks demonstrate the effectiveness of our proposed FactorLLM which achieves comparable performance to the source model securing up to 85% model performance while obtaining over a 30% increase in inference speed. Code: this https URL.

Title: Dynamic Adaptive Optimization for Effective Sentiment Analysis Fine-Tuning on Large Language Models

• Authors: Hongcheng Ding, Xuanze Zhao, Shamsul Nahar Abdullah, Deshinta Arrova Dewi, Zixiao Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11856
• Pdf URL: https://arxiv.org/pdf/2408.11856
• Copy Paste: [[2408.11856]] Dynamic Adaptive Optimization for Effective Sentiment Analysis Fine-Tuning on Large Language Models(https://arxiv.org/abs/2408.11856)
• Keywords: large language model
• Abstract: Sentiment analysis plays a crucial role in various domains, such as business intelligence and financial forecasting. Large language models (LLMs) have become a popular paradigm for sentiment analysis, leveraging multi-task learning to address specific tasks concurrently. However, LLMs with fine-tuning for sentiment analysis often underperforms due to the inherent challenges in managing diverse task complexities. Moreover, constant-weight approaches in multi-task learning struggle to adapt to variations in data characteristics, further complicating model effectiveness. To address these issues, we propose a novel multi-task learning framework with a dynamic adaptive optimization (DAO) module. This module is designed as a plug-and-play component that can be seamlessly integrated into existing models, providing an effective and flexible solution for multi-task learning. The key component of the DAO module is dynamic adaptive loss, which dynamically adjusts the weights assigned to different tasks based on their relative importance and data characteristics during training. Sentiment analyses on a standard and customized financial text dataset demonstrate that the proposed framework achieves superior performance. Specifically, this work improves the Mean Squared Error (MSE) and Accuracy (ACC) by 15.58% and 1.24% respectively, compared with previous work.

Title: Hermes 3 Technical Report

• Authors: Ryan Teknium, Jeffrey Quesnelle, Chen Guang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11857
• Pdf URL: https://arxiv.org/pdf/2408.11857
• Copy Paste: [[2408.11857]] Hermes 3 Technical Report(https://arxiv.org/abs/2408.11857)
• Keywords: large language model
• Abstract: Instruct (or "chat") tuned models have become the primary way in which most people interact with large language models. As opposed to "base" or "foundation" models, instruct-tuned models are optimized to respond to imperative statements. We present Hermes 3, a neutrally-aligned generalist instruct and tool use model with strong reasoning and creative abilities. Its largest version, Hermes 3 405B, achieves state of the art performance among open weight models on several public benchmarks.

Title: Convexity-based Pruning of Speech Representation Models

• Authors: Teresa Dorszewski, Lenka Tětková, Lars Kai Hansen
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.11858
• Pdf URL: https://arxiv.org/pdf/2408.11858
• Copy Paste: [[2408.11858]] Convexity-based Pruning of Speech Representation Models(https://arxiv.org/abs/2408.11858)
• Keywords: transformer
• Abstract: Speech representation models based on the transformer architecture and trained by self-supervised learning have shown great promise for solving tasks such as speech and speaker recognition, keyword spotting, emotion detection, and more. Typically, it is found that larger models lead to better performance. However, the significant computational effort involved in such large transformer systems is a challenge for embedded and real-world applications. Recent work has shown that there is significant redundancy in the transformer models for NLP and massive layer pruning is feasible (Sajjad et al., 2023). Here, we investigate layer pruning in audio models. We base the pruning decision on a convexity criterion. Convexity of classification regions has recently been proposed as an indicator of subsequent fine-tuning performance in a range of application domains, including NLP and audio. In empirical investigations, we find a massive reduction in the computational effort with no loss of performance or even improvements in certain cases.

Title: Speaking the Same Language: Leveraging LLMs in Standardizing Clinical Data for AI

• Authors: Arindam Sett, Somaye Hashemifar, Mrunal Yadav, Yogesh Pandit, Mohsen Hejrati
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11861
• Pdf URL: https://arxiv.org/pdf/2408.11861
• Copy Paste: [[2408.11861]] Speaking the Same Language: Leveraging LLMs in Standardizing Clinical Data for AI(https://arxiv.org/abs/2408.11861)
• Keywords: large language model
• Abstract: The implementation of Artificial Intelligence (AI) in the healthcare industry has garnered considerable attention, attributable to its prospective enhancement of clinical outcomes, expansion of access to superior healthcare, cost reduction, and elevation of patient satisfaction. Nevertheless, the primary hurdle that persists is related to the quality of accessible multi-modal healthcare data in conjunction with the evolution of AI methodologies. This study delves into the adoption of large language models to address specific challenges, specifically, the standardization of healthcare data. We advocate the use of these models to identify and map clinical data schemas to established data standard attributes, such as the Fast Healthcare Interoperability Resources. Our results illustrate that employing large language models significantly diminishes the necessity for manual data curation and elevates the efficacy of the data standardization process. Consequently, the proposed methodology has the propensity to expedite the integration of AI in healthcare, ameliorate the quality of patient care, whilst minimizing the time and financial resources necessary for the preparation of data for AI.

Title: Sentiment analysis of preservice teachers' reflections using a large language model

• Authors: Yunsoo Park, Younkyung Hong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11862
• Pdf URL: https://arxiv.org/pdf/2408.11862
• Copy Paste: [[2408.11862]] Sentiment analysis of preservice teachers' reflections using a large language model(https://arxiv.org/abs/2408.11862)
• Keywords: large language model
• Abstract: In this study, the emotion and tone of preservice teachers' reflections were analyzed using sentiment analysis with LLMs: GPT-4, Gemini, and BERT. We compared the results to understand how each tool categorizes and describes individual reflections and multiple reflections as a whole. This study aims to explore ways to bridge the gaps between qualitative, quantitative, and computational analyses of reflective practices in teacher education. This study finds that to effectively integrate LLM analysis into teacher education, developing an analysis method and result format that are both comprehensive and relevant for preservice teachers and teacher educators is crucial.

Title: Unraveling Text Generation in LLMs: A Stochastic Differential Equation Approach

• Authors: Yukun Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11863
• Pdf URL: https://arxiv.org/pdf/2408.11863
• Copy Paste: [[2408.11863]] Unraveling Text Generation in LLMs: A Stochastic Differential Equation Approach(https://arxiv.org/abs/2408.11863)
• Keywords: diffusion, large language model
• Abstract: This paper explores the application of Stochastic Differential Equations (SDE) to interpret the text generation process of Large Language Models (LLMs) such as GPT-4. Text generation in LLMs is modeled as a stochastic process where each step depends on previously generated content and model parameters, sampling the next word from a vocabulary distribution. We represent this generation process using SDE to capture both deterministic trends and stochastic perturbations. The drift term describes the deterministic trends in the generation process, while the diffusion term captures the stochastic variations. We fit these functions using neural networks and validate the model on real-world text corpora. Through numerical simulations and comprehensive analyses, including drift and diffusion analysis, stochastic process property evaluation, and phase space exploration, we provide deep insights into the dynamics of text generation. This approach not only enhances the understanding of the inner workings of LLMs but also offers a novel mathematical perspective on language generation, which is crucial for diagnosing, optimizing, and controlling the quality of generated text.

Title: How Susceptible are LLMs to Influence in Prompts?

• Authors: Sotiris Anagnostidis, Jannis Bulian
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11865
• Pdf URL: https://arxiv.org/pdf/2408.11865
• Copy Paste: [[2408.11865]] How Susceptible are LLMs to Influence in Prompts?(https://arxiv.org/abs/2408.11865)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) are highly sensitive to prompts, including additional context provided therein. As LLMs grow in capability, understanding their prompt-sensitivity becomes increasingly crucial for ensuring reliable and robust performance, particularly since evaluating these models becomes more challenging. In this work, we investigate how current models (Llama, Mixtral, Falcon) respond when presented with additional input from another model, mimicking a scenario where a more capable model -- or a system with access to more external information -- provides supplementary information to the target model. Across a diverse spectrum of question-answering tasks, we study how an LLM's response to multiple-choice questions changes when the prompt includes a prediction and explanation from another model. Specifically, we explore the influence of the presence of an explanation, the stated authoritativeness of the source, and the stated confidence of the supplementary input. Our findings reveal that models are strongly influenced, and when explanations are provided they are swayed irrespective of the quality of the explanation. The models are more likely to be swayed if the input is presented as being authoritative or confident, but the effect is small in size. This study underscores the significant prompt-sensitivity of LLMs and highlights the potential risks of incorporating outputs from external sources without thorough scrutiny and further validation. As LLMs continue to advance, understanding and mitigating such sensitivities will be crucial for their reliable and trustworthy deployment.

Title: Crossing New Frontiers: Knowledge-Augmented Large Language Model Prompting for Zero-Shot Text-Based De Novo Molecule Design

• Authors: Sakhinana Sagar Srinivas, Venkataramana Runkana
• Subjects: cs.CL, cs.AI, cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2408.11866
• Pdf URL: https://arxiv.org/pdf/2408.11866
• Copy Paste: [[2408.11866]] Crossing New Frontiers: Knowledge-Augmented Large Language Model Prompting for Zero-Shot Text-Based De Novo Molecule Design(https://arxiv.org/abs/2408.11866)
• Keywords: large language model
• Abstract: Molecule design is a multifaceted approach that leverages computational methods and experiments to optimize molecular properties, fast-tracking new drug discoveries, innovative material development, and more efficient chemical processes. Recently, text-based molecule design has emerged, inspired by next-generation AI tasks analogous to foundational vision-language models. Our study explores the use of knowledge-augmented prompting of large language models (LLMs) for the zero-shot text-conditional de novo molecular generation task. Our approach uses task-specific instructions and a few demonstrations to address distributional shift challenges when constructing augmented prompts for querying LLMs to generate molecules consistent with technical descriptions. Our framework proves effective, outperforming state-of-the-art (SOTA) baseline models on benchmark datasets.

Title: Enhance Lifelong Model Editing with Continuous Data-Adapter Association

• Authors: Jiaang Li, Quan Wang, Zhongnan Wang, Yongdong Zhang, Zhendong Mao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11869
• Pdf URL: https://arxiv.org/pdf/2408.11869
• Copy Paste: [[2408.11869]] Enhance Lifelong Model Editing with Continuous Data-Adapter Association(https://arxiv.org/abs/2408.11869)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors. Most model editing methods are solely designed for single-time use and lead to a significant forgetting effect after sequential edits over time, referred to as lifelong editing. Current approaches manage sequential edits by freezing original parameters and allocating new adapters for each knowledge modification. However, these methods lack robustness to minor input variations. To address this challenge, we propose ELDER, \textbf{E}nhancing \textbf{L}ifelong mo\textbf{D}el \textbf{E}diting with mixtu\textbf{R}e of Low-Rank Adapter (LoRA). ELDER is an adaptive approach that integrates multiple LoRAs through a router network. It learns to create a continuous and smooth association between data and adapters, thereby enhancing robustness and generalization to semantically equivalent inputs. Additionally, we introduce a novel loss to help learn associations between adapter allocations and edit semantics. A deferral mechanism is also proposed to retain the original LLM capabilities post-edit. Extensive experiments on GPT-2 XL and LLaMA2-7B demonstrate that ELDER effectively edits models in the lifelong setting and exhibits strong scalability, while retaining LLM's general abilities on downstream tasks.

Title: MegaFake: A Theory-Driven Dataset of Fake News Generated by Large Language Models

• Authors: Lionel Z. Wang, Yiming Ma, Renfei Gao, Beichen Guo, Zhuoran Li, Han Zhu, Wenqi Fan, Zexin Lu, Ka Chung Ng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11871
• Pdf URL: https://arxiv.org/pdf/2408.11871
• Copy Paste: [[2408.11871]] MegaFake: A Theory-Driven Dataset of Fake News Generated by Large Language Models(https://arxiv.org/abs/2408.11871)
• Keywords: large language model
• Abstract: The advent of large language models (LLMs) has revolutionized online content creation, making it much easier to generate high-quality fake news. This misuse threatens the integrity of our digital environment and ethical standards. Therefore, understanding the motivations and mechanisms behind LLM-generated fake news is crucial. In this study, we analyze the creation of fake news from a social psychology perspective and develop a comprehensive LLM-based theoretical framework, LLM-Fake Theory. We introduce a novel pipeline that automates the generation of fake news using LLMs, thereby eliminating the need for manual annotation. Utilizing this pipeline, we create a theoretically informed Machine-generated Fake news dataset, MegaFake, derived from the GossipCop dataset. We conduct comprehensive analyses to evaluate our MegaFake dataset. We believe that our dataset and insights will provide valuable contributions to future research focused on the detection and governance of fake news in the era of LLMs.

Title: Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

• Authors: Qianqian Xie, Dong Li, Mengxi Xiao, Zihao Jiang, Ruoyu Xiang, Xiao Zhang, Zhengyu Chen, Yueru He, Weiguang Han, Yuzhe Yang, Shunian Chen, Yifei Zhang, Lihang Shen, Daniel Kim, Zhiwei Liu, Zheheng Luo, Yangyang Yu, Yupeng Cao, Zhiyang Deng, Zhiyuan Yao, Haohang Li, Duanyu Feng, Yongfu Dai, VijayaSai Somasundaram, Peng Lu, Yilun Zhao, Yitao Long, Guojun Xiong, Kaleb Smith, Honghai Yu, Yanzhao Lai, Min Peng, Jianyun Nie, Jordan W. Suchow, Xiao-Yang Liu, Benyou Wang, Alejandro Lopez-Lira, Jimin Huang, Sophia Ananiadou
• Subjects: cs.CL, cs.CE, q-fin.CP
• Abstract URL: https://arxiv.org/abs/2408.11878
• Pdf URL: https://arxiv.org/pdf/2408.11878
• Copy Paste: [[2408.11878]] Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications(https://arxiv.org/abs/2408.11878)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have advanced financial applications, yet they often lack sufficient financial knowledge and struggle with tasks involving multi-modal inputs like tables and time series data. To address these limitations, we introduce \textit{Open-FinLLMs}, a series of Financial LLMs. We begin with FinLLaMA, pre-trained on a 52 billion token financial corpus, incorporating text, tables, and time-series data to embed comprehensive financial knowledge. FinLLaMA is then instruction fine-tuned with 573K financial instructions, resulting in FinLLaMA-instruct, which enhances task performance. Finally, we present FinLLaVA, a multimodal LLM trained with 1.43M image-text instructions to handle complex financial data types. Extensive evaluations demonstrate FinLLaMA's superior performance over LLaMA3-8B, LLaMA3.1-8B, and BloombergGPT in both zero-shot and few-shot settings across 19 and 4 datasets, respectively. FinLLaMA-instruct outperforms GPT-4 and other Financial LLMs on 15 datasets. FinLLaVA excels in understanding tables and charts across 4 multimodal tasks. Additionally, FinLLaMA achieves impressive Sharpe Ratios in trading simulations, highlighting its robust financial application capabilities. We will continually maintain and improve our models and benchmarks to support ongoing innovation in academia and industry.

Title: Beyond Labels: Aligning Large Language Models with Human-like Reasoning

• Authors: Muhammad Rafsan Kabir, Rafeed Mohammad Sultan, Ihsanul Haque Asif, Jawad Ibn Ahad, Fuad Rahman, Mohammad Ruhul Amin, Nabeel Mohammed, Shafin Rahman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11879
• Pdf URL: https://arxiv.org/pdf/2408.11879
• Copy Paste: [[2408.11879]] Beyond Labels: Aligning Large Language Models with Human-like Reasoning(https://arxiv.org/abs/2408.11879)
• Keywords: large language model
• Abstract: Aligning large language models (LLMs) with a human reasoning approach ensures that LLMs produce morally correct and human-like decisions. Ethical concerns are raised because current models are prone to generating false positives and providing malicious responses. To contribute to this issue, we have curated an ethics dataset named Dataset for Aligning Reasons (DFAR), designed to aid in aligning language models to generate human-like reasons. The dataset comprises statements with ethical-unethical labels and their corresponding reasons. In this study, we employed a unique and novel fine-tuning approach that utilizes ethics labels and their corresponding reasons (L+R), in contrast to the existing fine-tuning approach that only uses labels (L). The original pre-trained versions, the existing fine-tuned versions, and our proposed fine-tuned versions of LLMs were then evaluated on an ethical-unethical classification task and a reason-generation task. Our proposed fine-tuning strategy notably outperforms the others in both tasks, achieving significantly higher accuracy scores in the classification task and lower misalignment rates in the reason-generation task. The increase in classification accuracies and decrease in misalignment rates indicate that the L+R fine-tuned models align more with human ethics. Hence, this study illustrates that injecting reasons has substantially improved the alignment of LLMs, resulting in more human-like responses. We have made the DFAR dataset and corresponding codes publicly available at this https URL.

Title: Ancient Wisdom, Modern Tools: Exploring Retrieval-Augmented LLMs for Ancient Indian Philosophy

• Authors: Priyanka Mandikal
• Subjects: cs.CL, cs.CY, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.11903
• Pdf URL: https://arxiv.org/pdf/2408.11903
• Copy Paste: [[2408.11903]] Ancient Wisdom, Modern Tools: Exploring Retrieval-Augmented LLMs for Ancient Indian Philosophy(https://arxiv.org/abs/2408.11903)
• Keywords: large language model
• Abstract: LLMs have revolutionized the landscape of information retrieval and knowledge dissemination. However, their application in specialized areas is often hindered by factual inaccuracies and hallucinations, especially in long-tail knowledge distributions. We explore the potential of retrieval-augmented generation (RAG) models for long-form question answering (LFQA) in a specialized knowledge domain. We present VedantaNY-10M, a dataset curated from extensive public discourses on the ancient Indian philosophy of Advaita Vedanta. We develop and benchmark a RAG model against a standard, non-RAG LLM, focusing on transcription, retrieval, and generation performance. Human evaluations by computational linguists and domain experts show that the RAG model significantly outperforms the standard model in producing factual and comprehensive responses having fewer hallucinations. In addition, a keyword-based hybrid retriever that emphasizes unique low-frequency terms further improves results. Our study provides insights into effectively integrating modern large language models with ancient knowledge systems. Project page with dataset and code: this https URL

Title: Neural Symbolic Logical Rule Learner for Interpretable Learning

• Authors: Bowen Wei, Ziwei Zhu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11918
• Pdf URL: https://arxiv.org/pdf/2408.11918
• Copy Paste: [[2408.11918]] Neural Symbolic Logical Rule Learner for Interpretable Learning(https://arxiv.org/abs/2408.11918)
• Keywords: interpretability
• Abstract: Rule-based neural networks stand out for enabling interpretable classification by learning logical rules for both prediction and interpretation. However, existing models often lack flexibility due to the fixed model structure. Addressing this, we introduce the Normal Form Rule Learner (NFRL) algorithm, leveraging a selective discrete neural network, that treat weight parameters as hard selectors, to learn rules in both Conjunctive Normal Form (CNF) and Disjunctive Normal Form (DNF) for enhanced accuracy and interpretability. Instead of adopting a deep, complex structure, the NFRL incorporates two specialized Normal Form Layers (NFLs) with adaptable AND/OR neurons, a Negation Layer for input negations, and a Normal Form Constraint (NFC) to streamline neuron connections. We also show the novel network architecture can be optimized using adaptive gradient update together with Straight-Through Estimator to overcome the gradient vanishing challenge. Through extensive experiments on 11 datasets, NFRL demonstrates superior classification performance, quality of learned rules, efficiency and interpretability compared to 12 state-of-the-art alternatives. Code and data are available at \url{https://anonymous.4open.science/r/NFRL-27B4/}.

Title: Evaluation of Hash Algorithm Performance for Cryptocurrency Exchanges Based on Blockchain System

• Authors: Abel C. H. Chen
• Subjects: cs.CR, cs.PF
• Abstract URL: https://arxiv.org/abs/2408.11950
• Pdf URL: https://arxiv.org/pdf/2408.11950
• Copy Paste: [[2408.11950]] Evaluation of Hash Algorithm Performance for Cryptocurrency Exchanges Based on Blockchain System(https://arxiv.org/abs/2408.11950)
• Keywords: security
• Abstract: The blockchain system has emerged as one of the focal points of research in recent years, particularly in applications and services such as cryptocurrencies and smart contracts. In this context, the hash value serves as a crucial element in linking blocks within the blockchain, ensuring the integrity of block contents. Therefore, hash algorithms represent a vital security technology for ensuring the integrity and security of blockchain systems. This study primarily focuses on analyzing the security and execution efficiency of mainstream hash algorithms in the Proof of Work (PoW) calculations within blockchain systems. It proposes an evaluation factor and conducts comparative experiments to evaluate each hash algorithm. The experimental results indicate that there are no significant differences in the security aspects among SHA-2, SHA-3, and BLAKE2. However, SHA-2 and BLAKE2 demonstrate shorter computation times, indicating higher efficiency in execution.

Title: Decoding SEC Actions: Enforcement Trends through Analyzing Blockchain litigation using LLM-based Thematic Factor Mapping

• Authors: Junliang Luo, Xihan Xiong, William Knottenbelt, Xue Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11961
• Pdf URL: https://arxiv.org/pdf/2408.11961
• Copy Paste: [[2408.11961]] Decoding SEC Actions: Enforcement Trends through Analyzing Blockchain litigation using LLM-based Thematic Factor Mapping(https://arxiv.org/abs/2408.11961)
• Keywords: large language model
• Abstract: The proliferation of blockchain entities (persons or enterprises) exposes them to potential regulatory actions (e.g., being litigated) by regulatory authorities. Regulatory frameworks for crypto assets are actively being developed and refined, increasing the likelihood of such actions. The lack of systematic analysis of the factors driving litigation against blockchain entities leaves companies in need of clarity to navigate compliance risks. This absence of insight also deprives investors of the information for informed decision-making. This study focuses on U.S. litigation against blockchain entities, particularly by the U.S. Securities and Exchange Commission (SEC) given its influence on global crypto regulation. Utilizing frontier pretrained language models and large language models, we systematically map all SEC complaints against blockchain companies from 2012 to 2024 to thematic factors conceptualized by our study to delineate the factors driving SEC actions. We quantify the thematic factors and assess their influence on specific legal Acts cited within the complaints on an annual basis, allowing us to discern the regulatory emphasis, patterns and conduct trend analysis.

Title: Real-Time Incremental Explanations for Object Detectors

• Authors: Santiago Calderón-Peña, Hana Chockler, David A. Kelly
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11963
• Pdf URL: https://arxiv.org/pdf/2408.11963
• Copy Paste: [[2408.11963]] Real-Time Incremental Explanations for Object Detectors(https://arxiv.org/abs/2408.11963)
• Keywords: explainability
• Abstract: Existing black box explainability tools for object detectors rely on multiple calls to the model, which prevents them from computing explanations in real time. In this paper we introduce IncX, an algorithm for real-time incremental approximations of explanations, based on linear transformations of saliency maps. We implement IncX on top of D-RISE, a state-of-the-art black-box explainability tool for object detectors. We show that IncX's explanations are comparable in quality to those of D-RISE, with insertion curves being within 8%, and are computed two orders of magnitude faster that D-RISE's explanations.

Title: Visual Localization in 3D Maps: Comparing Point Cloud, Mesh, and NeRF Representations

• Authors: Lintong Zhang, Yifu Tao, Jiarong Lin, Fu Zhang, Maurice Fallon
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.11966
• Pdf URL: https://arxiv.org/pdf/2408.11966
• Copy Paste: [[2408.11966]] Visual Localization in 3D Maps: Comparing Point Cloud, Mesh, and NeRF Representations(https://arxiv.org/abs/2408.11966)
• Keywords: robust
• Abstract: This paper introduces and assesses a cross-modal global visual localization system that can localize camera images within a color 3D map representation built using both visual and lidar sensing. We present three different state-of-the-art methods for creating the color 3D maps: point clouds, meshes, and neural radiance fields (NeRF). Our system constructs a database of synthetic RGB and depth image pairs from these representations. This database serves as the basis for global localization. We present an automatic approach that builds this database by synthesizing novel images of the scene and exploiting the 3D structure encoded in the different representations. Next, we present a global localization system that relies on the synthetic image database to accurately estimate the 6 DoF camera poses of monocular query images. Our localization approach relies on different learning-based global descriptors and feature detectors which enable robust image retrieval and matching despite the domain gap between (real) query camera images and the synthetic database images. We assess the system's performance through extensive real-world experiments in both indoor and outdoor settings, in order to evaluate the effectiveness of each map representation and the benefits against traditional structure-from-motion localization approaches. Our results show that all three map representations can achieve consistent localization success rates of 55% and higher across various environments. NeRF synthesized images show superior performance, localizing query images at an average success rate of 72%. Furthermore, we demonstrate that our synthesized database enables global localization even when the map creation data and the localization sequence are captured when travelling in opposite directions. Our system, operating in real-time on a mobile laptop equipped with a GPU, achieves a processing rate of 1Hz.

Title: Two-Timescale Gradient Descent Ascent Algorithms for Nonconvex Minimax Optimization

• Authors: Tianyi Lin, Chi Jin, Michael. I. Jordan
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2408.11974
• Pdf URL: https://arxiv.org/pdf/2408.11974
• Copy Paste: [[2408.11974]] Two-Timescale Gradient Descent Ascent Algorithms for Nonconvex Minimax Optimization(https://arxiv.org/abs/2408.11974)
• Keywords: generative
• Abstract: We provide a unified analysis of two-timescale gradient descent ascent (TTGDA) for solving structured nonconvex minimax optimization problems in the form of $\min_\textbf{x} \max_{\textbf{y} \in Y} f(\textbf{x}, \textbf{y})$, where the objective function $f(\textbf{x}, \textbf{y})$ is nonconvex in $\textbf{x}$ and concave in $\textbf{y}$, and the constraint set $Y \subseteq \mathbb{R}^n$ is convex and bounded. In the convex-concave setting, the single-timescale GDA achieves strong convergence guarantees and has been used for solving application problems arising from operations research and computer science. However, it can fail to converge in more general settings. Our contribution in this paper is to design the simple deterministic and stochastic TTGDA algorithms that efficiently find one stationary point of the function $\Phi(\cdot) := \max_{\textbf{y} \in Y} f(\cdot, \textbf{y})$. Specifically, we prove the theoretical bounds on the complexity of solving both smooth and nonsmooth nonconvex-concave minimax optimization problems. To our knowledge, this is the first systematic analysis of TTGDA for nonconvex minimax optimization, shedding light on its superior performance in training generative adversarial networks (GANs) and in solving other real-world application problems.

Title: Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?

• Authors: Francesco Innocenti, El Mehdi Achour, Ryan Singh, Christopher L. Buckley
• Subjects: cs.LG, cs.AI, cs.NE, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.11979
• Pdf URL: https://arxiv.org/pdf/2408.11979
• Copy Paste: [[2408.11979]] Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?(https://arxiv.org/abs/2408.11979)
• Keywords: robust
• Abstract: Predictive coding (PC) is an energy-based learning algorithm that performs iterative inference over network activities before weight updates. Recent work suggests that PC can converge in fewer learning steps than backpropagation thanks to its inference procedure. However, these advantages are not always observed, and the impact of PC inference on learning is theoretically not well understood. Here, we study the geometry of the PC energy landscape at the (inference) equilibrium of the network activities. For deep linear networks, we first show that the equilibrated energy is simply a rescaled mean squared error loss with a weight-dependent rescaling. We then prove that many highly degenerate (non-strict) saddles of the loss including the origin become much easier to escape (strict) in the equilibrated energy. Our theory is validated by experiments on both linear and non-linear networks. Based on these results, we conjecture that all the saddles of the equilibrated energy are strict. Overall, this work suggests that PC inference makes the loss landscape more benign and robust to vanishing gradients, while also highlighting the challenge of speeding up PC inference on large-scale models.

Title: Large Language Models for Page Stream Segmentation

• Authors: Hunter Heidenreich, Ratish Dalvi, Rohith Mukku, Nikhil Verma, Neven Pičuljan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11981
• Pdf URL: https://arxiv.org/pdf/2408.11981
• Copy Paste: [[2408.11981]] Large Language Models for Page Stream Segmentation(https://arxiv.org/abs/2408.11981)
• Keywords: robust, large language model, segmentation
• Abstract: Page Stream Segmentation (PSS) is an essential prerequisite for automated document processing at scale. However, research progress has been limited by the absence of realistic public benchmarks. This paper works towards addressing this gap by introducing TABME++, an enhanced benchmark featuring commercial Optical Character Recognition (OCR) annotations. We evaluate the performance of large language models (LLMs) on PSS, focusing on decoder-based models fine-tuned with parameter-efficient methods. Our results show that decoder-based LLMs outperform smaller multimodal encoders. Through a review of existing PSS research and datasets, we identify key challenges and advancements in the field. Our findings highlight the key importance of robust OCR, providing valuable insights for the development of more effective document processing systems.

Title: Time Series Foundation Models and Deep Learning Architectures for Earthquake Temporal and Spatial Nowcasting

• Authors: Alireza Jafari, Geoffrey Fox, John B. Rundle, Andrea Donnellan, Lisa Grant Ludwig
• Subjects: cs.LG, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2408.11990
• Pdf URL: https://arxiv.org/pdf/2408.11990
• Copy Paste: [[2408.11990]] Time Series Foundation Models and Deep Learning Architectures for Earthquake Temporal and Spatial Nowcasting(https://arxiv.org/abs/2408.11990)
• Keywords: transformer
• Abstract: Advancing the capabilities of earthquake nowcasting, the real-time forecasting of seismic activities remains a crucial and enduring objective aimed at reducing casualties. This multifaceted challenge has recently gained attention within the deep learning domain, facilitated by the availability of extensive, long-term earthquake datasets. Despite significant advancements, existing literature on earthquake nowcasting lacks comprehensive evaluations of pre-trained foundation models and modern deep learning architectures. These architectures, such as transformers or graph neural networks, uniquely focus on different aspects of data, including spatial relationships, temporal patterns, and multi-scale dependencies. This paper addresses the mentioned gap by analyzing different architectures and introducing two innovation approaches called MultiFoundationQuake and GNNCoder. We formulate earthquake nowcasting as a time series forecasting problem for the next 14 days within 0.1-degree spatial bins in Southern California, spanning from 1986 to 2024. Earthquake time series is forecasted as a function of logarithm energy released by quakes. Our comprehensive evaluation employs several key performance metrics, notably Nash-Sutcliffe Efficiency and Mean Squared Error, over time in each spatial region. The results demonstrate that our introduced models outperform other custom architectures by effectively capturing temporal-spatial relationships inherent in seismic data. The performance of existing foundation models varies significantly based on the pre-training datasets, emphasizing the need for careful dataset selection. However, we introduce a new general approach termed MultiFoundationPattern that combines a bespoke pattern with foundation model results handled as auxiliary streams. In the earthquake case, the resultant MultiFoundationQuake model achieves the best overall performance.

Title: RAG-Optimized Tibetan Tourism LLMs: Enhancing Accuracy and Personalization

• Authors: Jinhu Qi, Shuai Yan, Yibo Zhang, Wentao Zhang, Rong Jin, Yuwei Hu, Ke Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12003
• Pdf URL: https://arxiv.org/pdf/2408.12003
• Copy Paste: [[2408.12003]] RAG-Optimized Tibetan Tourism LLMs: Enhancing Accuracy and Personalization(https://arxiv.org/abs/2408.12003)
• Keywords: large language model
• Abstract: With the development of the modern social economy, tourism has become an important way to meet people's spiritual needs, bringing development opportunities to the tourism industry. However, existing large language models (LLMs) face challenges in personalized recommendation capabilities and the generation of content that can sometimes produce hallucinations. This study proposes an optimization scheme for Tibet tourism LLMs based on retrieval-augmented generation (RAG) technology. By constructing a database of tourist viewpoints and processing the data using vectorization techniques, we have significantly improved retrieval accuracy. The application of RAG technology effectively addresses the hallucination problem in content generation. The optimized model shows significant improvements in fluency, accuracy, and relevance of content generation. This research demonstrates the potential of RAG technology in the standardization of cultural tourism information and data analysis, providing theoretical and technical support for the development of intelligent cultural tourism service systems.

Title: Energy Estimation of Last Mile Electric Vehicle Routes

• Authors: André Snoeck, Aniruddha Bhargava, Daniel Merchan, Josiah Davis, Julian Pachon
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12006
• Pdf URL: https://arxiv.org/pdf/2408.12006
• Copy Paste: [[2408.12006]] Energy Estimation of Last Mile Electric Vehicle Routes(https://arxiv.org/abs/2408.12006)
• Keywords: transformer
• Abstract: Last-mile carriers increasingly incorporate electric vehicles (EVs) into their delivery fleet to achieve sustainability goals. This goal presents many challenges across multiple planning spaces including but not limited to how to plan EV routes. In this paper, we address the problem of predicting energy consumption of EVs for Last-Mile delivery routes using deep learning. We demonstrate the need to move away from thinking about range and we propose using energy as the basic unit of analysis. We share a range of deep learning solutions, beginning with a Feed Forward Neural Network (NN) and Recurrent Neural Network (RNN) and demonstrate significant accuracy improvements relative to pure physics-based and distance-based approaches. Finally, we present Route Energy Transformer (RET) a decoder-only Transformer model sized according to Chinchilla scaling laws. RET yields a +217 Basis Points (bps) improvement in Mean Absolute Percentage Error (MAPE) relative to the Feed Forward NN and a +105 bps improvement relative to the RNN.

Title: QuaCK-TSF: Quantum-Classical Kernelized Time Series Forecasting

• Authors: Abdallah Aaraba, Soumaya Cherkaoui, Ola Ahmad, Jean-Frédéric Laprade, Olivier Nahman-Lévesque, Alexis Vieloszynski, Shengrui Wang
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.12007
• Pdf URL: https://arxiv.org/pdf/2408.12007
• Copy Paste: [[2408.12007]] QuaCK-TSF: Quantum-Classical Kernelized Time Series Forecasting(https://arxiv.org/abs/2408.12007)
• Keywords: robust
• Abstract: Forecasting in probabilistic time series is a complex endeavor that extends beyond predicting future values to also quantifying the uncertainty inherent in these predictions. Gaussian process regression stands out as a Bayesian machine learning technique adept at addressing this multifaceted challenge. This paper introduces a novel approach that blends the robustness of this Bayesian technique with the nuanced insights provided by the kernel perspective on quantum models, aimed at advancing quantum kernelized probabilistic forecasting. We incorporate a quantum feature map inspired by Ising interactions and demonstrate its effectiveness in capturing the temporal dependencies critical for precise forecasting. The optimization of our model's hyperparameters circumvents the need for computationally intensive gradient descent by employing gradient-free Bayesian optimization. Comparative benchmarks against established classical kernel models are provided, affirming that our quantum-enhanced approach achieves competitive performance.

Title: CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion

• Authors: Yunlong Tang, Gen Zhan, Li Yang, Yiting Liao, Chenliang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12009
• Pdf URL: https://arxiv.org/pdf/2408.12009
• Copy Paste: [[2408.12009]] CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion(https://arxiv.org/abs/2408.12009)
• Keywords: diffusion, large language model
• Abstract: Video saliency prediction aims to identify the regions in a video that attract human attention and gaze, driven by bottom-up features from the video and top-down processes like memory and cognition. Among these top-down influences, language plays a crucial role in guiding attention by shaping how visual information is interpreted. Existing methods primarily focus on modeling perceptual information while neglecting the reasoning process facilitated by language, where ranking cues are crucial outcomes of this process and practical guidance for saliency prediction. In this paper, we propose CaRDiff (Caption, Rank, and generate with Diffusion), a framework that imitates the process by integrating a multimodal large language model (MLLM), a grounding module, and a diffusion model, to enhance video saliency prediction. Specifically, we introduce a novel prompting method VSOR-CoT (Video Salient Object Ranking Chain of Thought), which utilizes an MLLM with a grounding module to caption video content and infer salient objects along with their rankings and positions. This process derives ranking maps that can be sufficiently leveraged by the diffusion model to decode the saliency maps for the given video accurately. Extensive experiments show the effectiveness of VSOR-CoT in improving the performance of video saliency prediction. The proposed CaRDiff performs better than state-of-the-art models on the MVS dataset and demonstrates cross-dataset capabilities on the DHF1k dataset through zero-shot evaluation.

Title: Confounding Privacy and Inverse Composition

• Authors: Tao Zhang, Bradley A. Malin, Netanel Raviv, Yevgeniy Vorobeychik
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.12010
• Pdf URL: https://arxiv.org/pdf/2408.12010
• Copy Paste: [[2408.12010]] Confounding Privacy and Inverse Composition(https://arxiv.org/abs/2408.12010)
• Keywords: privacy
• Abstract: We introduce a novel privacy notion of ($\epsilon, \delta$)-confounding privacy that generalizes both differential privacy and Pufferfish privacy. In differential privacy, sensitive information is contained in the dataset while in Pufferfish privacy, sensitive information determines data distribution. Consequently, both assume a chain-rule relationship between the sensitive information and the output of privacy mechanisms. Confounding privacy, in contrast, considers general causal relationships between the dataset and sensitive information. One of the key properties of differential privacy is that it can be easily composed over multiple interactions with the mechanism that maps private data to publicly shared information. In contrast, we show that the quantification of the privacy loss under the composition of independent ($\epsilon, \delta$)-confounding private mechanisms using the optimal composition of differential privacy \emph{underestimates} true privacy loss. To address this, we characterize an inverse composition framework to tightly implement a target global ($\epsilon_{g}, \delta_{g}$)-confounding privacy under composition while keeping individual mechanisms independent and private. In particular, we propose a novel copula-perturbation method which ensures that (1) each individual mechanism $i$ satisfies a target local ($\epsilon_{i}, \delta_{i}$)-confounding privacy and (2) the target global ($\epsilon_{g}, \delta_{g}$)-confounding privacy is tightly implemented by solving an optimization problem. Finally, we study inverse composition empirically on real datasets.

Title: R-STELLAR: A Resilient Synthesizable Signature Attenuation SCA Protection on AES-256 with built-in Attack-on-Countermeasure Detection

• Authors: Archisman Ghosh, Dong-Hyun Seo, Debayan Das, Santosh Ghosh, Shreyas Sen
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.12021
• Pdf URL: https://arxiv.org/pdf/2408.12021
• Copy Paste: [[2408.12021]] R-STELLAR: A Resilient Synthesizable Signature Attenuation SCA Protection on AES-256 with built-in Attack-on-Countermeasure Detection(https://arxiv.org/abs/2408.12021)
• Keywords: secure, security, protect, attack
• Abstract: Side channel attacks (SCAs) remain a significant threat to the security of cryptographic systems in modern embedded devices. Even mathematically secure cryptographic algorithms, when implemented in hardware, inadvertently leak information through physical side channel signatures such as power consumption, electromagnetic (EM) radiation, light emissions, and acoustic emanations. Exploiting these side channels significantly reduces the search space of the attacker. In recent years, physical countermeasures have significantly increased the minimum traces to disclosure (MTD) to 1 billion. Among them, signature attenuation is the first method to achieve this mark. Signature attenuation often relies on analog techniques, and digital signature attenuation reduces MTD to 20 million, requiring additional methods for high resilience. We focus on improving the digital signature attenuation by an order of magnitude (MTD 200M). Additionally, we explore possible attacks against signature attenuation countermeasure. We introduce a Voltage drop Linear region Biasing (VLB) attack technique that reduces the MTD to over 2000 times less than the previous threshold. This is the first known attack against a physical side-channel attack (SCA) countermeasure. We have implemented an attack detector with a response time of 0.8 milliseconds to detect such attacks, limiting SCA leakage window to sub-ms, which is insufficient for a successful attack.

Title: Understanding Epistemic Language with a Bayesian Theory of Mind

• Authors: Lance Ying, Tan Zhi-Xuan, Lionel Wong, Vikash Mansinghka, Joshua B. Tenenbaum
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12022
• Pdf URL: https://arxiv.org/pdf/2408.12022
• Copy Paste: [[2408.12022]] Understanding Epistemic Language with a Bayesian Theory of Mind(https://arxiv.org/abs/2408.12022)
• Keywords: generative
• Abstract: How do people understand and evaluate claims about others' beliefs, even though these beliefs cannot be directly observed? In this paper, we introduce a cognitive model of epistemic language interpretation, grounded in Bayesian inferences about other agents' goals, beliefs, and intentions: a language-augmented Bayesian theory-of-mind (LaBToM). By translating natural language into an epistemic ``language-of-thought'', then evaluating these translations against the inferences produced by inverting a probabilistic generative model of rational action and perception, LaBToM captures graded plausibility judgments about epistemic claims. We validate our model in an experiment where participants watch an agent navigate a maze to find keys hidden in boxes needed to reach their goal, then rate sentences about the agent's beliefs. In contrast with multimodal LLMs (GPT-4o, Gemini Pro) and ablated models, our model correlates highly with human judgments for a wide range of expressions, including modal language, uncertainty expressions, knowledge claims, likelihood comparisons, and attributions of false belief.

Title: ISETHDR: A Physics-based Synthetic Radiance Dataset for High Dynamic Range Driving Scenes

• Authors: Zhenyi Liu, Devesh Shah, Brian Wandell
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.12048
• Pdf URL: https://arxiv.org/pdf/2408.12048
• Copy Paste: [[2408.12048]] ISETHDR: A Physics-based Synthetic Radiance Dataset for High Dynamic Range Driving Scenes(https://arxiv.org/abs/2408.12048)
• Keywords: segmentation
• Abstract: This paper describes a physics-based end-to-end software simulation for image systems. We use the software to explore sensors designed to enhance performance in high dynamic range (HDR) environments, such as driving through daytime tunnels and under nighttime conditions. We synthesize physically realistic HDR spectral radiance images and use them as the input to digital twins that model the optics and sensors of different systems. This paper makes three main contributions: (a) We create a labeled (instance segmentation and depth), synthetic radiance dataset of HDR driving scenes. (b) We describe the development and validation of the end-to-end simulation framework. (c) We present a comparative analysis of two single-shot sensors designed for HDR. We open-source both the dataset and the software.

Title: Aligning (Medical) LLMs for (Counterfactual) Fairness

• Authors: Raphael Poulain, Hamed Fayyaz, Rahmatollah Beheshti
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12055
• Pdf URL: https://arxiv.org/pdf/2408.12055
• Copy Paste: [[2408.12055]] Aligning (Medical) LLMs for (Counterfactual) Fairness(https://arxiv.org/abs/2408.12055)
• Keywords: protect, fair, large language model
• Abstract: Large Language Models (LLMs) have emerged as promising solutions for a variety of medical and clinical decision support applications. However, LLMs are often subject to different types of biases, which can lead to unfair treatment of individuals, worsening health disparities, and reducing trust in AI-augmented medical tools. Aiming to address this important issue, in this study, we present a new model alignment approach for aligning LLMs using a preference optimization method within a knowledge distillation framework. Prior to presenting our proposed method, we first use an evaluation framework to conduct a comprehensive (largest to our knowledge) empirical evaluation to reveal the type and nature of existing biases in LLMs used for medical applications. We then offer a bias mitigation technique to reduce the unfair patterns in LLM outputs across different subgroups identified by the protected attributes. We show that our mitigation method is effective in significantly reducing observed biased patterns. Our code is publicly available at \url{this https URL}.

Title: Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs

• Authors: Ronit Singhal, Pransh Patwa, Parth Patwa, Aman Chadha, Amitava Das
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12060
• Pdf URL: https://arxiv.org/pdf/2408.12060
• Copy Paste: [[2408.12060]] Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs(https://arxiv.org/abs/2408.12060)
• Keywords: large language model
• Abstract: Given the widespread dissemination of misinformation on social media, implementing fact-checking mechanisms for online claims is essential. Manually verifying every claim is highly challenging, underscoring the need for an automated fact-checking system. This paper presents our system designed to address this issue. We utilize the Averitec dataset to assess the veracity of claims. In addition to veracity prediction, our system provides supporting evidence, which is extracted from the dataset. We develop a Retrieve and Generate (RAG) pipeline to extract relevant evidence sentences from a knowledge base, which are then inputted along with the claim into a large language model (LLM) for classification. We also evaluate the few-shot In-Context Learning (ICL) capabilities of multiple LLMs. Our system achieves an 'Averitec' score of 0.33, which is a 22% absolute improvement over the baseline. All code will be made available on All code will be made available on this https URL.

Title: Enhancing Sampling Protocol for Robust Point Cloud Classification

• Authors: Chongshou Li, Pin Tang, Xinke Li, Tianrui Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12062
• Pdf URL: https://arxiv.org/pdf/2408.12062
• Copy Paste: [[2408.12062]] Enhancing Sampling Protocol for Robust Point Cloud Classification(https://arxiv.org/abs/2408.12062)
• Keywords: robust
• Abstract: Established sampling protocols for 3D point cloud learning, such as Farthest Point Sampling (FPS) and Fixed Sample Size (FSS), have long been recognized and utilized. However, real-world data often suffer from corrputions such as sensor noise, which violates the benignness assumption of point cloud in current protocols. Consequently, they are notably vulnerable to noise, posing significant safety risks in critical applications like autonomous driving. To address these issues, we propose an enhanced point cloud sampling protocol, PointDR, which comprises two components: 1) Downsampling for key point identification and 2) Resampling for flexible sample size. Furthermore, differentiated strategies are implemented for training and inference processes. Particularly, an isolation-rated weight considering local density is designed for the downsampling method, assisting it in performing random key points selection in the training phase and bypassing noise in the inference phase. A local-geometry-preserved upsampling is incorporated into resampling, facilitating it to maintain a stochastic sample size in the training stage and complete insufficient data in the inference. It is crucial to note that the proposed protocol is free of model architecture altering and extra learning, thus minimal efforts are demanded for its replacement of the existing one. Despite the simplicity, it substantially improves the robustness of point cloud learning, showcased by outperforming the state-of-the-art methods on multiple benchmarks of corrupted point cloud classification. The code will be available upon the paper's acceptance.

Title: ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM

• Authors: Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, Yu Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12076
• Pdf URL: https://arxiv.org/pdf/2408.12076
• Copy Paste: [[2408.12076]] ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM(https://arxiv.org/abs/2408.12076)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved impressive advancements across numerous disciplines, yet the critical issue of knowledge conflicts, a major source of hallucinations, has rarely been studied. Only a few research explored the conflicts between the inherent knowledge of LLMs and the retrieved contextual knowledge. However, a thorough assessment of knowledge conflict in LLMs is still missing. Motivated by this research gap, we present ConflictBank, the first comprehensive benchmark developed to systematically evaluate knowledge conflicts from three aspects: (i) conflicts encountered in retrieved knowledge, (ii) conflicts within the models' encoded knowledge, and (iii) the interplay between these conflict forms. Our investigation delves into four model families and twelve LLM instances, meticulously analyzing conflicts stemming from misinformation, temporal discrepancies, and semantic divergences. Based on our proposed novel construction framework, we create 7,453,853 claim-evidence pairs and 553,117 QA pairs. We present numerous findings on model scale, conflict causes, and conflict types. We hope our ConflictBank benchmark will help the community better understand model behavior in conflicts and develop more reliable LLMs.

Title: High-Quality Data Augmentation for Low-Resource NMT: Combining a Translation Memory, a GAN Generator, and Filtering

• Authors: Hengjie Liu, Ruibo Hou, Yves Lepage
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12079
• Pdf URL: https://arxiv.org/pdf/2408.12079
• Copy Paste: [[2408.12079]] High-Quality Data Augmentation for Low-Resource NMT: Combining a Translation Memory, a GAN Generator, and Filtering(https://arxiv.org/abs/2408.12079)
• Keywords: generative
• Abstract: Back translation, as a technique for extending a dataset, is widely used by researchers in low-resource language translation tasks. It typically translates from the target to the source language to ensure high-quality translation results. This paper proposes a novel way of utilizing a monolingual corpus on the source side to assist Neural Machine Translation (NMT) in low-resource settings. We realize this concept by employing a Generative Adversarial Network (GAN), which augments the training data for the discriminator while mitigating the interference of low-quality synthetic monolingual translations with the generator. Additionally, this paper integrates Translation Memory (TM) with NMT, increasing the amount of data available to the generator. Moreover, we propose a novel procedure to filter the synthetic sentence pairs during the augmentation process, ensuring the high quality of the data.

Title: Towards Threat Modelling of IoT Context-Sharing Platforms

• Authors: Mohammad Goudarzi, Arash Shaghaghi, Simon Finn, Burkhard Stiller, Sanjay Jha
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.12081
• Pdf URL: https://arxiv.org/pdf/2408.12081
• Copy Paste: [[2408.12081]] Towards Threat Modelling of IoT Context-Sharing Platforms(https://arxiv.org/abs/2408.12081)
• Keywords: security
• Abstract: The Internet of Things (IoT) involves complex, interconnected systems and devices that depend on context-sharing platforms for interoperability and information exchange. These platforms are, therefore, critical components of real-world IoT deployments, making their security essential to ensure the resilience and reliability of these 'systems of systems'. In this paper, we take the first steps toward systematically and comprehensively addressing the security of IoT context-sharing platforms. We propose a framework for threat modelling and security analysis of a generic IoT context-sharing solution, employing the MITRE ATT&CK framework. Through an evaluation of various industry-funded projects and academic research, we identify significant security challenges in the design of IoT context-sharing platforms. Our threat modelling provides an in-depth analysis of the techniques and sub-techniques adversaries may use to exploit these systems, offering valuable insights for future research aimed at developing resilient solutions. Additionally, we have developed an open-source threat analysis tool that incorporates our detailed threat modelling, which can be used to evaluate and enhance the security of existing context-sharing platforms.

Title: Vision-Based Detection of Uncooperative Targets and Components on Small Satellites

• Authors: Hannah Grauer, Elena-Sorina Lupu, Connor Lee, Soon-Jo Chung, Darren Rowen, Benjamen Bycroft, Phaedrus Leeds, John Brader
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12084
• Pdf URL: https://arxiv.org/pdf/2408.12084
• Copy Paste: [[2408.12084]] Vision-Based Detection of Uncooperative Targets and Components on Small Satellites(https://arxiv.org/abs/2408.12084)
• Keywords: robust, segmentation
• Abstract: Space debris and inactive satellites pose a threat to the safety and integrity of operational spacecraft and motivate the need for space situational awareness techniques. These uncooperative targets create a challenging tracking and detection problem due to a lack of prior knowledge of their features, trajectories, or even existence. Recent advancements in computer vision models can be used to improve upon existing methods for tracking such uncooperative targets to make them more robust and reliable to the wide-ranging nature of the target. This paper introduces an autonomous detection model designed to identify and monitor these objects using learning and computer vision. The autonomous detection method aims to identify and accurately track the uncooperative targets in varied circumstances, including different camera spectral sensitivities, lighting, and backgrounds. Our method adapts to the relative distance between the observing spacecraft and the target, and different detection strategies are adjusted based on distance. At larger distances, we utilize You Only Look Once (YOLOv8), a multitask Convolutional Neural Network (CNN), for zero-shot and domain-specific single-shot real time detection of the target. At shorter distances, we use knowledge distillation to combine visual foundation models with a lightweight fast segmentation CNN (Fast-SCNN) to segment the spacecraft components with low storage requirements and fast inference times, and to enable weight updates from earth and possible onboard training. Lastly, we test our method on a custom dataset simulating the unique conditions encountered in space, as well as a publicly-available dataset.

Title: Unlocking Attributes' Contribution to Successful Camouflage: A Combined Textual and VisualAnalysis Strategy

• Authors: Hong Zhang, Yixuan Lyu, Qian Yu, Hanyang Liu, Huimin Ma, Ding Yuan, Yifan Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12086
• Pdf URL: https://arxiv.org/pdf/2408.12086
• Copy Paste: [[2408.12086]] Unlocking Attributes' Contribution to Successful Camouflage: A Combined Textual and VisualAnalysis Strategy(https://arxiv.org/abs/2408.12086)
• Keywords: robust, segmentation
• Abstract: In the domain of Camouflaged Object Segmentation (COS), despite continuous improvements in segmentation performance, the underlying mechanisms of effective camouflage remain poorly understood, akin to a black box. To address this gap, we present the first comprehensive study to examine the impact of camouflage attributes on the effectiveness of camouflage patterns, offering a quantitative framework for the evaluation of camouflage designs. To support this analysis, we have compiled the first dataset comprising descriptions of camouflaged objects and their attribute contributions, termed COD-Text And X-attributions (COD-TAX). Moreover, drawing inspiration from the hierarchical process by which humans process information: from high-level textual descriptions of overarching scenarios, through mid-level summaries of local areas, to low-level pixel data for detailed analysis. We have developed a robust framework that combines textual and visual information for the task of COS, named Attribution CUe Modeling with Eye-fixation Network (ACUMEN). ACUMEN demonstrates superior performance, outperforming nine leading methods across three widely-used datasets. We conclude by highlighting key insights derived from the attributes identified in our study. Code: this https URL.

Title: Extraction of Research Objectives, Machine Learning Model Names, and Dataset Names from Academic Papers and Analysis of Their Interrelationships Using LLM and Network Analysis

• Authors: S. Nishio, H. Nonaka, N. Tsuchiya, A. Migita, Y. Banno, T. Hayashi, H. Sakaji, T. Sakumoto, K. Watabe
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12097
• Pdf URL: https://arxiv.org/pdf/2408.12097
• Copy Paste: [[2408.12097]] Extraction of Research Objectives, Machine Learning Model Names, and Dataset Names from Academic Papers and Analysis of Their Interrelationships Using LLM and Network Analysis(https://arxiv.org/abs/2408.12097)
• Keywords: extraction
• Abstract: Machine learning is widely utilized across various industries. Identifying the appropriate machine learning models and datasets for specific tasks is crucial for the effective industrial application of machine learning. However, this requires expertise in both machine learning and the relevant domain, leading to a high learning cost. Therefore, research focused on extracting combinations of tasks, machine learning models, and datasets from academic papers is critically important, as it can facilitate the automatic recommendation of suitable methods. Conventional information extraction methods from academic papers have been limited to identifying machine learning models and other entities as named entities. To address this issue, this study proposes a methodology extracting tasks, machine learning methods, and dataset names from scientific papers and analyzing the relationships between these information by using LLM, embedding model, and network clustering. The proposed method's expression extraction performance, when using Llama3, achieves an F-score exceeding 0.8 across various categories, confirming its practical utility. Benchmarking results on financial domain papers have demonstrated the effectiveness of this method, providing insights into the use of the latest datasets, including those related to ESG (Environmental, Social, and Governance) data.

Title: Query-Efficient Video Adversarial Attack with Stylized Logo

• Authors: Duoxun Tang, Yuxin Cao, Xi Xiao, Derui Wang, Sheng Wen, Tianqing Zhu
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.12099
• Pdf URL: https://arxiv.org/pdf/2408.12099
• Copy Paste: [[2408.12099]] Query-Efficient Video Adversarial Attack with Stylized Logo(https://arxiv.org/abs/2408.12099)
• Keywords: defense, attack
• Abstract: Video classification systems based on Deep Neural Networks (DNNs) have demonstrated excellent performance in accurately verifying video content. However, recent studies have shown that DNNs are highly vulnerable to adversarial examples. Therefore, a deep understanding of adversarial attacks can better respond to emergency situations. In order to improve attack performance, many style-transfer-based attacks and patch-based attacks have been proposed. However, the global perturbation of the former will bring unnatural global color, while the latter is difficult to achieve success in targeted attacks due to the limited perturbation space. Moreover, compared to a plethora of methods targeting image classifiers, video adversarial attacks are still not that popular. Therefore, to generate adversarial examples with a low budget and to provide them with a higher verisimilitude, we propose a novel black-box video attack framework, called Stylized Logo Attack (SLA). SLA is conducted through three steps. The first step involves building a style references set for logos, which can not only make the generated examples more natural, but also carry more target class features in the targeted attacks. Then, reinforcement learning (RL) is employed to determine the style reference and position parameters of the logo within the video, which ensures that the stylized logo is placed in the video with optimal attributes. Finally, perturbation optimization is designed to optimize perturbations to improve the fooling rate in a step-by-step manner. Sufficient experimental results indicate that, SLA can achieve better performance than state-of-the-art methods and still maintain good deception effects when facing various defense methods.

Title: RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data

• Authors: Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang, Qiaozhi He, Tong Xiao, Chunliang Zhang, Tongran Liu, Quan Du, Di Yang, Jingbo Zhu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12109
• Pdf URL: https://arxiv.org/pdf/2408.12109
• Copy Paste: [[2408.12109]] RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data(https://arxiv.org/abs/2408.12109)
• Keywords: robust
• Abstract: Large vision-language models (LVLMs) often fail to align with human preferences, leading to issues like generating misleading content without proper visual context (also known as hallucination). A promising solution to this problem is using human-preference alignment techniques, such as best-of-n sampling and reinforcement learning. However, these techniques face the difficulty arising from the scarcity of visual preference data, which is required to train a visual reward model (VRM). In this work, we continue the line of research. We present a Robust Visual Reward Model (RoVRM) which improves human-preference alignment for LVLMs. RoVRM leverages auxiliary textual preference data through a three-phase progressive training and optimal transport-based preference data selection to effectively mitigate the scarcity of visual preference data. We experiment with RoVRM on the commonly used vision-language tasks based on the LLaVA-1.5-7B and -13B models. Experimental results demonstrate that RoVRM consistently outperforms traditional VRMs. Furthermore, our three-phase progressive training and preference data selection approaches can yield consistent performance gains over ranking-based alignment techniques, such as direct preference optimization.

Title: Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation

• Authors: Woo Kyung Kim, Minjong Yoo, Honguk Woo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12110
• Pdf URL: https://arxiv.org/pdf/2408.12110
• Copy Paste: [[2408.12110]] Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation(https://arxiv.org/abs/2408.12110)
• Keywords: diffusion
• Abstract: Data-driven offline reinforcement learning and imitation learning approaches have been gaining popularity in addressing sequential decision-making problems. Yet, these approaches rarely consider learning Pareto-optimal policies from a limited pool of expert datasets. This becomes particularly marked due to practical limitations in obtaining comprehensive datasets for all preferences, where multiple conflicting objectives exist and each expert might hold a unique optimization preference for these objectives. In this paper, we adapt inverse reinforcement learning (IRL) by using reward distance estimates for regularizing the discriminator. This enables progressive generation of a set of policies that accommodate diverse preferences on the multiple objectives, while using only two distinct datasets, each associated with a different expert preference. In doing so, we present a Pareto IRL framework (ParIRL) that establishes a Pareto policy set from these limited datasets. In the framework, the Pareto policy set is then distilled into a single, preference-conditioned diffusion model, thus allowing users to immediately specify which expert's patterns they prefer. Through experiments, we show that ParIRL outperforms other IRL algorithms for various multi-objective control tasks, achieving the dense approximation of the Pareto frontier. We also demonstrate the applicability of ParIRL with autonomous driving in CARLA.

Title: ZipGait: Bridging Skeleton and Silhouette with Diffusion Model for Advancing Gait Recognition

• Authors: Fanxu Min, Qing Cai, Shaoxiang Guo, Yang Yu, Hao Fan, Junyu Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12111
• Pdf URL: https://arxiv.org/pdf/2408.12111
• Copy Paste: [[2408.12111]] ZipGait: Bridging Skeleton and Silhouette with Diffusion Model for Advancing Gait Recognition(https://arxiv.org/abs/2408.12111)
• Keywords: diffusion
• Abstract: Current gait recognition research predominantly focuses on extracting appearance features effectively, but the performance is severely compromised by the vulnerability of silhouettes under unconstrained scenes. Consequently, numerous studies have explored how to harness information from various models, particularly by sufficiently utilizing the intrinsic information of skeleton sequences. While these model-based methods have achieved significant performance, there is still a huge gap compared to appearance-based methods, which implies the potential value of bridging silhouettes and skeletons. In this work, we make the first attempt to reconstruct dense body shapes from discrete skeleton distributions via the diffusion model, demonstrating a new approach that connects cross-modal features rather than focusing solely on intrinsic features to improve model-based methods. To realize this idea, we propose a novel gait diffusion model named DiffGait, which has been designed with four specific adaptations suitable for gait recognition. Furthermore, to effectively utilize the reconstructed silhouettes and skeletons, we introduce Perception Gait Integration (PGI) to integrate different gait features through a two-stage process. Incorporating those modifications leads to an efficient model-based gait recognition framework called ZipGait. Through extensive experiments on four public benchmarks, ZipGait demonstrates superior performance, outperforming the state-of-the-art methods by a large margin under both cross-domain and intra-domain settings, while achieving significant plug-and-play performance improvements.

Title: Understanding Data Reconstruction Leakage in Federated Learning from a Theoretical Perspective

• Authors: Zifan Wang, Binghui Zhang, Meng Pang, Yuan Hong, Binghui Wang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12119
• Pdf URL: https://arxiv.org/pdf/2408.12119
• Copy Paste: [[2408.12119]] Understanding Data Reconstruction Leakage in Federated Learning from a Theoretical Perspective(https://arxiv.org/abs/2408.12119)
• Keywords: privacy, protect, attack, federate, fair
• Abstract: Federated learning (FL) is an emerging collaborative learning paradigm that aims to protect data privacy. Unfortunately, recent works show FL algorithms are vulnerable to the serious data reconstruction attacks. However, existing works lack a theoretical foundation on to what extent the devices' data can be reconstructed and the effectiveness of these attacks cannot be compared fairly due to their unstable performance. To address this deficiency, we propose a theoretical framework to understand data reconstruction attacks to FL. Our framework involves bounding the data reconstruction error and an attack's error bound reflects its inherent attack effectiveness. Under the framework, we can theoretically compare the effectiveness of existing attacks. For instance, our results on multiple datasets validate that the iDLG attack inherently outperforms the DLG attack.

Title: On the Credibility of Backdoor Attacks Against Object Detectors in the Physical World

• Authors: Bao Gia Doan, Dang Quang Nguyen, Callum Lindquist, Paul Montague, Tamas Abraham, Olivier De Vel, Seyit Camtepe, Salil S. Kanhere, Ehsan Abbasnejad, Damith C. Ranasinghe
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.12122
• Pdf URL: https://arxiv.org/pdf/2408.12122
• Copy Paste: [[2408.12122]] On the Credibility of Backdoor Attacks Against Object Detectors in the Physical World(https://arxiv.org/abs/2408.12122)
• Keywords: defense, attack
• Abstract: Object detectors are vulnerable to backdoor attacks. In contrast to classifiers, detectors possess unique characteristics, architecturally and in task execution; often operating in challenging conditions, for instance, detecting traffic signs in autonomous cars. But, our knowledge dominates attacks against classifiers and tests in the "digital domain". To address this critical gap, we conducted an extensive empirical study targeting multiple detector architectures and two challenging detection tasks in real-world settings: traffic signs and vehicles. Using the diverse, methodically collected videos captured from driving cars and flying drones, incorporating physical object trigger deployments in authentic scenes, we investigated the viability of physical object-triggered backdoor attacks in application settings. Our findings revealed 8 key insights. Importantly, the prevalent "digital" data poisoning method for injecting backdoors into models does not lead to effective attacks against detectors in the real world, although proven effective in classification tasks. We construct a new, cost-efficient attack method, dubbed MORPHING, incorporating the unique nature of detection tasks; ours is remarkably successful in injecting physical object-triggered backdoors, even capable of poisoning triggers with clean label annotations or invisible triggers without diminishing the success of physical object triggered backdoors. We discovered that the defenses curated are ill-equipped to safeguard detectors against such attacks. To underscore the severity of the threat and foster further research, we, for the first time, release an extensive video test set of real-world backdoor attacks. Our study not only establishes the credibility and seriousness of this threat but also serves as a clarion call to the research community to advance backdoor defenses in the context of object detection.

Title: Recording Brain Activity While Listening to Music Using Wearable EEG Devices Combined with Bidirectional Long Short-Term Memory Networks

• Authors: Jingyi Wang, Zhiqun Wang, Guiran Liu
• Subjects: cs.LG, cs.HC, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.12124
• Pdf URL: https://arxiv.org/pdf/2408.12124
• Copy Paste: [[2408.12124]] Recording Brain Activity While Listening to Music Using Wearable EEG Devices Combined with Bidirectional Long Short-Term Memory Networks(https://arxiv.org/abs/2408.12124)
• Keywords: robust, extraction
• Abstract: Electroencephalography (EEG) signals are crucial for investigating brain function and cognitive processes. This study aims to address the challenges of efficiently recording and analyzing high-dimensional EEG signals while listening to music to recognize emotional states. We propose a method combining Bidirectional Long Short-Term Memory (Bi-LSTM) networks with attention mechanisms for EEG signal processing. Using wearable EEG devices, we collected brain activity data from participants listening to music. The data was preprocessed, segmented, and Differential Entropy (DE) features were extracted. We then constructed and trained a Bi-LSTM model to enhance key feature extraction and improve emotion recognition accuracy. Experiments were conducted on the SEED and DEAP datasets. The Bi-LSTM-AttGW model achieved 98.28% accuracy on the SEED dataset and 92.46% on the DEAP dataset in multi-class emotion recognition tasks, significantly outperforming traditional models such as SVM and EEG-Net. This study demonstrates the effectiveness of combining Bi-LSTM with attention mechanisms, providing robust technical support for applications in brain-computer interfaces (BCI) and affective computing. Future work will focus on improving device design, incorporating multimodal data, and further enhancing emotion recognition accuracy, aiming to achieve practical applications in real-world scenarios.

Title: Deep Analysis of Time Series Data for Smart Grid Startup Strategies: A Transformer-LSTM-PSO Model Approach

• Authors: Zecheng Zhang
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.12129
• Pdf URL: https://arxiv.org/pdf/2408.12129
• Copy Paste: [[2408.12129]] Deep Analysis of Time Series Data for Smart Grid Startup Strategies: A Transformer-LSTM-PSO Model Approach(https://arxiv.org/abs/2408.12129)
• Keywords: transformer
• Abstract: Grid startup, an integral component of the power system, holds strategic importance for ensuring the reliability and efficiency of the electrical grid. However, current methodologies for in-depth analysis and precise prediction of grid startup scenarios are inadequate. To address these challenges, we propose a novel method based on the Transformer-LSTM-PSO model. This model uniquely combines the Transformer's self-attention mechanism, LSTM's temporal modeling capabilities, and the parameter tuning features of the particle swarm optimization algorithm. It is designed to more effectively capture the complex temporal relationships in grid startup schemes. Our experiments demonstrate significant improvements, with our model achieving lower RMSE and MAE values across multiple datasets compared to existing benchmarks, particularly in the NYISO Electric Market dataset where the RMSE was reduced by approximately 15% and the MAE by 20% compared to conventional models. Our main contribution is the development of a Transformer-LSTM-PSO model that significantly enhances the accuracy and efficiency of smart grid startup predictions. The application of the Transformer-LSTM-PSO model represents a significant advancement in smart grid predictive analytics, concurrently fostering the development of more reliable and intelligent grid management systems.

Title: DRExplainer: Quantifiable Interpretability in Drug Response Prediction with Directed Graph Convolutional Network

• Authors: Haoyuan Shi, Tao Xu, Xiaodi Li, Qian Gao, Junfeng Xia, Zhenyu Yue
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12139
• Pdf URL: https://arxiv.org/pdf/2408.12139
• Copy Paste: [[2408.12139]] DRExplainer: Quantifiable Interpretability in Drug Response Prediction with Directed Graph Convolutional Network(https://arxiv.org/abs/2408.12139)
• Keywords: interpretability
• Abstract: Predicting the response of a cancer cell line to a therapeutic drug is pivotal for personalized medicine. Despite numerous deep learning methods that have been developed for drug response prediction, integrating diverse information about biological entities and predicting the directional response remain major challenges. Here, we propose a novel interpretable predictive model, DRExplainer, which leverages a directed graph convolutional network to enhance the prediction in a directed bipartite network framework. DRExplainer constructs a directed bipartite network integrating multi-omics profiles of cell lines, the chemical structure of drugs and known drug response to achieve directed prediction. Then, DRExplainer identifies the most relevant subgraph to each prediction in this directed bipartite network by learning a mask, facilitating critical medical decision-making. Additionally, we introduce a quantifiable method for model interpretability that leverages a ground truth benchmark dataset curated from biological features. In computational experiments, DRExplainer outperforms state-of-the-art predictive methods and another graph-based explanation method under the same experimental setting. Finally, the case studies further validate the interpretability and the effectiveness of DRExplainer in predictive novel drug response. Our code is available at: this https URL.

Title: TRRG: Towards Truthful Radiology Report Generation With Cross-modal Disease Clue Enhanced Large Language Model

• Authors: Yuhao Wang, Chao Hao, Yawen Cui, Xinqi Su, Weicheng Xie, Tao Tan, Zitong Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12141
• Pdf URL: https://arxiv.org/pdf/2408.12141
• Copy Paste: [[2408.12141]] TRRG: Towards Truthful Radiology Report Generation With Cross-modal Disease Clue Enhanced Large Language Model(https://arxiv.org/abs/2408.12141)
• Keywords: robust, large language model
• Abstract: The vision-language modeling capability of multi-modal large language models has attracted wide attention from the community. However, in medical domain, radiology report generation using vision-language models still faces significant challenges due to the imbalanced data distribution caused by numerous negated descriptions in radiology reports and issues such as rough alignment between radiology reports and radiography. In this paper, we propose a truthful radiology report generation framework, namely TRRG, based on stage-wise training for cross-modal disease clue injection into large language models. In pre-training stage, During the pre-training phase, contrastive learning is employed to enhance the ability of visual encoder to perceive fine-grained disease details. In fine-tuning stage, the clue injection module we proposed significantly enhances the disease-oriented perception capability of the large language model by effectively incorporating the robust zero-shot disease perception. Finally, through the cross-modal clue interaction module, our model effectively achieves the multi-granular interaction of visual embeddings and an arbitrary number of disease clue embeddings. This significantly enhances the report generation capability and clinical effectiveness of multi-modal large language models in the field of radiology reportgeneration. Experimental results demonstrate that our proposed pre-training and fine-tuning framework achieves state-of-the-art performance in radiology report generation on datasets such as IU-Xray and MIMIC-CXR. Further analysis indicates that our proposed method can effectively enhance the model to perceive diseases and improve its clinical effectiveness.

Title: MDD-5k: A New Diagnostic Conversation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agents

• Authors: Congchi Yin, Feng Li, Shu Zhang, Zike Wang, Jun Shao, Piji Li, Jianhua Chen, Xun Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12142
• Pdf URL: https://arxiv.org/pdf/2408.12142
• Copy Paste: [[2408.12142]] MDD-5k: A New Diagnostic Conversation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agents(https://arxiv.org/abs/2408.12142)
• Keywords: privacy, large language model
• Abstract: The clinical diagnosis of most mental disorders primarily relies on the conversations between psychiatrist and patient. The creation of such diagnostic conversation datasets is promising to boost the AI mental healthcare community. However, directly collecting the conversations in real diagnosis scenarios is near impossible due to stringent privacy and ethical considerations. To address this issue, we seek to synthesize diagnostic conversation by exploiting anonymous patient cases that are easier to access. Specifically, we design a neuro-symbolic multi-agent framework for synthesizing the diagnostic conversation of mental disorders with large language models. It takes patient case as input and is capable of generating multiple diverse conversations with one single patient case. The framework basically involves the interaction between a doctor agent and a patient agent, and achieves text generation under symbolic control via a dynamic diagnosis tree from a tool agent. By applying the proposed framework, we develop the largest Chinese mental disorders diagnosis dataset MDD-5k, which is built upon 1000 cleaned real patient cases by cooperating with a pioneering psychiatric hospital, and contains 5000 high-quality long conversations with diagnosis results as labels. To the best of our knowledge, it's also the first labelled Chinese mental disorders diagnosis dataset. Human evaluation demonstrates the proposed MDD-5k dataset successfully simulates human-like diagnostic process of mental disorders. The dataset and code will become publicly accessible in this https URL.

Title: Implicit Sentiment Analysis Based on Chain of Thought Prompting

• Authors: Zhihua Duan, Jialin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12157
• Pdf URL: https://arxiv.org/pdf/2408.12157
• Copy Paste: [[2408.12157]] Implicit Sentiment Analysis Based on Chain of Thought Prompting(https://arxiv.org/abs/2408.12157)
• Keywords: large language model
• Abstract: Implicit Sentiment Analysis (ISA) is a crucial research area in natural language processing. Inspired by the idea of large language model Chain of Thought (CoT), this paper introduces a Sentiment Analysis of Thinking (SAoT) framework. The framework first analyzes the implicit aspects and opinions in the text using common sense and thinking chain capabilities. Then, it reflects on the process of implicit sentiment analysis and finally deduces the polarity of sentiment. The model is evaluated on the SemEval 2014 dataset, consisting of 1120 restaurant reviews and 638 laptop reviews. The experimental results demonstrate that the utilization of the ERNIE-Bot-4+SAoT model yields a notable performance improvement. Specifically, on the restaurant dataset, the F1 score reaches 75.27, accompanied by an ISA score of 66.29. Similarly, on the computer dataset, the F1 score achieves 76.50, while the ISA score amounts to 73.46. Comparatively, the ERNIE-Bot-4+SAoT model surpasses the BERTAsp + SCAPt baseline by an average margin of 47.99%.

Title: Preference-Guided Reflective Sampling for Aligning Language Models

• Authors: Hai Ye, Hwee Tou Ng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12163
• Pdf URL: https://arxiv.org/pdf/2408.12163
• Copy Paste: [[2408.12163]] Preference-Guided Reflective Sampling for Aligning Language Models(https://arxiv.org/abs/2408.12163)
• Keywords: large language model
• Abstract: Large language models (LLMs) are aligned with human preferences by reinforcement learning from human feedback (RLHF). Effective data sampling is crucial for RLHF, as it determines the efficiency of model training, ensuring that models learn from the informative samples. To achieve better data generation, we propose a new sampling method called Preference-Guided Reflective Sampling (PRS). PRS frames the response generation as an optimization process to the explicitly specified user preference described in natural language. It employs a tree-based generation framework to enable an efficient sampling process, which guides the direction of generation through preference and better explores the sampling space with adaptive self-refinement. Notably, PRS can align LLMs to diverse preferences. We study preference-controlled text generation for instruction following and keyword-focused document summarization. Our findings indicate that PRS, across different LLM policies, generates training data with much higher rewards than strong baselines. PRS also excels in post-RL training.

Title: FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation

• Authors: KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Omer Raza
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12168
• Pdf URL: https://arxiv.org/pdf/2408.12168
• Copy Paste: [[2408.12168]] FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation(https://arxiv.org/abs/2408.12168)
• Keywords: large language model
• Abstract: Large language models (LLMs) have become increasingly prevalent in our daily lives, leading to an expectation for LLMs to be trustworthy -- - both accurate and well-calibrated (the prediction confidence should align with its ground truth correctness likelihood). Nowadays, fine-tuning has become the most popular method for adapting a model to practical usage by significantly increasing accuracy on downstream tasks. Despite the great accuracy it achieves, we found fine-tuning is still far away from satisfactory trustworthiness due to "tuning-induced mis-calibration". In this paper, we delve deeply into why and how mis-calibration exists in fine-tuned models, and how distillation can alleviate the issue. Then we further propose a brand new method named Efficient Trustworthy Distillation (FIRST), which utilizes a small portion of teacher's knowledge to obtain a reliable language model in a cost-efficient way. Specifically, we identify the "concentrated knowledge" phenomenon during distillation, which can significantly reduce the computational burden. Then we apply a "trustworthy maximization" process to optimize the utilization of this small portion of concentrated knowledge before transferring it to the student. Experimental results demonstrate the effectiveness of our method, where better accuracy (+2.3%) and less mis-calibration (-10%) are achieved on average across both in-domain and out-of-domain scenarios, indicating better trustworthiness.

Title: Rank and Align: Towards Effective Source-free Graph Domain Adaptation

• Authors: Junyu Luo, Zhiping Xiao, Yifan Wang, Xiao Luo, Jingyang Yuan, Wei Ju, Langechuan Liu, Ming Zhang
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.12185
• Pdf URL: https://arxiv.org/pdf/2408.12185
• Copy Paste: [[2408.12185]] Rank and Align: Towards Effective Source-free Graph Domain Adaptation(https://arxiv.org/abs/2408.12185)
• Keywords: privacy, robust, extraction
• Abstract: Graph neural networks (GNNs) have achieved impressive performance in graph domain adaptation. However, extensive source graphs could be unavailable in real-world scenarios due to privacy and storage concerns. To this end, we investigate an underexplored yet practical problem of source-free graph domain adaptation, which transfers knowledge from source models instead of source graphs to a target domain. To solve this problem, we introduce a novel GNN-based approach called Rank and Align (RNA), which ranks graph similarities with spectral seriation for robust semantics learning, and aligns inharmonic graphs with harmonic graphs which close to the source domain for subgraph extraction. In particular, to overcome label scarcity, we employ the spectral seriation algorithm to infer the robust pairwise rankings, which can guide semantic learning using a similarity learning objective. To depict distribution shifts, we utilize spectral clustering and the silhouette coefficient to detect harmonic graphs, which the source model can easily classify. To reduce potential domain discrepancy, we extract domain-invariant subgraphs from inharmonic graphs by an adversarial edge sampling process, which guides the invariant learning of GNNs. Extensive experiments on several benchmark datasets demonstrate the effectiveness of our proposed RNA.

Title: Reasoning Factual Knowledge in Structured Data with Large Language Models

• Authors: Sirui Huang, Yanggan Gu, Xuming Hu, Zhonghao Li, Qing Li, Guandong Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12188
• Pdf URL: https://arxiv.org/pdf/2408.12188
• Copy Paste: [[2408.12188]] Reasoning Factual Knowledge in Structured Data with Large Language Models(https://arxiv.org/abs/2408.12188)
• Keywords: large language model
• Abstract: Large language models (LLMs) have made remarkable progress in various natural language processing tasks as a benefit of their capability to comprehend and reason with factual knowledge. However, a significant amount of factual knowledge is stored in structured data, which possesses unique characteristics that differ from the unstructured texts used for pretraining. This difference can introduce imperceptible inference parameter deviations, posing challenges for LLMs in effectively utilizing and reasoning with structured data to accurately infer factual knowledge. To this end, we propose a benchmark named StructFact, to evaluate the structural reasoning capabilities of LLMs in inferring factual knowledge. StructFact comprises 8,340 factual questions encompassing various tasks, domains, timelines, and regions. This benchmark allows us to investigate the capability of LLMs across five factual tasks derived from the unique characteristics of structural facts. Extensive experiments on a set of LLMs with different training strategies reveal the limitations of current LLMs in inferring factual knowledge from structured data. We present this benchmark as a compass to navigate the strengths and weaknesses of LLMs in reasoning with structured data for knowledge-sensitive tasks, and to encourage advancements in related real-world applications. Please find our code at this https URL.

Title: Transientangelo: Few-Viewpoint Surface Reconstruction Using Single-Photon Lidar

• Authors: Weihan Luo, Anagh Malik, David B. Lindell
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12191
• Pdf URL: https://arxiv.org/pdf/2408.12191
• Copy Paste: [[2408.12191]] Transientangelo: Few-Viewpoint Surface Reconstruction Using Single-Photon Lidar(https://arxiv.org/abs/2408.12191)
• Keywords: robust
• Abstract: We consider the problem of few-viewpoint 3D surface reconstruction using raw measurements from a lidar system. Lidar captures 3D scene geometry by emitting pulses of light to a target and recording the speed-of-light time delay of the reflected light. However, conventional lidar systems do not output the raw, captured waveforms of backscattered light; instead, they pre-process these data into a 3D point cloud. Since this procedure typically does not accurately model the noise statistics of the system, exploit spatial priors, or incorporate information about downstream tasks, it ultimately discards useful information that is encoded in raw measurements of backscattered light. Here, we propose to leverage raw measurements captured with a single-photon lidar system from multiple viewpoints to optimize a neural surface representation of a scene. The measurements consist of time-resolved photon count histograms, or transients, which capture information about backscattered light at picosecond time scales. Additionally, we develop new regularization strategies that improve robustness to photon noise, enabling accurate surface reconstruction with as few as 10 photons per pixel. Our method outperforms other techniques for few-viewpoint 3D reconstruction based on depth maps, point clouds, or conventional lidar as demonstrated in simulation and with captured data.

Title: Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

• Authors: Kun Luo, Minghao Qin, Zheng Liu, Shitao Xiao, Jun Zhao, Kang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12194
• Pdf URL: https://arxiv.org/pdf/2408.12194
• Copy Paste: [[2408.12194]] Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment(https://arxiv.org/abs/2408.12194)
• Keywords: large language model
• Abstract: Pretrained language models like BERT and T5 serve as crucial backbone encoders for dense retrieval. However, these models often exhibit limited generalization capabilities and face challenges in improving in domain accuracy. Recent research has explored using large language models (LLMs) as retrievers, achieving SOTA performance across various tasks. Despite these advancements, the specific benefits of LLMs over traditional retrievers and the impact of different LLM configurations, such as parameter sizes, pretraining duration, and alignment processes on retrieval tasks remain unclear. In this work, we conduct a comprehensive empirical study on a wide range of retrieval tasks, including in domain accuracy, data efficiency, zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. We evaluate over 15 different backbone LLMs and non LLMs. Our findings reveal that larger models and extensive pretraining consistently enhance in domain accuracy and data efficiency. Additionally, larger models demonstrate significant potential in zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. These results underscore the advantages of LLMs as versatile and effective backbone encoders in dense retrieval, providing valuable insights for future research and development in this field.

Title: Computer-Aided Fall Recognition Using a Three-Stream Spatial-Temporal GCN Model with Adaptive Feature Aggregation

• Authors: Jungpil Shin, Abu Saleh Musa Miah, Rei Egawa1, Koki Hirooka, Md. Al Mehedi Hasan, Yoichi Tomioka, Yong Seok Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12211
• Pdf URL: https://arxiv.org/pdf/2408.12211
• Copy Paste: [[2408.12211]] Computer-Aided Fall Recognition Using a Three-Stream Spatial-Temporal GCN Model with Adaptive Feature Aggregation(https://arxiv.org/abs/2408.12211)
• Keywords: robust
• Abstract: The prevention of falls is paramount in modern healthcare, particularly for the elderly, as falls can lead to severe injuries or even fatalities. Additionally, the growing incidence of falls among the elderly, coupled with the urgent need to prevent suicide attempts resulting from medication overdose, underscores the critical importance of accurate and efficient fall detection methods. In this scenario, a computer-aided fall detection system is inevitable to save elderly people's lives worldwide. Many researchers have been working to develop fall detection systems. However, the existing fall detection systems often struggle with issues such as unsatisfactory performance accuracy, limited robustness, high computational complexity, and sensitivity to environmental factors due to a lack of effective features. In response to these challenges, this paper proposes a novel three-stream spatial-temporal feature-based fall detection system. Our system incorporates joint skeleton-based spatial and temporal Graph Convolutional Network (GCN) features, joint motion-based spatial and temporal GCN features, and residual connections-based features. Each stream employs adaptive graph-based feature aggregation and consecutive separable convolutional neural networks (Sep-TCN), significantly reducing computational complexity and model parameters compared to prior systems. Experimental results across multiple datasets demonstrate the superior effectiveness and efficiency of our proposed system, with accuracies of 99.51\%, 99.15\%, 99.79\% and 99.85 \% achieved on the ImViA, UR-Fall, Fall-UP and FU-Kinect datasets, respectively. The remarkable performance of our system highlights its superiority, efficiency, and generalizability in real-world fall detection scenarios, offering significant advancements in healthcare and societal well-being.

Title: Quantifying Psychological Sophistication of Malicious Emails

• Authors: Theodore Longtchi, Rosana Montañez Rodriguez, Kora Gwartney, Ekzhin Ear, David P. Azari, Christopher P. Kelley, Shouhuai Xu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.12217
• Pdf URL: https://arxiv.org/pdf/2408.12217
• Copy Paste: [[2408.12217]] Quantifying Psychological Sophistication of Malicious Emails(https://arxiv.org/abs/2408.12217)
• Keywords: defense, attack
• Abstract: Malicious emails including Phishing, Spam, and Scam are one significant class of cyber social engineering attacks. Despite numerous defenses to counter them, the problem remains largely open. The ineffectiveness of current defenses can be attributed to our superficial understanding of the psychological properties that make these attacks successful. This problem motivates us to investigate the psychological sophistication, or sophistication for short, of malicious emails. We propose an innovative framework that accommodates two important and complementary aspects of sophistication, dubbed Psychological Techniques, PTechs, and Psychological Tactics, PTacs. We propose metrics and grading rules for human experts to assess the sophistication of malicious emails via the lens of these PTechs and PTacs. To demonstrate the usefulness of the framework, we conduct a case study based on 1,036 malicious emails assessed by four independent graders. Our results show that malicious emails are psychologically sophisticated, while exhibiting both commonalities and different patterns in terms of their PTechs and PTacs. Results also show that previous studies might have focused on dealing with the less proliferated PTechs such as Persuasion and PTacs such as Reward, rather than the most proliferated PTechs such as Attention Grabbing and Impersonation, and PTacs such as Fit and Form and Familiarity that are identified in this study. We also found among others that social events are widely exploited by attackers in contextualizing their malicious emails. These findings could be leveraged to guide the design of effective defenses against malicious emails.

Title: EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for Automated Scoring of CEFR B2 Speaking Assessment Transcripts

• Authors: Nicy Scaria, Silvester John Joseph Kennedy, Thomas Latinovich, Deepak Subramani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12226
• Pdf URL: https://arxiv.org/pdf/2408.12226
• Copy Paste: [[2408.12226]] EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for Automated Scoring of CEFR B2 Speaking Assessment Transcripts(https://arxiv.org/abs/2408.12226)
• Keywords: large language model
• Abstract: Relying on human experts to evaluate CEFR speaking assessments in an e-learning environment creates scalability challenges, as it limits how quickly and widely assessments can be conducted. We aim to automate the evaluation of CEFR B2 English speaking assessments in e-learning environments from conversation transcripts. First, we evaluate the capability of leading open source and commercial Large Language Models (LLMs) to score a candidate's performance across various criteria in the CEFR B2 speaking exam in both global and India-specific contexts. Next, we create a new expert-validated, CEFR-aligned synthetic conversational dataset with transcripts that are rated at different assessment scores. In addition, new instruction-tuned datasets are developed from the English Vocabulary Profile (up to CEFR B2 level) and the CEFR-SP WikiAuto datasets. Finally, using these new datasets, we perform parameter efficient instruction tuning of Mistral Instruct 7B v0.2 to develop a family of models called EvalYaks. Four models in this family are for assessing the four sections of the CEFR B2 speaking exam, one for identifying the CEFR level of vocabulary and generating level-specific vocabulary, and another for detecting the CEFR level of text and generating level-specific text. EvalYaks achieved an average acceptable accuracy of 96%, a degree of variation of 0.35 levels, and performed 3 times better than the next best model. This demonstrates that a 7B parameter LLM instruction tuned with high-quality CEFR-aligned assessment data can effectively evaluate and score CEFR B2 English speaking assessments, offering a promising solution for scalable, automated language proficiency evaluation.

Title: Scalable Autoregressive Image Generation with Mamba

• Authors: Haopeng Li, Jinyue Yang, Kexin Wang, Xuerui Qiu, Yuhong Chou, Xin Li, Guoqi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12245
• Pdf URL: https://arxiv.org/pdf/2408.12245
• Copy Paste: [[2408.12245]] Scalable Autoregressive Image Generation with Mamba(https://arxiv.org/abs/2408.12245)
• Keywords: diffusion, transformer, generative
• Abstract: We introduce AiM, an autoregressive (AR) image generative model based on Mamba architecture. AiM employs Mamba, a novel state-space model characterized by its exceptional performance for long-sequence modeling with linear time complexity, to supplant the commonly utilized Transformers in AR image generation models, aiming to achieve both superior generation quality and enhanced inference speed. Unlike existing methods that adapt Mamba to handle two-dimensional signals via multi-directional scan, AiM directly utilizes the next-token prediction paradigm for autoregressive image generation. This approach circumvents the need for extensive modifications to enable Mamba to learn 2D spatial representations. By implementing straightforward yet strategically targeted modifications for visual generative tasks, we preserve Mamba's core structure, fully exploiting its efficient long-sequence modeling capabilities and scalability. We provide AiM models in various scales, with parameter counts ranging from 148M to 1.3B. On the ImageNet1K 256*256 benchmark, our best AiM model achieves a FID of 2.21, surpassing all existing AR models of comparable parameter counts and demonstrating significant competitiveness against diffusion models, with 2 to 10 times faster inference speed. Code is available at this https URL

Title: OVA-DETR: Open Vocabulary Aerial Object Detection Using Image-Text Alignment and Fusion

• Authors: Guoting Wei, Xia Yuan, Yu Liu, Zhenhao Shang, Kelu Yao, Chao Li, Qingsen Yan, Chunxia Zhao, Haokui Zhang, Rong Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12246
• Pdf URL: https://arxiv.org/pdf/2408.12246
• Copy Paste: [[2408.12246]] OVA-DETR: Open Vocabulary Aerial Object Detection Using Image-Text Alignment and Fusion(https://arxiv.org/abs/2408.12246)
• Keywords: extraction
• Abstract: Aerial object detection has been a hot topic for many years due to its wide application requirements. However, most existing approaches can only handle predefined categories, which limits their applicability for the open scenarios in real-world. In this paper, we extend aerial object detection to open scenarios by exploiting the relationship between image and text, and propose OVA-DETR, a high-efficiency open-vocabulary detector for aerial images. Specifically, based on the idea of image-text alignment, we propose region-text contrastive loss to replace the category regression loss in the traditional detection framework, which breaks the category limitation. Then, we propose Bidirectional Vision-Language Fusion (Bi-VLF), which includes a dual-attention fusion encoder and a multi-level text-guided Fusion Decoder. The dual-attention fusion encoder enhances the feature extraction process in the encoder part. The multi-level text-guided Fusion Decoder is designed to improve the detection ability for small objects, which frequently appear in aerial object detection scenarios. Experimental results on three widely used benchmark datasets show that our proposed method significantly improves the mAP and recall, while enjoying faster inference speed. For instance, in zero shot detection experiments on DIOR, the proposed OVA-DETR outperforms DescReg and YOLO-World by 37.4% and 33.1%, respectively, while achieving 87 FPS inference speed, which is 7.9x faster than DescReg and 3x faster than YOLO-world. The code is available at this https URL.

Title: PRG: Prompt-Based Distillation Without Annotation via Proxy Relational Graph

• Authors: Yijin Xu, Jialun Liu, Hualiang Wei, Wenhui Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12248
• Pdf URL: https://arxiv.org/pdf/2408.12248
• Copy Paste: [[2408.12248]] PRG: Prompt-Based Distillation Without Annotation via Proxy Relational Graph(https://arxiv.org/abs/2408.12248)
• Keywords: extraction
• Abstract: In this paper, we propose a new distillation method for extracting knowledge from Large Foundation Models (LFM) into lightweight models, introducing a novel supervision mode that does not require manually annotated data. While LFMs exhibit exceptional zero-shot classification abilities across datasets, relying solely on LFM-generated embeddings for distillation poses two main challenges: LFM's task-irrelevant knowledge and the high density of features. The transfer of task-irrelevant knowledge could compromise the student model's discriminative capabilities, and the high density of features within target domains obstructs the extraction of discriminative knowledge essential for the task. To address this issue, we introduce the Proxy Relational Graph (PRG) method. We initially extract task-relevant knowledge from LFMs by calculating a weighted average of logits obtained through text prompt embeddings. Then we construct sample-class proxy graphs for LFM and student models, respectively, to model the correlation between samples and class proxies. Then, we achieve the distillation of selective knowledge by aligning the relational graphs produced by both the LFM and the student model. Specifically, the distillation from LFM to the student model is achieved through two types of alignment: 1) aligning the sample nodes produced by the student model with those produced by the LFM, and 2) aligning the edge relationships in the student model's graph with those in the LFM's graph. Our experimental results validate the effectiveness of PRG, demonstrating its ability to leverage the extensive knowledge base of LFMs while skillfully circumventing their inherent limitations in focused learning scenarios. Notably, in our annotation-free framework, PRG achieves an accuracy of 76.23\% (T: 77.9\%) on CIFAR-100 and 72.44\% (T: 75.3\%) on the ImageNet-1K.

Title: LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction

• Authors: Aishik Nagar, Viktor Schlegel, Thanh-Tung Nguyen, Hao Li, Yuping Wu, Kuluhan Binici, Stefan Winkler
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12249
• Pdf URL: https://arxiv.org/pdf/2408.12249
• Copy Paste: [[2408.12249]] LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction(https://arxiv.org/abs/2408.12249)
• Keywords: extraction, large language model
• Abstract: Large Language Models (LLMs) are increasingly adopted for applications in healthcare, reaching the performance of domain experts on tasks such as question answering and document summarisation. Despite their success on these tasks, it is unclear how well LLMs perform on tasks that are traditionally pursued in the biomedical domain, such as structured information extration. To breach this gap, in this paper, we systematically benchmark LLM performance in Medical Classification and Named Entity Recognition (NER) tasks. We aim to disentangle the contribution of different factors to the performance, particularly the impact of LLMs' task knowledge and reasoning capabilities, their (parametric) domain knowledge, and addition of external knowledge. To this end we evaluate various open LLMs -- including BioMistral and Llama-2 models -- on a diverse set of biomedical datasets, using standard prompting, Chain-of-Thought (CoT) and Self-Consistency based reasoning as well as Retrieval-Augmented Generation (RAG) with PubMed and Wikipedia corpora. Counter-intuitively, our results reveal that standard prompting consistently outperforms more complex techniques across both tasks, laying bare the limitations in the current application of CoT, self-consistency and RAG in the biomedical domain. Our findings suggest that advanced prompting methods developed for knowledge- or reasoning-intensive tasks, such as CoT or RAG, are not easily portable to biomedical tasks where precise structured outputs are required. This highlights the need for more effective integration of external knowledge and reasoning mechanisms in LLMs to enhance their performance in real-world biomedical applications.

Title: Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning

• Authors: Ziming Liu, Jingcai Guo, Song Guo, Xiaocheng Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12253
• Pdf URL: https://arxiv.org/pdf/2408.12253
• Copy Paste: [[2408.12253]] Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning(https://arxiv.org/abs/2408.12253)
• Keywords: robust
• Abstract: This paper investigates a challenging problem of zero-shot learning in the multi-label scenario (MLZSL), wherein the model is trained to recognize multiple unseen classes within a sample (e.g., an image) based on seen classes and auxiliary knowledge, e.g., semantic information. Existing methods usually resort to analyzing the relationship of various seen classes residing in a sample from the dimension of spatial or semantic characteristics and transferring the learned model to unseen ones. However, they neglect the integrity of local and global features. Although the use of the attention structure will accurately locate local features, especially objects, it will significantly lose its integrity, and the relationship between classes will also be affected. Rough processing of global features will also directly affect comprehensiveness. This neglect will make the model lose its grasp of the main components of the image. Relying only on the local existence of seen classes during the inference stage introduces unavoidable bias. In this paper, we propose a novel and comprehensive visual-semantic framework for MLZSL, dubbed Epsilon, to fully make use of such properties and enable a more accurate and robust visual-semantic projection. In terms of spatial information, we achieve effective refinement by group aggregating image features into several semantic prompts. It can aggregate semantic information rather than class information, preserving the correlation between semantics. In terms of global semantics, we use global forward propagation to collect as much information as possible to ensure that semantics are not omitted. Experiments on large-scale MLZSL benchmark datasets NUS-Wide and Open-Images-v4 demonstrate that the proposed Epsilon outperforms other state-of-the-art methods with large margins.

Title: A Language-agnostic Model of Child Language Acquisition

• Authors: Louis Mahon, Omri Abend, Uri Berger, Katherine Demuth, Mark Johnson, Mark Steedman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12254
• Pdf URL: https://arxiv.org/pdf/2408.12254
• Copy Paste: [[2408.12254]] A Language-agnostic Model of Child Language Acquisition(https://arxiv.org/abs/2408.12254)
• Keywords: robust
• Abstract: This work reimplements a recent semantic bootstrapping child-language acquisition model, which was originally designed for English, and trains it to learn a new language: Hebrew. The model learns from pairs of utterances and logical forms as meaning representations, and acquires both syntax and word meanings simultaneously. The results show that the model mostly transfers to Hebrew, but that a number of factors, including the richer morphology in Hebrew, makes the learning slower and less robust. This suggests that a clear direction for future work is to enable the model to leverage the similarities between different word forms.

Title: Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates

• Authors: Yusuke Sakai, Adam Nohejl, Jiangnan Hang, Hidetaka Kamigaito, Taro Watanabe
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12263
• Pdf URL: https://arxiv.org/pdf/2408.12263
• Copy Paste: [[2408.12263]] Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates(https://arxiv.org/abs/2408.12263)
• Keywords: fair, large language model
• Abstract: The natural language understanding (NLU) performance of large language models (LLMs) has been evaluated across various tasks and datasets. The existing evaluation methods, however, do not take into account the variance in scores due to differences in prompts, which leads to unfair evaluation and comparison of NLU performance. Moreover, evaluation designed for specific prompts is inappropriate for instruction tuning, which aims to perform well with any prompt. It is therefore necessary to find a way to measure NLU performance in a fair manner, considering score variance between different instruction templates. In this study, we provide English and Japanese cross-lingual datasets for evaluating the NLU performance of LLMs, which include multiple instruction templates for fair evaluation of each task, along with regular expressions to constrain the output format. Furthermore, we propose the Sharpe score as an evaluation metric that takes into account the variance in scores between templates. Comprehensive analysis of English and Japanese LLMs reveals that the high variance among templates has a significant impact on the fair evaluation of LLMs.

Title: Variance reduction of diffusion model's gradients with Taylor approximation-based control variate

• Authors: Paul Jeha, Will Grathwohl, Michael Riis Andersen, Carl Henrik Ek, Jes Frellsen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12270
• Pdf URL: https://arxiv.org/pdf/2408.12270
• Copy Paste: [[2408.12270]] Variance reduction of diffusion model's gradients with Taylor approximation-based control variate(https://arxiv.org/abs/2408.12270)
• Keywords: diffusion
• Abstract: Score-based models, trained with denoising score matching, are remarkably effective in generating high dimensional data. However, the high variance of their training objective hinders optimisation. We attempt to reduce it with a control variate, derived via a $k$-th order Taylor expansion on the training objective and its gradient. We prove an equivalence between the two and demonstrate empirically the effectiveness of our approach on a low dimensional problem setting; and study its effect on larger problems.

Title: Tackling Data Heterogeneity in Federated Learning via Loss Decomposition

• Authors: Shuang Zeng, Pengxin Guo, Shuai Wang, Jianbo Wang, Yuyin Zhou, Liangqiong Qu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12300
• Pdf URL: https://arxiv.org/pdf/2408.12300
• Copy Paste: [[2408.12300]] Tackling Data Heterogeneity in Federated Learning via Loss Decomposition(https://arxiv.org/abs/2408.12300)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) is a rising approach towards collaborative and privacy-preserving machine learning where large-scale medical datasets remain localized to each client. However, the issue of data heterogeneity among clients often compels local models to diverge, leading to suboptimal global models. To mitigate the impact of data heterogeneity on FL performance, we start with analyzing how FL training influence FL performance by decomposing the global loss into three terms: local loss, distribution shift loss and aggregation loss. Remarkably, our loss decomposition reveals that existing local training-based FL methods attempt to reduce the distribution shift loss, while the global aggregation-based FL methods propose better aggregation strategies to reduce the aggregation loss. Nevertheless, a comprehensive joint effort to minimize all three terms is currently limited in the literature, leading to subpar performance when dealing with data heterogeneity challenges. To fill this gap, we propose a novel FL method based on global loss decomposition, called FedLD, to jointly reduce these three loss terms. Our FedLD involves a margin control regularization in local training to reduce the distribution shift loss, and a principal gradient-based server aggregation strategy to reduce the aggregation loss. Notably, under different levels of data heterogeneity, our strategies achieve better and more robust performance on retinal and chest X-ray classification compared to other FL algorithms. Our code is available at \href{this https URL}{this https URL}.

Title: MakeupAttack: Feature Space Black-box Backdoor Attack on Face Recognition via Makeup Transfer

• Authors: Ming Sun, Lihua Jing, Zixuan Zhu, Rui Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12312
• Pdf URL: https://arxiv.org/pdf/2408.12312
• Copy Paste: [[2408.12312]] MakeupAttack: Feature Space Black-box Backdoor Attack on Face Recognition via Makeup Transfer(https://arxiv.org/abs/2408.12312)
• Keywords: defense, attack, robust, steal
• Abstract: Backdoor attacks pose a significant threat to the training process of deep neural networks (DNNs). As a widely-used DNN-based application in real-world scenarios, face recognition systems once implanted into the backdoor, may cause serious consequences. Backdoor research on face recognition is still in its early stages, and the existing backdoor triggers are relatively simple and visible. Furthermore, due to the perceptibility, diversity, and similarity of facial datasets, many state-of-the-art backdoor attacks lose effectiveness on face recognition tasks. In this work, we propose a novel feature space backdoor attack against face recognition via makeup transfer, dubbed MakeupAttack. In contrast to many feature space attacks that demand full access to target models, our method only requires model queries, adhering to black-box attack principles. In our attack, we design an iterative training paradigm to learn the subtle features of the proposed makeup-style trigger. Additionally, MakeupAttack promotes trigger diversity using the adaptive selection method, dispersing the feature distribution of malicious samples to bypass existing defense methods. Extensive experiments were conducted on two widely-used facial datasets targeting multiple models. The results demonstrate that our proposed attack method can bypass existing state-of-the-art defenses while maintaining effectiveness, robustness, naturalness, and stealthiness, without compromising model performance.

Title: MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model

• Authors: Chaoya Jiang, Jia Hongrui, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang
• Subjects: cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.12321
• Pdf URL: https://arxiv.org/pdf/2408.12321
• Copy Paste: [[2408.12321]] MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model(https://arxiv.org/abs/2408.12321)
• Keywords: large language model
• Abstract: This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. Current MLLMs primarily focus on single-image visual understanding, limiting their ability to interpret and integrate information across multiple images. MaVEn addresses this limitation by combining discrete visual symbol sequences, which abstract coarse-grained semantic concepts, with traditional continuous representation sequences that model fine-grained features. This dual approach bridges the semantic gap between visual and textual data, thereby improving the model's ability to process and interpret information from multiple images effectively. Additionally, we design a dynamic reduction mechanism by for long-sequence continuous features to enhance multi-image processing efficiency. Experimental results demonstrate that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.

Title: Multimodal Foundational Models for Unsupervised 3D General Obstacle Detection

• Authors: Tamás Matuszka, Péter Hajas, Dávid Szeghy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12322
• Pdf URL: https://arxiv.org/pdf/2408.12322
• Copy Paste: [[2408.12322]] Multimodal Foundational Models for Unsupervised 3D General Obstacle Detection(https://arxiv.org/abs/2408.12322)
• Keywords: segmentation
• Abstract: Current autonomous driving perception models primarily rely on supervised learning with predefined categories. However, these models struggle to detect general obstacles not included in the fixed category set due to their variability and numerous edge cases. To address this issue, we propose a combination of multimodal foundational model-based obstacle segmentation with traditional unsupervised computational geometry-based outlier detection. Our approach operates offline, allowing us to leverage non-causality, and utilizes training-free methods. This enables the detection of general obstacles in 3D without the need for expensive retraining. To overcome the limitations of publicly available obstacle detection datasets, we collected and annotated our dataset, which includes various obstacles even in distant regions.

Title: Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparators

• Authors: Dingkang Yang, Dongling Xiao, Jinjie Wei, Mingcheng Li, Zhaoyu Chen, Ke Li, Lihua Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12325
• Pdf URL: https://arxiv.org/pdf/2408.12325
• Copy Paste: [[2408.12325]] Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparators(https://arxiv.org/abs/2408.12325)
• Keywords: robust, large language model
• Abstract: Despite their remarkable capabilities, Large Language Models (LLMs) are prone to generate responses that contradict verifiable facts, i.e., unfaithful hallucination content. Existing efforts generally focus on optimizing model parameters or editing semantic representations, which compromise the internal factual knowledge of target LLMs. In addition, hallucinations typically exhibit multifaceted patterns in downstream tasks, limiting the model's holistic performance across tasks. In this paper, we propose a Comparator-driven Decoding-Time (CDT) framework to alleviate the response hallucination. Firstly, we construct hallucinatory and truthful comparators with multi-task fine-tuning samples. In this case, we present an instruction prototype-guided mixture of experts strategy to enhance the ability of the corresponding comparators to capture different hallucination or truthfulness patterns in distinct task instructions. CDT constrains next-token predictions to factuality-robust distributions by contrasting the logit differences between the target LLMs and these comparators. Systematic experiments on multiple downstream tasks show that our framework can significantly improve the model performance and response factuality.

Title: Interactive DualChecker for Mitigating Hallucinations in Distilling Large Language Models

• Authors: Meiyun Wang, Masahiro Suzuki, Hiroki Sakaji, Kiyoshi Izumi
• Subjects: cs.CL, cs.AI, cs.CE, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.12326
• Pdf URL: https://arxiv.org/pdf/2408.12326
• Copy Paste: [[2408.12326]] Interactive DualChecker for Mitigating Hallucinations in Distilling Large Language Models(https://arxiv.org/abs/2408.12326)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities across various machine learning (ML) tasks. Given the high costs of creating annotated datasets for supervised learning, LLMs offer a valuable alternative by enabling effective few-shot in-context learning. However, these models can produce hallucinations, particularly in domains with incomplete knowledge. Additionally, current methods for knowledge distillation using LLMs often struggle to enhance the effectiveness of both teacher and student models. To address these challenges, we introduce DualChecker, an innovative framework designed to mitigate hallucinations and improve the performance of both teacher and student models during knowledge distillation. DualChecker employs ContextAligner to ensure that the context provided by teacher models aligns with human labeling standards. It also features a dynamic checker system that enhances model interaction: one component re-prompts teacher models with more detailed content when they show low confidence, and another identifies borderline cases from student models to refine the teaching templates. This interactive process promotes continuous improvement and effective knowledge transfer between the models. We evaluate DualChecker using a green innovation textual dataset that includes binary, multiclass, and token classification tasks. The experimental results show that DualChecker significantly outperforms existing state-of-the-art methods, achieving up to a 17% improvement in F1 score for teacher models and 10% for student models. Notably, student models fine-tuned with LLM predictions perform comparably to those fine-tuned with actual data, even in a challenging domain. We make all datasets, models, and code from this research publicly available.

Title: Enhanced Expressivity in Graph Neural Networks with Lanczos-Based Linear Constraints

• Authors: Niloofar Azizi, Nils Kriege, Horst Bischof
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12334
• Pdf URL: https://arxiv.org/pdf/2408.12334
• Copy Paste: [[2408.12334]] Enhanced Expressivity in Graph Neural Networks with Lanczos-Based Linear Constraints(https://arxiv.org/abs/2408.12334)
• Keywords: extraction
• Abstract: Graph Neural Networks (GNNs) excel in handling graph-structured data but often underperform in link prediction tasks compared to classical methods, mainly due to the limitations of the commonly used Message Passing GNNs (MPNNs). Notably, their ability to distinguish non-isomorphic graphs is limited by the 1-dimensional Weisfeiler-Lehman test. Our study presents a novel method to enhance the expressivity of GNNs by embedding induced subgraphs into the graph Laplacian matrix's eigenbasis. We introduce a Learnable Lanczos algorithm with Linear Constraints (LLwLC), proposing two novel subgraph extraction strategies: encoding vertex-deleted subgraphs and applying Neumann eigenvalue constraints. For the former, we conjecture that LLwLC establishes a universal approximator, offering efficient time complexity. The latter focuses on link representations enabling differentiation between $k$-regular graphs and node automorphism, a vital aspect for link prediction tasks. Our approach results in an extremely lightweight architecture, reducing the need for extensive training datasets. Empirically, our method improves performance in challenging link prediction tasks across benchmark datasets, establishing its practical utility and supporting our theoretical findings. Notably, LLwLC achieves 20x and 10x speedup by only requiring 5% and 10% data from the PubMed and OGBL-Vessel datasets while comparing to the state-of-the-art.

Title: Fine-tuning Smaller Language Models for Question Answering over Financial Documents

• Authors: Karmvir Singh Phogat, Sai Akhil Puranam, Sridhar Dasaratha, Chetan Harsha, Shashishekar Ramakrishna
• Subjects: cs.CL, cs.AI, cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.12337
• Pdf URL: https://arxiv.org/pdf/2408.12337
• Copy Paste: [[2408.12337]] Fine-tuning Smaller Language Models for Question Answering over Financial Documents(https://arxiv.org/abs/2408.12337)
• Keywords: extraction
• Abstract: Recent research has shown that smaller language models can acquire substantial reasoning abilities when fine-tuned with reasoning exemplars crafted by a significantly larger teacher model. We explore this paradigm for the financial domain, focusing on the challenge of answering questions that require multi-hop numerical reasoning over financial texts. We assess the performance of several smaller models that have been fine-tuned to generate programs that encode the required financial reasoning and calculations. Our findings demonstrate that these fine-tuned smaller models approach the performance of the teacher model. To provide a granular analysis of model performance, we propose an approach to investigate the specific student model capabilities that are enhanced by fine-tuning. Our empirical analysis indicates that fine-tuning refines the student models ability to express and apply the required financial concepts along with adapting the entity extraction for the specific data format. In addition, we hypothesize and demonstrate that comparable financial reasoning capability can be induced using relatively smaller datasets.

Title: VTON-HandFit: Virtual Try-on for Arbitrary Hand Pose Guided by Hand Priors Embedding

• Authors: Yujie Liang, Xiaobin Hu, Boyuan Jiang, Donghao Luo, Kai WU, Wenhui Han, Taisong Jin, Chengjie Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12340
• Pdf URL: https://arxiv.org/pdf/2408.12340
• Copy Paste: [[2408.12340]] VTON-HandFit: Virtual Try-on for Arbitrary Hand Pose Guided by Hand Priors Embedding(https://arxiv.org/abs/2408.12340)
• Keywords: diffusion
• Abstract: Although diffusion-based image virtual try-on has made considerable progress, emerging approaches still struggle to effectively address the issue of hand occlusion (i.e., clothing regions occluded by the hand part), leading to a notable degradation of the try-on performance. To tackle this issue widely existing in real-world scenarios, we propose VTON-HandFit, leveraging the power of hand priors to reconstruct the appearance and structure for hand occlusion cases. Firstly, we tailor a Handpose Aggregation Net using the ControlNet-based structure explicitly and adaptively encoding the global hand and pose priors. Besides, to fully exploit the hand-related structure and appearance information, we propose Hand-feature Disentanglement Embedding module to disentangle the hand priors into the hand structure-parametric and visual-appearance features, and customize a masked cross attention for further decoupled feature embedding. Lastly, we customize a hand-canny constraint loss to better learn the structure edge knowledge from the hand template of model image. VTON-HandFit outperforms the baselines in qualitative and quantitative evaluations on the public dataset and our self-collected hand-occlusion Handfit-3K dataset particularly for the arbitrary hand pose occlusion cases in real-world scenarios. Code and dataset will be made publicly available.

Title: GarmentAligner: Text-to-Garment Generation via Retrieval-augmented Multi-level Corrections

• Authors: Shiyue Zhang, Zheng Chong, Xujie Zhang, Hanhui Li, Yuhao Cheng, Yiqiang Yan, Xiaodan Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12352
• Pdf URL: https://arxiv.org/pdf/2408.12352
• Copy Paste: [[2408.12352]] GarmentAligner: Text-to-Garment Generation via Retrieval-augmented Multi-level Corrections(https://arxiv.org/abs/2408.12352)
• Keywords: extraction, diffusion
• Abstract: General text-to-image models bring revolutionary innovation to the fields of arts, design, and media. However, when applied to garment generation, even the state-of-the-art text-to-image models suffer from fine-grained semantic misalignment, particularly concerning the quantity, position, and interrelations of garment components. Addressing this, we propose GarmentAligner, a text-to-garment diffusion model trained with retrieval-augmented multi-level corrections. To achieve semantic alignment at the component level, we introduce an automatic component extraction pipeline to obtain spatial and quantitative information of garment components from corresponding images and captions. Subsequently, to exploit component relationships within the garment images, we construct retrieval subsets for each garment by retrieval augmentation based on component-level similarity ranking and conduct contrastive learning to enhance the model perception of components from positive and negative samples. To further enhance the alignment of components across semantic, spatial, and quantitative granularities, we propose the utilization of multi-level correction losses that leverage detailed component information. The experimental findings demonstrate that GarmentAligner achieves superior fidelity and fine-grained semantic alignment when compared to existing competitors.

Title: SoK: An Introspective Analysis of RPKI Security

• Authors: Donika Mirdita, Haya Schulmann, Michael Waidner
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.12359
• Pdf URL: https://arxiv.org/pdf/2408.12359
• Copy Paste: [[2408.12359]] SoK: An Introspective Analysis of RPKI Security(https://arxiv.org/abs/2408.12359)
• Keywords: security, protect, robust
• Abstract: The Resource Public Key Infrastructure (RPKI) is the main mechanism to protect inter-domain routing with BGP from prefix hijacks. It has already been widely deployed by large providers and the adoption rate is getting to a critical point. Almost half of all the global prefixes are now covered by RPKI and measurements show that 27% of networks are already using RPKI to validate BGP announcements. Over the past 10 years, there has been much research effort in RPKI, analyzing different facets of the protocol, such as software vulnerabilities, robustness of the infrastructure or the proliferation of RPKI validation. In this work we compile the first systemic overview of the vulnerabilities and misconfigurations in RPKI and quantify the security landscape of the global RPKI deployments based on our measurements and analysis. Our study discovers that 56% of the global RPKI validators suffer from at least one documented vulnerability. We also do a systematization of knowledge for existing RPKI security research and complement the existing knowledge with novel measurements in which we discover new trends in availability of RPKI repositories, and their communication patterns with the RPKI validators. We weave together the results of existing research and our study, to provide a comprehensive tableau of vulnerabilities, their sources, and to derive future research paths necessary to prepare RPKI for full global deployment.

Title: SAM-SP: Self-Prompting Makes SAM Great Again

• Authors: Chunpeng Zhou, Kangjie Ning, Qianqian Shen, Sheng Zhou, Zhi Yu, Haishuai Wang
• Subjects: cs.CV, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2408.12364
• Pdf URL: https://arxiv.org/pdf/2408.12364
• Copy Paste: [[2408.12364]] SAM-SP: Self-Prompting Makes SAM Great Again(https://arxiv.org/abs/2408.12364)
• Keywords: segmentation
• Abstract: The recently introduced Segment Anything Model (SAM), a Visual Foundation Model (VFM), has demonstrated impressive capabilities in zero-shot segmentation tasks across diverse natural image datasets. Despite its success, SAM encounters noticeably performance degradation when applied to specific domains, such as medical images. Current efforts to address this issue have involved fine-tuning strategies, intended to bolster the generalizability of the vanilla SAM. However, these approaches still predominantly necessitate the utilization of domain specific expert-level prompts during the evaluation phase, which severely constrains the model's practicality. To overcome this limitation, we introduce a novel self-prompting based fine-tuning approach, called SAM-SP, tailored for extending the vanilla SAM model. Specifically, SAM-SP leverages the output from the previous iteration of the model itself as prompts to guide subsequent iteration of the model. This self-prompting module endeavors to learn how to generate useful prompts autonomously and alleviates the dependence on expert prompts during the evaluation phase, significantly broadening SAM's applicability. Additionally, we integrate a self-distillation module to enhance the self-prompting process further. Extensive experiments across various domain specific datasets validate the effectiveness of the proposed SAM-SP. Our SAM-SP not only alleviates the reliance on expert prompts but also exhibits superior segmentation performance comparing to the state-of-the-art task-specific segmentation approaches, the vanilla SAM, and SAM-based approaches.

Title: Robust Principal Component Analysis via Discriminant Sample Weight Learning

• Authors: Yingzhuo Deng, Ke Hu, Bo Li, Yao Zhang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12366
• Pdf URL: https://arxiv.org/pdf/2408.12366
• Copy Paste: [[2408.12366]] Robust Principal Component Analysis via Discriminant Sample Weight Learning(https://arxiv.org/abs/2408.12366)
• Keywords: robust, extraction
• Abstract: Principal component analysis (PCA) is a classical feature extraction method, but it may be adversely affected by outliers, resulting in inaccurate learning of the projection matrix. This paper proposes a robust method to estimate both the data mean and the PCA projection matrix by learning discriminant sample weights from data containing outliers. Each sample in the dataset is assigned a weight, and the proposed algorithm iteratively learns the weights, the mean, and the projection matrix, respectively. Specifically, when the mean and the projection matrix are available, via fine-grained analysis of outliers, a weight for each sample is learned hierarchically so that outliers have small weights while normal samples have large weights. With the learned weights available, a weighted optimization problem is solved to estimate both the data mean and the projection matrix. Because the learned weights discriminate outliers from normal samples, the adverse influence of outliers is mitigated due to the corresponding small weights. Experiments on toy data, UCI dataset, and face dataset demonstrate the effectiveness of the proposed method in estimating the mean and the projection matrix from the data containing outliers.

Title: UMERegRobust -- Universal Manifold Embedding Compatible Features for Robust Point Cloud Registration

• Authors: Yuval Haitman, Amit Efraim, Joseph M. Francos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12380
• Pdf URL: https://arxiv.org/pdf/2408.12380
• Copy Paste: [[2408.12380]] UMERegRobust -- Universal Manifold Embedding Compatible Features for Robust Point Cloud Registration(https://arxiv.org/abs/2408.12380)
• Keywords: robust, extraction
• Abstract: In this paper, we adopt the Universal Manifold Embedding (UME) framework for the estimation of rigid transformations and extend it, so that it can accommodate scenarios involving partial overlap and differently sampled point clouds. UME is a methodology designed for mapping observations of the same object, related by rigid transformations, into a single low-dimensional linear subspace. This process yields a transformation-invariant representation of the observations, with its matrix form representation being covariant (i.e. equivariant) with the transformation. We extend the UME framework by introducing a UME-compatible feature extraction method augmented with a unique UME contrastive loss and a sampling equalizer. These components are integrated into a comprehensive and robust registration pipeline, named UMERegRobust. We propose the RotKITTI registration benchmark, specifically tailored to evaluate registration methods for scenarios involving large rotations. UMERegRobust achieves better than state-of-the-art performance on the KITTI benchmark, especially when strict precision of (1°, 10cm) is considered (with an average gain of +9%), and notably outperform SOTA methods on the RotKITTI benchmark (with +45% gain compared the most recent SOTA method). Our code is available at this https URL.

Title: Makeup-Guided Facial Privacy Protection via Untrained Neural Network Priors

• Authors: Fahad Shamshad, Muzammal Naseer, Karthik Nandakumar
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12387
• Pdf URL: https://arxiv.org/pdf/2408.12387
• Copy Paste: [[2408.12387]] Makeup-Guided Facial Privacy Protection via Untrained Neural Network Priors(https://arxiv.org/abs/2408.12387)
• Keywords: privacy, protect, attack
• Abstract: Deep learning-based face recognition (FR) systems pose significant privacy risks by tracking users without their consent. While adversarial attacks can protect privacy, they often produce visible artifacts compromising user experience. To mitigate this issue, recent facial privacy protection approaches advocate embedding adversarial noise into the natural looking makeup styles. However, these methods require training on large-scale makeup datasets that are not always readily available. In addition, these approaches also suffer from dataset bias. For instance, training on makeup data that predominantly contains female faces could compromise protection efficacy for male faces. To handle these issues, we propose a test-time optimization approach that solely optimizes an untrained neural network to transfer makeup style from a reference to a source image in an adversarial manner. We introduce two key modules: a correspondence module that aligns regions between reference and source images in latent space, and a decoder with conditional makeup layers. The untrained decoder, optimized via carefully designed structural and makeup consistency losses, generates a protected image that resembles the source but incorporates adversarial makeup to deceive FR models. As our approach does not rely on training with makeup face datasets, it avoids potential male/female dataset biases while providing effective protection. We further extend the proposed approach to videos by leveraging on temporal correlations. Experiments on benchmark datasets demonstrate superior performance in face verification and identification tasks and effectiveness against commercial FR systems. Our code and models will be available at this https URL

Title: Fredholm Integral Equations Neural Operator (FIE-NO) for Data-Driven Boundary Value Problems

• Authors: Haoyang Jiang, Yongzhi Qu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12389
• Pdf URL: https://arxiv.org/pdf/2408.12389
• Copy Paste: [[2408.12389]] Fredholm Integral Equations Neural Operator (FIE-NO) for Data-Driven Boundary Value Problems(https://arxiv.org/abs/2408.12389)
• Keywords: robust
• Abstract: In this paper, we present a novel Fredholm Integral Equation Neural Operator (FIE-NO) method, an integration of Random Fourier Features and Fredholm Integral Equations (FIE) into the deep learning framework, tailored for solving data-driven Boundary Value Problems (BVPs) with irregular boundaries. Unlike traditional computational approaches that struggle with the computational intensity and complexity of such problems, our method offers a robust, efficient, and accurate solution mechanism, using a physics inspired design of the learning structure. We demonstrate that the proposed physics-guided operator learning method (FIE-NO) achieves superior performance in addressing BVPs. Notably, our approach can generalize across multiple scenarios, including those with unknown equation forms and intricate boundary shapes, after being trained only on one boundary condition. Experimental validation demonstrates that the FIE-NO method performs well in simulated examples, including Darcy flow equation and typical partial differential equations such as the Laplace and Helmholtz equations. The proposed method exhibits robust performance across different boundary conditions. Experimental results indicate that FIE-NO achieves higher accuracy and stability compared to other methods when addressing complex boundary value problems with varying numbers of interior points.

Title: Multi-Style Facial Sketch Synthesis through Masked Generative Modeling

• Authors: Bowen Sun, Guo Lu, Shibao Zheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12400
• Pdf URL: https://arxiv.org/pdf/2408.12400
• Copy Paste: [[2408.12400]] Multi-Style Facial Sketch Synthesis through Masked Generative Modeling(https://arxiv.org/abs/2408.12400)
• Keywords: extraction, transformer, generative
• Abstract: The facial sketch synthesis (FSS) model, capable of generating sketch portraits from given facial photographs, holds profound implications across multiple domains, encompassing cross-modal face recognition, entertainment, art, media, among others. However, the production of high-quality sketches remains a formidable task, primarily due to the challenges and flaws associated with three key factors: (1) the scarcity of artist-drawn data, (2) the constraints imposed by limited style types, and (3) the deficiencies of processing input information in existing models. To address these difficulties, we propose a lightweight end-to-end synthesis model that efficiently converts images to corresponding multi-stylized sketches, obviating the necessity for any supplementary inputs (\eg, 3D geometry). In this study, we overcome the issue of data insufficiency by incorporating semi-supervised learning into the training process. Additionally, we employ a feature extraction module and style embeddings to proficiently steer the generative transformer during the iterative prediction of masked image tokens, thus achieving a continuous stylized output that retains facial features accurately in sketches. The extensive experiments demonstrate that our method consistently outperforms previous algorithms across multiple benchmarks, exhibiting a discernible disparity.

Title: Generalized SAM: Efficient Fine-Tuning of SAM for Variable Input Image Sizes

• Authors: Sota Kato, Hinako Mitsuoka, Kazuhiro Hotta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12406
• Pdf URL: https://arxiv.org/pdf/2408.12406
• Copy Paste: [[2408.12406]] Generalized SAM: Efficient Fine-Tuning of SAM for Variable Input Image Sizes(https://arxiv.org/abs/2408.12406)
• Keywords: segmentation
• Abstract: There has been a lot of recent research on improving the efficiency of fine-tuning foundation models. In this paper, we propose a novel efficient fine-tuning method that allows the input image size of Segment Anything Model (SAM) to be variable. SAM is a powerful foundational model for image segmentation trained on huge datasets, but it requires fine-tuning to recognize arbitrary classes. The input image size of SAM is fixed at 1024 x 1024, resulting in substantial computational demands during training. Furthermore, the fixed input image size may result in the loss of image information, e.g. due to fixed aspect ratios. To address this problem, we propose Generalized SAM (GSAM). Different from the previous methods, GSAM is the first to apply random cropping during training with SAM, thereby significantly reducing the computational cost of training. Experiments on datasets of various types and various pixel counts have shown that GSAM can train more efficiently than SAM and other fine-tuning methods for SAM, achieving comparable or higher accuracy.

Title: An Evaluation of Deep Learning Models for Stock Market Trend Prediction

• Authors: Gonzalo Lopez Gil, Paul Duhamel-Sebline, Andrew McCarren
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12408
• Pdf URL: https://arxiv.org/pdf/2408.12408
• Copy Paste: [[2408.12408]] An Evaluation of Deep Learning Models for Stock Market Trend Prediction(https://arxiv.org/abs/2408.12408)
• Keywords: transformer
• Abstract: The stock market is a fundamental component of financial systems, reflecting economic health, providing investment opportunities, and influencing global dynamics. Accurate stock market predictions can lead to significant gains and promote better investment decisions. However, predicting stock market trends is challenging due to their non-linear and stochastic nature. This study investigates the efficacy of advanced deep learning models for short-term trend forecasting using daily and hourly closing prices from the S&P 500 index and the Brazilian ETF EWZ. The models explored include Temporal Convolutional Networks (TCN), Neural Basis Expansion Analysis for Time Series Forecasting (N-BEATS), Temporal Fusion Transformers (TFT), Neural Hierarchical Interpolation for Time Series Forecasting (N-HiTS), and Time-series Dense Encoder (TiDE). Furthermore, we introduce the Extended Long Short-Term Memory for Time Series (xLSTM-TS) model, an xLSTM adaptation optimised for time series prediction. Wavelet denoising techniques were applied to smooth the signal and reduce minor fluctuations, providing cleaner data as input for all approaches. Denoising significantly improved performance in predicting stock price direction. Among the models tested, xLSTM-TS consistently outperformed others. For example, it achieved a test accuracy of 72.82% and an F1 score of 73.16% on the EWZ daily dataset. By leveraging advanced deep learning models and effective data preprocessing techniques, this research provides valuable insights into the application of machine learning for market movement forecasting, highlighting both the potential and the challenges involved.

Title: CODE: Confident Ordinary Differential Editing

• Authors: Bastien van Delft, Tommaso Martorella, Alexandre Alahi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12418
• Pdf URL: https://arxiv.org/pdf/2408.12418
• Copy Paste: [[2408.12418]] CODE: Confident Ordinary Differential Editing(https://arxiv.org/abs/2408.12418)
• Keywords: diffusion, generative
• Abstract: Conditioning image generation facilitates seamless editing and the creation of photorealistic images. However, conditioning on noisy or Out-of-Distribution (OoD) images poses significant challenges, particularly in balancing fidelity to the input and realism of the output. We introduce Confident Ordinary Differential Editing (CODE), a novel approach for image synthesis that effectively handles OoD guidance images. Utilizing a diffusion model as a generative prior, CODE enhances images through score-based updates along the probability-flow Ordinary Differential Equation (ODE) trajectory. This method requires no task-specific training, no handcrafted modules, and no assumptions regarding the corruptions affecting the conditioning image. Our method is compatible with any diffusion model. Positioned at the intersection of conditional image generation and blind image restoration, CODE operates in a fully blind manner, relying solely on a pre-trained generative model. Our method introduces an alternative approach to blind restoration: instead of targeting a specific ground truth image based on assumptions about the underlying corruption, CODE aims to increase the likelihood of the input image while maintaining fidelity. This results in the most probable in-distribution image around the input. Our contributions are twofold. First, CODE introduces a novel editing method based on ODE, providing enhanced control, realism, and fidelity compared to its SDE-based counterpart. Second, we introduce a confidence interval-based clipping method, which improves CODE's effectiveness by allowing it to disregard certain pixels or information, thus enhancing the restoration process in a blind manner. Experimental results demonstrate CODE's effectiveness over existing methods, particularly in scenarios involving severe degradation or OoD inputs.

Title: 4D Diffusion for Dynamic Protein Structure Prediction with Reference Guided Motion Alignment

• Authors: Kaihui Cheng, Ce Liu, Qingkun Su, Jun Wang, Liwei Zhang, Yining Tang, Yao Yao, Siyu Zhu, Yuan Qi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12419
• Pdf URL: https://arxiv.org/pdf/2408.12419
• Copy Paste: [[2408.12419]] 4D Diffusion for Dynamic Protein Structure Prediction with Reference Guided Motion Alignment(https://arxiv.org/abs/2408.12419)
• Keywords: diffusion
• Abstract: Protein structure prediction is pivotal for understanding the structure-function relationship of proteins, advancing biological research, and facilitating pharmaceutical development and experimental design. While deep learning methods and the expanded availability of experimental 3D protein structures have accelerated structure prediction, the dynamic nature of protein structures has received limited attention. This study introduces an innovative 4D diffusion model incorporating molecular dynamics (MD) simulation data to learn dynamic protein structures. Our approach is distinguished by the following components: (1) a unified diffusion model capable of generating dynamic protein structures, including both the backbone and side chains, utilizing atomic grouping and side-chain dihedral angle predictions; (2) a reference network that enhances structural consistency by integrating the latent embeddings of the initial 3D protein structures; and (3) a motion alignment module aimed at improving temporal structural coherence across multiple time steps. To our knowledge, this is the first diffusion-based model aimed at predicting protein trajectories across multiple time steps simultaneously. Validation on benchmark datasets demonstrates that our model exhibits high accuracy in predicting dynamic 3D structures of proteins containing up to 256 amino acids over 32 time steps, effectively capturing both local flexibility in stable states and significant conformational changes.

Title: Enhanced Infield Agriculture with Interpretable Machine Learning Approaches for Crop Classification

• Authors: Sudi Murindanyi, Joyce Nakatumba-Nabende, Rahman Sanya, Rose Nakibuule, Andrew Katumba
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12426
• Pdf URL: https://arxiv.org/pdf/2408.12426
• Copy Paste: [[2408.12426]] Enhanced Infield Agriculture with Interpretable Machine Learning Approaches for Crop Classification(https://arxiv.org/abs/2408.12426)
• Keywords: extraction, explainability, transformer
• Abstract: The increasing popularity of Artificial Intelligence in recent years has led to a surge in interest in image classification, especially in the agricultural sector. With the help of Computer Vision, Machine Learning, and Deep Learning, the sector has undergone a significant transformation, leading to the development of new techniques for crop classification in the field. Despite the extensive research on various image classification techniques, most have limitations such as low accuracy, limited use of data, and a lack of reporting model size and prediction. The most significant limitation of all is the need for model explainability. This research evaluates four different approaches for crop classification, namely traditional ML with handcrafted feature extraction methods like SIFT, ORB, and Color Histogram; Custom Designed CNN and established DL architecture like AlexNet; transfer learning on five models pre-trained using ImageNet such as EfficientNetV2, ResNet152V2, Xception, Inception-ResNetV2, MobileNetV3; and cutting-edge foundation models like YOLOv8 and DINOv2, a self-supervised Vision Transformer Model. All models performed well, but Xception outperformed all of them in terms of generalization, achieving 98% accuracy on the test data, with a model size of 80.03 MB and a prediction time of 0.0633 seconds. A key aspect of this research was the application of Explainable AI to provide the explainability of all the models. This journal presents the explainability of Xception model with LIME, SHAP, and GradCAM, ensuring transparency and trustworthiness in the models' predictions. This study highlights the importance of selecting the right model according to task-specific needs. It also underscores the important role of explainability in deploying AI in agriculture, providing insightful information to help enhance AI-driven crop management strategies.

Title: FlexEdit: Marrying Free-Shape Masks to VLLM for Flexible Image Editing

• Authors: Jue Wang, Yuxiang Lin, Tianshuo Yuan, Zhi-Qi Cheng, Xiaolong Wang, Jiao GH, Wei Chen, Xiaojiang Peng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12429
• Pdf URL: https://arxiv.org/pdf/2408.12429
• Copy Paste: [[2408.12429]] FlexEdit: Marrying Free-Shape Masks to VLLM for Flexible Image Editing(https://arxiv.org/abs/2408.12429)
• Keywords: diffusion, large language model
• Abstract: Combining Vision Large Language Models (VLLMs) with diffusion models offers a powerful method for executing image editing tasks based on human language instructions. However, language instructions alone often fall short in accurately conveying user requirements, particularly when users want to add, replace elements in specific areas of an image. Luckily, masks can effectively indicate the exact locations or elements to be edited, while they require users to precisely draw the shapes at the desired locations, which is highly user-unfriendly. To address this, we propose FlexEdit, an end-to-end image editing method that leverages both free-shape masks and language instructions for Flexible Editing. Our approach employs a VLLM in comprehending the image content, mask, and user instructions. Additionally, we introduce the Mask Enhance Adapter (MEA) that fuses the embeddings of the VLLM with the image data, ensuring a seamless integration of mask information and model output embeddings. Furthermore, we construct FSMI-Edit, a benchmark specifically tailored for free-shape mask, including 8 types of free-shape mask. Extensive experiments show that our method achieves state-of-the-art (SOTA) performance in LLM-based image editing, and our simple prompting technique stands out in its effectiveness. The code and data can be found at this https URL.

Title: Verifiable Homomorphic Linear Combinations in Multi-Instance Time-Lock Puzzles

• Authors: Aydin Abadi
• Subjects: cs.CR, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12444
• Pdf URL: https://arxiv.org/pdf/2408.12444
• Copy Paste: [[2408.12444]] Verifiable Homomorphic Linear Combinations in Multi-Instance Time-Lock Puzzles(https://arxiv.org/abs/2408.12444)
• Keywords: secure
• Abstract: Time-Lock Puzzles (TLPs) have been developed to securely transmit sensitive information into the future without relying on a trusted third party. Multi-instance TLP is a scalable variant of TLP that enables a server to efficiently find solutions to different puzzles provided by a client at once. Nevertheless, existing multi-instance TLPs lack support for (verifiable) homomorphic computation. To address this limitation, we introduce the "Multi-Instance partially Homomorphic TLP" (MH-TLP), a multi-instance TLP supporting efficient verifiable homomorphic linear combinations of puzzles belonging to a client. It ensures anyone can verify the correctness of computations and solutions. Building on MH-TLP, we further propose the "Multi-instance Multi-client verifiable partially Homomorphic TLP" (MMH-TLP). It not only supports all the features of MH-TLP but also allows for verifiable homomorphic linear combinations of puzzles from different clients. Our schemes refrain from using asymmetric-key cryptography for verification and, unlike most homomorphic TLPs, do not require a trusted third party. A comprehensive cost analysis demonstrates that our schemes scale linearly with the number of clients and puzzles.

Title: The 2nd Solution for LSVOS Challenge RVOS Track: Spatial-temporal Refinement for Consistent Semantic Segmentation

• Authors: Tuyen Tran
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12447
• Pdf URL: https://arxiv.org/pdf/2408.12447
• Copy Paste: [[2408.12447]] The 2nd Solution for LSVOS Challenge RVOS Track: Spatial-temporal Refinement for Consistent Semantic Segmentation(https://arxiv.org/abs/2408.12447)
• Keywords: segmentation
• Abstract: Referring Video Object Segmentation (RVOS) is a challenging task due to its requirement for temporal understanding. Due to the obstacle of computational complexity, many state-of-the-art models are trained on short time intervals. During testing, while these models can effectively process information over short time steps, they struggle to maintain consistent perception over prolonged time sequences, leading to inconsistencies in the resulting semantic segmentation masks. To address this challenge, we take a step further in this work by leveraging the tracking capabilities of the newly introduced Segment Anything Model version 2 (SAM-v2) to enhance the temporal consistency of the referring object segmentation model. Our method achieved a score of 60.40 \mathcal{J\text{\&}F} on the test set of the MeViS dataset, placing 2nd place in the final ranking of the RVOS Track at the ECCV 2024 LSVOS Challenge.

Title: Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing

• Authors: Mengqi Zhang, Bowen Fang, Qiang Liu, Pengjie Ren, Shu Wu, Zhumin Chen, Liang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12456
• Pdf URL: https://arxiv.org/pdf/2408.12456
• Copy Paste: [[2408.12456]] Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing(https://arxiv.org/abs/2408.12456)
• Keywords: large language model
• Abstract: Large language models (LLMs) face challenges with internal knowledge inaccuracies and outdated information. Knowledge editing has emerged as a pivotal approach to mitigate these issues. Although current knowledge editing techniques exhibit promising performance in single-hop reasoning tasks, they show limitations when applied to multi-hop reasoning. Drawing on cognitive neuroscience and the operational mechanisms of LLMs, we hypothesize that the residual single-hop knowledge after editing causes edited models to revert to their original answers when processing multi-hop questions, thereby undermining their performance in multihop reasoning tasks. To validate this hypothesis, we conduct a series of experiments that empirically confirm our assumptions. Building on the validated hypothesis, we propose a novel knowledge editing method that incorporates a Knowledge Erasure mechanism for Large language model Editing (KELE). Specifically, we design an erasure function for residual knowledge and an injection function for new knowledge. Through joint optimization, we derive the optimal recall vector, which is subsequently utilized within a rank-one editing framework to update the parameters of targeted model layers. Extensive experiments on GPT-J and GPT-2 XL demonstrate that KELE substantially enhances the multi-hop reasoning capability of edited LLMs.

Title: WCEbleedGen: A wireless capsule endoscopy dataset and its benchmarking for automatic bleeding classification, detection, and segmentation

• Authors: Palak Handa, Manas Dhir, Amirreza Mahbod, Florian Schwarzhans, Ramona Woitek, Nidhi Goel, Deepak Gunjan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12466
• Pdf URL: https://arxiv.org/pdf/2408.12466
• Copy Paste: [[2408.12466]] WCEbleedGen: A wireless capsule endoscopy dataset and its benchmarking for automatic bleeding classification, detection, and segmentation(https://arxiv.org/abs/2408.12466)
• Keywords: segmentation
• Abstract: Computer-based analysis of Wireless Capsule Endoscopy (WCE) is crucial. However, a medically annotated WCE dataset for training and evaluation of automatic classification, detection, and segmentation of bleeding and non-bleeding frames is currently lacking. The present work focused on development of a medically annotated WCE dataset called WCEbleedGen for automatic classification, detection, and segmentation of bleeding and non-bleeding frames. It comprises 2,618 WCE bleeding and non-bleeding frames which were collected from various internet resources and existing WCE datasets. A comprehensive benchmarking and evaluation of the developed dataset was done using nine classification-based, three detection-based, and three segmentation-based deep learning models. The dataset is of high-quality, is class-balanced and contains single and multiple bleeding sites. Overall, our standard benchmark results show that Visual Geometric Group (VGG) 19, You Only Look Once version 8 nano (YOLOv8n), and Link network (Linknet) performed best in automatic classification, detection, and segmentation-based evaluations, respectively. Automatic bleeding diagnosis is crucial for WCE video interpretations. This diverse dataset will aid in developing of real-time, multi-task learning-based innovative solutions for automatic bleeding diagnosis in WCE. The dataset and code are publicly available at this https URL and this https URL.

Title: Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning

• Authors: Mushui Liu, Fangtai Wu, Bozheng Li, Ziqian Lu, Yunlong Yu, Xi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12469
• Pdf URL: https://arxiv.org/pdf/2408.12469
• Copy Paste: [[2408.12469]] Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning(https://arxiv.org/abs/2408.12469)
• Keywords: extraction, large language model
• Abstract: Few-shot learning (FSL) aims to recognize new concepts using a limited number of visual samples. Existing approaches attempt to incorporate semantic information into the limited visual data for category understanding. However, these methods often enrich class-level feature representations with abstract category names, failing to capture the nuanced features essential for effective generalization. To address this issue, we propose a novel framework for FSL, which incorporates both the abstract class semantics and the concrete class entities extracted from Large Language Models (LLMs), to enhance the representation of the class prototypes. Specifically, our framework composes a Semantic-guided Visual Pattern Extraction (SVPE) module and a Prototype-Calibration (PC) module, where the SVPE meticulously extracts semantic-aware visual patterns across diverse scales, while the PC module seamlessly integrates these patterns to refine the visual prototype, enhancing its representativeness. Extensive experiments on four few-shot classification benchmarks and the BSCD-FSL cross-domain benchmarks showcase remarkable advancements over the current state-of-the-art methods. Notably, for the challenging one-shot setting, our approach, utilizing the ResNet-12 backbone, achieves an impressive average improvement of 1.95% over the second-best competitor.

Title: Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition

• Authors: Bozheng Li, Mushui Liu, Gaoang Wang, Yunlong Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12475
• Pdf URL: https://arxiv.org/pdf/2408.12475
• Copy Paste: [[2408.12475]] Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition(https://arxiv.org/abs/2408.12475)
• Keywords: large language model
• Abstract: In this paper, we propose a novel Temporal Sequence-Aware Model (TSAM) for few-shot action recognition (FSAR), which incorporates a sequential perceiver adapter into the pre-training framework, to integrate both the spatial information and the sequential temporal dynamics into the feature embeddings. Different from the existing fine-tuning approaches that capture temporal information by exploring the relationships among all the frames, our perceiver-based adapter recurrently captures the sequential dynamics alongside the timeline, which could perceive the order change. To obtain the discriminative representations for each class, we extend a textual corpus for each class derived from the large language models (LLMs) and enrich the visual prototypes by integrating the contextual semantic information. Besides, We introduce an unbalanced optimal transport strategy for feature matching that mitigates the impact of class-unrelated features, thereby facilitating more effective decision-making. Experimental results on five FSAR datasets demonstrate that our method set a new benchmark, beating the second-best competitors with large margins.

Title: Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese

• Authors: Khang T. Doan, Bao G. Huynh, Dung T. Hoang, Thuc D. Pham, Nhat H. Pham, Quan T.M. Nguyen, Bang Q. Vo, Suong N. Hoang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12480
• Pdf URL: https://arxiv.org/pdf/2408.12480
• Copy Paste: [[2408.12480]] Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese(https://arxiv.org/abs/2408.12480)
• Keywords: robust, extraction, large language model
• Abstract: In this report, we introduce Vintern-1B, a reliable 1-billion-parameters multimodal large language model (MLLM) for Vietnamese language tasks. By integrating the Qwen2-0.5B-Instruct language model with the InternViT-300M-448px visual model, Vintern-1B is optimized for a range of applications, including optical character recognition (OCR), document extraction, and general question-answering in Vietnamese context. The model is fine-tuned on an extensive dataset of over 3 million image-question-answer pairs, achieving robust performance and reliable results across multiple Vietnamese language benchmarks like OpenViVQA and ViTextVQA. Vintern-1B is small enough to fit into various on-device applications easily. Additionally, we have open-sourced several Vietnamese vision question answering (VQA) datasets for text and diagrams, created with Gemini 1.5 Flash. Our models are available at: this https URL.

Title: Scribbles for All: Benchmarking Scribble Supervised Segmentation Across Datasets

• Authors: Wolfgang Boettcher, Lukas Hoyer, Ozan Unal, Jan Eric Lenssen, Bernt Schiele
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12489
• Pdf URL: https://arxiv.org/pdf/2408.12489
• Copy Paste: [[2408.12489]] Scribbles for All: Benchmarking Scribble Supervised Segmentation Across Datasets(https://arxiv.org/abs/2408.12489)
• Keywords: segmentation
• Abstract: In this work, we introduce Scribbles for All, a label and training data generation algorithm for semantic segmentation trained on scribble labels. Training or fine-tuning semantic segmentation models with weak supervision has become an important topic recently and was subject to significant advances in model quality. In this setting, scribbles are a promising label type to achieve high quality segmentation results while requiring a much lower annotation effort than usual pixel-wise dense semantic segmentation annotations. The main limitation of scribbles as source for weak supervision is the lack of challenging datasets for scribble segmentation, which hinders the development of novel methods and conclusive evaluations. To overcome this limitation, Scribbles for All provides scribble labels for several popular segmentation datasets and provides an algorithm to automatically generate scribble labels for any dataset with dense annotations, paving the way for new insights and model advancements in the field of weakly supervised segmentation. In addition to providing datasets and algorithm, we evaluate state-of-the-art segmentation models on our datasets and show that models trained with our synthetic labels perform competitively with respect to models trained on manual labels. Thus, our datasets enable state-of-the-art research into methods for scribble-labeled semantic segmentation. The datasets, scribble generation algorithm, and baselines are publicly available at this https URL

Title: GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models

• Authors: Kunsheng Tang, Wenbo Zhou, Jie Zhang, Aishan Liu, Gelei Deng, Shuai Li, Peigui Qi, Weiming Zhang, Tianwei Zhang, Nenghai Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12494
• Pdf URL: https://arxiv.org/pdf/2408.12494
• Copy Paste: [[2408.12494]] GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models(https://arxiv.org/abs/2408.12494)
• Keywords: robust, fair, explainability, large language model
• Abstract: Large language models (LLMs) have exhibited remarkable capabilities in natural language generation, but they have also been observed to magnify societal biases, particularly those related to gender. In response to this issue, several benchmarks have been proposed to assess gender bias in LLMs. However, these benchmarks often lack practical flexibility or inadvertently introduce biases. To address these shortcomings, we introduce GenderCARE, a comprehensive framework that encompasses innovative Criteria, bias Assessment, Reduction techniques, and Evaluation metrics for quantifying and mitigating gender bias in LLMs. To begin, we establish pioneering criteria for gender equality benchmarks, spanning dimensions such as inclusivity, diversity, explainability, objectivity, robustness, and realisticity. Guided by these criteria, we construct GenderPair, a novel pair-based benchmark designed to assess gender bias in LLMs comprehensively. Our benchmark provides standardized and realistic evaluations, including previously overlooked gender groups such as transgender and non-binary individuals. Furthermore, we develop effective debiasing techniques that incorporate counterfactual data augmentation and specialized fine-tuning strategies to reduce gender bias in LLMs without compromising their overall performance. Extensive experiments demonstrate a significant reduction in various gender bias benchmarks, with reductions peaking at over 90% and averaging above 35% across 17 different LLMs. Importantly, these reductions come with minimal variability in mainstream language tasks, remaining below 2%. By offering a realistic assessment and tailored reduction of gender biases, we hope that our GenderCARE can represent a significant step towards achieving fairness and equity in LLMs. More details are available at this https URL.

Title: PCGRL+: Scaling, Control and Generalization in Reinforcement Learning Level Generators

• Authors: Sam Earle, Zehua Jiang, Julian Togelius
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12525
• Pdf URL: https://arxiv.org/pdf/2408.12525
• Copy Paste: [[2408.12525]] PCGRL+: Scaling, Control and Generalization in Reinforcement Learning Level Generators(https://arxiv.org/abs/2408.12525)
• Keywords: robust
• Abstract: Procedural Content Generation via Reinforcement Learning (PCGRL) has been introduced as a means by which controllable designer agents can be trained based only on a set of computable metrics acting as a proxy for the level's quality and key characteristics. While PCGRL offers a unique set of affordances for game designers, it is constrained by the compute-intensive process of training RL agents, and has so far been limited to generating relatively small levels. To address this issue of scale, we implement several PCGRL environments in Jax so that all aspects of learning and simulation happen in parallel on the GPU, resulting in faster environment simulation; removing the CPU-GPU transfer of information bottleneck during RL training; and ultimately resulting in significantly improved training speed. We replicate several key results from prior works in this new framework, letting models train for much longer than previously studied, and evaluating their behavior after 1 billion timesteps. Aiming for greater control for human designers, we introduce randomized level sizes and frozen "pinpoints" of pivotal game tiles as further ways of countering overfitting. To test the generalization ability of learned generators, we evaluate models on large, out-of-distribution map sizes, and find that partial observation sizes learn more robust design strategies.

Title: Show-o: One Single Transformer to Unify Multimodal Understanding and Generation

• Authors: Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12528
• Pdf URL: https://arxiv.org/pdf/2408.12528
• Copy Paste: [[2408.12528]] Show-o: One Single Transformer to Unify Multimodal Understanding and Generation(https://arxiv.org/abs/2408.12528)
• Keywords: diffusion, transformer
• Abstract: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model. Code and models are released at this https URL.

Title: Towards Evaluating and Building Versatile Large Language Models for Medicine

• Authors: Chaoyi Wu, Pengcheng Qiu, Jinxin Liu, Hongfei Gu, Na Li, Ya Zhang, Yanfeng Wang, Weidi Xie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12547
• Pdf URL: https://arxiv.org/pdf/2408.12547
• Copy Paste: [[2408.12547]] Towards Evaluating and Building Versatile Large Language Models for Medicine(https://arxiv.org/abs/2408.12547)
• Keywords: large language model
• Abstract: In this study, we present MedS-Bench, a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. Unlike existing benchmarks that focus on multiple-choice question answering, MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation, among others. We evaluated six leading LLMs, e.g., MEDITRON, Mistral, InternLM 2, Llama 3, GPT-4, and Claude-3.5 using few-shot prompting, and found that even the most sophisticated models struggle with these complex tasks. To address these limitations, we developed MedS-Ins, a large-scale instruction tuning dataset for medicine. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks. To demonstrate the dataset's utility, we conducted a proof-of-concept experiment by performing instruction tuning on a lightweight, open-source medical language model. The resulting model, MMedIns-Llama 3, significantly outperformed existing models across nearly all clinical tasks. To promote further advancements in the application of LLMs to clinical challenges, we have made the MedS-Ins dataset fully accessible and invite the research community to contribute to its expansion.Additionally, we have launched a dynamic leaderboard for MedS-Bench, which we plan to regularly update the test set to track progress and enhance the adaptation of general LLMs to the medical domain. Leaderboard: this https URL. Github: this https URL.

Title: Human-In-The-Loop Machine Learning for Safe and Ethical Autonomous Vehicles: Principles, Challenges, and Opportunities

• Authors: Yousef Emami, Kai Li, Luis Almeida, Wei Ni, Zhu Han
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12548
• Pdf URL: https://arxiv.org/pdf/2408.12548
• Copy Paste: [[2408.12548]] Human-In-The-Loop Machine Learning for Safe and Ethical Autonomous Vehicles: Principles, Challenges, and Opportunities(https://arxiv.org/abs/2408.12548)
• Keywords: robust
• Abstract: Rapid advances in Machine Learning (ML) have triggered new trends in Autonomous Vehicles (AVs). ML algorithms play a crucial role in interpreting sensor data, predicting potential hazards, and optimizing navigation strategies. However, achieving full autonomy in cluttered and complex situations, such as intricate intersections, diverse sceneries, varied trajectories, and complex missions, is still challenging, and the cost of data labeling remains a significant bottleneck. The adaptability and robustness of humans in complex scenarios motivate the inclusion of humans in ML process, leveraging their creativity, ethical power, and emotional intelligence to improve ML effectiveness. The scientific community knows this approach as Human-In-The-Loop Machine Learning (HITL-ML). Towards safe and ethical autonomy, we present a review of HITL-ML for AVs, focusing on Curriculum Learning (CL), Human-In-The-Loop Reinforcement Learning (HITL-RL), Active Learning (AL), and ethical principles. In CL, human experts systematically train ML models by starting with simple tasks and gradually progressing to more difficult ones. HITL-RL significantly enhances the RL process by incorporating human input through techniques like reward shaping, action injection, and interactive learning. AL streamlines the annotation process by targeting specific instances that need to be labeled with human oversight, reducing the overall time and cost associated with training. Ethical principles must be embedded in AVs to align their behavior with societal values and norms. In addition, we provide insights and specify future research directions.

Title: Comparing YOLOv5 Variants for Vehicle Detection: A Performance Analysis

• Authors: Athulya Sundaresan Geetha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12550
• Pdf URL: https://arxiv.org/pdf/2408.12550
• Copy Paste: [[2408.12550]] Comparing YOLOv5 Variants for Vehicle Detection: A Performance Analysis(https://arxiv.org/abs/2408.12550)
• Keywords: robust
• Abstract: Vehicle detection is an important task in the management of traffic and automatic vehicles. This study provides a comparative analysis of five YOLOv5 variants, YOLOv5n6s, YOLOv5s6s, YOLOv5m6s, YOLOv5l6s, and YOLOv5x6s, for vehicle detection in various environments. The research focuses on evaluating the effectiveness of these models in detecting different types of vehicles, such as Car, Bus, Truck, Bicycle, and Motorcycle, under varying conditions including lighting, occlusion, and weather. Performance metrics such as precision, recall, F1-score, and mean Average Precision are utilized to assess the accuracy and reliability of each model. YOLOv5n6s demonstrated a strong balance between precision and recall, particularly in detecting Cars. YOLOv5s6s and YOLOv5m6s showed improvements in recall, enhancing their ability to detect all relevant objects. YOLOv5l6s, with its larger capacity, provided robust performance, especially in detecting Cars, but not good with identifying Motorcycles and Bicycles. YOLOv5x6s was effective in recognizing Buses and Cars but faced challenges with Motorcycle class.

Title: ssProp: Energy-Efficient Training for Convolutional Neural Networks with Scheduled Sparse Back Propagation

• Authors: Lujia Zhong, Shuo Huang, Yonggang Shi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12561
• Pdf URL: https://arxiv.org/pdf/2408.12561
• Copy Paste: [[2408.12561]] ssProp: Energy-Efficient Training for Convolutional Neural Networks with Scheduled Sparse Back Propagation(https://arxiv.org/abs/2408.12561)
• Keywords: diffusion, generative, large language model
• Abstract: Recently, deep learning has made remarkable strides, especially with generative modeling, such as large language models and probabilistic diffusion models. However, training these models often involves significant computational resources, requiring billions of petaFLOPs. This high resource consumption results in substantial energy usage and a large carbon footprint, raising critical environmental concerns. Back-propagation (BP) is a major source of computational expense during training deep learning models. To advance research on energy-efficient training and allow for sparse learning on any machine and device, we propose a general, energy-efficient convolution module that can be seamlessly integrated into any deep learning architecture. Specifically, we introduce channel-wise sparsity with additional gradient selection schedulers during backward based on the assumption that BP is often dense and inefficient, which can lead to over-fitting and high computational consumption. Our experiments demonstrate that our approach reduces 40\% computations while potentially improving model performance, validated on image classification and generation tasks. This reduction can lead to significant energy savings and a lower carbon footprint during the research and development phases of large-scale AI systems. Additionally, our method mitigates over-fitting in a manner distinct from Dropout, allowing it to be combined with Dropout to further enhance model performance and reduce computational resource usage. Extensive experiments validate that our method generalizes to a variety of datasets and tasks and is compatible with a wide range of deep learning architectures and modules. Code is publicly available at this https URL.

Title: Sapiens: Foundation for Human Vision Models

• Authors: Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, Shunsuke Saito
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12569
• Pdf URL: https://arxiv.org/pdf/2408.12569
• Copy Paste: [[2408.12569]] Sapiens: Foundation for Human Vision Models(https://arxiv.org/abs/2408.12569)
• Keywords: segmentation
• Abstract: We present Sapiens, a family of models for four fundamental human-centric vision tasks - 2D pose estimation, body-part segmentation, depth estimation, and surface normal prediction. Our models natively support 1K high-resolution inference and are extremely easy to adapt for individual tasks by simply fine-tuning models pretrained on over 300 million in-the-wild human images. We observe that, given the same computational budget, self-supervised pretraining on a curated dataset of human images significantly boosts the performance for a diverse set of human-centric tasks. The resulting models exhibit remarkable generalization to in-the-wild data, even when labeled data is scarce or entirely synthetic. Our simple model design also brings scalability - model performance across tasks improves as we scale the number of parameters from 0.3 to 2 billion. Sapiens consistently surpasses existing baselines across various human-centric benchmarks. We achieve significant improvements over the prior state-of-the-art on Humans-5K (pose) by 7.6 mAP, Humans-2K (part-seg) by 17.1 mIoU, Hi4D (depth) by 22.4% relative RMSE, and THuman2 (normal) by 53.5% relative angular error.

Title: Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

• Authors: Jamba Team: Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zusman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Opher Lieber, Or Dagan, Orit Cohavi, Raz Alon, Ro'i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shaked Meirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Yehoshua Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, Yoav Shoham
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12570
• Pdf URL: https://arxiv.org/pdf/2408.12570
• Copy Paste: [[2408.12570]] Jamba-1.5: Hybrid Transformer-Mamba Models at Scale(https://arxiv.org/abs/2408.12570)
• Keywords: transformer, large language model
• Abstract: We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.

Title: Enhanced Parking Perception by Multi-Task Fisheye Cross-view Transformers

• Authors: Antonyo Musabini, Ivan Novikov, Sana Soula, Christel Leonet, Lihao Wang, Rachid Benmokhtar, Fabian Burger, Thomas Boulay, Xavier Perrotton
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12575
• Pdf URL: https://arxiv.org/pdf/2408.12575
• Copy Paste: [[2408.12575]] Enhanced Parking Perception by Multi-Task Fisheye Cross-view Transformers(https://arxiv.org/abs/2408.12575)
• Keywords: robust, transformer, segmentation
• Abstract: Current parking area perception algorithms primarily focus on detecting vacant slots within a limited range, relying on error-prone homographic projection for both labeling and inference. However, recent advancements in Advanced Driver Assistance System (ADAS) require interaction with end-users through comprehensive and intelligent Human-Machine Interfaces (HMIs). These interfaces should present a complete perception of the parking area going from distinguishing vacant slots' entry lines to the orientation of other parked vehicles. This paper introduces Multi-Task Fisheye Cross View Transformers (MT F-CVT), which leverages features from a four-camera fisheye Surround-view Camera System (SVCS) with multihead attentions to create a detailed Bird-Eye View (BEV) grid feature map. Features are processed by both a segmentation decoder and a Polygon-Yolo based object detection decoder for parking slots and vehicles. Trained on data labeled using LiDAR, MT F-CVT positions objects within a 25m x 25m real open-road scenes with an average error of only 20 cm. Our larger model achieves an F-1 score of 0.89. Moreover the smaller model operates at 16 fps on an Nvidia Jetson Orin embedded board, with similar detection results to the larger one. MT F-CVT demonstrates robust generalization capability across different vehicles and camera rig configurations. A demo video from an unseen vehicle and camera rig is available at: this https URL.

Title: A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language

• Authors: Ekdeep Singh Lubana, Kyogo Kawaguchi, Robert P. Dick, Hidenori Tanaka
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12578
• Pdf URL: https://arxiv.org/pdf/2408.12578
• Copy Paste: [[2408.12578]] A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language(https://arxiv.org/abs/2408.12578)
• Keywords: transformer
• Abstract: Increase in data, size, or compute can lead to sudden learning of specific capabilities by a neural network -- a phenomenon often called "emergence". Beyond scientific understanding, establishing the causal factors underlying such emergent capabilities is crucial to enable risk regulation frameworks for AI. In this work, we seek inspiration from study of emergent properties in other fields and propose a phenomenological definition for the concept in the context of neural networks. Our definition implicates the acquisition of specific structures underlying the data-generating process as a cause of sudden performance growth for specific, narrower tasks. We empirically investigate this definition by proposing an experimental system grounded in a context-sensitive formal language and find that Transformers trained to perform tasks on top of strings from this language indeed exhibit emergent capabilities. Specifically, we show that once the language's underlying grammar and context-sensitivity inducing structures are learned by the model, performance on narrower tasks suddenly begins to improve. We then analogize our network's learning dynamics with the process of percolation on a bipartite graph, establishing a formal phase transition model that predicts the shift in the point of emergence observed in experiment when changing the data structure. Overall, our experimental and theoretical frameworks yield a step towards better defining, characterizing, and predicting emergence in neural networks.

Title: RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment

• Authors: Xiaohan Wang, Xiaoyan Yang, Yuqi Zhu, Yue Shen, Jian Wang, Peng Wei, Lei Liang, Jinjie Gu, Huajun Chen, Ningyu Zhang
• Subjects: cs.CL, cs.AI, cs.HC, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12579
• Pdf URL: https://arxiv.org/pdf/2408.12579
• Copy Paste: [[2408.12579]] RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment(https://arxiv.org/abs/2408.12579)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) like GPT-4, MedPaLM-2, and Med-Gemini achieve performance competitively with human experts across various medical benchmarks. However, they still face challenges in making professional diagnoses akin to physicians, particularly in efficiently gathering patient information and reasoning the final diagnosis. To this end, we introduce the RuleAlign framework, designed to align LLMs with specific diagnostic rules. We develop a medical dialogue dataset comprising rule-based communications between patients and physicians and design an alignment learning approach through preference learning. Experimental results demonstrate the effectiveness of the proposed approach. We hope that our work can serve as an inspiration for exploring the potential of LLMs as AI physicians.

Title: Identifying the Best Arm in the Presence of Global Environment Shifts

• Authors: Phurinut Srisawad, Juergen Branke, Long Tran-Thanh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12581
• Pdf URL: https://arxiv.org/pdf/2408.12581
• Copy Paste: [[2408.12581]] Identifying the Best Arm in the Presence of Global Environment Shifts(https://arxiv.org/abs/2408.12581)
• Keywords: robust
• Abstract: This paper formulates a new Best-Arm Identification problem in the non-stationary stochastic bandits setting, where the means of all arms are shifted in the same way due to a global influence of the environment. The aim is to identify the unique best arm across environmental change given a fixed total budget. While this setting can be regarded as a special case of Adversarial Bandits or Corrupted Bandits, we demonstrate that existing solutions tailored to those settings do not fully utilise the nature of this global influence, and thus, do not work well in practice (despite their theoretical guarantees). To overcome this issue, in this paper we develop a novel selection policy that is consistent and robust in dealing with global environmental shifts. We then propose an allocation policy, LinLUCB, which exploits information about global shifts across all arms in each environment. Empirical tests depict a significant improvement in our policies against other existing methods.

Title: Real-Time Video Generation with Pyramid Attention Broadcast

• Authors: Xuanlei Zhao, Xiaolong Jin, Kai Wang, Yang You
• Subjects: cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.12588
• Pdf URL: https://arxiv.org/pdf/2408.12588
• Copy Paste: [[2408.12588]] Real-Time Video Generation with Pyramid Attention Broadcast(https://arxiv.org/abs/2408.12588)
• Keywords: robust, diffusion
• Abstract: We present Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free approach for DiT-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style. It applies different broadcast strategies to each attention based on their variance for best efficiency. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates superior results across three models compared to baselines, achieving real-time generation for up to 720p videos. We anticipate that our simple yet effective method will serve as a robust baseline and facilitate future research and application for video generation.

Title: xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations

• Authors: Can Qin, Congying Xia, Krithika Ramakrishnan, Michael Ryoo, Lifu Tu, Yihao Feng, Manli Shu, Honglu Zhou, Anas Awadalla, Jun Wang, Senthil Purushwalkam, Le Xue, Yingbo Zhou, Huan Wang, Silvio Savarese, Juan Carlos Niebles, Zeyuan Chen, Ran Xu, Caiming Xiong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12590
• Pdf URL: https://arxiv.org/pdf/2408.12590
• Copy Paste: [[2408.12590]] xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations(https://arxiv.org/abs/2408.12590)
• Keywords: robust, diffusion, transformer
• Abstract: We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. Building on recent advancements, such as OpenAI's Sora, we explore the latent diffusion model (LDM) architecture and introduce a video variational autoencoder (VidVAE). VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens and the computational demands associated with generating long-sequence videos. To further address the computational costs, we propose a divide-and-merge strategy that maintains temporal consistency across video segments. Our Diffusion Transformer (DiT) model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios. We have devised a data processing pipeline from the very beginning and collected over 13M high-quality video-text pairs. The pipeline includes multiple steps such as clipping, text detection, motion estimation, aesthetics scoring, and dense captioning based on our in-house video-LLM model. Training the VidVAE and DiT models required approximately 40 and 642 H100 days, respectively. Our model supports over 14-second 720p video generation in an end-to-end way and demonstrates competitive performance against state-of-the-art T2V models.

Title: Controllable Text Generation for Large Language Models: A Survey

• Authors: Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, Zhiyu Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12599
• Pdf URL: https://arxiv.org/pdf/2408.12599
• Copy Paste: [[2408.12599]] Controllable Text Generation for Large Language Models: A Survey(https://arxiv.org/abs/2408.12599)
• Keywords: large language model
• Abstract: In Natural Language Processing (NLP), Large Language Models (LLMs) have demonstrated high text generation quality. However, in real-world applications, LLMs must meet increasingly complex requirements. Beyond avoiding misleading or inappropriate content, LLMs are also expected to cater to specific user needs, such as imitating particular writing styles or generating text with poetic richness. These varied demands have driven the development of Controllable Text Generation (CTG) techniques, which ensure that outputs adhere to predefined control conditions--such as safety, sentiment, thematic consistency, and linguistic style--while maintaining high standards of helpfulness, fluency, and diversity. This paper systematically reviews the latest advancements in CTG for LLMs, offering a comprehensive definition of its core concepts and clarifying the requirements for control conditions and text quality. We categorize CTG tasks into two primary types: content control and attribute control. The key methods are discussed, including model retraining, fine-tuning, reinforcement learning, prompt engineering, latent space manipulation, and decoding-time intervention. We analyze each method's characteristics, advantages, and limitations, providing nuanced insights for achieving generation control. Additionally, we review CTG evaluation methods, summarize its applications across domains, and address key challenges in current research, including reduced fluency and practicality. We also propose several appeals, such as placing greater emphasis on real-world applications in future research. This paper aims to offer valuable guidance to researchers and developers in the field. Our reference list and Chinese version are open-sourced at this https URL.

Title: DreamCinema: Cinematic Transfer with Free Camera and 3D Character

• Authors: Weiliang Chen, Fangfu Liu, Diankun Wu, Haowen Sun, Haixu Song, Yueqi Duan
• Subjects: cs.CV, cs.GR, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.12601
• Pdf URL: https://arxiv.org/pdf/2408.12601
• Copy Paste: [[2408.12601]] DreamCinema: Cinematic Transfer with Free Camera and 3D Character(https://arxiv.org/abs/2408.12601)
• Keywords: generative
• Abstract: We are living in a flourishing era of digital media, where everyone has the potential to become a personal filmmaker. Current research on cinematic transfer empowers filmmakers to reproduce and manipulate the visual elements (e.g., cinematography and character behaviors) from classic shots. However, characters in the reimagined films still rely on manual crafting, which involves significant technical complexity and high costs, making it unattainable for ordinary users. Furthermore, their estimated cinematography lacks smoothness due to inadequate capturing of inter-frame motion and modeling of physical trajectories. Fortunately, the remarkable success of 2D and 3D AIGC has opened up the possibility of efficiently generating characters tailored to users' needs, diversifying cinematography. In this paper, we propose DreamCinema, a novel cinematic transfer framework that pioneers generative AI into the film production paradigm, aiming at facilitating user-friendly film creation. Specifically, we first extract cinematic elements (i.e., human and camera pose) and optimize the camera trajectory. Then, we apply a character generator to efficiently create 3D high-quality characters with a human structure prior. Finally, we develop a structure-guided motion transfer strategy to incorporate generated characters into film creation and transfer it via 3D graphics engines smoothly. Extensive experiments demonstrate the effectiveness of our method for creating high-quality films with free camera and 3D characters.