2024-09-30

Title: SSP-RACL: Classification of Noisy Fundus Images with Self-Supervised Pretraining and Robust Adaptive Credal Loss

• Authors: Mengwen Ye, Yingzi Huangfu, You Li, Zekuan Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18147
• Pdf URL: https://arxiv.org/pdf/2409.18147
• Copy Paste: [[2409.18147]] SSP-RACL: Classification of Noisy Fundus Images with Self-Supervised Pretraining and Robust Adaptive Credal Loss(https://arxiv.org/abs/2409.18147)
• Keywords: robust
• Abstract: Fundus image classification is crucial in the computer aided diagnosis tasks, but label noise significantly impairs the performance of deep neural networks. To address this challenge, we propose a robust framework, Self-Supervised Pre-training with Robust Adaptive Credal Loss (SSP-RACL), for handling label noise in fundus image datasets. First, we use Masked Autoencoders (MAE) for pre-training to extract features, unaffected by label noise. Subsequently, RACL employ a superset learning framework, setting confidence thresholds and adaptive label relaxation parameter to construct possibility distributions and provide more reliable ground-truth estimates, thus effectively suppressing the memorization effect. Additionally, we introduce clinical knowledge-based asymmetric noise generation to simulate real-world noisy fundus image datasets. Experimental results demonstrate that our proposed method outperforms existing approaches in handling label noise, showing superior performance.

Title: A Survey on Neural Architecture Search Based on Reinforcement Learning

• Authors: Wenzhu Shao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18163
• Pdf URL: https://arxiv.org/pdf/2409.18163
• Copy Paste: [[2409.18163]] A Survey on Neural Architecture Search Based on Reinforcement Learning(https://arxiv.org/abs/2409.18163)
• Keywords: extraction
• Abstract: The automation of feature extraction of machine learning has been successfully realized by the explosive development of deep learning. However, the structures and hyperparameters of deep neural network architectures also make huge difference on the performance in different tasks. The process of exploring optimal structures and hyperparameters often involves a lot of tedious human intervene. As a result, a legitimate question is to ask for the automation of searching for optimal network structures and hyperparameters. The work of automation of exploring optimal hyperparameters is done by Hyperparameter Optimization. Neural Architecture Search is aimed to automatically find the best network structure given specific tasks. In this paper, we firstly introduced the overall development of Neural Architecture Search and then focus mainly on providing an overall and understandable survey about Neural Architecture Search works that are relevant with reinforcement learning, including improvements and variants based on the hope of satisfying more complex structures and resource-insufficient environment.

Title: Jump Diffusion-Informed Neural Networks with Transfer Learning for Accurate American Option Pricing under Data Scarcity

• Authors: Qiguo Sun, Hanyue Huang, XiBei Yang, Yuwei Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18168
• Pdf URL: https://arxiv.org/pdf/2409.18168
• Copy Paste: [[2409.18168]] Jump Diffusion-Informed Neural Networks with Transfer Learning for Accurate American Option Pricing under Data Scarcity(https://arxiv.org/abs/2409.18168)
• Keywords: diffusion
• Abstract: Option pricing models, essential in financial mathematics and risk management, have been extensively studied and recently advanced by AI methodologies. However, American option pricing remains challenging due to the complexity of determining optimal exercise times and modeling non-linear payoffs resulting from stochastic paths. Moreover, the prevalent use of the Black-Scholes formula in hybrid models fails to accurately capture the discontinuity in the price process, limiting model performance, especially under scarce data conditions. To address these issues, this study presents a comprehensive framework for American option pricing consisting of six interrelated modules, which combine nonlinear optimization algorithms, analytical and numerical models, and neural networks to improve pricing performance. Additionally, to handle the scarce data challenge, this framework integrates the transfer learning through numerical data augmentation and a physically constrained, jump diffusion process-informed neural network to capture the leptokurtosis of the log return distribution. To increase training efficiency, a warm-up period using Bayesian optimization is designed to provide optimal data loss and physical loss coefficients. Experimental results of six case studies demonstrate the accuracy, convergence, physical effectiveness, and generalization of the framework. Moreover, the proposed model shows superior performance in pricing deep out-of-the-money options.

Title: Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey

• Authors: Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Ling Liu
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18169
• Pdf URL: https://arxiv.org/pdf/2409.18169
• Copy Paste: [[2409.18169]] Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey(https://arxiv.org/abs/2409.18169)
• Keywords: defense, attack, large language model
• Abstract: Recent research demonstrates that the nascent fine-tuning-as-a-service business model exposes serious safety concerns -- fine-tuning over a few harmful data uploaded by the users can compromise the safety alignment of the model. The attack, known as harmful fine-tuning, has raised a broad research interest among the community. However, as the attack is still new, \textbf{we observe from our miserable submission experience that there are general misunderstandings within the research community.} We in this paper aim to clear some common concerns for the attack setting, and formally establish the research problem. Specifically, we first present the threat model of the problem, and introduce the harmful fine-tuning attack and its variants. Then we systematically survey the existing literature on attacks/defenses/mechanical analysis of the problem. Finally, we outline future research directions that might contribute to the development of the field. Additionally, we present a list of questions of interest, which might be useful to refer to when reviewers in the peer review process question the realism of the experiment/attack/defense setting. A curated list of relevant papers is maintained and made accessible at: \url{this https URL}.

Title: Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review

• Authors: Emma Croxford, Yanjun Gao, Nicholas Pellegrino, Karen K. Wong, Graham Wills, Elliot First, Frank J. Liao, Cherodeep Goswami, Brian Patterson, Majid Afshar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18170
• Pdf URL: https://arxiv.org/pdf/2409.18170
• Copy Paste: [[2409.18170]] Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review(https://arxiv.org/abs/2409.18170)
• Keywords: large language model
• Abstract: Large Language Models have advanced clinical Natural Language Generation, creating opportunities to manage the volume of medical text. However, the high-stakes nature of medicine requires reliable evaluation, which remains a challenge. In this narrative review, we assess the current evaluation state for clinical summarization tasks and propose future directions to address the resource constraints of expert human evaluation.

Title: LowREm: A Repository of Word Embeddings for 87 Low-Resource Languages Enhanced with Multilingual Graph Knowledge

• Authors: Daniil Gurgurov, Rishu Kumar, Simon Ostermann
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18193
• Pdf URL: https://arxiv.org/pdf/2409.18193
• Copy Paste: [[2409.18193]] LowREm: A Repository of Word Embeddings for 87 Low-Resource Languages Enhanced with Multilingual Graph Knowledge(https://arxiv.org/abs/2409.18193)
• Keywords: large language model
• Abstract: Contextualized embeddings based on large language models (LLMs) are available for various languages, but their coverage is often limited for lower resourced languages. Training LLMs for such languages is often difficult due to insufficient data and high computational cost. Especially for very low resource languages, static word embeddings thus still offer a viable alternative. There is, however, a notable lack of comprehensive repositories with such embeddings for diverse languages. To address this, we present LowREm, a centralized repository of static embeddings for 87 low-resource languages. We also propose a novel method to enhance GloVe-based embeddings by integrating multilingual graph knowledge, utilizing another source of knowledge. We demonstrate the superior performance of our enhanced embeddings as compared to contextualized embeddings extracted from XLM-R on sentiment analysis. Our code and data are publicly available under this https URL.

Title: LangSAMP: Language-Script Aware Multilingual Pretraining

• Authors: Yihong Liu, Haotian Ye, Chunlan Ma, Mingyang Wang, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18199
• Pdf URL: https://arxiv.org/pdf/2409.18199
• Copy Paste: [[2409.18199]] LangSAMP: Language-Script Aware Multilingual Pretraining(https://arxiv.org/abs/2409.18199)
• Keywords: transformer
• Abstract: Recent multilingual pretrained language models (mPLMs) often avoid using language embeddings -- learnable vectors assigned to different languages. These embeddings are discarded for two main reasons: (1) mPLMs are expected to have a single, unified parameter set across all languages, and (2) they need to function seamlessly as universal text encoders without requiring language IDs as input. However, this removal increases the burden on token embeddings to encode all language-specific information, which may hinder the model's ability to produce more language-neutral representations. To address this challenge, we propose Language-Script Aware Multilingual Pretraining (LangSAMP), a method that incorporates both language and script embeddings to enhance representation learning while maintaining a simple architecture. Specifically, we integrate these embeddings into the output of the transformer blocks before passing the final representations to the language modeling head for prediction. We apply LangSAMP to the continual pretraining of XLM-R on a highly multilingual corpus covering more than 500 languages. The resulting model consistently outperforms the baseline. Extensive analysis further shows that language/script embeddings encode language/script-specific information, which improves the selection of source languages for crosslingual transfer. We make our code and models publicly available at \url{this https URL}.

Title: Evaluation of Security of ML-based Watermarking: Copy and Removal Attacks

• Authors: Vitaliy Kinakh, Brian Pulfer, Yury Belousov, Pierre Fernandez, Teddy Furon, Slava Voloshynovskiy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18211
• Pdf URL: https://arxiv.org/pdf/2409.18211
• Copy Paste: [[2409.18211]] Evaluation of Security of ML-based Watermarking: Copy and Removal Attacks(https://arxiv.org/abs/2409.18211)
• Keywords: security, protect, attack, robust, watermark
• Abstract: The vast amounts of digital content captured from the real world or AI-generated media necessitate methods for copyright protection, traceability, or data provenance verification. Digital watermarking serves as a crucial approach to address these challenges. Its evolution spans three generations: handcrafted, autoencoder-based, and foundation model based methods. %Its evolution spans three generations: handcrafted methods, autoencoder-based schemes, and methods based on foundation models. While the robustness of these systems is well-documented, the security against adversarial attacks remains underexplored. This paper evaluates the security of foundation models' latent space digital watermarking systems that utilize adversarial embedding techniques. A series of experiments investigate the security dimensions under copy and removal attacks, providing empirical insights into these systems' vulnerabilities. All experimental codes and results are available at this https URL}{repository

Title: Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey

• Authors: Yi Zhang, Zhen Chen, Chih-Hong Cheng, Wenjie Ruan, Xiaowei Huang, Dezong Zhao, David Flynn, Siddartha Khastgir, Xingyu Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18214
• Pdf URL: https://arxiv.org/pdf/2409.18214
• Copy Paste: [[2409.18214]] Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey(https://arxiv.org/abs/2409.18214)
• Keywords: security, privacy, robust, fair, explainability, diffusion
• Abstract: Text-to-Image (T2I) Diffusion Models (DMs) have garnered widespread attention for their impressive advancements in image generation. However, their growing popularity has raised ethical and social concerns related to key non-functional properties of trustworthiness, such as robustness, fairness, security, privacy, factuality, and explainability, similar to those in traditional deep learning (DL) tasks. Conventional approaches for studying trustworthiness in DL tasks often fall short due to the unique characteristics of T2I DMs, e.g., the multi-modal nature. Given the challenge, recent efforts have been made to develop new methods for investigating trustworthiness in T2I DMs via various means, including falsification, enhancement, verification \& validation and assessment. However, there is a notable lack of in-depth analysis concerning those non-functional properties and means. In this survey, we provide a timely and focused review of the literature on trustworthy T2I DMs, covering a concise-structured taxonomy from the perspectives of property, means, benchmarks and applications. Our review begins with an introduction to essential preliminaries of T2I DMs, and then we summarise key definitions/metrics specific to T2I tasks and analyses the means proposed in recent literature based on these definitions/metrics. Additionally, we review benchmarks and domain applications of T2I DMs. Finally, we highlight the gaps in current research, discuss the limitations of existing methods, and propose future research directions to advance the development of trustworthy T2I DMs. Furthermore, we keep up-to-date updates in this field to track the latest developments and maintain our GitHub repository at: this https URL

Title: Revolutionizing Payload Inspection: A Self-Supervised Journey to Precision with Few Shots

• Authors: Kyle Stein, Arash Mahyari, Guillermo Francia III, Eman El-Sheikh
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18219
• Pdf URL: https://arxiv.org/pdf/2409.18219
• Copy Paste: [[2409.18219]] Revolutionizing Payload Inspection: A Self-Supervised Journey to Precision with Few Shots(https://arxiv.org/abs/2409.18219)
• Keywords: security, attack, transformer
• Abstract: As networks continue to expand and become more interconnected, the need for novel malware detection methods becomes more pronounced. Traditional security measures are increasingly inadequate against the sophistication of modern cyber attacks. Deep Packet Inspection (DPI) has been pivotal in enhancing network security, offering an in-depth analysis of network traffic that surpasses conventional monitoring techniques. DPI not only examines the metadata of network packets, but also dives into the actual content being carried within the packet payloads, providing a comprehensive view of the data flowing through networks. The integration of advanced deep learning techniques with DPI has introduced modern methodologies into malware detection. However, the challenge with the state-of-the-art supervised learning approaches is that they prevent the generalization to unseen attacks embedded in the payloads, prohibiting them from accurately detecting new attacks and transferring knowledge learned from previous attacks to the new attacks with small labeled sample sizes. This paper leverages the recent advancements in self-supervised learning and few-shot learning. Our proposed self-supervised approach trains a transformer to learn the embedding of the payloads from a vast amount of unlabeled datasets by masking portions of payloads, leading to a learnt representation that well generalizes to various downstream tasks. Once the representation is extracted from payloads, they are used to train a malware detection algorithm. The representation obtained from the transformer is then used to adapt the malware detector to novel types of attacks using few-shot learning approaches. Our experimental results across several datasets show the great success and generalization of the proposed approach to novel scenarios.

Title: Visual Concept Networks: A Graph-Based Approach to Detecting Anomalous Data in Deep Neural Networks

• Authors: Debargha Ganguly, Debayan Gupta, Vipin Chaudhary
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18235
• Pdf URL: https://arxiv.org/pdf/2409.18235
• Copy Paste: [[2409.18235]] Visual Concept Networks: A Graph-Based Approach to Detecting Anomalous Data in Deep Neural Networks(https://arxiv.org/abs/2409.18235)
• Keywords: robust
• Abstract: Deep neural networks (DNNs), while increasingly deployed in many applications, struggle with robustness against anomalous and out-of-distribution (OOD) data. Current OOD benchmarks often oversimplify, focusing on single-object tasks and not fully representing complex real-world anomalies. This paper introduces a new, straightforward method employing graph structures and topological features to effectively detect both far-OOD and near-OOD data. We convert images into networks of interconnected human understandable features or visual concepts. Through extensive testing on two novel tasks, including ablation studies with large vocabularies and diverse tasks, we demonstrate the method's effectiveness. This approach enhances DNN resilience to OOD data and promises improved performance in various applications.

Title: Development of an Edge Resilient ML Ensemble to Tolerate ICS Adversarial Attacks

• Authors: Likai Yao, Qinxuan Shi, Zhanglong Yang, Sicong Shao, Salim Hariri
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18244
• Pdf URL: https://arxiv.org/pdf/2409.18244
• Copy Paste: [[2409.18244]] Development of an Edge Resilient ML Ensemble to Tolerate ICS Adversarial Attacks(https://arxiv.org/abs/2409.18244)
• Keywords: security, privacy, defense, attack
• Abstract: Deploying machine learning (ML) in dynamic data-driven applications systems (DDDAS) can improve the security of industrial control systems (ICS). However, ML-based DDDAS are vulnerable to adversarial attacks because adversaries can alter the input data slightly so that the ML models predict a different result. In this paper, our goal is to build a resilient edge machine learning (reML) architecture that is designed to withstand adversarial attacks by performing Data Air Gap Transformation (DAGT) to anonymize data feature spaces using deep neural networks and randomize the ML models used for predictions. The reML is based on the Resilient DDDAS paradigm, Moving Target Defense (MTD) theory, and TinyML and is applied to combat adversarial attacks on ICS. Furthermore, the proposed approach is power-efficient and privacy-preserving and, therefore, can be deployed on power-constrained devices to enhance ICS security. This approach enables resilient ML inference at the edge by shifting the computation from the computing-intensive platforms to the resource-constrained edge devices. The incorporation of TinyML with TensorFlow Lite ensures efficient resource utilization and, consequently, makes reML suitable for deployment in various industrial control environments. Furthermore, the dynamic nature of reML, facilitated by the resilient DDDAS development environment, allows for continuous adaptation and improvement in response to emerging threats. Lastly, we evaluate our approach on an ICS dataset and demonstrate that reML provides a viable and effective solution for resilient ML inference at the edge devices.

Title: Discovering New Shadow Patterns for Black-Box Attacks on Lane Detection of Autonomous Vehicles

• Authors: Pedram MohajerAnsari, Alkim Domeke, Jan de Voor, Arkajyoti Mitra, Grace Johnson, Amir Salarpour, Habeeb Olufowobi, Mohammad Hamad, Mert D. Pesé
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18248
• Pdf URL: https://arxiv.org/pdf/2409.18248
• Copy Paste: [[2409.18248]] Discovering New Shadow Patterns for Black-Box Attacks on Lane Detection of Autonomous Vehicles(https://arxiv.org/abs/2409.18248)
• Keywords: security, attack, steal
• Abstract: Ensuring autonomous vehicle (AV) security remains a critical concern. An area of paramount importance is the study of physical-world adversarial examples (AEs) aimed at exploiting vulnerabilities in perception systems. However, most of the prevailing research on AEs has neglected considerations of stealthiness and legality, resulting in scenarios where human drivers would promptly intervene or attackers would be swiftly detected and punished. These limitations hinder the applicability of such examples in real-life settings. In this paper, we introduce a novel approach to generate AEs using what we term negative shadows: deceptive patterns of light on the road created by strategically blocking sunlight, which then cast artificial lane-like patterns. These shadows are inconspicuous to a driver while deceiving AV perception systems, particularly those reliant on lane detection algorithms. By prioritizing the stealthy nature of attacks to minimize driver interventions and ensuring their legality from an attacker's standpoint, a more plausible range of scenarios is established. In multiple scenarios, including at low speeds, our method shows a high safety violation rate. Using a 20-meter negative shadow, it can direct a vehicle off-road with a 100% violation rate at speeds over 10 mph. Other attack scenarios, such as causing collisions, can be performed with at least 30 meters of negative shadow, achieving a 60-100% success rate. The attack also maintains an average stealthiness of 83.6% as measured through a human subject experiment, ensuring its efficacy in covert settings.

Title: Bridging the Protection Gap: Innovative Approaches to Shield Older Adults from AI-Enhanced Scams

• Authors: LD Herrera, London Van Sickle, Ashley Podhradsky
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18249
• Pdf URL: https://arxiv.org/pdf/2409.18249
• Copy Paste: [[2409.18249]] Bridging the Protection Gap: Innovative Approaches to Shield Older Adults from AI-Enhanced Scams(https://arxiv.org/abs/2409.18249)
• Keywords: protect, generative
• Abstract: Artificial Intelligence (AI) is rapidly gaining popularity as individuals, groups, and organizations discover and apply its expanding capabilities. Generative AI creates or alters various content types including text, image, audio, and video that are realistic and challenging to identify as AI-generated constructs. However, guardrails preventing malicious use of AI are easily bypassed. Numerous indications suggest that scammers are already using AI to enhance already successful scams, improving scam effectiveness, speed and credibility, while reducing detectability of scams that target older adults, who are known to be slow to adopt new technologies. Through hypothetical cases analysis of two leading scams, the tech support scams and the romance scams, this paper explores the future of AI in scams affecting older adults by identifying current vulnerabilities and recommending updated defensive measures focusing the establishment of a reliable support network offering elevated support to increase confidence and ability to defend against AI-enhanced scams.

Title: Amodal Instance Segmentation with Diffusion Shape Prior Estimation

• Authors: Minh Tran, Khoa Vo, Tri Nguyen, Ngan Le
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18256
• Pdf URL: https://arxiv.org/pdf/2409.18256
• Copy Paste: [[2409.18256]] Amodal Instance Segmentation with Diffusion Shape Prior Estimation(https://arxiv.org/abs/2409.18256)
• Keywords: diffusion, segmentation
• Abstract: Amodal Instance Segmentation (AIS) presents an intriguing challenge, including the segmentation prediction of both visible and occluded parts of objects within images. Previous methods have often relied on shape prior information gleaned from training data to enhance amodal segmentation. However, these approaches are susceptible to overfitting and disregard object category details. Recent advancements highlight the potential of conditioned diffusion models, pretrained on extensive datasets, to generate images from latent space. Drawing inspiration from this, we propose AISDiff with a Diffusion Shape Prior Estimation (DiffSP) module. AISDiff begins with the prediction of the visible segmentation mask and object category, alongside occlusion-aware processing through the prediction of occluding masks. Subsequently, these elements are inputted into our DiffSP module to infer the shape prior of the object. DiffSP utilizes conditioned diffusion models pretrained on extensive datasets to extract rich visual features for shape prior estimation. Additionally, we introduce the Shape Prior Amodal Predictor, which utilizes attention-based feature maps from the shape prior to refine amodal segmentation. Experiments across various AIS benchmarks demonstrate the effectiveness of our AISDiff.

Title: PCEvE: Part Contribution Evaluation Based Model Explanation for Human Figure Drawing Assessment and Beyond

• Authors: Jongseo Lee, Geo Ahn, Jinwoo Choi, Seongtae Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18260
• Pdf URL: https://arxiv.org/pdf/2409.18260
• Copy Paste: [[2409.18260]] PCEvE: Part Contribution Evaluation Based Model Explanation for Human Figure Drawing Assessment and Beyond(https://arxiv.org/abs/2409.18260)
• Keywords: explainability
• Abstract: For automatic human figure drawing (HFD) assessment tasks, such as diagnosing autism spectrum disorder (ASD) using HFD images, the clarity and explainability of a model decision are crucial. Existing pixel-level attribution-based explainable AI (XAI) approaches demand considerable effort from users to interpret the semantic information of a region in an image, which can be often time-consuming and impractical. To overcome this challenge, we propose a part contribution evaluation based model explanation (PCEvE) framework. On top of the part detection, we measure the Shapley Value of each individual part to evaluate the contribution to a model decision. Unlike existing attribution-based XAI approaches, the PCEvE provides a straightforward explanation of a model decision, i.e., a part contribution histogram. Furthermore, the PCEvE expands the scope of explanations beyond the conventional sample-level to include class-level and task-level insights, offering a richer, more comprehensive understanding of model behavior. We rigorously validate the PCEvE via extensive experiments on multiple HFD assessment datasets. Also, we sanity-check the proposed method with a set of controlled experiments. Additionally, we demonstrate the versatility and applicability of our method to other domains by applying it to a photo-realistic dataset, the Stanford Cars.

Title: Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing

• Authors: Huthaifa I. Ashqar, Ahmed Jaber, Taqwa I. Alhadidi, Mohammed Elhenawy
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18286
• Pdf URL: https://arxiv.org/pdf/2409.18286
• Copy Paste: [[2409.18286]] Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing(https://arxiv.org/abs/2409.18286)
• Keywords: extraction, large language model
• Abstract: This study aims to comprehensively review and empirically evaluate the application of multimodal large language models (MLLMs) and Large Vision Models (VLMs) in object detection for transportation systems. In the first fold, we provide a background about the potential benefits of MLLMs in transportation applications and conduct a comprehensive review of current MLLM technologies in previous studies. We highlight their effectiveness and limitations in object detection within various transportation scenarios. The second fold involves providing an overview of the taxonomy of end-to-end object detection in transportation applications and future directions. Building on this, we proposed empirical analysis for testing MLLMs on three real-world transportation problems that include object detection tasks namely, road safety attributes extraction, safety-critical event detection, and visual reasoning of thermal images. Our findings provide a detailed assessment of MLLM performance, uncovering both strengths and areas for improvement. Finally, we discuss practical limitations and challenges of MLLMs in enhancing object detection in transportation, thereby offering a roadmap for future research and development in this critical area.

Title: Efficient Microscopic Image Instance Segmentation for Food Crystal Quality Control

• Authors: Xiaoyu Ji, Jan P Allebach, Ali Shakouri, Fengqing Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18291
• Pdf URL: https://arxiv.org/pdf/2409.18291
• Copy Paste: [[2409.18291]] Efficient Microscopic Image Instance Segmentation for Food Crystal Quality Control(https://arxiv.org/abs/2409.18291)
• Keywords: segmentation
• Abstract: This paper is directed towards the food crystal quality control area for manufacturing, focusing on efficiently predicting food crystal counts and size distributions. Previously, manufacturers used the manual counting method on microscopic images of food liquid products, which requires substantial human effort and suffers from inconsistency issues. Food crystal segmentation is a challenging problem due to the diverse shapes of crystals and their surrounding hard mimics. To address this challenge, we propose an efficient instance segmentation method based on object detection. Experimental results show that the predicted crystal counting accuracy of our method is comparable with existing segmentation methods, while being five times faster. Based on our experiments, we also define objective criteria for separating hard mimics and food crystals, which could benefit manual annotation tasks on similar dataset.

Title: Causality-based Subject and Task Fingerprints using fMRI Time-series Data

• Authors: Dachuan Song, Li Shen, Duy Duong-Tran, Xuan Wang
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2409.18298
• Pdf URL: https://arxiv.org/pdf/2409.18298
• Copy Paste: [[2409.18298]] Causality-based Subject and Task Fingerprints using fMRI Time-series Data(https://arxiv.org/abs/2409.18298)
• Keywords: generative
• Abstract: Recently, there has been a revived interest in system neuroscience causation models due to their unique capability to unravel complex relationships in multi-scale brain networks. In this paper, our goal is to verify the feasibility and effectiveness of using a causality-based approach for fMRI fingerprinting. Specifically, we propose an innovative method that utilizes the causal dynamics activities of the brain to identify the unique cognitive patterns of individuals (e.g., subject fingerprint) and fMRI tasks (e.g., task fingerprint). The key novelty of our approach stems from the development of a two-timescale linear state-space model to extract 'spatio-temporal' (aka causal) signatures from an individual's fMRI time series data. To the best of our knowledge, we pioneer and subsequently quantify, in this paper, the concept of 'causal fingerprint.' Our method is well-separated from other fingerprint studies as we quantify fingerprints from a cause-and-effect perspective, which are then incorporated with a modal decomposition and projection method to perform subject identification and a GNN-based (Graph Neural Network) model to perform task identification. Finally, we show that the experimental results and comparisons with non-causality-based methods demonstrate the effectiveness of the proposed methods. We visualize the obtained causal signatures and discuss their biological relevance in light of the existing understanding of brain functionalities. Collectively, our work paves the way for further studies on causal fingerprints with potential applications in both healthy controls and neurodegenerative diseases.

Title: Harnessing Wavelet Transformations for Generalizable Deepfake Forgery Detection

• Authors: Lalith Bharadwaj Baru, Shilhora Akshay Patel, Rohit Boddeda
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18301
• Pdf URL: https://arxiv.org/pdf/2409.18301
• Copy Paste: [[2409.18301]] Harnessing Wavelet Transformations for Generalizable Deepfake Forgery Detection(https://arxiv.org/abs/2409.18301)
• Keywords: robust, diffusion, generative
• Abstract: The evolution of digital image manipulation, particularly with the advancement of deep generative models, significantly challenges existing deepfake detection methods, especially when the origin of the deepfake is obscure. To tackle the increasing complexity of these forgeries, we propose \textbf{Wavelet-CLIP}, a deepfake detection framework that integrates wavelet transforms with features derived from the ViT-L/14 architecture, pre-trained in the CLIP fashion. Wavelet-CLIP utilizes Wavelet Transforms to deeply analyze both spatial and frequency features from images, thus enhancing the model's capability to detect sophisticated deepfakes. To verify the effectiveness of our approach, we conducted extensive evaluations against existing state-of-the-art methods for cross-dataset generalization and detection of unseen images generated by standard diffusion models. Our method showcases outstanding performance, achieving an average AUC of 0.749 for cross-data generalization and 0.893 for robustness against unseen deepfakes, outperforming all compared methods. The code can be reproduced from the repo: \url{this https URL}

Title: Towards the Mitigation of Confirmation Bias in Semi-supervised Learning: a Debiased Training Perspective

• Authors: Yu Wang, Yuxuan Yin, Peng Li
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.18316
• Pdf URL: https://arxiv.org/pdf/2409.18316
• Copy Paste: [[2409.18316]] Towards the Mitigation of Confirmation Bias in Semi-supervised Learning: a Debiased Training Perspective(https://arxiv.org/abs/2409.18316)
• Keywords: robust
• Abstract: Semi-supervised learning (SSL) commonly exhibits confirmation bias, where models disproportionately favor certain classes, leading to errors in predicted pseudo labels that accumulate under a self-training paradigm. Unlike supervised settings, which benefit from a rich, static data distribution, SSL inherently lacks mechanisms to correct this self-reinforced bias, necessitating debiased interventions at each training step. Although the generation of debiased pseudo labels has been extensively studied, their effective utilization remains underexplored. Our analysis indicates that data from biased classes should have a reduced influence on parameter updates, while more attention should be given to underrepresented classes. To address these challenges, we introduce TaMatch, a unified framework for debiased training in SSL. TaMatch employs a scaling ratio derived from both a prior target distribution and the model's learning status to estimate and correct bias at each training step. This ratio adjusts the raw predictions on unlabeled data to produce debiased pseudo labels. In the utilization phase, these labels are differently weighted according to their predicted class, enhancing training equity and minimizing class bias. Additionally, TaMatch dynamically adjust the target distribution in response to the model's learning progress, facilitating robust handling of practical scenarios where the prior distribution is unknown. Empirical evaluations show that TaMatch significantly outperforms existing state-of-the-art methods across a range of challenging image classification tasks, highlighting the critical importance of both the debiased generation and utilization of pseudo labels in SSL.

Title: Automated Segmentation and Analysis of Microscopy Images of Laser Powder Bed Fusion Melt Tracks

• Authors: Aagam Shah, Reimar Weissbach, David A. Griggs, A. John Hart, Elif Ertekin, Sameh Tawfick
• Subjects: cs.CV, physics.app-ph
• Abstract URL: https://arxiv.org/abs/2409.18326
• Pdf URL: https://arxiv.org/pdf/2409.18326
• Copy Paste: [[2409.18326]] Automated Segmentation and Analysis of Microscopy Images of Laser Powder Bed Fusion Melt Tracks(https://arxiv.org/abs/2409.18326)
• Keywords: robust, segmentation
• Abstract: With the increasing adoption of metal additive manufacturing (AM), researchers and practitioners are turning to data-driven approaches to optimise printing conditions. Cross-sectional images of melt tracks provide valuable information for tuning process parameters, developing parameter scaling data, and identifying defects. Here we present an image segmentation neural network that automatically identifies and measures melt track dimensions from a cross-section image. We use a U-Net architecture to train on a data set of 62 pre-labelled images obtained from different labs, machines, and materials coupled with image augmentation. When neural network hyperparameters such as batch size and learning rate are properly tuned, the learned model shows an accuracy for classification of over 99% and an F1 score over 90%. The neural network exhibits robustness when tested on images captured by various users, printed on different machines, and acquired using different microscopes. A post-processing module extracts the height and width of the melt pool, and the wetting angles. We discuss opportunities to improve model performance and avenues for transfer learning, such as extension to other AM processes such as directed energy deposition.

Title: DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors

• Authors: Joseph Ortiz, Antoine Dedieu, Wolfgang Lehrach, Swaroop Guntupalli, Carter Wendelken, Ahmad Humayun, Guangyao Zhou, Sivaramakrishnan Swaminathan, Miguel Lázaro-Gredilla, Kevin Murphy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18330
• Pdf URL: https://arxiv.org/pdf/2409.18330
• Copy Paste: [[2409.18330]] DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors(https://arxiv.org/abs/2409.18330)
• Keywords: robust
• Abstract: Learning from previously collected data via behavioral cloning or offline reinforcement learning (RL) is a powerful recipe for scaling generalist agents by avoiding the need for expensive online learning. Despite strong generalization in some respects, agents are often remarkably brittle to minor visual variations in control-irrelevant factors such as the background or camera viewpoint. In this paper, we present theDeepMind Control Visual Benchmark (DMC-VB), a dataset collected in the DeepMind Control Suite to evaluate the robustness of offline RL agents for solving continuous control tasks from visual input in the presence of visual distractors. In contrast to prior works, our dataset (a) combines locomotion and navigation tasks of varying difficulties, (b) includes static and dynamic visual variations, (c) considers data generated by policies with different skill levels, (d) systematically returns pairs of state and pixel observation, (e) is an order of magnitude larger, and (f) includes tasks with hidden goals. Accompanying our dataset, we propose three benchmarks to evaluate representation learning methods for pretraining, and carry out experiments on several recently proposed methods. First, we find that pretrained representations do not help policy learning on DMC-VB, and we highlight a large representation gap between policies learned on pixel observations and on states. Second, we demonstrate when expert data is limited, policy learning can benefit from representations pretrained on (a) suboptimal data, and (b) tasks with stochastic hidden goals. Our dataset and benchmark code to train and evaluate agents are available at: this https URL.

Title: DeBaRA: Denoising-Based 3D Room Arrangement Generation

• Authors: Léopold Maillard, Nicolas Sereyjol-Garros, Tom Durand, Maks Ovsjanikov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18336
• Pdf URL: https://arxiv.org/pdf/2409.18336
• Copy Paste: [[2409.18336]] DeBaRA: Denoising-Based 3D Room Arrangement Generation(https://arxiv.org/abs/2409.18336)
• Keywords: diffusion, generative
• Abstract: Generating realistic and diverse layouts of furnished indoor 3D scenes unlocks multiple interactive applications impacting a wide range of industries. The inherent complexity of object interactions, the limited amount of available data and the requirement to fulfill spatial constraints all make generative modeling for 3D scene synthesis and arrangement challenging. Current methods address these challenges autoregressively or by using off-the-shelf diffusion objectives by simultaneously predicting all attributes without 3D reasoning considerations. In this paper, we introduce DeBaRA, a score-based model specifically tailored for precise, controllable and flexible arrangement generation in a bounded environment. We argue that the most critical component of a scene synthesis system is to accurately establish the size and position of various objects within a restricted area. Based on this insight, we propose a lightweight conditional score-based model designed with 3D spatial awareness at its core. We demonstrate that by focusing on spatial attributes of objects, a single trained DeBaRA model can be leveraged at test time to perform several downstream applications such as scene synthesis, completion and re-arrangement. Further, we introduce a novel Self Score Evaluation procedure so it can be optimally employed alongside external LLM models. We evaluate our approach through extensive experiments and demonstrate significant improvement upon state-of-the-art approaches in a range of scenarios.

Title: AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models

• Authors: Xin Hong, Yuan Gong, Vidhyasaharan Sethu, Ting Dang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18339
• Pdf URL: https://arxiv.org/pdf/2409.18339
• Copy Paste: [[2409.18339]] AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models(https://arxiv.org/abs/2409.18339)
• Keywords: large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have demonstrated great success in many Natural Language Processing (NLP) tasks. In addition to their cognitive intelligence, exploring their capabilities in emotional intelligence is also crucial, as it enables more natural and empathetic conversational AI. Recent studies have shown LLMs' capability in recognizing emotions, but they often focus on single emotion labels and overlook the complex and ambiguous nature of human emotions. This study is the first to address this gap by exploring the potential of LLMs in recognizing ambiguous emotions, leveraging their strong generalization capabilities and in-context learning. We design zero-shot and few-shot prompting and incorporate past dialogue as context information for ambiguous emotion recognition. Experiments conducted using three datasets indicate significant potential for LLMs in recognizing ambiguous emotions, and highlight the substantial benefits of including context information. Furthermore, our findings indicate that LLMs demonstrate a high degree of effectiveness in recognizing less ambiguous emotions and exhibit potential for identifying more ambiguous emotions, paralleling human perceptual capabilities.

Title: A Generalized LLM-Augmented BIM Framework: Application to a Speech-to-BIM system

• Authors: Ghang Lee, Suhyung Jang, Seokho Hyun
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.18345
• Pdf URL: https://arxiv.org/pdf/2409.18345
• Copy Paste: [[2409.18345]] A Generalized LLM-Augmented BIM Framework: Application to a Speech-to-BIM system(https://arxiv.org/abs/2409.18345)
• Keywords: large language model
• Abstract: Performing building information modeling (BIM) tasks is a complex process that imposes a steep learning curve and a heavy cognitive load due to the necessity of remembering sequences of numerous commands. With the rapid advancement of large language models (LLMs), it is foreseeable that BIM tasks, including querying and managing BIM data, 4D and 5D BIM, design compliance checking, or authoring a design, using written or spoken natural language (i.e., text-to-BIM or speech-to-BIM), will soon supplant traditional graphical user interfaces. This paper proposes a generalized LLM-augmented BIM framework to expedite the development of LLM-enhanced BIM applications by providing a step-by-step development process. The proposed framework consists of six steps: interpret-fill-match-structure-execute-check. The paper demonstrates the applicability of the proposed framework through implementing a speech-to-BIM application, NADIA-S (Natural-language-based Architectural Detailing through Interaction with Artificial Intelligence via Speech), using exterior wall detailing as an example.

Title: MultiClimate: Multimodal Stance Detection on Climate Change Videos

• Authors: Jiawen Wang, Longfei Zuo, Siyao Peng, Barbara Plank
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18346
• Pdf URL: https://arxiv.org/pdf/2409.18346
• Copy Paste: [[2409.18346]] MultiClimate: Multimodal Stance Detection on Climate Change Videos(https://arxiv.org/abs/2409.18346)
• Keywords: large language model
• Abstract: Climate change (CC) has attracted increasing attention in NLP in recent years. However, detecting the stance on CC in multimodal data is understudied and remains challenging due to a lack of reliable datasets. To improve the understanding of public opinions and communication strategies, this paper presents MultiClimate, the first open-source manually-annotated stance detection dataset with $100$ CC-related YouTube videos and $4,209$ frame-transcript pairs. We deploy state-of-the-art vision and language models, as well as multimodal models for MultiClimate stance detection. Results show that text-only BERT significantly outperforms image-only ResNet50 and ViT. Combining both modalities achieves state-of-the-art, $0.747$/$0.749$ in accuracy/F1. Our 100M-sized fusion models also beat CLIP and BLIP, as well as the much larger 9B-sized multimodal IDEFICS and text-only Llama3 and Gemma2, indicating that multimodal stance detection remains challenging for large language models. Our code, dataset, as well as supplementary materials, are available at this https URL.

Title: SinoSynth: A Physics-based Domain Randomization Approach for Generalizable CBCT Image Enhancement

• Authors: Yunkui Pang, Yilin Liu, Xu Chen, Pew-Thian Yap, Jun Lian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18355
• Pdf URL: https://arxiv.org/pdf/2409.18355
• Copy Paste: [[2409.18355]] SinoSynth: A Physics-based Domain Randomization Approach for Generalizable CBCT Image Enhancement(https://arxiv.org/abs/2409.18355)
• Keywords: generative
• Abstract: Cone Beam Computed Tomography (CBCT) finds diverse applications in medicine. Ensuring high image quality in CBCT scans is essential for accurate diagnosis and treatment delivery. Yet, the susceptibility of CBCT images to noise and artifacts undermines both their usefulness and reliability. Existing methods typically address CBCT artifacts through image-to-image translation approaches. These methods, however, are limited by the artifact types present in the training data, which may not cover the complete spectrum of CBCT degradations stemming from variations in imaging protocols. Gathering additional data to encompass all possible scenarios can often pose a challenge. To address this, we present SinoSynth, a physics-based degradation model that simulates various CBCT-specific artifacts to generate a diverse set of synthetic CBCT images from high-quality CT images without requiring pre-aligned data. Through extensive experiments, we demonstrate that several different generative networks trained on our synthesized data achieve remarkable results on heterogeneous multi-institutional datasets, outperforming even the same networks trained on actual data. We further show that our degradation model conveniently provides an avenue to enforce anatomical constraints in conditional generative models, yielding high-quality and structure-preserving synthetic CT images.

Title: FedDCL: a federated data collaboration learning as a hybrid-type privacy-preserving framework based on federated learning and data collaboration

• Authors: Akira Imakura, Tetsuya Sakurai
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18356
• Pdf URL: https://arxiv.org/pdf/2409.18356
• Copy Paste: [[2409.18356]] FedDCL: a federated data collaboration learning as a hybrid-type privacy-preserving framework based on federated learning and data collaboration(https://arxiv.org/abs/2409.18356)
• Keywords: privacy, federate
• Abstract: Recently, federated learning has attracted much attention as a privacy-preserving integrated analysis that enables integrated analysis of data held by multiple institutions without sharing raw data. On the other hand, federated learning requires iterative communication across institutions and has a big challenge for implementation in situations where continuous communication with the outside world is extremely difficult. In this study, we propose a federated data collaboration learning (FedDCL), which solves such communication issues by combining federated learning with recently proposed non-model share-type federated learning named as data collaboration analysis. In the proposed FedDCL framework, each user institution independently constructs dimensionality-reduced intermediate representations and shares them with neighboring institutions on intra-group DC servers. On each intra-group DC server, intermediate representations are transformed to incorporable forms called collaboration representations. Federated learning is then conducted between intra-group DC servers. The proposed FedDCL framework does not require iterative communication by user institutions and can be implemented in situations where continuous communication with the outside world is extremely difficult. The experimental results show that the performance of the proposed FedDCL is comparable to that of existing federated learning.

Title: Generative AI for fast and accurate Statistical Computation of Fluids

• Authors: Roberto Molinaro, Samuel Lanthaler, Bogdan Raonić, Tobias Rohner, Victor Armegioiu, Zhong Yi Wan, Fei Sha, Siddhartha Mishra, Leonardo Zepeda-Núñez
• Subjects: cs.LG, math.NA, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2409.18359
• Pdf URL: https://arxiv.org/pdf/2409.18359
• Copy Paste: [[2409.18359]] Generative AI for fast and accurate Statistical Computation of Fluids(https://arxiv.org/abs/2409.18359)
• Keywords: robust, diffusion, generative
• Abstract: We present a generative AI algorithm for addressing the challenging task of fast, accurate and robust statistical computation of three-dimensional turbulent fluid flows. Our algorithm, termed as GenCFD, is based on a conditional score-based diffusion model. Through extensive numerical experimentation with both incompressible and compressible fluid flows, we demonstrate that GenCFD provides very accurate approximation of statistical quantities of interest such as mean, variance, point pdfs, higher-order moments, while also generating high quality realistic samples of turbulent fluid flows and ensuring excellent spectral resolution. In contrast, ensembles of operator learning baselines which are trained to minimize mean (absolute) square errors regress to the mean flow. We present rigorous theoretical results uncovering the surprising mechanisms through which diffusion models accurately generate fluid flows. These mechanisms are illustrated with solvable toy models that exhibit the relevant features of turbulent fluid flows while being amenable to explicit analytical formulas.

Title: Architecture for Protecting Data Privacy in Decentralized Social Networks

• Authors: Quang Cao, Katerina Vgena, Aikaterini-Georgia Mavroeidi, Christos Kalloniatis, Xun Yi, Son Hoang Dau
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18360
• Pdf URL: https://arxiv.org/pdf/2409.18360
• Copy Paste: [[2409.18360]] Architecture for Protecting Data Privacy in Decentralized Social Networks(https://arxiv.org/abs/2409.18360)
• Keywords: privacy, protect
• Abstract: Centralized social networks have experienced a transformative impact on our digital era communication, connection, and information-sharing information. However, it has also raised significant concerns regarding users' privacy and individual rights. In response to these concerns, this paper proposes a novel Decentralized Social Network employing Blockchain technology and Decentralized Storage Networks completed by Access Control Smart Contracts. The initial phase comprises a comprehensive literature review, delving into decentralized social networks, explaining the review methodology, and presenting the resulting findings. Building upon these findings and an analysis of previous research gaps, we propose a novel architecture for decentralized social networks. In conclusion, the principal results highlight the benefit of our decentralized social network to protect user privacy. Moreover, the users have all rights to their posted information following the General Data Protection Regulation (GDPR).

Title: Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images

• Authors: Donghwan Kim, Tae-Kyun Kim
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18364
• Pdf URL: https://arxiv.org/pdf/2409.18364
• Copy Paste: [[2409.18364]] Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images(https://arxiv.org/abs/2409.18364)
• Keywords: diffusion
• Abstract: 3D human shape reconstruction under severe occlusion due to human-object or human-human interaction is a challenging problem. Parametric models i.e., SMPL(-X), which are based on the statistics across human shapes, can represent whole human body shapes but are limited to minimally-clothed human shapes. Implicit-function-based methods extract features from the parametric models to employ prior knowledge of human bodies and can capture geometric details such as clothing and hair. However, they often struggle to handle misaligned parametric models and inpaint occluded regions given a single RGB image. In this work, we propose a novel pipeline, MHCDIFF, Multi-hypotheses Conditioned Point Cloud Diffusion, composed of point cloud diffusion conditioned on probabilistic distributions for pixel-aligned detailed 3D human reconstruction under occlusion. Compared to previous implicit-function-based methods, the point cloud diffusion model can capture the global consistent features to generate the occluded regions, and the denoising process corrects the misaligned SMPL meshes. The core of MHCDIFF is extracting local features from multiple hypothesized SMPL(-X) meshes and aggregating the set of features to condition the diffusion model. In the experiments on CAPE and MultiHuman datasets, the proposed method outperforms various SOTA methods based on SMPL, implicit functions, point cloud diffusion, and their combined, under synthetic and real occlusions.

Title: Discovery and inversion of the viscoelastic wave equation in inhomogeneous media

• Authors: Su Chen, Yi Ding, Hiroe Miyake, Xiaojun Li
• Subjects: cs.LG, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2409.18370
• Pdf URL: https://arxiv.org/pdf/2409.18370
• Copy Paste: [[2409.18370]] Discovery and inversion of the viscoelastic wave equation in inhomogeneous media(https://arxiv.org/abs/2409.18370)
• Keywords: robust
• Abstract: In scientific machine learning, the task of identifying partial differential equations accurately from sparse and noisy data poses a significant challenge. Current sparse regression methods may identify inaccurate equations on sparse and noisy datasets and are not suitable for varying coefficients. To address this issue, we propose a hybrid framework that combines two alternating direction optimization phases: discovery and embedding. The discovery phase employs current well-developed sparse regression techniques to preliminarily identify governing equations from observations. The embedding phase implements a recurrent convolutional neural network (RCNN), enabling efficient processes for time-space iterations involved in discretized forms of wave equation. The RCNN model further optimizes the imperfect sparse regression results to obtain more accurate functional terms and coefficients. Through alternating update of discovery-embedding phases, essential physical equations can be robustly identified from noisy and low-resolution measurements. To assess the performance of proposed framework, numerical experiments are conducted on various scenarios involving wave equation in elastic/viscoelastic and homogeneous/inhomogeneous media. The results demonstrate that the proposed method exhibits excellent robustness and accuracy, even when faced with high levels of noise and limited data availability in both spatial and temporal domains.

Title: You Only Speak Once to See

• Authors: Wenhao Yang, Jianguo Wei, Wenhuan Lu, Lei Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18372
• Pdf URL: https://arxiv.org/pdf/2409.18372
• Copy Paste: [[2409.18372]] You Only Speak Once to See(https://arxiv.org/abs/2409.18372)
• Keywords: robust
• Abstract: Grounding objects in images using visual cues is a well-established approach in computer vision, yet the potential of audio as a modality for object recognition and grounding remains underexplored. We introduce YOSS, "You Only Speak Once to See," to leverage audio for grounding objects in visual scenes, termed Audio Grounding. By integrating pre-trained audio models with visual models using contrastive learning and multi-modal alignment, our approach captures speech commands or descriptions and maps them directly to corresponding objects within images. Experimental results indicate that audio guidance can be effectively applied to object grounding, suggesting that incorporating audio guidance may enhance the precision and robustness of current object grounding methods and improve the performance of robotic systems and computer vision applications. This finding opens new possibilities for advanced object recognition, scene understanding, and the development of more intuitive and capable robotic systems.

Title: Code Vulnerability Repair with Large Language Model using Context-Aware Prompt Tuning

• Authors: Arshiya Khan, Guannan Liu, Xing Gao
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18395
• Pdf URL: https://arxiv.org/pdf/2409.18395
• Copy Paste: [[2409.18395]] Code Vulnerability Repair with Large Language Model using Context-Aware Prompt Tuning(https://arxiv.org/abs/2409.18395)
• Keywords: security, large language model
• Abstract: Large Language Models (LLMs) have shown significant challenges in detecting and repairing vulnerable code, particularly when dealing with vulnerabilities involving multiple aspects, such as variables, code flows, and code structures. In this study, we utilize GitHub Copilot as the LLM and focus on buffer overflow vulnerabilities. Our experiments reveal a notable gap in Copilot's abilities when dealing with buffer overflow vulnerabilities, with a 76% vulnerability detection rate but only a 15% vulnerability repair rate. To address this issue, we propose context-aware prompt tuning techniques designed to enhance LLM performance in repairing buffer overflow. By injecting a sequence of domain knowledge about the vulnerability, including various security and code contexts, we demonstrate that Copilot's successful repair rate increases to 63%, representing more than four times the improvement compared to repairs without domain knowledge.

Title: GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation

• Authors: Jiawei Lu, Yingpeng Zhang, Zengjun Zhao, He Wang, Kun Zhou, Tianjia Shao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18401
• Pdf URL: https://arxiv.org/pdf/2409.18401
• Copy Paste: [[2409.18401]] GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation(https://arxiv.org/abs/2409.18401)
• Keywords: diffusion
• Abstract: Large-scale text-guided image diffusion models have shown astonishing results in text-to-image (T2I) generation. However, applying these models to synthesize textures for 3D geometries remains challenging due to the domain gap between 2D images and textures on a 3D surface. Early works that used a projecting-and-inpainting approach managed to preserve generation diversity but often resulted in noticeable artifacts and style inconsistencies. While recent methods have attempted to address these inconsistencies, they often introduce other issues, such as blurring, over-saturation, or over-smoothing. To overcome these challenges, we propose a novel text-to-texture synthesis framework that leverages pretrained diffusion models. We first introduce a local attention reweighing mechanism in the self-attention layers to guide the model in concentrating on spatial-correlated patches across different views, thereby enhancing local details while preserving cross-view consistency. Additionally, we propose a novel latent space merge pipeline, which further ensures consistency across different viewpoints without sacrificing too much diversity. Our method significantly outperforms existing state-of-the-art techniques regarding texture consistency and visual quality, while delivering results much faster than distillation-based methods. Importantly, our framework does not require additional training or fine-tuning, making it highly adaptable to a wide range of models available on public platforms.

Title: SpecCFA: Enhancing Control Flow Attestation/Auditing via Application-Aware Sub-Path Speculation

• Authors: Adam Caulfield, Liam Tyler, Ivan De Oliveira Nunes
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18403
• Pdf URL: https://arxiv.org/pdf/2409.18403
• Copy Paste: [[2409.18403]] SpecCFA: Enhancing Control Flow Attestation/Auditing via Application-Aware Sub-Path Speculation(https://arxiv.org/abs/2409.18403)
• Keywords: secure, security
• Abstract: At the edge of modern cyber-physical systems, Micro-Controller Units (MCUs) are responsible for safety-critical sensing/actuation. However, MCU cost constraints rule out the usual security mechanisms of general-purpose computers. Thus, various low-cost security architectures have been proposed to remotely verify MCU software integrity. Control Flow Attestation (CFA) enables a Verifier (Vrf) to remotely assess the run-time behavior of a prover MCU (Prv), generating an authenticated trace of all of Prv control flow transfers (CFLog). Further, Control Flow Auditing architectures augment CFA by guaranteeing the delivery of evidence to Vrf. Unfortunately, a limitation of existing CFA lies in the cost to store and transmit CFLog, as even simple MCU software may generate large traces. Given these issues, prior work has proposed static (context-insensitive) optimizations. However, they do not support configurable program-specific optimizations. In this work, we note that programs may produce unique predictable control flow sub-paths and argue that program-specific predictability can be leveraged to dynamically optimize CFA while retaining all security guarantees. Therefore, we propose SpecCFA: an approach for dynamic sub-path speculation in CFA. SpecCFA allows Vrf to securely speculate on likely control flow sub-paths for each attested program. At run-time, when a sub-path in CFLog matches a pre-defined speculation, the entire sub-path is replaced by a reserved symbol. SpecCFA can speculate on multiple variable-length control flow sub-paths simultaneously. We implement SpecCFA atop two open-source control flow auditing architectures: one based on a custom hardware design and one based on a commodity Trusted Execution Environment (ARM TrustZone-M). In both cases, SpecCFA significantly lowers storage/performance costs that are critical to resource-constrained MCUs.

Title: SciDFM: A Large Language Model with Mixture-of-Experts for Science

• Authors: Liangtai Sun, Danyu Luo, Da Ma, Zihan Zhao, Baocai Chen, Zhennan Shen, Su Zhu, Lu Chen, Xin Chen, Kai Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18412
• Pdf URL: https://arxiv.org/pdf/2409.18412
• Copy Paste: [[2409.18412]] SciDFM: A Large Language Model with Mixture-of-Experts for Science(https://arxiv.org/abs/2409.18412)
• Keywords: large language model
• Abstract: Recently, there has been a significant upsurge of interest in leveraging large language models (LLMs) to assist scientific discovery. However, most LLMs only focus on general science, while they lack domain-specific knowledge, such as chemical molecules and amino acid sequences. To bridge these gaps, we introduce SciDFM, a mixture-of-experts LLM, which is trained from scratch and is able to conduct college-level scientific reasoning and understand molecules and amino acid sequences. We collect a large-scale training corpus containing numerous scientific papers and books from different disciplines as well as data from domain-specific databases. We further fine-tune the pre-trained model on lots of instruction data to improve performances on downstream benchmarks. From experiment results, we show that SciDFM achieves strong performance on general scientific benchmarks such as SciEval and SciQ, and it reaches a SOTA performance on domain-specific benchmarks among models of similar size. We further analyze the expert layers and show that the results of expert selection vary with data from different disciplines. To benefit the broader research community, we open-source SciDFM at this https URL.

Title: VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback

• Authors: Guoxi Zhang, Jiuding Duan
• Subjects: cs.LG, cs.AI, cs.CL, cs.GT, econ.GN
• Abstract URL: https://arxiv.org/abs/2409.18417
• Pdf URL: https://arxiv.org/pdf/2409.18417
• Copy Paste: [[2409.18417]] VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback(https://arxiv.org/abs/2409.18417)
• Keywords: large language model
• Abstract: This paper addresses the cost-efficiency aspect of Reinforcement Learning from Human Feedback (RLHF). RLHF leverages datasets of human preferences over outputs of large language models (LLM) to instill human expectations into LLMs. While preference annotation comes with a monetized cost, the economic utility of a preference dataset has not been considered by far. What exacerbates this situation is that given complex intransitive or cyclic relationships in preference datasets, existing algorithms for fine-tuning LLMs are still far from capturing comprehensive preferences. This raises severe cost-efficiency concerns in production environments, where preference data accumulate over time. In this paper, we see the fine-tuning of LLMs as a monetized economy and introduce an auction mechanism to improve the efficiency of the preference data collection in dollar terms. We show that introducing an auction mechanism can play an essential role in enhancing the cost-efficiency of RLHF while maintaining satisfactory model performance. Experimental results demonstrate that our proposed auction-based protocol is cost-efficient for fine-tuning LLMs by concentrating on high-quality feedback.

Title: A3: Active Adversarial Alignment for Source-Free Domain Adaptation

• Authors: Chrisantus Eze, Christopher Crick
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18418
• Pdf URL: https://arxiv.org/pdf/2409.18418
• Copy Paste: [[2409.18418]] A3: Active Adversarial Alignment for Source-Free Domain Adaptation(https://arxiv.org/abs/2409.18418)
• Keywords: robust
• Abstract: Unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Recent works have focused on source-free UDA, where only target data is available. This is challenging as models rely on noisy pseudo-labels and struggle with distribution shifts. We propose Active Adversarial Alignment (A3), a novel framework combining self-supervised learning, adversarial training, and active learning for robust source-free UDA. A3 actively samples informative and diverse data using an acquisition function for training. It adapts models via adversarial losses and consistency regularization, aligning distributions without source data access. A3 advances source-free UDA through its synergistic integration of active and adversarial learning for effective domain alignment and noise reduction.

Title: Robust Network Learning via Inverse Scale Variational Sparsification

• Authors: Zhiling Zhou, Zirui Liu, Chengming Xu, Yanwei Fu, Xinwei Sun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18419
• Pdf URL: https://arxiv.org/pdf/2409.18419
• Copy Paste: [[2409.18419]] Robust Network Learning via Inverse Scale Variational Sparsification(https://arxiv.org/abs/2409.18419)
• Keywords: robust
• Abstract: While neural networks have made significant strides in many AI tasks, they remain vulnerable to a range of noise types, including natural corruptions, adversarial noise, and low-resolution artifacts. Many existing approaches focus on enhancing robustness against specific noise types, limiting their adaptability to others. Previous studies have addressed general robustness by adopting a spectral perspective, which tends to blur crucial features like texture and object contours. Our proposed solution, however, introduces an inverse scale variational sparsification framework within a time-continuous inverse scale space formulation. This framework progressively learns finer-scale features by discerning variational differences between pixels, ultimately preserving only large-scale features in the smoothed image. Unlike frequency-based methods, our approach not only removes noise by smoothing small-scale features where corruptions often occur but also retains high-contrast details such as textures and object contours. Moreover, our framework offers simplicity and efficiency in implementation. By integrating this algorithm into neural network training, we guide the model to prioritize learning large-scale features. We show the efficacy of our approach through enhanced robustness against various noise types.

Title: A physics-driven sensor placement optimization methodology for temperature field reconstruction

• Authors: Xu Liu, Wen Yao, Wei Peng, Zhuojia Fu, Zixue Xiang, Xiaoqian Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18423
• Pdf URL: https://arxiv.org/pdf/2409.18423
• Copy Paste: [[2409.18423]] A physics-driven sensor placement optimization methodology for temperature field reconstruction(https://arxiv.org/abs/2409.18423)
• Keywords: data-free
• Abstract: Perceiving the global field from sparse sensors has been a grand challenge in the monitoring, analysis, and design of physical systems. In this context, sensor placement optimization is a crucial issue. Most existing works require large and sufficient data to construct data-based criteria, which are intractable in data-free scenarios without numerical and experimental data. To this end, we propose a novel physics-driven sensor placement optimization (PSPO) method for temperature field reconstruction using a physics-based criterion to optimize sensor locations. In our methodological framework, we firstly derive the theoretical upper and lower bounds of the reconstruction error under noise scenarios by analyzing the optimal solution, proving that error bounds correlate with the condition number determined by sensor locations. Furthermore, the condition number, as the physics-based criterion, is used to optimize sensor locations by the genetic algorithm. Finally, the best sensors are validated by reconstruction models, including non-invasive end-to-end models, non-invasive reduced-order models, and physics-informed models. Experimental results, both on a numerical and an application case, demonstrate that the PSPO method significantly outperforms random and uniform selection methods, improving the reconstruction accuracy by nearly an order of magnitude. Moreover, the PSPO method can achieve comparable reconstruction accuracy to the existing data-driven placement optimization methods.

Title: Neural Collaborative Filtering to Detect Anomalies in Human Semantic Trajectories

• Authors: Yueyang Liu, Lance Kennedy, Hossein Amiri, Andreas Züfle
• Subjects: cs.LG, cs.IR, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.18427
• Pdf URL: https://arxiv.org/pdf/2409.18427
• Copy Paste: [[2409.18427]] Neural Collaborative Filtering to Detect Anomalies in Human Semantic Trajectories(https://arxiv.org/abs/2409.18427)
• Keywords: security, robust
• Abstract: Human trajectory anomaly detection has become increasingly important across a wide range of applications, including security surveillance and public health. However, existing trajectory anomaly detection methods are primarily focused on vehicle-level traffic, while human-level trajectory anomaly detection remains under-explored. Since human trajectory data is often very sparse, machine learning methods have become the preferred approach for identifying complex patterns. However, concerns regarding potential biases and the robustness of these models have intensified the demand for more transparent and explainable alternatives. In response to these challenges, our research focuses on developing a lightweight anomaly detection model specifically designed to detect anomalies in human trajectories. We propose a Neural Collaborative Filtering approach to model and predict normal mobility. Our method is designed to model users' daily patterns of life without requiring prior knowledge, thereby enhancing performance in scenarios where data is sparse or incomplete, such as in cold start situations. Our algorithm consists of two main modules. The first is the collaborative filtering module, which applies collaborative filtering to model normal mobility of individual humans to places of interest. The second is the neural module, responsible for interpreting the complex spatio-temporal relationships inherent in human trajectory data. To validate our approach, we conducted extensive experiments using simulated and real-world datasets comparing to numerous state-of-the-art trajectory anomaly detection approaches.

Title: Search3D: Hierarchical Open-Vocabulary 3D Segmentation

• Authors: Ayca Takmaz, Alexandros Delitzas, Robert W. Sumner, Francis Engelmann, Johanna Wald, Federico Tombari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18431
• Pdf URL: https://arxiv.org/pdf/2409.18431
• Copy Paste: [[2409.18431]] Search3D: Hierarchical Open-Vocabulary 3D Segmentation(https://arxiv.org/abs/2409.18431)
• Keywords: segmentation
• Abstract: Open-vocabulary 3D segmentation enables the exploration of 3D spaces using free-form text descriptions. Existing methods for open-vocabulary 3D instance segmentation primarily focus on identifying object-level instances in a scene. However, they face challenges when it comes to understanding more fine-grained scene entities such as object parts, or regions described by generic attributes. In this work, we introduce Search3D, an approach that builds a hierarchical open-vocabulary 3D scene representation, enabling the search for entities at varying levels of granularity: fine-grained object parts, entire objects, or regions described by attributes like materials. Our method aims to expand the capabilities of open vocabulary instance-level 3D segmentation by shifting towards a more flexible open-vocabulary 3D search setting less anchored to explicit object-centric queries, compared to prior work. To ensure a systematic evaluation, we also contribute a scene-scale open-vocabulary 3D part segmentation benchmark based on MultiScan, along with a set of open-vocabulary fine-grained part annotations on ScanNet++. We verify the effectiveness of Search3D across several tasks, demonstrating that our approach outperforms baselines in scene-scale open-vocabulary 3D part segmentation, while maintaining strong performance in segmenting 3D objects and materials.

Title: Gradient-free Decoder Inversion in Latent Diffusion Models

• Authors: Seongmin Hong, Suh Yoon Jeon, Kyeonghyun Lee, Ernest K. Ryu, Se Young Chun
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18442
• Pdf URL: https://arxiv.org/pdf/2409.18442
• Copy Paste: [[2409.18442]] Gradient-free Decoder Inversion in Latent Diffusion Models(https://arxiv.org/abs/2409.18442)
• Keywords: watermark, diffusion, generative
• Abstract: In latent diffusion models (LDMs), denoising diffusion process efficiently takes place on latent space whose dimension is lower than that of pixel space. Decoder is typically used to transform the representation in latent space to that in pixel space. While a decoder is assumed to have an encoder as an accurate inverse, exact encoder-decoder pair rarely exists in practice even though applications often require precise inversion of decoder. Prior works for decoder inversion in LDMs employed gradient descent inspired by inversions of generative adversarial networks. However, gradient-based methods require larger GPU memory and longer computation time for larger latent space. For example, recent video LDMs can generate more than 16 frames, but GPUs with 24 GB memory can only perform gradient-based decoder inversion for 4 frames. Here, we propose an efficient gradient-free decoder inversion for LDMs, which can be applied to diverse latent models. Theoretical convergence property of our proposed inversion has been investigated not only for the forward step method, but also for the inertial Krasnoselskii-Mann (KM) iterations under mild assumption on cocoercivity that is satisfied by recent LDMs. Our proposed gradient-free method with Adam optimizer and learning rate scheduling significantly reduced computation time and memory usage over prior gradient-based methods and enabled efficient computation in applications such as noise-space watermarking while achieving comparable error levels.

Title: Exploring Language Model Generalization in Low-Resource Extractive QA

• Authors: Saptarshi Sengupta, Wenpeng Yin, Preslav Nakov, Shreya Ghosh, Suhang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18446
• Pdf URL: https://arxiv.org/pdf/2409.18446
• Copy Paste: [[2409.18446]] Exploring Language Model Generalization in Low-Resource Extractive QA(https://arxiv.org/abs/2409.18446)
• Keywords: large language model
• Abstract: In this paper, we investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift, i.e., can LLMs generalize well to closed-domains that require specific knowledge such as medicine and law in a zero-shot fashion without additional in-domain training? To this end, we devise a series of experiments to empirically explain the performance gap. Our findings suggest that: a) LLMs struggle with dataset demands of closed-domains such as retrieving long answer-spans; b) Certain LLMs, despite showing strong overall performance, display weaknesses in meeting basic requirements as discriminating between domain-specific senses of words which we link to pre-processing decisions; c) Scaling model parameters is not always effective for cross-domain generalization; and d) Closed-domain datasets are quantitatively much different than open-domain EQA datasets and current LLMs struggle to deal with them. Our findings point out important directions for improving existing LLMs.

Title: Hierarchical Federated Learning with Multi-Timescale Gradient Correction

• Authors: Wenzhi Fang, Dong-Jun Han, Evan Chen, Shiqiang Wang, Christopher G. Brinton
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18448
• Pdf URL: https://arxiv.org/pdf/2409.18448
• Copy Paste: [[2409.18448]] Hierarchical Federated Learning with Multi-Timescale Gradient Correction(https://arxiv.org/abs/2409.18448)
• Keywords: federate
• Abstract: While traditional federated learning (FL) typically focuses on a star topology where clients are directly connected to a central server, real-world distributed systems often exhibit hierarchical architectures. Hierarchical FL (HFL) has emerged as a promising solution to bridge this gap, leveraging aggregation points at multiple levels of the system. However, existing algorithms for HFL encounter challenges in dealing with multi-timescale model drift, i.e., model drift occurring across hierarchical levels of data heterogeneity. In this paper, we propose a multi-timescale gradient correction (MTGC) methodology to resolve this issue. Our key idea is to introduce distinct control variables to (i) correct the client gradient towards the group gradient, i.e., to reduce client model drift caused by local updates based on individual datasets, and (ii) correct the group gradient towards the global gradient, i.e., to reduce group model drift caused by FL over clients within the group. We analytically characterize the convergence behavior of MTGC under general non-convex settings, overcoming challenges associated with couplings between correction terms. We show that our convergence bound is immune to the extent of data heterogeneity, confirming the stability of the proposed algorithm against multi-level non-i.i.d. data. Through extensive experiments on various datasets and models, we validate the effectiveness of MTGC in diverse HFL settings. The code for this project is available at \href{this https URL}{this https URL}.

Title: Towards Personal Data Sharing Autonomy:A Task-driven Data Capsule Sharing System

• Authors: Qiuyun Lyu, Yilong Zhou, Yizhi Ren, Zheng Wang, Yunchuan Guo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18449
• Pdf URL: https://arxiv.org/pdf/2409.18449
• Copy Paste: [[2409.18449]] Towards Personal Data Sharing Autonomy:A Task-driven Data Capsule Sharing System(https://arxiv.org/abs/2409.18449)
• Keywords: secure, security, privacy, protect, attack
• Abstract: Personal data custodian services enable data owners to share their data with data consumers in a convenient manner, anytime and anywhere. However, with data hosted in these services being beyond the control of the data owners, it raises significant concerns about privacy in personal data sharing. Many schemes have been proposed to realize fine-grained access control and privacy protection in data sharing. However, they fail to protect the rights of data owners to their data under the law, since their designs focus on the management of system administrators rather than enhancing the data owners' privacy. In this paper, we introduce a novel task-driven personal data sharing system based on the data capsule paradigm realizing personal data sharing autonomy. It enables data owners in our system to fully control their data, and share it autonomously. Specifically, we present a tamper-resistant data capsule encapsulation method, where the data capsule is the minimal unit for independent and secure personal data storage and sharing. Additionally, to realize selective sharing and informed-consent based authorization, we propose a task-driven data sharing mechanism that is resistant to collusion and EDoS attacks. Furthermore, by updating parts of the data capsules, the permissions granted to data consumers can be immediately revoked. Finally, we conduct a security and performance analysis, proving that our scheme is correct, sound, and secure, as well as revealing more advantageous features in practicality, compared with the state-of-the-art schemes.

Title: Leveraging Long-Context Large Language Models for Multi-Document Understanding and Summarization in Enterprise Applications

• Authors: Aditi Godbole, Jabin Geevarghese George, Smita Shandilya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18454
• Pdf URL: https://arxiv.org/pdf/2409.18454
• Copy Paste: [[2409.18454]] Leveraging Long-Context Large Language Models for Multi-Document Understanding and Summarization in Enterprise Applications(https://arxiv.org/abs/2409.18454)
• Keywords: large language model
• Abstract: The rapid increase in unstructured data across various fields has made multi-document comprehension and summarization a critical task. Traditional approaches often fail to capture relevant context, maintain logical consistency, and extract essential information from lengthy documents. This paper explores the use of Long-context Large Language Models (LLMs) for multi-document summarization, demonstrating their exceptional capacity to grasp extensive connections, provide cohesive summaries, and adapt to various industry domains and integration with enterprise applications/systems. The paper discusses the workflow of multi-document summarization for effectively deploying long-context LLMs, supported by case studies in legal applications, enterprise functions such as HR, finance, and sourcing, as well as in the medical and news domains. These case studies show notable enhancements in both efficiency and accuracy. Technical obstacles, such as dataset diversity, model scalability, and ethical considerations like bias mitigation and factual accuracy, are carefully analyzed. Prospective research avenues are suggested to augment the functionalities and applications of long-context LLMs, establishing them as pivotal tools for transforming information processing across diverse sectors and enterprise applications.

Title: Review of Digital Asset Development with Graph Neural Network Unlearning

• Authors: Zara Lisbon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18455
• Pdf URL: https://arxiv.org/pdf/2409.18455
• Copy Paste: [[2409.18455]] Review of Digital Asset Development with Graph Neural Network Unlearning(https://arxiv.org/abs/2409.18455)
• Keywords: secure, privacy, robust
• Abstract: In the rapidly evolving landscape of digital assets, the imperative for robust data privacy and compliance with regulatory frameworks has intensified. This paper investigates the critical role of Graph Neural Networks (GNNs) in the management of digital assets and introduces innovative unlearning techniques specifically tailored to GNN architectures. We categorize unlearning strategies into two primary classes: data-driven approximation, which manipulates the graph structure to isolate and remove the influence of specific nodes, and model-driven approximation, which modifies the internal parameters and architecture of the GNN itself. By examining recent advancements in these unlearning methodologies, we highlight their applicability in various use cases, including fraud detection, risk assessment, token relationship prediction, and decentralized governance. We discuss the challenges inherent in balancing model performance with the requirements for data unlearning, particularly in the context of real-time financial applications. Furthermore, we propose a hybrid approach that combines the strengths of both unlearning strategies to enhance the efficiency and effectiveness of GNNs in digital asset ecosystems. Ultimately, this paper aims to provide a comprehensive framework for understanding and implementing GNN unlearning techniques, paving the way for secure and compliant deployment of machine learning in the digital asset domain.

Title: FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation

• Authors: Yuki Imajuku, Yoko Yamakata, Kiyoharu Aizawa
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.18459
• Pdf URL: https://arxiv.org/pdf/2409.18459
• Copy Paste: [[2409.18459]] FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation(https://arxiv.org/abs/2409.18459)
• Keywords: large language model
• Abstract: Research on food image understanding using recipe data has been a long-standing focus due to the diversity and complexity of the data. Moreover, food is inextricably linked to people's lives, making it a vital research area for practical applications such as dietary management. Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities, not only in their vast knowledge but also in their ability to handle languages naturally. While English is predominantly used, they can also support multiple languages including Japanese. This suggests that MLLMs are expected to significantly improve performance in food image understanding tasks. We fine-tuned open MLLMs LLaVA-1.5 and Phi-3 Vision on a Japanese recipe dataset and benchmarked their performance against the closed model GPT-4o. We then evaluated the content of generated recipes, including ingredients and cooking procedures, using 5,000 evaluation samples that comprehensively cover Japanese food culture. Our evaluation demonstrates that the open models trained on recipe data outperform GPT-4o, the current state-of-the-art model, in ingredient generation. Our model achieved F1 score of 0.531, surpassing GPT-4o's F1 score of 0.481, indicating a higher level of accuracy. Furthermore, our model exhibited comparable performance to GPT-4o in generating cooking procedure text.

Title: Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration

• Authors: Mahdi Morafah, Vyacheslav Kungurtsev, Hojin Chang, Chen Chen, Bill Lin
• Subjects: cs.LG, cs.AI, cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.18461
• Pdf URL: https://arxiv.org/pdf/2409.18461
• Copy Paste: [[2409.18461]] Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration(https://arxiv.org/abs/2409.18461)
• Keywords: privacy, federate
• Abstract: Federated Learning has emerged as a promising paradigm for collaborative machine learning, while preserving user data privacy. Despite its potential, standard FL lacks support for diverse heterogeneous device prototypes, which vary significantly in model and dataset sizes -- from small IoT devices to large workstations. This limitation is only partially addressed by existing knowledge distillation techniques, which often fail to transfer knowledge effectively across a broad spectrum of device prototypes with varied capabilities. This failure primarily stems from two issues: the dilution of informative logits from more capable devices by those from less capable ones, and the use of a single integrated logits as the distillation target across all devices, which neglects their individual learning capacities and and the unique contributions of each. To address these challenges, we introduce TAKFL, a novel KD-based framework that treats the knowledge transfer from each device prototype's ensemble as a separate task, independently distilling each to preserve its unique contributions and avoid dilution. TAKFL also incorporates a KD-based self-regularization technique to mitigate the issues related to the noisy and unsupervised ensemble distillation process. To integrate the separately distilled knowledge, we introduce an adaptive task arithmetic knowledge integration process, allowing each student model to customize the knowledge integration for optimal performance. Additionally, we present theoretical results demonstrating the effectiveness of task arithmetic in transferring knowledge across heterogeneous devices with varying capacities. Comprehensive evaluations of our method across both CV and NLP tasks demonstrate that TAKFL achieves SOTA results in a variety of datasets and settings, significantly outperforming existing KD-based methods. Code is released at this https URL

Title: A TextGCN-Based Decoding Approach for Improving Remote Sensing Image Captioning

• Authors: Swadhin Das, Raksha Sharma
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18467
• Pdf URL: https://arxiv.org/pdf/2409.18467
• Copy Paste: [[2409.18467]] A TextGCN-Based Decoding Approach for Improving Remote Sensing Image Captioning(https://arxiv.org/abs/2409.18467)
• Keywords: security, fair
• Abstract: Remote sensing images are highly valued for their ability to address complex real-world issues such as risk management, security, and meteorology. However, manually captioning these images is challenging and requires specialized knowledge across various domains. This letter presents an approach for automatically describing (captioning) remote sensing images. We propose a novel encoder-decoder setup that deploys a Text Graph Convolutional Network (TextGCN) and multi-layer LSTMs. The embeddings generated by TextGCN enhance the decoder's understanding by capturing the semantic relationships among words at both the sentence and corpus levels. Furthermore, we advance our approach with a comparison-based beam search method to ensure fairness in the search strategy for generating the final caption. We present an extensive evaluation of our approach against various other state-of-the-art encoder-decoder frameworks. We evaluated our method across three datasets using seven metrics: BLEU-1 to BLEU-4, METEOR, ROUGE-L, and CIDEr. The results demonstrate that our approach significantly outperforms other state-of-the-art encoder-decoder methods.

Title: Fairness without Sensitive Attributes via Knowledge Sharing

• Authors: Hongliang Ni, Lei Han, Tong Chen, Shazia Sadiq, Gianluca Demartini
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18470
• Pdf URL: https://arxiv.org/pdf/2409.18470
• Copy Paste: [[2409.18470]] Fairness without Sensitive Attributes via Knowledge Sharing(https://arxiv.org/abs/2409.18470)
• Keywords: privacy, fair
• Abstract: While model fairness improvement has been explored previously, existing methods invariably rely on adjusting explicit sensitive attribute values in order to improve model fairness in downstream tasks. However, we observe a trend in which sensitive demographic information becomes inaccessible as public concerns around data privacy grow. In this paper, we propose a confidence-based hierarchical classifier structure called "Reckoner" for reliable fair model learning under the assumption of missing sensitive attributes. We first present results showing that if the dataset contains biased labels or other hidden biases, classifiers significantly increase the bias gap across different demographic groups in the subset with higher prediction confidence. Inspired by these findings, we devised a dual-model system in which a version of the model initialised with a high-confidence data subset learns from a version of the model initialised with a low-confidence data subset, enabling it to avoid biased predictions. Our experimental results show that Reckoner consistently outperforms state-of-the-art baselines in COMPAS dataset and New Adult dataset, considering both accuracy and fairness metrics.

Title: URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Base

• Authors: Aditya Khan, Mason Shipton, David Anugraha, Kaiyao Duan, Phuong H. Hoang, Eric Khiu, A. Seza Doğruöz, En-Shiun Annie Lee
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18472
• Pdf URL: https://arxiv.org/pdf/2409.18472
• Copy Paste: [[2409.18472]] URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Base(https://arxiv.org/abs/2409.18472)
• Keywords: robust
• Abstract: URIEL is a knowledge base offering geographical, phylogenetic, and typological vector representations for 7970 languages. It includes distance measures between these vectors for 4005 languages, which are accessible via the lang2vec tool. Despite being frequently cited, URIEL is limited in terms of linguistic inclusion and overall usability. To tackle these challenges, we introduce URIEL+, an enhanced version of URIEL and lang2vec addressing these limitations. In addition to expanding typological feature coverage for 2898 languages, URIEL+ improves user experience with robust, customizable distance calculations to better suit the needs of the users. These upgrades also offer competitive performance on downstream tasks and provide distances that better align with linguistic distance studies.

Title: Underwater Image Enhancement with Physical-based Denoising Diffusion Implicit Models

• Authors: Nguyen Gia Bach, Chanh Minh Tran, Eiji Kamioka, Phan Xuan Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18476
• Pdf URL: https://arxiv.org/pdf/2409.18476
• Copy Paste: [[2409.18476]] Underwater Image Enhancement with Physical-based Denoising Diffusion Implicit Models(https://arxiv.org/abs/2409.18476)
• Keywords: robust, diffusion, generative
• Abstract: Underwater vision is crucial for autonomous underwater vehicles (AUVs), and enhancing degraded underwater images in real-time on a resource-constrained AUV is a key challenge due to factors like light absorption and scattering, or the sufficient model computational complexity to resolve such factors. Traditional image enhancement techniques lack adaptability to varying underwater conditions, while learning-based methods, particularly those using convolutional neural networks (CNNs) and generative adversarial networks (GANs), offer more robust solutions but face limitations such as inadequate enhancement, unstable training, or mode collapse. Denoising diffusion probabilistic models (DDPMs) have emerged as a state-of-the-art approach in image-to-image tasks but require intensive computational complexity to achieve the desired underwater image enhancement (UIE) using the recent UW-DDPM solution. To address these challenges, this paper introduces UW-DiffPhys, a novel physical-based and diffusion-based UIE approach. UW-DiffPhys combines light-computation physical-based UIE network components with a denoising U-Net to replace the computationally intensive distribution transformation U-Net in the existing UW-DDPM framework, reducing complexity while maintaining performance. Additionally, the Denoising Diffusion Implicit Model (DDIM) is employed to accelerate the inference process through non-Markovian sampling. Experimental results demonstrate that UW-DiffPhys achieved a substantial reduction in computational complexity and inference time compared to UW-DDPM, with competitive performance in key metrics such as PSNR, SSIM, UCIQE, and an improvement in the overall underwater image quality UIQM metric. The implementation code can be found at the following repository: this https URL

Title: Temporal2Seq: A Unified Framework for Temporal Video Understanding Tasks

• Authors: Min Yang, Zichen Zhang, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18478
• Pdf URL: https://arxiv.org/pdf/2409.18478
• Copy Paste: [[2409.18478]] Temporal2Seq: A Unified Framework for Temporal Video Understanding Tasks(https://arxiv.org/abs/2409.18478)
• Keywords: segmentation
• Abstract: With the development of video understanding, there is a proliferation of tasks for clip-level temporal video analysis, including temporal action detection (TAD), temporal action segmentation (TAS), and generic event boundary detection (GEBD). While task-specific video understanding models have exhibited outstanding performance in each task, there remains a dearth of a unified framework capable of simultaneously addressing multiple tasks, which is a promising direction for the next generation of AI. To this end, in this paper, we propose a single unified framework, coined as Temporal2Seq, to formulate the output of these temporal video understanding tasks as a sequence of discrete tokens. With this unified token representation, Temporal2Seq can train a generalist model within a single architecture on different video understanding tasks. In the absence of multi-task learning (MTL) benchmarks, we compile a comprehensive co-training dataset by borrowing the datasets from TAD, TAS, and GEBD tasks. We evaluate our Temporal2Seq generalist model on the corresponding test sets of three tasks, demonstrating that Temporal2Seq can produce reasonable results on various tasks and achieve advantages compared with single-task training on this framework. We also investigate the generalization performance of our generalist model on new datasets from different tasks, which yields superior performance to the specific model.

Title: CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns

• Authors: Shengsheng Lin, Weiwei Lin, Xinyi Hu, Wentai Wu, Ruichao Mo, Haocheng Zhong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18479
• Pdf URL: https://arxiv.org/pdf/2409.18479
• Copy Paste: [[2409.18479]] CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns(https://arxiv.org/abs/2409.18479)
• Keywords: transformer
• Abstract: The stable periodic patterns present in time series data serve as the foundation for conducting long-horizon forecasts. In this paper, we pioneer the exploration of explicitly modeling this periodicity to enhance the performance of models in long-term time series forecasting (LTSF) tasks. Specifically, we introduce the Residual Cycle Forecasting (RCF) technique, which utilizes learnable recurrent cycles to model the inherent periodic patterns within sequences, and then performs predictions on the residual components of the modeled cycles. Combining RCF with a Linear layer or a shallow MLP forms the simple yet powerful method proposed in this paper, called CycleNet. CycleNet achieves state-of-the-art prediction accuracy in multiple domains including electricity, weather, and energy, while offering significant efficiency advantages by reducing over 90% of the required parameter quantity. Furthermore, as a novel plug-and-play technique, the RCF can also significantly improve the prediction accuracy of existing models, including PatchTST and iTransformer. The source code is available at: this https URL.

Title: Deep Heterogeneous Contrastive Hyper-Graph Learning for In-the-Wild Context-Aware Human Activity Recognition

• Authors: Wen Ge, Guanyi Mou, Emmanuel O. Agu, Kyumin Lee
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18481
• Pdf URL: https://arxiv.org/pdf/2409.18481
• Copy Paste: [[2409.18481]] Deep Heterogeneous Contrastive Hyper-Graph Learning for In-the-Wild Context-Aware Human Activity Recognition(https://arxiv.org/abs/2409.18481)
• Keywords: explainability
• Abstract: Human Activity Recognition (HAR) is a challenging, multi-label classification problem as activities may co-occur and sensor signals corresponding to the same activity may vary in different contexts (e.g., different device placements). This paper proposes a Deep Heterogeneous Contrastive Hyper-Graph Learning (DHC-HGL) framework that captures heterogenous Context-Aware HAR (CA-HAR) hypergraph properties in a message-passing and neighborhood-aggregation fashion. Prior work only explored homogeneous or shallow-node-heterogeneous graphs. DHC-HGL handles heterogeneous CA-HAR data by innovatively 1) Constructing three different types of sub-hypergraphs that are each passed through different custom HyperGraph Convolution (HGC) layers designed to handle edge-heterogeneity and 2) Adopting a contrastive loss function to ensure node-heterogeneity. In rigorous evaluation on two CA-HAR datasets, DHC-HGL significantly outperformed state-of-the-art baselines by 5.8% to 16.7% on Matthews Correlation Coefficient (MCC) and 3.0% to 8.4% on Macro F1 scores. UMAP visualizations of learned CA-HAR node embeddings are also presented to enhance model explainability.

Title: HSTFL: A Heterogeneous Federated Learning Framework for Misaligned Spatiotemporal Forecasting

• Authors: Shuowei Cai, Hao Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18482
• Pdf URL: https://arxiv.org/pdf/2409.18482
• Copy Paste: [[2409.18482]] HSTFL: A Heterogeneous Federated Learning Framework for Misaligned Spatiotemporal Forecasting(https://arxiv.org/abs/2409.18482)
• Keywords: privacy, attack, federate
• Abstract: Spatiotemporal forecasting has emerged as an indispensable building block of diverse smart city applications, such as intelligent transportation and smart energy management. Recent advancements have uncovered that the performance of spatiotemporal forecasting can be significantly improved by integrating knowledge in geo-distributed time series data from different domains, \eg enhancing real-estate appraisal with human mobility data; joint taxi and bike demand predictions. While effective, existing approaches assume a centralized data collection and exploitation environment, overlooking the privacy and commercial interest concerns associated with data owned by different parties. In this paper, we investigate multi-party collaborative spatiotemporal forecasting without direct access to multi-source private data. However, this task is challenging due to 1) cross-domain feature heterogeneity and 2) cross-client geographical heterogeneity, where standard horizontal or vertical federated learning is inapplicable. To this end, we propose a Heterogeneous SpatioTemporal Federated Learning (HSTFL) framework to enable multiple clients to collaboratively harness geo-distributed time series data from different domains while preserving privacy. Specifically, we first devise vertical federated spatiotemporal representation learning to locally preserve spatiotemporal dependencies among individual participants and generate effective representations for heterogeneous data. Then we propose a cross-client virtual node alignment block to incorporate cross-client spatiotemporal dependencies via a multi-level knowledge fusion scheme. Extensive privacy analysis and experimental evaluations demonstrate that HSTFL not only effectively resists inference attacks but also provides a significant improvement against various baselines.

Title: Evaluation of OpenAI o1: Opportunities and Challenges of AGI

• Authors: Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao, Zhenyuan Yang, Jingyuan Chen, Peilong Wang, Wei Ruan, Hui Wang, Huan Zhao, Jing Zhang, Yiming Ren, Shihuan Qin, Tong Chen, Jiaxi Li, Arif Hassan Zidan, Afrar Jahin, Minheng Chen, Sichen Xia, Jason Holmes, Yan Zhuang, Jiaqi Wang, Bochen Xu, Weiran Xia, Jichao Yu, Kaibo Tang, Yaxuan Yang, Bolun Sun, Tao Yang, Guoyu Lu, Xianqiao Wang, Lilong Chai, He Li, Jin Lu, Lichao Sun, Xin Zhang, Bao Ge, Xintao Hu, Lian Zhang, Hua Zhou, Lu Zhang, Shu Zhang, Ninghao Liu, Bei Jiang, Linglong Kong, Zhen Xiang, Yudan Ren, Jun Liu, Xi Jiang, Yu Bao, Wei Zhang, Xiang Li, Gang Li, Wei Liu, Dinggang Shen, Andrea Sikora, Xiaoming Zhai, Dajiang Zhu, Tianming Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18486
• Pdf URL: https://arxiv.org/pdf/2409.18486
• Copy Paste: [[2409.18486]] Evaluation of OpenAI o1: Opportunities and Challenges of AGI(https://arxiv.org/abs/2409.18486)
• Keywords: large language model
• Abstract: This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natural sciences, medicine, linguistics, and social sciences. Through rigorous testing, o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance in areas ranging from coding challenges to scientific reasoning and from language processing to creative problem-solving. Key findings include: -83.3% success rate in solving complex competitive programming problems, surpassing many human experts. -Superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models. -100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions. -Advanced natural language inference capabilities across general and specialized domains like medicine. -Impressive performance in chip design tasks, outperforming specialized models in areas such as EDA script generation and bug analysis. -Remarkable proficiency in anthropology and geology, demonstrating deep understanding and reasoning in these specialized fields. -Strong capabilities in quantitative investing. O1 has comprehensive financial knowledge and statistical modeling skills. -Effective performance in social media analysis, including sentiment analysis and emotion recognition. The model excelled particularly in tasks requiring intricate reasoning and knowledge integration across various fields. While some limitations were observed, including occasional errors on simpler problems and challenges with certain highly specialized concepts, the overall results indicate significant progress towards artificial general intelligence.

Title: Treating Brain-inspired Memories as Priors for Diffusion Model to Forecast Multivariate Time Series

• Authors: Muyao Wang, Wenchao Chen, Zhibin Duan, Bo Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18491
• Pdf URL: https://arxiv.org/pdf/2409.18491
• Copy Paste: [[2409.18491]] Treating Brain-inspired Memories as Priors for Diffusion Model to Forecast Multivariate Time Series(https://arxiv.org/abs/2409.18491)
• Keywords: robust, diffusion
• Abstract: Forecasting Multivariate Time Series (MTS) involves significant challenges in various application domains. One immediate challenge is modeling temporal patterns with the finite length of the input. These temporal patterns usually involve periodic and sudden events that recur across different channels. To better capture temporal patterns, we get inspiration from humans' memory mechanisms and propose a channel-shared, brain-inspired memory module for MTS. Specifically, brain-inspired memory comprises semantic and episodic memory, where the former is used to capture general patterns, such as periodic events, and the latter is employed to capture special patterns, such as sudden events, respectively. Meanwhile, we design corresponding recall and update mechanisms to better utilize these patterns. Furthermore, acknowledging the capacity of diffusion models to leverage memory as a prior, we present a brain-inspired memory-augmented diffusion model. This innovative model retrieves relevant memories for different channels, utilizing them as distinct priors for MTS predictions. This incorporation significantly enhances the accuracy and robustness of predictions. Experimental results on eight datasets consistently validate the superiority of our approach in capturing and leveraging diverse recurrent temporal patterns across different channels.

Title: Fairness-aware Multiobjective Evolutionary Learning

• Authors: Qingquan Zhang, Jialin Liu, Xin Yao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18499
• Pdf URL: https://arxiv.org/pdf/2409.18499
• Copy Paste: [[2409.18499]] Fairness-aware Multiobjective Evolutionary Learning(https://arxiv.org/abs/2409.18499)
• Keywords: fair
• Abstract: Multiobjective evolutionary learning (MOEL) has demonstrated its advantages of training fairer machine learning models considering a predefined set of conflicting objectives, including accuracy and different fairness measures. Recent works propose to construct a representative subset of fairness measures as optimisation objectives of MOEL throughout model training. However, the determination of a representative measure set relies on dataset, prior knowledge and requires substantial computational costs. What's more, those representative measures may differ across different model training processes. Instead of using a static predefined set determined before model training, this paper proposes to dynamically and adaptively determine a representative measure set online during model training. The dynamically determined representative set is then used as optimising objectives of the MOEL framework and can vary with time. Extensive experimental results on 12 well-known benchmark datasets demonstrate that our proposed framework achieves outstanding performance compared to state-of-the-art approaches for mitigating unfairness in terms of accuracy as well as 25 fairness measures although only a few of them were dynamically selected and used as optimisation objectives. The results indicate the importance of setting optimisation objectives dynamically during training.

Title: Do We Need Domain-Specific Embedding Models? An Empirical Investigation

• Authors: Yixuan Tang, Yi Yang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.18511
• Pdf URL: https://arxiv.org/pdf/2409.18511
• Copy Paste: [[2409.18511]] Do We Need Domain-Specific Embedding Models? An Empirical Investigation(https://arxiv.org/abs/2409.18511)
• Keywords: large language model
• Abstract: Embedding models play a crucial role in representing and retrieving information across various NLP applications. Recent advancements in Large Language Models (LLMs) have further enhanced the performance of embedding models, which are trained on massive amounts of text covering almost every domain. These models are often benchmarked on general-purpose datasets like Massive Text Embedding Benchmark (MTEB), where they demonstrate superior performance. However, a critical question arises: Is the development of domain-specific embedding models necessary when general-purpose models are trained on vast corpora that already include specialized domain texts? In this paper, we empirically investigate this question, choosing the finance domain as an example. We introduce the Finance Massive Text Embedding Benchmark (FinMTEB), a counterpart to MTEB that consists of financial domain-specific text datasets. We evaluate the performance of seven state-of-the-art embedding models on FinMTEB and observe a significant performance drop compared to their performance on MTEB. To account for the possibility that this drop is driven by FinMTEB's higher complexity, we propose four measures to quantify dataset complexity and control for this factor in our analysis. Our analysis provides compelling evidence that state-of-the-art embedding models struggle to capture domain-specific linguistic and semantic patterns, even when trained on large general-purpose corpora. This study sheds light on the necessity of developing domain-specific embedding models in the LLM era, offering valuable insights for researchers and practitioners.

Title: Token Caching for Diffusion Transformer Acceleration

• Authors: Jinming Lou, Wenyang Luo, Yufan Liu, Bing Li, Xinmiao Ding, Weiming Hu, Jiajiong Cao, Yuming Li, Chenguang Ma
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18523
• Pdf URL: https://arxiv.org/pdf/2409.18523
• Copy Paste: [[2409.18523]] Token Caching for Diffusion Transformer Acceleration(https://arxiv.org/abs/2409.18523)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion transformers have gained substantial interest in diffusion generative modeling due to their outstanding performance. However, their high computational cost, arising from the quadratic computational complexity of attention mechanisms and multi-step inference, presents a significant bottleneck. To address this challenge, we propose TokenCache, a novel post-training acceleration method that leverages the token-based multi-block architecture of transformers to reduce redundant computations among tokens across inference steps. TokenCache specifically addresses three critical questions in the context of diffusion transformers: (1) which tokens should be pruned to eliminate redundancy, (2) which blocks should be targeted for efficient pruning, and (3) at which time steps caching should be applied to balance speed and quality. In response to these challenges, TokenCache introduces a Cache Predictor that assigns importance scores to tokens, enabling selective pruning without compromising model performance. Furthermore, we propose an adaptive block selection strategy to focus on blocks with minimal impact on the network's output, along with a Two-Phase Round-Robin (TPRR) scheduling policy to optimize caching intervals throughout the denoising process. Experimental results across various models demonstrate that TokenCache achieves an effective trade-off between generation quality and inference speed for diffusion transformers. Our code will be publicly available.

Title: Security Analysis of Top-Ranked mHealth Fitness Apps: An Empirical Study

• Authors: Albin Forsberg, Leonardo Horn Iwaya
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18528
• Pdf URL: https://arxiv.org/pdf/2409.18528
• Copy Paste: [[2409.18528]] Security Analysis of Top-Ranked mHealth Fitness Apps: An Empirical Study(https://arxiv.org/abs/2409.18528)
• Keywords: secure, security, privacy
• Abstract: Mobile health applications (mHealth apps), particularly in the health and fitness category, have experienced an increase in popularity due to their convenience and availability. However, this widespread adoption raises concerns regarding the security of the user's data. In this study, we investigate the security vulnerabilities of ten top-ranked Android health and fitness apps, a set that accounts for 237 million downloads. We performed several static and dynamic security analyses using tools such as the Mobile Security Framework (MobSF) and Android emulators. We also checked the server's security levels with Qualys SSL, which allowed us to gain insights into the security posture of the servers communicating with the mHealth fitness apps. Our findings revealed many vulnerabilities, such as insecure coding, hardcoded sensitive information, over-privileged permissions, misconfiguration, and excessive communication with third-party domains. For instance, some apps store their database API key directly in the code while also exposing their database URL. We found insecure encryption methods in six apps, such as using AES with ECB mode. Two apps communicated with an alarming number of approximately 230 domains each, and a third app with over 100 domains, exacerbating privacy linkability threats. The study underscores the importance of continuous security assessments of top-ranked mHealth fitness apps to better understand the threat landscape and inform app developers.

Title: How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks?

• Authors: Jose Sosa, Mohamed Aloulou, Danila Rukhovich, Rim Sleimi, Boonyarit Changaival, Anis Kacem, Djamila Aouada
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18536
• Pdf URL: https://arxiv.org/pdf/2409.18536
• Copy Paste: [[2409.18536]] How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks?(https://arxiv.org/abs/2409.18536)
• Keywords: transformer, segmentation
• Abstract: Self-supervised pre-training has proven highly effective for many computer vision tasks, particularly when labelled data are scarce. In the context of Earth Observation (EO), foundation models and various other Vision Transformer (ViT)-based approaches have been successfully applied for transfer learning to downstream tasks. However, it remains unclear under which conditions pre-trained models offer significant advantages over training from scratch. In this study, we investigate the effectiveness of pre-training ViT-based Masked Autoencoders (MAE) for downstream EO tasks, focusing on reconstruction, segmentation, and classification. We consider two large ViT-based MAE pre-trained models: a foundation model (Prithvi) and SatMAE. We evaluate Prithvi on reconstruction and segmentation-based downstream tasks, and for SatMAE we assess its performance on a classification downstream task. Our findings suggest that pre-training is particularly beneficial when the fine-tuning task closely resembles the pre-training task, e.g. reconstruction. In contrast, for tasks such as segmentation or classification, training from scratch with specific hyperparameter adjustments proved to be equally or more effective.

Title: A Survey on Complex Tasks for Goal-Directed Interactive Agents

• Authors: Mareike Hartmann, Alexander Koller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18538
• Pdf URL: https://arxiv.org/pdf/2409.18538
• Copy Paste: [[2409.18538]] A Survey on Complex Tasks for Goal-Directed Interactive Agents(https://arxiv.org/abs/2409.18538)
• Keywords: large language model
• Abstract: Goal-directed interactive agents, which autonomously complete tasks through interactions with their environment, can assist humans in various domains of their daily lives. Recent advances in large language models (LLMs) led to a surge of new, more and more challenging tasks to evaluate such agents. To properly contextualize performance across these tasks, it is imperative to understand the different challenges they pose to agents. To this end, this survey compiles relevant tasks and environments for evaluating goal-directed interactive agents, structuring them along dimensions relevant for understanding current obstacles. An up-to-date compilation of relevant resources can be found on our project website: this https URL.

Title: Reducing Semantic Ambiguity In Domain Adaptive Semantic Segmentation Via Probabilistic Prototypical Pixel Contrast

• Authors: Xiaoke Hao, Shiyu Liu, Chuanbo Feng, Ye Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18543
• Pdf URL: https://arxiv.org/pdf/2409.18543
• Copy Paste: [[2409.18543]] Reducing Semantic Ambiguity In Domain Adaptive Semantic Segmentation Via Probabilistic Prototypical Pixel Contrast(https://arxiv.org/abs/2409.18543)
• Keywords: segmentation
• Abstract: Domain adaptation aims to reduce the model degradation on the target domain caused by the domain shift between the source and target domains. Although encouraging performance has been achieved by combining cognitive learning with the self-training paradigm, they suffer from ambiguous scenarios caused by scale, illumination, or overlapping when deploying deterministic embedding. To address these issues, we propose probabilistic proto-typical pixel contrast (PPPC), a universal adaptation framework that models each pixel embedding as a probability via multivariate Gaussian distribution to fully exploit the uncertainty within them, eventually improving the representation quality of the model. In addition, we derive prototypes from probability estimation posterior probability estimation which helps to push the decision boundary away from the ambiguity points. Moreover, we employ an efficient method to compute similarity between distributions, eliminating the need for sampling and reparameterization, thereby significantly reducing computational overhead. Further, we dynamically select the ambiguous crops at the image level to enlarge the number of boundary points involved in contrastive learning, which benefits the establishment of precise distributions for each category. Extensive experimentation demonstrates that PPPC not only helps to address ambiguity at the pixel level, yielding discriminative representations but also achieves significant improvements in both synthetic-to-real and day-to-night adaptation tasks. It surpasses the previous state-of-the-art (SOTA) by +5.2% mIoU in the most challenging daytime-to-nighttime adaptation scenario, exhibiting stronger generalization on other unseen datasets. The code and models are available at this https URL.

Title: Research on Predicting Public Opinion Event Heat Levels Based on Large Language Models

• Authors: Yi Ren, Tianyi Zhang, Weibin Li, DuoMu Zhou, Chenhao Qin, FangCheng Dong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18548
• Pdf URL: https://arxiv.org/pdf/2409.18548
• Copy Paste: [[2409.18548]] Research on Predicting Public Opinion Event Heat Levels Based on Large Language Models(https://arxiv.org/abs/2409.18548)
• Keywords: robust, large language model
• Abstract: In recent years, with the rapid development of large language models, serval models such as GPT-4o have demonstrated extraordinary capabilities, surpassing human performance in various language tasks. As a result, many researchers have begun exploring their potential applications in the field of public opinion analysis. This study proposes a novel large-language-models-based method for public opinion event heat level prediction. First, we preprocessed and classified 62,836 Chinese hot event data collected between July 2022 and December 2023. Then, based on each event's online dissemination heat index, we used the MiniBatchKMeans algorithm to automatically cluster the events and categorize them into four heat levels (ranging from low heat to very high heat). Next, we randomly selected 250 events from each heat level, totalling 1,000 events, to build the evaluation dataset. During the evaluation process, we employed various large language models to assess their accuracy in predicting event heat levels in two scenarios: without reference cases and with similar case references. The results showed that GPT-4o and DeepseekV2 performed the best in the latter case, achieving prediction accuracies of 41.4% and 41.5%, respectively. Although the overall prediction accuracy remains relatively low, it is worth noting that for low-heat (Level 1) events, the prediction accuracies of these two models reached 73.6% and 70.4%, respectively. Additionally, the prediction accuracy showed a downward trend from Level 1 to Level 4, which correlates with the uneven distribution of data across the heat levels in the actual dataset. This suggests that with the more robust dataset, public opinion event heat level prediction based on large language models will have significant research potential for the future.

Title: Efficient Noise Mitigation for Enhancing Inference Accuracy in DNNs on Mixed-Signal Accelerators

• Authors: Seyedarmin Azizi, Mohammad Erfan Sadeghi, Mehdi Kamal, Massoud Pedram
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18553
• Pdf URL: https://arxiv.org/pdf/2409.18553
• Copy Paste: [[2409.18553]] Efficient Noise Mitigation for Enhancing Inference Accuracy in DNNs on Mixed-Signal Accelerators(https://arxiv.org/abs/2409.18553)
• Keywords: robust
• Abstract: In this paper, we propose a framework to enhance the robustness of the neural models by mitigating the effects of process-induced and aging-related variations of analog computing components on the accuracy of the analog neural networks. We model these variations as the noise affecting the precision of the activations and introduce a denoising block inserted between selected layers of a pre-trained model. We demonstrate that training the denoising block significantly increases the model's robustness against various noise levels. To minimize the overhead associated with adding these blocks, we present an exploration algorithm to identify optimal insertion points for the denoising blocks. Additionally, we propose a specialized architecture to efficiently execute the denoising blocks, which can be integrated into mixed-signal accelerators. We evaluate the effectiveness of our approach using Deep Neural Network (DNN) models trained on the ImageNet and CIFAR-10 datasets. The results show that on average, by accepting 2.03% parameter count overhead, the accuracy drop due to the variations reduces from 31.7% to 1.15%.

Title: An Enhanced Federated Prototype Learning Method under Domain Shift

• Authors: Liang Kuang, Kuangpu Guo, Jian Liang, Jianguo Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18578
• Pdf URL: https://arxiv.org/pdf/2409.18578
• Copy Paste: [[2409.18578]] An Enhanced Federated Prototype Learning Method under Domain Shift(https://arxiv.org/abs/2409.18578)
• Keywords: federate
• Abstract: Federated Learning (FL) allows collaborative machine learning training without sharing private data. Numerous studies have shown that one significant factor affecting the performance of federated learning models is the heterogeneity of data across different clients, especially when the data is sampled from various domains. A recent paper introduces variance-aware dual-level prototype clustering and uses a novel $\alpha$-sparsity prototype loss, which increases intra-class similarity and reduces inter-class similarity. To ensure that the features converge within specific clusters, we introduce an improved algorithm, Federated Prototype Learning with Convergent Clusters, abbreviated as FedPLCC. To increase inter-class distances, we weight each prototype with the size of the cluster it represents. To reduce intra-class distances, considering that prototypes with larger distances might come from different domains, we select only a certain proportion of prototypes for the loss function calculation. Evaluations on the Digit-5, Office-10, and DomainNet datasets show that our method performs better than existing approaches.

Title: Hit the Sweet Spot! Span-Level Ensemble for Large Language Models

• Authors: Yangyifan Xu, Jianghao Chen, Junhong Wu, Jiajun Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18583
• Pdf URL: https://arxiv.org/pdf/2409.18583
• Copy Paste: [[2409.18583]] Hit the Sweet Spot! Span-Level Ensemble for Large Language Models(https://arxiv.org/abs/2409.18583)
• Keywords: robust, large language model
• Abstract: Ensembling various LLMs to unlock their complementary potential and leverage their individual strengths is highly valuable. Previous studies typically focus on two main paradigms: sample-level and token-level ensembles. Sample-level ensemble methods either select or blend fully generated outputs, which hinders dynamic correction and enhancement of outputs during the generation process. On the other hand, token-level ensemble methods enable real-time correction through fine-grained ensemble at each generation step. However, the information carried by an individual token is quite limited, leading to suboptimal decisions at each step. To address these issues, we propose SweetSpan, a span-level ensemble method that effectively balances the need for real-time adjustments and the information required for accurate ensemble decisions. Our approach involves two key steps: First, we have each candidate model independently generate candidate spans based on the shared prefix. Second, we calculate perplexity scores to facilitate mutual evaluation among the candidate models and achieve robust span selection by filtering out unfaithful scores. To comprehensively evaluate ensemble methods, we propose a new challenging setting (ensemble models with significant performance gaps) in addition to the standard setting (ensemble the best-performing models) to assess the performance of model ensembles in more realistic scenarios. Experimental results in both standard and challenging settings across various language generation tasks demonstrate the effectiveness, robustness, and versatility of our approach compared with previous ensemble methods.

Title: TemporalPaD: a reinforcement-learning framework for temporal feature representation and dimension reduction

• Authors: Xuechen Mu, Zhenyu Huang, Kewei Li, Haotian Zhang, Xiuli Wang, Yusi Fan, Kai Zhang, Fengfeng Zhou
• Subjects: cs.LG, cs.AI, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2409.18597
• Pdf URL: https://arxiv.org/pdf/2409.18597
• Copy Paste: [[2409.18597]] TemporalPaD: a reinforcement-learning framework for temporal feature representation and dimension reduction(https://arxiv.org/abs/2409.18597)
• Keywords: extraction
• Abstract: Recent advancements in feature representation and dimension reduction have highlighted their crucial role in enhancing the efficacy of predictive modeling. This work introduces TemporalPaD, a novel end-to-end deep learning framework designed for temporal pattern datasets. TemporalPaD integrates reinforcement learning (RL) with neural networks to achieve concurrent feature representation and feature reduction. The framework consists of three cooperative modules: a Policy Module, a Representation Module, and a Classification Module, structured based on the Actor-Critic (AC) framework. The Policy Module, responsible for dimensionality reduction through RL, functions as the actor, while the Representation Module for feature extraction and the Classification Module collectively serve as the critic. We comprehensively evaluate TemporalPaD using 29 UCI datasets, a well-known benchmark for validating feature reduction algorithms, through 10 independent tests and 10-fold cross-validation. Additionally, given that TemporalPaD is specifically designed for time series data, we apply it to a real-world DNA classification problem involving enhancer category and enhancer strength. The results demonstrate that TemporalPaD is an efficient and effective framework for achieving feature reduction, applicable to both structured data and sequence datasets. The source code of the proposed TemporalPaD is freely available as supplementary material to this article and at this http URL.

Title: Privacy-Preserving Quantum Annealing for Quadratic Unconstrained Binary Optimization (QUBO) Problems

• Authors: Moyang Xie, Yuan Zhang, Sheng Zhong, Qun Li
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2409.18601
• Pdf URL: https://arxiv.org/pdf/2409.18601
• Copy Paste: [[2409.18601]] Privacy-Preserving Quantum Annealing for Quadratic Unconstrained Binary Optimization (QUBO) Problems(https://arxiv.org/abs/2409.18601)
• Keywords: privacy
• Abstract: Quantum annealers offer a promising approach to solve Quadratic Unconstrained Binary Optimization (QUBO) problems, which have a wide range of applications. However, when a user submits its QUBO problem to a third-party quantum annealer, the problem itself may disclose the user's private information to the quantum annealing service provider. To mitigate this risk, we introduce a privacy-preserving QUBO framework and propose a novel solution method. Our approach employs a combination of digit-wise splitting and matrix permutation to obfuscate the QUBO problem's model matrix $Q$, effectively concealing the matrix elements. In addition, based on the solution to the obfuscated version of the QUBO problem, we can reconstruct the solution to the original problem with high accuracy. Theoretical analysis and empirical tests confirm the efficacy and efficiency of our proposed technique, demonstrating its potential for preserving user privacy in quantum annealing services.

Title: Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations

• Authors: Nicolò Penzo, Maryam Sajedinia, Bruno Lepri, Sara Tonelli, Marco Guerini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18602
• Pdf URL: https://arxiv.org/pdf/2409.18602
• Copy Paste: [[2409.18602]] Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations(https://arxiv.org/abs/2409.18602)
• Keywords: privacy
• Abstract: Assessing the performance of systems to classify Multi-Party Conversations (MPC) is challenging due to the interconnection between linguistic and structural characteristics of conversations. Conventional evaluation methods often overlook variances in model behavior across different levels of structural complexity on interaction graphs. In this work, we propose a methodological pipeline to investigate model performance across specific structural attributes of conversations. As a proof of concept we focus on Response Selection and Addressee Recognition tasks, to diagnose model weaknesses. To this end, we extract representative diagnostic subdatasets with a fixed number of users and a good structural variety from a large and open corpus of online MPCs. We further frame our work in terms of data minimization, avoiding the use of original usernames to preserve privacy, and propose alternatives to using original text messages. Results show that response selection relies more on the textual content of conversations, while addressee recognition requires capturing their structural dimension. Using an LLM in a zero-shot setting, we further highlight how sensitivity to prompt variations is task-dependent.

Title: Differentially Private Non Parametric Copulas: Generating synthetic data with non parametric copulas under privacy guarantees

• Authors: Pablo A. Osorio-Marulanda, John Esteban Castro Ramirez, Mikel Hernández Jiménez, Nicolas Moreno Reyes, Gorka Epelde Unanue
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2409.18611
• Pdf URL: https://arxiv.org/pdf/2409.18611
• Copy Paste: [[2409.18611]] Differentially Private Non Parametric Copulas: Generating synthetic data with non parametric copulas under privacy guarantees(https://arxiv.org/abs/2409.18611)
• Keywords: privacy, attack
• Abstract: Creation of synthetic data models has represented a significant advancement across diverse scientific fields, but this technology also brings important privacy considerations for users. This work focuses on enhancing a non-parametric copula-based synthetic data generation model, DPNPC, by incorporating Differential Privacy through an Enhanced Fourier Perturbation method. The model generates synthetic data for mixed tabular databases while preserving privacy. We compare DPNPC with three other models (PrivBayes, DP-Copula, and DP-Histogram) across three public datasets, evaluating privacy, utility, and execution time. DPNPC outperforms others in modeling multivariate dependencies, maintaining privacy for small $\epsilon$ values, and reducing training times. However, limitations include the need to assess the model's performance with different encoding methods and consider additional privacy attacks. Future research should address these areas to enhance privacy-preserving synthetic data generation.

Title: Model-based Preference Optimization in Abstractive Summarization without Human Feedback

• Authors: Jaepill Choi, Kyubyung Chae, Jiwoo Song, Yohan Jo, Taesup Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18618
• Pdf URL: https://arxiv.org/pdf/2409.18618
• Copy Paste: [[2409.18618]] Model-based Preference Optimization in Abstractive Summarization without Human Feedback(https://arxiv.org/abs/2409.18618)
• Keywords: large language model
• Abstract: In abstractive summarization, the challenge of producing concise and accurate summaries arises from the vast amount of information contained in the source document. Consequently, although Large Language Models (LLMs) can generate fluent text, they often introduce inaccuracies by hallucinating content not found in the original source. While supervised fine-tuning methods that maximize likelihood contribute to this issue, they do not consistently enhance the faithfulness of the summaries. Preference-based optimization methods, such as Direct Preference Optimization (DPO), can further refine the model to align with human preferences. However, these methods still heavily depend on costly human feedback. In this work, we introduce a novel and straightforward approach called Model-based Preference Optimization (MPO) to fine-tune LLMs for improved summarization abilities without any human feedback. By leveraging the model's inherent summarization capabilities, we create a preference dataset that is fully generated by the model using different decoding strategies. Our experiments on standard summarization datasets and various metrics demonstrate that our proposed MPO significantly enhances the quality of generated summaries without relying on human feedback.

Title: Unsupervised Fingerphoto Presentation Attack Detection With Diffusion Models

• Authors: Hailin Li, Raghavendra Ramachandra, Mohamed Ragab, Soumik Mondal, Yong Kiam Tan, Khin Mi Mi Aung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18636
• Pdf URL: https://arxiv.org/pdf/2409.18636
• Copy Paste: [[2409.18636]] Unsupervised Fingerphoto Presentation Attack Detection With Diffusion Models(https://arxiv.org/abs/2409.18636)
• Keywords: attack, biometric, diffusion
• Abstract: Smartphone-based contactless fingerphoto authentication has become a reliable alternative to traditional contact-based fingerprint biometric systems owing to rapid advances in smartphone camera technology. Despite its convenience, fingerprint authentication through fingerphotos is more vulnerable to presentation attacks, which has motivated recent research efforts towards developing fingerphoto Presentation Attack Detection (PAD) techniques. However, prior PAD approaches utilized supervised learning methods that require labeled training data for both bona fide and attack samples. This can suffer from two key issues, namely (i) generalization:the detection of novel presentation attack instruments (PAIs) unseen in the training data, and (ii) scalability:the collection of a large dataset of attack samples using different PAIs. To address these challenges, we propose a novel unsupervised approach based on a state-of-the-art deep-learning-based diffusion model, the Denoising Diffusion Probabilistic Model (DDPM), which is trained solely on bona fide samples. The proposed approach detects Presentation Attacks (PA) by calculating the reconstruction similarity between the input and output pairs of the DDPM. We present extensive experiments across three PAI datasets to test the accuracy and generalization capability of our approach. The results show that the proposed DDPM-based PAD method achieves significantly better detection error rates on several PAI classes compared to other baseline unsupervised approaches.

Title: Enhanced Convolution Neural Network with Optimized Pooling and Hyperparameter Tuning for Network Intrusion Detection

• Authors: Ayush Kumar Sharma, Sourav Patel, Supriya Bharat Wakchaure, Abirami S
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18642
• Pdf URL: https://arxiv.org/pdf/2409.18642
• Copy Paste: [[2409.18642]] Enhanced Convolution Neural Network with Optimized Pooling and Hyperparameter Tuning for Network Intrusion Detection(https://arxiv.org/abs/2409.18642)
• Keywords: security, protect, attack, robust
• Abstract: Network Intrusion Detection Systems (NIDS) are essential for protecting computer networks from malicious activities, including Denial of Service (DoS), Probing, User-to-Root (U2R), and Remote-to-Local (R2L) attacks. Without effective NIDS, networks are vulnerable to significant security breaches and data loss. Machine learning techniques provide a promising approach to enhance NIDS by automating threat detection and improving accuracy. In this research, we propose an Enhanced Convolutional Neural Network (EnCNN) for NIDS and evaluate its performance using the KDDCUP'99 dataset. Our methodology includes comprehensive data preprocessing, exploratory data analysis (EDA), and feature engineering. We compare EnCNN with various machine learning algorithms, including Logistic Regression, Decision Trees, Support Vector Machines (SVM), and ensemble methods like Random Forest, AdaBoost, and Voting Ensemble. The results show that EnCNN significantly improves detection accuracy, with a notable 10% increase over state-of-art approaches. This demonstrates the effectiveness of EnCNN in real-time network intrusion detection, offering a robust solution for identifying and mitigating security threats, and enhancing overall network resilience.

Title: When SAM2 Meets Video Camouflaged Object Segmentation: A Comprehensive Evaluation and Adaptation

• Authors: Yuli Zhou, Guolei Sun, Yawei Li, Luca Benini, Ender Konukoglu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18653
• Pdf URL: https://arxiv.org/pdf/2409.18653
• Copy Paste: [[2409.18653]] When SAM2 Meets Video Camouflaged Object Segmentation: A Comprehensive Evaluation and Adaptation(https://arxiv.org/abs/2409.18653)
• Keywords: large language model, segmentation
• Abstract: This study investigates the application and performance of the Segment Anything Model 2 (SAM2) in the challenging task of video camouflaged object segmentation (VCOS). VCOS involves detecting objects that blend seamlessly in the surroundings for videos, due to similar colors and textures, poor light conditions, etc. Compared to the objects in normal scenes, camouflaged objects are much more difficult to detect. SAM2, a video foundation model, has shown potential in various tasks. But its effectiveness in dynamic camouflaged scenarios remains under-explored. This study presents a comprehensive study on SAM2's ability in VCOS. First, we assess SAM2's performance on camouflaged video datasets using different models and prompts (click, box, and mask). Second, we explore the integration of SAM2 with existing multimodal large language models (MLLMs) and VCOS methods. Third, we specifically adapt SAM2 by fine-tuning it on the video camouflaged dataset. Our comprehensive experiments demonstrate that SAM2 has excellent zero-shot ability of detecting camouflaged objects in videos. We also show that this ability could be further improved by specifically adjusting SAM2's parameters for VCOS. The code will be available at this https URL

Title: Image-guided topic modeling for interpretable privacy classification

• Authors: Alina Elena Baia, Andrea Cavallaro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18674
• Pdf URL: https://arxiv.org/pdf/2409.18674
• Copy Paste: [[2409.18674]] Image-guided topic modeling for interpretable privacy classification(https://arxiv.org/abs/2409.18674)
• Keywords: privacy, large language model
• Abstract: Predicting and explaining the private information contained in an image in human-understandable terms is a complex and contextual task. This task is challenging even for large language models. To facilitate the understanding of privacy decisions, we propose to predict image privacy based on a set of natural language content descriptors. These content descriptors are associated with privacy scores that reflect how people perceive image content. We generate descriptors with our novel Image-guided Topic Modeling (ITM) approach. ITM leverages, via multimodality alignment, both vision information and image textual descriptions from a vision language model. We use the ITM-generated descriptors to learn a privacy predictor, Priv$\times$ITM, whose decisions are interpretable by design. Our Priv$\times$ITM classifier outperforms the reference interpretable method by 5 percentage points in accuracy and performs comparably to the current non-interpretable state-of-the-art model.

Title: Rehearsing Answers to Probable Questions with Perspective-Taking

• Authors: Yung-Yu Shih, Ziwei Xu, Hiroya Takamura, Yun-Nung Chen, Chung-Chi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18678
• Pdf URL: https://arxiv.org/pdf/2409.18678
• Copy Paste: [[2409.18678]] Rehearsing Answers to Probable Questions with Perspective-Taking(https://arxiv.org/abs/2409.18678)
• Keywords: large language model
• Abstract: Question answering (QA) has been a long-standing focus in the NLP field, predominantly addressing reading comprehension and common sense QA. However, scenarios involving the preparation of answers to probable questions during professional oral presentations remain underexplored. In this paper, we pioneer the examination of this crucial yet overlooked topic by utilizing real-world QA conversation transcripts between company managers and professional analysts. We explore the proposed task using three causal knowledge graphs (KGs) and three large language models (LLMs). This work provides foundational insights into the application of LLMs in professional QA scenarios, highlighting the importance of causal KGs and perspective-taking in generating effective responses.

Title: "Why" Has the Least Side Effect on Model Editing

• Authors: Tsung-Hsuan Pan, Chung-Chi Chen, Hen-Hsen Huang, Hsin-Hsi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18679
• Pdf URL: https://arxiv.org/pdf/2409.18679
• Copy Paste: [[2409.18679]] "Why" Has the Least Side Effect on Model Editing(https://arxiv.org/abs/2409.18679)
• Keywords: large language model
• Abstract: Training large language models (LLMs) from scratch is an expensive endeavor, particularly as world knowledge continually evolves. To maintain relevance and accuracy of LLMs, model editing has emerged as a pivotal research area. While these methods hold promise, they can also produce unintended side effects. Their underlying factors and causes remain largely unexplored. This paper delves into a critical factor-question type-by categorizing model editing questions. Our findings reveal that the extent of performance degradation varies significantly across different question types, providing new insights for experimental design in knowledge editing. Furthermore, we investigate whether insights from smaller models can be extrapolated to larger models. Our results indicate discrepancies in findings between models of different sizes, suggesting that insights from smaller models may not necessarily apply to larger models. Additionally, we examine the impact of batch size on side effects, discovering that increasing the batch size can mitigate performance drops.

Title: A Novel Unified Architecture for Low-Shot Counting by Detection and Segmentation

• Authors: Jer Pelhan, Alan Lukežič, Vitjan Zavrtanik, Matej Kristan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18686
• Pdf URL: https://arxiv.org/pdf/2409.18686
• Copy Paste: [[2409.18686]] A Novel Unified Architecture for Low-Shot Counting by Detection and Segmentation(https://arxiv.org/abs/2409.18686)
• Keywords: robust, segmentation
• Abstract: Low-shot object counters estimate the number of objects in an image using few or no annotated exemplars. Objects are localized by matching them to prototypes, which are constructed by unsupervised image-wide object appearance aggregation. Due to potentially diverse object appearances, the existing approaches often lead to overgeneralization and false positive detections. Furthermore, the best-performing methods train object localization by a surrogate loss, that predicts a unit Gaussian at each object center. This loss is sensitive to annotation error, hyperparameters and does not directly optimize the detection task, leading to suboptimal counts. We introduce GeCo, a novel low-shot counter that achieves accurate object detection, segmentation, and count estimation in a unified architecture. GeCo robustly generalizes the prototypes across objects appearances through a novel dense object query formulation. In addition, a novel counting loss is proposed, that directly optimizes the detection task and avoids the issues of the standard surrogate loss. GeCo surpasses the leading few-shot detection-based counters by $\sim$25\% in the total count MAE, achieves superior detection accuracy and sets a new solid state-of-the-art result across all low-shot counting setups.

Title: Learning from Pattern Completion: Self-supervised Controllable Generation

• Authors: Zhiqiang Chen, Guofan Fan, Jinying Gao, Lei Ma, Bo Lei, Tiejun Huang, Shan Yu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18694
• Pdf URL: https://arxiv.org/pdf/2409.18694
• Copy Paste: [[2409.18694]] Learning from Pattern Completion: Self-supervised Controllable Generation(https://arxiv.org/abs/2409.18694)
• Keywords: robust, segmentation
• Abstract: The human brain exhibits a strong ability to spontaneously associate different visual attributes of the same or similar visual scene, such as associating sketches and graffiti with real-world visual objects, usually without supervising information. In contrast, in the field of artificial intelligence, controllable generation methods like ControlNet heavily rely on annotated training datasets such as depth maps, semantic segmentation maps, and poses, which limits the method's scalability. Inspired by the neural mechanisms that may contribute to the brain's associative power, specifically the cortical modularization and hippocampal pattern completion, here we propose a self-supervised controllable generation (SCG) framework. Firstly, we introduce an equivariant constraint to promote inter-module independence and intra-module correlation in a modular autoencoder network, thereby achieving functional specialization. Subsequently, based on these specialized modules, we employ a self-supervised pattern completion approach for controllable generation training. Experimental results demonstrate that the proposed modular autoencoder effectively achieves functional specialization, including the modular processing of color, brightness, and edge detection, and exhibits brain-like features including orientation selectivity, color antagonism, and center-surround receptive fields. Through self-supervised training, associative generation capabilities spontaneously emerge in SCG, demonstrating excellent generalization ability to various tasks such as associative generation on painting, sketches, and ancient graffiti. Compared to the previous representative method ControlNet, our proposed approach not only demonstrates superior robustness in more challenging high-noise scenarios but also possesses more promising scalability potential due to its self-supervised manner.

Title: Rethinking the Power of Timestamps for Robust Time Series Forecasting: A Global-Local Fusion Perspective

• Authors: Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Jianxin Liao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18696
• Pdf URL: https://arxiv.org/pdf/2409.18696
• Copy Paste: [[2409.18696]] Rethinking the Power of Timestamps for Robust Time Series Forecasting: A Global-Local Fusion Perspective(https://arxiv.org/abs/2409.18696)
• Keywords: robust
• Abstract: Time series forecasting has played a pivotal role across various industries, including finance, transportation, energy, healthcare, and climate. Due to the abundant seasonal information they contain, timestamps possess the potential to offer robust global guidance for forecasting techniques. However, existing works primarily focus on local observations, with timestamps being treated merely as an optional supplement that remains underutilized. When data gathered from the real world is polluted, the absence of global information will damage the robust prediction capability of these algorithms. To address these problems, we propose a novel framework named GLAFF. Within this framework, the timestamps are modeled individually to capture the global dependencies. Working as a plugin, GLAFF adaptively adjusts the combined weights for global and local information, enabling seamless collaboration with any time series forecasting backbone. Extensive experiments conducted on nine real-world datasets demonstrate that GLAFF significantly enhances the average performance of widely used mainstream forecasting models by 12.5%, surpassing the previous state-of-the-art method by 5.5%.

Title: Read Over the Lines: Attacking LLMs and Toxicity Detection Systems with ASCII Art to Mask Profanity

• Authors: Sergey Berezin, Reza Farahbakhsh, Noel Crespi
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18708
• Pdf URL: https://arxiv.org/pdf/2409.18708
• Copy Paste: [[2409.18708]] Read Over the Lines: Attacking LLMs and Toxicity Detection Systems with ASCII Art to Mask Profanity(https://arxiv.org/abs/2409.18708)
• Keywords: attack
• Abstract: We introduce a novel family of adversarial attacks that exploit the inability of language models to interpret ASCII art. To evaluate these attacks, we propose the ToxASCII benchmark and develop two custom ASCII art fonts: one leveraging special tokens and another using text-filled letter shapes. Our attacks achieve a perfect 1.0 Attack Success Rate across ten models, including OpenAI's o1-preview and LLaMA 3.1. Warning: this paper contains examples of toxic language used for research purposes.

Title: Adversarial Challenges in Network Intrusion Detection Systems: Research Insights and Future Prospects

• Authors: Sabrine Ennaji, Fabio De Gaspari, Dorjan Hitaj, Alicia K/Bidi, Luigi V. Mancini
• Subjects: cs.CR, cs.ET, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.18736
• Pdf URL: https://arxiv.org/pdf/2409.18736
• Copy Paste: [[2409.18736]] Adversarial Challenges in Network Intrusion Detection Systems: Research Insights and Future Prospects(https://arxiv.org/abs/2409.18736)
• Keywords: security, defense, attack, robust
• Abstract: Machine learning has brought significant advances in cybersecurity, particularly in the area of intrusion detection systems. This improvements can be mostly attributed to the ability of machine learning algorithms to identify complex relations between features in the data and to generalize well to unseen samples. Deep neural networks in particular contributed to this progress by enabling the analysis of large amounts of training data, significantly enhancing detection performance. However, machine learning models are vulnerable to adversarial attacks: manipulations of input data designed to mislead the models into making incorrect predictions. While much attention has been given to adversarial threats in unstructured data such as text and images, their effectiveness in structured data such as network traffic has not been as thoroughly explored. This survey seeks to fill this gap by providing an critical review of machine learning-based Network Intrusion Detection Systems (NIDS) and a thorough analysis of their vulnerability to adversarial attacks. We critically review existing NIDS research, highlighting key trends, strengths, and limitations, and we identify gaps in understanding that require further exploration. We further discuss emerging challenges and offer insights for developing more robust and resilient NIDS models. In summary, this paper aims to enhance understanding of adversarial attacks and defenses in NIDS and guide future research in improving the robustness of machine learning models in cybersecurity applications.

Title: Cottention: Linear Transformers With Cosine Attention

• Authors: Gabriel Mongaras, Trevor Dohm, Eric C. Larson
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18747
• Pdf URL: https://arxiv.org/pdf/2409.18747
• Copy Paste: [[2409.18747]] Cottention: Linear Transformers With Cosine Attention(https://arxiv.org/abs/2409.18747)
• Keywords: transformer
• Abstract: Attention mechanisms, particularly softmax attention, have been instrumental in the success of transformer-based models such as GPT. However, the quadratic memory complexity of softmax attention with respect to sequence length poses significant challenges for processing longer sequences. We introduce Cottention, a novel attention mechanism that replaces the softmax operation with cosine similarity. By leveraging the properties of cosine similarity and rearranging the attention equation, Cottention achieves native linear memory complexity with respect to sequence length, making it inherently more memory-efficient than softmax attention. We demonstrate that Cottention can be reformulated as a recurrent neural network (RNN) with a finite hidden state, allowing for constant memory usage during inference. We evaluate Cottention on both the bidirectional BERT and causal GPT tasks, demonstrating comparable performance to softmax attention while significantly reducing memory requirements. To ensure efficient computation, we develop a custom CUDA kernel for Cottention. Our results show that Cottention is a promising alternative to softmax attention, enabling the processing of longer sequences without sacrificing performance, due to its native linear memory complexity and ability to maintain a constant memory footprint during inference.

Title: Enhancing Explainability in Multimodal Large Language Models Using Ontological Context

• Authors: Jihen Amara, Birgitta König-Ries, Sheeba Samuel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18753
• Pdf URL: https://arxiv.org/pdf/2409.18753
• Copy Paste: [[2409.18753]] Enhancing Explainability in Multimodal Large Language Models Using Ontological Context(https://arxiv.org/abs/2409.18753)
• Keywords: explainability, large language model
• Abstract: Recently, there has been a growing interest in Multimodal Large Language Models (MLLMs) due to their remarkable potential in various tasks integrating different modalities, such as image and text, as well as applications such as image captioning and visual question answering. However, such models still face challenges in accurately captioning and interpreting specific visual concepts and classes, particularly in domain-specific applications. We argue that integrating domain knowledge in the form of an ontology can significantly address these issues. In this work, as a proof of concept, we propose a new framework that combines ontology with MLLMs to classify images of plant diseases. Our method uses concepts about plant diseases from an existing disease ontology to query MLLMs and extract relevant visual concepts from images. Then, we use the reasoning capabilities of the ontology to classify the disease according to the identified concepts. Ensuring that the model accurately uses the concepts describing the disease is crucial in domain-specific applications. By employing an ontology, we can assist in verifying this alignment. Additionally, using the ontology's inference capabilities increases transparency, explainability, and trust in the decision-making process while serving as a judge by checking if the annotations of the concepts by MLLMs are aligned with those in the ontology and displaying the rationales behind their errors. Our framework offers a new direction for synergizing ontologies and MLLMs, supported by an empirical study using different well-known MLLMs.

Title: State-of-the-Art Periorbital Distance Prediction and Disease Classification Using Periorbital Features

• Authors: George R. Nahass, Ghasem Yazdanpanah, Madison Cheung, Alex Palacios, Jeffery Peterson, Kevin Heinze, Sasha Hubschman, Chad A. Purnell, Pete Setabutr, Ann Q. Tran, Darvin Yi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18769
• Pdf URL: https://arxiv.org/pdf/2409.18769
• Copy Paste: [[2409.18769]] State-of-the-Art Periorbital Distance Prediction and Disease Classification Using Periorbital Features(https://arxiv.org/abs/2409.18769)
• Keywords: robust, segmentation
• Abstract: Periorbital distances and features around the eyes and lids hold valuable information for disease quantification and monitoring of surgical and medical intervention. These distances are commonly measured manually, a process that is both subjective and highly time-consuming. Here, we set out to developed three deep-learning methods for segmentation and periorbital distance prediction, and also evaluate the utility of periorbital distances for disease classification. The MAE of our deep learning predicted distances was less than or very close to the error observed between trained human annotators. We compared our models to the current state-of-the-art (SOTA) method for periorbital distance prediction and found that our methods outperformed SOTA on all of our datasets on all but one periorbital measurement. We also show that robust segmentation can be achieved on diseased eyes using models trained on open-source, healthy eyes, and that periorbital distances have can be used as high-quality features in downstream classification models. Leveraging segmentation networks as intermediary steps in classification has broad implications for increasing the generalizability of classification models in ophthalmic plastic and craniofacial surgery by avoiding the out-of-distribution problem observed in traditional convolutional neural networks.

Title: HardCore Generation: Generating Hard UNSAT Problems for Data Augmentation

• Authors: Joseph Cotnareanu, Zhanguang Zhang, Hui-Ling Zhen, Yingxue Zhang, Mark Coates
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18778
• Pdf URL: https://arxiv.org/pdf/2409.18778
• Copy Paste: [[2409.18778]] HardCore Generation: Generating Hard UNSAT Problems for Data Augmentation(https://arxiv.org/abs/2409.18778)
• Keywords: generative
• Abstract: Efficiently determining the satisfiability of a boolean equation -- known as the SAT problem for brevity -- is crucial in various industrial problems. Recently, the advent of deep learning methods has introduced significant potential for enhancing SAT solving. However, a major barrier to the advancement of this field has been the scarcity of large, realistic datasets. The majority of current public datasets are either randomly generated or extremely limited, containing only a few examples from unrelated problem families. These datasets are inadequate for meaningful training of deep learning methods. In light of this, researchers have started exploring generative techniques to create data that more accurately reflect SAT problems encountered in practical situations. These methods have so far suffered from either the inability to produce challenging SAT problems or time-scalability obstacles. In this paper we address both by identifying and manipulating the key contributors to a problem's ``hardness'', known as cores. Although some previous work has addressed cores, the time costs are unacceptably high due to the expense of traditional heuristic core detection techniques. We introduce a fast core detection procedure that uses a graph neural network. Our empirical results demonstrate that we can efficiently generate problems that remain hard to solve and retain key attributes of the original example problems. We show via experiment that the generated synthetic SAT problems can be used in a data augmentation setting to provide improved prediction of solver runtimes.

Title: A Survey on the Honesty of Large Language Models

• Authors: Siheng Li, Cheng Yang, Taiqiang Wu, Chufan Shi, Yuji Zhang, Xinyu Zhu, Zesen Cheng, Deng Cai, Mo Yu, Lemao Liu, Jie Zhou, Yujiu Yang, Ngai Wong, Xixin Wu, Wai Lam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18786
• Pdf URL: https://arxiv.org/pdf/2409.18786
• Copy Paste: [[2409.18786]] A Survey on the Honesty of Large Language Models(https://arxiv.org/abs/2409.18786)
• Keywords: large language model
• Abstract: Honesty is a fundamental principle for aligning large language models (LLMs) with human values, requiring these models to recognize what they know and don't know and be able to faithfully express their knowledge. Despite promising, current LLMs still exhibit significant dishonest behaviors, such as confidently presenting wrong answers or failing to express what they know. In addition, research on the honesty of LLMs also faces challenges, including varying definitions of honesty, difficulties in distinguishing between known and unknown knowledge, and a lack of comprehensive understanding of related research. To address these issues, we provide a survey on the honesty of LLMs, covering its clarification, evaluation approaches, and strategies for improvement. Moreover, we offer insights for future research, aiming to inspire further exploration in this important area.

Title: Hierarchical Federated ADMM

• Authors: Seyed Mohammad Azimi-Abarghouyi, Nicola Bastianello, Karl H. Johansson, Viktoria Fodor
• Subjects: cs.LG, cs.AI, cs.DC, cs.IT, eess.SY
• Abstract URL: https://arxiv.org/abs/2409.18796
• Pdf URL: https://arxiv.org/pdf/2409.18796
• Copy Paste: [[2409.18796]] Hierarchical Federated ADMM(https://arxiv.org/abs/2409.18796)
• Keywords: privacy, federate
• Abstract: In this paper, we depart from the widely-used gradient descent-based hierarchical federated learning (FL) algorithms to develop a novel hierarchical FL framework based on the alternating direction method of multipliers (ADMM). Within this framework, we propose two novel FL algorithms, which both use ADMM in the top layer: one that employs ADMM in the lower layer and another that uses the conventional gradient descent-based approach. The proposed framework enhances privacy, and experiments demonstrate the superiority of the proposed algorithms compared to the conventional algorithms in terms of learning convergence and accuracy. Additionally, gradient descent on the lower layer performs well even if the number of local steps is very limited, while ADMM on both layers lead to better performance otherwise.

Title: LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis

• Authors: Hamed Babaei Giglou, Jennifer D'Souza, Sören Auer
• Subjects: cs.CL, cs.AI, cs.DL
• Abstract URL: https://arxiv.org/abs/2409.18812
• Pdf URL: https://arxiv.org/pdf/2409.18812
• Copy Paste: [[2409.18812]] LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis(https://arxiv.org/abs/2409.18812)
• Keywords: large language model
• Abstract: In response to the growing complexity and volume of scientific literature, this paper introduces the LLMs4Synthesis framework, designed to enhance the capabilities of Large Language Models (LLMs) in generating high-quality scientific syntheses. This framework addresses the need for rapid, coherent, and contextually rich integration of scientific insights, leveraging both open-source and proprietary LLMs. It also examines the effectiveness of LLMs in evaluating the integrity and reliability of these syntheses, alleviating inadequacies in current quantitative metrics. Our study contributes to this field by developing a novel methodology for processing scientific papers, defining new synthesis types, and establishing nine detailed quality criteria for evaluating syntheses. The integration of LLMs with reinforcement learning and AI feedback is proposed to optimize synthesis quality, ensuring alignment with established criteria. The LLMs4Synthesis framework and its components are made available, promising to enhance both the generation and evaluation processes in scientific research synthesis.

Title: EyeTrAES: Fine-grained, Low-Latency Eye Tracking via Adaptive Event Slicing

• Authors: Argha Sen, Nuwan Bandara, Ila Gokarn, Thivya Kandappu, Archan Misra
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.18813
• Pdf URL: https://arxiv.org/pdf/2409.18813
• Copy Paste: [[2409.18813]] EyeTrAES: Fine-grained, Low-Latency Eye Tracking via Adaptive Event Slicing(https://arxiv.org/abs/2409.18813)
• Keywords: robust, biometric, segmentation
• Abstract: Eye-tracking technology has gained significant attention in recent years due to its wide range of applications in human-computer interaction, virtual and augmented reality, and wearable health. Traditional RGB camera-based eye-tracking systems often struggle with poor temporal resolution and computational constraints, limiting their effectiveness in capturing rapid eye movements. To address these limitations, we propose EyeTrAES, a novel approach using neuromorphic event cameras for high-fidelity tracking of natural pupillary movement that shows significant kinematic variance. One of EyeTrAES's highlights is the use of a novel adaptive windowing/slicing algorithm that ensures just the right amount of descriptive asynchronous event data accumulation within an event frame, across a wide range of eye movement patterns. EyeTrAES then applies lightweight image processing functions over accumulated event frames from just a single eye to perform pupil segmentation and tracking. We show that these methods boost pupil tracking fidelity by 6+%, achieving IoU~=92%, while incurring at least 3x lower latency than competing pure event-based eye tracking alternatives [38]. We additionally demonstrate that the microscopic pupillary motion captured by EyeTrAES exhibits distinctive variations across individuals and can thus serve as a biometric fingerprint. For robust user authentication, we train a lightweight per-user Random Forest classifier using a novel feature vector of short-term pupillary kinematics, comprising a sliding window of pupil (location, velocity, acceleration) triples. Experimental studies with two different datasets demonstrate that the EyeTrAES-based authentication technique can simultaneously achieve high authentication accuracy (~=0.82) and low processing latency (~=12ms), and significantly outperform multiple state-of-the-art competitive baselines.

Title: Local Transcription Models in Home Care Nursing in Switzerland: an Interdisciplinary Case Study

• Authors: Jeremy Kramer, Tetiana Kravchenko, Beatrice Kaufmann, Friederike J.S. Thilo, Mascha Kurpicz-Briki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18819
• Pdf URL: https://arxiv.org/pdf/2409.18819
• Copy Paste: [[2409.18819]] Local Transcription Models in Home Care Nursing in Switzerland: an Interdisciplinary Case Study(https://arxiv.org/abs/2409.18819)
• Keywords: privacy
• Abstract: Latest advances in the field of natural language processing (NLP) enable new use cases for different domains, including the medical sector. In particular, transcription can be used to support automation in the nursing documentation process and give nurses more time to interact with the patients. However, different challenges including (a) data privacy, (b) local languages and dialects, and (c) domain-specific vocabulary need to be addressed. In this case study, we investigate the case of home care nursing documentation in Switzerland. We assessed different transcription tools and models, and conducted several experiments with OpenAI Whisper, involving different variations of German (i.e., dialects, foreign accent) and manually curated example texts by a domain expert of home care nursing. Our results indicate that even the used out-of-the-box model performs sufficiently well to be a good starting point for future research in the field.

Title: MinerU: An Open-Source Solution for Precise Document Content Extraction

• Authors: Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, Conghui He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18839
• Pdf URL: https://arxiv.org/pdf/2409.18839
• Copy Paste: [[2409.18839]] MinerU: An Open-Source Solution for Precise Document Content Extraction(https://arxiv.org/abs/2409.18839)
• Keywords: extraction
• Abstract: Document content analysis has been a crucial research area in computer vision. Despite significant advancements in methods such as OCR, layout detection, and formula recognition, existing open-source solutions struggle to consistently deliver high-quality content extraction due to the diversity in document types and content. To address these challenges, we present MinerU, an open-source solution for high-precision document content extraction. MinerU leverages the sophisticated PDF-Extract-Kit models to extract content from diverse documents effectively and employs finely-tuned preprocessing and postprocessing rules to ensure the accuracy of the final results. Experimental results demonstrate that MinerU consistently achieves high performance across various document types, significantly enhancing the quality and consistency of content extraction. The MinerU open-source project is available at this https URL.

Title: Predicting and analyzing memorization within fine-tuned Large Language Models

• Authors: Jérémie Dentan, Davide Buscaldi, Aymen Shabou, Sonia Vanier
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18858
• Pdf URL: https://arxiv.org/pdf/2409.18858
• Copy Paste: [[2409.18858]] Predicting and analyzing memorization within fine-tuned Large Language Models(https://arxiv.org/abs/2409.18858)
• Keywords: protect, large language model
• Abstract: Large Language Models have received significant attention due to their abilities to solve a wide range of complex tasks. However these models memorize a significant proportion of their training data, posing a serious threat when disclosed at inference time. To mitigate this unintended memorization, it is crucial to understand what elements are memorized and why. Most existing works provide a posteriori explanations, which has a limited interest in practice. To address this gap, we propose a new approach based on sliced mutual information to detect memorized samples a priori, in a classification setting. It is efficient from the early stages of training, and is readily adaptable to practical scenarios. Our method is supported by new theoretical results that we demonstrate, and requires a low computational budget. We obtain strong empirical results, paving the way for systematic inspection and protection of these vulnerable samples before memorization happens.

Title: Challenges of Generating Structurally Diverse Graphs

• Authors: Fedor Velikonivtsev, Mikhail Mironov, Liudmila Prokhorenkova
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18859
• Pdf URL: https://arxiv.org/pdf/2409.18859
• Copy Paste: [[2409.18859]] Challenges of Generating Structurally Diverse Graphs(https://arxiv.org/abs/2409.18859)
• Keywords: generative
• Abstract: For many graph-related problems, it can be essential to have a set of structurally diverse graphs. For instance, such graphs can be used for testing graph algorithms or their neural approximations. However, to the best of our knowledge, the problem of generating structurally diverse graphs has not been explored in the literature. In this paper, we fill this gap. First, we discuss how to define diversity for a set of graphs, why this task is non-trivial, and how one can choose a proper diversity measure. Then, for a given diversity measure, we propose and compare several algorithms optimizing it: we consider approaches based on standard random graph models, local graph optimization, genetic algorithms, and neural generative models. We show that it is possible to significantly improve diversity over basic random graph generators. Additionally, our analysis of generated graphs allows us to better understand the properties of graph distances: depending on which diversity measure is used for optimization, the obtained graphs may possess very different structural properties which gives insights about the sensitivity of the graph distance underlying the diversity measure.

Title: Emu3: Next-Token Prediction is All You Need

• Authors: Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, Zhongyuan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18869
• Pdf URL: https://arxiv.org/pdf/2409.18869
• Copy Paste: [[2409.18869]] Emu3: Next-Token Prediction is All You Need(https://arxiv.org/abs/2409.18869)
• Keywords: diffusion, transformer
• Abstract: While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.

Title: CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting

• Authors: Josef Koumar, Karel Hynek, Tomáš Čejka, Pavel Šiška
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18874
• Pdf URL: https://arxiv.org/pdf/2409.18874
• Copy Paste: [[2409.18874]] CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting(https://arxiv.org/abs/2409.18874)
• Keywords: security
• Abstract: Anomaly detection in network traffic is crucial for maintaining the security of computer networks and identifying malicious activities. One of the primary approaches to anomaly detection are methods based on forecasting. Nevertheless, extensive real-world network datasets for forecasting and anomaly detection techniques are missing, potentially causing performance overestimation of anomaly detection algorithms. This manuscript addresses this gap by introducing a dataset comprising time series data of network entities' behavior, collected from the CESNET3 network. The dataset was created from 40 weeks of network traffic of 275 thousand active IP addresses. The ISP origin of the presented data ensures a high level of variability among network entities, which forms a unique and authentic challenge for forecasting and anomaly detection models. It provides valuable insights into the practical deployment of forecast-based anomaly detection approaches.

Title: CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition

• Authors: Zhonglin Sun, Siyang Song, Ioannis Patras, Georgios Tzimiropoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18876
• Pdf URL: https://arxiv.org/pdf/2409.18876
• Copy Paste: [[2409.18876]] CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition(https://arxiv.org/abs/2409.18876)
• Keywords: privacy, diffusion, generative
• Abstract: Privacy issue is a main concern in developing face recognition techniques. Although synthetic face images can partially mitigate potential legal risks while maintaining effective face recognition (FR) performance, FR models trained by face images synthesized by existing generative approaches frequently suffer from performance degradation problems due to the insufficient discriminative quality of these synthesized samples. In this paper, we systematically investigate what contributes to solid face recognition model training, and reveal that face images with certain degree of similarities to their identity centers show great effectiveness in the performance of trained FR models. Inspired by this, we propose a novel diffusion-based approach (namely Center-based Semi-hard Synthetic Face Generation (CemiFace)) which produces facial samples with various levels of similarity to the subject center, thus allowing to generate face datasets containing effective discriminative samples for training face recognition. Experimental results show that with a modest degree of similarity, training on the generated dataset can produce competitive performance compared to previous generation methods.

Title: Explainable Artifacts for Synthetic Western Blot Source Attribution

• Authors: João Phillipe Cardenuto, Sara Mandelli, Daniel Moreira, Paolo Bestagini, Edward Delp, Anderson Rocha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18881
• Pdf URL: https://arxiv.org/pdf/2409.18881
• Copy Paste: [[2409.18881]] Explainable Artifacts for Synthetic Western Blot Source Attribution(https://arxiv.org/abs/2409.18881)
• Keywords: diffusion, generative
• Abstract: Recent advancements in artificial intelligence have enabled generative models to produce synthetic scientific images that are indistinguishable from pristine ones, posing a challenge even for expert scientists habituated to working with such content. When exploited by organizations known as paper mills, which systematically generate fraudulent articles, these technologies can significantly contribute to the spread of misinformation about ungrounded science, potentially undermining trust in scientific research. While previous studies have explored black-box solutions, such as Convolutional Neural Networks, for identifying synthetic content, only some have addressed the challenge of generalizing across different models and providing insight into the artifacts in synthetic images that inform the detection process. This study aims to identify explainable artifacts generated by state-of-the-art generative models (e.g., Generative Adversarial Networks and Diffusion Models) and leverage them for open-set identification and source attribution (i.e., pointing to the model that created the image).

Title: IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation

• Authors: Fan Lin, Shuyi Xie, Yong Dai, Wenlin Yao, Tianjiao Lang, Zishan Xu, Zhichao Hu, Xiao Xiao, Yuhong Liu, Yu Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18892
• Pdf URL: https://arxiv.org/pdf/2409.18892
• Copy Paste: [[2409.18892]] IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation(https://arxiv.org/abs/2409.18892)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) grow increasingly adept at managing complex tasks, the evaluation set must keep pace with these advancements to ensure it remains sufficiently discriminative. Item Discrimination (ID) theory, which is widely used in educational assessment, measures the ability of individual test items to differentiate between high and low performers. Inspired by this theory, we propose an ID-induced prompt synthesis framework for evaluating LLMs to ensure the evaluation set can continually update and refine according to model abilities. Our data synthesis framework prioritizes both breadth and specificity. It can generate prompts that comprehensively evaluate the capabilities of LLMs while revealing meaningful performance differences between models, allowing for effective discrimination of their relative strengths and weaknesses across various tasks and domains. To produce high-quality data, we incorporate a self-correct mechanism into our generalization framework, and develop two models to predict prompt discrimination and difficulty score to facilitate our data synthesis framework, contributing valuable tools to evaluation data synthesis research. We apply our generated data to evaluate five SOTA models. Our data achieves an average score of 51.92, accompanied by a variance of 10.06. By contrast, previous works (i.e., SELF-INSTRUCT and WizardLM) obtain an average score exceeding 67, with a variance below 3.2. The results demonstrate that the data generated by our framework is more challenging and discriminative compared to previous works. We will release a dataset of over 3,000 carefully crafted prompts to facilitate evaluation research of LLMs.

Title: Multi-Source Hard and Soft Information Fusion Approach for Accurate Cryptocurrency Price Movement Prediction

• Authors: Saeed Mohammadi Dashtaki, Mehdi Hosseini Chagahi, Behzad Moshiri, Md. Jalil Piran
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18895
• Pdf URL: https://arxiv.org/pdf/2409.18895
• Copy Paste: [[2409.18895]] Multi-Source Hard and Soft Information Fusion Approach for Accurate Cryptocurrency Price Movement Prediction(https://arxiv.org/abs/2409.18895)
• Keywords: transformer
• Abstract: One of the most important challenges in the financial and cryptocurrency field is accurately predicting cryptocurrency price trends. Leveraging artificial intelligence (AI) is beneficial in addressing this challenge. Cryptocurrency markets, marked by substantial growth and volatility, attract investors and scholars keen on deciphering and forecasting cryptocurrency price movements. The vast and diverse array of data available for such predictions increases the complexity of the task. In our study, we introduce a novel approach termed hard and soft information fusion (HSIF) to enhance the accuracy of cryptocurrency price movement forecasts. The hard information component of our approach encompasses historical price records alongside technical indicators. Complementing this, the soft data component extracts from X (formerly Twitter), encompassing news headlines and tweets about the cryptocurrency. To use this data, we use the Bidirectional Encoder Representations from Transformers (BERT)-based sentiment analysis method, financial BERT (FinBERT), which performs best. Finally, our model feeds on the information set including processed hard and soft data. We employ the bidirectional long short-term memory (BiLSTM) model because processing information in both forward and backward directions can capture long-term dependencies in sequential information. Our empirical findings emphasize the superiority of the HSIF approach over models dependent on single-source data by testing on Bitcoin-related data. By fusing hard and soft information on Bitcoin dataset, our model has about 96.8\% accuracy in predicting price movement. Incorporating information enables our model to grasp the influence of social sentiment on price fluctuations, thereby supplementing the technical analysis-based predictions derived from hard information.

Title: Detecting Dataset Abuse in Fine-Tuning Stable Diffusion Models for Text-to-Image Synthesis

• Authors: Songrui Wang, Yubo Zhu, Wei Tong, Sheng Zhong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18897
• Pdf URL: https://arxiv.org/pdf/2409.18897
• Copy Paste: [[2409.18897]] Detecting Dataset Abuse in Fine-Tuning Stable Diffusion Models for Text-to-Image Synthesis(https://arxiv.org/abs/2409.18897)
• Keywords: robust, watermark, diffusion, generative
• Abstract: Text-to-image synthesis has become highly popular for generating realistic and stylized images, often requiring fine-tuning generative models with domain-specific datasets for specialized tasks. However, these valuable datasets face risks of unauthorized usage and unapproved sharing, compromising the rights of the owners. In this paper, we address the issue of dataset abuse during the fine-tuning of Stable Diffusion models for text-to-image synthesis. We present a dataset watermarking framework designed to detect unauthorized usage and trace data leaks. The framework employs two key strategies across multiple watermarking schemes and is effective for large-scale dataset authorization. Extensive experiments demonstrate the framework's effectiveness, minimal impact on the dataset (only 2% of the data required to be modified for high detection accuracy), and ability to trace data leaks. Our results also highlight the robustness and transferability of the framework, proving its practical applicability in detecting dataset abuse.

Title: Unsupervised Low-light Image Enhancement with Lookup Tables and Diffusion Priors

• Authors: Yunlong Lin, Zhenqi Fu, Kairun Wen, Tian Ye, Sixiang Chen, Ge Meng, Yingying Wang, Yue Huang, Xiaotong Tu, Xinghao Ding
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.18899
• Pdf URL: https://arxiv.org/pdf/2409.18899
• Copy Paste: [[2409.18899]] Unsupervised Low-light Image Enhancement with Lookup Tables and Diffusion Priors(https://arxiv.org/abs/2409.18899)
• Keywords: diffusion
• Abstract: Low-light image enhancement (LIE) aims at precisely and efficiently recovering an image degraded in poor illumination environments. Recent advanced LIE techniques are using deep neural networks, which require lots of low-normal light image pairs, network parameters, and computational resources. As a result, their practicality is limited. In this work, we devise a novel unsupervised LIE framework based on diffusion priors and lookup tables (DPLUT) to achieve efficient low-light image recovery. The proposed approach comprises two critical components: a light adjustment lookup table (LLUT) and a noise suppression lookup table (NLUT). LLUT is optimized with a set of unsupervised losses. It aims at predicting pixel-wise curve parameters for the dynamic range adjustment of a specific image. NLUT is designed to remove the amplified noise after the light brightens. As diffusion models are sensitive to noise, diffusion priors are introduced to achieve high-performance noise suppression. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods in terms of visual quality and efficiency.

Title: In-depth Analysis of Privacy Threats in Federated Learning for Medical Data

• Authors: Badhan Chandra Das, M. Hadi Amini, Yanzhao Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18907
• Pdf URL: https://arxiv.org/pdf/2409.18907
• Copy Paste: [[2409.18907]] In-depth Analysis of Privacy Threats in Federated Learning for Medical Data(https://arxiv.org/abs/2409.18907)
• Keywords: privacy, protect, defense, attack, federate
• Abstract: Federated learning is emerging as a promising machine learning technique in the medical field for analyzing medical images, as it is considered an effective method to safeguard sensitive patient data and comply with privacy regulations. However, recent studies have revealed that the default settings of federated learning may inadvertently expose private training data to privacy attacks. Thus, the intensity of such privacy risks and potential mitigation strategies in the medical domain remain unclear. In this paper, we make three original contributions to privacy risk analysis and mitigation in federated learning for medical data. First, we propose a holistic framework, MedPFL, for analyzing privacy risks in processing medical data in the federated learning environment and developing effective mitigation strategies for protecting privacy. Second, through our empirical analysis, we demonstrate the severe privacy risks in federated learning to process medical images, where adversaries can accurately reconstruct private medical images by performing privacy attacks. Third, we illustrate that the prevalent defense mechanism of adding random noises may not always be effective in protecting medical images against privacy attacks in federated learning, which poses unique and pressing challenges related to protecting the privacy of medical data. Furthermore, the paper discusses several unique research questions related to the privacy protection of medical data in the federated learning environment. We conduct extensive experiments on several benchmark medical image datasets to analyze and mitigate the privacy risks associated with federated learning for medical data.

Title: Soft Measures for Extracting Causal Collective Intelligence

• Authors: Maryam Berijanian, Spencer Dork, Kuldeep Singh, Michael Riley Millikan, Ashlin Riggs, Aadarsh Swaminathan, Sarah L. Gibbs, Scott E. Friedman, Nathan Brugnone
• Subjects: cs.CL, cs.AI, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.18911
• Pdf URL: https://arxiv.org/pdf/2409.18911
• Copy Paste: [[2409.18911]] Soft Measures for Extracting Causal Collective Intelligence(https://arxiv.org/abs/2409.18911)
• Keywords: extraction, large language model
• Abstract: Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.

Title: A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needs

• Authors: Yan Sun, Li Shen, Dacheng Tao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18915
• Pdf URL: https://arxiv.org/pdf/2409.18915
• Copy Paste: [[2409.18915]] A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needs(https://arxiv.org/abs/2409.18915)
• Keywords: security, privacy, federate
• Abstract: As a popular paradigm for juggling data privacy and collaborative training, federated learning (FL) is flourishing to distributively process the large scale of heterogeneous datasets on edged clients. Due to bandwidth limitations and security considerations, it ingeniously splits the original problem into multiple subproblems to be solved in parallel, which empowers primal dual solutions to great application values in FL. In this paper, we review the recent development of classical federated primal dual methods and point out a serious common defect of such methods in non-convex scenarios, which we say is a "dual drift" caused by dual hysteresis of those longstanding inactive clients under partial participation training. To further address this problem, we propose a novel Aligned Federated Primal Dual (A-FedPD) method, which constructs virtual dual updates to align global consensus and local dual variables for those protracted unparticipated local clients. Meanwhile, we provide a comprehensive analysis of the optimization and generalization efficiency for the A-FedPD method on smooth non-convex objectives, which confirms its high efficiency and practicality. Extensive experiments are conducted on several classical FL setups to validate the effectiveness of our proposed method.

Title: Cluster-BPI: Efficient Fine-Grain Blind Power Identification for Defending against Hardware Thermal Trojans in Multicore SoCs

• Authors: Mohamed R. Elshamy, Mehdi Elahi, Ahmad Patooghy, Abdel-Hameed A. Badawy
• Subjects: cs.CR, cs.PF, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.18921
• Pdf URL: https://arxiv.org/pdf/2409.18921
• Copy Paste: [[2409.18921]] Cluster-BPI: Efficient Fine-Grain Blind Power Identification for Defending against Hardware Thermal Trojans in Multicore SoCs(https://arxiv.org/abs/2409.18921)
• Keywords: security, attack, robust
• Abstract: Modern multicore System-on-Chips (SoCs) feature hardware monitoring mechanisms that measure total power consumption. However, these aggregate measurements are often insufficient for fine-grained thermal and power management. This paper presents an enhanced Clustering Blind Power Identification (ICBPI) approach, designed to improve the sensitivity and robustness of the traditional Blind Power Identification (BPI) method. BPI estimates the power consumption of individual cores and models the thermal behavior of an SoC using only thermal sensor data and total power measurements. The proposed ICBPI approach refines BPI's initialization process, particularly improving the non-negative matrix factorization (NNMF) step, which is critical to the accuracy of BPI. ICBPI introduces density-based spatial clustering of applications with noise (DBSCAN) to better align temperature and power consumption data, thereby providing more accurate power consumption estimates. We validate the ICBPI method through two key tasks. The first task evaluates power estimation accuracy across four different multicore architectures, including a heterogeneous processor. Results show that ICBPI significantly enhances accuracy, reducing error rates by 77.56% compared to the original BPI and by 68.44% compared to the state-of-the-art BPISS method. The second task focuses on improving the detection and localization of malicious thermal sensor attacks in heterogeneous processors. The results demonstrate that ICBPI enhances the security and robustness of multicore SoCs against such attacks.

Title: AIPatient: Simulating Patients with EHRs and LLM Powered Agentic Workflow

• Authors: Huizi Yu, Jiayan Zhou, Lingyao Li, Shan Chen, Jack Gallifant, Anye Shi, Xiang Li, Wenyue Hua, Mingyu Jin, Guang Chen, Yang Zhou, Zhao Li, Trisha Gupte, Ming-Li Chen, Zahra Azizi, Yongfeng Zhang, Themistocles L. Assimes, Xin Ma, Danielle S. Bitterman, Lin Lu, Lizhou Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18924
• Pdf URL: https://arxiv.org/pdf/2409.18924
• Copy Paste: [[2409.18924]] AIPatient: Simulating Patients with EHRs and LLM Powered Agentic Workflow(https://arxiv.org/abs/2409.18924)
• Keywords: robust, diffusion, large language model
• Abstract: Simulated patient systems play a crucial role in modern medical education and research, providing safe, integrative learning environments and enabling clinical decision-making simulations. Large Language Models (LLM) could advance simulated patient systems by replicating medical conditions and patient-doctor interactions with high fidelity and low cost. However, ensuring the effectiveness and trustworthiness of these systems remains a challenge, as they require a large, diverse, and precise patient knowledgebase, along with a robust and stable knowledge diffusion to users. Here, we developed AIPatient, an advanced simulated patient system with AIPatient Knowledge Graph (AIPatient KG) as the input and the Reasoning Retrieval-Augmented Generation (Reasoning RAG) agentic workflow as the generation backbone. AIPatient KG samples data from Electronic Health Records (EHRs) in the Medical Information Mart for Intensive Care (MIMIC)-III database, producing a clinically diverse and relevant cohort of 1,495 patients with high knowledgebase validity (F1 0.89). Reasoning RAG leverages six LLM powered agents spanning tasks including retrieval, KG query generation, abstraction, checker, rewrite, and summarization. This agentic framework reaches an overall accuracy of 94.15% in EHR-based medical Question Answering (QA), outperforming benchmarks that use either no agent or only partial agent integration. Our system also presents high readability (median Flesch Reading Ease 77.23; median Flesch Kincaid Grade 5.6), robustness (ANOVA F-value 0.6126, p<0.1), and stability (ANOVA F-value 0.782, p<0.1). The promising performance of the AIPatient system highlights its potential to support a wide range of applications, including medical education, model evaluation, and system integration.

Title: ReviveDiff: A Universal Diffusion Model for Restoring Images in Adverse Weather Conditions

• Authors: Wenfeng Huang, Guoan Xu, Wenjing Jia, Stuart Perry, Guangwei Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18932
• Pdf URL: https://arxiv.org/pdf/2409.18932
• Copy Paste: [[2409.18932]] ReviveDiff: A Universal Diffusion Model for Restoring Images in Adverse Weather Conditions(https://arxiv.org/abs/2409.18932)
• Keywords: diffusion
• Abstract: Images captured in challenging environments--such as nighttime, foggy, rainy weather, and underwater--often suffer from significant degradation, resulting in a substantial loss of visual quality. Effective restoration of these degraded images is critical for the subsequent vision tasks. While many existing approaches have successfully incorporated specific priors for individual tasks, these tailored solutions limit their applicability to other degradations. In this work, we propose a universal network architecture, dubbed "ReviveDiff", which can address a wide range of degradations and bring images back to life by enhancing and restoring their quality. Our approach is inspired by the observation that, unlike degradation caused by movement or electronic issues, quality degradation under adverse conditions primarily stems from natural media (such as fog, water, and low luminance), which generally preserves the original structures of objects. To restore the quality of such images, we leveraged the latest advancements in diffusion models and developed ReviveDiff to restore image quality from both macro and micro levels across some key factors determining image quality, such as sharpness, distortion, noise level, dynamic range, and color accuracy. We rigorously evaluated ReviveDiff on seven benchmark datasets covering five types of degrading conditions: Rainy, Underwater, Low-light, Smoke, and Nighttime Hazy. Our experimental results demonstrate that ReviveDiff outperforms the state-of-the-art methods both quantitatively and visually.

Title: From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding

• Authors: Heqing Zou, Tianze Luo, Guiyang Xie, Victor (Xiao Jie)Zhang, Fengmao Lv, Guangcong Wang, Juanyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18938
• Pdf URL: https://arxiv.org/pdf/2409.18938
• Copy Paste: [[2409.18938]] From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding(https://arxiv.org/abs/2409.18938)
• Keywords: large language model
• Abstract: The integration of Large Language Models (LLMs) with visual encoders has recently shown promising performance in visual understanding tasks, leveraging their inherent capability to comprehend and generate human-like text for visual reasoning. Given the diverse nature of visual data, MultiModal Large Language Models (MM-LLMs) exhibit variations in model designing and training for understanding images, short videos, and long videos. Our paper focuses on the substantial differences and unique challenges posed by long video understanding compared to static image and short video understanding. Unlike static images, short videos encompass sequential frames with both spatial and within-event temporal information, while long videos consist of multiple events with between-event and long-term temporal information. In this survey, we aim to trace and summarize the advancements of MM-LLMs from image understanding to long video understanding. We review the differences among various visual understanding tasks and highlight the challenges in long video understanding, including more fine-grained spatiotemporal details, dynamic events, and long-term dependencies. We then provide a detailed summary of the advancements in MM-LLMs in terms of model design and training methodologies for understanding long videos. Finally, we compare the performance of existing MM-LLMs on video understanding benchmarks of various lengths and discuss potential future directions for MM-LLMs in long video understanding.

Title: Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models

• Authors: Jiaming Li, Lei Zhang, Yunshui Li, Ziqiang Liu, yuelin bai, Run Luo, Longze Chen, Min Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18943
• Pdf URL: https://arxiv.org/pdf/2409.18943
• Copy Paste: [[2409.18943]] Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models(https://arxiv.org/abs/2409.18943)
• Keywords: large language model
• Abstract: The instruction-following ability of large language models enables humans to interact with AI agents in a natural way. However, when required to generate responses of a specific length, large language models often struggle to meet users' needs due to their inherent difficulty in accurately perceiving numerical constraints. To explore the ability of large language models to control the length of generated responses, we propose the Target Length Generation Task (TLG) and design two metrics, Precise Match (PM) and Flexible Match (FM) to evaluate the model's performance in adhering to specified response lengths. Furthermore, we introduce a novel, model-agnostic approach called Ruler, which employs Meta Length Tokens (MLTs) to enhance the instruction-following ability of large language models under length-constrained instructions. Specifically, Ruler equips LLMs with the ability to generate responses of a specified length based on length constraints within the instructions. Moreover, Ruler can automatically generate appropriate MLT when length constraints are not explicitly provided, demonstrating excellent versatility and generalization. Comprehensive experiments show the effectiveness of Ruler across different LLMs on Target Length Generation Task, e.g., at All Level 27.97 average gain on PM, 29.57 average gain on FM. In addition, we conduct extensive ablation experiments to further substantiate the efficacy and generalization of Ruler. Our code and data is available at this https URL.

Title: LML: Language Model Learning a Dataset for Data-Augmented Prediction

• Authors: Praneeth Vadlapati
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18957
• Pdf URL: https://arxiv.org/pdf/2409.18957
• Copy Paste: [[2409.18957]] LML: Language Model Learning a Dataset for Data-Augmented Prediction(https://arxiv.org/abs/2409.18957)
• Keywords: interpretability, large language model
• Abstract: This paper introduces a new approach to using Large Language Models (LLMs) for classification tasks, which are typically handled using Machine Learning (ML) models. Unlike ML models that rely heavily on data cleaning and feature engineering, this method streamlines the process using LLMs. This paper proposes a new concept called "Language Model Learning (LML)" powered by a new method called "Data-Augmented Prediction (DAP)". The classification is performed by LLMs using a method similar to humans manually exploring and understanding the data and deciding classifications using data as a reference. Training data is summarized and evaluated to determine the features that lead to the classification of each label the most. In the process of DAP, the system uses the data summary to automatically create a query, which is used to retrieve relevant rows from the dataset. A classification is generated by the LLM using data summary and relevant rows, ensuring satisfactory accuracy even with complex data. Usage of data summary and similar data in DAP ensures context-aware decision-making. The proposed method uses the words "Act as an Explainable Machine Learning Model" in the prompt to enhance the interpretability of the predictions by allowing users to review the logic behind each prediction. In some test cases, the system scored an accuracy above 90%, proving the effectiveness of the system and its potential to outperform conventional ML models in various scenarios. The code is available at this https URL

Title: $O(d/T)$ Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions

• Authors: Gen Li, Yuling Yan
• Subjects: cs.LG, cs.AI, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.18959
• Pdf URL: https://arxiv.org/pdf/2409.18959
• Copy Paste: [[2409.18959]] $O(d/T)$ Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions(https://arxiv.org/abs/2409.18959)
• Keywords: diffusion, generative
• Abstract: Score-based diffusion models, which generate new data by learning to reverse a diffusion process that perturbs data from the target distribution into noise, have achieved remarkable success across various generative tasks. Despite their superior empirical performance, existing theoretical guarantees are often constrained by stringent assumptions or suboptimal convergence rates. In this paper, we establish a fast convergence theory for a popular SDE-based sampler under minimal assumptions. Our analysis shows that, provided $\ell_{2}$-accurate estimates of the score functions, the total variation distance between the target and generated distributions is upper bounded by $O(d/T)$ (ignoring logarithmic factors), where $d$ is the data dimensionality and $T$ is the number of steps. This result holds for any target distribution with finite first-order moment. To our knowledge, this improves upon existing convergence theory for both the SDE-based sampler and another ODE-based sampler, while imposing minimal assumptions on the target data distribution and score estimates. This is achieved through a novel set of analytical tools that provides a fine-grained characterization of how the error propagates at each step of the reverse process.

Title: ProMerge: Prompt and Merge for Unsupervised Instance Segmentation

• Authors: Dylan Li, Gyungin Shin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18961
• Pdf URL: https://arxiv.org/pdf/2409.18961
• Copy Paste: [[2409.18961]] ProMerge: Prompt and Merge for Unsupervised Instance Segmentation(https://arxiv.org/abs/2409.18961)
• Keywords: segmentation
• Abstract: Unsupervised instance segmentation aims to segment distinct object instances in an image without relying on human-labeled data. This field has recently seen significant advancements, partly due to the strong local correspondences afforded by rich visual feature representations from self-supervised models (e.g., DINO). Recent state-of-the-art approaches use self-supervised features to represent images as graphs and solve a generalized eigenvalue system (i.e., normalized-cut) to generate foreground masks. While effective, this strategy is limited by its attendant computational demands, leading to slow inference speeds. In this paper, we propose Prompt and Merge (ProMerge), which leverages self-supervised visual features to obtain initial groupings of patches and applies a strategic merging to these segments, aided by a sophisticated background-based mask pruning technique. ProMerge not only yields competitive results but also offers a significant reduction in inference time compared to state-of-the-art normalized-cut-based approaches. Furthermore, when training an object detector using our mask predictions as pseudo-labels, the resulting detector surpasses the current leading unsupervised model on various challenging instance segmentation benchmarks.

Title: Exploring Token Pruning in Vision State Space Models

• Authors: Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zichong Meng, Hangyu Zheng, Xuan Shen, Stratis Ioannidis, Wei Niu, Pu Zhao, Yanzhi Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18962
• Pdf URL: https://arxiv.org/pdf/2409.18962
• Copy Paste: [[2409.18962]] Exploring Token Pruning in Vision State Space Models(https://arxiv.org/abs/2409.18962)
• Keywords: transformer
• Abstract: State Space Models (SSMs) have the advantage of keeping linear computational complexity compared to attention modules in transformers, and have been applied to vision tasks as a new type of powerful vision foundation model. Inspired by the observations that the final prediction in vision transformers (ViTs) is only based on a subset of most informative tokens, we take the novel step of enhancing the efficiency of SSM-based vision models through token-based pruning. However, direct applications of existing token pruning techniques designed for ViTs fail to deliver good performance, even with extensive fine-tuning. To address this issue, we revisit the unique computational characteristics of SSMs and discover that naive application disrupts the sequential token positions. This insight motivates us to design a novel and general token pruning method specifically for SSM-based vision models. We first introduce a pruning-aware hidden state alignment method to stabilize the neighborhood of remaining tokens for performance enhancement. Besides, based on our detailed analysis, we propose a token importance evaluation method adapted for SSM models, to guide the token pruning. With efficient implementation and practical acceleration methods, our method brings actual speedup. Extensive experiments demonstrate that our approach can achieve significant computation reduction with minimal impact on performance across different tasks. Notably, we achieve 81.7\% accuracy on ImageNet with a 41.6\% reduction in the FLOPs for pruned PlainMamba-L3. Furthermore, our work provides deeper insights into understanding the behavior of SSM-based vision models for future research.

Title: PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation

• Authors: Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, Shenlong Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18964
• Pdf URL: https://arxiv.org/pdf/2409.18964
• Copy Paste: [[2409.18964]] PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation(https://arxiv.org/abs/2409.18964)
• Keywords: diffusion, generative
• Abstract: We present PhysGen, a novel image-to-video generation method that converts a single image and an input condition (e.g., force and torque applied to an object in the image) to produce a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process, enabling plausible image-space dynamics. At the heart of our system are three core components: (i) an image understanding module that effectively captures the geometry, materials, and physical parameters of the image; (ii) an image-space dynamics simulation model that utilizes rigid-body physics and inferred parameters to simulate realistic behaviors; and (iii) an image-based rendering and refinement module that leverages generative video diffusion to produce realistic video footage featuring the simulated motion. The resulting videos are realistic in both physics and appearance and are even precisely controllable, showcasing superior results over existing data-driven image-to-video generation works through quantitative comparison and comprehensive user study. PhysGen's resulting videos can be used for various downstream applications, such as turning an image into a realistic animation or allowing users to interact with the image and create various dynamics. Project page: this https URL