2024-09-26

Title: Explaining Human Comparisons using Alignment-Importance Heatmaps

• Authors: Nhut Truong, Dario Pesenti, Uri Hasson
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16292
• Pdf URL: https://arxiv.org/pdf/2409.16292
• Copy Paste: [[2409.16292]] Explaining Human Comparisons using Alignment-Importance Heatmaps(https://arxiv.org/abs/2409.16292)
• Keywords: explainability
• Abstract: We present a computational explainability approach for human comparison tasks, using Alignment Importance Score (AIS) heatmaps derived from deep-vision models. The AIS reflects a feature-map's unique contribution to the alignment between Deep Neural Network's (DNN) representational geometry and that of humans. We first validate the AIS by showing that prediction of out-of-sample human similarity judgments is improved when constructing representations using only higher-scoring AIS feature maps identified from a training set. We then compute image-specific heatmaps that visually indicate the areas that correspond to feature-maps with higher AIS scores. These maps provide an intuitive explanation of which image areas are more important when it is compared to other images in a cohort. We observe a correspondence between these heatmaps and saliency maps produced by a gaze-prediction model. However, in some cases, meaningful differences emerge, as the dimensions relevant for comparison are not necessarily the most visually salient. To conclude, Alignment Importance improves prediction of human similarity judgments from DNN embeddings, and provides interpretable insights into the relevant information in image space.

Title: GenCAD: Image-Conditioned Computer-Aided Design Generation with Transformer-Based Contrastive Representation and Diffusion Priors

• Authors: Md Ferdous Alam, Faez Ahmed
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16294
• Pdf URL: https://arxiv.org/pdf/2409.16294
• Copy Paste: [[2409.16294]] GenCAD: Image-Conditioned Computer-Aided Design Generation with Transformer-Based Contrastive Representation and Diffusion Priors(https://arxiv.org/abs/2409.16294)
• Keywords: diffusion, transformer, generative
• Abstract: The creation of manufacturable and editable 3D shapes through Computer-Aided Design (CAD) remains a highly manual and time-consuming task, hampered by the complex topology of boundary representations of 3D solids and unintuitive design tools. This paper introduces GenCAD, a generative model that employs autoregressive transformers and latent diffusion models to transform image inputs into parametric CAD command sequences, resulting in editable 3D shape representations. GenCAD integrates an autoregressive transformer-based architecture with a contrastive learning framework, enhancing the generation of CAD programs from input images and providing a representation learning framework for multiple data modalities relevant to engineering designs. Extensive evaluations demonstrate that GenCAD significantly outperforms existing state-of-the-art methods in terms of the precision and modifiability of generated 3D shapes. Notably, GenCAD shows a marked improvement in the accuracy of 3D shape generation for long sequences, supporting its application in complex design tasks. Additionally, the contrastive embedding feature of GenCAD facilitates the retrieval of CAD models using image queries from databases which is a critical challenge within the CAD community. While most work in the 3D shape generation literature focuses on representations like meshes, voxels, or point clouds, practical engineering applications demand modifiability and the ability for multi-modal conditional generation. Our results provide a significant step forward in this direction, highlighting the potential of generative models to expedite the entire design-to-production pipeline and seamlessly integrate different design modalities.

Title: DeepScore: A Comprehensive Approach to Measuring Quality in AI-Generated Clinical Documentation

• Authors: Jon Oleson
• Subjects: cs.CL, cs.AI, stat.AP
• Abstract URL: https://arxiv.org/abs/2409.16307
• Pdf URL: https://arxiv.org/pdf/2409.16307
• Copy Paste: [[2409.16307]] DeepScore: A Comprehensive Approach to Measuring Quality in AI-Generated Clinical Documentation(https://arxiv.org/abs/2409.16307)
• Keywords: generative
• Abstract: Medical practitioners are rapidly adopting generative AI solutions for clinical documentation, leading to significant time savings and reduced stress. However, evaluating the quality of AI-generated documentation is a complex and ongoing challenge. This paper presents an overview of DeepScribe's methodologies for assessing and managing note quality, focusing on various metrics and the composite "DeepScore", an overall index of quality and accuracy. These methodologies aim to enhance the quality of patient care documentation through accountability and continuous improvement.

Title: Automated Spatio-Temporal Weather Modeling for Load Forecasting

• Authors: Julie Keisler (CRIStAL, EDF R\&D OSIRIS, EDF R\&D), Margaux Bregere (EDF R\&D, EDF R\&D OSIRIS, LPSM (UMR\_8001))
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.16326
• Pdf URL: https://arxiv.org/pdf/2409.16326
• Copy Paste: [[2409.16326]] Automated Spatio-Temporal Weather Modeling for Load Forecasting(https://arxiv.org/abs/2409.16326)
• Keywords: extraction
• Abstract: Electricity is difficult to store, except at prohibitive cost, and therefore the balance between generation and load must be maintained at all times. Electricity is traditionally managed by anticipating demand and intermittent production (wind, solar) and matching flexible production (hydro, nuclear, coal and gas). Accurate forecasting of electricity load and renewable production is therefore essential to ensure grid performance and stability. Both are highly dependent on meteorological variables (temperature, wind, sunshine). These dependencies are complex and difficult to model. On the one hand, spatial variations do not have a uniform impact because population, industry, and wind and solar farms are not evenly distributed across the territory. On the other hand, temporal variations can have delayed effects on load (due to the thermal inertia of buildings). With access to observations from different weather stations and simulated data from meteorological models, we believe that both phenomena can be modeled together. In today's state-of-the-art load forecasting models, the spatio-temporal modeling of the weather is fixed. In this work, we aim to take advantage of the automated representation and spatio-temporal feature extraction capabilities of deep neural networks to improve spatio-temporal weather modeling for load forecasting. We compare our deep learning-based methodology with the state-of-the-art on French national load. This methodology could also be fully adapted to forecasting renewable energy production.

Title: Exploring the traditional NMT model and Large Language Model for chat translation

• Authors: Jinlong Yang, Hengchao Shang, Daimeng Wei, Jiaxin Guo, Zongyao Li, Zhanglin Wu, Zhiqiang Rao, Shaojun Li, Yuhao Xie, Yuanchang Luo, Jiawei Zheng, Bin Wei, Hao Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16331
• Pdf URL: https://arxiv.org/pdf/2409.16331
• Copy Paste: [[2409.16331]] Exploring the traditional NMT model and Large Language Model for chat translation(https://arxiv.org/abs/2409.16331)
• Keywords: large language model
• Abstract: This paper describes the submissions of Huawei Translation Services Center(HW-TSC) to WMT24 chat translation shared task on English$\leftrightarrow$Germany (en-de) bidirection. The experiments involved fine-tuning models using chat data and exploring various strategies, including Minimum Bayesian Risk (MBR) decoding and self-training. The results show significant performance improvements in certain directions, with the MBR self-training method achieving the best results. The Large Language Model also discusses the challenges and potential avenues for further research in the field of chat translation.

Title: Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs

• Authors: Shadi Iskander, Nachshon Cohen, Zohar Karnin, Ori Shapira, Sofia Tolmach
• Subjects: cs.LG, cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.16341
• Pdf URL: https://arxiv.org/pdf/2409.16341
• Copy Paste: [[2409.16341]] Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs(https://arxiv.org/abs/2409.16341)
• Keywords: large language model
• Abstract: Training large language models (LLMs) for external tool usage is a rapidly expanding field, with recent research focusing on generating synthetic data to address the shortage of available data. However, the absence of systematic data quality checks poses complications for properly training and testing models. To that end, we propose two approaches for assessing the reliability of data for training LLMs to use external tools. The first approach uses intuitive, human-defined correctness criteria. The second approach uses a model-driven assessment with in-context evaluation. We conduct a thorough evaluation of data quality on two popular benchmarks, followed by an extrinsic evaluation that showcases the impact of data quality on model performance. Our results demonstrate that models trained on high-quality data outperform those trained on unvalidated data, even when trained with a smaller quantity of data. These findings empirically support the significance of assessing and ensuring the reliability of training data for tool-using LLMs.

Title: Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs

• Authors: Amartya Roy, Danush Khanna, Devanshu Mahapatra, Vasanthakumar, Avirup Das, Kripabandhu Ghosh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16371
• Pdf URL: https://arxiv.org/pdf/2409.16371
• Copy Paste: [[2409.16371]] Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs(https://arxiv.org/abs/2409.16371)
• Keywords: robust
• Abstract: This paper tackles the challenge of building robust and generalizable bias mitigation models for language. Recognizing the limitations of existing datasets, we introduce ANUBIS, a novel dataset with 1507 carefully curated sentence pairs encompassing nine social bias categories. We evaluate state-of-the-art models like T5, utilizing Supervised Fine-Tuning (SFT), Reinforcement Learning (PPO, DPO), and In-Context Learning (ICL) for effective bias mitigation. Our analysis focuses on multi-class social bias reduction, cross-dataset generalizability, and environmental impact of the trained models. ANUBIS and our findings offer valuable resources for building more equitable AI systems and contribute to the development of responsible and unbiased technologies with broad societal impact.

Title: Instance Segmentation of Reinforced Concrete Bridges with Synthetic Point Clouds

• Authors: Asad Ur Rahman, Vedhus Hoskere
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.16381
• Pdf URL: https://arxiv.org/pdf/2409.16381
• Copy Paste: [[2409.16381]] Instance Segmentation of Reinforced Concrete Bridges with Synthetic Point Clouds(https://arxiv.org/abs/2409.16381)
• Keywords: transformer, segmentation
• Abstract: The National Bridge Inspection Standards require detailed element-level bridge inspections. Traditionally, inspectors manually assign condition ratings by rating structural components based on damage, but this process is labor-intensive and time-consuming. Automating the element-level bridge inspection process can facilitate more comprehensive condition documentation to improve overall bridge management. While semantic segmentation of bridge point clouds has been studied, research on instance segmentation of bridge elements is limited, partly due to the lack of annotated datasets, and the difficulty in generalizing trained models. To address this, we propose a novel approach for generating synthetic data using three distinct methods. Our framework leverages the Mask3D transformer model, optimized with hyperparameter tuning and a novel occlusion technique. The model achieves state-of-the-art performance on real LiDAR and photogrammetry bridge point clouds, respectively, demonstrating the potential of the framework for automating element-level bridge inspections.

Title: Towards Synthetic Data Generation for Improved Pain Recognition in Videos under Patient Constraints

• Authors: Jonas Nasimzada, Jens Kleesiek, Ken Herrmann, Alina Roitberg, Constantin Seibold
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16382
• Pdf URL: https://arxiv.org/pdf/2409.16382
• Copy Paste: [[2409.16382]] Towards Synthetic Data Generation for Improved Pain Recognition in Videos under Patient Constraints(https://arxiv.org/abs/2409.16382)
• Keywords: privacy
• Abstract: Recognizing pain in video is crucial for improving patient-computer interaction systems, yet traditional data collection in this domain raises significant ethical and logistical challenges. This study introduces a novel approach that leverages synthetic data to enhance video-based pain recognition models, providing an ethical and scalable alternative. We present a pipeline that synthesizes realistic 3D facial models by capturing nuanced facial movements from a small participant pool, and mapping these onto diverse synthetic avatars. This process generates 8,600 synthetic faces, accurately reflecting genuine pain expressions from varied angles and perspectives. Utilizing advanced facial capture techniques, and leveraging public datasets like CelebV-HQ and FFHQ-UV for demographic diversity, our new synthetic dataset significantly enhances model training while ensuring privacy by anonymizing identities through facial replacements. Experimental results demonstrate that models trained on combinations of synthetic data paired with a small amount of real participants achieve superior performance in pain recognition, effectively bridging the gap between synthetic simulations and real-world applications. Our approach addresses data scarcity and ethical concerns, offering a new solution for pain detection and opening new avenues for research in privacy-preserving dataset generation. All resources are publicly available to encourage further innovation in this field.

Title: Chasing the Shadows: TTPs in Action to Attribute Advanced Persistent Threats

• Authors: Nanda Rani, Bikash Saha, Vikas Maurya, Sandeep Kumar Shukla
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16400
• Pdf URL: https://arxiv.org/pdf/2409.16400
• Copy Paste: [[2409.16400]] Chasing the Shadows: TTPs in Action to Attribute Advanced Persistent Threats(https://arxiv.org/abs/2409.16400)
• Keywords: attack
• Abstract: The current state of Advanced Persistent Threats (APT) attribution primarily relies on time-consuming manual processes. These include mapping incident artifacts onto threat attribution frameworks and employing expert reasoning to uncover the most likely responsible APT groups. This research aims to assist the threat analyst in the attribution process by presenting an attribution method named CAPTAIN (Comprehensive Advanced Persistent Threat AttrIbutioN). This novel APT attribution approach leverages the Tactics, Techniques, and Procedures (TTPs) employed by various APT groups in past attacks. CAPTAIN follows two significant development steps: baseline establishment and similarity measure for attack pattern matching. This method starts by maintaining a TTP database of APTs seen in past attacks as baseline behaviour of threat groups. The attribution process leverages the contextual information added by TTP sequences, which reflects the sequence of behaviours threat actors demonstrated during the attack on different kill-chain stages. Then, it compares the provided TTPs with established baseline to identify the most closely matching threat group. CAPTAIN introduces a novel similarity measure for APT group attack-pattern matching that calculates the similarity between TTP sequences. The proposed approach outperforms traditional similarity measures like Cosine, Euclidean, and Longest Common Subsequence (LCS) in performing attribution. Overall, CAPTAIN performs attribution with the precision of 61.36% (top-1) and 69.98% (top-2), surpassing the existing state-of-the-art attribution methods.

Title: Evaluating Blocking Biases in Entity Matching

• Authors: Mohammad Hossein Moslemi, Harini Balamurugan, Mostafa Milani
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2409.16410
• Pdf URL: https://arxiv.org/pdf/2409.16410
• Copy Paste: [[2409.16410]] Evaluating Blocking Biases in Entity Matching(https://arxiv.org/abs/2409.16410)
• Keywords: fair
• Abstract: Entity Matching (EM) is crucial for identifying equivalent data entities across different sources, a task that becomes increasingly challenging with the growth and heterogeneity of data. Blocking techniques, which reduce the computational complexity of EM, play a vital role in making this process scalable. Despite advancements in blocking methods, the issue of fairness; where blocking may inadvertently favor certain demographic groups; has been largely overlooked. This study extends traditional blocking metrics to incorporate fairness, providing a framework for assessing bias in blocking techniques. Through experimental analysis, we evaluate the effectiveness and fairness of various blocking methods, offering insights into their potential biases. Our findings highlight the importance of considering fairness in EM, particularly in the blocking phase, to ensure equitable outcomes in data integration tasks.

Title: Leveraging Local Structure for Improving Model Explanations: An Information Propagation Approach

• Authors: Ruo Yang, Binghui Wang, Mustafa Bilgic
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16429
• Pdf URL: https://arxiv.org/pdf/2409.16429
• Copy Paste: [[2409.16429]] Leveraging Local Structure for Improving Model Explanations: An Information Propagation Approach(https://arxiv.org/abs/2409.16429)
• Keywords: interpretability
• Abstract: Numerous explanation methods have been recently developed to interpret the decisions made by deep neural network (DNN) models. For image classifiers, these methods typically provide an attribution score to each pixel in the image to quantify its contribution to the prediction. However, most of these explanation methods appropriate attribution scores to pixels independently, even though both humans and DNNs make decisions by analyzing a set of closely related pixels simultaneously. Hence, the attribution score of a pixel should be evaluated jointly by considering itself and its structurally-similar pixels. We propose a method called IProp, which models each pixel's individual attribution score as a source of explanatory information and explains the image prediction through the dynamic propagation of information across all pixels. To formulate the information propagation, IProp adopts the Markov Reward Process, which guarantees convergence, and the final status indicates the desired pixels' attribution scores. Furthermore, IProp is compatible with any existing attribution-based explanation method. Extensive experiments on various explanation methods and DNN models verify that IProp significantly improves them on a variety of interpretability metrics.

Title: A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions

• Authors: Rajesh Ranjan, Shailja Gupta, Surya Narayan Singh
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.16430
• Pdf URL: https://arxiv.org/pdf/2409.16430
• Copy Paste: [[2409.16430]] A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions(https://arxiv.org/abs/2409.16430)
• Keywords: fair, large language model
• Abstract: Large Language Models(LLMs) have revolutionized various applications in natural language processing (NLP) by providing unprecedented text generation, translation, and comprehension capabilities. However, their widespread deployment has brought to light significant concerns regarding biases embedded within these models. This paper presents a comprehensive survey of biases in LLMs, aiming to provide an extensive review of the types, sources, impacts, and mitigation strategies related to these biases. We systematically categorize biases into several dimensions. Our survey synthesizes current research findings and discusses the implications of biases in real-world applications. Additionally, we critically assess existing bias mitigation techniques and propose future research directions to enhance fairness and equity in LLMs. This survey serves as a foundational resource for researchers, practitioners, and policymakers concerned with addressing and understanding biases in LLMs.

Title: Lessons Learned from a Unifying Empirical Study of Parameter-Efficient Transfer Learning (PETL) in Visual Recognition

• Authors: Zheda Mai, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Li Zhang, Wei-Lun Chao
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16434
• Pdf URL: https://arxiv.org/pdf/2409.16434
• Copy Paste: [[2409.16434]] Lessons Learned from a Unifying Empirical Study of Parameter-Efficient Transfer Learning (PETL) in Visual Recognition(https://arxiv.org/abs/2409.16434)
• Keywords: robust, fair, transformer
• Abstract: Parameter-efficient transfer learning (PETL) has attracted significant attention lately, due to the increasing size of pre-trained models and the need to fine-tune (FT) them for superior downstream performance. This community-wide enthusiasm has sparked a plethora of new methods. Nevertheless, a systematic study to understand their performance and suitable application scenarios is lacking, leaving questions like when to apply PETL and which method to use largely unanswered. In this paper, we conduct a unifying empirical study of representative PETL methods in the context of Vision Transformers. We systematically tune their hyper-parameters to fairly compare their accuracy on downstream tasks. Our study not only offers a valuable user guide but also unveils several new insights. First, if tuned carefully, different PETL methods can obtain quite similar accuracy in the low-shot benchmark VTAB-1K. This includes simple methods like FT the bias terms that were reported inferior. Second, though with similar accuracy, we find that PETL methods make different mistakes and high-confidence predictions, likely due to their different inductive biases. Such an inconsistency (or complementariness) opens up the opportunity for ensemble methods, and we make preliminary attempts at this. Third, going beyond the commonly used low-shot tasks, we find that PETL is also useful in many-shot regimes -- it achieves comparable and sometimes better accuracy than full FT, using much fewer learnable parameters. Last but not least, we investigate PETL's ability to preserve a pre-trained model's robustness to distribution shifts (e.g., a CLIP backbone). Perhaps not surprisingly, PETL methods outperform full FT alone. However, with weight-space ensembles, the fully FT model can achieve a better balance between downstream and out-of-distribution performance, suggesting a future research direction for PETL.

Title: Glitch in Time: Exploiting Temporal Misalignment of IMU For Eavesdropping

• Authors: Ahmed Najeeb, Abdul Rafay, Naveed Anwar Bhatti, Muhammad Hamad Alizai
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16438
• Pdf URL: https://arxiv.org/pdf/2409.16438
• Copy Paste: [[2409.16438]] Glitch in Time: Exploiting Temporal Misalignment of IMU For Eavesdropping(https://arxiv.org/abs/2409.16438)
• Keywords: security, privacy, protect, attack
• Abstract: The increasing use of voice assistants and related applications has raised significant concerns about the security of Inertial Measurement Units (IMUs) in smartphones. These devices are vulnerable to acoustic eavesdropping attacks, jeopardizing user privacy. In response, Google imposed a rate limit of 200 Hz on permission-free access to IMUs, aiming to neutralize such side-channel attacks. Our research introduces a novel exploit, STAG, which circumvents these protections. It induces a temporal misalignment between the gyroscope and accelerometer, cleverly combining their data to resample at higher rates and reviving the potential for eavesdropping attacks previously curtailed by Google's security enhancements. Compared to prior methods, STAG achieves an 83.4% reduction in word error rate, highlighting its effectiveness in exploiting IMU data under restricted access and emphasizing the persistent security risks associated with these sensors.

Title: Underground Mapping and Localization Based on Ground-Penetrating Radar

• Authors: Jinchang Zhang, Guoyu Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16446
• Pdf URL: https://arxiv.org/pdf/2409.16446
• Copy Paste: [[2409.16446]] Underground Mapping and Localization Based on Ground-Penetrating Radar(https://arxiv.org/abs/2409.16446)
• Keywords: segmentation
• Abstract: 3D object reconstruction based on deep neural networks has gained increasing attention in recent years. However, 3D reconstruction of underground objects to generate point cloud maps remains a challenge. Ground Penetrating Radar (GPR) is one of the most powerful and extensively used tools for detecting and locating underground objects such as plant root systems and pipelines, with its cost-effectiveness and continuously evolving technology. This paper introduces a parabolic signal detection network based on deep convolutional neural networks, utilizing B-scan images from GPR sensors. The detected keypoints can aid in accurately fitting parabolic curves used to interpret the original GPR B-scan images as cross-sections of the object model. Additionally, a multi-task point cloud network was designed to perform both point cloud segmentation and completion simultaneously, filling in sparse point cloud maps. For unknown locations, GPR A-scan data can be used to match corresponding A-scan data in the constructed map, pinpointing the position to verify the accuracy of the map construction by the model. Experimental results demonstrate the effectiveness of our method.

Title: FMDLlama: Financial Misinformation Detection based on Large Language Models

• Authors: Zhiwei Liu, Xin Zhang, Kailai Yang, Qianqian Xie, Jimin Huang, Sophia Ananiadou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16452
• Pdf URL: https://arxiv.org/pdf/2409.16452
• Copy Paste: [[2409.16452]] FMDLlama: Financial Misinformation Detection based on Large Language Models(https://arxiv.org/abs/2409.16452)
• Keywords: large language model
• Abstract: The emergence of social media has made the spread of misinformation easier. In the financial domain, the accuracy of information is crucial for various aspects of financial market, which has made financial misinformation detection (FMD) an urgent problem that needs to be addressed. Large language models (LLMs) have demonstrated outstanding performance in various fields. However, current studies mostly rely on traditional methods and have not explored the application of LLMs in the field of FMD. The main reason is the lack of FMD instruction tuning datasets and evaluation benchmarks. In this paper, we propose FMDLlama, the first open-sourced instruction-following LLMs for FMD task based on fine-tuning Llama3.1 with instruction data, the first multi-task FMD instruction dataset (FMDID) to support LLM instruction tuning, and a comprehensive FMD evaluation benchmark (FMD-B) with classification and explanation generation tasks to test the FMD ability of LLMs. We compare our models with a variety of LLMs on FMD-B, where our model outperforms all other open-sourced LLMs as well as ChatGPT.

Title: Communication and Energy Efficient Federated Learning using Zero-Order Optimization Technique

• Authors: Elissa Mhanna, Mohamad Assaad
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.16456
• Pdf URL: https://arxiv.org/pdf/2409.16456
• Copy Paste: [[2409.16456]] Communication and Energy Efficient Federated Learning using Zero-Order Optimization Technique(https://arxiv.org/abs/2409.16456)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) is a popular machine learning technique that enables multiple users to collaboratively train a model while maintaining the user data privacy. A significant challenge in FL is the communication bottleneck in the upload direction, and thus the corresponding energy consumption of the devices, attributed to the increasing size of the model/gradient. In this paper, we address this issue by proposing a zero-order (ZO) optimization method that requires the upload of a quantized single scalar per iteration by each device instead of the whole gradient vector. We prove its theoretical convergence and find an upper bound on its convergence rate in the non-convex setting, and we discuss its implementation in practical scenarios. Our FL method and the corresponding convergence analysis take into account the impact of quantization and packet dropping due to wireless errors. We show also the superiority of our method, in terms of communication overhead and energy consumption, as compared to standard gradient-based FL methods.

Title: Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verification

• Authors: Ramya Keerthy Thatikonda, Jiuzhou Han, Wray Buntine, Ehsan Shareghi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16461
• Pdf URL: https://arxiv.org/pdf/2409.16461
• Copy Paste: [[2409.16461]] Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verification(https://arxiv.org/abs/2409.16461)
• Keywords: large language model
• Abstract: Logical reasoning is a fundamental task in natural language processing that presents significant challenges to Large Language Models (LLMs). The inherent characteristics of logical reasoning makes it well-suited for symbolic representations such as first-order logic (FOL). Research in symbolic logical reasoning explored FOL generation using state-of-the-art LLMs (i.e., GPT-4) to produce FOL translations of natural language (NL) statements, but errors in translation are usually not the focus. We address this by categorizing the translation errors in FOL statements generated by LLMs. To make progress towards improving the quality of FOL translations for smaller language models such as LLaMA-2 13B and Mistral 7B, we create ProofFOL, a high-quality FOL-annotated subset of ProofWriter dataset using GPT-4o. The models fine-tuned on this silver standard data achieve a significant gain in performance when compared to larger language models such as LLaMA-2 70B. In addition to improving the model using large data, we also tackle the issue of data scarcity and introduce an incremental framework encompassing of data augmentation and verification steps. In the augmentation process, a single pair of (premises, conclusion) is split into multiple new instances based on the predicates and FOLs. This data is used for fine-tuning, and the inference on this model generates FOLs with fewer errors over the model trained on the original data. Our investigation on the translation errors leads to generation of a perturbation dataset, which is used to train a verifier that corrects potential syntactic and semantic FOL translation errors. We demonstrate an efficient method for making the most of a limited existing human-annotated dataset. Our results show state-of-the-art performance for ProofWriter and ProntoQA datasets using ProofFOL on LLaMA-2 and Mistral models.

Title: Spelling Correction through Rewriting of Non-Autoregressive ASR Lattices

• Authors: Leonid Velikovich, Christopher Li, Diamantino Caseiro, Shankar Kumar, Pat Rondon, Kandarp Joshi, Xavier Velez
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.16469
• Pdf URL: https://arxiv.org/pdf/2409.16469
• Copy Paste: [[2409.16469]] Spelling Correction through Rewriting of Non-Autoregressive ASR Lattices(https://arxiv.org/abs/2409.16469)
• Keywords: transformer
• Abstract: For end-to-end Automatic Speech Recognition (ASR) models, recognizing personal or rare phrases can be hard. A promising way to improve accuracy is through spelling correction (or rewriting) of the ASR lattice, where potentially misrecognized phrases are replaced with acoustically similar and contextually relevant alternatives. However, rewriting is challenging for ASR models trained with connectionist temporal classification (CTC) due to noisy hypotheses produced by a non-autoregressive, context-independent beam search. We present a finite-state transducer (FST) technique for rewriting wordpiece lattices generated by Transformer-based CTC models. Our algorithm performs grapheme-to-phoneme (G2P) conversion directly from wordpieces into phonemes, avoiding explicit word representations and exploiting the richness of the CTC lattice. Our approach requires no retraining or modification of the ASR model. We achieved up to a 15.2% relative reduction in sentence error rate (SER) on a test set with contextually relevant entities.

Title: Generative AI-driven forecasting of oil production

• Authors: Yash Gandhi, Kexin Zheng, Birendra Jha, Ken-ichi Nomura, Aiichiro Nakano, Priya Vashishta, Rajiv K. Kalia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16482
• Pdf URL: https://arxiv.org/pdf/2409.16482
• Copy Paste: [[2409.16482]] Generative AI-driven forecasting of oil production(https://arxiv.org/abs/2409.16482)
• Keywords: extraction, transformer, generative
• Abstract: Forecasting oil production from oilfields with multiple wells is an important problem in petroleum and geothermal energy extraction, as well as energy storage technologies. The accuracy of oil forecasts is a critical determinant of economic projections, hydrocarbon reserves estimation, construction of fluid processing facilities, and energy price fluctuations. Leveraging generative AI techniques, we model time series forecasting of oil and water productions across four multi-well sites spanning four decades. Our goal is to effectively model uncertainties and make precise forecasts to inform decision-making processes at the field scale. We utilize an autoregressive model known as TimeGrad and a variant of a transformer architecture named Informer, tailored specifically for forecasting long sequence time series data. Predictions from both TimeGrad and Informer closely align with the ground truth data. The overall performance of the Informer stands out, demonstrating greater efficiency compared to TimeGrad in forecasting oil production rates across all sites.

Title: Exploring Knowledge Tracing in Tutor-Student Dialogues

• Authors: Alexander Scarlatos, Andrew Lan
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16490
• Pdf URL: https://arxiv.org/pdf/2409.16490
• Copy Paste: [[2409.16490]] Exploring Knowledge Tracing in Tutor-Student Dialogues(https://arxiv.org/abs/2409.16490)
• Keywords: large language model
• Abstract: Recent advances in large language models (LLMs) have led to the development of artificial intelligence (AI)-powered tutoring chatbots, showing promise in providing broad access to high-quality personalized education. Existing works have primarily studied how to make LLMs follow tutoring principles but not how to model student behavior in dialogues. However, analyzing student dialogue turns can serve as a formative assessment, since open-ended student discourse may indicate their knowledge levels and reveal specific misconceptions. In this work, we present a first attempt at performing knowledge tracing (KT) in tutor-student dialogues. We propose LLM prompting methods to identify the knowledge components/skills involved in each dialogue turn and diagnose whether the student responds correctly to the tutor, and verify the LLM's effectiveness via an expert human evaluation. We then apply a range of KT methods on the resulting labeled data to track student knowledge levels over an entire dialogue. We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues. We perform extensive qualitative analyses to highlight the challenges in dialogue KT and outline multiple avenues for future work.

Title: Proactive Schemes: A Survey of Adversarial Attacks for Social Good

• Authors: Vishal Asnani, Xi Yin, Xiaoming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16491
• Pdf URL: https://arxiv.org/pdf/2409.16491
• Copy Paste: [[2409.16491]] Proactive Schemes: A Survey of Adversarial Attacks for Social Good(https://arxiv.org/abs/2409.16491)
• Keywords: secure, attack
• Abstract: Adversarial attacks in computer vision exploit the vulnerabilities of machine learning models by introducing subtle perturbations to input data, often leading to incorrect predictions or classifications. These attacks have evolved in sophistication with the advent of deep learning, presenting significant challenges in critical applications, which can be harmful for society. However, there is also a rich line of research from a transformative perspective that leverages adversarial techniques for social good. Specifically, we examine the rise of proactive schemes-methods that encrypt input data using additional signals termed templates, to enhance the performance of deep learning models. By embedding these imperceptible templates into digital media, proactive schemes are applied across various applications, from simple image enhancements to complicated deep learning frameworks to aid performance, as compared to the passive schemes, which don't change the input data distribution for their framework. The survey delves into the methodologies behind these proactive schemes, the encryption and learning processes, and their application to modern computer vision and natural language processing applications. Additionally, it discusses the challenges, potential vulnerabilities, and future directions for proactive schemes, ultimately highlighting their potential to foster the responsible and secure advancement of deep learning technologies.

Title: Flight: A FaaS-Based Framework for Complex and Hierarchical Federated Learning

• Authors: Nathaniel Hudson, Valerie Hayot-Sasson, Yadu Babuji, Matt Baughman, J. Gregory Pauloski, Ryan Chard, Ian Foster, Kyle Chard
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.16495
• Pdf URL: https://arxiv.org/pdf/2409.16495
• Copy Paste: [[2409.16495]] Flight: A FaaS-Based Framework for Complex and Hierarchical Federated Learning(https://arxiv.org/abs/2409.16495)
• Keywords: federate
• Abstract: Federated Learning (FL) is a decentralized machine learning paradigm where models are trained on distributed devices and are aggregated at a central server. Existing FL frameworks assume simple two-tier network topologies where end devices are directly connected to the aggregation server. While this is a practical mental model, it does not exploit the inherent topology of real-world distributed systems like the Internet-of-Things. We present Flight, a novel FL framework that supports complex hierarchical multi-tier topologies, asynchronous aggregation, and decouples the control plane from the data plane. We compare the performance of Flight against Flower, a state-of-the-art FL framework. Our results show that Flight scales beyond Flower, supporting up to 2048 simultaneous devices, and reduces FL makespan across several models. Finally, we show that Flight's hierarchical FL model can reduce communication overheads by more than 60%.

Title: Real-Time Detection of Electronic Components in Waste Printed Circuit Boards: A Transformer-Based Approach

• Authors: Muhammad Mohsin, Stefano Rovetta, Francesco Masulli, Alberto Cabri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16496
• Pdf URL: https://arxiv.org/pdf/2409.16496
• Copy Paste: [[2409.16496]] Real-Time Detection of Electronic Components in Waste Printed Circuit Boards: A Transformer-Based Approach(https://arxiv.org/abs/2409.16496)
• Keywords: extraction, transformer
• Abstract: Critical Raw Materials (CRMs) such as copper, manganese, gallium, and various rare earths have great importance for the electronic industry. To increase the concentration of individual CRMs and thus make their extraction from Waste Printed Circuit Boards (WPCBs) convenient, we have proposed a practical approach that involves selective disassembling of the different types of electronic components from WPCBs using mechatronic systems guided by artificial vision techniques. In this paper we evaluate the real-time accuracy of electronic component detection and localization of the Real-Time DEtection TRansformer model architecture. Transformers have recently become very popular for the extraordinary results obtained in natural language processing and machine translation. Also in this case, the transformer model achieves very good performances, often superior to those of the latest state of the art object detection and localization models YOLOv8 and YOLOv9.

Title: Low Latency Point Cloud Rendering with Learned Splatting

• Authors: Yueyu Hu, Ran Gong, Qi Sun, Yao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16504
• Pdf URL: https://arxiv.org/pdf/2409.16504
• Copy Paste: [[2409.16504]] Low Latency Point Cloud Rendering with Learned Splatting(https://arxiv.org/abs/2409.16504)
• Keywords: robust
• Abstract: Point cloud is a critical 3D representation with many emerging applications. Because of the point sparsity and irregularity, high-quality rendering of point clouds is challenging and often requires complex computations to recover the continuous surface representation. On the other hand, to avoid visual discomfort, the motion-to-photon latency has to be very short, under 10 ms. Existing rendering solutions lack in either quality or speed. To tackle these challenges, we present a framework that unlocks interactive, free-viewing and high-fidelity point cloud rendering. We train a generic neural network to estimate 3D elliptical Gaussians from arbitrary point clouds and use differentiable surface splatting to render smooth texture and surface normal for arbitrary views. Our approach does not require per-scene optimization, and enable real-time rendering of dynamic point cloud. Experimental results demonstrate the proposed solution enjoys superior visual quality and speed, as well as generalizability to different scene content and robustness to compression artifacts. The code is available at this https URL .

Title: Understanding the Cognitive Complexity in Language Elicited by Product Images

• Authors: Yan-Ying Chen, Shabnam Hakimi, Monica Van, Francine Chen, Matthew Hong, Matt Klenk, Charlene Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16521
• Pdf URL: https://arxiv.org/pdf/2409.16521
• Copy Paste: [[2409.16521]] Understanding the Cognitive Complexity in Language Elicited by Product Images(https://arxiv.org/abs/2409.16521)
• Keywords: large language model
• Abstract: Product images (e.g., a phone) can be used to elicit a diverse set of consumer-reported features expressed through language, including surface-level perceptual attributes (e.g., "white") and more complex ones, like perceived utility (e.g., "battery"). The cognitive complexity of elicited language reveals the nature of cognitive processes and the context required to understand them; cognitive complexity also predicts consumers' subsequent choices. This work offers an approach for measuring and validating the cognitive complexity of human language elicited by product images, providing a tool for understanding the cognitive processes of human as well as virtual respondents simulated by Large Language Models (LLMs). We also introduce a large dataset that includes diverse descriptive labels for product images, including human-rated complexity. We demonstrate that human-rated cognitive complexity can be approximated using a set of natural language models that, combined, roughly capture the complexity construct. Moreover, this approach is minimally supervised and scalable, even in use cases with limited human assessment of complexity.

Title: APILOT: Navigating Large Language Models to Generate Secure Code by Sidestepping Outdated API Pitfalls

• Authors: Weiheng Bai, Keyang Xuan, Pengxiang Huang, Qiushi Wu, Jianing Wen, Jingjing Wu, Kangjie Lu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16526
• Pdf URL: https://arxiv.org/pdf/2409.16526
• Copy Paste: [[2409.16526]] APILOT: Navigating Large Language Models to Generate Secure Code by Sidestepping Outdated API Pitfalls(https://arxiv.org/abs/2409.16526)
• Keywords: secure, security, large language model
• Abstract: With the rapid development of large language models (LLMs), their applications have expanded into diverse fields, such as code assistance. However, the substantial size of LLMs makes their training highly resource- and time-intensive, rendering frequent retraining or updates impractical. Consequently, time-sensitive data can become outdated, potentially misleading LLMs in time-aware tasks. For example, new vulnerabilities are discovered in various programs every day. Without updating their knowledge, LLMs may inadvertently generate code that includes these newly discovered vulnerabilities. Current strategies, such as prompt engineering and fine-tuning, do not effectively address this issue. To address this issue, we propose solution, named APILOT, which maintains a realtime, quickly updatable dataset of outdated APIs. Additionally, APILOT utilizes an augmented generation method that leverages this dataset to navigate LLMs in generating secure, version-aware code. We conducted a comprehensive evaluation to measure the effectiveness of APILOT in reducing the incidence of outdated API recommendations across seven different state-of-the-art LLMs. The evaluation results indicate that APILOT can reduce outdated code recommendations by 89.42% on average with limited performance overhead. Interestingly, while enhancing security, APILOT also improves the usability of the code generated by LLMs, showing an average increase of 27.54% in usability. This underscores APILOT's dual capability to enhance both the safety and practical utility of code suggestions in contemporary software development environments.

Title: T2Pair++: Secure and Usable IoT Pairing with Zero Information Loss

• Authors: Chuxiong Wu, Xiaopeng Li, Lannan Luo, Qiang Zeng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16530
• Pdf URL: https://arxiv.org/pdf/2409.16530
• Copy Paste: [[2409.16530]] T2Pair++: Secure and Usable IoT Pairing with Zero Information Loss(https://arxiv.org/abs/2409.16530)
• Keywords: secure, security
• Abstract: Secure pairing is crucial for ensuring the trustworthy deployment and operation of Internet of Things (IoT) devices. However, traditional pairing methods are often unsuitable for IoT devices due to their lack of conventional user interfaces, such as keyboards. Proximity-based pairing approaches are usable but vulnerable to exploitation by co-located malicious devices. While methods based on a user's physical operations (such as shaking) on IoT devices offer greater security, they typically rely on inertial sensors to sense the operations, which most IoT devices lack. We introduce a novel technique called Universal Operation Sensing, enabling IoT devices to sense the user's physical operations without the need for inertial sensors. With this technique, users can complete the pairing process within seconds using simple actions such as pressing a button or twisting a knob, whether they are holding a smartphone or wearing a smartwatch. Moreover, we reveal an inaccuracy issue in the fuzzy commitment protocol, which is frequently used for pairing. To address it, we propose an accurate pairing protocol, which does not use fuzzy commitment and incurs zero information loss. The comprehensive evaluation shows that it is secure, usable and efficient.

Title: Prompt Sliders for Fine-Grained Control, Editing and Erasing of Concepts in Diffusion Models

• Authors: Deepak Sridhar, Nuno Vasconcelos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16535
• Pdf URL: https://arxiv.org/pdf/2409.16535
• Copy Paste: [[2409.16535]] Prompt Sliders for Fine-Grained Control, Editing and Erasing of Concepts in Diffusion Models(https://arxiv.org/abs/2409.16535)
• Keywords: diffusion
• Abstract: Diffusion models have recently surpassed GANs in image synthesis and editing, offering superior image quality and diversity. However, achieving precise control over attributes in generated images remains a challenge. Concept Sliders introduced a method for fine-grained image control and editing by learning concepts (attributes/objects). However, this approach adds parameters and increases inference time due to the loading and unloading of Low-Rank Adapters (LoRAs) used for learning concepts. These adapters are model-specific and require retraining for different architectures, such as Stable Diffusion (SD) v1.5 and SD-XL. In this paper, we propose a straightforward textual inversion method to learn concepts through text embeddings, which are generalizable across models that share the same text encoder, including different versions of the SD model. We refer to our method as Prompt Sliders. Besides learning new concepts, we also show that Prompt Sliders can be used to erase undesirable concepts such as artistic styles or mature content. Our method is 30% faster than using LoRAs because it eliminates the need to load and unload adapters and introduces no additional parameters aside from the target concept text embedding. Each concept embedding only requires 3KB of storage compared to the 8922KB or more required for each LoRA adapter, making our approach more computationally efficient. Project Page: this https URL

Title: Time Constant: Actuator Fingerprinting using Transient Response of Device and Process in ICS

• Authors: Chuadhry Mujeeb Ahmed, Matthew Calder, Sean Gunawan, Jay Prakash, Shishir Nagaraja, Jianying Zhou
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16536
• Pdf URL: https://arxiv.org/pdf/2409.16536
• Copy Paste: [[2409.16536]] Time Constant: Actuator Fingerprinting using Transient Response of Device and Process in ICS(https://arxiv.org/abs/2409.16536)
• Keywords: attack
• Abstract: Command injection and replay attacks are key threats in Cyber Physical Systems (CPS). We develop a novel actuator fingerprinting technique named Time Constant. Time Constant captures the transient dynamics of an actuator and physical process. The transient behavior is device-specific. We combine process and device transient characteristics to develop a copy-resistant actuator fingerprint that resists command injection and replay attacks in the face of insider adversaries. We validated the proposed scheme on data from a real water treatment testbed, as well as through real-time attack detection in the live plant. Our results show that we can uniquely distinguish between process states and actuators based on their Time Constant.

Title: Source-Free Domain Adaptation for YOLO Object Detection

• Authors: Simon Varailhon, Masih Aminbeidokhti, Marco Pedersoli, Eric Granger
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16538
• Pdf URL: https://arxiv.org/pdf/2409.16538
• Copy Paste: [[2409.16538]] Source-Free Domain Adaptation for YOLO Object Detection(https://arxiv.org/abs/2409.16538)
• Keywords: privacy
• Abstract: Source-free domain adaptation (SFDA) is a challenging problem in object detection, where a pre-trained source model is adapted to a new target domain without using any source domain data for privacy and efficiency reasons. Most state-of-the-art SFDA methods for object detection have been proposed for Faster-RCNN, a detector that is known to have high computational complexity. This paper focuses on domain adaptation techniques for real-world vision systems, particularly for the YOLO family of single-shot detectors known for their fast baselines and practical applications. Our proposed SFDA method - Source-Free YOLO (SF-YOLO) - relies on a teacher-student framework in which the student receives images with a learned, target domain-specific augmentation, allowing the model to be trained with only unlabeled target data and without requiring feature alignment. A challenge with self-training using a mean-teacher architecture in the absence of labels is the rapid decline of accuracy due to noisy or drifting pseudo-labels. To address this issue, a teacher-to-student communication mechanism is introduced to help stabilize the training and reduce the reliance on annotated target data for model selection. Despite its simplicity, our approach is competitive with state-of-the-art detectors on several challenging benchmark datasets, even sometimes outperforming methods that use source data for adaptation.

Title: Monge-Kantorovich Fitting With Sobolev Budgets

• Authors: Forest Kobayashi, Jonathan Hayase, Young-Heon Kim
• Subjects: cs.LG, math.AP
• Abstract URL: https://arxiv.org/abs/2409.16541
• Pdf URL: https://arxiv.org/pdf/2409.16541
• Copy Paste: [[2409.16541]] Monge-Kantorovich Fitting With Sobolev Budgets(https://arxiv.org/abs/2409.16541)
• Keywords: generative
• Abstract: We consider the problem of finding the ``best'' approximation of an $n$-dimensional probability measure $\rho$ using a measure $\nu$ whose support is parametrized by $f : \mathbb{R}^m \to \mathbb{R}^n$ where $m < n$. We quantify the performance of the approximation with the Monge-Kantorovich $p$-cost (also called the Wasserstein $p$-cost) $\mathbb{W}_p^p(\rho, \nu)$, and constrain the complexity of the approximation by bounding the $W^{k,q}$ Sobolev norm of $f$, which acts as a ``budget.'' We may then reformulate the problem as minimizing a functional $\mathscr{J}_p(f)$ under a constraint on the Sobolev budget. We treat general $k \geq 1$ for the Sobolev differentiability order (though $q, m$ are chosen to restrict $W^{k,q}$ to the supercritical regime $k q > m$ to guarantee existence of optimizers). The problem is closely related to (but distinct from) principal curves with length constraints when $m=1, k = 1$ and smoothing splines when $k > 1$. New aspects and challenges arise from the higher order differentiability condition. We study the gradient of $\mathscr{J}_p$, which is given by a vector field along $f$ we call the barycenter field. We use it to construct improvements to a given $f$, which gives a nontrivial (almost) strict monotonicty relation between the functional $\mathscr{J}_p$ and the Sobolev budget. We also provide a natural discretization scheme and establish its consistency. We use this scheme to model a generative learning task; in particular, we demonstrate that adding a constraint like ours as a soft penalty yields substantial improvement in training a GAN to produce images of handwritten digits, with performance competitive with weight-decay.

Title: EMIT- Event-Based Masked Auto Encoding for Irregular Time Series

• Authors: Hrishikesh Patel, Ruihong Qiu, Adam Irwin, Shazia Sadiq, Sen Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16554
• Pdf URL: https://arxiv.org/pdf/2409.16554
• Copy Paste: [[2409.16554]] EMIT- Event-Based Masked Auto Encoding for Irregular Time Series(https://arxiv.org/abs/2409.16554)
• Keywords: robust
• Abstract: Irregular time series, where data points are recorded at uneven intervals, are prevalent in healthcare settings, such as emergency wards where vital signs and laboratory results are captured at varying times. This variability, which reflects critical fluctuations in patient health, is essential for informed clinical decision-making. Existing self-supervised learning research on irregular time series often relies on generic pretext tasks like forecasting, which may not fully utilise the signal provided by irregular time series. There is a significant need for specialised pretext tasks designed for the characteristics of irregular time series to enhance model performance and robustness, especially in scenarios with limited data availability. This paper proposes a novel pretraining framework, EMIT, an event-based masking for irregular time series. EMIT focuses on masking-based reconstruction in the latent space, selecting masking points based on the rate of change in the data. This method preserves the natural variability and timing of measurements while enhancing the model's ability to process irregular intervals without losing essential information. Extensive experiments on the MIMIC-III and PhysioNet Challenge datasets demonstrate the superior performance of our event-based masking strategy. The code has been released at this https URL .

Title: Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)

• Authors: Yuchen Li, Haoyi Xiong, Linghe Kong, Zeyi Sun, Hongyang Chen, Shuaiqiang Wang, Dawei Yin
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.16590
• Pdf URL: https://arxiv.org/pdf/2409.16590
• Copy Paste: [[2409.16590]] Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)(https://arxiv.org/abs/2409.16590)
• Keywords: transformer
• Abstract: Both Transformer and Graph Neural Networks (GNNs) have been employed in the domain of learning to rank (LTR). However, these approaches adhere to two distinct yet complementary problem formulations: ranking score regression based on query-webpage pairs, and link prediction within query-webpage bipartite graphs, respectively. While it is possible to pre-train GNNs or Transformers on source datasets and subsequently fine-tune them on sparsely annotated LTR datasets, the distributional shifts between the pair-based and bipartite graph domains present significant challenges in integrating these heterogeneous models into a unified LTR framework at web scale. To address this, we introduce the novel MPGraf model, which leverages a modular and capsule-based pre-training strategy, aiming to cohesively integrate the regression capabilities of Transformers with the link prediction strengths of GNNs. We conduct extensive offline and online experiments to rigorously evaluate the performance of MPGraf.

Title: EventHallusion: Diagnosing Event Hallucinations in Video LLMs

• Authors: Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, Yu-Gang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16597
• Pdf URL: https://arxiv.org/pdf/2409.16597
• Copy Paste: [[2409.16597]] EventHallusion: Diagnosing Event Hallucinations in Video LLMs(https://arxiv.org/abs/2409.16597)
• Keywords: large language model
• Abstract: Recently, Multimodal Large Language Models (MLLMs) have made significant progress in the video comprehension field. Despite remarkable content reasoning and instruction following capabilities they demonstrated, the hallucination problem of these VideoLLMs is less explored compared with its counterpart in the image domain. To mitigate this gap, we first propose EventHallusion, a novel benchmark that focuses on assessing the VideoLMMs' hallucination phenomenon on video event comprehension. Based on the observation that existing VideoLLMs are entangled with the priors stemming from their foundation models, our EventHallusion is curated by meticulously collecting videos and annotating questions to intentionally mislead the VideoLLMs into interpreting events based on these priors rather than accurately understanding the video content. On the other hand, we also propose a simple yet effective method, called Temporal Contrastive Decoding (TCD), to tackle the hallucination problems of VideoLLMs. The proposed TCD suppresses the model's preference toward their priors by comparing the original video with a constructed counterpart, whose temporal cues are disrupted, during the autoregressive decoding stage. Through comprehensive evaluation of eight open-source and two closed-source VideoLLMs on the proposed EventHallusion benchmark, we find that the open-source models suffer significantly from hallucination problems, whereas the closed-source models perform markedly better. By further equipping open-sourced VideoLLMs with the proposed TCD approach, evident performance improvements are achieved across most metrics in the EventHallusion benchmark. Our codes and benchmark data are available at this https URL.

Title: Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications

• Authors: Ethan Lin, Zhiyuan Peng, Yi Fang
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16605
• Pdf URL: https://arxiv.org/pdf/2409.16605
• Copy Paste: [[2409.16605]] Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications(https://arxiv.org/abs/2409.16605)
• Keywords: large language model
• Abstract: Recent studies have evaluated the creativity/novelty of large language models (LLMs) primarily from a semantic perspective, using benchmarks from cognitive science. However, accessing the novelty in scholarly publications is a largely unexplored area in evaluating LLMs. In this paper, we introduce a scholarly novelty benchmark (SchNovel) to evaluate LLMs' ability to assess novelty in scholarly papers. SchNovel consists of 15000 pairs of papers across six fields sampled from the arXiv dataset with publication dates spanning 2 to 10 years apart. In each pair, the more recently published paper is assumed to be more novel. Additionally, we propose RAG-Novelty, which simulates the review process taken by human reviewers by leveraging the retrieval of similar papers to assess novelty. Extensive experiments provide insights into the capabilities of different LLMs to assess novelty and demonstrate that RAG-Novelty outperforms recent baseline models.

Title: DeformStream: Deformation-based Adaptive Volumetric Video Streaming

• Authors: Boyan Li, Yongting Chen, Dayou Zhang, Fangxin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16615
• Pdf URL: https://arxiv.org/pdf/2409.16615
• Copy Paste: [[2409.16615]] DeformStream: Deformation-based Adaptive Volumetric Video Streaming(https://arxiv.org/abs/2409.16615)
• Keywords: robust
• Abstract: Volumetric video streaming offers immersive 3D experiences but faces significant challenges due to high bandwidth requirements and latency issues in transmitting detailed content in real time. Traditional methods like point cloud streaming compromise visual quality when zoomed in, and neural rendering techniques are too computationally intensive for real-time use. Though mesh-based streaming stands out by preserving surface detail and connectivity, offering a more refined representation for 3D content, traditional mesh streaming methods typically transmit data on a per-frame basis, failing to take full advantage of temporal redundancies across frames. This results in inefficient bandwidth usage and poor adaptability to fluctuating network conditions. We introduce Deformation-based Adaptive Volumetric Video Streaming, a novel framework that enhances volumetric video streaming performance by leveraging the inherent deformability of mesh-based representations. DeformStream uses embedded deformation to reconstruct subsequent frames from inter-frame motion, significantly reducing bandwidth usage while ensuring visual coherence between frames. To address frame reconstruction overhead and network adaptability, we formulate a new QoE model that accounts for client-side deformation latency and design a dynamic programming algorithm to optimize the trade-off between visual quality and bandwidth consumption under varying network conditions. Our evaluation demonstrates that Deformation-based Adaptive Volumetric Video Streaming outperforms existing mesh-based streaming systems in both bandwidth efficiency and visual quality, offering a robust solution for real-time volumetric video applications.

Title: Claim-Guided Textual Backdoor Attack for Practical Applications

• Authors: Minkyoo Song, Hanna Kim, Jaehan Kim, Youngjin Jin, Seungwon Shin
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16618
• Pdf URL: https://arxiv.org/pdf/2409.16618
• Copy Paste: [[2409.16618]] Claim-Guided Textual Backdoor Attack for Practical Applications(https://arxiv.org/abs/2409.16618)
• Keywords: security, attack, steal, extraction, large language model
• Abstract: Recent advances in natural language processing and the increased use of large language models have exposed new security vulnerabilities, such as backdoor attacks. Previous backdoor attacks require input manipulation after model distribution to activate the backdoor, posing limitations in real-world applicability. Addressing this gap, we introduce a novel Claim-Guided Backdoor Attack (CGBA), which eliminates the need for such manipulations by utilizing inherent textual claims as triggers. CGBA leverages claim extraction, clustering, and targeted training to trick models to misbehave on targeted claims without affecting their performance on clean data. CGBA demonstrates its effectiveness and stealthiness across various datasets and models, significantly enhancing the feasibility of practical backdoor attacks. Our code and data will be available at this https URL.

Title: Ascend HiFloat8 Format for Deep Learning

• Authors: Yuanyong Luo, Zhongxing Zhang, Richard Wu, Hu Liu, Ying Jin, Kai Zheng, Minmin Wang, Zhanying He, Guipeng Hu, Luyao Chen, Tianchi Hu, Junsong Wang, Minqi Chen, Mikhaylov Dmitry, Korviakov Vladimir, Bobrin Maxim, Yuhao Hu, Guanfu Chen, Zeyi Huang
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2409.16626
• Pdf URL: https://arxiv.org/pdf/2409.16626
• Copy Paste: [[2409.16626]] Ascend HiFloat8 Format for Deep Learning(https://arxiv.org/abs/2409.16626)
• Keywords: large language model
• Abstract: This preliminary white paper proposes a novel 8-bit floating-point data format HiFloat8 (abbreviated as HiF8) for deep learning. HiF8 features tapered precision. For normal value encoding, it provides 7 exponents with 3-bit mantissa, 8 exponents with 2-bit mantissa, and 16 exponents with 1-bit mantissa. For denormal or subnormal value encoding, it extends the dynamic range by 7 extra powers of 2, from 31 to 38 binades (notice that FP16 covers 40 binades). Meanwhile, HiF8 encodes all the special values except that positive zero and negative zero are represented by only one bit-pattern. Thanks to the better balance between precision and dynamic range, HiF8 can be simultaneously used in both forward and backward passes of AI training. In this paper, we will describe the definition and rounding methods of HiF8, as well as the tentative training and inference solutions. To demonstrate the efficacy of HiF8 format, massive simulation results on various neural networks, including traditional neural networks and large language models (LLMs), will also be presented.

Title: Functional Stochastic Gradient MCMC for Bayesian Neural Networks

• Authors: Mengjing Wu, Junyu Xuan, Jie Lu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16632
• Pdf URL: https://arxiv.org/pdf/2409.16632
• Copy Paste: [[2409.16632]] Functional Stochastic Gradient MCMC for Bayesian Neural Networks(https://arxiv.org/abs/2409.16632)
• Keywords: diffusion
• Abstract: Classical variational inference for Bayesian neural networks (BNNs) in parameter space usually suffers from unresolved prior issues such as knowledge encoding intractability and pathological behaviors in deep networks, which could lead to an improper posterior inference. Hence, functional variational inference has been proposed recently to resolve these issues via stochastic process priors. Beyond variational inference, stochastic gradient Markov Chain Monte Carlo (SGMCMC) is another scalable and effective inference method for BNNs to asymptotically generate samples from true posterior by simulating a continuous dynamic. However, the existing SGMCMC methods only work in parametric space, which has the same issues of parameter-space variational inference, and extending the parameter-space dynamics to function-space dynamics is not a trivial undertaking. In this paper, we introduce a new functional SGMCMC scheme via newly designed diffusion dynamics, which can incorporate more informative functional priors. Moreover, we prove that the stationary distribution of these functional dynamics is the target posterior distribution over functions. We demonstrate better performance in both accuracy and uncertainty quantification of our functional SGMCMC on several tasks compared with naive SGMCMC and functional variational inference methods.

Title: Training Language Models to Win Debates with Self-Play Improves Judge Accuracy

• Authors: Samuel Arnesen, David Rein, Julian Michael
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16636
• Pdf URL: https://arxiv.org/pdf/2409.16636
• Copy Paste: [[2409.16636]] Training Language Models to Win Debates with Self-Play Improves Judge Accuracy(https://arxiv.org/abs/2409.16636)
• Keywords: robust
• Abstract: We test the robustness of debate as a method of scalable oversight by training models to debate with data generated via self-play. In a long-context reading comprehension task, we find that language model based evaluators answer questions more accurately when judging models optimized to win debates. By contrast, we find no such relationship for consultancy models trained to persuade a judge without an opposing debater present. In quantitative and qualitative comparisons between our debate models and novel consultancy baselines, we find evidence that debate training encourages stronger and more informative arguments, showing promise that it can help provide high-quality supervision for tasks that are difficult to directly evaluate.

Title: Examining the Rat in the Tunnel: Interpretable Multi-Label Classification of Tor-based Malware

• Authors: Ishan Karunanayake, Mashael AlSabah, Nadeem Ahmed, Sanjay Jha
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16639
• Pdf URL: https://arxiv.org/pdf/2409.16639
• Copy Paste: [[2409.16639]] Examining the Rat in the Tunnel: Interpretable Multi-Label Classification of Tor-based Malware(https://arxiv.org/abs/2409.16639)
• Keywords: privacy, attack, robust
• Abstract: Despite being the most popular privacy-enhancing network, Tor is increasingly adopted by cybercriminals to obfuscate malicious traffic, hindering the identification of malware-related communications between compromised devices and Command and Control (C&C) servers. This malicious traffic can induce congestion and reduce Tor's performance, while encouraging network administrators to block Tor traffic. Recent research, however, demonstrates the potential for accurately classifying captured Tor traffic as malicious or benign. While existing efforts have addressed malware class identification, their performance remains limited, with micro-average precision and recall values around 70%. Accurately classifying specific malware classes is crucial for effective attack prevention and mitigation. Furthermore, understanding the unique patterns and attack vectors employed by different malware classes helps the development of robust and adaptable defence mechanisms. We utilise a multi-label classification technique based on Message-Passing Neural Networks, demonstrating its superiority over previous approaches such as Binary Relevance, Classifier Chains, and Label Powerset, by achieving micro-average precision (MAP) and recall (MAR) exceeding 90%. Compared to previous work, we significantly improve performance by 19.98%, 10.15%, and 59.21% in MAP, MAR, and Hamming Loss, respectively. Next, we employ Explainable Artificial Intelligence (XAI) techniques to interpret the decision-making process within these models. Finally, we assess the robustness of all techniques by crafting adversarial perturbations capable of manipulating classifier predictions and generating false positives and negatives.

Title: Progressive Representation Learning for Real-Time UAV Tracking

• Authors: Changhong Fu, Xiang Lei, Haobo Zuo, Liangliang Yao, Guangze Zheng, Jia Pan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16652
• Pdf URL: https://arxiv.org/pdf/2409.16652
• Copy Paste: [[2409.16652]] Progressive Representation Learning for Real-Time UAV Tracking(https://arxiv.org/abs/2409.16652)
• Keywords: robust
• Abstract: Visual object tracking has significantly promoted autonomous applications for unmanned aerial vehicles (UAVs). However, learning robust object representations for UAV tracking is especially challenging in complex dynamic environments, when confronted with aspect ratio change and occlusion. These challenges severely alter the original information of the object. To handle the above issues, this work proposes a novel progressive representation learning framework for UAV tracking, i.e., PRL-Track. Specifically, PRL-Track is divided into coarse representation learning and fine representation learning. For coarse representation learning, two innovative regulators, which rely on appearance and semantic information, are designed to mitigate appearance interference and capture semantic information. Furthermore, for fine representation learning, a new hierarchical modeling generator is developed to intertwine coarse object representations. Exhaustive experiments demonstrate that the proposed PRL-Track delivers exceptional performance on three authoritative UAV tracking benchmarks. Real-world tests indicate that the proposed PRL-Track realizes superior tracking performance with 42.6 frames per second on the typical UAV platform equipped with an edge smart camera. The code, model, and demo videos are available at \url{this https URL}.

Title: The Credibility Transformer

• Authors: Ronald Richman, Salvatore Scognamiglio, Mario V. Wüthrich
• Subjects: cs.LG, q-fin.GN
• Abstract URL: https://arxiv.org/abs/2409.16653
• Pdf URL: https://arxiv.org/pdf/2409.16653
• Copy Paste: [[2409.16653]] The Credibility Transformer(https://arxiv.org/abs/2409.16653)
• Keywords: transformer, large language model
• Abstract: Inspired by the large success of Transformers in Large Language Models, these architectures are increasingly applied to tabular data. This is achieved by embedding tabular data into low-dimensional Euclidean spaces resulting in similar structures as time-series data. We introduce a novel credibility mechanism to this Transformer architecture. This credibility mechanism is based on a special token that should be seen as an encoder that consists of a credibility weighted average of prior information and observation based information. We demonstrate that this novel credibility mechanism is very beneficial to stabilize training, and our Credibility Transformer leads to predictive models that are superior to state-of-the-art deep learning models.

Title: TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans

• Authors: Aggelina Chatziagapi, Bindita Chaudhuri, Amit Kumar, Rakesh Ranjan, Dimitris Samaras, Nikolaos Sarafianos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16666
• Pdf URL: https://arxiv.org/pdf/2409.16666
• Copy Paste: [[2409.16666]] TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans(https://arxiv.org/abs/2409.16666)
• Keywords: robust
• Abstract: We introduce a novel framework that learns a dynamic neural radiance field (NeRF) for full-body talking humans from monocular videos. Prior work represents only the body pose or the face. However, humans communicate with their full body, combining body pose, hand gestures, as well as facial expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network that represents the holistic 4D human motion. Given a monocular video of a subject, we learn corresponding modules for the body, face, and hands, that are combined together to generate the final result. To capture complex finger articulation, we learn an additional deformation field for the hands. Our multi-identity representation enables simultaneous training for multiple subjects, as well as robust animation under completely unseen poses. It can also generalize to novel identities, given only a short video as input. We demonstrate state-of-the-art performance for animating full-body talking humans, with fine-grained hand articulation and facial expressions.

Title: A Character-Centric Creative Story Generation via Imagination

• Authors: Kyeongman Park, Minbeom Kim, Kyomin Jung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16667
• Pdf URL: https://arxiv.org/pdf/2409.16667
• Copy Paste: [[2409.16667]] A Character-Centric Creative Story Generation via Imagination(https://arxiv.org/abs/2409.16667)
• Keywords: large language model
• Abstract: Creative story generation with diverse and detailed story elements is a long-standing goal for large language models. While existing methodologies generate long and coherent stories, they fall significantly short of human capabilities in terms of diversity and character detail. To address this, we introduce a novel story generation framework called CCI (Character-centric Creative story generation via Imagination). CCI features two innovative modules for creative story generation: IG (Image-Guided Imagination) and MW (Multi-Writer model). In the IG module, we utilize DALL-E 3 to create visual representations of key story elements. The IG generates more novel and concrete characters, backgrounds, and main plots than text-only methods. The MW module uses these story elements created by IG to generate multiple description candidates for the protagonist and select the best one. This method incorporates vivid and rich character descriptions into the story. We compared the stories generated by CCI and baseline models through human evaluation and statistical analysis. The results showed significant improvements in the creativity. Furthermore, by enabling interactive multi-modal story generation with users, we have opened up possibilities for human-LLM integration in cultural development.

Title: GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learning

• Authors: Zhe-Rui Yang, Jindong Han, Chang-Dong Wang, Hao Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16670
• Pdf URL: https://arxiv.org/pdf/2409.16670
• Copy Paste: [[2409.16670]] GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learning(https://arxiv.org/abs/2409.16670)
• Keywords: large language model
• Abstract: Graph Neural Networks (GNNs) have demonstrated remarkable proficiency in handling a range of graph analytical tasks across various domains, such as e-commerce and social networks. Despite their versatility, GNNs face significant challenges in transferability, limiting their utility in real-world applications. Existing research in GNN transfer learning overlooks discrepancies in distribution among various graph datasets, facing challenges when transferring across different distributions. How to effectively adopt a well-trained GNN to new graphs with varying feature and structural distributions remains an under-explored problem. Taking inspiration from the success of Low-Rank Adaptation (LoRA) in adapting large language models to various domains, we propose GraphLoRA, an effective and parameter-efficient method for transferring well-trained GNNs to diverse graph domains. Specifically, we first propose a Structure-aware Maximum Mean Discrepancy (SMMD) to align divergent node feature distributions across source and target graphs. Moreover, we introduce low-rank adaptation by injecting a small trainable GNN alongside the pre-trained one, effectively bridging structural distribution gaps while mitigating the catastrophic forgetting. Additionally, a structure-aware regularization objective is proposed to enhance the adaptability of the pre-trained GNN to target graph with scarce supervision labels. Extensive experiments on six real-world datasets demonstrate the effectiveness of GraphLoRA against eleven baselines by tuning only 20% of parameters, even across disparate graph domains. The code is available at https://anonymous.4open.science/r/GraphLoRA.

Title: SWE2: SubWord Enriched and Significant Word Emphasized Framework for Hate Speech Detection

• Authors: Guanyi Mou, Pengyi Ye, Kyumin Lee
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16673
• Pdf URL: https://arxiv.org/pdf/2409.16673
• Copy Paste: [[2409.16673]] SWE2: SubWord Enriched and Significant Word Emphasized Framework for Hate Speech Detection(https://arxiv.org/abs/2409.16673)
• Keywords: attack, robust
• Abstract: Hate speech detection on online social networks has become one of the emerging hot topics in recent years. With the broad spread and fast propagation speed across online social networks, hate speech makes significant impacts on society by increasing prejudice and hurting people. Therefore, there are aroused attention and concern from both industry and academia. In this paper, we address the hate speech problem and propose a novel hate speech detection framework called SWE2, which only relies on the content of messages and automatically identifies hate speech. In particular, our framework exploits both word-level semantic information and sub-word knowledge. It is intuitively persuasive and also practically performs well under a situation with/without character-level adversarial attack. Experimental results show that our proposed model achieves 0.975 accuracy and 0.953 macro F1, outperforming 7 state-of-the-art baselines under no adversarial attack. Our model robustly and significantly performed well under extreme adversarial attack (manipulation of 50% messages), achieving 0.967 accuracy and 0.934 macro F1.

Title: CryptoTrain: Fast Secure Training on Encrypted Datase

• Authors: Jiaqi Xue, Yancheng Zhang, Yanshan Wang, Xueqiang Wang, Hao Zheng, Qian Lou
• Subjects: cs.CR, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16675
• Pdf URL: https://arxiv.org/pdf/2409.16675
• Copy Paste: [[2409.16675]] CryptoTrain: Fast Secure Training on Encrypted Datase(https://arxiv.org/abs/2409.16675)
• Keywords: secure, protect
• Abstract: Secure training, while protecting the confidentiality of both data and model weights, typically incurs significant training overhead. Traditional Fully Homomorphic Encryption (FHE)-based non-inter-active training models are heavily burdened by computationally demanding bootstrapping. To develop an efficient secure training system, we established a foundational framework, CryptoTrain-B, utilizing a hybrid cryptographic protocol that merges FHE with Oblivious Transfer (OT) for handling linear and non-linear operations, respectively. This integration eliminates the need for costly bootstrapping. Although CryptoTrain-B sets a new baseline in performance, reducing its training overhead remains essential. We found that ciphertext-ciphertext multiplication (CCMul) is a critical bottleneck in operations involving encrypted inputs and models. Our solution, the CCMul-Precompute technique, involves precomputing CCMul offline and resorting to the less resource-intensive ciphertext-plaintext multiplication (CPMul) during private training. Furthermore, conventional polynomial convolution in FHE systems tends to encode irrelevant and redundant values into polynomial slots, necessitating additional polynomials and ciphertexts for input representation and leading to extra multiplications. Addressing this, we introduce correlated polynomial convolution, which encodes only related input values into polynomials, thus drastically reducing the number of computations and overheads. By integrating CCMul-Precompute and correlated polynomial convolution into CryptoTrain-B, we facilitate a rapid and efficient secure training framework, CryptoTrain. Extensive experiments demonstrate that CryptoTrain achieves a ~5.3X training time reduction compared to prior methods.

Title: Erase then Rectify: A Training-Free Parameter Editing Approach for Cost-Effective Graph Unlearning

• Authors: Zhe-Rui Yang, Jindong Han, Chang-Dong Wang, Hao Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16684
• Pdf URL: https://arxiv.org/pdf/2409.16684
• Copy Paste: [[2409.16684]] Erase then Rectify: A Training-Free Parameter Editing Approach for Cost-Effective Graph Unlearning(https://arxiv.org/abs/2409.16684)
• Keywords: privacy
• Abstract: Graph unlearning, which aims to eliminate the influence of specific nodes, edges, or attributes from a trained Graph Neural Network (GNN), is essential in applications where privacy, bias, or data obsolescence is a concern. However, existing graph unlearning techniques often necessitate additional training on the remaining data, leading to significant computational costs, particularly with large-scale graphs. To address these challenges, we propose a two-stage training-free approach, Erase then Rectify (ETR), designed for efficient and scalable graph unlearning while preserving the model utility. Specifically, we first build a theoretical foundation showing that masking parameters critical for unlearned samples enables effective unlearning. Building on this insight, the Erase stage strategically edits model parameters to eliminate the impact of unlearned samples and their propagated influence on intercorrelated nodes. To further ensure the GNN's utility, the Rectify stage devises a gradient approximation method to estimate the model's gradient on the remaining dataset, which is then used to enhance model performance. Overall, ETR achieves graph unlearning without additional training or full training data access, significantly reducing computational overhead and preserving data privacy. Extensive experiments on seven public datasets demonstrate the consistent superiority of ETR in model utility, unlearning efficiency, and unlearning effectiveness, establishing it as a promising solution for real-world graph unlearning challenges.

Title: Cycle Counting under Local Differential Privacy for Degeneracy-bounded Graphs

• Authors: Quentin Hillebrand, Vorapong Suppakitpaisarn, Tetsuo Shibuya
• Subjects: cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2409.16688
• Pdf URL: https://arxiv.org/pdf/2409.16688
• Copy Paste: [[2409.16688]] Cycle Counting under Local Differential Privacy for Degeneracy-bounded Graphs(https://arxiv.org/abs/2409.16688)
• Keywords: privacy
• Abstract: We propose an algorithm for counting the number of cycles under local differential privacy for degeneracy-bounded input graphs. Numerous studies have focused on counting the number of triangles under the privacy notion, demonstrating that the expected \(\ell_2\)-error of these algorithms is \(\Omega(n^{1.5})\), where \(n\) is the number of nodes in the graph. When parameterized by the number of cycles of length four (\(C_4\)), the best existing triangle counting algorithm has an error of \(O(n^{1.5} + \sqrt{C_4}) = O(n^2)\). In this paper, we introduce an algorithm with an expected \(\ell_2\)-error of \(O(\delta^{1.5} n^{0.5} + \delta^{0.5} d_{\max}^{0.5} n^{0.5})\), where \(\delta\) is the degeneracy and \(d_{\max}\) is the maximum degree of the graph. For degeneracy-bounded graphs (\(\delta \in \Theta(1)\)) commonly found in practical social networks, our algorithm achieves an expected \(\ell_2\)-error of \(O(d_{\max}^{0.5} n^{0.5}) = O(n)\). Our algorithm's core idea is a precise count of triangles following a preprocessing step that approximately sorts the degree of all nodes. This approach can be extended to approximate the number of cycles of length \(k\), maintaining a similar \(\ell_2\)-error, namely $O(\delta^{(k-2)/2} d_{\max}^{0.5} n^{(k-2)/2} + \delta^{k/2} n^{(k-2)/2})$ or $O(d_{\max}^{0.5} n^{(k-2)/2}) = O(n^{(k-1)/2})$ for degeneracy-bounded graphs.

Title: Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model

• Authors: Shoma Iwai, Atsuki Osanai, Shunsuke Kitada, Shinichiro Omachi
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16689
• Pdf URL: https://arxiv.org/pdf/2409.16689
• Copy Paste: [[2409.16689]] Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model(https://arxiv.org/abs/2409.16689)
• Keywords: diffusion
• Abstract: Layout generation is a task to synthesize a harmonious layout with elements characterized by attributes such as category, position, and size. Human designers experiment with the placement and modification of elements to create aesthetic layouts, however, we observed that current discrete diffusion models (DDMs) struggle to correct inharmonious layouts after they have been generated. In this paper, we first provide novel insights into layout sticking phenomenon in DDMs and then propose a simple yet effective layout-assessment module Layout-Corrector, which works in conjunction with existing DDMs to address the layout sticking problem. We present a learning-based module capable of identifying inharmonious elements within layouts, considering overall layout harmony characterized by complex composition. During the generation process, Layout-Corrector evaluates the correctness of each token in the generated layout, reinitializing those with low scores to the ungenerated state. The DDM then uses the high-scored tokens as clues to regenerate the harmonized tokens. Layout-Corrector, tested on common benchmarks, consistently boosts layout-generation performance when in conjunction with various state-of-the-art DDMs. Furthermore, our extensive analysis demonstrates that the Layout-Corrector (1) successfully identifies erroneous tokens, (2) facilitates control over the fidelity-diversity trade-off, and (3) significantly mitigates the performance drop associated with fast sampling.

Title: Probing Omissions and Distortions in Transformer-based RDF-to-Text Models

• Authors: Juliette Faille, Albert Gatt, Claire Gardent
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16707
• Pdf URL: https://arxiv.org/pdf/2409.16707
• Copy Paste: [[2409.16707]] Probing Omissions and Distortions in Transformer-based RDF-to-Text Models(https://arxiv.org/abs/2409.16707)
• Keywords: transformer
• Abstract: In Natural Language Generation (NLG), important information is sometimes omitted in the output text. To better understand and analyse how this type of mistake arises, we focus on RDF-to-Text generation and explore two methods of probing omissions in the encoder output of BART (Lewis et al, 2020) and of T5 (Raffel et al, 2019): (i) a novel parameter-free probing method based on the computation of cosine similarity between embeddings of RDF graphs and of RDF graphs in which we removed some entities and (ii) a parametric probe which performs binary classification on the encoder embeddings to detect omitted entities. We also extend our analysis to distorted entities, i.e. entities that are not fully correctly mentioned in the generated text (e.g. misspelling of entity, wrong units of measurement). We found that both omitted and distorted entities can be probed in the encoder's output embeddings. This suggests that the encoder emits a weaker signal for these entities and therefore is responsible for some loss of information. This also shows that probing methods can be used to detect mistakes in the output of NLG models.

Title: EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models

• Authors: Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, Yu-Gang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16723
• Pdf URL: https://arxiv.org/pdf/2409.16723
• Copy Paste: [[2409.16723]] EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models(https://arxiv.org/abs/2409.16723)
• Keywords: large language model
• Abstract: Recently, Multimodal Large Language Models (MLLMs) have sparked great research interests owing to their exceptional content-reasoning and instruction-following capabilities. To effectively instruct an MLLM, in addition to conventional language expressions, the practice of referring to objects by painting with brushes on images has emerged as a prevalent tool (referred to as "referring visual prompts") due to its efficacy in aligning the user's intention with specific image regions. To accommodate the most common referring visual prompts, namely points, boxes, and masks, existing approaches initially utilize specialized feature encoding modules to capture the semantics of the highlighted areas indicated by these prompts. Subsequently, these encoded region features are adapted to MLLMs through fine-tuning on a meticulously curated multimodal instruction dataset. However, such designs suffer from redundancy in architecture. Moreover, they face challenges in effectively generalizing when encountering a diverse range of arbitrary referring visual prompts in real-life scenarios. To address the above issues, we propose EAGLE, a novel MLLM that empowers comprehension of arbitrary referring visual prompts with less training efforts than existing approaches. Specifically, our EAGLE maintains the innate format of the referring visual prompts as colored patches rendered on the given image for conducting the instruction tuning. Our approach embeds referring visual prompts as spatial concepts conveying specific spatial areas comprehensible to the MLLM, with the semantic comprehension of these regions originating from the MLLM itself. Besides, we also propose a Geometry-Agnostic Learning paradigm (GAL) to further disentangle the MLLM's region-level comprehension with the specific formats of referring visual prompts. Extensive experiments are conducted to prove the effectiveness of our proposed method.

Title: Verified Relative Safety Margins for Neural Network Twins

• Authors: Anahita Baninajjar, Kamran Hosseini, Ahmed Rezine, Amir Aminifar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16726
• Pdf URL: https://arxiv.org/pdf/2409.16726
• Copy Paste: [[2409.16726]] Verified Relative Safety Margins for Neural Network Twins(https://arxiv.org/abs/2409.16726)
• Keywords: robust
• Abstract: Given two Deep Neural Network (DNN) classifiers with the same input and output domains, our goal is to quantify the robustness of the two networks in relation to each other. Towards this, we introduce the notion of Relative Safety Margins (RSMs). Intuitively, given two classes and a common input, RSM of one classifier with respect to another reflects the relative margins with which decisions are made. The proposed notion is relevant in the context of several applications domains, including to compare a trained network and its corresponding compact network (e.g., pruned, quantized, distilled network). Not only can RSMs establish whether decisions are preserved, but they can also quantify their qualities. We also propose a framework to establish safe bounds on RSM gains or losses given an input and a family of perturbations. We evaluate our approach using the MNIST, CIFAR10, and two real-world medical datasets, to show the relevance of our results.

Title: RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems

• Authors: Yihong Tang, Bo Wang, Xu Wang, Dongming Zhao, Jing Liu, Jijun Zhang, Ruifang He, Yuexian Hou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16727
• Pdf URL: https://arxiv.org/pdf/2409.16727
• Copy Paste: [[2409.16727]] RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems(https://arxiv.org/abs/2409.16727)
• Keywords: attack, large language model
• Abstract: Role-playing systems powered by large language models (LLMs) have become increasingly influential in emotional communication applications. However, these systems are susceptible to character hallucinations, where the model deviates from predefined character roles and generates responses that are inconsistent with the intended persona. This paper presents the first systematic analysis of character hallucination from an attack perspective, introducing the RoleBreak framework. Our framework identifies two core mechanisms-query sparsity and role-query conflict-as key factors driving character hallucination. Leveraging these insights, we construct a novel dataset, RoleBreakEval, to evaluate existing hallucination mitigation techniques. Our experiments reveal that even enhanced models trained to minimize hallucination remain vulnerable to attacks. To address these vulnerabilities, we propose a novel defence strategy, the Narrator Mode, which generates supplemental context through narration to mitigate role-query conflicts and improve query generalization. Experimental results demonstrate that Narrator Mode significantly outperforms traditional refusal-based strategies by reducing hallucinations, enhancing fidelity to character roles and queries, and improving overall narrative coherence.

Title: GB-RVFL: Fusion of Randomized Neural Network and Granular Ball Computing

• Authors: M. Sajid, A. Quadir, M. Tanveer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16735
• Pdf URL: https://arxiv.org/pdf/2409.16735
• Copy Paste: [[2409.16735]] GB-RVFL: Fusion of Randomized Neural Network and Granular Ball Computing(https://arxiv.org/abs/2409.16735)
• Keywords: robust
• Abstract: The random vector functional link (RVFL) network is a prominent classification model with strong generalization ability. However, RVFL treats all samples uniformly, ignoring whether they are pure or noisy, and its scalability is limited due to the need for inverting the entire training matrix. To address these issues, we propose granular ball RVFL (GB-RVFL) model, which uses granular balls (GBs) as inputs instead of training samples. This approach enhances scalability by requiring only the inverse of the GB center matrix and improves robustness against noise and outliers through the coarse granularity of GBs. Furthermore, RVFL overlooks the dataset's geometric structure. To address this, we propose graph embedding GB-RVFL (GE-GB-RVFL) model, which fuses granular computing and graph embedding (GE) to preserve the topological structure of GBs. The proposed GB-RVFL and GE-GB-RVFL models are evaluated on KEEL, UCI, NDC and biomedical datasets, demonstrating superior performance compared to baseline models.

Title: E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL

• Authors: Hasan Alp Caferoğlu, Özgür Ulusoy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16751
• Pdf URL: https://arxiv.org/pdf/2409.16751
• Copy Paste: [[2409.16751]] E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL(https://arxiv.org/abs/2409.16751)
• Keywords: large language model
• Abstract: Translating Natural Language Queries into Structured Query Language (Text-to-SQL or NLQ-to-SQL) is a critical task extensively studied by both the natural language processing and database communities, aimed at providing a natural language interface to databases (NLIDB) and lowering the barrier for non-experts. Despite recent advancements made through the use of Large Language Models (LLMs), significant challenges remain. These include handling complex database schemas, resolving ambiguity in user queries, and generating SQL queries with intricate structures that accurately reflect the user's intent. In this work, we introduce E-SQL, a novel pipeline specifically designed to address these challenges through direct schema linking and candidate predicate augmentation. E-SQL enhances the natural language query by incorporating relevant database items (i.e., tables, columns, and values) and conditions directly into the question, bridging the gap between the query and the database structure. The pipeline leverages candidate predicate augmentation to mitigate erroneous or incomplete predicates in generated SQLs. We further investigate the impact of schema filtering, a technique widely explored in previous work, and demonstrate its diminishing returns when applied alongside advanced large language models. Comprehensive evaluations on the BIRD benchmark illustrate that E-SQL achieves competitive performance, particularly excelling in complex queries with a 66.29% execution accuracy on the test set. All code required to reproduce the reported results is publicly available on our GitHub repository.

Title: Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics

• Authors: Lukas Klein, Carsten T. Lüth, Udo Schlegel, Till J. Bungert, Mennatallah El-Assady, Paul F. Jäger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16756
• Pdf URL: https://arxiv.org/pdf/2409.16756
• Copy Paste: [[2409.16756]] Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics(https://arxiv.org/abs/2409.16756)
• Keywords: robust
• Abstract: Explainable AI (XAI) is a rapidly growing domain with a myriad of proposed methods as well as metrics aiming to evaluate their efficacy. However, current studies are often of limited scope, examining only a handful of XAI methods and ignoring underlying design parameters for performance, such as the model architecture or the nature of input data. Moreover, they often rely on one or a few metrics and neglect thorough validation, increasing the risk of selection bias and ignoring discrepancies among metrics. These shortcomings leave practitioners confused about which method to choose for their problem. In response, we introduce LATEC, a large-scale benchmark that critically evaluates 17 prominent XAI methods using 20 distinct metrics. We systematically incorporate vital design parameters like varied architectures and diverse input modalities, resulting in 7,560 examined combinations. Through LATEC, we showcase the high risk of conflicting metrics leading to unreliable rankings and consequently propose a more robust evaluation scheme. Further, we comprehensively evaluate various XAI methods to assist practitioners in selecting appropriate methods aligning with their needs. Curiously, the emerging top-performing method, Expected Gradients, is not examined in any relevant related study. LATEC reinforces its role in future XAI research by publicly releasing all 326k saliency maps and 378k metric scores as a (meta-)evaluation dataset.

Title: MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features

• Authors: Katharina Anderer, Andreas Reich, Matthias Wölfel
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.16765
• Pdf URL: https://arxiv.org/pdf/2409.16765
• Copy Paste: [[2409.16765]] MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features(https://arxiv.org/abs/2409.16765)
• Keywords: robust
• Abstract: This paper presents a benchmark dataset for aligning lecture videos with corresponding slides and introduces a novel multimodal algorithm leveraging features from speech, text, and images. It achieves an average accuracy of 0.82 in comparison to SIFT (0.56) while being approximately 11 times faster. Using dynamic programming the algorithm tries to determine the optimal slide sequence. The results show that penalizing slide transitions increases accuracy. Features obtained via optical character recognition (OCR) contribute the most to a high matching accuracy, followed by image features. The findings highlight that audio transcripts alone provide valuable information for alignment and are beneficial if OCR data is lacking. Variations in matching accuracy across different lectures highlight the challenges associated with video quality and lecture style. The novel multimodal algorithm demonstrates robustness to some of these challenges, underscoring the potential of the approach.

Title: Interpreting Deep Neural Network-Based Receiver Under Varying Signal-To-Noise Ratios

• Authors: Marko Tuononen, Dani Korpi, Ville Hautamäki
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16768
• Pdf URL: https://arxiv.org/pdf/2409.16768
• Copy Paste: [[2409.16768]] Interpreting Deep Neural Network-Based Receiver Under Varying Signal-To-Noise Ratios(https://arxiv.org/abs/2409.16768)
• Keywords: robust
• Abstract: We propose a novel method for interpreting neural networks, focusing on convolutional neural network-based receiver model. The method identifies which unit or units of the model contain most (or least) information about the channel parameter(s) of the interest, providing insights at both global and local levels -- with global explanations aggregating local ones. Experiments on link-level simulations demonstrate the method's effectiveness in identifying units that contribute most (and least) to signal-to-noise ratio processing. Although we focus on a radio receiver model, the method generalizes to other neural network architectures and applications, offering robust estimation even in high-dimensional settings.

Title: MixPolyp: Integrating Mask, Box and Scribble Supervision for Enhanced Polyp Segmentation

• Authors: Yiwen Hu, Jun Wei, Yuncheng Jiang, Haoyang Li, Shuguang Cui, Zhen Li, Song Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16774
• Pdf URL: https://arxiv.org/pdf/2409.16774
• Copy Paste: [[2409.16774]] MixPolyp: Integrating Mask, Box and Scribble Supervision for Enhanced Polyp Segmentation(https://arxiv.org/abs/2409.16774)
• Keywords: segmentation
• Abstract: Limited by the expensive labeling, polyp segmentation models are plagued by data shortages. To tackle this, we propose the mixed supervised polyp segmentation paradigm (MixPolyp). Unlike traditional models relying on a single type of annotation, MixPolyp combines diverse annotation types (mask, box, and scribble) within a single model, thereby expanding the range of available data and reducing labeling costs. To achieve this, MixPolyp introduces three novel supervision losses to handle various annotations: Subspace Projection loss (L_SP), Binary Minimum Entropy loss (L_BME), and Linear Regularization loss (L_LR). For box annotations, L_SP eliminates shape inconsistencies between the prediction and the supervision. For scribble annotations, L_BME provides supervision for unlabeled pixels through minimum entropy constraint, thereby alleviating supervision sparsity. Furthermore, L_LR provides dense supervision by enforcing consistency among the predictions, thus reducing the non-uniqueness. These losses are independent of the model structure, making them generally applicable. They are used only during training, adding no computational cost during inference. Extensive experiments on five datasets demonstrate MixPolyp's effectiveness.

Title: PhD Forum: Efficient Privacy-Preserving Processing via Memory-Centric Computing

• Authors: Mpoki Mwaisela
• Subjects: cs.CR, cs.AR, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.16777
• Pdf URL: https://arxiv.org/pdf/2409.16777
• Copy Paste: [[2409.16777]] PhD Forum: Efficient Privacy-Preserving Processing via Memory-Centric Computing(https://arxiv.org/abs/2409.16777)
• Keywords: secure, security, privacy
• Abstract: Privacy-preserving computation techniques like homomorphic encryption (HE) and secure multi-party computation (SMPC) enhance data security by enabling processing on encrypted data. However, the significant computational and CPU-DRAM data movement overhead resulting from the underlying cryptographic algorithms impedes the adoption of these techniques in practice. Existing approaches focus on improving computational overhead using specialized hardware like GPUs and FPGAs, but these methods still suffer from the same processor-DRAM overhead. Novel hardware technologies that support in-memory processing have the potential to address this problem. Memory-centric computing, or processing-in-memory (PIM), brings computation closer to data by introducing low-power processors called data processing units (DPUs) into memory. Besides its in-memory computation capability, PIM provides extensive parallelism, resulting in significant performance improvement over state-of-the-art approaches. We propose a framework that uses recently available PIM hardware to achieve efficient privacy-preserving computation. Our design consists of a four-layer architecture: (1) an application layer that decouples privacy-preserving applications from the underlying protocols and hardware; (2) a protocol layer that implements existing secure computation protocols (HE and MPC); (3) a data orchestration layer that leverages data compression techniques to mitigate the data transfer overhead between DPUs and host memory; (4) a computation layer which implements DPU kernels on which secure computation algorithms are built.

Title: Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction

• Authors: Jinchuan Zhang, Yan Zhou, Yaxin Liu, Ziming Li, Songlin Hu
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16783
• Pdf URL: https://arxiv.org/pdf/2409.16783
• Copy Paste: [[2409.16783]] Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction(https://arxiv.org/abs/2409.16783)
• Keywords: attack, large language model
• Abstract: Automated red teaming is an effective method for identifying misaligned behaviors in large language models (LLMs). Existing approaches, however, often focus primarily on improving attack success rates while overlooking the need for comprehensive test case coverage. Additionally, most of these methods are limited to single-turn red teaming, failing to capture the multi-turn dynamics of real-world human-machine interactions. To overcome these limitations, we propose HARM (Holistic Automated Red teaMing), which scales up the diversity of test cases using a top-down approach based on an extensible, fine-grained risk taxonomy. Our method also leverages a novel fine-tuning strategy and reinforcement learning techniques to facilitate multi-turn adversarial probing in a human-like manner. Experimental results demonstrate that our framework enables a more systematic understanding of model vulnerabilities and offers more targeted guidance for the alignment process.

Title: Enhancing Feature Selection and Interpretability in AI Regression Tasks Through Feature Attribution

• Authors: Alexander Hinterleitner, Thomas Bartz-Beielstein, Richard Schulz, Sebastian Spengler, Thomas Winter, Christoph Leitenmeier
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16787
• Pdf URL: https://arxiv.org/pdf/2409.16787
• Copy Paste: [[2409.16787]] Enhancing Feature Selection and Interpretability in AI Regression Tasks Through Feature Attribution(https://arxiv.org/abs/2409.16787)
• Keywords: security, robust, interpretability
• Abstract: Research in Explainable Artificial Intelligence (XAI) is increasing, aiming to make deep learning models more transparent. Most XAI methods focus on justifying the decisions made by Artificial Intelligence (AI) systems in security-relevant applications. However, relatively little attention has been given to using these methods to improve the performance and robustness of deep learning algorithms. Additionally, much of the existing XAI work primarily addresses classification problems. In this study, we investigate the potential of feature attribution methods to filter out uninformative features in input data for regression problems, thereby improving the accuracy and stability of predictions. We introduce a feature selection pipeline that combines Integrated Gradients with k-means clustering to select an optimal set of variables from the initial data space. To validate the effectiveness of this approach, we apply it to a real-world industrial problem - blade vibration analysis in the development process of turbo machinery.

Title: Mitigating the Bias of Large Language Model Evaluation

• Authors: Hongli Zhou, Hui Huang, Yunfei Long, Bing Xu, Conghui Zhu, Hailong Cao, Muyun Yang, Tiejun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16788
• Pdf URL: https://arxiv.org/pdf/2409.16788
• Copy Paste: [[2409.16788]] Mitigating the Bias of Large Language Model Evaluation(https://arxiv.org/abs/2409.16788)
• Keywords: large language model
• Abstract: Recently, there has been a trend of evaluating the Large Language Model (LLM) quality in the flavor of LLM-as-a-Judge, namely leveraging another LLM to evaluate the current output quality. However, existing judges are proven to be biased, namely they would favor answers which present better superficial quality (such as verbosity, fluency) while ignoring the instruction following ability. In this work, we propose systematic research about the bias of LLM-as-a-Judge. Specifically, for closed-source judge models, we apply calibration to mitigate the significance of superficial quality, both on probability level and prompt level. For open-source judge models, we propose to mitigate the bias by contrastive training, with curated negative samples that deviate from instruction but present better superficial quality. We apply our methods on the bias evaluation benchmark, and experiment results show our methods mitigate the bias by a large margin while maintaining a satisfactory evaluation accuracy.

Title: Topological SLAM in colonoscopies leveraging deep features and topological priors

• Authors: Javier Morlana, Juan D. Tardós, José M. M. Montiel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16806
• Pdf URL: https://arxiv.org/pdf/2409.16806
• Copy Paste: [[2409.16806]] Topological SLAM in colonoscopies leveraging deep features and topological priors(https://arxiv.org/abs/2409.16806)
• Keywords: transformer
• Abstract: We introduce ColonSLAM, a system that combines classical multiple-map metric SLAM with deep features and topological priors to create topological maps of the whole colon. The SLAM pipeline by itself is able to create disconnected individual metric submaps representing locations from short video subsections of the colon, but is not able to merge covisible submaps due to deformations and the limited performance of the SIFT descriptor in the medical domain. ColonSLAM is guided by topological priors and combines a deep localization network trained to distinguish if two images come from the same place or not and the soft verification of a transformer-based matching network, being able to relate far-in-time submaps during an exploration, grouping them in nodes imaging the same colon place, building more complex maps than any other approach in the literature. We demonstrate our approach in the Endomapper dataset, showing its potential for producing maps of the whole colon in real human explorations. Code and models are available at: this https URL.

Title: A Few Hypocrites: Few-Shot Learning and Subtype Definitions for Detecting Hypocrisy Accusations in Online Climate Change Debates

• Authors: Paulina Garcia Corral, Avishai Green, Hendrik Meyer, Anke Stoll, Xiaoyue Yan, Myrthe Reuver
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16807
• Pdf URL: https://arxiv.org/pdf/2409.16807
• Copy Paste: [[2409.16807]] A Few Hypocrites: Few-Shot Learning and Subtype Definitions for Detecting Hypocrisy Accusations in Online Climate Change Debates(https://arxiv.org/abs/2409.16807)
• Keywords: large language model
• Abstract: The climate crisis is a salient issue in online discussions, and hypocrisy accusations are a central rhetorical element in these debates. However, for large-scale text analysis, hypocrisy accusation detection is an understudied tool, most often defined as a smaller subtask of fallacious argument detection. In this paper, we define hypocrisy accusation detection as an independent task in NLP, and identify different relevant subtypes of hypocrisy accusations. Our Climate Hypocrisy Accusation Corpus (CHAC) consists of 420 Reddit climate debate comments, expert-annotated into two different types of hypocrisy accusations: personal versus political hypocrisy. We evaluate few-shot in-context learning with 6 shots and 3 instruction-tuned Large Language Models (LLMs) for detecting hypocrisy accusations in this dataset. Results indicate that the GPT-4o and Llama-3 models in particular show promise in detecting hypocrisy accusations (F1 reaching 0.68, while previous work shows F1 of 0.44). However, context matters for a complex semantic concept such as hypocrisy accusations, and we find models struggle especially at identifying political hypocrisy accusations compared to personal moral hypocrisy. Our study contributes new insights in hypocrisy detection and climate change discourse, and is a stepping stone for large-scale analysis of hypocrisy accusation in online climate debates.

Title: A parametric framework for kernel-based dynamic mode decomposition using deep learning

• Authors: Konstantinos Kevopoulos, Dongwei Ye
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2409.16817
• Pdf URL: https://arxiv.org/pdf/2409.16817
• Copy Paste: [[2409.16817]] A parametric framework for kernel-based dynamic mode decomposition using deep learning(https://arxiv.org/abs/2409.16817)
• Keywords: diffusion
• Abstract: Surrogate modelling is widely applied in computational science and engineering to mitigate computational efficiency issues for the real-time simulations of complex and large-scale computational models or for many-query scenarios, such as uncertainty quantification and design optimisation. In this work, we propose a parametric framework for kernel-based dynamic mode decomposition method based on the linear and nonlinear disambiguation optimization (LANDO) algorithm. The proposed parametric framework consists of two stages, offline and online. The offline stage prepares the essential component for prediction, namely a series of LANDO models that emulate the dynamics of the system with particular parameters from a training dataset. The online stage leverages those LANDO models to generate new data at a desired time instant, and approximate the mapping between parameters and the state with the data using deep learning techniques. Moreover, dimensionality reduction technique is applied to high-dimensional dynamical systems to reduce the computational cost of training. Three numerical examples including Lotka-Volterra model, heat equation and reaction-diffusion equation are presented to demonstrate the efficiency and effectiveness of the proposed framework.

Title: Spotlight Text Detector: Spotlight on Candidate Regions Like a Camera

• Authors: Xu Han, Junyu Gao, Chuang Yang, Yuan Yuan, Qi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16820
• Pdf URL: https://arxiv.org/pdf/2409.16820
• Copy Paste: [[2409.16820]] Spotlight Text Detector: Spotlight on Candidate Regions Like a Camera(https://arxiv.org/abs/2409.16820)
• Keywords: extraction, segmentation
• Abstract: The irregular contour representation is one of the tough challenges in scene text detection. Although segmentation-based methods have achieved significant progress with the help of flexible pixel prediction, the overlap of geographically close texts hinders detecting them separately. To alleviate this problem, some shrink-based methods predict text kernels and expand them to restructure texts. However, the text kernel is an artificial object with incomplete semantic features that are prone to incorrect or missing detection. In addition, different from the general objects, the geometry features (aspect ratio, scale, and shape) of scene texts vary significantly, which makes it difficult to detect them accurately. To consider the above problems, we propose an effective spotlight text detector (STD), which consists of a spotlight calibration module (SCM) and a multivariate information extraction module (MIEM). The former concentrates efforts on the candidate kernel, like a camera focus on the target. It obtains candidate features through a mapping filter and calibrates them precisely to eliminate some false positive samples. The latter designs different shape schemes to explore multiple geometric features for scene texts. It helps extract various spatial relationships to improve the model's ability to recognize kernel regions. Ablation studies prove the effectiveness of the designed SCM and MIEM. Extensive experiments verify that our STD is superior to existing state-of-the-art methods on various datasets, including ICDAR2015, CTW1500, MSRA-TD500, and Total-Text.

Title: Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability

• Authors: Carlos E. Luis, Alessandro G. Bottero, Julia Vinogradska, Felix Berkenkamp, Jan Peters
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16824
• Pdf URL: https://arxiv.org/pdf/2409.16824
• Copy Paste: [[2409.16824]] Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability(https://arxiv.org/abs/2409.16824)
• Keywords: transformer
• Abstract: Optimal decision-making under partial observability requires reasoning about the uncertainty of the environment's hidden state. However, most reinforcement learning architectures handle partial observability with sequence models that have no internal mechanism to incorporate uncertainty in their hidden state representation, such as recurrent neural networks, deterministic state-space models and transformers. Inspired by advances in probabilistic world models for reinforcement learning, we propose a standalone Kalman filter layer that performs closed-form Gaussian inference in linear state-space models and train it end-to-end within a model-free architecture to maximize returns. Similar to efficient linear recurrent layers, the Kalman filter layer processes sequential data using a parallel scan, which scales logarithmically with the sequence length. By design, Kalman filter layers are a drop-in replacement for other recurrent layers in standard model-free architectures, but importantly they include an explicit mechanism for probabilistic filtering of the latent state representation. Experiments in a wide variety of tasks with partial observability show that Kalman filter layers excel in problems where uncertainty reasoning is key for decision-making, outperforming other stateful models.

Title: Focus Entirety and Perceive Environment for Arbitrary-Shaped Text Detection

• Authors: Xu Han, Junyu Gao, Chuang Yang, Yuan Yuan, Qi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16827
• Pdf URL: https://arxiv.org/pdf/2409.16827
• Copy Paste: [[2409.16827]] Focus Entirety and Perceive Environment for Arbitrary-Shaped Text Detection(https://arxiv.org/abs/2409.16827)
• Keywords: segmentation
• Abstract: Due to the diversity of scene text in aspects such as font, color, shape, and size, accurately and efficiently detecting text is still a formidable challenge. Among the various detection approaches, segmentation-based approaches have emerged as prominent contenders owing to their flexible pixel-level predictions. However, these methods typically model text instances in a bottom-up manner, which is highly susceptible to noise. In addition, the prediction of pixels is isolated without introducing pixel-feature interaction, which also influences the detection performance. To alleviate these problems, we propose a multi-information level arbitrary-shaped text detector consisting of a focus entirety module (FEM) and a perceive environment module (PEM). The former extracts instance-level features and adopts a top-down scheme to model texts to reduce the influence of noises. Specifically, it assigns consistent entirety information to pixels within the same instance to improve their cohesion. In addition, it emphasizes the scale information, enabling the model to distinguish varying scale texts effectively. The latter extracts region-level information and encourages the model to focus on the distribution of positive samples in the vicinity of a pixel, which perceives environment information. It treats the kernel pixels as positive samples and helps the model differentiate text and kernel features. Extensive experiments demonstrate the FEM's ability to efficiently support the model in handling different scale texts and confirm the PEM can assist in perceiving pixels more accurately by focusing on pixel vicinities. Comparisons show the proposed model outperforms existing state-of-the-art approaches on four public datasets.

Title: Explicitly Modeling Pre-Cortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness

• Authors: Lucas Piper, Arlindo L. Oliveira, Tiago Marques
• Subjects: cs.CV, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2409.16838
• Pdf URL: https://arxiv.org/pdf/2409.16838
• Copy Paste: [[2409.16838]] Explicitly Modeling Pre-Cortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness(https://arxiv.org/abs/2409.16838)
• Keywords: robust
• Abstract: While convolutional neural networks (CNNs) excel at clean image classification, they struggle to classify images corrupted with different common corruptions, limiting their real-world applicability. Recent work has shown that incorporating a CNN front-end block that simulates some features of the primate primary visual cortex (V1) can improve overall model robustness. Here, we expand on this approach by introducing two novel biologically-inspired CNN model families that incorporate a new front-end block designed to simulate pre-cortical visual processing. RetinaNet, a hybrid architecture containing the novel front-end followed by a standard CNN back-end, shows a relative robustness improvement of 12.3% when compared to the standard model; and EVNet, which further adds a V1 block after the pre-cortical front-end, shows a relative gain of 18.5%. The improvement in robustness was observed for all the different corruption categories, though accompanied by a small decrease in clean image accuracy, and generalized to a different back-end architecture. These findings show that simulating multiple stages of early visual processing in CNN early layers provides cumulative benefits for model robustness.

Title: IRASNet: Improved Feature-Level Clutter Reduction for Domain Generalized SAR-ATR

• Authors: Oh-Tae Jang, Hae-Kang Song, Min-Jun Kim, Kyung-Hwan Lee, Geon Lee, Sung-Ho Kim, Kyung-Tae Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16845
• Pdf URL: https://arxiv.org/pdf/2409.16845
• Copy Paste: [[2409.16845]] IRASNet: Improved Feature-Level Clutter Reduction for Domain Generalized SAR-ATR(https://arxiv.org/abs/2409.16845)
• Keywords: extraction
• Abstract: Recently, computer-aided design models and electromagnetic simulations have been used to augment synthetic aperture radar (SAR) data for deep learning. However, an automatic target recognition (ATR) model struggles with domain shift when using synthetic data because the model learns specific clutter patterns present in such data, which disturbs performance when applied to measured data with different clutter distributions. This study proposes a framework particularly designed for domain-generalized SAR-ATR called IRASNet, enabling effective feature-level clutter reduction and domain-invariant feature learning. First, we propose a clutter reduction module (CRM) that maximizes the signal-to-clutter ratio on feature maps. The module reduces the impact of clutter at the feature level while preserving target and shadow information, thereby improving ATR performance. Second, we integrate adversarial learning with CRM to extract clutter-reduced domain-invariant features. The integration bridges the gap between synthetic and measured datasets without requiring measured data during training. Third, we improve feature extraction from target and shadow regions by implementing a positional supervision task using mask ground truth encoding. The improvement enhances the ability of the model to discriminate between classes. Our proposed IRASNet presents new state-of-the-art public SAR datasets utilizing target and shadow information to achieve superior performance across various test conditions. IRASNet not only enhances generalization performance but also significantly improves feature-level clutter reduction, making it a valuable advancement in the field of radar image pattern recognition.

Title: Robust Scene Change Detection Using Visual Foundation Models and Cross-Attention Mechanisms

• Authors: Chun-Jung Lin, Sourav Garg, Tat-Jun Chin, Feras Dayoub
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16850
• Pdf URL: https://arxiv.org/pdf/2409.16850
• Copy Paste: [[2409.16850]] Robust Scene Change Detection Using Visual Foundation Models and Cross-Attention Mechanisms(https://arxiv.org/abs/2409.16850)
• Keywords: robust, extraction
• Abstract: We present a novel method for scene change detection that leverages the robust feature extraction capabilities of a visual foundational model, DINOv2, and integrates full-image cross-attention to address key challenges such as varying lighting, seasonal variations, and viewpoint differences. In order to effectively learn correspondences and mis-correspondences between an image pair for the change detection task, we propose to a) ``freeze'' the backbone in order to retain the generality of dense foundation features, and b) employ ``full-image'' cross-attention to better tackle the viewpoint variations between the image pair. We evaluate our approach on two benchmark datasets, VL-CMU-CD and PSCD, along with their viewpoint-varied versions. Our experiments demonstrate significant improvements in F1-score, particularly in scenarios involving geometric changes between image pairs. The results indicate our method's superior generalization capabilities over existing state-of-the-art approaches, showing robustness against photometric and geometric variations as well as better overall generalization when fine-tuned to adapt to new environments. Detailed ablation studies further validate the contributions of each component in our architecture. Source code will be made publicly available upon acceptance.

Title: A Versatile and Differentiable Hand-Object Interaction Representation

• Authors: Théo Morales, Omid Taheri, Gerard Lacey
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16855
• Pdf URL: https://arxiv.org/pdf/2409.16855
• Copy Paste: [[2409.16855]] A Versatile and Differentiable Hand-Object Interaction Representation(https://arxiv.org/abs/2409.16855)
• Keywords: diffusion
• Abstract: Synthesizing accurate hands-object interactions (HOI) is critical for applications in Computer Vision, Augmented Reality (AR), and Mixed Reality (MR). Despite recent advances, the accuracy of reconstructed or generated HOI leaves room for refinement. Some techniques have improved the accuracy of dense correspondences by shifting focus from generating explicit contacts to using rich HOI fields. Still, they lack full differentiability or continuity and are tailored to specific tasks. In contrast, we present a Coarse Hand-Object Interaction Representation (CHOIR), a novel, versatile and fully differentiable field for HOI modelling. CHOIR leverages discrete unsigned distances for continuous shape and pose encoding, alongside multivariate Gaussian distributions to represent dense contact maps with few parameters. To demonstrate the versatility of CHOIR we design JointDiffusion, a diffusion model to learn a grasp distribution conditioned on noisy hand-object interactions or only object geometries, for both refinement and synthesis applications. We demonstrate JointDiffusion's improvements over the SOTA in both applications: it increases the contact F1 score by $5\%$ for refinement and decreases the sim. displacement by $46\%$ for synthesis. Our experiments show that JointDiffusion with CHOIR yield superior contact accuracy and physical realism compared to SOTA methods designed for specific tasks. Our models and code will be publicly available to the research community.

Title: The Role of Language Models in Modern Healthcare: A Comprehensive Review

• Authors: Amna Khalid, Ayma Khalid, Umar Khalid
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16860
• Pdf URL: https://arxiv.org/pdf/2409.16860
• Copy Paste: [[2409.16860]] The Role of Language Models in Modern Healthcare: A Comprehensive Review(https://arxiv.org/abs/2409.16860)
• Keywords: privacy, large language model
• Abstract: The application of large language models (LLMs) in healthcare has gained significant attention due to their ability to process complex medical data and provide insights for clinical decision-making. These models have demonstrated substantial capabilities in understanding and generating natural language, which is crucial for medical documentation, diagnostics, and patient interaction. This review examines the trajectory of language models from their early stages to the current state-of-the-art LLMs, highlighting their strengths in healthcare applications and discussing challenges such as data privacy, bias, and ethical considerations. The potential of LLMs to enhance healthcare delivery is explored, alongside the necessary steps to ensure their ethical and effective integration into medical practice.

Title: Towards Unified 3D Hair Reconstruction from Single-View Portraits

• Authors: Yujian Zheng, Yuda Qiu, Leyang Jin, Chongyang Ma, Haibin Huang, Di Zhang, Pengfei Wan, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16863
• Pdf URL: https://arxiv.org/pdf/2409.16863
• Copy Paste: [[2409.16863]] Towards Unified 3D Hair Reconstruction from Single-View Portraits(https://arxiv.org/abs/2409.16863)
• Keywords: diffusion
• Abstract: Single-view 3D hair reconstruction is challenging, due to the wide range of shape variations among diverse hairstyles. Current state-of-the-art methods are specialized in recovering un-braided 3D hairs and often take braided styles as their failure cases, because of the inherent difficulty to define priors for complex hairstyles, whether rule-based or data-based. We propose a novel strategy to enable single-view 3D reconstruction for a variety of hair types via a unified pipeline. To achieve this, we first collect a large-scale synthetic multi-view hair dataset SynMvHair with diverse 3D hair in both braided and un-braided styles, and learn two diffusion priors specialized on hair. Then we optimize 3D Gaussian-based hair from the priors with two specially designed modules, i.e. view-wise and pixel-wise Gaussian refinement. Our experiments demonstrate that reconstructing braided and un-braided 3D hair from single-view images via a unified approach is possible and our method achieves the state-of-the-art performance in recovering complex hairstyles. It is worth to mention that our method shows good generalization ability to real images, although it learns hair priors from synthetic data.

Title: Linking in Style: Understanding learned features in deep learning models

• Authors: Maren H. Wehrheim, Pamela Osuna-Vargas, Matthias Kaschube
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16865
• Pdf URL: https://arxiv.org/pdf/2409.16865
• Copy Paste: [[2409.16865]] Linking in Style: Understanding learned features in deep learning models(https://arxiv.org/abs/2409.16865)
• Keywords: generative
• Abstract: Convolutional neural networks (CNNs) learn abstract features to perform object classification, but understanding these features remains challenging due to difficult-to-interpret results or high computational costs. We propose an automatic method to visualize and systematically analyze learned features in CNNs. Specifically, we introduce a linking network that maps the penultimate layer of a pre-trained classifier to the latent space of a generative model (StyleGAN-XL), thereby enabling an interpretable, human-friendly visualization of the classifier's representations. Our findings indicate a congruent semantic order in both spaces, enabling a direct linear mapping between them. Training the linking network is computationally inexpensive and decoupled from training both the GAN and the classifier. We introduce an automatic pipeline that utilizes such GAN-based visualizations to quantify learned representations by analyzing activation changes in the classifier in the image domain. This quantification allows us to systematically study the learned representations in several thousand units simultaneously and to extract and visualize units selective for specific semantic concepts. Further, we illustrate how our method can be used to quantify and interpret the classifier's decision boundary using counterfactual examples. Overall, our method offers systematic and objective perspectives on learned abstract representations in CNNs. this https URL

Title: HVT: A Comprehensive Vision Framework for Learning in Non-Euclidean Space

• Authors: Jacob Fein-Ashley, Ethan Feng, Minh Pham
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16897
• Pdf URL: https://arxiv.org/pdf/2409.16897
• Copy Paste: [[2409.16897]] HVT: A Comprehensive Vision Framework for Learning in Non-Euclidean Space(https://arxiv.org/abs/2409.16897)
• Keywords: transformer
• Abstract: Data representation in non-Euclidean spaces has proven effective for capturing hierarchical and complex relationships in real-world datasets. Hyperbolic spaces, in particular, provide efficient embeddings for hierarchical structures. This paper introduces the Hyperbolic Vision Transformer (HVT), a novel extension of the Vision Transformer (ViT) that integrates hyperbolic geometry. While traditional ViTs operate in Euclidean space, our method enhances the self-attention mechanism by leveraging hyperbolic distance and Möbius transformations. This enables more effective modeling of hierarchical and relational dependencies in image data. We present rigorous mathematical formulations, showing how hyperbolic geometry can be incorporated into attention layers, feed-forward networks, and optimization. We offer improved performance for image classification using the ImageNet dataset.

Title: Towards Underwater Camouflaged Object Tracking: An Experimental Evaluation of SAM and SAM 2

• Authors: Chunhui Zhang, Li Liu, Guanjie Huang, Hao Wen, Xi Zhou, Yanfeng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16902
• Pdf URL: https://arxiv.org/pdf/2409.16902
• Copy Paste: [[2409.16902]] Towards Underwater Camouflaged Object Tracking: An Experimental Evaluation of SAM and SAM 2(https://arxiv.org/abs/2409.16902)
• Keywords: segmentation
• Abstract: Over the past decade, significant progress has been made in visual object tracking, largely due to the availability of large-scale training datasets. However, existing tracking datasets are primarily focused on open-air scenarios, which greatly limits the development of object tracking in underwater environments. To address this issue, we take a step forward by proposing the first large-scale underwater camouflaged object tracking dataset, namely UW-COT. Based on the proposed dataset, this paper presents an experimental evaluation of several advanced visual object tracking methods and the latest advancements in image and video segmentation. Specifically, we compare the performance of the Segment Anything Model (SAM) and its updated version, SAM 2, in challenging underwater environments. Our findings highlight the improvements in SAM 2 over SAM, demonstrating its enhanced capability to handle the complexities of underwater camouflaged objects. Compared to current advanced visual object tracking methods, the latest video segmentation foundation model SAM 2 also exhibits significant advantages, providing valuable insights into the development of more effective tracking technologies for underwater scenarios. The dataset will be accessible at \color{magenta}{this https URL}.

Title: Enhancing Temporal Sensitivity and Reasoning for Time-Sensitive Question Answering

• Authors: Wanqi Yang, Yanda Li, Meng Fang, Ling Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16909
• Pdf URL: https://arxiv.org/pdf/2409.16909
• Copy Paste: [[2409.16909]] Enhancing Temporal Sensitivity and Reasoning for Time-Sensitive Question Answering(https://arxiv.org/abs/2409.16909)
• Keywords: large language model
• Abstract: Time-Sensitive Question Answering (TSQA) demands the effective utilization of specific temporal contexts, encompassing multiple time-evolving facts, to address time-sensitive questions. This necessitates not only the parsing of temporal information within questions but also the identification and understanding of time-evolving facts to generate accurate answers. However, current large language models still have limited sensitivity to temporal information and their inadequate temporal reasoning this http URL this paper, we propose a novel framework that enhances temporal awareness and reasoning through Temporal Information-Aware Embedding and Granular Contrastive Reinforcement Learning. Experimental results on four TSQA datasets demonstrate that our framework significantly outperforms existing LLMs in TSQA tasks, marking a step forward in bridging the performance gap between machine and human temporal understanding and reasoning.

Title: Pruning Multilingual Large Language Models for Multilingual Inference

• Authors: Hwichan Kim, Jun Suzuki, Tosho Hirasawa, Mamoru Komachi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16911
• Pdf URL: https://arxiv.org/pdf/2409.16911
• Copy Paste: [[2409.16911]] Pruning Multilingual Large Language Models for Multilingual Inference(https://arxiv.org/abs/2409.16911)
• Keywords: large language model
• Abstract: Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. A distinctive characteristic of MLLMs is their high-quality translation capabilities, indicating an acquired proficiency in aligning between languages. This study explores how to enhance the zero-shot performance of MLLMs in non-English languages by leveraging their alignment capability between English and non-English languages. To achieve this, we first analyze the behavior of MLLMs when performing translation and reveal that there are large magnitude features that play a critical role in the translation process. Inspired by these findings, we retain the weights associated with operations involving the large magnitude features and prune other weights to force MLLMs to rely on these features for tasks beyond translation. We empirically demonstrate that this pruning strategy can enhance the MLLMs' performance in non-English language.

Title: Zero-Shot Detection of LLM-Generated Text using Token Cohesiveness

• Authors: Shixuan Ma, Quan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16914
• Pdf URL: https://arxiv.org/pdf/2409.16914
• Copy Paste: [[2409.16914]] Zero-Shot Detection of LLM-Generated Text using Token Cohesiveness(https://arxiv.org/abs/2409.16914)
• Keywords: large language model
• Abstract: The increasing capability and widespread usage of large language models (LLMs) highlight the desirability of automatic detection of LLM-generated text. Zero-shot detectors, due to their training-free nature, have received considerable attention and notable success. In this paper, we identify a new feature, token cohesiveness, that is useful for zero-shot detection, and we demonstrate that LLM-generated text tends to exhibit higher token cohesiveness than human-written text. Based on this observation, we devise TOCSIN, a generic dual-channel detection paradigm that uses token cohesiveness as a plug-and-play module to improve existing zero-shot detectors. To calculate token cohesiveness, TOCSIN only requires a few rounds of random token deletion and semantic difference measurement, making it particularly suitable for a practical black-box setting where the source model used for generation is not accessible. Extensive experiments with four state-of-the-art base detectors on various datasets, source models, and evaluation settings demonstrate the effectiveness and generality of the proposed approach. Code available at: \url{this https URL}.

Title: Game4Loc: A UAV Geo-Localization Benchmark from Game Data

• Authors: Yuxiang Ji, Boyong He, Zhuoyue Tan, Liaoni Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16925
• Pdf URL: https://arxiv.org/pdf/2409.16925
• Copy Paste: [[2409.16925]] Game4Loc: A UAV Geo-Localization Benchmark from Game Data(https://arxiv.org/abs/2409.16925)
• Keywords: privacy
• Abstract: The vision-based geo-localization technology for UAV, serving as a secondary source of GPS information in addition to the global navigation satellite systems (GNSS), can still operate independently in the GPS-denied environment. Recent deep learning based methods attribute this as the task of image matching and retrieval. By retrieving drone-view images in geo-tagged satellite image database, approximate localization information can be obtained. However, due to high costs and privacy concerns, it is usually difficult to obtain large quantities of drone-view images from a continuous area. Existing drone-view datasets are mostly composed of small-scale aerial photography with a strong assumption that there exists a perfect one-to-one aligned reference image for any query, leaving a significant gap from the practical localization scenario. In this work, we construct a large-range contiguous area UAV geo-localization dataset named GTA-UAV, featuring multiple flight altitudes, attitudes, scenes, and targets using modern computer games. Based on this dataset, we introduce a more practical UAV geo-localization task including partial matches of cross-view paired data, and expand the image-level retrieval to the actual localization in terms of distance (meters). For the construction of drone-view and satellite-view pairs, we adopt a weight-based contrastive learning approach, which allows for effective learning while avoiding additional post-processing matching steps. Experiments demonstrate the effectiveness of our data and training method for UAV geo-localization, as well as the generalization capabilities to real-world scenarios.

Title: Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents

• Authors: Emanuela Boros, Maud Ehrmann
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16934
• Pdf URL: https://arxiv.org/pdf/2409.16934
• Copy Paste: [[2409.16934]] Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents(https://arxiv.org/abs/2409.16934)
• Keywords: transformer, large language model
• Abstract: This paper investigates the presence of OCR-sensitive neurons within the Transformer architecture and their influence on named entity recognition (NER) performance on historical documents. By analysing neuron activation patterns in response to clean and noisy text inputs, we identify and then neutralise OCR-sensitive neurons to improve model performance. Based on two open access large language models (Llama2 and Mistral), experiments demonstrate the existence of OCR-sensitive regions and show improvements in NER performance on historical newspapers and classical commentaries, highlighting the potential of targeted neuron modulation to improve models' performance on noisy text.

Title: Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Model

• Authors: Hongliang Zhong, Can Wang, Jingbo Zhang, Jing Liao
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.16938
• Pdf URL: https://arxiv.org/pdf/2409.16938
• Copy Paste: [[2409.16938]] Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Model(https://arxiv.org/abs/2409.16938)
• Keywords: diffusion, generative
• Abstract: Generating and inserting new objects into 3D content is a compelling approach for achieving versatile scene recreation. Existing methods, which rely on SDS optimization or single-view inpainting, often struggle to produce high-quality results. To address this, we propose a novel method for object insertion in 3D content represented by Gaussian Splatting. Our approach introduces a multi-view diffusion model, dubbed MVInpainter, which is built upon a pre-trained stable video diffusion model to facilitate view-consistent object inpainting. Within MVInpainter, we incorporate a ControlNet-based conditional injection module to enable controlled and more predictable multi-view generation. After generating the multi-view inpainted results, we further propose a mask-aware 3D reconstruction technique to refine Gaussian Splatting reconstruction from these sparse inpainted views. By leveraging these fabricate techniques, our approach yields diverse results, ensures view-consistent and harmonious insertions, and produces better object quality. Extensive experiments demonstrate that our approach outperforms existing methods.

Title: Face Forgery Detection with Elaborate Backbone

• Authors: Zonghui Guo, Yingjie Liu, Jie Zhang, Haiyong Zheng, Shiguang Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16945
• Pdf URL: https://arxiv.org/pdf/2409.16945
• Copy Paste: [[2409.16945]] Face Forgery Detection with Elaborate Backbone(https://arxiv.org/abs/2409.16945)
• Keywords: attack
• Abstract: Face Forgery Detection (FFD), or Deepfake detection, aims to determine whether a digital face is real or fake. Due to different face synthesis algorithms with diverse forgery patterns, FFD models often overfit specific patterns in training datasets, resulting in poor generalization to other unseen forgeries. This severe challenge requires FFD models to possess strong capabilities in representing complex facial features and extracting subtle forgery cues. Although previous FFD models directly employ existing backbones to represent and extract facial forgery cues, the critical role of backbones is often overlooked, particularly as their knowledge and capabilities are insufficient to address FFD challenges, inevitably limiting generalization. Therefore, it is essential to integrate the backbone pre-training configurations and seek practical solutions by revisiting the complete FFD workflow, from backbone pre-training and fine-tuning to inference of discriminant results. Specifically, we analyze the crucial contributions of backbones with different configurations in FFD task and propose leveraging the ViT network with self-supervised learning on real-face datasets to pre-train a backbone, equipping it with superior facial representation capabilities. We then build a competitive backbone fine-tuning framework that strengthens the backbone's ability to extract diverse forgery cues within a competitive learning mechanism. Moreover, we devise a threshold optimization mechanism that utilizes prediction confidence to improve the inference reliability. Comprehensive experiments demonstrate that our FFD model with the elaborate backbone achieves excellent performance in FFD and extra face-related tasks, i.e., presentation attack detection. Code and models are available at this https URL.

Title: DALDA: Data Augmentation Leveraging Diffusion Model and LLM with Adaptive Guidance Scaling

• Authors: Kyuheon Jung, Yongdeuk Seo, Seongwoo Cho, Jaeyoung Kim, Hyun-seok Min, Sungchul Choi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16949
• Pdf URL: https://arxiv.org/pdf/2409.16949
• Copy Paste: [[2409.16949]] DALDA: Data Augmentation Leveraging Diffusion Model and LLM with Adaptive Guidance Scaling(https://arxiv.org/abs/2409.16949)
• Keywords: diffusion, large language model
• Abstract: In this paper, we present an effective data augmentation framework leveraging the Large Language Model (LLM) and Diffusion Model (DM) to tackle the challenges inherent in data-scarce scenarios. Recently, DMs have opened up the possibility of generating synthetic images to complement a few training images. However, increasing the diversity of synthetic images also raises the risk of generating samples outside the target distribution. Our approach addresses this issue by embedding novel semantic information into text prompts via LLM and utilizing real images as visual prompts, thus generating semantically rich images. To ensure that the generated images remain within the target distribution, we dynamically adjust the guidance weight based on each image's CLIPScore to control the diversity. Experimental results show that our method produces synthetic images with enhanced diversity while maintaining adherence to the target distribution. Consequently, our approach proves to be more efficient in the few-shot setting on several benchmarks. Our code is available at this https URL .

Title: RESAA: A Removal and Structural Analysis Attack Against Compound Logic Locking

• Authors: Felipe Almeida, Levent Aksoy, Samuel Pagliarini
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16959
• Pdf URL: https://arxiv.org/pdf/2409.16959
• Copy Paste: [[2409.16959]] RESAA: A Removal and Structural Analysis Attack Against Compound Logic Locking(https://arxiv.org/abs/2409.16959)
• Keywords: security, protect, attack
• Abstract: The semiconductor industry's paradigm shift towards fabless integrated circuit (IC) manufacturing has introduced security threats, including piracy, counterfeiting, hardware Trojans, and overproduction. In response to these challenges, various countermeasures, including Logic locking (LL), have been proposed to protect designs and mitigate security risks. LL is likely the most researched form of intellectual property (IP) protection for ICs. A significant advance has been made with the introduction of compound logic locking (CLL), where two LL techniques are concurrently utilized for improved resiliency against attacks. However, the vulnerabilities of LL techniques, particularly CLL, need to be explored further. This paper presents a novel framework, RESAA, designed to classify CLL-locked designs, identify critical gates, and execute various attacks to uncover secret keys. RESAA is agnostic to specific LL techniques, offering comprehensive insights into CLL's security scenarios. Experimental results demonstrate RESAA's efficacy in identifying critical gates, distinguishing segments corresponding to different LL techniques, and determining associated keys based on different threat models. In particular, for the oracle-less threat model, RESAA can achieve up to 92.6% accuracy on a relatively complex ITC'99 benchmark circuit. The results reported in this paper emphasize the significance of evaluation and thoughtful selection of LL techniques, as all studied CLL variants demonstrated vulnerability to our framework. RESAA is also open-sourced for the community at large.

Title: ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods

• Authors: MaryBeth Defrance, Maarten Buyl, Tijl De Bie
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.16965
• Pdf URL: https://arxiv.org/pdf/2409.16965
• Copy Paste: [[2409.16965]] ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods(https://arxiv.org/abs/2409.16965)
• Keywords: fair
• Abstract: Numerous methods have been implemented that pursue fairness with respect to sensitive features by mitigating biases in machine learning. Yet, the problem settings that each method tackles vary significantly, including the stage of intervention, the composition of sensitive features, the fairness notion, and the distribution of the output. Even in binary classification, these subtle differences make it highly complicated to benchmark fairness methods, as their performance can strongly depend on exactly how the bias mitigation problem was originally framed. Hence, we introduce ABCFair, a benchmark approach which allows adapting to the desiderata of the real-world problem setting, enabling proper comparability between methods for any use case. We apply ABCFair to a range of pre-, in-, and postprocessing methods on both large-scale, traditional datasets and on a dual label (biased and unbiased) dataset to sidestep the fairness-accuracy trade-off.

Title: Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization

• Authors: Rafael Mendoza, Isabella Cruz, Richard Liu, Aarav Deshmukh, David Williams, Jesscia Peng, Rohan Iyer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16973
• Pdf URL: https://arxiv.org/pdf/2409.16973
• Copy Paste: [[2409.16973]] Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization(https://arxiv.org/abs/2409.16973)
• Keywords: large language model
• Abstract: Large language models (LLMs) have revolutionized how we interact with technology, but their personalization to individual user preferences remains a significant challenge, particularly in on-device applications. Traditional methods often depend heavily on labeled datasets and can be resource-intensive. To address these issues, we present Adaptive Self-Supervised Learning Strategies (ASLS), which utilizes self-supervised learning techniques to personalize LLMs dynamically. The framework comprises a user profiling layer for collecting interaction data and a neural adaptation layer for real-time model fine-tuning. This innovative approach enables continuous learning from user feedback, allowing the model to generate responses that align closely with user-specific contexts. The adaptive mechanisms of ASLS minimize computational demands and enhance personalization efficiency. Experimental results across various user scenarios illustrate the superior performance of ASLS in boosting user engagement and satisfaction, highlighting its potential to redefine LLMs as highly responsive and context-aware systems on-device.

Title: Decoding Large-Language Models: A Systematic Overview of Socio-Technical Impacts, Constraints, and Emerging Questions

• Authors: Zeyneb N. Kaya, Souvick Ghosh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16974
• Pdf URL: https://arxiv.org/pdf/2409.16974
• Copy Paste: [[2409.16974]] Decoding Large-Language Models: A Systematic Overview of Socio-Technical Impacts, Constraints, and Emerging Questions(https://arxiv.org/abs/2409.16974)
• Keywords: large language model
• Abstract: There have been rapid advancements in the capabilities of large language models (LLMs) in recent years, greatly revolutionizing the field of natural language processing (NLP) and artificial intelligence (AI) to understand and interact with human language. Therefore, in this work, we conduct a systematic investigation of the literature to identify the prominent themes and directions of LLM developments, impacts, and limitations. Our findings illustrate the aims, methodologies, limitations, and future directions of LLM research. It includes responsible development considerations, algorithmic improvements, ethical challenges, and societal implications of LLM development. Overall, this paper provides a rigorous and comprehensive overview of current research in LLM and identifies potential directions for future development. The article highlights the application areas that could have a positive impact on society along with the ethical considerations.

Title: Single Image, Any Face: Generalisable 3D Face Generation

• Authors: Wenqing Wang, Haosen Yang, Josef Kittler, Xiatian Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16990
• Pdf URL: https://arxiv.org/pdf/2409.16990
• Copy Paste: [[2409.16990]] Single Image, Any Face: Generalisable 3D Face Generation(https://arxiv.org/abs/2409.16990)
• Keywords: diffusion, generative
• Abstract: The creation of 3D human face avatars from a single unconstrained image is a fundamental task that underlies numerous real-world vision and graphics applications. Despite the significant progress made in generative models, existing methods are either less suited in design for human faces or fail to generalise from the restrictive training domain to unconstrained facial images. To address these limitations, we propose a novel model, Gen3D-Face, which generates 3D human faces with unconstrained single image input within a multi-view consistent diffusion framework. Given a specific input image, our model first produces multi-view images, followed by neural surface construction. To incorporate face geometry information in a generalisable manner, we utilise input-conditioned mesh estimation instead of ground-truth mesh along with synthetic multi-view training data. Importantly, we introduce a multi-view joint generation scheme to enhance appearance consistency among different views. To the best of our knowledge, this is the first attempt and benchmark for creating photorealistic 3D human face avatars from single images for generic human subject across domains. Extensive experiments demonstrate the superiority of our method over previous alternatives for out-of-domain singe image 3D face generation and top competition for in-domain setting.

Title: INT-FlashAttention: Enabling Flash Attention for INT8 Quantization

• Authors: Shimao Chen, Zirui Liu, Zhiying Wu, Ce Zheng, Peizhuang Cong, Zihan Jiang, Lei Su, Tong Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16997
• Pdf URL: https://arxiv.org/pdf/2409.16997
• Copy Paste: [[2409.16997]] INT-FlashAttention: Enabling Flash Attention for INT8 Quantization(https://arxiv.org/abs/2409.16997)
• Keywords: large language model
• Abstract: As the foundation of large language models (LLMs), self-attention module faces the challenge of quadratic time and memory complexity with respect to sequence length. FlashAttention accelerates attention computation and reduces its memory usage by leveraging the GPU memory hierarchy. A promising research direction is to integrate FlashAttention with quantization methods. This paper introduces INT-FlashAttention, the first INT8 quantization architecture compatible with the forward workflow of FlashAttention, which significantly improves the inference speed of FlashAttention on Ampere GPUs. We implement our INT-FlashAttention prototype with fully INT8 activations and general matrix-multiplication (GEMM) kernels, making it the first attention operator with fully INT8 input. As a general token-level post-training quantization framework, INT-FlashAttention is also compatible with other data formats like INT4, etc. Experimental results show INT-FlashAttention achieves 72% faster inference speed and 82% smaller quantization error compared to standard FlashAttention with FP16 and FP8 data format.

Title: LLM-CARD: Towards a Description and Landscape of Large Language Models

• Authors: Shengwei Tian, Lifeng Han, Erick Mendez Guzman, Goran Nenadic
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2409.17011
• Pdf URL: https://arxiv.org/pdf/2409.17011
• Copy Paste: [[2409.17011]] LLM-CARD: Towards a Description and Landscape of Large Language Models(https://arxiv.org/abs/2409.17011)
• Keywords: extraction, large language model
• Abstract: With the rapid growth of the Natural Language Processing (NLP) field, a vast variety of Large Language Models (LLMs) continue to emerge for diverse NLP tasks. As an increasing number of papers are presented, researchers and developers face the challenge of information overload. Thus, it is particularly important to develop a system that can automatically extract and organise key information about LLMs from academic papers (\textbf{LLM model card}). This work is to develop such a pioneer system by using Named Entity Recognition (\textbf{NER}) and Relation Extraction (\textbf{RE}) methods that automatically extract key information about large language models from the papers, helping researchers to efficiently access information about LLMs. These features include model \textit{licence}, model \textit{name}, and model \textit{application}. With these features, we can form a model card for each paper. \textbf{Data-contribution} wise, 106 academic papers were processed by defining three dictionaries - LLMs name, licence, and application. 11,051 sentences were extracted through dictionary lookup, and the dataset was constructed through manual review of the final selection of 129 sentences that have a link between the name and the licence, and 106 sentences that have a link between the model name and the application.

Title: PTQ4RIS: Post-Training Quantization for Referring Image Segmentation

• Authors: Xiaoyan Jiang, Hang Yang, Kaiying Zhu, Xihe Qiu, Shibo Zhao, Sifan Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17020
• Pdf URL: https://arxiv.org/pdf/2409.17020
• Copy Paste: [[2409.17020]] PTQ4RIS: Post-Training Quantization for Referring Image Segmentation(https://arxiv.org/abs/2409.17020)
• Keywords: segmentation
• Abstract: Referring Image Segmentation (RIS), aims to segment the object referred by a given sentence in an image by understanding both visual and linguistic information. However, existing RIS methods tend to explore top-performance models, disregarding considerations for practical applications on resources-limited edge devices. This oversight poses a significant challenge for on-device RIS inference. To this end, we propose an effective and efficient post-training quantization framework termed PTQ4RIS. Specifically, we first conduct an in-depth analysis of the root causes of performance degradation in RIS model quantization and propose dual-region quantization (DRQ) and reorder-based outlier-retained quantization (RORQ) to address the quantization difficulties in visual and text encoders. Extensive experiments on three benchmarks with different bits settings (from 8 to 4 bits) demonstrates its superior performance. Importantly, we are the first PTQ method specifically designed for the RIS task, highlighting the feasibility of PTQ in RIS applications. Code will be available at {this https URL}.

Title: Enhanced Wavelet Scattering Network for image inpainting detection

• Authors: Barglazan Adrian-Alin, Brad Remus
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17023
• Pdf URL: https://arxiv.org/pdf/2409.17023
• Copy Paste: [[2409.17023]] Enhanced Wavelet Scattering Network for image inpainting detection(https://arxiv.org/abs/2409.17023)
• Keywords: robust, extraction, segmentation
• Abstract: The rapid advancement of image inpainting tools, especially those aimed at removing artifacts, has made digital image manipulation alarmingly accessible. This paper proposes several innovative ideas for detecting inpainting forgeries based on low level noise analysis by combining Dual-Tree Complex Wavelet Transform (DT-CWT) for feature extraction with convolutional neural networks (CNN) for forged area detection and localization, and lastly by employing an innovative combination of texture segmentation with noise variance estimations. The DT-CWT offers significant advantages due to its shift-invariance, enhancing its robustness against subtle manipulations during the inpainting process. Furthermore, its directional selectivity allows for the detection of subtle artifacts introduced by inpainting within specific frequency bands and orientations. Various neural network architectures were evaluated and proposed. Lastly, we propose a fusion detection module that combines texture analysis with noise variance estimation to give the forged area. Our approach was benchmarked against state-of-the-art methods and demonstrated superior performance over all cited alternatives. The training code (with pretrained model weights) as long as the dataset will be available at this https URL

Title: Counterfactual Token Generation in Large Language Models

• Authors: Ivi Chatzi, Nina Corvelo Benz, Eleni Straitouri, Stratis Tsirtsis, Manuel Gomez-Rodriguez
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17027
• Pdf URL: https://arxiv.org/pdf/2409.17027
• Copy Paste: [[2409.17027]] Counterfactual Token Generation in Large Language Models(https://arxiv.org/abs/2409.17027)
• Keywords: large language model
• Abstract: "Sure, I am happy to generate a story for you: Captain Lyra stood at the helm of her trusty ship, the Maelstrom's Fury, gazing out at the endless sea. [...] Lyra's eyes welled up with tears as she realized the bitter truth - she had sacrificed everything for fleeting riches, and lost the love of her crew, her family, and herself." Although this story, generated by a large language model, is captivating, one may wonder -- how would the story have unfolded if the model had chosen "Captain Maeve" as the protagonist instead? We cannot know. State-of-the-art large language models are stateless -- they maintain no internal memory or state. Given a prompt, they generate a sequence of tokens as an output using an autoregressive process. As a consequence, they cannot reason about counterfactual alternatives to tokens they have generated in the past. In this work, our goal is to enhance them with this functionality. To this end, we develop a causal model of token generation that builds upon the Gumbel-Max structural causal model. Our model allows any large language model to perform counterfactual token generation at almost no cost in comparison with vanilla token generation, it is embarrassingly simple to implement, and it does not require any fine-tuning nor prompt engineering. We implement our model on Llama 3 8B-instruct and conduct both qualitative and quantitative analyses of counterfactually generated text. We conclude with a demonstrative application of counterfactual token generation for bias detection, unveiling interesting insights about the model of the world constructed by large language models.

Title: EventHDR: from Event to High-Speed HDR Videos and Beyond

• Authors: Yunhao Zou, Ying Fu, Tsuyoshi Takatani, Yinqiang Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17029
• Pdf URL: https://arxiv.org/pdf/2409.17029
• Copy Paste: [[2409.17029]] EventHDR: from Event to High-Speed HDR Videos and Beyond(https://arxiv.org/abs/2409.17029)
• Keywords: segmentation
• Abstract: Event cameras are innovative neuromorphic sensors that asynchronously capture the scene dynamics. Due to the event-triggering mechanism, such cameras record event streams with much shorter response latency and higher intensity sensitivity compared to conventional cameras. On the basis of these features, previous works have attempted to reconstruct high dynamic range (HDR) videos from events, but have either suffered from unrealistic artifacts or failed to provide sufficiently high frame rates. In this paper, we present a recurrent convolutional neural network that reconstruct high-speed HDR videos from event sequences, with a key frame guidance to prevent potential error accumulation caused by the sparse event data. Additionally, to address the problem of severely limited real dataset, we develop a new optical system to collect a real-world dataset with paired high-speed HDR videos and event streams, facilitating future research in this field. Our dataset provides the first real paired dataset for event-to-HDR reconstruction, avoiding potential inaccuracies from simulation strategies. Experimental results demonstrate that our method can generate high-quality, high-speed HDR videos. We further explore the potential of our work in cross-camera reconstruction and downstream computer vision tasks, including object detection, panoramic segmentation, optical flow estimation, and monocular depth estimation under HDR scenarios.

Title: How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not

• Authors: Francesco Verdini, Pierfrancesco Melucci, Stefano Perna, Francesco Cariaggi, Marco Gaido, Sara Papi, Szymon Mazurek, Marek Kasztelnik, Luisa Bentivogli, Sébastien Bratières, Paolo Merialdo, Simone Scardapane
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17044
• Pdf URL: https://arxiv.org/pdf/2409.17044
• Copy Paste: [[2409.17044]] How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not(https://arxiv.org/abs/2409.17044)
• Keywords: large language model
• Abstract: The remarkable performance achieved by Large Language Models (LLM) has driven research efforts to leverage them for a wide range of tasks and input modalities. In speech-to-text (S2T) tasks, the emerging solution consists of projecting the output of the encoder of a Speech Foundational Model (SFM) into the LLM embedding space through an adapter module. However, no work has yet investigated how much the downstream-task performance depends on each component (SFM, adapter, LLM) nor whether the best design of the adapter depends on the chosen SFM and LLM. To fill this gap, we evaluate the combination of 5 adapter modules, 2 LLMs (Mistral and Llama), and 2 SFMs (Whisper and SeamlessM4T) on two widespread S2T tasks, namely Automatic Speech Recognition and Speech Translation. Our results demonstrate that the SFM plays a pivotal role in downstream performance, while the adapter choice has moderate impact and depends on the SFM and LLM.

Title: GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Design

• Authors: Phillip Mueller, Sebastian Mueller, Lars Mikelsons
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17045
• Pdf URL: https://arxiv.org/pdf/2409.17045
• Copy Paste: [[2409.17045]] GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Design(https://arxiv.org/abs/2409.17045)
• Keywords: diffusion, generative
• Abstract: We provide a dataset for enabling Deep Generative Models (DGMs) in engineering design and propose methods to automate data labeling by utilizing large-scale foundation models. GeoBiked is curated to contain 4 355 bicycle images, annotated with structural and technical features and is used to investigate two automated labeling techniques: The utilization of consolidated latent features (Hyperfeatures) from image-generation models to detect geometric correspondences (e.g. the position of the wheel center) in structural images and the generation of diverse text descriptions for structural images. GPT-4o, a vision-language-model (VLM), is instructed to analyze images and produce diverse descriptions aligned with the system-prompt. By representing technical images as Diffusion-Hyperfeatures, drawing geometric correspondences between them is possible. The detection accuracy of geometric points in unseen samples is improved by presenting multiple annotated source images. GPT-4o has sufficient capabilities to generate accurate descriptions of technical images. Grounding the generation only on images leads to diverse descriptions but causes hallucinations, while grounding it on categorical labels restricts the diversity. Using both as input balances creativity and accuracy. Successfully using Hyperfeatures for geometric correspondence suggests that this approach can be used for general point-detection and annotation tasks in technical images. Labeling such images with text descriptions using VLMs is possible, but dependent on the models detection capabilities, careful prompt-engineering and the selection of input information. Applying foundation models in engineering design is largely unexplored. We aim to bridge this gap with a dataset to explore training, finetuning and conditioning DGMs in this field and suggesting approaches to bootstrap foundation models to process technical images.

Title: ControlCity: A Multimodal Diffusion Model Based Approach for Accurate Geospatial Data Generation and Urban Morphology Analysis

• Authors: Fangshuo Zhou, Huaxia Li, Rui Hu, Sensen Wu, Hailin Feng, Zhenhong Du, Liuchang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17049
• Pdf URL: https://arxiv.org/pdf/2409.17049
• Copy Paste: [[2409.17049]] ControlCity: A Multimodal Diffusion Model Based Approach for Accurate Geospatial Data Generation and Urban Morphology Analysis(https://arxiv.org/abs/2409.17049)
• Keywords: diffusion
• Abstract: Volunteer Geographic Information (VGI), with its rich variety, large volume, rapid updates, and diverse sources, has become a critical source of geospatial data. However, VGI data from platforms like OSM exhibit significant quality heterogeneity across different data types, particularly with urban building data. To address this, we propose a multi-source geographic data transformation solution, utilizing accessible and complete VGI data to assist in generating urban building footprint data. We also employ a multimodal data generation framework to improve accuracy. First, we introduce a pipeline for constructing an 'image-text-metadata-building footprint' dataset, primarily based on road network data and supplemented by other multimodal data. We then present ControlCity, a geographic data transformation method based on a multimodal diffusion model. This method first uses a pre-trained text-to-image model to align text, metadata, and building footprint data. An improved ControlNet further integrates road network and land-use imagery, producing refined building footprint data. Experiments across 22 global cities demonstrate that ControlCity successfully simulates real urban building patterns, achieving state-of-the-art performance. Specifically, our method achieves an average FID score of 50.94, reducing error by 71.01% compared to leading methods, and a MIoU score of 0.36, an improvement of 38.46%. Additionally, our model excels in tasks like urban morphology transfer, zero-shot city generation, and spatial data completeness assessment. In the zero-shot city task, our method accurately predicts and generates similar urban structures, demonstrating strong generalization. This study confirms the effectiveness of our approach in generating urban building footprint data and capturing complex city characteristics.

Title: Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors

• Authors: Aiping Zhang, Zongsheng Yue, Renjing Pei, Wenqi Ren, Xiaochun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17058
• Pdf URL: https://arxiv.org/pdf/2409.17058
• Copy Paste: [[2409.17058]] Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors(https://arxiv.org/abs/2409.17058)
• Keywords: diffusion, generative
• Abstract: Diffusion-based image super-resolution (SR) methods have achieved remarkable success by leveraging large pre-trained text-to-image diffusion models as priors. However, these methods still face two challenges: the requirement for dozens of sampling steps to achieve satisfactory results, which limits efficiency in real scenarios, and the neglect of degradation models, which are critical auxiliary information in solving the SR problem. In this work, we introduced a novel one-step SR model, which significantly addresses the efficiency issue of diffusion-based SR methods. Unlike existing fine-tuning strategies, we designed a degradation-guided Low-Rank Adaptation (LoRA) module specifically for SR, which corrects the model parameters based on the pre-estimated degradation information from low-resolution images. This module not only facilitates a powerful data-dependent or degradation-dependent SR model but also preserves the generative prior of the pre-trained diffusion model as much as possible. Furthermore, we tailor a novel training pipeline by introducing an online negative sample generation strategy. Combined with the classifier-free guidance strategy during inference, it largely improves the perceptual quality of the super-resolution results. Extensive experiments have demonstrated the superior efficiency and effectiveness of the proposed model compared to recent state-of-the-art methods.

Title: Benchmarking Domain Generalization Algorithms in Computational Pathology

• Authors: Neda Zamanitajeddin, Mostafa Jahanifar, Kesi Xu, Fouzia Siraj, Nasir Rajpoot
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17063
• Pdf URL: https://arxiv.org/pdf/2409.17063
• Copy Paste: [[2409.17063]] Benchmarking Domain Generalization Algorithms in Computational Pathology(https://arxiv.org/abs/2409.17063)
• Keywords: robust
• Abstract: Deep learning models have shown immense promise in computational pathology (CPath) tasks, but their performance often suffers when applied to unseen data due to domain shifts. Addressing this requires domain generalization (DG) algorithms. However, a systematic evaluation of DG algorithms in the CPath context is lacking. This study aims to benchmark the effectiveness of 30 DG algorithms on 3 CPath tasks of varying difficulty through 7,560 cross-validation runs. We evaluate these algorithms using a unified and robust platform, incorporating modality-specific techniques and recent advances like pretrained foundation models. Our extensive cross-validation experiments provide insights into the relative performance of various DG strategies. We observe that self-supervised learning and stain augmentation consistently outperform other methods, highlighting the potential of pretrained models and data augmentation. Furthermore, we introduce a new pan-cancer tumor detection dataset (HISTOPANTUM) as a benchmark for future research. This study offers valuable guidance to researchers in selecting appropriate DG approaches for CPath tasks.

Title: Efficient Feature Interactions with Transformers: Improving User Spending Propensity Predictions in Gaming

• Authors: Ved Prakash, Kartavya Kothari
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17077
• Pdf URL: https://arxiv.org/pdf/2409.17077
• Copy Paste: [[2409.17077]] Efficient Feature Interactions with Transformers: Improving User Spending Propensity Predictions in Gaming(https://arxiv.org/abs/2409.17077)
• Keywords: transformer
• Abstract: Dream11 is a fantasy sports platform that allows users to create their own virtual teams for real-life sports events. We host multiple sports and matches for our 200M+ user base. In this RMG (real money gaming) setting, users pay an entry amount to participate in various contest products that we provide to users. In our current work, we discuss the problem of predicting the user's propensity to spend in a gaming round, so it can be utilized for various downstream applications. e.g. Upselling users by incentivizing them marginally as per their spending propensity, or personalizing the product listing based on the user's propensity to spend. We aim to model the spending propensity of each user based on past transaction data. In this paper, we benchmark tree-based and deep-learning models that show good results on structured data, and we propose a new architecture change that is specifically designed to capture the rich interactions among the input features. We show that our proposed architecture outperforms the existing models on the task of predicting the user's propensity to spend in a gaming round. Our new transformer model surpasses the state-of-the-art FT-Transformer, improving MAE by 2.5\% and MSE by 21.8\%.

Title: Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimation

• Authors: Richard D. Paul, Alessio Quercia, Vincent Fortuin, Katharina Nöh, Hanno Scharr
• Subjects: cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.17085
• Pdf URL: https://arxiv.org/pdf/2409.17085
• Copy Paste: [[2409.17085]] Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimation(https://arxiv.org/abs/2409.17085)
• Keywords: robust, transformer
• Abstract: State-of-the-art computer vision tasks, like monocular depth estimation (MDE), rely heavily on large, modern Transformer-based architectures. However, their application in safety-critical domains demands reliable predictive performance and uncertainty quantification. While Bayesian neural networks provide a conceptually simple approach to serve those requirements, they suffer from the high dimensionality of the parameter space. Parameter-efficient fine-tuning (PEFT) methods, in particular low-rank adaptations (LoRA), have emerged as a popular strategy for adapting large-scale models to down-stream tasks by performing parameter inference on lower-dimensional subspaces. In this work, we investigate the suitability of PEFT methods for subspace Bayesian inference in large-scale Transformer-based vision models. We show that, indeed, combining BitFit, DiffFit, LoRA, and CoLoRA, a novel LoRA-inspired PEFT method, with Bayesian inference enables more robust and reliable predictive performance in MDE.

Title: Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classification

• Authors: Xinrui Zhou, Yuhao Huang, Haoran Dou, Shijing Chen, Ao Chang, Jia Liu, Weiran Long, Jian Zheng, Erjiao Xu, Jie Ren, Ruobing Huang, Jun Cheng, Wufeng Xue, Dong Ni
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17091
• Pdf URL: https://arxiv.org/pdf/2409.17091
• Copy Paste: [[2409.17091]] Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classification(https://arxiv.org/abs/2409.17091)
• Keywords: diffusion, generative
• Abstract: In the medical field, the limited availability of large-scale datasets and labor-intensive annotation processes hinder the performance of deep models. Diffusion-based generative augmentation approaches present a promising solution to this issue, having been proven effective in advancing downstream medical recognition tasks. Nevertheless, existing works lack sufficient semantic and sequential steerability for challenging video/3D sequence generation, and neglect quality control of noisy synthesized samples, resulting in unreliable synthetic databases and severely limiting the performance of downstream tasks. In this work, we present Ctrl-GenAug, a novel and general generative augmentation framework that enables highly semantic- and sequential-customized sequence synthesis and suppresses incorrectly synthesized samples, to aid medical sequence classification. Specifically, we first design a multimodal conditions-guided sequence generator for controllably synthesizing diagnosis-promotive samples. A sequential augmentation module is integrated to enhance the temporal/stereoscopic coherence of generated samples. Then, we propose a noisy synthetic data filter to suppress unreliable cases at semantic and sequential levels. Extensive experiments on 3 medical datasets, using 11 networks trained on 3 paradigms, comprehensively analyze the effectiveness and generality of Ctrl-GenAug, particularly in underrepresented high-risk populations and out-domain conditions.

Title: Accumulator-Aware Post-Training Quantization

• Authors: Ian Colbert, Fabian Grob, Giuseppe Franco, Jinjie Zhang, Rayan Saab
• Subjects: cs.LG, cs.AI, cs.DM
• Abstract URL: https://arxiv.org/abs/2409.17092
• Pdf URL: https://arxiv.org/pdf/2409.17092
• Copy Paste: [[2409.17092]] Accumulator-Aware Post-Training Quantization(https://arxiv.org/abs/2409.17092)
• Keywords: large language model
• Abstract: Several recent studies have investigated low-precision accumulation, reporting improvements in throughput, power, and area across various platforms. However, the accompanying proposals have only considered the quantization-aware training (QAT) paradigm, in which models are fine-tuned or trained from scratch with quantization in the loop. As models continue to grow in size, QAT techniques become increasingly more expensive, which has motivated the recent surge in post-training quantization (PTQ) research. To the best of our knowledge, ours marks the first formal study of accumulator-aware quantization in the PTQ setting. To bridge this gap, we introduce AXE, a practical framework of accumulator-aware extensions designed to endow overflow avoidance guarantees to existing layer-wise PTQ algorithms. We theoretically motivate AXE and demonstrate its flexibility by implementing it on top of two state-of-the-art PTQ algorithms: GPFQ and OPTQ. We further generalize AXE to support multi-stage accumulation for the first time, opening the door for full datapath optimization and scaling to large language models (LLMs). We evaluate AXE across image classification and language generation models, and observe significant improvements in the trade-off between accumulator bit width and model accuracy over baseline methods.

Title: BitQ: Tailoring Block Floating Point Precision for Improved DNN Efficiency on Resource-Constrained Devices

• Authors: Yongqi Xu, Yujian Lee, Gao Yi, Bosheng Liu, Yucong Chen, Peng Liu, Jigang Wu, Xiaoming Chen, Yinhe Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17093
• Pdf URL: https://arxiv.org/pdf/2409.17093
• Copy Paste: [[2409.17093]] BitQ: Tailoring Block Floating Point Precision for Improved DNN Efficiency on Resource-Constrained Devices(https://arxiv.org/abs/2409.17093)
• Keywords: segmentation
• Abstract: Deep neural networks (DNNs) are powerful for cognitive tasks such as image classification, object detection, and scene segmentation. One drawback however is the significant high computational complexity and memory consumption, which makes them unfeasible to run real-time on embedded platforms because of the limited hardware resources. Block floating point (BFP) quantization is one of the representative compression approaches for reducing the memory and computational burden owing to their capability to effectively capture the broad data distribution of DNN models. Unfortunately, prior works on BFP-based quantization empirically choose the block size and the precision that preserve accuracy. In this paper, we develop a BFP-based bitwidth-aware analytical modeling framework (called ``BitQ'') for the best BFP implementation of DNN inference on embedded platforms. We formulate and resolve an optimization problem to identify the optimal BFP block size and bitwidth distribution by the trade-off of both accuracy and performance loss. Experimental results show that compared with an equal bitwidth setting, the BFP DNNs with optimized bitwidth allocation provide efficient computation, preserving accuracy on famous benchmarks. The source code and data are available at this https URL.

Title: General Detection-based Text Line Recognition

• Authors: Raphael Baena, Syrine Kalleli, Mathieu Aubry
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17095
• Pdf URL: https://arxiv.org/pdf/2409.17095
• Copy Paste: [[2409.17095]] General Detection-based Text Line Recognition(https://arxiv.org/abs/2409.17095)
• Keywords: transformer
• Abstract: We introduce a general detection-based approach to text line recognition, be it printed (OCR) or handwritten (HTR), with Latin, Chinese, or ciphered characters. Detection-based approaches have until now been largely discarded for HTR because reading characters separately is often challenging, and character-level annotation is difficult and expensive. We overcome these challenges thanks to three main insights: (i) synthetic pre-training with sufficiently diverse data enables learning reasonable character localization for any script; (ii) modern transformer-based detectors can jointly detect a large number of instances, and, if trained with an adequate masking strategy, leverage consistency between the different detections; (iii) once a pre-trained detection model with approximate character localization is available, it is possible to fine-tune it with line-level annotation on real data, even with a different alphabet. Our approach, dubbed DTLR, builds on a completely different paradigm than state-of-the-art HTR methods, which rely on autoregressive decoding, predicting character values one by one, while we treat a complete line in parallel. Remarkably, we demonstrate good performance on a large range of scripts, usually tackled with specialized approaches. In particular, we improve state-of-the-art performances for Chinese script recognition on the CASIA v2 dataset, and for cipher recognition on the Borg and Copiale datasets. Our code and models are available at this https URL.

Title: Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Prompts

• Authors: Mohammad Sadil Khan, Sankalp Sinha, Talha Uddin Sheikh, Didier Stricker, Sk Aziz Ali, Muhammad Zeshan Afzal
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.17106
• Pdf URL: https://arxiv.org/pdf/2409.17106
• Copy Paste: [[2409.17106]] Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Prompts(https://arxiv.org/abs/2409.17106)
• Keywords: transformer
• Abstract: Prototyping complex computer-aided design (CAD) models in modern softwares can be very time-consuming. This is due to the lack of intelligent systems that can quickly generate simpler intermediate parts. We propose Text2CAD, the first AI framework for generating text-to-parametric CAD models using designer-friendly instructions for all skill levels. Furthermore, we introduce a data annotation pipeline for generating text prompts based on natural language instructions for the DeepCAD dataset using Mistral and LLaVA-NeXT. The dataset contains $\sim170$K models and $\sim660$K text annotations, from abstract CAD descriptions (e.g., generate two concentric cylinders) to detailed specifications (e.g., draw two circles with center $(x,y)$ and radius $r_{1}$, $r_{2}$, and extrude along the normal by $d$...). Within the Text2CAD framework, we propose an end-to-end transformer-based auto-regressive network to generate parametric CAD models from input texts. We evaluate the performance of our model through a mixture of metrics, including visual quality, parametric precision, and geometrical accuracy. Our proposed framework shows great potential in AI-aided design applications. Our source code and annotations will be publicly available.

Title: MorphoSeg: An Uncertainty-Aware Deep Learning Method for Biomedical Segmentation of Complex Cellular Morphologies

• Authors: Tianhao Zhang, Heather J. McCourty, Berardo M. Sanchez-Tafolla, Anton Nikolaev, Lyudmila S. Mihaylova
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17110
• Pdf URL: https://arxiv.org/pdf/2409.17110
• Copy Paste: [[2409.17110]] MorphoSeg: An Uncertainty-Aware Deep Learning Method for Biomedical Segmentation of Complex Cellular Morphologies(https://arxiv.org/abs/2409.17110)
• Keywords: segmentation
• Abstract: Deep learning has revolutionized medical and biological imaging, particularly in segmentation tasks. However, segmenting biological cells remains challenging due to the high variability and complexity of cell shapes. Addressing this challenge requires high-quality datasets that accurately represent the diverse morphologies found in biological cells. Existing cell segmentation datasets are often limited by their focus on regular and uniform shapes. In this paper, we introduce a novel benchmark dataset of Ntera-2 (NT2) cells, a pluripotent carcinoma cell line, exhibiting diverse morphologies across multiple stages of differentiation, capturing the intricate and heterogeneous cellular structures that complicate segmentation tasks. To address these challenges, we propose an uncertainty-aware deep learning framework for complex cellular morphology segmentation (MorphoSeg) by incorporating sampling of virtual outliers from low-likelihood regions during training. Our comprehensive experimental evaluations against state-of-the-art baselines demonstrate that MorphoSeg significantly enhances segmentation accuracy, achieving up to a 7.74% increase in the Dice Similarity Coefficient (DSC) and a 28.36% reduction in the Hausdorff Distance. These findings highlight the effectiveness of our dataset and methodology in advancing cell segmentation capabilities, especially for complex and variable cell morphologies. The dataset and source code is publicly available at this https URL.

Title: Characterizing stable regions in the residual stream of LLMs

• Authors: Jett Janiak, Jacek Karwowski, Chatrik Singh Mangat, Giorgi Giglemiani, Nora Petrova, Stefan Heimersheim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17113
• Pdf URL: https://arxiv.org/pdf/2409.17113
• Copy Paste: [[2409.17113]] Characterizing stable regions in the residual stream of LLMs(https://arxiv.org/abs/2409.17113)
• Keywords: transformer
• Abstract: We identify "stable regions" in the residual stream of Transformers, where the model's output remains insensitive to small activation changes, but exhibits high sensitivity at region boundaries. These regions emerge during training and become more defined as training progresses or model size increases. The regions appear to be much larger than previously studied polytopes. Our analysis suggests that these stable regions align with semantic distinctions, where similar prompts cluster within regions, and activations from the same region lead to similar next token predictions.

Title: Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale

• Authors: Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, Pengfei Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17115
• Pdf URL: https://arxiv.org/pdf/2409.17115
• Copy Paste: [[2409.17115]] Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale(https://arxiv.org/abs/2409.17115)
• Keywords: large language model
• Abstract: Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date. However, these rules lack the flexibility to address the unique characteristics of individual example effectively. Meanwhile, applying tailored rules to every example is impractical for human experts. In this paper, we demonstrate that even small language models, with as few as 0.3B parameters, can exhibit substantial data refining capabilities comparable to those of human experts. We introduce Programming Every Example (ProX), a novel framework that treats data refinement as a programming task, enabling models to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale. Experimental results show that models pre-trained on ProX-curated data outperform either original data or data filtered by other selection methods by more than 2% across various downstream benchmarks. Its effectiveness spans various model sizes and pre-training corpora, including C4, RedPajama-V2, and FineWeb. Furthermore, ProX exhibits significant potential in domain-specific continual pre-training: without domain specific design, models trained on OpenWebMath refined by ProX outperform human-crafted rule-based methods, improving average accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B trained on 200B tokens. Further analysis highlights that ProX significantly saves training FLOPs, offering a promising path for efficient LLM pre-training.We are open-sourcing ProX with >100B corpus, models, and sharing all training and implementation details for reproducible research and future innovation. Code: this https URL

Title: Deep Learning and Machine Learning, Advancing Big Data Analytics and Management: Handy Appetizer

• Authors: Benji Peng, Xuanhe Pan, Yizhu Wen, Ziqian Bi, Keyu Chen, Ming Li, Ming Liu, Qian Niu, Junyu Liu, Jinlang Wang, Sen Zhang, Jiawei Xu, Pohsun Feng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17120
• Pdf URL: https://arxiv.org/pdf/2409.17120
• Copy Paste: [[2409.17120]] Deep Learning and Machine Learning, Advancing Big Data Analytics and Management: Handy Appetizer(https://arxiv.org/abs/2409.17120)
• Keywords: transformer
• Abstract: This book explores the role of Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL) in driving the progress of big data analytics and management. The book focuses on simplifying the complex mathematical concepts behind deep learning, offering intuitive visualizations and practical case studies to help readers understand how neural networks and technologies like Convolutional Neural Networks (CNNs) work. It introduces several classic models and technologies such as Transformers, GPT, ResNet, BERT, and YOLO, highlighting their applications in fields like natural language processing, image recognition, and autonomous driving. The book also emphasizes the importance of pre-trained models and how they can enhance model performance and accuracy, with instructions on how to apply these models in various real-world scenarios. Additionally, it provides an overview of key big data management technologies like SQL and NoSQL databases, as well as distributed computing frameworks such as Apache Hadoop and Spark, explaining their importance in managing and processing vast amounts of data. Ultimately, the book underscores the value of mastering deep learning and big data management skills as critical tools for the future workforce, making it an essential resource for both beginners and experienced professionals.

Title: Assessing the Level of Toxicity Against Distinct Groups in Bangla Social Media Comments: A Comprehensive Investigation

• Authors: Mukaffi Bin Moin, Pronay Debnath, Usafa Akther Rifa, Rijeet Bin Anis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17130
• Pdf URL: https://arxiv.org/pdf/2409.17130
• Copy Paste: [[2409.17130]] Assessing the Level of Toxicity Against Distinct Groups in Bangla Social Media Comments: A Comprehensive Investigation(https://arxiv.org/abs/2409.17130)
• Keywords: transformer
• Abstract: Social media platforms have a vital role in the modern world, serving as conduits for communication, the exchange of ideas, and the establishment of networks. However, the misuse of these platforms through toxic comments, which can range from offensive remarks to hate speech, is a concerning issue. This study focuses on identifying toxic comments in the Bengali language targeting three specific groups: transgender people, indigenous people, and migrant people, from multiple social media sources. The study delves into the intricate process of identifying and categorizing toxic language while considering the varying degrees of toxicity: high, medium, and low. The methodology involves creating a dataset, manual annotation, and employing pre-trained transformer models like Bangla-BERT, bangla-bert-base, distil-BERT, and Bert-base-multilingual-cased for classification. Diverse assessment metrics such as accuracy, recall, precision, and F1-score are employed to evaluate the model's effectiveness. The experimental findings reveal that Bangla-BERT surpasses alternative models, achieving an F1-score of 0.8903. This research exposes the complexity of toxicity in Bangla social media dialogues, revealing its differing impacts on diverse demographic groups.

Title: Streaming Neural Images

• Authors: Marcos V. Conde, Andy Bigos, Radu Timofte
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.17134
• Pdf URL: https://arxiv.org/pdf/2409.17134
• Copy Paste: [[2409.17134]] Streaming Neural Images(https://arxiv.org/abs/2409.17134)
• Keywords: robust
• Abstract: Implicit Neural Representations (INRs) are a novel paradigm for signal representation that have attracted considerable interest for image compression. INRs offer unprecedented advantages in signal resolution and memory efficiency, enabling new possibilities for compression techniques. However, the existing limitations of INRs for image compression have not been sufficiently addressed in the literature. In this work, we explore the critical yet overlooked limiting factors of INRs, such as computational cost, unstable performance, and robustness. Through extensive experiments and empirical analysis, we provide a deeper and more nuanced understanding of implicit neural image compression methods such as Fourier Feature Networks and Siren. Our work also offers valuable insights for future research in this area.

Title: PACE: marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization

• Authors: Yao Ni, Shan Zhang, Piotr Koniusz
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17137
• Pdf URL: https://arxiv.org/pdf/2409.17137
• Copy Paste: [[2409.17137]] PACE: marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization(https://arxiv.org/abs/2409.17137)
• Keywords: transformer
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) effectively adapts pre-trained vision transformers to downstream tasks. However, the optimization for tasks performance often comes at the cost of generalizability in fine-tuned models. To address this issue, we theoretically connect smaller weight gradient norms during training and larger datasets to the improved model generalization. Motivated by this connection, we propose reducing gradient norms for enhanced generalization and aligning fine-tuned model with the pre-trained counterpart to retain knowledge from large-scale pre-training data. Yet, naive alignment does not guarantee gradient reduction and can potentially cause gradient explosion, complicating efforts to manage gradients. To address such issues, we propose PACE, marrying generalization of PArameter-efficient fine-tuning with Consistency rEgularization. We perturb features learned from the adapter with the multiplicative noise and ensure the fine-tuned model remains consistent for same sample under different perturbations. Theoretical analysis shows that PACE not only implicitly regularizes gradients for enhanced generalization, but also implicitly aligns the fine-tuned and pre-trained models to retain knowledge. Experimental evidence supports our theories. PACE outperforms existing PEFT methods in four visual adaptation tasks: VTAB-1k, FGVC, few-shot learning and domain adaptation. Code will be available at this https URL

Title: FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression

• Authors: Fazal Mittu, Yihuan Bu, Akshat Gupta, Ashok Devireddy, Alp Eren Ozdarendeli, Anant Singh, Gopala Anumanchipalli
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17141
• Pdf URL: https://arxiv.org/pdf/2409.17141
• Copy Paste: [[2409.17141]] FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression(https://arxiv.org/abs/2409.17141)
• Keywords: transformer, large language model
• Abstract: While the language modeling objective has been shown to be deeply connected with compression, it is surprising that modern LLMs are not employed in practical text compression systems. In this paper, we provide an in-depth analysis of neural network and transformer-based compression techniques to answer this question. We compare traditional text compression systems with neural network and LLM-based text compression methods. Although LLM-based systems significantly outperform conventional compression methods, they are highly impractical. Specifically, LLMZip, a recent text compression system using Llama3-8B requires 9.5 days to compress just 10 MB of text, although with huge improvements in compression ratios. To overcome this, we present FineZip - a novel LLM-based text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely. FineZip can compress the above corpus in approximately 4 hours compared to 9.5 days, a 54 times improvement over LLMZip and comparable performance. FineZip outperforms traditional algorithmic compression methods with a large margin, improving compression ratios by approximately 50\%. With this work, we take the first step towards making lossless text compression with LLMs a reality. While FineZip presents a significant step in that direction, LLMs are still not a viable solution for large-scale text compression. We hope our work paves the way for future research and innovation to solve this problem.

Title: Attention Prompting on Image for Large Vision-Language Models

• Authors: Runpeng Yu, Weihao Yu, Xinchao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17143
• Pdf URL: https://arxiv.org/pdf/2409.17143
• Copy Paste: [[2409.17143]] Attention Prompting on Image for Large Vision-Language Models(https://arxiv.org/abs/2409.17143)
• Keywords: large language model
• Abstract: Compared with Large Language Models (LLMs), Large Vision-Language Models (LVLMs) can also accept images as input, thus showcasing more interesting emergent capabilities and demonstrating impressive performance on various vision-language tasks. Motivated by text prompting in LLMs, visual prompting has been explored to enhance LVLMs' capabilities of perceiving visual information. However, previous visual prompting techniques solely process visual inputs without considering text queries, limiting the models' ability to follow text instructions to complete tasks. To fill this gap, in this work, we propose a new prompting technique named Attention Prompting on Image, which just simply overlays a text-query-guided attention heatmap on the original input image and effectively enhances LVLM on various tasks. Specifically, we generate an attention heatmap for the input image dependent on the text query with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel values of the original image to obtain the actual input image for the LVLM. Extensive experiments on various vison-language benchmarks verify the effectiveness of our technique. For example, Attention Prompting on Image improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks, respectively.

Title: Differential Privacy Regularization: Protecting Training Data Through Loss Function Regularization

• Authors: Francisco Aguilera-Martínez, Fernando Berzal
• Subjects: cs.LG, cs.AI, cs.CR, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.17144
• Pdf URL: https://arxiv.org/pdf/2409.17144
• Copy Paste: [[2409.17144]] Differential Privacy Regularization: Protecting Training Data Through Loss Function Regularization(https://arxiv.org/abs/2409.17144)
• Keywords: privacy, protect
• Abstract: Training machine learning models based on neural networks requires large datasets, which may contain sensitive information. The models, however, should not expose private information from these datasets. Differentially private SGD [DP-SGD] requires the modification of the standard stochastic gradient descent [SGD] algorithm for training new models. In this short paper, a novel regularization strategy is proposed to achieve the same goal in a more efficient manner.

Title: DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion

• Authors: Yukun Huang, Jianan Wang, Ailing Zeng, Zheng-Jun Zha, Lei Zhang, Xihui Liu
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17145
• Pdf URL: https://arxiv.org/pdf/2409.17145
• Copy Paste: [[2409.17145]] DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion(https://arxiv.org/abs/2409.17145)
• Keywords: diffusion
• Abstract: Leveraging pretrained 2D diffusion models and score distillation sampling (SDS), recent methods have shown promising results for text-to-3D avatar generation. However, generating high-quality 3D avatars capable of expressive animation remains challenging. In this work, we present DreamWaltz-G, a novel learning framework for animatable 3D avatar generation from text. The core of this framework lies in Skeleton-guided Score Distillation and Hybrid 3D Gaussian Avatar representation. Specifically, the proposed skeleton-guided score distillation integrates skeleton controls from 3D human templates into 2D diffusion models, enhancing the consistency of SDS supervision in terms of view and human pose. This facilitates the generation of high-quality avatars, mitigating issues such as multiple faces, extra limbs, and blurring. The proposed hybrid 3D Gaussian avatar representation builds on the efficient 3D Gaussians, combining neural implicit fields and parameterized 3D meshes to enable real-time rendering, stable SDS optimization, and expressive animation. Extensive experiments demonstrate that DreamWaltz-G is highly effective in generating and animating 3D avatars, outperforming existing methods in both visual quality and animation expressiveness. Our framework further supports diverse applications, including human video reenactment and multi-subject scene composition.