2024-11-20

Title: DiHuR: Diffusion-Guided Generalizable Human Reconstruction

• Authors: Jinnan Chen, Chen Li, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11903
• Pdf URL: https://arxiv.org/pdf/2411.11903
• Copy Paste: [[2411.11903]] DiHuR: Diffusion-Guided Generalizable Human Reconstruction(https://arxiv.org/abs/2411.11903)
• Keywords: diffusion
• Abstract: We introduce DiHuR, a novel Diffusion-guided model for generalizable Human 3D Reconstruction and view synthesis from sparse, minimally overlapping images. While existing generalizable human radiance fields excel at novel view synthesis, they often struggle with comprehensive 3D reconstruction. Similarly, directly optimizing implicit Signed Distance Function (SDF) fields from sparse-view images typically yields poor results due to limited overlap. To enhance 3D reconstruction quality, we propose using learnable tokens associated with SMPL vertices to aggregate sparse view features and then to guide SDF prediction. These tokens learn a generalizable prior across different identities in training datasets, leveraging the consistent projection of SMPL vertices onto similar semantic areas across various human identities. This consistency enables effective knowledge transfer to unseen identities during inference. Recognizing SMPL's limitations in capturing clothing details, we incorporate a diffusion model as an additional prior to fill in missing information, particularly for complex clothing geometries. Our method integrates two key priors in a coherent manner: the prior from generalizable feed-forward models and the 2D diffusion prior, and it requires only multi-view image training, without 3D supervision. DiHuR demonstrates superior performance in both within-dataset and cross-dataset generalization settings, as validated on THuman, ZJU-MoCap, and HuMMan datasets compared to existing methods.

Title: GeoGround: A Unified Large Vision-Language Model. for Remote Sensing Visual Grounding

• Authors: Yue Zhou, Mengcheng Lan, Xiang Li, Yiping Ke, Xue Jiang, Litong Feng, Wayne Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11904
• Pdf URL: https://arxiv.org/pdf/2411.11904
• Copy Paste: [[2411.11904]] GeoGround: A Unified Large Vision-Language Model. for Remote Sensing Visual Grounding(https://arxiv.org/abs/2411.11904)
• Keywords: segmentation
• Abstract: Remote sensing (RS) visual grounding aims to use natural language expression to locate specific objects (in the form of the bounding box or segmentation mask) in RS images, enhancing human interaction with intelligent RS interpretation systems. Early research in this area was primarily based on horizontal bounding boxes (HBBs), but as more diverse RS datasets have become available, tasks involving oriented bounding boxes (OBBs) and segmentation masks have emerged. In practical applications, different targets require different grounding types: HBB can localize an object's position, OBB provides its orientation, and mask depicts its shape. However, existing specialized methods are typically tailored to a single type of RS visual grounding task and are hard to generalize across tasks. In contrast, large vision-language models (VLMs) exhibit powerful multi-task learning capabilities but struggle to handle dense prediction tasks like segmentation. This paper proposes GeoGround, a novel framework that unifies support for HBB, OBB, and mask RS visual grounding tasks, allowing flexible output selection. Rather than customizing the architecture of VLM, our work aims to elegantly support pixel-level visual grounding output through the Text-Mask technique. We define prompt-assisted and geometry-guided learning to enhance consistency across different signals. To support model training, we present refGeo, a large-scale RS visual instruction-following dataset containing 161k image-text pairs. Experimental results show that GeoGround demonstrates strong performance across four RS visual grounding tasks, matching or surpassing the performance of specialized methods on multiple benchmarks. Code available at this https URL

Title: $\text{S}^{3}$Mamba: Arbitrary-Scale Super-Resolution via Scaleable State Space Model

• Authors: Peizhe Xia, Long Peng, Xin Di, Renjing Pei, Yang Wang, Yang Cao, Zheng-Jun Zha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11906
• Pdf URL: https://arxiv.org/pdf/2411.11906
• Copy Paste: [[2411.11906]] $\text{S}^{3}$Mamba: Arbitrary-Scale Super-Resolution via Scaleable State Space Model(https://arxiv.org/abs/2411.11906)
• Keywords: transformer
• Abstract: Arbitrary scale super-resolution (ASSR) aims to super-resolve low-resolution images to high-resolution images at any scale using a single model, addressing the limitations of traditional super-resolution methods that are restricted to fixed-scale factors (e.g., $\times2$, $\times4$). The advent of Implicit Neural Representations (INR) has brought forth a plethora of novel methodologies for ASSR, which facilitate the reconstruction of original continuous signals by modeling a continuous representation space for coordinates and pixel values, thereby enabling arbitrary-scale super-resolution. Consequently, the primary objective of ASSR is to construct a continuous representation space derived from low-resolution inputs. However, existing methods, primarily based on CNNs and Transformers, face significant challenges such as high computational complexity and inadequate modeling of long-range dependencies, which hinder their effectiveness in real-world applications. To overcome these limitations, we propose a novel arbitrary-scale super-resolution method, called $\text{S}^{3}$Mamba, to construct a scalable continuous representation space. Specifically, we propose a Scalable State Space Model (SSSM) to modulate the state transition matrix and the sampling matrix of step size during the discretization process, achieving scalable and continuous representation modeling with linear computational complexity. Additionally, we propose a novel scale-aware self-attention mechanism to further enhance the network's ability to perceive global important features at different scales, thereby building the $\text{S}^{3}$Mamba to achieve superior arbitrary-scale super-resolution. Extensive experiments on both synthetic and real-world benchmarks demonstrate that our method achieves state-of-the-art performance and superior generalization capabilities at arbitrary super-resolution scales.

Title: LoRA Unlearns More and Retains More (Student Abstract)

• Authors: Atharv Mittal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.11907
• Pdf URL: https://arxiv.org/pdf/2411.11907
• Copy Paste: [[2411.11907]] LoRA Unlearns More and Retains More (Student Abstract)(https://arxiv.org/abs/2411.11907)
• Keywords: privacy
• Abstract: Due to increasing privacy regulations and regulatory compliance, Machine Unlearning (MU) has become essential. The goal of unlearning is to remove information related to a specific class from a model. Traditional approaches achieve exact unlearning by retraining the model on the remaining dataset, but incur high computational costs. This has driven the development of more efficient unlearning techniques, including model sparsification techniques, which boost computational efficiency, but degrade the model's performance on the remaining classes. To mitigate these issues, we propose a novel method, PruneLoRA which introduces a new MU paradigm, termed prune first, then adapt, then unlearn. LoRA (Hu et al. 2022) reduces the need for large-scale parameter updates by applying low-rank updates to the model. We leverage LoRA to selectively modify a subset of the pruned model's parameters, thereby reducing the computational cost, memory requirements and improving the model's ability to retain performance on the remaining classes. Experimental Results across various metrics showcase that our method outperforms other approximate MU methods and bridges the gap between exact and approximate unlearning. Our code is available at this https URL.

Title: SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization

• Authors: Hongrui Jia, Chaoya Jiang, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11909
• Pdf URL: https://arxiv.org/pdf/2411.11909
• Copy Paste: [[2411.11909]] SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization(https://arxiv.org/abs/2411.11909)
• Keywords: large language model
• Abstract: As language models continue to scale, Large Language Models (LLMs) have exhibited emerging capabilities in In-Context Learning (ICL), enabling them to solve language tasks by prefixing a few in-context demonstrations (ICDs) as context. Inspired by these advancements, researchers have extended these techniques to develop Large Multimodal Models (LMMs) with ICL capabilities. However, existing LMMs face a critical issue: they often fail to effectively leverage the visual context in multimodal demonstrations and instead simply follow textual patterns. This indicates that LMMs do not achieve effective alignment between multimodal demonstrations and model outputs. To address this problem, we propose Symbol Demonstration Direct Preference Optimization (SymDPO). Specifically, SymDPO aims to break the traditional paradigm of constructing multimodal demonstrations by using random symbols to replace text answers within instances. This forces the model to carefully understand the demonstration images and establish a relationship between the images and the symbols to answer questions correctly. We validate the effectiveness of this method on multiple benchmarks, demonstrating that with SymDPO, LMMs can more effectively understand the multimodal context within examples and utilize this knowledge to answer questions better.

Title: AIGS: Generating Science from AI-Powered Automated Falsification

• Authors: Zijun Liu, Kaiming Liu, Yiqi Zhu, Xuanyu Lei, Zonghan Yang, Zhenhe Zhang, Peng Li, Yang Liu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.11910
• Pdf URL: https://arxiv.org/pdf/2411.11910
• Copy Paste: [[2411.11910]] AIGS: Generating Science from AI-Powered Automated Falsification(https://arxiv.org/abs/2411.11910)
• Keywords: large language model
• Abstract: Rapid development of artificial intelligence has drastically accelerated the development of scientific discovery. Trained with large-scale observation data, deep neural networks extract the underlying patterns in an end-to-end manner and assist human researchers with highly-precised predictions in unseen scenarios. The recent rise of Large Language Models (LLMs) and the empowered autonomous agents enable scientists to gain help through interaction in different stages of their research, including but not limited to literature review, research ideation, idea implementation, and academic writing. However, AI researchers instantiated by foundation model empowered agents with full-process autonomy are still in their infancy. In this paper, we study $\textbf{AI-Generated Science}$ (AIGS), where agents independently and autonomously complete the entire research process and discover scientific laws. By revisiting the definition of scientific research, we argue that $\textit{falsification}$ is the essence of both human research process and the design of an AIGS system. Through the lens of falsification, prior systems attempting towards AI-Generated Science either lack the part in their design, or rely heavily on existing verification engines that narrow the use in specialized domains. In this work, we propose Baby-AIGS as a baby-step demonstration of a full-process AIGS system, which is a multi-agent system with agents in roles representing key research process. By introducing FalsificationAgent, which identify and then verify possible scientific discoveries, we empower the system with explicit falsification. Experiments on three tasks preliminarily show that Baby-AIGS could produce meaningful scientific discoveries, though not on par with experienced human researchers. Finally, we discuss on the limitations of current Baby-AIGS, actionable insights, and related ethical issues in detail.

Title: F$^3$OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics

• Authors: Pramit Saha, Felix Wagner, Divyanshu Mishra, Can Peng, Anshul Thakur, David Clifton, Konstantinos Kamnitsas, J. Alison Noble
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.11912
• Pdf URL: https://arxiv.org/pdf/2411.11912
• Copy Paste: [[2411.11912]] F$^3$OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics(https://arxiv.org/abs/2411.11912)
• Keywords: federate, data-free
• Abstract: Effective training of large Vision-Language Models (VLMs) on resource-constrained client devices in Federated Learning (FL) requires the usage of parameter-efficient fine-tuning (PEFT) strategies. To this end, we demonstrate the impact of two factors \textit{viz.}, client-specific layer importance score that selects the most important VLM layers for fine-tuning and inter-client layer diversity score that encourages diverse layer selection across clients for optimal VLM layer selection. We first theoretically motivate and leverage the principal eigenvalue magnitude of layerwise Neural Tangent Kernels and show its effectiveness as client-specific layer importance score. Next, we propose a novel layer updating strategy dubbed F$^3$OCUS that jointly optimizes the layer importance and diversity factors by employing a data-free, multi-objective, meta-heuristic optimization on the server. We explore 5 different meta-heuristic algorithms and compare their effectiveness for selecting model layers and adapter layers towards PEFT-FL. Furthermore, we release a new MedVQA-FL dataset involving overall 707,962 VQA triplets and 9 modality-specific clients and utilize it to train and evaluate our method. Overall, we conduct more than 10,000 client-level experiments on 6 Vision-Language FL task settings involving 58 medical image datasets and 4 different VLM architectures of varying sizes to demonstrate the effectiveness of the proposed method.

Title: FCC: Fully Connected Correlation for Few-Shot Segmentation

• Authors: Seonghyeon Moon, Haein Kong, Muhammad Haris Khan, Yuewei Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11917
• Pdf URL: https://arxiv.org/pdf/2411.11917
• Copy Paste: [[2411.11917]] FCC: Fully Connected Correlation for Few-Shot Segmentation(https://arxiv.org/abs/2411.11917)
• Keywords: transformer, segmentation
• Abstract: Few-shot segmentation (FSS) aims to segment the target object in a query image using only a small set of support images and masks. Therefore, having strong prior information for the target object using the support set is essential for guiding the initial training of FSS, which leads to the success of few-shot segmentation in challenging cases, such as when the target object shows considerable variation in appearance, texture, or scale across the support and query images. Previous methods have tried to obtain prior information by creating correlation maps from pixel-level correlation on final-layer or same-layer features. However, we found these approaches can offer limited and partial information when advanced models like Vision Transformers are used as the backbone. Vision Transformer encoders have a multi-layer structure with identical shapes in their intermediate layers. Leveraging the feature comparison from all layers in the encoder can enhance the performance of few-shot segmentation. We introduce FCC (Fully Connected Correlation) to integrate pixel-level correlations between support and query features, capturing associations that reveal target-specific patterns and correspondences in both same-layers and cross-layers. FCC captures previously inaccessible target information, effectively addressing the limitations of support mask. Our approach consistently demonstrates state-of-the-art performance on PASCAL, COCO, and domain shift tests. We conducted an ablation study and cross-layer correlation analysis to validate FCC's core methodology. These findings reveal the effectiveness of FCC in enhancing prior information and overall model performance.

Title: Artificial Intelligence Mangrove Monitoring System Based on Deep Learning and Sentinel-2 Satellite Data in the UAE (2017-2024)

• Authors: Linlin Tan, Haishan Wu
• Subjects: cs.LG, stat.CO
• Abstract URL: https://arxiv.org/abs/2411.11918
• Pdf URL: https://arxiv.org/pdf/2411.11918
• Copy Paste: [[2411.11918]] Artificial Intelligence Mangrove Monitoring System Based on Deep Learning and Sentinel-2 Satellite Data in the UAE (2017-2024)(https://arxiv.org/abs/2411.11918)
• Keywords: protect
• Abstract: Mangroves play a crucial role in maintaining coastal ecosystem health and protecting biodiversity. Therefore, continuous mapping of mangroves is essential for understanding their dynamics. Earth observation imagery typically provides a cost-effective way to monitor mangrove dynamics. However, there is a lack of regional studies on mangrove areas in the UAE. This study utilizes the UNet++ deep learning model combined with Sentinel-2 multispectral data and manually annotated labels to monitor the spatiotemporal dynamics of densely distributed mangroves (coverage greater than 70%) in the UAE from 2017 to 2024, achieving an mIoU of 87.8% on the validation set. Results show that the total mangrove area in the UAE in 2024 was approximately 9,142.21 hectares, an increase of 2,061.33 hectares compared to 2017, with carbon sequestration increasing by approximately 194,383.42 tons. Abu Dhabi has the largest mangrove area and plays a dominant role in the UAE's mangrove growth, increasing by 1,855.6 hectares between 2017-2024, while other emirates have also contributed to mangrove expansion through stable and sustainable growth in mangrove areas. This comprehensive growth pattern reflects the collective efforts of all emirates in mangrove restoration.

Title: SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory

• Authors: Cheng-Yen Yang, Hsiang-Wei Huang, Wenhao Chai, Zhongyu Jiang, Jenq-Neng Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11922
• Pdf URL: https://arxiv.org/pdf/2411.11922
• Copy Paste: [[2411.11922]] SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory(https://arxiv.org/abs/2411.11922)
• Keywords: robust, segmentation
• Abstract: The Segment Anything Model 2 (SAM 2) has demonstrated strong performance in object segmentation tasks but faces challenges in visual object tracking, particularly when managing crowded scenes with fast-moving or self-occluding objects. Furthermore, the fixed-window memory approach in the original model does not consider the quality of memories selected to condition the image features for the next frame, leading to error propagation in videos. This paper introduces SAMURAI, an enhanced adaptation of SAM 2 specifically designed for visual object tracking. By incorporating temporal motion cues with the proposed motion-aware memory selection mechanism, SAMURAI effectively predicts object motion and refines mask selection, achieving robust, accurate tracking without the need for retraining or fine-tuning. SAMURAI operates in real-time and demonstrates strong zero-shot performance across diverse benchmark datasets, showcasing its ability to generalize without fine-tuning. In evaluations, SAMURAI achieves significant improvements in success rate and precision over existing trackers, with a 7.1% AUC gain on LaSOT$_{\text{ext}}$ and a 3.5% AO gain on GOT-10k. Moreover, it achieves competitive results compared to fully supervised methods on LaSOT, underscoring its robustness in complex tracking scenarios and its potential for real-world applications in dynamic environments. Code and results are available at this https URL.

Title: Dataset Distillers Are Good Label Denoisers In the Wild

• Authors: Lechao Cheng, Kaifeng Chen, Jiyang Li, Shengeng Tang, Shufei Zhang, Meng Wang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11924
• Pdf URL: https://arxiv.org/pdf/2411.11924
• Copy Paste: [[2411.11924]] Dataset Distillers Are Good Label Denoisers In the Wild(https://arxiv.org/abs/2411.11924)
• Keywords: privacy, protect, robust
• Abstract: Learning from noisy data has become essential for adapting deep learning models to real-world applications. Traditional methods often involve first evaluating the noise and then applying strategies such as discarding noisy samples, re-weighting, or re-labeling. However, these methods can fall into a vicious cycle when the initial noise evaluation is inaccurate, leading to suboptimal performance. To address this, we propose a novel approach that leverages dataset distillation for noise removal. This method avoids the feedback loop common in existing techniques and enhances training efficiency, while also providing strong privacy protection through offline processing. We rigorously evaluate three representative dataset distillation methods (DATM, DANCE, and RCIG) under various noise conditions, including symmetric noise, asymmetric noise, and real-world natural noise. Our empirical findings reveal that dataset distillation effectively serves as a denoising tool in random noise scenarios but may struggle with structured asymmetric noise patterns, which can be absorbed into the distilled samples. Additionally, clean but challenging samples, such as those from tail classes in imbalanced datasets, may undergo lossy compression during distillation. Despite these challenges, our results highlight that dataset distillation holds significant promise for robust model training, especially in high-privacy environments where noise is prevalent.

Title: Continuous Speculative Decoding for Autoregressive Image Generation

• Authors: Zili Wang, Robert Zhang, Kun Ding, Qi Yang, Fei Li, Shiming Xiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11925
• Pdf URL: https://arxiv.org/pdf/2411.11925
• Copy Paste: [[2411.11925]] Continuous Speculative Decoding for Autoregressive Image Generation(https://arxiv.org/abs/2411.11925)
• Keywords: diffusion, large language model
• Abstract: Continuous-valued Autoregressive (AR) image generation models have demonstrated notable superiority over their discrete-token counterparts, showcasing considerable reconstruction quality and higher generation fidelity. However, the computational demands of the autoregressive framework result in significant inference overhead. While speculative decoding has proven effective in accelerating Large Language Models (LLMs), their adaptation to continuous-valued visual autoregressive models remains unexplored. This work generalizes the speculative decoding algorithm from discrete tokens to continuous space. By analyzing the intrinsic properties of output distribution, we establish a tailored acceptance criterion for the diffusion distributions prevalent in such models. To overcome the inconsistency that occurred in speculative decoding output distributions, we introduce denoising trajectory alignment and token pre-filling methods. Additionally, we identify the hard-to-sample distribution in the rejection phase. To mitigate this issue, we propose a meticulous acceptance-rejection sampling method with a proper upper bound, thereby circumventing complex integration. Experimental results show that our continuous speculative decoding achieves a remarkable $2.33\times$ speed-up on off-the-shelf models while maintaining the output distribution. Codes will be available at this https URL

Title: KAN-Mamba FusionNet: Redefining Medical Image Segmentation with Non-Linear Modeling

• Authors: Akansh Agrawal, Akshan Agrawal, Shashwat Gupta, Priyanka Bagade
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11926
• Pdf URL: https://arxiv.org/pdf/2411.11926
• Copy Paste: [[2411.11926]] KAN-Mamba FusionNet: Redefining Medical Image Segmentation with Non-Linear Modeling(https://arxiv.org/abs/2411.11926)
• Keywords: interpretability, segmentation
• Abstract: Medical image segmentation is crucial in robotic surgeries, disease diagnosis, and treatment plans. This research presents an innovative methodology that combines Kolmogorov-Arnold Networks (KAN) with an adapted Mamba layer for medical image segmentation. The proposed KAN-Mamba FusionNet framework improves image segmentation by integrating attention-driven mechanisms with convolutional parallel training and autoregressive deployment, while preserving interpretability, in contrast to the state-of-the-art techniques that depend exclusively on Mamba for ailment localization and accurate diagnosis. We evaluated our proposed KAN-Mamba FusionNet model on three distinct medical image segmentation datasets, BUSI, Kvasir-Seg and GlaS. The results indicated that the KAN-Mamba FusionNet consistently yields better IoU and F1 scores in comparison to the state-of-the-art methods. Further, we offer insights into the model's behavior via ablation studies, examining the effects of various components and assessing their contributions to the overall performance of the proposed model. The findings illustrate the strength and effectiveness of this methodology for dependable medical image segmentation, providing a unique approach to address intricate visual data issues in healthcare.

Title: FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training

• Authors: Anjia Cao, Xing Wei, Zhiheng Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11927
• Pdf URL: https://arxiv.org/pdf/2411.11927
• Copy Paste: [[2411.11927]] FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training(https://arxiv.org/abs/2411.11927)
• Keywords: large language model
• Abstract: Language-image pre-training faces significant challenges due to limited data in specific formats and the constrained capacities of text encoders. While prevailing methods attempt to address these issues through data augmentation and architecture modifications, they continue to struggle with processing long-form text inputs, and the inherent limitations of traditional CLIP text encoders lead to suboptimal downstream generalization. In this paper, we propose FLAME (Frozen Large lAnguage Models Enable data-efficient language-image pre-training) that leverages frozen large language models as text encoders, naturally processing long text inputs and demonstrating impressive multilingual generalization. FLAME comprises two key components: 1) a multifaceted prompt distillation technique for extracting diverse semantic representations from long captions, which better aligns with the multifaceted nature of images, and 2) a facet-decoupled attention mechanism, complemented by an offline embedding strategy, to ensure efficient computation. Extensive empirical evaluations demonstrate FLAME's superior performance. When trained on CC3M, FLAME surpasses the previous state-of-the-art by 4.9\% in ImageNet top-1 accuracy. On YFCC15M, FLAME surpasses the WIT-400M-trained CLIP by 44.4\% in average image-to-text recall@1 across 36 languages, and by 34.6\% in text-to-image recall@1 for long-context retrieval on Urban-1k. Code is available at \url{this https URL}.

Title: ChatHTTPFuzz: Large Language Model-Assisted IoT HTTP Fuzzing

• Authors: Zhe Yang, Hao Peng, Yanling Jiang, Xingwei Li, Haohua Du, Shuhai Wang, Jianwei Liu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.11929
• Pdf URL: https://arxiv.org/pdf/2411.11929
• Copy Paste: [[2411.11929]] ChatHTTPFuzz: Large Language Model-Assisted IoT HTTP Fuzzing(https://arxiv.org/abs/2411.11929)
• Keywords: security, large language model
• Abstract: Internet of Things (IoT) devices offer convenience through web interfaces, web VPNs, and other web-based services, all relying on the HTTP protocol. However, these externally exposed HTTP services resent significant security risks. Although fuzzing has shown some effectiveness in identifying vulnerabilities in IoT HTTP services, most state-of-the-art tools still rely on random mutation trategies, leading to difficulties in accurately understanding the HTTP protocol's structure and generating many invalid test cases. Furthermore, These fuzzers rely on a limited set of initial seeds for testing. While this approach initiates testing, the limited number and diversity of seeds hinder comprehensive coverage of complex scenarios in IoT HTTP services. In this paper, we investigate and find that large language models (LLMs) excel in parsing HTTP protocol data and analyzing code logic. Based on these findings, we propose a novel LLM-guided IoT HTTP fuzzing method, ChatHTTPFuzz, which automatically parses protocol fields and analyzes service code logic to generate protocol-compliant test cases. Specifically, we use LLMs to label fields in HTTP protocol data, creating seed templates. Second, The LLM analyzes service code to guide the generation of additional packets aligned with the code logic, enriching the seed templates and their field values. Finally, we design an enhanced Thompson sampling algorithm based on the exploration balance factor and mutation potential factor to schedule seed templates. We evaluate ChatHTTPFuzz on 14 different real-world IoT devices. It finds more vulnerabilities than SNIPUZZ, BOOFUZZ, and MUTINY. ChatHTTPFuzz has discovered 103 vulnerabilities, of which 68 are unique, and 23 have been assigned CVEs.

Title: AtomThink: A Slow Thinking Framework for Multimodal Mathematical Reasoning

• Authors: Kun Xiang, Zhili Liu, Zihao Jiang, Yunshuang Nie, Runhui Huang, Haoxiang Fan, Hanhui Li, Weiran Huang, Yihan Zeng, Jianhua Han, Lanqing Hong, Hang Xu, Xiaodan Liang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11930
• Pdf URL: https://arxiv.org/pdf/2411.11930
• Copy Paste: [[2411.11930]] AtomThink: A Slow Thinking Framework for Multimodal Mathematical Reasoning(https://arxiv.org/abs/2411.11930)
• Keywords: large language model
• Abstract: In this paper, we address the challenging task of multimodal mathematical reasoning by incorporating the ability of ``slow thinking" into multimodal large language models (MLLMs). Contrary to existing methods that rely on direct or fast thinking, our key idea is to construct long chains of thought (CoT) consisting of atomic actions in a step-by-step manner, guiding MLLMs to perform complex reasoning. To this end, we design a novel AtomThink framework composed of three key modules: (i) a CoT annotation engine that automatically generates high-quality CoT annotations to address the lack of high-quality visual mathematical data; (ii) an atomic step fine-tuning strategy that jointly optimizes an MLLM and a policy reward model (PRM) for step-wise reasoning; and (iii) four different search strategies that can be applied with the PRM to complete reasoning. Additionally, we propose AtomMATH, a large-scale multimodal dataset of long CoTs, and an atomic capability evaluation metric for mathematical tasks. Extensive experimental results show that the proposed AtomThink significantly improves the performance of baseline MLLMs, achieving approximately 50\% relative accuracy gains on MathVista and 120\% on MathVerse. To support the advancement of multimodal slow-thinking models, we will make our code and dataset publicly available on this https URL.

Title: METEOR: Evolutionary Journey of Large Language Models from Guidance to Self-Growth

• Authors: Jiawei Li, Chong Feng, Yang Gao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11933
• Pdf URL: https://arxiv.org/pdf/2411.11933
• Copy Paste: [[2411.11933]] METEOR: Evolutionary Journey of Large Language Models from Guidance to Self-Growth(https://arxiv.org/abs/2411.11933)
• Keywords: large language model
• Abstract: Model evolution enables learning from feedback to refine experiences and update skills, transforming models from having no domain knowledge to becoming domain experts. However, there is currently no unified and effective method for guiding this evolutionary process. To address this gap, we propose the Meteor method, which includes three training phases: weak-to-strong data distillation, iterative training, and self-evolution strategies. Each phase maximizes the model's inherent domain capabilities, allowing it to autonomously refine its domain knowledge and enhance performance. Experiments demonstrate that our approach significantly improves accuracy, completeness, relevance, coherence, and reliability across domain-specific tasks.

Title: SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input

• Authors: Zhen Lv, Yangqi Long, Congzhentao Huang, Cao Li, Chengfei Lv, Hao Ren, Dian Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11934
• Pdf URL: https://arxiv.org/pdf/2411.11934
• Copy Paste: [[2411.11934]] SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input(https://arxiv.org/abs/2411.11934)
• Keywords: diffusion
• Abstract: Stereo video synthesis from a monocular input is a demanding task in the fields of spatial computing and virtual reality. The main challenges of this task lie on the insufficiency of high-quality paired stereo videos for training and the difficulty of maintaining the spatio-temporal consistency between frames. Existing methods primarily address these issues by directly applying novel view synthesis (NVS) techniques to video, while facing limitations such as the inability to effectively represent dynamic scenes and the requirement for large amounts of training data. In this paper, we introduce a novel self-supervised stereo video synthesis paradigm via a video diffusion model, termed SpatialDreamer, which meets the challenges head-on. Firstly, to address the stereo video data insufficiency, we propose a Depth based Video Generation module DVG, which employs a forward-backward rendering mechanism to generate paired videos with geometric and temporal priors. Leveraging data generated by DVG, we propose RefinerNet along with a self-supervised synthetic framework designed to facilitate efficient and dedicated training. More importantly, we devise a consistency control module, which consists of a metric of stereo deviation strength and a Temporal Interaction Learning module TIL for geometric and temporal consistency ensurance respectively. We evaluated the proposed method against various benchmark methods, with the results showcasing its superior performance.

Title: Calibrated and Efficient Sampling-Free Confidence Estimation for LiDAR Scene Semantic Segmentation

• Authors: Hanieh Shojaei Miandashti, Qianqian Zou, Claus Brenner
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.11935
• Pdf URL: https://arxiv.org/pdf/2411.11935
• Copy Paste: [[2411.11935]] Calibrated and Efficient Sampling-Free Confidence Estimation for LiDAR Scene Semantic Segmentation(https://arxiv.org/abs/2411.11935)
• Keywords: segmentation
• Abstract: Reliable deep learning models require not only accurate predictions but also well-calibrated confidence estimates to ensure dependable uncertainty estimation. This is crucial in safety-critical applications like autonomous driving, which depend on rapid and precise semantic segmentation of LiDAR point clouds for real-time 3D scene understanding. In this work, we introduce a sampling-free approach for estimating well-calibrated confidence values for classification tasks, achieving alignment with true classification accuracy and significantly reducing inference time compared to sampling-based methods. Our evaluation using the Adaptive Calibration Error (ACE) metric for LiDAR semantic segmentation shows that our approach maintains well-calibrated confidence values while achieving increased processing speed compared to a sampling baseline. Additionally, reliability diagrams reveal that our method produces underconfidence rather than overconfident predictions, an advantage for safety-critical applications. Our sampling-free approach offers well-calibrated and time-efficient predictions for LiDAR scene semantic segmentation.

Title: Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets

• Authors: Ike Obi, Rohan Pant, Srishti Shekhar Agrawal, Maham Ghazanfar, Aaron Basiletti
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11937
• Pdf URL: https://arxiv.org/pdf/2411.11937
• Copy Paste: [[2411.11937]] Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets(https://arxiv.org/abs/2411.11937)
• Keywords: transformer
• Abstract: LLMs are increasingly fine-tuned using RLHF datasets to align them with human preferences and values. However, very limited research has investigated which specific human values are operationalized through these datasets. In this paper, we introduce Value Imprint, a framework for auditing and classifying the human values embedded within RLHF datasets. To investigate the viability of this framework, we conducted three case study experiments by auditing the Anthropic/hh-rlhf, OpenAI WebGPT Comparisons, and Alpaca GPT-4-LLM datasets to examine the human values embedded within them. Our analysis involved a two-phase process. During the first phase, we developed a taxonomy of human values through an integrated review of prior works from philosophy, axiology, and ethics. Then, we applied this taxonomy to annotate 6,501 RLHF preferences. During the second phase, we employed the labels generated from the annotation as ground truth data for training a transformer-based machine learning model to audit and classify the three RLHF datasets. Through this approach, we discovered that information-utility values, including Wisdom/Knowledge and Information Seeking, were the most dominant human values within all three RLHF datasets. In contrast, prosocial and democratic values, including Well-being, Justice, and Human/Animal Rights, were the least represented human values. These findings have significant implications for developing language models that align with societal values and norms. We contribute our datasets to support further research in this area.

Title: Fair Distillation: Teaching Fairness from Biased Teachers in Medical Imaging

• Authors: Milad Masroor, Tahir Hassan, Yu Tian, Kevin Wells, David Rosewarne, Thanh-Toan Do, Gustavo Carneiro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11939
• Pdf URL: https://arxiv.org/pdf/2411.11939
• Copy Paste: [[2411.11939]] Fair Distillation: Teaching Fairness from Biased Teachers in Medical Imaging(https://arxiv.org/abs/2411.11939)
• Keywords: fair, segmentation
• Abstract: Deep learning has achieved remarkable success in image classification and segmentation tasks. However, fairness concerns persist, as models often exhibit biases that disproportionately affect demographic groups defined by sensitive attributes such as race, gender, or age. Existing bias-mitigation techniques, including Subgroup Re-balancing, Adversarial Training, and Domain Generalization, aim to balance accuracy across demographic groups, but often fail to simultaneously improve overall accuracy, group-specific accuracy, and fairness due to conflicts among these interdependent objectives. We propose the Fair Distillation (FairDi) method, a novel fairness approach that decomposes these objectives by leveraging biased ``teacher'' models, each optimized for a specific demographic group. These teacher models then guide the training of a unified ``student'' model, which distills their knowledge to maximize overall and group-specific accuracies, while minimizing inter-group disparities. Experiments on medical imaging datasets show that FairDi achieves significant gains in both overall and group-specific accuracy, along with improved fairness, compared to existing methods. FairDi is adaptable to various medical tasks, such as classification and segmentation, and provides an effective solution for equitable model performance.

Title: TimeFormer: Capturing Temporal Relationships of Deformable 3D Gaussians for Robust Reconstruction

• Authors: DaDong Jiang, Zhihui Ke, Xiaobo Zhou, Zhi Hou, Xianghui Yang, Wenbo Hu, Tie Qiu, Chunchao Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.11941
• Pdf URL: https://arxiv.org/pdf/2411.11941
• Copy Paste: [[2411.11941]] TimeFormer: Capturing Temporal Relationships of Deformable 3D Gaussians for Robust Reconstruction(https://arxiv.org/abs/2411.11941)
• Keywords: robust, transformer
• Abstract: Dynamic scene reconstruction is a long-term challenge in 3D vision. Recent methods extend 3D Gaussian Splatting to dynamic scenes via additional deformation fields and apply explicit constraints like motion flow to guide the deformation. However, they learn motion changes from individual timestamps independently, making it challenging to reconstruct complex scenes, particularly when dealing with violent movement, extreme-shaped geometries, or reflective surfaces. To address the above issue, we design a plug-and-play module called TimeFormer to enable existing deformable 3D Gaussians reconstruction methods with the ability to implicitly model motion patterns from a learning perspective. Specifically, TimeFormer includes a Cross-Temporal Transformer Encoder, which adaptively learns the temporal relationships of deformable 3D Gaussians. Furthermore, we propose a two-stream optimization strategy that transfers the motion knowledge learned from TimeFormer to the base stream during the training phase. This allows us to remove TimeFormer during inference, thereby preserving the original rendering speed. Extensive experiments in the multi-view and monocular dynamic scenes validate qualitative and quantitative improvement brought by TimeFormer. Project Page: this https URL

Title: Medical Video Generation for Disease Progression Simulation

• Authors: Xu Cao, Kaizhao Liang, Kuei-Da Liao, Tianren Gao, Wenqian Ye, Jintai Chen, Zhiguang Ding, Jianguo Cao, James M. Rehg, Jimeng Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.11943
• Pdf URL: https://arxiv.org/pdf/2411.11943
• Copy Paste: [[2411.11943]] Medical Video Generation for Disease Progression Simulation(https://arxiv.org/abs/2411.11943)
• Keywords: diffusion, large language model
• Abstract: Modeling disease progression is crucial for improving the quality and efficacy of clinical diagnosis and prognosis, but it is often hindered by a lack of longitudinal medical image monitoring for individual patients. To address this challenge, we propose the first Medical Video Generation (MVG) framework that enables controlled manipulation of disease-related image and video features, allowing precise, realistic, and personalized simulations of disease progression. Our approach begins by leveraging large language models (LLMs) to recaption prompt for disease trajectory. Next, a controllable multi-round diffusion model simulates the disease progression state for each patient, creating realistic intermediate disease state sequence. Finally, a diffusion-based video transition generation model interpolates disease progression between these states. We validate our framework across three medical imaging domains: chest X-ray, fundus photography, and skin image. Our results demonstrate that MVG significantly outperforms baseline models in generating coherent and clinically plausible disease trajectories. Two user studies by veteran physicians, provide further validation and insights into the clinical utility of the generated sequences. MVG has the potential to assist healthcare providers in modeling disease trajectories, interpolating missing medical image data, and enhancing medical education through realistic, dynamic visualizations of disease progression.

Title: Transmission Line Outage Probability Prediction Under Extreme Events Using Peter-Clark Bayesian Structural Learning

• Authors: Xiaolin Chen, Qiuhua Huang, Yuqi Zhou
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.11980
• Pdf URL: https://arxiv.org/pdf/2411.11980
• Copy Paste: [[2411.11980]] Transmission Line Outage Probability Prediction Under Extreme Events Using Peter-Clark Bayesian Structural Learning(https://arxiv.org/abs/2411.11980)
• Keywords: robust
• Abstract: Recent years have seen a notable increase in the frequency and intensity of extreme weather events. With a rising number of power outages caused by these events, accurate prediction of power line outages is essential for safe and reliable operation of power grids. The Bayesian network is a probabilistic model that is very effective for predicting line outages under weather-related uncertainties. However, most existing studies in this area offer general risk assessments, but fall short of providing specific outage probabilities. In this work, we introduce a novel approach for predicting transmission line outage probabilities using a Bayesian network combined with Peter-Clark (PC) structural learning. Our approach not only enables precise outage probability calculations, but also demonstrates better scalability and robust performance, even with limited data. Case studies using data from BPA and NOAA show the effectiveness of this approach, while comparisons with several existing methods further highlight its advantages.

Title: Understanding Chain-of-Thought in LLMs through Information Theory

• Authors: Jean-Francois Ton, Muhammad Faaiz Taufiq, Yang Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.11984
• Pdf URL: https://arxiv.org/pdf/2411.11984
• Copy Paste: [[2411.11984]] Understanding Chain-of-Thought in LLMs through Information Theory(https://arxiv.org/abs/2411.11984)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown impressive performance in complex reasoning tasks through Chain-of-Thought (CoT) reasoning, allowing models to break down problems into manageable sub-tasks. However, existing CoT evaluation techniques either require annotated CoT data or fall short in accurately assessing intermediate reasoning steps, leading to high rates of false positives. In this paper, we formalize CoT reasoning in LLMs through an information-theoretic lens. Specifically, our framework quantifies the `information gain' at each reasoning step, enabling the identification of failure modes in LLMs without the need for expensive annotated datasets. We demonstrate the efficacy of our approach through extensive experiments on toy and GSM-8K data, where it significantly outperforms existing outcome-based methods by providing more accurate insights into model performance on individual tasks.

Title: ByteScience: Bridging Unstructured Scientific Literature and Structured Data with Auto Fine-tuned Large Language Model in Token Granularity

• Authors: Tong Xie, Hanzhi Zhang, Shaozhou Wang, Yuwei Wan, Imran Razzak, Chunyu Kit, Wenjie Zhangand Bram Hoex
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12000
• Pdf URL: https://arxiv.org/pdf/2411.12000
• Copy Paste: [[2411.12000]] ByteScience: Bridging Unstructured Scientific Literature and Structured Data with Auto Fine-tuned Large Language Model in Token Granularity(https://arxiv.org/abs/2411.12000)
• Keywords: extraction, large language model
• Abstract: Natural Language Processing (NLP) is widely used to supply summarization ability from long context to structured information. However, extracting structured knowledge from scientific text by NLP models remains a challenge because of its domain-specific nature to complex data preprocessing and the granularity of multi-layered device-level information. To address this, we introduce ByteScience, a non-profit cloud-based auto fine-tuned Large Language Model (LLM) platform, which is designed to extract structured scientific data and synthesize new scientific knowledge from vast scientific corpora. The platform capitalizes on DARWIN, an open-source, fine-tuned LLM dedicated to natural science. The platform was built on Amazon Web Services (AWS) and provides an automated, user-friendly workflow for custom model development and data extraction. The platform achieves remarkable accuracy with only a small amount of well-annotated articles. This innovative tool streamlines the transition from the science literature to structured knowledge and data and benefits the advancements in natural informatics.

Title: Analyzing and Improving the Skin Tone Consistency and Bias in Implicit 3D Relightable Face Generators

• Authors: Libing Zeng, Nima Khademi Kalantari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12002
• Pdf URL: https://arxiv.org/pdf/2411.12002
• Copy Paste: [[2411.12002]] Analyzing and Improving the Skin Tone Consistency and Bias in Implicit 3D Relightable Face Generators(https://arxiv.org/abs/2411.12002)
• Keywords: generative
• Abstract: With the advances in generative adversarial networks (GANs) and neural rendering, 3D relightable face generation has received significant attention. Among the existing methods, a particularly successful technique uses an implicit lighting representation and generates relit images through the product of synthesized albedo and light-dependent shading images. While this approach produces high-quality results with intricate shading details, it often has difficulty producing relit images with consistent skin tones, particularly when the lighting condition is extracted from images of individuals with dark skin. Additionally, this technique is biased towards producing albedo images with lighter skin tones. Our main observation is that this problem is rooted in the biased spherical harmonics (SH) coefficients, used during training. Following this observation, we conduct an analysis and demonstrate that the bias appears not only in band 0 (DC term), but also in the other bands of the estimated SH coefficients. We then propose a simple, but effective, strategy to mitigate the problem. Specifically, we normalize the SH coefficients by their DC term to eliminate the inherent magnitude bias, while statistically align the coefficients in the other bands to alleviate the directional bias. We also propose a scaling strategy to match the distribution of illumination magnitude in the generated images with the training data. Through extensive experiments, we demonstrate the effectiveness of our solution in increasing the skin tone consistency and mitigating bias.

Title: In-Situ Melt Pool Characterization via Thermal Imaging for Defect Detection in Directed Energy Deposition Using Vision Transformers

• Authors: Israt Zarin Era, Fan Zhou, Ahmed Shoyeb Raihan, Imtiaz Ahmed, Alan Abul-Haj, James Craig, Srinjoy Das, Zhichao Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12028
• Pdf URL: https://arxiv.org/pdf/2411.12028
• Copy Paste: [[2411.12028]] In-Situ Melt Pool Characterization via Thermal Imaging for Defect Detection in Directed Energy Deposition Using Vision Transformers(https://arxiv.org/abs/2411.12028)
• Keywords: transformer
• Abstract: Directed Energy Deposition (DED) offers significant potential for manufacturing complex and multi-material parts. However, internal defects such as porosity and cracks can compromise mechanical properties and overall performance. This study focuses on in-situ monitoring and characterization of melt pools associated with porosity, aiming to improve defect detection and quality control in DED-printed parts. Traditional machine learning approaches for defect identification rely on extensive labeled datasets, often scarce and expensive to generate in real-world manufacturing. To address this, our framework employs self-supervised learning on unlabeled melt pool data using a Vision Transformer-based Masked Autoencoder (MAE) to produce highly representative embeddings. These fine-tuned embeddings are leveraged via transfer learning to train classifiers on a limited labeled dataset, enabling the effective identification of melt pool anomalies. We evaluate two classifiers: (1) a Vision Transformer (ViT) classifier utilizing the fine-tuned MAE Encoder's parameters and (2) the fine-tuned MAE Encoder combined with an MLP classifier head. Our framework achieves overall accuracy ranging from 95.44% to 99.17% and an average F1 score exceeding 80%, with the ViT Classifier slightly outperforming the MAE Encoder Classifier. This demonstrates the scalability and cost-effectiveness of our approach for automated quality control in DED, effectively detecting defects with minimal labeled data.

Title: Machine Learning Evaluation Metric Discrepancies across Programming Languages and Their Components: Need for Standardization

• Authors: Mohammad R. Salmanpour, Morteza Alizadeh, Ghazal Mousavi, Saba Sadeghi, Sajad Amiri, Mehrdad Oveisi, Arman Rahmim, Ilker Hacihaliloglu
• Subjects: cs.LG, cs.SE, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2411.12032
• Pdf URL: https://arxiv.org/pdf/2411.12032
• Copy Paste: [[2411.12032]] Machine Learning Evaluation Metric Discrepancies across Programming Languages and Their Components: Need for Standardization(https://arxiv.org/abs/2411.12032)
• Keywords: segmentation
• Abstract: This study evaluates metrics for tasks such as classification, regression, clustering, correlation analysis, statistical tests, segmentation, and image-to-image (I2I) translation. Metrics were compared across Python libraries, R packages, and Matlab functions to assess their consistency and highlight discrepancies. The findings underscore the need for a unified roadmap to standardize metrics, ensuring reliable and reproducible ML evaluations across platforms. This study examined a wide range of evaluation metrics across various tasks and found only some to be consistent across platforms, such as (i) Accuracy, Balanced Accuracy, Cohens Kappa, F-beta Score, MCC, Geometric Mean, AUC, and Log Loss in binary classification; (ii) Accuracy, Cohens Kappa, and F-beta Score in multi-class classification; (iii) MAE, MSE, RMSE, MAPE, Explained Variance, Median AE, MSLE, and Huber in regression; (iv) Davies-Bouldin Index and Calinski-Harabasz Index in clustering; (v) Pearson, Spearman, Kendall's Tau, Mutual Information, Distance Correlation, Percbend, Shepherd, and Partial Correlation in correlation analysis; (vi) Paired t-test, Chi-Square Test, ANOVA, Kruskal-Wallis Test, Shapiro-Wilk Test, Welchs t-test, and Bartlett's test in statistical tests; (vii) Accuracy, Precision, and Recall in 2D segmentation; (viii) Accuracy in 3D segmentation; (ix) MAE, MSE, RMSE, and R-Squared in 2D-I2I translation; and (x) MAE, MSE, and RMSE in 3D-I2I translation. Given observation of discrepancies in a number of metrics (e.g. precision, recall and F1 score in binary classification, WCSS in clustering, multiple statistical tests, and IoU in segmentation, amongst multiple metrics), this study concludes that ML evaluation metrics require standardization and recommends that future research use consistent metrics for different tasks to effectively compare ML techniques and solutions.

Title: Scaling Deep Learning Research with Kubernetes on the NRP Nautilus HyperCluster

• Authors: J. Alex Hurt, Anes Ouadou, Mariam Alshehri, Grant J. Scott
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.12038
• Pdf URL: https://arxiv.org/pdf/2411.12038
• Copy Paste: [[2411.12038]] Scaling Deep Learning Research with Kubernetes on the NRP Nautilus HyperCluster(https://arxiv.org/abs/2411.12038)
• Keywords: robust, segmentation
• Abstract: Throughout the scientific computing space, deep learning algorithms have shown excellent performance in a wide range of applications. As these deep neural networks (DNNs) continue to mature, the necessary compute required to train them has continued to grow. Today, modern DNNs require millions of FLOPs and days to weeks of training to generate a well-trained model. The training times required for DNNs are oftentimes a bottleneck in DNN research for a variety of deep learning applications, and as such, accelerating and scaling DNN training enables more robust and accelerated research. To that end, in this work, we explore utilizing the NRP Nautilus HyperCluster to automate and scale deep learning model training for three separate applications of DNNs, including overhead object detection, burned area segmentation, and deforestation detection. In total, 234 deep neural models are trained on Nautilus, for a total time of 4,040 hours

Title: ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancements

• Authors: M. Arda Aydın, Efe Mert Çırpar, Elvin Abdinli, Gozde Unal, Yusuf H. Sahin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12044
• Pdf URL: https://arxiv.org/pdf/2411.12044
• Copy Paste: [[2411.12044]] ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancements(https://arxiv.org/abs/2411.12044)
• Keywords: large language model, segmentation
• Abstract: Recent advances in foundational Vision Language Models (VLMs) have reshaped the evaluation paradigm in computer vision tasks. These foundational models, especially CLIP, have accelerated research in open-vocabulary computer vision tasks, including Open-Vocabulary Semantic Segmentation (OVSS). Although the initial results are promising, the dense prediction capabilities of VLMs still require further improvement. In this study, we enhance the semantic segmentation performance of CLIP by introducing new modules and modifications: 1) architectural changes in the last layer of ViT and the incorporation of attention maps from the middle layers with the last layer, 2) Image Engineering: applying data augmentations to enrich input image representations, and 3) using Large Language Models (LLMs) to generate definitions and synonyms for each class name to leverage CLIP's open-vocabulary capabilities. Our training-free method, ITACLIP, outperforms current state-of-the-art approaches on segmentation benchmarks such as COCO-Stuff, COCO-Object, Pascal Context, and Pascal VOC. Our code is available at this https URL.

Title: Fingerprinting and Tracing Shadows: The Development and Impact of Browser Fingerprinting on Digital Privacy

• Authors: Alexander Lawall
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.12045
• Pdf URL: https://arxiv.org/pdf/2411.12045
• Copy Paste: [[2411.12045]] Fingerprinting and Tracing Shadows: The Development and Impact of Browser Fingerprinting on Digital Privacy(https://arxiv.org/abs/2411.12045)
• Keywords: privacy
• Abstract: Browser fingerprinting is a growing technique for identifying and tracking users online without traditional methods like cookies. This paper gives an overview by examining the various fingerprinting techniques and analyzes the entropy and uniqueness of the collected data. The analysis highlights that browser fingerprinting poses a complex challenge from both technical and privacy perspectives, as users often have no control over the collection and use of their data. In addition, it raises significant privacy concerns as users are often tracked without their knowledge or consent.

Title: Higher Order Graph Attention Probabilistic Walk Networks

• Authors: Thomas Bailie, Yun Sing Koh, Karthik Mukkavilli
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12052
• Pdf URL: https://arxiv.org/pdf/2411.12052
• Copy Paste: [[2411.12052]] Higher Order Graph Attention Probabilistic Walk Networks(https://arxiv.org/abs/2411.12052)
• Keywords: robust
• Abstract: Graphs inherently capture dependencies between nodes or variables through their topological structure, with paths between any two nodes indicating a sequential dependency on the nodes traversed. Message Passing Neural Networks (MPNNs) leverage these latent relationships embedded in graph structures, and have become widely adopted across diverse applications. However, many existing methods predominantly rely on local information within the $1$-hop neighborhood. This approach has notable limitations; for example, $1$-hop aggregation schemes inherently lose long-distance information, and are limited in expressive power as defined by the $k$-Weisfeiler-Leman ($k$-WL) isomorphism test. To address these issues, we propose the Higher Order Graphical Attention (HoGA) module, which assigns weights to variable-length paths sampled based on feature-vector diversity, effectively reconstructing the $k$-hop neighborhood. HoGA represents higher-order relationships as a robust form of self-attention, applicable to any single-hop attention mechanism. In empirical studies, applying HoGA to existing attention-based models consistently leads to significant accuracy improvements on benchmark node classification datasets. Furthermore, we observe that the performance degradation typically associated with additional message-passing steps may be mitigated.

Title: Benchmarking pre-trained text embedding models in aligning built asset information

• Authors: Mehrzad Shahinmoghadam, Ali Motamedi
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12056
• Pdf URL: https://arxiv.org/pdf/2411.12056
• Copy Paste: [[2411.12056]] Benchmarking pre-trained text embedding models in aligning built asset information(https://arxiv.org/abs/2411.12056)
• Keywords: large language model
• Abstract: Accurate mapping of the built asset information to established data classification systems and taxonomies is crucial for effective asset management, whether for compliance at project handover or ad-hoc data integration scenarios. Due to the complex nature of built asset data, which predominantly comprises technical text elements, this process remains largely manual and reliant on domain expert input. Recent breakthroughs in contextual text representation learning (text embedding), particularly through pre-trained large language models, offer promising approaches that can facilitate the automation of cross-mapping of the built asset data. However, no comprehensive evaluation has yet been conducted to assess these models' ability to effectively represent the complex semantics specific to built asset technical terminology. This study presents a comparative benchmark of state-of-the-art text embedding models to evaluate their effectiveness in aligning built asset information with domain-specific technical concepts. Our proposed datasets are derived from two renowned built asset data classification dictionaries. The results of our benchmarking across six proposed datasets, covering three tasks of clustering, retrieval, and reranking, highlight the need for future research on domain adaptation techniques. The benchmarking resources are published as an open-source library, which will be maintained and extended to support future evaluations in this field.

Title: Autoassociative Learning of Structural Representations for Modeling and Classification in Medical Imaging

• Authors: Zuzanna Buchnajzer, Kacper Dobek, Stanisław Hapke, Daniel Jankowski, Krzysztof Krawiec
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12070
• Pdf URL: https://arxiv.org/pdf/2411.12070
• Copy Paste: [[2411.12070]] Autoassociative Learning of Structural Representations for Modeling and Classification in Medical Imaging(https://arxiv.org/abs/2411.12070)
• Keywords: robust
• Abstract: Deep learning architectures based on convolutional neural networks tend to rely on continuous, smooth features. While this characteristics provides significant robustness and proves useful in many real-world tasks, it is strikingly incompatible with the physical characteristic of the world, which, at the scale in which humans operate, comprises crisp objects, typically representing well-defined categories. This study proposes a class of neurosymbolic systems that learn by reconstructing the observed images in terms of visual primitives and are thus forced to form high-level, structural explanations of them. When applied to the task of diagnosing abnormalities in histological imaging, the method proved superior to a conventional deep learning architecture in terms of classification accuracy, while being more transparent.

Title: Theoretical Corrections and the Leveraging of Reinforcement Learning to Enhance Triangle Attack

• Authors: Nicole Meng, Caleb Manicke, David Chen, Yingjie Lao, Caiwen Ding, Pengyu Hong, Kaleel Mahmood
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12071
• Pdf URL: https://arxiv.org/pdf/2411.12071
• Copy Paste: [[2411.12071]] Theoretical Corrections and the Leveraging of Reinforcement Learning to Enhance Triangle Attack(https://arxiv.org/abs/2411.12071)
• Keywords: defense, attack
• Abstract: Adversarial examples represent a serious issue for the application of machine learning models in many sensitive domains. For generating adversarial examples, decision based black-box attacks are one of the most practical techniques as they only require query access to the model. One of the most recently proposed state-of-the-art decision based black-box attacks is Triangle Attack (TA). In this paper, we offer a high-level description of TA and explain potential theoretical limitations. We then propose a new decision based black-box attack, Triangle Attack with Reinforcement Learning (TARL). Our new attack addresses the limits of TA by leveraging reinforcement learning. This creates an attack that can achieve similar, if not better, attack accuracy than TA with half as many queries on state-of-the-art classifiers and defenses across ImageNet and CIFAR-10.

Title: Zoomed In, Diffused Out: Towards Local Degradation-Aware Multi-Diffusion for Extreme Image Super-Resolution

• Authors: Brian B. Moser, Stanislav Frolov, Tobias C. Nauen, Federico Raue, Andreas Dengel
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.12072
• Pdf URL: https://arxiv.org/pdf/2411.12072
• Copy Paste: [[2411.12072]] Zoomed In, Diffused Out: Towards Local Degradation-Aware Multi-Diffusion for Extreme Image Super-Resolution(https://arxiv.org/abs/2411.12072)
• Keywords: extraction, diffusion
• Abstract: Large-scale, pre-trained Text-to-Image (T2I) diffusion models have gained significant popularity in image generation tasks and have shown unexpected potential in image Super-Resolution (SR). However, most existing T2I diffusion models are trained with a resolution limit of 512x512, making scaling beyond this resolution an unresolved but necessary challenge for image SR. In this work, we introduce a novel approach that, for the first time, enables these models to generate 2K, 4K, and even 8K images without any additional training. Our method leverages MultiDiffusion, which distributes the generation across multiple diffusion paths to ensure global coherence at larger scales, and local degradation-aware prompt extraction, which guides the T2I model to reconstruct fine local structures according to its low-resolution input. These innovations unlock higher resolutions, allowing T2I diffusion models to be applied to image SR tasks without limitation on resolution.

Title: Just Leaf It: Accelerating Diffusion Classifiers with Hierarchical Class Pruning

• Authors: Arundhati S. Shanbhag, Brian B. Moser, Tobias C. Nauen, Stanislav Frolov, Federico Raue, Andreas Dengel
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12073
• Pdf URL: https://arxiv.org/pdf/2411.12073
• Copy Paste: [[2411.12073]] Just Leaf It: Accelerating Diffusion Classifiers with Hierarchical Class Pruning(https://arxiv.org/abs/2411.12073)
• Keywords: diffusion, generative
• Abstract: Diffusion models, known for their generative capabilities, have recently shown unexpected potential in image classification tasks by using Bayes' theorem. However, most diffusion classifiers require evaluating all class labels for a single classification, leading to significant computational costs that can hinder their application in large-scale scenarios. To address this, we present a Hierarchical Diffusion Classifier (HDC) that exploits the inherent hierarchical label structure of a dataset. By progressively pruning irrelevant high-level categories and refining predictions only within relevant subcategories, i.e., leaf nodes, HDC reduces the total number of class evaluations. As a result, HDC can accelerate inference by up to 60% while maintaining and, in some cases, improving classification accuracy. Our work enables a new control mechanism of the trade-off between speed and precision, making diffusion-based classification more viable for real-world applications, particularly in large-scale image classification tasks.

Title: Molecule Generation with Fragment Retrieval Augmentation

• Authors: Seul Lee, Karsten Kreis, Srimukh Prasad Veccham, Meng Liu, Danny Reidenbach, Saee Paliwal, Arash Vahdat, Weili Nie
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12078
• Pdf URL: https://arxiv.org/pdf/2411.12078
• Copy Paste: [[2411.12078]] Molecule Generation with Fragment Retrieval Augmentation(https://arxiv.org/abs/2411.12078)
• Keywords: generative
• Abstract: Fragment-based drug discovery, in which molecular fragments are assembled into new molecules with desirable biochemical properties, has achieved great success. However, many fragment-based molecule generation methods show limited exploration beyond the existing fragments in the database as they only reassemble or slightly modify the given ones. To tackle this problem, we propose a new fragment-based molecule generation framework with retrieval augmentation, namely Fragment Retrieval-Augmented Generation (f-RAG). f-RAG is based on a pre-trained molecular generative model that proposes additional fragments from input fragments to complete and generate a new molecule. Given a fragment vocabulary, f-RAG retrieves two types of fragments: (1) hard fragments, which serve as building blocks that will be explicitly included in the newly generated molecule, and (2) soft fragments, which serve as reference to guide the generation of new fragments through a trainable fragment injection module. To extrapolate beyond the existing fragments, f-RAG updates the fragment vocabulary with generated fragments via an iterative refinement process which is further enhanced with post-hoc genetic fragment modification. f-RAG can achieve an improved exploration-exploitation trade-off by maintaining a pool of fragments and expanding it with novel and high-quality fragments through a strong generative prior.

Title: FruitNinja: 3D Object Interior Texture Generation with Gaussian Splatting

• Authors: Fangyu Wu, Yuhao Chen
• Subjects: cs.CV, cs.GR, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.12089
• Pdf URL: https://arxiv.org/pdf/2411.12089
• Copy Paste: [[2411.12089]] FruitNinja: 3D Object Interior Texture Generation with Gaussian Splatting(https://arxiv.org/abs/2411.12089)
• Keywords: diffusion, generative
• Abstract: In the real world, objects reveal internal textures when sliced or cut, yet this behavior is not well-studied in 3D generation tasks today. For example, slicing a virtual 3D watermelon should reveal flesh and seeds. Given that no available dataset captures an object's full internal structure and collecting data from all slices is impractical, generative methods become the obvious approach. However, current 3D generation and inpainting methods often focus on visible appearance and overlook internal textures. To bridge this gap, we introduce FruitNinja, the first method to generate internal textures for 3D objects undergoing geometric and topological changes. Our approach produces objects via 3D Gaussian Splatting (3DGS) with both surface and interior textures synthesized, enabling real-time slicing and rendering without additional optimization. FruitNinja leverages a pre-trained diffusion model to progressively inpaint cross-sectional views and applies voxel-grid-based smoothing to achieve cohesive textures throughout the object. Our OpaqueAtom GS strategy overcomes 3DGS limitations by employing densely distributed opaque Gaussians, avoiding biases toward larger particles that destabilize training and sharp color transitions for fine-grained textures. Experimental results show that FruitNinja substantially outperforms existing approaches, showcasing unmatched visual quality in real-time rendered internal views across arbitrary geometry manipulations.

Title: Federated Contrastive Learning of Graph-Level Representations

• Authors: Xiang Li, Gagan Agrawal, Rajiv Ramnath, Ruoming Jin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12098
• Pdf URL: https://arxiv.org/pdf/2411.12098
• Copy Paste: [[2411.12098]] Federated Contrastive Learning of Graph-Level Representations(https://arxiv.org/abs/2411.12098)
• Keywords: privacy, federate
• Abstract: Graph-level representations (and clustering/classification based on these representations) are required in a variety of applications. Examples include identifying malicious network traffic, prediction of protein properties, and many others. Often, data has to stay in isolated local systems (i.e., cannot be centrally shared for analysis) due to a variety of considerations like privacy concerns, lack of trust between the parties, regulations, or simply because the data is too large to be shared sufficiently quickly. This points to the need for federated learning for graph-level representations, a topic that has not been explored much, especially in an unsupervised setting. Addressing this problem, this paper presents a new framework we refer to as Federated Contrastive Learning of Graph-level Representations (FCLG). As the name suggests, our approach builds on contrastive learning. However, what is unique is that we apply contrastive learning at two levels. The first application is for local unsupervised learning of graph representations. The second level is to address the challenge associated with data distribution variation (i.e. the ``Non-IID issue") when combining local models. Through extensive experiments on the downstream task of graph-level clustering, we demonstrate FCLG outperforms baselines (which apply existing federated methods on existing graph-level clustering methods) with significant margins.

Title: Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods

• Authors: Jai Doshi, Asa Cooper Stickland
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12103
• Pdf URL: https://arxiv.org/pdf/2411.12103
• Copy Paste: [[2411.12103]] Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods(https://arxiv.org/abs/2411.12103)
• Keywords: robust, large language model
• Abstract: Large language model unlearning aims to remove harmful information that LLMs have learnt to prevent their use for malicious purposes. LLMU and RMU have been proposed as two methods for LLM unlearning, achieving impressive results on unlearning benchmarks. We study in detail the efficacy of these methods by evaluating their impact on general model capabilities on the WMDP benchmark as well as a biology benchmark we create. Our experiments show that RMU generally leads to better preservation of model capabilities, for similar or better unlearning. We further test the robustness of these methods and find that doing 5-shot prompting or rephrasing the question in simple ways can lead to an over ten-fold increase in accuracy on unlearning benchmarks. Finally, we show that training on unrelated data can almost completely recover pre-unlearning performance, demonstrating that these methods fail at truly unlearning. The code is available at $\href{this https URL}{this\, https\, URL}$.

Title: Distill the Best, Ignore the Rest: Improving Dataset Distillation with Loss-Value-Based Pruning

• Authors: Brian B. Moser, Federico Raue, Tobias C. Nauen, Stanislav Frolov, Andreas Dengel
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12115
• Pdf URL: https://arxiv.org/pdf/2411.12115
• Copy Paste: [[2411.12115]] Distill the Best, Ignore the Rest: Improving Dataset Distillation with Loss-Value-Based Pruning(https://arxiv.org/abs/2411.12115)
• Keywords: robust, generative
• Abstract: Dataset distillation has gained significant interest in recent years, yet existing approaches typically distill from the entire dataset, potentially including non-beneficial samples. We introduce a novel "Prune First, Distill After" framework that systematically prunes datasets via loss-based sampling prior to distillation. By leveraging pruning before classical distillation techniques and generative priors, we create a representative core-set that leads to enhanced generalization for unseen architectures - a significant challenge of current distillation methods. More specifically, our proposed framework significantly boosts distilled quality, achieving up to a 5.2 percentage points accuracy increase even with substantial dataset pruning, i.e., removing 80% of the original dataset prior to distillation. Overall, our experimental results highlight the advantages of our easy-sample prioritization and cross-architecture robustness, paving the way for more effective and high-quality dataset distillation.

Title: Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers

• Authors: Tiberiu Musat
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12118
• Pdf URL: https://arxiv.org/pdf/2411.12118
• Copy Paste: [[2411.12118]] Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers(https://arxiv.org/abs/2411.12118)
• Keywords: transformer, large language model
• Abstract: In this paper, I introduce the retrieval problem, a simple reasoning task that can be solved only by transformers with a minimum number of layers. The task has an adjustable difficulty that can further increase the required number of layers to any arbitrary value. I demonstrate that large language models can solve the task under different prompting formulations without any fine-tuning. To understand how transformers solve the retrieval problem, I train several transformers on a minimal formulation. I find that successful learning occurs only under the presence of an implicit curriculum. I uncover the learned mechanisms by studying the attention maps in the trained transformers. I also study the training process, uncovering that attention heads always emerge in a specific sequence.

Title: Visualizing Loss Functions as Topological Landscape Profiles

• Authors: Caleb Geniesse, Jiaqing Chen, Tiankai Xie, Ge Shi, Yaoqing Yang, Dmitriy Morozov, Talita Perciano, Michael W. Mahoney, Ross Maciejewski, Gunther H. Weber
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12136
• Pdf URL: https://arxiv.org/pdf/2411.12136
• Copy Paste: [[2411.12136]] Visualizing Loss Functions as Topological Landscape Profiles(https://arxiv.org/abs/2411.12136)
• Keywords: segmentation
• Abstract: In machine learning, a loss function measures the difference between model predictions and ground-truth (or target) values. For neural network models, visualizing how this loss changes as model parameters are varied can provide insights into the local structure of the so-called loss landscape (e.g., smoothness) as well as global properties of the underlying model (e.g., generalization performance). While various methods for visualizing the loss landscape have been proposed, many approaches limit sampling to just one or two directions, ignoring potentially relevant information in this extremely high-dimensional space. This paper introduces a new representation based on topological data analysis that enables the visualization of higher-dimensional loss landscapes. After describing this new topological landscape profile representation, we show how the shape of loss landscapes can reveal new details about model performance and learning dynamics, highlighting several use cases, including image segmentation (e.g., UNet) and scientific machine learning (e.g., physics-informed neural networks). Through these examples, we provide new insights into how loss landscapes vary across distinct hyperparameter spaces: we find that the topology of the loss landscape is simpler for better-performing models; and we observe greater variation in the shape of loss landscapes near transitions from low to high model performance.

Title: A Computational Method for Measuring "Open Codes" in Qualitative Analysis

• Authors: John Chen, Alexandros Lotsos, Lexie Zhao, Jessica Hullman, Bruce Sherin, Uri Wilensky, Michael Horn
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12142
• Pdf URL: https://arxiv.org/pdf/2411.12142
• Copy Paste: [[2411.12142]] A Computational Method for Measuring "Open Codes" in Qualitative Analysis(https://arxiv.org/abs/2411.12142)
• Keywords: generative
• Abstract: Qualitative analysis is critical to understanding human datasets in many social science disciplines. Open coding is an inductive qualitative process that identifies and interprets "open codes" from datasets. Yet, meeting methodological expectations (such as "as exhaustive as possible") can be challenging. While many machine learning (ML)/generative AI (GAI) studies have attempted to support open coding, few have systematically measured or evaluated GAI outcomes, increasing potential bias risks. Building on Grounded Theory and Thematic Analysis theories, we present a computational method to measure and identify potential biases from "open codes" systematically. Instead of operationalizing human expert results as the "ground truth," our method is built upon a team-based approach between human and machine coders. We experiment with two HCI datasets to establish this method's reliability by 1) comparing it with human analysis, and 2) analyzing its output stability. We present evidence-based suggestions and example workflows for ML/GAI to support open coding.

Title: Self-Supervised Learning in Deep Networks: A Pathway to Robust Few-Shot Classification

• Authors: Yuyang Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12151
• Pdf URL: https://arxiv.org/pdf/2411.12151
• Copy Paste: [[2411.12151]] Self-Supervised Learning in Deep Networks: A Pathway to Robust Few-Shot Classification(https://arxiv.org/abs/2411.12151)
• Keywords: robust, extraction
• Abstract: This study aims to optimize the few-shot image classification task and improve the model's feature extraction and classification performance by combining self-supervised learning with the deep network model ResNet-101. During the training process, we first pre-train the model with self-supervision to enable it to learn common feature expressions on a large amount of unlabeled data; then fine-tune it on the few-shot dataset Mini-ImageNet to improve the model's accuracy and generalization ability under limited data. The experimental results show that compared with traditional convolutional neural networks, ResNet-50, DenseNet, and other models, our method has achieved excellent performance of about 95.12% in classification accuracy (ACC) and F1 score, verifying the effectiveness of self-supervised learning in few-shot classification. This method provides an efficient and reliable solution for the field of few-shot image classification.

Title: A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation

• Authors: Jiajing Chen, Shuo Wang, Zhen Qi, Zhenhong Zhang, Chihang Wang, Hongye Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12157
• Pdf URL: https://arxiv.org/pdf/2411.12157
• Copy Paste: [[2411.12157]] A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation(https://arxiv.org/abs/2411.12157)
• Keywords: transformer, generative
• Abstract: This research introduces a novel text generation model that combines BERT's semantic interpretation strengths with GPT-4's generative capabilities, establishing a high standard in generating coherent, contextually accurate language. Through the combined architecture, the model enhances semantic depth and maintains smooth, human-like text flow, overcoming limitations seen in prior models. Experimental benchmarks reveal that BERT-GPT-4 surpasses traditional models, including GPT-3, T5, BART, Transformer-XL, and CTRL, in key metrics like Perplexity and BLEU, showcasing its superior natural language generation performance. By fully utilizing contextual information, this hybrid model generates text that is not only logically coherent but also aligns closely with human language patterns, providing an advanced solution for text generation tasks. This research highlights the potential of integrating semantic understanding with advanced generative models, contributing new insights for NLP, and setting a foundation for broader applications of large-scale generative architectures in areas such as automated writing, question-answer systems, and adaptive conversational agents.

Title: Microsegmented Cloud Network Architecture Using Open-Source Tools for a Zero Trust Foundation

• Authors: Sunil Arora, John Hastings
• Subjects: cs.CR, cs.DC, cs.NI, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.12162
• Pdf URL: https://arxiv.org/pdf/2411.12162
• Copy Paste: [[2411.12162]] Microsegmented Cloud Network Architecture Using Open-Source Tools for a Zero Trust Foundation(https://arxiv.org/abs/2411.12162)
• Keywords: secure, security, segmentation
• Abstract: This paper presents a multi-cloud networking architecture built on zero trust principles and micro-segmentation to provide secure connectivity with authentication, authorization, and encryption in transit. The proposed design includes the multi-cloud network to support a wide range of applications and workload use cases, compute resources including containers, virtual machines, and cloud-native services, including IaaS (Infrastructure as a Service (IaaS), PaaS (Platform as a service). Furthermore, open-source tools provide flexibility, agility, and independence from locking to one vendor technology. The paper provides a secure architecture with micro-segmentation and follows zero trust principles to solve multi-fold security and operational challenges.

Title: UrbanDiT: A Foundation Model for Open-World Urban Spatio-Temporal Learning

• Authors: Yuan Yuan, Chonghua Han, Jingtao Ding, Depeng Jin, Yong Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12164
• Pdf URL: https://arxiv.org/pdf/2411.12164
• Copy Paste: [[2411.12164]] UrbanDiT: A Foundation Model for Open-World Urban Spatio-Temporal Learning(https://arxiv.org/abs/2411.12164)
• Keywords: diffusion, transformer
• Abstract: The urban environment is characterized by complex spatio-temporal dynamics arising from diverse human activities and interactions. Effectively modeling these dynamics is essential for understanding and optimizing urban systems In this work, we introduce UrbanDiT, a foundation model for open-world urban spatio-temporal learning that successfully scale up diffusion transformers in this field. UrbanDiT pioneers a unified model that integrates diverse spatio-temporal data sources and types while learning universal spatio-temporal patterns across different cities and scenarios. This allows the model to unify both multi-data and multi-task learning, and effectively support a wide range of spatio-temporal applications. Its key innovation lies in the elaborated prompt learning framework, which adaptively generates both data-driven and task-specific prompts, guiding the model to deliver superior performance across various urban applications. UrbanDiT offers three primary advantages: 1) It unifies diverse data types, such as grid-based and graph-based data, into a sequential format, allowing to capture spatio-temporal dynamics across diverse scenarios of different cities; 2) With masking strategies and task-specific prompts, it supports a wide range of tasks, including bi-directional spatio-temporal prediction, temporal interpolation, spatial extrapolation, and spatio-temporal imputation; and 3) It generalizes effectively to open-world scenarios, with its powerful zero-shot capabilities outperforming nearly all baselines with training data. These features allow UrbanDiT to achieves state-of-the-art performance in different domains such as transportation traffic, crowd flows, taxi demand, bike usage, and cellular traffic, across multiple cities and tasks. UrbanDiT sets up a new benchmark for foundation models in the urban spatio-temporal domain.

Title: Sketch-guided Cage-based 3D Gaussian Splatting Deformation

• Authors: Tianhao Xie, Noam Aigerman, Eugene Belilovsky, Tiberiu Popa
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.12168
• Pdf URL: https://arxiv.org/pdf/2411.12168
• Copy Paste: [[2411.12168]] Sketch-guided Cage-based 3D Gaussian Splatting Deformation(https://arxiv.org/abs/2411.12168)
• Keywords: diffusion
• Abstract: 3D Gaussian Splatting (GS) is one of the most promising novel 3D representations that has received great interest in computer graphics and computer vision. While various systems have introduced editing capabilities for 3D GS, such as those guided by text prompts, fine-grained control over deformation remains an open challenge. In this work, we present a novel sketch-guided 3D GS deformation system that allows users to intuitively modify the geometry of a 3D GS model by drawing a silhouette sketch from a single viewpoint. Our approach introduces a new deformation method that combines cage-based deformations with a variant of Neural Jacobian Fields, enabling precise, fine-grained control. Additionally, it leverages large-scale 2D diffusion priors and ControlNet to ensure the generated deformations are semantically plausible. Through a series of experiments, we demonstrate the effectiveness of our method and showcase its ability to animate static 3D GS models as one of its key applications.

Title: SkillTree: Explainable Skill-Based Deep Reinforcement Learning for Long-Horizon Control Tasks

• Authors: Yongyan Wen, Siyuan Li, Rongchang Zuo, Lei Yuan, Hangyu Mao, Peng Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12173
• Pdf URL: https://arxiv.org/pdf/2411.12173
• Copy Paste: [[2411.12173]] SkillTree: Explainable Skill-Based Deep Reinforcement Learning for Long-Horizon Control Tasks(https://arxiv.org/abs/2411.12173)
• Keywords: explainability
• Abstract: Deep reinforcement learning (DRL) has achieved remarkable success in various research domains. However, its reliance on neural networks results in a lack of transparency, which limits its practical applications. To achieve explainability, decision trees have emerged as a popular and promising alternative to neural networks. Nonetheless, due to their limited expressiveness, traditional decision trees struggle with high-dimensional long-horizon continuous control tasks. In this paper, we proposes SkillTree, a novel framework that reduces complex continuous action spaces into discrete skill spaces. Our hierarchical approach integrates a differentiable decision tree within the high-level policy to generate skill embeddings, which subsequently guide the low-level policy in executing skills. By making skill decisions explainable, we achieve skill-level explainability, enhancing the understanding of the decision-making process in complex tasks. Experimental results demonstrate that our method achieves performance comparable to skill-based neural networks in complex robotic arm control domains. Furthermore, SkillTree offers explanations at the skill level, thereby increasing the transparency of the decision-making process.

Title: Robust 3D Semantic Occupancy Prediction with Calibration-free Spatial Transformation

• Authors: Zhuangwei Zhuang, Ziyin Wang, Sitao Chen, Lizhao Liu, Hui Luo, Mingkui Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12177
• Pdf URL: https://arxiv.org/pdf/2411.12177
• Copy Paste: [[2411.12177]] Robust 3D Semantic Occupancy Prediction with Calibration-free Spatial Transformation(https://arxiv.org/abs/2411.12177)
• Keywords: robust
• Abstract: 3D semantic occupancy prediction, which seeks to provide accurate and comprehensive representations of environment scenes, is important to autonomous driving systems. For autonomous cars equipped with multi-camera and LiDAR, it is critical to aggregate multi-sensor information into a unified 3D space for accurate and robust predictions. Recent methods are mainly built on the 2D-to-3D transformation that relies on sensor calibration to project the 2D image information into the 3D space. These methods, however, suffer from two major limitations: First, they rely on accurate sensor calibration and are sensitive to the calibration noise, which limits their application in real complex environments. Second, the spatial transformation layers are computationally expensive and limit their running on an autonomous vehicle. In this work, we attempt to exploit a Robust and Efficient 3D semantic Occupancy (REO) prediction scheme. To this end, we propose a calibration-free spatial transformation based on vanilla attention to implicitly model the spatial correspondence. In this way, we robustly project the 2D features to a predefined BEV plane without using sensor calibration as input. Then, we introduce 2D and 3D auxiliary training tasks to enhance the discrimination power of 2D backbones on spatial, semantic, and texture features. Last, we propose a query-based prediction scheme to efficiently generate large-scale fine-grained occupancy predictions. By fusing point clouds that provide complementary spatial information, our REO surpasses the existing methods by a large margin on three benchmarks, including OpenOccupancy, Occ3D-nuScenes, and SemanticKITTI Scene Completion. For instance, our REO achieves 19.8$\times$ speedup compared to Co-Occ, with 1.1 improvements in geometry IoU on OpenOccupancy. Our code will be available at this https URL.

Title: Diffusion-Inspired Cold Start with Sufficient Prior in Computerized Adaptive Testing

• Authors: Haiping Ma, Aoqing Xia, Changqian Wang, Hai Wang, Xingyi Zhang
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.12182
• Pdf URL: https://arxiv.org/pdf/2411.12182
• Copy Paste: [[2411.12182]] Diffusion-Inspired Cold Start with Sufficient Prior in Computerized Adaptive Testing(https://arxiv.org/abs/2411.12182)
• Keywords: diffusion
• Abstract: Computerized Adaptive Testing (CAT) aims to select the most appropriate questions based on the examinee's ability and is widely used in online education. However, existing CAT systems often lack initial understanding of the examinee's ability, requiring random probing questions. This can lead to poorly matched questions, extending the test duration and negatively impacting the examinee's mindset, a phenomenon referred to as the Cold Start with Insufficient Prior (CSIP) task. This issue occurs because CAT systems do not effectively utilize the abundant prior information about the examinee available from other courses on online platforms. These response records, due to the commonality of cognitive states across different knowledge domains, can provide valuable prior information for the target domain. However, no prior work has explored solutions for the CSIP task. In response to this gap, we propose Diffusion Cognitive States TransfeR Framework (DCSR), a novel domain transfer framework based on Diffusion Models (DMs) to address the CSIP task. Specifically, we construct a cognitive state transition bridge between domains, guided by the common cognitive states of examinees, encouraging the model to reconstruct the initial ability state in the target domain. To enrich the expressive power of the generated data, we analyze the causal relationships in the generation process from a causal perspective. Redundant and extraneous cognitive states can lead to limited transfer and negative transfer effects. Our DCSR can seamlessly apply the generated initial ability states in the target domain to existing question selection algorithms, thus improving the cold start performance of the CAT system. Extensive experiments conducted on five real-world datasets demonstrate that DCSR significantly outperforms existing baseline methods in addressing the CSIP task.

Title: Constant Rate Schedule: Constant-Rate Distributional Change for Efficient Training and Sampling in Diffusion Models

• Authors: Shuntaro Okada, Kenji Doi, Ryota Yoshihashi, Hirokatsu Kataoka, Tomohiro Tanaka
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12188
• Pdf URL: https://arxiv.org/pdf/2411.12188
• Copy Paste: [[2411.12188]] Constant Rate Schedule: Constant-Rate Distributional Change for Efficient Training and Sampling in Diffusion Models(https://arxiv.org/abs/2411.12188)
• Keywords: diffusion
• Abstract: We propose a noise schedule that ensures a constant rate of change in the probability distribution of diffused data throughout the diffusion process. To obtain this noise schedule, we measure the rate of change in the probability distribution of the forward process and use it to determine the noise schedule before training diffusion models. The functional form of the noise schedule is automatically determined and tailored to each dataset and type of diffusion model. We evaluate the effectiveness of our noise schedule on unconditional and class-conditional image generation tasks using the LSUN (bedroom/church/cat/horse), ImageNet, and FFHQ datasets. Through extensive experiments, we confirmed that our noise schedule broadly improves the performance of the diffusion models regardless of the dataset, sampler, number of function evaluations, or type of diffusion model.

Title: A Survey of Medical Vision-and-Language Applications and Their Techniques

• Authors: Qi Chen, Ruoshan Zhao, Sinuo Wang, Vu Minh Hieu Phan, Anton van den Hengel, Johan Verjans, Zhibin Liao, Minh-Son To, Yong Xia, Jian Chen, Yutong Xie, Qi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12195
• Pdf URL: https://arxiv.org/pdf/2411.12195
• Copy Paste: [[2411.12195]] A Survey of Medical Vision-and-Language Applications and Their Techniques(https://arxiv.org/abs/2411.12195)
• Keywords: segmentation
• Abstract: Medical vision-and-language models (MVLMs) have attracted substantial interest due to their capability to offer a natural language interface for interpreting complex medical data. Their applications are versatile and have the potential to improve diagnostic accuracy and decision-making for individual patients while also contributing to enhanced public health monitoring, disease surveillance, and policy-making through more efficient analysis of large data sets. MVLMS integrate natural language processing with medical images to enable a more comprehensive and contextual understanding of medical images alongside their corresponding textual information. Unlike general vision-and-language models trained on diverse, non-specialized datasets, MVLMs are purpose-built for the medical domain, automatically extracting and interpreting critical information from medical images and textual reports to support clinical decision-making. Popular clinical applications of MVLMs include automated medical report generation, medical visual question answering, medical multimodal segmentation, diagnosis and prognosis and medical image-text retrieval. Here, we provide a comprehensive overview of MVLMs and the various medical tasks to which they have been applied. We conduct a detailed analysis of various vision-and-language model architectures, focusing on their distinct strategies for cross-modal integration/exploitation of medical visual and textual features. We also examine the datasets used for these tasks and compare the performance of different models based on standardized evaluation metrics. Furthermore, we highlight potential challenges and summarize future research trends and directions. The full collection of papers and codes is available at: this https URL.

Title: CCIS-Diff: A Generative Model with Stable Diffusion Prior for Controlled Colonoscopy Image Synthesis

• Authors: Yifan Xie, Jingge Wang, Tao Feng, Fei Ma, Yang Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12198
• Pdf URL: https://arxiv.org/pdf/2411.12198
• Copy Paste: [[2411.12198]] CCIS-Diff: A Generative Model with Stable Diffusion Prior for Controlled Colonoscopy Image Synthesis(https://arxiv.org/abs/2411.12198)
• Keywords: robust, diffusion, generative, segmentation
• Abstract: Colonoscopy is crucial for identifying adenomatous polyps and preventing colorectal cancer. However, developing robust models for polyp detection is challenging by the limited size and accessibility of existing colonoscopy datasets. While previous efforts have attempted to synthesize colonoscopy images, current methods suffer from instability and insufficient data diversity. Moreover, these approaches lack precise control over the generation process, resulting in images that fail to meet clinical quality standards. To address these challenges, we propose CCIS-DIFF, a Controlled generative model for high-quality Colonoscopy Image Synthesis based on a Diffusion architecture. Our method offers precise control over both the spatial attributes (polyp location and shape) and clinical characteristics of polyps that align with clinical descriptions. Specifically, we introduce a blur mask weighting strategy to seamlessly blend synthesized polyps with the colonic mucosa, and a text-aware attention mechanism to guide the generated images to reflect clinical characteristics. Notably, to achieve this, we construct a new multi-modal colonoscopy dataset that integrates images, mask annotations, and corresponding clinical text descriptions. Experimental results demonstrate that our method generates high-quality, diverse colonoscopy images with fine control over both spatial constraints and clinical consistency, offering valuable support for downstream segmentation and diagnostic tasks.

Title: RoSIS: Robust Framework for Text-Promptable Surgical Instrument Segmentation Using Vision-Language Fusion

• Authors: Tae-Min Choi, Juyoun Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12199
• Pdf URL: https://arxiv.org/pdf/2411.12199
• Copy Paste: [[2411.12199]] RoSIS: Robust Framework for Text-Promptable Surgical Instrument Segmentation Using Vision-Language Fusion(https://arxiv.org/abs/2411.12199)
• Keywords: robust, fair, segmentation
• Abstract: Surgical instrument segmentation (SIS) is an essential task in computer-assisted surgeries, with deep learning-based research improving accuracy in complex environments. Recently, text-promptable segmentation methods have been introduced to generate masks based on text prompts describing target objects. However, these methods assume that the object described by a given text prompt exists in the scene. This results in mask generation whenever a related text prompt is provided, even if the object is absent from the image. Existing methods handle this by using prompts only for objects known to be present in the image, which introduces inaccessible information in a vision-based method setting and results in unfair comparisons. For fair comparison, we redefine existing text-promptable SIS settings to robust conditions, called Robust text-promptable SIS (R-SIS), designed to forward prompts of all classes and determine the existence of an object from a given text prompt for the fair comparison. Furthermore, we propose a novel framework, Robust Surgical Instrument Segmentation (RoSIS), which combines visual and language features for promptable segmentation in the R-SIS setting. RoSIS employs an encoder-decoder architecture with a Multi-Modal Fusion Block (MMFB) and a Selective Gate Block (SGB) to achieve balanced integration of vision and language features. Additionally, we introduce an iterative inference strategy that refines segmentation masks in two steps: an initial pass using name-based prompts, followed by a refinement step using location prompts. Experiments on various datasets and settings demonstrate that RoSIS outperforms existing vision-based and promptable methods under robust conditions.

Title: Invariant Shape Representation Learning For Image Classification

• Authors: Tonmoy Hossain, Jing Ma, Jundong Li, Miaomiao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12201
• Pdf URL: https://arxiv.org/pdf/2411.12201
• Copy Paste: [[2411.12201]] Invariant Shape Representation Learning For Image Classification(https://arxiv.org/abs/2411.12201)
• Keywords: robust, generative
• Abstract: Geometric shape features have been widely used as strong predictors for image classification. Nevertheless, most existing classifiers such as deep neural networks (DNNs) directly leverage the statistical correlations between these shape features and target variables. However, these correlations can often be spurious and unstable across different environments (e.g., in different age groups, certain types of brain changes have unstable relations with neurodegenerative disease); hence leading to biased or inaccurate predictions. In this paper, we introduce a novel framework that for the first time develops invariant shape representation learning (ISRL) to further strengthen the robustness of image classifiers. In contrast to existing approaches that mainly derive features in the image space, our model ISRL is designed to jointly capture invariant features in latent shape spaces parameterized by deformable transformations. To achieve this goal, we develop a new learning paradigm based on invariant risk minimization (IRM) to learn invariant representations of image and shape features across multiple training distributions/environments. By embedding the features that are invariant with regard to target variables in different environments, our model consistently offers more accurate predictions. We validate our method by performing classification tasks on both simulated 2D images, real 3D brain and cine cardiovascular magnetic resonance images (MRIs). Our code is publicly available at this https URL.

Title: DeTrigger: A Gradient-Centric Approach to Backdoor Attack Mitigation in Federated Learning

• Authors: Kichang Lee, Yujin Shin, Jonghyuk Yun, Jun Han, JeongGil Ko
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12220
• Pdf URL: https://arxiv.org/pdf/2411.12220
• Copy Paste: [[2411.12220]] DeTrigger: A Gradient-Centric Approach to Backdoor Attack Mitigation in Federated Learning(https://arxiv.org/abs/2411.12220)
• Keywords: privacy, protect, attack, robust, federate
• Abstract: Federated Learning (FL) enables collaborative model training across distributed devices while preserving local data privacy, making it ideal for mobile and embedded systems. However, the decentralized nature of FL also opens vulnerabilities to model poisoning attacks, particularly backdoor attacks, where adversaries implant trigger patterns to manipulate model predictions. In this paper, we propose DeTrigger, a scalable and efficient backdoor-robust federated learning framework that leverages insights from adversarial attack methodologies. By employing gradient analysis with temperature scaling, DeTrigger detects and isolates backdoor triggers, allowing for precise model weight pruning of backdoor activations without sacrificing benign model knowledge. Extensive evaluations across four widely used datasets demonstrate that DeTrigger achieves up to 251x faster detection than traditional methods and mitigates backdoor attacks by up to 98.9%, with minimal impact on global model accuracy. Our findings establish DeTrigger as a robust and scalable solution to protect federated learning environments against sophisticated backdoor threats.

Title: Perception of Digital Privacy Protection: An Empirical Study using GDPR Framework

• Authors: Hamoud Alhazmi, Ahmed Imran, Mohammad Abu Alsheikh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.12223
• Pdf URL: https://arxiv.org/pdf/2411.12223
• Copy Paste: [[2411.12223]] Perception of Digital Privacy Protection: An Empirical Study using GDPR Framework(https://arxiv.org/abs/2411.12223)
• Keywords: privacy, protect
• Abstract: Perception of privacy is a contested concept, which is also evolving along with the rapid proliferation and expansion of technological advancements. Information systems (IS) applications incorporate various sensing infrastructures, high-speed networks, and computing components that enable pervasive data collection about people. Any digital privacy breach within such systems can result in harmful and far-reaching impacts on individuals and societies. Accordingly, IS organisations have a legal and ethical responsibility to respect and protect individuals digital privacy rights. This study investigates people perception of digital privacy protection of government data using the General Data Protection Regulation (GDPR) framework. Findings suggest a dichotomy of perception in protecting people privacy rights. For example, people perceive the right to be informed as the most respected and protected in Information Technology (IT) systems. On the contrary, the right to object by granting and with-drawing consent is perceived as the least protected. Second, the study shows evidence of a social dilemma in people perception of digital privacy based on their context and culture.

Title: Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

• Authors: S. Tamang, D. J. Bora
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12240
• Pdf URL: https://arxiv.org/pdf/2411.12240
• Copy Paste: [[2411.12240]] Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages(https://arxiv.org/abs/2411.12240)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) based on transformer architectures have revolutionized a variety of domains, with tokenization playing a pivotal role in their pre-processing and fine-tuning stages. In multilingual models, particularly those tailored for Indic languages, effective tokenization is crucial for optimizing performance. This paper presents a comprehensive evaluation of tokenizers used by 12 LLMs across all 22 official languages of India, with a focus on comparing the efficiency of their tokenization processes. We employed the Normalized Sequence Length (NSL) as a key metric in our analysis. Our findings reveal that the SUTRA tokenizer outperforms all other models, including several Indic-specific models, excelling in 14 languages. Notable insights include the SUTRA tokenizer's superior handling of Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing Indian languages, and the limited performance of Project Indus in certain languages. This study underscores the critical importance of developing targeted tokenization strategies for multilingual and Indic-centric models, laying the groundwork for future improvements in tokenizer design to enhance linguistic coverage and model efficiency.

Title: Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism

• Authors: Yasaman Saadati, M. Hadi Amini
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.12244
• Pdf URL: https://arxiv.org/pdf/2411.12244
• Copy Paste: [[2411.12244]] Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism(https://arxiv.org/abs/2411.12244)
• Keywords: privacy, protect, federate
• Abstract: Federated Learning (FL) is a decentralized learning approach that protects sensitive information by utilizing local model parameters rather than sharing clients' raw datasets. While this privacy-preserving method is widely employed across various applications, it still requires significant development and optimization. Automated Machine Learning (Auto-ML) has been adapted for reducing the need for manual adjustments. Previous studies have explored the integration of AutoML with different FL algorithms to evaluate their effectiveness in enhancing FL settings. However, Automated FL (Auto-FL) faces additional challenges due to the involvement of a large cohort of clients and global training rounds between clients and the server, rendering the tuning process time-consuming and nearly impossible on resource-constrained edge devices (e.g., IoT devices). This paper investigates the deployment and integration of two lightweight Hyper-Parameter Optimization (HPO) tools, Raytune and Optuna, within the context of FL settings. A step-wise feedback mechanism has also been designed to accelerate the hyper-parameter tuning process and coordinate AutoML toolkits with the FL server. To this end, both local and global feedback mechanisms are integrated to limit the search space and expedite the HPO process. Further, a novel client selection technique is introduced to mitigate the straggler effect in Auto-FL. The selected hyper-parameter tuning tools are evaluated using two benchmark datasets, FEMNIST, and CIFAR10. Further, the paper discusses the essential properties of successful HPO tools, the integration mechanism with the FL pipeline, and the challenges posed by the distributed and heterogeneous nature of FL environments.

Title: Neuro-3D: Towards 3D Visual Decoding from EEG Signals

• Authors: Zhanqiang Guo, Jiamin Wu, Yonghao Song, Weijian Mai, Qihao Zheng, Wanli Ouyang, Chunfeng Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12248
• Pdf URL: https://arxiv.org/pdf/2411.12248
• Copy Paste: [[2411.12248]] Neuro-3D: Towards 3D Visual Decoding from EEG Signals(https://arxiv.org/abs/2411.12248)
• Keywords: robust, diffusion
• Abstract: Human's perception of the visual world is shaped by the stereo processing of 3D information. Understanding how the brain perceives and processes 3D visual stimuli in the real world has been a longstanding endeavor in neuroscience. Towards this goal, we introduce a new neuroscience task: decoding 3D visual perception from EEG signals, a neuroimaging technique that enables real-time monitoring of neural dynamics enriched with complex visual cues. To provide the essential benchmark, we first present EEG-3D, a pioneering dataset featuring multimodal analysis data and extensive EEG recordings from 12 subjects viewing 72 categories of 3D objects rendered in both videos and images. Furthermore, we propose Neuro-3D, a 3D visual decoding framework based on EEG signals. This framework adaptively integrates EEG features derived from static and dynamic stimuli to learn complementary and robust neural representations, which are subsequently utilized to recover both the shape and color of 3D objects through the proposed diffusion-based colored point cloud decoder. To the best of our knowledge, we are the first to explore EEG-based 3D visual decoding. Experiments indicate that Neuro-3D not only reconstructs colored 3D objects with high fidelity, but also learns effective neural representations that enable insightful brain region analysis. The dataset and associated code will be made publicly available.

Title: ADV2E: Bridging the Gap Between Analogue Circuit and Discrete Frames in the Video-to-Events Simulator

• Authors: Xiao Jiang, Fei Zhou, Jiongzhi Lin
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.12250
• Pdf URL: https://arxiv.org/pdf/2411.12250
• Copy Paste: [[2411.12250]] ADV2E: Bridging the Gap Between Analogue Circuit and Discrete Frames in the Video-to-Events Simulator(https://arxiv.org/abs/2411.12250)
• Keywords: segmentation
• Abstract: Event cameras operate fundamentally differently from traditional Active Pixel Sensor (APS) cameras, offering significant advantages. Recent research has developed simulators to convert video frames into events, addressing the shortage of real event datasets. Current simulators primarily focus on the logical behavior of event cameras. However, the fundamental analogue properties of pixel circuits are seldom considered in simulator design. The gap between analogue pixel circuit and discrete video frames causes the degeneration of synthetic events, particularly in high-contrast scenes. In this paper, we propose a novel method of generating reliable event data based on a detailed analysis of the pixel circuitry in event cameras. We incorporate the analogue properties of event camera pixel circuits into the simulator design: (1) analogue filtering of signals from light intensity to events, and (2) a cutoff frequency that is independent of video frame rate. Experimental results on two relevant tasks, including semantic segmentation and image reconstruction, validate the reliability of simulated event data, even in high-contrast scenes. This demonstrates that deep neural networks exhibit strong generalization from simulated to real event data, confirming that the synthetic events generated by the proposed method are both realistic and well-suited for effective training.

Title: A Review on Generative AI Models for Synthetic Medical Text, Time Series, and Longitudinal Data

• Authors: Mohammad Loni, Fatemeh Poursalim, Mehdi Asadi, Arash Gharehbaghi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12274
• Pdf URL: https://arxiv.org/pdf/2411.12274
• Copy Paste: [[2411.12274]] A Review on Generative AI Models for Synthetic Medical Text, Time Series, and Longitudinal Data(https://arxiv.org/abs/2411.12274)
• Keywords: privacy, generative, large language model
• Abstract: This paper presents the results of a novel scoping review on the practical models for generating three different types of synthetic health records (SHRs): medical text, time series, and longitudinal data. The innovative aspects of the review, which incorporate study objectives, data modality, and research methodology of the reviewed studies, uncover the importance and the scope of the topic for the digital medicine context. In total, 52 publications met the eligibility criteria for generating medical time series (22), longitudinal data (17), and medical text (13). Privacy preservation was found to be the main research objective of the studied papers, along with class imbalance, data scarcity, and data imputation as the other objectives. The adversarial network-based, probabilistic, and large language models exhibited superiority for generating synthetic longitudinal data, time series, and medical texts, respectively. Finding a reliable performance measure to quantify SHR re-identification risk is the major research gap of the topic.

Title: HouseLLM: LLM-Assisted Two-Phase Text-to-Floorplan Generation

• Authors: Ziyang Zong, Zhaohuan Zhan, Guang Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12279
• Pdf URL: https://arxiv.org/pdf/2411.12279
• Copy Paste: [[2411.12279]] HouseLLM: LLM-Assisted Two-Phase Text-to-Floorplan Generation(https://arxiv.org/abs/2411.12279)
• Keywords: diffusion, large language model
• Abstract: This paper proposes a two-phase text-to-floorplan generation method, which guides a Large Language Model (LLM) to generate an initial layout (Layout-LLM) and refines them into the final floorplans through conditional diffusion model. We incorporate a Chain-of-Thought approach to prompt the LLM based on user text specifications, enabling a more user-friendly and intuitive house layout design. This method allows users to describe their needs in natural language, enhancing accessibility and providing clearer geometric constraints. The final floorplans generated by Layout-LLM through conditional diffusion refinement are more accurate and better meet user requirements. Experimental results demonstrate that our approach achieves state-of-the-art performance across all metrics, validating its effectiveness in practical home design applications. We plan to release our code for public use.

Title: CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model

• Authors: Dongyoung Go, Taesun Whang, Chanhee Lee, Hwayeon Kim, Sunghoon Park, Seunghwan Ji, Dongchan Kim, Young-Bum Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12287
• Pdf URL: https://arxiv.org/pdf/2411.12287
• Copy Paste: [[2411.12287]] CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model(https://arxiv.org/abs/2411.12287)
• Keywords: robust, large language model
• Abstract: The integration of Retrieval-Augmented Generation (RAG) with Multimodal Large Language Models (MLLMs) has expanded the scope of multimodal query resolution. However, current systems struggle with intent understanding, information retrieval, and safety filtering, limiting their effectiveness. This paper introduces Contextual Understanding and Enhanced Search with MLLM (CUE-M), a novel multimodal search pipeline that addresses these challenges through a multi-stage framework comprising image context enrichment, intent refinement, contextual query generation, external API integration, and relevance-based filtering. CUE-M incorporates a robust safety framework combining image-based, text-based, and multimodal classifiers, dynamically adapting to instance- and category-specific risks. Evaluations on a multimodal Q&A dataset and a public safety benchmark demonstrate that CUE-M outperforms baselines in accuracy, knowledge integration, and safety, advancing the capabilities of multimodal retrieval systems.

Title: SSEditor: Controllable Mask-to-Scene Generation with Diffusion Model

• Authors: Haowen Zheng, Yanyan Liang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12290
• Pdf URL: https://arxiv.org/pdf/2411.12290
• Copy Paste: [[2411.12290]] SSEditor: Controllable Mask-to-Scene Generation with Diffusion Model(https://arxiv.org/abs/2411.12290)
• Keywords: diffusion
• Abstract: Recent advancements in 3D diffusion-based semantic scene generation have gained attention. However, existing methods rely on unconditional generation and require multiple resampling steps when editing scenes, which significantly limits their controllability and flexibility. To this end, we propose SSEditor, a controllable Semantic Scene Editor that can generate specified target categories without multiple-step resampling. SSEditor employs a two-stage diffusion-based framework: (1) a 3D scene autoencoder is trained to obtain latent triplane features, and (2) a mask-conditional diffusion model is trained for customizable 3D semantic scene generation. In the second stage, we introduce a geometric-semantic fusion module that enhance the model's ability to learn geometric and semantic information. This ensures that objects are generated with correct positions, sizes, and categories. Extensive experiments on SemanticKITTI and CarlaSC demonstrate that SSEditor outperforms previous approaches in terms of controllability and flexibility in target generation, as well as the quality of semantic scene generation and reconstruction. More importantly, experiments on the unseen Occ-3D Waymo dataset show that SSEditor is capable of generating novel urban scenes, enabling the rapid construction of 3D scenes.

Title: Generative Timelines for Instructed Visual Assembly

• Authors: Alejandro Pardo, Jui-Hsien Wang, Bernard Ghanem, Josef Sivic, Bryan Russell, Fabian Caba Heilbron
• Subjects: cs.CV, cs.HC, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.12293
• Pdf URL: https://arxiv.org/pdf/2411.12293
• Copy Paste: [[2411.12293]] Generative Timelines for Instructed Visual Assembly(https://arxiv.org/abs/2411.12293)
• Keywords: generative
• Abstract: The objective of this work is to manipulate visual timelines (e.g. a video) through natural language instructions, making complex timeline editing tasks accessible to non-expert or potentially even disabled users. We call this task Instructed visual assembly. This task is challenging as it requires (i) identifying relevant visual content in the input timeline as well as retrieving relevant visual content in a given input (video) collection, (ii) understanding the input natural language instruction, and (iii) performing the desired edits of the input visual timeline to produce an output timeline. To address these challenges, we propose the Timeline Assembler, a generative model trained to perform instructed visual assembly tasks. The contributions of this work are three-fold. First, we develop a large multimodal language model, which is designed to process visual content, compactly represent timelines and accurately interpret timeline editing instructions. Second, we introduce a novel method for automatically generating datasets for visual assembly tasks, enabling efficient training of our model without the need for human-labeled data. Third, we validate our approach by creating two novel datasets for image and video assembly, demonstrating that the Timeline Assembler substantially outperforms established baseline models, including the recent GPT-4o, in accurately executing complex assembly instructions across various real-world inspired scenarios.

Title: Diffusion Product Quantization

• Authors: Jie Shao, Hanxiao Zhang, Jianxin Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12306
• Pdf URL: https://arxiv.org/pdf/2411.12306
• Copy Paste: [[2411.12306]] Diffusion Product Quantization(https://arxiv.org/abs/2411.12306)
• Keywords: diffusion, generative
• Abstract: In this work, we explore the quantization of diffusion models in extreme compression regimes to reduce model size while maintaining performance. We begin by investigating classical vector quantization but find that diffusion models are particularly susceptible to quantization error, with the codebook size limiting generation quality. To address this, we introduce product quantization, which offers improved reconstruction precision and larger capacity -- crucial for preserving the generative capabilities of diffusion models. Furthermore, we propose a method to compress the codebook by evaluating the importance of each vector and removing redundancy, ensuring the model size remaining within the desired range. We also introduce an end-to-end calibration approach that adjusts assignments during the forward pass and optimizes the codebook using the DDPM loss. By compressing the model to as low as 1 bit (resulting in over 24 times reduction in model size), we achieve a balance between compression and quality. We apply our compression method to the DiT model on ImageNet and consistently outperform other quantization approaches, demonstrating competitive generative performance.

Title: Balancing Accuracy and Efficiency in Multi-Turn Intent Classification for LLM-Powered Dialog Systems in Production

• Authors: Junhua Liu, Yong Keat Tan, Bin Fu, Kwan Hui Lim
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.12307
• Pdf URL: https://arxiv.org/pdf/2411.12307
• Copy Paste: [[2411.12307]] Balancing Accuracy and Efficiency in Multi-Turn Intent Classification for LLM-Powered Dialog Systems in Production(https://arxiv.org/abs/2411.12307)
• Keywords: large language model
• Abstract: Accurate multi-turn intent classification is essential for advancing conversational AI systems. However, challenges such as the scarcity of comprehensive datasets and the complexity of contextual dependencies across dialogue turns hinder progress. This paper presents two novel approaches leveraging Large Language Models (LLMs) to enhance scalability and reduce latency in production dialogue systems. First, we introduce Symbol Tuning, which simplifies intent labels to reduce task complexity and improve performance in multi-turn dialogues. Second, we propose C-LARA (Consistency-aware, Linguistics Adaptive Retrieval Augmentation), a framework that employs LLMs for data augmentation and pseudo-labeling to generate synthetic multi-turn dialogues. These enriched datasets are used to fine-tune a small, efficient model suitable for deployment. Experiments conducted on multilingual dialogue datasets demonstrate significant improvements in classification accuracy and resource efficiency. Our methods enhance multi-turn intent classification accuracy by 5.09%, reduce annotation costs by 40%, and enable scalable deployment in low-resource multilingual industrial systems, highlighting their practicality and impact.

Title: CLIP Unreasonable Potential in Single-Shot Face Recognition

• Authors: Nhan T. Luu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12319
• Pdf URL: https://arxiv.org/pdf/2411.12319
• Copy Paste: [[2411.12319]] CLIP Unreasonable Potential in Single-Shot Face Recognition(https://arxiv.org/abs/2411.12319)
• Keywords: security, extraction
• Abstract: Face recognition is a core task in computer vision designed to identify and authenticate individuals by analyzing facial patterns and features. This field intersects with artificial intelligence image processing and machine learning with applications in security authentication and personalization. Traditional approaches in facial recognition focus on capturing facial features like the eyes, nose and mouth and matching these against a database to verify identities However challenges such as high false positive rates have persisted often due to the similarity among individuals facial features. Recently Contrastive Language Image Pretraining (CLIP) a model developed by OpenAI has shown promising advancements by linking natural language processing with vision tasks allowing it to generalize across modalities. Using CLIP's vision language correspondence and single-shot finetuning the model can achieve lower false positive rates upon deployment without the need of mass facial features extraction. This integration demonstrating CLIP's potential to address persistent issues in face recognition model performance without complicating our training paradigm.

Title: Enhancing Blind Source Separation with Dissociative Principal Component Analysis

• Authors: Muhammad Usman Khalid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12321
• Pdf URL: https://arxiv.org/pdf/2411.12321
• Copy Paste: [[2411.12321]] Enhancing Blind Source Separation with Dissociative Principal Component Analysis(https://arxiv.org/abs/2411.12321)
• Keywords: extraction, interpretability
• Abstract: Sparse principal component analysis (sPCA) enhances the interpretability of principal components (PCs) by imposing sparsity constraints on loading vectors (LVs). However, when used as a precursor to independent component analysis (ICA) for blind source separation (BSS), sPCA may underperform due to its focus on simplicity, potentially disregarding some statistical information essential for effective ICA. To overcome this limitation, a sophisticated approach is proposed that preserves the interpretability advantages of sPCA while significantly enhancing its source extraction capabilities. This consists of two tailored algorithms, dissociative PCA (DPCA1 and DPCA2), which employ adaptive and firm thresholding alongside gradient and coordinate descent approaches to optimize the proposed model dynamically. These algorithms integrate left and right singular vectors from singular value decomposition (SVD) through dissociation matrices (DMs) that replace traditional singular values, thus capturing latent interdependencies effectively to model complex source relationships. This leads to refined PCs and LVs that more accurately represent the underlying data structure. The proposed approach avoids focusing on individual eigenvectors, instead, it collaboratively combines multiple eigenvectors to disentangle interdependencies within each SVD variate. The superior performance of the proposed DPCA algorithms is demonstrated across four varied imaging applications including functional magnetic resonance imaging (fMRI) source retrieval, foreground-background separation, image reconstruction, and image inpainting. They outperformed traditional methods such as PCA+ICA, PPCA+ICA, SPCA+ICA, PMD, and GPower.

Title: Learning from Label Proportions and Covariate-shifted Instances

• Authors: Sagalpreet Singh, Navodita Sharma, Shreyas Havaldar, Rishi Saket, Aravindan Raghuveer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12334
• Pdf URL: https://arxiv.org/pdf/2411.12334
• Copy Paste: [[2411.12334]] Learning from Label Proportions and Covariate-shifted Instances(https://arxiv.org/abs/2411.12334)
• Keywords: privacy
• Abstract: In many applications, especially due to lack of supervision or privacy concerns, the training data is grouped into bags of instances (feature-vectors) and for each bag we have only an aggregate label derived from the instance-labels in the bag. In learning from label proportions (LLP) the aggregate label is the average of the instance-labels in a bag, and a significant body of work has focused on training models in the LLP setting to predict instance-labels. In practice however, the training data may have fully supervised albeit covariate-shifted source data, along with the usual target data with bag-labels, and we wish to train a good instance-level predictor on the target domain. We call this the covariate-shifted hybrid LLP problem. Fully supervised covariate shifted data often has useful training signals and the goal is to leverage them for better predictive performance in the hybrid LLP setting. To achieve this, we develop methods for hybrid LLP which naturally incorporate the target bag-labels along with the source instance-labels, in the domain adaptation framework. Apart from proving theoretical guarantees bounding the target generalization error, we also conduct experiments on several publicly available datasets showing that our methods outperform LLP and domain adaptation baselines as well techniques from previous related work.

Title: DiM: $f$-Divergence Minimization Guided Sharpness-Aware Optimization for Semi-supervised Medical Image Segmentation

• Authors: Bingli Wang, Houcheng Su, Nan Yin, Mengzhu Wang, Li Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12350
• Pdf URL: https://arxiv.org/pdf/2411.12350
• Copy Paste: [[2411.12350]] DiM: $f$-Divergence Minimization Guided Sharpness-Aware Optimization for Semi-supervised Medical Image Segmentation(https://arxiv.org/abs/2411.12350)
• Keywords: segmentation
• Abstract: As a technique to alleviate the pressure of data annotation, semi-supervised learning (SSL) has attracted widespread attention. In the specific domain of medical image segmentation, semi-supervised methods (SSMIS) have become a research hotspot due to their ability to reduce the need for large amounts of precisely annotated data. SSMIS focuses on enhancing the model's generalization performance by leveraging a small number of labeled samples and a large number of unlabeled samples. The latest sharpness-aware optimization (SAM) technique, which optimizes the model by reducing the sharpness of the loss function, has shown significant success in SSMIS. However, SAM and its variants may not fully account for the distribution differences between different datasets. To address this issue, we propose a sharpness-aware optimization method based on $f$-divergence minimization (DiM) for semi-supervised medical image segmentation. This method enhances the model's stability by fine-tuning the sensitivity of model parameters and improves the model's adaptability to different datasets through the introduction of $f$-divergence. By reducing $f$-divergence, the DiM method not only improves the performance balance between the source and target datasets but also prevents performance degradation due to overfitting on the source dataset.

Title: Ultra-Sparse Memory Network

• Authors: Zihao Huang, Qiyang Min, Hongzhi Huang, Defa Zhu, Yutao Zeng, Ran Guo, Xun Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12364
• Pdf URL: https://arxiv.org/pdf/2411.12364
• Copy Paste: [[2411.12364]] Ultra-Sparse Memory Network(https://arxiv.org/abs/2411.12364)
• Keywords: transformer
• Abstract: It is widely acknowledged that the performance of Transformer models is exponentially related to their number of parameters and computational complexity. While approaches like Mixture of Experts (MoE) decouple parameter count from computational complexity, they still face challenges in inference due to high memory access costs. This work introduces UltraMem, incorporating large-scale, ultra-sparse memory layer to address these limitations. Our approach significantly reduces inference latency while maintaining model performance. We also investigate the scaling laws of this new architecture, demonstrating that it not only exhibits favorable scaling properties but outperforms traditional models. In our experiments, we train networks with up to 20 million memory slots. The results show that our method achieves state-of-the-art inference speed and model performance within a given computational budget.

Title: RedPajama: an Open Dataset for Training Large Language Models

• Authors: Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, Ce Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12372
• Pdf URL: https://arxiv.org/pdf/2411.12372
• Copy Paste: [[2411.12372]] RedPajama: an Open Dataset for Training Large Language Models(https://arxiv.org/abs/2411.12372)
• Keywords: large language model
• Abstract: Large language models are increasingly becoming a cornerstone technology in artificial intelligence, the sciences, and society as a whole, yet the optimal strategies for dataset composition and filtering remain largely elusive. Many of the top-performing models lack transparency in their dataset curation and model development processes, posing an obstacle to the development of fully open language models. In this paper, we identify three core data-related challenges that must be addressed to advance open-source language models. These include (1) transparency in model development, including the data curation process, (2) access to large quantities of high-quality data, and (3) availability of artifacts and metadata for dataset curation and analysis. To address these challenges, we release RedPajama-V1, an open reproduction of the LLaMA training dataset. In addition, we release RedPajama-V2, a massive web-only dataset consisting of raw, unfiltered text data together with quality signals and metadata. Together, the RedPajama datasets comprise over 100 trillion tokens spanning multiple domains and with their quality signals facilitate the filtering of data, aiming to inspire the development of numerous new datasets. To date, these datasets have already been used in the training of strong language models used in production, such as Snowflake Arctic, Salesforce's XGen and AI2's OLMo. To provide insight into the quality of RedPajama, we present a series of analyses and ablation studies with decoder-only language models with up to 1.6B parameters. Our findings demonstrate how quality signals for web data can be effectively leveraged to curate high-quality subsets of the dataset, underscoring the potential of RedPajama to advance the development of transparent and high-performing language models at scale.

Title: Non-IID data in Federated Learning: A Systematic Review with Taxonomy, Metrics, Methods, Frameworks and Future Directions

• Authors: Daniel M. Jimenez G., David Solans, Mikko Heikkila, Andrea Vitaletti, Nicolas Kourtellis, Aris Anagnostopoulos, Ioannis Chatzigiannakis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12377
• Pdf URL: https://arxiv.org/pdf/2411.12377
• Copy Paste: [[2411.12377]] Non-IID data in Federated Learning: A Systematic Review with Taxonomy, Metrics, Methods, Frameworks and Future Directions(https://arxiv.org/abs/2411.12377)
• Keywords: privacy, federate
• Abstract: Recent advances in machine learning have highlighted Federated Learning (FL) as a promising approach that enables multiple distributed users (so-called clients) to collectively train ML models without sharing their private data. While this privacy-preserving method shows potential, it struggles when data across clients is not independent and identically distributed (non-IID) data. The latter remains an unsolved challenge that can result in poorer model performance and slower training times. Despite the significance of non-IID data in FL, there is a lack of consensus among researchers about its classification and quantification. This systematic review aims to fill that gap by providing a detailed taxonomy for non-IID data, partition protocols, and metrics to quantify data heterogeneity. Additionally, we describe popular solutions to address non-IID data and standardized frameworks employed in FL with heterogeneous data. Based on our state-of-the-art review, we present key lessons learned and suggest promising future research directions.

Title: Combinational Backdoor Attack against Customized Text-to-Image Models

• Authors: Wenbo Jiang, Jiaming He, Hongwei Li, Guowen Xu, Rui Zhang, Hanxiao Chen, Meng Hao, Haomiao Yang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.12389
• Pdf URL: https://arxiv.org/pdf/2411.12389
• Copy Paste: [[2411.12389]] Combinational Backdoor Attack against Customized Text-to-Image Models(https://arxiv.org/abs/2411.12389)
• Keywords: attack, steal, diffusion
• Abstract: Recently, Text-to-Image (T2I) synthesis technology has made tremendous strides. Numerous representative T2I models have emerged and achieved promising application outcomes, such as DALL-E, Stable Diffusion, Imagen, etc. In practice, it has become increasingly popular for model developers to selectively adopt various pre-trained text encoders and conditional diffusion models from third-party platforms, integrating them to build customized (personalized) T2I models. However, such an adoption approach is vulnerable to backdoor attacks. In this work, we propose a Combinational Backdoor Attack against Customized T2I models (CBACT2I) targeting this application scenario. Different from previous backdoor attacks against T2I models, CBACT2I embeds the backdoor into the text encoder and the conditional diffusion model separately. The customized T2I model exhibits backdoor behaviors only when the backdoor text encoder is used in combination with the backdoor conditional diffusion model. These properties make CBACT2I more stealthy and flexible than prior backdoor attacks against T2I models. Extensive experiments demonstrate the effectiveness of CBACT2I with different backdoor triggers and different backdoor targets on the open-sourced Stable Diffusion model. This work reveals the backdoor vulnerabilities of customized T2I models and urges countermeasures to mitigate backdoor threats in this scenario.

Title: Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering

• Authors: Aryan Keluskar, Amrita Bhattacharjee, Huan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12395
• Pdf URL: https://arxiv.org/pdf/2411.12395
• Copy Paste: [[2411.12395]] Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering(https://arxiv.org/abs/2411.12395)
• Keywords: extraction, large language model
• Abstract: Ambiguity in natural language poses significant challenges to Large Language Models (LLMs) used for open-domain question answering. LLMs often struggle with the inherent uncertainties of human communication, leading to misinterpretations, miscommunications, hallucinations, and biased responses. This significantly weakens their ability to be used for tasks like fact-checking, question answering, feature extraction, and sentiment analysis. Using open-domain question answering as a test case, we compare off-the-shelf and few-shot LLM performance, focusing on measuring the impact of explicit disambiguation strategies. We demonstrate how simple, training-free, token-level disambiguation methods may be effectively used to improve LLM performance for ambiguous question answering tasks. We empirically show our findings and discuss best practices and broader impacts regarding ambiguity in LLMs.

Title: Evaluating the Prompt Steerability of Large Language Models

• Authors: Erik Miehling, Michael Desmond, Karthikeyan Natesan Ramamurthy, Elizabeth M. Daly, Pierre Dognin, Jesus Rios, Djallel Bouneffouf, Miao Liu
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.12405
• Pdf URL: https://arxiv.org/pdf/2411.12405
• Copy Paste: [[2411.12405]] Evaluating the Prompt Steerability of Large Language Models(https://arxiv.org/abs/2411.12405)
• Keywords: large language model
• Abstract: Building pluralistic AI requires designing models that are able to be shaped to represent a wide range of value systems and cultures. Achieving this requires first being able to evaluate the degree to which a given model is capable of reflecting various personas. To this end, we propose a benchmark for evaluating the steerability of model personas as a function of prompting. Our design is based on a formal definition of prompt steerability, which analyzes the degree to which a model's joint behavioral distribution can be shifted from its baseline behavior. By defining steerability indices and inspecting how these indices change as a function of steering effort, we can estimate the steerability of a model across various persona dimensions and directions. Our benchmark reveals that the steerability of many current models is limited -- due to both a skew in their baseline behavior and an asymmetry in their steerability across many persona dimensions. We release an implementation of our benchmark at this https URL.

Title: Motif Channel Opened in a White-Box: Stereo Matching via Motif Correlation Graph

• Authors: Ziyang Chen, Yongjun Zhang, Wenting Li, Bingshu Wang, Yong Zhao, C. L. Philip Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12426
• Pdf URL: https://arxiv.org/pdf/2411.12426
• Copy Paste: [[2411.12426]] Motif Channel Opened in a White-Box: Stereo Matching via Motif Correlation Graph(https://arxiv.org/abs/2411.12426)
• Keywords: interpretability
• Abstract: Real-world applications of stereo matching, such as autonomous driving, place stringent demands on both safety and accuracy. However, learning-based stereo matching methods inherently suffer from the loss of geometric structures in certain feature channels, creating a bottleneck in achieving precise detail matching. Additionally, these methods lack interpretability due to the black-box nature of deep learning. In this paper, we propose MoCha-V2, a novel learning-based paradigm for stereo matching. MoCha-V2 introduces the Motif Correlation Graph (MCG) to capture recurring textures, which are referred to as ``motifs" within feature channels. These motifs reconstruct geometric structures and are learned in a more interpretable way. Subsequently, we integrate features from multiple frequency domains through wavelet inverse transformation. The resulting motif features are utilized to restore geometric structures in the stereo matching process. Experimental results demonstrate the effectiveness of MoCha-V2. MoCha-V2 achieved 1st place on the Middlebury benchmark at the time of its release. Code is available at this https URL.

Title: Large Language Models for Lossless Image Compression: Next-Pixel Prediction in Language Space is All You Need

• Authors: Kecheng Chen, Pingping Zhang, Hui Liu, Jie Liu, Yibing Liu, Jixin Huang, Shiqi Wang, Hong Yan, Haoliang Li
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.12448
• Pdf URL: https://arxiv.org/pdf/2411.12448
• Copy Paste: [[2411.12448]] Large Language Models for Lossless Image Compression: Next-Pixel Prediction in Language Space is All You Need(https://arxiv.org/abs/2411.12448)
• Keywords: large language model
• Abstract: We have recently witnessed that ``Intelligence" and `` Compression" are the two sides of the same coin, where the language large model (LLM) with unprecedented intelligence is a general-purpose lossless compressor for various data modalities. This attribute particularly appeals to the lossless image compression community, given the increasing need to compress high-resolution images in the current streaming media era. Consequently, a spontaneous envision emerges: Can the compression performance of the LLM elevate lossless image compression to new heights? However, our findings indicate that the naive application of LLM-based lossless image compressors suffers from a considerable performance gap compared with existing state-of-the-art (SOTA) codecs on common benchmark datasets. In light of this, we are dedicated to fulfilling the unprecedented intelligence (compression) capacity of the LLM for lossless image compression tasks, thereby bridging the gap between theoretical and practical compression performance. Specifically, we propose P$^{2}$-LLM, a next-pixel prediction-based LLM, which integrates various elaborated insights and methodologies, \textit{e.g.,} pixel-level priors, the in-context ability of LLM, and a pixel-level semantic preservation strategy, to enhance the understanding capacity of pixel sequences for better next-pixel predictions. Extensive experiments on benchmark datasets demonstrate that P$^{2}$-LLM can beat SOTA classical and learned codecs.

Title: \textsc{Neon}: News Entity-Interaction Extraction for Enhanced Question Answering

• Authors: Sneha Singhania, Silviu Cucerzan, Allen Herring, Sujay Kumar Jauhar
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.12449
• Pdf URL: https://arxiv.org/pdf/2411.12449
• Copy Paste: [[2411.12449]] \textsc{Neon}: News Entity-Interaction Extraction for Enhanced Question Answering(https://arxiv.org/abs/2411.12449)
• Keywords: extraction, large language model
• Abstract: Capturing fresh information in near real-time and using it to augment existing large language models (LLMs) is essential to generate up-to-date, grounded, and reliable output. This problem becomes particularly challenging when LLMs are used for informational tasks in rapidly evolving fields, such as Web search related to recent or unfolding events involving entities, where generating temporally relevant responses requires access to up-to-the-hour news sources. However, the information modeled by the parametric memory of LLMs is often outdated, and Web results from prototypical retrieval systems may fail to capture the latest relevant information and struggle to handle conflicting reports in evolving news. To address this challenge, we present the NEON framework, designed to extract emerging entity interactions -- such as events or activities -- as described in news articles. NEON constructs an entity-centric timestamped knowledge graph that captures such interactions, thereby facilitating enhanced QA capabilities related to news events. Our framework innovates by integrating open Information Extraction (openIE) style tuples into LLMs to enable in-context retrieval-augmented generation. This integration demonstrates substantial improvements in QA performance when tackling temporal, entity-centric search queries. Through NEON, LLMs can deliver more accurate, reliable, and up-to-date responses.

Title: Frequency-Aware Guidance for Blind Image Restoration via Diffusion Models

• Authors: Jun Xiao, Zihang Lyu, Hao Xie, Cong Zhang, Yakun Ju, Changjian Shui, Kin-Man Lam
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.12450
• Pdf URL: https://arxiv.org/pdf/2411.12450
• Copy Paste: [[2411.12450]] Frequency-Aware Guidance for Blind Image Restoration via Diffusion Models(https://arxiv.org/abs/2411.12450)
• Keywords: diffusion, generative
• Abstract: Blind image restoration remains a significant challenge in low-level vision tasks. Recently, denoising diffusion models have shown remarkable performance in image synthesis. Guided diffusion models, leveraging the potent generative priors of pre-trained models along with a differential guidance loss, have achieved promising results in blind image restoration. However, these models typically consider data consistency solely in the spatial domain, often resulting in distorted image content. In this paper, we propose a novel frequency-aware guidance loss that can be integrated into various diffusion models in a plug-and-play manner. Our proposed guidance loss, based on 2D discrete wavelet transform, simultaneously enforces content consistency in both the spatial and frequency domains. Experimental results demonstrate the effectiveness of our method in three blind restoration tasks: blind image deblurring, imaging through turbulence, and blind restoration for multiple degradations. Notably, our method achieves a significant improvement in PSNR score, with a remarkable enhancement of 3.72\,dB in image deblurring. Moreover, our method exhibits superior capability in generating images with rich details and reduced distortion, leading to the best visual quality.

Title: Empirical Privacy Evaluations of Generative and Predictive Machine Learning Models -- A review and challenges for practice

• Authors: Flavio Hafner, Chang Sun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12451
• Pdf URL: https://arxiv.org/pdf/2411.12451
• Copy Paste: [[2411.12451]] Empirical Privacy Evaluations of Generative and Predictive Machine Learning Models -- A review and challenges for practice(https://arxiv.org/abs/2411.12451)
• Keywords: privacy, generative
• Abstract: Synthetic data generators, when trained using privacy-preserving techniques like differential privacy, promise to produce synthetic data with formal privacy guarantees, facilitating the sharing of sensitive data. However, it is crucial to empirically assess the privacy risks associated with the generated synthetic data before deploying generative technologies. This paper outlines the key concepts and assumptions underlying empirical privacy evaluation in machine learning-based generative and predictive models. Then, this paper explores the practical challenges for privacy evaluations of generative models for use cases with millions of training records, such as data from statistical agencies and healthcare providers. Our findings indicate that methods designed to verify the correct operation of the training algorithm are effective for large datasets, but they often assume an adversary that is unrealistic in many scenarios. Based on the findings, we highlight a crucial trade-off between the computational feasibility of the evaluation and the level of realism of the assumed threat model. Finally, we conclude with ideas and suggestions for future research.

Title: StrTune: Data Dependence-based Code Slicing for Binary Similarity Detection with Fine-tuned Representation

• Authors: Kaiyan He, Yikun Hu, Xuehui Li, Yunhao Song, Yubo Zhao, Dawu Gu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.12454
• Pdf URL: https://arxiv.org/pdf/2411.12454
• Copy Paste: [[2411.12454]] StrTune: Data Dependence-based Code Slicing for Binary Similarity Detection with Fine-tuned Representation(https://arxiv.org/abs/2411.12454)
• Keywords: security
• Abstract: Binary Code Similarity Detection (BCSD) is significant for software security as it can address binary tasks such as malicious code snippets identification and binary patch analysis by comparing code patterns. Recently, there has been a growing focus on artificial intelligence-based approaches in BCSD due to their scalability and generalization. Because binaries are compiled with different compilation configurations, existing approaches still face notable limitations when comparing binary similarity. First, BCSD requires analysis on code behavior, and existing work claims to extract semantic, but actually still makes analysis in terms of syntax. Second, directly extracting features from assembly sequences, existing work cannot address the issues of instruction reordering and different syntax expressions caused by various compilation configurations. In this paper, we propose StrTune, which slices binary code based on data dependence and perform slice-level fine-tuning. To address the first limitation, StrTune performs backward slicing based on data dependence to capture how a value is computed along the execution. Each slice reflects the collecting semantics of the code, which is stable across different compilation configurations. StrTune introduces flow types to emphasize the independence of computations between slices, forming a graph representation. To overcome the second limitation, based on slices corresponding to the same value computation but having different syntax representation, StrTune utilizes a Siamese Network to fine-tune such pairs, making their representations closer in the feature space.

Title: Guide-to-Explain for Controllable Summarization

• Authors: Sangwon Ryu, Heejin Do, Daehee Kim, Yunsu Kim, Gary Geunbae Lee, Jungseul Ok
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12460
• Pdf URL: https://arxiv.org/pdf/2411.12460
• Copy Paste: [[2411.12460]] Guide-to-Explain for Controllable Summarization(https://arxiv.org/abs/2411.12460)
• Keywords: large language model
• Abstract: Recently, large language models (LLMs) have demonstrated remarkable performance in abstractive summarization tasks. However, controllable summarization with LLMs remains underexplored, limiting their ability to generate summaries that align with specific user preferences. In this paper, we first investigate the capability of LLMs to control diverse attributes, revealing that they encounter greater challenges with numerical attributes, such as length and extractiveness, compared to linguistic attributes. To address this challenge, we propose a guide-to-explain framework (GTE) for controllable summarization. Our GTE framework enables the model to identify misaligned attributes in the initial draft and guides it in explaining errors in the previous output. Based on this reflection, the model generates a well-adjusted summary. As a result, by allowing the model to reflect on its misalignment, we generate summaries that satisfy the desired attributes in surprisingly fewer iterations than other iterative methods solely using LLMs.

Title: NMT-Obfuscator Attack: Ignore a sentence in translation with only one word

• Authors: Sahar Sadrizadeh, César Descalzo, Ljiljana Dolamic, Pascal Frossard
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12473
• Pdf URL: https://arxiv.org/pdf/2411.12473
• Copy Paste: [[2411.12473]] NMT-Obfuscator Attack: Ignore a sentence in translation with only one word(https://arxiv.org/abs/2411.12473)
• Keywords: attack
• Abstract: Neural Machine Translation systems are used in diverse applications due to their impressive performance. However, recent studies have shown that these systems are vulnerable to carefully crafted small perturbations to their inputs, known as adversarial attacks. In this paper, we propose a new type of adversarial attack against NMT models. In this attack, we find a word to be added between two sentences such that the second sentence is ignored and not translated by the NMT model. The word added between the two sentences is such that the whole adversarial text is natural in the source language. This type of attack can be harmful in practical scenarios since the attacker can hide malicious information in the automatic translation made by the target NMT model. Our experiments show that different NMT models and translation tasks are vulnerable to this type of attack. Our attack can successfully force the NMT models to ignore the second part of the input in the translation for more than 50% of all cases while being able to maintain low perplexity for the whole input.

Title: Comparing Prior and Learned Time Representations in Transformer Models of Timeseries

• Authors: Natalia Koliou, Tatiana Boura, Stasinos Konstantopoulos, George Meramveliotakis, George Kosmadakis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12476
• Pdf URL: https://arxiv.org/pdf/2411.12476
• Copy Paste: [[2411.12476]] Comparing Prior and Learned Time Representations in Transformer Models of Timeseries(https://arxiv.org/abs/2411.12476)
• Keywords: robust, transformer
• Abstract: What sets timeseries analysis apart from other machine learning exercises is that time representation becomes a primary aspect of the experiment setup, as it must adequately represent the temporal relations that are relevant for the application at hand. In the work described here we study wo different variations of the Transformer architecture: one where we use the fixed time representation proposed in the literature and one where the time representation is learned from the data. Our experiments use data from predicting the energy output of solar panels, a task that exhibits known periodicities (daily and seasonal) that is straight-forward to encode in the fixed time representation. Our results indicate that even in an experiment where the phenomenon is well-understood, it is difficult to encode prior knowledge due to side-effects that are difficult to mitigate. We conclude that research work is needed to work the human into the learning loop in ways that improve the robustness and trust-worthiness of the network.

Title: Bias Free Sentiment Analysis

• Authors: Hubert Plisiecki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12493
• Pdf URL: https://arxiv.org/pdf/2411.12493
• Copy Paste: [[2411.12493]] Bias Free Sentiment Analysis(https://arxiv.org/abs/2411.12493)
• Keywords: robust, fair, explainability, transformer
• Abstract: This paper introduces the Semantic Propagation Graph Neural Network (SProp GNN), a machine learning sentiment analysis (SA) architecture that relies exclusively on syntactic structures and word-level emotional cues to predict emotions in text. By semantically blinding the model to information about specific words, it is robust to biases such as political or gender bias that have been plaguing previous machine learning-based SA systems. The SProp GNN shows performance superior to lexicon-based alternatives such as VADER and EmoAtlas on two different prediction tasks, and across two languages. Additionally, it approaches the accuracy of transformer-based models while significantly reducing bias in emotion prediction tasks. By offering improved explainability and reducing bias, the SProp GNN bridges the methodological gap between interpretable lexicon approaches and powerful, yet often opaque, deep learning models, offering a robust tool for fair and effective emotion analysis in understanding human behavior through text.

Title: Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus

• Authors: Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, Yasuhiro Sogawa
• Subjects: cs.LG, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2411.12498
• Pdf URL: https://arxiv.org/pdf/2411.12498
• Copy Paste: [[2411.12498]] Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus(https://arxiv.org/abs/2411.12498)
• Keywords: large language model
• Abstract: Large language models (LLMs) are capable of solving a wide range of tasks, yet they have struggled with reasoning. To address this, we propose $\textbf{Additional Logic Training (ALT)}$, which aims to enhance LLMs' reasoning capabilities by program-generated logical reasoning samples. We first establish principles for designing high-quality samples by integrating symbolic logic theory and previous empirical insights. Then, based on these principles, we construct a synthetic corpus named $\textbf{Formal Logic Deduction Diverse}$ ($\textbf{FLD}$$^{\times 2}$), comprising numerous samples of multi-step deduction with unknown facts, diverse reasoning rules, diverse linguistic expressions, and challenging distractors. Finally, we empirically show that ALT on FLD$^{\times2}$ substantially enhances the reasoning capabilities of state-of-the-art LLMs, including LLaMA-3.1-70B. Improvements include gains of up to 30 points on logical reasoning benchmarks, up to 10 points on math and coding benchmarks, and 5 points on the benchmark suite BBH.

Title: Transformer Neural Processes -- Kernel Regression

• Authors: Daniel Jenson, Jhonathan Navott, Mengyan Zhang, Makkunda Sharma, Elizaveta Semenova, Seth Flaxman
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.12502
• Pdf URL: https://arxiv.org/pdf/2411.12502
• Copy Paste: [[2411.12502]] Transformer Neural Processes -- Kernel Regression(https://arxiv.org/abs/2411.12502)
• Keywords: transformer
• Abstract: Stochastic processes model various natural phenomena from disease transmission to stock prices, but simulating and quantifying their uncertainty can be computationally challenging. For example, modeling a Gaussian Process with standard statistical methods incurs an $\mathcal{O}(n^3)$ penalty, and even using state-of-the-art Neural Processes (NPs) incurs an $\mathcal{O}(n^2)$ penalty due to the attention mechanism. We introduce the Transformer Neural Process - Kernel Regression (TNP-KR), a new architecture that incorporates a novel transformer block we call a Kernel Regression Block (KRBlock), which reduces the computational complexity of attention in transformer-based Neural Processes (TNPs) from $\mathcal{O}((n_C+n_T)^2)$ to $O(n_C^2+n_Cn_T)$ by eliminating masked computations, where $n_C$ is the number of context, and $n_T$ is the number of test points, respectively, and a fast attention variant that further reduces all attention calculations to $\mathcal{O}(n_C)$ in space and time complexity. In benchmarks spanning such tasks as meta-regression, Bayesian optimization, and image completion, we demonstrate that the full variant matches the performance of state-of-the-art methods while training faster and scaling two orders of magnitude higher in number of test points, and the fast variant nearly matches that performance while scaling to millions of both test and context points on consumer hardware.

Title: Probe-Me-Not: Protecting Pre-trained Encoders from Malicious Probing

• Authors: Ruyi Ding, Tong Zhou, Lili Su, Aidong Adam Ding, Xiaolin Xu, Yunsi Fei
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.12508
• Pdf URL: https://arxiv.org/pdf/2411.12508
• Copy Paste: [[2411.12508]] Probe-Me-Not: Protecting Pre-trained Encoders from Malicious Probing(https://arxiv.org/abs/2411.12508)
• Keywords: protect, transformer
• Abstract: Adapting pre-trained deep learning models to customized tasks has become a popular choice for developers to cope with limited computational resources and data volume. More specifically, probing--training a downstream head on a pre-trained encoder--has been widely adopted in transfer learning, which helps to prevent overfitting and catastrophic forgetting. However, such generalizability of pre-trained encoders raises concerns about the potential misuse of probing for harmful intentions, such as discriminatory speculation and warfare applications. In this work, we introduce EncoderLock, a novel applicability authorization method designed to protect pre-trained encoders from malicious probing, i.e., yielding poor performance on specified prohibited domains while maintaining their utility in authorized ones. Achieving this balance is challenging because of the opposite optimization objectives and the variety of downstream heads that adversaries can utilize adaptively. To address these challenges, EncoderLock employs two techniques: domain-aware weight selection and updating to restrict applications on prohibited domains/tasks, and self-challenging training scheme that iteratively strengthens resistance against any potential downstream classifiers that adversaries may apply. Moreover, recognizing the potential lack of data from prohibited domains in practical scenarios, we introduce three EncoderLock variants with different levels of data accessibility: supervised (prohibited domain data with labels), unsupervised (prohibited domain data without labels), and zero-shot (no data or labels available). We verify EncoderLock's effectiveness and practicality with a real-world pre-trained Vision Transformer (ViT) encoder from Facebook. These results underscore the valuable contributions EncoderLock brings to the development of responsible AI.

Title: Data Pruning in Generative Diffusion Models

• Authors: Rania Briq, Jiangtao Wang, Steffan Kesselheim
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12523
• Pdf URL: https://arxiv.org/pdf/2411.12523
• Copy Paste: [[2411.12523]] Data Pruning in Generative Diffusion Models(https://arxiv.org/abs/2411.12523)
• Keywords: fair, diffusion, generative
• Abstract: Data pruning is the problem of identifying a core subset that is most beneficial to training and discarding the remainder. While pruning strategies are well studied for discriminative models like those used in classification, little research has gone into their application to generative models. Generative models aim to estimate the underlying distribution of the data, so presumably they should benefit from larger datasets. In this work we aim to shed light on the accuracy of this statement, specifically answer the question of whether data pruning for generative diffusion models could have a positive impact. Contrary to intuition, we show that eliminating redundant or noisy data in large datasets is beneficial particularly when done strategically. We experiment with several pruning methods including recent-state-of-art methods, and evaluate over CelebA-HQ and ImageNet datasets. We demonstrate that a simple clustering method outperforms other sophisticated and computationally demanding methods. We further exhibit how we can leverage clustering to balance skewed datasets in an unsupervised manner to allow fair sampling for underrepresented populations in the data distribution, which is a crucial problem in generative models.

Title: Rethinking Top Probability from Multi-view for Distracted Driver Behaviour Localization

• Authors: Quang Vinh Nguyen, Vo Hoang Thanh Son, Chau Truong Vinh Hoang, Duc Duy Nguyen, Nhat Huy Nguyen Minh, Soo-Hyung Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12525
• Pdf URL: https://arxiv.org/pdf/2411.12525
• Copy Paste: [[2411.12525]] Rethinking Top Probability from Multi-view for Distracted Driver Behaviour Localization(https://arxiv.org/abs/2411.12525)
• Keywords: robust
• Abstract: Naturalistic driving action localization task aims to recognize and comprehend human behaviors and actions from video data captured during real-world driving scenarios. Previous studies have shown great action localization performance by applying a recognition model followed by probability-based post-processing. Nevertheless, the probabilities provided by the recognition model frequently contain confused information causing challenge for post-processing. In this work, we adopt an action recognition model based on self-supervise learning to detect distracted activities and give potential action probabilities. Subsequently, a constraint ensemble strategy takes advantages of multi-camera views to provide robust predictions. Finally, we introduce a conditional post-processing operation to locate distracted behaviours and action temporal boundaries precisely. Experimenting on test set A2, our method obtains the sixth position on the public leaderboard of track 3 of the 2024 AI City Challenge.

Title: Contourlet Refinement Gate Framework for Thermal Spectrum Distribution Regularized Infrared Image Super-Resolution

• Authors: Yang Zou, Zhixin Chen, Zhipeng Zhang, Xingyuan Li, Long Ma, Jinyuan Liu, Peng Wang, Yanning Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12530
• Pdf URL: https://arxiv.org/pdf/2411.12530
• Copy Paste: [[2411.12530]] Contourlet Refinement Gate Framework for Thermal Spectrum Distribution Regularized Infrared Image Super-Resolution(https://arxiv.org/abs/2411.12530)
• Keywords: diffusion, transformer
• Abstract: Image super-resolution (SR) is a classical yet still active low-level vision problem that aims to reconstruct high-resolution (HR) images from their low-resolution (LR) counterparts, serving as a key technique for image enhancement. Current approaches to address SR tasks, such as transformer-based and diffusion-based methods, are either dedicated to extracting RGB image features or assuming similar degradation patterns, neglecting the inherent modal disparities between infrared and visible images. When directly applied to infrared image SR tasks, these methods inevitably distort the infrared spectral distribution, compromising the machine perception in downstream tasks. In this work, we emphasize the infrared spectral distribution fidelity and propose a Contourlet refinement gate framework to restore infrared modal-specific features while preserving spectral distribution fidelity. Our approach captures high-pass subbands from multi-scale and multi-directional infrared spectral decomposition to recover infrared-degraded information through a gate architecture. The proposed Spectral Fidelity Loss regularizes the spectral frequency distribution during reconstruction, which ensures the preservation of both high- and low-frequency components and maintains the fidelity of infrared-specific features. We propose a two-stage prompt-learning optimization to guide the model in learning infrared HR characteristics from LR degradation. Extensive experiments demonstrate that our approach outperforms existing image SR models in both visual and perceptual tasks while notably enhancing machine perception in downstream tasks. Our code is available at this https URL.

Title: Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues

• Authors: Riccardo Grazzi, Julien Siems, Jörg K.H. Franke, Arber Zela, Frank Hutter, Massimiliano Pontil
• Subjects: cs.LG, cs.CL, cs.FL
• Abstract URL: https://arxiv.org/abs/2411.12537
• Pdf URL: https://arxiv.org/pdf/2411.12537
• Copy Paste: [[2411.12537]] Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues(https://arxiv.org/abs/2411.12537)
• Keywords: transformer, large language model
• Abstract: Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers in large language modeling, offering linear scaling with sequence length and improved training efficiency. However, LRNNs struggle to perform state-tracking which may impair performance in tasks such as code evaluation or tracking a chess game. Even parity, the simplest state-tracking task, which non-linear RNNs like LSTM handle effectively, cannot be solved by current LRNNs. Recently, Sarrof et al. (2024) demonstrated that the failure of LRNNs like Mamba to solve parity stems from restricting the value range of their diagonal state-transition matrices to $[0, 1]$ and that incorporating negative values can resolve this issue. We extend this result to non-diagonal LRNNs, which have recently shown promise in models such as DeltaNet. We prove that finite precision LRNNs with state-transition matrices having only positive eigenvalues cannot solve parity, while complex eigenvalues are needed to count modulo $3$. Notably, we also prove that LRNNs can learn any regular language when their state-transition matrices are products of identity minus vector outer product matrices, each with eigenvalues in the range $[-1, 1]$. Our empirical results confirm that extending the eigenvalue range of models like Mamba and DeltaNet to include negative values not only enables them to solve parity but consistently improves their performance on state-tracking tasks. Furthermore, pre-training LRNNs with an extended eigenvalue range for language modeling achieves comparable performance and stability while showing promise on code and math data. Our work enhances the expressivity of modern LRNNs, broadening their applicability without changing the cost of training or inference.

Title: UMGAD: Unsupervised Multiplex Graph Anomaly Detection

• Authors: Xiang Li, Jianpeng Qi, Zhongying Zhao, Guanjie Zheng, Lei Cao, Junyu Dong, Yanwei Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12556
• Pdf URL: https://arxiv.org/pdf/2411.12556
• Copy Paste: [[2411.12556]] UMGAD: Unsupervised Multiplex Graph Anomaly Detection(https://arxiv.org/abs/2411.12556)
• Keywords: extraction
• Abstract: Graph anomaly detection (GAD) is a critical task in graph machine learning, with the primary objective of identifying anomalous nodes that deviate significantly from the majority. This task is widely applied in various real-world scenarios, including fraud detection and social network analysis. However, existing GAD methods still face two major challenges: (1) They are often limited to detecting anomalies in single-type interaction graphs and struggle with multiple interaction types in multiplex heterogeneous graphs; (2) In unsupervised scenarios, selecting appropriate anomaly score thresholds remains a significant challenge for accurate anomaly detection. To address the above challenges, we propose a novel Unsupervised Multiplex Graph Anomaly Detection method, named UMGAD. We first learn multi-relational correlations among nodes in multiplex heterogeneous graphs and capture anomaly information during node attribute and structure reconstruction through graph-masked autoencoder (GMAE). Then, to further weaken the influence of noise and redundant information on abnormal information extraction, we generate attribute-level and subgraph-level augmented-view graphs respectively, and perform attribute and structure reconstruction through GMAE. Finally, We learn to optimize node attributes and structural features through contrastive learning between original-view and augmented-view graphs to improve the model's ability to capture anomalies. Meanwhile, we also propose a new anomaly score threshold selection strategy, which allows the model to be independent of the ground truth in real unsupervised scenarios. Extensive experiments on four datasets show that our \model significantly outperforms state-of-the-art methods, achieving average improvements of 13.48% in AUC and 11.68% in Macro-F1 across all datasets.

Title: Recall and Refine: A Simple but Effective Source-free Open-set Domain Adaptation Framework

• Authors: Ismail Nejjar, Hao Dong, Olga Fink
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12558
• Pdf URL: https://arxiv.org/pdf/2411.12558
• Copy Paste: [[2411.12558]] Recall and Refine: A Simple but Effective Source-free Open-set Domain Adaptation Framework(https://arxiv.org/abs/2411.12558)
• Keywords: privacy
• Abstract: Open-set Domain Adaptation (OSDA) aims to adapt a model from a labeled source domain to an unlabeled target domain, where novel classes - also referred to as target-private unknown classes - are present. Source-free Open-set Domain Adaptation (SF-OSDA) methods address OSDA without accessing labeled source data, making them particularly relevant under privacy constraints. However, SF-OSDA presents significant challenges due to distribution shifts and the introduction of novel classes. Existing SF-OSDA methods typically rely on thresholding the prediction entropy of a sample to identify it as either a known or unknown class but fail to explicitly learn discriminative features for the target-private unknown classes. We propose Recall and Refine (RRDA), a novel SF-OSDA framework designed to address these limitations by explicitly learning features for target-private unknown classes. RRDA employs a two-step process. First, we enhance the model's capacity to recognize unknown classes by training a target classifier with an additional decision boundary, guided by synthetic samples generated from target domain features. This enables the classifier to effectively separate known and unknown classes. In the second step, we adapt the entire model to the target domain, addressing both domain shifts and improving generalization to unknown classes. Any off-the-shelf source-free domain adaptation method (e.g., SHOT, AaD) can be seamlessly integrated into our framework at this stage. Extensive experiments on three benchmark datasets demonstrate that RRDA significantly outperforms existing SF-OSDA and OSDA methods.

Title: Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

• Authors: Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, Max Bartolo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12580
• Pdf URL: https://arxiv.org/pdf/2411.12580
• Copy Paste: [[2411.12580]] Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models(https://arxiv.org/abs/2411.12580)
• Keywords: robust, large language model
• Abstract: The capabilities and limitations of Large Language Models have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies. The sheer volume of data used in the design of LLMs has precluded us from applying the method traditionally used to measure generalisation: train-test set separation. To overcome this, we study what kind of generalisation strategies LLMs employ when performing reasoning tasks by investigating the pretraining data they rely on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining tokens, we identify what documents influence the model outputs for three simple mathematical reasoning tasks and contrast this to the data that are influential for answering factual questions. We find that, while the models rely on mostly distinct sets of data for each factual question, a document often has a similar influence across different reasoning questions within the same task, indicating the presence of procedural knowledge. We further find that the answers to factual questions often show up in the most influential data. However, for reasoning questions the answers usually do not show up as highly influential, nor do the answers to the intermediate reasoning steps. When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain a solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning.

Title: Leveraging MLLM Embeddings and Attribute Smoothing for Compositional Zero-Shot Learning

• Authors: Xudong Yan, Songhe Feng, Yang Zhang, Jian Yang, Yueguan Lin, Haojun Fei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12584
• Pdf URL: https://arxiv.org/pdf/2411.12584
• Copy Paste: [[2411.12584]] Leveraging MLLM Embeddings and Attribute Smoothing for Compositional Zero-Shot Learning(https://arxiv.org/abs/2411.12584)
• Keywords: large language model
• Abstract: Compositional zero-shot learning (CZSL) aims to recognize novel compositions of attributes and objects learned from seen compositions. Previous works disentangle attribute and object by extracting shared and exclusive parts between image pairs sharing the same attribute (object), as well as aligning them with pretrained word embeddings to improve unseen attribute-object recognition. Despite the significant achievements of existing efforts, they are hampered by three limitations: (1) the efficacy of disentanglement is compromised due to the influence of the background and the intricate entanglement of attribute with object in the same parts. (2) existing word embeddings fail to capture complex multimodal semantic information. (3) overconfidence exhibited by existing models in seen compositions hinders their generalization to novel compositions. Being aware of these, we propose a novel framework named Multimodal Large Language Model (MLLM) embeddings and attribute smoothing guided disentanglement (TRIDENT) for CZSL. First, we leverage feature adaptive aggregation modules to mitigate the impact of background, and utilize learnable condition masks to capture multigranularity features for disentanglement. Then, the last hidden states of MLLM are employed as word embeddings for their superior representation capabilities. Moreover, we propose attribute smoothing with auxiliary attributes generated by Large Language Model (LLM) for seen compositions, addressing the issue of overconfidence by encouraging the model to learn more attributes in one given composition. Extensive experiments demonstrate that TRIDENT achieves state-of-the-art performance on three benchmarks.

Title: Whisper Finetuning on Nepali Language

• Authors: Sanjay Rijal, Shital Adhikari, Manish Dahal, Manish Awale, Vaghawan Ojha
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12587
• Pdf URL: https://arxiv.org/pdf/2411.12587
• Copy Paste: [[2411.12587]] Whisper Finetuning on Nepali Language(https://arxiv.org/abs/2411.12587)
• Keywords: robust
• Abstract: Despite the growing advancements in Automatic Speech Recognition (ASR) models, the development of robust models for underrepresented languages, such as Nepali, remains a challenge. This research focuses on making an exhaustive and generalized dataset followed by fine-tuning OpenAI's Whisper models of different sizes to improve transcription (speech-to-text) accuracy for the Nepali language. We leverage publicly available ASR datasets and self-recorded custom datasets with a diverse range of accents, dialects, and speaking styles further enriched through augmentation. Our experimental results demonstrate that fine-tuning Whisper models on our curated custom dataset substantially reduces the Word Error Rate (WER) across all model sizes attributed to larger data variations in terms of speaker's age, gender, and sentiment, acoustic environment, dialect, denser audio segments (15-30 seconds) that are more compatible with Whisper's input, and manual curation of audios and transcriptions. Notably, our approach outperforms Whisper's baseline models trained on Fleur's dataset, achieving WER reductions of up to 36.2% on the small and 23.8% on medium models. Furthermore, we show that data augmentation plays a significant role in enhancing model robustness. Our approach underlines the importance of dataset quality, variation, and augmentation in the adaptation of state-of-the-art models to underrepresented languages for developing accurate ASR systems.

Title: ULTra: Unveiling Latent Token Interpretability in Transformer Based Understanding

• Authors: Hesam Hosseini, Ghazal Hosseini Mighan, Amirabbas Afzali, Sajjad Amini, Amir Houmansadr
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12589
• Pdf URL: https://arxiv.org/pdf/2411.12589
• Copy Paste: [[2411.12589]] ULTra: Unveiling Latent Token Interpretability in Transformer Based Understanding(https://arxiv.org/abs/2411.12589)
• Keywords: robust, interpretability, transformer, segmentation
• Abstract: Transformers have revolutionized Computer Vision (CV) and Natural Language Processing (NLP) through self-attention mechanisms. However, due to their complexity, their latent token representations are often difficult to interpret. We introduce a novel framework that interprets Transformer embeddings, uncovering meaningful semantic patterns within them. Based on this framework, we demonstrate that zero-shot unsupervised semantic segmentation can be performed effectively without any fine-tuning using a model pre-trained for tasks other than segmentation. Our method reveals the inherent capacity of Transformer models for understanding input semantics and achieves state-of-the-art performance in semantic segmentation, outperforming traditional segmentation models. Specifically, our approach achieves an accuracy of 67.2 % and an mIoU of 32.9 % on the COCO-Stuff dataset, as well as an mIoU of 51.9 % on the PASCAL VOC dataset. Additionally, we validate our interpretability framework on LLMs for text summarization, demonstrating its broad applicability and robustness.

Title: Debias your Large Multi-Modal Model at Test-Time with Non-Contrastive Visual Attribute Steering

• Authors: Neale Ratzlaff, Matthew Lyle Olson, Musashi Hinck, Estelle Aflalo, Shao-Yen Tseng, Vasudev Lal, Phillip Howard
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12590
• Pdf URL: https://arxiv.org/pdf/2411.12590
• Copy Paste: [[2411.12590]] Debias your Large Multi-Modal Model at Test-Time with Non-Contrastive Visual Attribute Steering(https://arxiv.org/abs/2411.12590)
• Keywords: protect
• Abstract: Large Multi-Modal Models (LMMs) have demonstrated impressive capabilities as general-purpose chatbots that can engage in conversations about a provided input, such as an image. However, their responses are influenced by societal biases present in their training datasets, leading to undesirable differences in how the model responds when presented with images depicting people of different demographics. In this work, we propose a novel debiasing framework for LMMs that directly removes biased representations during text generation to decrease outputs related to protected attributes, or even representing them internally. Our proposed method is training-free; given a single image and a list of target attributes, we can ablate the corresponding representations with just one step of gradient descent on the image itself. Our experiments show that not only can we can minimize the propensity of LMMs to generate text related to protected attributes, but we can improve sentiment and even simply use synthetic data to inform the ablation while retaining language modeling capabilities on real data such as COCO or FACET. Furthermore, we find the resulting generations from a debiased LMM exhibit similar accuracy as a baseline biased model, showing that debiasing effects can be achieved without sacrificing model performance.

Title: Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination

• Authors: Haojie Zheng, Tianyang Xu, Hanchi Sun, Shu Pu, Ruoxi Chen, Lichao Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12591
• Pdf URL: https://arxiv.org/pdf/2411.12591
• Copy Paste: [[2411.12591]] Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination(https://arxiv.org/abs/2411.12591)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) have advanced the integration of visual and linguistic modalities, establishing themselves as the dominant paradigm for visual-language tasks. Current approaches like chain of thought (CoT) reasoning have augmented the cognitive capabilities of large language models (LLMs), yet their adaptation to MLLMs is hindered by heightened risks of hallucination in cross-modality comprehension. In this paper, we find that the thinking while looking paradigm in current multimodal CoT approaches--where reasoning chains are generated alongside visual input--fails to mitigate hallucinations caused by misleading images. To address these limitations, we propose the Visual Inference Chain (VIC) framework, a novel approach that constructs reasoning chains using textual context alone before introducing visual input, effectively reducing cross-modal biases and enhancing multimodal reasoning accuracy. Comprehensive evaluations demonstrate that VIC significantly improves zero-shot performance across various vision-related tasks, mitigating hallucinations while refining the reasoning capabilities of MLLMs. Our code repository can be found at this https URL.

Title: AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction

• Authors: Yuanbin Man, Ying Huang, Chengming Zhang, Bingzhe Li, Wei Niu, Miao Yin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12593
• Pdf URL: https://arxiv.org/pdf/2411.12593
• Copy Paste: [[2411.12593]] AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction(https://arxiv.org/abs/2411.12593)
• Keywords: large language model
• Abstract: The advancements in large language models (LLMs) have propelled the improvement of video understanding tasks by incorporating LLMs with visual models. However, most existing LLM-based models (e.g., VideoLLaMA, VideoChat) are constrained to processing short-duration videos. Recent attempts to understand long-term videos by extracting and compressing visual features into a fixed memory size. Nevertheless, those methods leverage only visual modality to merge video tokens and overlook the correlation between visual and textual queries, leading to difficulties in effectively handling complex question-answering tasks. To address the challenges of long videos and complex prompts, we propose AdaCM$^2$, which, for the first time, introduces an adaptive cross-modality memory reduction approach to video-text alignment in an auto-regressive manner on video streams. Our extensive experiments on various video understanding tasks, such as video captioning, video question answering, and video classification, demonstrate that AdaCM$^2$ achieves state-of-the-art performance across multiple datasets while significantly reducing memory usage. Notably, it achieves a 4.5% improvement across multiple tasks in the LVU dataset with a GPU memory consumption reduction of up to 65%.

Title: SAM Carries the Burden: A Semi-Supervised Approach Refining Pseudo Labels for Medical Segmentation

• Authors: Ron Keuth, Lasse Hansen, Maren Balks, Ronja Jäger, Anne-Nele Schröder, Ludger Tüshaus, Mattias Heinrich
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12602
• Pdf URL: https://arxiv.org/pdf/2411.12602
• Copy Paste: [[2411.12602]] SAM Carries the Burden: A Semi-Supervised Approach Refining Pseudo Labels for Medical Segmentation(https://arxiv.org/abs/2411.12602)
• Keywords: segmentation
• Abstract: Semantic segmentation is a crucial task in medical imaging. Although supervised learning techniques have proven to be effective in performing this task, they heavily depend on large amounts of annotated training data. The recently introduced Segment Anything Model (SAM) enables prompt-based segmentation and offers zero-shot generalization to unfamiliar objects. In our work, we leverage SAM's abstract object understanding for medical image segmentation to provide pseudo labels for semi-supervised learning, thereby mitigating the need for extensive annotated training data. Our approach refines initial segmentations that are derived from a limited amount of annotated data (comprising up to 43 cases) by extracting bounding boxes and seed points as prompts forwarded to SAM. Thus, it enables the generation of dense segmentation masks as pseudo labels for unlabelled data. The results show that training with our pseudo labels yields an improvement in Dice score from $74.29\,\%$ to $84.17\,\%$ and from $66.63\,\%$ to $74.87\,\%$ for the segmentation of bones of the paediatric wrist and teeth in dental radiographs, respectively. As a result, our method outperforms intensity-based post-processing methods, state-of-the-art supervised learning for segmentation (nnU-Net), and the semi-supervised mean teacher approach. Our Code is available on GitHub.

Title: STREAM: A Universal State-Space Model for Sparse Geometric Data

• Authors: Mark Schöne, Yash Bhisikar, Karan Bania, Khaleelulla Khan Nazeer, Christian Mayr, Anand Subramoney, David Kappel
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2411.12603
• Pdf URL: https://arxiv.org/pdf/2411.12603
• Copy Paste: [[2411.12603]] STREAM: A Universal State-Space Model for Sparse Geometric Data(https://arxiv.org/abs/2411.12603)
• Keywords: transformer
• Abstract: Handling sparse and unstructured geometric data, such as point clouds or event-based vision, is a pressing challenge in the field of machine vision. Recently, sequence models such as Transformers and state-space models entered the domain of geometric data. These methods require specialized preprocessing to create a sequential view of a set of points. Furthermore, prior works involving sequence models iterate geometric data with either uniform or learned step sizes, implicitly relying on the model to infer the underlying geometric structure. In this work, we propose to encode geometric structure explicitly into the parameterization of a state-space model. State-space models are based on linear dynamics governed by a one-dimensional variable such as time or a spatial coordinate. We exploit this dynamic variable to inject relative differences of coordinates into the step size of the state-space model. The resulting geometric operation computes interactions between all pairs of N points in O(N) steps. Our model deploys the Mamba selective state-space model with a modified CUDA kernel to efficiently map sparse geometric data to modern hardware. The resulting sequence model, which we call STREAM, achieves competitive results on a range of benchmarks from point-cloud classification to event-based vision and audio classification. STREAM demonstrates a powerful inductive bias for sparse geometric data by improving the PointMamba baseline when trained from scratch on the ModelNet40 and ScanObjectNN point cloud analysis datasets. It further achieves, for the first time, 100% test accuracy on all 11 classes of the DVS128 Gestures dataset.

Title: SG-LRA: Self-Generating Automatic Scoliosis Cobb Angle Measurement with Low-Rank Approximation

• Authors: Zhiwen Shao, Yichen Yuan, Lizhuang Ma, Dit-Yan Yeung, Xiaojia Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12604
• Pdf URL: https://arxiv.org/pdf/2411.12604
• Copy Paste: [[2411.12604]] SG-LRA: Self-Generating Automatic Scoliosis Cobb Angle Measurement with Low-Rank Approximation(https://arxiv.org/abs/2411.12604)
• Keywords: privacy, segmentation
• Abstract: Automatic Cobb angle measurement from X-ray images is crucial for scoliosis screening and diagnosis. However, most existing regression-based methods and segmentation-based methods struggle with inaccurate spine representations or mask connectivity/fragmentation issues. Besides, landmark-based methods suffer from insufficient training data and annotations. To address these challenges, we propose a novel framework including Self-Generation pipeline and Low-Rank Approximation representation (SG-LRA) for automatic Cobb angle measurement. Specifically, we propose a parameterized spine contour representation based on LRA, which enables eigen-spine decomposition and spine contour reconstruction. We can directly obtain spine contour with only regressed LRA coefficients, which form a more accurate spine representation than rectangular boxes. Also, we combine LRA coefficient regression with anchor box classification to solve inaccurate predictions and mask connectivity issues. Moreover, we develop a data engine with automatic annotation and automatic selection in an iterative manner, which is trained on a private Spinal2023 dataset. With our data engine, we generate the largest scoliosis X-ray dataset named Spinal-AI2024 largely without privacy leaks. Extensive experiments on public AASCE2019, private Spinal2023, and generated Spinal-AI2024 datasets demonstrate that our method achieves state-of-the-art Cobb angle measurement performance. Our code and Spinal-AI2024 dataset are available at this https URL and this https URL, respectively.

Title: A Multimodal Approach Combining Structural and Cross-domain Textual Guidance for Weakly Supervised OCT Segmentation

• Authors: Jiaqi Yang, Nitish Mehta, Xiaoling Hu, Chao Chen, Chia-Ling Tsai
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12615
• Pdf URL: https://arxiv.org/pdf/2411.12615
• Copy Paste: [[2411.12615]] A Multimodal Approach Combining Structural and Cross-domain Textual Guidance for Weakly Supervised OCT Segmentation(https://arxiv.org/abs/2411.12615)
• Keywords: segmentation
• Abstract: Accurate segmentation of Optical Coherence Tomography (OCT) images is crucial for diagnosing and monitoring retinal diseases. However, the labor-intensive nature of pixel-level annotation limits the scalability of supervised learning with large datasets. Weakly Supervised Semantic Segmentation (WSSS) provides a promising alternative by leveraging image-level labels. In this study, we propose a novel WSSS approach that integrates structural guidance with text-driven strategies to generate high-quality pseudo labels, significantly improving segmentation performance. In terms of visual information, our method employs two processing modules that exchange raw image features and structural features from OCT images, guiding the model to identify where lesions are likely to occur. In terms of textual information, we utilize large-scale pretrained models from cross-domain sources to implement label-informed textual guidance and synthetic descriptive integration with two textual processing modules that combine local semantic features with consistent synthetic descriptions. By fusing these visual and textual components within a multimodal framework, our approach enhances lesion localization accuracy. Experimental results on three OCT datasets demonstrate that our method achieves state-of-the-art performance, highlighting its potential to improve diagnostic accuracy and efficiency in medical imaging.

Title: Exploring the Manifold of Neural Networks Using Diffusion Geometry

• Authors: Elliott Abel, Peyton Crevasse, Yvan Grinspan, Selma Mazioud, Folu Ogundipe, Kristof Reimann, Ellie Schueler, Andrew J. Steindl, Ellen Zhang, Dhananjay Bhaskar, Siddharth Viswanath, Yanlei Zhang, Tim G. J. Rudner, Ian Adelstein, Smita Krishnaswamy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12626
• Pdf URL: https://arxiv.org/pdf/2411.12626
• Copy Paste: [[2411.12626]] Exploring the Manifold of Neural Networks Using Diffusion Geometry(https://arxiv.org/abs/2411.12626)
• Keywords: diffusion
• Abstract: Drawing motivation from the manifold hypothesis, which posits that most high-dimensional data lies on or near low-dimensional manifolds, we apply manifold learning to the space of neural networks. We learn manifolds where datapoints are neural networks by introducing a distance between the hidden layer representations of the neural networks. These distances are then fed to the non-linear dimensionality reduction algorithm PHATE to create a manifold of neural networks. We characterize this manifold using features of the representation, including class separation, hierarchical cluster structure, spectral entropy, and topological structure. Our analysis reveals that high-performing networks cluster together in the manifold, displaying consistent embedding patterns across all these features. Finally, we demonstrate the utility of this approach for guiding hyperparameter optimization and neural architecture search by sampling from the manifold.

Title: Securing Satellite Link Segment: A Secure-by-Component Design

• Authors: Olfa Ben Yahia, William Ferguson, Sumit Chakravarty, Nesrine Benchoubane, Gunes Karabulut Kurt, Gürkan Gür, Gregory Falco
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.12632
• Pdf URL: https://arxiv.org/pdf/2411.12632
• Copy Paste: [[2411.12632]] Securing Satellite Link Segment: A Secure-by-Component Design(https://arxiv.org/abs/2411.12632)
• Keywords: secure, security, attack
• Abstract: The rapid evolution of communication technologies, compounded by recent geopolitical events such as the Viasat cyberattack in February 2022, has highlighted the urgent need for fast and reliable satellite missions for military and civil security operations. Consequently, this paper examines two Earth observation (EO) missions: one utilizing a single low Earth orbit (LEO) satellite and another through a network of LEO satellites, employing a secure-by-component design strategy. This approach begins by defining the scope of technical security engineering, decomposing the system into components and data flows, and enumerating attack surfaces. Then it proceeds by identifying threats to low-level components, applying secure-by-design principles, redesigning components into secure blocks in alignment with the Space Attack Research & Tactic Analysis (SPARTA) framework, and crafting shall statements to refactor the system design, with a particular focus on improving the security of the link segment.

Title: M3D: Dual-Stream Selective State Spaces and Depth-Driven Framework for High-Fidelity Single-View 3D Reconstruction

• Authors: Luoxi Zhang, Pragyan Shrestha, Yu Zhou, Chun Xie, Itaru Kitahara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12635
• Pdf URL: https://arxiv.org/pdf/2411.12635
• Copy Paste: [[2411.12635]] M3D: Dual-Stream Selective State Spaces and Depth-Driven Framework for High-Fidelity Single-View 3D Reconstruction(https://arxiv.org/abs/2411.12635)
• Keywords: extraction
• Abstract: The precise reconstruction of 3D objects from a single RGB image in complex scenes presents a critical challenge in virtual reality, autonomous driving, and robotics. Existing neural implicit 3D representation methods face significant difficulties in balancing the extraction of global and local features, particularly in diverse and complex environments, leading to insufficient reconstruction precision and quality. We propose M3D, a novel single-view 3D reconstruction framework, to tackle these challenges. This framework adopts a dual-stream feature extraction strategy based on Selective State Spaces to effectively balance the extraction of global and local features, thereby improving scene comprehension and representation precision. Additionally, a parallel branch extracts depth information, effectively integrating visual and geometric features to enhance reconstruction quality and preserve intricate details. Experimental results indicate that the fusion of multi-scale features with depth information via the dual-branch feature extraction significantly boosts geometric consistency and fidelity, achieving state-of-the-art reconstruction performance.

Title: DLBacktrace: A Model Agnostic Explainability for any Deep Learning Models

• Authors: Vinay Kumar Sankarapu, Chintan Chitroda, Yashwardhan Rathore, Neeraj Kumar Singh, Pratinav Seth
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12643
• Pdf URL: https://arxiv.org/pdf/2411.12643
• Copy Paste: [[2411.12643]] DLBacktrace: A Model Agnostic Explainability for any Deep Learning Models(https://arxiv.org/abs/2411.12643)
• Keywords: interpretability, explainability, large language model
• Abstract: The rapid advancement of artificial intelligence has led to increasingly sophisticated deep learning models, which frequently operate as opaque 'black boxes' with limited transparency in their decision-making processes. This lack of interpretability presents considerable challenges, especially in high-stakes applications where understanding the rationale behind a model's outputs is as essential as the outputs themselves. This study addresses the pressing need for interpretability in AI systems, emphasizing its role in fostering trust, ensuring accountability, and promoting responsible deployment in mission-critical fields. To address the interpretability challenge in deep learning, we introduce DLBacktrace, an innovative technique developed by the AryaXAI team to illuminate model decisions across a wide array of domains, including simple Multi Layer Perceptron (MLPs), Convolutional Neural Networks (CNNs), Large Language Models (LLMs), Computer Vision Models, and more. We provide a comprehensive overview of the DLBacktrace algorithm and present benchmarking results, comparing its performance against established interpretability methods, such as SHAP, LIME, GradCAM, Integrated Gradients, SmoothGrad, and Attention Rollout, using diverse task-based metrics. The proposed DLBacktrace technique is compatible with various model architectures built in PyTorch and TensorFlow, supporting models like Llama 3.2, other NLP architectures such as BERT and LSTMs, computer vision models like ResNet and U-Net, as well as custom deep neural network (DNN) models for tabular data. This flexibility underscores DLBacktrace's adaptability and effectiveness in enhancing model transparency across a broad spectrum of applications. The library is open-sourced and available at this https URL .

Title: PoM: Efficient Image and Video Generation with the Polynomial Mixer

• Authors: David Picard, Nicolas Dufour
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12663
• Pdf URL: https://arxiv.org/pdf/2411.12663
• Copy Paste: [[2411.12663]] PoM: Efficient Image and Video Generation with the Polynomial Mixer(https://arxiv.org/abs/2411.12663)
• Keywords: diffusion, transformer
• Abstract: Diffusion models based on Multi-Head Attention (MHA) have become ubiquitous to generate high quality images and videos. However, encoding an image or a video as a sequence of patches results in costly attention patterns, as the requirements both in terms of memory and compute grow quadratically. To alleviate this problem, we propose a drop-in replacement for MHA called the Polynomial Mixer (PoM) that has the benefit of encoding the entire sequence into an explicit state. PoM has a linear complexity with respect to the number of tokens. This explicit state also allows us to generate frames in a sequential fashion, minimizing memory and compute requirement, while still being able to train in parallel. We show the Polynomial Mixer is a universal sequence-to-sequence approximator, just like regular MHA. We adapt several Diffusion Transformers (DiT) for generating images and videos with PoM replacing MHA, and we obtain high quality samples while using less computational resources. The code is available at this https URL.

Title: Auto-Evaluation with Few Labels through Post-hoc Regression

• Authors: Benjamin Eyre, David Madras
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.12665
• Pdf URL: https://arxiv.org/pdf/2411.12665
• Copy Paste: [[2411.12665]] Auto-Evaluation with Few Labels through Post-hoc Regression(https://arxiv.org/abs/2411.12665)
• Keywords: robust, generative
• Abstract: Continually evaluating large generative models provides a unique challenge. Often, human annotations are necessary to evaluate high-level properties of these models (e.g. in text or images). However, collecting human annotations of samples can be resource intensive, and using other machine learning systems to provide the annotations, or automatic evaluation, can introduce systematic errors into the evaluation. The Prediction Powered Inference (PPI) framework provides a way of leveraging both the statistical power of automatic evaluation and a small pool of labelled data to produce a low-variance, unbiased estimate of the quantity being evaluated for. However, most work on PPI considers a relatively sizable set of labelled samples, which is not always practical to obtain. To this end, we present two new PPI-based techniques that leverage robust regressors to produce even lower variance estimators in the few-label regime.

Title: Machine Learning Approaches on Crop Pattern Recognition a Comparative Analysis

• Authors: Kazi Hasibul Kabir, Md. Zahiruddin Aqib, Sharmin Sultana, Shamim Akhter
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12667
• Pdf URL: https://arxiv.org/pdf/2411.12667
• Copy Paste: [[2411.12667]] Machine Learning Approaches on Crop Pattern Recognition a Comparative Analysis(https://arxiv.org/abs/2411.12667)
• Keywords: security
• Abstract: Monitoring agricultural activities is important to ensure food security. Remote sensing plays a significant role for large-scale continuous monitoring of cultivation activities. Time series remote sensing data were used for the generation of the cropping pattern. Classification algorithms are used to classify crop patterns and mapped agriculture land used. Some conventional classification methods including support vector machine (SVM) and decision trees were applied for crop pattern recognition. However, in this paper, we are proposing Deep Neural Network (DNN) based classification to improve the performance of crop pattern recognition and make a comparative analysis with two (2) other machine learning approaches including Naive Bayes and Random Forest.

Title: IoT-Based 3D Pose Estimation and Motion Optimization for Athletes: Application of C3D and OpenPose

• Authors: Fei Ren, Chao Ren, Tianyi Lyu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12676
• Pdf URL: https://arxiv.org/pdf/2411.12676
• Copy Paste: [[2411.12676]] IoT-Based 3D Pose Estimation and Motion Optimization for Athletes: Application of C3D and OpenPose(https://arxiv.org/abs/2411.12676)
• Keywords: robust, extraction
• Abstract: This study proposes the IoT-Enhanced Pose Optimization Network (IE-PONet) for high-precision 3D pose estimation and motion optimization of track and field athletes. IE-PONet integrates C3D for spatiotemporal feature extraction, OpenPose for real-time keypoint detection, and Bayesian optimization for hyperparameter tuning. Experimental results on NTURGB+D and FineGYM datasets demonstrate superior performance, with AP\(^p50\) scores of 90.5 and 91.0, and mAP scores of 74.3 and 74.0, respectively. Ablation studies confirm the essential roles of each module in enhancing model accuracy. IE-PONet provides a robust tool for athletic performance analysis and optimization, offering precise technical insights for training and injury prevention. Future work will focus on further model optimization, multimodal data integration, and developing real-time feedback mechanisms to enhance practical applications.

Title: Attribute Inference Attacks for Federated Regression Tasks

• Authors: Francesco Diana, Othmane Marfoq, Chuan Xu, Giovanni Neglia, Frédéric Giroire, Eoin Thomas
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.12697
• Pdf URL: https://arxiv.org/pdf/2411.12697
• Copy Paste: [[2411.12697]] Attribute Inference Attacks for Federated Regression Tasks(https://arxiv.org/abs/2411.12697)
• Keywords: privacy, attack, federate
• Abstract: Federated Learning (FL) enables multiple clients, such as mobile phones and IoT devices, to collaboratively train a global machine learning model while keeping their data localized. However, recent studies have revealed that the training phase of FL is vulnerable to reconstruction attacks, such as attribute inference attacks (AIA), where adversaries exploit exchanged messages and auxiliary public information to uncover sensitive attributes of targeted clients. While these attacks have been extensively studied in the context of classification tasks, their impact on regression tasks remains largely unexplored. In this paper, we address this gap by proposing novel model-based AIAs specifically designed for regression tasks in FL environments. Our approach considers scenarios where adversaries can either eavesdrop on exchanged messages or directly interfere with the training process. We benchmark our proposed attacks against state-of-the-art methods using real-world datasets. The results demonstrate a significant increase in reconstruction accuracy, particularly in heterogeneous client datasets, a common scenario in FL. The efficacy of our model-based AIAs makes them better candidates for empirically quantifying privacy leakage for federated regression tasks.

Title: When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanations

• Authors: Huaizhi Ge, Yiming Li, Qifan Wang, Yongfeng Zhang, Ruixiang Tang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12701
• Pdf URL: https://arxiv.org/pdf/2411.12701
• Copy Paste: [[2411.12701]] When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanations(https://arxiv.org/abs/2411.12701)
• Keywords: secure, attack, explainability, transformer, generative, large language model
• Abstract: Large Language Models (LLMs) are vulnerable to backdoor attacks, where hidden triggers can maliciously manipulate model behavior. While several backdoor attack methods have been proposed, the mechanisms by which backdoor functions operate in LLMs remain underexplored. In this paper, we move beyond attacking LLMs and investigate backdoor functionality through the novel lens of natural language explanations. Specifically, we leverage LLMs' generative capabilities to produce human-understandable explanations for their decisions, allowing us to compare explanations for clean and poisoned samples. We explore various backdoor attacks and embed the backdoor into LLaMA models for multiple tasks. Our experiments show that backdoored models produce higher-quality explanations for clean data compared to poisoned data, while generating significantly more consistent explanations for poisoned data than for clean data. We further analyze the explanation generation process, revealing that at the token level, the explanation token of poisoned samples only appears in the final few transformer layers of the LLM. At the sentence level, attention dynamics indicate that poisoned inputs shift attention from the input context when generating the explanation. These findings deepen our understanding of backdoor attack mechanisms in LLMs and offer a framework for detecting such vulnerabilities through explainability techniques, contributing to the development of more secure LLMs.

Title: Strengthening Fake News Detection: Leveraging SVM and Sophisticated Text Vectorization Techniques. Defying BERT?

• Authors: Ahmed Akib Jawad Karim, Kazi Hafiz Md Asad, Aznur Azam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12703
• Pdf URL: https://arxiv.org/pdf/2411.12703
• Copy Paste: [[2411.12703]] Strengthening Fake News Detection: Leveraging SVM and Sophisticated Text Vectorization Techniques. Defying BERT?(https://arxiv.org/abs/2411.12703)
• Keywords: transformer, large language model
• Abstract: The rapid spread of misinformation, particularly through online platforms, underscores the urgent need for reliable detection systems. This study explores the utilization of machine learning and natural language processing, specifically Support Vector Machines (SVM) and BERT, to detect news that are fake. We employ three distinct text vectorization methods for SVM: Term Frequency Inverse Document Frequency (TF-IDF), Word2Vec, and Bag of Words (BoW) evaluating their effectiveness in distinguishing between genuine and fake news. Additionally, we compare these methods against the transformer large language model, BERT. Our comprehensive approach includes detailed preprocessing steps, rigorous model implementation, and thorough evaluation to determine the most effective techniques. The results demonstrate that while BERT achieves superior accuracy with 99.98% and an F1-score of 0.9998, the SVM model with a linear kernel and BoW vectorization also performs exceptionally well, achieving 99.81% accuracy and an F1-score of 0.9980. These findings highlight that, despite BERT's superior performance, SVM models with BoW and TF-IDF vectorization methods come remarkably close, offering highly competitive performance with the advantage of lower computational requirements.

Title: Enhancing Multi-Class Disease Classification: Neoplasms, Cardiovascular, Nervous System, and Digestive Disorders Using Advanced LLMs

• Authors: Ahmed Akib Jawad Karim, Muhammad Zawad Mahmud, Samiha Islam, Aznur Azam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12712
• Pdf URL: https://arxiv.org/pdf/2411.12712
• Copy Paste: [[2411.12712]] Enhancing Multi-Class Disease Classification: Neoplasms, Cardiovascular, Nervous System, and Digestive Disorders Using Advanced LLMs(https://arxiv.org/abs/2411.12712)
• Keywords: transformer
• Abstract: In this research, we explored the improvement in terms of multi-class disease classification via pre-trained language models over Medical-Abstracts-TC-Corpus that spans five medical conditions. We excluded non-cancer conditions and examined four specific diseases. We assessed four LLMs, BioBERT, XLNet, and BERT, as well as a novel base model (Last-BERT). BioBERT, which was pre-trained on medical data, demonstrated superior performance in medical text classification (97% accuracy). Surprisingly, XLNet followed closely (96% accuracy), demonstrating its generalizability across domains even though it was not pre-trained on medical data. LastBERT, a custom model based on the lighter version of BERT, also proved competitive with 87.10% accuracy (just under BERT's 89.33%). Our findings confirm the importance of specialized models such as BioBERT and also support impressions around more general solutions like XLNet and well-tuned transformer architectures with fewer parameters (in this case, LastBERT) in medical domain tasks.

Title: CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs

• Authors: Zhehan Kan, Ce Zhang, Zihan Liao, Yapeng Tian, Wenming Yang, Junyuan Xiao, Xu Li, Dongmei Jiang, Yaowei Wang, Qingmin Liao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12713
• Pdf URL: https://arxiv.org/pdf/2411.12713
• Copy Paste: [[2411.12713]] CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs(https://arxiv.org/abs/2411.12713)
• Keywords: robust
• Abstract: Large Vision-Language Model (LVLM) systems have demonstrated impressive vision-language reasoning capabilities but suffer from pervasive and severe hallucination issues, posing significant risks in critical domains such as healthcare and autonomous systems. Despite previous efforts to mitigate hallucinations, a persistent issue remains: visual defect from vision-language misalignment, creating a bottleneck in visual processing capacity. To address this challenge, we develop Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs (CATCH), based on the Information Bottleneck theory. CATCH introduces Complementary Visual Decoupling (CVD) for visual information separation, Non-Visual Screening (NVS) for hallucination detection, and Adaptive Token-level Contrastive Decoding (ATCD) for hallucination mitigation. CATCH addresses issues related to visual defects that cause diminished fine-grained feature perception and cumulative hallucinations in open-ended scenarios. It is applicable to various visual question-answering tasks without requiring any specific data or prior knowledge, and generalizes robustly to new tasks without additional training, opening new possibilities for advancing LVLM in various challenging applications.

Title: Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation

• Authors: Praveen Srinivasa Varadhan, Amogh Gulati, Ashwin Sankar, Srija Anand, Anirudh Gupta, Anirudh Mukherjee, Shiva Kumar Marepally, Ankur Bhatia, Saloni Jaju, Suvrat Bhooshan, Mitesh M. Khapra
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.12719
• Pdf URL: https://arxiv.org/pdf/2411.12719
• Copy Paste: [[2411.12719]] Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation(https://arxiv.org/abs/2411.12719)
• Keywords: robust, fair
• Abstract: Despite rapid advancements in TTS models, a consistent and robust human evaluation framework is still lacking. For example, MOS tests fail to differentiate between similar models, and CMOS's pairwise comparisons are time-intensive. The MUSHRA test is a promising alternative for evaluating multiple TTS systems simultaneously, but in this work we show that its reliance on matching human reference speech unduly penalises the scores of modern TTS systems that can exceed human speech quality. More specifically, we conduct a comprehensive assessment of the MUSHRA test, focusing on its sensitivity to factors such as rater variability, listener fatigue, and reference bias. Based on our extensive evaluation involving 471 human listeners across Hindi and Tamil we identify two primary shortcomings: (i) reference-matching bias, where raters are unduly influenced by the human reference, and (ii) judgement ambiguity, arising from a lack of clear fine-grained guidelines. To address these issues, we propose two refined variants of the MUSHRA test. The first variant enables fairer ratings for synthesized samples that surpass human reference quality. The second variant reduces ambiguity, as indicated by the relatively lower variance across raters. By combining these approaches, we achieve both more reliable and more fine-grained assessments. We also release MANGO, a massive dataset of 47,100 human ratings, the first-of-its-kind collection for Indian languages, aiding in analyzing human preferences and developing automatic metrics for evaluating TTS systems.

Title: An AI-Enabled Side Channel Power Analysis Based Hardware Trojan Detection Method for Securing the Integrated Circuits in Cyber-Physical Systems

• Authors: Sefatun-Noor Puspa, Abyad Enan, Reek Majumdar, M Sabbir Salek, Gurcan Comert, Mashrur Chowdhury
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.12721
• Pdf URL: https://arxiv.org/pdf/2411.12721
• Copy Paste: [[2411.12721]] An AI-Enabled Side Channel Power Analysis Based Hardware Trojan Detection Method for Securing the Integrated Circuits in Cyber-Physical Systems(https://arxiv.org/abs/2411.12721)
• Keywords: security, attack, steal
• Abstract: Cyber-physical systems rely on sensors, communication, and computing, all powered by integrated circuits (ICs). ICs are largely susceptible to various hardware attacks with malicious intents. One of the stealthiest threats is the insertion of a hardware trojan into the IC, causing the circuit to malfunction or leak sensitive information. Due to supply chain vulnerabilities, ICs face risks of trojan insertion during various design and fabrication stages. These trojans typically remain inactive until triggered. Once triggered, trojans can severely compromise system safety and security. This paper presents a non-invasive method for hardware trojan detection based on side-channel power analysis. We utilize the dynamic power measurements for twelve hardware trojans from IEEE DataPort. Our approach applies to signal processing techniques to extract crucial time-domain and frequency-domain features from the power traces, which are then used for trojan detection leveraging Artificial Intelligence (AI) models. Comparison with a baseline detection approach indicates that our approach achieves higher detection accuracy than the baseline models used on the same side-channel power dataset.

Title: Information Theory of Meaningful Communication

• Authors: Doron Sivan, Misha Tsodyks
• Subjects: cs.CL, cs.IT
• Abstract URL: https://arxiv.org/abs/2411.12728
• Pdf URL: https://arxiv.org/pdf/2411.12728
• Copy Paste: [[2411.12728]] Information Theory of Meaningful Communication(https://arxiv.org/abs/2411.12728)
• Keywords: large language model
• Abstract: In Shannon's seminal paper, entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its printed form: (i) the units of information are not characters or even words but clauses, i.e. shortest meaningful parts of speech; and (ii) what is transmitted is principally the meaning of what is being said or written, while the precise phrasing that was used to communicate the meaning is typically ignored. In this study, we show that one can leverage recently developed large language models to quantify information communicated in meaningful narratives in terms of bits of meaning per clause.

Title: Benchmarking Positional Encodings for GNNs and Graph Transformers

• Authors: Florian Grötschla, Jiaqing Xie, Roger Wattenhofer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12732
• Pdf URL: https://arxiv.org/pdf/2411.12732
• Copy Paste: [[2411.12732]] Benchmarking Positional Encodings for GNNs and Graph Transformers(https://arxiv.org/abs/2411.12732)
• Keywords: transformer
• Abstract: Recent advances in Graph Neural Networks (GNNs) and Graph Transformers (GTs) have been driven by innovations in architectures and Positional Encodings (PEs), which are critical for augmenting node features and capturing graph topology. PEs are essential for GTs, where topological information would otherwise be lost without message-passing. However, PEs are often tested alongside novel architectures, making it difficult to isolate their effect on established models. To address this, we present a comprehensive benchmark of PEs in a unified framework that includes both message-passing GNNs and GTs. We also establish theoretical connections between MPNNs and GTs and introduce a sparsified GRIT attention mechanism to examine the influence of global connectivity. Our findings demonstrate that previously untested combinations of GNN architectures and PEs can outperform existing methods and offer a more comprehensive picture of the state-of-the-art. To support future research and experimentation in our framework, we make the code publicly available.

Title: ACING: Actor-Critic for Instruction Learning in Black-Box Large Language Models

• Authors: Salma Kharrat, Fares Fourati, Marco Canini
• Subjects: cs.CL, cs.AI, cs.LG, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2411.12736
• Pdf URL: https://arxiv.org/pdf/2411.12736
• Copy Paste: [[2411.12736]] ACING: Actor-Critic for Instruction Learning in Black-Box Large Language Models(https://arxiv.org/abs/2411.12736)
• Keywords: large language model
• Abstract: The effectiveness of Large Language Models (LLMs) in solving tasks vastly depends on the quality of the instructions, which often require fine-tuning through extensive human effort. This highlights the need for automated instruction optimization; however, this optimization is particularly challenging when dealing with black-box LLMs, where model parameters and gradients remain inaccessible. We propose ACING, a task-specific prompt optimization approach framed as a stateless continuous-action Reinforcement Learning (RL) problem, known as the continuum bandit setting. ACING leverages an actor-critic-based method to optimize prompts, learning from non-differentiable reward signals. We validate ACING by optimizing prompts for ChatGPT on 30 instruction-based tasks. ACING consistently outperforms baseline methods, achieving a median score improvement of 10 percentage points. Furthermore, ACING not only recovers but also surpasses human-crafted expert instructions, achieving up to a 39 percentage point improvement against human benchmarks.