2024-11-21

Title: An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2

• Authors: Pepijn de Reus, Ana Oprescu, Jelle Zuidema
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.12758
• Pdf URL: https://arxiv.org/pdf/2411.12758
• Copy Paste: [[2411.12758]] An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2(https://arxiv.org/abs/2411.12758)
• Keywords: large language model
• Abstract: This study examines quantisation and pruning strategies to reduce energy consumption in code Large Language Models (LLMs) inference. Using StarCoder2, we observe increased energy demands with quantization due to lower throughput and some accuracy losses. Conversely, pruning reduces energy usage but impairs performance. The results highlight challenges and trade-offs in LLM model compression. We suggest future work on hardware-optimized quantization to enhance efficiency with minimal loss in accuracy.

Title: A Novel Approach to Eliminating Hallucinations in Large Language Model-Assisted Causal Discovery

• Authors: Grace Sng, Yanming Zhang, Klaus Mueller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12759
• Pdf URL: https://arxiv.org/pdf/2411.12759
• Copy Paste: [[2411.12759]] A Novel Approach to Eliminating Hallucinations in Large Language Model-Assisted Causal Discovery(https://arxiv.org/abs/2411.12759)
• Keywords: large language model
• Abstract: The increasing use of large language models (LLMs) in causal discovery as a substitute for human domain experts highlights the need for optimal model selection. This paper presents the first hallucination survey of popular LLMs for causal discovery. We show that hallucinations exist when using LLMs in causal discovery so the choice of LLM is important. We propose using Retrieval Augmented Generation (RAG) to reduce hallucinations when quality data is available. Additionally, we introduce a novel method employing multiple LLMs with an arbiter in a debate to audit edges in causal graphs, achieving a comparable reduction in hallucinations to RAG.

Title: Playing Language Game with LLMs Leads to Jailbreaking

• Authors: Yu Peng, Zewen Long, Fangming Dong, Congyi Li, Shu Wu, Kai Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12762
• Pdf URL: https://arxiv.org/pdf/2411.12762
• Copy Paste: [[2411.12762]] Playing Language Game with LLMs Leads to Jailbreaking(https://arxiv.org/abs/2411.12762)
• Keywords: security, defense, attack, large language model
• Abstract: The advent of large language models (LLMs) has spurred the development of numerous jailbreak techniques aimed at circumventing their security defenses against malicious attacks. An effective jailbreak approach is to identify a domain where safety generalization fails, a phenomenon known as mismatched generalization. In this paper, we introduce two novel jailbreak methods based on mismatched generalization: natural language games and custom language games, both of which effectively bypass the safety mechanisms of LLMs, with various kinds and different variants, making them hard to defend and leading to high attack rates. Natural language games involve the use of synthetic linguistic constructs and the actions intertwined with these constructs, such as the Ubbi Dubbi language. Building on this phenomenon, we propose the custom language games method: by engaging with LLMs using a variety of custom rules, we successfully execute jailbreak attacks across multiple LLM platforms. Extensive experiments demonstrate the effectiveness of our methods, achieving success rates of 93% on GPT-4o, 89% on GPT-4o-mini and 83% on Claude-3.5-Sonnet. Furthermore, to investigate the generalizability of safety alignments, we fine-tuned Llama-3.1-70B with the custom language games to achieve safety alignment within our datasets and found that when interacting through other language games, the fine-tuned models still failed to identify harmful content. This finding indicates that the safety alignment knowledge embedded in LLMs fails to generalize across different linguistic formats, thus opening new avenues for future research in this area.

Title: SEFD: Semantic-Enhanced Framework for Detecting LLM-Generated Text

• Authors: Weiqing He, Bojian Hou, Tianqi Shang, Davoud Ataee Tarzanagh, Qi Long, Li Shen
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.12764
• Pdf URL: https://arxiv.org/pdf/2411.12764
• Copy Paste: [[2411.12764]] SEFD: Semantic-Enhanced Framework for Detecting LLM-Generated Text(https://arxiv.org/abs/2411.12764)
• Keywords: robust, large language model
• Abstract: The widespread adoption of large language models (LLMs) has created an urgent need for robust tools to detect LLM-generated text, especially in light of \textit{paraphrasing} techniques that often evade existing detection methods. To address this challenge, we present a novel semantic-enhanced framework for detecting LLM-generated text (SEFD) that leverages a retrieval-based mechanism to fully utilize text semantics. Our framework improves upon existing detection methods by systematically integrating retrieval-based techniques with traditional detectors, employing a carefully curated retrieval mechanism that strikes a balance between comprehensive coverage and computational efficiency. We showcase the effectiveness of our approach in sequential text scenarios common in real-world applications, such as online forums and Q\&A platforms. Through comprehensive experiments across various LLM-generated texts and detection methods, we demonstrate that our framework substantially enhances detection accuracy in paraphrasing scenarios while maintaining robustness for standard LLM-generated content.

Title: CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularization

• Authors: Nay Myat Min, Long H. Pham, Yige Li, Jun Sun
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12768
• Pdf URL: https://arxiv.org/pdf/2411.12768
• Copy Paste: [[2411.12768]] CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularization(https://arxiv.org/abs/2411.12768)
• Keywords: defense, attack, generative, large language model
• Abstract: Recent studies reveal that Large Language Models (LLMs) are susceptible to backdoor attacks, where adversaries embed hidden triggers that manipulate model responses. Existing backdoor defense methods are primarily designed for vision or classification tasks, and are thus ineffective for text generation tasks, leaving LLMs vulnerable. We introduce Internal Consistency Regularization (CROW), a novel defense using consistency regularization finetuning to address layer-wise inconsistencies caused by backdoor triggers. CROW leverages the intuition that clean models exhibit smooth, consistent transitions in hidden representations across layers, whereas backdoored models show noticeable fluctuation when triggered. By enforcing internal consistency through adversarial perturbations and regularization, CROW neutralizes backdoor effects without requiring clean reference models or prior trigger knowledge, relying only on a small set of clean data. This makes it practical for deployment across various LLM architectures. Experimental results demonstrate that CROW consistently achieves a significant reductions in attack success rates across diverse backdoor strategies and tasks, including negative sentiment, targeted refusal, and code injection, on models such as Llama-2 (7B, 13B), CodeLlama (7B, 13B) and Mistral-7B, while preserving the model's generative capabilities.

Title: Decoupling Training-Free Guided Diffusion by ADMM

• Authors: Youyuan Zhang, Zehua Liu, Zenan Li, Zhaoyu Li, James J. Clark, Xujie Si
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12773
• Pdf URL: https://arxiv.org/pdf/2411.12773
• Copy Paste: [[2411.12773]] Decoupling Training-Free Guided Diffusion by ADMM(https://arxiv.org/abs/2411.12773)
• Keywords: diffusion
• Abstract: In this paper, we consider the conditional generation problem by guiding off-the-shelf unconditional diffusion models with differentiable loss functions in a plug-and-play fashion. While previous research has primarily focused on balancing the unconditional diffusion model and the guided loss through a tuned weight hyperparameter, we propose a novel framework that distinctly decouples these two components. Specifically, we introduce two variables ${x}$ and ${z}$, to represent the generated samples governed by the unconditional generation model and the guidance function, respectively. This decoupling reformulates conditional generation into two manageable subproblems, unified by the constraint ${x} = {z}$. Leveraging this setup, we develop a new algorithm based on the Alternating Direction Method of Multipliers (ADMM) to adaptively balance these components. Additionally, we establish the equivalence between the diffusion reverse step and the proximal operator of ADMM and provide a detailed convergence analysis of our algorithm under certain mild assumptions. Our experiments demonstrate that our proposed method ADMMDiff consistently generates high-quality samples while ensuring strong adherence to the conditioning criteria. It outperforms existing methods across a range of conditional generation tasks, including image generation with various guidance and controllable motion synthesis.

Title: Faster Multi-GPU Training with PPLL: A Pipeline Parallelism Framework Leveraging Local Learning

• Authors: Xiuyuan Guo (1), Chengqi Xu (1), Guinan Guo (3), Feiyu Zhu (4), Changpeng Cai (5), Peizhe Wang (5), Xiaoming Wei (2), Junhao Su (2), Jialin Gao (2) ((1) University of Southern California, (2) Meituan, (3) Sun Yat-sen University, (4) University of Shanghai for Science and Technology, (5) Southeast University)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12780
• Pdf URL: https://arxiv.org/pdf/2411.12780
• Copy Paste: [[2411.12780]] Faster Multi-GPU Training with PPLL: A Pipeline Parallelism Framework Leveraging Local Learning(https://arxiv.org/abs/2411.12780)
• Keywords: transformer
• Abstract: Currently, training large-scale deep learning models is typically achieved through parallel training across multiple GPUs. However, due to the inherent communication overhead and synchronization delays in traditional model parallelism methods, seamless parallel training cannot be achieved, which, to some extent, affects overall training efficiency. To address this issue, we present PPLL (Pipeline Parallelism based on Local Learning), a novel framework that leverages local learning algorithms to enable effective parallel training across multiple GPUs. PPLL divides the model into several distinct blocks, each allocated to a separate GPU. By utilizing queues to manage data transfers between GPUs, PPLL ensures seamless cross-GPU communication, allowing multiple blocks to execute forward and backward passes in a pipelined manner. This design minimizes idle times and prevents bottlenecks typically caused by sequential gradient updates, thereby accelerating the overall training process. We validate PPLL through extensive experiments using ResNet and Vision Transformer (ViT) architectures on CIFAR-10, SVHN, and STL-10 datasets. Our results demonstrate that PPLL significantly enhances the training speed of the local learning method while achieving comparable or even superior training speed to traditional pipeline parallelism (PP) without sacrificing model performance. In a 4-GPU training setup, PPLL accelerated local learning training on ViT and ResNet by 162% and 33%, respectively, achieving 1.25x and 0.85x the speed of traditional pipeline parallelism.

Title: Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model

• Authors: Yiming Shi, Xun Zhu, Ying Hu, Chenyi Guo, Miao Li, Ji Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12783
• Pdf URL: https://arxiv.org/pdf/2411.12783
• Copy Paste: [[2411.12783]] Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model(https://arxiv.org/abs/2411.12783)
• Keywords: large language model
• Abstract: The analysis of 3D medical images is crucial for modern healthcare, yet traditional task-specific models are becoming increasingly inadequate due to limited generalizability across diverse clinical scenarios. Multimodal large language models (MLLMs) offer a promising solution to these challenges. However, existing MLLMs have limitations in fully leveraging the rich, hierarchical information embedded in 3D medical images. Inspired by clinical practice, where radiologists focus on both 3D spatial structure and 2D planar content, we propose Med-2E3, a novel MLLM for 3D medical image analysis that integrates 3D and 2D encoders. To aggregate 2D features more effectively, we design a Text-Guided Inter-Slice (TG-IS) scoring module, which scores the attention of each 2D slice based on slice contents and task instructions. To the best of our knowledge, Med-2E3 is the first MLLM to integrate both 3D and 2D features for 3D medical image analysis. Experiments on a large-scale, open-source 3D medical multimodal benchmark demonstrate that Med-2E3 exhibits task-specific attention distribution and significantly outperforms current state-of-the-art models, with a 14% improvement in report generation and a 5% gain in medical visual question answering (VQA), highlighting the model's potential in addressing complex multimodal clinical tasks. The code will be released upon acceptance.

Title: Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning

• Authors: Pengkun Jiao, Bin Zhu, Jingjing Chen, Chong-Wah Ngo, Yu-Gang Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12787
• Pdf URL: https://arxiv.org/pdf/2411.12787
• Copy Paste: [[2411.12787]] Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning(https://arxiv.org/abs/2411.12787)
• Keywords: large language model
• Abstract: Fine-tuning multimodal large language models (MLLMs) presents significant challenges, including a reliance on high-level visual features that limits fine-grained detail comprehension, and data conflicts that arise from task complexity. To address these issues, we propose an efficient fine-tuning framework with two novel approaches: Vision Cue Enhancement (VCE) and Dual Low-Rank Adaptation (Dual-LoRA). VCE enhances the vision projector by integrating multi-level visual cues, improving the model's ability to capture fine-grained visual features. Dual-LoRA introduces a dual low-rank structure for instruction tuning, decoupling learning into skill and task spaces to enable precise control and efficient adaptation across diverse tasks. Our method simplifies implementation, enhances visual comprehension, and improves adaptability. Experiments on both downstream tasks and general benchmarks demonstrate the effectiveness of our proposed approach.

Title: Automated 3D Physical Simulation of Open-world Scene with Gaussian Splatting

• Authors: Haoyu Zhao, Hao Wang, Xingyue Zhao, Hongqiu Wang, Zhiyu Wu, Chengjiang Long, Hua Zou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12789
• Pdf URL: https://arxiv.org/pdf/2411.12789
• Copy Paste: [[2411.12789]] Automated 3D Physical Simulation of Open-world Scene with Gaussian Splatting(https://arxiv.org/abs/2411.12789)
• Keywords: large language model, segmentation
• Abstract: Recent advancements in 3D generation models have opened new possibilities for simulating dynamic 3D object movements and customizing behaviors, yet creating this content remains challenging. Current methods often require manual assignment of precise physical properties for simulations or rely on video generation models to predict them, which is computationally intensive. In this paper, we rethink the usage of multi-modal large language model (MLLM) in physics-based simulation, and present Sim Anything, a physics-based approach that endows static 3D objects with interactive dynamics. We begin with detailed scene reconstruction and object-level 3D open-vocabulary segmentation, progressing to multi-view image in-painting. Inspired by human visual reasoning, we propose MLLM-based Physical Property Perception (MLLM-P3) to predict mean physical properties of objects in a zero-shot manner. Based on the mean values and the object's geometry, the Material Property Distribution Prediction model (MPDP) model then estimates the full distribution, reformulating the problem as probability distribution estimation to reduce computational costs. Finally, we simulate objects in an open-world scene with particles sampled via the Physical-Geometric Adaptive Sampling (PGAS) strategy, efficiently capturing complex deformations and significantly reducing computational costs. Extensive experiments and user studies demonstrate our Sim Anything achieves more realistic motion than state-of-the-art methods within 2 minutes on a single GPU.

Title: Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models

• Authors: Zhen Zeng, Leijiang Gu, Xun Yang, Zhangling Duan, Zenglin Shi, Meng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12790
• Pdf URL: https://arxiv.org/pdf/2411.12790
• Copy Paste: [[2411.12790]] Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models(https://arxiv.org/abs/2411.12790)
• Keywords: large language model
• Abstract: Knowledge editing aims to efficiently and cost-effectively correct inaccuracies and update outdated information. Recently, there has been growing interest in extending knowledge editing from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs), which integrate both textual and visual information, introducing additional editing complexities. Existing multimodal knowledge editing works primarily focus on text-oriented, coarse-grained scenarios, failing to address the unique challenges posed by multimodal contexts. In this paper, we propose a visual-oriented, fine-grained multimodal knowledge editing task that targets precise editing in images with multiple interacting entities. We introduce the Fine-Grained Visual Knowledge Editing (FGVEdit) benchmark to evaluate this task. Moreover, we propose a Multimodal Scope Classifier-based Knowledge Editor (MSCKE) framework. MSCKE leverages a multimodal scope classifier that integrates both visual and textual information to accurately identify and update knowledge related to specific entities within images. This approach ensures precise editing while preserving irrelevant information, overcoming the limitations of traditional text-only editing methods. Extensive experiments on the FGVEdit benchmark demonstrate that MSCKE outperforms existing methods, showcasing its effectiveness in solving the complex challenges of multimodal knowledge editing.

Title: Mitigating Perception Bias: A Training-Free Approach to Enhance LMM for Image Quality Assessment

• Authors: Siyi Pan, Baoliang Chen, Danni Huang, Hanwei Zhu, Lingyu Zhu, Xiangjie Sui, Shiqi Wang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.12791
• Pdf URL: https://arxiv.org/pdf/2411.12791
• Copy Paste: [[2411.12791]] Mitigating Perception Bias: A Training-Free Approach to Enhance LMM for Image Quality Assessment(https://arxiv.org/abs/2411.12791)
• Keywords: extraction
• Abstract: Despite the impressive performance of large multimodal models (LMMs) in high-level visual tasks, their capacity for image quality assessment (IQA) remains limited. One main reason is that LMMs are primarily trained for high-level tasks (e.g., image captioning), emphasizing unified image semantics extraction under varied quality. Such semantic-aware yet quality-insensitive perception bias inevitably leads to a heavy reliance on image semantics when those LMMs are forced for quality rating. In this paper, instead of retraining or tuning an LMM costly, we propose a training-free debiasing framework, in which the image quality prediction is rectified by mitigating the bias caused by image semantics. Specifically, we first explore several semantic-preserving distortions that can significantly degrade image quality while maintaining identifiable semantics. By applying these specific distortions to the query or test images, we ensure that the degraded images are recognized as poor quality while their semantics remain. During quality inference, both a query image and its corresponding degraded version are fed to the LMM along with a prompt indicating that the query image quality should be inferred under the condition that the degraded one is deemed poor this http URL prior condition effectively aligns the LMM's quality perception, as all degraded images are consistently rated as poor quality, regardless of their semantic this http URL, the quality scores of the query image inferred under different prior conditions (degraded versions) are aggregated using a conditional probability model. Extensive experiments on various IQA datasets show that our debiasing framework could consistently enhance the LMM performance and the code will be publicly available.

Title: Stylecodes: Encoding Stylistic Information For Image Generation

• Authors: Ciara Rowles
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12811
• Pdf URL: https://arxiv.org/pdf/2411.12811
• Copy Paste: [[2411.12811]] Stylecodes: Encoding Stylistic Information For Image Generation(https://arxiv.org/abs/2411.12811)
• Keywords: diffusion
• Abstract: Diffusion models excel in image generation, but controlling them remains a challenge. We focus on the problem of style-conditioned image generation. Although example images work, they are cumbersome: srefs (style-reference codes) from MidJourney solve this issue by expressing a specific image style in a short numeric code. These have seen widespread adoption throughout social media due to both their ease of sharing and the fact they allow using an image for style control, without having to post the source images themselves. However, users are not able to generate srefs from their own images, nor is the underlying training procedure public. We propose StyleCodes: an open-source and open-research style encoder architecture and training procedure to express image style as a 20-symbol base64 code. Our experiments show that our encoding results in minimal loss in quality compared to traditional image-to-style techniques.

Title: Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline

• Authors: Junlong Cheng, Bin Fu, Jin Ye, Guoan Wang, Tianbin Li, Haoyu Wang, Ruoyu Li, He Yao, Junren Chen, JingWen Li, Yanzhou Su, Min Zhu, Junjun He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12814
• Pdf URL: https://arxiv.org/pdf/2411.12814
• Copy Paste: [[2411.12814]] Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline(https://arxiv.org/abs/2411.12814)
• Keywords: segmentation
• Abstract: Interactive Medical Image Segmentation (IMIS) has long been constrained by the limited availability of large-scale, diverse, and densely annotated datasets, which hinders model generalization and consistent evaluation across different models. In this paper, we introduce the IMed-361M benchmark dataset, a significant advancement in general IMIS research. First, we collect and standardize over 6.4 million medical images and their corresponding ground truth masks from multiple data sources. Then, leveraging the strong object recognition capabilities of a vision foundational model, we automatically generated dense interactive masks for each image and ensured their quality through rigorous quality control and granularity management. Unlike previous datasets, which are limited by specific modalities or sparse annotations, IMed-361M spans 14 modalities and 204 segmentation targets, totaling 361 million masks-an average of 56 masks per image. Finally, we developed an IMIS baseline network on this dataset that supports high-quality mask generation through interactive inputs, including clicks, bounding boxes, text prompts, and their combinations. We evaluate its performance on medical image segmentation tasks from multiple perspectives, demonstrating superior accuracy and scalability compared to existing interactive segmentation models. To facilitate research on foundational models in medical computer vision, we release the IMed-361M and model at this https URL.

Title: Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction

• Authors: Sonny George, Chris Sypherd, Dylan Cashman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12828
• Pdf URL: https://arxiv.org/pdf/2411.12828
• Copy Paste: [[2411.12828]] Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction(https://arxiv.org/abs/2411.12828)
• Keywords: large language model
• Abstract: Large language model (LLM) agents show promise in an increasing number of domains. In many proposed applications, it is expected that the agent reasons over accumulated experience presented in an input prompt. We propose the OEDD (Operationalize Experience Despite Distraction) corpus, a human-annotator-validated body of scenarios with pre-scripted agent histories where the agent must make a decision based on disparate experiential information in the presence of a distractor. We evaluate three state-of-the-art LLMs (GPT-3.5 Turbo, GPT-4o, and Gemini 1.5 Pro) using a minimal chain-of-thought prompting strategy and observe that when (1) the input context contains over 1,615 tokens of historical interactions, (2) a crucially decision-informing premise is the rightful conclusion over two disparate environment premises, and (3) a trivial, but distracting red herring fact follows, all LLMs perform worse than random choice at selecting the better of two actions. Our code and test corpus are publicly available at: this https URL .

Title: Towards motion from video diffusion models

• Authors: Paul Janson, Tiberiu Popa, Eugene Belilovsky
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12831
• Pdf URL: https://arxiv.org/pdf/2411.12831
• Copy Paste: [[2411.12831]] Towards motion from video diffusion models(https://arxiv.org/abs/2411.12831)
• Keywords: diffusion
• Abstract: Text-conditioned video diffusion models have emerged as a powerful tool in the realm of video generation and editing. But their ability to capture the nuances of human movement remains under-explored. Indeed the ability of these models to faithfully model an array of text prompts can lead to a wide host of applications in human and character animation. In this work, we take initial steps to investigate whether these models can effectively guide the synthesis of realistic human body animations. Specifically we propose to synthesize human motion by deforming an SMPL-X body representation guided by Score distillation sampling (SDS) calculated using a video diffusion model. By analyzing the fidelity of the resulting animations, we gain insights into the extent to which we can obtain motion using publicly available text-to-video diffusion models using SDS. Our findings shed light on the potential and limitations of these models for generating diverse and plausible human motions, paving the way for further research in this exciting area.

Title: HyperGAN-CLIP: A Unified Framework for Domain Adaptation, Image Synthesis and Manipulation

• Authors: Abdul Basit Anees, Ahmet Canberk Baykal, Muhammed Burak Kizil, Duygu Ceylan, Erkut Erdem, Aykut Erdem
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12832
• Pdf URL: https://arxiv.org/pdf/2411.12832
• Copy Paste: [[2411.12832]] HyperGAN-CLIP: A Unified Framework for Domain Adaptation, Image Synthesis and Manipulation(https://arxiv.org/abs/2411.12832)
• Keywords: robust, generative
• Abstract: Generative Adversarial Networks (GANs), particularly StyleGAN and its variants, have demonstrated remarkable capabilities in generating highly realistic images. Despite their success, adapting these models to diverse tasks such as domain adaptation, reference-guided synthesis, and text-guided manipulation with limited training data remains challenging. Towards this end, in this study, we present a novel framework that significantly extends the capabilities of a pre-trained StyleGAN by integrating CLIP space via hypernetworks. This integration allows dynamic adaptation of StyleGAN to new domains defined by reference images or textual descriptions. Additionally, we introduce a CLIP-guided discriminator that enhances the alignment between generated images and target domains, ensuring superior image quality. Our approach demonstrates unprecedented flexibility, enabling text-guided image manipulation without the need for text-specific training data and facilitating seamless style transfer. Comprehensive qualitative and quantitative evaluations confirm the robustness and superior performance of our framework compared to existing methods.

Title: Data-to-Model Distillation: Data-Efficient Learning Framework

• Authors: Ahmad Sajedi, Samir Khaki, Lucy Z. Liu, Ehsan Amjadian, Yuri A. Lawryshyn, Konstantinos N. Plataniotis
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12841
• Pdf URL: https://arxiv.org/pdf/2411.12841
• Copy Paste: [[2411.12841]] Data-to-Model Distillation: Data-Efficient Learning Framework(https://arxiv.org/abs/2411.12841)
• Keywords: generative
• Abstract: Dataset distillation aims to distill the knowledge of a large-scale real dataset into small yet informative synthetic data such that a model trained on it performs as well as a model trained on the full dataset. Despite recent progress, existing dataset distillation methods often struggle with computational efficiency, scalability to complex high-resolution datasets, and generalizability to deep architectures. These approaches typically require retraining when the distillation ratio changes, as knowledge is embedded in raw pixels. In this paper, we propose a novel framework called Data-to-Model Distillation (D2M) to distill the real dataset's knowledge into the learnable parameters of a pre-trained generative model by aligning rich representations extracted from real and generated images. The learned generative model can then produce informative training images for different distillation ratios and deep architectures. Extensive experiments on 15 datasets of varying resolutions show D2M's superior performance, re-distillation efficiency, and cross-architecture generalizability. Our method effectively scales up to high-resolution 128x128 ImageNet-1K. Furthermore, we verify D2M's practical benefits for downstream applications in neural architecture search.

Title: Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

• Authors: Shang Liu, Yu Pan, Guanting Chen, Xiaocheng Li
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.12843
• Pdf URL: https://arxiv.org/pdf/2411.12843
• Copy Paste: [[2411.12843]] Reward Modeling with Ordinal Feedback: Wisdom of the Crowd(https://arxiv.org/abs/2411.12843)
• Keywords: large language model
• Abstract: Learning a reward model (RM) from human preferences has been an important component in aligning large language models (LLMs). The canonical setup of learning RMs from pairwise preference data is rooted in the classic Bradley-Terry (BT) model that accepts binary feedback, i.e., the label being either Response 1 is better than Response 2, or the opposite. Such a setup inevitably discards potentially useful samples (such as "tied" between the two responses) and loses more fine-grained information (such as "slightly better"). In this paper, we propose a framework for learning RMs under ordinal feedback which generalizes the case of binary preference feedback to any arbitrary granularity. Specifically, we first identify a marginal unbiasedness condition, which generalizes the assumption of the BT model in the existing binary feedback setting. The condition validates itself via the sociological concept of the wisdom of the crowd. Under the condition, we develop a natural probability model for pairwise preference data under ordinal feedback and analyze its properties. We prove the statistical benefits of ordinal feedback in terms of reducing the Rademacher complexity compared to the case of binary feedback. The proposed learning objective and the theory also extend to hinge loss and direct policy optimization (DPO). In particular, the theoretical analysis may be of independent interest when applying to a seemingly unrelated problem of knowledge distillation to interpret the bias-variance trade-off therein. The framework also sheds light on writing guidance for human annotators. Our numerical experiments validate that fine-grained feedback leads to better reward learning for both in-distribution and out-of-distribution settings. Further experiments show that incorporating a certain proportion of samples with tied preference boosts RM learning.

Title: CDI: Copyrighted Data Identification in Diffusion Models

• Authors: Jan Dubiński, Antoni Kowalczuk, Franziska Boenisch, Adam Dziedzic
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.12858
• Pdf URL: https://arxiv.org/pdf/2411.12858
• Copy Paste: [[2411.12858]] CDI: Copyrighted Data Identification in Diffusion Models(https://arxiv.org/abs/2411.12858)
• Keywords: protect, attack, membership infer, diffusion
• Abstract: Diffusion Models (DMs) benefit from large and diverse datasets for their training. Since this data is often scraped from the Internet without permission from the data owners, this raises concerns about copyright and intellectual property protections. While (illicit) use of data is easily detected for training samples perfectly re-created by a DM at inference time, it is much harder for data owners to verify if their data was used for training when the outputs from the suspect DM are not close replicas. Conceptually, membership inference attacks (MIAs), which detect if a given data point was used during training, present themselves as a suitable tool to address this challenge. However, we demonstrate that existing MIAs are not strong enough to reliably determine the membership of individual images in large, state-of-the-art DMs. To overcome this limitation, we propose CDI, a framework for data owners to identify whether their dataset was used to train a given DM. CDI relies on dataset inference techniques, i.e., instead of using the membership signal from a single data point, CDI leverages the fact that most data owners, such as providers of stock photography, visual media companies, or even individual artists, own datasets with multiple publicly exposed data points which might all be included in the training of a given DM. By selectively aggregating signals from existing MIAs and using new handcrafted methods to extract features for these datasets, feeding them to a scoring model, and applying rigorous statistical testing, CDI allows data owners with as little as 70 data points to identify with a confidence of more than 99% whether their data was used to train a given DM. Thereby, CDI represents a valuable tool for data owners to claim illegitimate use of their copyrighted data.

Title: AzSLD: Azerbaijani Sign Language Dataset for Fingerspelling, Word, and Sentence Translation with Baseline Software

• Authors: Nigar Alishzade, Jamaladdin Hasanov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12865
• Pdf URL: https://arxiv.org/pdf/2411.12865
• Copy Paste: [[2411.12865]] AzSLD: Azerbaijani Sign Language Dataset for Fingerspelling, Word, and Sentence Translation with Baseline Software(https://arxiv.org/abs/2411.12865)
• Keywords: robust
• Abstract: Sign language processing technology development relies on extensive and reliable datasets, instructions, and ethical guidelines. We present a comprehensive Azerbaijani Sign Language Dataset (AzSLD) collected from diverse sign language users and linguistic parameters to facilitate advancements in sign recognition and translation systems and support the local sign language community. The dataset was created within the framework of a vision-based AzSL translation project. This study introduces the dataset as a summary of the fingerspelling alphabet and sentence- and word-level sign language datasets. The dataset was collected from signers of different ages, genders, and signing styles, with videos recorded from two camera angles to capture each sign in full detail. This approach ensures robust training and evaluation of gesture recognition models. AzSLD contains 30,000 videos, each carefully annotated with accurate sign labels and corresponding linguistic translations. The dataset is accompanied by technical documentation and source code to facilitate its use in training and testing. This dataset offers a valuable resource of labeled data for researchers and developers working on sign language recognition, translation, or synthesis. Ethical guidelines were strictly followed throughout the project, with all participants providing informed consent for collecting, publishing, and using the data.

Title: From Text to Pose to Image: Improving Diffusion Model Control and Quality

• Authors: Clément Bonnett, Ariel N. Lee, Franck Wertel, Antoine Tamano, Tanguy Cizain, Pablo Ducru
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12872
• Pdf URL: https://arxiv.org/pdf/2411.12872
• Copy Paste: [[2411.12872]] From Text to Pose to Image: Improving Diffusion Model Control and Quality(https://arxiv.org/abs/2411.12872)
• Keywords: diffusion, generative
• Abstract: In the last two years, text-to-image diffusion models have become extremely popular. As their quality and usage increase, a major concern has been the need for better output control. In addition to prompt engineering, one effective method to improve the controllability of diffusion models has been to condition them on additional modalities such as image style, depth map, or keypoints. This forms the basis of ControlNets or Adapters. When attempting to apply these methods to control human poses in outputs of text-to-image diffusion models, two main challenges have arisen. The first challenge is generating poses following a wide range of semantic text descriptions, for which previous methods involved searching for a pose within a dataset of (caption, pose) pairs. The second challenge is conditioning image generation on a specified pose while keeping both high aesthetic and high pose fidelity. In this article, we fix these two main issues by introducing a text-to-pose (T2P) generative model alongside a new sampling algorithm, and a new pose adapter that incorporates more pose keypoints for higher pose fidelity. Together, these two new state-of-the-art models enable, for the first time, a generative text-to-pose-to-image framework for higher pose control in diffusion models. We release all models and the code used for the experiments at this https URL.

Title: ProSec: Fortifying Code LLMs with Proactive Security Alignment

• Authors: Xiangzhe Xu, Zian Su, Jinyao Guo, Kaiyuan Zhang, Zhenting Wang, Xiangyu Zhang
• Subjects: cs.CR, cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.12882
• Pdf URL: https://arxiv.org/pdf/2411.12882
• Copy Paste: [[2411.12882]] ProSec: Fortifying Code LLMs with Proactive Security Alignment(https://arxiv.org/abs/2411.12882)
• Keywords: secure, security, large language model
• Abstract: Recent advances in code-specific large language models (LLMs) have greatly enhanced code generation and refinement capabilities. However, the safety of code LLMs remains under-explored, posing potential risks as insecure code generated by these models may introduce vulnerabilities into real-world systems. Previous work proposes to collect security-focused instruction-tuning dataset from real-world vulnerabilities. It is constrained by the data sparsity of vulnerable code, and has limited applicability in the iterative post-training workflows of modern LLMs. In this paper, we propose ProSec, a novel proactive security alignment approach designed to align code LLMs with secure coding practices. ProSec systematically exposes the vulnerabilities in a code LLM by synthesizing error-inducing coding scenarios from Common Weakness Enumerations (CWEs), and generates fixes to vulnerable code snippets, allowing the model to learn secure practices through advanced preference learning objectives. The scenarios synthesized by ProSec triggers 25 times more vulnerable code than a normal instruction-tuning dataset, resulting in a security-focused alignment dataset 7 times larger than the previous work. Experiments show that models trained with ProSec is 29.2% to 35.5% more secure compared to previous work, with a marginal negative effect of less than 2 percentage points on model's utility.

Title: Selective Attention: Enhancing Transformer through Principled Context Control

• Authors: Xuechen Zhang, Xiangyu Chang, Mingchen Li, Amit Roy-Chowdhury, Jiasi Chen, Samet Oymak
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12892
• Pdf URL: https://arxiv.org/pdf/2411.12892
• Copy Paste: [[2411.12892]] Selective Attention: Enhancing Transformer through Principled Context Control(https://arxiv.org/abs/2411.12892)
• Keywords: transformer
• Abstract: The attention mechanism within the transformer architecture enables the model to weigh and combine tokens based on their relevance to the query. While self-attention has enjoyed major success, it notably treats all queries $q$ in the same way by applying the mapping $V^\top\text{softmax}(Kq)$, where $V,K$ are the value and key embeddings respectively. In this work, we argue that this uniform treatment hinders the ability to control contextual sparsity and relevance. As a solution, we introduce the $\textit{Selective Self-Attention}$ (SSA) layer that augments the softmax nonlinearity with a principled temperature scaling strategy. By controlling temperature, SSA adapts the contextual sparsity of the attention map to the query embedding and its position in the context window. Through theory and experiments, we demonstrate that this alleviates attention dilution, aids the optimization process, and enhances the model's ability to control softmax spikiness of individual queries. We also incorporate temperature scaling for value embeddings and show that it boosts the model's ability to suppress irrelevant/noisy tokens. Notably, SSA is a lightweight method which introduces less than 0.5% new parameters through a weight-sharing strategy and can be fine-tuned on existing LLMs. Extensive empirical evaluations demonstrate that SSA-equipped models achieve a noticeable and consistent accuracy improvement on language modeling benchmarks.

Title: Tree Species Classification using Machine Learning and 3D Tomographic SAR -- a case study in Northern Europe

• Authors: Colverd Grace, Schade Laura, Takami Jumpei, Bot Karol, Gallego Joseph
• Subjects: cs.LG, cs.CV, physics.data-an
• Abstract URL: https://arxiv.org/abs/2411.12897
• Pdf URL: https://arxiv.org/pdf/2411.12897
• Copy Paste: [[2411.12897]] Tree Species Classification using Machine Learning and 3D Tomographic SAR -- a case study in Northern Europe(https://arxiv.org/abs/2411.12897)
• Keywords: protect
• Abstract: Tree species classification plays an important role in nature conservation, forest inventories, forest management, and the protection of endangered species. Over the past four decades, remote sensing technologies have been extensively utilized for tree species classification, with Synthetic Aperture Radar (SAR) emerging as a key technique. In this study, we employed TomoSense, a 3D tomographic dataset, which utilizes a stack of single-look complex (SLC) images, a byproduct of SAR, captured at different incidence angles to generate a three-dimensional representation of the terrain. Our research focuses on evaluating multiple tabular machine-learning models using the height information derived from the tomographic image intensities to classify eight distinct tree species. The SLC data and tomographic imagery were analyzed across different polarimetric configurations and geosplit configurations. We investigated the impact of these variations on classification accuracy, comparing the performance of various tabular machine-learning models and optimizing them using Bayesian optimization. Additionally, we incorporated a proxy for actual tree height using point cloud data from Light Detection and Ranging (LiDAR) to provide height statistics associated with the model's predictions. This comparison offers insights into the reliability of tomographic data in predicting tree species classification based on height.

Title: Signformer is all you need: Towards Edge AI for Sign Language

• Authors: Eta Yang
• Subjects: cs.CL, cs.CV, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12901
• Pdf URL: https://arxiv.org/pdf/2411.12901
• Copy Paste: [[2411.12901]] Signformer is all you need: Towards Edge AI for Sign Language(https://arxiv.org/abs/2411.12901)
• Keywords: transformer, large language model
• Abstract: Sign language translation, especially in gloss-free paradigm, is confronting a dilemma of impracticality and unsustainability due to growing resource-intensive methodologies. Contemporary state-of-the-arts (SOTAs) have significantly hinged on pretrained sophiscated backbones such as Large Language Models (LLMs), embedding sources, or extensive datasets, inducing considerable parametric and computational inefficiency for sustainable use in real-world scenario. Despite their success, following this research direction undermines the overarching mission of this domain to create substantial value to bridge hard-hearing and common populations. Committing to the prevailing trend of LLM and Natural Language Processing (NLP) studies, we pursue a profound essential change in architecture to achieve ground-up improvements without external aid from pretrained models, prior knowledge transfer, or any NLP strategies considered not-from-scratch. Introducing Signformer, a from-scratch Feather-Giant transforming the area towards Edge AI that redefines extremities of performance and efficiency with LLM-competence and edgy-deployable compactness. In this paper, we present nature analysis of sign languages to inform our algorithmic design and deliver a scalable transformer pipeline with convolution and attention novelty. We achieve new 2nd place on leaderboard with a parametric reduction of 467-1807x against the finests as of 2024 and outcompete almost every other methods in a lighter configuration of 0.57 million parameters.

Title: MLDGG: Meta-Learning for Domain Generalization on Graphs

• Authors: Qin Tian, Chen Zhao, Minglai Shao, Wenjun Wang, Yujie Lin, Dong Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12913
• Pdf URL: https://arxiv.org/pdf/2411.12913
• Copy Paste: [[2411.12913]] MLDGG: Meta-Learning for Domain Generalization on Graphs(https://arxiv.org/abs/2411.12913)
• Keywords: robust
• Abstract: Domain generalization on graphs aims to develop models with robust generalization capabilities, ensuring effective performance on the testing set despite disparities between testing and training distributions. However, existing methods often rely on static encoders directly applied to the target domain, constraining its flexible adaptability. In contrast to conventional methodologies, which concentrate on developing specific generalized models, our framework, MLDGG, endeavors to achieve adaptable generalization across diverse domains by integrating cross-multi-domain meta-learning with structure learning and semantic identification. Initially, it introduces a generalized structure learner to mitigate the adverse effects of task-unrelated edges, enhancing the comprehensiveness of representations learned by Graph Neural Networks (GNNs) while capturing shared structural information across domains. Subsequently, a representation learner is designed to disentangle domain-invariant semantic and domain-specific variation information in node embedding by leveraging causal reasoning for semantic identification, further enhancing generalization. In the context of meta-learning, meta-parameters for both learners are optimized to facilitate knowledge transfer and enable effective adaptation to graphs through fine-tuning within the target domains, where target graphs are inaccessible during training. Our empirical results demonstrate that MLDGG surpasses baseline methods, showcasing its effectiveness in three different distribution shift settings.

Title: Trojan Cleansing with Neural Collapse

• Authors: Xihe Gu, Greg Fields, Yaman Jandali, Tara Javidi, Farinaz Koushanfar
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.12914
• Pdf URL: https://arxiv.org/pdf/2411.12914
• Copy Paste: [[2411.12914]] Trojan Cleansing with Neural Collapse(https://arxiv.org/abs/2411.12914)
• Keywords: attack
• Abstract: Trojan attacks are sophisticated training-time attacks on neural networks that embed backdoor triggers which force the network to produce a specific output on any input which includes the trigger. With the increasing relevance of deep networks which are too large to train with personal resources and which are trained on data too large to thoroughly audit, these training-time attacks pose a significant risk. In this work, we connect trojan attacks to Neural Collapse, a phenomenon wherein the final feature representations of over-parameterized neural networks converge to a simple geometric structure. We provide experimental evidence that trojan attacks disrupt this convergence for a variety of datasets and architectures. We then use this disruption to design a lightweight, broadly generalizable mechanism for cleansing trojan attacks from a wide variety of different network architectures and experimentally demonstrate its efficacy.

Title: VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge

• Authors: Vishwesh Nath, Wenqi Li, Dong Yang, Andriy Myronenko, Mingxin Zheng, Yao Lu, Zhijian Liu, Hongxu Yin, Yee Man Law, Yucheng Tang, Pengfei Guo, Can Zhao, Ziyue Xu, Yufan He, Greg Heinrich, Stephen Aylward, Marc Edgar, Michael Zephyr, Pavlo Molchanov, Baris Turkbey, Holger Roth, Daguang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12915
• Pdf URL: https://arxiv.org/pdf/2411.12915
• Copy Paste: [[2411.12915]] VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge(https://arxiv.org/abs/2411.12915)
• Keywords: segmentation
• Abstract: Generalist vision language models (VLMs) have made significant strides in computer vision, but they fall short in specialized fields like healthcare, where expert knowledge is essential. In traditional computer vision tasks, creative or approximate answers may be acceptable, but in healthcare, precision is this http URL large multimodal models like Gemini and GPT-4o are insufficient for medical tasks due to their reliance on memorized internet knowledge rather than the nuanced expertise required in healthcare. VLMs are usually trained in three stages: vision pre-training, vision-language pre-training, and instruction fine-tuning (IFT). IFT has been typically applied using a mixture of generic and healthcare data. In contrast, we propose that for medical VLMs, a fourth stage of specialized IFT is necessary, which focuses on medical data and includes information from domain expert models. Domain expert models developed for medical use are crucial because they are specifically trained for certain clinical tasks, e.g. to detect tumors and classify abnormalities through segmentation and classification, which learn fine-grained features of medical data$-$features that are often too intricate for a VLM to capture effectively especially in radiology. This paper introduces a new framework, VILA-M3, for medical VLMs that utilizes domain knowledge via expert models. Through our experiments, we show an improved state-of-the-art (SOTA) performance with an average improvement of ~9% over the prior SOTA model Med-Gemini and ~6% over models trained on the specific tasks. Our approach emphasizes the importance of domain expertise in creating precise, reliable VLMs for medical applications.

Title: LEDRO: LLM-Enhanced Design Space Reduction and Optimization for Analog Circuits

• Authors: Dimple Vijay Kochar, Hanrui Wang, Anantha Chandrakasan, Xin Zhang
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.12930
• Pdf URL: https://arxiv.org/pdf/2411.12930
• Copy Paste: [[2411.12930]] LEDRO: LLM-Enhanced Design Space Reduction and Optimization for Analog Circuits(https://arxiv.org/abs/2411.12930)
• Keywords: large language model
• Abstract: Traditional approaches for designing analog circuits are time-consuming and require significant human expertise. Existing automation efforts using methods like Bayesian Optimization (BO) and Reinforcement Learning (RL) are sub-optimal and costly to generalize across different topologies and technology nodes. In our work, we introduce a novel approach, LEDRO, utilizing Large Language Models (LLMs) in conjunction with optimization techniques to iteratively refine the design space for analog circuit sizing. LEDRO is highly generalizable compared to other RL and BO baselines, eliminating the need for design annotation or model training for different topologies or technology nodes. We conduct a comprehensive evaluation of our proposed framework and baseline on 22 different Op-Amp topologies across four FinFET technology nodes. Results demonstrate the superior performance of LEDRO as it outperforms our best baseline by an average of 13% FoM improvement with 2.15x speed-up on low complexity Op-Amps and 48% FoM improvement with 1.7x speed-up on high complexity Op-Amps. This highlights LEDRO's effective performance, efficiency, and generalizability.

Title: Enhancing Thermal MOT: A Novel Box Association Method Leveraging Thermal Identity and Motion Similarity

• Authors: Wassim El Ahmar, Dhanvin Kolhatkar, Farzan Nowruzi, Robert Laganiere
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12943
• Pdf URL: https://arxiv.org/pdf/2411.12943
• Copy Paste: [[2411.12943]] Enhancing Thermal MOT: A Novel Box Association Method Leveraging Thermal Identity and Motion Similarity(https://arxiv.org/abs/2411.12943)
• Keywords: robust
• Abstract: Multiple Object Tracking (MOT) in thermal imaging presents unique challenges due to the lack of visual features and the complexity of motion patterns. This paper introduces an innovative approach to improve MOT in the thermal domain by developing a novel box association method that utilizes both thermal object identity and motion similarity. Our method merges thermal feature sparsity and dynamic object tracking, enabling more accurate and robust MOT performance. Additionally, we present a new dataset comprised of a large-scale collection of thermal and RGB images captured in diverse urban environments, serving as both a benchmark for our method and a new resource for thermal imaging. We conduct extensive experiments to demonstrate the superiority of our approach over existing methods, showing significant improvements in tracking accuracy and robustness under various conditions. Our findings suggest that incorporating thermal identity with motion data enhances MOT performance. The newly collected dataset and source code is available at this https URL

Title: A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection

• Authors: Gabriel Chua, Shing Yee Chan, Shaun Khoo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12946
• Pdf URL: https://arxiv.org/pdf/2411.12946
• Copy Paste: [[2411.12946]] A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection(https://arxiv.org/abs/2411.12946)
• Keywords: data-free, large language model
• Abstract: Large Language Models are prone to off-topic misuse, where users may prompt these models to perform tasks beyond their intended scope. Current guardrails, which often rely on curated examples or custom classifiers, suffer from high false-positive rates, limited adaptability, and the impracticality of requiring real-world data that is not available in pre-production. In this paper, we introduce a flexible, data-free guardrail development methodology that addresses these challenges. By thoroughly defining the problem space qualitatively and passing this to an LLM to generate diverse prompts, we construct a synthetic dataset to benchmark and train off-topic guardrails that outperform heuristic approaches. Additionally, by framing the task as classifying whether the user prompt is relevant with respect to the system prompt, our guardrails effectively generalize to other misuse categories, including jailbreak and harmful prompts. Lastly, we further contribute to the field by open-sourcing both the synthetic dataset and the off-topic guardrail models, providing valuable resources for developing guardrails in pre-production environments and supporting future research and development in LLM safety.

Title: Machine learned reconstruction of tsunami dynamics from sparse observations

• Authors: Edward McDugald, Arvind Mohan, Darren Engwirda, Agnese Marcato, Javier Santos
• Subjects: cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2411.12948
• Pdf URL: https://arxiv.org/pdf/2411.12948
• Copy Paste: [[2411.12948]] Machine learned reconstruction of tsunami dynamics from sparse observations(https://arxiv.org/abs/2411.12948)
• Keywords: transformer
• Abstract: We investigate the use of the Senseiver, a transformer neural network designed for sparse sensing applications, to estimate full-field surface height measurements of tsunami waves from sparse observations. The model is trained on a large ensemble of simulated data generated via a shallow water equations solver, which we show to be a faithful reproduction for the underlying dynamics by comparison to historical events. We train the model on a dataset consisting of 8 tsunami simulations whose epicenters correspond to historical USGS earthquake records, and where the model inputs are restricted to measurements obtained at actively deployed buoy locations. We test the Senseiver on a dataset consisting of 8 simulations not included in training, demonstrating its capability for extrapolation. The results show remarkable resolution of fine scale phase and amplitude features from the true field, provided that at least a few of the sensors have obtained a non-zero signal. Throughout, we discuss which forecasting techniques can be improved by this method, and suggest ways in which the flexibility of the architecture can be leveraged to incorporate arbitrary remote sensing data (eg. HF Radar and satellite measurements) as well as investigate optimal sensor placements.

Title: On the Consistency of Video Large Language Models in Temporal Comprehension

• Authors: Minjoon Jung, Junbin Xiao, Byoung-Tak Zhang, Angela Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.12951
• Pdf URL: https://arxiv.org/pdf/2411.12951
• Copy Paste: [[2411.12951]] On the Consistency of Video Large Language Models in Temporal Comprehension(https://arxiv.org/abs/2411.12951)
• Keywords: robust, large language model
• Abstract: Video large language models (Video-LLMs) can temporally ground language queries and retrieve video moments. Yet, such temporal comprehension capabilities are neither well-studied nor understood. So we conduct a study on prediction consistency -- a key indicator for robustness and trustworthiness of temporal grounding. After the model identifies an initial moment within the video content, we apply a series of probes to check if the model's responses align with this initial grounding as an indicator of reliable comprehension. Our results reveal that current Video-LLMs are sensitive to variations in video contents, language queries, and task settings, unveiling severe deficiencies in maintaining consistency. We further explore common prompting and instruction-tuning methods as potential solutions, but find that their improvements are often unstable. To that end, we propose event temporal verification tuning that explicitly accounts for consistency, and demonstrate significant improvements for both grounding and consistency. Our data and code will be available at this https URL.

Title: A Foundation Model for Unified Urban Spatio-Temporal Flow Prediction

• Authors: Yuan Yuan, Jingtao Ding, Chonghua Han, Depeng Jin, Yong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12972
• Pdf URL: https://arxiv.org/pdf/2411.12972
• Copy Paste: [[2411.12972]] A Foundation Model for Unified Urban Spatio-Temporal Flow Prediction(https://arxiv.org/abs/2411.12972)
• Keywords: transformer
• Abstract: Urban spatio-temporal flow prediction, encompassing traffic flows and crowd flows, is crucial for optimizing city infrastructure and managing traffic and emergency responses. Traditional approaches have relied on separate models tailored to either grid-based data, representing cities as uniform cells, or graph-based data, modeling cities as networks of nodes and edges. In this paper, we build UniFlow, a foundational model for general urban flow prediction that unifies both grid-based and graphbased data. We first design a multi-view spatio-temporal patching mechanism to standardize different data into a consistent sequential format and then introduce a spatio-temporal transformer architecture to capture complex correlations and dynamics. To leverage shared spatio-temporal patterns across different data types and facilitate effective cross-learning, we propose SpatioTemporal Memory Retrieval Augmentation (ST-MRA). By creating structured memory modules to store shared spatio-temporal patterns, ST-MRA enhances predictions through adaptive memory retrieval. Extensive experiments demonstrate that UniFlow outperforms existing models in both grid-based and graph-based flow prediction, excelling particularly in scenarios with limited data availability, showcasing its superior performance and broad applicability. The datasets and code implementation have been released on this https URL.

Title: Adaptive Process-Guided Learning: An Application in Predicting Lake DO Concentrations

• Authors: Runlong Yu, Chonghao Qiu, Robert Ladwig, Paul C. Hanson, Yiqun Xie, Yanhua Li, Xiaowei Jia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12973
• Pdf URL: https://arxiv.org/pdf/2411.12973
• Copy Paste: [[2411.12973]] Adaptive Process-Guided Learning: An Application in Predicting Lake DO Concentrations(https://arxiv.org/abs/2411.12973)
• Keywords: robust
• Abstract: This paper introduces a \textit{Process-Guided Learning (Pril)} framework that integrates physical models with recurrent neural networks (RNNs) to enhance the prediction of dissolved oxygen (DO) concentrations in lakes, which is crucial for sustaining water quality and ecosystem health. Unlike traditional RNNs, which may deliver high accuracy but often lack physical consistency and broad applicability, the \textit{Pril} method incorporates differential DO equations for each lake layer, modeling it as a first-order linear solution using a forward Euler scheme with a daily timestep. However, this method is sensitive to numerical instabilities. When drastic fluctuations occur, the numerical integration is neither mass-conservative nor stable. Especially during stratified conditions, exogenous fluxes into each layer cause significant within-day changes in DO concentrations. To address this challenge, we further propose an \textit{Adaptive Process-Guided Learning (April)} model, which dynamically adjusts timesteps from daily to sub-daily intervals with the aim of mitigating the discrepancies caused by variations in entrainment fluxes. \textit{April} uses a generator-discriminator architecture to identify days with significant DO fluctuations and employs a multi-step Euler scheme with sub-daily timesteps to effectively manage these variations. We have tested our methods on a wide range of lakes in the Midwestern USA, and demonstrated robust capability in predicting DO concentrations even with limited training data. While primarily focused on aquatic ecosystems, this approach is broadly applicable to diverse scientific and engineering disciplines that utilize process-based models, such as power engineering, climate science, and biomedicine.

Title: Training Bilingual LMs with Data Constraints in the Targeted Language

• Authors: Skyler Seto, Maartje ter Hoeve, He Bai, Natalie Schluter, David Grangier
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12986
• Pdf URL: https://arxiv.org/pdf/2411.12986
• Copy Paste: [[2411.12986]] Training Bilingual LMs with Data Constraints in the Targeted Language(https://arxiv.org/abs/2411.12986)
• Keywords: large language model
• Abstract: Large language models are trained on massive scrapes of the web, as required by current scaling laws. Most progress is made for English, given its abundance of high-quality pretraining data. For most other languages, however, such high quality pretraining data is unavailable. In this work, we study how to boost pretrained model performance in a data constrained target language by enlisting data from an auxiliary language for which high quality data is available. We study this by quantifying the performance gap between training with data in a data-rich auxiliary language compared with training in the target language, exploring the benefits of translation systems, studying the limitations of model scaling for data constrained languages, and proposing new methods for upsampling data from the auxiliary language. Our results show that stronger auxiliary datasets result in performance gains without modification to the model or training objective for close languages, and, in particular, that performance gains due to the development of more information-rich English pretraining datasets can extend to targeted language settings with limited data.

Title: MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers

• Authors: Ning Ding, Yehui Tang, Haochen Qin, Zhenli Zhou, Chao Xu, Lin Li, Kai Han, Heng Liao, Yunhe Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12992
• Pdf URL: https://arxiv.org/pdf/2411.12992
• Copy Paste: [[2411.12992]] MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers(https://arxiv.org/abs/2411.12992)
• Keywords: transformer, large language model
• Abstract: In order to reduce the computational complexity of large language models, great efforts have been made to to improve the efficiency of transformer models such as linear attention and flash-attention. However, the model size and corresponding computational complexity are constantly scaled up in pursuit of higher performance. In this work, we present MemoryFormer, a novel transformer architecture which significantly reduces the computational complexity (FLOPs) from a new perspective. We eliminate nearly all the computations of the transformer model except for the necessary computation required by the multi-head attention operation. This is made possible by utilizing an alternative method for feature transformation to replace the linear projection of fully-connected layers. Specifically, we first construct a group of in-memory lookup tables that store a large amount of discrete vectors to replace the weight matrix used in linear projection. We then use a hash algorithm to retrieve a correlated subset of vectors dynamically based on the input embedding. The retrieved vectors combined together will form the output embedding, which provides an estimation of the result of matrix multiplication operation in a fully-connected layer. Compared to conducting matrix multiplication, retrieving data blocks from memory is a much cheaper operation which requires little computations. We train MemoryFormer from scratch and conduct extensive experiments on various benchmarks to demonstrate the effectiveness of the proposed model.

Title: MERLOT: A Distilled LLM-based Mixture-of-Experts Framework for Scalable Encrypted Traffic Classification

• Authors: Yuxuan Chen, Rongpeng Li, Zhifeng Zhao, Honggang Zhang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.13004
• Pdf URL: https://arxiv.org/pdf/2411.13004
• Copy Paste: [[2411.13004]] MERLOT: A Distilled LLM-based Mixture-of-Experts Framework for Scalable Encrypted Traffic Classification(https://arxiv.org/abs/2411.13004)
• Keywords: robust, large language model
• Abstract: We present MERLOT, a scalable mixture-of-expert (MoE) based refinement of distilled large language model optimized for encrypted traffic classification. By applying model distillation techniques in a teacher-student paradigm, compact models derived from GPT-2-base retain high classification accuracy while minimizing computational costs. These models function as specialized experts in an MoE architecture, dynamically assigned via a gating network. Unlike generation-based methods, our approach directly classifies encrypted traffic using the final decoder token with contextual feature embedding as input. Experiments on 10 datasets show superior or competitive performance over the state-of-the-art models while significantly reducing resource demands, underscoring its effectiveness and robustness.

Title: DT-LSD: Deformable Transformer-based Line Segment Detection

• Authors: Sebastian Janampa, Marios Pattichis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13005
• Pdf URL: https://arxiv.org/pdf/2411.13005
• Copy Paste: [[2411.13005]] DT-LSD: Deformable Transformer-based Line Segment Detection(https://arxiv.org/abs/2411.13005)
• Keywords: transformer
• Abstract: Line segment detection is a fundamental low-level task in computer vision, and improvements in this task can impact more advanced methods that depend on it. Most new methods developed for line segment detection are based on Convolutional Neural Networks (CNNs). Our paper seeks to address challenges that prevent the wider adoption of transformer-based methods for line segment detection. More specifically, we introduce a new model called Deformable Transformer-based Line Segment Detection (DT-LSD) that supports cross-scale interactions and can be trained quickly. This work proposes a novel Deformable Transformer-based Line Segment Detector (DT-LSD) that addresses LETR's drawbacks. For faster training, we introduce Line Contrastive DeNoising (LCDN), a technique that stabilizes the one-to-one matching process and speeds up training by 34$\times$. We show that DT-LSD is faster and more accurate than its predecessor transformer-based model (LETR) and outperforms all CNN-based models in terms of accuracy. In the Wireframe dataset, DT-LSD achieves 71.7 for $sAP^{10}$ and 73.9 for $sAP^{15}$; while 33.2 for $sAP^{10}$ and 35.1 for $sAP^{15}$ in the YorkUrban dataset.

Title: Evaluating LLMs Capabilities Towards Understanding Social Dynamics

• Authors: Anique Tahir, Lu Cheng, Manuel Sandoval, Yasin N. Silva, Deborah L. Hall, Huan Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13008
• Pdf URL: https://arxiv.org/pdf/2411.13008
• Copy Paste: [[2411.13008]] Evaluating LLMs Capabilities Towards Understanding Social Dynamics(https://arxiv.org/abs/2411.13008)
• Keywords: generative, large language model
• Abstract: Social media discourse involves people from different backgrounds, beliefs, and motives. Thus, often such discourse can devolve into toxic interactions. Generative Models, such as Llama and ChatGPT, have recently exploded in popularity due to their capabilities in zero-shot question-answering. Because these models are increasingly being used to ask questions of social significance, a crucial research question is whether they can understand social media dynamics. This work provides a critical analysis regarding generative LLM's ability to understand language and dynamics in social contexts, particularly considering cyberbullying and anti-cyberbullying (posts aimed at reducing cyberbullying) interactions. Specifically, we compare and contrast the capabilities of different large language models (LLMs) to understand three key aspects of social dynamics: language, directionality, and the occurrence of bullying/anti-bullying messages. We found that while fine-tuned LLMs exhibit promising results in some social media understanding tasks (understanding directionality), they presented mixed results in others (proper paraphrasing and bullying/anti-bullying detection). We also found that fine-tuning and prompt engineering mechanisms can have positive effects in some tasks. We believe that a understanding of LLM's capabilities is crucial to design future models that can be effectively used in social applications.

Title: LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts

• Authors: Zhuohan Gu, Jiayi Yao, Kuntai Du, Junchen Jiang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13009
• Pdf URL: https://arxiv.org/pdf/2411.13009
• Copy Paste: [[2411.13009]] LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts(https://arxiv.org/abs/2411.13009)
• Keywords: large language model
• Abstract: As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance efficiency and quality, we introduce LLMSteer, a fine-tuning-free framework that enhances LLMs through query-independent attention steering. Tested on popular LLMs and datasets, LLMSteer narrows the performance gap with baselines by 65.9% and reduces the runtime delay by up to 4.8x compared to recent attention steering methods.

Title: Open-World Amodal Appearance Completion

• Authors: Jiayang Ao, Yanbei Jiang, Qiuhong Ke, Krista A. Ehinger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13019
• Pdf URL: https://arxiv.org/pdf/2411.13019
• Copy Paste: [[2411.13019]] Open-World Amodal Appearance Completion(https://arxiv.org/abs/2411.13019)
• Keywords: segmentation
• Abstract: Understanding and reconstructing occluded objects is a challenging problem, especially in open-world scenarios where categories and contexts are diverse and unpredictable. Traditional methods, however, are typically restricted to closed sets of object categories, limiting their use in complex, open-world scenes. We introduce Open-World Amodal Appearance Completion, a training-free framework that expands amodal completion capabilities by accepting flexible text queries as input. Our approach generalizes to arbitrary objects specified by both direct terms and abstract queries. We term this capability reasoning amodal completion, where the system reconstructs the full appearance of the queried object based on the provided image and language query. Our framework unifies segmentation, occlusion analysis, and inpainting to handle complex occlusions and generates completed objects as RGBA elements, enabling seamless integration into applications such as 3D reconstruction and image editing. Extensive evaluations demonstrate the effectiveness of our approach in generalizing to novel objects and occlusions, establishing a new benchmark for amodal completion in open-world settings. The code and datasets will be released after paper acceptance.

Title: Enhancing Transportation Cyber-Physical Systems Security: A Shift to Post-Quantum Cryptography

• Authors: Abdullah Al Mamun, Akid Abrar, Mizanur Rahman, M Sabbir Salek, Mashrur Chowdhury
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.13023
• Pdf URL: https://arxiv.org/pdf/2411.13023
• Copy Paste: [[2411.13023]] Enhancing Transportation Cyber-Physical Systems Security: A Shift to Post-Quantum Cryptography(https://arxiv.org/abs/2411.13023)
• Keywords: secure, security, protect, attack, robust
• Abstract: The rise of quantum computing threatens traditional cryptographic algorithms that secure Transportation Cyber-Physical Systems (TCPS). Shor's algorithm poses a significant threat to RSA and ECC, while Grover's algorithm reduces the security of symmetric encryption schemes, such as AES. The objective of this paper is to underscore the urgency of transitioning to post-quantum cryptography (PQC) to mitigate these risks in TCPS by analyzing the vulnerabilities of traditional cryptographic schemes and the applicability of standardized PQC schemes in TCPS. We analyzed vulnerabilities in traditional cryptography against quantum attacks and reviewed the applicability of NIST-standardized PQC schemes, including CRYSTALS-Kyber, CRYSTALS-Dilithium, and SPHINCS+, in TCPS. We conducted a case study to analyze the vulnerabilities of a TCPS application from the Architecture Reference for Cooperative and Intelligent Transportation (ARC-IT) service package, i.e., Electronic Toll Collection, leveraging the Microsoft Threat Modeling tool. This case study highlights the cryptographic vulnerabilities of a TCPS application and presents how PQC can effectively counter these threats. Additionally, we evaluated CRYSTALS-Kyber's performance across wired and wireless TCPS data communication scenarios. While CRYSTALS-Kyber proves effective in securing TCPS applications over high-bandwidth, low-latency Ethernet networks, our analysis highlights challenges in meeting the stringent latency requirements of safety-critical wireless applications within TCPS. Future research should focus on developing lightweight PQC solutions and hybrid schemes that integrate traditional and PQC algorithms, to enhance compatibility, scalability, and real-time performance, ensuring robust protection against emerging quantum threats in TCPS.

Title: A Theory for Compressibility of Graph Transformers for Transductive Learning

• Authors: Hamed Shirzad, Honghao Lin, Ameya Velingker, Balaji Venkatachalam, David Woodruff, Danica Sutherland
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.13028
• Pdf URL: https://arxiv.org/pdf/2411.13028
• Copy Paste: [[2411.13028]] A Theory for Compressibility of Graph Transformers for Transductive Learning(https://arxiv.org/abs/2411.13028)
• Keywords: transformer
• Abstract: Transductive tasks on graphs differ fundamentally from typical supervised machine learning tasks, as the independent and identically distributed (i.i.d.) assumption does not hold among samples. Instead, all train/test/validation samples are present during training, making them more akin to a semi-supervised task. These differences make the analysis of the models substantially different from other models. Recently, Graph Transformers have significantly improved results on these datasets by overcoming long-range dependency problems. However, the quadratic complexity of full Transformers has driven the community to explore more efficient variants, such as those with sparser attention patterns. While the attention matrix has been extensively discussed, the hidden dimension or width of the network has received less attention. In this work, we establish some theoretical bounds on how and under what conditions the hidden dimension of these networks can be compressed. Our results apply to both sparse and dense variants of Graph Transformers.

Title: RobustFormer: Noise-Robust Pre-training for images and videos

• Authors: Ashish Bastola, Nishant Luitel, Hao Wang, Danda Pani Paudel, Roshani Poudel, Abolfazl Razi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13040
• Pdf URL: https://arxiv.org/pdf/2411.13040
• Copy Paste: [[2411.13040]] RobustFormer: Noise-Robust Pre-training for images and videos(https://arxiv.org/abs/2411.13040)
• Keywords: robust, transformer
• Abstract: While deep learning models are powerful tools that revolutionized many areas, they are also vulnerable to noise as they rely heavily on learning patterns and features from the exact details of the clean data. Transformers, which have become the backbone of modern vision models, are no exception. Current Discrete Wavelet Transforms (DWT) based methods do not benefit from masked autoencoder (MAE) pre-training since the inverse DWT (iDWT) introduced in these approaches is computationally inefficient and lacks compatibility with video inputs in transformer architectures. In this work, we present RobustFormer, a method that overcomes these limitations by enabling noise-robust pre-training for both images and videos; improving the efficiency of DWT-based methods by removing the need for computationally iDWT steps and simplifying the attention mechanism. To our knowledge, the proposed method is the first DWT-based method compatible with video inputs and masked pre-training. Our experiments show that MAE-based pre-training allows us to bypass the iDWT step, greatly reducing computation. Through extensive tests on benchmark datasets, RobustFormer achieves state-of-the-art results for both image and video tasks.

Title: Bounding-box Watermarking: Defense against Model Extraction Attacks on Object Detectors

• Authors: Satoru Koda, Ikuya Morikawa
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13047
• Pdf URL: https://arxiv.org/pdf/2411.13047
• Copy Paste: [[2411.13047]] Bounding-box Watermarking: Defense against Model Extraction Attacks on Object Detectors(https://arxiv.org/abs/2411.13047)
• Keywords: defense, attack, steal, extraction, watermark
• Abstract: Deep neural networks (DNNs) deployed in a cloud often allow users to query models via the APIs. However, these APIs expose the models to model extraction attacks (MEAs). In this attack, the attacker attempts to duplicate the target model by abusing the responses from the API. Backdoor-based DNN watermarking is known as a promising defense against MEAs, wherein the defender injects a backdoor into extracted models via API responses. The backdoor is used as a watermark of the model; if a suspicious model has the watermark (i.e., backdoor), it is verified as an extracted model. This work focuses on object detection (OD) models. Existing backdoor attacks on OD models are not applicable for model watermarking as the defense against MEAs on a realistic threat model. Our proposed approach involves inserting a backdoor into extracted models via APIs by stealthily modifying the bounding-boxes (BBs) of objects detected in queries while keeping the OD capability. In our experiments on three OD datasets, the proposed approach succeeded in identifying the extracted models with 100% accuracy in a wide variety of experimental scenarios.

Title: MEGL: Multimodal Explanation-Guided Learning

• Authors: Yifei Zhang, Tianxu Jiang, Bo Pan, Jingyu Wang, Guangji Bai, Liang Zhao
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13053
• Pdf URL: https://arxiv.org/pdf/2411.13053
• Copy Paste: [[2411.13053]] MEGL: Multimodal Explanation-Guided Learning(https://arxiv.org/abs/2411.13053)
• Keywords: interpretability
• Abstract: Explaining the decision-making processes of Artificial Intelligence (AI) models is crucial for addressing their "black box" nature, particularly in tasks like image classification. Traditional eXplainable AI (XAI) methods typically rely on unimodal explanations, either visual or textual, each with inherent limitations. Visual explanations highlight key regions but often lack rationale, while textual explanations provide context without spatial grounding. Further, both explanation types can be inconsistent or incomplete, limiting their reliability. To address these challenges, we propose a novel Multimodal Explanation-Guided Learning (MEGL) framework that leverages both visual and textual explanations to enhance model interpretability and improve classification performance. Our Saliency-Driven Textual Grounding (SDTG) approach integrates spatial information from visual explanations into textual rationales, providing spatially grounded and contextually rich explanations. Additionally, we introduce Textual Supervision on Visual Explanations to align visual explanations with textual rationales, even in cases where ground truth visual annotations are missing. A Visual Explanation Distribution Consistency loss further reinforces visual coherence by aligning the generated visual explanations with dataset-level patterns, enabling the model to effectively learn from incomplete multimodal supervision. We validate MEGL on two new datasets, Object-ME and Action-ME, for image classification with multimodal explanations. Experimental results demonstrate that MEGL outperforms previous approaches in prediction accuracy and explanation quality across both visual and textual domains. Our code will be made available upon the acceptance of the paper.

Title: Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

• Authors: Jared Fernandez, Luca Wehrstedt, Leonid Shamis, Mostafa Elhoushi, Kalyan Saladi, Yonatan Bisk, Emma Strubell, Jacob Kahn
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.13055
• Pdf URL: https://arxiv.org/pdf/2411.13055
• Copy Paste: [[2411.13055]] Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training(https://arxiv.org/abs/2411.13055)
• Keywords: large language model
• Abstract: Dramatic increases in the capabilities of neural network models in recent years are driven by scaling model size, training data, and corresponding computational resources. To develop the exceedingly large networks required in modern applications, such as large language models (LLMs), model training is distributed across tens of thousands of hardware accelerators (e.g. GPUs), requiring orchestration of computation and communication across large computing clusters. In this work, we demonstrate that careful consideration of hardware configuration and parallelization strategy is critical for effective (i.e. compute- and cost-efficient) scaling of model size, training data, and total computation. We conduct an extensive empirical study of the performance of large-scale LLM training workloads across model size, hardware configurations, and distributed parallelization strategies. We demonstrate that: (1) beyond certain scales, overhead incurred from certain distributed communication strategies leads parallelization strategies previously thought to be sub-optimal in fact become preferable; and (2) scaling the total number of accelerators for large model training quickly yields diminishing returns even when hardware and parallelization strategies are properly optimized, implying poor marginal performance per additional unit of power or GPU-hour.

Title: Efficient Masked AutoEncoder for Video Object Counting and A Large-Scale Benchmark

• Authors: Bing Cao, Quanhao Lu, Jiekang Feng, Pengfei Zhu, Qinghua Hu, Qilong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13056
• Pdf URL: https://arxiv.org/pdf/2411.13056
• Copy Paste: [[2411.13056]] Efficient Masked AutoEncoder for Video Object Counting and A Large-Scale Benchmark(https://arxiv.org/abs/2411.13056)
• Keywords: protect
• Abstract: The dynamic imbalance of the fore-background is a major challenge in video object counting, which is usually caused by the sparsity of foreground objects. This often leads to severe under- and over-prediction problems and has been less studied in existing works. To tackle this issue in video object counting, we propose a density-embedded Efficient Masked Autoencoder Counting (E-MAC) framework in this paper. To effectively capture the dynamic variations across frames, we utilize an optical flow-based temporal collaborative fusion that aligns features to derive multi-frame density residuals. The counting accuracy of the current frame is boosted by harnessing the information from adjacent frames. More importantly, to empower the representation ability of dynamic foreground objects for intra-frame, we first take the density map as an auxiliary modality to perform $\mathtt{D}$ensity-$\mathtt{E}$mbedded $\mathtt{M}$asked m$\mathtt{O}$deling ($\mathtt{DEMO}$) for multimodal self-representation learning to regress density map. However, as $\mathtt{DEMO}$ contributes effective cross-modal regression guidance, it also brings in redundant background information and hard to focus on foreground regions. To handle this dilemma, we further propose an efficient spatial adaptive masking derived from density maps to boost efficiency. In addition, considering most existing datasets are limited to human-centric scenarios, we first propose a large video bird counting dataset $\textit{DroneBird}$, in natural scenarios for migratory bird protection. Extensive experiments on three crowd datasets and our $\textit{DroneBird}$ validate our superiority against the counterparts.

Title: Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation

• Authors: Rohith Peddi, Saurabh, Ayush Abhay Shrivastava, Parag Singla, Vibhav Gogate
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13059
• Pdf URL: https://arxiv.org/pdf/2411.13059
• Copy Paste: [[2411.13059]] Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation(https://arxiv.org/abs/2411.13059)
• Keywords: robust
• Abstract: Spatio-Temporal Scene Graphs (STSGs) provide a concise and expressive representation of dynamic scenes by modelling objects and their evolving relationships over time. However, real-world visual relationships often exhibit a long-tailed distribution, causing existing methods for tasks like Video Scene Graph Generation (VidSGG) and Scene Graph Anticipation (SGA) to produce biased scene graphs. To this end, we propose ImparTail, a novel training framework that leverages curriculum learning and loss masking to mitigate bias in the generation and anticipation of spatio-temporal scene graphs. Our approach gradually decreases the dominance of the head relationship classes during training and focuses more on tail classes, leading to more balanced training. Furthermore, we introduce two new tasks, Robust Spatio-Temporal Scene Graph Generation and Robust Scene Graph Anticipation, designed to evaluate the robustness of STSG models against distribution shifts. Extensive experiments on the Action Genome dataset demonstrate that our framework significantly enhances the unbiased performance and robustness of STSG models compared to existing methods.

Title: Automatic marker-free registration based on similar tetrahedras for single-tree point clouds

• Authors: Jing Ren, Pei Wang, Hanlong Li, Yuhan Wu, Yuhang Gao, Wenxin Chen, Mingtai Zhang, Lingyun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13069
• Pdf URL: https://arxiv.org/pdf/2411.13069
• Copy Paste: [[2411.13069]] Automatic marker-free registration based on similar tetrahedras for single-tree point clouds(https://arxiv.org/abs/2411.13069)
• Keywords: robust
• Abstract: In recent years, terrestrial laser scanning technology has been widely used to collect tree point cloud data, aiding in measurements of diameter at breast height, biomass, and other forestry survey data. Since a single scan from terrestrial laser systems captures data from only one angle, multiple scans must be registered and fused to obtain complete tree point cloud data. This paper proposes a marker-free automatic registration method for single-tree point clouds based on similar tetrahedras. First, two point clouds from two scans of the same tree are used to generate tree skeletons, and key point sets are constructed from these skeletons. Tetrahedra are then filtered and matched according to similarity principles, with the vertices of these two matched tetrahedras selected as matching point pairs, thus completing the coarse registration of the point clouds from the two scans. Subsequently, the ICP method is applied to the coarse-registered leaf point clouds to obtain fine registration parameters, completing the precise registration of the two tree point clouds. Experiments were conducted using terrestrial laser scanning data from eight trees, each from different species and with varying shapes. The proposed method was evaluated using RMSE and Hausdorff distance, compared against the traditional ICP and NDT methods. The experimental results demonstrate that the proposed method significantly outperforms both ICP and NDT in registration accuracy, achieving speeds up to 593 times and 113 times faster than ICP and NDT, respectively. In summary, the proposed method shows good robustness in single-tree point cloud registration, with significant advantages in accuracy and speed compared to traditional ICP and NDT methods, indicating excellent application prospects in practical registration scenarios.

Title: Practical Compact Deep Compressed Sensing

• Authors: Bin Chen, Jian Zhang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.13081
• Pdf URL: https://arxiv.org/pdf/2411.13081
• Copy Paste: [[2411.13081]] Practical Compact Deep Compressed Sensing(https://arxiv.org/abs/2411.13081)
• Keywords: extraction, interpretability
• Abstract: Recent years have witnessed the success of deep networks in compressed sensing (CS), which allows for a significant reduction in sampling cost and has gained growing attention since its inception. In this paper, we propose a new practical and compact network dubbed PCNet for general image CS. Specifically, in PCNet, a novel collaborative sampling operator is designed, which consists of a deep conditional filtering step and a dual-branch fast sampling step. The former learns an implicit representation of a linear transformation matrix into a few convolutions and first performs adaptive local filtering on the input image, while the latter then uses a discrete cosine transform and a scrambled block-diagonal Gaussian matrix to generate under-sampled measurements. Our PCNet is equipped with an enhanced proximal gradient descent algorithm-unrolled network for reconstruction. It offers flexibility, interpretability, and strong recovery performance for arbitrary sampling rates once trained. Additionally, we provide a deployment-oriented extraction scheme for single-pixel CS imaging systems, which allows for the convenient conversion of any linear sampling operator to its matrix form to be loaded onto hardware like digital micro-mirror devices. Extensive experiments on natural image CS, quantized CS, and self-supervised CS demonstrate the superior reconstruction accuracy and generalization ability of PCNet compared to existing state-of-the-art methods, particularly for high-resolution images. Code is available at this https URL.

Title: Patience Is The Key to Large Language Model Reasoning

• Authors: Yijiong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13082
• Pdf URL: https://arxiv.org/pdf/2411.13082
• Copy Paste: [[2411.13082]] Patience Is The Key to Large Language Model Reasoning(https://arxiv.org/abs/2411.13082)
• Keywords: large language model
• Abstract: Recent advancements in the field of large language models, particularly through the Chain of Thought (CoT) approach, have demonstrated significant improvements in solving complex problems. However, existing models either tend to sacrifice detailed reasoning for brevity due to user preferences, or require extensive and expensive training data to learn complicated reasoning ability, limiting their potential in solving complex tasks. To bridge this gap, following the concept of scaling test-time, we propose a simple method by encouraging models to adopt a more patient reasoning style without the need of introducing new knowledge or skills. To employ a preference optimization approach, we generate detailed reasoning processes as positive examples and simple answers as negative examples, thereby training the model to favor thoroughness in its responses. Our results demonstrate a performance increase of up to 6.7% on GSM8k with training just on a lightweight dataset.

Title: DriveMLLM: A Benchmark for Spatial Understanding with Multimodal Large Language Models in Autonomous Driving

• Authors: Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Chenming Zhang, Shuai Liu, Long Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13112
• Pdf URL: https://arxiv.org/pdf/2411.13112
• Copy Paste: [[2411.13112]] DriveMLLM: A Benchmark for Spatial Understanding with Multimodal Large Language Models in Autonomous Driving(https://arxiv.org/abs/2411.13112)
• Keywords: large language model
• Abstract: Autonomous driving requires a comprehensive understanding of 3D environments to facilitate high-level tasks such as motion prediction, planning, and mapping. In this paper, we introduce DriveMLLM, a benchmark specifically designed to evaluate the spatial understanding capabilities of multimodal large language models (MLLMs) in autonomous driving. DriveMLLM includes 2,734 front-facing camera images and introduces both absolute and relative spatial reasoning tasks, accompanied by linguistically diverse natural language questions. To measure MLLMs' performance, we propose novel evaluation metrics focusing on spatial understanding. We evaluate several state-of-the-art MLLMs on DriveMLLM, and our results reveal the limitations of current models in understanding complex spatial relationships in driving contexts. We believe these findings underscore the need for more advanced MLLM-based spatial reasoning methods and highlight the potential for DriveMLLM to drive further research in autonomous driving. Code will be available at \url{this https URL}.

Title: Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning

• Authors: Zhi Luo, Xiyuan Yang, Pan Zhou, Di Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13116
• Pdf URL: https://arxiv.org/pdf/2411.13116
• Copy Paste: [[2411.13116]] Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning(https://arxiv.org/abs/2411.13116)
• Keywords: attack
• Abstract: Manipulating the interaction trajectories between the intelligent agent and the environment can control the agent's training and behavior, exposing the potential vulnerabilities of reinforcement learning (RL). For example, in Cyber-Physical Systems (CPS) controlled by RL, the attacker can manipulate the actions of the adopted RL to other actions during the training phase, which will lead to bad consequences. Existing work has studied action-manipulation attacks in tabular settings, where the states and actions are discrete. As seen in many up-and-coming RL applications, such as autonomous driving, continuous action space is widely accepted, however, its action-manipulation attacks have not been thoroughly investigated yet. In this paper, we consider this crucial problem in both white-box and black-box scenarios. Specifically, utilizing the knowledge derived exclusively from trajectories, we propose a black-box attack algorithm named LCBT, which uses the Monte Carlo tree search method for efficient action searching and manipulation. Additionally, we demonstrate that for an agent whose dynamic regret is sub-linearly related to the total number of steps, LCBT can teach the agent to converge to target policies with only sublinear attack cost, i.e., $O\left(\mathcal{R}(T) + MH^3K^E\log (MT)\right)(0

Title: Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders

• Authors: Charles O'Neill, David Klindt
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13117
• Pdf URL: https://arxiv.org/pdf/2411.13117
• Copy Paste: [[2411.13117]] Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders(https://arxiv.org/abs/2411.13117)
• Keywords: interpretability, large language model
• Abstract: A recent line of work has shown promise in using sparse autoencoders (SAEs) to uncover interpretable features in neural network representations. However, the simple linear-nonlinear encoding mechanism in SAEs limits their ability to perform accurate sparse inference. In this paper, we investigate sparse inference and learning in SAEs through the lens of sparse coding. Specifically, we show that SAEs perform amortised sparse inference with a computationally restricted encoder and, using compressed sensing theory, we prove that this mapping is inherently insufficient for accurate sparse inference, even in solvable cases. Building on this theory, we empirically explore conditions where more sophisticated sparse inference methods outperform traditional SAE encoders. Our key contribution is the decoupling of the encoding and decoding processes, which allows for a comparison of various sparse encoding strategies. We evaluate these strategies on two dimensions: alignment with true underlying sparse features and correct inference of sparse codes, while also accounting for computational costs during training and inference. Our results reveal that substantial performance gains can be achieved with minimal increases in compute cost. We demonstrate that this generalises to SAEs applied to large language models (LLMs), where advanced encoders achieve similar interpretability. This work opens new avenues for understanding neural network representations and offers important implications for improving the tools we use to analyse the activations of large language models.

Title: Virtual Staining of Label-Free Tissue in Imaging Mass Spectrometry

• Authors: Yijie Zhang, Luzhe Huang, Nir Pillar, Yuzhu Li, Lukasz G. Migas, Raf Van de Plas, Jeffrey M. Spraggins, Aydogan Ozcan
• Subjects: cs.CV, cs.LG, physics.med-ph, physics.optics
• Abstract URL: https://arxiv.org/abs/2411.13120
• Pdf URL: https://arxiv.org/pdf/2411.13120
• Copy Paste: [[2411.13120]] Virtual Staining of Label-Free Tissue in Imaging Mass Spectrometry(https://arxiv.org/abs/2411.13120)
• Keywords: diffusion
• Abstract: Imaging mass spectrometry (IMS) is a powerful tool for untargeted, highly multiplexed molecular mapping of tissue in biomedical research. IMS offers a means of mapping the spatial distributions of molecular species in biological tissue with unparalleled chemical specificity and sensitivity. However, most IMS platforms are not able to achieve microscopy-level spatial resolution and lack cellular morphological contrast, necessitating subsequent histochemical staining, microscopic imaging and advanced image registration steps to enable molecular distributions to be linked to specific tissue features and cell types. Here, we present a virtual histological staining approach that enhances spatial resolution and digitally introduces cellular morphological contrast into mass spectrometry images of label-free human tissue using a diffusion model. Blind testing on human kidney tissue demonstrated that the virtually stained images of label-free samples closely match their histochemically stained counterparts (with Periodic Acid-Schiff staining), showing high concordance in identifying key renal pathology structures despite utilizing IMS data with 10-fold larger pixel size. Additionally, our approach employs an optimized noise sampling technique during the diffusion model's inference process to reduce variance in the generated images, yielding reliable and repeatable virtual staining. We believe this virtual staining method will significantly expand the applicability of IMS in life sciences and open new avenues for mass spectrometry-based biomedical research.

Title: Adapting Vision Foundation Models for Robust Cloud Segmentation in Remote Sensing Images

• Authors: Xuechao Zou, Shun Zhang, Kai Li, Shiying Wang, Junliang Xing, Lei Jin, Congyan Lang, Pin Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13127
• Pdf URL: https://arxiv.org/pdf/2411.13127
• Copy Paste: [[2411.13127]] Adapting Vision Foundation Models for Robust Cloud Segmentation in Remote Sensing Images(https://arxiv.org/abs/2411.13127)
• Keywords: robust, transformer, segmentation
• Abstract: Cloud segmentation is a critical challenge in remote sensing image interpretation, as its accuracy directly impacts the effectiveness of subsequent data processing and analysis. Recently, vision foundation models (VFM) have demonstrated powerful generalization capabilities across various visual tasks. In this paper, we present a parameter-efficient adaptive approach, termed Cloud-Adapter, designed to enhance the accuracy and robustness of cloud segmentation. Our method leverages a VFM pretrained on general domain data, which remains frozen, eliminating the need for additional training. Cloud-Adapter incorporates a lightweight spatial perception module that initially utilizes a convolutional neural network (ConvNet) to extract dense spatial representations. These multi-scale features are then aggregated and serve as contextual inputs to an adapting module, which modulates the frozen transformer layers within the VFM. Experimental results demonstrate that the Cloud-Adapter approach, utilizing only 0.6% of the trainable parameters of the frozen backbone, achieves substantial performance gains. Cloud-Adapter consistently attains state-of-the-art (SOTA) performance across a wide variety of cloud segmentation datasets from multiple satellite sources, sensor series, data processing levels, land cover scenarios, and annotation granularities. We have released the source code and pretrained models at this https URL to support further research.

Title: TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models

• Authors: Xin Wang, Kai Chen, Jiaming Zhang, Jingjing Chen, Xingjun Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13136
• Pdf URL: https://arxiv.org/pdf/2411.13136
• Copy Paste: [[2411.13136]] TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models(https://arxiv.org/abs/2411.13136)
• Keywords: defense, attack, robust
• Abstract: Large pre-trained Vision-Language Models (VLMs) such as CLIP have demonstrated excellent zero-shot generalizability across various downstream tasks. However, recent studies have shown that the inference performance of CLIP can be greatly degraded by small adversarial perturbations, especially its visual modality, posing significant safety threats. To mitigate this vulnerability, in this paper, we propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks. TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP. Specifically, it is an unsupervised method that optimizes the defensive prompts for each test sample by minimizing a multi-view entropy and aligning adversarial-clean distributions. We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets, demonstrating that it enhances the zero-shot adversarial robustness of the original CLIP by at least 48.9% against AutoAttack (AA), while largely maintaining performance on clean examples. Moreover, TAPT outperforms existing adversarial prompt tuning methods across various backbones, achieving an average robustness improvement of at least 36.6%.

Title: SAGA: Synthetic Audit Log Generation for APT Campaigns

• Authors: Yi-Ting Huang, Ying-Ren Guo, Yu-Sheng Yang, Guo-Wei Wong, Yu-Zih Jheng, Yeali Sun, Jessemyn Modini, Timothy Lynar, Meng Chang Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.13138
• Pdf URL: https://arxiv.org/pdf/2411.13138
• Copy Paste: [[2411.13138]] SAGA: Synthetic Audit Log Generation for APT Campaigns(https://arxiv.org/abs/2411.13138)
• Keywords: attack, steal
• Abstract: With the increasing sophistication of Advanced Persistent Threats (APTs), the demand for effective detection and mitigation strategies and methods has escalated. Program execution leaves traces in the system audit log, which can be analyzed to detect malicious activities. However, collecting and analyzing large volumes of audit logs over extended periods is challenging, further compounded by insufficient labeling that hinders their usability. Addressing these challenges, this paper introduces SAGA (Synthetic Audit log Generation for APT campaigns), a novel approach for generating find-grained labeled synthetic audit logs that mimic real-world system logs while embedding stealthy APT attacks. SAGA generates configurable audit logs for arbitrary duration, blending benign logs from normal operations with malicious logs based on the definitions the MITRE ATT\&CK framework. Malicious audit logs follow an APT lifecycle, incorporating various attack techniques at each stage. These synthetic logs can serve as benchmark datasets for training machine learning models and assessing diverse APT detection methods. To demonstrate the usefulness of synthetic audit logs, we ran established baselines of event-based technique hunting and APT campaign detection using various synthetic audit logs. In addition, we show that a deep learning model trained on synthetic audit logs can detect previously unseen techniques within audit logs.

Title: CopyrightMeter: Revisiting Copyright Protection in Text-to-image Models

• Authors: Naen Xu, Changjiang Li, Tianyu Du, Minxi Li, Wenjie Luo, Jiacheng Liang, Yuyuan Li, Xuhong Zhang, Meng Han, Jianwei Yin, Ting Wang
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13144
• Pdf URL: https://arxiv.org/pdf/2411.13144
• Copy Paste: [[2411.13144]] CopyrightMeter: Revisiting Copyright Protection in Text-to-image Models(https://arxiv.org/abs/2411.13144)
• Keywords: protect, attack, robust, fair, watermark, diffusion
• Abstract: Text-to-image diffusion models have emerged as powerful tools for generating high-quality images from textual descriptions. However, their increasing popularity has raised significant copyright concerns, as these models can be misused to reproduce copyrighted content without authorization. In response, recent studies have proposed various copyright protection methods, including adversarial perturbation, concept erasure, and watermarking techniques. However, their effectiveness and robustness against advanced attacks remain largely unexplored. Moreover, the lack of unified evaluation frameworks has hindered systematic comparison and fair assessment of different approaches. To bridge this gap, we systematize existing copyright protection methods and attacks, providing a unified taxonomy of their design spaces. We then develop CopyrightMeter, a unified evaluation framework that incorporates 17 state-of-the-art protections and 16 representative attacks. Leveraging CopyrightMeter, we comprehensively evaluate protection methods across multiple dimensions, thereby uncovering how different design choices impact fidelity, efficacy, and resilience under attacks. Our analysis reveals several key findings: (i) most protections (16/17) are not resilient against attacks; (ii) the "best" protection varies depending on the target priority; (iii) more advanced attacks significantly promote the upgrading of protections. These insights provide concrete guidance for developing more robust protection methods, while its unified evaluation protocol establishes a standard benchmark for future copyright protection research in text-to-image generation.

Title: GraphCL: Graph-based Clustering for Semi-Supervised Medical Image Segmentation

• Authors: Mengzhu Wang, Jiao Li, Houcheng Su, Nan Yin, Shen Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13147
• Pdf URL: https://arxiv.org/pdf/2411.13147
• Copy Paste: [[2411.13147]] GraphCL: Graph-based Clustering for Semi-Supervised Medical Image Segmentation(https://arxiv.org/abs/2411.13147)
• Keywords: segmentation
• Abstract: Semi-supervised learning (SSL) has made notable advancements in medical image segmentation (MIS), particularly in scenarios with limited labeled data and significantly enhancing data utilization efficiency. Previous methods primarily focus on complex training strategies to utilize unlabeled data but neglect the importance of graph structural information. Different from existing methods, we propose a graph-based clustering for semi-supervised medical image segmentation (GraphCL) by jointly modeling graph data structure in a unified deep model. The proposed GraphCL model enjoys several advantages. Firstly, to the best of our knowledge, this is the first work to model the data structure information for semi-supervised medical image segmentation (SSMIS). Secondly, to get the clustered features across different graphs, we integrate both pairwise affinities between local image features and raw features as inputs. Extensive experimental results on three standard benchmarks show that the proposed GraphCL algorithm outperforms state-of-the-art semi-supervised medical image segmentation methods.

Title: YCB-LUMA: YCB Object Dataset with Luminance Keying for Object Localization

• Authors: Thomas Pöllabauer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13149
• Pdf URL: https://arxiv.org/pdf/2411.13149
• Copy Paste: [[2411.13149]] YCB-LUMA: YCB Object Dataset with Luminance Keying for Object Localization(https://arxiv.org/abs/2411.13149)
• Keywords: segmentation
• Abstract: Localizing target objects in images is an important task in computer vision. Often it is the first step towards solving a variety of applications in autonomous driving, maintenance, quality insurance, robotics, and augmented reality. Best in class solutions for this task rely on deep neural networks, which require a set of representative training data for best performance. Creating sets of sufficient quality, variety, and size is often difficult, error prone, and expensive. This is where the method of luminance keying can help: it provides a simple yet effective solution to record high quality data for training object detection and segmentation. We extend previous work that presented luminance keying on the common YCB-V set of household objects by recording the remaining objects of the YCB superset. The additional variety of objects - addition of transparency, multiple color variations, non-rigid objects - further demonstrates the usefulness of luminance keying and might be used to test the applicability of the approach on new 2D object detection and segmentation algorithms.

Title: RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generation

• Authors: Christoph Reinders, Radu Berdan, Beril Besbinar, Junji Otsuka, Daisuke Iso
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13150
• Pdf URL: https://arxiv.org/pdf/2411.13150
• Copy Paste: [[2411.13150]] RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generation(https://arxiv.org/abs/2411.13150)
• Keywords: extraction, diffusion
• Abstract: Current deep learning approaches in computer vision primarily focus on RGB data sacrificing information. In contrast, RAW images offer richer representation, which is crucial for precise recognition, particularly in challenging conditions like low-light environments. The resultant demand for comprehensive RAW image datasets contrasts with the labor-intensive process of creating specific datasets for individual sensors. To address this, we propose a novel diffusion-based method for generating RAW images guided by RGB images. Our approach integrates an RGB-guidance module for feature extraction from RGB inputs, then incorporates these features into the reverse diffusion process with RGB-guided residual blocks across various resolutions. This approach yields high-fidelity RAW images, enabling the creation of camera-specific RAW datasets. Our RGB2RAW experiments on four DSLR datasets demonstrate state-of-the-art performance. Moreover, RAW-Diffusion demonstrates exceptional data efficiency, achieving remarkable performance with as few as 25 training samples or even fewer. We extend our method to create BDD100K-RAW and Cityscapes-RAW datasets, revealing its effectiveness for object detection in RAW imagery, significantly reducing the amount of required RAW images.

Title: Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding

• Authors: Hyun Ryu, Eric Kim
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13157
• Pdf URL: https://arxiv.org/pdf/2411.13157
• Copy Paste: [[2411.13157]] Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding(https://arxiv.org/abs/2411.13157)
• Keywords: large language model
• Abstract: Efficient inference in large language models (LLMs) has become a critical focus as their scale and complexity grow. Traditional autoregressive decoding, while effective, suffers from computational inefficiencies due to its sequential token generation process. Speculative decoding addresses this bottleneck by introducing a two-stage framework: drafting and verification. A smaller, efficient model generates a preliminary draft, which is then refined by a larger, more sophisticated model. This paper provides a comprehensive survey of speculative decoding methods, categorizing them into draft-centric and model-centric approaches. We discuss key ideas associated with each method, highlighting their potential for scaling LLM inference. This survey aims to guide future research in optimizing speculative decoding and its integration into real-world LLM applications.

Title: Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM

• Authors: Jiawei Yu, Yuang Li, Xiaosong Qiao, Huan Zhao, Xiaofeng Zhao, Wei Tang, Min Zhang, Hao Yang, Jinsong Su
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.13159
• Pdf URL: https://arxiv.org/pdf/2411.13159
• Copy Paste: [[2411.13159]] Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM(https://arxiv.org/abs/2411.13159)
• Keywords: large language model
• Abstract: Text-to-speech (TTS) models have been widely adopted to enhance automatic speech recognition (ASR) systems using text-only corpora, thereby reducing the cost of labeling real speech data. Existing research primarily utilizes additional text data and predefined speech styles supported by TTS models. In this paper, we propose Hard-Synth, a novel ASR data augmentation method that leverages large language models (LLMs) and advanced zero-shot TTS. Our approach employs LLMs to generate diverse in-domain text through rewriting, without relying on additional text data. Rather than using predefined speech styles, we introduce a hard prompt selection method with zero-shot TTS to clone speech styles that the ASR model finds challenging to recognize. Experiments demonstrate that Hard-Synth significantly enhances the Conformer model, achieving relative word error rate (WER) reductions of 6.5\%/4.4\% on LibriSpeech dev/test-other subsets. Additionally, we show that Hard-Synth is data-efficient and capable of reducing bias in ASR.

Title: Unlocking Historical Clinical Trial Data with ALIGN: A Compositional Large Language Model System for Medical Coding

• Authors: Nabeel Seedat, Caterina Tozzi, Andrea Hita Ardiaca, Mihaela van der Schaar, James Weatherall, Adam Taylor
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13163
• Pdf URL: https://arxiv.org/pdf/2411.13163
• Copy Paste: [[2411.13163]] Unlocking Historical Clinical Trial Data with ALIGN: A Compositional Large Language Model System for Medical Coding(https://arxiv.org/abs/2411.13163)
• Keywords: large language model
• Abstract: The reuse of historical clinical trial data has significant potential to accelerate medical research and drug development. However, interoperability challenges, particularly with missing medical codes, hinders effective data integration across studies. While Large Language Models (LLMs) offer a promising solution for automated coding without labeled data, current approaches face challenges on complex coding tasks. We introduce ALIGN, a novel compositional LLM-based system for automated, zero-shot medical coding. ALIGN follows a three-step process: (1) diverse candidate code generation; (2) self-evaluation of codes and (3) confidence scoring and uncertainty estimation enabling human deferral to ensure reliability. We evaluate ALIGN on harmonizing medication terms into Anatomical Therapeutic Chemical (ATC) and medical history terms into Medical Dictionary for Regulatory Activities (MedDRA) codes extracted from 22 immunology trials. ALIGN outperformed the LLM baselines, while also providing capabilities for trustworthy deployment. For MedDRA coding, ALIGN achieved high accuracy across all levels, matching RAG and excelling at the most specific levels (87-90% for HLGT). For ATC coding, ALIGN demonstrated superior performance, particularly at lower hierarchy levels (ATC Level 4), with 72-73% overall accuracy and 86-89% accuracy for common medications, outperforming baselines by 7-22%. ALIGN's uncertainty-based deferral improved accuracy by 17% to 90% accuracy with 30% deferral, notably enhancing performance on uncommon medications. ALIGN achieves this cost-efficiently at \$0.0007 and \$0.02 per code for GPT-4o-mini and GPT-4o, reducing barriers to clinical adoption. ALIGN advances automated medical coding for clinical trial data, contributing to enhanced data interoperability and reusability, positioning it as a promising tool to improve clinical research and accelerate drug development.

Title: Cross-Camera Distracted Driver Classification through Feature Disentanglement and Contrastive Learning

• Authors: Simone Bianco, Luigi Celona, Paolo Napoletano
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.13181
• Pdf URL: https://arxiv.org/pdf/2411.13181
• Copy Paste: [[2411.13181]] Cross-Camera Distracted Driver Classification through Feature Disentanglement and Contrastive Learning(https://arxiv.org/abs/2411.13181)
• Keywords: robust
• Abstract: The classification of distracted drivers is pivotal for ensuring safe driving. Previous studies demonstrated the effectiveness of neural networks in automatically predicting driver distraction, fatigue, and potential hazards. However, recent research has uncovered a significant loss of accuracy in these models when applied to samples acquired under conditions that differ from the training data. In this paper, we introduce a robust model designed to withstand changes in camera position within the vehicle. Our Driver Behavior Monitoring Network (DBMNet) relies on a lightweight backbone and integrates a disentanglement module to discard camera view information from features, coupled with contrastive learning to enhance the encoding of various driver actions. Experiments conducted on the daytime and nighttime subsets of the 100-Driver dataset validate the effectiveness of our approach with an increment on average of 9\% in Top-1 accuracy in comparison with the state of the art. In addition, cross-dataset and cross-camera experiments conducted on three benchmark datasets, namely AUCDD-V1, EZZ2021 and SFD, demonstrate the superior generalization capability of the proposed method.

Title: Click; Single Object Tracking; Video Object Segmentation; Real-time Interaction

• Authors: Kuiran Wang, Xuehui Yu, Wenwen Yu, Guorong Li, Xiangyuan Lan, Qixiang Ye, Jianbin Jiao, Zhenjun Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13183
• Pdf URL: https://arxiv.org/pdf/2411.13183
• Copy Paste: [[2411.13183]] Click; Single Object Tracking; Video Object Segmentation; Real-time Interaction(https://arxiv.org/abs/2411.13183)
• Keywords: segmentation
• Abstract: Single object tracking(SOT) relies on precise object bounding box initialization. In this paper, we reconsidered the deficiencies in the current approaches to initializing single object trackers and propose a new paradigm for single object tracking algorithms, ClickTrack, a new paradigm using clicking interaction for real-time scenarios. Moreover, click as an input type inherently lack hierarchical information. To address ambiguity in certain special scenarios, we designed the Guided Click Refiner(GCR), which accepts point and optional textual information as inputs, transforming the point into the bounding box expected by the operator. The bounding box will be used as input of single object trackers. Experiments on LaSOT and GOT-10k benchmarks show that tracker combined with GCR achieves stable performance in real-time interactive scenarios. Furthermore, we explored the integration of GCR into the Segment Anything model(SAM), significantly reducing ambiguity issues when SAM receives point inputs.

Title: Engagement-Driven Content Generation with Large Language Models

• Authors: Erica Coppolillo, Marco Minici, Federico Cinus, Francesco Bonchi, Giuseppe Manco
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13187
• Pdf URL: https://arxiv.org/pdf/2411.13187
• Copy Paste: [[2411.13187]] Engagement-Driven Content Generation with Large Language Models(https://arxiv.org/abs/2411.13187)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit significant persuasion capabilities in one-on-one interactions, but their influence within social networks remains underexplored. This study investigates the potential social impact of LLMs in these environments, where interconnected users and complex opinion dynamics pose unique challenges. In particular, we address the following research question: can LLMs learn to generate meaningful content that maximizes user engagement on social networks? To answer this question, we define a pipeline to guide the LLM-based content generation which employs reinforcement learning with simulated feedback. In our framework, the reward is based on an engagement model borrowed from the literature on opinion dynamics and information propagation. Moreover, we force the text generated by the LLM to be aligned with a given topic and to satisfy a minimum fluency requirement. Using our framework, we analyze the capabilities and limitations of LLMs in tackling the given task, specifically considering the relative positions of the LLM as an agent within the social network and the distribution of opinions in the network on the given topic. Our findings show the full potential of LLMs in creating social engagement. Notable properties of our approach are that the learning procedure is adaptive to the opinion distribution of the underlying network and agnostic to the specifics of the engagement model, which is embedded as a plug-and-play component. In this regard, our approach can be easily refined for more complex engagement tasks and interventions in computational social science. The code used for the experiments is publicly available at this https URL.

Title: The Information Security Awareness of Large Language Models

• Authors: Ofir Cohen, Gil Ari Agmon, Asaf Shabtai, Rami Puzis
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13207
• Pdf URL: https://arxiv.org/pdf/2411.13207
• Copy Paste: [[2411.13207]] The Information Security Awareness of Large Language Models(https://arxiv.org/abs/2411.13207)
• Keywords: security, large language model
• Abstract: The popularity of large language models (LLMs) continues to increase, and LLM-based assistants have become ubiquitous, assisting people of diverse backgrounds in many aspects of life. Significant resources have been invested in the safety of LLMs and their alignment with social norms. However, research examining their behavior from the information security awareness (ISA) perspective is lacking. Chatbots and LLM-based assistants may put unwitting users in harm's way by facilitating unsafe behavior. We observe that the ISA inherent in some of today's most popular LLMs varies significantly, with most models requiring user prompts with a clear security context to utilize their security knowledge and provide safe responses to users. Based on this observation, we created a comprehensive set of 30 scenarios to assess the ISA of LLMs. These scenarios benchmark the evaluated models with respect to all focus areas defined in a mobile ISA taxonomy. Among our findings is that ISA is mildly affected by changing the model's temperature, whereas adjusting the system prompt can substantially impact it. This underscores the necessity of setting the right system prompt to mitigate ISA weaknesses. Our findings also highlight the importance of ISA assessment for the development of future LLM-based assistants.

Title: AIDBench: A benchmark for evaluating the authorship identification capability of large language models

• Authors: Zichen Wen, Dadi Guo, Huishuai Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13226
• Pdf URL: https://arxiv.org/pdf/2411.13226
• Copy Paste: [[2411.13226]] AIDBench: A benchmark for evaluating the authorship identification capability of large language models(https://arxiv.org/abs/2411.13226)
• Keywords: privacy, large language model
• Abstract: As large language models (LLMs) rapidly advance and integrate into daily life, the privacy risks they pose are attracting increasing attention. We focus on a specific privacy risk where LLMs may help identify the authorship of anonymous texts, which challenges the effectiveness of anonymity in real-world systems such as anonymous peer review systems. To investigate these risks, we present AIDBench, a new benchmark that incorporates several author identification datasets, including emails, blogs, reviews, articles, and research papers. AIDBench utilizes two evaluation methods: one-to-one authorship identification, which determines whether two texts are from the same author; and one-to-many authorship identification, which, given a query text and a list of candidate texts, identifies the candidate most likely written by the same author as the query text. We also introduce a Retrieval-Augmented Generation (RAG)-based method to enhance the large-scale authorship identification capabilities of LLMs, particularly when input lengths exceed the models' context windows, thereby establishing a new baseline for authorship identification using LLMs. Our experiments with AIDBench demonstrate that LLMs can correctly guess authorship at rates well above random chance, revealing new privacy risks posed by these powerful models. The source code and data will be made publicly available after acceptance.

Title: BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework

• Authors: Xu Zou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13237
• Pdf URL: https://arxiv.org/pdf/2411.13237
• Copy Paste: [[2411.13237]] BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework(https://arxiv.org/abs/2411.13237)
• Keywords: generative
• Abstract: Recently, generative pre-trained models have made significant strides, particularly highlighted by the release of ChatGPT and GPT-4, which exhibit superior cross-domain capabilities. However, these models still face challenges on constrained writing tasks like poem generation under open-domain titles. In response to this challenge, we introduce Block Inverse Prompting (BIPro) constrained generation framework. BIPro leverages two block inverse prompting methods, revise and rewrite, that mimic the process of human text writing using block generative models. It significantly improves the zero-shot generation quality on the formidable constrained generation task of open-domain traditional-form Chinese poem generation. Based on a less powerful block generative model GLM-10B-Chinese, poems composed via BIPro without priming or additional training outperform both most advanced direct generative systems like GPT-4 or GLM-4 and best domain-specific systems such as Yusheng, Shisanbai, or Baidu Poetry Helper in human evaluation by proficient poets. Finally, BIPro considerably narrows the gap between AI-generated works and short-listed human literary arts in another human evaluation, unveiling the promising potential of block generative models in improving the quality of constrained generation.

Title: XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation

• Authors: Ziyi Wang, Yanbo Wang, Xumin Yu, Jie Zhou, Jiwen Lu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13243
• Pdf URL: https://arxiv.org/pdf/2411.13243
• Copy Paste: [[2411.13243]] XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation(https://arxiv.org/abs/2411.13243)
• Keywords: diffusion, segmentation
• Abstract: Existing methodologies in open vocabulary 3D semantic segmentation primarily concentrate on establishing a unified feature space encompassing 3D, 2D, and textual modalities. Nevertheless, traditional techniques such as global feature alignment or vision-language model distillation tend to impose only approximate correspondence, struggling notably with delineating fine-grained segmentation boundaries. To address this gap, we propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. In our approach, we developed a mask generator based on the denoising UNet from a pre-trained diffusion model, leveraging its capability for precise textual control over dense pixel representations and enhancing the open-world adaptability of the generated masks. We further integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks with additional 3D geometry awareness. Subsequently, the generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings. Finally, we fuse complementary 2D and 3D mask features, resulting in competitive performance across multiple benchmarks for 3D open vocabulary semantic segmentation. Code is available at this https URL.

Title: Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL

• Authors: Zhibo Chu, Zichong Wang, Qitao Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13244
• Pdf URL: https://arxiv.org/pdf/2411.13244
• Copy Paste: [[2411.13244]] Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL(https://arxiv.org/abs/2411.13244)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit impressive problem-solving skills across many tasks, but they still underperform compared to humans in various downstream applications, such as text-to-SQL. On the BIRD benchmark leaderboard, human performance achieves an accuracy of 92.96\%, whereas the top-performing method reaches only 72.39\%. Notably, these state-of-the-art (SoTA) methods predominantly rely on in-context learning to simulate human-like reasoning. However, they overlook a critical human skill: continual learning. Inspired by the educational practice of maintaining mistake notebooks during our formative years, we propose LPE-SQL (Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL), a novel framework designed to augment LLMs by enabling continual learning without requiring parameter fine-tuning. LPE-SQL consists of four modules that \textbf{i)} retrieve relevant entries, \textbf{ii)} efficient sql generation, \textbf{iii)} generate the final result through a cross-consistency mechanism and \textbf{iv)} log successful and failed tasks along with their reasoning processes or reflection-generated tips. Importantly, the core module of LPE-SQL is the fourth one, while the other modules employ foundational methods, allowing LPE-SQL to be easily integrated with SoTA technologies to further enhance performance. Our experimental results demonstrate that this continual learning approach yields substantial performance gains, with the smaller Llama-3.1-70B model with surpassing the performance of the larger Llama-3.1-405B model using SoTA methods.

Title: I Blame Apple in Part for My False Expectations: An Autoethnographic Study of Apple's Lockdown Mode in iOS

• Authors: Benedikt Mader (1), Christian Eichenmüller (1), Gaston Pugliese (1), Dennis Eckhardt (1), Zinaida Benenson (1) ((1) Friedrich-Alexander-Universität Erlangen-Nürnberg)
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.13249
• Pdf URL: https://arxiv.org/pdf/2411.13249
• Copy Paste: [[2411.13249]] I Blame Apple in Part for My False Expectations: An Autoethnographic Study of Apple's Lockdown Mode in iOS(https://arxiv.org/abs/2411.13249)
• Keywords: security, protect
• Abstract: Lockdown Mode was introduced in 2022 as a hardening setting for Apple's operating systems, designed to strengthen the protection against ``some of the most sophisticated digital threats''. However, Apple never explained these threats further. We present the first academic exploration of Lockdown Mode based on a 3-month autoethnographic study. We obtained a nuanced understanding of user experience and identified issues that can be extrapolated to larger user groups. The lack of information from Apple about the underlying threat model and details on affected features may hinder adequate assessment of Lockdown Mode, making informed decisions on its use challenging. Besides encountering undocumented restrictions, we also experienced both too much and too little visibility of protection during Lockdown Mode use. Finally, we deem the paternalistic security approach by Apple's Lockdown Mode harmful, because without detailed knowledge about technical capabilities and boundaries, at-risk users may be lulled into a false sense of security.

Title: BelHouse3D: A Benchmark Dataset for Assessing Occlusion Robustness in 3D Point Cloud Semantic Segmentation

• Authors: Umamaheswaran Raman Kumar, Abdur Razzaq Fayjie, Jurgen Hannaert, Patrick Vandewalle
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.13251
• Pdf URL: https://arxiv.org/pdf/2411.13251
• Copy Paste: [[2411.13251]] BelHouse3D: A Benchmark Dataset for Assessing Occlusion Robustness in 3D Point Cloud Semantic Segmentation(https://arxiv.org/abs/2411.13251)
• Keywords: robust, segmentation
• Abstract: Large-scale 2D datasets have been instrumental in advancing machine learning; however, progress in 3D vision tasks has been relatively slow. This disparity is largely due to the limited availability of 3D benchmarking datasets. In particular, creating real-world point cloud datasets for indoor scene semantic segmentation presents considerable challenges, including data collection within confined spaces and the costly, often inaccurate process of per-point labeling to generate ground truths. While synthetic datasets address some of these challenges, they often fail to replicate real-world conditions, particularly the occlusions that occur in point clouds collected from real environments. Existing 3D benchmarking datasets typically evaluate deep learning models under the assumption that training and test data are independently and identically distributed (IID), which affects the models' usability for real-world point cloud segmentation. To address these challenges, we introduce the BelHouse3D dataset, a new synthetic point cloud dataset designed for 3D indoor scene semantic segmentation. This dataset is constructed using real-world references from 32 houses in Belgium, ensuring that the synthetic data closely aligns with real-world conditions. Additionally, we include a test set with data occlusion to simulate out-of-distribution (OOD) scenarios, reflecting the occlusions commonly encountered in real-world point clouds. We evaluate popular point-based semantic segmentation methods using our OOD setting and present a benchmark. We believe that BelHouse3D and its OOD setting will advance research in 3D point cloud semantic segmentation for indoor scenes, providing valuable insights for the development of more generalizable models.

Title: Transformers with Sparse Attention for Granger Causality

• Authors: Riya Mahesh, Rahul Vashisht, Chandrashekar Lakshminarayanan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13264
• Pdf URL: https://arxiv.org/pdf/2411.13264
• Copy Paste: [[2411.13264]] Transformers with Sparse Attention for Granger Causality(https://arxiv.org/abs/2411.13264)
• Keywords: transformer
• Abstract: Temporal causal analysis means understanding the underlying causes behind observed variables over time. Deep learning based methods such as transformers are increasingly used to capture temporal dynamics and causal relationships beyond mere correlations. Recent works suggest self-attention weights of transformers as a useful indicator of causal links. We leverage this to propose a novel modification to the self-attention module to establish causal links between the variables of multivariate time-series data with varying lag dependencies. Our Sparse Attention Transformer captures causal relationships using a two-fold approach - performing temporal attention first followed by attention between the variables across the time steps masking them individually to compute Granger Causality indices. The key novelty in our approach is the ability of the model to assert importance and pick the most significant past time instances for its prediction task against manually feeding a fixed time lag value. We demonstrate the effectiveness of our approach via extensive experimentation on several synthetic benchmark datasets. Furthermore, we compare the performance of our model with the traditional Vector Autoregression based Granger Causality method that assumes fixed lag length.

Title: VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation

• Authors: Ziyang Luo, Haoning Wu, Dongxu Li, Jing Ma, Mohan Kankanhalli, Junnan Li
• Subjects: cs.CV, cs.AI, cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.13281
• Pdf URL: https://arxiv.org/pdf/2411.13281
• Copy Paste: [[2411.13281]] VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation(https://arxiv.org/abs/2411.13281)
• Keywords: fair
• Abstract: Large multimodal models (LMMs) with advanced video analysis capabilities have recently garnered significant attention. However, most evaluations rely on traditional methods like multiple-choice questions in benchmarks such as VideoMME and LongVideoBench, which are prone to lack the depth needed to capture the complex demands of real-world users. To address this limitation-and due to the prohibitive cost and slow pace of human annotation for video tasks-we introduce VideoAutoArena, an arena-style benchmark inspired by LMSYS Chatbot Arena's framework, designed to automatically assess LMMs' video analysis abilities. VideoAutoArena utilizes user simulation to generate open-ended, adaptive questions that rigorously assess model performance in video understanding. The benchmark features an automated, scalable evaluation framework, incorporating a modified ELO Rating System for fair and continuous comparisons across multiple LMMs. To validate our automated judging system, we construct a 'gold standard' using a carefully curated subset of human annotations, demonstrating that our arena strongly aligns with human judgment while maintaining scalability. Additionally, we introduce a fault-driven evolution strategy, progressively increasing question complexity to push models toward handling more challenging video analysis scenarios. Experimental results demonstrate that VideoAutoArena effectively differentiates among state-of-the-art LMMs, providing insights into model strengths and areas for improvement. To further streamline our evaluation, we introduce VideoAutoBench as an auxiliary benchmark, where human annotators label winners in a subset of VideoAutoArena battles. We use GPT-4o as a judge to compare responses against these human-validated answers. Together, VideoAutoArena and VideoAutoBench offer a cost-effective, and scalable framework for evaluating LMMs in user-centric video analysis.

Title: Combining Autoregressive and Autoencoder Language Models for Text Classification

• Authors: João Gonçalves
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13282
• Pdf URL: https://arxiv.org/pdf/2411.13282
• Copy Paste: [[2411.13282]] Combining Autoregressive and Autoencoder Language Models for Text Classification(https://arxiv.org/abs/2411.13282)
• Keywords: large language model
• Abstract: This paper presents CAALM-TC (Combining Autoregressive and Autoencoder Language Models for Text Classification), a novel method that enhances text classification by integrating autoregressive and autoencoder language models. Autoregressive large language models such as Open AI's GPT, Meta's Llama or Microsoft's Phi offer promising prospects for content analysis practitioners, but they generally underperform supervised BERT based models for text classification. CAALM leverages autoregressive models to generate contextual information based on input texts, which is then combined with the original text and fed into an autoencoder model for classification. This hybrid approach capitalizes on the extensive contextual knowledge of autoregressive models and the efficient classification capabilities of autoencoders. Experimental results on four benchmark datasets demonstrate that CAALM consistently outperforms existing methods, particularly in tasks with smaller datasets and more abstract classification objectives. The findings indicate that CAALM offers a scalable and effective solution for automated content analysis in social science research that minimizes sample size requirements.

Title: DATAP-SfM: Dynamic-Aware Tracking Any Point for Robust Structure from Motion in the Wild

• Authors: Weicai Ye, Xinyu Chen, Ruohao Zhan, Di Huang, Xiaoshui Huang, Haoyi Zhu, Hujun Bao, Wanli Ouyang, Tong He, Guofeng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13291
• Pdf URL: https://arxiv.org/pdf/2411.13291
• Copy Paste: [[2411.13291]] DATAP-SfM: Dynamic-Aware Tracking Any Point for Robust Structure from Motion in the Wild(https://arxiv.org/abs/2411.13291)
• Keywords: robust, segmentation
• Abstract: This paper proposes a concise, elegant, and robust pipeline to estimate smooth camera trajectories and obtain dense point clouds for casual videos in the wild. Traditional frameworks, such as ParticleSfM~\cite{zhao2022particlesfm}, address this problem by sequentially computing the optical flow between adjacent frames to obtain point trajectories. They then remove dynamic trajectories through motion segmentation and perform global bundle adjustment. However, the process of estimating optical flow between two adjacent frames and chaining the matches can introduce cumulative errors. Additionally, motion segmentation combined with single-view depth estimation often faces challenges related to scale ambiguity. To tackle these challenges, we propose a dynamic-aware tracking any point (DATAP) method that leverages consistent video depth and point tracking. Specifically, our DATAP addresses these issues by estimating dense point tracking across the video sequence and predicting the visibility and dynamics of each point. By incorporating the consistent video depth prior, the performance of motion segmentation is enhanced. With the integration of DATAP, it becomes possible to estimate and optimize all camera poses simultaneously by performing global bundle adjustments for point tracking classified as static and visible, rather than relying on incremental camera registration. Extensive experiments on dynamic sequences, e.g., Sintel and TUM RGBD dynamic sequences, and on the wild video, e.g., DAVIS, demonstrate that the proposed method achieves state-of-the-art performance in terms of camera pose estimation even in complex dynamic challenge scenes.

Title: Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach

• Authors: Vaishnavi Khindkar, Vineeth Balasubramanian, Chetan Arora, Anbumani Subramanian, C.V. Jawahar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13302
• Pdf URL: https://arxiv.org/pdf/2411.13302
• Copy Paste: [[2411.13302]] Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach(https://arxiv.org/abs/2411.13302)
• Keywords: protect
• Abstract: With the increased importance of autonomous navigation systems has come an increasing need to protect the safety of Vulnerable Road Users (VRUs) such as pedestrians. Predicting pedestrian intent is one such challenging task, where prior work predicts the binary cross/no-cross intention with a fusion of visual and motion features. However, there has been no effort so far to hedge such predictions with human-understandable reasons. We address this issue by introducing a novel problem setting of exploring the intuitive reasoning behind a pedestrian's intent. In particular, we show that predicting the 'WHY' can be very useful in understanding the 'WHAT'. To this end, we propose a novel, reason-enriched PIE++ dataset consisting of multi-label textual explanations/reasons for pedestrian intent. We also introduce a novel multi-task learning framework called MINDREAD, which leverages a cross-modal representation learning framework for predicting pedestrian intent as well as the reason behind the intent. Our comprehensive experiments show significant improvement of 5.6% and 7% in accuracy and F1-score for the task of intent prediction on the PIE++ dataset using MINDREAD. We also achieved a 4.4% improvement in accuracy on a commonly used JAAD dataset. Extensive evaluation using quantitative/qualitative metrics and user studies shows the effectiveness of our approach.

Title: Verifying Machine Unlearning with Explainable AI

• Authors: Àlex Pujol Vidal, Anders S. Johansen, Mohammad N. S. Jahromi, Sergio Escalera, Kamal Nasrollahi, Thomas B. Moeslund
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13332
• Pdf URL: https://arxiv.org/pdf/2411.13332
• Copy Paste: [[2411.13332]] Verifying Machine Unlearning with Explainable AI(https://arxiv.org/abs/2411.13332)
• Keywords: privacy, protect
• Abstract: We investigate the effectiveness of Explainable AI (XAI) in verifying Machine Unlearning (MU) within the context of harbor front monitoring, focusing on data privacy and regulatory compliance. With the increasing need to adhere to privacy legislation such as the General Data Protection Regulation (GDPR), traditional methods of retraining ML models for data deletions prove impractical due to their complexity and resource demands. MU offers a solution by enabling models to selectively forget specific learned patterns without full retraining. We explore various removal techniques, including data relabeling, and model perturbation. Then, we leverage attribution-based XAI to discuss the effects of unlearning on model performance. Our proof-of-concept introduces feature importance as an innovative verification step for MU, expanding beyond traditional metrics and demonstrating techniques' ability to reduce reliance on undesired patterns. Additionally, we propose two novel XAI-based metrics, Heatmap Coverage (HC) and Attention Shift (AS), to evaluate the effectiveness of these methods. This approach not only highlights how XAI can complement MU by providing effective verification, but also sets the stage for future research to enhance their joint integration.

Title: Vertical Validation: Evaluating Implicit Generative Models for Graphs on Thin Support Regions

• Authors: Mai Elkady, Thu Bui, Bruno Ribeiro, David I. Inouye
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13358
• Pdf URL: https://arxiv.org/pdf/2411.13358
• Copy Paste: [[2411.13358]] Vertical Validation: Evaluating Implicit Generative Models for Graphs on Thin Support Regions(https://arxiv.org/abs/2411.13358)
• Keywords: generative
• Abstract: There has been a growing excitement that implicit graph generative models could be used to design or discover new molecules for medicine or material design. Because these molecules have not been discovered, they naturally lie in unexplored or scarcely supported regions of the distribution of known molecules. However, prior evaluation methods for implicit graph generative models have focused on validating statistics computed from the thick support (e.g., mean and variance of a graph property). Therefore, there is a mismatch between the goal of generating novel graphs and the evaluation methods. To address this evaluation gap, we design a novel evaluation method called Vertical Validation (VV) that systematically creates thin support regions during the train-test splitting procedure and then reweights generated samples so that they can be compared to the held-out test data. This procedure can be seen as a generalization of the standard train-test procedure except that the splits are dependent on sample features. We demonstrate that our method can be used to perform model selection if performance on thin support regions is the desired goal. As a side benefit, we also show that our approach can better detect overfitting as exemplified by memorization.

Title: On the Way to LLM Personalization: Learning to Remember User Conversations

• Authors: Lucie Charlotte Magister, Katherine Metcalf, Yizhe Zhang, Maartje ter Hoeve
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13405
• Pdf URL: https://arxiv.org/pdf/2411.13405
• Copy Paste: [[2411.13405]] On the Way to LLM Personalization: Learning to Remember User Conversations(https://arxiv.org/abs/2411.13405)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have quickly become an invaluable assistant for a variety of tasks. However, their effectiveness is constrained by their ability to tailor responses to human preferences and behaviors via personalization. Prior work in LLM personalization has largely focused on style transfer or incorporating small factoids about the user, as knowledge injection remains an open challenge. In this paper, we explore injecting knowledge of prior conversations into LLMs to enable future work on less redundant, personalized conversations. We identify two real-world constraints: (1) conversations are sequential in time and must be treated as such during training, and (2) per-user personalization is only viable in parameter-efficient settings. To this aim, we propose PLUM, a pipeline performing data augmentation for up-sampling conversations as question-answer pairs, that are then used to finetune a low-rank adaptation adapter with a weighted cross entropy loss. Even in this first exploration of the problem, we perform competitively with baselines such as RAG, attaining an accuracy of 81.5% across 100 conversations.

Title: Transformer-Based Contextualized Language Models Joint with Neural Networks for Natural Language Inference in Vietnamese

• Authors: Dat Van-Thanh Nguyen, Tin Van Huynh, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13407
• Pdf URL: https://arxiv.org/pdf/2411.13407
• Copy Paste: [[2411.13407]] Transformer-Based Contextualized Language Models Joint with Neural Networks for Natural Language Inference in Vietnamese(https://arxiv.org/abs/2411.13407)
• Keywords: transformer
• Abstract: Natural Language Inference (NLI) is a task within Natural Language Processing (NLP) that holds value for various AI applications. However, there have been limited studies on Natural Language Inference in Vietnamese that explore the concept of joint models. Therefore, we conducted experiments using various combinations of contextualized language models (CLM) and neural networks. We use CLM to create contextualized work presentations and use Neural Networks for classification. Furthermore, we have evaluated the strengths and weaknesses of each joint model and identified the model failure points in the Vietnamese context. The highest F1 score in this experiment, up to 82.78\% in the benchmark dataset (ViNLI). By conducting experiments with various models, the most considerable size of the CLM is XLM-R (355M). That combination has consistently demonstrated superior performance compared to fine-tuning strong pre-trained language models like PhoBERT (+6.58\%), mBERT (+19.08\%), and XLM-R (+0.94\%) in terms of F1-score. This article aims to introduce a novel approach or model that attains improved performance for Vietnamese NLI. Overall, we find that the joint approach of CLM and neural networks is simple yet capable of achieving high-quality performance, which makes it suitable for applications that require efficient resource utilization.

Title: Unification of Balti and trans-border sister dialects in the essence of LLMs and AI Technology

• Authors: Muhammad Sharif, Jiangyan Yi, Muhammad Shoaib
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13409
• Pdf URL: https://arxiv.org/pdf/2411.13409
• Copy Paste: [[2411.13409]] Unification of Balti and trans-border sister dialects in the essence of LLMs and AI Technology(https://arxiv.org/abs/2411.13409)
• Keywords: large language model
• Abstract: The language called Balti belongs to the Sino-Tibetan, specifically the Tibeto-Burman language family. It is understood with variations, across populations in India, China, Pakistan, Nepal, Tibet, Burma, and Bhutan, influenced by local cultures and producing various dialects. Considering the diverse cultural, socio-political, religious, and geographical impacts, it is important to step forward unifying the dialects, the basis of common root, lexica, and phonological perspectives, is vital. In the era of globalization and the increasingly frequent developments in AI technology, understanding the diversity and the efforts of dialect unification is important to understanding commonalities and shortening the gaps impacted by unavoidable circumstances. This article analyzes and examines how artificial intelligence AI in the essence of Large Language Models LLMs, can assist in analyzing, documenting, and standardizing the endangered Balti Language, based on the efforts made in different dialects so far.

Title: A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback

• Authors: Alireza Rashidi Laleh, Majid Nili Ahmadabadi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13410
• Pdf URL: https://arxiv.org/pdf/2411.13410
• Copy Paste: [[2411.13410]] A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback(https://arxiv.org/abs/2411.13410)
• Keywords: large language model
• Abstract: Reinforcement learning (RL) is one of the active fields in machine learning, demonstrating remarkable potential in tackling real-world challenges. Despite its promising prospects, this methodology has encountered with issues and challenges, hindering it from achieving the best performance. In particular, these approaches lack decent performance when navigating environments and solving tasks with large observation space, often resulting in sample-inefficiency and prolonged learning times. This issue, commonly referred to as the curse of dimensionality, complicates decision-making for RL agents, necessitating a careful balance between attention and decision-making. RL agents, when augmented with human or large language models' (LLMs) feedback, may exhibit resilience and adaptability, leading to enhanced performance and accelerated learning. Such feedback, conveyed through various modalities or granularities including natural language, serves as a guide for RL agents, aiding them in discerning relevant environmental cues and optimizing decision-making processes. In this survey paper, we mainly focus on problems of two-folds: firstly, we focus on humans or an LLMs assistance, investigating the ways in which these entities may collaborate with the RL agent in order to foster optimal behavior and expedite learning; secondly, we delve into the research papers dedicated to addressing the intricacies of environments characterized by large observation space.

Title: WaterPark: A Robustness Assessment of Language Model Watermarking

• Authors: Jiacheng Liang, Zian Wang, Lauren Hong, Shouling Ji, Ting Wang
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13425
• Pdf URL: https://arxiv.org/pdf/2411.13425
• Copy Paste: [[2411.13425]] WaterPark: A Robustness Assessment of Language Model Watermarking(https://arxiv.org/abs/2411.13425)
• Keywords: security, attack, robust, watermark, generative, large language model
• Abstract: To mitigate the misuse of large language models (LLMs), such as disinformation, automated phishing, and academic cheating, there is a pressing need for the capability of identifying LLM-generated texts. Watermarking emerges as one promising solution: it plants statistical signals into LLMs' generative processes and subsequently verifies whether LLMs produce given texts. Various watermarking methods (``watermarkers'') have been proposed; yet, due to the lack of unified evaluation platforms, many critical questions remain under-explored: i) What are the strengths/limitations of various watermarkers, especially their attack robustness? ii) How do various design choices impact their robustness? iii) How to optimally operate watermarkers in adversarial environments? To fill this gap, we systematize existing LLM watermarkers and watermark removal attacks, mapping out their design spaces. We then develop WaterPark, a unified platform that integrates 10 state-of-the-art watermarkers and 12 representative attacks. More importantly, leveraging WaterPark, we conduct a comprehensive assessment of existing watermarkers, unveiling the impact of various design choices on their attack robustness. For instance, a watermarker's resilience to increasingly intensive attacks hinges on its context dependency. We further explore the best practices to operate watermarkers in adversarial environments. For instance, using a generic detector alongside a watermark-specific detector improves the security of vulnerable watermarkers. We believe our study sheds light on current LLM watermarking techniques while WaterPark serves as a valuable testbed to facilitate future research.

Title: SynEHRgy: Synthesizing Mixed-Type Structured Electronic Health Records using Decoder-Only Transformers

• Authors: Hojjat Karami, David Atienza, Anisoara Ionescu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13428
• Pdf URL: https://arxiv.org/pdf/2411.13428
• Copy Paste: [[2411.13428]] SynEHRgy: Synthesizing Mixed-Type Structured Electronic Health Records using Decoder-Only Transformers(https://arxiv.org/abs/2411.13428)
• Keywords: privacy, transformer
• Abstract: Generating synthetic Electronic Health Records (EHRs) offers significant potential for data augmentation, privacy-preserving data sharing, and improving machine learning model training. We propose a novel tokenization strategy tailored for structured EHR data, which encompasses diverse data types such as covariates, ICD codes, and irregularly sampled time series. Using a GPT-like decoder-only transformer model, we demonstrate the generation of high-quality synthetic EHRs. Our approach is evaluated using the MIMIC-III dataset, and we benchmark the fidelity, utility, and privacy of the generated data against state-of-the-art models.

Title: Blockchain-Enhanced Framework for Secure Third-Party Vendor Risk Management and Vigilant Security Controls

• Authors: Deepti Gupta, Lavanya Elluri, Avi Jain, Shafika Showkat Moni, Omer Aslan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.13447
• Pdf URL: https://arxiv.org/pdf/2411.13447
• Copy Paste: [[2411.13447]] Blockchain-Enhanced Framework for Secure Third-Party Vendor Risk Management and Vigilant Security Controls(https://arxiv.org/abs/2411.13447)
• Keywords: secure, security, defense, robust
• Abstract: In an era of heightened digital interconnectedness, businesses increasingly rely on third-party vendors to enhance their operational capabilities. However, this growing dependency introduces significant security risks, making it crucial to develop a robust framework to mitigate potential vulnerabilities. This paper proposes a comprehensive secure framework for managing third-party vendor risk, integrating blockchain technology to ensure transparency, traceability, and immutability in vendor assessments and interactions. By leveraging blockchain, the framework enhances the integrity of vendor security audits, ensuring that vendor assessments remain up-to-date and tamperproof. This proposed framework leverages smart contracts to reduce human error while ensuring real-time monitoring of compliance and security controls. By evaluating critical security controls-such as data encryption, access control mechanisms, multi-factor authentication, and zero-trust architecture-this approach strengthens an organization's defense against emerging cyber threats. Additionally, continuous monitoring enabled by blockchain ensures the immutability and transparency of vendor compliance processes. In this paper, a case study on iHealth's transition to AWS Cloud demonstrates the practical implementation of the framework, showing a significant reduction in vulnerabilities and marked improvement in incident response times. Through the adoption of this blockchain-enabled approach, organizations can mitigate vendor risks, streamline compliance, and enhance their overall security posture.

Title: LIMBA: An Open-Source Framework for the Preservation and Valorization of Low-Resource Languages using Generative Models

• Authors: Salvatore Mario Carta, Stefano Chessa, Giulia Contu, Andrea Corriga, Andrea Deidda, Gianni Fenu, Luca Frigau, Alessandro Giuliani, Luca Grassi, Marco Manolo Manca, Mirko Marras, Francesco Mola, Bastianino Mossa, Piergiorgio Mura, Marco Ortu, Leonardo Piano, Simone Pisano, Alessia Pisu, Alessandro Sebastian Podda, Livio Pompianu, Simone Seu, Sandro Gabriele Tiddia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13453
• Pdf URL: https://arxiv.org/pdf/2411.13453
• Copy Paste: [[2411.13453]] LIMBA: An Open-Source Framework for the Preservation and Valorization of Low-Resource Languages using Generative Models(https://arxiv.org/abs/2411.13453)
• Keywords: generative
• Abstract: Minority languages are vital to preserving cultural heritage, yet they face growing risks of extinction due to limited digital resources and the dominance of artificial intelligence models trained on high-resource languages. This white paper proposes a framework to generate linguistic tools for low-resource languages, focusing on data creation to support the development of language models that can aid in preservation efforts. Sardinian, an endangered language, serves as the case study to demonstrate the framework's effectiveness. By addressing the data scarcity that hinders intelligent applications for such languages, we contribute to promoting linguistic diversity and support ongoing efforts in language standardization and revitalization through modern technologies.

Title: SoK: A Systems Perspective on Compound AI Threats and Countermeasures

• Authors: Sarbartha Banerjee, Prateek Sahu, Mulong Luo, Anjo Vahldiek-Oberwagner, Neeraja J. Yadwadkar, Mohit Tiwari
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13459
• Pdf URL: https://arxiv.org/pdf/2411.13459
• Copy Paste: [[2411.13459]] SoK: A Systems Perspective on Compound AI Threats and Countermeasures(https://arxiv.org/abs/2411.13459)
• Keywords: secure, defense, attack, large language model
• Abstract: Large language models (LLMs) used across enterprises often use proprietary models and operate on sensitive inputs and data. The wide range of attack vectors identified in prior research - targeting various software and hardware components used in training and inference - makes it extremely challenging to enforce confidentiality and integrity policies. As we advance towards constructing compound AI inference pipelines that integrate multiple large language models (LLMs), the attack surfaces expand significantly. Attackers now focus on the AI algorithms as well as the software and hardware components associated with these systems. While current research often examines these elements in isolation, we find that combining cross-layer attack observations can enable powerful end-to-end attacks with minimal assumptions about the threat model. Given, the sheer number of existing attacks at each layer, we need a holistic and systemized understanding of different attack vectors at each layer. This SoK discusses different software and hardware attacks applicable to compound AI systems and demonstrates how combining multiple attack mechanisms can reduce the threat model assumptions required for an isolated attack. Next, we systematize the ML attacks in lines with the Mitre Att&ck framework to better position each attack based on the threat model. Finally, we outline the existing countermeasures for both software and hardware layers and discuss the necessity of a comprehensive defense strategy to enable the secure and high-performance deployment of compound AI systems.

Title: When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

• Authors: Haonan Wang, Qian Liu, Chao Du, Tongyao Zhu, Cunxiao Du, Kenji Kawaguchi, Tianyu Pang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13476
• Pdf URL: https://arxiv.org/pdf/2411.13476
• Copy Paste: [[2411.13476]] When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training(https://arxiv.org/abs/2411.13476)
• Keywords: large language model
• Abstract: Extending context window sizes allows large language models (LLMs) to process longer sequences and handle more complex tasks. Rotary Positional Embedding (RoPE) has become the de facto standard due to its relative positional encoding properties that benefit long-context training. However, we observe that using RoPE with BFloat16 format results in numerical issues, causing it to deviate from its intended relative positional encoding, especially in long-context scenarios. This issue arises from BFloat16's limited precision and accumulates as context length increases, with the first token contributing significantly to this problem. To address this, we develop AnchorAttention, a plug-and-play attention method that alleviates numerical issues caused by BFloat16, improves long-context capabilities, and speeds up training. AnchorAttention reduces unnecessary attention computations, maintains semantic coherence, and boosts computational efficiency by treating the first token as a shared anchor with a consistent position ID, making it visible to all documents within the training context. Experiments on three types of LLMs demonstrate that AnchorAttention significantly improves long-context performance and reduces training time by over 50\% compared to standard full attention mechanisms, while preserving the original LLM's capabilities on general tasks. Our code is available at this https URL.

Title: PatentEdits: Framing Patent Novelty as Textual Entailment

• Authors: Ryan Lee, Alexander Spangher, Xuezhe Ma
• Subjects: cs.CL, cs.AI, cs.CY, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.13477
• Pdf URL: https://arxiv.org/pdf/2411.13477
• Copy Paste: [[2411.13477]] PatentEdits: Framing Patent Novelty as Textual Entailment(https://arxiv.org/abs/2411.13477)
• Keywords: large language model
• Abstract: A patent must be deemed novel and non-obvious in order to be granted by the US Patent Office (USPTO). If it is not, a US patent examiner will cite the prior work, or prior art, that invalidates the novelty and issue a non-final rejection. Predicting what claims of the invention should change given the prior art is an essential and crucial step in securing invention rights, yet has not been studied before as a learnable task. In this work we introduce the PatentEdits dataset, which contains 105K examples of successful revisions that overcome objections to novelty. We design algorithms to label edits sentence by sentence, then establish how well these edits can be predicted with large language models (LLMs). We demonstrate that evaluating textual entailment between cited references and draft sentences is especially effective in predicting which inventive claims remained unchanged or are novel in relation to prior art.

Title: Utilizing Large Language Models to Synthesize Product Desirability Datasets

• Authors: John D. Hastings, Sherri Weitl-Harms, Joseph Doty, Zachary L. Myers, Warren Thompson
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13485
• Pdf URL: https://arxiv.org/pdf/2411.13485
• Copy Paste: [[2411.13485]] Utilizing Large Language Models to Synthesize Product Desirability Datasets(https://arxiv.org/abs/2411.13485)
• Keywords: large language model
• Abstract: This research explores the application of large language models (LLMs) to generate synthetic datasets for Product Desirability Toolkit (PDT) testing, a key component in evaluating user sentiment and product experience. Utilizing gpt-4o-mini, a cost-effective alternative to larger commercial LLMs, three methods, Word+Review, Review+Word, and Supply-Word, were each used to synthesize 1000 product reviews. The generated datasets were assessed for sentiment alignment, textual diversity, and data generation cost. Results demonstrated high sentiment alignment across all methods, with Pearson correlations ranging from 0.93 to 0.97. Supply-Word exhibited the highest diversity and coverage of PDT terms, although with increased generation costs. Despite minor biases toward positive sentiments, in situations with limited test data, LLM-generated synthetic data offers significant advantages, including scalability, cost savings, and flexibility in dataset production.

Title: Advancing Heatwave Forecasting via Distribution Informed-Graph Neural Networks (DI-GNNs): Integrating Extreme Value Theory with GNNs

• Authors: Farrukh A. Chishtie, Dominique Brunet, Rachel H. White, Daniel Michelson, Jing Jiang, Vicky Lucas, Emily Ruboonga, Sayana Imaash, Melissa Westland, Timothy Chui, Rana Usman Ali, Mujtaba Hassan, Roland Stull, David Hudak
• Subjects: cs.LG, physics.ao-ph, physics.soc-ph
• Abstract URL: https://arxiv.org/abs/2411.13496
• Pdf URL: https://arxiv.org/pdf/2411.13496
• Copy Paste: [[2411.13496]] Advancing Heatwave Forecasting via Distribution Informed-Graph Neural Networks (DI-GNNs): Integrating Extreme Value Theory with GNNs(https://arxiv.org/abs/2411.13496)
• Keywords: robust
• Abstract: Heatwaves, prolonged periods of extreme heat, have intensified in frequency and severity due to climate change, posing substantial risks to public health, ecosystems, and infrastructure. Despite advancements in Machine Learning (ML) modeling, accurate heatwave forecasting at weather scales (1--15 days) remains challenging due to the non-linear interactions between atmospheric drivers and the rarity of these extreme events. Traditional models relying on heuristic feature engineering often fail to generalize across diverse climates and capture the complexities of heatwave dynamics. This study introduces the Distribution-Informed Graph Neural Network (DI-GNN), a novel framework that integrates principles from Extreme Value Theory (EVT) into the graph neural network architecture. DI-GNN incorporates Generalized Pareto Distribution (GPD)-derived descriptors into the feature space, adjacency matrix, and loss function to enhance its sensitivity to rare heatwave occurrences. By prioritizing the tails of climatic distributions, DI-GNN addresses the limitations of existing methods, particularly in imbalanced datasets where traditional metrics like accuracy are misleading. Empirical evaluations using weather station data from British Columbia, Canada, demonstrate the superior performance of DI-GNN compared to baseline models. DI-GNN achieved significant improvements in balanced accuracy, recall, and precision, with high AUC and average precision scores, reflecting its robustness in distinguishing heatwave events.

Title: VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models

• Authors: Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin, Yu Qiao, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13503
• Pdf URL: https://arxiv.org/pdf/2411.13503
• Copy Paste: [[2411.13503]] VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models(https://arxiv.org/abs/2411.13503)
• Keywords: fair, generative
• Abstract: Video generation has witnessed significant advancements, yet evaluating these models remains a challenge. A comprehensive evaluation benchmark for video generation is indispensable for two reasons: 1) Existing metrics do not fully align with human perceptions; 2) An ideal evaluation system should provide insights to inform future developments of video generation. To this end, we present VBench, a comprehensive benchmark suite that dissects "video generation quality" into specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. VBench has several appealing properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in video generation (e.g., subject identity inconsistency, motion smoothness, temporal flickering, and spatial relationship, etc). The evaluation metrics with fine-grained levels reveal individual models' strengths and weaknesses. 2) Human Alignment: We also provide a dataset of human preference annotations to validate our benchmarks' alignment with human perception, for each evaluation dimension respectively. 3) Valuable Insights: We look into current models' ability across various evaluation dimensions, and various content types. We also investigate the gaps between video and image generation models. 4) Versatile Benchmarking: VBench++ supports evaluating text-to-video and image-to-video. We introduce a high-quality Image Suite with an adaptive aspect ratio to enable fair evaluations across different image-to-video generation settings. Beyond assessing technical quality, VBench++ evaluates the trustworthiness of video generative models, providing a more holistic view of model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and continually add new video generation models to our leaderboard to drive forward the field of video generation.

Title: Disentangling Memory and Reasoning Ability in Large Language Models

• Authors: Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, Yongfeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13504
• Pdf URL: https://arxiv.org/pdf/2411.13504
• Copy Paste: [[2411.13504]] Disentangling Memory and Reasoning Ability in Large Language Models(https://arxiv.org/abs/2411.13504)
• Keywords: interpretability, large language model
• Abstract: Large Language Models (LLMs) have demonstrated strong performance in handling complex tasks requiring both extensive knowledge and reasoning abilities. However, the existing LLM inference pipeline operates as an opaque process without explicit separation between knowledge retrieval and reasoning steps, making the model's decision-making process unclear and disorganized. This ambiguity can lead to issues such as hallucinations and knowledge forgetting, which significantly impact the reliability of LLMs in high-stakes domains. In this paper, we propose a new inference paradigm that decomposes the complex inference process into two distinct and clear actions: (1) memory recall: which retrieves relevant knowledge, and (2) reasoning: which performs logical steps based on the recalled knowledge. To facilitate this decomposition, we introduce two special tokens memory and reason, guiding the model to distinguish between steps that require knowledge retrieval and those that involve reasoning. Our experiment results show that this decomposition not only improves model performance but also enhances the interpretability of the inference process, enabling users to identify sources of error and refine model responses effectively. The code is available at this https URL.

Title: Advancing Complex Medical Communication in Arabic with Sporo AraSum: Surpassing Existing Large Language Models

• Authors: Chanseo Lee, Sonu Kumar, Kimon A. Vogt, Sam Meraj, Antonia Vogt
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13518
• Pdf URL: https://arxiv.org/pdf/2411.13518
• Copy Paste: [[2411.13518]] Advancing Complex Medical Communication in Arabic with Sporo AraSum: Surpassing Existing Large Language Models(https://arxiv.org/abs/2411.13518)
• Keywords: large language model
• Abstract: The increasing demand for multilingual capabilities in healthcare underscores the need for AI models adept at processing diverse languages, particularly in clinical documentation and decision-making. Arabic, with its complex morphology, syntax, and diglossia, poses unique challenges for natural language processing (NLP) in medical contexts. This case study evaluates Sporo AraSum, a language model tailored for Arabic clinical documentation, against JAIS, the leading Arabic NLP model. Using synthetic datasets and modified PDQI-9 metrics modified ourselves for the purposes of assessing model performances in a different language. The study assessed the models' performance in summarizing patient-physician interactions, focusing on accuracy, comprehensiveness, clinical utility, and linguistic-cultural competence. Results indicate that Sporo AraSum significantly outperforms JAIS in AI-centric quantitative metrics and all qualitative attributes measured in our modified version of the PDQI-9. AraSum's architecture enables precise and culturally sensitive documentation, addressing the linguistic nuances of Arabic while mitigating risks of AI hallucinations. These findings suggest that Sporo AraSum is better suited to meet the demands of Arabic-speaking healthcare environments, offering a transformative solution for multilingual clinical workflows. Future research should incorporate real-world data to further validate these findings and explore broader integration into healthcare systems.

Title: Geometric Algebra Planes: Convex Implicit Neural Volumes

• Authors: Irmak Sivgin, Sara Fridovich-Keil, Gordon Wetzstein, Mert Pilanci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13525
• Pdf URL: https://arxiv.org/pdf/2411.13525
• Copy Paste: [[2411.13525]] Geometric Algebra Planes: Convex Implicit Neural Volumes(https://arxiv.org/abs/2411.13525)
• Keywords: segmentation
• Abstract: Volume parameterizations abound in recent literature, from the classic voxel grid to the implicit neural representation and everything in between. While implicit representations have shown impressive capacity and better memory efficiency compared to voxel grids, to date they require training via nonconvex optimization. This nonconvex training process can be slow to converge and sensitive to initialization and hyperparameter choices that affect the final converged result. We introduce a family of models, GA-Planes, that is the first class of implicit neural volume representations that can be trained by convex optimization. GA-Planes models include any combination of features stored in tensor basis elements, followed by a neural feature decoder. They generalize many existing representations and can be adapted for convex, semiconvex, or nonconvex training as needed for different inverse problems. In the 2D setting, we prove that GA-Planes is equivalent to a low-rank plus low-resolution matrix factorization; we show that this approximation outperforms the classic low-rank plus sparse decomposition for fitting a natural image. In 3D, we demonstrate GA-Planes' competitive performance in terms of expressiveness, model size, and optimizability across three volume fitting tasks: radiance field reconstruction, 3D segmentation, and video segmentation.

Title: Entropy Bootstrapping for Weakly Supervised Nuclei Detection

• Authors: James Willoughby, Irina Voiculescu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13528
• Pdf URL: https://arxiv.org/pdf/2411.13528
• Copy Paste: [[2411.13528]] Entropy Bootstrapping for Weakly Supervised Nuclei Detection(https://arxiv.org/abs/2411.13528)
• Keywords: segmentation
• Abstract: Microscopy structure segmentation, such as detecting cells or nuclei, generally requires a human to draw a ground truth contour around each instance. Weakly supervised approaches (e.g. consisting of only single point labels) have the potential to reduce this workload significantly. Our approach uses individual point labels for an entropy estimation to approximate an underlying distribution of cell pixels. We infer full cell masks from this distribution, and use Mask-RCNN to produce an instance segmentation output. We compare this point--annotated approach with training on the full ground truth masks. We show that our method achieves a comparatively good level of performance, despite a 95% reduction in pixel labels.

Title: Predictive Insights into LGBTQ+ Minority Stress: A Transductive Exploration of Social Media Discourse

• Authors: S. Chapagain, Y. Zhao, T. K. Rohleen, S. M. Hamdi, S. F. Boubrahimi, R. E. Flinn, E. M. Lund, D. Klooster, J. R. Scheer, C. J. Cascalheira
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13534
• Pdf URL: https://arxiv.org/pdf/2411.13534
• Copy Paste: [[2411.13534]] Predictive Insights into LGBTQ+ Minority Stress: A Transductive Exploration of Social Media Discourse(https://arxiv.org/abs/2411.13534)
• Keywords: extraction, transformer
• Abstract: Individuals who identify as sexual and gender minorities, including lesbian, gay, bisexual, transgender, queer, and others (LGBTQ+) are more likely to experience poorer health than their heterosexual and cisgender counterparts. One primary source that drives these health disparities is minority stress (i.e., chronic and social stressors unique to LGBTQ+ communities' experiences adapting to the dominant culture). This stress is frequently expressed in LGBTQ+ users' posts on social media platforms. However, these expressions are not just straightforward manifestations of minority stress. They involve linguistic complexity (e.g., idiom or lexical diversity), rendering them challenging for many traditional natural language processing methods to detect. In this work, we designed a hybrid model using Graph Neural Networks (GNN) and Bidirectional Encoder Representations from Transformers (BERT), a pre-trained deep language model to improve the classification performance of minority stress detection. We experimented with our model on a benchmark social media dataset for minority stress detection (LGBTQ+ MiSSoM+). The dataset is comprised of 5,789 human-annotated Reddit posts from LGBTQ+ subreddits. Our approach enables the extraction of hidden linguistic nuances through pretraining on a vast amount of raw data, while also engaging in transductive learning to jointly develop representations for both labeled training data and unlabeled test data. The RoBERTa-GCN model achieved an accuracy of 0.86 and an F1 score of 0.86, surpassing the performance of other baseline models in predicting LGBTQ+ minority stress. Improved prediction of minority stress expressions on social media could lead to digital health interventions to improve the wellbeing of LGBTQ+ people-a community with high rates of stress-sensitive health problems.

Title: Identity Preserving 3D Head Stylization with Multiview Score Distillation

• Authors: Bahri Batuhan Bilecen, Ahmet Berke Gokmen, Furkan Guzelant, Aysegul Dundar
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.13536
• Pdf URL: https://arxiv.org/pdf/2411.13536
• Copy Paste: [[2411.13536]] Identity Preserving 3D Head Stylization with Multiview Score Distillation(https://arxiv.org/abs/2411.13536)
• Keywords: diffusion
• Abstract: 3D head stylization transforms realistic facial features into artistic representations, enhancing user engagement across gaming and virtual reality applications. While 3D-aware generators have made significant advancements, many 3D stylization methods primarily provide near-frontal views and struggle to preserve the unique identities of original subjects, often resulting in outputs that lack diversity and individuality. This paper addresses these challenges by leveraging the PanoHead model, synthesizing images from a comprehensive 360-degree perspective. We propose a novel framework that employs negative log-likelihood distillation (LD) to enhance identity preservation and improve stylization quality. By integrating multi-view grid score and mirror gradients within the 3D GAN architecture and introducing a score rank weighing technique, our approach achieves substantial qualitative and quantitative improvements. Our findings not only advance the state of 3D head stylization but also provide valuable insights into effective distillation processes between diffusion models and GANs, focusing on the critical issue of identity preservation. Please visit the this https URL for more visuals.

Title: Metacognition for Unknown Situations and Environments (MUSE)

• Authors: Rodolfo Valiente, Praveen K. Pilly
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13537
• Pdf URL: https://arxiv.org/pdf/2411.13537
• Copy Paste: [[2411.13537]] Metacognition for Unknown Situations and Environments (MUSE)(https://arxiv.org/abs/2411.13537)
• Keywords: large language model
• Abstract: Metacognition--the awareness and regulation of one's cognitive processes--is central to human adaptability in unknown situations. In contrast, current autonomous agents often struggle in novel environments due to their limited capacity for adaptation. We hypothesize that metacognition is a critical missing ingredient in adaptive autonomous systems, equipping them with the cognitive flexibility needed to tackle unfamiliar challenges. Given the broad scope of metacognitive abilities, we focus on two key aspects: competence awareness and strategy selection for novel tasks. To this end, we propose the Metacognition for Unknown Situations and Environments (MUSE) framework, which integrates metacognitive processes--specifically self-awareness and self-regulation--into autonomous agents. We present two initial implementations of MUSE: one based on world modeling and another leveraging large language models (LLMs), both instantiating the metacognitive cycle. Our system continuously learns to assess its competence on a given task and uses this self-awareness to guide iterative cycles of strategy selection. MUSE agents show significant improvements in self-awareness and self-regulation, enabling them to solve novel, out-of-distribution tasks more effectively compared to Dreamer-v3-based reinforcement learning and purely prompt-based LLM agent approaches. This work highlights the promise of approaches inspired by cognitive and neural systems in enabling autonomous systems to adapt to new environments, overcoming the limitations of current methods that rely heavily on extensive training data.

Title: DIS-Mine: Instance Segmentation for Disaster-Awareness in Poor-Light Condition in Underground Mines

• Authors: Mizanur Rahman Jewel, Mohamed Elmahallawy, Sanjay Madria, Samuel Frimpong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13544
• Pdf URL: https://arxiv.org/pdf/2411.13544
• Copy Paste: [[2411.13544]] DIS-Mine: Instance Segmentation for Disaster-Awareness in Poor-Light Condition in Underground Mines(https://arxiv.org/abs/2411.13544)
• Keywords: segmentation
• Abstract: Detecting disasters in underground mining, such as explosions and structural damage, has been a persistent challenge over the years. This problem is compounded for first responders, who often have no clear information about the extent or nature of the damage within the mine. The poor-light or even total darkness inside the mines makes rescue efforts incredibly difficult, leading to a tragic loss of life. In this paper, we propose a novel instance segmentation method called DIS-Mine, specifically designed to identify disaster-affected areas within underground mines under low-light or poor visibility conditions, aiding first responders in rescue efforts. DIS-Mine is capable of detecting objects in images, even in complete darkness, by addressing challenges such as high noise, color distortions, and reduced contrast. The key innovations of DIS-Mine are built upon four core components: i) Image brightness improvement, ii) Instance segmentation with SAM integration, iii) Mask R-CNN-based segmentation, and iv) Mask alignment with feature matching. On top of that, we have collected real-world images from an experimental underground mine, introducing a new dataset named ImageMine, specifically gathered in low-visibility conditions. This dataset serves to validate the performance of DIS-Mine in realistic, challenging environments. Our comprehensive experiments on the ImageMine dataset, as well as on various other datasets demonstrate that DIS-Mine achieves a superior F1 score of 86.0% and mIoU of 72.0%, outperforming state-of-the-art instance segmentation methods, with at least 15x improvement and up to 80% higher precision in object detection.

Title: HF-Diff: High-Frequency Perceptual Loss and Distribution Matching for One-Step Diffusion-Based Image Super-Resolution

• Authors: Shoaib Meraj Sami, Md Mahedi Hasan, Jeremy Dawson, Nasser Nasrabadi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13548
• Pdf URL: https://arxiv.org/pdf/2411.13548
• Copy Paste: [[2411.13548]] HF-Diff: High-Frequency Perceptual Loss and Distribution Matching for One-Step Diffusion-Based Image Super-Resolution(https://arxiv.org/abs/2411.13548)
• Keywords: diffusion
• Abstract: Although recent diffusion-based single-step super-resolution methods achieve better performance as compared to SinSR, they are computationally complex. To improve the performance of SinSR, we investigate preserving the high-frequency detail features during super-resolution (SR) because the downgraded images lack detailed information. For this purpose, we introduce a high-frequency perceptual loss by utilizing an invertible neural network (INN) pretrained on the ImageNet dataset. Different feature maps of pretrained INN produce different high-frequency aspects of an image. During the training phase, we impose to preserve the high-frequency features of super-resolved and ground truth (GT) images that improve the SR image quality during inference. Furthermore, we also utilize the Jenson-Shannon divergence between GT and SR images in the pretrained DINO-v2 embedding space to match their distribution. By introducing the $\textbf{h}igh$- $\textbf{f}requency$ preserving loss and distribution matching constraint in the single-step $\textbf{diff}usion-based$ SR ($\textbf{HF-Diff}$), we achieve a state-of-the-art CLIPIQA score in the benchmark RealSR, RealSet65, DIV2K-Val, and ImageNet datasets. Furthermore, the experimental results in several datasets demonstrate that our high-frequency perceptual loss yields better SR image quality than LPIPS and VGG-based perceptual losses. Our code will be released at this https URL.

Title: Find Any Part in 3D

• Authors: Ziqi Ma, Yisong Yue, Georgia Gkioxari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13550
• Pdf URL: https://arxiv.org/pdf/2411.13550
• Copy Paste: [[2411.13550]] Find Any Part in 3D(https://arxiv.org/abs/2411.13550)
• Keywords: segmentation
• Abstract: We study open-world part segmentation in 3D: segmenting any part in any object based on any text query. Prior methods are limited in object categories and part vocabularies. Recent advances in AI have demonstrated effective open-world recognition capabilities in 2D. Inspired by this progress, we propose an open-world, direct-prediction model for 3D part segmentation that can be applied zero-shot to any object. Our approach, called Find3D, trains a general-category point embedding model on large-scale 3D assets from the internet without any human annotation. It combines a data engine, powered by foundation models for annotating data, with a contrastive training method. We achieve strong performance and generalization across multiple datasets, with up to a 3x improvement in mIoU over the next best method. Our model is 6x to over 300x faster than existing baselines. To encourage research in general-category open-world 3D part segmentation, we also release a benchmark for general objects and parts. Project website: this https URL

Title: REDUCIO! Generating 1024$\times$1024 Video within 16 Seconds using Extremely Compressed Motion Latents

• Authors: Rui Tian, Qi Dai, Jianmin Bao, Kai Qiu, Yifan Yang, Chong Luo, Zuxuan Wu, Yu-Gang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13552
• Pdf URL: https://arxiv.org/pdf/2411.13552
• Copy Paste: [[2411.13552]] REDUCIO! Generating 1024$\times$1024 Video within 16 Seconds using Extremely Compressed Motion Latents(https://arxiv.org/abs/2411.13552)
• Keywords: diffusion
• Abstract: Commercial video generation models have exhibited realistic, high-fidelity results but are still restricted to limited access. One crucial obstacle for large-scale applications is the expensive training and inference cost. In this paper, we argue that videos contain much more redundant information than images, thus can be encoded by very few motion latents based on a content image. Towards this goal, we design an image-conditioned VAE to encode a video to an extremely compressed motion latent space. This magic Reducio charm enables 64x reduction of latents compared to a common 2D VAE, without sacrificing the quality. Training diffusion models on such a compact representation easily allows for generating 1K resolution videos. We then adopt a two-stage video generation paradigm, which performs text-to-image and text-image-to-video sequentially. Extensive experiments show that our Reducio-DiT achieves strong performance in evaluation, though trained with limited GPU resources. More importantly, our method significantly boost the efficiency of video LDMs both in training and inference. We train Reducio-DiT in around 3.2K training hours in total and generate a 16-frame 1024*1024 video clip within 15.5 seconds on a single A100 GPU. Code released at this https URL .

Title: AI-generated Image Detection: Passive or Watermark?

• Authors: Moyang Guo, Yuepeng Hu, Zhengyuan Jiang, Zeyu Li, Amir Sadovnik, Arka Daw, Neil Gong
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13553
• Pdf URL: https://arxiv.org/pdf/2411.13553
• Copy Paste: [[2411.13553]] AI-generated Image Detection: Passive or Watermark?(https://arxiv.org/abs/2411.13553)
• Keywords: robust, watermark
• Abstract: While text-to-image models offer numerous benefits, they also pose significant societal risks. Detecting AI-generated images is crucial for mitigating these risks. Detection methods can be broadly categorized into passive and watermark-based approaches: passive detectors rely on artifacts present in AI-generated images, whereas watermark-based detectors proactively embed watermarks into such images. A key question is which type of detector performs better in terms of effectiveness, robustness, and efficiency. However, the current literature lacks a comprehensive understanding of this issue. In this work, we aim to bridge that gap by developing ImageDetectBench, the first comprehensive benchmark to compare the effectiveness, robustness, and efficiency of passive and watermark-based detectors. Our benchmark includes four datasets, each containing a mix of AI-generated and non-AI-generated images. We evaluate five passive detectors and four watermark-based detectors against eight types of common perturbations and three types of adversarial perturbations. Our benchmark results reveal several interesting findings. For instance, watermark-based detectors consistently outperform passive detectors, both in the presence and absence of perturbations. Based on these insights, we provide recommendations for detecting AI-generated images, e.g., when both types of detectors are applicable, watermark-based detectors should be the preferred choice.