2024-11-13

Title: ViTOC: Vision Transformer and Object-aware Captioner

• Authors: Feiyang Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07265
• Pdf URL: https://arxiv.org/pdf/2411.07265
• Copy Paste: [[2411.07265]] ViTOC: Vision Transformer and Object-aware Captioner(https://arxiv.org/abs/2411.07265)
• Keywords: transformer
• Abstract: This paper presents ViTOC (Vision Transformer and Object-aware Captioner), a novel vision-language model for image captioning that addresses the challenges of accuracy and diversity in generated descriptions. Unlike conventional approaches, ViTOC employs a dual-path architecture based on Vision Transformer and object detector, effectively fusing global visual features and local object information through learnable vectors. The model introduces an innovative object-aware prompting strategy that significantly enhances its capability in handling long-tail data. Experiments on the standard COCO dataset demonstrate that ViTOC outperforms baseline models across all evaluation metrics, achieving 71.26 and 17.82 on CIDEr and SPICE, respectively. Additionally, we propose a reference-free evaluation method based on CLIP to further validate the model's effectiveness. By utilizing pretrained visual model parameters, ViTOC achieves efficient end-to-end training.

Title: Target-driven Attack for Large Language Models

• Authors: Chong Zhang, Mingyu Jin, Dong Shu, Taowen Wang, Dongfang Liu, Xiaobo Jin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07268
• Pdf URL: https://arxiv.org/pdf/2411.07268
• Copy Paste: [[2411.07268]] Target-driven Attack for Large Language Models(https://arxiv.org/abs/2411.07268)
• Keywords: security, attack, robust, large language model
• Abstract: Current large language models (LLM) provide a strong foundation for large-scale user-oriented natural language tasks. Many users can easily inject adversarial text or instructions through the user interface, thus causing LLM model security challenges like the language model not giving the correct answer. Although there is currently a large amount of research on black-box attacks, most of these black-box attacks use random and heuristic strategies. It is unclear how these strategies relate to the success rate of attacks and thus effectively improve model robustness. To solve this problem, we propose our target-driven black-box attack method to maximize the KL divergence between the conditional probabilities of the clean text and the attack text to redefine the attack's goal. We transform the distance maximization problem into two convex optimization problems based on the attack goal to solve the attack text and estimate the covariance. Furthermore, the projected gradient descent algorithm solves the vector corresponding to the attack text. Our target-driven black-box attack approach includes two attack strategies: token manipulation and misinformation attack. Experimental results on multiple Large Language Models and datasets demonstrate the effectiveness of our attack method.

Title: Learning From Graph-Structured Data: Addressing Design Issues and Exploring Practical Applications in Graph Representation Learning

• Authors: Chenqing Hua
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07269
• Pdf URL: https://arxiv.org/pdf/2411.07269
• Copy Paste: [[2411.07269]] Learning From Graph-Structured Data: Addressing Design Issues and Exploring Practical Applications in Graph Representation Learning(https://arxiv.org/abs/2411.07269)
• Keywords: generative
• Abstract: Graphs serve as fundamental descriptors for systems composed of interacting elements, capturing a wide array of data types, from molecular interactions to social networks and knowledge graphs. In this paper, we present an exhaustive review of the latest advancements in graph representation learning and Graph Neural Networks (GNNs). GNNs, tailored to handle graph-structured data, excel in deriving insights and predictions from intricate relational information, making them invaluable for tasks involving such data. Graph representation learning, a pivotal approach in analyzing graph-structured data, facilitates numerous downstream tasks and applications across machine learning, data mining, biomedicine, and healthcare. Our work delves into the capabilities of GNNs, examining their foundational designs and their application in addressing real-world challenges. We introduce a GNN equipped with an advanced high-order pooling function, adept at capturing complex node interactions within graph-structured data. This pooling function significantly enhances the GNN's efficacy in both node- and graph-level tasks. Additionally, we propose a molecular graph generative model with a GNN as its core framework. This GNN backbone is proficient in learning invariant and equivariant molecular characteristics. Employing these features, the molecular graph generative model is capable of simultaneously learning and generating molecular graphs with atom-bond structures and precise atom positions. Our models undergo thorough experimental evaluations and comparisons with established methods, showcasing their superior performance in addressing diverse real-world challenges with various datasets.

Title: X-DFS: Explainable Artificial Intelligence Guided Design-for-Security Solution Space Exploration

• Authors: Tanzim Mahfuz, Swarup Bhunia, Prabuddha Chakraborty
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07308
• Pdf URL: https://arxiv.org/pdf/2411.07308
• Copy Paste: [[2411.07308]] X-DFS: Explainable Artificial Intelligence Guided Design-for-Security Solution Space Exploration(https://arxiv.org/abs/2411.07308)
• Keywords: security, attack, robust
• Abstract: Design and manufacturing of integrated circuits predominantly use a globally distributed semiconductor supply chain involving diverse entities. The modern semiconductor supply chain has been designed to boost production efficiency, but is filled with major security concerns such as malicious modifications (hardware Trojans), reverse engineering (RE), and cloning. While being deployed, digital systems are also subject to a plethora of threats such as power, timing, and electromagnetic (EM) side channel attacks. Many Design-for-Security (DFS) solutions have been proposed to deal with these vulnerabilities, and such solutions (DFS) relays on strategic modifications (e.g., logic locking, side channel resilient masking, and dummy logic insertion) of the digital designs for ensuring a higher level of security. However, most of these DFS strategies lack robust formalism, are often not human-understandable, and require an extensive amount of human expert effort during their development/use. All of these factors make it difficult to keep up with the ever growing number of microelectronic vulnerabilities. In this work, we propose X-DFS, an explainable Artificial Intelligence (AI) guided DFS solution-space exploration approach that can dramatically cut down the mitigation strategy development/use time while enriching our understanding of the vulnerability by providing human-understandable decision rationale. We implement X-DFS and comprehensively evaluate it for reverse engineering threats (SAIL, SWEEP, and OMLA) and formalize a generalized mechanism for applying X-DFS to defend against other threats such as hardware Trojans, fault attacks, and side channel attacks for seamless future extensions.

Title: Anomaly Detection in OKTA Logs using Autoencoders

• Authors: Jericho Cain, Hayden Beadles, Karthik Venkatesan
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.07314
• Pdf URL: https://arxiv.org/pdf/2411.07314
• Copy Paste: [[2411.07314]] Anomaly Detection in OKTA Logs using Autoencoders(https://arxiv.org/abs/2411.07314)
• Keywords: security
• Abstract: Okta logs are used today to detect cybersecurity events using various rule-based models with restricted look back periods. These functions have limitations, such as a limited retrospective analysis, a predefined rule set, and susceptibility to generating false positives. To address this, we adopt unsupervised techniques, specifically employing autoencoders. To properly use an autoencoder, we need to transform and simplify the complexity of the log data we receive from our users. This transformed and filtered data is then fed into the autoencoder, and the output is evaluated.

Title: SynRL: Aligning Synthetic Clinical Trial Data with Human-preferred Clinical Endpoints Using Reinforcement Learning

• Authors: Trisha Das, Zifeng Wang, Afrah Shafquat, Mandis Beigi, Jason Mezey, Jimeng Sun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07317
• Pdf URL: https://arxiv.org/pdf/2411.07317
• Copy Paste: [[2411.07317]] SynRL: Aligning Synthetic Clinical Trial Data with Human-preferred Clinical Endpoints Using Reinforcement Learning(https://arxiv.org/abs/2411.07317)
• Keywords: privacy
• Abstract: Each year, hundreds of clinical trials are conducted to evaluate new medical interventions, but sharing patient records from these trials with other institutions can be challenging due to privacy concerns and federal regulations. To help mitigate privacy concerns, researchers have proposed methods for generating synthetic patient data. However, existing approaches for generating synthetic clinical trial data disregard the usage requirements of these data, including maintaining specific properties of clinical outcomes, and only use post hoc assessments that are not coupled with the data generation process. In this paper, we propose SynRL which leverages reinforcement learning to improve the performance of patient data generators by customizing the generated data to meet the user-specified requirements for synthetic data outcomes and endpoints. Our method includes a data value critic function to evaluate the quality of the generated data and uses reinforcement learning to align the data generator with the users' needs based on the critic's feedback. We performed experiments on four clinical trial datasets and demonstrated the advantages of SynRL in improving the quality of the generated synthetic data while keeping the privacy risks low. We also show that SynRL can be utilized as a general framework that can customize data generation of multiple types of synthetic data generators. Our code is available at this https URL.

Title: Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendations

• Authors: Kirti Bhagat, Kinshuk Vasisht, Danish Pruthi
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07320
• Pdf URL: https://arxiv.org/pdf/2411.07320
• Copy Paste: [[2411.07320]] Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendations(https://arxiv.org/abs/2411.07320)
• Keywords: large language model
• Abstract: While a large body of work inspects language models for biases concerning gender, race, occupation and religion, biases of geographical nature are relatively less explored. Some recent studies benchmark the degree to which large language models encode geospatial knowledge. However, the impact of the encoded geographical knowledge (or lack thereof) on real-world applications has not been documented. In this work, we examine large language models for two common scenarios that require geographical knowledge: (a) travel recommendations and (b) geo-anchored story generation. Specifically, we study four popular language models, and across about $100$K travel requests, and $200$K story generations, we observe that travel recommendations corresponding to poorer countries are less unique with fewer location references, and stories from these regions more often convey emotions of hardship and sadness compared to those from wealthier nations.

Title: $SE(3)$ Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation

• Authors: Yinshuang Xu, Dian Chen, Katherine Liu, Sergey Zakharov, Rares Ambrus, Kostas Daniilidis, Vitor Guizilini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07326
• Pdf URL: https://arxiv.org/pdf/2411.07326
• Copy Paste: [[2411.07326]] $SE(3)$ Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation(https://arxiv.org/abs/2411.07326)
• Keywords: robust
• Abstract: Incorporating inductive bias by embedding geometric entities (such as rays) as input has proven successful in multi-view learning. However, the methods adopting this technique typically lack equivariance, which is crucial for effective 3D learning. Equivariance serves as a valuable inductive prior, aiding in the generation of robust multi-view features for 3D scene understanding. In this paper, we explore the application of equivariant multi-view learning to depth estimation, not only recognizing its significance for computer vision and robotics but also addressing the limitations of previous research. Most prior studies have either overlooked equivariance in this setting or achieved only approximate equivariance through data augmentation, which often leads to inconsistencies across different reference frames. To address this issue, we propose to embed $SE(3)$ equivariance into the Perceiver IO architecture. We employ Spherical Harmonics for positional encoding to ensure 3D rotation equivariance, and develop a specialized equivariant encoder and decoder within the Perceiver IO architecture. To validate our model, we applied it to the task of stereo depth estimation, achieving state of the art results on real-world datasets without explicit geometric constraints or extensive data augmentation.

Title: Multimodal Fusion Balancing Through Game-Theoretic Regularization

• Authors: Konstantinos Kontras, Thomas Strypsteen, Christos Chatzichristos, Paul P. Liang, Matthew Blaschko, Maarten De Vos
• Subjects: cs.LG, cs.AI, cs.CV, cs.GT, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.07335
• Pdf URL: https://arxiv.org/pdf/2411.07335
• Copy Paste: [[2411.07335]] Multimodal Fusion Balancing Through Game-Theoretic Regularization(https://arxiv.org/abs/2411.07335)
• Keywords: extraction
• Abstract: Multimodal learning can complete the picture of information extraction by uncovering key dependencies between data sources. However, current systems fail to fully leverage multiple modalities for optimal performance. This has been attributed to modality competition, where modalities strive for training resources, leaving some underoptimized. We show that current balancing methods struggle to train multimodal models that surpass even simple baselines, such as ensembles. This raises the question: how can we ensure that all modalities in multimodal training are sufficiently trained, and that learning from new modalities consistently improves performance? This paper proposes the Multimodal Competition Regularizer (MCR), a new loss component inspired by mutual information (MI) decomposition designed to prevent the adverse effects of competition in multimodal training. Our key contributions are: 1) Introducing game-theoretic principles in multimodal learning, where each modality acts as a player competing to maximize its influence on the final outcome, enabling automatic balancing of the MI terms. 2) Refining lower and upper bounds for each MI term to enhance the extraction of task-relevant unique and shared information across modalities. 3) Suggesting latent space permutations for conditional MI estimation, significantly improving computational efficiency. MCR outperforms all previously suggested training strategies and is the first to consistently improve multimodal learning beyond the ensemble baseline, clearly demonstrating that combining modalities leads to significant performance gains on both synthetic and large real-world datasets.

Title: SetLexSem Challenge: Using Set Operations to Evaluate the Lexical and Semantic Robustness of Language Models

• Authors: Bardiya Akhbari, Manish Gawali, Nicholas A. Dronen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07336
• Pdf URL: https://arxiv.org/pdf/2411.07336
• Copy Paste: [[2411.07336]] SetLexSem Challenge: Using Set Operations to Evaluate the Lexical and Semantic Robustness of Language Models(https://arxiv.org/abs/2411.07336)
• Keywords: robust, large language model
• Abstract: Set theory is foundational to mathematics and, when sets are finite, to reasoning about the world. An intelligent system should perform set operations consistently, regardless of superficial variations in the operands. Initially designed for semantically-oriented NLP tasks, large language models (LLMs) are now being evaluated on algorithmic tasks. Because sets are comprised of arbitrary symbols (e.g. numbers, words), they provide an opportunity to test, systematically, the invariance of LLMs' algorithmic abilities under simple lexical or semantic variations. To this end, we present the SetLexSem Challenge, a synthetic benchmark that evaluates the performance of LLMs on set operations. SetLexSem assesses the robustness of LLMs' instruction-following abilities under various conditions, focusing on the set operations and the nature and construction of the set members. Evaluating seven LLMs with SetLexSem, we find that they exhibit poor robustness to variation in both operation and operands. We show -- via the framework's systematic sampling of set members along lexical and semantic dimensions -- that LLMs are not only not robust to variation along these dimensions but demonstrate unique failure modes in particular, easy-to-create semantic groupings of "deceptive" sets. We find that rigorously measuring language model robustness to variation in frequency and length is challenging and present an analysis that measures them independently. The code for reproducing the results of this paper, and for generating the SetLexSem Challenge dataset, is available at \href{this https URL}{this https URL}.

Title: Warmstarting for Scaling Language Models

• Authors: Neeratyoy Mallik, Maciej Janowski, Johannes Hog, Herilalaina Rakotoarison, Aaron Klein, Josif Grabocka, Frank Hutter
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07340
• Pdf URL: https://arxiv.org/pdf/2411.07340
• Copy Paste: [[2411.07340]] Warmstarting for Scaling Language Models(https://arxiv.org/abs/2411.07340)
• Keywords: large language model
• Abstract: Scaling model sizes to scale performance has worked remarkably well for the current large language models paradigm. The research and empirical findings of various scaling studies led to novel scaling results and laws that guides subsequent research. High training costs for contemporary scales of data and models result in a lack of thorough understanding of how to tune and arrive at such training setups. One direction to ameliorate the cost of pretraining large models is to warmstart the large-scale training from smaller models that are cheaper to tune. In this work, we attempt to understand if the behavior of optimal hyperparameters can be retained under warmstarting for scaling. We explore simple operations that allow the application of theoretically motivated methods of zero-shot transfer of optimal hyperparameters using {\mu}Transfer. We investigate the aspects that contribute to the speedup in convergence and the preservation of stable training dynamics under warmstarting with {\mu}Transfer. We find that shrinking smaller model weights, zero-padding, and perturbing the resulting larger model with scaled initialization from {\mu}P enables effective warmstarting of $\mut{}$.

Title: Exploring Variational Autoencoders for Medical Image Generation: A Comprehensive Study

• Authors: Khadija Rais, Mohamed Amroune, Abdelmadjid Benmachiche, Mohamed Yassine Haouam
• Subjects: cs.LG, cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.07348
• Pdf URL: https://arxiv.org/pdf/2411.07348
• Copy Paste: [[2411.07348]] Exploring Variational Autoencoders for Medical Image Generation: A Comprehensive Study(https://arxiv.org/abs/2411.07348)
• Keywords: generative, segmentation
• Abstract: Variational autoencoder (VAE) is one of the most common techniques in the field of medical image generation, where this architecture has shown advanced researchers in recent years and has developed into various architectures. VAE has advantages including improving datasets by adding samples in smaller datasets and in datasets with imbalanced classes, and this is how data augmentation works. This paper provides a comprehensive review of studies on VAE in medical imaging, with a special focus on their ability to create synthetic images close to real data so that they can be used for data augmentation. This study reviews important architectures and methods used to develop VAEs for medical images and provides a comparison with other generative models such as GANs on issues such as image quality, and low diversity of generated samples. We discuss recent developments and applications in several medical fields highlighting the ability of VAEs to improve segmentation and classification accuracy.

Title: Federated Learning Client Pruning for Noisy Labels

• Authors: Mahdi Morafah, Hojin Chang, Chen Chen, Bill Lin
• Subjects: cs.LG, cs.AI, cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.07391
• Pdf URL: https://arxiv.org/pdf/2411.07391
• Copy Paste: [[2411.07391]] Federated Learning Client Pruning for Noisy Labels(https://arxiv.org/abs/2411.07391)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) enables collaborative model training across decentralized edge devices while preserving data privacy. However, existing FL methods often assume clean annotated datasets, impractical for resource-constrained edge devices. In reality, noisy labels are prevalent, posing significant challenges to FL performance. Prior approaches attempt label correction and robust training techniques but exhibit limited efficacy, particularly under high noise levels. This paper introduces ClipFL (Federated Learning Client Pruning), a novel framework addressing noisy labels from a fresh perspective. ClipFL identifies and excludes noisy clients based on their performance on a clean validation dataset, tracked using a Noise Candidacy Score (NCS). The framework comprises three phases: pre-client pruning to identify potential noisy clients and calculate their NCS, client pruning to exclude a percentage of clients with the highest NCS, and post-client pruning for fine-tuning the global model with standard FL on clean clients. Empirical evaluation demonstrates ClipFL's efficacy across diverse datasets and noise levels, achieving accurate noisy client identification, superior performance, faster convergence, and reduced communication costs compared to state-of-the-art FL methods. Our code is available at this https URL.

Title: Feature-Space Semantic Invariance: Enhanced OOD Detection for Open-Set Domain Generalization

• Authors: Haoliang Wang, Chen Zhao, Feng Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07392
• Pdf URL: https://arxiv.org/pdf/2411.07392
• Copy Paste: [[2411.07392]] Feature-Space Semantic Invariance: Enhanced OOD Detection for Open-Set Domain Generalization(https://arxiv.org/abs/2411.07392)
• Keywords: robust, generative
• Abstract: Open-set domain generalization addresses a real-world challenge: training a model to generalize across unseen domains (domain generalization) while also detecting samples from unknown classes not encountered during training (open-set recognition). However, most existing approaches tackle these issues separately, limiting their practical applicability. To overcome this limitation, we propose a unified framework for open-set domain generalization by introducing Feature-space Semantic Invariance (FSI). FSI maintains semantic consistency across different domains within the feature space, enabling more accurate detection of OOD instances in unseen domains. Additionally, we adopt a generative model to produce synthetic data with novel domain styles or class labels, enhancing model robustness. Initial experiments show that our method improves AUROC by 9.1% to 18.9% on ColoredMNIST, while also significantly increasing in-distribution classification accuracy.

Title: Toward Optimal Search and Retrieval for RAG

• Authors: Alexandria Leto, Cecilia Aguerrebere, Ishwar Bhati, Ted Willke, Mariano Tepper, Vy Ai Vo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07396
• Pdf URL: https://arxiv.org/pdf/2411.07396
• Copy Paste: [[2411.07396]] Toward Optimal Search and Retrieval for RAG(https://arxiv.org/abs/2411.07396)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) is a promising method for addressing some of the memory-related challenges associated with Large Language Models (LLMs). Two separate systems form the RAG pipeline, the retriever and the reader, and the impact of each on downstream task performance is not well-understood. Here, we work towards the goal of understanding how retrievers can be optimized for RAG pipelines for common tasks such as Question Answering (QA). We conduct experiments focused on the relationship between retrieval and RAG performance on QA and attributed QA and unveil a number of insights useful to practitioners developing high-performance RAG pipelines. For example, lowering search accuracy has minor implications for RAG performance while potentially increasing retrieval speed and memory efficiency.

Title: Beyond Keywords: A Context-based Hybrid Approach to Mining Ethical Concern-related App Reviews

• Authors: Aakash Sorathiya, Gouri Ginde
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.07398
• Pdf URL: https://arxiv.org/pdf/2411.07398
• Copy Paste: [[2411.07398]] Beyond Keywords: A Context-based Hybrid Approach to Mining Ethical Concern-related App Reviews(https://arxiv.org/abs/2411.07398)
• Keywords: privacy, extraction, large language model
• Abstract: With the increasing proliferation of mobile applications in our everyday experiences, the concerns surrounding ethics have surged significantly. Users generally communicate their feedback, report issues, and suggest new functionalities in application (app) reviews, frequently emphasizing safety, privacy, and accountability concerns. Incorporating these reviews is essential to developing successful products. However, app reviews related to ethical concerns generally use domain-specific language and are expressed using a more varied vocabulary. Thus making automated ethical concern-related app review extraction a challenging and time-consuming effort. This study proposes a novel Natural Language Processing (NLP) based approach that combines Natural Language Inference (NLI), which provides a deep comprehension of language nuances, and a decoder-only (LLaMA-like) Large Language Model (LLM) to extract ethical concern-related app reviews at scale. Utilizing 43,647 app reviews from the mental health domain, the proposed methodology 1) Evaluates four NLI models to extract potential privacy reviews and compares the results of domain-specific privacy hypotheses with generic privacy hypotheses; 2) Evaluates four LLMs for classifying app reviews to privacy concerns; and 3) Uses the best NLI and LLM models further to extract new privacy reviews from the dataset. Results show that the DeBERTa-v3-base-mnli-fever-anli NLI model with domain-specific hypotheses yields the best performance, and Llama3.1-8B-Instruct LLM performs best in the classification of app reviews. Then, using NLI+LLM, an additional 1,008 new privacy-related reviews were extracted that were not identified through the keyword-based approach in previous research, thus demonstrating the effectiveness of the proposed approach.

Title: Using Generative AI and Multi-Agents to Provide Automatic Feedback

• Authors: Shuchen Guo, Ehsan Latif, Yifan Zhou, Xuan Huang, Xiaoming Zhai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07407
• Pdf URL: https://arxiv.org/pdf/2411.07407
• Copy Paste: [[2411.07407]] Using Generative AI and Multi-Agents to Provide Automatic Feedback(https://arxiv.org/abs/2411.07407)
• Keywords: generative, large language model
• Abstract: This study investigates the use of generative AI and multi-agent systems to provide automatic feedback in educational contexts, particularly for student constructed responses in science assessments. The research addresses a key gap in the field by exploring how multi-agent systems, called AutoFeedback, can improve the quality of GenAI-generated feedback, overcoming known issues such as over-praise and over-inference that are common in single-agent large language models (LLMs). The study developed a multi-agent system consisting of two AI agents: one for generating feedback and another for validating and refining it. The system was tested on a dataset of 240 student responses, and its performance was compared to that of a single-agent LLM. Results showed that AutoFeedback significantly reduced the occurrence of over-praise and over-inference errors, providing more accurate and pedagogically sound feedback. The findings suggest that multi-agent systems can offer a more reliable solution for generating automated feedback in educational settings, highlighting their potential for scalable and personalized learning support. These results have important implications for educators and researchers seeking to leverage AI in formative assessments, offering a pathway to more effective feedback mechanisms that enhance student learning outcomes.

Title: XPoint: A Self-Supervised Visual-State-Space based Architecture for Multispectral Image Registration

• Authors: Ismail Can Yagmur, Hasan F. Ates, Bahadir K. Gunturk
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07430
• Pdf URL: https://arxiv.org/pdf/2411.07430
• Copy Paste: [[2411.07430]] XPoint: A Self-Supervised Visual-State-Space based Architecture for Multispectral Image Registration(https://arxiv.org/abs/2411.07430)
• Keywords: robust, extraction, segmentation
• Abstract: Accurate multispectral image matching presents significant challenges due to non-linear intensity variations across spectral modalities, extreme viewpoint changes, and the scarcity of labeled datasets. Current state-of-the-art methods are typically specialized for a single spectral difference, such as visibleinfrared, and struggle to adapt to other modalities due to their reliance on expensive supervision, such as depth maps or camera poses. To address the need for rapid adaptation across modalities, we introduce XPoint, a self-supervised, modular image-matching framework designed for adaptive training and fine-tuning on aligned multispectral datasets, allowing users to customize key components based on their specific tasks. XPoint employs modularity and self-supervision to allow for the adjustment of elements such as the base detector, which generates pseudoground truth keypoints invariant to viewpoint and spectrum variations. The framework integrates a VMamba encoder, pretrained on segmentation tasks, for robust feature extraction, and includes three joint decoder heads: two are dedicated to interest point and descriptor extraction; and a task-specific homography regression head imposes geometric constraints for superior performance in tasks like image registration. This flexible architecture enables quick adaptation to a wide range of modalities, demonstrated by training on Optical-Thermal data and fine-tuning on settings such as visual-near infrared, visual-infrared, visual-longwave infrared, and visual-synthetic aperture radar. Experimental results show that XPoint consistently outperforms or matches state-ofthe-art methods in feature matching and image registration tasks across five distinct multispectral datasets. Our source code is available at this https URL.

Title: SDN-Based Smart Cyber Switching (SCS) for Cyber Restoration of a Digital Substation

• Authors: Mansi Girdhar, Kuchan Park, Wencong Su, Junho Hong, Akila Herath, Chen-Ching Liu
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2411.07433
• Pdf URL: https://arxiv.org/pdf/2411.07433
• Copy Paste: [[2411.07433]] SDN-Based Smart Cyber Switching (SCS) for Cyber Restoration of a Digital Substation(https://arxiv.org/abs/2411.07433)
• Keywords: security, attack
• Abstract: In recent years, critical infrastructure and power grids have increasingly been targets of cyber-attacks, causing widespread and extended blackouts. Digital substations are particularly vulnerable to such cyber incursions, jeopardizing grid stability. This paper addresses these risks by proposing a cybersecurity framework that leverages software-defined networking (SDN) to bolster the resilience of substations based on the IEC-61850 standard. The research introduces a strategy involving smart cyber switching (SCS) for mitigation and concurrent intelligent electronic device (CIED) for restoration, ensuring ongoing operational integrity and cybersecurity within a substation. The SCS framework improves the physical network's behavior (i.e., leveraging commercial SDN capabilities) by incorporating an adaptive port controller (APC) module for dynamic port management and an intrusion detection system (IDS) to detect and counteract malicious IEC-61850-based sampled value (SV) and generic object-oriented system event (GOOSE) messages within the substation's communication network. The framework's effectiveness is validated through comprehensive simulations and a hardware-in-the-loop (HIL) testbed, demonstrating its ability to sustain substation operations during cyber-attacks and significantly improve the overall resilience of the power grid.

Title: All-in-one Weather-degraded Image Restoration via Adaptive Degradation-aware Self-prompting Model

• Authors: Yuanbo Wen, Tao Gao, Ziqi Li, Jing Zhang, Kaihao Zhang, Ting Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07445
• Pdf URL: https://arxiv.org/pdf/2411.07445
• Copy Paste: [[2411.07445]] All-in-one Weather-degraded Image Restoration via Adaptive Degradation-aware Self-prompting Model(https://arxiv.org/abs/2411.07445)
• Keywords: diffusion
• Abstract: Existing approaches for all-in-one weather-degraded image restoration suffer from inefficiencies in leveraging degradation-aware priors, resulting in sub-optimal performance in adapting to different weather conditions. To this end, we develop an adaptive degradation-aware self-prompting model (ADSM) for all-in-one weather-degraded image restoration. Specifically, our model employs the contrastive language-image pre-training model (CLIP) to facilitate the training of our proposed latent prompt generators (LPGs), which represent three types of latent prompts to characterize the degradation type, degradation property and image caption. Moreover, we integrate the acquired degradation-aware prompts into the time embedding of diffusion model to improve degradation perception. Meanwhile, we employ the latent caption prompt to guide the reverse sampling process using the cross-attention mechanism, thereby guiding the accurate image reconstruction. Furthermore, to accelerate the reverse sampling procedure of diffusion model and address the limitations of frequency perception, we introduce a wavelet-oriented noise estimating network (WNE-Net). Extensive experiments conducted on eight publicly available datasets demonstrate the effectiveness of our proposed approach in both task-specific and all-in-one applications.

Title: Efficient and Accurate Prompt Optimization: the Benefit of Memory in Exemplar-Guided Reflection

• Authors: Cilin Yan, Jingyun Wang, Lin Zhang, Ruihui Zhao, Xiaopu Wu, Kai Xiong, Qingsong Liu, Guoliang Kang, Yangyang Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07446
• Pdf URL: https://arxiv.org/pdf/2411.07446
• Copy Paste: [[2411.07446]] Efficient and Accurate Prompt Optimization: the Benefit of Memory in Exemplar-Guided Reflection(https://arxiv.org/abs/2411.07446)
• Keywords: large language model
• Abstract: Automatic prompt engineering aims to enhance the generation quality of large language models (LLMs). Recent works utilize feedbacks generated from erroneous cases to guide the prompt optimization. During inference, they may further retrieve several semantically-related exemplars and concatenate them to the optimized prompts to improve the performance. However, those works only utilize the feedback at the current step, ignoring historical and unseleccted feedbacks which are potentially beneficial. Moreover, the selection of exemplars only considers the general semantic relationship and may not be optimal in terms of task performance and matching with the optimized prompt. In this work, we propose an Exemplar-Guided Reflection with Memory mechanism (ERM) to realize more efficient and accurate prompt optimization. Specifically, we design an exemplar-guided reflection mechanism where the feedback generation is additionally guided by the generated exemplars. We further build two kinds of memory to fully utilize the historical feedback information and support more effective exemplar retrieval. Empirical evaluations show our method surpasses previous state-of-the-arts with less optimization steps, i.e., improving F1 score by 10.1 on LIAR dataset, and reducing half of the optimization steps on ProTeGi.

Title: Tracing the Roots: Leveraging Temporal Dynamics in Diffusion Trajectories for Origin Attribution

• Authors: Andreas Floros, Seyed-Mohsen Moosavi-Dezfooli, Pier Luigi Dragotti
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07449
• Pdf URL: https://arxiv.org/pdf/2411.07449
• Copy Paste: [[2411.07449]] Tracing the Roots: Leveraging Temporal Dynamics in Diffusion Trajectories for Origin Attribution(https://arxiv.org/abs/2411.07449)
• Keywords: robust, diffusion
• Abstract: Diffusion models have revolutionized image synthesis, garnering significant research interest in recent years. Diffusion is an iterative algorithm in which samples are generated step-by-step, starting from pure noise. This process introduces the notion of diffusion trajectories, i.e., paths from the standard Gaussian distribution to the target image distribution. In this context, we study discriminative algorithms operating on these trajectories. Specifically, given a pre-trained diffusion model, we consider the problem of classifying images as part of the training dataset, generated by the model or originating from an external source. Our approach demonstrates the presence of patterns across steps that can be leveraged for classification. We also conduct ablation studies, which reveal that using higher-order gradient features to characterize the trajectories leads to significant performance gains and more robust algorithms.

Title: DecoPrompt : Decoding Prompts Reduces Hallucinations when Large Language Models Meet False Premises

• Authors: Nan Xu, Xuezhe Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07457
• Pdf URL: https://arxiv.org/pdf/2411.07457
• Copy Paste: [[2411.07457]] DecoPrompt : Decoding Prompts Reduces Hallucinations when Large Language Models Meet False Premises(https://arxiv.org/abs/2411.07457)
• Keywords: large language model
• Abstract: While large language models (LLMs) have demonstrated increasing power, they have also called upon studies on their hallucinated outputs that deviate from factually correct statements. In this paper, we focus on one important scenario of false premises, where LLMs are distracted by misaligned claims although the model possesses the required factual knowledge to answer original questions accurately. Inspired by the observation that entropy of the false-premise prompt is closely related to its likelihood to elicit hallucination generation, we propose a new prompting algorithm, named DecoPrompt, to mitigate hallucination. DecoPrompt leverages LLMs to "decode" the false-premise prompts without really eliciting hallucination output from LLMs. We perform experiments on two datasets, demonstrating that DecoPrompt can reduce hallucinations effectively on outputs from different LLMs. Moreover, DecoPrompt exhibits cross-model transferability, which facilitates its applications to scenarios such as LLMs of large sizes or unavailable model logits.

Title: MureObjectStitch: Multi-reference Image Composition

• Authors: Jiaxuan Chen, Bo Zhang, Li Niu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07462
• Pdf URL: https://arxiv.org/pdf/2411.07462
• Copy Paste: [[2411.07462]] MureObjectStitch: Multi-reference Image Composition(https://arxiv.org/abs/2411.07462)
• Keywords: generative
• Abstract: Generative image composition aims to regenerate the given foreground object in the background image to produce a realistic composite image. In this work, we propose an effective finetuning strategy for generative image composition model, in which we finetune a pretrained model using one or more images containing the same foreground object. Moreover, we propose a multi-reference strategy, which allows the model to take in multiple reference images of the foreground object. The experiments on MureCOM dataset verify the effectiveness of our method.

Title: MSEG-VCUQ: Multimodal SEGmentation with Enhanced Vision Foundation Models, Convolutional Neural Networks, and Uncertainty Quantification for High-Speed Video Phase Detection Data

• Authors: Chika Maduabuchi, Ericmoore Jossou, Matteo Bucci
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.07463
• Pdf URL: https://arxiv.org/pdf/2411.07463
• Copy Paste: [[2411.07463]] MSEG-VCUQ: Multimodal SEGmentation with Enhanced Vision Foundation Models, Convolutional Neural Networks, and Uncertainty Quantification for High-Speed Video Phase Detection Data(https://arxiv.org/abs/2411.07463)
• Keywords: robust, extraction, transformer, segmentation
• Abstract: Purpose: High-speed video (HSV) phase detection (PD) segmentation is vital in nuclear reactors, chemical processing, and electronics cooling for detecting vapor, liquid, and microlayer phases. Traditional segmentation models face pixel-level accuracy and generalization issues in multimodal data. MSEG-VCUQ introduces VideoSAM, a hybrid framework leveraging convolutional neural networks (CNNs) and transformer-based vision models to enhance segmentation accuracy and generalizability across complex multimodal PD tasks. Methods: VideoSAM combines U-Net CNN and the Segment Anything Model (SAM) for advanced feature extraction and segmentation across diverse HSV PD modalities, spanning fluids like water, FC-72, nitrogen, and argon under varied heat flux conditions. The framework also incorporates uncertainty quantification (UQ) to assess pixel-based discretization errors, delivering reliable metrics such as contact line density and dry area fraction under experimental conditions. Results: VideoSAM outperforms SAM and modality-specific CNN models in segmentation accuracy, excelling in environments with complex phase boundaries, overlapping bubbles, and dynamic liquid-vapor interactions. Its hybrid architecture supports cross-dataset generalization, adapting effectively to varying modalities. The UQ module provides accurate error estimates, enhancing the reliability of segmentation outputs for advanced HSV PD research. Conclusion: MSEG-VCUQ, via VideoSAM, offers a robust solution for HSV PD segmentation, addressing previous limitations with advanced deep learning and UQ techniques. The open-source datasets and tools introduced enable scalable, precise, and adaptable segmentation for multimodal PD datasets, supporting advancements in HSV analysis and autonomous experimentation.

Title: Machines and Mathematical Mutations: Using GNNs to Characterize Quiver Mutation Classes

• Authors: Jesse He, Helen Jenne, Herman Chau, Davis Brown, Mark Raugas, Sara Billey, Henry Kvinge
• Subjects: cs.LG, hep-th, math.CO
• Abstract URL: https://arxiv.org/abs/2411.07467
• Pdf URL: https://arxiv.org/pdf/2411.07467
• Copy Paste: [[2411.07467]] Machines and Mathematical Mutations: Using GNNs to Characterize Quiver Mutation Classes(https://arxiv.org/abs/2411.07467)
• Keywords: explainability
• Abstract: Machine learning is becoming an increasingly valuable tool in mathematics, enabling one to identify subtle patterns across collections of examples so vast that they would be impossible for a single researcher to feasibly review and analyze. In this work, we use graph neural networks to investigate quiver mutation -- an operation that transforms one quiver (or directed multigraph) into another -- which is central to the theory of cluster algebras with deep connections to geometry, topology, and physics. In the study of cluster algebras, the question of mutation equivalence is of fundamental concern: given two quivers, can one efficiently determine if one quiver can be transformed into the other through a sequence of mutations? Currently, this question has only been resolved in specific cases. In this paper, we use graph neural networks and AI explainability techniques to discover mutation equivalence criteria for the previously unknown case of quivers of type $\tilde{D}_n$. Along the way, we also show that even without explicit training to do so, our model captures structure within its hidden representation that allows us to reconstruct known criteria from type $D_n$, adding to the growing evidence that modern machine learning models are capable of learning abstract and general rules from mathematical data.

Title: Privacy-Preserving Verifiable Neural Network Inference Service

• Authors: Arman Riasi, Jorge Guajardo, Thang Hoang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07468
• Pdf URL: https://arxiv.org/pdf/2411.07468
• Copy Paste: [[2411.07468]] Privacy-Preserving Verifiable Neural Network Inference Service(https://arxiv.org/abs/2411.07468)
• Keywords: secure, security, privacy, protect, fair
• Abstract: Machine learning has revolutionized data analysis and pattern recognition, but its resource-intensive training has limited accessibility. Machine Learning as a Service (MLaaS) simplifies this by enabling users to delegate their data samples to an MLaaS provider and obtain the inference result using a pre-trained model. Despite its convenience, leveraging MLaaS poses significant privacy and reliability concerns to the client. Specifically, sensitive information from the client inquiry data can be leaked to an adversarial MLaaS provider. Meanwhile, the lack of a verifiability guarantee can potentially result in biased inference results or even unfair payment issues. While existing trustworthy machine learning techniques, such as those relying on verifiable computation or secure computation, offer solutions to privacy and reliability concerns, they fall short of simultaneously protecting the privacy of client data and providing provable inference verifiability. In this paper, we propose vPIN, a privacy-preserving and verifiable CNN inference scheme that preserves privacy for client data samples while ensuring verifiability for the inference. vPIN makes use of partial homomorphic encryption and commit-and-prove succinct non-interactive argument of knowledge techniques to achieve desirable security properties. In vPIN, we develop various optimization techniques to minimize the proving circuit for homomorphic inference evaluation thereby, improving the efficiency and performance of our technique. We fully implemented and evaluated our vPIN scheme on standard datasets (e.g., MNIST, CIFAR-10). Our experimental results show that vPIN achieves high efficiency in terms of proving time, verification time, and proof size, while providing client data privacy guarantees and provable verifiability.

Title: Semi-Truths: A Large-Scale Dataset of AI-Augmented Images for Evaluating Robustness of AI-Generated Image detectors

• Authors: Anisha Pal, Julia Kruk, Mansi Phute, Manognya Bhattaram, Diyi Yang, Duen Horng Chau, Judy Hoffman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07472
• Pdf URL: https://arxiv.org/pdf/2411.07472
• Copy Paste: [[2411.07472]] Semi-Truths: A Large-Scale Dataset of AI-Augmented Images for Evaluating Robustness of AI-Generated Image detectors(https://arxiv.org/abs/2411.07472)
• Keywords: robust, diffusion
• Abstract: Text-to-image diffusion models have impactful applications in art, design, and entertainment, yet these technologies also pose significant risks by enabling the creation and dissemination of misinformation. Although recent advancements have produced AI-generated image detectors that claim robustness against various augmentations, their true effectiveness remains uncertain. Do these detectors reliably identify images with different levels of augmentation? Are they biased toward specific scenes or data distributions? To investigate, we introduce SEMI-TRUTHS, featuring 27,600 real images, 223,400 masks, and 1,472,700 AI-augmented images that feature targeted and localized perturbations produced using diverse augmentation techniques, diffusion models, and data distributions. Each augmented image is accompanied by metadata for standardized and targeted evaluation of detector robustness. Our findings suggest that state-of-the-art detectors exhibit varying sensitivities to the types and degrees of perturbations, data distributions, and augmentation methods used, offering new insights into their performance and limitations. The code for the augmentation and evaluation pipeline is available at this https URL.

Title: Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models

• Authors: Daria Kryvosheieva, Roger Levy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07474
• Pdf URL: https://arxiv.org/pdf/2411.07474
• Copy Paste: [[2411.07474]] Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models(https://arxiv.org/abs/2411.07474)
• Keywords: transformer
• Abstract: Language models (LMs) are capable of acquiring elements of human-like syntactic knowledge. Targeted syntactic evaluation tests have been employed to measure how well they form generalizations about syntactic phenomena in high-resource languages such as English. However, we still lack a thorough understanding of LMs' capacity for syntactic generalizations in low-resource languages, which are responsible for much of the diversity of syntactic patterns worldwide. In this study, we develop targeted syntactic evaluation tests for three low-resource languages (Basque, Hindi, and Swahili) and use them to evaluate five families of open-access multilingual Transformer LMs. We find that some syntactic tasks prove relatively easy for LMs while others (agreement in sentences containing indirect objects in Basque, agreement across a prepositional phrase in Swahili) are challenging. We additionally uncover issues with publicly available Transformers, including a bias toward the habitual aspect in Hindi in multilingual BERT and underperformance compared to similar-sized models in XGLM-4.5B.

Title: Developers Are Victims Too : A Comprehensive Analysis of The VS Code Extension Ecosystem

• Authors: Shehan Edirimannage, Charitha Elvitigala, Asitha Kottahachchi Kankanamge Don, Wathsara Daluwatta, Primal Wijesekara, Ibrahim Khalil
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.07479
• Pdf URL: https://arxiv.org/pdf/2411.07479
• Copy Paste: [[2411.07479]] Developers Are Victims Too : A Comprehensive Analysis of The VS Code Extension Ecosystem(https://arxiv.org/abs/2411.07479)
• Keywords: secure, security, attack
• Abstract: With the wave of high-profile supply chain attacks targeting development and client organizations, supply chain security has recently become a focal point. As a result, there is an elevated discussion on securing the development environment and increasing the transparency of the third-party code that runs in software products to minimize any negative impact from third-party code in a software product. However, the literature on secure software development lacks insight into how the third-party development tools used by every developer affect the security posture of the developer, the development organization, and, eventually, the end product. To that end, we have analyzed 52,880 third-party VS Code extensions to understand their threat to the developer, the code, and the development organizations. We found that ~5.6\% of the analyzed extensions have suspicious behavior, jeopardizing the integrity of the development environment and potentially leaking sensitive information on the developer's product. We also found that the VS Code hosting the third-party extensions lacks practical security controls and lets untrusted third-party code run unchecked and with questionable capabilities. We offer recommendations on possible avenues for fixing some of the issues uncovered during the analysis.

Title: Enhancing Link Prediction with Fuzzy Graph Attention Networks and Dynamic Negative Sampling

• Authors: Jinming Xing
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.07482
• Pdf URL: https://arxiv.org/pdf/2411.07482
• Copy Paste: [[2411.07482]] Enhancing Link Prediction with Fuzzy Graph Attention Networks and Dynamic Negative Sampling(https://arxiv.org/abs/2411.07482)
• Keywords: robust
• Abstract: Link prediction is crucial for understanding complex networks but traditional Graph Neural Networks (GNNs) often rely on random negative sampling, leading to suboptimal performance. This paper introduces Fuzzy Graph Attention Networks (FGAT), a novel approach integrating fuzzy rough sets for dynamic negative sampling and enhanced node feature aggregation. Fuzzy Negative Sampling (FNS) systematically selects high-quality negative edges based on fuzzy similarities, improving training efficiency. FGAT layer incorporates fuzzy rough set principles, enabling robust and discriminative node representations. Experiments on two research collaboration networks demonstrate FGAT's superior link prediction accuracy, outperforming state-of-the-art baselines by leveraging the power of fuzzy rough sets for effective negative sampling and node feature learning.

Title: Rapid Response: Mitigating LLM Jailbreaks with a Few Examples

• Authors: Alwin Peng, Julian Michael, Henry Sleight, Ethan Perez, Mrinank Sharma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07494
• Pdf URL: https://arxiv.org/pdf/2411.07494
• Copy Paste: [[2411.07494]] Rapid Response: Mitigating LLM Jailbreaks with a Few Examples(https://arxiv.org/abs/2411.07494)
• Keywords: defense, attack, robust, large language model
• Abstract: As large language models (LLMs) grow more powerful, ensuring their safety against misuse becomes crucial. While researchers have focused on developing robust defenses, no method has yet achieved complete invulnerability to attacks. We propose an alternative approach: instead of seeking perfect adversarial robustness, we develop rapid response techniques to look to block whole classes of jailbreaks after observing only a handful of attacks. To study this setting, we develop RapidResponseBench, a benchmark that measures a defense's robustness against various jailbreak strategies after adapting to a few observed examples. We evaluate five rapid response methods, all of which use jailbreak proliferation, where we automatically generate additional jailbreaks similar to the examples observed. Our strongest method, which fine-tunes an input classifier to block proliferated jailbreaks, reduces attack success rate by a factor greater than 240 on an in-distribution set of jailbreaks and a factor greater than 15 on an out-of-distribution set, having observed just one example of each jailbreaking strategy. Moreover, further studies suggest that the quality of proliferation model and number of proliferated examples play an key role in the effectiveness of this defense. Overall, our results highlight the potential of responding rapidly to novel jailbreaks to limit LLM misuse.

Title: Semantic Sleuth: Identifying Ponzi Contracts via Large Language Models

• Authors: Cong Wu, Jing Chen, Ziwei Wang, Ruichao Liang, Ruiying Du
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.07498
• Pdf URL: https://arxiv.org/pdf/2411.07498
• Copy Paste: [[2411.07498]] Semantic Sleuth: Identifying Ponzi Contracts via Large Language Models(https://arxiv.org/abs/2411.07498)
• Keywords: security, large language model
• Abstract: Smart contracts, self-executing agreements directly encoded in code, are fundamental to blockchain technology, especially in decentralized finance (DeFi) and Web3. However, the rise of Ponzi schemes in smart contracts poses significant risks, leading to substantial financial losses and eroding trust in blockchain systems. Existing detection methods, such as PonziGuard, depend on large amounts of labeled data and struggle to identify unseen Ponzi schemes, limiting their reliability and generalizability. In contrast, we introduce PonziSleuth, the first LLM-driven approach for detecting Ponzi smart contracts, which requires no labeled training data. PonziSleuth utilizes advanced language understanding capabilities of LLMs to analyze smart contract source code through a novel two-step zero-shot chain-of-thought prompting technique. Our extensive evaluation on benchmark datasets and real-world contracts demonstrates that PonziSleuth delivers comparable, and often superior, performance without the extensive data requirements, achieving a balanced detection accuracy of 96.06% with GPT-3.5-turbo, 93.91% with LLAMA3, and 94.27% with Mistral. In real-world detection, PonziSleuth successfully identified 15 new Ponzi schemes from 4,597 contracts verified by Etherscan in March 2024, with a false negative rate of 0% and a false positive rate of 0.29%. These results highlight PonziSleuth's capability to detect diverse and novel Ponzi schemes, marking a significant advancement in leveraging LLMs for enhancing blockchain security and mitigating financial scams.

Title: LAUREL: Learned Augmented Residual Layer

• Authors: Gaurav Menghani, Ravi Kumar, Sanjiv Kumar
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07501
• Pdf URL: https://arxiv.org/pdf/2411.07501
• Copy Paste: [[2411.07501]] LAUREL: Learned Augmented Residual Layer(https://arxiv.org/abs/2411.07501)
• Keywords: transformer
• Abstract: One of the core pillars of efficient deep learning methods is architectural improvements such as the residual/skip connection, which has led to significantly better model convergence and quality. Since then the residual connection has become ubiquitous in not just convolutional neural networks but also transformer-based architectures, the backbone of LLMs. In this paper we introduce \emph{Learned Augmented Residual Layer} (LAuReL) -- a novel generalization of the canonical residual connection -- with the goal to be an in-situ replacement of the latter while outperforming on both model quality and footprint metrics. Our experiments show that using \laurel can help boost performance for both vision and language models. For example, on the ResNet-50, ImageNet 1K task, it achieves $60\%$ of the gains from adding an extra layer, while only adding $0.003\%$ more parameters, and matches it while adding $2.6\times$ fewer parameters.

Title: FM-TS: Flow Matching for Time Series Generation

• Authors: Yang Hu, Xiao Wang, Lirong Wu, Huatian Zhang, Stan Z. Li, Sheng Wang, Tianlong Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07506
• Pdf URL: https://arxiv.org/pdf/2411.07506
• Copy Paste: [[2411.07506]] FM-TS: Flow Matching for Time Series Generation(https://arxiv.org/abs/2411.07506)
• Keywords: diffusion
• Abstract: Time series generation has emerged as an essential tool for analyzing temporal data across numerous fields. While diffusion models have recently gained significant attention in generating high-quality time series, they tend to be computationally demanding and reliant on complex stochastic processes. To address these limitations, we introduce FM-TS, a rectified Flow Matching-based framework for Time Series generation, which simplifies the time series generation process by directly optimizing continuous trajectories. This approach avoids the need for iterative sampling or complex noise schedules typically required in diffusion-based models. FM-TS is more efficient in terms of training and inference. Moreover, FM-TS is highly adaptive, supporting both conditional and unconditional time series generation. Notably, through our novel inference design, the model trained in an unconditional setting can seamlessly generalize to conditional tasks without the need for retraining. Extensive benchmarking across both settings demonstrates that FM-TS consistently delivers superior performance compared to existing approaches while being more efficient in terms of training and inference. For instance, in terms of discriminative score, FM-TS achieves 0.005, 0.019, 0.011, 0.005, 0.053, and 0.106 on the Sines, Stocks, ETTh, MuJoCo, Energy, and fMRI unconditional time series datasets, respectively, significantly outperforming the second-best method which achieves 0.006, 0.067, 0.061, 0.008, 0.122, and 0.167 on the same datasets. We have achieved superior performance in solar forecasting and MuJoCo imputation tasks, significantly enhanced by our innovative $t$ power sampling method. The code is available at this https URL.

Title: Robust Offline Reinforcement Learning for Non-Markovian Decision Processes

• Authors: Ruiquan Huang, Yingbin Liang, Jing Yang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.07514
• Pdf URL: https://arxiv.org/pdf/2411.07514
• Copy Paste: [[2411.07514]] Robust Offline Reinforcement Learning for Non-Markovian Decision Processes(https://arxiv.org/abs/2411.07514)
• Keywords: robust
• Abstract: Distributionally robust offline reinforcement learning (RL) aims to find a policy that performs the best under the worst environment within an uncertainty set using an offline dataset collected from a nominal model. While recent advances in robust RL focus on Markov decision processes (MDPs), robust non-Markovian RL is limited to planning problem where the transitions in the uncertainty set are known. In this paper, we study the learning problem of robust offline non-Markovian RL. Specifically, when the nominal model admits a low-rank structure, we propose a new algorithm, featuring a novel dataset distillation and a lower confidence bound (LCB) design for robust values under different types of the uncertainty set. We also derive new dual forms for these robust values in non-Markovian RL, making our algorithm more amenable to practical implementation. By further introducing a novel type-I concentrability coefficient tailored for offline low-rank non-Markovian decision processes, we prove that our algorithm can find an $\epsilon$-optimal robust policy using $O(1/\epsilon^2)$ offline samples. Moreover, we extend our algorithm to the case when the nominal model does not have specific structure. With a new type-II concentrability coefficient, the extended algorithm also enjoys polynomial sample efficiency under all different types of the uncertainty set.

Title: TIPS: Threat Actor Informed Prioritization of Applications using SecEncoder

• Authors: Muhammed Fatih Bulut, Acar Tamersoy, Naveed Ahmad, Yingqi Liu, Lloyd Greenwald
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07519
• Pdf URL: https://arxiv.org/pdf/2411.07519
• Copy Paste: [[2411.07519]] TIPS: Threat Actor Informed Prioritization of Applications using SecEncoder(https://arxiv.org/abs/2411.07519)
• Keywords: security
• Abstract: This paper introduces TIPS: Threat Actor Informed Prioritization using SecEncoder, a specialized language model for security. TIPS combines the strengths of both encoder and decoder language models to detect and prioritize compromised applications. By integrating threat actor intelligence, TIPS enhances the accuracy and relevance of its detections. Extensive experiments with a real-world benchmark dataset of applications demonstrate TIPS's high efficacy, achieving an F-1 score of 0.90 in identifying malicious applications. Additionally, in real-world scenarios, TIPS significantly reduces the backlog of investigations for security analysts by 87%, thereby streamlining the threat response process and improving overall security posture.

Title: Fair Summarization: Bridging Quality and Diversity in Extractive Summaries

• Authors: Sina Bagheri Nezhad, Sayan Bandyapadhyay, Ameeta Agrawal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07521
• Pdf URL: https://arxiv.org/pdf/2411.07521
• Copy Paste: [[2411.07521]] Fair Summarization: Bridging Quality and Diversity in Extractive Summaries(https://arxiv.org/abs/2411.07521)
• Keywords: fair
• Abstract: Fairness in multi-document summarization of user-generated content remains a critical challenge in natural language processing (NLP). Existing summarization methods often fail to ensure equitable representation across different social groups, leading to biased outputs. In this paper, we introduce two novel methods for fair extractive summarization: FairExtract, a clustering-based approach, and FairGPT, which leverages GPT-3.5-turbo with fairness constraints. We evaluate these methods using Divsumm summarization dataset of White-aligned, Hispanic, and African-American dialect tweets and compare them against relevant baselines. The results obtained using a comprehensive set of summarization quality metrics such as SUPERT, BLANC, SummaQA, BARTScore, and UniEval, as well as a fairness metric F, demonstrate that FairExtract and FairGPT achieve superior fairness while maintaining competitive summarization quality. Additionally, we introduce composite metrics (e.g., SUPERT+F, BLANC+F) that integrate quality and fairness into a single evaluation framework, offering a more nuanced understanding of the trade-offs between these objectives. This work highlights the importance of fairness in summarization and sets a benchmark for future research in fairness-aware NLP models.

Title: Collaborative and Federated Black-box Optimization: A Bayesian Optimization Perspective

• Authors: Raed Al Kontar
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.07523
• Pdf URL: https://arxiv.org/pdf/2411.07523
• Copy Paste: [[2411.07523]] Collaborative and Federated Black-box Optimization: A Bayesian Optimization Perspective(https://arxiv.org/abs/2411.07523)
• Keywords: privacy, federate
• Abstract: We focus on collaborative and federated black-box optimization (BBOpt), where agents optimize their heterogeneous black-box functions through collaborative sequential experimentation. From a Bayesian optimization perspective, we address the fundamental challenges of distributed experimentation, heterogeneity, and privacy within BBOpt, and propose three unifying frameworks to tackle these issues: (i) a global framework where experiments are centrally coordinated, (ii) a local framework that allows agents to make decisions based on minimal shared information, and (iii) a predictive framework that enhances local surrogates through collaboration to improve decision-making. We categorize existing methods within these frameworks and highlight key open questions to unlock the full potential of federated BBOpt. Our overarching goal is to shift federated learning from its predominantly descriptive/predictive paradigm to a prescriptive one, particularly in the context of BBOpt - an inherently sequential decision-making problem.

Title: Prompt-enhanced Network for Hateful Meme Classification

• Authors: Junxi Liu, Yanyan Feng, Jiehai Chen, Yun Xue, Fenghuan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07527
• Pdf URL: https://arxiv.org/pdf/2411.07527
• Copy Paste: [[2411.07527]] Prompt-enhanced Network for Hateful Meme Classification(https://arxiv.org/abs/2411.07527)
• Keywords: extraction
• Abstract: The dynamic expansion of social media has led to an inundation of hateful memes on media platforms, accentuating the growing need for efficient identification and removal. Acknowledging the constraints of conventional multimodal hateful meme classification, which heavily depends on external knowledge and poses the risk of including irrelevant or redundant content, we developed Pen -- a prompt-enhanced network framework based on the prompt learning approach. Specifically, after constructing the sequence through the prompt method and encoding it with a language model, we performed region information global extraction on the encoded sequence for multi-view perception. By capturing global information about inference instances and demonstrations, Pen facilitates category selection by fully leveraging sequence information. This approach significantly improves model classification accuracy. Additionally, to bolster the model's reasoning capabilities in the feature space, we introduced prompt-aware contrastive learning into the framework to improve the quality of sample feature distributions. Through extensive ablation experiments on two public datasets, we evaluate the effectiveness of the Pen framework, concurrently comparing it with state-of-the-art model baselines. Our research findings highlight that Pen surpasses manual prompt methods, showcasing superior generalization and classification accuracy in hateful meme classification tasks. Our code is available at this https URL.

Title: SecEncoder: Logs are All You Need in Security

• Authors: Muhammed Fatih Bulut, Yingqi Liu, Naveed Ahmad, Maximilian Turner, Sami Ait Ouahmane, Cameron Andrews, Lloyd Greenwald
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07528
• Pdf URL: https://arxiv.org/pdf/2411.07528
• Copy Paste: [[2411.07528]] SecEncoder: Logs are All You Need in Security(https://arxiv.org/abs/2411.07528)
• Keywords: security
• Abstract: Large and Small Language Models (LMs) are typically pretrained using extensive volumes of text, which are sourced from publicly accessible platforms such as Wikipedia, Book Corpus, or through web scraping. These models, due to their exposure to a wide range of language data, exhibit impressive generalization capabilities and can perform a multitude of tasks simultaneously. However, they often fall short when it comes to domain-specific tasks due to their broad training data. This paper introduces SecEncoder, a specialized small language model that is pretrained using security logs. SecEncoder is designed to address the domain-specific limitations of general LMs by focusing on the unique language and patterns found in security logs. Experimental results indicate that SecEncoder outperforms other LMs, such as BERTlarge, DeBERTa-v3-large and OpenAI's Embedding (textembedding-ada-002) models, which are pretrained mainly on natural language, across various tasks. Furthermore, although SecEncoder is primarily pretrained on log data, it outperforms models pretrained on natural language for a range of tasks beyond log analysis, such as incident prioritization and threat intelligence document retrieval. This suggests that domain specific pretraining with logs can significantly enhance the performance of LMs in security. These findings pave the way for future research into security-specific LMs and their potential applications.

Title: Large Language Models as Neurolinguistic Subjects: Identifying Internal Representations for Form and Meaning

• Authors: Linyang He, Ercong Nie, Helmut Schmid, Hinrich Schütze, Nima Mesgarani, Jonathan Brennan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07533
• Pdf URL: https://arxiv.org/pdf/2411.07533
• Copy Paste: [[2411.07533]] Large Language Models as Neurolinguistic Subjects: Identifying Internal Representations for Form and Meaning(https://arxiv.org/abs/2411.07533)
• Keywords: large language model
• Abstract: This study investigates the linguistic understanding of Large Language Models (LLMs) regarding signifier (form) and signified (meaning) by distinguishing two LLM evaluation paradigms: psycholinguistic and neurolinguistic. Traditional psycholinguistic evaluations often reflect statistical biases that may misrepresent LLMs' true linguistic capabilities. We introduce a neurolinguistic approach, utilizing a novel method that combines minimal pair and diagnostic probing to analyze activation patterns across model layers. This method allows for a detailed examination of how LLMs represent form and meaning, and whether these representations are consistent across languages. Our contributions are three-fold: (1) We compare neurolinguistic and psycholinguistic methods, revealing distinct patterns in LLM assessment; (2) We demonstrate that LLMs exhibit higher competence in form compared to meaning, with the latter largely correlated to the former; (3) We present new conceptual minimal pair datasets for Chinese (COMPS-ZH) and German (COMPS-DE), complementing existing English datasets.

Title: Double-Signed Fragmented DNSSEC for Countering Quantum Threat

• Authors: Syed W. Shah. Lei Pan, Din Duc Nha Nguyen, Robin Doss, Warren Armstrong, Praveen Gauravaram
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.07535
• Pdf URL: https://arxiv.org/pdf/2411.07535
• Copy Paste: [[2411.07535]] Double-Signed Fragmented DNSSEC for Countering Quantum Threat(https://arxiv.org/abs/2411.07535)
• Keywords: security, protect, attack
• Abstract: DNSSEC, a DNS security extension, is essential to accurately translating domain names to IP addresses. Digital signatures provide the foundation for this reliable translation, however, the evolution of 'Quantum Computers' has made traditional digital signatures vulnerable. In light of this, NIST has recently selected potential post-quantum digital signatures that can operate on conventional computers and resist attacks made with Quantum Computers. Since these post-quantum digital signatures are still in their early stages of development, replacing pre-quantum digital signature schemes in DNSSEC with post-quantum candidates is risky until the post-quantum candidates have undergone a thorough security analysis. Given this, herein, we investigate the viability of employing 'Double-Signatures' in DNSSEC, combining a post-quantum digital signature and a classic one. The rationale is that double-signatures will offer protection against quantum threats on conventional signature schemes as well as unknown non-quantum attacks on post-quantum signature schemes, hence even if one fails the other provides security guarantees. However, the inclusion of two signatures in the DNSSEC response message doesn't bode well with the maximum allowed size of DNSSEC responses (i.e., 1232B, a limitation enforced by MTU of physical links). To counter this issue, we leverage a way to do application-layer fragmentation of DNSSEC responses with two signatures. We implement our solution on top of OQS-BIND and through experiments show that the addition of two signatures in DNSSEC and application-layer fragmentation of all relevant resource records and their reassembly does not have any substantial impact on the efficiency of the resolution process and thus is suitable for the interim period at least until the quantum computers are fully realized.

Title: Model Stealing for Any Low-Rank Language Model

• Authors: Allen Liu, Ankur Moitra
• Subjects: cs.LG, cs.AI, cs.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.07536
• Pdf URL: https://arxiv.org/pdf/2411.07536
• Copy Paste: [[2411.07536]] Model Stealing for Any Low-Rank Language Model(https://arxiv.org/abs/2411.07536)
• Keywords: security, privacy, steal, large language model
• Abstract: Model stealing, where a learner tries to recover an unknown model via carefully chosen queries, is a critical problem in machine learning, as it threatens the security of proprietary models and the privacy of data they are trained on. In recent years, there has been particular interest in stealing large language models (LLMs). In this paper, we aim to build a theoretical understanding of stealing language models by studying a simple and mathematically tractable setting. We study model stealing for Hidden Markov Models (HMMs), and more generally low-rank language models. We assume that the learner works in the conditional query model, introduced by Kakade, Krishnamurthy, Mahajan and Zhang. Our main result is an efficient algorithm in the conditional query model, for learning any low-rank distribution. In other words, our algorithm succeeds at stealing any language model whose output distribution is low-rank. This improves upon the previous result by Kakade, Krishnamurthy, Mahajan and Zhang, which also requires the unknown distribution to have high "fidelity", a property that holds only in restricted cases. There are two key insights behind our algorithm: First, we represent the conditional distributions at each timestep by constructing barycentric spanners among a collection of vectors of exponentially large dimension. Second, for sampling from our representation, we iteratively solve a sequence of convex optimization problems that involve projection in relative entropy to prevent compounding of errors over the length of the sequence. This is an interesting example where, at least theoretically, allowing a machine learning model to solve more complex problems at inference time can lead to drastic improvements in its performance.

Title: Unraveling the Gradient Descent Dynamics of Transformers

• Authors: Bingqing Song, Boran Han, Shuai Zhang, Jie Ding, Mingyi Hong
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2411.07538
• Pdf URL: https://arxiv.org/pdf/2411.07538
• Copy Paste: [[2411.07538]] Unraveling the Gradient Descent Dynamics of Transformers(https://arxiv.org/abs/2411.07538)
• Keywords: transformer
• Abstract: While the Transformer architecture has achieved remarkable success across various domains, a thorough theoretical foundation explaining its optimization dynamics is yet to be fully developed. In this study, we aim to bridge this understanding gap by answering the following two core questions: (1) Which types of Transformer architectures allow Gradient Descent (GD) to achieve guaranteed convergence? and (2) Under what initial conditions and architectural specifics does the Transformer achieve rapid convergence during training? By analyzing the loss landscape of a single Transformer layer using Softmax and Gaussian attention kernels, our work provides concrete answers to these questions. Our findings demonstrate that, with appropriate weight initialization, GD can train a Transformer model (with either kernel type) to achieve a global optimal solution, especially when the input embedding dimension is large. Nonetheless, certain scenarios highlight potential pitfalls: training a Transformer using the Softmax attention kernel may sometimes lead to suboptimal local solutions. In contrast, the Gaussian attention kernel exhibits a much favorable behavior. Our empirical study further validate the theoretical findings.

Title: HiCoM: Hierarchical Coherent Motion for Streamable Dynamic Scene with 3D Gaussian Splatting

• Authors: Qiankun Gao, Jiarui Meng, Chengxiang Wen, Jie Chen, Jian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07541
• Pdf URL: https://arxiv.org/pdf/2411.07541
• Copy Paste: [[2411.07541]] HiCoM: Hierarchical Coherent Motion for Streamable Dynamic Scene with 3D Gaussian Splatting(https://arxiv.org/abs/2411.07541)
• Keywords: robust
• Abstract: The online reconstruction of dynamic scenes from multi-view streaming videos faces significant challenges in training, rendering and storage efficiency. Harnessing superior learning speed and real-time rendering capabilities, 3D Gaussian Splatting (3DGS) has recently demonstrated considerable potential in this field. However, 3DGS can be inefficient in terms of storage and prone to overfitting by excessively growing Gaussians, particularly with limited views. This paper proposes an efficient framework, dubbed HiCoM, with three key components. First, we construct a compact and robust initial 3DGS representation using a perturbation smoothing strategy. Next, we introduce a Hierarchical Coherent Motion mechanism that leverages the inherent non-uniform distribution and local consistency of 3D Gaussians to swiftly and accurately learn motions across frames. Finally, we continually refine the 3DGS with additional Gaussians, which are later merged into the initial 3DGS to maintain consistency with the evolving scene. To preserve a compact representation, an equivalent number of low-opacity Gaussians that minimally impact the representation are removed before processing subsequent frames. Extensive experiments conducted on two widely used datasets show that our framework improves learning efficiency of the state-of-the-art methods by about $20\%$ and reduces the data storage by $85\%$, achieving competitive free-viewpoint video synthesis quality but with higher robustness and stability. Moreover, by parallel learning multiple frames simultaneously, our HiCoM decreases the average training wall time to $<2$ seconds per frame with negligible performance degradation, substantially boosting real-world applicability and responsiveness.

Title: GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting

• Authors: Umangi Jain, Ashkan Mirzaei, Igor Gilitschenski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07555
• Pdf URL: https://arxiv.org/pdf/2411.07555
• Copy Paste: [[2411.07555]] GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting(https://arxiv.org/abs/2411.07555)
• Keywords: extraction, segmentation
• Abstract: We introduce GaussianCut, a new method for interactive multiview segmentation of scenes represented as 3D Gaussians. Our approach allows for selecting the objects to be segmented by interacting with a single view. It accepts intuitive user input, such as point clicks, coarse scribbles, or text. Using 3D Gaussian Splatting (3DGS) as the underlying scene representation simplifies the extraction of objects of interest which are considered to be a subset of the scene's Gaussians. Our key idea is to represent the scene as a graph and use the graph-cut algorithm to minimize an energy function to effectively partition the Gaussians into foreground and background. To achieve this, we construct a graph based on scene Gaussians and devise a segmentation-aligned energy function on the graph to combine user inputs with scene properties. To obtain an initial coarse segmentation, we leverage 2D image/video segmentation models and further refine these coarse estimates using our graph construction. Our empirical evaluations show the adaptability of GaussianCut across a diverse set of scenes. GaussianCut achieves competitive performance with state-of-the-art approaches for 3D segmentation without requiring any additional segmentation-aware training.

Title: Multi-task Feature Enhancement Network for No-Reference Image Quality Assessment

• Authors: Li Yu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.07556
• Pdf URL: https://arxiv.org/pdf/2411.07556
• Copy Paste: [[2411.07556]] Multi-task Feature Enhancement Network for No-Reference Image Quality Assessment(https://arxiv.org/abs/2411.07556)
• Keywords: robust, extraction
• Abstract: Due to the scarcity of labeled samples in Image Quality Assessment (IQA) datasets, numerous recent studies have proposed multi-task based strategies, which explore feature information from other tasks or domains to boost the IQA task. Nevertheless, multi-task strategies based No-Reference Image Quality Assessment (NR-IQA) methods encounter several challenges. First, existing methods have not explicitly exploited texture details, which significantly influence the image quality. Second, multi-task methods conventionally integrate features through simple operations such as addition or concatenation, thereby diminishing the network's capacity to accurately represent distorted features. To tackle these challenges, we introduce a novel multi-task NR-IQA framework. Our framework consists of three key components: a high-frequency extraction network, a quality estimation network, and a distortion-aware network. The high-frequency extraction network is designed to guide the model's focus towards high-frequency information, which is highly related to the texture details. Meanwhile, the distortion-aware network extracts distortion-related features to distinguish different distortion types. To effectively integrate features from different tasks, a feature fusion module is developed based on an attention mechanism. Empirical results from five standard IQA databases confirm that our method not only achieves high performance but also exhibits robust generalization ability.

Title: Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Models

• Authors: Tiejin Chen, Kaishen Wang, Hua Wei
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07559
• Pdf URL: https://arxiv.org/pdf/2411.07559
• Copy Paste: [[2411.07559]] Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Models(https://arxiv.org/abs/2411.07559)
• Keywords: attack, large language model
• Abstract: Jailbreaking methods, which induce Multi-modal Large Language Models (MLLMs) to output harmful responses, raise significant safety concerns. Among these methods, gradient-based approaches, which use gradients to generate malicious prompts, have been widely studied due to their high success rates in white-box settings, where full access to the model is available. However, these methods have notable limitations: they require white-box access, which is not always feasible, and involve high memory usage. To address scenarios where white-box access is unavailable, attackers often resort to transfer attacks. In transfer attacks, malicious inputs generated using white-box models are applied to black-box models, but this typically results in reduced attack performance. To overcome these challenges, we propose Zer0-Jack, a method that bypasses the need for white-box access by leveraging zeroth-order optimization. We propose patch coordinate descent to efficiently generate malicious image inputs to directly attack black-box MLLMs, which significantly reduces memory usage further. Through extensive experiments, Zer0-Jack achieves a high attack success rate across various models, surpassing previous transfer-based methods and performing comparably with existing white-box jailbreak techniques. Notably, Zer0-Jack achieves a 95\% attack success rate on MiniGPT-4 with the Harmful Behaviors Multi-modal Dataset on a black-box setting, demonstrating its effectiveness. Additionally, we show that Zer0-Jack can directly attack commercial MLLMs such as GPT-4o. Codes are provided in the supplement.

Title: Semantic segmentation on multi-resolution optical and microwave data using deep learning

• Authors: Jai G Singla, Bakul Vaghela
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.07581
• Pdf URL: https://arxiv.org/pdf/2411.07581
• Copy Paste: [[2411.07581]] Semantic segmentation on multi-resolution optical and microwave data using deep learning(https://arxiv.org/abs/2411.07581)
• Keywords: segmentation
• Abstract: Presently, deep learning and convolutional neural networks (CNNs) are widely used in the fields of image processing, image classification, object identification and many more. In this work, we implemented convolutional neural network based modified U-Net model and VGG-UNet model to automatically identify objects from satellite imagery captured using high resolution Indian remote sensing satellites and then to pixel wise classify satellite data into various classes. In this paper, Cartosat 2S (~1m spatial resolution) datasets were used and deep learning models were implemented to detect building shapes and ships from the test datasets with an accuracy of more than 95%. In another experiment, microwave data (varied resolution) from RISAT-1 was taken as an input and ships and trees were detected with an accuracy of >96% from these datasets. For the classification of images into multiple-classes, deep learning model was trained on multispectral Cartosat images. Model generated results were then tested using ground truth. Multi-label classification results were obtained with an accuracy (IoU) of better than 95%. Total six different problems were attempted using deep learning models and IoU accuracies in the range of 85% to 98% were achieved depending on the degree of complexity.

Title: Entropy Controllable Direct Preference Optimization

• Authors: Motoki Omura, Yasuhiro Fujita, Toshiki Kataoka
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07595
• Pdf URL: https://arxiv.org/pdf/2411.07595
• Copy Paste: [[2411.07595]] Entropy Controllable Direct Preference Optimization(https://arxiv.org/abs/2411.07595)
• Keywords: large language model
• Abstract: In the post-training of large language models (LLMs), Reinforcement Learning from Human Feedback (RLHF) is an effective approach to achieve generation aligned with human preferences. Direct Preference Optimization (DPO) allows for policy training with a simple binary cross-entropy loss without a reward model. The objective of DPO is regularized by reverse KL divergence that encourages mode-seeking fitting to the reference policy. Nonetheless, we indicate that minimizing reverse KL divergence could fail to capture a mode of the reference distribution, which may hurt the policy's performance. Based on this observation, we propose a simple modification to DPO, H-DPO, which allows for control over the entropy of the resulting policy, enhancing the distribution's sharpness and thereby enabling mode-seeking fitting more effectively. In our experiments, we show that H-DPO outperformed DPO across various tasks, demonstrating superior results in pass@$k$ evaluations for mathematical tasks. Moreover, H-DPO is simple to implement, requiring only minor modifications to the loss calculation of DPO, which makes it highly practical and promising for wide-ranging applications in the training of LLMs.

Title: A Survey on Adversarial Machine Learning for Code Data: Realistic Threats, Countermeasures, and Interpretations

• Authors: Yulong Yang, Haoran Fan, Chenhao Lin, Qian Li, Zhengyu Zhao, Chao Shen, Xiaohong Guan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.07597
• Pdf URL: https://arxiv.org/pdf/2411.07597
• Copy Paste: [[2411.07597]] A Survey on Adversarial Machine Learning for Code Data: Realistic Threats, Countermeasures, and Interpretations(https://arxiv.org/abs/2411.07597)
• Keywords: security, privacy, attack
• Abstract: Code Language Models (CLMs) have achieved tremendous progress in source code understanding and generation, leading to a significant increase in research interests focused on applying CLMs to real-world software engineering tasks in recent years. However, in realistic scenarios, CLMs are exposed to potential malicious adversaries, bringing risks to the confidentiality, integrity, and availability of CLM systems. Despite these risks, a comprehensive analysis of the security vulnerabilities of CLMs in the extremely adversarial environment has been lacking. To close this research gap, we categorize existing attack techniques into three types based on the CIA triad: poisoning attacks (integrity \& availability infringement), evasion attacks (integrity infringement), and privacy attacks (confidentiality infringement). We have collected so far the most comprehensive (79) papers related to adversarial machine learning for CLM from the research fields of artificial intelligence, computer security, and software engineering. Our analysis covers each type of risk, examining threat model categorization, attack techniques, and countermeasures, while also introducing novel perspectives on eXplainable AI (XAI) and exploring the interconnections between different risks. Finally, we identify current challenges and future research opportunities. This study aims to provide a comprehensive roadmap for both researchers and practitioners and pave the way towards more reliable CLMs for practical applications.

Title: Problem-Oriented Segmentation and Retrieval: Case Study on Tutoring Conversations

• Authors: Rose E. Wang, Pawan Wirawarn, Kenny Lam, Omar Khattab, Dorottya Demszky
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07598
• Pdf URL: https://arxiv.org/pdf/2411.07598
• Copy Paste: [[2411.07598]] Problem-Oriented Segmentation and Retrieval: Case Study on Tutoring Conversations(https://arxiv.org/abs/2411.07598)
• Keywords: large language model, segmentation
• Abstract: Many open-ended conversations (e.g., tutoring lessons or business meetings) revolve around pre-defined reference materials, like worksheets or meeting bullets. To provide a framework for studying such conversation structure, we introduce Problem-Oriented Segmentation & Retrieval (POSR), the task of jointly breaking down conversations into segments and linking each segment to the relevant reference item. As a case study, we apply POSR to education where effectively structuring lessons around problems is critical yet difficult. We present LessonLink, the first dataset of real-world tutoring lessons, featuring 3,500 segments, spanning 24,300 minutes of instruction and linked to 116 SAT math problems. We define and evaluate several joint and independent approaches for POSR, including segmentation (e.g., TextTiling), retrieval (e.g., ColBERT), and large language models (LLMs) methods. Our results highlight that modeling POSR as one joint task is essential: POSR methods outperform independent segmentation and retrieval pipelines by up to +76% on joint metrics and surpass traditional segmentation methods by up to +78% on segmentation metrics. We demonstrate POSR's practical impact on downstream education applications, deriving new insights on the language and time use in real-world lesson structures.

Title: Circuit Complexity Bounds for RoPE-based Transformer Architecture

• Authors: Bo Chen, Xiaoyu Li, Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song
• Subjects: cs.LG, cs.AI, cs.CC, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07602
• Pdf URL: https://arxiv.org/pdf/2411.07602
• Copy Paste: [[2411.07602]] Circuit Complexity Bounds for RoPE-based Transformer Architecture(https://arxiv.org/abs/2411.07602)
• Keywords: transformer, large language model
• Abstract: Characterizing the express power of the Transformer architecture is critical to understanding its capacity limits and scaling law. Recent works provide the circuit complexity bounds to Transformer-like architecture. On the other hand, Rotary Position Embedding ($\mathsf{RoPE}$) has emerged as a crucial technique in modern large language models, offering superior performance in capturing positional information compared to traditional position embeddings, which shows great potential in application prospects, particularly for the long context scenario. Empirical evidence also suggests that $\mathsf{RoPE}$-based Transformer architectures demonstrate greater generalization capabilities compared to conventional Transformer models. In this work, we establish a tighter circuit complexity bound for Transformers with $\mathsf{RoPE}$ attention. Our key contribution is that we show that unless $\mathsf{TC}^0 = \mathsf{NC}^1$, a $\mathsf{RoPE}$-based Transformer with $\mathrm{poly}(n)$-precision, $O(1)$ layers, hidden dimension $d \leq O(n)$ cannot solve the arithmetic problem or the Boolean formula value problem. This result significantly demonstrates the fundamental limitation of the expressivity of the $\mathsf{RoPE}$-based Transformer architecture, although it achieves giant empirical success. Our theoretical framework not only establishes tighter complexity bounds but also may instruct further work on the $\mathsf{RoPE}$-based Transformer.

Title: Quantum Information-Empowered Graph Neural Network for Hyperspectral Change Detection

• Authors: Chia-Hsiang Lin, Tzu-Hsuan Lin, Jocelyn Chanussot
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.07608
• Pdf URL: https://arxiv.org/pdf/2411.07608
• Copy Paste: [[2411.07608]] Quantum Information-Empowered Graph Neural Network for Hyperspectral Change Detection(https://arxiv.org/abs/2411.07608)
• Keywords: extraction
• Abstract: Change detection (CD) is a critical remote sensing technique for identifying changes in the Earth's surface over time. The outstanding substance identifiability of hyperspectral images (HSIs) has significantly enhanced the detection accuracy, making hyperspectral change detection (HCD) an essential technology. The detection accuracy can be further upgraded by leveraging the graph structure of HSIs, motivating us to adopt the graph neural networks (GNNs) in solving HCD. For the first time, this work introduces quantum deep network (QUEEN) into HCD. Unlike GNN and CNN, both extracting the affine-computing features, QUEEN provides fundamentally different unitary-computing features. We demonstrate that through the unitary feature extraction procedure, QUEEN provides radically new information for deciding whether there is a change or not. Hierarchically, a graph feature learning (GFL) module exploits the graph structure of the bitemporal HSIs at the superpixel level, while a quantum feature learning (QFL) module learns the quantum features at the pixel level, as a complementary to GFL by preserving pixel-level detailed spatial information not retained in the superpixels. In the final classification stage, a quantum classifier is designed to cooperate with a traditional fully connected classifier. The superior HCD performance of the proposed QUEEN-empowered GNN (i.e., QUEEN-G) will be experimentally demonstrated on real hyperspectral datasets.

Title: Multimodal Clinical Reasoning through Knowledge-augmented Rationale Generation

• Authors: Shuai Niu, Jing Ma, Liang Bai, Zhihua Wang, Yida Xu, Yunya Song, Xian Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07611
• Pdf URL: https://arxiv.org/pdf/2411.07611
• Copy Paste: [[2411.07611]] Multimodal Clinical Reasoning through Knowledge-augmented Rationale Generation(https://arxiv.org/abs/2411.07611)
• Keywords: large language model
• Abstract: Clinical rationales play a pivotal role in accurate disease diagnosis; however, many models predominantly use discriminative methods and overlook the importance of generating supportive rationales. Rationale distillation is a process that transfers knowledge from large language models (LLMs) to smaller language models (SLMs), thereby enhancing the latter's ability to break down complex tasks. Despite its benefits, rationale distillation alone is inadequate for addressing domain knowledge limitations in tasks requiring specialized expertise, such as disease diagnosis. Effectively embedding domain knowledge in SLMs poses a significant challenge. While current LLMs are primarily geared toward processing textual data, multimodal LLMs that incorporate time series data, especially electronic health records (EHRs), are still evolving. To tackle these limitations, we introduce ClinRaGen, an SLM optimized for multimodal rationale generation in disease diagnosis. ClinRaGen incorporates a unique knowledge-augmented attention mechanism to merge domain knowledge with time series EHR data, utilizing a stepwise rationale distillation strategy to produce both textual and time series-based clinical rationales. Our evaluations show that ClinRaGen markedly improves the SLM's capability to interpret multimodal EHR data and generate accurate clinical rationales, supporting more reliable disease diagnosis, advancing LLM applications in healthcare, and narrowing the performance divide between LLMs and SLMs.

Title: Artificial Intelligence for Biomedical Video Generation

• Authors: Linyuan Li, Jianing Qiu, Anujit Saha, Lin Li, Poyuan Li, Mengxian He, Ziyu Guo, Wu Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07619
• Pdf URL: https://arxiv.org/pdf/2411.07619
• Copy Paste: [[2411.07619]] Artificial Intelligence for Biomedical Video Generation(https://arxiv.org/abs/2411.07619)
• Keywords: generative
• Abstract: As a prominent subfield of Artificial Intelligence Generated Content (AIGC), video generation has achieved notable advancements in recent years. The introduction of Sora-alike models represents a pivotal breakthrough in video generation technologies, significantly enhancing the quality of synthesized videos. Particularly in the realm of biomedicine, video generation technology has shown immense potential such as medical concept explanation, disease simulation, and biomedical data augmentation. In this article, we thoroughly examine the latest developments in video generation models and explore their applications, challenges, and future opportunities in the biomedical sector. We have conducted an extensive review and compiled a comprehensive list of datasets from various sources to facilitate the development and evaluation of video generative models in biomedicine. Given the rapid progress in this field, we have also created a github repository to regularly update the advances of biomedical video generation at: this https URL

Title: Unraveling the Connections between Flow Matching and Diffusion Probabilistic Models in Training-free Conditional Generation

• Authors: Kaiyu Song, Hanjiang Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07625
• Pdf URL: https://arxiv.org/pdf/2411.07625
• Copy Paste: [[2411.07625]] Unraveling the Connections between Flow Matching and Diffusion Probabilistic Models in Training-free Conditional Generation(https://arxiv.org/abs/2411.07625)
• Keywords: diffusion
• Abstract: Training-free conditional generation aims to leverage the unconditional diffusion models to implement the conditional generation, where flow-matching (FM) and diffusion probabilistic models (DPMs) are two mature unconditional diffusion models that achieve high-quality generation. Two questions were asked in this paper: What are the underlying connections between FM and DPMs in training-free conditional generation? Can we leverage DPMs to improve the training-free conditional generation for FM? We first show that a probabilistic diffusion path can be associated with the FM and DPMs. Then, we reformulate the ordinary differential equation (ODE) of FM based on the score function of DPMs, and thus, the conditions in FM can be incorporated as those in DPMs. Finally, we propose two posterior sampling methods to estimate the conditional term and achieve a training-free conditional generation of FM. Experimental results show that our proposed method could be implemented for various conditional generation tasks. Our method can generate higher-quality results than the state-of-the-art methods.

Title: Leveraging Previous Steps: A Training-free Fast Solver for Flow Diffusion

• Authors: Kaiyu Song, Hanjiang Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07627
• Pdf URL: https://arxiv.org/pdf/2411.07627
• Copy Paste: [[2411.07627]] Leveraging Previous Steps: A Training-free Fast Solver for Flow Diffusion(https://arxiv.org/abs/2411.07627)
• Keywords: diffusion
• Abstract: Flow diffusion models (FDMs) have recently shown potential in generation tasks due to the high generation quality. However, the current ordinary differential equation (ODE) solver for FDMs, e.g., the Euler solver, still suffers from slow generation since ODE solvers need many number function evaluations (NFE) to keep high-quality generation. In this paper, we propose a novel training-free flow-solver to reduce NFE while maintaining high-quality generation. The key insight for the flow-solver is to leverage the previous steps to reduce the NFE, where a cache is created to reuse these results from the previous steps. Specifically, the Taylor expansion is first used to approximate the ODE. To calculate the high-order derivatives of Taylor expansion, the flow-solver proposes to use the previous steps and a polynomial interpolation to approximate it, where the number of orders we could approximate equals the number of previous steps we cached. We also prove that the flow-solver has a more minor approximation error and faster generation speed. Experimental results on the CIFAR-10, CelebA-HQ, LSUN-Bedroom, LSUN-Church, ImageNet, and real text-to-image generation prove the efficiency of the flow-solver. Specifically, the flow-solver improves the FID-30K from 13.79 to 6.75, from 46.64 to 19.49 with $\text{NFE}=10$ on CIFAR-10 and LSUN-Church, respectively.

Title: Breaking the Low-Rank Dilemma of Linear Attention

• Authors: Qihang Fan, Huaibo Huang, Ran He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07635
• Pdf URL: https://arxiv.org/pdf/2411.07635
• Copy Paste: [[2411.07635]] Breaking the Low-Rank Dilemma of Linear Attention(https://arxiv.org/abs/2411.07635)
• Keywords: transformer
• Abstract: The Softmax attention mechanism in Transformer models is notoriously computationally expensive, particularly due to its quadratic complexity, posing significant challenges in vision applications. In contrast, linear attention provides a far more efficient solution by reducing the complexity to linear levels. However, compared to Softmax attention, linear attention often experiences significant performance degradation. Our experiments indicate that this performance drop is due to the low-rank nature of linear attention's feature map, which hinders its ability to adequately model complex spatial information. In this paper, to break the low-rank dilemma of linear attention, we conduct rank analysis from two perspectives: the KV buffer and the output features. Consequently, we introduce Rank-Augmented Linear Attention (RALA), which rivals the performance of Softmax attention while maintaining linear complexity and high efficiency. Based on RALA, we construct the Rank-Augmented Vision Linear Transformer (RAVLT). Extensive experiments demonstrate that RAVLT achieves excellent performance across various vision tasks. Specifically, without using any additional labels, data, or supervision during training, RAVLT achieves an 84.4% Top-1 accuracy on ImageNet-1k with only 26M parameters and 4.6G FLOPs. This result significantly surpasses previous linear attention mechanisms, fully illustrating the potential of RALA. Code will be available at this https URL.

Title: Top-$n\sigma$: Not All Logits Are You Need

• Authors: Chenxia Tang, Jianchun Liu, Hongli Xu, Liusheng Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07641
• Pdf URL: https://arxiv.org/pdf/2411.07641
• Copy Paste: [[2411.07641]] Top-$n\sigma$: Not All Logits Are You Need(https://arxiv.org/abs/2411.07641)
• Keywords: large language model
• Abstract: Large language models (LLMs) typically employ greedy decoding or low-temperature sampling for reasoning tasks, reflecting a perceived trade-off between diversity and accuracy. We challenge this convention by introducing top-$n\sigma$, a novel sampling method that operates directly on pre-softmax logits by leveraging a statistical threshold. Our key insight is that logits naturally separate into a Gaussian-distributed noisy region and a distinct informative region, enabling efficient token filtering without complex probability manipulations. Unlike existing methods (e.g., top-$p$, min-$p$) that inadvertently include more noise tokens at higher temperatures, top-$n\sigma$ maintains a stable sampling space regardless of temperature scaling. We also provide a theoretical analysis of top-$n\sigma$ to better understand its behavior. The extensive experimental results across four reasoning-focused datasets demonstrate that our method not only outperforms existing sampling approaches but also surpasses greedy decoding, while maintaining consistent performance even at high temperatures.

Title: Understanding Audiovisual Deepfake Detection: Techniques, Challenges, Human Factors and Perceptual Insights

• Authors: Ammarah Hashmi, Sahibzada Adil Shahzad, Chia-Wen Lin, Yu Tsao, Hsin-Min Wang
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM, cs.SD, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.07650
• Pdf URL: https://arxiv.org/pdf/2411.07650
• Copy Paste: [[2411.07650]] Understanding Audiovisual Deepfake Detection: Techniques, Challenges, Human Factors and Perceptual Insights(https://arxiv.org/abs/2411.07650)
• Keywords: security
• Abstract: Deep Learning has been successfully applied in diverse fields, and its impact on deepfake detection is no exception. Deepfakes are fake yet realistic synthetic content that can be used deceitfully for political impersonation, phishing, slandering, or spreading misinformation. Despite extensive research on unimodal deepfake detection, identifying complex deepfakes through joint analysis of audio and visual streams remains relatively unexplored. To fill this gap, this survey first provides an overview of audiovisual deepfake generation techniques, applications, and their consequences, and then provides a comprehensive review of state-of-the-art methods that combine audio and visual modalities to enhance detection accuracy, summarizing and critically analyzing their strengths and limitations. Furthermore, we discuss existing open source datasets for a deeper understanding, which can contribute to the research community and provide necessary information to beginners who want to analyze deep learning-based audiovisual methods for video forensics. By bridging the gap between unimodal and multimodal approaches, this paper aims to improve the effectiveness of deepfake detection strategies and guide future research in cybersecurity and media integrity.

Title: Mitigating Bias in Queer Representation within Large Language Models: A Collaborative Agent Approach

• Authors: Tianyi Huang (1), Arya Somasundaram (1) ((1) App-In Club)
• Subjects: cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2411.07656
• Pdf URL: https://arxiv.org/pdf/2411.07656
• Copy Paste: [[2411.07656]] Mitigating Bias in Queer Representation within Large Language Models: A Collaborative Agent Approach(https://arxiv.org/abs/2411.07656)
• Keywords: fair, large language model
• Abstract: Large Language Models (LLMs) often perpetuate biases in pronoun usage, leading to misrepresentation or exclusion of queer individuals. This paper addresses the specific problem of biased pronoun usage in LLM outputs, particularly the inappropriate use of traditionally gendered pronouns ("he," "she") when inclusive language is needed to accurately represent all identities. We introduce a collaborative agent pipeline designed to mitigate these biases by analyzing and optimizing pronoun usage for inclusivity. Our multi-agent framework includes specialized agents for both bias detection and correction. Experimental evaluations using the Tango dataset-a benchmark focused on gender pronoun usage-demonstrate that our approach significantly improves inclusive pronoun classification, achieving a 32.6 percentage point increase over GPT-4o in correctly disagreeing with inappropriate traditionally gendered pronouns $(\chi^2 = 38.57, p < 0.0001)$. These results accentuate the potential of agent-driven frameworks in enhancing fairness and inclusivity in AI-generated content, demonstrating their efficacy in reducing biases and promoting socially responsible AI.

Title: Is Graph Convolution Always Beneficial For Every Feature?

• Authors: Yilun Zheng, Xiang Li, Sitao Luan, Xiaojiang Peng, Lihui Chen
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2411.07663
• Pdf URL: https://arxiv.org/pdf/2411.07663
• Copy Paste: [[2411.07663]] Is Graph Convolution Always Beneficial For Every Feature?(https://arxiv.org/abs/2411.07663)
• Keywords: robust, extraction
• Abstract: Graph Neural Networks (GNNs) have demonstrated strong capabilities in processing structured data. While traditional GNNs typically treat each feature dimension equally during graph convolution, we raise an important question: Is the graph convolution operation equally beneficial for each feature? If not, the convolution operation on certain feature dimensions can possibly lead to harmful effects, even worse than the convolution-free models. In prior studies, to assess the impacts of graph convolution on features, people proposed metrics based on feature homophily to measure feature consistency with the graph topology. However, these metrics have shown unsatisfactory alignment with GNN performance and have not been effectively employed to guide feature selection in GNNs. To address these limitations, we introduce a novel metric, Topological Feature Informativeness (TFI), to distinguish between GNN-favored and GNN-disfavored features, where its effectiveness is validated through both theoretical analysis and empirical observations. Based on TFI, we propose a simple yet effective Graph Feature Selection (GFS) method, which processes GNN-favored and GNN-disfavored features separately, using GNNs and non-GNN models. Compared to original GNNs, GFS significantly improves the extraction of useful topological information from each feature with comparable computational costs. Extensive experiments show that after applying GFS to 8 baseline and state-of-the-art (SOTA) GNN architectures across 10 datasets, 83.75% of the GFS-augmented cases show significant performance boosts. Furthermore, our proposed TFI metric outperforms other feature selection methods. These results validate the effectiveness of both GFS and TFI. Additionally, we demonstrate that GFS's improvements are robust to hyperparameter tuning, highlighting its potential as a universal method for enhancing various GNN architectures.

Title: Evaluating the Generation of Spatial Relations in Text and Image Generative Models

• Authors: Shang Hong Sim, Clarence Lee, Alvin Tan, Cheston Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07664
• Pdf URL: https://arxiv.org/pdf/2411.07664
• Copy Paste: [[2411.07664]] Evaluating the Generation of Spatial Relations in Text and Image Generative Models(https://arxiv.org/abs/2411.07664)
• Keywords: generative, large language model
• Abstract: Understanding spatial relations is a crucial cognitive ability for both humans and AI. While current research has predominantly focused on the benchmarking of text-to-image (T2I) models, we propose a more comprehensive evaluation that includes \textit{both} T2I and Large Language Models (LLMs). As spatial relations are naturally understood in a visuo-spatial manner, we develop an approach to convert LLM outputs into an image, thereby allowing us to evaluate both T2I models and LLMs \textit{visually}. We examined the spatial relation understanding of 8 prominent generative models (3 T2I models and 5 LLMs) on a set of 10 common prepositions, as well as assess the feasibility of automatic evaluation methods. Surprisingly, we found that T2I models only achieve subpar performance despite their impressive general image-generation abilities. Even more surprisingly, our results show that LLMs are significantly more accurate than T2I models in generating spatial relations, despite being primarily trained on textual data. We examined reasons for model failures and highlight gaps that can be filled to enable more spatially faithful generations.

Title: Rethinking Structure Learning For Graph Neural Networks

• Authors: Yilun Zheng, Zhuofan Zhang, Ziming Wang, Xiang Li, Sitao Luan, Xiaojiang Peng, Lihui Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07672
• Pdf URL: https://arxiv.org/pdf/2411.07672
• Copy Paste: [[2411.07672]] Rethinking Structure Learning For Graph Neural Networks(https://arxiv.org/abs/2411.07672)
• Keywords: fair
• Abstract: To improve the performance of Graph Neural Networks (GNNs), Graph Structure Learning (GSL) has been extensively applied to reconstruct or refine original graph structures, effectively addressing issues like heterophily, over-squashing, and noisy structures. While GSL is generally thought to improve GNN performance, it often leads to longer training times and more hyperparameter tuning. Besides, the distinctions among current GSL methods remain ambiguous from the perspective of GNN training, and there is a lack of theoretical analysis to quantify their effectiveness. Recent studies further suggest that, under fair comparisons with the same hyperparameter tuning, GSL does not consistently outperform baseline GNNs. This motivates us to ask a critical question: is GSL really useful for GNNs? To address this question, this paper makes two key contributions. First, we propose a new GSL framework, which includes three steps: GSL base (the representation used for GSL) construction, new structure construction, and view fusion, to better understand the effectiveness of GSL in GNNs. Second, after graph convolution, we analyze the differences in mutual information (MI) between node representations derived from the original topology and those from the newly constructed topology. Surprisingly, our empirical observations and theoretical analysis show that no matter which type of graph structure construction methods are used, after feeding the same GSL bases to the newly constructed graph, there is no MI gain compared to the original GSL bases. To fairly reassess the effectiveness of GSL, we conduct ablation experiments and find that it is the pretrained GSL bases that enhance GNN performance, and in most cases, GSL cannot improve GNN performance. This finding encourages us to rethink the essential components in GNNs, such as self-training and structural encoding, in GNN design rather than GSL.

Title: What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?

• Authors: Katie Kang, Amrith Setlur, Dibya Ghosh, Jacob Steinhardt, Claire Tomlin, Sergey Levine, Aviral Kumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07681
• Pdf URL: https://arxiv.org/pdf/2411.07681
• Copy Paste: [[2411.07681]] What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?(https://arxiv.org/abs/2411.07681)
• Keywords: robust, large language model
• Abstract: Despite the remarkable capabilities of modern large language models (LLMs), the mechanisms behind their problem-solving abilities remain elusive. In this work, we aim to better understand how the learning dynamics of LLM finetuning shapes downstream generalization. Our analysis focuses on reasoning tasks, whose problem structure allows us to distinguish between memorization (the exact replication of reasoning steps from the training data) and performance (the correctness of the final solution). We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy: the accuracy of model samples on training queries before they begin to copy the exact reasoning steps from the training set. On the dataset level, this metric is able to reliably predict test accuracy, achieving $R^2$ of around or exceeding 0.9 across various models (Llama3 8, Gemma2 9B), datasets (GSM8k, MATH), and training configurations. On a per-example level, this metric is also indicative of whether individual model predictions are robust to perturbations in the training query. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies. We focus on data curation as an example, and show that prioritizing examples with low pre-memorization accuracy leads to 1.5-2x improvements in data efficiency compared to i.i.d. data scaling, and outperforms other standard data curation techniques.

Title: Fast Disentangled Slim Tensor Learning for Multi-view Clustering

• Authors: Deng Xu, Chao Zhang, Zechao Li, Chunlin Chen, Huaxiong Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07685
• Pdf URL: https://arxiv.org/pdf/2411.07685
• Copy Paste: [[2411.07685]] Fast Disentangled Slim Tensor Learning for Multi-view Clustering(https://arxiv.org/abs/2411.07685)
• Keywords: robust
• Abstract: Tensor-based multi-view clustering has recently received significant attention due to its exceptional ability to explore cross-view high-order correlations. However, most existing methods still encounter some limitations. (1) Most of them explore the correlations among different affinity matrices, making them unscalable to large-scale data. (2) Although some methods address it by introducing bipartite graphs, they may result in sub-optimal solutions caused by an unstable anchor selection process. (3) They generally ignore the negative impact of latent semantic-unrelated information in each view. To tackle these issues, we propose a new approach termed fast Disentangled Slim Tensor Learning (DSTL) for multi-view clustering . Instead of focusing on the multi-view graph structures, DSTL directly explores the high-order correlations among multi-view latent semantic representations based on matrix factorization. To alleviate the negative influence of feature redundancy, inspired by robust PCA, DSTL disentangles the latent low-dimensional representation into a semantic-unrelated part and a semantic-related part for each view. Subsequently, two slim tensors are constructed with tensor-based regularization. To further enhance the quality of feature disentanglement, the semantic-related representations are aligned across views through a consensus alignment indicator. Our proposed model is computationally efficient and can be solved effectively. Extensive experiments demonstrate the superiority and efficiency of DSTL over state-of-the-art approaches. The code of DSTL is available at this https URL.

Title: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAG

• Authors: Zilun Zhang, Haozhan Shen, Tiancheng Zhao, Yuhao Wang, Bin Chen, Yuxiang Cai, Yongheng Shang, Jianwei Yin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07688
• Pdf URL: https://arxiv.org/pdf/2411.07688
• Copy Paste: [[2411.07688]] Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAG(https://arxiv.org/abs/2411.07688)
• Keywords: large language model
• Abstract: Ultra High Resolution (UHR) remote sensing imagery (RSI) (e.g. 100,000 $\times$ 100,000 pixels or more) poses a significant challenge for current Remote Sensing Multimodal Large Language Models (RSMLLMs). If choose to resize the UHR image to standard input image size, the extensive spatial and contextual information that UHR images contain will be neglected. Otherwise, the original size of these images often exceeds the token limits of standard RSMLLMs, making it difficult to process the entire image and capture long-range dependencies to answer the query based on the abundant visual context. In this paper, we introduce ImageRAG for RS, a training-free framework to address the complexities of analyzing UHR remote sensing imagery. By transforming UHR remote sensing image analysis task to image's long context selection task, we design an innovative image contextual retrieval mechanism based on the Retrieval-Augmented Generation (RAG) technique, denoted as ImageRAG. ImageRAG's core innovation lies in its ability to selectively retrieve and focus on the most relevant portions of the UHR image as visual contexts that pertain to a given query. Fast path and slow path are proposed in this framework to handle this task efficiently and effectively. ImageRAG allows RSMLLMs to manage extensive context and spatial information from UHR RSI, ensuring the analysis is both accurate and efficient.

Title: A Call to Reconsider Certification Authority Authorization (CAA)

• Authors: Pouyan Fotouhi Tehrani, Raphael Hiesgen, Thomas C. Schmidt, Matthias Wählisch
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.07702
• Pdf URL: https://arxiv.org/pdf/2411.07702
• Copy Paste: [[2411.07702]] A Call to Reconsider Certification Authority Authorization (CAA)(https://arxiv.org/abs/2411.07702)
• Keywords: security
• Abstract: Certification Authority Authentication (CAA) is a safeguard against illegitimate certificate issuance. We show how shortcomings in CAA concepts and operational aspects undermine its effectiveness in preventing certificate misissuance. Our discussion reveals pitfalls and highlights best practices when designing security protocols based on DNS.

Title: Emotion Classification of Children Expressions

• Authors: Sanchayan Vivekananthan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07708
• Pdf URL: https://arxiv.org/pdf/2411.07708
• Copy Paste: [[2411.07708]] Emotion Classification of Children Expressions(https://arxiv.org/abs/2411.07708)
• Keywords: robust, diffusion
• Abstract: This paper proposes a process for a classification model for the facial expressions. The proposed process would aid in specific categorisation of children's emotions from 2 emotions namely 'Happy' and 'Sad'. Since the existing emotion recognition systems algorithms primarily train on adult faces, the model developed is achieved by using advanced concepts of models with Squeeze-andExcitation blocks, Convolutional Block Attention modules, and robust data augmentation. Stable Diffusion image synthesis was used for expanding and diversifying the data set generating realistic and various training samples. The model designed using Batch Normalisation, Dropout, and SE Attention mechanisms for the classification of children's emotions achieved an accuracy rate of 89\% due to these methods improving the precision of emotion recognition in children. The relative importance of this issue is raised in this study with an emphasis on the call for a more specific model in emotion detection systems for the young generation with specific direction on how the young people can be assisted to manage emotions while online.

Title: OWLed: Outlier-weighed Layerwise Pruning for Efficient Autonomous Driving Framework

• Authors: Jiaxi Li, Lu Yin, Xilu Wang
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.07711
• Pdf URL: https://arxiv.org/pdf/2411.07711
• Copy Paste: [[2411.07711]] OWLed: Outlier-weighed Layerwise Pruning for Efficient Autonomous Driving Framework(https://arxiv.org/abs/2411.07711)
• Keywords: robust, large language model
• Abstract: The integration of Large Language Models (LLMs) into autonomous driving systems offers promising enhancements in environmental understanding and decision-making. However, the substantial computational demands of deploying LLMs locally on vehicles render this approach unfeasible for real-world automotive applications. To address this challenge, we introduce OWLed, the Outlier-Weighed Layerwise Pruning for Efficient Autonomous Driving Framework that leverages outlier-weighted layerwise sparsity for model compression. Our method assigns non-uniform sparsity ratios to different layers based on the distribution of outlier features, significantly reducing the model size without the need for fine-tuning. To ensure the compressed model adapts well to autonomous driving tasks, we incorporate driving environment data into both the calibration and pruning processes. Our empirical studies reveal that the encoder component is more sensitive to pruning than the LLM, highlighting its critical role in the system. Experimental results demonstrate that OWLed outperforms existing methods in perception, action prediction, and language understanding while substantially lowering computational requirements. These findings underscore the potential of combining advanced pruning techniques with LLMs to develop efficient and robust autonomous driving systems capable of handling complex scenarios. Code will be made publicly available.

Title: ALOcc: Adaptive Lifting-based 3D Semantic Occupancy and Cost Volume-based Flow Prediction

• Authors: Dubing Chen, Jin Fang, Wencheng Han, Xinjing Cheng, Junbo Yin, Chenzhong Xu, Fahad Shahbaz Khan, Jianbing Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07725
• Pdf URL: https://arxiv.org/pdf/2411.07725
• Copy Paste: [[2411.07725]] ALOcc: Adaptive Lifting-based 3D Semantic Occupancy and Cost Volume-based Flow Prediction(https://arxiv.org/abs/2411.07725)
• Keywords: robust
• Abstract: Vision-based semantic occupancy and flow prediction plays a crucial role in providing spatiotemporal cues for real-world tasks, such as autonomous driving. Existing methods prioritize higher accuracy to cater to the demands of these tasks. In this work, we strive to improve performance by introducing a series of targeted improvements for 3D semantic occupancy prediction and flow estimation. First, we introduce an occlusion-aware adaptive lifting mechanism with a depth denoising technique to improve the robustness of 2D-to-3D feature transformation and reduce the reliance on depth priors. Second, we strengthen the semantic consistency between 3D features and their original 2D modalities by utilizing shared semantic prototypes to jointly constrain both 2D and 3D features. This is complemented by confidence- and category-based sampling strategies to tackle long-tail challenges in 3D space. To alleviate the feature encoding burden in the joint prediction of semantics and flow, we propose a BEV cost volume-based prediction method that links flow and semantic features through a cost volume and employs a classification-regression supervision scheme to address the varying flow scales in dynamic scenes. Our purely convolutional architecture framework, named ALOcc, achieves an optimal tradeoff between speed and accuracy achieving state-of-the-art results on multiple benchmarks. On Occ3D and training without the camera visible mask, our ALOcc achieves an absolute gain of 2.5\% in terms of RayIoU while operating at a comparable speed compared to the state-of-the-art, using the same input size (256$\times$704) and ResNet-50 backbone. Our method also achieves 2nd place in the CVPR24 Occupancy and Flow Prediction Competition.

Title: Efficient 3D Perception on Multi-Sweep Point Cloud with Gumbel Spatial Pruning

• Authors: Jianhao Li, Tianyu Sun, Xueqian Zhang, Zhongdao Wang, Bailan Feng, Hengshuang Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07742
• Pdf URL: https://arxiv.org/pdf/2411.07742
• Copy Paste: [[2411.07742]] Efficient 3D Perception on Multi-Sweep Point Cloud with Gumbel Spatial Pruning(https://arxiv.org/abs/2411.07742)
• Keywords: segmentation
• Abstract: This paper studies point cloud perception within outdoor environments. Existing methods face limitations in recognizing objects located at a distance or occluded, due to the sparse nature of outdoor point clouds. In this work, we observe a significant mitigation of this problem by accumulating multiple temporally consecutive LiDAR sweeps, resulting in a remarkable improvement in perception accuracy. However, the computation cost also increases, hindering previous approaches from utilizing a large number of LiDAR sweeps. To tackle this challenge, we find that a considerable portion of points in the accumulated point cloud is redundant, and discarding these points has minimal impact on perception accuracy. We introduce a simple yet effective Gumbel Spatial Pruning (GSP) layer that dynamically prunes points based on a learned end-to-end sampling. The GSP layer is decoupled from other network components and thus can be seamlessly integrated into existing point cloud network architectures. Without incurring additional computational overhead, we increase the number of LiDAR sweeps from 10, a common practice, to as many as 40. Consequently, there is a significant enhancement in perception performance. For instance, in nuScenes 3D object detection and BEV map segmentation tasks, our pruning strategy improves the vanilla TransL baseline and other baseline methods.

Title: Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning

• Authors: Alexi Canesse, Mathieu Petitbois, Ludovic Denoyer, Sylvain Lamprier, Rémy Portelas
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.07760
• Pdf URL: https://arxiv.org/pdf/2411.07760
• Copy Paste: [[2411.07760]] Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning(https://arxiv.org/abs/2411.07760)
• Keywords: transformer
• Abstract: Offline Reinforcement Learning (RL) has emerged as a powerful alternative to imitation learning for behavior modeling in various domains, particularly in complex navigation tasks. An existing challenge with Offline RL is the signal-to-noise ratio, i.e. how to mitigate incorrect policy updates due to errors in value estimates. Towards this, multiple works have demonstrated the advantage of hierarchical offline RL methods, which decouples high-level path planning from low-level path following. In this work, we present a novel hierarchical transformer-based approach leveraging a learned quantizer of the space. This quantization enables the training of a simpler zone-conditioned low-level policy and simplifies planning, which is reduced to discrete autoregressive prediction. Among other benefits, zone-level reasoning in planning enables explicit trajectory stitching rather than implicit stitching based on noisy value function estimates. By combining this transformer-based planner with recent advancements in offline RL, our proposed approach achieves state-of-the-art results in complex long-distance navigation environments.

Title: ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization

• Authors: Weibo Zhao, Yubin Shi, Xinyu Lyu, Wanchen Sui, Shen Li, Yong Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07762
• Pdf URL: https://arxiv.org/pdf/2411.07762
• Copy Paste: [[2411.07762]] ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization(https://arxiv.org/abs/2411.07762)
• Keywords: extraction, large language model
• Abstract: Quantization stands as a pivotal technique for large language model (LLM) serving, yet it poses significant challenges particularly in achieving effective low-bit quantization. The limited numerical mapping makes the quantized model produce a non-trivial error, bringing out intolerable performance degration. This paper is anchored in the basic idea of model compression objectives, and delves into the layer-wise error distribution of LLMs during post-training quantization. Subsequently, we introduce ASER, an algorithm consisting of (1) Error Reconstruction: low-rank compensation for quantization error with LoRA-style matrices constructed by whitening SVD; (2) Activation Smoothing: outlier extraction to gain smooth activation and better error compensation. ASER is capable of quantizing typical LLMs to low-bit ones, particularly preserving accuracy even in W4A8 per-channel setup. Experimental results show that ASER is competitive among the state-of-the-art quantization algorithms, showing potential to activation quantization, with minor overhead.

Title: Novel View Synthesis with Pixel-Space Diffusion Models

• Authors: Noam Elata, Bahjat Kawar, Yaron Ostrovsky-Berman, Miriam Farber, Ron Sokolovsky
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07765
• Pdf URL: https://arxiv.org/pdf/2411.07765
• Copy Paste: [[2411.07765]] Novel View Synthesis with Pixel-Space Diffusion Models(https://arxiv.org/abs/2411.07765)
• Keywords: diffusion, generative
• Abstract: Synthesizing a novel view from a single input image is a challenging task. Traditionally, this task was approached by estimating scene depth, warping, and inpainting, with machine learning models enabling parts of the pipeline. More recently, generative models are being increasingly employed in novel view synthesis (NVS), often encompassing the entire end-to-end system. In this work, we adapt a modern diffusion model architecture for end-to-end NVS in the pixel space, substantially outperforming previous state-of-the-art (SOTA) techniques. We explore different ways to encode geometric information into the network. Our experiments show that while these methods may enhance performance, their impact is minor compared to utilizing improved generative models. Moreover, we introduce a novel NVS training scheme that utilizes single-view datasets, capitalizing on their relative abundance compared to their multi-view counterparts. This leads to improved generalization capabilities to scenes with out-of-domain content.

Title: Automatic Album Sequencing

• Authors: Vincent Herrmann, Dylan R. Ashley, Jürgen Schmidhuber
• Subjects: cs.LG, cs.AI, cs.CL, cs.MM, cs.SD
• Abstract URL: https://arxiv.org/abs/2411.07772
• Pdf URL: https://arxiv.org/pdf/2411.07772
• Copy Paste: [[2411.07772]] Automatic Album Sequencing(https://arxiv.org/abs/2411.07772)
• Keywords: transformer
• Abstract: Album sequencing is a critical part of the album production process. Recently, a data-driven approach was proposed that sequences general collections of independent media by extracting the narrative essence of the items in the collections. While this approach implies an album sequencing technique, it is not widely accessible to a less technical audience, requiring advanced knowledge of machine learning techniques to use. To address this, we introduce a new user-friendly web-based tool that allows a less technical audience to upload music tracks, execute this technique in one click, and subsequently presents the result in a clean visualization to the user. To both increase the number of templates available to the user and address shortcomings of previous work, we also introduce a new direct transformer-based album sequencing method. We find that our more direct method outperforms a random baseline but does not reach the same performance as the narrative essence approach. Both methods are included in our web-based user interface, and this -- alongside a full copy of our implementation -- is publicly available at this https URL

Title: Likelihood as a Performance Gauge for Retrieval-Augmented Generation

• Authors: Tianyu Liu, Jirui Qi, Paul He, Arianna Bisazza, Mrinmaya Sachan, Ryan Cotterell
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07773
• Pdf URL: https://arxiv.org/pdf/2411.07773
• Copy Paste: [[2411.07773]] Likelihood as a Performance Gauge for Retrieval-Augmented Generation(https://arxiv.org/abs/2411.07773)
• Keywords: large language model
• Abstract: Recent work finds that retrieval-augmented generation with large language models is prone to be influenced by the order of retrieved documents in the context. However, the lack of in-depth analysis limits the use of this phenomenon for prompt engineering in practice. In this study, we posit that likelihoods serve as an effective gauge for language model performance. Through experiments on two question-answering datasets with a variety of state-of-the-art language models, we reveal correlations between answer accuracy and the likelihood of the question at both the corpus level and the instance level. In addition, we find that question likelihood can also indicate the position of the task-relevant information in the context. Based on these findings, we propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance. We demonstrate their effectiveness with experiments. In addition, our likelihood-based methods are efficient, as they only need to compute the likelihood of the input, requiring much fewer language model passes than heuristic prompt engineering methods that require generating responses. Our analysis deepens our understanding of how input prompts affect model performance and provides a promising direction for efficient prompt optimization.

Title: Interaction Asymmetry: A General Principle for Learning Composable Abstractions

• Authors: Jack Brady, Julius von Kügelgen, Sébastien Lachapelle, Simon Buchholz, Thomas Kipf, Wieland Brendel
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07784
• Pdf URL: https://arxiv.org/pdf/2411.07784
• Copy Paste: [[2411.07784]] Interaction Asymmetry: A General Principle for Learning Composable Abstractions(https://arxiv.org/abs/2411.07784)
• Keywords: transformer
• Abstract: Learning disentangled representations of concepts and re-composing them in unseen ways is crucial for generalizing to out-of-domain situations. However, the underlying properties of concepts that enable such disentanglement and compositional generalization remain poorly understood. In this work, we propose the principle of interaction asymmetry which states: "Parts of the same concept have more complex interactions than parts of different concepts". We formalize this via block diagonality conditions on the $(n+1)$th order derivatives of the generator mapping concepts to observed data, where different orders of "complexity" correspond to different $n$. Using this formalism, we prove that interaction asymmetry enables both disentanglement and compositional generalization. Our results unify recent theoretical results for learning concepts of objects, which we show are recovered as special cases with $n\!=\!0$ or $1$. We provide results for up to $n\!=\!2$, thus extending these prior works to more flexible generator functions, and conjecture that the same proof strategies generalize to larger $n$. Practically, our theory suggests that, to disentangle concepts, an autoencoder should penalize its latent capacity and the interactions between concepts during decoding. We propose an implementation of these criteria using a flexible Transformer-based VAE, with a novel regularizer on the attention weights of the decoder. On synthetic image datasets consisting of objects, we provide evidence that this model can achieve comparable object disentanglement to existing models that use more explicit object-centric priors.

Title: Feature Fusion Transferability Aware Transformer for Unsupervised Domain Adaptation

• Authors: Xiaowei Yu, Zhe Huang, Zao Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07794
• Pdf URL: https://arxiv.org/pdf/2411.07794
• Copy Paste: [[2411.07794]] Feature Fusion Transferability Aware Transformer for Unsupervised Domain Adaptation(https://arxiv.org/abs/2411.07794)
• Keywords: transformer
• Abstract: Unsupervised domain adaptation (UDA) aims to leverage the knowledge learned from labeled source domains to improve performance on the unlabeled target domains. While Convolutional Neural Networks (CNNs) have been dominant in previous UDA methods, recent research has shown promise in applying Vision Transformers (ViTs) to this task. In this study, we propose a novel Feature Fusion Transferability Aware Transformer (FFTAT) to enhance ViT performance in UDA tasks. Our method introduces two key innovations: First, we introduce a patch discriminator to evaluate the transferability of patches, generating a transferability matrix. We integrate this matrix into self-attention, directing the model to focus on transferable patches. Second, we propose a feature fusion technique to fuse embeddings in the latent space, enabling each embedding to incorporate information from all others, thereby improving generalization. These two components work in synergy to enhance feature representation learning. Extensive experiments on widely used benchmarks demonstrate that our method significantly improves UDA performance, achieving state-of-the-art (SOTA) results.

Title: InvisMark: Invisible and Robust Watermarking for AI-generated Image Provenance

• Authors: Rui Xu, Mengya (Mia)Hu, Deren Lei, Yaxi Li, David Lowe, Alex Gorevski, Mingyu Wang, Emily Ching, Alex Deng
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07795
• Pdf URL: https://arxiv.org/pdf/2411.07795
• Copy Paste: [[2411.07795]] InvisMark: Invisible and Robust Watermarking for AI-generated Image Provenance(https://arxiv.org/abs/2411.07795)
• Keywords: attack, robust, watermark
• Abstract: The proliferation of AI-generated images has intensified the need for robust content authentication methods. We present InvisMark, a novel watermarking technique designed for high-resolution AI-generated images. Our approach leverages advanced neural network architectures and training strategies to embed imperceptible yet highly robust watermarks. InvisMark achieves state-of-the-art performance in imperceptibility (PSNR$\sim$51, SSIM $\sim$ 0.998) while maintaining over 97\% bit accuracy across various image manipulations. Notably, we demonstrate the successful encoding of 256-bit watermarks, significantly expanding payload capacity while preserving image quality. This enables the embedding of UUIDs with error correction codes, achieving near-perfect decoding success rates even under challenging image distortions. We also address potential vulnerabilities against advanced attacks and propose mitigation strategies. By combining high imperceptibility, extended payload capacity, and resilience to manipulations, InvisMark provides a robust foundation for ensuring media provenance in an era of increasingly sophisticated AI-generated content. Source code of this paper is available at: this https URL.

Title: Horticultural Temporal Fruit Monitoring via 3D Instance Segmentation and Re-Identification using Point Clouds

• Authors: Daniel Fusaro, Federico Magistri, Jens Behley, Alberto Pretto, Cyrill Stachniss
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.07799
• Pdf URL: https://arxiv.org/pdf/2411.07799
• Copy Paste: [[2411.07799]] Horticultural Temporal Fruit Monitoring via 3D Instance Segmentation and Re-Identification using Point Clouds(https://arxiv.org/abs/2411.07799)
• Keywords: segmentation
• Abstract: Robotic fruit monitoring is a key step toward automated agricultural production systems. Robots can significantly enhance plant and temporal fruit monitoring by providing precise, high-throughput assessments that overcome the limitations of traditional manual methods. Fruit monitoring is a challenging task due to the significant variation in size, shape, orientation, and occlusion of fruits. Also, fruits may be harvested or newly grown between recording sessions. Most methods are 2D image-based and they lack the 3D structure, depth, and spatial information, which represent key aspects of fruit monitoring. 3D colored point clouds, instead, can offer this information but they introduce challenges such as their sparsity and irregularity. In this paper, we present a novel approach for temporal fruit monitoring that addresses point clouds collected in a greenhouse over time. Our method segments fruits using a learning-based instance segmentation approach directly on the point cloud. Each segmented fruit is processed by a 3D sparse convolutional neural network to extract descriptors, which are used in an attention-based matching network to associate fruits with their instances from previous data collections. Experimental results on a real dataset of strawberries demonstrate that our approach outperforms other methods for fruits re-identification over time, allowing for precise temporal fruit monitoring in real and complex scenarios.

Title: Large-scale Remote Sensing Image Target Recognition and Automatic Annotation

• Authors: Wuzheng Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07802
• Pdf URL: https://arxiv.org/pdf/2411.07802
• Copy Paste: [[2411.07802]] Large-scale Remote Sensing Image Target Recognition and Automatic Annotation(https://arxiv.org/abs/2411.07802)
• Keywords: segmentation
• Abstract: This paper presents a method for object recognition and automatic labeling in large-area remote sensing images called LRSAA. The method integrates YOLOv11 and MobileNetV3-SSD object detection algorithms through ensemble learning to enhance model performance. Furthermore, it employs Poisson disk sampling segmentation techniques and the EIOU metric to optimize the training and inference processes of segmented images, followed by the integration of results. This approach not only reduces the demand for computational resources but also achieves a good balance between accuracy and speed. The source code for this project has been made publicly available on this https URL.

Title: Federated Low-Rank Adaptation with Differential Privacy over Wireless Networks

• Authors: Tianqu Kang, Zixin Wang, Hengtao He, Jun Zhang, Shenghui Song, Khaled B. Letaief
• Subjects: cs.LG, cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.07806
• Pdf URL: https://arxiv.org/pdf/2411.07806
• Copy Paste: [[2411.07806]] Federated Low-Rank Adaptation with Differential Privacy over Wireless Networks(https://arxiv.org/abs/2411.07806)
• Keywords: privacy, attack, federate
• Abstract: Fine-tuning large pre-trained foundation models (FMs) on distributed edge devices presents considerable computational and privacy challenges. Federated fine-tuning (FedFT) mitigates some privacy issues by facilitating collaborative model training without the need to share raw data. To lessen the computational burden on resource-limited devices, combining low-rank adaptation (LoRA) with federated learning enables parameter-efficient fine-tuning. Additionally, the split FedFT architecture partitions an FM between edge devices and a central server, reducing the necessity for complete model deployment on individual devices. However, the risk of privacy eavesdropping attacks in FedFT remains a concern, particularly in sensitive areas such as healthcare and finance. In this paper, we propose a split FedFT framework with differential privacy (DP) over wireless networks, where the inherent wireless channel noise in the uplink transmission is utilized to achieve DP guarantees without adding an extra artificial noise. We shall investigate the impact of the wireless noise on convergence performance of the proposed framework. We will also show that by updating only one of the low-rank matrices in the split FedFT with DP, the proposed method can mitigate the noise amplification effect. Simulation results will demonstrate that the proposed framework achieves higher accuracy under strict privacy budgets compared to baseline methods.

Title: Dual-Criterion Model Aggregation in Federated Learning: Balancing Data Quantity and Quality

• Authors: Haizhou Zhang, Xianjia Yu, Tomi Westerlund
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07816
• Pdf URL: https://arxiv.org/pdf/2411.07816
• Copy Paste: [[2411.07816]] Dual-Criterion Model Aggregation in Federated Learning: Balancing Data Quantity and Quality(https://arxiv.org/abs/2411.07816)
• Keywords: security, privacy, federate
• Abstract: Federated learning (FL) has become one of the key methods for privacy-preserving collaborative learning, as it enables the transfer of models without requiring local data exchange. Within the FL framework, an aggregation algorithm is recognized as one of the most crucial components for ensuring the efficacy and security of the system. Existing average aggregation algorithms typically assume that all client-trained data holds equal value or that weights are based solely on the quantity of data contributed by each client. In contrast, alternative approaches involve training the model locally after aggregation to enhance adaptability. However, these approaches fundamentally ignore the inherent heterogeneity between different clients' data and the complexity of variations in data at the aggregation stage, which may lead to a suboptimal global model. To address these issues, this study proposes a novel dual-criterion weighted aggregation algorithm involving the quantity and quality of data from the client node. Specifically, we quantify the data used for training and perform multiple rounds of local model inference accuracy evaluation on a specialized dataset to assess the data quality of each client. These two factors are utilized as weights within the aggregation process, applied through a dynamically weighted summation of these two factors. This approach allows the algorithm to adaptively adjust the weights, ensuring that every client can contribute to the global model, regardless of their data's size or initial quality. Our experiments show that the proposed algorithm outperforms several existing state-of-the-art aggregation approaches on both a general-purpose open-source dataset, CIFAR-10, and a dataset specific to visual obstacle avoidance.

Title: Query Optimization for Parametric Knowledge Refinement in Retrieval-Augmented Large Language Models

• Authors: Youan Cong, Cheng Wang, Pritom Saha Akash, Kevin Chen-Chuan Chang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.07820
• Pdf URL: https://arxiv.org/pdf/2411.07820
• Copy Paste: [[2411.07820]] Query Optimization for Parametric Knowledge Refinement in Retrieval-Augmented Large Language Models(https://arxiv.org/abs/2411.07820)
• Keywords: large language model
• Abstract: We introduce the \textit{Extract-Refine-Retrieve-Read} (ERRR) framework, a novel approach designed to bridge the pre-retrieval information gap in Retrieval-Augmented Generation (RAG) systems through query optimization tailored to meet the specific knowledge requirements of Large Language Models (LLMs). Unlike conventional query optimization techniques used in RAG, the ERRR framework begins by extracting parametric knowledge from LLMs, followed by using a specialized query optimizer for refining these queries. This process ensures the retrieval of only the most pertinent information essential for generating accurate responses. Moreover, to enhance flexibility and reduce computational costs, we propose a trainable scheme for our pipeline that utilizes a smaller, tunable model as the query optimizer, which is refined through knowledge distillation from a larger teacher model. Our evaluations on various question-answering (QA) datasets and with different retrieval systems show that ERRR consistently outperforms existing baselines, proving to be a versatile and cost-effective module for improving the utility and accuracy of RAG systems.

Title: Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices

• Authors: Kilian Pfeiffer, Mohamed Aboelenien Ahmed, Ramin Khalili, Jörg Henkel
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.07826
• Pdf URL: https://arxiv.org/pdf/2411.07826
• Copy Paste: [[2411.07826]] Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices(https://arxiv.org/abs/2411.07826)
• Keywords: federate, transformer, large language model
• Abstract: In recent years, Large Language Models (LLMs) through Transformer structures have dominated many machine learning tasks, especially text processing. However, these models require massive amounts of data for training and induce high resource requirements, particularly in terms of the large number of Floating Point Operations (FLOPs) and the high amounts of memory needed. To fine-tune such a model in a parameter-efficient way, techniques like Adapter or LoRA have been developed. However, we observe that the application of LoRA, when used in federated learning (FL), while still being parameter-efficient, is memory and FLOP inefficient. Based on that observation, we develop a novel layer finetuning scheme that allows devices in cross-device FL to make use of pretrained neural networks (NNs) while adhering to given resource constraints. We show that our presented scheme outperforms the current state of the art when dealing with homogeneous or heterogeneous computation and memory constraints and is on par with LoRA regarding limited communication, thereby achieving significantly higher accuracies in FL training.

Title: Suite-IN: Aggregating Motion Features from Apple Suite for Robust Inertial Navigation

• Authors: Lan Sun, Songpengcheng Xia, Junyuan Deng, Jiarui Yang, Zengyuan Lai, Qi Wu, Ling Pei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07828
• Pdf URL: https://arxiv.org/pdf/2411.07828
• Copy Paste: [[2411.07828]] Suite-IN: Aggregating Motion Features from Apple Suite for Robust Inertial Navigation(https://arxiv.org/abs/2411.07828)
• Keywords: robust
• Abstract: With the rapid development of wearable technology, devices like smartphones, smartwatches, and headphones equipped with IMUs have become essential for applications such as pedestrian positioning. However, traditional pedestrian dead reckoning (PDR) methods struggle with diverse motion patterns, while recent data-driven approaches, though improving accuracy, often lack robustness due to reliance on a single this http URL our work, we attempt to enhance the positioning performance using the low-cost commodity IMUs embedded in the wearable devices. We propose a multi-device deep learning framework named Suite-IN, aggregating motion data from Apple Suite for inertial navigation. Motion data captured by sensors on different body parts contains both local and global motion information, making it essential to reduce the negative effects of localized movements and extract global motion representations from multiple devices.

Title: Towards Vision Mixture of Experts for Wildlife Monitoring on the Edge

• Authors: Emmanuel Azuh Mensah, Anderson Lee, Haoran Zhang, Yitong Shan, Kurtis Heimerl
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07834
• Pdf URL: https://arxiv.org/pdf/2411.07834
• Copy Paste: [[2411.07834]] Towards Vision Mixture of Experts for Wildlife Monitoring on the Edge(https://arxiv.org/abs/2411.07834)
• Keywords: privacy, transformer
• Abstract: The explosion of IoT sensors in industrial, consumer and remote sensing use cases has come with unprecedented demand for computing infrastructure to transmit and to analyze petabytes of data. Concurrently, the world is slowly shifting its focus towards more sustainable computing. For these reasons, there has been a recent effort to reduce the footprint of related computing infrastructure, especially by deep learning algorithms, for advanced insight generation. The `TinyML' community is actively proposing methods to save communication bandwidth and excessive cloud storage costs while reducing algorithm inference latency and promoting data privacy. Such proposed approaches should ideally process multiple types of data, including time series, audio, satellite images, and video, near the network edge as multiple data streams has been shown to improve the discriminative ability of learning algorithms, especially for generating fine grained results. Incidentally, there has been recent work on data driven conditional computation of subnetworks that has shown real progress in using a single model to share parameters among very different types of inputs such as images and text, reducing the computation requirement of multi-tower multimodal networks. Inspired by such line of work, we explore similar per patch conditional computation for the first time for mobile vision transformers (vision only case), that will eventually be used for single-tower multimodal edge models. We evaluate the model on Cornell Sap Sucker Woods 60, a fine grained bird species discrimination dataset. Our initial experiments uses $4X$ fewer parameters compared to MobileViTV2-1.0 with a $1$% accuracy drop on the iNaturalist '21 birds test data provided as part of the SSW60 dataset.

Title: FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training

• Authors: Philip Zmushko, Aleksandr Beznosikov, Martin Takáč, Samuel Horváth
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07837
• Pdf URL: https://arxiv.org/pdf/2411.07837
• Copy Paste: [[2411.07837]] FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training(https://arxiv.org/abs/2411.07837)
• Keywords: large language model
• Abstract: With the increase in the number of parameters in large language models, the process of pre-training and fine-tuning increasingly demands larger volumes of GPU memory. A significant portion of this memory is typically consumed by the optimizer state. To overcome this challenge, recent approaches such as low-rank adaptation (LoRA (Hu et al., 2021)), low-rank gradient projection (GaLore (Zhao et al., 2024)), and blockwise optimization (BAdam (Luo et al., 2024)) have been proposed. However, in all these algorithms, the $\textit{effective rank of the weight updates remains low-rank}$, which can lead to a substantial loss of information from the gradient. This loss can be critically important, especially during the pre-training stage. In this paper, we introduce $\texttt{FRUGAL}$ ($\textbf{F}$ull-$\textbf{R}$ank $\textbf{U}$pdates with $\textbf{G}$r$\textbf{A}$dient sp$\textbf{L}$itting), a new memory-efficient optimization framework. $\texttt{FRUGAL}$ leverages gradient splitting to perform low-dimensional updates using advanced algorithms (such as Adam), while updates along the remaining directions are executed via state-free methods like SGD or signSGD (Bernstein et al., 2018). Our framework can be integrated with various low-rank update selection techniques, including GaLore and BAdam. We provide theoretical convergence guarantees for our framework when using SGDM for low-dimensional updates and SGD for state-free updates. Additionally, our method consistently outperforms concurrent approaches across various fixed memory budgets, achieving state-of-the-art results in pre-training and fine-tuning tasks while balancing memory efficiency and performance metrics.

Title: Chain Association-based Attacking and Shielding Natural Language Processing Systems

• Authors: Jiacheng Huang, Long Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07843
• Pdf URL: https://arxiv.org/pdf/2411.07843
• Copy Paste: [[2411.07843]] Chain Association-based Attacking and Shielding Natural Language Processing Systems(https://arxiv.org/abs/2411.07843)
• Keywords: attack, large language model
• Abstract: Association as a gift enables people do not have to mention something in completely straightforward words and allows others to understand what they intend to refer to. In this paper, we propose a chain association-based adversarial attack against natural language processing systems, utilizing the comprehension gap between humans and machines. We first generate a chain association graph for Chinese characters based on the association paradigm for building search space of potential adversarial examples. Then, we introduce an discrete particle swarm optimization algorithm to search for the optimal adversarial examples. We conduct comprehensive experiments and show that advanced natural language processing models and applications, including large language models, are vulnerable to our attack, while humans appear good at understanding the perturbed text. We also explore two methods, including adversarial training and associative graph-based recovery, to shield systems from chain association-based attack. Since a few examples that use some derogatory terms, this paper contains materials that may be offensive or upsetting to some people.

Title: IAE: Irony-based Adversarial Examples for Sentiment Analysis Systems

• Authors: Xiaoyin Yi, Jiacheng Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07850
• Pdf URL: https://arxiv.org/pdf/2411.07850
• Copy Paste: [[2411.07850]] IAE: Irony-based Adversarial Examples for Sentiment Analysis Systems(https://arxiv.org/abs/2411.07850)
• Keywords: security, attack
• Abstract: Adversarial examples, which are inputs deliberately perturbed with imperceptible changes to induce model errors, have raised serious concerns for the reliability and security of deep neural networks (DNNs). While adversarial attacks have been extensively studied in continuous data domains such as images, the discrete nature of text presents unique challenges. In this paper, we propose Irony-based Adversarial Examples (IAE), a method that transforms straightforward sentences into ironic ones to create adversarial text. This approach exploits the rhetorical device of irony, where the intended meaning is opposite to the literal interpretation, requiring a deeper understanding of context to detect. The IAE method is particularly challenging due to the need to accurately locate evaluation words, substitute them with appropriate collocations, and expand the text with suitable ironic elements while maintaining semantic coherence. Our research makes the following key contributions: (1) We introduce IAE, a strategy for generating textual adversarial examples using irony. This method does not rely on pre-existing irony corpora, making it a versatile tool for creating adversarial text in various NLP tasks. (2) We demonstrate that the performance of several state-of-the-art deep learning models on sentiment analysis tasks significantly deteriorates when subjected to IAE attacks. This finding underscores the susceptibility of current NLP systems to adversarial manipulation through irony. (3) We compare the impact of IAE on human judgment versus NLP systems, revealing that humans are less susceptible to the effects of irony in text.

Title: Evidential time-to-event prediction model with well-calibrated uncertainty estimation

• Authors: Ling Huang, Yucheng Xing, Swapnil Mishra, Thierry Denoeux, Mengling Feng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07853
• Pdf URL: https://arxiv.org/pdf/2411.07853
• Copy Paste: [[2411.07853]] Evidential time-to-event prediction model with well-calibrated uncertainty estimation(https://arxiv.org/abs/2411.07853)
• Keywords: robust
• Abstract: Time-to-event analysis, or Survival analysis, provides valuable insights into clinical prognosis and treatment recommendations. However, this task is typically more challenging than other regression tasks due to the censored observations. Moreover, concerns regarding the reliability of predictions persist among clinicians, mainly attributed to the absence of confidence assessment, robustness, and calibration of prediction. To address those challenges, we introduce an evidential regression model designed especially for time-to-event prediction tasks, with which the most plausible event time, is directly quantified by aggregated Gaussian random fuzzy numbers (GRFNs). The GRFNs are a newly introduced family of random fuzzy subsets of the real line that generalizes both Gaussian random variables and Gaussian possibility distributions. Different from conventional methods that construct models based on strict data distribution, e.g., proportional hazard function, our model only assumes the event time is encoded in a real line GFRN without any strict distribution assumption, therefore offering more flexibility in complex data scenarios. Furthermore, the epistemic and aleatory uncertainty regarding the event time is quantified within the aggregated GRFN as well. Our model can, therefore, provide more detailed clinical decision-making guidance with two more degrees of information. The model is fit by minimizing a generalized negative log-likelihood function that accounts for data censoring based on uncertainty evidence reasoning. Experimental results on simulated datasets with varying data distributions and censoring scenarios, as well as on real-world datasets across diverse clinical settings and tasks, demonstrate that our model achieves both accurate and reliable performance, outperforming state-of-the-art methods.

Title: Tucano: Advancing Neural Text Generation for Portuguese

• Authors: Nicholas Kluge Corrêa, Aniket Sen, Sophia Falk, Shiza Fatimah
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07854
• Pdf URL: https://arxiv.org/pdf/2411.07854
• Copy Paste: [[2411.07854]] Tucano: Advancing Neural Text Generation for Portuguese(https://arxiv.org/abs/2411.07854)
• Keywords: transformer, generative
• Abstract: Significant advances have been made in natural language processing in recent years. However, our current deep learning approach to language modeling requires substantial resources in terms of data and computation. One of the side effects of this data-hungry paradigm is the current schism between languages, separating those considered high-resource, where most of the development happens and resources are available, and the low-resource ones, which struggle to attain the same level of performance and autonomy. This study aims to introduce a new set of resources to stimulate the future development of neural text generation in Portuguese. In this work, we document the development of GigaVerbo, a concatenation of deduplicated Portuguese text corpora amounting to 200 billion tokens. Via this corpus, we trained a series of decoder-transformers named Tucano. Our models perform equal or superior to other Portuguese and multilingual language models of similar size in several Portuguese benchmarks. The evaluation of our models also reveals that model performance on many currently available benchmarks used by the Portuguese NLP community has little to no correlation with the scaling of token ingestion during training, highlighting the limitations of such evaluations when it comes to the assessment of Portuguese generative language models. All derivatives of our study are openly released on GitHub and Hugging Face. See this https URL

Title: Verbosity $\neq$ Veracity: Demystify Verbosity Compensation Behavior of Large Language Models

• Authors: Yusen Zhang, Sarkar Snigdha Sarathi Das, Rui Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07858
• Pdf URL: https://arxiv.org/pdf/2411.07858
• Copy Paste: [[2411.07858]] Verbosity $\neq$ Veracity: Demystify Verbosity Compensation Behavior of Large Language Models(https://arxiv.org/abs/2411.07858)
• Keywords: large language model
• Abstract: When unsure about an answer, humans often respond with more words than necessary, hoping that part of the response will be correct. We observe a similar behavior in large language models (LLMs), which we term "Verbosity Compensation" (VC). VC is harmful because it confuses the user understanding, leading to low efficiency, and influences the LLM services by increasing the latency and cost of generating useless tokens. In this paper, we present the first work that defines and analyzes Verbosity Compensation, explores its causes, and proposes a simple mitigating approach. We define Verbosity Compensation as the behavior of generating responses that can be compressed without information loss when prompted to write concisely. Our experiments, conducted on five datasets of knowledge and reasoning-based QA tasks with 14 newly developed LLMs, reveal three conclusions. 1) We reveal a pervasive presence of verbosity compensation across all models and all datasets. Notably, GPT-4 exhibits a VC frequency of 50.40%. 2) We reveal the large performance gap between verbose and concise responses, with a notable difference of 27.61% on the Qasper dataset. We also demonstrate that this difference does not naturally diminish as LLM capability increases. Both 1) and 2) highlight the urgent need to mitigate the frequency of VC behavior and disentangle verbosity with veracity. We propose a simple yet effective cascade algorithm that replaces the verbose responses with the other model-generated responses. The results show that our approach effectively alleviates the VC of the Mistral model from 63.81% to 16.16% on the Qasper dataset. 3) We also find that verbose responses exhibit higher uncertainty across all five datasets, suggesting a strong connection between verbosity and model uncertainty. Our dataset and code are available at this https URL.

Title: CDXFormer: Boosting Remote Sensing Change Detection with Extended Long Short-Term Memory

• Authors: Zhenkai Wu, Xiaowen Ma, Rongrong Lian, Zhentao Lin, Wei Zhang
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.07863
• Pdf URL: https://arxiv.org/pdf/2411.07863
• Copy Paste: [[2411.07863]] CDXFormer: Boosting Remote Sensing Change Detection with Extended Long Short-Term Memory(https://arxiv.org/abs/2411.07863)
• Keywords: transformer
• Abstract: In complex scenes and varied conditions, effectively integrating spatial-temporal context is crucial for accurately identifying changes. However, current RS-CD methods lack a balanced consideration of performance and efficiency. CNNs lack global context, Transformers have quadratic computational complexity, and Mambas are restricted by CUDA acceleration. In this paper, we propose CDXFormer, with a core component that is a powerful XLSTM-based feature enhancement layer, integrating the advantages of linear computational complexity, global context perception, and strong interpret-ability. Specifically, we introduce a scale-specific Feature Enhancer layer, incorporating a Cross-Temporal Global Perceptron customized for semantic-accurate deep features, and a Cross-Temporal Spatial Refiner customized for detail-rich shallow features. Additionally, we propose a Cross-Scale Interactive Fusion module to progressively interact global change representations with spatial responses. Extensive experimental results demonstrate that CDXFormer achieves state-of-the-art performance across three benchmark datasets, offering a compelling balance between efficiency and accuracy. Code is available at this https URL.

Title: Trustful LLMs: Customizing and Grounding Text Generation with Knowledge Bases and Dual Decoders

• Authors: Xiaofeng Zhu, Jaya Krishna Mandivarapu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07870
• Pdf URL: https://arxiv.org/pdf/2411.07870
• Copy Paste: [[2411.07870]] Trustful LLMs: Customizing and Grounding Text Generation with Knowledge Bases and Dual Decoders(https://arxiv.org/abs/2411.07870)
• Keywords: attack, large language model
• Abstract: Although people are impressed by the content generation skills of large language models, the use of LLMs, such as ChatGPT, is limited by the domain grounding of the content. The correctness and groundedness of the generated content need to be based on a verified context, such as results from Retrieval-Augmented Generation (RAG). One important issue when adapting LLMs to a customized domain is that the generated responses are often incomplete, or the additions are not verified and may even be hallucinated. Prior studies on hallucination detection have focused on evaluation metrics, which are not easily adaptable to dynamic domains and can be vulnerable to attacks like jail-breaking. In this work, we propose 1) a post-processing algorithm that leverages knowledge triplets in RAG context to correct hallucinations and 2) a dual-decoder model that fuses RAG context to guide the generation process.

Title: Diverse capability and scaling of diffusion and auto-regressive models when learning abstract rules

• Authors: Binxu Wang, Jiaqi Shang, Haim Sompolinsky
• Subjects: cs.LG, cs.AI, cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2411.07873
• Pdf URL: https://arxiv.org/pdf/2411.07873
• Copy Paste: [[2411.07873]] Diverse capability and scaling of diffusion and auto-regressive models when learning abstract rules(https://arxiv.org/abs/2411.07873)
• Keywords: diffusion, generative
• Abstract: Humans excel at discovering regular structures from limited samples and applying inferred rules to novel settings. We investigate whether modern generative models can similarly learn underlying rules from finite samples and perform reasoning through conditional sampling. Inspired by Raven's Progressive Matrices task, we designed GenRAVEN dataset, where each sample consists of three rows, and one of 40 relational rules governing the object position, number, or attributes applies to all rows. We trained generative models to learn the data distribution, where samples are encoded as integer arrays to focus on rule learning. We compared two generative model families: diffusion (EDM, DiT, SiT) and autoregressive models (GPT2, Mamba). We evaluated their ability to generate structurally consistent samples and perform panel completion via unconditional and conditional sampling. We found diffusion models excel at unconditional generation, producing more novel and consistent samples from scratch and memorizing less, but performing less well in panel completion, even with advanced conditional sampling methods. Conversely, autoregressive models excel at completing missing panels in a rule-consistent manner but generate less consistent samples unconditionally. We observe diverse data scaling behaviors: for both model families, rule learning emerges at a certain dataset size - around 1000s examples per rule. With more training data, diffusion models improve both their unconditional and conditional generation capabilities. However, for autoregressive models, while panel completion improves with more training data, unconditional generation consistency declines. Our findings highlight complementary capabilities and limitations of diffusion and autoregressive models in rule learning and reasoning tasks, suggesting avenues for further research into their mechanisms and potential for human-like reasoning.

Title: INTRABENCH: Interactive Radiological Benchmark

• Authors: Constantin Ulrich, Tassilo Wald, Emily Tempus, Maximilian Rokuss, Paul F. Jaeger, Klaus Maier-Hein
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07885
• Pdf URL: https://arxiv.org/pdf/2411.07885
• Copy Paste: [[2411.07885]] INTRABENCH: Interactive Radiological Benchmark(https://arxiv.org/abs/2411.07885)
• Keywords: fair, segmentation
• Abstract: Current interactive segmentation approaches, inspired by the success of META's Segment Anything model, have achieved notable advancements, however, they come with substantial limitations that hinder their practical application in real clinical scenarios. These include unrealistic human interaction requirements, such as slice-by-slice operations for 2D models on 3D data, a lack of iterative refinement, and insufficient evaluation experiments. These shortcomings prevent accurate assessment of model performance and lead to inconsistent outcomes across studies. IntRaBench overcomes these challenges by offering a comprehensive and reproducible framework for evaluating interactive segmentation methods in realistic, clinically relevant scenarios. It includes diverse datasets, target structures, and segmentation models, and provides a flexible codebase that allows seamless integration of new models and prompting strategies. Additionally, we introduce advanced techniques to minimize clinician interaction, ensuring fair comparisons between 2D and 3D models. By open-sourcing IntRaBench, we invite the research community to integrate their models and prompting techniques, ensuring continuous and transparent evaluation of interactive segmentation models in 3D medical imaging.

Title: A Stochastic Optimization Framework for Private and Fair Learning From Decentralized Data

• Authors: Devansh Gupta, A.S. Poornash, Andrew Lowy, Meisam Razaviyayn
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07889
• Pdf URL: https://arxiv.org/pdf/2411.07889
• Copy Paste: [[2411.07889]] A Stochastic Optimization Framework for Private and Fair Learning From Decentralized Data(https://arxiv.org/abs/2411.07889)
• Keywords: privacy, federate, fair
• Abstract: Machine learning models are often trained on sensitive data (e.g., medical records and race/gender) that is distributed across different "silos" (e.g., hospitals). These federated learning models may then be used to make consequential decisions, such as allocating healthcare resources. Two key challenges emerge in this setting: (i) maintaining the privacy of each person's data, even if other silos or an adversary with access to the central server tries to infer this data; (ii) ensuring that decisions are fair to different demographic groups (e.g., race/gender). In this paper, we develop a novel algorithm for private and fair federated learning (FL). Our algorithm satisfies inter-silo record-level differential privacy (ISRL-DP), a strong notion of private FL requiring that silo i's sent messages satisfy record-level differential privacy for all i. Our framework can be used to promote different fairness notions, including demographic parity and equalized odds. We prove that our algorithm converges under mild smoothness assumptions on the loss function, whereas prior work required strong convexity for convergence. As a byproduct of our analysis, we obtain the first convergence guarantee for ISRL-DP nonconvex-strongly concave min-max FL. Experiments demonstrate the state-of-the-art fairness-accuracy tradeoffs of our algorithm across different privacy levels.

Title: Joint multi-dimensional dynamic attention and transformer for general image restoration

• Authors: Huan Zhang, Xu Zhang, Nian Cai, Jianglei Di, Yun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07893
• Pdf URL: https://arxiv.org/pdf/2411.07893
• Copy Paste: [[2411.07893]] Joint multi-dimensional dynamic attention and transformer for general image restoration(https://arxiv.org/abs/2411.07893)
• Keywords: extraction, transformer
• Abstract: Outdoor images often suffer from severe degradation due to rain, haze, and noise, impairing image quality and challenging high-level tasks. Current image restoration methods struggle to handle complex degradation while maintaining efficiency. This paper introduces a novel image restoration architecture that combines multi-dimensional dynamic attention and self-attention within a U-Net framework. To leverage the global modeling capabilities of transformers and the local modeling capabilities of convolutions, we integrate sole CNNs in the encoder-decoder and sole transformers in the latent layer. Additionally, we design convolutional kernels with selected multi-dimensional dynamic attention to capture diverse degraded inputs efficiently. A transformer block with transposed self-attention further enhances global feature extraction while maintaining efficiency. Extensive experiments demonstrate that our method achieves a better balance between performance and computational complexity across five image restoration tasks: deraining, deblurring, denoising, dehazing, and enhancement, as well as superior performance for high-level vision tasks. The source code will be available at this https URL.

Title: CryptoLLM: Unleashing the Power of Prompted LLMs for SmartQnA and Classification of Crypto Posts

• Authors: Aniket Deroy, Subhankar Maity
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07917
• Pdf URL: https://arxiv.org/pdf/2411.07917
• Copy Paste: [[2411.07917]] CryptoLLM: Unleashing the Power of Prompted LLMs for SmartQnA and Classification of Crypto Posts(https://arxiv.org/abs/2411.07917)
• Keywords: robust
• Abstract: The rapid growth of social media has resulted in an large volume of user-generated content, particularly in niche domains such as cryptocurrency. This task focuses on developing robust classification models to accurately categorize cryptocurrency-related social media posts into predefined classes, including but not limited to objective, positive, negative, etc. Additionally, the task requires participants to identify the most relevant answers from a set of posts in response to specific questions. By leveraging advanced LLMs, this research aims to enhance the understanding and filtering of cryptocurrency discourse, thereby facilitating more informed decision-making in this volatile sector. We have used a prompt-based technique to solve the classification task for reddit posts and twitter posts. Also, we have used 64-shot technique along with prompts on GPT-4-Turbo model to determine whether a answer is relevant to a question or not.

Title: Isometric Transformations for Image Augmentation in Mueller Matrix Polarimetry

• Authors: Christopher Hahne, Omar Rodriguez-Nunez, Éléa Gros, Théotim Lucas, Ekkehard Hewer, Tatiana Novikova, Theoni Maragkou, Philippe Schucht, Richard McKinley
• Subjects: cs.CV, physics.med-ph
• Abstract URL: https://arxiv.org/abs/2411.07918
• Pdf URL: https://arxiv.org/pdf/2411.07918
• Copy Paste: [[2411.07918]] Isometric Transformations for Image Augmentation in Mueller Matrix Polarimetry(https://arxiv.org/abs/2411.07918)
• Keywords: robust, segmentation
• Abstract: Mueller matrix polarimetry captures essential information about polarized light interactions with a sample, presenting unique challenges for data augmentation in deep learning due to its distinct structure. While augmentations are an effective and affordable way to enhance dataset diversity and reduce overfitting, standard transformations like rotations and flips do not preserve the polarization properties in Mueller matrix images. To this end, we introduce a versatile simulation framework that applies physically consistent rotations and flips to Mueller matrices, tailored to maintain polarization fidelity. Our experimental results across multiple datasets reveal that conventional augmentations can lead to misleading results when applied to polarimetric data, underscoring the necessity of our physics-based approach. In our experiments, we first compare our polarization-specific augmentations against real-world captures to validate their physical consistency. We then apply these augmentations in a semantic segmentation task, achieving substantial improvements in model generalization and performance. This study underscores the necessity of physics-informed data augmentation for polarimetric imaging in deep learning (DL), paving the way for broader adoption and more robust applications across diverse research in the field. In particular, our framework unlocks the potential of DL models for polarimetric datasets with limited sample sizes. Our code implementation is available at this http URL.

Title: Learning Memory Mechanisms for Decision Making through Demonstrations

• Authors: William Yue, Bo Liu, Peter Stone
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.07954
• Pdf URL: https://arxiv.org/pdf/2411.07954
• Copy Paste: [[2411.07954]] Learning Memory Mechanisms for Decision Making through Demonstrations(https://arxiv.org/abs/2411.07954)
• Keywords: transformer
• Abstract: In Partially Observable Markov Decision Processes, integrating an agent's history into memory poses a significant challenge for decision-making. Traditional imitation learning, relying on observation-action pairs for expert demonstrations, fails to capture the expert's memory mechanisms used in decision-making. To capture memory processes as demonstrations, we introduce the concept of \textbf{memory dependency pairs} $(p, q)$ indicating that events at time $p$ are recalled for decision-making at time $q$. We introduce \textbf{AttentionTuner} to leverage memory dependency pairs in Transformers and find significant improvements across several tasks compared to standard Transformers when evaluated on Memory Gym and the Long-term Memory Benchmark. Code is available at this https URL .

Title: On the Convergence of Continual Federated Learning Using Incrementally Aggregated Gradients

• Authors: Satish Kumar Keshri, Nazreen Shah, Ranjitha Prasad
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.07959
• Pdf URL: https://arxiv.org/pdf/2411.07959
• Copy Paste: [[2411.07959]] On the Convergence of Continual Federated Learning Using Incrementally Aggregated Gradients(https://arxiv.org/abs/2411.07959)
• Keywords: privacy, federate
• Abstract: The holy grail of machine learning is to enable Continual Federated Learning (CFL) to enhance the efficiency, privacy, and scalability of AI systems while learning from streaming data. The primary challenge of a CFL system is to overcome global catastrophic forgetting, wherein the accuracy of the global model trained on new tasks declines on the old tasks. In this work, we propose Continual Federated Learning with Aggregated Gradients (C-FLAG), a novel replay-memory based federated strategy consisting of edge-based gradient updates on memory and aggregated gradients on the current data. We provide convergence analysis of the C-FLAG approach which addresses forgetting and bias while converging at a rate of $O(1/\sqrt{T})$ over $T$ communication rounds. We formulate an optimization sub-problem that minimizes catastrophic forgetting, translating CFL into an iterative algorithm with adaptive learning rates that ensure seamless learning across tasks. We empirically show that C-FLAG outperforms several state-of-the-art baselines on both task and class-incremental settings with respect to metrics such as accuracy and forgetting.

Title: From General to Specific: Utilizing General Hallucation to Automatically Measure the Role Relationship Fidelity for Specific Role-Play Agents

• Authors: Chuyi Kong, Ziyang Luo, Hongzhan Lin, Zhiyuan Fan, Yaxin Fan, Yuxi Sun, Jing Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07965
• Pdf URL: https://arxiv.org/pdf/2411.07965
• Copy Paste: [[2411.07965]] From General to Specific: Utilizing General Hallucation to Automatically Measure the Role Relationship Fidelity for Specific Role-Play Agents(https://arxiv.org/abs/2411.07965)
• Keywords: large language model
• Abstract: The advanced role-playing capabilities of Large Language Models (LLMs) have paved the way for developing Role-Playing Agents (RPAs). However, existing benchmarks, such as HPD, which incorporates manually scored character relationships into the context for LLMs to sort coherence, and SocialBench, which uses specific profiles generated by LLMs in the context of multiple-choice tasks to assess character preferences, face limitations like poor generalizability, implicit and inaccurate judgments, and excessive context length. To address the above issues, we propose an automatic, scalable, and generalizable paradigm. Specifically, we construct a benchmark by extracting relations from a general knowledge graph and leverage RPA's inherent hallucination properties to prompt it to interact across roles, employing ChatGPT for stance detection and defining relationship hallucination along with three related metrics. Extensive experiments validate the effectiveness and stability of our metrics. Our findings further explore factors influencing these metrics and discuss the trade-off between relationship hallucination and factuality.

Title: JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

• Authors: Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu, Chong Ruan
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07975
• Pdf URL: https://arxiv.org/pdf/2411.07975
• Copy Paste: [[2411.07975]] JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation(https://arxiv.org/abs/2411.07975)
• Keywords: generative, large language model
• Abstract: We present JanusFlow, a powerful framework that unifies image understanding and generation in a single model. JanusFlow introduces a minimalist architecture that integrates autoregressive language models with rectified flow, a state-of-the-art method in generative modeling. Our key finding demonstrates that rectified flow can be straightforwardly trained within the large language model framework, eliminating the need for complex architectural modifications. To further improve the performance of our unified model, we adopt two key strategies: (i) decoupling the understanding and generation encoders, and (ii) aligning their representations during unified training. Extensive experiments show that JanusFlow achieves comparable or superior performance to specialized models in their respective domains, while significantly outperforming existing unified approaches across standard benchmarks. This work represents a step toward more efficient and versatile vision-language models.

Title: Derivational Morphology Reveals Analogical Generalization in Large Language Models

• Authors: Valentin Hofmann, Leonie Weissweiler, David Mortensen, Hinrich Schütze, Janet Pierrehumbert
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07990
• Pdf URL: https://arxiv.org/pdf/2411.07990
• Copy Paste: [[2411.07990]] Derivational Morphology Reveals Analogical Generalization in Large Language Models(https://arxiv.org/abs/2411.07990)
• Keywords: large language model
• Abstract: What mechanisms underlie linguistic generalization in large language models (LLMs)? This question has attracted considerable attention, with most studies analyzing the extent to which the language skills of LLMs resemble rules. As of yet, it is not known whether linguistic generalization in LLMs could equally well be explained as the result of analogical processes, which can be formalized as similarity operations on stored exemplars. A key shortcoming of prior research is its focus on linguistic phenomena with a high degree of regularity, for which rule-based and analogical approaches make the same predictions. Here, we instead examine derivational morphology, specifically English adjective nominalization, which displays notable variability. We introduce a new method for investigating linguistic generalization in LLMs: focusing on GPT-J, we fit cognitive models that instantiate rule-based and analogical learning to the LLM training data and compare their predictions on a set of nonce adjectives with those of the LLM, allowing us to draw direct conclusions regarding underlying mechanisms. As expected, rule-based and analogical models explain the predictions of GPT-J equally well for adjectives with regular nominalization patterns. However, for adjectives with variable nominalization patterns, the analogical model provides a much better match. Furthermore, GPT-J's behavior is sensitive to the individual word frequencies, even for regular forms, a behavior that is consistent with an analogical account of regular forms but not a rule-based one. These findings refute the hypothesis that GPT-J's linguistic generalization on adjective nominalization involves rules, suggesting similarity operations on stored exemplars as the underlying mechanism. Overall, our study suggests that analogical processes play a bigger role in the linguistic generalization of LLMs than previously thought.

Title: ExpressivityArena: Can LLMs Express Information Implicitly?

• Authors: Joshua Tint, Som Sagar, Aditya Taparia, Kelly Raines, Bimsara Pathiraja, Caleb Liu, Ransalu Senanayake
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08010
• Pdf URL: https://arxiv.org/pdf/2411.08010
• Copy Paste: [[2411.08010]] ExpressivityArena: Can LLMs Express Information Implicitly?(https://arxiv.org/abs/2411.08010)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) have demonstrated remarkable performance in certain dimensions, their ability to express implicit language cues that human use for effective communication remains unclear. This paper presents ExpressivityArena, a Python library for measuring the implicit communication abilities of LLMs. We provide a comprehensive framework to evaluate expressivity of arbitrary LLMs and explore its practical implications. To this end, we refine the definition and measurements of ``expressivity,'' and use our framework in a set of small experiments. These experiments test LLMs in creative and logical tasks such as poetry, coding, and emotion-based responses. They are then evaluated by an automated grader, through ExpressivityArena, which we verify to be the most pragmatic for testing expressivity. Building on these experiments, we deepen our understanding of the expressivity of LLMs by assessing their ability to remain expressive in conversations. Our findings indicate that LLMs are capable of generating and understanding expressive content, however, with some limitations. These insights will inform the future development and deployment of expressive LLMs. We provide the code for ExpressivityArena alongside our paper.

Title: Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings

• Authors: Aditya Sanghi, Aliasghar Khani, Pradyumna Reddy, Arianna Rampini, Derek Cheung, Kamal Rahimi Malekshan, Kanika Madan, Hooman Shayani
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08017
• Pdf URL: https://arxiv.org/pdf/2411.08017
• Copy Paste: [[2411.08017]] Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings(https://arxiv.org/abs/2411.08017)
• Keywords: diffusion, generative
• Abstract: Large-scale 3D generative models require substantial computational resources yet often fall short in capturing fine details and complex geometries at high resolutions. We attribute this limitation to the inefficiency of current representations, which lack the compactness required to model the generative models effectively. To address this, we introduce a novel approach called Wavelet Latent Diffusion, or WaLa, that encodes 3D shapes into wavelet-based, compact latent encodings. Specifically, we compress a $256^3$ signed distance field into a $12^3 \times 4$ latent grid, achieving an impressive 2427x compression ratio with minimal loss of detail. This high level of compression allows our method to efficiently train large-scale generative networks without increasing the inference time. Our models, both conditional and unconditional, contain approximately one billion parameters and successfully generate high-quality 3D shapes at $256^3$ resolution. Moreover, WaLa offers rapid inference, producing shapes within two to four seconds depending on the condition, despite the model's scale. We demonstrate state-of-the-art performance across multiple datasets, with significant improvements in generation quality, diversity, and computational efficiency. We open-source our code and, to the best of our knowledge, release the largest pretrained 3D generative models across different modalities.

Title: Language Models as Causal Effect Generators

• Authors: Lucius E.J. Bynum, Kyunghyun Cho
• Subjects: cs.CL, cs.AI, cs.LG, stat.AP, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.08019
• Pdf URL: https://arxiv.org/pdf/2411.08019
• Copy Paste: [[2411.08019]] Language Models as Causal Effect Generators(https://arxiv.org/abs/2411.08019)
• Keywords: large language model
• Abstract: We present a framework for large language model (LLM) based data generation with controllable causal structure. In particular, we define a procedure for turning any language model and any directed acyclic graph (DAG) into a sequence-driven structural causal model (SD-SCM). Broadly speaking, an SD-SCM is a causal model with user-defined structure and LLM-defined structural equations. We characterize how an SD-SCM allows sampling from observational, interventional, and counterfactual distributions according to the desired causal structure. We then leverage this procedure to propose a new type of benchmark for causal inference methods, generating individual-level counterfactual data without needing to manually specify functional relationships between variables. We create an example benchmark consisting of thousands of datasets, and test a suite of popular estimation methods on these datasets for average, conditional average, and individual treatment effect estimation, both with and without hidden confounding. Apart from generating data, the same procedure also allows us to test for the presence of a causal effect that might be encoded in an LLM. This procedure can underpin auditing LLMs for misinformation, discrimination, or otherwise undesirable behavior. We believe SD-SCMs can serve as a useful tool in any application that would benefit from sequential data with controllable causal structure.

Title: LLMPhy: Complex Physical Reasoning Using Large Language Models and World Models

• Authors: Anoop Cherian, Radu Corcodel, Siddarth Jain, Diego Romeres
• Subjects: cs.LG, cs.AI, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.08027
• Pdf URL: https://arxiv.org/pdf/2411.08027
• Copy Paste: [[2411.08027]] LLMPhy: Complex Physical Reasoning Using Large Language Models and World Models(https://arxiv.org/abs/2411.08027)
• Keywords: large language model
• Abstract: Physical reasoning is an important skill needed for robotic agents when operating in the real world. However, solving such reasoning problems often involves hypothesizing and reflecting over complex multi-body interactions under the effect of a multitude of physical forces and thus learning all such interactions poses a significant hurdle for state-of-the-art machine learning frameworks, including large language models (LLMs). To study this problem, we propose a new physical reasoning task and a dataset, dubbed TraySim. Our task involves predicting the dynamics of several objects on a tray that is given an external impact -- the domino effect of the ensued object interactions and their dynamics thus offering a challenging yet controlled setup, with the goal of reasoning being to infer the stability of the objects after the impact. To solve this complex physical reasoning task, we present LLMPhy, a zero-shot black-box optimization framework that leverages the physics knowledge and program synthesis abilities of LLMs, and synergizes these abilities with the world models built into modern physics engines. Specifically, LLMPhy uses an LLM to generate code to iteratively estimate the physical hyperparameters of the system (friction, damping, layout, etc.) via an implicit analysis-by-synthesis approach using a (non-differentiable) simulator in the loop and uses the inferred parameters to imagine the dynamics of the scene towards solving the reasoning task. To show the effectiveness of LLMPhy, we present experiments on our TraySim dataset to predict the steady-state poses of the objects. Our results show that the combination of the LLM and the physics engine leads to state-of-the-art zero-shot physical reasoning performance, while demonstrating superior convergence against standard black-box optimization methods and better estimation of the physical parameters.

Title: GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation

• Authors: Yushi Lan, Shangchen Zhou, Zhaoyang Lyu, Fangzhou Hong, Shuai Yang, Bo Dai, Xingang Pan, Chen Change Loy
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.08033
• Pdf URL: https://arxiv.org/pdf/2411.08033
• Copy Paste: [[2411.08033]] GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation(https://arxiv.org/abs/2411.08033)
• Keywords: diffusion
• Abstract: While 3D content generation has advanced significantly, existing methods still face challenges with input formats, latent space design, and output representations. This paper introduces a novel 3D generation framework that addresses these challenges, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs a Variational Autoencoder (VAE) with multi-view posed RGB-D(epth)-N(ormal) renderings as input, using a unique latent space design that preserves 3D shape information, and incorporates a cascaded latent diffusion model for improved shape-texture disentanglement. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single/multi-view image inputs. Notably, the newly proposed latent space naturally enables geometry-texture disentanglement, thus allowing 3D-aware editing. Experimental results demonstrate the effectiveness of our approach on multiple datasets, outperforming existing methods in both text- and image-conditioned 3D generation.

Title: Scaling Properties of Diffusion Models for Perceptual Tasks

• Authors: Rahul Ravishankar, Zeeshan Patel, Jathushan Rajasegaran, Jitendra Malik
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08034
• Pdf URL: https://arxiv.org/pdf/2411.08034
• Copy Paste: [[2411.08034]] Scaling Properties of Diffusion Models for Perceptual Tasks(https://arxiv.org/abs/2411.08034)
• Keywords: diffusion, segmentation
• Abstract: In this paper, we argue that iterative computation with diffusion models offers a powerful paradigm for not only generation but also visual perception tasks. We unify tasks such as depth estimation, optical flow, and segmentation under image-to-image translation, and show how diffusion models benefit from scaling training and test-time compute for these perception tasks. Through a careful analysis of these scaling behaviors, we present various techniques to efficiently train diffusion models for visual perception tasks. Our models achieve improved or comparable performance to state-of-the-art methods using significantly less data and compute. To use our code and models, see this https URL .