2024-10-21

Title: Explaining an image classifier with a generative model conditioned by uncertainty

• Authors: Adrien Le Coz, Stéphane Herbin, Faouzi Adjed
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.13871
• Pdf URL: https://arxiv.org/pdf/2410.13871
• Copy Paste: [[2410.13871]] Explaining an image classifier with a generative model conditioned by uncertainty(https://arxiv.org/abs/2410.13871)
• Keywords: generative
• Abstract: We propose to condition a generative model by a given image classifier uncertainty in order to analyze and explain its behavior. Preliminary experiments on synthetic data and a corrupted version of MNIST dataset illustrate the idea.

Title: Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model

• Authors: Long Le, Jason Xie, William Liang, Hung-Ju Wang, Yue Yang, Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, Eric Eaton
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.13882
• Pdf URL: https://arxiv.org/pdf/2410.13882
• Copy Paste: [[2410.13882]] Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model(https://arxiv.org/abs/2410.13882)
• Keywords: foundation model
• Abstract: Interactive 3D simulated objects are crucial in AR/VR, animations, and robotics, driving immersive experiences and advanced automation. However, creating these articulated objects requires extensive human effort and expertise, limiting their broader applications. To overcome this challenge, we present Articulate-Anything, a system that automates the articulation of diverse, complex objects from many input modalities, including text, images, and videos. Articulate-Anything leverages vision-language models (VLMs) to generate code that can be compiled into an interactable digital twin for use in standard 3D simulators. Our system exploits existing 3D asset datasets via a mesh retrieval mechanism, along with an actor-critic system that iteratively proposes, evaluates, and refines solutions for articulating the objects, self-correcting errors to achieve a robust outcome. Qualitative evaluations demonstrate Articulate-Anything's capability to articulate complex and even ambiguous object affordances by leveraging rich grounded inputs. In extensive quantitative experiments on the standard PartNet-Mobility dataset, Articulate-Anything substantially outperforms prior work, increasing the success rate from 8.7-11.6% to 75% and setting a new bar for state-of-the-art performance. We further showcase the utility of our generated assets by using them to train robotic policies for fine-grained manipulation tasks that go beyond basic pick and place.

Title: Can LLMs be Scammed? A Baseline Measurement Study

• Authors: Udari Madhushani Sehwag, Kelly Patel, Francesca Mosca, Vineeth Ravi, Jessica Staddon
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.13893
• Pdf URL: https://arxiv.org/pdf/2410.13893
• Copy Paste: [[2410.13893]] Can LLMs be Scammed? A Baseline Measurement Study(https://arxiv.org/abs/2410.13893)
• Keywords: generative
• Abstract: Despite the importance of developing generative AI models that can effectively resist scams, current literature lacks a structured framework for evaluating their vulnerability to such threats. In this work, we address this gap by constructing a benchmark based on the FINRA taxonomy and systematically assessing Large Language Models' (LLMs') vulnerability to a variety of scam tactics. First, we incorporate 37 well-defined base scam scenarios reflecting the diverse scam categories identified by FINRA taxonomy, providing a focused evaluation of LLMs' scam detection capabilities. Second, we utilize representative proprietary (GPT-3.5, GPT-4) and open-source (Llama) models to analyze their performance in scam detection. Third, our research provides critical insights into which scam tactics are most effective against LLMs and how varying persona traits and persuasive techniques influence these vulnerabilities. We reveal distinct susceptibility patterns across different models and scenarios, underscoring the need for targeted enhancements in LLM design and deployment.

Title: A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation

• Authors: Aviral Srivastava, Sourav Panda
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.13897
• Pdf URL: https://arxiv.org/pdf/2410.13897
• Copy Paste: [[2410.13897]] A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation(https://arxiv.org/abs/2410.13897)
• Keywords: diffusion, generative, anomaly
• Abstract: As generative AI systems, including large language models (LLMs) and diffusion models, advance rapidly, their growing adoption has led to new and complex security risks often overlooked in traditional AI risk assessment frameworks. This paper introduces a novel formal framework for categorizing and mitigating these emergent security risks by integrating adaptive, real-time monitoring, and dynamic risk mitigation strategies tailored to generative models' unique vulnerabilities. We identify previously under-explored risks, including latent space exploitation, multi-modal cross-attack vectors, and feedback-loop-induced model degradation. Our framework employs a layered approach, incorporating anomaly detection, continuous red-teaming, and real-time adversarial simulation to mitigate these risks. We focus on formal verification methods to ensure model robustness and scalability in the face of evolving threats. Though theoretical, this work sets the stage for future empirical validation by establishing a detailed methodology and metrics for evaluating the performance of risk mitigation strategies in generative AI systems. This framework addresses existing gaps in AI safety, offering a comprehensive road map for future research and implementation.

Title: Security of and by Generative AI platforms

• Authors: Hari Hayagreevan, Souvik Khamaru
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.13899
• Pdf URL: https://arxiv.org/pdf/2410.13899
• Copy Paste: [[2410.13899]] Security of and by Generative AI platforms(https://arxiv.org/abs/2410.13899)
• Keywords: generative
• Abstract: This whitepaper highlights the dual importance of securing generative AI (genAI) platforms and leveraging genAI for cybersecurity. As genAI technologies proliferate, their misuse poses significant risks, including data breaches, model tampering, and malicious content generation. Securing these platforms is critical to protect sensitive data, ensure model integrity, and prevent adversarial attacks. Simultaneously, genAI presents opportunities for enhancing security by automating threat detection, vulnerability analysis, and incident response. The whitepaper explores strategies for robust security frameworks around genAI systems, while also showcasing how genAI can empower organizations to anticipate, detect, and mitigate sophisticated cyber threats.

Title: GraspDiffusion: Synthesizing Realistic Whole-body Hand-Object Interaction

• Authors: Patrick Kwon, Hanbyul Joo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.13911
• Pdf URL: https://arxiv.org/pdf/2410.13911
• Copy Paste: [[2410.13911]] GraspDiffusion: Synthesizing Realistic Whole-body Hand-Object Interaction(https://arxiv.org/abs/2410.13911)
• Keywords: diffusion, generative
• Abstract: Recent generative models can synthesize high-quality images but often fail to generate humans interacting with objects using their hands. This arises mostly from the model's misunderstanding of such interactions, and the hardships of synthesizing intricate regions of the body. In this paper, we propose GraspDiffusion, a novel generative method that creates realistic scenes of human-object interaction. Given a 3D object mesh, GraspDiffusion first constructs life-like whole-body poses with control over the object's location relative to the human body. This is achieved by separately leveraging the generative priors for 3D body and hand poses, optimizing them into a joint grasping pose. The resulting pose guides the image synthesis to correctly reflect the intended interaction, allowing the creation of realistic and diverse human-object interaction scenes. We demonstrate that GraspDiffusion can successfully tackle the relatively uninvestigated problem of generating full-bodied human-object interactions while outperforming previous methods. Code and models will be available at this https URL

Title: FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model

• Authors: ZiDong Wang, Zeyu Lu, Di Huang, Cai Zhou, Wanli Ouyang, and Lei Bai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.13925
• Pdf URL: https://arxiv.org/pdf/2410.13925
• Copy Paste: [[2410.13925]] FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model(https://arxiv.org/abs/2410.13925)
• Keywords: diffusion
• Abstract: \textit{Nature is infinitely resolution-free}. In the context of this reality, existing diffusion models, such as Diffusion Transformers, often face challenges when processing image resolutions outside of their trained domain. To address this limitation, we conceptualize images as sequences of tokens with dynamic sizes, rather than traditional methods that perceive images as fixed-resolution grids. This perspective enables a flexible training strategy that seamlessly accommodates various aspect ratios during both training and inference, thus promoting resolution generalization and eliminating biases introduced by image cropping. On this basis, we present the \textbf{Flexible Vision Transformer} (FiT), a transformer architecture specifically designed for generating images with \textit{unrestricted resolutions and aspect ratios}. We further upgrade the FiT to FiTv2 with several innovative designs, includingthe Query-Key vector normalization, the AdaLN-LoRA module, a rectified flow scheduler, and a Logit-Normal sampler. Enhanced by a meticulously adjusted network structure, FiTv2 exhibits $2\times$ convergence speed of FiT. When incorporating advanced training-free extrapolation techniques, FiTv2 demonstrates remarkable adaptability in both resolution extrapolation and diverse resolution generation. Additionally, our exploration of the scalability of the FiTv2 model reveals that larger models exhibit better computational efficiency. Furthermore, we introduce an efficient post-training strategy to adapt a pre-trained model for the high-resolution generation. Comprehensive experiments demonstrate the exceptional performance of FiTv2 across a broad range of resolutions. We have released all the codes and models at \url{this https URL} to promote the exploration of diffusion transformer models for arbitrary-resolution image generation.

Title: Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation

• Authors: Junhong Wu, Yang Zhao, Yangyifan Xu, Bing Liu, Chengqing Zong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.13944
• Pdf URL: https://arxiv.org/pdf/2410.13944
• Copy Paste: [[2410.13944]] Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation(https://arxiv.org/abs/2410.13944)
• Keywords: generative
• Abstract: Large Language Models (LLMs) have achieved impressive results across numerous NLP tasks but still encounter difficulties in machine translation. Traditional methods to improve translation have typically involved fine-tuning LLMs using parallel corpora. However, vanilla fine-tuning often leads to catastrophic forgetting of the instruction-following capabilities and alignment with human preferences, compromising their broad general abilities and introducing potential security risks. These abilities, which are developed using proprietary and unavailable training data, make existing continual instruction tuning methods ineffective. To overcome this issue, we propose a novel approach called RaDis (Rationale Distillation). RaDis harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then "replayed" to prevent forgetting. These rationales encapsulate general knowledge and safety principles, acting as self-distillation targets to regulate the training process. By jointly training on both reference translations and self-generated rationales, the model can learn new translation skills while preserving its overall general abilities. Extensive experiments demonstrate that our method enhances machine translation performance while maintaining the broader capabilities of LLMs across other tasks. This work presents a pathway for creating more versatile LLMs that excel in specialized tasks without compromising generality and safety.

Title: On Diffusion Models for Multi-Agent Partial Observability: Shared Attractors, Error Bounds, and Composite Flow

• Authors: Tonghan Wang, Heng Dong, Yanchen Jiang, David C. Parkes, Milind Tambe
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.13953
• Pdf URL: https://arxiv.org/pdf/2410.13953
• Copy Paste: [[2410.13953]] On Diffusion Models for Multi-Agent Partial Observability: Shared Attractors, Error Bounds, and Composite Flow(https://arxiv.org/abs/2410.13953)
• Keywords: diffusion
• Abstract: Multiagent systems grapple with partial observability (PO), and the decentralized POMDP (Dec-POMDP) model highlights the fundamental nature of this challenge. Whereas recent approaches to address PO have appealed to deep learning models, providing a rigorous understanding of how these models and their approximation errors affect agents' handling of PO and their interactions remain a challenge. In addressing this challenge, we investigate reconstructing global states from local action-observation histories in Dec-POMDPs using diffusion models. We first find that diffusion models conditioned on local history represent possible states as stable fixed points. In collectively observable (CO) Dec-POMDPs, individual diffusion models conditioned on agents' local histories share a unique fixed point corresponding to the global state, while in non-CO settings, the shared fixed points yield a distribution of possible states given joint history. We further find that, with deep learning approximation errors, fixed points can deviate from true states and the deviation is negatively correlated to the Jacobian rank. Inspired by this low-rank property, we bound the deviation by constructing a surrogate linear regression model that approximates the local behavior of diffusion models. With this bound, we propose a composite diffusion process iterating over agents with theoretical convergence guarantees to the true state.

Title: Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all

• Authors: Ihab Bendidi, Shawn Whitfield, Kian Kenyon-Dean, Hanene Ben Yedder, Yassir El Mesbahi, Emmanuel Noutahi, Alisandra K. Denton
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.13956
• Pdf URL: https://arxiv.org/pdf/2410.13956
• Copy Paste: [[2410.13956]] Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all(https://arxiv.org/abs/2410.13956)
• Keywords: foundation model
• Abstract: Understanding the relationships among genes, compounds, and their interactions in living organisms remains limited due to technological constraints and the complexity of biological data. Deep learning has shown promise in exploring these relationships using various data types. However, transcriptomics, which provides detailed insights into cellular states, is still underused due to its high noise levels and limited data availability. Recent advancements in transcriptomics sequencing provide new opportunities to uncover valuable insights, especially with the rise of many new foundation models for transcriptomics, yet no benchmark has been made to robustly evaluate the effectiveness of these rising models for perturbation analysis. This article presents a novel biologically motivated evaluation framework and a hierarchy of perturbation analysis tasks for comparing the performance of pretrained foundation models to each other and to more classical techniques of learning from transcriptomics data. We compile diverse public datasets from different sequencing techniques and cell lines to assess models performance. Our approach identifies scVI and PCA to be far better suited models for understanding biological perturbations in comparison to existing foundation models, especially in their application in real-world scenarios.

Title: On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery

• Authors: Renpu Liu, Ruida Zhou, Cong Shen, Jing Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.13981
• Pdf URL: https://arxiv.org/pdf/2410.13981
• Copy Paste: [[2410.13981]] On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery(https://arxiv.org/abs/2410.13981)
• Keywords: in-context
• Abstract: An intriguing property of the Transformer is its ability to perform in-context learning (ICL), where the Transformer can solve different inference tasks without parameter updating based on the contextual information provided by the corresponding input-output demonstration pairs. It has been theoretically proved that ICL is enabled by the capability of Transformers to perform gradient-descent algorithms (Von Oswald et al., 2023a; Bai et al., 2024). This work takes a step further and shows that Transformers can perform learning-to-optimize (L2O) algorithms. Specifically, for the ICL sparse recovery (formulated as LASSO) tasks, we show that a K-layer Transformer can perform an L2O algorithm with a provable convergence rate linear in K. This provides a new perspective explaining the superior ICL capability of Transformers, even with only a few layers, which cannot be achieved by the standard gradient-descent algorithms. Moreover, unlike the conventional L2O algorithms that require the measurement matrix involved in training to match that in testing, the trained Transformer is able to solve sparse recovery problems generated with different measurement matrices. Besides, Transformers as an L2O algorithm can leverage structural information embedded in the training tasks to accelerate its convergence during ICL, and generalize across different lengths of demonstration pairs, where conventional L2O algorithms typically struggle or fail. Such theoretical findings are supported by our experimental results.

Title: Personalized Adaptation via In-Context Preference Learning

• Authors: Allison Lau, Younwoo Choi, Vahid Balazadeh, Keertana Chidambaram, Vasilis Syrgkanis, Rahul G. Krishnan
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.14001
• Pdf URL: https://arxiv.org/pdf/2410.14001
• Copy Paste: [[2410.14001]] Personalized Adaptation via In-Context Preference Learning(https://arxiv.org/abs/2410.14001)
• Keywords: in-context
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is widely used to align Language Models (LMs) with human preferences. However, existing approaches often neglect individual user preferences, leading to suboptimal personalization. We present the Preference Pretrained Transformer (PPT), a novel approach for adaptive personalization using online user feedback. PPT leverages the in-context learning capabilities of transformers to dynamically adapt to individual preferences. Our approach consists of two phases: (1) an offline phase where we train a single policy model using a history-dependent loss function, and (2) an online phase where the model adapts to user preferences through in-context learning. We demonstrate PPT's effectiveness in a contextual bandit setting, showing that it achieves personalized adaptation superior to existing methods while significantly reducing the computational costs. Our results suggest the potential of in-context learning for scalable and efficient personalization in large language models.

Title: Latent Weight Diffusion: Generating Policies from Trajectories

• Authors: Shashank Hegde, Gautam Salhotra, Gaurav S. Sukhatme
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.14040
• Pdf URL: https://arxiv.org/pdf/2410.14040
• Copy Paste: [[2410.14040]] Latent Weight Diffusion: Generating Policies from Trajectories(https://arxiv.org/abs/2410.14040)
• Keywords: diffusion
• Abstract: With the increasing availability of open-source robotic data, imitation learning has emerged as a viable approach for both robot manipulation and locomotion. Currently, large generalized policies are trained to predict controls or trajectories using diffusion models, which have the desirable property of learning multimodal action distributions. However, generalizability comes with a cost - namely, larger model size and slower inference. Further, there is a known trade-off between performance and action horizon for Diffusion Policy (i.e., diffusing trajectories): fewer diffusion queries accumulate greater trajectory tracking errors. Thus, it is common practice to run these models at high inference frequency, subject to robot computational constraints. To address these limitations, we propose Latent Weight Diffusion (LWD), a method that uses diffusion to learn a distribution over policies for robotic tasks, rather than over trajectories. Our approach encodes demonstration trajectories into a latent space and then decodes them into policies using a hypernetwork. We employ a diffusion denoising model within this latent space to learn its distribution. We demonstrate that LWD can reconstruct the behaviors of the original policies that generated the trajectory dataset. LWD offers the benefits of considerably smaller policy networks during inference and requires fewer diffusion model queries. When tested on the Metaworld MT10 benchmark, LWD achieves a higher success rate compared to a vanilla multi-task policy, while using models up to ~18x smaller during inference. Additionally, since LWD generates closed-loop policies, we show that it outperforms Diffusion Policy in long action horizon settings, with reduced diffusion queries during rollout.

Title: Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles

• Authors: Xiao Pu, Tianxing He, Xiaojun Wan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.14042
• Pdf URL: https://arxiv.org/pdf/2410.14042
• Copy Paste: [[2410.14042]] Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles(https://arxiv.org/abs/2410.14042)
• Keywords: in-context
• Abstract: Prompt compression condenses contexts while maintaining their informativeness for different usage scenarios. It not only shortens the inference time and reduces computational costs during the usage of large language models, but also lowers expenses when using closed-source models. In a preliminary study, we discover that when instructing language models to compress prompts, different compression styles (e.g., extractive or abstractive) impact performance of compressed prompts on downstream tasks. Building on this insight, we propose Style-Compress, a lightweight framework that adapts a smaller language model to compress prompts for a larger model on a new task without additional training. Our approach iteratively generates and selects effective compressed prompts as task-specific demonstrations through style variation and in-context learning, enabling smaller models to act as efficient compressors with task-specific examples. Style-Compress outperforms two baseline compression models in four tasks: original prompt reconstruction, text summarization, multi-hop QA, and CoT reasoning. In addition, with only 10 samples and 100 queries for adaptation, prompts compressed by Style-Compress achieve performance on par with or better than original prompts at a compression ratio of 0.25 or 0.5.

Title: Learning Metadata-Agnostic Representations for Text-to-SQL In-Context Example Selection

• Authors: Chuhong Mai, Ro-ee Tal, Thahir Mohamed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.14049
• Pdf URL: https://arxiv.org/pdf/2410.14049
• Copy Paste: [[2410.14049]] Learning Metadata-Agnostic Representations for Text-to-SQL In-Context Example Selection(https://arxiv.org/abs/2410.14049)
• Keywords: in-context
• Abstract: In-context learning (ICL) is a powerful paradigm where large language models (LLMs) benefit from task demonstrations added to the prompt. Yet, selecting optimal demonstrations is not trivial, especially for complex or multi-modal tasks where input and output distributions differ. We hypothesize that forming task-specific representations of the input is key. In this paper, we propose a method to align representations of natural language questions and those of SQL queries in a shared embedding space. Our technique, dubbed MARLO - Metadata-Agnostic Representation Learning for Text-tO-SQL - uses query structure to model querying intent without over-indexing on underlying database metadata (i.e. tables, columns, or domain-specific entities of a database referenced in the question or query). This allows MARLO to select examples that are structurally and semantically relevant for the task rather than examples that are spuriously related to a certain domain or question phrasing. When used to retrieve examples based on question similarity, MARLO shows superior performance compared to generic embedding models (on average +2.9\%pt. in execution accuracy) on the Spider benchmark. It also outperforms the next best method that masks metadata information by +0.8\%pt. in execution accuracy on average, while imposing a significantly lower inference latency.

Title: On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods

• Authors: Hariprasath Govindarajan, Per Sidén, Jacob Roll, Fredrik Lindsten
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14060
• Pdf URL: https://arxiv.org/pdf/2410.14060
• Copy Paste: [[2410.14060]] On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods(https://arxiv.org/abs/2410.14060)
• Keywords: self-supervised
• Abstract: A prominent self-supervised learning paradigm is to model the representations as clusters, or more generally as a mixture model. Learning to map the data samples to compact representations and fitting the mixture model simultaneously leads to the representation collapse problem. Regularizing the distribution of data points over the clusters is the prevalent strategy to avoid this issue. While this is sufficient to prevent full representation collapse, we show that a partial prototype collapse problem still exists in the DINO family of methods, that leads to significant redundancies in the prototypes. Such prototype redundancies serve as shortcuts for the method to achieve a marginal latent class distribution that matches the prescribed prior. We show that by encouraging the model to use diverse prototypes, the partial prototype collapse can be mitigated. Effective utilization of the prototypes enables the methods to learn more fine-grained clusters, encouraging more informative representations. We demonstrate that this is especially beneficial when pre-training on a long-tailed fine-grained dataset.

Title: SAMReg: SAM-enabled Image Registration with ROI-based Correspondence

• Authors: Shiqi Huang, Tingfa Xu, Ziyi Shen, Shaheer Ullah Saeed, Wen Yan, Dean Barratt, Yipeng Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14083
• Pdf URL: https://arxiv.org/pdf/2410.14083
• Copy Paste: [[2410.14083]] SAMReg: SAM-enabled Image Registration with ROI-based Correspondence(https://arxiv.org/abs/2410.14083)
• Keywords: foundation model
• Abstract: This paper describes a new spatial correspondence representation based on paired regions-of-interest (ROIs), for medical image registration. The distinct properties of the proposed ROI-based correspondence are discussed, in the context of potential benefits in clinical applications following image registration, compared with alternative correspondence-representing approaches, such as those based on sampled displacements and spatial transformation functions. These benefits include a clear connection between learning-based image registration and segmentation, which in turn motivates two cases of image registration approaches using (pre-)trained segmentation networks. Based on the segment anything model (SAM), a vision foundation model for segmentation, we develop a new registration algorithm SAMReg, which does not require any training (or training data), gradient-based fine-tuning or prompt engineering. The proposed SAMReg models are evaluated across five real-world applications, including intra-subject registration tasks with cardiac MR and lung CT, challenging inter-subject registration scenarios with prostate MR and retinal imaging, and an additional evaluation with a non-clinical example with aerial image registration. The proposed methods outperform both intensity-based iterative algorithms and DDF-predicting learning-based networks across tested metrics including Dice and target registration errors on anatomical structures, and further demonstrates competitive performance compared to weakly-supervised registration approaches that rely on fully-segmented training data. Open source code and examples are available at: this https URL.

Title: In-context learning and Occam's razor

• Authors: Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, Guillaume Lajoie
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14086
• Pdf URL: https://arxiv.org/pdf/2410.14086
• Copy Paste: [[2410.14086]] In-context learning and Occam's razor(https://arxiv.org/abs/2410.14086)
• Keywords: in-context
• Abstract: The goal of machine learning is generalization. While the No Free Lunch Theorem states that we cannot obtain theoretical guarantees for generalization without further assumptions, in practice we observe that simple models which explain the training data generalize best: a principle called Occam's razor. Despite the need for simple models, most current approaches in machine learning only minimize the training error, and at best indirectly promote simplicity through regularization or architecture design. Here, we draw a connection between Occam's razor and in-context learning: an emergent ability of certain sequence models like Transformers to learn at inference time from past observations in a sequence. In particular, we show that the next-token prediction loss used to train in-context learners is directly equivalent to a data compression technique called prequential coding, and that minimizing this loss amounts to jointly minimizing both the training error and the complexity of the model that was implicitly learned from context. Our theory and the empirical experiments we use to support it not only provide a normative account of in-context learning, but also elucidate the shortcomings of current in-context learning methods, suggesting ways in which they can be improved. We make our code available at this https URL.

Title: MMAD-Purify: A Precision-Optimized Framework for Efficient and Scalable Multi-Modal Attacks

• Authors: Xinxin Liu, Zhongliang Guo, Siyuan Huang, Chun Pong Lau
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14089
• Pdf URL: https://arxiv.org/pdf/2410.14089
• Copy Paste: [[2410.14089]] MMAD-Purify: A Precision-Optimized Framework for Efficient and Scalable Multi-Modal Attacks(https://arxiv.org/abs/2410.14089)
• Keywords: diffusion, generative
• Abstract: Neural networks have achieved remarkable performance across a wide range of tasks, yet they remain susceptible to adversarial perturbations, which pose significant risks in safety-critical applications. With the rise of multimodality, diffusion models have emerged as powerful tools not only for generative tasks but also for various applications such as image editing, inpainting, and super-resolution. However, these models still lack robustness due to limited research on attacking them to enhance their resilience. Traditional attack techniques, such as gradient-based adversarial attacks and diffusion model-based methods, are hindered by computational inefficiencies and scalability issues due to their iterative nature. To address these challenges, we introduce an innovative framework that leverages the distilled backbone of diffusion models and incorporates a precision-optimized noise predictor to enhance the effectiveness of our attack framework. This approach not only enhances the attack's potency but also significantly reduces computational costs. Our framework provides a cutting-edge solution for multi-modal adversarial attacks, ensuring reduced latency and the generation of high-fidelity adversarial examples with superior success rates. Furthermore, we demonstrate that our framework achieves outstanding transferability and robustness against purification defenses, outperforming existing gradient-based attack models in both effectiveness and efficiency.

Title: Extreme Precipitation Nowcasting using Multi-Task Latent Diffusion Models

• Authors: Li Chaorong, Ling Xudong, Yang Qiang, Qin Fengqing, Huang Yuanyuan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14103
• Pdf URL: https://arxiv.org/pdf/2410.14103
• Copy Paste: [[2410.14103]] Extreme Precipitation Nowcasting using Multi-Task Latent Diffusion Models(https://arxiv.org/abs/2410.14103)
• Keywords: diffusion
• Abstract: Deep learning models have made remarkable strides in precipitation prediction, yet they continue to struggle with capturing the spatial details of the features of radar images, particularly over high precipitation intensity areas. This shortcoming is evident in the form of low forecast accuracy in the spatial positioning of radar echo images across varying precipitation intensity regions. To address this challenge, we introduce the multi-task latent diffusion model(MTLDM), a novel approach for precipitation prediction. The basic concept of the MTLDM is based on the understanding that the radar image representing precipitation is the result of multiple factors. Therefore, we adopt a divide-and-conquer approach, that is, we decompose the radar image using decomposition technology and then predict the decomposed sub-images separately. We conceptualize the precipitation image as a composition of various components corresponding to different precipitation intensities. The MTLDM decomposes the precipitation image into these distinct components and employs a dedicated task to predict each one. This method enables spatiotemporally consistent prediction of real-world precipitation areas up to 5-80 min in advance, outperforming existing state-of-the-art techniques across multiple evaluation metrics.

Title: Improving Graph Neural Networks by Learning Continuous Edge Directions

• Authors: Seong Ho Pahng, Sahand Hormoz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14109
• Pdf URL: https://arxiv.org/pdf/2410.14109
• Copy Paste: [[2410.14109]] Improving Graph Neural Networks by Learning Continuous Edge Directions(https://arxiv.org/abs/2410.14109)
• Keywords: diffusion
• Abstract: Graph Neural Networks (GNNs) traditionally employ a message-passing mechanism that resembles diffusion over undirected graphs, which often leads to homogenization of node features and reduced discriminative power in tasks such as node classification. Our key insight for addressing this limitation is to assign fuzzy edge directions -- that can vary continuously from node $i$ pointing to node $j$ to vice versa -- to the edges of a graph so that features can preferentially flow in one direction between nodes to enable long-range information transmission across the graph. We also introduce a novel complex-valued Laplacian for directed graphs with fuzzy edges where the real and imaginary parts represent information flow in opposite directions. Using this Laplacian, we propose a general framework, called Continuous Edge Direction (CoED) GNN, for learning on graphs with fuzzy edges and prove its expressivity limits using a generalization of the Weisfeiler-Leman (WL) graph isomorphism test for directed graphs with fuzzy edges. Our architecture aggregates neighbor features scaled by the learned edge directions and processes the aggregated messages from in-neighbors and out-neighbors separately alongside the self-features of the nodes. Since continuous edge directions are differentiable, they can be learned jointly with the GNN weights via gradient-based optimization. CoED GNN is particularly well-suited for graph ensemble data where the graph structure remains fixed but multiple realizations of node features are available, such as in gene regulatory networks, web connectivity graphs, and power grids. We demonstrate through extensive experiments on both synthetic and real datasets that learning continuous edge directions significantly improves performance both for undirected and directed graphs compared with existing methods.

Title: Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning

• Authors: Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, Lingpeng Kong
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14157
• Pdf URL: https://arxiv.org/pdf/2410.14157
• Copy Paste: [[2410.14157]] Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning(https://arxiv.org/abs/2410.14157)
• Keywords: diffusion
• Abstract: Autoregressive language models, despite their impressive capabilities, struggle with complex reasoning and long-term planning tasks. We introduce discrete diffusion models as a novel solution to these challenges. Through the lens of subgoal imbalance, we demonstrate how diffusion models effectively learn difficult subgoals that elude autoregressive approaches. We propose Multi-granularity Diffusion Modeling (MDM), which prioritizes subgoals based on difficulty during learning. On complex tasks like Countdown, Sudoku, and Boolean Satisfiability Problems, MDM significantly outperforms autoregressive models without using search techniques. For instance, MDM achieves 91.5\% and 100\% accuracy on Countdown and Sudoku, respectively, compared to 45.8\% and 20.7\% for autoregressive models. Our work highlights the potential of diffusion-based approaches in advancing AI capabilities for sophisticated language understanding and problem-solving tasks.

Title: Assessing Open-world Forgetting in Generative Image Model Customization

• Authors: Héctor Laria, Alex Gomez-Villa, Imad Eddine Marouf, Kai Wang, Bogdan Raducanu, Joost van de Weijer
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14159
• Pdf URL: https://arxiv.org/pdf/2410.14159
• Copy Paste: [[2410.14159]] Assessing Open-world Forgetting in Generative Image Model Customization(https://arxiv.org/abs/2410.14159)
• Keywords: diffusion, generative
• Abstract: Recent advances in diffusion models have significantly enhanced image generation capabilities. However, customizing these models with new classes often leads to unintended consequences that compromise their reliability. We introduce the concept of open-world forgetting to emphasize the vast scope of these unintended alterations, contrasting it with the well-studied closed-world forgetting, which is measurable by evaluating performance on a limited set of classes or skills. Our research presents the first comprehensive investigation into open-world forgetting in diffusion models, focusing on semantic and appearance drift of representations. We utilize zero-shot classification to analyze semantic drift, revealing that even minor model adaptations lead to unpredictable shifts affecting areas far beyond newly introduced concepts, with dramatic drops in zero-shot classification of up to 60%. Additionally, we observe significant changes in texture and color of generated content when analyzing appearance drift. To address these issues, we propose a mitigation strategy based on functional regularization, designed to preserve original capabilities while accommodating new concepts. Our study aims to raise awareness of unintended changes due to model customization and advocates for the analysis of open-world forgetting in future research on model customization and finetuning methods. Furthermore, we provide insights for developing more robust adaptation methodologies.

Title: LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems

• Authors: Nan Xu, Xuezhe Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14166
• Pdf URL: https://arxiv.org/pdf/2410.14166
• Copy Paste: [[2410.14166]] LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems(https://arxiv.org/abs/2410.14166)
• Keywords: in-context
• Abstract: Interestingly, LLMs yet struggle with some basic tasks that humans find trivial to handle, e.g., counting the number of character r's in the word "strawberry". There are several popular conjectures (e.g., tokenization, architecture and training data) regarding the reason for deficiency of LLMs in simple word-based counting problems, sharing the similar belief that such failure stems from model pretraining hence probably inevitable during deployment. In this paper, we carefully design multiple evaluation settings to investigate validity of prevalent conjectures. Meanwhile, we measure transferability of advanced mathematical and coding reasoning capabilities from specialized LLMs to simple counting tasks. Although specialized LLMs suffer from counting problems as well, we find conjectures about inherent deficiency of LLMs invalid and further seek opportunities to elicit knowledge and capabilities from LLMs that are beneficial to counting tasks. Compared with strategies such as finetuning and in-context learning that are commonly adopted to enhance performance on new or challenging tasks, we show that engaging reasoning is the most robust and efficient way to help LLMs better perceive tasks with more accurate responses. We hope our conjecture validation design could provide insights into the study of future critical failure modes of LLMs. Based on challenges in transferring advanced capabilities to much simpler tasks, we call for more attention to model capability acquisition and evaluation. We also highlight the importance of cultivating consciousness of "reasoning before responding" during model pretraining.

Title: Heavy-Tailed Diffusion Models

• Authors: Kushagra Pandey, Jaideep Pathak, Yilun Xu, Stephan Mandt, Michael Pritchard, Arash Vahdat, Morteza Mardani
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.14171
• Pdf URL: https://arxiv.org/pdf/2410.14171
• Copy Paste: [[2410.14171]] Heavy-Tailed Diffusion Models(https://arxiv.org/abs/2410.14171)
• Keywords: diffusion
• Abstract: Diffusion models achieve state-of-the-art generation quality across many applications, but their ability to capture rare or extreme events in heavy-tailed distributions remains unclear. In this work, we show that traditional diffusion and flow-matching models with standard Gaussian priors fail to capture heavy-tailed behavior. We address this by repurposing the diffusion framework for heavy-tail estimation using multivariate Student-t distributions. We develop a tailored perturbation kernel and derive the denoising posterior based on the conditional Student-t distribution for the backward process. Inspired by $\gamma$-divergence for heavy-tailed distributions, we derive a training objective for heavy-tailed denoisers. The resulting framework introduces controllable tail generation using only a single scalar hyperparameter, making it easily tunable for diverse real-world distributions. As specific instantiations of our framework, we introduce t-EDM and t-Flow, extensions of existing diffusion and flow models that employ a Student-t prior. Remarkably, our approach is readily compatible with standard Gaussian diffusion models and requires only minimal code changes. Empirically, we show that our t-EDM and t-Flow outperform standard diffusion models in heavy-tail estimation on high-resolution weather datasets in which generating rare and extreme events is crucial.

Title: Flexi-Fuzz least squares SVM for Alzheimer's diagnosis: Tackling noise, outliers, and class imbalance

• Authors: Mushir Akhtar, A. Quadir, M. Tanveer, Mohd. Arshad (for the Alzheimer's Disease Neuroimaging)
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14207
• Pdf URL: https://arxiv.org/pdf/2410.14207
• Copy Paste: [[2410.14207]] Flexi-Fuzz least squares SVM for Alzheimer's diagnosis: Tackling noise, outliers, and class imbalance(https://arxiv.org/abs/2410.14207)
• Keywords: generative
• Abstract: Alzheimer's disease (AD) is a leading neurodegenerative condition and the primary cause of dementia, characterized by progressive cognitive decline and memory loss. Its progression, marked by shrinkage in the cerebral cortex, is irreversible. Numerous machine learning algorithms have been proposed for the early diagnosis of AD. However, they often struggle with the issues of noise, outliers, and class imbalance. To tackle the aforementioned limitations, in this article, we introduce a novel, robust, and flexible membership scheme called Flexi-Fuzz. This scheme integrates a novel flexible weighting mechanism, class probability, and imbalance ratio. The proposed flexible weighting mechanism assigns the maximum weight to samples within a specific proximity to the center, with a gradual decrease in weight beyond a certain threshold. This approach ensures that samples near the class boundary still receive significant weight, maintaining their influence in the classification process. Class probability is used to mitigate the impact of noisy samples, while the imbalance ratio addresses class imbalance. Leveraging this, we incorporate the proposed Flexi-Fuzz membership scheme into the least squares support vector machines (LSSVM) framework, resulting in a robust and flexible model termed Flexi-Fuzz-LSSVM. We determine the class-center using two methods: the conventional mean approach and an innovative median approach, leading to two model variants, Flexi-Fuzz-LSSVM-I and Flexi-Fuzz-LSSVM-II. To validate the effectiveness of the proposed Flexi-Fuzz-LSSVM models, we evaluated them on benchmark UCI and KEEL datasets, both with and without label noise. Additionally, we tested the models on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset for AD diagnosis. Experimental results demonstrate the superiority of the Flexi-Fuzz-LSSVM models over baseline models.

Title: Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning

• Authors: Xiaochuan Li, Zichun Yu, Chenyan Xiong
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14208
• Pdf URL: https://arxiv.org/pdf/2410.14208
• Copy Paste: [[2410.14208]] Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning(https://arxiv.org/abs/2410.14208)
• Keywords: generative
• Abstract: Synthetic data has been widely used to train large language models, but their generative nature inevitably introduces noisy, non-informative, and misleading learning signals. In this paper, we propose Montessori-Instruct, a novel data synthesis framework that tailors the data synthesis ability of the teacher language model toward the student language model's learning process. Specifically, we utilize local data influence of synthetic training data points on students to characterize students' learning preferences. Then, we train the teacher model with Direct Preference Optimization (DPO) to generate synthetic data tailored toward student learning preferences. Experiments with Llama3-8B-Instruct (teacher) and Llama3-8B (student) on Alpaca Eval and MT-Bench demonstrate that Montessori-Instruct significantly outperforms standard synthesis methods by 18.35\% and 46.24\% relatively. Our method also beats data synthesized by a stronger teacher model, GPT-4o. Further analysis confirms the benefits of teacher's learning to generate more influential training data in the student's improved learning, the advantages of local data influence in accurately measuring student preferences, and the robustness of Montessori-Instruct across different student models. Our code and data are open-sourced at this https URL.

Title: G-NeuroDAVIS: A Neural Network model for generalized embedding, data visualization and sample generation

• Authors: Chayan Maitra, Rajat K. De
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14223
• Pdf URL: https://arxiv.org/pdf/2410.14223
• Copy Paste: [[2410.14223]] G-NeuroDAVIS: A Neural Network model for generalized embedding, data visualization and sample generation(https://arxiv.org/abs/2410.14223)
• Keywords: generative
• Abstract: Visualizing high-dimensional datasets through a generalized embedding has been a challenge for a long time. Several methods have shown up for the same, but still, they have not been able to generate a generalized embedding, which not only can reveal the hidden patterns present in the data but also generate realistic high-dimensional samples from it. Motivated by this aspect, in this study, a novel generative model, called G-NeuroDAVIS, has been developed, which is capable of visualizing high-dimensional data through a generalized embedding, and thereby generating new samples. The model leverages advanced generative techniques to produce high-quality embedding that captures the underlying structure of the data more effectively than existing methods. G-NeuroDAVIS can be trained in both supervised and unsupervised settings. We rigorously evaluated our model through a series of experiments, demonstrating superior performance in classification tasks, which highlights the robustness of the learned representations. Furthermore, the conditional sample generation capability of the model has been described through qualitative assessments, revealing a marked improvement in generating realistic and diverse samples. G-NeuroDAVIS has outperformed the Variational Autoencoder (VAE) significantly in multiple key aspects, including embedding quality, classification performance, and sample generation capability. These results underscore the potential of our generative model to serve as a powerful tool in various applications requiring high-quality data generation and representation learning.

Title: Unified Convergence Analysis for Score-Based Diffusion Models with Deterministic Samplers

• Authors: Runjia Li, Qiwei Di, Quanquan Gu
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.14237
• Pdf URL: https://arxiv.org/pdf/2410.14237
• Copy Paste: [[2410.14237]] Unified Convergence Analysis for Score-Based Diffusion Models with Deterministic Samplers(https://arxiv.org/abs/2410.14237)
• Keywords: diffusion
• Abstract: Score-based diffusion models have emerged as powerful techniques for generating samples from high-dimensional data distributions. These models involve a two-phase process: first, injecting noise to transform the data distribution into a known prior distribution, and second, sampling to recover the original data distribution from noise. Among the various sampling methods, deterministic samplers stand out for their enhanced efficiency. However, analyzing these deterministic samplers presents unique challenges, as they preclude the use of established techniques such as Girsanov's theorem, which are only applicable to stochastic samplers. Furthermore, existing analysis for deterministic samplers usually focuses on specific examples, lacking a generalized approach for general forward processes and various deterministic samplers. Our paper addresses these limitations by introducing a unified convergence analysis framework. To demonstrate the power of our framework, we analyze the variance-preserving (VP) forward process with the exponential integrator (EI) scheme, achieving iteration complexity of $\tilde O(d^2/\epsilon)$. Additionally, we provide a detailed analysis of Denoising Diffusion Implicit Models (DDIM)-type samplers, which have been underexplored in previous research, achieving polynomial iteration complexity.

Title: Pseudo-label Refinement for Improving Self-Supervised Learning Systems

• Authors: Zia-ur-Rehman, Arif Mahmood, Wenxiong Kang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14242
• Pdf URL: https://arxiv.org/pdf/2410.14242
• Copy Paste: [[2410.14242]] Pseudo-label Refinement for Improving Self-Supervised Learning Systems(https://arxiv.org/abs/2410.14242)
• Keywords: self-supervised
• Abstract: Self-supervised learning systems have gained significant attention in recent years by leveraging clustering-based pseudo-labels to provide supervision without the need for human annotations. However, the noise in these pseudo-labels caused by the clustering methods poses a challenge to the learning process leading to degraded performance. In this work, we propose a pseudo-label refinement (SLR) algorithm to address this issue. The cluster labels from the previous epoch are projected to the current epoch cluster-labels space and a linear combination of the new label and the projected label is computed as a soft refined label containing the information from the previous epoch clusters as well as from the current epoch. In contrast to the common practice of using the maximum value as a cluster/class indicator, we employ hierarchical clustering on these soft pseudo-labels to generate refined hard-labels. This approach better utilizes the information embedded in the soft labels, outperforming the simple maximum value approach for hard label generation. The effectiveness of the proposed SLR algorithm is evaluated in the context of person re-identification (Re-ID) using unsupervised domain adaptation (UDA). Experimental results demonstrate that the modified Re-ID baseline, incorporating the SLR algorithm, achieves significantly improved mean Average Precision (mAP) performance in various UDA tasks, including real-to-synthetic, synthetic-to-real, and different real-to-real scenarios. These findings highlight the efficacy of the SLR algorithm in enhancing the performance of self-supervised learning systems.

Title: ERDDCI: Exact Reversible Diffusion via Dual-Chain Inversion for High-Quality Image Editing

• Authors: Jimin Dai, Yingzhen Zhang, Shuo Chen, Jian Yang, Lei Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14247
• Pdf URL: https://arxiv.org/pdf/2410.14247
• Copy Paste: [[2410.14247]] ERDDCI: Exact Reversible Diffusion via Dual-Chain Inversion for High-Quality Image Editing(https://arxiv.org/abs/2410.14247)
• Keywords: diffusion
• Abstract: Diffusion models (DMs) have been successfully applied to real image editing. These models typically invert images into latent noise vectors used to reconstruct the original images (known as inversion), and then edit them during the inference process. However, recent popular DMs often rely on the assumption of local linearization, where the noise injected during the inversion process is expected to approximate the noise removed during the inference process. While DM efficiently generates images under this assumption, it can also accumulate errors during the diffusion process due to the assumption, ultimately negatively impacting the quality of real image reconstruction and editing. To address this issue, we propose a novel method, referred to as ERDDCI (Exact Reversible Diffusion via Dual-Chain Inversion). ERDDCI uses the new Dual-Chain Inversion (DCI) for joint inference to derive an exact reversible diffusion process. By using DCI, our method effectively avoids the cumbersome optimization process in existing inversion approaches and achieves high-quality image editing. Additionally, to accommodate image operations under high guidance scales, we introduce a dynamic control strategy that enables more refined image reconstruction and editing. Our experiments demonstrate that ERDDCI significantly outperforms state-of-the-art methods in a 50-step diffusion process. It achieves rapid and precise image reconstruction with an SSIM of 0.999 and an LPIPS of 0.001, and also delivers competitive results in image editing.

Title: HYPNOS : Highly Precise Foreground-focused Diffusion Finetuning for Inanimate Objects

• Authors: Oliverio Theophilus Nathanael, Jonathan Samuel Lumentut, Nicholas Hans Muliawan, Edbert Valencio Angky, Felix Indra Kurniadi, Alfi Yusrotis Zakiyyah, Jeklin Harefa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14265
• Pdf URL: https://arxiv.org/pdf/2410.14265
• Copy Paste: [[2410.14265]] HYPNOS : Highly Precise Foreground-focused Diffusion Finetuning for Inanimate Objects(https://arxiv.org/abs/2410.14265)
• Keywords: diffusion, generative
• Abstract: In recent years, personalized diffusion-based text-to-image generative tasks have been a hot topic in computer vision studies. A robust diffusion model is determined by its ability to perform near-perfect reconstruction of certain product outcomes given few related input samples. Unfortunately, the current prominent diffusion-based finetuning technique falls short in maintaining the foreground object consistency while being constrained to produce diverse backgrounds in the image outcome. In the worst scenario, the overfitting issue may occur, meaning that the foreground object is less controllable due to the condition above, for example, the input prompt information is transferred ambiguously to both foreground and background regions, instead of the supposed background region only. To tackle the issues above, we proposed Hypnos, a highly precise foreground-focused diffusion finetuning technique. On the image level, this strategy works best for inanimate object generation tasks, and to do so, Hypnos implements two main approaches, namely: (i) a content-centric prompting strategy and (ii) the utilization of our additional foreground-focused discriminative module. The utilized module is connected with the diffusion model and finetuned with our proposed set of supervision mechanism. Combining the strategies above yielded to the foreground-background disentanglement capability of the diffusion model. Our experimental results showed that the proposed strategy gave a more robust performance and visually pleasing results compared to the former technique. For better elaborations, we also provided extensive studies to assess the fruitful outcomes above, which reveal how personalization behaves in regard to several training conditions.

Title: ClearSR: Latent Low-Resolution Image Embeddings Help Diffusion-Based Real-World Super Resolution Models See Clearer

• Authors: Yuhao Wan, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, Ming-Ming Cheng, Bo Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14279
• Pdf URL: https://arxiv.org/pdf/2410.14279
• Copy Paste: [[2410.14279]] ClearSR: Latent Low-Resolution Image Embeddings Help Diffusion-Based Real-World Super Resolution Models See Clearer(https://arxiv.org/abs/2410.14279)
• Keywords: diffusion, generative
• Abstract: We present ClearSR, a new method that can better take advantage of latent low-resolution image (LR) embeddings for diffusion-based real-world image super-resolution (Real-ISR). Previous Real-ISR models mostly focus on how to activate more generative priors of text-to-image diffusion models to make the output high-resolution (HR) images look better. However, since these methods rely too much on the generative priors, the content of the output images is often inconsistent with the input LR ones. To mitigate the above issue, in this work, we explore using latent LR embeddings to constrain the control signals from ControlNet, and extract LR information at both detail and structure levels. We show that the proper use of latent LR embeddings can produce higher-quality control signals, which enables the super-resolution results to be more consistent with the LR image and leads to clearer visual results. In addition, we also show that latent LR embeddings can be used to control the inference stage, allowing for the improvement of fidelity and generation ability simultaneously. Experiments demonstrate that our model can achieve better performance across multiple metrics on several test sets and generate more consistent SR results with LR images than existing methods. Our code will be made publicly available.

Title: HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation

• Authors: Bo Cheng, Yuhang Ma, Liebucha Wu, Shanyuan Liu, Ao Ma, Xiaoyu Wu, Dawei Leng, Yuhui Yin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14324
• Pdf URL: https://arxiv.org/pdf/2410.14324
• Copy Paste: [[2410.14324]] HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation(https://arxiv.org/abs/2410.14324)
• Keywords: diffusion
• Abstract: The task of layout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object missing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, we propose a \textbf{Hi}erarchical \textbf{Co}ntrollable (HiCo) diffusion model for layout-to-image generation, featuring object seperable conditioning branch structure. Our key insight is to achieve spatial disentanglement through hierarchical modeling of layouts. We use a multi branch structure to represent hierarchy and aggregate them in fusion module. To evaluate the performance of multi-objective controllable layout generation in natural scenes, we introduce the HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned. this https URL.

Title: Croc: Pretraining Large Multimodal Models with Cross-Modal Comprehension

• Authors: Yin Xie, Kaicheng Yang, Ninghua Yang, Weimo Deng, Xiangzi Dai, Tiancheng Gu, Yumeng Wang, Xiang An, Yongle Zhao, Ziyong Feng, Jiankang Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14332
• Pdf URL: https://arxiv.org/pdf/2410.14332
• Copy Paste: [[2410.14332]] Croc: Pretraining Large Multimodal Models with Cross-Modal Comprehension(https://arxiv.org/abs/2410.14332)
• Keywords: foundation model
• Abstract: Recent advances in Large Language Models (LLMs) have catalyzed the development of Large Multimodal Models (LMMs). However, existing research primarily focuses on tuning language and image instructions, ignoring the critical pretraining phase where models learn to process textual and visual modalities jointly. In this paper, we propose a new pretraining paradigm for LMMs to enhance the visual comprehension capabilities of LLMs by introducing a novel cross-modal comprehension stage. Specifically, we design a dynamically learnable prompt token pool and employ the Hungarian algorithm to replace part of the original visual tokens with the most relevant prompt tokens. Then, we conceptualize visual tokens as analogous to a "foreign language" for the LLMs and propose a mixed attention mechanism with bidirectional visual attention and unidirectional textual attention to comprehensively enhance the understanding of visual tokens. Meanwhile, we integrate a detailed caption generation task, leveraging rich descriptions to further facilitate LLMs in understanding visual semantic information. After pretraining on 1.5 million publicly accessible data, we present a new foundation model called Croc. Experimental results demonstrate that Croc achieves new state-of-the-art performance on massive vision-language benchmarks. To support reproducibility and facilitate further research, we release the training code and pre-trained model weights at this https URL.

Title: AnomalyNCD: Towards Novel Anomaly Class Discovery in Industrial Scenarios

• Authors: Ziming Huang, Xurui Li, Haotian Liu, Feng Xue, Yuzhe Wang, Yu Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14379
• Pdf URL: https://arxiv.org/pdf/2410.14379
• Copy Paste: [[2410.14379]] AnomalyNCD: Towards Novel Anomaly Class Discovery in Industrial Scenarios(https://arxiv.org/abs/2410.14379)
• Keywords: self-supervised, anomaly
• Abstract: In the industrial scenario, anomaly detection could locate but cannot classify anomalies. To complete their capability, we study to automatically discover and recognize visual classes of industrial anomalies. In terms of multi-class anomaly classification, previous methods cluster anomalies represented by frozen pre-trained models but often fail due to poor discrimination. Novel class discovery (NCD) has the potential to tackle this. However, it struggles with non-prominent and semantically weak anomalies that challenge network learning focus. To address these, we introduce AnomalyNCD, a multi-class anomaly classification framework compatible with existing anomaly detection methods. This framework learns anomaly-specific features and classifies anomalies in a self-supervised manner. Initially, a technique called Main Element Binarization (MEBin) is first designed, which segments primary anomaly regions into masks to alleviate the impact of incorrect detections on learning. Subsequently, we employ mask-guided contrastive representation learning to improve feature discrimination, which focuses network attention on isolated anomalous regions and reduces the confusion of erroneous inputs through re-corrected pseudo labels. Finally, to enable flexible classification at both region and image levels during inference, we develop a region merging strategy that determines the overall image category based on the classified anomaly regions. Our method outperforms the state-of-the-art works on the MVTec AD and MTD datasets. Compared with the current methods, AnomalyNCD combined with zero-shot anomaly detection method achieves a 10.8% $F_1$ gain, 8.8% NMI gain, and 9.5% ARI gain on MVTec AD, 12.8% $F_1$ gain, 5.7% NMI gain, and 10.8% ARI gain on MTD. The source code is available at this https URL.

Title: Unscrambling disease progression at scale: fast inference of event permutations with optimal transport

• Authors: Peter A. Wijeratne, Daniel C. Alexander
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14388
• Pdf URL: https://arxiv.org/pdf/2410.14388
• Copy Paste: [[2410.14388]] Unscrambling disease progression at scale: fast inference of event permutations with optimal transport(https://arxiv.org/abs/2410.14388)
• Keywords: generative
• Abstract: Disease progression models infer group-level temporal trajectories of change in patients' features as a chronic degenerative condition plays out. They provide unique insight into disease biology and staging systems with individual-level clinical utility. Discrete models consider disease progression as a latent permutation of events, where each event corresponds to a feature becoming measurably abnormal. However, permutation inference using traditional maximum likelihood approaches becomes prohibitive due to combinatoric explosion, severely limiting model dimensionality and utility. Here we leverage ideas from optimal transport to model disease progression as a latent permutation matrix of events belonging to the Birkhoff polytope, facilitating fast inference via optimisation of the variational lower bound. This enables a factor of 1000 times faster inference than the current state of the art and, correspondingly, supports models with several orders of magnitude more features than the current state of the art can consider. Experiments demonstrate the increase in speed, accuracy and robustness to noise in simulation. Further experiments with real-world imaging data from two separate datasets, one from Alzheimer's disease patients, the other age-related macular degeneration, showcase, for the first time, pixel-level disease progression events in the brain and eye, respectively. Our method is low compute, interpretable and applicable to any progressive condition and data modality, giving it broad potential clinical utility.

Title: Generative AI, Pragmatics, and Authenticity in Second Language Learning

• Authors: Robert Godwin-Jones`
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.14395
• Pdf URL: https://arxiv.org/pdf/2410.14395
• Copy Paste: [[2410.14395]] Generative AI, Pragmatics, and Authenticity in Second Language Learning(https://arxiv.org/abs/2410.14395)
• Keywords: generative
• Abstract: There are obvious benefits to integrating generative AI (artificial intelligence) into language learning and teaching. Those include using AI as a language tutor, creating learning materials, or assessing learner output. However, due to how AI systems under-stand human language, based on a mathematical model using statistical probability, they lack the lived experience to be able to use language with the same social aware-ness as humans. Additionally, there are built-in linguistic and cultural biases based on their training data which is mostly in English and predominantly from Western sources. Those facts limit AI suitability for some language learning interactions. Stud-ies have clearly shown that systems such as ChatGPT often do not produce language that is pragmatically appropriate. The lack of linguistic and cultural authenticity has important implications for how AI is integrated into second language acquisition as well as in instruction targeting development of intercultural communication compe-tence.

Title: Dynamic Negative Guidance of Diffusion Models

• Authors: Felix Koulischer, Johannes Deleu, Gabriel Raya, Thomas Demeester, Luca Ambrogioni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14398
• Pdf URL: https://arxiv.org/pdf/2410.14398
• Copy Paste: [[2410.14398]] Dynamic Negative Guidance of Diffusion Models(https://arxiv.org/abs/2410.14398)
• Keywords: diffusion, generative
• Abstract: Negative Prompting (NP) is widely utilized in diffusion models, particularly in text-to-image applications, to prevent the generation of undesired features. In this paper, we show that conventional NP is limited by the assumption of a constant guidance scale, which may lead to highly suboptimal results, or even complete failure, due to the non-stationarity and state-dependence of the reverse process. Based on this analysis, we derive a principled technique called Dynamic Negative Guidance, which relies on a near-optimal time and state dependent modulation of the guidance without requiring additional training. Unlike NP, negative guidance requires estimating the posterior class probability during the denoising process, which is achieved with limited additional computational overhead by tracking the discrete Markov Chain during the generative process. We evaluate the performance of DNG class-removal on MNIST and CIFAR10, where we show that DNG leads to higher safety, preservation of class balance and image quality when compared with baseline methods. Furthermore, we show that it is possible to use DNG with Stable Diffusion to obtain more accurate and less invasive guidance than NP.

Title: SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning

• Authors: Magdalena Wysocka, Danilo S. Carvalho, Oskar Wysocki, Marco Valentino, Andre Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.14399
• Pdf URL: https://arxiv.org/pdf/2410.14399
• Copy Paste: [[2410.14399]] SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning(https://arxiv.org/abs/2410.14399)
• Keywords: in-context
• Abstract: Syllogistic reasoning is crucial for Natural Language Inference (NLI). This capability is particularly significant in specialized domains such as biomedicine, where it can support automatic evidence interpretation and scientific discovery. This paper presents SylloBio-NLI, a novel framework that leverages external ontologies to systematically instantiate diverse syllogistic arguments for biomedical NLI. We employ SylloBio-NLI to evaluate Large Language Models (LLMs) on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. Extensive experiments reveal that biomedical syllogistic reasoning is particularly challenging for zero-shot LLMs, which achieve an average accuracy between 70% on generalized modus ponens and 23% on disjunctive syllogism. At the same time, we found that few-shot prompting can boost the performance of different LLMs, including Gemma (+14%) and LLama-3 (+43%). However, a deeper analysis shows that both techniques exhibit high sensitivity to superficial lexical variations, highlighting a dependency between reliability, models' architecture, and pre-training regime. Overall, our results indicate that, while in-context examples have the potential to elicit syllogistic reasoning in LLMs, existing models are still far from achieving the robustness and consistency required for safe biomedical NLI applications.

Title: FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models

• Authors: Rui Hu, Qian He, Gaofeng He, Jiedong Zhuang, Huang Chen, Huafeng Liu, Huamin Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14429
• Pdf URL: https://arxiv.org/pdf/2410.14429
• Copy Paste: [[2410.14429]] FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models(https://arxiv.org/abs/2410.14429)
• Keywords: diffusion, generative
• Abstract: Modeling and producing lifelike clothed human images has attracted researchers' attention from different areas for decades, with the complexity from highly articulated and structured content. Rendering algorithms decompose and simulate the imaging process of a camera, while are limited by the accuracy of modeled variables and the efficiency of computation. Generative models can produce impressively vivid human images, however still lacking in controllability and editability. This paper studies photorealism enhancement of rendered images, leveraging generative power from diffusion models on the controlled basis of rendering. We introduce a novel framework to translate rendered images into their realistic counterparts, which consists of two stages: Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG). In DKI, we adopt positive (real) domain finetuning and negative (rendered) domain embedding to inject knowledge into a pretrained Text-to-image (T2I) diffusion model. In RIG, we generate the realistic image corresponding to the input rendered image, with a Texture-preserving Attention Control (TAC) to preserve fine-grained clothing textures, exploiting the decoupled features encoded in the UNet structure. Additionally, we introduce SynFashion dataset, featuring high-quality digital clothing images with diverse textures. Extensive experimental results demonstrate the superiority and effectiveness of our method in rendered-to-real image translation.

Title: Toward Generalizing Visual Brain Decoding to Unseen Subjects

• Authors: Xiangtao Kong, Kexin Huang, Ping Li, Lei Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14445
• Pdf URL: https://arxiv.org/pdf/2410.14445
• Copy Paste: [[2410.14445]] Toward Generalizing Visual Brain Decoding to Unseen Subjects(https://arxiv.org/abs/2410.14445)
• Keywords: foundation model
• Abstract: Visual brain decoding aims to decode visual information from human brain activities. Despite the great progress, one critical limitation of current brain decoding research lies in the lack of generalization capability to unseen subjects. Prior works typically focus on decoding brain activity of individuals based on the observation that different subjects exhibit different brain activities, while it remains unclear whether brain decoding can be generalized to unseen subjects. This study aims to answer this question. We first consolidate an image-fMRI dataset consisting of stimulus-image and fMRI-response pairs, involving 177 subjects in the movie-viewing task of the Human Connectome Project (HCP). This dataset allows us to investigate the brain decoding performance with the increase of participants. We then present a learning paradigm that applies uniform processing across all subjects, instead of employing different network heads or tokenizers for individuals as in previous methods, which can accommodate a large number of subjects to explore the generalization capability across different subjects. A series of experiments are conducted and we have the following findings. First, the network exhibits clear generalization capabilities with the increase of training subjects. Second, the generalization capability is common to popular network architectures (MLP, CNN and Transformer). Third, the generalization performance is affected by the similarity between subjects. Our findings reveal the inherent similarities in brain activities across individuals. With the emerging of larger and more comprehensive datasets, it is possible to train a brain decoding foundation model in the this http URL and models can be found at this https URL.

Title: LUDVIG: Learning-free Uplifting of 2D Visual features to Gaussian Splatting scenes

• Authors: Juliette Marrie, Romain Ménégaux, Michael Arbel, Diane Larlus, Julien Mairal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14462
• Pdf URL: https://arxiv.org/pdf/2410.14462
• Copy Paste: [[2410.14462]] LUDVIG: Learning-free Uplifting of 2D Visual features to Gaussian Splatting scenes(https://arxiv.org/abs/2410.14462)
• Keywords: diffusion
• Abstract: We address the task of uplifting visual features or semantic masks from 2D vision models to 3D scenes represented by Gaussian Splatting. Whereas common approaches rely on iterative optimization-based procedures, we show that a simple yet effective aggregation technique yields excellent results. Applied to semantic masks from Segment Anything (SAM), our uplifting approach leads to segmentation quality comparable to the state of the art. We then extend this method to generic DINOv2 features, integrating 3D scene geometry through graph diffusion, and achieve competitive segmentation results despite DINOv2 not being trained on millions of annotated masks like SAM.

Title: How Do Training Methods Influence the Utilization of Vision Models?

• Authors: Paul Gavrikov, Shashank Agnihotri, Margret Keuper, Janis Keuper
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.14470
• Pdf URL: https://arxiv.org/pdf/2410.14470
• Copy Paste: [[2410.14470]] How Do Training Methods Influence the Utilization of Vision Models?(https://arxiv.org/abs/2410.14470)
• Keywords: self-supervised
• Abstract: Not all learnable parameters (e.g., weights) contribute equally to a neural network's decision function. In fact, entire layers' parameters can sometimes be reset to random values with little to no impact on the model's decisions. We revisit earlier studies that examined how architecture and task complexity influence this phenomenon and ask: is this phenomenon also affected by how we train the model? We conducted experimental evaluations on a diverse set of ImageNet-1k classification models to explore this, keeping the architecture and training data constant but varying the training pipeline. Our findings reveal that the training method strongly influences which layers become critical to the decision function for a given task. For example, improved training regimes and self-supervised training increase the importance of early layers while significantly under-utilizing deeper layers. In contrast, methods such as adversarial training display an opposite trend. Our preliminary results extend previous findings, offering a more nuanced understanding of the inner mechanics of neural networks. Code: this https URL

Title: ANT: Adaptive Noise Schedule for Time Series Diffusion Models

• Authors: Seunghan Lee, Kibok Lee, Taeyoung Park
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.14488
• Pdf URL: https://arxiv.org/pdf/2410.14488
• Copy Paste: [[2410.14488]] ANT: Adaptive Noise Schedule for Time Series Diffusion Models(https://arxiv.org/abs/2410.14488)
• Keywords: diffusion, generative
• Abstract: Advances in diffusion models for generative artificial intelligence have recently propagated to the time series (TS) domain, demonstrating state-of-the-art performance on various tasks. However, prior works on TS diffusion models often borrow the framework of existing works proposed in other domains without considering the characteristics of TS data, leading to suboptimal performance. In this work, we propose Adaptive Noise schedule for Time series diffusion models (ANT), which automatically predetermines proper noise schedules for given TS datasets based on their statistics representing non-stationarity. Our intuition is that an optimal noise schedule should satisfy the following desiderata: 1) It linearly reduces the non-stationarity of TS data so that all diffusion steps are equally meaningful, 2) the data is corrupted to the random noise at the final step, and 3) the number of steps is sufficiently large. The proposed method is practical for use in that it eliminates the necessity of finding the optimal noise schedule with a small additional cost to compute the statistics for given datasets, which can be done offline before training. We validate the effectiveness of our method across various tasks, including TS forecasting, refinement, and generation, on datasets from diverse domains. Code is available at this repository: this https URL.

Title: LEAD: Latent Realignment for Human Motion Diffusion

• Authors: Nefeli Andreou, Xi Wang, Victoria Fernández Abrevaya, Marie-Paule Cani, Yiorgos Chrysanthou, Vicky Kalogeiton
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.14508
• Pdf URL: https://arxiv.org/pdf/2410.14508
• Copy Paste: [[2410.14508]] LEAD: Latent Realignment for Human Motion Diffusion(https://arxiv.org/abs/2410.14508)
• Keywords: diffusion
• Abstract: Our goal is to generate realistic human motion from natural language. Modern methods often face a trade-off between model expressiveness and text-to-motion alignment. Some align text and motion latent spaces but sacrifice expressiveness; others rely on diffusion models producing impressive motions, but lacking semantic meaning in their latent space. This may compromise realism, diversity, and applicability. Here, we address this by combining latent diffusion with a realignment mechanism, producing a novel, semantically structured space that encodes the semantics of language. Leveraging this capability, we introduce the task of textual motion inversion to capture novel motion concepts from a few examples. For motion synthesis, we evaluate LEAD on HumanML3D and KIT-ML and show comparable performance to the state-of-the-art in terms of realism, diversity, and text-motion consistency. Our qualitative analysis and user study reveal that our synthesized motions are sharper, more human-like and comply better with the text compared to modern methods. For motion textual inversion, our method demonstrates improved capacity in capturing out-of-distribution characteristics in comparison to traditional VAEs.

Title: Multi-modal Pose Diffuser: A Multimodal Generative Conditional Pose Prior

• Authors: Calvin-Khang Ta, Arindam Dutta, Rohit Kundu, Rohit Lal, Hannah Dela Cruz, Dripta S. Raychaudhuri, Amit Roy-Chowdhury
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14540
• Pdf URL: https://arxiv.org/pdf/2410.14540
• Copy Paste: [[2410.14540]] Multi-modal Pose Diffuser: A Multimodal Generative Conditional Pose Prior(https://arxiv.org/abs/2410.14540)
• Keywords: diffusion, generative
• Abstract: The Skinned Multi-Person Linear (SMPL) model plays a crucial role in 3D human pose estimation, providing a streamlined yet effective representation of the human body. However, ensuring the validity of SMPL configurations during tasks such as human mesh regression remains a significant challenge , highlighting the necessity for a robust human pose prior capable of discerning realistic human poses. To address this, we introduce MOPED: \underline{M}ulti-m\underline{O}dal \underline{P}os\underline{E} \underline{D}iffuser. MOPED is the first method to leverage a novel multi-modal conditional diffusion model as a prior for SMPL pose parameters. Our method offers powerful unconditional pose generation with the ability to condition on multi-modal inputs such as images and text. This capability enhances the applicability of our approach by incorporating additional context often overlooked in traditional pose priors. Extensive experiments across three distinct tasks-pose estimation, pose denoising, and pose completion-demonstrate that our multi-modal diffusion model-based prior significantly outperforms existing methods. These results indicate that our model captures a broader spectrum of plausible human poses.

Title: Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization

• Authors: Frederic Kirstein, Terry Ruas, Robert Kratel, Bela Gipp
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14545
• Pdf URL: https://arxiv.org/pdf/2410.14545
• Copy Paste: [[2410.14545]] Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization(https://arxiv.org/abs/2410.14545)
• Keywords: generative
• Abstract: Meeting summarization is crucial in digital communication, but existing solutions struggle with salience identification to generate personalized, workable summaries, and context understanding to fully comprehend the meetings' content. Previous attempts to address these issues by considering related supplementary resources (e.g., presentation slides) alongside transcripts are hindered by models' limited context sizes and handling the additional complexities of the multi-source tasks, such as identifying relevant information in additional files and seamlessly aligning it with the meeting content. This work explores multi-source meeting summarization considering supplementary materials through a three-stage large language model approach: identifying transcript passages needing additional context, inferring relevant details from supplementary materials and inserting them into the transcript, and generating a summary from this enriched transcript. Our multi-source approach enhances model understanding, increasing summary relevance by ~9% and producing more content-rich outputs. We introduce a personalization protocol that extracts participant characteristics and tailors summaries accordingly, improving informativeness by ~10%. This work further provides insights on performance-cost trade-offs across four leading model families, including edge-device capable options. Our approach can be extended to similar complex generative tasks benefitting from additional resources and personalization, such as dialogue systems and action planning.

Title: Towards Unsupervised Validation of Anomaly-Detection Models

• Authors: Lihi Idan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14579
• Pdf URL: https://arxiv.org/pdf/2410.14579
• Copy Paste: [[2410.14579]] Towards Unsupervised Validation of Anomaly-Detection Models(https://arxiv.org/abs/2410.14579)
• Keywords: anomaly
• Abstract: Unsupervised validation of anomaly-detection models is a highly challenging task. While the common practices for model validation involve a labeled validation set, such validation sets cannot be constructed when the underlying datasets are unlabeled. The lack of robust and efficient unsupervised model-validation techniques presents an acute challenge in the implementation of automated anomaly-detection pipelines, especially when there exists no prior knowledge of the model's performance on similar datasets. This work presents a new paradigm to automated validation of anomaly-detection models, inspired by real-world, collaborative decision-making mechanisms. We focus on two commonly-used, unsupervised model-validation tasks -- model selection and model evaluation -- and provide extensive experimental results that demonstrate the accuracy and robustness of our approach on both tasks.

Title: Neuro-Symbolic Traders: Assessing the Wisdom of AI Crowds in Markets

• Authors: Namid R. Stillman, Rory Baggott
• Subjects: cs.LG, q-fin.CP
• Abstract URL: https://arxiv.org/abs/2410.14587
• Pdf URL: https://arxiv.org/pdf/2410.14587
• Copy Paste: [[2410.14587]] Neuro-Symbolic Traders: Assessing the Wisdom of AI Crowds in Markets(https://arxiv.org/abs/2410.14587)
• Keywords: generative
• Abstract: Deep generative models are becoming increasingly used as tools for financial analysis. However, it is unclear how these models will influence financial markets, especially when they infer financial value in a semi-autonomous way. In this work, we explore the interplay between deep generative models and market dynamics. We develop a form of virtual traders that use deep generative models to make buy/sell decisions, which we term neuro-symbolic traders, and expose them to a virtual market. Under our framework, neuro-symbolic traders are agents that use vision-language models to discover a model of the fundamental value of an asset. Agents develop this model as a stochastic differential equation, calibrated to market data using gradient descent. We test our neuro-symbolic traders on both synthetic data and real financial time series, including an equity stock, commodity, and a foreign exchange pair. We then expose several groups of neuro-symbolic traders to a virtual market environment. This market environment allows for feedback between the traders belief of the underlying value to the observed price dynamics. We find that this leads to price suppression compared to the historical data, highlighting a future risk to market stability. Our work is a first step towards quantifying the effect of deep generative agents on markets dynamics and sets out some of the potential risks and benefits of this approach in the future.

Title: How Does Data Diversity Shape the Weight Landscape of Neural Networks?

• Authors: Yang Ba, Michelle V. Mancenido, Rong Pan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14602
• Pdf URL: https://arxiv.org/pdf/2410.14602
• Copy Paste: [[2410.14602]] How Does Data Diversity Shape the Weight Landscape of Neural Networks?(https://arxiv.org/abs/2410.14602)
• Keywords: generative
• Abstract: To enhance the generalization of machine learning models to unseen data, techniques such as dropout, weight decay ($L_2$ regularization), and noise augmentation are commonly employed. While regularization methods (i.e., dropout and weight decay) are geared toward adjusting model parameters to prevent overfitting, data augmentation increases the diversity of the input training set, a method purported to improve accuracy and calibration error. In this paper, we investigate the impact of each of these techniques on the parameter space of neural networks, with the goal of understanding how they alter the weight landscape in transfer learning scenarios. To accomplish this, we employ Random Matrix Theory to analyze the eigenvalue distributions of pre-trained models, fine-tuned using these techniques but using different levels of data diversity, for the same downstream tasks. We observe that diverse data influences the weight landscape in a similar fashion as dropout. Additionally, we compare commonly used data augmentation methods with synthetic data created by generative models. We conclude that synthetic data can bring more diversity into real input data, resulting in a better performance on out-of-distribution test instances.

Title: On the Regularization of Learnable Embeddings for Time Series Processing

• Authors: Luca Butera, Giovanni De Felice, Andrea Cini, Cesare Alippi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14630
• Pdf URL: https://arxiv.org/pdf/2410.14630
• Copy Paste: [[2410.14630]] On the Regularization of Learnable Embeddings for Time Series Processing(https://arxiv.org/abs/2410.14630)
• Keywords: foundation model
• Abstract: In processing multiple time series, accounting for the individual features of each sequence can be challenging. To address this, modern deep learning methods for time series analysis combine a shared (global) model with local layers, specific to each time series, often implemented as learnable embeddings. Ideally, these local embeddings should encode meaningful representations of the unique dynamics of each sequence. However, when these are learned end-to-end as parameters of a forecasting model, they may end up acting as mere sequence identifiers. Shared processing blocks may then become reliant on such identifiers, limiting their transferability to new contexts. In this paper, we address this issue by investigating methods to regularize the learning of local learnable embeddings for time series processing. Specifically, we perform the first extensive empirical study on the subject and show how such regularizations consistently improve performance in widely adopted architectures. Furthermore, we show that methods preventing the co-adaptation of local and global parameters are particularly effective in this context. This hypothesis is validated by comparing several methods preventing the downstream models from relying on sequence identifiers, going as far as completely resetting the embeddings during training. The obtained results provide an important contribution to understanding the interplay between learnable local parameters and shared processing layers: a key challenge in modern time series processing models and a step toward developing effective foundation models for time series.

Title: Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning

• Authors: Yuxiang Lu, Shengcao Cao, Yu-Xiong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.14633
• Pdf URL: https://arxiv.org/pdf/2410.14633
• Copy Paste: [[2410.14633]] Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning(https://arxiv.org/abs/2410.14633)
• Keywords: foundation model
• Abstract: Vision Foundation Models (VFMs) have demonstrated outstanding performance on numerous downstream tasks. However, due to their inherent representation biases originating from different training paradigms, VFMs exhibit advantages and disadvantages across distinct vision tasks. Although amalgamating the strengths of multiple VFMs for downstream tasks is an intuitive strategy, effectively exploiting these biases remains a significant challenge. In this paper, we propose a novel and versatile "Swiss Army Knife" (SAK) solution, which adaptively distills knowledge from a committee of VFMs to enhance multi-task learning. Unlike existing methods that use a single backbone for knowledge transfer, our approach preserves the unique representation bias of each teacher by collaborating the lightweight Teacher-Specific Adapter Path modules with the Teacher-Agnostic Stem. Through dynamic selection and combination of representations with Mixture-of-Representations Routers, our SAK is capable of synergizing the complementary strengths of multiple VFMs. Extensive experiments show that our SAK remarkably outperforms prior state of the arts in multi-task learning by 10% on the NYUD-v2 benchmark, while also providing a flexible and robust framework that can readily accommodate more advanced model designs.

Title: GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings

• Authors: Raghuveer Thirukovalluru, Bhuwan Dhingra
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14635
• Pdf URL: https://arxiv.org/pdf/2410.14635
• Copy Paste: [[2410.14635]] GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings(https://arxiv.org/abs/2410.14635)
• Keywords: generative
• Abstract: Training-free embedding methods directly leverage pretrained large language models (LLMs) to embed text, bypassing the costly and complex procedure of contrastive learning. Previous training-free embedding methods have mainly focused on optimizing embedding prompts and have overlooked the benefits of utilizing the generative abilities of LLMs. We propose a novel method, GenEOL, which uses LLMs to generate diverse transformations of a sentence that preserve its meaning, and aggregates the resulting embeddings of these transformations to enhance the overall sentence embedding. GenEOL significantly outperforms the existing training-free embedding methods by an average of 2.85 points across several LLMs on the sentence semantic text similarity (STS) benchmark. Our analysis shows that GenEOL stabilizes representation quality across LLM layers and is robust to perturbations of embedding prompts. GenEOL also achieves notable gains on multiple clustering, reranking and pair-classification tasks from the MTEB benchmark.

Title: BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities

• Authors: Shaozhe Hao, Xuantong Liu, Xianbiao Qi, Shihao Zhao, Bojia Zi, Rong Xiao, Kai Han, Kwan-Yee K. Wong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.14672
• Pdf URL: https://arxiv.org/pdf/2410.14672
• Copy Paste: [[2410.14672]] BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities(https://arxiv.org/abs/2410.14672)
• Keywords: generative
• Abstract: We introduce BiGR, a novel conditional image generation model using compact binary latent codes for generative training, focusing on enhancing both generation and representation capabilities. BiGR is the first conditional generative model that unifies generation and discrimination within the same framework. BiGR features a binary tokenizer, a masked modeling mechanism, and a binary transcoder for binary code prediction. Additionally, we introduce a novel entropy-ordered sampling method to enable efficient image generation. Extensive experiments validate BiGR's superior performance in generation quality, as measured by FID-50k, and representation capabilities, as evidenced by linear-probe accuracy. Moreover, BiGR showcases zero-shot generalization across various vision tasks, enabling applications such as image inpainting, outpainting, editing, interpolation, and enrichment, without the need for structural modifications. Our findings suggest that BiGR unifies generative and discriminative tasks effectively, paving the way for further advancements in the field.