2024-09-26

Title: GenCAD: Image-Conditioned Computer-Aided Design Generation with Transformer-Based Contrastive Representation and Diffusion Priors

• Authors: Md Ferdous Alam, Faez Ahmed
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16294
• Pdf URL: https://arxiv.org/pdf/2409.16294
• Copy Paste: [[2409.16294]] GenCAD: Image-Conditioned Computer-Aided Design Generation with Transformer-Based Contrastive Representation and Diffusion Priors(https://arxiv.org/abs/2409.16294)
• Keywords: diffusion, generative
• Abstract: The creation of manufacturable and editable 3D shapes through Computer-Aided Design (CAD) remains a highly manual and time-consuming task, hampered by the complex topology of boundary representations of 3D solids and unintuitive design tools. This paper introduces GenCAD, a generative model that employs autoregressive transformers and latent diffusion models to transform image inputs into parametric CAD command sequences, resulting in editable 3D shape representations. GenCAD integrates an autoregressive transformer-based architecture with a contrastive learning framework, enhancing the generation of CAD programs from input images and providing a representation learning framework for multiple data modalities relevant to engineering designs. Extensive evaluations demonstrate that GenCAD significantly outperforms existing state-of-the-art methods in terms of the precision and modifiability of generated 3D shapes. Notably, GenCAD shows a marked improvement in the accuracy of 3D shape generation for long sequences, supporting its application in complex design tasks. Additionally, the contrastive embedding feature of GenCAD facilitates the retrieval of CAD models using image queries from databases which is a critical challenge within the CAD community. While most work in the 3D shape generation literature focuses on representations like meshes, voxels, or point clouds, practical engineering applications demand modifiability and the ability for multi-modal conditional generation. Our results provide a significant step forward in this direction, highlighting the potential of generative models to expedite the entire design-to-production pipeline and seamlessly integrate different design modalities.

Title: DeepScore: A Comprehensive Approach to Measuring Quality in AI-Generated Clinical Documentation

• Authors: Jon Oleson
• Subjects: cs.CL, cs.AI, stat.AP
• Abstract URL: https://arxiv.org/abs/2409.16307
• Pdf URL: https://arxiv.org/pdf/2409.16307
• Copy Paste: [[2409.16307]] DeepScore: A Comprehensive Approach to Measuring Quality in AI-Generated Clinical Documentation(https://arxiv.org/abs/2409.16307)
• Keywords: generative
• Abstract: Medical practitioners are rapidly adopting generative AI solutions for clinical documentation, leading to significant time savings and reduced stress. However, evaluating the quality of AI-generated documentation is a complex and ongoing challenge. This paper presents an overview of DeepScribe's methodologies for assessing and managing note quality, focusing on various metrics and the composite "DeepScore", an overall index of quality and accuracy. These methodologies aim to enhance the quality of patient care documentation through accountability and continuous improvement.

Title: Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs

• Authors: Shadi Iskander, Nachshon Cohen, Zohar Karnin, Ori Shapira, Sofia Tolmach
• Subjects: cs.LG, cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.16341
• Pdf URL: https://arxiv.org/pdf/2409.16341
• Copy Paste: [[2409.16341]] Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs(https://arxiv.org/abs/2409.16341)
• Keywords: in-context
• Abstract: Training large language models (LLMs) for external tool usage is a rapidly expanding field, with recent research focusing on generating synthetic data to address the shortage of available data. However, the absence of systematic data quality checks poses complications for properly training and testing models. To that end, we propose two approaches for assessing the reliability of data for training LLMs to use external tools. The first approach uses intuitive, human-defined correctness criteria. The second approach uses a model-driven assessment with in-context evaluation. We conduct a thorough evaluation of data quality on two popular benchmarks, followed by an extrinsic evaluation that showcases the impact of data quality on model performance. Our results demonstrate that models trained on high-quality data outperform those trained on unvalidated data, even when trained with a smaller quantity of data. These findings empirically support the significance of assessing and ensuring the reliability of training data for tool-using LLMs.

Title: Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs

• Authors: Amartya Roy, Danush Khanna, Devanshu Mahapatra, Vasanthakumar, Avirup Das, Kripabandhu Ghosh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16371
• Pdf URL: https://arxiv.org/pdf/2409.16371
• Copy Paste: [[2409.16371]] Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs(https://arxiv.org/abs/2409.16371)
• Keywords: in-context
• Abstract: This paper tackles the challenge of building robust and generalizable bias mitigation models for language. Recognizing the limitations of existing datasets, we introduce ANUBIS, a novel dataset with 1507 carefully curated sentence pairs encompassing nine social bias categories. We evaluate state-of-the-art models like T5, utilizing Supervised Fine-Tuning (SFT), Reinforcement Learning (PPO, DPO), and In-Context Learning (ICL) for effective bias mitigation. Our analysis focuses on multi-class social bias reduction, cross-dataset generalizability, and environmental impact of the trained models. ANUBIS and our findings offer valuable resources for building more equitable AI systems and contribute to the development of responsible and unbiased technologies with broad societal impact.

Title: RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation

• Authors: Ioannis Panagiotopoulos, Giorgos Filandrianos, Maria Lymperaiou, Giorgos Stamou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16383
• Pdf URL: https://arxiv.org/pdf/2409.16383
• Copy Paste: [[2409.16383]] RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation(https://arxiv.org/abs/2409.16383)
• Keywords: in-context
• Abstract: Riddle-solving requires advanced reasoning skills, pushing LLMs to engage in abstract thinking and creative problem-solving, often revealing limitations in their cognitive abilities. In this paper, we examine the riddle-solving capabilities of LLMs using a multiple-choice format, exploring how different prompting techniques impact performance on riddles that demand diverse reasoning skills. To enhance results, we introduce RISCORE (RIddle Solving with COntext REcontruciton) a novel fully automated prompting method that generates and utilizes contextually reconstructed sentence-based puzzles in conjunction with the original examples to create few-shot exemplars. Our experiments demonstrate that RISCORE significantly improves the performance of language models in both vertical and lateral thinking tasks, surpassing traditional exemplar selection strategies across a variety of few-shot settings.

Title: Generative AI-driven forecasting of oil production

• Authors: Yash Gandhi, Kexin Zheng, Birendra Jha, Ken-ichi Nomura, Aiichiro Nakano, Priya Vashishta, Rajiv K. Kalia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16482
• Pdf URL: https://arxiv.org/pdf/2409.16482
• Copy Paste: [[2409.16482]] Generative AI-driven forecasting of oil production(https://arxiv.org/abs/2409.16482)
• Keywords: generative
• Abstract: Forecasting oil production from oilfields with multiple wells is an important problem in petroleum and geothermal energy extraction, as well as energy storage technologies. The accuracy of oil forecasts is a critical determinant of economic projections, hydrocarbon reserves estimation, construction of fluid processing facilities, and energy price fluctuations. Leveraging generative AI techniques, we model time series forecasting of oil and water productions across four multi-well sites spanning four decades. Our goal is to effectively model uncertainties and make precise forecasts to inform decision-making processes at the field scale. We utilize an autoregressive model known as TimeGrad and a variant of a transformer architecture named Informer, tailored specifically for forecasting long sequence time series data. Predictions from both TimeGrad and Informer closely align with the ground truth data. The overall performance of the Informer stands out, demonstrating greater efficiency compared to TimeGrad in forecasting oil production rates across all sites.

Title: Prompt Sliders for Fine-Grained Control, Editing and Erasing of Concepts in Diffusion Models

• Authors: Deepak Sridhar, Nuno Vasconcelos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16535
• Pdf URL: https://arxiv.org/pdf/2409.16535
• Copy Paste: [[2409.16535]] Prompt Sliders for Fine-Grained Control, Editing and Erasing of Concepts in Diffusion Models(https://arxiv.org/abs/2409.16535)
• Keywords: diffusion
• Abstract: Diffusion models have recently surpassed GANs in image synthesis and editing, offering superior image quality and diversity. However, achieving precise control over attributes in generated images remains a challenge. Concept Sliders introduced a method for fine-grained image control and editing by learning concepts (attributes/objects). However, this approach adds parameters and increases inference time due to the loading and unloading of Low-Rank Adapters (LoRAs) used for learning concepts. These adapters are model-specific and require retraining for different architectures, such as Stable Diffusion (SD) v1.5 and SD-XL. In this paper, we propose a straightforward textual inversion method to learn concepts through text embeddings, which are generalizable across models that share the same text encoder, including different versions of the SD model. We refer to our method as Prompt Sliders. Besides learning new concepts, we also show that Prompt Sliders can be used to erase undesirable concepts such as artistic styles or mature content. Our method is 30% faster than using LoRAs because it eliminates the need to load and unload adapters and introduces no additional parameters aside from the target concept text embedding. Each concept embedding only requires 3KB of storage compared to the 8922KB or more required for each LoRA adapter, making our approach more computationally efficient. Project Page: this https URL

Title: Monge-Kantorovich Fitting With Sobolev Budgets

• Authors: Forest Kobayashi, Jonathan Hayase, Young-Heon Kim
• Subjects: cs.LG, math.AP
• Abstract URL: https://arxiv.org/abs/2409.16541
• Pdf URL: https://arxiv.org/pdf/2409.16541
• Copy Paste: [[2409.16541]] Monge-Kantorovich Fitting With Sobolev Budgets(https://arxiv.org/abs/2409.16541)
• Keywords: generative
• Abstract: We consider the problem of finding the ``best'' approximation of an $n$-dimensional probability measure $\rho$ using a measure $\nu$ whose support is parametrized by $f : \mathbb{R}^m \to \mathbb{R}^n$ where $m < n$. We quantify the performance of the approximation with the Monge-Kantorovich $p$-cost (also called the Wasserstein $p$-cost) $\mathbb{W}_p^p(\rho, \nu)$, and constrain the complexity of the approximation by bounding the $W^{k,q}$ Sobolev norm of $f$, which acts as a ``budget.'' We may then reformulate the problem as minimizing a functional $\mathscr{J}_p(f)$ under a constraint on the Sobolev budget. We treat general $k \geq 1$ for the Sobolev differentiability order (though $q, m$ are chosen to restrict $W^{k,q}$ to the supercritical regime $k q > m$ to guarantee existence of optimizers). The problem is closely related to (but distinct from) principal curves with length constraints when $m=1, k = 1$ and smoothing splines when $k > 1$. New aspects and challenges arise from the higher order differentiability condition. We study the gradient of $\mathscr{J}_p$, which is given by a vector field along $f$ we call the barycenter field. We use it to construct improvements to a given $f$, which gives a nontrivial (almost) strict monotonicty relation between the functional $\mathscr{J}_p$ and the Sobolev budget. We also provide a natural discretization scheme and establish its consistency. We use this scheme to model a generative learning task; in particular, we demonstrate that adding a constraint like ours as a soft penalty yields substantial improvement in training a GAN to produce images of handwritten digits, with performance competitive with weight-decay.

Title: EMIT- Event-Based Masked Auto Encoding for Irregular Time Series

• Authors: Hrishikesh Patel, Ruihong Qiu, Adam Irwin, Shazia Sadiq, Sen Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16554
• Pdf URL: https://arxiv.org/pdf/2409.16554
• Copy Paste: [[2409.16554]] EMIT- Event-Based Masked Auto Encoding for Irregular Time Series(https://arxiv.org/abs/2409.16554)
• Keywords: self-supervised
• Abstract: Irregular time series, where data points are recorded at uneven intervals, are prevalent in healthcare settings, such as emergency wards where vital signs and laboratory results are captured at varying times. This variability, which reflects critical fluctuations in patient health, is essential for informed clinical decision-making. Existing self-supervised learning research on irregular time series often relies on generic pretext tasks like forecasting, which may not fully utilise the signal provided by irregular time series. There is a significant need for specialised pretext tasks designed for the characteristics of irregular time series to enhance model performance and robustness, especially in scenarios with limited data availability. This paper proposes a novel pretraining framework, EMIT, an event-based masking for irregular time series. EMIT focuses on masking-based reconstruction in the latent space, selecting masking points based on the rate of change in the data. This method preserves the natural variability and timing of measurements while enhancing the model's ability to process irregular intervals without losing essential information. Extensive experiments on the MIMIC-III and PhysioNet Challenge datasets demonstrate the superior performance of our event-based masking strategy. The code has been released at this https URL .

Title: EventHallusion: Diagnosing Event Hallucinations in Video LLMs

• Authors: Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, Yu-Gang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16597
• Pdf URL: https://arxiv.org/pdf/2409.16597
• Copy Paste: [[2409.16597]] EventHallusion: Diagnosing Event Hallucinations in Video LLMs(https://arxiv.org/abs/2409.16597)
• Keywords: foundation model
• Abstract: Recently, Multimodal Large Language Models (MLLMs) have made significant progress in the video comprehension field. Despite remarkable content reasoning and instruction following capabilities they demonstrated, the hallucination problem of these VideoLLMs is less explored compared with its counterpart in the image domain. To mitigate this gap, we first propose EventHallusion, a novel benchmark that focuses on assessing the VideoLMMs' hallucination phenomenon on video event comprehension. Based on the observation that existing VideoLLMs are entangled with the priors stemming from their foundation models, our EventHallusion is curated by meticulously collecting videos and annotating questions to intentionally mislead the VideoLLMs into interpreting events based on these priors rather than accurately understanding the video content. On the other hand, we also propose a simple yet effective method, called Temporal Contrastive Decoding (TCD), to tackle the hallucination problems of VideoLLMs. The proposed TCD suppresses the model's preference toward their priors by comparing the original video with a constructed counterpart, whose temporal cues are disrupted, during the autoregressive decoding stage. Through comprehensive evaluation of eight open-source and two closed-source VideoLLMs on the proposed EventHallusion benchmark, we find that the open-source models suffer significantly from hallucination problems, whereas the closed-source models perform markedly better. By further equipping open-sourced VideoLLMs with the proposed TCD approach, evident performance improvements are achieved across most metrics in the EventHallusion benchmark. Our codes and benchmark data are available at this https URL.

Title: FAFA: Frequency-Aware Flow-Aided Self-Supervision for Underwater Object Pose Estimation

• Authors: Jingyi Tang, Gu Wang, Zeyu Chen, Shengquan Li, Xiu Li, Xiangyang Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16600
• Pdf URL: https://arxiv.org/pdf/2409.16600
• Copy Paste: [[2409.16600]] FAFA: Frequency-Aware Flow-Aided Self-Supervision for Underwater Object Pose Estimation(https://arxiv.org/abs/2409.16600)
• Keywords: self-supervised
• Abstract: Although methods for estimating the pose of objects in indoor scenes have achieved great success, the pose estimation of underwater objects remains challenging due to difficulties brought by the complex underwater environment, such as degraded illumination, blurring, and the substantial cost of obtaining real annotations. In response, we introduce FAFA, a Frequency-Aware Flow-Aided self-supervised framework for 6D pose estimation of unmanned underwater vehicles (UUVs). Essentially, we first train a frequency-aware flow-based pose estimator on synthetic data, where an FFT-based augmentation approach is proposed to facilitate the network in capturing domain-invariant features and target domain styles from a frequency perspective. Further, we perform self-supervised training by enforcing flow-aided multi-level consistencies to adapt it to the real-world underwater environment. Our framework relies solely on the 3D model and RGB images, alleviating the need for any real pose annotations or other-modality data like depths. We evaluate the effectiveness of FAFA on common underwater object pose benchmarks and showcase significant performance improvements compared to state-of-the-art methods. Code is available at this http URL.

Title: Functional Stochastic Gradient MCMC for Bayesian Neural Networks

• Authors: Mengjing Wu, Junyu Xuan, Jie Lu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16632
• Pdf URL: https://arxiv.org/pdf/2409.16632
• Copy Paste: [[2409.16632]] Functional Stochastic Gradient MCMC for Bayesian Neural Networks(https://arxiv.org/abs/2409.16632)
• Keywords: diffusion
• Abstract: Classical variational inference for Bayesian neural networks (BNNs) in parameter space usually suffers from unresolved prior issues such as knowledge encoding intractability and pathological behaviors in deep networks, which could lead to an improper posterior inference. Hence, functional variational inference has been proposed recently to resolve these issues via stochastic process priors. Beyond variational inference, stochastic gradient Markov Chain Monte Carlo (SGMCMC) is another scalable and effective inference method for BNNs to asymptotically generate samples from true posterior by simulating a continuous dynamic. However, the existing SGMCMC methods only work in parametric space, which has the same issues of parameter-space variational inference, and extending the parameter-space dynamics to function-space dynamics is not a trivial undertaking. In this paper, we introduce a new functional SGMCMC scheme via newly designed diffusion dynamics, which can incorporate more informative functional priors. Moreover, we prove that the stationary distribution of these functional dynamics is the target posterior distribution over functions. We demonstrate better performance in both accuracy and uncertainty quantification of our functional SGMCMC on several tasks compared with naive SGMCMC and functional variational inference methods.

Title: Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model

• Authors: Shoma Iwai, Atsuki Osanai, Shunsuke Kitada, Shinichiro Omachi
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16689
• Pdf URL: https://arxiv.org/pdf/2409.16689
• Copy Paste: [[2409.16689]] Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model(https://arxiv.org/abs/2409.16689)
• Keywords: diffusion
• Abstract: Layout generation is a task to synthesize a harmonious layout with elements characterized by attributes such as category, position, and size. Human designers experiment with the placement and modification of elements to create aesthetic layouts, however, we observed that current discrete diffusion models (DDMs) struggle to correct inharmonious layouts after they have been generated. In this paper, we first provide novel insights into layout sticking phenomenon in DDMs and then propose a simple yet effective layout-assessment module Layout-Corrector, which works in conjunction with existing DDMs to address the layout sticking problem. We present a learning-based module capable of identifying inharmonious elements within layouts, considering overall layout harmony characterized by complex composition. During the generation process, Layout-Corrector evaluates the correctness of each token in the generated layout, reinitializing those with low scores to the ungenerated state. The DDM then uses the high-scored tokens as clues to regenerate the harmonized tokens. Layout-Corrector, tested on common benchmarks, consistently boosts layout-generation performance when in conjunction with various state-of-the-art DDMs. Furthermore, our extensive analysis demonstrates that the Layout-Corrector (1) successfully identifies erroneous tokens, (2) facilitates control over the fidelity-diversity trade-off, and (3) significantly mitigates the performance drop associated with fast sampling.

Title: Pix2Next: Leveraging Vision Foundation Models for RGB to NIR Image Translation

• Authors: Youngwan Jin, Incheol Park, Hanbin Song, Hyeongjin Ju, Yagiz Nalcakan, Shiho Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16706
• Pdf URL: https://arxiv.org/pdf/2409.16706
• Copy Paste: [[2409.16706]] Pix2Next: Leveraging Vision Foundation Models for RGB to NIR Image Translation(https://arxiv.org/abs/2409.16706)
• Keywords: foundation model
• Abstract: This paper proposes Pix2Next, a novel image-to-image translation framework designed to address the challenge of generating high-quality Near-Infrared (NIR) images from RGB inputs. Our approach leverages a state-of-the-art Vision Foundation Model (VFM) within an encoder-decoder architecture, incorporating cross-attention mechanisms to enhance feature integration. This design captures detailed global representations and preserves essential spectral characteristics, treating RGB-to-NIR translation as more than a simple domain transfer problem. A multi-scale PatchGAN discriminator ensures realistic image generation at various detail levels, while carefully designed loss functions couple global context understanding with local feature preservation. We performed experiments on the RANUS dataset to demonstrate Pix2Next's advantages in quantitative metrics and visual quality, improving the FID score by 34.81% compared to existing methods. Furthermore, we demonstrate the practical utility of Pix2Next by showing improved performance on a downstream object detection task using generated NIR data to augment limited real NIR datasets. The proposed approach enables the scaling up of NIR datasets without additional data acquisition or annotation efforts, potentially accelerating advancements in NIR-based computer vision applications.

Title: A Few Hypocrites: Few-Shot Learning and Subtype Definitions for Detecting Hypocrisy Accusations in Online Climate Change Debates

• Authors: Paulina Garcia Corral, Avishai Green, Hendrik Meyer, Anke Stoll, Xiaoyue Yan, Myrthe Reuver
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16807
• Pdf URL: https://arxiv.org/pdf/2409.16807
• Copy Paste: [[2409.16807]] A Few Hypocrites: Few-Shot Learning and Subtype Definitions for Detecting Hypocrisy Accusations in Online Climate Change Debates(https://arxiv.org/abs/2409.16807)
• Keywords: in-context
• Abstract: The climate crisis is a salient issue in online discussions, and hypocrisy accusations are a central rhetorical element in these debates. However, for large-scale text analysis, hypocrisy accusation detection is an understudied tool, most often defined as a smaller subtask of fallacious argument detection. In this paper, we define hypocrisy accusation detection as an independent task in NLP, and identify different relevant subtypes of hypocrisy accusations. Our Climate Hypocrisy Accusation Corpus (CHAC) consists of 420 Reddit climate debate comments, expert-annotated into two different types of hypocrisy accusations: personal versus political hypocrisy. We evaluate few-shot in-context learning with 6 shots and 3 instruction-tuned Large Language Models (LLMs) for detecting hypocrisy accusations in this dataset. Results indicate that the GPT-4o and Llama-3 models in particular show promise in detecting hypocrisy accusations (F1 reaching 0.68, while previous work shows F1 of 0.44). However, context matters for a complex semantic concept such as hypocrisy accusations, and we find models struggle especially at identifying political hypocrisy accusations compared to personal moral hypocrisy. Our study contributes new insights in hypocrisy detection and climate change discourse, and is a stepping stone for large-scale analysis of hypocrisy accusation in online climate debates.

Title: A parametric framework for kernel-based dynamic mode decomposition using deep learning

• Authors: Konstantinos Kevopoulos, Dongwei Ye
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2409.16817
• Pdf URL: https://arxiv.org/pdf/2409.16817
• Copy Paste: [[2409.16817]] A parametric framework for kernel-based dynamic mode decomposition using deep learning(https://arxiv.org/abs/2409.16817)
• Keywords: diffusion
• Abstract: Surrogate modelling is widely applied in computational science and engineering to mitigate computational efficiency issues for the real-time simulations of complex and large-scale computational models or for many-query scenarios, such as uncertainty quantification and design optimisation. In this work, we propose a parametric framework for kernel-based dynamic mode decomposition method based on the linear and nonlinear disambiguation optimization (LANDO) algorithm. The proposed parametric framework consists of two stages, offline and online. The offline stage prepares the essential component for prediction, namely a series of LANDO models that emulate the dynamics of the system with particular parameters from a training dataset. The online stage leverages those LANDO models to generate new data at a desired time instant, and approximate the mapping between parameters and the state with the data using deep learning techniques. Moreover, dimensionality reduction technique is applied to high-dimensional dynamical systems to reduce the computational cost of training. Three numerical examples including Lotka-Volterra model, heat equation and reaction-diffusion equation are presented to demonstrate the efficiency and effectiveness of the proposed framework.

Title: XAI-guided Insulator Anomaly Detection for Imbalanced Datasets

• Authors: Maximilian Andreas Hoefler, Karsten Mueller, Wojciech Samek
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16821
• Pdf URL: https://arxiv.org/pdf/2409.16821
• Copy Paste: [[2409.16821]] XAI-guided Insulator Anomaly Detection for Imbalanced Datasets(https://arxiv.org/abs/2409.16821)
• Keywords: anomaly
• Abstract: Power grids serve as a vital component in numerous industries, seamlessly delivering electrical energy to industrial processes and technologies, making their safe and reliable operation indispensable. However, powerlines can be hard to inspect due to difficult terrain or harsh climatic conditions. Therefore, unmanned aerial vehicles are increasingly deployed to inspect powerlines, resulting in a substantial stream of visual data which requires swift and accurate processing. Deep learning methods have become widely popular for this task, proving to be a valuable asset in fault detection. In particular, the detection of insulator defects is crucial for predicting powerline failures, since their malfunction can lead to transmission disruptions. It is therefore of great interest to continuously maintain and rigorously inspect insulator components. In this work we propose a novel pipeline to tackle this task. We utilize state-of-the-art object detection to detect and subsequently classify individual insulator anomalies. Our approach addresses dataset challenges such as imbalance and motion-blurred images through a fine-tuning methodology which allows us to alter the classification focus of the model by increasing the classification accuracy of anomalous insulators. In addition, we employ explainable-AI tools for precise localization and explanation of anomalies. This proposed method contributes to the field of anomaly detection, particularly vision-based industrial inspection and predictive maintenance. We significantly improve defect detection accuracy by up to 13%, while also offering a detailed analysis of model mis-classifications and localization quality, showcasing the potential of our method on real-world data.

Title: Robust Scene Change Detection Using Visual Foundation Models and Cross-Attention Mechanisms

• Authors: Chun-Jung Lin, Sourav Garg, Tat-Jun Chin, Feras Dayoub
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16850
• Pdf URL: https://arxiv.org/pdf/2409.16850
• Copy Paste: [[2409.16850]] Robust Scene Change Detection Using Visual Foundation Models and Cross-Attention Mechanisms(https://arxiv.org/abs/2409.16850)
• Keywords: foundation model
• Abstract: We present a novel method for scene change detection that leverages the robust feature extraction capabilities of a visual foundational model, DINOv2, and integrates full-image cross-attention to address key challenges such as varying lighting, seasonal variations, and viewpoint differences. In order to effectively learn correspondences and mis-correspondences between an image pair for the change detection task, we propose to a) ``freeze'' the backbone in order to retain the generality of dense foundation features, and b) employ ``full-image'' cross-attention to better tackle the viewpoint variations between the image pair. We evaluate our approach on two benchmark datasets, VL-CMU-CD and PSCD, along with their viewpoint-varied versions. Our experiments demonstrate significant improvements in F1-score, particularly in scenarios involving geometric changes between image pairs. The results indicate our method's superior generalization capabilities over existing state-of-the-art approaches, showing robustness against photometric and geometric variations as well as better overall generalization when fine-tuned to adapt to new environments. Detailed ablation studies further validate the contributions of each component in our architecture. Source code will be made publicly available upon acceptance.

Title: A Versatile and Differentiable Hand-Object Interaction Representation

• Authors: Théo Morales, Omid Taheri, Gerard Lacey
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16855
• Pdf URL: https://arxiv.org/pdf/2409.16855
• Copy Paste: [[2409.16855]] A Versatile and Differentiable Hand-Object Interaction Representation(https://arxiv.org/abs/2409.16855)
• Keywords: diffusion
• Abstract: Synthesizing accurate hands-object interactions (HOI) is critical for applications in Computer Vision, Augmented Reality (AR), and Mixed Reality (MR). Despite recent advances, the accuracy of reconstructed or generated HOI leaves room for refinement. Some techniques have improved the accuracy of dense correspondences by shifting focus from generating explicit contacts to using rich HOI fields. Still, they lack full differentiability or continuity and are tailored to specific tasks. In contrast, we present a Coarse Hand-Object Interaction Representation (CHOIR), a novel, versatile and fully differentiable field for HOI modelling. CHOIR leverages discrete unsigned distances for continuous shape and pose encoding, alongside multivariate Gaussian distributions to represent dense contact maps with few parameters. To demonstrate the versatility of CHOIR we design JointDiffusion, a diffusion model to learn a grasp distribution conditioned on noisy hand-object interactions or only object geometries, for both refinement and synthesis applications. We demonstrate JointDiffusion's improvements over the SOTA in both applications: it increases the contact F1 score by $5\%$ for refinement and decreases the sim. displacement by $46\%$ for synthesis. Our experiments show that JointDiffusion with CHOIR yield superior contact accuracy and physical realism compared to SOTA methods designed for specific tasks. Our models and code will be publicly available to the research community.

Title: Towards Unified 3D Hair Reconstruction from Single-View Portraits

• Authors: Yujian Zheng, Yuda Qiu, Leyang Jin, Chongyang Ma, Haibin Huang, Di Zhang, Pengfei Wan, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16863
• Pdf URL: https://arxiv.org/pdf/2409.16863
• Copy Paste: [[2409.16863]] Towards Unified 3D Hair Reconstruction from Single-View Portraits(https://arxiv.org/abs/2409.16863)
• Keywords: diffusion
• Abstract: Single-view 3D hair reconstruction is challenging, due to the wide range of shape variations among diverse hairstyles. Current state-of-the-art methods are specialized in recovering un-braided 3D hairs and often take braided styles as their failure cases, because of the inherent difficulty to define priors for complex hairstyles, whether rule-based or data-based. We propose a novel strategy to enable single-view 3D reconstruction for a variety of hair types via a unified pipeline. To achieve this, we first collect a large-scale synthetic multi-view hair dataset SynMvHair with diverse 3D hair in both braided and un-braided styles, and learn two diffusion priors specialized on hair. Then we optimize 3D Gaussian-based hair from the priors with two specially designed modules, i.e. view-wise and pixel-wise Gaussian refinement. Our experiments demonstrate that reconstructing braided and un-braided 3D hair from single-view images via a unified approach is possible and our method achieves the state-of-the-art performance in recovering complex hairstyles. It is worth to mention that our method shows good generalization ability to real images, although it learns hair priors from synthetic data.

Title: Linking in Style: Understanding learned features in deep learning models

• Authors: Maren H. Wehrheim, Pamela Osuna-Vargas, Matthias Kaschube
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16865
• Pdf URL: https://arxiv.org/pdf/2409.16865
• Copy Paste: [[2409.16865]] Linking in Style: Understanding learned features in deep learning models(https://arxiv.org/abs/2409.16865)
• Keywords: generative
• Abstract: Convolutional neural networks (CNNs) learn abstract features to perform object classification, but understanding these features remains challenging due to difficult-to-interpret results or high computational costs. We propose an automatic method to visualize and systematically analyze learned features in CNNs. Specifically, we introduce a linking network that maps the penultimate layer of a pre-trained classifier to the latent space of a generative model (StyleGAN-XL), thereby enabling an interpretable, human-friendly visualization of the classifier's representations. Our findings indicate a congruent semantic order in both spaces, enabling a direct linear mapping between them. Training the linking network is computationally inexpensive and decoupled from training both the GAN and the classifier. We introduce an automatic pipeline that utilizes such GAN-based visualizations to quantify learned representations by analyzing activation changes in the classifier in the image domain. This quantification allows us to systematically study the learned representations in several thousand units simultaneously and to extract and visualize units selective for specific semantic concepts. Further, we illustrate how our method can be used to quantify and interpret the classifier's decision boundary using counterfactual examples. Overall, our method offers systematic and objective perspectives on learned abstract representations in CNNs. this https URL

Title: Towards Underwater Camouflaged Object Tracking: An Experimental Evaluation of SAM and SAM 2

• Authors: Chunhui Zhang, Li Liu, Guanjie Huang, Hao Wen, Xi Zhou, Yanfeng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16902
• Pdf URL: https://arxiv.org/pdf/2409.16902
• Copy Paste: [[2409.16902]] Towards Underwater Camouflaged Object Tracking: An Experimental Evaluation of SAM and SAM 2(https://arxiv.org/abs/2409.16902)
• Keywords: foundation model
• Abstract: Over the past decade, significant progress has been made in visual object tracking, largely due to the availability of large-scale training datasets. However, existing tracking datasets are primarily focused on open-air scenarios, which greatly limits the development of object tracking in underwater environments. To address this issue, we take a step forward by proposing the first large-scale underwater camouflaged object tracking dataset, namely UW-COT. Based on the proposed dataset, this paper presents an experimental evaluation of several advanced visual object tracking methods and the latest advancements in image and video segmentation. Specifically, we compare the performance of the Segment Anything Model (SAM) and its updated version, SAM 2, in challenging underwater environments. Our findings highlight the improvements in SAM 2 over SAM, demonstrating its enhanced capability to handle the complexities of underwater camouflaged objects. Compared to current advanced visual object tracking methods, the latest video segmentation foundation model SAM 2 also exhibits significant advantages, providing valuable insights into the development of more effective tracking technologies for underwater scenarios. The dataset will be accessible at \color{magenta}{this https URL}.

Title: Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Model

• Authors: Hongliang Zhong, Can Wang, Jingbo Zhang, Jing Liao
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.16938
• Pdf URL: https://arxiv.org/pdf/2409.16938
• Copy Paste: [[2409.16938]] Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Model(https://arxiv.org/abs/2409.16938)
• Keywords: diffusion, generative
• Abstract: Generating and inserting new objects into 3D content is a compelling approach for achieving versatile scene recreation. Existing methods, which rely on SDS optimization or single-view inpainting, often struggle to produce high-quality results. To address this, we propose a novel method for object insertion in 3D content represented by Gaussian Splatting. Our approach introduces a multi-view diffusion model, dubbed MVInpainter, which is built upon a pre-trained stable video diffusion model to facilitate view-consistent object inpainting. Within MVInpainter, we incorporate a ControlNet-based conditional injection module to enable controlled and more predictable multi-view generation. After generating the multi-view inpainted results, we further propose a mask-aware 3D reconstruction technique to refine Gaussian Splatting reconstruction from these sparse inpainted views. By leveraging these fabricate techniques, our approach yields diverse results, ensures view-consistent and harmonious insertions, and produces better object quality. Extensive experiments demonstrate that our approach outperforms existing methods.

Title: Face Forgery Detection with Elaborate Backbone

• Authors: Zonghui Guo, Yingjie Liu, Jie Zhang, Haiyong Zheng, Shiguang Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16945
• Pdf URL: https://arxiv.org/pdf/2409.16945
• Copy Paste: [[2409.16945]] Face Forgery Detection with Elaborate Backbone(https://arxiv.org/abs/2409.16945)
• Keywords: self-supervised
• Abstract: Face Forgery Detection (FFD), or Deepfake detection, aims to determine whether a digital face is real or fake. Due to different face synthesis algorithms with diverse forgery patterns, FFD models often overfit specific patterns in training datasets, resulting in poor generalization to other unseen forgeries. This severe challenge requires FFD models to possess strong capabilities in representing complex facial features and extracting subtle forgery cues. Although previous FFD models directly employ existing backbones to represent and extract facial forgery cues, the critical role of backbones is often overlooked, particularly as their knowledge and capabilities are insufficient to address FFD challenges, inevitably limiting generalization. Therefore, it is essential to integrate the backbone pre-training configurations and seek practical solutions by revisiting the complete FFD workflow, from backbone pre-training and fine-tuning to inference of discriminant results. Specifically, we analyze the crucial contributions of backbones with different configurations in FFD task and propose leveraging the ViT network with self-supervised learning on real-face datasets to pre-train a backbone, equipping it with superior facial representation capabilities. We then build a competitive backbone fine-tuning framework that strengthens the backbone's ability to extract diverse forgery cues within a competitive learning mechanism. Moreover, we devise a threshold optimization mechanism that utilizes prediction confidence to improve the inference reliability. Comprehensive experiments demonstrate that our FFD model with the elaborate backbone achieves excellent performance in FFD and extra face-related tasks, i.e., presentation attack detection. Code and models are available at this https URL.

Title: DALDA: Data Augmentation Leveraging Diffusion Model and LLM with Adaptive Guidance Scaling

• Authors: Kyuheon Jung, Yongdeuk Seo, Seongwoo Cho, Jaeyoung Kim, Hyun-seok Min, Sungchul Choi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16949
• Pdf URL: https://arxiv.org/pdf/2409.16949
• Copy Paste: [[2409.16949]] DALDA: Data Augmentation Leveraging Diffusion Model and LLM with Adaptive Guidance Scaling(https://arxiv.org/abs/2409.16949)
• Keywords: diffusion
• Abstract: In this paper, we present an effective data augmentation framework leveraging the Large Language Model (LLM) and Diffusion Model (DM) to tackle the challenges inherent in data-scarce scenarios. Recently, DMs have opened up the possibility of generating synthetic images to complement a few training images. However, increasing the diversity of synthetic images also raises the risk of generating samples outside the target distribution. Our approach addresses this issue by embedding novel semantic information into text prompts via LLM and utilizing real images as visual prompts, thus generating semantically rich images. To ensure that the generated images remain within the target distribution, we dynamically adjust the guidance weight based on each image's CLIPScore to control the diversity. Experimental results show that our method produces synthetic images with enhanced diversity while maintaining adherence to the target distribution. Consequently, our approach proves to be more efficient in the few-shot setting on several benchmarks. Our code is available at this https URL .

Title: Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization

• Authors: Rafael Mendoza, Isabella Cruz, Richard Liu, Aarav Deshmukh, David Williams, Jesscia Peng, Rohan Iyer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16973
• Pdf URL: https://arxiv.org/pdf/2409.16973
• Copy Paste: [[2409.16973]] Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization(https://arxiv.org/abs/2409.16973)
• Keywords: self-supervised
• Abstract: Large language models (LLMs) have revolutionized how we interact with technology, but their personalization to individual user preferences remains a significant challenge, particularly in on-device applications. Traditional methods often depend heavily on labeled datasets and can be resource-intensive. To address these issues, we present Adaptive Self-Supervised Learning Strategies (ASLS), which utilizes self-supervised learning techniques to personalize LLMs dynamically. The framework comprises a user profiling layer for collecting interaction data and a neural adaptation layer for real-time model fine-tuning. This innovative approach enables continuous learning from user feedback, allowing the model to generate responses that align closely with user-specific contexts. The adaptive mechanisms of ASLS minimize computational demands and enhance personalization efficiency. Experimental results across various user scenarios illustrate the superior performance of ASLS in boosting user engagement and satisfaction, highlighting its potential to redefine LLMs as highly responsive and context-aware systems on-device.

Title: Single Image, Any Face: Generalisable 3D Face Generation

• Authors: Wenqing Wang, Haosen Yang, Josef Kittler, Xiatian Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16990
• Pdf URL: https://arxiv.org/pdf/2409.16990
• Copy Paste: [[2409.16990]] Single Image, Any Face: Generalisable 3D Face Generation(https://arxiv.org/abs/2409.16990)
• Keywords: diffusion, generative
• Abstract: The creation of 3D human face avatars from a single unconstrained image is a fundamental task that underlies numerous real-world vision and graphics applications. Despite the significant progress made in generative models, existing methods are either less suited in design for human faces or fail to generalise from the restrictive training domain to unconstrained facial images. To address these limitations, we propose a novel model, Gen3D-Face, which generates 3D human faces with unconstrained single image input within a multi-view consistent diffusion framework. Given a specific input image, our model first produces multi-view images, followed by neural surface construction. To incorporate face geometry information in a generalisable manner, we utilise input-conditioned mesh estimation instead of ground-truth mesh along with synthetic multi-view training data. Importantly, we introduce a multi-view joint generation scheme to enhance appearance consistency among different views. To the best of our knowledge, this is the first attempt and benchmark for creating photorealistic 3D human face avatars from single images for generic human subject across domains. Extensive experiments demonstrate the superiority of our method over previous alternatives for out-of-domain singe image 3D face generation and top competition for in-domain setting.

Title: How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not

• Authors: Francesco Verdini, Pierfrancesco Melucci, Stefano Perna, Francesco Cariaggi, Marco Gaido, Sara Papi, Szymon Mazurek, Marek Kasztelnik, Luisa Bentivogli, Sébastien Bratières, Paolo Merialdo, Simone Scardapane
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17044
• Pdf URL: https://arxiv.org/pdf/2409.17044
• Copy Paste: [[2409.17044]] How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not(https://arxiv.org/abs/2409.17044)
• Keywords: foundation model
• Abstract: The remarkable performance achieved by Large Language Models (LLM) has driven research efforts to leverage them for a wide range of tasks and input modalities. In speech-to-text (S2T) tasks, the emerging solution consists of projecting the output of the encoder of a Speech Foundational Model (SFM) into the LLM embedding space through an adapter module. However, no work has yet investigated how much the downstream-task performance depends on each component (SFM, adapter, LLM) nor whether the best design of the adapter depends on the chosen SFM and LLM. To fill this gap, we evaluate the combination of 5 adapter modules, 2 LLMs (Mistral and Llama), and 2 SFMs (Whisper and SeamlessM4T) on two widespread S2T tasks, namely Automatic Speech Recognition and Speech Translation. Our results demonstrate that the SFM plays a pivotal role in downstream performance, while the adapter choice has moderate impact and depends on the SFM and LLM.

Title: GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Design

• Authors: Phillip Mueller, Sebastian Mueller, Lars Mikelsons
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17045
• Pdf URL: https://arxiv.org/pdf/2409.17045
• Copy Paste: [[2409.17045]] GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Design(https://arxiv.org/abs/2409.17045)
• Keywords: diffusion, foundation model, generative
• Abstract: We provide a dataset for enabling Deep Generative Models (DGMs) in engineering design and propose methods to automate data labeling by utilizing large-scale foundation models. GeoBiked is curated to contain 4 355 bicycle images, annotated with structural and technical features and is used to investigate two automated labeling techniques: The utilization of consolidated latent features (Hyperfeatures) from image-generation models to detect geometric correspondences (e.g. the position of the wheel center) in structural images and the generation of diverse text descriptions for structural images. GPT-4o, a vision-language-model (VLM), is instructed to analyze images and produce diverse descriptions aligned with the system-prompt. By representing technical images as Diffusion-Hyperfeatures, drawing geometric correspondences between them is possible. The detection accuracy of geometric points in unseen samples is improved by presenting multiple annotated source images. GPT-4o has sufficient capabilities to generate accurate descriptions of technical images. Grounding the generation only on images leads to diverse descriptions but causes hallucinations, while grounding it on categorical labels restricts the diversity. Using both as input balances creativity and accuracy. Successfully using Hyperfeatures for geometric correspondence suggests that this approach can be used for general point-detection and annotation tasks in technical images. Labeling such images with text descriptions using VLMs is possible, but dependent on the models detection capabilities, careful prompt-engineering and the selection of input information. Applying foundation models in engineering design is largely unexplored. We aim to bridge this gap with a dataset to explore training, finetuning and conditioning DGMs in this field and suggesting approaches to bootstrap foundation models to process technical images.

Title: ControlCity: A Multimodal Diffusion Model Based Approach for Accurate Geospatial Data Generation and Urban Morphology Analysis

• Authors: Fangshuo Zhou, Huaxia Li, Rui Hu, Sensen Wu, Hailin Feng, Zhenhong Du, Liuchang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17049
• Pdf URL: https://arxiv.org/pdf/2409.17049
• Copy Paste: [[2409.17049]] ControlCity: A Multimodal Diffusion Model Based Approach for Accurate Geospatial Data Generation and Urban Morphology Analysis(https://arxiv.org/abs/2409.17049)
• Keywords: diffusion
• Abstract: Volunteer Geographic Information (VGI), with its rich variety, large volume, rapid updates, and diverse sources, has become a critical source of geospatial data. However, VGI data from platforms like OSM exhibit significant quality heterogeneity across different data types, particularly with urban building data. To address this, we propose a multi-source geographic data transformation solution, utilizing accessible and complete VGI data to assist in generating urban building footprint data. We also employ a multimodal data generation framework to improve accuracy. First, we introduce a pipeline for constructing an 'image-text-metadata-building footprint' dataset, primarily based on road network data and supplemented by other multimodal data. We then present ControlCity, a geographic data transformation method based on a multimodal diffusion model. This method first uses a pre-trained text-to-image model to align text, metadata, and building footprint data. An improved ControlNet further integrates road network and land-use imagery, producing refined building footprint data. Experiments across 22 global cities demonstrate that ControlCity successfully simulates real urban building patterns, achieving state-of-the-art performance. Specifically, our method achieves an average FID score of 50.94, reducing error by 71.01% compared to leading methods, and a MIoU score of 0.36, an improvement of 38.46%. Additionally, our model excels in tasks like urban morphology transfer, zero-shot city generation, and spatial data completeness assessment. In the zero-shot city task, our method accurately predicts and generates similar urban structures, demonstrating strong generalization. This study confirms the effectiveness of our approach in generating urban building footprint data and capturing complex city characteristics.

Title: Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors

• Authors: Aiping Zhang, Zongsheng Yue, Renjing Pei, Wenqi Ren, Xiaochun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17058
• Pdf URL: https://arxiv.org/pdf/2409.17058
• Copy Paste: [[2409.17058]] Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors(https://arxiv.org/abs/2409.17058)
• Keywords: diffusion, generative
• Abstract: Diffusion-based image super-resolution (SR) methods have achieved remarkable success by leveraging large pre-trained text-to-image diffusion models as priors. However, these methods still face two challenges: the requirement for dozens of sampling steps to achieve satisfactory results, which limits efficiency in real scenarios, and the neglect of degradation models, which are critical auxiliary information in solving the SR problem. In this work, we introduced a novel one-step SR model, which significantly addresses the efficiency issue of diffusion-based SR methods. Unlike existing fine-tuning strategies, we designed a degradation-guided Low-Rank Adaptation (LoRA) module specifically for SR, which corrects the model parameters based on the pre-estimated degradation information from low-resolution images. This module not only facilitates a powerful data-dependent or degradation-dependent SR model but also preserves the generative prior of the pre-trained diffusion model as much as possible. Furthermore, we tailor a novel training pipeline by introducing an online negative sample generation strategy. Combined with the classifier-free guidance strategy during inference, it largely improves the perceptual quality of the super-resolution results. Extensive experiments have demonstrated the superior efficiency and effectiveness of the proposed model compared to recent state-of-the-art methods.

Title: Benchmarking Domain Generalization Algorithms in Computational Pathology

• Authors: Neda Zamanitajeddin, Mostafa Jahanifar, Kesi Xu, Fouzia Siraj, Nasir Rajpoot
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17063
• Pdf URL: https://arxiv.org/pdf/2409.17063
• Copy Paste: [[2409.17063]] Benchmarking Domain Generalization Algorithms in Computational Pathology(https://arxiv.org/abs/2409.17063)
• Keywords: self-supervised, foundation model
• Abstract: Deep learning models have shown immense promise in computational pathology (CPath) tasks, but their performance often suffers when applied to unseen data due to domain shifts. Addressing this requires domain generalization (DG) algorithms. However, a systematic evaluation of DG algorithms in the CPath context is lacking. This study aims to benchmark the effectiveness of 30 DG algorithms on 3 CPath tasks of varying difficulty through 7,560 cross-validation runs. We evaluate these algorithms using a unified and robust platform, incorporating modality-specific techniques and recent advances like pretrained foundation models. Our extensive cross-validation experiments provide insights into the relative performance of various DG strategies. We observe that self-supervised learning and stain augmentation consistently outperform other methods, highlighting the potential of pretrained models and data augmentation. Furthermore, we introduce a new pan-cancer tumor detection dataset (HISTOPANTUM) as a benchmark for future research. This study offers valuable guidance to researchers in selecting appropriate DG approaches for CPath tasks.

Title: Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition

• Authors: Pritika Ramu, Koustava Goswami, Apoorv Saxena, Balaji Vasan Srinivavsan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17073
• Pdf URL: https://arxiv.org/pdf/2409.17073
• Copy Paste: [[2409.17073]] Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition(https://arxiv.org/abs/2409.17073)
• Keywords: in-context
• Abstract: Accurately attributing answer text to its source document is crucial for developing a reliable question-answering system. However, attribution for long documents remains largely unexplored. Post-hoc attribution systems are designed to map answer text back to the source document, yet the granularity of this mapping has not been addressed. Furthermore, a critical question arises: What precisely should be attributed, with an emphasis on identifying the information units within an answer that necessitate grounding? In this paper, we propose and investigate a novel approach to the factual decomposition of generated answers for attribution, employing template-based in-context learning. To accomplish this, we utilize the question and integrate negative sampling during few-shot in-context learning for decomposition. This approach enhances the semantic understanding of both abstractive and extractive answers. We examine the impact of answer decomposition by providing a thorough examination of various attribution approaches, ranging from retrieval-based techniques to LLM-based attributors.

Title: Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?

• Authors: Bowen Zhao, Leo Parker Dirac, Paulina Varshavskaya
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17080
• Pdf URL: https://arxiv.org/pdf/2409.17080
• Copy Paste: [[2409.17080]] Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?(https://arxiv.org/abs/2409.17080)
• Keywords: in-context
• Abstract: Large vision-language models (VLMs) have become state-of-the-art for many computer vision tasks, with in-context learning (ICL) as a popular adaptation strategy for new ones. But can VLMs learn novel concepts purely from visual demonstrations, or are they limited to adapting to the output format of ICL examples? We propose a new benchmark we call Spatial Visual Ambiguity Tasks (SVAT) that challenges state-of-the-art VLMs to learn new visuospatial tasks in-context. We find that VLMs fail to do this zero-shot, and sometimes continue to fail after finetuning. However, adding simpler data to the training by curriculum learning leads to improved ICL performance.

Title: Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classification

• Authors: Xinrui Zhou, Yuhao Huang, Haoran Dou, Shijing Chen, Ao Chang, Jia Liu, Weiran Long, Jian Zheng, Erjiao Xu, Jie Ren, Ruobing Huang, Jun Cheng, Wufeng Xue, Dong Ni
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17091
• Pdf URL: https://arxiv.org/pdf/2409.17091
• Copy Paste: [[2409.17091]] Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classification(https://arxiv.org/abs/2409.17091)
• Keywords: diffusion, generative
• Abstract: In the medical field, the limited availability of large-scale datasets and labor-intensive annotation processes hinder the performance of deep models. Diffusion-based generative augmentation approaches present a promising solution to this issue, having been proven effective in advancing downstream medical recognition tasks. Nevertheless, existing works lack sufficient semantic and sequential steerability for challenging video/3D sequence generation, and neglect quality control of noisy synthesized samples, resulting in unreliable synthetic databases and severely limiting the performance of downstream tasks. In this work, we present Ctrl-GenAug, a novel and general generative augmentation framework that enables highly semantic- and sequential-customized sequence synthesis and suppresses incorrectly synthesized samples, to aid medical sequence classification. Specifically, we first design a multimodal conditions-guided sequence generator for controllably synthesizing diagnosis-promotive samples. A sequential augmentation module is integrated to enhance the temporal/stereoscopic coherence of generated samples. Then, we propose a noisy synthetic data filter to suppress unreliable cases at semantic and sequential levels. Extensive experiments on 3 medical datasets, using 11 networks trained on 3 paradigms, comprehensively analyze the effectiveness and generality of Ctrl-GenAug, particularly in underrepresented high-risk populations and out-domain conditions.

Title: Unveiling Ontological Commitment in Multi-Modal Foundation Models

• Authors: Mert Keser, Gesina Schwalbe, Niki Amini-Naieni, Matthias Rottmann, Alois Knoll
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17109
• Pdf URL: https://arxiv.org/pdf/2409.17109
• Copy Paste: [[2409.17109]] Unveiling Ontological Commitment in Multi-Modal Foundation Models(https://arxiv.org/abs/2409.17109)
• Keywords: foundation model
• Abstract: Ontological commitment, i.e., used concepts, relations, and assumptions, are a corner stone of qualitative reasoning (QR) models. The state-of-the-art for processing raw inputs, though, are deep neural networks (DNNs), nowadays often based off from multimodal foundation models. These automatically learn rich representations of concepts and respective reasoning. Unfortunately, the learned qualitative knowledge is opaque, preventing easy inspection, validation, or adaptation against available QR models. So far, it is possible to associate pre-defined concepts with latent representations of DNNs, but extractable relations are mostly limited to semantic similarity. As a next step towards QR for validation and verification of DNNs: Concretely, we propose a method that extracts the learned superclass hierarchy from a multimodal DNN for a given set of leaf concepts. Under the hood we (1) obtain leaf concept embeddings using the DNN's textual input modality; (2) apply hierarchical clustering to them, using that DNNs encode semantic similarities via vector distances; and (3) label the such-obtained parent concepts using search in available ontologies from QR. An initial evaluation study shows that meaningful ontological class hierarchies can be extracted from state-of-the-art foundation models. Furthermore, we demonstrate how to validate and verify a DNN's learned representations against given ontologies. Lastly, we discuss potential future applications in the context of QR.

Title: DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion

• Authors: Yukun Huang, Jianan Wang, Ailing Zeng, Zheng-Jun Zha, Lei Zhang, Xihui Liu
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17145
• Pdf URL: https://arxiv.org/pdf/2409.17145
• Copy Paste: [[2409.17145]] DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion(https://arxiv.org/abs/2409.17145)
• Keywords: diffusion
• Abstract: Leveraging pretrained 2D diffusion models and score distillation sampling (SDS), recent methods have shown promising results for text-to-3D avatar generation. However, generating high-quality 3D avatars capable of expressive animation remains challenging. In this work, we present DreamWaltz-G, a novel learning framework for animatable 3D avatar generation from text. The core of this framework lies in Skeleton-guided Score Distillation and Hybrid 3D Gaussian Avatar representation. Specifically, the proposed skeleton-guided score distillation integrates skeleton controls from 3D human templates into 2D diffusion models, enhancing the consistency of SDS supervision in terms of view and human pose. This facilitates the generation of high-quality avatars, mitigating issues such as multiple faces, extra limbs, and blurring. The proposed hybrid 3D Gaussian avatar representation builds on the efficient 3D Gaussians, combining neural implicit fields and parameterized 3D meshes to enable real-time rendering, stable SDS optimization, and expressive animation. Extensive experiments demonstrate that DreamWaltz-G is highly effective in generating and animating 3D avatars, outperforming existing methods in both visual quality and animation expressiveness. Our framework further supports diverse applications, including human video reenactment and multi-subject scene composition.