2024-09-24

Title: A Knowledge-Centric Benchmarking Framework and Empirical Study for Retrieval-Augmented Generation

• Authors: Shuo Yu (1 and 2), Mingyue Cheng (1 and 2), Jiqian Yang (1 and 2), Jie Ouyang (1 and 2) ((1) Anhui Province Key Laboratory of Big Data Analysis and Application, University of Science and Technology of China (2) State Key Laboratory of Cognitive Intelligence)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13694
• Pdf URL: https://arxiv.org/pdf/2409.13694
• Copy Paste: [[2409.13694]] A Knowledge-Centric Benchmarking Framework and Empirical Study for Retrieval-Augmented Generation(https://arxiv.org/abs/2409.13694)
• Keywords: generative
• Abstract: Retrieval-Augmented Generation (RAG) enhances generative models by integrating retrieval mechanisms, which allow these models to access and utilize external knowledge sources. Despite its advantages, RAG encounters significant challenges, particularly in effectively handling real-world queries and mitigating hallucinations. The KDD Cup 2024 CRAG competition brings these issues to the forefront by incorporating both web pages and a mock API as knowledge sources, adding the complexity of parsing HTML before large language models (LLMs) can process the information. In this paper, we propose a novel RAG benchmark designed to address these challenges. Our work provides a comprehensive set of experimental results, offering valuable insights for the study of RAG. We thoroughly examine the entire RAG process, including knowledge source selection, retrieval, organization, and reasoning. Key findings from our study include the impact of automated knowledge source selection using agents and the influence of noise chunks on RAG reasoning. Additionally, we conduct detailed experiments to analyze the effects of various hyperparameters on RAG performance. To support further research, we have made our results, the associated code, and a parsed version of the CRAG dataset publicly available\footnote{this https URL}, contributing to the advancement of RAG methodologies and establishing a solid foundation for future work in this domain.

Title: Towards Safe Multilingual Frontier AI

• Authors: Artūrs Kanepajs, Vladimir Ivanov, Richard Moulange
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.13708
• Pdf URL: https://arxiv.org/pdf/2409.13708
• Copy Paste: [[2409.13708]] Towards Safe Multilingual Frontier AI(https://arxiv.org/abs/2409.13708)
• Keywords: diffusion
• Abstract: Linguistically inclusive LLMs -- which maintain good performance regardless of the language with which they are prompted -- are necessary for the diffusion of AI benefits around the world. Multilingual jailbreaks that rely on language translation to evade safety measures undermine the safe and inclusive deployment of AI systems. We provide policy recommendations to enhance the multilingual capabilities of AI while mitigating the risks of multilingual jailbreaks. We quantitatively assess the relationship between language resourcedness and model vulnerabilities to multilingual jailbreaks for five frontier large language models across 24 official EU languages. Building on prior research, we propose policy actions that align with the EU legal landscape and institutional framework to address multilingual jailbreaks, while promoting linguistic inclusivity. These include mandatory assessments of multilingual capabilities and vulnerabilities, public opinion research, and state support for multilingual AI development. The measures aim to improve AI safety and functionality through EU policy initiatives, guiding the implementation of the EU AI Act and informing regulatory efforts of the European AI Office.

Title: Good Idea or Not, Representation of LLM Could Tell

• Authors: Yi Xu, Bo Xue, Shuqian Sheng, Cheng Deng, Jiaxin Ding, Zanwei Shen, Luoyi Fu, Xinbing Wang, Chenghu Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13712
• Pdf URL: https://arxiv.org/pdf/2409.13712
• Copy Paste: [[2409.13712]] Good Idea or Not, Representation of LLM Could Tell(https://arxiv.org/abs/2409.13712)
• Keywords: generative
• Abstract: In the ever-expanding landscape of academic research, the proliferation of ideas presents a significant challenge for researchers: discerning valuable ideas from the less impactful ones. The ability to efficiently evaluate the potential of these ideas is crucial for the advancement of science and paper review. In this work, we focus on idea assessment, which aims to leverage the knowledge of large language models to assess the merit of scientific ideas. First, we investigate existing text evaluation research and define the problem of quantitative evaluation of ideas. Second, we curate and release a benchmark dataset from nearly four thousand manuscript papers with full texts, meticulously designed to train and evaluate the performance of different approaches to this task. Third, we establish a framework for quantifying the value of ideas by employing representations in a specific layer of large language models. Experimental results show that the scores predicted by our method are relatively consistent with those of humans. Our findings suggest that the representations of large language models hold more potential in quantifying the value of ideas than their generative outputs, demonstrating a promising avenue for automating the idea assessment process.

Title: Identity-related Speech Suppression in Generative AI Content Moderation

• Authors: Oghenefejiro Isaacs Anigboro, Charlie M. Crawford, Danaë Metaxa, Sorelle A. Friedler
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.13725
• Pdf URL: https://arxiv.org/pdf/2409.13725
• Copy Paste: [[2409.13725]] Identity-related Speech Suppression in Generative AI Content Moderation(https://arxiv.org/abs/2409.13725)
• Keywords: generative
• Abstract: Automated content moderation has long been used to help identify and filter undesired user-generated content online. Generative AI systems now use such filters to keep undesired generated content from being created by or shown to users. From classrooms to Hollywood, as generative AI is increasingly used for creative or expressive text generation, whose stories will these technologies allow to be told, and whose will they suppress? In this paper, we define and introduce measures of speech suppression, focusing on speech related to different identity groups incorrectly filtered by a range of content moderation APIs. Using both short-form, user-generated datasets traditional in content moderation and longer generative AI-focused data, including two datasets we introduce in this work, we create a benchmark for measurement of speech suppression for nine identity groups. Across one traditional and four generative AI-focused automated content moderation services tested, we find that identity-related speech is more likely to be incorrectly suppressed than other speech except in the cases of a few non-marginalized groups. Additionally, we find differences between APIs in their abilities to correctly moderate generative AI content.

Title: Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Prompts

• Authors: Anna Mészáros, Szilvia Ujváry, Wieland Brendel, Patrik Reizinger, Ferenc Huszár
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.13728
• Pdf URL: https://arxiv.org/pdf/2409.13728
• Copy Paste: [[2409.13728]] Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Prompts(https://arxiv.org/abs/2409.13728)
• Keywords: in-context
• Abstract: LLMs show remarkable emergent abilities, such as inferring concepts from presumably out-of-distribution prompts, known as in-context learning. Though this success is often attributed to the Transformer architecture, our systematic understanding is limited. In complex real-world data sets, even defining what is out-of-distribution is not obvious. To better understand the OOD behaviour of autoregressive LLMs, we focus on formal languages, which are defined by the intersection of rules. We define a new scenario of OOD compositional generalization, termed rule extrapolation. Rule extrapolation describes OOD scenarios, where the prompt violates at least one rule. We evaluate rule extrapolation in formal languages with varying complexity in linear and recurrent architectures, the Transformer, and state space models to understand the architectures' influence on rule extrapolation. We also lay the first stones of a normative theory of rule extrapolation, inspired by the Solomonoff prior in algorithmic information theory.

Title: Table-to-Text Generation with Pretrained Diffusion Models

• Authors: Aleksei S. Krylov, Oleg D. Somov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13739
• Pdf URL: https://arxiv.org/pdf/2409.13739
• Copy Paste: [[2409.13739]] Table-to-Text Generation with Pretrained Diffusion Models(https://arxiv.org/abs/2409.13739)
• Keywords: diffusion
• Abstract: Diffusion models have demonstrated significant potential in achieving state-of-the-art performance across various text generation tasks. In this systematic study, we investigate their application to the table-to-text problem by adapting the diffusion model to the task and conducting an in-depth analysis. Our experiments cover multiple aspects of diffusion models training. We explore sampling strategy influence by inducing recent diffusion model accelerator DPM-Solver++ into our core model. We have tested different prediction aggregation methods, like ROVER and Minimum Bayes-Risk (MBR). Our studies cover the impact of the pre-training phase in diffusion models and the generation length constraints influence. We also have compared diffusion model generation with auto-regressive text-to-text models with different temperature settings for diversity evaluation. Our key observation is that diffusion models demonstrate the balance between quality and diversity while auto-regressive text-to-text models are not successful at handling both at the same time. Furthermore, we found out that to achieve the highest quality possible, it is preferable to use a regular sampler with the strictest length constraint to create multiple samples, and then use MBR to aggregate the predictions. However, if you are prepared to give up high level of diversity and to accelerate the process, you can also utilize a fast sampler DPM-Solver++. Our findings reveal that diffusion models achieve comparable results in the table-to-text domain, highlighting their viability in the table-to-text challenge as a promising research direction.

Title: Context-Aware Membership Inference Attacks against Pre-trained Large Language Models

• Authors: Hongyan Chang, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, Reza Shokri
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.13745
• Pdf URL: https://arxiv.org/pdf/2409.13745
• Copy Paste: [[2409.13745]] Context-Aware Membership Inference Attacks against Pre-trained Large Language Models(https://arxiv.org/abs/2409.13745)
• Keywords: generative
• Abstract: Prior Membership Inference Attacks (MIAs) on pre-trained Large Language Models (LLMs), adapted from classification model attacks, fail due to ignoring the generative process of LLMs across token sequences. In this paper, we present a novel attack that adapts MIA statistical tests to the perplexity dynamics of subsequences within a data point. Our method significantly outperforms prior loss-based approaches, revealing context-dependent memorization patterns in pre-trained LLMs.

Title: Do Large Language Models Need a Content Delivery Network?

• Authors: Yihua Cheng, Kuntai Du, Jiayi Yao, Junchen Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13761
• Pdf URL: https://arxiv.org/pdf/2409.13761
• Copy Paste: [[2409.13761]] Do Large Language Models Need a Content Delivery Network?(https://arxiv.org/abs/2409.13761)
• Keywords: in-context
• Abstract: As the use of large language models (LLMs) expands rapidly, so does the range of knowledge needed to supplement various LLM queries. Thus, enabling flexible and efficient injection of new knowledge in LLM inference is critical. Three high-level options exist: (i) embedding the knowledge in LLM's weights (i.e., fine-tuning), (ii) including the knowledge as a part of LLM's text input (i.e., in-context learning), or (iii) injecting the KV caches of the new knowledge to LLM during prefill. This paper argues that, although fine-tuning and in-context learning are popular, using KV caches as the medium of knowledge could simultaneously enable more modular management of knowledge injection and more efficient LLM serving with low cost and fast response. To realize these benefits, we envision a Knowledge Delivery Network (KDN), a new system component in LLM services that dynamically optimizes the storage, transfer, and composition of KV cache across LLM engines and other compute and storage resources. We believe that, just like content delivery networks (CDNs), such as Akamai, enabled the success of the Internet ecosystem through their efficient data delivery, KDNs will be critical to the success of LLM applications through their efficient knowledge delivery. We have open-sourced a KDN prototype at this https URL.

Title: Trustworthy Intrusion Detection: Confidence Estimation Using Latent Space

• Authors: Ioannis Pitsiorlas, George Arvanitakis, Marios Kountouris
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13774
• Pdf URL: https://arxiv.org/pdf/2409.13774
• Copy Paste: [[2409.13774]] Trustworthy Intrusion Detection: Confidence Estimation Using Latent Space(https://arxiv.org/abs/2409.13774)
• Keywords: anomaly
• Abstract: This work introduces a novel method for enhancing confidence in anomaly detection in Intrusion Detection Systems (IDS) through the use of a Variational Autoencoder (VAE) architecture. By developing a confidence metric derived from latent space representations, we aim to improve the reliability of IDS predictions against cyberattacks. Applied to the NSL-KDD dataset, our approach focuses on binary classification tasks to effectively distinguish between normal and malicious network activities. The methodology demonstrates a significant enhancement in anomaly detection, evidenced by a notable correlation of 0.45 between the reconstruction error and the proposed metric. Our findings highlight the potential of employing VAEs for more accurate and trustworthy anomaly detection in network security.

Title: Revisiting Synthetic Human Trajectories: Imitative Generation and Benchmarks Beyond Datasaurus

• Authors: Bangchao Deng, Xin Jing, Tianyue Yang, Bingqing Qu, Philippe Cudre-Mauroux, Dingqi Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13790
• Pdf URL: https://arxiv.org/pdf/2409.13790
• Copy Paste: [[2409.13790]] Revisiting Synthetic Human Trajectories: Imitative Generation and Benchmarks Beyond Datasaurus(https://arxiv.org/abs/2409.13790)
• Keywords: generative
• Abstract: Human trajectory data, which plays a crucial role in various applications such as crowd management and epidemic prevention, is challenging to obtain due to practical constraints and privacy concerns. In this context, synthetic human trajectory data is generated to simulate as close as possible to real-world human trajectories, often under summary statistics and distributional similarities. However, the complexity of human mobility patterns is oversimplified by these similarities (a.k.a. ``Datasaurus''), resulting in intrinsic biases in both generative model design and benchmarks of the generated trajectories. Against this background, we propose MIRAGE, a huMan-Imitative tRAjectory GenErative model designed as a neural Temporal Point Process integrating an Exploration and Preferential Return model. It imitates the human decision-making process in trajectory generation, rather than fitting any specific statistical distributions as traditional methods do, thus avoiding the Datasaurus issue. Moreover, we also propose a comprehensive task-based evaluation protocol beyond Datasaurus to systematically benchmark trajectory generative models on four typical downstream tasks, integrating multiple techniques and evaluation metrics for each task, to comprehensively assess the ultimate utility of the generated trajectories. We conduct a thorough evaluation of MIRAGE on three real-world user trajectory datasets against a sizeable collection of baselines. Results show that compared to the best baselines, MIRAGE-generated trajectory data not only achieves the best statistical and distributional similarities with 59.0-71.5% improvement, but also yields the best performance in the task-based evaluation with 10.9-33.4% improvement.

Title: Multi-Modality Conditioned Variational U-Net for Field-of-View Extension in Brain Diffusion MRI

• Authors: Zhiyuan Li, Tianyuan Yao, Praitayini Kanakaraj, Chenyu Gao, Shunxing Bao, Lianrui Zuo, Michael E. Kim, Nancy R. Newlin, Gaurav Rudravaram, Nazirah M. Khairi, Yuankai Huo, Kurt G. Schilling, Walter A. Kukull, Arthur W. Toga, Derek B. Archer, Timothy J. Hohman, Bennett A. Landman
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13846
• Pdf URL: https://arxiv.org/pdf/2409.13846
• Copy Paste: [[2409.13846]] Multi-Modality Conditioned Variational U-Net for Field-of-View Extension in Brain Diffusion MRI(https://arxiv.org/abs/2409.13846)
• Keywords: diffusion, generative
• Abstract: An incomplete field-of-view (FOV) in diffusion magnetic resonance imaging (dMRI) can severely hinder the volumetric and bundle analyses of whole-brain white matter connectivity. Although existing works have investigated imputing the missing regions using deep generative models, it remains unclear how to specifically utilize additional information from paired multi-modality data and whether this can enhance the imputation quality and be useful for downstream tractography. To fill this gap, we propose a novel framework for imputing dMRI scans in the incomplete part of the FOV by integrating the learned diffusion features in the acquired part of the FOV to the complete brain anatomical structure. We hypothesize that by this design the proposed framework can enhance the imputation performance of the dMRI scans and therefore be useful for repairing whole-brain tractography in corrupted dMRI scans with incomplete FOV. We tested our framework on two cohorts from different sites with a total of 96 subjects and compared it with a baseline imputation method that treats the information from T1w and dMRI scans equally. The proposed framework achieved significant improvements in imputation performance, as demonstrated by angular correlation coefficient (p < 1E-5), and in downstream tractography accuracy, as demonstrated by Dice score (p < 0.01). Results suggest that the proposed framework improved imputation performance in dMRI scans by specifically utilizing additional information from paired multi-modality data, compared with the baseline method. The imputation achieved by the proposed framework enhances whole brain tractography, and therefore reduces the uncertainty when analyzing bundles associated with neurodegenerative.

Title: SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation

• Authors: Maying Shen, Nadine Chang, Sifei Liu, Jose M. Alvarez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.13860
• Pdf URL: https://arxiv.org/pdf/2409.13860
• Copy Paste: [[2409.13860]] SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation(https://arxiv.org/abs/2409.13860)
• Keywords: foundation model
• Abstract: In recent years, the data collected for artificial intelligence has grown to an unmanageable amount. Particularly within industrial applications, such as autonomous vehicles, model training computation budgets are being exceeded while model performance is saturating -- and yet more data continues to pour in. To navigate the flood of data, we propose a framework to select the most semantically diverse and important dataset portion. Then, we further semantically enrich it by discovering meaningful new data from a massive unlabeled data pool. Importantly, we can provide explainability by leveraging foundation models to generate semantics for every data point. We quantitatively show that our Semantic Selection and Enrichment framework (SSE) can a) successfully maintain model performance with a smaller training dataset and b) improve model performance by enriching the smaller dataset without exceeding the original dataset size. Consequently, we demonstrate that semantic diversity is imperative for optimal data selection and model performance.

Title: Tabular Data Generation using Binary Diffusion

• Authors: Vitaliy Kinakh, Slava Voloshynovskiy
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13882
• Pdf URL: https://arxiv.org/pdf/2409.13882
• Copy Paste: [[2409.13882]] Tabular Data Generation using Binary Diffusion(https://arxiv.org/abs/2409.13882)
• Keywords: diffusion, generative
• Abstract: Generating synthetic tabular data is critical in machine learning, especially when real data is limited or sensitive. Traditional generative models often face challenges due to the unique characteristics of tabular data, such as mixed data types and varied distributions, and require complex preprocessing or large pretrained models. In this paper, we introduce a novel, lossless binary transformation method that converts any tabular data into fixed-size binary representations, and a corresponding new generative model called Binary Diffusion, specifically designed for binary data. Binary Diffusion leverages the simplicity of XOR operations for noise addition and removal and employs binary cross-entropy loss for training. Our approach eliminates the need for extensive preprocessing, complex noise parameter tuning, and pretraining on large datasets. We evaluate our model on several popular tabular benchmark datasets, demonstrating that Binary Diffusion outperforms existing state-of-the-art models on Travel, Adult Income, and Diabetes datasets while being significantly smaller in size.

Title: LLM for Everyone: Representing the Underrepresented in Large Language Models

• Authors: Samuel Cahyawijaya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13897
• Pdf URL: https://arxiv.org/pdf/2409.13897
• Copy Paste: [[2409.13897]] LLM for Everyone: Representing the Underrepresented in Large Language Models(https://arxiv.org/abs/2409.13897)
• Keywords: in-context
• Abstract: Natural language processing (NLP) has witnessed a profound impact of large language models (LLMs) that excel in a multitude of tasks. However, the limitation of LLMs in multilingual settings, particularly in underrepresented languages, remains a significant hurdle. This thesis aims to bridge the gap in NLP research and development by focusing on underrepresented languages. A comprehensive evaluation of LLMs is conducted to assess their capabilities in these languages, revealing the challenges of multilingual and multicultural generalization. Addressing the multilingual generalization gap, this thesis proposes data-and-compute-efficient methods to mitigate the disparity in LLM ability in underrepresented languages, allowing better generalization on underrepresented languages without the loss of task generalization ability. The proposed solutions cover cross-lingual continual instruction tuning, retrieval-based cross-lingual in-context learning, and in-context query alignment. Furthermore, a novel method to measure cultural values alignment between LLMs operating in different languages is proposed, ensuring cultural sensitivity and inclusivity. These contributions aim to enhance the multilingual and multicultural alignment of LLMs in underrepresented languages, ultimately advancing the NLP field toward greater equality and inclusiveness.

Title: High-Resolution Flood Probability Mapping Using Generative Machine Learning with Large-Scale Synthetic Precipitation and Inundation Data

• Authors: Lipai Huang, Federico Antolini, Ali Mostafavi, Russell Blessing, Matthew Garcia, Samuel D. Brody
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13936
• Pdf URL: https://arxiv.org/pdf/2409.13936
• Copy Paste: [[2409.13936]] High-Resolution Flood Probability Mapping Using Generative Machine Learning with Large-Scale Synthetic Precipitation and Inundation Data(https://arxiv.org/abs/2409.13936)
• Keywords: generative
• Abstract: High-resolution flood probability maps are essential for addressing the limitations of existing flood risk assessment approaches but are often limited by the availability of historical event data. Also, producing simulated data needed for creating probabilistic flood maps using physics-based models involves significant computation and time effort inhibiting the feasibility. To address this gap, this study introduces Flood-Precip GAN (Flood-Precipitation Generative Adversarial Network), a novel methodology that leverages generative machine learning to simulate large-scale synthetic inundation data to produce probabilistic flood maps. With a focus on Harris County, Texas, Flood-Precip GAN begins with training a cell-wise depth estimator using a limited number of physics-based model-generated precipitation-flood events. This model, which emphasizes precipitation-based features, outperforms universal models. Subsequently, a Generative Adversarial Network (GAN) with constraints is employed to conditionally generate synthetic precipitation records. Strategic thresholds are established to filter these records, ensuring close alignment with true precipitation patterns. For each cell, synthetic events are smoothed using a K-nearest neighbors algorithm and processed through the depth estimator to derive synthetic depth distributions. By iterating this procedure and after generating 10,000 synthetic precipitation-flood events, we construct flood probability maps in various formats, considering different inundation depths. Validation through similarity and correlation metrics confirms the fidelity of the synthetic depth distributions relative to true data. Flood-Precip GAN provides a scalable solution for generating synthetic flood depth data needed to create high-resolution flood probability maps, significantly enhancing flood preparedness and mitigation efforts.

Title: Enhancing Advanced Visual Reasoning Ability of Large Language Models

• Authors: Zhiyuan Li, Dongnan Liu, Chaoyi Zhang, Heng Wang, Tengfei Xue, Weidong Cai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13980
• Pdf URL: https://arxiv.org/pdf/2409.13980
• Copy Paste: [[2409.13980]] Enhancing Advanced Visual Reasoning Ability of Large Language Models(https://arxiv.org/abs/2409.13980)
• Keywords: in-context
• Abstract: Recent advancements in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models' advanced reasoning ability. Traditional Vision-Language Models (VLMs) perform well in visual perception tasks while struggling with complex reasoning scenarios. Conversely, Large Language Models (LLMs) demonstrate robust text reasoning capabilities; however, they lack visual acuity. To bridge this gap, we propose Complex Visual Reasoning Large Language Models (CVR-LLM), capitalizing on VLMs' visual perception proficiency and LLMs' extensive reasoning capability. Unlike recent multimodal large language models (MLLMs) that require a projection layer, our approach transforms images into detailed, context-aware descriptions using an iterative self-refinement loop and leverages LLMs' text knowledge for accurate predictions without extra training. We also introduce a novel multi-modal in-context learning (ICL) methodology to enhance LLMs' contextual understanding and reasoning. Additionally, we introduce Chain-of-Comparison (CoC), a step-by-step comparison technique enabling contrasting various aspects of predictions. Our CVR-LLM presents the first comprehensive study across a wide array of complex visual reasoning tasks and achieves SOTA performance among all.

Title: GAInS: Gradient Anomaly-aware Biomedical Instance Segmentation

• Authors: Runsheng Liu, Hao Jiang, Yanning Zhou, Huangjing Lin, Liansheng Wang, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.13988
• Pdf URL: https://arxiv.org/pdf/2409.13988
• Copy Paste: [[2409.13988]] GAInS: Gradient Anomaly-aware Biomedical Instance Segmentation(https://arxiv.org/abs/2409.13988)
• Keywords: anomaly
• Abstract: Instance segmentation plays a vital role in the morphological quantification of biomedical entities such as tissues and cells, enabling precise identification and delineation of different structures. Current methods often address the challenges of touching, overlapping or crossing instances through individual modeling, while neglecting the intrinsic interrelation between these conditions. In this work, we propose a Gradient Anomaly-aware Biomedical Instance Segmentation approach (GAInS), which leverages instance gradient information to perceive local gradient anomaly regions, thus modeling the spatial relationship between instances and refining local region segmentation. Specifically, GAInS is firstly built on a Gradient Anomaly Mapping Module (GAMM), which encodes the radial fields of instances through window sliding to obtain instance gradient anomaly maps. To efficiently refine boundaries and regions with gradient anomaly attention, we propose an Adaptive Local Refinement Module (ALRM) with a gradient anomaly-aware loss function. Extensive comparisons and ablation experiments in three biomedical scenarios demonstrate that our proposed GAInS outperforms other state-of-the-art (SOTA) instance segmentation methods. The code is available at this https URL.

Title: ChronoGAN: Supervised and Embedded Generative Adversarial Networks for Time Series Generation

• Authors: MohammadReza EskandariNasab, Shah Muhammad Hamdi, Soukaina Filali Boubrahimi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14013
• Pdf URL: https://arxiv.org/pdf/2409.14013
• Copy Paste: [[2409.14013]] ChronoGAN: Supervised and Embedded Generative Adversarial Networks for Time Series Generation(https://arxiv.org/abs/2409.14013)
• Keywords: generative
• Abstract: Generating time series data using Generative Adversarial Networks (GANs) presents several prevalent challenges, such as slow convergence, information loss in embedding spaces, instability, and performance variability depending on the series length. To tackle these obstacles, we introduce a robust framework aimed at addressing and mitigating these issues effectively. This advanced framework integrates the benefits of an Autoencoder-generated embedding space with the adversarial training dynamics of GANs. This framework benefits from a time series-based loss function and oversight from a supervisory network, both of which capture the stepwise conditional distributions of the data effectively. The generator functions within the latent space, while the discriminator offers essential feedback based on the feature space. Moreover, we introduce an early generation algorithm and an improved neural network architecture to enhance stability and ensure effective generalization across both short and long time series. Through joint training, our framework consistently outperforms existing benchmarks, generating high-quality time series data across a range of real and synthetic datasets with diverse characteristics.

Title: Mitigating Exposure Bias in Score-Based Generation of Molecular Conformations

• Authors: Sijia Wang, Chen Wang, Zhenhao Zhao, Jiqiang Zhang, Weiran Cai
• Subjects: cs.LG, cs.AI, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2409.14014
• Pdf URL: https://arxiv.org/pdf/2409.14014
• Copy Paste: [[2409.14014]] Mitigating Exposure Bias in Score-Based Generation of Molecular Conformations(https://arxiv.org/abs/2409.14014)
• Keywords: diffusion, generative
• Abstract: Molecular conformation generation poses a significant challenge in the field of computational chemistry. Recently, Diffusion Probabilistic Models (DPMs) and Score-Based Generative Models (SGMs) are effectively used due to their capacity for generating accurate conformations far beyond conventional physics-based approaches. However, the discrepancy between training and inference rises a critical problem known as the exposure bias. While this issue has been extensively investigated in DPMs, the existence of exposure bias in SGMs and its effective measurement remain unsolved, which hinders the use of compensation methods for SGMs, including ConfGF and Torsional Diffusion as the representatives. In this work, we first propose a method for measuring exposure bias in SGMs used for molecular conformation generation, which confirms the significant existence of exposure bias in these models and measures its value. We design a new compensation algorithm Input Perturbation (IP), which is adapted from a method originally designed for DPMs only. Experimental results show that by introducing IP, SGM-based molecular conformation models can significantly improve both the accuracy and diversity of the generated conformations. Especially by using the IP-enhanced Torsional Diffusion model, we achieve new state-of-the-art performance on the GEOM-Drugs dataset and are on par on GEOM-QM9. We provide the code publicly at this https URL.

Title: BrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidance

• Authors: Ling Wang, Chen Wu, Lin Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14021
• Pdf URL: https://arxiv.org/pdf/2409.14021
• Copy Paste: [[2409.14021]] BrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidance(https://arxiv.org/abs/2409.14021)
• Keywords: diffusion, generative
• Abstract: Can we directly visualize what we imagine in our brain together with what we describe? The inherent nature of human perception reveals that, when we think, our body can combine language description and build a vivid picture in our brain. Intuitively, generative models should also hold such versatility. In this paper, we introduce BrainDreamer, a novel end-to-end language-guided generative framework that can mimic human reasoning and generate high-quality images from electroencephalogram (EEG) brain signals. Our method is superior in its capacity to eliminate the noise introduced by non-invasive EEG data acquisition and meanwhile achieve a more precise mapping between the EEG and image modality, thus leading to significantly better-generated images. Specifically, BrainDreamer consists of two key learning stages: 1) modality alignment and 2) image generation. In the alignment stage, we propose a novel mask-based triple contrastive learning strategy to effectively align EEG, text, and image embeddings to learn a unified representation. In the generation stage, we inject the EEG embeddings into the pre-trained Stable Diffusion model by designing a learnable EEG adapter to generate high-quality reasoning-coherent images. Moreover, BrainDreamer can accept textual descriptions (e.g., color, position, etc.) to achieve controllable image generation. Extensive experiments show that our method significantly outperforms prior arts in terms of generating quality and quantitative performance.

Title: Recovering Global Data Distribution Locally in Federated Learning

• Authors: Ziyu Yao
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14063
• Pdf URL: https://arxiv.org/pdf/2409.14063
• Copy Paste: [[2409.14063]] Recovering Global Data Distribution Locally in Federated Learning(https://arxiv.org/abs/2409.14063)
• Keywords: generative
• Abstract: Federated Learning (FL) is a distributed machine learning paradigm that enables collaboration among multiple clients to train a shared model without sharing raw data. However, a major challenge in FL is the label imbalance, where clients may exclusively possess certain classes while having numerous minority and missing classes. Previous works focus on optimizing local updates or global aggregation but ignore the underlying imbalanced label distribution across clients. In this paper, we propose a novel approach ReGL to address this challenge, whose key idea is to Recover the Global data distribution Locally. Specifically, each client uses generative models to synthesize images that complement the minority and missing classes, thereby alleviating label imbalance. Moreover, we adaptively fine-tune the image generation process using local real data, which makes the synthetic images align more closely with the global distribution. Importantly, both the generation and fine-tuning processes are conducted at the client-side without leaking data privacy. Through comprehensive experiments on various image classification datasets, we demonstrate the remarkable superiority of our approach over existing state-of-the-art works in fundamentally tackling label imbalance in FL.

Title: One-shot World Models Using a Transformer Trained on a Synthetic Prior

• Authors: Fabio Ferreira, Moreno Schlageter, Raghu Rajan, Andre Biedenkapp, Frank Hutter
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14084
• Pdf URL: https://arxiv.org/pdf/2409.14084
• Copy Paste: [[2409.14084]] One-shot World Models Using a Transformer Trained on a Synthetic Prior(https://arxiv.org/abs/2409.14084)
• Keywords: in-context
• Abstract: A World Model is a compressed spatial and temporal representation of a real world environment that allows one to train an agent or execute planning methods. However, world models are typically trained on observations from the real world environment, and they usually do not enable learning policies for other real environments. We propose One-Shot World Model (OSWM), a transformer world model that is learned in an in-context learning fashion from purely synthetic data sampled from a prior distribution. Our prior is composed of multiple randomly initialized neural networks, where each network models the dynamics of each state and reward dimension of a desired target environment. We adopt the supervised learning procedure of Prior-Fitted Networks by masking next-state and reward at random context positions and query OSWM to make probabilistic predictions based on the remaining transition context. During inference time, OSWM is able to quickly adapt to the dynamics of a simple grid world, as well as the CartPole gym and a custom control environment by providing 1k transition steps as context and is then able to successfully train environment-solving agent policies. However, transferring to more complex environments remains a challenge, currently. Despite these limitations, we see this work as an important stepping-stone in the pursuit of learning world models purely from synthetic data.

Title: Foundation Models for Amodal Video Instance Segmentation in Automated Driving

• Authors: Jasmin Breitenstein, Franz Jünger, Andreas Bär, Tim Fingscheidt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14095
• Pdf URL: https://arxiv.org/pdf/2409.14095
• Copy Paste: [[2409.14095]] Foundation Models for Amodal Video Instance Segmentation in Automated Driving(https://arxiv.org/abs/2409.14095)
• Keywords: foundation model
• Abstract: In this work, we study amodal video instance segmentation for automated driving. Previous works perform amodal video instance segmentation relying on methods trained on entirely labeled video data with techniques borrowed from standard video instance segmentation. Such amodally labeled video data is difficult and expensive to obtain and the resulting methods suffer from a trade-off between instance segmentation and tracking performance. To largely solve this issue, we propose to study the application of foundation models for this task. More precisely, we exploit the extensive knowledge of the Segment Anything Model (SAM), while fine-tuning it to the amodal instance segmentation task. Given an initial video instance segmentation, we sample points from the visible masks to prompt our amodal SAM. We use a point memory to store those points. If a previously observed instance is not predicted in a following frame, we retrieve its most recent points from the point memory and use a point tracking method to follow those points to the current frame, together with the corresponding last amodal instance mask. This way, while basing our method on an amodal instance segmentation, we nevertheless obtain video-level amodal instance segmentation results. Our resulting S-AModal method achieves state-of-the-art results in amodal video instance segmentation while resolving the need for amodal video-based labels. Code for S-AModal is available at this https URL.

Title: PoseAugment: Generative Human Pose Data Augmentation with Physical Plausibility for IMU-based Motion Capture

• Authors: Zhuojun Li, Chun Yu, Chen Liang, Yuanchun Shi
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.14101
• Pdf URL: https://arxiv.org/pdf/2409.14101
• Copy Paste: [[2409.14101]] PoseAugment: Generative Human Pose Data Augmentation with Physical Plausibility for IMU-based Motion Capture(https://arxiv.org/abs/2409.14101)
• Keywords: generative
• Abstract: The data scarcity problem is a crucial factor that hampers the model performance of IMU-based human motion capture. However, effective data augmentation for IMU-based motion capture is challenging, since it has to capture the physical relations and constraints of the human body, while maintaining the data distribution and quality. We propose PoseAugment, a novel pipeline incorporating VAE-based pose generation and physical optimization. Given a pose sequence, the VAE module generates infinite poses with both high fidelity and diversity, while keeping the data distribution. The physical module optimizes poses to satisfy physical constraints with minimal motion restrictions. High-quality IMU data are then synthesized from the augmented poses for training motion capture models. Experiments show that PoseAugment outperforms previous data augmentation and pose generation methods in terms of motion capture accuracy, revealing a strong potential of our method to alleviate the data collection burden for IMU-based motion capture and related tasks driven by human poses.

Title: ESDS: AI-Powered Early Stunting Detection and Monitoring System using Edited Radius-SMOTE Algorithm

• Authors: A.A. Gde Yogi Pramana, Haidar Muhammad Zidan, Muhammad Fazil Maulana, Oskar Natan
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.14105
• Pdf URL: https://arxiv.org/pdf/2409.14105
• Copy Paste: [[2409.14105]] ESDS: AI-Powered Early Stunting Detection and Monitoring System using Edited Radius-SMOTE Algorithm(https://arxiv.org/abs/2409.14105)
• Keywords: generative
• Abstract: Stunting detection is a significant issue in Indonesian healthcare, causing lower cognitive function, lower productivity, a weakened immunity, delayed neuro-development, and degenerative diseases. In regions with a high prevalence of stunting and limited welfare resources, identifying children in need of treatment is critical. The diagnostic process often raises challenges, such as the lack of experience in medical workers, incompatible anthropometric equipment, and inefficient medical bureaucracy. To counteract the issues, the use of load cell sensor and ultrasonic sensor can provide suitable anthropometric equipment and streamline the medical bureaucracy for stunting detection. This paper also employs machine learning for stunting detection based on sensor readings. The experiment results show that the sensitivity of the load cell sensor and the ultrasonic sensor is 0.9919 and 0.9986, respectively. Also, the machine learning test results have three classification classes, which are normal, stunted, and stunting with an accuracy rate of 98\%.

Title: Local Patterns Generalize Better for Novel Anomalies

• Authors: Yalong Jiang, Liquan Mao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14109
• Pdf URL: https://arxiv.org/pdf/2409.14109
• Copy Paste: [[2409.14109]] Local Patterns Generalize Better for Novel Anomalies(https://arxiv.org/abs/2409.14109)
• Keywords: anomaly
• Abstract: Video anomaly detection (VAD) aims at identifying novel actions or events which are unseen during training. Existing mainstream VAD techniques focus on the global patterns of events and cannot properly generalize to novel samples. In this paper, we propose a framework to identify the spatial local patterns which generalize to novel samples and model the dynamics of local patterns. In spatial part of the framework, the capability of extracting local patterns is gained from image-text contrastive learning with Image-Text Alignment Module (ITAM). To detect different types of anomalies, a two-branch framework is proposed for representing the local patterns in both actions and appearances. In temporal part of the framework, a State Machine Module (SMM) is proposed to model the dynamics of local patterns by decomposing their temporal variations into motion components. Different dynamics are represented with different weighted sums of a fixed set of motion components. The video sequences with either novel spatial distributions of local patterns or distinctive dynamics of local patterns are deemed as anomalies. Extensive experiments on popular benchmark datasets demonstrate that state-of-the-art performance can be achieved.

Title: Efficient and Effective Model Extraction

• Authors: Hongyu Zhu, Wentao Hu, Sichu Liang, Fangqi Li, Wenwen Wang, Shilin Wang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14122
• Pdf URL: https://arxiv.org/pdf/2409.14122
• Copy Paste: [[2409.14122]] Efficient and Effective Model Extraction(https://arxiv.org/abs/2409.14122)
• Keywords: generative
• Abstract: Model extraction aims to create a functionally similar copy from a machine learning as a service (MLaaS) API with minimal overhead, often for illicit purposes or as a precursor to further attacks, posing a significant threat to the MLaaS ecosystem. However, recent studies show that model extraction is inefficient, especially when the target task distribution is unavailable. In such cases, even significantly increasing the attack budget fails to yield a sufficiently similar model, reducing the adversary's incentive. In this paper, we revisit the basic design choices throughout the extraction process and propose an efficient and effective algorithm, Efficient and Effective Model Extraction (E3), which optimizes both query preparation and the training routine. E3 achieves superior generalization over state-of-the-art methods while minimizing computational costs. For example, with only 0.005 times the query budget and less than 0.2 times the runtime, E3 outperforms classical generative model-based data-free model extraction with over 50% absolute accuracy improvement on CIFAR-10. Our findings highlight the ongoing risk of model extraction and propose E3 as a useful benchmark for future security evaluations.

Title: JVID: Joint Video-Image Diffusion for Visual-Quality and Temporal-Consistency in Video Generation

• Authors: Hadrien Reynaud, Matthew Baugh, Mischa Dombrowski, Sarah Cechnicka, Qingjie Meng, Bernhard Kainz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14149
• Pdf URL: https://arxiv.org/pdf/2409.14149
• Copy Paste: [[2409.14149]] JVID: Joint Video-Image Diffusion for Visual-Quality and Temporal-Consistency in Video Generation(https://arxiv.org/abs/2409.14149)
• Keywords: diffusion
• Abstract: We introduce the Joint Video-Image Diffusion model (JVID), a novel approach to generating high-quality and temporally coherent videos. We achieve this by integrating two diffusion models: a Latent Image Diffusion Model (LIDM) trained on images and a Latent Video Diffusion Model (LVDM) trained on video data. Our method combines these models in the reverse diffusion process, where the LIDM enhances image quality and the LVDM ensures temporal consistency. This unique combination allows us to effectively handle the complex spatio-temporal dynamics in video generation. Our results demonstrate quantitative and qualitative improvements in producing realistic and coherent videos.

Title: Content-aware Tile Generation using Exterior Boundary Inpainting

• Authors: Sam Sartor, Pieter Peers
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.14184
• Pdf URL: https://arxiv.org/pdf/2409.14184
• Copy Paste: [[2409.14184]] Content-aware Tile Generation using Exterior Boundary Inpainting(https://arxiv.org/abs/2409.14184)
• Keywords: diffusion
• Abstract: We present a novel and flexible learning-based method for generating tileable image sets. Our method goes beyond simple self-tiling, supporting sets of mutually tileable images that exhibit a high degree of diversity. To promote diversity we decouple structure from content by foregoing explicit copying of patches from an exemplar image. Instead we leverage the prior knowledge of natural images and textures embedded in large-scale pretrained diffusion models to guide tile generation constrained by exterior boundary conditions and a text prompt to specify the content. By carefully designing and selecting the exterior boundary conditions, we can reformulate the tile generation process as an inpainting problem, allowing us to directly employ existing diffusion-based inpainting models without the need to retrain a model on a custom training set. We demonstrate the flexibility and efficacy of our content-aware tile generation method on different tiling schemes, such as Wang tiles, from only a text prompt. Furthermore, we introduce a novel Dual Wang tiling scheme that provides greater texture continuity and diversity than existing Wang tile variants.

Title: Advancing Employee Behavior Analysis through Synthetic Data: Leveraging ABMs, GANs, and Statistical Models for Enhanced Organizational Efficiency

• Authors: Rakshitha Jayashankar, Mahesh Balan
• Subjects: cs.LG, cs.FL, stat.OT
• Abstract URL: https://arxiv.org/abs/2409.14197
• Pdf URL: https://arxiv.org/pdf/2409.14197
• Copy Paste: [[2409.14197]] Advancing Employee Behavior Analysis through Synthetic Data: Leveraging ABMs, GANs, and Statistical Models for Enhanced Organizational Efficiency(https://arxiv.org/abs/2409.14197)
• Keywords: generative
• Abstract: Success in todays data-driven corporate climate requires a deep understanding of employee behavior. Companies aim to improve employee satisfaction, boost output, and optimize workflow. This research study delves into creating synthetic data, a powerful tool that allows us to comprehensively understand employee performance, flexibility, cooperation, and team dynamics. Synthetic data provides a detailed and accurate picture of employee activities while protecting individual privacy thanks to cutting-edge methods like agent-based models (ABMs), Generative Adversarial Networks (GANs), and statistical models. Through the creation of multiple situations, this method offers insightful viewpoints regarding increasing teamwork, improving adaptability, and accelerating overall productivity. We examine how synthetic data has evolved from a specialized field to an essential resource for researching employee behavior and enhancing management efficiency. Keywords: Agent-Based Model, Generative Adversarial Network, workflow optimization, organizational success

Title: Deep Learning Technology for Face Forgery Detection: A Survey

• Authors: Lixia Ma, Puning Yang, Yuting Xu, Ziming Yang, Peipei Li, Huaibo Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14289
• Pdf URL: https://arxiv.org/pdf/2409.14289
• Copy Paste: [[2409.14289]] Deep Learning Technology for Face Forgery Detection: A Survey(https://arxiv.org/abs/2409.14289)
• Keywords: generative
• Abstract: Currently, the rapid development of computer vision and deep learning has enabled the creation or manipulation of high-fidelity facial images and videos via deep generative approaches. This technology, also known as deepfake, has achieved dramatic progress and become increasingly popular in social media. However, the technology can generate threats to personal privacy and national security by spreading misinformation. To diminish the risks of deepfake, it is desirable to develop powerful forgery detection methods to distinguish fake faces from real faces. This paper presents a comprehensive survey of recent deep learning-based approaches for facial forgery detection. We attempt to provide the reader with a deeper understanding of the current advances as well as the major challenges for deepfake detection based on deep learning. We present an overview of deepfake techniques and analyse the characteristics of various deepfake datasets. We then provide a systematic review of different categories of deepfake detection and state-of-the-art deepfake detection methods. The drawbacks of existing detection methods are analyzed, and future research directions are discussed to address the challenges in improving both the performance and generalization of deepfake detection.

Title: DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation

• Authors: Xuewen Liu, Zhikai Li, Qingyi Gu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14307
• Pdf URL: https://arxiv.org/pdf/2409.14307
• Copy Paste: [[2409.14307]] DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation(https://arxiv.org/abs/2409.14307)
• Keywords: diffusion
• Abstract: Diffusion models have shown excellent performance on various image generation tasks, but the substantial computational costs and huge memory footprint hinder their low-latency applications in real-world scenarios. Quantization is a promising way to compress and accelerate models. Nevertheless, due to the wide range and time-varying activations in diffusion models, existing methods cannot maintain both accuracy and efficiency simultaneously for low-bit quantization. To tackle this issue, we propose DilateQuant, a novel quantization framework for diffusion models that offers comparable accuracy and high efficiency. Specifically, we keenly aware of numerous unsaturated in-channel weights, which can be cleverly exploited to reduce the range of activations without additional computation cost. Based on this insight, we propose Weight Dilation (WD) that maximally dilates the unsaturated in-channel weights to a constrained range through a mathematically equivalent scaling. WD costlessly absorbs the activation quantization errors into weight quantization. The range of activations decreases, which makes activations quantization easy. The range of weights remains constant, which makes model easy to converge in training stage. Considering the temporal network leads to time-varying activations, we design a Temporal Parallel Quantizer (TPQ), which sets time-step quantization parameters and supports parallel quantization for different time steps, significantly improving the performance and reducing time cost. To further enhance performance while preserving efficiency, we introduce a Block-wise Knowledge Distillation (BKD) to align the quantized models with the full-precision models at a block level. The simultaneous training of time-step quantization parameters and weights minimizes the time required, and the shorter backpropagation paths decreases the memory footprint of the quantization process.

Title: Anisotropic Diffusion Probabilistic Model for Imbalanced Image Classification

• Authors: Jingyu Kong, Yuan Guo, Yu Wang, Yuping Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14313
• Pdf URL: https://arxiv.org/pdf/2409.14313
• Copy Paste: [[2409.14313]] Anisotropic Diffusion Probabilistic Model for Imbalanced Image Classification(https://arxiv.org/abs/2409.14313)
• Keywords: diffusion, generative
• Abstract: Real-world data often has a long-tailed distribution, where the scarcity of tail samples significantly limits the model's generalization ability. Denoising Diffusion Probabilistic Models (DDPM) are generative models based on stochastic differential equation theory and have demonstrated impressive performance in image classification tasks. However, existing diffusion probabilistic models do not perform satisfactorily in classifying tail classes. In this work, we propose the Anisotropic Diffusion Probabilistic Model (ADPM) for imbalanced image classification problems. We utilize the data distribution to control the diffusion speed of different class samples during the forward process, effectively improving the classification accuracy of the denoiser in the reverse process. Specifically, we provide a theoretical strategy for selecting noise levels for different categories in the diffusion process based on error analysis theory to address the imbalanced classification problem. Furthermore, we integrate global and local image prior in the forward process to enhance the model's discriminative ability in the spatial dimension, while incorporate semantic-level contextual information in the reverse process to boost the model's discriminative power and robustness. Through comparisons with state-of-the-art methods on four medical benchmark datasets, we validate the effectiveness of the proposed method in handling long-tail data. Our results confirm that the anisotropic diffusion model significantly improves the classification accuracy of rare classes while maintaining the accuracy of head classes. On the skin lesion datasets, PAD-UFES and HAM10000, the F1-scores of our method improved by 4% and 3%, respectively compared to the original diffusion probabilistic model.

Title: Self-Supervised Audio-Visual Soundscape Stylization

• Authors: Tingle Li, Renhao Wang, Po-Yao Huang, Andrew Owens, Gopala Anumanchipalli
• Subjects: cs.CV, cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.14340
• Pdf URL: https://arxiv.org/pdf/2409.14340
• Copy Paste: [[2409.14340]] Self-Supervised Audio-Visual Soundscape Stylization(https://arxiv.org/abs/2409.14340)
• Keywords: diffusion, self-supervised
• Abstract: Speech sounds convey a great deal of information about the scenes, resulting in a variety of effects ranging from reverberation to additional ambient sounds. In this paper, we manipulate input speech to sound as though it was recorded within a different scene, given an audio-visual conditional example recorded from that scene. Our model learns through self-supervision, taking advantage of the fact that natural video contains recurring sound events and textures. We extract an audio clip from a video and apply speech enhancement. We then train a latent diffusion model to recover the original speech, using another audio-visual clip taken from elsewhere in the video as a conditional hint. Through this process, the model learns to transfer the conditional example's sound properties to the input speech. We show that our model can be successfully trained using unlabeled, in-the-wild videos, and that an additional visual signal can improve its sound prediction abilities. Please see our project webpage for video results: https://tinglok.netlify.app/files/avsoundscape/

Title: The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests

• Authors: Lior Madmoni, Amir Zait, Ilia Labzovsky, Danny Karmon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14371
• Pdf URL: https://arxiv.org/pdf/2409.14371
• Copy Paste: [[2409.14371]] The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests(https://arxiv.org/abs/2409.14371)
• Keywords: generative, in-context
• Abstract: Generative AI agents are often expected to respond to complex user requests that have No One Right Answer (NORA), e.g., "design a vegetarian meal plan below 1800 calories". Such requests may entail a set of constraints that the agent should adhere to. To successfully develop agents for NORA scenarios, an accurate automatic evaluation framework is essential, and specifically - one capable of validating the satisfaction of constraints in the agent's response. Recently, large language models (LLMs) have been adopted as versatile evaluators for many NORA tasks, but their ability to evaluate constraint-satisfaction in generated text remains unclear. To study this, we develop and release a novel Arithmetic Constraint-Satisfaction (ACS) benchmarking dataset. The dataset consists of complex user requests with corresponding constraints, agent responses and human labels indicating each constraint's satisfaction level in the response. A unique property of this dataset is that validating many of its constraints requires reviewing the response as a whole (in contrast to many other benchmarks that require the validation of a single independent item). Moreover, it assesses LLMs in performing reasoning, in-context data extraction, arithmetic calculations, and counting. We then benchmark both open and proprietary LLMs on evaluating constraint-satisfaction, and show that most models still have a significant headroom for improvement, and that errors primarily stem from reasoning issues. In addition, most models exhibit a skewed constraint-satisfaction prediction pattern, with higher accuracy where the ground-truth label is "satisfied". Lastly, few-shot prompting for our task proved to be rather challenging, since many of the studied models showed a degradation in performance when it was introduced.

Title: GroupDiff: Diffusion-based Group Portrait Editing

• Authors: Yuming Jiang, Nanxuan Zhao, Qing Liu, Krishna Kumar Singh, Shuai Yang, Chen Change Loy, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14379
• Pdf URL: https://arxiv.org/pdf/2409.14379
• Copy Paste: [[2409.14379]] GroupDiff: Diffusion-based Group Portrait Editing(https://arxiv.org/abs/2409.14379)
• Keywords: diffusion
• Abstract: Group portrait editing is highly desirable since users constantly want to add a person, delete a person, or manipulate existing persons. It is also challenging due to the intricate dynamics of human interactions and the diverse gestures. In this work, we present GroupDiff, a pioneering effort to tackle group photo editing with three dedicated contributions: 1) Data Engine: Since there is no labeled data for group photo editing, we create a data engine to generate paired data for training. The training data engine covers the diverse needs of group portrait editing. 2) Appearance Preservation: To keep the appearance consistent after editing, we inject the images of persons from the group photo into the attention modules and employ skeletons to provide intra-person guidance. 3) Control Flexibility: Bounding boxes indicating the locations of each person are used to reweight the attention matrix so that the features of each person can be injected into the correct places. This inter-person guidance provides flexible manners for manipulation. Extensive experiments demonstrate that GroupDiff exhibits state-of-the-art performance compared to existing methods. GroupDiff offers controllability for editing and maintains the fidelity of the original photos.

Title: A Visualized Malware Detection Framework with CNN and Conditional GAN

• Authors: Fang Wang (Florence Wong), Hussam Al Hamadi, Ernesto Damiani
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14439
• Pdf URL: https://arxiv.org/pdf/2409.14439
• Copy Paste: [[2409.14439]] A Visualized Malware Detection Framework with CNN and Conditional GAN(https://arxiv.org/abs/2409.14439)
• Keywords: generative
• Abstract: Malware visualization analysis incorporating with Machine Learning (ML) has been proven to be a promising solution for improving security defenses on different platforms. In this work, we propose an integrated framework for addressing common problems experienced by ML utilizers in developing malware detection systems. Namely, a pictorial presentation system with extensions is designed to preserve the identities of benign/malign samples by encoding each variable into binary digits and mapping them into black and white pixels. A conditional Generative Adversarial Network based model is adopted to produce synthetic images and mitigate issues of imbalance classes. Detection models architected by Convolutional Neural Networks are for validating performances while training on datasets with and without artifactual samples. Result demonstrates accuracy rates of 98.51% and 97.26% for these two training scenarios.

Title: Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding

• Authors: Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, Bo Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14485
• Pdf URL: https://arxiv.org/pdf/2409.14485
• Copy Paste: [[2409.14485]] Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding(https://arxiv.org/abs/2409.14485)
• Keywords: anomaly
• Abstract: Although current Multi-modal Large Language Models (MLLMs) demonstrate promising results in video understanding, processing extremely long videos remains an ongoing challenge. Typically, MLLMs struggle with handling thousands of tokens that exceed the maximum context length of LLMs, and they experience reduced visual clarity due to token aggregation. Another challenge is the high computational cost stemming from the large number of video tokens. To tackle these issues, we propose Video-XL, an extra-long vision language model designed for efficient hour-scale video understanding. Specifically, we argue that LLMs can be adapted as effective visual condensers and introduce Visual Context Latent Summarization, which condenses visual contexts into highly compact forms. Extensive experiments demonstrate that our model achieves promising results on popular long video understanding benchmarks, despite being trained on limited image data. Moreover, Video-XL strikes a promising balance between efficiency and effectiveness, processing 1024 frames on a single 80GB GPU while achieving nearly 100\% accuracy in the Needle-in-a-Haystack evaluation. We envision Video-XL becoming a valuable tool for long video applications such as video summarization, surveillance anomaly detection, and Ad placement identification.

Title: Unleashing the Power of Emojis in Texts via Self-supervised Graph Pre-Training

• Authors: Zhou Zhang, Dongzeng Tan, Jiaan Wang, Yilong Chen, Jiarong Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14552
• Pdf URL: https://arxiv.org/pdf/2409.14552
• Copy Paste: [[2409.14552]] Unleashing the Power of Emojis in Texts via Self-supervised Graph Pre-Training(https://arxiv.org/abs/2409.14552)
• Keywords: self-supervised
• Abstract: Emojis have gained immense popularity on social platforms, serving as a common means to supplement or replace text. However, existing data mining approaches generally either completely ignore or simply treat emojis as ordinary Unicode characters, which may limit the model's ability to grasp the rich semantic information in emojis and the interaction between emojis and texts. Thus, it is necessary to release the emoji's power in social media data mining. To this end, we first construct a heterogeneous graph consisting of three types of nodes, i.e. post, word and emoji nodes to improve the representation of different elements in posts. The edges are also well-defined to model how these three elements interact with each other. To facilitate the sharing of information among post, word and emoji nodes, we propose a graph pre-train framework for text and emoji co-modeling, which contains two graph pre-training tasks: node-level graph contrastive learning and edge-level link reconstruction learning. Extensive experiments on the Xiaohongshu and Twitter datasets with two types of downstream tasks demonstrate that our approach proves significant improvement over previous strong baseline methods.

Title: Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions

• Authors: Hongchen Wang, Kangming Li, Scott Ramsay, Yao Fehlis, Edward Kim, Jason Hattrick-Simpers
• Subjects: cs.CL, cond-mat.mtrl-sci, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14572
• Pdf URL: https://arxiv.org/pdf/2409.14572
• Copy Paste: [[2409.14572]] Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions(https://arxiv.org/abs/2409.14572)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have the potential to revolutionize scientific research, yet their robustness and reliability in domain-specific applications remain insufficiently explored. This study conducts a comprehensive evaluation and robustness analysis of LLMs within the field of materials science, focusing on domain-specific question answering and materials property prediction. Three distinct datasets are used in this study: 1) a set of multiple-choice questions from undergraduate-level materials science courses, 2) a dataset including various steel compositions and yield strengths, and 3) a band gap dataset, containing textual descriptions of material crystal structures and band gap values. The performance of LLMs is assessed using various prompting strategies, including zero-shot chain-of-thought, expert prompting, and few-shot in-context learning. The robustness of these models is tested against various forms of 'noise', ranging from realistic disturbances to intentionally adversarial manipulations, to evaluate their resilience and reliability under real-world conditions. Additionally, the study uncovers unique phenomena of LLMs during predictive tasks, such as mode collapse behavior when the proximity of prompt examples is altered and performance enhancement from train/test mismatch. The findings aim to provide informed skepticism for the broad use of LLMs in materials science and to inspire advancements that enhance their robustness and reliability for practical applications.

Title: URSimulator: Human-Perception-Driven Prompt Tuning for Enhanced Virtual Urban Renewal via Diffusion Models

• Authors: Chuanbo Hu, Shan Jia, Xin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14589
• Pdf URL: https://arxiv.org/pdf/2409.14589
• Copy Paste: [[2409.14589]] URSimulator: Human-Perception-Driven Prompt Tuning for Enhanced Virtual Urban Renewal via Diffusion Models(https://arxiv.org/abs/2409.14589)
• Keywords: diffusion
• Abstract: Tackling Urban Physical Disorder (e.g., abandoned buildings, litter, messy vegetation, graffiti) is essential, as it negatively impacts the safety, well-being, and psychological state of communities. Urban Renewal is the process of revitalizing these neglected and decayed areas within a city to improve the physical environment and quality of life for residents. Effective urban renewal efforts can transform these environments, enhancing their appeal and livability. However, current research lacks simulation tools that can quantitatively assess and visualize the impacts of renewal efforts, often relying on subjective judgments. Such tools are crucial for planning and implementing effective strategies by providing a clear visualization of potential changes and their impacts. This paper presents a novel framework addressing this gap by using human perception feedback to simulate street environment enhancement. We develop a prompt tuning approach that integrates text-driven Stable Diffusion with human perception feedback, iteratively editing local areas of street view images to better align with perceptions of beauty, liveliness, and safety. Our experiments show that this framework significantly improves perceptions of urban environments, with increases of 17.60% in safety, 31.15% in beauty, and 28.82% in liveliness. In contrast, advanced methods like DiffEdit achieve only 2.31%, 11.87%, and 15.84% improvements, respectively. We applied this framework across various virtual scenarios, including neighborhood improvement, building redevelopment, green space expansion, and community garden creation. The results demonstrate its effectiveness in simulating urban renewal, offering valuable insights for urban planning and policy-making.

Title: Implicit Dynamical Flow Fusion (IDFF) for Generative Modeling

• Authors: Mohammad R. Rezaei, Rahul G. Krishnan, Milos R. Popovic, Milad Lankarany
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14599
• Pdf URL: https://arxiv.org/pdf/2409.14599
• Copy Paste: [[2409.14599]] Implicit Dynamical Flow Fusion (IDFF) for Generative Modeling(https://arxiv.org/abs/2409.14599)
• Keywords: diffusion, generative
• Abstract: Conditional Flow Matching (CFM) models can generate high-quality samples from a non-informative prior, but they can be slow, often needing hundreds of network evaluations (NFE). To address this, we propose Implicit Dynamical Flow Fusion (IDFF); IDFF learns a new vector field with an additional momentum term that enables taking longer steps during sample generation while maintaining the fidelity of the generated distribution. Consequently, IDFFs reduce the NFEs by a factor of ten (relative to CFMs) without sacrificing sample quality, enabling rapid sampling and efficient handling of image and time-series data generation tasks. We evaluate IDFF on standard benchmarks such as CIFAR-10 and CelebA for image generation. We achieved likelihood and quality performance comparable to CFMs and diffusion-based models with fewer NFEs. IDFF also shows superior performance on time-series datasets modeling, including molecular simulation and sea surface temperature (SST) datasets, highlighting its versatility and effectiveness across different domains.

Title: Protein-Mamba: Biological Mamba Models for Protein Function Prediction

• Authors: Bohao Xu, Yingzhou Lu, Yoshitaka Inoue, Namkyeong Lee, Tianfan Fu, Jintai Chen
• Subjects: cs.LG, q-bio.BM, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2409.14617
• Pdf URL: https://arxiv.org/pdf/2409.14617
• Copy Paste: [[2409.14617]] Protein-Mamba: Biological Mamba Models for Protein Function Prediction(https://arxiv.org/abs/2409.14617)
• Keywords: self-supervised
• Abstract: Protein function prediction is a pivotal task in drug discovery, significantly impacting the development of effective and safe therapeutics. Traditional machine learning models often struggle with the complexity and variability inherent in predicting protein functions, necessitating more sophisticated approaches. In this work, we introduce Protein-Mamba, a novel two-stage model that leverages both self-supervised learning and fine-tuning to improve protein function prediction. The pre-training stage allows the model to capture general chemical structures and relationships from large, unlabeled datasets, while the fine-tuning stage refines these insights using specific labeled datasets, resulting in superior prediction performance. Our extensive experiments demonstrate that Protein-Mamba achieves competitive performance, compared with a couple of state-of-the-art methods across a range of protein function datasets. This model's ability to effectively utilize both unlabeled and labeled data highlights the potential of self-supervised learning in advancing protein function prediction and offers a promising direction for future research in drug discovery.

Title: SOS: Segment Object System for Open-World Instance Segmentation With Object Priors

• Authors: Christian Wilms, Tim Rolff, Maris Hillemann, Robert Johanson, Simone Frintrop
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14627
• Pdf URL: https://arxiv.org/pdf/2409.14627
• Copy Paste: [[2409.14627]] SOS: Segment Object System for Open-World Instance Segmentation With Object Priors(https://arxiv.org/abs/2409.14627)
• Keywords: self-supervised, foundation model
• Abstract: We propose an approach for Open-World Instance Segmentation (OWIS), a task that aims to segment arbitrary unknown objects in images by generalizing from a limited set of annotated object classes during training. Our Segment Object System (SOS) explicitly addresses the generalization ability and the low precision of state-of-the-art systems, which often generate background detections. To this end, we generate high-quality pseudo annotations based on the foundation model SAM. We thoroughly study various object priors to generate prompts for SAM, explicitly focusing the foundation model on objects. The strongest object priors were obtained by self-attention maps from self-supervised Vision Transformers, which we utilize for prompting SAM. Finally, the post-processed segments from SAM are used as pseudo annotations to train a standard instance segmentation system. Our approach shows strong generalization capabilities on COCO, LVIS, and ADE20k datasets and improves on the precision by up to 81.6% compared to the state-of-the-art. Source code is available at: this https URL

Title: Direct Judgement Preference Optimization

• Authors: Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, Shafiq Joty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14664
• Pdf URL: https://arxiv.org/pdf/2409.14664
• Copy Paste: [[2409.14664]] Direct Judgement Preference Optimization(https://arxiv.org/abs/2409.14664)
• Keywords: generative
• Abstract: Auto-evaluation is crucial for assessing response quality and offering feedback for model development. Recent studies have explored training large language models (LLMs) as generative judges to evaluate and critique other models' outputs. In this work, we investigate the idea of learning from both positive and negative data with preference optimization to enhance the evaluation capabilities of LLM judges across an array of different use cases. We achieve this by employing three approaches to collect the preference pairs for different use cases, each aimed at improving our generative judge from a different perspective. Our comprehensive study over a wide range of benchmarks demonstrates the effectiveness of our method. In particular, our generative judge achieves the best performance on 10 out of 13 benchmarks, outperforming strong baselines like GPT-4o and specialized judge models. Further analysis show that our judge model robustly counters inherent biases such as position and length bias, flexibly adapts to any evaluation protocol specified by practitioners, and provides helpful language feedback for improving downstream generator models.

Title: Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science

• Authors: Taihang Wang, Xiaoman Xu, Yimin Wang, Ye Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14673
• Pdf URL: https://arxiv.org/pdf/2409.14673
• Copy Paste: [[2409.14673]] Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science(https://arxiv.org/abs/2409.14673)
• Keywords: in-context
• Abstract: Real-world applications of large language models (LLMs) in computational social science (CSS) tasks primarily depend on the effectiveness of instruction tuning (IT) or in-context learning (ICL). While IT has shown highly effective at fine-tuning LLMs for various tasks, ICL offers a rapid alternative for task adaptation by learning from examples without explicit gradient updates. In this paper, we evaluate the classification performance of LLMs using IT versus ICL in few-shot CSS tasks. The experimental results indicate that ICL consistently outperforms IT in most CSS tasks. Additionally, we investigate the relationship between the increasing number of training samples and LLM performance. Our findings show that simply increasing the number of samples without considering their quality does not consistently enhance the performance of LLMs with either ICL or IT and can sometimes even result in a performance decline. Finally, we compare three prompting strategies, demonstrating that ICL is more effective than zero-shot and Chain-of-Thought (CoT). Our research highlights the significant advantages of ICL in handling CSS tasks in few-shot settings and emphasizes the importance of optimizing sample quality and prompting strategies to improve LLM classification performance. The code will be made available.

Title: Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections

• Authors: Ankit Dhiman, Manan Shah, Rishubh Parihar, Yash Bhalgat, Lokesh R Boregowda, R Venkatesh Babu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14677
• Pdf URL: https://arxiv.org/pdf/2409.14677
• Copy Paste: [[2409.14677]] Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections(https://arxiv.org/abs/2409.14677)
• Keywords: diffusion, generative
• Abstract: We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We formulate this problem as an image inpainting task, allowing for more user control over the placement of mirrors during the generation process. To enable this, we create SynMirror, a large-scale dataset of diverse synthetic scenes with objects placed in front of mirrors. SynMirror contains around 198K samples rendered from 66K unique 3D objects, along with their associated depth maps, normal maps and instance-wise segmentation masks, to capture relevant geometric properties of the scene. Using this dataset, we propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality geometrically consistent and photo-realistic mirror reflections given an input image and a mask depicting the mirror region. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis. To the best of our knowledge, we are the first to successfully tackle the challenging problem of generating controlled and faithful mirror reflections of an object in a scene using diffusion based models. SynMirror and MirrorFusion open up new avenues for image editing and augmented reality applications for practitioners and researchers alike.

Title: Adaptive and Robust Watermark for Generative Tabular Data

• Authors: Dung Daniel Ngo, Daniel Scott, Saheed Obitayo, Vamsi K. Potluru, Manuela Veloso
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14700
• Pdf URL: https://arxiv.org/pdf/2409.14700
• Copy Paste: [[2409.14700]] Adaptive and Robust Watermark for Generative Tabular Data(https://arxiv.org/abs/2409.14700)
• Keywords: generative
• Abstract: Recent developments in generative models have demonstrated its ability to create high-quality synthetic data. However, the pervasiveness of synthetic content online also brings forth growing concerns that it can be used for malicious purposes. To ensure the authenticity of the data, watermarking techniques have recently emerged as a promising solution due to their strong statistical guarantees. In this paper, we propose a flexible and robust watermarking mechanism for generative tabular data. Specifically, a data provider with knowledge of the downstream tasks can partition the feature space into pairs of $(key, value)$ columns. Within each pair, the data provider first uses elements in the $key$ column to generate a randomized set of ''green'' intervals, then encourages elements of the $value$ column to be in one of these ''green'' intervals. We show theoretically and empirically that the watermarked datasets (i) have negligible impact on the data quality and downstream utility, (ii) can be efficiently detected, and (iii) are robust against multiple attacks commonly observed in data science.

Title: ControlEdit: A MultiModal Local Clothing Image Editing Method

• Authors: Di Cheng, YingJie Shi, ShiXin Sun, JiaFu Zhang, WeiJing Wang, Yu Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14720
• Pdf URL: https://arxiv.org/pdf/2409.14720
• Copy Paste: [[2409.14720]] ControlEdit: A MultiModal Local Clothing Image Editing Method(https://arxiv.org/abs/2409.14720)
• Keywords: diffusion, self-supervised
• Abstract: Multimodal clothing image editing refers to the precise adjustment and modification of clothing images using data such as textual descriptions and visual images as control conditions, which effectively improves the work efficiency of designers and reduces the threshold for user design. In this paper, we propose a new image editing method ControlEdit, which transfers clothing image editing to multimodal-guided local inpainting of clothing images. We address the difficulty of collecting real image datasets by leveraging the self-supervised learning approach. Based on this learning approach, we extend the channels of the feature extraction network to ensure consistent clothing image style before and after editing, and we design an inverse latent loss function to achieve soft control over the content of non-edited areas. In addition, we adopt Blended Latent Diffusion as the sampling method to make the editing boundaries transition naturally and enforce consistency of non-edited area content. Extensive experiments demonstrate that ControlEdit surpasses baseline algorithms in both qualitative and quantitative evaluations.

Title: Research on Dynamic Data Flow Anomaly Detection based on Machine Learning

• Authors: Liyang Wang, Yu Cheng, Hao Gong, Jiacheng Hu, Xirui Tang, Iris Li
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14796
• Pdf URL: https://arxiv.org/pdf/2409.14796
• Copy Paste: [[2409.14796]] Research on Dynamic Data Flow Anomaly Detection based on Machine Learning(https://arxiv.org/abs/2409.14796)
• Keywords: anomaly
• Abstract: The sophistication and diversity of contemporary cyberattacks have rendered the use of proxies, gateways, firewalls, and encrypted tunnels as a standalone defensive strategy inadequate. Consequently, the proactive identification of data anomalies has emerged as a prominent area of research within the field of data security. The majority of extant studies concentrate on sample equilibrium data, with the consequence that the detection effect is not optimal in the context of unbalanced data. In this study, the unsupervised learning method is employed to identify anomalies in dynamic data flows. Initially, multi-dimensional features are extracted from real-time data, and a clustering algorithm is utilised to analyse the patterns of the data. This enables the potential outliers to be automatically identified. By clustering similar data, the model is able to detect data behaviour that deviates significantly from normal traffic without the need for labelled data. The results of the experiments demonstrate that the proposed method exhibits high accuracy in the detection of anomalies across a range of scenarios. Notably, it demonstrates robust and adaptable performance, particularly in the context of unbalanced data.

Title: VARADE: a Variational-based AutoRegressive model for Anomaly Detection on the Edge

• Authors: Alessio Mascolini, Sebastiano Gaiardelli, Francesco Ponzio, Nicola Dall'Ora, Enrico Macii, Sara Vinco, Santa Di Cataldo, Franco Fummi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14816
• Pdf URL: https://arxiv.org/pdf/2409.14816
• Copy Paste: [[2409.14816]] VARADE: a Variational-based AutoRegressive model for Anomaly Detection on the Edge(https://arxiv.org/abs/2409.14816)
• Keywords: anomaly
• Abstract: Detecting complex anomalies on massive amounts of data is a crucial task in Industry 4.0, best addressed by deep learning. However, available solutions are computationally demanding, requiring cloud architectures prone to latency and bandwidth issues. This work presents VARADE, a novel solution implementing a light autoregressive framework based on variational inference, which is best suited for real-time execution on the edge. The proposed approach was validated on a robotic arm, part of a pilot production line, and compared with several state-of-the-art algorithms, obtaining the best trade-off between anomaly detection accuracy, power consumption and inference frequency on two different edge platforms.

Title: GroCo: Ground Constraint for Metric Self-Supervised Monocular Depth

• Authors: Aurélien Cecille, Stefan Duffner, Franck Davoine, Thibault Neveu, Rémi Agier
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.14850
• Pdf URL: https://arxiv.org/pdf/2409.14850
• Copy Paste: [[2409.14850]] GroCo: Ground Constraint for Metric Self-Supervised Monocular Depth(https://arxiv.org/abs/2409.14850)
• Keywords: self-supervised
• Abstract: Monocular depth estimation has greatly improved in the recent years but models predicting metric depth still struggle to generalize across diverse camera poses and datasets. While recent supervised methods mitigate this issue by leveraging ground prior information at inference, their adaptability to self-supervised settings is limited due to the additional challenge of scale recovery. Addressing this gap, we propose in this paper a novel constraint on ground areas designed specifically for the self-supervised paradigm. This mechanism not only allows to accurately recover the scale but also ensures coherence between the depth prediction and the ground prior. Experimental results show that our method surpasses existing scale recovery techniques on the KITTI benchmark and significantly enhances model generalization capabilities. This improvement can be observed by its more robust performance across diverse camera rotations and its adaptability in zero-shot conditions with previously unseen driving datasets such as DDAD.

Title: Disentanglement with Factor Quantized Variational Autoencoders

• Authors: Gulcin Baykal, Melih Kandemir, Gozde Unal
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14851
• Pdf URL: https://arxiv.org/pdf/2409.14851
• Copy Paste: [[2409.14851]] Disentanglement with Factor Quantized Variational Autoencoders(https://arxiv.org/abs/2409.14851)
• Keywords: generative
• Abstract: Disentangled representation learning aims to represent the underlying generative factors of a dataset in a latent representation independently of one another. In our work, we propose a discrete variational autoencoder (VAE) based model where the ground truth information about the generative factors are not provided to the model. We demonstrate the advantages of learning discrete representations over learning continuous representations in facilitating disentanglement. Furthermore, we propose incorporating an inductive bias into the model to further enhance disentanglement. Precisely, we propose scalar quantization of the latent variables in a latent representation with scalar values from a global codebook, and we add a total correlation term to the optimization as an inductive bias. Our method called FactorQVAE is the first method that combines optimization based disentanglement approaches with discrete representation learning, and it outperforms the former disentanglement methods in terms of two disentanglement metrics (DCI and InfoMEC) while improving the reconstruction performance. Our code can be found at \url{this https URL}.

Title: Testing Dependency of Weighted Random Graphs

• Authors: Mor Oren, Vered Paslev, Wasim Huleihel
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2409.14870
• Pdf URL: https://arxiv.org/pdf/2409.14870
• Copy Paste: [[2409.14870]] Testing Dependency of Weighted Random Graphs(https://arxiv.org/abs/2409.14870)
• Keywords: generative
• Abstract: In this paper, we study the task of detecting the edge dependency between two weighted random graphs. We formulate this task as a simple hypothesis testing problem, where under the null hypothesis, the two observed graphs are statistically independent, whereas under the alternative, the edges of one graph are dependent on the edges of a randomly vertex-permuted version of the other graph. For general edge-weights distributions, we establish thresholds at which optimal testing is information-theoretically impossible and possible, as a function of the total number of nodes in the observed graphs and the generative distributions of the weights. Finally, we observe a statistical-computational gap in our problem, and we provide evidence that this is fundamental using the framework of low-degree polynomials.

Title: Advancing Video Quality Assessment for AIGC

• Authors: Xinli Yue, Jianhui Sun, Han Kong, Liangchao Yao, Tianyi Wang, Lei Li, Fengyun Rao, Jing Lv, Fan Xia, Yuetang Deng, Qian Wang, Lingchen Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14888
• Pdf URL: https://arxiv.org/pdf/2409.14888
• Copy Paste: [[2409.14888]] Advancing Video Quality Assessment for AIGC(https://arxiv.org/abs/2409.14888)
• Keywords: generative
• Abstract: In recent years, AI generative models have made remarkable progress across various domains, including text generation, image generation, and video generation. However, assessing the quality of text-to-video generation is still in its infancy, and existing evaluation frameworks fall short when compared to those for natural videos. Current video quality assessment (VQA) methods primarily focus on evaluating the overall quality of natural videos and fail to adequately account for the substantial quality discrepancies between frames in generated videos. To address this issue, we propose a novel loss function that combines mean absolute error with cross-entropy loss to mitigate inter-frame quality inconsistencies. Additionally, we introduce the innovative S2CNet technique to retain critical content, while leveraging adversarial training to enhance the model's generalization capabilities. Experimental results demonstrate that our method outperforms existing VQA techniques on the AIGC Video dataset, surpassing the previous state-of-the-art by 3.1% in terms of PLCC.

Title: Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization

• Authors: Aseem Srivastava, Smriti Joshi, Tanmoy Chakraborty, Md Shad Akhtar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14907
• Pdf URL: https://arxiv.org/pdf/2409.14907
• Copy Paste: [[2409.14907]] Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization(https://arxiv.org/abs/2409.14907)
• Keywords: generative
• Abstract: In mental health counseling, condensing dialogues into concise and relevant summaries (aka counseling notes) holds pivotal significance. Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks; however, their adaptation to domain-specific intricacies remains challenging, especially within mental health contexts. Unlike standard LLMs, mental health experts first plan to apply domain knowledge in writing summaries. Our work enhances LLMs' ability by introducing a novel planning engine to orchestrate structuring knowledge alignment. To achieve high-order planning, we divide knowledge encapsulation into two major phases: (i) holding dialogue structure and (ii) incorporating domain-specific knowledge. We employ a planning engine on Llama-2, resulting in a novel framework, PIECE. Our proposed system employs knowledge filtering-cum-scaffolding to encapsulate domain knowledge. Additionally, PIECE leverages sheaf convolution learning to enhance its understanding of the dialogue's structural nuances. We compare PIECE with 14 baseline methods and observe a significant improvement across ROUGE and Bleurt scores. Further, expert evaluation and analyses validate the generation quality to be effective, sometimes even surpassing the gold standard. We further benchmark PIECE with other LLMs and report improvement, including Llama-2 (+2.72%), Mistral (+2.04%), and Zephyr (+1.59%), to justify the generalizability of the planning engine.

Title: Improving Adversarial Robustness for 3D Point Cloud Recognition at Test-Time through Purified Self-Training

• Authors: Jinpeng Lin, Xulei Yang, Tianrui Li, Xun Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14940
• Pdf URL: https://arxiv.org/pdf/2409.14940
• Copy Paste: [[2409.14940]] Improving Adversarial Robustness for 3D Point Cloud Recognition at Test-Time through Purified Self-Training(https://arxiv.org/abs/2409.14940)
• Keywords: generative
• Abstract: Recognizing 3D point cloud plays a pivotal role in many real-world applications. However, deploying 3D point cloud deep learning model is vulnerable to adversarial attacks. Despite many efforts into developing robust model by adversarial training, they may become less effective against emerging attacks. This limitation motivates the development of adversarial purification which employs generative model to mitigate the impact of adversarial attacks. In this work, we highlight the remaining challenges from two perspectives. First, the purification based method requires retraining the classifier on purified samples which introduces additional computation overhead. Moreover, in a more realistic scenario, testing samples arrives in a streaming fashion and adversarial samples are not isolated from clean samples. These challenges motivates us to explore dynamically update model upon observing testing samples. We proposed a test-time purified self-training strategy to achieve this objective. Adaptive thresholding and feature distribution alignment are introduced to improve the robustness of self-training. Extensive results on different adversarial attacks suggest the proposed method is complementary to purification based method in handling continually changing adversarial attacks on the testing data stream.

Title: DepthART: Monocular Depth Estimation as Autoregressive Refinement Task

• Authors: Bulat Gabdullin, Nina Konovalova, Nikolay Patakin, Dmitry Senushkin, Anton Konushin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15010
• Pdf URL: https://arxiv.org/pdf/2409.15010
• Copy Paste: [[2409.15010]] DepthART: Monocular Depth Estimation as Autoregressive Refinement Task(https://arxiv.org/abs/2409.15010)
• Keywords: diffusion, generative
• Abstract: Despite recent success in discriminative approaches in monocular depth estimation its quality remains limited by training datasets. Generative approaches mitigate this issue by leveraging strong priors derived from training on internet-scale datasets. Recent studies have demonstrated that large text-to-image diffusion models achieve state-of-the-art results in depth estimation when fine-tuned on small depth datasets. Concurrently, autoregressive generative approaches, such as the Visual AutoRegressive modeling~(VAR), have shown promising results in conditioned image synthesis. Following the visual autoregressive modeling paradigm, we introduce the first autoregressive depth estimation model based on the visual autoregressive transformer. Our primary contribution is DepthART -- a novel training method formulated as Depth Autoregressive Refinement Task. Unlike the original VAR training procedure, which employs static targets, our method utilizes a dynamic target formulation that enables model self-refinement and incorporates multi-modal guidance during training. Specifically, we use model predictions as inputs instead of ground truth token maps during training, framing the objective as residual minimization. Our experiments demonstrate that the proposed training approach significantly outperforms visual autoregressive modeling via next-scale prediction in the depth estimation task. The Visual Autoregressive Transformer trained with our approach on Hypersim achieves superior results on a set of unseen benchmarks compared to other generative and discriminative baselines.

Title: Generative LLM Powered Conversational AI Application for Personalized Risk Assessment: A Case Study in COVID-19

• Authors: Mohammad Amin Roshani, Xiangyu Zhou, Yao Qiang, Srinivasan Suresh, Steve Hicks, Usha Sethuraman, Dongxiao Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15027
• Pdf URL: https://arxiv.org/pdf/2409.15027
• Copy Paste: [[2409.15027]] Generative LLM Powered Conversational AI Application for Personalized Risk Assessment: A Case Study in COVID-19(https://arxiv.org/abs/2409.15027)
• Keywords: generative
• Abstract: Large language models (LLMs) have shown remarkable capabilities in various natural language tasks and are increasingly being applied in healthcare domains. This work demonstrates a new LLM-powered disease risk assessment approach via streaming human-AI conversation, eliminating the need for programming required by traditional machine learning approaches. In a COVID-19 severity risk assessment case study, we fine-tune pre-trained generative LLMs (e.g., Llama2-7b and Flan-t5-xl) using a few shots of natural language examples, comparing their performance with traditional classifiers (i.e., Logistic Regression, XGBoost, Random Forest) that are trained de novo using tabular data across various experimental settings. We develop a mobile application that uses these fine-tuned LLMs as its generative AI (GenAI) core to facilitate real-time interaction between clinicians and patients, providing no-code risk assessment through conversational interfaces. This integration not only allows for the use of streaming Questions and Answers (QA) as inputs but also offers personalized feature importance analysis derived from the LLM's attention layers, enhancing the interpretability of risk assessments. By achieving high Area Under the Curve (AUC) scores with a limited number of fine-tuning samples, our results demonstrate the potential of generative LLMs to outperform discriminative classification methods in low-data regimes, highlighting their real-world adaptability and effectiveness. This work aims to fill the existing gap in leveraging generative LLMs for interactive no-code risk assessment and to encourage further research in this emerging field.

Title: Anomaly Detection from a Tensor Train Perspective

• Authors: Alejandro Mata Ali, Aitor Moreno Fdez. de Leceta, Jorge López Rubio
• Subjects: cs.LG, cs.CR, cs.ET, cs.IT, quant-ph
• Abstract URL: https://arxiv.org/abs/2409.15030
• Pdf URL: https://arxiv.org/pdf/2409.15030
• Copy Paste: [[2409.15030]] Anomaly Detection from a Tensor Train Perspective(https://arxiv.org/abs/2409.15030)
• Keywords: anomaly
• Abstract: We present a series of algorithms in tensor networks for anomaly detection in datasets, by using data compression in a Tensor Train representation. These algorithms consist of preserving the structure of normal data in compression and deleting the structure of anomalous data. The algorithms can be applied to any tensor network representation. We test the effectiveness of the methods with digits and Olivetti faces datasets and a cybersecurity dataset to determine cyber-attacks.

Title: FisheyeDepth: A Real Scale Self-Supervised Depth Estimation Model for Fisheye Camera

• Authors: Guoyang Zhao, Yuxuan Liu, Weiqing Qi, Fulong Ma, Ming Liu, Jun Ma
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.15054
• Pdf URL: https://arxiv.org/pdf/2409.15054
• Copy Paste: [[2409.15054]] FisheyeDepth: A Real Scale Self-Supervised Depth Estimation Model for Fisheye Camera(https://arxiv.org/abs/2409.15054)
• Keywords: self-supervised
• Abstract: Accurate depth estimation is crucial for 3D scene comprehension in robotics and autonomous vehicles. Fisheye cameras, known for their wide field of view, have inherent geometric benefits. However, their use in depth estimation is restricted by a scarcity of ground truth data and image distortions. We present FisheyeDepth, a self-supervised depth estimation model tailored for fisheye cameras. We incorporate a fisheye camera model into the projection and reprojection stages during training to handle image distortions, thereby improving depth estimation accuracy and training stability. Furthermore, we incorporate real-scale pose information into the geometric projection between consecutive frames, replacing the poses estimated by the conventional pose network. Essentially, this method offers the necessary physical depth for robotic tasks, and also streamlines the training and inference procedures. Additionally, we devise a multi-channel output strategy to improve robustness by adaptively fusing features at various scales, which reduces the noise from real pose data. We demonstrate the superior performance and robustness of our model in fisheye image depth estimation through evaluations on public datasets and real-world scenarios. The project website is available at: this https URL.

Title: Scientific Cross-Document Coreference and Hierarchy with Definition-Augmented Relational Reasoning

• Authors: Lior Forer, Tom Hope
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15113
• Pdf URL: https://arxiv.org/pdf/2409.15113
• Copy Paste: [[2409.15113]] Scientific Cross-Document Coreference and Hierarchy with Definition-Augmented Relational Reasoning(https://arxiv.org/abs/2409.15113)
• Keywords: in-context
• Abstract: We address the fundamental task of inferring cross-document coreference and hierarchy in scientific texts, which has important applications in knowledge graph construction, search, recommendation and discovery. LLMs still struggle when faced with many long-tail technical concepts with nuanced variations. We present a novel method which generates context-dependent definitions of concept mentions by retrieving full-text literature, and uses the definitions to enhance detection of cross-document mention relations. We further generate relational definitions, which describe how two concept mentions are related or different, and design an efficient re-ranking approach to address the combinatorial explosion involved in inferring links across papers. In both fine-tuning and in-context learning settings we achieve large gains in performance. We provide analysis of generated definitions, shedding light on the relational reasoning ability of LLMs over fine-grained scientific texts.

Title: Diffusion-based RGB-D Semantic Segmentation with Deformable Attention Transformer

• Authors: Minh Bui, Kostas Alexis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15117
• Pdf URL: https://arxiv.org/pdf/2409.15117
• Copy Paste: [[2409.15117]] Diffusion-based RGB-D Semantic Segmentation with Deformable Attention Transformer(https://arxiv.org/abs/2409.15117)
• Keywords: diffusion, generative
• Abstract: Vision-based perception and reasoning is essential for scene understanding in any autonomous system. RGB and depth images are commonly used to capture both the semantic and geometric features of the environment. Developing methods to reliably interpret this data is critical for real-world applications, where noisy measurements are often unavoidable. In this work, we introduce a diffusion-based framework to address the RGB-D semantic segmentation problem. Additionally, we demonstrate that utilizing a Deformable Attention Transformer as the encoder to extract features from depth images effectively captures the characteristics of invalid regions in depth measurements. Our generative framework shows a greater capacity to model the underlying distribution of RGB-D images, achieving robust performance in challenging scenarios with significantly less training time compared to discriminative methods. Experimental results indicate that our approach achieves State-of-the-Art performance on both the NYUv2 and SUN-RGBD datasets in general and especially in the most challenging of their image data. Our project page will be available at this https URL

Title: MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning

• Authors: Yue Han, Junwei Zhu, Yuxiang Feng, Xiaozhong Ji, Keke He, Xiangtai Li, zhucun xue, Yong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15179
• Pdf URL: https://arxiv.org/pdf/2409.15179
• Copy Paste: [[2409.15179]] MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning(https://arxiv.org/abs/2409.15179)
• Keywords: diffusion
• Abstract: Current diffusion-based face animation methods generally adopt a ReferenceNet (a copy of U-Net) and a large amount of curated self-acquired data to learn appearance features, as robust appearance features are vital for ensuring temporal stability. However, when trained on public datasets, the results often exhibit a noticeable performance gap in image quality and temporal consistency. To address this issue, we meticulously examine the essential appearance features in the facial animation tasks, which include motion-agnostic (e.g., clothing, background) and motion-related (e.g., facial details) texture components, along with high-level discriminative identity features. Drawing from this analysis, we introduce a Motion-Identity Modulated Appearance Learning Module (MIA) that modulates CLIP features at both motion and identity levels. Additionally, to tackle the semantic/ color discontinuities between clips, we design an Inter-clip Affinity Learning Module (ICA) to model temporal relationships across clips. Our method achieves precise facial motion control (i.e., expressions and gaze), faithful identity preservation, and generates animation videos that maintain both intra/inter-clip temporal consistency. Moreover, it easily adapts to various modalities of driving sources. Extensive experiments demonstrate the superiority of our method.

Title: PALLM: Evaluating and Enhancing PALLiative Care Conversations with Large Language Models

• Authors: Zhiyuan Wang, Fangxu Yuan, Virginia LeBaron, Tabor Flickinger, Laura E. Barnes
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.15188
• Pdf URL: https://arxiv.org/pdf/2409.15188
• Copy Paste: [[2409.15188]] PALLM: Evaluating and Enhancing PALLiative Care Conversations with Large Language Models(https://arxiv.org/abs/2409.15188)
• Keywords: in-context
• Abstract: Effective patient-provider communication is crucial in clinical care, directly impacting patient outcomes and quality of life. Traditional evaluation methods, such as human ratings, patient feedback, and provider self-assessments, are often limited by high costs and scalability issues. Although existing natural language processing (NLP) techniques show promise, they struggle with the nuances of clinical communication and require sensitive clinical data for training, reducing their effectiveness in real-world applications. Emerging large language models (LLMs) offer a new approach to assessing complex communication metrics, with the potential to advance the field through integration into passive sensing and just-in-time intervention systems. This study explores LLMs as evaluators of palliative care communication quality, leveraging their linguistic, in-context learning, and reasoning capabilities. Specifically, using simulated scripts crafted and labeled by healthcare professionals, we test proprietary models (e.g., GPT-4) and fine-tune open-source LLMs (e.g., LLaMA2) with a synthetic dataset generated by GPT-4 to evaluate clinical conversations, to identify key metrics such as `understanding' and `empathy'. Our findings demonstrated LLMs' superior performance in evaluating clinical communication, providing actionable feedback with reasoning, and demonstrating the feasibility and practical viability of developing in-house LLMs. This research highlights LLMs' potential to enhance patient-provider interactions and lays the groundwork for downstream steps in developing LLM-empowered clinical health systems.

Title: MotifDisco: Motif Causal Discovery For Time Series Motifs

• Authors: Josephine Lamp, Mark Derdzinski, Christopher Hannemann, Sam Hatfield, Joost van der Linden
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15219
• Pdf URL: https://arxiv.org/pdf/2409.15219
• Copy Paste: [[2409.15219]] MotifDisco: Motif Causal Discovery For Time Series Motifs(https://arxiv.org/abs/2409.15219)
• Keywords: generative, anomaly
• Abstract: Many time series, particularly health data streams, can be best understood as a sequence of phenomenon or events, which we call motifs. A time series motif is a short trace segment which may implicitly capture an underlying phenomenon within the time series. Specifically, we focus on glucose traces collected from continuous glucose monitors (CGMs), which inherently contain motifs representing underlying human behaviors such as eating and exercise. The ability to identify and quantify causal relationships amongst motifs can provide a mechanism to better understand and represent these patterns, useful for improving deep learning and generative models and for advanced technology development (e.g., personalized coaching and artificial insulin delivery systems). However, no previous work has developed causal discovery methods for time series motifs. Therefore, in this paper we develop MotifDisco (motif disco-very of causality), a novel causal discovery framework to learn causal relations amongst motifs from time series traces. We formalize a notion of Motif Causality (MC), inspired from Granger Causality and Transfer Entropy, and develop a Graph Neural Network-based framework that learns causality between motifs by solving an unsupervised link prediction problem. We also integrate MC with three model use cases of forecasting, anomaly detection and clustering, to showcase the use of MC as a building block for other downstream tasks. Finally, we evaluate our framework and find that Motif Causality provides a significant performance improvement in all use cases.

Title: ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models

• Authors: Sombit Dey, Jan-Nico Zaech, Nikolay Nikolov, Luc Van Gool, Danda Pani Paudel
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.15250
• Pdf URL: https://arxiv.org/pdf/2409.15250
• Copy Paste: [[2409.15250]] ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models(https://arxiv.org/abs/2409.15250)
• Keywords: foundation model
• Abstract: Recent progress in large language models and access to large-scale robotic datasets has sparked a paradigm shift in robotics models transforming them into generalists able to adapt to various tasks, scenes, and robot modalities. A large step for the community are open Vision Language Action models which showcase strong performance in a wide variety of tasks. In this work, we study the visual generalization capabilities of three existing robotic foundation models, and propose a corresponding evaluation framework. Our study shows that the existing models do not exhibit robustness to visual out-of-domain scenarios. This is potentially caused by limited variations in the training data and/or catastrophic forgetting, leading to domain limitations in the vision foundation models. We further explore OpenVLA, which uses two pre-trained vision foundation models and is, therefore, expected to generalize to out-of-domain experiments. However, we showcase catastrophic forgetting by DINO-v2 in OpenVLA through its failure to fulfill the task of depth regression. To overcome the aforementioned issue of visual catastrophic forgetting, we propose a gradual backbone reversal approach founded on model merging. This enables OpenVLA which requires the adaptation of the visual backbones during initial training -- to regain its visual generalization ability. Regaining this capability enables our ReVLA model to improve over OpenVLA by a factor of 77% and 66% for grasping and lifting in visual OOD tasks .

Title: S$^2$AG-Vid: Enhancing Multi-Motion Alignment in Video Diffusion Models via Spatial and Syntactic Attention-Based Guidance

• Authors: Yuanhang Li, Qi Mao, Lan Chen, Zhen Fang, Lei Tian, Xinyan Xiao, Libiao Jin, Hua Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15259
• Pdf URL: https://arxiv.org/pdf/2409.15259
• Copy Paste: [[2409.15259]] S$^2$AG-Vid: Enhancing Multi-Motion Alignment in Video Diffusion Models via Spatial and Syntactic Attention-Based Guidance(https://arxiv.org/abs/2409.15259)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-video (T2V) generation using diffusion models have garnered significant attention. However, existing T2V models primarily focus on simple scenes featuring a single object performing a single motion. Challenges arise in scenarios involving multiple objects with distinct motions, often leading to incorrect video-text alignment between subjects and their corresponding motions. To address this challenge, we propose \textbf{S$^2$AG-Vid}, a training-free inference-stage optimization method that improves the alignment of multiple objects with their corresponding motions in T2V models. S$^2$AG-Vid initially applies a spatial position-based, cross-attention (CA) constraint in the early stages of the denoising process, facilitating multiple nouns distinctly attending to the correct subject regions. To enhance the motion-subject binding, we implement a syntax-guided contrastive constraint in the subsequent denoising phase, aimed at improving the correlations between the CA maps of verbs and their corresponding nouns.Both qualitative and quantitative evaluations demonstrate that the proposed framework significantly outperforms baseline approaches, producing higher-quality videos with improved subject-motion consistency.

Title: UDA-Bench: Revisiting Common Assumptions in Unsupervised Domain Adaptation Using a Standardized Framework

• Authors: Tarun Kalluri, Sreyas Ravichandran, Manmohan Chandraker
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15264
• Pdf URL: https://arxiv.org/pdf/2409.15264
• Copy Paste: [[2409.15264]] UDA-Bench: Revisiting Common Assumptions in Unsupervised Domain Adaptation Using a Standardized Framework(https://arxiv.org/abs/2409.15264)
• Keywords: self-supervised
• Abstract: In this work, we take a deeper look into the diverse factors that influence the efficacy of modern unsupervised domain adaptation (UDA) methods using a large-scale, controlled empirical study. To facilitate our analysis, we first develop UDA-Bench, a novel PyTorch framework that standardizes training and evaluation for domain adaptation enabling fair comparisons across several UDA methods. Using UDA-Bench, our comprehensive empirical study into the impact of backbone architectures, unlabeled data quantity, and pre-training datasets reveals that: (i) the benefits of adaptation methods diminish with advanced backbones, (ii) current methods underutilize unlabeled data, and (iii) pre-training data significantly affects downstream adaptation in both supervised and self-supervised settings. In the context of unsupervised adaptation, these observations uncover several novel and surprising properties, while scientifically validating several others that were often considered empirical heuristics or practitioner intuitions in the absence of a standardized training and evaluation framework. The UDA-Bench framework and trained models are publicly available at this https URL.

Title: MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors

• Authors: Yehonathan Litman, Or Patashnik, Kangle Deng, Aviral Agrawal, Rushikesh Zawar, Fernando De la Torre, Shubham Tulsiani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15273
• Pdf URL: https://arxiv.org/pdf/2409.15273
• Copy Paste: [[2409.15273]] MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors(https://arxiv.org/abs/2409.15273)
• Keywords: diffusion
• Abstract: Recent works in inverse rendering have shown promise in using multi-view images of an object to recover shape, albedo, and materials. However, the recovered components often fail to render accurately under new lighting conditions due to the intrinsic challenge of disentangling albedo and material properties from input images. To address this challenge, we introduce MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that incorporates a 2D prior on texture and material properties. We present StableMaterial, a 2D diffusion model prior that refines multi-lit data to estimate the most likely albedo and material from given input appearances. This model is trained on albedo, material, and relit image data derived from a curated dataset of approximately ~12K artist-designed synthetic Blender objects called BlenderVault. we incorporate this diffusion prior with an inverse rendering framework where we use score distillation sampling (SDS) to guide the optimization of the albedo and materials, improving relighting performance in comparison with previous work. We validate MaterialFusion's relighting performance on 4 datasets of synthetic and real objects under diverse illumination conditions, showing our diffusion-aided approach significantly improves the appearance of reconstructed objects under novel lighting conditions. We intend to publicly release our BlenderVault dataset to support further research in this field.

Title: PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions

• Authors: Weifeng Lin, Xinyu Wei, Renrui Zhang, Le Zhuo, Shitian Zhao, Siyuan Huang, Junlin Xie, Yu Qiao, Peng Gao, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15278
• Pdf URL: https://arxiv.org/pdf/2409.15278
• Copy Paste: [[2409.15278]] PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions(https://arxiv.org/abs/2409.15278)
• Keywords: diffusion, foundation model, generative
• Abstract: This paper presents a versatile image-to-image visual assistant, PixWizard, designed for image generation, manipulation, and translation based on free-from language instructions. To this end, we tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction templates in natural language, we comprehensively include a large set of diverse vision tasks such as text-to-image generation, image restoration, image grounding, dense image prediction, image editing, controllable generation, inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with a flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes. The model also incorporates structure-aware and semantic-aware guidance to facilitate effective fusion of information from the input image. Our experiments demonstrate that PixWizard not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits promising generalization capabilities with unseen tasks and human instructions. The code and related resources are available at this https URL