2024-10-17

Title: Dual-frame Fluid Motion Estimation with Test-time Optimization and Zero-divergence Loss

• Authors: Yifei Zhang, Huan-ang Gao, Zhou Jiang, Hao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11934
• Pdf URL: https://arxiv.org/pdf/2410.11934
• Copy Paste: [[2410.11934]] Dual-frame Fluid Motion Estimation with Test-time Optimization and Zero-divergence Loss(https://arxiv.org/abs/2410.11934)
• Keywords: self-supervised
• Abstract: 3D particle tracking velocimetry (PTV) is a key technique for analyzing turbulent flow, one of the most challenging computational problems of our century. At the core of 3D PTV is the dual-frame fluid motion estimation algorithm, which tracks particles across two consecutive frames. Recently, deep learning-based methods have achieved impressive accuracy in dual-frame fluid motion estimation; however, they heavily depend on large volumes of labeled data. In this paper, we introduce a new method that is completely self-supervised and notably outperforms its fully-supervised counterparts while requiring only 1% of the training samples (without labels) used by previous methods. Our method features a novel zero-divergence loss that is specific to the domain of turbulent flow. Inspired by the success of splat operation in high-dimensional filtering and random fields, we propose a splat-based implementation for this loss which is both efficient and effective. The self-supervised nature of our method naturally supports test-time optimization, leading to the development of a tailored Dynamic Velocimetry Enhancer (DVE) module. We demonstrate that strong cross-domain robustness is achieved through test-time optimization on unseen leave-one-out synthetic domains and real physical/biological domains. Code, data and models are available at this https URL.

Title: CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning

• Authors: Qingqing Cao, Mahyar Najibi, Sachin Mehta
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.11963
• Pdf URL: https://arxiv.org/pdf/2410.11963
• Copy Paste: [[2410.11963]] CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning(https://arxiv.org/abs/2410.11963)
• Keywords: diffusion, foundation model
• Abstract: Pretraining robust vision or multimodal foundation models (e.g., CLIP) relies on large-scale datasets that may be noisy, potentially misaligned, and have long-tail distributions. Previous works have shown promising results in augmenting datasets by generating synthetic samples. However, they only support domain-specific ad hoc use cases (e.g., either image or text only, but not both), and are limited in data diversity due to a lack of fine-grained control over the synthesis process. In this paper, we design a \emph{controllable} image-text synthesis pipeline, CtrlSynth, for data-efficient and robust multimodal learning. The key idea is to decompose the visual semantics of an image into basic elements, apply user-specified control policies (e.g., remove, add, or replace operations), and recompose them to synthesize images or texts. The decompose and recompose feature in CtrlSynth allows users to control data synthesis in a fine-grained manner by defining customized control policies to manipulate the basic elements. CtrlSynth leverages the capabilities of pretrained foundation models such as large language models or diffusion models to reason and recompose basic elements such that synthetic samples are natural and composed in diverse ways. CtrlSynth is a closed-loop, training-free, and modular framework, making it easy to support different pretrained models. With extensive experiments on 31 datasets spanning different vision and vision-language tasks, we show that CtrlSynth substantially improves zero-shot classification, image-text retrieval, and compositional reasoning performance of CLIP models.

Title: A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection

• Authors: James Enouen, Mahito Sugiyama
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.11964
• Pdf URL: https://arxiv.org/pdf/2410.11964
• Copy Paste: [[2410.11964]] A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection(https://arxiv.org/abs/2410.11964)
• Keywords: generative
• Abstract: The log-linear model has received a significant amount of theoretical attention in previous decades and remains the fundamental tool used for learning probability distributions over discrete variables. Despite its large popularity in statistical mechanics and high-dimensional statistics, the vast majority of such energy-based modeling approaches only focus on the two-variable relationships, such as Boltzmann machines and Markov graphical models. Although these approaches have easier-to-solve structure learning problems and easier-to-optimize parametric distributions, they often ignore the rich structure which exists in the higher-order interactions between different variables. Using more recent tools from the field of information geometry, we revisit the classical formulation of the log-linear model with a focus on higher-order mode interactions, going beyond the 1-body modes of independent distributions and the 2-body modes of Boltzmann distributions. This perspective allows us to define a complete decomposition of the KL error. This then motivates the formulation of a sparse selection problem over the set of possible mode interactions. In the same way as sparse graph selection allows for better generalization, we find that our learned distributions are able to more efficiently use the finite amount of data which is available in practice. On both synthetic and real-world datasets, we demonstrate our algorithm's effectiveness in maximizing the log-likelihood for the generative task and also the ease of adaptability to the discriminative task of classification.

Title: DDIL: Improved Diffusion Distillation With Imitation Learning

• Authors: Risheek Garrepalli, Shweta Mahajan, Munawar Hayat, Fatih Porikli
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11971
• Pdf URL: https://arxiv.org/pdf/2410.11971
• Copy Paste: [[2410.11971]] DDIL: Improved Diffusion Distillation With Imitation Learning(https://arxiv.org/abs/2410.11971)
• Keywords: diffusion, generative
• Abstract: Diffusion models excel at generative modeling (e.g., text-to-image) but sampling requires multiple denoising network passes, limiting practicality. Efforts such as progressive distillation or consistency distillation have shown promise by reducing the number of passes at the expense of quality of the generated samples. In this work we identify co-variate shift as one of reason for poor performance of multi-step distilled models from compounding error at inference time. To address co-variate shift, we formulate diffusion distillation within imitation learning (DDIL) framework and enhance training distribution for distilling diffusion models on both data distribution (forward diffusion) and student induced distributions (backward diffusion). Training on data distribution helps to diversify the generations by preserving marginal data distribution and training on student distribution addresses compounding error by correcting covariate shift. In addition, we adopt reflected diffusion formulation for distillation and demonstrate improved performance, stable training across different distillation methods. We show that DDIL consistency improves on baseline algorithms of progressive distillation (PD), Latent consistency models (LCM) and Distribution Matching Distillation (DMD2).

Title: Beyond Labels: A Self-Supervised Framework with Masked Autoencoders and Random Cropping for Breast Cancer Subtype Classification

• Authors: Annalisa Chiocchetti, Marco Dossena, Christopher Irwin, Luigi Portinale
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12006
• Pdf URL: https://arxiv.org/pdf/2410.12006
• Copy Paste: [[2410.12006]] Beyond Labels: A Self-Supervised Framework with Masked Autoencoders and Random Cropping for Breast Cancer Subtype Classification(https://arxiv.org/abs/2410.12006)
• Keywords: self-supervised
• Abstract: This work contributes to breast cancer sub-type classification using histopathological images. We utilize masked autoencoders (MAEs) to learn a self-supervised embedding tailored for computer vision tasks in this domain. This embedding captures informative representations of histopathological data, facilitating feature learning without extensive labeled datasets. During pre-training, we investigate employing a random crop technique to generate a large dataset from WSIs automatically. Additionally, we assess the performance of linear probes for multi-class classification tasks of cancer sub-types using the representations learnt by the MAE. Our approach aims to achieve strong performance on downstream tasks by leveraging the complementary strengths of ViTs and autoencoders. We evaluate our model's performance on the BRACS dataset and compare it with existing benchmarks.

Title: Bias Similarity Across Large Language Models

• Authors: Hyejun Jeong, Shiqing Ma, Amir Houmansadr
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12010
• Pdf URL: https://arxiv.org/pdf/2410.12010
• Copy Paste: [[2410.12010]] Bias Similarity Across Large Language Models(https://arxiv.org/abs/2410.12010)
• Keywords: generative
• Abstract: Bias in machine learning models has been a chronic problem, especially as these models influence decision-making in human society. In generative AI, such as Large Language Models, the impact of bias is even more profound compared to the classification models. LLMs produce realistic and human-like content that users may unconsciously trust, which could perpetuate harmful stereotypes to the uncontrolled public. It becomes particularly concerning when utilized in journalism or education. While prior studies have explored and quantified bias in individual AI models, no work has yet compared bias similarity across different LLMs. To fill this gap, we take a comprehensive look at ten open- and closed-source LLMs from four model families, assessing the extent of biases through output distribution. Using two datasets-one containing 4k questions and another with one million questions for each of the four bias dimensions -- we measure functional similarity to understand how biases manifest across models. Our findings reveal that 1) fine-tuning does not significantly alter output distributions, which would limit its ability to mitigate bias, 2) LLMs within the same family tree do not produce similar output distributions, implying that addressing bias in one model could have limited implications for others in the same family, and 3) there is a possible risk of training data information leakage, raising concerns about privacy and data security. Our analysis provides insight into LLM behavior and highlights potential risks in real-world deployment.

Title: A Survey on Deep Tabular Learning

• Authors: Shriyank Somvanshi, Subasish Das, Syed Aaqib Javed, Gian Antariksa, Ahmed Hossain
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12034
• Pdf URL: https://arxiv.org/pdf/2410.12034
• Copy Paste: [[2410.12034]] A Survey on Deep Tabular Learning(https://arxiv.org/abs/2410.12034)
• Keywords: diffusion, self-supervised
• Abstract: Tabular data, widely used in industries like healthcare, finance, and transportation, presents unique challenges for deep learning due to its heterogeneous nature and lack of spatial structure. This survey reviews the evolution of deep learning models for tabular data, from early fully connected networks (FCNs) to advanced architectures like TabNet, SAINT, TabTranSELU, and MambaNet. These models incorporate attention mechanisms, feature embeddings, and hybrid architectures to address tabular data complexities. TabNet uses sequential attention for instance-wise feature selection, improving interpretability, while SAINT combines self-attention and intersample attention to capture complex interactions across features and data points, both advancing scalability and reducing computational overhead. Hybrid architectures such as TabTransformer and FT-Transformer integrate attention mechanisms with multi-layer perceptrons (MLPs) to handle categorical and numerical data, with FT-Transformer adapting transformers for tabular datasets. Research continues to balance performance and efficiency for large datasets. Graph-based models like GNN4TDL and GANDALF combine neural networks with decision trees or graph structures, enhancing feature representation and mitigating overfitting in small datasets through advanced regularization techniques. Diffusion-based models like the Tabular Denoising Diffusion Probabilistic Model (TabDDPM) generate synthetic data to address data scarcity, improving model robustness. Similarly, models like TabPFN and Ptab leverage pre-trained language models, incorporating transfer learning and self-supervised techniques into tabular tasks. This survey highlights key advancements and outlines future research directions on scalability, generalization, and interpretability in diverse tabular data applications.

Title: Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned

• Authors: Cassandra L. Jacobs, Loïc Grobol, Alvin Tsang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12057
• Pdf URL: https://arxiv.org/pdf/2410.12057
• Copy Paste: [[2410.12057]] Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned(https://arxiv.org/abs/2410.12057)
• Keywords: generative
• Abstract: In this work we compare the generative behavior at the next token prediction level in several language models by comparing them to human productions in the cloze task. We find that while large models trained for longer are typically better estimators of human productions, but they reliably under-estimate the probabilities of human responses, over-rank rare responses, under-rank top responses, and produce highly distinct semantic spaces. Altogether, this work demonstrates in a tractable, interpretable domain that LM generations can not be used as replacements of or models of the cloze task.

Title: De-jargonizing Science for Journalists with GPT-4: A Pilot Study

• Authors: Sachita Nishal, Eric Lee, Nicholas Diakopoulos
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.12069
• Pdf URL: https://arxiv.org/pdf/2410.12069
• Copy Paste: [[2410.12069]] De-jargonizing Science for Journalists with GPT-4: A Pilot Study(https://arxiv.org/abs/2410.12069)
• Keywords: generative
• Abstract: This study offers an initial evaluation of a human-in-the-loop system leveraging GPT-4 (a large language model or LLM), and Retrieval-Augmented Generation (RAG) to identify and define jargon terms in scientific abstracts, based on readers' self-reported knowledge. The system achieves fairly high recall in identifying jargon and preserves relative differences in readers' jargon identification, suggesting personalization as a feasible use-case for LLMs to support sense-making of complex information. Surprisingly, using only abstracts for context to generate definitions yields slightly more accurate and higher quality definitions than using RAG-based context from the fulltext of an article. The findings highlight the potential of generative AI for assisting science reporters, and can inform future work on developing tools to simplify dense documents.

Title: WeatherDG: LLM-assisted Procedural Weather Generation for Domain-Generalized Semantic Segmentation

• Authors: Chenghao Qian, Yuhu Guo, Yuhong Mo, Wenjing Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12075
• Pdf URL: https://arxiv.org/pdf/2410.12075
• Copy Paste: [[2410.12075]] WeatherDG: LLM-assisted Procedural Weather Generation for Domain-Generalized Semantic Segmentation(https://arxiv.org/abs/2410.12075)
• Keywords: diffusion, foundation model
• Abstract: In this work, we propose a novel approach, namely WeatherDG, that can generate realistic, weather-diverse, and driving-screen images based on the cooperation of two foundation models, i.e, Stable Diffusion (SD) and Large Language Model (LLM). Specifically, we first fine-tune the SD with source data, aligning the content and layout of generated samples with real-world driving scenarios. Then, we propose a procedural prompt generation method based on LLM, which can enrich scenario descriptions and help SD automatically generate more diverse, detailed images. In addition, we introduce a balanced generation strategy, which encourages the SD to generate high-quality objects of tailed classes under various weather conditions, such as riders and motorcycles. This segmentation-model-agnostic method can improve the generalization ability of existing models by additionally adapting them with the generated synthetic data. Experiments on three challenging datasets show that our method can significantly improve the segmentation performance of different state-of-the-art models on target domains. Notably, in the setting of ''Cityscapes to ACDC'', our method improves the baseline HRDA by 13.9% in mIoU.

Title: Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attacks

• Authors: Kevin Eykholt, Farhan Ahmed, Pratik Vaishnavi, Amir Rahmati
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.12076
• Pdf URL: https://arxiv.org/pdf/2410.12076
• Copy Paste: [[2410.12076]] Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attacks(https://arxiv.org/abs/2410.12076)
• Keywords: generative
• Abstract: The vulnerability of machine learning models in adversarial scenarios has garnered significant interest in the academic community over the past decade, resulting in a myriad of attacks and defenses. However, while the community appears to be overtly successful in devising new attacks across new contexts, the development of defenses has stalled. After a decade of research, we appear no closer to securing AI applications beyond additional training. Despite a lack of effective mitigations, AI development and its incorporation into existing systems charge full speed ahead with the rise of generative AI and large language models. Will our ineffectiveness in developing solutions to adversarial threats further extend to these new technologies? In this paper, we argue that overly permissive attack and overly restrictive defensive threat models have hampered defense development in the ML domain. Through the lens of adversarial evasion attacks against neural networks, we critically examine common attack assumptions, such as the ability to bypass any defense not explicitly built into the model. We argue that these flawed assumptions, seen as reasonable by the community based on paper acceptance, have encouraged the development of adversarial attacks that map poorly to real-world scenarios. In turn, new defenses evaluated against these very attacks are inadvertently required to be almost perfect and incorporated as part of the model. But do they need to? In practice, machine learning models are deployed as a small component of a larger system. We analyze adversarial machine learning from a system security perspective rather than an AI perspective and its implications for emerging AI paradigms.

Title: SplatPose+: Real-time Image-Based Pose-Agnostic 3D Anomaly Detection

• Authors: Yizhe Liu, Yan Song Hu, Yuhao Chen, John Zelek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12080
• Pdf URL: https://arxiv.org/pdf/2410.12080
• Copy Paste: [[2410.12080]] SplatPose+: Real-time Image-Based Pose-Agnostic 3D Anomaly Detection(https://arxiv.org/abs/2410.12080)
• Keywords: anomaly
• Abstract: Image-based Pose-Agnostic 3D Anomaly Detection is an important task that has emerged in industrial quality control. This task seeks to find anomalies from query images of a tested object given a set of reference images of an anomaly-free object. The challenge is that the query views (a.k.a poses) are unknown and can be different from the reference views. Currently, new methods such as OmniposeAD and SplatPose have emerged to bridge the gap by synthesizing pseudo reference images at the query views for pixel-to-pixel comparison. However, none of these methods can infer in real-time, which is critical in industrial quality control for massive production. For this reason, we propose SplatPose+, which employs a hybrid representation consisting of a Structure from Motion (SfM) model for localization and a 3D Gaussian Splatting (3DGS) model for Novel View Synthesis. Although our proposed pipeline requires the computation of an additional SfM model, it offers real-time inference speeds and faster training compared to SplatPose. Quality-wise, we achieved a new SOTA on the Pose-agnostic Anomaly Detection benchmark with the Multi-Pose Anomaly Detection (MAD-SIM) dataset.

Title: Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning

• Authors: Fengyu Gao, Ruida Zhou, Tianhao Wang, Cong Shen, Jing Yang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12085
• Pdf URL: https://arxiv.org/pdf/2410.12085
• Copy Paste: [[2410.12085]] Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning(https://arxiv.org/abs/2410.12085)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) rely on the contextual information embedded in examples/demonstrations to perform in-context learning (ICL). To mitigate the risk of LLMs potentially leaking private information contained in examples in the prompt, we introduce a novel data-adaptive differentially private algorithm called AdaDPSyn to generate synthetic examples from the private dataset and then use these synthetic examples to perform ICL. The objective of AdaDPSyn is to adaptively adjust the noise level in the data synthesis mechanism according to the inherent statistical properties of the data, thereby preserving high ICL accuracy while maintaining formal differential privacy guarantees. A key innovation in AdaDPSyn is the Precision-Focused Iterative Radius Reduction technique, which dynamically refines the aggregation radius - the scope of data grouping for noise addition - based on patterns observed in data clustering, thereby minimizing the amount of additive noise. We conduct extensive experiments on standard benchmarks and compare AdaDPSyn with DP few-shot generation algorithm (Tang et al., 2023). The experiments demonstrate that AdaDPSyn not only outperforms DP few-shot generation, but also maintains high accuracy levels close to those of non-private baselines, providing an effective solution for ICL with privacy protection.

Title: Preference Optimization with Multi-Sample Comparisons

• Authors: Chaoqi Wang, Zhuokai Zhao, Chen Zhu, Karthik Abinav Sankararaman, Michal Valko, Xuefei Cao, Zhaorun Chen, Madian Khabsa, Yuxin Chen, Hao Ma, Sinong Wang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12138
• Pdf URL: https://arxiv.org/pdf/2410.12138
• Copy Paste: [[2410.12138]] Preference Optimization with Multi-Sample Comparisons(https://arxiv.org/abs/2410.12138)
• Keywords: diffusion, generative
• Abstract: Recent advancements in generative models, particularly large language models (LLMs) and diffusion models, have been driven by extensive pretraining on large datasets followed by post-training. However, current post-training methods such as reinforcement learning from human feedback (RLHF) and direct alignment from preference methods (DAP) primarily utilize single-sample comparisons. These approaches often fail to capture critical characteristics such as generative diversity and bias, which are more accurately assessed through multiple samples. To address these limitations, we introduce a novel approach that extends post-training to include multi-sample comparisons. To achieve this, we propose Multi-sample Direct Preference Optimization (mDPO) and Multi-sample Identity Preference Optimization (mIPO). These methods improve traditional DAP methods by focusing on group-wise characteristics. Empirically, we demonstrate that multi-sample comparison is more effective in optimizing collective characteristics~(e.g., diversity and bias) for generative models than single-sample comparison. Additionally, our findings suggest that multi-sample comparisons provide a more robust optimization framework, particularly for dataset with label noise.

Title: SAM-Guided Masked Token Prediction for 3D Scene Understanding

• Authors: Zhimin Chen, Liang Yang, Yingwei Li, Longlong Jing, Bing Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12158
• Pdf URL: https://arxiv.org/pdf/2410.12158
• Copy Paste: [[2410.12158]] SAM-Guided Masked Token Prediction for 3D Scene Understanding(https://arxiv.org/abs/2410.12158)
• Keywords: self-supervised, foundation model
• Abstract: Foundation models have significantly enhanced 2D task performance, and recent works like Bridge3D have successfully applied these models to improve 3D scene understanding through knowledge distillation, marking considerable advancements. Nonetheless, challenges such as the misalignment between 2D and 3D representations and the persistent long-tail distribution in 3D datasets still restrict the effectiveness of knowledge distillation from 2D to 3D using foundation models. To tackle these issues, we introduce a novel SAM-guided tokenization method that seamlessly aligns 3D transformer structures with region-level knowledge distillation, replacing the traditional KNN-based tokenization techniques. Additionally, we implement a group-balanced re-weighting strategy to effectively address the long-tail problem in knowledge distillation. Furthermore, inspired by the recent success of masked feature prediction, our framework incorporates a two-stage masked token prediction process in which the student model predicts both the global embeddings and the token-wise local embeddings derived from the teacher models trained in the first stage. Our methodology has been validated across multiple datasets, including SUN RGB-D, ScanNet, and S3DIS, for tasks like 3D object detection and semantic segmentation. The results demonstrate significant improvements over current State-of-the-art self-supervised methods, establishing new benchmarks in this field.

Title: NSSI-Net: Multi-Concept Generative Adversarial Network for Non-Suicidal Self-Injury Detection Using High-Dimensional EEG Signals in a Semi-Supervised Learning Framework

• Authors: Zhen Liang, Weishan Ye, Qile Liu, Li Zhang, Gan Huang, Yongjie Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12159
• Pdf URL: https://arxiv.org/pdf/2410.12159
• Copy Paste: [[2410.12159]] NSSI-Net: Multi-Concept Generative Adversarial Network for Non-Suicidal Self-Injury Detection Using High-Dimensional EEG Signals in a Semi-Supervised Learning Framework(https://arxiv.org/abs/2410.12159)
• Keywords: generative
• Abstract: Non-suicidal self-injury (NSSI) is a serious threat to the physical and mental health of adolescents, significantly increasing the risk of suicide and attracting widespread public concern. Electroencephalography (EEG), as an objective tool for identifying brain disorders, holds great promise. However, extracting meaningful and reliable features from high-dimensional EEG data, especially by integrating spatiotemporal brain dynamics into informative representations, remains a major challenge. In this study, we introduce an advanced semi-supervised adversarial network, NSSI-Net, to effectively model EEG features related to NSSI. NSSI-Net consists of two key modules: a spatial-temporal feature extraction module and a multi-concept discriminator. In the spatial-temporal feature extraction module, an integrated 2D convolutional neural network (2D-CNN) and a bi-directional Gated Recurrent Unit (BiGRU) are used to capture both spatial and temporal dynamics in EEG data. In the multi-concept discriminator, signal, gender, domain, and disease levels are fully explored to extract meaningful EEG features, considering individual, demographic, disease variations across a diverse population. Based on self-collected NSSI data (n=114), the model's effectiveness and reliability are demonstrated, with a 7.44% improvement in performance compared to existing machine learning and deep learning methods. This study advances the understanding and early diagnosis of NSSI in adolescents with depression, enabling timely intervention. The source code is available at this https URL.

Title: Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Generator-Validator Fine-tuning

• Authors: Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Dongmei Zhang, Surajit Chaudhuri
• Subjects: cs.CL, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12164
• Pdf URL: https://arxiv.org/pdf/2410.12164
• Copy Paste: [[2410.12164]] Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Generator-Validator Fine-tuning(https://arxiv.org/abs/2410.12164)
• Keywords: generative
• Abstract: In this work, we propose Table-LLM-Specialist, or Table-Specialist for short, as a new self-trained fine-tuning paradigm specifically designed for table tasks. Our insight is that for each table task, there often exist two dual versions of the same task, one generative and one classification in nature. Leveraging their duality, we propose a Generator-Validator paradigm, to iteratively generate-then-validate training data from language-models, to fine-tune stronger \sys models that can specialize in a given task, without requiring manually-labeled data. Our extensive evaluations suggest that our Table-Specialist has (1) \textit{strong performance} on diverse table tasks over vanilla language-models -- for example, Table-Specialist fine-tuned on GPT-3.5 not only outperforms vanilla GPT-3.5, but can often match or surpass GPT-4 level quality, (2) \textit{lower cost} to deploy, because when Table-Specialist fine-tuned on GPT-3.5 achieve GPT-4 level quality, it becomes possible to deploy smaller models with lower latency and inference cost, with comparable quality, and (3) \textit{better generalizability} when evaluated across multiple benchmarks, since \sys is fine-tuned on a broad range of training data systematically generated from diverse real tables. Our code and data will be available at this https URL.

Title: Model Balancing Helps Low-data Training and Fine-tuning

• Authors: Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, Yaoqing Yang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12178
• Pdf URL: https://arxiv.org/pdf/2410.12178
• Copy Paste: [[2410.12178]] Model Balancing Helps Low-data Training and Fine-tuning(https://arxiv.org/abs/2410.12178)
• Keywords: foundation model
• Abstract: Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets. Studies on these foundation models underscore the importance of low-data training and fine-tuning. This topic, well-known in natural language processing (NLP), has also gained increasing attention in the emerging field of scientific machine learning (SciML). To address the limitations of low-data training and fine-tuning, we draw inspiration from Heavy-Tailed Self-Regularization (HT-SR) theory, analyzing the shape of empirical spectral densities (ESDs) and revealing an imbalance in training quality across different model layers. To mitigate this issue, we adapt a recently proposed layer-wise learning rate scheduler, TempBalance, which effectively balances training quality across layers and enhances low-data training and fine-tuning for both NLP and SciML tasks. Notably, TempBalance demonstrates increasing performance gains as the amount of available tuning data decreases. Comparative analyses further highlight the effectiveness of TempBalance and its adaptability as an "add-on" method for improving model performance.

Title: TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration

• Authors: Yiwei Guo, Shaobin Zhuang, Kunchang Li, Yu Qiao, Yali Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12183
• Pdf URL: https://arxiv.org/pdf/2410.12183
• Copy Paste: [[2410.12183]] TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration(https://arxiv.org/abs/2410.12183)
• Keywords: foundation model
• Abstract: Vision-language foundation models (such as CLIP) have recently shown their power in transfer learning, owing to large-scale image-text pre-training. However, target domain data in the downstream tasks can be highly different from the pre-training phase, which makes it hard for such a single model to generalize well. Alternatively, there exists a wide range of expert models that contain diversified vision and/or language knowledge pre-trained on different modalities, tasks, networks, and datasets. Unfortunately, these models are "isolated agents" with heterogeneous structures, and how to integrate their knowledge for generalizing CLIP-like models has not been fully explored. To bridge this gap, we propose a general and concise TransAgent framework, which transports the knowledge of the isolated agents in a unified manner, and effectively guides CLIP to generalize with multi-source knowledge distillation. With such a distinct framework, we flexibly collaborate with 11 heterogeneous agents to empower vision-language foundation models, without further cost in the inference phase. Finally, our TransAgent achieves state-of-the-art performance on 11 visual recognition datasets. Under the same low-shot setting, it outperforms the popular CoOp with around 10% on average, and 20% on EuroSAT which contains large domain shifts.

Title: Negative-Prompt-driven Alignment for Generative Language Model

• Authors: Shiqi Qiao, Ning Xv, Biao Liu, Xin Geng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12194
• Pdf URL: https://arxiv.org/pdf/2410.12194
• Copy Paste: [[2410.12194]] Negative-Prompt-driven Alignment for Generative Language Model(https://arxiv.org/abs/2410.12194)
• Keywords: generative
• Abstract: Large language models have achieved remarkable capabilities, but aligning their outputs with human values and preferences remains a significant challenge. Existing alignment methods primarily focus on positive examples while overlooking the importance of negative responses in guiding models away from undesirable behaviors. For instance, the widely-used alignment datasets reveals a scarcity of explicit negative examples that contradict human values, hindering its ability to discourage harmful or biased outputs during training. To address this limitation, we propose NEAT, i.e., NEgative-prompt-driven AlignmenT, to introduce negative prompts to generate undesirable responses alongside positive examples during the optimization process. NEAT explicitly penalizes the model for producing harmful outputs, guiding it not only toward desirable behaviors but also steering it away from generating undesirable, biased responses. This dual feedback mechanism enables better alignment with human preferences, crucial in contexts where avoiding harm is paramount. Starting from a pre-trained language model, NEAT performs online alignment by incorporating a ranking loss derived from an expanded preference dataset containing both positive and negative examples. Extensive experiments validate NEAT's effectiveness in significantly enhancing language models' alignment with human values and preferences.

Title: Abnormality Forecasting: Time Series Anomaly Prediction via Future Context Modeling

• Authors: Sinong Zhao, Wenrui Wang, Hongzuo Xu, Zhaoyang Yu, Qingsong Wen, Gang Wang, xiaoguang Liu, Guansong Pang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12206
• Pdf URL: https://arxiv.org/pdf/2410.12206
• Copy Paste: [[2410.12206]] Abnormality Forecasting: Time Series Anomaly Prediction via Future Context Modeling(https://arxiv.org/abs/2410.12206)
• Keywords: anomaly
• Abstract: Identifying anomalies from time series data plays an important role in various fields such as infrastructure security, intelligent operation and maintenance, and space exploration. Current research focuses on detecting the anomalies after they occur, which can lead to significant financial/reputation loss or infrastructure damage. In this work we instead study a more practical yet very challenging problem, time series anomaly prediction, aiming at providing early warnings for abnormal events before their occurrence. To tackle this problem, we introduce a novel principled approach, namely future context modeling (FCM). Its key insight is that the future abnormal events in a target window can be accurately predicted if their preceding observation window exhibits any subtle difference to normal data. To effectively capture such differences, FCM first leverages long-term forecasting models to generate a discriminative future context based on the observation data, aiming to amplify those subtle but unusual difference. It then models a normality correlation of the observation data with the forecasting future context to complement the normality modeling of the observation data in foreseeing possible abnormality in the target window. A joint variate-time attention learning is also introduced in FCM to leverage both temporal signals and features of the time series data for more discriminative normality modeling in the aforementioned two views. Comprehensive experiments on five datasets demonstrate that FCM gains good recall rate (70\%+) on multiple datasets and significantly outperforms all baselines in F1 score. Code is available at this https URL.

Title: Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree

• Authors: Harbani Jaggi, Kashyap Murali, Eve Fleisig, Erdem Bıyık
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12217
• Pdf URL: https://arxiv.org/pdf/2410.12217
• Copy Paste: [[2410.12217]] Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree(https://arxiv.org/abs/2410.12217)
• Keywords: in-context
• Abstract: When annotators disagree, predicting the labels given by individual annotators can capture nuances overlooked by traditional label aggregation. We introduce three approaches to predicting individual annotator ratings on the toxicity of text by incorporating individual annotator-specific information: a neural collaborative filtering (NCF) approach, an in-context learning (ICL) approach, and an intermediate embedding-based architecture. We also study the utility of demographic information for rating prediction. NCF showed limited utility; however, integrating annotator history, demographics, and survey information permits both the embedding-based architecture and ICL to substantially improve prediction accuracy, with the embedding-based architecture outperforming the other methods. We also find that, if demographics are predicted from survey information, using these imputed demographics as features performs comparably to using true demographic data. This suggests that demographics may not provide substantial information for modeling ratings beyond what is captured in survey responses. Our findings raise considerations about the relative utility of different types of annotator information and provide new approaches for modeling annotators in subjective NLP tasks.

Title: Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety

• Authors: Lucas Choi, Ross Greer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12225
• Pdf URL: https://arxiv.org/pdf/2410.12225
• Copy Paste: [[2410.12225]] Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety(https://arxiv.org/abs/2410.12225)
• Keywords: foundation model
• Abstract: This paper evaluates the use of vision-language models (VLMs) for zero-shot detection and association of hardhats to enhance construction safety. Given the significant risk of head injuries in construction, proper enforcement of hardhat use is critical. We investigate the applicability of foundation models, specifically OWLv2, for detecting hardhats in real-world construction site images. Our contributions include the creation of a new benchmark dataset, Hardhat Safety Detection Dataset, by filtering and combining existing datasets and the development of a cascaded detection approach. Experimental results on 5,210 images demonstrate that the OWLv2 model achieves an average precision of 0.6493 for hardhat detection. We further analyze the limitations and potential improvements for real-world applications, highlighting the strengths and weaknesses of current foundation models in safety perception domains.

Title: Off-dynamics Conditional Diffusion Planners

• Authors: Wen Zheng Terence Ng, Jianda Chen, Tianwei Zhang
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.12238
• Pdf URL: https://arxiv.org/pdf/2410.12238
• Copy Paste: [[2410.12238]] Off-dynamics Conditional Diffusion Planners(https://arxiv.org/abs/2410.12238)
• Keywords: diffusion
• Abstract: Offline Reinforcement Learning (RL) offers an attractive alternative to interactive data acquisition by leveraging pre-existing datasets. However, its effectiveness hinges on the quantity and quality of the data samples. This work explores the use of more readily available, albeit off-dynamics datasets, to address the challenge of data scarcity in Offline RL. We propose a novel approach using conditional Diffusion Probabilistic Models (DPMs) to learn the joint distribution of the large-scale off-dynamics dataset and the limited target dataset. To enable the model to capture the underlying dynamics structure, we introduce two contexts for the conditional model: (1) a continuous dynamics score allows for partial overlap between trajectories from both datasets, providing the model with richer information; (2) an inverse-dynamics context guides the model to generate trajectories that adhere to the target environment's dynamic constraints. Empirical results demonstrate that our method significantly outperforms several strong baselines. Ablation studies further reveal the critical role of each dynamics context. Additionally, our model demonstrates that by modifying the context, we can interpolate between source and target dynamics, making it more robust to subtle shifts in the environment.

Title: CATCH: Channel-Aware multivariate Time Series Anomaly Detection via Frequency Patching

• Authors: Xingjian Wu, Xiangfei Qiu, Zhengyu Li, Yihang Wang, Jilin Hu, Chenjuan Guo, Hui Xiong, Bin Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12261
• Pdf URL: https://arxiv.org/pdf/2410.12261
• Copy Paste: [[2410.12261]] CATCH: Channel-Aware multivariate Time Series Anomaly Detection via Frequency Patching(https://arxiv.org/abs/2410.12261)
• Keywords: anomaly
• Abstract: Anomaly detection in multivariate time series is challenging as heterogeneous subsequence anomalies may occur. Reconstruction-based methods, which focus on learning nomral patterns in the frequency domain to detect diverse abnormal subsequences, achieve promising resutls, while still falling short on capturing fine-grained frequency characteristics and channel correlations. To contend with the limitations, we introduce CATCH, a framework based on frequency patching. We propose to patchify the frequency domain into frequency bands, which enhances its ability to capture fine-grained frequency characteristics. To perceive appropriate channel correlations, we propose a Channel Fusion Module (CFM), which features a patch-wise mask generator and a masked-attention mechanism. Driven by a bi-level multi-objective optimization algorithm, the CFM is encouraged to iteratively discover appropriate patch-wise channel correlations, and to cluster relevant channels while isolating adverse effects from irrelevant channels. Extensive experiments on 9 real-world datasets and 12 synthetic datasets demonstrate that CATCH achieves state-of-the-art performance.

Title: DaDiff: Domain-aware Diffusion Model for Nighttime UAV Tracking

• Authors: Haobo Zuo, Changhong Fu, Guangze Zheng, Liangliang Yao, Kunhan Lu, Jia Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12270
• Pdf URL: https://arxiv.org/pdf/2410.12270
• Copy Paste: [[2410.12270]] DaDiff: Domain-aware Diffusion Model for Nighttime UAV Tracking(https://arxiv.org/abs/2410.12270)
• Keywords: diffusion
• Abstract: Domain adaptation is an inspiring solution to the misalignment issue of day/night image features for nighttime UAV tracking. However, the one-step adaptation paradigm is inadequate in addressing the prevalent difficulties posed by low-resolution (LR) objects when viewed from the UAVs at night, owing to the blurry edge contour and limited detail information. Moreover, these approaches struggle to perceive LR objects disturbed by nighttime noise. To address these challenges, this work proposes a novel progressive alignment paradigm, named domain-aware diffusion model (DaDiff), aligning nighttime LR object features to the daytime by virtue of progressive and stable generations. The proposed DaDiff includes an alignment encoder to enhance the detail information of nighttime LR objects, a tracking-oriented layer designed to achieve close collaboration with tracking tasks, and a successive distribution discriminator presented to distinguish different feature distributions at each diffusion timestep successively. Furthermore, an elaborate nighttime UAV tracking benchmark is constructed for LR objects, namely NUT-LR, consisting of 100 annotated sequences. Exhaustive experiments have demonstrated the robustness and feature alignment ability of the proposed DaDiff. The source code and video demo are available at this https URL.

Title: Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond

• Authors: Pengwei Liang, Junjun Jiang, Qing Ma, Xianming Liu, Jiayi Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12274
• Pdf URL: https://arxiv.org/pdf/2410.12274
• Copy Paste: [[2410.12274]] Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond(https://arxiv.org/abs/2410.12274)
• Keywords: self-supervised
• Abstract: Image fusion is famous as an alternative solution to generate one high-quality image from multiple images in addition to image restoration from a single degraded image. The essence of image fusion is to integrate complementary information from source images. Existing fusion methods struggle with generalization across various tasks and often require labor-intensive designs, in which it is difficult to identify and extract useful information from source images due to the diverse requirements of each fusion task. Additionally, these methods develop highly specialized features for different downstream applications, hindering the adaptation to new and diverse downstream tasks. To address these limitations, we introduce DeFusion++, a novel framework that leverages self-supervised learning (SSL) to enhance the versatility of feature representation for different image fusion tasks. DeFusion++ captures the image fusion task-friendly representations from large-scale data in a self-supervised way, overcoming the constraints of limited fusion datasets. Specifically, we introduce two innovative pretext tasks: common and unique decomposition (CUD) and masked feature modeling (MFM). CUD decomposes source images into abstract common and unique components, while MFM refines these components into robust fused features. Jointly training of these tasks enables DeFusion++ to produce adaptable representations that can effectively extract useful information from various source images, regardless of the fusion task. The resulting fused representations are also highly adaptable for a wide range of downstream tasks, including image segmentation and object detection. DeFusion++ stands out by producing versatile fused representations that can enhance both the quality of image fusion and the effectiveness of downstream high-level vision tasks, simplifying the process with the elegant fusion framework.

Title: Controlled Automatic Task-Specific Synthetic Data Generation for Hallucination Detection

• Authors: Yong Xie, Karan Aggarwal, Aitzaz Ahmad, Stephen Lau
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12278
• Pdf URL: https://arxiv.org/pdf/2410.12278
• Copy Paste: [[2410.12278]] Controlled Automatic Task-Specific Synthetic Data Generation for Hallucination Detection(https://arxiv.org/abs/2410.12278)
• Keywords: in-context
• Abstract: We present a novel approach to automatically generate non-trivial task-specific synthetic datasets for hallucination detection. Our approach features a two-step generation-selection pipeline, using hallucination pattern guidance and a language style alignment during generation. Hallucination pattern guidance leverages the most important task-specific hallucination patterns while language style alignment aligns the style of the synthetic dataset with benchmark text. To obtain robust supervised detectors from synthetic datasets, we also adopt a data mixture strategy to improve performance robustness and generalization. Our results on three datasets show that our generated hallucination text is more closely aligned with non-hallucinated text versus baselines, to train hallucination detectors with better generalization. Our hallucination detectors trained on synthetic datasets outperform in-context-learning (ICL)-based detectors by a large margin of 32%. Our extensive experiments confirm the benefits of our approach with cross-task and cross-generator generalization. Our data-mixture-based training further improves the generalization and robustness of hallucination detection.

Title: DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domain

• Authors: Fengpeng Li, Kemou Li, Haiwei Wu, Jinyu Tian, Jiantao Zhou
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12307
• Pdf URL: https://arxiv.org/pdf/2410.12307
• Copy Paste: [[2410.12307]] DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domain(https://arxiv.org/abs/2410.12307)
• Keywords: generative
• Abstract: To protect deep neural networks (DNNs) from adversarial attacks, adversarial training (AT) is developed by incorporating adversarial examples (AEs) into model training. Recent studies show that adversarial attacks disproportionately impact the patterns within the phase of the sample's frequency spectrum -- typically containing crucial semantic information -- more than those in the amplitude, resulting in the model's erroneous categorization of AEs. We find that, by mixing the amplitude of training samples' frequency spectrum with those of distractor images for AT, the model can be guided to focus on phase patterns unaffected by adversarial perturbations. As a result, the model's robustness can be improved. Unfortunately, it is still challenging to select appropriate distractor images, which should mix the amplitude without affecting the phase patterns. To this end, in this paper, we propose an optimized Adversarial Amplitude Generator (AAG) to achieve a better tradeoff between improving the model's robustness and retaining phase patterns. Based on this generator, together with an efficient AE production procedure, we design a new Dual Adversarial Training (DAT) strategy. Experiments on various datasets show that our proposed DAT leads to significantly improved robustness against diverse adversarial attacks.

Title: FaceChain-FACT: Face Adapter with Decoupled Training for Identity-preserved Personalization

• Authors: Cheng Yu, Haoyu Xie, Lei Shang, Yang Liu, Jun Dan, Baigui Sun, Liefeng Bo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12312
• Pdf URL: https://arxiv.org/pdf/2410.12312
• Copy Paste: [[2410.12312]] FaceChain-FACT: Face Adapter with Decoupled Training for Identity-preserved Personalization(https://arxiv.org/abs/2410.12312)
• Keywords: generative
• Abstract: In the field of human-centric personalized image generation, the adapter-based method obtains the ability to customize and generate portraits by text-to-image training on facial data. This allows for identity-preserved personalization without additional fine-tuning in inference. Although there are improvements in efficiency and fidelity, there is often a significant performance decrease in test following ability, controllability, and diversity of generated faces compared to the base model. In this paper, we analyze that the performance degradation is attributed to the failure to decouple identity features from other attributes during extraction, as well as the failure to decouple the portrait generation training from the overall generation task. To address these issues, we propose the Face Adapter with deCoupled Training (FACT) framework, focusing on both model architecture and training strategy. To decouple identity features from others, we leverage a transformer-based face-export encoder and harness fine-grained identity features. To decouple the portrait generation training, we propose Face Adapting Increment Regularization~(FAIR), which effectively constrains the effect of face adapters on the facial region, preserving the generative ability of the base model. Additionally, we incorporate a face condition drop and shuffle mechanism, combined with curriculum learning, to enhance facial controllability and diversity. As a result, FACT solely learns identity preservation from training data, thereby minimizing the impact on the original text-to-image capabilities of the base model. Extensive experiments show that FACT has both controllability and fidelity in both text-to-image generation and inpainting solutions for portrait generation.

Title: Revisited Large Language Model for Time Series Analysis through Modality Alignment

• Authors: Liangwei Nathan Zheng, Chang George Dong, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, Weitong Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12326
• Pdf URL: https://arxiv.org/pdf/2410.12326
• Copy Paste: [[2410.12326]] Revisited Large Language Model for Time Series Analysis through Modality Alignment(https://arxiv.org/abs/2410.12326)
• Keywords: anomaly
• Abstract: Large Language Models have demonstrated impressive performance in many pivotal web applications such as sensor data analysis. However, since LLMs are not designed for time series tasks, simpler models like linear regressions can often achieve comparable performance with far less complexity. In this study, we perform extensive experiments to assess the effectiveness of applying LLMs to key time series tasks, including forecasting, classification, imputation, and anomaly detection. We compare the performance of LLMs against simpler baseline models, such as single-layer linear models and randomly initialized LLMs. Our results reveal that LLMs offer minimal advantages for these core time series tasks and may even distort the temporal structure of the data. In contrast, simpler models consistently outperform LLMs while requiring far fewer parameters. Furthermore, we analyze existing reprogramming techniques and show, through data manifold analysis, that these methods fail to effectively align time series data with language and display pseudo-alignment behaviour in embedding space. Our findings suggest that the performance of LLM-based methods in time series tasks arises from the intrinsic characteristics and structure of time series data, rather than any meaningful alignment with the language model architecture.

Title: Neuron-based Personality Trait Induction in Large Language Models

• Authors: Jia Deng, Tianyi Tang, Yanbin Yin, Wenhao Yang, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12327
• Pdf URL: https://arxiv.org/pdf/2410.12327
• Copy Paste: [[2410.12327]] Neuron-based Personality Trait Induction in Large Language Models(https://arxiv.org/abs/2410.12327)
• Keywords: generative
• Abstract: Large language models (LLMs) have become increasingly proficient at simulating various personality traits, an important capability for supporting related applications (e.g., role-playing). To further improve this capacity, in this paper, we present a neuron-based approach for personality trait induction in LLMs, with three major technical contributions. First, we construct PersonalityBench, a large-scale dataset for identifying and evaluating personality traits in LLMs. This dataset is grounded in the Big Five personality traits from psychology and is designed to assess the generative capabilities of LLMs towards specific personality traits. Second, by leveraging PersonalityBench, we propose an efficient method for identifying personality-related neurons within LLMs by examining the opposite aspects of a given trait. Third, we develop a simple yet effective induction method that manipulates the values of these identified personality-related neurons. This method enables fine-grained control over the traits exhibited by LLMs without training and modifying model parameters. Extensive experiments validate the efficacy of our neuron identification and trait induction methods. Notably, our approach achieves comparable performance as fine-tuned models, offering a more efficient and flexible solution for personality trait induction in LLMs. We provide access to all the mentioned resources at this https URL.

Title: Improved Anomaly Detection through Conditional Latent Space VAE Ensembles

• Authors: Oskar Åström, Alexandros Sopasakis
• Subjects: cs.LG, cs.CV, math.PR
• Abstract URL: https://arxiv.org/abs/2410.12328
• Pdf URL: https://arxiv.org/pdf/2410.12328
• Copy Paste: [[2410.12328]] Improved Anomaly Detection through Conditional Latent Space VAE Ensembles(https://arxiv.org/abs/2410.12328)
• Keywords: anomaly
• Abstract: We propose a novel Conditional Latent space Variational Autoencoder (CL-VAE) to perform improved pre-processing for anomaly detection on data with known inlier classes and unknown outlier classes. This proposed variational autoencoder (VAE) improves latent space separation by conditioning on information within the data. The method fits a unique prior distribution to each class in the dataset, effectively expanding the classic prior distribution for VAEs to include a Gaussian mixture model. An ensemble of these VAEs are merged in the latent spaces to form a group consensus that greatly improves the accuracy of anomaly detection across data sets. Our approach is compared against the capabilities of a typical VAE, a CNN, and a PCA, with regards AUC for anomaly detection. The proposed model shows increased accuracy in anomaly detection, achieving an AUC of 97.4% on the MNIST dataset compared to 95.7% for the second best model. In addition, the CL-VAE shows increased benefits from ensembling, a more interpretable latent space, and an increased ability to learn patterns in complex data with limited model sizes.

Title: MAX: Masked Autoencoder for X-ray Fluorescence in Geological Investigation

• Authors: An-Sheng Lee, Yu-Wen Pao, Hsuan-Tien Lin, Sofia Ya Hsuan Liou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12330
• Pdf URL: https://arxiv.org/pdf/2410.12330
• Copy Paste: [[2410.12330]] MAX: Masked Autoencoder for X-ray Fluorescence in Geological Investigation(https://arxiv.org/abs/2410.12330)
• Keywords: self-supervised, foundation model
• Abstract: Pre-training foundation models has become the de-facto procedure for deep learning approaches, yet its application remains limited in the geological studies, where in needs of the model transferability to break the shackle of data scarcity. Here we target on the X-ray fluorescence (XRF) scanning data, a standard high-resolution measurement in extensive scientific drilling projects. We propose a scalable self-supervised learner, masked autoencoders on XRF spectra (MAX), to pre-train a foundation model covering geological records from multiple regions of the Pacific and Southern Ocean. In pre-training, we find that masking a high proportion of the input spectrum (50\%) yields a nontrivial and meaningful self-supervisory task. For downstream tasks, we select the quantification of XRF spectra into two costly geochemical measurements, CaCO$_3$ and total organic carbon, due to their importance in understanding the paleo-oceanic carbon system. Our results show that MAX, requiring only one-third of the data, outperforms models without pre-training in terms of quantification accuracy. Additionally, the model's generalizability improves by more than 60\% in zero-shot tests on new materials, with explainability further ensuring its robustness. Thus, our approach offers a promising pathway to overcome data scarcity in geological discovery by leveraging the self-supervised foundation model and fast-acquired XRF scanning data.

Title: MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs

• Authors: Yunqiu Xu, Linchao Zhu, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12332
• Pdf URL: https://arxiv.org/pdf/2410.12332
• Copy Paste: [[2410.12332]] MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs(https://arxiv.org/abs/2410.12332)
• Keywords: foundation model
• Abstract: While multimodal large language models (MLLMs) have demonstrated extraordinary vision-language understanding capabilities and shown potential to serve as general-purpose assistants, their abilities to solve instance-level visual-language problems beyond a single image warrant further exploration. In order to assess these unproven abilities of MLLMs, this paper proposes a new visual grounding task called multi-context visual grounding, which aims to localize instances of interest across multiple images based on open-ended text prompts. To facilitate this research, we meticulously construct a new dataset MC-Bench for benchmarking the visual grounding capabilities of MLLMs. MC-Bench features 2K high-quality and manually annotated samples, consisting of instance-level labeled image pairs and corresponding text prompts that indicate the target instances in the images. In total, there are three distinct styles of text prompts, covering 20 practical skills. We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities. Our evaluation reveals a non-trivial performance gap between existing MLLMs and humans across all metrics. We also observe that existing MLLMs typically outperform foundation models without LLMs only on image-level metrics, and the specialist MLLMs trained on single images often struggle to generalize to multi-image scenarios. Moreover, a simple stepwise baseline integrating advanced MLLM and a detector can significantly surpass prior end-to-end MLLMs. We hope our MC-Bench and empirical findings can encourage the research community to further explore and enhance the untapped potentials of MLLMs in instance-level tasks, particularly in multi-image contexts. Project page: this https URL.

Title: A linguistic analysis of undesirable outcomes in the era of generative AI

• Authors: Daniele Gambetta, Gizem Gezici, Fosca Giannotti, Dino Pedreschi, Alistair Knott, Luca Pappalardo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12341
• Pdf URL: https://arxiv.org/pdf/2410.12341
• Copy Paste: [[2410.12341]] A linguistic analysis of undesirable outcomes in the era of generative AI(https://arxiv.org/abs/2410.12341)
• Keywords: generative
• Abstract: Recent research has focused on the medium and long-term impacts of generative AI, posing scientific and societal challenges mainly due to the detection and reliability of machine-generated information, which is projected to form the major content on the Web soon. Prior studies show that LLMs exhibit a lower performance in generation tasks (model collapse) as they undergo a fine-tuning process across multiple generations on their own generated content (self-consuming loop). In this paper, we present a comprehensive simulation framework built upon the chat version of LLama2, focusing particularly on the linguistic aspects of the generated content, which has not been fully examined in existing studies. Our results show that the model produces less lexical rich content across generations, reducing diversity. The lexical richness has been measured using the linguistic measures of entropy and TTR as well as calculating the POSTags frequency. The generated content has also been examined with an $n$-gram analysis, which takes into account the word order, and semantic networks, which consider the relation between different words. These findings suggest that the model collapse occurs not only by decreasing the content diversity but also by distorting the underlying linguistic patterns of the generated text, which both highlight the critical importance of carefully choosing and curating the initial input text, which can alleviate the model collapse problem. Furthermore, we conduct a qualitative analysis of the fine-tuned models of the pipeline to compare their performances on generic NLP tasks to the original model. We find that autophagy transforms the initial model into a more creative, doubtful and confused one, which might provide inaccurate answers and include conspiracy theories in the model responses, spreading false and biased information on the Web.

Title: Towards Flexible and Efficient Diffusion Low Light Enhancer

• Authors: Guanzhou Lan, Qianli Ma, Yuqi Yang, Zhigang Wang, Dong Wang, Yuan Yuan, Bin Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12346
• Pdf URL: https://arxiv.org/pdf/2410.12346
• Copy Paste: [[2410.12346]] Towards Flexible and Efficient Diffusion Low Light Enhancer(https://arxiv.org/abs/2410.12346)
• Keywords: diffusion
• Abstract: Diffusion-based Low-Light Image Enhancement (LLIE) has demonstrated significant success in improving the visibility of low-light images. However, the substantial computational burden introduced by the iterative sampling process remains a major concern. Current acceleration methods, whether training-based or training-free, often lead to significant performance degradation. As a result, to achieve an efficient student model with performance comparable to that of existing multi-step teacher model, it is usually necessary to retrain a more capable teacher model. This approach introduces inflexibility, as it requires additional training to enhance the teacher's performance. To address these challenges, we propose \textbf{Re}flectance-aware \textbf{D}iffusion with \textbf{Di}stilled \textbf{T}rajectory (\textbf{ReDDiT}), a step distillation framework specifically designed for LLIE. ReDDiT trains a student model to replicate the teacher's trajectory in fewer steps while also possessing the ability to surpass the teacher's performance. Specifically, we first introduce a trajectory decoder from the teacher model to provide guidance. Subsequently, a reflectance-aware trajectory refinement module is incorporated into the distillation process to enable more deterministic guidance from the teacher model. Our framework achieves comparable performance to previous diffusion-based methods with redundant steps in just 2 steps while establishing new state-of-the-art (SOTA) results with 8 or 4 steps. Comprehensive experimental evaluations on 10 benchmark datasets validate the effectiveness of our method, consistently outperforming existing SOTA methods.

Title: Towards Neural Scaling Laws for Time Series Foundation Models

• Authors: Qingren Yao, Chao-Han Huck Yang, Renhe Jiang, Yuxuan Liang, Ming Jin, Shirui Pan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12360
• Pdf URL: https://arxiv.org/pdf/2410.12360
• Copy Paste: [[2410.12360]] Towards Neural Scaling Laws for Time Series Foundation Models(https://arxiv.org/abs/2410.12360)
• Keywords: foundation model
• Abstract: Scaling laws offer valuable insights into the design of time series foundation models (TSFMs). However, previous research has largely focused on the scaling laws of TSFMs for in-distribution (ID) data, leaving their out-of-distribution (OOD) scaling behavior and the influence of model architectures less explored. In this work, we examine two common TSFM architectures, encoder-only and decoder-only Transformers, and investigate their scaling behavior on both ID and OOD data. These models are trained and evaluated across varying parameter counts, compute budgets, and dataset sizes. Our experiments reveal that the log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID settings. We further compare the scaling properties across different architectures, incorporating two state-of-the-art TSFMs as case studies, showing that model architecture plays a significant role in scaling. The encoder-only Transformers demonstrate better scalability than the decoder-only Transformers, while the architectural enhancements in the two advanced TSFMs primarily improve ID performance but reduce OOD scalability. While scaling up TSFMs is expected to drive performance breakthroughs, the lack of a comprehensive understanding of TSFM scaling laws has hindered the development of a robust framework to guide model scaling. We fill this gap in this work by synthesizing our findings and providing practical guidelines for designing and scaling larger TSFMs with enhanced model capabilities.

Title: GAN Based Top-Down View Synthesis in Reinforcement Learning Environments

• Authors: Usama Younus, Vinoj Jayasundara, Shivam Mishra, Suleyman Aslan
• Subjects: cs.CV, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.12372
• Pdf URL: https://arxiv.org/pdf/2410.12372
• Copy Paste: [[2410.12372]] GAN Based Top-Down View Synthesis in Reinforcement Learning Environments(https://arxiv.org/abs/2410.12372)
• Keywords: generative
• Abstract: Human actions are based on the mental perception of the environment. Even when all the aspects of an environment are not visible, humans have an internal mental model that can generalize the partially visible scenes to fully constructed and connected views. This internal mental model uses learned abstract representations of spatial and temporal aspects of the environments encountered in the past. Artificial agents in reinforcement learning environments also benefit by learning a representation of the environment from experience. It provides the agent with viewpoints that are not directly visible to it, helping it make better policy decisions. It can also be used to predict the future state of the environment. This project explores learning the top-down view of an RL environment based on the artificial agent's first-person view observations with a generative adversarial network(GAN). The top-down view is useful as it provides a complete overview of the environment by building a map of the entire environment. It provides information about the objects' dimensions and shapes along with their relative positions with one another. Initially, when only a partial observation of the environment is visible to the agent, only a partial top-down view is generated. As the agent explores the environment through a set of actions, the generated top-down view becomes complete. This generated top-down view can assist the agent in deducing better policy decisions. The focus of the project is to learn the top-down view of an RL environment. It doesn't deal with any Reinforcement Learning task.

Title: HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims

• Authors: Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, Kunwoo Park
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.12377
• Pdf URL: https://arxiv.org/pdf/2410.12377
• Copy Paste: [[2410.12377]] HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims(https://arxiv.org/abs/2410.12377)
• Keywords: in-context
• Abstract: To tackle the AVeriTeC shared task hosted by the FEVER-24, we introduce a system that only employs publicly available large language models (LLMs) for each step of automated fact-checking, dubbed the Herd of Open LLMs for verifying real-world claims (HerO). HerO employs multiple LLMs for each step of automated fact-checking. For evidence retrieval, a language model is used to enhance a query by generating hypothetical fact-checking documents. We prompt pretrained and fine-tuned LLMs for question generation and veracity prediction by crafting prompts with retrieved in-context samples. HerO achieved 2nd place on the leaderboard with the AVeriTeC score of 0.57, suggesting the potential of open LLMs for verifying real-world claims. For future research, we make our code publicly available at this https URL.

Title: Feature Augmentation for Self-supervised Contrastive Learning: A Closer Look

• Authors: Yong Zhang, Rui Zhu, Shifeng Zhang, Xu Zhou, Shifeng Chen, Xiaofan Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12396
• Pdf URL: https://arxiv.org/pdf/2410.12396
• Copy Paste: [[2410.12396]] Feature Augmentation for Self-supervised Contrastive Learning: A Closer Look(https://arxiv.org/abs/2410.12396)
• Keywords: self-supervised
• Abstract: Self-supervised contrastive learning heavily relies on the view variance brought by data augmentation, so that it can learn a view-invariant pre-trained representation. Beyond increasing the view variance for contrast, this work focuses on improving the diversity of training data, to improve the generalization and robustness of the pre-trained models. To this end, we propose a unified framework to conduct data augmentation in the feature space, known as feature augmentation. This strategy is domain-agnostic, which augments similar features to the original ones and thus improves the data diversity. We perform a systematic investigation of various feature augmentation architectures, the gradient-flow skill, and the relationship between feature augmentation and traditional data augmentation. Our study reveals some practical principles for feature augmentation in self-contrastive learning. By integrating feature augmentation on the instance discrimination or the instance similarity paradigm, we consistently improve the performance of pre-trained feature learning and gain better generalization over the downstream image classification and object detection task.

Title: Training Neural Samplers with Reverse Diffusive KL Divergence

• Authors: Jiajun He, Wenlin Chen, Mingtian Zhang, David Barber, José Miguel Hernández-Lobato
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12456
• Pdf URL: https://arxiv.org/pdf/2410.12456
• Copy Paste: [[2410.12456]] Training Neural Samplers with Reverse Diffusive KL Divergence(https://arxiv.org/abs/2410.12456)
• Keywords: diffusion, generative
• Abstract: Training generative models to sample from unnormalized density functions is an important and challenging task in machine learning. Traditional training methods often rely on the reverse Kullback-Leibler (KL) divergence due to its tractability. However, the mode-seeking behavior of reverse KL hinders effective approximation of multi-modal target distributions. To address this, we propose to minimize the reverse KL along diffusion trajectories of both model and target densities. We refer to this objective as the reverse diffusive KL divergence, which allows the model to capture multiple modes. Leveraging this objective, we train neural samplers that can efficiently generate samples from the target distribution in one step. We demonstrate that our method enhances sampling performance across various Boltzmann distributions, including both synthetic multi-modal densities and n-body particle systems.

Title: HELM: Hierarchical Encoding for mRNA Language Modeling

• Authors: Mehdi Yazdani-Jahromi, Mangal Prakash, Tommaso Mansi, Artem Moskalev, Rui Liao
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2410.12459
• Pdf URL: https://arxiv.org/pdf/2410.12459
• Copy Paste: [[2410.12459]] HELM: Hierarchical Encoding for mRNA Language Modeling(https://arxiv.org/abs/2410.12459)
• Keywords: foundation model, generative
• Abstract: Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on six diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8\%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines.

Title: SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling

• Authors: Loris Gaven, Clement Romac, Thomas Carta, Sylvain Lamprier, Olivier Sigaud, Pierre-Yves Oudeyer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12481
• Pdf URL: https://arxiv.org/pdf/2410.12481
• Copy Paste: [[2410.12481]] SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling(https://arxiv.org/abs/2410.12481)
• Keywords: generative
• Abstract: The past years have seen Large Language Models (LLMs) strive not only as generative models but also as agents solving textual sequential decision-making tasks. When facing complex environments where their zero-shot abilities are insufficient, recent work showed online Reinforcement Learning (RL) could be used for the LLM agent to discover and learn efficient strategies interactively. However, most prior work sticks to on-policy algorithms, which greatly reduces the scope of methods such agents could use for both exploration and exploitation, such as experience replay and hindsight relabeling. Yet, such methods may be key for LLM learning agents, and in particular when designing autonomous intrinsically motivated agents sampling and pursuing their own goals (i.e. autotelic agents). This paper presents and studies an adaptation of Soft Actor-Critic and hindsight relabeling to LLM agents. Our method not only paves the path towards autotelic LLM agents that learn online but can also outperform on-policy methods in more classic multi-goal RL environments.

Title: Synthetic Augmentation for Anatomical Landmark Localization using DDPMs

• Authors: Arnela Hadzic, Lea Bogensperger, Simon Johannes Joham, Martin Urschler
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12489
• Pdf URL: https://arxiv.org/pdf/2410.12489
• Copy Paste: [[2410.12489]] Synthetic Augmentation for Anatomical Landmark Localization using DDPMs(https://arxiv.org/abs/2410.12489)
• Keywords: diffusion, generative
• Abstract: Deep learning techniques for anatomical landmark localization (ALL) have shown great success, but their reliance on large annotated datasets remains a problem due to the tedious and costly nature of medical data acquisition and annotation. While traditional data augmentation, variational autoencoders (VAEs), and generative adversarial networks (GANs) have already been used to synthetically expand medical datasets, diffusion-based generative models have recently started to gain attention for their ability to generate high-quality synthetic images. In this study, we explore the use of denoising diffusion probabilistic models (DDPMs) for generating medical images and their corresponding heatmaps of landmarks to enhance the training of a supervised deep learning model for ALL. Our novel approach involves a DDPM with a 2-channel input, incorporating both the original medical image and its heatmap of annotated landmarks. We also propose a novel way to assess the quality of the generated images using a Markov Random Field (MRF) model for landmark matching and a Statistical Shape Model (SSM) to check landmark plausibility, before we evaluate the DDPM-augmented dataset in the context of an ALL task involving hand X-Rays.

Title: Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective

• Authors: Yongxin Zhu, Bocheng Li, Hang Zhang, Xin Li, Linli Xu, Lidong Bing
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12490
• Pdf URL: https://arxiv.org/pdf/2410.12490
• Copy Paste: [[2410.12490]] Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective(https://arxiv.org/abs/2410.12490)
• Keywords: diffusion, generative
• Abstract: Latent-based image generative models, such as Latent Diffusion Models (LDMs) and Mask Image Models (MIMs), have achieved notable success in image generation tasks. These models typically leverage reconstructive autoencoders like VQGAN or VAE to encode pixels into a more compact latent space and learn the data distribution in the latent space instead of directly from pixels. However, this practice raises a pertinent question: Is it truly the optimal choice? In response, we begin with an intriguing observation: despite sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in image generation. This finding contrasts sharply with the field of NLP, where the autoregressive model GPT has established a commanding presence. To address this discrepancy, we introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling. Furthermore, we propose a simple but effective discrete image tokenizer to stabilize the latent space for image generative modeling. Experimental results show that image autoregressive modeling with our tokenizer (DiGIT) benefits both image understanding and image generation with the next token prediction principle, which is inherently straightforward for GPT models but challenging for other generative models. Remarkably, for the first time, a GPT-style autoregressive model for images outperforms LDMs, which also exhibits substantial improvement akin to GPT when scaling up model size. Our findings underscore the potential of an optimized latent space and the integration of discrete tokenization in advancing the capabilities of image generative models. The code is available at \url{this https URL}.

Title: DH-VTON: Deep Text-Driven Virtual Try-On via Hybrid Attention Learning

• Authors: Jiabao Wei, Zhiyuan Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12501
• Pdf URL: https://arxiv.org/pdf/2410.12501
• Copy Paste: [[2410.12501]] DH-VTON: Deep Text-Driven Virtual Try-On via Hybrid Attention Learning(https://arxiv.org/abs/2410.12501)
• Keywords: diffusion
• Abstract: Virtual Try-ON (VTON) aims to synthesis specific person images dressed in given garments, which recently receives numerous attention in online shopping scenarios. Currently, the core challenges of the VTON task mainly lie in the fine-grained semantic extraction (i.e.,deep semantics) of the given reference garments during depth estimation and effective texture preservation when the garments are synthesized and warped onto human body. To cope with these issues, we propose DH-VTON, a deep text-driven virtual try-on model featuring a special hybrid attention learning strategy and deep garment semantic preservation module. By standing on the shoulder of a well-built pre-trained paint-by-example (abbr. PBE) approach, we present our DH-VTON pipeline in this work. Specifically, to extract the deep semantics of the garments, we first introduce InternViT-6B as fine-grained feature learner, which can be trained to align with the large-scale intrinsic knowledge with deep text semantics (e.g.,"neckline" or "girdle") to make up for the deficiency of the commonly adopted CLIP encoder. Based on this, to enhance the customized dressing abilities, we further introduce Garment-Feature ControlNet Plus (abbr. GFC+) module and propose to leverage a fresh hybrid attention strategy for training, which can adaptively integrate fine-grained characteristics of the garments into the different layers of the VTON model, so as to achieve multi-scale features preservation effects. Extensive experiments on several representative datasets demonstrate that our method outperforms previous diffusion-based and GAN-based approaches, showing competitive performance in preserving garment details and generating authentic human images.

Title: MING: A Functional Approach to Learning Molecular Generative Models

• Authors: Van Khoa Nguyen, Maciej Falkiewicz, Giangiacomo Mercatali, Alexandros Kalousis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12522
• Pdf URL: https://arxiv.org/pdf/2410.12522
• Copy Paste: [[2410.12522]] MING: A Functional Approach to Learning Molecular Generative Models(https://arxiv.org/abs/2410.12522)
• Keywords: diffusion, generative
• Abstract: Traditional molecule generation methods often rely on sequence or graph-based representations, which can limit their expressive power or require complex permutation-equivariant architectures. This paper introduces a novel paradigm for learning molecule generative models based on functional representations. Specifically, we propose Molecular Implicit Neural Generation (MING), a diffusion-based model that learns molecular distributions in function space. Unlike standard diffusion processes in data space, MING employs a novel functional denoising probabilistic process, which jointly denoises the information in both the function's input and output spaces by leveraging an expectation-maximization procedure for latent implicit neural representations of data. This approach allows for a simple yet effective model design that accurately captures underlying function distributions. Experimental results on molecule-related datasets demonstrate MING's superior performance and ability to generate plausible molecular samples, surpassing state-of-the-art data-space methods while offering a more streamlined architecture and significantly faster generation times.

Title: Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing

• Authors: Mingce Guo, Jingxuan He, Shengeng Tang, Zhangye Wang, Lechao Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12526
• Pdf URL: https://arxiv.org/pdf/2410.12526
• Copy Paste: [[2410.12526]] Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing(https://arxiv.org/abs/2410.12526)
• Keywords: diffusion, generative
• Abstract: Text-driven video editing utilizing generative diffusion models has garnered significant attention due to their potential applications. However, existing approaches are constrained by the limited word embeddings provided in pre-training, which hinders nuanced editing targeting open concepts with specific attributes. Directly altering the keywords in target prompts often results in unintended disruptions to the attention mechanisms. To achieve more flexible editing easily, this work proposes an improved concept-augmented video editing approach that generates diverse and stable target videos flexibly by devising abstract conceptual pairs. Specifically, the framework involves concept-augmented textual inversion and a dual prior supervision mechanism. The former enables plug-and-play guidance of stable diffusion for video editing, effectively capturing target attributes for more stylized results. The dual prior supervision mechanism significantly enhances video stability and fidelity. Comprehensive evaluations demonstrate that our approach generates more stable and lifelike videos, outperforming state-of-the-art methods.

Title: One Step Diffusion via Shortcut Models

• Authors: Kevin Frans, Danijar Hafner, Sergey Levine, Pieter Abbeel
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12557
• Pdf URL: https://arxiv.org/pdf/2410.12557
• Copy Paste: [[2410.12557]] One Step Diffusion via Shortcut Models(https://arxiv.org/abs/2410.12557)
• Keywords: diffusion, generative
• Abstract: Diffusion models and flow-matching models have enabled generating diverse and realistic images by learning to transfer noise to data. However, sampling from these models involves iterative denoising over many neural network passes, making generation slow and expensive. Previous approaches for speeding up sampling require complex training regimes, such as multiple training phases, multiple networks, or fragile scheduling. We introduce shortcut models, a family of generative models that use a single network and training phase to produce high-quality samples in a single or multiple sampling steps. Shortcut models condition the network not only on the current noise level but also on the desired step size, allowing the model to skip ahead in the generation process. Across a wide range of sampling step budgets, shortcut models consistently produce higher quality samples than previous approaches, such as consistency models and reflow. Compared to distillation, shortcut models reduce complexity to a single network and training phase and additionally allow varying step budgets at inference time.

Title: Can We Reverse In-Context Knowledge Edits?

• Authors: Paul Youssef, Zhixue Zhao, Jörg Schlötterer, Christin Seifert
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12586
• Pdf URL: https://arxiv.org/pdf/2410.12586
• Copy Paste: [[2410.12586]] Can We Reverse In-Context Knowledge Edits?(https://arxiv.org/abs/2410.12586)
• Keywords: in-context
• Abstract: In-context knowledge editing (IKE) enables efficient modification of large language model (LLM) outputs without parameter changes and at zero-cost. However, it can be misused to manipulate responses opaquely, e.g., insert misinformation or offensive content. Such malicious interventions could be incorporated into high-level wrapped APIs where the final input prompt is not shown to end-users. To address this issue, we investigate the detection and reversal of IKE-edits. First, we demonstrate that IKE-edits can be detected with high accuracy (F1 > 80\%) using only the top-10 output probabilities of the next token, even in a black-box setting, e.g. proprietary LLMs with limited output information. Further, we introduce the novel task of reversing IKE-edits using specially tuned reversal tokens. We explore using both continuous and discrete reversal tokens, achieving over 80\% accuracy in recovering original, unedited outputs across multiple LLMs. Our continuous reversal tokens prove particularly effective, with minimal impact on unedited prompts. Through analysis of output distributions, attention patterns, and token rankings, we provide insights into IKE's effects on LLMs and how reversal tokens mitigate them. This work represents a significant step towards enhancing LLM resilience against potential misuse of in-context editing, improving their transparency and trustworthiness.

Title: CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

• Authors: Zhiyuan Ma, Jianjun Li, Guohui Li, Kaiyan Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12595
• Pdf URL: https://arxiv.org/pdf/2410.12595
• Copy Paste: [[2410.12595]] CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training(https://arxiv.org/abs/2410.12595)
• Keywords: self-supervised
• Abstract: With the flourishing of social media platforms, vision-language pre-training (VLP) recently has received great attention and many remarkable progresses have been achieved. The success of VLP largely benefits from the information complementation and enhancement between different modalities. However, most of recent studies focus on cross-modal contrastive learning (CMCL) to promote image-text alignment by pulling embeddings of positive sample pairs together while pushing those of negative pairs apart, which ignores the natural asymmetry property between different modalities and requires large-scale image-text corpus to achieve arduous progress. To mitigate this predicament, we propose CMAL, a Cross-Modal Associative Learning framework with anchor points detection and cross-modal associative learning for VLP. Specifically, we first respectively embed visual objects and textual tokens into separate hypersphere spaces to learn intra-modal hidden features, and then design a cross-modal associative prompt layer to perform anchor point masking and swap feature filling for constructing a hybrid cross-modal associative prompt. Afterwards, we exploit a unified semantic encoder to learn their cross-modal interactive features for context adaptation. Finally, we design an associative mapping classification layer to learn potential associative mappings between modalities at anchor points, within which we develop a fresh self-supervised associative mapping classification task to boost CMAL's performance. Experimental results verify the effectiveness of CMAL, showing that it achieves competitive performance against previous CMCL-based methods on four common downstream vision-and-language tasks, with significantly fewer corpus. Especially, CMAL obtains new state-of-the-art results on SNLI-VE and REC (testA).

Title: Self-Supervised Learning of Disentangled Representations for Multivariate Time-Series

• Authors: Ching Chang, Chiao-Tung Chan, Wei-Yao Wang, Wen-Chih Peng, Tien-Fu Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12606
• Pdf URL: https://arxiv.org/pdf/2410.12606
• Copy Paste: [[2410.12606]] Self-Supervised Learning of Disentangled Representations for Multivariate Time-Series(https://arxiv.org/abs/2410.12606)
• Keywords: self-supervised
• Abstract: Multivariate time-series data in fields like healthcare and industry are informative but challenging due to high dimensionality and lack of labels. Recent self-supervised learning methods excel in learning rich representations without labels but struggle with disentangled embeddings and inductive bias issues like transformation-invariance. To address these challenges, we introduce TimeDRL, a framework for multivariate time-series representation learning with dual-level disentangled embeddings. TimeDRL features: (i) disentangled timestamp-level and instance-level embeddings using a [CLS] token strategy; (ii) timestamp-predictive and instance-contrastive tasks for representation learning; and (iii) avoidance of augmentation methods to eliminate inductive biases. Experiments on forecasting and classification datasets show TimeDRL outperforms existing methods, with further validation in semi-supervised settings with limited labeled data.

Title: Towards Graph Foundation Models: The Perspective of Zero-shot Reasoning on Knowledge Graphs

• Authors: Kai Wang, Siqiang Luo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12609
• Pdf URL: https://arxiv.org/pdf/2410.12609
• Copy Paste: [[2410.12609]] Towards Graph Foundation Models: The Perspective of Zero-shot Reasoning on Knowledge Graphs(https://arxiv.org/abs/2410.12609)
• Keywords: foundation model
• Abstract: Inspired by the success of artificial general intelligence, there is a trend towards developing Graph Foundation Models that excel in generalization across various graph tasks and domains. However, current models often require extensive training or fine-tuning to capture structural and semantic insights on new graphs, which limits their versatility. In this work, we explore graph foundation models from the perspective of zero-shot reasoning on Knowledge Graphs (KGs). Our focus is on utilizing KGs as a unified topological structure to tackle diverse tasks, while addressing semantic isolation challenges in KG reasoning to effectively integrate diverse semantic and structural features. This brings us new methodological insights into KG reasoning, as well as high generalizability towards foundation models in practice. Methodologically, we introduce SCORE, a unified graph reasoning framework that effectively generalizes diverse graph tasks using zero-shot learning. At the core of SCORE is semantic conditional message passing, a technique designed to capture both structural and semantic invariances in graphs, with theoretical backing for its expressive power. Practically, we evaluate the zero-shot reasoning capability of SCORE using 38 diverse graph datasets, covering node-level, link-level, and graph-level tasks across multiple domains. Our experiments reveal a substantial performance improvement over prior foundation models and supervised baselines, highlighting the efficacy and adaptability of our approach.

Title: Constrained Posterior Sampling: Time Series Generation with Hard Constraints

• Authors: Sai Shankar Narasimhan, Shubhankar Agarwal, Litu Rout, Sanjay Shakkottai, Sandeep P. Chinchali
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.12652
• Pdf URL: https://arxiv.org/pdf/2410.12652
• Copy Paste: [[2410.12652]] Constrained Posterior Sampling: Time Series Generation with Hard Constraints(https://arxiv.org/abs/2410.12652)
• Keywords: diffusion
• Abstract: Generating realistic time series samples is crucial for stress-testing models and protecting user privacy by using synthetic data. In engineering and safety-critical applications, these samples must meet certain hard constraints that are domain-specific or naturally imposed by physics or nature. Consider, for example, generating electricity demand patterns with constraints on peak demand times. This can be used to stress-test the functioning of power grids during adverse weather conditions. Existing approaches for generating constrained time series are either not scalable or degrade sample quality. To address these challenges, we introduce Constrained Posterior Sampling (CPS), a diffusion-based sampling algorithm that aims to project the posterior mean estimate into the constraint set after each denoising update. Notably, CPS scales to a large number of constraints (~100) without requiring additional training. We provide theoretical justifications highlighting the impact of our projection step on sampling. Empirically, CPS outperforms state-of-the-art methods in sample quality and similarity to real time series by around 10% and 42%, respectively, on real-world stocks, traffic, and air quality datasets.

Title: Evaluating Morphological Compositional Generalization in Large Language Models

• Authors: Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Lonneke van der Plas, Duygu Ataman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12656
• Pdf URL: https://arxiv.org/pdf/2410.12656
• Copy Paste: [[2410.12656]] Evaluating Morphological Compositional Generalization in Large Language Models(https://arxiv.org/abs/2410.12656)
• Keywords: generative
• Abstract: Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabilities remain questionable, raising doubts about whether these models learn language similarly to humans. While humans exhibit compositional generalization and linguistic creativity in language use, the extent to which LLMs replicate these abilities, particularly in morphology, is under-explored. In this work, we systematically investigate the morphological generalization abilities of LLMs through the lens of compositionality. We define morphemes as compositional primitives and design a novel suite of generative and discriminative tasks to assess morphological productivity and systematicity. Focusing on agglutinative languages such as Turkish and Finnish, we evaluate several state-of-the-art instruction-finetuned multilingual models, including GPT-4 and Gemini. Our analysis shows that LLMs struggle with morphological compositional generalization particularly when applied to novel word roots, with performance declining sharply as morphological complexity increases. While models can identify individual morphological combinations better than chance, their performance lacks systematicity, leading to significant accuracy gaps compared to humans.

Title: Explanation-Preserving Augmentation for Semi-Supervised Graph Representation Learning

• Authors: Zhuomin Chen, Jingchao Ni, Hojat Allah Salehi, Xu Zheng, Esteban Schafir, Farhad Shirani, Dongsheng Luo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12657
• Pdf URL: https://arxiv.org/pdf/2410.12657
• Copy Paste: [[2410.12657]] Explanation-Preserving Augmentation for Semi-Supervised Graph Representation Learning(https://arxiv.org/abs/2410.12657)
• Keywords: self-supervised
• Abstract: Graph representation learning (GRL), enhanced by graph augmentation methods, has emerged as an effective technique achieving performance improvements in wide tasks such as node classification and graph classification. In self-supervised GRL, paired graph augmentations are generated from each graph. Its objective is to infer similar representations for augmentations of the same graph, but maximally distinguishable representations for augmentations of different graphs. Analogous to image and language domains, the desiderata of an ideal augmentation method include both (1) semantics-preservation; and (2) data-perturbation; i.e., an augmented graph should preserve the semantics of its original graph while carrying sufficient variance. However, most existing (un-)/self-supervised GRL methods focus on data perturbation but largely neglect semantics preservation. To address this challenge, in this paper, we propose a novel method, Explanation-Preserving Augmentation (EPA), that leverages graph explanation techniques for generating augmented graphs that can bridge the gap between semantics-preservation and data-perturbation. EPA first uses a small number of labels to train a graph explainer to infer the sub-structures (explanations) that are most relevant to a graph's semantics. These explanations are then used to generate semantics-preserving augmentations for self-supervised GRL, namely EPA-GRL. We demonstrate theoretically, using an analytical example, and through extensive experiments on a variety of benchmark datasets that EPA-GRL outperforms the state-of-the-art (SOTA) GRL methods, which are built upon semantics-agnostic data augmentations.

Title: Automatic Mapping of Anatomical Landmarks from Free-Text Using Large Language Models: Insights from Llama-2

• Authors: Mohamad Abdi, Gerardo Hemosillo Valadez, Halid Ziya Yerebakan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12686
• Pdf URL: https://arxiv.org/pdf/2410.12686
• Copy Paste: [[2410.12686]] Automatic Mapping of Anatomical Landmarks from Free-Text Using Large Language Models: Insights from Llama-2(https://arxiv.org/abs/2410.12686)
• Keywords: generative, anomaly
• Abstract: Anatomical landmarks are vital in medical imaging for navigation and anomaly detection. Modern large language models (LLMs), like Llama-2, offer promise for automating the mapping of these landmarks in free-text radiology reports to corresponding positions in image data. Recent studies propose LLMs may develop coherent representations of generative processes. Motivated by these insights, we investigated whether LLMs accurately represent the spatial positions of anatomical landmarks. Through experiments with Llama-2 models, we found that they can linearly represent anatomical landmarks in space with considerable robustness to different prompts. These results underscore the potential of LLMs to enhance the efficiency and accuracy of medical imaging workflows.

Title: MultiCamCows2024 -- A Multi-view Image Dataset for AI-driven Holstein-Friesian Cattle Re-Identification on a Working Farm

• Authors: Phoenix Yu, Tilo Burghardt, Andrew W Dowsey, Neill W Campbell
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12695
• Pdf URL: https://arxiv.org/pdf/2410.12695
• Copy Paste: [[2410.12695]] MultiCamCows2024 -- A Multi-view Image Dataset for AI-driven Holstein-Friesian Cattle Re-Identification on a Working Farm(https://arxiv.org/abs/2410.12695)
• Keywords: self-supervised
• Abstract: We present MultiCamCows2024, a farm-scale image dataset filmed across multiple cameras for the biometric identification of individual Holstein-Friesian cattle exploiting their unique black and white coat-patterns. Captured by three ceiling-mounted visual sensors covering adjacent barn areas over seven days on a working dairy farm, the dataset comprises 101, 329 images of 90 cows, plus the underlying original CCTV footage. The dataset is provided alongside full computer vision recognition baselines, that is both a supervised and self-supervised learning framework for individual cow identification trained on cattle tracklets. We report a performance above 96% single image identification accuracy from the dataset and demonstrate that combining data from multiple cameras during learning enhances self-supervised identification. We show that our framework enables fully automatic cattle identification, barring only the simple human verification of tracklet integrity during data collection. Crucially, our study highlights that multi-camera, supervised and self-supervised components in tandem not only deliver highly accurate individual cow identification but also achieve this efficiently with no labelling of cattle identities by humans at all. We argue that this improvement in efficacy has practical implications for livestock management, behaviour analysis, and agricultural monitoring. For full reproducibility and practical ease of use, we publish all key software and code including re-identification components and the species detector with this paper.

Title: AdaptiveDrag: Semantic-Driven Dragging on Diffusion-Based Image Editing

• Authors: DuoSheng Chen, Binghui Chen, Yifeng Geng, Liefeng Bo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12696
• Pdf URL: https://arxiv.org/pdf/2410.12696
• Copy Paste: [[2410.12696]] AdaptiveDrag: Semantic-Driven Dragging on Diffusion-Based Image Editing(https://arxiv.org/abs/2410.12696)
• Keywords: diffusion, generative
• Abstract: Recently, several point-based image editing methods (e.g., DragDiffusion, FreeDrag, DragNoise) have emerged, yielding precise and high-quality results based on user instructions. However, these methods often make insufficient use of semantic information, leading to less desirable results. In this paper, we proposed a novel mask-free point-based image editing method, AdaptiveDrag, which provides a more flexible editing approach and generates images that better align with user intent. Specifically, we design an auto mask generation module using super-pixel division for user-friendliness. Next, we leverage a pre-trained diffusion model to optimize the latent, enabling the dragging of features from handle points to target points. To ensure a comprehensive connection between the input image and the drag process, we have developed a semantic-driven optimization. We design adaptive steps that are supervised by the positions of the points and the semantic regions derived from super-pixel segmentation. This refined optimization process also leads to more realistic and accurate drag results. Furthermore, to address the limitations in the generative consistency of the diffusion model, we introduce an innovative corresponding loss during the sampling process. Building on these effective designs, our method delivers superior generation results using only the single input image and the handle-target point pairs. Extensive experiments have been conducted and demonstrate that the proposed method outperforms others in handling various drag instructions (e.g., resize, movement, extension) across different domains (e.g., animals, human face, land space, clothing).

Title: Embedding an Ethical Mind: Aligning Text-to-Image Synthesis via Lightweight Value Optimization

• Authors: Xingqi Wang, Xiaoyuan Yi, Xing Xie, Jia Jia
• Subjects: cs.CV, cs.AI, cs.CY, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.12700
• Pdf URL: https://arxiv.org/pdf/2410.12700
• Copy Paste: [[2410.12700]] Embedding an Ethical Mind: Aligning Text-to-Image Synthesis via Lightweight Value Optimization(https://arxiv.org/abs/2410.12700)
• Keywords: diffusion
• Abstract: Recent advancements in diffusion models trained on large-scale data have enabled the generation of indistinguishable human-level images, yet they often produce harmful content misaligned with human values, e.g., social bias, and offensive content. Despite extensive research on Large Language Models (LLMs), the challenge of Text-to-Image (T2I) model alignment remains largely unexplored. Addressing this problem, we propose LiVO (Lightweight Value Optimization), a novel lightweight method for aligning T2I models with human values. LiVO only optimizes a plug-and-play value encoder to integrate a specified value principle with the input prompt, allowing the control of generated images over both semantics and values. Specifically, we design a diffusion model-tailored preference optimization loss, which theoretically approximates the Bradley-Terry model used in LLM alignment but provides a more flexible trade-off between image quality and value conformity. To optimize the value encoder, we also develop a framework to automatically construct a text-image preference dataset of 86k (prompt, aligned image, violating image, value principle) samples. Without updating most model parameters and through adaptive value selection from the input prompt, LiVO significantly reduces harmful outputs and achieves faster convergence, surpassing several strong baselines and taking an initial step towards ethically aligned T2I models.

Title: Sarcasm Detection in a Less-Resourced Language

• Authors: Lazar Đoković, Marko Robnik-Šikonja
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12704
• Pdf URL: https://arxiv.org/pdf/2410.12704
• Copy Paste: [[2410.12704]] Sarcasm Detection in a Less-Resourced Language(https://arxiv.org/abs/2410.12704)
• Keywords: generative
• Abstract: The sarcasm detection task in natural language processing tries to classify whether an utterance is sarcastic or not. It is related to sentiment analysis since it often inverts surface sentiment. Because sarcastic sentences are highly dependent on context, and they are often accompanied by various non-verbal cues, the task is challenging. Most of related work focuses on high-resourced languages like English. To build a sarcasm detection dataset for a less-resourced language, such as Slovenian, we leverage two modern techniques: a machine translation specific medium-size transformer model, and a very large generative language model. We explore the viability of translated datasets and how the size of a pretrained transformer affects its ability to detect sarcasm. We train ensembles of detection models and evaluate models' performance. The results show that larger models generally outperform smaller ones and that ensembling can slightly improve sarcasm detection performance. Our best ensemble approach achieves an $\text{F}_1$-score of 0.765 which is close to annotators' agreement in the source language.

Title: Counterfactual Generative Modeling with Variational Causal Inference

• Authors: Yulun Wu, Louie McConnell, Claudia Iriondo
• Subjects: cs.LG, cs.AI, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12730
• Pdf URL: https://arxiv.org/pdf/2410.12730
• Copy Paste: [[2410.12730]] Counterfactual Generative Modeling with Variational Causal Inference(https://arxiv.org/abs/2410.12730)
• Keywords: generative
• Abstract: Estimating an individual's potential outcomes under counterfactual treatments is a challenging task for traditional causal inference and supervised learning approaches when the outcome is high-dimensional (e.g. gene expressions, facial images) and covariates are relatively limited. In this case, to predict one's outcomes under counterfactual treatments, it is crucial to leverage individual information contained in its high-dimensional observed outcome in addition to the covariates. Prior works using variational inference in counterfactual generative modeling have been focusing on neural adaptations and model variants within the conditional variational autoencoder formulation, which we argue is fundamentally ill-suited to the notion of counterfactual in causal inference. In this work, we present a novel variational Bayesian causal inference framework and its theoretical backings to properly handle counterfactual generative modeling tasks, through which we are able to conduct counterfactual supervision end-to-end during training without any counterfactual samples, and encourage latent disentanglement that aids the correct identification of causal effect in counterfactual generations. In experiments, we demonstrate the advantage of our framework compared to state-of-the-art models in counterfactual generative modeling on multiple benchmarks.

Title: SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation

• Authors: Jaehong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, Mohit Bansal
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12761
• Pdf URL: https://arxiv.org/pdf/2410.12761
• Copy Paste: [[2410.12761]] SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation(https://arxiv.org/abs/2410.12761)
• Keywords: diffusion
• Abstract: Recent advances in diffusion models have significantly enhanced their ability to generate high-quality images and videos, but they have also increased the risk of producing unsafe content. Existing unlearning/editing-based methods for safe generation remove harmful concepts from models but face several challenges: (1) They cannot instantly remove harmful concepts without training. (2) Their safe generation capabilities depend on collected training data. (3) They alter model weights, risking degradation in quality for content unrelated to toxic concepts. To address these, we propose SAFREE, a novel, training-free approach for safe T2I and T2V, that does not alter the model's weights. Specifically, we detect a subspace corresponding to a set of toxic concepts in the text embedding space and steer prompt embeddings away from this subspace, thereby filtering out harmful content while preserving intended semantics. To balance the trade-off between filtering toxicity and preserving safe concepts, SAFREE incorporates a novel self-validating filtering mechanism that dynamically adjusts the denoising steps when applying the filtered embeddings. Additionally, we incorporate adaptive re-attention mechanisms within the diffusion latent space to selectively diminish the influence of features related to toxic concepts at the pixel level. In the end, SAFREE ensures coherent safety checking, preserving the fidelity, quality, and safety of the output. SAFREE achieves SOTA performance in suppressing unsafe content in T2I generation compared to training-free baselines and effectively filters targeted concepts while maintaining high-quality images. It also shows competitive results against training-based methods. We extend SAFREE to various T2I backbones and T2V tasks, showcasing its flexibility and generalization. SAFREE provides a robust and adaptable safeguard for ensuring safe visual generation.

Title: The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse

• Authors: Ekansh Sharma, Daniel M. Roy, Gintare Karolina Dziugaite
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12766
• Pdf URL: https://arxiv.org/pdf/2410.12766
• Copy Paste: [[2410.12766]] The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse(https://arxiv.org/abs/2410.12766)
• Keywords: foundation model
• Abstract: Model merging aims to efficiently combine the weights of multiple expert models, each trained on a specific task, into a single multi-task model, with strong performance across all tasks. When applied to all but the last layer of weights, existing methods -- such as Task Arithmetic, TIES-merging, and TALL mask merging -- work well to combine expert models obtained by fine-tuning a common foundation model, operating within a "local" neighborhood of the foundation model. This work explores the more challenging scenario of "non-local" merging, which we find arises when an expert model changes significantly during pretraining or where the expert models do not even share a common foundation model. We observe that standard merging techniques often fail to generalize effectively in this non-local setting, even when accounting for permutation symmetries using standard techniques. We identify that this failure is, in part, due to "variance collapse", a phenomenon identified also in the setting of linear mode connectivity by Jordan et al. (2023). To address this, we propose a multi-task technique to re-scale and shift the output activations of the merged model for each task, aligning its output statistics with those of the corresponding task-specific expert models. Our experiments demonstrate that this correction significantly improves the performance of various model merging approaches in non-local settings, providing a strong baseline for future research on this problem.

Title: Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Concepts

• Authors: Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijie Deng, Min Lin
• Subjects: cs.CV, cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12777
• Pdf URL: https://arxiv.org/pdf/2410.12777
• Copy Paste: [[2410.12777]] Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Concepts(https://arxiv.org/abs/2410.12777)
• Keywords: diffusion
• Abstract: With the rapid progress of diffusion-based content generation, significant efforts are being made to unlearn harmful or copyrighted concepts from pretrained diffusion models (DMs) to prevent potential model misuse. However, it is observed that even when DMs are properly unlearned before release, malicious finetuning can compromise this process, causing DMs to relearn the unlearned concepts. This occurs partly because certain benign concepts (e.g., "skin") retained in DMs are related to the unlearned ones (e.g., "nudity"), facilitating their relearning via finetuning. To address this, we propose meta-unlearning on DMs. Intuitively, a meta-unlearned DM should behave like an unlearned DM when used as is; moreover, if the meta-unlearned DM undergoes malicious finetuning on unlearned concepts, the related benign concepts retained within it will be triggered to self-destruct, hindering the relearning of unlearned concepts. Our meta-unlearning framework is compatible with most existing unlearning methods, requiring only the addition of an easy-to-implement meta objective. We validate our approach through empirical experiments on meta-unlearning concepts from Stable Diffusion models (SD-v1-4 and SDXL), supported by extensive ablation studies. Our code is available at this https URL.

Title: Geometry-Aware Generative Autoencoders for Warped Riemannian Metric Learning and Generative Modeling on Data Manifolds

• Authors: Xingzhi Sun, Danqi Liao, Kincaid MacDonald, Yanlei Zhang, Chen Liu, Guillaume Huguet, Guy Wolf, Ian Adelstein, Tim G. J. Rudner, Smita Krishnaswamy
• Subjects: cs.LG, math.DG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12779
• Pdf URL: https://arxiv.org/pdf/2410.12779
• Copy Paste: [[2410.12779]] Geometry-Aware Generative Autoencoders for Warped Riemannian Metric Learning and Generative Modeling on Data Manifolds(https://arxiv.org/abs/2410.12779)
• Keywords: generative
• Abstract: Rapid growth of high-dimensional datasets in fields such as single-cell RNA sequencing and spatial genomics has led to unprecedented opportunities for scientific discovery, but it also presents unique computational and statistical challenges. Traditional methods struggle with geometry-aware data generation, interpolation along meaningful trajectories, and transporting populations via feasible paths. To address these issues, we introduce Geometry-Aware Generative Autoencoder (GAGA), a novel framework that combines extensible manifold learning with generative modeling. GAGA constructs a neural network embedding space that respects the intrinsic geometries discovered by manifold learning and learns a novel warped Riemannian metric on the data space. This warped metric is derived from both the points on the data manifold and negative samples off the manifold, allowing it to characterize a meaningful geometry across the entire latent space. Using this metric, GAGA can uniformly sample points on the manifold, generate points along geodesics, and interpolate between populations across the learned manifold. GAGA shows competitive performance in simulated and real world datasets, including a 30% improvement over the state-of-the-art methods in single-cell population-level trajectory inference.

Title: Context-Scaling versus Task-Scaling in In-Context Learning

• Authors: Amirhesam Abedsoltan, Adityanarayanan Radhakrishnan, Jingfeng Wu, Mikhail Belkin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12783
• Pdf URL: https://arxiv.org/pdf/2410.12783
• Copy Paste: [[2410.12783]] Context-Scaling versus Task-Scaling in In-Context Learning(https://arxiv.org/abs/2410.12783)
• Keywords: in-context
• Abstract: Transformers exhibit In-Context Learning (ICL), where these models solve new tasks by using examples in the prompt without additional training. In our work, we identify and analyze two key components of ICL: (1) context-scaling, where model performance improves as the number of in-context examples increases and (2) task-scaling, where model performance improves as the number of pre-training tasks increases. While transformers are capable of both context-scaling and task-scaling, we empirically show that standard Multi-Layer Perceptrons (MLPs) with vectorized input are only capable of task-scaling. To understand how transformers are capable of context-scaling, we first propose a significantly simplified transformer architecture without key, query, value weights. We show that it performs ICL comparably to the original GPT-2 model in various statistical learning tasks including linear regression, teacher-student settings. Furthermore, a single block of our simplified transformer can be viewed as data dependent feature map followed by an MLP. This feature map on its own is a powerful predictor that is capable of context-scaling but is not capable of task-scaling. We show empirically that concatenating the output of this feature map with vectorized data as an input to MLPs enables both context-scaling and task-scaling. This finding provides a simple setting to study context and task-scaling for ICL.