2024-10-25

Title: Graph Contrastive Learning via Cluster-refined Negative Sampling for Semi-supervised Text Classification

• Authors: Wei Ai, Jianbin Li, Ze Wang, Jiayi Du, Tao Meng, Yuntao Shou, Keqin Li
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18130
• Pdf URL: https://arxiv.org/pdf/2410.18130
• Copy Paste: [[2410.18130]] Graph Contrastive Learning via Cluster-refined Negative Sampling for Semi-supervised Text Classification(https://arxiv.org/abs/2410.18130)
• Keywords: self-supervised
• Abstract: Graph contrastive learning (GCL) has been widely applied to text classification tasks due to its ability to generate self-supervised signals from unlabeled data, thus facilitating model training. However, existing GCL-based text classification methods often suffer from negative sampling bias, where similar nodes are incorrectly paired as negative pairs. This can lead to over-clustering, where instances of the same class are divided into different clusters. To address the over-clustering issue, we propose an innovative GCL-based method of graph contrastive learning via cluster-refined negative sampling for semi-supervised text classification, namely ClusterText. Firstly, we combine the pre-trained model Bert with graph neural networks to learn text representations. Secondly, we introduce a clustering refinement strategy, which clusters the learned text representations to obtain pseudo labels. For each text node, its negative sample set is drawn from different clusters. Additionally, we propose a self-correction mechanism to mitigate the loss of true negative samples caused by clustering inconsistency. By calculating the Euclidean distance between each text node and other nodes within the same cluster, distant nodes are still selected as negative samples. Our proposed ClusterText demonstrates good scalable computing, as it can effectively extract important information from from a large amount of data. Experimental results demonstrate the superiority of ClusterText in text classification tasks.

Title: Advancing Super-Resolution in Neural Radiance Fields via Variational Diffusion Strategies

• Authors: Shrey Vishen, Jatin Sarabu, Chinmay Bharathulwar, Rithwick Lakshmanan, Vishnu Srinivas
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18137
• Pdf URL: https://arxiv.org/pdf/2410.18137
• Copy Paste: [[2410.18137]] Advancing Super-Resolution in Neural Radiance Fields via Variational Diffusion Strategies(https://arxiv.org/abs/2410.18137)
• Keywords: diffusion
• Abstract: We present a novel method for diffusion-guided frameworks for view-consistent super-resolution (SR) in neural rendering. Our approach leverages existing 2D SR models in conjunction with advanced techniques such as Variational Score Distilling (VSD) and a LoRA fine-tuning helper, with spatial training to significantly boost the quality and consistency of upscaled 2D images compared to the previous methods in the literature, such as Renoised Score Distillation (RSD) proposed in DiSR-NeRF (1), or SDS proposed in DreamFusion. The VSD score facilitates precise fine-tuning of SR models, resulting in high-quality, view-consistent images. To address the common challenge of inconsistencies among independent SR 2D images, we integrate Iterative 3D Synchronization (I3DS) from the DiSR-NeRF framework. Our quantitative benchmarks and qualitative results on the LLFF dataset demonstrate the superior performance of our system compared to existing methods such as DiSR-NeRF.

Title: Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction

• Authors: Nicholas Walker
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18160
• Pdf URL: https://arxiv.org/pdf/2410.18160
• Copy Paste: [[2410.18160]] Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction(https://arxiv.org/abs/2410.18160)
• Keywords: generative
• Abstract: Causal decoder-only transformer models used for generative language modelling, such as Generative Pre-trained Transformers (GPT), are trained to predict the next token in a sequence based only on its previous tokens. Despite this simple training objective, they have proved to be powerful AI tools. However, only predicting the next token results in top layer embedding vectors that are highly token-focused. There may be benefits in generating embedding vectors at each token position that better capture the overall meaning of longer sequences of future text. Recent studies matching brain scans with deep language models suggest that humans also predict upcoming words when listening or reading but consider multiple future tokens rather than just one. This research investigates a new pretraining method called Future Token Prediction (FTP). In FTP, a large transformer encoder generates top layer embedding vectors for each token position, which, instead of being passed to a language head, are linearly and expansively projected to a pseudo-sequence, which is cross attended to by a small transformer decoder to predict the next N tokens forward from that position in the sequence. The top layer embedding vectors from FTP models exhibit distinct properties compared to those from standard GPT models, varying smoothly along a text sequence as measured by cosine similarity between adjacent tokens. Text generated by FTP models show improved topic coherence compared to standard GPT-like models trained with the same prediction perplexity for the next single token. The vectors are shown to better represent the topic of text based on the results of text classification examples. On a toy, but complex, coding problem, FTP networks produce significantly better results than GPT networks.

Title: Gazelle: An Instruction Dataset for Arabic Writing Assistance

• Authors: Samar M. Magdy, Fakhraddin Alwajih, Sang Yun Kwon, Reem Abdel-Salam, Muhammad Abdul-Mageed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18163
• Pdf URL: https://arxiv.org/pdf/2410.18163
• Copy Paste: [[2410.18163]] Gazelle: An Instruction Dataset for Arabic Writing Assistance(https://arxiv.org/abs/2410.18163)
• Keywords: generative
• Abstract: Writing has long been considered a hallmark of human intelligence and remains a pinnacle task for artificial intelligence (AI) due to the intricate cognitive processes involved. Recently, rapid advancements in generative AI, particularly through the development of Large Language Models (LLMs), have significantly transformed the landscape of writing assistance. However, underrepresented languages like Arabic encounter significant challenges in the development of advanced AI writing tools, largely due to the limited availability of data. This scarcity constrains the training of effective models, impeding the creation of sophisticated writing assistance technologies. To address these issues, we present Gazelle, a comprehensive dataset for Arabic writing assistance. In addition, we offer an evaluation framework designed to enhance Arabic writing assistance tools. Our human evaluation of leading LLMs, including GPT-4, GPT-4o, Cohere Command R+, and Gemini 1.5 Pro, highlights their respective strengths and limitations in addressing the challenges of Arabic writing. Our findings underscore the need for continuous model training and dataset enrichment to manage the complexities of Arabic language processing, paving the way for more effective AI-powered Arabic writing tools.

Title: TabDPT: Scaling Tabular Foundation Models

• Authors: Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, Anthony L. Caterini
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.18164
• Pdf URL: https://arxiv.org/pdf/2410.18164
• Copy Paste: [[2410.18164]] TabDPT: Scaling Tabular Foundation Models(https://arxiv.org/abs/2410.18164)
• Keywords: self-supervised, foundation model, in-context
• Abstract: The challenges faced by neural networks on tabular data are well-documented and have hampered the progress of tabular foundation models. Techniques leveraging in-context learning (ICL) have shown promise here, allowing for dynamic adaptation to unseen data. ICL can provide predictions for entirely new datasets without further training or hyperparameter tuning, therefore providing very fast inference when encountering a novel task. However, scaling ICL for tabular data remains an issue: approaches based on large language models cannot efficiently process numeric tables, and tabular-specific techniques have not been able to effectively harness the power of real data to improve performance and generalization. We are able to overcome these challenges by training tabular-specific ICL-based architectures on real data with self-supervised learning and retrieval, combining the best of both worlds. Our resulting model -- the Tabular Discriminative Pre-trained Transformer (TabDPT) -- achieves state-of-the-art performance on the CC18 (classification) and CTR23 (regression) benchmarks with no task-specific fine-tuning, demonstrating the adapatability and speed of ICL once the model is pre-trained. TabDPT also demonstrates strong scaling as both model size and amount of available data increase, pointing towards future improvements simply through the curation of larger tabular pre-training datasets and training larger models.

Title: CorrectionLM: Self-Corrections with SLM for Dialogue State Tracking

• Authors: Chia-Hsuan Lee, Hao Cheng, Mari Ostendorf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18209
• Pdf URL: https://arxiv.org/pdf/2410.18209
• Copy Paste: [[2410.18209]] CorrectionLM: Self-Corrections with SLM for Dialogue State Tracking(https://arxiv.org/abs/2410.18209)
• Keywords: in-context
• Abstract: Large language models (LLMs) have demonstrated self-improvement capabilities via feedback and refinement, but current small language models (SLMs) have had limited success in this area. Existing correction approaches often rely on distilling knowledge from LLMs, which imposes significant computation demands. In this work, we introduce CORRECTIONLM, a novel correction framework that enables SLMs to self-correct using in-context exemplars without LLM involvement. Applied to two dialogue state tracking (DST) tasks in low-resource settings, CORRECTIONLM achieves results similar to a state-of-the-art LLM at a small fraction of the computation costs.

Title: DMTG: A Human-Like Mouse Trajectory Generation Bot Based on Entropy-Controlled Diffusion Networks

• Authors: Jiahua Liu, Zeyuan Cui, Wenhan Ge, Pengxiang Zhan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18233
• Pdf URL: https://arxiv.org/pdf/2410.18233
• Copy Paste: [[2410.18233]] DMTG: A Human-Like Mouse Trajectory Generation Bot Based on Entropy-Controlled Diffusion Networks(https://arxiv.org/abs/2410.18233)
• Keywords: diffusion
• Abstract: CAPTCHAs protect against resource misuse and data theft by distinguishing human activity from automated bots. Advances in machine learning have made traditional image and text-based CAPTCHAs vulnerable to attacks, leading modern CAPTCHAs, such as GeeTest and Akamai, to incorporate behavioral analysis like mouse trajectory detection. Existing bypass techniques struggle to fully mimic human behavior, making it difficult to evaluate the effectiveness of anti-bot measures. To address this, we propose a diffusion model-based mouse trajectory generation framework (DMTG), which controls trajectory complexity and produces realistic human-like mouse movements. DMTG also provides white-box and black-box testing methods to assess its ability to bypass CAPTCHA systems. In experiments, DMTG reduces bot detection accuracy by 4.75%-9.73% compared to other models. Additionally, it mimics physical human behaviors, such as slow initiation and directional force differences, demonstrating improved performance in both simulation and real-world CAPTCHA scenarios.

Title: Countering Autonomous Cyber Threats

• Authors: Kade M. Heckel, Adrian Weller
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.18312
• Pdf URL: https://arxiv.org/pdf/2410.18312
• Copy Paste: [[2410.18312]] Countering Autonomous Cyber Threats(https://arxiv.org/abs/2410.18312)
• Keywords: foundation model, generative
• Abstract: With the capability to write convincing and fluent natural language and generate code, Foundation Models present dual-use concerns broadly and within the cyber domain specifically. Generative AI has already begun to impact cyberspace through a broad illicit marketplace for assisting malware development and social engineering attacks through hundreds of malicious-AI-as-a-services tools. More alarming is that recent research has shown the potential for these advanced models to inform or independently execute offensive cyberspace operations. However, these previous investigations primarily focused on the threats posed by proprietary models due to the until recent lack of strong open-weight model and additionally leave the impacts of network defenses or potential countermeasures unexplored. Critically, understanding the aptitude of downloadable models to function as offensive cyber agents is vital given that they are far more difficult to govern and prevent their misuse. As such, this work evaluates several state-of-the-art FMs on their ability to compromise machines in an isolated network and investigates defensive mechanisms to defeat such AI-powered attacks. Using target machines from a commercial provider, the most recently released downloadable models are found to be on par with a leading proprietary model at conducting simple cyber attacks with common hacking tools against known vulnerabilities. To mitigate such LLM-powered threats, defensive prompt injection (DPI) payloads for disrupting the malicious cyber agent's workflow are demonstrated to be effective. From these results, the implications for AI safety and governance with respect to cybersecurity is analyzed.

Title: Self-Supervised Learning for Time Series: A Review & Critique of FITS

• Authors: Andreas Løvendahl Eefsen, Nicholas Erup Larsen, Oliver Glozmann Bork Hansen, Thor Højhus Avenstrup
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18318
• Pdf URL: https://arxiv.org/pdf/2410.18318
• Copy Paste: [[2410.18318]] Self-Supervised Learning for Time Series: A Review & Critique of FITS(https://arxiv.org/abs/2410.18318)
• Keywords: self-supervised
• Abstract: Accurate time series forecasting is a highly valuable endeavour with applications across many industries. Despite recent deep learning advancements, increased model complexity, and larger model sizes, many state-of-the-art models often perform worse or on par with simpler models. One of those cases is a recently proposed model, FITS, claiming competitive performance with significantly reduced parameter counts. By training a one-layer neural network in the complex frequency domain, we are able to replicate these results. Our experiments on a wide range of real-world datasets further reveal that FITS especially excels at capturing periodic and seasonal patterns, but struggles with trending, non-periodic, or random-resembling behavior. With our two novel hybrid approaches, where we attempt to remedy the weaknesses of FITS by combining it with DLinear, we achieve the best results of any known open-source model on multivariate regression and promising results in multiple/linear regression on price datasets, on top of vastly improving upon what FITS achieves as a standalone model.

Title: FedBaF: Federated Learning Aggregation Biased by a Foundation Model

• Authors: Jong-Ik Park, Srinivasa Pranav, José M. F. Moura, Carlee Joe-Wong
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.18352
• Pdf URL: https://arxiv.org/pdf/2410.18352
• Copy Paste: [[2410.18352]] FedBaF: Federated Learning Aggregation Biased by a Foundation Model(https://arxiv.org/abs/2410.18352)
• Keywords: foundation model
• Abstract: Foundation models are now a major focus of leading technology organizations due to their ability to generalize across diverse tasks. Existing approaches for adapting foundation models to new applications often rely on Federated Learning (FL) and disclose the foundation model weights to clients when using it to initialize the global model. While these methods ensure client data privacy, they compromise model and information security. In this paper, we introduce Federated Learning Aggregation Biased by a Foundation Model (FedBaF), a novel method for dynamically integrating pre-trained foundation model weights during the FL aggregation phase. Unlike conventional methods, FedBaF preserves the confidentiality of the foundation model while still leveraging its power to train more accurate models, especially in non-IID and adversarial scenarios. Our comprehensive experiments use Pre-ResNet and foundation models like Vision Transformer to demonstrate that FedBaF not only matches, but often surpasses the test accuracy of traditional weight initialization methods by up to 11.4\% in IID and up to 15.8\% in non-IID settings. Additionally, FedBaF applied to a Transformer-based language model significantly reduced perplexity by up to 39.2\%.

Title: Harnessing PU Learning for Enhanced Cloud-based DDoS Detection: A Comparative Analysis

• Authors: Robert Dilworth, Charan Gudla
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18380
• Pdf URL: https://arxiv.org/pdf/2410.18380
• Copy Paste: [[2410.18380]] Harnessing PU Learning for Enhanced Cloud-based DDoS Detection: A Comparative Analysis(https://arxiv.org/abs/2410.18380)
• Keywords: anomaly
• Abstract: This paper explores the application of Positive-Unlabeled (PU) learning for enhanced Distributed Denial-of-Service (DDoS) detection in cloud environments. Utilizing the $\texttt{BCCC-cPacket-Cloud-DDoS-2024}$ dataset, we implement PU learning with four machine learning algorithms: XGBoost, Random Forest, Support Vector Machine, and Naïve Bayes. Our results demonstrate the superior performance of ensemble methods, with XGBoost and Random Forest achieving $F_{1}$ scores exceeding 98%. We quantify the efficacy of each approach using metrics including $F_{1}$ score, ROC AUC, Recall, and Precision. This study bridges the gap between PU learning and cloud-based anomaly detection, providing a foundation for addressing Context-Aware DDoS Detection in multi-cloud environments. Our findings highlight the potential of PU learning in scenarios with limited labeled data, offering valuable insights for developing more robust and adaptive cloud security mechanisms.

Title: FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling

• Authors: Zhengqiang Zhang, Ruihuang Li, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18410
• Pdf URL: https://arxiv.org/pdf/2410.18410
• Copy Paste: [[2410.18410]] FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling(https://arxiv.org/abs/2410.18410)
• Keywords: diffusion
• Abstract: While image generation with diffusion models has achieved a great success, generating images of higher resolution than the training size remains a challenging task due to the high computational cost. Current methods typically perform the entire sampling process at full resolution and process all frequency components simultaneously, contradicting with the inherent coarse-to-fine nature of latent diffusion models and wasting computations on processing premature high-frequency details at early diffusion stages. To address this issue, we introduce an efficient $\textbf{Fre}$quency-aware $\textbf{Ca}$scaded $\textbf{S}$ampling framework, $\textbf{FreCaS}$ in short, for higher-resolution image generation. FreCaS decomposes the sampling process into cascaded stages with gradually increased resolutions, progressively expanding frequency bands and refining the corresponding details. We propose an innovative frequency-aware classifier-free guidance (FA-CFG) strategy to assign different guidance strengths for different frequency components, directing the diffusion model to add new details in the expanded frequency domain of each stage. Additionally, we fuse the cross-attention maps of previous and current stages to avoid synthesizing unfaithful layouts. Experiments demonstrate that FreCaS significantly outperforms state-of-the-art methods in image quality and generation speed. In particular, FreCaS is about 2.86$\times$ and 6.07$\times$ faster than ScaleCrafter and DemoFusion in generating a 2048$\times$2048 image using a pre-trained SDXL model and achieves an FID$_b$ improvement of 11.6 and 3.7, respectively. FreCaS can be easily extended to more complex models such as SD3. The source code of FreCaS can be found at $\href{\text{this https URL}}{this https URL}$.

Title: The Nature of Mathematical Modeling and Probabilistic Optimization Engineering in Generative AI

• Authors: Fulu Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18441
• Pdf URL: https://arxiv.org/pdf/2410.18441
• Copy Paste: [[2410.18441]] The Nature of Mathematical Modeling and Probabilistic Optimization Engineering in Generative AI(https://arxiv.org/abs/2410.18441)
• Keywords: generative
• Abstract: In this paper, we give an in-depth analysis on the mathematical problem formulations and the probabilistic optimization explorations for some of the key components in Transformer model [33] in the field of generative AI. We explore and discuss some potential further enhancement for current state of the art methods for some key underlying technologies of generative AI models from algorithmic and probabilistic optimization perspective. In particular, we present an optimal solution for sub-word encoding (SWE) based on similar initial settings as that of byte-pair encoding (BPE) algorithm in [9] with similar objectives as that of WordPiece approach in [28, 31] to maximize the likelihood of the training data. We also present cross entropy optimization method to optimize hyperparameters for word2vec model [17]. In addition, we propose a factored combination of rotary positional encoding (RoPE) [32] and attention with linear biases (ALiBi) [23] with a harmonic series. We also present a probabilistic FlashAttention [6, 7] (PrFlashAttention) method with a probability distribution over block distances in the matrix to decide which block is likely to participate in a given round of attention computation while maintaining the lower triangle shape of the tensor for autoregressive language models by re-shaping the tensors. Finally, we present staircase adaptive quantization (SAQ) of key-value (KV) cache for multi-query attention (MQA) based on the framework presented in [16] to have gradual quantization degradation while achieving reasonable model quality and cost savings.

Title: Graph Pre-Training Models Are Strong Anomaly Detectors

• Authors: Jiashun Cheng, Zinan Zheng, Yang Liu, Jianheng Tang, Hongwei Wang, Yu Rong, Jia Li, Fugee Tsung
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18487
• Pdf URL: https://arxiv.org/pdf/2410.18487
• Copy Paste: [[2410.18487]] Graph Pre-Training Models Are Strong Anomaly Detectors(https://arxiv.org/abs/2410.18487)
• Keywords: anomaly
• Abstract: Graph Anomaly Detection (GAD) is a challenging and practical research topic where Graph Neural Networks (GNNs) have recently shown promising results. The effectiveness of existing GNNs in GAD has been mainly attributed to the simultaneous learning of node representations and the classifier in an end-to-end manner. Meanwhile, graph pre-training, the two-stage learning paradigm such as DGI and GraphMAE, has shown potential in leveraging unlabeled graph data to enhance downstream tasks, yet its impact on GAD remains under-explored. In this work, we show that graph pre-training models are strong graph anomaly detectors. Specifically, we demonstrate that pre-training is highly competitive, markedly outperforming the state-of-the-art end-to-end training models when faced with limited supervision. To understand this phenomenon, we further uncover pre-training enhances the detection of distant, under-represented, unlabeled anomalies that go beyond 2-hop neighborhoods of known anomalies, shedding light on its superior performance against end-to-end models. Moreover, we extend our examination to the potential of pre-training in graph-level anomaly detection. We envision this work to stimulate a re-evaluation of pre-training's role in GAD and offer valuable insights for future research.

Title: Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics

• Authors: Jinghao Hu, Yuhe Zhang, GuoHua Geng, Liuyuxin Yang, JiaRui Yan, Jingtao Cheng, YaDong Zhang, Kang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18537
• Pdf URL: https://arxiv.org/pdf/2410.18537
• Copy Paste: [[2410.18537]] Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics(https://arxiv.org/abs/2410.18537)
• Keywords: diffusion
• Abstract: Traditionally, style has been primarily considered in terms of artistic elements such as colors, brushstrokes, and lighting. However, identical semantic subjects, like people, boats, and houses, can vary significantly across different artistic traditions, indicating that style also encompasses the underlying semantics. Therefore, in this study, we propose a zero-shot scheme for image variation with coordinated semantics. Specifically, our scheme transforms the image-to-image problem into an image-to-text-to-image problem. The image-to-text operation employs vision-language models e.g., BLIP) to generate text describing the content of the input image, including the objects and their positions. Subsequently, the input style keyword is elaborated into a detailed description of this style and then merged with the content text using the reasoning capabilities of ChatGPT. Finally, the text-to-image operation utilizes a Diffusion model to generate images based on the text prompt. To enable the Diffusion model to accommodate more styles, we propose a fine-tuning strategy that injects text and style constraints into cross-attention. This ensures that the output image exhibits similar semantics in the desired style. To validate the performance of the proposed scheme, we constructed a benchmark comprising images of various styles and scenes and introduced two novel metrics. Despite its simplicity, our scheme yields highly plausible results in a zero-shot manner, particularly for generating stylized images with high-fidelity semantics.

Title: SMITE: Segment Me In TimE

• Authors: Amirhossein Alimohammadi, Sauradip Nag, Saeid Asgari Taghanaki, Andrea Tagliasacchi, Ghassan Hamarneh, Ali Mahdavi Amiri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18538
• Pdf URL: https://arxiv.org/pdf/2410.18538
• Copy Paste: [[2410.18538]] SMITE: Segment Me In TimE(https://arxiv.org/abs/2410.18538)
• Keywords: diffusion
• Abstract: Segmenting an object in a video presents significant challenges. Each pixel must be accurately labelled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or a few sample images. In this paper, we address this issue by employing a pre-trained text to image diffusion model supplemented with an additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of-the-art alternatives.

Title: Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation

• Authors: Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, Remigiusz Kinas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18565
• Pdf URL: https://arxiv.org/pdf/2410.18565
• Copy Paste: [[2410.18565]] Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation(https://arxiv.org/abs/2410.18565)
• Keywords: generative
• Abstract: We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish language processing. Trained on curated Polish corpora, this model addresses key challenges in language model development through innovative techniques. These include Weighted Instruction Cross-Entropy Loss, which balances the learning of different instruction types, and Adaptive Learning Rate, which dynamically adjusts the learning rate based on training progress. To evaluate performance, we created the Open PL LLM Leaderboard and Polish MT-Bench, novel frameworks assessing various NLP tasks and conversational abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9 percentage point increase in average score compared to Mistral-7B-v0.1 on the RAG Reader task. It also excels in the Polish MT-Bench, particularly in Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model represents a substantial advancement in Polish language AI, offering a powerful tool for diverse linguistic applications and setting new benchmarks in the field.

Title: Taipan: Efficient and Expressive State Space Language Models with Selective Attention

• Authors: Chien Van Nguyen, Huy Huu Nguyen, Thang M. Pham, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Ryan A. Rossi, Trung Bui, Viet Dac Lai, Franck Dernoncourt, Thien Huu Nguyen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18572
• Pdf URL: https://arxiv.org/pdf/2410.18572
• Copy Paste: [[2410.18572]] Taipan: Efficient and Expressive State Space Language Models with Selective Attention(https://arxiv.org/abs/2410.18572)
• Keywords: in-context
• Abstract: Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long sequences due to quadratic computational complexity in training and linearly scaling memory costs during inference. Recent State Space Models (SSMs) such as Mamba offer alternatives with constant memory usage, but they underperform in tasks requiring extensive in-context retrieval. We introduce Taipan, a novel hybrid architecture that combines Mamba-2 with Selective Attention Layers (SALs). These SALs identify tokens requiring long-range interactions, remove less important features, and then augment their representations using the attention module. This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. By constraining the attention budget, Taipan extends accurate predictions to context lengths of up to 1 million tokens while preserving computational efficiency. Our experiments demonstrate Taipan's superior performance across various scales and tasks, offering a promising solution for efficient long-context language modeling.

Title: Understanding Players as if They Are Talking to the Game in a Customized Language: A Pilot Study

• Authors: Tianze Wang, Maryam Honari-Jahromi, Styliani Katsarou, Olga Mikheeva, Theodoros Panagiotakopoulos, Oleg Smirnov, Lele Cao, Sahar Asadi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18605
• Pdf URL: https://arxiv.org/pdf/2410.18605
• Copy Paste: [[2410.18605]] Understanding Players as if They Are Talking to the Game in a Customized Language: A Pilot Study(https://arxiv.org/abs/2410.18605)
• Keywords: self-supervised
• Abstract: This pilot study explores the application of language models (LMs) to model game event sequences, treating them as a customized natural language. We investigate a popular mobile game, transforming raw event data into textual sequences and pretraining a Longformer model on this data. Our approach captures the rich and nuanced interactions within game sessions, effectively identifying meaningful player segments. The results demonstrate the potential of self-supervised LMs in enhancing game design and personalization without relying on ground-truth labels.

Title: Environment Maps Editing using Inverse Rendering and Adversarial Implicit Functions

• Authors: Antonio D'Orazio, Davide Sforza, Fabio Pellacini, Iacopo Masi
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.18622
• Pdf URL: https://arxiv.org/pdf/2410.18622
• Copy Paste: [[2410.18622]] Environment Maps Editing using Inverse Rendering and Adversarial Implicit Functions(https://arxiv.org/abs/2410.18622)
• Keywords: generative
• Abstract: Editing High Dynamic Range (HDR) environment maps using an inverse differentiable rendering architecture is a complex inverse problem due to the sparsity of relevant pixels and the challenges in balancing light sources and background. The pixels illuminating the objects are a small fraction of the total image, leading to noise and convergence issues when the optimization directly involves pixel values. HDR images, with pixel values beyond the typical Standard Dynamic Range (SDR), pose additional challenges. Higher learning rates corrupt the background during optimization, while lower learning rates fail to manipulate light sources. Our work introduces a novel method for editing HDR environment maps using a differentiable rendering, addressing sparsity and variance between values. Instead of introducing strong priors that extract the relevant HDR pixels and separate the light sources, or using tricks such as optimizing the HDR image in the log space, we propose to model the optimized environment map with a new variant of implicit neural representations able to handle HDR images. The neural representation is trained with adversarial perturbations over the weights to ensure smooth changes in the output when it receives gradients from the inverse rendering. In this way, we obtain novel and cheap environment maps without relying on latent spaces of expensive generative models, maintaining the original visual consistency. Experimental results demonstrate the method's effectiveness in reconstructing the desired lighting effects while preserving the fidelity of the map and reflections on objects in the scene. Our approach can pave the way to interesting tasks, such as estimating a new environment map given a rendering with novel light sources, maintaining the initial perceptual features, and enabling brush stroke-based editing of existing environment maps.

Title: Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Model

• Authors: Jinxu Lin, Linwei Tao, Minjing Dong, Chang Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18639
• Pdf URL: https://arxiv.org/pdf/2410.18639
• Copy Paste: [[2410.18639]] Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Model(https://arxiv.org/abs/2410.18639)
• Keywords: diffusion, generative
• Abstract: As diffusion models become increasingly popular, the misuse of copyrighted and private images has emerged as a major concern. One promising solution to mitigate this issue is identifying the contribution of specific training samples in generative models, a process known as data attribution. Existing data attribution methods for diffusion models typically quantify the contribution of a training sample by evaluating the change in diffusion loss when the sample is included or excluded from the training process. However, we argue that the direct usage of diffusion loss cannot represent such a contribution accurately due to the calculation of diffusion loss. Specifically, these approaches measure the divergence between predicted and ground truth distributions, which leads to an indirect comparison between the predicted distributions and cannot represent the variances between model behaviors. To address these issues, we aim to measure the direct comparison between predicted distributions with an attribution score to analyse the training sample importance, which is achieved by Diffusion Attribution Score (DAS). Underpinned by rigorous theoretical analysis, we elucidate the effectiveness of DAS. Additionally, we explore strategies to accelerate DAS calculations, facilitating its application to large-scale diffusion models. Our extensive experiments across various datasets and diffusion models demonstrate that DAS significantly surpasses previous benchmarks in terms of the linear data-modelling score, establishing new state-of-the-art performance.

Title: DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation

• Authors: Yuang Ai, Xiaoqiang Zhou, Huaibo Huang, Xiaotian Han, Zhengyu Chen, Quanzeng You, Hongxia Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18666
• Pdf URL: https://arxiv.org/pdf/2410.18666
• Copy Paste: [[2410.18666]] DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation(https://arxiv.org/abs/2410.18666)
• Keywords: diffusion, generative
• Abstract: Image restoration (IR) in real-world scenarios presents significant challenges due to the lack of high-capacity models and comprehensive datasets. To tackle these issues, we present a dual strategy: GenIR, an innovative data curation pipeline, and DreamClear, a cutting-edge Diffusion Transformer (DiT)-based image restoration model. GenIR, our pioneering contribution, is a dual-prompt learning pipeline that overcomes the limitations of existing datasets, which typically comprise only a few thousand images and thus offer limited generalizability for larger models. GenIR streamlines the process into three stages: image-text pair construction, dual-prompt based fine-tuning, and data generation & filtering. This approach circumvents the laborious data crawling process, ensuring copyright compliance and providing a cost-effective, privacy-safe solution for IR dataset construction. The result is a large-scale dataset of one million high-quality images. Our second contribution, DreamClear, is a DiT-based image restoration model. It utilizes the generative priors of text-to-image (T2I) diffusion models and the robust perceptual capabilities of multi-modal large language models (MLLMs) to achieve photorealistic restoration. To boost the model's adaptability to diverse real-world degradations, we introduce the Mixture of Adaptive Modulator (MoAM). It employs token-wise degradation priors to dynamically integrate various restoration experts, thereby expanding the range of degradations the model can address. Our exhaustive experiments confirm DreamClear's superior performance, underlining the efficacy of our dual strategy for real-world image restoration. Code and pre-trained models will be available at: this https URL.

Title: Ali-AUG: Innovative Approaches to Labeled Data Augmentation using One-Step Diffusion Model

• Authors: Ali Hamza, Aizea Lojo, Adrian Núñez-Marcos, Aitziber Atutxa
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18678
• Pdf URL: https://arxiv.org/pdf/2410.18678
• Copy Paste: [[2410.18678]] Ali-AUG: Innovative Approaches to Labeled Data Augmentation using One-Step Diffusion Model(https://arxiv.org/abs/2410.18678)
• Keywords: diffusion
• Abstract: This paper introduces Ali-AUG, a novel single-step diffusion model for efficient labeled data augmentation in industrial applications. Our method addresses the challenge of limited labeled data by generating synthetic, labeled images with precise feature insertion. Ali-AUG utilizes a stable diffusion architecture enhanced with skip connections and LoRA modules to efficiently integrate masks and images, ensuring accurate feature placement without affecting unrelated image content. Experimental validation across various industrial datasets demonstrates Ali-AUG's superiority in generating high-quality, defect-enhanced images while maintaining rapid single-step inference. By offering precise control over feature insertion and minimizing required training steps, our technique significantly enhances data augmentation capabilities, providing a powerful tool for improving the performance of deep learning models in scenarios with limited labeled data. Ali-AUG is especially useful for use cases like defective product image generation to train AI-based models to improve their ability to detect defects in manufacturing processes. Using different data preparation strategies, including Classification Accuracy Score (CAS) and Naive Augmentation Score (NAS), we show that Ali-AUG improves model performance by 31% compared to other augmentation methods and by 45% compared to models without data augmentation. Notably, Ali-AUG reduces training time by 32% and supports both paired and unpaired datasets, enhancing flexibility in data preparation.

Title: Rigid Single-Slice-in-Volume registration via rotation-equivariant 2D/3D feature matching

• Authors: Stefan Brandstätter, Philipp Seeböck, Christoph Fürböck, Svitlana Pochepnia, Helmut Prosch, Georg Langs
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18683
• Pdf URL: https://arxiv.org/pdf/2410.18683
• Copy Paste: [[2410.18683]] Rigid Single-Slice-in-Volume registration via rotation-equivariant 2D/3D feature matching(https://arxiv.org/abs/2410.18683)
• Keywords: self-supervised
• Abstract: 2D to 3D registration is essential in tasks such as diagnosis, surgical navigation, environmental understanding, navigation in robotics, autonomous systems, or augmented reality. In medical imaging, the aim is often to place a 2D image in a 3D volumetric observation to w. Current approaches for rigid single slice in volume registration are limited by requirements such as pose initialization, stacks of adjacent slices, or reliable anatomical landmarks. Here, we propose a self-supervised 2D/3D registration approach to match a single 2D slice to the corresponding 3D volume. The method works in data without anatomical priors such as images of tumors. It addresses the dimensionality disparity and establishes correspondences between 2D in-plane and 3D out-of-plane rotation-equivariant features by using group equivariant CNNs. These rotation-equivariant features are extracted from the 2D query slice and aligned with their 3D counterparts. Results demonstrate the robustness of the proposed slice-in-volume registration on the NSCLC-Radiomics CT and KIRBY21 MRI datasets, attaining an absolute median angle error of less than 2 degrees and a mean-matching feature accuracy of 89% at a tolerance of 3 pixels.

Title: GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning

• Authors: Rita Ramos, Everlyn Asiko Chimoto, Maartje ter Hoeve, Natalie Schluter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18702
• Pdf URL: https://arxiv.org/pdf/2410.18702
• Copy Paste: [[2410.18702]] GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning(https://arxiv.org/abs/2410.18702)
• Keywords: in-context
• Abstract: We introduce GrammaMT, a grammatically-aware prompting approach for machine translation that uses Interlinear Glossed Text (IGT), a common form of linguistic description providing morphological and lexical annotations for source sentences. GrammaMT proposes three prompting strategies: gloss-shot, chain-gloss and model-gloss. All are training-free, requiring only a few examples that involve minimal effort to collect, and making them well-suited for low-resource setups. Experiments show that GrammaMT enhances translation performance on open-source instruction-tuned LLMs for various low- to high-resource languages across three benchmarks: (1) the largest IGT corpus, (2) the challenging 2023 SIGMORPHON Shared Task data over endangered languages, and (3) even in an out-of-domain setting with FLORES. Moreover, ablation studies reveal that leveraging gloss resources could substantially boost MT performance (by over 17 BLEU points) if LLMs accurately generate or access input sentence glosses.

Title: Exploiting Interpretable Capabilities with Concept-Enhanced Diffusion and Prototype Networks

• Authors: Alba Carballo-Castro, Sonia Laguna, Moritz Vandenhirtz, Julia E. Vogt
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18705
• Pdf URL: https://arxiv.org/pdf/2410.18705
• Copy Paste: [[2410.18705]] Exploiting Interpretable Capabilities with Concept-Enhanced Diffusion and Prototype Networks(https://arxiv.org/abs/2410.18705)
• Keywords: diffusion
• Abstract: Concept-based machine learning methods have increasingly gained importance due to the growing interest in making neural networks interpretable. However, concept annotations are generally challenging to obtain, making it crucial to leverage all their prior knowledge. By creating concept-enriched models that incorporate concept information into existing architectures, we exploit their interpretable capabilities to the fullest extent. In particular, we propose Concept-Guided Conditional Diffusion, which can generate visual representations of concepts, and Concept-Guided Prototype Networks, which can create a concept prototype dataset and leverage it to perform interpretable concept prediction. These results open up new lines of research by exploiting pre-existing information in the quest for rendering machine learning more human-understandable.

Title: Retrieval-Augmented Diffusion Models for Time Series Forecasting

• Authors: Jingwei Liu, Ling Yang, Hongyan Li, Shenda Hong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18712
• Pdf URL: https://arxiv.org/pdf/2410.18712
• Copy Paste: [[2410.18712]] Retrieval-Augmented Diffusion Models for Time Series Forecasting(https://arxiv.org/abs/2410.18712)
• Keywords: diffusion
• Abstract: While time series diffusion models have received considerable focus from many recent works, the performance of existing models remains highly unstable. Factors limiting time series diffusion models include insufficient time series datasets and the absence of guidance. To address these limitations, we propose a Retrieval- Augmented Time series Diffusion model (RATD). The framework of RATD consists of two parts: an embedding-based retrieval process and a reference-guided diffusion model. In the first part, RATD retrieves the time series that are most relevant to historical time series from the database as references. The references are utilized to guide the denoising process in the second part. Our approach allows leveraging meaningful samples within the database to aid in sampling, thus maximizing the utilization of datasets. Meanwhile, this reference-guided mechanism also compensates for the deficiencies of existing time series diffusion models in terms of guidance. Experiments and visualizations on multiple datasets demonstrate the effectiveness of our approach, particularly in complicated prediction tasks.

Title: ChatSearch: a Dataset and a Generative Retrieval Model for General Conversational Image Retrieval

• Authors: Zijia Zhao, Longteng Guo, Tongtian Yue, Erdong Hu, Shuai Shao, Zehuan Yuan, Hua Huang, Jing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18715
• Pdf URL: https://arxiv.org/pdf/2410.18715
• Copy Paste: [[2410.18715]] ChatSearch: a Dataset and a Generative Retrieval Model for General Conversational Image Retrieval(https://arxiv.org/abs/2410.18715)
• Keywords: generative
• Abstract: In this paper, we investigate the task of general conversational image retrieval on open-domain images. The objective is to search for images based on interactive conversations between humans and computers. To advance this task, we curate a dataset called ChatSearch. This dataset includes a multi-round multimodal conversational context query for each target image, thereby requiring the retrieval system to find the accurate image from database. Simultaneously, we propose a generative retrieval model named ChatSearcher, which is trained end-to-end to accept/produce interleaved image-text inputs/outputs. ChatSearcher exhibits strong capability in reasoning with multimodal context and can leverage world knowledge to yield visual retrieval results. It demonstrates superior performance on the ChatSearch dataset and also achieves competitive results on other image retrieval tasks and visual conversation tasks. We anticipate that this work will inspire further research on interactive multimodal retrieval systems. Our dataset will be available at this https URL.

Title: Low-Latency Video Anonymization for Crowd Anomaly Detection: Privacy vs. Performance

• Authors: Mulugeta Weldezgina Asres, Lei Jiao, Christian Walter Omlin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18717
• Pdf URL: https://arxiv.org/pdf/2410.18717
• Copy Paste: [[2410.18717]] Low-Latency Video Anonymization for Crowd Anomaly Detection: Privacy vs. Performance(https://arxiv.org/abs/2410.18717)
• Keywords: anomaly
• Abstract: Recent advancements in artificial intelligence promise ample potential in monitoring applications with surveillance cameras. However, concerns about privacy and model bias have made it challenging to utilize them in public. Although de-identification approaches have been proposed in the literature, aiming to achieve a certain level of anonymization, most of them employ deep learning models that are computationally demanding for real-time edge deployment. In this study, we revisit conventional anonymization solutions for privacy protection and real-time video anomaly detection (VAD) applications. We propose a novel lightweight adaptive anonymization for VAD (LA3D) that employs dynamic adjustment to enhance privacy protection. We evaluated the approaches on publicly available privacy and VAD data sets to examine the strengths and weaknesses of the different anonymization techniques and highlight the promising efficacy of our approach. Our experiment demonstrates that LA3D enables substantial improvement in the privacy anonymization capability without majorly degrading VAD efficacy.

Title: Rectified Diffusion Guidance for Conditional Generation

• Authors: Mengfei Xia, Nan Xue, Yujun Shen, Ran Yi, Tieliang Gong, Yong-Jin Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18737
• Pdf URL: https://arxiv.org/pdf/2410.18737
• Copy Paste: [[2410.18737]] Rectified Diffusion Guidance for Conditional Generation(https://arxiv.org/abs/2410.18737)
• Keywords: diffusion, generative
• Abstract: Classifier-Free Guidance (CFG), which combines the conditional and unconditional score functions with two coefficients summing to one, serves as a practical technique for diffusion model sampling. Theoretically, however, denoising with CFG cannot be expressed as a reciprocal diffusion process, which may consequently leave some hidden risks during use. In this work, we revisit the theory behind CFG and rigorously confirm that the improper configuration of the combination coefficients (i.e., the widely used summing-to-one version) brings about expectation shift of the generative distribution. To rectify this issue, we propose ReCFG with a relaxation on the guidance coefficients such that denoising with ReCFG strictly aligns with the diffusion theory. We further show that our approach enjoys a closed-form solution given the guidance strength. That way, the rectified coefficients can be readily pre-computed via traversing the observed data, leaving the sampling speed barely affected. Empirical evidence on real-world data demonstrate the compatibility of our post-hoc design with existing state-of-the-art diffusion models, including both class-conditioned ones (e.g., EDM2 on ImageNet) and text-conditioned ones (e.g., SD3 on CC12M), without any retraining. We will open-source the code to facilitate further research.

Title: Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing

• Authors: Haonan Lin, Mengmeng Wang, Jiahao Wang, Wenbin An, Yan Chen, Yong Liu, Feng Tian, Guang Dai, Jingdong Wang, Qianying Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18756
• Pdf URL: https://arxiv.org/pdf/2410.18756
• Copy Paste: [[2410.18756]] Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing(https://arxiv.org/abs/2410.18756)
• Keywords: diffusion
• Abstract: Text-guided diffusion models have significantly advanced image editing, enabling high-quality and diverse modifications driven by text prompts. However, effective editing requires inverting the source image into a latent space, a process often hindered by prediction errors inherent in DDIM inversion. These errors accumulate during the diffusion process, resulting in inferior content preservation and edit fidelity, especially with conditional inputs. We address these challenges by investigating the primary contributors to error accumulation in DDIM inversion and identify the singularity problem in traditional noise schedules as a key issue. To resolve this, we introduce the Logistic Schedule, a novel noise schedule designed to eliminate singularities, improve inversion stability, and provide a better noise space for image editing. This schedule reduces noise prediction errors, enabling more faithful editing that preserves the original content of the source image. Our approach requires no additional retraining and is compatible with various existing editing methods. Experiments across eight editing tasks demonstrate the Logistic Schedule's superior performance in content preservation and edit fidelity compared to traditional noise schedules, highlighting its adaptability and effectiveness.

Title: Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances

• Authors: Shilin Lu, Zihan Zhou, Jiayou Lu, Yuanzhi Zhu, Adams Wai-Kin Kong
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18775
• Pdf URL: https://arxiv.org/pdf/2410.18775
• Copy Paste: [[2410.18775]] Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances(https://arxiv.org/abs/2410.18775)
• Keywords: diffusion, generative
• Abstract: Current image watermarking methods are vulnerable to advanced image editing techniques enabled by large-scale text-to-image models. These models can distort embedded watermarks during editing, posing significant challenges to copyright protection. In this work, we introduce W-Bench, the first comprehensive benchmark designed to evaluate the robustness of watermarking methods against a wide range of image editing techniques, including image regeneration, global editing, local editing, and image-to-video generation. Through extensive evaluations of eleven representative watermarking methods against prevalent editing techniques, we demonstrate that most methods fail to detect watermarks after such edits. To address this limitation, we propose VINE, a watermarking method that significantly enhances robustness against various image editing techniques while maintaining high image quality. Our approach involves two key innovations: (1) we analyze the frequency characteristics of image editing and identify that blurring distortions exhibit similar frequency properties, which allows us to use them as surrogate attacks during training to bolster watermark robustness; (2) we leverage a large-scale pretrained diffusion model SDXL-Turbo, adapting it for the watermarking task to achieve more imperceptible and robust watermark embedding. Experimental results show that our method achieves outstanding watermarking performance under various image editing techniques, outperforming existing methods in both image quality and robustness. Code is available at this https URL.

Title: A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs

• Authors: Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh, Ayan Chakrabarti, Wittawat Jitkrittum, Vladimir Feinberg, Seungyeon Kim, Hrayr Harutyunyan, Nikunj Saunshi, Zachary Nado, Rakesh Shivanna, Sashank J. Reddi, Aditya Krishna Menon, Rohan Anil, Sanjiv Kumar
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18779
• Pdf URL: https://arxiv.org/pdf/2410.18779
• Copy Paste: [[2410.18779]] A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs(https://arxiv.org/abs/2410.18779)
• Keywords: self-supervised
• Abstract: A primary challenge in large language model (LLM) development is their onerous pre-training cost. Typically, such pre-training involves optimizing a self-supervised objective (such as next-token prediction) over a large corpus. This paper explores a promising paradigm to improve LLM pre-training efficiency and quality by suitably leveraging a small language model (SLM). In particular, this paradigm relies on an SLM to both (1) provide soft labels as additional training supervision, and (2) select a small subset of valuable ("informative" and "hard") training examples. Put together, this enables an effective transfer of the SLM's predictive distribution to the LLM, while prioritizing specific regions of the training data distribution. Empirically, this leads to reduced LLM training time compared to standard training, while improving the overall quality. Theoretically, we develop a statistical framework to systematically study the utility of SLMs in enabling efficient training of high-quality LLMs. In particular, our framework characterizes how the SLM's seemingly low-quality supervision can enhance the training of a much more capable LLM. Furthermore, it also highlights the need for an adaptive utilization of such supervision, by striking a balance between the bias and variance introduced by the SLM-provided soft labels. We corroborate our theoretical framework by improving the pre-training of an LLM with 2.8B parameters by utilizing a smaller LM with 1.5B parameters on the Pile dataset.

Title: Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality

• Authors: Zhihan Huang, Yuting Wei, Yuxin Chen
• Subjects: cs.LG, eess.SP, math.NA, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.18784
• Pdf URL: https://arxiv.org/pdf/2410.18784
• Copy Paste: [[2410.18784]] Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality(https://arxiv.org/abs/2410.18784)
• Keywords: diffusion, generative
• Abstract: The denoising diffusion probabilistic model (DDPM) has emerged as a mainstream generative model in generative AI. While sharp convergence guarantees have been established for the DDPM, the iteration complexity is, in general, proportional to the ambient data dimension, resulting in overly conservative theory that fails to explain its practical efficiency. This has motivated the recent work Li and Yan (2024a) to investigate how the DDPM can achieve sampling speed-ups through automatic exploitation of intrinsic low dimensionality of data. We strengthen this prior work by demonstrating, in some sense, optimal adaptivity to unknown low dimensionality. For a broad class of data distributions with intrinsic dimension $k$, we prove that the iteration complexity of the DDPM scales nearly linearly with $k$, which is optimal when using KL divergence to measure distributional discrepancy. Our theory is established based on a key observation: the DDPM update rule is equivalent to running a suitably parameterized SDE upon discretization, where the nonlinear component of the drift term is intrinsically low-dimensional.

Title: Fast constrained sampling in pre-trained diffusion models

• Authors: Alexandros Graikos, Nebojsa Jojic, Dimitris Samaras
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18804
• Pdf URL: https://arxiv.org/pdf/2410.18804
• Copy Paste: [[2410.18804]] Fast constrained sampling in pre-trained diffusion models(https://arxiv.org/abs/2410.18804)
• Keywords: diffusion, generative
• Abstract: Diffusion models have dominated the field of large, generative image models, with the prime examples of Stable Diffusion and DALL-E 3 being widely adopted. These models have been trained to perform text-conditioned generation on vast numbers of image-caption pairs and as a byproduct, have acquired general knowledge about natural image statistics. However, when confronted with the task of constrained sampling, e.g. generating the right half of an image conditioned on the known left half, applying these models is a delicate and slow process, with previously proposed algorithms relying on expensive iterative operations that are usually orders of magnitude slower than text-based inference. This is counter-intuitive, as image-conditioned generation should rely less on the difficult-to-learn semantic knowledge that links captions and imagery, and should instead be achievable by lower-level correlations among image pixels. In practice, inverse models are trained or tuned separately for each inverse problem, e.g. by providing parts of images during training as an additional condition, to allow their application in realistic settings. However, we argue that this is not necessary and propose an algorithm for fast-constrained sampling in large pre-trained diffusion models (Stable Diffusion) that requires no expensive backpropagation operations through the model and produces results comparable even to the state-of-the-art \emph{tuned} models. Our method is based on a novel optimization perspective to sampling under constraints and employs a numerical approximation to the expensive gradients, previously computed using backpropagation, incurring significant speed-ups.

Title: Multi-Scale Diffusion: Enhancing Spatial Layout in High-Resolution Panoramic Image Generation

• Authors: Xiaoyu Zhang, Teng Zhou, Xinlong Zhang, Jia Wei, Yongchuan Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18830
• Pdf URL: https://arxiv.org/pdf/2410.18830
• Copy Paste: [[2410.18830]] Multi-Scale Diffusion: Enhancing Spatial Layout in High-Resolution Panoramic Image Generation(https://arxiv.org/abs/2410.18830)
• Keywords: diffusion
• Abstract: Diffusion models have recently gained recognition for generating diverse and high-quality content, especially in the domain of image synthesis. These models excel not only in creating fixed-size images but also in producing panoramic images. However, existing methods often struggle with spatial layout consistency when producing high-resolution panoramas, due to the lack of guidance of the global image layout. In this paper, we introduce the Multi-Scale Diffusion (MSD) framework, a plug-and-play module that extends the existing panoramic image generation framework to multiple resolution levels. By utilizing gradient descent techniques, our method effectively incorporates structural information from low-resolution images into high-resolution outputs. A comprehensive evaluation of the proposed method was conducted, comparing it with the prior works in qualitative and quantitative dimensions. The evaluation results demonstrate that our method significantly outperforms others in generating coherent high-resolution panoramas.

Title: From Efficiency to Equity: Measuring Fairness in Preference Learning

• Authors: Shreeyash Gowaikar, Hugo Berard, Rashid Mushkani, Shin Koseki
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18841
• Pdf URL: https://arxiv.org/pdf/2410.18841
• Copy Paste: [[2410.18841]] From Efficiency to Equity: Measuring Fairness in Preference Learning(https://arxiv.org/abs/2410.18841)
• Keywords: generative
• Abstract: As AI systems, particularly generative models, increasingly influence decision-making, ensuring that they are able to fairly represent diverse human preferences becomes crucial. This paper introduces a novel framework for evaluating epistemic fairness in preference learning models inspired by economic theories of inequality and Rawlsian justice. We propose metrics adapted from the Gini Coefficient, Atkinson Index, and Kuznets Ratio to quantify fairness in these models. We validate our approach using two datasets: a custom visual preference dataset (AI-EDI-Space) and the Jester Jokes dataset. Our analysis reveals variations in model performance across users, highlighting potential epistemic injustices. We explore pre-processing and in-processing techniques to mitigate these inequalities, demonstrating a complex relationship between model efficiency and fairness. This work contributes to AI ethics by providing a framework for evaluating and improving epistemic fairness in preference learning models, offering insights for developing more inclusive AI systems in contexts where diverse human preferences are crucial.

Title: Provably Robust Watermarks for Open-Source Language Models

• Authors: Miranda Christ, Sam Gunn, Tal Malkin, Mariana Raykova
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18861
• Pdf URL: https://arxiv.org/pdf/2410.18861
• Copy Paste: [[2410.18861]] Provably Robust Watermarks for Open-Source Language Models(https://arxiv.org/abs/2410.18861)
• Keywords: generative
• Abstract: The recent explosion of high-quality language models has necessitated new methods for identifying AI-generated text. Watermarking is a leading solution and could prove to be an essential tool in the age of generative AI. Existing approaches embed watermarks at inference and crucially rely on the large language model (LLM) specification and parameters being secret, which makes them inapplicable to the open-source setting. In this work, we introduce the first watermarking scheme for open-source LLMs. Our scheme works by modifying the parameters of the model, but the watermark can be detected from just the outputs of the model. Perhaps surprisingly, we prove that our watermarks are unremovable under certain assumptions about the adversary's knowledge. To demonstrate the behavior of our construction under concrete parameter instantiations, we present experimental results with OPT-6.7B and OPT-1.3B. We demonstrate robustness to both token substitution and perturbation of the model parameters. We find that the stronger of these attacks, the model-perturbation attack, requires deteriorating the quality score to 0 out of 100 in order to bring the detection rate down to 50%.

Title: Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences

• Authors: Weijian Luo
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18881
• Pdf URL: https://arxiv.org/pdf/2410.18881
• Copy Paste: [[2410.18881]] Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences(https://arxiv.org/abs/2410.18881)
• Keywords: diffusion
• Abstract: One-step text-to-image generator models offer advantages such as swift inference efficiency, flexible architectures, and state-of-the-art generation performance. In this paper, we study the problem of aligning one-step generator models with human preferences for the first time. Inspired by the success of reinforcement learning using human feedback (RLHF), we formulate the alignment problem as maximizing expected human reward functions while adding an Integral Kullback-Leibler divergence term to prevent the generator from diverging. By overcoming technical challenges, we introduce Diff-Instruct++ (DI++), the first, fast-converging and image data-free human preference alignment method for one-step text-to-image generators. We also introduce novel theoretical insights, showing that using CFG for diffusion distillation is secretly doing RLHF with DI++. Such an interesting finding brings understanding and potential contributions to future research involving CFG. In the experiment sections, we align both UNet-based and DiT-based one-step generators using DI++, which use the Stable Diffusion 1.5 and the PixelArt-$\alpha$ as the reference diffusion processes. The resulting DiT-based one-step text-to-image model achieves a strong Aesthetic Score of 6.19 and an Image Reward of 1.24 on the COCO validation prompt dataset. It also achieves a leading Human preference Score (HPSv2.0) of 28.48, outperforming other open-sourced models such as Stable Diffusion XL, DMD2, SD-Turbo, as well as PixelArt-$\alpha$. Both theoretical contributions and empirical evidence indicate that DI++ is a strong human-preference alignment approach for one-step text-to-image models.

Title: Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code

• Authors: Jipeng Zhang, Jianshu Zhang, Yuanzhe Li, Renjie Pi, Rui Pan, Runtao Liu, Ziqiang Zheng, Tong Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18957
• Pdf URL: https://arxiv.org/pdf/2410.18957
• Copy Paste: [[2410.18957]] Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code(https://arxiv.org/abs/2410.18957)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) demonstrate strong proficiency in generating code for high-resource programming languages (HRPLs) like Python but struggle significantly with low-resource programming languages (LRPLs) such as Racket or D. This performance gap deepens the digital divide, preventing developers using LRPLs from benefiting equally from LLM advancements and reinforcing disparities in innovation within underrepresented programming communities. While generating additional training data for LRPLs is promising, it faces two key challenges: manual annotation is labor-intensive and costly, and LLM-generated LRPL code is often of subpar quality. The underlying cause of this issue is the gap between natural language to programming language gap (NL-PL Gap), which is especially pronounced in LRPLs due to limited aligned data. In this work, we introduce a novel approach called Bridge-Coder, which leverages LLMs' intrinsic capabilities to enhance the performance on LRPLs. Our method consists of two key stages. Bridge Generation, where we create high-quality dataset by utilizing LLMs' general knowledge understanding, proficiency in HRPLs, and in-context learning abilities. Then, we apply the Bridged Alignment, which progressively improves the alignment between NL instructions and LRPLs. Experimental results across multiple LRPLs show that Bridge-Coder significantly enhances model performance, demonstrating the effectiveness and generalization of our approach. Furthermore, we offer a detailed analysis of the key components of our method, providing valuable insights for future work aimed at addressing the challenges associated with LRPLs.

Title: Stable Consistency Tuning: Understanding and Improving Consistency Models

• Authors: Fu-Yun Wang, Zhengyang Geng, Hongsheng Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18958
• Pdf URL: https://arxiv.org/pdf/2410.18958
• Copy Paste: [[2410.18958]] Stable Consistency Tuning: Understanding and Improving Consistency Models(https://arxiv.org/abs/2410.18958)
• Keywords: diffusion, generative
• Abstract: Diffusion models achieve superior generation quality but suffer from slow generation speed due to the iterative nature of denoising. In contrast, consistency models, a new generative family, achieve competitive performance with significantly faster sampling. These models are trained either through consistency distillation, which leverages pretrained diffusion models, or consistency training/tuning directly from raw data. In this work, we propose a novel framework for understanding consistency models by modeling the denoising process of the diffusion model as a Markov Decision Process (MDP) and framing consistency model training as the value estimation through Temporal Difference~(TD) Learning. More importantly, this framework allows us to analyze the limitations of current consistency training/tuning strategies. Built upon Easy Consistency Tuning (ECT), we propose Stable Consistency Tuning (SCT), which incorporates variance-reduced learning using the score identity. SCT leads to significant performance improvements on benchmarks such as CIFAR-10 and ImageNet-64. On ImageNet-64, SCT achieves 1-step FID 2.42 and 2-step FID 1.55, a new SoTA for consistency models.

Title: Context is Key: A Benchmark for Forecasting with Essential Textual Information

• Authors: Andrew Robert Williams, Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requeima, Alexandre Lacoste, Irina Rish, Nicolas Chapados, Alexandre Drouin
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.18959
• Pdf URL: https://arxiv.org/pdf/2410.18959
• Copy Paste: [[2410.18959]] Context is Key: A Benchmark for Forecasting with Essential Textual Information(https://arxiv.org/abs/2410.18959)
• Keywords: foundation model
• Abstract: Forecasting is a critical task in decision making across various domains. While numerical data provides a foundation, it often lacks crucial context necessary for accurate predictions. Human forecasters frequently rely on additional information, such as background knowledge or constraints, which can be efficiently communicated through natural language. However, the ability of existing forecasting models to effectively integrate this textual information remains an open question. To address this, we introduce "Context is Key" (CiK), a time series forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context, requiring models to integrate both modalities. We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters, and propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark. Our experiments highlight the importance of incorporating contextual information, demonstrate surprising performance when using LLM-based forecasting models, and also reveal some of their critical shortcomings. By presenting this benchmark, we aim to advance multimodal forecasting, promoting models that are both accurate and accessible to decision-makers with varied technical expertise. The benchmark can be visualized at this https URL .

Title: Where Am I and What Will I See: An Auto-Regressive Model for Spatial Localization and View Prediction

• Authors: Junyi Chen, Di Huang, Weicai Ye, Wanli Ouyang, Tong He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18962
• Pdf URL: https://arxiv.org/pdf/2410.18962
• Copy Paste: [[2410.18962]] Where Am I and What Will I See: An Auto-Regressive Model for Spatial Localization and View Prediction(https://arxiv.org/abs/2410.18962)
• Keywords: generative
• Abstract: Spatial intelligence is the ability of a machine to perceive, reason, and act in three dimensions within space and time. Recent advancements in large-scale auto-regressive models have demonstrated remarkable capabilities across various reasoning tasks. However, these models often struggle with fundamental aspects of spatial reasoning, particularly in answering questions like "Where am I?" and "What will I see?". While some attempts have been done, existing approaches typically treat them as separate tasks, failing to capture their interconnected nature. In this paper, we present Generative Spatial Transformer (GST), a novel auto-regressive framework that jointly addresses spatial localization and view prediction. Our model simultaneously estimates the camera pose from a single image and predicts the view from a new camera pose, effectively bridging the gap between spatial awareness and visual prediction. The proposed innovative camera tokenization method enables the model to learn the joint distribution of 2D projections and their corresponding spatial perspectives in an auto-regressive manner. This unified training paradigm demonstrates that joint optimization of pose estimation and novel view synthesis leads to improved performance in both tasks, for the first time, highlighting the inherent relationship between spatial awareness and visual prediction.

Title: On the Crucial Role of Initialization for Matrix Factorization

• Authors: Bingcong Li, Liang Zhang, Aryan Mokhtari, Niao He
• Subjects: cs.LG, eess.SP, math.OC
• Abstract URL: https://arxiv.org/abs/2410.18965
• Pdf URL: https://arxiv.org/pdf/2410.18965
• Copy Paste: [[2410.18965]] On the Crucial Role of Initialization for Matrix Factorization(https://arxiv.org/abs/2410.18965)
• Keywords: diffusion, foundation model
• Abstract: This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth optimization. We introduce Nystrom initialization, which significantly improves the global convergence of Scaled Gradient Descent (ScaledGD) in both symmetric and asymmetric matrix factorization tasks. Specifically, we prove that ScaledGD with Nystrom initialization achieves quadratic convergence in cases where only linear rates were previously known. Furthermore, we extend this initialization to low-rank adapters (LoRA) commonly used for finetuning foundation models. Our approach, NoRA, i.e., LoRA with Nystrom initialization, demonstrates superior performance across various downstream tasks and model scales, from 1B to 7B parameters, in large language and diffusion models.

Title: Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques

• Authors: David Ortiz-Perez, Manuel Benavent-Lledo, Jose Garcia-Rodriguez, David Tomás, M. Flores Vizcaya-Moreno
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18972
• Pdf URL: https://arxiv.org/pdf/2410.18972
• Copy Paste: [[2410.18972]] Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques(https://arxiv.org/abs/2410.18972)
• Keywords: foundation model
• Abstract: Cognitive decline is a natural part of aging, often resulting in reduced cognitive abilities. In some cases, however, this decline is more pronounced, typically due to disorders such as Alzheimer's disease. Early detection of anomalous cognitive decline is crucial, as it can facilitate timely professional intervention. While medical data can help in this detection, it often involves invasive procedures. An alternative approach is to employ non-intrusive techniques such as speech or handwriting analysis, which do not necessarily affect daily activities. This survey reviews the most relevant methodologies that use deep learning techniques to automate the cognitive decline estimation task, including audio, text, and visual processing. We discuss the key features and advantages of each modality and methodology, including state-of-the-art approaches like Transformer architecture and foundation models. In addition, we present works that integrate different modalities to develop multimodal models. We also highlight the most significant datasets and the quantitative results from studies using these resources. From this review, several conclusions emerge. In most cases, the textual modality achieves the best results and is the most relevant for detecting cognitive decline. Moreover, combining various approaches from individual modalities into a multimodal model consistently enhances performance across nearly all scenarios.

Title: 3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation

• Authors: Hansheng Chen, Bokui Shen, Yulin Liu, Ruoxi Shi, Linqi Zhou, Connor Z. Lin, Jiayuan Gu, Hao Su, Gordon Wetzstein, Leonidas Guibas
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18974
• Pdf URL: https://arxiv.org/pdf/2410.18974
• Copy Paste: [[2410.18974]] 3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation(https://arxiv.org/abs/2410.18974)
• Keywords: diffusion
• Abstract: Multi-view image diffusion models have significantly advanced open-domain 3D object generation. However, most existing models rely on 2D network architectures that lack inherent 3D biases, resulting in compromised geometric consistency. To address this challenge, we introduce 3D-Adapter, a plug-in module designed to infuse 3D geometry awareness into pretrained image diffusion models. Central to our approach is the idea of 3D feedback augmentation: for each denoising step in the sampling loop, 3D-Adapter decodes intermediate multi-view features into a coherent 3D representation, then re-encodes the rendered RGBD views to augment the pretrained base model through feature addition. We study two variants of 3D-Adapter: a fast feed-forward version based on Gaussian splatting and a versatile training-free version utilizing neural fields and meshes. Our extensive experiments demonstrate that 3D-Adapter not only greatly enhances the geometry quality of text-to-multi-view models such as Instant3D and Zero123++, but also enables high-quality 3D generation using the plain text-to-image Stable Diffusion. Furthermore, we showcase the broad application potential of 3D-Adapter by presenting high quality results in text-to-3D, image-to-3D, text-to-texture, and text-to-avatar tasks.

Title: Unbounded: A Generative Infinite Game of Character Life Simulation

• Authors: Jialu Li, Yuanzhen Li, Neal Wadhwa, Yael Pritch, David E. Jacobs, Michael Rubinstein, Mohit Bansal, Nataniel Ruiz
• Subjects: cs.CV, cs.AI, cs.CL, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18975
• Pdf URL: https://arxiv.org/pdf/2410.18975
• Copy Paste: [[2410.18975]] Unbounded: A Generative Infinite Game of Character Life Simulation(https://arxiv.org/abs/2410.18975)
• Keywords: generative
• Abstract: We introduce the concept of a generative infinite game, a video game that transcends the traditional boundaries of finite, hard-coded systems by using generative models. Inspired by James P. Carse's distinction between finite and infinite games, we leverage recent advances in generative AI to create Unbounded: a game of character life simulation that is fully encapsulated in generative models. Specifically, Unbounded draws inspiration from sandbox life simulations and allows you to interact with your autonomous virtual character in a virtual world by feeding, playing with and guiding it - with open-ended mechanics generated by an LLM, some of which can be emergent. In order to develop Unbounded, we propose technical innovations in both the LLM and visual generation domains. Specifically, we present: (1) a specialized, distilled large language model (LLM) that dynamically generates game mechanics, narratives, and character interactions in real-time, and (2) a new dynamic regional image prompt Adapter (IP-Adapter) for vision models that ensures consistent yet flexible visual generation of a character across multiple environments. We evaluate our system through both qualitative and quantitative analysis, showing significant improvements in character life simulation, user instruction following, narrative coherence, and visual consistency for both characters and the environments compared to traditional related approaches.

Title: MotionCLR: Motion Generation and Training-free Editing via Understanding Attention Mechanisms

• Authors: Ling-Hao Chen, Wenxun Dai, Xuan Ju, Shunlin Lu, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18977
• Pdf URL: https://arxiv.org/pdf/2410.18977
• Copy Paste: [[2410.18977]] MotionCLR: Motion Generation and Training-free Editing via Understanding Attention Mechanisms(https://arxiv.org/abs/2410.18977)
• Keywords: diffusion
• Abstract: This research delves into the problem of interactive editing of human motion generation. Previous motion diffusion models lack explicit modeling of the word-level text-motion correspondence and good explainability, hence restricting their fine-grained editing ability. To address this issue, we propose an attention-based motion diffusion model, namely MotionCLR, with CLeaR modeling of attention mechanisms. Technically, MotionCLR models the in-modality and cross-modality interactions with self-attention and cross-attention, respectively. More specifically, the self-attention mechanism aims to measure the sequential similarity between frames and impacts the order of motion features. By contrast, the cross-attention mechanism works to find the fine-grained word-sequence correspondence and activate the corresponding timesteps in the motion sequence. Based on these key properties, we develop a versatile set of simple yet effective motion editing methods via manipulating attention maps, such as motion (de-)emphasizing, in-place motion replacement, and example-based motion generation, etc. For further verification of the explainability of the attention mechanism, we additionally explore the potential of action-counting and grounded motion generation ability via attention maps. Our experimental results show that our method enjoys good generation and editing ability with good explainability.