2024-09-06

Title: SDOoop: Capturing Periodical Patterns and Out-of-phase Anomalies in Streaming Data Analysis

• Authors: Alexander Hartl, Félix Iglesias Vázquez, Tanja Zseby
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.02973
• Pdf URL: https://arxiv.org/pdf/2409.02973
• Copy Paste: [[2409.02973]] SDOoop: Capturing Periodical Patterns and Out-of-phase Anomalies in Streaming Data Analysis(https://arxiv.org/abs/2409.02973)
• Keywords: anomaly
• Abstract: Streaming data analysis is increasingly required in applications, e.g., IoT, cybersecurity, robotics, mechatronics or cyber-physical systems. Despite its relevance, it is still an emerging field with open challenges. SDO is a recent anomaly detection method designed to meet requirements of speed, interpretability and intuitive parameterization. In this work, we present SDOoop, which extends the capabilities of SDO's streaming version to retain temporal information of data structures. SDOoop spots contextual anomalies undetectable by traditional algorithms, while enabling the inspection of data geometries, clusters and temporal patterns. We used SDOoop to model real network communications in critical infrastructures and extract patterns that disclose their dynamics. Moreover, we evaluated SDOoop with data from intrusion detection and natural science domains and obtained performances equivalent or superior to state-of-the-art approaches. Our results show the high potential of new model-based methods to analyze and explain streaming data. Since SDOoop operates with constant per-sample space and time complexity, it is ideal for big data, being able to instantly process large volumes of information. SDOoop conforms to next-generation machine learning, which, in addition to accuracy and speed, is expected to provide highly interpretable and informative models.

Title: NUMOSIM: A Synthetic Mobility Dataset with Anomaly Detection Benchmarks

• Authors: Chris Stanford, Suman Adari, Xishun Liao, Yueshuai He, Qinhua Jiang, Chenchen Kuai, Jiaqi Ma, Emmanuel Tung, Yinlong Qian, Lingyi Zhao, Zihao Zhou, Zeeshan Rasheed, Khurram Shafique
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03024
• Pdf URL: https://arxiv.org/pdf/2409.03024
• Copy Paste: [[2409.03024]] NUMOSIM: A Synthetic Mobility Dataset with Anomaly Detection Benchmarks(https://arxiv.org/abs/2409.03024)
• Keywords: anomaly
• Abstract: Collecting real-world mobility data is challenging. It is often fraught with privacy concerns, logistical difficulties, and inherent biases. Moreover, accurately annotating anomalies in large-scale data is nearly impossible, as it demands meticulous effort to distinguish subtle and complex patterns. These challenges significantly impede progress in geospatial anomaly detection research by restricting access to reliable data and complicating the rigorous evaluation, comparison, and benchmarking of methodologies. To address these limitations, we introduce a synthetic mobility dataset, NUMOSIM, that provides a controlled, ethical, and diverse environment for benchmarking anomaly detection techniques. NUMOSIM simulates a wide array of realistic mobility scenarios, encompassing both typical and anomalous behaviours, generated through advanced deep learning models trained on real mobility data. This approach allows NUMOSIM to accurately replicate the complexities of real-world movement patterns while strategically injecting anomalies to challenge and evaluate detection algorithms based on how effectively they capture the interplay between demographic, geospatial, and temporal factors. Our goal is to advance geospatial mobility analysis by offering a realistic benchmark for improving anomaly detection and mobility modeling techniques. To support this, we provide open access to the NUMOSIM dataset, along with comprehensive documentation, evaluation metrics, and benchmark results.

Title: MDNF: Multi-Diffusion-Nets for Neural Fields on Meshes

• Authors: Avigail Cohen Rimon, Tal Shnitzer, Mirela Ben Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03034
• Pdf URL: https://arxiv.org/pdf/2409.03034
• Copy Paste: [[2409.03034]] MDNF: Multi-Diffusion-Nets for Neural Fields on Meshes(https://arxiv.org/abs/2409.03034)
• Keywords: diffusion
• Abstract: We propose a novel framework for representing neural fields on triangle meshes that is multi-resolution across both spatial and frequency domains. Inspired by the Neural Fourier Filter Bank (NFFB), our architecture decomposes the spatial and frequency domains by associating finer spatial resolution levels with higher frequency bands, while coarser resolutions are mapped to lower frequencies. To achieve geometry-aware spatial decomposition we leverage multiple DiffusionNet components, each associated with a different spatial resolution level. Subsequently, we apply a Fourier feature mapping to encourage finer resolution levels to be associated with higher frequencies. The final signal is composed in a wavelet-inspired manner using a sine-activated MLP, aggregating higher-frequency signals on top of lower-frequency ones. Our architecture attains high accuracy in learning complex neural fields and is robust to discontinuities, exponential scale variations of the target field, and mesh modification. We demonstrate the effectiveness of our approach through its application to diverse neural fields, such as synthetic RGB functions, UV texture coordinates, and vertex normals, illustrating different challenges. To validate our method, we compare its performance against two alternatives, showcasing the advantages of our multi-resolution architecture.

Title: Can Your Generative Model Detect Out-of-Distribution Covariate Shift?

• Authors: Christiaan Viviers, Amaan Valiuddin, Francisco Caetano, Lemar Abdi, Lena Filatova, Peter de With, Fons van der Sommen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03043
• Pdf URL: https://arxiv.org/pdf/2409.03043
• Copy Paste: [[2409.03043]] Can Your Generative Model Detect Out-of-Distribution Covariate Shift?(https://arxiv.org/abs/2409.03043)
• Keywords: generative
• Abstract: Detecting Out-of-Distribution~(OOD) sensory data and covariate distribution shift aims to identify new test examples with different high-level image statistics to the captured, normal and In-Distribution (ID) set. Existing OOD detection literature largely focuses on semantic shift with little-to-no consensus over covariate shift. Generative models capture the ID data in an unsupervised manner, enabling them to effectively identify samples that deviate significantly from this learned distribution, irrespective of the downstream task. In this work, we elucidate the ability of generative models to detect and quantify domain-specific covariate shift through extensive analyses that involves a variety of models. To this end, we conjecture that it is sufficient to detect most occurring sensory faults (anomalies and deviations in global signals statistics) by solely modeling high-frequency signal-dependent and independent details. We propose a novel method, CovariateFlow, for OOD detection, specifically tailored to covariate heteroscedastic high-frequency image-components using conditional Normalizing Flows (cNFs). Our results on CIFAR10 vs. CIFAR10-C and ImageNet200 vs. ImageNet200-C demonstrate the effectiveness of the method by accurately detecting OOD covariate shift. This work contributes to enhancing the fidelity of imaging systems and aiding machine learning models in OOD detection in the presence of covariate shift.

Title: Oddballness: universal anomaly detection with language models

• Authors: Filip Graliński, Ryszard Staruch, Krzysztof Jurkiewicz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.03046
• Pdf URL: https://arxiv.org/pdf/2409.03046
• Copy Paste: [[2409.03046]] Oddballness: universal anomaly detection with language models(https://arxiv.org/abs/2409.03046)
• Keywords: anomaly
• Abstract: We present a new method to detect anomalies in texts (in general: in sequences of any data), using language models, in a totally unsupervised manner. The method considers probabilities (likelihoods) generated by a language model, but instead of focusing on low-likelihood tokens, it considers a new metric introduced in this paper: oddballness. Oddballness measures how ``strange'' a given token is according to the language model. We demonstrate in grammatical error detection tasks (a specific case of text anomaly detection) that oddballness is better than just considering low-likelihood events, if a totally unsupervised setup is assumed.

Title: Spatial Diffusion for Cell Layout Generation

• Authors: Chen Li, Xiaoling Hu, Shahira Abousamra, Meilong Xu, Chao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03106
• Pdf URL: https://arxiv.org/pdf/2409.03106
• Copy Paste: [[2409.03106]] Spatial Diffusion for Cell Layout Generation(https://arxiv.org/abs/2409.03106)
• Keywords: diffusion, generative
• Abstract: Generative models, such as GANs and diffusion models, have been used to augment training sets and boost performances in different tasks. We focus on generative models for cell detection instead, i.e., locating and classifying cells in given pathology images. One important information that has been largely overlooked is the spatial patterns of the cells. In this paper, we propose a spatial-pattern-guided generative model for cell layout generation. Specifically, a novel diffusion model guided by spatial features and generates realistic cell layouts has been proposed. We explore different density models as spatial features for the diffusion model. In downstream tasks, we show that the generated cell layouts can be used to guide the generation of high-quality pathology images. Augmenting with these images can significantly boost the performance of SOTA cell detection methods. The code is available at this https URL.

Title: Probing self-attention in self-supervised speech models for cross-linguistic differences

• Authors: Sai Gopinath, Joselyn Rodriguez
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03115
• Pdf URL: https://arxiv.org/pdf/2409.03115
• Copy Paste: [[2409.03115]] Probing self-attention in self-supervised speech models for cross-linguistic differences(https://arxiv.org/abs/2409.03115)
• Keywords: self-supervised
• Abstract: Speech models have gained traction thanks to increase in accuracy from novel transformer architectures. While this impressive increase in performance across automatic speech recognition (ASR) benchmarks is noteworthy, there is still much that is unknown about the use of attention mechanisms for speech-related tasks. For example, while it is assumed that these models are learning language-independent (i.e., universal) speech representations, there has not yet been an in-depth exploration of what it would mean for the models to be language-independent. In the current paper, we explore this question within the realm of self-attention mechanisms of one small self-supervised speech transformer model (TERA). We find that even with a small model, the attention heads learned are diverse ranging from almost entirely diagonal to almost entirely global regardless of the training language. We highlight some notable differences in attention patterns between Turkish and English and demonstrate that the models do learn important phonological information during pretraining. We also present a head ablation study which shows that models across languages primarily rely on diagonal heads to classify phonemes.

Title: A Scalable Matrix Visualization for Understanding Tree Ensemble Classifiers

• Authors: Zhen Li, Weikai Yang, Jun Yuan, Jing Wu, Changjian Chen, Yao Ming, Fan Yang, Hui Zhang, Shixia Liu
• Subjects: cs.LG, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.03164
• Pdf URL: https://arxiv.org/pdf/2409.03164
• Copy Paste: [[2409.03164]] A Scalable Matrix Visualization for Understanding Tree Ensemble Classifiers(https://arxiv.org/abs/2409.03164)
• Keywords: anomaly
• Abstract: The high performance of tree ensemble classifiers benefits from a large set of rules, which, in turn, makes the models hard to understand. To improve interpretability, existing methods extract a subset of rules for approximation using model reduction techniques. However, by focusing on the reduced rule set, these methods often lose fidelity and ignore anomalous rules that, despite their infrequency, play crucial roles in real-world applications. This paper introduces a scalable visual analysis method to explain tree ensemble classifiers that contain tens of thousands of rules. The key idea is to address the issue of losing fidelity by adaptively organizing the rules as a hierarchy rather than reducing them. To ensure the inclusion of anomalous rules, we develop an anomaly-biased model reduction method to prioritize these rules at each hierarchical level. Synergized with this hierarchical organization of rules, we develop a matrix-based hierarchical visualization to support exploration at different levels of detail. Our quantitative experiments and case studies demonstrate how our method fosters a deeper understanding of both common and anomalous rules, thereby enhancing interpretability without sacrificing comprehensiveness.

Title: RoomDiffusion: A Specialized Diffusion Model in the Interior Design Industry

• Authors: Zhaowei Wang, Ying Hao, Hao Wei, Qing Xiao, Lulu Chen, Yulong Li, Yue Yang, Tianyi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03198
• Pdf URL: https://arxiv.org/pdf/2409.03198
• Copy Paste: [[2409.03198]] RoomDiffusion: A Specialized Diffusion Model in the Interior Design Industry(https://arxiv.org/abs/2409.03198)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-image diffusion models have significantly transformed visual content generation, yet their application in specialized fields such as interior design remains underexplored. In this paper, we present RoomDiffusion, a pioneering diffusion model meticulously tailored for the interior design industry. To begin with, we build from scratch a whole data pipeline to update and evaluate data for iterative model optimization. Subsequently, techniques such as multiaspect training, multi-stage fine-tune and model fusion are applied to enhance both the visual appeal and precision of the generated results. Lastly, leveraging the latent consistency Distillation method, we distill and expedite the model for optimal efficiency. Unlike existing models optimized for general scenarios, RoomDiffusion addresses specific challenges in interior design, such as lack of fashion, high furniture duplication rate, and inaccurate style. Through our holistic human evaluation protocol with more than 20 professional human evaluators, RoomDiffusion demonstrates industry-leading performance in terms of aesthetics, accuracy, and efficiency, surpassing all existing open source models such as stable diffusion and SDXL.

Title: An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classification

• Authors: Zhuowei Chen, Lianxi Wang, Yuben Wu, Xinfeng Liao, Yujia Tian, Junyang Zhong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03203
• Pdf URL: https://arxiv.org/pdf/2409.03203
• Copy Paste: [[2409.03203]] An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classification(https://arxiv.org/abs/2409.03203)
• Keywords: diffusion
• Abstract: Sentiment classification (SC) often suffers from low-resource challenges such as domain-specific contexts, imbalanced label distributions, and few-shot scenarios. The potential of the diffusion language model (LM) for textual data augmentation (DA) remains unexplored, moreover, textual DA methods struggle to balance the diversity and consistency of new samples. Most DA methods either perform logical modifications or rephrase less important tokens in the original sequence with the language model. In the context of SC, strong emotional tokens could act critically on the sentiment of the whole sequence. Therefore, contrary to rephrasing less important context, we propose DiffusionCLS to leverage a diffusion LM to capture in-domain knowledge and generate pseudo samples by reconstructing strong label-related tokens. This approach ensures a balance between consistency and diversity, avoiding the introduction of noise and augmenting crucial features of datasets. DiffusionCLS also comprises a Noise-Resistant Training objective to help the model generalize. Experiments demonstrate the effectiveness of our method in various low-resource scenarios including domain-specific and domain-general problems. Ablation studies confirm the effectiveness of our framework's modules, and visualization studies highlight optimal deployment conditions, reinforcing our conclusions.

Title: iSeg: An Iterative Refinement-based Framework for Training-free Segmentation

• Authors: Lin Sun, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, Yanwei Pang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03209
• Pdf URL: https://arxiv.org/pdf/2409.03209
• Copy Paste: [[2409.03209]] iSeg: An Iterative Refinement-based Framework for Training-free Segmentation(https://arxiv.org/abs/2409.03209)
• Keywords: diffusion
• Abstract: Stable diffusion has demonstrated strong image synthesis ability to given text descriptions, suggesting it to contain strong semantic clue for grouping objects. Inspired by this, researchers have explored employing stable diffusion for trainingfree segmentation. Most existing approaches either simply employ cross-attention map or refine it by self-attention map, to generate segmentation masks. We believe that iterative refinement with self-attention map would lead to better results. However, we mpirically demonstrate that such a refinement is sub-optimal likely due to the self-attention map containing irrelevant global information which hampers accurately refining cross-attention map with multiple iterations. To address this, we propose an iterative refinement framework for training-free segmentation, named iSeg, having an entropy-reduced self-attention module which utilizes a gradient descent scheme to reduce the entropy of self-attention map, thereby suppressing the weak responses corresponding to irrelevant global information. Leveraging the entropy-reduced self-attention module, our iSeg stably improves refined crossattention map with iterative refinement. Further, we design a category-enhanced cross-attention module to generate accurate cross-attention map, providing a better initial input for iterative refinement. Extensive experiments across different datasets and diverse segmentation tasks reveal the merits of proposed contributions, leading to promising performance on diverse segmentation tasks. For unsupervised semantic segmentation on Cityscapes, our iSeg achieves an absolute gain of 3.8% in terms of mIoU compared to the best existing training-free approach in literature. Moreover, our proposed iSeg can support segmentation with different kind of images and interactions.

Title: Optimizing 3D Gaussian Splatting for Sparse Viewpoint Scene Reconstruction

• Authors: Shen Chen, Jiale Zhou, Lei Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03213
• Pdf URL: https://arxiv.org/pdf/2409.03213
• Copy Paste: [[2409.03213]] Optimizing 3D Gaussian Splatting for Sparse Viewpoint Scene Reconstruction(https://arxiv.org/abs/2409.03213)
• Keywords: diffusion
• Abstract: 3D Gaussian Splatting (3DGS) has emerged as a promising approach for 3D scene representation, offering a reduction in computational overhead compared to Neural Radiance Fields (NeRF). However, 3DGS is susceptible to high-frequency artifacts and demonstrates suboptimal performance under sparse viewpoint conditions, thereby limiting its applicability in robotics and computer vision. To address these limitations, we introduce SVS-GS, a novel framework for Sparse Viewpoint Scene reconstruction that integrates a 3D Gaussian smoothing filter to suppress artifacts. Furthermore, our approach incorporates a Depth Gradient Profile Prior (DGPP) loss with a dynamic depth mask to sharpen edges and 2D diffusion with Score Distillation Sampling (SDS) loss to enhance geometric consistency in novel view synthesis. Experimental evaluations on the MipNeRF-360 and SeaThru-NeRF datasets demonstrate that SVS-GS markedly improves 3D reconstruction from sparse viewpoints, offering a robust and efficient solution for scene understanding in robotics and computer vision applications.

Title: Unveiling Context-Related Anomalies: Knowledge Graph Empowered Decoupling of Scene and Action for Human-Related Video Anomaly Detection

• Authors: Chenglizhao Chen, Xinyu Liu, Mengke Song, Luming Li, Xu Yu, Shanchen Pang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03236
• Pdf URL: https://arxiv.org/pdf/2409.03236
• Copy Paste: [[2409.03236]] Unveiling Context-Related Anomalies: Knowledge Graph Empowered Decoupling of Scene and Action for Human-Related Video Anomaly Detection(https://arxiv.org/abs/2409.03236)
• Keywords: anomaly
• Abstract: Detecting anomalies in human-related videos is crucial for surveillance applications. Current methods primarily include appearance-based and action-based techniques. Appearance-based methods rely on low-level visual features such as color, texture, and shape. They learn a large number of pixel patterns and features related to known scenes during training, making them effective in detecting anomalies within these familiar contexts. However, when encountering new or significantly changed scenes, i.e., unknown scenes, they often fail because existing SOTA methods do not effectively capture the relationship between actions and their surrounding scenes, resulting in low generalization. In contrast, action-based methods focus on detecting anomalies in human actions but are usually less informative because they tend to overlook the relationship between actions and their scenes, leading to incorrect detection. For instance, the normal event of running on the beach and the abnormal event of running on the street might both be considered normal due to the lack of scene information. In short, current methods struggle to integrate low-level visual and high-level action features, leading to poor anomaly detection in varied and complex scenes. To address this challenge, we propose a novel decoupling-based architecture for human-related video anomaly detection (DecoAD). DecoAD significantly improves the integration of visual and action features through the decoupling and interweaving of scenes and actions, thereby enabling a more intuitive and accurate understanding of complex behaviors and scenes. DecoAD supports fully supervised, weakly supervised, and unsupervised settings.

Title: SVP: Style-Enhanced Vivid Portrait Talking Head Diffusion Model

• Authors: Weipeng Tan, Chuming Lin, Chengming Xu, Xiaozhong Ji, Junwei Zhu, Chengjie Wang, Yanwei Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03270
• Pdf URL: https://arxiv.org/pdf/2409.03270
• Copy Paste: [[2409.03270]] SVP: Style-Enhanced Vivid Portrait Talking Head Diffusion Model(https://arxiv.org/abs/2409.03270)
• Keywords: diffusion
• Abstract: Talking Head Generation (THG), typically driven by audio, is an important and challenging task with broad application prospects in various fields such as digital humans, film production, and virtual reality. While diffusion model-based THG methods present high quality and stable content generation, they often overlook the intrinsic style which encompasses personalized features such as speaking habits and facial expressions of a video. As consequence, the generated video content lacks diversity and vividness, thus being limited in real life scenarios. To address these issues, we propose a novel framework named Style-Enhanced Vivid Portrait (SVP) which fully leverages style-related information in THG. Specifically, we first introduce the novel probabilistic style prior learning to model the intrinsic style as a Gaussian distribution using facial expressions and audio embedding. The distribution is learned through the 'bespoked' contrastive objective, effectively capturing the dynamic style information in each video. Then we finetune a pretrained Stable Diffusion (SD) model to inject the learned intrinsic style as a controlling signal via cross attention. Experiments show that our model generates diverse, vivid, and high-quality videos with flexible control over intrinsic styles, outperforming existing state-of-the-art methods.

Title: OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving

• Authors: Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, Wenchao Ding
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.03272
• Pdf URL: https://arxiv.org/pdf/2409.03272
• Copy Paste: [[2409.03272]] OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving(https://arxiv.org/abs/2409.03272)
• Keywords: foundation model, generative
• Abstract: The rise of multi-modal large language models(MLLMs) has spurred their applications in autonomous driving. Recent MLLM-based methods perform action by learning a direct mapping from perception to action, neglecting the dynamics of the world and the relations between action and world dynamics. In contrast, human beings possess world model that enables them to simulate the future states based on 3D internal visual representation and plan actions accordingly. To this end, we propose OccLLaMA, an occupancy-language-action generative world model, which uses semantic occupancy as a general visual representation and unifies vision-language-action(VLA) modalities through an autoregressive model. Specifically, we introduce a novel VQVAE-like scene tokenizer to efficiently discretize and reconstruct semantic occupancy scenes, considering its sparsity and classes imbalance. Then, we build a unified multi-modal vocabulary for vision, language and action. Furthermore, we enhance LLM, specifically LLaMA, to perform the next token/scene prediction on the unified vocabulary to complete multiple tasks in autonomous driving. Extensive experiments demonstrate that OccLLaMA achieves competitive performance across multiple tasks, including 4D occupancy forecasting, motion planning, and visual question answering, showcasing its potential as a foundation model in autonomous driving.

Title: Enhancing User-Centric Privacy Protection: An Interactive Framework through Diffusion Models and Machine Unlearning

• Authors: Huaxi Huang, Xin Yuan, Qiyu Liao, Dadong Wang, Tongliang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03326
• Pdf URL: https://arxiv.org/pdf/2409.03326
• Copy Paste: [[2409.03326]] Enhancing User-Centric Privacy Protection: An Interactive Framework through Diffusion Models and Machine Unlearning(https://arxiv.org/abs/2409.03326)
• Keywords: diffusion, generative
• Abstract: In the realm of multimedia data analysis, the extensive use of image datasets has escalated concerns over privacy protection within such data. Current research predominantly focuses on privacy protection either in data sharing or upon the release of trained machine learning models. Our study pioneers a comprehensive privacy protection framework that safeguards image data privacy concurrently during data sharing and model publication. We propose an interactive image privacy protection framework that utilizes generative machine learning models to modify image information at the attribute level and employs machine unlearning algorithms for the privacy preservation of model parameters. This user-interactive framework allows for adjustments in privacy protection intensity based on user feedback on generated images, striking a balance between maximal privacy safeguarding and maintaining model performance. Within this framework, we instantiate two modules: a differential privacy diffusion model for protecting attribute information in images and a feature unlearning algorithm for efficient updates of the trained model on the revised image dataset. Our approach demonstrated superiority over existing methods on facial datasets across various attribute classifications.

Title: Sketch: A Toolkit for Streamlining LLM Operations

• Authors: Xin Jiang, Xiang Li, Wenjia Ma, Xuezhi Fang, Yiqun Yao, Naitong Yu, Xuying Meng, Peng Han, Jing Li, Aixin Sun, Yequan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03346
• Pdf URL: https://arxiv.org/pdf/2409.03346
• Copy Paste: [[2409.03346]] Sketch: A Toolkit for Streamlining LLM Operations(https://arxiv.org/abs/2409.03346)
• Keywords: generative
• Abstract: Large language models (LLMs) represented by GPT family have achieved remarkable success. The characteristics of LLMs lie in their ability to accommodate a wide range of tasks through a generative approach. However, the flexibility of their output format poses challenges in controlling and harnessing the model's outputs, thereby constraining the application of LLMs in various domains. In this work, we present Sketch, an innovative toolkit designed to streamline LLM operations across diverse fields. Sketch comprises the following components: (1) a suite of task description schemas and prompt templates encompassing various NLP tasks; (2) a user-friendly, interactive process for building structured output LLM services tailored to various NLP tasks; (3) an open-source dataset for output format control, along with tools for dataset construction; and (4) an open-source model based on LLaMA3-8B-Instruct that adeptly comprehends and adheres to output formatting instructions. We anticipate this initiative to bring considerable convenience to LLM users, achieving the goal of ''plug-and-play'' for various applications. The components of Sketch will be progressively open-sourced at this https URL.

Title: KAN See In the Dark

• Authors: Aoxiang Ning, Minglong Xue, Jinhong He, Chengyun Song
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03404
• Pdf URL: https://arxiv.org/pdf/2409.03404
• Copy Paste: [[2409.03404]] KAN See In the Dark(https://arxiv.org/abs/2409.03404)
• Keywords: diffusion
• Abstract: Existing low-light image enhancement methods are difficult to fit the complex nonlinear relationship between normal and low-light images due to uneven illumination and noise effects. The recently proposed Kolmogorov-Arnold networks (KANs) feature spline-based convolutional layers and learnable activation functions, which can effectively capture nonlinear dependencies. In this paper, we design a KAN-Block based on KANs and innovatively apply it to low-light image enhancement. This method effectively alleviates the limitations of current methods constrained by linear network structures and lack of interpretability, further demonstrating the potential of KANs in low-level vision tasks. Given the poor perception of current low-light image enhancement methods and the stochastic nature of the inverse diffusion process, we further introduce frequency-domain perception for visually oriented enhancement. Extensive experiments demonstrate the competitive performance of our method on benchmark datasets. The code will be available at: this https URL}{this https URL.

Title: Automatic occlusion removal from 3D maps for maritime situational awareness

• Authors: Felix Sattler, Borja Carrillo Perez, Maurice Stephan, Sarah Barnes
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03451
• Pdf URL: https://arxiv.org/pdf/2409.03451
• Copy Paste: [[2409.03451]] Automatic occlusion removal from 3D maps for maritime situational awareness(https://arxiv.org/abs/2409.03451)
• Keywords: generative
• Abstract: We introduce a novel method for updating 3D geospatial models, specifically targeting occlusion removal in large-scale maritime environments. Traditional 3D reconstruction techniques often face problems with dynamic objects, like cars or vessels, that obscure the true environment, leading to inaccurate models or requiring extensive manual editing. Our approach leverages deep learning techniques, including instance segmentation and generative inpainting, to directly modify both the texture and geometry of 3D meshes without the need for costly reprocessing. By selectively targeting occluding objects and preserving static elements, the method enhances both geometric and visual accuracy. This approach not only preserves structural and textural details of map data but also maintains compatibility with current geospatial standards, ensuring robust performance across diverse datasets. The results demonstrate significant improvements in 3D model fidelity, making this method highly applicable for maritime situational awareness and the dynamic display of auxiliary information.

Title: Data-free Distillation with Degradation-prompt Diffusion for Multi-weather Image Restoration

• Authors: Pei Wang, Xiaotong Luo, Yuan Xie, Yanyun Qu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03455
• Pdf URL: https://arxiv.org/pdf/2409.03455
• Copy Paste: [[2409.03455]] Data-free Distillation with Degradation-prompt Diffusion for Multi-weather Image Restoration(https://arxiv.org/abs/2409.03455)
• Keywords: diffusion
• Abstract: Multi-weather image restoration has witnessed incredible progress, while the increasing model capacity and expensive data acquisition impair its applications in memory-limited devices. Data-free distillation provides an alternative for allowing to learn a lightweight student model from a pre-trained teacher model without relying on the original training data. The existing data-free learning methods mainly optimize the models with the pseudo data generated by GANs or the real data collected from the Internet. However, they inevitably suffer from the problems of unstable training or domain shifts with the original data. In this paper, we propose a novel Data-free Distillation with Degradation-prompt Diffusion framework for multi-weather Image Restoration (D4IR). It replaces GANs with pre-trained diffusion models to avoid model collapse and incorporates a degradation-aware prompt adapter to facilitate content-driven conditional diffusion for generating domain-related images. Specifically, a contrast-based degradation prompt adapter is firstly designed to capture degradation-aware prompts from web-collected degraded images. Then, the collected unpaired clean images are perturbed to latent features of stable diffusion, and conditioned with the degradation-aware prompts to synthesize new domain-related degraded images for knowledge distillation. Experiments illustrate that our proposal achieves comparable performance to the model distilled with original training data, and is even superior to other mainstream unsupervised methods.

Title: LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priors

• Authors: Hanyang Yu, Xiaoxiao Long, Ping Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03456
• Pdf URL: https://arxiv.org/pdf/2409.03456
• Copy Paste: [[2409.03456]] LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priors(https://arxiv.org/abs/2409.03456)
• Keywords: diffusion
• Abstract: We aim to address sparse-view reconstruction of a 3D scene by leveraging priors from large-scale vision models. While recent advancements such as 3D Gaussian Splatting (3DGS) have demonstrated remarkable successes in 3D reconstruction, these methods typically necessitate hundreds of input images that densely capture the underlying scene, making them time-consuming and impractical for real-world applications. However, sparse-view reconstruction is inherently ill-posed and under-constrained, often resulting in inferior and incomplete outcomes. This is due to issues such as failed initialization, overfitting on input images, and a lack of details. To mitigate these challenges, we introduce LM-Gaussian, a method capable of generating high-quality reconstructions from a limited number of images. Specifically, we propose a robust initialization module that leverages stereo priors to aid in the recovery of camera poses and the reliable point clouds. Additionally, a diffusion-based refinement is iteratively applied to incorporate image diffusion priors into the Gaussian optimization process to preserve intricate scene details. Finally, we utilize video diffusion priors to further enhance the rendered images for realistic visual effects. Overall, our approach significantly reduces the data acquisition requirements compared to previous 3DGS methods. We validate the effectiveness of our framework through experiments on various public datasets, demonstrating its potential for high-quality 360-degree scene reconstruction. Visual results are on our website.

Title: ScreenMark: Watermarking Arbitrary Visual Content on Screen

• Authors: Xiujian Liang, Gaozhi Liu, Yichao Si, Xiaoxiao Hu, Zhenxing Qian, Xinpeng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03487
• Pdf URL: https://arxiv.org/pdf/2409.03487
• Copy Paste: [[2409.03487]] ScreenMark: Watermarking Arbitrary Visual Content on Screen(https://arxiv.org/abs/2409.03487)
• Keywords: diffusion
• Abstract: Digital watermarking has demonstrated its effectiveness in protecting multimedia content. However, existing watermarking are predominantly tailored for specific media types, rendering them less effective for the protection of content displayed on computer screens, which is often multimodal and dynamic. Visual Screen Content (VSC), is particularly susceptible to theft and leakage via screenshots, a vulnerability that current watermarking methods fail to adequately this http URL tackle these challenges, we propose ScreenMark, a robust and practical watermarking method designed specifically for arbitrary VSC protection. ScreenMark utilizes a three-stage progressive watermarking framework. Initially, inspired by diffusion principles, we initialize the mutual transformation between regular watermark information and irregular watermark patterns. Subsequently, these patterns are integrated with screen content using a pre-multiplication alpha blending technique, supported by a pre-trained screen decoder for accurate watermark retrieval. The progressively complex distorter enhances the robustness of the watermark in real-world screenshot scenarios. Finally, the model undergoes fine-tuning guided by a joint-level distorter to ensure optimal this http URL validate the effectiveness of ScreenMark, we compiled a dataset comprising 100,000 screenshots from various devices and resolutions. Extensive experiments across different datasets confirm the method's superior robustness, imperceptibility, and practical applicability.

Title: Blended Latent Diffusion under Attention Control for Real-World Video Editing

• Authors: Deyin Liu, Lin Yuanbo Wu, Xianghua Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03514
• Pdf URL: https://arxiv.org/pdf/2409.03514
• Copy Paste: [[2409.03514]] Blended Latent Diffusion under Attention Control for Real-World Video Editing(https://arxiv.org/abs/2409.03514)
• Keywords: diffusion
• Abstract: Due to lack of fully publicly available text-to-video models, current video editing methods tend to build on pre-trained text-to-image generation models, however, they still face grand challenges in dealing with the local editing of video with temporal information. First, although existing methods attempt to focus on local area editing by a pre-defined mask, the preservation of the outside-area background is non-ideal due to the spatially entire generation of each frame. In addition, specially providing a mask by user is an additional costly undertaking, so an autonomous masking strategy integrated into the editing process is desirable. Last but not least, image-level pretrained model hasn't learned temporal information across frames of a video which is vital for expressing the motion and dynamics. In this paper, we propose to adapt a image-level blended latent diffusion model to perform local video editing tasks. Specifically, we leverage DDIM inversion to acquire the latents as background latents instead of the randomly noised ones to better preserve the background information of the input video. We further introduce an autonomous mask manufacture mechanism derived from cross-attention maps in diffusion steps. Finally, we enhance the temporal consistency across video frames by transforming the self-attention blocks of U-Net into temporal-spatial blocks. Through extensive experiments, our proposed approach demonstrates effectiveness in different real-world video editing tasks.

Title: FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary Segmentation

• Authors: Xi Chen, Haosen Yang, Sheng Jin, Xiatian Zhu, Hongxun Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03525
• Pdf URL: https://arxiv.org/pdf/2409.03525
• Copy Paste: [[2409.03525]] FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary Segmentation(https://arxiv.org/abs/2409.03525)
• Keywords: foundation model
• Abstract: Open-vocabulary segmentation poses significant challenges, as it requires segmenting and recognizing objects across an open set of categories in unconstrained environments. Building on the success of powerful vision-language (ViL) foundation models, such as CLIP, recent efforts sought to harness their zero-short capabilities to recognize unseen categories. Despite notable performance improvements, these models still encounter the critical issue of generating precise mask proposals for unseen categories and scenarios, resulting in inferior segmentation performance eventually. To address this challenge, we introduce a novel approach, FrozenSeg, designed to integrate spatial knowledge from a localization foundation model (e.g., SAM) and semantic knowledge extracted from a ViL model (e.g., CLIP), in a synergistic framework. Taking the ViL model's visual encoder as the feature backbone, we inject the space-aware feature into the learnable queries and CLIP features within the transformer decoder. In addition, we devise a mask proposal ensemble strategy for further improving the recall rate and mask quality. To fully exploit pre-trained knowledge while minimizing training overhead, we freeze both foundation models, focusing optimization efforts solely on a lightweight transformer decoder for mask proposal generation-the performance bottleneck. Extensive experiments demonstrate that FrozenSeg advances state-of-the-art results across various segmentation benchmarks, trained exclusively on COCO panoptic data, and tested in a zero-shot manner. Code is available at this https URL.

Title: Risk-based Calibration for Probabilistic Classifiers

• Authors: Aritz Pérez, Carlos Echegoyen, Guzmán Santafé
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03542
• Pdf URL: https://arxiv.org/pdf/2409.03542
• Copy Paste: [[2409.03542]] Risk-based Calibration for Probabilistic Classifiers(https://arxiv.org/abs/2409.03542)
• Keywords: generative
• Abstract: We introduce a general iterative procedure called risk-based calibration (RC) designed to minimize the empirical risk under the 0-1 loss (empirical error) for probabilistic classifiers. These classifiers are based on modeling probability distributions, including those constructed from the joint distribution (generative) and those based on the class conditional distribution (conditional). RC can be particularized to any probabilistic classifier provided a specific learning algorithm that computes the classifier's parameters in closed form using data statistics. RC reinforces the statistics aligned with the true class while penalizing those associated with other classes, guided by the 0-1 loss. The proposed method has been empirically tested on 30 datasets using naïve Bayes, quadratic discriminant analysis, and logistic regression classifiers. RC improves the empirical error of the original closed-form learning algorithms and, more notably, consistently outperforms the gradient descent approach with the three classifiers.

Title: DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture

• Authors: Qianlong Xiang, Miao Zhang, Yuzhang Shang, Jianlong Wu, Yan Yan, Liqiang Nie
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03550
• Pdf URL: https://arxiv.org/pdf/2409.03550
• Copy Paste: [[2409.03550]] DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture(https://arxiv.org/abs/2409.03550)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DMs) have demonstrated exceptional generative capabilities across various areas, while they are hindered by slow inference speeds and high computational demands during deployment. The most common way to accelerate DMs involves reducing the number of denoising steps during generation, achieved through faster sampling solvers or knowledge distillation (KD). In contrast to prior approaches, we propose a novel method that transfers the capability of large pretrained DMs to faster architectures. Specifically, we employ KD in a distinct manner to compress DMs by distilling their generative ability into more rapid variants. Furthermore, considering that the source data is either unaccessible or too enormous to store for current generative models, we introduce a new paradigm for their distillation without source data, termed Data-Free Knowledge Distillation for Diffusion Models (DKDM). Generally, our established DKDM framework comprises two main components: 1) a DKDM objective that uses synthetic denoising data produced by pretrained DMs to optimize faster DMs without source data, and 2) a dynamic iterative distillation method that flexibly organizes the synthesis of denoising data, preventing it from slowing down the optimization process as the generation is slow. To our knowledge, this is the first attempt at using KD to distill DMs into any architecture in a data-free manner. Importantly, our DKDM is orthogonal to most existing acceleration methods, such as denoising step reduction, quantization and pruning. Experiments show that our DKDM is capable of deriving 2x faster DMs with performance remaining on par with the baseline. Notably, our DKDM enables pretrained DMs to function as "datasets" for training new DMs.

Title: Organized Grouped Discrete Representation for Object-Centric Learning

• Authors: Rongzhen Zhao, Vivienne Wang, Juho Kannala, Joni Pajarinen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03553
• Pdf URL: https://arxiv.org/pdf/2409.03553
• Copy Paste: [[2409.03553]] Organized Grouped Discrete Representation for Object-Centric Learning(https://arxiv.org/abs/2409.03553)
• Keywords: diffusion
• Abstract: Object-Centric Learning (OCL) represents dense image or video pixels as sparse object features. Representative methods utilize discrete representation composed of Variational Autoencoder (VAE) template features to suppress pixel-level information redundancy and guide object-level feature aggregation. The most recent advancement, Grouped Discrete Representation (GDR), further decomposes these template features into attributes. However, its naive channel grouping as decomposition may erroneously group channels belonging to different attributes together and discretize them as sub-optimal template attributes, which losses information and harms expressivity. We propose Organized GDR (OGDR) to organize channels belonging to the same attributes together for correct decomposition from features into attributes. In unsupervised segmentation experiments, OGDR is fully superior to GDR in augmentating classical transformer-based OCL methods; it even improves state-of-the-art diffusion-based ones. Codebook PCA and representation similarity analyses show that compared with GDR, our OGDR eliminates redundancy and preserves information better for guiding object representation learning. The source code is available in the supplementary material.

Title: TCDiff: Triple Condition Diffusion Model with 3D Constraints for Stylizing Synthetic Faces

• Authors: Bernardo Biesseck, Pedro Vidal, Luiz Coelho, Roger Granada, David Menotti|
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03600
• Pdf URL: https://arxiv.org/pdf/2409.03600
• Copy Paste: [[2409.03600]] TCDiff: Triple Condition Diffusion Model with 3D Constraints for Stylizing Synthetic Faces(https://arxiv.org/abs/2409.03600)
• Keywords: diffusion
• Abstract: A robust face recognition model must be trained using datasets that include a large number of subjects and numerous samples per subject under varying conditions (such as pose, expression, age, noise, and occlusion). Due to ethical and privacy concerns, large-scale real face datasets have been discontinued, such as MS1MV3, and synthetic face generators have been proposed, utilizing GANs and Diffusion Models, such as SYNFace, SFace, DigiFace-1M, IDiff-Face, DCFace, and GANDiffFace, aiming to supply this demand. Some of these methods can produce high-fidelity realistic faces, but with low intra-class variance, while others generate high-variance faces with low identity consistency. In this paper, we propose a Triple Condition Diffusion Model (TCDiff) to improve face style transfer from real to synthetic faces through 2D and 3D facial constraints, enhancing face identity consistency while keeping the necessary high intra-class variance. Face recognition experiments using 1k, 2k, and 5k classes of our new dataset for training outperform state-of-the-art synthetic datasets in real face benchmarks such as LFW, CFP-FP, AgeDB, and BUPT. Our source code is available at: this https URL.

Title: SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local Editing

• Authors: Lingyu Xiong, Xize Cheng, Jintao Tan, Xianjia Wu, Xiandong Li, Lei Zhu, Fei Ma, Minglei Li, Huang Xu, Zhihu Hu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.03605
• Pdf URL: https://arxiv.org/pdf/2409.03605
• Copy Paste: [[2409.03605]] SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local Editing(https://arxiv.org/abs/2409.03605)
• Keywords: generative
• Abstract: Audio-driven talking face generation aims to synthesize video with lip movements synchronized to input audio. However, current generative techniques face challenges in preserving intricate regional textures (skin, teeth). To address the aforementioned challenges, we propose a novel framework called SegTalker to decouple lip movements and image textures by introducing segmentation as intermediate representation. Specifically, given the mask of image employed by a parsing network, we first leverage the speech to drive the mask and generate talking segmentation. Then we disentangle semantic regions of image into style codes using a mask-guided encoder. Ultimately, we inject the previously generated talking segmentation and style codes into a mask-guided StyleGAN to synthesize video frame. In this way, most of textures are fully preserved. Moreover, our approach can inherently achieve background separation and facilitate mask-guided facial local editing. In particular, by editing the mask and swapping the region textures from a given reference image (e.g. hair, lip, eyebrows), our approach enables facial editing seamlessly when generating talking face video. Experiments demonstrate that our proposed approach can effectively preserve texture details and generate temporally consistent video while remaining competitive in lip synchronization. Quantitative and qualitative results on the HDTF and MEAD datasets illustrate the superior performance of our method over existing methods.

Title: VFLGAN-TS: Vertical Federated Learning-based Generative Adversarial Networks for Publication of Vertically Partitioned Time-Series Data

• Authors: Xun Yuan, Zilong Zhao, Prosanta Gope, Biplab Sikdar
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.03612
• Pdf URL: https://arxiv.org/pdf/2409.03612
• Copy Paste: [[2409.03612]] VFLGAN-TS: Vertical Federated Learning-based Generative Adversarial Networks for Publication of Vertically Partitioned Time-Series Data(https://arxiv.org/abs/2409.03612)
• Keywords: generative
• Abstract: In the current artificial intelligence (AI) era, the scale and quality of the dataset play a crucial role in training a high-quality AI model. However, often original data cannot be shared due to privacy concerns and regulations. A potential solution is to release a synthetic dataset with a similar distribution to the private dataset. Nevertheless, in some scenarios, the attributes required to train an AI model are distributed among different parties, and the parties cannot share the local data for synthetic data construction due to privacy regulations. In PETS 2024, we recently introduced the first Vertical Federated Learning-based Generative Adversarial Network (VFLGAN) for publishing vertically partitioned static data. However, VFLGAN cannot effectively handle time-series data, presenting both temporal and attribute dimensions. In this article, we proposed VFLGAN-TS, which combines the ideas of attribute discriminator and vertical federated learning to generate synthetic time-series data in the vertically partitioned scenario. The performance of VFLGAN-TS is close to that of its counterpart, which is trained in a centralized manner and represents the upper limit for VFLGAN-TS. To further protect privacy, we apply a Gaussian mechanism to make VFLGAN-TS satisfy an $(\epsilon,\delta)$-differential privacy. Besides, we develop an enhanced privacy auditing scheme to evaluate the potential privacy breach through the framework of VFLGAN-TS and synthetic datasets.

Title: RealisHuman: A Two-Stage Approach for Refining Malformed Human Parts in Generated Images

• Authors: Benzhi Wang, Jingkai Zhou, Jingqi Bai, Yang Yang, Weihua Chen, Fan Wang, Zhen Lei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03644
• Pdf URL: https://arxiv.org/pdf/2409.03644
• Copy Paste: [[2409.03644]] RealisHuman: A Two-Stage Approach for Refining Malformed Human Parts in Generated Images(https://arxiv.org/abs/2409.03644)
• Keywords: diffusion, generative
• Abstract: In recent years, diffusion models have revolutionized visual generation, outperforming traditional frameworks like Generative Adversarial Networks (GANs). However, generating images of humans with realistic semantic parts, such as hands and faces, remains a significant challenge due to their intricate structural complexity. To address this issue, we propose a novel post-processing solution named RealisHuman. The RealisHuman framework operates in two stages. First, it generates realistic human parts, such as hands or faces, using the original malformed parts as references, ensuring consistent details with the original image. Second, it seamlessly integrates the rectified human parts back into their corresponding positions by repainting the surrounding areas to ensure smooth and realistic blending. The RealisHuman framework significantly enhances the realism of human generation, as demonstrated by notable improvements in both qualitative and quantitative metrics. Code is available at this https URL.

Title: Unsupervised Anomaly Detection and Localization with Generative Adversarial Networks

• Authors: Khouloud Abdelli, Matteo Lonardi, Jurgen Gripp, Samuel Olsson, Fabien Boitier, Patricia Layec
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.03657
• Pdf URL: https://arxiv.org/pdf/2409.03657
• Copy Paste: [[2409.03657]] Unsupervised Anomaly Detection and Localization with Generative Adversarial Networks(https://arxiv.org/abs/2409.03657)
• Keywords: generative, anomaly
• Abstract: We propose a novel unsupervised anomaly detection approach using generative adversarial networks and SOP-derived spectrograms. Demonstrating remarkable efficacy, our method achieves over 97% accuracy on SOP datasets from both submarine and terrestrial fiber links, all achieved without the need for labelled data.

Title: The representation landscape of few-shot learning and fine-tuning in large language models

• Authors: Diego Doimo, Alessandro Serra, Alessio Ansuini, Alberto Cazzaniga
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03662
• Pdf URL: https://arxiv.org/pdf/2409.03662
• Copy Paste: [[2409.03662]] The representation landscape of few-shot learning and fine-tuning in large language models(https://arxiv.org/abs/2409.03662)
• Keywords: in-context
• Abstract: In-context learning (ICL) and supervised fine-tuning (SFT) are two common strategies for improving the performance of modern large language models (LLMs) on specific tasks. Despite their different natures, these strategies often lead to comparable performance gains. However, little is known about whether they induce similar representations inside LLMs. We approach this problem by analyzing the probability landscape of their hidden representations in the two cases. More specifically, we compare how LLMs solve the same question-answering task, finding that ICL and SFT create very different internal structures, in both cases undergoing a sharp transition in the middle of the network. In the first half of the network, ICL shapes interpretable representations hierarchically organized according to their semantic content. In contrast, the probability landscape obtained with SFT is fuzzier and semantically mixed. In the second half of the model, the fine-tuned representations develop probability modes that better encode the identity of answers, while the landscape of ICL representations is characterized by less defined peaks. Our approach reveals the diverse computational strategies developed inside LLMs to solve the same task across different conditions, allowing us to make a step towards designing optimal methods to extract information from language models.

Title: Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation

• Authors: Slava Elizarov, Ciara Rowles, Simon Donné
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.03718
• Pdf URL: https://arxiv.org/pdf/2409.03718
• Copy Paste: [[2409.03718]] Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation(https://arxiv.org/abs/2409.03718)
• Keywords: diffusion
• Abstract: Generating high-quality 3D objects from textual descriptions remains a challenging problem due to computational cost, the scarcity of 3D data, and complex 3D representations. We introduce Geometry Image Diffusion (GIMDiffusion), a novel Text-to-3D model that utilizes geometry images to efficiently represent 3D shapes using 2D images, thereby avoiding the need for complex 3D-aware architectures. By integrating a Collaborative Control mechanism, we exploit the rich 2D priors of existing Text-to-Image models such as Stable Diffusion. This enables strong generalization even with limited 3D training data (allowing us to use only high-quality training data) as well as retaining compatibility with guidance techniques such as IPAdapter. In short, GIMDiffusion enables the generation of 3D assets at speeds comparable to current Text-to-Image models. The generated objects consist of semantically meaningful, separate parts and include internal structures, enhancing both usability and versatility.

Title: ArtiFade: Learning to Generate High-quality Subject from Blemished Images

• Authors: Shuya Yang, Shaozhe Hao, Yukang Cao, Kwan-Yee K. Wong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03745
• Pdf URL: https://arxiv.org/pdf/2409.03745
• Copy Paste: [[2409.03745]] ArtiFade: Learning to Generate High-quality Subject from Blemished Images(https://arxiv.org/abs/2409.03745)
• Keywords: diffusion, generative
• Abstract: Subject-driven text-to-image generation has witnessed remarkable advancements in its ability to learn and capture characteristics of a subject using only a limited number of images. However, existing methods commonly rely on high-quality images for training and may struggle to generate reasonable images when the input images are blemished by artifacts. This is primarily attributed to the inadequate capability of current techniques in distinguishing subject-related features from disruptive artifacts. In this paper, we introduce ArtiFade to tackle this issue and successfully generate high-quality artifact-free images from blemished datasets. Specifically, ArtiFade exploits fine-tuning of a pre-trained text-to-image model, aiming to remove artifacts. The elimination of artifacts is achieved by utilizing a specialized dataset that encompasses both unblemished images and their corresponding blemished counterparts during fine-tuning. ArtiFade also ensures the preservation of the original generative capabilities inherent within the diffusion model, thereby enhancing the overall performance of subject-driven methods in generating high-quality and artifact-free images. We further devise evaluation benchmarks tailored for this task. Through extensive qualitative and quantitative experiments, we demonstrate the generalizability of ArtiFade in effective artifact removal under both in-distribution and out-of-distribution scenarios.

Title: Attention Heads of Large Language Models: A Survey

• Authors: Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Bo Tang, Feiyu Xiong, Zhiyu Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.03752
• Pdf URL: https://arxiv.org/pdf/2409.03752
• Copy Paste: [[2409.03752]] Attention Heads of Large Language Models: A Survey(https://arxiv.org/abs/2409.03752)
• Keywords: in-context
• Abstract: Since the advent of ChatGPT, Large Language Models (LLMs) have excelled in various tasks but remain largely as black-box systems. Consequently, their development relies heavily on data-driven approaches, limiting performance enhancement through changes in internal architecture and reasoning pathways. As a result, many researchers have begun exploring the potential internal mechanisms of LLMs, aiming to identify the essence of their reasoning bottlenecks, with most studies focusing on attention heads. Our survey aims to shed light on the internal reasoning processes of LLMs by concentrating on the interpretability and underlying mechanisms of attention heads. We first distill the human thought process into a four-stage framework: Knowledge Recalling, In-Context Identification, Latent Reasoning, and Expression Preparation. Using this framework, we systematically review existing research to identify and categorize the functions of specific attention heads. Furthermore, we summarize the experimental methodologies used to discover these special heads, dividing them into two categories: Modeling-Free methods and Modeling-Required methods. Also, we outline relevant evaluation methods and benchmarks. Finally, we discuss the limitations of current research and propose several potential future directions. Our reference list is open-sourced at \url{this https URL}.

Title: Foundation Model or Finetune? Evaluation of few-shot semantic segmentation for river pollution

• Authors: Marga Don, Stijn Pinson, Blanca Guillen Cebrian, Yuki M. Asano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03754
• Pdf URL: https://arxiv.org/pdf/2409.03754
• Copy Paste: [[2409.03754]] Foundation Model or Finetune? Evaluation of few-shot semantic segmentation for river pollution(https://arxiv.org/abs/2409.03754)
• Keywords: foundation model
• Abstract: Foundation models (FMs) are a popular topic of research in AI. Their ability to generalize to new tasks and datasets without retraining or needing an abundance of data makes them an appealing candidate for applications on specialist datasets. In this work, we compare the performance of FMs to finetuned pre-trained supervised models in the task of semantic segmentation on an entirely new dataset. We see that finetuned models consistently outperform the FMs tested, even in cases were data is scarce. We release the code and dataset for this work on GitHub.

Title: DC-Solver: Improving Predictor-Corrector Diffusion Sampler via Dynamic Compensation

• Authors: Wenliang Zhao, Haolin Wang, Jie Zhou, Jiwen Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03755
• Pdf URL: https://arxiv.org/pdf/2409.03755
• Copy Paste: [[2409.03755]] DC-Solver: Improving Predictor-Corrector Diffusion Sampler via Dynamic Compensation(https://arxiv.org/abs/2409.03755)
• Keywords: diffusion
• Abstract: Diffusion probabilistic models (DPMs) have shown remarkable performance in visual synthesis but are computationally expensive due to the need for multiple evaluations during the sampling. Recent predictor-corrector diffusion samplers have significantly reduced the required number of function evaluations (NFE), but inherently suffer from a misalignment issue caused by the extra corrector step, especially with a large classifier-free guidance scale (CFG). In this paper, we introduce a new fast DPM sampler called DC-Solver, which leverages dynamic compensation (DC) to mitigate the misalignment of the predictor-corrector samplers. The dynamic compensation is controlled by compensation ratios that are adaptive to the sampling steps and can be optimized on only 10 datapoints by pushing the sampling trajectory toward a ground truth trajectory. We further propose a cascade polynomial regression (CPR) which can instantly predict the compensation ratios on unseen sampling configurations. Additionally, we find that the proposed dynamic compensation can also serve as a plug-and-play module to boost the performance of predictor-only samplers. Extensive experiments on both unconditional sampling and conditional sampling demonstrate that our DC-Solver can consistently improve the sampling quality over previous methods on different DPMs with a wide range of resolutions up to 1024$\times$1024. Notably, we achieve 10.38 FID (NFE=5) on unconditional FFHQ and 0.394 MSE (NFE=5, CFG=7.5) on Stable-Diffusion-2.1. Code is available at this https URL

Title: Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding

• Authors: Yunze Man, Shuhong Zheng, Zhipeng Bao, Martial Hebert, Liang-Yan Gui, Yu-Xiong Wang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.03757
• Pdf URL: https://arxiv.org/pdf/2409.03757
• Copy Paste: [[2409.03757]] Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding(https://arxiv.org/abs/2409.03757)
• Keywords: diffusion, foundation model
• Abstract: Complex 3D scene understanding has gained increasing attention, with scene encoding strategies playing a crucial role in this success. However, the optimal scene encoding strategies for various scenarios remain unclear, particularly compared to their image-based counterparts. To address this issue, we present a comprehensive study that probes various visual encoding models for 3D scene understanding, identifying the strengths and limitations of each model across different scenarios. Our evaluation spans seven vision foundation encoders, including image-based, video-based, and 3D foundation models. We evaluate these models in four tasks: Vision-Language Scene Reasoning, Visual Grounding, Segmentation, and Registration, each focusing on different aspects of scene understanding. Our evaluations yield key findings: DINOv2 demonstrates superior performance, video models excel in object-level tasks, diffusion models benefit geometric tasks, and language-pretrained models show unexpected limitations in language-related tasks. These insights challenge some conventional understandings, provide novel perspectives on leveraging visual foundation models, and highlight the need for more flexible encoder selection in future vision-language and scene-understanding tasks.