2024-09-09

Title: A Dataset for Mechanical Mechanisms

• Authors: Farshid Ghezelbash, Amir Hossein Eskandari, Amir J Bidhendi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03763
• Pdf URL: https://arxiv.org/pdf/2409.03763
• Copy Paste: [[2409.03763]] A Dataset for Mechanical Mechanisms(https://arxiv.org/abs/2409.03763)
• Keywords: diffusion, generative
• Abstract: This study introduces a dataset consisting of approximately 9,000 images of mechanical mechanisms and their corresponding descriptions, aimed at supporting research in mechanism design. The dataset consists of a diverse collection of 2D and 3D sketches, meticulously curated to ensure relevance and quality. We demonstrate the application of this dataset by fine-tuning two models: 1) Stable Diffusion (for generating new mechanical designs), and 2) BLIP-2 (for captioning these designs). While the results from Stable Diffusion show promise, particularly in generating coherent 3D sketches, the model struggles with 2D sketches and occasionally produces nonsensical outputs. These limitations underscore the need for further development, particularly in expanding the dataset and refining model architectures. Nonetheless, this work serves as a step towards leveraging generative AI in mechanical design, highlighting both the potential and current limitations of these approaches.

Title: Assessing the Uncertainty and Robustness of Object Detection Models for Detecting Stickers on Laptops

• Authors: Chengjie Lu, Jiahui Wu, Shaukat Ali, Mikkel Labori Olsen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03782
• Pdf URL: https://arxiv.org/pdf/2409.03782
• Copy Paste: [[2409.03782]] Assessing the Uncertainty and Robustness of Object Detection Models for Detecting Stickers on Laptops(https://arxiv.org/abs/2409.03782)
• Keywords: diffusion
• Abstract: Refurbishing laptops extends their lives while contributing to reducing electronic waste, which promotes building a sustainable future. To this end, the Danish Technological Institute (DTI) focuses on the research and development of several applications, including laptop refurbishing. This has several steps, including cleaning, which involves identifying and removing stickers from laptop surfaces. DTI trained six sticker detection models (SDMs) based on open-source object detection models to identify such stickers precisely so these stickers can be removed automatically. However, given the diversity in types of stickers (e.g., shapes, colors, locations), identification of the stickers is highly uncertain, thereby requiring explicit quantification of uncertainty associated with the identified stickers. Such uncertainty quantification can help reduce risks in removing stickers, which, for example, could otherwise result in damaging laptop surfaces. For uncertainty quantification, we adopted the Monte Carlo Dropout method to evaluate the six SDMs from DTI using three datasets: the original image dataset from DTI and two datasets generated with vision language models, i.e., DALL-E-3 and Stable Diffusion-3. In addition, we presented novel robustness metrics concerning detection accuracy and uncertainty to assess the robustness of the SDMs based on adversarial datasets generated from the three datasets using a dense adversary method. Our evaluation results show that different SDMs perform differently regarding different metrics. Based on the results, we provide SDM selection guidelines and lessons learned from various perspectives.

Title: Protecting Activity Sensing Data Privacy Using Hierarchical Information Dissociation

• Authors: Guangjing Wang, Hanqing Guo, Yuanda Wang, Bocheng Chen, Ce Zhou, Qiben Yan
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03796
• Pdf URL: https://arxiv.org/pdf/2409.03796
• Copy Paste: [[2409.03796]] Protecting Activity Sensing Data Privacy Using Hierarchical Information Dissociation(https://arxiv.org/abs/2409.03796)
• Keywords: diffusion
• Abstract: Smartphones and wearable devices have been integrated into our daily lives, offering personalized services. However, many apps become overprivileged as their collected sensing data contains unnecessary sensitive information. For example, mobile sensing data could reveal private attributes (e.g., gender and age) and unintended sensitive features (e.g., hand gestures when entering passwords). To prevent sensitive information leakage, existing methods must obtain private labels and users need to specify privacy policies. However, they only achieve limited control over information disclosure. In this work, we present Hippo to dissociate hierarchical information including private metadata and multi-grained activity information from the sensing data. Hippo achieves fine-grained control over the disclosure of sensitive information without requiring private labels. Specifically, we design a latent guidance-based diffusion model, which generates multi-grained versions of raw sensor data conditioned on hierarchical latent activity features. Hippo enables users to control the disclosure of sensitive information in sensing data, ensuring their privacy while preserving the necessary features to meet the utility requirements of applications. Hippo is the first unified model that achieves two goals: perturbing the sensitive attributes and controlling the disclosure of sensitive information in mobile sensing data. Extensive experiments show that Hippo can anonymize personal attributes and transform activity information at various resolutions across different types of sensing data.

Title: Interpretable Cyber Threat Detection for Enterprise Industrial Networks: A Computational Design Science Approach

• Authors: Prabhat Kumar, A.K.M. Najmul Islam
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.03798
• Pdf URL: https://arxiv.org/pdf/2409.03798
• Copy Paste: [[2409.03798]] Interpretable Cyber Threat Detection for Enterprise Industrial Networks: A Computational Design Science Approach(https://arxiv.org/abs/2409.03798)
• Keywords: generative
• Abstract: Enterprise industrial networks face threats that risk data and operations. However, designing efficient threat detection system is challenging due to data scarcity, especially where privacy is a concern. The complexity of enterprise industrial network data adds to this challenge, causing high false positives and interpretation issues. Towards this, we use IS computational design science paradigm to develop a two-stage cyber threat detection system for enterprise-level IS that are both secure and capable of adapting to evolving technological and business environments. The first stage generates synthetic industrial network data using a modified generative adversarial network. The second stage develops a novel bidirectional gated recurrent unit and a modified attention mechanism for effective threat detection. We also use shapley additive explanations and a decision tree technique for enhancing interpretability. Our analysis on two public datasets shows the frameworks high precision in threat detection and offers practical cybersecurity solutions and methodological advancements.

Title: Neural Entropy

• Authors: Akhil Premkumar
• Subjects: cs.LG, cond-mat.stat-mech, cs.IT
• Abstract URL: https://arxiv.org/abs/2409.03817
• Pdf URL: https://arxiv.org/pdf/2409.03817
• Copy Paste: [[2409.03817]] Neural Entropy(https://arxiv.org/abs/2409.03817)
• Keywords: diffusion, generative
• Abstract: We examine the connection between deep learning and information theory through the paradigm of diffusion models. Using well-established principles from non-equilibrium thermodynamics we can characterize the amount of information required to reverse a diffusive process. Neural networks store this information and operate in a manner reminiscent of Maxwell's demon during the generative stage. We illustrate this cycle using a novel diffusion scheme we call the entropy matching model, wherein the information conveyed to the network during training exactly corresponds to the entropy that must be negated during reversal. We demonstrate that this entropy can be used to analyze the encoding efficiency and storage capacity of the network. This conceptual picture blends elements of stochastic optimal control, thermodynamics, information theory, and optimal transport, and raises the prospect of applying diffusion models as a test bench to understand neural networks.

Title: Few-shot Adaptation of Medical Vision-Language Models

• Authors: Fereshteh Shakeri, Yunshi Huang, Julio Silva-Rodríguez, Houda Bahig, An Tang, Jose Dolz, Ismail Ben Ayed
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03868
• Pdf URL: https://arxiv.org/pdf/2409.03868
• Copy Paste: [[2409.03868]] Few-shot Adaptation of Medical Vision-Language Models(https://arxiv.org/abs/2409.03868)
• Keywords: foundation model
• Abstract: Integrating image and text data through multi-modal learning has emerged as a new approach in medical imaging research, following its successful deployment in computer vision. While considerable efforts have been dedicated to establishing medical foundation models and their zero-shot transfer to downstream tasks, the popular few-shot setting remains relatively unexplored. Following on from the currently strong emergence of this setting in computer vision, we introduce the first structured benchmark for adapting medical vision-language models (VLMs) in a strict few-shot regime and investigate various adaptation strategies commonly used in the context of natural images. Furthermore, we evaluate a simple generalization of the linear-probe adaptation baseline, which seeks an optimal blending of the visual prototypes and text embeddings via learnable class-wise multipliers. Surprisingly, such a text-informed linear probe yields competitive performances in comparison to convoluted prompt-learning and adapter-based strategies, while running considerably faster and accommodating the black-box setting. Our extensive experiments span three different medical modalities and specialized foundation models, nine downstream tasks, and several state-of-the-art few-shot adaptation methods. We made our benchmark and code publicly available to trigger further developments in this emergent subject: \url{this https URL}.

Title: CACER: Clinical Concept Annotations for Cancer Events and Relations

• Authors: Yujuan Fu, Giridhar Kaushik Ramachandran, Ahmad Halwani, Bridget T. McInnes, Fei Xia, Kevin Lybarger, Meliha Yetisgen, Özlem Uzuner
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.03905
• Pdf URL: https://arxiv.org/pdf/2409.03905
• Copy Paste: [[2409.03905]] CACER: Clinical Concept Annotations for Cancer Events and Relations(https://arxiv.org/abs/2409.03905)
• Keywords: in-context
• Abstract: Clinical notes contain unstructured representations of patient histories, including the relationships between medical problems and prescription drugs. To investigate the relationship between cancer drugs and their associated symptom burden, we extract structured, semantic representations of medical problem and drug information from the clinical narratives of oncology notes. We present Clinical Concept Annotations for Cancer Events and Relations (CACER), a novel corpus with fine-grained annotations for over 48,000 medical problems and drug events and 10,000 drug-problem and problem-problem relations. Leveraging CACER, we develop and evaluate transformer-based information extraction (IE) models such as BERT, Flan-T5, Llama3, and GPT-4 using fine-tuning and in-context learning (ICL). In event extraction, the fine-tuned BERT and Llama3 models achieved the highest performance at 88.2-88.0 F1, which is comparable to the inter-annotator agreement (IAA) of 88.4 F1. In relation extraction, the fine-tuned BERT, Flan-T5, and Llama3 achieved the highest performance at 61.8-65.3 F1. GPT-4 with ICL achieved the worst performance across both tasks. The fine-tuned models significantly outperformed GPT-4 in ICL, highlighting the importance of annotated training data and model optimization. Furthermore, the BERT models performed similarly to Llama3. For our task, LLMs offer no performance advantage over the smaller BERT models. The results emphasize the need for annotated training data to optimize models. Multiple fine-tuned transformer models achieved performance comparable to IAA for several extraction tasks.

Title: The Role of Generative Systems in Historical Photography Management: A Case Study on Catalan Archives

• Authors: Èric Śanchez, Adrià Molina, Oriol Ramos Terrades
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03911
• Pdf URL: https://arxiv.org/pdf/2409.03911
• Copy Paste: [[2409.03911]] The Role of Generative Systems in Historical Photography Management: A Case Study on Catalan Archives(https://arxiv.org/abs/2409.03911)
• Keywords: generative
• Abstract: The use of image analysis in automated photography management is an increasing trend in heritage institutions. Such tools alleviate the human cost associated with the manual and expensive annotation of new data sources while facilitating fast access to the citizenship through online indexes and search engines. However, available tagging and description tools are usually designed around modern photographs in English, neglecting historical corpora in minoritized languages, each of which exhibits intrinsic particularities. The primary objective of this research is to study the quantitative contribution of generative systems in the description of historical sources. This is done by contextualizing the task of captioning historical photographs from the Catalan archives as a case study. Our findings provide practitioners with tools and directions on transfer learning for captioning models based on visual adaptation and linguistic proximity.

Title: Data-Efficient Generation for Dataset Distillation

• Authors: Zhe Li, Weitong Zhang, Sarah Cechnicka, Bernhard Kainz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03929
• Pdf URL: https://arxiv.org/pdf/2409.03929
• Copy Paste: [[2409.03929]] Data-Efficient Generation for Dataset Distillation(https://arxiv.org/abs/2409.03929)
• Keywords: diffusion
• Abstract: While deep learning techniques have proven successful in image-related tasks, the exponentially increased data storage and computation costs become a significant challenge. Dataset distillation addresses these challenges by synthesizing only a few images for each class that encapsulate all essential information. Most current methods focus on matching. The problems lie in the synthetic images not being human-readable and the dataset performance being insufficient for downstream learning tasks. Moreover, the distillation time can quickly get out of bounds when the number of synthetic images per class increases even slightly. To address this, we train a class conditional latent diffusion model capable of generating realistic synthetic images with labels. The sampling time can be reduced to several tens of images per seconds. We demonstrate that models can be effectively trained using only a small set of synthetic images and evaluated on a large real test set. Our approach achieved rank \(1\) in The First Dataset Distillation Challenge at ECCV 2024 on the CIFAR100 and TinyImageNet datasets.

Title: HUMOS: Human Motion Model Conditioned on Body Shape

• Authors: Shashank Tripathi, Omid Taheri, Christoph Lassner, Michael J. Black, Daniel Holden, Carsten Stoll
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03944
• Pdf URL: https://arxiv.org/pdf/2409.03944
• Copy Paste: [[2409.03944]] HUMOS: Human Motion Model Conditioned on Body Shape(https://arxiv.org/abs/2409.03944)
• Keywords: generative
• Abstract: Generating realistic human motion is essential for many computer vision and graphics applications. The wide variety of human body shapes and sizes greatly impacts how people move. However, most existing motion models ignore these differences, relying on a standardized, average body. This leads to uniform motion across different body types, where movements don't match their physical characteristics, limiting diversity. To solve this, we introduce a new approach to develop a generative motion model based on body shape. We show that it's possible to train this model using unpaired data by applying cycle consistency, intuitive physics, and stability constraints, which capture the relationship between identity and movement. The resulting model generates diverse, physically plausible, and dynamically stable human motions that are both quantitatively and qualitatively more realistic than current state-of-the-art methods. More details are available on our project page this https URL.

Title: Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance

• Authors: RenMing Huang, Shaochong Liu, Yunqiang Pei, Peng Wang, Guoqing Wang, Yang Yang, Hengtao Shen
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.03996
• Pdf URL: https://arxiv.org/pdf/2409.03996
• Copy Paste: [[2409.03996]] Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance(https://arxiv.org/abs/2409.03996)
• Keywords: diffusion
• Abstract: In this work, we address the challenging problem of long-horizon goal-reaching policy learning from non-expert, action-free observation data. Unlike fully labeled expert data, our data is more accessible and avoids the costly process of action labeling. Additionally, compared to online learning, which often involves aimless exploration, our data provides useful guidance for more efficient exploration. To achieve our goal, we propose a novel subgoal guidance learning strategy. The motivation behind this strategy is that long-horizon goals offer limited guidance for efficient exploration and accurate state transition. We develop a diffusion strategy-based high-level policy to generate reasonable subgoals as waypoints, preferring states that more easily lead to the final goal. Additionally, we learn state-goal value functions to encourage efficient subgoal reaching. These two components naturally integrate into the off-policy actor-critic framework, enabling efficient goal attainment through informative exploration. We evaluate our method on complex robotic navigation and manipulation tasks, demonstrating a significant performance advantage over existing methods. Our ablation study further shows that our method is robust to observation data with various corruptions.

Title: DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes

• Authors: Jianbiao Mei, Yukai Ma, Xuemeng Yang, Licheng Wen, Tiantian Wei, Min Dou, Botian Shi, Yong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04003
• Pdf URL: https://arxiv.org/pdf/2409.04003
• Copy Paste: [[2409.04003]] DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes(https://arxiv.org/abs/2409.04003)
• Keywords: diffusion
• Abstract: Recent advances in diffusion models have significantly enhanced the cotrollable generation of streetscapes for and facilitated downstream perception and planning tasks. However, challenges such as maintaining temporal coherence, generating long videos, and accurately modeling driving scenes persist. Accordingly, we propose DreamForge, an advanced diffusion-based autoregressive video generation model designed for the long-term generation of 3D-controllable and extensible video. In terms of controllability, our DreamForge supports flexible conditions such as text descriptions, camera poses, 3D bounding boxes, and road layouts, while also providing perspective guidance to produce driving scenes that are both geometrically and contextually accurate. For consistency, we ensure inter-view consistency through cross-view attention and temporal coherence via an autoregressive architecture enhanced with motion cues. Codes will be available at this https URL.

Title: One-Shot Diffusion Mimicker for Handwritten Text Generation

• Authors: Gang Dai, Yifan Zhang, Quhui Ke, Qiangya Guo, Shuangping Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04004
• Pdf URL: https://arxiv.org/pdf/2409.04004
• Copy Paste: [[2409.04004]] One-Shot Diffusion Mimicker for Handwritten Text Generation(https://arxiv.org/abs/2409.04004)
• Keywords: diffusion
• Abstract: Existing handwritten text generation methods often require more than ten handwriting samples as style references. However, in practical applications, users tend to prefer a handwriting generation model that operates with just a single reference sample for its convenience and efficiency. This approach, known as "one-shot generation", significantly simplifies the process but poses a significant challenge due to the difficulty of accurately capturing a writer's style from a single sample, especially when extracting fine details from the characters' edges amidst sparse foreground and undesired background noise. To address this problem, we propose a One-shot Diffusion Mimicker (One-DM) to generate handwritten text that can mimic any calligraphic style with only one reference sample. Inspired by the fact that high-frequency information of the individual sample often contains distinct style patterns (e.g., character slant and letter joining), we develop a novel style-enhanced module to improve the style extraction by incorporating high-frequency components from a single sample. We then fuse the style features with the text content as a merged condition for guiding the diffusion model to produce high-quality handwritten text images. Extensive experiments demonstrate that our method can successfully generate handwriting scripts with just one sample reference in multiple languages, even outperforming previous methods using over ten samples. Our source code is available at this https URL.

Title: Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens for Text-to-Any-Task

• Authors: Jing Wang, Ao Ma, Jiasong Feng, Dawei Leng, Yuhui Yin, Xiaodan Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04005
• Pdf URL: https://arxiv.org/pdf/2409.04005
• Copy Paste: [[2409.04005]] Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens for Text-to-Any-Task(https://arxiv.org/abs/2409.04005)
• Keywords: diffusion
• Abstract: The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity. To address this redundancy, we propose the Proxy Token Diffusion Transformer (PT-DiT), which employs sparse representative token attention (where the number of representative tokens is much smaller than the total number of tokens) to model global visual information efficiently. Specifically, in each transformer block, we randomly sample one token from each spatial-temporal window to serve as a proxy token for that region. The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention. Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism. Building on the well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a variety of models for T2I, T2V, and T2MV tasks. Experimental results show that PT-DiT achieves competitive performance while reducing the computational complexity in both image and video generation tasks (e.g., a 48% reduction compared to DiT and a 35% reduction compared to Pixart-alpha). Our source code is available at this https URL.

Title: Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamics

• Authors: Woojin Cho, Jihyun Lee, Minjae Yi, Minje Kim, Taeyun Woo, Donghwan Kim, Taewook Ha, Hyokeun Lee, Je-Hwan Ryu, Woontack Woo, Tae-Kyun Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04033
• Pdf URL: https://arxiv.org/pdf/2409.04033
• Copy Paste: [[2409.04033]] Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamics(https://arxiv.org/abs/2409.04033)
• Keywords: foundation model
• Abstract: Existing datasets for 3D hand-object interaction are limited either in the data cardinality, data variations in interaction scenarios, or the quality of annotations. In this work, we present a comprehensive new training dataset for hand-object interaction called HOGraspNet. It is the only real dataset that captures full grasp taxonomies, providing grasp annotation and wide intraclass variations. Using grasp taxonomies as atomic actions, their space and time combinatorial can represent complex hand activities around objects. We select 22 rigid objects from the YCB dataset and 8 other compound objects using shape and size taxonomies, ensuring coverage of all hand grasp configurations. The dataset includes diverse hand shapes from 99 participants aged 10 to 74, continuous video frames, and a 1.5M RGB-Depth of sparse frames with annotations. It offers labels for 3D hand and object meshes, 3D keypoints, contact maps, and \emph{grasp labels}. Accurate hand and object 3D meshes are obtained by fitting the hand parametric model (MANO) and the hand implicit function (HALO) to multi-view RGBD frames, with the MoCap system only for objects. Note that HALO fitting does not require any parameter tuning, enabling scalability to the dataset's size with comparable accuracy to MANO. We evaluate HOGraspNet on relevant tasks: grasp classification and 3D hand pose estimation. The result shows performance variations based on grasp type and object class, indicating the potential importance of the interaction space captured by our dataset. The provided data aims at learning universal shape priors or foundation models for 3D hand-object interaction. Our dataset and code are available at this https URL.

Title: Towards Safer Online Spaces: Simulating and Assessing Intervention Strategies for Eating Disorder Discussions

• Authors: Louis Penafiel, Hsien-Te Kao, Isabel Erickson, David Chu, Robert McCormack, Kristina Lerman, Svitlana Volkova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04043
• Pdf URL: https://arxiv.org/pdf/2409.04043
• Copy Paste: [[2409.04043]] Towards Safer Online Spaces: Simulating and Assessing Intervention Strategies for Eating Disorder Discussions(https://arxiv.org/abs/2409.04043)
• Keywords: generative
• Abstract: Eating disorders are complex mental health conditions that affect millions of people around the world. Effective interventions on social media platforms are crucial, yet testing strategies in situ can be risky. We present a novel LLM-driven experimental testbed for simulating and assessing intervention strategies in ED-related discussions. Our framework generates synthetic conversations across multiple platforms, models, and ED-related topics, allowing for controlled experimentation with diverse intervention approaches. We analyze the impact of various intervention strategies on conversation dynamics across four dimensions: intervention type, generative model, social media platform, and ED-related community/topic. We employ cognitive domain analysis metrics, including sentiment, emotions, etc., to evaluate the effectiveness of interventions. Our findings reveal that civility-focused interventions consistently improve positive sentiment and emotional tone across all dimensions, while insight-resetting approaches tend to increase negative emotions. We also uncover significant biases in LLM-generated conversations, with cognitive metrics varying notably between models (Claude-3 Haiku $>$ Mistral $>$ GPT-3.5-turbo $>$ LLaMA3) and even between versions of the same model. These variations highlight the importance of model selection in simulating realistic discussions related to ED. Our work provides valuable information on the complex dynamics of ED-related discussions and the effectiveness of various intervention strategies.

Title: D4: Text-guided diffusion model-based domain adaptive data augmentation for vineyard shoot detection

• Authors: Kentaro Hirahara, Chikahito Nakane, Hajime Ebisawa, Tsuyoshi Kuroda, Yohei Iwaki, Tomoyoshi Utsumi, Yuichiro Nomura, Makoto Koike, Hiroshi Mineno
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04060
• Pdf URL: https://arxiv.org/pdf/2409.04060
• Copy Paste: [[2409.04060]] D4: Text-guided diffusion model-based domain adaptive data augmentation for vineyard shoot detection(https://arxiv.org/abs/2409.04060)
• Keywords: diffusion, generative
• Abstract: In an agricultural field, plant phenotyping using object detection models is gaining attention. However, collecting the training data necessary to create generic and high-precision models is extremely challenging due to the difficulty of annotation and the diversity of domains. Furthermore, it is difficult to transfer training data across different crops, and although machine learning models effective for specific environments, conditions, or crops have been developed, they cannot be widely applied in actual fields. In this study, we propose a generative data augmentation method (D4) for vineyard shoot detection. D4 uses a pre-trained text-guided diffusion model based on a large number of original images culled from video data collected by unmanned ground vehicles or other means, and a small number of annotated datasets. The proposed method generates new annotated images with background information adapted to the target domain while retaining annotation information necessary for object detection. In addition, D4 overcomes the lack of training data in agriculture, including the difficulty of annotation and diversity of domains. We confirmed that this generative data augmentation method improved the mean average precision by up to 28.65% for the BBox detection task and the average precision by up to 13.73% for the keypoint detection task for vineyard shoot detection. Our generative data augmentation method D4 is expected to simultaneously solve the cost and domain diversity issues of training data generation in agriculture and improve the generalization performance of detection models.

Title: UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity

• Authors: Yicheng Fu, Raviteja Anantha, Prabal Vashisht, Jianpeng Cheng, Etai Littwin
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04081
• Pdf URL: https://arxiv.org/pdf/2409.04081
• Copy Paste: [[2409.04081]] UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity(https://arxiv.org/abs/2409.04081)
• Keywords: self-supervised
• Abstract: Generating user intent from a sequence of user interface (UI) actions is a core challenge in comprehensive UI understanding. Recent advancements in multimodal large language models (MLLMs) have led to substantial progress in this area, but their demands for extensive model parameters, computing power, and high latency makes them impractical for scenarios requiring lightweight, on-device solutions with low latency or heightened privacy. Additionally, the lack of high-quality datasets has hindered the development of such lightweight models. To address these challenges, we propose UI-JEPA, a novel framework that employs masking strategies to learn abstract UI embeddings from unlabeled data through self-supervised learning, combined with an LLM decoder fine-tuned for user intent prediction. We also introduce two new UI-grounded multimodal datasets, "Intent in the Wild" (IIW) and "Intent in the Tame" (IIT), designed for few-shot and zero-shot UI understanding tasks. IIW consists of 1.7K videos across 219 intent categories, while IIT contains 914 videos across 10 categories. We establish the first baselines for these datasets, showing that representations learned using a JEPA-style objective, combined with an LLM decoder, can achieve user intent predictions that match the performance of state-of-the-art large MLLMs, but with significantly reduced annotation and deployment resources. Measured by intent similarity scores, UI-JEPA outperforms GPT-4 Turbo and Claude 3.5 Sonnet by 10.0% and 7.2% respectively, averaged across two datasets. Notably, UI-JEPA accomplishes the performance with a 50.5x reduction in computational cost and a 6.6x improvement in latency in the IIW dataset. These results underscore the effectiveness of UI-JEPA, highlighting its potential for lightweight, high-performance UI understanding.

Title: Context is the Key: Backdoor Attacks for In-Context Learning with Vision Transformers

• Authors: Gorka Abad, Stjepan Picek, Lorenzo Cavallaro, Aitor Urbieta
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04142
• Pdf URL: https://arxiv.org/pdf/2409.04142
• Copy Paste: [[2409.04142]] Context is the Key: Backdoor Attacks for In-Context Learning with Vision Transformers(https://arxiv.org/abs/2409.04142)
• Keywords: in-context
• Abstract: Due to the high cost of training, large model (LM) practitioners commonly use pretrained models downloaded from untrusted sources, which could lead to owning compromised models. In-context learning is the ability of LMs to perform multiple tasks depending on the prompt or context. This can enable new attacks, such as backdoor attacks with dynamic behavior depending on how models are prompted. In this paper, we leverage the ability of vision transformers (ViTs) to perform different tasks depending on the prompts. Then, through data poisoning, we investigate two new threats: i) task-specific backdoors where the attacker chooses a target task to attack, and only the selected task is compromised at test time under the presence of the trigger. At the same time, any other task is not affected, even if prompted with the trigger. We succeeded in attacking every tested model, achieving up to 89.90\% degradation on the target task. ii) We generalize the attack, allowing the backdoor to affect \emph{any} task, even tasks unseen during the training phase. Our attack was successful on every tested model, achieving a maximum of $13\times$ degradation. Finally, we investigate the robustness of prompts and fine-tuning as techniques for removing the backdoors from the model. We found that these methods fall short and, in the best case, reduce the degradation from 89.90\% to 73.46\%.

Title: GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformers

• Authors: Lorenza Prospero, Abdullah Hamdi, Joao F. Henriques, Christian Rupprecht
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04196
• Pdf URL: https://arxiv.org/pdf/2409.04196
• Copy Paste: [[2409.04196]] GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformers(https://arxiv.org/abs/2409.04196)
• Keywords: diffusion
• Abstract: Reconstructing realistic 3D human models from monocular images has significant applications in creative industries, human-computer interfaces, and healthcare. We base our work on 3D Gaussian Splatting (3DGS), a scene representation composed of a mixture of Gaussians. Predicting such mixtures for a human from a single input image is challenging, as it is a non-uniform density (with a many-to-one relationship with input pixels) with strict physical constraints. At the same time, it needs to be flexible to accommodate a variety of clothes and poses. Our key observation is that the vertices of standardized human meshes (such as SMPL) can provide an adequate density and approximate initial position for Gaussians. We can then train a transformer model to jointly predict comparatively small adjustments to these positions, as well as the other Gaussians' attributes and the SMPL parameters. We show empirically that this combination (using only multi-view supervision) can achieve fast inference of 3D human models from a single image without test-time optimization, expensive diffusion models, or 3D points supervision. We also show that it can improve 3D pose estimation by better fitting human models that account for clothes and other variations. The code is available on the project website this https URL .

Title: Learning to Learn Transferable Generative Attack for Person Re-Identification

• Authors: Yuan Bian, Min Liu, Xueping Wang, Yunfeng Ma, Yaonan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04208
• Pdf URL: https://arxiv.org/pdf/2409.04208
• Copy Paste: [[2409.04208]] Learning to Learn Transferable Generative Attack for Person Re-Identification(https://arxiv.org/abs/2409.04208)
• Keywords: generative
• Abstract: Deep learning-based person re-identification (re-id) models are widely employed in surveillance systems and inevitably inherit the vulnerability of deep networks to adversarial attacks. Existing attacks merely consider cross-dataset and cross-model transferability, ignoring the cross-test capability to perturb models trained in different domains. To powerfully examine the robustness of real-world re-id models, the Meta Transferable Generative Attack (MTGA) method is proposed, which adopts meta-learning optimization to promote the generative attacker producing highly transferable adversarial examples by learning comprehensively simulated transfer-based cross-model\&dataset\&test black-box meta attack tasks. Specifically, cross-model\&dataset black-box attack tasks are first mimicked by selecting different re-id models and datasets for meta-train and meta-test attack processes. As different models may focus on different feature regions, the Perturbation Random Erasing module is further devised to prevent the attacker from learning to only corrupt model-specific features. To boost the attacker learning to possess cross-test transferability, the Normalization Mix strategy is introduced to imitate diverse feature embedding spaces by mixing multi-domain statistics of target models. Extensive experiments show the superiority of MTGA, especially in cross-model\&dataset and cross-model\&dataset\&test attacks, our MTGA outperforms the SOTA methods by 21.5\% and 11.3\% on mean mAP drop rate, respectively. The code of MTGA will be released after the paper is accepted.

Title: UniDet3D: Multi-dataset Indoor 3D Object Detection

• Authors: Maksim Kolodiazhnyi, Anna Vorontsova, Matvey Skripkin, Danila Rukhovich, Anton Konushin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04234
• Pdf URL: https://arxiv.org/pdf/2409.04234
• Copy Paste: [[2409.04234]] UniDet3D: Multi-dataset Indoor 3D Object Detection(https://arxiv.org/abs/2409.04234)
• Keywords: foundation model
• Abstract: Growing customer demand for smart solutions in robotics and augmented reality has attracted considerable attention to 3D object detection from point clouds. Yet, existing indoor datasets taken individually are too small and insufficiently diverse to train a powerful and general 3D object detection model. In the meantime, more general approaches utilizing foundation models are still inferior in quality to those based on supervised training for a specific task. In this work, we propose \ours{}, a simple yet effective 3D object detection model, which is trained on a mixture of indoor datasets and is capable of working in various indoor environments. By unifying different label spaces, \ours{} enables learning a strong representation across multiple datasets through a supervised joint training scheme. The proposed network architecture is built upon a vanilla transformer encoder, making it easy to run, customize and extend the prediction pipeline for practical use. Extensive experiments demonstrate that \ours{} obtains significant gains over existing 3D object detection methods in 6 indoor benchmarks: ScanNet (+1.1 mAP50), ARKitScenes (+19.4 mAP25), S3DIS (+9.1 mAP50), MultiScan (+9.3 mAP50), 3RScan (+3.2 mAP50), and ScanNet++ (+2.7 mAP50). Code is available at this https URL .

Title: Learning vs Retrieval: The Role of In-Context Examples in Regression with LLMs

• Authors: Aliakbar Nafar, Kristen Brent Venable, Parisa Kordjamshidi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04318
• Pdf URL: https://arxiv.org/pdf/2409.04318
• Copy Paste: [[2409.04318]] Learning vs Retrieval: The Role of In-Context Examples in Regression with LLMs(https://arxiv.org/abs/2409.04318)
• Keywords: generative, in-context
• Abstract: Generative Large Language Models (LLMs) are capable of being in-context learners. However, the underlying mechanism of in-context learning (ICL) is still a major research question, and experimental research results about how models exploit ICL are not always consistent. In this work, we propose a framework for evaluating in-context learning mechanisms, which we claim are a combination of retrieving internal knowledge and learning from in-context examples by focusing on regression tasks. First, we show that LLMs can perform regression on real-world datasets and then design experiments to measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. We argue that this process lies on a spectrum between these two extremes. We provide an in-depth analysis of the degrees to which these mechanisms are triggered depending on various factors, such as prior knowledge about the tasks and the type and richness of the information provided by the in-context examples. We employ three LLMs and utilize multiple datasets to corroborate the robustness of our findings. Our results shed light on how to engineer prompts to leverage meta-learning from in-context examples and foster knowledge retrieval depending on the problem being addressed.

Title: Empirical Bayesian image restoration by Langevin sampling with a denoising diffusion implicit prior

• Authors: Charlesquin Kemajou Mbakam, Jean-Francois Giovannelli, Marcelo Pereyra
• Subjects: cs.CV, cs.CL, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.04384
• Pdf URL: https://arxiv.org/pdf/2409.04384
• Copy Paste: [[2409.04384]] Empirical Bayesian image restoration by Langevin sampling with a denoising diffusion implicit prior(https://arxiv.org/abs/2409.04384)
• Keywords: diffusion
• Abstract: Score-based diffusion methods provide a powerful strategy to solve image restoration tasks by flexibly combining a pre-trained foundational prior model with a likelihood function specified during test time. Such methods are predominantly derived from two stochastic processes: reversing Ornstein-Uhlenbeck, which underpins the celebrated denoising diffusion probabilistic models (DDPM) and denoising diffusion implicit models (DDIM), and the Langevin diffusion process. The solutions delivered by DDPM and DDIM are often remarkably realistic, but they are not always consistent with measurements because of likelihood intractability issues and the associated required approximations. Alternatively, using a Langevin process circumvents the intractable likelihood issue, but usually leads to restoration results of inferior quality and longer computing times. This paper presents a novel and highly computationally efficient image restoration method that carefully embeds a foundational DDPM denoiser within an empirical Bayesian Langevin algorithm, which jointly calibrates key model hyper-parameters as it estimates the model's posterior mean. Extensive experimental results on three canonical tasks (image deblurring, super-resolution, and inpainting) demonstrate that the proposed approach improves on state-of-the-art strategies both in image estimation accuracy and computing time.

Title: VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

• Authors: Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, Yao Lu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04429
• Pdf URL: https://arxiv.org/pdf/2409.04429
• Copy Paste: [[2409.04429]] VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation(https://arxiv.org/abs/2409.04429)
• Keywords: diffusion, foundation model
• Abstract: VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation. Traditional visual language models (VLMs) use separate modules for understanding and generating visual content, which can lead to misalignment and increased complexity. In contrast, VILA-U employs a single autoregressive next-token prediction framework for both tasks, eliminating the need for additional components like diffusion models. This approach not only simplifies the model but also achieves near state-of-the-art performance in visual language understanding and generation. The success of VILA-U is attributed to two main factors: the unified vision tower that aligns discrete visual tokens with textual inputs during pretraining, which enhances visual perception, and autoregressive image generation can achieve similar quality as diffusion models with high-quality dataset. This allows VILA-U to perform comparably to more complex models using a fully token-based autoregressive framework.