2024-07-19

Title: "I understand why I got this grade": Automatic Short Answer Grading with Feedback

• Authors: Dishank Aggarwal, Pushpak Bhattacharyya, Bhaskaran Raman
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.12818
• Pdf URL: https://arxiv.org/pdf/2407.12818
• Copy Paste: [[2407.12818]] "I understand why I got this grade": Automatic Short Answer Grading with Feedback(https://arxiv.org/abs/2407.12818)
• Keywords: generative
• Abstract: The demand for efficient and accurate assessment methods has intensified as education systems transition to digital platforms. Providing feedback is essential in educational settings and goes beyond simply conveying marks as it justifies the assigned marks. In this context, we present a significant advancement in automated grading by introducing Engineering Short Answer Feedback (EngSAF) -- a dataset of 5.8k student answers accompanied by reference answers and questions for the Automatic Short Answer Grading (ASAG) task. The EngSAF dataset is meticulously curated to cover a diverse range of subjects, questions, and answer patterns from multiple engineering domains. We leverage state-of-the-art large language models' (LLMs) generative capabilities with our Label-Aware Synthetic Feedback Generation (LASFG) strategy to include feedback in our dataset. This paper underscores the importance of enhanced feedback in practical educational settings, outlines dataset annotation and feedback generation processes, conducts a thorough EngSAF analysis, and provides different LLMs-based zero-shot and finetuned baselines for future comparison. Additionally, we demonstrate the efficiency and effectiveness of the ASAG system through its deployment in a real-world end-semester exam at the Indian Institute of Technology Bombay (IITB), showcasing its practical viability and potential for broader implementation in educational institutions.

Title: AutoFlow: Automated Workflow Generation for Large Language Model Agents

• Authors: Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, Yongfeng Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12821
• Pdf URL: https://arxiv.org/pdf/2407.12821
• Copy Paste: [[2407.12821]] AutoFlow: Automated Workflow Generation for Large Language Model Agents(https://arxiv.org/abs/2407.12821)
• Keywords: in-context
• Abstract: Recent advancements in Large Language Models (LLMs) have shown significant progress in understanding complex natural language. One important application of LLM is LLM-based AI Agent, which leverages the ability of LLM as well as external tools for complex-task solving. To make sure LLM Agents follow an effective and reliable procedure to solve the given task, manually designed workflows are usually used to guide the working mechanism of agents. However, manually designing the workflows requires considerable efforts and domain knowledge, making it difficult to develop and deploy agents on massive scales. To address these issues, we propose AutoFlow, a framework designed to automatically generate workflows for agents to solve complex tasks. AutoFlow takes natural language program as the format of agent workflow and employs a workflow optimization procedure to iteratively optimize the workflow quality. Besides, this work offers two workflow generation methods: fine-tuning-based and in-context-based methods, making the AutoFlow framework applicable to both open-source and closed-source LLMs. Experimental results show that our framework can produce robust and reliable agent workflows. We believe that the automatic generation and interpretation of workflows in natural language represent a promising paradigm for solving complex tasks, particularly with the rapid development of LLMs. The source code of this work is available at this https URL.

Title: MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production

• Authors: Jian Ma, Wenguan Wang, Yi Yang, Feng Zheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12842
• Pdf URL: https://arxiv.org/pdf/2407.12842
• Copy Paste: [[2407.12842]] MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production(https://arxiv.org/abs/2407.12842)
• Keywords: diffusion
• Abstract: Sign language understanding has made significant strides; however, there is still no viable solution for generating sign sequences directly from entire spoken content, e.g., text or speech. In this paper, we propose a unified framework for continuous sign language production, easing communication between sign and non-sign language users. In particular, a sequence diffusion model, utilizing embeddings extracted from text or speech, is crafted to generate sign predictions step by step. Moreover, by creating a joint embedding space for text, audio, and sign, we bind these modalities and leverage the semantic consistency among them to provide informative feedback for the model training. This embedding-consistency learning strategy minimizes the reliance on sign triplets and ensures continuous model refinement, even with a missing audio modality. Experiments on How2Sign and PHOENIX14T datasets demonstrate that our model achieves competitive performance in sign language production.

Title: Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization

• Authors: Aniket Deroy, Kripabandhu Ghosh, Saptarshi Ghosh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12848
• Pdf URL: https://arxiv.org/pdf/2407.12848
• Copy Paste: [[2407.12848]] Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization(https://arxiv.org/abs/2407.12848)
• Keywords: generative
• Abstract: Automatic summarization of legal case judgements, which are known to be long and complex, has traditionally been tried via extractive summarization models. In recent years, generative models including abstractive summarization models and Large language models (LLMs) have gained huge popularity. In this paper, we explore the applicability of such models for legal case judgement summarization. We applied various domain specific abstractive summarization models and general domain LLMs as well as extractive summarization models over two sets of legal case judgements from the United Kingdom (UK) Supreme Court and the Indian (IN) Supreme Court and evaluated the quality of the generated summaries. We also perform experiments on a third dataset of legal documents of a different type, Government reports from the United States (US). Results show that abstractive summarization models and LLMs generally perform better than the extractive methods as per traditional metrics for evaluating summary quality. However, detailed investigation shows the presence of inconsistencies and hallucinations in the outputs of the generative models, and we explore ways to reduce the hallucinations and inconsistencies in the summaries. Overall, the investigation suggests that further improvements are needed to enhance the reliability of abstractive models and LLMs for legal case judgement summarization. At present, a human-in-the-loop technique is more suitable for performing manual checks to identify inconsistencies in the generated summaries.

Title: Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)

• Authors: Krishnaram Kenthapadi, Mehrnoosh Sameki, Ankur Taly
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12858
• Pdf URL: https://arxiv.org/pdf/2407.12858
• Copy Paste: [[2407.12858]] Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)(https://arxiv.org/abs/2407.12858)
• Keywords: generative
• Abstract: With the ongoing rapid adoption of Artificial Intelligence (AI)-based systems in high-stakes domains, ensuring the trustworthiness, safety, and observability of these systems has become crucial. It is essential to evaluate and monitor AI systems not only for accuracy and quality-related metrics but also for robustness, bias, security, interpretability, and other responsible AI dimensions. We focus on large language models (LLMs) and other generative AI models, which present additional challenges such as hallucinations, harmful and manipulative content, and copyright infringement. In this survey article accompanying our KDD 2024 tutorial, we highlight a wide range of harms associated with generative AI systems, and survey state of the art approaches (along with open challenges) to address these harms.

Title: STAGE: Simplified Text-Attributed Graph Embeddings Using Pre-trained LLMs

• Authors: Aaron Zolnai-Lucas, Jack Boylan, Chris Hokamp, Parsa Ghaffari
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12860
• Pdf URL: https://arxiv.org/pdf/2407.12860
• Copy Paste: [[2407.12860]] STAGE: Simplified Text-Attributed Graph Embeddings Using Pre-trained LLMs(https://arxiv.org/abs/2407.12860)
• Keywords: diffusion
• Abstract: We present Simplified Text-Attributed Graph Embeddings (STAGE), a straightforward yet effective method for enhancing node features in Graph Neural Network (GNN) models that encode Text-Attributed Graphs (TAGs). Our approach leverages Large-Language Models (LLMs) to generate embeddings for textual attributes. STAGE achieves competitive results on various node classification benchmarks while also maintaining a simplicity in implementation relative to current state-of-the-art (SoTA) techniques. We show that utilizing pre-trained LLMs as embedding generators provides robust features for ensemble GNN training, enabling pipelines that are simpler than current SoTA approaches which require multiple expensive training and prompting stages. We also implement diffusion-pattern GNNs in an effort to make this pipeline scalable to graphs beyond academic benchmarks.

Title: Bilingual Adaptation of Monolingual Foundation Models

• Authors: Gurpreet Gosal, Yishi Xu, Gokul Ramakrishnan, Rituraj Joshi, Avraham Sheinin, Zhiming (Charles)Chen, Biswajit Mishra, Natalia Vassilieva, Joel Hestness, Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Onkar Pandit, Samta Kamboj, Rahul Pal, Parvez Mullah, Soundar Doraiswamy, Mohamed El Karim Chami
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12869
• Pdf URL: https://arxiv.org/pdf/2407.12869
• Copy Paste: [[2407.12869]] Bilingual Adaptation of Monolingual Foundation Models(https://arxiv.org/abs/2407.12869)
• Keywords: foundation model
• Abstract: We present an efficient method for adapting a monolingual Large Language Model (LLM) to another language, addressing challenges of catastrophic forgetting and tokenizer limitations. We focus this study on adapting Llama 2 to Arabic. Our two-stage approach begins with expanding the vocabulary and training only the embeddings matrix, followed by full model continual pretraining on a bilingual corpus. By continually pretraining on a mix of Arabic and English corpora, the model retains its proficiency in English while acquiring capabilities in Arabic. Our approach results in significant improvements in Arabic and slight enhancements in English, demonstrating cost-effective cross-lingual transfer. We also perform extensive ablations on embedding initialization techniques, data mix ratios, and learning rates and release a detailed training recipe.

Title: MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation

• Authors: Xiaohan Wang, Dian Li, Yilin Zhao, Sinbadliu, Hui Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12871
• Pdf URL: https://arxiv.org/pdf/2407.12871
• Copy Paste: [[2407.12871]] MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation(https://arxiv.org/abs/2407.12871)
• Keywords: self-supervised, in-context
• Abstract: Utilizing complex tools with Large Language Models (LLMs) is a critical component for grounding AI agents in various real-world scenarios. The core challenge of manipulating tools lies in understanding their usage and functionality. The prevailing approach involves few-shot prompting with demonstrations or fine-tuning on expert trajectories. However, for complex tools and tasks, mere in-context demonstrations may fail to cover sufficient knowledge. Training-based methods are also constrained by the high cost of dataset construction and limited generalizability. In this paper, we introduce a new tool learning methodology (MetaTool) that is generalizable for mastering any reusable toolset. Our approach includes a self-supervised data augmentation technique that enables LLMs to gain a comprehensive understanding of various tools, thereby improving their ability to complete tasks effectively. We develop a series of meta-tasks that involve predicting masked factors of tool execution. These self-supervised tasks enable the automatic generation of high-quality QA data concerning tool comprehension. By incorporating meta-task data into the instruction tuning process, the proposed MetaTool model achieves significant superiority to open-source models and is comparable to GPT-4/GPT-3.5 on multiple tool-oriented tasks.

Title: Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detection

• Authors: Ye Jiang, Yimin Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.12879
• Pdf URL: https://arxiv.org/pdf/2407.12879
• Copy Paste: [[2407.12879]] Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detection(https://arxiv.org/abs/2407.12879)
• Keywords: in-context
• Abstract: Large visual-language models (LVLMs) exhibit exceptional performance in visual-language reasoning across diverse cross-modal benchmarks. Despite these advances, recent research indicates that Large Language Models (LLMs), like GPT-3.5-turbo, underachieve compared to well-trained smaller models, such as BERT, in Fake News Detection (FND), prompting inquiries into LVLMs' efficacy in FND tasks. Although performance could improve through fine-tuning LVLMs, the substantial parameters and requisite pre-trained weights render it a resource-heavy endeavor for FND applications. This paper initially assesses the FND capabilities of two notable LVLMs, CogVLM and GPT4V, in comparison to a smaller yet adeptly trained CLIP model in a zero-shot context. The findings demonstrate that LVLMs can attain performance competitive with that of the smaller model. Next, we integrate standard in-context learning (ICL) with LVLMs, noting improvements in FND performance, though limited in scope and consistency. To address this, we introduce the \textbf{I}n-context \textbf{M}ultimodal \textbf{F}ake \textbf{N}ews \textbf{D}etection (IMFND) framework, enriching in-context examples and test inputs with predictions and corresponding probabilities from a well-trained smaller model. This strategic integration directs the LVLMs' focus towards news segments associated with higher probabilities, thereby improving their analytical accuracy. The experimental results suggest that the IMFND framework significantly boosts the FND efficiency of LVLMs, achieving enhanced accuracy over the standard ICL approach across three publicly available FND datasets.

Title: BinaryAlign: Word Alignment as Binary Sequence Labeling

• Authors: Gaetan Lopez Latouche, Marc-André Carbonneau, Ben Swanson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.12881
• Pdf URL: https://arxiv.org/pdf/2407.12881
• Copy Paste: [[2407.12881]] BinaryAlign: Word Alignment as Binary Sequence Labeling(https://arxiv.org/abs/2407.12881)
• Keywords: foundation model
• Abstract: Real world deployments of word alignment are almost certain to cover both high and low resource languages. However, the state-of-the-art for this task recommends a different model class depending on the availability of gold alignment training data for a particular language pair. We propose BinaryAlign, a novel word alignment technique based on binary sequence labeling that outperforms existing approaches in both scenarios, offering a unifying approach to the task. Additionally, we vary the specific choice of multilingual foundation model, perform stratified error analysis over alignment error type, and explore the performance of BinaryAlign on non-English language pairs. We make our source code publicly available.

Title: GeoGuide: Geometric guidance of diffusion models

• Authors: Mateusz Poleski, Jacek Tabor, Przemysław Spurek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12889
• Pdf URL: https://arxiv.org/pdf/2407.12889
• Copy Paste: [[2407.12889]] GeoGuide: Geometric guidance of diffusion models(https://arxiv.org/abs/2407.12889)
• Keywords: diffusion
• Abstract: Diffusion models are among the most effective methods for image generation. This is in particular because, unlike GANs, they can be easily conditioned during training to produce elements with desired class or properties. However, guiding a pre-trained diffusion model to generate elements from previously unlabeled data is significantly more challenging. One of the possible solutions was given by the ADM-G guiding approach. Although ADM-G successfully generates elements from the given class, there is a significant quality gap compared to a model originally conditioned on this class. In particular, the FID score obtained by the ADM-G-guided diffusion model is nearly three times lower than the class-conditioned guidance. We demonstrate that this issue is partly due to ADM-G providing minimal guidance during the final stage of the denoising process. To address this problem, we propose GeoGuide, a guidance model based on tracing the distance of the diffusion model's trajectory from the data manifold. The main idea of GeoGuide is to produce normalized adjustments during the backward denoising process. As shown in the experiments, GeoGuide surpasses the probabilistic approach ADM-G with respect to both the FID scores and the quality of the generated images.

Title: DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion

• Authors: Huiguo He, Huan Yang, Zixi Tuo, Yuan Zhou, Qiuyue Wang, Yuhang Zhang, Zeyu Liu, Wenhao Huang, Hongyang Chao, Jian Yin
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.12899
• Pdf URL: https://arxiv.org/pdf/2407.12899
• Copy Paste: [[2407.12899]] DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion(https://arxiv.org/abs/2407.12899)
• Keywords: diffusion
• Abstract: Story visualization aims to create visually compelling images or videos corresponding to textual narratives. Despite recent advances in diffusion models yielding promising results, existing methods still struggle to create a coherent sequence of subject-consistent frames based solely on a story. To this end, we propose DreamStory, an automatic open-domain story visualization framework by leveraging the LLMs and a novel multi-subject consistent diffusion model. DreamStory consists of (1) an LLM acting as a story director and (2) an innovative Multi-Subject consistent Diffusion model (MSD) for generating consistent multi-subject across the images. First, DreamStory employs the LLM to generate descriptive prompts for subjects and scenes aligned with the story, annotating each scene's subjects for subsequent subject-consistent generation. Second, DreamStory utilizes these detailed subject descriptions to create portraits of the subjects, with these portraits and their corresponding textual information serving as multimodal anchors (guidance). Finally, the MSD uses these multimodal anchors to generate story scenes with consistent multi-subject. Specifically, the MSD includes Masked Mutual Self-Attention (MMSA) and Masked Mutual Cross-Attention (MMCA) modules. MMSA and MMCA modules ensure appearance and semantic consistency with reference images and text, respectively. Both modules employ masking mechanisms to prevent subject blending. To validate our approach and promote progress in story visualization, we established a benchmark, DS-500, which can assess the overall performance of the story visualization framework, subject-identification accuracy, and the consistency of the generation model. Extensive experiments validate the effectiveness of DreamStory in both subjective and objective evaluations. Please visit our project homepage at this https URL.

Title: The Foundation Model Transparency Index v1.1: May 2024

• Authors: Rishi Bommasani, Kevin Klyman, Sayash Kapoor, Shayne Longpre, Betty Xiong, Nestor Maslej, Percy Liang
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.12929
• Pdf URL: https://arxiv.org/pdf/2407.12929
• Copy Paste: [[2407.12929]] The Foundation Model Transparency Index v1.1: May 2024(https://arxiv.org/abs/2407.12929)
• Keywords: foundation model
• Abstract: Foundation models are increasingly consequential yet extremely opaque. To characterize the status quo, the Foundation Model Transparency Index was launched in October 2023 to measure the transparency of leading foundation model developers. The October 2023 Index (v1.0) assessed 10 major foundation model developers (e.g. OpenAI, Google) on 100 transparency indicators (e.g. does the developer disclose the wages it pays for data labor?). At the time, developers publicly disclosed very limited information with the average score being 37 out of 100. To understand how the status quo has changed, we conduct a follow-up study (v1.1) after 6 months: we score 14 developers against the same 100 indicators. While in v1.0 we searched for publicly available information, in v1.1 developers submit reports on the 100 transparency indicators, potentially including information that was not previously public. We find that developers now score 58 out of 100 on average, a 21 point improvement over v1.0. Much of this increase is driven by developers disclosing information during the v1.1 process: on average, developers disclosed information related to 16.6 indicators that was not previously public. We observe regions of sustained (i.e. across v1.0 and v1.1) and systemic (i.e. across most or all developers) opacity such as on copyright status, data access, data labor, and downstream impact. We publish transparency reports for each developer that consolidate information disclosures: these reports are based on the information disclosed to us via developers. Our findings demonstrate that transparency can be improved in this nascent ecosystem, the Foundation Model Transparency Index likely contributes to these improvements, and policymakers should consider interventions in areas where transparency has not improved.

Title: GenRC: Generative 3D Room Completion from Sparse Image Collections

• Authors: Ming-Feng Li, Yueh-Feng Ku, Hong-Xuan Yen, Chi Liu, Yu-Lun Liu, Albert Y. C. Chen, Cheng-Hao Kuo, Min Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12939
• Pdf URL: https://arxiv.org/pdf/2407.12939
• Copy Paste: [[2407.12939]] GenRC: Generative 3D Room Completion from Sparse Image Collections(https://arxiv.org/abs/2407.12939)
• Keywords: diffusion, generative
• Abstract: Sparse RGBD scene completion is a challenging task especially when considering consistent textures and geometries throughout the entire scene. Different from existing solutions that rely on human-designed text prompts or predefined camera trajectories, we propose GenRC, an automated training-free pipeline to complete a room-scale 3D mesh with high-fidelity textures. To achieve this, we first project the sparse RGBD images to a highly incomplete 3D mesh. Instead of iteratively generating novel views to fill in the void, we utilized our proposed E-Diffusion to generate a view-consistent panoramic RGBD image which ensures global geometry and appearance consistency. Furthermore, we maintain the input-output scene stylistic consistency through textual inversion to replace human-designed text prompts. To bridge the domain gap among datasets, E-Diffusion leverages models trained on large-scale datasets to generate diverse appearances. GenRC outperforms state-of-the-art methods under most appearance and geometric metrics on ScanNet and ARKitScenes datasets, even though GenRC is not trained on these datasets nor using predefined camera trajectories. Project page: \href{this https URL}{this https URL}

Title: Denoising Diffusions in Latent Space for Medical Image Segmentation

• Authors: Fahim Ahmed Zaman, Mathews Jacob, Amanda Chang, Kan Liu, Milan Sonka, Xiaodong Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.12952
• Pdf URL: https://arxiv.org/pdf/2407.12952
• Copy Paste: [[2407.12952]] Denoising Diffusions in Latent Space for Medical Image Segmentation(https://arxiv.org/abs/2407.12952)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DPMs) have demonstrated remarkable performance in image generation, often times outperforming other generative models. Since their introduction, the powerful noise-to-image denoising pipeline has been extended to various discriminative tasks, including image segmentation. In case of medical imaging, often times the images are large 3D scans, where segmenting one image using DPMs become extremely inefficient due to large memory consumption and time consuming iterative sampling process. In this work, we propose a novel conditional generative modeling framework (LDSeg) that performs diffusion in latent space for medical image segmentation. Our proposed framework leverages the learned inherent low-dimensional latent distribution of the target object shapes and source image embeddings. The conditional diffusion in latent space not only ensures accurate n-D image segmentation for multi-label objects, but also mitigates the major underlying problems of the traditional DPM based segmentation: (1) large memory consumption, (2) time consuming sampling process and (3) unnatural noise injection in forward/reverse process. LDSeg achieved state-of-the-art segmentation accuracy on three medical image datasets with different imaging modalities. Furthermore, we show that our proposed model is significantly more robust to noises, compared to the traditional deterministic segmentation models, which can be potential in solving the domain shift problems in the medical imaging domain. Codes are available at: this https URL.

Title: ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders

• Authors: Carlos Hinojosa, Shuming Liu, Bernard Ghanem
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.13036
• Pdf URL: https://arxiv.org/pdf/2407.13036
• Copy Paste: [[2407.13036]] ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders(https://arxiv.org/abs/2407.13036)
• Keywords: self-supervised
• Abstract: Masked AutoEncoders (MAE) have emerged as a robust self-supervised framework, offering remarkable performance across a wide range of downstream tasks. To increase the difficulty of the pretext task and learn richer visual representations, existing works have focused on replacing standard random masking with more sophisticated strategies, such as adversarial-guided and teacher-guided masking. However, these strategies depend on the input data thus commonly increasing the model complexity and requiring additional calculations to generate the mask patterns. This raises the question: Can we enhance MAE performance beyond random masking without relying on input data or incurring additional computational costs? In this work, we introduce a simple yet effective data-independent method, termed ColorMAE, which generates different binary mask patterns by filtering random noise. Drawing inspiration from color noise in image processing, we explore four types of filters to yield mask patterns with different spatial and semantic priors. ColorMAE requires no additional learnable parameters or computational overhead in the network, yet it significantly enhances the learned representations. We provide a comprehensive empirical evaluation, demonstrating our strategy's superiority in downstream tasks compared to random masking. Notably, we report an improvement of 2.72 in mIoU in semantic segmentation tasks relative to baseline MAE implementations.

Title: Universal Facial Encoding of Codec Avatars from VR Headsets

• Authors: Shaojie Bai, Te-Li Wang, Chenghui Li, Akshay Venkatesh, Tomas Simon, Chen Cao, Gabriel Schwartz, Ryan Wrench, Jason Saragih, Yaser Sheikh, Shih-En Wei
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13038
• Pdf URL: https://arxiv.org/pdf/2407.13038
• Copy Paste: [[2407.13038]] Universal Facial Encoding of Codec Avatars from VR Headsets(https://arxiv.org/abs/2407.13038)
• Keywords: self-supervised
• Abstract: Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces. In this paper, we present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a self-supervised learning approach, based on a cross-view reconstruction objective, that enables generalization to unseen users. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency. We present an improved parameterization for precise ground-truth generation that provides robustness to environmental variation. The resulting system produces accurate facial animation for unseen users wearing VR headsets in realtime. We compare our approach to prior face-encoding methods demonstrating significant improvements in both quantitative metrics and qualitative results.

Title: INTELLECT: Adapting Cyber Threat Detection to Heterogeneous Computing Environments

• Authors: Simone Magnani, Liubov Nedoshivina, Roberto Doriguzzi-Corin, Stefano Braghin, Domenico Siracusa
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13043
• Pdf URL: https://arxiv.org/pdf/2407.13043
• Copy Paste: [[2407.13043]] INTELLECT: Adapting Cyber Threat Detection to Heterogeneous Computing Environments(https://arxiv.org/abs/2407.13043)
• Keywords: anomaly
• Abstract: The widespread adoption of cloud computing, edge, and IoT has increased the attack surface for cyber threats. This is due to the large-scale deployment of often unsecured, heterogeneous devices with varying hardware and software configurations. The diversity of these devices attracts a wide array of potential attack methods, making it challenging for individual organizations to have comprehensive knowledge of all possible threats. In this context, powerful anomaly detection models can be developed by combining data from different parties using Federated Learning. FL enables the collaborative development of ML-based IDSs without requiring the parties to disclose sensitive training data, such as network traffic or sensor readings. However, deploying the resulting models can be challenging, as they may require more computational resources than those available on target devices with limited capacity or already allocated for other operations. Training device-specific models is not feasible for an organization because a significant portion of the training data is private to other participants in the FL process. To address these challenges, this paper introduces INTELLECT, a novel solution that integrates feature selection, model pruning, and fine-tuning techniques into a cohesive pipeline for the dynamic adaptation of pre-trained ML models and configurations for IDSs. Through empirical evaluation, we analyze the benefits of INTELLECT's approach in tailoring ML models to the specific resource constraints of an organization's devices and measure variations in traffic classification accuracy resulting from feature selection, pruning, and fine-tuning operations. Additionally, we demonstrate the advantages of incorporating knowledge distillation techniques while fine-tuning, enabling the ML model to consistently adapt to local network patterns while preserving historical knowledge.

Title: A Novel GAN Approach to Augment Limited Tabular Data for Short-Term Substance Use Prediction

• Authors: Nguyen Thach, Patrick Habecker, Bergen Johnston, Lillianna Cervantes, Anika Eisenbraun, Alex Mason, Kimberly Tyler, Bilal Khan, Hau Chan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13047
• Pdf URL: https://arxiv.org/pdf/2407.13047
• Copy Paste: [[2407.13047]] A Novel GAN Approach to Augment Limited Tabular Data for Short-Term Substance Use Prediction(https://arxiv.org/abs/2407.13047)
• Keywords: generative
• Abstract: Substance use is a global issue that negatively impacts millions of persons who use drugs (PWUDs). In practice, identifying vulnerable PWUDs for efficient allocation of appropriate resources is challenging due to their complex use patterns (e.g., their tendency to change usage within months) and the high acquisition costs for collecting PWUD-focused substance use data. Thus, there has been a paucity of machine learning models for accurately predicting short-term substance use behaviors of PWUDs. In this paper, using longitudinal survey data of 258 PWUDs in the U.S. Great Plains collected by our team, we design a novel GAN that deals with high-dimensional low-sample-size tabular data and survey skip logic to augment existing data to improve classification models' prediction on (A) whether the PWUDs would increase usage and (B) at which ordinal frequency they would use a particular drug within the next 12 months. Our evaluation results show that, when trained on augmented data from our proposed GAN, the classification models improve their predictive performance (AUROC) by up to 13.4% in Problem (A) and 15.8% in Problem (B) for usage of marijuana, meth, amphetamines, and cocaine, which outperform state-of-the-art generative models.

Title: Rethinking Video-Text Understanding: Retrieval from Counterfactually Augmented Data

• Authors: Wufei Ma, Kai Li, Zhongshi Jiang, Moustafa Meshry, Qihao Liu, Huiyu Wang, Christian Häne, Alan Yuille
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13094
• Pdf URL: https://arxiv.org/pdf/2407.13094
• Copy Paste: [[2407.13094]] Rethinking Video-Text Understanding: Retrieval from Counterfactually Augmented Data(https://arxiv.org/abs/2407.13094)
• Keywords: foundation model
• Abstract: Recent video-text foundation models have demonstrated strong performance on a wide variety of downstream video understanding tasks. Can these video-text models genuinely understand the contents of natural videos? Standard video-text evaluations could be misleading as many questions can be inferred merely from the objects and contexts in a single frame or biases inherent in the datasets. In this paper, we aim to better assess the capabilities of current video-text models and understand their limitations. We propose a novel evaluation task for video-text understanding, namely retrieval from counterfactually augmented data (RCAD), and a new Feint6K dataset. To succeed on our new evaluation task, models must derive a comprehensive understanding of the video from cross-frame reasoning. Analyses show that previous video-text foundation models can be easily fooled by counterfactually augmented data and are far behind human-level performance. In order to narrow the gap between video-text models and human performance on RCAD, we identify a key limitation of current contrastive approaches on video-text data and introduce LLM-teacher, a more effective approach to learn action semantics by leveraging knowledge obtained from a pretrained large language model. Experiments and analyses show that our approach successfully learn more discriminative action embeddings and improves results on Feint6K when applied to multiple video-text models. Our Feint6K dataset and project page is available at this https URL.

Title: FocusDiffuser: Perceiving Local Disparities for Camouflaged Object Detection

• Authors: Jianwei Zhao, Xin Li, Fan Yang, Qiang Zhai, Ao Luo, Zicheng Jiao, Hong Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13133
• Pdf URL: https://arxiv.org/pdf/2407.13133
• Copy Paste: [[2407.13133]] FocusDiffuser: Perceiving Local Disparities for Camouflaged Object Detection(https://arxiv.org/abs/2407.13133)
• Keywords: diffusion, generative
• Abstract: Detecting objects seamlessly blended into their surroundings represents a complex task for both human cognitive capabilities and advanced artificial intelligence algorithms. Currently, the majority of methodologies for detecting camouflaged objects mainly focus on utilizing discriminative models with various unique designs. However, it has been observed that generative models, such as Stable Diffusion, possess stronger capabilities for understanding various objects in complex environments; Yet their potential for the cognition and detection of camouflaged objects has not been extensively explored. In this study, we present a novel denoising diffusion model, namely FocusDiffuser, to investigate how generative models can enhance the detection and interpretation of camouflaged objects. We believe that the secret to spotting camouflaged objects lies in catching the subtle nuances in details. Consequently, our FocusDiffuser innovatively integrates specialized enhancements, notably the Boundary-Driven LookUp (BDLU) module and Cyclic Positioning (CP) module, to elevate standard diffusion models, significantly boosting the detail-oriented analytical capabilities. Our experiments demonstrate that FocusDiffuser, from a generative perspective, effectively addresses the challenge of camouflaged object detection, surpassing leading models on benchmarks like CAMO, COD10K and NC4K.

Title: Image Inpainting Models are Effective Tools for Instruction-guided Image Editing

• Authors: Xuan Ju, Junhao Zhuang, Zhaoyang Zhang, Yuxuan Bian, Qiang Xu, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13139
• Pdf URL: https://arxiv.org/pdf/2407.13139
• Copy Paste: [[2407.13139]] Image Inpainting Models are Effective Tools for Instruction-guided Image Editing(https://arxiv.org/abs/2407.13139)
• Keywords: diffusion
• Abstract: This is the technique report for the winning solution of the CVPR2024 GenAI Media Generation Challenge Workshop's Instruction-guided Image Editing track. Instruction-guided image editing has been largely studied in recent years. The most advanced methods, such as SmartEdit and MGIE, usually combine large language models with diffusion models through joint training, where the former provides text understanding ability, and the latter provides image generation ability. However, in our experiments, we find that simply connecting large language models and image generation models through intermediary guidance such as masks instead of joint fine-tuning leads to a better editing performance and success rate. We use a 4-step process IIIE (Inpainting-based Instruction-guided Image Editing): editing category classification, main editing object identification, editing mask acquisition, and image inpainting. Results show that through proper combinations of language models and image inpainting models, our pipeline can reach a high success rate with satisfying visual quality.

Title: Training-Free Large Model Priors for Multiple-in-One Image Restoration

• Authors: Xuanhua He, Lang Li, Yingying Wang, Hui Zheng, Ke Cao, Keyu Yan, Rui Li, Chengjun Xie, Jie Zhang, Man Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13181
• Pdf URL: https://arxiv.org/pdf/2407.13181
• Copy Paste: [[2407.13181]] Training-Free Large Model Priors for Multiple-in-One Image Restoration(https://arxiv.org/abs/2407.13181)
• Keywords: diffusion
• Abstract: Image restoration aims to reconstruct the latent clear images from their degraded versions. Despite the notable achievement, existing methods predominantly focus on handling specific degradation types and thus require specialized models, impeding real-world applications in dynamic degradation scenarios. To address this issue, we propose Large Model Driven Image Restoration framework (LMDIR), a novel multiple-in-one image restoration paradigm that leverages the generic priors from large multi-modal language models (MMLMs) and the pretrained diffusion models. In detail, LMDIR integrates three key prior knowledges: 1) global degradation knowledge from MMLMs, 2) scene-aware contextual descriptions generated by MMLMs, and 3) fine-grained high-quality reference images synthesized by diffusion models guided by MMLM descriptions. Standing on above priors, our architecture comprises a query-based prompt encoder, degradation-aware transformer block injecting global degradation knowledge, content-aware transformer block incorporating scene description, and reference-based transformer block incorporating fine-grained image priors. This design facilitates single-stage training paradigm to address various degradations while supporting both automatic and user-guided restoration. Extensive experiments demonstrate that our designed method outperforms state-of-the-art competitors on multiple evaluation benchmarks.

Title: SpaDiT: Diffusion Transformer for Spatial Gene Expression Prediction using scRNA-seq

• Authors: Xiaoyu Li, Fangfang Zhu, Wenwen Min
• Subjects: cs.LG, cs.AI, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2407.13182
• Pdf URL: https://arxiv.org/pdf/2407.13182
• Copy Paste: [[2407.13182]] SpaDiT: Diffusion Transformer for Spatial Gene Expression Prediction using scRNA-seq(https://arxiv.org/abs/2407.13182)
• Keywords: diffusion, generative
• Abstract: The rapid development of spatial transcriptomics (ST) technologies is revolutionizing our understanding of the spatial organization of biological tissues. Current ST methods, categorized into next-generation sequencing-based (seq-based) and fluorescence in situ hybridization-based (image-based) methods, offer innovative insights into the functional dynamics of biological tissues. However, these methods are limited by their cellular resolution and the quantity of genes they can detect. To address these limitations, we propose SpaDiT, a deep learning method that utilizes a diffusion generative model to integrate scRNA-seq and ST data for the prediction of undetected genes. By employing a Transformer-based diffusion model, SpaDiT not only accurately predicts unknown genes but also effectively generates the spatial structure of ST genes. We have demonstrated the effectiveness of SpaDiT through extensive experiments on both seq-based and image-based ST data. SpaDiT significantly contributes to ST gene prediction methods with its innovative approach. Compared to eight leading baseline methods, SpaDiT achieved state-of-the-art performance across multiple metrics, highlighting its substantial bioinformatics contribution.

Title: Safe-SD: Safe and Traceable Stable Diffusion with Text Prompt Trigger for Invisible Generative Watermarking

• Authors: Zhiyuan Ma, Guoli Jia, Biqing Qi, Bowen Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13188
• Pdf URL: https://arxiv.org/pdf/2407.13188
• Copy Paste: [[2407.13188]] Safe-SD: Safe and Traceable Stable Diffusion with Text Prompt Trigger for Invisible Generative Watermarking(https://arxiv.org/abs/2407.13188)
• Keywords: diffusion, generative
• Abstract: Recently, stable diffusion (SD) models have typically flourished in the field of image synthesis and personalized editing, with a range of photorealistic and unprecedented images being successfully generated. As a result, widespread interest has been ignited to develop and use various SD-based tools for visual content creation. However, the exposure of AI-created content on public platforms could raise both legal and ethical risks. In this regard, the traditional methods of adding watermarks to the already generated images (i.e. post-processing) may face a dilemma (e.g., being erased or modified) in terms of copyright protection and content monitoring, since the powerful image inversion and text-to-image editing techniques have been widely explored in SD-based methods. In this work, we propose a Safe and high-traceable Stable Diffusion framework (namely Safe-SD) to adaptively implant the graphical watermarks (e.g., QR code) into the imperceptible structure-related pixels during the generative diffusion process for supporting text-driven invisible watermarking and detection. Different from the previous high-cost injection-then-detection training framework, we design a simple and unified architecture, which makes it possible to simultaneously train watermark injection and detection in a single network, greatly improving the efficiency and convenience of use. Moreover, to further support text-driven generative watermarking and deeply explore its robustness and high-traceability, we elaborately design lambda sampling and encryption algorithm to fine-tune a latent diffuser wrapped by a VAE for balancing high-fidelity image synthesis and high-traceable watermark detection. We present our quantitative and qualitative results on two representative datasets LSUN, COCO and FFHQ, demonstrating state-of-the-art performance of Safe-SD and showing it significantly outperforms the previous approaches.

Title: Adaptive Foundation Models for Online Decisions: HyperAgent with Fast Incremental Uncertainty Estimation

• Authors: Yingru Li, Jiawei Xu, Zhi-Quan Luo
• Subjects: cs.LG, cs.AI, cs.HC, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.13195
• Pdf URL: https://arxiv.org/pdf/2407.13195
• Copy Paste: [[2407.13195]] Adaptive Foundation Models for Online Decisions: HyperAgent with Fast Incremental Uncertainty Estimation(https://arxiv.org/abs/2407.13195)
• Keywords: foundation model
• Abstract: Foundation models often struggle with uncertainty when faced with new situations in online decision-making, necessitating scalable and efficient exploration to resolve this uncertainty. We introduce GPT-HyperAgent, an augmentation of GPT with HyperAgent for uncertainty-aware, scalable exploration in contextual bandits, a fundamental online decision problem involving natural language input. We prove that HyperAgent achieves fast incremental uncertainty estimation with $\tilde{O}(\log T)$ per-step computational complexity over $T$ periods under the linear realizable assumption. Our analysis demonstrates that HyperAgent's regret order matches that of exact Thompson sampling in linear contextual bandits, closing a significant theoretical gap in scalable exploration. Empirical results in real-world contextual bandit tasks, such as automated content moderation with human feedback, validate the practical effectiveness of GPT-HyperAgent for safety-critical decisions. Our code is open-sourced at \url{this https URL}.

Title: NODER: Image Sequence Regression Based on Neural Ordinary Differential Equations

• Authors: Hao Bai, Yi Hong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13241
• Pdf URL: https://arxiv.org/pdf/2407.13241
• Copy Paste: [[2407.13241]] NODER: Image Sequence Regression Based on Neural Ordinary Differential Equations(https://arxiv.org/abs/2407.13241)
• Keywords: diffusion
• Abstract: Regression on medical image sequences can capture temporal image pattern changes and predict images at missing or future time points. However, existing geodesic regression methods limit their regression performance by a strong underlying assumption of linear dynamics, while diffusion-based methods have high computational costs and lack constraints to preserve image topology. In this paper, we propose an optimization-based new framework called NODER, which leverages neural ordinary differential equations to capture complex underlying dynamics and reduces its high computational cost of handling high-dimensional image volumes by introducing the latent space. We compare our NODER with two recent regression methods, and the experimental results on ADNI and ACDC datasets demonstrate that our method achieves the state-of-the-art performance in 3D image regression. Our model needs only a couple of images in a sequence for prediction, which is practical, especially for clinical situations where extremely limited image time series are available for analysis. Our source code is available at this https URL.

Title: Are Large Language Models Capable of Generating Human-Level Narratives?

• Authors: Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, Nanyun Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13248
• Pdf URL: https://arxiv.org/pdf/2407.13248
• Copy Paste: [[2407.13248]] Are Large Language Models Capable of Generating Human-Level Narratives?(https://arxiv.org/abs/2407.13248)
• Keywords: generative
• Abstract: This paper investigates the capability of LLMs in storytelling, focusing on narrative development and plot progression. We introduce a novel computational framework to analyze narratives through three discourse-level aspects: i) story arcs, ii) turning points, and iii) affective dimensions, including arousal and valence. By leveraging expert and automatic annotations, we uncover significant discrepancies between the LLM- and human- written stories. While human-written stories are suspenseful, arousing, and diverse in narrative structures, LLM stories are homogeneously positive and lack tension. Next, we measure narrative reasoning skills as a precursor to generative capacities, concluding that most LLMs fall short of human abilities in discourse understanding. Finally, we show that explicit integration of aforementioned discourse features can enhance storytelling, as is demonstrated by over 40% improvement in neural storytelling in terms of diversity, suspense, and arousal.

Title: Motif-Consistent Counterfactuals with Adversarial Refinement for Graph-Level Anomaly Detection

• Authors: Chunjing Xiao, Shikang Pang, Wenxin Tai, Yanlong Huang, Goce Trajcevski, Fan Zhou
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2407.13251
• Pdf URL: https://arxiv.org/pdf/2407.13251
• Copy Paste: [[2407.13251]] Motif-Consistent Counterfactuals with Adversarial Refinement for Graph-Level Anomaly Detection(https://arxiv.org/abs/2407.13251)
• Keywords: generative, anomaly
• Abstract: Graph-level anomaly detection is significant in diverse domains. To improve detection performance, counterfactual graphs have been exploited to benefit the generalization capacity by learning causal relations. Most existing studies directly introduce perturbations (e.g., flipping edges) to generate counterfactual graphs, which are prone to alter the semantics of generated examples and make them off the data manifold, resulting in sub-optimal performance. To address these issues, we propose a novel approach, Motif-consistent Counterfactuals with Adversarial Refinement (MotifCAR), for graph-level anomaly detection. The model combines the motif of one graph, the core subgraph containing the identification (category) information, and the contextual subgraph (non-motif) of another graph to produce a raw counterfactual graph. However, the produced raw graph might be distorted and cannot satisfy the important counterfactual properties: Realism, Validity, Proximity and Sparsity. Towards that, we present a Generative Adversarial Network (GAN)-based graph optimizer to refine the raw counterfactual graphs. It adopts the discriminator to guide the generator to generate graphs close to realistic data, i.e., meet the property Realism. Further, we design the motif consistency to force the motif of the generated graphs to be consistent with the realistic graphs, meeting the property Validity. Also, we devise the contextual loss and connection loss to control the contextual subgraph and the newly added links to meet the properties Proximity and Sparsity. As a result, the model can generate high-quality counterfactual graphs. Experiments demonstrate the superiority of MotifCAR.

Title: Unveiling Structural Memorization: Structural Membership Inference Attack for Text-to-Image Diffusion Models

• Authors: Qiao Li, Xiaomeng Fu, Xi Wang, Jin Liu, Xingyu Gao, Jiao Dai, Jizhong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13252
• Pdf URL: https://arxiv.org/pdf/2407.13252
• Copy Paste: [[2407.13252]] Unveiling Structural Memorization: Structural Membership Inference Attack for Text-to-Image Diffusion Models(https://arxiv.org/abs/2407.13252)
• Keywords: diffusion
• Abstract: With the rapid advancements of large-scale text-to-image diffusion models, various practical applications have emerged, bringing significant convenience to society. However, model developers may misuse the unauthorized data to train diffusion models. These data are at risk of being memorized by the models, thus potentially violating citizens' privacy rights. Therefore, in order to judge whether a specific image is utilized as a member of a model's training set, Membership Inference Attack (MIA) is proposed to serve as a tool for privacy protection. Current MIA methods predominantly utilize pixel-wise comparisons as distinguishing clues, considering the pixel-level memorization characteristic of diffusion models. However, it is practically impossible for text-to-image models to memorize all the pixel-level information in massive training sets. Therefore, we move to the more advanced structure-level memorization. Observations on the diffusion process show that the structures of members are better preserved compared to those of nonmembers, indicating that diffusion models possess the capability to remember the structures of member images from training sets. Drawing on these insights, we propose a simple yet effective MIA method tailored for text-to-image diffusion models. Extensive experimental results validate the efficacy of our approach. Compared to current pixel-level baselines, our approach not only achieves state-of-the-art performance but also demonstrates remarkable robustness against various distortions.

Title: SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learning

• Authors: Joseph Marvin Imperial, Harish Tayyar Madabushi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13297
• Pdf URL: https://arxiv.org/pdf/2407.13297
• Copy Paste: [[2407.13297]] SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learning(https://arxiv.org/abs/2407.13297)
• Keywords: in-context
• Abstract: Specialized lexicons are collections of words with associated constraints such as special definitions, specific roles, and intended target audiences. These constraints are necessary for content generation and documentation tasks (e.g., writing technical manuals or children's books), where the goal is to reduce the ambiguity of text content and increase its overall readability for a specific group of audience. Understanding how large language models can capture these constraints can help researchers build better, more impactful tools for wider use beyond the NLP community. Towards this end, we introduce SpeciaLex, a benchmark for evaluating a language model's ability to follow specialized lexicon-based constraints across 18 diverse subtasks with 1,285 test instances covering core tasks of Checking, Identification, Rewriting, and Open Generation. We present an empirical evaluation of 15 open and closed-source LLMs and discuss insights on how factors such as model scale, openness, setup, and recency affect performance upon evaluating with the benchmark.

Title: Learning-From-Mistakes Prompting for Indigenous Language Translation

• Authors: You-Cheng Liao, Chen-Jui Yu, Chi-Yi Lin, He-Feng Yun, Yen-Hsiang Wang, Hsiao-Min Li, Yao-Chung Fan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13343
• Pdf URL: https://arxiv.org/pdf/2407.13343
• Copy Paste: [[2407.13343]] Learning-From-Mistakes Prompting for Indigenous Language Translation(https://arxiv.org/abs/2407.13343)
• Keywords: in-context
• Abstract: Using large language models, this paper presents techniques to improve extremely low-resourced indigenous language translations. Our approaches are grounded in the use of (1) the presence of a datastore consisting of a limited number of parallel translation examples, (2) the inherent capabilities of LLMs like GPT-3.5, and (3) a word-level translation dictionary. We harness the potential of LLMs and in-context learning techniques in such a setting for using LLMs as universal translators for extremely low-resourced languages. Our methodology hinges on utilizing LLMs as language compilers for selected language pairs, hypothesizing that they could internalize syntactic structures to facilitate accurate translation. We introduce three techniques: KNNPrompting with Retrieved Prompting Context, Chain-of-Thought Prompting and Learningfrom-Mistakes Prompting, with the last method addressing past errors. The evaluation results suggest that, even with limited corpora, LLMs can effectively translate extremely low-resource languages when paired with proper prompting.

Title: EarlyMalDetect: A Novel Approach for Early Windows Malware Detection Based on Sequences of API Calls

• Authors: Pascal Maniriho, Abdun Naser Mahmood, Mohammad Jabed Morshed Chowdhury
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13355
• Pdf URL: https://arxiv.org/pdf/2407.13355
• Copy Paste: [[2407.13355]] EarlyMalDetect: A Novel Approach for Early Windows Malware Detection Based on Sequences of API Calls(https://arxiv.org/abs/2407.13355)
• Keywords: generative
• Abstract: In this work, we propose EarlyMalDetect, a novel approach for early Windows malware detection based on sequences of API calls. Our approach leverages generative transformer models and attention-guided deep recurrent neural networks to accurately identify and detect patterns of malicious behaviors in the early stage of malware execution. By analyzing the sequences of API calls invoked during execution, the proposed approach can classify executable files (programs) as malware or benign by predicting their behaviors based on a few shots (initial API calls) invoked during execution. EarlyMalDetect can predict and reveal what a malware program is going to perform on the target system before it occurs, which can help to stop it before executing its malicious payload and infecting the system. Specifically, EarlyMalDetect relies on a fine-tuned transformer model based on API calls which has the potential to predict the next API call functions to be used by a malware or benign executable program. Our extensive experimental evaluations show that the proposed approach is highly effective in predicting malware behaviors and can be used as a preventive measure against zero-day threats in Windows systems.

Title: Linear-Complexity Self-Supervised Learning for Speech Processing

• Authors: Shucong Zhang, Titouan Parcollet, Rogier van Dalen, Sourav Bhattacharya
• Subjects: cs.CL, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.13377
• Pdf URL: https://arxiv.org/pdf/2407.13377
• Copy Paste: [[2407.13377]] Linear-Complexity Self-Supervised Learning for Speech Processing(https://arxiv.org/abs/2407.13377)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) models usually require weeks of pre-training with dozens of high-end GPUs. These models typically have a multi-headed self-attention (MHSA) context encoder. However, MHSA takes quadratic time and space in the input length, contributing to the high pre-training cost. Linear-complexity alternatives to MHSA have been proposed. For instance, in supervised training, the SummaryMixing model is the first to outperform MHSA across multiple speech processing tasks. However, these cheaper alternatives have not been explored for SSL yet. This paper studies a linear-complexity context encoder for SSL for the first time. With better or equivalent performance for the downstream tasks of the MP3S benchmark, SummaryMixing reduces the pre-training time and peak VRAM of wav2vec 2.0 model by 18% and by 23%, respectively, leading to the pre-training of a 155M wav2vec 2.0 model finished within one week with 4 Tesla A100 GPUs. Code is available at this https URL.

Title: All Roads Lead to Rome? Exploring Representational Similarities Between Latent Spaces of Generative Image Models

• Authors: Charumathi Badrinath, Usha Bhalla, Alex Oesterling, Suraj Srinivas, Himabindu Lakkaraju
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13449
• Pdf URL: https://arxiv.org/pdf/2407.13449
• Copy Paste: [[2407.13449]] All Roads Lead to Rome? Exploring Representational Similarities Between Latent Spaces of Generative Image Models(https://arxiv.org/abs/2407.13449)
• Keywords: diffusion, generative
• Abstract: Do different generative image models secretly learn similar underlying representations? We investigate this by measuring the latent space similarity of four different models: VAEs, GANs, Normalizing Flows (NFs), and Diffusion Models (DMs). Our methodology involves training linear maps between frozen latent spaces to "stitch" arbitrary pairs of encoders and decoders and measuring output-based and probe-based metrics on the resulting "stitched'' models. Our main findings are that linear maps between latent spaces of performant models preserve most visual information even when latent sizes differ; for CelebA models, gender is the most similarly represented probe-able attribute. Finally we show on an NF that latent space representations converge early in training.

Title: SCAPE: A Simple and Strong Category-Agnostic Pose Estimator

• Authors: Yujia Liang, Zixuan Ye, Wenze Liu, Hao Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13483
• Pdf URL: https://arxiv.org/pdf/2407.13483
• Copy Paste: [[2407.13483]] SCAPE: A Simple and Strong Category-Agnostic Pose Estimator(https://arxiv.org/abs/2407.13483)
• Keywords: in-context
• Abstract: Category-Agnostic Pose Estimation (CAPE) aims to localize keypoints on an object of any category given few exemplars in an in-context manner. Prior arts involve sophisticated designs, e.g., sundry modules for similarity calculation and a two-stage framework, or takes in extra heatmap generation and supervision. We notice that CAPE is essentially a task about feature matching, which can be solved within the attention process. Therefore we first streamline the architecture into a simple baseline consisting of several pure self-attention layers and an MLP regression head -- this simplification means that one only needs to consider the attention quality to boost the performance of CAPE. Towards an effective attention process for CAPE, we further introduce two key modules: i) a global keypoint feature perceptor to inject global semantic information into support keypoints, and ii) a keypoint attention refiner to enhance inter-node correlation between keypoints. They jointly form a Simple and strong Category-Agnostic Pose Estimator (SCAPE). Experimental results show that SCAPE outperforms prior arts by 2.2 and 1.3 PCK under 1-shot and 5-shot settings with faster inference speed and lighter model capacity, excelling in both accuracy and efficiency. Code and models are available at this https URL

Title: Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?

• Authors: Stefanos-Iordanis Papadopoulos, Christos Koutlis, Symeon Papadopoulos, Panagiotis C. Petrantonakis
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.13488
• Pdf URL: https://arxiv.org/pdf/2407.13488
• Copy Paste: [[2407.13488]] Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?(https://arxiv.org/abs/2407.13488)
• Keywords: foundation model
• Abstract: Out-of-context (OOC) misinformation poses a significant challenge in multimodal fact-checking, where images are paired with texts that misrepresent their original context to support false narratives. Recent research in evidence-based OOC detection has seen a trend towards increasingly complex architectures, incorporating Transformers, foundation models, and large language models. In this study, we introduce a simple yet robust baseline, which assesses MUltimodal SimilaritiEs (MUSE), specifically the similarity between image-text pairs and external image and text evidence. Our results demonstrate that MUSE, when used with conventional classifiers like Decision Tree, Random Forest, and Multilayer Perceptron, can compete with and even surpass the state-of-the-art on the NewsCLIPpings and VERITE datasets. Furthermore, integrating MUSE in our proposed "Attentive Intermediate Transformer Representations" (AITR) significantly improved performance, by 3.3% and 7.5% on NewsCLIPpings and VERITE, respectively. Nevertheless, the success of MUSE, relying on surface-level patterns and shortcuts, without examining factuality and logical inconsistencies, raises critical questions about how we define the task, construct datasets, collect external evidence and overall, how we assess progress in the field. We release our code at: this https URL

Title: Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks

• Authors: Samy Ateia, Udo Kruschwitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13511
• Pdf URL: https://arxiv.org/pdf/2407.13511
• Copy Paste: [[2407.13511]] Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks(https://arxiv.org/abs/2407.13511)
• Keywords: in-context
• Abstract: Commercial large language models (LLMs), like OpenAI's GPT-4 powering ChatGPT and Anthropic's Claude 3 Opus, have dominated natural language processing (NLP) benchmarks across different domains. New competing Open-Source alternatives like Mixtral 8x7B or Llama 3 have emerged and seem to be closing the gap while often offering higher throughput and being less costly to use. Open-Source LLMs can also be self-hosted, which makes them interesting for enterprise and clinical use cases where sensitive data should not be processed by third parties. We participated in the 12th BioASQ challenge, which is a retrieval augmented generation (RAG) setting, and explored the performance of current GPT models Claude 3 Opus, GPT-3.5-turbo and Mixtral 8x7b with in-context learning (zero-shot, few-shot) and QLoRa fine-tuning. We also explored how additional relevant knowledge from Wikipedia added to the context-window of the LLM might improve their performance. Mixtral 8x7b was competitive in the 10-shot setting, both with and without fine-tuning, but failed to produce usable results in the zero-shot setting. QLoRa fine-tuning and Wikipedia context did not lead to measurable performance gains. Our results indicate that the performance gap between commercial and open-source models in RAG setups exists mainly in the zero-shot setting and can be closed by simply collecting few-shot examples for domain-specific use cases. The code needed to rerun these experiments is available through GitHub.

Title: EnergyDiff: Universal Time-Series Energy Data Generation using Diffusion Models

• Authors: Nan Lin, Peter Palensky, Pedro P. Vergara
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2407.13538
• Pdf URL: https://arxiv.org/pdf/2407.13538
• Copy Paste: [[2407.13538]] EnergyDiff: Universal Time-Series Energy Data Generation using Diffusion Models(https://arxiv.org/abs/2407.13538)
• Keywords: diffusion, generative
• Abstract: High-resolution time series data are crucial for operation and planning in energy systems such as electrical power systems and heating systems. However, due to data collection costs and privacy concerns, such data is often unavailable or insufficient for downstream tasks. Data synthesis is a potential solution for this data scarcity. With the recent development of generative AI, we propose EnergyDiff, a universal data generation framework for energy time series data. EnergyDiff builds on state-of-the-art denoising diffusion probabilistic models, utilizing a proposed denoising network dedicated to high-resolution time series data and introducing a novel Marginal Calibration technique. Our extensive experimental results demonstrate that EnergyDiff achieves significant improvement in capturing temporal dependencies and marginal distributions compared to baselines, particularly at the 1-minute resolution. Additionally, EnergyDiff consistently generates high-quality time series data across diverse energy domains, time resolutions, and at both customer and transformer levels with reduced computational need.

Title: On the Discriminability of Self-Supervised Representation Learning

• Authors: Zeen Song, Wenwen Qiang, Changwen Zheng, Fuchun Sun, Hui Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13541
• Pdf URL: https://arxiv.org/pdf/2407.13541
• Copy Paste: [[2407.13541]] On the Discriminability of Self-Supervised Representation Learning(https://arxiv.org/abs/2407.13541)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) has recently achieved significant success in downstream visual tasks. However, a notable gap still exists between SSL and supervised learning (SL), especially in complex downstream tasks. In this paper, we show that the features learned by SSL methods suffer from the crowding problem, where features of different classes are not distinctly separated, and features within the same class exhibit large intra-class variance. In contrast, SL ensures a clear separation between classes. We analyze this phenomenon and conclude that SSL objectives do not constrain the relationships between different samples and their augmentations. Our theoretical analysis delves into how SSL objectives fail to enforce the necessary constraints between samples and their augmentations, leading to poor performance in complex tasks. We provide a theoretical framework showing that the performance gap between SSL and SL mainly stems from the inability of SSL methods to capture the aggregation of similar augmentations and the separation of dissimilar augmentations. To address this issue, we propose a learnable regulator called Dynamic Semantic Adjuster (DSA). DSA aggregates and separates samples in the feature space while being robust to outliers. Through extensive empirical evaluations on multiple benchmark datasets, we demonstrate the superiority of DSA in enhancing feature aggregation and separation, ultimately closing the performance gap between SSL and SL.

Title: SAM-Driven Weakly Supervised Nodule Segmentation with Uncertainty-Aware Cross Teaching

• Authors: Xingyue Zhao, Peiqi Li, Xiangde Luo, Meng Yang, Shi Chang, Zhongyu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13553
• Pdf URL: https://arxiv.org/pdf/2407.13553
• Copy Paste: [[2407.13553]] SAM-Driven Weakly Supervised Nodule Segmentation with Uncertainty-Aware Cross Teaching(https://arxiv.org/abs/2407.13553)
• Keywords: foundation model
• Abstract: Automated nodule segmentation is essential for computer-assisted diagnosis in ultrasound images. Nevertheless, most existing methods depend on precise pixel-level annotations by medical professionals, a process that is both costly and labor-intensive. Recently, segmentation foundation models like SAM have shown impressive generalizability on natural images, suggesting their potential as pseudo-labelers. However, accurate prompts remain crucial for their success in medical images. In this work, we devise a novel weakly supervised framework that effectively utilizes the segmentation foundation model to generate pseudo-labels from aspect ration annotations for automatic nodule segmentation. Specifically, we develop three types of bounding box prompts based on scalable shape priors, followed by an adaptive pseudo-label selection module to fully exploit the prediction capabilities of the foundation model for nodules. We also present a SAM-driven uncertainty-aware cross-teaching strategy. This approach integrates SAM-based uncertainty estimation and label-space perturbations into cross-teaching to mitigate the impact of pseudo-label inaccuracies on model training. Extensive experiments on two clinically collected ultrasound datasets demonstrate the superior performance of our proposed method.

Title: Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition

• Authors: Gagan Bhatia, El Moatez Billah Nagoudi, Fakhraddin Alwajih, Muhammad Abdul-Mageed
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13559
• Pdf URL: https://arxiv.org/pdf/2407.13559
• Copy Paste: [[2407.13559]] Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition(https://arxiv.org/abs/2407.13559)
• Keywords: foundation model
• Abstract: Arabic Optical Character Recognition (OCR) and Handwriting Recognition (HWR) pose unique challenges due to the cursive and context-sensitive nature of the Arabic script. This study introduces Qalam, a novel foundation model designed for Arabic OCR and HWR, built on a SwinV2 encoder and RoBERTa decoder architecture. Our model significantly outperforms existing methods, achieving a Word Error Rate (WER) of just 0.80% in HWR tasks and 1.18% in OCR tasks. We train Qalam on a diverse dataset, including over 4.5 million images from Arabic manuscripts and a synthetic dataset comprising 60k image-text pairs. Notably, Qalam demonstrates exceptional handling of Arabic diacritics, a critical feature in Arabic scripts. Furthermore, it shows a remarkable ability to process high-resolution inputs, addressing a common limitation in current OCR systems. These advancements underscore Qalam's potential as a leading solution for Arabic script recognition, offering a significant leap in accuracy and efficiency.

Title: Research on Tibetan Tourism Viewpoints information generation system based on LLM

• Authors: Jinhu Qi, Shuai Yan, Wentao Zhang, Yibo Zhang, Zirui Liu, Ke Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13561
• Pdf URL: https://arxiv.org/pdf/2407.13561
• Copy Paste: [[2407.13561]] Research on Tibetan Tourism Viewpoints information generation system based on LLM(https://arxiv.org/abs/2407.13561)
• Keywords: generative
• Abstract: Tibet, ensconced within China's territorial expanse, is distinguished by its labyrinthine and heterogeneous topography, a testament to its profound historical heritage, and the cradle of a unique religious ethos. The very essence of these attributes, however, has impeded the advancement of Tibet's tourism service infrastructure, rendering existing smart tourism services inadequate for the region's visitors. This study delves into the ramifications of informational disparities at tourist sites on Tibetan tourism and addresses the challenge of establishing the Large Language Model (LLM) evaluation criteria. It introduces an innovative approach, the DualGen Bridge AI system, employing supervised fine-tuning techniques to bolster model functionality and enhance optimization processes. Furthermore, it pioneers a multi-structured generative results assessment framework. Empirical validation confirms the efficacy of this framework. The study also explores the application of the supervised fine-tuning method within the proprietary DualGen Bridge AI, aimed at refining the generation of tourist site information. The study's findings offer valuable insights for optimizing system performance and provide support and inspiration for the application of LLM technology in Tibet's tourism services and beyond, potentially revolutionizing the smart tourism industry with advanced, tailored information generation capabilities.

Title: Large Language Models as Reliable Knowledge Bases?

• Authors: Danna Zheng, Mirella Lapata, Jeff Z. Pan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13578
• Pdf URL: https://arxiv.org/pdf/2407.13578
• Copy Paste: [[2407.13578]] Large Language Models as Reliable Knowledge Bases?(https://arxiv.org/abs/2407.13578)
• Keywords: in-context
• Abstract: The NLP community has recently shown a growing interest in leveraging Large Language Models (LLMs) for knowledge-intensive tasks, viewing LLMs as potential knowledge bases (KBs). However, the reliability and extent to which LLMs can function as KBs remain underexplored. While previous studies suggest LLMs can encode knowledge within their parameters, the amount of parametric knowledge alone is not sufficient to evaluate their effectiveness as KBs. This study defines criteria that a reliable LLM-as-KB should meet, focusing on factuality and consistency, and covering both seen and unseen knowledge. We develop several metrics based on these criteria and use them to evaluate 26 popular LLMs, while providing a comprehensive analysis of the effects of model size, instruction tuning, and in-context learning (ICL). Our results paint a worrying picture. Even a high-performant model like GPT-3.5-turbo is not factual or consistent, and strategies like ICL and fine-tuning are unsuccessful at making LLMs better KBs.

Title: Training-free Composite Scene Generation for Layout-to-Image Synthesis

• Authors: Jiaqi Liu, Tao Huang, Chang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13609
• Pdf URL: https://arxiv.org/pdf/2407.13609
• Copy Paste: [[2407.13609]] Training-free Composite Scene Generation for Layout-to-Image Synthesis(https://arxiv.org/abs/2407.13609)
• Keywords: diffusion
• Abstract: Recent breakthroughs in text-to-image diffusion models have significantly advanced the generation of high-fidelity, photo-realistic images from textual descriptions. Yet, these models often struggle with interpreting spatial arrangements from text, hindering their ability to produce images with precise spatial configurations. To bridge this gap, layout-to-image generation has emerged as a promising direction. However, training-based approaches are limited by the need for extensively annotated datasets, leading to high data acquisition costs and a constrained conceptual scope. Conversely, training-free methods face challenges in accurately locating and generating semantically similar objects within complex compositions. This paper introduces a novel training-free approach designed to overcome adversarial semantic intersections during the diffusion conditioning phase. By refining intra-token loss with selective sampling and enhancing the diffusion process with attention redistribution, we propose two innovative constraints: 1) an inter-token constraint that resolves token conflicts to ensure accurate concept synthesis; and 2) a self-attention constraint that improves pixel-to-pixel relationships. Our evaluations confirm the effectiveness of leveraging layout information for guiding the diffusion process, generating content-rich images with enhanced fidelity and complexity. Code is available at this https URL.

Title: Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

• Authors: Xiaoyu Zhu, Hao Zhou, Pengfei Xing, Long Zhao, Hao Xu, Junwei Liang, Alexander Hauptmann, Ting Liu, Andrew Gallagher
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13642
• Pdf URL: https://arxiv.org/pdf/2407.13642
• Copy Paste: [[2407.13642]] Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models(https://arxiv.org/abs/2407.13642)
• Keywords: diffusion, generative
• Abstract: In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%.

Title: MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis

• Authors: Ziming Zhong, Yanxu Xu, Jing Li, Jiale Xu, Zhengxin Li, Chaohui Yu, Shenghua Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13675
• Pdf URL: https://arxiv.org/pdf/2407.13675
• Copy Paste: [[2407.13675]] MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis(https://arxiv.org/abs/2407.13675)
• Keywords: diffusion
• Abstract: We present MeshSegmenter, a simple yet effective framework designed for zero-shot 3D semantic segmentation. This model successfully extends the powerful capabilities of 2D segmentation models to 3D meshes, delivering accurate 3D segmentation across diverse meshes and segment descriptions. Specifically, our model leverages the Segment Anything Model (SAM) model to segment the target regions from images rendered from the 3D shape. In light of the importance of the texture for segmentation, we also leverage the pretrained stable diffusion model to generate images with textures from 3D shape, and leverage SAM to segment the target regions from images with textures. Textures supplement the shape for segmentation and facilitate accurate 3D segmentation even in geometrically non-prominent areas, such as segmenting a car door within a car mesh. To achieve the 3D segments, we render 2D images from different views and conduct segmentation for both textured and untextured images. Lastly, we develop a multi-view revoting scheme that integrates 2D segmentation results and confidence scores from various views onto the 3D mesh, ensuring the 3D consistency of segmentation results and eliminating inaccuracies from specific perspectives. Through these innovations, MeshSegmenter offers stable and reliable 3D segmentation results both quantitatively and qualitatively, highlighting its potential as a transformative tool in the field of 3D zero-shot segmentation. The code is available at \url{this https URL}.

Title: PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers

• Authors: Songlin Li, Despoina Paschalidou, Leonidas Guibas
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13677
• Pdf URL: https://arxiv.org/pdf/2407.13677
• Copy Paste: [[2407.13677]] PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers(https://arxiv.org/abs/2407.13677)
• Keywords: generative
• Abstract: The increased demand for tools that automate the 3D content creation process led to tremendous progress in deep generative models that can generate diverse 3D objects of high fidelity. In this paper, we present PASTA, an autoregressive transformer architecture for generating high quality 3D shapes. PASTA comprises two main components: An autoregressive transformer that generates objects as a sequence of cuboidal primitives and a blending network, implemented with a transformer decoder that composes the sequences of cuboids and synthesizes high quality meshes for each object. Our model is trained in two stages: First we train our autoregressive generative model using only annotated cuboidal parts as supervision and next, we train our blending network using explicit 3D supervision, in the form of watertight meshes. Evaluations on various ShapeNet objects showcase the ability of our model to perform shape generation from diverse inputs \eg from scratch, from a partial object, from text and images, as well size-guided generation, by explicitly conditioning on a bounding box that defines the object's boundaries. Moreover, as our model considers the underlying part-based structure of a 3D object, we are able to select a specific part and produce shapes with meaningful variations of this part. As evidenced by our experiments, our model generates 3D shapes that are both more realistic and diverse than existing part-based and non part-based methods, while at the same time is simpler to implement and train.

Title: HPix: Generating Vector Maps from Satellite Images

• Authors: Aditya Taparia, Keshab Nath
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.13680
• Pdf URL: https://arxiv.org/pdf/2407.13680
• Copy Paste: [[2407.13680]] HPix: Generating Vector Maps from Satellite Images(https://arxiv.org/abs/2407.13680)
• Keywords: generative
• Abstract: Vector maps find widespread utility across diverse domains due to their capacity to not only store but also represent discrete data boundaries such as building footprints, disaster impact analysis, digitization, urban planning, location points, transport links, and more. Although extensive research exists on identifying building footprints and road types from satellite imagery, the generation of vector maps from such imagery remains an area with limited exploration. Furthermore, conventional map generation techniques rely on labor-intensive manual feature extraction or rule-based approaches, which impose inherent limitations. To surmount these limitations, we propose a novel method called HPix, which utilizes modified Generative Adversarial Networks (GANs) to generate vector tile map from satellite images. HPix incorporates two hierarchical frameworks: one operating at the global level and the other at the local level, resulting in a comprehensive model. Through empirical evaluations, our proposed approach showcases its effectiveness in producing highly accurate and visually captivating vector tile maps derived from satellite images. We further extend our study's application to include mapping of road intersections and building footprints cluster based on their area.

Title: Cross-Task Attack: A Self-Supervision Generative Framework Based on Attention Shift

• Authors: Qingyuan Zeng, Yunpeng Gong, Min Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13700
• Pdf URL: https://arxiv.org/pdf/2407.13700
• Copy Paste: [[2407.13700]] Cross-Task Attack: A Self-Supervision Generative Framework Based on Attention Shift(https://arxiv.org/abs/2407.13700)
• Keywords: self-supervised, generative
• Abstract: Studying adversarial attacks on artificial intelligence (AI) systems helps discover model shortcomings, enabling the construction of a more robust system. Most existing adversarial attack methods only concentrate on single-task single-model or single-task cross-model scenarios, overlooking the multi-task characteristic of artificial intelligence systems. As a result, most of the existing attacks do not pose a practical threat to a comprehensive and collaborative AI system. However, implementing cross-task attacks is highly demanding and challenging due to the difficulty in obtaining the real labels of different tasks for the same picture and harmonizing the loss functions across different tasks. To address this issue, we propose a self-supervised Cross-Task Attack framework (CTA), which utilizes co-attention and anti-attention maps to generate cross-task adversarial perturbation. Specifically, the co-attention map reflects the area to which different visual task models pay attention, while the anti-attention map reflects the area that different visual task models neglect. CTA generates cross-task perturbations by shifting the attention area of samples away from the co-attention map and closer to the anti-attention map. We conduct extensive experiments on multiple vision tasks and the experimental results confirm the effectiveness of the proposed design for adversarial attacks.

Title: Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review

• Authors: Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, Sergey Levine
• Subjects: cs.LG, cs.AI, q-bio.QM, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.13734
• Pdf URL: https://arxiv.org/pdf/2407.13734
• Copy Paste: [[2407.13734]] Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review(https://arxiv.org/abs/2407.13734)
• Keywords: diffusion, generative
• Abstract: This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. While diffusion models are widely known to provide excellent generative modeling capability, practical applications in domains such as biology require generating samples that maximize some desired metric (e.g., translation efficiency in RNA, docking score in molecules, stability in protein). In these cases, the diffusion model can be optimized not only to generate realistic samples but also to explicitly maximize the measure of interest. Such methods are based on concepts from reinforcement learning (RL). We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning, tailored specifically for fine-tuning diffusion models. We aim to explore fundamental aspects such as the strengths and limitations of different RL-based fine-tuning algorithms across various scenarios, the benefits of RL-based fine-tuning compared to non-RL-based approaches, and the formal objectives of RL-based fine-tuning (target distributions). Additionally, we aim to examine their connections with related topics such as classifier guidance, Gflownets, flow-based diffusion models, path integral control theory, and sampling from unnormalized distributions such as MCMC. The code of this tutorial is available at this https URL

Title: LogoSticker: Inserting Logos into Diffusion Models for Customized Generation

• Authors: Mingkang Zhu, Xi Chen, Zhongdao Wang, Hengshuang Zhao, Jiaya Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13752
• Pdf URL: https://arxiv.org/pdf/2407.13752
• Copy Paste: [[2407.13752]] LogoSticker: Inserting Logos into Diffusion Models for Customized Generation(https://arxiv.org/abs/2407.13752)
• Keywords: diffusion
• Abstract: Recent advances in text-to-image model customization have underscored the importance of integrating new concepts with a few examples. Yet, these progresses are largely confined to widely recognized subjects, which can be learned with relative ease through models' adequate shared prior knowledge. In contrast, logos, characterized by unique patterns and textual elements, are hard to establish shared knowledge within diffusion models, thus presenting a unique challenge. To bridge this gap, we introduce the task of logo insertion. Our goal is to insert logo identities into diffusion models and enable their seamless synthesis in varied contexts. We present a novel two-phase pipeline LogoSticker to tackle this task. First, we propose the actor-critic relation pre-training algorithm, which addresses the nontrivial gaps in models' understanding of the potential spatial positioning of logos and interactions with other objects. Second, we propose a decoupled identity learning algorithm, which enables precise localization and identity extraction of logos. LogoSticker can generate logos accurately and harmoniously in diverse contexts. We comprehensively validate the effectiveness of LogoSticker over customization methods and large models such as DALLE~3. \href{this https URL}{Project page}.

Title: Black-Box Opinion Manipulation Attacks to Retrieval-Augmented Generation of Large Language Models

• Authors: Zhuo Chen, Jiawei Liu, Haotan Liu, Qikai Cheng, Fan Zhang, Wei Lu, Xiaozhong Liu
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13757
• Pdf URL: https://arxiv.org/pdf/2407.13757
• Copy Paste: [[2407.13757]] Black-Box Opinion Manipulation Attacks to Retrieval-Augmented Generation of Large Language Models(https://arxiv.org/abs/2407.13757)
• Keywords: generative
• Abstract: Retrieval-Augmented Generation (RAG) is applied to solve hallucination problems and real-time constraints of large language models, but it also induces vulnerabilities against retrieval corruption attacks. Existing research mainly explores the unreliability of RAG in white-box and closed-domain QA tasks. In this paper, we aim to reveal the vulnerabilities of Retrieval-Enhanced Generative (RAG) models when faced with black-box attacks for opinion manipulation. We explore the impact of such attacks on user cognition and decision-making, providing new insight to enhance the reliability and security of RAG models. We manipulate the ranking results of the retrieval model in RAG with instruction and use these results as data to train a surrogate model. By employing adversarial retrieval attack methods to the surrogate model, black-box transfer attacks on RAG are further realized. Experiments conducted on opinion datasets across multiple topics show that the proposed attack strategy can significantly alter the opinion polarity of the content generated by RAG. This demonstrates the model's vulnerability and, more importantly, reveals the potential negative impact on user cognition and decision-making, making it easier to mislead users into accepting incorrect or biased information.

Title: Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion

• Authors: Boyang Deng, Richard Tucker, Zhengqi Li, Leonidas Guibas, Noah Snavely, Gordon Wetzstein
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2407.13759
• Pdf URL: https://arxiv.org/pdf/2407.13759
• Copy Paste: [[2407.13759]] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion(https://arxiv.org/abs/2407.13759)
• Keywords: diffusion
• Abstract: We present a method for generating Streetscapes-long sequences of views through an on-the-fly synthesized city-scale scene. Our generation is conditioned by language input (e.g., city name, weather), as well as an underlying map/layout hosting the desired trajectory. Compared to recent models for video generation or 3D view synthesis, our method can scale to much longer-range camera trajectories, spanning several city blocks, while maintaining visual quality and consistency. To achieve this goal, we build on recent work on video diffusion, used within an autoregressive framework that can easily scale to long sequences. In particular, we introduce a new temporal imputation method that prevents our autoregressive approach from drifting from the distribution of realistic city imagery. We train our Streetscapes system on a compelling source of data-posed imagery from Google Street View, along with contextual map data-which allows users to generate city views conditioned on any desired city layout, with controllable camera poses. Please see more results at our project page at this https URL.