2024-04-09

Title: The Future of MEV

• Authors: Jonah Burian
• Subjects: cs.CR, cs.GT
• Abstract URL: https://arxiv.org/abs/2404.04262
• Pdf URL: https://arxiv.org/pdf/2404.04262
• Copy Paste: [[2404.04262]] The Future of MEV(https://arxiv.org/abs/2404.04262)
• Keywords: secure, robust
• Abstract: This paper analyzes the Execution Tickets proposal on Ethereum Research, unveiling its potential to revolutionize the Ethereum blockchain's economic model. At the core of this proposal lies a novel ticketing mechanism poised to redefine how the Ethereum protocol distributes the value associated with proposing execution payloads. This innovative approach enables the Ethereum protocol to directly broker Maximal Extractable Value (MEV), traditionally an external revenue stream for validators. The implementation of Execution Tickets goes beyond optimizing validator compensation; it also introduces a new Ethereum native asset with a market capitalization expected to correlate closely with the present value of all value associated with future block production. The analysis demonstrates that the Execution Ticket system can facilitate a more equitable distribution of value within the Ethereum ecosystem, and pave the way for a more secure and economically robust blockchain network.

Title: Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights

• Authors: Xianlong Zeng, Fanghao Song, Ang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04281
• Pdf URL: https://arxiv.org/pdf/2404.04281
• Copy Paste: [[2404.04281]] Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights(https://arxiv.org/abs/2404.04281)
• Keywords: extraction, generative, large language model
• Abstract: This study introduces a simple yet effective method for identifying similar data points across non-free text domains, such as tabular and image data, using Large Language Models (LLMs). Our two-step approach involves data point summarization and hidden state extraction. Initially, data is condensed via summarization using an LLM, reducing complexity and highlighting essential information in sentences. Subsequently, the summarization sentences are fed through another LLM to extract hidden states, serving as compact, feature-rich representations. This approach leverages the advanced comprehension and generative capabilities of LLMs, offering a scalable and efficient strategy for similarity identification across diverse datasets. We demonstrate the effectiveness of our method in identifying similar data points on multiple datasets. Additionally, our approach enables non-technical domain experts, such as fraud investigators or marketing operators, to quickly identify similar data points tailored to specific scenarios, demonstrating its utility in practical applications. In general, our results open new avenues for leveraging LLMs in data analysis across various domains.

Title: Translation-based Video-to-Video Synthesis

• Authors: Pratim Saha, Chengcui Zhang
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.04283
• Pdf URL: https://arxiv.org/pdf/2404.04283
• Copy Paste: [[2404.04283]] Translation-based Video-to-Video Synthesis(https://arxiv.org/abs/2404.04283)
• Keywords: segmentation
• Abstract: Translation-based Video Synthesis (TVS) has emerged as a vital research area in computer vision, aiming to facilitate the transformation of videos between distinct domains while preserving both temporal continuity and underlying content features. This technique has found wide-ranging applications, encompassing video super-resolution, colorization, segmentation, and more, by extending the capabilities of traditional image-to-image translation to the temporal domain. One of the principal challenges faced in TVS is the inherent risk of introducing flickering artifacts and inconsistencies between frames during the synthesis process. This is particularly challenging due to the necessity of ensuring smooth and coherent transitions between video frames. Efforts to tackle this challenge have induced the creation of diverse strategies and algorithms aimed at mitigating these unwanted consequences. This comprehensive review extensively examines the latest progress in the realm of TVS. It thoroughly investigates emerging methodologies, shedding light on the fundamental concepts and mechanisms utilized for proficient video synthesis. This survey also illuminates their inherent strengths, limitations, appropriate applications, and potential avenues for future development.

Title: MIMIR: A Streamlined Platform for Personalized Agent Tuning in Domain Expertise

• Authors: Chunyuan Deng, Xiangru Tang, Yilun Zhao, Hanming Wang, Haoran Wang, Wangchunshu Zhou, Arman Cohan, Mark Gerstein
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04285
• Pdf URL: https://arxiv.org/pdf/2404.04285
• Copy Paste: [[2404.04285]] MIMIR: A Streamlined Platform for Personalized Agent Tuning in Domain Expertise(https://arxiv.org/abs/2404.04285)
• Keywords: large language model
• Abstract: Recently, large language models (LLMs) have evolved into interactive agents, proficient in planning, tool use, and task execution across a wide variety of tasks. However, without specific agent tuning, open-source models like LLaMA currently struggle to match the efficiency of GPT- 4, particularly given the scarcity of agent-tuning datasets for fine-tuning. In response, we introduce \textsc{Mimir}: a streamlined platform offering a customizable pipeline that enables users to leverage both private knowledge and publicly available, legally compliant datasets at scale for \textbf{personalized agent tuning}. Additionally, \textsc{Mimir} supports the generation of general instruction-tuning datasets from the same input. This dual capability ensures that language agents developed through the platform possess both specific agent abilities and general competencies. \textsc{Mimir} integrates these features into a cohesive end-to-end platform, facilitating everything from the uploading of personalized files to one-click agent fine-tuning.

Title: Language Model Evolution: An Iterated Learning Perspective

• Authors: Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, Danica J. Sutherland
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04286
• Pdf URL: https://arxiv.org/pdf/2404.04286
• Copy Paste: [[2404.04286]] Language Model Evolution: An Iterated Learning Perspective(https://arxiv.org/abs/2404.04286)
• Keywords: large language model
• Abstract: With the widespread adoption of Large Language Models (LLMs), the prevalence of iterative interactions among these models is anticipated to increase. Notably, recent advancements in multi-round self-improving methods allow LLMs to generate new examples for training subsequent models. At the same time, multi-agent LLM systems, involving automated interactions among agents, are also increasing in prominence. Thus, in both short and long terms, LLMs may actively engage in an evolutionary process. We draw parallels between the behavior of LLMs and the evolution of human culture, as the latter has been extensively studied by cognitive scientists for decades. Our approach involves leveraging Iterated Learning (IL), a Bayesian framework that elucidates how subtle biases are magnified during human cultural evolution, to explain some behaviors of LLMs. This paper outlines key characteristics of agents' behavior in the Bayesian-IL framework, including predictions that are supported by experimental verification with various LLMs. This theoretical framework could help to more effectively predict and guide the evolution of LLMs in desired directions.

Title: CONFLARE: CONFormal LArge language model REtrieval

• Authors: Pouria Rouzrokh, Shahriar Faghani, Cooper U. Gamble, Moein Shariatnia, Bradley J. Erickson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04287
• Pdf URL: https://arxiv.org/pdf/2404.04287
• Copy Paste: [[2404.04287]] CONFLARE: CONFormal LArge language model REtrieval(https://arxiv.org/abs/2404.04287)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) frameworks enable large language models (LLMs) to retrieve relevant information from a knowledge base and incorporate it into the context for generating responses. This mitigates hallucinations and allows for the updating of knowledge without retraining the LLM. However, RAG does not guarantee valid responses if retrieval fails to identify the necessary information as the context for response generation. Also, if there is contradictory content, the RAG response will likely reflect only one of the two possible responses. Therefore, quantifying uncertainty in the retrieval process is crucial for ensuring RAG trustworthiness. In this report, we introduce a four-step framework for applying conformal prediction to quantify retrieval uncertainty in RAG frameworks. First, a calibration set of questions answerable from the knowledge base is constructed. Each question's embedding is compared against document embeddings to identify the most relevant document chunks containing the answer and record their similarity scores. Given a user-specified error rate ({\alpha}), these similarity scores are then analyzed to determine a similarity score cutoff threshold. During inference, all chunks with similarity exceeding this threshold are retrieved to provide context to the LLM, ensuring the true answer is captured in the context with a (1-{\alpha}) confidence level. We provide a Python package that enables users to implement the entire workflow proposed in our work, only using LLMs and without human intervention.

Title: Conversational Disease Diagnosis via External Planner-Controlled Large Language Models

• Authors: Zhoujian Sun, Cheng Luo, Zhengxing Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04292
• Pdf URL: https://arxiv.org/pdf/2404.04292
• Copy Paste: [[2404.04292]] Conversational Disease Diagnosis via External Planner-Controlled Large Language Models(https://arxiv.org/abs/2404.04292)
• Keywords: large language model
• Abstract: The advancement of medical artificial intelligence (AI) has set the stage for the realization of conversational diagnosis, where AI systems mimic human doctors by engaging in dialogue with patients to deduce diagnoses. This study introduces an innovative approach using external planners augmented with large language models (LLMs) to develop a medical task-oriented dialogue system. This system comprises a policy module for information gathering, a LLM based module for natural language understanding and generation, addressing the limitations of previous AI systems in these areas. By emulating the two-phase decision-making process of doctors disease screening and differential diagnosis. we designed two distinct planners. The first focuses on collecting patient symptoms to identify potential diseases, while the second delves into specific inquiries to confirm or exclude these diseases. Utilizing reinforcement learning and active learning with LLMs, we trained these planners to navigate medical dialogues effectively. Our evaluation on the MIMIC-IV dataset demonstrated the system's capability to outperform existing models, indicating a significant step towards achieving automated conversational disease diagnostics and enhancing the precision and accessibility of medical diagnoses.

Title: Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding

• Authors: Yanda Li, Dixuan Wang, Jiaqing Liang, Guochao Jiang, Qianyu He, Yanghua Xiao, Deqing Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04293
• Pdf URL: https://arxiv.org/pdf/2404.04293
• Copy Paste: [[2404.04293]] Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding(https://arxiv.org/abs/2404.04293)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated good performance in many reasoning tasks, but they still struggle with some complicated reasoning tasks including logical reasoning. One non-negligible reason for LLMs' suboptimal performance on logical reasoning is their overlooking of understanding logical fallacies correctly. To evaluate LLMs' capability of logical fallacy understanding (LFU), we propose five concrete tasks from three cognitive dimensions of WHAT, WHY, and HOW in this paper. Towards these LFU tasks, we have successfully constructed a new dataset LFUD based on GPT-4 accompanied by a little human effort. Our extensive experiments justify that our LFUD can be used not only to evaluate LLMs' LFU capability, but also to fine-tune LLMs to obtain significantly enhanced performance on logical reasoning.

Title: ProLoc: Robust Location Proofs in Hindsight

• Authors: Roberta De Viti, Pierfrancesco Ingo, Isaac Sheff, Peter Druschel, Deepak Garg
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04297
• Pdf URL: https://arxiv.org/pdf/2404.04297
• Copy Paste: [[2404.04297]] ProLoc: Robust Location Proofs in Hindsight(https://arxiv.org/abs/2404.04297)
• Keywords: attack, robust
• Abstract: Many online services rely on self-reported locations of user devices like smartphones. To mitigate harm from falsified self-reported locations, the literature has proposed location proof services (LPSs), which provide proof of a device's location by corroborating its self-reported location using short-range radio contacts with either trusted infrastructure or nearby devices that also report their locations. This paper presents ProLoc, a new LPS that extends prior work in two ways. First, ProLoc relaxes prior work's proofs that a device was at a given location to proofs that a device was within distance "d" of a given location. We argue that these weaker proofs, which we call "region proofs", are important because (i) region proofs can be constructed with few requirements on device reporting behavior as opposed to precise location proofs, and (ii) a quantitative bound on a device's distance from a known epicenter is useful for many applications. For example, in the context of citizen reporting near an unexpected event (earthquake, violent protest, etc.), knowing the verified distances of the reporting devices from the event's epicenter would be valuable for ranking the reports by relevance or flagging fake reports. Second, ProLoc includes a novel mechanism to prevent collusion attacks where a set of attacker-controlled devices corroborate each others' false locations. Ours is the first mechanism that does not need additional infrastructure to handle attacks with made-up devices, which an attacker can create in any number at any location without any cost. For this, we rely on a variant of TrustRank applied to the self-reported trajectories and encounters of devices. Our goal is to prevent retroactive attacks where the adversary cannot predict ahead of time which fake location it will want to report, which is the case for the reporting of unexpected events.

Title: CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering

• Authors: Nirmalie Wiratunga, Ramitha Abeyratne, Lasal Jayawardena, Kyle Martin, Stewart Massie, Ikechukwu Nkisi-Orji, Ruvan Weerasinghe, Anne Liret, Bruno Fleisch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04302
• Pdf URL: https://arxiv.org/pdf/2404.04302
• Copy Paste: [[2404.04302]] CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering(https://arxiv.org/abs/2404.04302)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) enhances Large Language Model (LLM) output by providing prior knowledge as context to input. This is beneficial for knowledge-intensive and expert reliant tasks, including legal question-answering, which require evidence to validate generated text outputs. We highlight that Case-Based Reasoning (CBR) presents key opportunities to structure retrieval as part of the RAG process in an LLM. We introduce CBR-RAG, where CBR cycle's initial retrieval stage, its indexing vocabulary, and similarity knowledge containers are used to enhance LLM queries with contextually relevant cases. This integration augments the original LLM query, providing a richer prompt. We present an evaluation of CBR-RAG, and examine different representations (i.e. general and domain-specific embeddings) and methods of comparison (i.e. inter, intra and hybrid similarity) on the task of legal question-answering. Our results indicate that the context provided by CBR's case reuse enforces similarity between relevant components of the questions and the evidence base leading to significant improvements in the quality of generated answers.

Title: AuditGPT: Auditing Smart Contracts with ChatGPT

• Authors: Shihao Xia, Shuai Shao, Mengting He, Tingting Yu, Linhai Song, Yiying Zhang
• Subjects: cs.CR, cs.AI, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.04306
• Pdf URL: https://arxiv.org/pdf/2404.04306
• Copy Paste: [[2404.04306]] AuditGPT: Auditing Smart Contracts with ChatGPT(https://arxiv.org/abs/2404.04306)
• Keywords: security, large language model
• Abstract: To govern smart contracts running on Ethereum, multiple Ethereum Request for Comment (ERC) standards have been developed, each containing a set of rules to guide the behaviors of smart contracts. Violating the ERC rules could cause serious security issues and financial loss, signifying the importance of verifying smart contracts follow ERCs. Today's practices of such verification are to either manually audit each single contract or use expert-developed, limited-scope program-analysis tools, both of which are far from being effective in identifying ERC rule violations. This paper presents a tool named AuditGPT that leverages large language models (LLMs) to automatically and comprehensively verify ERC rules against smart contracts. To build AuditGPT, we first conduct an empirical study on 222 ERC rules specified in four popular ERCs to understand their content, their security impacts, their specification in natural language, and their implementation in Solidity. Guided by the study, we construct AuditGPT by separating the large, complex auditing process into small, manageable tasks and design prompts specialized for each ERC rule type to enhance LLMs' auditing performance. In the evaluation, AuditGPT successfully pinpoints 418 ERC rule violations and only reports 18 false positives, showcasing its effectiveness and accuracy. Moreover, AuditGPT beats an auditing service provided by security experts in effectiveness, accuracy, and cost, demonstrating its advancement over state-of-the-art smart-contract auditing practices.

Title: Faraday: Synthetic Smart Meter Generator for the smart grid

• Authors: Sheng Chai, Gus Chadney
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04314
• Pdf URL: https://arxiv.org/pdf/2404.04314
• Copy Paste: [[2404.04314]] Faraday: Synthetic Smart Meter Generator for the smart grid(https://arxiv.org/abs/2404.04314)
• Keywords: privacy, protect
• Abstract: Access to smart meter data is essential to rapid and successful transitions to electrified grids, underpinned by flexibility delivered by low carbon technologies, such as electric vehicles (EV) and heat pumps, and powered by renewable energy. Yet little of this data is available for research and modelling purposes due consumer privacy protections. Whilst many are calling for raw datasets to be unlocked through regulatory changes, we believe this approach will take too long. Synthetic data addresses these challenges directly by overcoming privacy issues. In this paper, we present Faraday, a Variational Auto-encoder (VAE)-based model trained over 300 million smart meter data readings from an energy supplier in the UK, with information such as property type and low carbon technologies (LCTs) ownership. The model produces household-level synthetic load profiles conditioned on these labels, and we compare its outputs against actual substation readings to show how the model can be used for real-world applications by grid modellers interested in modelling energy grids of the future.

Title: Robust Depth Enhancement via Polarization Prompt Fusion Tuning

• Authors: Kei Ikemura, Yiming Huang, Felix Heide, Zhaoxiang Zhang, Qifeng Chen, Chenyang Lei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04318
• Pdf URL: https://arxiv.org/pdf/2404.04318
• Copy Paste: [[2404.04318]] Robust Depth Enhancement via Polarization Prompt Fusion Tuning(https://arxiv.org/abs/2404.04318)
• Keywords: robust
• Abstract: Existing depth sensors are imperfect and may provide inaccurate depth values in challenging scenarios, such as in the presence of transparent or reflective objects. In this work, we present a general framework that leverages polarization imaging to improve inaccurate depth measurements from various depth sensors. Previous polarization-based depth enhancement methods focus on utilizing pure physics-based formulas for a single sensor. In contrast, our method first adopts a learning-based strategy where a neural network is trained to estimate a dense and complete depth map from polarization data and a sensor depth map from different sensors. To further improve the performance, we propose a Polarization Prompt Fusion Tuning (PPFT) strategy to effectively utilize RGB-based models pre-trained on large-scale datasets, as the size of the polarization dataset is limited to train a strong model from scratch. We conducted extensive experiments on a public dataset, and the results demonstrate that the proposed method performs favorably compared to existing depth enhancement baselines. Code and demos are available at https://lastbasket.github.io/PPFT/.

Title: SpatialTracker: Tracking Any 2D Pixels in 3D Space

• Authors: Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, Xiaowei Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04319
• Pdf URL: https://arxiv.org/pdf/2404.04319
• Copy Paste: [[2404.04319]] SpatialTracker: Tracking Any 2D Pixels in 3D Space(https://arxiv.org/abs/2404.04319)
• Keywords: transformer
• Abstract: Recovering dense and long-range pixel motion in videos is a challenging problem. Part of the difficulty arises from the 3D-to-2D projection process, leading to occlusions and discontinuities in the 2D motion domain. While 2D motion can be intricate, we posit that the underlying 3D motion can often be simple and low-dimensional. In this work, we propose to estimate point trajectories in 3D space to mitigate the issues caused by image projection. Our method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth estimators, represents the 3D content of each frame efficiently using a triplane representation, and performs iterative updates using a transformer to estimate 3D trajectories. Tracking in 3D allows us to leverage as-rigid-as-possible (ARAP) constraints while simultaneously learning a rigidity embedding that clusters pixels into different rigid parts. Extensive evaluation shows that our approach achieves state-of-the-art tracking performance both qualitatively and quantitatively, particularly in challenging scenarios such as out-of-plane rotation.

Title: Scope Ambiguities in Large Language Models

• Authors: Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala, Siva Reddy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04332
• Pdf URL: https://arxiv.org/pdf/2404.04332
• Copy Paste: [[2404.04332]] Scope Ambiguities in Large Language Models(https://arxiv.org/abs/2404.04332)
• Keywords: large language model
• Abstract: Sentences containing multiple semantic operators with overlapping scope often create ambiguities in interpretation, known as scope ambiguities. These ambiguities offer rich insights into the interaction between semantic structure and world knowledge in language processing. Despite this, there has been little research into how modern large language models treat them. In this paper, we investigate how different versions of certain autoregressive language models -- GPT-2, GPT-3/3.5, Llama 2 and GPT-4 -- treat scope ambiguous sentences, and compare this with human judgments. We introduce novel datasets that contain a joint total of almost 1,000 unique scope-ambiguous sentences, containing interactions between a range of semantic operators, and annotated for human judgments. Using these datasets, we find evidence that several models (i) are sensitive to the meaning ambiguity in these sentences, in a way that patterns well with human judgments, and (ii) can successfully identify human-preferred readings at a high level of accuracy (over 90% in some cases).

Title: Koala: Key frame-conditioned long video-LLM

• Authors: Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan A. Plummer, Bryan Russell, Kate Saenko
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04346
• Pdf URL: https://arxiv.org/pdf/2404.04346
• Copy Paste: [[2404.04346]] Koala: Key frame-conditioned long video-LLM(https://arxiv.org/abs/2404.04346)
• Keywords: large language model
• Abstract: Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships. State-of-the-art video Large Language Models (vLLMs) hold promise as a viable solution due to their demonstrated emergent capabilities on new tasks. However, despite being trained on millions of short seconds-long videos, vLLMs are unable to understand minutes-long videos and accurately answer questions about them. To address this limitation, we propose a lightweight and self-supervised approach, Key frame-conditioned long video-LLM (Koala), that introduces learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to longer videos. Our approach introduces two new tokenizers that condition on visual tokens computed from sparse video key frames for understanding short and long video moments. We train our proposed approach on HowTo100M and demonstrate its effectiveness on zero-shot long video understanding benchmarks, where it outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across all tasks. Surprisingly, we also empirically show that our approach not only helps a pretrained vLLM to understand long videos but also improves its accuracy on short-term action recognition.

Title: Assisting humans in complex comparisons: automated information comparison at scale

• Authors: Truman Yuen, Graham A. Watt, Yuri Lawryshyn
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04351
• Pdf URL: https://arxiv.org/pdf/2404.04351
• Copy Paste: [[2404.04351]] Assisting humans in complex comparisons: automated information comparison at scale(https://arxiv.org/abs/2404.04351)
• Keywords: generative, large language model
• Abstract: Generative Large Language Models enable efficient analytics across knowledge domains, rivalling human experts in information comparisons. However, the applications of LLMs for information comparisons face scalability challenges due to the difficulties in maintaining information across large contexts and overcoming model token limitations. To address these challenges, we developed the novel Abstractive Summarization \& Criteria-driven Comparison Endpoint (ASC$^2$End) system to automate information comparison at scale. Our system employs Semantic Text Similarity comparisons for generating evidence-supported analyses. We utilize proven data-handling strategies such as abstractive summarization and retrieval augmented generation to overcome token limitations and retain relevant information during model inference. Prompts were designed using zero-shot strategies to contextualize information for improved model reasoning. We evaluated abstractive summarization using ROUGE scoring and assessed the generated comparison quality using survey responses. Models evaluated on the ASC$^2$End system show desirable results providing insights on the expected performance of the system. ASC$^2$End is a novel system and tool that enables accurate, automated information comparison at scale across knowledge domains, overcoming limitations in context length and retrieval.

Title: Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback

• Authors: Mo Kordzanganeh, Danial Keshvary, Nariman Arian
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04356
• Pdf URL: https://arxiv.org/pdf/2404.04356
• Copy Paste: [[2404.04356]] Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback(https://arxiv.org/abs/2404.04356)
• Keywords: diffusion
• Abstract: Latent diffusion models are the state-of-the-art for synthetic image generation. To align these models with human preferences, training the models using reinforcement learning on human feedback is crucial. Black et. al 2024 introduced denoising diffusion policy optimisation (DDPO), which accounts for the iterative denoising nature of the generation by modelling it as a Markov chain with a final reward. As the reward is a single value that determines the model's performance on the entire image, the model has to navigate a very sparse reward landscape and so requires a large sample count. In this work, we extend the DDPO by presenting the Pixel-wise Policy Optimisation (PXPO) algorithm, which can take feedback for each pixel, providing a more nuanced reward to the model.

Title: Prompt Public Large Language Models to Synthesize Data for Private On-device Applications

• Authors: Shanshan Wu, Zheng Xu, Yanxiang Zhang, Yuanbo Zhang, Daniel Ramage
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04360
• Pdf URL: https://arxiv.org/pdf/2404.04360
• Copy Paste: [[2404.04360]] Prompt Public Large Language Models to Synthesize Data for Private On-device Applications(https://arxiv.org/abs/2404.04360)
• Keywords: privacy, federate, large language model
• Abstract: Pre-training on public data is an effective method to improve the performance for federated learning (FL) with differential privacy (DP). This paper investigates how large language models (LLMs) trained on public data can improve the quality of pre-training data for the on-device language models trained with DP and FL. We carefully design LLM prompts to filter and transform existing public data, and generate new data to resemble the real user data distribution. The model pre-trained on our synthetic dataset achieves relative improvement of 19.0% and 22.8% in next word prediction accuracy compared to the baseline model pre-trained on a standard public dataset, when evaluated over the real user data in Gboard (Google Keyboard, a production mobile keyboard application). Furthermore, our method achieves evaluation accuracy better than or comparable to the baseline during the DP FL fine-tuning over millions of mobile devices, and our final model outperforms the baseline in production A/B testing. Our experiments demonstrate the strengths of LLMs in synthesizing data close to the private distribution even without accessing the private data, and also suggest future research directions to further reduce the distribution gap.

Title: Deciphering Political Entity Sentiment in News with Large Language Models: Zero-Shot and Few-Shot Strategies

• Authors: Alapan Kuila, Sudeshna Sarkar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04361
• Pdf URL: https://arxiv.org/pdf/2404.04361
• Copy Paste: [[2404.04361]] Deciphering Political Entity Sentiment in News with Large Language Models: Zero-Shot and Few-Shot Strategies(https://arxiv.org/abs/2404.04361)
• Keywords: large language model
• Abstract: Sentiment analysis plays a pivotal role in understanding public opinion, particularly in the political domain where the portrayal of entities in news articles influences public perception. In this paper, we investigate the effectiveness of Large Language Models (LLMs) in predicting entity-specific sentiment from political news articles. Leveraging zero-shot and few-shot strategies, we explore the capability of LLMs to discern sentiment towards political entities in news content. Employing a chain-of-thought (COT) approach augmented with rationale in few-shot in-context learning, we assess whether this method enhances sentiment prediction accuracy. Our evaluation on sentiment-labeled datasets demonstrates that LLMs, outperform fine-tuned BERT models in capturing entity-specific sentiment. We find that learning in-context significantly improves model performance, while the self-consistency mechanism enhances consistency in sentiment prediction. Despite the promising results, we observe inconsistencies in the effectiveness of the COT prompting method. Overall, our findings underscore the potential of LLMs in entity-centric sentiment analysis within the political news domain and highlight the importance of suitable prompting strategies and model architectures.

Title: Compositional Estimation of Lipschitz Constants for Deep Neural Networks

• Authors: Yuezhu Xu, S. Sivaranjani
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2404.04375
• Pdf URL: https://arxiv.org/pdf/2404.04375
• Copy Paste: [[2404.04375]] Compositional Estimation of Lipschitz Constants for Deep Neural Networks(https://arxiv.org/abs/2404.04375)
• Keywords: attack, robust
• Abstract: The Lipschitz constant plays a crucial role in certifying the robustness of neural networks to input perturbations and adversarial attacks, as well as the stability and safety of systems with neural network controllers. Therefore, estimation of tight bounds on the Lipschitz constant of neural networks is a well-studied topic. However, typical approaches involve solving a large matrix verification problem, the computational cost of which grows significantly for deeper networks. In this letter, we provide a compositional approach to estimate Lipschitz constants for deep feedforward neural networks by obtaining an exact decomposition of the large matrix verification problem into smaller sub-problems. We further obtain a closed-form solution that applies to most common neural network activation functions, which will enable rapid robustness and stability certificates for neural networks deployed in online control settings. Finally, we demonstrate through numerical experiments that our approach provides a steep reduction in computation time while yielding Lipschitz bounds that are very close to those achieved by state-of-the-art approaches.

Title: ClickDiffusion: Harnessing LLMs for Interactive Precise Image Editing

• Authors: Alec Helbling, Seongmin Lee, Polo Chau
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04376
• Pdf URL: https://arxiv.org/pdf/2404.04376
• Copy Paste: [[2404.04376]] ClickDiffusion: Harnessing LLMs for Interactive Precise Image Editing(https://arxiv.org/abs/2404.04376)
• Keywords: diffusion
• Abstract: Recently, researchers have proposed powerful systems for generating and manipulating images using natural language instructions. However, it is difficult to precisely specify many common classes of image transformations with text alone. For example, a user may wish to change the location and breed of a particular dog in an image with several similar dogs. This task is quite difficult with natural language alone, and would require a user to write a laboriously complex prompt that both disambiguates the target dog and describes the destination. We propose ClickDiffusion, a system for precise image manipulation and generation that combines natural language instructions with visual feedback provided by the user through a direct manipulation interface. We demonstrate that by serializing both an image and a multi-modal instruction into a textual representation it is possible to leverage LLMs to perform precise transformations of the layout and appearance of an image. Code available at https://github.com/poloclub/ClickDiffusion.

Title: Reconfigurable and Scalable Honeynet for Cyber-Physical Systems

• Authors: Luís Sousa, José Cecílio, Pedro Ferreira, Alan Oliveira
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04385
• Pdf URL: https://arxiv.org/pdf/2404.04385
• Copy Paste: [[2404.04385]] Reconfigurable and Scalable Honeynet for Cyber-Physical Systems(https://arxiv.org/abs/2404.04385)
• Keywords: security, attack
• Abstract: Industrial Control Systems (ICS) constitute the backbone of contemporary industrial operations, ranging from modest heating, ventilation, and air conditioning systems to expansive national power grids. Given their pivotal role in critical infrastructure, there has been a concerted effort to enhance security measures and deepen our comprehension of potential cyber threats within this domain. To address these challenges, numerous implementations of Honeypots and Honeynets intended to detect and understand attacks have been employed for ICS. This approach diverges from conventional methods by focusing on making a scalable and reconfigurable honeynet for cyber-physical systems. It will also automatically generate attacks on the honeynet to test and validate it. With the development of a scalable and reconfigurable Honeynet and automatic attack generation tools, it is also expected that the system will serve as a basis for producing datasets for training algorithms for detecting and classifying attacks in cyber-physical honeynets.

Title: Increased LLM Vulnerabilities from Fine-tuning and Quantization

• Authors: Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, Prashanth Harshangi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04392
• Pdf URL: https://arxiv.org/pdf/2404.04392
• Copy Paste: [[2404.04392]] Increased LLM Vulnerabilities from Fine-tuning and Quantization(https://arxiv.org/abs/2404.04392)
• Keywords: privacy, attack, large language model
• Abstract: Large Language Models (LLMs) have become very popular and have found use cases in many domains, such as chatbots, auto-task completion agents, and much more. However, LLMs are vulnerable to different types of attacks, such as jailbreaking, prompt injection attacks, and privacy leakage attacks. Foundational LLMs undergo adversarial and alignment training to learn not to generate malicious and toxic content. For specialized use cases, these foundational LLMs are subjected to fine-tuning or quantization for better performance and efficiency. We examine the impact of downstream tasks such as fine-tuning and quantization on LLM vulnerability. We test foundation models like Mistral, Llama, MosaicML, and their fine-tuned versions. Our research shows that fine-tuning and quantization reduces jailbreak resistance significantly, leading to increased LLM vulnerabilities. Finally, we demonstrate the utility of external guardrails in reducing LLM vulnerabilities.

Title: PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos

• Authors: Yufei Zhang, Jeffrey O. Kephart, Zijun Cui, Qiang Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04430
• Pdf URL: https://arxiv.org/pdf/2404.04430
• Copy Paste: [[2404.04430]] PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos(https://arxiv.org/abs/2404.04430)
• Keywords: transformer
• Abstract: While current methods have shown promising progress on estimating 3D human motion from monocular videos, their motion estimates are often physically unrealistic because they mainly consider kinematics. In this paper, we introduce Physics-aware Pretrained Transformer (PhysPT), which improves kinematics-based motion estimates and infers motion forces. PhysPT exploits a Transformer encoder-decoder backbone to effectively learn human dynamics in a self-supervised manner. Moreover, it incorporates physics principles governing human motion. Specifically, we build a physics-based body representation and contact force model. We leverage them to impose novel physics-inspired training losses (i.e., force loss, contact loss, and Euler-Lagrange loss), enabling PhysPT to capture physical properties of the human body and the forces it experiences. Experiments demonstrate that, once trained, PhysPT can be directly applied to kinematics-based estimates to significantly enhance their physical plausibility and generate favourable motion forces. Furthermore, we show that these physically meaningful quantities translate into improved accuracy of an important downstream task: human action recognition.

Title: Robust Few-Shot Ensemble Learning with Focal Diversity-Based Pruning

• Authors: Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, Ka-Ho Chow, Margaret L. Loper, Ling Liu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04434
• Pdf URL: https://arxiv.org/pdf/2404.04434
• Copy Paste: [[2404.04434]] Robust Few-Shot Ensemble Learning with Focal Diversity-Based Pruning(https://arxiv.org/abs/2404.04434)
• Keywords: robust
• Abstract: This paper presents FusionShot, a focal diversity optimized few-shot ensemble learning approach for boosting the robustness and generalization performance of pre-trained few-shot models. The paper makes three original contributions. First, we explore the unique characteristics of few-shot learning to ensemble multiple few-shot (FS) models by creating three alternative fusion channels. Second, we introduce the concept of focal error diversity to learn the most efficient ensemble teaming strategy, rather than assuming that an ensemble of a larger number of base models will outperform those sub-ensembles of smaller size. We develop a focal-diversity ensemble pruning method to effectively prune out the candidate ensembles with low ensemble error diversity and recommend top-$K$ FS ensembles with the highest focal error diversity. Finally, we capture the complex non-linear patterns of ensemble few-shot predictions by designing the learn-to-combine algorithm, which can learn the diverse weight assignments for robust ensemble fusion over different member models. Extensive experiments on representative few-shot benchmarks show that the top-K ensembles recommended by FusionShot can outperform the representative SOTA few-shot models on novel tasks (different distributions and unknown at training), and can prevail over existing few-shot learners in both cross-domain settings and adversarial settings. For reproducibility purposes, FusionShot trained models, results, and code are made available at https://github.com/sftekin/fusionshot

Title: Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation

• Authors: Fahmida Alam, Md Asiful Islam, Robert Vacareanu, Mihai Surdeanu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.04445
• Pdf URL: https://arxiv.org/pdf/2404.04445
• Copy Paste: [[2404.04445]] Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation(https://arxiv.org/abs/2404.04445)
• Keywords: extraction
• Abstract: We introduce a meta dataset for few-shot relation extraction, which includes two datasets derived from existing supervised relation extraction datasets NYT29 (Takanobu et al., 2019; Nayak and Ng, 2020) and WIKIDATA (Sorokin and Gurevych, 2017) as well as a few-shot form of the TACRED dataset (Sabo et al., 2021). Importantly, all these few-shot datasets were generated under realistic assumptions such as: the test relations are different from any relations a model might have seen before, limited training data, and a preponderance of candidate relation mentions that do not correspond to any of the relations of interest. Using this large resource, we conduct a comprehensive evaluation of six recent few-shot relation extraction methods, and observe that no method comes out as a clear winner. Further, the overall performance on this task is low, indicating substantial need for future research. We release all versions of the data, i.e., both supervised and few-shot, for future research.

Title: Vision Transformers in Domain Adaptation and Generalization: A Study of Robustness

• Authors: Shadi Alijani, Jamil Fayyad, Homayoun Najjaran
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04452
• Pdf URL: https://arxiv.org/pdf/2404.04452
• Copy Paste: [[2404.04452]] Vision Transformers in Domain Adaptation and Generalization: A Study of Robustness(https://arxiv.org/abs/2404.04452)
• Keywords: robust, transformer
• Abstract: Deep learning models are often evaluated in scenarios where the data distribution is different from those used in the training and validation phases. The discrepancy presents a challenge for accurately predicting the performance of models once deployed on the target distribution. Domain adaptation and generalization are widely recognized as effective strategies for addressing such shifts, thereby ensuring reliable performance. The recent promising results in applying vision transformers in computer vision tasks, coupled with advancements in self-attention mechanisms, have demonstrated their significant potential for robustness and generalization in handling distribution shifts. Motivated by the increased interest from the research community, our paper investigates the deployment of vision transformers in domain adaptation and domain generalization scenarios. For domain adaptation methods, we categorize research into feature-level, instance-level, model-level adaptations, and hybrid approaches, along with other categorizations with respect to diverse strategies for enhancing domain adaptation. Similarly, for domain generalization, we categorize research into multi-domain learning, meta-learning, regularization techniques, and data augmentation strategies. We further classify diverse strategies in research, underscoring the various approaches researchers have taken to address distribution shifts by integrating vision transformers. The inclusion of comprehensive tables summarizing these categories is a distinct feature of our work, offering valuable insights for researchers. These findings highlight the versatility of vision transformers in managing distribution shifts, crucial for real-world applications, especially in critical safety and decision-making scenarios.

Title: JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups

• Authors: Simindokht Jahangard, Zhixi Cai, Shiki Wen, Hamid Rezatofighi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04458
• Pdf URL: https://arxiv.org/pdf/2404.04458
• Copy Paste: [[2404.04458]] JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups(https://arxiv.org/abs/2404.04458)
• Keywords: large language model
• Abstract: Understanding human social behaviour is crucial in computer vision and robotics. Micro-level observations like individual actions fall short, necessitating a comprehensive approach that considers individual behaviour, intra-group dynamics, and social group levels for a thorough understanding. To address dataset limitations, this paper introduces JRDB-Social, an extension of JRDB. Designed to fill gaps in human understanding across diverse indoor and outdoor social contexts, JRDB-Social provides annotations at three levels: individual attributes, intra-group interactions, and social group context. This dataset aims to enhance our grasp of human social dynamics for robotic applications. Utilizing the recent cutting-edge multi-modal large language models, we evaluated our benchmark to explore their capacity to decipher social human behaviour.

Title: Automated Polyp Segmentation in Colonoscopy Images

• Authors: Swagat Ranjit, Jian Zhang, Bijaya B. Karki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04461
• Pdf URL: https://arxiv.org/pdf/2404.04461
• Copy Paste: [[2404.04461]] Automated Polyp Segmentation in Colonoscopy Images(https://arxiv.org/abs/2404.04461)
• Keywords: segmentation
• Abstract: It is important to find the polyps in a human system that helps to prevent cancer during medical diagnosis. This research discusses using a dilated convolution module along with a criss cross attention-based network to segment polyps from the endoscopic images of the colon. To gather the context information of all pixels in an image more efficiently, criss-cross attention module has played a vital role. In order to extract maximum information from dataset, data augmentation techniques are employed in the dataset. Rotations, flips, scaling, and contrast along with varying learning rates were implemented to make a better model. Global average pooling was applied over ResNet50 that helped to store the important details of encoder. In our experiment, the proposed architecture's performance was compared with existing models like U-Net, DeepLabV3, PraNet. This architecture outperformed other models on the subset of dataset which has irregular polyp shapes. The combination of dilated convolution module, RCCA, and global average pooling was found to be effective for irregular shapes. Our architecture demonstrates an enhancement, with an average improvement of 3.75% across all metrics when compared to existing models.

Title: Aligning Diffusion Models by Optimizing Human Utility

• Authors: Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, Kazuki Kozuka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04465
• Pdf URL: https://arxiv.org/pdf/2404.04465
• Copy Paste: [[2404.04465]] Aligning Diffusion Models by Optimizing Human Utility(https://arxiv.org/abs/2404.04465)
• Keywords: diffusion
• Abstract: We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.

Title: Cybersecurity for Modern Smart Grid against Emerging Threats

• Authors: Daisuke Mashima, Yao Chen, Muhammad M. Roomi, Subhash Lakshminarayana, Deming Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04466
• Pdf URL: https://arxiv.org/pdf/2404.04466
• Copy Paste: [[2404.04466]] Cybersecurity for Modern Smart Grid against Emerging Threats(https://arxiv.org/abs/2404.04466)
• Keywords: security
• Abstract: Smart Grid is a power grid system that uses digital communication technologies. By deploying intelligent devices throughout the power grid infrastructure,from power generation to consumption, and enabling communication among them, it revolutionizes the modern power grid industry with increased efficiency, reliability, and availability. However, reliance on information and communication technologies has also made the smart grids exposed to new vulnerabilities and complications that may negatively impact the availability and stability of electricity services, which are vital for people's daily lives. The purpose of this monograph is to provide an up-to-date and comprehensive survey and tutorial on the cybersecurity aspect of smart grids. The book focuses on the sources of the cybersecurity issues, the taxonomy of threats, and the survey of various approaches to overcome or mitigate such threats. It covers the state-of-the-art research results in recent years, along with remaining open challenges. We hope that this monograph can be used both as learning materials for beginners who are embarking on research in this area and as a useful reference for established researchers in this field.

Title: Mixed-Query Transformer: A Unified Image Segmentation Architecture

• Authors: Pei Wang, Zhaowei Cai, Hao Yang, Ashwin Swaminathan, R. Manmatha, Stefano Soatto
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04469
• Pdf URL: https://arxiv.org/pdf/2404.04469
• Copy Paste: [[2404.04469]] Mixed-Query Transformer: A Unified Image Segmentation Architecture(https://arxiv.org/abs/2404.04469)
• Keywords: transformer, segmentation
• Abstract: Existing unified image segmentation models either employ a unified architecture across multiple tasks but use separate weights tailored to each dataset, or apply a single set of weights to multiple datasets but are limited to a single task. In this paper, we introduce the Mixed-Query Transformer (MQ-Former), a unified architecture for multi-task and multi-dataset image segmentation using a single set of weights. To enable this, we propose a mixed query strategy, which can effectively and dynamically accommodate different types of objects without heuristic designs. In addition, the unified architecture allows us to use data augmentation with synthetic masks and captions to further improve model generalization. Experiments demonstrate that MQ-Former can not only effectively handle multiple segmentation datasets and tasks compared to specialized state-of-the-art models with competitive performance, but also generalize better to open-set segmentation tasks, evidenced by over 7 points higher performance than the prior art on the open-vocabulary SeginW benchmark.

Title: Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

• Authors: Yann Dubois, Balázs Galambosi, Percy Liang, Tatsunori B. Hashimoto
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.04475
• Pdf URL: https://arxiv.org/pdf/2404.04475
• Copy Paste: [[2404.04475]] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators(https://arxiv.org/abs/2404.04475)
• Keywords: robust
• Abstract: LLM-based auto-annotators have become a key component of the LLM development process due to their cost-effectiveness and scalability compared to human-based evaluation. However, these auto-annotators can introduce complex biases that are hard to remove. Even simple, known confounders such as preference for longer outputs remain in existing automated evaluation metrics. We propose a simple regression analysis approach for controlling biases in auto-evaluations. As a real case study, we focus on reducing the length bias of AlpacaEval, a fast and affordable benchmark for chat LLMs that uses LLMs to estimate response quality. Despite being highly correlated with human preferences, AlpacaEval is known to favor models that generate longer outputs. We introduce a length-controlled AlpacaEval that aims to answer the counterfactual question: "What would the preference be if the model's and baseline's output had the same length?". To achieve this, we first fit a generalized linear model to predict the biased output of interest (auto-annotator preferences) based on the mediators we want to control for (length difference) and other relevant features. We then obtain length-controlled preferences by predicting preferences while conditioning the GLM with a zero difference in lengths. Length-controlling not only improves the robustness of the metric to manipulations in model verbosity, we also find that it increases the Spearman correlation with LMSYS' Chatbot Arena from 0.94 to 0.98. We release the code and leaderboard at https://tatsu-lab.github.io/alpaca_eval/ .

Title: Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models

• Authors: Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, Junshi Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04478
• Pdf URL: https://arxiv.org/pdf/2404.04478
• Copy Paste: [[2404.04478]] Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models(https://arxiv.org/abs/2404.04478)
• Keywords: diffusion, transformer
• Abstract: Transformers have catalyzed advancements in computer vision and natural language processing (NLP) fields. However, substantial computational complexity poses limitations for their application in long-context tasks, such as high-resolution image generation. This paper introduces a series of architectures adapted from the RWKV model used in the NLP, with requisite modifications tailored for diffusion model applied to image generation tasks, referred to as Diffusion-RWKV. Similar to the diffusion with Transformers, our model is designed to efficiently handle patchnified inputs in a sequence with extra conditions, while also scaling up effectively, accommodating both large-scale parameters and extensive datasets. Its distinctive advantage manifests in its reduced spatial aggregation complexity, rendering it exceptionally adept at processing high-resolution images, thereby eliminating the necessity for windowing or group cached operations. Experimental results on both condition and unconditional image generation tasks demonstrate that Diffison-RWKV achieves performance on par with or surpasses existing CNN or Transformer-based diffusion models in FID and IS metrics while significantly reducing total computation FLOP usage.

Title: Hyperparameter Optimization for SecureBoost via Constrained Multi-Objective Federated Learning

• Authors: Yan Kang, Ziyao Ren, Lixin Fan, Linghua Yang, Yongxin Tong, Qiang Yang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04490
• Pdf URL: https://arxiv.org/pdf/2404.04490
• Copy Paste: [[2404.04490]] Hyperparameter Optimization for SecureBoost via Constrained Multi-Objective Federated Learning(https://arxiv.org/abs/2404.04490)
• Keywords: secure, privacy, protect, attack, federate
• Abstract: SecureBoost is a tree-boosting algorithm that leverages homomorphic encryption (HE) to protect data privacy in vertical federated learning. SecureBoost and its variants have been widely adopted in fields such as finance and healthcare. However, the hyperparameters of SecureBoost are typically configured heuristically for optimizing model performance (i.e., utility) solely, assuming that privacy is secured. Our study found that SecureBoost and some of its variants are still vulnerable to label leakage. This vulnerability may lead the current heuristic hyperparameter configuration of SecureBoost to a suboptimal trade-off between utility, privacy, and efficiency, which are pivotal elements toward a trustworthy federated learning system. To address this issue, we propose the Constrained Multi-Objective SecureBoost (CMOSB) algorithm, which aims to approximate Pareto optimal solutions that each solution is a set of hyperparameters achieving an optimal trade-off between utility loss, training cost, and privacy leakage. We design measurements of the three objectives, including a novel label inference attack named instance clustering attack (ICA) to measure the privacy leakage of SecureBoost. Additionally, we provide two countermeasures against ICA. The experimental results demonstrate that the CMOSB yields superior hyperparameters over those optimized by grid search and Bayesian optimization regarding the trade-off between utility loss, training cost, and privacy leakage.

Title: IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe Biomedical Natural Language Inference for Clinical Trials

• Authors: Shreyasi Mandal, Ashutosh Modi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04510
• Pdf URL: https://arxiv.org/pdf/2404.04510
• Copy Paste: [[2404.04510]] IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe Biomedical Natural Language Inference for Clinical Trials(https://arxiv.org/abs/2404.04510)
• Keywords: robust, large language model
• Abstract: Large Language models (LLMs) have demonstrated state-of-the-art performance in various natural language processing (NLP) tasks across multiple domains, yet they are prone to shortcut learning and factual inconsistencies. This research investigates LLMs' robustness, consistency, and faithful reasoning when performing Natural Language Inference (NLI) on breast cancer Clinical Trial Reports (CTRs) in the context of SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. We examine the reasoning capabilities of LLMs and their adeptness at logical problem-solving. A comparative analysis is conducted on pre-trained language models (PLMs), GPT-3.5, and Gemini Pro under zero-shot settings using Retrieval-Augmented Generation (RAG) framework, integrating various reasoning chains. The evaluation yields an F1 score of 0.69, consistency of 0.71, and a faithfulness score of 0.90 on the test dataset.

Title: Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models

• Authors: Songtao Jiang, Yan Zhang, Chenyi Zhou, Yeying Jin, Yang Feng, Jian Wu, Zuozhu Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04514
• Pdf URL: https://arxiv.org/pdf/2404.04514
• Copy Paste: [[2404.04514]] Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models(https://arxiv.org/abs/2404.04514)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA), particularly in object-oriented perception tasks which demand fine-grained understanding of object identities, locations or attributes, as indicated by empirical findings. This is mainly due to their limited capability to effectively integrate complex visual cues with textual information and potential object hallucinations. In this paper, we present a novel approach, Joint Visual and Text Prompting (VTPrompt), that employs fine-grained visual information to enhance the capability of MLLMs in VQA, especially for object-oriented perception. VTPrompt merges visual and text prompts to extract key concepts from textual questions and employs a detection model to highlight relevant objects as visual prompts in images. The processed images alongside text prompts are subsequently fed into MLLMs to produce more accurate answers. Our experiments with GPT-4V and Gemini Pro, on three benchmarks, i.e., MME , MMB and POPE, demonstrate significant improvements. Particularly, our method led to a score improvement of up to 183.5 for GPT-4V on MME and enhanced MMB performance by 8.17\% for GPT-4V and 15.69\% for Gemini Pro.

Title: Latent-based Diffusion Model for Long-tailed Recognition

• Authors: Pengxiao Han, Changkun Ye, Jieming Zhou, Jing Zhang, Jie Hong, Xuesong Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04517
• Pdf URL: https://arxiv.org/pdf/2404.04517
• Copy Paste: [[2404.04517]] Latent-based Diffusion Model for Long-tailed Recognition(https://arxiv.org/abs/2404.04517)
• Keywords: diffusion
• Abstract: Long-tailed imbalance distribution is a common issue in practical computer vision applications. Previous works proposed methods to address this problem, which can be categorized into several classes: re-sampling, re-weighting, transfer learning, and feature augmentation. In recent years, diffusion models have shown an impressive generation ability in many sub-problems of deep computer vision. However, its powerful generation has not been explored in long-tailed problems. We propose a new approach, the Latent-based Diffusion Model for Long-tailed Recognition (LDMLR), as a feature augmentation method to tackle the issue. First, we encode the imbalanced dataset into features using the baseline model. Then, we train a Denoising Diffusion Implicit Model (DDIM) using these encoded features to generate pseudo-features. Finally, we train the classifier using the encoded and pseudo-features from the previous two steps. The model's accuracy shows an improvement on the CIFAR-LT and ImageNet-LT datasets by using the proposed method.

Title: MedIAnomaly: A comparative study of anomaly detection in medical images

• Authors: Yu Cai, Weiwen Zhang, Hao Chen, Kwang-Ting Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04518
• Pdf URL: https://arxiv.org/pdf/2404.04518
• Copy Paste: [[2404.04518]] MedIAnomaly: A comparative study of anomaly detection in medical images(https://arxiv.org/abs/2404.04518)
• Keywords: fair, segmentation
• Abstract: Anomaly detection (AD) aims at detecting abnormal samples that deviate from the expected normal patterns. Generally, it can be trained on merely normal data without the requirement for abnormal samples, and thereby plays an important role in the recognition of rare diseases and health screening in the medical domain. Despite numerous related studies, we observe a lack of a fair and comprehensive evaluation, which causes some ambiguous conclusions and hinders the development of this field. This paper focuses on building a benchmark with unified implementation and comparison to address this problem. In particular, seven medical datasets with five image modalities, including chest X-rays, brain MRIs, retinal fundus images, dermatoscopic images, and histopathology whole slide images are organized for extensive evaluation. Twenty-seven typical AD methods, including reconstruction and self-supervised learning-based methods, are involved in comparison of image-level anomaly classification and pixel-level anomaly segmentation. Furthermore, we for the first time formally explore the effect of key components in existing methods, clearly revealing unresolved challenges and potential future directions. The datasets and code are available at \url{https://github.com/caiyu6666/MedIAnomaly}.

Title: Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models

• Authors: Zhiyuan Peng, Xuyang Wu, Qifan Wang, Sravanthi Rajanala, Yi Fang
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04522
• Pdf URL: https://arxiv.org/pdf/2404.04522
• Copy Paste: [[2404.04522]] Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models(https://arxiv.org/abs/2404.04522)
• Keywords: large language model
• Abstract: Parameter Efficient Fine-Tuning (PEFT) methods have been extensively utilized in Large Language Models (LLMs) to improve the down-streaming tasks without the cost of fine-tuing the whole LLMs. Recent studies have shown how to effectively use PEFT for fine-tuning LLMs in ranking tasks with convincing performance; there are some limitations, including the learned prompt being fixed for different documents, overfitting to specific tasks, and low adaptation ability. In this paper, we introduce a query-dependent parameter efficient fine-tuning (Q-PEFT) approach for text reranking to leak the information of the true queries to LLMs and then make the generation of true queries from input documents much easier. Specifically, we utilize the query to extract the top-$k$ tokens from concatenated documents, serving as contextual clues. We further augment Q-PEFT by substituting the retrieval mechanism with a multi-head attention layer to achieve end-to-end training and cover all the tokens in the documents, guiding the LLMs to generate more document-specific synthetic queries, thereby further improving the reranking performance. Extensive experiments are conducted on four public datasets, demonstrating the effectiveness of our proposed approach.

Title: IITK at SemEval-2024 Task 10: Who is the speaker? Improving Emotion Recognition and Flip Reasoning in Conversations via Speaker Embeddings

• Authors: Shubham Patel, Divyaksh Shukla, Ashutosh Modi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04525
• Pdf URL: https://arxiv.org/pdf/2404.04525
• Copy Paste: [[2404.04525]] IITK at SemEval-2024 Task 10: Who is the speaker? Improving Emotion Recognition and Flip Reasoning in Conversations via Speaker Embeddings(https://arxiv.org/abs/2404.04525)
• Keywords: transformer
• Abstract: This paper presents our approach for the SemEval-2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversations. For the Emotion Recognition in Conversations (ERC) task, we utilize a masked-memory network along with speaker participation. We propose a transformer-based speaker-centric model for the Emotion Flip Reasoning (EFR) task. We also introduce Probable Trigger Zone, a region of the conversation that is more likely to contain the utterances causing the emotion to flip. For sub-task 3, the proposed approach achieves a 5.9 (F1 score) improvement over the task baseline. The ablation study results highlight the significance of various design choices in the proposed method.

Title: DATENeRF: Depth-Aware Text-based Editing of NeRFs

• Authors: Sara Rojas, Julien Philip, Kai Zhang, Sai Bi, Fujun Luan, Bernard Ghanem, Kalyan Sunkavall
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04526
• Pdf URL: https://arxiv.org/pdf/2404.04526
• Copy Paste: [[2404.04526]] DATENeRF: Depth-Aware Text-based Editing of NeRFs(https://arxiv.org/abs/2404.04526)
• Keywords: robust, diffusion
• Abstract: Recent advancements in diffusion models have shown remarkable proficiency in editing 2D images based on text prompts. However, extending these techniques to edit scenes in Neural Radiance Fields (NeRF) is complex, as editing individual 2D frames can result in inconsistencies across multiple views. Our crucial insight is that a NeRF scene's geometry can serve as a bridge to integrate these 2D edits. Utilizing this geometry, we employ a depth-conditioned ControlNet to enhance the coherence of each 2D image modification. Moreover, we introduce an inpainting approach that leverages the depth information of NeRF scenes to distribute 2D edits across different images, ensuring robustness against errors and resampling challenges. Our results reveal that this methodology achieves more consistent, lifelike, and detailed edits than existing leading methods for text-driven NeRF scene editing.

Title: VTR: An Optimized Vision Transformer for SAR ATR Acceleration on FPGA

• Authors: Sachini Wickramasinghe, Dhruv Parikh, Bingyi Zhang, Rajgopal Kannan, Viktor Prasanna, Carl Busart
• Subjects: cs.CV, cs.AI, cs.AR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.04527
• Pdf URL: https://arxiv.org/pdf/2404.04527
• Copy Paste: [[2404.04527]] VTR: An Optimized Vision Transformer for SAR ATR Acceleration on FPGA(https://arxiv.org/abs/2404.04527)
• Keywords: transformer
• Abstract: Synthetic Aperture Radar (SAR) Automatic Target Recognition (ATR) is a key technique used in military applications like remote-sensing image recognition. Vision Transformers (ViTs) are the current state-of-the-art in various computer vision applications, outperforming their CNN counterparts. However, using ViTs for SAR ATR applications is challenging due to (1) standard ViTs require extensive training data to generalize well due to their low locality; the standard SAR datasets, however, have a limited number of labeled training data which reduces the learning capability of ViTs; (2) ViTs have a high parameter count and are computation intensive which makes their deployment on resource-constrained SAR platforms difficult. In this work, we develop a lightweight ViT model that can be trained directly on small datasets without any pre-training by utilizing the Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA) modules. We directly train this model on SAR datasets which have limited training samples to evaluate its effectiveness for SAR ATR applications. We evaluate our proposed model, that we call VTR (ViT for SAR ATR), on three widely used SAR datasets: MSTAR, SynthWakeSAR, and GBSAR. Further, we propose a novel FPGA accelerator for VTR, in order to enable deployment for real-time SAR ATR applications.

Title: Frequency Decomposition-Driven Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation

• Authors: Xianping Ma, Xiaokang Zhang, Xingchen Ding, Man-On Pun, Siwei Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04531
• Pdf URL: https://arxiv.org/pdf/2404.04531
• Copy Paste: [[2404.04531]] Frequency Decomposition-Driven Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation(https://arxiv.org/abs/2404.04531)
• Keywords: transformer, generative, segmentation
• Abstract: Cross-domain semantic segmentation of remote sensing (RS) imagery based on unsupervised domain adaptation (UDA) techniques has significantly advanced deep-learning applications in the geosciences. Recently, with its ingenious and versatile architecture, the Transformer model has been successfully applied in RS-UDA tasks. However, existing UDA methods mainly focus on domain alignment in the high-level feature space. It is still challenging to retain cross-domain local spatial details and global contextual semantics simultaneously, which is crucial for the RS image semantic segmentation task. To address these problems, we propose novel high/low-frequency decomposition (HLFD) techniques to guide representation alignment in cross-domain semantic segmentation. Specifically, HLFD attempts to decompose the feature maps into high- and low-frequency components before performing the domain alignment in the corresponding subspaces. Secondly, to further facilitate the alignment of decomposed features, we propose a fully global-local generative adversarial network, namely GLGAN, to learn domain-invariant detailed and semantic features across domains by leveraging global-local transformer blocks (GLTBs). By integrating HLFD techniques and the GLGAN, a novel UDA framework called FD-GLGAN is developed to improve the cross-domain transferability and generalization capability of semantic segmentation models. Extensive experiments on two fine-resolution benchmark datasets, namely ISPRS Potsdam and ISPRS Vaihingen, highlight the effectiveness and superiority of the proposed approach as compared to the state-of-the-art UDA methods. The source code for this work will be accessible at https://github.com/sstary/SSRS.

Title: Impact of Fairness Regulations on Institutions' Policies and Population Qualifications

• Authors: Hamidreza Montaseri, Amin Gohari
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.04534
• Pdf URL: https://arxiv.org/pdf/2404.04534
• Copy Paste: [[2404.04534]] Impact of Fairness Regulations on Institutions' Policies and Population Qualifications(https://arxiv.org/abs/2404.04534)
• Keywords: fair
• Abstract: The proliferation of algorithmic systems has fueled discussions surrounding the regulation and control of their social impact. Herein, we consider a system whose primary objective is to maximize utility by selecting the most qualified individuals. To promote demographic parity in the selection algorithm, we consider penalizing discrimination across social groups. We examine conditions under which a discrimination penalty can effectively reduce disparity in the selection. Additionally, we explore the implications of such a penalty when individual qualifications may evolve over time in response to the imposed penalizing policy. We identify scenarios where the penalty could hinder the natural attainment of equity within the population. Moreover, we propose certain conditions that can counteract this undesirable outcome, thus ensuring fairness.

Title: BeyondScene: Higher-Resolution Human-Centric Scene Generation With Pretrained Diffusion

• Authors: Gwanghyun Kim, Hayeon Kim, Hoigi Seo, Dong Un Kang, Se Young Chun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04544
• Pdf URL: https://arxiv.org/pdf/2404.04544
• Copy Paste: [[2404.04544]] BeyondScene: Higher-Resolution Human-Centric Scene Generation With Pretrained Diffusion(https://arxiv.org/abs/2404.04544)
• Keywords: diffusion
• Abstract: Generating higher-resolution human-centric scenes with details and controls remains a challenge for existing text-to-image diffusion models. This challenge stems from limited training image size, text encoder capacity (limited tokens), and the inherent difficulty of generating complex scenes involving multiple humans. While current methods attempted to address training size limit only, they often yielded human-centric scenes with severe artifacts. We propose BeyondScene, a novel framework that overcomes prior limitations, generating exquisite higher-resolution (over 8K) human-centric scenes with exceptional text-image correspondence and naturalness using existing pretrained diffusion models. BeyondScene employs a staged and hierarchical approach to initially generate a detailed base image focusing on crucial elements in instance creation for multiple humans and detailed descriptions beyond token limit of diffusion model, and then to seamlessly convert the base image to a higher-resolution output, exceeding training image size and incorporating details aware of text and instances via our novel instance-aware hierarchical enlargement process that consists of our proposed high-frequency injected forward diffusion and adaptive joint diffusion. BeyondScene surpasses existing methods in terms of correspondence with detailed text descriptions and naturalness, paving the way for advanced applications in higher-resolution human-centric scene creation beyond the capacity of pretrained diffusion models without costly retraining. Project page: https://janeyeon.github.io/beyond-scene.

Title: A self-attention model for robust rigid slice-to-volume registration of functional MRI

• Authors: Samah Khawaled, Simon K. Warfield, Moti Freiman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04546
• Pdf URL: https://arxiv.org/pdf/2404.04546
• Copy Paste: [[2404.04546]] A self-attention model for robust rigid slice-to-volume registration of functional MRI(https://arxiv.org/abs/2404.04546)
• Keywords: robust
• Abstract: Functional Magnetic Resonance Imaging (fMRI) is vital in neuroscience, enabling investigations into brain disorders, treatment monitoring, and brain function mapping. However, head motion during fMRI scans, occurring between shots of slice acquisition, can result in distortion, biased analyses, and increased costs due to the need for scan repetitions. Therefore, retrospective slice-level motion correction through slice-to-volume registration (SVR) is crucial. Previous studies have utilized deep learning (DL) based models to address the SVR task; however, they overlooked the uncertainty stemming from the input stack of slices and did not assign weighting or scoring to each slice. In this work, we introduce an end-to-end SVR model for aligning 2D fMRI slices with a 3D reference volume, incorporating a self-attention mechanism to enhance robustness against input data variations and uncertainties. It utilizes independent slice and volume encoders and a self-attention module to assign pixel-wise scores for each slice. We conducted evaluation experiments on 200 images involving synthetic rigid motion generated from 27 subjects belonging to the test set, from the publicly available Healthy Brain Network (HBN) dataset. Our experimental results demonstrate that our model achieves competitive performance in terms of alignment accuracy compared to state-of-the-art deep learning-based methods (Euclidean distance of $0.93$ [mm] vs. $1.86$ [mm]). Furthermore, our approach exhibits significantly faster registration speed compared to conventional iterative methods ($0.096$ sec. vs. $1.17$ sec.). Our end-to-end SVR model facilitates real-time head motion tracking during fMRI acquisition, ensuring reliability and robustness against uncertainties in inputs. source code, which includes the training and evaluations, will be available soon.

Title: Learning Instance-Aware Correspondences for Robust Multi-Instance Point Cloud Registration in Cluttered Scenes

• Authors: Zhiyuan Yu, Zheng Qin, Lintao Zheng, Kai Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04557
• Pdf URL: https://arxiv.org/pdf/2404.04557
• Copy Paste: [[2404.04557]] Learning Instance-Aware Correspondences for Robust Multi-Instance Point Cloud Registration in Cluttered Scenes(https://arxiv.org/abs/2404.04557)
• Keywords: robust, extraction, transformer
• Abstract: Multi-instance point cloud registration estimates the poses of multiple instances of a model point cloud in a scene point cloud. Extracting accurate point correspondence is to the center of the problem. Existing approaches usually treat the scene point cloud as a whole, overlooking the separation of instances. Therefore, point features could be easily polluted by other points from the background or different instances, leading to inaccurate correspondences oblivious to separate instances, especially in cluttered scenes. In this work, we propose MIRETR, Multi-Instance REgistration TRansformer, a coarse-to-fine approach to the extraction of instance-aware correspondences. At the coarse level, it jointly learns instance-aware superpoint features and predicts per-instance masks. With instance masks, the influence from outside of the instance being concerned is minimized, such that highly reliable superpoint correspondences can be extracted. The superpoint correspondences are then extended to instance candidates at the fine level according to the instance masks. At last, an efficient candidate selection and refinement algorithm is devised to obtain the final registrations. Extensive experiments on three public benchmarks demonstrate the efficacy of our approach. In particular, MIRETR outperforms the state of the arts by 16.6 points on F1 score on the challenging ROBI benchmark. Code and models are available at https://github.com/zhiyuanYU134/MIRETR.

Title: Diffusion Time-step Curriculum for One Image to 3D Generation

• Authors: Xuanyu Yi, Zike Wu, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, Hanwang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04562
• Pdf URL: https://arxiv.org/pdf/2404.04562
• Copy Paste: [[2404.04562]] Diffusion Time-step Curriculum for One Image to 3D Generation(https://arxiv.org/abs/2404.04562)
• Keywords: diffusion
• Abstract: Score distillation sampling~(SDS) has been widely adopted to overcome the absence of unseen views in reconstructing 3D objects from a \textbf{single} image. It leverages pre-trained 2D diffusion models as teacher to guide the reconstruction of student 3D models. Despite their remarkable success, SDS-based methods often encounter geometric artifacts and texture saturation. We find out the crux is the overlooked indiscriminate treatment of diffusion time-steps during optimization: it unreasonably treats the student-teacher knowledge distillation to be equal at all time-steps and thus entangles coarse-grained and fine-grained modeling. Therefore, we propose the Diffusion Time-step Curriculum one-image-to-3D pipeline (DTC123), which involves both the teacher and student models collaborating with the time-step curriculum in a coarse-to-fine manner. Extensive experiments on NeRF4, RealFusion15, GSO and Level50 benchmark demonstrate that DTC123 can produce multi-view consistent, high-quality, and diverse 3D assets. Codes and more generation demos will be released in https://github.com/yxymessi/DTC123.

Title: Optimization of Lightweight Malware Detection Models For AIoT Devices

• Authors: Felicia Lo, Shin-Ming Cheng, Rafael Kaliski
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04567
• Pdf URL: https://arxiv.org/pdf/2404.04567
• Copy Paste: [[2404.04567]] Optimization of Lightweight Malware Detection Models For AIoT Devices(https://arxiv.org/abs/2404.04567)
• Keywords: robust
• Abstract: Malware intrusion is problematic for Internet of Things (IoT) and Artificial Intelligence of Things (AIoT) devices as they often reside in an ecosystem of connected devices, such as a smart home. If any devices are infected, the whole ecosystem can be compromised. Although various Machine Learning (ML) models are deployed to detect malware and network intrusion, generally speaking, robust high-accuracy models tend to require resources not found in all IoT devices, compared to less robust models defined by weak learners. In order to combat this issue, Fadhilla proposed a meta-learner ensemble model comprised of less robust prediction results inherent with weak learner ML models to produce a highly robust meta-learning ensemble model. The main problem with the prior research is that it cannot be deployed in low-end AIoT devices due to the limited resources comprising processing power, storage, and memory (the required libraries quickly exhaust low-end AIoT devices' resources.) Hence, this research aims to optimize the proposed super learner meta-learning ensemble model to make it viable for low-end AIoT devices. We show the library and ML model memory requirements associated with each optimization stage and emphasize that optimization of current ML models is necessitated for low-end AIoT devices. Our results demonstrate that we can obtain similar accuracy and False Positive Rate (FPR) metrics from high-end AIoT devices running the derived ML model, with a lower inference duration and smaller memory footprint.

Title: To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO

• Authors: Zi-Hao Qiu, Siqi Guo, Mao Xu, Tuo Zhao, Lijun Zhang, Tianbao Yang
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2404.04575
• Pdf URL: https://arxiv.org/pdf/2404.04575
• Copy Paste: [[2404.04575]] To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO(https://arxiv.org/abs/2404.04575)
• Keywords: robust, large language model
• Abstract: The temperature parameter plays a profound role during training and/or inference with large foundation models (LFMs) such as large language models (LLMs) and CLIP models. Particularly, it adjusts the logits in the softmax function in LLMs, which is crucial for next token generation, and it scales the similarities in the contrastive loss for training CLIP models. A significant question remains: Is it viable to learn a neural network to predict a personalized temperature of any input data for enhancing LFMs"? In this paper, we present a principled framework for learning a small yet generalizable temperature prediction network (TempNet) to improve LFMs. Our solution is composed of a novel learning framework with a robust loss underpinned by constrained distributionally robust optimization (DRO), and a properly designed TempNet with theoretical inspiration. TempNet can be trained together with a large foundation model from scratch or learned separately given a pretrained foundation model. It is not only useful for predicting personalized temperature to promote the training of LFMs but also generalizable and transferable to new tasks. Our experiments on LLMs and CLIP models demonstrate that TempNet greatly improves the performance of existing solutions or models, e.g. Table 1. The code to reproduce the experimental results in this paper can be found at https://github.com/zhqiu/TempNet.

Title: GLCM-Based Feature Combination for Extraction Model Optimization in Object Detection Using Machine Learning

• Authors: Florentina Tatrin Kurniati, Daniel HF Manongga, Eko Sediyono, Sri Yulianto Joko Prasetyo, Roy Rudolf Huizen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04578
• Pdf URL: https://arxiv.org/pdf/2404.04578
• Copy Paste: [[2404.04578]] GLCM-Based Feature Combination for Extraction Model Optimization in Object Detection Using Machine Learning(https://arxiv.org/abs/2404.04578)
• Keywords: security, extraction
• Abstract: In the era of modern technology, object detection using the Gray Level Co-occurrence Matrix (GLCM) extraction method plays a crucial role in object recognition processes. It finds applications in real-time scenarios such as security surveillance and autonomous vehicle navigation, among others. Computational efficiency becomes a critical factor in achieving real-time object detection. Hence, there is a need for a detection model with low complexity and satisfactory accuracy. This research aims to enhance computational efficiency by selecting appropriate features within the GLCM framework. Two classification models, namely K-Nearest Neighbours (K-NN) and Support Vector Machine (SVM), were employed, with the results indicating that K-Nearest Neighbours (K-NN) outperforms SVM in terms of computational complexity. Specifically, K-NN, when utilizing a combination of Correlation, Energy, and Homogeneity features, achieves a 100% accuracy rate with low complexity. Moreover, when using a combination of Energy and Homogeneity features, K-NN attains an almost perfect accuracy level of 99.9889%, while maintaining low complexity. On the other hand, despite SVM achieving 100% accuracy in certain feature combinations, its high or very high complexity can pose challenges, particularly in real-time applications. Therefore, based on the trade-off between accuracy and complexity, the K-NN model with a combination of Correlation, Energy, and Homogeneity features emerges as a more suitable choice for real-time applications that demand high accuracy and low complexity. This research provides valuable insights for optimizing object detection in various applications requiring both high accuracy and rapid responsiveness.

Title: SDFR: Synthetic Data for Face Recognition Competition

• Authors: Hatef Otroshi Shahreza, Christophe Ecabert, Anjith George, Alexander Unnervik, Sébastien Marcel, Nicolò Di Domenico, Guido Borghi, Davide Maltoni, Fadi Boutros, Julia Vogel, Naser Damer, Ángela Sánchez-Pérez, EnriqueMas-Candela, Jorge Calvo-Zaragoza, Bernardo Biesseck, Pedro Vidal, Roger Granada, David Menotti, Ivan DeAndres-Tame, Simone Maurizio La Cava, Sara Concas, Pietro Melzi, Ruben Tolosana, Ruben Vera-Rodriguez, Gianpaolo Perelli, Giulia Orrù, Gian Luca Marcialis, Julian Fierrez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04580
• Pdf URL: https://arxiv.org/pdf/2404.04580
• Copy Paste: [[2404.04580]] SDFR: Synthetic Data for Face Recognition Competition(https://arxiv.org/abs/2404.04580)
• Keywords: privacy, generative
• Abstract: Large-scale face recognition datasets are collected by crawling the Internet and without individuals' consent, raising legal, ethical, and privacy concerns. With the recent advances in generative models, recently several works proposed generating synthetic face recognition datasets to mitigate concerns in web-crawled face recognition datasets. This paper presents the summary of the Synthetic Data for Face Recognition (SDFR) Competition held in conjunction with the 18th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2024) and established to investigate the use of synthetic data for training face recognition models. The SDFR competition was split into two tasks, allowing participants to train face recognition systems using new synthetic datasets and/or existing ones. In the first task, the face recognition backbone was fixed and the dataset size was limited, while the second task provided almost complete freedom on the model backbone, the dataset, and the training pipeline. The submitted models were trained on existing and also new synthetic datasets and used clever methods to improve training with synthetic data. The submissions were evaluated and ranked on a diverse set of seven benchmarking datasets. The paper gives an overview of the submitted face recognition models and reports achieved performance compared to baseline models trained on real and synthetic datasets. Furthermore, the evaluation of submissions is extended to bias assessment across different demography groups. Lastly, an outlook on the current state of the research in training face recognition models using synthetic data is presented, and existing problems as well as potential future directions are also discussed.

Title: D$^3$: Scaling Up Deepfake Detection by Learning from Discrepancy

• Authors: Yongqi Yang, Zhihao Qian, Ye Zhu, Yu Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04584
• Pdf URL: https://arxiv.org/pdf/2404.04584
• Copy Paste: [[2404.04584]] D$^3$: Scaling Up Deepfake Detection by Learning from Discrepancy(https://arxiv.org/abs/2404.04584)
• Keywords: robust, generative
• Abstract: The boom of Generative AI brings opportunities entangled with risks and concerns. In this work, we seek a step toward a universal deepfake detection system with better generalization and robustness, to accommodate the responsible deployment of diverse image generative models. We do so by first scaling up the existing detection task setup from the one-generator to multiple-generators in training, during which we disclose two challenges presented in prior methodological designs. Specifically, we reveal that the current methods tailored for training on one specific generator either struggle to learn comprehensive artifacts from multiple generators or tend to sacrifice their ability to identify fake images from seen generators (i.e., In-Domain performance) to exchange the generalization for unseen generators (i.e., Out-Of-Domain performance). To tackle the above challenges, we propose our Discrepancy Deepfake Detector (D$^3$) framework, whose core idea is to learn the universal artifacts from multiple generators by introducing a parallel network branch that takes a distorted image as extra discrepancy signal to supplement its original counterpart. Extensive scaled-up experiments on the merged UFD and GenImage datasets with six detection models demonstrate the effectiveness of our framework, achieving a 5.3% accuracy improvement in the OOD testing compared to the current SOTA methods while maintaining the ID performance.

Title: PIE: Physics-inspired Low-light Enhancement

• Authors: Dong Liang, Zhengyan Xu, Ling Li, Mingqiang Wei, Songcan Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04586
• Pdf URL: https://arxiv.org/pdf/2404.04586
• Copy Paste: [[2404.04586]] PIE: Physics-inspired Low-light Enhancement(https://arxiv.org/abs/2404.04586)
• Keywords: segmentation
• Abstract: In this paper, we propose a physics-inspired contrastive learning paradigm for low-light enhancement, called PIE. PIE primarily addresses three issues: (i) To resolve the problem of existing learning-based methods often training a LLE model with strict pixel-correspondence image pairs, we eliminate the need for pixel-correspondence paired training data and instead train with unpaired images. (ii) To address the disregard for negative samples and the inadequacy of their generation in existing methods, we incorporate physics-inspired contrastive learning for LLE and design the Bag of Curves (BoC) method to generate more reasonable negative samples that closely adhere to the underlying physical imaging principle. (iii) To overcome the reliance on semantic ground truths in existing methods, we propose an unsupervised regional segmentation module, ensuring regional brightness consistency while eliminating the dependency on semantic ground truths. Overall, the proposed PIE can effectively learn from unpaired positive/negative samples and smoothly realize non-semantic regional enhancement, which is clearly different from existing LLE efforts. Besides the novel architecture of PIE, we explore the gain of PIE on downstream tasks such as semantic segmentation and face detection. Training on readily available open data and extensive experiments demonstrate that our method surpasses the state-of-the-art LLE models over six independent cross-scenes datasets. PIE runs fast with reasonable GFLOPs in test time, making it easy to use on mobile devices.

Title: Exploiting Sequence Number Leakage: TCP Hijacking in NAT-Enabled Wi-Fi Networks

• Authors: Yuxiang Yang, Xuewei Feng, Qi Li, Kun Sun, Ziqiang Wang, Ke Xu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04601
• Pdf URL: https://arxiv.org/pdf/2404.04601
• Copy Paste: [[2404.04601]] Exploiting Sequence Number Leakage: TCP Hijacking in NAT-Enabled Wi-Fi Networks(https://arxiv.org/abs/2404.04601)
• Keywords: attack
• Abstract: In this paper, we uncover a new side-channel vulnerability in the widely used NAT port preservation strategy and an insufficient reverse path validation strategy of Wi-Fi routers, which allows an off-path attacker to infer if there is one victim client in the same network communicating with another host on the Internet using TCP. After detecting the presence of TCP connections between the victim client and the server, the attacker can evict the original NAT mapping and reconstruct a new mapping at the router by sending fake TCP packets due to the routers' vulnerability of disabling TCP window tracking strategy, which has been faithfully implemented in most of the routers for years. In this way, the attacker can intercept TCP packets from the server and obtain the current sequence and acknowledgment numbers, which in turn allows the attacker to forcibly close the connection, poison the traffic in plain text, or reroute the server's incoming packets to the attacker. We test 67 widely used routers from 30 vendors and discover that 52 of them are affected by this attack. Also, we conduct an extensive measurement study on 93 real-world Wi-Fi networks. The experimental results show that 75 of these evaluated Wi-Fi networks (81%) are fully vulnerable to our attack. Our case study shows that it takes about 17.5, 19.4, and 54.5 seconds on average to terminate an SSH connection, download private files from FTP servers, and inject fake HTTP response packets with success rates of 87.4%, 82.6%, and 76.1%. We responsibly disclose the vulnerability and suggest mitigation strategies to all affected vendors and have received positive feedback, including acknowledgments, CVEs, rewards, and adoption of our suggestions.

Title: Panoptic Perception: A Novel Task and Fine-grained Dataset for Universal Remote Sensing Image Interpretation

• Authors: Danpei Zhao, Bo Yuan, Ziqiang Chen, Tian Li, Zhuoran Liu, Wentao Li, Yue Gao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04608
• Pdf URL: https://arxiv.org/pdf/2404.04608
• Copy Paste: [[2404.04608]] Panoptic Perception: A Novel Task and Fine-grained Dataset for Universal Remote Sensing Image Interpretation(https://arxiv.org/abs/2404.04608)
• Keywords: segmentation
• Abstract: Current remote-sensing interpretation models often focus on a single task such as detection, segmentation, or caption. However, the task-specific designed models are unattainable to achieve the comprehensive multi-level interpretation of images. The field also lacks support for multi-task joint interpretation datasets. In this paper, we propose Panoptic Perception, a novel task and a new fine-grained dataset (FineGrip) to achieve a more thorough and universal interpretation for RSIs. The new task, 1) integrates pixel-level, instance-level, and image-level information for universal image perception, 2) captures image information from coarse to fine granularity, achieving deeper scene understanding and description, and 3) enables various independent tasks to complement and enhance each other through multi-task learning. By emphasizing multi-task interactions and the consistency of perception results, this task enables the simultaneous processing of fine-grained foreground instance segmentation, background semantic segmentation, and global fine-grained image captioning. Concretely, the FineGrip dataset includes 2,649 remote sensing images, 12,054 fine-grained instance segmentation masks belonging to 20 foreground things categories, 7,599 background semantic masks for 5 stuff classes and 13,245 captioning sentences. Furthermore, we propose a joint optimization-based panoptic perception model. Experimental results on FineGrip demonstrate the feasibility of the panoptic perception task and the beneficial effect of multi-task joint optimization on individual tasks. The dataset will be publicly available.

Title: Vanishing Variance Problem in Fully Decentralized Neural-Network Systems

• Authors: Yongding Tian, Zaid Al-Ars, Maksim Kitsak, Peter Hofstee
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.04616
• Pdf URL: https://arxiv.org/pdf/2404.04616
• Copy Paste: [[2404.04616]] Vanishing Variance Problem in Fully Decentralized Neural-Network Systems(https://arxiv.org/abs/2404.04616)
• Keywords: privacy, federate
• Abstract: Federated learning and gossip learning are emerging methodologies designed to mitigate data privacy concerns by retaining training data on client devices and exclusively sharing locally-trained machine learning (ML) models with others. The primary distinction between the two lies in their approach to model aggregation: federated learning employs a centralized parameter server, whereas gossip learning adopts a fully decentralized mechanism, enabling direct model exchanges among nodes. This decentralized nature often positions gossip learning as less efficient compared to federated learning. Both methodologies involve a critical step: computing a representation of received ML models and integrating this representation into the existing model. Conventionally, this representation is derived by averaging the received models, exemplified by the FedAVG algorithm. Our findings suggest that this averaging approach inherently introduces a potential delay in model convergence. We identify the underlying cause and refer to it as the "vanishing variance" problem, where averaging across uncorrelated ML models undermines the optimal variance established by the Xavier weight initialization. Unlike federated learning where the central server ensures model correlation, and unlike traditional gossip learning which circumvents this problem through model partitioning and sampling, our research introduces a variance-corrected model averaging algorithm. This novel algorithm preserves the optimal variance needed during model averaging, irrespective of network topology or non-IID data distributions. Our extensive simulation results demonstrate that our approach enables gossip learning to achieve convergence efficiency comparable to that of federated learning.

Title: Empowering Image Recovery_ A Multi-Attention Approach

• Authors: Juan Wen (1 and 2), Yawei Li (2), Chao Zhang (3), Weiyan Hou (1), Radu Timofte (2 and 4), Luc Van Gool (2) ((1) Zhengzhou University, (2) Computer Vision Lab, ETH Zurich, (3) LAN-XEN, Technology, INC., (4) Bayerische Julius-Maximilians-Universität Würzburg)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04617
• Pdf URL: https://arxiv.org/pdf/2404.04617
• Copy Paste: [[2404.04617]] Empowering Image Recovery_ A Multi-Attention Approach(https://arxiv.org/abs/2404.04617)
• Keywords: transformer
• Abstract: We propose Diverse Restormer (DART), a novel image restoration method that effectively integrates information from various sources (long sequences, local and global regions, feature dimensions, and positional dimensions) to address restoration challenges. While Transformer models have demonstrated excellent performance in image restoration due to their self-attention mechanism, they face limitations in complex scenarios. Leveraging recent advancements in Transformers and various attention mechanisms, our method utilizes customized attention mechanisms to enhance overall performance. DART, our novel network architecture, employs windowed attention to mimic the selective focusing mechanism of human eyes. By dynamically adjusting receptive fields, it optimally captures the fundamental features crucial for image resolution reconstruction. Efficiency and performance balance are achieved through the LongIR attention mechanism for long sequence image restoration. Integration of attention mechanisms across feature and positional dimensions further enhances the recovery of fine details. Evaluation across five restoration tasks consistently positions DART at the forefront. Upon acceptance, we commit to providing publicly accessible code and models to ensure reproducibility and facilitate further research.

Title: Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

• Authors: Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, Wenqiang Lei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04626
• Pdf URL: https://arxiv.org/pdf/2404.04626
• Copy Paste: [[2404.04626]] Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective(https://arxiv.org/abs/2404.04626)
• Keywords: large language model
• Abstract: Direct Preference Optimization (DPO), which derives reward signals directly from pairwise preference data, has shown its effectiveness on aligning Large Language Models (LLMs) with human preferences. Despite its widespread use across various tasks, DPO has been criticized for its sensitivity to the SFT's effectiveness and its hindrance to the learning capacity towards human-preferred responses, leading to less satisfactory performance. To overcome those limitations, the theoretical understanding of DPO are indispensable but still lacking. To this end, we take a step towards theoretically analyzing and understanding the limitations of DPO. Specifically, we provide an analytical framework using the field theory to analyze the optimization process of DPO. By analyzing the gradient vector field of the DPO loss function, we find that the DPO loss function decreases the probability of producing human dispreferred data at a faster rate than it increases the probability of producing preferred data. This provides theoretical insights for understanding the limitations of DPO discovered in the related research experiments, thereby setting the foundation for its improvement.

Title: Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement

• Authors: Zaid Khan, Vijay Kumar BG, Samuel Schulter, Yun Fu, Manmohan Chandraker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04627
• Pdf URL: https://arxiv.org/pdf/2404.04627
• Copy Paste: [[2404.04627]] Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement(https://arxiv.org/abs/2404.04627)
• Keywords: large language model
• Abstract: Visual program synthesis is a promising approach to exploit the reasoning abilities of large language models for compositional computer vision tasks. Previous work has used few-shot prompting with frozen LLMs to synthesize visual programs. Training an LLM to write better visual programs is an attractive prospect, but it is unclear how to accomplish this. No dataset of visual programs for training exists, and acquisition of a visual program dataset cannot be easily crowdsourced due to the need for expert annotators. To get around the lack of direct supervision, we explore improving the program synthesis abilities of an LLM using feedback from interactive experience. We propose a method where we exploit existing annotations for a vision-language task to improvise a coarse reward signal for that task, treat the LLM as a policy, and apply reinforced self-training to improve the visual program synthesis ability of the LLM for that task. We describe a series of experiments on object detection, compositional visual question answering, and image-text retrieval, and show that in each case, the self-trained LLM outperforms or performs on par with few-shot frozen LLMs that are an order of magnitude larger. Website: https://zaidkhan.me/ViReP

Title: DifFUSER: Diffusion Model for Robust Multi-Sensor Fusion in 3D Object Detection and BEV Segmentation

• Authors: Duy-Tho Le, Hengcan Shi, Jianfei Cai, Hamid Rezatofighi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04629
• Pdf URL: https://arxiv.org/pdf/2404.04629
• Copy Paste: [[2404.04629]] DifFUSER: Diffusion Model for Robust Multi-Sensor Fusion in 3D Object Detection and BEV Segmentation(https://arxiv.org/abs/2404.04629)
• Keywords: robust, diffusion, transformer, generative, segmentation
• Abstract: Diffusion models have recently gained prominence as powerful deep generative models, demonstrating unmatched performance across various domains. However, their potential in multi-sensor fusion remains largely unexplored. In this work, we introduce DifFUSER, a novel approach that leverages diffusion models for multi-modal fusion in 3D object detection and BEV map segmentation. Benefiting from the inherent denoising property of diffusion, DifFUSER is able to refine or even synthesize sensor features in case of sensor malfunction, thereby improving the quality of the fused output. In terms of architecture, our DifFUSER blocks are chained together in a hierarchical BiFPN fashion, termed cMini-BiFPN, offering an alternative architecture for latent diffusion. We further introduce a Gated Self-conditioned Modulated (GSM) latent diffusion module together with a Progressive Sensor Dropout Training (PSDT) paradigm, designed to add stronger conditioning to the diffusion process and robustness to sensor failures. Our extensive evaluations on the Nuscenes dataset reveal that DifFUSER not only achieves state-of-the-art performance with a 69.1% mIOU in BEV map segmentation tasks but also competes effectively with leading transformer-based fusion techniques in 3D object detection.

Title: On the Limitations of Large Language Models (LLMs): False Attribution

• Authors: Tosin Adewumi, Nudrat Habib, Lama Alkhaled, Elisa Barney
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04631
• Pdf URL: https://arxiv.org/pdf/2404.04631
• Copy Paste: [[2404.04631]] On the Limitations of Large Language Models (LLMs): False Attribution(https://arxiv.org/abs/2404.04631)
• Keywords: large language model
• Abstract: In this work, we provide insight into one important limitation of large language models (LLMs), i.e. false attribution, and introduce a new hallucination metric - Simple Hallucination Index (SHI). The task of automatic author attribution for relatively small chunks of text is an important NLP task but can be challenging. We empirically evaluate the power of 3 open SotA LLMs in zero-shot setting (LLaMA-2-13B, Mixtral 8x7B, and Gemma-7B), especially as human annotation can be costly. We collected the top 10 most popular books, according to Project Gutenberg, divided each one into equal chunks of 400 words, and asked each LLM to predict the author. We then randomly sampled 162 chunks for human evaluation from each of the annotated books, based on the error margin of 7% and a confidence level of 95% for the book with the most chunks (Great Expectations by Charles Dickens, having 922 chunks). The average results show that Mixtral 8x7B has the highest prediction accuracy, the lowest SHI, and a Pearson's correlation (r) of 0.737, 0.249, and -0.9996, respectively, followed by LLaMA-2-13B and Gemma-7B. However, Mixtral 8x7B suffers from high hallucinations for 3 books, rising as high as an SHI of 0.87 (in the range 0-1, where 1 is the worst). The strong negative correlation of accuracy and SHI, given by r, demonstrates the fidelity of the new hallucination metric, which is generalizable to other tasks. We publicly release the annotated chunks of data and our codes to aid the reproducibility and evaluation of other models.

Title: CANEDERLI: On The Impact of Adversarial Training and Transferability on CAN Intrusion Detection Systems

• Authors: Francesco Marchiori, Mauro Conti
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04648
• Pdf URL: https://arxiv.org/pdf/2404.04648
• Copy Paste: [[2404.04648]] CANEDERLI: On The Impact of Adversarial Training and Transferability on CAN Intrusion Detection Systems(https://arxiv.org/abs/2404.04648)
• Keywords: attack
• Abstract: The growing integration of vehicles with external networks has led to a surge in attacks targeting their Controller Area Network (CAN) internal bus. As a countermeasure, various Intrusion Detection Systems (IDSs) have been suggested in the literature to prevent and mitigate these threats. With the increasing volume of data facilitated by the integration of Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication networks, most of these systems rely on data-driven approaches such as Machine Learning (ML) and Deep Learning (DL) models. However, these systems are susceptible to adversarial evasion attacks. While many researchers have explored this vulnerability, their studies often involve unrealistic assumptions, lack consideration for a realistic threat model, and fail to provide effective solutions. In this paper, we present CANEDERLI (CAN Evasion Detection ResiLIence), a novel framework for securing CAN-based IDSs. Our system considers a realistic threat model and addresses the impact of adversarial attacks on DL-based detection systems. Our findings highlight strong transferability properties among diverse attack methodologies by considering multiple state-of-the-art attacks and model architectures. We analyze the impact of adversarial training in addressing this threat and propose an adaptive online adversarial training technique outclassing traditional fine-tuning methodologies with F1 scores up to 0.941. By making our framework publicly available, we aid practitioners and researchers in assessing the resilience of IDSs to a varied adversarial landscape.

Title: InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization

• Authors: Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, Di Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04650
• Pdf URL: https://arxiv.org/pdf/2404.04650
• Copy Paste: [[2404.04650]] InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization(https://arxiv.org/abs/2404.04650)
• Keywords: diffusion
• Abstract: Recent strides in the development of diffusion models, exemplified by advancements such as Stable Diffusion, have underscored their remarkable prowess in generating visually compelling images. However, the imperative of achieving a seamless alignment between the generated image and the provided prompt persists as a formidable challenge. This paper traces the root of these difficulties to invalid initial noise, and proposes a solution in the form of Initial Noise Optimization (InitNO), a paradigm that refines this noise. Considering text prompts, not all random noises are effective in synthesizing semantically-faithful images. We design the cross-attention response score and the self-attention conflict score to evaluate the initial noise, bifurcating the initial latent space into valid and invalid sectors. A strategically crafted noise optimization pipeline is developed to guide the initial noise towards valid regions. Our method, validated through rigorous experimentation, shows a commendable proficiency in generating images in strict accordance with text prompts. Our code is available at https://github.com/xiefan-guo/initno.

Title: HawkDrive: A Transformer-driven Visual Perception System for Autonomous Driving in Night Scene

• Authors: Ziang Guo, Stepan Perminov, Mikhail Konenkov, Dzmitry Tsetserukou
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.04653
• Pdf URL: https://arxiv.org/pdf/2404.04653
• Copy Paste: [[2404.04653]] HawkDrive: A Transformer-driven Visual Perception System for Autonomous Driving in Night Scene(https://arxiv.org/abs/2404.04653)
• Keywords: transformer, segmentation
• Abstract: Many established vision perception systems for autonomous driving scenarios ignore the influence of light conditions, one of the key elements for driving safety. To address this problem, we present HawkDrive, a novel perception system with hardware and software solutions. Hardware that utilizes stereo vision perception, which has been demonstrated to be a more reliable way of estimating depth information than monocular vision, is partnered with the edge computing device Nvidia Jetson Xavier AGX. Our software for low light enhancement, depth estimation, and semantic segmentation tasks, is a transformer-based neural network. Our software stack, which enables fast inference and noise reduction, is packaged into system modules in Robot Operating System 2 (ROS2). Our experimental results have shown that the proposed end-to-end system is effective in improving the depth estimation and semantic segmentation performance. Our dataset and codes will be released at https://github.com/ZionGo6/HawkDrive.

Title: Music Recommendation Based on Facial Emotion Recognition

• Authors: Rajesh B, Keerthana V, Narayana Darapaneni, Anwesh Reddy P
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.04654
• Pdf URL: https://arxiv.org/pdf/2404.04654
• Copy Paste: [[2404.04654]] Music Recommendation Based on Facial Emotion Recognition(https://arxiv.org/abs/2404.04654)
• Keywords: robust
• Abstract: Introduction: Music provides an incredible avenue for individuals to express their thoughts and emotions, while also serving as a delightful mode of entertainment for enthusiasts and music lovers. Objectives: This paper presents a comprehensive approach to enhancing the user experience through the integration of emotion recognition, music recommendation, and explainable AI using GRAD-CAM. Methods: The proposed methodology utilizes a ResNet50 model trained on the Facial Expression Recognition (FER) dataset, consisting of real images of individuals expressing various emotions. Results: The system achieves an accuracy of 82% in emotion classification. By leveraging GRAD-CAM, the model provides explanations for its predictions, allowing users to understand the reasoning behind the system's recommendations. The model is trained on both FER and real user datasets, which include labelled facial expressions, and real images of individuals expressing various emotions. The training process involves pre-processing the input images, extracting features through convolutional layers, reasoning with dense layers, and generating emotion predictions through the output layer Conclusion: The proposed methodology, leveraging the Resnet50 model with ROI-based analysis and explainable AI techniques, offers a robust and interpretable solution for facial emotion detection paper.

Title: Binary Classifier Optimization for Large Language Model Alignment

• Authors: Seungjae Jung, Gunsoo Han, Daniel Wontae Nam, Kyoung-Woon On
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04656
• Pdf URL: https://arxiv.org/pdf/2404.04656
• Copy Paste: [[2404.04656]] Binary Classifier Optimization for Large Language Model Alignment(https://arxiv.org/abs/2404.04656)
• Keywords: robust, large language model
• Abstract: Aligning Large Language Models (LLMs) to human preferences through preference optimization has been crucial but labor-intensive, necessitating for each prompt a comparison of both a chosen and a rejected text completion by evaluators. Recently, Kahneman-Tversky Optimization (KTO) has demonstrated that LLMs can be aligned using merely binary "thumbs-up" or "thumbs-down" signals on each prompt-completion pair. In this paper, we present theoretical foundations to explain the successful alignment achieved through these binary signals. Our analysis uncovers a new perspective: optimizing a binary classifier, whose logit is a reward, implicitly induces minimizing the Direct Preference Optimization (DPO) loss. In the process of this discovery, we identified two techniques for effective alignment: reward shift and underlying distribution matching. Consequently, we propose a new algorithm, \textit{Binary Classifier Optimization}, that integrates the techniques. We validate our methodology in two settings: first, on a paired preference dataset, where our method performs on par with DPO and KTO; and second, on binary signal datasets simulating real-world conditions with divergent underlying distributions between thumbs-up and thumbs-down data. Our model consistently demonstrates effective and robust alignment across two base LLMs and three different binary signal datasets, showcasing the strength of our approach to learning from binary feedback.

Title: Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly

• Authors: Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, Shujian Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04659
• Pdf URL: https://arxiv.org/pdf/2404.04659
• Copy Paste: [[2404.04659]] Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly(https://arxiv.org/abs/2404.04659)
• Keywords: large language model
• Abstract: Despite their strong ability to retrieve knowledge in English, current large language models show imbalance abilities in different languages. Two approaches are proposed to address this, i.e., multilingual pretraining and multilingual instruction tuning. However, whether and how do such methods contribute to the cross-lingual knowledge alignment inside the models is unknown. In this paper, we propose CLiKA, a systematic framework to assess the cross-lingual knowledge alignment of LLMs in the Performance, Consistency and Conductivity levels, and explored the effect of multilingual pretraining and instruction tuning on the degree of alignment. Results show that: while both multilingual pretraining and instruction tuning are beneficial for cross-lingual knowledge alignment, the training strategy needs to be carefully designed. Namely, continued pretraining improves the alignment of the target language at the cost of other languages, while mixed pretraining affect other languages less. Also, the overall cross-lingual knowledge alignment, especially in the conductivity level, is unsatisfactory for all tested LLMs, and neither multilingual pretraining nor instruction tuning can substantially improve the cross-lingual knowledge conductivity.

Title: Learning Minimal NAP Specifications for Neural Network Verification

• Authors: Chuqin Geng, Zhaoyue Wang, Haolin Ye, Saifei Liao, Xujie Si
• Subjects: cs.LG, cs.PL
• Abstract URL: https://arxiv.org/abs/2404.04662
• Pdf URL: https://arxiv.org/pdf/2404.04662
• Copy Paste: [[2404.04662]] Learning Minimal NAP Specifications for Neural Network Verification(https://arxiv.org/abs/2404.04662)
• Keywords: robust
• Abstract: Specifications play a crucial role in neural network verification. They define the precise input regions we aim to verify, typically represented as L-infinity norm balls. While recent research suggests using neural activation patterns (NAPs) as specifications for verifying unseen test set data, it focuses on computing the most refined NAPs, often limited to very small regions in the input space. In this paper, we study the following problem: Given a neural network, find a minimal (coarsest) NAP that is sufficient for formal verification of the network's robustness. Finding the minimal NAP specification not only expands verifiable bounds but also provides insights into which neurons contribute to the model's robustness. To address this problem, we propose several exact and approximate approaches. Our exact approaches leverage the verification tool to find minimal NAP specifications in either a deterministic or statistical manner. Whereas the approximate methods efficiently estimate minimal NAPs using adversarial examples and local gradients, without making calls to the verification tool. This allows us to inspect potential causal links between neurons and the robustness of state-of-the-art neural networks, a task for which existing verification frameworks fail to scale. Our experimental results suggest that minimal NAP specifications require much smaller fractions of neurons compared to the most refined NAP specifications, yet they can significantly expand the verifiable boundaries to several orders of magnitude larger.

Title: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

• Authors: Nicolas Yax, Pierre-Yves Oudeyer, Stefano Palminteri
• Subjects: cs.CL, cs.LG, q-bio.PE
• Abstract URL: https://arxiv.org/abs/2404.04671
• Pdf URL: https://arxiv.org/pdf/2404.04671
• Copy Paste: [[2404.04671]] Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks(https://arxiv.org/abs/2404.04671)
• Keywords: large language model
• Abstract: This paper introduces PhyloLM, a method applying phylogenetic algorithms to Large Language Models to explore their finetuning relationships, and predict their performance characteristics. By leveraging the phylogenetic distance metric, we construct dendrograms, which satisfactorily capture distinct LLM families (across a set of 77 open-source and 22 closed models). Furthermore, phylogenetic distance predicts performances in benchmarks (we test MMLU and ARC), thus enabling a time and cost-effective estimation of LLM capabilities. The approach translates genetic concepts to machine learning, offering tools to infer LLM development, relationships, and capabilities, even in the absence of transparent training information.

Title: Order-Based Pre-training Strategies for Procedural Text Understanding

• Authors: Abhilash Nandy, Yash Kulkarni, Pawan Goyal, Niloy Ganguly
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04676
• Pdf URL: https://arxiv.org/pdf/2404.04676
• Copy Paste: [[2404.04676]] Order-Based Pre-training Strategies for Procedural Text Understanding(https://arxiv.org/abs/2404.04676)
• Keywords: transformer
• Abstract: In this paper, we propose sequence-based pretraining methods to enhance procedural understanding in natural language processing. Procedural text, containing sequential instructions to accomplish a task, is difficult to understand due to the changing attributes of entities in the context. We focus on recipes, which are commonly represented as ordered instructions, and use this order as a supervision signal. Our work is one of the first to compare several 'order as-supervision' transformer pre-training methods, including Permutation Classification, Embedding Regression, and Skip-Clip, and shows that these methods give improved results compared to the baselines and SoTA LLMs on two downstream Entity-Tracking datasets: NPN-Cooking dataset in recipe domain and ProPara dataset in open domain. Our proposed methods address the non-trivial Entity Tracking Task that requires prediction of entity states across procedure steps, which requires understanding the order of steps. These methods show an improvement over the best baseline by 1.6% and 7-9% on NPN-Cooking and ProPara Datasets respectively across metrics.

Title: Salient Sparse Visual Odometry With Pose-Only Supervision

• Authors: Siyu Chen, Kangcheng Liu, Chen Wang, Shenghai Yuan, Jianfei Yang, Lihua Xie
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.04677
• Pdf URL: https://arxiv.org/pdf/2404.04677
• Copy Paste: [[2404.04677]] Salient Sparse Visual Odometry With Pose-Only Supervision(https://arxiv.org/abs/2404.04677)
• Keywords: robust, extraction
• Abstract: Visual Odometry (VO) is vital for the navigation of autonomous systems, providing accurate position and orientation estimates at reasonable costs. While traditional VO methods excel in some conditions, they struggle with challenges like variable lighting and motion blur. Deep learning-based VO, though more adaptable, can face generalization problems in new environments. Addressing these drawbacks, this paper presents a novel hybrid visual odometry (VO) framework that leverages pose-only supervision, offering a balanced solution between robustness and the need for extensive labeling. We propose two cost-effective and innovative designs: a self-supervised homographic pre-training for enhancing optical flow learning from pose-only labels and a random patch-based salient point detection strategy for more accurate optical flow patch extraction. These designs eliminate the need for dense optical flow labels for training and significantly improve the generalization capability of the system in diverse and challenging environments. Our pose-only supervised method achieves competitive performance on standard datasets and greater robustness and generalization ability in extreme and unseen scenarios, even compared to dense optical flow-supervised state-of-the-art methods.

Title: OmniColor: A Global Camera Pose Optimization Approach of LiDAR-360Camera Fusion for Colorizing Point Clouds

• Authors: Bonan Liu, Guoyang Zhao, Jianhao Jiao, Guang Cai, Chengyang Li, Handi Yin, Yuyang Wang, Ming Liu, Pan Hui
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.04693
• Pdf URL: https://arxiv.org/pdf/2404.04693
• Copy Paste: [[2404.04693]] OmniColor: A Global Camera Pose Optimization Approach of LiDAR-360Camera Fusion for Colorizing Point Clouds(https://arxiv.org/abs/2404.04693)
• Keywords: extraction
• Abstract: A Colored point cloud, as a simple and efficient 3D representation, has many advantages in various fields, including robotic navigation and scene reconstruction. This representation is now commonly used in 3D reconstruction tasks relying on cameras and LiDARs. However, fusing data from these two types of sensors is poorly performed in many existing frameworks, leading to unsatisfactory mapping results, mainly due to inaccurate camera poses. This paper presents OmniColor, a novel and efficient algorithm to colorize point clouds using an independent 360-degree camera. Given a LiDAR-based point cloud and a sequence of panorama images with initial coarse camera poses, our objective is to jointly optimize the poses of all frames for mapping images onto geometric reconstructions. Our pipeline works in an off-the-shelf manner that does not require any feature extraction or matching process. Instead, we find optimal poses by directly maximizing the photometric consistency of LiDAR maps. In experiments, we show that our method can overcome the severe visual distortion of omnidirectional images and greatly benefit from the wide field of view (FOV) of 360-degree cameras to reconstruct various scenarios with accuracy and stability. The code will be released at https://github.com/liubonan123/OmniColor/.

Title: Advances in Differential Privacy and Differentially Private Machine Learning

• Authors: Saswat Das, Subhankar Mishra
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04706
• Pdf URL: https://arxiv.org/pdf/2404.04706
• Copy Paste: [[2404.04706]] Advances in Differential Privacy and Differentially Private Machine Learning(https://arxiv.org/abs/2404.04706)
• Keywords: privacy
• Abstract: There has been an explosion of research on differential privacy (DP) and its various applications in recent years, ranging from novel variants and accounting techniques in differential privacy to the thriving field of differentially private machine learning (DPML) to newer implementations in practice, like those by various companies and organisations such as census bureaus. Most recent surveys focus on the applications of differential privacy in particular contexts like data publishing, specific machine learning tasks, analysis of unstructured data, location privacy, etc. This work thus seeks to fill the gap for a survey that primarily discusses recent developments in the theory of differential privacy along with newer DP variants, viz. Renyi DP and Concentrated DP, novel mechanisms and techniques, and the theoretical developments in differentially private machine learning in proper detail. In addition, this survey discusses its applications to privacy-preserving machine learning in practice and a few practical implementations of DP.

Title: Data Poisoning Attacks on Off-Policy Policy Evaluation Methods

• Authors: Elita Lobo, Harvineet Singh, Marek Petrik, Cynthia Rudin, Himabindu Lakkaraju
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04714
• Pdf URL: https://arxiv.org/pdf/2404.04714
• Copy Paste: [[2404.04714]] Data Poisoning Attacks on Off-Policy Policy Evaluation Methods(https://arxiv.org/abs/2404.04714)
• Keywords: attack, robust
• Abstract: Off-policy Evaluation (OPE) methods are a crucial tool for evaluating policies in high-stakes domains such as healthcare, where exploration is often infeasible, unethical, or expensive. However, the extent to which such methods can be trusted under adversarial threats to data quality is largely unexplored. In this work, we make the first attempt at investigating the sensitivity of OPE methods to marginal adversarial perturbations to the data. We design a generic data poisoning attack framework leveraging influence functions from robust statistics to carefully construct perturbations that maximize error in the policy value estimates. We carry out extensive experimentation with multiple healthcare and control datasets. Our results demonstrate that many existing OPE methods are highly prone to generating value estimates with large errors when subject to data poisoning attacks, even for small adversarial perturbations. These findings question the reliability of policy values derived using OPE methods and motivate the need for developing OPE methods that are statistically robust to train-time data poisoning attacks.

Title: PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics

• Authors: Derui Zhu, Dingfan Chen, Qing Li, Zongxiong Chen, Lei Ma, Jens Grossklags, Mario Fritz
• Subjects: cs.CL, cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.04722
• Pdf URL: https://arxiv.org/pdf/2404.04722
• Copy Paste: [[2404.04722]] PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics(https://arxiv.org/abs/2404.04722)
• Keywords: large language model
• Abstract: Despite tremendous advancements in large language models (LLMs) over recent years, a notably urgent challenge for their practical deployment is the phenomenon of hallucination, where the model fabricates facts and produces non-factual statements. In response, we propose PoLLMgraph, a Polygraph for LLMs, as an effective model-based white-box detection and forecasting approach. PoLLMgraph distinctly differs from the large body of existing research that concentrates on addressing such challenges through black-box evaluations. In particular, we demonstrate that hallucination can be effectively detected by analyzing the LLM's internal state transition dynamics during generation via tractable probabilistic models. Experimental results on various open-source LLMs confirm the efficacy of PoLLMgraph, outperforming state-of-the-art methods by a considerable margin, evidenced by over 20% improvement in AUC-ROC on common benchmarking datasets like TruthfulQA. Our work paves a new way for model-based white-box analysis of LLMs, motivating the research community to further explore, understand, and refine the intricate dynamics of LLM behaviors.

Title: We need to aim at the top: Factors associated with cybersecurity awareness of cyber and information security decision-makers

• Authors: Simon Vrhovec, Blaž Markelj
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04725
• Pdf URL: https://arxiv.org/pdf/2404.04725
• Copy Paste: [[2404.04725]] We need to aim at the top: Factors associated with cybersecurity awareness of cyber and information security decision-makers(https://arxiv.org/abs/2404.04725)
• Keywords: security, attack
• Abstract: Cyberattacks pose a significant business risk to organizations. Although there is ample literature focusing on why people pose a major risk to organizational cybersecurity and how to deal with it, there is surprisingly little we know about cyber and information security decision-makers who are essentially the people in charge of setting up and maintaining organizational cybersecurity. In this paper, we study cybersecurity awareness of cyber and information security decision-makers, and investigate factors associated with it. We conducted an online survey among Slovenian cyber and information security decision-makers (N=283) to (1) determine whether their cybersecurity awareness is associated with adoption of antimalware solutions in their organizations, and (2) explore which organizational factors and personal characteristics are associated with their cybersecurity awareness. Our findings indicate that awareness of well-known threats and solutions seems to be quite low for individuals in decision-making roles. They also provide insights into which threats and solutions are cyber and information security decision-makers the least aware of. We uncovered that awareness of certain threats and solutions is positively associated with either adoption of advanced antimalware solutions with EDR/XDR capabilities or adoption of SOC. Additionally, we identified significant organizational factors (organizational role type) and personal characteristics (gender, age, experience with information security and experience with IT) related to cybersecurity awareness of cyber and information security decision-makers. Organization size and formal education were not significant. These results offer insights that can be leveraged in targeted cybersecurity training tailored to the needs of groups of cyber and information security decision-makers based on these key factors.

Title: Navigating the Landscape of Hint Generation Research: From the Past to the Future

• Authors: Anubhav Jangra, Jamshid Mozafari, Adam Jatowt, Smaranda Muresan
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.04728
• Pdf URL: https://arxiv.org/pdf/2404.04728
• Copy Paste: [[2404.04728]] Navigating the Landscape of Hint Generation Research: From the Past to the Future(https://arxiv.org/abs/2404.04728)
• Keywords: large language model
• Abstract: Digital education has gained popularity in the last decade, especially after the COVID-19 pandemic. With the improving capabilities of large language models to reason and communicate with users, envisioning intelligent tutoring systems (ITSs) that can facilitate self-learning is not very far-fetched. One integral component to fulfill this vision is the ability to give accurate and effective feedback via hints to scaffold the learning process. In this survey article, we present a comprehensive review of prior research on hint generation, aiming to bridge the gap between research in education and cognitive science, and research in AI and Natural Language Processing. Informed by our findings, we propose a formal definition of the hint generation task, and discuss the roadmap of building an effective hint generation system aligned with the formal definition, including open challenges, future directions and ethical considerations.

Title: ProtoAL: Interpretable Deep Active Learning with prototypes for medical imaging

• Authors: Iury B. de A. Santos, André C.P.L.F. de Carvalho
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04736
• Pdf URL: https://arxiv.org/pdf/2404.04736
• Copy Paste: [[2404.04736]] ProtoAL: Interpretable Deep Active Learning with prototypes for medical imaging(https://arxiv.org/abs/2404.04736)
• Keywords: interpretability
• Abstract: The adoption of Deep Learning algorithms in the medical imaging field is a prominent area of research, with high potential for advancing AI-based Computer-aided diagnosis (AI-CAD) solutions. However, current solutions face challenges due to a lack of interpretability features and high data demands, prompting recent efforts to address these issues. In this study, we propose the ProtoAL method, where we integrate an interpretable DL model into the Deep Active Learning (DAL) framework. This approach aims to address both challenges by focusing on the medical imaging context and utilizing an inherently interpretable model based on prototypes. We evaluated ProtoAL on the Messidor dataset, achieving an area under the precision-recall curve of 0.79 while utilizing only 76.54\% of the available labeled data. These capabilities can enhances the practical usability of a DL model in the medical field, providing a means of trust calibration in domain experts and a suitable solution for learning in the data scarcity context often found.

Title: Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind

• Authors: Hongchuan Zeng, Hongshen Xu, Lu Chen, Kai Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04748
• Pdf URL: https://arxiv.org/pdf/2404.04748
• Copy Paste: [[2404.04748]] Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind(https://arxiv.org/abs/2404.04748)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have ushered in a new era in Natural Language Processing, but their massive size demands effective compression techniques for practicality. Although numerous model compression techniques have been investigated, they typically rely on a calibration set that overlooks the multilingual context and results in significant accuracy degradation for low-resource languages. This paper introduces Multilingual Brain Surgeon (MBS), a novel calibration data sampling method for multilingual LLMs compression. MBS overcomes the English-centric limitations of existing methods by sampling calibration data from various languages proportionally to the language distribution of the model training datasets. Our experiments, conducted on the BLOOM multilingual LLM, demonstrate that MBS improves the performance of existing English-centric compression methods, especially for low-resource languages. We also uncover the dynamics of language interaction during compression, revealing that the larger the proportion of a language in the training set and the more similar the language is to the calibration language, the better performance the language retains after compression. In conclusion, MBS presents an innovative approach to compressing multilingual LLMs, addressing the performance disparities and improving the language inclusivity of existing compression techniques.

Title: GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling

• Authors: Hritik Bansal, Po-Nien Kung, P. Jeffrey Brantingham, Kai-Wei Chang, Nanyun Peng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04763
• Pdf URL: https://arxiv.org/pdf/2404.04763
• Copy Paste: [[2404.04763]] GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling(https://arxiv.org/abs/2404.04763)
• Keywords: generative, large language model
• Abstract: Multimodal event argument role labeling (EARL), a task that assigns a role for each event participant (object) in an image is a complex challenge. It requires reasoning over the entire image, the depicted event, and the interactions between various objects participating in the event. Existing models heavily rely on high-quality event-annotated training data to understand the event semantics and structures, and they fail to generalize to new event types and domains. In this paper, we propose GenEARL, a training-free generative framework that harness the power of the modern generative models to understand event task descriptions given image contexts to perform the EARL task. Specifically, GenEARL comprises two stages of generative prompting with a frozen vision-language model (VLM) and a frozen large language model (LLM). First, a generative VLM learns the semantics of the event argument roles and generates event-centric object descriptions based on the image. Subsequently, a LLM is prompted with the generated object descriptions with a predefined template for EARL (i.e., assign an object with an event argument role). We show that GenEARL outperforms the contrastive pretraining (CLIP) baseline by 9.4% and 14.2% accuracy for zero-shot EARL on the M2E2 and SwiG datasets, respectively. In addition, we outperform CLIP-Event by 22% precision on M2E2 dataset. The framework also allows flexible adaptation and generalization to unseen domains.

Title: Safeguarding Voice Privacy: Harnessing Near-Ultrasonic Interference To Protect Against Unauthorized Audio Recording

• Authors: Forrest McKee, David Noever
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04769
• Pdf URL: https://arxiv.org/pdf/2404.04769
• Copy Paste: [[2404.04769]] Safeguarding Voice Privacy: Harnessing Near-Ultrasonic Interference To Protect Against Unauthorized Audio Recording(https://arxiv.org/abs/2404.04769)
• Keywords: security, privacy, protect, robust
• Abstract: The widespread adoption of voice-activated systems has modified routine human-machine interaction but has also introduced new vulnerabilities. This paper investigates the susceptibility of automatic speech recognition (ASR) algorithms in these systems to interference from near-ultrasonic noise. Building upon prior research that demonstrated the ability of near-ultrasonic frequencies (16 kHz - 22 kHz) to exploit the inherent properties of microelectromechanical systems (MEMS) microphones, our study explores alternative privacy enforcement means using this interference phenomenon. We expose a critical vulnerability in the most common microphones used in modern voice-activated devices, which inadvertently demodulate near-ultrasonic frequencies into the audible spectrum, disrupting the ASR process. Through a systematic analysis of the impact of near-ultrasonic noise on various ASR systems, we demonstrate that this vulnerability is consistent across different devices and under varying conditions, such as broadcast distance and specific phoneme structures. Our findings highlight the need to develop robust countermeasures to protect voice-activated systems from malicious exploitation of this vulnerability. Furthermore, we explore the potential applications of this phenomenon in enhancing privacy by disrupting unauthorized audio recording or eavesdropping. This research underscores the importance of a comprehensive approach to securing voice-activated systems, combining technological innovation, responsible development practices, and informed policy decisions to ensure the privacy and security of users in an increasingly connected world.

Title: Generating Uncontextualized and Contextualized Questions for Document-Level Event Argument Extraction

• Authors: Md Nayem Uddin, Enfa Rose George, Eduardo Blanco, Steven Corman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04770
• Pdf URL: https://arxiv.org/pdf/2404.04770
• Copy Paste: [[2404.04770]] Generating Uncontextualized and Contextualized Questions for Document-Level Event Argument Extraction(https://arxiv.org/abs/2404.04770)
• Keywords: extraction
• Abstract: This paper presents multiple question generation strategies for document-level event argument extraction. These strategies do not require human involvement and result in uncontextualized questions as well as contextualized questions grounded on the event and document of interest. Experimental results show that combining uncontextualized and contextualized questions is beneficial, especially when event triggers and arguments appear in different sentences. Our approach does not have corpus-specific components, in particular, the question generation strategies transfer across corpora. We also present a qualitative analysis of the most common errors made by our best model.

Title: Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution

• Authors: Guangyuan Li, Chen Rao, Juncheng Mo, Zhanjie Zhang, Wei Xing, Lei Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04785
• Pdf URL: https://arxiv.org/pdf/2404.04785
• Copy Paste: [[2404.04785]] Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution(https://arxiv.org/abs/2404.04785)
• Keywords: diffusion, transformer
• Abstract: Recently, diffusion models (DM) have been applied in magnetic resonance imaging (MRI) super-resolution (SR) reconstruction, exhibiting impressive performance, especially with regard to detailed reconstruction. However, the current DM-based SR reconstruction methods still face the following issues: (1) They require a large number of iterations to reconstruct the final image, which is inefficient and consumes a significant amount of computational resources. (2) The results reconstructed by these methods are often misaligned with the real high-resolution images, leading to remarkable distortion in the reconstructed MR images. To address the aforementioned issues, we propose an efficient diffusion model for multi-contrast MRI SR, named as DiffMSR. Specifically, we apply DM in a highly compact low-dimensional latent space to generate prior knowledge with high-frequency detail information. The highly compact latent space ensures that DM requires only a few simple iterations to produce accurate prior knowledge. In addition, we design the Prior-Guide Large Window Transformer (PLWformer) as the decoder for DM, which can extend the receptive field while fully utilizing the prior knowledge generated by DM to ensure that the reconstructed MR image remains undistorted. Extensive experiments on public and clinical datasets demonstrate that our DiffMSR outperforms state-of-the-art methods.

Title: SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

• Authors: Zihao Wang, Shaoduo Gan
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04793
• Pdf URL: https://arxiv.org/pdf/2404.04793
• Copy Paste: [[2404.04793]] SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget(https://arxiv.org/abs/2404.04793)
• Keywords: large language model
• Abstract: Optimizing the Key-Value (KV) cache of the Large Language Model (LLM) has been considered critical to saving the cost of inference. Most of the existing KV-cache compression algorithms attempted to sparsify the sequence of tokens by taking advantage of the different importance of tokens. In this work, we found that by identifying the importance of attention layers, we could optimize the KV-cache jointly from two dimensions. Based on our observations regarding layer-wise importance in inference, we propose SqueezeAttention to precisely optimize the allocation of KV-cache budget among layers on-the-fly and then incorporate three representative token sparsification algorithms to compress the KV-cache for each layer with its very own budget. By optimizing the KV-cache from both sequence's and layer's dimensions, SqueezeAttention achieves around 30% to 70% of the memory reductions and up to 2.2 times of throughput improvements in a wide range of LLMs and benchmarks. The code is available at https://github.com/hetailang/SqueezeAttention.

Title: Coordinated Sparse Recovery of Label Noise

• Authors: Yukun Yang, Naihao Wang, Haixin Yang, Ruirui Li
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.04800
• Pdf URL: https://arxiv.org/pdf/2404.04800
• Copy Paste: [[2404.04800]] Coordinated Sparse Recovery of Label Noise(https://arxiv.org/abs/2404.04800)
• Keywords: robust
• Abstract: Label noise is a common issue in real-world datasets that inevitably impacts the generalization of models. This study focuses on robust classification tasks where the label noise is instance-dependent. Estimating the transition matrix accurately in this task is challenging, and methods based on sample selection often exhibit confirmation bias to varying degrees. Sparse over-parameterized training (SOP) has been theoretically effective in estimating and recovering label noise, offering a novel solution for noise-label learning. However, this study empirically observes and verifies a technical flaw of SOP: the lack of coordination between model predictions and noise recovery leads to increased generalization error. To address this, we propose a method called Coordinated Sparse Recovery (CSR). CSR introduces a collaboration matrix and confidence weights to coordinate model predictions and noise recovery, reducing error leakage. Based on CSR, this study designs a joint sample selection strategy and constructs a comprehensive and powerful learning framework called CSR+. CSR+ significantly reduces confirmation bias, especially for datasets with more classes and a high proportion of instance-specific noise. Experimental results on simulated and real-world noisy datasets demonstrate that both CSR and CSR+ achieve outstanding performance compared to methods at the same level.

Title: Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving

• Authors: Jinlong Li, Baolu Li, Zhengzhong Tu, Xinyu Liu, Qing Guo, Felix Juefei-Xu, Runsheng Xu, Hongkai Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04804
• Pdf URL: https://arxiv.org/pdf/2404.04804
• Copy Paste: [[2404.04804]] Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving(https://arxiv.org/abs/2404.04804)
• Keywords: diffusion
• Abstract: Vision-centric perception systems for autonomous driving have gained considerable attention recently due to their cost-effectiveness and scalability, especially compared to LiDAR-based systems. However, these systems often struggle in low-light conditions, potentially compromising their performance and safety. To address this, our paper introduces LightDiff, a domain-tailored framework designed to enhance the low-light image quality for autonomous driving applications. Specifically, we employ a multi-condition controlled diffusion model. LightDiff works without any human-collected paired data, leveraging a dynamic data degradation process instead. It incorporates a novel multi-condition adapter that adaptively controls the input weights from different modalities, including depth maps, RGB images, and text captions, to effectively illuminate dark scenes while maintaining context consistency. Furthermore, to align the enhanced images with the detection model's knowledge, LightDiff employs perception-specific scores as rewards to guide the diffusion training process through reinforcement learning. Extensive experiments on the nuScenes datasets demonstrate that LightDiff can significantly improve the performance of several state-of-the-art 3D detectors in night-time conditions while achieving high visual quality scores, highlighting its potential to safeguard autonomous driving.

Title: D2SL: Decouple Defogging and Semantic Learning for Foggy Domain-Adaptive Segmentation

• Authors: Xuan Sun, Zhanfu An, Yuyu Liu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.04807
• Pdf URL: https://arxiv.org/pdf/2404.04807
• Copy Paste: [[2404.04807]] D2SL: Decouple Defogging and Semantic Learning for Foggy Domain-Adaptive Segmentation(https://arxiv.org/abs/2404.04807)
• Keywords: segmentation
• Abstract: We investigated domain adaptive semantic segmentation in foggy weather scenarios, which aims to enhance the utilization of unlabeled foggy data and improve the model's adaptability to foggy conditions. Current methods rely on clear images as references, jointly learning defogging and segmentation for foggy images. Despite making some progress, there are still two main drawbacks: (1) the coupling of segmentation and defogging feature representations, resulting in a decrease in semantic representation capability, and (2) the failure to leverage real fog priors in unlabeled foggy data, leading to insufficient model generalization ability. To address these issues, we propose a novel training framework, Decouple Defogging and Semantic learning, called D2SL, aiming to alleviate the adverse impact of defogging tasks on the final segmentation task. In this framework, we introduce a domain-consistent transfer strategy to establish a connection between defogging and segmentation tasks. Furthermore, we design a real fog transfer strategy to improve defogging effects by fully leveraging the fog priors from real foggy images. Our approach enhances the semantic representations required for segmentation during the defogging learning process and maximizes the representation capability of fog invariance by effectively utilizing real fog data. Comprehensive experiments validate the effectiveness of the proposed method.

Title: Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language

• Authors: Raphaël Merx, Aso Mahmudi, Katrina Langford, Leo Alberto de Araujo, Ekaterina Vylomova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04809
• Pdf URL: https://arxiv.org/pdf/2404.04809
• Copy Paste: [[2404.04809]] Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language(https://arxiv.org/abs/2404.04809)
• Keywords: large language model
• Abstract: This study explores the use of large language models (LLMs) for translating English into Mambai, a low-resource Austronesian language spoken in Timor-Leste, with approximately 200,000 native speakers. Leveraging a novel corpus derived from a Mambai language manual and additional sentences translated by a native speaker, we examine the efficacy of few-shot LLM prompting for machine translation (MT) in this low-resource context. Our methodology involves the strategic selection of parallel sentences and dictionary entries for prompting, aiming to enhance translation accuracy, using open-source and proprietary LLMs (LlaMa 2 70b, Mixtral 8x7B, GPT-4). We find that including dictionary entries in prompts and a mix of sentences retrieved through TF-IDF and semantic embeddings significantly improves translation quality. However, our findings reveal stark disparities in translation performance across test sets, with BLEU scores reaching as high as 21.2 on materials from the language manual, in contrast to a maximum of 4.4 on a test set provided by a native speaker. These results underscore the importance of diverse and representative corpora in assessing MT for low-resource languages. Our research provides insights into few-shot LLM prompting for low-resource MT, and makes available an initial corpus for the Mambai language.

Title: Inference-Time Rule Eraser: Distilling and Removing Bias Rules to Mitigate Bias in Deployed Models

• Authors: Yi Zhang, Jitao Sang
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.04814
• Pdf URL: https://arxiv.org/pdf/2404.04814
• Copy Paste: [[2404.04814]] Inference-Time Rule Eraser: Distilling and Removing Bias Rules to Mitigate Bias in Deployed Models(https://arxiv.org/abs/2404.04814)
• Keywords: fair
• Abstract: Fairness is critical for artificial intelligence systems, especially for those deployed in high-stakes applications such as hiring and justice. Existing efforts toward fairness in machine learning fairness require retraining or fine-tuning the neural network weights to meet the fairness criteria. However, this is often not feasible in practice for regular model users due to the inability to access and modify model weights. In this paper, we propose a more flexible fairness paradigm, Inference-Time Rule Eraser, or simply Eraser, which considers the case where model weights can not be accessed and tackles fairness issues from the perspective of biased rules removal at inference-time. We first verified the feasibility of modifying the model output to wipe the biased rule through Bayesian analysis, and deduced Inference-Time Rule Eraser via subtracting the logarithmic value associated with unfair rules (i.e., the model's response to biased features) from the model's logits output as a means of removing biased rules. Moreover, we present a specific implementation of Rule Eraser that involves two stages: (1) limited queries are performed on the model with inaccessible weights to distill its biased rules into an additional patched model, and (2) during inference time, the biased rules already distilled into the patched model are excluded from the output of the original model, guided by the removal strategy outlined in Rule Eraser. Exhaustive experimental evaluation demonstrates the effectiveness and superior performance of the proposed Rule Eraser in addressing fairness concerns.

Title: FRACTAL: Fine-Grained Scoring from Aggregate Text Labels

• Authors: Yukti Makhija, Priyanka Agrawal, Rishi Saket, Aravindan Raghuveer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04817
• Pdf URL: https://arxiv.org/pdf/2404.04817
• Copy Paste: [[2404.04817]] FRACTAL: Fine-Grained Scoring from Aggregate Text Labels(https://arxiv.org/abs/2404.04817)
• Keywords: large language model
• Abstract: Large language models (LLMs) are being increasingly tuned to power complex generation tasks such as writing, fact-seeking, querying and reasoning. Traditionally, human or model feedback for evaluating and further tuning LLM performance has been provided at the response level, enabling faster and more cost-effective assessments. However, recent works (Amplayo et al. [2022], Wu et al. [2023]) indicate that sentence-level labels may provide more accurate and interpretable feedback for LLM optimization. In this work, we introduce methods to disaggregate response-level labels into sentence-level (pseudo-)labels. Our approach leverages multiple instance learning (MIL) and learning from label proportions (LLP) techniques in conjunction with prior information (e.g., document-sentence cosine similarity) to train a specialized model for sentence-level scoring. We also employ techniques which use model predictions to pseudo-label the train-set at the sentence-level for model training to further improve performance. We conduct extensive evaluations of our methods across six datasets and four tasks: retrieval, question answering, summarization, and math reasoning. Our results demonstrate improved performance compared to multiple baselines across most of these tasks. Our work is the first to develop response-level feedback to sentence-level scoring techniques, leveraging sentence-level prior information, along with comprehensive evaluations on multiple tasks as well as end-to-end finetuning evaluation showing performance comparable to a model trained on fine-grained human annotated labels.

Title: Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer

• Authors: Hyeongjin Nam, Daniel Sungho Jung, Gyeongsik Moon, Kyoung Mu Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04819
• Pdf URL: https://arxiv.org/pdf/2404.04819
• Copy Paste: [[2404.04819]] Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer(https://arxiv.org/abs/2404.04819)
• Keywords: transformer
• Abstract: Human-object contact serves as a strong cue to understand how humans physically interact with objects. Nevertheless, it is not widely explored to utilize human-object contact information for the joint reconstruction of 3D human and object from a single image. In this work, we present a novel joint 3D human-object reconstruction method (CONTHO) that effectively exploits contact information between humans and objects. There are two core designs in our system: 1) 3D-guided contact estimation and 2) contact-based 3D human and object refinement. First, for accurate human-object contact estimation, CONTHO initially reconstructs 3D humans and objects and utilizes them as explicit 3D guidance for contact estimation. Second, to refine the initial reconstructions of 3D human and object, we propose a novel contact-based refinement Transformer that effectively aggregates human features and object features based on the estimated human-object contact. The proposed contact-based refinement prevents the learning of erroneous correlation between human and object, which enables accurate 3D reconstruction. As a result, our CONTHO achieves state-of-the-art performance in both human-object contact estimation and joint reconstruction of 3D human and object. The code is publicly available at https://github.com/dqj5182/CONTHO_RELEASE.

Title: 3D Building Reconstruction from Monocular Remote Sensing Images with Multi-level Supervisions

• Authors: Weijia Li, Haote Yang, Zhenghao Hu, Juepeng Zheng, Gui-Song Xia, Conghui He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04823
• Pdf URL: https://arxiv.org/pdf/2404.04823
• Copy Paste: [[2404.04823]] 3D Building Reconstruction from Monocular Remote Sensing Images with Multi-level Supervisions(https://arxiv.org/abs/2404.04823)
• Keywords: extraction
• Abstract: 3D building reconstruction from monocular remote sensing images is an important and challenging research problem that has received increasing attention in recent years, owing to its low cost of data acquisition and availability for large-scale applications. However, existing methods rely on expensive 3D-annotated samples for fully-supervised training, restricting their application to large-scale cross-city scenarios. In this work, we propose MLS-BRN, a multi-level supervised building reconstruction network that can flexibly utilize training samples with different annotation levels to achieve better reconstruction results in an end-to-end manner. To alleviate the demand on full 3D supervision, we design two new modules, Pseudo Building Bbox Calculator and Roof-Offset guided Footprint Extractor, as well as new tasks and training strategies for different types of samples. Experimental results on several public and new datasets demonstrate that our proposed MLS-BRN achieves competitive performance using much fewer 3D-annotated samples, and significantly improves the footprint extraction and 3D reconstruction performance compared with current state-of-the-art. The code and datasets of this work will be released at https://github.com/opendatalab/MLS-BRN.git.

Title: Strictly-ID-Preserved and Controllable Accessory Advertising Image Generation

• Authors: Youze Xue, Binghui Chen, Yifeng Geng, Xuansong Xie, Jiansheng Chen, Hongbing Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04828
• Pdf URL: https://arxiv.org/pdf/2404.04828
• Copy Paste: [[2404.04828]] Strictly-ID-Preserved and Controllable Accessory Advertising Image Generation(https://arxiv.org/abs/2404.04828)
• Keywords: generative
• Abstract: Customized generative text-to-image models have the ability to produce images that closely resemble a given subject. However, in the context of generating advertising images for e-commerce scenarios, it is crucial that the generated subject's identity aligns perfectly with the product being advertised. In order to address the need for strictly-ID preserved advertising image generation, we have developed a Control-Net based customized image generation pipeline and have taken earring model advertising as an example. Our approach facilitates a seamless interaction between the earrings and the model's face, while ensuring that the identity of the earrings remains intact. Furthermore, to achieve a diverse and controllable display, we have proposed a multi-branch cross-attention architecture, which allows for control over the scale, pose, and appearance of the model, going beyond the limitations of text prompts. Our method manages to achieve fine-grained control of the generated model's face, resulting in controllable and captivating advertising effects.

Title: ShoeModel: Learning to Wear on the User-specified Shoes via Diffusion Model

• Authors: Binghui Chen, Wenyu Li, Yifeng Geng, Xuansong Xie, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04833
• Pdf URL: https://arxiv.org/pdf/2404.04833
• Copy Paste: [[2404.04833]] ShoeModel: Learning to Wear on the User-specified Shoes via Diffusion Model(https://arxiv.org/abs/2404.04833)
• Keywords: diffusion
• Abstract: With the development of the large-scale diffusion model, Artificial Intelligence Generated Content (AIGC) techniques are popular recently. However, how to truly make it serve our daily lives remains an open question. To this end, in this paper, we focus on employing AIGC techniques in one filed of E-commerce marketing, i.e., generating hyper-realistic advertising images for displaying user-specified shoes by human. Specifically, we propose a shoe-wearing system, called Shoe-Model, to generate plausible images of human legs interacting with the given shoes. It consists of three modules: (1) shoe wearable-area detection module (WD), (2) leg-pose synthesis module (LpS) and the final (3) shoe-wearing image generation module (SW). Them three are performed in ordered stages. Compared to baselines, our ShoeModel is shown to generalize better to different type of shoes and has ability of keeping the ID-consistency of the given shoes, as well as automatically producing reasonable interactions with human. Extensive experiments show the effectiveness of our proposed shoe-wearing system. Figure 1 shows the input and output examples of our ShoeModel.

Title: Data Bias According to Bipol: Men are Naturally Right and It is the Role of Women to Follow Their Lead

• Authors: Irene Pagliai, Goya van Boven, Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Isabella Södergren, Elisa Barney
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04838
• Pdf URL: https://arxiv.org/pdf/2404.04838
• Copy Paste: [[2404.04838]] Data Bias According to Bipol: Men are Naturally Right and It is the Role of Women to Follow Their Lead(https://arxiv.org/abs/2404.04838)
• Keywords: secure, explainability, large language model
• Abstract: We introduce new large labeled datasets on bias in 3 languages and show in experiments that bias exists in all 10 datasets of 5 languages evaluated, including benchmark datasets on the English GLUE/SuperGLUE leaderboards. The 3 new languages give a total of almost 6 million labeled samples and we benchmark on these datasets using SotA multilingual pretrained models: mT5 and mBERT. The challenge of social bias, based on prejudice, is ubiquitous, as recent events with AI and large language models (LLMs) have shown. Motivated by this challenge, we set out to estimate bias in multiple datasets. We compare some recent bias metrics and use bipol, which has explainability in the metric. We also confirm the unverified assumption that bias exists in toxic comments by randomly sampling 200 samples from a toxic dataset population using the confidence level of 95% and error margin of 7%. Thirty gold samples were randomly distributed in the 200 samples to secure the quality of the annotation. Our findings confirm that many of the datasets have male bias (prejudice against women), besides other types of bias. We publicly release our new datasets, lexica, models, and codes.

Title: SLPL SHROOM at SemEval\-2024 Task 06: A comprehensive study on models ability to detect hallucination

• Authors: Pouya Fallah, Soroush Gooran, Mohammad Jafarinasab, Pouya Sadeghi, Reza Farnia, Amirreza Tarabkhah, Zainab Sadat Taghavi, Hossein Sameti
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04845
• Pdf URL: https://arxiv.org/pdf/2404.04845
• Copy Paste: [[2404.04845]] SLPL SHROOM at SemEval\-2024 Task 06: A comprehensive study on models ability to detect hallucination(https://arxiv.org/abs/2404.04845)
• Keywords: generative
• Abstract: Language models, particularly generative models, are susceptible to hallucinations, generating outputs that contradict factual knowledge or the source text. This study explores methods for detecting hallucinations in three SemEval-2024 Task 6 tasks: Machine Translation, Definition Modeling, and Paraphrase Generation. We evaluate two methods: semantic similarity between the generated text and factual references, and an ensemble of language models that judge each other's outputs. Our results show that semantic similarity achieves moderate accuracy and correlation scores in trial data, while the ensemble method offers insights into the complexities of hallucination detection but falls short of expectations. This work highlights the challenges of hallucination detection and underscores the need for further research in this critical area.

Title: F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation

• Authors: Junhong Wu, Yuchen Liu, Chengqing Zong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04846
• Pdf URL: https://arxiv.org/pdf/2404.04846
• Copy Paste: [[2404.04846]] F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation(https://arxiv.org/abs/2404.04846)
• Keywords: robust
• Abstract: In the evolving landscape of Neural Machine Translation (NMT), the pretrain-then-finetune paradigm has yielded impressive results. However, the persistent challenge of Catastrophic Forgetting (CF) remains a hurdle. While previous work has introduced Continual Learning (CL) methods to address CF, these approaches grapple with the delicate balance between avoiding forgetting and maintaining system extensibility. To address this, we propose a CL method, named $\textbf{F-MALLOC}$ ($\textbf{F}$eed-forward $\textbf{M}$emory $\textbf{ALLOC}ation)$. F-MALLOC is inspired by recent insights highlighting that feed-forward layers emulate neural memories and encapsulate crucial translation knowledge. It decomposes feed-forward layers into discrete memory cells and allocates these memories to different tasks. By learning to allocate and safeguard these memories, our method effectively alleviates CF while ensuring robust extendability. Besides, we propose a comprehensive assessment protocol for multi-stage CL of NMT systems. Experiments conducted following this new protocol showcase the superior performance of F-MALLOC, evidenced by higher BLEU scores and almost zero forgetting.

Title: Hidden You Malicious Goal Into Benigh Narratives: Jailbreak Large Language Models through Logic Chain Injection

• Authors: Zhilong Wang, Yebo Cao, Peng Liu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04849
• Pdf URL: https://arxiv.org/pdf/2404.04849
• Copy Paste: [[2404.04849]] Hidden You Malicious Goal Into Benigh Narratives: Jailbreak Large Language Models through Logic Chain Injection(https://arxiv.org/abs/2404.04849)
• Keywords: security, attack
• Abstract: Jailbreak attacks on Language Model Models (LLMs) entail crafting prompts aimed at exploiting the models to generate malicious content. Existing jailbreak attacks can successfully deceive the LLMs, however they cannot deceive the human. This paper proposes a new type of jailbreak attacks which can deceive both the LLMs and human (i.e., security analyst). The key insight of our idea is borrowed from the social psychology - that is human are easily deceived if the lie is hidden in truth. Based on this insight, we proposed the logic-chain injection attacks to inject malicious intention into benign truth. Logic-chain injection attack firstly dissembles its malicious target into a chain of benign narrations, and then distribute narrations into a related benign article, with undoubted facts. In this way, newly generate prompt cannot only deceive the LLMs, but also deceive human.

Title: Lucky 52: How Many Languages Are Needed to Instruction Fine-Tune Large Language Models?

• Authors: Shaoxiong Ji, Pinzhen Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04850
• Pdf URL: https://arxiv.org/pdf/2404.04850
• Copy Paste: [[2404.04850]] Lucky 52: How Many Languages Are Needed to Instruction Fine-Tune Large Language Models?(https://arxiv.org/abs/2404.04850)
• Keywords: large language model
• Abstract: Fine-tuning large language models for multilingual downstream tasks requires a diverse set of languages to capture the nuances and structures of different linguistic contexts effectively. While the specific number varies depending on the desired scope and target languages, we argue that the number of languages, language exposure, and similarity that incorporate the selection of languages for fine-tuning are some important aspects to examine. By fine-tuning large multilingual models on 1 to 52 languages, this paper answers one question: How many languages are needed in instruction fine-tuning for multilingual tasks? We investigate how multilingual instruction fine-tuned models behave on multilingual benchmarks with an increasing number of languages and discuss our findings from the perspective of language exposure and similarity.

Title: Contextual Chart Generation for Cyber Deception

• Authors: David D. Nguyen, David Liebowitz, Surya Nepal, Salil S. Kanhere, Sharif Abuadbba
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04854
• Pdf URL: https://arxiv.org/pdf/2404.04854
• Copy Paste: [[2404.04854]] Contextual Chart Generation for Cyber Deception(https://arxiv.org/abs/2404.04854)
• Keywords: security, transformer, generative, large language model
• Abstract: Honeyfiles are security assets designed to attract and detect intruders on compromised systems. Honeyfiles are a type of honeypot that mimic real, sensitive documents, creating the illusion of the presence of valuable data. Interaction with a honeyfile reveals the presence of an intruder, and can provide insights into their goals and intentions. Their practical use, however, is limited by the time, cost and effort associated with manually creating realistic content. The introduction of large language models has made high-quality text generation accessible, but honeyfiles contain a variety of content including charts, tables and images. This content needs to be plausible and realistic, as well as semantically consistent both within honeyfiles and with the real documents they mimic, to successfully deceive an intruder. In this paper, we focus on an important component of the honeyfile content generation problem: document charts. Charts are ubiquitous in corporate documents and are commonly used to communicate quantitative and scientific data. Existing image generation models, such as DALL-E, are rather prone to generating charts with incomprehensible text and unconvincing data. We take a multi-modal approach to this problem by combining two purpose-built generative models: a multitask Transformer and a specialized multi-head autoencoder. The Transformer generates realistic captions and plot text, while the autoencoder generates the underlying tabular data for the plot. To advance the field of automated honeyplot generation, we also release a new document-chart dataset and propose a novel metric Keyword Semantic Matching (KSM). This metric measures the semantic consistency between keywords of a corpus and a smaller bag of words. Extensive experiments demonstrate excellent performance against multiple large language models, including ChatGPT and GPT4.

Title: Msmsfnet: a multi-stream and multi-scale fusion net for edge detection

• Authors: Chenguang Liu, Chisheng Wang, Feifei Dong, Xin Su, Chuanhua Zhu, Dejin Zhang, Qingquan Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04856
• Pdf URL: https://arxiv.org/pdf/2404.04856
• Copy Paste: [[2404.04856]] Msmsfnet: a multi-stream and multi-scale fusion net for edge detection(https://arxiv.org/abs/2404.04856)
• Keywords: fair
• Abstract: Edge detection is a long standing problem in computer vision. Recent deep learning based algorithms achieve state of-the-art performance in publicly available datasets. Despite the efficiency of these algorithms, their performance, however, relies heavily on the pretrained weights of the backbone network on the ImageNet dataset. This limits heavily the design space of deep learning based edge detectors. Whenever we want to devise a new model, we have to train this new model on the ImageNet dataset first, and then fine tune the model using the edge detection datasets. The comparison would be unfair otherwise. However, it is usually not feasible for many researchers to train a model on the ImageNet dataset due to the limited computation resources. In this work, we study the performance that can be achieved by state-of-the-art deep learning based edge detectors in publicly available datasets when they are trained from scratch, and devise a new network architecture, the multi-stream and multi scale fusion net (msmsfnet), for edge detection. We show in our experiments that by training all models from scratch to ensure the fairness of comparison, out model outperforms state-of-the art deep learning based edge detectors in three publicly available datasets.

Title: ByteEdit: Boost, Comply and Accelerate Generative Image Editing

• Authors: Yuxi Ren, Jie Wu, Yanzuo Lu, Huafeng Kuang, Xin Xia, Xionghui Wang, Qianqian Wang, Yixing Zhu, Pan Xie, Shiyin Wang, Xuefeng Xiao, Yitong Wang, Min Zheng, Lean Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04860
• Pdf URL: https://arxiv.org/pdf/2404.04860
• Copy Paste: [[2404.04860]] ByteEdit: Boost, Comply and Accelerate Generative Image Editing(https://arxiv.org/abs/2404.04860)
• Keywords: diffusion, generative
• Abstract: Recent advancements in diffusion-based generative image editing have sparked a profound revolution, reshaping the landscape of image outpainting and inpainting tasks. Despite these strides, the field grapples with inherent challenges, including: i) inferior quality; ii) poor consistency; iii) insufficient instrcution adherence; iv) suboptimal generation efficiency. To address these obstacles, we present ByteEdit, an innovative feedback learning framework meticulously designed to Boost, Comply, and Accelerate Generative Image Editing tasks. ByteEdit seamlessly integrates image reward models dedicated to enhancing aesthetics and image-text alignment, while also introducing a dense, pixel-level reward model tailored to foster coherence in the output. Furthermore, we propose a pioneering adversarial and progressive feedback learning strategy to expedite the model's inference speed. Through extensive large-scale user evaluations, we demonstrate that ByteEdit surpasses leading generative image editing products, including Adobe, Canva, and MeiTu, in both generation quality and consistency. ByteEdit-Outpainting exhibits a remarkable enhancement of 388% and 135% in quality and consistency, respectively, when compared to the baseline model. Experiments also verfied that our acceleration models maintains excellent performance results in terms of quality and consistency.

Title: Privacy-Preserving Traceable Functional Encryption for Inner Product

• Authors: Muyao Qiu, Jinguang Han
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04861
• Pdf URL: https://arxiv.org/pdf/2404.04861
• Copy Paste: [[2404.04861]] Privacy-Preserving Traceable Functional Encryption for Inner Product(https://arxiv.org/abs/2404.04861)
• Keywords: secure, security, privacy, protect
• Abstract: Functional encryption introduces a new paradigm of public key encryption that decryption only reveals the function value of encrypted data. To curb key leakage issues and trace users in FE-IP, a new primitive called traceable functional encryption for inner product (TFE-IP) has been proposed. However, the privacy protection of user's identities has not been considered in the existing TFE-IP schemes. In order to balance privacy and accountability, we propose the concept of privacy-preserving traceable functional encryption for inner product (PPTFE-IP) and give a concrete construction. Our scheme provides the following features: (1) To prevent key sharing, a user's key is bound with both his/her identity and a vector; (2) The key generation center (KGC) and a user execute a two-party secure computing protocol to generate a key without the former knowing anything about the latter's identity; (3) Each user can verify the correctness of his/her key; (4) A user can calculate the inner product of the two vectors embedded in his/her key and in a ciphertext; (5) Only the tracer can trace the identity embedded in a key. The security of our scheme is formally reduced to well-known complexity assumptions, and the implementation is conducted to evaluate its efficiency. The novelty of our scheme is to protect users' privacy and provide traceability if required.

Title: Signal-noise separation using unsupervised reservoir computing

• Authors: Jaesung Choi, Pilwon Kim
• Subjects: cs.LG, eess.SP, nlin.CD
• Abstract URL: https://arxiv.org/abs/2404.04870
• Pdf URL: https://arxiv.org/pdf/2404.04870
• Copy Paste: [[2404.04870]] Signal-noise separation using unsupervised reservoir computing(https://arxiv.org/abs/2404.04870)
• Keywords: robust
• Abstract: Removing noise from a signal without knowing the characteristics of the noise is a challenging task. This paper introduces a signal-noise separation method based on time series prediction. We use Reservoir Computing (RC) to extract the maximum portion of "predictable information" from a given signal. Reproducing the deterministic component of the signal using RC, we estimate the noise distribution from the difference between the original signal and reconstructed one. The method is based on a machine learning approach and requires no prior knowledge of either the deterministic signal or the noise distribution. It provides a way to identify additivity/multiplicativity of noise and to estimate the signal-to-noise ratio (SNR) indirectly. The method works successfully for combinations of various signal and noise, including chaotic signal and highly oscillating sinusoidal signal which are corrupted by non-Gaussian additive/ multiplicative noise. The separation performances are robust and notably outstanding for signals with strong noise, even for those with negative SNR.

Title: Data Stream Sampling with Fuzzy Task Boundaries and Noisy Labels

• Authors: Yu-Hsi Chen
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04871
• Pdf URL: https://arxiv.org/pdf/2404.04871
• Copy Paste: [[2404.04871]] Data Stream Sampling with Fuzzy Task Boundaries and Noisy Labels(https://arxiv.org/abs/2404.04871)
• Keywords: robust, fair
• Abstract: In the realm of continual learning, the presence of noisy labels within data streams represents a notable obstacle to model reliability and fairness. We focus on the data stream scenario outlined in pertinent literature, characterized by fuzzy task boundaries and noisy labels. To address this challenge, we introduce a novel and intuitive sampling method called Noisy Test Debiasing (NTD) to mitigate noisy labels in evolving data streams and establish a fair and robust continual learning algorithm. NTD is straightforward to implement, making it feasible across various scenarios. Our experiments benchmark four datasets, including two synthetic noise datasets (CIFAR10 and CIFAR100) and real-world noise datasets (mini-WebVision and Food-101N). The results validate the efficacy of NTD for online continual learning in scenarios with noisy labels in data streams. Compared to the previous leading approach, NTD achieves a training speedup enhancement over two times while maintaining or surpassing accuracy levels. Moreover, NTD utilizes less than one-fifth of the GPU memory resources compared to previous leading methods.

Title: HiLo: Detailed and Robust 3D Clothed Human Reconstruction with High-and Low-Frequency Information of Parametric Models

• Authors: Yifan Yang, Dong Liu, Shuhai Zhang, Zeshuai Deng, Zixiong Huang, Mingkui Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04876
• Pdf URL: https://arxiv.org/pdf/2404.04876
• Copy Paste: [[2404.04876]] HiLo: Detailed and Robust 3D Clothed Human Reconstruction with High-and Low-Frequency Information of Parametric Models(https://arxiv.org/abs/2404.04876)
• Keywords: robust
• Abstract: Reconstructing 3D clothed human involves creating a detailed geometry of individuals in clothing, with applications ranging from virtual try-on, movies, to games. To enable practical and widespread applications, recent advances propose to generate a clothed human from an RGB image. However, they struggle to reconstruct detailed and robust avatars simultaneously. We empirically find that the high-frequency (HF) and low-frequency (LF) information from a parametric model has the potential to enhance geometry details and improve robustness to noise, respectively. Based on this, we propose HiLo, namely clothed human reconstruction with high- and low-frequency information, which contains two components. 1) To recover detailed geometry using HF information, we propose a progressive HF Signed Distance Function to enhance the detailed 3D geometry of a clothed human. We analyze that our progressive learning manner alleviates large gradients that hinder model convergence. 2) To achieve robust reconstruction against inaccurate estimation of the parametric model by using LF information, we propose a spatial interaction implicit function. This function effectively exploits the complementary spatial information from a low-resolution voxel grid of the parametric model. Experimental results demonstrate that HiLo outperforms the state-of-the-art methods by 10.43% and 9.54% in terms of Chamfer distance on the Thuman2.0 and CAPE datasets, respectively. Additionally, HiLo demonstrates robustness to noise from the parametric model, challenging poses, and various clothing styles.

Title: GauU-Scene V2: Expanse Lidar Image Dataset Shows Unreliable Geometric Reconstruction Using Gaussian Splatting and NeRF

• Authors: Butian Xiong, Nanjun Zheng, Zhen Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04880
• Pdf URL: https://arxiv.org/pdf/2404.04880
• Copy Paste: [[2404.04880]] GauU-Scene V2: Expanse Lidar Image Dataset Shows Unreliable Geometric Reconstruction Using Gaussian Splatting and NeRF(https://arxiv.org/abs/2404.04880)
• Keywords: extraction
• Abstract: We introduce a novel large-scale scene reconstruction benchmark that utilizes newly developed 3D representation approaches: Gaussian Splatting and Neural Radiance Fields, on our expansive GauU-Scene V2 dataset. GauU-Scene V2 encompasses over 6.5 square kilometers and features a comprehensive RGB dataset coupled with LiDAR ground truth. This dataset offers a unique blend of urban and academic environments for advanced spatial analysis, covering more than 6.5 km2. We also provide detailed supplementary information on data collection protocols. Furthermore, we present an easy-to-follow pipeline to align the COLMAP sparse point cloud with the detailed LiDAR dataset. Our evaluation of U-Scene, which includes a detailed analysis across various novel viewpoints using image-based metrics such as SSIM, LPIPS, and PSNR, shows contradictory results when applying geometric-based metrics, such as Chamfer distance. This leads to doubts about the reliability of current image-based measurement matrices and geometric extraction methods on Gaussian Splatting. We also make the dataset available on the following anonymous project page

Title: Mixture of Low-rank Experts for Transferable AI-Generated Image Detection

• Authors: Zihan Liu, Hanyi Wang, Yaoyu Kang, Shilin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04883
• Pdf URL: https://arxiv.org/pdf/2404.04883
• Copy Paste: [[2404.04883]] Mixture of Low-rank Experts for Transferable AI-Generated Image Detection(https://arxiv.org/abs/2404.04883)
• Keywords: robust, diffusion, generative
• Abstract: Generative models have shown a giant leap in synthesizing photo-realistic images with minimal expertise, sparking concerns about the authenticity of online information. This study aims to develop a universal AI-generated image detector capable of identifying images from diverse sources. Existing methods struggle to generalize across unseen generative models when provided with limited sample sources. Inspired by the zero-shot transferability of pre-trained vision-language models, we seek to harness the nontrivial visual-world knowledge and descriptive proficiency of CLIP-ViT to generalize over unknown domains. This paper presents a novel parameter-efficient fine-tuning approach, mixture of low-rank experts, to fully exploit CLIP-ViT's potential while preserving knowledge and expanding capacity for transferable detection. We adapt only the MLP layers of deeper ViT blocks via an integration of shared and separate LoRAs within an MoE-based structure. Extensive experiments on public benchmarks show that our method achieves superiority over state-of-the-art approaches in cross-generator generalization and robustness to perturbations. Remarkably, our best-performing ViT-L/14 variant requires training only 0.08% of its parameters to surpass the leading baseline by +3.64% mAP and +12.72% avg.Acc across unseen diffusion and autoregressive models. This even outperforms the baseline with just 0.28% of the training data. Our code and pre-trained models will be available at https://github.com/zhliuworks/CLIPMoLE.

Title: LRNet: Change detection of high-resolution remote sensing imagery via strategy of localization-then-refinement

• Authors: Huan Zhong, Chen Wu, Ziqi Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04884
• Pdf URL: https://arxiv.org/pdf/2404.04884
• Copy Paste: [[2404.04884]] LRNet: Change detection of high-resolution remote sensing imagery via strategy of localization-then-refinement(https://arxiv.org/abs/2404.04884)
• Keywords: extraction
• Abstract: Change detection, as a research hotspot in the field of remote sensing, has witnessed continuous development and progress. However, the discrimination of boundary details remains a significant bottleneck due to the complexity of surrounding elements between change areas and backgrounds. Discriminating the boundaries of large change areas results in misalignment, while connecting boundaries occurs for small change targets. To address the above issues, a novel network based on the localization-then-refinement strategy is proposed in this paper, namely LRNet. LRNet consists of two stages: localization and refinement. In the localization stage, a three-branch encoder simultaneously extracts original image features and their differential features for interactive localization of the position of each change area. To minimize information loss during feature extraction, learnable optimal pooling (LOP) is proposed to replace the widely used max-pooling. Additionally, this process is trainable and contributes to the overall optimization of the network. To effectively interact features from different branches and accurately locate change areas of various sizes, change alignment attention (C2A) and hierarchical change alignment module (HCA) are proposed. In the refinement stage, the localization results from the localization stage are corrected by constraining the change areas and change edges through the edge-area alignment module (E2A). Subsequently, the decoder, combined with the difference features strengthened by C2A in the localization phase, refines change areas of different sizes, ultimately achieving accurate boundary discrimination of change areas. The proposed LRNet outperforms 13 other state-of-the-art methods in terms of comprehensive evaluation metrics and provides the most precise boundary discrimination results on the LEVIR-CD and WHU-CD datasets.

Title: TimeGPT in Load Forecasting: A Large Time Series Model Perspective

• Authors: Wenlong Liao, Fernando Porte-Agel, Jiannong Fang, Christian Rehtanz, Shouxiang Wang, Dechang Yang, Zhe Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04885
• Pdf URL: https://arxiv.org/pdf/2404.04885
• Copy Paste: [[2404.04885]] TimeGPT in Load Forecasting: A Large Time Series Model Perspective(https://arxiv.org/abs/2404.04885)
• Keywords: transformer, generative, large language model
• Abstract: Machine learning models have made significant progress in load forecasting, but their forecast accuracy is limited in cases where historical load data is scarce. Inspired by the outstanding performance of large language models (LLMs) in computer vision and natural language processing, this paper aims to discuss the potential of large time series models in load forecasting with scarce historical data. Specifically, the large time series model is constructed as a time series generative pre-trained transformer (TimeGPT), which is trained on massive and diverse time series datasets consisting of 100 billion data points (e.g., finance, transportation, banking, web traffic, weather, energy, healthcare, etc.). Then, the scarce historical load data is used to fine-tune the TimeGPT, which helps it to adapt to the data distribution and characteristics associated with load forecasting. Simulation results show that TimeGPT outperforms the benchmarks (e.g., popular machine learning models and statistical models) for load forecasting on several real datasets with scarce training samples, particularly for short look-ahead times. However, it cannot be guaranteed that TimeGPT is always superior to benchmarks for load forecasting with scarce data, since the performance of TimeGPT may be affected by the distribution differences between the load data and the training data. In practical applications, we can divide the historical data into a training set and a validation set, and then use the validation set loss to decide whether TimeGPT is the best choice for a specific dataset.

Title: PagPassGPT: Pattern Guided Password Guessing via Generative Pretrained Transformer

• Authors: Xingyu Su, Xiaojie Zhu, Yang Li, Yong Li, Chi Chen, Paulo Esteves-Veríssimo
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04886
• Pdf URL: https://arxiv.org/pdf/2404.04886
• Copy Paste: [[2404.04886]] PagPassGPT: Pattern Guided Password Guessing via Generative Pretrained Transformer(https://arxiv.org/abs/2404.04886)
• Keywords: transformer, generative
• Abstract: Amidst the surge in deep learning-based password guessing models, challenges of generating high-quality passwords and reducing duplicate passwords persist. To address these challenges, we present PagPassGPT, a password guessing model constructed on Generative Pretrained Transformer (GPT). It can perform pattern guided guessing by incorporating pattern structure information as background knowledge, resulting in a significant increase in the hit rate. Furthermore, we propose D&C-GEN to reduce the repeat rate of generated passwords, which adopts the concept of a divide-and-conquer approach. The primary task of guessing passwords is recursively divided into non-overlapping subtasks. Each subtask inherits the knowledge from the parent task and predicts succeeding tokens. In comparison to the state-of-the-art model, our proposed scheme exhibits the capability to correctly guess 12% more passwords while producing 25% fewer duplicates.

Title: A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals

• Authors: Jiangnan Tang, Jingya Wang, Kaiyang Ji, Lan Xu, Jingyi Yu, Ye Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04890
• Pdf URL: https://arxiv.org/pdf/2404.04890
• Copy Paste: [[2404.04890]] A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals(https://arxiv.org/abs/2404.04890)
• Keywords: diffusion
• Abstract: Estimating full-body human motion via sparse tracking signals from head-mounted displays and hand controllers in 3D scenes is crucial to applications in AR/VR. One of the biggest challenges to this task is the one-to-many mapping from sparse observations to dense full-body motions, which endowed inherent ambiguities. To help resolve this ambiguous problem, we introduce a new framework to combine rich contextual information provided by scenes to benefit full-body motion tracking from sparse observations. To estimate plausible human motions given sparse tracking signals and 3D scenes, we develop $\text{S}^2$Fusion, a unified framework fusing \underline{S}cene and sparse \underline{S}ignals with a conditional dif\underline{Fusion} model. $\text{S}^2$Fusion first extracts the spatial-temporal relations residing in the sparse signals via a periodic autoencoder, and then produces time-alignment feature embedding as additional inputs. Subsequently, by drawing initial noisy motion from a pre-trained prior, $\text{S}^2$Fusion utilizes conditional diffusion to fuse scene geometry and sparse tracking signals to generate full-body scene-aware motions. The sampling procedure of $\text{S}^2$Fusion is further guided by a specially designed scene-penetration loss and phase-matching loss, which effectively regularizes the motion of the lower body even in the absence of any tracking signals, making the generated motion much more plausible and coherent. Extensive experimental results have demonstrated that our $\text{S}^2$Fusion outperforms the state-of-the-art in terms of estimation quality and smoothness.

Title: DL-EWF: Deep Learning Empowering Women's Fashion with Grounded-Segment-Anything Segmentation for Body Shape Classification

• Authors: Fatemeh Asghari, Mohammad Reza Soheili, Faezeh Gholamrezaie
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04891
• Pdf URL: https://arxiv.org/pdf/2404.04891
• Copy Paste: [[2404.04891]] DL-EWF: Deep Learning Empowering Women's Fashion with Grounded-Segment-Anything Segmentation for Body Shape Classification(https://arxiv.org/abs/2404.04891)
• Keywords: segmentation
• Abstract: The global fashion industry plays a pivotal role in the global economy, and addressing fundamental issues within the industry is crucial for developing innovative solutions. One of the most pressing challenges in the fashion industry is the mismatch between body shapes and the garments of individuals they purchase. This issue is particularly prevalent among individuals with non-ideal body shapes, exacerbating the challenges faced. Considering inter-individual variability in body shapes is essential for designing and producing garments that are widely accepted by consumers. Traditional methods for determining human body shape are limited due to their low accuracy, high costs, and time-consuming nature. New approaches, utilizing digital imaging and deep neural networks (DNN), have been introduced to identify human body shape. In this study, the Style4BodyShape dataset is used for classifying body shapes into five categories: Rectangle, Triangle, Inverted Triangle, Hourglass, and Apple. In this paper, the body shape segmentation of a person is extracted from the image, disregarding the surroundings and background. Then, Various pre-trained models, such as ResNet18, ResNet34, ResNet50, VGG16, VGG19, and Inception v3, are used to classify the segmentation results. Among these pre-trained models, the Inception V3 model demonstrates superior performance regarding f1-score evaluation metric and accuracy compared to the other models.

Title: Radial Networks: Dynamic Layer Routing for High-Performance Large Language Models

• Authors: Jordan Dotzel, Yash Akhauri, Ahmed S. AbouElhamayed, Carly Jiang, Mohamed Abdelfattah, Zhiru Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04900
• Pdf URL: https://arxiv.org/pdf/2404.04900
• Copy Paste: [[2404.04900]] Radial Networks: Dynamic Layer Routing for High-Performance Large Language Models(https://arxiv.org/abs/2404.04900)
• Keywords: large language model
• Abstract: Large language models (LLMs) often struggle with strict memory, latency, and power demands. To meet these demands, various forms of dynamic sparsity have been proposed that reduce compute on an input-by-input basis. These methods improve over static methods by exploiting the variance across individual inputs, which has steadily grown with the exponential increase in training data. Yet, the increasing depth within modern models, currently with hundreds of layers, has opened opportunities for dynamic layer sparsity, which skips the computation for entire layers. In this work, we explore the practicality of layer sparsity by profiling residual connections and establish the relationship between model depth and layer sparsity. For example, the residual blocks in the OPT-66B model have a median contribution of 5% to its output. We then take advantage of this dynamic sparsity and propose Radial Networks, which perform token-level routing between layers guided by a trained router module. These networks can be used in a post-training distillation from sequential networks or trained from scratch to co-learn the router and layer weights. They enable scaling to larger model sizes by decoupling the number of layers from the dynamic depth of the network, and their design allows for layer reuse. By varying the compute token by token, they reduce the overall resources needed for generating entire sequences. Overall, this leads to larger capacity networks with significantly lower compute and serving costs for large language models.

Title: Regularized Conditional Diffusion Model for Multi-Task Preference Alignment

• Authors: Xudong Yu, Chenjia Bai, Haoran He, Changhong Wang, Xuelong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04920
• Pdf URL: https://arxiv.org/pdf/2404.04920
• Copy Paste: [[2404.04920]] Regularized Conditional Diffusion Model for Multi-Task Preference Alignment(https://arxiv.org/abs/2404.04920)
• Keywords: diffusion
• Abstract: Sequential decision-making is desired to align with human intents and exhibit versatility across various tasks. Previous methods formulate it as a conditional generation process, utilizing return-conditioned diffusion models to directly model trajectory distributions. Nevertheless, the return-conditioned paradigm relies on pre-defined reward functions, facing challenges when applied in multi-task settings characterized by varying reward functions (versatility) and showing limited controllability concerning human preferences (alignment). In this work, we adopt multi-task preferences as a unified condition for both single- and multi-task decision-making, and propose preference representations aligned with preference labels. The learned representations are used to guide the conditional generation process of diffusion models, and we introduce an auxiliary objective to maximize the mutual information between representations and corresponding generated trajectories, improving alignment between trajectories and preferences. Extensive experiments in D4RL and Meta-World demonstrate that our method presents favorable performance in single- and multi-task scenarios, and exhibits superior alignment with preferences.

Title: GvT: A Graph-based Vision Transformer with Talking-Heads Utilizing Sparsity, Trained from Scratch on Small Datasets

• Authors: Dongjing Shan, guiqiang chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04924
• Pdf URL: https://arxiv.org/pdf/2404.04924
• Copy Paste: [[2404.04924]] GvT: A Graph-based Vision Transformer with Talking-Heads Utilizing Sparsity, Trained from Scratch on Small Datasets(https://arxiv.org/abs/2404.04924)
• Keywords: transformer
• Abstract: Vision Transformers (ViTs) have achieved impressive results in large-scale image classification. However, when training from scratch on small datasets, there is still a significant performance gap between ViTs and Convolutional Neural Networks (CNNs), which is attributed to the lack of inductive bias. To address this issue, we propose a Graph-based Vision Transformer (GvT) that utilizes graph convolutional projection and graph-pooling. In each block, queries and keys are calculated through graph convolutional projection based on the spatial adjacency matrix, while dot-product attention is used in another graph convolution to generate values. When using more attention heads, the queries and keys become lower-dimensional, making their dot product an uninformative matching function. To overcome this low-rank bottleneck in attention heads, we employ talking-heads technology based on bilinear pooled features and sparse selection of attention tensors. This allows interaction among filtered attention scores and enables each attention mechanism to depend on all queries and keys. Additionally, we apply graph-pooling between two intermediate blocks to reduce the number of tokens and aggregate semantic information more effectively. Our experimental results show that GvT produces comparable or superior outcomes to deep convolutional networks and surpasses vision transformers without pre-training on large datasets. The code for our proposed model is publicly available on the website.

Title: Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers

• Authors: Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, Philip S. Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04925
• Pdf URL: https://arxiv.org/pdf/2404.04925
• Copy Paste: [[2404.04925]] Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers(https://arxiv.org/abs/2404.04925)
• Keywords: large language model
• Abstract: Multilingual Large Language Models are capable of using powerful Large Language Models to handle and respond to queries in multiple languages, which achieves remarkable success in multilingual natural language processing tasks. Despite these breakthroughs, there still remains a lack of a comprehensive survey to summarize existing approaches and recent developments in this field. To this end, in this paper, we present a thorough review and provide a unified perspective to summarize the recent progress as well as emerging trends in multilingual large language models (MLLMs) literature. The contributions of this paper can be summarized: (1) First survey: to our knowledge, we take the first step and present a thorough review in MLLMs research field according to multi-lingual alignment; (2) New taxonomy: we offer a new and unified perspective to summarize the current progress of MLLMs; (3) New frontiers: we highlight several emerging frontiers and discuss the corresponding challenges; (4) Abundant resources: we collect abundant open-source resources, including relevant papers, data corpora, and leaderboards. We hope our work can provide the community with quick access and spur breakthrough research in MLLMs.

Title: Anomaly Detection in Electrocardiograms: Advancing Clinical Diagnosis Through Self-Supervised Learning

• Authors: Aofan Jiang, Chaoqin Huang, Qing Cao, Yuchen Xu, Zi Zeng, Kang Chen, Ya Zhang, Yanfeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04935
• Pdf URL: https://arxiv.org/pdf/2404.04935
• Copy Paste: [[2404.04935]] Anomaly Detection in Electrocardiograms: Advancing Clinical Diagnosis Through Self-Supervised Learning(https://arxiv.org/abs/2404.04935)
• Keywords: robust
• Abstract: The electrocardiogram (ECG) is an essential tool for diagnosing heart disease, with computer-aided systems improving diagnostic accuracy and reducing healthcare costs. Despite advancements, existing systems often miss rare cardiac anomalies that could be precursors to serious, life-threatening issues or alterations in the cardiac macro/microstructure. We address this gap by focusing on self-supervised anomaly detection (AD), training exclusively on normal ECGs to recognize deviations indicating anomalies. We introduce a novel self-supervised learning framework for ECG AD, utilizing a vast dataset of normal ECGs to autonomously detect and localize cardiac anomalies. It proposes a novel masking and restoration technique alongside a multi-scale cross-attention module, enhancing the model's ability to integrate global and local signal features. The framework emphasizes accurate localization of anomalies within ECG signals, ensuring the method's clinical relevance and reliability. To reduce the impact of individual variability, the approach further incorporates crucial patient-specific information from ECG reports, such as age and gender, thus enabling accurate identification of a broad spectrum of cardiac anomalies, including rare ones. Utilizing an extensive dataset of 478,803 ECG graphic reports from real-world clinical practice, our method has demonstrated exceptional effectiveness in AD across all tested conditions, regardless of their frequency of occurrence, significantly outperforming existing models. It achieved superior performance metrics, including an AUROC of 91.2%, an F1 score of 83.7%, a sensitivity rate of 84.2%, a specificity of 83.0%, and a precision of 75.6% with a fixed recall rate of 90%. It has also demonstrated robust localization capabilities, with an AUROC of 76.5% and a Dice coefficient of 65.3% for anomaly localization.

Title: Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models

• Authors: Weiwei Cao, Jianpeng Zhang, Yingda Xia, Tony C. W. Mok, Zi Li, Xianghua Ye, Le Lu, Jian Zheng, Yuxing Tang, Ling Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04936
• Pdf URL: https://arxiv.org/pdf/2404.04936
• Copy Paste: [[2404.04936]] Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models(https://arxiv.org/abs/2404.04936)
• Keywords: robust
• Abstract: Radiologists highly desire fully automated versatile AI for medical imaging interpretation. However, the lack of extensively annotated large-scale multi-disease datasets has hindered the achievement of this goal. In this paper, we explore the feasibility of leveraging language as a naturally high-quality supervision for chest CT imaging. In light of the limited availability of image-report pairs, we bootstrap the understanding of 3D chest CT images by distilling chest-related diagnostic knowledge from an extensively pre-trained 2D X-ray expert model. Specifically, we propose a language-guided retrieval method to match each 3D CT image with its semantically closest 2D X-ray image, and perform pair-wise and semantic relation knowledge distillation. Subsequently, we use contrastive learning to align images and reports within the same patient while distinguishing them from the other patients. However, the challenge arises when patients have similar semantic diagnoses, such as healthy patients, potentially confusing if treated as negatives. We introduce a robust contrastive learning that identifies and corrects these false negatives. We train our model with over 12,000 pairs of chest CT images and radiology reports. Extensive experiments across multiple scenarios, including zero-shot learning, report generation, and fine-tuning processes, demonstrate the model's feasibility in interpreting chest CT images.

Title: Optimizing Information Propagation for Blockchain-empowered Mobile AIGC: A Graph Attention Network Approach

• Authors: Jiana Liao, Jinbo Wen, Jiawen Kang, Yang Zhang, Jianbo Du, Qihao Li, Weiting Zhang, Dong Yang
• Subjects: cs.CR, cs.GT
• Abstract URL: https://arxiv.org/abs/2404.04937
• Pdf URL: https://arxiv.org/pdf/2404.04937
• Copy Paste: [[2404.04937]] Optimizing Information Propagation for Blockchain-empowered Mobile AIGC: A Graph Attention Network Approach(https://arxiv.org/abs/2404.04937)
• Keywords: secure
• Abstract: Artificial Intelligence-Generated Content (AIGC) is a rapidly evolving field that utilizes advanced AI algorithms to generate content. Through integration with mobile edge networks, mobile AIGC networks have gained significant attention, which can provide real-time customized and personalized AIGC services and products. Since blockchains can facilitate decentralized and transparent data management, AIGC products can be securely managed by blockchain to avoid tampering and plagiarization. However, the evolution of blockchain-empowered mobile AIGC is still in its nascent phase, grappling with challenges such as improving information propagation efficiency to enable blockchain-empowered mobile AIGC. In this paper, we design a Graph Attention Network (GAT)-based information propagation optimization framework for blockchain-empowered mobile AIGC. We first innovatively apply age of information as a data-freshness metric to measure information propagation efficiency in public blockchains. Considering that GATs possess the excellent ability to process graph-structured data, we utilize the GAT to obtain the optimal information propagation trajectory. Numerical results demonstrate that the proposed scheme exhibits the most outstanding information propagation efficiency compared with traditional routing mechanisms.

Title: Fuzzy K-Means Clustering without Cluster Centroids

• Authors: Han Lu, Fangfang Li, Quanxue Gao, Cheng Deng, Chris Ding, Qianqian Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04940
• Pdf URL: https://arxiv.org/pdf/2404.04940
• Copy Paste: [[2404.04940]] Fuzzy K-Means Clustering without Cluster Centroids(https://arxiv.org/abs/2404.04940)
• Keywords: robust
• Abstract: Fuzzy K-Means clustering is a critical technique in unsupervised data analysis. However, the performance of popular Fuzzy K-Means algorithms is sensitive to the selection of initial cluster centroids and is also affected by noise when updating mean cluster centroids. To address these challenges, this paper proposes a novel Fuzzy K-Means clustering algorithm that entirely eliminates the reliance on cluster centroids, obtaining membership matrices solely through distance matrix computation. This innovation enhances flexibility in distance measurement between sample points, thus improving the algorithm's performance and robustness. The paper also establishes theoretical connections between the proposed model and popular Fuzzy K-Means clustering techniques. Experimental results on several real datasets demonstrate the effectiveness of the algorithm.

Title: Prompting Large Language Models for Zero-shot Essay Scoring via Multi-trait Specialization

• Authors: Sanwoo Lee, Yida Cai, Desong Meng, Ziyang Wang, Yunfang Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04941
• Pdf URL: https://arxiv.org/pdf/2404.04941
• Copy Paste: [[2404.04941]] Prompting Large Language Models for Zero-shot Essay Scoring via Multi-trait Specialization(https://arxiv.org/abs/2404.04941)
• Keywords: large language model
• Abstract: Advances in automated essay scoring (AES) have traditionally relied on labeled essays, requiring tremendous cost and expertise for their acquisition. Recently, large language models (LLMs) have achieved great success in various tasks, but their potential is less explored in AES. In this paper, we propose Multi Trait Specialization (MTS), a zero-shot prompting framework to elicit essay scoring capabilities in LLMs. Specifically, we leverage ChatGPT to decompose writing proficiency into distinct traits and generate scoring criteria for each trait. Then, an LLM is prompted to extract trait scores from several conversational rounds, each round scoring one of the traits based on the scoring criteria. Finally, we derive the overall score via trait averaging and min-max scaling. Experimental results on two benchmark datasets demonstrate that MTS consistently outperforms straightforward prompting (Vanilla) in average QWK across all LLMs and datasets, with maximum gains of 0.437 on TOEFL11 and 0.355 on ASAP. Additionally, with the help of MTS, the small-sized Llama2-13b-chat substantially outperforms ChatGPT, facilitating an effective deployment in real applications.

Title: AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment

• Authors: Yuanfeng Xu, Yuhao Chen, Zhongzhan Huang, Zijian He, Guangrun Wang, Philip Torr, Liang Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04946
• Pdf URL: https://arxiv.org/pdf/2404.04946
• Copy Paste: [[2404.04946]] AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment(https://arxiv.org/abs/2404.04946)
• Keywords: extraction, diffusion
• Abstract: Recent video editing advancements rely on accurate pose sequences to animate subjects. However, these efforts are not suitable for cross-species animation due to pose misalignment between species (for example, the poses of a cat differs greatly from that of a pig due to differences in body structure). In this paper, we present AnimateZoo, a zero-shot diffusion-based video generator to address this challenging cross-species animation issue, aiming to accurately produce animal animations while preserving the background. The key technique used in our AnimateZoo is subject alignment, which includes two steps. First, we improve appearance feature extraction by integrating a Laplacian detail booster and a prompt-tuning identity extractor. These components are specifically designed to capture essential appearance information, including identity and fine details. Second, we align shape features and address conflicts from differing subjects by introducing a scale-information remover. This ensures accurate cross-species animation. Moreover, we introduce two high-quality animal video datasets featuring a wide variety of species. Trained on these extensive datasets, our model is capable of generating videos characterized by accurate movements, consistent appearance, and high-fidelity frames, without the need for the pre-inference fine-tuning that prior arts required. Extensive experiments showcase the outstanding performance of our method in cross-species action following tasks, demonstrating exceptional shape adaptation capability. The project page is available at https://justinxu0.github.io/AnimateZoo/.

Title: Iniva: Inclusive and Incentive-compatible Vote Aggregation

• Authors: Arian Baloochestani, Hanish Gogada, Leander Jehl, Hein Meling
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.04948
• Pdf URL: https://arxiv.org/pdf/2404.04948
• Copy Paste: [[2404.04948]] Iniva: Inclusive and Incentive-compatible Vote Aggregation(https://arxiv.org/abs/2404.04948)
• Keywords: security, attack, robust
• Abstract: Many blockchain platforms use committee-based consensus for scalability, finality, and security. In this consensus scheme, a committee decides which blocks get appended to the chain, typically through several voting phases. Platforms typically leverage the committee members' recorded votes to reward, punish, or detect failures. A common approach is to let the block proposer decide which votes to include, opening the door to possible attacks. For example, a malicious proposer can omit votes from targeted committee members, resulting in lost profits and, ultimately, their departure from the system. This paper presents Iniva, an inclusive and incentive-compatible vote aggregation scheme that prevents such vote omission attacks. Iniva relies on a tree overlay with carefully selected fallback paths, making it robust against process failures without needing reconfiguration or additional redundancy. Our analysis shows that Iniva significantly reduces the chance to omit individual votes while ensuring that omitting many votes incurs a significant cost. In addition, our experimental results show that Iniva enjoys robustness, scalability, and reasonable throughput.

Title: SilverSight: A Multi-Task Chinese Financial Large Language Model Based on Adaptive Semantic Space Learning

• Authors: Yuhang Zhou, Zeping Li, Siyu Tian, Yuchen Ni, Sen Liu, Guangnan Ye, Hongfeng Chai
• Subjects: cs.CL, cs.CE
• Abstract URL: https://arxiv.org/abs/2404.04949
• Pdf URL: https://arxiv.org/pdf/2404.04949
• Copy Paste: [[2404.04949]] SilverSight: A Multi-Task Chinese Financial Large Language Model Based on Adaptive Semantic Space Learning(https://arxiv.org/abs/2404.04949)
• Keywords: large language model
• Abstract: Large language models (LLMs) are increasingly being applied across various specialized fields, leveraging their extensive knowledge to empower a multitude of scenarios within these domains. However, each field encompasses a variety of specific tasks that require learning, and the diverse, heterogeneous data across these domains can lead to conflicts during model task transfer. In response to this challenge, our study introduces an Adaptive Semantic Space Learning (ASSL) framework, which utilizes the adaptive reorganization of data distributions within the semantic space to enhance the performance and selection efficacy of multi-expert models. Utilizing this framework, we trained a financial multi-task LLM named "SilverSight". Our research findings demonstrate that our framework can achieve results close to those obtained with full data training using only 10% of the data, while also exhibiting strong generalization capabilities.

Title: High-Discriminative Attribute Feature Learning for Generalized Zero-Shot Learning

• Authors: Yu Lei, Guoshuai Sheng, Fangfang Li, Quanxue Gao, Cheng Deng, Qin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04953
• Pdf URL: https://arxiv.org/pdf/2404.04953
• Copy Paste: [[2404.04953]] High-Discriminative Attribute Feature Learning for Generalized Zero-Shot Learning(https://arxiv.org/abs/2404.04953)
• Keywords: transformer
• Abstract: Zero-shot learning(ZSL) aims to recognize new classes without prior exposure to their samples, relying on semantic knowledge from observed classes. However, current attention-based models may overlook the transferability of visual features and the distinctiveness of attribute localization when learning regional features in images. Additionally, they often overlook shared attributes among different objects. Highly discriminative attribute features are crucial for identifying and distinguishing unseen classes. To address these issues, we propose an innovative approach called High-Discriminative Attribute Feature Learning for Generalized Zero-Shot Learning (HDAFL). HDAFL optimizes visual features by learning attribute features to obtain discriminative visual embeddings. Specifically, HDAFL utilizes multiple convolutional kernels to automatically learn discriminative regions highly correlated with attributes in images, eliminating irrelevant interference in image features. Furthermore, we introduce a Transformer-based attribute discrimination encoder to enhance the discriminative capability among attributes. Simultaneously, the method employs contrastive loss to alleviate dataset biases and enhance the transferability of visual features, facilitating better semantic transfer between seen and unseen classes. Experimental results demonstrate the effectiveness of HDAFL across three widely used datasets.

Title: Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models

• Authors: Zijin Yang, Kai Zeng, Kejiang Chen, Han Fang, Weiming Zhang, Nenghai Yu
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04956
• Pdf URL: https://arxiv.org/pdf/2404.04956
• Copy Paste: [[2404.04956]] Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models(https://arxiv.org/abs/2404.04956)
• Keywords: protect, robust, watermark, diffusion
• Abstract: Ethical concerns surrounding copyright protection and inappropriate content generation pose challenges for the practical implementation of diffusion models. One effective solution involves watermarking the generated images. However, existing methods often compromise the model performance or require additional training, which is undesirable for operators and users. To address this issue, we propose Gaussian Shading, a diffusion model watermarking technique that is both performance-lossless and training-free, while serving the dual purpose of copyright protection and tracing of offending content. Our watermark embedding is free of model parameter modifications and thus is plug-and-play. We map the watermark to latent representations following a standard Gaussian distribution, which is indistinguishable from latent representations obtained from the non-watermarked diffusion model. Therefore we can achieve watermark embedding with lossless performance, for which we also provide theoretical proof. Furthermore, since the watermark is intricately linked with image semantics, it exhibits resilience to lossy processing and erasure attempts. The watermark can be extracted by Denoising Diffusion Implicit Models (DDIM) inversion and inverse sampling. We evaluate Gaussian Shading on multiple versions of Stable Diffusion, and the results demonstrate that Gaussian Shading not only is performance-lossless but also outperforms existing methods in terms of robustness.

Title: A Two Dimensional Feature Engineering Method for Relation Extraction

• Authors: Hao Wang, Yanping Chen, Weizhe Yang, Yongbin Qin, Ruizhang Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04959
• Pdf URL: https://arxiv.org/pdf/2404.04959
• Copy Paste: [[2404.04959]] A Two Dimensional Feature Engineering Method for Relation Extraction(https://arxiv.org/abs/2404.04959)
• Keywords: extraction
• Abstract: Transforming a sentence into a two-dimensional (2D) representation (e.g., the table filling) has the ability to unfold a semantic plane, where an element of the plane is a word-pair representation of a sentence which may denote a possible relation representation composed of two named entities. The 2D representation is effective in resolving overlapped relation instances. However, in related works, the representation is directly transformed from a raw input. It is weak to utilize prior knowledge, which is important to support the relation extraction task. In this paper, we propose a two-dimensional feature engineering method in the 2D sentence representation for relation extraction. Our proposed method is evaluated on three public datasets (ACE05 Chinese, ACE05 English, and SanWen) and achieves the state-of-the-art performance. The results indicate that two-dimensional feature engineering can take advantage of a two-dimensional sentence representation and make full use of prior knowledge in traditional feature engineering. Our code is publicly available at https://github.com/Wang-ck123/A-Two-Dimensional-Feature-Engineering-Method-for-Entity-Relation-Extraction

Title: PairAug: What Can Augmented Image-Text Pairs Do for Radiology?

• Authors: Yutong Xie, Qi Chen, Sinuo Wang, Minh-Son To, Iris Lee, Ee Win Khoo, Kerolos Hendy, Daniel Koh, Yong Xia, Qi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04960
• Pdf URL: https://arxiv.org/pdf/2404.04960
• Copy Paste: [[2404.04960]] PairAug: What Can Augmented Image-Text Pairs Do for Radiology?(https://arxiv.org/abs/2404.04960)
• Keywords: privacy, large language model
• Abstract: Current vision-language pre-training (VLP) methodologies predominantly depend on paired image-text datasets, a resource that is challenging to acquire in radiology due to privacy considerations and labelling complexities. Data augmentation provides a practical solution to overcome the issue of data scarcity, however, most augmentation methods exhibit a limited focus, prioritising either image or text augmentation exclusively. Acknowledging this limitation, our objective is to devise a framework capable of concurrently augmenting medical image and text data. We design a Pairwise Augmentation (PairAug) approach that contains an Inter-patient Augmentation (InterAug) branch and an Intra-patient Augmentation (IntraAug) branch. Specifically, the InterAug branch of our approach generates radiology images using synthesised yet plausible reports derived from a Large Language Model (LLM). The generated pairs can be considered a collection of new patient cases since they are artificially created and may not exist in the original dataset. In contrast, the IntraAug branch uses newly generated reports to manipulate images. This process allows us to create new paired data for each individual with diverse medical conditions. Our extensive experiments on various downstream tasks covering medical image classification zero-shot and fine-tuning analysis demonstrate that our PairAug, concurrently expanding both image and text data, substantially outperforms image-/text-only expansion baselines and advanced medical VLP baselines. Our code is released at \url{https://github.com/YtongXie/PairAug}.

Title: SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials

• Authors: Mael Jullien, Marco Valentino, André Freitas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04963
• Pdf URL: https://arxiv.org/pdf/2404.04963
• Copy Paste: [[2404.04963]] SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials(https://arxiv.org/abs/2404.04963)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) are at the forefront of NLP achievements but fall short in dealing with shortcut learning, factual inconsistency, and vulnerability to adversarial inputs.These shortcomings are especially critical in medical contexts, where they can misrepresent actual model capabilities. Addressing this, we present SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for ClinicalTrials. Our contributions include the refined NLI4CT-P dataset (i.e., Natural Language Inference for Clinical Trials - Perturbed), designed to challenge LLMs with interventional and causal reasoning tasks, along with a comprehensive evaluation of methods and results for participant submissions. A total of 106 participants registered for the task contributing to over 1200 individual submissions and 25 system overview papers. This initiative aims to advance the robustness and applicability of NLI models in healthcare, ensuring safer and more dependable AI assistance in clinical decision-making. We anticipate that the dataset, models, and outcomes of this task can support future research in the field of biomedical NLI. The dataset, competition leaderboard, and website are publicly available.

Title: Temporal Generalization Estimation in Evolving Graphs

• Authors: Bin Lu, Tingyan Ma, Xiaoying Gan, Xinbing Wang, Yunqiang Zhu, Chenghu Zhou, Shiyu Liang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04969
• Pdf URL: https://arxiv.org/pdf/2404.04969
• Copy Paste: [[2404.04969]] Temporal Generalization Estimation in Evolving Graphs(https://arxiv.org/abs/2404.04969)
• Keywords: extraction
• Abstract: Graph Neural Networks (GNNs) are widely deployed in vast fields, but they often struggle to maintain accurate representations as graphs evolve. We theoretically establish a lower bound, proving that under mild conditions, representation distortion inevitably occurs over time. To estimate the temporal distortion without human annotation after deployment, one naive approach is to pre-train a recurrent model (e.g., RNN) before deployment and use this model afterwards, but the estimation is far from satisfactory. In this paper, we analyze the representation distortion from an information theory perspective, and attribute it primarily to inaccurate feature extraction during evolution. Consequently, we introduce Smart, a straightforward and effective baseline enhanced by an adaptive feature extractor through self-supervised graph reconstruction. In synthetic random graphs, we further refine the former lower bound to show the inevitable distortion over time and empirically observe that Smart achieves good estimation performance. Moreover, we observe that Smart consistently shows outstanding generalization estimation on four real-world evolving graphs. The ablation studies underscore the necessity of graph reconstruction. For example, on OGB-arXiv dataset, the estimation metric MAPE deteriorates from 2.19% to 8.00% without reconstruction.

Title: FPL+: Filtered Pseudo Label-based Unsupervised Cross-Modality Adaptation for 3D Medical Image Segmentation

• Authors: Jianghao Wu, Dong Guo, Guotai Wang, Qiang Yue, Huijun Yu, Kang Li, Shaoting Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04971
• Pdf URL: https://arxiv.org/pdf/2404.04971
• Copy Paste: [[2404.04971]] FPL+: Filtered Pseudo Label-based Unsupervised Cross-Modality Adaptation for 3D Medical Image Segmentation(https://arxiv.org/abs/2404.04971)
• Keywords: segmentation
• Abstract: Adapting a medical image segmentation model to a new domain is important for improving its cross-domain transferability, and due to the expensive annotation process, Unsupervised Domain Adaptation (UDA) is appealing where only unlabeled images are needed for the adaptation. Existing UDA methods are mainly based on image or feature alignment with adversarial training for regularization, and they are limited by insufficient supervision in the target domain. In this paper, we propose an enhanced Filtered Pseudo Label (FPL+)-based UDA method for 3D medical image segmentation. It first uses cross-domain data augmentation to translate labeled images in the source domain to a dual-domain training set consisting of a pseudo source-domain set and a pseudo target-domain set. To leverage the dual-domain augmented images to train a pseudo label generator, domain-specific batch normalization layers are used to deal with the domain shift while learning the domain-invariant structure features, generating high-quality pseudo labels for target-domain images. We then combine labeled source-domain images and target-domain images with pseudo labels to train a final segmentor, where image-level weighting based on uncertainty estimation and pixel-level weighting based on dual-domain consensus are proposed to mitigate the adverse effect of noisy pseudo labels. Experiments on three public multi-modal datasets for Vestibular Schwannoma, brain tumor and whole heart segmentation show that our method surpassed ten state-of-the-art UDA methods, and it even achieved better results than fully supervised learning in the target domain in some cases.

Title: MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models

• Authors: Zihao Wei, Jingcheng Deng, Liang Pang, Hanxing Ding, Huawei Shen, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04990
• Pdf URL: https://arxiv.org/pdf/2404.04990
• Copy Paste: [[2404.04990]] MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models(https://arxiv.org/abs/2404.04990)
• Keywords: large language model
• Abstract: The extensive utilization of large language models (LLMs) underscores the crucial necessity for precise and contemporary knowledge embedded within their intrinsic parameters. Existing research on knowledge editing primarily concentrates on monolingual scenarios, neglecting the complexities presented by multilingual contexts and multi-hop reasoning. To address these challenges, our study introduces MLaKE (Multilingual Language Knowledge Editing), a novel benchmark comprising 4072 multi-hop and 5360 single-hop questions designed to evaluate the adaptability of knowledge editing methods across five languages: English, Chinese, Japanese, French, and German. MLaKE aggregates fact chains from Wikipedia across languages and utilizes LLMs to generate questions in both free-form and multiple-choice. We evaluate the multilingual knowledge editing generalization capabilities of existing methods on MLaKE. Existing knowledge editing methods demonstrate higher success rates in English samples compared to other languages. However, their generalization capabilities are limited in multi-language experiments. Notably, existing knowledge editing methods often show relatively high generalization for languages within the same language family compared to languages from different language families. These results underscore the imperative need for advancements in multilingual knowledge editing and we hope MLaKE can serve as a valuable resource for benchmarking and solution development.

Title: OSS Malicious Package Analysis in the Wild

• Authors: Xiaoyan Zhou, Ying Zhang, Wenjia Niu, Jiqiang Liu, Haining Wang, Qiang Li
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.04991
• Pdf URL: https://arxiv.org/pdf/2404.04991
• Copy Paste: [[2404.04991]] OSS Malicious Package Analysis in the Wild(https://arxiv.org/abs/2404.04991)
• Keywords: security, attack
• Abstract: The open-source software (OSS) ecosystem suffers from various security threats and risks, and malicious packages play a central role in software supply chain (SSC) attacks. Although malware research has a history of over thirty years, less attention has been paid to OSS malware. Its existing research has three limitations: a lack of high-quality datasets, malware diversity, and attack campaign context. In this paper, we first build and curate the largest dataset of 23,425 malicious packages from scattered online sources. We then propose a knowledge graph to represent the OSS malware corpus and conduct malicious package analysis in the wild. Our main findings include (1) it is essential to collect malicious packages from various online sources because there is little data overlap between different sources; (2) despite the sheer volume of SSC attack campaigns, many malicious packages are similar, and unknown/sophisticated attack behaviors have yet to emerge or be detected; (3) OSS malicious package has its distinct life cycle, denoted as {changing->release->detection->removal}, and slightly changing the package (different name) is a widespread attack manner; (4) while malicious packages often lack context about how and who released them, security reports disclose the information about corresponding SSC attack campaigns.

Title: Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM

• Authors: Pingping Zhang, Tianyu Yan, Yang Liu, Huchuan Lu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.04996
• Pdf URL: https://arxiv.org/pdf/2404.04996
• Copy Paste: [[2404.04996]] Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM(https://arxiv.org/abs/2404.04996)
• Keywords: segmentation
• Abstract: As an important pillar of underwater intelligence, Marine Animal Segmentation (MAS) involves segmenting animals within marine environments. Previous methods don't excel in extracting long-range contextual features and overlook the connectivity between discrete pixels. Recently, Segment Anything Model (SAM) offers a universal framework for general segmentation tasks. Unfortunately, trained with natural images, SAM does not obtain the prior knowledge from marine images. In addition, the single-position prompt of SAM is very insufficient for prior guidance. To address these issues, we propose a novel feature learning framework, named Dual-SAM for high-performance MAS. To this end, we first introduce a dual structure with SAM's paradigm to enhance feature learning of marine images. Then, we propose a Multi-level Coupled Prompt (MCP) strategy to instruct comprehensive underwater prior information, and enhance the multi-level features of SAM's encoder with adapters. Subsequently, we design a Dilated Fusion Attention Module (DFAM) to progressively integrate multi-level features from SAM's encoder. Finally, instead of directly predicting the masks of marine animals, we propose a Criss-Cross Connectivity Prediction (C$^3$P) paradigm to capture the inter-connectivity between discrete pixels. With dual decoders, it generates pseudo-labels and achieves mutual supervision for complementary feature representations, resulting in considerable improvements over previous techniques. Extensive experiments verify that our proposed method achieves state-of-the-art performances on five widely-used MAS datasets. The code is available at https://github.com/Drchip61/Dual_SAM.

Title: Adapting LLMs for Efficient Context Processing through Soft Prompt Compression

• Authors: Cangqing Wang, Yutian Yang, Ruisi Li, Dan Sun, Ruicong Cai, Yuzhu Zhang, Chengqian Fu, Lillian Floyd
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04997
• Pdf URL: https://arxiv.org/pdf/2404.04997
• Copy Paste: [[2404.04997]] Adapting LLMs for Efficient Context Processing through Soft Prompt Compression(https://arxiv.org/abs/2404.04997)
• Keywords: robust, large language model
• Abstract: The rapid advancement of Large Language Models (LLMs) has inaugurated a transformative epoch in natural language processing, fostering unprecedented proficiency in text generation, comprehension, and contextual scrutiny. Nevertheless, effectively handling extensive contexts, crucial for myriad applications, poses a formidable obstacle owing to the intrinsic constraints of the models' context window sizes and the computational burdens entailed by their operations. This investigation presents an innovative framework that strategically tailors LLMs for streamlined context processing by harnessing the synergies among natural language summarization, soft prompt compression, and augmented utility preservation mechanisms. Our methodology, dubbed SoftPromptComp, amalgamates natural language prompts extracted from summarization methodologies with dynamically generated soft prompts to forge a concise yet semantically robust depiction of protracted contexts. This depiction undergoes further refinement via a weighting mechanism optimizing information retention and utility for subsequent tasks. We substantiate that our framework markedly diminishes computational overhead and enhances LLMs' efficacy across various benchmarks, while upholding or even augmenting the caliber of the produced content. By amalgamating soft prompt compression with sophisticated summarization, SoftPromptComp confronts the dual challenges of managing lengthy contexts and ensuring model scalability. Our findings point towards a propitious trajectory for augmenting LLMs' applicability and efficiency, rendering them more versatile and pragmatic for real-world applications. This research enriches the ongoing discourse on optimizing language models, providing insights into the potency of soft prompts and summarization techniques as pivotal instruments for the forthcoming generation of NLP solutions.

Title: Dual-Scale Transformer for Large-Scale Single-Pixel Imaging

• Authors: Gang Qu, Ping Wang, Xin Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05001
• Pdf URL: https://arxiv.org/pdf/2404.05001
• Copy Paste: [[2404.05001]] Dual-Scale Transformer for Large-Scale Single-Pixel Imaging(https://arxiv.org/abs/2404.05001)
• Keywords: transformer
• Abstract: Single-pixel imaging (SPI) is a potential computational imaging technique which produces image by solving an illposed reconstruction problem from few measurements captured by a single-pixel detector. Deep learning has achieved impressive success on SPI reconstruction. However, previous poor reconstruction performance and impractical imaging model limit its real-world applications. In this paper, we propose a deep unfolding network with hybrid-attention Transformer on Kronecker SPI model, dubbed HATNet, to improve the imaging quality of real SPI cameras. Specifically, we unfold the computation graph of the iterative shrinkagethresholding algorithm (ISTA) into two alternative modules: efficient tensor gradient descent and hybrid-attention multiscale denoising. By virtue of Kronecker SPI, the gradient descent module can avoid high computational overheads rooted in previous gradient descent modules based on vectorized SPI. The denoising module is an encoder-decoder architecture powered by dual-scale spatial attention for high- and low-frequency aggregation and channel attention for global information recalibration. Moreover, we build a SPI prototype to verify the effectiveness of the proposed method. Extensive experiments on synthetic and real data demonstrate that our method achieves the state-of-the-art performance. The source code and pre-trained models are available at https://github.com/Gang-Qu/HATNet-SPI.

Title: MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators

• Authors: Shenghai Yuan, Jinfa Huang, Yujun Shi, Yongqi Xu, Ruijie Zhu, Bin Lin, Xinhua Cheng, Li Yuan, Jiebo Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05014
• Pdf URL: https://arxiv.org/pdf/2404.05014
• Copy Paste: [[2404.05014]] MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators(https://arxiv.org/abs/2404.05014)
• Keywords: extraction
• Abstract: Recent advances in Text-to-Video generation (T2V) have achieved remarkable success in synthesizing high-quality general videos from textual descriptions. A largely overlooked problem in T2V is that existing models have not adequately encoded physical knowledge of the real world, thus generated videos tend to have limited motion and poor variations. In this paper, we propose \textbf{MagicTime}, a metamorphic time-lapse video generation model, which learns real-world physics knowledge from time-lapse videos and implements metamorphic generation. First, we design a MagicAdapter scheme to decouple spatial and temporal training, encode more physical knowledge from metamorphic videos, and transform pre-trained T2V models to generate metamorphic videos. Second, we introduce a Dynamic Frames Extraction strategy to adapt to metamorphic time-lapse videos, which have a wider variation range and cover dramatic object metamorphic processes, thus embodying more physical knowledge than general videos. Finally, we introduce a Magic Text-Encoder to improve the understanding of metamorphic video prompts. Furthermore, we create a time-lapse video-text dataset called \textbf{ChronoMagic}, specifically curated to unlock the metamorphic video generation ability. Extensive experiments demonstrate the superiority and effectiveness of MagicTime for generating high-quality and dynamic metamorphic videos, suggesting time-lapse video generation is a promising path toward building metamorphic simulators of the physical world.

Title: LOGO: A Long-Form Video Dataset for Group Action Quality Assessment

• Authors: Shiyi Zhang, Wenxun Dai, Sujia Wang, Xiangwei Shen, Jiwen Lu, Jie Zhou, Yansong Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05029
• Pdf URL: https://arxiv.org/pdf/2404.05029
• Copy Paste: [[2404.05029]] LOGO: A Long-Form Video Dataset for Group Action Quality Assessment(https://arxiv.org/abs/2404.05029)
• Keywords: segmentation
• Abstract: Action quality assessment (AQA) has become an emerging topic since it can be extensively applied in numerous scenarios. However, most existing methods and datasets focus on single-person short-sequence scenes, hindering the application of AQA in more complex situations. To address this issue, we construct a new multi-person long-form video dataset for action quality assessment named LOGO. Distinguished in scenario complexity, our dataset contains 200 videos from 26 artistic swimming events with 8 athletes in each sample along with an average duration of 204.2 seconds. As for richness in annotations, LOGO includes formation labels to depict group information of multiple athletes and detailed annotations on action procedures. Furthermore, we propose a simple yet effective method to model relations among athletes and reason about the potential temporal logic in long-form videos. Specifically, we design a group-aware attention module, which can be easily plugged into existing AQA methods, to enrich the clip-wise representations based on contextual group information. To benchmark LOGO, we systematically conduct investigations on the performance of several popular methods in AQA and action segmentation. The results reveal the challenges our dataset brings. Extensive experiments also show that our approach achieves state-of-the-art on the LOGO dataset. The dataset and code will be released at \url{https://github.com/shiyi-zh0408/LOGO }.

Title: Optimizing Privacy and Utility Tradeoffs for Group Interests Through Harmonization

• Authors: Bishwas Mandal, George Amariucai, Shuangqing Wei
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05043
• Pdf URL: https://arxiv.org/pdf/2404.05043
• Copy Paste: [[2404.05043]] Optimizing Privacy and Utility Tradeoffs for Group Interests Through Harmonization(https://arxiv.org/abs/2404.05043)
• Keywords: privacy
• Abstract: We propose a novel problem formulation to address the privacy-utility tradeoff, specifically when dealing with two distinct user groups characterized by unique sets of private and utility attributes. Unlike previous studies that primarily focus on scenarios where all users share identical private and utility attributes and often rely on auxiliary datasets or manual annotations, we introduce a collaborative data-sharing mechanism between two user groups through a trusted third party. This third party uses adversarial privacy techniques with our proposed data-sharing mechanism to internally sanitize data for both groups and eliminates the need for manual annotation or auxiliary datasets. Our methodology ensures that private attributes cannot be accurately inferred while enabling highly accurate predictions of utility features. Importantly, even if analysts or adversaries possess auxiliary datasets containing raw data, they are unable to accurately deduce private features. Additionally, our data-sharing mechanism is compatible with various existing adversarially trained privacy techniques. We empirically demonstrate the effectiveness of our approach using synthetic and real-world datasets, showcasing its ability to balance the conflicting goals of privacy and utility.

Title: Initial Exploration of Zero-Shot Privacy Utility Tradeoffs in Tabular Data Using GPT-4

• Authors: Bishwas Mandal, George Amariucai, Shuangqing Wei
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05047
• Pdf URL: https://arxiv.org/pdf/2404.05047
• Copy Paste: [[2404.05047]] Initial Exploration of Zero-Shot Privacy Utility Tradeoffs in Tabular Data Using GPT-4(https://arxiv.org/abs/2404.05047)
• Keywords: privacy, fair, large language model
• Abstract: We investigate the application of large language models (LLMs), specifically GPT-4, to scenarios involving the tradeoff between privacy and utility in tabular data. Our approach entails prompting GPT-4 by transforming tabular data points into textual format, followed by the inclusion of precise sanitization instructions in a zero-shot manner. The primary objective is to sanitize the tabular data in such a way that it hinders existing machine learning models from accurately inferring private features while allowing models to accurately infer utility-related attributes. We explore various sanitization instructions. Notably, we discover that this relatively simple approach yields performance comparable to more complex adversarial optimization methods used for managing privacy-utility tradeoffs. Furthermore, while the prompts successfully obscure private features from the detection capabilities of existing machine learning models, we observe that this obscuration alone does not necessarily meet a range of fairness metrics. Nevertheless, our research indicates the potential effectiveness of LLMs in adhering to these fairness metrics, with some of our experimental results aligning with those achieved by well-established adversarial optimization techniques.

Title: PlateSegFL: A Privacy-Preserving License Plate Detection Using Federated Segmentation Learning

• Authors: Md. Shahriar Rahman Anuvab, Mishkat Sultana, Md. Atif Hossain, Shashwata Das, Suvarthi Chowdhury, Rafeed Rahman, Dibyo Fabian Dofadar, Shahriar Rahman Rana
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05049
• Pdf URL: https://arxiv.org/pdf/2404.05049
• Copy Paste: [[2404.05049]] PlateSegFL: A Privacy-Preserving License Plate Detection Using Federated Segmentation Learning(https://arxiv.org/abs/2404.05049)
• Keywords: secure, privacy, federate, segmentation
• Abstract: Automatic License Plate Recognition (ALPR) is an integral component of an intelligent transport system with extensive applications in secure transportation, vehicle-to-vehicle communication, stolen vehicles detection, traffic violations, and traffic flow management. The existing license plate detection system focuses on one-shot learners or pre-trained models that operate with a geometric bounding box, limiting the model's performance. Furthermore, continuous video data streams uploaded to the central server result in network and complexity issues. To combat this, PlateSegFL was introduced, which implements U-Net-based segmentation along with Federated Learning (FL). U-Net is well-suited for multi-class image segmentation tasks because it can analyze a large number of classes and generate a pixel-level segmentation map for each class. Federated Learning is used to reduce the quantity of data required while safeguarding the user's privacy. Different computing platforms, such as mobile phones, are able to collaborate on the development of a standard prediction model where it makes efficient use of one's time; incorporates more diverse data; delivers projections in real-time; and requires no physical effort from the user; resulting around 95% F1 score.

Title: Facial Affective Behavior Analysis with Instruction Tuning

• Authors: Yifan Li, Anh Dao, Wentao Bao, Zhen Tan, Tianlong Chen, Huan Liu, Yu Kong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05052
• Pdf URL: https://arxiv.org/pdf/2404.05052
• Copy Paste: [[2404.05052]] Facial Affective Behavior Analysis with Instruction Tuning(https://arxiv.org/abs/2404.05052)
• Keywords: interpretability, large language model
• Abstract: Facial affective behavior analysis (FABA) is crucial for understanding human mental states from images. However, traditional approaches primarily deploy models to discriminate among discrete emotion categories, and lack the fine granularity and reasoning capability for complex facial behaviors. The advent of Multi-modal Large Language Models (MLLMs) has been proven successful in general visual understanding tasks. However, directly harnessing MLLMs for FABA is challenging due to the scarcity of datasets and benchmarks, neglecting facial prior knowledge, and low training efficiency. To address these challenges, we introduce (i) an instruction-following dataset for two FABA tasks, e.g., emotion and action unit recognition, (ii) a benchmark FABA-Bench with a new metric considering both recognition and generation ability, and (iii) a new MLLM "EmoLA" as a strong baseline to the community. Our initiative on the dataset and benchmarks reveal the nature and rationale of facial affective behaviors, i.e., fine-grained facial movement, interpretability, and reasoning. Moreover, to build an effective and efficient FABA MLLM, we introduce a facial prior expert module with face structure knowledge and a low-rank adaptation module into pre-trained MLLM. We conduct extensive experiments on FABA-Bench and four commonly-used FABA datasets. The results demonstrate that the proposed facial prior expert can boost the performance and EmoLA achieves the best results on our FABA-Bench. On commonly-used FABA datasets, EmoLA is competitive rivaling task-specific state-of-the-art models.

Title: Percentile Criterion Optimization in Offline Reinforcement Learning

• Authors: Elita A. Lobo, Cyrus Cousins, Yair Zick, Marek Petrik
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05055
• Pdf URL: https://arxiv.org/pdf/2404.05055
• Copy Paste: [[2404.05055]] Percentile Criterion Optimization in Offline Reinforcement Learning(https://arxiv.org/abs/2404.05055)
• Keywords: robust
• Abstract: In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the \emph{percentile criterion}. The percentile criterion is approximately solved by constructing an \emph{ambiguity set} that contains the true model with high probability and optimizing the policy for the worst model in the set. Since the percentile criterion is non-convex, constructing ambiguity sets is often challenging. Existing work uses \emph{Bayesian credible regions} as ambiguity sets, but they are often unnecessarily large and result in learning overly conservative policies. To overcome these shortcomings, we propose a novel Value-at-Risk based dynamic programming algorithm to optimize the percentile criterion without explicitly constructing any ambiguity sets. Our theoretical and empirical results show that our algorithm implicitly constructs much smaller ambiguity sets and learns less conservative robust policies.

Title: A robust assessment for invariant representations

• Authors: Wenlu Tang, Zicheng Liu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.05058
• Pdf URL: https://arxiv.org/pdf/2404.05058
• Copy Paste: [[2404.05058]] A robust assessment for invariant representations(https://arxiv.org/abs/2404.05058)
• Keywords: robust
• Abstract: The performance of machine learning models can be impacted by changes in data over time. A promising approach to address this challenge is invariant learning, with a particular focus on a method known as invariant risk minimization (IRM). This technique aims to identify a stable data representation that remains effective with out-of-distribution (OOD) data. While numerous studies have developed IRM-based methods adaptive to data augmentation scenarios, there has been limited attention on directly assessing how well these representations preserve their invariant performance under varying conditions. In our paper, we propose a novel method to evaluate invariant performance, specifically tailored for IRM-based methods. We establish a bridge between the conditional expectation of an invariant predictor across different environments through the likelihood ratio. Our proposed criterion offers a robust basis for evaluating invariant performance. We validate our approach with theoretical support and demonstrate its effectiveness through extensive numerical studies.These experiments illustrate how our method can assess the invariant performance of various representation techniques.

Title: Automated Prediction of Breast Cancer Response to Neoadjuvant Chemotherapy from DWI Data

• Authors: Shir Nitzan, Maya Gilad, Moti Freiman
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05061
• Pdf URL: https://arxiv.org/pdf/2404.05061
• Copy Paste: [[2404.05061]] Automated Prediction of Breast Cancer Response to Neoadjuvant Chemotherapy from DWI Data(https://arxiv.org/abs/2404.05061)
• Keywords: robust, diffusion, segmentation
• Abstract: Effective surgical planning for breast cancer hinges on accurately predicting pathological complete response (pCR) to neoadjuvant chemotherapy (NAC). Diffusion-weighted MRI (DWI) and machine learning offer a non-invasive approach for early pCR assessment. However, most machine-learning models require manual tumor segmentation, a cumbersome and error-prone task. We propose a deep learning model employing "Size-Adaptive Lesion Weighting" for automatic DWI tumor segmentation to enhance pCR prediction accuracy. Despite histopathological changes during NAC complicating DWI image segmentation, our model demonstrates robust performance. Utilizing the BMMR2 challenge dataset, it matches human experts in pCR prediction pre-NAC with an area under the curve (AUC) of 0.76 vs. 0.796, and surpasses standard automated methods mid-NAC, with an AUC of 0.729 vs. 0.654 and 0.576. Our approach represents a significant advancement in automating breast cancer treatment planning, enabling more reliable pCR predictions without manual segmentation.

Title: AirShot: Efficient Few-Shot Detection for Autonomous Exploration

• Authors: Zihan Wang, Bowen Li, Chen Wang, Sebastian Scherer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05069
• Pdf URL: https://arxiv.org/pdf/2404.05069
• Copy Paste: [[2404.05069]] AirShot: Efficient Few-Shot Detection for Autonomous Exploration(https://arxiv.org/abs/2404.05069)
• Keywords: robust
• Abstract: Few-shot object detection has drawn increasing attention in the field of robotic exploration, where robots are required to find unseen objects with a few online provided examples. Despite recent efforts have been made to yield online processing capabilities, slow inference speeds of low-powered robots fail to meet the demands of real-time detection-making them impractical for autonomous exploration. Existing methods still face performance and efficiency challenges, mainly due to unreliable features and exhaustive class loops. In this work, we propose a new paradigm AirShot, and discover that, by fully exploiting the valuable correlation map, AirShot can result in a more robust and faster few-shot object detection system, which is more applicable to robotics community. The core module Top Prediction Filter (TPF) can operate on multi-scale correlation maps in both the training and inference stages. During training, TPF supervises the generation of a more representative correlation map, while during inference, it reduces looping iterations by selecting top-ranked classes, thus cutting down on computational costs with better performance. Surprisingly, this dual functionality exhibits general effectiveness and efficiency on various off-the-shelf models. Exhaustive experiments on COCO2017, VOC2014, and SubT datasets demonstrate that TPF can significantly boost the efficacy and efficiency of most off-the-shelf models, achieving up to 36.4% precision improvements along with 56.3% faster inference speed. Code and Data are at: https://github.com/ImNotPrepared/AirShot.

Title: Test-Time Training for Depression Detection

• Authors: Sri Harsha Dumpala, Chandramouli Shama Sastry, Rudolf Uher, Sageev Oore
• Subjects: cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.05071
• Pdf URL: https://arxiv.org/pdf/2404.05071
• Copy Paste: [[2404.05071]] Test-Time Training for Depression Detection(https://arxiv.org/abs/2404.05071)
• Keywords: robust
• Abstract: Previous works on depression detection use datasets collected in similar environments to train and test the models. In practice, however, the train and test distributions cannot be guaranteed to be identical. Distribution shifts can be introduced due to variations such as recording environment (e.g., background noise) and demographics (e.g., gender, age, etc). Such distributional shifts can surprisingly lead to severe performance degradation of the depression detection models. In this paper, we analyze the application of test-time training (TTT) to improve robustness of models trained for depression detection. When compared to regular testing of the models, we find TTT can significantly improve the robustness of the model under a variety of distributional shifts introduced due to: (a) background-noise, (b) gender-bias, and (c) data collection and curation procedure (i.e., train and test samples are from separate datasets).

Title: HaVTR: Improving Video-Text Retrieval Through Augmentation Using Large Foundation Models

• Authors: Yimu Wang, Shuai Yuan, Xiangru Jian, Wei Pang, Mushi Wang, Ning Yu
• Subjects: cs.CV, cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05083
• Pdf URL: https://arxiv.org/pdf/2404.05083
• Copy Paste: [[2404.05083]] HaVTR: Improving Video-Text Retrieval Through Augmentation Using Large Foundation Models(https://arxiv.org/abs/2404.05083)
• Keywords: generative, large language model
• Abstract: While recent progress in video-text retrieval has been driven by the exploration of powerful model architectures and training strategies, the representation learning ability of video-text retrieval models is still limited due to low-quality and scarce training data annotations. To address this issue, we present a novel video-text learning paradigm, HaVTR, which augments video and text data to learn more generalized features. Specifically, we first adopt a simple augmentation method, which generates self-similar data by randomly duplicating or dropping subwords and frames. In addition, inspired by the recent advancement in visual and language generative models, we propose a more powerful augmentation method through textual paraphrasing and video stylization using large language models (LLMs) and visual generative models (VGMs). Further, to bring richer information into video and text, we propose a hallucination-based augmentation method, where we use LLMs and VGMs to generate and add new relevant information to the original data. Benefiting from the enriched data, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of HaVTR over existing methods.

Title: A Note on LoRA

• Authors: Vlad Fomenko, Han Yu, Jongho Lee, Stanley Hsieh, Weizhu Chen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05086
• Pdf URL: https://arxiv.org/pdf/2404.05086
• Copy Paste: [[2404.05086]] A Note on LoRA(https://arxiv.org/abs/2404.05086)
• Keywords: large language model
• Abstract: LoRA (Low-Rank Adaptation) has emerged as a preferred method for efficiently adapting Large Language Models (LLMs) with remarkable simplicity and efficacy. This note extends the original LoRA paper by offering new perspectives that were not initially discussed and presents a series of insights for deploying LoRA at scale. Without introducing new experiments, we aim to improve the understanding and application of LoRA.

Title: How much reliable is ChatGPT's prediction on Information Extraction under Input Perturbations?

• Authors: Ishani Mondal, Abhilasha Sancheti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05088
• Pdf URL: https://arxiv.org/pdf/2404.05088
• Copy Paste: [[2404.05088]] How much reliable is ChatGPT's prediction on Information Extraction under Input Perturbations?(https://arxiv.org/abs/2404.05088)
• Keywords: robust, extraction
• Abstract: In this paper, we assess the robustness (reliability) of ChatGPT under input perturbations for one of the most fundamental tasks of Information Extraction (IE) i.e. Named Entity Recognition (NER). Despite the hype, the majority of the researchers have vouched for its language understanding and generation capabilities; a little attention has been paid to understand its robustness: How the input-perturbations affect 1) the predictions, 2) the confidence of predictions and 3) the quality of rationale behind its prediction. We perform a systematic analysis of ChatGPT's robustness (under both zero-shot and few-shot setup) on two NER datasets using both automatic and human evaluation. Based on automatic evaluation metrics, we find that 1) ChatGPT is more brittle on Drug or Disease replacements (rare entities) compared to the perturbations on widely known Person or Location entities, 2) the quality of explanations for the same entity considerably differ under different types of "Entity-Specific" and "Context-Specific" perturbations and the quality can be significantly improved using in-context learning, and 3) it is overconfident for majority of the incorrect predictions, and hence it could lead to misguidance of the end-users.

Title: VMambaMorph: a Visual Mamba-based Framework with Cross-Scan Module for Deformable 3D Image Registration

• Authors: Ziyang Wang, Jian-Qing Zheng, Chao Ma, Tao Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05105
• Pdf URL: https://arxiv.org/pdf/2404.05105
• Copy Paste: [[2404.05105]] VMambaMorph: a Visual Mamba-based Framework with Cross-Scan Module for Deformable 3D Image Registration(https://arxiv.org/abs/2404.05105)
• Keywords: extraction, transformer
• Abstract: Image registration, a critical process in medical imaging, involves aligning different sets of medical imaging data into a single unified coordinate system. Deep learning networks, such as the Convolutional Neural Network (CNN)-based VoxelMorph, Vision Transformer (ViT)-based TransMorph, and State Space Model (SSM)-based MambaMorph, have demonstrated effective performance in this domain. The recent Visual State Space Model (VMamba), which incorporates a cross-scan module with SSM, has exhibited promising improvements in modeling global-range dependencies with efficient computational cost in computer vision tasks. This paper hereby introduces an exploration of VMamba with image registration, named VMambaMorph. This novel hybrid VMamba-CNN network is designed specifically for 3D image registration. Utilizing a U-shaped network architecture, VMambaMorph computes the deformation field based on target and source volumes. The VMamba-based block with 2D cross-scan module is redesigned for 3D volumetric feature processing, and a fine-grained feature extraction module is proposed for high-dimensional feature learning. We validate VMambaMorph using a public benchmark brain MR-CT registration dataset, comparing its performance against current state-of-the-art methods. The results indicate that VMambaMorph achieves competitive registration quality. The code for VMambaMorph is available on GitHub.

Title: Stop Stealing My Data: Sanitizing Stego Channels in 3D Printing Design Files

• Authors: Aleksandr Dolgavin, Mark Yampolskiy, Moti Yung
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05106
• Pdf URL: https://arxiv.org/pdf/2404.05106
• Copy Paste: [[2404.05106]] Stop Stealing My Data: Sanitizing Stego Channels in 3D Printing Design Files(https://arxiv.org/abs/2404.05106)
• Keywords: secure, security, steal
• Abstract: The increased adoption of additive manufacturing (AM) and the acceptance of AM outsourcing created an ecosystem in which the sending and receiving of digital designs by different actors became normal. It has recently been shown that the STL design files -- most commonly used in AM -- contain steganographic channels. Such channels can allow additional data to be embedded within the STL files without changing the printed model. These factors create a threat of misusing the design files as a covert communication channel to either exfiltrate stolen sensitive digital data from organizations or infiltrate malicious software into a secure environment. This paper addresses this security threat by designing and evaluating a \emph{sanitizer} that erases hidden content where steganographic channels might exist. The proposed sanitizer takes into account a set of specific constraints imposed by the application domain, such as not affecting the ability to manufacture part of the required quality using the sanitized design.

Title: Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by Unsupervised Learning

• Authors: Yujian Xiong, Wenhui Zhu, Zhong-Lin Lu, Yalin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05107
• Pdf URL: https://arxiv.org/pdf/2404.05107
• Copy Paste: [[2404.05107]] Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by Unsupervised Learning(https://arxiv.org/abs/2404.05107)
• Keywords: interpretability, generative
• Abstract: The reconstruction of human visual inputs from brain activity, particularly through functional Magnetic Resonance Imaging (fMRI), holds promising avenues for unraveling the mechanisms of the human visual system. Despite the significant strides made by deep learning methods in improving the quality and interpretability of visual reconstruction, there remains a substantial demand for high-quality, long-duration, subject-specific 7-Tesla fMRI experiments. The challenge arises in integrating diverse smaller 3-Tesla datasets or accommodating new subjects with brief and low-quality fMRI scans. In response to these constraints, we propose a novel framework that generates enhanced 3T fMRI data through an unsupervised Generative Adversarial Network (GAN), leveraging unpaired training across two distinct fMRI datasets in 7T and 3T, respectively. This approach aims to overcome the limitations of the scarcity of high-quality 7-Tesla data and the challenges associated with brief and low-quality scans in 3-Tesla experiments. In this paper, we demonstrate the reconstruction capabilities of the enhanced 3T fMRI data, highlighting its proficiency in generating superior input visual images compared to data-intensive methods trained and tested on a single subject.

Title: Class Similarity Transition: Decoupling Class Similarities and Imbalance from Generalized Few-shot Segmentation

• Authors: Shihong Wang, Ruixun Liu, Kaiyu Li, Jiawei Jiang, Xiangyong Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05111
• Pdf URL: https://arxiv.org/pdf/2404.05111
• Copy Paste: [[2404.05111]] Class Similarity Transition: Decoupling Class Similarities and Imbalance from Generalized Few-shot Segmentation(https://arxiv.org/abs/2404.05111)
• Keywords: segmentation
• Abstract: In Generalized Few-shot Segmentation (GFSS), a model is trained with a large corpus of base class samples and then adapted on limited samples of novel classes. This paper focuses on the relevance between base and novel classes, and improves GFSS in two aspects: 1) mining the similarity between base and novel classes to promote the learning of novel classes, and 2) mitigating the class imbalance issue caused by the volume difference between the support set and the training set. Specifically, we first propose a similarity transition matrix to guide the learning of novel classes with base class knowledge. Then, we leverage the Label-Distribution-Aware Margin (LDAM) loss and Transductive Inference to the GFSS task to address the problem of class imbalance as well as overfitting the support set. In addition, by extending the probability transition matrix, the proposed method can mitigate the catastrophic forgetting of base classes when learning novel classes. With a simple training phase, our proposed method can be applied to any segmentation network trained on base classes. We validated our methods on the adapted version of OpenEarthMap. Compared to existing GFSS baselines, our method excels them all from 3% to 7% and ranks second in the OpenEarthMap Land Cover Mapping Few-Shot Challenge at the completion of this paper. Code: https://github.com/earth-insights/ClassTrans

Title: Image-based Agarwood Resinous Area Segmentation using Deep Learning

• Authors: Irwandi Hipiny, Johari Abdullah, Noor Alamshah Bolhassan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05129
• Pdf URL: https://arxiv.org/pdf/2404.05129
• Copy Paste: [[2404.05129]] Image-based Agarwood Resinous Area Segmentation using Deep Learning(https://arxiv.org/abs/2404.05129)
• Keywords: extraction, segmentation
• Abstract: The manual extraction method of Agarwood resinous compound is laborious work, requires skilled workers, and is subject to human errors. Commercial Agarwood industries have been actively exploring using Computer Numerical Control (CNC) machines to replace human effort for this particular task. The CNC machine accepts a G-code script produced from a binary image in which the wood region that needs to be chiselled off is marked with (0, 0, 0) as its RGB value. Rather than requiring a human expert to perform the region marking, we propose using a Deep learning image segmentation method instead. Our setup involves a camera that captures the cross-section image and then passes the image file to a computer. The computer performs the automated image segmentation and feeds the CNC machine with a G-code script. In this article, we report the initial segmentation results achieved using a state-of-the-art Deep learning segmentation method and discuss potential improvements to refine the segmentation accuracy.

Title: Enabling Privacy-Preserving Cyber Threat Detection with Federated Learning

• Authors: Yu Bi, Yekai Li, Xuan Feng, Xianghang Mi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05130
• Pdf URL: https://arxiv.org/pdf/2404.05130
• Copy Paste: [[2404.05130]] Enabling Privacy-Preserving Cyber Threat Detection with Federated Learning(https://arxiv.org/abs/2404.05130)
• Keywords: security, privacy, protect, attack, federate
• Abstract: Despite achieving good performance and wide adoption, machine learning based security detection models (e.g., malware classifiers) are subject to concept drift and evasive evolution of attackers, which renders up-to-date threat data as a necessity. However, due to enforcement of various privacy protection regulations (e.g., GDPR), it is becoming increasingly challenging or even prohibitive for security vendors to collect individual-relevant and privacy-sensitive threat datasets, e.g., SMS spam/non-spam messages from mobile devices. To address such obstacles, this study systematically profiles the (in)feasibility of federated learning for privacy-preserving cyber threat detection in terms of effectiveness, byzantine resilience, and efficiency. This is made possible by the build-up of multiple threat datasets and threat detection models, and more importantly, the design of realistic and security-specific experiments. We evaluate FL on two representative threat detection tasks, namely SMS spam detection and Android malware detection. It shows that FL-trained detection models can achieve a performance that is comparable to centrally trained counterparts. Also, most non-IID data distributions have either minor or negligible impact on the model performance, while a label-based non-IID distribution of a high extent can incur non-negligible fluctuation and delay in FL training. Then, under a realistic threat model, FL turns out to be adversary-resistant to attacks of both data poisoning and model poisoning. Particularly, the attacking impact of a practical data poisoning attack is no more than 0.14\% loss in model accuracy. Regarding FL efficiency, a bootstrapping strategy turns out to be effective to mitigate the training delay as observed in label-based non-IID scenarios.

Title: Self-Supervised Multi-Object Tracking with Path Consistency

• Authors: Zijia Lu, Bing Shuai, Yanbei Chen, Zhenlin Xu, Davide Modolo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05136
• Pdf URL: https://arxiv.org/pdf/2404.05136
• Copy Paste: [[2404.05136]] Self-Supervised Multi-Object Tracking with Path Consistency(https://arxiv.org/abs/2404.05136)
• Keywords: robust
• Abstract: In this paper, we propose a novel concept of path consistency to learn robust object matching without using manual object identity supervision. Our key idea is that, to track a object through frames, we can obtain multiple different association results from a model by varying the frames it can observe, i.e., skipping frames in observation. As the differences in observations do not alter the identities of objects, the obtained association results should be consistent. Based on this rationale, we generate multiple observation paths, each specifying a different set of frames to be skipped, and formulate the Path Consistency Loss that enforces the association results are consistent across different observation paths. We use the proposed loss to train our object matching model with only self-supervision. By extensive experiments on three tracking datasets (MOT17, PersonPath22, KITTI), we demonstrate that our method outperforms existing unsupervised methods with consistent margins on various evaluation metrics, and even achieves performance close to supervised methods.

Title: Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation

• Authors: Rohan Deepak Ajwani, Zining Zhu, Jonathan Rose, Frank Rudzicz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05143
• Pdf URL: https://arxiv.org/pdf/2404.05143
• Copy Paste: [[2404.05143]] Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation(https://arxiv.org/abs/2404.05143)
• Keywords: transformer, large language model
• Abstract: Transformer-based Large Language Models (LLMs) have shown exceptional language generation capabilities in response to text-based prompts. However, controlling the direction of generation via textual prompts has been challenging, especially with smaller models. In this work, we explore the use of Prompt Tuning to achieve controlled language generation. Generated text is steered using prompt embeddings, which are trained using a small language model, used as a discriminator. Moreover, we demonstrate that these prompt embeddings can be trained with a very small dataset, with as low as a few hundred training examples. Our method thus offers a data and parameter efficient solution towards controlling language model outputs. We carry out extensive evaluation on four datasets: SST-5 and Yelp (sentiment analysis), GYAFC (formality) and JIGSAW (toxic language). Finally, we demonstrate the efficacy of our method towards mitigating harmful, toxic, and biased text generated by language models.

Title: Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients

• Authors: HyoJe Jung, Yunha Kim, Heejung Choi, Hyeram Seo, Minkyoung Kim, JiYe Han, Gaeun Kee, Seohyun Park, Soyoung Ko, Byeolhee Kim, Suyeon Kim, Tae Joon Jun, Young-Hak Kim
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05144
• Pdf URL: https://arxiv.org/pdf/2404.05144
• Copy Paste: [[2404.05144]] Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients(https://arxiv.org/abs/2404.05144)
• Keywords: large language model
• Abstract: Medical documentation, including discharge notes, is crucial for ensuring patient care quality, continuity, and effective medical communication. However, the manual creation of these documents is not only time-consuming but also prone to inconsistencies and potential errors. The automation of this documentation process using artificial intelligence (AI) represents a promising area of innovation in healthcare. This study directly addresses the inefficiencies and inaccuracies in creating discharge notes manually, particularly for cardiac patients, by employing AI techniques, specifically large language model (LLM). Utilizing a substantial dataset from a cardiology center, encompassing wide-ranging medical records and physician assessments, our research evaluates the capability of LLM to enhance the documentation process. Among the various models assessed, Mistral-7B distinguished itself by accurately generating discharge notes that significantly improve both documentation efficiency and the continuity of care for patients. These notes underwent rigorous qualitative evaluation by medical expert, receiving high marks for their clinical relevance, completeness, readability, and contribution to informed decision-making and care planning. Coupled with quantitative analyses, these results confirm Mistral-7B's efficacy in distilling complex medical information into concise, coherent summaries. Overall, our findings illuminate the considerable promise of specialized LLM, such as Mistral-7B, in refining healthcare documentation workflows and advancing patient care. This study lays the groundwork for further integrating advanced AI technologies in healthcare, demonstrating their potential to revolutionize patient documentation and support better care outcomes.

Title: UniMix: Towards Domain Adaptive and Generalizable LiDAR Semantic Segmentation in Adverse Weather

• Authors: Haimei Zhao, Jing Zhang, Zhuo Chen, Shanshan Zhao, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05145
• Pdf URL: https://arxiv.org/pdf/2404.05145
• Copy Paste: [[2404.05145]] UniMix: Towards Domain Adaptive and Generalizable LiDAR Semantic Segmentation in Adverse Weather(https://arxiv.org/abs/2404.05145)
• Keywords: robust, segmentation
• Abstract: LiDAR semantic segmentation (LSS) is a critical task in autonomous driving and has achieved promising progress. However, prior LSS methods are conventionally investigated and evaluated on datasets within the same domain in clear weather. The robustness of LSS models in unseen scenes and all weather conditions is crucial for ensuring safety and reliability in real applications. To this end, we propose UniMix, a universal method that enhances the adaptability and generalizability of LSS models. UniMix first leverages physically valid adverse weather simulation to construct a Bridge Domain, which serves to bridge the domain gap between the clear weather scenes and the adverse weather scenes. Then, a Universal Mixing operator is defined regarding spatial, intensity, and semantic distributions to create the intermediate domain with mixed samples from given domains. Integrating the proposed two techniques into a teacher-student framework, UniMix efficiently mitigates the domain gap and enables LSS models to learn weather-robust and domain-invariant representations. We devote UniMix to two main setups: 1) unsupervised domain adaption, adapting the model from the clear weather source domain to the adverse weather target domain; 2) domain generalization, learning a model that generalizes well to unseen scenes in adverse weather. Extensive experiments validate the effectiveness of UniMix across different tasks and datasets, all achieving superior performance over state-of-the-art methods. The code will be released.

Title: Semantic Stealth: Adversarial Text Attacks on NLP Using Several Methods

• Authors: Roopkatha Dey, Aivy Debnath, Sayak Kumar Dutta, Kaustav Ghosh, Arijit Mitra, Arghya Roy Chowdhury, Jaydip Sen
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05159
• Pdf URL: https://arxiv.org/pdf/2404.05159
• Copy Paste: [[2404.05159]] Semantic Stealth: Adversarial Text Attacks on NLP Using Several Methods(https://arxiv.org/abs/2404.05159)
• Keywords: attack, robust, steal, interpretability
• Abstract: In various real-world applications such as machine translation, sentiment analysis, and question answering, a pivotal role is played by NLP models, facilitating efficient communication and decision-making processes in domains ranging from healthcare to finance. However, a significant challenge is posed to the robustness of these natural language processing models by text adversarial attacks. These attacks involve the deliberate manipulation of input text to mislead the predictions of the model while maintaining human interpretability. Despite the remarkable performance achieved by state-of-the-art models like BERT in various natural language processing tasks, they are found to remain vulnerable to adversarial perturbations in the input text. In addressing the vulnerability of text classifiers to adversarial attacks, three distinct attack mechanisms are explored in this paper using the victim model BERT: BERT-on-BERT attack, PWWS attack, and Fraud Bargain's Attack (FBA). Leveraging the IMDB, AG News, and SST2 datasets, a thorough comparative analysis is conducted to assess the effectiveness of these attacks on the BERT classifier model. It is revealed by the analysis that PWWS emerges as the most potent adversary, consistently outperforming other methods across multiple evaluation scenarios, thereby emphasizing its efficacy in generating adversarial examples for text classification. Through comprehensive experimentation, the performance of these attacks is assessed and the findings indicate that the PWWS attack outperforms others, demonstrating lower runtime, higher accuracy, and favorable semantic similarity scores. The key insight of this paper lies in the assessment of the relative performances of three prevalent state-of-the-art attack mechanisms.

Title: Linguistic Changes in Spontaneous Speech for Detecting Parkinsons Disease Using Large Language Models

• Authors: Jonathan Crawford
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.05160
• Pdf URL: https://arxiv.org/pdf/2404.05160
• Copy Paste: [[2404.05160]] Linguistic Changes in Spontaneous Speech for Detecting Parkinsons Disease Using Large Language Models(https://arxiv.org/abs/2404.05160)
• Keywords: generative, large language model
• Abstract: Parkinsons disease is the second most prevalent neurodegenerative disorder with over ten million active cases worldwide and one million new diagnoses per year. Detecting and subsequently diagnosing the disease is challenging because of symptom heterogeneity with respect to complexity, as well as the type and timing of phenotypic manifestations. Typically, language impairment can present in the prodromal phase and precede motor symptoms suggesting that a linguistic-based approach could serve as a diagnostic method for incipient Parkinsons disease. Additionally, improved linguistic models may enhance other approaches through ensemble techniques. The field of large language models is advancing rapidly, presenting the opportunity to explore the use of these new models for detecting Parkinsons disease and to improve on current linguistic approaches with high-dimensional representations of linguistics. We evaluate the application of state-of-the-art large language models to detect Parkinsons disease automatically from spontaneous speech with up to 73% accuracy.

Title: QMix: Quality-aware Learning with Mixed Noise for Robust Retinal Disease Diagnosis

• Authors: Junlin Hou, Jilan Xu, Rui Feng, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05169
• Pdf URL: https://arxiv.org/pdf/2404.05169
• Copy Paste: [[2404.05169]] QMix: Quality-aware Learning with Mixed Noise for Robust Retinal Disease Diagnosis(https://arxiv.org/abs/2404.05169)
• Keywords: robust, noise learning
• Abstract: Due to the complexity of medical image acquisition and the difficulty of annotation, medical image datasets inevitably contain noise. Noisy data with wrong labels affects the robustness and generalization ability of deep neural networks. Previous noise learning methods mainly considered noise arising from images being mislabeled, i.e. label noise, assuming that all mislabeled images are of high image quality. However, medical images are prone to suffering extreme quality issues, i.e. data noise, where discriminative visual features are missing for disease diagnosis. In this paper, we propose a noise learning framework, termed as QMix, that learns a robust disease diagnosis model under mixed noise. QMix alternates between sample separation and quality-aware semisupervised training in each training epoch. In the sample separation phase, we design a joint uncertainty-loss criterion to effectively separate (1) correctly labeled images; (2) mislabeled images with high quality and (3) mislabeled images with low quality. In the semi-supervised training phase, we train a disease diagnosis model to learn robust feature representation from the separated samples. Specifically, we devise a sample-reweighing loss to mitigate the effect of mislabeled images with low quality during training. Meanwhile, a contrastive enhancement loss is proposed to further distinguish mislabeled images with low quality from correctly labeled images. QMix achieved state-of-the-art disease diagnosis performance on five public retinal image datasets and exhibited substantial improvement on robustness against mixed noise.

Title: GloSoFarID: Global multispectral dataset for Solar Farm IDentification in satellite imagery

• Authors: Zhiyuan Yang, Ryan Rad
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05180
• Pdf URL: https://arxiv.org/pdf/2404.05180
• Copy Paste: [[2404.05180]] GloSoFarID: Global multispectral dataset for Solar Farm IDentification in satellite imagery(https://arxiv.org/abs/2404.05180)
• Keywords: robust
• Abstract: Solar Photovoltaic (PV) technology is increasingly recognized as a pivotal solution in the global pursuit of clean and renewable energy. This technology addresses the urgent need for sustainable energy alternatives by converting solar power into electricity without greenhouse gas emissions. It not only curtails global carbon emissions but also reduces reliance on finite, non-renewable energy sources. In this context, monitoring solar panel farms becomes essential for understanding and facilitating the worldwide shift toward clean energy. This study contributes to this effort by developing the first comprehensive global dataset of multispectral satellite imagery of solar panel farms. This dataset is intended to form the basis for training robust machine learning models, which can accurately map and analyze the expansion and distribution of solar panel farms globally. The insights gained from this endeavor will be instrumental in guiding informed decision-making for a sustainable energy future. https://github.com/yzyly1992/GloSoFarID

Title: DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model

• Authors: Chao Gao, Sai Qian Zhang
• Subjects: cs.LG, cs.AI, cs.CL, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.05182
• Pdf URL: https://arxiv.org/pdf/2404.05182
• Copy Paste: [[2404.05182]] DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model(https://arxiv.org/abs/2404.05182)
• Keywords: privacy, protect, large language model
• Abstract: To enhance the performance of large language models (LLM) on downstream tasks, one solution is to fine-tune certain LLM parameters and make it better align with the characteristics of the training dataset. This process is commonly known as parameter-efficient fine-tuning (PEFT). Due to the scale of LLM, PEFT operations are usually executed in the public environment (e.g., cloud server). This necessitates the sharing of sensitive user data across public environments, thereby raising potential privacy concerns. To tackle these challenges, we propose a distributed PEFT framework called DLoRA. DLoRA enables scalable PEFT operations to be performed collaboratively between the cloud and user devices. Coupled with the proposed Kill and Revive algorithm, the evaluation results demonstrate that DLoRA can significantly reduce the computation and communication workload over the user devices while achieving superior accuracy and privacy protection.

Title: Have You Merged My Model? On The Robustness of Large Language Model IP Protection Methods Against Model Merging

• Authors: Tianshuo Cong, Delong Ran, Zesen Liu, Xinlei He, Jinyuan Liu, Yichen Gong, Qi Li, Anyu Wang, Xiaoyun Wang
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05188
• Pdf URL: https://arxiv.org/pdf/2404.05188
• Copy Paste: [[2404.05188]] Have You Merged My Model? On The Robustness of Large Language Model IP Protection Methods Against Model Merging(https://arxiv.org/abs/2404.05188)
• Keywords: protect, robust, watermark, large language model
• Abstract: Model merging is a promising lightweight model empowerment technique that does not rely on expensive computing devices (e.g., GPUs) or require the collection of specific training data. Instead, it involves editing different upstream model parameters to absorb their downstream task capabilities. However, uncertified model merging can infringe upon the Intellectual Property (IP) rights of the original upstream models. In this paper, we conduct the first study on the robustness of IP protection methods in model merging scenarios. We investigate two state-of-the-art IP protection techniques: Quantization Watermarking and Instructional Fingerprint, along with various advanced model merging technologies, such as Task Arithmetic, TIES-MERGING, and so on. Experimental results indicate that current Large Language Model (LLM) watermarking techniques cannot survive in the merged models, whereas model fingerprinting techniques can. Our research aims to highlight that model merging should be an indispensable consideration in the robustness assessment of model IP protection techniques, thereby promoting the healthy development of the open-source LLM community.

Title: HSViT: Horizontally Scalable Vision Transformer

• Authors: Chenhao Xu, Chang-Tsun Li, Chee Peng Lim, Douglas Creighton
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05196
• Pdf URL: https://arxiv.org/pdf/2404.05196
• Copy Paste: [[2404.05196]] HSViT: Horizontally Scalable Vision Transformer(https://arxiv.org/abs/2404.05196)
• Keywords: transformer
• Abstract: While the Vision Transformer (ViT) architecture gains prominence in computer vision and attracts significant attention from multimedia communities, its deficiency in prior knowledge (inductive bias) regarding shift, scale, and rotational invariance necessitates pre-training on large-scale datasets. Furthermore, the growing layers and parameters in both ViT and convolutional neural networks (CNNs) impede their applicability to mobile multimedia services, primarily owing to the constrained computational resources on edge devices. To mitigate the aforementioned challenges, this paper introduces a novel horizontally scalable vision transformer (HSViT). Specifically, a novel image-level feature embedding allows ViT to better leverage the inductive bias inherent in the convolutional layers. Based on this, an innovative horizontally scalable architecture is designed, which reduces the number of layers and parameters of the models while facilitating collaborative training and inference of ViT models across multiple nodes. The experimental results depict that, without pre-training on large-scale datasets, HSViT achieves up to 10% higher top-1 accuracy than state-of-the-art schemes, ascertaining its superior preservation of inductive bias. The code is available at https://github.com/xuchenhao001/HSViT.

Title: A secure and private ensemble matcher using multi-vault obfuscated templates

• Authors: Babak Poorebrahim Gilkalaye, Shubhabrata Mukherjee, Reza Derakhshani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05205
• Pdf URL: https://arxiv.org/pdf/2404.05205
• Copy Paste: [[2404.05205]] A secure and private ensemble matcher using multi-vault obfuscated templates(https://arxiv.org/abs/2404.05205)
• Keywords: secure, security, privacy, protect, biometric, generative
• Abstract: Given the irrevocability of biometric samples and mounting privacy concerns, biometric template security and secure matching are among the essential features of any well-designed modern biometric system. In this paper, we propose an obfuscation method that hides the biometric template information with just enough chaff. The main idea is to reduce the number of chaff points to a practical level by creating n sub-templates from the original template and hiding each sub-template with m chaff points. During verification, s closest vectors to the biometric query are retrieved from each vault and then combined to generate hash values that are compared with the stored hash value. We demonstrate the effectiveness of synthetic facial images, generated by a Generative Adversarial Network (GAN), as ``random chaff points'' within a secure-vault authorization system. This approach safeguards user identities during training and deployment. We tested our protocol using the AT&T, GT, and LFW face datasets, with the ROC areas under the curve being 0.99, 0.99, and 0.90, respectively. These numbers were close to those of the unprotected templates, showing that our method does not adversely affect accuracy.

Title: iVPT: Improving Task-relevant Information Sharing in Visual Prompt Tuning by Cross-layer Dynamic Connection

• Authors: Nan Zhou, Jiaxin Chen, Di Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05207
• Pdf URL: https://arxiv.org/pdf/2404.05207
• Copy Paste: [[2404.05207]] iVPT: Improving Task-relevant Information Sharing in Visual Prompt Tuning by Cross-layer Dynamic Connection(https://arxiv.org/abs/2404.05207)
• Keywords: transformer, segmentation
• Abstract: Recent progress has shown great potential of visual prompt tuning (VPT) when adapting pre-trained vision transformers to various downstream tasks. However, most existing solutions independently optimize prompts at each layer, thereby neglecting the usage of task-relevant information encoded in prompt tokens across layers. Additionally, existing prompt structures are prone to interference from task-irrelevant noise in input images, which can do harm to the sharing of task-relevant information. In this paper, we propose a novel VPT approach, \textbf{iVPT}. It innovatively incorporates a cross-layer dynamic connection (CDC) for input prompt tokens from adjacent layers, enabling effective sharing of task-relevant information. Furthermore, we design a dynamic aggregation (DA) module that facilitates selective sharing of information between layers. The combination of CDC and DA enhances the flexibility of the attention process within the VPT framework. Building upon these foundations, iVPT introduces an attentive reinforcement (AR) mechanism, by automatically identifying salient image tokens, which are further enhanced by prompt tokens in an additive manner. Extensive experiments on 24 image classification and semantic segmentation benchmarks clearly demonstrate the advantage of the proposed iVPT, compared to the state-of-the-art counterparts.

Title: Bidirectional Long-Range Parser for Sequential Data Understanding

• Authors: George Leotescu, Daniel Voinea, Alin-Ionut Popa
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05210
• Pdf URL: https://arxiv.org/pdf/2404.05210
• Copy Paste: [[2404.05210]] Bidirectional Long-Range Parser for Sequential Data Understanding(https://arxiv.org/abs/2404.05210)
• Keywords: transformer
• Abstract: The transformer is a powerful data modelling framework responsible for remarkable performance on a wide range of tasks. However, they are limited in terms of scalability as it is suboptimal and inefficient to process long-sequence data. To this purpose we introduce BLRP (Bidirectional Long-Range Parser), a novel and versatile attention mechanism designed to increase performance and efficiency on long-sequence tasks. It leverages short and long range heuristics in the form of a local sliding window approach combined with a global bidirectional latent space synthesis technique. We show the benefits and versatility of our approach on vision and language domains by demonstrating competitive results against state-of-the-art methods on the Long-Range-Arena and CIFAR benchmarks together with ablations demonstrating the computational efficiency.

Title: Multi-level Graph Subspace Contrastive Learning for Hyperspectral Image Clustering

• Authors: Jingxin Wang, Renxiang Guan, Kainan Gao, Zihao Li, Hao Li, Xianju Li, Chang Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05211
• Pdf URL: https://arxiv.org/pdf/2404.05211
• Copy Paste: [[2404.05211]] Multi-level Graph Subspace Contrastive Learning for Hyperspectral Image Clustering(https://arxiv.org/abs/2404.05211)
• Keywords: robust
• Abstract: Hyperspectral image (HSI) clustering is a challenging task due to its high complexity. Despite subspace clustering shows impressive performance for HSI, traditional methods tend to ignore the global-local interaction in HSI data. In this study, we proposed a multi-level graph subspace contrastive learning (MLGSC) for HSI clustering. The model is divided into the following main parts. Graph convolution subspace construction: utilizing spectral and texture feautures to construct two graph convolution views. Local-global graph representation: local graph representations were obtained by step-by-step convolutions and a more representative global graph representation was obtained using an attention-based pooling strategy. Multi-level graph subspace contrastive learning: multi-level contrastive learning was conducted to obtain local-global joint graph representations, to improve the consistency of the positive samples between views, and to obtain more robust graph embeddings. Specifically, graph-level contrastive learning is used to better learn global representations of HSI data. Node-level intra-view and inter-view contrastive learning is designed to learn joint representations of local regions of HSI. The proposed model is evaluated on four popular HSI datasets: Indian Pines, Pavia University, Houston, and Xu Zhou. The overall accuracies are 97.75%, 99.96%, 92.28%, and 95.73%, which significantly outperforms the current state-of-the-art clustering methods.

Title: DiffCJK: Conditional Diffusion Model for High-Quality and Wide-coverage CJK Character Generation

• Authors: Yingtao Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05212
• Pdf URL: https://arxiv.org/pdf/2404.05212
• Copy Paste: [[2404.05212]] DiffCJK: Conditional Diffusion Model for High-Quality and Wide-coverage CJK Character Generation(https://arxiv.org/abs/2404.05212)
• Keywords: diffusion, generative
• Abstract: Chinese, Japanese, and Korean (CJK), with a vast number of native speakers, has profound influence on society and culture. The typesetting of CJK languages carries a wide range of requirements due to the complexity of their scripts and unique literary traditions. A critical aspect of this typesetting process is that CJK fonts need to provide a set of consistent-looking glyphs for approximately one hundred thousand characters. However, creating such a font is inherently labor-intensive and expensive, which significantly hampers the development of new CJK fonts for typesetting, historical, aesthetic, or artistic purposes. To bridge this gap, we are motivated by recent advancements in diffusion-based generative models and propose a novel diffusion method for generating glyphs in a targeted style from a \emph{single} conditioned, standard glyph form. Our experiments show that our method is capable of generating fonts of both printed and hand-written styles, the latter of which presents a greater challenge. Moreover, our approach shows remarkable zero-shot generalization capabilities for non-CJK but Chinese-inspired scripts. We also show our method facilitates smooth style interpolation and generates bitmap images suitable for vectorization, which is crucial in the font creation process. In summary, our proposed method opens the door to high-quality, generative model-assisted font creation for CJK characters, for both typesetting and artistic endeavors.

Title: Out-of-Distribution Data: An Acquaintance of Adversarial Examples -- A Survey

• Authors: Naveen Karunanayake, Ravin Gunawardena, Suranga Seneviratne, Sanjay Chawla
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05219
• Pdf URL: https://arxiv.org/pdf/2404.05219
• Copy Paste: [[2404.05219]] Out-of-Distribution Data: An Acquaintance of Adversarial Examples -- A Survey(https://arxiv.org/abs/2404.05219)
• Keywords: attack, robust
• Abstract: Deep neural networks (DNNs) deployed in real-world applications can encounter out-of-distribution (OOD) data and adversarial examples. These represent distinct forms of distributional shifts that can significantly impact DNNs' reliability and robustness. Traditionally, research has addressed OOD detection and adversarial robustness as separate challenges. This survey focuses on the intersection of these two areas, examining how the research community has investigated them together. Consequently, we identify two key research directions: robust OOD detection and unified robustness. Robust OOD detection aims to differentiate between in-distribution (ID) data and OOD data, even when they are adversarially manipulated to deceive the OOD detector. Unified robustness seeks a single approach to make DNNs robust against both adversarial attacks and OOD inputs. Accordingly, first, we establish a taxonomy based on the concept of distributional shifts. This framework clarifies how robust OOD detection and unified robustness relate to other research areas addressing distributional shifts, such as OOD detection, open set recognition, and anomaly detection. Subsequently, we review existing work on robust OOD detection and unified robustness. Finally, we highlight the limitations of the existing work and propose promising research directions that explore adversarial and OOD inputs within a unified framework.

Title: LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models

• Authors: Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, Zhiting Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05221
• Pdf URL: https://arxiv.org/pdf/2404.05221
• Copy Paste: [[2404.05221]] LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models(https://arxiv.org/abs/2404.05221)
• Keywords: robust, interpretability, large language model
• Abstract: Generating accurate step-by-step reasoning is essential for Large Language Models (LLMs) to address complex problems and enhance robustness and interpretability. Despite the flux of research on developing advanced reasoning approaches, systematically analyzing the diverse LLMs and reasoning strategies in generating reasoning chains remains a significant challenge. The difficulties stem from the lack of two key elements: (1) an automatic method for evaluating the generated reasoning chains on different tasks, and (2) a unified formalism and implementation of the diverse reasoning approaches for systematic comparison. This paper aims to close the gap: (1) We introduce AutoRace for fully automated reasoning chain evaluation. Existing metrics rely on expensive human annotations or pre-defined LLM prompts not adaptable to different tasks. In contrast, AutoRace automatically creates detailed evaluation criteria tailored for each task, and uses GPT-4 for accurate evaluation following the criteria. (2) We develop LLM Reasoners, a library for standardized modular implementation of existing and new reasoning algorithms, under a unified formulation of the search, reward, and world model components. With the new evaluation and library, (3) we conduct extensive study of different reasoning approaches (e.g., CoT, ToT, RAP). The analysis reveals interesting findings about different factors contributing to reasoning, including the reward-guidance, breadth-vs-depth in search, world model, and prompt formats, etc.

Title: LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding

• Authors: Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, Cong Yao
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05225
• Pdf URL: https://arxiv.org/pdf/2404.05225
• Copy Paste: [[2404.05225]] LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding(https://arxiv.org/abs/2404.05225)
• Keywords: interpretability, large language model
• Abstract: Recently, leveraging large language models (LLMs) or multimodal large language models (MLLMs) for document understanding has been proven very promising. However, previous works that employ LLMs/MLLMs for document understanding have not fully explored and utilized the document layout information, which is vital for precise document understanding. In this paper, we propose LayoutLLM, an LLM/MLLM based method for document understanding. The core of LayoutLLM is a layout instruction tuning strategy, which is specially designed to enhance the comprehension and utilization of document layouts. The proposed layout instruction tuning strategy consists of two components: Layout-aware Pre-training and Layout-aware Supervised Fine-tuning. To capture the characteristics of document layout in Layout-aware Pre-training, three groups of pre-training tasks, corresponding to document-level, region-level and segment-level information, are introduced. Furthermore, a novel module called layout chain-of-thought (LayoutCoT) is devised to enable LayoutLLM to focus on regions relevant to the question and generate accurate answers. LayoutCoT is effective for boosting the performance of document understanding. Meanwhile, it brings a certain degree of interpretability, which could facilitate manual inspection and correction. Experiments on standard benchmarks show that the proposed LayoutLLM significantly outperforms existing methods that adopt open-source 7B LLMs/MLLMs for document understanding. The training data of the LayoutLLM is publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/LayoutLLM

Title: Empirical Upscaling of Point-scale Soil Moisture Measurements for Spatial Evaluation of Model Simulations and Satellite Retrievals

• Authors: Yi Yu, Brendan P. Malone, Luigi J. Renzullo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05229
• Pdf URL: https://arxiv.org/pdf/2404.05229
• Copy Paste: [[2404.05229]] Empirical Upscaling of Point-scale Soil Moisture Measurements for Spatial Evaluation of Model Simulations and Satellite Retrievals(https://arxiv.org/abs/2404.05229)
• Keywords: fair
• Abstract: The evaluation of modelled or satellite-derived soil moisture (SM) estimates is usually dependent on comparisons against in-situ SM measurements. However, the inherent mismatch in spatial support (i.e., scale) necessitates a cautious interpretation of point-to-pixel comparisons. The upscaling of the in-situ measurements to a commensurate resolution to that of the modelled or retrieved SM will lead to a fairer comparison and statistically more defensible evaluation. In this study, we presented an upscaling approach that combines spatiotemporal fusion with machine learning to extrapolate point-scale SM measurements from 28 in-situ sites to a 100 m resolution for an agricultural area of 100 km by 100 km. We conducted a four-fold cross-validation, which consistently demonstrated comparable correlation performance across folds, ranging from 0.6 to 0.9. The proposed approach was further validated based on a cross-cluster strategy by using two spatial subsets within the study area, denoted as cluster A and B, each of which equally comprised of 12 in-situ sites. The cross-cluster validation underscored the capability of the upscaling approach to map the spatial variability of SM within areas that were not covered by in-situ sites, with correlation performance ranging between 0.6 and 0.8. In general, our proposed upscaling approach offers an avenue to extrapolate point measurements of SM to a spatial scale more akin to climatic model grids or remotely sensed observations. Future investigations should delve into a further evaluation of the upscaling approach using independent data, such as model simulations, satellite retrievals or field campaign data.

Title: Supervised Gradual Machine Learning for Aspect Category Detection

• Authors: Murtadha Ahmed, Qun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05245
• Pdf URL: https://arxiv.org/pdf/2404.05245
• Copy Paste: [[2404.05245]] Supervised Gradual Machine Learning for Aspect Category Detection(https://arxiv.org/abs/2404.05245)
• Keywords: extraction
• Abstract: Aspect Category Detection (ACD) aims to identify implicit and explicit aspects in a given review sentence. The state-of-the-art approaches for ACD use Deep Neural Networks (DNNs) to address the problem as a multi-label classification task. However, learning category-specific representations heavily rely on the amount of labeled examples, which may not readily available in real-world scenarios. In this paper, we propose a novel approach to tackle the ACD task by combining DNNs with Gradual Machine Learning (GML) in a supervised setting. we aim to leverage the strength of DNN in semantic relation modeling, which can facilitate effective knowledge transfer between labeled and unlabeled instances during the gradual inference of GML. To achieve this, we first analyze the learned latent space of the DNN to model the relations, i.e., similar or opposite, between instances. We then represent these relations as binary features in a factor graph to efficiently convey knowledge. Finally, we conduct a comparative study of our proposed solution on real benchmark datasets and demonstrate that the GML approach, in collaboration with DNNs for feature extraction, consistently outperforms pure DNN solutions.

Title: CodeEnhance: A Codebook-Driven Approach for Low-Light Image Enhancement

• Authors: Xu Wu, XianXu Hou, Zhihui Lai, Jie Zhou, Ya-nan Zhang, Witold Pedrycz, Linlin Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05253
• Pdf URL: https://arxiv.org/pdf/2404.05253
• Copy Paste: [[2404.05253]] CodeEnhance: A Codebook-Driven Approach for Low-Light Image Enhancement(https://arxiv.org/abs/2404.05253)
• Keywords: robust
• Abstract: Low-light image enhancement (LLIE) aims to improve low-illumination images. However, existing methods face two challenges: (1) uncertainty in restoration from diverse brightness degradations; (2) loss of texture and color information caused by noise suppression and light enhancement. In this paper, we propose a novel enhancement approach, CodeEnhance, by leveraging quantized priors and image refinement to address these challenges. In particular, we reframe LLIE as learning an image-to-code mapping from low-light images to discrete codebook, which has been learned from high-quality images. To enhance this process, a Semantic Embedding Module (SEM) is introduced to integrate semantic information with low-level features, and a Codebook Shift (CS) mechanism, designed to adapt the pre-learned codebook to better suit the distinct characteristics of our low-light dataset. Additionally, we present an Interactive Feature Transformation (IFT) module to refine texture and color information during image reconstruction, allowing for interactive enhancement based on user preferences. Extensive experiments on both real-world and synthetic benchmarks demonstrate that the incorporation of prior knowledge and controllable information transfer significantly enhances LLIE performance in terms of quality and fidelity. The proposed CodeEnhance exhibits superior robustness to various degradations, including uneven illumination, noise, and color distortion.

Title: Text-to-Image Synthesis for Any Artistic Styles: Advancements in Personalized Artistic Image Generation via Subdivision and Dual Binding

• Authors: Junseo Park, Beomseok Ko, Hyeryung Jang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05256
• Pdf URL: https://arxiv.org/pdf/2404.05256
• Copy Paste: [[2404.05256]] Text-to-Image Synthesis for Any Artistic Styles: Advancements in Personalized Artistic Image Generation via Subdivision and Dual Binding(https://arxiv.org/abs/2404.05256)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-image models, such as Stable Diffusion, have demonstrated their ability to synthesize visual images through natural language prompts. One approach of personalizing text-to-image models, exemplified by DreamBooth, fine-tunes the pre-trained model by binding unique text identifiers with a few images of a specific subject. Although existing fine-tuning methods have demonstrated competence in rendering images according to the styles of famous painters, it is still challenging to learn to produce images encapsulating distinct art styles due to abstract and broad visual perceptions of stylistic attributes such as lines, shapes, textures, and colors. In this paper, we introduce a new method, Single-StyleForge, for personalization. It fine-tunes pre-trained text-to-image diffusion models to generate diverse images in specified styles from text prompts. By using around 15-20 images of the target style, the approach establishes a foundational binding of a unique token identifier with a broad range of the target style. It also utilizes auxiliary images to strengthen this binding, resulting in offering specific guidance on representing elements such as persons in a target style-consistent manner. In addition, we present ways to improve the quality of style and text-image alignment through a method called Multi-StyleForge, which inherits the strategy used in StyleForge and learns tokens in multiple. Experimental evaluation conducted on six distinct artistic styles demonstrates substantial improvements in both the quality of generated images and the perceptual fidelity metrics, such as FID, KID, and CLIP scores.

Title: Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Security

• Authors: Yihe Fan, Yuxin Cao, Ziyu Zhao, Ziyao Liu, Shaofeng Li
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05264
• Pdf URL: https://arxiv.org/pdf/2404.05264
• Copy Paste: [[2404.05264]] Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Security(https://arxiv.org/abs/2404.05264)
• Keywords: security, defense, attack, large language model
• Abstract: Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities that increasingly influence various aspects of our daily lives, constantly defining the new boundary of Artificial General Intelligence (AGI). Image modalities, enriched with profound semantic information and a more continuous mathematical nature compared to other modalities, greatly enhance the functionalities of MLLMs when integrated. However, this integration serves as a double-edged sword, providing attackers with expansive vulnerabilities to exploit for highly covert and harmful attacks. The pursuit of reliable AI systems like powerful MLLMs has emerged as a pivotal area of contemporary research. In this paper, we endeavor to demostrate the multifaceted risks associated with the incorporation of image modalities into MLLMs. Initially, we delineate the foundational components and training processes of MLLMs. Subsequently, we construct a threat model, outlining the security vulnerabilities intrinsic to MLLMs. Moreover, we analyze and summarize existing scholarly discourses on MLLMs' attack and defense mechanisms, culminating in suggestions for the future research on MLLM security. Through this comprehensive analysis, we aim to deepen the academic understanding of MLLM security challenges and propel forward the development of trustworthy MLLM systems.

Title: Deep Optics for Video Snapshot Compressive Imaging

• Authors: Ping Wang, Lishun Wang, Xin Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05274
• Pdf URL: https://arxiv.org/pdf/2404.05274
• Copy Paste: [[2404.05274]] Deep Optics for Video Snapshot Compressive Imaging(https://arxiv.org/abs/2404.05274)
• Keywords: transformer
• Abstract: Video snapshot compressive imaging (SCI) aims to capture a sequence of video frames with only a single shot of a 2D detector, whose backbones rest in optical modulation patterns (also known as masks) and a computational reconstruction algorithm. Advanced deep learning algorithms and mature hardware are putting video SCI into practical applications. Yet, there are two clouds in the sunshine of SCI: i) low dynamic range as a victim of high temporal multiplexing, and ii) existing deep learning algorithms' degradation on real system. To address these challenges, this paper presents a deep optics framework to jointly optimize masks and a reconstruction network. Specifically, we first propose a new type of structural mask to realize motion-aware and full-dynamic-range measurement. Considering the motion awareness property in measurement domain, we develop an efficient network for video SCI reconstruction using Transformer to capture long-term temporal dependencies, dubbed Res2former. Moreover, sensor response is introduced into the forward model of video SCI to guarantee end-to-end model training close to real system. Finally, we implement the learned structural masks on a digital micro-mirror device. Experimental results on synthetic and real data validate the effectiveness of the proposed framework. We believe this is a milestone for real-world video SCI. The source code and data are available at https://github.com/pwangcs/DeepOpticsSCI.

Title: MOSE: Boosting Vision-based Roadside 3D Object Detection with Scene Cues

• Authors: Xiahan Chen, Mingjian Chen, Sanli Tang, Yi Niu, Jiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05280
• Pdf URL: https://arxiv.org/pdf/2404.05280
• Copy Paste: [[2404.05280]] MOSE: Boosting Vision-based Roadside 3D Object Detection with Scene Cues(https://arxiv.org/abs/2404.05280)
• Keywords: transformer
• Abstract: 3D object detection based on roadside cameras is an additional way for autonomous driving to alleviate the challenges of occlusion and short perception range from vehicle cameras. Previous methods for roadside 3D object detection mainly focus on modeling the depth or height of objects, neglecting the stationary of cameras and the characteristic of inter-frame consistency. In this work, we propose a novel framework, namely MOSE, for MOnocular 3D object detection with Scene cuEs. The scene cues are the frame-invariant scene-specific features, which are crucial for object localization and can be intuitively regarded as the height between the surface of the real road and the virtual ground plane. In the proposed framework, a scene cue bank is designed to aggregate scene cues from multiple frames of the same scene with a carefully designed extrinsic augmentation strategy. Then, a transformer-based decoder lifts the aggregated scene cues as well as the 3D position embeddings for 3D object location, which boosts generalization ability in heterologous scenes. The extensive experiment results on two public benchmarks demonstrate the state-of-the-art performance of the proposed method, which surpasses the existing methods by a large margin.

Title: Multi-Task Learning for Features Extraction in Financial Annual Reports

• Authors: Syrielle Montariol, Matej Martinc, Andraž Pelicon, Senja Pollak, Boshko Koloski, Igor Lončarski, Aljoša Valentinčič
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05281
• Pdf URL: https://arxiv.org/pdf/2404.05281
• Copy Paste: [[2404.05281]] Multi-Task Learning for Features Extraction in Financial Annual Reports(https://arxiv.org/abs/2404.05281)
• Keywords: extraction
• Abstract: For assessing various performance indicators of companies, the focus is shifting from strictly financial (quantitative) publicly disclosed information to qualitative (textual) information. This textual data can provide valuable weak signals, for example through stylistic features, which can complement the quantitative data on financial performance or on Environmental, Social and Governance (ESG) criteria. In this work, we use various multi-task learning methods for financial text classification with the focus on financial sentiment, objectivity, forward-looking sentence prediction and ESG-content detection. We propose different methods to combine the information extracted from training jointly on different tasks; our best-performing method highlights the positive effect of explicitly adding auxiliary task predictions as features for the final target task during the multi-task training. Next, we use these classifiers to extract textual features from annual reports of FTSE350 companies and investigate the link between ESG quantitative scores and these features.

Title: Detecting Every Object from Events

• Authors: Haitian Zhang, Chang Xu, Xinya Wang, Bingde Liu, Guang Hua, Lei Yu, Wen Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05285
• Pdf URL: https://arxiv.org/pdf/2404.05285
• Copy Paste: [[2404.05285]] Detecting Every Object from Events(https://arxiv.org/abs/2404.05285)
• Keywords: robust, transformer
• Abstract: Object detection is critical in autonomous driving, and it is more practical yet challenging to localize objects of unknown categories: an endeavour known as Class-Agnostic Object Detection (CAOD). Existing studies on CAOD predominantly rely on ordinary cameras, but these frame-based sensors usually have high latency and limited dynamic range, leading to safety risks in real-world scenarios. In this study, we turn to a new modality enabled by the so-called event camera, featured by its sub-millisecond latency and high dynamic range, for robust CAOD. We propose Detecting Every Object in Events (DEOE), an approach tailored for achieving high-speed, class-agnostic open-world object detection in event-based vision. Built upon the fast event-based backbone: recurrent vision transformer, we jointly consider the spatial and temporal consistencies to identify potential objects. The discovered potential objects are assimilated as soft positive samples to avoid being suppressed as background. Moreover, we introduce a disentangled objectness head to separate the foreground-background classification and novel object discovery tasks, enhancing the model's generalization in localizing novel objects while maintaining a strong ability to filter out the background. Extensive experiments confirm the superiority of our proposed DEOE in comparison with three strong baseline methods that integrate the state-of-the-art event-based object detector with advancements in RGB-based CAOD. Our code is available at https://github.com/Hatins/DEOE.

Title: Automated Attack Synthesis for Constant Product Market Makers

• Authors: Sujin Han, Jinseo Kim, Sung-Ju Lee, Insu Yun
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.05297
• Pdf URL: https://arxiv.org/pdf/2404.05297
• Copy Paste: [[2404.05297]] Automated Attack Synthesis for Constant Product Market Makers(https://arxiv.org/abs/2404.05297)
• Keywords: attack
• Abstract: Decentralized Finance enables many novel applications that were impossible in traditional finances. However, it also introduces new types of vulnerabilities, such as composability bugs. The composability bugs refer to issues that lead to erroneous behaviors when multiple smart contracts operate together. One typical example of composability bugs is those between token contracts and Constant Product Market Makers (CPMM), the most widely used model for Decentralized Exchanges. Since 2022, 23 exploits of such kind have resulted in a total loss of 2.2M USD. BlockSec, a smart contract auditing company, once reported that 138 exploits of such kind occurred just in February 2023. We propose CPMM-Exploiter, which automatically detects and generates end-to-end exploits for CPMM composability bugs. Generating such end-to-end exploits is challenging due to the large search space of multiple contracts and various fees involved with financial services. To tackle this, we investigated real-world exploits regarding these vulnerabilities and identified that they arise due to violating two safety invariants. Based on this observation, we implemented CPMM-Exploiter, a new grammar-based fuzzer targeting the detection of these bugs. CPMM-Exploiter uses fuzzing to find transactions that break the invariants. It then refines these transactions to make them profitable for the attacker. We evaluated CPMM-Exploiter on two real-world exploit datasets. CPMM-Exploiter obtained recalls of 0.91 and 0.89, respectively, while five baselines achieved maximum recalls of 0.36 and 0.58, respectively. We further evaluated CPMM-Exploiter by running it on the latest blocks of the Ethereum and Binance networks. It successfully generated 18 new exploits, which can result in 12.9K USD profit in total.

Title: Texture Classification Network Integrating Adaptive Wavelet Transform

• Authors: Su-Xi Yu, Jing-Yuan He, Yi Wang, Yu-Jiao Cai, Jun Yang, Bo Lin, Wei-Bin Yang, Jian Ruan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05300
• Pdf URL: https://arxiv.org/pdf/2404.05300
• Copy Paste: [[2404.05300]] Texture Classification Network Integrating Adaptive Wavelet Transform(https://arxiv.org/abs/2404.05300)
• Keywords: extraction
• Abstract: Graves' disease is a common condition that is diagnosed clinically by determining the smoothness of the thyroid texture and its morphology in ultrasound images. Currently, the most widely used approach for the automated diagnosis of Graves' disease utilizes Convolutional Neural Networks (CNNs) for both feature extraction and classification. However, these methods demonstrate limited efficacy in capturing texture features. Given the high capacity of wavelets in describing texture features, this research integrates learnable wavelet modules utilizing the Lifting Scheme into CNNs and incorporates a parallel wavelet branch into the ResNet18 model to enhance texture feature extraction. Our model can analyze texture features in spatial and frequency domains simultaneously, leading to optimized classification accuracy. We conducted experiments on collected ultrasound datasets and publicly available natural image texture datasets, our proposed network achieved 97.27% accuracy and 95.60% recall on ultrasound datasets, 60.765% accuracy on natural image texture datasets, surpassing the accuracy of ResNet and conrming the effectiveness of our approach.

Title: Human Detection from 4D Radar Data in Low-Visibility Field Conditions

• Authors: Mikael Skog, Oleksandr Kotlyar, Vladimír Kubelka, Martin Magnusson
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.05307
• Pdf URL: https://arxiv.org/pdf/2404.05307
• Copy Paste: [[2404.05307]] Human Detection from 4D Radar Data in Low-Visibility Field Conditions(https://arxiv.org/abs/2404.05307)
• Keywords: segmentation
• Abstract: Autonomous driving technology is increasingly being used on public roads and in industrial settings such as mines. While it is essential to detect pedestrians, vehicles, or other obstacles, adverse field conditions negatively affect the performance of classical sensors such as cameras or lidars. Radar, on the other hand, is a promising modality that is less affected by, e.g., dust, smoke, water mist or fog. In particular, modern 4D imaging radars provide target responses across the range, vertical angle, horizontal angle and Doppler velocity dimensions. We propose TMVA4D, a CNN architecture that leverages this 4D radar modality for semantic segmentation. The CNN is trained to distinguish between the background and person classes based on a series of 2D projections of the 4D radar data that include the elevation, azimuth, range, and Doppler velocity dimensions. We also outline the process of compiling a novel dataset consisting of data collected in industrial settings with a car-mounted 4D radar and describe how the ground-truth labels were generated from reference thermal images. Using TMVA4D on this dataset, we achieve an mIoU score of 78.2% and an mDice score of 86.1%, evaluated on the two classes background and person

Title: CLIPping the Limits: Finding the Sweet Spot for Relevant Images in Automated Driving Systems Perception Testing

• Authors: Philipp Rigoll, Laurenz Adolph, Lennart Ries, Eric Sax
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.05309
• Pdf URL: https://arxiv.org/pdf/2404.05309
• Copy Paste: [[2404.05309]] CLIPping the Limits: Finding the Sweet Spot for Relevant Images in Automated Driving Systems Perception Testing(https://arxiv.org/abs/2404.05309)
• Keywords: robust
• Abstract: Perception systems, especially cameras, are the eyes of automated driving systems. Ensuring that they function reliably and robustly is therefore an important building block in the automation of vehicles. There are various approaches to test the perception of automated driving systems. Ultimately, however, it always comes down to the investigation of the behavior of perception systems under specific input data. Camera images are a crucial part of the input data. Image data sets are therefore collected for the testing of automated driving systems, but it is non-trivial to find specific images in these data sets. Thanks to recent developments in neural networks, there are now methods for sorting the images in a data set according to their similarity to a prompt in natural language. In order to further automate the provision of search results, we make a contribution by automating the threshold definition in these sorted results and returning only the images relevant to the prompt as a result. Our focus is on preventing false positives and false negatives equally. It is also important that our method is robust and in the case that our assumptions are not fulfilled, we provide a fallback solution.

Title: BruSLeAttack: A Query-Efficient Score-Based Black-Box Sparse Adversarial Attack

• Authors: Viet Quoc Vo, Ehsan Abbasnejad, Damith C. Ranasinghe
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05311
• Pdf URL: https://arxiv.org/pdf/2404.05311
• Copy Paste: [[2404.05311]] BruSLeAttack: A Query-Efficient Score-Based Black-Box Sparse Adversarial Attack(https://arxiv.org/abs/2404.05311)
• Keywords: security, defense, attack, robust
• Abstract: We study the unique, less-well understood problem of generating sparse adversarial samples simply by observing the score-based replies to model queries. Sparse attacks aim to discover a minimum number-the l0 bounded-perturbations to model inputs to craft adversarial examples and misguide model decisions. But, in contrast to query-based dense attack counterparts against black-box models, constructing sparse adversarial perturbations, even when models serve confidence score information to queries in a score-based setting, is non-trivial. Because, such an attack leads to i) an NP-hard problem; and ii) a non-differentiable search space. We develop the BruSLeAttack-a new, faster (more query-efficient) Bayesian algorithm for the problem. We conduct extensive attack evaluations including an attack demonstration against a Machine Learning as a Service (MLaaS) offering exemplified by Google Cloud Vision and robustness testing of adversarial training regimes and a recent defense against black-box attacks. The proposed attack scales to achieve state-of-the-art attack success rates and query efficiency on standard computer vision tasks such as ImageNet across different model architectures. Our artefacts and DIY attack samples are available on GitHub. Importantly, our work facilitates faster evaluation of model vulnerabilities and raises our vigilance on the safety, security and reliability of deployed systems.

Title: Reflected Search Poisoning for Illicit Promotion

• Authors: Sangyi Wu, Jialong Xue, Shaoxuan Zhou, Xianghang Mi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05320
• Pdf URL: https://arxiv.org/pdf/2404.05320
• Copy Paste: [[2404.05320]] Reflected Search Poisoning for Illicit Promotion(https://arxiv.org/abs/2404.05320)
• Keywords: security, steal
• Abstract: As an emerging black hat search engine optimization (SEO) technique, reflected search poisoning (RSP) allows a miscreant to free-ride the reputation of high-ranking websites, poisoning search engines with illicit promotion texts (IPTs) in an efficient and stealthy manner, while avoiding the burden of continuous website compromise as required by traditional promotion infections. However, little is known about the security implications of RSP, e.g., what illicit promotion campaigns are being distributed by RSP, and to what extent regular search users can be exposed to illicit promotion texts distributed by RSP. In this study, we conduct the first security study on RSP-based illicit promotion, which is made possible through an end-to-end methodology for capturing, analyzing, and infiltrating IPTs. As a result, IPTs distributed via RSP are found to be large-scale, continuously growing, and diverse in both illicit categories and natural languages. Particularly, we have identified over 11 million distinct IPTs belonging to 14 different illicit categories, with typical examples including drug trading, data theft, counterfeit goods, and hacking services. Also, the underlying RSP cases have abused tens of thousands of high-ranking websites, as well as extensively poisoning all four popular search engines we studied, especially Google Search and Bing. Furthermore, it is observed that benign search users are being exposed to IPTs at a concerning extent. To facilitate interaction with potential customers (victim search users), miscreants tend to embed various types of contacts in IPTs, especially instant messaging accounts. Further infiltration of these IPT contacts reveals that the underlying illicit campaigns are operated on a large scale. All these findings highlight the negative security implications of IPTs and RSPs, and thus call for more efforts to mitigate RSP-driven illicit promotion.

Title: Mask-ControlNet: Higher-Quality Image Generation with An Additional Mask Prompt

• Authors: Zhiqi Huang, Huixin Xiong, Haoyu Wang, Longguang Wang, Zhiheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05331
• Pdf URL: https://arxiv.org/pdf/2404.05331
• Copy Paste: [[2404.05331]] Mask-ControlNet: Higher-Quality Image Generation with An Additional Mask Prompt(https://arxiv.org/abs/2404.05331)
• Keywords: diffusion
• Abstract: Text-to-image generation has witnessed great progress, especially with the recent advancements in diffusion models. Since texts cannot provide detailed conditions like object appearance, reference images are usually leveraged for the control of objects in the generated images. However, existing methods still suffer limited accuracy when the relationship between the foreground and background is complicated. To address this issue, we develop a framework termed Mask-ControlNet by introducing an additional mask prompt. Specifically, we first employ large vision models to obtain masks to segment the objects of interest in the reference image. Then, the object images are employed as additional prompts to facilitate the diffusion model to better understand the relationship between foreground and background regions during image generation. Experiments show that the mask prompts enhance the controllability of the diffusion model to maintain higher fidelity to the reference image while achieving better image quality. Comparison with previous text-to-image generation methods demonstrates our method's superior quantitative and qualitative performance on the benchmark datasets.

Title: Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level

• Authors: Chenxu Wang, Bin Dai, Huaping Liu, Baoyuan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05337
• Pdf URL: https://arxiv.org/pdf/2404.05337
• Copy Paste: [[2404.05337]] Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level(https://arxiv.org/abs/2404.05337)
• Keywords: large language model
• Abstract: Prominent large language models have exhibited human-level performance in many domains, even enabling the derived agents to simulate human and social interactions. While practical works have substantiated the practicability of grounding language agents in sandbox simulation or embodied simulators, current social intelligence benchmarks either stay at the language level or use subjective metrics. In pursuit of a more realistic and objective evaluation, we introduce the Social Tasks in Sandbox Simulation (STSS) benchmark, which assesses language agents \textbf{objectively} at the \textbf{action level} by scrutinizing the goal achievements within the multi-agent simulation. Additionally, we sample conversation scenarios to build a language-level benchmark to provide an economically prudent preliminary evaluation and align with prevailing benchmarks. To gauge the significance of agent architecture, we implement a target-driven planning (TDP) module as an adjunct to the existing agent. Our evaluative findings highlight that the STSS benchmark is challenging for state-of-the-art language agents. Furthermore, it effectively discriminates between distinct language agents, suggesting its usefulness as a benchmark for evaluating both language models and agent architectures.

Title: Certified PEFTSmoothing: Parameter-Efficient Fine-Tuning with Randomized Smoothing

• Authors: Chengyan Fu, Wenjie Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05350
• Pdf URL: https://arxiv.org/pdf/2404.05350
• Copy Paste: [[2404.05350]] Certified PEFTSmoothing: Parameter-Efficient Fine-Tuning with Randomized Smoothing(https://arxiv.org/abs/2404.05350)
• Keywords: robust, diffusion
• Abstract: Randomized smoothing is the primary certified robustness method for accessing the robustness of deep learning models to adversarial perturbations in the l2-norm, by adding isotropic Gaussian noise to the input image and returning the majority votes over the base classifier. Theoretically, it provides a certified norm bound, ensuring predictions of adversarial examples are stable within this bound. A notable constraint limiting widespread adoption is the necessity to retrain base models entirely from scratch to attain a robust version. This is because the base model fails to learn the noise-augmented data distribution to give an accurate vote. One intuitive way to overcome this challenge is to involve a custom-trained denoiser to eliminate the noise. However, this approach is inefficient and sub-optimal. Inspired by recent large model training procedures, we explore an alternative way named PEFTSmoothing to adapt the base model to learn the Gaussian noise-augmented data with Parameter-Efficient Fine-Tuning (PEFT) methods in both white-box and black-box settings. Extensive results demonstrate the effectiveness and efficiency of PEFTSmoothing, which allow us to certify over 98% accuracy for ViT on CIFAR-10, 20% higher than SoTA denoised smoothing, and over 61% accuracy on ImageNet which is 30% higher than CNN-based denoiser and comparable to the Diffusion-based denoiser.

Title: CNN-based Game State Detection for a Foosball Table

• Authors: David Hagens, Jan Knaup, Elke Hergenröther, Andreas Weinmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05357
• Pdf URL: https://arxiv.org/pdf/2404.05357
• Copy Paste: [[2404.05357]] CNN-based Game State Detection for a Foosball Table(https://arxiv.org/abs/2404.05357)
• Keywords: extraction
• Abstract: The automation of games using Deep Reinforcement Learning Strategies (DRL) is a well-known challenge in AI research. While for feature extraction in a video game typically the whole image is used, this is hardly practical for many real world games. Instead, using a smaller game state reducing the dimension of the parameter space to include essential parameters only seems to be a promising approach. In the game of Foosball, a compact and comprehensive game state description consists of the positional shifts and rotations of the figures and the position of the ball over time. In particular, velocities and accelerations can be derived from consecutive time samples of the game state. In this paper, a figure detection system to determine the game state in Foosball is presented. We capture a dataset containing the rotations of the rods which were measured using accelerometers and the positional shifts were derived using traditional Computer Vision techniques (in a laboratory setting). This dataset is utilized to train Convolutional Neural Network (CNN) based end-to-end regression models to predict the rotations and shifts of each rod. We present an evaluation of our system using different state-of-the-art CNNs as base architectures for the regression model. We show that our system is able to predict the game state with high accuracy. By providing data for both black and white teams, the presented system is intended to provide the required data for future developments of Imitation Learning techniques w.r.t. to observing human players.

Title: Multi-head Attention-based Deep Multiple Instance Learning

• Authors: Hassan Keshvarikhojasteh, Josien Pluim, Mitko Veta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05362
• Pdf URL: https://arxiv.org/pdf/2404.05362
• Copy Paste: [[2404.05362]] Multi-head Attention-based Deep Multiple Instance Learning(https://arxiv.org/abs/2404.05362)
• Keywords: interpretability, transformer
• Abstract: This paper introduces MAD-MIL, a Multi-head Attention-based Deep Multiple Instance Learning model, designed for weakly supervised Whole Slide Images (WSIs) classification in digital pathology. Inspired by the multi-head attention mechanism of the Transformer, MAD-MIL simplifies model complexity while achieving competitive results against advanced models like CLAM and DS-MIL. Evaluated on the MNIST-BAGS and public datasets, including TUPAC16, TCGA BRCA, TCGA LUNG, and TCGA KIDNEY, MAD-MIL consistently outperforms ABMIL. This demonstrates enhanced information diversity, interpretability, and efficiency in slide representation. The model's effectiveness, coupled with fewer trainable parameters and lower computational complexity makes it a promising solution for automated pathology workflows. Our code is available at https://github.com/tueimage/MAD-MIL.

Title: Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance

• Authors: Dazhong Shen, Guanglu Song, Zeyue Xue, Fu-Yun Wang, Yu Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05384
• Pdf URL: https://arxiv.org/pdf/2404.05384
• Copy Paste: [[2404.05384]] Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance(https://arxiv.org/abs/2404.05384)
• Keywords: diffusion, segmentation
• Abstract: Classifier-Free Guidance (CFG) has been widely used in text-to-image diffusion models, where the CFG scale is introduced to control the strength of text guidance on the whole image space. However, we argue that a global CFG scale results in spatial inconsistency on varying semantic strengths and suboptimal image quality. To address this problem, we present a novel approach, Semantic-aware Classifier-Free Guidance (S-CFG), to customize the guidance degrees for different semantic units in text-to-image diffusion models. Specifically, we first design a training-free semantic segmentation method to partition the latent image into relatively independent semantic regions at each denoising step. In particular, the cross-attention map in the denoising U-net backbone is renormalized for assigning each patch to the corresponding token, while the self-attention map is used to complete the semantic regions. Then, to balance the amplification of diverse semantic units, we adaptively adjust the CFG scales across different semantic regions to rescale the text guidance degrees into a uniform level. Finally, extensive experiments demonstrate the superiority of S-CFG over the original CFG strategy on various text-to-image diffusion models, without requiring any extra training cost. our codes are available at https://github.com/SmilesDZgk/S-CFG.

Title: PAT: Pixel-wise Adaptive Training for Long-tailed Segmentation

• Authors: Khoi Do, Duong Nguyen, Nguyen H. Tran, Viet Dung Nguyen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05393
• Pdf URL: https://arxiv.org/pdf/2404.05393
• Copy Paste: [[2404.05393]] PAT: Pixel-wise Adaptive Training for Long-tailed Segmentation(https://arxiv.org/abs/2404.05393)
• Keywords: robust, segmentation
• Abstract: Beyond class frequency, we recognize the impact of class-wise relationships among various class-specific predictions and the imbalance in label masks on long-tailed segmentation learning. To address these challenges, we propose an innovative Pixel-wise Adaptive Training (PAT) technique tailored for long-tailed segmentation. PAT has two key features: 1) class-wise gradient magnitude homogenization, and 2) pixel-wise class-specific loss adaptation (PCLA). First, the class-wise gradient magnitude homogenization helps alleviate the imbalance among label masks by ensuring equal consideration of the class-wise impact on model updates. Second, PCLA tackles the detrimental impact of both rare classes within the long-tailed distribution and inaccurate predictions from previous training stages by encouraging learning classes with low prediction confidence and guarding against forgetting classes with high confidence. This combined approach fosters robust learning while preventing the model from forgetting previously learned knowledge. PAT exhibits significant performance improvements, surpassing the current state-of-the-art by 2.2% in the NyU dataset. Moreover, it enhances overall pixel-wise accuracy by 2.85% and intersection over union value by 2.07%, with a particularly notable declination of 0.39% in detecting rare classes compared to Balance Logits Variation, as demonstrated on the three popular datasets, i.e., OxfordPetIII, CityScape, and NYU.

Title: SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety

• Authors: Paul Röttger, Fabio Pernisi, Bertie Vidgen, Dirk Hovy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05399
• Pdf URL: https://arxiv.org/pdf/2404.05399
• Copy Paste: [[2404.05399]] SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety(https://arxiv.org/abs/2404.05399)
• Keywords: large language model
• Abstract: The last two years have seen a rapid growth in concerns around the safety of large language models (LLMs). Researchers and practitioners have met these concerns by introducing an abundance of new datasets for evaluating and improving LLM safety. However, much of this work has happened in parallel, and with very different goals in mind, ranging from the mitigation of near-term risks around bias and toxic content generation to the assessment of longer-term catastrophic risk potential. This makes it difficult for researchers and practitioners to find the most relevant datasets for a given use case, and to identify gaps in dataset coverage that future work may fill. To remedy these issues, we conduct a first systematic review of open datasets for evaluating and improving LLM safety. We review 102 datasets, which we identified through an iterative and community-driven process over the course of several months. We highlight patterns and trends, such as a a trend towards fully synthetic datasets, as well as gaps in dataset coverage, such as a clear lack of non-English datasets. We also examine how LLM safety datasets are used in practice -- in LLM release publications and popular LLM benchmarks -- finding that current evaluation practices are highly idiosyncratic and make use of only a small fraction of available datasets. Our contributions are based on SafetyPrompts.com, a living catalogue of open datasets for LLM safety, which we commit to updating continuously as the field of LLM safety develops.

Title: SoK: Gradient Leakage in Federated Learning

• Authors: Jiacheng Du, Jiahui Hu, Zhibo Wang, Peng Sun, Neil Zhenqiang Gong, Kui Ren
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05403
• Pdf URL: https://arxiv.org/pdf/2404.05403
• Copy Paste: [[2404.05403]] SoK: Gradient Leakage in Federated Learning(https://arxiv.org/abs/2404.05403)
• Keywords: defense, attack, federate
• Abstract: Federated learning (FL) enables collaborative model training among multiple clients without raw data exposure. However, recent studies have shown that clients' private training data can be reconstructed from the gradients they share in FL, known as gradient inversion attacks (GIAs). While GIAs have demonstrated effectiveness under \emph{ideal settings and auxiliary assumptions}, their actual efficacy against \emph{practical FL systems} remains under-explored. To address this gap, we conduct a comprehensive study on GIAs in this work. We start with a survey of GIAs that establishes a milestone to trace their evolution and develops a systematization to uncover their inherent threats. Specifically, we categorize the auxiliary assumptions used by existing GIAs based on their practical accessibility to potential adversaries. To facilitate deeper analysis, we highlight the challenges that GIAs face in practical FL systems from three perspectives: \textit{local training}, \textit{model}, and \textit{post-processing}. We then perform extensive theoretical and empirical evaluations of state-of-the-art GIAs across diverse settings, utilizing eight datasets and thirteen models. Our findings indicate that GIAs have inherent limitations when reconstructing data under practical local training settings. Furthermore, their efficacy is sensitive to the trained model, and even simple post-processing measures applied to gradients can be effective defenses. Overall, our work provides crucial insights into the limited effectiveness of GIAs in practical FL systems. By rectifying prior misconceptions, we hope to inspire more accurate and realistic investigations on this topic.

Title: PerkwE_COQA: enhance Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Models

• Authors: Pardis Moradbeiki, Nasser Ghadiri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05406
• Pdf URL: https://arxiv.org/pdf/2404.05406
• Copy Paste: [[2404.05406]] PerkwE_COQA: enhance Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Models(https://arxiv.org/abs/2404.05406)
• Keywords: extraction, large language model
• Abstract: Smart cities need the involvement of their residents to enhance quality of life. Conversational query-answering is an emerging approach for user engagement. There is an increasing demand of an advanced conversational question-answering that goes beyond classic systems. Existing approaches have shown that LLMs offer promising capabilities for CQA, but may struggle to capture the nuances of conversational contexts. The new approach involves understanding the content and engaging in a multi-step conversation with the user to fulfill their needs. This paper presents a novel method to elevate the performance of Persian Conversational question-answering (CQA) systems. It combines the strengths of Large Language Models (LLMs) with contextual keyword extraction. Our method extracts keywords specific to the conversational flow, providing the LLM with additional context to understand the user's intent and generate more relevant and coherent responses. We evaluated the effectiveness of this combined approach through various metrics, demonstrating significant improvements in CQA performance compared to an LLM-only baseline. The proposed method effectively handles implicit questions, delivers contextually relevant answers, and tackles complex questions that rely heavily on conversational context. The findings indicate that our method outperformed the evaluation benchmarks up to 8% higher than existing methods and the LLM-only baseline.

Title: Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networks

• Authors: Maksym Ivashechkin, Oscar Mendez, Richard Bowden
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05414
• Pdf URL: https://arxiv.org/pdf/2404.05414
• Copy Paste: [[2404.05414]] Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networks(https://arxiv.org/abs/2404.05414)
• Keywords: extraction
• Abstract: 3D hand pose estimation from images has seen considerable interest from the literature, with new methods improving overall 3D accuracy. One current challenge is to address hand-to-hand interaction where self-occlusions and finger articulation pose a significant problem to estimation. Little work has applied physical constraints that minimize the hand intersections that occur as a result of noisy estimation. This work addresses the intersection of hands by exploiting an occupancy network that represents the hand's volume as a continuous manifold. This allows us to model the probability distribution of points being inside a hand. We designed an intersection loss function to minimize the likelihood of hand-to-point intersections. Moreover, we propose a new hand mesh parameterization that is superior to the commonly used MANO model in many respects including lower mesh complexity, underlying 3D skeleton extraction, watertightness, etc. On the benchmark InterHand2.6M dataset, the models trained using our intersection loss achieve better results than the state-of-the-art by significantly decreasing the number of hand intersections while lowering the mean per-joint positional error. Additionally, we demonstrate superior performance for 3D hand uplift on Re:InterHand and SMILE datasets and show reduced hand-to-hand intersections for complex domains such as sign-language pose estimation.

Title: Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations

• Authors: Yiming Li, Xueqing Peng, Jianfu Li, Xu Zuo, Suyuan Peng, Donghong Pei, Cui Tao, Hua Xu, Na Hong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05415
• Pdf URL: https://arxiv.org/pdf/2404.05415
• Copy Paste: [[2404.05415]] Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations(https://arxiv.org/abs/2404.05415)
• Keywords: extraction, transformer, generative, large language model
• Abstract: In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPT) present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to compare the performance of GPT with traditional deep learning models (Long Short-Term Memory (LSTM) and Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT)) in extracting acupoint-related location relations and assess the impact of pretraining and fine-tuning on GPT's performance. We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations ('direction_of,' 'distance_of,' 'part_of,' 'near_acupoint,' and 'located_near') (n= 3,174) between acupoints were annotated. Five models were compared: BioBERT, LSTM, pre-trained GPT-3.5, and fine-tuned GPT-3.5, as well as pre-trained GPT-4. Performance metrics included micro-average exact match precision, recall, and F1 scores. Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92. This study underscores the effectiveness of LLMs like GPT in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.

Title: XL$^2$Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies

• Authors: Xuanfan Ni, Hengyi Cai, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Piji Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05446
• Pdf URL: https://arxiv.org/pdf/2404.05446
• Copy Paste: [[2404.05446]] XL$^2$Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies(https://arxiv.org/abs/2404.05446)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks but are constrained by their small context window sizes. Various efforts have been proposed to expand the context window to accommodate even up to 200K input tokens. Meanwhile, building high-quality benchmarks with much longer text lengths and more demanding tasks to provide comprehensive evaluations is of immense practical interest to facilitate long context understanding research of LLMs. However, prior benchmarks create datasets that ostensibly cater to long-text comprehension by expanding the input of traditional tasks, which falls short to exhibit the unique characteristics of long-text understanding, including long dependency tasks and longer text length compatible with modern LLMs' context window size. In this paper, we introduce a benchmark for extremely long context understanding with long-range dependencies, XL$^2$Bench, which includes three scenarios: Fiction Reading, Paper Reading, and Law Reading, and four tasks of increasing complexity: Memory Retrieval, Detailed Understanding, Overall Understanding, and Open-ended Generation, covering 27 subtasks in English and Chinese. It has an average length of 100K+ words (English) and 200K+ characters (Chinese). Evaluating six leading LLMs on XL$^2$Bench, we find that their performance significantly lags behind human levels. Moreover, the observed decline in performance across both the original and enhanced datasets underscores the efficacy of our approach to mitigating data contamination.

Title: RoT: Enhancing Large Language Models with Reflection on Search Trees

• Authors: Wenyang Hui, Yan Wang, Kewei Tu, Chengyue Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05449
• Pdf URL: https://arxiv.org/pdf/2404.05449
• Copy Paste: [[2404.05449]] RoT: Enhancing Large Language Models with Reflection on Search Trees(https://arxiv.org/abs/2404.05449)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated impressive capability in reasoning and planning when integrated with tree-search-based prompting methods. However, since these methods ignore the previous search experiences, they often make the same mistakes in the search process. To address this issue, we introduce Reflection on search Trees (RoT), an LLM reflection framework designed to improve the performance of tree-search-based prompting methods. It uses a strong LLM to summarize guidelines from previous tree search experiences to enhance the ability of a weak LLM. The guidelines are instructions about solving this task through tree search which can prevent the weak LLMs from making similar mistakes in the past search process. In addition, we proposed a novel state selection method, which identifies the critical information from historical search processes to help RoT generate more specific and meaningful guidelines. In our extensive experiments, we find that RoT significantly improves the performance of LLMs in reasoning or planning tasks with various tree-search-based prompting methods (e.g., BFS and MCTS). Non-tree-search-based prompting methods such as Chain-of-Thought (CoT) can also benefit from RoT guidelines since RoT can provide task-specific knowledge collected from the search experience.

Title: HAMMR: HierArchical MultiModal React agents for generic VQA

• Authors: Lluis Castrejon, Thomas Mensink, Howard Zhou, Vittorio Ferrari, Andre Araujo, Jasper Uijlings
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05465
• Pdf URL: https://arxiv.org/pdf/2404.05465
• Copy Paste: [[2404.05465]] HAMMR: HierArchical MultiModal React agents for generic VQA(https://arxiv.org/abs/2404.05465)
• Keywords: large language model
• Abstract: Combining Large Language Models (LLMs) with external specialized tools (LLMs+tools) is a recent paradigm to solve multimodal tasks such as Visual Question Answering (VQA). While this approach was demonstrated to work well when optimized and evaluated for each individual benchmark, in practice it is crucial for the next generation of real-world AI systems to handle a broad range of multimodal problems. Therefore we pose the VQA problem from a unified perspective and evaluate a single system on a varied suite of VQA tasks including counting, spatial reasoning, OCR-based reasoning, visual pointing, external knowledge, and more. In this setting, we demonstrate that naively applying the LLM+tools approach using the combined set of all tools leads to poor results. This motivates us to introduce HAMMR: HierArchical MultiModal React. We start from a multimodal ReAct-based system and make it hierarchical by enabling our HAMMR agents to call upon other specialized agents. This enhances the compositionality of the LLM+tools approach, which we show to be critical for obtaining high accuracy on generic VQA. Concretely, on our generic VQA suite, HAMMR outperforms the naive LLM+tools approach by 19.5%. Additionally, HAMMR achieves state-of-the-art results on this task, outperforming the generic standalone PaLI-X VQA model by 5.0%.

Title: Enhancing Lip Reading with Multi-Scale Video and Multi-Encoder

• Authors: He Wang, Pengcheng Guo, Xucheng Wan, Huan Zhou, Lei Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05466
• Pdf URL: https://arxiv.org/pdf/2404.05466
• Copy Paste: [[2404.05466]] Enhancing Lip Reading with Multi-Scale Video and Multi-Encoder(https://arxiv.org/abs/2404.05466)
• Keywords: extraction, transformer
• Abstract: Automatic lip-reading (ALR) aims to automatically transcribe spoken content from a speaker's silent lip motion captured in video. Current mainstream lip-reading approaches only use a single visual encoder to model input videos of a single scale. In this paper, we propose to enhance lipreading by incorporating multi-scale video data and multi-encoder. Specifically, we first propose a novel multi-scale lip extraction algorithm based on the size of the speaker's face and an enhanced ResNet3D visual front-end (VFE) to extract lip features at different scales. For the multi-encoder, in addition to the mainstream Transformer and Conformer, we also incorporate the recently proposed Branchformer and EBranchformer as visual encoders. In the experiments, we explore the influence of different video data scales and encoders on ALR system performance and fuse the texts transcribed by all ALR systems using recognizer output voting error reduction (ROVER). Finally, our proposed approach placed second in the ICME 2024 ChatCLR Challenge Task 2, with a 21.52% reduction in character error rate (CER) compared to the official baseline on the evaluation set.

Title: WaveCatBoost for Probabilistic Forecasting of Regional Air Quality Data

• Authors: Jintu Borah, Tanujit Chakraborty, Md. Shahrul Md. Nadzir, Mylene G. Cayetano, Shubhankar Majumdar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05482
• Pdf URL: https://arxiv.org/pdf/2404.05482
• Copy Paste: [[2404.05482]] WaveCatBoost for Probabilistic Forecasting of Regional Air Quality Data(https://arxiv.org/abs/2404.05482)
• Keywords: protect, robust
• Abstract: Accurate and reliable air quality forecasting is essential for protecting public health, sustainable development, pollution control, and enhanced urban planning. This letter presents a novel WaveCatBoost architecture designed to forecast the real-time concentrations of air pollutants by combining the maximal overlapping discrete wavelet transform (MODWT) with the CatBoost model. This hybrid approach efficiently transforms time series into high-frequency and low-frequency components, thereby extracting signal from noise and improving prediction accuracy and robustness. Evaluation of two distinct regional datasets, from the Central Air Pollution Control Board (CPCB) sensor network and a low-cost air quality sensor system (LAQS), underscores the superior performance of our proposed methodology in real-time forecasting compared to the state-of-the-art statistical and deep learning architectures. Moreover, we employ a conformal prediction strategy to provide probabilistic bands with our forecasts.

Title: PetKaz at SemEval-2024 Task 3: Advancing Emotion Classification with an LLM for Emotion-Cause Pair Extraction in Conversations

• Authors: Roman Kazakov, Kseniia Petukhova, Ekaterina Kochmar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05502
• Pdf URL: https://arxiv.org/pdf/2404.05502
• Copy Paste: [[2404.05502]] PetKaz at SemEval-2024 Task 3: Advancing Emotion Classification with an LLM for Emotion-Cause Pair Extraction in Conversations(https://arxiv.org/abs/2404.05502)
• Keywords: extraction
• Abstract: In this paper, we present our submission to the SemEval-2023 Task~3 "The Competition of Multimodal Emotion Cause Analysis in Conversations", focusing on extracting emotion-cause pairs from dialogs. Specifically, our approach relies on combining fine-tuned GPT-3.5 for emotion classification and a BiLSTM-based neural network to detect causes. We score 2nd in the ranking for Subtask 1, demonstrating the effectiveness of our approach through one of the highest weighted-average proportional F1 scores recorded at 0.264.

Title: Taming Transformers for Realistic Lidar Point Cloud Generation

• Authors: Hamed Haghighi, Amir Samadi, Mehrdad Dianati, Valentina Donzella, Kurt Debattista
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.05505
• Pdf URL: https://arxiv.org/pdf/2404.05505
• Copy Paste: [[2404.05505]] Taming Transformers for Realistic Lidar Point Cloud Generation(https://arxiv.org/abs/2404.05505)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion Models (DMs) have achieved State-Of-The-Art (SOTA) results in the Lidar point cloud generation task, benefiting from their stable training and iterative refinement during sampling. However, DMs often fail to realistically model Lidar raydrop noise due to their inherent denoising process. To retain the strength of iterative sampling while enhancing the generation of raydrop noise, we introduce LidarGRIT, a generative model that uses auto-regressive transformers to iteratively sample the range images in the latent space rather than image space. Furthermore, LidarGRIT utilises VQ-VAE to separately decode range images and raydrop masks. Our results show that LidarGRIT achieves superior performance compared to SOTA models on KITTI-360 and KITTI odometry datasets. Code available at:https://github.com/hamedhaghighi/LidarGRIT.

Title: Impact of LiDAR visualisations on semantic segmentation of archaeological objects

• Authors: Raveerat Jaturapitpornchai, Giulio Poggi, Gregory Sech, Ziga Kokalj, Marco Fiorucci, Arianna Traviglia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05512
• Pdf URL: https://arxiv.org/pdf/2404.05512
• Copy Paste: [[2404.05512]] Impact of LiDAR visualisations on semantic segmentation of archaeological objects(https://arxiv.org/abs/2404.05512)
• Keywords: segmentation
• Abstract: Deep learning methods in LiDAR-based archaeological research often leverage visualisation techniques derived from Digital Elevation Models to enhance characteristics of archaeological objects present in the images. This paper investigates the impact of visualisations on deep learning performance through a comprehensive testing framework. The study involves the use of eight semantic segmentation models to evaluate seven diverse visualisations across two study areas, encompassing five archaeological classes. Experimental results reveal that the choice of appropriate visualisations can influence performance by up to 8%. Yet, pinpointing one visualisation that outperforms the others in segmenting all archaeological classes proves challenging. The observed performance variation, reaching up to 25% across different model configurations, underscores the importance of thoughtfully selecting model configurations and LiDAR visualisations for successfully segmenting archaeological objects.

Title: Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models

• Authors: Saman Motamed, Wouter Van Gansbeke, Luc Van Gool
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05519
• Pdf URL: https://arxiv.org/pdf/2404.05519
• Copy Paste: [[2404.05519]] Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models(https://arxiv.org/abs/2404.05519)
• Keywords: diffusion
• Abstract: With recent advances in image and video diffusion models for content creation, a plethora of techniques have been proposed for customizing their generated content. In particular, manipulating the cross-attention layers of Text-to-Image (T2I) diffusion models has shown great promise in controlling the shape and location of objects in the scene. Transferring image-editing techniques to the video domain, however, is extremely challenging as object motion and temporal consistency are difficult to capture accurately. In this work, we take a first look at the role of cross-attention in Text-to-Video (T2V) diffusion models for zero-shot video editing. While one-shot models have shown potential in controlling motion and camera movement, we demonstrate zero-shot control over object shape, position and movement in T2V models. We show that despite the limitations of current T2V models, cross-attention guidance can be a promising approach for editing videos.

Title: Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

• Authors: Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05530
• Pdf URL: https://arxiv.org/pdf/2404.05530
• Copy Paste: [[2404.05530]] Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data(https://arxiv.org/abs/2404.05530)
• Keywords: attack
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is a popular method for aligning Language Models (LM) with human values and preferences. RLHF requires a large number of preference pairs as training data, which are often used in both the Supervised Fine-Tuning and Reward Model training, and therefore publicly available datasets are commonly used. In this work, we study to what extent a malicious actor can manipulate the LMs generations by poisoning the preferences, i.e., injecting poisonous preference pairs into these datasets and the RLHF training process. We propose strategies to build poisonous preference pairs and test their performance by poisoning two widely used preference datasets. Our results show that preference poisoning is highly effective: by injecting a small amount of poisonous data (1-5% of the original dataset), we can effectively manipulate the LM to generate a target entity in a target sentiment (positive or negative). The findings from our experiments also shed light on strategies to defend against the preference poisoning attack.

Title: Evaluating Interventional Reasoning Capabilities of Large Language Models

• Authors: Tejas Kasetty, Divyat Mahajan, Gintare Karolina Dziugaite, Alexandre Drouin, Dhanya Sridhar
• Subjects: cs.LG, cs.AI, cs.CL, stat.ME
• Abstract URL: https://arxiv.org/abs/2404.05545
• Pdf URL: https://arxiv.org/pdf/2404.05545
• Copy Paste: [[2404.05545]] Evaluating Interventional Reasoning Capabilities of Large Language Models(https://arxiv.org/abs/2404.05545)
• Keywords: large language model
• Abstract: Numerous decision-making tasks require estimating causal effects under interventions on different parts of a system. As practitioners consider using large language models (LLMs) to automate decisions, studying their causal reasoning capabilities becomes crucial. A recent line of work evaluates LLMs ability to retrieve commonsense causal facts, but these evaluations do not sufficiently assess how LLMs reason about interventions. Motivated by the role that interventions play in causal inference, in this paper, we conduct empirical analyses to evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention. We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types, and enable a study of intervention-based reasoning. These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts. Our analysis on four LLMs highlights that while GPT- 4 models show promising accuracy at predicting the intervention effects, they remain sensitive to distracting factors in the prompts.

Title: TIM: A Time Interval Machine for Audio-Visual Action Recognition

• Authors: Jacob Chalk, Jaesung Huh, Evangelos Kazakos, Andrew Zisserman, Dima Damen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05559
• Pdf URL: https://arxiv.org/pdf/2404.05559
• Copy Paste: [[2404.05559]] TIM: A Time Interval Machine for Audio-Visual Action Recognition(https://arxiv.org/abs/2404.05559)
• Keywords: transformer
• Abstract: Diverse actions give rise to rich audio-visual signals in long videos. Recent works showcase that the two modalities of audio and video exhibit different temporal extents of events and distinct labels. We address the interplay between the two modalities in long videos by explicitly modelling the temporal extents of audio and visual events. We propose the Time Interval Machine (TIM) where a modality-specific time interval poses as a query to a transformer encoder that ingests a long video input. The encoder then attends to the specified interval, as well as the surrounding context in both modalities, in order to recognise the ongoing action. We test TIM on three long audio-visual video datasets: EPIC-KITCHENS, Perception Test, and AVE, reporting state-of-the-art (SOTA) for recognition. On EPIC-KITCHENS, we beat previous SOTA that utilises LLMs and significantly larger pre-training by 2.9% top-1 action recognition accuracy. Additionally, we show that TIM can be adapted for action detection, using dense multi-scale interval queries, outperforming SOTA on EPIC-KITCHENS-100 for most metrics, and showing strong performance on the Perception Test. Our ablations show the critical role of integrating the two modalities and modelling their time intervals in achieving this performance. Code and models at: https://github.com/JacobChalk/TIM

Title: Dynamic Backtracking in GFlowNet: Enhancing Decision Steps with Reward-Dependent Adjustment Mechanisms

• Authors: Shuai Guo, Jielei Chu, Lei Zhu, Tianrui Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05576
• Pdf URL: https://arxiv.org/pdf/2404.05576
• Copy Paste: [[2404.05576]] Dynamic Backtracking in GFlowNet: Enhancing Decision Steps with Reward-Dependent Adjustment Mechanisms(https://arxiv.org/abs/2404.05576)
• Keywords: generative
• Abstract: Generative Flow Networks (GFlowNets) are probabilistic models predicated on Markov flows, employing specific amortization algorithms to learn stochastic policies that generate compositional substances including biomolecules, chemical materials, and more. Demonstrating formidable prowess in generating high-performance biochemical molecules, GFlowNets accelerate the discovery of scientific substances, effectively circumventing the time-consuming, labor-intensive, and costly shortcomings intrinsic to conventional material discovery. However, previous work often struggles to accumulate exploratory experience and is prone to becoming disoriented within expansive sampling spaces. Attempts to address this issue, such as LS-GFN, are limited to local greedy searches and lack broader global adjustments. This paper introduces a novel GFlowNet variant, the Dynamic Backtracking GFN (DB-GFN), which enhances the adaptability of decision-making steps through a reward-based dynamic backtracking mechanism. DB-GFN permits backtracking during the network construction process according to the current state's reward value, thus correcting disadvantageous decisions and exploring alternative pathways during the exploration process. Applied to generative tasks of biochemical molecules and genetic material sequences, DB-GFN surpasses existing GFlowNet models and traditional reinforcement learning methods in terms of sample quality, exploration sample quantity, and training convergence speed. Furthermore, the orthogonal nature of DB-GFN suggests its potential as a powerful tool for future improvements in GFN networks, with the promise of integrating with other strategies to achieve more efficient search performance.

Title: Social-MAE: Social Masked Autoencoder for Multi-person Motion Representation Learning

• Authors: Mahsa Ehsanpour, Ian Reid, Hamid Rezatofighi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05578
• Pdf URL: https://arxiv.org/pdf/2404.05578
• Copy Paste: [[2404.05578]] Social-MAE: Social Masked Autoencoder for Multi-person Motion Representation Learning(https://arxiv.org/abs/2404.05578)
• Keywords: transformer
• Abstract: For a complete comprehension of multi-person scenes, it is essential to go beyond basic tasks like detection and tracking. Higher-level tasks, such as understanding the interactions and social activities among individuals, are also crucial. Progress towards models that can fully understand scenes involving multiple people is hindered by a lack of sufficient annotated data for such high-level tasks. To address this challenge, we introduce Social-MAE, a simple yet effective transformer-based masked autoencoder framework for multi-person human motion data. The framework uses masked modeling to pre-train the encoder to reconstruct masked human joint trajectories, enabling it to learn generalizable and data efficient representations of motion in human crowded scenes. Social-MAE comprises a transformer as the MAE encoder and a lighter-weight transformer as the MAE decoder which operates on multi-person joints' trajectory in the frequency domain. After the reconstruction task, the MAE decoder is replaced with a task-specific decoder and the model is fine-tuned end-to-end for a variety of high-level social tasks. Our proposed model combined with our pre-training approach achieves the state-of-the-art results on various high-level social tasks, including multi-person pose forecasting, social grouping, and social action understanding. These improvements are demonstrated across four popular multi-person datasets encompassing both human 2D and 3D body pose.

Title: Robust Data Pruning: Uncovering and Overcoming Implicit Bias

• Authors: Artem Vysogorets, Kartik Ahuja, Julia Kempe
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05579
• Pdf URL: https://arxiv.org/pdf/2404.05579
• Copy Paste: [[2404.05579]] Robust Data Pruning: Uncovering and Overcoming Implicit Bias(https://arxiv.org/abs/2404.05579)
• Keywords: robust, fair
• Abstract: In the era of exceptionally data-hungry models, careful selection of the training data is essential to mitigate the extensive costs of deep learning. Data pruning offers a solution by removing redundant or uninformative samples from the dataset, which yields faster convergence and improved neural scaling laws. However, little is known about its impact on classification bias of the trained models. We conduct the first systematic study of this effect and reveal that existing data pruning algorithms can produce highly biased classifiers. At the same time, we argue that random data pruning with appropriate class ratios has potential to improve the worst-class performance. We propose a "fairness-aware" approach to pruning and empirically demonstrate its performance on standard computer vision benchmarks. In sharp contrast to existing algorithms, our proposed method continues improving robustness at a tolerable drop of average performance as we prune more from the datasets. We present theoretical analysis of the classification risk in a mixture of Gaussians to further motivate our algorithm and support our findings.

Title: Responsible Visual Editing

• Authors: Minheng Ni, Yeli Shen, Lei Zhang, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05580
• Pdf URL: https://arxiv.org/pdf/2404.05580
• Copy Paste: [[2404.05580]] Responsible Visual Editing(https://arxiv.org/abs/2404.05580)
• Keywords: privacy
• Abstract: With recent advancements in visual synthesis, there is a growing risk of encountering images with detrimental effects, such as hate, discrimination, or privacy violations. The research on transforming harmful images into responsible ones remains unexplored. In this paper, we formulate a new task, responsible visual editing, which entails modifying specific concepts within an image to render it more responsible while minimizing changes. However, the concept that needs to be edited is often abstract, making it challenging to locate what needs to be modified and plan how to modify it. To tackle these challenges, we propose a Cognitive Editor (CoEditor) that harnesses the large multimodal model through a two-stage cognitive process: (1) a perceptual cognitive process to focus on what needs to be modified and (2) a behavioral cognitive process to strategize how to modify. To mitigate the negative implications of harmful images on research, we create a transparent and public dataset, AltBear, which expresses harmful information using teddy bears instead of humans. Experiments demonstrate that CoEditor can effectively comprehend abstract concepts within complex scenes and significantly surpass the performance of baseline models for responsible visual editing. We find that the AltBear dataset corresponds well to the harmful content found in real images, offering a consistent experimental evaluation, thereby providing a safer benchmark for future research. Moreover, CoEditor also shows great results in general editing. We release our code and dataset at https://github.com/kodenii/Responsible-Visual-Editing.

Title: Towards More General Video-based Deepfake Detection through Facial Feature Guided Adaptation for Foundation Model

• Authors: Yue-Hua Han, Tai-Ming Huang, Shu-Tzu Lo, Po-Han Huang, Kai-Lung Hua, Jun-Cheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05583
• Pdf URL: https://arxiv.org/pdf/2404.05583
• Copy Paste: [[2404.05583]] Towards More General Video-based Deepfake Detection through Facial Feature Guided Adaptation for Foundation Model(https://arxiv.org/abs/2404.05583)
• Keywords: secure, robust, generative
• Abstract: With the rise of deep learning, generative models have enabled the creation of highly realistic synthetic images, presenting challenges due to their potential misuse. While research in Deepfake detection has grown rapidly in response, many detection methods struggle with unseen Deepfakes generated by new synthesis techniques. To address this generalisation challenge, we propose a novel Deepfake detection approach by adapting rich information encoded inside the Foundation Models with rich information encoded inside, specifically using the image encoder from CLIP which has demonstrated strong zero-shot capability for downstream tasks. Inspired by the recent advances of parameter efficient fine-tuning, we propose a novel side-network-based decoder to extract spatial and temporal cues from the given video clip, with the promotion of the Facial Component Guidance (FCG) to guidencourage the spatial feature to include features of key facial parts for more robust and general Deepfake detection. Through extensive cross-dataset evaluations, our approach exhibits superior effectiveness in identifying unseen Deepfake samples, achieving notable performance improvementsuccess even with limited training samples and manipulation types. Our model secures an average performance enhancement of 0.9% AUROC in cross-dataset assessments comparing with state-of-the-art methods, especiallytablishing a significant lead of achieving 4.4% improvement on the challenging DFDC dataset.

Title: Neural Cellular Automata for Lightweight, Robust and Explainable Classification of White Blood Cell Images

• Authors: Michael Deutges, Ario Sadafi, Nassir Navab, Carsten Marr
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.05584
• Pdf URL: https://arxiv.org/pdf/2404.05584
• Copy Paste: [[2404.05584]] Neural Cellular Automata for Lightweight, Robust and Explainable Classification of White Blood Cell Images(https://arxiv.org/abs/2404.05584)
• Keywords: robust, explainability
• Abstract: Diagnosis of hematological malignancies depends on accurate identification of white blood cells in peripheral blood smears. Deep learning techniques are emerging as a viable solution to scale and optimize this process by automatic identification of cells in laboratories. However, these techniques face several challenges such as limited generalizability, sensitivity to domain shifts and lack of explainability. Here, we are introducing a novel approach based on neural cellular automata (NCA) for white blood cell classification. We test our approach on three datasets of white blood cell images and show that we achieve competitive performance compared to conventional methods. Our NCA-based method is significantly smaller in terms of parameters and exhibits robustness to domain shifts. Furthermore, the architecture is inherently explainable, providing insights into the decision process for each classification, helping experts understand and validate model predictions. Results demonstrate that NCA not only can be used for image classification, but also address key challenges of conventional methods, indicating a high potential for applicability in clinical practice.

Title: Enhancing Software Related Information Extraction with Generative Language Models through Single-Choice Question Answering

• Authors: Wolfgang Otto, Sharmila Upadhyaya, Stefan Dietze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05587
• Pdf URL: https://arxiv.org/pdf/2404.05587
• Copy Paste: [[2404.05587]] Enhancing Software Related Information Extraction with Generative Language Models through Single-Choice Question Answering(https://arxiv.org/abs/2404.05587)
• Keywords: extraction, generative
• Abstract: This paper describes our participation in the Shared Task on Software Mentions Disambiguation (SOMD), with a focus on improving relation extraction in scholarly texts through Generative Language Models (GLMs) using single-choice question-answering. The methodology prioritises the use of in-context learning capabilities of GLMs to extract software-related entities and their descriptive attributes, such as distributive information. Our approach uses Retrieval-Augmented Generation (RAG) techniques and GLMs for Named Entity Recognition (NER) and Attributive NER to identify relationships between extracted software entities, providing a structured solution for analysing software citations in academic literature. The paper provides a detailed description of our approach, demonstrating how using GLMs in a single-choice QA paradigm can greatly enhance IE methodologies. Our participation in the SOMD shared task highlights the importance of precise software citation practices and showcases our system's ability to overcome the challenges of disambiguating and extracting relationships between software mentions. This sets the groundwork for future research and development in this field.

Title: MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering

• Authors: Iñigo Alonso, Maite Oronoz, Rodrigo Agerri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05590
• Pdf URL: https://arxiv.org/pdf/2404.05590
• Copy Paste: [[2404.05590]] MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering(https://arxiv.org/abs/2404.05590)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have the potential of facilitating the development of Artificial Intelligence technology to assist medical experts for interactive decision support, which has been demonstrated by their competitive performances in Medical QA. However, while impressive, the required quality bar for medical applications remains far from being achieved. Currently, LLMs remain challenged by outdated knowledge and by their tendency to generate hallucinated content. Furthermore, most benchmarks to assess medical knowledge lack reference gold explanations which means that it is not possible to evaluate the reasoning of LLMs predictions. Finally, the situation is particularly grim if we consider benchmarking LLMs for languages other than English which remains, as far as we know, a totally neglected topic. In order to address these shortcomings, in this paper we present MedExpQA, the first multilingual benchmark based on medical exams to evaluate LLMs in Medical Question Answering. To the best of our knowledge, MedExpQA includes for the first time reference gold explanations written by medical doctors which can be leveraged to establish various gold-based upper-bounds for comparison with LLMs performance. Comprehensive multilingual experimentation using both the gold reference explanations and Retrieval Augmented Generation (RAG) approaches show that performance of LLMs still has large room for improvement, especially for languages other than English. Furthermore, and despite using state-of-the-art RAG methods, our results also demonstrate the difficulty of obtaining and integrating readily available medical knowledge that may positively impact results on downstream evaluations for Medical Question Answering. So far the benchmark is available in four languages, but we hope that this work may encourage further development to other languages.

Title: UniFL: Improve Stable Diffusion via Unified Feedback Learning

• Authors: Jiacheng Zhang, Jie Wu, Yuxi Ren, Xin Xia, Huafeng Kuang, Pan Xie, Jiashi Li, Xuefeng Xiao, Weilin Huang, Min Zheng, Lean Fu, Guanbin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05595
• Pdf URL: https://arxiv.org/pdf/2404.05595
• Copy Paste: [[2404.05595]] UniFL: Improve Stable Diffusion via Unified Feedback Learning(https://arxiv.org/abs/2404.05595)
• Keywords: diffusion
• Abstract: Diffusion models have revolutionized the field of image generation, leading to the proliferation of high-quality models and diverse downstream applications. However, despite these significant advancements, the current competitive solutions still suffer from several limitations, including inferior visual quality, a lack of aesthetic appeal, and inefficient inference, without a comprehensive solution in sight. To address these challenges, we present UniFL, a unified framework that leverages feedback learning to enhance diffusion models comprehensively. UniFL stands out as a universal, effective, and generalizable solution applicable to various diffusion models, such as SD1.5 and SDXL. Notably, UniFL incorporates three key components: perceptual feedback learning, which enhances visual quality; decoupled feedback learning, which improves aesthetic appeal; and adversarial feedback learning, which optimizes inference speed. In-depth experiments and extensive user studies validate the superior performance of our proposed method in enhancing both the quality of generated models and their acceleration. For instance, UniFL surpasses ImageReward by 17% user preference in terms of generation quality and outperforms LCM and SDXL Turbo by 57% and 20% in 4-step inference. Moreover, we have verified the efficacy of our approach in downstream tasks, including Lora, ControlNet, and AnimateDiff.

Title: Hook-in Privacy Techniques for gRPC-based Microservice Communication

• Authors: Louis Loechel, Siar-Remzi Akbayin, Elias Grünewald, Jannis Kiesel, Inga Strelnikova, Thomas Janke, Frank Pallas
• Subjects: cs.CR, cs.CY, cs.DC, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.05598
• Pdf URL: https://arxiv.org/pdf/2404.05598
• Copy Paste: [[2404.05598]] Hook-in Privacy Techniques for gRPC-based Microservice Communication(https://arxiv.org/abs/2404.05598)
• Keywords: privacy
• Abstract: gRPC is at the heart of modern distributed system architectures. Based on HTTP/2 and Protocol Buffers, it provides highly performant, standardized, and polyglot communication across loosely coupled microservices and is increasingly preferred over REST- or GraphQL-based service APIs in practice. Despite its widespread adoption, gRPC lacks any advanced privacy techniques beyond transport encryption and basic token-based authentication. Such advanced techniques are, however, increasingly important for fulfilling regulatory requirements. For instance, anonymizing or otherwise minimizing (personal) data before responding to requests, or pre-processing data based on the purpose of the access may be crucial in certain usecases. In this paper, we therefore propose a novel approach for integrating such advanced privacy techniques into the gRPC framework in a practically viable way. Specifically, we present a general approach along with a working prototype that implements privacy techniques, such as data minimization and purpose limitation, in a configurable, extensible, and gRPC-native way utilizing a gRPC interceptor. We also showcase how to integrate this contribution into a realistic example of a food delivery use case. Alongside these implementations, a preliminary performance evaluation shows practical applicability with reasonable overheads. Altogether, we present a viable solution for integrating advanced privacy techniques into real-world gRPC-based microservice architectures, thereby facilitating regulatory compliance ``by design''.

Title: SpeechAlign: Aligning Speech Generation to Human Preferences

• Authors: Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, Xipeng Qiu
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.05600
• Pdf URL: https://arxiv.org/pdf/2404.05600
• Copy Paste: [[2404.05600]] SpeechAlign: Aligning Speech Generation to Human Preferences(https://arxiv.org/abs/2404.05600)
• Keywords: robust
• Abstract: Speech language models have significantly advanced in generating realistic speech, with neural codec language models standing out. However, the integration of human feedback to align speech outputs to human preferences is often neglected. This paper addresses this gap by first analyzing the distribution gap in codec language models, highlighting how it leads to discrepancies between the training and inference phases, which negatively affects performance. Then we explore leveraging learning from human feedback to bridge the distribution gap. We introduce SpeechAlign, an iterative self-improvement strategy that aligns speech language models to human preferences. SpeechAlign involves constructing a preference codec dataset contrasting golden codec tokens against synthetic tokens, followed by preference optimization to improve the codec language model. This cycle of improvement is carried out iteratively to steadily convert weak models to strong ones. Through both subjective and objective evaluations, we show that SpeechAlign can bridge the distribution gap and facilitating continuous self-improvement of the speech language model. Moreover, SpeechAlign exhibits robust generalization capabilities and works for smaller models. Code and models will be available at https://github.com/0nutation/SpeechGPT.

Title: AI-Enabled System for Efficient and Effective Cyber Incident Detection and Response in Cloud Environments

• Authors: Mohammed A. M. Farzaan, Mohamed Chahine Ghanem, Ayman El-Hajjar
• Subjects: cs.CR, cs.ET, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.05602
• Pdf URL: https://arxiv.org/pdf/2404.05602
• Copy Paste: [[2404.05602]] AI-Enabled System for Efficient and Effective Cyber Incident Detection and Response in Cloud Environments(https://arxiv.org/abs/2404.05602)
• Keywords: security, robust
• Abstract: The escalating sophistication and volume of cyber threats in cloud environments necessitate a paradigm shift in strategies. Recognising the need for an automated and precise response to cyber threats, this research explores the application of AI and ML and proposes an AI-powered cyber incident response system for cloud environments. This system, encompassing Network Traffic Classification, Web Intrusion Detection, and post-incident Malware Analysis (built as a Flask application), achieves seamless integration across platforms like Google Cloud and Microsoft Azure. The findings from this research highlight the effectiveness of the Random Forest model, achieving an accuracy of 90% for the Network Traffic Classifier and 96% for the Malware Analysis Dual Model application. Our research highlights the strengths of AI-powered cyber security. The Random Forest model excels at classifying cyber threats, offering an efficient and robust solution. Deep learning models significantly improve accuracy, and their resource demands can be managed using cloud-based TPUs and GPUs. Cloud environments themselves provide a perfect platform for hosting these AI/ML systems, while container technology ensures both efficiency and scalability. These findings demonstrate the contribution of the AI-led system in guaranteeing a robust and scalable cyber incident response solution in the cloud.

Title: Technical Report: The Graph Spectral Token -- Enhancing Graph Transformers with Spectral Information

• Authors: Zihan Pengmei, Zimu Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05604
• Pdf URL: https://arxiv.org/pdf/2404.05604
• Copy Paste: [[2404.05604]] Technical Report: The Graph Spectral Token -- Enhancing Graph Transformers with Spectral Information(https://arxiv.org/abs/2404.05604)
• Keywords: transformer
• Abstract: Graph Transformers have emerged as a powerful alternative to Message-Passing Graph Neural Networks (MP-GNNs) to address limitations such as over-squashing of information exchange. However, incorporating graph inductive bias into transformer architectures remains a significant challenge. In this report, we propose the Graph Spectral Token, a novel approach to directly encode graph spectral information, which captures the global structure of the graph, into the transformer architecture. By parameterizing the auxiliary [CLS] token and leaving other tokens representing graph nodes, our method seamlessly integrates spectral information into the learning process. We benchmark the effectiveness of our approach by enhancing two existing graph transformers, GraphTrans and SubFormer. The improved GraphTrans, dubbed GraphTrans-Spec, achieves over 10% improvements on large graph benchmark datasets while maintaining efficiency comparable to MP-GNNs. SubFormer-Spec demonstrates strong performance across various datasets.

Title: A Training-Free Plug-and-Play Watermark Framework for Stable Diffusion

• Authors: Guokai Zhang, Lanjun Wang, Yuting Su, An-An Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05607
• Pdf URL: https://arxiv.org/pdf/2404.05607
• Copy Paste: [[2404.05607]] A Training-Free Plug-and-Play Watermark Framework for Stable Diffusion(https://arxiv.org/abs/2404.05607)
• Keywords: security, attack, robust, watermark, diffusion
• Abstract: Nowadays, the family of Stable Diffusion (SD) models has gained prominence for its high quality outputs and scalability. This has also raised security concerns on social media, as malicious users can create and disseminate harmful content. Existing approaches involve training components or entire SDs to embed a watermark in generated images for traceability and responsibility attribution. However, in the era of AI-generated content (AIGC), the rapid iteration of SDs renders retraining with watermark models costly. To address this, we propose a training-free plug-and-play watermark framework for SDs. Without modifying any components of SDs, we embed diverse watermarks in the latent space, adapting to the denoising process. Our experimental findings reveal that our method effectively harmonizes image quality and watermark invisibility. Furthermore, it performs robustly under various attacks. We also have validated that our method is generalized to multiple versions of SDs, even without retraining the watermark model.

Title: LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking

• Authors: Faren Yan, Peng Yu, Xin Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05624
• Pdf URL: https://arxiv.org/pdf/2404.05624
• Copy Paste: [[2404.05624]] LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking(https://arxiv.org/abs/2404.05624)
• Keywords: large language model
• Abstract: The use of LLMs for natural language processing has become a popular trend in the past two years, driven by their formidable capacity for context comprehension and learning, which has inspired a wave of research from academics and industry professionals. However, for certain NLP tasks, such as NER, the performance of LLMs still falls short when compared to supervised learning methods. In our research, we developed a NER processing framework called LTNER that incorporates a revolutionary Contextualized Entity Marking Gen Method. By leveraging the cost-effective GPT-3.5 coupled with context learning that does not require additional training, we significantly improved the accuracy of LLMs in handling NER tasks. The F1 score on the CoNLL03 dataset increased from the initial 85.9% to 91.9%, approaching the performance of supervised fine-tuning. This outcome has led to a deeper understanding of the potential of LLMs.

Title: Learning a Category-level Object Pose Estimator without Pose Annotations

• Authors: Fengrui Tian, Yaoyao Liu, Adam Kortylewski, Yueqi Duan, Shaoyi Du, Alan Yuille, Angtian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05626
• Pdf URL: https://arxiv.org/pdf/2404.05626
• Copy Paste: [[2404.05626]] Learning a Category-level Object Pose Estimator without Pose Annotations(https://arxiv.org/abs/2404.05626)
• Keywords: diffusion
• Abstract: 3D object pose estimation is a challenging task. Previous works always require thousands of object images with annotated poses for learning the 3D pose correspondence, which is laborious and time-consuming for labeling. In this paper, we propose to learn a category-level 3D object pose estimator without pose annotations. Instead of using manually annotated images, we leverage diffusion models (e.g., Zero-1-to-3) to generate a set of images under controlled pose differences and propose to learn our object pose estimator with those images. Directly using the original diffusion model leads to images with noisy poses and artifacts. To tackle this issue, firstly, we exploit an image encoder, which is learned from a specially designed contrastive pose learning, to filter the unreasonable details and extract image feature maps. Additionally, we propose a novel learning strategy that allows the model to learn object poses from those generated image sets without knowing the alignment of their canonical poses. Experimental results show that our method has the capability of category-level object pose estimation from a single shot setting (as pose definition), while significantly outperforming other state-of-the-art methods on the few-shot category-level object pose estimation benchmarks.

Title: Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data

• Authors: Haitham Hammami, Louis Baligand, Bojan Petrovski
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05632
• Pdf URL: https://arxiv.org/pdf/2404.05632
• Copy Paste: [[2404.05632]] Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data(https://arxiv.org/abs/2404.05632)
• Keywords: robust, transformer, generative, large language model
• Abstract: In the financial industry, identifying the location of parties involved in payments is a major challenge in the context of various regulatory requirements. For this purpose address parsing entails extracting fields such as street, postal code, or country from free text message attributes. While payment processing platforms are updating their standards with more structured formats such as SWIFT with ISO 20022, address parsing remains essential for a considerable volume of messages. With the emergence of Transformers and Generative Large Language Models (LLM), we explore the performance of state-of-the-art solutions given the constraint of processing a vast amount of daily data. This paper also aims to show the need for training robust models capable of dealing with real-world noisy transactional data. Our results suggest that a well fine-tuned Transformer model using early-stopping significantly outperforms other approaches. Nevertheless, generative LLMs demonstrate strong zero-shot performance and warrant further investigations.

Title: Investigating the Impact of Quantization on Adversarial Robustness

• Authors: Qun Li, Yuan Meng, Chen Tang, Jiacheng Jiang, Zhi Wang
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05639
• Pdf URL: https://arxiv.org/pdf/2404.05639
• Copy Paste: [[2404.05639]] Investigating the Impact of Quantization on Adversarial Robustness(https://arxiv.org/abs/2404.05639)
• Keywords: secure, security, attack, robust
• Abstract: Quantization is a promising technique for reducing the bit-width of deep models to improve their runtime performance and storage efficiency, and thus becomes a fundamental step for deployment. In real-world scenarios, quantized models are often faced with adversarial attacks which cause the model to make incorrect inferences by introducing slight perturbations. However, recent studies have paid less attention to the impact of quantization on the model robustness. More surprisingly, existing studies on this topic even present inconsistent conclusions, which prompted our in-depth investigation. In this paper, we conduct a first-time analysis of the impact of the quantization pipeline components that can incorporate robust optimization under the settings of Post-Training Quantization and Quantization-Aware Training. Through our detailed analysis, we discovered that this inconsistency arises from the use of different pipelines in different studies, specifically regarding whether robust optimization is performed and at which quantization stage it occurs. Our research findings contribute insights into deploying more secure and robust quantized networks, assisting practitioners in reference for scenarios with high-security requirements and limited resources.

Title: Causality Extraction from Nuclear Licensee Event Reports Using a Hybrid Framework

• Authors: Sohag Rahman, Sai Zhang, Min Xian, Shoukun Sun, Fei Xu, Zhegang Ma
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05656
• Pdf URL: https://arxiv.org/pdf/2404.05656
• Copy Paste: [[2404.05656]] Causality Extraction from Nuclear Licensee Event Reports Using a Hybrid Framework(https://arxiv.org/abs/2404.05656)
• Keywords: extraction
• Abstract: Industry-wide nuclear power plant operating experience is a critical source of raw data for performing parameter estimations in reliability and risk models. Much operating experience information pertains to failure events and is stored as reports containing unstructured data, such as narratives. Event reports are essential for understanding how failures are initiated and propagated, including the numerous causal relations involved. Causal relation extraction using deep learning represents a significant frontier in the field of natural language processing (NLP), and is crucial since it enables the interpretation of intricate narratives and connections contained within vast amounts of written information. This paper proposed a hybrid framework for causality detection and extraction from nuclear licensee event reports. The main contributions include: (1) we compiled an LER corpus with 20,129 text samples for causality analysis, (2) developed an interactive tool for labeling cause effect pairs, (3) built a deep-learning-based approach for causal relation detection, and (4) developed a knowledge based cause-effect extraction approach.

Title: MLP Can Be A Good Transformer Learner

• Authors: Sihao Lin, Pumeng Lyu, Dongrui Liu, Tao Tang, Xiaodan Liang, Andy Song, Xiaojun Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05657
• Pdf URL: https://arxiv.org/pdf/2404.05657
• Copy Paste: [[2404.05657]] MLP Can Be A Good Transformer Learner(https://arxiv.org/abs/2404.05657)
• Keywords: transformer
• Abstract: Self-attention mechanism is the key of the Transformer but often criticized for its computation demands. Previous token pruning works motivate their methods from the view of computation redundancy but still need to load the full network and require same memory costs. This paper introduces a novel strategy that simplifies vision transformers and reduces computational load through the selective removal of non-essential attention layers, guided by entropy considerations. We identify that regarding the attention layer in bottom blocks, their subsequent MLP layers, i.e. two feed-forward layers, can elicit the same entropy quantity. Meanwhile, the accompanied MLPs are under-exploited since they exhibit smaller feature entropy compared to those MLPs in the top blocks. Therefore, we propose to integrate the uninformative attention layers into their subsequent counterparts by degenerating them into identical mapping, yielding only MLP in certain transformer blocks. Experimental results on ImageNet-1k show that the proposed method can remove 40% attention layer of DeiT-B, improving throughput and memory bound without performance compromise. Code is available at https://github.com/sihaoevery/lambda_vit.

Title: VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain

• Authors: Khai Le-Duc
• Subjects: cs.CL, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.05659
• Pdf URL: https://arxiv.org/pdf/2404.05659
• Copy Paste: [[2404.05659]] VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain(https://arxiv.org/abs/2404.05659)
• Keywords: privacy
• Abstract: Due to privacy restrictions, there's a shortage of publicly available speech recognition datasets in the medical domain. In this work, we present VietMed - a Vietnamese speech recognition dataset in the medical domain comprising 16h of labeled medical speech, 1000h of unlabeled medical speech and 1200h of unlabeled general-domain speech. To our best knowledge, VietMed is by far the world's largest public medical speech recognition dataset in 7 aspects: total duration, number of speakers, diseases, recording conditions, speaker roles, unique medical terms and accents. VietMed is also by far the largest public Vietnamese speech dataset in terms of total duration. Additionally, we are the first to present a medical ASR dataset covering all ICD-10 disease groups and all accents within a country. Moreover, we release the first public large-scale pre-trained models for Vietnamese ASR, w2v2-Viet and XLSR-53-Viet, along with the first public large-scale fine-tuned models for medical ASR. Even without any medical data in unsupervised pre-training, our best pre-trained model XLSR-53-Viet generalizes very well to the medical domain by outperforming state-of-the-art XLSR-53, from 51.8% to 29.6% WER on test set (a relative reduction of more than 40%). All code, data and models are made publicly available here: https://github.com/leduckhai/MultiMed.

Title: BinaryDM: Towards Accurate Binarization of Diffusion Model

• Authors: Xingyu Zheng, Haotong Qin, Xudong Ma, Mingyuan Zhang, Haojie Hao, Jiakai Wang, Zixiang Zhao, Jinyang Guo, Xianglong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05662
• Pdf URL: https://arxiv.org/pdf/2404.05662
• Copy Paste: [[2404.05662]] BinaryDM: Towards Accurate Binarization of Diffusion Model(https://arxiv.org/abs/2404.05662)
• Keywords: diffusion
• Abstract: With the advancement of diffusion models (DMs) and the substantially increased computational requirements, quantization emerges as a practical solution to obtain compact and efficient low-bit DMs. However, the highly discrete representation leads to severe accuracy degradation, hindering the quantization of diffusion models to ultra-low bit-widths. In this paper, we propose BinaryDM, a novel accurate quantization-aware training approach to push the weights of diffusion models towards the limit of 1-bit. Firstly, we present a Learnable Multi-basis Binarizer (LMB) to recover the representations generated by the binarized DM, which improves the information in details of representations crucial to the DM. Secondly, a Low-rank Representation Mimicking (LRM) is applied to enhance the binarization-aware optimization of the DM, alleviating the optimization direction ambiguity caused by fine-grained alignment. Moreover, a progressive initialization strategy is applied to training DMs to avoid convergence difficulties. Comprehensive experiments demonstrate that BinaryDM achieves significant accuracy and efficiency gains compared to SOTA quantization methods of DMs under ultra-low bit-widths. As the first binarization method for diffusion models, BinaryDM achieves impressive 16.0 times FLOPs and 27.1 times storage savings with 1-bit weight and 4-bit activation, showcasing its substantial advantages and potential for deploying DMs on resource-limited scenarios.

Title: YaART: Yet Another ART Rendering Technology

• Authors: Sergey Kastryulin, Artem Konev, Alexander Shishenya, Eugene Lyapustin, Artem Khurshudov, Alexander Tselousov, Nikita Vinokurov, Denis Kuznedelev, Alexander Markovich, Grigoriy Livshits, Alexey Kirillov, Anastasiia Tabisheva, Liubov Chubarova, Marina Kaminskaia, Alexander Ustyuzhanin, Artemii Shvetsov, Daniil Shlenskii, Valerii Startsev, Dmitrii Kornilov, Mikhail Romanov, Artem Babenko, Sergei Ovcharenko, Valentin Khrulkov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05666
• Pdf URL: https://arxiv.org/pdf/2404.05666
• Copy Paste: [[2404.05666]] YaART: Yet Another ART Rendering Technology(https://arxiv.org/abs/2404.05666)
• Keywords: diffusion, generative
• Abstract: In the rapidly progressing field of generative models, the development of efficient and high-fidelity text-to-image diffusion systems represents a significant frontier. This study introduces YaART, a novel production-grade text-to-image cascaded diffusion model aligned to human preferences using Reinforcement Learning from Human Feedback (RLHF). During the development of YaART, we especially focus on the choices of the model and training dataset sizes, the aspects that were not systematically investigated for text-to-image cascaded diffusion models before. In particular, we comprehensively analyze how these choices affect both the efficiency of the training process and the quality of the generated images, which are highly important in practice. Furthermore, we demonstrate that models trained on smaller datasets of higher-quality images can successfully compete with those trained on larger datasets, establishing a more efficient scenario of diffusion models training. From the quality perspective, YaART is consistently preferred by users over many existing state-of-the-art models.

Title: AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation

• Authors: Jiannan Ge, Lingxi Xie, Hongtao Xie, Pandeng Li, Xiaopeng Zhang, Yongdong Zhang, Qi Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05667
• Pdf URL: https://arxiv.org/pdf/2404.05667
• Copy Paste: [[2404.05667]] AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation(https://arxiv.org/abs/2404.05667)
• Keywords: extraction, segmentation
• Abstract: A serious issue that harms the performance of zero-shot visual recognition is named objective misalignment, i.e., the learning objective prioritizes improving the recognition accuracy of seen classes rather than unseen classes, while the latter is the true target to pursue. This issue becomes more significant in zero-shot image segmentation because the stronger (i.e., pixel-level) supervision brings a larger gap between seen and unseen classes. To mitigate it, we propose a novel architecture named AlignZeg, which embodies a comprehensive improvement of the segmentation pipeline, including proposal extraction, classification, and correction, to better fit the goal of zero-shot segmentation. (1) Mutually-Refined Proposal Extraction. AlignZeg harnesses a mutual interaction between mask queries and visual features, facilitating detailed class-agnostic mask proposal extraction. (2) Generalization-Enhanced Proposal Classification. AlignZeg introduces synthetic data and incorporates multiple background prototypes to allocate a more generalizable feature space. (3) Predictive Bias Correction. During the inference stage, AlignZeg uses a class indicator to find potential unseen class proposals followed by a prediction postprocess to correct the prediction bias. Experiments demonstrate that AlignZeg markedly enhances zero-shot semantic segmentation, as shown by an average 3.8% increase in hIoU, primarily attributed to a 7.1% improvement in identifying unseen classes, and we further validate that the improvement comes from alleviating the objective misalignment issue.

Title: NAF-DPM: A Nonlinear Activation-Free Diffusion Probabilistic Model for Document Enhancement

• Authors: Giordano Cicchetti, Danilo Comminiello
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05669
• Pdf URL: https://arxiv.org/pdf/2404.05669
• Copy Paste: [[2404.05669]] NAF-DPM: A Nonlinear Activation-Free Diffusion Probabilistic Model for Document Enhancement(https://arxiv.org/abs/2404.05669)
• Keywords: diffusion, generative
• Abstract: Real-world documents may suffer various forms of degradation, often resulting in lower accuracy in optical character recognition (OCR) systems. Therefore, a crucial preprocessing step is essential to eliminate noise while preserving text and key features of documents. In this paper, we propose NAF-DPM, a novel generative framework based on a diffusion probabilistic model (DPM) designed to restore the original quality of degraded documents. While DPMs are recognized for their high-quality generated images, they are also known for their large inference time. To mitigate this problem we provide the DPM with an efficient nonlinear activation-free (NAF) network and we employ as a sampler a fast solver of ordinary differential equations, which can converge in a few iterations. To better preserve text characters, we introduce an additional differentiable module based on convolutional recurrent neural networks, simulating the behavior of an OCR system during training. Experiments conducted on various datasets showcase the superiority of our approach, achieving state-of-the-art performance in terms of pixel-level and perceptual similarity metrics. Furthermore, the results demonstrate a notable character error reduction made by OCR systems when transcribing real-world document images enhanced by our framework. Code and pre-trained models are available at https://github.com/ispamm/NAF-DPM.

Title: CoReS: Orchestrating the Dance of Reasoning and Segmentation

• Authors: Xiaoyi Bao, Siyang Sun, Shuailei Ma, Kecheng Zheng, Yuxin Guo, Guosheng Zhao, Yun Zheng, Xingang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05673
• Pdf URL: https://arxiv.org/pdf/2404.05673
• Copy Paste: [[2404.05673]] CoReS: Orchestrating the Dance of Reasoning and Segmentation(https://arxiv.org/abs/2404.05673)
• Keywords: large language model, segmentation
• Abstract: The reasoning segmentation task, which demands a nuanced comprehension of intricate queries to accurately pinpoint object regions, is attracting increasing attention. However, Multi-modal Large Language Models (MLLM) often find it difficult to accurately localize the objects described in complex reasoning contexts. We believe that the act of reasoning segmentation should mirror the cognitive stages of human visual search, where each step is a progressive refinement of thought toward the final object. Thus we introduce the Chains of Reasoning and Segmenting (CoReS) and find this top-down visual hierarchy indeed enhances the visual search process. Specifically, we propose a dual-chain structure that generates multi-modal, chain-like outputs to aid the segmentation process. Furthermore, to steer the MLLM's outputs into this intended hierarchy, we incorporate in-context inputs as guidance. Extensive experiments demonstrate the superior performance of our CoReS, which surpasses the state-of-the-art method by 7.1\% on the ReasonSeg dataset. The code will be released at https://github.com/baoxiaoyi/CoReS.

Title: MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

• Authors: Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, Xiao Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05674
• Pdf URL: https://arxiv.org/pdf/2404.05674
• Copy Paste: [[2404.05674]] MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation(https://arxiv.org/abs/2404.05674)
• Keywords: robust, diffusion, large language model
• Abstract: In this paper, we present MoMA: an open-vocabulary, training-free personalized image model that boasts flexible zero-shot capabilities. As foundational text-to-image models rapidly evolve, the demand for robust image-to-image translation grows. Addressing this need, MoMA specializes in subject-driven personalized image generation. Utilizing an open-source, Multimodal Large Language Model (MLLM), we train MoMA to serve a dual role as both a feature extractor and a generator. This approach effectively synergizes reference image and text prompt information to produce valuable image features, facilitating an image diffusion model. To better leverage the generated features, we further introduce a novel self-attention shortcut method that efficiently transfers image features to an image diffusion model, improving the resemblance of the target object in generated images. Remarkably, as a tuning-free plug-and-play module, our model requires only a single reference image and outperforms existing methods in generating images with high detail fidelity, enhanced identity-preservation and prompt faithfulness. Our work is open-source, thereby providing universal access to these advancements.

Title: SphereHead: Stable 3D Full-head Synthesis with Spherical Tri-plane Representation

• Authors: Heyuan Li, Ce Chen, Tianhao Shi, Yuda Qiu, Sizhe An, Guanying Chen, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05680
• Pdf URL: https://arxiv.org/pdf/2404.05680
• Copy Paste: [[2404.05680]] SphereHead: Stable 3D Full-head Synthesis with Spherical Tri-plane Representation(https://arxiv.org/abs/2404.05680)
• Keywords: generative
• Abstract: While recent advances in 3D-aware Generative Adversarial Networks (GANs) have aided the development of near-frontal view human face synthesis, the challenge of comprehensively synthesizing a full 3D head viewable from all angles still persists. Although PanoHead proves the possibilities of using a large-scale dataset with images of both frontal and back views for full-head synthesis, it often causes artifacts for back views. Based on our in-depth analysis, we found the reasons are mainly twofold. First, from network architecture perspective, we found each plane in the utilized tri-plane/tri-grid representation space tends to confuse the features from both sides, causing "mirroring" artifacts (e.g., the glasses appear in the back). Second, from data supervision aspect, we found that existing discriminator training in 3D GANs mainly focuses on the quality of the rendered image itself, and does not care much about its plausibility with the perspective from which it was rendered. This makes it possible to generate "face" in non-frontal views, due to its easiness to fool the discriminator. In response, we propose SphereHead, a novel tri-plane representation in the spherical coordinate system that fits the human head's geometric characteristics and efficiently mitigates many of the generated artifacts. We further introduce a view-image consistency loss for the discriminator to emphasize the correspondence of the camera parameters and the images. The combination of these efforts results in visually superior outcomes with significantly fewer artifacts. Our code and dataset are publicly available at https://lhyfst.github.io/spherehead.

Title: Retrieval-Augmented Open-Vocabulary Object Detection

• Authors: Jooyeon Kim, Eulrang Cho, Sehyung Kim, Hyunwoo J. Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05687
• Pdf URL: https://arxiv.org/pdf/2404.05687
• Copy Paste: [[2404.05687]] Retrieval-Augmented Open-Vocabulary Object Detection(https://arxiv.org/abs/2404.05687)
• Keywords: large language model
• Abstract: Open-vocabulary object detection (OVD) has been studied with Vision-Language Models (VLMs) to detect novel objects beyond the pre-trained categories. Previous approaches improve the generalization ability to expand the knowledge of the detector, using 'positive' pseudo-labels with additional 'class' names, e.g., sock, iPod, and alligator. To extend the previous methods in two aspects, we propose Retrieval-Augmented Losses and visual Features (RALF). Our method retrieves related 'negative' classes and augments loss functions. Also, visual features are augmented with 'verbalized concepts' of classes, e.g., worn on the feet, handheld music player, and sharp teeth. Specifically, RALF consists of two modules: Retrieval Augmented Losses (RAL) and Retrieval-Augmented visual Features (RAF). RAL constitutes two losses reflecting the semantic similarity with negative vocabularies. In addition, RAF augments visual features with the verbalized concepts from a large language model (LLM). Our experiments demonstrate the effectiveness of RALF on COCO and LVIS benchmark datasets. We achieve improvement up to 3.4 box AP$_{50}^{\text{N}}$ on novel categories of the COCO dataset and 3.6 mask AP$_{\text{r}}$ gains on the LVIS dataset. Code is available at https://github.com/mlvlab/RALF .

Title: David and Goliath: An Empirical Evaluation of Attacks and Defenses for QNNs at the Deep Edge

• Authors: Miguel Costa, Sandro Pinto
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05688
• Pdf URL: https://arxiv.org/pdf/2404.05688
• Copy Paste: [[2404.05688]] David and Goliath: An Empirical Evaluation of Attacks and Defenses for QNNs at the Deep Edge(https://arxiv.org/abs/2404.05688)
• Keywords: defense, attack, robust
• Abstract: ML is shifting from the cloud to the edge. Edge computing reduces the surface exposing private data and enables reliable throughput guarantees in real-time applications. Of the panoply of devices deployed at the edge, resource-constrained MCUs, e.g., Arm Cortex-M, are more prevalent, orders of magnitude cheaper, and less power-hungry than application processors or GPUs. Thus, enabling intelligence at the deep edge is the zeitgeist, with researchers focusing on unveiling novel approaches to deploy ANNs on these constrained devices. Quantization is a well-established technique that has proved effective in enabling the deployment of neural networks on MCUs; however, it is still an open question to understand the robustness of QNNs in the face of adversarial examples. To fill this gap, we empirically evaluate the effectiveness of attacks and defenses from (full-precision) ANNs on (constrained) QNNs. Our evaluation includes three QNNs targeting TinyML applications, ten attacks, and six defenses. With this study, we draw a set of interesting findings. First, quantization increases the point distance to the decision boundary and leads the gradient estimated by some attacks to explode or vanish. Second, quantization can act as a noise attenuator or amplifier, depending on the noise magnitude, and causes gradient misalignment. Regarding adversarial defenses, we conclude that input pre-processing defenses show impressive results on small perturbations; however, they fall short as the perturbation increases. At the same time, train-based defenses increase the average point distance to the decision boundary, which holds after quantization. However, we argue that train-based defenses still need to smooth the quantization-shift and gradient misalignment phenomenons to counteract adversarial example transferability to QNNs. All artifacts are open-sourced to enable independent validation of results.

Title: Evaluating Mathematical Reasoning Beyond Accuracy

• Authors: Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, Pengfei Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05692
• Pdf URL: https://arxiv.org/pdf/2404.05692
• Copy Paste: [[2404.05692]] Evaluating Mathematical Reasoning Beyond Accuracy(https://arxiv.org/abs/2404.05692)
• Keywords: large language model
• Abstract: The leaderboard of Large Language Models (LLMs) in mathematical tasks has been continuously updated. However, the majority of evaluations focus solely on the final results, neglecting the quality of the intermediate steps. This oversight can mask underlying problems, such as logical errors or unnecessary steps in the reasoning process. To measure reasoning beyond final-answer accuracy, we introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps. ReasonEval employs $\textit{validity}$ and $\textit{redundancy}$ to characterize the reasoning quality, as well as accompanying LLMs to assess them automatically. Instantiated by base models that possess strong mathematical knowledge and trained with high-quality labeled data, ReasonEval achieves state-of-the-art performance on human-labeled datasets and can accurately detect different types of errors generated by perturbation. When applied to evaluate LLMs specialized in math, we find that an increase in final-answer accuracy does not necessarily guarantee an improvement in the overall quality of the reasoning steps for challenging mathematical problems. Additionally, we observe that ReasonEval can play a significant role in data selection. We release the best-performing model, meta-evaluation script, and all evaluation results at https://github.com/GAIR-NLP/ReasonEval.

Title: Evaluating the Efficacy of Cut-and-Paste Data Augmentation in Semantic Segmentation for Satellite Imagery

• Authors: Ionut M. Motoi, Leonardo Saraceni, Daniele Nardi, Thomas A. Ciarfuglia
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.05693
• Pdf URL: https://arxiv.org/pdf/2404.05693
• Copy Paste: [[2404.05693]] Evaluating the Efficacy of Cut-and-Paste Data Augmentation in Semantic Segmentation for Satellite Imagery(https://arxiv.org/abs/2404.05693)
• Keywords: segmentation
• Abstract: Satellite imagery is crucial for tasks like environmental monitoring and urban planning. Typically, it relies on semantic segmentation or Land Use Land Cover (LULC) classification to categorize each pixel. Despite the advancements brought about by Deep Neural Networks (DNNs), their performance in segmentation tasks is hindered by challenges such as limited availability of labeled data, class imbalance and the inherent variability and complexity of satellite images. In order to mitigate those issues, our study explores the effectiveness of a Cut-and-Paste augmentation technique for semantic segmentation in satellite images. We adapt this augmentation, which usually requires labeled instances, to the case of semantic segmentation. By leveraging the connected components in the semantic segmentation labels, we extract instances that are then randomly pasted during training. Using the DynamicEarthNet dataset and a U-Net model for evaluation, we found that this augmentation significantly enhances the mIoU score on the test set from 37.9 to 44.1. This finding highlights the potential of the Cut-and-Paste augmentation to improve the generalization capabilities of semantic segmentation models in satellite imagery.

Title: Case Study: Neural Network Malware Detection Verification for Feature and Image Datasets

• Authors: Preston K. Robinette, Diego Manzanas Lopez, Serena Serbinowska, Kevin Leach, Taylor T. Johnson
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05703
• Pdf URL: https://arxiv.org/pdf/2404.05703
• Copy Paste: [[2404.05703]] Case Study: Neural Network Malware Detection Verification for Feature and Image Datasets(https://arxiv.org/abs/2404.05703)
• Keywords: protect, defense, attack, robust
• Abstract: Malware, or software designed with harmful intent, is an ever-evolving threat that can have drastic effects on both individuals and institutions. Neural network malware classification systems are key tools for combating these threats but are vulnerable to adversarial machine learning attacks. These attacks perturb input data to cause misclassification, bypassing protective systems. Existing defenses often rely on enhancing the training process, thereby increasing the model's robustness to these perturbations, which is quantified using verification. While training improvements are necessary, we propose focusing on the verification process used to evaluate improvements to training. As such, we present a case study that evaluates a novel verification domain that will help to ensure tangible safeguards against adversaries and provide a more reliable means of evaluating the robustness and effectiveness of anti-malware systems. To do so, we describe malware classification and two types of common malware datasets (feature and image datasets), demonstrate the certified robustness accuracy of malware classifiers using the Neural Network Verification (NNV) and Neural Network Enumeration (nnenum) tools, and outline the challenges and future considerations necessary for the improvement and refinement of the verification of malware classification. By evaluating this novel domain as a case study, we hope to increase its visibility, encourage further research and scrutiny, and ultimately enhance the resilience of digital systems against malicious attacks.

Title: Learning 3D-Aware GANs from Unposed Images with Template Feature Field

• Authors: Xinya Chen, Hanlei Guo, Yanrui Bin, Shangzhan Zhang, Yuanbo Yang, Yue Wang, Yujun Shen, Yiyi Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05705
• Pdf URL: https://arxiv.org/pdf/2404.05705
• Copy Paste: [[2404.05705]] Learning 3D-Aware GANs from Unposed Images with Template Feature Field(https://arxiv.org/abs/2404.05705)
• Keywords: generative
• Abstract: Collecting accurate camera poses of training images has been shown to well serve the learning of 3D-aware generative adversarial networks (GANs) yet can be quite expensive in practice. This work targets learning 3D-aware GANs from unposed images, for which we propose to perform on-the-fly pose estimation of training images with a learned template feature field (TeFF). Concretely, in addition to a generative radiance field as in previous approaches, we ask the generator to also learn a field from 2D semantic features while sharing the density from the radiance field. Such a framework allows us to acquire a canonical 3D feature template leveraging the dataset mean discovered by the generative model, and further efficiently estimate the pose parameters on real data. Experimental results on various challenging datasets demonstrate the superiority of our approach over state-of-the-art alternatives from both the qualitative and the quantitative perspectives.

Title: Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

• Authors: Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, Zhe Gan
• Subjects: cs.CV, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.05719
• Pdf URL: https://arxiv.org/pdf/2404.05719
• Copy Paste: [[2404.05719]] Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs(https://arxiv.org/abs/2404.05719)
• Keywords: large language model
• Abstract: Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest (e.g., icons, texts) than natural images, we incorporate "any resolution" on top of Ferret to magnify details and leverage enhanced visual features. Specifically, each screen is divided into 2 sub-images based on the original aspect ratio (i.e., horizontal division for portrait screens and vertical division for landscape screens). Both sub-images are encoded separately before being sent to LLMs. We meticulously gather training samples from an extensive range of elementary UI tasks, such as icon recognition, find text, and widget listing. These samples are formatted for instruction-following with region annotations to facilitate precise referring and grounding. To augment the model's reasoning ability, we further compile a dataset for advanced tasks, including detailed description, perception/interaction conversations, and function inference. After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions. For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks. Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.

Title: MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding

• Authors: Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, Ser-Nam Lim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05726
• Pdf URL: https://arxiv.org/pdf/2404.05726
• Copy Paste: [[2404.05726]] MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding(https://arxiv.org/abs/2404.05726)
• Keywords: large language model
• Abstract: With the success of large language models (LLMs), integrating the vision model into LLMs to build vision-language foundation models has gained much more interest recently. However, existing LLM-based large multimodal models (e.g., Video-LLaMA, VideoChat) can only take in a limited number of frames for short video understanding. In this study, we mainly focus on designing an efficient and effective model for long-term video understanding. Instead of trying to process more frames simultaneously like most existing work, we propose to process videos in an online manner and store past video information in a memory bank. This allows our model to reference historical video content for long-term analysis without exceeding LLMs' context length constraints or GPU memory limits. Our memory bank can be seamlessly integrated into current multimodal LLMs in an off-the-shelf manner. We conduct extensive experiments on various video understanding tasks, such as long-video understanding, video question answering, and video captioning, and our model can achieve state-of-the-art performances across multiple datasets. Code available at https://boheumd.github.io/MA-LMM/.

Title: A Large-Scale Exploration of $μ$-Transfer

• Authors: Lucas Lingle
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05728
• Pdf URL: https://arxiv.org/pdf/2404.05728
• Copy Paste: [[2404.05728]] A Large-Scale Exploration of $μ$-Transfer(https://arxiv.org/abs/2404.05728)
• Keywords: transformer
• Abstract: Large neural network models have become a mainstay of natural language processing and computer vision, yet their initialization and learning rates are set in a largely heuristic fashion, potentially varying from paper to paper and one model size to the next. The $\mu$-Parameterization ($\mu$P) offers a potential solution to these challenges, yielding scaling rules for model initialization and learning rates, and reportedly enabling zero-shot hyperparameter transfer from small to large models in a variety of cases. Despite the evident promise, the $\mu$P scaling rules are not yet widely adopted, perhaps due to higher implementation complexity, many variations, or complex theoretical background. This work investigates $\mu$P empirically, focusing on the ubiquitous transformer architecture, and aims to answer a simple question: does $\mu$-Transfer yield optimal learning rates in practice? From models with 2M to 10B parameters, we show that $\mu$-Transfer works as intended for the majority of important cases, but also identify some surprising cases where it may not.