2024-05-03

Title: SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuning

• Authors: Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang, Ang Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00705
• Pdf URL: https://arxiv.org/pdf/2405.00705
• Copy Paste: [[2405.00705]] SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuning(https://arxiv.org/abs/2405.00705)
• Keywords: large language model
• Abstract: The pre-trained Large Language Models (LLMs) can be adapted for many downstream tasks and tailored to align with human preferences through fine-tuning. Recent studies have discovered that LLMs can achieve desirable performance with only a small amount of high-quality data, suggesting that a large amount of the data in these extensive datasets is redundant or even harmful. Identifying high-quality data from vast datasets to curate small yet effective datasets has emerged as a critical challenge. In this paper, we introduce SHED, an automated dataset refinement framework based on Shapley value for instruction fine-tuning. SHED eliminates the need for human intervention or the use of commercial LLMs. Moreover, the datasets curated through SHED exhibit transferability, indicating they can be reused across different LLMs with consistently high performance. We conduct extensive experiments to evaluate the datasets curated by SHED. The results demonstrate SHED's superiority over state-of-the-art methods across various tasks and LLMs; notably, datasets comprising only 10% of the original data selected by SHED achieve performance comparable to or surpassing that of the full datasets.

Title: Science Written by Generative AI is Perceived as Less Intelligent, but More Credible and Trustworthy than Science Written by Humans

• Authors: David M. Markowitz
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2405.00706
• Pdf URL: https://arxiv.org/pdf/2405.00706
• Copy Paste: [[2405.00706]] Science Written by Generative AI is Perceived as Less Intelligent, but More Credible and Trustworthy than Science Written by Humans(https://arxiv.org/abs/2405.00706)
• Keywords: generative
• Abstract: This paper evaluated the effectiveness of using generative AI to simplify science communication and enhance public trust in science. By comparing lay summaries of journal articles from PNAS, yoked to those generated by AI, this work assessed linguistic simplicity across such summaries and public perceptions. Study 1a analyzed simplicity features of PNAS abstracts (scientific summaries) and significance statements (lay summaries), observing that lay summaries were indeed linguistically simpler, but effect size differences were small. Study 1b used GPT-4 to create significance statements based on paper abstracts and this more than doubled the average effect size without fine-tuning. Finally, Study 2 experimentally demonstrated that simply-written GPT summaries facilitated more favorable public perceptions of scientists (their credibility, trustworthiness) than more complexly-written human PNAS summaries. AI has the potential to engage scientific communities and the public via a simple language heuristic, advocating for its integration into scientific dissemination for a more informed society.

Title: Evaluating Tool-Augmented Agents in Remote Sensing Platforms

• Authors: Simranjit Singh, Michael Fore, Dimitrios Stamoulis
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00709
• Pdf URL: https://arxiv.org/pdf/2405.00709
• Copy Paste: [[2405.00709]] Evaluating Tool-Augmented Agents in Remote Sensing Platforms(https://arxiv.org/abs/2405.00709)
• Keywords: large language model
• Abstract: Tool-augmented Large Language Models (LLMs) have shown impressive capabilities in remote sensing (RS) applications. However, existing benchmarks assume question-answering input templates over predefined image-text data pairs. These standalone instructions neglect the intricacies of realistic user-grounded tasks. Consider a geospatial analyst: they zoom in a map area, they draw a region over which to collect satellite imagery, and they succinctly ask "Detect all objects here". Where is `here`, if it is not explicitly hardcoded in the image-text template, but instead is implied by the system state, e.g., the live map positioning? To bridge this gap, we present GeoLLM-QA, a benchmark designed to capture long sequences of verbal, visual, and click-based actions on a real UI platform. Through in-depth evaluation of state-of-the-art LLMs over a diverse set of 1,000 tasks, we offer insights towards stronger agents for RS applications.

Title: Homonym Sense Disambiguation in the Georgian Language

• Authors: Davit Melikidze, Alexander Gamkrelidze
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00710
• Pdf URL: https://arxiv.org/pdf/2405.00710
• Copy Paste: [[2405.00710]] Homonym Sense Disambiguation in the Georgian Language(https://arxiv.org/abs/2405.00710)
• Keywords: large language model
• Abstract: This research proposes a novel approach to the Word Sense Disambiguation (WSD) task in the Georgian language, based on supervised fine-tuning of a pre-trained Large Language Model (LLM) on a dataset formed by filtering the Georgian Common Crawls corpus. The dataset is used to train a classifier for words with multiple senses. Additionally, we present experimental results of using LSTM for WSD. Accurately disambiguating homonyms is crucial in natural language processing. Georgian, an agglutinative language belonging to the Kartvelian language family, presents unique challenges in this context. The aim of this paper is to highlight the specific problems concerning homonym disambiguation in the Georgian language and to present our approach to solving them. The techniques discussed in the article achieve 95% accuracy for predicting lexical meanings of homonyms using a hand-classified dataset of over 7500 sentences.

Title: Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunities

• Authors: Xiaomin Yu, Yezhaohui Wang, Yanfang Chen, Zhen Tao, Dinghao Xi, Shichao Song, Simin Niu
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2405.00711
• Pdf URL: https://arxiv.org/pdf/2405.00711
• Copy Paste: [[2405.00711]] Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunities(https://arxiv.org/abs/2405.00711)
• Keywords: diffusion, generative, large language model
• Abstract: In recent years, generative artificial intelligence models, represented by Large Language Models (LLMs) and Diffusion Models (DMs), have revolutionized content production methods. These artificial intelligence-generated content (AIGC) have become deeply embedded in various aspects of daily life and work, spanning texts, images, videos, and audio. The authenticity of AI-generated content is progressively enhancing, approaching human-level creative standards. However, these technologies have also led to the emergence of Fake Artificial Intelligence Generated Content (FAIGC), posing new challenges in distinguishing genuine information. It is crucial to recognize that AIGC technology is akin to a double-edged sword; its potent generative capabilities, while beneficial, also pose risks for the creation and dissemination of FAIGC. In this survey, We propose a new taxonomy that provides a more comprehensive breakdown of the space of FAIGC methods today. Next, we explore the modalities and generative technologies of FAIGC, categorized under AI-generated disinformation and AI-generated misinformation. From various perspectives, we then introduce FAIGC detection methods, including Deceptive FAIGC Detection, Deepfake Detection, and Hallucination-based FAIGC Detection. Finally, we discuss outstanding challenges and promising areas for future research.

Title: Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation

• Authors: Hanyin Wang, Chufan Gao, Bolun Liu, Qiping Xu, Guleid Hussein, Mohamad El Labban, Kingsley Iheasirim, Hariprasad Korsapati, Jimeng Sun
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00715
• Pdf URL: https://arxiv.org/pdf/2405.00715
• Copy Paste: [[2405.00715]] Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation(https://arxiv.org/abs/2405.00715)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown promising capabilities in handling clinical text summarization tasks. In this study, we demonstrate that a small open-source LLM can be effectively trained to generate high-quality clinical notes from outpatient patient-doctor dialogues. We achieve this through a comprehensive domain- and task-specific adaptation process for the LLaMA-2 13 billion parameter model. This process incorporates continued pre-training, supervised fine-tuning, and reinforcement learning from both AI and human feedback. We introduced an enhanced approach, termed DistillDirect, for performing on-policy reinforcement learning with Gemini Pro serving as the teacher model. Our resulting model, LLaMA-Clinic, is capable of generating clinical notes that are comparable in quality to those authored by physicians. In a blinded physician reader study, the majority (90.4%) of individual evaluations rated the notes generated by LLaMA-Clinic as "acceptable" or higher across all three criteria: real-world readiness, completeness, and accuracy. Notably, in the more challenging "Assessment and Plan" section, LLaMA-Clinic scored higher (4.2/5) in real-world readiness compared to physician-authored notes (4.1/5). Additionally, we identified caveats in public clinical note datasets, such as ACI-BENCH. We highlight key considerations for future clinical note-generation tasks, emphasizing the importance of pre-defining a best-practice note format. Overall, our research demonstrates the potential and feasibility of training smaller, open-source LLMs to assist with clinical documentation, capitalizing on healthcare institutions' access to patient records and domain expertise. We have made our newly created synthetic clinic dialogue-note dataset and the physician feedback dataset publicly available to foster future research in this field.

Title: Large Language Models in Healthcare: A Comprehensive Benchmark

• Authors: Andrew Liu, Hongjian Zhou, Yining Hua, Omid Rohanian, Lei Clifton, David A. Clifton
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00716
• Pdf URL: https://arxiv.org/pdf/2405.00716
• Copy Paste: [[2405.00716]] Large Language Models in Healthcare: A Comprehensive Benchmark(https://arxiv.org/abs/2405.00716)
• Keywords: robust, large language model
• Abstract: The adoption of large language models (LLMs) to assist clinicians has attracted remarkable attention. Existing works mainly adopt the close-ended question-answering task with answer options for evaluation. However, in real clinical settings, many clinical decisions, such as treatment recommendations, involve answering open-ended questions without pre-set options. Meanwhile, existing studies mainly use accuracy to assess model performance. In this paper, we comprehensively benchmark diverse LLMs in healthcare, to clearly understand their strengths and weaknesses. Our benchmark contains seven tasks and thirteen datasets across medical language generation, understanding, and reasoning. We conduct a detailed evaluation of the existing sixteen LLMs in healthcare under both zero-shot and few-shot (i.e., 1,3,5-shot) learning settings. We report the results on five metrics (i.e. matching, faithfulness, comprehensiveness, generalizability, and robustness) that are critical in achieving trust from clinical users. We further invite medical experts to conduct human evaluation.

Title: Can't say cant? Measuring and Reasoning of Dark Jargons in Large Language Models

• Authors: Xu Ji, Jianyi Zhang, Ziyin Zhou, Zhangchi Zhao, Qianqian Qiao, Kaiying Han, Md Imran Hossen, Xiali Hei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00718
• Pdf URL: https://arxiv.org/pdf/2405.00718
• Copy Paste: [[2405.00718]] Can't say cant? Measuring and Reasoning of Dark Jargons in Large Language Models(https://arxiv.org/abs/2405.00718)
• Keywords: diffusion, large language model
• Abstract: Ensuring the resilience of Large Language Models (LLMs) against malicious exploitation is paramount, with recent focus on mitigating offensive responses. Yet, the understanding of cant or dark jargon remains unexplored. This paper introduces a domain-specific Cant dataset and CantCounter evaluation framework, employing Fine-Tuning, Co-Tuning, Data-Diffusion, and Data-Analysis stages. Experiments reveal LLMs, including ChatGPT, are susceptible to cant bypassing filters, with varying recognition accuracy influenced by question types, setups, and prompt clues. Updated models exhibit higher acceptance rates for cant queries. Moreover, LLM reactions differ across domains, e.g., reluctance to engage in racism versus LGBT topics. These findings underscore LLMs' understanding of cant and reflect training data characteristics and vendor approaches to sensitive topics. Additionally, we assess LLMs' ability to demonstrate reasoning capabilities. Access to our datasets and code is available at https://github.com/cistineup/CantCounter.

Title: LLMs for Generating and Evaluating Counterfactuals: A Comprehensive Study

• Authors: Van Bach Nguyen, Paul Youssef, Jörg Schlötterer, Christin Seifert
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00722
• Pdf URL: https://arxiv.org/pdf/2405.00722
• Copy Paste: [[2405.00722]] LLMs for Generating and Evaluating Counterfactuals: A Comprehensive Study(https://arxiv.org/abs/2405.00722)
• Keywords: robust, large language model
• Abstract: As NLP models become more complex, understanding their decisions becomes more crucial. Counterfactuals (CFs), where minimal changes to inputs flip a model's prediction, offer a way to explain these models. While Large Language Models (LLMs) have shown remarkable performance in NLP tasks, their efficacy in generating high-quality CFs remains uncertain. This work fills this gap by investigating how well LLMs generate CFs for two NLU tasks. We conduct a comprehensive comparison of several common LLMs, and evaluate their CFs, assessing both intrinsic metrics, and the impact of these CFs on data augmentation. Moreover, we analyze differences between human and LLM-generated CFs, providing insights for future research directions. Our results show that LLMs generate fluent CFs, but struggle to keep the induced changes minimal. Generating CFs for Sentiment Analysis (SA) is less challenging than NLI where LLMs show weaknesses in generating CFs that flip the original label. This also reflects on the data augmentation performance, where we observe a large gap between augmenting with human and LLMs CFs. Furthermore, we evaluate LLMs' ability to assess CFs in a mislabelled data setting, and show that they have a strong bias towards agreeing with the provided labels. GPT4 is more robust against this bias and its scores correlate well with automatic metrics. Our findings reveal several limitations and point to potential future work directions.

Title: Evaluating the Application of ChatGPT in Outpatient Triage Guidance: A Comparative Study

• Authors: Dou Liu, Ying Han, Xiandi Wang, Xiaomei Tan, Di Liu, Guangwu Qian, Kang Li, Dan Pu, Rong Yin
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2405.00728
• Pdf URL: https://arxiv.org/pdf/2405.00728
• Copy Paste: [[2405.00728]] Evaluating the Application of ChatGPT in Outpatient Triage Guidance: A Comparative Study(https://arxiv.org/abs/2405.00728)
• Keywords: large language model
• Abstract: The integration of Artificial Intelligence (AI) in healthcare presents a transformative potential for enhancing operational efficiency and health outcomes. Large Language Models (LLMs), such as ChatGPT, have shown their capabilities in supporting medical decision-making. Embedding LLMs in medical systems is becoming a promising trend in healthcare development. The potential of ChatGPT to address the triage problem in emergency departments has been examined, while few studies have explored its application in outpatient departments. With a focus on streamlining workflows and enhancing efficiency for outpatient triage, this study specifically aims to evaluate the consistency of responses provided by ChatGPT in outpatient guidance, including both within-version response analysis and between-version comparisons. For within-version, the results indicate that the internal response consistency for ChatGPT-4.0 is significantly higher than ChatGPT-3.5 (p=0.03) and both have a moderate consistency (71.2% for 4.0 and 59.6% for 3.5) in their top recommendation. However, the between-version consistency is relatively low (mean consistency score=1.43/3, median=1), indicating few recommendations match between the two versions. Also, only 50% top recommendations match perfectly in the comparisons. Interestingly, ChatGPT-3.5 responses are more likely to be complete than those from ChatGPT-4.0 (p=0.02), suggesting possible differences in information processing and response generation between the two versions. The findings offer insights into AI-assisted outpatient operations, while also facilitating the exploration of potentials and limitations of LLMs in healthcare utilization. Future research may focus on carefully optimizing LLMs and AI integration in healthcare systems based on ergonomic and human factors principles, precisely aligning with the specific needs of effective outpatient triage.

Title: LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report

• Authors: Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, Devvret Rishi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00732
• Pdf URL: https://arxiv.org/pdf/2405.00732
• Copy Paste: [[2405.00732]] LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report(https://arxiv.org/abs/2405.00732)
• Keywords: large language model
• Abstract: Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted methods for Parameter Efficient Fine-Tuning (PEFT) of Large Language Models (LLMs). LoRA reduces the number of trainable parameters and memory usage while achieving comparable performance to full fine-tuning. We aim to assess the viability of training and serving LLMs fine-tuned with LoRA in real-world applications. First, we measure the quality of LLMs fine-tuned with quantized low rank adapters across 10 base models and 31 tasks for a total of 310 models. We find that 4-bit LoRA fine-tuned models outperform base models by 34 points and GPT-4 by 10 points on average. Second, we investigate the most effective base models for fine-tuning and assess the correlative and predictive capacities of task complexity heuristics in forecasting the outcomes of fine-tuning. Finally, we evaluate the latency and concurrency capabilities of LoRAX, an open-source Multi-LoRA inference server that facilitates the deployment of multiple LoRA fine-tuned models on a single GPU using shared base model weights and dynamic adapter loading. LoRAX powers LoRA Land, a web application that hosts 25 LoRA fine-tuned Mistral-7B LLMs on a single NVIDIA A100 GPU with 80GB memory. LoRA Land highlights the quality and cost-effectiveness of employing multiple specialized LLMs over a single, general-purpose LLM.

Title: Federated Graph Learning for EV Charging Demand Forecasting with Personalization Against Cyberattacks

• Authors: Yi Li, Renyou Xie, Chaojie Li, Yi Wang, Zhaoyang Dong
• Subjects: cs.CR, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2405.00742
• Pdf URL: https://arxiv.org/pdf/2405.00742
• Copy Paste: [[2405.00742]] Federated Graph Learning for EV Charging Demand Forecasting with Personalization Against Cyberattacks(https://arxiv.org/abs/2405.00742)
• Keywords: security, privacy, attack, robust, federate
• Abstract: Mitigating cybersecurity risk in electric vehicle (EV) charging demand forecasting plays a crucial role in the safe operation of collective EV chargings, the stability of the power grid, and the cost-effective infrastructure expansion. However, existing methods either suffer from the data privacy issue and the susceptibility to cyberattacks or fail to consider the spatial correlation among different stations. To address these challenges, a federated graph learning approach involving multiple charging stations is proposed to collaboratively train a more generalized deep learning model for demand forecasting while capturing spatial correlations among various stations and enhancing robustness against potential attacks. Firstly, for better model performance, a Graph Neural Network (GNN) model is leveraged to characterize the geographic correlation among different charging stations in a federated manner. Secondly, to ensure robustness and deal with the data heterogeneity in a federated setting, a message passing that utilizes a global attention mechanism to aggregate personalized models for each client is proposed. Thirdly, by concerning cyberattacks, a special credit-based function is designed to mitigate potential threats from malicious clients or unwanted attacks. Extensive experiments on a public EV charging dataset are conducted using various deep learning techniques and federated learning methods to demonstrate the prediction accuracy and robustness of the proposed approach.

Title: Soft Preference Optimization: Aligning Language Models to Expert Distributions

• Authors: Arsalan Sharifnassab, Sina Ghiassian, Saber Salehkaleybar, Surya Kanoria, Dale Schuurmans
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00747
• Pdf URL: https://arxiv.org/pdf/2405.00747
• Copy Paste: [[2405.00747]] Soft Preference Optimization: Aligning Language Models to Expert Distributions(https://arxiv.org/abs/2405.00747)
• Keywords: generative, large language model
• Abstract: We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.

Title: CLIPArTT: Light-weight Adaptation of CLIP to New Domains at Test Time

• Authors: Gustavo Adolfo Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, Christian Desrosiers
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00754
• Pdf URL: https://arxiv.org/pdf/2405.00754
• Copy Paste: [[2405.00754]] CLIPArTT: Light-weight Adaptation of CLIP to New Domains at Test Time(https://arxiv.org/abs/2405.00754)
• Keywords: robust
• Abstract: Pre-trained vision-language models (VLMs), exemplified by CLIP, demonstrate remarkable adaptability across zero-shot classification tasks without additional training. However, their performance diminishes in the presence of domain shifts. In this study, we introduce CLIP Adaptation duRing Test-Time (CLIPArTT), a fully test-time adaptation (TTA) approach for CLIP, which involves automatic text prompts construction during inference for their use as text supervision. Our method employs a unique, minimally invasive text prompt tuning process, wherein multiple predicted classes are aggregated into a single new text prompt, used as pseudo label to re-classify inputs in a transductive manner. Additionally, we pioneer the standardization of TTA benchmarks (e.g., TENT) in the realm of VLMs. Our findings demonstrate that, without requiring additional transformations nor new trainable modules, CLIPArTT enhances performance dynamically across non-corrupted datasets such as CIFAR-10, corrupted datasets like CIFAR-10-C and CIFAR-10.1, alongside synthetic datasets such as VisDA-C. This research underscores the potential for improving VLMs' adaptability through novel test-time strategies, offering insights for robust performance across varied datasets and environments. The code can be found at: https://github.com/dosowiechi/CLIPArTT.git

Title: Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models

• Authors: Xiaoshi Wu, Yiming Hao, Manyuan Zhang, Keqiang Sun, Zhaoyang Huang, Guanglu Song, Yu Liu, Hongsheng Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00760
• Pdf URL: https://arxiv.org/pdf/2405.00760
• Copy Paste: [[2405.00760]] Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models(https://arxiv.org/abs/2405.00760)
• Keywords: diffusion
• Abstract: Optimizing a text-to-image diffusion model with a given reward function is an important but underexplored research area. In this study, we propose Deep Reward Tuning (DRTune), an algorithm that directly supervises the final output image of a text-to-image diffusion model and back-propagates through the iterative sampling process to the input noise. We find that training earlier steps in the sampling process is crucial for low-level rewards, and deep supervision can be achieved efficiently and effectively by stopping the gradient of the denoising network input. DRTune is extensively evaluated on various reward models. It consistently outperforms other algorithms, particularly for low-level control signals, where all shallow supervision methods fail. Additionally, we fine-tune Stable Diffusion XL 1.0 (SDXL 1.0) model via DRTune to optimize Human Preference Score v2.1, resulting in the Favorable Diffusion XL 1.0 (FDXL 1.0) model. FDXL 1.0 significantly enhances image quality compared to SDXL 1.0 and reaches comparable quality compared with Midjourney v5.2.

Title: Obtaining Favorable Layouts for Multiple Object Generation

• Authors: Barak Battash, Amit Rozner, Lior Wolf, Ofir Lindenbaum
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00791
• Pdf URL: https://arxiv.org/pdf/2405.00791
• Copy Paste: [[2405.00791]] Obtaining Favorable Layouts for Multiple Object Generation(https://arxiv.org/abs/2405.00791)
• Keywords: diffusion
• Abstract: Large-scale text-to-image models that can generate high-quality and diverse images based on textual prompts have shown remarkable success. These models aim ultimately to create complex scenes, and addressing the challenge of multi-subject generation is a critical step towards this goal. However, the existing state-of-the-art diffusion models face difficulty when generating images that involve multiple subjects. When presented with a prompt containing more than one subject, these models may omit some subjects or merge them together. To address this challenge, we propose a novel approach based on a guiding principle. We allow the diffusion model to initially propose a layout, and then we rearrange the layout grid. This is achieved by enforcing cross-attention maps (XAMs) to adhere to proposed masks and by migrating pixels from latent maps to new locations determined by us. We introduce new loss terms aimed at reducing XAM entropy for clearer spatial definition of subjects, reduce the overlap between XAMs, and ensure that XAMs align with their respective masks. We contrast our approach with several alternative methods and show that it more faithfully captures the desired concepts across a variety of text prompts.

Title: The Impact of IMSI Catcher Deployments on Cellular Network Security: Challenges and Countermeasures in 4G and 5G Networks

• Authors: Karwan Mustafa Kareem
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.00793
• Pdf URL: https://arxiv.org/pdf/2405.00793
• Copy Paste: [[2405.00793]] The Impact of IMSI Catcher Deployments on Cellular Network Security: Challenges and Countermeasures in 4G and 5G Networks(https://arxiv.org/abs/2405.00793)
• Keywords: security, privacy, protect
• Abstract: IMSI (International Mobile Subscriber Identity) catchers, also known as "Stingrays" or "cell site simulators," are rogue devices that pose a significant threat to cellular network security [1]. IMSI catchers can intercept and manipulate cellular communications, compromising the privacy and security of mobile devices and their users. With the advent of 4G and 5G networks, IMSI catchers have become more sophisticated and pose new challenges to cellular network security [2]. This paper provides an overview of the impact of IMSI catcher deployments on cellular network security in the context of 4G and 5G networks. It discusses the challenges posed by IMSI catchers, including the unauthorized collection of IMSI numbers, interception of communications, and potential misuse of subscriber information. It also highlights the potential consequences of IMSI catcher deployments, including the compromise of user privacy, financial fraud, and unauthorized surveillance. The paper further reviews the countermeasures that can be employed to mitigate the risks posed by IMSI catchers. These countermeasures include network-based solutions such as signal analysis, encryption, and authentication mechanisms, as well as user-based solutions such as mobile applications and device settings. The paper also discusses the limitations and effectiveness of these countermeasures in the context of 4G and 5G networks. Finally, the paper identifies research gaps and future directions for enhancing cellular network security against IMSI catchers in the era of 4G and 5G networks. This includes the need for improved encryption algorithms, authentication mechanisms, and detection techniques to effectively detect and prevent IMSI catcher deployments. The paper also emphasizes the importance of regulatory and policy measures to govern the deployment and use of IMSI catchers to protect user privacy and security.

Title: "Ask Me Anything": How Comcast Uses LLMs to Assist Agents in Real Time

• Authors: Scott Rome, Tianwen Chen, Raphael Tang, Luwei Zhou, Ferhan Ture
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.00801
• Pdf URL: https://arxiv.org/pdf/2405.00801
• Copy Paste: [[2405.00801]] "Ask Me Anything": How Comcast Uses LLMs to Assist Agents in Real Time(https://arxiv.org/abs/2405.00801)
• Keywords: large language model
• Abstract: Customer service is how companies interface with their customers. It can contribute heavily towards the overall customer satisfaction. However, high-quality service can become expensive, creating an incentive to make it as cost efficient as possible and prompting most companies to utilize AI-powered assistants, or "chat bots". On the other hand, human-to-human interaction is still desired by customers, especially when it comes to complex scenarios such as disputes and sensitive topics like bill payment. This raises the bar for customer service agents. They need to accurately understand the customer's question or concern, identify a solution that is acceptable yet feasible (and within the company's policy), all while handling multiple conversations at once. In this work, we introduce "Ask Me Anything" (AMA) as an add-on feature to an agent-facing customer service interface. AMA allows agents to ask questions to a large language model (LLM) on demand, as they are handling customer conversations -- the LLM provides accurate responses in real-time, reducing the amount of context switching the agent needs. In our internal experiments, we find that agents using AMA versus a traditional search experience spend approximately 10% fewer seconds per conversation containing a search, translating to millions of dollars of savings annually. Agents that used the AMA feature provided positive feedback nearly 80% of the time, demonstrating its usefulness as an AI-assisted feature for customer care.

Title: ICU Bloodstream Infection Prediction: A Transformer-Based Approach for EHR Analysis

• Authors: Ortal Hirszowicz, Dvir Aran
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00819
• Pdf URL: https://arxiv.org/pdf/2405.00819
• Copy Paste: [[2405.00819]] ICU Bloodstream Infection Prediction: A Transformer-Based Approach for EHR Analysis(https://arxiv.org/abs/2405.00819)
• Keywords: transformer
• Abstract: We introduce RatchetEHR, a novel transformer-based framework designed for the predictive analysis of electronic health records (EHR) data in intensive care unit (ICU) settings, with a specific focus on bloodstream infection (BSI) prediction. Leveraging the MIMIC-IV dataset, RatchetEHR demonstrates superior predictive performance compared to other methods, including RNN, LSTM, and XGBoost, particularly due to its advanced handling of sequential and temporal EHR data. A key innovation in RatchetEHR is the integration of the Graph Convolutional Transformer (GCT) component, which significantly enhances the ability to identify hidden structural relationships within EHR data, resulting in more accurate clinical predictions. Through SHAP value analysis, we provide insights into influential features for BSI prediction. RatchetEHR integrates multiple advancements in deep learning which together provide accurate predictions even with a relatively small sample size and highly imbalanced dataset. This study contributes to medical informatics by showcasing the application of advanced AI techniques in healthcare and sets a foundation for further research to optimize these capabilities in EHR data analysis.

Title: WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting

• Authors: Olly Styles, Sam Miller, Patricio Cerda-Mardini, Tanaya Guha, Victor Sanchez, Bertie Vidgen
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2405.00823
• Pdf URL: https://arxiv.org/pdf/2405.00823
• Copy Paste: [[2405.00823]] WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting(https://arxiv.org/abs/2405.00823)
• Keywords: robust
• Abstract: We introduce WorkBench: a benchmark dataset for evaluating agents' ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents' errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents' ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/olly-styles/WorkBench.

Title: WIBA: What Is Being Argued? A Comprehensive Approach to Argument Mining

• Authors: Arman Irani, Ju Yeon Park, Kevin Esterling, Michalis Faloutsos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.00828
• Pdf URL: https://arxiv.org/pdf/2405.00828
• Copy Paste: [[2405.00828]] WIBA: What Is Being Argued? A Comprehensive Approach to Argument Mining(https://arxiv.org/abs/2405.00828)
• Keywords: large language model
• Abstract: We propose WIBA, a novel framework and suite of methods that enable the comprehensive understanding of "What Is Being Argued" across contexts. Our approach develops a comprehensive framework that detects: (a) the existence, (b) the topic, and (c) the stance of an argument, correctly accounting for the logical dependence among the three tasks. Our algorithm leverages the fine-tuning and prompt-engineering of Large Language Models. We evaluate our approach and show that it performs well in all the three capabilities. First, we develop and release an Argument Detection model that can classify a piece of text as an argument with an F1 score between 79% and 86% on three different benchmark datasets. Second, we release a language model that can identify the topic being argued in a sentence, be it implicit or explicit, with an average similarity score of 71%, outperforming current naive methods by nearly 40%. Finally, we develop a method for Argument Stance Classification, and evaluate the capability of our approach, showing it achieves a classification F1 score between 71% and 78% across three diverse benchmark datasets. Our evaluation demonstrates that WIBA allows the comprehensive understanding of What Is Being Argued in large corpora across diverse contexts, which is of core interest to many applications in linguistics, communication, and social and computer science. To facilitate accessibility to the advancements outlined in this work, we release WIBA as a free open access platform (wiba.dev).

Title: Locality Regularized Reconstruction: Structured Sparsity and Delaunay Triangulations

• Authors: Marshall Mueller, James M. Murphy, Abiy Tasissa
• Subjects: cs.LG, eess.SP, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2405.00837
• Pdf URL: https://arxiv.org/pdf/2405.00837
• Copy Paste: [[2405.00837]] Locality Regularized Reconstruction: Structured Sparsity and Delaunay Triangulations(https://arxiv.org/abs/2405.00837)
• Keywords: extraction
• Abstract: Linear representation learning is widely studied due to its conceptual simplicity and empirical utility in tasks such as compression, classification, and feature extraction. Given a set of points $[\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n] = \mathbf{X} \in \mathbb{R}^{d \times n}$ and a vector $\mathbf{y} \in \mathbb{R}^d$, the goal is to find coefficients $\mathbf{w} \in \mathbb{R}^n$ so that $\mathbf{X} \mathbf{w} \approx \mathbf{y}$, subject to some desired structure on $\mathbf{w}$. In this work we seek $\mathbf{w}$ that forms a local reconstruction of $\mathbf{y}$ by solving a regularized least squares regression problem. We obtain local solutions through a locality function that promotes the use of columns of $\mathbf{X}$ that are close to $\mathbf{y}$ when used as a regularization term. We prove that, for all levels of regularization and under a mild condition that the columns of $\mathbf{X}$ have a unique Delaunay triangulation, the optimal coefficients' number of non-zero entries is upper bounded by $d+1$, thereby providing local sparse solutions when $d \ll n$. Under the same condition we also show that for any $\mathbf{y}$ contained in the convex hull of $\mathbf{X}$ there exists a regime of regularization parameter such that the optimal coefficients are supported on the vertices of the Delaunay simplex containing $\mathbf{y}$. This provides an interpretation of the sparsity as having structure obtained implicitly from the Delaunay triangulation of $\mathbf{X}$. We demonstrate that our locality regularized problem can be solved in comparable time to other methods that identify the containing Delaunay simplex.

Title: Communication-Efficient Training Workload Balancing for Decentralized Multi-Agent Learning

• Authors: Seyed Mahmoud Sajjadi Mohammadabadi, Lei Yang, Feng Yan, Junshan Zhang
• Subjects: cs.LG, cs.AI, cs.DC, cs.MA, cs.PF
• Abstract URL: https://arxiv.org/abs/2405.00839
• Pdf URL: https://arxiv.org/pdf/2405.00839
• Copy Paste: [[2405.00839]] Communication-Efficient Training Workload Balancing for Decentralized Multi-Agent Learning(https://arxiv.org/abs/2405.00839)
• Keywords: privacy, protect, robust
• Abstract: Decentralized Multi-agent Learning (DML) enables collaborative model training while preserving data privacy. However, inherent heterogeneity in agents' resources (computation, communication, and task size) may lead to substantial variations in training time. This heterogeneity creates a bottleneck, lengthening the overall training time due to straggler effects and potentially wasting spare resources of faster agents. To minimize training time in heterogeneous environments, we present a Communication-Efficient Training Workload Balancing for Decentralized Multi-Agent Learning (ComDML), which balances the workload among agents through a decentralized approach. Leveraging local-loss split training, ComDML enables parallel updates, where slower agents offload part of their workload to faster agents. To minimize the overall training time, ComDML optimizes the workload balancing by jointly considering the communication and computation capacities of agents, which hinges upon integer programming. A dynamic decentralized pairing scheduler is developed to efficiently pair agents and determine optimal offloading amounts. We prove that in ComDML, both slower and faster agents' models converge, for convex and non-convex functions. Furthermore, extensive experimental results on popular datasets (CIFAR-10, CIFAR-100, and CINIC-10) and their non-I.I.D. variants, with large models such as ResNet-56 and ResNet-110, demonstrate that ComDML can significantly reduce the overall training time while maintaining model accuracy, compared to state-of-the-art methods. ComDML demonstrates robustness in heterogeneous environments, and privacy measures can be seamlessly integrated for enhanced data protection.

Title: A Blockchain-Based Audit Mechanism for Trust and Integrity in IoT-Fog Environments

• Authors: Ismael Martinez, Abdelhakim Senhaji Hafid, Michel Gendreau
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.00844
• Pdf URL: https://arxiv.org/pdf/2405.00844
• Copy Paste: [[2405.00844]] A Blockchain-Based Audit Mechanism for Trust and Integrity in IoT-Fog Environments(https://arxiv.org/abs/2405.00844)
• Keywords: secure, security
• Abstract: The full realization of smart city technology is dependent on the secure and honest collaboration between IoT applications and edge-computing. In particular, resource constrained IoT devices may rely on fog-computing to alleviate the computing load of IoT tasks. Mutual authentication is needed between IoT and fog to preserve IoT data security, and monetization of fog services is needed to promote the fog service ecosystem. However, there is no guarantee that fog nodes will always respond to IoT requests correctly, either intentionally or accidentally. In the public decentralized IoT-fog environment, it is crucial to enforce integrity among fog nodes. In this paper, we propose a blockchain-based system that 1) streamlines the mutual authentication service monetization between IoT and fog, 2) verifies the integrity of fog nodes via service audits, and 3) discourages malicious activity and promotes honesty among fog nodes through incentives and penalties.

Title: Brighteye: Glaucoma Screening with Color Fundus Photographs based on Vision Transformer

• Authors: Hui Lin, Charilaos Apostolidis, Aggelos K. Katsaggelos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00857
• Pdf URL: https://arxiv.org/pdf/2405.00857
• Copy Paste: [[2405.00857]] Brighteye: Glaucoma Screening with Color Fundus Photographs based on Vision Transformer(https://arxiv.org/abs/2405.00857)
• Keywords: secure, transformer
• Abstract: Differences in image quality, lighting conditions, and patient demographics pose challenges to automated glaucoma detection from color fundus photography. Brighteye, a method based on Vision Transformer, is proposed for glaucoma detection and glaucomatous feature classification. Brighteye learns long-range relationships among pixels within large fundus images using a self-attention mechanism. Prior to being input into Brighteye, the optic disc is localized using YOLOv8, and the region of interest (ROI) around the disc center is cropped to ensure alignment with clinical practice. Optic disc detection improves the sensitivity at 95% specificity from 79.20% to 85.70% for glaucoma detection and the Hamming distance from 0.2470 to 0.1250 for glaucomatous feature classification. In the developmental stage of the Justified Referral in AI Glaucoma Screening (JustRAIGS) challenge, the overall outcome secured the fifth position out of 226 entries.

Title: Guided Conditional Diffusion Classifier (ConDiff) for Enhanced Prediction of Infection in Diabetic Foot Ulcers

• Authors: Palawat Busaranuvong, Emmanuel Agu, Deepak Kumar, Shefalika Gautam, Reza Saadati Fard, Bengisu Tulu, Diane Strong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00858
• Pdf URL: https://arxiv.org/pdf/2405.00858
• Copy Paste: [[2405.00858]] Guided Conditional Diffusion Classifier (ConDiff) for Enhanced Prediction of Infection in Diabetic Foot Ulcers(https://arxiv.org/abs/2405.00858)
• Keywords: diffusion, generative
• Abstract: To detect infected wounds in Diabetic Foot Ulcers (DFUs) from photographs, preventing severe complications and amputations. Methods: This paper proposes the Guided Conditional Diffusion Classifier (ConDiff), a novel deep-learning infection detection model that combines guided image synthesis with a denoising diffusion model and distance-based classification. The process involves (1) generating guided conditional synthetic images by injecting Gaussian noise to a guide image, followed by denoising the noise-perturbed image through a reverse diffusion process, conditioned on infection status and (2) classifying infections based on the minimum Euclidean distance between synthesized images and the original guide image in embedding space. Results: ConDiff demonstrated superior performance with an accuracy of 83% and an F1-score of 0.858, outperforming state-of-the-art models by at least 3%. The use of a triplet loss function reduces overfitting in the distance-based classifier. Conclusions: ConDiff not only enhances diagnostic accuracy for DFU infections but also pioneers the use of generative discriminative models for detailed medical image analysis, offering a promising approach for improving patient outcomes.

Title: Math Multiple Choice Question Generation via Human-Large Language Model Collaboration

• Authors: Jaewook Lee, Digory Smith, Simon Woodhead, Andrew Lan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.00864
• Pdf URL: https://arxiv.org/pdf/2405.00864
• Copy Paste: [[2405.00864]] Math Multiple Choice Question Generation via Human-Large Language Model Collaboration(https://arxiv.org/abs/2405.00864)
• Keywords: large language model
• Abstract: Multiple choice questions (MCQs) are a popular method for evaluating students' knowledge due to their efficiency in administration and grading. Crafting high-quality math MCQs is a labor-intensive process that requires educators to formulate precise stems and plausible distractors. Recent advances in large language models (LLMs) have sparked interest in automating MCQ creation, but challenges persist in ensuring mathematical accuracy and addressing student errors. This paper introduces a prototype tool designed to facilitate collaboration between LLMs and educators for streamlining the math MCQ generation process. We conduct a pilot study involving math educators to investigate how the tool can help them simplify the process of crafting high-quality math MCQs. We found that while LLMs can generate well-formulated question stems, their ability to generate distractors that capture common student errors and misconceptions is limited. Nevertheless, a human-AI collaboration has the potential to enhance the efficiency and effectiveness of MCQ generation.

Title: Hiding Sensitive Information Using PDF Steganography

• Authors: Ryan Klemm, Bo Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.00865
• Pdf URL: https://arxiv.org/pdf/2405.00865
• Copy Paste: [[2405.00865]] Hiding Sensitive Information Using PDF Steganography(https://arxiv.org/abs/2405.00865)
• Keywords: security
• Abstract: The use of steganography to transmit secret data is becoming increasingly common in security products and malware today. Despite being extremely popular, PDF files are not often the focus of steganography research, as most applications utilize digital image, audio, and video files as their cover data. However, the PDF file format is promising for usage in medium-capacity steganography applications. In this paper, we present a novel PDF steganography algorithm based upon least-significant bit insertion into the real-valued operands of PDF stream operators. Where prior research has only considered a small subset of these operators, we take an extensive look at all the possible operators defined in the Adobe PDF standard to evaluate their usability in our steganography algorithm. We also provide a case study which embeds malware into a given cover PDF document.

Title: Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis

• Authors: Prateek Verma, Minh-Hao Van, Xintao Wu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00876
• Pdf URL: https://arxiv.org/pdf/2405.00876
• Copy Paste: [[2405.00876]] Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis(https://arxiv.org/abs/2405.00876)
• Keywords: segmentation
• Abstract: Vision language models (VLMs) have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data. VLMs such as LLaVA, ChatGPT-4, and Gemini have recently shown impressive performance on tasks such as natural image captioning, visual question answering (VQA), and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Since medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is indubitably essential to test the performance of VLMs and foundation models such as SAM, on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM with classification, segmentation, counting, and VQA tasks on a variety of microscopy images. We observe that ChatGPT and Gemini are impressively able to comprehend the visual features in microscopy images, while SAM is quite capable at isolating artefacts in a general sense. However, the performance is not close to that of a domain expert - the models are readily encumbered by the introduction of impurities, defects, artefact overlaps and diversity present in the images.

Title: SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models

• Authors: Burak Can Biner, Farrin Marouf Sofian, Umur Berkay Karakaş, Duygu Ceylan, Erkut Erdem, Aykut Erdem
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00878
• Pdf URL: https://arxiv.org/pdf/2405.00878
• Copy Paste: [[2405.00878]] SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models(https://arxiv.org/abs/2405.00878)
• Keywords: diffusion
• Abstract: We are witnessing a revolution in conditional image synthesis with the recent success of large scale text-to-image generation methods. This success also opens up new opportunities in controlling the generation and editing process using multi-modal input. While spatial control using cues such as depth, sketch, and other images has attracted a lot of research, we argue that another equally effective modality is audio since sound and sight are two main components of human perception. Hence, we propose a method to enable audio-conditioning in large scale image diffusion models. Our method first maps features obtained from audio clips to tokens that can be injected into the diffusion model in a fashion similar to text tokens. We introduce additional audio-image cross attention layers which we finetune while freezing the weights of the original layers of the diffusion model. In addition to audio conditioned image generation, our method can also be utilized in conjuction with diffusion based editing methods to enable audio conditioned image editing. We demonstrate our method on a wide range of audio and image datasets. We perform extensive comparisons with recent methods and show favorable performance.

Title: WHALE-FL: Wireless and Heterogeneity Aware Latency Efficient Federated Learning over Mobile Devices via Adaptive Subnetwork Scheduling

• Authors: Huai-an Su, Jiaxiang Geng, Liang Li, Xiaoqi Qin, Yanzhao Hou, Xin Fu, Miao Pan
• Subjects: cs.LG, cs.NI, eess.IV
• Abstract URL: https://arxiv.org/abs/2405.00885
• Pdf URL: https://arxiv.org/pdf/2405.00885
• Copy Paste: [[2405.00885]] WHALE-FL: Wireless and Heterogeneity Aware Latency Efficient Federated Learning over Mobile Devices via Adaptive Subnetwork Scheduling(https://arxiv.org/abs/2405.00885)
• Keywords: federate
• Abstract: As a popular distributed learning paradigm, federated learning (FL) over mobile devices fosters numerous applications, while their practical deployment is hindered by participating devices' computing and communication heterogeneity. Some pioneering research efforts proposed to extract subnetworks from the global model, and assign as large a subnetwork as possible to the device for local training based on its full computing and communications capacity. Although such fixed size subnetwork assignment enables FL training over heterogeneous mobile devices, it is unaware of (i) the dynamic changes of devices' communication and computing conditions and (ii) FL training progress and its dynamic requirements of local training contributions, both of which may cause very long FL training delay. Motivated by those dynamics, in this paper, we develop a wireless and heterogeneity aware latency efficient FL (WHALE-FL) approach to accelerate FL training through adaptive subnetwork scheduling. Instead of sticking to the fixed size subnetwork, WHALE-FL introduces a novel subnetwork selection utility function to capture device and FL training dynamics, and guides the mobile device to adaptively select the subnetwork size for local training based on (a) its computing and communication capacity, (b) its dynamic computing and/or communication conditions, and (c) FL training status and its corresponding requirements for local training contributions. Our evaluation shows that, compared with peer designs, WHALE-FL effectively accelerates FL training without sacrificing learning accuracy.

Title: Wake Vision: A Large-scale, Diverse Dataset and Benchmark Suite for TinyML Person Detection

• Authors: Colby Banbury, Emil Njor, Matthew Stewart, Pete Warden, Manjunath Kudlur, Nat Jeffries, Xenofon Fafoutis, Vijay Janapa Reddi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00892
• Pdf URL: https://arxiv.org/pdf/2405.00892
• Copy Paste: [[2405.00892]] Wake Vision: A Large-scale, Diverse Dataset and Benchmark Suite for TinyML Person Detection(https://arxiv.org/abs/2405.00892)
• Keywords: robust
• Abstract: Machine learning applications on extremely low-power devices, commonly referred to as tiny machine learning (TinyML), promises a smarter and more connected world. However, the advancement of current TinyML research is hindered by the limited size and quality of pertinent datasets. To address this challenge, we introduce Wake Vision, a large-scale, diverse dataset tailored for person detection -- the canonical task for TinyML visual sensing. Wake Vision comprises over 6 million images, which is a hundredfold increase compared to the previous standard, and has undergone thorough quality filtering. Using Wake Vision for training results in a 2.41\% increase in accuracy compared to the established benchmark. Alongside the dataset, we provide a collection of five detailed benchmark sets that assess model performance on specific segments of the test data, such as varying lighting conditions, distances from the camera, and demographic characteristics of subjects. These novel fine-grained benchmarks facilitate the evaluation of model quality in challenging real-world scenarios that are often ignored when focusing solely on overall accuracy. Through an evaluation of a MobileNetV2 TinyML model on the benchmarks, we show that the input resolution plays a more crucial role than the model width in detecting distant subjects and that the impact of quantization on model robustness is minimal, thanks to the dataset quality. These findings underscore the importance of a detailed evaluation to identify essential factors for model development. The dataset, benchmark suite, code, and models are publicly available under the CC-BY 4.0 license, enabling their use for commercial use cases.

Title: DiL-NeRF: Delving into Lidar for Neural Radiance Field on Street Scenes

• Authors: Shanlin Sun, Bingbing Zhuang, Ziyu Jiang, Buyu Liu, Xiaohui Xie, Manmohan Chandraker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00900
• Pdf URL: https://arxiv.org/pdf/2405.00900
• Copy Paste: [[2405.00900]] DiL-NeRF: Delving into Lidar for Neural Radiance Field on Street Scenes(https://arxiv.org/abs/2405.00900)
• Keywords: robust
• Abstract: Photorealistic simulation plays a crucial role in applications such as autonomous driving, where advances in neural radiance fields (NeRFs) may allow better scalability through the automatic creation of digital 3D assets. However, reconstruction quality suffers on street scenes due to largely collinear camera motions and sparser samplings at higher speeds. On the other hand, the application often demands rendering from camera views that deviate from the inputs to accurately simulate behaviors like lane changes. In this paper, we propose several insights that allow a better utilization of Lidar data to improve NeRF quality on street scenes. First, our framework learns a geometric scene representation from Lidar, which is fused with the implicit grid-based representation for radiance decoding, thereby supplying stronger geometric information offered by explicit point cloud. Second, we put forth a robust occlusion-aware depth supervision scheme, which allows utilizing densified Lidar points by accumulation. Third, we generate augmented training views from Lidar points for further improvement. Our insights translate to largely improved novel view synthesis under real driving scenes.

Title: A Named Entity Recognition and Topic Modeling-based Solution for Locating and Better Assessment of Natural Disasters in Social Media

• Authors: Ayaz Mehmood, Muhammad Tayyab Zamir, Muhammad Asif Ayub, Nasir Ahmad, Kashif Ahmad
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.00903
• Pdf URL: https://arxiv.org/pdf/2405.00903
• Copy Paste: [[2405.00903]] A Named Entity Recognition and Topic Modeling-based Solution for Locating and Better Assessment of Natural Disasters in Social Media(https://arxiv.org/abs/2405.00903)
• Keywords: extraction
• Abstract: Over the last decade, similar to other application domains, social media content has been proven very effective in disaster informatics. However, due to the unstructured nature of the data, several challenges are associated with disaster analysis in social media content. To fully explore the potential of social media content in disaster informatics, access to relevant content and the correct geo-location information is very critical. In this paper, we propose a three-step solution to tackling these challenges. Firstly, the proposed solution aims to classify social media posts into relevant and irrelevant posts followed by the automatic extraction of location information from the posts' text through Named Entity Recognition (NER) analysis. Finally, to quickly analyze the topics covered in large volumes of social media posts, we perform topic modeling resulting in a list of top keywords, that highlight the issues discussed in the tweet. For the Relevant Classification of Twitter Posts (RCTP), we proposed a merit-based fusion framework combining the capabilities of four different models namely BERT, RoBERTa, Distil BERT, and ALBERT obtaining the highest F1-score of 0.933 on a benchmark dataset. For the Location Extraction from Twitter Text (LETT), we evaluated four models namely BERT, RoBERTa, Distil BERTA, and Electra in an NER framework obtaining the highest F1-score of 0.960. For topic modeling, we used the BERTopic library to discover the hidden topic patterns in the relevant tweets. The experimental results of all the components of the proposed end-to-end solution are very encouraging and hint at the potential of social media content and NLP in disaster management.

Title: LOTUS: Improving Transformer Efficiency with Sparsity Pruning and Data Lottery Tickets

• Authors: Ojasw Upadhyay
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00906
• Pdf URL: https://arxiv.org/pdf/2405.00906
• Copy Paste: [[2405.00906]] LOTUS: Improving Transformer Efficiency with Sparsity Pruning and Data Lottery Tickets(https://arxiv.org/abs/2405.00906)
• Keywords: transformer
• Abstract: Vision transformers have revolutionized computer vision, but their computational demands present challenges for training and deployment. This paper introduces LOTUS (LOttery Transformers with Ultra Sparsity), a novel method that leverages data lottery ticket selection and sparsity pruning to accelerate vision transformer training while maintaining accuracy. Our approach focuses on identifying and utilizing the most informative data subsets and eliminating redundant model parameters to optimize the training process. Through extensive experiments, we demonstrate the effectiveness of LOTUS in achieving rapid convergence and high accuracy with significantly reduced computational requirements. This work highlights the potential of combining data selection and sparsity techniques for efficient vision transformer training, opening doors for further research and development in this area.

Title: Transformer-Based Self-Supervised Learning for Histopathological Classification of Ischemic Stroke Clot Origin

• Authors: K. Yeh, M. S. Jabal, V. Gupta, D. F. Kallmes, W. Brinjikji, B. S. Erdal
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00908
• Pdf URL: https://arxiv.org/pdf/2405.00908
• Copy Paste: [[2405.00908]] Transformer-Based Self-Supervised Learning for Histopathological Classification of Ischemic Stroke Clot Origin(https://arxiv.org/abs/2405.00908)
• Keywords: interpretability, transformer
• Abstract: Background and Purpose: Identifying the thromboembolism source in ischemic stroke is crucial for treatment and secondary prevention yet is often undetermined. This study describes a self-supervised deep learning approach in digital pathology of emboli for classifying ischemic stroke clot origin from histopathological images. Methods: The dataset included whole slide images (WSI) from the STRIP AI Kaggle challenge, consisting of retrieved clots from ischemic stroke patients following mechanical thrombectomy. Transformer-based deep learning models were developed using transfer learning and self-supervised pretraining for classifying WSI. Customizations included an attention pooling layer, weighted loss function, and threshold optimization. Various model architectures were tested and compared, and model performances were primarily evaluated using weighted logarithmic loss. Results: The model achieved a logloss score of 0.662 in cross-validation and 0.659 on the test set. Different model backbones were compared, with the swin_large_patch4_window12_384 showed higher performance. Thresholding techniques for clot origin classification were employed to balance false positives and negatives. Conclusion: The study demonstrates the extent of efficacy of transformer-based deep learning models in identifying ischemic stroke clot origins from histopathological images and emphasizes the need for refined modeling techniques specifically adapted to thrombi WSI. Further research is needed to improve model performance, interpretability, validate its effectiveness. Future enhancement could include integrating larger patient cohorts, advanced preprocessing strategies, and exploring ensemble multimodal methods for enhanced diagnostic accuracy.

Title: Quantum Federated Learning Experiments in the Cloud with Data Encoding

• Authors: Shiva Raj Pokhrel, Naman Yash, Jonathan Kua, Gang Li, Lei Pan
• Subjects: cs.LG, cs.ET, quant-ph
• Abstract URL: https://arxiv.org/abs/2405.00909
• Pdf URL: https://arxiv.org/pdf/2405.00909
• Copy Paste: [[2405.00909]] Quantum Federated Learning Experiments in the Cloud with Data Encoding(https://arxiv.org/abs/2405.00909)
• Keywords: privacy, federate
• Abstract: Quantum Federated Learning (QFL) is an emerging concept that aims to unfold federated learning (FL) over quantum networks, enabling collaborative quantum model training along with local data privacy. We explore the challenges of deploying QFL on cloud platforms, emphasizing quantum intricacies and platform limitations. The proposed data-encoding-driven QFL, with a proof of concept (GitHub Open Source) using genomic data sets on quantum simulators, shows promising results.

Title: De-Biasing Models of Biased Decisions: A Comparison of Methods Using Mortgage Application Data

• Authors: Nicholas Tenev
• Subjects: cs.LG, cs.CY, econ.EM
• Abstract URL: https://arxiv.org/abs/2405.00910
• Pdf URL: https://arxiv.org/pdf/2405.00910
• Copy Paste: [[2405.00910]] De-Biasing Models of Biased Decisions: A Comparison of Methods Using Mortgage Application Data(https://arxiv.org/abs/2405.00910)
• Keywords: protect
• Abstract: Prediction models can improve efficiency by automating decisions such as the approval of loan applications. However, they may inherit bias against protected groups from the data they are trained on. This paper adds counterfactual (simulated) ethnic bias to real data on mortgage application decisions, and shows that this bias is replicated by a machine learning model (XGBoost) even when ethnicity is not used as a predictive variable. Next, several other de-biasing methods are compared: averaging over prohibited variables, taking the most favorable prediction over prohibited variables (a novel method), and jointly minimizing errors as well as the association between predictions and prohibited variables. De-biasing can recover some of the original decisions, but the results are sensitive to whether the bias is effected through a proxy.

Title: EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion

• Authors: Guangyao Zhai, Evin Pınar Örnek, Dave Zhenyu Chen, Ruotong Liao, Yan Di, Nassir Navab, Federico Tombari, Benjamin Busam
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00915
• Pdf URL: https://arxiv.org/pdf/2405.00915
• Copy Paste: [[2405.00915]] EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion(https://arxiv.org/abs/2405.00915)
• Keywords: diffusion, generative
• Abstract: We present EchoScene, an interactive and controllable generative model that generates 3D indoor scenes on scene graphs. EchoScene leverages a dual-branch diffusion model that dynamically adapts to scene graphs. Existing methods struggle to handle scene graphs due to varying numbers of nodes, multiple edge combinations, and manipulator-induced node-edge operations. EchoScene overcomes this by associating each node with a denoising process and enables collaborative information exchange, enhancing controllable and consistent generation aware of global constraints. This is achieved through an information echo scheme in both shape and layout branches. At every denoising step, all processes share their denoising data with an information exchange unit that combines these updates using graph convolution. The scheme ensures that the denoising processes are influenced by a holistic understanding of the scene graph, facilitating the generation of globally coherent scenes. The resulting scenes can be manipulated during inference by editing the input scene graph and sampling the noise in the diffusion model. Extensive experiments validate our approach, which maintains scene controllability and surpasses previous methods in generation fidelity. Moreover, the generated scenes are of high quality and thus directly compatible with off-the-shelf texture generation. Code and trained models are open-sourced.

Title: LLaVA Finds Free Lunch: Teaching Human Behavior Improves Content Understanding Abilities Of LLMs

• Authors: Somesh Singh, Harini S I, Yaman K Singla, Veeky Baths, Rajiv Ratn Shah, Changyou Chen, Balaji Krishnamurthy
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2405.00942
• Pdf URL: https://arxiv.org/pdf/2405.00942
• Copy Paste: [[2405.00942]] LLaVA Finds Free Lunch: Teaching Human Behavior Improves Content Understanding Abilities Of LLMs(https://arxiv.org/abs/2405.00942)
• Keywords: large language model
• Abstract: Communication is defined as ``Who says what to whom with what effect.'' A message from a communicator generates downstream receiver effects, also known as behavior. Receiver behavior, being a downstream effect of the message, carries rich signals about it. Even after carrying signals about the message, the behavior data is often ignored while training large language models. We show that training LLMs on receiver behavior can actually help improve their content-understanding abilities. Specifically, we show that training LLMs to predict the receiver behavior of likes and comments improves the LLM's performance on a wide variety of downstream content understanding tasks. We show this performance increase over 40 video and image understanding tasks over 23 benchmark datasets across both 0-shot and fine-tuning settings, outperforming many supervised baselines. Moreover, since receiver behavior, such as likes and comments, is collected by default on the internet and does not need any human annotations to be useful, the performance improvement we get after training on this data is essentially free-lunch. We release the receiver behavior cleaned comments and likes of 750k images and videos collected from multiple platforms along with our instruction-tuning data.

Title: The Role of Model Architecture and Scale in Predicting Molecular Properties: Insights from Fine-Tuning RoBERTa, BART, and LLaMA

• Authors: Lee Youngmin, Lang S.I.D. Andrew, Cai Duoduo, Wheat R. Stephen
• Subjects: cs.LG, cs.CL, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2405.00949
• Pdf URL: https://arxiv.org/pdf/2405.00949
• Copy Paste: [[2405.00949]] The Role of Model Architecture and Scale in Predicting Molecular Properties: Insights from Fine-Tuning RoBERTa, BART, and LLaMA(https://arxiv.org/abs/2405.00949)
• Keywords: robust, large language model
• Abstract: This study introduces a systematic framework to compare the efficacy of Large Language Models (LLMs) for fine-tuning across various cheminformatics tasks. Employing a uniform training methodology, we assessed three well-known models-RoBERTa, BART, and LLaMA-on their ability to predict molecular properties using the Simplified Molecular Input Line Entry System (SMILES) as a universal molecular representation format. Our comparative analysis involved pre-training 18 configurations of these models, with varying parameter sizes and dataset scales, followed by fine-tuning them on six benchmarking tasks from DeepChem. We maintained consistent training environments across models to ensure reliable comparisons. This approach allowed us to assess the influence of model type, size, and training dataset size on model performance. Specifically, we found that LLaMA-based models generally offered the lowest validation loss, suggesting their superior adaptability across tasks and scales. However, we observed that absolute validation loss is not a definitive indicator of model performance - contradicts previous research - at least for fine-tuning tasks: instead, model size plays a crucial role. Through rigorous replication and validation, involving multiple training and fine-tuning cycles, our study not only delineates the strengths and limitations of each model type but also provides a robust methodology for selecting the most suitable LLM for specific cheminformatics applications. This research underscores the importance of considering model architecture and dataset characteristics in deploying AI for molecular property prediction, paving the way for more informed and effective utilization of AI in drug discovery and related fields.

Title: Recovering Labels from Local Updates in Federated Learning

• Authors: Huancheng Chen, Haris Vikalo
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2405.00955
• Pdf URL: https://arxiv.org/pdf/2405.00955
• Copy Paste: [[2405.00955]] Recovering Labels from Local Updates in Federated Learning(https://arxiv.org/abs/2405.00955)
• Keywords: privacy, attack, extraction, federate
• Abstract: Gradient inversion (GI) attacks present a threat to the privacy of clients in federated learning (FL) by aiming to enable reconstruction of the clients' data from communicated model updates. A number of such techniques attempts to accelerate data recovery by first reconstructing labels of the samples used in local training. However, existing label extraction methods make strong assumptions that typically do not hold in realistic FL settings. In this paper we present a novel label recovery scheme, Recovering Labels from Local Updates (RLU), which provides near-perfect accuracy when attacking untrained (most vulnerable) models. More significantly, RLU achieves high performance even in realistic real-world settings where the clients in an FL system run multiple local epochs, train on heterogeneous data, and deploy various optimizers to minimize different objective functions. Specifically, RLU estimates labels by solving a least-square problem that emerges from the analysis of the correlation between labels of the data points used in a training round and the resulting update of the output layer. The experimental results on several datasets, architectures, and data heterogeneity scenarios demonstrate that the proposed method consistently outperforms existing baselines, and helps improve quality of the reconstructed images in GI attacks in terms of both PSNR and LPIPS.

Title: Generative manufacturing systems using diffusion models and ChatGPT

• Authors: Xingyu Li, Fei Tao, Wei Ye, Aydin Nassehi, John W. Sutherland
• Subjects: cs.LG, cs.AI, cs.HC, eess.SY
• Abstract URL: https://arxiv.org/abs/2405.00958
• Pdf URL: https://arxiv.org/pdf/2405.00958
• Copy Paste: [[2405.00958]] Generative manufacturing systems using diffusion models and ChatGPT(https://arxiv.org/abs/2405.00958)
• Keywords: diffusion, generative
• Abstract: In this study, we introduce Generative Manufacturing Systems (GMS) as a novel approach to effectively manage and coordinate autonomous manufacturing assets, thereby enhancing their responsiveness and flexibility to address a wide array of production objectives and human preferences. Deviating from traditional explicit modeling, GMS employs generative AI, including diffusion models and ChatGPT, for implicit learning from envisioned futures, marking a shift from a model-optimum to a training-sampling decision-making. Through the integration of generative AI, GMS enables complex decision-making through interactive dialogue with humans, allowing manufacturing assets to generate multiple high-quality global decisions that can be iteratively refined based on human feedback. Empirical findings showcase GMS's substantial improvement in system resilience and responsiveness to uncertainties, with decision times reduced from seconds to milliseconds. The study underscores the inherent creativity and diversity in the generated solutions, facilitating human-centric decision-making through seamless and continuous human-machine interactions.

Title: Robust Decentralized Learning with Local Updates and Gradient Tracking

• Authors: Sajjad Ghiasvand, Amirhossein Reisizadeh, Mahnoosh Alizadeh, Ramtin Pedarsani
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2405.00965
• Pdf URL: https://arxiv.org/pdf/2405.00965
• Copy Paste: [[2405.00965]] Robust Decentralized Learning with Local Updates and Gradient Tracking(https://arxiv.org/abs/2405.00965)
• Keywords: robust, federate
• Abstract: As distributed learning applications such as Federated Learning, the Internet of Things (IoT), and Edge Computing grow, it is critical to address the shortcomings of such technologies from a theoretical perspective. As an abstraction, we consider decentralized learning over a network of communicating clients or nodes and tackle two major challenges: data heterogeneity and adversarial robustness. We propose a decentralized minimax optimization method that employs two important modules: local updates and gradient tracking. Minimax optimization is the key tool to enable adversarial training for ensuring robustness. Having local updates is essential in Federated Learning (FL) applications to mitigate the communication bottleneck, and utilizing gradient tracking is essential to proving convergence in the case of data heterogeneity. We analyze the performance of the proposed algorithm, Dec-FedTrack, in the case of nonconvex-strongly concave minimax optimization, and prove that it converges a stationary point. We also conduct numerical experiments to support our theoretical findings.

Title: Efficient Compression of Multitask Multilingual Speech Models

• Authors: Thomas Palmeira Ferraz
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2405.00966
• Pdf URL: https://arxiv.org/pdf/2405.00966
• Copy Paste: [[2405.00966]] Efficient Compression of Multitask Multilingual Speech Models(https://arxiv.org/abs/2405.00966)
• Keywords: robust
• Abstract: Whisper is a multitask and multilingual speech model covering 99 languages. It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still underperforms on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions. In this work, we examine its limitations, demonstrating the presence of speaker-related (gender, age) and model-related (resourcefulness and model size) bias. Despite that, we show that only model-related bias are amplified by quantization, impacting more low-resource languages and smaller models. Searching for a better compression approach, we propose DistilWhisper, an approach that is able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. This dual approach allows us to effectively boost ASR performance while keeping the robustness inherited from the multitask and multilingual pre-training. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters, boosting performance in the targeted languages for both in- and out-of-domain test sets, while introducing only a negligible parameter overhead at inference.

Title: How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses

• Authors: Jionghao Lin, Zifei Han, Danielle R. Thomas, Ashish Gurung, Shivang Gupta, Vincent Aleven, Kenneth R. Koedinger
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2405.00970
• Pdf URL: https://arxiv.org/pdf/2405.00970
• Copy Paste: [[2405.00970]] How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses(https://arxiv.org/abs/2405.00970)
• Keywords: large language model
• Abstract: One-on-one tutoring is widely acknowledged as an effective instructional method, conditioned on qualified tutors. However, the high demand for qualified tutors remains a challenge, often necessitating the training of novice tutors (i.e., trainees) to ensure effective tutoring. Research suggests that providing timely explanatory feedback can facilitate the training process for trainees. However, it presents challenges due to the time-consuming nature of assessing trainee performance by human experts. Inspired by the recent advancements of large language models (LLMs), our study employed the GPT-4 model to build an explanatory feedback system. This system identifies trainees' responses in binary form (i.e., correct/incorrect) and automatically provides template-based feedback with responses appropriately rephrased by the GPT-4 model. We conducted our study on 410 responses from trainees across three training lessons: Giving Effective Praise, Reacting to Errors, and Determining What Students Know. Our findings indicate that: 1) using a few-shot approach, the GPT-4 model effectively identifies correct/incorrect trainees' responses from three training lessons with an average F1 score of 0.84 and an AUC score of 0.85; and 2) using the few-shot approach, the GPT-4 model adeptly rephrases incorrect trainees' responses into desired responses, achieving performance comparable to that of human experts.

Title: CACTUS: Chemistry Agent Connecting Tool-Usage to Science

• Authors: Andrew D. McNaughton, Gautham Ramalaxmi, Agustin Kruel, Carter R. Knutson, Rohith A. Varikoti, Neeraj Kumar
• Subjects: cs.CL, cs.AI, cs.LG, physics.chem-ph, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2405.00972
• Pdf URL: https://arxiv.org/pdf/2405.00972
• Copy Paste: [[2405.00972]] CACTUS: Chemistry Agent Connecting Tool-Usage to Science(https://arxiv.org/abs/2405.00972)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown remarkable potential in various domains, but they often lack the ability to access and reason over domain-specific knowledge and tools. In this paper, we introduced CACTUS (Chemistry Agent Connecting Tool-Usage to Science), an LLM-based agent that integrates cheminformatics tools to enable advanced reasoning and problem-solving in chemistry and molecular discovery. We evaluate the performance of CACTUS using a diverse set of open-source LLMs, including Gemma-7b, Falcon-7b, MPT-7b, Llama2-7b, and Mistral-7b, on a benchmark of thousands of chemistry questions. Our results demonstrate that CACTUS significantly outperforms baseline LLMs, with the Gemma-7b and Mistral-7b models achieving the highest accuracy regardless of the prompting strategy used. Moreover, we explore the impact of domain-specific prompting and hardware configurations on model performance, highlighting the importance of prompt engineering and the potential for deploying smaller models on consumer-grade hardware without significant loss in accuracy. By combining the cognitive capabilities of open-source LLMs with domain-specific tools, CACTUS can assist researchers in tasks such as molecular property prediction, similarity searching, and drug-likeness assessment. Furthermore, CACTUS represents a significant milestone in the field of cheminformatics, offering an adaptable tool for researchers engaged in chemistry and molecular discovery. By integrating the strengths of open-source LLMs with domain-specific tools, CACTUS has the potential to accelerate scientific advancement and unlock new frontiers in the exploration of novel, effective, and safe therapeutic candidates, catalysts, and materials. Moreover, CACTUS's ability to integrate with automated experimentation platforms and make data-driven decisions in real time opens up new possibilities for autonomous discovery.

Title: On the Evaluation of Machine-Generated Reports

• Authors: James Mayfield, Eugene Yang, Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Kayi, Kate Sanders, Marc Mason, Noah Hibbler
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2405.00982
• Pdf URL: https://arxiv.org/pdf/2405.00982
• Copy Paste: [[2405.00982]] On the Evaluation of Machine-Generated Reports(https://arxiv.org/abs/2405.00982)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have enabled new ways to satisfy information needs. Although great strides have been made in applying them to settings like document ranking and short-form text generation, they still struggle to compose complete, accurate, and verifiable long-form reports. Reports with these qualities are necessary to satisfy the complex, nuanced, or multi-faceted information needs of users. In this perspective paper, we draw together opinions from industry and academia, and from a variety of related research areas, to present our vision for automatic report generation, and -- critically -- a flexible framework by which such reports can be evaluated. In contrast with other summarization tasks, automatic report generation starts with a detailed description of an information need, stating the necessary background, requirements, and scope of the report. Further, the generated reports should be complete, accurate, and verifiable. These qualities, which are desirable -- if not required -- in many analytic report-writing settings, require rethinking how to build and evaluate systems that exhibit these qualities. To foster new efforts in building these systems, we present an evaluation framework that draws on ideas found in various evaluations. To test completeness and accuracy, the framework uses nuggets of information, expressed as questions and answers, that need to be part of any high-quality generated report. Additionally, evaluation of citations that map claims made in the report to their source documents ensures verifiability.

Title: LLM-AD: Large Language Model based Audio Description System

• Authors: Peng Chu, Jiang Wang, Andre Abrantes
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00983
• Pdf URL: https://arxiv.org/pdf/2405.00983
• Copy Paste: [[2405.00983]] LLM-AD: Large Language Model based Audio Description System(https://arxiv.org/abs/2405.00983)
• Keywords: large language model
• Abstract: The development of Audio Description (AD) has been a pivotal step forward in making video content more accessible and inclusive. Traditionally, AD production has demanded a considerable amount of skilled labor, while existing automated approaches still necessitate extensive training to integrate multimodal inputs and tailor the output from a captioning style to an AD style. In this paper, we introduce an automated AD generation pipeline that harnesses the potent multimodal and instruction-following capacities of GPT-4V(ision). Notably, our methodology employs readily available components, eliminating the need for additional training. It produces ADs that not only comply with established natural language AD production standards but also maintain contextually consistent character information across frames, courtesy of a tracking-based character recognition module. A thorough analysis on the MAD dataset reveals that our approach achieves a performance on par with learning-based methods in automated AD production, as substantiated by a CIDEr score of 20.5.

Title: FREE: Faster and Better Data-Free Meta-Learning

• Authors: Yongxian Wei, Zixuan Hu, Zhenyi Wang, Li Shen, Chun Yuan, Dacheng Tao
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00984
• Pdf URL: https://arxiv.org/pdf/2405.00984
• Copy Paste: [[2405.00984]] FREE: Faster and Better Data-Free Meta-Learning(https://arxiv.org/abs/2405.00984)
• Keywords: privacy, data-free
• Abstract: Data-Free Meta-Learning (DFML) aims to extract knowledge from a collection of pre-trained models without requiring the original data, presenting practical benefits in contexts constrained by data privacy concerns. Current DFML methods primarily focus on the data recovery from these pre-trained models. However, they suffer from slow recovery speed and overlook gaps inherent in heterogeneous pre-trained models. In response to these challenges, we introduce the Faster and Better Data-Free Meta-Learning (FREE) framework, which contains: (i) a meta-generator for rapidly recovering training tasks from pre-trained models; and (ii) a meta-learner for generalizing to new unseen tasks. Specifically, within the module Faster Inversion via Meta-Generator, each pre-trained model is perceived as a distinct task. The meta-generator can rapidly adapt to a specific task in just five steps, significantly accelerating the data recovery. Furthermore, we propose Better Generalization via Meta-Learner and introduce an implicit gradient alignment algorithm to optimize the meta-learner. This is achieved as aligned gradient directions alleviate potential conflicts among tasks from heterogeneous pre-trained models. Empirical experiments on multiple benchmarks affirm the superiority of our approach, marking a notable speed-up (20$\times$) and performance enhancement (1.42\% $\sim$ 4.78\%) in comparison to the state-of-the-art.

Title: S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic

• Authors: Safa Messaoud, Billel Mokeddem, Zhenghai Xue, Linsey Pang, Bo An, Haipeng Chen, Sanjay Chawla
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00987
• Pdf URL: https://arxiv.org/pdf/2405.00987
• Copy Paste: [[2405.00987]] S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic(https://arxiv.org/abs/2405.00987)
• Keywords: robust
• Abstract: Learning expressive stochastic policies instead of deterministic ones has been proposed to achieve better stability, sample complexity, and robustness. Notably, in Maximum Entropy Reinforcement Learning (MaxEnt RL), the policy is modeled as an expressive Energy-Based Model (EBM) over the Q-values. However, this formulation requires the estimation of the entropy of such EBMs, which is an open problem. To address this, previous MaxEnt RL methods either implicitly estimate the entropy, resulting in high computational complexity and variance (SQL), or follow a variational inference procedure that fits simplified actor distributions (e.g., Gaussian) for tractability (SAC). We propose Stein Soft Actor-Critic (S$^2$AC), a MaxEnt RL algorithm that learns expressive policies without compromising efficiency. Specifically, S$^2$AC uses parameterized Stein Variational Gradient Descent (SVGD) as the underlying policy. We derive a closed-form expression of the entropy of such policies. Our formula is computationally efficient and only depends on first-order derivatives and vector products. Empirical results show that S$^2$AC yields more optimal solutions to the MaxEnt objective than SQL and SAC in the multi-goal environment, and outperforms SAC and SQL on the MuJoCo benchmark. Our code is available at: https://github.com/SafaMessaoud/S2AC-Energy-Based-RL-with-Stein-Soft-Actor-Critic

Title: Context-Aware Clustering using Large Language Models

• Authors: Sindhu Tipirneni, Ravinarayana Adkathimar, Nurendra Choudhary, Gaurush Hiranandani, Rana Ali Amjad, Vassilis N. Ioannidis, Changhe Yuan, Chandan K. Reddy
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00988
• Pdf URL: https://arxiv.org/pdf/2405.00988
• Copy Paste: [[2405.00988]] Context-Aware Clustering using Large Language Models(https://arxiv.org/abs/2405.00988)
• Keywords: large language model
• Abstract: Despite the remarkable success of Large Language Models (LLMs) in text understanding and generation, their potential for text clustering tasks remains underexplored. We observed that powerful closed-source LLMs provide good quality clusterings of entity sets but are not scalable due to the massive compute power required and the associated costs. Thus, we propose CACTUS (Context-Aware ClusTering with aUgmented triplet losS), a systematic approach that leverages open-source LLMs for efficient and effective supervised clustering of entity subsets, particularly focusing on text-based entities. Existing text clustering methods fail to effectively capture the context provided by the entity subset. Moreover, though there are several language modeling based approaches for clustering, very few are designed for the task of supervised clustering. This paper introduces a novel approach towards clustering entity subsets using LLMs by capturing context via a scalable inter-entity attention mechanism. We propose a novel augmented triplet loss function tailored for supervised clustering, which addresses the inherent challenges of directly applying the triplet loss to this problem. Furthermore, we introduce a self-supervised clustering task based on text augmentation techniques to improve the generalization of our model. For evaluation, we collect ground truth clusterings from a closed-source LLM and transfer this knowledge to an open-source LLM under the supervised clustering framework, allowing a faster and cheaper open-source model to perform the same task. Experiments on various e-commerce query and product clustering datasets demonstrate that our proposed approach significantly outperforms existing unsupervised and supervised baselines under various external clustering evaluation metrics.

Title: The IgboAPI Dataset: Empowering Igbo Language Technologies through Multi-dialectal Enrichment

• Authors: Chris Chinenye Emezue, Ifeoma Okoh, Chinedu Mbonu, Chiamaka Chukwuneke, Daisy Lal, Ignatius Ezeani, Paul Rayson, Ijemma Onwuzulike, Chukwuma Okeke, Gerald Nweya, Bright Ogbonna, Chukwuebuka Oraegbunam, Esther Chidinma Awo-Ndubuisi, Akudo Amarachukwu Osuagwu, Obioha Nmezi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.00997
• Pdf URL: https://arxiv.org/pdf/2405.00997
• Copy Paste: [[2405.00997]] The IgboAPI Dataset: Empowering Igbo Language Technologies through Multi-dialectal Enrichment(https://arxiv.org/abs/2405.00997)
• Keywords: robust
• Abstract: The Igbo language is facing a risk of becoming endangered, as indicated by a 2025 UNESCO study. This highlights the need to develop language technologies for Igbo to foster communication, learning and preservation. To create robust, impactful, and widely adopted language technologies for Igbo, it is essential to incorporate the multi-dialectal nature of the language. The primary obstacle in achieving dialectal-aware language technologies is the lack of comprehensive dialectal datasets. In response, we present the IgboAPI dataset, a multi-dialectal Igbo-English dictionary dataset, developed with the aim of enhancing the representation of Igbo dialects. Furthermore, we illustrate the practicality of the IgboAPI dataset through two distinct studies: one focusing on Igbo semantic lexicon and the other on machine translation. In the semantic lexicon project, we successfully establish an initial Igbo semantic lexicon for the Igbo semantic tagger, while in the machine translation study, we demonstrate that by finetuning existing machine translation systems using the IgboAPI dataset, we significantly improve their ability to handle dialectal variations in sentences.

Title: Part-aware Shape Generation with Latent 3D Diffusion of Neural Voxel Fields

• Authors: Yuhang Huang, SHilong Zou, Xinwang Liu, Kai Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00998
• Pdf URL: https://arxiv.org/pdf/2405.00998
• Copy Paste: [[2405.00998]] Part-aware Shape Generation with Latent 3D Diffusion of Neural Voxel Fields(https://arxiv.org/abs/2405.00998)
• Keywords: diffusion, generative
• Abstract: This paper presents a novel latent 3D diffusion model for the generation of neural voxel fields, aiming to achieve accurate part-aware structures. Compared to existing methods, there are two key designs to ensure high-quality and accurate part-aware generation. On one hand, we introduce a latent 3D diffusion process for neural voxel fields, enabling generation at significantly higher resolutions that can accurately capture rich textural and geometric details. On the other hand, a part-aware shape decoder is introduced to integrate the part codes into the neural voxel fields, guiding the accurate part decomposition and producing high-quality rendering results. Through extensive experimentation and comparisons with state-of-the-art methods, we evaluate our approach across four different classes of data. The results demonstrate the superior generative capabilities of our proposed method in part-aware shape generation, outperforming existing state-of-the-art methods.

Title: Spider: A Unified Framework for Context-dependent Concept Understanding

• Authors: Xiaoqi Zhao, Youwei Pang, Wei Ji, Baicheng Sheng, Jiaming Zuo, Lihe Zhang, Huchuan Lu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01002
• Pdf URL: https://arxiv.org/pdf/2405.01002
• Copy Paste: [[2405.01002]] Spider: A Unified Framework for Context-dependent Concept Understanding(https://arxiv.org/abs/2405.01002)
• Keywords: segmentation
• Abstract: Different from the context-independent (CI) concepts such as human, car, and airplane, context-dependent (CD) concepts require higher visual understanding ability, such as camouflaged object and medical lesion. Despite the rapid advance of many CD understanding tasks in respective branches, the isolated evolution leads to their limited cross-domain generalisation and repetitive technique innovation. Since there is a strong coupling relationship between foreground and background context in CD tasks, existing methods require to train separate models in their focused domains. This restricts their real-world CD concept understanding towards artificial general intelligence (AGI). We propose a unified model with a single set of parameters, Spider, which only needs to be trained once. With the help of the proposed concept filter driven by the image-mask group prompt, Spider is able to understand and distinguish diverse strong context-dependent concepts to accurately capture the Prompter's intention. Without bells and whistles, Spider significantly outperforms the state-of-the-art specialized models in 8 different context-dependent segmentation tasks, including 4 natural scenes (salient, camouflaged, and transparent objects and shadow) and 4 medical lesions (COVID-19, polyp, breast, and skin lesion with color colonoscopy, CT, ultrasound, and dermoscopy modalities). Besides, Spider shows obvious advantages in continuous learning. It can easily complete the training of new tasks by fine-tuning parameters less than 1\% and bring a tolerable performance degradation of less than 5\% for all old tasks. The source code will be publicly available at \href{https://github.com/Xiaoqi-Zhao-DLUT/Spider-UniCDSeg}{Spider-UniCDSeg}.

Title: On Mechanistic Knowledge Localization in Text-to-Image Generative Models

• Authors: Samyadeep Basu, Keivan Rezaei, Ryan Rossi, Cherry Zhao, Vlad Morariu, Varun Manjunatha, Soheil Feizi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01008
• Pdf URL: https://arxiv.org/pdf/2405.01008
• Copy Paste: [[2405.01008]] On Mechanistic Knowledge Localization in Text-to-Image Generative Models(https://arxiv.org/abs/2405.01008)
• Keywords: diffusion, generative
• Abstract: Identifying layers within text-to-image models which control visual attributes can facilitate efficient model editing through closed-form updates. Recent work, leveraging causal tracing show that early Stable-Diffusion variants confine knowledge primarily to the first layer of the CLIP text-encoder, while it diffuses throughout the UNet.Extending this framework, we observe that for recent models (e.g., SD-XL, DeepFloyd), causal tracing fails in pinpointing localized knowledge, highlighting challenges in model editing. To address this issue, we introduce the concept of Mechanistic Localization in text-to-image models, where knowledge about various visual attributes (e.g., ``style", ``objects", ``facts") can be mechanistically localized to a small fraction of layers in the UNet, thus facilitating efficient model editing. We localize knowledge using our method LocoGen which measures the direct effect of intermediate layers to output generation by performing interventions in the cross-attention layers of the UNet. We then employ LocoEdit, a fast closed-form editing method across popular open-source text-to-image models (including the latest SD-XL)and explore the possibilities of neuron-level model editing. Using Mechanistic Localization, our work offers a better view of successes and failures in localization-based text-to-image model editing. Code will be available at \href{https://github.com/samyadeepbasu/LocoGen}{https://github.com/samyadeepbasu/LocoGen}.

Title: Non-clairvoyant Scheduling with Partial Predictions

• Authors: Ziyad Benomar, Vianney Perchet
• Subjects: cs.LG, cs.AI, cs.DS
• Abstract URL: https://arxiv.org/abs/2405.01013
• Pdf URL: https://arxiv.org/pdf/2405.01013
• Copy Paste: [[2405.01013]] Non-clairvoyant Scheduling with Partial Predictions(https://arxiv.org/abs/2405.01013)
• Keywords: robust
• Abstract: The non-clairvoyant scheduling problem has gained new interest within learning-augmented algorithms, where the decision-maker is equipped with predictions without any quality guarantees. In practical settings, access to predictions may be reduced to specific instances, due to cost or data limitations. Our investigation focuses on scenarios where predictions for only $B$ job sizes out of $n$ are available to the algorithm. We first establish near-optimal lower bounds and algorithms in the case of perfect predictions. Subsequently, we present a learning-augmented algorithm satisfying the robustness, consistency, and smoothness criteria, and revealing a novel tradeoff between consistency and smoothness inherent in the scenario with a restricted number of predictions.

Title: Technical Report of NICE Challenge at CVPR 2024: Caption Re-ranking Evaluation Using Ensembled CLIP and Consensus Scores

• Authors: Kiyoon Jeong, Woojun Lee, Woongchan Nam, Minjeong Ma, Pilsung Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01028
• Pdf URL: https://arxiv.org/pdf/2405.01028
• Copy Paste: [[2405.01028]] Technical Report of NICE Challenge at CVPR 2024: Caption Re-ranking Evaluation Using Ensembled CLIP and Consensus Scores(https://arxiv.org/abs/2405.01028)
• Keywords: secure
• Abstract: This report presents the ECO (Ensembled Clip score and cOnsensus score) pipeline from team DSBA LAB, which is a new framework used to evaluate and rank captions for a given image. ECO selects the most accurate caption describing image. It is made possible by combining an Ensembled CLIP score, which considers the semantic alignment between the image and captions, with a Consensus score that accounts for the essentialness of the captions. Using this framework, we achieved notable success in the CVPR 2024 Workshop Challenge on Caption Re-ranking Evaluation at the New Frontiers for Zero-Shot Image Captioning Evaluation (NICE). Specifically, we secured third place based on the CIDEr metric, second in both the SPICE and METEOR metrics, and first in the ROUGE-L and all BLEU Score metrics. The code and configuration for the ECO framework are available at https://github.com/ DSBA-Lab/ECO .

Title: Towards Trust Proof for Secure Confidential Virtual Machines

• Authors: Jingkai Mao, Haoran Zhu, Junchao Fan, Lin Li, Xiaolin Chang
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2405.01030
• Pdf URL: https://arxiv.org/pdf/2405.01030
• Copy Paste: [[2405.01030]] Towards Trust Proof for Secure Confidential Virtual Machines(https://arxiv.org/abs/2405.01030)
• Keywords: secure, security, privacy, protect
• Abstract: The Virtual Machine (VM)-based Trusted-Execution-Environment (TEE) technology, like AMD Secure-Encrypted-Virtualization (SEV), enables the establishment of Confidential VMs (CVMs) to protect data privacy. But CVM lacks ways to provide the trust proof of its running state, degrading the user confidence of using CVM. The technology of virtual Trusted Platform Module (vTPM) can be used to generate trust proof for CVM. However, the existing vTPM-based approaches have the weaknesses like lack of a well-defined root-of-trust, lack of vTPM protection, and lack of vTPM's trust proof. These weaknesses prevent the generation of the trust proof of the CVM. This paper proposes an approach to generate the trust proof for AMD SEV-based CVM so as to ensure its security by using a secure vTPM to construct Trusted Complete Chain for the CVM (T3CVM). T3CVM consists of three components: 1) TR-Manager, as the well-defined root-of-trust, helps to build complete trust chains for CVMs; 2) CN-TPMCVM, a special CVM provides secure vTPMs; 3) CN-CDriver, an enhanced TPM driver. Our approach overcomes the weaknesses of existing approaches and enables trusted computing-based applications to run seamlessly in the trusted CVM. We perform a formal security analysis of T3CVM, and implement a prototype system to evaluate its performance.

Title: The Privacy Power of Correlated Noise in Decentralized Learning

• Authors: Youssef Allouah, Anastasia Koloskova, Aymane El Firdoussi, Martin Jaggi, Rachid Guerraoui
• Subjects: cs.LG, cs.CR, cs.DC, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2405.01031
• Pdf URL: https://arxiv.org/pdf/2405.01031
• Copy Paste: [[2405.01031]] The Privacy Power of Correlated Noise in Decentralized Learning(https://arxiv.org/abs/2405.01031)
• Keywords: secure, privacy, protect
• Abstract: Decentralized learning is appealing as it enables the scalable usage of large amounts of distributed data and resources (without resorting to any central entity), while promoting privacy since every user minimizes the direct exposure of their data. Yet, without additional precautions, curious users can still leverage models obtained from their peers to violate privacy. In this paper, we propose Decor, a variant of decentralized SGD with differential privacy (DP) guarantees. Essentially, in Decor, users securely exchange randomness seeds in one communication round to generate pairwise-canceling correlated Gaussian noises, which are injected to protect local models at every communication round. We theoretically and empirically show that, for arbitrary connected graphs, Decor matches the central DP optimal privacy-utility trade-off. We do so under SecLDP, our new relaxation of local DP, which protects all user communications against an external eavesdropper and curious users, assuming that every pair of connected users shares a secret, i.e., an information hidden to all others. The main theoretical challenge is to control the accumulation of non-canceling correlated noise due to network sparsity. We also propose a companion SecLDP privacy accountant for public use.

Title: CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes

• Authors: Seong-Joon Park, Hee-Youl Kwak, Sang-Hyo Kim, Yongjune Kim, Jong-Seon No
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2405.01033
• Pdf URL: https://arxiv.org/pdf/2405.01033
• Copy Paste: [[2405.01033]] CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes(https://arxiv.org/abs/2405.01033)
• Keywords: transformer
• Abstract: Error correcting codes~(ECCs) are indispensable for reliable transmission in communication systems. The recent advancements in deep learning have catalyzed the exploration of ECC decoders based on neural networks. Among these, transformer-based neural decoders have achieved state-of-the-art decoding performance. In this paper, we propose a novel Cross-attention Message-Passing Transformer~(CrossMPT). CrossMPT iteratively updates two types of input vectors (i.e., magnitude and syndrome vectors) using two masked cross-attention blocks. The mask matrices in these cross-attention blocks are determined by the code's parity-check matrix that delineates the relationship between magnitude and syndrome vectors. Our experimental results show that CrossMPT significantly outperforms existing neural network-based decoders, particularly in decoding low-density parity-check codes. Notably, CrossMPT also achieves a significant reduction in computational complexity, achieving over a 50\% decrease in its attention layers compared to the original transformer-based decoder, while retaining the computational complexity of the remaining layers.

Title: Development of Cybersecurity Simulator-Based Platform for the Protection of Critical Infrastructures

• Authors: Tero Vartiainen, Duong Dang, Mike Mekkanen, Emmanuel Anti
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01046
• Pdf URL: https://arxiv.org/pdf/2405.01046
• Copy Paste: [[2405.01046]] Development of Cybersecurity Simulator-Based Platform for the Protection of Critical Infrastructures(https://arxiv.org/abs/2405.01046)
• Keywords: security, protect, attack
• Abstract: Critical infrastructures (CNI) are vulnerable to cyberattacks due to their interconnected communication systems. We are developing a platform using real-time simulation of cyber-physical systems to enhance CNI resilience and security. The platform, initiated in the Vaasa Harbor Microgrid, allows creation of a digital twin and real-time execution of its functions. It provides a co-simulation environment for simulating cyberattack scenarios, aiding in the design of a cybersecurity simulator-based platform and offering services for CNI stakeholders.

Title: Leverage Multi-source Traffic Demand Data Fusion with Transformer Model for Urban Parking Prediction

• Authors: Yin Huang, Yongqi Dong, Youhua Tang, Li Li
• Subjects: cs.LG, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2405.01055
• Pdf URL: https://arxiv.org/pdf/2405.01055
• Copy Paste: [[2405.01055]] Leverage Multi-source Traffic Demand Data Fusion with Transformer Model for Urban Parking Prediction(https://arxiv.org/abs/2405.01055)
• Keywords: transformer
• Abstract: The escalation in urban private car ownership has worsened the urban parking predicament, necessitating effective parking availability prediction for urban planning and management. However, the existing prediction methods suffer from low prediction accuracy with the lack of spatial-temporal correlation features related to parking volume, and neglect of flow patterns and correlations between similar parking lots within certain areas. To address these challenges, this study proposes a parking availability prediction framework integrating spatial-temporal deep learning with multi-source data fusion, encompassing traffic demand data from multiple sources (e.g., metro, bus, taxi services), and parking lot data. The framework is based on the Transformer as the spatial-temporal deep learning model and leverages K-means clustering to establish parking cluster zones, extracting and integrating traffic demand characteristics from various transportation modes (i.e., metro, bus, online ride-hailing, and taxi) connected to parking lots. Real-world empirical data was used to verify the effectiveness of the proposed method compared with different machine learning, deep learning, and traditional statistical models for predicting parking availability. Experimental results reveal that, with the proposed pipeline, the developed Transformer model outperforms other models in terms of various metrics, e.g., Mean Squared Error (MSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE). By fusing multi-source demanding data with spatial-temporal deep learning techniques, this approach offers the potential to develop parking availability prediction systems that furnish more accurate and timely information to both drivers and urban planners, thereby fostering more efficient and sustainable urban mobility.

Title: A text-based, generative deep learning model for soil reflectance spectrum simulation in the VIS-NIR (400-2499 nm) bands

• Authors: Tong Lei, Brian N. Bailey
• Subjects: cs.LG, cs.AI, cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2405.01060
• Pdf URL: https://arxiv.org/pdf/2405.01060
• Copy Paste: [[2405.01060]] A text-based, generative deep learning model for soil reflectance spectrum simulation in the VIS-NIR (400-2499 nm) bands(https://arxiv.org/abs/2405.01060)
• Keywords: diffusion, generative
• Abstract: Simulating soil reflectance spectra is invaluable for soil-plant radiative modeling and training machine learning models, yet it is difficult as the intricate relationships between soil structure and its constituents. To address this, a fully data-driven soil optics generative model (SOGM) for simulation of soil reflectance spectra based on soil property inputs was developed. The model is trained on an extensive dataset comprising nearly 180,000 soil spectra-property pairs from 17 datasets. It generates soil reflectance spectra from text-based inputs describing soil properties and their values rather than only numerical values and labels in binary vector format. The generative model can simulate output spectra based on an incomplete set of input properties. SOGM is based on the denoising diffusion probabilistic model (DDPM). Two additional sub-models were also built to complement the SOGM: a spectral padding model that can fill in the gaps for spectra shorter than the full visible-near-infrared range (VIS-NIR; 400 to 2499 nm), and a wet soil spectra model that can estimate the effects of water content on soil reflectance spectra given the dry spectrum predicted by the SOGM. The SOGM was up-scaled by coupling with the Helios 3D plant modeling software, which allowed for generation of synthetic aerial images of simulated soil and plant scenes. It can also be easily integrated with soil-plant radiation model used for remote sensin research like PROSAIL. The testing results of the SOGM on new datasets that not included in model training proved that the model can generate reasonable soil reflectance spectra based on available property inputs. The presented models are openly accessible on: https://github.com/GEMINI-Breeding/SOGM_soil_spectra_simulation.

Title: MFDS-Net: Multi-Scale Feature Depth-Supervised Network for Remote Sensing Change Detection with Global Semantic and Detail Information

• Authors: Zhenyang Huang, Zhaojin Fu, Song Jintao, Genji Yuan, Jinjiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01065
• Pdf URL: https://arxiv.org/pdf/2405.01065
• Copy Paste: [[2405.01065]] MFDS-Net: Multi-Scale Feature Depth-Supervised Network for Remote Sensing Change Detection with Global Semantic and Detail Information(https://arxiv.org/abs/2405.01065)
• Keywords: extraction
• Abstract: Change detection as an interdisciplinary discipline in the field of computer vision and remote sensing at present has been receiving extensive attention and research. Due to the rapid development of society, the geographic information captured by remote sensing satellites is changing faster and more complex, which undoubtedly poses a higher challenge and highlights the value of change detection tasks. We propose MFDS-Net: Multi-Scale Feature Depth-Supervised Network for Remote Sensing Change Detection with Global Semantic and Detail Information (MFDS-Net) with the aim of achieving a more refined description of changing buildings as well as geographic information, enhancing the localisation of changing targets and the acquisition of weak features. To achieve the research objectives, we use a modified ResNet_34 as backbone network to perform feature extraction and DO-Conv as an alternative to traditional convolution to better focus on the association between feature information and to obtain better training results. We propose the Global Semantic Enhancement Module (GSEM) to enhance the processing of high-level semantic information from a global perspective. The Differential Feature Integration Module (DFIM) is proposed to strengthen the fusion of different depth feature information, achieving learning and extraction of differential features. The entire network is trained and optimized using a deep supervision mechanism. The experimental outcomes of MFDS-Net surpass those of current mainstream change detection networks. On the LEVIR dataset, it achieved an F1 score of 91.589 and IoU of 84.483, on the WHU dataset, the scores were F1: 92.384 and IoU: 86.807, and on the GZ-CD dataset, the scores were F1: 86.377 and IoU: 76.021. The code is available at https://github.com/AOZAKIiii/MFDS-Net

Title: Callico: a Versatile Open-Source Document Image Annotation Platform

• Authors: Christopher Kermorvant, Eva Bardou, Manon Blanco, Bastien Abadie
• Subjects: cs.CV, cs.DL
• Abstract URL: https://arxiv.org/abs/2405.01071
• Pdf URL: https://arxiv.org/pdf/2405.01071
• Copy Paste: [[2405.01071]] Callico: a Versatile Open-Source Document Image Annotation Platform(https://arxiv.org/abs/2405.01071)
• Keywords: extraction
• Abstract: This paper presents Callico, a web-based open source platform designed to simplify the annotation process in document recognition projects. The move towards data-centric AI in machine learning and deep learning underscores the importance of high-quality data, and the need for specialised tools that increase the efficiency and effectiveness of generating such data. For document image annotation, Callico offers dual-display annotation for digitised documents, enabling simultaneous visualisation and annotation of scanned images and text. This capability is critical for OCR and HTR model training, document layout analysis, named entity recognition, form-based key value annotation or hierarchical structure annotation with element grouping. The platform supports collaborative annotation with versatile features backed by a commitment to open source development, high-quality code standards and easy deployment via Docker. Illustrative use cases - including the transcription of the Belfort municipal registers, the indexing of French World War II prisoners for the ICRC, and the extraction of personal information from the Socface project's census lists - demonstrate Callico's applicability and utility.

Title: Poisoning Attacks on Federated Learning for Autonomous Driving

• Authors: Sonakshi Garg, Hugo Jönsson, Gustav Kalander, Axel Nilsson, Bhhaanu Pirange, Viktor Valadi, Johan Östman
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01073
• Pdf URL: https://arxiv.org/pdf/2405.01073
• Copy Paste: [[2405.01073]] Poisoning Attacks on Federated Learning for Autonomous Driving(https://arxiv.org/abs/2405.01073)
• Keywords: defense, attack, steal, federate
• Abstract: Federated Learning (FL) is a decentralized learning paradigm, enabling parties to collaboratively train models while keeping their data confidential. Within autonomous driving, it brings the potential of reducing data storage costs, reducing bandwidth requirements, and to accelerate the learning. FL is, however, susceptible to poisoning attacks. In this paper, we introduce two novel poisoning attacks on FL tailored to regression tasks within autonomous driving: FLStealth and Off-Track Attack (OTA). FLStealth, an untargeted attack, aims at providing model updates that deteriorate the global model performance while appearing benign. OTA, on the other hand, is a targeted attack with the objective to change the global model's behavior when exposed to a certain trigger. We demonstrate the effectiveness of our attacks by conducting comprehensive experiments pertaining to the task of vehicle trajectory prediction. In particular, we show that, among five different untargeted attacks, FLStealth is the most successful at bypassing the considered defenses employed by the server. For OTA, we demonstrate the inability of common defense strategies to mitigate the attack, highlighting the critical need for new defensive mechanisms against targeted attacks within FL for autonomous driving.

Title: KDPrint: Passive Authentication using Keystroke Dynamics-to-Image Encoding via Standardization

• Authors: Yooshin Kim, Namhyeok Kwon, Donghoon Shin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01080
• Pdf URL: https://arxiv.org/pdf/2405.01080
• Copy Paste: [[2405.01080]] KDPrint: Passive Authentication using Keystroke Dynamics-to-Image Encoding via Standardization(https://arxiv.org/abs/2405.01080)
• Keywords: attack, biometric
• Abstract: In contemporary mobile user authentication systems, verifying user legitimacy has become paramount due to the widespread use of smartphones. Although fingerprint and facial recognition are widely used for mobile authentication, PIN-based authentication is still employed as a fallback option if biometric authentication fails after multiple attempts. Consequently, the system remains susceptible to attacks targeting the PIN when biometric methods are unsuccessful. In response to these concerns, two-factor authentication has been proposed, albeit with the caveat of increased user effort. To address these challenges, this paper proposes a passive authentication system that utilizes keystroke data, a byproduct of primary authentication methods, for background user authentication. Additionally, we introduce a novel image encoding technique to capture the temporal dynamics of keystroke data, overcoming the performance limitations of deep learning models. Furthermore, we present a methodology for selecting suitable behavioral biometric features for image representation. The resulting images, depicting the user's PIN input patterns, enhance the model's ability to uniquely identify users through the secondary channel with high accuracy. Experimental results demonstrate that the proposed imaging approach surpasses existing methods in terms of information capacity. In self-collected dataset experiments, incorporating features from prior research, our method achieved an Equal Error Rate (EER) of 6.7\%, outperforming the existing method's 47.7\%. Moreover, our imaging technique attained a True Acceptance Rate (TAR) of 94.4\% and a False Acceptance Rate (FAR) of 8\% for 17 users.

Title: Single Image Super-Resolution Based on Global-Local Information Synergy

• Authors: Nianzu Qiao, Lamei Di, Changyin Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01085
• Pdf URL: https://arxiv.org/pdf/2405.01085
• Copy Paste: [[2405.01085]] Single Image Super-Resolution Based on Global-Local Information Synergy(https://arxiv.org/abs/2405.01085)
• Keywords: extraction, transformer
• Abstract: Although several image super-resolution solutions exist, they still face many challenges. CNN-based algorithms, despite the reduction in computational complexity, still need to improve their accuracy. While Transformer-based algorithms have higher accuracy, their ultra-high computational complexity makes them difficult to be accepted in practical applications. To overcome the existing challenges, a novel super-resolution reconstruction algorithm is proposed in this paper. The algorithm achieves a significant increase in accuracy through a unique design while maintaining a low complexity. The core of the algorithm lies in its cleverly designed Global-Local Information Extraction Module and Basic Block Module. By combining global and local information, the Global-Local Information Extraction Module aims to understand the image content more comprehensively so as to recover the global structure and local details in the image more accurately, which provides rich information support for the subsequent reconstruction process. Experimental results show that the comprehensive performance of the algorithm proposed in this paper is optimal, providing an efficient and practical new solution in the field of super-resolution reconstruction.

Title: Type2Branch: Keystroke Biometrics based on a Dual-branch Architecture with Attention Mechanisms and Set2set Loss

• Authors: Nahuel González, Giuseppe Stragapede, Rubén Vera-Rodriguez, Rubén Tolosana
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01088
• Pdf URL: https://arxiv.org/pdf/2405.01088
• Copy Paste: [[2405.01088]] Type2Branch: Keystroke Biometrics based on a Dual-branch Architecture with Attention Mechanisms and Set2set Loss(https://arxiv.org/abs/2405.01088)
• Keywords: robust, biometric, fair
• Abstract: In 2021, the pioneering work on TypeNet showed that keystroke dynamics verification could scale to hundreds of thousands of users with minimal performance degradation. Recently, the KVC-onGoing competition has provided an open and robust experimental protocol for evaluating keystroke dynamics verification systems of such scale, including considerations of algorithmic fairness. This article describes Type2Branch, the model and techniques that achieved the lowest error rates at the KVC-onGoing, in both desktop and mobile scenarios. The novelty aspects of the proposed Type2Branch include: i) synthesized timing features emphasizing user behavior deviation from the general population, ii) a dual-branch architecture combining recurrent and convolutional paths with various attention mechanisms, iii) a new loss function named Set2set that captures the global structure of the embedding space, and iv) a training curriculum of increasing difficulty. Considering five enrollment samples per subject of approximately 50 characters typed, the proposed Type2Branch achieves state-of-the-art performance with mean per-subject EERs of 0.77% and 1.03% on evaluation sets of respectively 15,000 and 5,000 subjects for desktop and mobile scenarios. With a uniform global threshold for all subjects, the EERs are 3.25% for desktop and 3.61% for mobile, outperforming previous approaches by a significant margin.

Title: Learning Object States from Actions via Large Language Models

• Authors: Masatoshi Tateno, Takuma Yagi, Ryosuke Furuta, Yoichi Sato
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01090
• Pdf URL: https://arxiv.org/pdf/2405.01090
• Copy Paste: [[2405.01090]] Learning Object States from Actions via Large Language Models(https://arxiv.org/abs/2405.01090)
• Keywords: large language model
• Abstract: Temporally localizing the presence of object states in videos is crucial in understanding human activities beyond actions and objects. This task has suffered from a lack of training data due to object states' inherent ambiguity and variety. To avoid exhaustive annotation, learning from transcribed narrations in instructional videos would be intriguing. However, object states are less described in narrations compared to actions, making them less effective. In this work, we propose to extract the object state information from action information included in narrations, using large language models (LLMs). Our observation is that LLMs include world knowledge on the relationship between actions and their resulting object states, and can infer the presence of object states from past action sequences. The proposed LLM-based framework offers flexibility to generate plausible pseudo-object state labels against arbitrary categories. We evaluate our method with our newly collected Multiple Object States Transition (MOST) dataset including dense temporal annotation of 60 object state categories. Our model trained by the generated pseudo-labels demonstrates significant improvement of over 29% in mAP against strong zero-shot vision-language models, showing the effectiveness of explicitly extracting object state information from actions through LLMs.

Title: Transformers Fusion across Disjoint Samples for Hyperspectral Image Classification

• Authors: Muhammad Ahmad, Manuel Mazzara, Salvatore Distifano
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2405.01095
• Pdf URL: https://arxiv.org/pdf/2405.01095
• Copy Paste: [[2405.01095]] Transformers Fusion across Disjoint Samples for Hyperspectral Image Classification(https://arxiv.org/abs/2405.01095)
• Keywords: robust, transformer
• Abstract: 3D Swin Transformer (3D-ST) known for its hierarchical attention and window-based processing, excels in capturing intricate spatial relationships within images. Spatial-spectral Transformer (SST), meanwhile, specializes in modeling long-range dependencies through self-attention mechanisms. Therefore, this paper introduces a novel method: an attentional fusion of these two transformers to significantly enhance the classification performance of Hyperspectral Images (HSIs). What sets this approach apart is its emphasis on the integration of attentional mechanisms from both architectures. This integration not only refines the modeling of spatial and spectral information but also contributes to achieving more precise and accurate classification results. The experimentation and evaluation of benchmark HSI datasets underscore the importance of employing disjoint training, validation, and test samples. The results demonstrate the effectiveness of the fusion approach, showcasing its superiority over traditional methods and individual transformers. Incorporating disjoint samples enhances the robustness and reliability of the proposed methodology, emphasizing its potential for advancing hyperspectral image classification.

Title: Enhancing Person Re-Identification via Uncertainty Feature Fusion and Wise Distance Aggregation

• Authors: Quang-Huy Che, Le-Chuong Nguyen, Vinh-Tiep Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01101
• Pdf URL: https://arxiv.org/pdf/2405.01101
• Copy Paste: [[2405.01101]] Enhancing Person Re-Identification via Uncertainty Feature Fusion and Wise Distance Aggregation(https://arxiv.org/abs/2405.01101)
• Keywords: security, robust
• Abstract: The quest for robust Person re-identification (Re-ID) systems capable of accurately identifying subjects across diverse scenarios remains a formidable challenge in surveillance and security applications. This study presents a novel methodology that significantly enhances Person Re-Identification (Re-ID) by integrating Uncertainty Feature Fusion (UFFM) with Wise Distance Aggregation (WDA). Tested on benchmark datasets - Market-1501, DukeMTMC-ReID, and MSMT17 - our approach demonstrates substantial improvements in Rank-1 accuracy and mean Average Precision (mAP). Specifically, UFFM capitalizes on the power of feature synthesis from multiple images to overcome the limitations imposed by the variability of subject appearances across different views. WDA further refines the process by intelligently aggregating similarity metrics, thereby enhancing the system's ability to discern subtle but critical differences between subjects. The empirical results affirm the superiority of our method over existing approaches, achieving new performance benchmarks across all evaluated datasets. Code is available on Github.

Title: Less is More: on the Over-Globalizing Problem in Graph Transformers

• Authors: Yujie Xing, Xiao Wang, Yibo Li, Hai Huang, Chuan Shi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01102
• Pdf URL: https://arxiv.org/pdf/2405.01102
• Copy Paste: [[2405.01102]] Less is More: on the Over-Globalizing Problem in Graph Transformers(https://arxiv.org/abs/2405.01102)
• Keywords: transformer
• Abstract: Graph Transformer, due to its global attention mechanism, has emerged as a new tool in dealing with graph-structured data. It is well recognized that the global attention mechanism considers a wider receptive field in a fully connected graph, leading many to believe that useful information can be extracted from all the nodes. In this paper, we challenge this belief: does the globalizing property always benefit Graph Transformers? We reveal the over-globalizing problem in Graph Transformer by presenting both empirical evidence and theoretical analysis, i.e., the current attention mechanism overly focuses on those distant nodes, while the near nodes, which actually contain most of the useful information, are relatively weakened. Then we propose a novel Bi-Level Global Graph Transformer with Collaborative Training (CoBFormer), including the inter-cluster and intra-cluster Transformers, to prevent the over-globalizing problem while keeping the ability to extract valuable information from distant nodes. Moreover, the collaborative training is proposed to improve the model's generalization ability with a theoretical guarantee. Extensive experiments on various graphs well validate the effectiveness of our proposed CoBFormer.

Title: Image segmentation of treated and untreated tumor spheroids by Fully Convolutional Networks

• Authors: Matthias Streller, Soňa Michlíková, Willy Ciecior, Katharina Lönnecke, Leoni A. Kunz-Schughart, Steffen Lange, Anja Voss-Böhme
• Subjects: cs.CV, q-bio.QM, q-bio.TO
• Abstract URL: https://arxiv.org/abs/2405.01105
• Pdf URL: https://arxiv.org/pdf/2405.01105
• Copy Paste: [[2405.01105]] Image segmentation of treated and untreated tumor spheroids by Fully Convolutional Networks(https://arxiv.org/abs/2405.01105)
• Keywords: segmentation
• Abstract: Multicellular tumor spheroids (MCTS) are advanced cell culture systems for assessing the impact of combinatorial radio(chemo)therapy. They exhibit therapeutically relevant in-vivo-like characteristics from 3D cell-cell and cell-matrix interactions to radial pathophysiological gradients related to proliferative activity and nutrient/oxygen supply, altering cellular radioresponse. State-of-the-art assays quantify long-term curative endpoints based on collected brightfield image time series from large treated spheroid populations per irradiation dose and treatment arm. Here, spheroid control probabilities are documented analogous to in-vivo tumor control probabilities based on Kaplan-Meier curves. This analyses require laborious spheroid segmentation of up to 100.000 images per treatment arm to extract relevant structural information from the images, e.g., diameter, area, volume and circularity. While several image analysis algorithms are available for spheroid segmentation, they all focus on compact MCTS with clearly distinguishable outer rim throughout growth. However, treated MCTS may partly be detached and destroyed and are usually obscured by dead cell debris. We successfully train two Fully Convolutional Networks, UNet and HRNet, and optimize their hyperparameters to develop an automatic segmentation for both untreated and treated MCTS. We systematically validate the automatic segmentation on larger, independent data sets of spheroids derived from two human head-and-neck cancer cell lines. We find an excellent overlap between manual and automatic segmentation for most images, quantified by Jaccard indices at around 90%. For images with smaller overlap of the segmentations, we demonstrate that this error is comparable to the variations across segmentations from different biological experts, suggesting that these images represent biologically unclear or ambiguous cases.

Title: Federated Learning with Heterogeneous Data Handling for Robust Vehicular Object Detection

• Authors: Ahmad Khalil, Tizian Dege, Pegah Golchin, Rostyslav Olshevskyi, Antonio Fernandez Anta, Tobias Meuser
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01108
• Pdf URL: https://arxiv.org/pdf/2405.01108
• Copy Paste: [[2405.01108]] Federated Learning with Heterogeneous Data Handling for Robust Vehicular Object Detection(https://arxiv.org/abs/2405.01108)
• Keywords: robust, federate
• Abstract: In the pursuit of refining precise perception models for fully autonomous driving, continual online model training becomes essential. Federated Learning (FL) within vehicular networks offers an efficient mechanism for model training while preserving raw sensory data integrity. Yet, FL struggles with non-identically distributed data (e.g., quantity skew), leading to suboptimal convergence rates during model training. In previous work, we introduced FedLA, an innovative Label-Aware aggregation method addressing data heterogeneity in FL for generic scenarios. In this paper, we introduce FedProx+LA, a novel FL method building upon the state-of-the-art FedProx and FedLA to tackle data heterogeneity, which is specifically tailored for vehicular networks. We evaluate the efficacy of FedProx+LA in continuous online object detection model training. Through a comparative analysis against conventional and state-of-the-art methods, our findings reveal the superior convergence rate of FedProx+LA. Notably, if the label distribution is very heterogeneous, our FedProx+LA approach shows substantial improvements in detection performance compared to baseline methods, also outperforming our previous FedLA approach. Moreover, both FedLA and FedProx+LA increase convergence speed by 30% compared to baseline methods.

Title: Mining REST APIs for Potential Mass Assignment Vulnerabilities

• Authors: Arash Mazidi, Davide Corradini, Mohammad Ghafari
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01111
• Pdf URL: https://arxiv.org/pdf/2405.01111
• Copy Paste: [[2405.01111]] Mining REST APIs for Potential Mass Assignment Vulnerabilities(https://arxiv.org/abs/2405.01111)
• Keywords: security, protect
• Abstract: REST APIs have a pivotal role in accessing protected resources within cyberspace. Despite the availability of security testing tools, mass assignment vulnerabilities are common, yielding unauthorized access to sensitive data. We propose a lightweight approach to mine the REST API specifications and identify operations and attributes that are prone to mass assignment. We conducted a preliminary study on 100 APIs and found 25 prone to this vulnerability. We confirmed nine real vulnerable operations in six open-source APIs.

Title: Sports Analysis and VR Viewing System Based on Player Tracking and Pose Estimation with Multimodal and Multiview Sensors

• Authors: Wenxuan Guo, Zhiyu Pan, Ziheng Xi, Alapati Tuerxun, Jianjiang Feng, Jie Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01112
• Pdf URL: https://arxiv.org/pdf/2405.01112
• Copy Paste: [[2405.01112]] Sports Analysis and VR Viewing System Based on Player Tracking and Pose Estimation with Multimodal and Multiview Sensors(https://arxiv.org/abs/2405.01112)
• Keywords: robust
• Abstract: Sports analysis and viewing play a pivotal role in the current sports domain, offering significant value not only to coaches and athletes but also to fans and the media. In recent years, the rapid development of virtual reality (VR) and augmented reality (AR) technologies have introduced a new platform for watching games. Visualization of sports competitions in VR/AR represents a revolutionary technology, providing audiences with a novel immersive viewing experience. However, there is still a lack of related research in this area. In this work, we present for the first time a comprehensive system for sports competition analysis and real-time visualization on VR/AR platforms. First, we utilize multiview LiDARs and cameras to collect multimodal game data. Subsequently, we propose a framework for multi-player tracking and pose estimation based on a limited amount of supervised data, which extracts precise player positions and movements from point clouds and images. Moreover, we perform avatar modeling of players to obtain their 3D models. Ultimately, using these 3D player data, we conduct competition analysis and real-time visualization on VR/AR. Extensive quantitative experiments demonstrate the accuracy and robustness of our multi-player tracking and pose estimation framework. The visualization results showcase the immense potential of our sports visualization system on the domain of watching games on VR/AR devices. The multimodal competition dataset we collected and all related code will be released soon.

Title: A Survey of the Overlooked Dangers of Template Engines

• Authors: Lorenzo Pisu, Davide Maiorca, Giorgio Giacinto
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01118
• Pdf URL: https://arxiv.org/pdf/2405.01118
• Copy Paste: [[2405.01118]] A Survey of the Overlooked Dangers of Template Engines(https://arxiv.org/abs/2405.01118)
• Keywords: security, attack
• Abstract: Template engines play a pivotal role in modern web application development, facilitating the dynamic rendering of content, products, and user interfaces. Nowadays, template engines are essential in any website that deals with dynamic data, from e-commerce platforms to social media. However, their widespread use also makes them attractive targets for attackers seeking to exploit vulnerabilities and gain unauthorized access to web servers. This paper presents a comprehensive survey of template engines, focusing on their susceptibility to Remote Code Execution (RCE) attacks, a critical security concern in web application development.

Title: Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts

• Authors: Lotem Golany, Filippo Galgani, Maya Mamo, Nimrod Parasol, Omer Vandsburger, Nadav Bar, Ido Dagan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01121
• Pdf URL: https://arxiv.org/pdf/2405.01121
• Copy Paste: [[2405.01121]] Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts(https://arxiv.org/abs/2405.01121)
• Keywords: large language model
• Abstract: Existing methods for creating source-grounded information-seeking dialog datasets are often costly and hard to implement due to their sole reliance on human annotators. We propose combining large language models (LLMs) prompting with human expertise for more efficient and reliable data generation. Instead of the labor-intensive Wizard-of-Oz (WOZ) method, where two annotators generate a dialog from scratch, role-playing agent and user, we use LLM generation to simulate the two roles. Annotators then verify the output and augment it with attribution data. We demonstrate our method by constructing MISeD -- Meeting Information Seeking Dialogs dataset -- the first information-seeking dialog dataset focused on meeting transcripts. Models finetuned with MISeD demonstrate superior performance on our test set, as well as on a novel fully-manual WOZ test set and an existing query-based summarization benchmark, suggesting the utility of our approach.

Title: Automated Virtual Product Placement and Assessment in Images using Diffusion Models

• Authors: Mohammad Mahmudul Alam, Negin Sokhandan, Emmett Goodman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01130
• Pdf URL: https://arxiv.org/pdf/2405.01130
• Copy Paste: [[2405.01130]] Automated Virtual Product Placement and Assessment in Images using Diffusion Models(https://arxiv.org/abs/2405.01130)
• Keywords: diffusion, segmentation
• Abstract: In Virtual Product Placement (VPP) applications, the discrete integration of specific brand products into images or videos has emerged as a challenging yet important task. This paper introduces a novel three-stage fully automated VPP system. In the first stage, a language-guided image segmentation model identifies optimal regions within images for product inpainting. In the second stage, Stable Diffusion (SD), fine-tuned with a few example product images, is used to inpaint the product into the previously identified candidate regions. The final stage introduces an "Alignment Module", which is designed to effectively sieve out low-quality images. Comprehensive experiments demonstrate that the Alignment Module ensures the presence of the intended product in every generated image and enhances the average quality of images by 35%. The results presented in this paper demonstrate the effectiveness of the proposed VPP system, which holds significant potential for transforming the landscape of virtual advertising and marketing strategies.

Title: Sharp Bounds for Sequential Federated Learning on Heterogeneous Data

• Authors: Yipeng Li, Xinchen Lyu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01142
• Pdf URL: https://arxiv.org/pdf/2405.01142
• Copy Paste: [[2405.01142]] Sharp Bounds for Sequential Federated Learning on Heterogeneous Data(https://arxiv.org/abs/2405.01142)
• Keywords: federate
• Abstract: There are two paradigms in Federated Learning (FL): parallel FL (PFL), where models are trained in a parallel manner across clients; and sequential FL (SFL), where models are trained in a sequential manner across clients. In contrast to that of PFL, the convergence theory of SFL on heterogeneous data is still lacking. To resolve the theoretical dilemma of SFL, we establish sharp convergence guarantees for SFL on heterogeneous data with both upper and lower bounds. Specifically, we derive the upper bounds for strongly convex, general convex and non-convex objective functions, and construct the matching lower bounds for the strongly convex and general convex objective functions. Then, we compare the upper bounds of SFL with those of PFL, showing that SFL outperforms PFL (at least, when the level of heterogeneity is relatively high). Experimental results on quadratic functions and real data sets validate the counterintuitive comparison result.

Title: Boosting Communication Efficiency of Federated Learning's Secure Aggregation

• Authors: Niousha Nazemi, Omid Tavallaie, Shuaijun Chen, Albert Y. Zomaya, Ralph Holz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01144
• Pdf URL: https://arxiv.org/pdf/2405.01144
• Copy Paste: [[2405.01144]] Boosting Communication Efficiency of Federated Learning's Secure Aggregation(https://arxiv.org/abs/2405.01144)
• Keywords: secure, privacy, attack, federate
• Abstract: Federated Learning (FL) is a decentralized machine learning approach where client devices train models locally and send them to a server that performs aggregation to generate a global model. FL is vulnerable to model inversion attacks, where the server can infer sensitive client data from trained models. Google's Secure Aggregation (SecAgg) protocol addresses this data privacy issue by masking each client's trained model using shared secrets and individual elements generated locally on the client's device. Although SecAgg effectively preserves privacy, it imposes considerable communication and computation overhead, especially as network size increases. Building upon SecAgg, this poster introduces a Communication-Efficient Secure Aggregation (CESA) protocol that substantially reduces this overhead by using only two shared secrets per client to mask the model. We propose our method for stable networks with low delay variation and limited client dropouts. CESA is independent of the data distribution and network size (for higher than 6 nodes), preventing the honest-but-curious server from accessing unmasked models. Our initial evaluation reveals that CESA significantly reduces the communication cost compared to SecAgg.

Title: SynFlowNet: Towards Molecule Design with Guaranteed Synthesis Pathways

• Authors: Miruna Cretu, Charles Harris, Julien Roy, Emmanuel Bengio, Pietro Liò
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2405.01155
• Pdf URL: https://arxiv.org/pdf/2405.01155
• Copy Paste: [[2405.01155]] SynFlowNet: Towards Molecule Design with Guaranteed Synthesis Pathways(https://arxiv.org/abs/2405.01155)
• Keywords: generative
• Abstract: Recent breakthroughs in generative modelling have led to a number of works proposing molecular generation models for drug discovery. While these models perform well at capturing drug-like motifs, they are known to often produce synthetically inaccessible molecules. This is because they are trained to compose atoms or fragments in a way that approximates the training distribution, but they are not explicitly aware of the synthesis constraints that come with making molecules in the lab. To address this issue, we introduce SynFlowNet, a GFlowNet model whose action space uses chemically validated reactions and reactants to sequentially build new molecules. We evaluate our approach using synthetic accessibility scores and an independent retrosynthesis tool. SynFlowNet consistently samples synthetically feasible molecules, while still being able to find diverse and high-utility candidates. Furthermore, we compare molecules designed with SynFlowNet to experimentally validated actives, and find that they show comparable properties of interest, such as molecular weight, SA score and predicted protein binding affinity.

Title: Self-Supervised Learning for Interventional Image Analytics: Towards Robust Device Trackers

• Authors: Saahil Islam, Venkatesh N. Murthy, Dominik Neumann, Badhan Kumar Das, Puneet Sharma, Andreas Maier, Dorin Comaniciu, Florin C. Ghesu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01156
• Pdf URL: https://arxiv.org/pdf/2405.01156
• Copy Paste: [[2405.01156]] Self-Supervised Learning for Interventional Image Analytics: Towards Robust Device Trackers(https://arxiv.org/abs/2405.01156)
• Keywords: robust
• Abstract: An accurate detection and tracking of devices such as guiding catheters in live X-ray image acquisitions is an essential prerequisite for endovascular cardiac interventions. This information is leveraged for procedural guidance, e.g., directing stent placements. To ensure procedural safety and efficacy, there is a need for high robustness no failures during tracking. To achieve that, one needs to efficiently tackle challenges, such as: device obscuration by contrast agent or other external devices or wires, changes in field-of-view or acquisition angle, as well as the continuous movement due to cardiac and respiratory motion. To overcome the aforementioned challenges, we propose a novel approach to learn spatio-temporal features from a very large data cohort of over 16 million interventional X-ray frames using self-supervision for image sequence data. Our approach is based on a masked image modeling technique that leverages frame interpolation based reconstruction to learn fine inter-frame temporal correspondences. The features encoded in the resulting model are fine-tuned downstream. Our approach achieves state-of-the-art performance and in particular robustness compared to ultra optimized reference solutions (that use multi-stage feature fusion, multi-task and flow regularization). The experiments show that our method achieves 66.31% reduction in maximum tracking error against reference solutions (23.20% when flow regularization is used); achieving a success score of 97.95% at a 3x faster inference speed of 42 frames-per-second (on GPU). The results encourage the use of our approach in various other tasks within interventional image analytics that require effective understanding of spatio-temporal semantics.

Title: GroupedMixer: An Entropy Model with Group-wise Token-Mixers for Learned Image Compression

• Authors: Daxin Li, Yuanchao Bai, Kai Wang, Junjun Jiang, Xianming Liu, Wen Gao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2405.01170
• Pdf URL: https://arxiv.org/pdf/2405.01170
• Copy Paste: [[2405.01170]] GroupedMixer: An Entropy Model with Group-wise Token-Mixers for Learned Image Compression(https://arxiv.org/abs/2405.01170)
• Keywords: transformer
• Abstract: Transformer-based entropy models have gained prominence in recent years due to their superior ability to capture long-range dependencies in probability distribution estimation compared to convolution-based methods. However, previous transformer-based entropy models suffer from a sluggish coding process due to pixel-wise autoregression or duplicated computation during inference. In this paper, we propose a novel transformer-based entropy model called GroupedMixer, which enjoys both faster coding speed and better compression performance than previous transformer-based methods. Specifically, our approach builds upon group-wise autoregression by first partitioning the latent variables into groups along spatial-channel dimensions, and then entropy coding the groups with the proposed transformer-based entropy model. The global causal self-attention is decomposed into more efficient group-wise interactions, implemented using inner-group and cross-group token-mixers. The inner-group token-mixer incorporates contextual elements within a group while the cross-group token-mixer interacts with previously decoded groups. Alternate arrangement of two token-mixers enables global contextual reference. To further expedite the network inference, we introduce context cache optimization to GroupedMixer, which caches attention activation values in cross-group token-mixers and avoids complex and duplicated computation. Experimental results demonstrate that the proposed GroupedMixer yields the state-of-the-art rate-distortion performance with fast compression speed.

Title: Uncertainty-aware self-training with expectation maximization basis transformation

• Authors: Zijia Wang, Wenbin Yang, Zhisong Liu, Zhen Jia
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01175
• Pdf URL: https://arxiv.org/pdf/2405.01175
• Copy Paste: [[2405.01175]] Uncertainty-aware self-training with expectation maximization basis transformation(https://arxiv.org/abs/2405.01175)
• Keywords: extraction, segmentation
• Abstract: Self-training is a powerful approach to deep learning. The key process is to find a pseudo-label for modeling. However, previous self-training algorithms suffer from the over-confidence issue brought by the hard labels, even some confidence-related regularizers cannot comprehensively catch the uncertainty. Therefore, we propose a new self-training framework to combine uncertainty information of both model and dataset. Specifically, we propose to use Expectation-Maximization (EM) to smooth the labels and comprehensively estimate the uncertainty information. We further design a basis extraction network to estimate the initial basis from the dataset. The obtained basis with uncertainty can be filtered based on uncertainty information. It can then be transformed into the real hard label to iteratively update the model and basis in the retraining process. Experiments on image classification and semantic segmentation show the advantages of our methods among confidence-aware self-training algorithms with 1-3 percentage improvement on different datasets.

Title: Potential Energy based Mixture Model for Noisy Label Learning

• Authors: Zijia Wang, Wenbin Yang, Zhisong Liu, Zhen Jia
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01186
• Pdf URL: https://arxiv.org/pdf/2405.01186
• Copy Paste: [[2405.01186]] Potential Energy based Mixture Model for Noisy Label Learning(https://arxiv.org/abs/2405.01186)
• Keywords: robust
• Abstract: Training deep neural networks (DNNs) from noisy labels is an important and challenging task. However, most existing approaches focus on the corrupted labels and ignore the importance of inherent data structure. To bridge the gap between noisy labels and data, inspired by the concept of potential energy in physics, we propose a novel Potential Energy based Mixture Model (PEMM) for noise-labels learning. We innovate a distance-based classifier with the potential energy regularization on its class centers. Embedding our proposed classifier with existing deep learning backbones, we can have robust networks with better feature representations. They can preserve intrinsic structures from the data, resulting in a superior noisy tolerance. We conducted extensive experiments to analyze the efficiency of our proposed model on several real-world datasets. Quantitative results show that it can achieve state-of-the-art performance.

Title: Gradient-Congruity Guided Federated Sparse Training

• Authors: Chris Xing Tian, Yibing Liu, Haoliang Li, Ray C.C. Cheung, Shiqi Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01189
• Pdf URL: https://arxiv.org/pdf/2405.01189
• Copy Paste: [[2405.01189]] Gradient-Congruity Guided Federated Sparse Training(https://arxiv.org/abs/2405.01189)
• Keywords: privacy, federate
• Abstract: Edge computing allows artificial intelligence and machine learning models to be deployed on edge devices, where they can learn from local data and collaborate to form a global model. Federated learning (FL) is a distributed machine learning technique that facilitates this process while preserving data privacy. However, FL also faces challenges such as high computational and communication costs regarding resource-constrained devices, and poor generalization performance due to the heterogeneity of data across edge clients and the presence of out-of-distribution data. In this paper, we propose the Gradient-Congruity Guided Federated Sparse Training (FedSGC), a novel method that integrates dynamic sparse training and gradient congruity inspection into federated learning framework to address these issues. Our method leverages the idea that the neurons, in which the associated gradients with conflicting directions with respect to the global model contain irrelevant or less generalized information for other clients, and could be pruned during the sparse training process. Conversely, the neurons where the associated gradients with consistent directions could be grown in a higher priority. In this way, FedSGC can greatly reduce the local computation and communication overheads while, at the same time, enhancing the generalization abilities of FL. We evaluate our method on challenging non-i.i.d settings and show that it achieves competitive accuracy with state-of-the-art FL methods across various scenarios while minimizing computation and communication costs.

Title: Decoupling Feature Extraction and Classification Layers for Calibrated Neural Networks

• Authors: Mikkel Jordahn, Pablo Olmos
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2405.01196
• Pdf URL: https://arxiv.org/pdf/2405.01196
• Copy Paste: [[2405.01196]] Decoupling Feature Extraction and Classification Layers for Calibrated Neural Networks(https://arxiv.org/abs/2405.01196)
• Keywords: extraction, transformer
• Abstract: Deep Neural Networks (DNN) have shown great promise in many classification applications, yet are widely known to have poorly calibrated predictions when they are over-parametrized. Improving DNN calibration without comprising on model accuracy is of extreme importance and interest in safety critical applications such as in the health-care sector. In this work, we show that decoupling the training of feature extraction layers and classification layers in over-parametrized DNN architectures such as Wide Residual Networks (WRN) and Visual Transformers (ViT) significantly improves model calibration whilst retaining accuracy, and at a low training cost. In addition, we show that placing a Gaussian prior on the last hidden layer outputs of a DNN, and training the model variationally in the classification training stage, even further improves calibration. We illustrate these methods improve calibration across ViT and WRN architectures for several image classification benchmark datasets.

Title: Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies

• Authors: Finn Rietz, Erik Schaffernicht, Stefan Heinrich, Johannes A. Stork
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01198
• Pdf URL: https://arxiv.org/pdf/2405.01198
• Copy Paste: [[2405.01198]] Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies(https://arxiv.org/abs/2405.01198)
• Keywords: interpretability
• Abstract: Reinforcement learning policies are typically represented by black-box neural networks, which are non-interpretable and not well-suited for safety-critical domains. To address both of these issues, we propose constrained normalizing flow policies as interpretable and safe-by-construction policy models. We achieve safety for reinforcement learning problems with instantaneous safety constraints, for which we can exploit domain knowledge by analytically constructing a normalizing flow that ensures constraint satisfaction. The normalizing flow corresponds to an interpretable sequence of transformations on action samples, each ensuring alignment with respect to a particular constraint. Our experiments reveal benefits beyond interpretability in an easier learning objective and maintained constraint satisfaction throughout the entire learning process. Our approach leverages constraints over reward engineering while offering enhanced interpretability, safety, and direct means of providing domain knowledge to the agent without relying on complex reward functions.

Title: Latent Fingerprint Matching via Dense Minutia Descriptor

• Authors: Zhiyu Pan, Yongjie Duan, Xiongjun Guan, Jianjiang Feng, Jie Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01199
• Pdf URL: https://arxiv.org/pdf/2405.01199
• Copy Paste: [[2405.01199]] Latent Fingerprint Matching via Dense Minutia Descriptor(https://arxiv.org/abs/2405.01199)
• Keywords: extraction, segmentation
• Abstract: Latent fingerprint matching is a daunting task, primarily due to the poor quality of latent fingerprints. In this study, we propose a deep-learning based dense minutia descriptor (DMD) for latent fingerprint matching. A DMD is obtained by extracting the fingerprint patch aligned by its central minutia, capturing detailed minutia information and texture information. Our dense descriptor takes the form of a three-dimensional representation, with two dimensions associated with the original image plane and the other dimension representing the abstract features. Additionally, the extraction process outputs the fingerprint segmentation map, ensuring that the descriptor is only valid in the foreground region. The matching between two descriptors occurs in their overlapping regions, with a score normalization strategy to reduce the impact brought by the differences outside the valid area. Our descriptor achieves state-of-the-art performance on several latent fingerprint datasets. Overall, our DMD is more representative and interpretable compared to previous methods.

Title: Towards Cross-Scale Attention and Surface Supervision for Fractured Bone Segmentation in CT

• Authors: Yu Zhou, Xiahao Zou, Yi Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01204
• Pdf URL: https://arxiv.org/pdf/2405.01204
• Copy Paste: [[2405.01204]] Towards Cross-Scale Attention and Surface Supervision for Fractured Bone Segmentation in CT(https://arxiv.org/abs/2405.01204)
• Keywords: segmentation
• Abstract: Bone segmentation is an essential step for the preoperative planning of fracture trauma surgery. The automated segmentation of fractured bone from computed tomography (CT) scans remains challenging, due to the large differences of fractures in position and morphology, and also the inherent anatomical characteristics of different bone structures. To alleviate these issues, we propose a cross-scale attention mechanism as well as a surface supervision strategy for fractured bone segmentation in CT. Specifically, a cross-scale attention mechanism is introduced to effectively aggregate the features among different scales to provide more powerful fracture representation. Moreover, a surface supervision strategy is employed, which explicitly constrains the network to pay more attention to the bone boundary. The efficacy of the proposed method is evaluated on a public dataset containing CT scans with hip fractures. The evaluation metrics are Dice similarity coefficient (DSC), average symmetric surface distance (ASSD), and Hausdorff distance (95HD). The proposed method achieves an average DSC of 93.36%, ASSD of 0.85mm, 95HD of 7.51mm. Our method offers an effective fracture segmentation approach for the pelvic CT examinations, and has the potential to be used for improving the segmentation performance of other types of fractures.

Title: Improving Membership Inference in ASR Model Auditing with Perturbed Loss Features

• Authors: Francisco Teixeira, Karla Pizzi, Raphael Olivier, Alberto Abad, Bhiksha Raj, Isabel Trancoso
• Subjects: cs.LG, cs.CR, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2405.01207
• Pdf URL: https://arxiv.org/pdf/2405.01207
• Copy Paste: [[2405.01207]] Improving Membership Inference in ASR Model Auditing with Perturbed Loss Features(https://arxiv.org/abs/2405.01207)
• Keywords: privacy, membership infer
• Abstract: Membership Inference (MI) poses a substantial privacy threat to the training data of Automatic Speech Recognition (ASR) systems, while also offering an opportunity to audit these models with regard to user data. This paper explores the effectiveness of loss-based features in combination with Gaussian and adversarial perturbations to perform MI in ASR models. To the best of our knowledge, this approach has not yet been investigated. We compare our proposed features with commonly used error-based features and find that the proposed features greatly enhance performance for sample-level MI. For speaker-level MI, these features improve results, though by a smaller margin, as error-based features already obtained a high performance for this task. Our findings emphasise the importance of considering different feature sets and levels of access to target models for effective MI in ASR systems, providing valuable insights for auditing such models.

Title: CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentation

• Authors: Chenying Liu, Conrad Albrecht, Yi Wang, Xiao Xiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01217
• Pdf URL: https://arxiv.org/pdf/2405.01217
• Copy Paste: [[2405.01217]] CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentation(https://arxiv.org/abs/2405.01217)
• Keywords: segmentation
• Abstract: We study the potential of noisy labels y to pretrain semantic segmentation models in a multi-modal learning framework for geospatial applications. Specifically, we propose a novel Cross-modal Sample Selection method (CromSS) that utilizes the class distributions P^{(d)}(x,c) over pixels x and classes c modelled by multiple sensors/modalities d of a given geospatial scene. Consistency of predictions across sensors $d$ is jointly informed by the entropy of P^{(d)}(x,c). Noisy label sampling we determine by the confidence of each sensor d in the noisy class label, P^{(d)}(x,c=y(x)). To verify the performance of our approach, we conduct experiments with Sentinel-1 (radar) and Sentinel-2 (optical) satellite imagery from the globally-sampled SSL4EO-S12 dataset. We pair those scenes with 9-class noisy labels sourced from the Google Dynamic World project for pretraining. Transfer learning evaluations (downstream task) on the DFC2020 dataset confirm the effectiveness of the proposed method for remote sensing image segmentation.

Title: RaffeSDG: Random Frequency Filtering enabled Single-source Domain Generalization for Medical Image Segmentation

• Authors: Heng Li, Haojin Li, Jianyu Chen, Zhongxi Qiu, Huazhu Fu, Lidai Wang, Yan Hu, Jiang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01228
• Pdf URL: https://arxiv.org/pdf/2405.01228
• Copy Paste: [[2405.01228]] RaffeSDG: Random Frequency Filtering enabled Single-source Domain Generalization for Medical Image Segmentation(https://arxiv.org/abs/2405.01228)
• Keywords: robust, segmentation
• Abstract: Deep learning models often encounter challenges in making accurate inferences when there are domain shifts between the source and target data. This issue is particularly pronounced in clinical settings due to the scarcity of annotated data resulting from the professional and private nature of medical data. Despite the existence of decent solutions, many of them are hindered in clinical settings due to limitations in data collection and computational complexity. To tackle domain shifts in data-scarce medical scenarios, we propose a Random frequency filtering enabled Single-source Domain Generalization algorithm (RaffeSDG), which promises robust out-of-domain inference with segmentation models trained on a single-source domain. A filter-based data augmentation strategy is first proposed to promote domain variability within a single-source domain by introducing variations in frequency space and blending homologous samples. Then Gaussian filter-based structural saliency is also leveraged to learn robust representations across augmented samples, further facilitating the training of generalizable segmentation models. To validate the effectiveness of RaffeSDG, we conducted extensive experiments involving out-of-domain inference on segmentation tasks for three human tissues imaged by four diverse modalities. Through thorough investigations and comparisons, compelling evidence was observed in these experiments, demonstrating the potential and generalizability of RaffeSDG. The code is available at https://github.com/liamheng/Non-IID_Medical_Image_Segmentation.

Title: Boosting Jailbreak Attack with Momentum

• Authors: Yihao Zhang, Zeming Wei
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR, math.OC
• Abstract URL: https://arxiv.org/abs/2405.01229
• Pdf URL: https://arxiv.org/pdf/2405.01229
• Copy Paste: [[2405.01229]] Boosting Jailbreak Attack with Momentum(https://arxiv.org/abs/2405.01229)
• Keywords: attack, large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable success across diverse tasks, yet they remain vulnerable to adversarial attacks, notably the well-documented \textit{jailbreak} attack. Recently, the Greedy Coordinate Gradient (GCG) attack has demonstrated efficacy in exploiting this vulnerability by optimizing adversarial prompts through a combination of gradient heuristics and greedy search. However, the efficiency of this attack has become a bottleneck in the attacking process. To mitigate this limitation, in this paper we rethink the generation of adversarial prompts through an optimization lens, aiming to stabilize the optimization process and harness more heuristic insights from previous iterations. Specifically, we introduce the \textbf{M}omentum \textbf{A}ccelerated G\textbf{C}G (\textbf{MAC}) attack, which incorporates a momentum term into the gradient heuristic. Experimental results showcase the notable enhancement achieved by MAP in gradient-based attacks on aligned language models. Our code is available at https://github.com/weizeming/momentum-attack-llm.

Title: Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices

• Authors: Jamil Zaghir, Marco Naguib, Mina Bjelogrlic, Aurélie Névéol, Xavier Tannier, Christian Lovis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01249
• Pdf URL: https://arxiv.org/pdf/2405.01249
• Copy Paste: [[2405.01249]] Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices(https://arxiv.org/abs/2405.01249)
• Keywords: large language model
• Abstract: Prompt engineering is crucial for harnessing the potential of large language models (LLMs), especially in the medical domain where specialized terminology and phrasing is used. However, the efficacy of prompt engineering in the medical domain remains to be explored. In this work, 114 recent studies (2022-2024) applying prompt engineering in medicine, covering prompt learning (PL), prompt tuning (PT), and prompt design (PD) are reviewed. PD is the most prevalent (78 articles). In 12 papers, PD, PL, and PT terms were used interchangeably. ChatGPT is the most commonly used LLM, with seven papers using it for processing sensitive clinical data. Chain-of-Thought emerges as the most common prompt engineering technique. While PL and PT articles typically provide a baseline for evaluating prompt-based approaches, 64% of PD studies lack non-prompt-related baselines. We provide tables and figures summarizing existing work, and reporting recommendations to guide future research contributions.

Title: Towards Consistent Object Detection via LiDAR-Camera Synergy

• Authors: Kai Luo, Hao Wu, Kefu Yi, Kailun Yang, Wei Hao, Rongdong Hu
• Subjects: cs.CV, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2405.01258
• Pdf URL: https://arxiv.org/pdf/2405.01258
• Copy Paste: [[2405.01258]] Towards Consistent Object Detection via LiDAR-Camera Synergy(https://arxiv.org/abs/2405.01258)
• Keywords: robust
• Abstract: As human-machine interaction continues to evolve, the capacity for environmental perception is becoming increasingly crucial. Integrating the two most common types of sensory data, images, and point clouds, can enhance detection accuracy. However, currently, no model exists that can simultaneously detect an object's position in both point clouds and images and ascertain their corresponding relationship. This information is invaluable for human-machine interactions, offering new possibilities for their enhancement. In light of this, this paper introduces an end-to-end Consistency Object Detection (COD) algorithm framework that requires only a single forward inference to simultaneously obtain an object's position in both point clouds and images and establish their correlation. Furthermore, to assess the accuracy of the object correlation between point clouds and images, this paper proposes a new evaluation metric, Consistency Precision (CP). To verify the effectiveness of the proposed framework, an extensive set of experiments has been conducted on the KITTI and DAIR-V2X datasets. The study also explored how the proposed consistency detection method performs on images when the calibration parameters between images and point clouds are disturbed, compared to existing post-processing methods. The experimental results demonstrate that the proposed method exhibits excellent detection performance and robustness, achieving end-to-end consistency detection. The source code will be made publicly available at https://github.com/xifen523/COD.

Title: Causal Influence in Federated Edge Inference

• Authors: Mert Kayaalp, Yunus Inan, Visa Koivunen, Ali H. Sayed
• Subjects: cs.LG, cs.MA, eess.SP, eess.SY
• Abstract URL: https://arxiv.org/abs/2405.01260
• Pdf URL: https://arxiv.org/pdf/2405.01260
• Copy Paste: [[2405.01260]] Causal Influence in Federated Edge Inference(https://arxiv.org/abs/2405.01260)
• Keywords: attack, federate
• Abstract: In this paper, we consider a setting where heterogeneous agents with connectivity are performing inference using unlabeled streaming data. Observed data are only partially informative about the target variable of interest. In order to overcome the uncertainty, agents cooperate with each other by exchanging their local inferences with and through a fusion center. To evaluate how each agent influences the overall decision, we adopt a causal framework in order to distinguish the actual influence of agents from mere correlations within the decision-making process. Various scenarios reflecting different agent participation patterns and fusion center policies are investigated. We derive expressions to quantify the causal impact of each agent on the joint decision, which could be beneficial for anticipating and addressing atypical scenarios, such as adversarial attacks or system malfunctions. We validate our theoretical results with numerical simulations and a real-world application of multi-camera crowd counting.

Title: Towards Inclusive Face Recognition Through Synthetic Ethnicity Alteration

• Authors: Praveen Kumar Chandaliya, Kiran Raja, Raghavendra Ramachandra, Zahid Akhtar, Christoph Busch
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01273
• Pdf URL: https://arxiv.org/pdf/2405.01273
• Copy Paste: [[2405.01273]] Towards Inclusive Face Recognition Through Synthetic Ethnicity Alteration(https://arxiv.org/abs/2405.01273)
• Keywords: privacy, generative
• Abstract: Numerous studies have shown that existing Face Recognition Systems (FRS), including commercial ones, often exhibit biases toward certain ethnicities due to under-represented data. In this work, we explore ethnicity alteration and skin tone modification using synthetic face image generation methods to increase the diversity of datasets. We conduct a detailed analysis by first constructing a balanced face image dataset representing three ethnicities: Asian, Black, and Indian. We then make use of existing Generative Adversarial Network-based (GAN) image-to-image translation and manifold learning models to alter the ethnicity from one to another. A systematic analysis is further conducted to assess the suitability of such datasets for FRS by studying the realistic skin-tone representation using Individual Typology Angle (ITA). Further, we also analyze the quality characteristics using existing Face image quality assessment (FIQA) approaches. We then provide a holistic FRS performance analysis using four different systems. Our findings pave the way for future research works in (i) developing both specific ethnicity and general (any to any) ethnicity alteration models, (ii) expanding such approaches to create databases with diverse skin tones, (iii) creating datasets representing various ethnicities which further can help in mitigating bias while addressing privacy concerns.

Title: Reinforcement Learning for Edit-Based Non-Autoregressive Neural Machine Translation

• Authors: Hao Wang, Tetsuro Morimura, Ukyo Honda, Daisuke Kawahara
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.01280
• Pdf URL: https://arxiv.org/pdf/2405.01280
• Copy Paste: [[2405.01280]] Reinforcement Learning for Edit-Based Non-Autoregressive Neural Machine Translation(https://arxiv.org/abs/2405.01280)
• Keywords: transformer
• Abstract: Non-autoregressive (NAR) language models are known for their low latency in neural machine translation (NMT). However, a performance gap exists between NAR and autoregressive models due to the large decoding space and difficulty in capturing dependency between target words accurately. Compounding this, preparing appropriate training data for NAR models is a non-trivial task, often exacerbating exposure bias. To address these challenges, we apply reinforcement learning (RL) to Levenshtein Transformer, a representative edit-based NAR model, demonstrating that RL with self-generated data can enhance the performance of edit-based NAR models. We explore two RL approaches: stepwise reward maximization and episodic reward maximization. We discuss the respective pros and cons of these two approaches and empirically verify them. Moreover, we experimentally investigate the impact of temperature setting on performance, confirming the importance of proper temperature setting for NAR models' training.

Title: Measuring the Exploitation of Weaknesses in the Wild

• Authors: Peter Mell, Irena Bojanova, Carlos Galhardo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01289
• Pdf URL: https://arxiv.org/pdf/2405.01289
• Copy Paste: [[2405.01289]] Measuring the Exploitation of Weaknesses in the Wild(https://arxiv.org/abs/2405.01289)
• Keywords: security, attack
• Abstract: Identifying the software weaknesses exploited by attacks supports efforts to reduce developer introduction of vulnerabilities and to guide security code review efforts. A weakness is a bug or fault type that can be exploited through an operation that results in a security-relevant error. Ideally, the security community would measure the prevalence of the software weaknesses used in actual exploitation. This work advances that goal by introducing a simple metric that utilizes public data feeds to determine the probability of a weakness being exploited in the wild for any 30-day window. The metric is evaluated on a set of 130 weaknesses that were commonly found in vulnerabilities between April 2021 and March 2024. Our analysis reveals that 92 % of the weaknesses are not being constantly exploited.

Title: The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation

• Authors: Maja Pavlovic, Massimo Poesio
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01299
• Pdf URL: https://arxiv.org/pdf/2405.01299
• Copy Paste: [[2405.01299]] The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation(https://arxiv.org/abs/2405.01299)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have emerged as powerful support tools across various natural language tasks and a range of application domains. Recent studies focus on exploring their capabilities for data annotation. This paper provides a comparative overview of twelve studies investigating the potential of LLMs in labelling data. While the models demonstrate promising cost and time-saving benefits, there exist considerable limitations, such as representativeness, bias, sensitivity to prompt variations and English language preference. Leveraging insights from these studies, our empirical analysis further examines the alignment between human and GPT-generated opinion distributions across four subjective datasets. In contrast to the studies examining representation, our methodology directly obtains the opinion distribution from GPT. Our analysis thereby supports the minority of studies that are considering diverse perspectives when evaluating data annotation tasks and highlights the need for further research in this direction.

Title: Constrained Reinforcement Learning Under Model Mismatch

• Authors: Zhongchang Sun, Sihong He, Fei Miao, Shaofeng Zou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01327
• Pdf URL: https://arxiv.org/pdf/2405.01327
• Copy Paste: [[2405.01327]] Constrained Reinforcement Learning Under Model Mismatch(https://arxiv.org/abs/2405.01327)
• Keywords: robust
• Abstract: Existing studies on constrained reinforcement learning (RL) may obtain a well-performing policy in the training environment. However, when deployed in a real environment, it may easily violate constraints that were originally satisfied during training because there might be model mismatch between the training and real environments. To address the above challenge, we formulate the problem as constrained RL under model uncertainty, where the goal is to learn a good policy that optimizes the reward and at the same time satisfy the constraint under model mismatch. We develop a Robust Constrained Policy Optimization (RCPO) algorithm, which is the first algorithm that applies to large/continuous state space and has theoretical guarantees on worst-case reward improvement and constraint violation at each iteration during the training. We demonstrate the effectiveness of our algorithm on a set of RL tasks with constraints.

Title: Decentralization of Ethereum's Builder Market

• Authors: Sen Yang, Kartik Nayak, Fan Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01329
• Pdf URL: https://arxiv.org/pdf/2405.01329
• Copy Paste: [[2405.01329]] Decentralization of Ethereum's Builder Market(https://arxiv.org/abs/2405.01329)
• Keywords: security, protect
• Abstract: Blockchains protect an ecosystem worth more than $500bn with their strong security properties derived from the principle of decentralization. Is today's blockchain really decentralized? In this paper, we empirically studied one of the {\em least decentralized} parts of Ethereum -- the most used blockchain system in practice -- and shed light on the decentralization issue from a new perspective. To avoid centralization caused by Maximal Extractable Value (MEV), Ethereum adopts a novel mechanism that produces blocks through a {\em builder market}. After two years in operation, however, the builder market has evolved to a highly centralized one with three builders producing more than 90% of blocks. {\em Why does the builder market centralize, given that it is permissionless and anyone can join?} Moreover, {\em what are the security implications of a centralized builder market to MEV-Boost auctions?} Through a rigorous empirical study of the builder market's core mechanism, MEV-Boost auctions, we answered these two questions using a large-scale auction dataset we curated since 2022. Unlike previous works that focus on {\em who} wins the auctions, we focus on {\em why} they win, to shed light on the {openness, competitiveness, and efficiency} of MEV-Boost auctions. Our findings also help identify directions for improving the decentralization of builder markets.

Title: Multi-view Action Recognition via Directed Gromov-Wasserstein Discrepancy

• Authors: Hoang-Quan Nguyen, Thanh-Dat Truong, Khoa Luu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01337
• Pdf URL: https://arxiv.org/pdf/2405.01337
• Copy Paste: [[2405.01337]] Multi-view Action Recognition via Directed Gromov-Wasserstein Discrepancy(https://arxiv.org/abs/2405.01337)
• Keywords: transformer
• Abstract: Action recognition has become one of the popular research topics in computer vision. There are various methods based on Convolutional Networks and self-attention mechanisms as Transformers to solve both spatial and temporal dimensions problems of action recognition tasks that achieve competitive performances. However, these methods lack a guarantee of the correctness of the action subject that the models give attention to, i.e., how to ensure an action recognition model focuses on the proper action subject to make a reasonable action prediction. In this paper, we propose a multi-view attention consistency method that computes the similarity between two attentions from two different views of the action videos using Directed Gromov-Wasserstein Discrepancy. Furthermore, our approach applies the idea of Neural Radiance Field to implicitly render the features from novel views when training on single-view datasets. Therefore, the contributions in this work are three-fold. Firstly, we introduce the multi-view attention consistency to solve the problem of reasonable prediction in action recognition. Secondly, we define a new metric for multi-view consistent attention using Directed Gromov-Wasserstein Discrepancy. Thirdly, we built an action recognition model based on Video Transformers and Neural Radiance Fields. Compared to the recent action recognition methods, the proposed approach achieves state-of-the-art results on three large-scale datasets, i.e., Jester, Something-Something V2, and Kinetics-400.

Title: The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights

• Authors: Wenhao Zhu, Shujian Huang, Fei Yuan, Cheng Chen, Jiajun Chen, Alexandra Birch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.01345
• Pdf URL: https://arxiv.org/pdf/2405.01345
• Copy Paste: [[2405.01345]] The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights(https://arxiv.org/abs/2405.01345)
• Keywords: large language model
• Abstract: Bridging the significant gap between large language model's English and non-English performance presents a great challenge. While some previous studies attempt to mitigate this gap with translated training data, the recently proposed question alignment approach leverages the model's English expertise to improve multilingual performance with minimum usage of expensive, error-prone translation. In this paper, we explore how broadly this method can be applied by examining its effects in reasoning with executable code and reasoning with common sense. We also explore how to apply this approach efficiently to extremely large language models using proxy-tuning. Experiment results on multilingual reasoning benchmarks mGSM, mSVAMP and xCSQA demonstrate that the question alignment approach can be used to boost multilingual performance across diverse reasoning scenarios, model families, and sizes. For instance, when applied to the LLaMA2 models, our method brings an average accuracy improvements of 12.2% on mGSM even with the 70B model. To understand the mechanism of its success, we analyze representation space, chain-of-thought and translation data scales, which reveals how question translation training strengthens language alignment within LLMs and shapes their working patterns.

Title: Position Paper: Beyond Robustness Against Single Attack Types

• Authors: Sihui Dai, Chong Xiang, Tong Wu, Prateek Mittal
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01349
• Pdf URL: https://arxiv.org/pdf/2405.01349
• Copy Paste: [[2405.01349]] Position Paper: Beyond Robustness Against Single Attack Types(https://arxiv.org/abs/2405.01349)
• Keywords: defense, attack, robust
• Abstract: Current research on defending against adversarial examples focuses primarily on achieving robustness against a single attack type such as $\ell_2$ or $\ell_{\infty}$-bounded attacks. However, the space of possible perturbations is much larger and currently cannot be modeled by a single attack type. The discrepancy between the focus of current defenses and the space of attacks of interest calls to question the practicality of existing defenses and the reliability of their evaluation. In this position paper, we argue that the research community should look beyond single attack robustness, and we draw attention to three potential directions involving robustness against multiple attacks: simultaneous multiattack robustness, unforeseen attack robustness, and a newly defined problem setting which we call continual adaptive robustness. We provide a unified framework which rigorously defines these problem settings, synthesize existing research in these fields, and outline open directions. We hope that our position paper inspires more research in simultaneous multiattack, unforeseen attack, and continual adaptive robustness.

Title: Community-Invariant Graph Contrastive Learning

• Authors: Shiyin Tan, Dongyuan Li, Renhe Jiang, Ying Zhang, Manabu Okumura
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2405.01350
• Pdf URL: https://arxiv.org/pdf/2405.01350
• Copy Paste: [[2405.01350]] Community-Invariant Graph Contrastive Learning(https://arxiv.org/abs/2405.01350)
• Keywords: robust
• Abstract: Graph augmentation has received great attention in recent years for graph contrastive learning (GCL) to learn well-generalized node/graph representations. However, mainstream GCL methods often favor randomly disrupting graphs for augmentation, which shows limited generalization and inevitably leads to the corruption of high-level graph information, i.e., the graph community. Moreover, current knowledge-based graph augmentation methods can only focus on either topology or node features, causing the model to lack robustness against various types of noise. To address these limitations, this research investigated the role of the graph community in graph augmentation and figured out its crucial advantage for learnable graph augmentation. Based on our observations, we propose a community-invariant GCL framework to maintain graph community structure during learnable graph augmentation. By maximizing the spectral changes, this framework unifies the constraints of both topology and feature augmentation, enhancing the model's robustness. Empirical evidence on 21 benchmark datasets demonstrates the exclusive merits of our framework. Code is released on Github (https://github.com/ShiyinTan/CI-GCL.git).

Title: GAIA: A General AI Assistant for Intelligent Accelerator Operations

• Authors: Frank Mayet
• Subjects: cs.CL, physics.acc-ph
• Abstract URL: https://arxiv.org/abs/2405.01359
• Pdf URL: https://arxiv.org/pdf/2405.01359
• Copy Paste: [[2405.01359]] GAIA: A General AI Assistant for Intelligent Accelerator Operations(https://arxiv.org/abs/2405.01359)
• Keywords: large language model
• Abstract: Large-scale machines like particle accelerators are usually run by a team of experienced operators. In case of a particle accelerator, these operators possess suitable background knowledge on both accelerator physics and the technology comprising the machine. Due to the complexity of the machine, particular subsystems of the machine are taken care of by experts, who the operators can turn to. In this work the reasoning and action (ReAct) prompting paradigm is used to couple an open-weights large language model (LLM) with a high-level machine control system framework and other tools, e.g. the electronic logbook or machine design documentation. By doing so, a multi-expert retrieval augmented generation (RAG) system is implemented, which assists operators in knowledge retrieval tasks, interacts with the machine directly if needed, or writes high level control system scripts. This consolidation of expert knowledge and machine interaction can simplify and speed up machine operation tasks for both new and experienced human operators.

Title: Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving

• Authors: Xin Quan, Marco Valentino, Louise A. Dennis, André Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.01379
• Pdf URL: https://arxiv.org/pdf/2405.01379
• Copy Paste: [[2405.01379]] Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving(https://arxiv.org/abs/2405.01379)
• Keywords: large language model
• Abstract: Natural language explanations have become a proxy for evaluating explainable and multi-step Natural Language Inference (NLI) models. However, assessing the validity of explanations for NLI is challenging as it typically involves the crowd-sourcing of apposite datasets, a process that is time-consuming and prone to logical errors. To address existing limitations, this paper investigates the verification and refinement of natural language explanations through the integration of Large Language Models (LLMs) and Theorem Provers (TPs). Specifically, we present a neuro-symbolic framework, named Explanation-Refiner, that augments a TP with LLMs to generate and formalise explanatory sentences and suggest potential inference strategies for NLI. In turn, the TP is employed to provide formal guarantees on the logical validity of the explanations and to generate feedback for subsequent improvements. We demonstrate how Explanation-Refiner can be jointly used to evaluate explanatory reasoning, autoformalisation, and error correction mechanisms of state-of-the-art LLMs as well as to automatically enhance the quality of human-annotated explanations of variable complexity in different domains.

Title: Invariant Risk Minimization Is A Total Variation Model

• Authors: Zhao-Rong Lai, Wei-Wen Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01389
• Pdf URL: https://arxiv.org/pdf/2405.01389
• Copy Paste: [[2405.01389]] Invariant Risk Minimization Is A Total Variation Model(https://arxiv.org/abs/2405.01389)
• Keywords: robust
• Abstract: Invariant risk minimization (IRM) is an arising approach to generalize invariant features to different environments in machine learning. While most related works focus on new IRM settings or new application scenarios, the mathematical essence of IRM remains to be properly explained. We verify that IRM is essentially a total variation based on $L^2$ norm (TV-$\ell_2$) of the learning risk with respect to the classifier variable. Moreover, we propose a novel IRM framework based on the TV-$\ell_1$ model. It not only expands the classes of functions that can be used as the learning risk, but also has robust performance in denoising and invariant feature preservation based on the coarea formula. We also illustrate some requirements for IRM-TV-$\ell_1$ to achieve out-of-distribution generalization. Experimental results show that the proposed framework achieves competitive performance in several benchmark machine learning scenarios.

Title: IDPFilter: Mitigating Interdependent Privacy Issues in Third-Party Apps

• Authors: Shuaishuai Liu, Gergely Biczók
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01411
• Pdf URL: https://arxiv.org/pdf/2405.01411
• Copy Paste: [[2405.01411]] IDPFilter: Mitigating Interdependent Privacy Issues in Third-Party Apps(https://arxiv.org/abs/2405.01411)
• Keywords: privacy, protect
• Abstract: Third-party applications have become an essential part of today's online ecosystem, enhancing the functionality of popular platforms. However, the intensive data exchange underlying their proliferation has increased concerns about interdependent privacy (IDP). This paper provides a comprehensive investigation into the previously underinvestigated IDP issues of third-party apps. Specifically, first, we analyze the permission structure of multiple app platforms, identifying permissions that have the potential to cause interdependent privacy issues by enabling a user to share someone else's personal data with an app. Second, we collect datasets and characterize the extent to which existing apps request these permissions, revealing the relationship between characteristics such as the respective app platform, the app's type, and the number of interdependent privacy-related permissions it requests. Third, we analyze the various reasons IDP is neglected by both data protection regulations and app platforms and then devise principles that should be followed when designing a mitigation solution. Finally, based on these principles and satisfying clearly defined objectives, we propose IDPFilter, a platform-agnostic API that enables application providers to minimize collateral information collection by filtering out data collected from their users but implicating others as data subjects. We implement a proof-of-concept prototype, IDPTextFilter, that implements the filtering logic on textual data, and provide its initial performance evaluation with regard to privacy, accuracy, and efficiency.

Title: Applying Transparent Shaping for Zero Trust Architecture Implementation in AWS: A Case Study

• Authors: Wenjia Wang, Seyed Masoud Sadjadi, Naphtali Rishe
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2405.01412
• Pdf URL: https://arxiv.org/pdf/2405.01412
• Copy Paste: [[2405.01412]] Applying Transparent Shaping for Zero Trust Architecture Implementation in AWS: A Case Study(https://arxiv.org/abs/2405.01412)
• Keywords: security
• Abstract: This study introduces a methodology integrating Zero Trust Architecture (ZTA) principles and Transparent Shaping into an AWS-hosted Online File Manager (OFM) application, enhancing security without substantial code modifications. We evaluate our approach with the Mozilla Observatory, highlighting significant security improvements and outlining a promising direction for applying Transparent Shaping and ZTA in cloud environments.

Title: MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors

• Authors: Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, Min Chen
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01413
• Pdf URL: https://arxiv.org/pdf/2405.01413
• Copy Paste: [[2405.01413]] MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors(https://arxiv.org/abs/2405.01413)
• Keywords: large language model
• Abstract: Large 2D vision-language models (2D-LLMs) have gained significant attention by bridging Large Language Models (LLMs) with images using a simple projector. Inspired by their success, large 3D point cloud-language models (3D-LLMs) also integrate point clouds into LLMs. However, directly aligning point clouds with LLM requires expensive training costs, typically in hundreds of GPU-hours on A100, which hinders the development of 3D-LLMs. In this paper, we introduce MiniGPT-3D, an efficient and powerful 3D-LLM that achieves multiple SOTA results while training for only 27 hours on one RTX 3090. Specifically, we propose to align 3D point clouds with LLMs using 2D priors from 2D-LLMs, which can leverage the similarity between 2D and 3D visual information. We introduce a novel four-stage training strategy for modality alignment in a cascaded way, and a mixture of query experts module to adaptively aggregate features with high efficiency. Moreover, we utilize parameter-efficient fine-tuning methods LoRA and Norm fine-tuning, resulting in only 47.8M learnable parameters, which is up to 260x fewer than existing methods. Extensive experiments show that MiniGPT-3D achieves SOTA on 3D object classification and captioning tasks, with significantly cheaper training costs. Notably, MiniGPT-3D gains an 8.12 increase on GPT-4 evaluation score for the challenging object captioning task compared to ShapeLLM-13B, while the latter costs 160 total GPU-hours on 8 A800. We are the first to explore the efficient 3D-LLM, offering new insights to the community. Code and weights are available at https://github.com/TangYuan96/MiniGPT-3D.

Title: StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation

• Authors: Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, Qibin Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01434
• Pdf URL: https://arxiv.org/pdf/2405.01434
• Copy Paste: [[2405.01434]] StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation(https://arxiv.org/abs/2405.01434)
• Keywords: diffusion, generative
• Abstract: For recent diffusion-based generative models, maintaining consistent content across a series of generated images, especially those containing subjects and complex details, presents a significant challenge. In this paper, we propose a new way of self-attention calculation, termed Consistent Self-Attention, that significantly boosts the consistency between the generated images and augments prevalent pretrained diffusion-based text-to-image models in a zero-shot manner. To extend our method to long-range video generation, we further introduce a novel semantic space temporal motion prediction module, named Semantic Motion Predictor. It is trained to estimate the motion conditions between two provided images in the semantic spaces. This module converts the generated sequence of images into videos with smooth transitions and consistent subjects that are significantly more stable than the modules based on latent spaces only, especially in the context of long video generation. By merging these two novel components, our framework, referred to as StoryDiffusion, can describe a text-based story with consistent images or videos encompassing a rich variety of contents. The proposed StoryDiffusion encompasses pioneering explorations in visual story generation with the presentation of images and videos, which we hope could inspire more research from the aspect of architectural modifications. Our code is made publicly available at https://github.com/HVision-NKU/StoryDiffusion.

Title: An Exploratory Case Study on Data Breach Journalism

• Authors: Jukka Ruohonen, Kalle Hjerppe, Maximilian von Zastrow
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2405.01446
• Pdf URL: https://arxiv.org/pdf/2405.01446
• Copy Paste: [[2405.01446]] An Exploratory Case Study on Data Breach Journalism(https://arxiv.org/abs/2405.01446)
• Keywords: fair
• Abstract: This paper explores the novel topic of data breach journalism and data breach news through the case of databreaches.net, a news outlet dedicated to data breaches and related cyber crime. Motivated by the issues in traditional crime news and crime journalism, the case is explored by the means of text mining. According to the results, the outlet has kept a steady publishing pace, mainly focusing on plain and short reporting but with generally high-quality source material for the news articles. Despite these characteristics, the news articles exhibit fairly strong sentiments, which is partially expected due to the presence of emotionally laden crime and the long history of sensationalism in crime news. The news site has also covered the full scope of data breaches, although many of these are fairly traditional, exposing personal identifiers and financial details of the victims. Also hospitals and the healthcare sector stand out. With these results, the paper advances the study of data breaches by considering these from the perspective of media and journalism.

Title: Unconditionally Safe Light Client

• Authors: Niusha Moshrefi, Peiyao Sheng, Soubhik Deb, Sreeram Kannan, Pramod Viswanath
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01459
• Pdf URL: https://arxiv.org/pdf/2405.01459
• Copy Paste: [[2405.01459]] Unconditionally Safe Light Client(https://arxiv.org/abs/2405.01459)
• Keywords: security, protect, robust
• Abstract: Blockchain applications often rely on lightweight clients to access and verify on-chain data efficiently without the need to run a resource-intensive full node. These light clients must maintain robust security to protect the blockchain's integrity for users of applications built upon it, achieving this with minimal resources and without significant latency. Moreover, different applications have varying security needs. This work focuses on addressing these two key requirements in the context of Proof-of-Stake (PoS) blockchains and identifying the fundamental cost-latency trade-offs to achieve tailored, optimal security for each light client. The key security guarantee of PoS blockchains is economic (implied by the "stake"). In this paper we formalize this cryptoeconomic security to light clients, ensuring that the cost of corrupting the data provided to light clients must outweigh the potential profit, thereby economically deterring malicious actors. We further introduce "insured" cryptoeconomic security to light clients, providing unconditional protection via the attribution of adversarial actions and the consequent slashing of stakes. The divisible and fungible nature of stake facilitates programmable security, allowing for customization of the security level and insurance amount according to the specific needs of different applications. We implemented the protocols in less than 1000 lines of Solidity and TypeScript code and evaluated their gas cost, latency, and the computational overhead. For example, for a transaction with value of \$32k, the light client can choose between zero cost with a latency of 5 hours or instant confirmation with an insurance cost of \$7.45. Thus, the client can select the optimal point on the latency-cost trade-off spectrum that best aligns with its needs. Light clients require negligible storage and face minimal computational costs,...

Title: Purify Unlearnable Examples via Rate-Constrained Variational Autoencoders

• Authors: Yi Yu, Yufei Wang, Song Xia, Wenhan Yang, Shijian Lu, Yap-Peng Tan, Alex C. Kot
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01460
• Pdf URL: https://arxiv.org/pdf/2405.01460
• Copy Paste: [[2405.01460]] Purify Unlearnable Examples via Rate-Constrained Variational Autoencoders(https://arxiv.org/abs/2405.01460)
• Keywords: defense, attack, robust
• Abstract: Unlearnable examples (UEs) seek to maximize testing error by making subtle modifications to training examples that are correctly labeled. Defenses against these poisoning attacks can be categorized based on whether specific interventions are adopted during training. The first approach is training-time defense, such as adversarial training, which can mitigate poisoning effects but is computationally intensive. The other approach is pre-training purification, e.g., image short squeezing, which consists of several simple compressions but often encounters challenges in dealing with various UEs. Our work provides a novel disentanglement mechanism to build an efficient pre-training purification method. Firstly, we uncover rate-constrained variational autoencoders (VAEs), demonstrating a clear tendency to suppress the perturbations in UEs. We subsequently conduct a theoretical analysis for this phenomenon. Building upon these insights, we introduce a disentangle variational autoencoder (D-VAE), capable of disentangling the perturbations with learnable class-wise embeddings. Based on this network, a two-stage purification approach is naturally developed. The first stage focuses on roughly eliminating perturbations, while the second stage produces refined, poison-free results, ensuring effectiveness and robustness across various scenarios. Extensive experiments demonstrate the remarkable performance of our method across CIFAR-10, CIFAR-100, and a 100-class ImageNet-subset. Code is available at https://github.com/yuyi-sd/D-VAE.

Title: SATO: Stable Text-to-Motion Framework

• Authors: Wenshuo Chen, Hongru Xiao, Erhang Zhang, Lijie Hu, Lei Wang, Mengyuan Liu, Chen Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01461
• Pdf URL: https://arxiv.org/pdf/2405.01461
• Copy Paste: [[2405.01461]] SATO: Stable Text-to-Motion Framework(https://arxiv.org/abs/2405.01461)
• Keywords: robust
• Abstract: Is the Text to Motion model robust? Recent advancements in Text to Motion models primarily stem from more accurate predictions of specific actions. However, the text modality typically relies solely on pre-trained Contrastive Language-Image Pretraining (CLIP) models. Our research has uncovered a significant issue with the text-to-motion model: its predictions often exhibit inconsistent outputs, resulting in vastly different or even incorrect poses when presented with semantically similar or identical text inputs. In this paper, we undertake an analysis to elucidate the underlying causes of this instability, establishing a clear link between the unpredictability of model outputs and the erratic attention patterns of the text encoder module. Consequently, we introduce a formal framework aimed at addressing this issue, which we term the Stable Text-to-Motion Framework (SATO). SATO consists of three modules, each dedicated to stable attention, stable prediction, and maintaining a balance between accuracy and robustness trade-off. We present a methodology for constructing an SATO that satisfies the stability of attention and prediction. To verify the stability of the model, we introduced a new textual synonym perturbation dataset based on HumanML3D and KIT-ML. Results show that SATO is significantly more stable against synonyms and other slight perturbations while keeping its high accuracy performance.

Title: Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning

• Authors: Théo Moutakanni, Piotr Bojanowski, Guillaume Chassagnon, Céline Hudelot, Armand Joulin, Yann LeCun, Matthew Muckley, Maxime Oquab, Marie-Pierre Revel, Maria Vakalopoulou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01469
• Pdf URL: https://arxiv.org/pdf/2405.01469
• Copy Paste: [[2405.01469]] Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning(https://arxiv.org/abs/2405.01469)
• Keywords: robust, segmentation
• Abstract: AI Foundation models are gaining traction in various applications, including medical fields like radiology. However, medical foundation models are often tested on limited tasks, leaving their generalisability and biases unexplored. We present RayDINO, a large visual encoder trained by self-supervision on 873k chest X-rays. We compare RayDINO to previous state-of-the-art models across nine radiology tasks, from classification and dense segmentation to text generation, and provide an in depth analysis of population, age and sex biases of our model. Our findings suggest that self-supervision allows patient-centric AI proving useful in clinical workflows and interpreting X-rays holistically. With RayDINO and small task-specific adapters, we reach state-of-the-art results and improve generalization to unseen populations while mitigating bias, illustrating the true promise of foundation models: versatility and robustness.

Title: NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment

• Authors: Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, Oleksii Kuchaiev
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01481
• Pdf URL: https://arxiv.org/pdf/2405.01481
• Copy Paste: [[2405.01481]] NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment(https://arxiv.org/abs/2405.01481)
• Keywords: large language model
• Abstract: Aligning Large Language Models (LLMs) with human values and preferences is essential for making them helpful and safe. However, building efficient tools to perform alignment can be challenging, especially for the largest and most competent LLMs which often contain tens or hundreds of billions of parameters. We create NeMo-Aligner, a toolkit for model alignment that can efficiently scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly optimized and scalable implementations for major paradigms of model alignment such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally, our toolkit supports running most of the alignment techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for extensibility, allowing support for other alignment techniques with minimal effort. It is open-sourced with Apache 2.0 License and we invite community contributions at https://github.com/NVIDIA/NeMo-Aligner

Title: Digital Twin Generators for Disease Modeling

• Authors: Nameyeh Alam, Jake Basilico, Daniele Bertolini, Satish Casie Chetty, Heather D'Angelo, Ryan Douglas, Charles K. Fisher, Franklin Fuller, Melissa Gomes, Rishabh Gupta, Alex Lang, Anton Loukianov, Rachel Mak-McCully, Cary Murray, Hanalei Pham, Susanna Qiao, Elena Ryapolova-Webb, Aaron Smith, Dimitri Theoharatos, Anil Tolwani, Eric W. Tramel, Anna Vidovszky, Judy Viduya, Jonathan R. Walsh
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2405.01488
• Pdf URL: https://arxiv.org/pdf/2405.01488
• Copy Paste: [[2405.01488]] Digital Twin Generators for Disease Modeling(https://arxiv.org/abs/2405.01488)
• Keywords: generative
• Abstract: A patient's digital twin is a computational model that describes the evolution of their health over time. Digital twins have the potential to revolutionize medicine by enabling individual-level computer simulations of human health, which can be used to conduct more efficient clinical trials or to recommend personalized treatment options. Due to the overwhelming complexity of human biology, machine learning approaches that leverage large datasets of historical patients' longitudinal health records to generate patients' digital twins are more tractable than potential mechanistic models. In this manuscript, we describe a neural network architecture that can learn conditional generative models of clinical trajectories, which we call Digital Twin Generators (DTGs), that can create digital twins of individual patients. We show that the same neural network architecture can be trained to generate accurate digital twins for patients across 13 different indications simply by changing the training set and tuning hyperparameters. By introducing a general purpose architecture, we aim to unlock the ability to scale machine learning approaches to larger datasets and across more indications so that a digital twin could be created for any patient in the world.

Title: Controllable Text Generation in the Instruction-Tuning Era

• Authors: Dhananjay Ashok, Barnabas Poczos
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01490
• Pdf URL: https://arxiv.org/pdf/2405.01490
• Copy Paste: [[2405.01490]] Controllable Text Generation in the Instruction-Tuning Era(https://arxiv.org/abs/2405.01490)
• Keywords: large language model
• Abstract: While most research on controllable text generation has focused on steering base Language Models, the emerging instruction-tuning and prompting paradigm offers an alternate approach to controllability. We compile and release ConGenBench, a testbed of 17 different controllable generation tasks, using a subset of it to benchmark the performance of 9 different baselines and methods on Instruction-tuned Language Models. To our surprise, we find that prompting-based approaches outperform controllable text generation methods on most datasets and tasks, highlighting a need for research on controllable text generation with Instruction-tuned Language Models in specific. Prompt-based approaches match human performance on most stylistic tasks while lagging on structural tasks, foregrounding a need to study more varied constraints and more challenging stylistic tasks. To facilitate such research, we provide an algorithm that uses only a task dataset and a Large Language Model with in-context capabilities to automatically generate a constraint dataset. This method eliminates the fields dependence on pre-curated constraint datasets, hence vastly expanding the range of constraints that can be studied in the future.

Title: Navigating Heterogeneity and Privacy in One-Shot Federated Learning with Diffusion Models

• Authors: Matias Mendieta, Guangyu Sun, Chen Chen
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01494
• Pdf URL: https://arxiv.org/pdf/2405.01494
• Copy Paste: [[2405.01494]] Navigating Heterogeneity and Privacy in One-Shot Federated Learning with Diffusion Models(https://arxiv.org/abs/2405.01494)
• Keywords: security, privacy, attack, federate, diffusion
• Abstract: Federated learning (FL) enables multiple clients to train models collectively while preserving data privacy. However, FL faces challenges in terms of communication cost and data heterogeneity. One-shot federated learning has emerged as a solution by reducing communication rounds, improving efficiency, and providing better security against eavesdropping attacks. Nevertheless, data heterogeneity remains a significant challenge, impacting performance. This work explores the effectiveness of diffusion models in one-shot FL, demonstrating their applicability in addressing data heterogeneity and improving FL performance. Additionally, we investigate the utility of our diffusion model approach, FedDiff, compared to other one-shot FL methods under differential privacy (DP). Furthermore, to improve generated sample quality under DP settings, we propose a pragmatic Fourier Magnitude Filtering (FMF) method, enhancing the effectiveness of generated data for global model training.

Title: LocInv: Localization-aware Inversion for Text-Guided Image Editing

• Authors: Chuanming Tang, Kai Wang, Fei Yang, Joost van de Weijer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01496
• Pdf URL: https://arxiv.org/pdf/2405.01496
• Copy Paste: [[2405.01496]] LocInv: Localization-aware Inversion for Text-Guided Image Editing(https://arxiv.org/abs/2405.01496)
• Keywords: diffusion, segmentation
• Abstract: Large-scale Text-to-Image (T2I) diffusion models demonstrate significant generation capabilities based on textual prompts. Based on the T2I diffusion models, text-guided image editing research aims to empower users to manipulate generated images by altering the text prompts. However, existing image editing techniques are prone to editing over unintentional regions that are beyond the intended target area, primarily due to inaccuracies in cross-attention maps. To address this problem, we propose Localization-aware Inversion (LocInv), which exploits segmentation maps or bounding boxes as extra localization priors to refine the cross-attention maps in the denoising phases of the diffusion process. Through the dynamic updating of tokens corresponding to noun words in the textual input, we are compelling the cross-attention maps to closely align with the correct noun and adjective words in the text prompt. Based on this technique, we achieve fine-grained image editing over particular objects while preventing undesired changes to other regions. Our method LocInv, based on the publicly available Stable Diffusion, is extensively evaluated on a subset of the COCO dataset, and consistently obtains superior results both quantitatively and qualitatively.The code will be released at https://github.com/wangkai930418/DPL

Title: Analyzing the Role of Semantic Representations in the Era of Large Language Models

• Authors: Zhijing Jin, Yuen Chen, Fernando Gonzalez, Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Schölkopf, Mona Diab
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01502
• Pdf URL: https://arxiv.org/pdf/2405.01502
• Copy Paste: [[2405.01502]] Analyzing the Role of Semantic Representations in the Era of Large Language Models(https://arxiv.org/abs/2405.01502)
• Keywords: large language model
• Abstract: Traditionally, natural language processing (NLP) models often use a rich set of features created by linguistic expertise, such as semantic representations. However, in the era of large language models (LLMs), more and more tasks are turned into generic, end-to-end sequence generation problems. In this paper, we investigate the question: what is the role of semantic representations in the era of LLMs? Specifically, we investigate the effect of Abstract Meaning Representation (AMR) across five diverse NLP tasks. We propose an AMR-driven chain-of-thought prompting method, which we call AMRCoT, and find that it generally hurts performance more than it helps. To investigate what AMR may have to offer on these tasks, we conduct a series of analysis experiments. We find that it is difficult to predict which input examples AMR may help or hurt on, but errors tend to arise with multi-word expressions, named entities, and in the final inference step where the LLM must connect its reasoning over the AMR to its prediction. We recommend focusing on these areas for future work in semantic representations for LLMs. Our code: https://github.com/causalNLP/amr_llm.

Title: Learnable Linguistic Watermarks for Tracing Model Extraction Attacks on Large Language Models

• Authors: Minhao Bai, Kaiyi Pang, Yongfeng Huang
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2405.01509
• Pdf URL: https://arxiv.org/pdf/2405.01509
• Copy Paste: [[2405.01509]] Learnable Linguistic Watermarks for Tracing Model Extraction Attacks on Large Language Models(https://arxiv.org/abs/2405.01509)
• Keywords: attack, robust, extraction, watermark, large language model
• Abstract: In the rapidly evolving domain of artificial intelligence, safeguarding the intellectual property of Large Language Models (LLMs) is increasingly crucial. Current watermarking techniques against model extraction attacks, which rely on signal insertion in model logits or post-processing of generated text, remain largely heuristic. We propose a novel method for embedding learnable linguistic watermarks in LLMs, aimed at tracing and preventing model extraction attacks. Our approach subtly modifies the LLM's output distribution by introducing controlled noise into token frequency distributions, embedding an statistically identifiable controllable watermark.We leverage statistical hypothesis testing and information theory, particularly focusing on Kullback-Leibler Divergence, to differentiate between original and modified distributions effectively. Our watermarking method strikes a delicate well balance between robustness and output quality, maintaining low false positive/negative rates and preserving the LLM's original performance.

Title: Transformer-Aided Semantic Communications

• Authors: Matin Mortaheb, Erciyes Karakaya, Mohammad A. Amir Khojastepour, Sennur Ulukus
• Subjects: cs.CV, cs.IT, cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2405.01521
• Pdf URL: https://arxiv.org/pdf/2405.01521
• Copy Paste: [[2405.01521]] Transformer-Aided Semantic Communications(https://arxiv.org/abs/2405.01521)
• Keywords: transformer, large language model
• Abstract: The transformer structure employed in large language models (LLMs), as a specialized category of deep neural networks (DNNs) featuring attention mechanisms, stands out for their ability to identify and highlight the most relevant aspects of input data. Such a capability is particularly beneficial in addressing a variety of communication challenges, notably in the realm of semantic communication where proper encoding of the relevant data is critical especially in systems with limited bandwidth. In this work, we employ vision transformers specifically for the purpose of compression and compact representation of the input image, with the goal of preserving semantic information throughout the transmission process. Through the use of the attention mechanism inherent in transformers, we create an attention mask. This mask effectively prioritizes critical segments of images for transmission, ensuring that the reconstruction phase focuses on key objects highlighted by the mask. Our methodology significantly improves the quality of semantic communication and optimizes bandwidth usage by encoding different parts of the data in accordance with their semantic information content, thus enhancing overall efficiency. We evaluate the effectiveness of our proposed framework using the TinyImageNet dataset, focusing on both reconstruction quality and accuracy. Our evaluation results demonstrate that our framework successfully preserves semantic information, even when only a fraction of the encoded data is transmitted, according to the intended compression rates.

Title: FLAME: Factuality-Aware Alignment for Large Language Models

• Authors: Sheng-Chieh Lin, Luyu Gao, Barlas Oguz, Wenhan Xiong, Jimmy Lin, Wen-tau Yih, Xilun Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01525
• Pdf URL: https://arxiv.org/pdf/2405.01525
• Copy Paste: [[2405.01525]] FLAME: Factuality-Aware Alignment for Large Language Models(https://arxiv.org/abs/2405.01525)
• Keywords: large language model
• Abstract: Alignment is a standard procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants. We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e. hallucination). In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both alignment steps:\ supervised fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that training the LLM on new knowledge or unfamiliar texts can encourage hallucination. This makes SFT less factual as it trains on human labeled data that may be novel to the LLM. Furthermore, reward functions used in standard RL can also encourage hallucination, because it guides the LLM to provide more helpful responses on a diverse set of instructions, often preferring longer and more detailed responses. Based on these observations, we propose factuality-aware alignment, comprised of factuality-aware SFT and factuality-aware RL through direct preference optimization. Experiments show that our proposed factuality-aware alignment guides LLMs to output more factual responses while maintaining instruction-following capability.

Title: OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning

• Authors: Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, Jose M. Alvarez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01533
• Pdf URL: https://arxiv.org/pdf/2405.01533
• Copy Paste: [[2405.01533]] OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning(https://arxiv.org/abs/2405.01533)
• Keywords: large language model
• Abstract: The advances in multimodal large language models (MLLMs) have led to growing interests in LLM-based autonomous driving agents to leverage their strong reasoning capabilities. However, capitalizing on MLLMs' strong reasoning capabilities for improved planning behavior is challenging since planning requires full 3D situational awareness beyond 2D reasoning. To address this challenge, our work proposes a holistic framework for strong alignment between agent models and 3D driving tasks. Our framework starts with a novel 3D MLLM architecture that uses sparse queries to lift and compress visual representations into 3D before feeding them into an LLM. This query-based representation allows us to jointly encode dynamic objects and static map elements (e.g., traffic lanes), providing a condensed world model for perception-action alignment in 3D. We further propose OmniDrive-nuScenes, a new visual question-answering dataset challenging the true 3D situational awareness of a model with comprehensive visual question-answering (VQA) tasks, including scene description, traffic regulation, 3D grounding, counterfactual reasoning, decision making and planning. Extensive studies show the effectiveness of the proposed architecture as well as the importance of the VQA tasks for reasoning and planning in complex 3D scenes.

Title: Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks

• Authors: Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, Ruslan Salakhutdinov
• Subjects: cs.LG, cs.AI, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2405.01534
• Pdf URL: https://arxiv.org/pdf/2405.01534
• Copy Paste: [[2405.01534]] Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks(https://arxiv.org/abs/2405.01534)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been shown to be capable of performing high-level planning for long-horizon robotics tasks, yet existing methods require access to a pre-defined skill library (e.g. picking, placing, pulling, pushing, navigating). However, LLM planning does not address how to design or learn those behaviors, which remains challenging particularly in long-horizon settings. Furthermore, for many tasks of interest, the robot needs to be able to adjust its behavior in a fine-grained manner, requiring the agent to be capable of modifying low-level control actions. Can we instead use the internet-scale knowledge from LLMs for high-level policies, guiding reinforcement learning (RL) policies to efficiently solve robotic control tasks online without requiring a pre-determined set of skills? In this paper, we propose Plan-Seq-Learn (PSL): a modular approach that uses motion planning to bridge the gap between abstract language and learned low-level control for solving long-horizon robotics tasks from scratch. We demonstrate that PSL achieves state-of-the-art results on over 25 challenging robotics tasks with up to 10 stages. PSL solves long-horizon tasks from raw visual input spanning four benchmarks at success rates of over 85%, out-performing language-based, classical, and end-to-end approaches. Video results and code at https://mihdalal.github.io/planseqlearn/

Title: Customizing Text-to-Image Models with a Single Image Pair

• Authors: Maxwell Jones, Sheng-Yu Wang, Nupur Kumari, David Bau, Jun-Yan Zhu
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01536
• Pdf URL: https://arxiv.org/pdf/2405.01536
• Copy Paste: [[2405.01536]] Customizing Text-to-Image Models with a Single Image Pair(https://arxiv.org/abs/2405.01536)
• Keywords: diffusion, generative
• Abstract: Art reinterpretation is the practice of creating a variation of a reference work, making a paired artwork that exhibits a distinct artistic style. We ask if such an image pair can be used to customize a generative model to capture the demonstrated stylistic difference. We propose Pair Customization, a new customization method that learns stylistic difference from a single image pair and then applies the acquired style to the generation process. Unlike existing methods that learn to mimic a single concept from a collection of images, our method captures the stylistic difference between paired images. This allows us to apply a stylistic change without overfitting to the specific image content in the examples. To address this new task, we employ a joint optimization method that explicitly separates the style and content into distinct LoRA weight spaces. We optimize these style and content weights to reproduce the style and content images while encouraging their orthogonality. During inference, we modify the diffusion process via a new style guidance based on our learned weights. Both qualitative and quantitative experiments show that our method can effectively learn style while avoiding overfitting to image content, highlighting the potential of modeling such stylistic differences from a single image pair.

Title: Multi-Space Alignments Towards Universal LiDAR Segmentation

• Authors: Youquan Liu, Lingdong Kong, Xiaoyang Wu, Runnan Chen, Xin Li, Liang Pan, Ziwei Liu, Yuexin Ma
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2405.01538
• Pdf URL: https://arxiv.org/pdf/2405.01538
• Copy Paste: [[2405.01538]] Multi-Space Alignments Towards Universal LiDAR Segmentation(https://arxiv.org/abs/2405.01538)
• Keywords: robust, segmentation
• Abstract: A unified and versatile LiDAR segmentation model with strong robustness and generalizability is desirable for safe autonomous driving perception. This work presents M3Net, a one-of-a-kind framework for fulfilling multi-task, multi-dataset, multi-modality LiDAR segmentation in a universal manner using just a single set of parameters. To better exploit data volume and diversity, we first combine large-scale driving datasets acquired by different types of sensors from diverse scenes and then conduct alignments in three spaces, namely data, feature, and label spaces, during the training. As a result, M3Net is capable of taming heterogeneous data for training state-of-the-art LiDAR segmentation models. Extensive experiments on twelve LiDAR segmentation datasets verify our effectiveness. Notably, using a shared set of parameters, M3Net achieves 75.1%, 83.1%, and 72.4% mIoU scores, respectively, on the official benchmarks of SemanticKITTI, nuScenes, and Waymo Open.