2024-09-18

Title: Harnessing Artificial Intelligence for Wildlife Conservation

• Authors: Paul Fergus, Carl Chalmers, Steve Longmore, Serge Wich
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10523
• Pdf URL: https://arxiv.org/pdf/2409.10523
• Copy Paste: [[2409.10523]] Harnessing Artificial Intelligence for Wildlife Conservation(https://arxiv.org/abs/2409.10523)
• Keywords: transformer
• Abstract: The rapid decline in global biodiversity demands innovative conservation strategies. This paper examines the use of artificial intelligence (AI) in wildlife conservation, focusing on the Conservation AI platform. Leveraging machine learning and computer vision, Conservation AI detects and classifies animals, humans, and poaching-related objects using visual spectrum and thermal infrared cameras. The platform processes this data with convolutional neural networks (CNNs) and Transformer architectures to monitor species, including those which are critically endangered. Real-time detection provides the immediate responses required for time-critical situations (e.g. poaching), while non-real-time analysis supports long-term wildlife monitoring and habitat health assessment. Case studies from Europe, North America, Africa, and Southeast Asia highlight the platform's success in species identification, biodiversity monitoring, and poaching prevention. The paper also discusses challenges related to data quality, model accuracy, and logistical constraints, while outlining future directions involving technological advancements, expansion into new geographical regions, and deeper collaboration with local communities and policymakers. Conservation AI represents a significant step forward in addressing the urgent challenges of wildlife conservation, offering a scalable and adaptable solution that can be implemented globally.

Title: Ethical Challenges in Computer Vision: Ensuring Privacy and Mitigating Bias in Publicly Available Datasets

• Authors: Ghalib Ahmed Tahir
• Subjects: cs.CV, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.10533
• Pdf URL: https://arxiv.org/pdf/2409.10533
• Copy Paste: [[2409.10533]] Ethical Challenges in Computer Vision: Ensuring Privacy and Mitigating Bias in Publicly Available Datasets(https://arxiv.org/abs/2409.10533)
• Keywords: security, privacy, protect
• Abstract: This paper aims to shed light on the ethical problems of creating and deploying computer vision tech, particularly in using publicly available datasets. Due to the rapid growth of machine learning and artificial intelligence, computer vision has become a vital tool in many industries, including medical care, security systems, and trade. However, extensive use of visual data that is often collected without consent due to an informed discussion of its ramifications raises significant concerns about privacy and bias. The paper also examines these issues by analyzing popular datasets such as COCO, LFW, ImageNet, CelebA, PASCAL VOC, etc., that are usually used for training computer vision models. We offer a comprehensive ethical framework that addresses these challenges regarding the protection of individual rights, minimization of bias as well as openness and responsibility. We aim to encourage AI development that will take into account societal values as well as ethical standards to avoid any public harm.

Title: OxML Challenge 2023: Carcinoma classification using data augmentation

• Authors: Kislay Raj, Teerath Kumar, Alessandra Mileo, Malika Bendechache
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.10544
• Pdf URL: https://arxiv.org/pdf/2409.10544
• Copy Paste: [[2409.10544]] OxML Challenge 2023: Carcinoma classification using data augmentation(https://arxiv.org/abs/2409.10544)
• Keywords: privacy
• Abstract: Carcinoma is the prevailing type of cancer and can manifest in various body parts. It is widespread and can potentially develop in numerous locations within the body. In the medical domain, data for carcinoma cancer is often limited or unavailable due to privacy concerns. Moreover, when available, it is highly imbalanced, with a scarcity of positive class samples and an abundance of negative ones. The OXML 2023 challenge provides a small and imbalanced dataset, presenting significant challenges for carcinoma classification. To tackle these issues, participants in the challenge have employed various approaches, relying on pre-trained models, preprocessing techniques, and few-shot learning. Our work proposes a novel technique that combines padding augmentation and ensembling to address the carcinoma classification challenge. In our proposed method, we utilize ensembles of five neural networks and implement padding as a data augmentation technique, taking into account varying image sizes to enhance the classifier's performance. Using our approach, we made place into top three and declared as winner.

Title: NoPhish: Efficient Chrome Extension for Phishing Detection Using Machine Learning Techniques

• Authors: Leand Thaqi, Arbnor Halili, Kamer Vishi, Blerim Rexha
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10547
• Pdf URL: https://arxiv.org/pdf/2409.10547
• Copy Paste: [[2409.10547]] NoPhish: Efficient Chrome Extension for Phishing Detection Using Machine Learning Techniques(https://arxiv.org/abs/2409.10547)
• Keywords: security, attack
• Abstract: The growth of digitalization services via web browsers has simplified our daily routine of doing business. But at the same time, it has made the web browser very attractive for several cyber-attacks. Web phishing is a well-known cyberattack that is used by attackers camouflaging as trustworthy web servers to obtain sensitive user information such as credit card numbers, bank information, personal ID, social security number, and username and passwords. In recent years many techniques have been developed to identify the authentic web pages that users visit and warn them when the webpage is phishing. In this paper, we have developed an extension for Chrome the most favorite web browser, that will serve as a middleware between the user and phishing websites. The Chrome extension named "NoPhish" shall identify a phishing webpage based on several Machine Learning techniques. We have used the training dataset from "PhishTank" and extracted the 22 most popular features as rated by the Alexa database. The training algorithms used are Random Forest, Support Vector Machine, and k-Nearest Neighbor. The performance results show that Random Forest delivers the best precision.

Title: Convolutional Networks as Extremely Small Foundation Models: Visual Prompting and Theoretical Perspective

• Authors: Jianqiao Wangni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.10555
• Pdf URL: https://arxiv.org/pdf/2409.10555
• Copy Paste: [[2409.10555]] Convolutional Networks as Extremely Small Foundation Models: Visual Prompting and Theoretical Perspective(https://arxiv.org/abs/2409.10555)
• Keywords: segmentation
• Abstract: Comparing to deep neural networks trained for specific tasks, those foundational deep networks trained on generic datasets such as ImageNet classification, benefits from larger-scale datasets, simpler network structure and easier training techniques. In this paper, we design a prompting module which performs few-shot adaptation of generic deep networks to new tasks. Driven by learning theory, we derive prompting modules that are as simple as possible, as they generalize better under the same training error. We use a case study on video object segmentation to experiment. We give a concrete prompting module, the Semi-parametric Deep Forest (SDForest) that combines several nonparametric methods such as correlation filter, random forest, image-guided filter, with a deep network trained for ImageNet classification task. From a learning-theoretical point of view, all these models are of significantly smaller VC dimension or complexity so tend to generalize better, as long as the empirical studies show that the training error of this simple ensemble can achieve comparable results from a end-to-end trained deep network. We also propose a novel methods of analyzing the generalization under the setting of video object segmentation to make the bound tighter. In practice, SDForest has extremely low computation cost and achieves real-time even on CPU. We test on video object segmentation tasks and achieve competitive performance at DAVIS2016 and DAVIS2017 with purely deep learning approaches, without any training or fine-tuning.

Title: Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers

• Authors: Siyu Chen, Heejune Sheen, Tianhao Wang, Zhuoran Yang
• Subjects: cs.LG, cs.AI, cs.CL, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.10559
• Pdf URL: https://arxiv.org/pdf/2409.10559
• Copy Paste: [[2409.10559]] Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers(https://arxiv.org/abs/2409.10559)
• Keywords: transformer, large language model
• Abstract: In-context learning (ICL) is a cornerstone of large language model (LLM) functionality, yet its theoretical foundations remain elusive due to the complexity of transformer architectures. In particular, most existing work only theoretically explains how the attention mechanism facilitates ICL under certain data models. It remains unclear how the other building blocks of the transformer contribute to ICL. To address this question, we study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data, where each token in the Markov chain statistically depends on the previous $n$ tokens. We analyze a sophisticated transformer model featuring relative positional embedding, multi-head softmax attention, and a feed-forward layer with normalization. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model that performs a generalized version of the induction head mechanism with a learned feature, resulting from the congruous contribution of all the building blocks. In the limiting model, the first attention layer acts as a $\mathit{copier}$, copying past tokens within a given window to each position, and the feed-forward network with normalization acts as a $\mathit{selector}$ that generates a feature vector by only looking at informationally relevant parents from the window. Finally, the second attention layer is a $\mathit{classifier}$ that compares these features with the feature at the output position, and uses the resulting similarity scores to generate the desired output. Our theory is further validated by experiments.

Title: DrLLM: Prompt-Enhanced Distributed Denial-of-Service Resistance Method with Large Language Models

• Authors: Zhenyu Yin, Shang Liu, Guangyuan Xu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10561
• Pdf URL: https://arxiv.org/pdf/2409.10561
• Copy Paste: [[2409.10561]] DrLLM: Prompt-Enhanced Distributed Denial-of-Service Resistance Method with Large Language Models(https://arxiv.org/abs/2409.10561)
• Keywords: security, attack, large language model
• Abstract: The increasing number of Distributed Denial of Service (DDoS) attacks poses a major threat to the Internet, highlighting the importance of DDoS mitigation. Most existing approaches require complex training methods to learn data features, which increases the complexity and generality of the application. In this paper, we propose DrLLM, which aims to mine anomalous traffic information in zero-shot scenarios through Large Language Models (LLMs). To bridge the gap between DrLLM and existing approaches, we embed the global and local information of the traffic data into the reasoning paradigm and design three modules, namely Knowledge Embedding, Token Embedding, and Progressive Role Reasoning, for data representation and reasoning. In addition we explore the generalization of prompt engineering in the cybersecurity domain to improve the classification capability of DrLLM. Our ablation experiments demonstrate the applicability of DrLLM in zero-shot scenarios and further demonstrate the potential of LLMs in the network domains. DrLLM implementation code has been open-sourced at this https URL.

Title: Are Existing Road Design Guidelines Suitable for Autonomous Vehicles?

• Authors: Yang Sun, Christopher M. Poskitt, Jun Sun
• Subjects: cs.CV, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.10562
• Pdf URL: https://arxiv.org/pdf/2409.10562
• Copy Paste: [[2409.10562]] Are Existing Road Design Guidelines Suitable for Autonomous Vehicles?(https://arxiv.org/abs/2409.10562)
• Keywords: attack
• Abstract: The emergence of Autonomous Vehicles (AVs) has spurred research into testing the resilience of their perception systems, i.e. to ensure they are not susceptible to making critical misjudgements. It is important that they are tested not only with respect to other vehicles on the road, but also those objects placed on the roadside. Trash bins, billboards, and greenery are all examples of such objects, typically placed according to guidelines that were developed for the human visual system, and which may not align perfectly with the needs of AVs. Existing tests, however, usually focus on adversarial objects with conspicuous shapes/patches, that are ultimately unrealistic given their unnatural appearances and the need for white box knowledge. In this work, we introduce a black box attack on the perception systems of AVs, in which the objective is to create realistic adversarial scenarios (i.e. satisfying road design guidelines) by manipulating the positions of common roadside objects, and without resorting to `unnatural' adversarial patches. In particular, we propose TrashFuzz , a fuzzing algorithm to find scenarios in which the placement of these objects leads to substantial misperceptions by the AV -- such as mistaking a traffic light's colour -- with overall the goal of causing it to violate traffic laws. To ensure the realism of these scenarios, they must satisfy several rules encoding regulatory guidelines about the placement of objects on public streets. We implemented and evaluated these attacks for the Apollo, finding that TrashFuzz induced it into violating 15 out of 24 different traffic laws.

Title: Applying Action Masking and Curriculum Learning Techniques to Improve Data Efficiency and Overall Performance in Operational Technology Cyber Security using Reinforcement Learning

• Authors: Alec Wilson, William Holmes, Ryan Menzies, Kez Smithson Whitehead
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10563
• Pdf URL: https://arxiv.org/pdf/2409.10563
• Copy Paste: [[2409.10563]] Applying Action Masking and Curriculum Learning Techniques to Improve Data Efficiency and Overall Performance in Operational Technology Cyber Security using Reinforcement Learning(https://arxiv.org/abs/2409.10563)
• Keywords: security, attack
• Abstract: In previous work, the IPMSRL environment (Integrated Platform Management System Reinforcement Learning environment) was developed with the aim of training defensive RL agents in a simulator representing a subset of an IPMS on a maritime vessel under a cyber-attack. This paper extends the use of IPMSRL to enhance realism including the additional dynamics of false positive alerts and alert delay. Applying curriculum learning, in the most difficult environment tested, resulted in an episode reward mean increasing from a baseline result of -2.791 to -0.569. Applying action masking, in the most difficult environment tested, resulted in an episode reward mean increasing from a baseline result of -2.791 to -0.743. Importantly, this level of performance was reached in less than 1 million timesteps, which was far more data efficient than vanilla PPO which reached a lower level of performance after 2.5 million timesteps. The training method which resulted in the highest level of performance observed in this paper was a combination of the application of curriculum learning and action masking, with a mean episode reward of 0.137. This paper also introduces a basic hardcoded defensive agent encoding a representation of cyber security best practice, which provides context to the episode reward mean figures reached by the RL agents. The hardcoded agent managed an episode reward mean of -1.895. This paper therefore shows that applications of curriculum learning and action masking, both independently and in tandem, present a way to overcome the complex real-world dynamics that are present in operational technology cyber security threat remediation.

Title: Eureka: Evaluating and Understanding Large Foundation Models

• Authors: Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, Safoora Yousefi
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.10566
• Pdf URL: https://arxiv.org/pdf/2409.10566
• Copy Paste: [[2409.10566]] Eureka: Evaluating and Understanding Large Foundation Models(https://arxiv.org/abs/2409.10566)
• Keywords: generative
• Abstract: Rigorous and reproducible evaluation is critical for assessing the state of the art and for guiding scientific advances in Artificial Intelligence. Evaluation is challenging in practice due to several reasons, including benchmark saturation, lack of transparency in methods used for measurement, development challenges in extracting measurements for generative tasks, and, more generally, the extensive number of capabilities required for a well-rounded comparison across models. We make three contributions to alleviate the above challenges. First, we present Eureka, an open-source framework for standardizing evaluations of large foundation models beyond single-score reporting and rankings. Second, we introduce Eureka-Bench as an extensible collection of benchmarks testing capabilities that (i) are still challenging for state-of-the-art models and (ii) represent fundamental but overlooked language and multimodal capabilities. The inherent space for improvement in non-saturated benchmarks enables us to discover meaningful differences between models at a capability level. Third, using Eureka, we conduct an analysis of 12 state-of-the-art models, providing in-depth insights into failure understanding and model comparison, which can be leveraged to plan targeted improvements. In contrast to recent trends in reports and leaderboards showing absolute rankings and claims for one model or another to be the best, our analysis shows that there is no such best model. Different models have different strengths, but there are models that appear more often than others as best performers for some capabilities. Despite the recent improvements, current models still struggle with several fundamental capabilities including detailed image understanding, benefiting from multimodal input when available rather than fully relying on language, factuality and grounding for information retrieval, and over refusals.

Title: Protecting Copyright of Medical Pre-trained Language Models: Training-Free Backdoor Watermarking

• Authors: Cong Kong, Rui Xu, Weixi Chen, Jiawei Chen, Zhaoxia Yin
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.10570
• Pdf URL: https://arxiv.org/pdf/2409.10570
• Copy Paste: [[2409.10570]] Protecting Copyright of Medical Pre-trained Language Models: Training-Free Backdoor Watermarking(https://arxiv.org/abs/2409.10570)
• Keywords: protect, attack, robust, watermark
• Abstract: Pre-training language models followed by fine-tuning on specific tasks is standard in NLP, but traditional models often underperform when applied to the medical domain, leading to the development of specialized medical pre-trained language models (Med-PLMs). These models are valuable assets but are vulnerable to misuse and theft, requiring copyright protection. However, no existing watermarking methods are tailored for Med-PLMs, and adapting general PLMs watermarking techniques to the medical domain faces challenges such as task incompatibility, loss of fidelity, and inefficiency. To address these issues, we propose the first training-free backdoor watermarking method for Med-PLMs. Our method uses rare special symbols as trigger words, which do not impact downstream task performance, embedding watermarks by replacing their original embeddings with those of specific medical terms in the Med-PLMs' word embeddings layer. After fine-tuning the watermarked Med-PLMs on various medical downstream tasks, the final models (FMs) respond to the trigger words in the same way they would to the corresponding medical terms. This property can be utilized to extract the watermark. Experiments demonstrate that our method achieves high fidelity while effectively extracting watermarks across various medical downstream tasks. Additionally, our method demonstrates robustness against various attacks and significantly enhances the efficiency of watermark embedding, reducing the embedding time from 10 hours to 10 seconds.

Title: ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood

• Authors: Ruoyu Wang, Jiachen Sun, Shaowei Hua, Quan Fang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10571
• Pdf URL: https://arxiv.org/pdf/2409.10571
• Copy Paste: [[2409.10571]] ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood(https://arxiv.org/abs/2409.10571)
• Keywords: large language model
• Abstract: Direct Preference Optimization (DPO) is a method for enhancing model performance by directly optimizing for the preferences or rankings of outcomes, instead of traditional loss functions. This approach has proven effective in aligning Large Language Models (LLMs) with human preferences. Despite its widespread use across various tasks, DPO has been criticized for its sensitivity to the effectiveness of Supervised Fine-Tuning (SFT) and its limitations in enabling models to learn human-preferred responses, leading to less satisfactory performance. To address these limitations, we propose Aligned Supervised Fine-Tuning (ASFT), an effective approach that better aligns LLMs with pair-wise datasets by optimizing absolute likelihood for each response, rather than using the Bradley-Terry model, and eliminates the need for a reference model. Through theoretical gradient analysis, we demonstrate that ASFT mitigates the issue where the DPO loss function decreases the probability of generating human-dispreferred data at a faster rate than it increases the probability of producing preferred data. Additionally, we compare ASFT to DPO and its latest variants, such as the single-step approach ORPO, using the latest instruction-tuned model Llama3, which has been fine-tuned on UltraFeedback and HH-RLHF. We evaluated performance on instruction-following benchmarks like MT-Bench and traditional text generation metrics such as BLEU-4 and ROUGE-L. Extensive experiments demonstrate that ASFT is an effective alignment approach, consistently outperforming existing methods.

Title: Detection Made Easy: Potentials of Large Language Models for Solidity Vulnerabilities

• Authors: Md Tauseef Alam, Raju Halder, Abyayananda Maiti
• Subjects: cs.CR, cs.AI, cs.ET, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10574
• Pdf URL: https://arxiv.org/pdf/2409.10574
• Copy Paste: [[2409.10574]] Detection Made Easy: Potentials of Large Language Models for Solidity Vulnerabilities(https://arxiv.org/abs/2409.10574)
• Keywords: attack, generative, large language model
• Abstract: The large-scale deployment of Solidity smart contracts on the Ethereum mainnet has increasingly attracted financially-motivated attackers in recent years. A few now-infamous attacks in Ethereum's history includes DAO attack in 2016 (50 million dollars lost), Parity Wallet hack in 2017 (146 million dollars locked), Beautychain's token BEC in 2018 (900 million dollars market value fell to 0), and NFT gaming blockchain breach in 2022 ($600 million in Ether stolen). This paper presents a comprehensive investigation of the use of large language models (LLMs) and their capabilities in detecting OWASP Top Ten vulnerabilities in Solidity. We introduce a novel, class-balanced, structured, and labeled dataset named VulSmart, which we use to benchmark and compare the performance of open-source LLMs such as CodeLlama, Llama2, CodeT5 and Falcon, alongside closed-source models like GPT-3.5 Turbo and GPT-4o Mini. Our proposed SmartVD framework is rigorously tested against these models through extensive automated and manual evaluations, utilizing BLEU and ROUGE metrics to assess the effectiveness of vulnerability detection in smart contracts. We also explore three distinct prompting strategies-zero-shot, few-shot, and chain-of-thought-to evaluate the multi-class classification and generative capabilities of the SmartVD framework. Our findings reveal that SmartVD outperforms its open-source counterparts and even exceeds the performance of closed-source base models like GPT-3.5 and GPT-4 Mini. After fine-tuning, the closed-source models, GPT-3.5 Turbo and GPT-4o Mini, achieved remarkable performance with 99% accuracy in detecting vulnerabilities, 94% in identifying their types, and 98% in determining severity. Notably, SmartVD performs best with the `chain-of-thought' prompting technique, whereas the fine-tuned closed-source models excel with the `zero-shot' prompting approach.

Title: Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports

• Authors: Mohamed Sobhi Jabal, Pranav Warman, Jikai Zhang, Kartikeye Gupta, Ayush Jain, Maciej Mazurowski, Walter Wiggins, Kirti Magudia, Evan Calabrese
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10576
• Pdf URL: https://arxiv.org/pdf/2409.10576
• Copy Paste: [[2409.10576]] Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports(https://arxiv.org/abs/2409.10576)
• Keywords: privacy, extraction, large language model
• Abstract: Purpose: To develop and evaluate an automated system for extracting structured clinical information from unstructured radiology and pathology reports using open-weights large language models (LMs) and retrieval augmented generation (RAG), and to assess the effects of model configuration variables on extraction performance. Methods and Materials: The study utilized two datasets: 7,294 radiology reports annotated for Brain Tumor Reporting and Data System (BT-RADS) scores and 2,154 pathology reports annotated for isocitrate dehydrogenase (IDH) mutation status. An automated pipeline was developed to benchmark the performance of various LMs and RAG configurations. The impact of model size, quantization, prompting strategies, output formatting, and inference parameters was systematically evaluated. Results: The best performing models achieved over 98% accuracy in extracting BT-RADS scores from radiology reports and over 90% for IDH mutation status extraction from pathology reports. The top model being medical fine-tuned llama3. Larger, newer, and domain fine-tuned models consistently outperformed older and smaller models. Model quantization had minimal impact on performance. Few-shot prompting significantly improved accuracy. RAG improved performance for complex pathology reports but not for shorter radiology reports. Conclusions: Open LMs demonstrate significant potential for automated extraction of structured clinical data from unstructured clinical reports with local privacy-preserving application. Careful model selection, prompt engineering, and semi-automated optimization using annotated data are critical for optimal performance. These approaches could be reliable enough for practical use in research workflows, highlighting the potential for human-machine collaboration in healthcare data extraction.

Title: GLEAN: Generative Learning for Eliminating Adversarial Noise

• Authors: Justin Lyu Kim, Kyoungwan Woo
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10578
• Pdf URL: https://arxiv.org/pdf/2409.10578
• Copy Paste: [[2409.10578]] GLEAN: Generative Learning for Eliminating Adversarial Noise(https://arxiv.org/abs/2409.10578)
• Keywords: protect, attack, diffusion, generative
• Abstract: In the age of powerful diffusion models such as DALL-E and Stable Diffusion, many in the digital art community have suffered style mimicry attacks due to fine-tuning these models on their works. The ability to mimic an artist's style via text-to-image diffusion models raises serious ethical issues, especially without explicit consent. Glaze, a tool that applies various ranges of perturbations to digital art, has shown significant success in preventing style mimicry attacks, at the cost of artifacts ranging from imperceptible noise to severe quality degradation. The release of Glaze has sparked further discussions regarding the effectiveness of similar protection methods. In this paper, we propose GLEAN- applying I2I generative networks to strip perturbations from Glazed images, evaluating the performance of style mimicry attacks before and after GLEAN on the results of Glaze. GLEAN aims to support and enhance Glaze by highlighting its limitations and encouraging further development.

Title: Veridical Data Science for Medical Foundation Models

• Authors: Ahmed Alaa, Bin Yu
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.10580
• Pdf URL: https://arxiv.org/pdf/2409.10580
• Copy Paste: [[2409.10580]] Veridical Data Science for Medical Foundation Models(https://arxiv.org/abs/2409.10580)
• Keywords: large language model
• Abstract: The advent of foundation models (FMs) such as large language models (LLMs) has led to a cultural shift in data science, both in medicine and beyond. This shift involves moving away from specialized predictive models trained for specific, well-defined domain questions to generalist FMs pre-trained on vast amounts of unstructured data, which can then be adapted to various clinical tasks and questions. As a result, the standard data science workflow in medicine has been fundamentally altered; the foundation model lifecycle (FMLC) now includes distinct upstream and downstream processes, in which computational resources, model and data access, and decision-making power are distributed among multiple stakeholders. At their core, FMs are fundamentally statistical models, and this new workflow challenges the principles of Veridical Data Science (VDS), hindering the rigorous statistical analysis expected in transparent and scientifically reproducible data science practices. We critically examine the medical FMLC in light of the core principles of VDS: predictability, computability, and stability (PCS), and explain how it deviates from the standard data science workflow. Finally, we propose recommendations for a reimagined medical FMLC that expands and refines the PCS principles for VDS including considering the computational and accessibility constraints inherent to FMs.

Title: CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios

• Authors: Luning Wang, Shiyao Li, Xuefei Ning, Zhihang Yuan, Shengen Yan, Guohao Dai, Yu Wang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10593
• Pdf URL: https://arxiv.org/pdf/2409.10593
• Copy Paste: [[2409.10593]] CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios(https://arxiv.org/abs/2409.10593)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been widely adopted to process long-context tasks. However, the large memory overhead of the key-value (KV) cache poses significant challenges in long-context scenarios. Existing training-free KV cache compression methods typically focus on quantization and token pruning, which have compression limits, and excessive sparsity can lead to severe performance degradation. Other methods design new architectures with less KV overhead but require significant training overhead. To address the above two drawbacks, we further explore the redundancy in the channel dimension and apply an architecture-level design with minor training costs. Therefore, we introduce CSKV, a training-efficient Channel Shrinking technique for KV cache compression: (1) We first analyze the singular value distribution of the KV cache, revealing significant redundancy and compression potential along the channel dimension. Based on this observation, we propose using low-rank decomposition for key and value layers and storing the low-dimension features. (2) To preserve model performance, we introduce a bi-branch KV cache, including a window-based full-precision KV cache and a low-precision compressed KV cache. (3) To reduce the training costs, we minimize the layer-wise reconstruction loss for the compressed KV cache instead of retraining the entire LLMs. Extensive experiments show that CSKV can reduce the memory overhead of the KV cache by 80% while maintaining the model's long-context capability. Moreover, we show that our method can be seamlessly combined with quantization to further reduce the memory overhead, achieving a compression ratio of up to 95%.

Title: Kolmogorov-Arnold Transformer

• Authors: Xingyi Yang, Xinchao Wang
• Subjects: cs.LG, cs.AI, cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.10594
• Pdf URL: https://arxiv.org/pdf/2409.10594
• Copy Paste: [[2409.10594]] Kolmogorov-Arnold Transformer(https://arxiv.org/abs/2409.10594)
• Keywords: transformer
• Abstract: Transformers stand as the cornerstone of mordern deep learning. Traditionally, these models rely on multi-layer perceptron (MLP) layers to mix the information between channels. In this paper, we introduce the Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces MLP layers with Kolmogorov-Arnold Network (KAN) layers to enhance the expressiveness and performance of the model. Integrating KANs into transformers, however, is no easy feat, especially when scaled up. Specifically, we identify three key challenges: (C1) Base function. The standard B-spline function used in KANs is not optimized for parallel computing on modern hardware, resulting in slower inference speeds. (C2) Parameter and Computation Inefficiency. KAN requires a unique function for each input-output pair, making the computation extremely large. (C3) Weight initialization. The initialization of weights in KANs is particularly challenging due to their learnable activation functions, which are critical for achieving convergence in deep neural networks. To overcome the aforementioned challenges, we propose three key solutions: (S1) Rational basis. We replace B-spline functions with rational functions to improve compatibility with modern GPUs. By implementing this in CUDA, we achieve faster computations. (S2) Group KAN. We share the activation weights through a group of neurons, to reduce the computational load without sacrificing performance. (S3) Variance-preserving initialization. We carefully initialize the activation weights to make sure that the activation variance is maintained across layers. With these designs, KAT scales effectively and readily outperforms traditional MLP-based transformers.

Title: Optimizing Resource Consumption in Diffusion Models through Hallucination Early Detection

• Authors: Federico Betti, Lorenzo Baraldi, Lorenzo Baraldi, Rita Cucchiara, Nicu Sebe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.10597
• Pdf URL: https://arxiv.org/pdf/2409.10597
• Copy Paste: [[2409.10597]] Optimizing Resource Consumption in Diffusion Models through Hallucination Early Detection(https://arxiv.org/abs/2409.10597)
• Keywords: diffusion, generative
• Abstract: Diffusion models have significantly advanced generative AI, but they encounter difficulties when generating complex combinations of multiple objects. As the final result heavily depends on the initial seed, accurately ensuring the desired output can require multiple iterations of the generation process. This repetition not only leads to a waste of time but also increases energy consumption, echoing the challenges of efficiency and accuracy in complex generative tasks. To tackle this issue, we introduce HEaD (Hallucination Early Detection), a new paradigm designed to swiftly detect incorrect generations at the beginning of the diffusion process. The HEaD pipeline combines cross-attention maps with a new indicator, the Predicted Final Image, to forecast the final outcome by leveraging the information available at early stages of the generation process. We demonstrate that using HEaD saves computational resources and accelerates the generation process to get a complete image, i.e. an image where all requested objects are accurately depicted. Our findings reveal that HEaD can save up to 12% of the generation time on a two objects scenario and underscore the importance of early detection mechanisms in generative models.

Title: Exploring Fine-tuned Generative Models for Keyphrase Selection: A Case Study for Russian

• Authors: Anna Glazkova, Dmitry Morozov
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10640
• Pdf URL: https://arxiv.org/pdf/2409.10640
• Copy Paste: [[2409.10640]] Exploring Fine-tuned Generative Models for Keyphrase Selection: A Case Study for Russian(https://arxiv.org/abs/2409.10640)
• Keywords: extraction, transformer, generative
• Abstract: Keyphrase selection plays a pivotal role within the domain of scholarly texts, facilitating efficient information retrieval, summarization, and indexing. In this work, we explored how to apply fine-tuned generative transformer-based models to the specific task of keyphrase selection within Russian scientific texts. We experimented with four distinct generative models, such as ruT5, ruGPT, mT5, and mBART, and evaluated their performance in both in-domain and cross-domain settings. The experiments were conducted on the texts of Russian scientific abstracts from four domains: mathematics \& computer science, history, medicine, and linguistics. The use of generative models, namely mBART, led to gains in in-domain performance (up to 4.9\% in BERTScore, 9.0\% in ROUGE-1, and 12.2\% in F1-score) over three keyphrase extraction baselines for the Russian language. Although the results for cross-domain usage were significantly lower, they still demonstrated the capability to surpass baseline performances in several cases, underscoring the promising potential for further exploration and refinement in this research field.

Title: HAVANA: Hierarchical stochastic neighbor embedding for Accelerated Video ANnotAtions

• Authors: Alexandru Bobe, Jan C. van Gemert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.10641
• Pdf URL: https://arxiv.org/pdf/2409.10641
• Copy Paste: [[2409.10641]] HAVANA: Hierarchical stochastic neighbor embedding for Accelerated Video ANnotAtions(https://arxiv.org/abs/2409.10641)
• Keywords: robust
• Abstract: Video annotation is a critical and time-consuming task in computer vision research and applications. This paper presents a novel annotation pipeline that uses pre-extracted features and dimensionality reduction to accelerate the temporal video annotation process. Our approach uses Hierarchical Stochastic Neighbor Embedding (HSNE) to create a multi-scale representation of video features, allowing annotators to efficiently explore and label large video datasets. We demonstrate significant improvements in annotation effort compared to traditional linear methods, achieving more than a 10x reduction in clicks required for annotating over 12 hours of video. Our experiments on multiple datasets show the effectiveness and robustness of our pipeline across various scenarios. Moreover, we investigate the optimal configuration of HSNE parameters for different datasets. Our work provides a promising direction for scaling up video annotation efforts in the era of video understanding.

Title: CaBaGe: Data-Free Model Extraction using ClAss BAlanced Generator Ensemble

• Authors: Jonathan Rosenthal, Shanchao Liang, Kevin Zhang, Lin Tan
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10643
• Pdf URL: https://arxiv.org/pdf/2409.10643
• Copy Paste: [[2409.10643]] CaBaGe: Data-Free Model Extraction using ClAss BAlanced Generator Ensemble(https://arxiv.org/abs/2409.10643)
• Keywords: attack, extraction, data-free
• Abstract: Machine Learning as a Service (MLaaS) is often provided as a pay-per-query, black-box system to clients. Such a black-box approach not only hinders open replication, validation, and interpretation of model results, but also makes it harder for white-hat researchers to identify vulnerabilities in the MLaaS systems. Model extraction is a promising technique to address these challenges by reverse-engineering black-box models. Since training data is typically unavailable for MLaaS models, this paper focuses on the realistic version of it: data-free model extraction. We propose a data-free model extraction approach, CaBaGe, to achieve higher model extraction accuracy with a small number of queries. Our innovations include (1) a novel experience replay for focusing on difficult training samples; (2) an ensemble of generators for steadily producing diverse synthetic data; and (3) a selective filtering process for querying the victim model with harder, more balanced samples. In addition, we create a more realistic setting, for the first time, where the attacker has no knowledge of the number of classes in the victim training data, and create a solution to learn the number of classes on the fly. Our evaluation shows that CaBaGe outperforms existing techniques on seven datasets -- MNIST, FMNIST, SVHN, CIFAR-10, CIFAR-100, ImageNet-subset, and Tiny ImageNet -- with an accuracy improvement of the extracted models by up to 43.13%. Furthermore, the number of queries required to extract a clone model matching the final accuracy of prior work is reduced by up to 75.7%.

Title: Improving Multi-candidate Speculative Decoding

• Authors: Xiaofan Lu, Yixiao Zeng, Feiyang Ma, Zixu Yu, Marco Levorato
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10644
• Pdf URL: https://arxiv.org/pdf/2409.10644
• Copy Paste: [[2409.10644]] Improving Multi-candidate Speculative Decoding(https://arxiv.org/abs/2409.10644)
• Keywords: large language model
• Abstract: Speculative Decoding (SD) is a technique to accelerate the inference of Large Language Models (LLMs) by using a lower complexity draft model to propose candidate tokens verified by a larger target model. To further improve efficiency, Multi-Candidate Speculative Decoding (MCSD) improves upon this by sampling multiple candidate tokens from the draft model at each step and verifying them in parallel, thus increasing the chances of accepting a token and reducing generation time. Existing MCSD methods rely on the draft model to initialize the multi-candidate sequences and use static length and tree attention structure for draft generation. However, such an approach suffers from the draft and target model's output distribution differences, especially in dynamic generation context. In this work, we introduce an improved version of MCSD that includes a target model initialized multi-candidate process, dynamic sliced topology-aware causal mask for dynamic length adjustment, and decision models to optimize early stopping. Our framework improves the acceptance rate, defined as the ratio of the longest draft sequence length accepted by the target model over the maximum draft sequence length, by a maximum of 164% and gains a maximum of 75% generation speed up over the MCSD baseline. We also conduct an ablation study to evaluate the impact of the decision model.

Title: Benchmarking Secure Sampling Protocols for Differential Privacy

• Authors: Yucheng Fu, Tianhao Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.10667
• Pdf URL: https://arxiv.org/pdf/2409.10667
• Copy Paste: [[2409.10667]] Benchmarking Secure Sampling Protocols for Differential Privacy(https://arxiv.org/abs/2409.10667)
• Keywords: secure, security, privacy, protect, attack, fair
• Abstract: Differential privacy (DP) is widely employed to provide privacy protection for individuals by limiting information leakage from the aggregated data. Two well-known models of DP are the central model and the local model. The former requires a trustworthy server for data aggregation, while the latter requires individuals to add noise, significantly decreasing the utility of aggregated results. Recently, many studies have proposed to achieve DP with Secure Multi-party Computation (MPC) in distributed settings, namely, the distributed model, which has utility comparable to central model while, under specific security assumptions, preventing parties from obtaining others' information. One challenge of realizing DP in distributed model is efficiently sampling noise with MPC. Although many secure sampling methods have been proposed, they have different security assumptions and isolated theoretical analyses. There is a lack of experimental evaluations to measure and compare their performances. We fill this gap by benchmarking existing sampling protocols in MPC and performing comprehensive measurements of their efficiency. First, we present a taxonomy of the underlying techniques of these sampling protocols. Second, we extend widely used distributed noise generation protocols to be resilient against Byzantine attackers. Third, we implement discrete sampling protocols and align their security settings for a fair comparison. We then conduct an extensive evaluation to study their efficiency and utility.

Title: Toward Mitigating Sex Bias in Pilot Trainees' Stress and Fatigue Modeling

• Authors: Rachel Pfeifer, Sudip Vhaduri, Mark Wilson, Julius Keller
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.10676
• Pdf URL: https://arxiv.org/pdf/2409.10676
• Copy Paste: [[2409.10676]] Toward Mitigating Sex Bias in Pilot Trainees' Stress and Fatigue Modeling(https://arxiv.org/abs/2409.10676)
• Keywords: fair
• Abstract: While researchers have been trying to understand the stress and fatigue among pilots, especially pilot trainees, and to develop stress/fatigue models to automate the process of detecting stress/fatigue, they often do not consider biases such as sex in those models. However, in a critical profession like aviation, where the demographic distribution is disproportionately skewed to one sex, it is urgent to mitigate biases for fair and safe model predictions. In this work, we investigate the perceived stress/fatigue of 69 college students, including 40 pilot trainees with around 63% male. We construct models with decision trees first without bias mitigation and then with bias mitigation using a threshold optimizer with demographic parity and equalized odds constraints 30 times with random instances. Using bias mitigation, we achieve improvements of 88.31% (demographic parity difference) and 54.26% (equalized odds difference), which are also found to be statistically significant.

Title: Mitigating Sex Bias in Audio Data-driven COPD and COVID-19 Breathing Pattern Detection Models

• Authors: Rachel Pfeifer, Sudip Vhaduri, James Eric Dietz
• Subjects: cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.10677
• Pdf URL: https://arxiv.org/pdf/2409.10677
• Copy Paste: [[2409.10677]] Mitigating Sex Bias in Audio Data-driven COPD and COVID-19 Breathing Pattern Detection Models(https://arxiv.org/abs/2409.10677)
• Keywords: fair
• Abstract: In the healthcare industry, researchers have been developing machine learning models to automate diagnosing patients with respiratory illnesses based on their breathing patterns. However, these models do not consider the demographic biases, particularly sex bias, that often occur when models are trained with a skewed patient dataset. Hence, it is essential in such an important industry to reduce this bias so that models can make fair diagnoses. In this work, we examine the bias in models used to detect breathing patterns of two major respiratory diseases, i.e., chronic obstructive pulmonary disease (COPD) and COVID-19. Using decision tree models trained with audio recordings of breathing patterns obtained from two open-source datasets consisting of 29 COPD and 680 COVID-19-positive patients, we analyze the effect of sex bias on the models. With a threshold optimizer and two constraints (demographic parity and equalized odds) to mitigate the bias, we witness 81.43% (demographic parity difference) and 71.81% (equalized odds difference) improvements. These findings are statistically significant.

Title: Mitigating Partial Observability in Adaptive Traffic Signal Control with Transformers

• Authors: Xiaoyu Wang, Ayal Taitler, Scott Sanner, Baher Abdulhai
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2409.10693
• Pdf URL: https://arxiv.org/pdf/2409.10693
• Copy Paste: [[2409.10693]] Mitigating Partial Observability in Adaptive Traffic Signal Control with Transformers(https://arxiv.org/abs/2409.10693)
• Keywords: transformer
• Abstract: Efficient traffic signal control is essential for managing urban transportation, minimizing congestion, and improving safety and sustainability. Reinforcement Learning (RL) has emerged as a promising approach to enhancing adaptive traffic signal control (ATSC) systems, allowing controllers to learn optimal policies through interaction with the environment. However, challenges arise due to partial observability (PO) in traffic networks, where agents have limited visibility, hindering effectiveness. This paper presents the integration of Transformer-based controllers into ATSC systems to address PO effectively. We propose strategies to enhance training efficiency and effectiveness, demonstrating improved coordination capabilities in real-world scenarios. The results showcase the Transformer-based model's ability to capture significant information from historical observations, leading to better control policies and improved traffic flow. This study highlights the potential of leveraging the advanced Transformer architecture to enhance urban transportation management.

Title: Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models

• Authors: Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Joao Souza, Suhail Doshi, Daiqing Li
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.10695
• Pdf URL: https://arxiv.org/pdf/2409.10695
• Copy Paste: [[2409.10695]] Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models(https://arxiv.org/abs/2409.10695)
• Keywords: robust, generative, large language model
• Abstract: We introduce Playground v3 (PGv3), our latest text-to-image model that achieves state-of-the-art (SoTA) performance across multiple testing benchmarks, excels in graphic design abilities and introduces new capabilities. Unlike traditional text-to-image generative models that rely on pre-trained language models like T5 or CLIP text encoders, our approach fully integrates Large Language Models (LLMs) with a novel structure that leverages text conditions exclusively from a decoder-only LLM. Additionally, to enhance image captioning quality-we developed an in-house captioner, capable of generating captions with varying levels of detail, enriching the diversity of text structures. We also introduce a new benchmark CapsBench to evaluate detailed image captioning performance. Experimental results demonstrate that PGv3 excels in text prompt adherence, complex reasoning, and accurate text rendering. User preference studies indicate the super-human graphic design ability of our model for common design applications, such as stickers, posters, and logo designs. Furthermore, PGv3 introduces new capabilities, including precise RGB color control and robust multilingual understanding.

Title: CoMamba: Real-time Cooperative Perception Unlocked with State Space Models

• Authors: Jinlong Li, Xinyu Liu, Baolu Li, Runsheng Xu, Jiachen Li, Hongkai Yu, Zhengzhong Tu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.10699
• Pdf URL: https://arxiv.org/pdf/2409.10699
• Copy Paste: [[2409.10699]] CoMamba: Real-time Cooperative Perception Unlocked with State Space Models(https://arxiv.org/abs/2409.10699)
• Keywords: transformer
• Abstract: Cooperative perception systems play a vital role in enhancing the safety and efficiency of vehicular autonomy. Although recent studies have highlighted the efficacy of vehicle-to-everything (V2X) communication techniques in autonomous driving, a significant challenge persists: how to efficiently integrate multiple high-bandwidth features across an expanding network of connected agents such as vehicles and infrastructure. In this paper, we introduce CoMamba, a novel cooperative 3D detection framework designed to leverage state-space models for real-time onboard vehicle perception. Compared to prior state-of-the-art transformer-based models, CoMamba enjoys being a more scalable 3D model using bidirectional state space models, bypassing the quadratic complexity pain-point of attention mechanisms. Through extensive experimentation on V2X/V2V datasets, CoMamba achieves superior performance compared to existing methods while maintaining real-time processing capabilities. The proposed framework not only enhances object detection accuracy but also significantly reduces processing time, making it a promising solution for next-generation cooperative perception systems in intelligent transportation networks.

Title: Self-Attention Limits Working Memory Capacity of Transformer-Based Models

• Authors: Dongyu Gong, Hantao Zhang
• Subjects: cs.CL, cs.AI, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2409.10715
• Pdf URL: https://arxiv.org/pdf/2409.10715
• Copy Paste: [[2409.10715]] Self-Attention Limits Working Memory Capacity of Transformer-Based Models(https://arxiv.org/abs/2409.10715)
• Keywords: interpretability, transformer, large language model
• Abstract: Recent work on Transformer-based large language models (LLMs) has revealed striking limits in their working memory capacity, similar to what has been found in human behavioral studies. Specifically, these models' performance drops significantly on N-back tasks as N increases. However, there is still a lack of mechanistic interpretability as to why this phenomenon would arise. Inspired by the executive attention theory from behavioral sciences, we hypothesize that the self-attention mechanism within Transformer-based models might be responsible for their working memory capacity limits. To test this hypothesis, we train vanilla decoder-only transformers to perform N-back tasks and find that attention scores gradually aggregate to the N-back positions over training, suggesting that the model masters the task by learning a strategy to pay attention to the relationship between the current position and the N-back position. Critically, we find that the total entropy of the attention score matrix increases as N increases, suggesting that the dispersion of attention scores might be the cause of the capacity limit observed in N-back tasks.

Title: Benchmarking VLMs' Reasoning About Persuasive Atypical Images

• Authors: Sina Malakouti, Aysan Aghazadeh, Ashmit Khandelwal, Adriana Kovashka
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.10719
• Pdf URL: https://arxiv.org/pdf/2409.10719
• Copy Paste: [[2409.10719]] Benchmarking VLMs' Reasoning About Persuasive Atypical Images(https://arxiv.org/abs/2409.10719)
• Keywords: large language model
• Abstract: Vision language models (VLMs) have shown strong zero-shot generalization across various tasks, especially when integrated with large language models (LLMs). However, their ability to comprehend rhetorical and persuasive visual media, such as advertisements, remains understudied. Ads often employ atypical imagery, using surprising object juxtapositions to convey shared properties. For example, Fig. 1 (e) shows a beer with a feather-like texture. This requires advanced reasoning to deduce that this atypical representation signifies the beer's lightness. We introduce three novel tasks, Multi-label Atypicality Classification, Atypicality Statement Retrieval, and Aypical Object Recognition, to benchmark VLMs' understanding of atypicality in persuasive images. We evaluate how well VLMs use atypicality to infer an ad's message and test their reasoning abilities by employing semantically challenging negatives. Finally, we pioneer atypicality-aware verbalization by extracting comprehensive image descriptions sensitive to atypical elements. Our findings reveal that: (1) VLMs lack advanced reasoning capabilities compared to LLMs; (2) simple, effective strategies can extract atypicality-aware information, leading to comprehensive image verbalization; (3) atypicality aids persuasive advertisement understanding. Code and data will be made available.

Title: On the effects of similarity metrics in decentralized deep learning under distributional shift

• Authors: Edvin Listo Zec, Tom Hagander, Eric Ihre-Thomason, Sarunas Girdzijauskas
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10720
• Pdf URL: https://arxiv.org/pdf/2409.10720
• Copy Paste: [[2409.10720]] On the effects of similarity metrics in decentralized deep learning under distributional shift(https://arxiv.org/abs/2409.10720)
• Keywords: privacy, robust
• Abstract: Decentralized Learning (DL) enables privacy-preserving collaboration among organizations or users to enhance the performance of local deep learning models. However, model aggregation becomes challenging when client data is heterogeneous, and identifying compatible collaborators without direct data exchange remains a pressing issue. In this paper, we investigate the effectiveness of various similarity metrics in DL for identifying peers for model merging, conducting an empirical analysis across multiple datasets with distribution shifts. Our research provides insights into the performance of these metrics, examining their role in facilitating effective collaboration. By exploring the strengths and limitations of these metrics, we contribute to the development of robust DL methods.

Title: A Missing Data Imputation GAN for Character Sprite Generation

• Authors: Flávio Coutinho, Luiz Chaimowicz
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.10721
• Pdf URL: https://arxiv.org/pdf/2409.10721
• Copy Paste: [[2409.10721]] A Missing Data Imputation GAN for Character Sprite Generation(https://arxiv.org/abs/2409.10721)
• Keywords: generative
• Abstract: Creating and updating pixel art character sprites with many frames spanning different animations and poses takes time and can quickly become repetitive. However, that can be partially automated to allow artists to focus on more creative tasks. In this work, we concentrate on creating pixel art character sprites in a target pose from images of them facing other three directions. We present a novel approach to character generation by framing the problem as a missing data imputation task. Our proposed generative adversarial networks model receives the images of a character in all available domains and produces the image of the missing pose. We evaluated our approach in the scenarios with one, two, and three missing images, achieving similar or better results to the state-of-the-art when more images are available. We also evaluate the impact of the proposed changes to the base architecture.

Title: Depth from Coupled Optical Differentiation

• Authors: Junjie Luo, Yuxuan Liu, Emma Alexander, Qi Guo
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.10725
• Pdf URL: https://arxiv.org/pdf/2409.10725
• Copy Paste: [[2409.10725]] Depth from Coupled Optical Differentiation(https://arxiv.org/abs/2409.10725)
• Keywords: robust
• Abstract: We propose depth from coupled optical differentiation, a low-computation passive-lighting 3D sensing mechanism. It is based on our discovery that per-pixel object distance can be rigorously determined by a coupled pair of optical derivatives of a defocused image using a simple, closed-form relationship. Unlike previous depth-from-defocus (DfD) methods that leverage spatial derivatives of the image to estimate scene depths, the proposed mechanism's use of only optical derivatives makes it significantly more robust to noise. Furthermore, unlike many previous DfD algorithms with requirements on aperture code, this relationship is proved to be universal to a broad range of aperture codes. We build the first 3D sensor based on depth from coupled optical differentiation. Its optical assembly includes a deformable lens and a motorized iris, which enables dynamic adjustments to the optical power and aperture radius. The sensor captures two pairs of images: one pair with a differential change of optical power and the other with a differential change of aperture scale. From the four images, a depth and confidence map can be generated with only 36 floating point operations per output pixel (FLOPOP), more than ten times lower than the previous lowest passive-lighting depth sensing solution to our knowledge. Additionally, the depth map generated by the proposed sensor demonstrates more than twice the working range of previous DfD methods while using significantly lower computation.

Title: Semantics Preserving Emoji Recommendation with Large Language Models

• Authors: Zhongyi Qiu, Kangyi Qiu, Hanjia Lyu, Wei Xiong, Jiebo Luo
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.10760
• Pdf URL: https://arxiv.org/pdf/2409.10760
• Copy Paste: [[2409.10760]] Semantics Preserving Emoji Recommendation with Large Language Models(https://arxiv.org/abs/2409.10760)
• Keywords: large language model
• Abstract: Emojis have become an integral part of digital communication, enriching text by conveying emotions, tone, and intent. Existing emoji recommendation methods are primarily evaluated based on their ability to match the exact emoji a user chooses in the original text. However, they ignore the essence of users' behavior on social media in that each text can correspond to multiple reasonable emojis. To better assess a model's ability to align with such real-world emoji usage, we propose a new semantics preserving evaluation framework for emoji recommendation, which measures a model's ability to recommend emojis that maintain the semantic consistency with the user's text. To evaluate how well a model preserves semantics, we assess whether the predicted affective state, demographic profile, and attitudinal stance of the user remain unchanged. If these attributes are preserved, we consider the recommended emojis to have maintained the original semantics. The advanced abilities of Large Language Models (LLMs) in understanding and generating nuanced, contextually relevant output make them well-suited for handling the complexities of semantics preserving emoji recommendation. To this end, we construct a comprehensive benchmark to systematically assess the performance of six proprietary and open-source LLMs using different prompting techniques on our task. Our experiments demonstrate that GPT-4o outperforms other LLMs, achieving a semantics preservation score of 79.23%. Additionally, we conduct case studies to analyze model biases in downstream classification tasks and evaluate the diversity of the recommended emojis.

Title: Federated Learning for Smart Grid: A Survey on Applications and Potential Vulnerabilities

• Authors: Zikai Zhang, Suman Rath, Jiaohao Xu, Tingsong Xiao
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.10764
• Pdf URL: https://arxiv.org/pdf/2409.10764
• Copy Paste: [[2409.10764]] Federated Learning for Smart Grid: A Survey on Applications and Potential Vulnerabilities(https://arxiv.org/abs/2409.10764)
• Keywords: security, privacy, defense, attack, robust, federate
• Abstract: The Smart Grid (SG) is a critical energy infrastructure that collects real-time electricity usage data to forecast future energy demands using information and communication technologies (ICT). Due to growing concerns about data security and privacy in SGs, federated learning (FL) has emerged as a promising training framework. FL offers a balance between privacy, efficiency, and accuracy in SGs by enabling collaborative model training without sharing private data from IoT devices. In this survey, we thoroughly review recent advancements in designing FL-based SG systems across three stages: generation, transmission and distribution, and consumption. Additionally, we explore potential vulnerabilities that may arise when implementing FL in these stages. Finally, we discuss the gap between state-of-the-art FL research and its practical applications in SGs and propose future research directions. These focus on potential attack and defense strategies for FL-based SG systems and the need to build a robust FL-based SG infrastructure. Unlike traditional surveys that address security issues in centralized machine learning methods for SG systems, this survey specifically examines the applications and security concerns in FL-based SG systems for the first time. Our aim is to inspire further research into applications and improvements in the robustness of FL-based SG systems.

Title: Are Deep Learning Models Robust to Partial Object Occlusion in Visual Recognition Tasks?

• Authors: Kaleb Kassaw, Francesco Luzi, Leslie M. Collins, Jordan M. Malof
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10775
• Pdf URL: https://arxiv.org/pdf/2409.10775
• Copy Paste: [[2409.10775]] Are Deep Learning Models Robust to Partial Object Occlusion in Visual Recognition Tasks?(https://arxiv.org/abs/2409.10775)
• Keywords: robust, transformer, segmentation
• Abstract: Image classification models, including convolutional neural networks (CNNs), perform well on a variety of classification tasks but struggle under conditions of partial occlusion, i.e., conditions in which objects are partially covered from the view of a camera. Methods to improve performance under occlusion, including data augmentation, part-based clustering, and more inherently robust architectures, including Vision Transformer (ViT) models, have, to some extent, been evaluated on their ability to classify objects under partial occlusion. However, evaluations of these methods have largely relied on images containing artificial occlusion, which are typically computer-generated and therefore inexpensive to label. Additionally, methods are rarely compared against each other, and many methods are compared against early, now outdated, deep learning models. We contribute the Image Recognition Under Occlusion (IRUO) dataset, based on the recently developed Occluded Video Instance Segmentation (OVIS) dataset (arXiv:2102.01558). IRUO utilizes real-world and artificially occluded images to test and benchmark leading methods' robustness to partial occlusion in visual recognition tasks. In addition, we contribute the design and results of a human study using images from IRUO that evaluates human classification performance at multiple levels and types of occlusion. We find that modern CNN-based models show improved recognition accuracy on occluded images compared to earlier CNN-based models, and ViT-based models are more accurate than CNN-based models on occluded images, performing only modestly worse than human accuracy. We also find that certain types of occlusion, including diffuse occlusion, where relevant objects are seen through "holes" in occluders such as fences and leaves, can greatly reduce the accuracy of deep recognition models as compared to humans, especially those with CNN backbones.

Title: Physics-Informed Neural Networks with Trust-Region Sequential Quadratic Programming

• Authors: Xiaoran Cheng, Sen Na
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2409.10777
• Pdf URL: https://arxiv.org/pdf/2409.10777
• Copy Paste: [[2409.10777]] Physics-Informed Neural Networks with Trust-Region Sequential Quadratic Programming(https://arxiv.org/abs/2409.10777)
• Keywords: robust
• Abstract: Physics-Informed Neural Networks (PINNs) represent a significant advancement in Scientific Machine Learning (SciML), which integrate physical domain knowledge into an empirical loss function as soft constraints and apply existing machine learning methods to train the model. However, recent research has noted that PINNs may fail to learn relatively complex Partial Differential Equations (PDEs). This paper addresses the failure modes of PINNs by introducing a novel, hard-constrained deep learning method -- trust-region Sequential Quadratic Programming (trSQP-PINN). In contrast to directly training the penalized soft-constrained loss as in PINNs, our method performs a linear-quadratic approximation of the hard-constrained loss, while leveraging the soft-constrained loss to adaptively adjust the trust-region radius. We only trust our model approximations and make updates within the trust region, and such an updating manner can overcome the ill-conditioning issue of PINNs. We also address the computational bottleneck of second-order SQP methods by employing quasi-Newton updates for second-order information, and importantly, we introduce a simple pretraining step to further enhance training efficiency of our method. We demonstrate the effectiveness of trSQP-PINN through extensive experiments. Compared to existing hard-constrained methods for PINNs, such as penalty methods and augmented Lagrangian methods, trSQP-PINN significantly improves the accuracy of the learned PDE solutions, achieving up to 1-3 orders of magnitude lower errors. Additionally, our pretraining step is generally effective for other hard-constrained methods, and experiments have shown the robustness of our method against both problem-specific parameters and algorithm tuning parameters.

Title: Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering

• Authors: Qingru Zhang, Xiaodong Yu, Chandan Singh, Xiaodong Liu, Liyuan Liu, Jianfeng Gao, Tuo Zhao, Dan Roth, Hao Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10790
• Pdf URL: https://arxiv.org/pdf/2409.10790
• Copy Paste: [[2409.10790]] Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering(https://arxiv.org/abs/2409.10790)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across various real-world tasks. However, they often struggle to fully comprehend and effectively utilize their input contexts, resulting in responses that are unfaithful or hallucinated. This difficulty increases for contexts that are long or contain distracting information, which can divert LLMs from fully capturing essential evidence. To address this issue, many works use prompting to help LLMs utilize contextual information more faithfully. For instance, iterative prompting highlights key information in two steps that first ask the LLM to identify important pieces of context and then derive answers accordingly. However, prompting methods are constrained to highlighting key information implicitly in token space, which is often insufficient to fully steer the model's attention. To improve model faithfulness more reliably, we propose AutoPASTA, a method that automatically identifies key contextual information and explicitly highlights it by steering an LLM's attention scores. Like prompting, AutoPASTA is applied at inference time and does not require changing any model parameters. Our experiments on open-book QA demonstrate that AutoPASTA effectively enables models to grasp essential contextual information, leading to substantially improved model faithfulness and performance, e.g., an average improvement of 7.95% for LLAMA3-70B-Instruct. Code will be publicly available at this https URL .

Title: Quantum Machine Learning for Semiconductor Fabrication: Modeling GaN HEMT Contact Process

• Authors: Zeheng Wang, Fangzhou Wang, Liang Li, Zirui Wang, Timothy van der Laan, Ross C. C. Leon, Jing-Kai Huang, Muhammad Usman
• Subjects: cs.LG, cs.ET, quant-ph
• Abstract URL: https://arxiv.org/abs/2409.10803
• Pdf URL: https://arxiv.org/pdf/2409.10803
• Copy Paste: [[2409.10803]] Quantum Machine Learning for Semiconductor Fabrication: Modeling GaN HEMT Contact Process(https://arxiv.org/abs/2409.10803)
• Keywords: robust
• Abstract: This paper pioneers the use of quantum machine learning (QML) for modeling the Ohmic contact process in GaN high-electron-mobility transistors (HEMTs) for the first time. Utilizing data from 159 devices and variational auto-encoder-based augmentation, we developed a quantum kernel-based regressor (QKR) with a 2-level ZZ-feature map. Benchmarking against six classical machine learning (CML) models, our QKR consistently demonstrated the lowest mean absolute error (MAE), mean squared error (MSE), and root mean squared error (RMSE). Repeated statistical analysis confirmed its robustness. Additionally, experiments verified an MAE of 0.314 ohm-mm, underscoring the QKR's superior performance and potential for semiconductor applications, and demonstrating significant advancements over traditional CML methods.

Title: Fast and Post-Quantum Authentication for Real-time Next Generation Networks with Bloom Filter

• Authors: Kiarash Sedghighadikolaei, Attila A Yavuz
• Subjects: cs.CR, cs.NI, eess.SY
• Abstract URL: https://arxiv.org/abs/2409.10813
• Pdf URL: https://arxiv.org/pdf/2409.10813
• Copy Paste: [[2409.10813]] Fast and Post-Quantum Authentication for Real-time Next Generation Networks with Bloom Filter(https://arxiv.org/abs/2409.10813)
• Keywords: secure, security
• Abstract: Large-scale next-generation networked systems like smart grids and vehicular networks facilitate extensive automation and autonomy through real-time communication of sensitive messages. Digital signatures are vital for such applications since they offer scalable broadcast authentication with non-repudiation. Yet, even conventional secure signatures (e.g., ECDSA, RSA) introduce significant cryptographic delays that can disrupt the safety of such delay-aware systems. With the rise of quantum computers breaking conventional intractability problems, these traditional cryptosystems must be replaced with post-quantum (PQ) secure ones. However, PQ-secure signatures are significantly costlier than their conventional counterparts, vastly exacerbating delay hurdles for real-time applications. We propose a new signature called Time Valid Probabilistic Data Structure HORS (TVPD-HORS) that achieves significantly lower end-to-end delay with a tunable PQ-security for real-time applications. We harness special probabilistic data structures as an efficient one-way function at the heart of our novelty, thereby vastly fastening HORS as a primitive for NIST PQ cryptography standards. TVPD-HORS permits tunable and fast processing for varying input sizes via One-hash Bloom Filter, excelling in time valid cases, wherein authentication with shorter security parameters is used for short-lived yet safety-critical messages. We show that TVPD-HORS verification is 2.7x and 5x faster than HORS in high-security and time valid settings, respectively. TVPD-HORS key generation is also faster, with a similar signing speed to HORS. Moreover, TVPD-HORS can increase the speed of HORS variants over a magnitude of time. These features make TVPD-HORS an ideal primitive to raise high-speed time valid versions of PQ-safe standards like XMSS and SPHINCS+, paving the way for real-time authentication of next-generation networks.

Title: AutoCRAT: Automatic Cumulative Reconstruction of Alert Trees

• Authors: Eric Ficke, Raymond M. Bateman, Shouhuai Xu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.10828
• Pdf URL: https://arxiv.org/pdf/2409.10828
• Copy Paste: [[2409.10828]] AutoCRAT: Automatic Cumulative Reconstruction of Alert Trees(https://arxiv.org/abs/2409.10828)
• Keywords: security, attack
• Abstract: When a network is attacked, cyber defenders need to precisely identify which systems (i.e., computers or devices) were compromised and what damage may have been inflicted. This process is sometimes referred to as cyber triage and is an important part of the incident response procedure. Cyber triage is challenging because the impacts of a network breach can be far-reaching with unpredictable consequences. This highlights the importance of automating this process. In this paper we propose AutoCRAT, a system for quantifying the breadth and severity of threats posed by a network exposure, and for prioritizing cyber triage activities during incident response. Specifically, AutoCRAT automatically reconstructs what we call alert trees, which track network security events emanating from, or leading to, a particular computer on the network. We validate the usefulness of AutoCRAT using a real-world dataset. Experimental results show that our prototype system can reconstruct alert trees efficiently and can facilitate data visualization in both incident response and threat intelligence analysis.

Title: ReXErr: Synthesizing Clinically Meaningful Errors in Diagnostic Radiology Reports

• Authors: Vishwanatha M. Rao, Serena Zhang, Julian N. Acosta, Subathra Adithan, Pranav Rajpurkar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10829
• Pdf URL: https://arxiv.org/pdf/2409.10829
• Copy Paste: [[2409.10829]] ReXErr: Synthesizing Clinically Meaningful Errors in Diagnostic Radiology Reports(https://arxiv.org/abs/2409.10829)
• Keywords: large language model
• Abstract: Accurately interpreting medical images and writing radiology reports is a critical but challenging task in healthcare. Both human-written and AI-generated reports can contain errors, ranging from clinical inaccuracies to linguistic mistakes. To address this, we introduce ReXErr, a methodology that leverages Large Language Models to generate representative errors within chest X-ray reports. Working with board-certified radiologists, we developed error categories that capture common mistakes in both human and AI-generated reports. Our approach uses a novel sampling scheme to inject diverse errors while maintaining clinical plausibility. ReXErr demonstrates consistency across error categories and produces errors that closely mimic those found in real-world scenarios. This method has the potential to aid in the development and evaluation of report correction algorithms, potentially enhancing the quality and reliability of radiology reporting.

Title: Machine Learning for Public Good: Predicting Urban Crime Patterns to Enhance Community Safety

• Authors: Sia Gupta, Simeon Sayer
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.10838
• Pdf URL: https://arxiv.org/pdf/2409.10838
• Copy Paste: [[2409.10838]] Machine Learning for Public Good: Predicting Urban Crime Patterns to Enhance Community Safety(https://arxiv.org/abs/2409.10838)
• Keywords: extraction
• Abstract: In recent years, urban safety has become a paramount concern for city planners and law enforcement agencies. Accurate prediction of likely crime occurrences can significantly enhance preventive measures and resource allocation. However, many law enforcement departments lack the tools to analyze and apply advanced AI and ML techniques that can support city planners, watch programs, and safety leaders to take proactive steps towards overall community safety. This paper explores the effectiveness of ML techniques to predict spatial and temporal patterns of crimes in urban areas. Leveraging police dispatch call data from San Jose, CA, the research goal is to achieve a high degree of accuracy in categorizing calls into priority levels particularly for more dangerous situations that require an immediate law enforcement response. This categorization is informed by the time, place, and nature of the call. The research steps include data extraction, preprocessing, feature engineering, exploratory data analysis, implementation, optimization and tuning of different supervised machine learning models and neural networks. The accuracy and precision are examined for different models and features at varying granularity of crime categories and location precision. The results demonstrate that when compared to a variety of other models, Random Forest classification models are most effective in identifying dangerous situations and their corresponding priority levels with high accuracy (Accuracy = 85%, AUC = 0.92) at a local level while ensuring a minimum amount of false negatives. While further research and data gathering is needed to include other social and economic factors, these results provide valuable insights for law enforcement agencies to optimize resources, develop proactive deployment approaches, and adjust response patterns to enhance overall public safety outcomes in an unbiased way.

Title: Implicit Reasoning in Deep Time Series Forecasting

• Authors: Willa Potosnak, Cristian Challu, Mononito Goswami, Michał Wiliński, Nina Żukowska
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10840
• Pdf URL: https://arxiv.org/pdf/2409.10840
• Copy Paste: [[2409.10840]] Implicit Reasoning in Deep Time Series Forecasting(https://arxiv.org/abs/2409.10840)
• Keywords: transformer
• Abstract: Recently, time series foundation models have shown promising zero-shot forecasting performance on time series from a wide range of domains. However, it remains unclear whether their success stems from a true understanding of temporal dynamics or simply from memorizing the training data. While implicit reasoning in language models has been studied, similar evaluations for time series models have been largely unexplored. This work takes an initial step toward assessing the reasoning abilities of deep time series forecasting models. We find that certain linear, MLP-based, and patch-based Transformer models generalize effectively in systematically orchestrated out-of-distribution scenarios, suggesting underexplored reasoning capabilities beyond simple pattern memorization.

Title: BAD: Bidirectional Auto-regressive Diffusion for Text-to-Motion Generation

• Authors: S. Rohollah Hosseyni, Ali Ahmad Rahmani, S. Jamal Seyedmohammadi, Sanaz Seyedin, Arash Mohammadi
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10847
• Pdf URL: https://arxiv.org/pdf/2409.10847
• Copy Paste: [[2409.10847]] BAD: Bidirectional Auto-regressive Diffusion for Text-to-Motion Generation(https://arxiv.org/abs/2409.10847)
• Keywords: diffusion, generative
• Abstract: Autoregressive models excel in modeling sequential dependencies by enforcing causal constraints, yet they struggle to capture complex bidirectional patterns due to their unidirectional nature. In contrast, mask-based models leverage bidirectional context, enabling richer dependency modeling. However, they often assume token independence during prediction, which undermines the modeling of sequential dependencies. Additionally, the corruption of sequences through masking or absorption can introduce unnatural distortions, complicating the learning process. To address these issues, we propose Bidirectional Autoregressive Diffusion (BAD), a novel approach that unifies the strengths of autoregressive and mask-based generative models. BAD utilizes a permutation-based corruption technique that preserves the natural sequence structure while enforcing causal dependencies through randomized ordering, enabling the effective capture of both sequential and bidirectional relationships. Comprehensive experiments show that BAD outperforms autoregressive and mask-based models in text-to-motion generation, suggesting a novel pre-training strategy for sequence modeling. The codebase for BAD is available on this https URL.

Title: 3DFacePolicy: Speech-Driven 3D Facial Animation with Diffusion Policy

• Authors: Xuanmeng Sha, Liyun Zhang, Tomohiro Mashita, Yuki Uranishi
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.10848
• Pdf URL: https://arxiv.org/pdf/2409.10848
• Copy Paste: [[2409.10848]] 3DFacePolicy: Speech-Driven 3D Facial Animation with Diffusion Policy(https://arxiv.org/abs/2409.10848)
• Keywords: diffusion, transformer
• Abstract: Audio-driven 3D facial animation has made immersive progress both in research and application developments. The newest approaches focus on Transformer-based methods and diffusion-based methods, however, there is still gap in the vividness and emotional expression between the generated animation and real human face. To tackle this limitation, we propose 3DFacePolicy, a diffusion policy model for 3D facial animation prediction. This method generates variable and realistic human facial movements by predicting the 3D vertex trajectory on the 3D facial template with diffusion policy instead of facial generation for every frame. It takes audio and vertex states as observations to predict the vertex trajectory and imitate real human facial expressions, which keeps the continuous and natural flow of human emotions. The experiments show that our approach is effective in variable and dynamic facial motion synthesizing.

Title: An Anti-disguise Authentication System Using the First Impression of Avatar in Metaverse

• Authors: Zhenyong Zhang, Kedi Yang, Youliang Tian, Jianfeng Ma
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.10850
• Pdf URL: https://arxiv.org/pdf/2409.10850
• Copy Paste: [[2409.10850]] An Anti-disguise Authentication System Using the First Impression of Avatar in Metaverse(https://arxiv.org/abs/2409.10850)
• Keywords: security, attack
• Abstract: Metaverse is a vast virtual world parallel to the physical world, where the user acts as an avatar to enjoy various services that break through the temporal and spatial limitations of the physical world. Metaverse allows users to create arbitrary digital appearances as their own avatars by which an adversary may disguise his/her avatar to fraud others. In this paper, we propose an anti-disguise authentication method that draws on the idea of the first impression from the physical world to recognize an old friend. Specifically, the first meeting scenario in the metaverse is stored and recalled to help the authentication between avatars. To prevent the adversary from replacing and forging the first impression, we construct a chameleon-based signcryption mechanism and design a ciphertext authentication protocol to ensure the public verifiability of encrypted identities. The security analysis shows that the proposed signcryption mechanism meets not only the security requirement but also the public verifiability. Besides, the ciphertext authentication protocol has the capability of defending against the replacing and forging attacks on the first impression. Extensive experiments show that the proposed avatar authentication system is able to achieve anti-disguise authentication at a low storage consumption on the blockchain.

Title: Adaptive Large Language Models By Layerwise Attention Shortcuts

• Authors: Prateek Verma, Mert Pilanci
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.10870
• Pdf URL: https://arxiv.org/pdf/2409.10870
• Copy Paste: [[2409.10870]] Adaptive Large Language Models By Layerwise Attention Shortcuts(https://arxiv.org/abs/2409.10870)
• Keywords: transformer, large language model
• Abstract: Transformer architectures are the backbone of the modern AI revolution. However, they are based on simply stacking the same blocks in dozens of layers and processing information sequentially from one block to another. In this paper, we propose to challenge this and introduce adaptive computations for LLM-like setups, which allow the final layer to attend to all of the intermediate layers as it deems fit through the attention mechanism, thereby introducing computational \textbf{attention shortcuts}. These shortcuts can thus make the architecture depth and context adaptive. We showcase four different datasets, namely acoustic tokens, natural language, and symbolic music, and we achieve superior performance for GPT-like architecture. We give evidence via attention maps that the models learn complex dependencies across layers that are adaptive in context and depth depending on the input tokens.

Title: American Sign Language to Text Translation using Transformer and Seq2Seq with LSTM

• Authors: Gregorius Guntur Sunardi Putra, Adifa Widyadhani Chanda D'Layla, Dimas Wahono, Riyanarto Sarno, Agus Tri Haryono
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10874
• Pdf URL: https://arxiv.org/pdf/2409.10874
• Copy Paste: [[2409.10874]] American Sign Language to Text Translation using Transformer and Seq2Seq with LSTM(https://arxiv.org/abs/2409.10874)
• Keywords: transformer
• Abstract: Sign language translation is one of the important issues in communication between deaf and hearing people, as it expresses words through hand, body, and mouth movements. American Sign Language is one of the sign languages used, one of which is the alphabetic sign. The development of neural machine translation technology is moving towards sign language translation. Transformer became the state-of-the-art in natural language processing. This study compares the Transformer with the Sequence-to-Sequence (Seq2Seq) model in translating sign language to text. In addition, an experiment was conducted by adding Residual Long Short-Term Memory (ResidualLSTM) in the Transformer. The addition of ResidualLSTM to the Transformer reduces the performance of the Transformer model by 23.37% based on the BLEU Score value. In comparison, the Transformer itself increases the BLEU Score value by 28.14 compared to the Seq2Seq model.

Title: CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization

• Authors: Ziwei Gong, Lin Ai, Harshsaiprasad Deshpande, Alexander Johnson, Emmy Phung, Zehui Wu, Ahmad Emami, Julia Hirschberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10883
• Pdf URL: https://arxiv.org/pdf/2409.10883
• Copy Paste: [[2409.10883]] CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization(https://arxiv.org/abs/2409.10883)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have spurred interest in automatic evaluation methods for summarization, offering a faster, more cost-effective alternative to human evaluation. However, existing methods often fall short when applied to complex tasks like long-context summarizations and dialogue-based meeting summarizations. In this paper, we introduce CREAM (Comparison-Based Reference-Free Elo-Ranked Automatic Evaluation for Meeting Summarization), a novel framework that addresses the unique challenges of evaluating meeting summaries. CREAM leverages a combination of chain-of-thought reasoning and key facts alignment to assess conciseness and completeness of model-generated summaries without requiring reference. By employing an ELO ranking system, our approach provides a robust mechanism for comparing the quality of different models or prompt configurations.

Title: Shaking the Fake: Detecting Deepfake Videos in Real Time via Active Probes

• Authors: Zhixin Xie, Jun Luo
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.10889
• Pdf URL: https://arxiv.org/pdf/2409.10889
• Copy Paste: [[2409.10889]] Shaking the Fake: Detecting Deepfake Videos in Real Time via Active Probes(https://arxiv.org/abs/2409.10889)
• Keywords: generative
• Abstract: Real-time deepfake, a type of generative AI, is capable of "creating" non-existing contents (e.g., swapping one's face with another) in a video. It has been, very unfortunately, misused to produce deepfake videos (during web conferences, video calls, and identity authentication) for malicious purposes, including financial scams and political misinformation. Deepfake detection, as the countermeasure against deepfake, has attracted considerable attention from the academic community, yet existing works typically rely on learning passive features that may perform poorly beyond seen datasets. In this paper, we propose SFake, a new real-time deepfake detection method that innovatively exploits deepfake models' inability to adapt to physical interference. Specifically, SFake actively sends probes to trigger mechanical vibrations on the smartphone, resulting in the controllable feature on the footage. Consequently, SFake determines whether the face is swapped by deepfake based on the consistency of the facial area with the probe pattern. We implement SFake, evaluate its effectiveness on a self-built dataset, and compare it with six other detection methods. The results show that SFake outperforms other detection methods with higher detection accuracy, faster process speed, and lower memory consumption.

Title: Technical Upgrades to and Enhancements of a System Vulnerability Analysis Tool Based on the Blackboard Architecture

• Authors: Matthew Tassava, Cameron Kolodjski, Jeremy Straub
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.10892
• Pdf URL: https://arxiv.org/pdf/2409.10892
• Copy Paste: [[2409.10892]] Technical Upgrades to and Enhancements of a System Vulnerability Analysis Tool Based on the Blackboard Architecture(https://arxiv.org/abs/2409.10892)
• Keywords: security, attack
• Abstract: A system vulnerability analysis technique (SVAT) for the analysis of complex mission critical systems (CMCS) that cannot be taken offline or subjected to the risks posed by traditional penetration testing was previously developed. This system uses path-based analysis of vulnerabilities to identify potential threats to system security. Generalization logic building on the Blackboard Architecture's rule-fact paradigm was implemented in this system, the software for operation and network attack results review (SONARR). This paper presents an overview of additional functionality that has been added to this tool and the experimentation that was conducted to analyze their efficacy and the performance benefits of the new in-memory processing capabilities of the SONARR algorithm. The results of the performance tests and their relation to networks' architecture are discussed. The paper concludes with a discussion of avenues of future work, including the implementation of multithreading, additional analysis metrics like confidentiality, integrity, and availability, and improved heuristic development.

Title: Enhancing Security Testing Software for Systems that Cannot be Subjected to the Risks of Penetration Testing Through the Incorporation of Multi-threading and and Other Capabilities

• Authors: Matthew Tassava, Cameron Kolodjski, Jordan Milbrath, Jeremy Straub
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.10893
• Pdf URL: https://arxiv.org/pdf/2409.10893
• Copy Paste: [[2409.10893]] Enhancing Security Testing Software for Systems that Cannot be Subjected to the Risks of Penetration Testing Through the Incorporation of Multi-threading and and Other Capabilities(https://arxiv.org/abs/2409.10893)
• Keywords: security, attack
• Abstract: The development of a system vulnerability analysis tool (SVAT) for complex mission critical systems (CMCS) produced the software for operation and network attack results review (SONARR). This software builds upon the Blackboard Architecture and uses its a rule-fact logic to assess model networks to identify potential pathways that an attacker might take through them via the exploitation of vulnerabilities within the network. The SONARR objects and algorithm were developed previously; however, performance was insufficient for analyzing large networks. This paper describes and analyzes the performance of a multi-threaded SONARR algorithm and other enhancements which were developed to increase SONARR's performance and facilitate the analysis of large networks.

Title: AutoSpec: Automated Generation of Neural Network Specifications

• Authors: Shuowei Jin, Francis Y. Yan, Cheng Tan, Anuj Kalia, Xenofon Foukas, Z. Morley Mao
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.10897
• Pdf URL: https://arxiv.org/pdf/2409.10897
• Copy Paste: [[2409.10897]] AutoSpec: Automated Generation of Neural Network Specifications(https://arxiv.org/abs/2409.10897)
• Keywords: robust
• Abstract: The increasing adoption of neural networks in learning-augmented systems highlights the importance of model safety and robustness, particularly in safety-critical domains. Despite progress in the formal verification of neural networks, current practices require users to manually define model specifications -- properties that dictate expected model behavior in various scenarios. This manual process, however, is prone to human error, limited in scope, and time-consuming. In this paper, we introduce AutoSpec, the first framework to automatically generate comprehensive and accurate specifications for neural networks in learning-augmented systems. We also propose the first set of metrics for assessing the accuracy and coverage of model specifications, establishing a benchmark for future comparisons. Our evaluation across four distinct applications shows that AutoSpec outperforms human-defined specifications as well as two baseline approaches introduced in this study.

Title: Attention-Seeker: Dynamic Self-Attention Scoring for Unsupervised Keyphrase Extraction

• Authors: Erwin D. López Z., Cheng Tang, Atsushi Shimada
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.10907
• Pdf URL: https://arxiv.org/pdf/2409.10907
• Copy Paste: [[2409.10907]] Attention-Seeker: Dynamic Self-Attention Scoring for Unsupervised Keyphrase Extraction(https://arxiv.org/abs/2409.10907)
• Keywords: extraction, large language model
• Abstract: This paper proposes Attention-Seeker, an unsupervised keyphrase extraction method that leverages self-attention maps from a Large Language Model to estimate the importance of candidate phrases. Our approach identifies specific components - such as layers, heads, and attention vectors - where the model pays significant attention to the key topics of the text. The attention weights provided by these components are then used to score the candidate phrases. Unlike previous models that require manual tuning of parameters (e.g., selection of heads, prompts, hyperparameters), Attention-Seeker dynamically adapts to the input text without any manual adjustments, enhancing its practical applicability. We evaluate Attention-Seeker on four publicly available datasets: Inspec, SemEval2010, SemEval2017, and Krapivin. Our results demonstrate that, even without parameter tuning, Attention-Seeker outperforms most baseline models, achieving state-of-the-art performance on three out of four datasets, particularly excelling in extracting keyphrases from long documents.

Title: Anti-ESIA: Analyzing and Mitigating Impacts of Electromagnetic Signal Injection Attacks

• Authors: Denglin Kang, Youqian Zhang, Wai Cheong Tam, Eugene Y. Fu
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.10922
• Pdf URL: https://arxiv.org/pdf/2409.10922
• Copy Paste: [[2409.10922]] Anti-ESIA: Analyzing and Mitigating Impacts of Electromagnetic Signal Injection Attacks(https://arxiv.org/abs/2409.10922)
• Keywords: security, attack
• Abstract: Cameras are integral components of many critical intelligent systems. However, a growing threat, known as Electromagnetic Signal Injection Attacks (ESIA), poses a significant risk to these systems, where ESIA enables attackers to remotely manipulate images captured by cameras, potentially leading to malicious actions and catastrophic consequences. Despite the severity of this threat, the underlying reasons for ESIA's effectiveness remain poorly understood, and effective countermeasures are lacking. This paper aims to address these gaps by investigating ESIA from two distinct aspects: pixel loss and color strips. By analyzing these aspects separately on image classification tasks, we gain a deeper understanding of how ESIA can compromise intelligent systems. Additionally, we explore a lightweight solution to mitigate the effects of ESIA while acknowledging its limitations. Our findings provide valuable insights for future research and development in the field of camera security and intelligent systems.

Title: HGSLoc: 3DGS-based Heuristic Camera Pose Refinement

• Authors: Zhongyan Niu, Zhen Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.10925
• Pdf URL: https://arxiv.org/pdf/2409.10925
• Copy Paste: [[2409.10925]] HGSLoc: 3DGS-based Heuristic Camera Pose Refinement(https://arxiv.org/abs/2409.10925)
• Keywords: robust
• Abstract: Visual localization refers to the process of determining camera poses and orientation within a known scene representation. This task is often complicated by factors such as illumination changes and variations in viewing angles. In this paper, we propose HGSLoc, a novel lightweight, plug and-play pose optimization framework, which integrates 3D reconstruction with a heuristic refinement strategy to achieve higher pose estimation accuracy. Specifically, we introduce an explicit geometric map for 3D representation and high-fidelity rendering, allowing the generation of high-quality synthesized views to support accurate visual localization. Our method demonstrates a faster rendering speed and higher localization accuracy compared to NeRF-based neural rendering localization approaches. We introduce a heuristic refinement strategy, its efficient optimization capability can quickly locate the target node, while we set the step-level optimization step to enhance the pose accuracy in the scenarios with small errors. With carefully designed heuristic functions, it offers efficient optimization capabilities, enabling rapid error reduction in rough localization estimations. Our method mitigates the dependence on complex neural network models while demonstrating improved robustness against noise and higher localization accuracy in challenging environments, as compared to neural network joint optimization strategies. The optimization framework proposed in this paper introduces novel approaches to visual localization by integrating the advantages of 3D reconstruction and heuristic refinement strategy, which demonstrates strong performance across multiple benchmark datasets, including 7Scenes and DB dataset.

Title: Propulsion: Steering LLM with Tiny Fine-Tuning

• Authors: Md Kowsher, Nusrat Jahan Prottasha, Prakash Bhat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10927
• Pdf URL: https://arxiv.org/pdf/2409.10927
• Copy Paste: [[2409.10927]] Propulsion: Steering LLM with Tiny Fine-Tuning(https://arxiv.org/abs/2409.10927)
• Keywords: large language model
• Abstract: The rapid advancements in Large Language Models (LLMs) have revolutionized natural language processing (NLP) and related fields. However, fine-tuning these models for specific tasks remains computationally expensive and risks degrading pre-learned features. To address these challenges, we propose Propulsion, a novel parameter efficient fine-tuning (PEFT) method designed to optimize task-specific performance while drastically reducing computational overhead. Inspired by the concept of controlled adjustments in physical motion, Propulsion selectively re-scales specific dimensions of a pre-trained model, guiding output predictions toward task objectives without modifying the model's parameters. By introducing lightweight, trainable Propulsion parameters at the pre-trained layer, we minimize the number of parameters updated during fine-tuning, preventing overfitting or overwriting of existing knowledge. Our theoretical analysis, supported by Neural Tangent Kernel (NTK) theory, shows that Propulsion approximates the performance of full fine-tuning with far fewer trainable parameters. Empirically, Propulsion reduces the parameter count from 355.3 million to just 0.086 million, achieving over a 10x reduction compared to standard approaches like LoRA while maintaining competitive performance across benchmarks.

Title: An Enhanced Online Certificate Status Protocol for Public Key Infrastructure with Smart Grid and Energy Storage System

• Authors: Hong-Sheng Huang, Cheng-Che Chuang, Jhih-Zen Shih, Hsuan-Tung Chen, Hung-Min Sun
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.10929
• Pdf URL: https://arxiv.org/pdf/2409.10929
• Copy Paste: [[2409.10929]] An Enhanced Online Certificate Status Protocol for Public Key Infrastructure with Smart Grid and Energy Storage System(https://arxiv.org/abs/2409.10929)
• Keywords: secure, security, protect, robust
• Abstract: The efficiency of checking certificate status is one of the key indicators in the public key infrastructure (PKI). This prompted researchers to design the Online Certificate Status Protocol (OCSP) standard, defined in RFC 6960, to guide developers in implementing OCSP components. However, as the environment increasingly relies on PKI for identity authentication, it is essential to protect the communication between clients and servers from rogue elements. This can be achieved by using SSL/TLS techniques to establish a secure channel, allowing Certificate Authorities (CAs) to safely transfer certificate status information. In this work, we introduce the OCSP Stapling approach to optimize OCSP query costs in our smart grid environment. This approach reduces the number of queries from the Device Language Message Specification (DLMS) server to the OCSP server. Our experimental results show that OCSP stapling increases both efficiency and security, creating a more robust architecture for the smart grid.

Title: Early Detection of Coronary Heart Disease Using Hybrid Quantum Machine Learning Approach

• Authors: Mehroush Banday, Sherin Zafar, Parul Agarwal, M Afshar Alam, Abubeker K M
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10932
• Pdf URL: https://arxiv.org/pdf/2409.10932
• Copy Paste: [[2409.10932]] Early Detection of Coronary Heart Disease Using Hybrid Quantum Machine Learning Approach(https://arxiv.org/abs/2409.10932)
• Keywords: robust
• Abstract: Coronary heart disease (CHD) is a severe cardiac disease, and hence, its early diagnosis is essential as it improves treatment results and saves money on medical care. The prevailing development of quantum computing and machine learning (ML) technologies may bring practical improvement to the performance of CHD diagnosis. Quantum machine learning (QML) is receiving tremendous interest in various disciplines due to its higher performance and capabilities. A quantum leap in the healthcare industry will increase processing power and optimise multiple models. Techniques for QML have the potential to forecast cardiac disease and help in early detection. To predict the risk of coronary heart disease, a hybrid approach utilizing an ensemble machine learning model based on QML classifiers is presented in this paper. Our approach, with its unique ability to address multidimensional healthcare data, reassures the method's robustness by fusing quantum and classical ML algorithms in a multi-step inferential framework. The marked rise in heart disease and death rates impacts worldwide human health and the global economy. Reducing cardiac morbidity and mortality requires early detection of heart disease. In this research, a hybrid approach utilizes techniques with quantum computing capabilities to tackle complex problems that are not amenable to conventional machine learning algorithms and to minimize computational expenses. The proposed method has been developed in the Raspberry Pi 5 Graphics Processing Unit (GPU) platform and tested on a broad dataset that integrates clinical and imaging data from patients suffering from CHD and healthy controls. Compared to classical machine learning models, the accuracy, sensitivity, F1 score, and specificity of the proposed hybrid QML model used with CHD are manifold higher.

Title: Optimizing TinyML: The Impact of Reduced Data Acquisition Rates for Time Series Classification on Microcontrollers

• Authors: Riya Samanta, Bidyut Saha, Soumya K. Ghosh, Ram Babu Roy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10942
• Pdf URL: https://arxiv.org/pdf/2409.10942
• Copy Paste: [[2409.10942]] Optimizing TinyML: The Impact of Reduced Data Acquisition Rates for Time Series Classification on Microcontrollers(https://arxiv.org/abs/2409.10942)
• Keywords: privacy
• Abstract: Tiny Machine Learning (TinyML) enables efficient, lowcost, and privacy preserving machine learning inference directly on microcontroller units (MCUs) connected to sensors. Optimizing models for these constrained environments is crucial. This paper investigates how reducing data acquisition rates affects TinyML models for time series classification, focusing on resource-constrained, battery operated IoT devices. By lowering data sampling frequency, we aim to reduce computational demands RAM usage, energy consumption, latency, and MAC operations by approximately fourfold while maintaining similar classification accuracies. Our experiments with six benchmark datasets (UCIHAR, WISDM, PAMAP2, MHEALTH, MITBIH, and PTB) showed that reducing data acquisition rates significantly cut energy consumption and computational load, with minimal accuracy loss. For example, a 75\% reduction in acquisition rate for MITBIH and PTB datasets led to a 60\% decrease in RAM usage, 75\% reduction in MAC operations, 74\% decrease in latency, and 70\% reduction in energy consumption, without accuracy loss. These results offer valuable insights for deploying efficient TinyML models in constrained environments.

Title: Contrasformer: A Brain Network Contrastive Transformer for Neurodegenerative Condition Identification

• Authors: Jiaxing Xu, Kai He, Mengcheng Lan, Qingtian Bian, Wei Li, Tieying Li, Yiping Ke, Miao Qiao
• Subjects: cs.LG, cs.AI, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2409.10944
• Pdf URL: https://arxiv.org/pdf/2409.10944
• Copy Paste: [[2409.10944]] Contrasformer: A Brain Network Contrastive Transformer for Neurodegenerative Condition Identification(https://arxiv.org/abs/2409.10944)
• Keywords: interpretability, transformer, generative
• Abstract: Understanding neurological disorder is a fundamental problem in neuroscience, which often requires the analysis of brain networks derived from functional magnetic resonance imaging (fMRI) data. Despite the prevalence of Graph Neural Networks (GNNs) and Graph Transformers in various domains, applying them to brain networks faces challenges. Specifically, the datasets are severely impacted by the noises caused by distribution shifts across sub-populations and the neglect of node identities, both obstruct the identification of disease-specific patterns. To tackle these challenges, we propose Contrasformer, a novel contrastive brain network Transformer. It generates a prior-knowledge-enhanced contrast graph to address the distribution shifts across sub-populations by a two-stream attention mechanism. A cross attention with identity embedding highlights the identity of nodes, and three auxiliary losses ensure group consistency. Evaluated on 4 functional brain network datasets over 4 different diseases, Contrasformer outperforms the state-of-the-art methods for brain networks by achieving up to 10.8\% improvement in accuracy, which demonstrates its efficacy in neurological disorder identification. Case studies illustrate its interpretability, especially in the context of neuroscience. This paper provides a solution for analyzing brain networks, offering valuable insights into neurological disorders. Our code is available at \url{this https URL}.

Title: Fair Anomaly Detection For Imbalanced Groups

• Authors: Ziwei Wu, Lecheng Zheng, Yuancheng Yu, Ruizhong Qiu, John Birge, Jingrui He
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10951
• Pdf URL: https://arxiv.org/pdf/2409.10951
• Copy Paste: [[2409.10951]] Fair Anomaly Detection For Imbalanced Groups(https://arxiv.org/abs/2409.10951)
• Keywords: security, protect, fair
• Abstract: Anomaly detection (AD) has been widely studied for decades in many real-world applications, including fraud detection in finance, and intrusion detection for cybersecurity, etc. Due to the imbalanced nature between protected and unprotected groups and the imbalanced distributions of normal examples and anomalies, the learning objectives of most existing anomaly detection methods tend to solely concentrate on the dominating unprotected group. Thus, it has been recognized by many researchers about the significance of ensuring model fairness in anomaly detection. However, the existing fair anomaly detection methods tend to erroneously label most normal examples from the protected group as anomalies in the imbalanced scenario where the unprotected group is more abundant than the protected group. This phenomenon is caused by the improper design of learning objectives, which statistically focus on learning the frequent patterns (i.e., the unprotected group) while overlooking the under-represented patterns (i.e., the protected group). To address these issues, we propose FairAD, a fairness-aware anomaly detection method targeting the imbalanced scenario. It consists of a fairness-aware contrastive learning module and a rebalancing autoencoder module to ensure fairness and handle the imbalanced data issue, respectively. Moreover, we provide the theoretical analysis that shows our proposed contrastive learning regularization guarantees group fairness. Empirical studies demonstrate the effectiveness and efficiency of FairAD across multiple real-world datasets.

Title: Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Style

• Authors: Yuepei Li, Kang Zhou, Qiao Qiao, Bach Nguyen, Qing Wang, Qi Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10955
• Pdf URL: https://arxiv.org/pdf/2409.10955
• Copy Paste: [[2409.10955]] Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Style(https://arxiv.org/abs/2409.10955)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) improves Large Language Models (LLMs) by incorporating external information into the response generation process. However, how context-faithful LLMs are and what factors influence LLMs' context-faithfulness remain largely unexplored. In this study, we investigate the impact of memory strength and evidence presentation on LLMs' receptiveness to external evidence. We introduce a method to quantify the memory strength of LLMs by measuring the divergence in LLMs' responses to different paraphrases of the same question, which is not considered by previous works. We also generate evidence in various styles to evaluate the effects of evidence in different styles. Two datasets are used for evaluation: Natural Questions (NQ) with popular questions and popQA featuring long-tail questions. Our results show that for questions with high memory strength, LLMs are more likely to rely on internal memory, particularly for larger LLMs such as GPT-4. On the other hand, presenting paraphrased evidence significantly increases LLMs' receptiveness compared to simple repetition or adding details.

Title: Cross-lingual transfer of multilingual models on low resource African Languages

• Authors: Harish Thangaraj, Ananya Chenat, Jaskaran Singh Walia, Vukosi Marivate
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10965
• Pdf URL: https://arxiv.org/pdf/2409.10965
• Copy Paste: [[2409.10965]] Cross-lingual transfer of multilingual models on low resource African Languages(https://arxiv.org/abs/2409.10965)
• Keywords: transformer
• Abstract: Large multilingual models have significantly advanced natural language processing (NLP) research. However, their high resource demands and potential biases from diverse data sources have raised concerns about their effectiveness across low-resource languages. In contrast, monolingual models, trained on a single language, may better capture the nuances of the target language, potentially providing more accurate results. This study benchmarks the cross-lingual transfer capabilities from a high-resource language to a low-resource language for both, monolingual and multilingual models, focusing on Kinyarwanda and Kirundi, two Bantu languages. We evaluate the performance of transformer based architectures like Multilingual BERT (mBERT), AfriBERT, and BantuBERTa against neural-based architectures such as BiGRU, CNN, and char-CNN. The models were trained on Kinyarwanda and tested on Kirundi, with fine-tuning applied to assess the extent of performance improvement and catastrophic forgetting. AfriBERT achieved the highest cross-lingual accuracy of 88.3% after fine-tuning, while BiGRU emerged as the best-performing neural model with 83.3% accuracy. We also analyze the degree of forgetting in the original language post-fine-tuning. While monolingual models remain competitive, this study highlights that multilingual models offer strong cross-lingual transfer capabilities in resource limited settings.

Title: Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs

• Authors: Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, Benyou Wang
• Subjects: cs.CL, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.10994
• Pdf URL: https://arxiv.org/pdf/2409.10994
• Copy Paste: [[2409.10994]] Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs(https://arxiv.org/abs/2409.10994)
• Keywords: large language model
• Abstract: The rapid advancement of Multimodal Large Language Models (MLLMs) has led to remarkable performances across various domains. However, this progress is accompanied by a substantial surge in the resource consumption of these models. We address this pressing issue by introducing a new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance. Inspired by human attention patterns in Visual Question Answering (VQA) tasks, TRIM presents a fresh perspective on the selection and reduction of image tokens. The TRIM method has been extensively tested across 12 datasets, and the results demonstrate a significant reduction in computational overhead while maintaining a consistent level of performance. This research marks a critical stride in efficient MLLM development, promoting greater accessibility and sustainability of high-performing models.

Title: GINTRIP: Interpretable Temporal Graph Regression using Information bottleneck and Prototype-based method

• Authors: Ali Royat, Seyed Mohamad Moghadas, Lesley De Cruz, Adrian Munteanu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10996
• Pdf URL: https://arxiv.org/pdf/2409.10996
• Copy Paste: [[2409.10996]] GINTRIP: Interpretable Temporal Graph Regression using Information bottleneck and Prototype-based method(https://arxiv.org/abs/2409.10996)
• Keywords: interpretability
• Abstract: Deep neural networks (DNNs) have demonstrated remarkable performance across various domains, yet their application to temporal graph regression tasks faces significant challenges regarding interpretability. This critical issue, rooted in the inherent complexity of both DNNs and underlying spatio-temporal patterns in the graph, calls for innovative solutions. While interpretability concerns in Graph Neural Networks (GNNs) mirror those of DNNs, to the best of our knowledge, no notable work has addressed the interpretability of temporal GNNs using a combination of Information Bottleneck (IB) principles and prototype-based methods. Our research introduces a novel approach that uniquely integrates these techniques to enhance the interpretability of temporal graph regression models. The key contributions of our work are threefold: We introduce the \underline{G}raph \underline{IN}terpretability in \underline{T}emporal \underline{R}egression task using \underline{I}nformation bottleneck and \underline{P}rototype (GINTRIP) framework, the first combined application of IB and prototype-based methods for interpretable temporal graph tasks. We derive a novel theoretical bound on mutual information (MI), extending the applicability of IB principles to graph regression tasks. We incorporate an unsupervised auxiliary classification head, fostering multi-task learning and diverse concept representation, which enhances the model bottleneck's interpretability. Our model is evaluated on real-world traffic datasets, outperforming existing methods in both forecasting accuracy and interpretability-related metrics.

Title: Contextual Breach: Assessing the Robustness of Transformer-based QA Models

• Authors: Asir Saadat, Nahian Ibn Asad, Md Farhan Ishmam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10997
• Pdf URL: https://arxiv.org/pdf/2409.10997
• Copy Paste: [[2409.10997]] Contextual Breach: Assessing the Robustness of Transformer-based QA Models(https://arxiv.org/abs/2409.10997)
• Keywords: robust, transformer
• Abstract: Contextual question-answering models are susceptible to adversarial perturbations to input context, commonly observed in real-world scenarios. These adversarial noises are designed to degrade the performance of the model by distorting the textual input. We introduce a unique dataset that incorporates seven distinct types of adversarial noise into the context, each applied at five different intensity levels on the SQuAD dataset. To quantify the robustness, we utilize robustness metrics providing a standardized measure for assessing model performance across varying noise types and levels. Experiments on transformer-based question-answering models reveal robustness vulnerabilities and important insights into the model's performance in realistic textual input.

Title: Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models

• Authors: Potsawee Manakul, Guangzhi Sun, Warit Sirichotedumrong, Kasima Tharnpipitchai, Kunat Pipatanakul
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.10999
• Pdf URL: https://arxiv.org/pdf/2409.10999
• Copy Paste: [[2409.10999]] Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models(https://arxiv.org/abs/2409.10999)
• Keywords: large language model
• Abstract: Audio language models can understand audio inputs and perform a range of audio-related tasks based on instructions, such as speech recognition and audio captioning, where the instructions are usually textual prompts. Audio language models are mostly initialized from pre-trained audio encoders and large language models (LLMs). Although these pre-trained components were developed to support multiple languages, audio-language models are trained predominantly on English data, which may limit their usability to only English instructions or English speech inputs. First, this paper examines the performance of existing audio language models in an underserved language using Thai as an example. This paper demonstrates that, despite being built on multilingual backbones, audio language models do not exhibit cross-lingual emergent abilities to low-resource languages. Second, this paper studies data mixture for developing audio language models that are optimized for a target language as well as English. In addition. this paper integrates audio comprehension and speech instruction-following capabilities into a single unified model. Our experiments provide insights into data mixture for enhancing instruction-following capabilities in both a low-resource language and English. Our model, Typhoon-Audio, outperforms existing open-source audio language models by a considerable margin, and it is comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai languages.

Title: MM2Latent: Text-to-facial image generation and editing in GANs with multimodal assistance

• Authors: Debin Meng, Christos Tzelepis, Ioannis Patras, Georgios Tzimiropoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11010
• Pdf URL: https://arxiv.org/pdf/2409.11010
• Copy Paste: [[2409.11010]] MM2Latent: Text-to-facial image generation and editing in GANs with multimodal assistance(https://arxiv.org/abs/2409.11010)
• Keywords: diffusion
• Abstract: Generating human portraits is a hot topic in the image generation area, e.g. mask-to-face generation and text-to-face generation. However, these unimodal generation methods lack controllability in image generation. Controllability can be enhanced by exploring the advantages and complementarities of various modalities. For instance, we can utilize the advantages of text in controlling diverse attributes and masks in controlling spatial locations. Current state-of-the-art methods in multimodal generation face limitations due to their reliance on extensive hyperparameters, manual operations during the inference stage, substantial computational demands during training and inference, or inability to edit real images. In this paper, we propose a practical framework - MM2Latent - for multimodal image generation and editing. We use StyleGAN2 as our image generator, FaRL for text encoding, and train an autoencoders for spatial modalities like mask, sketch and 3DMM. We propose a strategy that involves training a mapping network to map the multimodal input into the w latent space of StyleGAN. The proposed framework 1) eliminates hyperparameters and manual operations in the inference stage, 2) ensures fast inference speeds, and 3) enables the editing of real images. Extensive experiments demonstrate that our method exhibits superior performance in multimodal image generation, surpassing recent GAN- and diffusion-based methods. Also, it proves effective in multimodal image editing and is faster than GAN- and diffusion-based methods. We make the code publicly available at: this https URL

Title: Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation

• Authors: Rui Yu, Runkai Zhao, Jiagen Li, Qingsong Zhao, Songhao Zhu, HuaiCheng Yan, Meng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11018
• Pdf URL: https://arxiv.org/pdf/2409.11018
• Copy Paste: [[2409.11018]] Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation(https://arxiv.org/abs/2409.11018)
• Keywords: robust, extraction, transformer
• Abstract: The LiDAR-based 3D object detector that strikes a balance between accuracy and speed is crucial for achieving real-time perception in autonomous driving and robotic navigation systems. To enhance the accuracy of point cloud detection, integrating global context for visual understanding improves the point clouds ability to grasp overall spatial information. However, many existing LiDAR detection models depend on intricate feature transformation and extraction processes, leading to poor real-time performance and high resource consumption, which limits their practical effectiveness. In this work, we propose a Faster LiDAR 3D object detection framework, called FASD, which implements heterogeneous model distillation by adaptively uniform cross-model voxel features. We aim to distill the transformer's capacity for high-performance sequence modeling into Mamba models with low FLOPs, achieving a significant improvement in accuracy through knowledge transfer. Specifically, Dynamic Voxel Group and Adaptive Attention strategies are integrated into the sparse backbone, creating a robust teacher model with scale-adaptive attention for effective global visual context modeling. Following feature alignment with the Adapter, we transfer knowledge from the Transformer to the Mamba through latent space feature supervision and span-head distillation, resulting in improved performance and an efficient student model. We evaluated the framework on the Waymo and nuScenes datasets, achieving a 4x reduction in resource consumption and a 1-2\% performance improvement over the current SoTA methods.

Title: GEIC: Universal and Multilingual Named Entity Recognition with Large Language Models

• Authors: Hanjun Luo, Yibing Jin, Xuecheng Liu, Tong Shang, Ruizhe Chen, Zuozhu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11022
• Pdf URL: https://arxiv.org/pdf/2409.11022
• Copy Paste: [[2409.11022]] GEIC: Universal and Multilingual Named Entity Recognition with Large Language Models(https://arxiv.org/abs/2409.11022)
• Keywords: extraction, large language model
• Abstract: Large Language Models (LLMs) have supplanted traditional methods in numerous natural language processing tasks. Nonetheless, in Named Entity Recognition (NER), existing LLM-based methods underperform compared to baselines and require significantly more computational resources, limiting their application. In this paper, we introduce the task of generation-based extraction and in-context classification (GEIC), designed to leverage LLMs' prior knowledge and self-attention mechanisms for NER tasks. We then propose CascadeNER, a universal and multilingual GEIC framework for few-shot and zero-shot NER. CascadeNER employs model cascading to utilize two small-parameter LLMs to extract and classify independently, reducing resource consumption while enhancing accuracy. We also introduce AnythingNER, the first NER dataset specifically designed for LLMs, including 8 languages, 155 entity types and a novel dynamic categorization system. Experiments show that CascadeNER achieves state-of-the-art performance on low-resource and fine-grained scenarios, including CrossNER and FewNERD. Our work is openly accessible.

Title: D2Vformer: A Flexible Time Series Prediction Model Based on Time Position Embedding

• Authors: Xiaobao Song, Hao Wang, Liwei Deng, Yuxin He, Wenming Cao, Chi-Sing Leungc
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11024
• Pdf URL: https://arxiv.org/pdf/2409.11024
• Copy Paste: [[2409.11024]] D2Vformer: A Flexible Time Series Prediction Model Based on Time Position Embedding(https://arxiv.org/abs/2409.11024)
• Keywords: transformer
• Abstract: Time position embeddings capture the positional information of time steps, often serving as auxiliary inputs to enhance the predictive capabilities of time series models. However, existing models exhibit limitations in capturing intricate time positional information and effectively utilizing these embeddings. To address these limitations, this paper proposes a novel model called D2Vformer. Unlike typical prediction methods that rely on RNNs or Transformers, this approach can directly handle scenarios where the predicted sequence is not adjacent to the input sequence or where its length dynamically changes. In comparison to conventional methods, D2Vformer undoubtedly saves a significant amount of training resources. In D2Vformer, the Date2Vec module uses the timestamp information and feature sequences to generate time position embeddings. Afterward, D2Vformer introduces a new fusion block that utilizes an attention mechanism to explore the similarity in time positions between the embeddings of the input sequence and the predicted sequence, thereby generating predictions based on this similarity. Through extensive experiments on six datasets, we demonstrate that Date2Vec outperforms other time position embedding methods, and D2Vformer surpasses state-of-the-art methods in both fixed-length and variable-length prediction tasks.

Title: Prompt Obfuscation for Large Language Models

• Authors: David Pape, Thorsten Eisenhofer, Lea Schönherr
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11026
• Pdf URL: https://arxiv.org/pdf/2409.11026
• Copy Paste: [[2409.11026]] Prompt Obfuscation for Large Language Models(https://arxiv.org/abs/2409.11026)
• Keywords: protect, attack, steal, extraction, large language model
• Abstract: System prompts that include detailed instructions to describe the task performed by the underlying large language model (LLM) can easily transform foundation models into tools and services with minimal overhead. Because of their crucial impact on the utility, they are often considered intellectual property, similar to the code of a software product. However, extracting system prompts is easily possible by using prompt injection. As of today, there is no effective countermeasure to prevent the stealing of system prompts and all safeguarding efforts could be evaded with carefully crafted prompt injections that bypass all protection this http URL this work, we propose an alternative to conventional system prompts. We introduce prompt obfuscation to prevent the extraction of the system prompt while maintaining the utility of the system itself with only little overhead. The core idea is to find a representation of the original system prompt that leads to the same functionality, while the obfuscated system prompt does not contain any information that allows conclusions to be drawn about the original system prompt. We implement an optimization-based method to find an obfuscated prompt representation while maintaining the functionality. To evaluate our approach, we investigate eight different metrics to compare the performance of a system using the original and the obfuscated system prompts, and we show that the obfuscated version is constantly on par with the original one. We further perform three different deobfuscation attacks and show that with access to the obfuscated prompt and the LLM itself, we are not able to consistently extract meaningful information. Overall, we showed that prompt obfuscation can be an effective method to protect intellectual property while maintaining the same utility as the original system prompt.

Title: Hierarchical Narrative Analysis: Unraveling Perceptions of Generative AI

• Authors: Riona Matsuoka, Hiroki Matsumoto, Takahiro Yoshida, Tomohiro Watanabe, Ryoma Kondo, Ryohei Hisano
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11032
• Pdf URL: https://arxiv.org/pdf/2409.11032
• Copy Paste: [[2409.11032]] Hierarchical Narrative Analysis: Unraveling Perceptions of Generative AI(https://arxiv.org/abs/2409.11032)
• Keywords: fair, generative, large language model
• Abstract: Written texts reflect an author's perspective, making the thorough analysis of literature a key research method in fields such as the humanities and social sciences. However, conventional text mining techniques like sentiment analysis and topic modeling are limited in their ability to capture the hierarchical narrative structures that reveal deeper argumentative patterns. To address this gap, we propose a method that leverages large language models (LLMs) to extract and organize these structures into a hierarchical framework. We validate this approach by analyzing public opinions on generative AI collected by Japan's Agency for Cultural Affairs, comparing the narratives of supporters and critics. Our analysis provides clearer visualization of the factors influencing divergent opinions on generative AI, offering deeper insights into the structures of agreement and disagreement.

Title: Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming

• Authors: Kranti Chalamalasetti, Sherzod Hakimov, David Schlangen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11041
• Pdf URL: https://arxiv.org/pdf/2409.11041
• Copy Paste: [[2409.11041]] Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming(https://arxiv.org/abs/2409.11041)
• Keywords: large language model
• Abstract: While there has been a lot of research recently on robots in household environments, at the present time, most robots in existence can be found on shop floors, and most interactions between humans and robots happen there. ``Collaborative robots'' (cobots) designed to work alongside humans on assembly lines traditionally require expert programming, limiting ability to make changes, or manual guidance, limiting expressivity of the resulting programs. To address these limitations, we explore using Large Language Models (LLMs), and in particular, their abilities of doing in-context learning, for conversational code generation. As a first step, we define RATS, the ``Repetitive Assembly Task'', a 2D building task designed to lay the foundation for simulating industry assembly scenarios. In this task, a `programmer' instructs a cobot, using natural language, on how a certain assembly is to be built; that is, the programmer induces a program, through natural language. We create a dataset that pairs target structures with various example instructions (human-authored, template-based, and model-generated) and example code. With this, we systematically evaluate the capabilities of state-of-the-art LLMs for synthesising this kind of code, given in-context examples. Evaluating in a simulated environment, we find that LLMs are capable of generating accurate `first order code' (instruction sequences), but have problems producing `higher-order code' (abstractions such as functions, or use of loops).

Title: A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language Models: An Experimental Analysis up to 405B

• Authors: Jemin Lee, Sihyeong Park, Jinse Kwon, Jihun Oh, Yongin Kwon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11055
• Pdf URL: https://arxiv.org/pdf/2409.11055
• Copy Paste: [[2409.11055]] A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language Models: An Experimental Analysis up to 405B(https://arxiv.org/abs/2409.11055)
• Keywords: large language model
• Abstract: Prior research works have evaluated quantized LLMs using limited metrics such as perplexity or a few basic knowledge tasks and old datasets. Additionally, recent large-scale models such as Llama 3.1 with up to 405B have not been thoroughly examined. This paper evaluates the performance of instruction-tuned LLMs across various quantization methods (GPTQ, AWQ, SmoothQuant, and FP8) on models ranging from 7B to 405B. Using 13 benchmarks, we assess performance across six task types: commonsense Q\&A, knowledge and language understanding, instruction following, hallucination detection, mathematics, and dialogue. Our key findings reveal that (1) quantizing a larger LLM to a similar size as a smaller FP16 LLM generally performs better across most benchmarks, except for hallucination detection and instruction following; (2) performance varies significantly with different quantization methods, model size, and bit-width, with weight-only methods often yielding better results in larger models; (3) task difficulty does not significantly impact accuracy degradation due to quantization; and (4) the MT-Bench evaluation method has limited discriminatory power among recent high-performing LLMs.

Title: Large Language Models are Good Multi-lingual Learners : When LLMs Meet Cross-lingual Prompts

• Authors: Teng Wang, Zhenqi He, Wing-Yin Yu, Xiaojin Fu, Xiongwei Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11056
• Pdf URL: https://arxiv.org/pdf/2409.11056
• Copy Paste: [[2409.11056]] Large Language Models are Good Multi-lingual Learners : When LLMs Meet Cross-lingual Prompts(https://arxiv.org/abs/2409.11056)
• Keywords: large language model
• Abstract: With the advent of Large Language Models (LLMs), generating rule-based data for real-world applications has become more accessible. Due to the inherent ambiguity of natural language and the complexity of rule sets, especially in long contexts, LLMs often struggle to follow all specified rules, frequently omitting at least one. To enhance the reasoning and understanding of LLMs on long and complex contexts, we propose a novel prompting strategy Multi-Lingual Prompt, namely MLPrompt, which automatically translates the error-prone rule that an LLM struggles to follow into another language, thus drawing greater attention to it. Experimental results on public datasets across various tasks have shown MLPrompt can outperform state-of-the-art prompting methods such as Chain of Thought, Tree of Thought, and Self-Consistency. Additionally, we introduce a framework integrating MLPrompt with an auto-checking mechanism for structured data generation, with a specific case study in text-to-MIP instances. Further, we extend the proposed framework for text-to-SQL to demonstrate its generation ability towards structured data synthesis.

Title: KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models

• Authors: Bo Lv, Quan Zhou, Xuanang Ding, Yan Wang, Zeming Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11057
• Pdf URL: https://arxiv.org/pdf/2409.11057
• Copy Paste: [[2409.11057]] KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models(https://arxiv.org/abs/2409.11057)
• Keywords: large language model
• Abstract: The bottleneck associated with the key-value(KV) cache presents a significant challenge during the inference processes of large language models. While depth pruning accelerates inference, it requires extensive recovery training, which can take up to two weeks. On the other hand, width pruning retains much of the performance but offers slight speed gains. To tackle these challenges, we propose KVPruner to improve model efficiency while maintaining performance. Our method uses global perplexity-based analysis to determine the importance ratio for each block and provides multiple strategies to prune non-essential KV channels within blocks. Compared to the original model, KVPruner reduces runtime memory usage by 50% and boosts throughput by over 35%. Additionally, our method requires only two hours of LoRA fine-tuning on small datasets to recover most of the performance.

Title: HMF: A Hybrid Multi-Factor Framework for Dynamic Intraoperative Hypotension Prediction

• Authors: Mingyue Cheng, Jintao Zhang, Zhiding Liu, Chunli Liu, Yanhu Xie
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.11064
• Pdf URL: https://arxiv.org/pdf/2409.11064
• Copy Paste: [[2409.11064]] HMF: A Hybrid Multi-Factor Framework for Dynamic Intraoperative Hypotension Prediction(https://arxiv.org/abs/2409.11064)
• Keywords: robust, transformer
• Abstract: Intraoperative hypotension (IOH) prediction using Mean Arterial Pressure (MAP) is a critical research area with significant implications for patient outcomes during surgery. However, existing approaches predominantly employ static modeling paradigms that overlook the dynamic nature of physiological signals. In this paper, we introduce a novel Hybrid Multi-Factor (HMF) framework that reformulates IOH prediction as a blood pressure forecasting task. Our framework leverages a Transformer encoder, specifically designed to effectively capture the temporal evolution of MAP series through a patch-based input representation, which segments the input physiological series into informative patches for accurate analysis. To address the challenges of distribution shift in physiological series, our approach incorporates two key innovations: (1) Symmetric normalization and de-normalization processes help mitigate distributional drift in statistical properties, thereby ensuring the model's robustness across varying conditions, and (2) Sequence decomposition, which disaggregates the input series into trend and seasonal components, allowing for a more precise modeling of inherent sequence dependencies. Extensive experiments conducted on two real-world datasets demonstrate the superior performance of our approach compared to competitive baselines, particularly in capturing the nuanced variations in input series that are crucial for accurate IOH prediction.

Title: MonoKAN: Certified Monotonic Kolmogorov-Arnold Network

• Authors: Alejandro Polo-Molina, David Alfaya, Jose Portela
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.11078
• Pdf URL: https://arxiv.org/pdf/2409.11078
• Copy Paste: [[2409.11078]] MonoKAN: Certified Monotonic Kolmogorov-Arnold Network(https://arxiv.org/abs/2409.11078)
• Keywords: interpretability
• Abstract: Artificial Neural Networks (ANNs) have significantly advanced various fields by effectively recognizing patterns and solving complex problems. Despite these advancements, their interpretability remains a critical challenge, especially in applications where transparency and accountability are essential. To address this, explainable AI (XAI) has made progress in demystifying ANNs, yet interpretability alone is often insufficient. In certain applications, model predictions must align with expert-imposed requirements, sometimes exemplified by partial monotonicity constraints. While monotonic approaches are found in the literature for traditional Multi-layer Perceptrons (MLPs), they still face difficulties in achieving both interpretability and certified partial monotonicity. Recently, the Kolmogorov-Arnold Network (KAN) architecture, based on learnable activation functions parametrized as splines, has been proposed as a more interpretable alternative to MLPs. Building on this, we introduce a novel ANN architecture called MonoKAN, which is based on the KAN architecture and achieves certified partial monotonicity while enhancing interpretability. To achieve this, we employ cubic Hermite splines, which guarantee monotonicity through a set of straightforward conditions. Additionally, by using positive weights in the linear combinations of these splines, we ensure that the network preserves the monotonic relationships between input and output. Our experiments demonstrate that MonoKAN not only enhances interpretability but also improves predictive performance across the majority of benchmarks, outperforming state-of-the-art monotonic MLP approaches.

Title: Fractional Naive Bayes (FNB): non-convex optimization for a parsimonious weighted selective naive Bayes classifier

• Authors: Carine Hue, Marc Boullé
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.11100
• Pdf URL: https://arxiv.org/pdf/2409.11100
• Copy Paste: [[2409.11100]] Fractional Naive Bayes (FNB): non-convex optimization for a parsimonious weighted selective naive Bayes classifier(https://arxiv.org/abs/2409.11100)
• Keywords: robust
• Abstract: We study supervised classification for datasets with a very large number of input variables. The naïve Bayes classifier is attractive for its simplicity, scalability and effectiveness in many real data applications. When the strong naïve Bayes assumption of conditional independence of the input variables given the target variable is not valid, variable selection and model averaging are two common ways to improve the performance. In the case of the naïve Bayes classifier, the resulting weighting scheme on the models reduces to a weighting scheme on the variables. Here we focus on direct estimation of variable weights in such a weighted naïve Bayes classifier. We propose a sparse regularization of the model log-likelihood, which takes into account prior penalization costs related to each input variable. Compared to averaging based classifiers used up until now, our main goal is to obtain parsimonious robust models with less variables and equivalent performance. The direct estimation of the variable weights amounts to a non-convex optimization problem for which we propose and compare several two-stage algorithms. First, the criterion obtained by convex relaxation is minimized using several variants of standard gradient methods. Then, the initial non-convex optimization problem is solved using local optimization methods initialized with the result of the first stage. The various proposed algorithms result in optimization-based weighted naïve Bayes classifiers, that are evaluated on benchmark datasets and positioned w.r.t. to a reference averaging-based classifier.

Title: Depth-based Privileged Information for Boosting 3D Human Pose Estimation on RGB

• Authors: Alessandro Simoni, Francesco Marchetti, Guido Borghi, Federico Becattini, Davide Davoli, Lorenzo Garattoni, Gianpiero Francesca, Lorenzo Seidenari, Roberto Vezzani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11104
• Pdf URL: https://arxiv.org/pdf/2409.11104
• Copy Paste: [[2409.11104]] Depth-based Privileged Information for Boosting 3D Human Pose Estimation on RGB(https://arxiv.org/abs/2409.11104)
• Keywords: extraction
• Abstract: Despite the recent advances in computer vision research, estimating the 3D human pose from single RGB images remains a challenging task, as multiple 3D poses can correspond to the same 2D projection on the image. In this context, depth data could help to disambiguate the 2D information by providing additional constraints about the distance between objects in the scene and the camera. Unfortunately, the acquisition of accurate depth data is limited to indoor spaces and usually is tied to specific depth technologies and devices, thus limiting generalization capabilities. In this paper, we propose a method able to leverage the benefits of depth information without compromising its broader applicability and adaptability in a predominantly RGB-camera-centric landscape. Our approach consists of a heatmap-based 3D pose estimator that, leveraging the paradigm of Privileged Information, is able to hallucinate depth information from the RGB frames given at inference time. More precisely, depth information is used exclusively during training by enforcing our RGB-based hallucination network to learn similar features to a backbone pre-trained only on depth data. This approach proves to be effective even when dealing with limited and small datasets. Experimental results reveal that the paradigm of Privileged Information significantly enhances the model's performance, enabling efficient extraction of depth information by using only RGB images.

Title: Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games

• Authors: Matīss Rikters, Sanita Reinsone
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.11112
• Pdf URL: https://arxiv.org/pdf/2409.11112
• Copy Paste: [[2409.11112]] Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games(https://arxiv.org/abs/2409.11112)
• Keywords: large language model
• Abstract: At the beginning of 2022, a simplistic word-guessing game took the world by storm and was further adapted to many languages beyond the original English version. In this paper, we examine the strategies of daily word-guessing game players that have evolved during a period of over two years. A survey gathered from 25% of frequent players reveals their strategies and motivations for continuing the daily journey. We also explore the capability of several popular open-access large language model systems and open-source models at comprehending and playing the game in two different languages. Results highlight the struggles of certain models to maintain correct guess length and generate repetitions, as well as hallucinations of non-existent words and inflections.

Title: Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detection

• Authors: Bo Liu, Liming Zhan, Yujie Feng, Zexin Lu, Chengqiang Xie, Lei Xue, Xiao-Ming Wu, Albert Y.S. Lam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11114
• Pdf URL: https://arxiv.org/pdf/2409.11114
• Copy Paste: [[2409.11114]] Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detection(https://arxiv.org/abs/2409.11114)
• Keywords: robust, large language model
• Abstract: In the realm of task-oriented dialogue systems, a robust intent detection mechanism must effectively handle malformed utterances encountered in real-world scenarios. This study presents a novel fine-tuning framework for large language models (LLMs) aimed at enhancing in-distribution (ID) intent classification and out-of-distribution (OOD) intent detection, which utilizes semantic matching with prototypes derived from ID class names. By harnessing the highly distinguishable representations of LLMs, we construct semantic prototypes for each ID class using a diversity-grounded prompt tuning approach. We rigorously test our framework in a challenging OOD context, where ID and OOD classes are semantically close yet distinct, referred to as \emph{near} OOD detection. For a thorough assessment, we benchmark our method against the prevalent fine-tuning approaches. The experimental findings reveal that our method demonstrates superior performance in both few-shot ID intent classification and near-OOD intent detection tasks.

Title: Learning Generalized Hamiltonians using fully Symplectic Mappings

• Authors: Harsh Choudhary, Chandan Gupta, Vyacheslav kungrutsev, Melvin Leok, Georgios Korpas
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11138
• Pdf URL: https://arxiv.org/pdf/2409.11138
• Copy Paste: [[2409.11138]] Learning Generalized Hamiltonians using fully Symplectic Mappings(https://arxiv.org/abs/2409.11138)
• Keywords: robust
• Abstract: Many important physical systems can be described as the evolution of a Hamiltonian system, which has the important property of being conservative, that is, energy is conserved throughout the evolution. Physics Informed Neural Networks and in particular Hamiltonian Neural Networks have emerged as a mechanism to incorporate structural inductive bias into the NN model. By ensuring physical invariances are conserved, the models exhibit significantly better sample complexity and out-of-distribution accuracy than standard NNs. Learning the Hamiltonian as a function of its canonical variables, typically position and velocity, from sample observations of the system thus becomes a critical task in system identification and long-term prediction of system behavior. However, to truly preserve the long-run physical conservation properties of Hamiltonian systems, one must use symplectic integrators for a forward pass of the system's simulation. While symplectic schemes have been used in the literature, they are thus far limited to situations when they reduce to explicit algorithms, which include the case of separable Hamiltonians or augmented non-separable Hamiltonians. We extend it to generalized non-separable Hamiltonians, and noting the self-adjoint property of symplectic integrators, we bypass computationally intensive backpropagation through an ODE solver. We show that the method is robust to noise and provides a good approximation of the system Hamiltonian when the state variables are sampled from a noisy observation. In the numerical results, we show the performance of the method concerning Hamiltonian reconstruction and conservation, indicating its particular advantage for non-separable systems.

Title: Scale generalisation properties of extended scale-covariant and scale-invariant Gaussian derivative networks on image datasets with spatial scaling variations

• Authors: Andrzej Perzanowski, Tony Lindeberg
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11140
• Pdf URL: https://arxiv.org/pdf/2409.11140
• Copy Paste: [[2409.11140]] Scale generalisation properties of extended scale-covariant and scale-invariant Gaussian derivative networks on image datasets with spatial scaling variations(https://arxiv.org/abs/2409.11140)
• Keywords: explainability
• Abstract: This paper presents an in-depth analysis of the scale generalisation properties of the scale-covariant and scale-invariant Gaussian derivative networks, complemented with both conceptual and algorithmic extensions. For this purpose, Gaussian derivative networks are evaluated on new rescaled versions of the Fashion-MNIST and the CIFAR-10 datasets, with spatial scaling variations over a factor of 4 in the testing data, that are not present in the training data. Additionally, evaluations on the previously existing STIR datasets show that the Gaussian derivative networks achieve better scale generalisation than previously reported for these datasets for other types of deep networks. We first experimentally demonstrate that the Gaussian derivative networks have quite good scale generalisation properties on the new datasets, and that average pooling of feature responses over scales may sometimes also lead to better results than the previously used approach of max pooling over scales. Then, we demonstrate that using a spatial max pooling mechanism after the final layer enables localisation of non-centred objects in image domain, with maintained scale generalisation properties. We also show that regularisation during training, by applying dropout across the scale channels, referred to as scale-channel dropout, improves both the performance and the scale generalisation. In additional ablation studies, we demonstrate that discretisations of Gaussian derivative networks, based on the discrete analogue of the Gaussian kernel in combination with central difference operators, perform best or among the best, compared to a set of other discrete approximations of the Gaussian derivative kernels. Finally, by visualising the activation maps and the learned receptive fields, we demonstrate that the Gaussian derivative networks have very good explainability properties.

Title: Semformer: Transformer Language Models with Semantic Planning

• Authors: Yongjing Yin, Junran Ding, Kai Song, Yue Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11143
• Pdf URL: https://arxiv.org/pdf/2409.11143
• Copy Paste: [[2409.11143]] Semformer: Transformer Language Models with Semantic Planning(https://arxiv.org/abs/2409.11143)
• Keywords: transformer
• Abstract: Next-token prediction serves as the dominant component in current neural language models. During the training phase, the model employs teacher forcing, which predicts tokens based on all preceding ground truth tokens. However, this approach has been found to create shortcuts, utilizing the revealed prefix to spuriously fit future tokens, potentially compromising the accuracy of the next-token predictor. In this paper, we introduce Semformer, a novel method of training a Transformer language model that explicitly models the semantic planning of response. Specifically, we incorporate a sequence of planning tokens into the prefix, guiding the planning token representations to predict the latent semantic representations of the response, which are induced by an autoencoder. In a minimal planning task (i.e., graph path-finding), our model exhibits near-perfect performance and effectively mitigates shortcut learning, a feat that standard training methods and baseline models have been unable to accomplish. Furthermore, we pretrain Semformer from scratch with 125M parameters, demonstrating its efficacy through measures of perplexity, in-context learning, and fine-tuning on summarization tasks.

Title: Reasoning Graph Enhanced Exemplars Retrieval for In-Context Learning

• Authors: Yukang Lin, Bingchen Zhong, Shuoran Jiang, Joanna Siebert, Qingcai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11147
• Pdf URL: https://arxiv.org/pdf/2409.11147
• Copy Paste: [[2409.11147]] Reasoning Graph Enhanced Exemplars Retrieval for In-Context Learning(https://arxiv.org/abs/2409.11147)
• Keywords: large language model
• Abstract: Large language models(LLMs) have exhibited remarkable few-shot learning capabilities and unified the paradigm of NLP tasks through the in-context learning(ICL) technique. Despite the success of ICL, the quality of the exemplar demonstrations can significantly influence the LLM's performance. Existing exemplar selection methods mainly focus on the semantic similarity between queries and candidate exemplars. On the other hand, the logical connections between reasoning steps can be beneficial to depict the problem-solving process as well. In this paper, we proposes a novel method named Reasoning Graph-enhanced Exemplar Retrieval(RGER). RGER first quires LLM to generate an initial response, then expresses intermediate problem-solving steps to a graph structure. After that, it employs graph kernel to select exemplars with semantic and structural similarity. Extensive experiments demonstrate the structural relationship is helpful to the alignment of queries and candidate exemplars. The efficacy of RGER on math and logit reasoning tasks showcases its superiority over state-of-the-art retrieval-based approaches. Our code is released at this https URL.

Title: Improving the Efficiency of Visually Augmented Language Models

• Authors: Paula Ontalvilla, Aitor Ormazabal, Gorka Azkune
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11148
• Pdf URL: https://arxiv.org/pdf/2409.11148
• Copy Paste: [[2409.11148]] Improving the Efficiency of Visually Augmented Language Models(https://arxiv.org/abs/2409.11148)
• Keywords: fair
• Abstract: Despite the impressive performance of autoregressive Language Models (LM) it has been shown that due to reporting bias, LMs lack visual knowledge, i.e. they do not know much about the visual world and its properties. To augment LMs with visual knowledge, existing solutions often rely on explicit images, requiring time-consuming retrieval or image generation systems. This paper shows that explicit images are not necessary to visually augment an LM. Instead, we use visually-grounded text representations obtained from the well-known CLIP multimodal system. For a fair comparison, we modify VALM, a visually-augmented LM which uses image retrieval and representation, to work directly with visually-grounded text representations. We name this new model BLIND-VALM. We show that BLIND-VALM performs on par with VALM for Visual Language Understanding (VLU), Natural Language Understanding (NLU) and Language Modeling tasks, despite being significantly more efficient and simpler. We also show that scaling up our model within the compute budget of VALM, either increasing the model or pre-training corpus size, we outperform VALM for all the evaluation tasks.

Title: SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibration

• Authors: Xin Guan, Nathaniel Demchak, Saloni Gupta, Ze Wang, Ediz Ertekin Jr., Adriano Koshiyama, Emre Kazim, Zekun Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11149
• Pdf URL: https://arxiv.org/pdf/2409.11149
• Copy Paste: [[2409.11149]] SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibration(https://arxiv.org/abs/2409.11149)
• Keywords: fair, large language model
• Abstract: The development of unbiased large language models is widely recognized as crucial, yet existing benchmarks fall short in detecting biases due to limited scope, contamination, and lack of a fairness baseline. SAGED(-Bias) is the first holistic benchmarking pipeline to address these problems. The pipeline encompasses five core stages: scraping materials, assembling benchmarks, generating responses, extracting numeric features, and diagnosing with disparity metrics. SAGED includes metrics for max disparity, such as impact ratio, and bias concentration, such as Max Z-scores. Noticing that assessment tool bias and contextual bias in prompts can distort evaluation, SAGED implements counterfactual branching and baseline calibration for mitigation. For demonstration, we use SAGED on G20 Countries with popular 8b-level models including Gemma2, Llama3.1, Mistral, and Qwen2. With sentiment analysis, we find that while Mistral and Qwen2 show lower max disparity and higher bias concentration than Gemma2 and Llama3.1, all models are notably biased against countries like Russia and (except for Qwen2) China. With further experiments to have models role-playing U.S. (vice-/former-) presidents, we see bias amplifies and shifts in heterogeneous directions. Moreover, we see Qwen2 and Mistral not engage in role-playing, while Llama3.1 and Gemma2 role-play Trump notably more intensively than Biden and Harris, indicating role-playing performance bias in these models.

Title: UltimateDO: An Efficient Framework to Marry Occupancy Prediction with 3D Object Detection via Channel2height

• Authors: Zichen Yu, Changyong Shu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11160
• Pdf URL: https://arxiv.org/pdf/2409.11160
• Copy Paste: [[2409.11160]] UltimateDO: An Efficient Framework to Marry Occupancy Prediction with 3D Object Detection via Channel2height(https://arxiv.org/abs/2409.11160)
• Keywords: transformer
• Abstract: Occupancy and 3D object detection are characterized as two standard tasks in modern autonomous driving system. In order to deploy them on a series of edge chips with better precision and time-consuming trade-off, contemporary approaches either deploy standalone models for individual tasks, or design a multi-task paradigm with separate heads. However, they might suffer from deployment difficulties (i.e., 3D convolution, transformer and so on) or deficiencies in task coordination. Instead, we argue that a favorable framework should be devised in pursuit of ease deployment on diverse chips and high precision with little time-consuming. Oriented at this, we revisit the paradigm for interaction between 3D object detection and occupancy prediction, reformulate the model with 2D convolution and prioritize the tasks such that each contributes to other. Thus, we propose a method to achieve fast 3D object detection and occupancy prediction (UltimateDO), wherein the light occupancy prediction head in FlashOcc is married to 3D object detection network, with negligible additional timeconsuming of only 1.1ms while facilitating each other. We instantiate UltimateDO on the challenging nuScenes-series benchmarks.

Title: Synthetic data augmentation for robotic mobility aids to support blind and low vision people

• Authors: Hochul Hwang, Krisha Adhikari, Satya Shodhaka, Donghyun Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11164
• Pdf URL: https://arxiv.org/pdf/2409.11164
• Copy Paste: [[2409.11164]] Synthetic data augmentation for robotic mobility aids to support blind and low vision people(https://arxiv.org/abs/2409.11164)
• Keywords: robust
• Abstract: Robotic mobility aids for blind and low-vision (BLV) individuals rely heavily on deep learning-based vision models specialized for various navigational tasks. However, the performance of these models is often constrained by the availability and diversity of real-world datasets, which are challenging to collect in sufficient quantities for different tasks. In this study, we investigate the effectiveness of synthetic data, generated using Unreal Engine 4, for training robust vision models for this safety-critical application. Our findings demonstrate that synthetic data can enhance model performance across multiple tasks, showcasing both its potential and its limitations when compared to real-world data. We offer valuable insights into optimizing synthetic data generation for developing robotic mobility aids. Additionally, we publicly release our generated synthetic dataset to support ongoing research in assistive technologies for BLV individuals, available at this https URL.

Title: Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving

• Authors: Yunsheng Ma, Amr Abdelraouf, Rohit Gupta, Ziran Wang, Kyungtae Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11182
• Pdf URL: https://arxiv.org/pdf/2409.11182
• Copy Paste: [[2409.11182]] Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving(https://arxiv.org/abs/2409.11182)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable potential for enhancing scene understanding in autonomous driving systems through powerful logical reasoning capabilities. However, the deployment of these models faces significant challenges due to their substantial parameter sizes and computational demands, which often exceed the constraints of onboard computation. One major limitation arises from the large number of visual tokens required to capture fine-grained and long-context visual information, leading to increased latency and memory consumption. To address this issue, we propose Video Token Sparsification (VTS), a novel approach that leverages the inherent redundancy in consecutive video frames to significantly reduce the total number of visual tokens while preserving the most salient information. VTS employs a lightweight CNN-based proposal model to adaptively identify key frames and prune less informative tokens, effectively mitigating hallucinations and increasing inference throughput without compromising performance. We conduct comprehensive experiments on the DRAMA and LingoQA benchmarks, demonstrating the effectiveness of VTS in achieving up to a 33\% improvement in inference throughput and a 28\% reduction in memory usage compared to the baseline without compromising performance.

Title: LASERS: LAtent Space Encoding for Representations with Sparsity for Generative Modeling

• Authors: Xin Li, Anand Sarwate
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11184
• Pdf URL: https://arxiv.org/pdf/2409.11184
• Copy Paste: [[2409.11184]] LASERS: LAtent Space Encoding for Representations with Sparsity for Generative Modeling(https://arxiv.org/abs/2409.11184)
• Keywords: generative
• Abstract: Learning compact and meaningful latent space representations has been shown to be very useful in generative modeling tasks for visual data. One particular example is applying Vector Quantization (VQ) in variational autoencoders (VQ-VAEs, VQ-GANs, etc.), which has demonstrated state-of-the-art performance in many modern generative modeling applications. Quantizing the latent space has been justified by the assumption that the data themselves are inherently discrete in the latent space (like pixel values). In this paper, we propose an alternative representation of the latent space by relaxing the structural assumption than the VQ formulation. Specifically, we assume that the latent space can be approximated by a union of subspaces model corresponding to a dictionary-based representation under a sparsity constraint. The dictionary is learned/updated during the training process. We apply this approach to look at two models: Dictionary Learning Variational Autoencoders (DL-VAEs) and DL-VAEs with Generative Adversarial Networks (DL-GANs). We show empirically that our more latent space is more expressive and has leads to better representations than the VQ approach in terms of reconstruction quality at the expense of a small computational overhead for the latent space computation. Our results thus suggest that the true benefit of the VQ approach might not be from discretization of the latent space, but rather the lossy compression of the latent space. We confirm this hypothesis by showing that our sparse representations also address the codebook collapse issue as found common in VQ-family models.

Title: Deep Learning tools to support deforestation monitoring in the Ivory Coast using SAR and Optical satellite imagery

• Authors: Gabriele Sartor, Matteo Salis, Stefano Pinardi, Ozgur Saracik, Rosa Meo
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11186
• Pdf URL: https://arxiv.org/pdf/2409.11186
• Copy Paste: [[2409.11186]] Deep Learning tools to support deforestation monitoring in the Ivory Coast using SAR and Optical satellite imagery(https://arxiv.org/abs/2409.11186)
• Keywords: segmentation
• Abstract: Deforestation is gaining an increasingly importance due to its strong influence on the sorrounding environment, especially in developing countries where population has a disadvantaged economic condition and agriculture is the main source of income. In Ivory Coast, for instance, where the cocoa production is the most remunerative activity, it is not rare to assist to the replacement of portion of ancient forests with new cocoa plantations. In order to monitor this type of deleterious activities, satellites can be employed to recognize the disappearance of the forest to prevent it from expand its area of interest. In this study, Forest-Non-Forest map (FNF) has been used as ground truth for models based on Sentinel images input. State-of-the-art models U-Net, Attention U-Net, Segnet and FCN32 are compared over different years combining Sentinel-1, Sentinel-2 and cloud probability to create forest/non-forest segmentation. Although Ivory Coast lacks of forest coverage datasets and is partially covered by Sentinel images, it is demonstrated the feasibility to create models classifying forest and non-forests pixels over the area using open datasets to predict where deforestation could have occurred. Although a significant portion of the deforestation research is carried out on visible bands, SAR acquisitions are employed to overcome the limits of RGB images over areas often covered by clouds. Finally, the most promising model is employed to estimate the hectares of forest has been cut between 2019 and 2020.

Title: HS3-Bench: A Benchmark and Strong Baseline for Hyperspectral Semantic Segmentation in Driving Scenarios

• Authors: Nick Theisen, Robin Bartsch, Dietrich Paulus, Peer Neubert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11205
• Pdf URL: https://arxiv.org/pdf/2409.11205
• Copy Paste: [[2409.11205]] HS3-Bench: A Benchmark and Strong Baseline for Hyperspectral Semantic Segmentation in Driving Scenarios(https://arxiv.org/abs/2409.11205)
• Keywords: segmentation
• Abstract: Semantic segmentation is an essential step for many vision applications in order to understand a scene and the objects within. Recent progress in hyperspectral imaging technology enables the application in driving scenarios and the hope is that the devices perceptive abilities provide an advantage over RGB-cameras. Even though some datasets exist, there is no standard benchmark available to systematically measure progress on this task and evaluate the benefit of hyperspectral data. In this paper, we work towards closing this gap by providing the HyperSpectral Semantic Segmentation benchmark (HS3-Bench). It combines annotated hyperspectral images from three driving scenario datasets and provides standardized metrics, implementations, and evaluation protocols. We use the benchmark to derive two strong baseline models that surpass the previous state-of-the-art performances with and without pre-training on the individual datasets. Further, our results indicate that the existing learning-based methods benefit more from leveraging additional RGB training data than from leveraging the additional hyperspectral channels. This poses important questions for future research on hyperspectral imaging for semantic segmentation in driving scenarios. Code to run the benchmark and the strong baseline approaches are available under this https URL.

Title: High-Order Evolving Graphs for Enhanced Representation of Traffic Dynamics

• Authors: Aditya Humnabadkar, Arindam Sikdar, Benjamin Cave, Huaizhong Zhang, Paul Bakaki, Ardhendu Behera
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11206
• Pdf URL: https://arxiv.org/pdf/2409.11206
• Copy Paste: [[2409.11206]] High-Order Evolving Graphs for Enhanced Representation of Traffic Dynamics(https://arxiv.org/abs/2409.11206)
• Keywords: robust
• Abstract: We present an innovative framework for traffic dynamics analysis using High-Order Evolving Graphs, designed to improve spatio-temporal representations in autonomous driving contexts. Our approach constructs temporal bidirectional bipartite graphs that effectively model the complex interactions within traffic scenes in real-time. By integrating Graph Neural Networks (GNNs) with high-order multi-aggregation strategies, we significantly enhance the modeling of traffic scene dynamics, providing a more accurate and detailed analysis of these interactions. Additionally, we incorporate inductive learning techniques inspired by the GraphSAGE framework, enabling our model to adapt to new and unseen traffic scenarios without the need for retraining, thus ensuring robust generalization. Through extensive experiments on the ROAD and ROAD Waymo datasets, we establish a comprehensive baseline for further developments, demonstrating the potential of our method in accurately capturing traffic behavior. Our results emphasize the value of high-order statistical moments and feature-gated attention mechanisms in improving traffic behavior analysis, laying the groundwork for advancing autonomous driving technologies. Our source code is available at: this https URL\_Order\_Graphs

Title: Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization

• Authors: Jianing Wang, Yang Zhou, Xiaocheng Zhang, Mengjiao Bao, Peng Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11212
• Pdf URL: https://arxiv.org/pdf/2409.11212
• Copy Paste: [[2409.11212]] Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization(https://arxiv.org/abs/2409.11212)
• Keywords: robust, large language model
• Abstract: Iterative preference optimization has recently become one of the de-facto training paradigms for large language models (LLMs), but the performance is still underwhelming due to too much noisy preference data yielded in the loop. To combat this issue, we present an \textbf{U}ncertainty-enhanced \textbf{P}reference \textbf{O}ptimization (UPO) framework to make the LLM self-evolve with reliable feedback. The key idea is mitigating the noisy preference data derived from the current policy and reward models by performing pair-wise uncertainty estimation and judiciously reliable feedback sampling. To reach this goal, we thus introduce an estimator model, which incorporates Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation for the preference data derived from the LLM policy. Compared to the existing methods that directly filter generated responses based on the reward score, the estimator focuses on the model uncertainty in a pair-wise manner and effectively bypasses the confirmation bias problem of the reward model. Additionally, we also propose an uncertainty-enhanced self-evolution algorithm to improve the robustness of preference optimization and encourage the LLM to generate responses with both high reward and certainty. Extensive experiments over multiple benchmarks demonstrate that our framework substantially alleviates the noisy problem and improves the performance of iterative preference optimization.

Title: Exploring ChatGPT-based Augmentation Strategies for Contrastive Aspect-based Sentiment Analysis

• Authors: Lingling Xu, Haoran Xie, S. Joe Qin, Fu Lee Wang, Xiaohui Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11218
• Pdf URL: https://arxiv.org/pdf/2409.11218
• Copy Paste: [[2409.11218]] Exploring ChatGPT-based Augmentation Strategies for Contrastive Aspect-based Sentiment Analysis(https://arxiv.org/abs/2409.11218)
• Keywords: large language model
• Abstract: Aspect-based sentiment analysis (ABSA) involves identifying sentiment towards specific aspect terms in a sentence and allows us to uncover nuanced perspectives and attitudes on particular aspects of a product, service, or topic. However, the scarcity of labeled data poses a significant challenge to training high-quality models. To address this issue, we explore the potential of data augmentation using ChatGPT, a well-performing large language model (LLM), to enhance the sentiment classification performance towards aspect terms. Specifically, we explore three data augmentation strategies based on ChatGPT: context-focused, aspect-focused, and context-aspect data augmentation techniques. Context-focused data augmentation focuses on changing the word expression of context words in the sentence while keeping aspect terms unchanged. In contrast, aspect-focused data augmentation aims to change aspect terms but keep context words unchanged. Context-Aspect data augmentation integrates the above two data augmentations to generate augmented samples. Furthermore, we incorporate contrastive learning into the ABSA tasks to improve performance. Extensive experiments show that all three data augmentation techniques lead to performance improvements, with the context-aspect data augmentation strategy performing best and surpassing the performance of the baseline models.

Title: Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models

• Authors: Tianqi Chen, Shujian Zhang, Mingyuan Zhou
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11219
• Pdf URL: https://arxiv.org/pdf/2409.11219
• Copy Paste: [[2409.11219]] Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models(https://arxiv.org/abs/2409.11219)
• Keywords: secure, diffusion, data-free, generative
• Abstract: The machine learning community is increasingly recognizing the importance of fostering trust and safety in modern generative AI (GenAI) models. We posit machine unlearning (MU) as a crucial foundation for developing safe, secure, and trustworthy GenAI models. Traditional MU methods often rely on stringent assumptions and require access to real data. This paper introduces Score Forgetting Distillation (SFD), an innovative MU approach that promotes the forgetting of undesirable information in diffusion models by aligning the conditional scores of ``unsafe'' classes or concepts with those of ``safe'' ones. To eliminate the need for real data, our SFD framework incorporates a score-based MU loss into the score distillation objective of a pretrained diffusion model. This serves as a regularization term that preserves desired generation capabilities while enabling the production of synthetic data through a one-step generator. Our experiments on pretrained label-conditional and text-to-image diffusion models demonstrate that our method effectively accelerates the forgetting of target classes or concepts during generation, while preserving the quality of other classes or concepts. This unlearned and distilled diffusion not only pioneers a novel concept in MU but also accelerates the generation speed of diffusion models. Our experiments and studies on a range of diffusion models and datasets confirm that our approach is generalizable, effective, and advantageous for MU in diffusion models.

Title: Multimodal Attention-Enhanced Feature Fusion-based Weekly Supervised Anomaly Violence Detection

• Authors: Yuta Kaneko, Abu Saleh Musa Miah, Najmul Hassan, Hyoun-Sup Lee, Si-Woong Jang, Jungpil Shin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11223
• Pdf URL: https://arxiv.org/pdf/2409.11223
• Copy Paste: [[2409.11223]] Multimodal Attention-Enhanced Feature Fusion-based Weekly Supervised Anomaly Violence Detection(https://arxiv.org/abs/2409.11223)
• Keywords: robust
• Abstract: Weakly supervised video anomaly detection (WS-VAD) is a crucial area in computer vision for developing intelligent surveillance systems. This system uses three feature streams: RGB video, optical flow, and audio signals, where each stream extracts complementary spatial and temporal features using an enhanced attention module to improve detection accuracy and robustness. In the first stream, we employed an attention-based, multi-stage feature enhancement approach to improve spatial and temporal features from the RGB video where the first stage consists of a ViT-based CLIP module, with top-k features concatenated in parallel with I3D and Temporal Contextual Aggregation (TCA) based rich spatiotemporal features. The second stage effectively captures temporal dependencies using the Uncertainty-Regulated Dual Memory Units (UR-DMU) model, which learns representations of normal and abnormal data simultaneously, and the third stage is employed to select the most relevant spatiotemporal features. The second stream extracted enhanced attention-based spatiotemporal features from the flow data modality-based feature by taking advantage of the integration of the deep learning and attention module. The audio stream captures auditory cues using an attention module integrated with the VGGish model, aiming to detect anomalies based on sound patterns. These streams enrich the model by incorporating motion and audio signals often indicative of abnormal events undetectable through visual analysis alone. The concatenation of the multimodal fusion leverages the strengths of each modality, resulting in a comprehensive feature set that significantly improves anomaly detection accuracy and robustness across three datasets. The extensive experiment and high performance with the three benchmark datasets proved the effectiveness of the proposed system over the existing state-of-the-art system.

Title: A Human-Centered Risk Evaluation of Biometric Systems Using Conjoint Analysis

• Authors: Tetsushi Ohki, Narishige Abe, Hidetsugu Uchida, Shigefumi Yamada
• Subjects: cs.CV, cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.11224
• Pdf URL: https://arxiv.org/pdf/2409.11224
• Copy Paste: [[2409.11224]] A Human-Centered Risk Evaluation of Biometric Systems Using Conjoint Analysis(https://arxiv.org/abs/2409.11224)
• Keywords: security, attack, biometric
• Abstract: Biometric recognition systems, known for their convenience, are widely adopted across various fields. However, their security faces risks depending on the authentication algorithm and deployment environment. Current risk assessment methods faces significant challenges in incorporating the crucial factor of attacker's motivation, leading to incomplete evaluations. This paper presents a novel human-centered risk evaluation framework using conjoint analysis to quantify the impact of risk factors, such as surveillance cameras, on attacker's motivation. Our framework calculates risk values incorporating the False Acceptance Rate (FAR) and attack probability, allowing comprehensive comparisons across use cases. A survey of 600 Japanese participants demonstrates our method's effectiveness, showing how security measures influence attacker's motivation. This approach helps decision-makers customize biometric systems to enhance security while maintaining usability.

Title: Generalized Few-Shot Semantic Segmentation in Remote Sensing: Challenge and Benchmark

• Authors: Clifford Broni-Bediako, Junshi Xia, Jian Song, Hongruixuan Chen, Mennatullah Siam, Naoto Yokoya
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11227
• Pdf URL: https://arxiv.org/pdf/2409.11227
• Copy Paste: [[2409.11227]] Generalized Few-Shot Semantic Segmentation in Remote Sensing: Challenge and Benchmark(https://arxiv.org/abs/2409.11227)
• Keywords: segmentation
• Abstract: Learning with limited labelled data is a challenging problem in various applications, including remote sensing. Few-shot semantic segmentation is one approach that can encourage deep learning models to learn from few labelled examples for novel classes not seen during the training. The generalized few-shot segmentation setting has an additional challenge which encourages models not only to adapt to the novel classes but also to maintain strong performance on the training base classes. While previous datasets and benchmarks discussed the few-shot segmentation setting in remote sensing, we are the first to propose a generalized few-shot segmentation benchmark for remote sensing. The generalized setting is more realistic and challenging, which necessitates exploring it within the remote sensing context. We release the dataset augmenting OpenEarthMap with additional classes labelled for the generalized few-shot evaluation setting. The dataset is released during the OpenEarthMap land cover mapping generalized few-shot challenge in the L3D-IVU workshop in conjunction with CVPR 2024. In this work, we summarize the dataset and challenge details in addition to providing the benchmark results on the two phases of the challenge for the validation and test sets.

Title: Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models

• Authors: Bishwash Khanal, Jeffery M. Capone
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11233
• Pdf URL: https://arxiv.org/pdf/2409.11233
• Copy Paste: [[2409.11233]] Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models(https://arxiv.org/abs/2409.11233)
• Keywords: large language model
• Abstract: Large language models (LLMs) offer powerful capabilities but incur substantial computational costs, driving the need for efficient compression techniques. This study evaluates the impact of popular compression methods - Magnitude Pruning, SparseGPT, and Wanda - on the LLaMA-2-7B model, focusing on the trade-offs between model size reduction, downstream task performance, and the role of calibration data. Our findings reveal that while SparseGPT and Wanda preserve perplexity even at 50% sparsity, they suffer significant degradation on downstream tasks, highlighting the inadequacy of perplexity as the sole evaluation metric. To address this, we introduce Jensen-Shannon (JS) Divergence as a more comprehensive metric that captures nuanced changes in model behavior post-compression. We further demonstrate that task-specific calibration data significantly enhances the downstream performance of compressed models compared to general calibration data. This research underscores the necessity for diverse evaluation metrics and careful calibration data selection to fully understand the complexities of LLM compression and its implications for practical applications.

Title: LLM-as-a-Judge & Reward Model: What They Can and Cannot Do

• Authors: Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, Seunghyeok Hong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11239
• Pdf URL: https://arxiv.org/pdf/2409.11239
• Copy Paste: [[2409.11239]] LLM-as-a-Judge & Reward Model: What They Can and Cannot Do(https://arxiv.org/abs/2409.11239)
• Keywords: large language model
• Abstract: LLM-as-a-Judge and reward models are widely used alternatives of multiple-choice questions or human annotators for large language model (LLM) evaluation. Their efficacy shines in evaluating long-form responses, serving a critical role as evaluators of leaderboards and as proxies to align LLMs via reinforcement learning. However, despite their popularity, their effectiveness outside of English remains largely unexplored. In this paper, we conduct a comprehensive analysis on automated evaluators, reporting key findings on their behavior in a non-English environment. First, we discover that English evaluation capabilities significantly influence language-specific capabilities, often more than the language proficiency itself, enabling evaluators trained in English to easily transfer their skills to other languages. Second, we identify critical shortcomings, where LLMs fail to detect and penalize errors, such as factual inaccuracies, cultural misrepresentations, and the presence of unwanted language. Finally, we release Kudge, the first non-English meta-evaluation dataset containing 5,012 human annotations in Korean.

Title: Federated Learning with Integrated Sensing, Communication, and Computation: Frameworks and Performance Analysis

• Authors: Yipeng Liang, Qimei Chen, Hao Jiang
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.11240
• Pdf URL: https://arxiv.org/pdf/2409.11240
• Copy Paste: [[2409.11240]] Federated Learning with Integrated Sensing, Communication, and Computation: Frameworks and Performance Analysis(https://arxiv.org/abs/2409.11240)
• Keywords: robust, federate
• Abstract: With the emergence of integrated sensing, communication, and computation (ISCC) in the upcoming 6G era, federated learning with ISCC (FL-ISCC), integrating sample collection, local training, and parameter exchange and aggregation, has garnered increasing interest for enhancing training efficiency. Currently, FL-ISCC primarily includes two algorithms: FedAVG-ISCC and FedSGD-ISCC. However, the theoretical understanding of the performance and advantages of these algorithms remains limited. To address this gap, we investigate a general FL-ISCC framework, implementing both FedAVG-ISCC and FedSGD-ISCC. We experimentally demonstrate the substantial potential of the ISCC framework in reducing latency and energy consumption in FL. Furthermore, we provide a theoretical analysis and comparison. The results reveal that:1) Both sample collection and communication errors negatively impact algorithm performance, highlighting the need for careful design to optimize FL-ISCC applications. 2) FedAVG-ISCC performs better than FedSGD-ISCC under IID data due to its advantage with multiple local updates. 3) FedSGD-ISCC is more robust than FedAVG-ISCC under non-IID data, where the multiple local updates in FedAVG-ISCC worsen performance as non-IID data increases. FedSGD-ISCC maintains performance levels similar to IID conditions. 4) FedSGD-ISCC is more resilient to communication errors than FedAVG-ISCC, which suffers from significant performance degradation as communication errors increase.Extensive simulations confirm the effectiveness of the FL-ISCC framework and validate our theoretical analysis.

Title: Linear Recency Bias During Training Improves Transformers' Fit to Reading Times

• Authors: Christian Clark, Byung-Doh Oh, William Schuler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11250
• Pdf URL: https://arxiv.org/pdf/2409.11250
• Copy Paste: [[2409.11250]] Linear Recency Bias During Training Improves Transformers' Fit to Reading Times(https://arxiv.org/abs/2409.11250)
• Keywords: transformer
• Abstract: Recent psycholinguistic research has compared human reading times to surprisal estimates from language models to study the factors shaping human sentence processing difficulty. Previous studies have shown a strong fit between surprisal values from Transformers and reading times. However, standard Transformers work with a lossless representation of the entire previous linguistic context, unlike models of human language processing that include memory decay. To bridge this gap, this paper evaluates a modification of the Transformer model that uses ALiBi (Press et al., 2022), a recency bias added to attention scores. Surprisal estimates with ALiBi show an improved fit to human reading times compared to a standard Transformer baseline. A subsequent analysis of attention heads suggests that ALiBi's mixture of slopes -- which determine the rate of memory decay in each attention head -- may play a role in the improvement by helping models with ALiBi to track different kinds of linguistic dependencies.

Title: WER We Stand: Benchmarking Urdu ASR Models

• Authors: Samee Arif, Aamina Jamal Khan, Mustafa Abbas, Agha Ali Raza, Awais Athar
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.11252
• Pdf URL: https://arxiv.org/pdf/2409.11252
• Copy Paste: [[2409.11252]] WER We Stand: Benchmarking Urdu ASR Models(https://arxiv.org/abs/2409.11252)
• Keywords: robust
• Abstract: This paper presents a comprehensive evaluation of Urdu Automatic Speech Recognition (ASR) models. We analyze the performance of three ASR model families: Whisper, MMS, and Seamless-M4T using Word Error Rate (WER), along with a detailed examination of the most frequent wrong words and error types including insertions, deletions, and substitutions. Our analysis is conducted using two types of datasets, read speech and conversational speech. Notably, we present the first conversational speech dataset designed for benchmarking Urdu ASR models. We find that seamless-large outperforms other ASR models on the read speech dataset, while whisper-large performs best on the conversational speech dataset. Furthermore, this evaluation highlights the complexities of assessing ASR models for low-resource languages like Urdu using quantitative metrics alone and emphasizes the need for a robust Urdu text normalization system. Our findings contribute valuable insights for developing robust ASR systems for low-resource languages like Urdu.

Title: Norm of Mean Contextualized Embeddings Determines their Variance

• Authors: Hiroaki Yamagiwa, Hidetoshi Shimodaira
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11253
• Pdf URL: https://arxiv.org/pdf/2409.11253
• Copy Paste: [[2409.11253]] Norm of Mean Contextualized Embeddings Determines their Variance(https://arxiv.org/abs/2409.11253)
• Keywords: transformer
• Abstract: Contextualized embeddings vary by context, even for the same token, and form a distribution in the embedding space. To analyze this distribution, we focus on the norm of the mean embedding and the variance of the embeddings. In this study, we first demonstrate that these values follow the well-known formula for variance in statistics and provide an efficient sequential computation method. Then, by observing embeddings from intermediate layers of several Transformer models, we found a strong trade-off relationship between the norm and the variance: as the mean embedding becomes closer to the origin, the variance increases. This trade-off is likely influenced by the layer normalization mechanism used in Transformer models. Furthermore, when the sets of token embeddings are treated as clusters, we show that the variance of the entire embedding set can theoretically be decomposed into the within-cluster variance and the between-cluster variance. We found experimentally that as the layers of Transformer models deepen, the embeddings move farther from the origin, the between-cluster variance relatively decreases, and the within-cluster variance relatively increases. These results are consistent with existing studies on the anisotropy of the embedding spaces across layers.

Title: Towards Novel Malicious Packet Recognition: A Few-Shot Learning Approach

• Authors: Kyle Stein, Andrew A. Mahyari, Guillermo Francia III, Eman El-Sheikh
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11254
• Pdf URL: https://arxiv.org/pdf/2409.11254
• Copy Paste: [[2409.11254]] Towards Novel Malicious Packet Recognition: A Few-Shot Learning Approach(https://arxiv.org/abs/2409.11254)
• Keywords: security, defense, attack, robust, extraction, large language model
• Abstract: As the complexity and connectivity of networks increase, the need for novel malware detection approaches becomes imperative. Traditional security defenses are becoming less effective against the advanced tactics of today's cyberattacks. Deep Packet Inspection (DPI) has emerged as a key technology in strengthening network security, offering detailed analysis of network traffic that goes beyond simple metadata analysis. DPI examines not only the packet headers but also the payload content within, offering a thorough insight into the data traversing the network. This study proposes a novel approach that leverages a large language model (LLM) and few-shot learning to accurately recognizes novel, unseen malware types with few labels samples. Our proposed approach uses a pretrained LLM on known malware types to extract the embeddings from packets. The embeddings are then used alongside few labeled samples of an unseen malware type. This technique is designed to acclimate the model to different malware representations, further enabling it to generate robust embeddings for each trained and unseen classes. Following the extraction of embeddings from the LLM, few-shot learning is utilized to enhance performance with minimal labeled data. Our evaluation, which utilized two renowned datasets, focused on identifying malware types within network traffic and Internet of Things (IoT) environments. Our approach shows promising results with an average accuracy of 86.35% and F1-Score of 86.40% on different malware types across the two datasets.

Title: Attacking Slicing Network via Side-channel Reinforcement Learning Attack

• Authors: Wei Shao, Chandra Thapa, Rayne Holland, Sarah Ali Siddiqui, Seyit Camtepe
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11258
• Pdf URL: https://arxiv.org/pdf/2409.11258
• Copy Paste: [[2409.11258]] Attacking Slicing Network via Side-channel Reinforcement Learning Attack(https://arxiv.org/abs/2409.11258)
• Keywords: security, attack, robust
• Abstract: Network slicing in 5G and the future 6G networks will enable the creation of multiple virtualized networks on a shared physical infrastructure. This innovative approach enables the provision of tailored networks to accommodate specific business types or industry users, thus delivering more customized and efficient services. However, the shared memory and cache in network slicing introduce security vulnerabilities that have yet to be fully addressed. In this paper, we introduce a reinforcement learning-based side-channel cache attack framework specifically designed for network slicing environments. Unlike traditional cache attack methods, our framework leverages reinforcement learning to dynamically identify and exploit cache locations storing sensitive information, such as authentication keys and user registration data. We assume that one slice network is compromised and demonstrate how the attacker can induce another shared slice to send registration requests, thereby estimating the cache locations of critical data. By formulating the cache timing channel attack as a reinforcement learning-driven guessing game between the attack slice and the victim slice, our model efficiently explores possible actions to pinpoint memory blocks containing sensitive information. Experimental results showcase the superiority of our approach, achieving a success rate of approximately 95\% to 98\% in accurately identifying the storage locations of sensitive data. This high level of accuracy underscores the potential risks in shared network slicing environments and highlights the need for robust security measures to safeguard against such advanced side-channel attacks.

Title: The Art of Storytelling: Multi-Agent Generative AI for Dynamic Multimodal Narratives

• Authors: Samee Arif, Taimoor Arif, Aamina Jamal Khan, Muhammad Saad Haroon, Agha Ali Raza, Awais Athar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11261
• Pdf URL: https://arxiv.org/pdf/2409.11261
• Copy Paste: [[2409.11261]] The Art of Storytelling: Multi-Agent Generative AI for Dynamic Multimodal Narratives(https://arxiv.org/abs/2409.11261)
• Keywords: generative
• Abstract: This paper introduces the concept of an education tool that utilizes Generative Artificial Intelligence (GenAI) to enhance storytelling for children. The system combines GenAI-driven narrative co-creation, text-to-speech conversion, and text-to-video generation to produce an engaging experience for learners. We describe the co-creation process, the adaptation of narratives into spoken words using text-to-speech models, and the transformation of these narratives into contextually relevant visuals through text-to-video technology. Our evaluation covers the linguistics of the generated stories, the text-to-speech conversion quality, and the accuracy of the generated visuals.

Title: LOLA -- An Open-Source Massively Multilingual Large Language Model

• Authors: Nikit Srivastava, Denis Kuchelev, Tatiana Moteu, Kshitij Shetty, Michael Roeder, Diego Moussallem, Hamada Zahera, Axel-Cyrille Ngonga Ngomo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11272
• Pdf URL: https://arxiv.org/pdf/2409.11272
• Copy Paste: [[2409.11272]] LOLA -- An Open-Source Massively Multilingual Large Language Model(https://arxiv.org/abs/2409.11272)
• Keywords: robust, transformer, large language model
• Abstract: This paper presents LOLA, a massively multilingual large language model trained on more than 160 languages using a sparse Mixture-of-Experts Transformer architecture. Our architectural and implementation choices address the challenge of harnessing linguistic diversity while maintaining efficiency and avoiding the common pitfalls of multilinguality. Our analysis of the evaluation results shows competitive performance in natural language generation and understanding tasks. Additionally, we demonstrate how the learned expert-routing mechanism exploits implicit phylogenetic linguistic patterns to potentially alleviate the curse of multilinguality. We provide an in-depth look at the training process, an analysis of the datasets, and a balanced exploration of the model's strengths and limitations. As an open-source model, LOLA promotes reproducibility and serves as a robust foundation for future research. Our findings enable the development of compute-efficient multilingual models with strong, scalable performance across languages.

Title: Task Arithmetic for Language Expansion in Speech Translation

• Authors: Yao-Fei Cheng, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Wen Shen Teo, Siddhant Arora, Shinji Watanabe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11274
• Pdf URL: https://arxiv.org/pdf/2409.11274
• Copy Paste: [[2409.11274]] Task Arithmetic for Language Expansion in Speech Translation(https://arxiv.org/abs/2409.11274)
• Keywords: large language model
• Abstract: Recent advances in large language models (LLMs) have gained interest in speech-text multimodal foundation models, achieving strong performance on instruction-based speech translation (ST). However, expanding language pairs from an existing instruction-tuned ST system is costly due to the necessity of re-training on a combination of new and previous datasets. We propose to expand new language pairs by merging the model trained on new language pairs and the existing model, using task arithmetic. We find that the direct application of task arithmetic for ST causes the merged model to fail to follow instructions; thus, generating translation in incorrect languages. To eliminate language confusion, we propose an augmented task arithmetic method that merges an additional language control model. It is trained to generate the correct target language token following the instructions. Our experiments demonstrate that our proposed language control model can achieve language expansion by eliminating language confusion. In our MuST-C and CoVoST-2 experiments, it shows up to 4.66 and 4.92 BLEU scores improvement, respectively. In addition, we demonstrate the use of our task arithmetic framework can expand to a language pair where neither paired ST training data nor a pre-trained ST model is available. We first synthesize the ST system from machine translation (MT) systems via task analogy, then merge the synthesized ST system to the existing ST model.

Title: Hackphyr: A Local Fine-Tuned LLM Agent for Network Security Environments

• Authors: Maria Rigaki, Carlos Catania, Sebastian Garcia
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11276
• Pdf URL: https://arxiv.org/pdf/2409.11276
• Copy Paste: [[2409.11276]] Hackphyr: A Local Fine-Tuned LLM Agent for Network Security Environments(https://arxiv.org/abs/2409.11276)
• Keywords: security, privacy, large language model
• Abstract: Large Language Models (LLMs) have shown remarkable potential across various domains, including cybersecurity. Using commercial cloud-based LLMs may be undesirable due to privacy concerns, costs, and network connectivity constraints. In this paper, we present Hackphyr, a locally fine-tuned LLM to be used as a red-team agent within network security environments. Our fine-tuned 7 billion parameter model can run on a single GPU card and achieves performance comparable with much larger and more powerful commercial models such as GPT-4. Hackphyr clearly outperforms other models, including GPT-3.5-turbo, and baselines, such as Q-learning agents in complex, previously unseen scenarios. To achieve this performance, we generated a new task-specific cybersecurity dataset to enhance the base model's capabilities. Finally, we conducted a comprehensive analysis of the agents' behaviors that provides insights into the planning abilities and potential shortcomings of such agents, contributing to the broader understanding of LLM-based agents in cybersecurity contexts

Title: Leveraging Distillation Techniques for Document Understanding: A Case Study with FLAN-T5

• Authors: Marcel Lamott, Muhammad Armaghan Shakir
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11282
• Pdf URL: https://arxiv.org/pdf/2409.11282
• Copy Paste: [[2409.11282]] Leveraging Distillation Techniques for Document Understanding: A Case Study with FLAN-T5(https://arxiv.org/abs/2409.11282)
• Keywords: large language model
• Abstract: The surge of digital documents in various formats, including less standardized documents such as business reports and environmental assessments, underscores the growing importance of Document Understanding. While Large Language Models (LLMs) have showcased prowess across diverse natural language processing tasks, their direct application to Document Understanding remains a challenge. Previous research has demonstrated the utility of LLMs in this domain, yet their significant computational demands make them challenging to deploy effectively. Additionally, proprietary Blackbox LLMs often outperform their open-source counterparts, posing a barrier to widespread accessibility. In this paper, we delve into the realm of document understanding, leveraging distillation methods to harness the power of large LLMs while accommodating computational limitations. Specifically, we present a novel approach wherein we distill document understanding knowledge from the proprietary LLM ChatGPT into FLAN-T5. Our methodology integrates labeling and curriculum-learning mechanisms to facilitate efficient knowledge transfer. This work contributes to the advancement of document understanding methodologies by offering a scalable solution that bridges the gap between resource-intensive LLMs and practical applications. Our findings underscore the potential of distillation techniques in facilitating the deployment of sophisticated language models in real-world scenarios, thereby fostering advancements in natural language processing and document comprehension domains.

Title: Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling

• Authors: Xinyue Fang, Zhen Huang, Zhiliang Tian, Minghui Fang, Ziyi Pan, Quntian Fang, Zhihua Wen, Hengyue Pan, Dongsheng Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11283
• Pdf URL: https://arxiv.org/pdf/2409.11283
• Copy Paste: [[2409.11283]] Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling(https://arxiv.org/abs/2409.11283)
• Keywords: segmentation
• Abstract: LLMs obtain remarkable performance but suffer from hallucinations. Most research on detecting hallucination focuses on the questions with short and concrete correct answers that are easy to check the faithfulness. Hallucination detections for text generation with open-ended answers are more challenging. Some researchers use external knowledge to detect hallucinations in generated texts, but external resources for specific scenarios are hard to access. Recent studies on detecting hallucinations in long text without external resources conduct consistency comparison among multiple sampled outputs. To handle long texts, researchers split long texts into multiple facts and individually compare the consistency of each pairs of facts. However, these methods (1) hardly achieve alignment among multiple facts; (2) overlook dependencies between multiple contextual facts. In this paper, we propose a graph-based context-aware (GCA) hallucination detection for text generations, which aligns knowledge facts and considers the dependencies between contextual knowledge triples in consistency comparison. Particularly, to align multiple facts, we conduct a triple-oriented response segmentation to extract multiple knowledge triples. To model dependencies among contextual knowledge triple (facts), we construct contextual triple into a graph and enhance triples' interactions via message passing and aggregating via RGCN. To avoid the omission of knowledge triples in long text, we conduct a LLM-based reverse verification via reconstructing the knowledge triples. Experiments show that our model enhances hallucination detection and excels all baselines.

Title: EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage

• Authors: Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, Huan Sun
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11295
• Pdf URL: https://arxiv.org/pdf/2409.11295
• Copy Paste: [[2409.11295]] EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage(https://arxiv.org/abs/2409.11295)
• Keywords: security, privacy, defense, attack, steal
• Abstract: Generalist web agents have evolved rapidly and demonstrated remarkable potential. However, there are unprecedented safety risks associated with these them, which are nearly unexplored so far. In this work, we aim to narrow this gap by conducting the first study on the privacy risks of generalist web agents in adversarial environments. First, we present a threat model that discusses the adversarial targets, constraints, and attack scenarios. Particularly, we consider two types of adversarial targets: stealing users' specific personally identifiable information (PII) or stealing the entire user request. To achieve these objectives, we propose a novel attack method, termed Environmental Injection Attack (EIA). This attack injects malicious content designed to adapt well to different environments where the agents operate, causing them to perform unintended actions. This work instantiates EIA specifically for the privacy scenario. It inserts malicious web elements alongside persuasive instructions that mislead web agents into leaking private information, and can further leverage CSS and JavaScript features to remain stealthy. We collect 177 actions steps that involve diverse PII categories on realistic websites from the Mind2Web dataset, and conduct extensive experiments using one of the most capable generalist web agent frameworks to date, SeeAct. The results demonstrate that EIA achieves up to 70% ASR in stealing users' specific PII. Stealing full user requests is more challenging, but a relaxed version of EIA can still achieve 16% ASR. Despite these concerning results, it is important to note that the attack can still be detectable through careful human inspection, highlighting a trade-off between high autonomy and security. This leads to our detailed discussion on the efficacy of EIA under different levels of human supervision as well as implications on defenses for generalist web agents.

Title: Decentralized Biometric Authentication based on Fuzzy Commitments and Blockchain

• Authors: Nibras Abo Alzahab, Giulia Rafaiani, Massimo Battaglioni, Franco Chiaraluce, Marco Baldi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11303
• Pdf URL: https://arxiv.org/pdf/2409.11303
• Copy Paste: [[2409.11303]] Decentralized Biometric Authentication based on Fuzzy Commitments and Blockchain(https://arxiv.org/abs/2409.11303)
• Keywords: security, privacy, protect, attack, biometric
• Abstract: Blockchain technology, which was introduced for supporting cryptocurrencies, today provides a decentralized infrastructure for general information storage and execution of algorithms, thus enabling the conversion of many applications and services from a centralized and intermediated model to a decentralized and disintermediated one. In this paper we focus on biometric authentication, which is classically performed using centralized systems, and could hence benefit from decentralization. For such a purpose, however, an inherent contradiction between biometric applications and blockchain technology must be overcome, as the former require keeping biometric features private, while blockchain is a public infrastructure. We propose a blockchain-based biometric authentication protocol that enables decentralization and resilience while protecting the privacy, personal data, and, in particular, biometric features of users. The protocol we propose leverages fuzzy commitment schemes to allow biometric authentication to be performed without disclosing biometric data. We also analyze the security of the protocol we propose by considering some relevant attacks.

Title: GS-Net: Generalizable Plug-and-Play 3D Gaussian Splatting Module

• Authors: Yichen Zhang, Zihan Wang, Jiali Han, Peilin Li, Jiaxun Zhang, Jianqiang Wang, Lei He, Keqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11307
• Pdf URL: https://arxiv.org/pdf/2409.11307
• Copy Paste: [[2409.11307]] GS-Net: Generalizable Plug-and-Play 3D Gaussian Splatting Module(https://arxiv.org/abs/2409.11307)
• Keywords: robust
• Abstract: 3D Gaussian Splatting (3DGS) integrates the strengths of primitive-based representations and volumetric rendering techniques, enabling real-time, high-quality rendering. However, 3DGS models typically overfit to single-scene training and are highly sensitive to the initialization of Gaussian ellipsoids, heuristically derived from Structure from Motion (SfM) point clouds, which limits both generalization and practicality. To address these limitations, we propose GS-Net, a generalizable, plug-and-play 3DGS module that densifies Gaussian ellipsoids from sparse SfM point clouds, enhancing geometric structure representation. To the best of our knowledge, GS-Net is the first plug-and-play 3DGS module with cross-scene generalization capabilities. Additionally, we introduce the CARLA-NVS dataset, which incorporates additional camera viewpoints to thoroughly evaluate reconstruction and rendering quality. Extensive experiments demonstrate that applying GS-Net to 3DGS yields a PSNR improvement of 2.08 dB for conventional viewpoints and 1.86 dB for novel viewpoints, confirming the method's effectiveness and robustness.

Title: SpMis: An Investigation of Synthetic Spoken Misinformation Detection

• Authors: Peizhuo Liu, Li Wang, Renqiang He, Haorui He, Lei Wang, Huadi Zheng, Jie Shi, Tong Xiao, Zhizheng Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11308
• Pdf URL: https://arxiv.org/pdf/2409.11308
• Copy Paste: [[2409.11308]] SpMis: An Investigation of Synthetic Spoken Misinformation Detection(https://arxiv.org/abs/2409.11308)
• Keywords: generative
• Abstract: In recent years, speech generation technology has advanced rapidly, fueled by generative models and large-scale training techniques. While these developments have enabled the production of high-quality synthetic speech, they have also raised concerns about the misuse of this technology, particularly for generating synthetic misinformation. Current research primarily focuses on distinguishing machine-generated speech from human-produced speech, but the more urgent challenge is detecting misinformation within spoken content. This task requires a thorough analysis of factors such as speaker identity, topic, and synthesis. To address this need, we conduct an initial investigation into synthetic spoken misinformation detection by introducing an open-source dataset, SpMis. SpMis includes speech synthesized from over 1,000 speakers across five common topics, utilizing state-of-the-art text-to-speech systems. Although our results show promising detection capabilities, they also reveal substantial challenges for practical implementation, underscoring the importance of ongoing research in this critical area.

Title: fMRI-3D: A Comprehensive Dataset for Enhancing fMRI-based 3D Reconstruction

• Authors: Jianxiong Gao, Yuqian Fu, Yun Wang, Xuelin Qian, Jianfeng Feng, Yanwei Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11315
• Pdf URL: https://arxiv.org/pdf/2409.11315
• Copy Paste: [[2409.11315]] fMRI-3D: A Comprehensive Dataset for Enhancing fMRI-based 3D Reconstruction(https://arxiv.org/abs/2409.11315)
• Keywords: diffusion, transformer, generative
• Abstract: Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI) data, introduced as Recon3DMind in our conference work, is of significant interest to both cognitive neuroscience and computer vision. To advance this task, we present the fMRI-3D dataset, which includes data from 15 participants and showcases a total of 4768 3D objects. The dataset comprises two components: fMRI-Shape, previously introduced and accessible at this https URL, and fMRI-Objaverse, proposed in this paper and available at this https URL. fMRI-Objaverse includes data from 5 subjects, 4 of whom are also part of the Core set in fMRI-Shape, with each subject viewing 3142 3D objects across 117 categories, all accompanied by text captions. This significantly enhances the diversity and potential applications of the dataset. Additionally, we propose MinD-3D, a novel framework designed to decode 3D visual information from fMRI signals. The framework first extracts and aggregates features from fMRI data using a neuro-fusion encoder, then employs a feature-bridge diffusion model to generate visual features, and finally reconstructs the 3D object using a generative transformer decoder. We establish new benchmarks by designing metrics at both semantic and structural levels to evaluate model performance. Furthermore, we assess our model's effectiveness in an Out-of-Distribution setting and analyze the attribution of the extracted features and the visual ROIs in fMRI signals. Our experiments demonstrate that MinD-3D not only reconstructs 3D objects with high semantic and spatial accuracy but also deepens our understanding of how human brain processes 3D visual information. Project page at: this https URL.

Title: MSDNet: Multi-Scale Decoder for Few-Shot Semantic Segmentation via Transformer-Guided Prototyping

• Authors: Amirreza Fateh, Mohammad Reza Mohammadi, Mohammad Reza Jahed Motlagh
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11316
• Pdf URL: https://arxiv.org/pdf/2409.11316
• Copy Paste: [[2409.11316]] MSDNet: Multi-Scale Decoder for Few-Shot Semantic Segmentation via Transformer-Guided Prototyping(https://arxiv.org/abs/2409.11316)
• Keywords: transformer, segmentation
• Abstract: Few-shot Semantic Segmentation addresses the challenge of segmenting objects in query images with only a handful of annotated examples. However, many previous state-of-the-art methods either have to discard intricate local semantic features or suffer from high computational complexity. To address these challenges, we propose a new Few-shot Semantic Segmentation framework based on the transformer architecture. Our approach introduces the spatial transformer decoder and the contextual mask generation module to improve the relational understanding between support and query images. Moreover, we introduce a multi-scale decoder to refine the segmentation mask by incorporating features from different resolutions in a hierarchical manner. Additionally, our approach integrates global features from intermediate encoder stages to improve contextual understanding, while maintaining a lightweight structure to reduce complexity. This balance between performance and efficiency enables our method to achieve state-of-the-art results on benchmark datasets such as $PASCAL-5^i$ and $COCO-20^i$ in both 1-shot and 5-shot settings. Notably, our model with only 1.5 million parameters demonstrates competitive performance while overcoming limitations of existing methodologies. this https URL

Title: LPT++: Efficient Training on Mixture of Long-tailed Experts

• Authors: Bowen Dong, Pan Zhou, Wangmeng Zuo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11323
• Pdf URL: https://arxiv.org/pdf/2409.11323
• Copy Paste: [[2409.11323]] LPT++: Efficient Training on Mixture of Long-tailed Experts(https://arxiv.org/abs/2409.11323)
• Keywords: transformer
• Abstract: We introduce LPT++, a comprehensive framework for long-tailed classification that combines parameter-efficient fine-tuning (PEFT) with a learnable model ensemble. LPT++ enhances frozen Vision Transformers (ViTs) through the integration of three core components. The first is a universal long-tailed adaptation module, which aggregates long-tailed prompts and visual adapters to adapt the pretrained model to the target domain, meanwhile improving its discriminative ability. The second is the mixture of long-tailed experts framework with a mixture-of-experts (MoE) scorer, which adaptively calculates reweighting coefficients for confidence scores from both visual-only and visual-language (VL) model experts to generate more accurate predictions. Finally, LPT++ employs a three-phase training framework, wherein each critical module is learned separately, resulting in a stable and effective long-tailed classification training paradigm. Besides, we also propose the simple version of LPT++ namely LPT, which only integrates visual-only pretrained ViT and long-tailed prompts to formulate a single model method. LPT can clearly illustrate how long-tailed prompts works meanwhile achieving comparable performance without VL pretrained models. Experiments show that, with only ~1% extra trainable parameters, LPT++ achieves comparable accuracy against all the counterparts.

Title: TopoMaskV2: Enhanced Instance-Mask-Based Formulation for the Road Topology Problem

• Authors: M. Esat Kalfaoglu, Halil Ibrahim Ozturk, Ozsel Kilinc, Alptekin Temizel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11325
• Pdf URL: https://arxiv.org/pdf/2409.11325
• Copy Paste: [[2409.11325]] TopoMaskV2: Enhanced Instance-Mask-Based Formulation for the Road Topology Problem(https://arxiv.org/abs/2409.11325)
• Keywords: transformer
• Abstract: Recently, the centerline has become a popular representation of lanes due to its advantages in solving the road topology problem. To enhance centerline prediction, we have developed a new approach called TopoMask. Unlike previous methods that rely on keypoints or parametric methods, TopoMask utilizes an instance-mask-based formulation coupled with a masked-attention-based transformer architecture. We introduce a quad-direction label representation to enrich the mask instances with flow information and design a corresponding post-processing technique for mask-to-centerline conversion. Additionally, we demonstrate that the instance-mask formulation provides complementary information to parametric Bezier regressions, and fusing both outputs leads to improved detection and topology performance. Moreover, we analyze the shortcomings of the pillar assumption in the Lift Splat technique and adapt a multi-height bin configuration. Experimental results show that TopoMask achieves state-of-the-art performance in the OpenLane-V2 dataset, increasing from 44.1 to 49.4 for Subset-A and 44.7 to 51.8 for Subset-B in the V1.1 OLS baseline.

Title: CLIP Adaptation by Intra-modal Overlap Reduction

• Authors: Alexey Kravets, Vinay Namboodiri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11338
• Pdf URL: https://arxiv.org/pdf/2409.11338
• Copy Paste: [[2409.11338]] CLIP Adaptation by Intra-modal Overlap Reduction(https://arxiv.org/abs/2409.11338)
• Keywords: robust
• Abstract: Numerous methods have been proposed to adapt a pre-trained foundational CLIP model for few-shot classification. As CLIP is trained on a large corpus, it generalises well through adaptation to few-shot classification. In this work, we analyse the intra-modal overlap in image space in terms of embedding representation. Our analysis shows that, due to contrastive learning, embeddings from CLIP model exhibit high cosine similarity distribution overlap in the image space between paired and unpaired examples affecting the performance of few-shot training-free classification methods which rely on similarity in the image space for their predictions. To tackle intra-modal overlap we propose to train a lightweight adapter on a generic set of samples from the Google Open Images dataset demonstrating that this improves accuracy for few-shot training-free classification. We validate our contribution through extensive empirical analysis and demonstrate that reducing the intra-modal overlap leads to a) improved performance on a number of standard datasets, b) increased robustness to distribution shift and c) higher feature variance rendering the features more discriminative for downstream tasks.

Title: OmniGen: Unified Image Generation

• Authors: Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, Zheng Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11340
• Pdf URL: https://arxiv.org/pdf/2409.11340
• Copy Paste: [[2409.11340]] OmniGen: Unified Image Generation(https://arxiv.org/abs/2409.11340)
• Keywords: diffusion
• Abstract: In this work, we introduce OmniGen, a new diffusion model for unified image generation. Unlike popular diffusion models (e.g., Stable Diffusion), OmniGen no longer requires additional modules such as ControlNet or IP-Adapter to process diverse control conditions. OmniGenis characterized by the following features: 1) Unification: OmniGen not only demonstrates text-to-image generation capabilities but also inherently supports other downstream tasks, such as image editing, subject-driven generation, and visual-conditional generation. Additionally, OmniGen can handle classical computer vision tasks by transforming them into image generation tasks, such as edge detection and human pose recognition. 2) Simplicity: The architecture of OmniGen is highly simplified, eliminating the need for additional text encoders. Moreover, it is more user-friendly compared to existing diffusion models, enabling complex tasks to be accomplished through instructions without the need for extra preprocessing steps (e.g., human pose estimation), thereby significantly simplifying the workflow of image generation. 3) Knowledge Transfer: Through learning in a unified format, OmniGen effectively transfers knowledge across different tasks, manages unseen tasks and domains, and exhibits novel capabilities. We also explore the model's reasoning capabilities and potential applications of chain-of-thought mechanism. This work represents the first attempt at a general-purpose image generation model, and there remain several unresolved issues. We will open-source the related resources at this https URL to foster advancements in this field.

Title: THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models

• Authors: Mengfei Liang, Archish Arun, Zekun Wu, Cristian Munoz, Jonathan Lutch, Emre Kazim, Adriano Koshiyama, Philip Treleaven
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11353
• Pdf URL: https://arxiv.org/pdf/2409.11353
• Copy Paste: [[2409.11353]] THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models(https://arxiv.org/abs/2409.11353)
• Keywords: large language model
• Abstract: Hallucination, the generation of factually incorrect content, is a growing challenge in Large Language Models (LLMs). Existing detection and mitigation methods are often isolated and insufficient for domain-specific needs, lacking a standardized pipeline. This paper introduces THaMES (Tool for Hallucination Mitigations and EvaluationS), an integrated framework and library addressing this gap. THaMES offers an end-to-end solution for evaluating and mitigating hallucinations in LLMs, featuring automated test set generation, multifaceted benchmarking, and adaptable mitigation strategies. It automates test set creation from any corpus, ensuring high data quality, diversity, and cost-efficiency through techniques like batch processing, weighted sampling, and counterfactual validation. THaMES assesses a model's ability to detect and reduce hallucinations across various tasks, including text generation and binary classification, applying optimal mitigation strategies like In-Context Learning (ICL), Retrieval Augmented Generation (RAG), and Parameter-Efficient Fine-tuning (PEFT). Evaluations of state-of-the-art LLMs using a knowledge base of academic papers, political news, and Wikipedia reveal that commercial models like GPT-4o benefit more from RAG than ICL, while open-weight models like Llama-3.1-8B-Instruct and Mistral-Nemo gain more from ICL. Additionally, PEFT significantly enhances the performance of Llama-3.1-8B-Instruct in both evaluation tasks.

Title: Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think

• Authors: Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt, Daan de Geus, Alexander Hermans, Bastian Leibe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11355
• Pdf URL: https://arxiv.org/pdf/2409.11355
• Copy Paste: [[2409.11355]] Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think(https://arxiv.org/abs/2409.11355)
• Keywords: diffusion
• Abstract: Recent work showed that large diffusion models can be reused as highly precise monocular depth estimators by casting depth estimation as an image-conditional image generation task. While the proposed model achieved state-of-the-art results, high computational demands due to multi-step inference limited its use in many scenarios. In this paper, we show that the perceived inefficiency was caused by a flaw in the inference pipeline that has so far gone unnoticed. The fixed model performs comparably to the best previously reported configuration while being more than 200$\times$ faster. To optimize for downstream task performance, we perform end-to-end fine-tuning on top of the single-step model with task-specific losses and get a deterministic model that outperforms all other diffusion-based depth and normal estimation models on common zero-shot benchmarks. We surprisingly find that this fine-tuning protocol also works directly on Stable Diffusion and achieves comparable performance to current state-of-the-art diffusion-based depth and normal estimation models, calling into question some of the conclusions drawn from prior works.

Title: RenderWorld: World Model with Self-Supervised 3D Label

• Authors: Ziyang Yan, Wenzhen Dong, Yihua Shao, Yuhang Lu, Liu Haiyang, Jingwen Liu, Haozhe Wang, Zhe Wang, Yan Wang, Fabio Remondino, Yuexin Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11356
• Pdf URL: https://arxiv.org/pdf/2409.11356
• Copy Paste: [[2409.11356]] RenderWorld: World Model with Self-Supervised 3D Label(https://arxiv.org/abs/2409.11356)
• Keywords: robust, segmentation
• Abstract: End-to-end autonomous driving with vision-only is not only more cost-effective compared to LiDAR-vision fusion but also more reliable than traditional methods. To achieve a economical and robust purely visual autonomous driving system, we propose RenderWorld, a vision-only end-to-end autonomous driving framework, which generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes the labels by AM-VAE, and uses world model for forecasting and planning. RenderWorld employs Gaussian Splatting to represent 3D scenes and render 2D images greatly improves segmentation accuracy and reduces GPU memory consumption compared with NeRF-based methods. By applying AM-VAE to encode air and non-air separately, RenderWorld achieves more fine-grained scene element representation, leading to state-of-the-art performance in both 4D occupancy forecasting and motion planning from autoregressive world model.

Title: CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration

• Authors: Jiahui Gao, Renjie Pi, Tianyang Han, Han Wu, Lanqing Hong, Lingpeng Kong, Xin Jiang, Zhenguo Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11365
• Pdf URL: https://arxiv.org/pdf/2409.11365
• Copy Paste: [[2409.11365]] CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration(https://arxiv.org/abs/2409.11365)
• Keywords: large language model
• Abstract: The deployment of multimodal large language models (MLLMs) has demonstrated remarkable success in engaging in conversations involving visual inputs, thanks to the superior power of large language models (LLMs). Those MLLMs are typically built based on the LLMs, with an image encoder to process images into the token embedding space of the LLMs. However, the integration of visual modality has introduced a unique vulnerability: the MLLM becomes susceptible to malicious visual inputs and prone to generating sensitive or harmful responses, even though the LLM has been trained on textual dataset to align with human value. In this paper, we first raise the question: ``Do the MLLMs possess safety-awareness against malicious image inputs?". We find that after adding a principle that specifies the safety requirement into the input of the MLLM, the model's safety awareness becomes boosted. This phenomenon verifies the existence of MLLM's safety-awareness against image inputs, it is only weakened by the modality gap. We then introduce a simple yet effective technique termed CoCA, which amplifies the safety-awareness of the MLLM by calibrating its output distribution. Our proposed strategy helps the model reclaim its original safety awareness without losing its original capabilities. We verify the effectiveness of our approach on both multimodal safety and understanding benchmarks.

Title: OSV: One Step is Enough for High-Quality Image to Video Generation

• Authors: Xiaofeng Mao, Zhengkai Jiang, Fu-Yun Wang, Wenbing Zhu, Jiangning Zhang, Hao Chen, Mingmin Chi, Yabiao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11367
• Pdf URL: https://arxiv.org/pdf/2409.11367
• Copy Paste: [[2409.11367]] OSV: One Step is Enough for High-Quality Image to Video Generation(https://arxiv.org/abs/2409.11367)
• Keywords: diffusion
• Abstract: Video diffusion models have shown great potential in generating high-quality videos, making them an increasingly popular focus. However, their inherent iterative nature leads to substantial computational and time costs. While efforts have been made to accelerate video diffusion by reducing inference steps (through techniques like consistency distillation) and GAN training (these approaches often fall short in either performance or training stability). In this work, we introduce a two-stage training framework that effectively combines consistency distillation with GAN training to address these challenges. Additionally, we propose a novel video discriminator design, which eliminates the need for decoding the video latents and improves the final performance. Our model is capable of producing high-quality videos in merely one-step, with the flexibility to perform multi-step refinement for further performance enhancement. Our quantitative evaluation on the OpenWebVid-1M benchmark shows that our model significantly outperforms existing methods. Notably, our 1-step performance(FVD 171.15) exceeds the 8-step performance of the consistency distillation based method, AnimateLCM (FVD 184.79), and approaches the 25-step performance of advanced Stable Video Diffusion (FVD 156.94).

Title: Uncertainty and Prediction Quality Estimation for Semantic Segmentation via Graph Neural Networks

• Authors: Edgar Heinert, Stephan Tilgner, Timo Palm, Matthias Rottmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11373
• Pdf URL: https://arxiv.org/pdf/2409.11373
• Copy Paste: [[2409.11373]] Uncertainty and Prediction Quality Estimation for Semantic Segmentation via Graph Neural Networks(https://arxiv.org/abs/2409.11373)
• Keywords: segmentation
• Abstract: When employing deep neural networks (DNNs) for semantic segmentation in safety-critical applications like automotive perception or medical imaging, it is important to estimate their performance at runtime, e.g. via uncertainty estimates or prediction quality estimates. Previous works mostly performed uncertainty estimation on pixel-level. In a line of research, a connected-component-wise (segment-wise) perspective was taken, approaching uncertainty estimation on an object-level by performing so-called meta classification and regression to estimate uncertainty and prediction quality, respectively. In those works, each predicted segment is considered individually to estimate its uncertainty or prediction quality. However, the neighboring segments may provide additional hints on whether a given predicted segment is of high quality, which we study in the present work. On the basis of uncertainty indicating metrics on segment-level, we use graph neural networks (GNNs) to model the relationship of a given segment's quality as a function of the given segment's metrics as well as those of its neighboring segments. We compare different GNN architectures and achieve a notable performance improvement.

Title: Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification

• Authors: Fatema-E- Jannat, Sina Gholami, Jennifer I. Lim, Theodore Leng, Minhaj Nur Alam, Hamed Tabkhi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11375
• Pdf URL: https://arxiv.org/pdf/2409.11375
• Copy Paste: [[2409.11375]] Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification(https://arxiv.org/abs/2409.11375)
• Keywords: privacy, robust, large language model
• Abstract: In the medical domain, acquiring large datasets poses significant challenges due to privacy concerns. Nonetheless, the development of a robust deep-learning model for retinal disease diagnosis necessitates a substantial dataset for training. The capacity to generalize effectively on smaller datasets remains a persistent challenge. The scarcity of data presents a significant barrier to the practical implementation of scalable medical AI solutions. To address this issue, we've combined a wide range of data sources to improve performance and generalization to new data by giving it a deeper understanding of the data representation from multi-modal datasets and developed a self-supervised framework based on large language models (LLMs), SwinV2 to gain a deeper understanding of multi-modal dataset representations, enhancing the model's ability to extrapolate to new data for the detection of eye diseases using optical coherence tomography (OCT) images. We adopt a two-phase training methodology, self-supervised pre-training, and fine-tuning on a downstream supervised classifier. An ablation study conducted across three datasets employing various encoder backbones, without data fusion, with low data availability setting, and without self-supervised pre-training scenarios, highlights the robustness of our method. Our findings demonstrate consistent performance across these diverse conditions, showcasing superior generalization capabilities compared to the baseline model, ResNet-50.

Title: Towards Time Series Reasoning with LLMs

• Authors: Winnie Chow, Lauren Gardiner, Haraldur T. Hallgrímsson, Maxwell A. Xu, Shirley You Ren
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11376
• Pdf URL: https://arxiv.org/pdf/2409.11376
• Copy Paste: [[2409.11376]] Towards Time Series Reasoning with LLMs(https://arxiv.org/abs/2409.11376)
• Keywords: large language model
• Abstract: Multi-modal large language models (MLLMs) have enabled numerous advances in understanding and reasoning in domains like vision, but we have not yet seen this broad success for time-series. Although prior works on time-series MLLMs have shown promising performance in time-series forecasting, very few works show how an LLM could be used for time-series reasoning in natural language. We propose a novel multi-modal time-series LLM approach that learns generalizable information across various domains with powerful zero-shot performance. First, we train a lightweight time-series encoder on top of an LLM to directly extract time-series information. Then, we fine-tune our model with chain-of-thought augmented time-series tasks to encourage the model to generate reasoning paths. We show that our model learns a latent representation that reflects specific time-series features (e.g. slope, frequency), as well as outperforming GPT-4o on a set of zero-shot reasoning tasks on a variety of domains.

Title: Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement

• Authors: Simon Yu, Liangyu Chen, Sara Ahmadian, Marzieh Fadaee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11378
• Pdf URL: https://arxiv.org/pdf/2409.11378
• Copy Paste: [[2409.11378]] Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement(https://arxiv.org/abs/2409.11378)
• Keywords: large language model
• Abstract: Finetuning large language models on instruction data is crucial for enhancing pre-trained knowledge and improving instruction-following capabilities. As instruction datasets proliferate, selecting optimal data for effective training becomes increasingly important. This work addresses the question: How can we determine the optimal subset of data for effective training? While existing research often emphasizes local criteria like instance quality for subset selection, we argue that a global approach focused on data diversity is more critical. Our method employs k-means clustering to ensure the selected subset effectively represents the full dataset. We propose an iterative refinement method inspired by active learning techniques to resample instances from clusters, reassessing each cluster's importance and sampling weight in every training iteration. This approach reduces the effect of outliers and automatically filters out clusters containing low-quality data. Through extensive evaluation across natural language reasoning, general world knowledge, code and math reasoning tasks, and by fine-tuning models from various families, we observe consistent improvements, achieving a 7% increase over random selection and a 3.8% improvement over state-of-the-art sampling methods. Our work highlights the significance of diversity-first sampling when finetuning LLMs to enhance performance across a broad array of evaluation tasks. Our code is available at this https URL.

Title: Ultrasound Image Enhancement with the Variance of Diffusion Models

• Authors: Yuxin Zhang, Clément Huneau, Jérôme Idier, Diana Mateus
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11380
• Pdf URL: https://arxiv.org/pdf/2409.11380
• Copy Paste: [[2409.11380]] Ultrasound Image Enhancement with the Variance of Diffusion Models(https://arxiv.org/abs/2409.11380)
• Keywords: diffusion
• Abstract: Ultrasound imaging, despite its widespread use in medicine, often suffers from various sources of noise and artifacts that impact the signal-to-noise ratio and overall image quality. Enhancing ultrasound images requires a delicate balance between contrast, resolution, and speckle preservation. This paper introduces a novel approach that integrates adaptive beamforming with denoising diffusion-based variance imaging to address this challenge. By applying Eigenspace-Based Minimum Variance (EBMV) beamforming and employing a denoising diffusion model fine-tuned on ultrasound data, our method computes the variance across multiple diffusion-denoised samples to produce high-quality despeckled images. This approach leverages both the inherent multiplicative noise of ultrasound and the stochastic nature of diffusion models. Experimental results on a publicly available dataset demonstrate the effectiveness of our method in achieving superior image reconstructions from single plane-wave acquisitions. The code is available at: this https URL.

Title: Training Datasets Generation for Machine Learning: Application to Vision Based Navigation

• Authors: Jérémy Lebreton, Ingo Ahrns, Roland Brochard, Christoph Haskamp, Matthieu Le Goff, Nicolas Menga, Nicolas Ollagnier, Ralf Regele, Francesco Capolupo, Massimo Casasco
• Subjects: cs.CV, astro-ph.EP, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11383
• Pdf URL: https://arxiv.org/pdf/2409.11383
• Copy Paste: [[2409.11383]] Training Datasets Generation for Machine Learning: Application to Vision Based Navigation(https://arxiv.org/abs/2409.11383)
• Keywords: robust, generative
• Abstract: Vision Based Navigation consists in utilizing cameras as precision sensors for GNC after extracting information from images. To enable the adoption of machine learning for space applications, one of obstacles is the demonstration that available training datasets are adequate to validate the algorithms. The objective of the study is to generate datasets of images and metadata suitable for training machine learning algorithms. Two use cases were selected and a robust methodology was developed to validate the datasets including the ground truth. The first use case is in-orbit rendezvous with a man-made object: a mockup of satellite ENVISAT. The second use case is a Lunar landing scenario. Datasets were produced from archival datasets (Chang'e 3), from the laboratory at DLR TRON facility and at Airbus Robotic laboratory, from SurRender software high fidelity image simulator using Model Capture and from Generative Adversarial Networks. The use case definition included the selection of algorithms as benchmark: an AI-based pose estimation algorithm and a dense optical flow algorithm were selected. Eventually it is demonstrated that datasets produced with SurRender and selected laboratory facilities are adequate to train machine learning algorithms.

Title: Says Who? Effective Zero-Shot Annotation of Focalization

• Authors: Rebecca M. M. Hicke, Yuri Bizzoni, Pascale Feldkamp, Ross Deans Kristensen-McLachlan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11390
• Pdf URL: https://arxiv.org/pdf/2409.11390
• Copy Paste: [[2409.11390]] Says Who? Effective Zero-Shot Annotation of Focalization(https://arxiv.org/abs/2409.11390)
• Keywords: large language model
• Abstract: Focalization, the perspective through which narrative is presented, is encoded via a wide range of lexico-grammatical features and is subject to reader interpretation. Moreover, trained readers regularly disagree on interpretations, suggesting that this problem may be computationally intractable. In this paper, we provide experiments to test how well contemporary Large Language Models (LLMs) perform when annotating literary texts for focalization mode. Despite the challenging nature of the task, LLMs show comparable performance to trained human annotators in our experiments. We provide a case study working with the novels of Stephen King to demonstrate the usefulness of this approach for computational literary studies, illustrating how focalization can be studied at scale.

Title: NVLM: Open Frontier-Class Multimodal LLMs

• Authors: Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.11402
• Pdf URL: https://arxiv.org/pdf/2409.11402
• Copy Paste: [[2409.11402]] NVLM: Open Frontier-Class Multimodal LLMs(https://arxiv.org/abs/2409.11402)
• Keywords: large language model
• Abstract: We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training. In terms of model design, we perform a comprehensive comparison between decoder-only multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g., Flamingo). Based on the strengths and weaknesses of both approaches, we propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. Furthermore, we introduce a 1-D tile-tagging design for tile-based dynamic high-resolution images, which significantly boosts performance on multimodal reasoning and OCR-related tasks. Regarding training data, we meticulously curate and provide detailed information on our multimodal pretraining and supervised fine-tuning datasets. Our findings indicate that dataset quality and task diversity are more important than scale, even during the pretraining phase, across all architectures. Notably, we develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks while maintaining and even improving text-only performance compared to their LLM backbones. To achieve this, we craft and integrate a high-quality text-only dataset into multimodal training, alongside a substantial amount of multimodal math and reasoning data, leading to enhanced math and coding capabilities across modalities. To advance research in the field, we are releasing the model weights and will open-source the code for the community: this https URL.

Title: AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs

• Authors: Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, Firoj Alam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11404
• Pdf URL: https://arxiv.org/pdf/2409.11404
• Copy Paste: [[2409.11404]] AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs(https://arxiv.org/abs/2409.11404)
• Keywords: large language model
• Abstract: Arabic, with its rich diversity of dialects, remains significantly underrepresented in Large Language Models, particularly in dialectal variations. We address this gap by introducing seven synthetic datasets in dialects alongside Modern Standard Arabic (MSA), created using Machine Translation (MT) combined with human post-editing. We present AraDiCE, a benchmark for Arabic Dialect and Cultural Evaluation. We evaluate LLMs on dialect comprehension and generation, focusing specifically on low-resource Arabic dialects. Additionally, we introduce the first-ever fine-grained benchmark designed to evaluate cultural awareness across the Gulf, Egypt, and Levant regions, providing a novel dimension to LLM evaluation. Our findings demonstrate that while Arabic-specific models like Jais and AceGPT outperform multilingual models on dialectal tasks, significant challenges persist in dialect identification, generation, and translation. This work contributes ~45K post-edited samples, a cultural benchmark, and highlights the importance of tailored training to improve LLM performance in capturing the nuances of diverse Arabic dialects and cultural contexts. We will release the dialectal translation models and benchmarks curated in this study.

Title: Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion

• Authors: Zhenwei Wang, Tengfei Wang, Zexin He, Gerhard Hancke, Ziwei Liu, Rynson W.H. Lau
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11406
• Pdf URL: https://arxiv.org/pdf/2409.11406
• Copy Paste: [[2409.11406]] Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion(https://arxiv.org/abs/2409.11406)
• Keywords: diffusion, generative
• Abstract: In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.