2024-09-25

Title: Global Context Enhanced Anomaly Detection of Cyber Attacks via Decoupled Graph Neural Networks

• Authors: Ahmad Hafez
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15304
• Pdf URL: https://arxiv.org/pdf/2409.15304
• Copy Paste: [[2409.15304]] Global Context Enhanced Anomaly Detection of Cyber Attacks via Decoupled Graph Neural Networks(https://arxiv.org/abs/2409.15304)
• Keywords: attack
• Abstract: Recently, there has been a substantial amount of interest in GNN-based anomaly detection. Existing efforts have focused on simultaneously mastering the node representations and the classifier necessary for identifying abnormalities with relatively shallow models to create an embedding. Therefore, the existing state-of-the-art models are incapable of capturing nonlinear network information and producing suboptimal outcomes. In this thesis, we deploy decoupled GNNs to overcome this issue. Specifically, we decouple the essential node representations and classifier for detecting anomalies. In addition, for node representation learning, we develop a GNN architecture with two modules for aggregating node feature information to produce the final node embedding. Finally, we conduct empirical experiments to verify the effectiveness of our proposed approach. The findings demonstrate that decoupled training along with the global context enhanced representation of the nodes is superior to the state-of-the-art models in terms of AUC and introduces a novel way of capturing the node information.

Title: Visual Prompting in Multimodal Large Language Models: A Survey

• Authors: Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A. Rossi, Ruiyi Zhang, Subrata Mitra, Dimitris N. Metaxas, Lina Yao, Jingbo Shang, Julian McAuley
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15310
• Pdf URL: https://arxiv.org/pdf/2409.15310
• Copy Paste: [[2409.15310]] Visual Prompting in Multimodal Large Language Models: A Survey(https://arxiv.org/abs/2409.15310)
• Keywords: generative, large language model
• Abstract: Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompting methods that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, object referring, and compositional reasoning abilities. In addition, we provide a summary of model training and in-context learning methods to improve MLLM's perception and understanding of visual prompts. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods.

Title: Enhancing coastal water body segmentation with Landsat Irish Coastal Segmentation (LICS) dataset

• Authors: Conor O'Sullivan, Ambrish Kashyap, Seamus Coveney, Xavier Monteys, Soumyabrata Dev
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.15311
• Pdf URL: https://arxiv.org/pdf/2409.15311
• Copy Paste: [[2409.15311]] Enhancing coastal water body segmentation with Landsat Irish Coastal Segmentation (LICS) dataset(https://arxiv.org/abs/2409.15311)
• Keywords: segmentation
• Abstract: Ireland's coastline, a critical and dynamic resource, is facing challenges such as erosion, sedimentation, and human activities. Monitoring these changes is a complex task we approach using a combination of satellite imagery and deep learning methods. However, limited research exists in this area, particularly for Ireland. This paper presents the Landsat Irish Coastal Segmentation (LICS) dataset, which aims to facilitate the development of deep learning methods for coastal water body segmentation while addressing modelling challenges specific to Irish meteorology and coastal types. The dataset is used to evaluate various automated approaches for segmentation, with U-NET achieving the highest accuracy of 95.0% among deep learning methods. Nevertheless, the Normalised Difference Water Index (NDWI) benchmark outperformed U-NET with an average accuracy of 97.2%. The study suggests that deep learning approaches can be further improved with more accurate training data and by considering alternative measurements of erosion. The LICS dataset and code are freely available to support reproducible research and further advancements in coastal monitoring efforts.

Title: Deep Transfer Learning for Breast Cancer Classification

• Authors: Prudence Djagba, J. K. Buwa Mbouobda
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15313
• Pdf URL: https://arxiv.org/pdf/2409.15313
• Copy Paste: [[2409.15313]] Deep Transfer Learning for Breast Cancer Classification(https://arxiv.org/abs/2409.15313)
• Keywords: transformer
• Abstract: Breast cancer is a major global health issue that affects millions of women worldwide. Classification of breast cancer as early and accurately as possible is crucial for effective treatment and enhanced patient outcomes. Deep transfer learning has emerged as a promising technique for improving breast cancer classification by utilizing pre-trained models and transferring knowledge across related tasks. In this study, we examine the use of a VGG, Vision Transformers (ViT) and Resnet to classify images for Invasive Ductal Carcinoma (IDC) cancer and make a comparative analysis of the algorithms. The result shows a great advantage of Resnet-34 with an accuracy of $90.40\%$ in classifying cancer images. However, the pretrained VGG-16 demonstrates a higher F1-score because there is less parameters to update. We believe that the field of breast cancer diagnosis stands to benefit greatly from the use of deep transfer learning. Transfer learning may assist to increase the accuracy and accessibility of breast cancer screening by allowing deep learning models to be trained with little data.

Title: Electrooptical Image Synthesis from SAR Imagery Using Generative Adversarial Networks

• Authors: Grant Rosario, David Noever
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.15331
• Pdf URL: https://arxiv.org/pdf/2409.15331
• Copy Paste: [[2409.15331]] Electrooptical Image Synthesis from SAR Imagery Using Generative Adversarial Networks(https://arxiv.org/abs/2409.15331)
• Keywords: robust, interpretability, transformer, generative
• Abstract: The utility of Synthetic Aperture Radar (SAR) imagery in remote sensing and satellite image analysis is well established, offering robustness under various weather and lighting conditions. However, SAR images, characterized by their unique structural and texture characteristics, often pose interpretability challenges for analysts accustomed to electrooptical (EO) imagery. This application compares state-of-the-art Generative Adversarial Networks (GANs) including Pix2Pix, CycleGan, S-CycleGan, and a novel dual?generator GAN utilizing partial convolutions and a novel dual-generator architecture utilizing transformers. These models are designed to progressively refine the realism in the translated optical images, thereby enhancing the visual interpretability of SAR data. We demonstrate the efficacy of our approach through qualitative and quantitative evaluations, comparing the synthesized EO images with actual EO images in terms of visual fidelity and feature preservation. The results show significant improvements in interpretability, making SAR data more accessible for analysts familiar with EO imagery. Furthermore, we explore the potential of this technology in various applications, including environmental monitoring, urban planning, and military reconnaissance, where rapid, accurate interpretation of SAR data is crucial. Our research contributes to the field of remote sensing by bridging the gap between SAR and EO imagery, offering a novel tool for enhanced data interpretation and broader application of SAR technology in various domains.

Title: Evaluating Large Language Models with Tests of Spanish as a Foreign Language: Pass or Fail?

• Authors: Marina Mayor-Rocher, Nina Melero, Elena Merino-Gómez, María Grandury, Javier Conde, Pedro Reviriego
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15334
• Pdf URL: https://arxiv.org/pdf/2409.15334
• Copy Paste: [[2409.15334]] Evaluating Large Language Models with Tests of Spanish as a Foreign Language: Pass or Fail?(https://arxiv.org/abs/2409.15334)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been profusely evaluated on their ability to answer questions on many topics and their performance on different natural language understanding tasks. Those tests are usually conducted in English, but most LLM users are not native English speakers. Therefore, it is of interest to analyze how LLMs understand other languages at different levels: from paragraphs to morphems. In this paper, we evaluate the performance of state-of-the-art LLMs in TELEIA, a recently released benchmark with similar questions to those of Spanish exams for foreign students, covering topics such as reading comprehension, word formation, meaning and compositional semantics, and grammar. The results show that LLMs perform well at understanding Spanish but are still far from achieving the level of a native speaker in terms of grammatical competence.

Title: Ultrafast vision perception by neuromorphic optical flow

• Authors: Shengbo Wang, Shuo Gao, Tongming Pu, Liangbing Zhao, Arokia Nathan
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.15345
• Pdf URL: https://arxiv.org/pdf/2409.15345
• Copy Paste: [[2409.15345]] Ultrafast vision perception by neuromorphic optical flow(https://arxiv.org/abs/2409.15345)
• Keywords: segmentation
• Abstract: Optical flow is crucial for robotic visual perception, yet current methods primarily operate in a 2D format, capturing movement velocities only in horizontal and vertical dimensions. This limitation results in incomplete motion cues, such as missing regions of interest or detailed motion analysis of different regions, leading to delays in processing high-volume visual data in real-world settings. Here, we report a 3D neuromorphic optical flow method that leverages the time-domain processing capability of memristors to embed external motion features directly into hardware, thereby completing motion cues and dramatically accelerating the computation of movement velocities and subsequent task-specific algorithms. In our demonstration, this approach reduces visual data processing time by an average of 0.3 seconds while maintaining or improving the accuracy of motion prediction, object tracking, and object segmentation. Interframe visual processing is achieved for the first time in UAV scenarios. Furthermore, the neuromorphic optical flow algorithm's flexibility allows seamless integration with existing algorithms, ensuring broad applicability. These advancements open unprecedented avenues for robotic perception, without the trade-off between accuracy and efficiency.

Title: Block-Attention for Low-Latency RAG

• Authors: East Sun, Yan Wang, Lan Tian
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15355
• Pdf URL: https://arxiv.org/pdf/2409.15355
• Copy Paste: [[2409.15355]] Block-Attention for Low-Latency RAG(https://arxiv.org/abs/2409.15355)
• Keywords: segmentation
• Abstract: We introduce Block-Attention, an attention mechanism designed to address the increased inference latency in Retrieval-Augmented Generation (RAG) scenarios. Its main idea lies in dividing the input sequence into blocks, where each block calculates its key-value (KV) states independently except for the final block. In RAG scenarios, by defining each passage as a block, Block-Attention enables us to pre-compute the KV states for all passages and cache them in memory. The implementation involves block segmentation, positional encoding calculation, and fine-tuning the LLM to adapt to the Block-Attention mechanism. Experiments on four RAG benchmarks demonstrate that after block fine-tuning, the Block Attention model can achieve performance comparable to (68.4\% vs 67.9\% on Llama3) or even better (62.8\% vs 59.6\% on Mistral) than self-attention models. Notably, Block-Attention reduces the TTFT to a very low level. It only takes 45 ms to output the first token for an input sequence with a total length of 32K. Compared with the self-attention model, the time consumption is reduced by 98.7\%.

Title: Reward-Robust RLHF in LLMs

• Authors: Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, Yuan Shen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15360
• Pdf URL: https://arxiv.org/pdf/2409.15360
• Copy Paste: [[2409.15360]] Reward-Robust RLHF in LLMs(https://arxiv.org/abs/2409.15360)
• Keywords: robust, large language model
• Abstract: As Large Language Models (LLMs) continue to progress toward more advanced forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is increasingly seen as a key pathway toward achieving Artificial General Intelligence (AGI). However, the reliance on reward-model-based (RM-based) alignment methods introduces significant challenges due to the inherent instability and imperfections of Reward Models (RMs), which can lead to critical issues such as reward hacking and misalignment with human intentions. In this paper, we introduce a reward-robust RLHF framework aimed at addressing these fundamental challenges, paving the way for more reliable and resilient learning in LLMs. Our approach introduces a novel optimization objective that carefully balances performance and robustness by incorporating Bayesian Reward Model Ensembles (BRME) to model the uncertainty set of reward functions. This allows the framework to integrate both nominal performance and minimum reward signals, ensuring more stable learning even with imperfect reward models. Empirical results demonstrate that our framework consistently outperforms traditional RLHF across diverse benchmarks, showing improved accuracy and long-term stability. We also provide a theoretical analysis, demonstrating that reward-robust RLHF approaches the stability of constant reward settings, which proves to be effective in a stochastic-case analysis. Together, these contributions highlight the framework potential to enhance both the performance and stability of LLM alignment with RLHF.

Title: Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning

• Authors: Essa Jan, Nouar AlDahoul, Moiz Ali, Faizan Ahmad, Fareed Zaffar, Yasir Zaki
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15361
• Pdf URL: https://arxiv.org/pdf/2409.15361
• Copy Paste: [[2409.15361]] Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning(https://arxiv.org/abs/2409.15361)
• Keywords: attack, robust, large language model
• Abstract: Recent breakthroughs in Large Language Models (LLMs) have led to their adoption across a wide range of tasks, ranging from code generation to machine translation and sentiment analysis, etc. Red teaming/Safety alignment efforts show that fine-tuning models on benign (non-harmful) data could compromise safety. However, it remains unclear to what extent this phenomenon is influenced by different variables, including fine-tuning task, model calibrations, etc. This paper explores the task-wise safety degradation due to fine-tuning on downstream tasks such as summarization, code generation, translation, and classification across various calibration. Our results reveal that: 1) Fine-tuning LLMs for code generation and translation leads to the highest degradation in safety guardrails. 2) LLMs generally have weaker guardrails for translation and classification, with 73-92% of harmful prompts answered, across baseline and other calibrations, falling into one of two concern categories. 3) Current solutions, including guards and safety tuning datasets, lack cross-task robustness. To address these issues, we developed a new multitask safety dataset effectively reducing attack success rates across a range of tasks without compromising the model's overall helpfulness. Our work underscores the need for generalized alignment measures to ensure safer and more robust models.

Title: VERA: Validation and Enhancement for Retrieval Augmented systems

• Authors: Nitin Aravind Birur, Tanay Baswa, Divyanshu Kumar, Jatan Loya, Sahil Agarwal, Prashanth Harshangi
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.15364
• Pdf URL: https://arxiv.org/pdf/2409.15364
• Copy Paste: [[2409.15364]] VERA: Validation and Enhancement for Retrieval Augmented systems(https://arxiv.org/abs/2409.15364)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) exhibit remarkable capabilities but often produce inaccurate responses, as they rely solely on their embedded knowledge. Retrieval-Augmented Generation (RAG) enhances LLMs by incorporating an external information retrieval system, supplying additional context along with the query to mitigate inaccuracies for a particular context. However, accuracy issues still remain, as the model may rely on irrelevant documents or extrapolate incorrectly from its training knowledge. To assess and improve the performance of both the retrieval system and the LLM in a RAG framework, we propose \textbf{VERA} (\textbf{V}alidation and \textbf{E}nhancement for \textbf{R}etrieval \textbf{A}ugmented systems), a system designed to: 1) Evaluate and enhance the retrieved context before response generation, and 2) Evaluate and refine the LLM-generated response to ensure precision and minimize errors. VERA employs an evaluator-cum-enhancer LLM that first checks if external retrieval is necessary, evaluates the relevance and redundancy of the retrieved context, and refines it to eliminate non-essential information. Post-response generation, VERA splits the response into atomic statements, assesses their relevance to the query, and ensures adherence to the context. Our experiments demonstrate VERA's remarkable efficacy not only in improving the performance of smaller open-source models, but also larger state-of-the art models. These enhancements underscore VERA's potential to produce accurate and relevant responses, advancing the state-of-the-art in retrieval-augmented language modeling. VERA's robust methodology, combining multiple evaluation and refinement steps, effectively mitigates hallucinations and improves retrieval and response processes, making it a valuable tool for applications demanding high accuracy and reliability in information generation. .

Title: Novel Saliency Analysis for the Forward Forward Algorithm

• Authors: Mitra Bakhshi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15365
• Pdf URL: https://arxiv.org/pdf/2409.15365
• Copy Paste: [[2409.15365]] Novel Saliency Analysis for the Forward Forward Algorithm(https://arxiv.org/abs/2409.15365)
• Keywords: robust
• Abstract: Incorporating the Forward Forward algorithm into neural network training represents a transformative shift from traditional methods, introducing a dual forward mechanism that streamlines the learning process by bypassing the complexities of derivative propagation. This method is noted for its simplicity and efficiency and involves executing two forward passes the first with actual data to promote positive reinforcement, and the second with synthetically generated negative data to enable discriminative learning. Our experiments confirm that the Forward Forward algorithm is not merely an experimental novelty but a viable training strategy that competes robustly with conventional multi layer perceptron (MLP) architectures. To overcome the limitations inherent in traditional saliency techniques, which predominantly rely on gradient based methods, we developed a bespoke saliency algorithm specifically tailored for the Forward Forward framework. This innovative algorithm enhances the intuitive understanding of feature importance and network decision-making, providing clear visualizations of the data features most influential in model predictions. By leveraging this specialized saliency method, we gain deeper insights into the internal workings of the model, significantly enhancing our interpretative capabilities beyond those offered by standard approaches. Our evaluations, utilizing the MNIST and Fashion MNIST datasets, demonstrate that our method performs comparably to traditional MLP-based models.

Title: Trajectory Anomaly Detection with Language Models

• Authors: Jonathan Mbuya, Dieter Pfoser, Antonios Anastasopoulos
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15366
• Pdf URL: https://arxiv.org/pdf/2409.15366
• Copy Paste: [[2409.15366]] Trajectory Anomaly Detection with Language Models(https://arxiv.org/abs/2409.15366)
• Keywords: robust
• Abstract: This paper presents a novel approach for trajectory anomaly detection using an autoregressive causal-attention model, termed LM-TAD. This method leverages the similarities between language statements and trajectories, both of which consist of ordered elements requiring coherence through external rules and contextual variations. By treating trajectories as sequences of tokens, our model learns the probability distributions over trajectories, enabling the identification of anomalous locations with high precision. We incorporate user-specific tokens to account for individual behavior patterns, enhancing anomaly detection tailored to user context. Our experiments demonstrate the effectiveness of LM-TAD on both synthetic and real-world datasets. In particular, the model outperforms existing methods on the Pattern of Life (PoL) dataset by detecting user-contextual anomalies and achieves competitive results on the Porto taxi dataset, highlighting its adaptability and robustness. Additionally, we introduce the use of perplexity and surprisal rate metrics for detecting outliers and pinpointing specific anomalous locations within trajectories. The LM-TAD framework supports various trajectory representations, including GPS coordinates, staypoints, and activity types, proving its versatility in handling diverse trajectory data. Moreover, our approach is well-suited for online trajectory anomaly detection, significantly reducing computational latency by caching key-value states of the attention mechanism, thereby avoiding repeated computations.

Title: Fine-Tuning a Time Series Foundation Model with Wasserstein Loss

• Authors: Andrei Chernov
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15367
• Pdf URL: https://arxiv.org/pdf/2409.15367
• Copy Paste: [[2409.15367]] Fine-Tuning a Time Series Foundation Model with Wasserstein Loss(https://arxiv.org/abs/2409.15367)
• Keywords: large language model
• Abstract: Inspired by recent advancements in large language models (LLMs) for Natural Language Processing (NLP), there has been a surge in research focused on developing foundational models for time series forecasting. One approach involves training LLM architectures on tokenized time series data using cross-entropy loss. Although this method has demonstrated promising results, cross-entropy loss is primarily designed for classification tasks and does not account for the distance between classes. To address this limitation, we propose using the Wasserstein loss for such architectures. To validate our approach, we fine-tuned a foundational time series model on $22$ zero-shot datasets, comparing the performance of cross-entropy loss with that of Wasserstein loss. Our results demonstrate that replacing cross-entropy loss with Wasserstein loss significantly improves point estimation.

Title: MedCodER: A Generative AI Assistant for Medical Coding

• Authors: Krishanu Das Baksi, Elijah Soba, John J. Higgins, Ravi Saini, Jaden Wood, Jane Cook, Jack Scott, Nirmala Pudota, Tim Weninger, Edward Bowen, Sanmitra Bhattacharya
• Subjects: cs.CL, cs.AI, cs.ET, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15368
• Pdf URL: https://arxiv.org/pdf/2409.15368
• Copy Paste: [[2409.15368]] MedCodER: A Generative AI Assistant for Medical Coding(https://arxiv.org/abs/2409.15368)
• Keywords: extraction, generative
• Abstract: Medical coding is essential for standardizing clinical data and communication but is often time-consuming and prone to errors. Traditional Natural Language Processing (NLP) methods struggle with automating coding due to the large label space, lengthy text inputs, and the absence of supporting evidence annotations that justify code selection. Recent advancements in Generative Artificial Intelligence (AI) offer promising solutions to these challenges. In this work, we introduce MedCodER, a Generative AI framework for automatic medical coding that leverages extraction, retrieval, and re-ranking techniques as core components. MedCodER achieves a micro-F1 score of 0.60 on International Classification of Diseases (ICD) code prediction, significantly outperforming state-of-the-art methods. Additionally, we present a new dataset containing medical records annotated with disease diagnoses, ICD codes, and supporting evidence texts (this https URL). Ablation tests confirm that MedCodER's performance depends on the integration of each of its aforementioned components, as performance declines when these components are evaluated in isolation.

Title: Smirk: An Atomically Complete Tokenizer for Molecular Foundation Models

• Authors: Alexius Wadell, Anoushka Bhutani, Venkatasubramanian Viswanathan
• Subjects: cs.LG, cs.AI, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2409.15370
• Pdf URL: https://arxiv.org/pdf/2409.15370
• Copy Paste: [[2409.15370]] Smirk: An Atomically Complete Tokenizer for Molecular Foundation Models(https://arxiv.org/abs/2409.15370)
• Keywords: transformer
• Abstract: Molecular Foundation Models are emerging as powerful tools for accelerating molecular design, material science, and cheminformatics, leveraging transformer architectures to speed up the discovery of new materials and drugs while reducing the computational cost of traditional ab initio methods. However, current models are constrained by closed-vocabulary tokenizers that fail to capture the full diversity of molecular structures. In this work, we systematically evaluate thirteen chemistry-specific tokenizers for their coverage of the SMILES language, uncovering substantial gaps. Using N-gram language models, we accessed the impact of tokenizer choice on model performance and quantified the information loss of unknown tokens. We introduce two new tokenizers, smirk and smirk-gpe, which can represent the entirety of the OpenSMILES specification while avoiding the pitfalls of existing tokenizers. Our work highlights the importance of open-vocabulary modeling for molecular foundation models and the need for chemically diverse benchmarks for cheminformatics.

Title: Bone: Block Affine Transformation as Parameter Efficient Fine-tuning Methods for Large Language Models

• Authors: Jiale Kang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15371
• Pdf URL: https://arxiv.org/pdf/2409.15371
• Copy Paste: [[2409.15371]] Bone: Block Affine Transformation as Parameter Efficient Fine-tuning Methods for Large Language Models(https://arxiv.org/abs/2409.15371)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) continue to grow in size, their computational and memory requirements increase correspondingly. Consequently, the exploration of cost-effective and efficient fine-tuning methods has become increasingly important. Low-Rank Adaptation (LoRA) has achieved remarkable training results by freezing the original weights and training only low-rank matrices, establishing itself as the predominant fine-tuning method for LLMs. In pursuit of performance closer to full-parameter training, a series of LoRA variants have emerged, such as LoRA+, PISSA, Olora, and LoRA-GA. However, these methods also make the fine-tuning initialization process more complex, and it remains challenging to surpass the performance ceiling of full fine-tuning. To address these issues, this paper introduces an innovative method called Bone (Block Affine), which not only reduces memory overhead but also emphasizes the internal connections between weights, leading to faster convergence and better data fitting. Experimental comparisons across two different LLM architectures (LLaMA2, RWKV6) and various parameter scales demonstrate that the Bone structure can achieve rapid convergence and superior data fitting without the need for complex initialization. For example, when fine-tuning LLaMA2-7B on the MetaMathQA dataset and validating on GSM8k and math benchmarks, Bone achieved fine-tuning scores of 49.36 and 8.8, respectively, outperforming PISSA by 5.84\% and 1.96\%.

Title: ControlMath: Controllable Data Generation Promotes Math Generalist Models

• Authors: Nuo Chen, Ning Wu, Jianhui Chang, Jia Li
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15376
• Pdf URL: https://arxiv.org/pdf/2409.15376
• Copy Paste: [[2409.15376]] ControlMath: Controllable Data Generation Promotes Math Generalist Models(https://arxiv.org/abs/2409.15376)
• Keywords: large language model
• Abstract: Utilizing large language models (LLMs) for data augmentation has yielded encouraging results in mathematical reasoning. However, these approaches face constraints in problem diversity, potentially restricting them to in-domain/distribution data generation. To this end, we propose ControlMath, an iterative method involving an equation-generator module and two LLM-based agents. The module creates diverse equations, which the Problem-Crafter agent then transforms into math word problems. The Reverse-Agent filters and selects high-quality data, adhering to the "less is more" principle, achieving better results with fewer data points. This approach enables the generation of diverse math problems, not limited to specific domains or distributions. As a result, we collect ControlMathQA, which involves 190k math word problems. Extensive results prove that combining our dataset with in-domain datasets like GSM8K can help improve the model's mathematical ability to generalize, leading to improved performances both within and beyond specific domains.

Title: Prompting Large Language Models for Supporting the Differential Diagnosis of Anemia

• Authors: Elisa Castagnari (HeKA), Lillian Muyama (HeKA), Adrien Coulet (HeKA)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15377
• Pdf URL: https://arxiv.org/pdf/2409.15377
• Copy Paste: [[2409.15377]] Prompting Large Language Models for Supporting the Differential Diagnosis of Anemia(https://arxiv.org/abs/2409.15377)
• Keywords: transformer, generative, large language model
• Abstract: In practice, clinicians achieve a diagnosis by following a sequence of steps, such as laboratory exams, observations, or imaging. The pathways to reach diagnosis decisions are documented by guidelines authored by expert organizations, which guide clinicians to reach a correct diagnosis through these sequences of steps. While these guidelines are beneficial for following medical reasoning and consolidating medical knowledge, they have some drawbacks. They often fail to address patients with uncommon conditions due to their focus on the majority population, and are slow and costly to update, making them unsuitable for rapidly emerging diseases or new practices. Inspired by clinical guidelines, our study aimed to develop pathways similar to those that can be obtained in clinical guidelines. We tested three Large Language Models (LLMs) -Generative Pretrained Transformer 4 (GPT-4), Large Language Model Meta AI (LLaMA), and Mistral -on a synthetic yet realistic dataset to differentially diagnose anemia and its subtypes. By using advanced prompting techniques to enhance the decision-making process, we generated diagnostic pathways using these models. Experimental results indicate that LLMs hold huge potential in clinical pathway discovery from patient data, with GPT-4 exhibiting the best performance in all conducted experiments.

Title: Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino

• Authors: Jann Railey Montalan, Jian Gang Ngui, Wei Qi Leong, Yosephine Susanto, Hamsawardhini Rengarajan, William Chandra Tjhi, Alham Fikri Aji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15380
• Pdf URL: https://arxiv.org/pdf/2409.15380
• Copy Paste: [[2409.15380]] Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino(https://arxiv.org/abs/2409.15380)
• Keywords: large language model
• Abstract: Multilingual large language models (LLMs) today may not necessarily provide culturally appropriate and relevant responses to its Filipino users. We introduce Kalahi, a cultural LLM evaluation suite collaboratively created by native Filipino speakers. It is composed of 150 high-quality, handcrafted and nuanced prompts that test LLMs for generations that are relevant to shared Filipino cultural knowledge and values. Strong LLM performance in Kalahi indicates a model's ability to generate responses similar to what an average Filipino would say or do in a given situation. We conducted experiments on LLMs with multilingual and Filipino language support. Results show that Kalahi, while trivial for Filipinos, is challenging for LLMs, with the best model answering only 46.0% of the questions correctly compared to native Filipino performance of 89.10%. Thus, Kalahi can be used to accurately and reliably evaluate Filipino cultural representation in LLMs.

Title: Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generation

• Authors: G M Shahariar, Jia Chen, Jiachen Li, Yue Dong
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15381
• Pdf URL: https://arxiv.org/pdf/2409.15381
• Copy Paste: [[2409.15381]] Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generation(https://arxiv.org/abs/2409.15381)
• Keywords: attack
• Abstract: Recent studies show that text-to-image (T2I) models are vulnerable to adversarial attacks, especially with noun perturbations in text prompts. In this study, we investigate the impact of adversarial attacks on different POS tags within text prompts on the images generated by T2I models. We create a high-quality dataset for realistic POS tag token swapping and perform gradient-based attacks to find adversarial suffixes that mislead T2I models into generating images with altered tokens. Our empirical results show that the attack success rate (ASR) varies significantly among different POS tag categories, with nouns, proper nouns, and adjectives being the easiest to attack. We explore the mechanism behind the steering effect of adversarial suffixes, finding that the number of critical tokens and content fusion vary among POS tags, while features like suffix transferability are consistent across categories. We have made our implementation publicly available at - this https URL.

Title: Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient

• Authors: Shaoqi Wang, Chunjie Yang, Siwei Lou
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.15393
• Pdf URL: https://arxiv.org/pdf/2409.15393
• Copy Paste: [[2409.15393]] Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient(https://arxiv.org/abs/2409.15393)
• Keywords: robust, extraction, interpretability
• Abstract: Neural networks (NN) are extensively studied in cutting-edge soft sensor models due to their feature extraction and function approximation capabilities. Current research into network-based methods primarily focuses on models' offline accuracy. Notably, in industrial soft sensor context, online optimizing stability and interpretability are prioritized, followed by accuracy. This requires a clearer understanding of network's training process. To bridge this gap, we propose a novel NN named the Approximated Orthogonal Projection Unit (AOPU) which has solid mathematical basis and presents superior training stability. AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. We further prove that AOPU attains minimum variance estimation (MVE) in NN, wherein the truncated gradient approximates the natural gradient (NG). Empirical results on two chemical process datasets clearly show that AOPU outperforms other models in achieving stable convergence, marking a significant advancement in soft sensor field.

Title: Parse Trees Guided LLM Prompt Compression

• Authors: Wenhao Mao, Chengbin Hou, Tianyu Zhang, Xinyu Lin, Ke Tang, Hairong Lv
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15395
• Pdf URL: https://arxiv.org/pdf/2409.15395
• Copy Paste: [[2409.15395]] Parse Trees Guided LLM Prompt Compression(https://arxiv.org/abs/2409.15395)
• Keywords: generative, large language model
• Abstract: Offering rich contexts to Large Language Models (LLMs) has shown to boost the performance in various tasks, but the resulting longer prompt would increase the computational cost and might exceed the input limit of LLMs. Recently, some prompt compression methods have been suggested to shorten the length of prompts by using language models to generate shorter prompts or by developing computational models to select important parts of original prompt. The generative compression methods would suffer from issues like hallucination, while the selective compression methods have not involved linguistic rules and overlook the global structure of prompt. To this end, we propose a novel selective compression method called PartPrompt. It first obtains a parse tree for each sentence based on linguistic rules, and calculates local information entropy for each node in a parse tree. These local parse trees are then organized into a global tree according to the hierarchical structure such as the dependency of sentences, paragraphs, and sections. After that, the root-ward propagation and leaf-ward propagation are proposed to adjust node values over the global tree. Finally, a recursive algorithm is developed to prune the global tree based on the adjusted node values. The experiments show that PartPrompt receives the state-of-the-art performance across various datasets, metrics, compression ratios, and target LLMs for inference. The in-depth ablation studies confirm the effectiveness of designs in PartPrompt, and other additional experiments also demonstrate its superiority in terms of the coherence of compressed prompts and in the extreme long prompt scenario.

Title: Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI

• Authors: Ambrish Rawat, Stefan Schoepf, Giulio Zizzo, Giandomenico Cornacchia, Muhammad Zaid Hameed, Kieran Fraser, Erik Miehling, Beat Buesser, Elizabeth M. Daly, Mark Purcell, Prasanna Sattigeri, Pin-Yu Chen, Kush R. Varshney
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15398
• Pdf URL: https://arxiv.org/pdf/2409.15398
• Copy Paste: [[2409.15398]] Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI(https://arxiv.org/abs/2409.15398)
• Keywords: security, protect, defense, attack, generative, large language model
• Abstract: As generative AI, particularly large language models (LLMs), become increasingly integrated into production applications, new attack surfaces and vulnerabilities emerge and put a focus on adversarial threats in natural language and multi-modal systems. Red-teaming has gained importance in proactively identifying weaknesses in these systems, while blue-teaming works to protect against such adversarial attacks. Despite growing academic interest in adversarial risks for generative AI, there is limited guidance tailored for practitioners to assess and mitigate these challenges in real-world environments. To address this, our contributions include: (1) a practical examination of red- and blue-teaming strategies for securing generative AI, (2) identification of key challenges and open questions in defense development and evaluation, and (3) the Attack Atlas, an intuitive framework that brings a practical approach to analyzing single-turn input attacks, placing it at the forefront for practitioners. This work aims to bridge the gap between academic insights and practical security measures for the protection of generative AI systems.

Title: Revealing an Unattractivity Bias in Mental Reconstruction of Occluded Faces using Generative Image Models

• Authors: Frederik Riedmann, Bernhard Egger, Tim Rohe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15443
• Pdf URL: https://arxiv.org/pdf/2409.15443
• Copy Paste: [[2409.15443]] Revealing an Unattractivity Bias in Mental Reconstruction of Occluded Faces using Generative Image Models(https://arxiv.org/abs/2409.15443)
• Keywords: diffusion, generative
• Abstract: Previous studies have shown that faces are rated as more attractive when they are partially occluded. The cause of this observation remains unclear. One explanation is a mental reconstruction of the occluded face parts which is biased towards a more attractive percept as shown in face-attractiveness rating tasks. We aimed to test for this hypothesis by using a delayed matching-to-sample task, which directly requires mental reconstruction. In two online experiments, we presented observers with unattractive, neutral or attractive synthetic reconstructions of the occluded face parts using a state-of-the-art diffusion-based image generator. Our experiments do not support the initial hypothesis and reveal an unattractiveness bias for occluded faces instead. This suggests that facial attractiveness rating tasks do not prompt reconstructions. Rather, the attractivity bias may arise from global image features, and faces may actually be reconstructed with unattractive properties when mental reconstruction is applied.

Title: CUTE: Measuring LLMs' Understanding of Their Tokens

• Authors: Lukas Edman, Helmut Schmid, Alexander Fraser
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15452
• Pdf URL: https://arxiv.org/pdf/2409.15452
• Copy Paste: [[2409.15452]] CUTE: Measuring LLMs' Understanding of Their Tokens(https://arxiv.org/abs/2409.15452)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direct access to individual characters. This raises the question: To what extent can LLMs learn orthographic information? To answer this, we propose a new benchmark, CUTE, which features a collection of tasks designed to test the orthographic knowledge of LLMs. We evaluate popular LLMs on CUTE, finding that most of them seem to know the spelling of their tokens, yet fail to use this information effectively to manipulate text, calling into question how much of this knowledge is generalizable.

Title: In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models

• Authors: Pengrui Han, Peiyang Song, Haofei Yu, Jiaxuan You
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15454
• Pdf URL: https://arxiv.org/pdf/2409.15454
• Copy Paste: [[2409.15454]] In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models(https://arxiv.org/abs/2409.15454)
• Keywords: large language model
• Abstract: Recent advancements in artificial intelligence have led to the creation of highly capable large language models (LLMs) that can perform tasks in a human-like manner. However, LLMs exhibit only infant-level cognitive abilities in certain areas. One such area is the A-Not-B error, a phenomenon seen in infants where they repeat a previously rewarded behavior despite well-observed changed conditions. This highlights their lack of inhibitory control -- the ability to stop a habitual or impulsive response. In our work, we design a text-based multi-choice QA scenario similar to the A-Not-B experimental settings to systematically test the inhibitory control abilities of LLMs. We found that state-of-the-art LLMs (like Llama3-8b) perform consistently well with in-context learning (ICL) but make errors and show a significant drop of as many as 83.3% in reasoning tasks when the context changes trivially. This suggests that LLMs only have inhibitory control abilities on par with human infants in this regard, often failing to suppress the previously established response pattern during ICL.

Title: Preventing Rowhammer Exploits via Low-Cost Domain-Aware Memory Allocation

• Authors: Anish Saxena, Walter Wang, Alexandros Daglis
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.15463
• Pdf URL: https://arxiv.org/pdf/2409.15463
• Copy Paste: [[2409.15463]] Preventing Rowhammer Exploits via Low-Cost Domain-Aware Memory Allocation(https://arxiv.org/abs/2409.15463)
• Keywords: security, protect, defense, attack
• Abstract: Rowhammer is a hardware security vulnerability at the heart of every system with modern DRAM-based memory. Despite its discovery a decade ago, comprehensive defenses remain elusive, while the probability of successful attacks grows with DRAM density. Hardware-based defenses have been ineffective, due to considerable cost, delays in commercial adoption, and attackers' repeated ability to circumvent them. Meanwhile, more flexible software-based solutions either incur substantial performance and memory capacity overheads, or offer limited forms of protection. Citadel is a new memory allocator design that prevents Rowhammer-initiated security exploits by addressing the vulnerability's root cause: physical adjacency of DRAM rows. Citadel enables creation of flexible security domains and isolates different domains in physically disjoint memory regions, guaranteeing security by design. On a server system, Citadel supports thousands of security domains at a modest 7.4% average memory overhead and no performance loss. In contrast, recent domain isolation schemes fail to support many workload scenarios due to excessive overheads, and incur 4--6x higher overheads for supported scenarios. As a software solution, Citadel offers readily deployable Rowhammer-aware isolation on legacy, current, and future systems.

Title: MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models

• Authors: Mohammad Shahab Sepehri, Zalan Fabian, Maryam Soltanolkotabi, Mahdi Soltanolkotabi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15477
• Pdf URL: https://arxiv.org/pdf/2409.15477
• Copy Paste: [[2409.15477]] MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models(https://arxiv.org/abs/2409.15477)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have tremendous potential to improve the accuracy, availability, and cost-effectiveness of healthcare by providing automated solutions or serving as aids to medical professionals. Despite promising first steps in developing medical MLLMs in the past few years, their capabilities and limitations are not well-understood. Recently, many benchmark datasets have been proposed that test the general medical knowledge of such models across a variety of medical areas. However, the systematic failure modes and vulnerabilities of such models are severely underexplored with most medical benchmarks failing to expose the shortcomings of existing models in this safety-critical domain. In this paper, we introduce MediConfusion, a challenging medical Visual Question Answering (VQA) benchmark dataset, that probes the failure modes of medical MLLMs from a vision perspective. We reveal that state-of-the-art models are easily confused by image pairs that are otherwise visually dissimilar and clearly distinct for medical experts. Strikingly, all available models (open-source or proprietary) achieve performance below random guessing on MediConfusion, raising serious concerns about the reliability of existing medical MLLMs for healthcare deployment. We also extract common patterns of model failure that may help the design of a new generation of more trustworthy and reliable MLLMs in healthcare.

Title: VLMine: Long-Tail Data Mining with Vision Language Models

• Authors: Mao Ye, Gregory P. Meyer, Zaiwei Zhang, Dennis Park, Siva Karthik Mustikovela, Yuning Chai, Eric M Wolff
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15486
• Pdf URL: https://arxiv.org/pdf/2409.15486
• Copy Paste: [[2409.15486]] VLMine: Long-Tail Data Mining with Vision Language Models(https://arxiv.org/abs/2409.15486)
• Keywords: robust
• Abstract: Ensuring robust performance on long-tail examples is an important problem for many real-world applications of machine learning, such as autonomous driving. This work focuses on the problem of identifying rare examples within a corpus of unlabeled data. We propose a simple and scalable data mining approach that leverages the knowledge contained within a large vision language model (VLM). Our approach utilizes a VLM to summarize the content of an image into a set of keywords, and we identify rare examples based on keyword frequency. We find that the VLM offers a distinct signal for identifying long-tail examples when compared to conventional methods based on model uncertainty. Therefore, we propose a simple and general approach for integrating signals from multiple mining algorithms. We evaluate the proposed method on two diverse tasks: 2D image classification, in which inter-class variation is the primary source of data diversity, and on 3D object detection, where intra-class variation is the main concern. Furthermore, through the detection task, we demonstrate that the knowledge extracted from 2D images is transferable to the 3D domain. Our experiments consistently show large improvements (between 10\% and 50\%) over the baseline techniques on several representative benchmarks: ImageNet-LT, Places-LT, and the Waymo Open Dataset.

Title: Analysis of Human Perception in Distinguishing Real and AI-Generated Faces: An Eye-Tracking Based Study

• Authors: Jin Huang, Subhadra Gopalakrishnan, Trisha Mittal, Jake Zuena, Jaclyn Pytlarz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15498
• Pdf URL: https://arxiv.org/pdf/2409.15498
• Copy Paste: [[2409.15498]] Analysis of Human Perception in Distinguishing Real and AI-Generated Faces: An Eye-Tracking Based Study(https://arxiv.org/abs/2409.15498)
• Keywords: generative
• Abstract: Recent advancements in Artificial Intelligence have led to remarkable improvements in generating realistic human faces. While these advancements demonstrate significant progress in generative models, they also raise concerns about the potential misuse of these generated images. In this study, we investigate how humans perceive and distinguish between real and fake images. We designed a perceptual experiment using eye-tracking technology to analyze how individuals differentiate real faces from those generated by AI. Our analysis of StyleGAN-3 generated images reveals that participants can distinguish real from fake faces with an average accuracy of 76.80%. Additionally, we found that participants scrutinize images more closely when they suspect an image to be fake. We believe this study offers valuable insights into human perception of AI-generated media.

Title: PixelBytes: Catching Unified Embedding for Multimodal Generation

• Authors: Fabien Furfaro
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15512
• Pdf URL: https://arxiv.org/pdf/2409.15512
• Copy Paste: [[2409.15512]] PixelBytes: Catching Unified Embedding for Multimodal Generation(https://arxiv.org/abs/2409.15512)
• Keywords: transformer
• Abstract: This report introduces PixelBytes Embedding, a novel approach for unified multimodal representation learning. Our method captures diverse inputs in a single, cohesive representation, enabling emergent properties for multimodal sequence generation, particularly for text and pixelated images. Inspired by state-of-the-art sequence models such as Image Transformers, PixelCNN, and Mamba-Bytes, PixelBytes aims to address the challenges of integrating different data types. We explore various model architectures, including Recurrent Neural Networks (RNNs), State Space Models (SSMs), and Attention-based models, focusing on bidirectional processing and our innovative PxBy embedding technique. Our experiments, conducted on a specialized PixelBytes Pok{é}mon dataset, demonstrate that bidirectional sequence models with PxBy embedding and convolutional layers can generate coherent multimodal sequences. This work contributes to the advancement of integrated AI models capable of understanding and generating multimodal data in a unified manner.

Title: Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA

• Authors: Nirmal Roy, Leonardo F. R. Ribeiro, Rexhina Blloshmi, Kevin Small
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15515
• Pdf URL: https://arxiv.org/pdf/2409.15515
• Copy Paste: [[2409.15515]] Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA(https://arxiv.org/abs/2409.15515)
• Keywords: large language model
• Abstract: Augmenting Large Language Models (LLMs) with information retrieval capabilities (i.e., Retrieval-Augmented Generation (RAG)) has proven beneficial for knowledge-intensive tasks. However, understanding users' contextual search intent when generating responses is an understudied topic for conversational question answering (QA). This conversational extension leads to additional concerns when compared to single-turn QA as it is more challenging for systems to comprehend conversational context and manage retrieved passages over multiple turns. In this work, we propose a method for enabling LLMs to decide when to retrieve in RAG settings given a conversational context. When retrieval is deemed necessary, the LLM then rewrites the conversation for passage retrieval and judges the relevance of returned passages before response generation. Operationally, we build on the single-turn SELF-RAG framework (Asai et al., 2023) and propose SELF-multi-RAG for conversational settings. SELF-multi-RAG demonstrates improved capabilities over single-turn variants with respect to retrieving relevant passages (by using summarized conversational context) and assessing the quality of generated responses. Experiments on three conversational QA datasets validate the enhanced response generation capabilities of SELF-multi-RAG, with improvements of ~13% measured by human annotation.

Title: Eagle: Efficient Training-Free Router for Multi-LLM Inference

• Authors: Zesen Zhao, Shuowei Jin, Z. Morley Mao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15518
• Pdf URL: https://arxiv.org/pdf/2409.15518
• Copy Paste: [[2409.15518]] Eagle: Efficient Training-Free Router for Multi-LLM Inference(https://arxiv.org/abs/2409.15518)
• Keywords: large language model
• Abstract: The proliferation of Large Language Models (LLMs) with varying capabilities and costs has created a need for efficient model selection in AI systems. LLM routers address this need by dynamically choosing the most suitable model for a given query based on task requirements and budget constraints. However, existing routers face challenges in scalability and real-time adaptation, particularly in high-volume online environments. We present Eagle, a novel LLM routing approach that combines global and local ELO ranking modules to overcome these limitations. By evaluating both general and specialized LLM abilities, Eagle provides a scalable, training-free solution that enhances model selection quality while reducing computational overhead. Our experiments across multiple datasets show Eagle consistently outperforms baseline methods, with improvements of up to 23.52 percent in Area Under Curve (AUC) scores. Moreover, Eagle demonstrates remarkable efficiency, requiring only 1/20 of baseline methods' time for initialization and 100 to 200 times faster incremental updates in online scenarios, making it well-suited for dynamic, high-volume online serving environments.

Title: Enabling Resource-Efficient On-Device Fine-Tuning of LLMs Using Only Inference Engines

• Authors: Lei Gao, Amir Ziashahabi, Yue Niu, Salman Avestimehr, Murali Annavaram
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15520
• Pdf URL: https://arxiv.org/pdf/2409.15520
• Copy Paste: [[2409.15520]] Enabling Resource-Efficient On-Device Fine-Tuning of LLMs Using Only Inference Engines(https://arxiv.org/abs/2409.15520)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional performance in automating various tasks, such as text generation and summarization. Currently LLMs are trained and fine-tuned on large cloud server. Deploying and fine-tuning these models on resource-constrained edge devices remains a significant challenge due to their substantial memory and computational requirements. This paper introduces a resource-efficient zeroth-order optimization approach that lowers the barriers for fine-tuning LLMs in such constrained environments. Our method features a parallelized randomized gradient estimation (P-RGE) technique, which performs gradient estimation with high parallel efficiency. P-RGE leverages outer-loop and inner-loop parallelization to perform multiple function queries and forward passes in parallel, reducing the wall-clock end-to-end training time. By integrating this technique with parameter-efficient fine-tuning methods (e.g., LoRA) and on-device inference engines (e.g., ExecuTorch), we demonstrate efficient fine-tuning of LLMs on both server-side and edge devices. Experiments show that P-RGE achieves significant runtime speedups and memory savings while maintaining fine-tuning accuracy, which paves the way for more practical deployment of LLMs in real-time, on-device applications.

Title: Ditto: Elastic Confidential VMs with Secure and Dynamic CPU Scaling

• Authors: Shixuan Zhao, Mengyuan Li, Mengjia Yan, Zhiqiang Lin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.15542
• Pdf URL: https://arxiv.org/pdf/2409.15542
• Copy Paste: [[2409.15542]] Ditto: Elastic Confidential VMs with Secure and Dynamic CPU Scaling(https://arxiv.org/abs/2409.15542)
• Keywords: secure, security
• Abstract: Confidential Virtual Machines (CVMs) are a type of VMbased Trusted Execution Environments (TEEs) designed to enhance the security of cloud-based VMs, safeguarding them even from malicious hypervisors. Although CVMs have been widely adopted by major cloud service providers, current CVM designs face significant challenges in runtime resource management due to their fixed capacities and lack of transparency. These limitations hamper efficient cloud resource management, leading to increased operational costs and reduced agility in responding to fluctuating workloads. This paper introduces a dynamic CPU resource management approach, featuring the novel concept of "Elastic CVM. This approach allows for hypervisor-assisted runtime adjustment of CPU resources using a specialized vCPU type, termed Worker vCPU. This new approach enhances CPU resource adaptability and operational efficiency without compromising security. Additionally, we introduce a Worker vCPU Abstraction Layer to simplify Worker vCPU deployment and management. To demonstrate the effectiveness of our approach, we have designed and implemented a serverless computing prototype platform, called Ditto. We show that Ditto significantly improves performance and efficiency through finergrain resource management. The concept of "Elastic CVM" and the Worker vCPU design not only optimize cloud resource utilization but also pave the way for more flexible and cost-effective confidential computing environments.

Title: SOFI: Multi-Scale Deformable Transformer for Camera Calibration with Enhanced Line Queries

• Authors: Sebastian Janampa, Marios Pattichis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15553
• Pdf URL: https://arxiv.org/pdf/2409.15553
• Copy Paste: [[2409.15553]] SOFI: Multi-Scale Deformable Transformer for Camera Calibration with Enhanced Line Queries(https://arxiv.org/abs/2409.15553)
• Keywords: transformer
• Abstract: Camera calibration consists of estimating camera parameters such as the zenith vanishing point and horizon line. Estimating the camera parameters allows other tasks like 3D rendering, artificial reality effects, and object insertion in an image. Transformer-based models have provided promising results; however, they lack cross-scale interaction. In this work, we introduce \textit{multi-Scale defOrmable transFormer for camera calibratIon with enhanced line queries}, SOFI. SOFI improves the line queries used in CTRL-C and MSCC by using both line content and line geometric features. Moreover, SOFI's line queries allow transformer models to adopt the multi-scale deformable attention mechanism to promote cross-scale interaction between the feature maps produced by the backbone. SOFI outperforms existing methods on the \textit {Google Street View}, \textit {Horizon Line in the Wild}, and \textit {Holicity} datasets while keeping a competitive inference speed.

Title: Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection

• Authors: Alireza Ganjdanesh, Yan Kang, Yuchen Liu, Richard Zhang, Zhe Lin, Heng Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15557
• Pdf URL: https://arxiv.org/pdf/2409.15557
• Copy Paste: [[2409.15557]] Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection(https://arxiv.org/abs/2409.15557)
• Keywords: diffusion
• Abstract: Diffusion probabilistic models can generate high-quality samples. Yet, their sampling process requires numerous denoising steps, making it slow and computationally intensive. We propose to reduce the sampling cost by pruning a pretrained diffusion model into a mixture of efficient experts. First, we study the similarities between pairs of denoising timesteps, observing a natural clustering, even across different datasets. This suggests that rather than having a single model for all time steps, separate models can serve as ``experts'' for their respective time intervals. As such, we separately fine-tune the pretrained model on each interval, with elastic dimensions in depth and width, to obtain experts specialized in their corresponding denoising interval. To optimize the resource usage between experts, we introduce our Expert Routing Agent, which learns to select a set of proper network configurations. By doing so, our method can allocate the computing budget between the experts in an end-to-end manner without requiring manual heuristics. Finally, with a selected configuration, we fine-tune our pruned experts to obtain our mixture of efficient experts. We demonstrate the effectiveness of our method, DiffPruning, across several datasets, LSUN-Church, LSUN-Beds, FFHQ, and ImageNet, on the Latent Diffusion Model architecture.

Title: Stalactite: Toolbox for Fast Prototyping of Vertical Federated Learning Systems

• Authors: Anastasiia Zakharova, Dmitriy Alexandrov, Maria Khodorchenko, Nikolay Butakov, Alexey Vasilev, Maxim Savchenko, Alexander Grigorievskiy
• Subjects: cs.LG, cs.DC, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.15558
• Pdf URL: https://arxiv.org/pdf/2409.15558
• Copy Paste: [[2409.15558]] Stalactite: Toolbox for Fast Prototyping of Vertical Federated Learning Systems(https://arxiv.org/abs/2409.15558)
• Keywords: federate
• Abstract: Machine learning (ML) models trained on datasets owned by different organizations and physically located in remote databases offer benefits in many real-world use cases. State regulations or business requirements often prevent data transfer to a central location, making it difficult to utilize standard machine learning algorithms. Federated Learning (FL) is a technique that enables models to learn from distributed datasets without revealing the original data. Vertical Federated learning (VFL) is a type of FL where data samples are divided by features across several data owners. For instance, in a recommendation task, a user can interact with various sets of items, and the logs of these interactions are stored by different organizations. In this demo paper, we present \emph{Stalactite} - an open-source framework for VFL that provides the necessary functionality for building prototypes of VFL systems. It has several advantages over the existing frameworks. In particular, it allows researchers to focus on the algorithmic side rather than engineering and to easily deploy learning in a distributed environment. It implements several VFL algorithms and has a built-in homomorphic encryption layer. We demonstrate its use on a real-world recommendation datasets.

Title: Analyzing Privacy Implications of Data Collection in Android Automotive OS

• Authors: Bulut Gözübüyük, Brian Tang, Kang G. Shin, Mert D. Pesé
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.15561
• Pdf URL: https://arxiv.org/pdf/2409.15561
• Copy Paste: [[2409.15561]] Analyzing Privacy Implications of Data Collection in Android Automotive OS(https://arxiv.org/abs/2409.15561)
• Keywords: privacy, protect
• Abstract: Modern vehicles have become sophisticated computation and sensor systems, as evidenced by advanced driver assistance systems, in-car infotainment, and autonomous driving capabilities. They collect and process vast amounts of data through various embedded subsystems. One significant player in this landscape is Android Automotive OS (AAOS), which has been integrated into over 100M vehicles and has become a dominant force in the in-vehicle infotainment market. With this extensive data collection, privacy has become increasingly crucial. The volume of data gathered by these systems raises questions about how this information is stored, used, and protected, making privacy a critical issue for manufacturers and consumers. However, very little has been done on vehicle data privacy. This paper focuses on the privacy implications of AAOS, examining the exact nature and scope of data collection and the corresponding privacy policies from the original equipment manufacturers (OEMs). We develop a novel automotive privacy analysis tool called PriDrive which employs three methodological approaches: network traffic inspection, and both static and dynamic analyses of Android images using rooted emulators from various OEMs. These methodologies are followed by an assessment of whether the collected data types were properly disclosed in OEMs and 3rd party apps' privacy policies (to identify any discrepancies or violations). Our evaluation on three different OEM platforms reveals that vehicle speed is collected at a sampling rate of roughly 25 Hz. Other properties such as model info, climate & AC, and seat data are collected in a batch 30 seconds into vehicle startup. In addition, several vehicle property types were collected without disclosure in their respective privacy policies. For example, OEM A's policies only covers 110 vehicle properties or 13.02% of the properties found in our static analysis.

Title: CauSkelNet: Causal Representation Learning for Human Behaviour Analysis

• Authors: Xingrui Gu, Chuyi Jiang, Erte Wang, Zekun Wu, Qiang Cui, Leimin Tian, Lianlong Wu, Siyang Song, Chuang Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15564
• Pdf URL: https://arxiv.org/pdf/2409.15564
• Copy Paste: [[2409.15564]] CauSkelNet: Causal Representation Learning for Human Behaviour Analysis(https://arxiv.org/abs/2409.15564)
• Keywords: protect, robust, interpretability
• Abstract: Constrained by the lack of model interpretability and a deep understanding of human movement in traditional movement recognition machine learning methods, this study introduces a novel representation learning method based on causal inference to better understand human joint dynamics and complex behaviors. We propose a two-stage framework that combines the Peter-Clark (PC) algorithm and Kullback-Leibler (KL) divergence to identify and quantify causal relationships between joints. Our method effectively captures interactions and produces interpretable, robust representations. Experiments on the EmoPain dataset show that our causal GCN outperforms traditional GCNs in accuracy, F1 score, and recall, especially in detecting protective behaviors. The model is also highly invariant to data scale changes, enhancing its reliability in practical applications. Our approach advances human motion analysis and paves the way for more adaptive intelligent healthcare solutions.

Title: GEM-RAG: Graphical Eigen Memories For Retrieval Augmented Generation

• Authors: Brendan Hogan Rappazzo, Yingheng Wang, Aaron Ferber, Carla Gomes
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15566
• Pdf URL: https://arxiv.org/pdf/2409.15566
• Copy Paste: [[2409.15566]] GEM-RAG: Graphical Eigen Memories For Retrieval Augmented Generation(https://arxiv.org/abs/2409.15566)
• Keywords: robust, large language model
• Abstract: The ability to form, retrieve, and reason about memories in response to stimuli serves as the cornerstone for general intelligence - shaping entities capable of learning, adaptation, and intuitive insight. Large Language Models (LLMs) have proven their ability, given the proper memories or context, to reason and respond meaningfully to stimuli. However, they are still unable to optimally encode, store, and retrieve memories - the ability to do this would unlock their full ability to operate as AI agents, and to specialize to niche domains. To remedy this, one promising area of research is Retrieval Augmented Generation (RAG), which aims to augment LLMs by providing them with rich in-context examples and information. In question-answering (QA) applications, RAG methods embed the text of interest in chunks, and retrieve the most relevant chunks for a prompt using text embeddings. Motivated by human memory encoding and retrieval, we aim to improve over standard RAG methods by generating and encoding higher-level information and tagging the chunks by their utility to answer questions. We introduce Graphical Eigen Memories For Retrieval Augmented Generation (GEM-RAG). GEM-RAG works by tagging each chunk of text in a given text corpus with LLM generated ``utility'' questions, connecting chunks in a graph based on the similarity of both their text and utility questions, and then using the eigendecomposition of the memory graph to build higher level summary nodes that capture the main themes of the text. We evaluate GEM-RAG, using both UnifiedQA and GPT-3.5 Turbo as the LLMs, with SBERT, and OpenAI's text encoders on two standard QA tasks, showing that GEM-RAG outperforms other state-of-the-art RAG methods on these tasks. We also discuss the implications of having a robust RAG system and future directions.

Title: Clinical-grade Multi-Organ Pathology Report Generation for Multi-scale Whole Slide Images via a Semantically Guided Medical Text Foundation Model

• Authors: Jing Wei Tan, SeungKyu Kim, Eunsu Kim, Sung Hak Lee, Sangjeong Ahn, Won-Ki Jeong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15574
• Pdf URL: https://arxiv.org/pdf/2409.15574
• Copy Paste: [[2409.15574]] Clinical-grade Multi-Organ Pathology Report Generation for Multi-scale Whole Slide Images via a Semantically Guided Medical Text Foundation Model(https://arxiv.org/abs/2409.15574)
• Keywords: transformer
• Abstract: Vision language models (VLM) have achieved success in both natural language comprehension and image recognition tasks. However, their use in pathology report generation for whole slide images (WSIs) is still limited due to the huge size of multi-scale WSIs and the high cost of WSI annotation. Moreover, in most of the existing research on pathology report generation, sufficient validation regarding clinical efficacy has not been conducted. Herein, we propose a novel Patient-level Multi-organ Pathology Report Generation (PMPRG) model, which utilizes the multi-scale WSI features from our proposed multi-scale regional vision transformer (MR-ViT) model and their real pathology reports to guide VLM training for accurate pathology report generation. The model then automatically generates a report based on the provided key features attended regional features. We assessed our model using a WSI dataset consisting of multiple organs, including the colon and kidney. Our model achieved a METEOR score of 0.68, demonstrating the effectiveness of our approach. This model allows pathologists to efficiently generate pathology reports for patients, regardless of the number of WSIs involved.

Title: Polyatomic Complexes: A topologically-informed learning representation for atomistic systems

• Authors: Rahul Khorana, Marcus Noack, Jin Qian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15600
• Pdf URL: https://arxiv.org/pdf/2409.15600
• Copy Paste: [[2409.15600]] Polyatomic Complexes: A topologically-informed learning representation for atomistic systems(https://arxiv.org/abs/2409.15600)
• Keywords: robust
• Abstract: Developing robust physics-informed representations of chemical structures that enable models to learn topological inductive biases is challenging. In this manuscript, we present a representation of atomistic systems. We begin by proving that our representation satisfies all structural, geometric, efficiency, and generalizability constraints. Afterward, we provide a general algorithm to encode any atomistic system. Finally, we report performance comparable to state-of-the-art methods on numerous tasks. We open-source all code and datasets. The code and data are available at this https URL.

Title: Revolutionizing Biomarker Discovery: Leveraging Generative AI for Bio-Knowledge-Embedded Continuous Space Exploration

• Authors: Wangyang Ying, Dongjie Wang, Xuanming Hu, Ji Qiu, Jin Park, Yanjie Fu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15612
• Pdf URL: https://arxiv.org/pdf/2409.15612
• Copy Paste: [[2409.15612]] Revolutionizing Biomarker Discovery: Leveraging Generative AI for Bio-Knowledge-Embedded Continuous Space Exploration(https://arxiv.org/abs/2409.15612)
• Keywords: robust, generative
• Abstract: Biomarker discovery is vital in advancing personalized medicine, offering insights into disease diagnosis, prognosis, and therapeutic efficacy. Traditionally, the identification and validation of biomarkers heavily depend on extensive experiments and statistical analyses. These approaches are time-consuming, demand extensive domain expertise, and are constrained by the complexity of biological systems. These limitations motivate us to ask: Can we automatically identify the effective biomarker subset without substantial human efforts? Inspired by the success of generative AI, we think that the intricate knowledge of biomarker identification can be compressed into a continuous embedding space, thus enhancing the search for better biomarkers. Thus, we propose a new biomarker identification framework with two important modules:1) training data preparation and 2) embedding-optimization-generation. The first module uses a multi-agent system to automatically collect pairs of biomarker subsets and their corresponding prediction accuracy as training data. These data establish a strong knowledge base for biomarker identification. The second module employs an encoder-evaluator-decoder learning paradigm to compress the knowledge of the collected data into a continuous space. Then, it utilizes gradient-based search techniques and autoregressive-based reconstruction to efficiently identify the optimal subset of biomarkers. Finally, we conduct extensive experiments on three real-world datasets to show the efficiency, robustness, and effectiveness of our method.

Title: KISS-Matcher: Fast and Robust Point Cloud Registration Revisited

• Authors: Hyungtae Lim, Daebeom Kim, Gunhee Shin, Jingnan Shi, Ignacio Vizzo, Hyun Myung, Jaesik Park, and Luca Carlone
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.15615
• Pdf URL: https://arxiv.org/pdf/2409.15615
• Copy Paste: [[2409.15615]] KISS-Matcher: Fast and Robust Point Cloud Registration Revisited(https://arxiv.org/abs/2409.15615)
• Keywords: robust, extraction
• Abstract: While global point cloud registration systems have advanced significantly in all aspects, many studies have focused on specific components, such as feature extraction, graph-theoretic pruning, or pose solvers. In this paper, we take a holistic view on the registration problem and develop an open-source and versatile C++ library for point cloud registration, called \textit{KISS-Matcher}. KISS-Matcher combines a novel feature detector, \textit{Faster-PFH}, that improves over the classical fast point feature histogram (FPFH). Moreover, it adopts a $k$-core-based graph-theoretic pruning to reduce the time complexity of rejecting outlier correspondences. Finally, it combines these modules in a complete, user-friendly, and ready-to-use pipeline. As verified by extensive experiments, KISS-Matcher has superior scalability and broad applicability, achieving a substantial speed-up compared to state-of-the-art outlier-robust registration pipelines while preserving accuracy. Our code will be available at \href{this https URL}{\texttt{this https URL}}.

Title: Data Augmentation for Sparse Multidimensional Learning Performance Data Using Generative AI

• Authors: Liang Zhang, Jionghao Lin, John Sabatini, Conrad Borchers, Daniel Weitekamp, Meng Cao, John Hollander, Xiangen Hu, Arthur C. Graesser
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15631
• Pdf URL: https://arxiv.org/pdf/2409.15631
• Copy Paste: [[2409.15631]] Data Augmentation for Sparse Multidimensional Learning Performance Data Using Generative AI(https://arxiv.org/abs/2409.15631)
• Keywords: transformer, generative
• Abstract: Learning performance data describe correct and incorrect answers or problem-solving attempts in adaptive learning, such as in intelligent tutoring systems (ITSs). Learning performance data tend to be highly sparse (80\%\(\sim\)90\% missing observations) in most real-world applications due to adaptive item selection. This data sparsity presents challenges to using learner models to effectively predict future performance explore new hypotheses about learning. This article proposes a systematic framework for augmenting learner data to address data sparsity in learning performance data. First, learning performance is represented as a three-dimensional tensor of learners' questions, answers, and attempts, capturing longitudinal knowledge states during learning. Second, a tensor factorization method is used to impute missing values in sparse tensors of collected learner data, thereby grounding the imputation on knowledge tracing tasks that predict missing performance values based on real observations. Third, a module for generating patterns of learning is used. This study contrasts two forms of generative Artificial Intelligence (AI), including Generative Adversarial Networks (GANs) and Generate Pre-Trained Transformers (GPT) to generate data associated with different clusters of learner data. We tested this approach on an adult literacy dataset from AutoTutor lessons developed for Adult Reading Comprehension (ARC). We found that: (1) tensor factorization improved the performance in tracing and predicting knowledge mastery compared with other knowledge tracing techniques without data augmentation, showing higher relative fidelity for this imputation method, and (2) the GAN-based simulation showed greater overall stability and less statistical bias based on a divergence evaluation with varying simulation sample sizes compared to GPT.

Title: Personalized Federated Learning via Backbone Self-Distillation

• Authors: Pengju Wang, Bochao Liu, Dan Zeng, Chenggang Yan, Shiming Ge
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15636
• Pdf URL: https://arxiv.org/pdf/2409.15636
• Copy Paste: [[2409.15636]] Personalized Federated Learning via Backbone Self-Distillation(https://arxiv.org/abs/2409.15636)
• Keywords: federate
• Abstract: In practical scenarios, federated learning frequently necessitates training personalized models for each client using heterogeneous data. This paper proposes a backbone self-distillation approach to facilitate personalized federated learning. In this approach, each client trains its local model and only sends the backbone weights to the server. These weights are then aggregated to create a global backbone, which is returned to each client for updating. However, the client's local backbone lacks personalization because of the common representation. To solve this problem, each client further performs backbone self-distillation by using the global backbone as a teacher and transferring knowledge to update the local backbone. This process involves learning two components: the shared backbone for common representation and the private head for local personalization, which enables effective global knowledge transfer. Extensive experiments and comparisons with 12 state-of-the-art approaches demonstrate the effectiveness of our approach.

Title: Looped Transformers for Length Generalization

• Authors: Ying Fan, Yilun Du, Kannan Ramchandran, Kangwook Lee
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15647
• Pdf URL: https://arxiv.org/pdf/2409.15647
• Copy Paste: [[2409.15647]] Looped Transformers for Length Generalization(https://arxiv.org/abs/2409.15647)
• Keywords: transformer
• Abstract: Recent work has shown that Transformers trained from scratch can successfully solve various arithmetic and algorithmic tasks, such as adding numbers and computing parity. While these Transformers generalize well on unseen inputs of the same length, they struggle with length generalization, i.e., handling inputs of unseen lengths. In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization. We focus on tasks with a known iterative solution, involving multiple iterations of a RASP-L operation - a length-generalizable operation that can be expressed by a finite-sized Transformer. We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks.

Title: ImPoster: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Models

• Authors: Divya Kothandaraman, Kuldeep Kulkarni, Sumit Shekhar, Balaji Vasan Srinivasan, Dinesh Manocha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15650
• Pdf URL: https://arxiv.org/pdf/2409.15650
• Copy Paste: [[2409.15650]] ImPoster: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Models(https://arxiv.org/abs/2409.15650)
• Keywords: diffusion
• Abstract: We present ImPoster, a novel algorithm for generating a target image of a 'source' subject performing a 'driving' action. The inputs to our algorithm are a single pair of a source image with the subject that we wish to edit and a driving image with a subject of an arbitrary class performing the driving action, along with the text descriptions of the two images. Our approach is completely unsupervised and does not require any access to additional annotations like keypoints or pose. Our approach builds on a pretrained text-to-image latent diffusion model and learns the characteristics of the source and the driving image by finetuning the diffusion model for a small number of iterations. At inference time, ImPoster performs step-wise text prompting i.e. it denoises by first moving in the direction of the image manifold corresponding to the driving image followed by the direction of the image manifold corresponding to the text description of the desired target image. We propose a novel diffusion guidance formulation, image frequency guidance, to steer the generation towards the manifold of the source subject and the driving action at every step of the inference denoising. Our frequency guidance formulations are derived from the frequency domain properties of images. We extensively evaluate ImPoster on a diverse set of source-driving image pairs to demonstrate improvements over baselines. To the best of our knowledge, ImPoster is the first approach towards achieving both subject-driven as well as action-driven image personalization. Code and data is available at this https URL.

Title: Identified-and-Targeted: The First Early Evidence of the Privacy-Invasive Use of Browser Fingerprinting for Online Tracking

• Authors: Zengrui Liu, Jimmy Dani, Shujiang Wu, Yinzhi Cao, Nitesh Saxena
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.15656
• Pdf URL: https://arxiv.org/pdf/2409.15656
• Copy Paste: [[2409.15656]] Identified-and-Targeted: The First Early Evidence of the Privacy-Invasive Use of Browser Fingerprinting for Online Tracking(https://arxiv.org/abs/2409.15656)
• Keywords: security, privacy, robust
• Abstract: While advertising has become commonplace in today's online interactions, there is a notable dearth of research investigating the extent to which browser fingerprinting is harnessed for user tracking and targeted advertising. Prior studies only measured whether fingerprinting-related scripts are being run on the websites but that in itself does not necessarily mean that fingerprinting is being used for the privacy-invasive purpose of online tracking because fingerprinting might be deployed for the defensive purposes of bot/fraud detection and user authentication. It is imperative to address the mounting concerns regarding the utilization of browser fingerprinting in the realm of online advertising. To understand the privacy-invasive use of fingerprinting for user tracking, this paper introduces a new framework ``FPTrace'' (fingerprinting-based tracking assessment and comprehensive evaluation framework) designed to identify alterations in advertisements resulting from adjustments in browser fingerprinting settings. Our approach involves emulating genuine user interactions, capturing advertiser bid data, and closely monitoring HTTP information. Using FPTrace we conduct a large-scale measurement study to identify whether browser fingerprinting is being used for the purpose of user tracking and ad targeting. The results we have obtained provide robust evidence supporting the utilization of browser fingerprinting for the purposes of advertisement tracking and targeting. This is substantiated by significant disparities in bid values and a reduction in HTTP records subsequent to changes in fingerprinting. In conclusion, our research unveils the widespread employment of browser fingerprinting in online advertising, prompting critical considerations regarding user privacy and data security within the digital advertising landscape.

Title: Double-Path Adaptive-correlation Spatial-Temporal Inverted Transformer for Stock Time Series Forecasting

• Authors: Wenbo Yan, Ying Tan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15662
• Pdf URL: https://arxiv.org/pdf/2409.15662
• Copy Paste: [[2409.15662]] Double-Path Adaptive-correlation Spatial-Temporal Inverted Transformer for Stock Time Series Forecasting(https://arxiv.org/abs/2409.15662)
• Keywords: transformer
• Abstract: Spatial-temporal graph neural networks (STGNNs) have achieved significant success in various time series forecasting tasks. However, due to the lack of explicit and fixed spatial relationships in stock prediction tasks, many STGNNs fail to perform effectively in this domain. While some STGNNs learn spatial relationships from time series, they often lack comprehensiveness. Research indicates that modeling time series using feature changes as tokens reveals entirely different information compared to using time steps as tokens. To more comprehensively extract dynamic spatial information from stock data, we propose a Double-Path Adaptive-correlation Spatial-Temporal Inverted Transformer (DPA-STIFormer). DPA-STIFormer models each node via continuous changes in features as tokens and introduces a Double Direction Self-adaptation Fusion mechanism. This mechanism decomposes node encoding into temporal and feature representations, simultaneously extracting different spatial correlations from a double path approach, and proposes a Double-path gating mechanism to fuse these two types of correlation information. Experiments conducted on four stock market datasets demonstrate state-of-the-art results, validating the model's superior capability in uncovering latent temporal-correlation patterns.

Title: Data Poisoning-based Backdoor Attack Framework against Supervised Learning Rules of Spiking Neural Networks

• Authors: Lingxin Jin, Meiyu Lin, Wei Jiang, Jinyu Zhan
• Subjects: cs.CR, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.15670
• Pdf URL: https://arxiv.org/pdf/2409.15670
• Copy Paste: [[2409.15670]] Data Poisoning-based Backdoor Attack Framework against Supervised Learning Rules of Spiking Neural Networks(https://arxiv.org/abs/2409.15670)
• Keywords: security, attack, robust, steal
• Abstract: Spiking Neural Networks (SNNs), the third generation neural networks, are known for their low energy consumption and high robustness. SNNs are developing rapidly and can compete with Artificial Neural Networks (ANNs) in many fields. To ensure that the widespread use of SNNs does not cause serious security incidents, much research has been conducted to explore the robustness of SNNs under adversarial sample attacks. However, many other unassessed security threats exist, such as highly stealthy backdoor attacks. Therefore, to fill the research gap in this and further explore the security vulnerabilities of SNNs, this paper explores the robustness performance of SNNs trained by supervised learning rules under backdoor attacks. Specifically, the work herein includes: i) We propose a generic backdoor attack framework that can be launched against the training process of existing supervised learning rules and covers all learnable dataset types of SNNs. ii) We analyze the robustness differences between different learning rules and between SNN and ANN, which suggests that SNN no longer has inherent robustness under backdoor attacks. iii) We reveal the vulnerability of conversion-dependent learning rules caused by backdoor migration and further analyze the migration ability during the conversion process, finding that the backdoor migration rate can even exceed 99%. iv) Finally, we discuss potential countermeasures against this kind of backdoor attack and its technical challenges and point out several promising research directions.

Title: A Survey of Stance Detection on Social Media: New Directions and Perspectives

• Authors: Bowen Zhang, Genan Dai, Fuqiang Niu, Nan Yin, Xiaomao Fan, Hu Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15690
• Pdf URL: https://arxiv.org/pdf/2409.15690
• Copy Paste: [[2409.15690]] A Survey of Stance Detection on Social Media: New Directions and Perspectives(https://arxiv.org/abs/2409.15690)
• Keywords: robust, large language model
• Abstract: In modern digital environments, users frequently express opinions on contentious topics, providing a wealth of information on prevailing attitudes. The systematic analysis of these opinions offers valuable insights for decision-making in various sectors, including marketing and politics. As a result, stance detection has emerged as a crucial subfield within affective computing, enabling the automatic detection of user stances in social media conversations and providing a nuanced understanding of public sentiment on complex issues. Recent years have seen a surge of research interest in developing effective stance detection methods, with contributions from multiple communities, including natural language processing, web science, and social computing. This paper provides a comprehensive survey of stance detection techniques on social media, covering task definitions, datasets, approaches, and future works. We review traditional stance detection models, as well as state-of-the-art methods based on large language models, and discuss their strengths and limitations. Our survey highlights the importance of stance detection in understanding public opinion and sentiment, and identifies gaps in current research. We conclude by outlining potential future directions for stance detection on social media, including the need for more robust and generalizable models, and the importance of addressing emerging challenges such as multi-modal stance detection and stance detection in low-resource languages.

Title: GraphGI:A GNN Explanation Method using Game Interaction

• Authors: Xingping Xian, Jianlu Liu, Tao Wu, Lin Yuan, Chao Wang, Baiyun Chen
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.15698
• Pdf URL: https://arxiv.org/pdf/2409.15698
• Copy Paste: [[2409.15698]] GraphGI:A GNN Explanation Method using Game Interaction(https://arxiv.org/abs/2409.15698)
• Keywords: interpretability
• Abstract: Graph Neural Networks (GNNs) have garnered significant attention and have been extensively utilized across various domains. However, similar to other deep learning models, GNNs are often viewed as black-box models, making it challenging to interpret their prediction mechanisms. Current graph explanation techniques focus on identifying key nodes or edges, attributing the critical data features that drive model predictions. Nevertheless, these features do not independently influence the model's outcomes; rather, they interact with one another to collectively affect predictions. In this work, we propose a novel explanatory method GraphGI, which identifies the coalition with the highest interaction strength and presents it as an explanatory subgraph. Given a trained model and an input graph, our method explains predictions by gradually incorporating significant edges into the selected subgraph. We utilize game-theoretic interaction values to assess the interaction strength after edge additions, ensuring that the newly added edges confer maximum interaction strength to the explanatory subgraph. To enhance computational efficiency, we adopt effective approximation techniques for calculating Shapley values and game-theoretic interaction values. Empirical evaluations demonstrate that our method achieves superior fidelity and sparsity, maintaining the interpretability of the results at a comprehensible level.

Title: Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generation

• Authors: Zheng Liu, Chenyuan Wu, Ninglu Shao, Shitao Xiao, Chaozhuo Li, Defu Lian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15699
• Pdf URL: https://arxiv.org/pdf/2409.15699
• Copy Paste: [[2409.15699]] Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generation(https://arxiv.org/abs/2409.15699)
• Keywords: large language model
• Abstract: The existing Retrieval-Augmented Generation (RAG) systems face significant challenges in terms of cost and effectiveness. On one hand, they need to encode the lengthy retrieved contexts before responding to the input tasks, which imposes substantial computational overhead. On the other hand, directly using generic Large Language Models (LLMs) often leads to sub-optimal answers, while task-specific fine-tuning may compromise the LLMs' general capabilities. To address these challenges, we introduce a novel approach called FlexRAG (Flexible Context Adaptation for RAG). In this approach, the retrieved contexts are compressed into compact embeddings before being encoded by the LLMs. Simultaneously, these compressed embeddings are optimized to enhance downstream RAG performance. A key feature of FlexRAG is its flexibility, which enables effective support for diverse compression ratios and selective preservation of important contexts. Thanks to these technical designs, FlexRAG achieves superior generation quality while significantly reducing running costs. Comprehensive experiments on various question-answering datasets validate our approach as a cost-effective and flexible solution for RAG systems.

Title: Adversarial Federated Consensus Learning for Surface Defect Classification Under Data Heterogeneity in IIoT

• Authors: Jixuan Cui, Jun Li, Zhen Mei, Yiyang Ni, Wen Chen, Zengxiang Li
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.15711
• Pdf URL: https://arxiv.org/pdf/2409.15711
• Copy Paste: [[2409.15711]] Adversarial Federated Consensus Learning for Surface Defect Classification Under Data Heterogeneity in IIoT(https://arxiv.org/abs/2409.15711)
• Keywords: privacy, federate
• Abstract: The challenge of data scarcity hinders the application of deep learning in industrial surface defect classification (SDC), as it's difficult to collect and centralize sufficient training data from various entities in Industrial Internet of Things (IIoT) due to privacy concerns. Federated learning (FL) provides a solution by enabling collaborative global model training across clients while maintaining privacy. However, performance may suffer due to data heterogeneity--discrepancies in data distributions among clients. In this paper, we propose a novel personalized FL (PFL) approach, named Adversarial Federated Consensus Learning (AFedCL), for the challenge of data heterogeneity across different clients in SDC. First, we develop a dynamic consensus construction strategy to mitigate the performance degradation caused by data heterogeneity. Through adversarial training, local models from different clients utilize the global model as a bridge to achieve distribution alignment, alleviating the problem of global knowledge forgetting. Complementing this strategy, we propose a consensus-aware aggregation mechanism. It assigns aggregation weights to different clients based on their efficacy in global knowledge learning, thereby enhancing the global model's generalization capabilities. Finally, we design an adaptive feature fusion module to further enhance global knowledge utilization efficiency. Personalized fusion weights are gradually adjusted for each client to optimally balance global and local features, tailored to their individual global knowledge learning efficacy. Compared with state-of-the-art FL methods like FedALA, the proposed AFedCL method achieves an accuracy increase of up to 5.67% on three SDC datasets.

Title: Disentangled Generation and Aggregation for Robust Radiance Fields

• Authors: Shihe Shen, Huachen Gao, Wangze Xu, Rui Peng, Luyang Tang, Kaiqiang Xiong, Jianbo Jiao, Ronggang Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.15715
• Pdf URL: https://arxiv.org/pdf/2409.15715
• Copy Paste: [[2409.15715]] Disentangled Generation and Aggregation for Robust Radiance Fields(https://arxiv.org/abs/2409.15715)
• Keywords: robust
• Abstract: The utilization of the triplane-based radiance fields has gained attention in recent years due to its ability to effectively disentangle 3D scenes with a high-quality representation and low computation cost. A key requirement of this method is the precise input of camera poses. However, due to the local update property of the triplane, a similar joint estimation as previous joint pose-NeRF optimization works easily results in local minima. To this end, we propose the Disentangled Triplane Generation module to introduce global feature context and smoothness into triplane learning, which mitigates errors caused by local updating. Then, we propose the Disentangled Plane Aggregation to mitigate the entanglement caused by the common triplane feature aggregation during camera pose updating. In addition, we introduce a two-stage warm-start training strategy to reduce the implicit constraints caused by the triplane generator. Quantitative and qualitative results demonstrate that our proposed method achieves state-of-the-art performance in novel view synthesis with noisy or unknown camera poses, as well as efficient convergence of optimization. Project page: this https URL.

Title: Federated Large Language Models: Current Progress and Future Directions

• Authors: Yuhang Yao, Jianyi Zhang, Junda Wu, Chengkai Huang, Yu Xia, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Ang Li, Lina Yao, Julian McAuley, Yiran Chen, Carlee Joe-Wong
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15723
• Pdf URL: https://arxiv.org/pdf/2409.15723
• Copy Paste: [[2409.15723]] Federated Large Language Models: Current Progress and Future Directions(https://arxiv.org/abs/2409.15723)
• Keywords: privacy, federate, large language model
• Abstract: Large language models are rapidly gaining popularity and have been widely adopted in real-world applications. While the quality of training data is essential, privacy concerns arise during data collection. Federated learning offers a solution by allowing multiple clients to collaboratively train LLMs without sharing local data. However, FL introduces new challenges, such as model convergence issues due to heterogeneous data and high communication costs. A comprehensive study is required to address these challenges and guide future research. This paper surveys Federated learning for LLMs (FedLLM), highlighting recent advances and future directions. We focus on two key aspects: fine-tuning and prompt learning in a federated setting, discussing existing work and associated research challenges. We finally propose potential research directions for federated LLMs, including pre-training and how LLMs can further enhance federated learning.

Title: LaPose: Laplacian Mixture Shape Modeling for RGB-Based Category-Level Object Pose Estimation

• Authors: Ruida Zhang, Ziqin Huang, Gu Wang, Chenyangguang Zhang, Yan Di, Xingxing Zuo, Jiwen Tang, Xiangyang Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15727
• Pdf URL: https://arxiv.org/pdf/2409.15727
• Copy Paste: [[2409.15727]] LaPose: Laplacian Mixture Shape Modeling for RGB-Based Category-Level Object Pose Estimation(https://arxiv.org/abs/2409.15727)
• Keywords: robust
• Abstract: While RGBD-based methods for category-level object pose estimation hold promise, their reliance on depth data limits their applicability in diverse scenarios. In response, recent efforts have turned to RGB-based methods; however, they face significant challenges stemming from the absence of depth information. On one hand, the lack of depth exacerbates the difficulty in handling intra-class shape variation, resulting in increased uncertainty in shape predictions. On the other hand, RGB-only inputs introduce inherent scale ambiguity, rendering the estimation of object size and translation an ill-posed problem. To tackle these challenges, we propose LaPose, a novel framework that models the object shape as the Laplacian mixture model for Pose estimation. By representing each point as a probabilistic distribution, we explicitly quantify the shape uncertainty. LaPose leverages both a generalized 3D information stream and a specialized feature stream to independently predict the Laplacian distribution for each point, capturing different aspects of object geometry. These two distributions are then integrated as a Laplacian mixture model to establish the 2D-3D correspondences, which are utilized to solve the pose via the PnP module. In order to mitigate scale ambiguity, we introduce a scale-agnostic representation for object size and translation, enhancing training efficiency and overall robustness. Extensive experiments on the NOCS datasets validate the effectiveness of LaPose, yielding state-of-the-art performance in RGB-based category-level object pose estimation. Codes are released at this https URL

Title: EvoFA: Evolvable Fast Adaptation for EEG Emotion Recognition

• Authors: Ming Jin, Danni Zhang, Gangming Zhao, Changde Du, Jinpeng Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15733
• Pdf URL: https://arxiv.org/pdf/2409.15733
• Copy Paste: [[2409.15733]] EvoFA: Evolvable Fast Adaptation for EEG Emotion Recognition(https://arxiv.org/abs/2409.15733)
• Keywords: robust
• Abstract: Electroencephalography (EEG)-based emotion recognition has gained significant traction due to its accuracy and objectivity. However, the non-stationary nature of EEG signals leads to distribution drift over time, causing severe performance degradation when the model is reused. While numerous domain adaptation (DA) approaches have been proposed in recent years to address this issue, their reliance on large amounts of target data for calibration restricts them to offline scenarios, rendering them unsuitable for real-time applications. To address this challenge, this paper proposes Evolvable Fast Adaptation (EvoFA), an online adaptive framework tailored for EEG data. EvoFA organically integrates the rapid adaptation of Few-Shot Learning (FSL) and the distribution matching of Domain Adaptation (DA) through a two-stage generalization process. During the training phase, a robust base meta-learning model is constructed for strong generalization. In the testing phase, a designed evolvable meta-adaptation module iteratively aligns the marginal distribution of target (testing) data with the evolving source (training) data within a model-agnostic meta-learning framework, enabling the model to learn the evolving trends of testing data relative to training data and improving online testing performance. Experimental results demonstrate that EvoFA achieves significant improvements compared to the basic FSL method and previous online methods. The introduction of EvoFA paves the way for broader adoption of EEG-based emotion recognition in real-world applications. Our code will be released upon publication.

Title: LSAST - Enhancing Cybersecurity through LLM-supported Static Application Security Testing

• Authors: Mete Keltek, Ziyue Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.15735
• Pdf URL: https://arxiv.org/pdf/2409.15735
• Copy Paste: [[2409.15735]] LSAST - Enhancing Cybersecurity through LLM-supported Static Application Security Testing(https://arxiv.org/abs/2409.15735)
• Keywords: security, privacy, large language model
• Abstract: In the fast-evolving landscape of cybersecurity, Large Language Models (LLMs) play a pivotal role, continually improving their ability to analyze software code. This paper introduces a novel approach to vulnerability scanning by integrating conservative SAST (Static Application Security Testing) scanners with LLM capabilities, resulting in the creation of LSAST (LLM-supported Static Application Security Testing). Our approach significantly enhances the performance of LLMs in vulnerability scanning, establishing a new standard in this field. We benchmark LSAST's efficiency and compare its results with a state-of-the-art LLM. Additionally, we address the inherent drawbacks of LLMs in vulnerability scanning: their reliance on static training datasets, which leads to the exclusion of the latest vulnerabilities, and the privacy concerns associated with sending code to third-party LLM providers. To mitigate these issues, we utilize an open-source LLM to ensure privacy and employ a novel approach to gather relevant vulnerability information, thereby equipping the LLM with up-to-date knowledge.

Title: Teaching Tailored to Talent: Adverse Weather Restoration via Prompt Pool and Depth-Anything Constraint

• Authors: Sixiang Chen, Tian Ye, Kai Zhang, Zhaohu Xing, Yunlong Lin, Lei Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15739
• Pdf URL: https://arxiv.org/pdf/2409.15739
• Copy Paste: [[2409.15739]] Teaching Tailored to Talent: Adverse Weather Restoration via Prompt Pool and Depth-Anything Constraint(https://arxiv.org/abs/2409.15739)
• Keywords: diffusion
• Abstract: Recent advancements in adverse weather restoration have shown potential, yet the unpredictable and varied combinations of weather degradations in the real world pose significant challenges. Previous methods typically struggle with dynamically handling intricate degradation combinations and carrying on background reconstruction precisely, leading to performance and generalization limitations. Drawing inspiration from prompt learning and the "Teaching Tailored to Talent" concept, we introduce a novel pipeline, T3-DiffWeather. Specifically, we employ a prompt pool that allows the network to autonomously combine sub-prompts to construct weather-prompts, harnessing the necessary attributes to adaptively tackle unforeseen weather input. Moreover, from a scene modeling perspective, we incorporate general prompts constrained by Depth-Anything feature to provide the scene-specific condition for the diffusion process. Furthermore, by incorporating contrastive prompt loss, we ensures distinctive representations for both types of prompts by a mutual pushing strategy. Experimental results demonstrate that our method achieves state-of-the-art performance across various synthetic and real-world datasets, markedly outperforming existing diffusion techniques in terms of computational efficiency.

Title: ManiNeg: Manifestation-guided Multimodal Pretraining for Mammography Classification

• Authors: Xujun Li, Xin Wei, Jing Jiang, Danxiang Chen, Wei Zhang, Jinpeng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15745
• Pdf URL: https://arxiv.org/pdf/2409.15745
• Copy Paste: [[2409.15745]] ManiNeg: Manifestation-guided Multimodal Pretraining for Mammography Classification(https://arxiv.org/abs/2409.15745)
• Keywords: robust
• Abstract: Breast cancer is a significant threat to human health. Contrastive learning has emerged as an effective method to extract critical lesion features from mammograms, thereby offering a potent tool for breast cancer screening and analysis. A crucial aspect of contrastive learning involves negative sampling, where the selection of appropriate hard negative samples is essential for driving representations to retain detailed information about lesions. In contrastive learning, it is often assumed that features can sufficiently capture semantic content, and that each minibatch inherently includes ideal hard negative samples. However, the characteristics of breast lumps challenge these assumptions. In response, we introduce ManiNeg, a novel approach that leverages manifestations as proxies to mine hard negative samples. Manifestations, which refer to the observable symptoms or signs of a disease, provide a knowledge-driven and robust basis for choosing hard negative samples. This approach benefits from its invariance to model optimization, facilitating efficient sampling. To support ManiNeg and future research endeavors, we developed the MVKL dataset, which includes multi-view mammograms, corresponding reports, meticulously annotated manifestations, and pathologically confirmed benign-malignant outcomes. We evaluate ManiNeg on the benign and malignant classification task. Our results demonstrate that ManiNeg not only improves representation in both unimodal and multimodal contexts but also shows generalization across datasets. The MVKL dataset and our codes are publicly available at this https URL.

Title: Training Neural Networks for Modularity aids Interpretability

• Authors: Satvik Golechha, Dylan Cope, Nandi Schoots
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15747
• Pdf URL: https://arxiv.org/pdf/2409.15747
• Copy Paste: [[2409.15747]] Training Neural Networks for Modularity aids Interpretability(https://arxiv.org/abs/2409.15747)
• Keywords: interpretability
• Abstract: An approach to improve network interpretability is via clusterability, i.e., splitting a model into disjoint clusters that can be studied independently. We find pretrained models to be highly unclusterable and thus train models to be more modular using an ``enmeshment loss'' function that encourages the formation of non-interacting clusters. Using automated interpretability measures, we show that our method finds clusters that learn different, disjoint, and smaller circuits for CIFAR-10 labels. Our approach provides a promising direction for making neural networks easier to interpret.

Title: The Roles of Generative Artificial Intelligence in Internet of Electric Vehicles

• Authors: Hanwen Zhang, Dusit Niyato, Wei Zhang, Changyuan Zhao, Hongyang Du, Abbas Jamalipour, Sumei Sun, Yiyang Pei
• Subjects: cs.LG, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2409.15750
• Pdf URL: https://arxiv.org/pdf/2409.15750
• Copy Paste: [[2409.15750]] The Roles of Generative Artificial Intelligence in Internet of Electric Vehicles(https://arxiv.org/abs/2409.15750)
• Keywords: security, robust, generative
• Abstract: With the advancement of generative artificial intelligence (GenAI) models, their capability to generate content is seeing significant enhancement, leading to widespread applications in the field of data generation and forecasting. Furthermore, GenAI has strong capabilities in data modeling and analysis, which enhances Internet of electric vehicles (IoEV) applications in various aspects. In this paper, we investigate and survey applications of GenAI in the IoEV. Specifically, we categorize GenAI for IoEV into four different layers namely, EV's battery layer, individual electric vehicle (EV) layer, smart grid with EV layer, and security layer. We first introduce various GenAI techniques used in each layer of IoEV applications. Subsequently, public datasets available for training the GenAI models are summarized. Finally, we provide recommendations for future directions. This survey not only categorizes the applications of GenAI in IoEV across different layers but also serves as a valuable resource for researchers and practitioners by highlighting the design and implementation challenges within each layer. Furthermore, it provides a roadmap for future research directions, enabling the development of more robust and efficient IoEV systems through the integration of advanced GenAI techniques.

Title: Smart Grid Security: A Verified Deep Reinforcement Learning Framework to Counter Cyber-Physical Attacks

• Authors: Suman Maiti, Soumyajit Dey
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.15757
• Pdf URL: https://arxiv.org/pdf/2409.15757
• Copy Paste: [[2409.15757]] Smart Grid Security: A Verified Deep Reinforcement Learning Framework to Counter Cyber-Physical Attacks(https://arxiv.org/abs/2409.15757)
• Keywords: security, protect, attack
• Abstract: The distributed nature of smart grids, combined with sophisticated sensors, control algorithms, and data collection facilities at Supervisory Control and Data Acquisition (SCADA) centers, makes them vulnerable to strategically crafted cyber-physical attacks. These malicious attacks can manipulate power demands using high-wattage Internet of Things (IoT) botnet devices, such as refrigerators and air conditioners, or introduce false values into transmission line power flow sensor readings. Consequently, grids experience blackouts and high power flow oscillations. Existing grid protection mechanisms, originally designed to tackle natural faults in transmission lines and generator outages, are ineffective against such intelligently crafted attacks. This is because grid operators overlook potential scenarios of cyber-physical attacks during their design phase. In this work, we propose a safe Deep Reinforcement Learning (DRL)-based framework for mitigating attacks on smart grids. The DRL agent effectively neutralizes cyber-physical attacks on grid surfaces by triggering appropriate sequences of existing protection schemes. The safety of the DRL agent is formally verified through a reachability analysis method. Additionally, our framework is designed for deployment on CUDA-enabled GPU systems, which enables faster execution of these protection sequences and their real-time validation. Our framework establishes a new set of protection rules for grid models, successfully thwarting existing cyber-physical attacks.

Title: TFG: Unified Training-Free Guidance for Diffusion Models

• Authors: Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma, James Zou, Stefano Ermon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15761
• Pdf URL: https://arxiv.org/pdf/2409.15761
• Copy Paste: [[2409.15761]] TFG: Unified Training-Free Guidance for Diffusion Models(https://arxiv.org/abs/2409.15761)
• Keywords: diffusion
• Abstract: Given an unconditional diffusion model and a predictor for a target property of interest (e.g., a classifier), the goal of training-free guidance is to generate samples with desirable target properties without additional training. Existing methods, though effective in various individual applications, often lack theoretical grounding and rigorous testing on extensive benchmarks. As a result, they could even fail on simple tasks, and applying them to a new problem becomes unavoidably difficult. This paper introduces a novel algorithmic framework encompassing existing methods as special cases, unifying the study of training-free guidance into the analysis of an algorithm-agnostic design space. Via theoretical and empirical investigation, we propose an efficient and effective hyper-parameter searching strategy that can be readily applied to any downstream task. We systematically benchmark across 7 diffusion models on 16 tasks with 40 targets, and improve performance by 8.5% on average. Our framework and benchmark offer a solid foundation for conditional generation in a training-free manner.

Title: XTRUST: On the Multilingual Trustworthiness of Large Language Models

• Authors: Yahan Li, Yi Wang, Yi Chang, Yuan Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15762
• Pdf URL: https://arxiv.org/pdf/2409.15762
• Copy Paste: [[2409.15762]] XTRUST: On the Multilingual Trustworthiness of Large Language Models(https://arxiv.org/abs/2409.15762)
• Keywords: privacy, robust, fair, large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a range of natural language processing (NLP) tasks, capturing the attention of both practitioners and the broader public. A key question that now preoccupies the AI community concerns the capabilities and limitations of these models, with trustworthiness emerging as a central issue, particularly as LLMs are increasingly applied in sensitive fields like healthcare and finance, where errors can have serious consequences. However, most previous studies on the trustworthiness of LLMs have been limited to a single language, typically the predominant one in the dataset, such as English. In response to the growing global deployment of LLMs, we introduce XTRUST, the first comprehensive multilingual trustworthiness benchmark. XTRUST encompasses a diverse range of topics, including illegal activities, hallucination, out-of-distribution (OOD) robustness, physical and mental health, toxicity, fairness, misinformation, privacy, and machine ethics, across 10 different languages. Using XTRUST, we conduct an empirical evaluation of the multilingual trustworthiness of five widely used LLMs, offering an in-depth analysis of their performance across languages and tasks. Our results indicate that many LLMs struggle with certain low-resource languages, such as Arabic and Russian, highlighting the considerable room for improvement in the multilingual trustworthiness of current language models. The code is available at this https URL.

Title: CHBench: A Chinese Dataset for Evaluating Health in Large Language Models

• Authors: Chenlu Guo, Nuo Xu, Yi Chang, Yuan Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15766
• Pdf URL: https://arxiv.org/pdf/2409.15766
• Copy Paste: [[2409.15766]] CHBench: A Chinese Dataset for Evaluating Health in Large Language Models(https://arxiv.org/abs/2409.15766)
• Keywords: large language model
• Abstract: With the rapid development of large language models (LLMs), assessing their performance on health-related inquiries has become increasingly essential. It is critical that these models provide accurate and trustworthy health information, as their application in real-world contexts--where misinformation can have serious consequences for individuals seeking medical advice and support--depends on their reliability. In this work, we present CHBench, the first comprehensive Chinese Health-related Benchmark designed to evaluate LLMs' capabilities in understanding physical and mental health across diverse scenarios. CHBench includes 6,493 entries related to mental health and 2,999 entries focused on physical health, covering a broad spectrum of topics. This dataset serves as a foundation for evaluating Chinese LLMs' capacity to comprehend and generate accurate health-related information. Our extensive evaluations of four popular Chinese LLMs demonstrate that there remains considerable room for improvement in their understanding of health-related information. The code is available at this https URL.

Title: Zero-shot forecasting of chaotic systems

• Authors: Yuanzhao Zhang, William Gilpin
• Subjects: cs.LG, nlin.CD, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2409.15771
• Pdf URL: https://arxiv.org/pdf/2409.15771
• Copy Paste: [[2409.15771]] Zero-shot forecasting of chaotic systems(https://arxiv.org/abs/2409.15771)
• Keywords: large language model
• Abstract: Time-series forecasting is a challenging task that traditionally requires specialized models custom-trained for the specific task at hand. Recently, inspired by the success of large language models, foundation models pre-trained on vast amounts of time-series data from diverse domains have emerged as a promising candidate for general-purpose time-series forecasting. The defining characteristic of these foundation models is their ability to perform zero-shot learning, that is, forecasting a new system from limited context data without explicit re-training or fine-tuning. Here, we evaluate whether the zero-shot learning paradigm extends to the challenging task of forecasting chaotic systems. Across 135 distinct chaotic dynamical systems and $10^8$ timepoints, we find that foundation models produce competitive forecasts compared to custom-trained models (including NBEATS, TiDE, etc.), particularly when training data is limited. Interestingly, even after point forecasts fail, foundation models preserve the geometric and statistical properties of the chaotic attractors, demonstrating a surprisingly strong ability to capture the long-term behavior of chaotic dynamical systems. Our results highlight the promises and pitfalls of foundation models in making zero-shot forecasts of chaotic systems.

Title: Training Data Attribution: Was Your Model Secretly Trained On Data Created By Mine?

• Authors: Likun Zhang, Hao Wu, Lingcui Zhang, Fengyuan Xu, Jin Cao, Fenghua Li, Ben Niu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15781
• Pdf URL: https://arxiv.org/pdf/2409.15781
• Copy Paste: [[2409.15781]] Training Data Attribution: Was Your Model Secretly Trained On Data Created By Mine?(https://arxiv.org/abs/2409.15781)
• Keywords: security, watermark
• Abstract: The emergence of text-to-image models has recently sparked significant interest, but the attendant is a looming shadow of potential infringement by violating the user terms. Specifically, an adversary may exploit data created by a commercial model to train their own without proper authorization. To address such risk, it is crucial to investigate the attribution of a suspicious model's training data by determining whether its training data originates, wholly or partially, from a specific source model. To trace the generated data, existing methods require applying extra watermarks during either the training or inference phases of the source model. However, these methods are impractical for pre-trained models that have been released, especially when model owners lack security expertise. To tackle this challenge, we propose an injection-free training data attribution method for text-to-image models. It can identify whether a suspicious model's training data stems from a source model, without additional modifications on the source model. The crux of our method lies in the inherent memorization characteristic of text-to-image models. Our core insight is that the memorization of the training dataset is passed down through the data generated by the source model to the model trained on that data, making the source model and the infringing model exhibit consistent behaviors on specific samples. Therefore, our approach involves developing algorithms to uncover these distinct samples and using them as inherent watermarks to verify if a suspicious model originates from the source model. Our experiments demonstrate that our method achieves an accuracy of over 80\% in identifying the source of a suspicious model's training data, without interfering the original training or generation process of the source model.

Title: Small Language Models: Survey, Measurements, and Insights

• Authors: Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, Mengwei Xu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15790
• Pdf URL: https://arxiv.org/pdf/2409.15790
• Copy Paste: [[2409.15790]] Small Language Models: Survey, Measurements, and Insights(https://arxiv.org/abs/2409.15790)
• Keywords: transformer, large language model
• Abstract: Small language models (SLMs), despite their widespread adoption in modern smart devices, have received significantly less academic attention compared to their large language model (LLM) counterparts, which are predominantly deployed in data centers and cloud environments. While researchers continue to improve the capabilities of LLMs in the pursuit of artificial general intelligence, SLM research aims to make machine intelligence more accessible, affordable, and efficient for everyday tasks. Focusing on transformer-based, decoder-only language models with 100M-5B parameters, we survey 59 state-of-the-art open-source SLMs, analyzing their technical innovations across three axes: architectures, training datasets, and training algorithms. In addition, we evaluate their capabilities in various domains, including commonsense reasoning, in-context learning, mathematics, and coding. To gain further insight into their on-device runtime costs, we benchmark their inference latency and memory footprints. Through in-depth analysis of our benchmarking data, we offer valuable insights to advance research in this field.

Title: Towards Universal Large-Scale Foundational Model for Natural Gas Demand Forecasting

• Authors: Xinxing Zhou, Jiaqi Ye, Shubao Zhao, Ming Jin, Zhaoxiang Hou, Chengyi Yang, Zengxiang Li, Yanlong Wen, Xiaojie Yuan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15794
• Pdf URL: https://arxiv.org/pdf/2409.15794
• Copy Paste: [[2409.15794]] Towards Universal Large-Scale Foundational Model for Natural Gas Demand Forecasting(https://arxiv.org/abs/2409.15794)
• Keywords: robust
• Abstract: In the context of global energy strategy, accurate natural gas demand forecasting is crucial for ensuring efficient resource allocation and operational planning. Traditional forecasting methods struggle to cope with the growing complexity and variability of gas consumption patterns across diverse industries and commercial sectors. To address these challenges, we propose the first foundation model specifically tailored for natural gas demand forecasting. Foundation models, known for their ability to generalize across tasks and datasets, offer a robust solution to the limitations of traditional methods, such as the need for separate models for different customer segments and their limited generalization capabilities. Our approach leverages contrastive learning to improve prediction accuracy in real-world scenarios, particularly by tackling issues such as noise in historical consumption data and the potential misclassification of similar data samples, which can lead to degradation in the quaility of the representation and thus the accuracy of downstream forecasting tasks. By integrating advanced noise filtering techniques within the contrastive learning framework, our model enhances the quality of learned representations, leading to more accurate predictions. Furthermore, the model undergoes industry-specific fine-tuning during pretraining, enabling it to better capture the unique characteristics of gas consumption across various sectors. We conducted extensive experiments using a large-scale dataset from ENN Group, which includes data from over 10,000 industrial, commercial, and welfare-related customers across multiple regions. Our model outperformed existing state-of-the-art methods, demonstrating a relative improvement in MSE by 3.68\% and in MASE by 6.15\% compared to the best available model.

Title: DIAL: Dense Image-text ALignment for Weakly Supervised Semantic Segmentation

• Authors: Soojin Jang, Jungmin Yun, Junehyoung Kwon, Eunju Lee, Youngbin Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15801
• Pdf URL: https://arxiv.org/pdf/2409.15801
• Copy Paste: [[2409.15801]] DIAL: Dense Image-text ALignment for Weakly Supervised Semantic Segmentation(https://arxiv.org/abs/2409.15801)
• Keywords: segmentation
• Abstract: Weakly supervised semantic segmentation (WSSS) approaches typically rely on class activation maps (CAMs) for initial seed generation, which often fail to capture global context due to limited supervision from image-level labels. To address this issue, we introduce DALNet, Dense Alignment Learning Network that leverages text embeddings to enhance the comprehensive understanding and precise localization of objects across different levels of granularity. Our key insight is to employ a dual-level alignment strategy: (1) Global Implicit Alignment (GIA) to capture global semantics by maximizing the similarity between the class token and the corresponding text embeddings while minimizing the similarity with background embeddings, and (2) Local Explicit Alignment (LEA) to improve object localization by utilizing spatial information from patch tokens. Moreover, we propose a cross-contrastive learning approach that aligns foreground features between image and text modalities while separating them from the background, encouraging activation in missing regions and suppressing distractions. Through extensive experiments on the PASCAL VOC and MS COCO datasets, we demonstrate that DALNet significantly outperforms state-of-the-art WSSS methods. Our approach, in particular, allows for more efficient end-to-end process as a single-stage method.

Title: 3D-JEPA: A Joint Embedding Predictive Architecture for 3D Self-Supervised Representation Learning

• Authors: Naiwen Hu, Haozhe Cheng, Yifan Xie, Shiqi Li, Jihua Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15803
• Pdf URL: https://arxiv.org/pdf/2409.15803
• Copy Paste: [[2409.15803]] 3D-JEPA: A Joint Embedding Predictive Architecture for 3D Self-Supervised Representation Learning(https://arxiv.org/abs/2409.15803)
• Keywords: generative
• Abstract: Invariance-based and generative methods have shown a conspicuous performance for 3D self-supervised representation learning (SSRL). However, the former relies on hand-crafted data augmentations that introduce bias not universally applicable to all downstream tasks, and the latter indiscriminately reconstructs masked regions, resulting in irrelevant details being saved in the representation space. To solve the problem above, we introduce 3D-JEPA, a novel non-generative 3D SSRL framework. Specifically, we propose a multi-block sampling strategy that produces a sufficiently informative context block and several representative target blocks. We present the context-aware decoder to enhance the reconstruction of the target blocks. Concretely, the context information is fed to the decoder continuously, facilitating the encoder in learning semantic modeling rather than memorizing the context information related to target blocks. Overall, 3D-JEPA predicts the representation of target blocks from a context block using the encoder and context-aware decoder architecture. Various downstream tasks on different datasets demonstrate 3D-JEPA's effectiveness and efficiency, achieving higher accuracy with fewer pretraining epochs, e.g., 88.65% accuracy on PB_T50_RS with 150 pretraining epochs.

Title: NER-Luxury: Named entity recognition for the fashion and luxury domain

• Authors: Akim Mousterou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15804
• Pdf URL: https://arxiv.org/pdf/2409.15804
• Copy Paste: [[2409.15804]] NER-Luxury: Named entity recognition for the fashion and luxury domain(https://arxiv.org/abs/2409.15804)
• Keywords: large language model
• Abstract: In this study, we address multiple challenges of developing a named-entity recognition model in English for the fashion and luxury industry, namely the entity disambiguation, French technical jargon in multiple sub-sectors, scarcity of the ESG methodology, and a disparate company structures of the sector with small and medium-sized luxury houses to large conglomerate leveraging economy of scale. In this work, we introduce a taxonomy of 36+ entity types with a luxury-oriented annotation scheme, and create a dataset of more than 40K sentences respecting a clear hierarchical classification. We also present five supervised fine-tuned models NER-Luxury for fashion, beauty, watches, jewelry, fragrances, cosmetics, and overall luxury, focusing equally on the aesthetic side and the quantitative side. In an additional experiment, we compare in a quantitative empirical assessment of the NER performance of our models against the state-of-the-art open-source large language models that show promising results and highlights the benefits of incorporating a bespoke NER model in existing machine learning pipelines.

Title: A Computer Vision Approach for Autonomous Cars to Drive Safe at Construction Zone

• Authors: Abu Shad Ahammed, Md Shahi Amran Hossain, Roman Obermaisser
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.15809
• Pdf URL: https://arxiv.org/pdf/2409.15809
• Copy Paste: [[2409.15809]] A Computer Vision Approach for Autonomous Cars to Drive Safe at Construction Zone(https://arxiv.org/abs/2409.15809)
• Keywords: secure, robust
• Abstract: To build a smarter and safer city, a secure, efficient, and sustainable transportation system is a key requirement. The autonomous driving system (ADS) plays an important role in the development of smart transportation and is considered one of the major challenges facing the automotive sector in recent decades. A car equipped with an autonomous driving system (ADS) comes with various cutting-edge functionalities such as adaptive cruise control, collision alerts, automated parking, and more. A primary area of research within ADAS involves identifying road obstacles in construction zones regardless of the driving environment. This paper presents an innovative and highly accurate road obstacle detection model utilizing computer vision technology that can be activated in construction zones and functions under diverse drift conditions, ultimately contributing to build a safer road transportation system. The model developed with the YOLO framework achieved a mean average precision exceeding 94\% and demonstrated an inference time of 1.6 milliseconds on the validation dataset, underscoring the robustness of the methodology applied to mitigate hazards and risks for autonomous vehicles.

Title: Aided design of bridge aesthetics based on Stable Diffusion fine-tuning

• Authors: Leye Zhang, Xiangxiang Tian, Chengli Zhang, Hongjun Zhang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15812
• Pdf URL: https://arxiv.org/pdf/2409.15812
• Copy Paste: [[2409.15812]] Aided design of bridge aesthetics based on Stable Diffusion fine-tuning(https://arxiv.org/abs/2409.15812)
• Keywords: diffusion
• Abstract: Stable Diffusion fine-tuning technique is tried to assist bridge-type innovation. The bridge real photo dataset is built, and Stable Diffusion is fine tuned by using four methods that are Textual Inversion, Dreambooth, Hypernetwork and Lora. All of them can capture the main characteristics of dataset images and realize the personalized customization of Stable Diffusion. Through fine-tuning, Stable Diffusion is not only a drawing tool, but also has the designer's innovative thinking ability. The fine tuned model can generate a large number of innovative new bridge types, which can provide rich inspiration for human designers. The result shows that this technology can be used as an engine of creativity and a power multiplier for human designers.

Title: Layer-wise Model Merging for Unsupervised Domain Adaptation in Segmentation Tasks

• Authors: Roberto Alcover-Couso, Juan C. SanMiguel, Marcos Escudero-Viñolo, Jose M Martínez
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.15813
• Pdf URL: https://arxiv.org/pdf/2409.15813
• Copy Paste: [[2409.15813]] Layer-wise Model Merging for Unsupervised Domain Adaptation in Segmentation Tasks(https://arxiv.org/abs/2409.15813)
• Keywords: robust, extraction, segmentation
• Abstract: Merging parameters of multiple models has resurfaced as an effective strategy to enhance task performance and robustness, but prior work is limited by the high costs of ensemble creation and inference. In this paper, we leverage the abundance of freely accessible trained models to introduce a cost-free approach to model merging. It focuses on a layer-wise integration of merged models, aiming to maintain the distinctiveness of the task-specific final layers while unifying the initial layers, which are primarily associated with feature extraction. This approach ensures parameter consistency across all layers, essential for boosting performance. Moreover, it facilitates seamless integration of knowledge, enabling effective merging of models from different datasets and tasks. Specifically, we investigate its applicability in Unsupervised Domain Adaptation (UDA), an unexplored area for model merging, for Semantic and Panoptic Segmentation. Experimental results demonstrate substantial UDA improvements without additional costs for merging same-architecture models from distinct datasets ($\uparrow 2.6\%$ mIoU) and different-architecture models with a shared backbone ($\uparrow 6.8\%$ mIoU). Furthermore, merging Semantic and Panoptic Segmentation models increases mPQ by $\uparrow 7\%$. These findings are validated across a wide variety of UDA strategies, architectures, and datasets.

Title: Empirical Insights on Fine-Tuning Large Language Models for Question-Answering

• Authors: Junjie Ye, Yuming Yang, Qi Zhang, Tao Gui, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15825
• Pdf URL: https://arxiv.org/pdf/2409.15825
• Copy Paste: [[2409.15825]] Empirical Insights on Fine-Tuning Large Language Models for Question-Answering(https://arxiv.org/abs/2409.15825)
• Keywords: large language model
• Abstract: Large language models (LLMs) encode extensive world knowledge through pre-training on massive datasets, which can then be fine-tuned for the question-answering (QA) task. However, effective strategies for fine-tuning LLMs for the QA task remain largely unexplored. To address this gap, we categorize supervised fine-tuning (SFT) data based on the extent of knowledge memorized by the pretrained LLMs and conduct a series of empirical analyses. Our experiments, involving four LLMs from three different model families, focus on three key factors: the amount of data required for SFT, the impact of different SFT datasets on model performance, and how data requirements vary across LLMs. The results show that as few as 60 data points during the SFT stage can activate the knowledge encoded during pre-training, enabling LLMs to perform the QA task. Additionally, SFT with data of varying memory levels has a significant impact on LLM performance, with the optimal dataset differing based on the specific model being fine-tuned. Future research will delve deeper into the mechanisms underlying these phenomena.

Title: Unveiling Language Competence Neurons: A Psycholinguistic Approach to Model Interpretability

• Authors: Xufeng Duan, Xinyu Zhou, Bei Xiao, Zhenguang G. Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15827
• Pdf URL: https://arxiv.org/pdf/2409.15827
• Copy Paste: [[2409.15827]] Unveiling Language Competence Neurons: A Psycholinguistic Approach to Model Interpretability(https://arxiv.org/abs/2409.15827)
• Keywords: interpretability, transformer, large language model
• Abstract: As large language models (LLMs) become advance in their linguistic capacity, understanding how they capture aspects of language competence remains a significant challenge. This study therefore employs psycholinguistic paradigms, which are well-suited for probing deeper cognitive aspects of language processing, to explore neuron-level representations in language model across three tasks: sound-shape association, sound-gender association, and implicit causality. Our findings indicate that while GPT-2-XL struggles with the sound-shape task, it demonstrates human-like abilities in both sound-gender association and implicit causality. Targeted neuron ablation and activation manipulation reveal a crucial relationship: when GPT-2-XL displays a linguistic ability, specific neurons correspond to that competence; conversely, the absence of such an ability indicates a lack of specialized neurons. This study is the first to utilize psycholinguistic experiments to investigate deep language competence at the neuron level, providing a new level of granularity in model interpretability and insights into the internal mechanisms driving language ability in transformer based LLMs.

Title: Potential Field as Scene Affordance for Behavior Change-Based Visual Risk Object Identification

• Authors: Pang-Yuan Pao, Shu-Wei Lu, Ze-Yan Lu, Yi-Ting Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15846
• Pdf URL: https://arxiv.org/pdf/2409.15846
• Copy Paste: [[2409.15846]] Potential Field as Scene Affordance for Behavior Change-Based Visual Risk Object Identification(https://arxiv.org/abs/2409.15846)
• Keywords: segmentation
• Abstract: We study behavior change-based visual risk object identification (Visual-ROI), a critical framework designed to detect potential hazards for intelligent driving systems. Existing methods often show significant limitations in spatial accuracy and temporal consistency, stemming from an incomplete understanding of scene affordance. For example, these methods frequently misidentify vehicles that do not impact the ego vehicle as risk objects. Furthermore, existing behavior change-based methods are inefficient because they implement causal inference in the perspective image space. We propose a new framework with a Bird's Eye View (BEV) representation to overcome the above challenges. Specifically, we utilize potential fields as scene affordance, involving repulsive forces derived from road infrastructure and traffic participants, along with attractive forces sourced from target destinations. In this work, we compute potential fields by assigning different energy levels according to the semantic labels obtained from BEV semantic segmentation. We conduct thorough experiments and ablation studies, comparing the proposed method with various state-of-the-art algorithms on both synthetic and real-world datasets. Our results show a notable increase in spatial and temporal consistency, with enhancements of 20.3% and 11.6% on the RiskBench dataset, respectively. Additionally, we can improve computational efficiency by 88%. We achieve improvements of 5.4% in spatial accuracy and 7.2% in temporal consistency on the nuScenes dataset.

Title: iGAiVA: Integrated Generative AI and Visual Analytics in a Machine Learning Workflow for Text Classification

• Authors: Yuanzhe Jin, Adrian Carrasco-Revilla, Min Chen
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15848
• Pdf URL: https://arxiv.org/pdf/2409.15848
• Copy Paste: [[2409.15848]] iGAiVA: Integrated Generative AI and Visual Analytics in a Machine Learning Workflow for Text Classification(https://arxiv.org/abs/2409.15848)
• Keywords: generative, large language model
• Abstract: In developing machine learning (ML) models for text classification, one common challenge is that the collected data is often not ideally distributed, especially when new classes are introduced in response to changes of data and tasks. In this paper, we present a solution for using visual analytics (VA) to guide the generation of synthetic data using large language models. As VA enables model developers to identify data-related deficiency, data synthesis can be targeted to address such deficiency. We discuss different types of data deficiency, describe different VA techniques for supporting their identification, and demonstrate the effectiveness of targeted data synthesis in improving model accuracy. In addition, we present a software tool, iGAiVA, which maps four groups of ML tasks into four VA views, integrating generative AI and VA into an ML workflow for developing and improving text classification models.

Title: A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding

• Authors: Abdulfattah Safa, Gözde Gül Şahin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15861
• Pdf URL: https://arxiv.org/pdf/2409.15861
• Copy Paste: [[2409.15861]] A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding(https://arxiv.org/abs/2409.15861)
• Keywords: large language model
• Abstract: Dialogue State Tracking (DST) is crucial for understanding user needs and executing appro- priate system actions in task-oriented dialogues. Majority of existing DST methods are designed to work within predefined ontologies and as- sume the availability of gold domain labels, struggling with adapting to new slots values. While Large Language Models (LLMs)-based systems show promising zero-shot DST perfor- mance, they either require extensive computa- tional resources or they underperform existing fully-trained systems, limiting their practical- ity. To address these limitations, we propose a zero-shot, open-vocabulary system that in- tegrates domain classification and DST in a single pipeline. Our approach includes refor- mulating DST as a question-answering task for less capable models and employing self- refining prompts for more adaptable ones. Our system does not rely on fixed slot values de- fined in the ontology allowing the system to adapt dynamically. We compare our approach with existing SOTA, and show that it provides up to 20% better Joint Goal Accuracy (JGA) over previous methods on datasets like Multi- WOZ 2.1, with up to 90% fewer requests to the LLM API.

Title: Privacy Evaluation Benchmarks for NLP Models

• Authors: Wei Huang, Yinggui Wang, Cen Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15868
• Pdf URL: https://arxiv.org/pdf/2409.15868
• Copy Paste: [[2409.15868]] Privacy Evaluation Benchmarks for NLP Models(https://arxiv.org/abs/2409.15868)
• Keywords: privacy, defense, attack, large language model
• Abstract: By inducing privacy attacks on NLP models, attackers can obtain sensitive information such as training data and model parameters, etc. Although researchers have studied, in-depth, several kinds of attacks in NLP models, they are non-systematic analyses. It lacks a comprehensive understanding of the impact caused by the attacks. For example, we must consider which scenarios can apply to which attacks, what the common factors are that affect the performance of different attacks, the nature of the relationships between different attacks, and the influence of various datasets and models on the effectiveness of the attacks, etc. Therefore, we need a benchmark to holistically assess the privacy risks faced by NLP models. In this paper, we present a privacy attack and defense evaluation benchmark in the field of NLP, which includes the conventional/small models and large language models (LLMs). This benchmark supports a variety of models, datasets, and protocols, along with standardized modules for comprehensive evaluation of attacks and defense strategies. Based on the above framework, we present a study on the association between auxiliary data from different domains and the strength of privacy attacks. And we provide an improved attack method in this scenario with the help of Knowledge Distillation (KD). Furthermore, we propose a chained framework for privacy attacks. Allowing a practitioner to chain multiple attacks to achieve a higher-level attack objective. Based on this, we provide some defense and enhanced attack strategies. The code for reproducing the results can be found at this https URL.

Title: Zero-Shot Detection of AI-Generated Images

• Authors: Davide Cozzolino, Giovanni Poggi, Matthias Nießner, Luisa Verdoliva
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15875
• Pdf URL: https://arxiv.org/pdf/2409.15875
• Copy Paste: [[2409.15875]] Zero-Shot Detection of AI-Generated Images(https://arxiv.org/abs/2409.15875)
• Keywords: diffusion, generative
• Abstract: Detecting AI-generated images has become an extraordinarily difficult challenge as new generative architectures emerge on a daily basis with more and more capabilities and unprecedented realism. New versions of many commercial tools, such as DALLE, Midjourney, and Stable Diffusion, have been released recently, and it is impractical to continually update and retrain supervised forensic detectors to handle such a large variety of models. To address this challenge, we propose a zero-shot entropy-based detector (ZED) that neither needs AI-generated training data nor relies on knowledge of generative architectures to artificially synthesize their artifacts. Inspired by recent works on machine-generated text detection, our idea is to measure how surprising the image under analysis is compared to a model of real images. To this end, we rely on a lossless image encoder that estimates the probability distribution of each pixel given its context. To ensure computational efficiency, the encoder has a multi-resolution architecture and contexts comprise mostly pixels of the lower-resolution version of the image.Since only real images are needed to learn the model, the detector is independent of generator architectures and synthetic training data. Using a single discriminative feature, the proposed detector achieves state-of-the-art performance. On a wide variety of generative models it achieves an average improvement of more than 3% over the SoTA in terms of accuracy. Code is available at this https URL.

Title: Exploring VQ-VAE with Prosody Parameters for Speaker Anonymization

• Authors: Sotheara Leang (CADT, M-PSI), Anderson Augusma (M-PSI, SVH), Eric Castelli (M-PSI), Frédérique Letué (SAM), Sethserey Sam (CADT), Dominique Vaufreydaz (M-PSI)
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.15882
• Pdf URL: https://arxiv.org/pdf/2409.15882
• Copy Paste: [[2409.15882]] Exploring VQ-VAE with Prosody Parameters for Speaker Anonymization(https://arxiv.org/abs/2409.15882)
• Keywords: privacy
• Abstract: Human speech conveys prosody, linguistic content, and speaker identity. This article investigates a novel speaker anonymization approach using an end-to-end network based on a Vector-Quantized Variational Auto-Encoder (VQ-VAE) to deal with these speech components. This approach is designed to disentangle these components to specifically target and modify the speaker identity while preserving the linguistic and emotionalcontent. To do so, three separate branches compute embeddings for content, prosody, and speaker identity respectively. During synthesis, taking these embeddings, the decoder of the proposed architecture is conditioned on both speaker and prosody information, allowing for capturing more nuanced emotional states and precise adjustments to speaker identification. Findings indicate that this method outperforms most baseline techniques in preserving emotional information. However, it exhibits more limited performance on other voice privacy tasks, emphasizing the need for further improvements.

Title: CAD: Memory Efficient Convolutional Adapter for Segment Anything

• Authors: Joohyeok Kim, Joonhyeon Song, Seohwan Yun, Seongho Yoon, Sangmin Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15889
• Pdf URL: https://arxiv.org/pdf/2409.15889
• Copy Paste: [[2409.15889]] CAD: Memory Efficient Convolutional Adapter for Segment Anything(https://arxiv.org/abs/2409.15889)
• Keywords: segmentation
• Abstract: The Foundation model for image segmentation, Segment Anything (SAM), has been actively researched in various fields since its proposal. Various researches have been proposed to adapt SAM to specific domains, with one notable approach involving the addition and training of lightweight adapter modules. While adapter-based fine-tuning approaches have reported parameter efficiency and significant performance improvements, they face a often overlooked issue: the excessive consumption of GPU memory relative to the number of trainable parameters. Addressing this issue, this paper proposes a memory-efficient parallel convolutional adapter architecture. This architecture connects in parallel with SAM's image encoder, eliminating the need to store activations and gradients of the image encoder during model training. Our proposed architecture demonstrated competitive experimental results while using less than half the GPU memory compared to SAM Adapter, indicating its value as an alternative to simple decoder fine-tuning when hardware limitations preclude adapter-based learning. Our code implementation is available at our github.

Title: HLB: Benchmarking LLMs' Humanlikeness in Language Use

• Authors: Xufeng Duan, Bei Xiao, Xuemei Tang, Zhenguang G. Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15890
• Pdf URL: https://arxiv.org/pdf/2409.15890
• Copy Paste: [[2409.15890]] HLB: Benchmarking LLMs' Humanlikeness in Language Use(https://arxiv.org/abs/2409.15890)
• Keywords: extraction, large language model
• Abstract: As synthetic data becomes increasingly prevalent in training language models, particularly through generated dialogue, concerns have emerged that these models may deviate from authentic human language patterns, potentially losing the richness and creativity inherent in human communication. This highlights the critical need to assess the humanlikeness of language models in real-world language use. In this paper, we present a comprehensive humanlikeness benchmark (HLB) evaluating 20 large language models (LLMs) using 10 psycholinguistic experiments designed to probe core linguistic aspects, including sound, word, syntax, semantics, and discourse (see this https URL). To anchor these comparisons, we collected responses from over 2,000 human participants and compared them to outputs from the LLMs in these experiments. For rigorous evaluation, we developed a coding algorithm that accurately identified language use patterns, enabling the extraction of response distributions for each task. By comparing the response distributions between human participants and LLMs, we quantified humanlikeness through distributional similarity. Our results reveal fine-grained differences in how well LLMs replicate human responses across various linguistic levels. Importantly, we found that improvements in other performance metrics did not necessarily lead to greater humanlikeness, and in some cases, even resulted in a decline. By introducing psycholinguistic methods to model evaluation, this benchmark offers the first framework for systematically assessing the humanlikeness of LLMs in language use.

Title: Unsupervised Attention Regularization Based Domain Adaptation for Oracle Character Recognition

• Authors: Mei Wang, Weihong Deng, Jiani Hu, Sen Su
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15893
• Pdf URL: https://arxiv.org/pdf/2409.15893
• Copy Paste: [[2409.15893]] Unsupervised Attention Regularization Based Domain Adaptation for Oracle Character Recognition(https://arxiv.org/abs/2409.15893)
• Keywords: robust, interpretability
• Abstract: The study of oracle characters plays an important role in Chinese archaeology and philology. However, the difficulty of collecting and annotating real-world scanned oracle characters hinders the development of oracle character recognition. In this paper, we develop a novel unsupervised domain adaptation (UDA) method, i.e., unsupervised attention regularization net?work (UARN), to transfer recognition knowledge from labeled handprinted oracle characters to unlabeled scanned data. First, we experimentally prove that existing UDA methods are not always consistent with human priors and cannot achieve optimal performance on the target domain. For these oracle characters with flip-insensitivity and high inter-class similarity, model interpretations are not flip-consistent and class-separable. To tackle this challenge, we take into consideration visual perceptual plausibility when adapting. Specifically, our method enforces attention consistency between the original and flipped images to achieve the model robustness to flipping. Simultaneously, we constrain attention separability between the pseudo class and the most confusing class to improve the model discriminability. Extensive experiments demonstrate that UARN shows better interpretability and achieves state-of-the-art performance on Oracle-241 dataset, substantially outperforming the previously structure-texture separation network by 8.5%.

Title: FedRepOpt: Gradient Re-parametrized Optimizers in Federated Learning

• Authors: Kin Wai Lau, Yasar Abbas Ur Rehman, Pedro Porto Buarque de Gusmão, Lai-Man Po, Lan Ma, Yuyang Xie
• Subjects: cs.LG, cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.15898
• Pdf URL: https://arxiv.org/pdf/2409.15898
• Copy Paste: [[2409.15898]] FedRepOpt: Gradient Re-parametrized Optimizers in Federated Learning(https://arxiv.org/abs/2409.15898)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) has emerged as a privacy-preserving method for training machine learning models in a distributed manner on edge devices. However, on-device models face inherent computational power and memory limitations, potentially resulting in constrained gradient updates. As the model's size increases, the frequency of gradient updates on edge devices decreases, ultimately leading to suboptimal training outcomes during any particular FL round. This limits the feasibility of deploying advanced and large-scale models on edge devices, hindering the potential for performance enhancements. To address this issue, we propose FedRepOpt, a gradient re-parameterized optimizer for FL. The gradient re-parameterized method allows training a simple local model with a similar performance as a complex model by modifying the optimizer's gradients according to a set of model-specific hyperparameters obtained from the complex models. In this work, we focus on VGG-style and Ghost-style models in the FL environment. Extensive experiments demonstrate that models using FedRepOpt obtain a significant boost in performance of 16.7% and 11.4% compared to the RepGhost-style and RepVGG-style networks, while also demonstrating a faster convergence time of 11.7% and 57.4% compared to their complex structure.

Title: Konstruktor: A Strong Baseline for Simple Knowledge Graph Question Answering

• Authors: Maria Lysyuk, Mikhail Salnikov, Pavel Braslavski, Alexander Panchenko
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15902
• Pdf URL: https://arxiv.org/pdf/2409.15902
• Copy Paste: [[2409.15902]] Konstruktor: A Strong Baseline for Simple Knowledge Graph Question Answering(https://arxiv.org/abs/2409.15902)
• Keywords: robust, extraction, interpretability, large language model
• Abstract: While being one of the most popular question types, simple questions such as "Who is the author of Cinderella?", are still not completely solved. Surprisingly, even the most powerful modern Large Language Models are prone to errors when dealing with such questions, especially when dealing with rare entities. At the same time, as an answer may be one hop away from the question entity, one can try to develop a method that uses structured knowledge graphs (KGs) to answer such questions. In this paper, we introduce Konstruktor - an efficient and robust approach that breaks down the problem into three steps: (i) entity extraction and entity linking, (ii) relation prediction, and (iii) querying the knowledge graph. Our approach integrates language models and knowledge graphs, exploiting the power of the former and the interpretability of the latter. We experiment with two named entity recognition and entity linking methods and several relation detection techniques. We show that for relation detection, the most challenging step of the workflow, a combination of relation classification/generation and ranking outperforms other methods. We report Konstruktor's strong results on four datasets.

Title: Enhancing Text-to-SQL Capabilities of Large Language Models via Domain Database Knowledge Injection

• Authors: Xingyu Ma, Xin Tian, Lingxiang Wu, Xuepeng Wang, Xueming Tang, Jinqiao Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15907
• Pdf URL: https://arxiv.org/pdf/2409.15907
• Copy Paste: [[2409.15907]] Enhancing Text-to-SQL Capabilities of Large Language Models via Domain Database Knowledge Injection(https://arxiv.org/abs/2409.15907)
• Keywords: large language model
• Abstract: Text-to-SQL is a subtask in semantic parsing that has seen rapid progress with the evolution of Large Language Models (LLMs). However, LLMs face challenges due to hallucination issues and a lack of domain-specific database knowledge(such as table schema and cell values). As a result, they can make errors in generating table names, columns, and matching values to the correct columns in SQL statements. This paper introduces a method of knowledge injection to enhance LLMs' ability to understand schema contents by incorporating prior knowledge. This approach improves their performance in Text-to-SQL tasks. Experimental results show that pre-training LLMs on domain-specific database knowledge and fine-tuning them on downstream Text-to-SQL tasks significantly improves the Execution Match (EX) and Exact Match (EM) metrics across various models. This effectively reduces errors in generating column names and matching values to the columns. Furthermore, the knowledge-injected models can be applied to many downstream Text-to-SQL tasks, demonstrating the generalizability of the approach presented in this paper.

Title: A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translation

• Authors: Xiaoqian Liu, Yangfan Du, Jianjin Wang, Yuan Ge, Chen Xu, Tong Xiao, Guocheng Chen, Jingbo Zhu
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.15911
• Pdf URL: https://arxiv.org/pdf/2409.15911
• Copy Paste: [[2409.15911]] A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translation(https://arxiv.org/abs/2409.15911)
• Keywords: robust
• Abstract: Simultaneous Speech Translation (SimulST) involves generating target language text while continuously processing streaming speech input, presenting significant real-time challenges. Multi-task learning is often employed to enhance SimulST performance but introduces optimization conflicts between primary and auxiliary tasks, potentially compromising overall efficiency. The existing model-level conflict resolution methods are not well-suited for this task which exacerbates inefficiencies and leads to high GPU memory consumption. To address these challenges, we propose a Modular Gradient Conflict Mitigation (MGCM) strategy that detects conflicts at a finer-grained modular level and resolves them utilizing gradient projection. Experimental results demonstrate that MGCM significantly improves SimulST performance, particularly under medium and high latency conditions, achieving a 0.68 BLEU score gain in offline tasks. Additionally, MGCM reduces GPU memory consumption by over 95\% compared to other conflict mitigation methods, establishing it as a robust solution for SimulST tasks.

Title: Learning Compact Channel Correlation Representation for LiDAR Place Recognition

• Authors: Saimunur Rahman, Peyman Moghadam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15919
• Pdf URL: https://arxiv.org/pdf/2409.15919
• Copy Paste: [[2409.15919]] Learning Compact Channel Correlation Representation for LiDAR Place Recognition(https://arxiv.org/abs/2409.15919)
• Keywords: robust
• Abstract: This paper presents a novel approach to learn compact channel correlation representation for LiDAR place recognition, called C3R, aimed at reducing the computational burden and dimensionality associated with traditional covariance pooling methods for place recognition tasks. Our method partitions the feature matrix into smaller groups, computes group-wise covariance matrices, and aggregates them via a learnable aggregation strategy. Matrix power normalization is applied to ensure stability. Theoretical analyses are also given to demonstrate the effectiveness of the proposed method, including its ability to preserve permutation invariance and maintain high mutual information between the original features and the aggregated representation. We conduct extensive experiments on four large-scale, public LiDAR place recognition datasets including Oxford RobotCar, In-house, MulRan, and WildPlaces datasets to validate our approach's superiority in accuracy, and robustness. Furthermore, we provide the quantitative results of our approach for a deeper understanding. The code will be released upon acceptance.

Title: Multilingual Transfer and Domain Adaptation for Low-Resource Languages of Spain

• Authors: Yuanchang Luo, Zhanglin Wu, Daimeng Wei, Hengchao Shang, Zongyao Li, Jiaxin Guo, Zhiqiang Rao, Shaojun Li, Jinlong Yang, Yuhao Xie, Jiawei Zheng Bin Wei, Hao Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15924
• Pdf URL: https://arxiv.org/pdf/2409.15924
• Copy Paste: [[2409.15924]] Multilingual Transfer and Domain Adaptation for Low-Resource Languages of Spain(https://arxiv.org/abs/2409.15924)
• Keywords: transformer
• Abstract: This article introduces the submission status of the Translation into Low-Resource Languages of Spain task at (WMT 2024) by Huawei Translation Service Center (HW-TSC). We participated in three translation tasks: spanish to aragonese (es-arg), spanish to aranese (es-arn), and spanish to asturian (es-ast). For these three translation tasks, we use training strategies such as multilingual transfer, regularized dropout, forward translation and back translation, labse denoising, transduction ensemble learning and other strategies to neural machine translation (NMT) model based on training deep transformer-big architecture. By using these enhancement strategies, our submission achieved a competitive result in the final evaluation.

Title: SLIMER-IT: Zero-Shot NER on Italian Language

• Authors: Andrew Zamai, Leonardo Rigutini, Marco Maggini, Andrea Zugarini
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.15933
• Pdf URL: https://arxiv.org/pdf/2409.15933
• Copy Paste: [[2409.15933]] SLIMER-IT: Zero-Shot NER on Italian Language(https://arxiv.org/abs/2409.15933)
• Keywords: large language model
• Abstract: Traditional approaches to Named Entity Recognition (NER) frame the task into a BIO sequence labeling problem. Although these systems often excel in the downstream task at hand, they require extensive annotated data and struggle to generalize to out-of-distribution input domains and unseen entity types. On the contrary, Large Language Models (LLMs) have demonstrated strong zero-shot capabilities. While several works address Zero-Shot NER in English, little has been done in other languages. In this paper, we define an evaluation framework for Zero-Shot NER, applying it to the Italian language. Furthermore, we introduce SLIMER-IT, the Italian version of SLIMER, an instruction-tuning approach for zero-shot NER leveraging prompts enriched with definition and guidelines. Comparisons with other state-of-the-art models, demonstrate the superiority of SLIMER-IT on never-seen-before entity tags.

Title: A Formalization of Image Vectorization by Region Merging

• Authors: Roy Y. He, Sung Ha Kang, Jean-Michel Morel
• Subjects: cs.CV, cs.GR, math.NA
• Abstract URL: https://arxiv.org/abs/2409.15940
• Pdf URL: https://arxiv.org/pdf/2409.15940
• Copy Paste: [[2409.15940]] A Formalization of Image Vectorization by Region Merging(https://arxiv.org/abs/2409.15940)
• Keywords: segmentation
• Abstract: Image vectorization converts raster images into vector graphics composed of regions separated by curves. Typical vectorization methods first define the regions by grouping similar colored regions via color quantization, then approximate their boundaries by Bezier curves. In that way, the raster input is converted into an SVG format parameterizing the regions' colors and the Bezier control points. This compact representation has many graphical applications thanks to its universality and resolution-independence. In this paper, we remark that image vectorization is nothing but an image segmentation, and that it can be built by fine to coarse region merging. Our analysis of the problem leads us to propose a vectorization method alternating region merging and curve smoothing. We formalize the method by alternate operations on the dual and primal graph induced from any domain partition. In that way, we address a limitation of current vectorization methods, which separate the update of regional information from curve approximation. We formalize region merging methods by associating them with various gain functionals, including the classic Beaulieu-Goldberg and Mumford-Shah functionals. More generally, we introduce and compare region merging criteria involving region number, scale, area, and internal standard deviation. We also show that the curve smoothing, implicit in all vectorization methods, can be performed by the shape-preserving affine scale space. We extend this flow to a network of curves and give a sufficient condition for the topological preservation of the segmentation. The general vectorization method that follows from this analysis shows explainable behaviors, explicitly controlled by a few intuitive parameters. It is experimentally compared to state-of-the-art software and proved to have comparable or superior fidelity and cost efficiency.

Title: Vulnerabilities that arise from poor governance in Distributed Ledger Technologies

• Authors: Aida Manzano Kharman, William Sanders
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.15947
• Pdf URL: https://arxiv.org/pdf/2409.15947
• Copy Paste: [[2409.15947]] Vulnerabilities that arise from poor governance in Distributed Ledger Technologies(https://arxiv.org/abs/2409.15947)
• Keywords: security, attack
• Abstract: Current implementations of governance in Distributed Ledger Technologies leave them susceptible to a number of attacks. We survey the state of the art of Distributed Ledger Technologies (DLTs) governance protocols and work carried out to systematise good governance properties in the context of DLTs. We then select the most appropriate taxonomy of good governance properties and point to formal security notions that good governance protocols should satisfy. We point practitioners to existing solutions to deliver them, where possible. Furthermore, we outline a number of vulnerabilities that arise in the absence of good governance properties. We call on the research community and DLT research practitioners to prioritise delivering these good governance properties and continue to develop tools to do so, to avoid attacks to DLT protocols that exploit their poor governance models.

Title: Mind the Prompt: A Novel Benchmark for Prompt-based Class-Agnostic Counting

• Authors: Luca Ciampi, Nicola Messina, Matteo Pierucci, Giuseppe Amato, Marco Avvenuti, Fabrizio Falchi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15953
• Pdf URL: https://arxiv.org/pdf/2409.15953
• Copy Paste: [[2409.15953]] Mind the Prompt: A Novel Benchmark for Prompt-based Class-Agnostic Counting(https://arxiv.org/abs/2409.15953)
• Keywords: robust
• Abstract: Class-agnostic counting (CAC) is a recent task in computer vision that aims to estimate the number of instances of arbitrary object classes never seen during model training. With the recent advancement of robust vision-and-language foundation models, there is a growing interest in prompt-based CAC, where object categories to be counted can be specified using natural language. However, we identify significant limitations in current benchmarks for evaluating this task, which hinder both accurate assessment and the development of more effective solutions. Specifically, we argue that the current evaluation protocols do not measure the ability of the model to understand which object has to be counted. This is due to two main factors: (i) the shortcomings of CAC datasets, which primarily consist of images containing objects from a single class, and (ii) the limitations of current counting performance evaluators, which are based on traditional class-specific counting and focus solely on counting errors. To fill this gap, we introduce the Prompt-Aware Counting (PrACo) benchmark, which comprises two targeted tests, each accompanied by appropriate evaluation metrics. We evaluate state-of-the-art methods and demonstrate that, although some achieve impressive results on standard class-specific counting metrics, they exhibit a significant deficiency in understanding the input prompt, indicating the need for more careful training procedures or revised designs. The code for reproducing our results is available at this https URL.

Title: Historical Trajectory Assisted Zeroth-Order Federated Optimization

• Authors: Xiaoyu He, Chenlin Wu, Zike Li, Zibin Zheng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15955
• Pdf URL: https://arxiv.org/pdf/2409.15955
• Copy Paste: [[2409.15955]] Historical Trajectory Assisted Zeroth-Order Federated Optimization(https://arxiv.org/abs/2409.15955)
• Keywords: federate
• Abstract: Federated learning is a distributed learning framework which enables clients to train models individually and to upload their model updates for aggregation. The local training process heavily relies on distributed gradient descent techniques. In the situation where gradient information is not available, the gradients need to be estimated from zeroth-order information, which typically involves computing finite-differences along isotropic random directions. This method suffers from high estimation errors, as the geometric features of the objective landscape may be overlooked during the isotropic sampling. In this work, we propose a non-isotropic sampling method to improve the gradient estimation procedure. Gradients in our method are estimated in a subspace spanned by historical trajectories of solutions, aiming to encourage the exploration of promising regions and hence improve the convergence. We implement this method in zeroth-order federated settings, and show that the convergence rate aligns with existing ones while introducing no significant overheads in communication or local computation. The effectiveness of our proposal is verified on several numerical experiments in comparison to several commonly-used zeroth-order federated optimization algorithms.

Title: Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality

• Authors: Hannah Schieber, Jacob Young, Tobias Langlotz, Stefanie Zollmann, Daniel Roth
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.15959
• Pdf URL: https://arxiv.org/pdf/2409.15959
• Copy Paste: [[2409.15959]] Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality(https://arxiv.org/abs/2409.15959)
• Keywords: extraction, segmentation
• Abstract: Advancements in 3D rendering like Gaussian Splatting (GS) allow novel view synthesis and real-time rendering in virtual reality (VR). However, GS-created 3D environments are often difficult to edit. For scene enhancement or to incorporate 3D assets, segmenting Gaussians by class is essential. Existing segmentation approaches are typically limited to certain types of scenes, e.g., ''circular'' scenes, to determine clear object boundaries. However, this method is ineffective when removing large objects in non-''circling'' scenes such as large outdoor scenes. We propose Semantics-Controlled GS (SCGS), a segmentation-driven GS approach, enabling the separation of large scene parts in uncontrolled, natural environments. SCGS allows scene editing and the extraction of scene parts for VR. Additionally, we introduce a challenging outdoor dataset, overcoming the ''circling'' setup. We outperform the state-of-the-art in visual quality on our dataset and in segmentation quality on the 3D-OVS dataset. We conducted an exploratory user study, comparing a 360-video, plain GS, and SCGS in VR with a fixed viewpoint. In our subsequent main study, users were allowed to move freely, evaluating plain GS and SCGS. Our main study results show that participants clearly prefer SCGS over plain GS. We overall present an innovative approach that surpasses the state-of-the-art both technically and in user experience.

Title: Adversarial Backdoor Defense in CLIP

• Authors: Junhao Kuang, Siyuan Liang, Jiawei Liang, Kuanrong Liu, Xiaochun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15968
• Pdf URL: https://arxiv.org/pdf/2409.15968
• Copy Paste: [[2409.15968]] Adversarial Backdoor Defense in CLIP(https://arxiv.org/abs/2409.15968)
• Keywords: defense, attack, robust
• Abstract: Multimodal contrastive pretraining, exemplified by models like CLIP, has been found to be vulnerable to backdoor attacks. While current backdoor defense methods primarily employ conventional data augmentation to create augmented samples aimed at feature alignment, these methods fail to capture the distinct features of backdoor samples, resulting in suboptimal defense performance. Observations reveal that adversarial examples and backdoor samples exhibit similarities in the feature space within the compromised models. Building on this insight, we propose Adversarial Backdoor Defense (ABD), a novel data augmentation strategy that aligns features with meticulously crafted adversarial examples. This approach effectively disrupts the backdoor association. Our experiments demonstrate that ABD provides robust defense against both traditional uni-modal and multimodal backdoor attacks targeting CLIP. Compared to the current state-of-the-art defense method, CleanCLIP, ABD reduces the attack success rate by 8.66% for BadNet, 10.52% for Blended, and 53.64% for BadCLIP, while maintaining a minimal average decrease of just 1.73% in clean accuracy.

Title: Edge-device Collaborative Computing for Multi-view Classification

• Authors: Marco Palena, Tania Cerquitelli, Carla Fabiana Chiasserini
• Subjects: cs.LG, cs.AI, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.15973
• Pdf URL: https://arxiv.org/pdf/2409.15973
• Copy Paste: [[2409.15973]] Edge-device Collaborative Computing for Multi-view Classification(https://arxiv.org/abs/2409.15973)
• Keywords: privacy, robust
• Abstract: Motivated by the proliferation of Internet-of-Thing (IoT) devices and the rapid advances in the field of deep learning, there is a growing interest in pushing deep learning computations, conventionally handled by the cloud, to the edge of the network to deliver faster responses to end users, reduce bandwidth consumption to the cloud, and address privacy concerns. However, to fully realize deep learning at the edge, two main challenges still need to be addressed: (i) how to meet the high resource requirements of deep learning on resource-constrained devices, and (ii) how to leverage the availability of multiple streams of spatially correlated data, to increase the effectiveness of deep learning and improve application-level performance. To address the above challenges, we explore collaborative inference at the edge, in which edge nodes and end devices share correlated data and the inference computational burden by leveraging different ways to split computation and fuse data. Besides traditional centralized and distributed schemes for edge-end device collaborative inference, we introduce selective schemes that decrease bandwidth resource consumption by effectively reducing data redundancy. As a reference scenario, we focus on multi-view classification in a networked system in which sensing nodes can capture overlapping fields of view. The proposed schemes are compared in terms of accuracy, computational expenditure at the nodes, communication overhead, inference latency, robustness, and noise sensitivity. Experimental results highlight that selective collaborative schemes can achieve different trade-offs between the above performance metrics, with some of them bringing substantial communication savings (from 18% to 74% of the transmitted data with respect to centralized inference) while still keeping the inference accuracy well above 90%.

Title: Finetuning LLMs for Comparative Assessment Tasks

• Authors: Vatsal Raina, Adian Liusie, Mark Gales
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15979
• Pdf URL: https://arxiv.org/pdf/2409.15979
• Copy Paste: [[2409.15979]] Finetuning LLMs for Comparative Assessment Tasks(https://arxiv.org/abs/2409.15979)
• Keywords: large language model
• Abstract: Automated assessment in natural language generation is a challenging task. Instruction-tuned large language models (LLMs) have shown promise in reference-free evaluation, particularly through comparative assessment. However, the quadratic computational complexity of pairwise comparisons limits its scalability. To address this, efficient comparative assessment has been explored by applying comparative strategies on zero-shot LLM probabilities. We propose a framework for finetuning LLMs for comparative assessment to align the model's output with the target distribution of comparative probabilities. By training on soft probabilities, our approach improves state-of-the-art performance while maintaining high performance with an efficient subset of comparisons.

Title: Improvements to SDXL in NovelAI Diffusion V3

• Authors: Juan Ossa, Eren Doğan, Alex Birch, F. Johnson
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15997
• Pdf URL: https://arxiv.org/pdf/2409.15997
• Copy Paste: [[2409.15997]] Improvements to SDXL in NovelAI Diffusion V3(https://arxiv.org/abs/2409.15997)
• Keywords: diffusion
• Abstract: In this technical report, we document the changes we made to SDXL in the process of training NovelAI Diffusion V3, our state of the art anime image generation model.

Title: Unleashing the Potential of Synthetic Images: A Study on Histopathology Image Classification

• Authors: Leire Benito-Del-Valle, Aitor Alvarez-Gila, Itziar Eguskiza, Cristina L. Saratxaga
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16002
• Pdf URL: https://arxiv.org/pdf/2409.16002
• Copy Paste: [[2409.16002]] Unleashing the Potential of Synthetic Images: A Study on Histopathology Image Classification(https://arxiv.org/abs/2409.16002)
• Keywords: diffusion, transformer, generative
• Abstract: Histopathology image classification is crucial for the accurate identification and diagnosis of various diseases but requires large and diverse datasets. Obtaining such datasets, however, is often costly and time-consuming due to the need for expert annotations and ethical constraints. To address this, we examine the suitability of different generative models and image selection approaches to create realistic synthetic histopathology image patches conditioned on class labels. Our findings highlight the importance of selecting an appropriate generative model type and architecture to enhance performance. Our experiments over the PCam dataset show that diffusion models are effective for transfer learning, while GAN-generated samples are better suited for augmentation. Additionally, transformer-based generative models do not require image filtering, in contrast to those derived from Convolutional Neural Networks (CNNs), which benefit from realism score-based selection. Therefore, we show that synthetic images can effectively augment existing datasets, ultimately improving the performance of the downstream histopathology image classification task.

Title: Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs

• Authors: Yang Yuhang, Peng Yizhou, Eng Siong Chng, Xionghu Zhong
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.16005
• Pdf URL: https://arxiv.org/pdf/2409.16005
• Copy Paste: [[2409.16005]] Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs(https://arxiv.org/abs/2409.16005)
• Keywords: large language model
• Abstract: The integration of large language models (LLMs) with pre-trained speech models has opened up new avenues in automatic speech recognition (ASR). While LLMs excel in multimodal understanding tasks, effectively leveraging their capabilities for ASR remains a significant challenge. This paper presents a novel training approach to enhance LLM performance in ASR tasks. We propose pre-training LLMs on Pinyin embedding sequences, which represent pronunciation features, to generate corresponding Chinese characters. This step enables the LLM to adapt to generating text from pronunciation features before encountering real speech data. Furthermore, we fine-tune the LoRA parameters to enhance the LLM's understanding of speech modality information. In AISHELL-1 corpus, our approach yields a 9.5% relative improvement in ASR tasks compared to the baseline without Pinyi-to-Character pre-training. Additionally, incorporating auxiliary text data for Pinyi-to-Character pre-training further boosts performance, achieving a 19.0% relative improvement.

Title: Lattice-Based Vulnerabilities in Lee Metric Post-Quantum Cryptosystems

• Authors: Anna-Lena Horlemann, Karan Khathuria, Marc Newman, Amin Sakzad, Carlos Vela Cabello
• Subjects: cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2409.16018
• Pdf URL: https://arxiv.org/pdf/2409.16018
• Copy Paste: [[2409.16018]] Lattice-Based Vulnerabilities in Lee Metric Post-Quantum Cryptosystems(https://arxiv.org/abs/2409.16018)
• Keywords: secure, security, attack
• Abstract: Post-quantum cryptography has gained attention due to the need for secure cryptographic systems in the face of quantum computing. Code-based and lattice-based cryptography are two prominent approaches, both heavily studied within the NIST standardization project. Code-based cryptography -- most prominently exemplified by the McEliece cryptosystem -- is based on the hardness of decoding random linear error-correcting codes. Despite the McEliece cryptosystem having been unbroken for several decades, it suffers from large key sizes, which has led to exploring variants using metrics than the Hamming metric, such as the Lee metric. This alternative metric may allow for smaller key sizes, but requires further analysis for potential vulnerabilities to lattice- based attack techniques. In this paper, we consider a generic Lee metric based McEliece type cryptosystem and evaluate its security against lattice-based attacks.

Title: AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessment

• Authors: Nuo Chen, Jiqun Liu, Xiaoyu Dong, Qijiong Liu, Tetsuya Sakai, Xiao-Ming Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16022
• Pdf URL: https://arxiv.org/pdf/2409.16022
• Copy Paste: [[2409.16022]] AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessment(https://arxiv.org/abs/2409.16022)
• Keywords: large language model
• Abstract: Cognitive biases are systematic deviations in thinking that lead to irrational judgments and problematic decision-making, extensively studied across various fields. Recently, large language models (LLMs) have shown advanced understanding capabilities but may inherit human biases from their training data. While social biases in LLMs have been well-studied, cognitive biases have received less attention, with existing research focusing on specific scenarios. The broader impact of cognitive biases on LLMs in various decision- making contexts remains underexplored. We investigated whether LLMs are influenced by the threshold priming effect in relevance judgments, a core task and widely-discussed research topic in the Information Retrieval (IR) coummunity. The priming effect occurs when exposure to certain stimuli unconsciously affects subsequent behavior and decisions. Our experiment employed 10 topics from the TREC 2019 Deep Learning passage track collection, and tested AI judgments under different document relevance scores, batch lengths, and LLM models, including GPT-3.5, GPT-4, LLaMa2-13B and LLaMa2-70B. Results showed that LLMs tend to give lower scores to later documents if earlier ones have high relevance, and vice versa, regardless of the combination and model used. Our finding demonstrates that LLM%u2019s judgments, similar to human judgments, are also influenced by threshold priming biases, and suggests that researchers and system engineers should take into account potential human-like cognitive biases in designing, evaluating, and auditing LLMs in IR tasks and beyond.

Title: Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts

• Authors: Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, Ming Jin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16040
• Pdf URL: https://arxiv.org/pdf/2409.16040
• Copy Paste: [[2409.16040]] Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts(https://arxiv.org/abs/2409.16040)
• Keywords: transformer
• Abstract: Deep learning for time series forecasting has seen significant advancements over the past decades. However, despite the success of large-scale pre-training in language and vision domains, pre-trained time series models remain limited in scale and operate at a high cost, hindering the development of larger capable forecasting models in real-world applications. In response, we introduce Time-MoE, a scalable and unified architecture designed to pre-train larger, more capable forecasting foundation models while reducing inference costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE enhances computational efficiency by activating only a subset of networks for each prediction, reducing computational load while maintaining high model capacity. This allows Time-MoE to scale effectively without a corresponding increase in inference costs. Time-MoE comprises a family of decoder-only transformer models that operate in an auto-regressive manner and support flexible forecasting horizons with varying input context lengths. We pre-trained these models on our newly introduced large-scale data Time-300B, which spans over 9 domains and encompassing over 300 billion time points. For the first time, we scaled a time series foundation model up to 2.4 billion parameters, achieving significantly improved forecasting precision. Our results validate the applicability of scaling laws for training tokens and model size in the context of time series forecasting. Compared to dense models with the same number of activated parameters or equivalent computation budgets, our models consistently outperform them by large margin. These advancements position Time-MoE as a state-of-the-art solution for tackling real-world time series forecasting challenges with superior capability, efficiency, and flexibility.

Title: Adversarial Watermarking for Face Recognition

• Authors: Yuguang Yao, Anil Jain, Sijia Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16056
• Pdf URL: https://arxiv.org/pdf/2409.16056
• Copy Paste: [[2409.16056]] Adversarial Watermarking for Face Recognition(https://arxiv.org/abs/2409.16056)
• Keywords: security, attack, steal, watermark
• Abstract: Watermarking is an essential technique for embedding an identifier (i.e., watermark message) within digital images to assert ownership and monitor unauthorized alterations. In face recognition systems, watermarking plays a pivotal role in ensuring data integrity and security. However, an adversary could potentially interfere with the watermarking process, significantly impairing recognition performance. We explore the interaction between watermarking and adversarial attacks on face recognition models. Our findings reveal that while watermarking or input-level perturbation alone may have a negligible effect on recognition accuracy, the combined effect of watermarking and perturbation can result in an adversarial watermarking attack, significantly degrading recognition performance. Specifically, we introduce a novel threat model, the adversarial watermarking attack, which remains stealthy in the absence of watermarking, allowing images to be correctly recognized initially. However, once watermarking is applied, the attack is activated, causing recognition failures. Our study reveals a previously unrecognized vulnerability: adversarial perturbations can exploit the watermark message to evade face recognition systems. Evaluated on the CASIA-WebFace dataset, our proposed adversarial watermarking attack reduces face matching accuracy by 67.2% with an $\ell_\infty$ norm-measured perturbation strength of ${2}/{255}$ and by 95.9% with a strength of ${4}/{255}$.

Title: Towards Robust Object Detection: Identifying and Removing Backdoors via Module Inconsistency Analysis

• Authors: Xianda Zhang, Siyuan Liang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16057
• Pdf URL: https://arxiv.org/pdf/2409.16057
• Copy Paste: [[2409.16057]] Towards Robust Object Detection: Identifying and Removing Backdoors via Module Inconsistency Analysis(https://arxiv.org/abs/2409.16057)
• Keywords: security, defense, attack, robust
• Abstract: Object detection models, widely used in security-critical applications, are vulnerable to backdoor attacks that cause targeted misclassifications when triggered by specific patterns. Existing backdoor defense techniques, primarily designed for simpler models like image classifiers, often fail to effectively detect and remove backdoors in object detectors. We propose a backdoor defense framework tailored to object detection models, based on the observation that backdoor attacks cause significant inconsistencies between local modules' behaviors, such as the Region Proposal Network (RPN) and classification head. By quantifying and analyzing these inconsistencies, we develop an algorithm to detect backdoors. We find that the inconsistent module is usually the main source of backdoor behavior, leading to a removal method that localizes the affected module, resets its parameters, and fine-tunes the model on a small clean dataset. Extensive experiments with state-of-the-art two-stage object detectors show our method achieves a 90% improvement in backdoor removal rate over fine-tuning baselines, while limiting clean data accuracy loss to less than 4%. To the best of our knowledge, this work presents the first approach that addresses both the detection and removal of backdoors in two-stage object detection models, advancing the field of securing these complex systems against backdoor attacks.

Title: Generative 3D Cardiac Shape Modelling for In-Silico Trials

• Authors: Andrei Gasparovici, Alex Serban
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.16058
• Pdf URL: https://arxiv.org/pdf/2409.16058
• Copy Paste: [[2409.16058]] Generative 3D Cardiac Shape Modelling for In-Silico Trials(https://arxiv.org/abs/2409.16058)
• Keywords: generative
• Abstract: We propose a deep learning method to model and generate synthetic aortic shapes based on representing shapes as the zero-level set of a neural signed distance field, conditioned by a family of trainable embedding vectors with encode the geometric features of each shape. The network is trained on a dataset of aortic root meshes reconstructed from CT images by making the neural field vanish on sampled surface points and enforcing its spatial gradient to have unit norm. Empirical results show that our model can represent aortic shapes with high fidelity. Moreover, by sampling from the learned embedding vectors, we can generate novel shapes that resemble real patient anatomies, which can be used for in-silico trials.

Title: Benchmarking Robustness of Endoscopic Depth Estimation with Synthetically Corrupted Data

• Authors: An Wang, Haochen Yin, Beilei Cui, Mengya Xu, Hongliang Ren
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.16063
• Pdf URL: https://arxiv.org/pdf/2409.16063
• Copy Paste: [[2409.16063]] Benchmarking Robustness of Endoscopic Depth Estimation with Synthetically Corrupted Data(https://arxiv.org/abs/2409.16063)
• Keywords: robust
• Abstract: Accurate depth perception is crucial for patient outcomes in endoscopic surgery, yet it is compromised by image distortions common in surgical settings. To tackle this issue, our study presents a benchmark for assessing the robustness of endoscopic depth estimation models. We have compiled a comprehensive dataset that reflects real-world conditions, incorporating a range of synthetically induced corruptions at varying severity levels. To further this effort, we introduce the Depth Estimation Robustness Score (DERS), a novel metric that combines measures of error, accuracy, and robustness to meet the multifaceted requirements of surgical applications. This metric acts as a foundational element for evaluating performance, establishing a new paradigm for the comparative analysis of depth estimation technologies. Additionally, we set forth a benchmark focused on robustness for the evaluation of depth estimation in endoscopic surgery, with the aim of driving progress in model refinement. A thorough analysis of two monocular depth estimation models using our framework reveals crucial information about their reliability under adverse conditions. Our results emphasize the essential need for algorithms that can tolerate data corruption, thereby advancing discussions on improving model robustness. The impact of this research transcends theoretical frameworks, providing concrete gains in surgical precision and patient safety. This study establishes a benchmark for the robustness of depth estimation and serves as a foundation for developing more resilient surgical support technologies. Code is available at this https URL.

Title: Machine learning approaches for automatic defect detection in photovoltaic systems

• Authors: Swayam Rajat Mohanty, Moin Uddin Maruf, Vaibhav Singh, Zeeshan Ahmad
• Subjects: cs.CV, physics.app-ph
• Abstract URL: https://arxiv.org/abs/2409.16069
• Pdf URL: https://arxiv.org/pdf/2409.16069
• Copy Paste: [[2409.16069]] Machine learning approaches for automatic defect detection in photovoltaic systems(https://arxiv.org/abs/2409.16069)
• Keywords: robust, interpretability, generative
• Abstract: Solar photovoltaic (PV) modules are prone to damage during manufacturing, installation and operation which reduces their power conversion efficiency. This diminishes their positive environmental impact over the lifecycle. Continuous monitoring of PV modules during operation via unmanned aerial vehicles is essential to ensure that defective panels are promptly replaced or repaired to maintain high power conversion efficiencies. Computer vision provides an automatic, non-destructive and cost-effective tool for monitoring defects in large-scale PV plants. We review the current landscape of deep learning-based computer vision techniques used for detecting defects in solar modules. We compare and evaluate the existing approaches at different levels, namely the type of images used, data collection and processing method, deep learning architectures employed, and model interpretability. Most approaches use convolutional neural networks together with data augmentation or generative adversarial network-based techniques. We evaluate the deep learning approaches by performing interpretability analysis on classification tasks. This analysis reveals that the model focuses on the darker regions of the image to perform the classification. We find clear gaps in the existing approaches while also laying out the groundwork for mitigating these challenges when building new models. We conclude with the relevant research gaps that need to be addressed and approaches for progress in this field: integrating geometric deep learning with existing approaches for building more robust and reliable models, leveraging physics-based neural networks that combine domain expertise of physical laws to build more domain-aware deep learning models, and incorporating interpretability as a factor for building models that can be trusted. The review points towards a clear roadmap for making this technology commercially relevant.

Title: Open-World Object Detection with Instance Representation Learning

• Authors: Sunoh Lee, Minsik Jeon, Jihong Min, Junwon Seo
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.16073
• Pdf URL: https://arxiv.org/pdf/2409.16073
• Copy Paste: [[2409.16073]] Open-World Object Detection with Instance Representation Learning(https://arxiv.org/abs/2409.16073)
• Keywords: robust, extraction
• Abstract: While humans naturally identify novel objects and understand their relationships, deep learning-based object detectors struggle to detect and relate objects that are not observed during training. To overcome this issue, Open World Object Detection(OWOD) has been introduced to enable models to detect unknown objects in open-world scenarios. However, OWOD methods fail to capture the fine-grained relationships between detected objects, which are crucial for comprehensive scene understanding and applications such as class discovery and tracking. In this paper, we propose a method to train an object detector that can both detect novel objects and extract semantically rich features in open-world conditions by leveraging the knowledge of Vision Foundation Models(VFM). We first utilize the semantic masks from the Segment Anything Model to supervise the box regression of unknown objects, ensuring accurate localization. By transferring the instance-wise similarities obtained from the VFM features to the detector's instance embeddings, our method then learns a semantically rich feature space of these embeddings. Extensive experiments show that our method learns a robust and generalizable feature space, outperforming other OWOD-based feature extraction methods. Additionally, we demonstrate that the enhanced feature from our model increases the detector's applicability to tasks such as open-world tracking.

Title: Assessing Simplification Levels in Neural Networks: The Impact of Hyperparameter Configurations on Complexity and Sensitivity

• Authors: (Joy)Huixin Guan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16086
• Pdf URL: https://arxiv.org/pdf/2409.16086
• Copy Paste: [[2409.16086]] Assessing Simplification Levels in Neural Networks: The Impact of Hyperparameter Configurations on Complexity and Sensitivity(https://arxiv.org/abs/2409.16086)
• Keywords: robust
• Abstract: This paper presents an experimental study focused on understanding the simplification properties of neural networks under different hyperparameter configurations, specifically investigating the effects on Lempel Ziv complexity and sensitivity. By adjusting key hyperparameters such as activation functions, hidden layers, and learning rate, this study evaluates how these parameters impact the complexity of network outputs and their robustness to input perturbations. The experiments conducted using the MNIST dataset aim to provide insights into the relationships between hyperparameters, complexity, and sensitivity, contributing to a deeper theoretical understanding of these concepts in neural networks.

Title: From Pixels to Words: Leveraging Explainability in Face Recognition through Interactive Natural Language Processing

• Authors: Ivan DeAndres-Tame, Muhammad Faisal, Ruben Tolosana, Rouqaiah Al-Refai, Ruben Vera-Rodriguez, Philipp Terhörst
• Subjects: cs.CV, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16089
• Pdf URL: https://arxiv.org/pdf/2409.16089
• Copy Paste: [[2409.16089]] From Pixels to Words: Leveraging Explainability in Face Recognition through Interactive Natural Language Processing(https://arxiv.org/abs/2409.16089)
• Keywords: fair, interpretability, explainability
• Abstract: Face Recognition (FR) has advanced significantly with the development of deep learning, achieving high accuracy in several applications. However, the lack of interpretability of these systems raises concerns about their accountability, fairness, and reliability. In the present study, we propose an interactive framework to enhance the explainability of FR models by combining model-agnostic Explainable Artificial Intelligence (XAI) and Natural Language Processing (NLP) techniques. The proposed framework is able to accurately answer various questions of the user through an interactive chatbot. In particular, the explanations generated by our proposed method are in the form of natural language text and visual representations, which for example can describe how different facial regions contribute to the similarity measure between two faces. This is achieved through the automatic analysis of the output's saliency heatmaps of the face images and a BERT question-answering model, providing users with an interface that facilitates a comprehensive understanding of the FR decisions. The proposed approach is interactive, allowing the users to ask questions to get more precise information based on the user's background knowledge. More importantly, in contrast to previous studies, our solution does not decrease the face recognition performance. We demonstrate the effectiveness of the method through different experiments, highlighting its potential to make FR systems more interpretable and user-friendly, especially in sensitive applications where decision-making transparency is crucial.

Title: Exploring Hint Generation Approaches in Open-Domain Question Answering

• Authors: Jamshid Mozafari, Abdelrahman Abdallah, Bhawna Piryani, Adam Jatowt
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.16096
• Pdf URL: https://arxiv.org/pdf/2409.16096
• Copy Paste: [[2409.16096]] Exploring Hint Generation Approaches in Open-Domain Question Answering(https://arxiv.org/abs/2409.16096)
• Keywords: generative, large language model
• Abstract: Automatic Question Answering (QA) systems rely on contextual information to provide accurate answers. Commonly, contexts are prepared through either retrieval-based or generation-based methods. The former involves retrieving relevant documents from a corpus like Wikipedia, whereas the latter uses generative models such as Large Language Models (LLMs) to generate the context. In this paper, we introduce a novel context preparation approach called HINTQA, which employs Automatic Hint Generation (HG) techniques. Unlike traditional methods, HINTQA prompts LLMs to produce hints about potential answers for the question rather than generating relevant context. We evaluate our approach across three QA datasets including TriviaQA, NaturalQuestions, and Web Questions, examining how the number and order of hints impact performance. Our findings show that the HINTQA surpasses both retrieval-based and generation-based approaches. We demonstrate that hints enhance the accuracy of answers more than retrieved and generated contexts.

Title: Neuromorphic Drone Detection: an Event-RGB Multimodal Approach

• Authors: Gabriele Magrini, Federico Becattini, Pietro Pala, Alberto Del Bimbo, Antonio Porta
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16099
• Pdf URL: https://arxiv.org/pdf/2409.16099
• Copy Paste: [[2409.16099]] Neuromorphic Drone Detection: an Event-RGB Multimodal Approach(https://arxiv.org/abs/2409.16099)
• Keywords: attack
• Abstract: In recent years, drone detection has quickly become a subject of extreme interest: the potential for fast-moving objects of contained dimensions to be used for malicious intents or even terrorist attacks has posed attention to the necessity for precise and resilient systems for detecting and identifying such elements. While extensive literature and works exist on object detection based on RGB data, it is also critical to recognize the limits of such modality when applied to UAVs detection. Detecting drones indeed poses several challenges such as fast-moving objects and scenes with a high dynamic range or, even worse, scarce illumination levels. Neuromorphic cameras, on the other hand, can retain precise and rich spatio-temporal information in situations that are challenging for RGB cameras. They are resilient to both high-speed moving objects and scarce illumination settings, while prone to suffer a rapid loss of information when the objects in the scene are static. In this context, we present a novel model for integrating both domains together, leveraging multimodal data to take advantage of the best of both worlds. To this end, we also release NeRDD (Neuromorphic-RGB Drone Detection), a novel spatio-temporally synchronized Event-RGB Drone detection dataset of more than 3.5 hours of multimodal annotated recordings.

Title: Ciphertext Malleability in Lattice-Based KEMs as a Countermeasure to Side Channel Analysis

• Authors: Pierre-Augustin Berthet
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16107
• Pdf URL: https://arxiv.org/pdf/2409.16107
• Copy Paste: [[2409.16107]] Ciphertext Malleability in Lattice-Based KEMs as a Countermeasure to Side Channel Analysis(https://arxiv.org/abs/2409.16107)
• Keywords: security, attack
• Abstract: Due to developments in quantum computing, classical asymmetric cryptography is at risk of being breached. Consequently, new Post-Quantum Cryptography (PQC) primitives using lattices are studied. Another point of scrutiny is the resilience of these new primitives to Side Channel Analysis (SCA), where an attacker can study physical leakages. In this work we discuss a SCA vulnerability due to the ciphertext malleability of some PQC primitives exposed by a work from Ravi et al. We propose a novel countermeasure to this vulnerability exploiting the same ciphertext malleability and discuss its practical application to several PQC primitives. We also extend the seminal work of Ravi et al. by detailling their attack on the different security levels of a post-quantum Key Encapsulation Mechanism (KEM), namely FrodoKEM.

Title: Self-attention as an attractor network: transient memories without backpropagation

• Authors: Francesco D'Amico, Matteo Negri
• Subjects: cs.LG, cond-mat.dis-nn
• Abstract URL: https://arxiv.org/abs/2409.16112
• Pdf URL: https://arxiv.org/pdf/2409.16112
• Copy Paste: [[2409.16112]] Self-attention as an attractor network: transient memories without backpropagation(https://arxiv.org/abs/2409.16112)
• Keywords: transformer
• Abstract: Transformers are one of the most successful architectures of modern neural networks. At their core there is the so-called attention mechanism, which recently interested the physics community as it can be written as the derivative of an energy function in certain cases: while it is possible to write the cross-attention layer as a modern Hopfield network, the same is not possible for the self-attention, which is used in the GPT architectures and other autoregressive models. In this work we show that it is possible to obtain the self-attention layer as the derivative of local energy terms, which resemble a pseudo-likelihood. We leverage the analogy with pseudo-likelihood to design a recurrent model that can be trained without backpropagation: the dynamics shows transient states that are strongly correlated with both train and test examples. Overall we present a novel framework to interpret self-attention as an attractor network, potentially paving the way for new theoretical approaches inspired from physics to understand transformers.

Title: TabEBM: A Tabular Data Augmentation Method with Distinct Class-Specific Energy-Based Models

• Authors: Andrei Margeloiu, Xiangjian Jiang, Nikola Simidjievski, Mateja Jamnik
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16118
• Pdf URL: https://arxiv.org/pdf/2409.16118
• Copy Paste: [[2409.16118]] TabEBM: A Tabular Data Augmentation Method with Distinct Class-Specific Energy-Based Models(https://arxiv.org/abs/2409.16118)
• Keywords: robust, generative
• Abstract: Data collection is often difficult in critical fields such as medicine, physics, and chemistry. As a result, classification methods usually perform poorly with these small datasets, leading to weak predictive performance. Increasing the training set with additional synthetic data, similar to data augmentation in images, is commonly believed to improve downstream classification performance. However, current tabular generative methods that learn either the joint distribution $ p(\mathbf{x}, y) $ or the class-conditional distribution $ p(\mathbf{x} \mid y) $ often overfit on small datasets, resulting in poor-quality synthetic data, usually worsening classification performance compared to using real data alone. To solve these challenges, we introduce TabEBM, a novel class-conditional generative method using Energy-Based Models (EBMs). Unlike existing methods that use a shared model to approximate all class-conditional densities, our key innovation is to create distinct EBM generative models for each class, each modelling its class-specific data distribution individually. This approach creates robust energy landscapes, even in ambiguous class distributions. Our experiments show that TabEBM generates synthetic data with higher quality and better statistical fidelity than existing methods. When used for data augmentation, our synthetic data consistently improves the classification performance across diverse datasets of various sizes, especially small ones.

Title: VisioPhysioENet: Multimodal Engagement Detection using Visual and Physiological Signals

• Authors: Alakhsimar Singh, Nischay Verma, Kanav Goyal, Amritpal Singh, Puneet Kumar, Xiaobai Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16126
• Pdf URL: https://arxiv.org/pdf/2409.16126
• Copy Paste: [[2409.16126]] VisioPhysioENet: Multimodal Engagement Detection using Visual and Physiological Signals(https://arxiv.org/abs/2409.16126)
• Keywords: extraction
• Abstract: This paper presents VisioPhysioENet, a novel multimodal system that leverages visual cues and physiological signals to detect learner engagement. It employs a two-level approach for visual feature extraction using the Dlib library for facial landmark extraction and the OpenCV library for further estimations. This is complemented by extracting physiological signals using the plane-orthogonal-to-skin method to assess cardiovascular activity. These features are integrated using advanced machine learning classifiers, enhancing the detection of various engagement levels. We rigorously evaluate VisioPhysioENet on the DAiSEE dataset, where it achieves an accuracy of 63.09%, demonstrating a superior ability to discern various levels of engagement compared to existing methodologies. The proposed system's code can be accessed at this https URL.

Title: Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment

• Authors: Yuxiao Chen, Kai Li, Wentao Bao, Deep Patel, Yu Kong, Martin Renqiang Min, Dimitris N. Metaxas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16145
• Pdf URL: https://arxiv.org/pdf/2409.16145
• Copy Paste: [[2409.16145]] Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment(https://arxiv.org/abs/2409.16145)
• Keywords: large language model
• Abstract: Learning to localize temporal boundaries of procedure steps in instructional videos is challenging due to the limited availability of annotated large-scale training videos. Recent works focus on learning the cross-modal alignment between video segments and ASR-transcripted narration texts through contrastive learning. However, these methods fail to account for the alignment noise, i.e., irrelevant narrations to the instructional task in videos and unreliable timestamps in narrations. To address these challenges, this work proposes a novel training framework. Motivated by the strong capabilities of Large Language Models (LLMs) in procedure understanding and text summarization, we first apply an LLM to filter out task-irrelevant information and summarize task-related procedure steps (LLM-steps) from narrations. To further generate reliable pseudo-matching between the LLM-steps and the video for training, we propose the Multi-Pathway Text-Video Alignment (MPTVA) strategy. The key idea is to measure alignment between LLM-steps and videos via multiple pathways, including: (1) step-narration-video alignment using narration timestamps, (2) direct step-to-video alignment based on their long-term semantic similarity, and (3) direct step-to-video alignment focusing on short-term fine-grained semantic similarity learned from general video domains. The results from different pathways are fused to generate reliable pseudo step-video matching. We conducted extensive experiments across various tasks and problem settings to evaluate our proposed method. Our approach surpasses state-of-the-art methods in three downstream tasks: procedure step grounding, step localization, and narration grounding by 5.9\%, 3.1\%, and 2.8\%.

Title: Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework

• Authors: Lu Chen, Ruqing Zhang, Jiafeng Guo, Yixing Fan, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16146
• Pdf URL: https://arxiv.org/pdf/2409.16146
• Copy Paste: [[2409.16146]] Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework(https://arxiv.org/abs/2409.16146)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) has emerged as a popular solution to mitigate the hallucination issues of large language models. However, existing studies on RAG seldom address the issue of predictive uncertainty, i.e., how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications. In this work, we emphasize the importance of risk control, ensuring that RAG models proactively refuse to answer questions with low confidence. Our research identifies two critical latent factors affecting RAG's confidence in its predictions: the quality of the retrieved results and the manner in which these results are utilized. To guide RAG models in assessing their own confidence based on these two latent factors, we develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers. We also introduce a benchmarking procedure to collect answers with the option to abstain, facilitating a series of experiments. For evaluation, we introduce several risk-related metrics and the experimental results demonstrate the effectiveness of our approach.

Title: ComiCap: A VLMs pipeline for dense captioning of Comic Panels

• Authors: Emanuele Vivoli, Niccolò Biondi, Marco Bertini, Dimosthenis Karatzas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16159
• Pdf URL: https://arxiv.org/pdf/2409.16159
• Copy Paste: [[2409.16159]] ComiCap: A VLMs pipeline for dense captioning of Comic Panels(https://arxiv.org/abs/2409.16159)
• Keywords: fair
• Abstract: The comic domain is rapidly advancing with the development of single- and multi-page analysis and synthesis models. Recent benchmarks and datasets have been introduced to support and assess models' capabilities in tasks such as detection (panels, characters, text), linking (character re-identification and speaker identification), and analysis of comic elements (e.g., dialog transcription). However, to provide a comprehensive understanding of the storyline, a model must not only extract elements but also understand their relationships and generate highly informative captions. In this work, we propose a pipeline that leverages Vision-Language Models (VLMs) to obtain dense, grounded captions. To construct our pipeline, we introduce an attribute-retaining metric that assesses whether all important attributes are identified in the caption. Additionally, we created a densely annotated test set to fairly evaluate open-source VLMs and select the best captioning model according to our metric. Our pipeline generates dense captions with bounding boxes that are quantitatively and qualitatively superior to those produced by specifically trained models, without requiring any additional training. Using this pipeline, we annotated over 2 million panels across 13,000 books, which will be available on the project page this https URL.

Title: MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling

• Authors: Yifang Men, Yuan Yao, Miaomiao Cui, Liefeng Bo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16160
• Pdf URL: https://arxiv.org/pdf/2409.16160
• Copy Paste: [[2409.16160]] MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling(https://arxiv.org/abs/2409.16160)
• Keywords: robust, diffusion
• Abstract: Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

Title: Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

• Authors: Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, Fei Wu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16167
• Pdf URL: https://arxiv.org/pdf/2409.16167
• Copy Paste: [[2409.16167]] Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering(https://arxiv.org/abs/2409.16167)
• Keywords: large language model
• Abstract: Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to enhance LLM capabilities. However, existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading to parameter interference and performance degradation. In this paper, we investigate the feasibility of disassembling and reassembling multiple LoRAs at a finer granularity, analogous to assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. These MSUs demonstrate permutation invariance and concatenation-summation equivalence properties, enabling flexible combinations to create new LoRAs. Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into $k$ clusters. The centroid of each cluster serves as a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of $k$. Additionally, we apply a dual reweighting strategy to optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.

Title: Fine Tuning Text-to-Image Diffusion Models for Correcting Anomalous Images

• Authors: Hyunwoo Yoo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16174
• Pdf URL: https://arxiv.org/pdf/2409.16174
• Copy Paste: [[2409.16174]] Fine Tuning Text-to-Image Diffusion Models for Correcting Anomalous Images(https://arxiv.org/abs/2409.16174)
• Keywords: diffusion
• Abstract: Since the advent of GANs and VAEs, image generation models have continuously evolved, opening up various real-world applications with the introduction of Stable Diffusion and DALL-E models. These text-to-image models can generate high-quality images for fields such as art, design, and advertising. However, they often produce aberrant images for certain prompts. This study proposes a method to mitigate such issues by fine-tuning the Stable Diffusion 3 model using the DreamBooth technique. Experimental results targeting the prompt "lying on the grass/street" demonstrate that the fine-tuned model shows improved performance in visual evaluation and metrics such as Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Frechet Inception Distance (FID). User surveys also indicated a higher preference for the fine-tuned model. This research is expected to make contributions to enhancing the practicality and reliability of text-to-image models.

Title: Cyber Knowledge Completion Using Large Language Models

• Authors: Braden K Webb, Sumit Purohit, Rounak Meyur
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16176
• Pdf URL: https://arxiv.org/pdf/2409.16176
• Copy Paste: [[2409.16176]] Cyber Knowledge Completion Using Large Language Models(https://arxiv.org/abs/2409.16176)
• Keywords: security, attack, large language model
• Abstract: The integration of the Internet of Things (IoT) into Cyber-Physical Systems (CPSs) has expanded their cyber-attack surface, introducing new and sophisticated threats with potential to exploit emerging vulnerabilities. Assessing the risks of CPSs is increasingly difficult due to incomplete and outdated cybersecurity knowledge. This highlights the urgent need for better-informed risk assessments and mitigation strategies. While previous efforts have relied on rule-based natural language processing (NLP) tools to map vulnerabilities, weaknesses, and attack patterns, recent advancements in Large Language Models (LLMs) present a unique opportunity to enhance cyber-attack knowledge completion through improved reasoning, inference, and summarization capabilities. We apply embedding models to encapsulate information on attack patterns and adversarial techniques, generating mappings between them using vector embeddings. Additionally, we propose a Retrieval-Augmented Generation (RAG)-based approach that leverages pre-trained models to create structured mappings between different taxonomies of threat patterns. Further, we use a small hand-labeled dataset to compare the proposed RAG-based approach to a baseline standard binary classification model. Thus, the proposed approach provides a comprehensive framework to address the challenge of cyber-attack knowledge graph completion.

Title: HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

• Authors: Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16191
• Pdf URL: https://arxiv.org/pdf/2409.16191
• Copy Paste: [[2409.16191]] HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models(https://arxiv.org/abs/2409.16191)
• Keywords: large language model
• Abstract: In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks (e.g., long-context understanding), and many benchmarks have been proposed. However, we observe that long text generation capabilities are not well investigated. Therefore, we introduce the Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive, in-the-wild, and open-ended benchmark to evaluate LLMs' performance in generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation. Besides, we propose Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation method that significantly reduces the time and effort required for human evaluation while maintaining a high correlation with human evaluation. We have conducted extensive experiments across around 30 mainstream LLMs and observed that the current LLMs lack long text generation capabilities. Specifically, first, regardless of whether the instructions include explicit or implicit length constraints, we observe that most LLMs cannot generate text that is longer than 4000 words. Second, we observe that while some LLMs can generate longer text, many issues exist (e.g., severe repetition and quality degradation). Third, to demonstrate the effectiveness of HelloEval, we compare HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge methods, which show that HelloEval has the highest correlation with human evaluation. We release our code in this https URL.

Title: Segmentation Strategies in Deep Learning for Prostate Cancer Diagnosis: A Comparative Study of Mamba, SAM, and YOLO

• Authors: Ali Badiezadeh, Amin Malekmohammadi, Seyed Mostafa Mirhassani, Parisa Gifani, Majid Vafaeezadeh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16205
• Pdf URL: https://arxiv.org/pdf/2409.16205
• Copy Paste: [[2409.16205]] Segmentation Strategies in Deep Learning for Prostate Cancer Diagnosis: A Comparative Study of Mamba, SAM, and YOLO(https://arxiv.org/abs/2409.16205)
• Keywords: robust, segmentation
• Abstract: Accurate segmentation of prostate cancer histopathology images is crucial for diagnosis and treatment planning. This study presents a comparative analysis of three deep learning-based methods, Mamba, SAM, and YOLO, for segmenting prostate cancer histopathology images. We evaluated the performance of these models on two comprehensive datasets, Gleason 2019 and SICAPv2, using Dice score, precision, and recall metrics. Our results show that the High-order Vision Mamba UNet (H-vmunet) model outperforms the other two models, achieving the highest scores across all metrics on both datasets. The H-vmunet model's advanced architecture, which integrates high-order visual state spaces and 2D-selective-scan operations, enables efficient and sensitive lesion detection across different scales. Our study demonstrates the potential of the H-vmunet model for clinical applications and highlights the importance of robust validation and comparison of deep learning-based methods for medical image analysis. The findings of this study contribute to the development of accurate and reliable computer-aided diagnosis systems for prostate cancer. The code is available at this http URL.

Title: LLMCount: Enhancing Stationary mmWave Detection with Multimodal-LLM

• Authors: Boyan Li, Shengyi Ding, Deen Ma, Yixuan Wu, Hongjie Liao, Kaiyuan Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16209
• Pdf URL: https://arxiv.org/pdf/2409.16209
• Copy Paste: [[2409.16209]] LLMCount: Enhancing Stationary mmWave Detection with Multimodal-LLM(https://arxiv.org/abs/2409.16209)
• Keywords: privacy
• Abstract: Millimeter wave sensing provides people with the capability of sensing the surrounding crowds in a non-invasive and privacy-preserving manner, which holds huge application potential. However, detecting stationary crowds remains challenging due to several factors such as minimal movements (like breathing or casual fidgets), which can be easily treated as noise clusters during data collection and consequently filtered in the following processing procedures. Additionally, the uneven distribution of signal power due to signal power attenuation and interferences resulting from external reflectors or absorbers further complicates accurate detection. To address these challenges and enable stationary crowd detection across various application scenarios requiring specialized domain adaption, we introduce LLMCount, the first system to harness the capabilities of large-language models (LLMs) to enhance crowd detection performance. By exploiting the decision-making capability of LLM, we can successfully compensate the signal power to acquire a uniform distribution and thereby achieve a detection with higher accuracy. To assess the system's performance, comprehensive evaluations are conducted under diversified scenarios like hall, meeting room, and cinema. The evaluation results show that our proposed approach reaches high detection accuracy with lower overall latency compared with previous methods.

Title: MaskBit: Embedding-free Image Generation via Bit Tokens

• Authors: Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, Liang-Chieh Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16211
• Pdf URL: https://arxiv.org/pdf/2409.16211
• Copy Paste: [[2409.16211]] MaskBit: Embedding-free Image Generation via Bit Tokens(https://arxiv.org/abs/2409.16211)
• Keywords: diffusion, transformer
• Abstract: Masked transformer models for class-conditional image generation have become a compelling alternative to diffusion models. Typically comprising two stages - an initial VQGAN model for transitioning between latent space and image space, and a subsequent Transformer model for image generation within latent space - these frameworks offer promising avenues for image synthesis. In this study, we present two primary contributions: Firstly, an empirical and systematic examination of VQGANs, leading to a modernized VQGAN. Secondly, a novel embedding-free generation network operating directly on bit tokens - a binary quantized representation of tokens with rich semantics. The first contribution furnishes a transparent, reproducible, and high-performing VQGAN model, enhancing accessibility and matching the performance of current state-of-the-art methods while revealing previously undisclosed details. The second contribution demonstrates that embedding-free image generation using bit tokens achieves a new state-of-the-art FID of 1.52 on the ImageNet 256x256 benchmark, with a compact generator model of mere 305M parameters.

Title: Deep Learning for Precision Agriculture: Post-Spraying Evaluation and Deposition Estimation

• Authors: Harry Rogers, Tahmina Zebin, Grzegorz Cielniak, Beatriz De La Iglesia, Ben Magri
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16213
• Pdf URL: https://arxiv.org/pdf/2409.16213
• Copy Paste: [[2409.16213]] Deep Learning for Precision Agriculture: Post-Spraying Evaluation and Deposition Estimation(https://arxiv.org/abs/2409.16213)
• Keywords: interpretability
• Abstract: Precision spraying evaluation requires automation primarily in post-spraying imagery. In this paper we propose an eXplainable Artificial Intelligence (XAI) computer vision pipeline to evaluate a precision spraying system post-spraying without the need for traditional agricultural methods. The developed system can semantically segment potential targets such as lettuce, chickweed, and meadowgrass and correctly identify if targets have been sprayed. Furthermore, this pipeline evaluates using a domain-specific Weakly Supervised Deposition Estimation task, allowing for class-specific quantification of spray deposit weights in {\mu}L. Estimation of coverage rates of spray deposition in a class-wise manner allows for further understanding of effectiveness of precision spraying systems. Our study evaluates different Class Activation Mapping techniques, namely AblationCAM and ScoreCAM, to determine which is more effective and interpretable for these tasks. In the pipeline, inference-only feature fusion is used to allow for further interpretability and to enable the automation of precision spraying evaluation post-spray. Our findings indicate that a Fully Convolutional Network with an EfficientNet-B0 backbone and inference-only feature fusion achieves an average absolute difference in deposition values of 156.8 {\mu}L across three classes in our test set. The dataset curated in this paper is publicly available at this https URL

Title: Fine-Tuning is Fine, if Calibrated

• Authors: Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Vardaan Pahuja, Tanya Berger-Wolf, Song Gao, Charles Stewart, Yu Su, Wei-Lun Chao
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16223
• Pdf URL: https://arxiv.org/pdf/2409.16223
• Copy Paste: [[2409.16223]] Fine-Tuning is Fine, if Calibrated(https://arxiv.org/abs/2409.16223)
• Keywords: robust
• Abstract: Fine-tuning is arguably the most straightforward way to tailor a pre-trained model (e.g., a foundation model) to downstream applications, but it also comes with the risk of losing valuable knowledge the model had learned in pre-training. For example, fine-tuning a pre-trained classifier capable of recognizing a large number of classes to master a subset of classes at hand is shown to drastically degrade the model's accuracy in the other classes it had previously learned. As such, it is hard to further use the fine-tuned model when it encounters classes beyond the fine-tuning data. In this paper, we systematically dissect the issue, aiming to answer the fundamental question, ''What has been damaged in the fine-tuned model?'' To our surprise, we find that the fine-tuned model neither forgets the relationship among the other classes nor degrades the features to recognize these classes. Instead, the fine-tuned model often produces more discriminative features for these other classes, even if they were missing during fine-tuning! {What really hurts the accuracy is the discrepant logit scales between the fine-tuning classes and the other classes}, implying that a simple post-processing calibration would bring back the pre-trained model's capability and at the same time unveil the feature improvement over all classes. We conduct an extensive empirical study to demonstrate the robustness of our findings and provide preliminary explanations underlying them, suggesting new directions for future theoretical analysis. Our code is available at this https URL.

Title: Low-degree Security of the Planted Random Subgraph Problem

• Authors: Andrej Bogdanov, Chris Jones, Alon Rosen, Ilias Zadik
• Subjects: cs.CR, cs.DS, math.ST
• Abstract URL: https://arxiv.org/abs/2409.16227
• Pdf URL: https://arxiv.org/pdf/2409.16227
• Copy Paste: [[2409.16227]] Low-degree Security of the Planted Random Subgraph Problem(https://arxiv.org/abs/2409.16227)
• Keywords: secure, security
• Abstract: The planted random subgraph detection conjecture of Abram et al. (TCC 2023) asserts the pseudorandomness of a pair of graphs $(H, G)$, where $G$ is an Erdos-Renyi random graph on $n$ vertices, and $H$ is a random induced subgraph of $G$ on $k$ vertices. Assuming the hardness of distinguishing these two distributions (with two leaked vertices), Abram et al. construct communication-efficient, computationally secure (1) 2-party private simultaneous messages (PSM) and (2) secret sharing for forbidden graph structures. We prove the low-degree hardness of detecting planted random subgraphs all the way up to $k\leq n^{1 - \Omega(1)}$. This improves over Abram et al.'s analysis for $k \leq n^{1/2 - \Omega(1)}$. The hardness extends to $r$-uniform hypergraphs for constant $r$. Our analysis is tight in the distinguisher's degree, its advantage, and in the number of leaked vertices. Extending the constructions of Abram et al, we apply the conjecture towards (1) communication-optimal multiparty PSM protocols for random functions and (2) bit secret sharing with share size $(1 + \epsilon)\log n$ for any $\epsilon > 0$ in which arbitrary minimal coalitions of up to $r$ parties can reconstruct and secrecy holds against all unqualified subsets of up to $\ell = o(\epsilon \log n)^{1/(r-1)}$ parties.

Title: Predicting Deterioration in Mild Cognitive Impairment with Survival Transformers, Extreme Gradient Boosting and Cox Proportional Hazard Modelling

• Authors: Henry Musto, Daniel Stamate, Doina Logofatu, Daniel Stahl
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.16231
• Pdf URL: https://arxiv.org/pdf/2409.16231
• Copy Paste: [[2409.16231]] Predicting Deterioration in Mild Cognitive Impairment with Survival Transformers, Extreme Gradient Boosting and Cox Proportional Hazard Modelling(https://arxiv.org/abs/2409.16231)
• Keywords: transformer
• Abstract: The paper proposes a novel approach of survival transformers and extreme gradient boosting models in predicting cognitive deterioration in individuals with mild cognitive impairment (MCI) using metabolomics data in the ADNI cohort. By leveraging advanced machine learning and transformer-based techniques applied in survival analysis, the proposed approach highlights the potential of these techniques for more accurate early detection and intervention in Alzheimer's dementia disease. This research also underscores the importance of non-invasive biomarkers and innovative modelling tools in enhancing the accuracy of dementia risk assessments, offering new avenues for clinical practice and patient care. A comprehensive Monte Carlo simulation procedure consisting of 100 repetitions of a nested cross-validation in which models were trained and evaluated, indicates that the survival machine learning models based on Transformer and XGBoost achieved the highest mean C-index performances, namely 0.85 and 0.8, respectively, and that they are superior to the conventional survival analysis Cox Proportional Hazards model which achieved a mean C-Index of 0.77. Moreover, based on the standard deviations of the C-Index performances obtained in the Monte Carlo simulation, we established that both survival machine learning models above are more stable than the conventional statistical model.

Title: Label-Augmented Dataset Distillation

• Authors: Seoungyoon Kang, Youngsun Lim, Hyunjung Shim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16239
• Pdf URL: https://arxiv.org/pdf/2409.16239
• Copy Paste: [[2409.16239]] Label-Augmented Dataset Distillation(https://arxiv.org/abs/2409.16239)
• Keywords: robust
• Abstract: Traditional dataset distillation primarily focuses on image representation while often overlooking the important role of labels. In this study, we introduce Label-Augmented Dataset Distillation (LADD), a new dataset distillation framework enhancing dataset distillation with label augmentations. LADD sub-samples each synthetic image, generating additional dense labels to capture rich semantics. These dense labels require only a 2.5% increase in storage (ImageNet subsets) with significant performance benefits, providing strong learning signals. Our label generation strategy can complement existing dataset distillation methods for significantly enhancing their training efficiency and performance. Experimental results demonstrate that LADD outperforms existing methods in terms of computational overhead and accuracy. With three high-performance dataset distillation algorithms, LADD achieves remarkable gains by an average of 14.9% in accuracy. Furthermore, the effectiveness of our method is proven across various datasets, distillation hyperparameters, and algorithms. Finally, our method improves the cross-architecture robustness of the distilled dataset, which is important in the application scenario.

Title: Fields of The World: A Machine Learning Benchmark Dataset For Global Agricultural Field Boundary Segmentation

• Authors: Hannah Kerner, Snehal Chaudhari, Aninda Ghosh, Caleb Robinson, Adeel Ahmad, Eddie Choi, Nathan Jacobs, Chris Holmes, Matthias Mohr, Rahul Dodhia, Juan M. Lavista Ferres, Jennifer Marcus
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16252
• Pdf URL: https://arxiv.org/pdf/2409.16252
• Copy Paste: [[2409.16252]] Fields of The World: A Machine Learning Benchmark Dataset For Global Agricultural Field Boundary Segmentation(https://arxiv.org/abs/2409.16252)
• Keywords: segmentation
• Abstract: Crop field boundaries are foundational datasets for agricultural monitoring and assessments but are expensive to collect manually. Machine learning (ML) methods for automatically extracting field boundaries from remotely sensed images could help realize the demand for these datasets at a global scale. However, current ML methods for field instance segmentation lack sufficient geographic coverage, accuracy, and generalization capabilities. Further, research on improving ML methods is restricted by the lack of labeled datasets representing the diversity of global agricultural fields. We present Fields of The World (FTW) -- a novel ML benchmark dataset for agricultural field instance segmentation spanning 24 countries on four continents (Europe, Africa, Asia, and South America). FTW is an order of magnitude larger than previous datasets with 70,462 samples, each containing instance and semantic segmentation masks paired with multi-date, multi-spectral Sentinel-2 satellite images. We provide results from baseline models for the new FTW benchmark, show that models trained on FTW have better zero-shot and fine-tuning performance in held-out countries than models that aren't pre-trained with diverse datasets, and show positive qualitative zero-shot results of FTW models in a real-world scenario -- running on Sentinel-2 scenes over Ethiopia.

Title: Semantic Refocused Tuning for Open-Vocabulary Panoptic Segmentation

• Authors: Yong Xien Chng, Xuchong Qiu, Yizeng Han, Kai Ding, Wan Ding, Gao Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16278
• Pdf URL: https://arxiv.org/pdf/2409.16278
• Copy Paste: [[2409.16278]] Semantic Refocused Tuning for Open-Vocabulary Panoptic Segmentation(https://arxiv.org/abs/2409.16278)
• Keywords: extraction, segmentation
• Abstract: Open-vocabulary panoptic segmentation is an emerging task aiming to accurately segment the image into semantically meaningful masks based on a set of texts. Despite existing efforts, it remains challenging to develop a high-performing method that generalizes effectively across new domains and requires minimal training resources. Our in-depth analysis of current methods reveals a crucial insight: mask classification is the main performance bottleneck for open-vocab. panoptic segmentation. Based on this, we propose Semantic Refocused Tuning (SMART), a novel framework that greatly enhances open-vocab. panoptic segmentation by improving mask classification through two key innovations. First, SMART adopts a multimodal Semantic-guided Mask Attention mechanism that injects task-awareness into the regional information extraction process. This enables the model to capture task-specific and contextually relevant information for more effective mask classification. Second, it incorporates Query Projection Tuning, which strategically fine-tunes the query projection layers within the Vision Language Model (VLM) used for mask classification. This adjustment allows the model to adapt the image focus of mask tokens to new distributions with minimal training resources, while preserving the VLM's pre-trained knowledge. Extensive ablation studies confirm the superiority of our approach. Notably, SMART sets new state-of-the-art results, demonstrating improvements of up to +1.3 PQ and +5.4 mIoU across representative benchmarks, while reducing training costs by nearly 10x compared to the previous best method. Our code and data will be released.

Title: MonoFormer: One Transformer for Both Diffusion and Autoregression

• Authors: Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, Jingdong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16280
• Pdf URL: https://arxiv.org/pdf/2409.16280
• Copy Paste: [[2409.16280]] MonoFormer: One Transformer for Both Diffusion and Autoregression(https://arxiv.org/abs/2409.16280)
• Keywords: diffusion, transformer
• Abstract: Most existing multimodality methods use separate backbones for autoregression-based discrete text generation and diffusion-based continuous visual generation, or the same backbone by discretizing the visual data to use autoregression for both text and visual generation. In this paper, we propose to study a simple idea: share one transformer for both autoregression and diffusion. The feasibility comes from two main aspects: (i) Transformer is successfully applied to diffusion for visual generation, and (ii) transformer training for autoregression and diffusion is very similar, and the difference merely lies in that diffusion uses bidirectional attention mask and autoregression uses causal attention mask. Experimental results show that our approach achieves comparable image generation performance to current state-of-the-art methods as well as maintains the text generation capability. The project is publicly available at this https URL.

Title: Self-Supervised Any-Point Tracking by Contrastive Random Walks

• Authors: Ayush Shrivastava, Andrew Owens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.16288
• Pdf URL: https://arxiv.org/pdf/2409.16288
• Copy Paste: [[2409.16288]] Self-Supervised Any-Point Tracking by Contrastive Random Walks(https://arxiv.org/abs/2409.16288)
• Keywords: transformer
• Abstract: We present a simple, self-supervised approach to the Tracking Any Point (TAP) problem. We train a global matching transformer to find cycle consistent tracks through video via contrastive random walks, using the transformer's attention-based global matching to define the transition matrices for a random walk on a space-time graph. The ability to perform "all pairs" comparisons between points allows the model to obtain high spatial precision and to obtain a strong contrastive learning signal, while avoiding many of the complexities of recent approaches (such as coarse-to-fine matching). To do this, we propose a number of design decisions that allow global matching architectures to be trained through self-supervision using cycle consistency. For example, we identify that transformer-based methods are sensitive to shortcut solutions, and propose a data augmentation scheme to address them. Our method achieves strong performance on the TapVid benchmarks, outperforming previous self-supervised tracking methods, such as DIFT, and is competitive with several supervised methods.