2024-11-11

Title: The Impact of Quantum-Safe Cryptography (QSC) on Website Response

• Authors: Ananya Tadepalli
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.05024
• Pdf URL: https://arxiv.org/pdf/2411.05024
• Copy Paste: [[2411.05024]] The Impact of Quantum-Safe Cryptography (QSC) on Website Response(https://arxiv.org/abs/2411.05024)
• Keywords: secure, security
• Abstract: Modern web traffic relies on 2048-bit RSA encryption to secure our data in transit. Rapid advances in Quantum Computing pose a grave challenge by allowing hackers to break this encryption in hours. In August of 2024, the National Institute of Standards and Technology published Quantum-Safe Cryptography (QSC) standards, including CRYSTALS-Kyber for general encryption and CRYSTALS-Dilithium, FALCON, and SPHINCS+ for digital signatures. Despite this proactive approach, the slow adoption of encryption protocols remains a concern, leaving a significant portion of data vulnerable to interception. In this context, this study aims to evaluate the impact of NIST's Quantum-Resistant Cryptographic Algorithms on website response times, particularly focusing on SSL handshake time and total download time under varying network conditions. By assessing the performance of these algorithms, this research seeks to provide empirical evidence and a reusable framework for validating the efficacy of QSC in real-world scenarios. It was found that the QSC algorithms outperformed the classical algorithm under normal and congested network conditions. There was also found to be an improvement in the total download time for larger file sizes, and a better performance by QSC under higher latency and packet loss conditions. Therefore, this study recommends that websites switch to QSC when the standards are ratified. These insights are crucial for accelerating the adoption of QSC and ensuring the security of data in the face of quantum computing threats.

Title: LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions

• Authors: Zhehui Liao, Maria Antoniak, Inyoung Cheong, Evie Yu-Yen Cheng, Ai-Heng Lee, Kyle Lo, Joseph Chee Chang, Amy X. Zhang
• Subjects: cs.CL, cs.AI, cs.CY, cs.DL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.05025
• Pdf URL: https://arxiv.org/pdf/2411.05025
• Copy Paste: [[2411.05025]] LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions(https://arxiv.org/abs/2411.05025)
• Keywords: large language model
• Abstract: The rise of large language models (LLMs) has led many researchers to consider their usage for scientific work. Some have found benefits using LLMs to augment or automate aspects of their research pipeline, while others have urged caution due to risks and ethical concerns. Yet little work has sought to quantify and characterize how researchers use LLMs and why. We present the first large-scale survey of 816 verified research article authors to understand how the research community leverages and perceives LLMs as research tools. We examine participants' self-reported LLM usage, finding that 81% of researchers have already incorporated LLMs into different aspects of their research workflow. We also find that traditionally disadvantaged groups in academia (non-White, junior, and non-native English speaking researchers) report higher LLM usage and perceived benefits, suggesting potential for improved research equity. However, women, non-binary, and senior researchers have greater ethical concerns, potentially hindering adoption.

Title: Deep Learning and Machine Learning -- Natural Language Processing: From Theory to Application

• Authors: Keyu Chen, Cheng Fei, Ziqian Bi, Junyu Liu, Benji Peng, Sen Zhang, Xuanhe Pan, Jiawei Xu, Jinlang Wang, Caitlyn Heqi Yin, Yichao Zhang, Pohsun Feng, Yizhu Wen, Tianyang Wang, Ming Li, Jintao Ren, Qian Niu, Silin Chen, Weiche Hsieh, Lawrence K.Q. Yan, Chia Xin Liang, Han Xu, Hong-Ming Tseng, Xinyuan Song, Ming Liu
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.05026
• Pdf URL: https://arxiv.org/pdf/2411.05026
• Copy Paste: [[2411.05026]] Deep Learning and Machine Learning -- Natural Language Processing: From Theory to Application(https://arxiv.org/abs/2411.05026)
• Keywords: robust, transformer, large language model
• Abstract: With a focus on natural language processing (NLP) and the role of large language models (LLMs), we explore the intersection of machine learning, deep learning, and artificial intelligence. As artificial intelligence continues to revolutionize fields from healthcare to finance, NLP techniques such as tokenization, text classification, and entity recognition are essential for processing and understanding human language. This paper discusses advanced data preprocessing techniques and the use of frameworks like Hugging Face for implementing transformer-based models. Additionally, it highlights challenges such as handling multilingual data, reducing bias, and ensuring model robustness. By addressing key aspects of data processing and model fine-tuning, this work aims to provide insights into deploying effective and ethically sound AI solutions.

Title: Generative Artificial Intelligence Meets Synthetic Aperture Radar: A Survey

• Authors: Zhongling Huang, Xidan Zhang, Zuqian Tang, Feng Xu, Mihai Datcu, Junwei Han
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.05027
• Pdf URL: https://arxiv.org/pdf/2411.05027
• Copy Paste: [[2411.05027]] Generative Artificial Intelligence Meets Synthetic Aperture Radar: A Survey(https://arxiv.org/abs/2411.05027)
• Keywords: generative
• Abstract: SAR images possess unique attributes that present challenges for both human observers and vision AI models to interpret, owing to their electromagnetic characteristics. The interpretation of SAR images encounters various hurdles, with one of the primary obstacles being the data itself, which includes issues related to both the quantity and quality of the data. The challenges can be addressed using generative AI technologies. Generative AI, often known as GenAI, is a very advanced and powerful technology in the field of artificial intelligence that has gained significant attention. The advancement has created possibilities for the creation of texts, photorealistic pictures, videos, and material in various modalities. This paper aims to comprehensively investigate the intersection of GenAI and SAR. First, we illustrate the common data generation-based applications in SAR field and compare them with computer vision tasks, analyzing the similarity, difference, and general challenges of them. Then, an overview of the latest GenAI models is systematically reviewed, including various basic models and their variations targeting the general challenges. Additionally, the corresponding applications in SAR domain are also included. Specifically, we propose to summarize the physical model based simulation approaches for SAR, and analyze the hybrid modeling methods that combine the GenAI and interpretable models. The evaluation methods that have been or could be applied to SAR, are also explored. Finally, the potential challenges and future prospects are discussed. To our best knowledge, this survey is the first exhaustive examination of the interdiscipline of SAR and GenAI, encompassing a wide range of topics, including deep neural networks, physical models, computer vision, and SAR images. The resources of this survey are open-source at \url{this https URL}.

Title: Mitigating Privacy Risks in LLM Embeddings from Embedding Inversion

• Authors: Tiantian Liu, Hongwei Yao, Tong Wu, Zhan Qin, Feng Lin, Kui Ren, Chun Chen
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05034
• Pdf URL: https://arxiv.org/pdf/2411.05034
• Copy Paste: [[2411.05034]] Mitigating Privacy Risks in LLM Embeddings from Embedding Inversion(https://arxiv.org/abs/2411.05034)
• Keywords: security, privacy, protect, defense, attack, transformer, large language model
• Abstract: Embeddings have become a cornerstone in the functionality of large language models (LLMs) due to their ability to transform text data into rich, dense numerical representations that capture semantic and syntactic properties. These embedding vector databases serve as the long-term memory of LLMs, enabling efficient handling of a wide range of natural language processing tasks. However, the surge in popularity of embedding vector databases in LLMs has been accompanied by significant concerns about privacy leakage. Embedding vector databases are particularly vulnerable to embedding inversion attacks, where adversaries can exploit the embeddings to reverse-engineer and extract sensitive information from the original text data. Existing defense mechanisms have shown limitations, often struggling to balance security with the performance of downstream tasks. To address these challenges, we introduce Eguard, a novel defense mechanism designed to mitigate embedding inversion attacks. Eguard employs a transformer-based projection network and text mutual information optimization to safeguard embeddings while preserving the utility of LLMs. Our approach significantly reduces privacy risks, protecting over 95% of tokens from inversion while maintaining high performance across downstream tasks consistent with original embeddings.

Title: From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models

• Authors: Charles Zhang, Benji Peng, Xintian Sun, Qian Niu, Junyu Liu, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Ming Liu, Yichao Zhang, Cheng Fei, Caitlyn Heqi Yin, Lawrence KQ Yan, Tianyang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05036
• Pdf URL: https://arxiv.org/pdf/2411.05036
• Copy Paste: [[2411.05036]] From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models(https://arxiv.org/abs/2411.05036)
• Keywords: robust, interpretability, generative, large language model
• Abstract: Word embeddings and language models have transformed natural language processing (NLP) by facilitating the representation of linguistic elements in continuous vector spaces. This review visits foundational concepts such as the distributional hypothesis and contextual similarity, tracing the evolution from sparse representations like one-hot encoding to dense embeddings including Word2Vec, GloVe, and fastText. We examine both static and contextualized embeddings, underscoring advancements in models such as ELMo, BERT, and GPT and their adaptations for cross-lingual and personalized applications. The discussion extends to sentence and document embeddings, covering aggregation methods and generative topic models, along with the application of embeddings in multimodal domains, including vision, robotics, and cognitive science. Advanced topics such as model compression, interpretability, numerical encoding, and bias mitigation are analyzed, addressing both technical challenges and ethical implications. Additionally, we identify future research directions, emphasizing the need for scalable training techniques, enhanced interpretability, and robust grounding in non-textual modalities. By synthesizing current methodologies and emerging trends, this survey offers researchers and practitioners an in-depth resource to push the boundaries of embedding-based language models.

Title: YouTube Comments Decoded: Leveraging LLMs for Low Resource Language Classification

• Authors: Aniket Deroy, Subhankar Maity
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05039
• Pdf URL: https://arxiv.org/pdf/2411.05039
• Copy Paste: [[2411.05039]] YouTube Comments Decoded: Leveraging LLMs for Low Resource Language Classification(https://arxiv.org/abs/2411.05039)
• Keywords: large language model
• Abstract: Sarcasm detection is a significant challenge in sentiment analysis, particularly due to its nature of conveying opinions where the intended meaning deviates from the literal expression. This challenge is heightened in social media contexts where code-mixing, especially in Dravidian languages, is prevalent. Code-mixing involves the blending of multiple languages within a single utterance, often with non-native scripts, complicating the task for systems trained on monolingual data. This shared task introduces a novel gold standard corpus designed for sarcasm and sentiment detection within code-mixed texts, specifically in Tamil-English and Malayalam-English languages. The primary objective of this task is to identify sarcasm and sentiment polarity within a code-mixed dataset of Tamil-English and Malayalam-English comments and posts collected from social media platforms. Each comment or post is annotated at the message level for sentiment polarity, with particular attention to the challenges posed by class imbalance, reflecting real-world this http URL this work, we experiment with state-of-the-art large language models like GPT-3.5 Turbo via prompting to classify comments into sarcastic or non-sarcastic categories. We obtained a macro-F1 score of 0.61 for Tamil language. We obtained a macro-F1 score of 0.50 for Malayalam language.

Title: Bottom-Up and Top-Down Analysis of Values, Agendas, and Observations in Corpora and LLMs

• Authors: Scott E. Friedman, Noam Benkler, Drisana Mosaphir, Jeffrey Rye, Sonja M. Schmer-Galunder, Micah Goldwater, Matthew McLure, Ruta Wheelock, Jeremy Gottlieb, Robert P. Goldman, Christopher Miller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05040
• Pdf URL: https://arxiv.org/pdf/2411.05040
• Copy Paste: [[2411.05040]] Bottom-Up and Top-Down Analysis of Values, Agendas, and Observations in Corpora and LLMs(https://arxiv.org/abs/2411.05040)
• Keywords: large language model
• Abstract: Large language models (LLMs) generate diverse, situated, persuasive texts from a plurality of potential perspectives, influenced heavily by their prompts and training data. As part of LLM adoption, we seek to characterize - and ideally, manage - the socio-cultural values that they express, for reasons of safety, accuracy, inclusion, and cultural fidelity. We present a validated approach to automatically (1) extracting heterogeneous latent value propositions from texts, (2) assessing resonance and conflict of values with texts, and (3) combining these operations to characterize the pluralistic value alignment of human-sourced and LLM-sourced textual data.

Title: Improving Radiology Report Conciseness and Structure via Local Large Language Models

• Authors: Iryna Hartsock, Cyrillo Araujo, Les Folio, Ghulam Rasool
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05042
• Pdf URL: https://arxiv.org/pdf/2411.05042
• Copy Paste: [[2411.05042]] Improving Radiology Report Conciseness and Structure via Local Large Language Models(https://arxiv.org/abs/2411.05042)
• Keywords: security, privacy, large language model
• Abstract: In this study, we aim to enhance radiology reporting by improving both the conciseness and structured organization of findings (also referred to as templating), specifically by organizing information according to anatomical regions. This structured approach allows physicians to locate relevant information quickly, increasing the report's utility. We utilize Large Language Models (LLMs) such as Mixtral, Mistral, and Llama to generate concise, well-structured reports. Among these, we primarily focus on the Mixtral model due to its superior adherence to specific formatting requirements compared to other models. To maintain data security and privacy, we run these LLMs locally behind our institution's firewall. We leverage the LangChain framework and apply five distinct prompting strategies to enforce a consistent structure in radiology reports, aiming to eliminate extraneous language and achieve a high level of conciseness. We also introduce a novel metric, the Conciseness Percentage (CP) score, to evaluate report brevity. Our dataset comprises 814 radiology reports authored by seven board-certified body radiologists at our cancer center. In evaluating the different prompting methods, we discovered that the most effective approach for generating concise, well-structured reports involves first instructing the LLM to condense the report, followed by a prompt to structure the content according to specific guidelines. We assessed all prompting strategies based on their ability to handle formatting issues, reduce report length, and adhere to formatting instructions. Our findings demonstrate that open-source, locally deployed LLMs can significantly improve radiology report conciseness and structure while conforming to specified formatting standards.

Title: Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale

• Authors: Flavio Di Palo, Prateek Singhi, Bilal Fadlallah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05045
• Pdf URL: https://arxiv.org/pdf/2411.05045
• Copy Paste: [[2411.05045]] Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale(https://arxiv.org/abs/2411.05045)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) face significant challenges at inference time due to their high computational demands. To address this, we present Performance-Guided Knowledge Distillation (PGKD), a cost-effective and high-throughput solution for production text classification applications. PGKD utilizes teacher-student Knowledge Distillation to distill the knowledge of LLMs into smaller, task-specific models. PGKD establishes an active learning routine between the student model and the LLM; the LLM continuously generates new training data leveraging hard-negative mining, student model validation performance, and early-stopping protocols to inform the data generation. By employing a cyclical, performance-aware approach tailored for highly multi-class, sparsely annotated datasets prevalent in industrial text classification, PGKD effectively addresses training challenges and outperforms traditional BERT-base models and other knowledge distillation methods on several multi-class classification datasets. Additionally, cost and latency benchmarking reveals that models fine-tuned with PGKD are up to 130X faster and 25X less expensive than LLMs for inference on the same classification task. While PGKD is showcased for text classification tasks, its versatile framework can be extended to any LLM distillation task, including language generation, making it a powerful tool for optimizing performance across a wide range of AI applications.

Title: Leveraging LLMs to Enable Natural Language Search on Go-to-market Platforms

• Authors: Jesse Yao, Saurav Acharya, Priyaranjan Parida, Srinivas Attipalli, Ali Dasdan
• Subjects: cs.CL, cs.AI, cs.DB, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05048
• Pdf URL: https://arxiv.org/pdf/2411.05048
• Copy Paste: [[2411.05048]] Leveraging LLMs to Enable Natural Language Search on Go-to-market Platforms(https://arxiv.org/abs/2411.05048)
• Keywords: extraction, large language model
• Abstract: Enterprise searches require users to have complex knowledge of queries, configurations, and metadata, rendering it difficult for them to access information as needed. Most go-to-market (GTM) platforms utilize advanced search, an interface that enables users to filter queries by various fields using categories or keywords, which, historically, however, has proven to be exceedingly cumbersome, as users are faced with seemingly hundreds of options, fields, and buttons. Consequently, querying with natural language has long been ideal, a notion further empowered by Large Language Models (LLMs). In this paper, we implement and evaluate a solution for the Zoominfo product for sellers, which prompts the LLM with natural language, producing search fields through entity extraction that are then converted into a search query. The intermediary search fields offer numerous advantages for each query, including the elimination of syntax errors, simpler ground truths, and an intuitive format for the LLM to interpret. We paired this pipeline with many advanced prompt engineering strategies, featuring an intricate system message, few-shot prompting, chain-of-thought (CoT) reasoning, and execution refinement. Furthermore, we manually created the ground truth for 500+ natural language queries, enabling the supervised fine-tuning of Llama-3-8B-Instruct and the introduction of sophisticated numerical metrics. Comprehensive experiments with closed, open source, and fine-tuned LLM models were conducted through exact, Jaccard, cosine, and semantic similarity on individual search entities to demonstrate the efficacy of our approach. Overall, the most accurate closed model had an average accuracy of 97% per query, with only one field performing under 90%, with comparable results observed from the fine-tuned models.

Title: Intellectual Property Protection for Deep Learning Model and Dataset Intelligence

• Authors: Yongqi Jiang, Yansong Gao, Chunyi Zhou, Hongsheng Hu, Anmin Fu, Willy Susilo
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05051
• Pdf URL: https://arxiv.org/pdf/2411.05051
• Copy Paste: [[2411.05051]] Intellectual Property Protection for Deep Learning Model and Dataset Intelligence(https://arxiv.org/abs/2411.05051)
• Keywords: protect, attack, large language model
• Abstract: With the growing applications of Deep Learning (DL), especially recent spectacular achievements of Large Language Models (LLMs) such as ChatGPT and LLaMA, the commercial significance of these remarkable models has soared. However, acquiring well-trained models is costly and resource-intensive. It requires a considerable high-quality dataset, substantial investment in dedicated architecture design, expensive computational resources, and efforts to develop technical expertise. Consequently, safeguarding the Intellectual Property (IP) of well-trained models is attracting increasing attention. In contrast to existing surveys overwhelmingly focusing on model IPP mainly, this survey not only encompasses the protection on model level intelligence but also valuable dataset intelligence. Firstly, according to the requirements for effective IPP design, this work systematically summarizes the general and scheme-specific performance evaluation metrics. Secondly, from proactive IP infringement prevention and reactive IP ownership verification perspectives, it comprehensively investigates and analyzes the existing IPP methods for both dataset and model intelligence. Additionally, from the standpoint of training settings, it delves into the unique challenges that distributed settings pose to IPP compared to centralized settings. Furthermore, this work examines various attacks faced by deep IPP techniques. Finally, we outline prospects for promising future directions that may act as a guide for innovative research.

Title: FMEA Builder: Expert Guided Text Generation for Equipment Maintenance

• Authors: Karol Lynch, Fabio Lorenzi, John Sheehan, Duygu Kabakci-Zorlu, Bradley Eck
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05054
• Pdf URL: https://arxiv.org/pdf/2411.05054
• Copy Paste: [[2411.05054]] FMEA Builder: Expert Guided Text Generation for Equipment Maintenance(https://arxiv.org/abs/2411.05054)
• Keywords: generative, large language model
• Abstract: Foundation models show great promise for generative tasks in many domains. Here we discuss the use of foundation models to generate structured documents related to critical assets. A Failure Mode and Effects Analysis (FMEA) captures the composition of an asset or piece of equipment, the ways it may fail and the consequences thereof. Our system uses large language models to enable fast and expert supervised generation of new FMEA documents. Empirical analysis shows that foundation models can correctly generate over half of an FMEA's key content. Results from polling audiences of reliability professionals show a positive outlook on using generative AI to create these documents for critical assets.

Title: Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Models

• Authors: Pete Janowczyk, Linda Laurier, Ave Giulietta, Arlo Octavia, Meade Cleti
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05056
• Pdf URL: https://arxiv.org/pdf/2411.05056
• Copy Paste: [[2411.05056]] Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Models(https://arxiv.org/abs/2411.05056)
• Keywords: secure, security, privacy, protect, defense, attack, robust
• Abstract: Multi-Modal Language Models (MLLMs) have transformed artificial intelligence by combining visual and text data, making applications like image captioning, visual question answering, and multi-modal content creation possible. This ability to understand and work with complex information has made MLLMs useful in areas such as healthcare, autonomous systems, and digital content. However, integrating multiple types of data also creates security risks. Attackers can manipulate either the visual or text inputs, or both, to make the model produce unintended or even harmful responses. This paper reviews how visual inputs in MLLMs can be exploited by various attack strategies. We break down these attacks into categories: simple visual tweaks and cross-modal manipulations, as well as advanced strategies like VLATTACK, HADES, and Collaborative Multimodal Adversarial Attack (Co-Attack). These attacks can mislead even the most robust models while looking nearly identical to the original visuals, making them hard to detect. We also discuss the broader security risks, including threats to privacy and safety in important applications. To counter these risks, we review current defense methods like the SmoothVLM framework, pixel-wise randomization, and MirrorCheck, looking at their strengths and limitations. We also discuss new methods to make MLLMs more secure, including adaptive defenses, better evaluation tools, and security approaches that protect both visual and text data. By bringing together recent developments and identifying key areas for improvement, this review aims to support the creation of more secure and reliable multi-modal AI systems for real-world use.

Title: A Brief History of Named Entity Recognition

• Authors: Monica Munnangi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05057
• Pdf URL: https://arxiv.org/pdf/2411.05057
• Copy Paste: [[2411.05057]] A Brief History of Named Entity Recognition(https://arxiv.org/abs/2411.05057)
• Keywords: extraction
• Abstract: A large amount of information in today's world is now stored in knowledge bases. Named Entity Recognition (NER) is a process of extracting, disambiguation, and linking an entity from raw text to insightful and structured knowledge bases. More concretely, it is identifying and classifying entities in the text that are crucial for Information Extraction, Semantic Annotation, Question Answering, Ontology Population, and so on. The process of NER has evolved in the last three decades since it first appeared in 1996. In this survey, we study the evolution of techniques employed for NER and compare the results, starting from supervised to the developing unsupervised learning methods.

Title: FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?

• Authors: Eric Wu, Kevin Wu, James Zou
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.05059
• Pdf URL: https://arxiv.org/pdf/2411.05059
• Copy Paste: [[2411.05059]] FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?(https://arxiv.org/abs/2411.05059)
• Keywords: large language model
• Abstract: There is great interest in fine-tuning frontier large language models (LLMs) to inject new information and update existing knowledge. While commercial LLM fine-tuning APIs from providers such as OpenAI and Google promise flexible adaptation for various applications, the efficacy of fine-tuning remains unclear. In this study, we introduce FineTuneBench, an evaluation framework and dataset for understanding how well commercial fine-tuning APIs can successfully learn new and updated knowledge. We analyze five frontier LLMs with commercially available fine-tuning APIs, including GPT-4o and Gemini 1.5 Pro, on their effectiveness in two settings: (1) ingesting novel information, such as recent news events and new people profiles, and (2) updating existing knowledge, such as updated medical guidelines and code frameworks. Our results reveal substantial shortcomings in all the models' abilities to effectively learn new information through fine-tuning, with an average generalization accuracy of 37% across all models. When updating existing knowledge, such as incorporating medical guideline updates, commercial fine-tuning APIs show even more limited capability (average generalization accuracy of 19%). Overall, fine-tuning GPT-4o mini is the most effective for infusing new knowledge and updating knowledge, followed by GPT-3.5 Turbo and GPT-4o. The fine-tuning APIs for Gemini 1.5 Flesh and Gemini 1.5 Pro are unable to learn new knowledge or update existing knowledge. These findings underscore a major shortcoming in using current commercial fine-tuning services to achieve reliable knowledge infusion in common scenarios. We open source the FineTuneBench dataset at this https URL.

Title: Watermarking Language Models through Language Models

• Authors: Xin Zhong, Agnibh Dasgupta, Abdullah Tanvir
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05091
• Pdf URL: https://arxiv.org/pdf/2411.05091
• Copy Paste: [[2411.05091]] Watermarking Language Models through Language Models(https://arxiv.org/abs/2411.05091)
• Keywords: protect, watermark
• Abstract: This paper presents a novel framework for watermarking language models through prompts generated by language models. The proposed approach utilizes a multi-model setup, incorporating a Prompting language model to generate watermarking instructions, a Marking language model to embed watermarks within generated content, and a Detecting language model to verify the presence of these watermarks. Experiments are conducted using ChatGPT and Mistral as the Prompting and Marking language models, with detection accuracy evaluated using a pretrained classifier model. Results demonstrate that the proposed framework achieves high classification accuracy across various configurations, with 95% accuracy for ChatGPT, 88.79% for Mistral. These findings validate the and adaptability of the proposed watermarking strategy across different language model architectures. Hence the proposed framework holds promise for applications in content attribution, copyright protection, and model authentication.

Title: The impact of mobility, beam sweeping and smart jammers on security vulnerabilities of 5G cells

• Authors: Ghazal Asemian, Michel Kulhandjian, Mohammadreza Amini, Burak Kantarci, Claude D'Amours, Melike Erol-Kantarci
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05131
• Pdf URL: https://arxiv.org/pdf/2411.05131
• Copy Paste: [[2411.05131]] The impact of mobility, beam sweeping and smart jammers on security vulnerabilities of 5G cells(https://arxiv.org/abs/2411.05131)
• Keywords: security, attack
• Abstract: The vulnerability of 5G networks to jamming attacks has emerged as a significant concern. This paper contributes in two primary aspects. Firstly, it investigates the effect of a multi-jammer on 5G cell metrics, specifically throughput and goodput. The investigation is conducted within the context of a mobility model for user equipment (UE), with a focus on scenarios involving connected vehicles (CVs) engaged in a mission. Secondly, the vulnerability of synchronization signal block (SSB) components is examined concerning jamming power and beam sweeping. Notably, the study reveals that increasing jamming power beyond 40 dBm in our specific scenario configuration no longer decreases network throughput due to the re-transmission of packets through the hybrid automatic repeat request (HARQ) process. Furthermore, it is observed that under the same jamming power, the physical downlink shared channel (PDSCH) is more vulnerable than the primary synchronization signal (PSS) and secondary synchronization signal (SSS). However, a smart jammer can disrupt the cell search process by injecting less power and targeting PSS-SSS or physical broadcast channel (PBCH) data compared to a barrage jammer. On the other hand, beam sweeping proves effective in mitigating the impact of a smart jammer, reducing the error vector magnitude root mean square from 51.59% to 23.36% under the same jamming power.

Title: EPIC: Enhancing Privacy through Iterative Collaboration

• Authors: Prakash Chourasia, Heramb Lonkar, Sarwan Ali, Murray Patterson
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05167
• Pdf URL: https://arxiv.org/pdf/2411.05167
• Copy Paste: [[2411.05167]] EPIC: Enhancing Privacy through Iterative Collaboration(https://arxiv.org/abs/2411.05167)
• Keywords: privacy, federate
• Abstract: Advancements in genomics technology lead to a rising volume of viral (e.g., SARS-CoV-2) sequence data, resulting in increased usage of machine learning (ML) in bioinformatics. Traditional ML techniques require centralized data collection and processing, posing challenges in realistic healthcare scenarios. Additionally, privacy, ownership, and stringent regulation issues exist when pooling medical data into centralized storage to train a powerful deep learning (DL) model. The Federated learning (FL) approach overcomes such issues by setting up a central aggregator server and a shared global model. It also facilitates data privacy by extracting knowledge while keeping the actual data private. This work proposes a cutting-edge Privacy enhancement through Iterative Collaboration (EPIC) architecture. The network is divided and distributed between local and centralized servers. We demonstrate the EPIC approach to resolve a supervised classification problem to estimate SARS-CoV-2 genomic sequence data lineage without explicitly transferring raw sequence data. We aim to create a universal decentralized optimization framework that allows various data holders to work together and converge to a single predictive model. The findings demonstrate that privacy-preserving strategies can be successfully used with aggregation approaches without materially altering the degree of learning convergence. Finally, we highlight a few potential issues and prospects for study in FL-based approaches to healthcare applications.

Title: ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Language

• Authors: Yuxin Wang, Xiaomeng Zhu, Weimin Lyu, Saeed Hassanpour, Soroush Vosoughi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05172
• Pdf URL: https://arxiv.org/pdf/2411.05172
• Copy Paste: [[2411.05172]] ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Language(https://arxiv.org/abs/2411.05172)
• Keywords: robust, large language model
• Abstract: Handling implicit language is essential for natural language processing systems to achieve precise text understanding and facilitate natural interactions with users. Despite its importance, the absence of a robust metric for accurately measuring the implicitness of language significantly constrains the depth of analysis possible in evaluating models' comprehension capabilities. This paper addresses this gap by developing a scalar metric that quantifies the implicitness level of language without relying on external references. Drawing on principles from traditional linguistics, we define ''implicitness'' as the divergence between semantic meaning and pragmatic interpretation. To operationalize this definition, we introduce ImpScore, a novel, reference-free metric formulated through an interpretable regression model. This model is trained using pairwise contrastive learning on a specially curated dataset comprising $112,580$ (implicit sentence, explicit sentence) pairs. We validate ImpScore through a user study that compares its assessments with human evaluations on out-of-distribution data, demonstrating its accuracy and strong correlation with human judgments. Additionally, we apply ImpScore to hate speech detection datasets, illustrating its utility and highlighting significant limitations in current large language models' ability to understand highly implicit content. The metric model and its training data are available at this https URL.

Title: DWFL: Enhancing Federated Learning through Dynamic Weighted Averaging

• Authors: Prakash Chourasia, Tamkanat E Ali, Sarwan Ali, Murray Pattersn
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05173
• Pdf URL: https://arxiv.org/pdf/2411.05173
• Copy Paste: [[2411.05173]] DWFL: Enhancing Federated Learning through Dynamic Weighted Averaging(https://arxiv.org/abs/2411.05173)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) is a distributed learning technique that maintains data privacy by providing a decentralized training method for machine learning models using distributed big data. This promising Federated Learning approach has also gained popularity in bioinformatics, where the privacy of biomedical data holds immense importance, especially when patient data is involved. Despite the successful implementation of Federated learning in biological sequence analysis, rigorous consideration is still required to improve accuracy in a way that data privacy should not be compromised. Additionally, the optimal integration of federated learning, especially in protein sequence analysis, has not been fully explored. We propose a deep feed-forward neural network-based enhanced federated learning method for protein sequence classification to overcome these challenges. Our method introduces novel enhancements to improve classification accuracy. We introduce dynamic weighted federated learning (DWFL) which is a federated learning-based approach, where local model weights are adjusted using weighted averaging based on their performance metrics. By assigning higher weights to well-performing models, we aim to create a more potent initial global model for the federated learning process, leading to improved accuracy. We conduct experiments using real-world protein sequence datasets to assess the effectiveness of DWFL. The results obtained using our proposed approach demonstrate significant improvements in model accuracy, making federated learning a preferred, more robust, and privacy-preserving approach for collaborative machine-learning tasks.

Title: Interpretable Measurement of CNN Deep Feature Density using Copula and the Generalized Characteristic Function

• Authors: David Chapman, Parniyan Farvardin
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05183
• Pdf URL: https://arxiv.org/pdf/2411.05183
• Copy Paste: [[2411.05183]] Interpretable Measurement of CNN Deep Feature Density using Copula and the Generalized Characteristic Function(https://arxiv.org/abs/2411.05183)
• Keywords: robust
• Abstract: We present a novel empirical approach toward measuring the Probability Density Function (PDF) of the deep features of Convolutional Neural Networks (CNNs). Measurement of the deep feature PDF is a valuable problem for several reasons. Notably, a. Understanding the deep feature PDF yields new insight into deep representations. b. Feature density methods are important for tasks such as anomaly detection which can improve the robustness of deep learning models in the wild. Interpretable measurement of the deep feature PDF is challenging due to the Curse of Dimensionality (CoD), and the Spatial intuition Limitation. Our novel measurement technique combines copula analysis with the Method of Orthogonal Moments (MOM), in order to directly measure the Generalized Characteristic Function (GCF) of the multivariate deep feature PDF. We find that, surprisingly, the one-dimensional marginals of non-negative deep CNN features after major blocks are not well approximated by a Gaussian distribution, and that these features increasingly approximate an exponential distribution with increasing network depth. Furthermore, we observe that deep features become increasingly independent with increasing network depth within their typical ranges. However, we surprisingly also observe that many deep features exhibit strong dependence (either correlation or anti-correlation) with other extremely strong detections, even if these features are independent within typical ranges. We elaborate on these findings in our discussion, where we propose a new hypothesis that exponentially infrequent large valued features correspond to strong computer vision detections of semantic targets, which would imply that these large-valued features are not outliers but rather an important detection signal.

Title: PentestAgent: Incorporating LLM Agents to Automated Penetration Testing

• Authors: Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, Wei Ruan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05185
• Pdf URL: https://arxiv.org/pdf/2411.05185
• Copy Paste: [[2411.05185]] PentestAgent: Incorporating LLM Agents to Automated Penetration Testing(https://arxiv.org/abs/2411.05185)
• Keywords: security, large language model
• Abstract: Penetration testing is a critical technique for identifying security vulnerabilities, traditionally performed manually by skilled security specialists. This complex process involves gathering information about the target system, identifying entry points, exploiting the system, and reporting findings. Despite its effectiveness, manual penetration testing is time-consuming and expensive, often requiring significant expertise and resources that many organizations cannot afford. While automated penetration testing methods have been proposed, they often fall short in real-world applications due to limitations in flexibility, adaptability, and implementation. Recent advancements in large language models (LLMs) offer new opportunities for enhancing penetration testing through increased intelligence and automation. However, current LLM-based approaches still face significant challenges, including limited penetration testing knowledge and a lack of comprehensive automation capabilities. To address these gaps, we propose PentestAgent, a novel LLM-based automated penetration testing framework that leverages the power of LLMs and various LLM-based techniques like Retrieval Augmented Generation (RAG) to enhance penetration testing knowledge and automate various tasks. Our framework leverages multi-agent collaboration to automate intelligence gathering, vulnerability analysis, and exploitation stages, reducing manual intervention. We evaluate PentestAgent using a comprehensive benchmark, demonstrating superior performance in task completion and overall efficiency. This work significantly advances the practical applicability of automated penetration testing systems.

Title: Adversarial Robustness of In-Context Learning in Transformers for Linear Regression

• Authors: Usman Anwar, Johannes Von Oswald, Louis Kirsch, David Krueger, Spencer Frei
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05189
• Pdf URL: https://arxiv.org/pdf/2411.05189
• Copy Paste: [[2411.05189]] Adversarial Robustness of In-Context Learning in Transformers for Linear Regression(https://arxiv.org/abs/2411.05189)
• Keywords: attack, robust, transformer
• Abstract: Transformers have demonstrated remarkable in-context learning capabilities across various domains, including statistical learning tasks. While previous work has shown that transformers can implement common learning algorithms, the adversarial robustness of these learned algorithms remains unexplored. This work investigates the vulnerability of in-context learning in transformers to \textit{hijacking attacks} focusing on the setting of linear regression tasks. Hijacking attacks are prompt-manipulation attacks in which the adversary's goal is to manipulate the prompt to force the transformer to generate a specific output. We first prove that single-layer linear transformers, known to implement gradient descent in-context, are non-robust and can be manipulated to output arbitrary predictions by perturbing a single example in the in-context training set. While our experiments show these attacks succeed on linear transformers, we find they do not transfer to more complex transformers with GPT-2 architectures. Nonetheless, we show that these transformers can be hijacked using gradient-based adversarial attacks. We then demonstrate that adversarial training enhances transformers' robustness against hijacking attacks, even when just applied during finetuning. Additionally, we find that in some settings, adversarial training against a weaker attack model can lead to robustness to a stronger attack model. Lastly, we investigate the transferability of hijacking attacks across transformers of varying scales and initialization seeds, as well as between transformers and ordinary least squares (OLS). We find that while attacks transfer effectively between small-scale transformers, they show poor transferability in other scenarios (small-to-large scale, large-to-large scale, and between transformers and OLS).

Title: Explaining Mixtures of Sources in News Articles

• Authors: Alexander Spangher, James Youn, Matt DeButts, Nanyun Peng, Emilio Ferrara, Jonathan May
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05192
• Pdf URL: https://arxiv.org/pdf/2411.05192
• Copy Paste: [[2411.05192]] Explaining Mixtures of Sources in News Articles(https://arxiv.org/abs/2411.05192)
• Keywords: generative, large language model
• Abstract: Human writers plan, then write. For large language models (LLMs) to play a role in longer-form article generation, we must understand the planning steps humans make before writing. We explore one kind of planning, source-selection in news, as a case-study for evaluating plans in long-form generation. We ask: why do specific stories call for specific kinds of sources? We imagine a generative process for story writing where a source-selection schema is first selected by a journalist, and then sources are chosen based on categories in that schema. Learning the article's plan means predicting the schema initially chosen by the journalist. Working with professional journalists, we adapt five existing schemata and introduce three new ones to describe journalistic plans for the inclusion of sources in documents. Then, inspired by Bayesian latent-variable modeling, we develop metrics to select the most likely plan, or schema, underlying a story, which we use to compare schemata. We find that two schemata: stance and social affiliation best explain source plans in most documents. However, other schemata like textual entailment explain source plans in factually rich topics like "Science". Finally, we find we can predict the most suitable schema given just the article's headline with reasonable accuracy. We see this as an important case-study for human planning, and provides a framework and approach for evaluating other kinds of plans. We release a corpora, NewsSources, with annotations for 4M articles.

Title: Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning

• Authors: Joey Hong, Anca Dragan, Sergey Levine
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05193
• Pdf URL: https://arxiv.org/pdf/2411.05193
• Copy Paste: [[2411.05193]] Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning(https://arxiv.org/abs/2411.05193)
• Keywords: large language model
• Abstract: Value-based reinforcement learning (RL) can in principle learn effective policies for a wide range of multi-turn problems, from games to dialogue to robotic control, including via offline RL from static previously collected datasets. However, despite the widespread use of policy gradient methods to train large language models for single turn tasks (e.g., question answering), value-based methods for multi-turn RL in an off-policy or offline setting have proven particularly challenging to scale to the setting of large language models. This setting requires effectively leveraging pretraining, scaling to large architectures with billions of parameters, and training on large datasets, all of which represent major challenges for current value-based RL methods. In this work, we propose a novel offline RL algorithm that addresses these drawbacks, casting Q-learning as a modified supervised fine-tuning (SFT) problem where the probabilities of tokens directly translate to Q-values. In this way we obtain an algorithm that smoothly transitions from maximizing the likelihood of the data during pretraining to learning a near-optimal Q-function during finetuning. Our algorithm has strong theoretical foundations, enjoying performance bounds similar to state-of-the-art Q-learning methods, while in practice utilizing an objective that closely resembles SFT. Because of this, our approach can enjoy the full benefits of the pretraining of language models, without the need to reinitialize any weights before RL finetuning, and without the need to initialize new heads for predicting values or advantages. Empirically, we evaluate our method on both pretrained LLMs and VLMs, on a variety of tasks including both natural language dialogue and robotic manipulation and navigation from images.

Title: Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations

• Authors: Joey Hong, Jessica Lin, Anca Dragan, Sergey Levine
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05194
• Pdf URL: https://arxiv.org/pdf/2411.05194
• Copy Paste: [[2411.05194]] Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations(https://arxiv.org/abs/2411.05194)
• Keywords: large language model
• Abstract: Recent progress on large language models (LLMs) has enabled dialogue agents to generate highly naturalistic and plausible text. However, current LLM language generation focuses on responding accurately to questions and requests with a single effective response. In reality, many real dialogues are interactive, meaning an agent's utterances will influence their conversational partner, elicit information, or change their opinion. Accounting for how an agent can effectively steer a conversation is a crucial ability in many dialogue tasks, from healthcare to preference elicitation. Existing methods for fine-tuning dialogue agents to accomplish such tasks would rely on curating some amount of expert data. However, doing so often requires understanding the underlying cognitive processes of the conversational partner, which is a skill neither humans nor LLMs trained on human data can reliably do. Our key insight is that while LLMs may not be adept at identifying effective strategies for steering conversations a priori, or in the middle of an ongoing conversation, they can do so post-hoc, or in hindsight, after seeing how their conversational partner responds. We use this fact to rewrite and augment existing suboptimal data, and train via offline reinforcement learning (RL) an agent that outperforms both prompting and learning from unaltered human demonstrations. We apply our approach to two domains that require understanding human mental state, intelligent interaction, and persuasion: mental health support, and soliciting charitable donations. Our results in a user study with real humans show that our approach greatly outperforms existing state-of-the-art dialogue agents.

Title: On Erroneous Agreements of CLIP Image Embeddings

• Authors: Siting Li, Pang Wei Koh, Simon Shaolei Du
• Subjects: cs.LG, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05195
• Pdf URL: https://arxiv.org/pdf/2411.05195
• Copy Paste: [[2411.05195]] On Erroneous Agreements of CLIP Image Embeddings(https://arxiv.org/abs/2411.05195)
• Keywords: large language model
• Abstract: Recent research suggests that the failures of Vision-Language Models (VLMs) at visual reasoning often stem from erroneous agreements -- when semantically distinct images are ambiguously encoded by the CLIP image encoder into embeddings with high cosine similarity. In this paper, we show that erroneous agreements are not always the main culprit, as Multimodal Large Language Models (MLLMs) can still extract distinct information from them. For instance, when distinguishing objects on the left vs right in the What'sUp benchmark, the CLIP image embeddings of the left/right pairs have an average cosine similarity $>0.99$, and CLIP performs at random chance; but LLaVA-1.5-7B, which uses the same CLIP image encoder, achieves nearly $100\%$ accuracy. We find that the extractable information in CLIP image embeddings is likely obscured by CLIP's inadequate vision-language alignment: Its matching score learned by the contrastive objective might not capture all diverse image-text correspondences. We also study the MMVP benchmark, on which prior work has shown that LLaVA-1.5 cannot distinguish image pairs with high cosine similarity. We observe a performance gain brought by attending more to visual input through an alternative decoding algorithm. Further, the accuracy significantly increases if the model can take both images as input to emphasize their nuanced differences. Both findings indicate that LLaVA-1.5 did not utilize extracted visual information sufficiently. In conclusion, our findings suggest that while improving image encoders could benefit VLMs, there is still room to enhance models with a fixed image encoder by applying better strategies for extracting and utilizing visual information.

Title: Hardware and Software Platform Inference

• Authors: Cheng Zhang, Hanna Foerster, Robert D. Mullins, Yiren Zhao, Ilia Shumailov
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05197
• Pdf URL: https://arxiv.org/pdf/2411.05197
• Copy Paste: [[2411.05197]] Hardware and Software Platform Inference(https://arxiv.org/abs/2411.05197)
• Keywords: large language model
• Abstract: It is now a common business practice to buy access to large language model (LLM) inference rather than self-host, because of significant upfront hardware infrastructure and energy costs. However, as a buyer, there is no mechanism to verify the authenticity of the advertised service including the serving hardware platform, e.g. that it is actually being served using an NVIDIA H100. Furthermore, there are reports suggesting that model providers may deliver models that differ slightly from the advertised ones, often to make them run on less expensive hardware. That way, a client pays premium for a capable model access on more expensive hardware, yet ends up being served by a (potentially less capable) cheaper model on cheaper hardware. In this paper we introduce \textit{\textbf{hardware and software platform inference (HSPI)}} -- a method for identifying the underlying \GPU{} architecture and software stack of a (black-box) machine learning model solely based on its input-output behavior. Our method leverages the inherent differences of various \GPU{} architectures and compilers to distinguish between different \GPU{} types and software stacks. By analyzing the numerical patterns in the model's outputs, we propose a classification framework capable of accurately identifying the \GPU{} used for model inference as well as the underlying software configuration. Our findings demonstrate the feasibility of inferring \GPU{} type from black-box models. We evaluate HSPI against models served on different real hardware and find that in a white-box setting we can distinguish between different \GPU{}s with between $83.9\%$ and $100\%$ accuracy. Even in a black-box setting we are able to achieve results that are up to three times higher than random guess accuracy.

Title: Private Algorithms for Stochastic Saddle Points and Variational Inequalities: Beyond Euclidean Geometry

• Authors: Raef Bassily, Cristóbal Guzmán, Michael Menart
• Subjects: cs.LG, cs.CR, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.05198
• Pdf URL: https://arxiv.org/pdf/2411.05198
• Copy Paste: [[2411.05198]] Private Algorithms for Stochastic Saddle Points and Variational Inequalities: Beyond Euclidean Geometry(https://arxiv.org/abs/2411.05198)
• Keywords: privacy
• Abstract: In this work, we conduct a systematic study of stochastic saddle point problems (SSP) and stochastic variational inequalities (SVI) under the constraint of $(\epsilon,\delta)$-differential privacy (DP) in both Euclidean and non-Euclidean setups. We first consider Lipschitz convex-concave SSPs in the $\ell_p/\ell_q$ setup, $p,q\in[1,2]$. Here, we obtain a bound of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$ on the strong SP-gap, where $n$ is the number of samples and $d$ is the dimension. This rate is nearly optimal for any $p,q\in[1,2]$. Without additional assumptions, such as smoothness or linearity requirements, prior work under DP has only obtained this rate when $p=q=2$ (i.e., only in the Euclidean setup). Further, existing algorithms have each only been shown to work for specific settings of $p$ and $q$ and under certain assumptions on the loss and the feasible set, whereas we provide a general algorithm for DP SSPs whenever $p,q\in[1,2]$. Our result is obtained via a novel analysis of the recursive regularization algorithm. In particular, we develop new tools for analyzing generalization, which may be of independent interest. Next, we turn our attention towards SVIs with a monotone, bounded and Lipschitz operator and consider $\ell_p$-setups, $p\in[1,2]$. Here, we provide the first analysis which obtains a bound on the strong VI-gap of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$. For $p-1=\Omega(1)$, this rate is near optimal due to existing lower bounds. To obtain this result, we develop a modified version of recursive regularization. Our analysis builds on the techniques we develop for SSPs as well as employing additional novel components which handle difficulties arising from adapting the recursive regularization framework to SVIs.

Title: CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement

• Authors: Leitian Tao, Xiang Chen, Tong Yu, Tung Mai, Ryan Rossi, Yixuan Li, Saayan Mitra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05199
• Pdf URL: https://arxiv.org/pdf/2411.05199
• Copy Paste: [[2411.05199]] CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement(https://arxiv.org/abs/2411.05199)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have significantly advanced code generation but often require substantial resources and tend to over-generalize, limiting their efficiency for specific tasks. Fine-tuning smaller, open-source LLMs presents a viable alternative; however, it typically lags behind cutting-edge models due to supervised fine-tuning's reliance solely on correct code examples, which restricts the model's ability to learn from its own mistakes and adapt to diverse programming challenges. To bridge this gap, we introduce CodeLutra, a novel framework that enhances low-performing LLMs by leveraging both successful and failed code generation attempts. Unlike conventional fine-tuning, CodeLutra employs an iterative preference learning mechanism to compare correct and incorrect solutions as well as maximize the likelihood of correct codes. Through continuous iterative refinement, CodeLutra enables smaller LLMs to match or surpass GPT-4's performance in various code generation tasks without relying on vast external datasets or larger auxiliary models. On a challenging data analysis task, using just 500 samples improved Llama-3-8B's accuracy from 28.2% to 48.6%, approaching GPT-4's performance. These results highlight CodeLutra's potential to close the gap between open-source and closed-source models, making it a promising approach in the field of code generation.

Title: STAND-Guard: A Small Task-Adaptive Content Moderation Model

• Authors: Minjia Wang, Pingping Lin, Siqi Cai, Shengnan An, Shengjie Ma, Zeqi Lin, Congrui Huang, Bixiong Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05214
• Pdf URL: https://arxiv.org/pdf/2411.05214
• Copy Paste: [[2411.05214]] STAND-Guard: A Small Task-Adaptive Content Moderation Model(https://arxiv.org/abs/2411.05214)
• Keywords: large language model
• Abstract: Content moderation, the process of reviewing and monitoring the safety of generated content, is important for development of welcoming online platforms and responsible large language models. Content moderation contains various tasks, each with its unique requirements tailored to specific scenarios. Therefore, it is crucial to develop a model that can be easily adapted to novel or customized content moderation tasks accurately without extensive model tuning. This paper presents STAND-GUARD, a Small Task-Adaptive coNtent moDeration model. The basic motivation is: by performing instruct tuning on various content moderation tasks, we can unleash the power of small language models (SLMs) on unseen (out-of-distribution) content moderation tasks. We also carefully study the effects of training tasks and model size on the efficacy of cross-task fine-tuning mechanism. Experiments demonstrate STAND-Guard is comparable to GPT-3.5-Turbo across over 40 public datasets, as well as proprietary datasets derived from real-world business scenarios. Remarkably, STAND-Guard achieved nearly equivalent results to GPT-4-Turbo on unseen English binary classification tasks

Title: Anticipatory Understanding of Resilient Agriculture to Climate

• Authors: David Willmes, Nick Krall, James Tanis, Zachary Terner, Fernando Tavares, Joe Haberlin III, Matt Crichton, Alexander Schlichting
• Subjects: cs.CV, cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2411.05219
• Pdf URL: https://arxiv.org/pdf/2411.05219
• Copy Paste: [[2411.05219]] Anticipatory Understanding of Resilient Agriculture to Climate(https://arxiv.org/abs/2411.05219)
• Keywords: security
• Abstract: With billions of people facing moderate or severe food insecurity, the resilience of the global food supply will be of increasing concern due to the effects of climate change and geopolitical events. In this paper we describe a framework to better identify food security hotspots using a combination of remote sensing, deep learning, crop yield modeling, and causal modeling of the food distribution system. While we feel that the methods are adaptable to other regions of the world, we focus our analysis on the wheat breadbasket of northern India, which supplies a large percentage of the world's population. We present a quantitative analysis of deep learning domain adaptation methods for wheat farm identification based on curated remote sensing data from France. We model climate change impacts on crop yields using the existing crop yield modeling tool WOFOST and we identify key drivers of crop simulation error using a longitudinal penalized functional regression. A description of a system dynamics model of the food distribution system in India is also presented, along with results of food insecurity identification based on seeding this model with the predicted crop yields.

Title: Don't Look Twice: Faster Video Transformers with Run-Length Tokenization

• Authors: Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niinuma, Kris M. Kitani, László Jeni
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05222
• Pdf URL: https://arxiv.org/pdf/2411.05222
• Copy Paste: [[2411.05222]] Don't Look Twice: Faster Video Transformers with Run-Length Tokenization(https://arxiv.org/abs/2411.05222)
• Keywords: transformer
• Abstract: Transformers are slow to train on videos due to extremely large numbers of input tokens, even though many video tokens are repeated over time. Existing methods to remove such uninformative tokens either have significant overhead, negating any speedup, or require tuning for different datasets and examples. We present Run-Length Tokenization (RLT), a simple approach to speed up video transformers inspired by run-length encoding for data compression. RLT efficiently finds and removes runs of patches that are repeated over time prior to model inference, then replaces them with a single patch and a positional encoding to represent the resulting token's new length. Our method is content-aware, requiring no tuning for different datasets, and fast, incurring negligible overhead. RLT yields a large speedup in training, reducing the wall-clock time to fine-tune a video transformer by 30% while matching baseline model performance. RLT also works without any training, increasing model throughput by 35% with only 0.1% drop in accuracy. RLT speeds up training at 30 FPS by more than 100%, and on longer video datasets, can reduce the token count by up to 80%. Our project page is at this https URL.

Title: Generalizable Single-Source Cross-modality Medical Image Segmentation via Invariant Causal Mechanisms

• Authors: Boqi Chen, Yuanzhi Zhu, Yunke Ao, Sebastiano Caprara, Reto Sutter, Gunnar Rätsch, Ender Konukoglu, Anna Susmelj
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05223
• Pdf URL: https://arxiv.org/pdf/2411.05223
• Copy Paste: [[2411.05223]] Generalizable Single-Source Cross-modality Medical Image Segmentation via Invariant Causal Mechanisms(https://arxiv.org/abs/2411.05223)
• Keywords: diffusion, generative, segmentation
• Abstract: Single-source domain generalization (SDG) aims to learn a model from a single source domain that can generalize well on unseen target domains. This is an important task in computer vision, particularly relevant to medical imaging where domain shifts are common. In this work, we consider a challenging yet practical setting: SDG for cross-modality medical image segmentation. We combine causality-inspired theoretical insights on learning domain-invariant representations with recent advancements in diffusion-based augmentation to improve generalization across diverse imaging modalities. Guided by the ``intervention-augmentation equivariant'' principle, we use controlled diffusion models (DMs) to simulate diverse imaging styles while preserving the content, leveraging rich generative priors in large-scale pretrained DMs to comprehensively perturb the multidimensional style variable. Extensive experiments on challenging cross-modality segmentation tasks demonstrate that our approach consistently outperforms state-of-the-art SDG methods across three distinct anatomies and imaging modalities. The source code is available at \href{this https URL}{this https URL}.

Title: Beyond the Numbers: Transparency in Relation Extraction Benchmark Creation and Leaderboards

• Authors: Varvara Arzt, Allan Hanbury
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05224
• Pdf URL: https://arxiv.org/pdf/2411.05224
• Copy Paste: [[2411.05224]] Beyond the Numbers: Transparency in Relation Extraction Benchmark Creation and Leaderboards(https://arxiv.org/abs/2411.05224)
• Keywords: extraction
• Abstract: This paper investigates the transparency in the creation of benchmarks and the use of leaderboards for measuring progress in NLP, with a focus on the relation extraction (RE) task. Existing RE benchmarks often suffer from insufficient documentation, lacking crucial details such as data sources, inter-annotator agreement, the algorithms used for the selection of instances for datasets, and information on potential biases like dataset imbalance. Progress in RE is frequently measured by leaderboards that rank systems based on evaluation methods, typically limited to aggregate metrics like F1-score. However, the absence of detailed performance analysis beyond these metrics can obscure the true generalisation capabilities of models. Our analysis reveals that widely used RE benchmarks, such as TACRED and NYT, tend to be highly imbalanced and contain noisy labels. Moreover, the lack of class-based performance metrics fails to accurately reflect model performance across datasets with a large number of relation types. These limitations should be carefully considered when reporting progress in RE. While our discussion centers on the transparency of RE benchmarks and leaderboards, the observations we discuss are broadly applicable to other NLP tasks as well. Rather than undermining the significance and value of existing RE benchmarks and the development of new models, this paper advocates for improved documentation and more rigorous evaluation to advance the field.

Title: Breaking The Ice: Video Segmentation for Close-Range Ice-Covered Waters

• Authors: Corwin Grant Jeon MacMillan, K. Andrea Scott, Zhao Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05225
• Pdf URL: https://arxiv.org/pdf/2411.05225
• Copy Paste: [[2411.05225]] Breaking The Ice: Video Segmentation for Close-Range Ice-Covered Waters(https://arxiv.org/abs/2411.05225)
• Keywords: robust, segmentation
• Abstract: Rapid ice recession in the Arctic Ocean, with predictions of ice-free summers by 2060, opens new maritime routes but requires reliable navigation solutions. Current approaches rely heavily on subjective expert judgment, underscoring the need for automated, data-driven solutions. This study leverages machine learning to assess ice conditions using ship-borne optical data, introducing a finely annotated dataset of 946 images, and a semi-manual, region-based annotation technique. The proposed video segmentation model, UPerFlow, advances the SegFlow architecture by incorporating a six-channel ResNet encoder, two UPerNet-based segmentation decoders for each image, PWCNet as the optical flow encoder, and cross-connections that integrate bi-directional flow features without loss of latent information. The proposed architecture outperforms baseline image segmentation networks by an average 38\% in occluded regions, demonstrating the robustness of video segmentation in addressing challenging Arctic conditions.

Title: CHATTER: A Character Attribution Dataset for Narrative Understanding

• Authors: Sabyasachee Baruah, Shrikanth Narayanan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05227
• Pdf URL: https://arxiv.org/pdf/2411.05227
• Copy Paste: [[2411.05227]] CHATTER: A Character Attribution Dataset for Narrative Understanding(https://arxiv.org/abs/2411.05227)
• Keywords: robust
• Abstract: Computational narrative understanding studies the identification, description, and interaction of the elements of a narrative: characters, attributes, events, and relations. Narrative research has given considerable attention to defining and classifying character types. However, these character-type taxonomies do not generalize well because they are small, too simple, or specific to a domain. We require robust and reliable benchmarks to test whether narrative models truly understand the nuances of the character's development in the story. Our work addresses this by curating the Chatter dataset that labels whether a character portrays some attribute for 88148 character-attribute pairs, encompassing 2998 characters, 13324 attributes and 660 movies. We validate a subset of Chatter, called ChatterEval, using human annotations to serve as an evaluation benchmark for the character attribution task in movie scripts. ChatterEval assesses narrative understanding and the long-context modeling capacity of language models.

Title: Abstract2Appendix: Academic Reviews Enhance LLM Long-Context Capabilities

• Authors: Shengzhi Li, Kittipat Kampa, Rongyu Lin, Bohang Li, Shichao Pei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05232
• Pdf URL: https://arxiv.org/pdf/2411.05232
• Copy Paste: [[2411.05232]] Abstract2Appendix: Academic Reviews Enhance LLM Long-Context Capabilities(https://arxiv.org/abs/2411.05232)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown remarkable performance across various tasks, yet their ability to handle long-context reading remains challenging. This study explores the effectiveness of leveraging high-quality academic peer review data for fine-tuning LLMs to enhance their long-context capabilities. We compare the Direct Preference Optimization (DPO) method with the Supervised Fine-Tuning (SFT) method, demonstrating DPO's superiority and data efficiency. Our experiments show that the fine-tuned model achieves a 4.04-point improvement over phi-3 and a 2.6\% increase on the Qasper benchmark using only 2000 samples. Despite facing limitations in data scale and processing costs, this study underscores the potential of DPO and high-quality data in advancing LLM performance. Additionally, the zero-shot benchmark results indicate that aggregated high-quality human reviews are overwhelmingly preferred over LLM-generated responses, even for the most capable models like GPT-4o. This suggests that high-quality human reviews are extremely rich in information, reasoning, and long-context retrieval, capabilities that even the most advanced models have not fully captured. These findings highlight the high utility of leveraging human reviews to further advance the field.

Title: Generating Highly Designable Proteins with Geometric Algebra Flow Matching

• Authors: Simon Wagner, Leif Seute, Vsevolod Viliuga, Nicolas Wolf, Frauke Gräter, Jan Stühmer
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.05238
• Pdf URL: https://arxiv.org/pdf/2411.05238
• Copy Paste: [[2411.05238]] Generating Highly Designable Proteins with Geometric Algebra Flow Matching(https://arxiv.org/abs/2411.05238)
• Keywords: generative
• Abstract: We introduce a generative model for protein backbone design utilizing geometric products and higher order message passing. In particular, we propose Clifford Frame Attention (CFA), an extension of the invariant point attention (IPA) architecture from AlphaFold2, in which the backbone residue frames and geometric features are represented in the projective geometric algebra. This enables to construct geometrically expressive messages between residues, including higher order terms, using the bilinear operations of the algebra. We evaluate our architecture by incorporating it into the framework of FrameFlow, a state-of-the-art flow matching model for protein backbone generation. The proposed model achieves high designability, diversity and novelty, while also sampling protein backbones that follow the statistical distribution of secondary structure elements found in naturally occurring proteins, a property so far only insufficiently achieved by many state-of-the-art generative models.

Title: What talking you?: Translating Code-Mixed Messaging Texts to English

• Authors: Lynnette Hui Xian Ng, Luo Qi Chan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05253
• Pdf URL: https://arxiv.org/pdf/2411.05253
• Copy Paste: [[2411.05253]] What talking you?: Translating Code-Mixed Messaging Texts to English(https://arxiv.org/abs/2411.05253)
• Keywords: large language model
• Abstract: Translation of code-mixed texts to formal English allow a wider audience to understand these code-mixed languages, and facilitate downstream analysis applications such as sentiment analysis. In this work, we look at translating Singlish, which is colloquial Singaporean English, to formal standard English. Singlish is formed through the code-mixing of multiple Asian languages and dialects. We analysed the presence of other Asian languages and variants which can facilitate translation. Our dataset is short message texts, written as informal communication between Singlish speakers. We use a multi-step prompting scheme on five Large Language Models (LLMs) for language detection and translation. Our analysis show that LLMs do not perform well in this task, and we describe the challenges involved in translation of code-mixed languages. We also release our dataset in this link this https URL.

Title: Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding

• Authors: Jaeyoo Park, Jin Young Choi, Jeonghyung Park, Bohyung Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05254
• Pdf URL: https://arxiv.org/pdf/2411.05254
• Copy Paste: [[2411.05254]] Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding(https://arxiv.org/abs/2411.05254)
• Keywords: large language model
• Abstract: We present a novel OCR-free document understanding framework based on pretrained Multimodal Large Language Models (MLLMs). Our approach employs multi-scale visual features to effectively handle various font sizes within document images. To address the increasing costs of considering the multi-scale visual inputs for MLLMs, we propose the Hierarchical Visual Feature Aggregation (HVFA) module, designed to reduce the number of input tokens to LLMs. Leveraging a feature pyramid with cross-attentive pooling, our approach effectively manages the trade-off between information loss and efficiency without being affected by varying document image sizes. Furthermore, we introduce a novel instruction tuning task, which facilitates the model's text-reading capability by learning to predict the relative positions of input text, eventually minimizing the risk of truncated text caused by the limited capacity of LLMs. Comprehensive experiments validate the effectiveness of our approach, demonstrating superior performance in various document understanding tasks.

Title: QuanCrypt-FL: Quantized Homomorphic Encryption with Pruning for Secure Federated Learning

• Authors: Md Jueal Mia, M. Hadi Amini
• Subjects: cs.CR, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.05260
• Pdf URL: https://arxiv.org/pdf/2411.05260
• Copy Paste: [[2411.05260]] QuanCrypt-FL: Quantized Homomorphic Encryption with Pruning for Secure Federated Learning(https://arxiv.org/abs/2411.05260)
• Keywords: secure, privacy, protect, attack, membership infer, federate
• Abstract: Federated Learning has emerged as a leading approach for decentralized machine learning, enabling multiple clients to collaboratively train a shared model without exchanging private data. While FL enhances data privacy, it remains vulnerable to inference attacks, such as gradient inversion and membership inference, during both training and inference phases. Homomorphic Encryption provides a promising solution by encrypting model updates to protect against such attacks, but it introduces substantial communication overhead, slowing down training and increasing computational costs. To address these challenges, we propose QuanCrypt-FL, a novel algorithm that combines low-bit quantization and pruning techniques to enhance protection against attacks while significantly reducing computational costs during training. Further, we propose and implement mean-based clipping to mitigate quantization overflow or errors. By integrating these methods, QuanCrypt-FL creates a communication-efficient FL framework that ensures privacy protection with minimal impact on model accuracy, thereby improving both computational efficiency and attack resilience. We validate our approach on MNIST, CIFAR-10, and CIFAR-100 datasets, demonstrating superior performance compared to state-of-the-art methods. QuanCrypt-FL consistently outperforms existing method and matches Vanilla-FL in terms of accuracy across varying client. Further, QuanCrypt-FL achieves up to 9x faster encryption, 16x faster decryption, and 1.5x faster inference compared to BatchCrypt, with training time reduced by up to 3x.

Title: Decoding Report Generators: A Cyclic Vision-Language Adapter for Counterfactual Explanations

• Authors: Yingying Fang, Zihao Jin, Shaojie Guo, Jinda Liu, Yijian Gao, Junzhi Ning, Zhiling Yue, Zhi Li, Simon LF Walsh, Guang Yang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05261
• Pdf URL: https://arxiv.org/pdf/2411.05261
• Copy Paste: [[2411.05261]] Decoding Report Generators: A Cyclic Vision-Language Adapter for Counterfactual Explanations(https://arxiv.org/abs/2411.05261)
• Keywords: interpretability, explainability
• Abstract: Despite significant advancements in report generation methods, a critical limitation remains: the lack of interpretability in the generated text. This paper introduces an innovative approach to enhance the explainability of text generated by report generation models. Our method employs cyclic text manipulation and visual comparison to identify and elucidate the features in the original content that influence the generated text. By manipulating the generated reports and producing corresponding images, we create a comparative framework that highlights key attributes and their impact on the text generation process. This approach not only identifies the image features aligned to the generated text but also improves transparency but also provides deeper insights into the decision-making mechanisms of the report generation models. Our findings demonstrate the potential of this method to significantly enhance the interpretability and transparency of AI-generated reports.

Title: Cancer-Net SCa-Synth: An Open Access Synthetically Generated 2D Skin Lesion Dataset for Skin Cancer Classification

• Authors: Chi-en Amy Tai, Oustan Ding, Alexander Wong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05269
• Pdf URL: https://arxiv.org/pdf/2411.05269
• Copy Paste: [[2411.05269]] Cancer-Net SCa-Synth: An Open Access Synthetically Generated 2D Skin Lesion Dataset for Skin Cancer Classification(https://arxiv.org/abs/2411.05269)
• Keywords: diffusion, generative
• Abstract: In the United States, skin cancer ranks as the most commonly diagnosed cancer, presenting a significant public health issue due to its high rates of occurrence and the risk of serious complications if not caught early. Recent advancements in dataset curation and deep learning have shown promise in quick and accurate detection of skin cancer. However, current open-source datasets have significant class imbalances which impedes the effectiveness of these deep learning models. In healthcare, generative artificial intelligence (AI) models have been employed to create synthetic data, addressing data imbalance in datasets by augmenting underrepresented classes and enhancing the overall quality and performance of machine learning models. In this paper, we build on top of previous work by leveraging new advancements in generative AI, notably Stable Diffusion and DreamBooth. We introduce Cancer-Net SCa-Synth, an open access synthetically generated 2D skin lesion dataset for skin cancer classification. Further analysis on the data effectiveness by comparing the ISIC 2020 test set performance for training with and without these synthetic images for a simple model highlights the benefits of leveraging synthetic data to improve performance. Cancer-Net SCa-Synth is publicly available at this https URL as part of a global open-source initiative for accelerating machine learning for cancer care.

Title: Seeing Through the Fog: A Cost-Effectiveness Analysis of Hallucination Detection Systems

• Authors: Alexander Thomas, Seth Rosen, Vishnu Vettrivel
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05270
• Pdf URL: https://arxiv.org/pdf/2411.05270
• Copy Paste: [[2411.05270]] Seeing Through the Fog: A Cost-Effectiveness Analysis of Hallucination Detection Systems(https://arxiv.org/abs/2411.05270)
• Keywords: large language model
• Abstract: This paper presents a comparative analysis of hallucination detection systems for AI, focusing on automatic summarization and question answering tasks for Large Language Models (LLMs). We evaluate different hallucination detection systems using the diagnostic odds ratio (DOR) and cost-effectiveness metrics. Our results indicate that although advanced models can perform better they come at a much higher cost. We also demonstrate how an ideal hallucination detection system needs to maintain performance across different model sizes. Our findings highlight the importance of choosing a detection system aligned with specific application needs and resource constraints. Future research will explore hybrid systems and automated identification of underperforming components to enhance AI reliability and efficiency in detecting and mitigating hallucinations.

Title: Distributed-Order Fractional Graph Operating Network

• Authors: Kai Zhao, Xuhao Li, Qiyu Kang, Feng Ji, Qinxu Ding, Yanan Zhao, Wenfei Liang, Wee Peng Tay
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05274
• Pdf URL: https://arxiv.org/pdf/2411.05274
• Copy Paste: [[2411.05274]] Distributed-Order Fractional Graph Operating Network(https://arxiv.org/abs/2411.05274)
• Keywords: diffusion
• Abstract: We introduce the Distributed-order fRActional Graph Operating Network (DRAGON), a novel continuous Graph Neural Network (GNN) framework that incorporates distributed-order fractional calculus. Unlike traditional continuous GNNs that utilize integer-order or single fractional-order differential equations, DRAGON uses a learnable probability distribution over a range of real numbers for the derivative orders. By allowing a flexible and learnable superposition of multiple derivative orders, our framework captures complex graph feature updating dynamics beyond the reach of conventional models. We provide a comprehensive interpretation of our framework's capability to capture intricate dynamics through the lens of a non-Markovian graph random walk with node feature updating driven by an anomalous diffusion process over the graph. Furthermore, to highlight the versatility of the DRAGON framework, we conduct empirical evaluations across a range of graph learning tasks. The results consistently demonstrate superior performance when compared to traditional continuous GNN models. The implementation code is available at \url{this https URL}.

Title: GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching

• Authors: Sajal Regmi, Chetan Phakami Pun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05276
• Pdf URL: https://arxiv.org/pdf/2411.05276
• Copy Paste: [[2411.05276]] GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching(https://arxiv.org/abs/2411.05276)
• Keywords: large language model
• Abstract: Large Language Models (LLMs), such as GPT (Radford et al., 2019), have significantly advanced artificial intelligence by enabling sophisticated natural language understanding and generation. However, the high computational and financial costs associated with frequent API calls to these models present a substantial bottleneck, especially for applications like customer service chatbots that handle repetitive queries. In this paper, we introduce GPT Semantic Cache, a method that leverages semantic caching of query embeddings in in-memory storage (Redis). By storing embeddings of user queries, our approach efficiently identifies semantically similar questions, allowing for the retrieval of pre-generated responses without redundant API calls to the LLM. This technique reduces operational costs and improves response times, enhancing the efficiency of LLM-powered applications.

Title: Revisiting the Robustness of Watermarking to Paraphrasing Attacks

• Authors: Saksham Rastogi, Danish Pruthi
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05277
• Pdf URL: https://arxiv.org/pdf/2411.05277
• Copy Paste: [[2411.05277]] Revisiting the Robustness of Watermarking to Paraphrasing Attacks(https://arxiv.org/abs/2411.05277)
• Keywords: attack, robust, watermark
• Abstract: Amidst rising concerns about the internet being proliferated with content generated from language models (LMs), watermarking is seen as a principled way to certify whether text was generated from a model. Many recent watermarking techniques slightly modify the output probabilities of LMs to embed a signal in the generated output that can later be detected. Since early proposals for text watermarking, questions about their robustness to paraphrasing have been prominently discussed. Lately, some techniques are deliberately designed and claimed to be robust to paraphrasing. However, such watermarking schemes do not adequately account for the ease with which they can be reverse-engineered. We show that with access to only a limited number of generations from a black-box watermarked model, we can drastically increase the effectiveness of paraphrasing attacks to evade watermark detection, thereby rendering the watermark ineffective.

Title: SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding

• Authors: Ryan Sun, Tianyi Zhou, Xun Chen, Lichao Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05289
• Pdf URL: https://arxiv.org/pdf/2411.05289
• Copy Paste: [[2411.05289]] SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding(https://arxiv.org/abs/2411.05289)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have become essential in advancing natural language processing (NLP) tasks, but their sequential token generation limits inference speed. Multi-Draft Speculative Decoding (MDSD) offers a promising solution by using a smaller draft model to generate multiple token sequences, which the target LLM verifies in parallel. However, current heuristic approaches, such as Recursive Rejection Sampling (RRS), suffer from low acceptance rates in subsequent drafts, limiting the advantages of using multiple drafts. Meanwhile, Optimal Transport with Membership Cost (OTM) can theoretically improve acceptance rates, but its computational cost is too high for real-time use. We present SpecHub, a novel, efficient sampling-verification method for MDSD that improves acceptance rates with only linear computational overhead. By simplifying the OTM problem into a compact Linear Programming model, SpecHub significantly reduces computational complexity. It further accelerates sampling by leveraging a sparse joint distribution, focusing computation on high-probability token sequences. In extensive experiments, Spechub consistently generates 0.05-0.27 and 0.02-0.16 more tokens per step than RRS and RRS without replacement. We attach our code at \url{this https URL}.

Title: Revisiting Network Perturbation for Semi-Supervised Semantic Segmentation

• Authors: Sien Li, Tao Wang, Ruizhe Hu, Wenxi Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05307
• Pdf URL: https://arxiv.org/pdf/2411.05307
• Copy Paste: [[2411.05307]] Revisiting Network Perturbation for Semi-Supervised Semantic Segmentation(https://arxiv.org/abs/2411.05307)
• Keywords: segmentation
• Abstract: In semi-supervised semantic segmentation (SSS), weak-to-strong consistency regularization techniques are widely utilized in recent works, typically combined with input-level and feature-level perturbations. However, the integration between weak-to-strong consistency regularization and network perturbation has been relatively rare. We note several problems with existing network perturbations in SSS that may contribute to this phenomenon. By revisiting network perturbations, we introduce a new approach for network perturbation to expand the existing weak-to-strong consistency regularization for unlabeled data. Additionally, we present a volatile learning process for labeled data, which is uncommon in existing research. Building upon previous work that includes input-level and feature-level perturbations, we present MLPMatch (Multi-Level-Perturbation Match), an easy-to-implement and efficient framework for semi-supervised semantic segmentation. MLPMatch has been validated on the Pascal VOC and Cityscapes datasets, achieving state-of-the-art performance. Code is available from this https URL.

Title: Exploring the Alignment Landscape: LLMs and Geometric Deep Models in Protein Representation

• Authors: Dong Shu, Bingbing Duan, Kai Guo, Kaixiong Zhou, Jiliang Tang, Mengnan Du
• Subjects: cs.LG, cs.AI, cs.CE, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2411.05316
• Pdf URL: https://arxiv.org/pdf/2411.05316
• Copy Paste: [[2411.05316]] Exploring the Alignment Landscape: LLMs and Geometric Deep Models in Protein Representation(https://arxiv.org/abs/2411.05316)
• Keywords: large language model
• Abstract: Latent representation alignment has become a foundational technique for constructing multimodal large language models (MLLM) by mapping embeddings from different modalities into a shared space, often aligned with the embedding space of large language models (LLMs) to enable effective cross-modal understanding. While preliminary protein-focused MLLMs have emerged, they have predominantly relied on heuristic approaches, lacking a fundamental understanding of optimal alignment practices across representations. In this study, we explore the alignment of multimodal representations between LLMs and Geometric Deep Models (GDMs) in the protein domain. We comprehensively evaluate three state-of-the-art LLMs (Gemma2-2B, LLaMa3.1-8B, and LLaMa3.1-70B) with four protein-specialized GDMs (GearNet, GVP, ScanNet, GAT). Our work examines alignment factors from both model and protein perspectives, identifying challenges in current alignment methodologies and proposing strategies to improve the alignment process. Our key findings reveal that GDMs incorporating both graph and 3D structural information align better with LLMs, larger LLMs demonstrate improved alignment capabilities, and protein rarity significantly impacts alignment performance. We also find that increasing GDM embedding dimensions, using two-layer projection heads, and fine-tuning LLMs on protein-specific data substantially enhance alignment quality. These strategies offer potential enhancements to the performance of protein-related multimodal models. Our code and data are available at this https URL.

Title: Fairness in Monotone $k$-submodular Maximization: Algorithms and Applications

• Authors: Yanhui Zhu, Samik Basu, A. Pavan
• Subjects: cs.LG, cs.DS
• Abstract URL: https://arxiv.org/abs/2411.05318
• Pdf URL: https://arxiv.org/pdf/2411.05318
• Copy Paste: [[2411.05318]] Fairness in Monotone $k$-submodular Maximization: Algorithms and Applications(https://arxiv.org/abs/2411.05318)
• Keywords: fair
• Abstract: Submodular optimization has become increasingly prominent in machine learning and fairness has drawn much attention. In this paper, we propose to study the fair $k$-submodular maximization problem and develop a $\frac{1}{3}$-approximation greedy algorithm with a running time of $\mathcal{O}(knB)$. To the best of our knowledge, our work is the first to incorporate fairness in the context of $k$-submodular maximization, and our theoretical guarantee matches the best-known $k$-submodular maximization results without fairness constraints. In addition, we have developed a faster threshold-based algorithm that achieves a $(\frac{1}{3} - \epsilon)$ approximation with $\mathcal{O}(\frac{kn}{\epsilon} \log \frac{B}{\epsilon})$ evaluations of the function $f$. Furthermore, for both algorithms, we provide approximation guarantees when the $k$-submodular function is not accessible but only can be approximately accessed. We have extensively validated our theoretical findings through empirical research and examined the practical implications of fairness. Specifically, we have addressed the question: ``What is the price of fairness?" through case studies on influence maximization with $k$ topics and sensor placement with $k$ types. The experimental results show that the fairness constraints do not significantly undermine the quality of solutions.

Title: SASWISE-UE: Segmentation and Synthesis with Interpretable Scalable Ensembles for Uncertainty Estimation

• Authors: Weijie Chen, Alan McMillan
• Subjects: cs.LG, cs.CV, stat.ME
• Abstract URL: https://arxiv.org/abs/2411.05324
• Pdf URL: https://arxiv.org/pdf/2411.05324
• Copy Paste: [[2411.05324]] SASWISE-UE: Segmentation and Synthesis with Interpretable Scalable Ensembles for Uncertainty Estimation(https://arxiv.org/abs/2411.05324)
• Keywords: robust, interpretability, transformer, segmentation
• Abstract: This paper introduces an efficient sub-model ensemble framework aimed at enhancing the interpretability of medical deep learning models, thus increasing their clinical applicability. By generating uncertainty maps, this framework enables end-users to evaluate the reliability of model outputs. We developed a strategy to develop diverse models from a single well-trained checkpoint, facilitating the training of a model family. This involves producing multiple outputs from a single input, fusing them into a final output, and estimating uncertainty based on output disagreements. Implemented using U-Net and UNETR models for segmentation and synthesis tasks, this approach was tested on CT body segmentation and MR-CT synthesis datasets. It achieved a mean Dice coefficient of 0.814 in segmentation and a Mean Absolute Error of 88.17 HU in synthesis, improved from 89.43 HU by pruning. Additionally, the framework was evaluated under corruption and undersampling, maintaining correlation between uncertainty and error, which highlights its robustness. These results suggest that the proposed approach not only maintains the performance of well-trained models but also enhances interpretability through effective uncertainty estimation, applicable to both convolutional and transformer models in a range of imaging tasks.

Title: Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning

• Authors: Dharmendra Prajapat, Durga Toshniwal
• Subjects: cs.CL, cs.AI, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.05340
• Pdf URL: https://arxiv.org/pdf/2411.05340
• Copy Paste: [[2411.05340]] Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning(https://arxiv.org/abs/2411.05340)
• Keywords: large language model
• Abstract: Task-oriented dialogue (TOD) system is designed to accomplish user-defined tasks through dialogues. The TOD system has progressed towards end-to-end modeling by leveraging pre-trained large language models. Fine-tuning the pre-trained language models using only supervised learning leads to the exposure bias and token loss problem and it deviates the models from completing the user's task. To address these issues, we propose a TOD system that leverages a unified pre-trained language model, GPT2, as a base model. It is optimized using supervised learning and reinforcement learning (RL). The issues in the TOD system are mitigated using a non-differentiable reward function. The reward is calculated using the weighted sum of the success rate and BLEU evaluation metrics. The success rate and BLEU metrics in reward calculation guide the language model for user task completion while ensuring a coherent and fluent response. Our model is acquired by fine-tuning a pre-trained model on the dialogue-session level which comprises user utterance, belief state, system act, and system response. Experimental results on MultiWOZ2.1 demonstrate that our model increases the inform rate by 1.60% and the success rate by 3.17% compared to the baseline.

Title: Reasoning Robustness of LLMs to Adversarial Typographical Errors

• Authors: Esther Gan, Yiran Zhao, Liying Cheng, Yancan Mao, Anirudh Goyal, Kenji Kawaguchi, Min-Yen Kan, Michael Shieh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05345
• Pdf URL: https://arxiv.org/pdf/2411.05345
• Copy Paste: [[2411.05345]] Reasoning Robustness of LLMs to Adversarial Typographical Errors(https://arxiv.org/abs/2411.05345)
• Keywords: attack, robust, large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning using Chain-of-Thought (CoT) prompting. However, CoT can be biased by users' instruction. In this work, we study the reasoning robustness of LLMs to typographical errors, which can naturally occur in users' queries. We design an Adversarial Typo Attack ($\texttt{ATA}$) algorithm that iteratively samples typos for words that are important to the query and selects the edit that is most likely to succeed in attacking. It shows that LLMs are sensitive to minimal adversarial typographical changes. Notably, with 1 character edit, Mistral-7B-Instruct's accuracy drops from 43.7% to 38.6% on GSM8K, while with 8 character edits the performance further drops to 19.2%. To extend our evaluation to larger and closed-source LLMs, we develop the $\texttt{R$^2$ATA}$ benchmark, which assesses models' $\underline{R}$easoning $\underline{R}$obustness to $\underline{\texttt{ATA}}$. It includes adversarial typographical questions derived from three widely used reasoning datasets-GSM8K, BBH, and MMLU-by applying $\texttt{ATA}$ to open-source LLMs. $\texttt{R$^2$ATA}$ demonstrates remarkable transferability and causes notable performance drops across multiple super large and closed-source LLMs.

Title: RED: Residual Estimation Diffusion for Low-Dose PET Sinogram Reconstruction

• Authors: Xingyu Ai, Bin Huang, Fang Chen, Liu Shi, Binxuan Li, Shaoyu Wang, Qiegen Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05354
• Pdf URL: https://arxiv.org/pdf/2411.05354
• Copy Paste: [[2411.05354]] RED: Residual Estimation Diffusion for Low-Dose PET Sinogram Reconstruction(https://arxiv.org/abs/2411.05354)
• Keywords: diffusion, generative
• Abstract: Recent advances in diffusion models have demonstrated exceptional performance in generative tasks across vari-ous fields. In positron emission tomography (PET), the reduction in tracer dose leads to information loss in sino-grams. Using diffusion models to reconstruct missing in-formation can improve imaging quality. Traditional diffu-sion models effectively use Gaussian noise for image re-constructions. However, in low-dose PET reconstruction, Gaussian noise can worsen the already sparse data by introducing artifacts and inconsistencies. To address this issue, we propose a diffusion model named residual esti-mation diffusion (RED). From the perspective of diffusion mechanism, RED uses the residual between sinograms to replace Gaussian noise in diffusion process, respectively sets the low-dose and full-dose sinograms as the starting point and endpoint of reconstruction. This mechanism helps preserve the original information in the low-dose sinogram, thereby enhancing reconstruction reliability. From the perspective of data consistency, RED introduces a drift correction strategy to reduce accumulated prediction errors during the reverse process. Calibrating the inter-mediate results of reverse iterations helps maintain the data consistency and enhances the stability of reconstruc-tion process. Experimental results show that RED effec-tively improves the quality of low-dose sinograms as well as the reconstruction results. The code is available at: this https URL.

Title: Enhancing Visual Classification using Comparative Descriptors

• Authors: Hankyeol Lee, Gawon Seo, Wonseok Choi, Geunyoung Jung, Kyungwoo Song, Jiyoung Jung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05357
• Pdf URL: https://arxiv.org/pdf/2411.05357
• Copy Paste: [[2411.05357]] Enhancing Visual Classification using Comparative Descriptors(https://arxiv.org/abs/2411.05357)
• Keywords: robust, large language model
• Abstract: The performance of vision-language models (VLMs), such as CLIP, in visual classification tasks, has been enhanced by leveraging semantic knowledge from large language models (LLMs), including GPT. Recent studies have shown that in zero-shot classification tasks, descriptors incorporating additional cues, high-level concepts, or even random characters often outperform those using only the category name. In many classification tasks, while the top-1 accuracy may be relatively low, the top-5 accuracy is often significantly higher. This gap implies that most misclassifications occur among a few similar classes, highlighting the model's difficulty in distinguishing between classes with subtle differences. To address this challenge, we introduce a novel concept of comparative descriptors. These descriptors emphasize the unique features of a target class against its most similar classes, enhancing differentiation. By generating and integrating these comparative descriptors into the classification framework, we refine the semantic focus and improve classification accuracy. An additional filtering process ensures that these descriptors are closer to the image embeddings in the CLIP space, further enhancing performance. Our approach demonstrates improved accuracy and robustness in visual classification tasks by addressing the specific challenge of subtle inter-class differences.

Title: Agricultural Landscape Understanding At Country-Scale

• Authors: Radhika Dua, Nikita Saxena, Aditi Agarwal, Alex Wilson, Gaurav Singh, Hoang Tran, Ishan Deshpande, Amandeep Kaur, Gaurav Aggarwal, Chandan Nath, Arnab Basu, Vishal Batchu, Sharath Holla, Bindiya Kurle, Olana Missura, Rahul Aggarwal, Shubhika Garg, Nishi Shah, Avneet Singh, Dinesh Tewari, Agata Dondzik, Bharat Adsul, Milind Sohoni, Asim Rama Praveen, Aaryan Dangi, Lisan Kadivar, E Abhishek, Niranjan Sudhansu, Kamlakar Hattekar, Sameer Datar, Musty Krishna Chaithanya, Anumas Ranjith Reddy, Aashish Kumar, Betala Laxmi Tirumala, Alok Talekar
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.05359
• Pdf URL: https://arxiv.org/pdf/2411.05359
• Copy Paste: [[2411.05359]] Agricultural Landscape Understanding At Country-Scale(https://arxiv.org/abs/2411.05359)
• Keywords: segmentation
• Abstract: Agricultural landscapes are quite complex, especially in the Global South where fields are smaller, and agricultural practices are more varied. In this paper we report on our progress in digitizing the agricultural landscape (natural and man-made) in our study region of India. We use high resolution imagery and a UNet style segmentation model to generate the first of its kind national-scale multi-class panoptic segmentation output. Through this work we have been able to identify individual fields across 151.7M hectares, and delineating key features such as water resources and vegetation. We share how this output was validated by our team and externally by downstream users, including some sample use cases that can lead to targeted data driven decision making. We believe this dataset will contribute towards digitizing agriculture by generating the foundational baselayer.

Title: Quantum Rewinding for IOP-Based Succinct Arguments

• Authors: Alessandro Chiesa, Marcel Dall Agnol, Zijing Di, Ziyi Guan, Nicholas Spooner
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2411.05360
• Pdf URL: https://arxiv.org/pdf/2411.05360
• Copy Paste: [[2411.05360]] Quantum Rewinding for IOP-Based Succinct Arguments(https://arxiv.org/abs/2411.05360)
• Keywords: secure, security
• Abstract: We analyze the post-quantum security of succinct interactive arguments constructed from interactive oracle proofs (IOPs) and vector commitment schemes. We prove that an interactive variant of the BCS transformation is secure in the standard model against quantum adversaries when the vector commitment scheme is collapsing. Our proof builds on and extends prior work on the post-quantum security of Kilians succinct interactive argument, which is instead based on probabilistically checkable proofs (PCPs). We introduce a new quantum rewinding strategy that works across any number of rounds. As a consequence of our results, we obtain standard-model post-quantum secure succinct arguments with the best asymptotic complexity known.

Title: AuthFormer: Adaptive Multimodal biometric authentication transformer for middle-aged and elderly people

• Authors: Yang rui, Meng ling-tao, Zhang qiu-yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05395
• Pdf URL: https://arxiv.org/pdf/2411.05395
• Copy Paste: [[2411.05395]] AuthFormer: Adaptive Multimodal biometric authentication transformer for middle-aged and elderly people(https://arxiv.org/abs/2411.05395)
• Keywords: security, robust, biometric, transformer
• Abstract: Multimodal biometric authentication methods address the limitations of unimodal biometric technologies in security, robustness, and user adaptability. However, most existing methods depend on fixed combinations and numbers of biometric modalities, which restricts flexibility and adaptability in real-world applications. To overcome these challenges, we propose an adaptive multimodal biometric authentication model, AuthFormer, tailored for elderly users. AuthFormer is trained on the LUTBIO multimodal biometric database, containing biometric data from elderly individuals. By incorporating a cross-attention mechanism and a Gated Residual Network (GRN), the model improves adaptability to physiological variations in elderly users. Experiments show that AuthFormer achieves an accuracy of 99.73%. Additionally, its encoder requires only two layers to perform optimally, reducing complexity compared to traditional Transformer-based models.

Title: Post-Hoc Robustness Enhancement in Graph Neural Networks with Conditional Random Fields

• Authors: Yassine Abbahaddou, Sofiane Ennadir, Johannes F. Lutzeyer, Fragkiskos D. Malliaros, Michalis Vazirgiannis
• Subjects: cs.LG, cs.SI, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.05399
• Pdf URL: https://arxiv.org/pdf/2411.05399
• Copy Paste: [[2411.05399]] Post-Hoc Robustness Enhancement in Graph Neural Networks with Conditional Random Fields(https://arxiv.org/abs/2411.05399)
• Keywords: defense, attack, robust
• Abstract: Graph Neural Networks (GNNs), which are nowadays the benchmark approach in graph representation learning, have been shown to be vulnerable to adversarial attacks, raising concerns about their real-world applicability. While existing defense techniques primarily concentrate on the training phase of GNNs, involving adjustments to message passing architectures or pre-processing methods, there is a noticeable gap in methods focusing on increasing robustness during inference. In this context, this study introduces RobustCRF, a post-hoc approach aiming to enhance the robustness of GNNs at the inference stage. Our proposed method, founded on statistical relational learning using a Conditional Random Field, is model-agnostic and does not require prior knowledge about the underlying model architecture. We validate the efficacy of this approach across various models, leveraging benchmark node classification datasets.

Title: Palermo: Improving the Performance of Oblivious Memory using Protocol-Hardware Co-Design

• Authors: Haojie Ye, Yuchen Xia, Yuhan Chen, Kuan-Yu Chen, Yichao Yuan, Shuwen Deng, Baris Kasikci, Trevor Mudge, Nishil Talati
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2411.05400
• Pdf URL: https://arxiv.org/pdf/2411.05400
• Copy Paste: [[2411.05400]] Palermo: Improving the Performance of Oblivious Memory using Protocol-Hardware Co-Design(https://arxiv.org/abs/2411.05400)
• Keywords: security, privacy, attack
• Abstract: Oblivious RAM (ORAM) hides the memory access patterns, enhancing data privacy by preventing attackers from discovering sensitive information based on the sequence of memory accesses. The performance of ORAM is often limited by its inherent trade-off between security and efficiency, as concealing memory access patterns imposes significant computational and memory overhead. While prior works focus on improving the ORAM performance by prefetching and eliminating ORAM requests, we find that their performance is very sensitive to workload locality behavior and incurs additional management overhead caused by the ORAM stash pressure. This paper presents Palermo: a protocol-hardware co-design to improve ORAM performance. The key observation in Palermo is that classical ORAM protocols enforce restrictive dependencies between memory operations that result in low memory bandwidth utilization. Palermo introduces a new protocol that overlaps large portions of memory operations, within a single and between multiple ORAM requests, without breaking correctness and security guarantees. Subsequently, we propose an ORAM controller architecture that executes the proposed protocol to service ORAM requests. The hardware is responsible for concurrently issuing memory requests as well as imposing the necessary dependencies to ensure a consistent view of the ORAM tree across requests. Using a rich workload mix, we demonstrate that Palermo outperforms the RingORAM baseline by 2.8x, on average, incurring a negligible area overhead of 5.78mm^2 (less than 2% in 12th generation Intel CPU after technology scaling) and 2.14W without sacrificing security. We further show that Palermo also outperforms the state-of-the-art works PageORAM, PrORAM, and IR-ORAM.

Title: Benchmarking Distributional Alignment of Large Language Models

• Authors: Nicole Meister, Carlos Guestrin, Tatsunori Hashimoto
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05403
• Pdf URL: https://arxiv.org/pdf/2411.05403
• Copy Paste: [[2411.05403]] Benchmarking Distributional Alignment of Large Language Models(https://arxiv.org/abs/2411.05403)
• Keywords: large language model
• Abstract: Language models (LMs) are increasingly used as simulacra for people, yet their ability to match the distribution of views of a specific demographic group and be \textit{distributionally aligned} remains uncertain. This notion of distributional alignment is complex, as there is significant variation in the types of attributes that are simulated. Prior works have underexplored the role of three critical variables -- the question domain, steering method, and distribution expression method -- which motivates our contribution of a benchmark explicitly addressing these dimensions. We construct a dataset expanding beyond political values, create human baselines for this task, and evaluate the extent to which an LM can align with a particular group's opinion distribution to inform design choices of such simulation systems. Our analysis reveals open problems regarding if, and how, LMs can be used to simulate humans, and that LLMs can more accurately describe the opinion distribution than simulate such distributions.

Title: Gap-Filling Prompting Enhances Code-Assisted Mathematical Reasoning

• Authors: Mohammad Ghiasvand Mohammadkhani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05407
• Pdf URL: https://arxiv.org/pdf/2411.05407
• Copy Paste: [[2411.05407]] Gap-Filling Prompting Enhances Code-Assisted Mathematical Reasoning(https://arxiv.org/abs/2411.05407)
• Keywords: large language model
• Abstract: Despite the strong performance of large language models (LLMs) in tasks like mathematical reasoning, their practical use is limited by high computational demands and proprietary restrictions. Chain-of-thought (CoT) and program-of-thought (PoT) fine-tuning are common methods to transfer LLM knowledge to small language models (SLMs). However, CoT often leads to calculation errors in SLMs, while PoT has shown more promise. While most PoT-based approaches focus on direct problem-to-code conversion or extracting only the key information from questions and then providing code solution for it, this work emphasizes filling the gaps in the question to clearly illustrate the solution path, which can be challenging for an SLM to understand when such information is not explicitly provided. Therefore, this paper introduces Gap-Filling Prompting (GFP), a novel two-step prompting strategy designed to enhance the problem-solving process for SLMs. The first step identifies these gaps and provides hints for filling them, while the second step adds the hints to the question to generate a final code solution. Experimental results on two benchmark datasets demonstrate that GFP significantly improves the mathematical reasoning abilities of SLMs.

Title: POC-SLT: Partial Object Completion with SDF Latent Transformers

• Authors: Faezeh Zakeri, Raphael Braun, Lukas Ruppert, Henrik P.A. Lensch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05419
• Pdf URL: https://arxiv.org/pdf/2411.05419
• Copy Paste: [[2411.05419]] POC-SLT: Partial Object Completion with SDF Latent Transformers(https://arxiv.org/abs/2411.05419)
• Keywords: transformer
• Abstract: 3D geometric shape completion hinges on representation learning and a deep understanding of geometric data. Without profound insights into the three-dimensional nature of the data, this task remains unattainable. Our work addresses this challenge of 3D shape completion given partial observations by proposing a transformer operating on the latent space representing Signed Distance Fields (SDFs). Instead of a monolithic volume, the SDF of an object is partitioned into smaller high-resolution patches leading to a sequence of latent codes. The approach relies on a smooth latent space encoding learned via a variational autoencoder (VAE), trained on millions of 3D patches. We employ an efficient masked autoencoder transformer to complete partial sequences into comprehensive shapes in latent space. Our approach is extensively evaluated on partial observations from ShapeNet and the ABC dataset where only fractions of the objects are given. The proposed POC-SLT architecture compares favorably with several baseline state-of-the-art methods, demonstrating a significant improvement in 3D shape completion, both qualitatively and quantitatively.

Title: WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning

• Authors: Xiangyu Zhao, Zhiwang Zhou, Wenlong Zhang, Yihao Liu, Xiangyu Chen, Junchao Gong, Hao Chen, Ben Fei, Shiqi Chen, Wanli Ouyang, Xiao-Ming Wu, Lei Bai
• Subjects: cs.LG, cs.AI, cs.CV, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2411.05420
• Pdf URL: https://arxiv.org/pdf/2411.05420
• Copy Paste: [[2411.05420]] WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning(https://arxiv.org/abs/2411.05420)
• Keywords: large language model
• Abstract: The Earth's weather system encompasses intricate weather data modalities and diverse weather understanding tasks, which hold significant value to human life. Existing data-driven models focus on single weather understanding tasks (e.g., weather forecasting). Although these models have achieved promising results, they fail to tackle various complex tasks within a single and unified model. Moreover, the paradigm that relies on limited real observations for a single scenario hinders the model's performance upper bound. In response to these limitations, we draw inspiration from the in-context learning paradigm employed in state-of-the-art visual foundation models and large language models. In this paper, we introduce the first generalist weather foundation model (WeatherGFM), designed to address a wide spectrum of weather understanding tasks in a unified manner. More specifically, we initially unify the representation and definition of the diverse weather understanding tasks. Subsequently, we devised weather prompt formats to manage different weather data modalities, namely single, multiple, and temporal modalities. Finally, we adopt a visual prompting question-answering paradigm for the training of unified weather understanding tasks. Extensive experiments indicate that our WeatherGFM can effectively handle up to ten weather understanding tasks, including weather forecasting, super-resolution, weather image translation, and post-processing. Our method also showcases generalization ability on unseen tasks.

Title: VISTA: Visual Integrated System for Tailored Automation in Math Problem Generation Using LLM

• Authors: Jeongwoo Lee, Kwangsuk Park, Jihyeon Park
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05423
• Pdf URL: https://arxiv.org/pdf/2411.05423
• Copy Paste: [[2411.05423]] VISTA: Visual Integrated System for Tailored Automation in Math Problem Generation Using LLM(https://arxiv.org/abs/2411.05423)
• Keywords: large language model
• Abstract: Generating accurate and consistent visual aids is a critical challenge in mathematics education, where visual representations like geometric shapes and functions play a pivotal role in enhancing student comprehension. This paper introduces a novel multi-agent framework that leverages Large Language Models (LLMs) to automate the creation of complex mathematical visualizations alongside coherent problem text. Our approach not only simplifies the generation of precise visual aids but also aligns these aids with the problem's core mathematical concepts, improving both problem creation and assessment. By integrating multiple agents, each responsible for distinct tasks such as numeric calculation, geometry validation, and visualization, our system delivers mathematically accurate and contextually relevant problems with visual aids. Evaluation across Geometry and Function problem types shows that our method significantly outperforms basic LLMs in terms of text coherence, consistency, relevance and similarity, while maintaining the essential geometrical and functional integrity of the original problems. Although some challenges remain in ensuring consistent visual outputs, our framework demonstrates the immense potential of LLMs in transforming the way educators generate and utilize visual aids in math education.

Title: Dave: a decentralized, secure, and lively fraud-proof algorithm

• Authors: Diego Nehab, Gabriel Coutinho de Paula, Augusto Teixeira
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.05463
• Pdf URL: https://arxiv.org/pdf/2411.05463
• Copy Paste: [[2411.05463]] Dave: a decentralized, secure, and lively fraud-proof algorithm(https://arxiv.org/abs/2411.05463)
• Keywords: secure, security
• Abstract: In this paper, we introduce a new fraud-proof algorithm that offers an unprecedented combination of decentralization, security, and liveness. The resources that must be mobilized by an honest participant to defeat an adversary grow only logarithmically with what the adversary ultimately loses. As a consequence, there is no need to introduce high bonds that prevent an adversary from creating too many Sybils. This makes the system very inclusive and frees participants from having to pool resources among themselves to engage the protocol. Finally, the maximum delay to finalization also grows only logarithmically with total adversarial expenditure, with the smallest multiplicative factor to date. In summary: the entire dispute completes in 2--5 challenge periods, the only way to break consensus is to censor the honest party for more than one challenge period, and the costs of engaging in the dispute are minimal.

Title: Bridging the Gap between Learning and Inference for Diffusion-Based Molecule Generation

• Authors: Peidong Liu, Wenbo Zhang, Xue Zhe, Jiancheng Lv, Xianggen Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05472
• Pdf URL: https://arxiv.org/pdf/2411.05472
• Copy Paste: [[2411.05472]] Bridging the Gap between Learning and Inference for Diffusion-Based Molecule Generation(https://arxiv.org/abs/2411.05472)
• Keywords: diffusion
• Abstract: The efficacy of diffusion models in generating a spectrum of data modalities, including images, text, and videos, has spurred inquiries into their utility in molecular generation, yielding significant advancements in the field. However, the molecular generation process with diffusion models involves multiple autoregressive steps over a finite time horizon, leading to exposure bias issues inherently. To address the exposure bias issue, we propose a training framework named GapDiff. The core idea of GapDiff is to utilize model-predicted conformations as ground truth probabilistically during training, aiming to mitigate the data distributional disparity between training and inference, thereby enhancing the affinity of generated molecules. We conduct experiments using a 3D molecular generation model on the CrossDocked2020 dataset, and the vina energy and diversity demonstrate the potency of our framework with superior affinity. GapDiff is available at \url{this https URL}.

Title: Improving image synthesis with diffusion-negative sampling

• Authors: Alakh Desai, Nuno Vasconcelos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05473
• Pdf URL: https://arxiv.org/pdf/2411.05473
• Copy Paste: [[2411.05473]] Improving image synthesis with diffusion-negative sampling(https://arxiv.org/abs/2411.05473)
• Keywords: diffusion
• Abstract: For image generation with diffusion models (DMs), a negative prompt n can be used to complement the text prompt p, helping define properties not desired in the synthesized image. While this improves prompt adherence and image quality, finding good negative prompts is challenging. We argue that this is due to a semantic gap between humans and DMs, which makes good negative prompts for DMs appear unintuitive to humans. To bridge this gap, we propose a new diffusion-negative prompting (DNP) strategy. DNP is based on a new procedure to sample images that are least compliant with p under the distribution of the DM, denoted as diffusion-negative sampling (DNS). Given p, one such image is sampled, which is then translated into natural language by the user or a captioning model, to produce the negative prompt n*. The pair (p, n*) is finally used to prompt the DM. DNS is straightforward to implement and requires no training. Experiments and human evaluations show that DNP performs well both quantitatively and qualitatively and can be easily combined with several DM variants.

Title: EUREKHA: Enhancing User Representation for Key Hackers Identification in Underground Forums

• Authors: Abdoul Nasser Hassane Amadou, Anas Motii, Saida Elouardi, EL Houcine Bergou
• Subjects: cs.CR, cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2411.05479
• Pdf URL: https://arxiv.org/pdf/2411.05479
• Copy Paste: [[2411.05479]] EUREKHA: Enhancing User Representation for Key Hackers Identification in Underground Forums(https://arxiv.org/abs/2411.05479)
• Keywords: transformer, large language model
• Abstract: Underground forums serve as hubs for cybercriminal activities, offering a space for anonymity and evasion of conventional online oversight. In these hidden communities, malicious actors collaborate to exchange illicit knowledge, tools, and tactics, driving a range of cyber threats from hacking techniques to the sale of stolen data, malware, and zero-day exploits. Identifying the key instigators (i.e., key hackers), behind these operations is essential but remains a complex challenge. This paper presents a novel method called EUREKHA (Enhancing User Representation for Key Hacker Identification in Underground Forums), designed to identify these key hackers by modeling each user as a textual sequence. This sequence is processed through a large language model (LLM) for domain-specific adaptation, with LLMs acting as feature extractors. These extracted features are then fed into a Graph Neural Network (GNN) to model user structural relationships, significantly improving identification accuracy. Furthermore, we employ BERTopic (Bidirectional Encoder Representations from Transformers Topic Modeling) to extract personalized topics from user-generated content, enabling multiple textual representations per user and optimizing the selection of the most representative sequence. Our study demonstrates that fine-tuned LLMs outperform state-of-the-art methods in identifying key hackers. Additionally, when combined with GNNs, our model achieves significant improvements, resulting in approximately 6% and 10% increases in accuracy and F1-score, respectively, over existing methods. EUREKHA was tested on the Hack-Forums dataset, and we provide open-source access to our code.

Title: The Limits of Differential Privacy in Online Learning

• Authors: Bo Li, Wei Wang, Peng Ye
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05483
• Pdf URL: https://arxiv.org/pdf/2411.05483
• Copy Paste: [[2411.05483]] The Limits of Differential Privacy in Online Learning(https://arxiv.org/abs/2411.05483)
• Keywords: privacy
• Abstract: Differential privacy (DP) is a formal notion that restricts the privacy leakage of an algorithm when running on sensitive data, in which privacy-utility trade-off is one of the central problems in private data analysis. In this work, we investigate the fundamental limits of differential privacy in online learning algorithms and present evidence that separates three types of constraints: no DP, pure DP, and approximate DP. We first describe a hypothesis class that is online learnable under approximate DP but not online learnable under pure DP under the adaptive adversarial setting. This indicates that approximate DP must be adopted when dealing with adaptive adversaries. We then prove that any private online learner must make an infinite number of mistakes for almost all hypothesis classes. This essentially generalizes previous results and shows a strong separation between private and non-private settings since a finite mistake bound is always attainable (as long as the class is online learnable) when there is no privacy requirement.

Title: Do Histopathological Foundation Models Eliminate Batch Effects? A Comparative Study

• Authors: Jonah Kömen, Hannah Marienwald, Jonas Dippel, Julius Hense
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05489
• Pdf URL: https://arxiv.org/pdf/2411.05489
• Copy Paste: [[2411.05489]] Do Histopathological Foundation Models Eliminate Batch Effects? A Comparative Study(https://arxiv.org/abs/2411.05489)
• Keywords: robust
• Abstract: Deep learning has led to remarkable advancements in computational histopathology, e.g., in diagnostics, biomarker prediction, and outcome prognosis. Yet, the lack of annotated data and the impact of batch effects, e.g., systematic technical data differences across hospitals, hamper model robustness and generalization. Recent histopathological foundation models -- pretrained on millions to billions of images -- have been reported to improve generalization performances on various downstream tasks. However, it has not been systematically assessed whether they fully eliminate batch effects. In this study, we empirically show that the feature embeddings of the foundation models still contain distinct hospital signatures that can lead to biased predictions and misclassifications. We further find that the signatures are not removed by stain normalization methods, dominate distances in feature space, and are evident across various principal components. Our work provides a novel perspective on the evaluation of medical foundation models, paving the way for more robust pretraining strategies and downstream predictors.

Title: KyrgyzNLP: Challenges, Progress, and Future

• Authors: Anton Alekseev, Timur Turatali
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05503
• Pdf URL: https://arxiv.org/pdf/2411.05503
• Copy Paste: [[2411.05503]] KyrgyzNLP: Challenges, Progress, and Future(https://arxiv.org/abs/2411.05503)
• Keywords: large language model
• Abstract: Large language models (LLMs) have excelled in numerous benchmarks, advancing AI applications in both linguistic and non-linguistic tasks. However, this has primarily benefited well-resourced languages, leaving less-resourced ones (LRLs) at a disadvantage. In this paper, we highlight the current state of the NLP field in the specific LRL: kyrgyz tili. Human evaluation, including annotated datasets created by native speakers, remains an irreplaceable component of reliable NLP performance, especially for LRLs where automatic evaluations can fall short. In recent assessments of the resources for Turkic languages, Kyrgyz is labeled with the status 'Scraping By', a severely under-resourced language spoken by millions. This is concerning given the growing importance of the language, not only in Kyrgyzstan but also among diaspora communities where it holds no official status. We review prior efforts in the field, noting that many of the publicly available resources have only recently been developed, with few exceptions beyond dictionaries (the processed data used for the analysis is presented at this https URL). While recent papers have made some headway, much more remains to be done. Despite interest and support from both business and government sectors in the Kyrgyz Republic, the situation for Kyrgyz language resources remains challenging. We stress the importance of community-driven efforts to build these resources, ensuring the future advancement sustainability. We then share our view of the most pressing challenges in Kyrgyz NLP. Finally, we propose a roadmap for future development in terms of research topics and language resources.

Title: LBPE: Long-token-first Tokenization to Improve Large Language Models

• Authors: Haoran Lian, Yizhe Xiong, Zijia Lin, Jianwei Niu, Shasha Mo, Hui Chen, Peng Liu, Guiguang Ding
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05504
• Pdf URL: https://arxiv.org/pdf/2411.05504
• Copy Paste: [[2411.05504]] LBPE: Long-token-first Tokenization to Improve Large Language Models(https://arxiv.org/abs/2411.05504)
• Keywords: robust, large language model
• Abstract: The prevalent use of Byte Pair Encoding (BPE) in Large Language Models (LLMs) facilitates robust handling of subword units and avoids issues of out-of-vocabulary words. Despite its success, a critical challenge persists: long tokens, rich in semantic information, have fewer occurrences in tokenized datasets compared to short tokens, which can result in imbalanced learning issue across different tokens. To address that, we propose LBPE, which prioritizes long tokens during the encoding process. LBPE generates tokens according to their reverse ranks of token length rather than their ranks in the vocabulary, granting longer tokens higher priority during the encoding process. Consequently, LBPE smooths the frequency differences between short and long tokens, and thus mitigates the learning imbalance. Extensive experiments across diverse language modeling tasks demonstrate that LBPE consistently outperforms the original BPE, well demonstrating its effectiveness.

Title: Towards Lifelong Few-Shot Customization of Text-to-Image Diffusion

• Authors: Nan Song, Xiaofeng Yang, Ze Yang, Guosheng Lin
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05544
• Pdf URL: https://arxiv.org/pdf/2411.05544
• Copy Paste: [[2411.05544]] Towards Lifelong Few-Shot Customization of Text-to-Image Diffusion(https://arxiv.org/abs/2411.05544)
• Keywords: diffusion, data-free
• Abstract: Lifelong few-shot customization for text-to-image diffusion aims to continually generalize existing models for new tasks with minimal data while preserving old knowledge. Current customization diffusion models excel in few-shot tasks but struggle with catastrophic forgetting problems in lifelong generations. In this study, we identify and categorize the catastrophic forgetting problems into two folds: relevant concepts forgetting and previous concepts forgetting. To address these challenges, we first devise a data-free knowledge distillation strategy to tackle relevant concepts forgetting. Unlike existing methods that rely on additional real data or offline replay of original concept data, our approach enables on-the-fly knowledge distillation to retain the previous concepts while learning new ones, without accessing any previous data. Second, we develop an In-Context Generation (ICGen) paradigm that allows the diffusion model to be conditioned upon the input vision context, which facilitates the few-shot generation and mitigates the issue of previous concepts forgetting. Extensive experiments show that the proposed Lifelong Few-Shot Diffusion (LFS-Diffusion) method can produce high-quality and accurate images while maintaining previously learned knowledge.

Title: Assessing the Answerability of Queries in Retrieval-Augmented Code Generation

• Authors: Geonmin Kim, Jaeyeon Kim, Hancheol Park, Wooksu Shin, Tae-Ho Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05547
• Pdf URL: https://arxiv.org/pdf/2411.05547
• Copy Paste: [[2411.05547]] Assessing the Answerability of Queries in Retrieval-Augmented Code Generation(https://arxiv.org/abs/2411.05547)
• Keywords: large language model
• Abstract: Thanks to unprecedented language understanding and generation capabilities of large language model (LLM), Retrieval-augmented Code Generation (RaCG) has recently been widely utilized among software developers. While this has increased productivity, there are still frequent instances of incorrect codes being provided. In particular, there are cases where plausible yet incorrect codes are generated for queries from users that cannot be answered with the given queries and API descriptions. This study proposes a task for evaluating answerability, which assesses whether valid answers can be generated based on users' queries and retrieved APIs in RaCG. Additionally, we build a benchmark dataset called Retrieval-augmented Code Generability Evaluation (RaCGEval) to evaluate the performance of models performing this task. Experimental results show that this task remains at a very challenging level, with baseline models exhibiting a low performance of 46.7%. Furthermore, this study discusses methods that could significantly improve performance.

Title: DeepArUco++: Improved detection of square fiducial markers in challenging lighting conditions

• Authors: Rafael Berral-Soler, Rafael Muñoz-Salinas, Rafael Medina-Carnicer, Manuel J. Marín-Jiménez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05552
• Pdf URL: https://arxiv.org/pdf/2411.05552
• Copy Paste: [[2411.05552]] DeepArUco++: Improved detection of square fiducial markers in challenging lighting conditions(https://arxiv.org/abs/2411.05552)
• Keywords: robust
• Abstract: Fiducial markers are a computer vision tool used for object pose estimation and detection. These markers are highly useful in fields such as industry, medicine and logistics. However, optimal lighting conditions are not always available,and other factors such as blur or sensor noise can affect image quality. Classical computer vision techniques that precisely locate and decode fiducial markers often fail under difficult illumination conditions (e.g. extreme variations of lighting within the same frame). Hence, we propose DeepArUco++, a deep learning-based framework that leverages the robustness of Convolutional Neural Networks to perform marker detection and decoding in challenging lighting conditions. The framework is based on a pipeline using different Neural Network models at each step, namely marker detection, corner refinement and marker decoding. Additionally, we propose a simple method for generating synthetic data for training the different models that compose the proposed pipeline, and we present a second, real-life dataset of ArUco markers in challenging lighting conditions used to evaluate our system. The developed method outperforms other state-of-the-art methods in such tasks and remains competitive even when testing on the datasets used to develop those methods. Code available in GitHub: this https URL

Title: Open-set object detection: towards unified problem formulation and benchmarking

• Authors: Hejer Ammar, Nikita Kiselov, Guillaume Lapouge, Romaric Audigier
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05564
• Pdf URL: https://arxiv.org/pdf/2411.05564
• Copy Paste: [[2411.05564]] Open-set object detection: towards unified problem formulation and benchmarking(https://arxiv.org/abs/2411.05564)
• Keywords: transformer
• Abstract: In real-world applications where confidence is key, like autonomous driving, the accurate detection and appropriate handling of classes differing from those used during training are crucial. Despite the proposal of various unknown object detection approaches, we have observed widespread inconsistencies among them regarding the datasets, metrics, and scenarios used, alongside a notable absence of a clear definition for unknown objects, which hampers meaningful evaluation. To counter these issues, we introduce two benchmarks: a unified VOC-COCO evaluation, and the new OpenImagesRoad benchmark which provides clear hierarchical object definition besides new evaluation metrics. Complementing the benchmark, we exploit recent self-supervised Vision Transformers performance, to improve pseudo-labeling-based OpenSet Object Detection (OSOD), through OW-DETR++. State-of-the-art methods are extensively evaluated on the proposed benchmarks. This study provides a clear problem definition, ensures consistent evaluations, and draws new conclusions about effectiveness of OSOD strategies.

Title: Obfuscation as Instruction Decorrelation

• Authors: Ali Ajorian, Erick Lavoie, Christian Tschudin
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2411.05570
• Pdf URL: https://arxiv.org/pdf/2411.05570
• Copy Paste: [[2411.05570]] Obfuscation as Instruction Decorrelation(https://arxiv.org/abs/2411.05570)
• Keywords: secure, protect
• Abstract: Obfuscation of computer programs has historically been approached either as a practical but \textit{ad hoc} craft to make reverse engineering subjectively difficult, or as a sound theoretical investigation unfortunately detached from the numerous existing constraints of engineering practical systems. In this paper, we propose \textit{instruction decorrelation} as a new approach that makes the instructions of a set of real-world programs appear independent from one another. We contribute: a formal definition of \textit{instruction independence} with multiple instantiations for various aspects of programs; a combination of program transformations that meet the corresponding instances of instruction independence against an honest-but-curious adversary, specifically random interleaving and memory access obfuscation; and an implementation of an interpreter that uses a trusted execution environment (TEE) only to perform memory address translation and memory shuffling, leaving instructions execution outside the TEE. These first steps highlight the practicality of our approach. Combined with additional techniques to protect the content of memory and to hopefully lower the requirements on TEEs, this work could potentially lead to more secure obfuscation techniques that could execute on commonly available hardware.

Title: Evaluating and Adapting Large Language Models to Represent Folktales in Low-Resource Languages

• Authors: JA Meaney, Beatrice Alex, William Lamb
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05593
• Pdf URL: https://arxiv.org/pdf/2411.05593
• Copy Paste: [[2411.05593]] Evaluating and Adapting Large Language Models to Represent Folktales in Low-Resource Languages(https://arxiv.org/abs/2411.05593)
• Keywords: large language model
• Abstract: Folktales are a rich resource of knowledge about the society and culture of a civilisation. Digital folklore research aims to use automated techniques to better understand these folktales, and it relies on abstract representations of the textual data. Although a number of large language models (LLMs) claim to be able to represent low-resource langauges such as Irish and Gaelic, we present two classification tasks to explore how useful these representations are, and three adaptations to improve the performance of these models. We find that adapting the models to work with longer sequences, and continuing pre-training on the domain of folktales improves classification performance, although these findings are tempered by the impressive performance of a baseline SVM with non-contextual features.

Title: A Two-Step Concept-Based Approach for Enhanced Interpretability and Trust in Skin Lesion Diagnosis

• Authors: Cristiano Patrício, Luís F. Teixeira, João C. Neves
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05609
• Pdf URL: https://arxiv.org/pdf/2411.05609
• Copy Paste: [[2411.05609]] A Two-Step Concept-Based Approach for Enhanced Interpretability and Trust in Skin Lesion Diagnosis(https://arxiv.org/abs/2411.05609)
• Keywords: interpretability, large language model
• Abstract: The main challenges hindering the adoption of deep learning-based systems in clinical settings are the scarcity of annotated data and the lack of interpretability and trust in these systems. Concept Bottleneck Models (CBMs) offer inherent interpretability by constraining the final disease prediction on a set of human-understandable concepts. However, this inherent interpretability comes at the cost of greater annotation burden. Additionally, adding new concepts requires retraining the entire system. In this work, we introduce a novel two-step methodology that addresses both of these challenges. By simulating the two stages of a CBM, we utilize a pretrained Vision Language Model (VLM) to automatically predict clinical concepts, and a Large Language Model (LLM) to generate disease diagnoses based on the predicted concepts. We validate our approach on three skin lesion datasets, demonstrating that it outperforms traditional CBMs and state-of-the-art explainable methods, all without requiring any training and utilizing only a few annotated examples. The code is available at this https URL.

Title: WHALE: Towards Generalizable and Scalable World Models for Embodied Decision-making

• Authors: Zhilong Zhang, Ruifeng Chen, Junyin Ye, Yihao Sun, Pengyuan Wang, Jingcheng Pang, Kaiyuan Li, Tianshuo Liu, Haoxin Lin, Yang Yu, Zhi-Hua Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05619
• Pdf URL: https://arxiv.org/pdf/2411.05619
• Copy Paste: [[2411.05619]] WHALE: Towards Generalizable and Scalable World Models for Embodied Decision-making(https://arxiv.org/abs/2411.05619)
• Keywords: transformer
• Abstract: World models play a crucial role in decision-making within embodied environments, enabling cost-free explorations that would otherwise be expensive in the real world. To facilitate effective decision-making, world models must be equipped with strong generalizability to support faithful imagination in out-of-distribution (OOD) regions and provide reliable uncertainty estimation to assess the credibility of the simulated experiences, both of which present significant challenges for prior scalable approaches. This paper introduces WHALE, a framework for learning generalizable world models, consisting of two key techniques: behavior-conditioning and retracing-rollout. Behavior-conditioning addresses the policy distribution shift, one of the primary sources of the world model generalization error, while retracing-rollout enables efficient uncertainty estimation without the necessity of model ensembles. These techniques are universal and can be combined with any neural network architecture for world model learning. Incorporating these two techniques, we present Whale-ST, a scalable spatial-temporal transformer-based world model with enhanced generalizability. We demonstrate the superiority of Whale-ST in simulation tasks by evaluating both value estimation accuracy and video generation fidelity. Additionally, we examine the effectiveness of our uncertainty estimation technique, which enhances model-based policy optimization in fully offline scenarios. Furthermore, we propose Whale-X, a 414M parameter world model trained on 970K trajectories from Open X-Embodiment datasets. We show that Whale-X exhibits promising scalability and strong generalizability in real-world manipulation scenarios using minimal demonstrations.

Title: SynDroneVision: A Synthetic Dataset for Image-Based Drone Detection

• Authors: Tamara R. Lenhard, Andreas Weinmann, Kai Franke, Tobias Koch
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.05633
• Pdf URL: https://arxiv.org/pdf/2411.05633
• Copy Paste: [[2411.05633]] SynDroneVision: A Synthetic Dataset for Image-Based Drone Detection(https://arxiv.org/abs/2411.05633)
• Keywords: robust
• Abstract: Developing robust drone detection systems is often constrained by the limited availability of large-scale annotated training data and the high costs associated with real-world data collection. However, leveraging synthetic data generated via game engine-based simulations provides a promising and cost-effective solution to overcome this issue. Therefore, we present SynDroneVision, a synthetic dataset specifically designed for RGB-based drone detection in surveillance applications. Featuring diverse backgrounds, lighting conditions, and drone models, SynDroneVision offers a comprehensive training foundation for deep learning algorithms. To evaluate the dataset's effectiveness, we perform a comparative analysis across a selection of recent YOLO detection models. Our findings demonstrate that SynDroneVision is a valuable resource for real-world data enrichment, achieving notable enhancements in model performance and robustness, while significantly reducing the time and costs of real-world data acquisition. SynDroneVision will be publicly released upon paper acceptance.

Title: Video RWKV:Video Action Recognition Based RWKV

• Authors: Zhuowen Yin, Chengru Li, Xingbo Dong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05636
• Pdf URL: https://arxiv.org/pdf/2411.05636
• Copy Paste: [[2411.05636]] Video RWKV:Video Action Recognition Based RWKV(https://arxiv.org/abs/2411.05636)
• Keywords: transformer
• Abstract: To address the challenges of high computational costs and long-distance dependencies in exist ing video understanding methods, such as CNNs and Transformers, this work introduces RWKV to the video domain in a novel way. We propose a LSTM CrossRWKV (LCR) framework, designed for spatiotemporal representation learning to tackle the video understanding task. Specifically, the proposed linear complexity LCR incorporates a novel Cross RWKV gate to facilitate interaction be tween current frame edge information and past features, enhancing the focus on the subject through edge features and globally aggregating inter-frame features over time. LCR stores long-term mem ory for video processing through an enhanced LSTM recurrent execution mechanism. By leveraging the Cross RWKV gate and recurrent execution, LCR effectively captures both spatial and temporal features. Additionally, the edge information serves as a forgetting gate for LSTM, guiding long-term memory this http URL masking strategy reduces redundant information in food and reduces this http URL advantages enable LSTM CrossRWKV to set a new benchmark in video under standing, offering a scalable and efficient solution for comprehensive video analysis. All code and models are publicly available.

Title: Assessing Open-Source Large Language Models on Argumentation Mining Subtasks

• Authors: Mohammad Yeghaneh Abkenar, Weixing Wang, Hendrik Graupner, Manfred Stede
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05639
• Pdf URL: https://arxiv.org/pdf/2411.05639
• Copy Paste: [[2411.05639]] Assessing Open-Source Large Language Models on Argumentation Mining Subtasks(https://arxiv.org/abs/2411.05639)
• Keywords: large language model
• Abstract: We explore the capability of four open-sourcelarge language models (LLMs) in argumentation mining (AM). We conduct experiments on three different corpora; persuasive essays(PE), argumentative microtexts (AMT) Part 1 and Part 2, based on two argumentation mining sub-tasks: (i) argumentative discourse units classifications (ADUC), and (ii) argumentative relation classification (ARC). This work aims to assess the argumentation capability of open-source LLMs, including Mistral 7B, Mixtral8x7B, LlamA2 7B and LlamA3 8B in both, zero-shot and few-shot scenarios. Our analysis contributes to further assessing computational argumentation with open-source LLMs in future research efforts.

Title: Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation

• Authors: Long Truong To, Hung Tuan Le, Dat Van-Thanh Nguyen, Manh Trong Nguyen, Tri Thien Nguyen, Tin Van Huynh, Kiet Van Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05641
• Pdf URL: https://arxiv.org/pdf/2411.05641
• Copy Paste: [[2411.05641]] Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation(https://arxiv.org/abs/2411.05641)
• Keywords: large language model
• Abstract: Large Language Models (LLMs), with gradually improving reading comprehension and reasoning capabilities, are being applied to a range of complex language tasks, including the automatic generation of language data for various purposes. However, research on applying LLMs for automatic data generation in low-resource languages like Vietnamese is still underdeveloped and lacks comprehensive evaluation. In this paper, we explore the use of LLMs for automatic data generation for the Vietnamese fact-checking task, which faces significant data limitations. Specifically, we focus on fact-checking data where claims are synthesized from multiple evidence sentences to assess the information synthesis capabilities of LLMs. We develop an automatic data construction process using simple prompt techniques on LLMs and explore several methods to improve the quality of the generated data. To evaluate the quality of the data generated by LLMs, we conduct both manual quality assessments and performance evaluations using language models. Experimental results and manual evaluations illustrate that while the quality of the generated data has significantly improved through fine-tuning techniques, LLMs still cannot match the data quality produced by humans.

Title: Enhancing Model Fairness and Accuracy with Similarity Networks: A Methodological Approach

• Authors: Samira Maghool, Paolo Ceravolo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05648
• Pdf URL: https://arxiv.org/pdf/2411.05648
• Copy Paste: [[2411.05648]] Enhancing Model Fairness and Accuracy with Similarity Networks: A Methodological Approach(https://arxiv.org/abs/2411.05648)
• Keywords: fair
• Abstract: In this paper, we propose an innovative approach to thoroughly explore dataset features that introduce bias in downstream machine-learning tasks. Depending on the data format, we use different techniques to map instances into a similarity feature space. Our method's ability to adjust the resolution of pairwise similarity provides clear insights into the relationship between the dataset classification complexity and model fairness. Experimental results confirm the promising applicability of the similarity network in promoting fair models. Moreover, leveraging our methodology not only seems promising in providing a fair downstream task such as classification, it also performs well in imputation and augmentation of the dataset satisfying the fairness criteria such as demographic parity and imbalanced classes.

Title: Towards a Re-evaluation of Data Forging Attacks in Practice

• Authors: Mohamed Suliman, Anisa Halimi, Swanand Kadhe, Nathalie Baracaldo, Douglas Leith
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05658
• Pdf URL: https://arxiv.org/pdf/2411.05658
• Copy Paste: [[2411.05658]] Towards a Re-evaluation of Data Forging Attacks in Practice(https://arxiv.org/abs/2411.05658)
• Keywords: privacy, attack
• Abstract: Data forging attacks provide counterfactual proof that a model was trained on a given dataset, when in fact, it was trained on another. These attacks work by forging (replacing) mini-batches with ones containing distinct training examples that produce nearly identical gradients. Data forging appears to break any potential avenues for data governance, as adversarial model owners may forge their training set from a dataset that is not compliant to one that is. Given these serious implications on data auditing and compliance, we critically analyse data forging from both a practical and theoretical point of view, finding that a key practical limitation of current attack methods makes them easily detectable by a verifier; namely that they cannot produce sufficiently identical gradients. Theoretically, we analyse the question of whether two distinct mini-batches can produce the same gradient. Generally, we find that while there may exist an infinite number of distinct mini-batches with real-valued training examples and labels that produce the same gradient, finding those that are within the allowed domain e.g. pixel values between 0-255 and one hot labels is a non trivial task. Our results call for the reevaluation of the strength of existing attacks, and for additional research into successful data forging, given the serious consequences it may have on machine learning and privacy.

Title: Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation

• Authors: Xiwen Wei, Guihong Li, Radu Marculescu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05663
• Pdf URL: https://arxiv.org/pdf/2411.05663
• Copy Paste: [[2411.05663]] Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation(https://arxiv.org/abs/2411.05663)
• Keywords: privacy, robust, transformer
• Abstract: Catastrophic forgetting is a significant challenge in online continual learning (OCL), especially for non-stationary data streams that do not have well-defined task boundaries. This challenge is exacerbated by the memory constraints and privacy concerns inherent in rehearsal buffers. To tackle catastrophic forgetting, in this paper, we introduce Online-LoRA, a novel framework for task-free OCL. Online-LoRA allows to finetune pre-trained Vision Transformer (ViT) models in real-time to address the limitations of rehearsal buffers and leverage pre-trained models' performance benefits. As the main contribution, our approach features a novel online weight regularization strategy to identify and consolidate important model parameters. Moreover, Online-LoRA leverages the training dynamics of loss values to enable the automatic recognition of the data distribution shifts. Extensive experiments across many task-free OCL scenarios and benchmark datasets (including CIFAR-100, ImageNet-R, ImageNet-S, CUB-200 and CORe50) demonstrate that Online-LoRA can be robustly adapted to various ViT architectures, while achieving better performance compared to SOTA methods. Our code will be publicly available at: this https URL.

Title: Unmasking the Limits of Large Language Models: A Systematic Evaluation of Masked Text Processing Ability through MskQA and MskCal

• Authors: Fuka Matsuzaki, Haru-Tada Sato
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05665
• Pdf URL: https://arxiv.org/pdf/2411.05665
• Copy Paste: [[2411.05665]] Unmasking the Limits of Large Language Models: A Systematic Evaluation of Masked Text Processing Ability through MskQA and MskCal(https://arxiv.org/abs/2411.05665)
• Keywords: robust, large language model
• Abstract: This paper sheds light on the limitations of Large Language Models (LLMs) by rigorously evaluating their ability to process masked text. We introduce two novel tasks: MskQA, measuring reasoning on masked question-answering datasets like RealtimeQA, and MskCal, assessing numerical reasoning on masked arithmetic this http URL GPT-4o and 4o-mini reveals that while LLMs exhibit some resilience to masked text, their performance is highly contingent on masking rates and semantic cues. Specifically, "solid masking," where semantic clues are entirely absent, leads to a significant performance drop compared to "partial lifting," where some semantic information is retained, indicating LLMs' reliance on surface-level patterns. Interestingly, GPT-4o consistently outperforms 4o-mini, particularly in MskCal, demonstrating a greater ability to handle numerical reasoning with masked text. This underscores the crucial role of semantic cues in the reasoning process of LLMs. Our study illuminates the interplay between background knowledge and reasoning ability in masked text processing, paving the way for a deeper understanding of LLM capabilities and limitations, and highlighting the need for more robust evaluation methods to accurately assess their true comprehension abilities.

Title: Improving Molecular Graph Generation with Flow Matching and Optimal Transport

• Authors: Xiaoyang Hou, Tian Zhu, Milong Ren, Dongbo Bu, Xin Gao, Chunming Zhang, Shiwei Sun
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05676
• Pdf URL: https://arxiv.org/pdf/2411.05676
• Copy Paste: [[2411.05676]] Improving Molecular Graph Generation with Flow Matching and Optimal Transport(https://arxiv.org/abs/2411.05676)
• Keywords: diffusion, transformer, generative
• Abstract: Generating molecular graphs is crucial in drug design and discovery but remains challenging due to the complex interdependencies between nodes and edges. While diffusion models have demonstrated their potentiality in molecular graph design, they often suffer from unstable training and inefficient sampling. To enhance generation performance and training stability, we propose GGFlow, a discrete flow matching generative model incorporating optimal transport for molecular graphs and it incorporates an edge-augmented graph transformer to enable the direct communications among chemical bounds. Additionally, GGFlow introduces a novel goal-guided generation framework to control the generative trajectory of our model, aiming to design novel molecular structures with the desired properties. GGFlow demonstrates superior performance on both unconditional and conditional molecule generation tasks, outperforming existing baselines and underscoring its effectiveness and potential for wider application.

Title: Tell What You Hear From What You See -- Video to Audio Generation Through Text

• Authors: Xiulong Liu, Kun Su, Eli Shlizerman
• Subjects: cs.CV, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.05679
• Pdf URL: https://arxiv.org/pdf/2411.05679
• Copy Paste: [[2411.05679]] Tell What You Hear From What You See -- Video to Audio Generation Through Text(https://arxiv.org/abs/2411.05679)
• Keywords: transformer, generative
• Abstract: The content of visual and audio scenes is multi-faceted such that a video can be paired with various audio and vice-versa. Thereby, in video-to-audio generation task, it is imperative to introduce steering approaches for controlling the generated audio. While Video-to-Audio generation is a well-established generative task, existing methods lack such controllability. In this work, we propose VATT, a multi-modal generative framework that takes a video and an optional text prompt as input, and generates audio and optional textual description of the audio. Such a framework has two advantages: i) Video-to-Audio generation process can be refined and controlled via text which complements the context of visual information, and ii) The model can suggest what audio to generate for the video by generating audio captions. VATT consists of two key modules: VATT Converter, a LLM that is fine-tuned for instructions and includes a projection layer that maps video features to the LLM vector space; and VATT Audio, a transformer that generates audio tokens from visual frames and from optional text prompt using iterative parallel decoding. The audio tokens are converted to a waveform by pretrained neural codec. Experiments show that when VATT is compared to existing video-to-audio generation methods in objective metrics, it achieves competitive performance when the audio caption is not provided. When the audio caption is provided as a prompt, VATT achieves even more refined performance (lowest KLD score of 1.41). Furthermore, subjective studies show that VATT Audio has been chosen as preferred generated audio than audio generated by existing methods. VATT enables controllable video-to-audio generation through text as well as suggesting text prompts for videos through audio captions, unlocking novel applications such as text-guided video-to-audio generation and video-to-audio captioning.

Title: A Survey of AI-Related Cyber Security Risks and Countermeasures in Mobility-as-a-Service

• Authors: Kai-Fung Chu, Haiyue Yuan, Jinsheng Yuan, Weisi Guo, Nazmiye Balta-Ozkan, Shujun Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05681
• Pdf URL: https://arxiv.org/pdf/2411.05681
• Copy Paste: [[2411.05681]] A Survey of AI-Related Cyber Security Risks and Countermeasures in Mobility-as-a-Service(https://arxiv.org/abs/2411.05681)
• Keywords: security, privacy, attack, extraction
• Abstract: Mobility-as-a-Service (MaaS) integrates different transport modalities and can support more personalisation of travellers' journey planning based on their individual preferences, behaviours and wishes. To fully achieve the potential of MaaS, a range of AI (including machine learning and data mining) algorithms are needed to learn personal requirements and needs, to optimise journey planning of each traveller and all travellers as a whole, to help transport service operators and relevant governmental bodies to operate and plan their services, and to detect and prevent cyber attacks from various threat actors including dishonest and malicious travellers and transport operators. The increasing use of different AI and data processing algorithms in both centralised and distributed settings opens the MaaS ecosystem up to diverse cyber and privacy attacks at both the AI algorithm level and the connectivity surfaces. In this paper, we present the first comprehensive review on the coupling between AI-driven MaaS design and the diverse cyber security challenges related to cyber attacks and countermeasures. In particular, we focus on how current and emerging AI-facilitated privacy risks (profiling, inference, and third-party threats) and adversarial AI attacks (evasion, extraction, and gamification) may impact the MaaS ecosystem. These risks often combine novel attacks (e.g., inverse learning) with traditional attack vectors (e.g., man-in-the-middle attacks), exacerbating the risks for the wider participation actors and the emergence of new business models.

Title: Autoregressive Adaptive Hypergraph Transformer for Skeleton-based Activity Recognition

• Authors: Abhisek Ray, Ayush Raj, Maheshkumar H. Kolekar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05692
• Pdf URL: https://arxiv.org/pdf/2411.05692
• Copy Paste: [[2411.05692]] Autoregressive Adaptive Hypergraph Transformer for Skeleton-based Activity Recognition(https://arxiv.org/abs/2411.05692)
• Keywords: robust, transformer
• Abstract: Extracting multiscale contextual information and higher-order correlations among skeleton sequences using Graph Convolutional Networks (GCNs) alone is inadequate for effective action classification. Hypergraph convolution addresses the above issues but cannot harness the long-range dependencies. Transformer proves to be effective in capturing these dependencies and making complex contextual features accessible. We propose an Autoregressive Adaptive HyperGraph Transformer (AutoregAd-HGformer) model for in-phase (autoregressive and discrete) and out-phase (adaptive) hypergraph generation. The vector quantized in-phase hypergraph equipped with powerful autoregressive learned priors produces a more robust and informative representation suitable for hyperedge formation. The out-phase hypergraph generator provides a model-agnostic hyperedge learning technique to align the attributes with input skeleton embedding. The hybrid (supervised and unsupervised) learning in AutoregAd-HGformer explores the action-dependent feature along spatial, temporal, and channel dimensions. The extensive experimental results and ablation study indicate the superiority of our model over state-of-the-art hypergraph architectures on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.

Title: Visual-TCAV: Concept-based Attribution and Saliency Maps for Post-hoc Explainability in Image Classification

• Authors: Antonio De Santis, Riccardo Campi, Matteo Bianchi, Marco Brambilla
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05698
• Pdf URL: https://arxiv.org/pdf/2411.05698
• Copy Paste: [[2411.05698]] Visual-TCAV: Concept-based Attribution and Saliency Maps for Post-hoc Explainability in Image Classification(https://arxiv.org/abs/2411.05698)
• Keywords: explainability
• Abstract: Convolutional Neural Networks (CNNs) have seen significant performance improvements in recent years. However, due to their size and complexity, they function as black-boxes, leading to transparency concerns. State-of-the-art saliency methods generate local explanations that highlight the area in the input image where a class is identified but cannot explain how a concept of interest contributes to the prediction, which is essential for bias mitigation. On the other hand, concept-based methods, such as TCAV (Testing with Concept Activation Vectors), provide insights into how sensitive is the network to a concept, but cannot compute its attribution in a specific prediction nor show its location within the input image. This paper introduces a novel post-hoc explainability framework, Visual-TCAV, which aims to bridge the gap between these methods by providing both local and global explanations for CNN-based image classification. Visual-TCAV uses Concept Activation Vectors (CAVs) to generate saliency maps that show where concepts are recognized by the network. Moreover, it can estimate the attribution of these concepts to the output of any class using a generalization of Integrated Gradients. This framework is evaluated on popular CNN architectures, with its validity further confirmed via experiments where ground truth for explanations is known, and a comparison with TCAV. Our code will be made available soon.

Title: Image inpainting enhancement by replacing the original mask with a self-attended region from the input image

• Authors: Kourosh Kiani, Razieh Rastgoo, Alireza Chaji, Sergio Escalera
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.05705
• Pdf URL: https://arxiv.org/pdf/2411.05705
• Copy Paste: [[2411.05705]] Image inpainting enhancement by replacing the original mask with a self-attended region from the input image(https://arxiv.org/abs/2411.05705)
• Keywords: transformer
• Abstract: Image inpainting, the process of restoring missing or corrupted regions of an image by reconstructing pixel information, has recently seen considerable advancements through deep learning-based approaches. In this paper, we introduce a novel deep learning-based pre-processing methodology for image inpainting utilizing the Vision Transformer (ViT). Our approach involves replacing masked pixel values with those generated by the ViT, leveraging diverse visual patches within the attention matrix to capture discriminative spatial features. To the best of our knowledge, this is the first instance of such a pre-processing model being proposed for image inpainting tasks. Furthermore, we show that our methodology can be effectively applied using the pre-trained ViT model with pre-defined patch size. To evaluate the generalization capability of the proposed methodology, we provide experimental results comparing our approach with four standard models across four public datasets, demonstrating the efficacy of our pre-processing technique in enhancing inpainting performance.

Title: Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models

• Authors: Jia-Hong Huang, Hongyi Zhu, Yixian Shen, Stevan Rudinac, Evangelos Kanoulas
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05706
• Pdf URL: https://arxiv.org/pdf/2411.05706
• Copy Paste: [[2411.05706]] Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models(https://arxiv.org/abs/2411.05706)
• Keywords: diffusion
• Abstract: Evaluating the quality of automatically generated image descriptions is a complex task that requires metrics capturing various dimensions, such as grammaticality, coverage, accuracy, and truthfulness. Although human evaluation provides valuable insights, its cost and time-consuming nature pose limitations. Existing automated metrics like BLEU, ROUGE, METEOR, and CIDEr attempt to fill this gap, but they often exhibit weak correlations with human judgment. To address this challenge, we propose a novel evaluation framework called Image2Text2Image, which leverages diffusion models, such as Stable Diffusion or DALL-E, for text-to-image generation. In the Image2Text2Image framework, an input image is first processed by a selected image captioning model, chosen for evaluation, to generate a textual description. Using this generated description, a diffusion model then creates a new image. By comparing features extracted from the original and generated images, we measure their similarity using a designated similarity metric. A high similarity score suggests that the model has produced a faithful textual description, while a low score highlights discrepancies, revealing potential weaknesses in the model's performance. Notably, our framework does not rely on human-annotated reference captions, making it a valuable tool for assessing image captioning models. Extensive experiments and human evaluations validate the efficacy of our proposed Image2Text2Image evaluation framework. The code and dataset will be published to support further research in the community.

Title: Sample and Computationally Efficient Robust Learning of Gaussian Single-Index Models

• Authors: Puqian Wang, Nikos Zarifis, Ilias Diakonikolas, Jelena Diakonikolas
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05708
• Pdf URL: https://arxiv.org/pdf/2411.05708
• Copy Paste: [[2411.05708]] Sample and Computationally Efficient Robust Learning of Gaussian Single-Index Models(https://arxiv.org/abs/2411.05708)
• Keywords: robust
• Abstract: A single-index model (SIM) is a function of the form $\sigma(\mathbf{w}^{\ast} \cdot \mathbf{x})$, where $\sigma: \mathbb{R} \to \mathbb{R}$ is a known link function and $\mathbf{w}^{\ast}$ is a hidden unit vector. We study the task of learning SIMs in the agnostic (a.k.a. adversarial label noise) model with respect to the $L^2_2$-loss under the Gaussian distribution. Our main result is a sample and computationally efficient agnostic proper learner that attains $L^2_2$-error of $O(\mathrm{OPT})+\epsilon$, where $\mathrm{OPT}$ is the optimal loss. The sample complexity of our algorithm is $\tilde{O}(d^{\lceil k^{\ast}/2\rceil}+d/\epsilon)$, where $k^{\ast}$ is the information-exponent of $\sigma$ corresponding to the degree of its first non-zero Hermite coefficient. This sample bound nearly matches known CSQ lower bounds, even in the realizable setting. Prior algorithmic work in this setting had focused on learning in the realizable case or in the presence of semi-random noise. Prior computationally efficient robust learners required significantly stronger assumptions on the link function.

Title: STARS: Sensor-agnostic Transformer Architecture for Remote Sensing

• Authors: Ethan King, Jaime Rodriguez, Diego Llanes, Timothy Doster, Tegan Emerson, James Koch
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.05714
• Pdf URL: https://arxiv.org/pdf/2411.05714
• Copy Paste: [[2411.05714]] STARS: Sensor-agnostic Transformer Architecture for Remote Sensing(https://arxiv.org/abs/2411.05714)
• Keywords: transformer
• Abstract: We present a sensor-agnostic spectral transformer as the basis for spectral foundation models. To that end, we introduce a Universal Spectral Representation (USR) that leverages sensor meta-data, such as sensing kernel specifications and sensing wavelengths, to encode spectra obtained from any spectral instrument into a common representation, such that a single model can ingest data from any sensor. Furthermore, we develop a methodology for pre-training such models in a self-supervised manner using a novel random sensor-augmentation and reconstruction pipeline to learn spectral features independent of the sensing paradigm. We demonstrate that our architecture can learn sensor independent spectral features that generalize effectively to sensors not seen during training. This work sets the stage for training foundation models that can both leverage and be effective for the growing diversity of spectral data.

Title: PEP-GS: Perceptually-Enhanced Precise Structured 3D Gaussians for View-Adaptive Rendering

• Authors: Junxi Jin, Xiulai Li, Haiping Huang, Lianjun Liu, Yujie Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05731
• Pdf URL: https://arxiv.org/pdf/2411.05731
• Copy Paste: [[2411.05731]] PEP-GS: Perceptually-Enhanced Precise Structured 3D Gaussians for View-Adaptive Rendering(https://arxiv.org/abs/2411.05731)
• Keywords: interpretability
• Abstract: Recent advances in structured 3D Gaussians for view-adaptive rendering, particularly through methods like Scaffold-GS, have demonstrated promising results in neural scene representation. However, existing approaches still face challenges in perceptual consistency and precise view-dependent effects. We present PEP-GS, a novel framework that enhances structured 3D Gaussians through three key innovations: (1) a Local-Enhanced Multi-head Self-Attention (LEMSA) mechanism that replaces spherical harmonics for more accurate view-dependent color decoding, and (2) Kolmogorov-Arnold Networks (KAN) that optimize Gaussian opacity and covariance functions for enhanced interpretability and splatting precision. (3) a Neural Laplacian Pyramid Decomposition (NLPD) that improves perceptual similarity across views. Our comprehensive evaluation across multiple datasets indicates that, compared to the current state-of-the-art methods, these improvements are particularly evident in challenging scenarios such as view-dependent effects, specular reflections, fine-scale details and false geometry generation.

Title: Differential Privacy Under Class Imbalance: Methods and Empirical Insights

• Authors: Lucas Rosenblatt, Yuliia Lut, Eitan Turok, Marco Avella-Medina, Rachel Cummings
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05733
• Pdf URL: https://arxiv.org/pdf/2411.05733
• Copy Paste: [[2411.05733]] Differential Privacy Under Class Imbalance: Methods and Empirical Insights(https://arxiv.org/abs/2411.05733)
• Keywords: privacy, protect
• Abstract: Imbalanced learning occurs in classification settings where the distribution of class-labels is highly skewed in the training data, such as when predicting rare diseases or in fraud detection. This class imbalance presents a significant algorithmic challenge, which can be further exacerbated when privacy-preserving techniques such as differential privacy are applied to protect sensitive training data. Our work formalizes these challenges and provides a number of algorithmic solutions. We consider DP variants of pre-processing methods that privately augment the original dataset to reduce the class imbalance; these include oversampling, SMOTE, and private synthetic data generation. We also consider DP variants of in-processing techniques, which adjust the learning algorithm to account for the imbalance; these include model bagging, class-weighted empirical risk minimization and class-weighted deep learning. For each method, we either adapt an existing imbalanced learning technique to the private setting or demonstrate its incompatibility with differential privacy. Finally, we empirically evaluate these privacy-preserving imbalanced learning methods under various data and distributional settings. We find that private synthetic data methods perform well as a data pre-processing step, while class-weighted ERMs are an alternative in higher-dimensional settings where private synthetic data suffers from the curse of dimensionality.

Title: StdGEN: Semantic-Decomposed 3D Character Generation from Single Images

• Authors: Yuze He, Yanning Zhou, Wang Zhao, Zhongkai Wu, Kaiwen Xiao, Wei Yang, Yong-Jin Liu, Xiao Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05738
• Pdf URL: https://arxiv.org/pdf/2411.05738
• Copy Paste: [[2411.05738]] StdGEN: Semantic-Decomposed 3D Character Generation from Single Images(https://arxiv.org/abs/2411.05738)
• Keywords: extraction, diffusion, transformer
• Abstract: We present StdGEN, an innovative pipeline for generating semantically decomposed high-quality 3D characters from single images, enabling broad applications in virtual reality, gaming, and filmmaking, etc. Unlike previous methods which struggle with limited decomposability, unsatisfactory quality, and long optimization times, StdGEN features decomposability, effectiveness and efficiency; i.e., it generates intricately detailed 3D characters with separated semantic components such as the body, clothes, and hair, in three minutes. At the core of StdGEN is our proposed Semantic-aware Large Reconstruction Model (S-LRM), a transformer-based generalizable model that jointly reconstructs geometry, color and semantics from multi-view images in a feed-forward manner. A differentiable multi-layer semantic surface extraction scheme is introduced to acquire meshes from hybrid implicit fields reconstructed by our S-LRM. Additionally, a specialized efficient multi-view diffusion model and an iterative multi-layer surface refinement module are integrated into the pipeline to facilitate high-quality, decomposable 3D character generation. Extensive experiments demonstrate our state-of-the-art performance in 3D anime character generation, surpassing existing baselines by a significant margin in geometry, texture and decomposability. StdGEN offers ready-to-use semantic-decomposed 3D characters and enables flexible customization for a wide range of applications. Project page: this https URL

Title: Topology-aware Reinforcement Feature Space Reconstruction for Graph Data

• Authors: Wangyang Ying, Haoyue Bai, Kunpeng Liu, Yanjie Fu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05742
• Pdf URL: https://arxiv.org/pdf/2411.05742
• Copy Paste: [[2411.05742]] Topology-aware Reinforcement Feature Space Reconstruction for Graph Data(https://arxiv.org/abs/2411.05742)
• Keywords: extraction
• Abstract: Feature space is an environment where data points are vectorized to represent the original dataset. Reconstructing a good feature space is essential to augment the AI power of data, improve model generalization, and increase the availability of downstream ML models. Existing literature, such as feature transformation and feature selection, is labor-intensive (e.g., heavy reliance on empirical experience) and mostly designed for tabular data. Moreover, these methods regard data samples as independent, which ignores the unique topological structure when applied to graph data, thus resulting in a suboptimal reconstruction feature space. Can we consider the topological information to automatically reconstruct feature space for graph data without heavy experiential knowledge? To fill this gap, we leverage topology-aware reinforcement learning to automate and optimize feature space reconstruction for graph data. Our approach combines the extraction of core subgraphs to capture essential structural information with a graph neural network (GNN) to encode topological features and reduce computing complexity. Then we introduce three reinforcement agents within a hierarchical structure to systematically generate meaningful features through an iterative process, effectively reconstructing the feature space. This framework provides a principled solution for attributed graph feature space reconstruction. The extensive experiments demonstrate the effectiveness and efficiency of including topological awareness.

Title: Free Record-Level Privacy Risk Evaluation Through Artifact-Based Methods

• Authors: Joseph Pollock, Igor Shilov, Euodia Dodd, Yves-Alexandre de Montjoye
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05743
• Pdf URL: https://arxiv.org/pdf/2411.05743
• Copy Paste: [[2411.05743]] Free Record-Level Privacy Risk Evaluation Through Artifact-Based Methods(https://arxiv.org/abs/2411.05743)
• Keywords: privacy, attack, robust, membership infer
• Abstract: Membership inference attacks (MIAs) are widely used to empirically assess the privacy risks of samples used to train a target machine learning model. State-of-the-art methods however require training hundreds of shadow models, with the same size and architecture of the target model, solely to evaluate the privacy risk. While one might be able to afford this for small models, the cost often becomes prohibitive for medium and large models. We here instead propose a novel approach to identify the at-risk samples using only artifacts available during training, with little to no additional computational overhead. Our method analyzes individual per-sample loss traces and uses them to identify the vulnerable data samples. We demonstrate the effectiveness of our artifact-based approach through experiments on the CIFAR10 dataset, showing high precision in identifying vulnerable samples as determined by a SOTA shadow model-based MIA (LiRA). Impressively, our method reaches the same precision as another SOTA MIA when measured against LiRA, despite it being orders of magnitude cheaper. We then show LT-IQR to outperform alternative loss aggregation methods, perform ablation studies on hyperparameters, and validate the robustness of our method to the target metric. Finally, we study the evolution of the vulnerability score distribution throughout training as a metric for model-level risk assessment.

Title: WavShadow: Wavelet Based Shadow Segmentation and Removal

• Authors: Shreyans Jain, Aadya Arora, Viraj Vekaria, Karan Gandhi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05747
• Pdf URL: https://arxiv.org/pdf/2411.05747
• Copy Paste: [[2411.05747]] WavShadow: Wavelet Based Shadow Segmentation and Removal(https://arxiv.org/abs/2411.05747)
• Keywords: robust, segmentation
• Abstract: Shadow removal and segmentation remain challenging tasks in computer vision, particularly in complex real-world scenarios. This study presents a novel approach that enhances the ShadowFormer model by incorporating Masked Autoencoder (MAE) priors and Fast Fourier Convolution (FFC) blocks, leading to significantly faster convergence and improved performance. We introduce key innovations: (1) integration of MAE priors trained on Places2 dataset for better context understanding, (2) adoption of Haar wavelet features for enhanced edge detection and multi-scale analysis, and (3) implementation of a modified SAM Adapter for robust shadow segmentation. Extensive experiments on the challenging DESOBA dataset demonstrate that our approach achieves state-of-the-art results, with notable improvements in both convergence speed and shadow removal quality.

Title: Tract-RLFormer: A Tract-Specific RL policy based Decoder-only Transformer Network

• Authors: Ankita Joshi, Ashutosh Sharma, Anoushkrit Goel, Ranjeet Ranjan Jha, Chirag Ahuja, Arnav Bhavsar, Aditya Nigam
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05757
• Pdf URL: https://arxiv.org/pdf/2411.05757
• Copy Paste: [[2411.05757]] Tract-RLFormer: A Tract-Specific RL policy based Decoder-only Transformer Network(https://arxiv.org/abs/2411.05757)
• Keywords: diffusion, transformer, segmentation
• Abstract: Fiber tractography is a cornerstone of neuroimaging, enabling the detailed mapping of the brain's white matter pathways through diffusion MRI. This is crucial for understanding brain connectivity and function, making it a valuable tool in neurological applications. Despite its importance, tractography faces challenges due to its complexity and susceptibility to false positives, misrepresenting vital pathways. To address these issues, recent strategies have shifted towards deep learning, utilizing supervised learning, which depends on precise ground truth, or reinforcement learning, which operates without it. In this work, we propose Tract-RLFormer, a network utilizing both supervised and reinforcement learning, in a two-stage policy refinement process that markedly improves the accuracy and generalizability across various data-sets. By employing a tract-specific approach, our network directly delineates the tracts of interest, bypassing the traditional segmentation process. Through rigorous validation on datasets such as TractoInferno, HCP, and ISMRM-2015, our methodology demonstrates a leap forward in tractography, showcasing its ability to accurately map the brain's white matter tracts.

Title: Multi-hop Evidence Pursuit Meets the Web: Team Papelo at FEVER 2024

• Authors: Christopher Malon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05762
• Pdf URL: https://arxiv.org/pdf/2411.05762
• Copy Paste: [[2411.05762]] Multi-hop Evidence Pursuit Meets the Web: Team Papelo at FEVER 2024(https://arxiv.org/abs/2411.05762)
• Keywords: large language model
• Abstract: Separating disinformation from fact on the web has long challenged both the search and the reasoning powers of humans. We show that the reasoning power of large language models (LLMs) and the retrieval power of modern search engines can be combined to automate this process and explainably verify claims. We integrate LLMs and search under a multi-hop evidence pursuit strategy. This strategy generates an initial question based on an input claim using a sequence to sequence model, searches and formulates an answer to the question, and iteratively generates follow-up questions to pursue the evidence that is missing using an LLM. We demonstrate our system on the FEVER 2024 (AVeriTeC) shared task. Compared to a strategy of generating all the questions at once, our method obtains .045 higher label accuracy and .155 higher AVeriTeC score (evaluating the adequacy of the evidence). Through ablations, we show the importance of various design choices, such as the question generation method, medium-sized context, reasoning with one document at a time, adding metadata, paraphrasing, reducing the problem to two classes, and reconsidering the final verdict. Our submitted system achieves .510 AVeriTeC score on the dev set and .477 AVeriTeC score on the test set.

Title: FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents

• Authors: Yilun Zhao, Yitao Long, Yuru Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Yiming Zhang, Xiangru Tang, Chen Zhao, Arman Cohan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05764
• Pdf URL: https://arxiv.org/pdf/2411.05764
• Copy Paste: [[2411.05764]] FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents(https://arxiv.org/abs/2411.05764)
• Keywords: extraction
• Abstract: We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FinDVer contains 2,400 expert-annotated examples, divided into three subsets: information extraction, numerical reasoning, and knowledge-intensive reasoning, each addressing common scenarios encountered in real-world financial contexts. We assess a broad spectrum of LLMs under long-context and RAG settings. Our results show that even the current best-performing system, GPT-4o, still lags behind human experts. We further provide in-depth analysis on long-context and RAG setting, Chain-of-Thought reasoning, and model reasoning errors, offering insights to drive future advancements. We believe that FinDVer can serve as a valuable benchmark for evaluating LLMs in claim verification over complex, expert-domain documents.

Title: Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?

• Authors: Veronica Chatrath, Marcelo Lotif, Shaina Raza
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05775
• Pdf URL: https://arxiv.org/pdf/2411.05775
• Copy Paste: [[2411.05775]] Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?(https://arxiv.org/abs/2411.05775)
• Keywords: robust, large language model
• Abstract: Political misinformation poses significant challenges to democratic processes, shaping public opinion and trust in media. Manual fact-checking methods face issues of scalability and annotator bias, while machine learning models require large, costly labelled datasets. This study investigates the use of state-of-the-art large language models (LLMs) as reliable annotators for detecting political factuality in news articles. Using open-source LLMs, we create a politically diverse dataset, labelled for bias through LLM-generated annotations. These annotations are validated by human experts and further evaluated by LLM-based judges to assess the accuracy and reliability of the annotations. Our approach offers a scalable and robust alternative to traditional fact-checking, enhancing transparency and public trust in media.

Title: Quantitative Assessment of Intersectional Empathetic Bias and Understanding

• Authors: Vojtech Formanek, Ondrej Sotolar
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.05777
• Pdf URL: https://arxiv.org/pdf/2411.05777
• Copy Paste: [[2411.05777]] Quantitative Assessment of Intersectional Empathetic Bias and Understanding(https://arxiv.org/abs/2411.05777)
• Keywords: robust
• Abstract: A growing amount of literature critiques the current operationalizations of empathy based on loose definitions of the construct. Such definitions negatively affect dataset quality, model robustness, and evaluation reliability. We propose an empathy evaluation framework that operationalizes empathy close to its psychological origins. The framework measures the variance in responses of LLMs to prompts using existing metrics for empathy and emotional valence. The variance is introduced through the controlled generation of the prompts by varying social biases affecting context understanding, thus impacting empathetic understanding. The control over generation ensures high theoretical validity of the constructs in the prompt dataset. Also, it makes high-quality translation, especially into languages that currently have little-to-no way of evaluating empathy or bias, such as the Slavonic family, more manageable. Using chosen LLMs and various prompt types, we demonstrate the empathy evaluation with the framework, including multiple-choice answers and free generation. The variance in our initial evaluation sample is small and we were unable to measure convincing differences between the empathetic understanding in contexts given by different social groups. However, the results are promising because the models showed significant alterations their reasoning chains needed to capture the relatively subtle changes in the prompts. This provides the basis for future research into the construction of the evaluation sample and statistical methods for measuring the results.

Title: Curriculum Learning for Few-Shot Domain Adaptation in CT-based Airway Tree Segmentation

• Authors: Maxime Jacovella, Ali Keshavarzi, Elsa Angelini
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05779
• Pdf URL: https://arxiv.org/pdf/2411.05779
• Copy Paste: [[2411.05779]] Curriculum Learning for Few-Shot Domain Adaptation in CT-based Airway Tree Segmentation(https://arxiv.org/abs/2411.05779)
• Keywords: segmentation
• Abstract: Despite advances with deep learning (DL), automated airway segmentation from chest CT scans continues to face challenges in segmentation quality and generalization across cohorts. To address these, we propose integrating Curriculum Learning (CL) into airway segmentation networks, distributing the training set into batches according to ad-hoc complexity scores derived from CT scans and corresponding ground-truth tree features. We specifically investigate few-shot domain adaptation, targeting scenarios where manual annotation of a full fine-tuning dataset is prohibitively expensive. Results are reported on two large open-cohorts (ATM22 and AIIB23) with high performance using CL for full training (Source domain) and few-shot fine-tuning (Target domain), but with also some insights on potential detrimental effects if using a classic Bootstrapping scoring function or if not using proper scan sequencing.

Title: GazeSearch: Radiology Findings Search Benchmark

• Authors: Trong Thang Pham, Tien-Phat Nguyen, Yuki Ikebe, Akash Awasthi, Zhigang Deng, Carol C. Wu, Hien Nguyen, Ngan Le
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05780
• Pdf URL: https://arxiv.org/pdf/2411.05780
• Copy Paste: [[2411.05780]] GazeSearch: Radiology Findings Search Benchmark(https://arxiv.org/abs/2411.05780)
• Keywords: interpretability
• Abstract: Medical eye-tracking data is an important information source for understanding how radiologists visually interpret medical images. This information not only improves the accuracy of deep learning models for X-ray analysis but also their interpretability, enhancing transparency in decision-making. However, the current eye-tracking data is dispersed, unprocessed, and ambiguous, making it difficult to derive meaningful insights. Therefore, there is a need to create a new dataset with more focus and purposeful eyetracking data, improving its utility for diagnostic applications. In this work, we propose a refinement method inspired by the target-present visual search challenge: there is a specific finding and fixations are guided to locate it. After refining the existing eye-tracking datasets, we transform them into a curated visual search dataset, called GazeSearch, specifically for radiology findings, where each fixation sequence is purposefully aligned to the task of locating a particular finding. Subsequently, we introduce a scan path prediction baseline, called ChestSearch, specifically tailored to GazeSearch. Finally, we employ the newly introduced GazeSearch as a benchmark to evaluate the performance of current state-of-the-art methods, offering a comprehensive assessment for visual search in the medical imaging domain.

Title: Recycled Attention: Efficient inference for long-context language models

• Authors: Fangyuan Xu, Tanya Goyal, Eunsol Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05787
• Pdf URL: https://arxiv.org/pdf/2411.05787
• Copy Paste: [[2411.05787]] Recycled Attention: Efficient inference for long-context language models(https://arxiv.org/abs/2411.05787)
• Keywords: large language model
• Abstract: Generating long sequences of tokens given a long-context input imposes a heavy computational burden for large language models (LLMs). One of the computational bottleneck comes from computing attention over a long sequence of input at each generation step. In this paper, we propose Recycled Attention, an inference-time method which alternates between full context attention and attention over a subset of input tokens. When performing partial attention, we recycle the attention pattern of a previous token that has performed full attention and attend only to the top K most attended tokens, reducing the cost of data movement and attention computation. Compared to previously proposed inference-time acceleration method which attends only to local context or tokens with high accumulative attention scores, our approach flexibly chooses tokens that are relevant to the current decoding step. We evaluate our methods on RULER, a suite of tasks designed to comprehensively evaluate long-context abilities, and long-context language modeling tasks. Applying our method to off-the-shelf LLMs achieves comparable speedup to baselines which only consider local context while improving the performance by 2x. We further explore two ideas to improve performance-efficiency trade-offs: (1) dynamically decide when to perform recycled or full attention step based on the query similarities and (2) continued pre-training the model with Recycled Attention.