2024-12-23

Title: Image Privacy Protection: A Survey

• Authors: Wenying Wen, Ziye Yuan, Yushu Zhang, Tao Wang, Xiangli Xiao, Ruoyu Zhao, Yuming Fang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15228
• Pdf URL: https://arxiv.org/pdf/2412.15228
• Copy Paste: [[2412.15228]] Image Privacy Protection: A Survey(https://arxiv.org/abs/2412.15228)
• Keywords: privacy, protect
• Abstract: Images serve as a crucial medium for communication, presenting information in a visually engaging format that facilitates rapid comprehension of key points. Meanwhile, during transmission and storage, they contain significant sensitive information. If not managed properly, this information may be vulnerable to exploitation for personal gain, potentially infringing on privacy rights and other legal entitlements. Consequently, researchers continue to propose some approaches for preserving image privacy and publish reviews that provide comprehensive and methodical summaries of these approaches. However, existing reviews tend to categorize either by specific scenarios, or by specific privacy objectives. This classification somewhat restricts the reader's ability to grasp a holistic view of image privacy protection and poses challenges in developing a total understanding of the subject that transcends different scenarios and privacy objectives. Instead of examining image privacy protection from a single aspect, it is more desirable to consider user needs for a comprehensive understanding. To fill this gap, we conduct a systematic review of image privacy protection approaches based on privacy protection goals. Specifically, we define the attribute known as privacy sensitive domains and use it as the core classification dimension to construct a comprehensive framework for image privacy protection that encompasses various scenarios and privacy objectives. This framework offers a deep understanding of the multi-layered aspects of image privacy, categorizing its protection into three primary levels: data-level, content-level, and feature-level. For each category, we analyze the main approaches and features of image privacy protection and systematically review representative solutions. Finally, we discuss the challenges and future directions of image privacy protection.

Title: OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Models

• Authors: Kartik Sharma, Peeyush Kumar, Yunqing Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15235
• Pdf URL: https://arxiv.org/pdf/2412.15235
• Copy Paste: [[2412.15235]] OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Models(https://arxiv.org/abs/2412.15235)
• Keywords: large language model
• Abstract: This paper presents OG-RAG, an Ontology-Grounded Retrieval Augmented Generation method designed to enhance LLM-generated responses by anchoring retrieval processes in domain-specific ontologies. While LLMs are widely used for tasks like question answering and search, they struggle to adapt to specialized knowledge, such as industrial workflows or knowledge work, without expensive fine-tuning or sub-optimal retrieval methods. Existing retrieval-augmented models, such as RAG, offer improvements but fail to account for structured domain knowledge, leading to suboptimal context generation. Ontologies, which conceptually organize domain knowledge by defining entities and their interrelationships, offer a structured representation to address this gap. OG-RAG constructs a hypergraph representation of domain documents, where each hyperedge encapsulates clusters of factual knowledge grounded using domain-specific ontology. An optimization algorithm then retrieves the minimal set of hyperedges that constructs a precise, conceptually grounded context for the LLM. This method enables efficient retrieval while preserving the complex relationships between entities. OG-RAG applies to domains where fact-based reasoning is essential, particularly in tasks that require workflows or decision-making steps to follow predefined rules and procedures. These include industrial workflows in healthcare, legal, and agricultural sectors, as well as knowledge-driven tasks such as news journalism, investigative research, consulting and more. Our evaluations demonstrate that OG-RAG increases the recall of accurate facts by 55% and improves response correctness by 40% across four different LLMs. Additionally, OG-RAG enables 30% faster attribution of responses to context and boosts fact-based reasoning accuracy by 27% compared to baseline methods.

Title: algoTRIC: Symmetric and asymmetric encryption algorithms for Cryptography -- A comparative analysis in AI era

• Authors: Naresh Kshetri, Mir Mehedi Rahman, Md Masud Rana, Omar Faruq Osama, James Hutson
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15237
• Pdf URL: https://arxiv.org/pdf/2412.15237
• Copy Paste: [[2412.15237]] algoTRIC: Symmetric and asymmetric encryption algorithms for Cryptography -- A comparative analysis in AI era(https://arxiv.org/abs/2412.15237)
• Keywords: security
• Abstract: The increasing integration of artificial intelligence (AI) within cybersecurity has necessitated stronger encryption methods to ensure data security. This paper presents a comparative analysis of symmetric (SE) and asymmetric encryption (AE) algorithms, focusing on their role in securing sensitive information in AI-driven environments. Through an in-depth study of various encryption algorithms such as AES, RSA, and others, this research evaluates the efficiency, complexity, and security of these algorithms within modern cybersecurity frameworks. Utilizing both qualitative and quantitative analysis, this research explores the historical evolution of encryption algorithms and their growing relevance in AI applications. The comparison of SE and AE algorithms focuses on key factors such as processing speed, scalability, and security resilience in the face of evolving threats. Special attention is given to how these algorithms are integrated into AI systems and how they manage the challenges posed by large-scale data processing in multi-agent environments. Our results highlight that while SE algorithms demonstrate high-speed performance and lower computational demands, AE algorithms provide superior security, particularly in scenarios requiring enhanced encryption for AI-based networks. The paper concludes by addressing the security concerns that encryption algorithms must tackle in the age of AI and outlines future research directions aimed at enhancing encryption techniques for cybersecurity.

Title: Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks

• Authors: Gregory Kang Ruey Lau, Wenyang Hu, Diwen Liu, Jizhuo Chen, See-Kiong Ng, Bryan Kian Hsiang Low
• Subjects: cs.CL, cs.AI, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2412.15238
• Pdf URL: https://arxiv.org/pdf/2412.15238
• Copy Paste: [[2412.15238]] Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks(https://arxiv.org/abs/2412.15238)
• Keywords: large language model
• Abstract: Large Language Models still encounter substantial challenges in reasoning tasks, especially for smaller models, which many users may be restricted to due to resource constraints (e.g. GPU memory restrictions). Inference-time methods to boost LLM performance, such as prompting methods to invoke certain reasoning pathways in responses, have been shown effective in past works, though they largely rely on sequential queries. The ensemble method, which consists of multiple constituent models running in parallel, is a promising approach to achieving better inference-time performance, especially given recent developments that enabled significant speed-ups in LLM batch inference. In this work, we propose a novel, training-free LLM ensemble framework where a single LLM model is fed an optimized, diverse set of prompts in parallel, effectively producing an ensemble at inference time to achieve performance improvement in reasoning tasks. We empirically demonstrate that our method leads to significant gains on math reasoning tasks, e.g., on MATH, where our ensemble consisting of a few small models (e.g., three Qwen2-MATH-1.5B-it models) can outperform a larger model (e.g., Qwen2-MATH-7B-it).

Title: Modeling Story Expectations to Understand Engagement: A Generative Framework Using LLMs

• Authors: Hortense Fong, George Gui
• Subjects: cs.CL, cs.AI, econ.GN, stat.ME
• Abstract URL: https://arxiv.org/abs/2412.15239
• Pdf URL: https://arxiv.org/pdf/2412.15239
• Copy Paste: [[2412.15239]] Modeling Story Expectations to Understand Engagement: A Generative Framework Using LLMs(https://arxiv.org/abs/2412.15239)
• Keywords: extraction, generative, large language model
• Abstract: Understanding when and why consumers engage with stories is crucial for content creators and platforms. While existing theories suggest that audience beliefs of what is going to happen should play an important role in engagement decisions, empirical work has mostly focused on developing techniques to directly extract features from actual content, rather than capturing forward-looking beliefs, due to the lack of a principled way to model such beliefs in unstructured narrative data. To complement existing feature extraction techniques, this paper introduces a novel framework that leverages large language models to model audience forward-looking beliefs about how stories might unfold. Our method generates multiple potential continuations for each story and extracts features related to expectations, uncertainty, and surprise using established content analysis techniques. Applying our method to over 30,000 book chapters from Wattpad, we demonstrate that our framework complements existing feature engineering techniques by amplifying their marginal explanatory power on average by 31%. The results reveal that different types of engagement-continuing to read, commenting, and voting-are driven by distinct combinations of current and anticipated content features. Our framework provides a novel way to study and explore how audience forward-looking beliefs shape their engagement with narrative media, with implications for marketing strategy in content-focused industries.

Title: ChainStream: An LLM-based Framework for Unified Synthetic Sensing

• Authors: Jiacheng Liu, Yuanchun Li, Liangyan Li, Yi Sun, Hao Wen, Xiangyu Li, Yao Guo, Yunxin Liu
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.15240
• Pdf URL: https://arxiv.org/pdf/2412.15240
• Copy Paste: [[2412.15240]] ChainStream: An LLM-based Framework for Unified Synthetic Sensing(https://arxiv.org/abs/2412.15240)
• Keywords: privacy, generative, large language model
• Abstract: Many applications demand context sensing to offer personalized and timely services. Yet, developing sensing programs can be challenging for developers and using them is privacy-concerning for end-users. In this paper, we propose to use natural language as the unified interface to process personal data and sense user context, which can effectively ease app development and make the data pipeline more transparent. Our work is inspired by large language models (LLMs) and other generative models, while directly applying them does not solve the problem - letting the model directly process the data cannot handle complex sensing requests and letting the model write the data processing program suffers error-prone code generation. We address the problem with 1) a unified data processing framework that makes context-sensing programs simpler and 2) a feedback-guided query optimizer that makes data query more informative. To evaluate the performance of natural language-based context sensing, we create a benchmark that contains 133 context sensing tasks. Extensive evaluation has shown that our approach is able to automatically solve the context-sensing tasks efficiently and precisely. The code is opensourced at this https URL.

Title: Quantifying Positional Biases in Text Embedding Models

• Authors: Samarth Goel, Reagan J. Lee, Kannan Ramchandran
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15241
• Pdf URL: https://arxiv.org/pdf/2412.15241
• Copy Paste: [[2412.15241]] Quantifying Positional Biases in Text Embedding Models(https://arxiv.org/abs/2412.15241)
• Keywords: robust
• Abstract: Embedding models are crucial for tasks in Information Retrieval (IR) and semantic similarity measurement, yet their handling of longer texts and associated positional biases remains underexplored. In this study, we investigate the impact of content position and input size on text embeddings. Our experiments reveal that embedding models, irrespective of their positional encoding mechanisms, disproportionately prioritize the beginning of an input. Ablation studies demonstrate that insertion of irrelevant text or removal at the start of a document reduces cosine similarity between altered and original embeddings by up to 12.3\% more than ablations at the end. Regression analysis further confirms this bias, with sentence importance declining as position moves further from the start, even with with content-agnosticity. We hypothesize that this effect arises from pre-processing strategies and chosen positional encoding techniques. These findings quantify the sensitivity of retrieval systems and suggest a new lens towards embedding model robustness.

Title: Script-Based Dialog Policy Planning for LLM-Powered Conversational Agents: A Basic Architecture for an "AI Therapist"

• Authors: Robert Wasenmüller, Kevin Hilbert, Christoph Benzmüller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15242
• Pdf URL: https://arxiv.org/pdf/2412.15242
• Copy Paste: [[2412.15242]] Script-Based Dialog Policy Planning for LLM-Powered Conversational Agents: A Basic Architecture for an "AI Therapist"(https://arxiv.org/abs/2412.15242)
• Keywords: large language model
• Abstract: Large Language Model (LLM)-Powered Conversational Agents have the potential to provide users with scaled behavioral healthcare support, and potentially even deliver full-scale "AI therapy'" in the future. While such agents can already conduct fluent and proactive emotional support conversations, they inherently lack the ability to (a) consistently and reliably act by predefined rules to align their conversation with an overarching therapeutic concept and (b) make their decision paths inspectable for risk management and clinical evaluation -- both essential requirements for an "AI Therapist". In this work, we introduce a novel paradigm for dialog policy planning in conversational agents enabling them to (a) act according to an expert-written "script" that outlines the therapeutic approach and (b) explicitly transition through a finite set of states over the course of the conversation. The script acts as a deterministic component, constraining the LLM's behavior in desirable ways and establishing a basic architecture for an AI Therapist. We implement two variants of Script-Based Dialog Policy Planning using different prompting techniques and synthesize a total of 100 conversations with LLM-simulated patients. The results demonstrate the feasibility of this new technology and provide insights into the efficiency and effectiveness of different implementation variants.

Title: MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples

• Authors: Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15244
• Pdf URL: https://arxiv.org/pdf/2412.15244
• Copy Paste: [[2412.15244]] MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples(https://arxiv.org/abs/2412.15244)
• Keywords: large language model
• Abstract: Aligning Large Language Models (LLMs) with human feedback is crucial for their development. Existing preference optimization methods such as DPO and KTO, while improved based on Reinforcement Learning from Human Feedback (RLHF), are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. Meanwhile, current preference optimization research mainly targets single-question scenarios with two replies, neglecting optimization with multiple replies, which leads to a waste of data in the application. This study introduces the MPPO algorithm, which leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. Through a comparison of Point-wise, Pair-wise, and List-wise implementations, we found that the Pair-wise approach achieves the best performance, significantly enhancing the quality of model responses. Experimental results demonstrate MPPO's outstanding performance across various benchmarks. On MT-Bench, MPPO outperforms DPO, ORPO, and SimPO. Notably, on Arena-Hard, MPPO surpasses DPO and ORPO by substantial margins. These achievements underscore the remarkable advantages of MPPO in preference optimization tasks.

Title: Accelerating Retrieval-Augmented Generation

• Authors: Derrick Quinn, Mohammad Nouri, Neel Patel, John Salihu, Alireza Salemi, Sukhan Lee, Hamed Zamani, Mohammad Alian
• Subjects: cs.CL, cs.AI, cs.AR, cs.DC, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15246
• Pdf URL: https://arxiv.org/pdf/2412.15246
• Copy Paste: [[2412.15246]] Accelerating Retrieval-Augmented Generation(https://arxiv.org/abs/2412.15246)
• Keywords: generative, large language model
• Abstract: An evolving solution to address hallucination and enhance accuracy in large language models (LLMs) is Retrieval-Augmented Generation (RAG), which involves augmenting LLMs with information retrieved from an external knowledge source, such as the web. This paper profiles several RAG execution pipelines and demystifies the complex interplay between their retrieval and generation phases. We demonstrate that while exact retrieval schemes are expensive, they can reduce inference time compared to approximate retrieval variants because an exact retrieval model can send a smaller but more accurate list of documents to the generative model while maintaining the same end-to-end accuracy. This observation motivates the acceleration of the exact nearest neighbor search for RAG. In this work, we design Intelligent Knowledge Store (IKS), a type-2 CXL device that implements a scale-out near-memory acceleration architecture with a novel cache-coherent interface between the host CPU and near-memory accelerators. IKS offers 13.4-27.9x faster exact nearest neighbor search over a 512GB vector database compared with executing the search on Intel Sapphire Rapids CPUs. This higher search performance translates to 1.7-26.3x lower end-to-end inference time for representative RAG applications. IKS is inherently a memory expander; its internal DRAM can be disaggregated and used for other applications running on the server to prevent DRAM, which is the most expensive component in today's servers, from being stranded.

Title: Streamlining Systematic Reviews: A Novel Application of Large Language Models

• Authors: Fouad Trad, Ryan Yammine, Jana Charafeddine, Marlene Chakhtoura, Maya Rahme, Ghada El-Hajj Fuleihan, Ali Chehab
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15247
• Pdf URL: https://arxiv.org/pdf/2412.15247
• Copy Paste: [[2412.15247]] Streamlining Systematic Reviews: A Novel Application of Large Language Models(https://arxiv.org/abs/2412.15247)
• Keywords: large language model
• Abstract: Systematic reviews (SRs) are essential for evidence-based guidelines but are often limited by the time-consuming nature of literature screening. We propose and evaluate an in-house system based on Large Language Models (LLMs) for automating both title/abstract and full-text screening, addressing a critical gap in the literature. Using a completed SR on Vitamin D and falls (14,439 articles), the LLM-based system employed prompt engineering for title/abstract screening and Retrieval-Augmented Generation (RAG) for full-text screening. The system achieved an article exclusion rate (AER) of 99.5%, specificity of 99.6%, a false negative rate (FNR) of 0%, and a negative predictive value (NPV) of 100%. After screening, only 78 articles required manual review, including all 20 identified by traditional methods, reducing manual screening time by 95.5%. For comparison, Rayyan, a commercial tool for title/abstract screening, achieved an AER of 72.1% and FNR of 5% when including articles Rayyan considered as undecided or likely to include. Lowering Rayyan's inclusion thresholds improved FNR to 0% but increased screening time. By addressing both screening phases, the LLM-based system significantly outperformed Rayyan and traditional methods, reducing total screening time to 25.5 hours while maintaining high accuracy. These findings highlight the transformative potential of LLMs in SR workflows by offering a scalable, efficient, and accurate solution, particularly for the full-text screening phase, which has lacked automation tools.

Title: RoundTripOCR: A Data Generation Technique for Enhancing Post-OCR Error Correction in Low-Resource Devanagari Languages

• Authors: Harshvivek Kashid, Pushpak Bhattacharyya
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15248
• Pdf URL: https://arxiv.org/pdf/2412.15248
• Copy Paste: [[2412.15248]] RoundTripOCR: A Data Generation Technique for Enhancing Post-OCR Error Correction in Low-Resource Devanagari Languages(https://arxiv.org/abs/2412.15248)
• Keywords: extraction, transformer
• Abstract: Optical Character Recognition (OCR) technology has revolutionized the digitization of printed text, enabling efficient data extraction and analysis across various domains. Just like Machine Translation systems, OCR systems are prone to errors. In this work, we address the challenge of data generation and post-OCR error correction, specifically for low-resource languages. We propose an approach for synthetic data generation for Devanagari languages, RoundTripOCR, that tackles the scarcity of the post-OCR Error Correction datasets for low-resource languages. We release post-OCR text correction datasets for Hindi, Marathi, Bodo, Nepali, Konkani and Sanskrit. We also present a novel approach for OCR error correction by leveraging techniques from machine translation. Our method involves translating erroneous OCR output into a corrected form by treating the OCR errors as mistranslations in a parallel text corpus, employing pre-trained transformer models to learn the mapping from erroneous to correct text pairs, effectively correcting OCR errors.

Title: LLMs for Literature Review: Are we there yet?

• Authors: Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, Christopher Pal
• Subjects: cs.CL, cs.AI, cs.DL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15249
• Pdf URL: https://arxiv.org/pdf/2412.15249
• Copy Paste: [[2412.15249]] LLMs for Literature Review: Are we there yet?(https://arxiv.org/abs/2412.15249)
• Keywords: large language model
• Abstract: Literature reviews are an essential component of scientific research, but they remain time-intensive and challenging to write, especially due to the recent influx of research papers. This paper explores the zero-shot abilities of recent Large Language Models (LLMs) in assisting with the writing of literature reviews based on an abstract. We decompose the task into two components: 1. Retrieving related works given a query abstract, and 2. Writing a literature review based on the retrieved results. We analyze how effective LLMs are for both components. For retrieval, we introduce a novel two-step search strategy that first uses an LLM to extract meaningful keywords from the abstract of a paper and then retrieves potentially relevant papers by querying an external knowledge base. Additionally, we study a prompting-based re-ranking mechanism with attribution and show that re-ranking doubles the normalized recall compared to naive search methods, while providing insights into the LLM's decision-making process. In the generation phase, we propose a two-step approach that first outlines a plan for the review and then executes steps in the plan to generate the actual review. To evaluate different LLM-based literature review methods, we create test sets from arXiv papers using a protocol designed for rolling use with newly released LLMs to avoid test set contamination in zero-shot evaluations. We release this evaluation protocol to promote additional research and development in this regard. Our empirical results suggest that LLMs show promising potential for writing literature reviews when the task is decomposed into smaller components of retrieval and planning. Further, we demonstrate that our planning-based approach achieves higher-quality reviews by minimizing hallucinated references in the generated review by 18-26% compared to existing simpler LLM-based generation methods.

Title: An Enhanced Text Compression Approach Using Transformer-based Language Models

• Authors: Chowdhury Mofizur Rahman, Mahbub E Sobhani, Anika Tasnim Rodela, Swakkhar Shatabda
• Subjects: cs.CL, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15250
• Pdf URL: https://arxiv.org/pdf/2412.15250
• Copy Paste: [[2412.15250]] An Enhanced Text Compression Approach Using Transformer-based Language Models(https://arxiv.org/abs/2412.15250)
• Keywords: transformer
• Abstract: Text compression shrinks textual data while keeping crucial information, eradicating constraints on storage, bandwidth, and computational efficacy. The integration of lossless compression techniques with transformer-based text decompression has received negligible attention, despite the increasing volume of English text data in communication. The primary barrier in advancing text compression and restoration involves optimizing transformer-based approaches with efficient pre-processing and integrating lossless compression algorithms, that remained unresolved in the prior attempts. Here, we propose a transformer-based method named RejuvenateForme for text decompression, addressing prior issues by harnessing a new pre-processing technique and a lossless compression method. Our meticulous pre-processing technique incorporating the Lempel-Ziv-Welch algorithm achieves compression ratios of 12.57, 13.38, and 11.42 on the BookCorpus, EN-DE, and EN-FR corpora, thus showing state-of-the-art compression ratios compared to other deep learning and traditional approaches. Furthermore, the RejuvenateForme achieves a BLEU score of 27.31, 25.78, and 50.45 on the EN-DE, EN-FR, and BookCorpus corpora, showcasing its comprehensive efficacy. In contrast, the pre-trained T5-Small exhibits better performance over prior state-of-the-art models.

Title: AgentPS: Agentic Process Supervision for Multi-modal Content Quality Assurance through Multi-round QA

• Authors: Gorden Liu, Yu Sun, Ruixiao Sun, Xin Dong, Hongyu Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15251
• Pdf URL: https://arxiv.org/pdf/2412.15251
• Copy Paste: [[2412.15251]] AgentPS: Agentic Process Supervision for Multi-modal Content Quality Assurance through Multi-round QA(https://arxiv.org/abs/2412.15251)
• Keywords: large language model
• Abstract: The advanced processing and reasoning capabilities of multimodal large language models (MLLMs) have driven substantial progress in vision-language (VL) understanding tasks. However, while effective for tasks governed by straightforward logic, MLLMs often encounter challenges when reasoning over complex, interdependent logic structures. To address this limitation, we introduce \textit{AgentPS}, a novel framework that integrates Agentic Process Supervision into MLLMs via multi-round question answering during fine-tuning. \textit{AgentPS} demonstrates significant performance improvements over baseline MLLMs on proprietary TikTok datasets, due to its integration of process supervision and structured sequential reasoning. Furthermore, we show that replacing human-annotated labels with LLM-generated labels retains much of the performance gain, highlighting the framework's practical scalability in industrial applications. These results position \textit{AgentPS} as a highly effective and efficient architecture for multimodal classification tasks. Its adaptability and scalability, especially when enhanced by automated annotation generation, make it a powerful tool for handling large-scale, real-world challenges.

Title: NER- RoBERTa: Fine-Tuning RoBERTa for Named Entity Recognition (NER) within low-resource languages

• Authors: Abdulhady Abas Abdullah, Srwa Hasan Abdulla, Dalia Mohammad Toufiq, Halgurd S. Maghdid, Tarik A. Rashid, Pakshan F. Farho, Shadan Sh. Sabr, Akar H. Taher, Darya S. Hamad, Hadi Veisi, Aras T. Asaad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15252
• Pdf URL: https://arxiv.org/pdf/2412.15252
• Copy Paste: [[2412.15252]] NER- RoBERTa: Fine-Tuning RoBERTa for Named Entity Recognition (NER) within low-resource languages(https://arxiv.org/abs/2412.15252)
• Keywords: generative
• Abstract: Nowadays, Natural Language Processing (NLP) is an important tool for most people's daily life routines, ranging from understanding speech, translation, named entity recognition (NER), and text categorization, to generative text models such as ChatGPT. Due to the existence of big data and consequently large corpora for widely used languages like English, Spanish, Turkish, Persian, and many more, these applications have been developed accurately. However, the Kurdish language still requires more corpora and large datasets to be included in NLP applications. This is because Kurdish has a rich linguistic structure, varied dialects, and a limited dataset, which poses unique challenges for Kurdish NLP (KNLP) application development. While several studies have been conducted in KNLP for various applications, Kurdish NER (KNER) remains a challenge for many KNLP tasks, including text analysis and classification. In this work, we address this limitation by proposing a methodology for fine-tuning the pre-trained RoBERTa model for KNER. To this end, we first create a Kurdish corpus, followed by designing a modified model architecture and implementing the training procedures. To evaluate the trained model, a set of experiments is conducted to demonstrate the performance of the KNER model using different tokenization methods and trained models. The experimental results show that fine-tuned RoBERTa with the SentencePiece tokenization method substantially improves KNER performance, achieving a 12.8% improvement in F1-score compared to traditional models, and consequently establishes a new benchmark for KNLP.

Title: Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction

• Authors: Andrea Cristina McGlinchey, Peter J Barclay
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15253
• Pdf URL: https://arxiv.org/pdf/2412.15253
• Copy Paste: [[2412.15253]] Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction(https://arxiv.org/abs/2412.15253)
• Keywords: protect, generative, large language model
• Abstract: Following the universal availability of generative AI systems with the release of ChatGPT, automatic detection of deceptive text created by Large Language Models has focused on domains such as academic plagiarism and "fake news". However, generative AI also poses a threat to the livelihood of creative writers, and perhaps to literary culture in general, through reduction in quality of published material. Training a Large Language Model on writers' output to generate "sham books" in a particular style seems to constitute a new form of plagiarism. This problem has been little researched. In this study, we trained Machine Learning classifier models to distinguish short samples of human-written from machine-generated creative fiction, focusing on classic detective novels. Our results show that a Naive Bayes and a Multi-Layer Perceptron classifier achieved a high degree of success (accuracy > 95%), significantly outperforming human judges (accuracy < 55%). This approach worked well with short text samples (around 100 words), which previous research has shown to be difficult to classify. We have deployed an online proof-of-concept classifier tool, AI Detective, as a first step towards developing lightweight and reliable applications for use by editors and publishers, with the aim of protecting the economic and cultural contribution of human authors.

Title: RIRO: Reshaping Inputs, Refining Outputs Unlocking the Potential of Large Language Models in Data-Scarce Contexts

• Authors: Ali Hamdi, Hozaifa Kassab, Mohamed Bahaa, Marwa Mohamed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15254
• Pdf URL: https://arxiv.org/pdf/2412.15254
• Copy Paste: [[2412.15254]] RIRO: Reshaping Inputs, Refining Outputs Unlocking the Potential of Large Language Models in Data-Scarce Contexts(https://arxiv.org/abs/2412.15254)
• Keywords: large language model
• Abstract: Large language models (LLMs) have significantly advanced natural language processing, excelling in areas like text generation, summarization, and question-answering. Despite their capabilities, these models face challenges when fine-tuned on small, domain-specific datasets, often struggling to generalize and deliver accurate results with unfamiliar inputs. To tackle this issue, we introduce RIRO, a novel two-layer architecture designed to improve performance in data-scarce environments. The first layer leverages advanced prompt engineering to reformulate inputs, ensuring better alignment with training data, while the second layer focuses on refining outputs to minimize inconsistencies. Through fine-tuning models like Phi-2, Falcon 7B, and Falcon 1B, with Phi-2 outperforming the others. Additionally, we introduce a benchmark using evaluation metrics such as cosine similarity, Levenshtein distance, BLEU score, ROUGE-1, ROUGE-2, and ROUGE-L. While these advancements improve performance, challenges like computational demands and overfitting persist, limiting the potential of LLMs in data-scarce, high-stakes environments such as healthcare, legal documentation, and software testing.

Title: Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation

• Authors: Jonibek Mansurov, Akhmed Sakip, Alham Fikri Aji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15255
• Pdf URL: https://arxiv.org/pdf/2412.15255
• Copy Paste: [[2412.15255]] Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation(https://arxiv.org/abs/2412.15255)
• Keywords: robust
• Abstract: In this paper, we show that knowledge distillation can be subverted to manipulate language model benchmark scores, revealing a critical vulnerability in current evaluation practices. We introduce "Data Laundering," a three-phase process analogous to financial money laundering, that enables the covert transfer of benchmark-specific knowledge through seemingly legitimate intermediate training steps. Through extensive experiments with a 2-layer BERT student model, we show how this approach can achieve substantial improvements in benchmark accuracy (up to 75\% on GPQA) without developing genuine reasoning capabilities. Notably, this method can be exploited intentionally or even unintentionally, as researchers may inadvertently adopt this method that inflates scores using knowledge distillation without realizing the implications. While our findings demonstrate the effectiveness of this technique, we present them as a cautionary tale highlighting the urgent need for more robust evaluation methods in AI. This work aims to contribute to the ongoing discussion about evaluation integrity in AI development and the need for benchmarks that more accurately reflect true model capabilities. The code is available at \url{this https URL}.

Title: Structured Extraction of Real World Medical Knowledge using LLMs for Summarization and Search

• Authors: Edward Kim, Manil Shrestha, Richard Foty, Tom DeLay, Vicki Seyfert-Margolis
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15256
• Pdf URL: https://arxiv.org/pdf/2412.15256
• Copy Paste: [[2412.15256]] Structured Extraction of Real World Medical Knowledge using LLMs for Summarization and Search(https://arxiv.org/abs/2412.15256)
• Keywords: extraction, large language model
• Abstract: Creation and curation of knowledge graphs can accelerate disease discovery and analysis in real-world data. While disease ontologies aid in biological data annotation, codified categories (SNOMED-CT, ICD10, CPT) may not capture patient condition nuances or rare diseases. Multiple disease definitions across data sources complicate ontology mapping and disease clustering. We propose creating patient knowledge graphs using large language model extraction techniques, allowing data extraction via natural language rather than rigid ontological hierarchies. Our method maps to existing ontologies (MeSH, SNOMED-CT, RxNORM, HPO) to ground extracted entities. Using a large ambulatory care EHR database with 33.6M patients, we demonstrate our method through the patient search for Dravet syndrome, which received ICD10 recognition in October 2020. We describe our construction of patient-specific knowledge graphs and symptom-based patient searches. Using confirmed Dravet syndrome ICD10 codes as ground truth, we employ LLM-based entity extraction to characterize patients in grounded ontologies. We then apply this method to identify Beta-propeller protein-associated neurodegeneration (BPAN) patients, demonstrating real-world discovery where no ground truth exists.

Title: DisEmbed: Transforming Disease Understanding through Embeddings

• Authors: Salman Faroz
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15258
• Pdf URL: https://arxiv.org/pdf/2412.15258
• Copy Paste: [[2412.15258]] DisEmbed: Transforming Disease Understanding through Embeddings(https://arxiv.org/abs/2412.15258)
• Keywords: robust
• Abstract: The medical domain is vast and diverse, with many existing embedding models focused on general healthcare applications. However, these models often struggle to capture a deep understanding of diseases due to their broad generalization across the entire medical field. To address this gap, I present DisEmbed, a disease-focused embedding model. DisEmbed is trained on a synthetic dataset specifically curated to include disease descriptions, symptoms, and disease-related Q\&A pairs, making it uniquely suited for disease-related tasks. For evaluation, I benchmarked DisEmbed against existing medical models using disease-specific datasets and the triplet evaluation method. My results demonstrate that DisEmbed outperforms other models, particularly in identifying disease-related contexts and distinguishing between similar diseases. This makes DisEmbed highly valuable for disease-specific use cases, including retrieval-augmented generation (RAG) tasks, where its performance is particularly robust.

Title: Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice

• Authors: Hannes Westermann, Jaromir Savelka
• Subjects: cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.15260
• Pdf URL: https://arxiv.org/pdf/2412.15260
• Copy Paste: [[2412.15260]] Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice(https://arxiv.org/abs/2412.15260)
• Keywords: large language model
• Abstract: Interacting with the legal system and the government requires the assembly and analysis of various pieces of information that can be spread across different (paper) documents, such as forms, certificates and contracts (e.g. leases). This information is required in order to understand one's legal rights, as well as to fill out forms to file claims in court or obtain government benefits. However, finding the right information, locating the correct forms and filling them out can be challenging for laypeople. Large language models (LLMs) have emerged as a powerful technology that has the potential to address this gap, but still rely on the user to provide the correct information, which may be challenging and error-prone if the information is only available in complex paper documents. We present an investigation into utilizing multi-modal LLMs to analyze images of handwritten paper forms, in order to automatically extract relevant information in a structured format. Our initial results are promising, but reveal some limitations (e.g., when the image quality is low). Our work demonstrates the potential of integrating multi-modal LLMs to support laypeople and self-represented litigants in finding and assembling relevant information.

Title: Advanced ingestion process powered by LLM parsing for RAG system

• Authors: Arnau Perez, Xavier Vizcaino
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15262
• Pdf URL: https://arxiv.org/pdf/2412.15262
• Copy Paste: [[2412.15262]] Advanced ingestion process powered by LLM parsing for RAG system(https://arxiv.org/abs/2412.15262)
• Keywords: extraction
• Abstract: Retrieval Augmented Generation (RAG) systems struggle with processing multimodal documents of varying structural complexity. This paper introduces a novel multi-strategy parsing approach using LLM-powered OCR to extract content from diverse document types, including presentations and high text density files both scanned or not. The methodology employs a node-based extraction technique that creates relationships between different information types and generates context-aware metadata. By implementing a Multimodal Assembler Agent and a flexible embedding strategy, the system enhances document comprehension and retrieval capabilities. Experimental evaluations across multiple knowledge bases demonstrate the approach's effectiveness, showing improvements in answer relevancy and information faithfulness.

Title: ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports

• Authors: Romain Hardy, Sung Eun Kim, Pranav Rajpurkar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15264
• Pdf URL: https://arxiv.org/pdf/2412.15264
• Copy Paste: [[2412.15264]] ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports(https://arxiv.org/abs/2412.15264)
• Keywords: robust
• Abstract: The increasing adoption of AI-generated radiology reports necessitates robust methods for detecting hallucinations--false or unfounded statements that could impact patient care. We present ReXTrust, a novel framework for fine-grained hallucination detection in AI-generated radiology reports. Our approach leverages sequences of hidden states from large vision-language models to produce finding-level hallucination risk scores. We evaluate ReXTrust on a subset of the MIMIC-CXR dataset and demonstrate superior performance compared to existing approaches, achieving an AUROC of 0.8751 across all findings and 0.8963 on clinically significant findings. Our results show that white-box approaches leveraging model hidden states can provide reliable hallucination detection for medical AI systems, potentially improving the safety and reliability of automated radiology reporting.

Title: Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models

• Authors: Yingshui Tan, Boren Zheng, Baihui Zheng, Kerui Cao, Huiyun Jing, Jincheng Wei, Jiaheng Liu, Yancheng He, Wenbo Su, Xiangyong Zhu, Bo Zheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15265
• Pdf URL: https://arxiv.org/pdf/2412.15265
• Copy Paste: [[2412.15265]] Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models(https://arxiv.org/abs/2412.15265)
• Keywords: attack, robust, large language model
• Abstract: With the rapid advancement of Large Language Models (LLMs), significant safety concerns have emerged. Fundamentally, the safety of large language models is closely linked to the accuracy, comprehensiveness, and clarity of their understanding of safety knowledge, particularly in domains such as law, policy and ethics. This factuality ability is crucial in determining whether these models can be deployed and applied safely and compliantly within specific regions. To address these challenges and better evaluate the factuality ability of LLMs to answer short questions, we introduce the Chinese SafetyQA benchmark. Chinese SafetyQA has several properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate, Safety-related, Harmless). Based on Chinese SafetyQA, we perform a comprehensive evaluation on the factuality abilities of existing LLMs and analyze how these capabilities relate to LLM abilities, e.g., RAG ability and robustness against attacks.

Title: On the Structural Memory of LLM Agents

• Authors: Ruihong Zeng, Jinyuan Fang, Siwei Liu, Zaiqiao Meng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15266
• Pdf URL: https://arxiv.org/pdf/2412.15266
• Copy Paste: [[2412.15266]] On the Structural Memory of LLM Agents(https://arxiv.org/abs/2412.15266)
• Keywords: large language model
• Abstract: Memory plays a pivotal role in enabling large language model~(LLM)-based agents to engage in complex and long-term interactions, such as question answering (QA) and dialogue systems. While various memory modules have been proposed for these tasks, the impact of different memory structures across tasks remains insufficiently explored. This paper investigates how memory structures and memory retrieval methods affect the performance of LLM-based agents. Specifically, we evaluate four types of memory structures, including chunks, knowledge triples, atomic facts, and summaries, along with mixed memory that combines these components. In addition, we evaluate three widely used memory retrieval methods: single-step retrieval, reranking, and iterative retrieval. Extensive experiments conducted across four tasks and six datasets yield the following key insights: (1) Different memory structures offer distinct advantages, enabling them to be tailored to specific tasks; (2) Mixed memory structures demonstrate remarkable resilience in noisy environments; (3) Iterative retrieval consistently outperforms other methods across various scenarios. Our investigation aims to inspire further research into the design of memory systems for LLM-based agents.

Title: Toxicity Detection towards Adaptability to Changing Perturbations

• Authors: Hankun Kang, Jianhao Chen, Yongqi Li, Xin Miao, Mayi Xu, Ming Zhong, Yuanyuan Zhu, Tieyun Qian
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15267
• Pdf URL: https://arxiv.org/pdf/2412.15267
• Copy Paste: [[2412.15267]] Toxicity Detection towards Adaptability to Changing Perturbations(https://arxiv.org/abs/2412.15267)
• Keywords: security, robust, large language model
• Abstract: Toxicity detection is crucial for maintaining the peace of the society. While existing methods perform well on normal toxic contents or those generated by specific perturbation methods, they are vulnerable to evolving perturbation patterns. However, in real-world scenarios, malicious users tend to create new perturbation patterns for fooling the detectors. For example, some users may circumvent the detector of large language models (LLMs) by adding `I am a scientist' at the beginning of the prompt. In this paper, we introduce a novel problem, i.e., continual learning jailbreak perturbation patterns, into the toxicity detection field. To tackle this problem, we first construct a new dataset generated by 9 types of perturbation patterns, 7 of them are summarized from prior work and 2 of them are developed by us. We then systematically validate the vulnerability of current methods on this new perturbation pattern-aware dataset via both the zero-shot and fine tuned cross-pattern detection. Upon this, we present the domain incremental learning paradigm and the corresponding benchmark to ensure the detector's robustness to dynamically emerging types of perturbed toxic text. Our code and dataset are provided in the appendix and will be publicly available at GitHub, by which we wish to offer new research opportunities for the security-relevant communities.

Title: Enhancing LLM-based Hatred and Toxicity Detection with Meta-Toxic Knowledge Graph

• Authors: Yibo Zhao, Jiapeng Zhu, Can Xu, Xiang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15268
• Pdf URL: https://arxiv.org/pdf/2412.15268
• Copy Paste: [[2412.15268]] Enhancing LLM-based Hatred and Toxicity Detection with Meta-Toxic Knowledge Graph(https://arxiv.org/abs/2412.15268)
• Keywords: large language model
• Abstract: The rapid growth of social media platforms has raised significant concerns regarding online content toxicity. When Large Language Models (LLMs) are used for toxicity detection, two key challenges emerge: 1) the absence of domain-specific toxic knowledge leads to false negatives; 2) the excessive sensitivity of LLMs to toxic speech results in false positives, limiting freedom of speech. To address these issues, we propose a novel method called MetaTox, leveraging graph search on a meta-toxic knowledge graph to enhance hatred and toxicity detection. First, we construct a comprehensive meta-toxic knowledge graph by utilizing LLMs to extract toxic information through a three-step pipeline, with toxic benchmark datasets serving as corpora. Second, we query the graph via retrieval and ranking processes to supplement accurate, relevant toxic knowledge. Extensive experiments and in-depth case studies across multiple datasets demonstrate that our MetaTox significantly decreases the false positive rate while boosting overall toxicity detection performance. Our code will be available soon.

Title: The Reliability Paradox: Exploring How Shortcut Learning Undermines Language Model Calibration

• Authors: Geetanjali Bihani, Julia Rayz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15269
• Pdf URL: https://arxiv.org/pdf/2412.15269
• Copy Paste: [[2412.15269]] The Reliability Paradox: Exploring How Shortcut Learning Undermines Language Model Calibration(https://arxiv.org/abs/2412.15269)
• Keywords: robust
• Abstract: The advent of pre-trained language models (PLMs) has enabled significant performance gains in the field of natural language processing. However, recent studies have found PLMs to suffer from miscalibration, indicating a lack of accuracy in the confidence estimates provided by these models. Current evaluation methods for PLM calibration often assume that lower calibration error estimates indicate more reliable predictions. However, fine-tuned PLMs often resort to shortcuts, leading to overconfident predictions that create the illusion of enhanced performance but lack generalizability in their decision rules. The relationship between PLM reliability, as measured by calibration error, and shortcut learning, has not been thoroughly explored thus far. This paper aims to investigate this relationship, studying whether lower calibration error implies reliable decision rules for a language model. Our findings reveal that models with seemingly superior calibration portray higher levels of non-generalizable decision rules. This challenges the prevailing notion that well-calibrated models are inherently reliable. Our study highlights the need to bridge the current gap between language model calibration and generalization objectives, urging the development of comprehensive frameworks to achieve truly robust and reliable language models.

Title: Baichuan4-Finance Technical Report

• Authors: Hanyu Zhang, Boyu Qiu, Yuhao Feng, Shuqi Li, Qian Ma, Xiyuan Zhang, Qiang Ju, Dong Yan, Jian Xie
• Subjects: cs.CL, cs.AI, cs.CE, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15270
• Pdf URL: https://arxiv.org/pdf/2412.15270
• Copy Paste: [[2412.15270]] Baichuan4-Finance Technical Report(https://arxiv.org/abs/2412.15270)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated strong capabilities in language understanding, generation, and reasoning, yet their potential in finance remains underexplored due to the complexity and specialization of financial knowledge. In this work, we report the development of the Baichuan4-Finance series, including a comprehensive suite of foundational Baichuan4-Finance-Base and an aligned language model Baichuan4-Finance, which are built upon Baichuan4-Turbo base model and tailored for finance domain. Firstly, we have dedicated significant effort to building a detailed pipeline for improving data quality. Moreover, in the continual pre-training phase, we propose a novel domain self-constraint training strategy, which enables Baichuan4-Finance-Base to acquire financial knowledge without losing general capabilities. After Supervised Fine-tuning and Reinforcement Learning from Human Feedback and AI Feedback, the chat model Baichuan4-Finance is able to tackle various financial certification questions and real-world scenario applications. We evaluate Baichuan4-Finance on many widely used general datasets and two holistic financial benchmarks. The evaluation results show that Baichuan4-Finance-Base surpasses almost all competitive baselines on financial tasks by significant margins without sacrificing performance on general LLM benchmarks. At the same time, Baichuan4-Finance demonstrates even more impressive performance on financial application scenarios, showcasing its potential to foster community innovation in the financial LLM field.

Title: A MapReduce Approach to Effectively Utilize Long Context Information in Retrieval Augmented Language Models

• Authors: Gongbo Zhang, Zihan Xu, Qiao Jin, Fangyi Chen, Yilu Fang, Yi Liu, Justin F. Rousseau, Ziyang Xu, Zhiyong Lu, Chunhua Weng, Yifan Peng
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15271
• Pdf URL: https://arxiv.org/pdf/2412.15271
• Copy Paste: [[2412.15271]] A MapReduce Approach to Effectively Utilize Long Context Information in Retrieval Augmented Language Models(https://arxiv.org/abs/2412.15271)
• Keywords: robust, large language model
• Abstract: While holding great promise for improving and facilitating healthcare, large language models (LLMs) struggle to produce up-to-date responses on evolving topics due to outdated knowledge or hallucination. Retrieval-augmented generation (RAG) is a pivotal innovation that improves the accuracy and relevance of LLM responses by integrating LLMs with a search engine and external sources of knowledge. However, the quality of RAG responses can be largely impacted by the rank and density of key information in the retrieval results, such as the "lost-in-the-middle" problem. In this work, we aim to improve the robustness and reliability of the RAG workflow in the medical domain. Specifically, we propose a map-reduce strategy, BriefContext, to combat the "lost-in-the-middle" issue without modifying the model weights. We demonstrated the advantage of the workflow with various LLM backbones and on multiple QA datasets. This method promises to improve the safety and reliability of LLMs deployed in healthcare domains.

Title: SimGRAG: Leveraging Similar Subgraphs for Knowledge Graphs Driven Retrieval-Augmented Generation

• Authors: Yuzheng Cai, Zhenyue Guo, Yiwen Pei, Wanrui Bian, Weiguo Zheng
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15272
• Pdf URL: https://arxiv.org/pdf/2412.15272
• Copy Paste: [[2412.15272]] SimGRAG: Leveraging Similar Subgraphs for Knowledge Graphs Driven Retrieval-Augmented Generation(https://arxiv.org/abs/2412.15272)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have shown impressive versatility across various tasks. To eliminate its hallucinations, retrieval-augmented generation (RAG) has emerged as a powerful approach, leveraging external knowledge sources like knowledge graphs (KGs). In this paper, we study the task of KG-driven RAG and propose a novel Similar Graph Enhanced Retrieval-Augmented Generation (SimGRAG) method. It effectively addresses the challenge of aligning query texts and KG structures through a two-stage process: (1) query-to-pattern, which uses an LLM to transform queries into a desired graph pattern, and (2) pattern-to-subgraph, which quantifies the alignment between the pattern and candidate subgraphs using a graph semantic distance (GSD) metric. We also develop an optimized retrieval algorithm that efficiently identifies the top-$k$ subgraphs within 1-second latency on a 10-million-scale KG. Extensive experiments show that SimGRAG outperforms state-of-the-art KG-driven RAG methods in both question answering and fact verification, offering superior plug-and-play usability and scalability.

Title: Memory-Augmented Agent Training for Business Document Understanding

• Authors: Jiale Liu, Yifan Zeng, Malte Højmark-Bertelsen, Marie Normann Gadeberg, Huazheng Wang, Qingyun Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15274
• Pdf URL: https://arxiv.org/pdf/2412.15274
• Copy Paste: [[2412.15274]] Memory-Augmented Agent Training for Business Document Understanding(https://arxiv.org/abs/2412.15274)
• Keywords: extraction, large language model
• Abstract: Traditional enterprises face significant challenges in processing business documents, where tasks like extracting transport references from invoices remain largely manual despite their crucial role in logistics operations. While Large Language Models offer potential automation, their direct application to specialized business domains often yields unsatisfactory results. We introduce Matrix (Memory-Augmented agent Training through Reasoning and Iterative eXploration), a novel paradigm that enables LLM agents to progressively build domain expertise through experience-driven memory refinement and iterative learning. To validate this approach, we collaborate with one of the world's largest logistics companies to create a dataset of Universal Business Language format invoice documents, focusing on the task of transport reference extraction. Experiments demonstrate that Matrix outperforms prompting a single LLM by 30.3%, vanilla LLM agent by 35.2%. We further analyze the metrics of the optimized systems and observe that the agent system requires less API calls, fewer costs and can analyze longer documents on average. Our methods establish a new approach to transform general-purpose LLMs into specialized business tools through systematic memory enhancement in document processing tasks.

Title: Fooling LLM graders into giving better grades through neural activity guided adversarial prompting

• Authors: Atsushi Yamamura, Surya Ganguli
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15275
• Pdf URL: https://arxiv.org/pdf/2412.15275
• Copy Paste: [[2412.15275]] Fooling LLM graders into giving better grades through neural activity guided adversarial prompting(https://arxiv.org/abs/2412.15275)
• Keywords: security, attack, large language model
• Abstract: The deployment of artificial intelligence (AI) in critical decision-making and evaluation processes raises concerns about inherent biases that malicious actors could exploit to distort decision outcomes. We propose a systematic method to reveal such biases in AI evaluation systems and apply it to automated essay grading as an example. Our approach first identifies hidden neural activity patterns that predict distorted decision outcomes and then optimizes an adversarial input suffix to amplify such patterns. We demonstrate that this combination can effectively fool large language model (LLM) graders into assigning much higher grades than humans would. We further show that this white-box attack transfers to black-box attacks on other models, including commercial closed-source models like Gemini. They further reveal the existence of a "magic word" that plays a pivotal role in the efficacy of the attack. We trace the origin of this magic word bias to the structure of commonly-used chat templates for supervised fine-tuning of LLMs and show that a minor change in the template can drastically reduce the bias. This work not only uncovers vulnerabilities in current LLMs but also proposes a systematic method to identify and remove hidden biases, contributing to the goal of ensuring AI safety and security.

Title: Exploring Query Efficient Data Generation towards Data-free Model Stealing in Hard Label Setting

• Authors: Gaozheng Pei, Shaojie lyu, Ke Ma, Pinci Yang, Qianqian Xu, Yingfei Sun
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15276
• Pdf URL: https://arxiv.org/pdf/2412.15276
• Copy Paste: [[2412.15276]] Exploring Query Efficient Data Generation towards Data-free Model Stealing in Hard Label Setting(https://arxiv.org/abs/2412.15276)
• Keywords: attack, steal, data-free
• Abstract: Data-free model stealing involves replicating the functionality of a target model into a substitute model without accessing the target model's structure, parameters, or training data. The adversary can only access the target model's predictions for generated samples. Once the substitute model closely approximates the behavior of the target model, attackers can exploit its white-box characteristics for subsequent malicious activities, such as adversarial attacks. Existing methods within cooperative game frameworks often produce samples with high confidence for the prediction of the substitute model, which makes it difficult for the substitute model to replicate the behavior of the target model. This paper presents a new data-free model stealing approach called Query Efficient Data Generation (\textbf{QEDG}). We introduce two distinct loss functions to ensure the generation of sufficient samples that closely and uniformly align with the target model's decision boundary across multiple classes. Building on the limitation of current methods, which typically yield only one piece of supervised information per query, we propose the query-free sample augmentation that enables the acquisition of additional supervised information without increasing the number of queries. Motivated by theoretical analysis, we adopt the consistency rate metric, which more accurately evaluates the similarity between the substitute and target models. We conducted extensive experiments to verify the effectiveness of our proposed method, which achieved better performance with fewer queries compared to the state-of-the-art methods on the real \textbf{MLaaS} scenario and five datasets.

Title: Context-DPO: Aligning Language Models for Context-Faithfulness

• Authors: Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Shenghua Liu
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15280
• Pdf URL: https://arxiv.org/pdf/2412.15280
• Copy Paste: [[2412.15280]] Context-DPO: Aligning Language Models for Context-Faithfulness(https://arxiv.org/abs/2412.15280)
• Keywords: generative, large language model
• Abstract: Reliable responses from large language models (LLMs) require adherence to user instructions and retrieved information. While alignment techniques help LLMs align with human intentions and values, improving context-faithfulness through alignment remains underexplored. To address this, we propose $\textbf{Context-DPO}$, the first alignment method specifically designed to enhance LLMs' context-faithfulness. We introduce $\textbf{ConFiQA}$, a benchmark that simulates Retrieval-Augmented Generation (RAG) scenarios with knowledge conflicts to evaluate context-faithfulness. By leveraging faithful and stubborn responses to questions with provided context from ConFiQA, our Context-DPO aligns LLMs through direct preference optimization. Extensive experiments demonstrate that our Context-DPO significantly improves context-faithfulness, achieving 35% to 280% improvements on popular open-source models. Further analysis demonstrates that Context-DPO preserves LLMs' generative capabilities while providing interpretable insights into context utilization. Our code and data are released at this https URL

Title: A Systematic Examination of Preference Learning through the Lens of Instruction-Following

• Authors: Joongwon Kim, Anirudh Goyal, Aston Zhang, Bo Xiong, Rui Hou, Melanie Kambadur, Dhruv Mahajan, Hannaneh Hajishirzi, Liang Tan
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15282
• Pdf URL: https://arxiv.org/pdf/2412.15282
• Copy Paste: [[2412.15282]] A Systematic Examination of Preference Learning through the Lens of Instruction-Following(https://arxiv.org/abs/2412.15282)
• Keywords: large language model
• Abstract: Preference learning is a widely adopted post-training technique that aligns large language models (LLMs) to human preferences and improves specific downstream task capabilities. In this work we systematically investigate how specific attributes of preference datasets affect the alignment and downstream performance of LLMs in instruction-following tasks. We use a novel synthetic data generation pipeline to generate 48,000 unique instruction-following prompts with combinations of 23 verifiable constraints that enable fine-grained and automated quality assessments of model responses. With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS) - to obtain pairs of (chosen, rejected) responses. Then, we perform experiments investigating the effects of (1) the presence of shared prefixes between the chosen and rejected responses, (2) the contrast and quality of the chosen, rejected responses and (3) the complexity of the training prompts. Our experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements and greater stability across challenging training configurations. High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance by balancing diversity and learning efficiency. Additionally, training on prompts of moderate difficulty leads to better generalization across tasks, even for more complex evaluation scenarios, compared to overly challenging prompts. Our findings provide actionable insights into optimizing preference data curation for instruction-following tasks, offering a scalable and effective framework for enhancing LLM training and alignment.

Title: Channel Merging: Preserving Specialization for Merged Experts

• Authors: Mingyang Zhang, Jing Liu, Ganggui Ding, Xinyi Yu, Linlin Ou, Bohan Zhuang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15283
• Pdf URL: https://arxiv.org/pdf/2412.15283
• Copy Paste: [[2412.15283]] Channel Merging: Preserving Specialization for Merged Experts(https://arxiv.org/abs/2412.15283)
• Keywords: large language model
• Abstract: Lately, the practice of utilizing task-specific fine-tuning has been implemented to improve the performance of large language models (LLM) in subsequent tasks. Through the integration of diverse LLMs, the overall competency of LLMs is significantly boosted. Nevertheless, traditional ensemble methods are notably memory-intensive, necessitating the simultaneous loading of all specialized models into GPU memory. To address the inefficiency, model merging strategies have emerged, merging all LLMs into one model to reduce the memory footprint during inference. Despite these advances, model merging often leads to parameter conflicts and performance decline as the number of experts increases. Previous methods to mitigate these conflicts include post-pruning and partial merging. However, both approaches have limitations, particularly in terms of performance and storage efficiency when merged experts increase. To address these challenges, we introduce Channel Merging, a novel strategy designed to minimize parameter conflicts while enhancing storage efficiency. This method clusters and merges channel parameters based on their similarity to form several groups offline. By ensuring that only highly similar parameters are merged within each group, it significantly reduces parameter conflicts. During inference, we can instantly look up the expert parameters from the merged groups, preserving specialized knowledge. Our experiments demonstrate that Channel Merging consistently delivers high performance, matching unmerged models in tasks like English and Chinese reasoning, mathematical reasoning, and code generation. Moreover, it obtains results comparable to model ensemble with just 53% parameters when used with a task-specific router.

Title: Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining

• Authors: Steven Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15285
• Pdf URL: https://arxiv.org/pdf/2412.15285
• Copy Paste: [[2412.15285]] Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining(https://arxiv.org/abs/2412.15285)
• Keywords: large language model
• Abstract: Pretraining large language models effectively requires strategic data selection, blending and ordering. However, key details about data mixtures especially their scalability to longer token horizons and larger model sizes remain underexplored due to limited disclosure by model developers. To address this, we formalize the concept of two-phase pretraining and conduct an extensive systematic study on how to select and mix data to maximize model accuracies for the two phases. Our findings illustrate that a two-phase approach for pretraining outperforms random data ordering and natural distribution of tokens by 3.4% and 17% on average accuracies. We provide in-depth guidance on crafting optimal blends based on quality of the data source and the number of epochs to be seen. We propose to design blends using downsampled data at a smaller scale of 1T tokens and then demonstrate effective scaling of our approach to larger token horizon of 15T tokens and larger model size of 25B model size. These insights provide a series of steps practitioners can follow to design and scale their data blends.

Title: Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models

• Authors: Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, Aleksandra Faust
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15287
• Pdf URL: https://arxiv.org/pdf/2412.15287
• Copy Paste: [[2412.15287]] Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models(https://arxiv.org/abs/2412.15287)
• Keywords: large language model
• Abstract: Recent studies have indicated that effectively utilizing inference-time compute is crucial for attaining better performance from large language models (LLMs). In this work, we propose a novel inference-aware fine-tuning paradigm, in which the model is fine-tuned in a manner that directly optimizes the performance of the inference-time strategy. We study this paradigm using the simple yet effective Best-of-N (BoN) inference strategy, in which a verifier selects the best out of a set of LLM-generated responses. We devise the first imitation learning and reinforcement learning~(RL) methods for BoN-aware fine-tuning, overcoming the challenging, non-differentiable argmax operator within BoN. We empirically demonstrate that our BoN-aware models implicitly learn a meta-strategy that interleaves best responses with more diverse responses that might be better suited to a test-time input -- a process reminiscent of the exploration-exploitation trade-off in RL. Our experiments demonstrate the effectiveness of BoN-aware fine-tuning in terms of improved performance and inference-time compute. In particular, we show that our methods improve the Bo32 performance of Gemma 2B on Hendrycks MATH from 26.8% to 30.8%, and pass@32 from 60.0% to 67.0%, as well as the pass@16 on HumanEval from 61.6% to 67.1%.

Title: SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage

• Authors: Xiaoning Dong, Wenbo Hu, Wei Xu, Tianxing He
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15289
• Pdf URL: https://arxiv.org/pdf/2412.15289
• Copy Paste: [[2412.15289]] SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage(https://arxiv.org/abs/2412.15289)
• Keywords: secure, attack, large language model
• Abstract: Large language models (LLMs) have made significant advancements across various tasks, but their safety alignment remain a major concern. Exploring jailbreak prompts can expose LLMs' vulnerabilities and guide efforts to secure them. Existing methods primarily design sophisticated instructions for the LLM to follow, or rely on multiple iterations, which could hinder the performance and efficiency of jailbreaks. In this work, we propose a novel jailbreak paradigm, Simple Assistive Task Linkage (SATA), which can effectively circumvent LLM safeguards and elicit harmful responses. Specifically, SATA first masks harmful keywords within a malicious query to generate a relatively benign query containing one or multiple [MASK] special tokens. It then employs a simple assistive task such as a masked language model task or an element lookup by position task to encode the semantics of the masked keywords. Finally, SATA links the assistive task with the masked query to jointly perform the jailbreak. Extensive experiments show that SATA achieves state-of-the-art performance and outperforms baselines by a large margin. Specifically, on AdvBench dataset, with mask language model (MLM) assistive task, SATA achieves an overall attack success rate (ASR) of 85% and harmful score (HS) of 4.57, and with element lookup by position (ELP) assistive task, SATA attains an overall ASR of 76% and HS of 4.43.

Title: A Large-scale Empirical Study on Large Language Models for Election Prediction

• Authors: Chenxiao Yu, Zhaotian Weng, Yuangang Li, Zheng Li, Xiyang Hu, Yue Zhao
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2412.15291
• Pdf URL: https://arxiv.org/pdf/2412.15291
• Copy Paste: [[2412.15291]] A Large-scale Empirical Study on Large Language Models for Election Prediction(https://arxiv.org/abs/2412.15291)
• Keywords: large language model
• Abstract: Can Large Language Models (LLMs) accurately predict election outcomes? While LLMs have demonstrated impressive performance in healthcare, legal analysis, and creative applications, their capabilities in election forecasting remain uncertain. Notably, election prediction poses unique challenges: limited voter-level data, evolving political contexts, and the complexity of modeling human behavior. In the first part of this paper, we explore and introduce a multi-step reasoning framework for election prediction, which systematically integrates demographic, ideological, and time-sensitive factors. Validated on 2016 and 2020 real-world data and extensive synthetic personas, our approach adapts to changing political landscapes, reducing bias and significantly improving predictive accuracy. We further apply our pipeline to the 2024 U.S. presidential election, illustrating its ability to generalize beyond observed historical data. Beyond enhancing accuracy, the second part of the paper provides insights into the broader implications of LLM-based election forecasting. We identify potential political biases embedded in pretrained corpora, examine how demographic patterns can become exaggerated, and suggest strategies for mitigating these issues. Together, this project, a large-scale LLM empirical study, advances the accuracy of election predictions and establishes directions for more balanced, transparent, and context-aware modeling in political science research and practice.

Title: A Universal Model for Human Mobility Prediction

• Authors: Qingyue Long, Yuan Yuan, Yong Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15294
• Pdf URL: https://arxiv.org/pdf/2412.15294
• Copy Paste: [[2412.15294]] A Universal Model for Human Mobility Prediction(https://arxiv.org/abs/2412.15294)
• Keywords: diffusion, transformer
• Abstract: Predicting human mobility is crucial for urban planning, traffic control, and emergency response. Mobility behaviors can be categorized into individual and collective, and these behaviors are recorded by diverse mobility data, such as individual trajectory and crowd flow. As different modalities of mobility data, individual trajectory and crowd flow have a close coupling relationship. Crowd flows originate from the bottom-up aggregation of individual trajectories, while the constraints imposed by crowd flows shape these individual trajectories. Existing mobility prediction methods are limited to single tasks due to modal gaps between individual trajectory and crowd flow. In this work, we aim to unify mobility prediction to break through the limitations of task-specific models. We propose a universal human mobility prediction model (named UniMob), which can be applied to both individual trajectory and crowd flow. UniMob leverages a multi-view mobility tokenizer that transforms both trajectory and flow data into spatiotemporal tokens, facilitating unified sequential modeling through a diffusion transformer architecture. To bridge the gap between the different characteristics of these two data modalities, we implement a novel bidirectional individual and collective alignment mechanism. This mechanism enables learning common spatiotemporal patterns from different mobility data, facilitating mutual enhancement of both trajectory and flow predictions. Extensive experiments on real-world datasets validate the superiority of our model over state-of-the-art baselines in trajectory and flow prediction. Especially in noisy and scarce data scenarios, our model achieves the highest performance improvement of more than 14% and 25% in MAPE and Accuracy@5.

Title: Confidence in the Reasoning of Large Language Models

• Authors: Yudi Pawitan, Chris Holmes
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15296
• Pdf URL: https://arxiv.org/pdf/2412.15296
• Copy Paste: [[2412.15296]] Confidence in the Reasoning of Large Language Models(https://arxiv.org/abs/2412.15296)
• Keywords: large language model
• Abstract: There is a growing literature on reasoning by large language models (LLMs), but the discussion on the uncertainty in their responses is still lacking. Our aim is to assess the extent of confidence that LLMs have in their answers and how it correlates with accuracy. Confidence is measured (i) qualitatively in terms of persistence in keeping their answer when prompted to reconsider, and (ii) quantitatively in terms of self-reported confidence score. We investigate the performance of three LLMs -- GPT4o, GPT4-turbo and Mistral -- on two benchmark sets of questions on causal judgement and formal fallacies and a set of probability and statistical puzzles and paradoxes. Although the LLMs show significantly better performance than random guessing, there is a wide variability in their tendency to change their initial answers. There is a positive correlation between qualitative confidence and accuracy, but the overall accuracy for the second answer is often worse than for the first answer. There is a strong tendency to overstate the self-reported confidence score. Confidence is only partially explained by the underlying token-level probability. The material effects of prompting on qualitative confidence and the strong tendency for overconfidence indicate that current LLMs do not have any internally coherent sense of confidence.

Title: A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation

• Authors: Bhaskarjit Sarmah, Kriti Dutta, Anna Grigoryan, Sachin Tiwari, Stefano Pasquali, Dhagash Mehta
• Subjects: cs.CL, cs.AI, cs.LG, q-fin.ST, stat.ME
• Abstract URL: https://arxiv.org/abs/2412.15298
• Pdf URL: https://arxiv.org/pdf/2412.15298
• Copy Paste: [[2412.15298]] A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation(https://arxiv.org/abs/2412.15298)
• Keywords: large language model
• Abstract: We argue that the Declarative Self-improving Python (DSPy) optimizers are a way to align the large language model (LLM) prompts and their evaluations to the human annotations. We present a comparative analysis of five teleprompter algorithms, namely, Cooperative Prompt Optimization (COPRO), Multi-Stage Instruction Prompt Optimization (MIPRO), BootstrapFewShot, BootstrapFewShot with Optuna, and K-Nearest Neighbor Few Shot, within the DSPy framework with respect to their ability to align with human evaluations. As a concrete example, we focus on optimizing the prompt to align hallucination detection (using LLM as a judge) to human annotated ground truth labels for a publicly available benchmark dataset. Our experiments demonstrate that optimized prompts can outperform various benchmark methods to detect hallucination, and certain telemprompters outperform the others in at least these experiments.

Title: Tokenphormer: Structure-aware Multi-token Graph Transformer for Node Classification

• Authors: Zijie Zhou, Zhaoqi Lu, Xuekai Wei, Rongqin Chen, Shenghui Zhang, Pak Lon Ip, Leong Hou U
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15302
• Pdf URL: https://arxiv.org/pdf/2412.15302
• Copy Paste: [[2412.15302]] Tokenphormer: Structure-aware Multi-token Graph Transformer for Node Classification(https://arxiv.org/abs/2412.15302)
• Keywords: transformer
• Abstract: Graph Neural Networks (GNNs) are widely used in graph data mining tasks. Traditional GNNs follow a message passing scheme that can effectively utilize local and structural information. However, the phenomena of over-smoothing and over-squashing limit the receptive field in message passing processes. Graph Transformers were introduced to address these issues, achieving a global receptive field but suffering from the noise of irrelevant nodes and loss of structural information. Therefore, drawing inspiration from fine-grained token-based representation learning in Natural Language Processing (NLP), we propose the Structure-aware Multi-token Graph Transformer (Tokenphormer), which generates multiple tokens to effectively capture local and structural information and explore global information at different levels of granularity. Specifically, we first introduce the walk-token generated by mixed walks consisting of four walk types to explore the graph and capture structure and contextual information flexibly. To ensure local and global information coverage, we also introduce the SGPM-token (obtained through the Self-supervised Graph Pre-train Model, SGPM) and the hop-token, extending the length and density limit of the walk-token, respectively. Finally, these expressive tokens are fed into the Transformer model to learn node representations collaboratively. Experimental results demonstrate that the capability of the proposed Tokenphormer can achieve state-of-the-art performance on node classification tasks.

Title: Self-Evolution Knowledge Distillation for LLM-based Machine Translation

• Authors: Yuncheng Song, Liang Ding, Changtong Zan, Shujian Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15303
• Pdf URL: https://arxiv.org/pdf/2412.15303
• Copy Paste: [[2412.15303]] Self-Evolution Knowledge Distillation for LLM-based Machine Translation(https://arxiv.org/abs/2412.15303)
• Keywords: large language model
• Abstract: Knowledge distillation (KD) has shown great promise in transferring knowledge from larger teacher models to smaller student models. However, existing KD strategies for large language models often minimize output distributions between student and teacher models indiscriminately for each token. This overlooks the imbalanced nature of tokens and their varying transfer difficulties. In response, we propose a distillation strategy called Self-Evolution KD. The core of this approach involves dynamically integrating teacher distribution and one-hot distribution of ground truth into the student distribution as prior knowledge, which promotes the distillation process. It adjusts the ratio of prior knowledge based on token learning difficulty, fully leveraging the teacher model's potential. Experimental results show our method brings an average improvement of approximately 1.4 SacreBLEU points across four translation directions in the WMT22 test sets. Further analysis indicates that the improvement comes from better knowledge transfer from teachers, confirming our hypothesis.

Title: TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers

• Authors: Savitha Viswanadh Kandala, Pramuka Medaranga, Ambuj Varshney
• Subjects: cs.LG, cs.DC, cs.ET, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.15304
• Pdf URL: https://arxiv.org/pdf/2412.15304
• Copy Paste: [[2412.15304]] TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers(https://arxiv.org/abs/2412.15304)
• Keywords: privacy
• Abstract: Language models have gained significant interest due to their general-purpose capabilities, which appear to emerge as models are scaled to increasingly larger parameter sizes. However, these large models impose stringent requirements on computing systems, necessitating significant memory and processing requirements for inference. This makes performing inference on mobile and edge devices challenging, often requiring invocating remotely-hosted models via network calls. Remote inference, in turn, introduces issues like latency, unreliable network connectivity, and privacy concerns. To address these challenges, we explored the possibility of deviating from the trend of increasing model size. Instead, we hypothesize that much smaller models (~30-120M parameters) can outperform their larger counterparts for specific tasks by carefully curating the data used for pre-training and fine-tuning. We investigate this within the context of deploying edge-device models to support sensing applications. We trained several foundational models through a systematic study and found that small models can run locally on edge devices, achieving high token rates and accuracy. Based on these findings, we developed a framework that allows users to train foundational models tailored to their specific applications and deploy them at the edge.

Title: MIETT: Multi-Instance Encrypted Traffic Transformer for Encrypted Traffic Classification

• Authors: Xu-Yang Chen, Lu Han, De-Chuan Zhan, Han-Jia Ye
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15306
• Pdf URL: https://arxiv.org/pdf/2412.15306
• Copy Paste: [[2412.15306]] MIETT: Multi-Instance Encrypted Traffic Transformer for Encrypted Traffic Classification(https://arxiv.org/abs/2412.15306)
• Keywords: security, transformer
• Abstract: Network traffic includes data transmitted across a network, such as web browsing and file transfers, and is organized into packets (small units of data) and flows (sequences of packets exchanged between two endpoints). Classifying encrypted traffic is essential for detecting security threats and optimizing network management. Recent advancements have highlighted the superiority of foundation models in this task, particularly for their ability to leverage large amounts of unlabeled data and demonstrate strong generalization to unseen data. However, existing methods that focus on token-level relationships fail to capture broader flow patterns, as tokens, defined as sequences of hexadecimal digits, typically carry limited semantic information in encrypted traffic. These flow patterns, which are crucial for traffic classification, arise from the interactions between packets within a flow, not just their internal structure. To address this limitation, we propose a Multi-Instance Encrypted Traffic Transformer (MIETT), which adopts a multi-instance approach where each packet is treated as a distinct instance within a larger bag representing the entire flow. This enables the model to capture both token-level and packet-level relationships more effectively through Two-Level Attention (TLA) layers, improving the model's ability to learn complex packet dynamics and flow patterns. We further enhance the model's understanding of temporal and flow-specific dynamics by introducing two novel pre-training tasks: Packet Relative Position Prediction (PRPP) and Flow Contrastive Learning (FCL). After fine-tuning, MIETT achieves state-of-the-art (SOTA) results across five datasets, demonstrating its effectiveness in classifying encrypted traffic and understanding complex network behaviors. Code is available at \url{this https URL}.

Title: ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese

• Authors: Tran Thai Hoa, Tran Quang Duy, Khanh Quoc Tran, Kiet Van Nguyen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15308
• Pdf URL: https://arxiv.org/pdf/2412.15308
• Copy Paste: [[2412.15308]] ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese(https://arxiv.org/abs/2412.15308)
• Keywords: robust, large language model
• Abstract: The rapid spread of information in the digital age highlights the critical need for effective fact-checking tools, particularly for languages with limited resources, such as Vietnamese. In response to this challenge, we introduce ViFactCheck, the first publicly available benchmark dataset designed specifically for Vietnamese fact-checking across multiple online news domains. This dataset contains 7,232 human-annotated pairs of claim-evidence combinations sourced from reputable Vietnamese online news, covering 12 diverse topics. It has been subjected to a meticulous annotation process to ensure high quality and reliability, achieving a Fleiss Kappa inter-annotator agreement score of 0.83. Our evaluation leverages state-of-the-art pre-trained and large language models, employing fine-tuning and prompting techniques to assess performance. Notably, the Gemma model demonstrated superior effectiveness, with an impressive macro F1 score of 89.90%, thereby establishing a new standard for fact-checking benchmarks. This result highlights the robust capabilities of Gemma in accurately identifying and verifying facts in Vietnamese. To further promote advances in fact-checking technology and improve the reliability of digital media, we have made the ViFactCheck dataset, model checkpoints, fact-checking pipelines, and source code freely available on GitHub. This initiative aims to inspire further research and enhance the accuracy of information in low-resource languages.

Title: Conceptual In-Context Learning and Chain of Concepts: Solving Complex Conceptual Problems Using Large Language Models

• Authors: Nishtha N. Vaidya, Thomas Runkler, Thomas Hubauer, Veronika Haderlein-Hoegberg, Maja Mlicic Brandt
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15309
• Pdf URL: https://arxiv.org/pdf/2412.15309
• Copy Paste: [[2412.15309]] Conceptual In-Context Learning and Chain of Concepts: Solving Complex Conceptual Problems Using Large Language Models(https://arxiv.org/abs/2412.15309)
• Keywords: large language model
• Abstract: Science and engineering problems fall in the category of complex conceptual problems that require specific conceptual information (CI) like math/logic -related know-how, process information, or engineering guidelines to solve them. Large Language Models (LLMs) are promising agents to solve such complex conceptual problems due to their implications in advancing engineering and science tasks like assisted problem-solving. But vanilla LLMs, trained on open-world data, lack the necessary CI. In this work, we specifically explore shallow customization methods (SCMs) of LLMs for solving complex conceptual problems. We propose two novel SCM algorithms for LLM, to augment LLMs with CI and enable LLMs to solve complex conceptual problems: Conceptual In-Context Learning (C-ICL) and Chain of Concepts (CoC). The problem tackled in this paper is generation of proprietary data models in the engineering/industry domain based on conceptual information in data modelling guidelines. We evaluate our algorithms on varied sizes of the OpenAI LLMs against four evaluation metrics related to syntactic and semantic correctness, time and cost incurred. The proposed algorithms perform better than currently popular LLM SCMs like In-context Learning (ICL) and Chain of Thoughts (CoT). It was observed that as compared to CoT, response correctness increased by 30.6% and 29.88% for the new SCMs C-ICL and CoC respectively. Qualitative analysis suggests that the proposed new SCMs activate emergent capabilities in LLMs, previously unobserved in the existing SCMs. They make problem-solving processes more transparent and reduce hallucinations and the tendency of model responses to copy examples from prompts (parroting).

Title: Re-evaluating Group Robustness via Adaptive Class-Specific Scaling

• Authors: Seonguk Seo, Bohyung Han
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15311
• Pdf URL: https://arxiv.org/pdf/2412.15311
• Copy Paste: [[2412.15311]] Re-evaluating Group Robustness via Adaptive Class-Specific Scaling(https://arxiv.org/abs/2412.15311)
• Keywords: robust
• Abstract: Group distributionally robust optimization, which aims to improve robust accuracies -- worst-group and unbiased accuracies -- is a prominent algorithm used to mitigate spurious correlations and address dataset bias. Although existing approaches have reported improvements in robust accuracies, these gains often come at the cost of average accuracy due to inherent trade-offs. To control this trade-off flexibly and efficiently, we propose a simple class-specific scaling strategy, directly applicable to existing debiasing algorithms with no additional training. We further develop an instance-wise adaptive scaling technique to alleviate this trade-off, even leading to improvements in both robust and average accuracies. Our approach reveals that a naïve ERM baseline matches or even outperforms the recent debiasing methods by simply adopting the class-specific scaling technique. Additionally, we introduce a novel unified metric that quantifies the trade-off between the two accuracies as a scalar value, allowing for a comprehensive evaluation of existing algorithms. By tackling the inherent trade-off and offering a performance landscape, our approach provides valuable insights into robust techniques beyond just robust accuracy. We validate the effectiveness of our framework through experiments across datasets in computer vision and natural language processing domains.

Title: PCA-Featured Transformer for Jamming Detection in 5G UAV Networks

• Authors: Joseanne Viana, Hamed Farkhari, Pedro Sebastiao, Victor P Gil Jimenez, Lester Ho
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15312
• Pdf URL: https://arxiv.org/pdf/2412.15312
• Copy Paste: [[2412.15312]] PCA-Featured Transformer for Jamming Detection in 5G UAV Networks(https://arxiv.org/abs/2412.15312)
• Keywords: attack, robust, transformer
• Abstract: Jamming attacks pose a threat to Unmanned Aerial Vehicle (UAV) wireless communication systems, potentially disrupting essential services and compromising network reliability. Current detection approaches struggle with sophisticated artificial intelligence (AI) jamming techniques that adapt their patterns while existing machine learning solutions often require extensive feature engineering and fail to capture complex temporal dependencies in attack signatures. Furthermore, 5G networks using either Time Division Duplex (TDD) or Frequency Division Duplex (FDD) methods can face service degradation from intentional interference sources. To address these challenges, we present a novel transformer-based deep learning framework for jamming detection with Principal Component Analysis (PCA) added features. Our architecture leverages the transformer's self-attention mechanism to capture complex temporal dependencies and spatial correlations in wireless signal characteristics, enabling more robust jamming detection techniques. The U-shaped model incorporates a modified transformer encoder that processes signal features including received signal strength indicator (RSSI) and signal-to-noise ratio (SINR) measurements, alongside a specialized positional encoding scheme that accounts for the periodic nature of wireless signals. In addition, we propose a batch size scheduler and implement chunking techniques to optimize training convergence for time series data. These advancements contribute to achieving up to a ten times improvement in training speed within the advanced U-shaped encoder-decoder model introduced. Simulation results demonstrate that our approach achieves a detection accuracy of 90.33 \% in Line-of-Sight (LoS) and 84.35 % in non-Line-of-Sight (NLoS) and outperforms machine learning methods and existing deep learning solutions such as the XGBoost (XGB) classifier in approximately 4%.

Title: Eliciting Causal Abilities in Large Language Models for Reasoning Tasks

• Authors: Yajing Wang, Zongwei Luo, Jingzhe Wang, Zhanke Zhou, Yongqiang Chen, Bo Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15314
• Pdf URL: https://arxiv.org/pdf/2412.15314
• Copy Paste: [[2412.15314]] Eliciting Causal Abilities in Large Language Models for Reasoning Tasks(https://arxiv.org/abs/2412.15314)
• Keywords: interpretability, large language model
• Abstract: Prompt optimization automatically refines prompting expressions, unlocking the full potential of LLMs in downstream tasks. However, current prompt optimization methods are costly to train and lack sufficient interpretability. This paper proposes enhancing LLMs' reasoning performance by eliciting their causal inference ability from prompting instructions to correct answers. Specifically, we introduce the Self-Causal Instruction Enhancement (SCIE) method, which enables LLMs to generate high-quality, low-quantity observational data, then estimates the causal effect based on these data, and ultimately generates instructions with the optimized causal effect. In SCIE, the instructions are treated as the treatment, and textual features are used to process natural language, establishing causal relationships through treatments between instructions and downstream tasks. Additionally, we propose applying Object-Relational (OR) principles, where the uncovered causal relationships are treated as the inheritable class across task objects, ensuring low-cost reusability. Extensive experiments demonstrate that our method effectively generates instructions that enhance reasoning performance with reduced training cost of prompts, leveraging interpretable textual features to provide actionable insights.

Title: Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models

• Authors: Reza Shirkavand, Peiran Yu, Shangqian Gao, Gowthami Somepalli, Tom Goldstein, Heng Huang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15341
• Pdf URL: https://arxiv.org/pdf/2412.15341
• Copy Paste: [[2412.15341]] Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models(https://arxiv.org/abs/2412.15341)
• Keywords: diffusion, generative
• Abstract: Recent advances in diffusion generative models have yielded remarkable progress. While the quality of generated content continues to improve, these models have grown considerably in size and complexity. This increasing computational burden poses significant challenges, particularly in resource-constrained deployment scenarios such as mobile devices. The combination of model pruning and knowledge distillation has emerged as a promising solution to reduce computational demands while preserving generation quality. However, this technique inadvertently propagates undesirable behaviors, including the generation of copyrighted content and unsafe concepts, even when such instances are absent from the fine-tuning dataset. In this paper, we propose a novel bilevel optimization framework for pruned diffusion models that consolidates the fine-tuning and unlearning processes into a unified phase. Our approach maintains the principal advantages of distillation-namely, efficient convergence and style transfer capabilities-while selectively suppressing the generation of unwanted content. This plug-in framework is compatible with various pruning and concept unlearning methods, facilitating efficient, safe deployment of diffusion models in controlled environments.

Title: Large Language Models on Small Resource-Constrained Systems: Performance Characterization, Analysis and Trade-offs

• Authors: Liam Seymour, Basar Kutukcu, Sabur Baidya
• Subjects: cs.LG, cs.CC
• Abstract URL: https://arxiv.org/abs/2412.15352
• Pdf URL: https://arxiv.org/pdf/2412.15352
• Copy Paste: [[2412.15352]] Large Language Models on Small Resource-Constrained Systems: Performance Characterization, Analysis and Trade-offs(https://arxiv.org/abs/2412.15352)
• Keywords: security, privacy, transformer, generative, large language model
• Abstract: Generative AI like the Large Language Models (LLMs) has become more available for the general consumer in recent years. Publicly available services, e.g., ChatGPT, perform token generation on networked cloud server hardware, effectively removing the hardware entry cost for end users. However, the reliance on network access for these services, privacy and security risks involved, and sometimes the needs of the application make it necessary to run LLMs locally on edge devices. A significant amount of research has been done on optimization of LLMs and other transformer-based models on non-networked, resource-constrained devices, but they typically target older hardware. Our research intends to provide a 'baseline' characterization of more recent commercially available embedded hardware for LLMs, and to provide a simple utility to facilitate batch testing LLMs on recent Jetson hardware. We focus on the latest line of NVIDIA Jetson devices (Jetson Orin), and a set of publicly available LLMs (Pythia) ranging between 70 million and 1.4 billion parameters. Through detailed experimental evaluation with varying software and hardware parameters, we showcase trade-off spaces and optimization choices. Additionally, we design our testing structure to facilitate further research that involves performing batch LLM testing on Jetson hardware.

Title: GeoPro-Net: Learning Interpretable Spatiotemporal Prediction Models through Statistically-Guided Geo-Prototyping

• Authors: Bang An, Xun Zhou, Zirui Zhou, Ronilo Ragodos, Zenglin Xu, Jun Luo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15353
• Pdf URL: https://arxiv.org/pdf/2412.15353
• Copy Paste: [[2412.15353]] GeoPro-Net: Learning Interpretable Spatiotemporal Prediction Models through Statistically-Guided Geo-Prototyping(https://arxiv.org/abs/2412.15353)
• Keywords: interpretability
• Abstract: The problem of forecasting spatiotemporal events such as crimes and accidents is crucial to public safety and city management. Besides accuracy, interpretability is also a key requirement for spatiotemporal forecasting models to justify the decisions. Interpretation of the spatiotemporal forecasting mechanism is, however, challenging due to the complexity of multi-source spatiotemporal features, the non-intuitive nature of spatiotemporal patterns for non-expert users, and the presence of spatial heterogeneity in the data. Currently, no existing deep learning model intrinsically interprets the complex predictive process learned from multi-source spatiotemporal features. To bridge the gap, we propose GeoPro-Net, an intrinsically interpretable spatiotemporal model for spatiotemporal event forecasting problems. GeoPro-Net introduces a novel Geo-concept convolution operation, which employs statistical tests to extract predictive patterns in the input as Geo-concepts, and condenses the Geo-concept-encoded input through interpretable channel fusion and geographic-based pooling. In addition, GeoPro-Net learns different sets of prototypes of concepts inherently, and projects them to real-world cases for interpretation. Comprehensive experiments and case studies on four real-world datasets demonstrate that GeoPro-Net provides better interpretability while still achieving competitive prediction performance compared with state-of-the-art baselines.

Title: Dataset Augmentation by Mixing Visual Concepts

• Authors: Abdullah Al Rahat, Hemanth Venkateswara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15358
• Pdf URL: https://arxiv.org/pdf/2412.15358
• Copy Paste: [[2412.15358]] Dataset Augmentation by Mixing Visual Concepts(https://arxiv.org/abs/2412.15358)
• Keywords: diffusion
• Abstract: This paper proposes a dataset augmentation method by fine-tuning pre-trained diffusion models. Generating images using a pre-trained diffusion model with textual conditioning often results in domain discrepancy between real data and generated images. We propose a fine-tuning approach where we adapt the diffusion model by conditioning it with real images and novel text embeddings. We introduce a unique procedure called Mixing Visual Concepts (MVC) where we create novel text embeddings from image captions. The MVC enables us to generate multiple images which are diverse and yet similar to the real data enabling us to perform effective dataset augmentation. We perform comprehensive qualitative and quantitative evaluations with the proposed dataset augmentation approach showcasing both coarse-grained and finegrained changes in generated images. Our approach outperforms state-of-the-art augmentation techniques on benchmark classification tasks.

Title: Decade of Natural Language Processing in Chronic Pain: A Systematic Review

• Authors: Swati Rajwal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15360
• Pdf URL: https://arxiv.org/pdf/2412.15360
• Copy Paste: [[2412.15360]] Decade of Natural Language Processing in Chronic Pain: A Systematic Review(https://arxiv.org/abs/2412.15360)
• Keywords: transformer
• Abstract: In recent years, the intersection of Natural Language Processing (NLP) and public health has opened innovative pathways for investigating various domains, including chronic pain in textual datasets. Despite the promise of NLP in chronic pain, the literature is dispersed across various disciplines, and there is a need to consolidate existing knowledge, identify knowledge gaps in the literature, and inform future research directions in this emerging field. This review aims to investigate the state of the research on NLP-based interventions designed for chronic pain research. A search strategy was formulated and executed across PubMed, Web of Science, IEEE Xplore, Scopus, and ACL Anthology to find studies published in English between 2014 and 2024. After screening 132 papers, 26 studies were included in the final review. Key findings from this review underscore the significant potential of NLP techniques to address pressing challenges in chronic pain research. The past 10 years in this field have showcased the utilization of advanced methods (transformers like RoBERTa and BERT) achieving high-performance metrics (e.g., F1>0.8) in classification tasks, while unsupervised approaches like Latent Dirichlet Allocation (LDA) and k-means clustering have proven effective for exploratory analyses. Results also reveal persistent challenges such as limited dataset diversity, inadequate sample sizes, and insufficient representation of underrepresented populations. Future research studies should explore multimodal data validation systems, context-aware mechanistic modeling, and the development of standardized evaluation metrics to enhance reproducibility and equity in chronic pain research.

Title: Spatiotemporally Coherent Probabilistic Generation of Weather from Climate

• Authors: Jonathan Schmidt, Luca Schmidt, Felix Strnad, Nicole Ludwig, Philipp Hennig
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2412.15361
• Pdf URL: https://arxiv.org/pdf/2412.15361
• Copy Paste: [[2412.15361]] Spatiotemporally Coherent Probabilistic Generation of Weather from Climate(https://arxiv.org/abs/2412.15361)
• Keywords: diffusion, generative
• Abstract: Local climate information is crucial for impact assessment and decision-making, yet coarse global climate simulations cannot capture small-scale phenomena. Current statistical downscaling methods infer these phenomena as temporally decoupled spatial patches. However, to preserve physical properties, estimating spatio-temporally coherent high-resolution weather dynamics for multiple variables across long time horizons is crucial. We present a novel generative approach that uses a score-based diffusion model trained on high-resolution reanalysis data to capture the statistical properties of local weather dynamics. After training, we condition on coarse climate model data to generate weather patterns consistent with the aggregate information. As this inference task is inherently uncertain, we leverage the probabilistic nature of diffusion models and sample multiple trajectories. We evaluate our approach with high-resolution reanalysis information before applying it to the climate model downscaling task. We then demonstrate that the model generates spatially and temporally coherent weather dynamics that align with global climate output.

Title: Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation

• Authors: Joanne Boisson, Zara Siddique, Hsuvas Borkakoty, Dimosthenis Antypas, Luis Espinosa Anke, Jose Camacho-Collados
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15375
• Pdf URL: https://arxiv.org/pdf/2412.15375
• Copy Paste: [[2412.15375]] Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation(https://arxiv.org/abs/2412.15375)
• Keywords: extraction, large language model
• Abstract: Extracting metaphors and analogies from free text requires high-level reasoning abilities such as abstraction and language understanding. Our study focuses on the extraction of the concepts that form metaphoric analogies in literary texts. To this end, we construct a novel dataset in this domain with the help of domain experts. We compare the out-of-the-box ability of recent large language models (LLMs) to structure metaphoric mappings from fragments of texts containing proportional analogies. The models are further evaluated on the generation of implicit elements of the analogy, which are indirectly suggested in the texts and inferred by human readers. The competitive results obtained by LLMs in our experiments are encouraging and open up new avenues such as automatically extracting analogies and metaphors from text instead of investing resources in domain experts to manually label data.

Title: Uncertainty-Guided Cross Attention Ensemble Mean Teacher for Semi-supervised Medical Image Segmentation

• Authors: Meghana Karri, Amit Soni Arya, Koushik Biswas, Nicol`o Gennaro, Vedat Cicek, Gorkem Durak, Yuri S. Velichko, Ulas Bagci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15380
• Pdf URL: https://arxiv.org/pdf/2412.15380
• Copy Paste: [[2412.15380]] Uncertainty-Guided Cross Attention Ensemble Mean Teacher for Semi-supervised Medical Image Segmentation(https://arxiv.org/abs/2412.15380)
• Keywords: robust, transformer, segmentation
• Abstract: This work proposes a novel framework, Uncertainty-Guided Cross Attention Ensemble Mean Teacher (UG-CEMT), for achieving state-of-the-art performance in semi-supervised medical image segmentation. UG-CEMT leverages the strengths of co-training and knowledge distillation by combining a Cross-attention Ensemble Mean Teacher framework (CEMT) inspired by Vision Transformers (ViT) with uncertainty-guided consistency regularization and Sharpness-Aware Minimization emphasizing uncertainty. UG-CEMT improves semi-supervised performance while maintaining a consistent network architecture and task setting by fostering high disparity between sub-networks. Experiments demonstrate significant advantages over existing methods like Mean Teacher and Cross-pseudo Supervision in terms of disparity, domain generalization, and medical image segmentation performance. UG-CEMT achieves state-of-the-art results on multi-center prostate MRI and cardiac MRI datasets, where object segmentation is particularly challenging. Our results show that using only 10\% labeled data, UG-CEMT approaches the performance of fully supervised methods, demonstrating its effectiveness in exploiting unlabeled data for robust medical image segmentation. The code is publicly available at \url{this https URL}

Title: Recovering WPA-3 Network Password by Bypassing the Simultaneous Authentication of Equals Handshake using Social Engineering Captive Portal

• Authors: Kyle Chadee, Wayne Goodridge, Koffka Khan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15381
• Pdf URL: https://arxiv.org/pdf/2412.15381
• Copy Paste: [[2412.15381]] Recovering WPA-3 Network Password by Bypassing the Simultaneous Authentication of Equals Handshake using Social Engineering Captive Portal(https://arxiv.org/abs/2412.15381)
• Keywords: security, protect, attack
• Abstract: Wi-Fi Protected Access 3 (WPA3) is the accepted standard for next generation wireless security. WPA3 comes with exciting new features that allows for increased security of Wi-Fi networks. One such feature is the Simultaneous Authentication of Equals (SAE) which is a protocol whereby passphrases are hashed using a Password Authenticated Key Exchange with keys from both the Access Point and the Client making the password resistant to offline dictionary attacks. (Harkins, Dan. 2019) This objective of this research paper seeks to bypass WPA3-SAE to acquire network password via Man-in-the-Middle attack and Social Engineering. This method can prove to be useful given that majority of network attacks stem from social engineering. For this research we would be looking at the security of WPA3 personal transition mode and capture the network password via a captive portal. Breaching the WPA3 network can be possible by building on various security flaws that was disclosed on WPA3 in 2021. Due to the discovery of Dragonblood downgrade attacks disclosed in 2019, identified that WPA2/3Handshakes could be acquired. A Man in the Middle attack proposed set up is carried out by using race conditions to deauthentication WPA3 network and then using a Raspberry Pi to spawn a rouge WPA3 network. As such, the handshake acquired can then be utilized as to verify the password that would be entered in the captive portal of the rouge WPA3 network. This research identified that the Password was able to be recovered from Social Engineering Captive Portal when Protected Management Frames are not implemented. This research also indicates that some devices are not able to connect to a WPA 3 transition network which is contradicting the Wi-Fi Alliance claim that it is backwards compatible with WPA2.

Title: Systematic Evaluation of Long-Context LLMs on Financial Concepts

• Authors: Lavanya Gupta, Saket Sharma, Yiyun Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15386
• Pdf URL: https://arxiv.org/pdf/2412.15386
• Copy Paste: [[2412.15386]] Systematic Evaluation of Long-Context LLMs on Financial Concepts(https://arxiv.org/abs/2412.15386)
• Keywords: robust, large language model
• Abstract: Long-context large language models (LC LLMs) promise to increase reliability of LLMs in real-world tasks requiring processing and understanding of long input documents. However, this ability of LC LLMs to reliably utilize their growing context windows remains under investigation. In this work, we evaluate the performance of state-of-the-art GPT-4 suite of LC LLMs in solving a series of progressively challenging tasks, as a function of factors such as context length, task difficulty, and position of key information by creating a real world financial news dataset. Our findings indicate that LC LLMs exhibit brittleness at longer context lengths even for simple tasks, with performance deteriorating sharply as task complexity increases. At longer context lengths, these state-of-the-art models experience catastrophic failures in instruction following resulting in degenerate outputs. Our prompt ablations also reveal unfortunate continued sensitivity to both the placement of the task instruction in the context window as well as minor markdown formatting. Finally, we advocate for more rigorous evaluation of LC LLMs by employing holistic metrics such as F1 (rather than recall) and reporting confidence intervals, thereby ensuring robust and conclusive findings.

Title: Maximising Histopathology Segmentation using Minimal Labels via Self-Supervision

• Authors: Zeeshan Nisar, Thomas Lampert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15389
• Pdf URL: https://arxiv.org/pdf/2412.15389
• Copy Paste: [[2412.15389]] Maximising Histopathology Segmentation using Minimal Labels via Self-Supervision(https://arxiv.org/abs/2412.15389)
• Keywords: segmentation
• Abstract: Histopathology, the microscopic examination of tissue samples, is essential for disease diagnosis and prognosis. Accurate segmentation and identification of key regions in histopathology images are crucial for developing automated solutions. However, state-of-art deep learning segmentation methods like UNet require extensive labels, which is both costly and time-consuming, particularly when dealing with multiple stainings. To mitigate this, multi-stain segmentation methods such as MDS1 and UDAGAN have been developed, which reduce the need for labels by requiring only one (source) stain to be labelled. Nonetheless, obtaining source stain labels can still be challenging, and segmentation models fail when they are unavailable. This article shows that through self-supervised pre-training, including SimCLR, BYOL, and a novel approach, HR-CS-CO, the performance of these segmentation methods (UNet, MDS1, and UDAGAN) can be retained even with 95% fewer labels. Notably, with self-supervised pre-training and using only 5% labels, the performance drops are minimal: 5.9% for UNet, 4.5% for MDS1, and 6.2% for UDAGAN, compared to their respective fully supervised counterparts (without pre-training, using 100% labels). The code is available from this https URL [to be made public upon acceptance].

Title: LG-Sleep: Local and Global Temporal Dependencies for Mice Sleep Scoring

• Authors: Shadi Sartipi, Mie Andersen, Natalie Hauglund, Celia Kjaerby, Verena Untiet, Maiken Nedergaard, Mujdat Cetin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15412
• Pdf URL: https://arxiv.org/pdf/2412.15412
• Copy Paste: [[2412.15412]] LG-Sleep: Local and Global Temporal Dependencies for Mice Sleep Scoring(https://arxiv.org/abs/2412.15412)
• Keywords: robust
• Abstract: Efficiently identifying sleep stages is crucial for unraveling the intricacies of sleep in both preclinical and clinical research. The labor-intensive nature of manual sleep scoring, demanding substantial expertise, has prompted a surge of interest in automated alternatives. Sleep studies in mice play a significant role in understanding sleep patterns and disorders and underscore the need for robust scoring methodologies. In response, this study introduces LG-Sleep, a novel subject-independent deep neural network architecture designed for mice sleep scoring through electroencephalogram (EEG) signals. LG-Sleep extracts local and global temporal transitions within EEG signals to categorize sleep data into three stages: wake, rapid eye movement (REM) sleep, and non-rapid eye movement (NREM) sleep. The model leverages local and global temporal information by employing time-distributed convolutional neural networks to discern local temporal transitions in EEG data. Subsequently, features derived from the convolutional filters traverse long short-term memory blocks, capturing global transitions over extended periods. Crucially, the model is optimized in an autoencoder-decoder fashion, facilitating generalization across distinct subjects and adapting to limited training samples. Experimental findings demonstrate superior performance of LG-Sleep compared to conventional deep neural networks. Moreover, the model exhibits good performance across different sleep stages even when tasked with scoring based on limited training samples.

Title: An Explorative Study of Pig Butchering Scams

• Authors: Bhupendra Acharya, Thorsten Holz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15423
• Pdf URL: https://arxiv.org/pdf/2412.15423
• Copy Paste: [[2412.15423]] An Explorative Study of Pig Butchering Scams(https://arxiv.org/abs/2412.15423)
• Keywords: attack
• Abstract: In the recent past, so-called pig-butchering scams are on the rise. This term is based on a translation of the Chinese term "Sha Zhu Pan", where scammers refer to victims as "pig" which are to be "fattened up before slaughter" so that scammer can siphon off as much monetary value as possible. In this type of scam, attackers perform social engineering tricks on victims over an extended period to build credibility or relationships. After a certain period, when victims transfer larger amounts of money to scammers, the fraudsters' platforms or profiles go permanently offline and the victims' money is lost. In this work, we provide the first comprehensive study of pig-butchering scams from multiple vantage points. Our study analyzes the direct victims' narratives shared on multiple social media platforms, public abuse report databases, and case studies from news outlets. Between March 2024 to October 2024, we collected data related to pig butchering scams from (i) four social media platforms comprised of more than 430,000 social media accounts and 770,000 posts; (ii) more than 3,200 public abuse reports narratives, and (iii) about 1,000 news articles. Through automated and qualitative evaluation, we provide an evaluation of victims of pig-butchering scams, finding 146 social media scammed users, 2,570 abuse reports narratives, and 50 case studies of 834 souls from news outlets. In total, we approximated losses of over \$521 million related to such scams. To complement this analysis, we performed a survey on crowdsourcing platforms with 584 users to broaden the insights on comparative analysis of pig-butchering scams with other types of scams. Our research highlights that these attacks are sophisticated and often require multiple entities, including policymakers and law enforcement, to work together alongside user education to create a proactive detection of such scams.

Title: AdaCred: Adaptive Causal Decision Transformers with Feature Crediting

• Authors: Hemant Kumawat, Saibal Mukhopadhyay
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.15427
• Pdf URL: https://arxiv.org/pdf/2412.15427
• Copy Paste: [[2412.15427]] AdaCred: Adaptive Causal Decision Transformers with Feature Crediting(https://arxiv.org/abs/2412.15427)
• Keywords: transformer
• Abstract: Reinforcement learning (RL) can be formulated as a sequence modeling problem, where models predict future actions based on historical state-action-reward sequences. Current approaches typically require long trajectory sequences to model the environment in offline RL settings. However, these models tend to over-rely on memorizing long-term representations, which impairs their ability to effectively attribute importance to trajectories and learned representations based on task-specific relevance. In this work, we introduce AdaCred, a novel approach that represents trajectories as causal graphs built from short-term action-reward-state sequences. Our model adaptively learns control policy by crediting and pruning low-importance representations, retaining only those most relevant for the downstream task. Our experiments demonstrate that AdaCred-based policies require shorter trajectory sequences and consistently outperform conventional methods in both offline reinforcement learning and imitation learning environments.

Title: Time Will Tell: Timing Side Channels via Output Token Count in Large Language Models

• Authors: Tianchen Zhang, Gururaj Saileshwar, David Lie
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15431
• Pdf URL: https://arxiv.org/pdf/2412.15431
• Copy Paste: [[2412.15431]] Time Will Tell: Timing Side Channels via Output Token Count in Large Language Models(https://arxiv.org/abs/2412.15431)
• Keywords: attack, large language model
• Abstract: This paper demonstrates a new side-channel that enables an adversary to extract sensitive information about inference inputs in large language models (LLMs) based on the number of output tokens in the LLM response. We construct attacks using this side-channel in two common LLM tasks: recovering the target language in machine translation tasks and recovering the output class in classification tasks. In addition, due to the auto-regressive generation mechanism in LLMs, an adversary can recover the output token count reliably using a timing channel, even over the network against a popular closed-source commercial LLM. Our experiments show that an adversary can learn the output language in translation tasks with more than 75% precision across three different models (Tower, M2M100, MBart50). Using this side-channel, we also show the input class in text classification tasks can be leaked out with more than 70% precision from open-source LLMs like Llama-3.1, Llama-3.2, Gemma2, and production models like GPT-4o. Finally, we propose tokenizer-, system-, and prompt-based mitigations against the output token count side-channel.

Title: SKETCH: Structured Knowledge Enhanced Text Comprehension for Holistic Retrieval

• Authors: Aakash Mahalingam, Vinesh Kumar Gande, Aman Chadha, Vinija Jain, Divya Chaudhary
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15443
• Pdf URL: https://arxiv.org/pdf/2412.15443
• Copy Paste: [[2412.15443]] SKETCH: Structured Knowledge Enhanced Text Comprehension for Holistic Retrieval(https://arxiv.org/abs/2412.15443)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) systems have become pivotal in leveraging vast corpora to generate informed and contextually relevant responses, notably reducing hallucinations in Large Language Models. Despite significant advancements, these systems struggle to efficiently process and retrieve information from large datasets while maintaining a comprehensive understanding of the context. This paper introduces SKETCH, a novel methodology that enhances the RAG retrieval process by integrating semantic text retrieval with knowledge graphs, thereby merging structured and unstructured data for a more holistic comprehension. SKETCH, demonstrates substantial improvements in retrieval performance and maintains superior context integrity compared to traditional methods. Evaluated across four diverse datasets: QuALITY, QASPER, NarrativeQA, and Italian Cuisine-SKETCH consistently outperforms baseline approaches on key RAGAS metrics such as answer_relevancy, faithfulness, context_precision and context_recall. Notably, on the Italian Cuisine dataset, SKETCH achieved an answer relevancy of 0.94 and a context precision of 0.99, representing the highest performance across all evaluated metrics. These results highlight SKETCH's capability in delivering more accurate and contextually relevant responses, setting new benchmarks for future retrieval systems.

Title: How to Manage My Data? With Machine--Interpretable GDPR Rights!

• Authors: Beatriz Esteves, Harshvardhan J. Pandit, Georg P. Krog, Paul Ryan
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.15451
• Pdf URL: https://arxiv.org/pdf/2412.15451
• Copy Paste: [[2412.15451]] How to Manage My Data? With Machine--Interpretable GDPR Rights!(https://arxiv.org/abs/2412.15451)
• Keywords: privacy
• Abstract: The EU GDPR is a landmark regulation that introduced several rights for individuals to obtain information and control how their personal data is being processed, as well as receive a copy of it. However, there are gaps in the effective use of rights due to each organisation developing custom methods for rights declaration and management. Simultaneously, there is a technological gap as there is no single consistent standards-based mechanism that can automate the handling of rights for both organisations and individuals. In this article, we present a specification for exercising and managing rights in a machine-interpretable format based on semantic web standards. Our approach uses the comprehensive Data Privacy Vocabulary to create a streamlined workflow for individuals to understand what rights exist, how and where to exercise them, and for organisations to effectively manage them. This work pushes the state of the art in GDPR rights management and is crucial for data reuse and rights management under technologically intensive developments, such as Data Spaces.

Title: Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization

• Authors: Sahil Wadhwa, Chengtian Xu, Haoming Chen, Aakash Mahalingam, Akankshya Kar, Divya Chaudhary
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15453
• Pdf URL: https://arxiv.org/pdf/2412.15453
• Copy Paste: [[2412.15453]] Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization(https://arxiv.org/abs/2412.15453)
• Keywords: large language model
• Abstract: The automatic generation of counter-speech (CS) is a critical strategy for addressing hate speech by providing constructive and informed responses. However, existing methods often fail to generate high-quality, impactful, and scalable CS, particularly across diverse linguistic contexts. In this paper, we propose a novel methodology to enhance CS generation by aligning Large Language Models (LLMs) using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Our approach leverages DPO to align LLM outputs with human preferences, ensuring contextually appropriate and linguistically adaptable responses. Additionally, we incorporate knowledge grounding to enhance the factual accuracy and relevance of generated CS. Experimental results demonstrate that DPO-aligned models significantly outperform SFT baselines on CS benchmarks while scaling effectively to multiple languages. These findings highlight the potential of preference-based alignment techniques to advance CS generation across varied linguistic settings. The model supervision and alignment is done in English and the same model is used for reporting metrics across other languages like Basque, Italian, and Spanish.

Title: Non-Uniform Parameter-Wise Model Merging

• Authors: Albert Manuel Orozco Camacho, Stefan Horoi, Guy Wolf, Eugene Belilovsky
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15467
• Pdf URL: https://arxiv.org/pdf/2412.15467
• Copy Paste: [[2412.15467]] Non-Uniform Parameter-Wise Model Merging(https://arxiv.org/abs/2412.15467)
• Keywords: robust
• Abstract: Combining multiple machine learning models has long been a technique for enhancing performance, particularly in distributed settings. Traditional approaches, such as model ensembles, work well, but are expensive in terms of memory and compute. Recently, methods based on averaging model parameters have achieved good results in some settings and have gained popularity. However, merging models initialized differently that do not share a part of their training trajectories can yield worse results than simply using the base models, even after aligning their neurons. In this paper, we introduce a novel approach, Non-uniform Parameter-wise Model Merging, or NP Merge, which merges models by learning the contribution of each parameter to the final model using gradient-based optimization. We empirically demonstrate the effectiveness of our method for merging models of various architectures in multiple settings, outperforming past methods. We also extend NP Merge to handle the merging of multiple models, showcasing its scalability and robustness.

Title: Continual Learning Using Only Large Language Model Prompting

• Authors: Jiabao Qiu, Zixuan Ke, Bing Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15479
• Pdf URL: https://arxiv.org/pdf/2412.15479
• Copy Paste: [[2412.15479]] Continual Learning Using Only Large Language Model Prompting(https://arxiv.org/abs/2412.15479)
• Keywords: large language model
• Abstract: We introduce CLOB, a novel continual learning (CL) paradigm wherein a large language model (LLM) is regarded as a black box. Learning is done incrementally via only verbal prompting. CLOB does not fine-tune any part of the LLM or add any trainable parameters to it. It is particularly suitable for LLMs that are accessible via APIs. We also propose a new CL technique, called CIS, based on incremental summarization that also overcomes the LLM's input length limit. Experiments show CIS outperforms baselines by a very large margin.

Title: Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage

• Authors: Saehyung Lee, Seunghyun Yoon, Trung Bui, Jing Shi, Sungroh Yoon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15484
• Pdf URL: https://arxiv.org/pdf/2412.15484
• Copy Paste: [[2412.15484]] Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage(https://arxiv.org/abs/2412.15484)
• Keywords: robust, large language model
• Abstract: Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations. Our analysis reveals that existing hallucination detection methods struggle with detailed captions. We attribute this to the increasing reliance of MLLMs on their generated text, rather than the input image, as the sequence length grows. To address this issue, we propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions. Additionally, we introduce an evaluation framework and a benchmark dataset to facilitate the systematic analysis of detailed captions. Our experiments demonstrate that our proposed evaluation method better aligns with human judgments of factuality than existing metrics and that existing approaches to improve the MLLM factuality may fall short in hyper-detailed image captioning tasks. In contrast, our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V. Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating that an MLLM's performance on VQA benchmarks may not correlate with its ability to generate detailed image captions.

Title: GCA-3D: Towards Generalized and Consistent Domain Adaptation of 3D Generators

• Authors: Hengjia Li, Yang Liu, Yibo Zhao, Haoran Cheng, Yang Yang, Linxuan Xia, Zekai Luo, Qibo Qiu, Boxi Wu, Tu Zheng, Zheng Yang, Deng Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15491
• Pdf URL: https://arxiv.org/pdf/2412.15491
• Copy Paste: [[2412.15491]] GCA-3D: Towards Generalized and Consistent Domain Adaptation of 3D Generators(https://arxiv.org/abs/2412.15491)
• Keywords: diffusion, generative
• Abstract: Recently, 3D generative domain adaptation has emerged to adapt the pre-trained generator to other domains without collecting massive datasets and camera pose distributions. Typically, they leverage large-scale pre-trained text-to-image diffusion models to synthesize images for the target domain and then fine-tune the 3D model. However, they suffer from the tedious pipeline of data generation, which inevitably introduces pose bias between the source domain and synthetic dataset. Furthermore, they are not generalized to support one-shot image-guided domain adaptation, which is more challenging due to the more severe pose bias and additional identity bias introduced by the single image reference. To address these issues, we propose GCA-3D, a generalized and consistent 3D domain adaptation method without the intricate pipeline of data generation. Different from previous pipeline methods, we introduce multi-modal depth-aware score distillation sampling loss to efficiently adapt 3D generative models in a non-adversarial manner. This multi-modal loss enables GCA-3D in both text prompt and one-shot image prompt adaptation. Besides, it leverages per-instance depth maps from the volume rendering module to mitigate the overfitting problem and retain the diversity of results. To enhance the pose and identity consistency, we further propose a hierarchical spatial consistency loss to align the spatial structure between the generated images in the source and target domain. Experiments demonstrate that GCA-3D outperforms previous methods in terms of efficiency, generalization, pose accuracy, and identity consistency.

Title: TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use

• Authors: Junjie Ye, Yilong Wu, Sixian Li, Yuming Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan, Zhengyin Du
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15495
• Pdf URL: https://arxiv.org/pdf/2412.15495
• Copy Paste: [[2412.15495]] TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use(https://arxiv.org/abs/2412.15495)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) achieve remarkable advancements by leveraging tools to interact with external environments, a critical step toward generalized AI. However, the standard supervised fine-tuning (SFT) approach, which relies on large-scale datasets, often overlooks task-specific characteristics in tool use, leading to performance bottlenecks. To address this issue, we analyze three existing LLMs and uncover key insights: training data can inadvertently impede tool-use behavior, token importance is distributed unevenly, and errors in tool calls fall into a small set of distinct categories. Building on these findings, we propose TL-Training, a task-feature-based framework that mitigates the effects of suboptimal training data, dynamically adjusts token weights to prioritize key tokens during SFT, and incorporates a robust reward mechanism tailored to error categories, optimized through proximal policy optimization. We validate TL-Training by training CodeLLaMA-2-7B and evaluating it on four diverse open-source test sets. Our results demonstrate that the LLM trained by our method matches or surpasses both open- and closed-source LLMs in tool-use performance using only 1,217 training data points. Additionally, our method enhances robustness in noisy environments and improves general task performance, offering a scalable and efficient paradigm for tool-use training in LLMs. The code and data are available at this https URL.

Title: The First Multilingual Model For The Detection of Suicide Texts

• Authors: Rodolfo Zevallos, Annika Schoene, John E. Ortega
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15498
• Pdf URL: https://arxiv.org/pdf/2412.15498
• Copy Paste: [[2412.15498]] The First Multilingual Model For The Detection of Suicide Texts(https://arxiv.org/abs/2412.15498)
• Keywords: transformer
• Abstract: Suicidal ideation is a serious health problem affecting millions of people worldwide. Social networks provide information about these mental health problems through users' emotional expressions. We propose a multilingual model leveraging transformer architectures like mBERT, XML-R, and mT5 to detect suicidal text across posts in six languages - Spanish, English, German, Catalan, Portuguese and Italian. A Spanish suicide ideation tweet dataset was translated into five other languages using SeamlessM4T. Each model was fine-tuned on this multilingual data and evaluated across classification metrics. Results showed mT5 achieving the best performance overall with F1 scores above 85%, highlighting capabilities for cross-lingual transfer learning. The English and Spanish translations also displayed high quality based on perplexity. Our exploration underscores the importance of considering linguistic diversity in developing automated multilingual tools to identify suicidal risk. Limitations exist around semantic fidelity in translations and ethical implications which provide guidance for future human-in-the-loop evaluations.

Title: A Robust Prototype-Based Network with Interpretable RBF Classifier Foundations

• Authors: Sascha Saralajew, Ashish Rana, Thomas Villmann, Ammar Shaker
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15499
• Pdf URL: https://arxiv.org/pdf/2412.15499
• Copy Paste: [[2412.15499]] A Robust Prototype-Based Network with Interpretable RBF Classifier Foundations(https://arxiv.org/abs/2412.15499)
• Keywords: robust, interpretability
• Abstract: Prototype-based classification learning methods are known to be inherently interpretable. However, this paradigm suffers from major limitations compared to deep models, such as lower performance. This led to the development of the so-called deep Prototype-Based Networks (PBNs), also known as prototypical parts models. In this work, we analyze these models with respect to different properties, including interpretability. In particular, we focus on the Classification-by-Components (CBC) approach, which uses a probabilistic model to ensure interpretability and can be used as a shallow or deep architecture. We show that this model has several shortcomings, like creating contradicting explanations. Based on these findings, we propose an extension of CBC that solves these issues. Moreover, we prove that this extension has robustness guarantees and derive a loss that optimizes robustness. Additionally, our analysis shows that most (deep) PBNs are related to (deep) RBF classifiers, which implies that our robustness guarantees generalize to shallow RBF classifiers. The empirical evaluation demonstrates that our deep PBN yields state-of-the-art classification accuracy on different benchmarks while resolving the interpretability shortcomings of other approaches. Further, our shallow PBN variant outperforms other shallow PBNs while being inherently interpretable and exhibiting provable robustness guarantees.

Title: Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models

• Authors: Zhisheng Tang, Mayank Kejriwal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15501
• Pdf URL: https://arxiv.org/pdf/2412.15501
• Copy Paste: [[2412.15501]] Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models(https://arxiv.org/abs/2412.15501)
• Keywords: large language model
• Abstract: Research on emergent patterns in Large Language Models (LLMs) has gained significant traction in both psychology and artificial intelligence, motivating the need for a comprehensive review that offers a synthesis of this complex landscape. In this article, we systematically review LLMs' capabilities across three important cognitive domains: decision-making biases, reasoning, and creativity. We use empirical studies drawing on established psychological tests and compare LLMs' performance to human benchmarks. On decision-making, our synthesis reveals that while LLMs demonstrate several human-like biases, some biases observed in humans are absent, indicating cognitive patterns that only partially align with human decision-making. On reasoning, advanced LLMs like GPT-4 exhibit deliberative reasoning akin to human System-2 thinking, while smaller models fall short of human-level performance. A distinct dichotomy emerges in creativity: while LLMs excel in language-based creative tasks, such as storytelling, they struggle with divergent thinking tasks that require real-world context. Nonetheless, studies suggest that LLMs hold considerable potential as collaborators, augmenting creativity in human-machine problem-solving settings. Discussing key limitations, we also offer guidance for future research in areas such as memory, attention, and open-source model development.

Title: Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggers

• Authors: Ruofei Wang, Hongzhan Lin, Ziyuan Luo, Ka Chun Cheung, Simon See, Jing Ma, Renjie Wan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15503
• Pdf URL: https://arxiv.org/pdf/2412.15503
• Copy Paste: [[2412.15503]] Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggers(https://arxiv.org/abs/2412.15503)
• Keywords: attack, steal
• Abstract: Hateful meme detection aims to prevent the proliferation of hateful memes on various social media platforms. Considering its impact on social environments, this paper introduces a previously ignored but significant threat to hateful meme detection: backdoor attacks. By injecting specific triggers into meme samples, backdoor attackers can manipulate the detector to output their desired outcomes. To explore this, we propose the Meme Trojan framework to initiate backdoor attacks on hateful meme detection. Meme Trojan involves creating a novel Cross-Modal Trigger (CMT) and a learnable trigger augmentor to enhance the trigger pattern according to each input sample. Due to the cross-modal property, the proposed CMT can effectively initiate backdoor attacks on hateful meme detectors under an automatic application scenario. Additionally, the injection position and size of our triggers are adaptive to the texts contained in the meme, which ensures that the trigger is seamlessly integrated with the meme content. Our approach outperforms the state-of-the-art backdoor attack methods, showing significant improvements in effectiveness and stealthiness. We believe that this paper will draw more attention to the potential threat posed by backdoor attacks on hateful meme detection.

Title: Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework

• Authors: Zhenjie Xu (1), Wenqing Chen (1), Yi Tang (1), Xuanying Li (2), Cheng Hu (1), Zhixuan Chu (3), Kui Ren (3), Zibin Zheng (1), Zhichao Lu (4) ((1) School of Software Engineering, Sun Yat-sen University, (2) School of Physics and Astronomy, Sun Yat-sen University, (3) School of Cyber Science and Technology, Zhejiang University, (4) Department of Computer Science, City University of Hong Kong)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15504
• Pdf URL: https://arxiv.org/pdf/2412.15504
• Copy Paste: [[2412.15504]] Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework(https://arxiv.org/abs/2412.15504)
• Keywords: large language model
• Abstract: Natural language processing (NLP) has seen remarkable advancements with the development of large language models (LLMs). Despite these advancements, LLMs often produce socially biased outputs. Recent studies have mainly addressed this problem by prompting LLMs to behave ethically, but this approach results in unacceptable performance degradation. In this paper, we propose a multi-objective approach within a multi-agent framework (MOMA) to mitigate social bias in LLMs without significantly compromising their performance. The key idea of MOMA involves deploying multiple agents to perform causal interventions on bias-related contents of the input questions, breaking the shortcut connection between these contents and the corresponding answers. Unlike traditional debiasing techniques leading to performance degradation, MOMA substantially reduces bias while maintaining accuracy in downstream tasks. Our experiments conducted on two datasets and two models demonstrate that MOMA reduces bias scores by up to 87.7%, with only a marginal performance degradation of up to 6.8% in the BBQ dataset. Additionally, it significantly enhances the multi-objective metric icat in the StereoSet dataset by up to 58.1%. Code will be made available at this https URL.

Title: Stylish and Functional: Guided Interpolation Subject to Physical Constraints

• Authors: Yan-Ying Chen, Nikos Arechiga, Chenyang Yuan, Matthew Hong, Matt Klenk, Charlene Wu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15507
• Pdf URL: https://arxiv.org/pdf/2412.15507
• Copy Paste: [[2412.15507]] Stylish and Functional: Guided Interpolation Subject to Physical Constraints(https://arxiv.org/abs/2412.15507)
• Keywords: diffusion, generative
• Abstract: Generative AI is revolutionizing engineering design practices by enabling rapid prototyping and manipulation of designs. One example of design manipulation involves taking two reference design images and using them as prompts to generate a design image that combines aspects of both. Real engineering designs have physical constraints and functional requirements in addition to aesthetic design considerations. Internet-scale foundation models commonly used for image generation, however, are unable to take these physical constraints and functional requirements into consideration as part of the generation process. We consider the problem of generating a design inspired by two input designs, and propose a zero-shot framework toward enforcing physical, functional requirements over the generation process by leveraging a pretrained diffusion model as the backbone. As a case study, we consider the example of rotational symmetry in generation of wheel designs. Automotive wheels are required to be rotationally symmetric for physical stability. We formulate the requirement of rotational symmetry by the use of a symmetrizer, and we use this symmetrizer to guide the diffusion process towards symmetric wheel generations. Our experimental results find that the proposed approach makes generated interpolations with higher realism than methods in related work, as evaluated by Fréchet inception distance (FID). We also find that our approach generates designs that more closely satisfy physical and functional requirements than generating without the symmetry guidance.

Title: ADEQA: A Question Answer based approach for joint ADE-Suspect Extraction using Sequence-To-Sequence Transformers

• Authors: Vinayak Arannil, Tomal Deb, Atanu Roy
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15510
• Pdf URL: https://arxiv.org/pdf/2412.15510
• Copy Paste: [[2412.15510]] ADEQA: A Question Answer based approach for joint ADE-Suspect Extraction using Sequence-To-Sequence Transformers(https://arxiv.org/abs/2412.15510)
• Keywords: extraction, transformer
• Abstract: Early identification of Adverse Drug Events (ADE) is critical for taking prompt actions while introducing new drugs into the market. These ADEs information are available through various unstructured data sources like clinical study reports, patient health records, social media posts, etc. Extracting ADEs and the related suspect drugs using machine learning is a challenging task due to the complex linguistic relations between drug ADE pairs in textual data and unavailability of large corpus of labelled datasets. This paper introduces ADEQA, a question-answer(QA) based approach using quasi supervised labelled data and sequence-to-sequence transformers to extract ADEs, drug suspects and the relationships between them. Unlike traditional QA models, natural language generation (NLG) based models don't require extensive token level labelling and thereby reduces the adoption barrier significantly. On a public ADE corpus, we were able to achieve state-of-the-art results with an F1 score of 94% on establishing the relationships between ADEs and the respective suspects.

Title: PreNeT: Leveraging Computational Features to Predict Deep Neural Network Training Time

• Authors: Alireza Pourali, Arian Boukani, Hamzeh Khazaei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15519
• Pdf URL: https://arxiv.org/pdf/2412.15519
• Copy Paste: [[2412.15519]] PreNeT: Leveraging Computational Features to Predict Deep Neural Network Training Time(https://arxiv.org/abs/2412.15519)
• Keywords: transformer, large language model
• Abstract: Training deep learning models, particularly Transformer-based architectures such as Large Language Models (LLMs), demands substantial computational resources and extended training periods. While optimal configuration and infrastructure selection can significantly reduce associated costs, this optimization requires preliminary analysis tools. This paper introduces PreNeT, a novel predictive framework designed to address this optimization challenge. PreNeT facilitates training optimization by integrating comprehensive computational metrics, including layer-specific parameters, arithmetic operations and memory utilization. A key feature of PreNeT is its capacity to accurately predict training duration on previously unexamined hardware infrastructures, including novel accelerator architectures. This framework employs a sophisticated approach to capture and analyze the distinct characteristics of various neural network layers, thereby enhancing existing prediction methodologies. Through proactive implementation of PreNeT, researchers and practitioners can determine optimal configurations, parameter settings, and hardware specifications to maximize cost-efficiency and minimize training duration. Experimental results demonstrate that PreNeT achieves up to 72% improvement in prediction accuracy compared to contemporary state-of-the-art frameworks.

Title: HREF: Human Response-Guided Evaluation of Instruction Following in Language Models

• Authors: Xinxi Lyu, Yizhong Wang, Hannaneh Hajishirzi, Pradeep Dasigi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15524
• Pdf URL: https://arxiv.org/pdf/2412.15524
• Copy Paste: [[2412.15524]] HREF: Human Response-Guided Evaluation of Instruction Following in Language Models(https://arxiv.org/abs/2412.15524)
• Keywords: large language model
• Abstract: Evaluating the capability of Large Language Models (LLMs) in following instructions has heavily relied on a powerful LLM as the judge, introducing unresolved biases that deviate the judgments from human judges. In this work, we reevaluate various choices for automatic evaluation on a wide range of instruction-following tasks. We experiment with methods that leverage human-written responses and observe that they enhance the reliability of automatic evaluations across a wide range of tasks, resulting in up to a 3.2% improvement in agreement with human judges. We also discovered that human-written responses offer an orthogonal perspective to model-generated responses in following instructions and should be used as an additional context when comparing model responses. Based on these observations, we develop a new evaluation benchmark, Human Response-Guided Evaluation of Instruction Following (HREF), comprising 4,258 samples across 11 task categories with a composite evaluation setup, employing a composite evaluation setup that selects the most reliable method for each category. In addition to providing reliable evaluation, HREF emphasizes individual task performance and is free from contamination. Finally, we study the impact of key design choices in HREF, including the size of the evaluation set, the judge model, the baseline model, and the prompt template. We host a live leaderboard that evaluates LLMs on the private evaluation set of HREF.

Title: SGTC: Semantic-Guided Triplet Co-training for Sparsely Annotated Semi-Supervised Medical Image Segmentation

• Authors: Ke Yan, Qing Cai, Fan Zhang, Ziyan Cao, Zhi Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15526
• Pdf URL: https://arxiv.org/pdf/2412.15526
• Copy Paste: [[2412.15526]] SGTC: Semantic-Guided Triplet Co-training for Sparsely Annotated Semi-Supervised Medical Image Segmentation(https://arxiv.org/abs/2412.15526)
• Keywords: robust, segmentation
• Abstract: Although semi-supervised learning has made significant advances in the field of medical image segmentation, fully annotating a volumetric sample slice by slice remains a costly and time-consuming task. Even worse, most of the existing approaches pay much attention to image-level information and ignore semantic features, resulting in the inability to perceive weak boundaries. To address these issues, we propose a novel Semantic-Guided Triplet Co-training (SGTC) framework, which achieves high-end medical image segmentation by only annotating three orthogonal slices of a few volumetric samples, significantly alleviating the burden of radiologists. Our method consist of two main components. Specifically, to enable semantic-aware, fine-granular segmentation and enhance the quality of pseudo-labels, a novel semantic-guided auxiliary learning mechanism is proposed based on the pretrained CLIP. In addition, focusing on a more challenging but clinically realistic scenario, a new triple-view disparity training strategy is proposed, which uses sparse annotations (i.e., only three labeled slices of a few volumes) to perform co-training between three sub-networks, significantly improving the robustness. Extensive experiments on three public medical datasets demonstrate that our method outperforms most state-of-the-art semi-supervised counterparts under sparse annotation settings. The source code is available at this https URL.

Title: XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation

• Authors: Qianren Mao, Yangyifei Luo, Jinlong Zhang, Hanwen Hao, Zhilong Cao, Xiaolong Wang, Xiao Guan, Zhenting Huang, Weifeng Jiang, Shuyu Guo, Zhentao Han, Qili Zhang, Siyuan Tao, Yujie Liu, Junnan Liu, Zhixing Tan, Jie Sun, Bo Li, Xudong Liu, Richong Zhang, Jianxin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15529
• Pdf URL: https://arxiv.org/pdf/2412.15529
• Copy Paste: [[2412.15529]] XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation(https://arxiv.org/abs/2412.15529)
• Keywords: generative, large language model
• Abstract: Retrieval-augmented generation (RAG) synergizes the retrieval of pertinent data with the generative capabilities of Large Language Models (LLMs), ensuring that the generated output is not only contextually relevant but also accurate and this http URL introduce XRAG, an open-source, modular codebase that facilitates exhaustive evaluation of the performance of foundational components of advanced RAG modules. These components are systematically categorized into four core phases: pre-retrieval, retrieval, post-retrieval, and generation. We systematically analyse them across reconfigured datasets, providing a comprehensive benchmark for their effectiveness. Given the escalating complexity of RAG systems, we underscore the necessity of identifying potential failure points of RAG modules. We formulate a suite of experimental methodologies and diagnostic testing protocols to dissect the failure points inherent in the engineering of RAG modules. Subsequently, we proffer bespoke solutions that are designed to augment the validation processes and bolster the overall performance of these modules. Our work thoroughly evaluates the performance of core advanced components in RAG systems, providing insights into optimizations for prevalent failure points.

Title: FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF

• Authors: Flint Xiaofeng Fan, Cheston Tan, Yew-Soon Ong, Roger Wattenhofer, Wei-Tsang Ooi
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15538
• Pdf URL: https://arxiv.org/pdf/2412.15538
• Copy Paste: [[2412.15538]] FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF(https://arxiv.org/abs/2412.15538)
• Keywords: privacy, robust, federate
• Abstract: In the era of increasing privacy concerns and demand for personalized experiences, traditional Reinforcement Learning with Human Feedback (RLHF) frameworks face significant challenges due to their reliance on centralized data. We introduce Federated Reinforcement Learning with Human Feedback (FedRLHF), a novel framework that decentralizes the RLHF process. FedRLHF enables collaborative policy learning across multiple clients without necessitating the sharing of raw data or human feedback, thereby ensuring robust privacy preservation. Leveraging federated reinforcement learning, each client integrates human feedback locally into their reward functions and updates their policies through personalized RLHF processes. We establish rigorous theoretical foundations for FedRLHF, providing convergence guarantees, and deriving sample complexity bounds that scale efficiently with the number of clients. Empirical evaluations on the MovieLens and IMDb datasets demonstrate that FedRLHF not only preserves user privacy but also achieves performance on par with centralized RLHF, while enhancing personalization across diverse client environments.

Title: MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering

• Authors: Zhang Siyue, Xue Yuxiang, Zhang Yiming, Wu Xiaobao, Luu Anh Tuan, Zhao Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15540
• Pdf URL: https://arxiv.org/pdf/2412.15540
• Copy Paste: [[2412.15540]] MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering(https://arxiv.org/abs/2412.15540)
• Keywords: large language model
• Abstract: Understanding temporal relations and answering time-sensitive questions is crucial yet a challenging task for question-answering systems powered by large language models (LLMs). Existing approaches either update the parametric knowledge of LLMs with new facts, which is resource-intensive and often impractical, or integrate LLMs with external knowledge retrieval (i.e., retrieval-augmented generation). However, off-the-shelf retrievers often struggle to identify relevant documents that require intensive temporal reasoning. To systematically study time-sensitive question answering, we introduce the TempRAGEval benchmark, which repurposes existing datasets by incorporating temporal perturbations and gold evidence labels. As anticipated, all existing retrieval methods struggle with these temporal reasoning-intensive questions. We further propose Modular Retrieval (MRAG), a trainless framework that includes three modules: (1) Question Processing that decomposes question into a main content and a temporal constraint; (2) Retrieval and Summarization that retrieves evidence and uses LLMs to summarize according to the main content; (3) Semantic-Temporal Hybrid Ranking that scores each evidence summarization based on both semantic and temporal relevance. On TempRAGEval, MRAG significantly outperforms baseline retrievers in retrieval performance, leading to further improvements in final answer accuracy.

Title: ChangeDiff: A Multi-Temporal Change Detection Data Generator with Flexible Text Prompts via Diffusion Model

• Authors: Qi Zang, Jiayi Yang, Shuang Wang, Dong Zhao, Wenjun Yi, Zhun Zhong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15541
• Pdf URL: https://arxiv.org/pdf/2412.15541
• Copy Paste: [[2412.15541]] ChangeDiff: A Multi-Temporal Change Detection Data Generator with Flexible Text Prompts via Diffusion Model(https://arxiv.org/abs/2412.15541)
• Keywords: diffusion, generative
• Abstract: Data-driven deep learning models have enabled tremendous progress in change detection (CD) with the support of pixel-level annotations. However, collecting diverse data and manually annotating them is costly, laborious, and knowledge-intensive. Existing generative methods for CD data synthesis show competitive potential in addressing this issue but still face the following limitations: 1) difficulty in flexibly controlling change events, 2) dependence on additional data to train the data generators, 3) focus on specific change detection tasks. To this end, this paper focuses on the semantic CD (SCD) task and develops a multi-temporal SCD data generator ChangeDiff by exploring powerful diffusion models. ChangeDiff innovatively generates change data in two steps: first, it uses text prompts and a text-to-layout (T2L) model to create continuous layouts, and then it employs layout-to-image (L2I) to convert these layouts into images. Specifically, we propose multi-class distribution-guided text prompts (MCDG-TP), allowing for layouts to be generated flexibly through controllable classes and their corresponding ratios. Subsequently, to generalize the T2L model to the proposed MCDG-TP, a class distribution refinement loss is further designed as training supervision. %For the former, a multi-classdistribution-guided text prompt (MCDG-TP) is proposed to complement via controllable classes and ratios. To generalize the text-to-image diffusion model to the proposed MCDG-TP, a class distribution refinement loss is designed as training supervision. For the latter, MCDG-TP in three modes is proposed to synthesize new layout masks from various texts. Our generated data shows significant progress in temporal continuity, spatial diversity, and quality realism, empowering change detectors with accuracy and transferability. The code is available at this https URL

Title: NGQA: A Nutritional Graph Question Answering Benchmark for Personalized Health-aware Nutritional Reasoning

• Authors: Zheyuan Zhang, Yiyang Li, Nhi Ha Lan Le, Zehong Wang, Tianyi Ma, Vincent Galassi, Keerthiram Murugesan, Nuno Moniz, Werner Geyer, Nitesh V Chawla, Chuxu Zhang, Yanfang Ye
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15547
• Pdf URL: https://arxiv.org/pdf/2412.15547
• Copy Paste: [[2412.15547]] NGQA: A Nutritional Graph Question Answering Benchmark for Personalized Health-aware Nutritional Reasoning(https://arxiv.org/abs/2412.15547)
• Keywords: large language model
• Abstract: Diet plays a critical role in human health, yet tailoring dietary reasoning to individual health conditions remains a major challenge. Nutrition Question Answering (QA) has emerged as a popular method for addressing this problem. However, current research faces two critical limitations. On one hand, the absence of datasets involving user-specific medical information severely limits \textit{personalization}. This challenge is further compounded by the wide variability in individual health needs. On the other hand, while large language models (LLMs), a popular solution for this task, demonstrate strong reasoning abilities, they struggle with the domain-specific complexities of personalized healthy dietary reasoning, and existing benchmarks fail to capture these challenges. To address these gaps, we introduce the Nutritional Graph Question Answering (NGQA) benchmark, the first graph question answering dataset designed for personalized nutritional health reasoning. NGQA leverages data from the National Health and Nutrition Examination Survey (NHANES) and the Food and Nutrient Database for Dietary Studies (FNDDS) to evaluate whether a food is healthy for a specific user, supported by explanations of the key contributing nutrients. The benchmark incorporates three question complexity settings and evaluates reasoning across three downstream tasks. Extensive experiments with LLM backbones and baseline models demonstrate that the NGQA benchmark effectively challenges existing models. In sum, NGQA addresses a critical real-world problem while advancing GraphQA research with a novel domain-specific benchmark.

Title: AutoRank: MCDA Based Rank Personalization for LoRA-Enabled Distributed Learning

• Authors: Shuaijun Chen, Omid Tavallaie, Niousha Nazemi, Xin Chen, Albert Y. Zomaya
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.15553
• Pdf URL: https://arxiv.org/pdf/2412.15553
• Copy Paste: [[2412.15553]] AutoRank: MCDA Based Rank Personalization for LoRA-Enabled Distributed Learning(https://arxiv.org/abs/2412.15553)
• Keywords: robust, federate
• Abstract: As data volumes expand rapidly, distributed machine learning has become essential for addressing the growing computational demands of modern AI systems. However, training models in distributed environments is challenging with participants hold skew, Non-Independent-Identically distributed (Non-IID) data. Low-Rank Adaptation (LoRA) offers a promising solution to this problem by personalizing low-rank updates rather than optimizing the entire model, LoRA-enabled distributed learning minimizes computational and maximize personalization for each participant. Enabling more robust and efficient training in distributed learning settings, especially in large-scale, heterogeneous systems. Despite the strengths of current state-of-the-art methods, they often require manual configuration of the initial rank, which is increasingly impractical as the number of participants grows. This manual tuning is not only time-consuming but also prone to suboptimal configurations. To address this limitation, we propose AutoRank, an adaptive rank-setting algorithm inspired by the bias-variance trade-off. AutoRank leverages the MCDA method TOPSIS to dynamically assign local ranks based on the complexity of each participant's data. By evaluating data distribution and complexity through our proposed data complexity metrics, AutoRank provides fine-grained adjustments to the rank of each participant's local LoRA model. This adaptive approach effectively mitigates the challenges of double-imbalanced, non-IID data. Experimental results demonstrate that AutoRank significantly reduces computational overhead, enhances model performance, and accelerates convergence in highly heterogeneous federated learning environments. Through its strong adaptability, AutoRank offers a scalable and flexible solution for distributed machine learning.

Title: In-context Continual Learning Assisted by an External Continual Learner

• Authors: Saleh Momeni, Sahisnu Mazumder, Zixuan Ke, Bing Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15563
• Pdf URL: https://arxiv.org/pdf/2412.15563
• Copy Paste: [[2412.15563]] In-context Continual Learning Assisted by an External Continual Learner(https://arxiv.org/abs/2412.15563)
• Keywords: large language model
• Abstract: Existing continual learning (CL) methods mainly rely on fine-tuning or adapting large language models (LLMs). They still suffer from catastrophic forgetting (CF). Little work has been done to exploit in-context learning (ICL) to leverage the extensive knowledge within LLMs for CL without updating any parameters. However, incrementally learning each new task in ICL necessitates adding training examples from each class of the task to the prompt, which hampers scalability as the prompt length increases. This issue not only leads to excessively long prompts that exceed the input token limit of the underlying LLM but also degrades the model's performance due to the overextended context. To address this, we introduce InCA, a novel approach that integrates an external continual learner (ECL) with ICL to enable scalable CL without CF. The ECL is built incrementally to pre-select a small subset of likely classes for each test instance. By restricting the ICL prompt to only these selected classes, InCA prevents prompt lengths from becoming excessively long, while maintaining high performance. Experimental results demonstrate that InCA significantly outperforms existing CL baselines, achieving substantial performance gains.

Title: DefFiller: Mask-Conditioned Diffusion for Salient Steel Surface Defect Generation

• Authors: Yichun Tai, Zhenzhen Huang, Tao Peng, Zhijiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15570
• Pdf URL: https://arxiv.org/pdf/2412.15570
• Copy Paste: [[2412.15570]] DefFiller: Mask-Conditioned Diffusion for Salient Steel Surface Defect Generation(https://arxiv.org/abs/2412.15570)
• Keywords: diffusion, generative
• Abstract: Current saliency-based defect detection methods show promise in industrial settings, but the unpredictability of defects in steel production environments complicates dataset creation, hampering model performance. Existing data augmentation approaches using generative models often require pixel-level annotations, which are time-consuming and resource-intensive. To address this, we introduce DefFiller, a mask-conditioned defect generation method that leverages a layout-to-image diffusion model. DefFiller generates defect samples paired with mask conditions, eliminating the need for pixel-level annotations and enabling direct use in model training. We also develop an evaluation framework to assess the quality of generated samples and their impact on detection performance. Experimental results on the SD-Saliency-900 dataset demonstrate that DefFiller produces high-quality defect images that accurately match the provided mask conditions, significantly enhancing the performance of saliency-based defect detection models trained on the augmented dataset.

Title: J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM

• Authors: Takero Yoshida, Yuikazu Ito, Yoshihiro Fujiwara, Shinji Tsuchida, Daisuke Sugiyama, Daisuke Matsuoka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15574
• Pdf URL: https://arxiv.org/pdf/2412.15574
• Copy Paste: [[2412.15574]] J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM(https://arxiv.org/abs/2412.15574)
• Keywords: large language model
• Abstract: Japan Agency for Marine-Earth Science and Technology (JAMSTEC) has made available the JAMSTEC Earth Deep-sea Image (J-EDI), a deep-sea video and image archive (this https URL). This archive serves as a valuable resource for researchers and scholars interested in deep-sea imagery. The dataset comprises images and videos of deep-sea phenomena, predominantly of marine organisms, but also of the seafloor and physical processes. In this study, we propose J-EDI QA, a benchmark for understanding images of deep-sea organisms using a multimodal large language model (LLM). The benchmark is comprised of 100 images, accompanied by questions and answers with four options by JAMSTEC researchers for each image. The QA pairs are provided in Japanese, and the benchmark assesses the ability to understand deep-sea species in Japanese. In the evaluation presented in this paper, OpenAI o1 achieved a 50% correct response rate. This result indicates that even with the capabilities of state-of-the-art models as of December 2024, deep-sea species comprehension is not yet at an expert level. Further advances in deep-sea species-specific LLMs are therefore required.

Title: SaliencyI2PLoc: saliency-guided image-point cloud localization using contrastive learning

• Authors: Yuhao Li, Jianping Li, Zhen Dong, Yuan Wang, Bisheng Yang
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.15577
• Pdf URL: https://arxiv.org/pdf/2412.15577
• Copy Paste: [[2412.15577]] SaliencyI2PLoc: saliency-guided image-point cloud localization using contrastive learning(https://arxiv.org/abs/2412.15577)
• Keywords: robust
• Abstract: Image to point cloud global localization is crucial for robot navigation in GNSS-denied environments and has become increasingly important for multi-robot map fusion and urban asset management. The modality gap between images and point clouds poses significant challenges for cross-modality fusion. Current cross-modality global localization solutions either require modality unification, which leads to information loss, or rely on engineered training schemes to encode multi-modality features, which often lack feature alignment and relation consistency. To address these limitations, we propose, SaliencyI2PLoc, a novel contrastive learning based architecture that fuses the saliency map into feature aggregation and maintains the feature relation consistency on multi-manifold spaces. To alleviate the pre-process of data mining, the contrastive learning framework is applied which efficiently achieves cross-modality feature mapping. The context saliency-guided local feature aggregation module is designed, which fully leverages the contribution of the stationary information in the scene generating a more representative global feature. Furthermore, to enhance the cross-modality feature alignment during contrastive learning, the consistency of relative relationships between samples in different manifold spaces is also taken into account. Experiments conducted on urban and highway scenario datasets demonstrate the effectiveness and robustness of our method. Specifically, our method achieves a Recall@1 of 78.92% and a Recall@20 of 97.59% on the urban scenario evaluation dataset, showing an improvement of 37.35% and 18.07%, compared to the baseline method. This demonstrates that our architecture efficiently fuses images and point clouds and represents a significant step forward in cross-modality global localization. The project page and code will be released.

Title: A Deep Probabilistic Framework for Continuous Time Dynamic Graph Generation

• Authors: Ryien Hosseini, Filippo Simini, Venkatram Vishwanath, Henry Hoffmann
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15582
• Pdf URL: https://arxiv.org/pdf/2412.15582
• Copy Paste: [[2412.15582]] A Deep Probabilistic Framework for Continuous Time Dynamic Graph Generation(https://arxiv.org/abs/2412.15582)
• Keywords: generative
• Abstract: Recent advancements in graph representation learning have shifted attention towards dynamic graphs, which exhibit evolving topologies and features over time. The increased use of such graphs creates a paramount need for generative models suitable for applications such as data augmentation, obfuscation, and anomaly detection. However, there are few generative techniques that handle continuously changing temporal graph data; existing work largely relies on augmenting static graphs with additional temporal information to model dynamic interactions between nodes. In this work, we propose a fundamentally different approach: We instead directly model interactions as a joint probability of an edge forming between two nodes at a given time. This allows us to autoregressively generate new synthetic dynamic graphs in a largely assumption free, scalable, and inductive manner. We formalize this approach as DG-Gen, a generative framework for continuous time dynamic graphs, and demonstrate its effectiveness over five datasets. Our experiments demonstrate that DG-Gen not only generates higher fidelity graphs compared to traditional methods but also significantly advances link prediction tasks.

Title: NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization

• Authors: Danial Kamali, Elham J. Barezi, Parisa Kordjamshidi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15588
• Pdf URL: https://arxiv.org/pdf/2412.15588
• Copy Paste: [[2412.15588]] NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization(https://arxiv.org/abs/2412.15588)
• Keywords: robust, large language model
• Abstract: Compositional generalization is crucial for artificial intelligence agents to solve complex vision-language reasoning tasks. Neuro-symbolic approaches have demonstrated promise in capturing compositional structures, but they face critical challenges: (a) reliance on predefined predicates for symbolic representations that limit adaptability, (b) difficulty in extracting predicates from raw data, and (c) using non-differentiable operations for combining primitive concepts. To address these issues, we propose NeSyCoCo, a neuro-symbolic framework that leverages large language models (LLMs) to generate symbolic representations and map them to differentiable neural computations. NeSyCoCo introduces three innovations: (a) augmenting natural language inputs with dependency structures to enhance the alignment with symbolic representations, (b) employing distributed word representations to link diverse, linguistically motivated logical predicates to neural modules, and (c) using the soft composition of normalized predicate scores to align symbolic and differentiable reasoning. Our framework achieves state-of-the-art results on the ReaSCAN and CLEVR-CoGenT compositional generalization benchmarks and demonstrates robust performance with novel concepts in the CLEVR-SYN benchmark.

Title: SemDP: Semantic-level Differential Privacy Protection for Face Datasets

• Authors: Xiaoting Zhang, Tao Wang, Junhao Ji
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15590
• Pdf URL: https://arxiv.org/pdf/2412.15590
• Copy Paste: [[2412.15590]] SemDP: Semantic-level Differential Privacy Protection for Face Datasets(https://arxiv.org/abs/2412.15590)
• Keywords: privacy, protect
• Abstract: While large-scale face datasets have advanced deep learning-based face analysis, they also raise privacy concerns due to the sensitive personal information they contain. Recent schemes have implemented differential privacy to protect face datasets. However, these schemes generally treat each image as a separate database, which does not fully meet the core requirements of differential privacy. In this paper, we propose a semantic-level differential privacy protection scheme that applies to the entire face dataset. Unlike pixel-level differential privacy approaches, our scheme guarantees that semantic privacy in faces is not compromised. The key idea is to convert unstructured data into structured data to enable the application of differential privacy. Specifically, we first extract semantic information from the face dataset to build an attribute database, then apply differential perturbations to obscure this attribute data, and finally use an image synthesis model to generate a protected face dataset. Extensive experimental results show that our scheme can maintain visual naturalness and balance the privacy-utility trade-off compared to the mainstream schemes.

Title: Machine Learning Techniques for Pattern Recognition in High-Dimensional Data Mining

• Authors: Pochun Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15593
• Pdf URL: https://arxiv.org/pdf/2412.15593
• Copy Paste: [[2412.15593]] Machine Learning Techniques for Pattern Recognition in High-Dimensional Data Mining(https://arxiv.org/abs/2412.15593)
• Keywords: robust, extraction
• Abstract: This paper proposes a frequent pattern data mining algorithm based on support vector machine (SVM), aiming to solve the performance bottleneck of traditional frequent pattern mining algorithms in high-dimensional and sparse data environments. By converting the frequent pattern mining task into a classification problem, the SVM model is introduced to improve the accuracy and robustness of pattern extraction. In terms of method design, the kernel function is used to map the data to a high-dimensional feature space, so as to construct the optimal classification hyperplane, realize the nonlinear separation of patterns and the accurate mining of frequent items. In the experiment, two public datasets, Retail and Mushroom, were selected to compare and analyze the proposed algorithm with traditional FP-Growth, FP-Tree, decision tree and random forest models. The experimental results show that the algorithm in this paper is significantly better than the traditional model in terms of three key indicators: support, confidence and lift, showing strong pattern recognition ability and rule extraction effect. The study shows that the SVM model has excellent performance advantages in an environment with high data sparsity and a large number of transactions, and can effectively cope with complex pattern mining tasks. At the same time, this paper also points out the potential direction of future research, including the introduction of deep learning and ensemble learning frameworks to further improve the scalability and adaptability of the algorithm. This research not only provides a new idea for frequent pattern mining, but also provides important technical support for solving pattern discovery and association rule mining problems in practical applications.

Title: Template-Driven LLM-Paraphrased Framework for Tabular Math Word Problem Generation

• Authors: Xiaoqiang Kang, Zimu Wang, Xiaobo Jin, Wei Wang, Kaizhu Huang, Qiufeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15594
• Pdf URL: https://arxiv.org/pdf/2412.15594
• Copy Paste: [[2412.15594]] Template-Driven LLM-Paraphrased Framework for Tabular Math Word Problem Generation(https://arxiv.org/abs/2412.15594)
• Keywords: large language model
• Abstract: Solving tabular math word problems (TMWPs) has become a critical role in evaluating the mathematical reasoning ability of large language models (LLMs), where large-scale TMWP samples are commonly required for LLM fine-tuning. Since the collection of high-quality TMWP datasets is costly and time-consuming, recent research has concentrated on automatic TMWP generation. However, current generated samples usually suffer from issues of either correctness or diversity. In this paper, we propose a Template-driven LLM-paraphrased (TeLL) framework for generating high-quality TMWP samples with diverse backgrounds and accurate tables, questions, answers, and solutions. To this end, we first extract templates from existing real samples to generate initial problems, ensuring correctness. Then, we adopt an LLM to extend templates and paraphrase problems, obtaining diverse TMWP samples. Furthermore, we find the reasoning annotation is important for solving TMWPs. Therefore, we propose to enrich each solution with illustrative reasoning steps. Through the proposed framework, we construct a high-quality dataset TabMWP-TeLL by adhering to the question types in the TabMWP dataset, and we conduct extensive experiments on a variety of LLMs to demonstrate the effectiveness of TabMWP-TeLL in improving TMWP solving performance. The code and data of this paper are available at: this https URL.

Title: Mask-RadarNet: Enhancing Transformer With Spatial-Temporal Semantic Context for Radar Object Detection in Autonomous Driving

• Authors: Yuzhi Wu, Jun Liu, Guangfeng Jiang, Weijian Liu, Danilo Orlando
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15595
• Pdf URL: https://arxiv.org/pdf/2412.15595
• Copy Paste: [[2412.15595]] Mask-RadarNet: Enhancing Transformer With Spatial-Temporal Semantic Context for Radar Object Detection in Autonomous Driving(https://arxiv.org/abs/2412.15595)
• Keywords: robust, transformer
• Abstract: As a cost-effective and robust technology, automotive radar has seen steady improvement during the last years, making it an appealing complement to commonly used sensors like camera and LiDAR in autonomous driving. Radio frequency data with rich semantic information are attracting more and more attention. Most current radar-based models take radio frequency image sequences as the input. However, these models heavily rely on convolutional neural networks and leave out the spatial-temporal semantic context during the encoding stage. To solve these problems, we propose a model called Mask-RadarNet to fully utilize the hierarchical semantic features from the input radar data. Mask-RadarNet exploits the combination of interleaved convolution and attention operations to replace the traditional architecture in transformer-based models. In addition, patch shift is introduced to the Mask-RadarNet for efficient spatial-temporal feature learning. By shifting part of patches with a specific mosaic pattern in the temporal dimension, Mask-RadarNet achieves competitive performance while reducing the computational burden of the spatial-temporal modeling. In order to capture the spatial-temporal semantic contextual information, we design the class masking attention module (CMAM) in our encoder. Moreover, a lightweight auxiliary decoder is added to our model to aggregate prior maps generated from the CMAM. Experiments on the CRUW dataset demonstrate the superiority of the proposed method to some state-of-the-art radar-based object detection algorithms. With relatively lower computational complexity and fewer parameters, the proposed Mask-RadarNet achieves higher recognition accuracy for object detection in autonomous driving.

Title: Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classification

• Authors: Gyutae Park, Ingeol Baek, ByeongJeong Kim, Joongbo Shin, Hwanhee Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15603
• Pdf URL: https://arxiv.org/pdf/2412.15603
• Copy Paste: [[2412.15603]] Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classification(https://arxiv.org/abs/2412.15603)
• Keywords: large language model
• Abstract: Dialogue intent classification aims to identify the underlying purpose or intent of a user's input in a conversation. Current intent classification systems encounter considerable challenges, primarily due to the vast number of possible intents and the significant semantic overlap among similar intent classes. In this paper, we propose a novel approach to few-shot dialogue intent classification through in-context learning, incorporating dynamic label refinement to address these challenges. Our method retrieves relevant examples for a test input from the training set and leverages a large language model to dynamically refine intent labels based on semantic understanding, ensuring that intents are clearly distinguishable from one another. Experimental results demonstrate that our approach effectively resolves confusion between semantically similar intents, resulting in significantly enhanced performance across multiple datasets compared to baselines. We also show that our method generates more interpretable intent labels, and has a better semantic coherence in capturing underlying user intents compared to baselines.

Title: Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks

• Authors: Brian J Chan, Chao-Ting Chen, Jui-Hung Cheng, Hen-Hsen Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15605
• Pdf URL: https://arxiv.org/pdf/2412.15605
• Copy Paste: [[2412.15605]] Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks(https://arxiv.org/abs/2412.15605)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) has gained traction as a powerful approach for enhancing language models by integrating external knowledge sources. However, RAG introduces challenges such as retrieval latency, potential errors in document selection, and increased system complexity. With the advent of large language models (LLMs) featuring significantly extended context windows, this paper proposes an alternative paradigm, cache-augmented generation (CAG) that bypasses real-time retrieval. Our method involves preloading all relevant resources, especially when the documents or knowledge for retrieval are of a limited and manageable size, into the LLM's extended context and caching its runtime parameters. During inference, the model utilizes these preloaded parameters to answer queries without additional retrieval steps. Comparative analyses reveal that CAG eliminates retrieval latency and minimizes retrieval errors while maintaining context relevance. Performance evaluations across multiple benchmarks highlight scenarios where long-context LLMs either outperform or complement traditional RAG pipelines. These findings suggest that, for certain applications, particularly those with a constrained knowledge base, CAG provide a streamlined and efficient alternative to RAG, achieving comparable or superior results with reduced complexity.

Title: A Fusion Approach of Dependency Syntax and Sentiment Polarity for Feature Label Extraction in Commodity Reviews

• Authors: Jianfei Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15610
• Pdf URL: https://arxiv.org/pdf/2412.15610
• Copy Paste: [[2412.15610]] A Fusion Approach of Dependency Syntax and Sentiment Polarity for Feature Label Extraction in Commodity Reviews(https://arxiv.org/abs/2412.15610)
• Keywords: robust, extraction
• Abstract: This study analyzes 13,218 product reviews from this http URL, covering four categories: mobile phones, computers, cosmetics, and food. A novel method for feature label extraction is proposed by integrating dependency parsing and sentiment polarity analysis. The proposed method addresses the challenges of low robustness in existing extraction algorithms and significantly enhances extraction accuracy. Experimental results show that the method achieves an accuracy of 0.7, with recall and F-score both stabilizing at 0.8, demonstrating its effectiveness. However, challenges such as dependence on matching dictionaries and the limited scope of extracted feature tags require further investigation in future research.

Title: Technical Report for ICML 2024 TiFA Workshop MLLM Attack Challenge: Suffix Injection and Projected Gradient Descent Can Easily Fool An MLLM

• Authors: Yangyang Guo, Ziwei Xu, Xilie Xu, YongKang Wong, Liqiang Nie, Mohan Kankanhalli
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15614
• Pdf URL: https://arxiv.org/pdf/2412.15614
• Copy Paste: [[2412.15614]] Technical Report for ICML 2024 TiFA Workshop MLLM Attack Challenge: Suffix Injection and Projected Gradient Descent Can Easily Fool An MLLM(https://arxiv.org/abs/2412.15614)
• Keywords: attack
• Abstract: This technical report introduces our top-ranked solution that employs two approaches, \ie suffix injection and projected gradient descent (PGD) , to address the TiFA workshop MLLM attack challenge. Specifically, we first append the text from an incorrectly labeled option (pseudo-labeled) to the original query as a suffix. Using this modified query, our second approach applies the PGD method to add imperceptible perturbations to the image. Combining these two techniques enables successful attacks on the LLaVA 1.5 model.

Title: Pirates of Charity: Exploring Donation-based Abuses in Social Media Platforms

• Authors: Bhupendra Acharya, Dario Lazzaro, Antonio Emanuele Cinà, Thorsten Holz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15621
• Pdf URL: https://arxiv.org/pdf/2412.15621
• Copy Paste: [[2412.15621]] Pirates of Charity: Exploring Donation-based Abuses in Social Media Platforms(https://arxiv.org/abs/2412.15621)
• Keywords: security, protect
• Abstract: With the widespread use of social media, organizations, and individuals use these platforms to raise funds and support causes. Unfortunately, this has led to the rise of scammers in soliciting fraudulent donations. In this study, we conduct a large-scale analysis of donation-based scams on social media platforms. More specifically, we studied profile creation and scam operation fraudulent donation solicitation on X, Instagram, Facebook, YouTube, and Telegram. By collecting data from 151,966 accounts and their 3,053,333 posts related to donations between March 2024 and May 2024, we identified 832 scammers using various techniques to deceive users into making fraudulent donations. Analyzing the fraud communication channels such as phone number, email, and external URL linked, we show that these scamming accounts perform various fraudulent donation schemes, including classic abuse such as fake fundraising website setup, crowdsourcing fundraising, and asking users to communicate via email, phone, and pay via various payment methods. Through collaboration with industry partners PayPal and cryptocurrency abuse database Chainabuse, we further validated the scams and measured the financial losses on these platforms. Our study highlights significant weaknesses in social media platforms' ability to protect users from fraudulent donations. Additionally, we recommended social media platforms, and financial services for taking proactive steps to block these fraudulent activities. Our study provides a foundation for the security community and researchers to automate detecting and mitigating fraudulent donation solicitation on social media platforms.

Title: JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs

• Authors: Hongyi Li, Jiawei Ye, Jie Wu, Tianjie Yan, Chu Wang, Zhixin Li
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15623
• Pdf URL: https://arxiv.org/pdf/2412.15623
• Copy Paste: [[2412.15623]] JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs(https://arxiv.org/abs/2412.15623)
• Keywords: security, defense, attack, robust, large language model
• Abstract: Large Language Models (LLMs) aligned with human feedback have recently garnered significant attention. However, it remains vulnerable to jailbreak attacks, where adversaries manipulate prompts to induce harmful outputs. Exploring jailbreak attacks enables us to investigate the vulnerabilities of LLMs and further guides us in enhancing their security. Unfortunately, existing techniques mainly rely on handcrafted templates or generated-based optimization, posing challenges in scalability, efficiency and universality. To address these issues, we present JailPO, a novel black-box jailbreak framework to examine LLM alignment. For scalability and universality, JailPO meticulously trains attack models to automatically generate covert jailbreak prompts. Furthermore, we introduce a preference optimization-based attack method to enhance the jailbreak effectiveness, thereby improving efficiency. To analyze model vulnerabilities, we provide three flexible jailbreak patterns. Extensive experiments demonstrate that JailPO not only automates the attack process while maintaining effectiveness but also exhibits superior performance in efficiency, universality, and robustness against defenses compared to baselines. Additionally, our analysis of the three JailPO patterns reveals that attacks based on complex templates exhibit higher attack strength, whereas covert question transformations elicit riskier responses and are more likely to bypass defense mechanisms.

Title: Can Input Attributions Interpret the Inductive Reasoning Process Elicited in In-Context Learning?

• Authors: Mengyu Ye, Tatsuki Kuribayashi, Goro Kobayashi, Jun Suzuki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15628
• Pdf URL: https://arxiv.org/pdf/2412.15628
• Copy Paste: [[2412.15628]] Can Input Attributions Interpret the Inductive Reasoning Process Elicited in In-Context Learning?(https://arxiv.org/abs/2412.15628)
• Keywords: large language model
• Abstract: Elucidating the rationale behind neural models' outputs has been challenging in the machine learning field, which is indeed applicable in this age of large language models (LLMs) and in-context learning (ICL). When it comes to estimating input attributions (IA), ICL poses a new issue of interpreting which example in the prompt, consisting of a set of examples, contributed to identifying the task/rule to be solved. To this end, in this paper, we introduce synthetic diagnostic tasks inspired by the poverty of the stimulus design in inductive reasoning; here, most in-context examples are ambiguous w.r.t. their underlying rule, and one critical example disambiguates the task demonstrated. The question is whether conventional IA methods can identify such an example in interpreting the inductive reasoning process in ICL. Our experiments provide several practical findings; for example, a certain simple IA method works the best, and the larger the model, the generally harder it is to interpret the ICL with gradient-based IA methods.

Title: CrackUDA: Incremental Unsupervised Domain Adaptation for Improved Crack Segmentation in Civil Structures

• Authors: Kushagra Srivastava, Damodar Datta Kancharla, Rizvi Tahereen, Pradeep Kumar Ramancharla, Ravi Kiran Sarvadevabhatla, Harikumar Kandath
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15637
• Pdf URL: https://arxiv.org/pdf/2412.15637
• Copy Paste: [[2412.15637]] CrackUDA: Incremental Unsupervised Domain Adaptation for Improved Crack Segmentation in Civil Structures(https://arxiv.org/abs/2412.15637)
• Keywords: robust, segmentation
• Abstract: Crack segmentation plays a crucial role in ensuring the structural integrity and seismic safety of civil structures. However, existing crack segmentation algorithms encounter challenges in maintaining accuracy with domain shifts across datasets. To address this issue, we propose a novel deep network that employs incremental training with unsupervised domain adaptation (UDA) using adversarial learning, without a significant drop in accuracy in the source domain. Our approach leverages an encoder-decoder architecture, consisting of both domain-invariant and domain-specific parameters. The encoder learns shared crack features across all domains, ensuring robustness to domain variations. Simultaneously, the decoder's domain-specific parameters capture domain-specific features unique to each domain. By combining these components, our model achieves improved crack segmentation performance. Furthermore, we introduce BuildCrack, a new crack dataset comparable to sub-datasets of the well-established CrackSeg9K dataset in terms of image count and crack percentage. We evaluate our proposed approach against state-of-the-art UDA methods using different sub-datasets of CrackSeg9K and our custom dataset. Our experimental results demonstrate a significant improvement in crack segmentation accuracy and generalization across target domains compared to other UDA methods - specifically, an improvement of 0.65 and 2.7 mIoU on source and target domains respectively.

Title: CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training

• Authors: Xiuli Bi, Jian Lu, Bo Liu, Xiaodong Cun, Yong Zhang, Weisheng Li, Bin Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15646
• Pdf URL: https://arxiv.org/pdf/2412.15646
• Copy Paste: [[2412.15646]] CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training(https://arxiv.org/abs/2412.15646)
• Keywords: diffusion
• Abstract: Benefiting from large-scale pre-training of text-video pairs, current text-to-video (T2V) diffusion models can generate high-quality videos from the text description. Besides, given some reference images or videos, the parameter-efficient fine-tuning method, i.e. LoRA, can generate high-quality customized concepts, e.g., the specific subject or the motions from a reference video. However, combining the trained multiple concepts from different references into a single network shows obvious artifacts. To this end, we propose CustomTTT, where we can joint custom the appearance and the motion of the given video easily. In detail, we first analyze the prompt influence in the current video diffusion model and find the LoRAs are only needed for the specific layers for appearance and motion customization. Besides, since each LoRA is trained individually, we propose a novel test-time training technique to update parameters after combination utilizing the trained customized models. We conduct detailed experiments to verify the effectiveness of the proposed methods. Our method outperforms several state-of-the-art works in both qualitative and quantitative evaluations.

Title: Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution

• Authors: Wentao Tan, Qiong Cao, Yibing Zhan, Chao Xue, Changxing Ding
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15650
• Pdf URL: https://arxiv.org/pdf/2412.15650
• Copy Paste: [[2412.15650]] Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution(https://arxiv.org/abs/2412.15650)
• Keywords: large language model
• Abstract: Human preference alignment can greatly enhance Multimodal Large Language Models (MLLMs), but collecting high-quality preference data is costly. A promising solution is the self-evolution strategy, where models are iteratively trained on data they generate. However, current techniques still rely on human- or GPT-annotated data and sometimes require additional models or ground truth answers. To address these issues, we propose a novel multimodal self-evolution framework that enables the model to autonomously generate high-quality questions and answers using only unannotated images. First, we implement an image-driven self-questioning mechanism, allowing the model to create and evaluate questions based on image content, regenerating them if they are irrelevant or unanswerable. This sets a strong foundation for answer generation. Second, we introduce an answer self-enhancement technique, starting with image captioning to improve answer quality. We also use corrupted images to generate rejected answers, forming distinct preference pairs for optimization. Finally, we incorporate an image content alignment loss function alongside Direct Preference Optimization (DPO) loss to reduce hallucinations, ensuring the model focuses on image content. Experiments show that our framework performs competitively with methods using external information, offering a more efficient and scalable approach to MLLMs.

Title: MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula

• Authors: Sieun Hyeon, Kyudan Jung, Jaehee Won, Nam-Joon Kim, Hyun Gon Ryu, Hyuk-Jae Lee, Jaeyoung Do
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15655
• Pdf URL: https://arxiv.org/pdf/2412.15655
• Copy Paste: [[2412.15655]] MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula(https://arxiv.org/abs/2412.15655)
• Keywords: large language model
• Abstract: In various academic and professional settings, such as mathematics lectures or research presentations, it is often necessary to convey mathematical expressions orally. However, reading mathematical expressions aloud without accompanying visuals can significantly hinder comprehension, especially for those who are hearing-impaired or rely on subtitles due to language barriers. For instance, when a presenter reads Euler's Formula, current Automatic Speech Recognition (ASR) models often produce a verbose and error-prone textual description (e.g., e to the power of i x equals cosine of x plus i $\textit{side}$ of x), instead of the concise $\LaTeX{}$ format (i.e., $ e^{ix} = \cos(x) + i\sin(x) $), which hampers clear understanding and communication. To address this issue, we introduce MathSpeech, a novel pipeline that integrates ASR models with small Language Models (sLMs) to correct errors in mathematical expressions and accurately convert spoken expressions into structured $\LaTeX{}$ representations. Evaluated on a new dataset derived from lecture recordings, MathSpeech demonstrates $\LaTeX{}$ generation capabilities comparable to leading commercial Large Language Models (LLMs), while leveraging fine-tuned small language models of only 120M parameters. Specifically, in terms of CER, BLEU, and ROUGE scores for $\LaTeX{}$ translation, MathSpeech demonstrated significantly superior capabilities compared to GPT-4o. We observed a decrease in CER from 0.390 to 0.298, and higher ROUGE/BLEU scores compared to GPT-4o.

Title: Synthetic Tabular Data Generation for Imbalanced Classification: The Surprising Effectiveness of an Overlap Class

• Authors: Annie D'souza, Swetha M, Sunita Sarawagi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15657
• Pdf URL: https://arxiv.org/pdf/2412.15657
• Copy Paste: [[2412.15657]] Synthetic Tabular Data Generation for Imbalanced Classification: The Surprising Effectiveness of an Overlap Class(https://arxiv.org/abs/2412.15657)
• Keywords: diffusion, generative
• Abstract: Handling imbalance in class distribution when building a classifier over tabular data has been a problem of long-standing interest. One popular approach is augmenting the training dataset with synthetically generated data. While classical augmentation techniques were limited to linear interpolation of existing minority class examples, recently higher capacity deep generative models are providing greater promise. However, handling of imbalance in class distribution when building a deep generative model is also a challenging problem, that has not been studied as extensively as imbalanced classifier model training. We show that state-of-the-art deep generative models yield significantly lower-quality minority examples than majority examples. %In this paper, we start with the observation that imbalanced data training of generative models trained imbalanced dataset which under-represent the minority class. We propose a novel technique of converting the binary class labels to ternary class labels by introducing a class for the region where minority and majority distributions overlap. We show that just this pre-processing of the training set, significantly improves the quality of data generated spanning several state-of-the-art diffusion and GAN-based models. While training the classifier using synthetic data, we remove the overlap class from the training data and justify the reasons behind the enhanced accuracy. We perform extensive experiments on four real-life datasets, five different classifiers, and five generative models demonstrating that our method enhances not only the synthesizer performance of state-of-the-art models but also the classifier performance.

Title: SCENIC: Scene-aware Semantic Navigation with Instruction-guided Control

• Authors: Xiaohan Zhang, Sebastian Starke, Vladimir Guzov, Zhensong Zhang, Eduardo Pérez Pellitero, Gerard Pons-Moll
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15664
• Pdf URL: https://arxiv.org/pdf/2412.15664
• Copy Paste: [[2412.15664]] SCENIC: Scene-aware Semantic Navigation with Instruction-guided Control(https://arxiv.org/abs/2412.15664)
• Keywords: diffusion
• Abstract: Synthesizing natural human motion that adapts to complex environments while allowing creative control remains a fundamental challenge in motion synthesis. Existing models often fall short, either by assuming flat terrain or lacking the ability to control motion semantics through text. To address these limitations, we introduce SCENIC, a diffusion model designed to generate human motion that adapts to dynamic terrains within virtual scenes while enabling semantic control through natural language. The key technical challenge lies in simultaneously reasoning about complex scene geometry while maintaining text control. This requires understanding both high-level navigation goals and fine-grained environmental constraints. The model must ensure physical plausibility and precise navigation across varied terrain, while also preserving user-specified text control, such as ``carefully stepping over obstacles" or ``walking upstairs like a zombie." Our solution introduces a hierarchical scene reasoning approach. At its core is a novel scene-dependent, goal-centric canonicalization that handles high-level goal constraint, and is complemented by an ego-centric distance field that captures local geometric details. This dual representation enables our model to generate physically plausible motion across diverse 3D scenes. By implementing frame-wise text alignment, our system achieves seamless transitions between different motion styles while maintaining scene constraints. Experiments demonstrate our novel diffusion model generates arbitrarily long human motions that both adapt to complex scenes with varying terrain surfaces and respond to textual prompts. Additionally, we show SCENIC can generalize to four real-scene datasets. Our code, dataset, and models will be released at \url{this https URL}.

Title: Learning Group Interactions and Semantic Intentions for Multi-Object Trajectory Prediction

• Authors: Mengshi Qi, Yuxin Yang, Huadong Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15673
• Pdf URL: https://arxiv.org/pdf/2412.15673
• Copy Paste: [[2412.15673]] Learning Group Interactions and Semantic Intentions for Multi-Object Trajectory Prediction(https://arxiv.org/abs/2412.15673)
• Keywords: diffusion
• Abstract: Effective modeling of group interactions and dynamic semantic intentions is crucial for forecasting behaviors like trajectories or movements. In complex scenarios like sports, agents' trajectories are influenced by group interactions and intentions, including team strategies and opponent actions. To this end, we propose a novel diffusion-based trajectory prediction framework that integrates group-level interactions into a conditional diffusion model, enabling the generation of diverse trajectories aligned with specific group activity. To capture dynamic semantic intentions, we frame group interaction prediction as a cooperative game, using Banzhaf interaction to model cooperation trends. We then fuse semantic intentions with enhanced agent embeddings, which are refined through both global and local aggregation. Furthermore, we expand the NBA SportVU dataset by adding human annotations of team-level tactics for trajectory and tactic prediction tasks. Extensive experiments on three widely-adopted datasets demonstrate that our model outperforms state-of-the-art methods. Our source code and data are available at this https URL.

Title: PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium

• Authors: Xinzhe Li, Jiahui Zhan, Shengfeng He, Yangyang Xu, Junyu Dong, Huaidong Zhang, Yong Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15674
• Pdf URL: https://arxiv.org/pdf/2412.15674
• Copy Paste: [[2412.15674]] PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium(https://arxiv.org/abs/2412.15674)
• Keywords: robust, generative
• Abstract: Personalized image generation has made significant strides in adapting content to novel concepts. However, a persistent challenge remains: balancing the accurate reconstruction of unseen concepts with the need for editability according to the prompt, especially when dealing with the complex nuances of facial features. In this study, we delve into the temporal dynamics of the text-to-image conditioning process, emphasizing the crucial role of stage partitioning in introducing new concepts. We present PersonaMagic, a stage-regulated generative technique designed for high-fidelity face customization. Using a simple MLP network, our method learns a series of embeddings within a specific timestep interval to capture face concepts. Additionally, we develop a Tandem Equilibrium mechanism that adjusts self-attention responses in the text encoder, balancing text description and identity preservation, improving both areas. Extensive experiments confirm the superiority of PersonaMagic over state-of-the-art methods in both qualitative and quantitative evaluations. Moreover, its robustness and flexibility are validated in non-facial domains, and it can also serve as a valuable plug-in for enhancing the performance of pretrained personalization models.

Title: DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization

• Authors: Zihan Ding, Chi Jin, Difan Liu, Haitian Zheng, Krishna Kumar Singh, Qiang Zhang, Yan Kang, Zhe Lin, Yuchen Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15689
• Pdf URL: https://arxiv.org/pdf/2412.15689
• Copy Paste: [[2412.15689]] DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization(https://arxiv.org/abs/2412.15689)
• Keywords: diffusion
• Abstract: Diffusion probabilistic models have shown significant progress in video generation; however, their computational efficiency is limited by the large number of sampling steps required. Reducing sampling steps often compromises video quality or generation diversity. In this work, we introduce a distillation method that combines variational score distillation and consistency distillation to achieve few-step video generation, maintaining both high quality and diversity. We also propose a latent reward model fine-tuning approach to further enhance video generation performance according to any specified reward metric. This approach reduces memory usage and does not require the reward to be differentiable. Our method demonstrates state-of-the-art performance in few-step generation for 10-second videos (128 frames at 12 FPS). The distilled student model achieves a score of 82.57 on VBench, surpassing the teacher model as well as baseline models Gen-3, T2V-Turbo, and Kling. One-step distillation accelerates the teacher model's diffusion sampling by up to 278.6 times, enabling near real-time generation. Human evaluations further validate the superior performance of our 4-step student models compared to teacher model using 50-step DDIM sampling.

Title: Revealing the Black Box of Device Search Engine: Scanning Assets, Strategies, and Ethical Consideration

• Authors: Mengying Wu, Geng Hong, Jinsong Chen, Qi Liu, Shujun Tang, Youhao Li, Baojun Liu, Haixin Duan, Min Yang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15696
• Pdf URL: https://arxiv.org/pdf/2412.15696
• Copy Paste: [[2412.15696]] Revealing the Black Box of Device Search Engine: Scanning Assets, Strategies, and Ethical Consideration(https://arxiv.org/abs/2412.15696)
• Keywords: security, privacy, protect
• Abstract: In the digital age, device search engines such as Censys and Shodan play crucial roles by scanning the internet to catalog online devices, aiding in the understanding and mitigation of network security risks. While previous research has used these tools to detect devices and assess vulnerabilities, there remains uncertainty regarding the assets they scan, the strategies they employ, and whether they adhere to ethical guidelines. This study presents the first comprehensive examination of these engines' operational and ethical dimensions. We developed a novel framework to trace the IP addresses utilized by these engines and collected 1,407 scanner IPs. By uncovering their IPs, we gain deep insights into the actions of device search engines for the first time and gain original findings. By employing 28 honeypots to monitor their scanning activities extensively in one year, we demonstrate that users can hardly evade scans by blocklisting scanner IPs or migrating service ports. Our findings reveal significant ethical concerns, including a lack of transparency, harmlessness, and anonymity. Notably, these engines often fail to provide transparency and do not allow users to opt out of scans. Further, the engines send malformed requests, attempt to access excessive details without authorization, and even publish personally identifiable information (PII) and screenshots on search results. These practices compromise user privacy and expose devices to further risks by potentially aiding malicious entities. This paper emphasizes the urgent need for stricter ethical standards and enhanced transparency in the operations of device search engines, offering crucial insights into safeguarding against invasive scanning practices and protecting digital infrastructures.

Title: PoisonCatcher: Revealing and Identifying LDP Poisoning Attacks in IIoT

• Authors: Lisha Shuai, Shaofeng Tan, Nan Zhang, Jiamin Zhang, Min Zhang, Xiaolong Yang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15704
• Pdf URL: https://arxiv.org/pdf/2412.15704
• Copy Paste: [[2412.15704]] PoisonCatcher: Revealing and Identifying LDP Poisoning Attacks in IIoT(https://arxiv.org/abs/2412.15704)
• Keywords: privacy, attack, robust, steal
• Abstract: Local Differential Privacy (LDP) is widely adopted in the Industrial Internet of Things (IIoT) for its lightweight, decentralized, and scalable nature. However, its perturbation-based privacy mechanism makes it difficult to distinguish between uncontaminated and tainted data, encouraging adversaries to launch poisoning attacks. While LDP provides some resilience against minor poisoning, it lacks robustness in IIoT with dynamic networks and substantial real-time data flows. Effective countermeasures for such attacks are still underdeveloped. This work narrows the critical gap by revealing and identifying LDP poisoning attacks in IIoT. We begin by deepening the understanding of such attacks, revealing novel threats that arise from the interplay between LDP indistinguishability and IIoT complexity. This exploration uncovers a novel rule-poisoning attack, and presents a general attack formulation by unifying it with input-poisoning and output-poisoning. Furthermore, two key attack impacts, i.e., Statistical Query Result (SQR) accuracy degradation and inter-dataset correlations disruption, along with two characteristics: attack patterns unstable and poisoned data stealth are revealed. From this, we propose PoisonCatcher, a four-stage solution that detects LDP poisoning attacks and identifies specific contaminated data points. It utilizes temporal similarity, attribute correlation, and time-series stability analysis to detect datasets exhibiting SQR accuracy degradation, inter-dataset disruptions, and unstable patterns. Enhanced feature engineering is used to extract subtle poisoning signatures, enabling machine learning models to identify specific contamination. Experimental evaluations show the effectiveness, achieving state-of-the-art performance with average precision and recall rates of 86.17% and 97.5%, respectively, across six representative attack scenarios.

Title: Contrastive Learning for Task-Independent SpeechLLM-Pretraining

• Authors: Maike Züfle, Jan Niehues
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.15712
• Pdf URL: https://arxiv.org/pdf/2412.15712
• Copy Paste: [[2412.15712]] Contrastive Learning for Task-Independent SpeechLLM-Pretraining(https://arxiv.org/abs/2412.15712)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs. To address these challenges, we propose a scalable, two-stage training approach: (1) A task-independent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a task-specific fine-tuning stage requiring minimal data. This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.

Title: Towards Secure AI-driven Industrial Metaverse with NFT Digital Twins

• Authors: Ravi Prakash, Tony Thomas
• Subjects: cs.CR, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.15716
• Pdf URL: https://arxiv.org/pdf/2412.15716
• Copy Paste: [[2412.15716]] Towards Secure AI-driven Industrial Metaverse with NFT Digital Twins(https://arxiv.org/abs/2412.15716)
• Keywords: secure, security
• Abstract: The rise of the industrial metaverse has brought digital twins (DTs) to the forefront. Blockchain-powered non-fungible tokens (NFTs) offer a decentralized approach to creating and owning these cloneable DTs. However, the potential for unauthorized duplication, or counterfeiting, poses a significant threat to the security of NFT-DTs. Existing NFT clone detection methods often rely on static information like metadata and images, which can be easily manipulated. To address these limitations, we propose a novel deep-learning-based solution as a combination of an autoencoder and RNN-based classifier. This solution enables real-time pattern recognition to detect fake NFT-DTs. Additionally, we introduce the concept of dynamic metadata, providing a more reliable way to verify authenticity through AI-integrated smart contracts. By effectively identifying counterfeit DTs, our system contributes to strengthening the security of NFT-based assets in the metaverse.

Title: Fine-tuning Whisper on Low-Resource Languages for Real-World Applications

• Authors: Vincenzo Timmel, Claudio Paonessa, Reza Kakooee, Manfred Vogel, Daniel Perruchoud
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.15726
• Pdf URL: https://arxiv.org/pdf/2412.15726
• Copy Paste: [[2412.15726]] Fine-tuning Whisper on Low-Resource Languages for Real-World Applications(https://arxiv.org/abs/2412.15726)
• Keywords: segmentation
• Abstract: This paper presents a new approach to fine-tuning OpenAI's Whisper model for low-resource languages by introducing a novel data generation method that converts sentence-level data into a long-form corpus, using Swiss German as a case study. Non-sentence-level data, which could improve the performance of long-form audio, is difficult to obtain and often restricted by copyright laws. Our method bridges this gap by transforming more accessible sentence-level data into a format that preserves the model's ability to handle long-form audio and perform segmentation without requiring non-sentence-level data. Our data generation process improves performance in several real-world applications and leads to the development of a new state-of-the-art speech-to-text (STT) model for Swiss German. We compare our model with a non-fine-tuned Whisper and our previous state-of-the-art Swiss German STT models, where our new model achieves higher BLEU scores. Our results also indicate that the proposed method is adaptable to other low-resource languages, supported by written guidance and code that allows the creation of fine-tuned Whisper models, which keep segmentation capabilities and allow the transcription of longer audio files using only sentence-level data with high quality.

Title: fluke: Federated Learning Utility frameworK for Experimentation and research

• Authors: Mirko Polato
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15728
• Pdf URL: https://arxiv.org/pdf/2412.15728
• Copy Paste: [[2412.15728]] fluke: Federated Learning Utility frameworK for Experimentation and research(https://arxiv.org/abs/2412.15728)
• Keywords: federate
• Abstract: Since its inception in 2016, Federated Learning (FL) has been gaining tremendous popularity in the machine learning community. Several frameworks have been proposed to facilitate the development of FL algorithms, but researchers often resort to implementing their algorithms from scratch, including all baselines and experiments. This is because existing frameworks are not flexible enough to support their needs or the learning curve to extend them is too steep. In this paper, we present \fluke, a Python package designed to simplify the development of new FL algorithms. fluke is specifically designed for prototyping purposes and is meant for researchers or practitioners focusing on the learning components of a federated system. fluke is open-source, and it can be either used out of the box or extended with new algorithms with minimal overhead.

Title: The Role of Recurrency in Image Segmentation for Noisy and Limited Sample Settings

• Authors: David Calhas, João Marques, Arlindo L. Oliveira
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15734
• Pdf URL: https://arxiv.org/pdf/2412.15734
• Copy Paste: [[2412.15734]] The Role of Recurrency in Image Segmentation for Noisy and Limited Sample Settings(https://arxiv.org/abs/2412.15734)
• Keywords: segmentation
• Abstract: The biological brain has inspired multiple advances in machine learning. However, most state-of-the-art models in computer vision do not operate like the human brain, simply because they are not capable of changing or improving their decisions/outputs based on a deeper analysis. The brain is recurrent, while these models are not. It is therefore relevant to explore what would be the impact of adding recurrent mechanisms to existing state-of-the-art architectures and to answer the question of whether recurrency can improve existing architectures. To this end, we build on a feed-forward segmentation model and explore multiple types of recurrency for image segmentation. We explore self-organizing, relational, and memory retrieval types of recurrency that minimize a specific energy function. In our experiments, we tested these models on artificial and medical imaging data, while analyzing the impact of high levels of noise and few-shot learning settings. Our results do not validate our initial hypothesis that recurrent models should perform better in these settings, suggesting that these recurrent architectures, by themselves, are not sufficient to surpass state-of-the-art feed-forward versions and that additional work needs to be done on the topic.

Title: Prompt-based Unifying Inference Attack on Graph Neural Networks

• Authors: Yuecen Wei, Xingcheng Fu, Lingyun Liu, Qingyun Sun, Hao Peng, Chunming Hu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15735
• Pdf URL: https://arxiv.org/pdf/2412.15735
• Copy Paste: [[2412.15735]] Prompt-based Unifying Inference Attack on Graph Neural Networks(https://arxiv.org/abs/2412.15735)
• Keywords: attack
• Abstract: Graph neural networks (GNNs) provide important prospective insights in applications such as social behavior analysis and financial risk analysis based on their powerful learning capabilities on graph data. Nevertheless, GNNs' predictive performance relies on the quality of task-specific node labels, so it is common practice to improve the model's generalization ability in the downstream execution of decision-making tasks through pre-training. Graph prompting is a prudent choice but risky without taking measures to prevent data leakage. In other words, in high-risk decision scenarios, prompt learning can infer private information by accessing model parameters trained on private data (publishing model parameters in pre-training, i.e., without directly leaking the raw data, is a tacitly accepted trend). However, myriad graph inference attacks necessitate tailored module design and processing to enhance inference capabilities due to variations in supervision signals. In this paper, we propose a novel Prompt-based unifying Inference Attack framework on GNNs, named ProIA. Specifically, ProIA retains the crucial topological information of the graph during pre-training, enhancing the background knowledge of the inference attack model. It then utilizes a unified prompt and introduces additional disentanglement factors in downstream attacks to adapt to task-relevant knowledge. Finally, extensive experiments show that ProIA enhances attack capabilities and demonstrates remarkable adaptability to various inference attacks.

Title: VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Models

• Authors: Dexter Neo, Tsuhan Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15739
• Pdf URL: https://arxiv.org/pdf/2412.15739
• Copy Paste: [[2412.15739]] VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Models(https://arxiv.org/abs/2412.15739)
• Keywords: large language model
• Abstract: Large Vision-Language Models (LVLMs) have made remarkable developments along with the recent surge of large language models. Despite their advancements, LVLMs have a tendency to generate plausible yet inaccurate or inconsistent information based on the provided source content. This phenomenon, also known as ``hallucinations" can have serious downstream implications during the deployment of LVLMs. To address this, we present VORD a simple and effective method that alleviates hallucinations by calibrating token predictions based on ordinal relationships between modified image pairs. VORD is presented in two forms: 1.) a minimalist training-free variant which eliminates implausible tokens from modified image pairs, and 2.) a trainable objective function that penalizes unlikely tokens. Our experiments demonstrate that VORD delivers better calibration and effectively mitigates object hallucinations on a wide-range of LVLM benchmarks.

Title: Critique of Impure Reason: Unveiling the reasoning behaviour of medical Large Language Models

• Authors: Shamus Sim, Tyrone Chen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15748
• Pdf URL: https://arxiv.org/pdf/2412.15748
• Copy Paste: [[2412.15748]] Critique of Impure Reason: Unveiling the reasoning behaviour of medical Large Language Models(https://arxiv.org/abs/2412.15748)
• Keywords: large language model
• Abstract: Background: Despite the current ubiquity of Large Language Models (LLMs) across the medical domain, there is a surprising lack of studies which address their reasoning behaviour. We emphasise the importance of understanding reasoning behaviour as opposed to high-level prediction accuracies, since it is equivalent to explainable AI (XAI) in this context. In particular, achieving XAI in medical LLMs used in the clinical domain will have a significant impact across the healthcare sector. Results: Therefore, we define the concept of reasoning behaviour in the specific context of medical LLMs. We then categorise and discuss the current state of the art of methods which evaluate reasoning behaviour in medical LLMs. Finally, we propose theoretical frameworks which can empower medical professionals or machine learning engineers to gain insight into the low-level reasoning operations of these previously obscure models. Conclusion: The subsequent increased transparency and trust in medical machine learning models by clinicians as well as patients will accelerate the integration, application as well as further development of medical AI for the healthcare system as a whole

Title: Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference

• Authors: Jorge García-Carrasco, Alejandro Maté, Juan Trujillo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15750
• Pdf URL: https://arxiv.org/pdf/2412.15750
• Copy Paste: [[2412.15750]] Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference(https://arxiv.org/abs/2412.15750)
• Keywords: interpretability, large language model
• Abstract: Large Language Models (LLMs) have shown impressive performance across a wide range of tasks. However, the size of LLMs is steadily increasing, hindering their application on computationally constrained environments. On the other hand, despite their general capabilities, there are many situations where only one specific task is performed, rendering all other capabilities unnecessary and wasteful. This leads us to the following question: Is it possible to extract the minimal subset from an LLM that is able to perform a specific task in a faster, standalone manner? Recent works on Mechanistic Interpretability (MI) have shown that specific tasks are performed by a localized subset of components, or circuit. However, current techniques used to identify the circuit cannot be used to extract it for its standalone usage. In this work, we propose a novel approach to automatically extract the subset of the LLM that properly performs a targeted task requiring no additional training and a small amount of data samples. We evaluate our approach on different tasks and show that the resulting models are (i) considerably smaller, reducing the number of parameters up to 82.77% and (ii) more interpretable, as they focus on the circuit that is used to carry out the specific task, and can therefore be understood using MI techniques.

Title: Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles

• Authors: Sophie Steger, Christian Knoll, Bernhard Klein, Holger Fröning, Franz Pernkopf
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15758
• Pdf URL: https://arxiv.org/pdf/2412.15758
• Copy Paste: [[2412.15758]] Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles(https://arxiv.org/abs/2412.15758)
• Keywords: robust
• Abstract: Bayesian inference in function space has gained attention due to its robustness against overparameterization in neural networks. However, approximating the infinite-dimensional function space introduces several challenges. In this work, we discuss function space inference via particle optimization and present practical modifications that improve uncertainty estimation and, most importantly, make it applicable for large and pretrained networks. First, we demonstrate that the input samples, where particle predictions are enforced to be diverse, are detrimental to the model performance. While diversity on training data itself can lead to underfitting, the use of label-destroying data augmentation, or unlabeled out-of-distribution data can improve prediction diversity and uncertainty estimates. Furthermore, we take advantage of the function space formulation, which imposes no restrictions on network parameterization other than sufficient flexibility. Instead of using full deep ensembles to represent particles, we propose a single multi-headed network that introduces a minimal increase in parameters and computation. This allows seamless integration to pretrained networks, where this repulsive last-layer ensemble can be used for uncertainty aware fine-tuning at minimal additional cost. We achieve competitive results in disentangling aleatoric and epistemic uncertainty for active learning, detecting out-of-domain data, and providing calibrated uncertainty estimates under distribution shifts with minimal compute and memory.

Title: Linguistic Features Extracted by GPT-4 Improve Alzheimer's Disease Detection based on Spontaneous Speech

• Authors: Jonathan Heitz, Gerold Schneider, Nicolas Langer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15772
• Pdf URL: https://arxiv.org/pdf/2412.15772
• Copy Paste: [[2412.15772]] Linguistic Features Extracted by GPT-4 Improve Alzheimer's Disease Detection based on Spontaneous Speech(https://arxiv.org/abs/2412.15772)
• Keywords: large language model
• Abstract: Alzheimer's Disease (AD) is a significant and growing public health concern. Investigating alterations in speech and language patterns offers a promising path towards cost-effective and non-invasive early detection of AD on a large scale. Large language models (LLMs), such as GPT, have enabled powerful new possibilities for semantic text analysis. In this study, we leverage GPT-4 to extract five semantic features from transcripts of spontaneous patient speech. The features capture known symptoms of AD, but they are difficult to quantify effectively using traditional methods of computational linguistics. We demonstrate the clinical significance of these features and further validate one of them ("Word-Finding Difficulties") against a proxy measure and human raters. When combined with established linguistic features and a Random Forest classifier, the GPT-derived features significantly improve the detection of AD. Our approach proves effective for both manually transcribed and automatically generated transcripts, representing a novel and impactful use of recent advancements in LLMs for AD speech analysis.

Title: Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning

• Authors: Sungjin Park, Xiao Liu, Yeyun Gong, Edward Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15797
• Pdf URL: https://arxiv.org/pdf/2412.15797
• Copy Paste: [[2412.15797]] Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning(https://arxiv.org/abs/2412.15797)
• Keywords: large language model
• Abstract: Despite recent advances in large language models, open-source models often struggle to consistently perform well on complex reasoning tasks. Existing ensemble methods, whether applied at the token or output levels, fail to address these challenges. In response, we present Language model Ensemble with Monte Carlo Tree Search (LE-MCTS), a novel framework for process-level ensembling of language models. LE-MCTS formulates step-by-step reasoning with an ensemble of language models as a Markov decision process. In this framework, states represent intermediate reasoning paths, while actions consist of generating the next reasoning step using one of the language models selected from a predefined pool. Guided by a process-based reward model, LE-MCTS performs a tree search over the reasoning steps generated by different language models, identifying the most accurate reasoning chain. Experimental results on five mathematical reasoning benchmarks demonstrate that our approach outperforms both single language model decoding algorithms and language model ensemble methods. Notably, LE-MCTS improves performance by 3.6% and 4.3% on the MATH and MQA datasets, respectively, highlighting its effectiveness in solving complex reasoning problems.

Title: Diffusion-Based Conditional Image Editing through Optimized Inference with Guidance

• Authors: Hyunsoo Lee, Minsoo Kang, Bohyung Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15798
• Pdf URL: https://arxiv.org/pdf/2412.15798
• Copy Paste: [[2412.15798]] Diffusion-Based Conditional Image Editing through Optimized Inference with Guidance(https://arxiv.org/abs/2412.15798)
• Keywords: diffusion
• Abstract: We present a simple but effective training-free approach for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. Our goal is to generate an image that aligns with the target task while preserving the structure and background of a source image. To this end, we derive the representation guidance with a combination of two objectives: maximizing the similarity to the target prompt based on the CLIP score and minimizing the structural distance to the source latent variable. This guidance improves the fidelity of the generated target image to the given target prompt while maintaining the structure integrity of the source image. To incorporate the representation guidance component, we optimize the target latent variable of diffusion model's reverse process with the guidance. Experimental results demonstrate that our method achieves outstanding image-to-image translation performance on various tasks when combined with the pretrained Stable Diffusion model.

Title: WebLLM: A High-Performance In-Browser LLM Inference Engine

• Authors: Charlie F. Ruan, Yucheng Qin, Xun Zhou, Ruihang Lai, Hongyi Jin, Yixin Dong, Bohan Hou, Meng-Shiun Yu, Yiyan Zhai, Sudeep Agarwal, Hangrui Cao, Siyuan Feng, Tianqi Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15803
• Pdf URL: https://arxiv.org/pdf/2412.15803
• Copy Paste: [[2412.15803]] WebLLM: A High-Performance In-Browser LLM Inference Engine(https://arxiv.org/abs/2412.15803)
• Keywords: privacy, large language model
• Abstract: Advancements in large language models (LLMs) have unlocked remarkable capabilities. While deploying these models typically requires server-grade GPUs and cloud-based inference, the recent emergence of smaller open-source models and increasingly powerful consumer devices have made on-device deployment practical. The web browser as a platform for on-device deployment is universally accessible, provides a natural agentic environment, and conveniently abstracts out the different backends from diverse device vendors. To address this opportunity, we introduce WebLLM, an open-source JavaScript framework that enables high-performance LLM inference entirely within web browsers. WebLLM provides an OpenAI-style API for seamless integration into web applications, and leverages WebGPU for efficient local GPU acceleration and WebAssembly for performant CPU computation. With machine learning compilers MLC-LLM and Apache TVM, WebLLM leverages optimized WebGPU kernels, overcoming the absence of performant WebGPU kernel libraries. Evaluations show that WebLLM can retain up to 80% native performance on the same device, with room to further close the gap. WebLLM paves the way for universally accessible, privacy-preserving, personalized, and locally powered LLM applications in web browsers. The code is available at: this https URL.

Title: Cross-Modal Few-Shot Learning with Second-Order Neural Ordinary Differential Equations

• Authors: Yi Zhang, Chun-Wun Cheng, Junyi He, Zhihai He, Carola-Bibiane Schönlieb, Yuyan Chen, Angelica I Aviles-Rivero
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15813
• Pdf URL: https://arxiv.org/pdf/2412.15813
• Copy Paste: [[2412.15813]] Cross-Modal Few-Shot Learning with Second-Order Neural Ordinary Differential Equations(https://arxiv.org/abs/2412.15813)
• Keywords: robust
• Abstract: We introduce SONO, a novel method leveraging Second-Order Neural Ordinary Differential Equations (Second-Order NODEs) to enhance cross-modal few-shot learning. By employing a simple yet effective architecture consisting of a Second-Order NODEs model paired with a cross-modal classifier, SONO addresses the significant challenge of overfitting, which is common in few-shot scenarios due to limited training examples. Our second-order approach can approximate a broader class of functions, enhancing the model's expressive power and feature generalization capabilities. We initialize our cross-modal classifier with text embeddings derived from class-relevant prompts, streamlining training efficiency by avoiding the need for frequent text encoder processing. Additionally, we utilize text-based image augmentation, exploiting CLIP's robust image-text correlation to enrich training data significantly. Extensive experiments across multiple datasets demonstrate that SONO outperforms existing state-of-the-art methods in few-shot learning performance.

Title: Unveiling the Mechanisms of DAI: A Logic-Based Approach to Stablecoin Analysis

• Authors: Francesco De Sclavis, Giuseppe Galano, Aldo Glielmo, Matteo Nardelli
• Subjects: cs.CR, cs.DC, cs.LO
• Abstract URL: https://arxiv.org/abs/2412.15814
• Pdf URL: https://arxiv.org/pdf/2412.15814
• Copy Paste: [[2412.15814]] Unveiling the Mechanisms of DAI: A Logic-Based Approach to Stablecoin Analysis(https://arxiv.org/abs/2412.15814)
• Keywords: robust
• Abstract: Stablecoins are digital assets designed to maintain a stable value, typically pegged to traditional currencies. Despite their growing prominence, many stablecoins have struggled to consistently meet stability expectations, and their underlying mechanisms often remain opaque and challenging to analyze. This paper focuses on the DAI stablecoin, which combines crypto-collateralization and algorithmic mechanisms. We propose a formal logic-based framework for representing the policies and operations of DAI, implemented in Prolog and released as open-source software. Our framework enables detailed analysis and simulation of DAI's stability mechanisms, providing a foundation for understanding its robustness and identifying potential vulnerabilities.

Title: Robustness-enhanced Myoelectric Control with GAN-based Open-set Recognition

• Authors: Cheng Wang, Ziyang Feng, Pin Zhang, Manjiang Cao, Yiming Yuan, Tengfei Chang
• Subjects: cs.CV, cs.HC, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.15819
• Pdf URL: https://arxiv.org/pdf/2412.15819
• Copy Paste: [[2412.15819]] Robustness-enhanced Myoelectric Control with GAN-based Open-set Recognition(https://arxiv.org/abs/2412.15819)
• Keywords: robust, generative
• Abstract: Electromyography (EMG) signals are widely used in human motion recognition and medical rehabilitation, yet their variability and susceptibility to noise significantly limit the reliability of myoelectric control systems. Existing recognition algorithms often fail to handle unfamiliar actions effectively, leading to system instability and errors. This paper proposes a novel framework based on Generative Adversarial Networks (GANs) to enhance the robustness and usability of myoelectric control systems by enabling open-set recognition. The method incorporates a GAN-based discriminator to identify and reject unknown actions, maintaining system stability by preventing misclassifications. Experimental evaluations on publicly available and self-collected datasets demonstrate a recognition accuracy of 97.6\% for known actions and a 23.6\% improvement in Active Error Rate (AER) after rejecting unknown actions. The proposed approach is computationally efficient and suitable for deployment on edge devices, making it practical for real-world applications.

Title: S$^2$DN: Learning to Denoise Unconvincing Knowledge for Inductive Knowledge Graph Completion

• Authors: Tengfei Ma, Yujie Chen, Liang Wang, Xuan Lin, Bosheng Song, Xiangxiang Zeng
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15822
• Pdf URL: https://arxiv.org/pdf/2412.15822
• Copy Paste: [[2412.15822]] S$^2$DN: Learning to Denoise Unconvincing Knowledge for Inductive Knowledge Graph Completion(https://arxiv.org/abs/2412.15822)
• Keywords: robust
• Abstract: Inductive Knowledge Graph Completion (KGC) aims to infer missing facts between newly emerged entities within knowledge graphs (KGs), posing a significant challenge. While recent studies have shown promising results in inferring such entities through knowledge subgraph reasoning, they suffer from (i) the semantic inconsistencies of similar relations, and (ii) noisy interactions inherent in KGs due to the presence of unconvincing knowledge for emerging entities. To address these challenges, we propose a Semantic Structure-aware Denoising Network (S$^2$DN) for inductive KGC. Our goal is to learn adaptable general semantics and reliable structures to distill consistent semantic knowledge while preserving reliable interactions within KGs. Specifically, we introduce a semantic smoothing module over the enclosing subgraphs to retain the universal semantic knowledge of relations. We incorporate a structure refining module to filter out unreliable interactions and offer additional knowledge, retaining robust structure surrounding target links. Extensive experiments conducted on three benchmark KGs demonstrate that S$^2$DN surpasses the performance of state-of-the-art models. These results demonstrate the effectiveness of S$^2$DN in preserving semantic consistency and enhancing the robustness of filtering out unreliable interactions in contaminated KGs.

Title: Measuring Cross-Modal Interactions in Multimodal Models

• Authors: Laura Wenderoth, Konstantin Hemker, Nikola Simidjievski, Mateja Jamnik
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15828
• Pdf URL: https://arxiv.org/pdf/2412.15828
• Copy Paste: [[2412.15828]] Measuring Cross-Modal Interactions in Multimodal Models(https://arxiv.org/abs/2412.15828)
• Keywords: explainability
• Abstract: Integrating AI in healthcare can greatly improve patient care and system efficiency. However, the lack of explainability in AI systems (XAI) hinders their clinical adoption, especially in multimodal settings that use increasingly complex model architectures. Most existing XAI methods focus on unimodal models, which fail to capture cross-modal interactions crucial for understanding the combined impact of multiple data sources. Existing methods for quantifying cross-modal interactions are limited to two modalities, rely on labelled data, and depend on model performance. This is problematic in healthcare, where XAI must handle multiple data sources and provide individualised explanations. This paper introduces InterSHAP, a cross-modal interaction score that addresses the limitations of existing approaches. InterSHAP uses the Shapley interaction index to precisely separate and quantify the contributions of the individual modalities and their interactions without approximations. By integrating an open-source implementation with the SHAP package, we enhance reproducibility and ease of use. We show that InterSHAP accurately measures the presence of cross-modal interactions, can handle multiple modalities, and provides detailed explanations at a local level for individual samples. Furthermore, we apply InterSHAP to multimodal medical datasets and demonstrate its applicability for individualised explanations.

Title: Enhancing Generalized Few-Shot Semantic Segmentation via Effective Knowledge Transfer

• Authors: Xinyue Chen, Miaojing Shi, Zijian Zhou, Lianghua He, Sophia Tsoka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15835
• Pdf URL: https://arxiv.org/pdf/2412.15835
• Copy Paste: [[2412.15835]] Enhancing Generalized Few-Shot Semantic Segmentation via Effective Knowledge Transfer(https://arxiv.org/abs/2412.15835)
• Keywords: segmentation
• Abstract: Generalized few-shot semantic segmentation (GFSS) aims to segment objects of both base and novel classes, using sufficient samples of base classes and few samples of novel classes. Representative GFSS approaches typically employ a two-phase training scheme, involving base class pre-training followed by novel class fine-tuning, to learn the classifiers for base and novel classes respectively. Nevertheless, distribution gap exists between base and novel classes in this process. To narrow this gap, we exploit effective knowledge transfer from base to novel classes. First, a novel prototype modulation module is designed to modulate novel class prototypes by exploiting the correlations between base and novel classes. Second, a novel classifier calibration module is proposed to calibrate the weight distribution of the novel classifier according to that of the base classifier. Furthermore, existing GFSS approaches suffer from a lack of contextual information for novel classes due to their limited samples, we thereby introduce a context consistency learning scheme to transfer the contextual knowledge from base to novel classes. Extensive experiments on PASCAL-5$^i$ and COCO-20$^i$ demonstrate that our approach significantly enhances the state of the art in the GFSS setting. The code is available at: this https URL.

Title: Multi-dimensional Visual Prompt Enhanced Image Restoration via Mamba-Transformer Aggregation

• Authors: Aiwen Jiang, Hourong Chen, Zhiwen Chen, Jihua Ye, Mingwen Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15845
• Pdf URL: https://arxiv.org/pdf/2412.15845
• Copy Paste: [[2412.15845]] Multi-dimensional Visual Prompt Enhanced Image Restoration via Mamba-Transformer Aggregation(https://arxiv.org/abs/2412.15845)
• Keywords: transformer
• Abstract: Recent efforts on image restoration have focused on developing "all-in-one" models that can handle different degradation types and levels within single model. However, most of mainstream Transformer-based ones confronted with dilemma between model capabilities and computation burdens, since self-attention mechanism quadratically increase in computational complexity with respect to image size, and has inadequacies in capturing long-range dependencies. Most of Mamba-related ones solely scanned feature map in spatial dimension for global modeling, failing to fully utilize information in channel dimension. To address aforementioned problems, this paper has proposed to fully utilize complementary advantages from Mamba and Transformer without sacrificing computation efficiency. Specifically, the selective scanning mechanism of Mamba is employed to focus on spatial modeling, enabling capture long-range spatial dependencies under linear complexity. The self-attention mechanism of Transformer is applied to focus on channel modeling, avoiding high computation burdens that are in quadratic growth with image's spatial dimensions. Moreover, to enrich informative prompts for effective image restoration, multi-dimensional prompt learning modules are proposed to learn prompt-flows from multi-scale encoder/decoder layers, benefiting for revealing underlying characteristic of various degradations from both spatial and channel perspectives, therefore, enhancing the capabilities of "all-in-one" model to solve various restoration tasks. Extensive experiment results on several image restoration benchmark tasks such as image denoising, dehazing, and deraining, have demonstrated that the proposed method can achieve new state-of-the-art performance, compared with many popular mainstream methods. Related source codes and pre-trained parameters will be public on github this https URL.

Title: Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation

• Authors: Kai Brandenbusch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15853
• Pdf URL: https://arxiv.org/pdf/2412.15853
• Copy Paste: [[2412.15853]] Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation(https://arxiv.org/abs/2412.15853)
• Keywords: diffusion
• Abstract: The generation of images of realistic looking, readable handwritten text is a challenging task which is referred to as handwritten text generation (HTG). Given a string and examples from a writer, the goal is to synthesize an image depicting the correctly spelled word in handwriting with the calligraphic style of the desired writer. An important application of HTG is the generation of training images in order to adapt downstream models for new data sets. With their success in natural image generation, diffusion models (DMs) have become the state-of-the-art approach in HTG. In this work, we present an extension of a latent DM for HTG to enable generation of writing styles not seen during training by learning style conditioning with a masked auto encoder. Our proposed content encoder allows for different ways of conditioning the DM on textual and calligraphic features. Additionally, we employ classifier-free guidance and explore the influence on the quality of the generated training images. For adapting the model to a new unlabeled data set, we propose a semi-supervised training scheme. We evaluate our approach on the IAM-database and use the RIMES-database to examine the generation of data not seen during training achieving improvements in this particularly promising application of DMs for HTG.

Title: TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain

• Authors: Camille Barboule, Viet-Phi Huynh, Adrien Bufort, Yoan Chabot, Géraldine Damnati, Gwénolé Lecorvé
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15891
• Pdf URL: https://arxiv.org/pdf/2412.15891
• Copy Paste: [[2412.15891]] TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain(https://arxiv.org/abs/2412.15891)
• Keywords: large language model
• Abstract: Despite outstanding processes in many tasks, Large Language Models (LLMs) still lack accuracy when dealing with highly technical domains. Especially, telecommunications (telco) is a particularly challenging domain due the large amount of lexical, semantic and conceptual peculiarities. Yet, this domain holds many valuable use cases, directly linked to industrial needs. Hence, this paper studies how LLMs can be adapted to the telco domain. It reports our effort to (i) collect a massive corpus of domain-specific data (800M tokens, 80K instructions), (ii) perform adaptation using various methodologies, and (iii) benchmark them against larger generalist models in downstream tasks that require extensive knowledge of telecommunications. Our experiments on Llama-2-7b show that domain-adapted models can challenge the large generalist models. They also suggest that adaptation can be restricted to a unique instruction-tuning step, dicarding the need for any fine-tuning on raw texts beforehand.

Title: A Thorough Investigation into the Application of Deep CNN for Enhancing Natural Language Processing Capabilities

• Authors: Chang Weng, Scott Rood, Mehdi Ali Ramezani, Amir Aslani, Reza Zarrab, Wang Zwuo, Sanjeev Salimans, Tim Satheesh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15900
• Pdf URL: https://arxiv.org/pdf/2412.15900
• Copy Paste: [[2412.15900]] A Thorough Investigation into the Application of Deep CNN for Enhancing Natural Language Processing Capabilities(https://arxiv.org/abs/2412.15900)
• Keywords: generative, segmentation
• Abstract: Natural Language Processing (NLP) is widely used in fields like machine translation and sentiment analysis. However, traditional NLP models struggle with accuracy and efficiency. This paper introduces Deep Convolutional Neural Networks (DCNN) into NLP to address these issues. By integrating DCNN, machine learning (ML) algorithms, and generative adversarial networks (GAN), the study improves language understanding, reduces ambiguity, and enhances task performance. The high-performance NLP model shows a 10% improvement in segmentation accuracy and a 4% increase in recall rate compared to traditional models. This integrated approach excels in tasks such as word segmentation, part-of-speech tagging, machine translation, and text classification, offering better recognition accuracy and processing efficiency.

Title: Vulnerability Detection in Popular Programming Languages with Language Models

• Authors: Syafiq Al Atiiq, Kevin Dahlén, Christian Gehrmann
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.15905
• Pdf URL: https://arxiv.org/pdf/2412.15905
• Copy Paste: [[2412.15905]] Vulnerability Detection in Popular Programming Languages with Language Models(https://arxiv.org/abs/2412.15905)
• Keywords: security
• Abstract: Vulnerability detection is crucial for maintaining software security, and recent research has explored the use of Language Models (LMs) for this task. While LMs have shown promising results, their performance has been inconsistent across datasets, particularly when generalizing to unseen code. Moreover, most studies have focused on the C/C++ programming language, with limited attention given to other popular languages. This paper addresses this gap by investigating the effectiveness of LMs for vulnerability detection in JavaScript, Java, Python, PHP, and Go, in addition to C/C++ for comparison. We utilize the CVEFixes dataset to create a diverse collection of language-specific vulnerabilities and preprocess the data to ensure quality and integrity. We fine-tune and evaluate state-of-the-art LMs across the selected languages and find that the performance of vulnerability detection varies significantly. JavaScript exhibits the best performance, with considerably better and more practical detection capabilities compared to C/C++. We also examine the relationship between code complexity and detection performance across the six languages and find only a weak correlation between code complexity metrics and the models' F1 scores.

Title: Development of a Large-scale Dataset of Chest Computed Tomography Reports in Japanese and a High-performance Finding Classification Model

• Authors: Yosuke Yamagishi, Yuta Nakamura, Tomohiro Kikuchi, Yuki Sonoda, Hiroshi Hirakawa, Shintaro Kano, Satoshi Nakamura, Shouhei Hanaoka, Takeharu Yoshikawa, Osamu Abe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15907
• Pdf URL: https://arxiv.org/pdf/2412.15907
• Copy Paste: [[2412.15907]] Development of a Large-scale Dataset of Chest Computed Tomography Reports in Japanese and a High-performance Finding Classification Model(https://arxiv.org/abs/2412.15907)
• Keywords: robust, large language model
• Abstract: Background: Recent advances in large language models highlight the need for high-quality multilingual medical datasets. While Japan leads globally in CT scanner deployment and utilization, the lack of large-scale Japanese radiology datasets has hindered the development of specialized language models for medical imaging analysis. Objective: To develop a comprehensive Japanese CT report dataset through machine translation and establish a specialized language model for structured finding classification. Additionally, to create a rigorously validated evaluation dataset through expert radiologist review. Methods: We translated the CT-RATE dataset (24,283 CT reports from 21,304 patients) into Japanese using GPT-4o mini. The training dataset consisted of 22,778 machine-translated reports, while the validation dataset included 150 radiologist-revised reports. We developed CT-BERT-JPN based on "tohoku-nlp/bert-base-japanese-v3" architecture for extracting 18 structured findings from Japanese radiology reports. Results: Translation metrics showed strong performance with BLEU scores of 0.731 and 0.690, and ROUGE scores ranging from 0.770 to 0.876 for Findings and from 0.748 to 0.857 for Impression sections. CT-BERT-JPN demonstrated superior performance compared to GPT-4o in 11 out of 18 conditions, including lymphadenopathy (+14.2%), interlobular septal thickening (+10.9%), and atelectasis (+7.4%). The model maintained F1 scores exceeding 0.95 in 14 out of 18 conditions and achieved perfect scores in four conditions. Conclusions: Our study establishes a robust Japanese CT report dataset and demonstrates the effectiveness of a specialized language model for structured finding classification. The hybrid approach of machine translation and expert validation enables the creation of large-scale medical datasets while maintaining high quality.

Title: Watertox: The Art of Simplicity in Universal Attacks A Cross-Model Framework for Robust Adversarial Generation

• Authors: Zhenghao Gao, Shengjie Xu, Meixi Chen, Fangyao Zhao
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15924
• Pdf URL: https://arxiv.org/pdf/2412.15924
• Copy Paste: [[2412.15924]] Watertox: The Art of Simplicity in Universal Attacks A Cross-Model Framework for Robust Adversarial Generation(https://arxiv.org/abs/2412.15924)
• Keywords: security, attack, robust
• Abstract: Contemporary adversarial attack methods face significant limitations in cross-model transferability and practical applicability. We present Watertox, an elegant adversarial attack framework achieving remarkable effectiveness through architectural diversity and precision-controlled perturbations. Our two-stage Fast Gradient Sign Method combines uniform baseline perturbations ($\epsilon_1 = 0.1$) with targeted enhancements ($\epsilon_2 = 0.4$). The framework leverages an ensemble of complementary architectures, from VGG to ConvNeXt, synthesizing diverse perspectives through an innovative voting mechanism. Against state-of-the-art architectures, Watertox reduces model accuracy from 70.6% to 16.0%, with zero-shot attacks achieving up to 98.8% accuracy reduction against unseen architectures. These results establish Watertox as a significant advancement in adversarial methodologies, with promising applications in visual security systems and CAPTCHA generation.

Title: MiniGPT-Pancreas: Multimodal Large Language Model for Pancreas Cancer Classification and Detection

• Authors: Andrea Moglia, Elia Clement Nastasio, Luca Mainardi, Pietro Cerveri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15925
• Pdf URL: https://arxiv.org/pdf/2412.15925
• Copy Paste: [[2412.15925]] MiniGPT-Pancreas: Multimodal Large Language Model for Pancreas Cancer Classification and Detection(https://arxiv.org/abs/2412.15925)
• Keywords: large language model, segmentation
• Abstract: Problem: Pancreas radiological imaging is challenging due to the small size, blurred boundaries, and variability of shape and position of the organ among patients. Goal: In this work we present MiniGPT-Pancreas, a Multimodal Large Language Model (MLLM), as an interactive chatbot to support clinicians in pancreas cancer diagnosis by integrating visual and textual information. Methods: MiniGPT-v2, a general-purpose MLLM, was fine-tuned in a cascaded way for pancreas detection, tumor classification, and tumor detection with multimodal prompts combining questions and computed tomography scans from the National Institute of Health (NIH), and Medical Segmentation Decathlon (MSD) datasets. The AbdomenCT-1k dataset was used to detect the liver, spleen, kidney, and pancreas. Results: MiniGPT-Pancreas achieved an Intersection over Union (IoU) of 0.595 and 0.550 for the detection of pancreas on NIH and MSD datasets, respectively. For the pancreas cancer classification task on the MSD dataset, accuracy, precision, and recall were 0.876, 0.874, and 0.878, respectively. When evaluating MiniGPT-Pancreas on the AbdomenCT-1k dataset for multi-organ detection, the IoU was 0.8399 for the liver, 0.722 for the kidney, 0.705 for the spleen, and 0.497 for the pancreas. For the pancreas tumor detection task, the IoU score was 0.168 on the MSD dataset. Conclusions: MiniGPT-Pancreas represents a promising solution to support clinicians in the classification of pancreas images with pancreas tumors. Future research is needed to improve the score on the detection task, especially for pancreas tumors.

Title: Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation

• Authors: Gautier Evennou, Antoine Chaffin, Vivien Chappelier, Ewa Kijak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15939
• Pdf URL: https://arxiv.org/pdf/2412.15939
• Copy Paste: [[2412.15939]] Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation(https://arxiv.org/abs/2412.15939)
• Keywords: generative
• Abstract: The rise of the generative models quality during the past years enabled the generation of edited variations of images at an important scale. To counter the harmful effects of such technology, the Image Difference Captioning (IDC) task aims to describe the differences between two images. While this task is successfully handled for simple 3D rendered images, it struggles on real-world images. The reason is twofold: the training data-scarcity, and the difficulty to capture fine-grained differences between complex images. To address those issues, we propose in this paper a simple yet effective framework to both adapt existing image captioning models to the IDC task and augment IDC datasets. We introduce BLIP2IDC, an adaptation of BLIP2 to the IDC task at low computational cost, and show it outperforms two-streams approaches by a significant margin on real-world IDC datasets. We also propose to use synthetic augmentation to improve the performance of IDC models in an agnostic fashion. We show that our synthetic augmentation strategy provides high quality data, leading to a challenging new dataset well-suited for IDC named Syned1.

Title: From General to Specific: Tailoring Large Language Models for Personalized Healthcare

• Authors: Ruize Shi, Hong Huang, Wei Zhou, Kehan Yin, Kai Zhao, Yun Zhao
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.15957
• Pdf URL: https://arxiv.org/pdf/2412.15957
• Copy Paste: [[2412.15957]] From General to Specific: Tailoring Large Language Models for Personalized Healthcare(https://arxiv.org/abs/2412.15957)
• Keywords: large language model
• Abstract: The rapid development of large language models (LLMs) has transformed many industries, including healthcare. However, previous medical LLMs have largely focused on leveraging general medical knowledge to provide responses, without accounting for patient variability and lacking true personalization at the individual level. To address this, we propose a novel method called personalized medical language model (PMLM), which explores and optimizes personalized LLMs through recommendation systems and reinforcement learning (RL). Specifically, by utilizing self-informed and peer-informed personalization, PMLM captures changes in behaviors and preferences to design initial personalized prompts tailored to individual needs. We further refine these initial personalized prompts through RL, ultimately enhancing the precision of LLM guidance. Notably, the personalized prompt are hard prompt, which grants PMLM high adaptability and reusability, allowing it to directly leverage high-quality proprietary LLMs. We evaluate PMLM using real-world obstetrics and gynecology data, and the experimental results demonstrate that PMLM achieves personalized responses, and it provides more refined and individualized services, offering a potential way for personalized medical LLMs.

Title: BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models

• Authors: Patrick Haller, Jonas Golde, Alan Akbik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.15978
• Pdf URL: https://arxiv.org/pdf/2412.15978
• Copy Paste: [[2412.15978]] BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models(https://arxiv.org/abs/2412.15978)
• Keywords: transformer
• Abstract: This paper explores the potential of recurrent neural networks (RNNs) and other subquadratic architectures as competitive alternatives to transformer-based models in low-resource language modeling scenarios. We utilize HGRN2 (Qin et al., 2024), a recently proposed RNN-based architecture, and comparatively evaluate its effectiveness against transformer-based baselines and other subquadratic architectures (LSTM, xLSTM, Mamba). Our experimental results show that BABYHGRN, our HGRN2 language model, outperforms transformer-based models in both the 10M and 100M word tracks of the challenge, as measured by their performance on the BLiMP, EWoK, GLUE and BEAR benchmarks. Further, we show the positive impact of knowledge distillation. Our findings challenge the prevailing focus on transformer architectures and indicate the viability of RNN-based models, particularly in resource-constrained environments.

Title: Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling

• Authors: Maximillian Chen, Ruoxi Sun, Sercan Ö. Arık
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.15995
• Pdf URL: https://arxiv.org/pdf/2412.15995
• Copy Paste: [[2412.15995]] Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling(https://arxiv.org/abs/2412.15995)
• Keywords: robust
• Abstract: Conversational assistants are increasingly popular across diverse real-world applications, highlighting the need for advanced multimodal speech modeling. Speech, as a natural mode of communication, encodes rich user-specific characteristics such as speaking rate and pitch, making it critical for effective interaction. Our work introduces a data-centric customization approach for efficiently enhancing multimodal understanding in conversational speech modeling. Central to our contributions is a novel multi-task learning paradigm that involves designing auxiliary tasks to utilize a small amount of speech data. Our approach achieves state-of-the-art performance on the Spoken-SQuAD benchmark, using only 10% of the training data with open-weight models, establishing a robust and efficient framework for audio-centric conversational modeling. We also introduce ASK-QA, the first dataset for multi-turn spoken dialogue with ambiguous user requests and dynamic evaluation inputs. Code and data forthcoming.

Title: CNN-LSTM Hybrid Deep Learning Model for Remaining Useful Life Estimation

• Authors: Muthukumar G, Jyosna Philip
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15998
• Pdf URL: https://arxiv.org/pdf/2412.15998
• Copy Paste: [[2412.15998]] CNN-LSTM Hybrid Deep Learning Model for Remaining Useful Life Estimation(https://arxiv.org/abs/2412.15998)
• Keywords: robust
• Abstract: Remaining Useful Life (RUL) of a component or a system is defined as the length from the current time to the end of the useful life. Accurate RUL estimation plays a crucial role in Predictive Maintenance applications. Traditional regression methods, both linear and non-linear, have struggled to achieve high accuracy in this domain. While Convolutional Neural Networks (CNNs) have shown improved accuracy, they often overlook the sequential nature of the data, relying instead on features derived from sliding windows. Since RUL prediction inherently involves multivariate time series analysis, robust sequence learning is essential. In this work, we propose a hybrid approach combining Convolutional Neural Networks with Long Short-Term Memory (LSTM) networks for RUL estimation. Although CNN-based LSTM models have been applied to sequence prediction tasks in financial forecasting, this is the first attempt to adopt this approach for RUL estimation in prognostics. In this approach, CNN is first employed to efficiently extract features from the data, followed by LSTM, which uses these extracted features to predict RUL. This method effectively leverages sensor sequence information, uncovering hidden patterns within the data, even under multiple operating conditions and fault scenarios. Our results demonstrate that the hybrid CNN-LSTM model achieves the highest accuracy, offering a superior score compared to the other methods.

Title: Choose Your Explanation: A Comparison of SHAP and GradCAM in Human Activity Recognition

• Authors: Felix Tempel, Daniel Groos, Espen Alexander F. Ihlen, Lars Adde, Inga Strümke
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16003
• Pdf URL: https://arxiv.org/pdf/2412.16003
• Copy Paste: [[2412.16003]] Choose Your Explanation: A Comparison of SHAP and GradCAM in Human Activity Recognition(https://arxiv.org/abs/2412.16003)
• Keywords: interpretability, explainability
• Abstract: Explaining machine learning (ML) models using eXplainable AI (XAI) techniques has become essential to make them more transparent and trustworthy. This is especially important in high-stakes domains like healthcare, where understanding model decisions is critical to ensure ethical, sound, and trustworthy outcome predictions. However, users are often confused about which explanability method to choose for their specific use case. We present a comparative analysis of widely used explainability methods, Shapley Additive Explanations (SHAP) and Gradient-weighted Class Activation Mapping (GradCAM), within the domain of human activity recognition (HAR) utilizing graph convolutional networks (GCNs). By evaluating these methods on skeleton-based data from two real-world datasets, including a healthcare-critical cerebral palsy (CP) case, this study provides vital insights into both approaches' strengths, limitations, and differences, offering a roadmap for selecting the most appropriate explanation method based on specific models and applications. We quantitatively and quantitatively compare these methods, focusing on feature importance ranking, interpretability, and model sensitivity through perturbation experiments. While SHAP provides detailed input feature attribution, GradCAM delivers faster, spatially oriented explanations, making both methods complementary depending on the application's requirements. Given the importance of XAI in enhancing trust and transparency in ML models, particularly in sensitive environments like healthcare, our research demonstrates how SHAP and GradCAM could complement each other to provide more interpretable and actionable model explanations.

Title: Detection of Aerial Spoofing Attacks to LEO Satellite Systems via Deep Learning

• Authors: Jos Wigchert, Savio Sciancalepore, Gabriele Oligeri
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16008
• Pdf URL: https://arxiv.org/pdf/2412.16008
• Copy Paste: [[2412.16008]] Detection of Aerial Spoofing Attacks to LEO Satellite Systems via Deep Learning(https://arxiv.org/abs/2412.16008)
• Keywords: security, attack, robust
• Abstract: Detecting spoofing attacks to Low-Earth-Orbit (LEO) satellite systems is a cornerstone to assessing the authenticity of the received information and guaranteeing robust service delivery in several application domains. The solutions available today for spoofing detection either rely on additional communication systems, receivers, and antennas, or require mobile deployments. Detection systems working at the Physical (PHY) layer of the satellite communication link also require time-consuming and energy-hungry training processes on all satellites of the constellation, and rely on the availability of spoofed data, which are often challenging to collect. Moreover, none of such contributions investigate the feasibility of aerial spoofing attacks launched via drones operating at various altitudes. In this paper, we propose a new spoofing detection technique for LEO satellite constellation systems, applying anomaly detection on the received PHY signal via autoencoders. We validate our solution through an extensive measurement campaign involving the deployment of an actual spoofer (Software-Defined Radio) installed on a drone and injecting rogue IRIDIUM messages while flying at different altitudes with various movement patterns. Our results demonstrate that the proposed technique can reliably detect LEO spoofing attacks launched at different altitudes, while state-of-the-art competing approaches simply fail. We also release the collected data as open source, fostering further research on satellite security.

Title: The Only Way is Ethics: A Guide to Ethical Research with Large Language Models

• Authors: Eddie L. Ungless, Nikolas Vitsakis, Zeerak Talat, James Garforth, Björn Ross, Arno Onken, Atoosa Kasirzadeh, Alexandra Birch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16022
• Pdf URL: https://arxiv.org/pdf/2412.16022
• Copy Paste: [[2412.16022]] The Only Way is Ethics: A Guide to Ethical Research with Large Language Models(https://arxiv.org/abs/2412.16022)
• Keywords: security, privacy, large language model
• Abstract: There is a significant body of work looking at the ethical considerations of large language models (LLMs): critiquing tools to measure performance and harms; proposing toolkits to aid in ideation; discussing the risks to workers; considering legislation around privacy and security etc. As yet there is no work that integrates these resources into a single practical guide that focuses on LLMs; we attempt this ambitious goal. We introduce 'LLM Ethics Whitepaper', which we provide as an open and living resource for NLP practitioners, and those tasked with evaluating the ethical implications of others' work. Our goal is to translate ethics literature into concrete recommendations and provocations for thinking with clear first steps, aimed at computer scientists. 'LLM Ethics Whitepaper' distils a thorough literature review into clear Do's and Don'ts, which we present also in this paper. We likewise identify useful toolkits to support ethical work. We refer the interested reader to the full LLM Ethics Whitepaper, which provides a succinct discussion of ethical considerations at each stage in a project lifecycle, as well as citations for the hundreds of papers from which we drew our recommendations. The present paper can be thought of as a pocket guide to conducting ethical research with LLMs.

Title: CoCoGaussian: Leveraging Circle of Confusion for Gaussian Splatting from Defocused Images

• Authors: Jungho Lee, Suhwan Cho, Taeoh Kim, Ho-Deok Jang, Minhyeok Lee, Geonho Cha, Dongyoon Wee, Dogyoon Lee, Sangyoun Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16028
• Pdf URL: https://arxiv.org/pdf/2412.16028
• Copy Paste: [[2412.16028]] CoCoGaussian: Leveraging Circle of Confusion for Gaussian Splatting from Defocused Images(https://arxiv.org/abs/2412.16028)
• Keywords: robust
• Abstract: 3D Gaussian Splatting (3DGS) has attracted significant attention for its high-quality novel view rendering, inspiring research to address real-world challenges. While conventional methods depend on sharp images for accurate scene reconstruction, real-world scenarios are often affected by defocus blur due to finite depth of field, making it essential to account for realistic 3D scene representation. In this study, we propose CoCoGaussian, a Circle of Confusion-aware Gaussian Splatting that enables precise 3D scene representation using only defocused images. CoCoGaussian addresses the challenge of defocus blur by modeling the Circle of Confusion (CoC) through a physically grounded approach based on the principles of photographic defocus. Exploiting 3D Gaussians, we compute the CoC diameter from depth and learnable aperture information, generating multiple Gaussians to precisely capture the CoC shape. Furthermore, we introduce a learnable scaling factor to enhance robustness and provide more flexibility in handling unreliable depth in scenes with reflective or refractive surfaces. Experiments on both synthetic and real-world datasets demonstrate that CoCoGaussian achieves state-of-the-art performance across multiple benchmarks.

Title: SafeCFG: Redirecting Harmful Classifier-Free Guidance for Safe Generation

• Authors: Jiadong Pan, Hongcheng Gao, Liang Li, Zheng-Jun Zha, Qingming Huang, Jiebo Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16039
• Pdf URL: https://arxiv.org/pdf/2412.16039
• Copy Paste: [[2412.16039]] SafeCFG: Redirecting Harmful Classifier-Free Guidance for Safe Generation(https://arxiv.org/abs/2412.16039)
• Keywords: diffusion
• Abstract: Diffusion models (DMs) have demonstrated exceptional performance in text-to-image (T2I) tasks, leading to their widespread use. With the introduction of classifier-free guidance (CFG), the quality of images generated by DMs is improved. However, DMs can generate more harmful images by maliciously guiding the image generation process through CFG. Some safe guidance methods aim to mitigate the risk of generating harmful images but often reduce the quality of clean image generation. To address this issue, we introduce the Harmful Guidance Redirector (HGR), which redirects harmful CFG direction while preserving clean CFG direction during image generation, transforming CFG into SafeCFG and achieving high safety and quality generation. We train HGR to redirect multiple harmful CFG directions simultaneously, demonstrating its ability to eliminate various harmful elements while preserving high-quality generation. Additionally, we find that HGR can detect image harmfulness, allowing for unsupervised fine-tuning of safe diffusion models without pre-defined clean or harmful labels. Experimental results show that by incorporating HGR, images generated by diffusion models achieve both high quality and strong safety, and safe DMs trained through unsupervised methods according to the harmfulness detected by HGR also exhibit good safety performance. The codes will be publicly available.

Title: Segmentation of arbitrary features in very high resolution remote sensing imagery

• Authors: Henry Cording, Yves Plancherel, Pablo Brito-Parada
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16046
• Pdf URL: https://arxiv.org/pdf/2412.16046
• Copy Paste: [[2412.16046]] Segmentation of arbitrary features in very high resolution remote sensing imagery(https://arxiv.org/abs/2412.16046)
• Keywords: segmentation
• Abstract: Very high resolution (VHR) mapping through remote sensing (RS) imagery presents a new opportunity to inform decision-making and sustainable practices in countless domains. Efficient processing of big VHR data requires automated tools applicable to numerous geographic regions and features. Contemporary RS studies address this challenge by employing deep learning (DL) models for specific datasets or features, which limits their applicability across contexts. The present research aims to overcome this limitation by introducing EcoMapper, a scalable solution to segment arbitrary features in VHR RS imagery. EcoMapper fully automates processing of geospatial data, DL model training, and inference. Models trained with EcoMapper successfully segmented two distinct features in a real-world UAV dataset, achieving scores competitive with prior studies which employed context-specific models. To evaluate EcoMapper, many additional models were trained on permutations of principal field survey characteristics (FSCs). A relationship was discovered allowing derivation of optimal ground sampling distance from feature size, termed Cording Index (CI). A comprehensive methodology for field surveys was developed to ensure DL methods can be applied effectively to collected data. The EcoMapper code accompanying this work is available at this https URL .

Title: Label-Efficient Data Augmentation with Video Diffusion Models for Guidewire Segmentation in Cardiac Fluoroscopy

• Authors: Shaoyan Pan, Yikang Liu, Lin Zhao, Eric Z. Chen, Xiao Chen, Terrence Chen, Shanhui Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16050
• Pdf URL: https://arxiv.org/pdf/2412.16050
• Copy Paste: [[2412.16050]] Label-Efficient Data Augmentation with Video Diffusion Models for Guidewire Segmentation in Cardiac Fluoroscopy(https://arxiv.org/abs/2412.16050)
• Keywords: robust, diffusion, segmentation
• Abstract: The accurate segmentation of guidewires in interventional cardiac fluoroscopy videos is crucial for computer-aided navigation tasks. Although deep learning methods have demonstrated high accuracy and robustness in wire segmentation, they require substantial annotated datasets for generalizability, underscoring the need for extensive labeled data to enhance model performance. To address this challenge, we propose the Segmentation-guided Frame-consistency Video Diffusion Model (SF-VD) to generate large collections of labeled fluoroscopy videos, augmenting the training data for wire segmentation networks. SF-VD leverages videos with limited annotations by independently modeling scene distribution and motion distribution. It first samples the scene distribution by generating 2D fluoroscopy images with wires positioned according to a specified input mask, and then samples the motion distribution by progressively generating subsequent frames, ensuring frame-to-frame coherence through a frame-consistency strategy. A segmentation-guided mechanism further refines the process by adjusting wire contrast, ensuring a diverse range of visibility in the synthesized image. Evaluation on a fluoroscopy dataset confirms the superior quality of the generated videos and shows significant improvements in guidewire segmentation.

Title: VaulTor: Putting the TEE in Tor

• Authors: Humza Ikram, Rumaisa Habib, Muaz Ali, Zartash Afzal Uzmi
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.16064
• Pdf URL: https://arxiv.org/pdf/2412.16064
• Copy Paste: [[2412.16064]] VaulTor: Putting the TEE in Tor(https://arxiv.org/abs/2412.16064)
• Keywords: security, attack
• Abstract: Online services that desire to operate anonymously routinely host themselves as 'Hidden Services' in the Tor network. However, these services are frequently threatened by deanonymization attacks, whereby their IP address and location may be inferred by the authorities. We present VaulTor, a novel architecture for the Tor network to ensure an extra layer of security for the Hidden Services against deanonymization attacks. In this new architecture, a volunteer (vault) is incentivized to host the web application content on behalf of the Hidden Service. The vault runs the hosted application in a Trusted Execution Environment (TEE) and becomes the point of contact for interested clients. This setup can substantially reduce the uptime requirement of the original Hidden Service provider and hence significantly decrease the chance of deanonymization attacks against them. We also show that the VaulTor architecture does not cause any noticeable performance degradation in accessing the hosted content (the performance degradation ranges from 2.6-5.5%).

Title: SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data

• Authors: Xinwei Ju, Rema Daher, Razvan Caramalau, Baoru Huang, Danail Stoyanov, Francisco Vasconcelos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16078
• Pdf URL: https://arxiv.org/pdf/2412.16078
• Copy Paste: [[2412.16078]] SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data(https://arxiv.org/abs/2412.16078)
• Keywords: segmentation
• Abstract: Colorectal cancer (CRC) remains a leading cause of cancer-related deaths worldwide, with polyp removal being an effective early screening method. However, navigating the colon for thorough polyp detection poses significant challenges. To advance camera navigation in colonoscopy, we propose the Semantic Segmentation for Tools and Fold Edges in Colonoscopy (SegCol) Challenge. This challenge introduces a dataset from the EndoMapper repository, featuring manually annotated, pixel-level semantic labels for colon folds and endoscopic tools across selected frames from 96 colonoscopy videos. By providing fold edges as anatomical landmarks and depth discontinuity information from both fold and tool labels, the dataset is aimed to improve depth perception and localization methods. Hosted as part of the Endovis Challenge at MICCAI 2024, SegCol aims to drive innovation in colonoscopy navigation systems. Details are available at this https URL, and code resources at this https URL .

Title: Fair Distributed Machine Learning with Imbalanced Data as a Stackelberg Evolutionary Game

• Authors: Sebastian Niehaus, Ingo Roeder, Nico Scherf
• Subjects: cs.LG, cs.CV, cs.GT, cs.NE
• Abstract URL: https://arxiv.org/abs/2412.16079
• Pdf URL: https://arxiv.org/pdf/2412.16079
• Copy Paste: [[2412.16079]] Fair Distributed Machine Learning with Imbalanced Data as a Stackelberg Evolutionary Game(https://arxiv.org/abs/2412.16079)
• Keywords: privacy, fair
• Abstract: Decentralised learning enables the training of deep learning algorithms without centralising data sets, resulting in benefits such as improved data privacy, operational efficiency and the fostering of data ownership policies. However, significant data imbalances pose a challenge in this framework. Participants with smaller datasets in distributed learning environments often achieve poorer results than participants with larger datasets. Data imbalances are particularly pronounced in medical fields and are caused by different patient populations, technological inequalities and divergent data collection practices. In this paper, we consider distributed learning as an Stackelberg evolutionary game. We present two algorithms for setting the weights of each node's contribution to the global model in each training round: the Deterministic Stackelberg Weighting Model (DSWM) and the Adaptive Stackelberg Weighting Model (ASWM). We use three medical datasets to highlight the impact of dynamic weighting on underrepresented nodes in distributed learning. Our results show that the ASWM significantly favours underrepresented nodes by improving their performance by 2.713% in AUC. Meanwhile, nodes with larger datasets experience only a modest average performance decrease of 0.441%.

Title: Differentially Private Federated Learning of Diffusion Models for Synthetic Tabular Data Generation

• Authors: Timur Sattarov, Marco Schreyer, Damian Borth
• Subjects: cs.LG, q-fin.ST
• Abstract URL: https://arxiv.org/abs/2412.16083
• Pdf URL: https://arxiv.org/pdf/2412.16083
• Copy Paste: [[2412.16083]] Differentially Private Federated Learning of Diffusion Models for Synthetic Tabular Data Generation(https://arxiv.org/abs/2412.16083)
• Keywords: secure, privacy, robust, federate, diffusion
• Abstract: The increasing demand for privacy-preserving data analytics in finance necessitates solutions for synthetic data generation that rigorously uphold privacy standards. We introduce DP-Fed-FinDiff framework, a novel integration of Differential Privacy, Federated Learning and Denoising Diffusion Probabilistic Models designed to generate high-fidelity synthetic tabular data. This framework ensures compliance with stringent privacy regulations while maintaining data utility. We demonstrate the effectiveness of DP-Fed-FinDiff on multiple real-world financial datasets, achieving significant improvements in privacy guarantees without compromising data quality. Our empirical evaluations reveal the optimal trade-offs between privacy budgets, client configurations, and federated optimization strategies. The results affirm the potential of DP-Fed-FinDiff to enable secure data sharing and robust analytics in highly regulated domains, paving the way for further advances in federated learning and privacy-preserving data synthesis.

Title: Explainable AI for Multivariate Time Series Pattern Exploration: Latent Space Visual Analytics with Time Fusion Transformer and Variational Autoencoders in Power Grid Event Diagnosis

• Authors: Haowen Xu, Ali Boyaci, Jianming Lian, Aaron Wilson
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16098
• Pdf URL: https://arxiv.org/pdf/2412.16098
• Copy Paste: [[2412.16098]] Explainable AI for Multivariate Time Series Pattern Exploration: Latent Space Visual Analytics with Time Fusion Transformer and Variational Autoencoders in Power Grid Event Diagnosis(https://arxiv.org/abs/2412.16098)
• Keywords: interpretability, transformer, generative
• Abstract: Detecting and analyzing complex patterns in multivariate time-series data is crucial for decision-making in urban and environmental system operations. However, challenges arise from the high dimensionality, intricate complexity, and interconnected nature of complex patterns, which hinder the understanding of their underlying physical processes. Existing AI methods often face limitations in interpretability, computational efficiency, and scalability, reducing their applicability in real-world scenarios. This paper proposes a novel visual analytics framework that integrates two generative AI models, Time Fusion Transformer (TFT) and Variational Autoencoders (VAEs), to reduce complex patterns into lower-dimensional latent spaces and visualize them in 2D using dimensionality reduction techniques such as PCA, t-SNE, and UMAP with DBSCAN. These visualizations, presented through coordinated and interactive views and tailored glyphs, enable intuitive exploration of complex multivariate temporal patterns, identifying patterns' similarities and uncover their potential correlations for a better interpretability of the AI outputs. The framework is demonstrated through a case study on power grid signal data, where it identifies multi-label grid event signatures, including faults and anomalies with diverse root causes. Additionally, novel metrics and visualizations are introduced to validate the models and evaluate the performance, efficiency, and consistency of latent maps generated by TFT and VAE under different configurations. These analyses provide actionable insights for model parameter tuning and reliability improvements. Comparative results highlight that TFT achieves shorter run times and superior scalability to diverse time-series data shapes compared to VAE. This work advances fault diagnosis in multivariate time series, fostering explainable AI to support critical system operations.

Title: Logical Consistency of Large Language Models in Fact-checking

• Authors: Bishwamittra Ghosh, Sarah Hasan, Naheed Anjum Arafat, Arijit Khan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16100
• Pdf URL: https://arxiv.org/pdf/2412.16100
• Copy Paste: [[2412.16100]] Logical Consistency of Large Language Models in Fact-checking(https://arxiv.org/abs/2412.16100)
• Keywords: large language model
• Abstract: In recent years, large language models (LLMs) have demonstrated significant success in performing varied natural language tasks such as language translation, question-answering, summarizing, fact-checking, etc. Despite LLMs' impressive ability to generate human-like texts, LLMs are infamous for their inconsistent responses -- a meaning-preserving change in the input query results in an inconsistent response and attributes to vulnerabilities of LLMs such as hallucination, jailbreaking, etc. Consequently, existing research focuses on simple paraphrasing-based consistency assessment of LLMs, and ignores complex queries that necessitates an even better understanding of logical reasoning by an LLM. Our work therefore addresses the logical inconsistency of LLMs under complex logical queries with primitive logical operators, e.g., negation, conjunction, and disjunction. As a test bed, we consider retrieval-augmented LLMs on a fact-checking task involving propositional logic queries from real-world knowledge graphs (KGs). Our contributions are three-fold. Benchmark: We introduce three logical fact-checking datasets over KGs for community development towards logically consistent LLMs. Assessment: We propose consistency measures of LLMs on propositional logic queries as input and demonstrate that existing LLMs lack logical consistency, specially on complex queries. Improvement: We employ supervised fine-tuning to improve the logical consistency of LLMs on the complex fact-checking task with KG contexts.

Title: Demystifying the Potential of ChatGPT-4 Vision for Construction Progress Monitoring

• Authors: Ahmet Bahaddin Ersoz
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16108
• Pdf URL: https://arxiv.org/pdf/2412.16108
• Copy Paste: [[2412.16108]] Demystifying the Potential of ChatGPT-4 Vision for Construction Progress Monitoring(https://arxiv.org/abs/2412.16108)
• Keywords: segmentation
• Abstract: The integration of Large Vision-Language Models (LVLMs) such as OpenAI's GPT-4 Vision into various sectors has marked a significant evolution in the field of artificial intelligence, particularly in the analysis and interpretation of visual data. This paper explores the practical application of GPT-4 Vision in the construction industry, focusing on its capabilities in monitoring and tracking the progress of construction projects. Utilizing high-resolution aerial imagery of construction sites, the study examines how GPT-4 Vision performs detailed scene analysis and tracks developmental changes over time. The findings demonstrate that while GPT-4 Vision is proficient in identifying construction stages, materials, and machinery, it faces challenges with precise object localization and segmentation. Despite these limitations, the potential for future advancements in this technology is considerable. This research not only highlights the current state and opportunities of using LVLMs in construction but also discusses future directions for enhancing the model's utility through domain-specific training and integration with other computer vision techniques and digital twins.

Title: CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up

• Authors: Songhua Liu, Zhenxiong Tan, Xinchao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16112
• Pdf URL: https://arxiv.org/pdf/2412.16112
• Copy Paste: [[2412.16112]] CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up(https://arxiv.org/abs/2412.16112)
• Keywords: diffusion, transformer
• Abstract: Diffusion Transformers (DiT) have become a leading architecture in image generation. However, the quadratic complexity of attention mechanisms, which are responsible for modeling token-wise relationships, results in significant latency when generating high-resolution images. To address this issue, we aim at a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs to linear. We begin our exploration with a comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. Based on these insights, we introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token, and thus achieves linear complexity. Our experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity, yielding results comparable to the teacher model. Simultaneously, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images. Furthermore, we investigate favorable properties in the distilled attention layers, such as zero-shot generalization cross various models and plugins, and improved support for multi-GPU parallel inference. Models and codes are available here: this https URL.

Title: PruneVid: Visual Token Pruning for Efficient Video Large Language Models

• Authors: Xiaohu Huang, Hao Zhou, Kai Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16117
• Pdf URL: https://arxiv.org/pdf/2412.16117
• Copy Paste: [[2412.16117]] PruneVid: Visual Token Pruning for Efficient Video Large Language Models(https://arxiv.org/abs/2412.16117)
• Keywords: large language model
• Abstract: In this paper, we introduce PruneVid, a visual token pruning method designed to enhance the efficiency of multi-modal video understanding. Large Language Models (LLMs) have shown promising performance in video tasks due to their extended capabilities in comprehending visual modalities. However, the substantial redundancy in video data presents significant computational challenges for LLMs. To address this issue, we introduce a training-free method that 1) minimizes video redundancy by merging spatial-temporal tokens, and 2) leverages LLMs' reasoning capabilities to selectively prune visual features relevant to question tokens, enhancing model efficiency. We validate our method across multiple video benchmarks, which demonstrate that PruneVid can prune over 80% of tokens while maintaining competitive performance combined with different model networks. This highlights its superior effectiveness and efficiency compared to existing pruning methods. Code: this https URL.

Title: Deciphering the Underserved: Benchmarking LLM OCR for Low-Resource Scripts

• Authors: Muhammad Abdullah Sohail, Salaar Masood, Hamza Iqbal
• Subjects: cs.LG, cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.16119
• Pdf URL: https://arxiv.org/pdf/2412.16119
• Copy Paste: [[2412.16119]] Deciphering the Underserved: Benchmarking LLM OCR for Low-Resource Scripts(https://arxiv.org/abs/2412.16119)
• Keywords: robust, large language model
• Abstract: This study investigates the potential of Large Language Models (LLMs), particularly GPT-4o, for Optical Character Recognition (OCR) in low-resource scripts such as Urdu, Albanian, and Tajik, with English serving as a benchmark. Using a meticulously curated dataset of 2,520 images incorporating controlled variations in text length, font size, background color, and blur, the research simulates diverse real-world challenges. Results emphasize the limitations of zero-shot LLM-based OCR, particularly for linguistically complex scripts, highlighting the need for annotated datasets and fine-tuned models. This work underscores the urgency of addressing accessibility gaps in text digitization, paving the way for inclusive and robust OCR solutions for underserved languages.

Title: PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics

• Authors: Daniil Larionov, Steffen Eger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16120
• Pdf URL: https://arxiv.org/pdf/2412.16120
• Copy Paste: [[2412.16120]] PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics(https://arxiv.org/abs/2412.16120)
• Keywords: large language model
• Abstract: Evaluating the quality of machine-generated natural language content is a challenging task in Natural Language Processing (NLP). Recently, large language models (LLMs) like GPT-4 have been employed for this purpose, but they are computationally expensive due to the extensive token usage required by complex evaluation prompts. In this paper, we propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation. Our method involves a two-stage fine-tuning process: supervised fine-tuning followed by preference optimization to refine the model's outputs based on human preferences. We focus on Machine Translation (MT) evaluation and utilize the GEMBA-MQM metric as a starting point. Our results show a $2.37\times$ reduction in token usage without any loss in evaluation quality. This work makes state-of-the-art LLM-based metrics like GEMBA-MQM more cost-effective and efficient, enhancing their accessibility for broader use.

Title: LEDA: Log-Euclidean Diffeomorphic Autoencoder for Efficient Statistical Analysis of Diffeomorphism

• Authors: Krithika Iyer, Shireen Elhabian, Sarang Joshi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16129
• Pdf URL: https://arxiv.org/pdf/2412.16129
• Copy Paste: [[2412.16129]] LEDA: Log-Euclidean Diffeomorphic Autoencoder for Efficient Statistical Analysis of Diffeomorphism(https://arxiv.org/abs/2412.16129)
• Keywords: robust
• Abstract: Image registration is a core task in computational anatomy that establishes correspondences between images. Invertible deformable registration, which computes a deformation field and handles complex, non-linear transformation, is essential for tracking anatomical variations, especially in neuroimaging applications where inter-subject differences and longitudinal changes are key. Analyzing the deformation fields is challenging due to their non-linearity, limiting statistical analysis. However, traditional approaches for analyzing deformation fields are computationally expensive, sensitive to initialization, and prone to numerical errors, especially when the deformation is far from the identity. To address these limitations, we propose the Log-Euclidean Diffeomorphic Autoencoder (LEDA), an innovative framework designed to compute the principal logarithm of deformation fields by efficiently predicting consecutive square roots. LEDA operates within a linearized latent space that adheres to the diffeomorphisms group action laws, enhancing our model's robustness and applicability. We also introduce a loss function to enforce inverse consistency, ensuring accurate latent representations of deformation fields. Extensive experiments with the OASIS-1 dataset demonstrate the effectiveness of LEDA in accurately modeling and analyzing complex non-linear deformations while maintaining inverse consistency. Additionally, we evaluate its ability to capture and incorporate clinical variables, enhancing its relevance for clinical applications.

Title: Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscation

• Authors: Seyedreza Mohseni, Seyedali Mohammadi, Deepa Tilwani, Yash Saxena, Gerald Ndwula, Sriram Vema, Edward Raff, Manas Gaur
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16135
• Pdf URL: https://arxiv.org/pdf/2412.16135
• Copy Paste: [[2412.16135]] Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscation(https://arxiv.org/abs/2412.16135)
• Keywords: attack, large language model
• Abstract: Malware authors often employ code obfuscations to make their malware harder to detect. Existing tools for generating obfuscated code often require access to the original source code (e.g., C++ or Java), and adding new obfuscations is a non-trivial, labor-intensive process. In this study, we ask the following question: Can Large Language Models (LLMs) potentially generate a new obfuscated assembly code? If so, this poses a risk to anti-virus engines and potentially increases the flexibility of attackers to create new obfuscation patterns. We answer this in the affirmative by developing the MetamorphASM benchmark comprising MetamorphASM Dataset (MAD) along with three code obfuscation techniques: dead code, register substitution, and control flow change. The MetamorphASM systematically evaluates the ability of LLMs to generate and analyze obfuscated code using MAD, which contains 328,200 obfuscated assembly code samples. We release this dataset and analyze the success rate of various LLMs (e.g., GPT-3.5/4, GPT-4o-mini, Starcoder, CodeGemma, CodeLlama, CodeT5, and LLaMA 3.1) in generating obfuscated assembly code. The evaluation was performed using established information-theoretic metrics and manual human review to ensure correctness and provide the foundation for researchers to study and develop remediations to this risk. The source code can be found at the following GitHub link: this https URL.

Title: Camera-Based Localization and Enhanced Normalized Mutual Information

• Authors: Vishnu Teja Kunde, Jean-Francois Chamberland, Siddharth Agarwal
• Subjects: cs.CV, eess.SP, stat.AP
• Abstract URL: https://arxiv.org/abs/2412.16137
• Pdf URL: https://arxiv.org/pdf/2412.16137
• Copy Paste: [[2412.16137]] Camera-Based Localization and Enhanced Normalized Mutual Information(https://arxiv.org/abs/2412.16137)
• Keywords: robust
• Abstract: Robust and fine localization algorithms are crucial for autonomous driving. For the production of such vehicles as a commodity, affordable sensing solutions and reliable localization algorithms must be designed. This work considers scenarios where the sensor data comes from images captured by an inexpensive camera mounted on the vehicle and where the vehicle contains a fine global map. Such localization algorithms typically involve finding the section in the global map that best matches the captured image. In harsh environments, both the global map and the captured image can be noisy. Because of physical constraints on camera placement, the image captured by the camera can be viewed as a noisy perspective transformed version of the road in the global map. Thus, an optimal algorithm should take into account the unequal noise power in various regions of the captured image, and the intrinsic uncertainty in the global map due to environmental variations. This article briefly reviews two matching methods: (i) standard inner product (SIP) and (ii) normalized mutual information (NMI). It then proposes novel and principled modifications to improve the performance of these algorithms significantly in noisy environments. These enhancements are inspired by the physical constraints associated with autonomous vehicles. They are grounded in statistical signal processing and, in some context, are provably better. Numerical simulations demonstrate the effectiveness of such modifications.

Title: FedGAT: A Privacy-Preserving Federated Approximation Algorithm for Graph Attention Networks

• Authors: Siddharth Ambekar, Yuhang Yao, Ryan Li, Carlee Joe-Wong
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.16144
• Pdf URL: https://arxiv.org/pdf/2412.16144
• Copy Paste: [[2412.16144]] FedGAT: A Privacy-Preserving Federated Approximation Algorithm for Graph Attention Networks(https://arxiv.org/abs/2412.16144)
• Keywords: privacy, robust, federate
• Abstract: Federated training methods have gained popularity for graph learning with applications including friendship graphs of social media sites and customer-merchant interaction graphs of huge online marketplaces. However, privacy regulations often require locally generated data to be stored on local clients. The graph is then naturally partitioned across clients, with no client permitted access to information stored on another. Cross-client edges arise naturally in such cases and present an interesting challenge to federated training methods, as training a graph model at one client requires feature information of nodes on the other end of cross-client edges. Attempting to retain such edges often incurs significant communication overhead, and dropping them altogether reduces model performance. In simpler models such as Graph Convolutional Networks, this can be fixed by communicating a limited amount of feature information across clients before training, but GATs (Graph Attention Networks) require additional information that cannot be pre-communicated, as it changes from training round to round. We introduce the Federated Graph Attention Network (FedGAT) algorithm for semi-supervised node classification, which approximates the behavior of GATs with provable bounds on the approximation error. FedGAT requires only one pre-training communication round, significantly reducing the communication overhead for federated GAT training. We then analyze the error in the approximation and examine the communication overhead and computational complexity of the algorithm. Experiments show that FedGAT achieves nearly the same accuracy as a GAT model in a centralised setting, and its performance is robust to the number of clients as well as data distribution.

Title: Offline Reinforcement Learning for LLM Multi-Step Reasoning

• Authors: Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, Yi Wu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16145
• Pdf URL: https://arxiv.org/pdf/2412.16145
• Copy Paste: [[2412.16145]] Offline Reinforcement Learning for LLM Multi-Step Reasoning(https://arxiv.org/abs/2412.16145)
• Keywords: large language model
• Abstract: Improving the multi-step reasoning ability of large language models (LLMs) with offline reinforcement learning (RL) is essential for quickly adapting them to complex tasks. While Direct Preference Optimization (DPO) has shown promise in aligning LLMs with human preferences, it is less suitable for multi-step reasoning tasks because (1) DPO relies on paired preference data, which is not readily available for multi-step reasoning tasks, and (2) it treats all tokens uniformly, making it ineffective for credit assignment in multi-step reasoning tasks, which often come with sparse reward. In this work, we propose OREO (Offline Reasoning Optimization), an offline RL method for enhancing LLM multi-step reasoning. Building on insights from previous works of maximum entropy reinforcement learning, it jointly learns a policy model and value function by optimizing the soft Bellman Equation. We show in principle that it reduces the need to collect pairwise data and enables better credit assignment. Empirically, OREO surpasses existing offline learning methods on multi-step reasoning benchmarks, including mathematical reasoning tasks (GSM8K, MATH) and embodied agent control (ALFWorld). The approach can be extended to a multi-iteration framework when additional resources are available. Furthermore, the learned value function can be leveraged to guide the tree search for free, which can further boost performance during test time.

Title: Mamba2D: A Natively Multi-Dimensional State-Space Model for Vision Tasks

• Authors: Enis Baty, Alejandro Hernández Díaz, Chris Bridges, Rebecca Davidson, Steve Eckersley, Simon Hadfield
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16146
• Pdf URL: https://arxiv.org/pdf/2412.16146
• Copy Paste: [[2412.16146]] Mamba2D: A Natively Multi-Dimensional State-Space Model for Vision Tasks(https://arxiv.org/abs/2412.16146)
• Keywords: transformer
• Abstract: State-Space Models (SSMs) have recently emerged as a powerful and efficient alternative to the long-standing transformer architecture. However, existing SSM conceptualizations retain deeply rooted biases from their roots in natural language processing. This constrains their ability to appropriately model the spatially-dependent characteristics of visual inputs. In this paper, we address these limitations by re-deriving modern selective state-space techniques, starting from a natively multidimensional formulation. Currently, prior works attempt to apply natively 1D SSMs to 2D data (i.e. images) by relying on arbitrary combinations of 1D scan directions to capture spatial dependencies. In contrast, Mamba2D improves upon this with a single 2D scan direction that factors in both dimensions of the input natively, effectively modelling spatial dependencies when constructing hidden states. Mamba2D shows comparable performance to prior adaptations of SSMs for vision tasks, on standard image classification evaluations with the ImageNet-1K dataset.

Title: SeagrassFinder: Deep Learning for Eelgrass Detection and Coverage Estimation in the Wild

• Authors: Jannik Elsäßer, Laura Weihl, Veronika Cheplygina, Lisbeth Tangaa Nielsen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16147
• Pdf URL: https://arxiv.org/pdf/2412.16147
• Copy Paste: [[2412.16147]] SeagrassFinder: Deep Learning for Eelgrass Detection and Coverage Estimation in the Wild(https://arxiv.org/abs/2412.16147)
• Keywords: transformer
• Abstract: Seagrass meadows play a crucial role in marine ecosystems, providing important services such as carbon sequestration, water quality improvement, and habitat provision. Monitoring the distribution and abundance of seagrass is essential for environmental impact assessments and conservation efforts. However, the current manual methods of analyzing underwater video transects to assess seagrass coverage are time-consuming and subjective. This work explores the use of deep learning models to automate the process of seagrass detection and coverage estimation from underwater video data. A dataset of over 8,300 annotated underwater images was created, and several deep learning architectures, including ResNet, InceptionNetV3, DenseNet, and Vision Transformer, were evaluated for the task of binary classification of ``Eelgrass Present'' and ``Eelgrass Absent'' images. The results demonstrate that deep learning models, particularly the Vision Transformer, can achieve high performance in predicting eelgrass presence, with AUROC scores exceeding 0.95 on the final test dataset. The use of transfer learning and the application of the Deep WaveNet underwater image enhancement model further improved the models' capabilities. The proposed methodology allows for the efficient processing of large volumes of video data, enabling the acquisition of much more detailed information on seagrass distributions compared to current manual methods. This information is crucial for environmental impact assessments and monitoring programs, as seagrasses are important indicators of coastal ecosystem health. Overall, this project demonstrates the value that deep learning can bring to the field of marine ecology and environmental monitoring.

Title: MotiF: Making Text Count in Image Animation with Motion Focal Loss

• Authors: Shijie Wang, Samaneh Azadi, Rohit Girdhar, Saketh Rambhatla, Chen Sun, Xi Yin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16153
• Pdf URL: https://arxiv.org/pdf/2412.16153
• Copy Paste: [[2412.16153]] MotiF: Making Text Count in Image Animation with Motion Focal Loss(https://arxiv.org/abs/2412.16153)
• Keywords: robust
• Abstract: Text-Image-to-Video (TI2V) generation aims to generate a video from an image following a text description, which is also referred to as text-guided image animation. Most existing methods struggle to generate videos that align well with the text prompts, particularly when motion is specified. To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation. We use optical flow to generate a motion heatmap and weight the loss according to the intensity of the motion. This modified objective leads to noticeable improvements and complements existing methods that utilize motion priors as model inputs. Additionally, due to the lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V Bench, a dataset consists of 320 image-text pairs for robust evaluation. We present a human evaluation protocol that asks the annotators to select an overall preference between two videos followed by their justifications. Through a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced models, achieving an average preference of 72%. The TI2V Bench is released in this https URL.

Title: Can Generative Video Models Help Pose Estimation?

• Authors: Ruojin Cai, Jason Y. Zhang, Philipp Henzler, Zhengqi Li, Noah Snavely, Ricardo Martin-Brualla
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16155
• Pdf URL: https://arxiv.org/pdf/2412.16155
• Copy Paste: [[2412.16155]] Can Generative Video Models Help Pose Estimation?(https://arxiv.org/abs/2412.16155)
• Keywords: generative
• Abstract: Pairwise pose estimation from images with little or no overlap is an open challenge in computer vision. Existing methods, even those trained on large-scale datasets, struggle in these scenarios due to the lack of identifiable correspondences or visual overlap. Inspired by the human ability to infer spatial relationships from diverse scenes, we propose a novel approach, InterPose, that leverages the rich priors encoded within pre-trained generative video models. We propose to use a video model to hallucinate intermediate frames between two input images, effectively creating a dense, visual transition, which significantly simplifies the problem of pose estimation. Since current video models can still produce implausible motion or inconsistent geometry, we introduce a self-consistency score that evaluates the consistency of pose predictions from sampled videos. We demonstrate that our approach generalizes among three state-of-the-art video models and show consistent improvements over the state-of-the-art DUSt3R on four diverse datasets encompassing indoor, outdoor, and object-centric scenes. Our findings suggest a promising avenue for improving pose estimation models by leveraging large generative models trained on vast amounts of video data, which is more readily available than 3D data. See our project page for results: this https URL.

Title: Personalized Representation from Personalized Generation

• Authors: Shobhita Sundaram, Julia Chae, Yonglong Tian, Sara Beery, Phillip Isola
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16156
• Pdf URL: https://arxiv.org/pdf/2412.16156
• Copy Paste: [[2412.16156]] Personalized Representation from Personalized Generation(https://arxiv.org/abs/2412.16156)
• Keywords: diffusion, segmentation
• Abstract: Modern vision models excel at general purpose downstream tasks. It is unclear, however, how they may be used for personalized vision tasks, which are both fine-grained and data-scarce. Recent works have successfully applied synthetic data to general-purpose representation learning, while advances in T2I diffusion models have enabled the generation of personalized images from just a few real examples. Here, we explore a potential connection between these ideas, and formalize the challenge of using personalized synthetic data to learn personalized representations, which encode knowledge about an object of interest and may be flexibly applied to any downstream task relating to the target object. We introduce an evaluation suite for this challenge, including reformulations of two existing datasets and a novel dataset explicitly constructed for this purpose, and propose a contrastive learning approach that makes creative use of image generators. We show that our method improves personalized representation learning for diverse downstream tasks, from recognition to segmentation, and analyze characteristics of image generation approaches that are key to this gain.

Title: HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding

• Authors: Chenxin Tao, Shiqian Su, Xizhou Zhu, Chenyu Zhang, Zhe Chen, Jiawen Liu, Wenhai Wang, Lewei Lu, Gao Huang, Yu Qiao, Jifeng Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16158
• Pdf URL: https://arxiv.org/pdf/2412.16158
• Copy Paste: [[2412.16158]] HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding(https://arxiv.org/abs/2412.16158)
• Keywords: large language model
• Abstract: The rapid advance of Large Language Models (LLMs) has catalyzed the development of Vision-Language Models (VLMs). Monolithic VLMs, which avoid modality-specific encoders, offer a promising alternative to the compositional ones but face the challenge of inferior performance. Most existing monolithic VLMs require tuning pre-trained LLMs to acquire vision abilities, which may degrade their language capabilities. To address this dilemma, this paper presents a novel high-performance monolithic VLM named HoVLE. We note that LLMs have been shown capable of interpreting images, when image embeddings are aligned with text embeddings. The challenge for current monolithic VLMs actually lies in the lack of a holistic embedding module for both vision and language inputs. Therefore, HoVLE introduces a holistic embedding module that converts visual and textual inputs into a shared space, allowing LLMs to process images in the same way as texts. Furthermore, a multi-stage training strategy is carefully designed to empower the holistic embedding module. It is first trained to distill visual features from a pre-trained vision encoder and text embeddings from the LLM, enabling large-scale training with unpaired random images and text tokens. The whole model further undergoes next-token prediction on multi-modal data to align the embeddings. Finally, an instruction-tuning stage is incorporated. Our experiments show that HoVLE achieves performance close to leading compositional models on various benchmarks, outperforming previous monolithic models by a large margin. Model available at this https URL.