2024-09-24

Title: A Knowledge-Centric Benchmarking Framework and Empirical Study for Retrieval-Augmented Generation

• Authors: Shuo Yu (1 and 2), Mingyue Cheng (1 and 2), Jiqian Yang (1 and 2), Jie Ouyang (1 and 2) ((1) Anhui Province Key Laboratory of Big Data Analysis and Application, University of Science and Technology of China (2) State Key Laboratory of Cognitive Intelligence)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13694
• Pdf URL: https://arxiv.org/pdf/2409.13694
• Copy Paste: [[2409.13694]] A Knowledge-Centric Benchmarking Framework and Empirical Study for Retrieval-Augmented Generation(https://arxiv.org/abs/2409.13694)
• Keywords: generative, large language model
• Abstract: Retrieval-Augmented Generation (RAG) enhances generative models by integrating retrieval mechanisms, which allow these models to access and utilize external knowledge sources. Despite its advantages, RAG encounters significant challenges, particularly in effectively handling real-world queries and mitigating hallucinations. The KDD Cup 2024 CRAG competition brings these issues to the forefront by incorporating both web pages and a mock API as knowledge sources, adding the complexity of parsing HTML before large language models (LLMs) can process the information. In this paper, we propose a novel RAG benchmark designed to address these challenges. Our work provides a comprehensive set of experimental results, offering valuable insights for the study of RAG. We thoroughly examine the entire RAG process, including knowledge source selection, retrieval, organization, and reasoning. Key findings from our study include the impact of automated knowledge source selection using agents and the influence of noise chunks on RAG reasoning. Additionally, we conduct detailed experiments to analyze the effects of various hyperparameters on RAG performance. To support further research, we have made our results, the associated code, and a parsed version of the CRAG dataset publicly available\footnote{this https URL}, contributing to the advancement of RAG methodologies and establishing a solid foundation for future work in this domain.

Title: You Only Use Reactive Attention Slice For Long Context Retrieval

• Authors: Yun Joon Soh, Hanxian Huang, Yuandong Tian, Jishen Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13695
• Pdf URL: https://arxiv.org/pdf/2409.13695
• Copy Paste: [[2409.13695]] You Only Use Reactive Attention Slice For Long Context Retrieval(https://arxiv.org/abs/2409.13695)
• Keywords: large language model
• Abstract: Supporting longer context for Large Language Models (LLM) is a promising direction to advance LLMs. As training a model for a longer context window is computationally expensive, many alternative solutions, such as Retrieval Augmented Generation (RAG), have been used. However, most existing RAG methods adopt embedding-based retrieval that falls short on long contexts. To address such challenges, we propose an attention-based retrieval technique, You Only Use Reactive Attention slice (YOURA). YOURA leverages a novel retrieval heuristic called reaction score to rank the relevance of each sentence in the input context with the query sentence. Intuitively, we measure how the per-token attention score "reacts" to the query and greedily retrieves the most reactive sentences. Internally, YOURA generates a token-indexed vector (called reaction vector) for the whole input context. To map each sentence to the token-indexed vector, we propose an Embedding-Agnostic Sentence Yield (EASY), a best-effort token wiggling algorithm. We evaluate our retrieval technique on three open-source pre-trained LLM models across six LongBench QA datasets. Our technique achieves up to 30% vLLM inference throughput improvement for serving long-context queries with a nearly identical quality score to the simple yet effective truncate-middle approach.

Title: CA-BERT: Leveraging Context Awareness for Enhanced Multi-Turn Chat Interaction

• Authors: Minghao Liu, Mingxiu Sui, Cangqing Wang, Zhejie Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13701
• Pdf URL: https://arxiv.org/pdf/2409.13701
• Copy Paste: [[2409.13701]] CA-BERT: Leveraging Context Awareness for Enhanced Multi-Turn Chat Interaction(https://arxiv.org/abs/2409.13701)
• Keywords: robust, transformer
• Abstract: Effective communication in automated chat systems hinges on the ability to understand and respond to context. Traditional models often struggle with determining when additional context is necessary for generating appropriate responses. This paper introduces Context-Aware BERT (CA-BERT), a transformer-based model specifically fine-tuned to address this challenge. CA-BERT innovatively applies deep learning techniques to discern context necessity in multi-turn chat interactions, enhancing both the relevance and accuracy of responses. We describe the development of CA-BERT, which adapts the robust architecture of BERT with a novel training regimen focused on a specialized dataset of chat dialogues. The model is evaluated on its ability to classify context necessity, demonstrating superior performance over baseline BERT models in terms of accuracy and efficiency. Furthermore, CA-BERT's implementation showcases significant reductions in training time and resource usage, making it feasible for real-time applications. The results indicate that CA-BERT can effectively enhance the functionality of chatbots by providing a nuanced understanding of context, thereby improving user experience and interaction quality in automated systems. This study not only advances the field of NLP in chat applications but also provides a framework for future research into context-sensitive AI developments.

Title: Shaping the Future of Endangered and Low-Resource Languages -- Our Role in the Age of LLMs: A Keynote at ECIR 2024

• Authors: Josiane Mothe (IRIT-SIG)
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.13702
• Pdf URL: https://arxiv.org/pdf/2409.13702
• Copy Paste: [[2409.13702]] Shaping the Future of Endangered and Low-Resource Languages -- Our Role in the Age of LLMs: A Keynote at ECIR 2024(https://arxiv.org/abs/2409.13702)
• Keywords: large language model
• Abstract: Isidore of Seville is credited with the adage that it is language that gives birth to a people, and not the other way around , underlining the profound role played by language in the formation of cultural and social identity. Today, of the more than 7100 languages listed, a significant number are endangered. Since the 1970s, linguists, information seekers and enthusiasts have helped develop digital resources and automatic tools to support a wide range of languages, including endangered ones. The advent of Large Language Model (LLM) technologies holds both promise and peril. They offer unprecedented possibilities for the translation and generation of content and resources, key elements in the preservation and revitalisation of languages. They also present threat of homogenisation, cultural oversimplification and the further marginalisation of already vulnerable languages. The talk this paper is based on has proposed an initiatory journey, exploring the potential paths and partnerships between technology and tradition, with a particular focus on the Occitan language. Occitan is a language from Southern France, parts of Spain and Italy that played a major cultural and economic role, particularly in the Middle Ages. It is now endangered according to UNESCO. The talk critically has examined how human expertise and artificial intelligence can work together to offer hope for preserving the linguistic diversity that forms the foundation of our global and especially our European heritage while addressing some of the ethical and practical challenges that accompany the use of these powerful technologies. This paper is based on the keynote I gave at the 46th European Conference on Information Retrieval (ECIR 2024). As an alternative to reading this paper, a video talk is available online. 1 Date: 26 March 2024.

Title: Entity Extraction from High-Level Corruption Schemes via Large Language Models

• Authors: Panagiotis Koletsis, Panagiotis-Konstantinos Gemos, Christos Chronis, Iraklis Varlamis, Vasilis Efthymiou, Georgios Th. Papadopoulos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13704
• Pdf URL: https://arxiv.org/pdf/2409.13704
• Copy Paste: [[2409.13704]] Entity Extraction from High-Level Corruption Schemes via Large Language Models(https://arxiv.org/abs/2409.13704)
• Keywords: extraction, large language model
• Abstract: The rise of financial crime that has been observed in recent years has created an increasing concern around the topic and many people, organizations and governments are more and more frequently trying to combat it. Despite the increase of interest in this area, there is a lack of specialized datasets that can be used to train and evaluate works that try to tackle those problems. This article proposes a new micro-benchmark dataset for algorithms and models that identify individuals and organizations, and their multiple writings, in news articles, and presents an approach that assists in its creation. Experimental efforts are also reported, using this dataset, to identify individuals and organizations in financial-crime-related articles using various low-billion parameter Large Language Models (LLMs). For these experiments, standard metrics (Accuracy, Precision, Recall, F1 Score) are reported and various prompt variants comprising the best practices of prompt engineering are tested. In addition, to address the problem of ambiguous entity mentions, a simple, yet effective LLM-based disambiguation method is proposed, ensuring that the evaluation aligns with reality. Finally, the proposed approach is compared against a widely used state-of-the-art open-source baseline, showing the superiority of the proposed method.

Title: Debiasing Text Safety Classifiers through a Fairness-Aware Ensemble

• Authors: Olivia Sturman, Aparna Joshi, Bhaktipriya Radharapu, Piyush Kumar, Renee Shelby
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13705
• Pdf URL: https://arxiv.org/pdf/2409.13705
• Copy Paste: [[2409.13705]] Debiasing Text Safety Classifiers through a Fairness-Aware Ensemble(https://arxiv.org/abs/2409.13705)
• Keywords: fair, large language model
• Abstract: Increasing use of large language models (LLMs) demand performant guardrails to ensure the safety of inputs and outputs of LLMs. When these safeguards are trained on imbalanced data, they can learn the societal biases. We present a light-weight, post-processing method for mitigating counterfactual fairness in closed-source text safety classifiers. Our approach involves building an ensemble that not only outperforms the input classifiers and policy-aligns them, but also acts as a debiasing regularizer. We introduce two threshold-agnostic metrics to assess the counterfactual fairness of a model, and demonstrate how combining these metrics with Fair Data Reweighting (FDW) helps mitigate biases. We create an expanded Open AI dataset, and a new templated LLM-generated dataset based on user-prompts, both of which are counterfactually balanced across identity groups and cover four key areas of safety; we will work towards publicly releasing these datasets. Our results show that our approach improves counterfactual fairness with minimal impact on model performance.

Title: Towards Safe Multilingual Frontier AI

• Authors: Artūrs Kanepajs, Vladimir Ivanov, Richard Moulange
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.13708
• Pdf URL: https://arxiv.org/pdf/2409.13708
• Copy Paste: [[2409.13708]] Towards Safe Multilingual Frontier AI(https://arxiv.org/abs/2409.13708)
• Keywords: diffusion, large language model
• Abstract: Linguistically inclusive LLMs -- which maintain good performance regardless of the language with which they are prompted -- are necessary for the diffusion of AI benefits around the world. Multilingual jailbreaks that rely on language translation to evade safety measures undermine the safe and inclusive deployment of AI systems. We provide policy recommendations to enhance the multilingual capabilities of AI while mitigating the risks of multilingual jailbreaks. We quantitatively assess the relationship between language resourcedness and model vulnerabilities to multilingual jailbreaks for five frontier large language models across 24 official EU languages. Building on prior research, we propose policy actions that align with the EU legal landscape and institutional framework to address multilingual jailbreaks, while promoting linguistic inclusivity. These include mandatory assessments of multilingual capabilities and vulnerabilities, public opinion research, and state support for multilingual AI development. The measures aim to improve AI safety and functionality through EU policy initiatives, guiding the implementation of the EU AI Act and informing regulatory efforts of the European AI Office.

Title: Column Vocabulary Association (CVA): semantic interpretation of dataless tables

• Authors: Margherita Martorana, Xueli Pan, Benno Kruit, Tobias Kuhn, Jacco van Ossenbruggen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13709
• Pdf URL: https://arxiv.org/pdf/2409.13709
• Copy Paste: [[2409.13709]] Column Vocabulary Association (CVA): semantic interpretation of dataless tables(https://arxiv.org/abs/2409.13709)
• Keywords: large language model
• Abstract: Traditional Semantic Table Interpretation (STI) methods rely primarily on the underlying table data to create semantic annotations. This year's SemTab challenge introduced the ``Metadata to KG'' track, which focuses on performing STI by using only metadata information, without access to the underlying data. In response to this new challenge, we introduce a new term: Column Vocabulary Association (CVA). This term refers to the task of semantic annotation of column headers solely based on metadata information. In this study, we evaluate the performance of various methods in executing the CVA task, including a Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) approach, as well as a more traditional similarity approach with SemanticBERT. Our methodology uses a zero-shot setting, with no pretraining or examples passed to the Large Language Models (LLMs), as we aim to avoid a domain-specific setting. We investigate a total of 7 different LLMs, of which three commercial GPT models (i.e. gpt-3.5-turbo-0.125, gpt-4o and gpt-4-turbo) and four open source models (i.e. llama3-80b, llama3-7b, gemma-7b and mixtral-8x7b). We integrate this models with RAG systems, and we explore how variations in temperature settings affect performances. Moreover, we continue our investigation by performing the CVA task utilizing SemanticBERT, analyzing how various metadata information influence its performance. Initial findings indicate that LLMs generally perform well at temperatures below 1.0, achieving an accuracy of 100\% in certain cases. Nevertheless, our investigation also reveal that the nature of the data significantly influences CVA task outcomes. In fact, in cases where the input data and glossary are related (for example by being created by the same organizations) traditional methods appear to surpass the performance of LLMs.

Title: You can remove GPT2's LayerNorm by fine-tuning

• Authors: Stefan Heimersheim
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13710
• Pdf URL: https://arxiv.org/pdf/2409.13710
• Copy Paste: [[2409.13710]] You can remove GPT2's LayerNorm by fine-tuning(https://arxiv.org/abs/2409.13710)
• Keywords: interpretability, transformer, large language model
• Abstract: The LayerNorm (LN) layer in GPT-style transformer models has long been a hindrance to mechanistic interpretability. LN is a crucial component required to stabilize the training of large language models, and LN or the similar RMSNorm have been used in practically all large language models based on the transformer architecture. The non-linear nature of the LN layers is a hindrance for mechanistic interpretability as it hinders interpretation of the residual stream, and makes it difficult to decompose the model into circuits. Some research have gone so far as to name "reasons interpretability researchers hate layer norm". In this paper we show that it is possible to remove the LN layers from a pre-trained GPT2-small model by fine-tuning on a fraction (500M tokens) of the training data. We demonstrate that this LN-free model achieves similar performance to the original model on the OpenWebText and ThePile datasets (-0.05 cross-entropy loss), and the Hellaswag benchmark (-0.5% accuracy). We provide the fine-tuning procedure and a Hugging Face repository with the fine-tuned GPT2-small models. Our work not only provides a simplified model for mechanistic interpretability research, but also provides evidence that the LN layers, at inference time, do not play a crucial role in transformer models.

Title: Good Idea or Not, Representation of LLM Could Tell

• Authors: Yi Xu, Bo Xue, Shuqian Sheng, Cheng Deng, Jiaxin Ding, Zanwei Shen, Luoyi Fu, Xinbing Wang, Chenghu Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13712
• Pdf URL: https://arxiv.org/pdf/2409.13712
• Copy Paste: [[2409.13712]] Good Idea or Not, Representation of LLM Could Tell(https://arxiv.org/abs/2409.13712)
• Keywords: generative, large language model
• Abstract: In the ever-expanding landscape of academic research, the proliferation of ideas presents a significant challenge for researchers: discerning valuable ideas from the less impactful ones. The ability to efficiently evaluate the potential of these ideas is crucial for the advancement of science and paper review. In this work, we focus on idea assessment, which aims to leverage the knowledge of large language models to assess the merit of scientific ideas. First, we investigate existing text evaluation research and define the problem of quantitative evaluation of ideas. Second, we curate and release a benchmark dataset from nearly four thousand manuscript papers with full texts, meticulously designed to train and evaluate the performance of different approaches to this task. Third, we establish a framework for quantifying the value of ideas by employing representations in a specific layer of large language models. Experimental results show that the scores predicted by our method are relatively consistent with those of humans. Our findings suggest that the representations of large language models hold more potential in quantifying the value of ideas than their generative outputs, demonstrating a promising avenue for automating the idea assessment process.

Title: Sentiment Informed Sentence BERT-Ensemble Algorithm for Depression Detection

• Authors: Bayode Ogunleye, Hemlata Sharma, Olamilekan Shobayo
• Subjects: cs.CL, cs.LG, math.ST, stat.AP
• Abstract URL: https://arxiv.org/abs/2409.13713
• Pdf URL: https://arxiv.org/pdf/2409.13713
• Copy Paste: [[2409.13713]] Sentiment Informed Sentence BERT-Ensemble Algorithm for Depression Detection(https://arxiv.org/abs/2409.13713)
• Keywords: transformer
• Abstract: The World Health Organisation (WHO) revealed approximately 280 million people in the world suffer from depression. Yet, existing studies on early-stage depression detection using machine learning (ML) techniques are limited. Prior studies have applied a single stand-alone algorithm, which is unable to deal with data complexities, prone to overfitting, and limited in generalization. To this end, our paper examined the performance of several ML algorithms for early-stage depression detection using two benchmark social media datasets (D1 and D2). More specifically, we incorporated sentiment indicators to improve our model performance. Our experimental results showed that sentence bidirectional encoder representations from transformers (SBERT) numerical vectors fitted into the stacking ensemble model achieved comparable F1 scores of 69% in the dataset (D1) and 76% in the dataset (D2). Our findings suggest that utilizing sentiment indicators as an additional feature for depression detection yields an improved model performance, and thus, we recommend the development of a depressive term corpus for future work.

Title: TracrBench: Generating Interpretability Testbeds with Large Language Models

• Authors: Hannes Thurnherr, Jérémy Scheurer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13714
• Pdf URL: https://arxiv.org/pdf/2409.13714
• Copy Paste: [[2409.13714]] TracrBench: Generating Interpretability Testbeds with Large Language Models(https://arxiv.org/abs/2409.13714)
• Keywords: interpretability, transformer, large language model
• Abstract: Achieving a mechanistic understanding of transformer-based language models is an open challenge, especially due to their large number of parameters. Moreover, the lack of ground truth mappings between model weights and their functional roles hinders the effective evaluation of interpretability methods, impeding overall progress. Tracr, a method for generating compiled transformers with inherent ground truth mappings in RASP, has been proposed to address this issue. However, manually creating a large number of models needed for verifying interpretability methods is labour-intensive and time-consuming. In this work, we present a novel approach for generating interpretability test beds using large language models (LLMs) and introduce TracrBench, a novel dataset consisting of 121 manually written and LLM-generated, human-validated RASP programs and their corresponding transformer weights. During this process, we evaluate the ability of frontier LLMs to autonomously generate RASP programs and find that this task poses significant challenges. GPT-4-turbo, with a 20-shot prompt and best-of-5 sampling, correctly implements only 57 out of 101 test programs, necessitating the manual implementation of the remaining programs. With its 121 samples, TracrBench aims to serve as a valuable testbed for evaluating and comparing interpretability methods.

Title: DiVA-DocRE: A Discriminative and Voice-Aware Paradigm for Document-Level Relation Extraction

• Authors: Yiheng Wu, Roman Yangarber, Xian Mao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13717
• Pdf URL: https://arxiv.org/pdf/2409.13717
• Copy Paste: [[2409.13717]] DiVA-DocRE: A Discriminative and Voice-Aware Paradigm for Document-Level Relation Extraction(https://arxiv.org/abs/2409.13717)
• Keywords: extraction, large language model
• Abstract: The remarkable capabilities of Large Language Models (LLMs) in text comprehension and generation have revolutionized Information Extraction (IE). One such advancement is in Document-level Relation Triplet Extraction (DocRTE), a critical task in information systems that aims to extract entities and their semantic relationships from documents. However, existing methods are primarily designed for Sentence level Relation Triplet Extraction (SentRTE), which typically handles a limited set of relations and triplet facts within a single sentence. Additionally, some approaches treat relations as candidate choices integrated into prompt templates, resulting in inefficient processing and suboptimal performance when determining the relation elements in triplets. To address these limitations, we introduce a Discriminative and Voice Aware Paradigm DiVA. DiVA involves only two steps: performing document-level relation extraction (DocRE) and then identifying the subject object entities based on the relation. No additional processing is required simply input the document to directly obtain the triplets. This streamlined process more accurately reflects real-world scenarios for triplet extraction. Our innovation lies in transforming DocRE into a discriminative task, where the model pays attention to each relation and to the often overlooked issue of active vs. passive voice within the triplet. Our experiments on the Re-DocRED and DocRED datasets demonstrate state-of-the-art results for the DocRTE task.

Title: LegiLM: A Fine-Tuned Legal Language Model for Data Compliance

• Authors: Linkai Zhu, Lu Yang, Chaofan Li, Shanwen Hu, Lu Liu, Bin Yin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13721
• Pdf URL: https://arxiv.org/pdf/2409.13721
• Copy Paste: [[2409.13721]] LegiLM: A Fine-Tuned Legal Language Model for Data Compliance(https://arxiv.org/abs/2409.13721)
• Keywords: security, privacy, protect
• Abstract: Ensuring compliance with international data protection standards for privacy and data security is a crucial but complex task, often requiring substantial legal expertise. This paper introduces LegiLM, a novel legal language model specifically tailored for consulting on data or information compliance. LegiLM leverages a pre-trained GDPR Fines dataset and has been fine-tuned to automatically assess whether particular actions or events breach data security and privacy regulations. By incorporating a specialized dataset that includes global data protection laws, meticulously annotated policy documents, and relevant privacy policies, LegiLM is optimized for addressing data compliance challenges. The model integrates advanced legal reasoning methods and information retrieval enhancements to enhance accuracy and reliability in practical legal consulting scenarios. Our evaluation using a custom benchmark dataset demonstrates that LegiLM excels in detecting data regulation breaches, offering sound legal justifications, and recommending necessary compliance modifications, setting a new benchmark for AI-driven legal compliance solutions. Our resources are publicly available at this https URL

Title: Explainable Malware Analysis: Concepts, Approaches and Challenges

• Authors: Harikha Manthena, Shaghayegh Shajarian, Jeffrey Kimmell, Mahmoud Abdelsalam, Sajad Khorsandroo, Maanak Gupta
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13723
• Pdf URL: https://arxiv.org/pdf/2409.13723
• Copy Paste: [[2409.13723]] Explainable Malware Analysis: Concepts, Approaches and Challenges(https://arxiv.org/abs/2409.13723)
• Keywords: security, attack
• Abstract: Machine learning (ML) has seen exponential growth in recent years, finding applications in various domains such as finance, medicine, and cybersecurity. Malware remains a significant threat to modern computing, frequently used by attackers to compromise systems. While numerous machine learning-based approaches for malware detection achieve high performance, they often lack transparency and fail to explain their predictions. This is a critical drawback in malware analysis, where understanding the rationale behind detections is essential for security analysts to verify and disseminate information. Explainable AI (XAI) addresses this issue by maintaining high accuracy while producing models that provide clear, understandable explanations for their decisions. In this survey, we comprehensively review the current state-of-the-art ML-based malware detection techniques and popular XAI approaches. Additionally, we discuss research implementations and the challenges of explainable malware analysis. This theoretical survey serves as an entry point for researchers interested in XAI applications in malware detection. By analyzing recent advancements in explainable malware analysis, we offer a broad overview of the progress in this field, positioning our work as the first to extensively cover XAI methods for malware classification and detection.

Title: Logically Consistent Language Models via Neuro-Symbolic Integration

• Authors: Diego Calanzone, Stefano Teso, Antonio Vergari
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13724
• Pdf URL: https://arxiv.org/pdf/2409.13724
• Copy Paste: [[2409.13724]] Logically Consistent Language Models via Neuro-Symbolic Integration(https://arxiv.org/abs/2409.13724)
• Keywords: large language model
• Abstract: Large language models (LLMs) are a promising venue for natural language understanding and generation. However, current LLMs are far from reliable: they are prone to generating non-factual information and, more crucially, to contradicting themselves when prompted to reason about relations between entities of the world. These problems are currently addressed with large scale fine-tuning or by delegating reasoning to external tools. In this work, we strive for a middle ground and introduce a loss based on neuro-symbolic reasoning that teaches an LLM to be logically consistent with an external set of facts and rules and improves self-consistency even when the LLM is fine-tuned on a limited set of facts. Our approach also allows to easily combine multiple logical constraints at once in a principled way, delivering LLMs that are more consistent w.r.t. all constraints and improve over several baselines w.r.t. a given constraint. Moreover, our method allows LLMs to extrapolate to unseen but semantically similar factual knowledge, represented in unseen datasets, more systematically.

Title: Identity-related Speech Suppression in Generative AI Content Moderation

• Authors: Oghenefejiro Isaacs Anigboro, Charlie M. Crawford, Danaë Metaxa, Sorelle A. Friedler
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.13725
• Pdf URL: https://arxiv.org/pdf/2409.13725
• Copy Paste: [[2409.13725]] Identity-related Speech Suppression in Generative AI Content Moderation(https://arxiv.org/abs/2409.13725)
• Keywords: generative
• Abstract: Automated content moderation has long been used to help identify and filter undesired user-generated content online. Generative AI systems now use such filters to keep undesired generated content from being created by or shown to users. From classrooms to Hollywood, as generative AI is increasingly used for creative or expressive text generation, whose stories will these technologies allow to be told, and whose will they suppress? In this paper, we define and introduce measures of speech suppression, focusing on speech related to different identity groups incorrectly filtered by a range of content moderation APIs. Using both short-form, user-generated datasets traditional in content moderation and longer generative AI-focused data, including two datasets we introduce in this work, we create a benchmark for measurement of speech suppression for nine identity groups. Across one traditional and four generative AI-focused automated content moderation services tested, we find that identity-related speech is more likely to be incorrectly suppressed than other speech except in the cases of a few non-marginalized groups. Additionally, we find differences between APIs in their abilities to correctly moderate generative AI content.

Title: Classification performance and reproducibility of GPT-4 omni for information extraction from veterinary electronic health records

• Authors: Judit M Wulcan, Kevin L Jacques, Mary Ann Lee, Samantha L Kovacs, Nicole Dausend, Lauren E Prince, Jonatan Wulcan, Sina Marsilio, Stefan M Keller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13727
• Pdf URL: https://arxiv.org/pdf/2409.13727
• Copy Paste: [[2409.13727]] Classification performance and reproducibility of GPT-4 omni for information extraction from veterinary electronic health records(https://arxiv.org/abs/2409.13727)
• Keywords: extraction, large language model
• Abstract: Large language models (LLMs) can extract information from veterinary electronic health records (EHRs), but performance differences between models, the effect of temperature settings, and the influence of text ambiguity have not been previously evaluated. This study addresses these gaps by comparing the performance of GPT-4 omni (GPT-4o) and GPT-3.5 Turbo under different conditions and investigating the relationship between human interobserver agreement and LLM errors. The LLMs and five humans were tasked with identifying six clinical signs associated with Feline chronic enteropathy in 250 EHRs from a veterinary referral hospital. At temperature 0, the performance of GPT-4o compared to the majority opinion of human respondents, achieved 96.9% sensitivity (interquartile range [IQR] 92.9-99.3%), 97.6% specificity (IQR 96.5-98.5%), 80.7% positive predictive value (IQR 70.8-84.6%), 99.5% negative predictive value (IQR 99.0-99.9%), 84.4% F1 score (IQR 77.3-90.4%), and 96.3% balanced accuracy (IQR 95.0-97.9%). The performance of GPT-4o was significantly better than that of its predecessor, GPT-3.5 Turbo, particularly with respect to sensitivity where GPT-3.5 Turbo only achieved 81.7% (IQR 78.9-84.8%). Adjusting the temperature for GPT-4o did not significantly impact classification performance. GPT-4o demonstrated greater reproducibility than human pairs regardless of temperature, with an average Cohen's kappa of 0.98 (IQR 0.98-0.99) at temperature 0 compared to 0.8 (IQR 0.78-0.81) for humans. Most GPT-4o errors occurred in instances where humans disagreed (35/43 errors, 81.4%), suggesting that these errors were more likely caused by ambiguity of the EHR than explicit model faults. Using GPT-4o to automate information extraction from veterinary EHRs is a viable alternative to manual extraction.

Title: Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Prompts

• Authors: Anna Mészáros, Szilvia Ujváry, Wieland Brendel, Patrik Reizinger, Ferenc Huszár
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.13728
• Pdf URL: https://arxiv.org/pdf/2409.13728
• Copy Paste: [[2409.13728]] Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Prompts(https://arxiv.org/abs/2409.13728)
• Keywords: transformer
• Abstract: LLMs show remarkable emergent abilities, such as inferring concepts from presumably out-of-distribution prompts, known as in-context learning. Though this success is often attributed to the Transformer architecture, our systematic understanding is limited. In complex real-world data sets, even defining what is out-of-distribution is not obvious. To better understand the OOD behaviour of autoregressive LLMs, we focus on formal languages, which are defined by the intersection of rules. We define a new scenario of OOD compositional generalization, termed rule extrapolation. Rule extrapolation describes OOD scenarios, where the prompt violates at least one rule. We evaluate rule extrapolation in formal languages with varying complexity in linear and recurrent architectures, the Transformer, and state space models to understand the architectures' influence on rule extrapolation. We also lay the first stones of a normative theory of rule extrapolation, inspired by the Solomonoff prior in algorithmic information theory.

Title: MathGLM-Vision: Solving Mathematical Problems with Multi-Modal Large Language Model

• Authors: Zhen Yang, Jinhao Chen, Zhengxiao Du, Wenmeng Yu, Weihan Wang, Wenyi Hong, Zhihuan Jiang, Bin Xu, Yuxiao Dong, Jie Tang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13729
• Pdf URL: https://arxiv.org/pdf/2409.13729
• Copy Paste: [[2409.13729]] MathGLM-Vision: Solving Mathematical Problems with Multi-Modal Large Language Model(https://arxiv.org/abs/2409.13729)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated significant capabilities in mathematical reasoning, particularly with text-based mathematical problems. However, current multi-modal large language models (MLLMs), especially those specialized in mathematics, tend to focus predominantly on solving geometric problems but ignore the diversity of visual information available in other areas of mathematics. Moreover, the geometric information for these specialized mathematical MLLMs is derived from several public datasets, which are typically limited in diversity and complexity. To address these limitations, we aim to construct a fine-tuning dataset named MathVL, and develop a series of specialized mathematical MLLMs termed MathGLM-Vision by conducting Supervised Fine-Tuning (SFT) on MathVL with various parameter-scale backbones. To extensively evaluate the effectiveness of MathGLM-Vision, we conduct experiments on several public benchmarks and our curated MathVL-test consisting of 2,000 problems. Experimental results demonstrate that MathGLM-Vision achieves significant improvements compared with some existing models, including backbone models and open-source mathematical MLLMs. These findings indicate the importance of diversity dataset in enhancing the mathematical reasoning abilities of MLLMs.

Title: KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation

• Authors: Lei Liang, Mengshu Sun, Zhengke Gui, Zhongshu Zhu, Zhouyu Jiang, Ling Zhong, Yuan Qu, Peilong Zhao, Zhongpu Bo, Jin Yang, Huaidong Xiong, Lin Yuan, Jun Xu, Zaoyang Wang, Wen Zhang, Huajun Chen, Zhiqiang Zhang, Jun Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13731
• Pdf URL: https://arxiv.org/pdf/2409.13731
• Copy Paste: [[2409.13731]] KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation(https://arxiv.org/abs/2409.13731)
• Keywords: large language model
• Abstract: The recently developed retrieval-augmented generation (RAG) technology enables the efficient construction of domain-specific applications. However, it faces limitations due to fuzzy retrieval processes, the "hallucination" problem of understanding and reasoning capabilities of general language models, and cascading losses in complex systems. These challenges hinder the effectiveness of specialized knowledge services. However, in scenarios such as scientific computing, medicine, and law, the accuracy of knowledge, the completeness of information, and the logical rigor of rules, time, and values are particularly critical. We Introduce professional domain knowledge service framework: Knowledge Augmented Generation(KAG) to improve generation and reasoning performance by bidirectionally enhancing large language model(LLM)s and knowledge graph(KG)s, including five key enhancements: 1) LLM-friendly knowledge semantic representation, 2) mutual indexing between knowledge graph and original chunks, 3) logicalform-guided hybrid reasoning and solving, 4) Knowledge alignment based on semantic reasoning, 5) Model for KAG. We compared KAG with existing RAG methods in multi-hop question answering. The results show that KAG performs significantly better than the state-of-the-art methods, with a relative improvement from 19.6% to 33.4% in F1. We apply KAG to two professional knowledge Q&A tasks of Ant Group, including E-Goverment Q&A and E-Health Q&A, and has achieved significant improvement in professionalism compared with NaiveRAG. We will soon natively support KAG on the open source KG engine OpenSPG, allowing developers to more easily build rigorous knowledge decision-making or convenient information retrieval services.

Title: TopoChat: Enhancing Topological Materials Retrieval With Large Language Model and Multi-Source Knowledge

• Authors: HuangChao Xu, Baohua Zhang, Zhong Jin, Tiannian Zhu, Quansheng Wu, Hongming Weng
• Subjects: cs.CL, cond-mat.mtrl-sci, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13732
• Pdf URL: https://arxiv.org/pdf/2409.13732
• Copy Paste: [[2409.13732]] TopoChat: Enhancing Topological Materials Retrieval With Large Language Model and Multi-Source Knowledge(https://arxiv.org/abs/2409.13732)
• Keywords: large language model
• Abstract: Large language models (LLMs), such as ChatGPT, have demonstrated impressive performance in the text generation task, showing the ability to understand and respond to complex instructions. However, the performance of naive LLMs in speciffc domains is limited due to the scarcity of domain-speciffc corpora and specialized training. Moreover, training a specialized large-scale model necessitates signiffcant hardware resources, which restricts researchers from leveraging such models to drive advances. Hence, it is crucial to further improve and optimize LLMs to meet speciffc domain demands and enhance their scalability. Based on the condensed matter data center, we establish a material knowledge graph (MaterialsKG) and integrate it with literature. Using large language models and prompt learning, we develop a specialized dialogue system for topological materials called TopoChat. Compared to naive LLMs, TopoChat exhibits superior performance in structural and property querying, material recommendation, and complex relational reasoning. This system enables efffcient and precise retrieval of information and facilitates knowledge interaction, thereby encouraging the advancement on the ffeld of condensed matter materials.

Title: RNR: Teaching Large Language Models to Follow Roles and Rules

• Authors: Kuan Wang, Alexander Bukharin, Haoming Jiang, Qingyu Yin, Zhengyang Wang, Tuo Zhao, Jingbo Shang, Chao Zhang, Bing Yin, Xian Li, Jianshu Chen, Shiyang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13733
• Pdf URL: https://arxiv.org/pdf/2409.13733
• Copy Paste: [[2409.13733]] RNR: Teaching Large Language Models to Follow Roles and Rules(https://arxiv.org/abs/2409.13733)
• Keywords: large language model
• Abstract: Instruction fine-tuning (IFT) elicits instruction following capabilities and steers the behavior of large language models (LLMs) via supervised learning. However, existing models trained on open-source IFT datasets only have the ability to follow instructions from users, and often fail to follow complex role and rules specified by developers, a.k.a. system prompts. The ability to follow these roles and rules is essential for deployment, as it ensures that the model safely interacts with users within developer defined guidelines. To improve such role and rule following ability, we propose \model, an automated data generation pipeline that generates diverse roles and rules from existing IFT instructions, along with corresponding responses. This data can then be used to train models that follow complex system prompts. The models are evaluated on our newly created benchmarks for role and rule following ability, as well as standard instruction-following benchmarks and general NLP tasks. Our framework significantly improves role and rule following capability in LLMs, as evidenced by over 25% increase in pass-rate on rule adherence, i.e. following all requirements, in our experiments with the Alpaca and Ultrachat datasets. Moreover, our models achieves this increase without any regression on popular instruction following benchmarks.

Title: Analysis of Socially Unacceptable Discourse with Zero-shot Learning

• Authors: Rayane Ghilene, Dimitra Niaouri, Michele Linardi, Julien Longhi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13735
• Pdf URL: https://arxiv.org/pdf/2409.13735
• Copy Paste: [[2409.13735]] Analysis of Socially Unacceptable Discourse with Zero-shot Learning(https://arxiv.org/abs/2409.13735)
• Keywords: robust, transformer
• Abstract: Socially Unacceptable Discourse (SUD) analysis is crucial for maintaining online positive environments. We investigate the effectiveness of Entailment-based zero-shot text classification (unsupervised method) for SUD detection and characterization by leveraging pre-trained transformer models and prompting techniques. The results demonstrate good generalization capabilities of these models to unseen data and highlight the promising nature of this approach for generating labeled datasets for the analysis and characterization of extremist narratives. The findings of this research contribute to the development of robust tools for studying SUD and promoting responsible communication online.

Title: NLP4PBM: A Systematic Review on Process Extraction using Natural Language Processing with Rule-based, Machine and Deep Learning Methods

• Authors: William Van Woensel, Soroor Motie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13738
• Pdf URL: https://arxiv.org/pdf/2409.13738
• Copy Paste: [[2409.13738]] NLP4PBM: A Systematic Review on Process Extraction using Natural Language Processing with Rule-based, Machine and Deep Learning Methods(https://arxiv.org/abs/2409.13738)
• Keywords: extraction
• Abstract: This literature review studies the field of automated process extraction, i.e., transforming textual descriptions into structured processes using Natural Language Processing (NLP). We found that Machine Learning (ML) / Deep Learning (DL) methods are being increasingly used for the NLP component. In some cases, they were chosen for their suitability towards process extraction, and results show that they can outperform classic rule-based methods. We also found a paucity of gold-standard, scalable annotated datasets, which currently hinders objective evaluations as well as the training or fine-tuning of ML / DL methods. Finally, we discuss preliminary work on the application of LLMs for automated process extraction, as well as promising developments in this field.

Title: Table-to-Text Generation with Pretrained Diffusion Models

• Authors: Aleksei S. Krylov, Oleg D. Somov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13739
• Pdf URL: https://arxiv.org/pdf/2409.13739
• Copy Paste: [[2409.13739]] Table-to-Text Generation with Pretrained Diffusion Models(https://arxiv.org/abs/2409.13739)
• Keywords: diffusion
• Abstract: Diffusion models have demonstrated significant potential in achieving state-of-the-art performance across various text generation tasks. In this systematic study, we investigate their application to the table-to-text problem by adapting the diffusion model to the task and conducting an in-depth analysis. Our experiments cover multiple aspects of diffusion models training. We explore sampling strategy influence by inducing recent diffusion model accelerator DPM-Solver++ into our core model. We have tested different prediction aggregation methods, like ROVER and Minimum Bayes-Risk (MBR). Our studies cover the impact of the pre-training phase in diffusion models and the generation length constraints influence. We also have compared diffusion model generation with auto-regressive text-to-text models with different temperature settings for diversity evaluation. Our key observation is that diffusion models demonstrate the balance between quality and diversity while auto-regressive text-to-text models are not successful at handling both at the same time. Furthermore, we found out that to achieve the highest quality possible, it is preferable to use a regular sampler with the strictest length constraint to create multiple samples, and then use MBR to aggregate the predictions. However, if you are prepared to give up high level of diversity and to accelerate the process, you can also utilize a fast sampler DPM-Solver++. Our findings reveal that diffusion models achieve comparable results in the table-to-text domain, highlighting their viability in the table-to-text challenge as a promising research direction.

Title: Knowing When to Ask -- Bridging Large Language Models and Data

• Authors: Prashanth Radhakrishnan, Jennifer Chen, Bo Xu, Prem Ramaswami, Hannah Pho, Adriana Olmos, James Manyika, R. V. Guha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13741
• Pdf URL: https://arxiv.org/pdf/2409.13741
• Copy Paste: [[2409.13741]] Knowing When to Ask -- Bridging Large Language Models and Data(https://arxiv.org/abs/2409.13741)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are prone to generating factually incorrect information when responding to queries that involve numerical and statistical data or other timely facts. In this paper, we present an approach for enhancing the accuracy of LLMs by integrating them with Data Commons, a vast, open-source repository of public statistics from trusted organizations like the United Nations (UN), Center for Disease Control and Prevention (CDC) and global census bureaus. We explore two primary methods: Retrieval Interleaved Generation (RIG), where the LLM is trained to produce natural language queries to retrieve data from Data Commons, and Retrieval Augmented Generation (RAG), where relevant data tables are fetched from Data Commons and used to augment the LLM's prompt. We evaluate these methods on a diverse set of queries, demonstrating their effectiveness in improving the factual accuracy of LLM outputs. Our work represents an early step towards building more trustworthy and reliable LLMs that are grounded in verifiable statistical data and capable of complex factual reasoning.

Title: Distinguishability Investigation on Longa's Atomic Patterns when used as a Basis for Implementing Elliptic Curve Scalar Multiplication Algorithms

• Authors: Sze Hei Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.13742
• Pdf URL: https://arxiv.org/pdf/2409.13742
• Copy Paste: [[2409.13742]] Distinguishability Investigation on Longa's Atomic Patterns when used as a Basis for Implementing Elliptic Curve Scalar Multiplication Algorithms(https://arxiv.org/abs/2409.13742)
• Keywords: security, attack, robust
• Abstract: In the evolving landscape of cryptographic security, the robustness of Elliptic Curve Cryptography (ECC) against side-channel analysis (SCA) attacks is of paramount importance due to the widespread use of ECC and the growing sophistication of SCAs. This thesis delves into the investigation of Longa's atomic patterns applied within Elliptic Curve scalar multiplication algorithms, assessing their resistance to horizontal SCAs. The research employs these atomic patterns in practical implementation on a microcontroller (Texas Instruments Launchpad F28379 board) using the open-source cryptographic library FLECC in C. In our analysis, we only focused on the distinguishability of the first atomic block in the Elliptic Curve point doubling and point addition patterns. Due to various technical limitations, we were unable to determine significant differences in the execution time and the shapes of the atomic blocks. Further investigations of the SCA-resistance can be performed based on this work. A significant contribution of this work is the identification and correction of several discrepancies in Longa's original atomic patterns. This thesis marks the first practical implementation of Longa's patterns, extending the theoretical research into empirical analysis.

Title: A Simplified Retriever to Improve Accuracy of Phenotype Normalizations by Large Language Models

• Authors: Daniel B. Hier, Thanh Son Do, Tayo Obafemi-Ajayi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13744
• Pdf URL: https://arxiv.org/pdf/2409.13744
• Copy Paste: [[2409.13744]] A Simplified Retriever to Improve Accuracy of Phenotype Normalizations by Large Language Models(https://arxiv.org/abs/2409.13744)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown improved accuracy in phenotype term normalization tasks when augmented with retrievers that suggest candidate normalizations based on term definitions. In this work, we introduce a simplified retriever that enhances LLM accuracy by searching the Human Phenotype Ontology (HPO) for candidate matches using contextual word embeddings from BioBERT without the need for explicit term definitions. Testing this method on terms derived from the clinical synopses of Online Mendelian Inheritance in Man (OMIM), we demonstrate that the normalization accuracy of a state-of-the-art LLM increases from a baseline of 62.3% without augmentation to 90.3% with retriever augmentation. This approach is potentially generalizable to other biomedical term normalization tasks and offers an efficient alternative to more complex retrieval methods.

Title: Context-Aware Membership Inference Attacks against Pre-trained Large Language Models

• Authors: Hongyan Chang, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, Reza Shokri
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.13745
• Pdf URL: https://arxiv.org/pdf/2409.13745
• Copy Paste: [[2409.13745]] Context-Aware Membership Inference Attacks against Pre-trained Large Language Models(https://arxiv.org/abs/2409.13745)
• Keywords: attack, membership infer, generative, large language model
• Abstract: Prior Membership Inference Attacks (MIAs) on pre-trained Large Language Models (LLMs), adapted from classification model attacks, fail due to ignoring the generative process of LLMs across token sequences. In this paper, we present a novel attack that adapts MIA statistical tests to the perplexity dynamics of subsequences within a data point. Our method significantly outperforms prior loss-based approaches, revealing context-dependent memorization patterns in pre-trained LLMs.

Title: When Less Is Not More: Large Language Models Normalize Less-Frequent Terms with Lower Accuracy

• Authors: Daniel B. Hier, Thanh Son Do, Tayo Obafemi-Ajayi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13746
• Pdf URL: https://arxiv.org/pdf/2409.13746
• Copy Paste: [[2409.13746]] When Less Is Not More: Large Language Models Normalize Less-Frequent Terms with Lower Accuracy(https://arxiv.org/abs/2409.13746)
• Keywords: large language model
• Abstract: Term normalization is the process of mapping a term from free text to a standardized concept and its machine-readable code in an ontology. Accurate normalization of terms that capture phenotypic differences between patients and diseases is critical to the success of precision medicine initiatives. A large language model (LLM), such as GPT-4o, can normalize terms to the Human Phenotype Ontology (HPO), but it may retrieve incorrect HPO IDs. Reported accuracy rates for LLMs on these tasks may be inflated due to imbalanced test datasets skewed towards high-frequency terms. In our study, using a comprehensive dataset of 268,776 phenotype annotations for 12,655 diseases from the HPO, GPT-4o achieved an accuracy of 13.1% in normalizing 11,225 unique terms. However, the accuracy was unevenly distributed, with higher-frequency and shorter terms normalized more accurately than lower-frequency and longer terms. Feature importance analysis, using SHAP and permutation methods, identified low-term frequency as the most significant predictor of normalization errors. These findings suggest that training and evaluation datasets for LLM-based term normalization should balance low- and high-frequency terms to improve model performance, particularly for infrequent terms critical to precision medicine.

Title: Machine Translation with Large Language Models: Decoder Only vs. Encoder-Decoder

• Authors: Abhinav P.M., SujayKumar Reddy M, Oswald Christopher
• Subjects: cs.CL, cs.ET, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13747
• Pdf URL: https://arxiv.org/pdf/2409.13747
• Copy Paste: [[2409.13747]] Machine Translation with Large Language Models: Decoder Only vs. Encoder-Decoder(https://arxiv.org/abs/2409.13747)
• Keywords: large language model
• Abstract: This project, titled "Machine Translation with Large Language Models: Decoder-only vs. Encoder-Decoder," aims to develop a multilingual machine translation (MT) model. Focused on Indian regional languages, especially Telugu, Tamil, and Malayalam, the model seeks to enable accurate and contextually appropriate translations across diverse language pairs. By comparing Decoder-only and Encoder-Decoder architectures, the project aims to optimize translation quality and efficiency, advancing cross-linguistic communication tools.The primary objective is to develop a model capable of delivering high-quality translations that are accurate and contextually appropriate. By leveraging large language models, specifically comparing the effectiveness of Decoder-only and Encoder-Decoder architectures, the project seeks to optimize translation performance and efficiency across multilingual contexts. Through rigorous experimentation and analysis, this project aims to advance the field of machine translation, contributing valuable insights into the effectiveness of different model architectures and paving the way for enhanced cross-linguistic communication tools.

Title: TheraGen: Therapy for Every Generation

• Authors: Kartikey Doshi, Jimit Shah, Narendra Shekokar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13748
• Pdf URL: https://arxiv.org/pdf/2409.13748
• Copy Paste: [[2409.13748]] TheraGen: Therapy for Every Generation(https://arxiv.org/abs/2409.13748)
• Keywords: transformer
• Abstract: We present TheraGen, an advanced AI-powered mental health chatbot utilizing the LLaMA 2 7B model. This approach builds upon recent advancements in language models and transformer architectures. TheraGen provides all-day personalized, compassionate mental health care by leveraging a large dataset of 1 million conversational entries, combining anonymized therapy transcripts, online mental health discussions, and psychological literature, including APA resources. Our implementation employs transfer learning, fine-tuning, and advanced training techniques to optimize performance. TheraGen offers a user-friendly interface for seamless interaction, providing empathetic responses and evidence-based coping strategies. Evaluation results demonstrate high user satisfaction rates, with 94% of users reporting improved mental well-being. The system achieved a BLEU score of 0.67 and a ROUGE score of 0.62, indicating strong response accuracy. With an average response time of 1395 milliseconds, TheraGen ensures real-time, efficient support. While not a replacement for professional therapy, TheraGen serves as a valuable complementary tool, significantly improving user well-being and addressing the accessibility gap in mental health treatments. This paper details TheraGen's architecture, training methodology, ethical considerations, and future directions, contributing to the growing field of AI-assisted mental healthcare and offering a scalable solution to the pressing need for mental health support.

Title: KodeXv0.1: A Family of State-of-the-Art Financial Large Language Models

• Authors: Neel Rajani, Lilli Kiessling, Aleksandr Ogaltsov, Claus Lang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13749
• Pdf URL: https://arxiv.org/pdf/2409.13749
• Copy Paste: [[2409.13749]] KodeXv0.1: A Family of State-of-the-Art Financial Large Language Models(https://arxiv.org/abs/2409.13749)
• Keywords: large language model
• Abstract: Although powerful, current cutting-edge LLMs may not fulfil the needs of highly specialised sectors. We introduce KodeXv0.1, a family of large language models that outclass GPT-4 in financial question answering. We utilise the base variants of Llama 3.1 8B and 70B and adapt them to the financial domain through a custom training regime. To this end, we collect and process a large number of publicly available financial documents such as earnings calls and business reports. These are used to generate a high-quality, synthetic dataset consisting of Context-Question-Answer triplets which closely mirror real-world financial tasks. Using the train split of this dataset, we perform RAG-aware 4bit LoRA instruction tuning runs of Llama 3.1 base variants to produce KodeX-8Bv0.1 and KodeX-70Bv0.1. We then complete extensive model evaluations using FinanceBench, FinQABench and the withheld test split of our dataset. Our results show that KodeX-8Bv0.1 is more reliable in financial contexts than cutting-edge instruct models in the same parameter regime, surpassing them by up to 9.24%. In addition, it is even capable of outperforming state-of-the-art proprietary models such as GPT-4 by up to 7.07%. KodeX-70Bv0.1 represents a further improvement upon this, exceeding GPT-4's performance on every tested benchmark.

Title: Thinking Before Speaking: A Role-playing Model with Mindset

• Authors: Baohua Zhang, Yongyi Huang, Wenyao Cui, Huaping Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13752
• Pdf URL: https://arxiv.org/pdf/2409.13752
• Copy Paste: [[2409.13752]] Thinking Before Speaking: A Role-playing Model with Mindset(https://arxiv.org/abs/2409.13752)
• Keywords: large language model
• Abstract: Role-playing is an easy task for Large Language Models (LLMs), as they are skilled at simulating human behaviors. Many current studies have enabled LLMs to generate responses in the tone of a specific role by fine-tuning the models or using specialized prompts. However, it is typically easy to recognize when a role is being played by LLMs. These models tend to perform poorly when confronted with knowledge that the assumed role does not possess, or a question that requires the specific experience or logic of the role to answer. To address this problem and make LLMs act more like real roles, we propose a Thinking Before Speaking (TBS) model in this paper. Unlike other studies, we first extend the data based on the character's real-life scenarios and the historical dialogue, supplementing each pair of dialogue with the character's mindset. Then we add few data points that include elements beyond the role's knowledge, and fine-tune the LLMs. This approach can help LLMs adopt the role's thought process and logic, avoiding responses that fall outside the role's knowledge base. We have also prepared a dataset and evaluation metrics to test these capabilities. Experimental results show that our TBS model can better emulate a role in terms of tone, knowledge, and mindset.

Title: Entity-Aware Self-Attention and Contextualized GCN for Enhanced Relation Extraction in Long Sentences

• Authors: Xin Wang, Xinyi Bai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13755
• Pdf URL: https://arxiv.org/pdf/2409.13755
• Copy Paste: [[2409.13755]] Entity-Aware Self-Attention and Contextualized GCN for Enhanced Relation Extraction in Long Sentences(https://arxiv.org/abs/2409.13755)
• Keywords: extraction
• Abstract: Relation extraction as an important natural Language processing (NLP) task is to identify relations between named entities in text. Recently, graph convolutional networks over dependency trees have been widely used to capture syntactic features and achieved attractive performance. However, most existing dependency-based approaches ignore the positive influence of the words outside the dependency trees, sometimes conveying rich and useful information on relation extraction. In this paper, we propose a novel model, Entity-aware Self-attention Contextualized GCN (ESC-GCN), which efficiently incorporates syntactic structure of input sentences and semantic context of sequences. To be specific, relative position self-attention obtains the overall semantic pairwise correlation related to word position, and contextualized graph convolutional networks capture rich intra-sentence dependencies between words by adequately pruning operations. Furthermore, entity-aware attention layer dynamically selects which token is more decisive to make final relation prediction. In this way, our proposed model not only reduces the noisy impact from dependency trees, but also obtains easily-ignored entity-related semantic representation. Extensive experiments on various tasks demonstrate that our model achieves encouraging performance as compared to existing dependency-based and sequence-based models. Specially, our model excels in extracting relations between entities of long sentences.

Title: Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance

• Authors: Adarsh MS, Jithin VG, Ditto PS
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13757
• Pdf URL: https://arxiv.org/pdf/2409.13757
• Copy Paste: [[2409.13757]] Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance(https://arxiv.org/abs/2409.13757)
• Keywords: large language model
• Abstract: Large language models (LLMs) are known for their exceptional performance across a range of natural language processing tasks, but their deployment comes at a high computational and financial cost. On the other hand, smaller language models (SLMs), which can be deployed on lower-cost edge devices, struggle to match the performance of their larger counterparts. This paper presents a novel hybrid inference approach that leverages the strengths of both model types while minimizing reliance on costly cloud-based LLMs. Unlike existing methods that route entire queries to either an SLM or a cloud LLM, our approach introduces a reward-based mechanism to dynamically determine the involvement of the cloud LLM during token generation. Specifically, each token predicted by the SLM is evaluated against a reward score, and only when this score falls below a certain threshold is the cloud LLM consulted for assistance in the next token prediction. This method not only reduces the traffic to the cloud LLM, thereby lowering costs, but also allows for flexible control over response quality depending on the reward score threshold. Experimental results demonstrate that our approach significantly reduces cloud LLM usage with minimal impact on overall response quality, offering a cost-effective solution for deploying high-performance language models

Title: Do Large Language Models Need a Content Delivery Network?

• Authors: Yihua Cheng, Kuntai Du, Jiayi Yao, Junchen Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13761
• Pdf URL: https://arxiv.org/pdf/2409.13761
• Copy Paste: [[2409.13761]] Do Large Language Models Need a Content Delivery Network?(https://arxiv.org/abs/2409.13761)
• Keywords: large language model
• Abstract: As the use of large language models (LLMs) expands rapidly, so does the range of knowledge needed to supplement various LLM queries. Thus, enabling flexible and efficient injection of new knowledge in LLM inference is critical. Three high-level options exist: (i) embedding the knowledge in LLM's weights (i.e., fine-tuning), (ii) including the knowledge as a part of LLM's text input (i.e., in-context learning), or (iii) injecting the KV caches of the new knowledge to LLM during prefill. This paper argues that, although fine-tuning and in-context learning are popular, using KV caches as the medium of knowledge could simultaneously enable more modular management of knowledge injection and more efficient LLM serving with low cost and fast response. To realize these benefits, we envision a Knowledge Delivery Network (KDN), a new system component in LLM services that dynamically optimizes the storage, transfer, and composition of KV cache across LLM engines and other compute and storage resources. We believe that, just like content delivery networks (CDNs), such as Akamai, enabled the success of the Internet ecosystem through their efficient data delivery, KDNs will be critical to the success of LLM applications through their efficient knowledge delivery. We have open-sourced a KDN prototype at this https URL.

Title: Local Explanations and Self-Explanations for Assessing Faithfulness in black-box LLMs

• Authors: Christos Fragkathoulas, Odysseas S. Chlapanis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13764
• Pdf URL: https://arxiv.org/pdf/2409.13764
• Copy Paste: [[2409.13764]] Local Explanations and Self-Explanations for Assessing Faithfulness in black-box LLMs(https://arxiv.org/abs/2409.13764)
• Keywords: explainability, large language model
• Abstract: This paper introduces a novel task to assess the faithfulness of large language models (LLMs) using local perturbations and self-explanations. Many LLMs often require additional context to answer certain questions correctly. For this purpose, we propose a new efficient alternative explainability technique, inspired by the commonly used leave-one-out approach. Using this approach, we identify the sufficient and necessary parts for the LLM to generate correct answers, serving as explanations. We propose a metric for assessing faithfulness that compares these crucial parts with the self-explanations of the model. Using the Natural Questions dataset, we validate our approach, demonstrating its effectiveness in explaining model decisions and assessing faithfulness.

Title: Magika: AI-Powered Content-Type Detection

• Authors: Yanick Fratantonio, Luca Invernizzi, Loua Farah, Kurt Thomas, Marina Zhang, Ange Albertini, Francois Galilee, Giancarlo Metitieri, Julien Cretin, Alex Petit-Bianco, David Tao, Elie Bursztein
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13768
• Pdf URL: https://arxiv.org/pdf/2409.13768
• Copy Paste: [[2409.13768]] Magika: AI-Powered Content-Type Detection(https://arxiv.org/abs/2409.13768)
• Keywords: security
• Abstract: The task of content-type detection -- which entails identifying the data encoded in an arbitrary byte sequence -- is critical for operating systems, development, reverse engineering environments, and a variety of security applications. In this paper, we introduce Magika, a novel AI-powered content-type detection tool. Under the hood, Magika employs a deep learning model that can execute on a single CPU with just 1MB of memory to store the model's weights. We show that Magika achieves an average F1 score of 99% across over a hundred content types and a test set of more than 1M files, outperforming all existing content-type detection tools today. In order to foster adoption and improvements, we open source Magika under an Apache 2 license on GitHub and make our model and training pipeline publicly available. Our tool has already seen adoption by the Gmail email provider for attachment scanning, and it has been integrated with VirusTotal to aid with malware analysis. We note that this paper discusses the first iteration of Magika, and a more recent version already supports more than 200 content types. The interested reader can see the latest development on the Magika GitHub repository, available at this https URL.

Title: A constrained optimization approach to improve robustness of neural networks

• Authors: Shudian Zhao, Jan Kronqvist
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2409.13770
• Pdf URL: https://arxiv.org/pdf/2409.13770
• Copy Paste: [[2409.13770]] A constrained optimization approach to improve robustness of neural networks(https://arxiv.org/abs/2409.13770)
• Keywords: attack, robust
• Abstract: In this paper, we present a novel nonlinear programming-based approach to fine-tune pre-trained neural networks to improve robustness against adversarial attacks while maintaining high accuracy on clean data. Our method introduces adversary-correction constraints to ensure correct classification of adversarial data and minimizes changes to the model parameters. We propose an efficient cutting-plane-based algorithm to iteratively solve the large-scale nonconvex optimization problem by approximating the feasible region through polyhedral cuts and balancing between robustness and accuracy. Computational experiments on standard datasets such as MNIST and CIFAR10 demonstrate that the proposed approach significantly improves robustness, even with a very small set of adversarial data, while maintaining minimal impact on accuracy.

Title: Trustworthy Intrusion Detection: Confidence Estimation Using Latent Space

• Authors: Ioannis Pitsiorlas, George Arvanitakis, Marios Kountouris
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13774
• Pdf URL: https://arxiv.org/pdf/2409.13774
• Copy Paste: [[2409.13774]] Trustworthy Intrusion Detection: Confidence Estimation Using Latent Space(https://arxiv.org/abs/2409.13774)
• Keywords: security, attack
• Abstract: This work introduces a novel method for enhancing confidence in anomaly detection in Intrusion Detection Systems (IDS) through the use of a Variational Autoencoder (VAE) architecture. By developing a confidence metric derived from latent space representations, we aim to improve the reliability of IDS predictions against cyberattacks. Applied to the NSL-KDD dataset, our approach focuses on binary classification tasks to effectively distinguish between normal and malicious network activities. The methodology demonstrates a significant enhancement in anomaly detection, evidenced by a notable correlation of 0.45 between the reconstruction error and the proposed metric. Our findings highlight the potential of employing VAEs for more accurate and trustworthy anomaly detection in network security.

Title: Revisiting Synthetic Human Trajectories: Imitative Generation and Benchmarks Beyond Datasaurus

• Authors: Bangchao Deng, Xin Jing, Tianyue Yang, Bingqing Qu, Philippe Cudre-Mauroux, Dingqi Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13790
• Pdf URL: https://arxiv.org/pdf/2409.13790
• Copy Paste: [[2409.13790]] Revisiting Synthetic Human Trajectories: Imitative Generation and Benchmarks Beyond Datasaurus(https://arxiv.org/abs/2409.13790)
• Keywords: privacy, generative
• Abstract: Human trajectory data, which plays a crucial role in various applications such as crowd management and epidemic prevention, is challenging to obtain due to practical constraints and privacy concerns. In this context, synthetic human trajectory data is generated to simulate as close as possible to real-world human trajectories, often under summary statistics and distributional similarities. However, the complexity of human mobility patterns is oversimplified by these similarities (a.k.a. ``Datasaurus''), resulting in intrinsic biases in both generative model design and benchmarks of the generated trajectories. Against this background, we propose MIRAGE, a huMan-Imitative tRAjectory GenErative model designed as a neural Temporal Point Process integrating an Exploration and Preferential Return model. It imitates the human decision-making process in trajectory generation, rather than fitting any specific statistical distributions as traditional methods do, thus avoiding the Datasaurus issue. Moreover, we also propose a comprehensive task-based evaluation protocol beyond Datasaurus to systematically benchmark trajectory generative models on four typical downstream tasks, integrating multiple techniques and evaluation metrics for each task, to comprehensively assess the ultimate utility of the generated trajectories. We conduct a thorough evaluation of MIRAGE on three real-world user trajectory datasets against a sizeable collection of baselines. Results show that compared to the best baselines, MIRAGE-generated trajectory data not only achieves the best statistical and distributional similarities with 59.0-71.5% improvement, but also yields the best performance in the task-based evaluation with 10.9-33.4% improvement.

Title: Continual Learning for Multimodal Data Fusion of a Soft Gripper

• Authors: Nilay Kushawaha, Egidio Falotico
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.13792
• Pdf URL: https://arxiv.org/pdf/2409.13792
• Copy Paste: [[2409.13792]] Continual Learning for Multimodal Data Fusion of a Soft Gripper(https://arxiv.org/abs/2409.13792)
• Keywords: robust
• Abstract: Continual learning (CL) refers to the ability of an algorithm to continuously and incrementally acquire new knowledge from its environment while retaining previously learned information. A model trained on one data modality often fails when tested with a different modality. A straightforward approach might be to fuse the two modalities by concatenating their features and training the model on the fused data. However, this requires retraining the model from scratch each time it encounters a new domain. In this paper, we introduce a continual learning algorithm capable of incrementally learning different data modalities by leveraging both class-incremental and domain-incremental learning scenarios in an artificial environment where labeled data is scarce, yet non-iid (independent and identical distribution) unlabeled data from the environment is plentiful. The proposed algorithm is efficient and only requires storing prototypes for each class. We evaluate the algorithm's effectiveness on a challenging custom multimodal dataset comprising of tactile data from a soft pneumatic gripper, and visual data from non-stationary images of objects extracted from video sequences. Additionally, we conduct an ablation study on the custom dataset and the Core50 dataset to highlight the contributions of different components of the algorithm. To further demonstrate the robustness of the algorithm, we perform a real-time experiment for object classification using the soft gripper and an external independent camera setup, all synchronized with the Robot Operating System (ROS) framework.

Title: On the Feasibility of Fully AI-automated Vishing Attacks

• Authors: João Figueiredo, Afonso Carvalho, Daniel Castro, Daniel Gonçalves, Nuno Santos
• Subjects: cs.CR, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.13793
• Pdf URL: https://arxiv.org/pdf/2409.13793
• Copy Paste: [[2409.13793]] On the Feasibility of Fully AI-automated Vishing Attacks(https://arxiv.org/abs/2409.13793)
• Keywords: security, protect, attack, large language model
• Abstract: A vishing attack is a form of social engineering where attackers use phone calls to deceive individuals into disclosing sensitive information, such as personal data, financial information, or security credentials. Attackers exploit the perceived urgency and authenticity of voice communication to manipulate victims, often posing as legitimate entities like banks or tech support. Vishing is a particularly serious threat as it bypasses security controls designed to protect information. In this work, we study the potential for vishing attacks to escalate with the advent of AI. In theory, AI-powered software bots may have the ability to automate these attacks by initiating conversations with potential victims via phone calls and deceiving them into disclosing sensitive information. To validate this thesis, we introduce ViKing, an AI-powered vishing system developed using publicly available AI technology. It relies on a Large Language Model (LLM) as its core cognitive processor to steer conversations with victims, complemented by a pipeline of speech-to-text and text-to-speech modules that facilitate audio-text conversion in phone calls. Through a controlled social experiment involving 240 participants, we discovered that ViKing has successfully persuaded many participants to reveal sensitive information, even those who had been explicitly warned about the risk of vishing campaigns. Interactions with ViKing's bots were generally considered realistic. From these findings, we conclude that tools like ViKing may already be accessible to potential malicious actors, while also serving as an invaluable resource for cyber awareness programs.

Title: ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer

• Authors: Shihua Sun, Kenechukwu Nwodo, Shridatt Sugrim, Angelos Stavrou, Haining Wang
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.13828
• Pdf URL: https://arxiv.org/pdf/2409.13828
• Copy Paste: [[2409.13828]] ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer(https://arxiv.org/abs/2409.13828)
• Keywords: attack, robust, transformer
• Abstract: The use of transformers for vision tasks has challenged the traditional dominant role of convolutional neural networks (CNN) in computer vision (CV). For image classification tasks, Vision Transformer (ViT) effectively establishes spatial relationships between patches within images, directing attention to important areas for accurate predictions. However, similar to CNNs, ViTs are vulnerable to adversarial attacks, which mislead the image classifier into making incorrect decisions on images with carefully designed perturbations. Moreover, adversarial patch attacks, which introduce arbitrary perturbations within a small area, pose a more serious threat to ViTs. Even worse, traditional detection methods, originally designed for CNN models, are impractical or suffer significant performance degradation when applied to ViTs, and they generally overlook patch attacks. In this paper, we propose ViTGuard as a general detection method for defending ViT models against adversarial attacks, including typical attacks where perturbations spread over the entire input and patch attacks. ViTGuard uses a Masked Autoencoder (MAE) model to recover randomly masked patches from the unmasked regions, providing a flexible image reconstruction strategy. Then, threshold-based detectors leverage distinctive ViT features, including attention maps and classification (CLS) token representations, to distinguish between normal and adversarial samples. The MAE model does not involve any adversarial samples during training, ensuring the effectiveness of our detectors against unseen attacks. ViTGuard is compared with seven existing detection methods under nine attacks across three datasets. The evaluation results show the superiority of ViTGuard over existing detectors. Finally, considering the potential detection evasion, we further demonstrate ViTGuard's robustness against adaptive attacks for evasion.

Title: Measuring Copyright Risks of Large Language Model via Partial Information Probing

• Authors: Weijie Zhao, Huajie Shao, Zhaozhuo Xu, Suzhen Duan, Denghui Zhang
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.13831
• Pdf URL: https://arxiv.org/pdf/2409.13831
• Copy Paste: [[2409.13831]] Measuring Copyright Risks of Large Language Model via Partial Information Probing(https://arxiv.org/abs/2409.13831)
• Keywords: large language model
• Abstract: Exploring the data sources used to train Large Language Models (LLMs) is a crucial direction in investigating potential copyright infringement by these models. While this approach can identify the possible use of copyrighted materials in training data, it does not directly measure infringing risks. Recent research has shifted towards testing whether LLMs can directly output copyrighted content. Addressing this direction, we investigate and assess LLMs' capacity to generate infringing content by providing them with partial information from copyrighted materials, and try to use iterative prompting to get LLMs to generate more infringing content. Specifically, we input a portion of a copyrighted text into LLMs, prompt them to complete it, and then analyze the overlap between the generated content and the original copyrighted material. Our findings demonstrate that LLMs can indeed generate content highly overlapping with copyrighted materials based on these partial inputs.

Title: STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions

• Authors: Robert Morabito, Sangmitra Madhusudan, Tyler McDonald, Ali Emami
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.13843
• Pdf URL: https://arxiv.org/pdf/2409.13843
• Copy Paste: [[2409.13843]] STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions(https://arxiv.org/abs/2409.13843)
• Keywords: fair, large language model
• Abstract: Mitigating explicit and implicit biases in Large Language Models (LLMs) has become a critical focus in the field of natural language processing. However, many current methodologies evaluate scenarios in isolation, without considering the broader context or the spectrum of potential biases within each situation. To address this, we introduce the Sensitivity Testing on Offensive Progressions (STOP) dataset, which includes 450 offensive progressions containing 2,700 unique sentences of varying severity that progressively escalate from less to more explicitly offensive. Covering a broad spectrum of 9 demographics and 46 sub-demographics, STOP ensures inclusivity and comprehensive coverage. We evaluate several leading closed- and open-source models, including GPT-4, Mixtral, and Llama 3. Our findings reveal that even the best-performing models detect bias inconsistently, with success rates ranging from 19.3% to 69.8%. We also demonstrate how aligning models with human judgments on STOP can improve model answer rates on sensitive tasks such as BBQ, StereoSet, and CrowS-Pairs by up to 191%, while maintaining or even improving performance. STOP presents a novel framework for assessing the complex nature of biases in LLMs, which will enable more effective bias mitigation strategies and facilitates the creation of fairer language models.

Title: Multi-Modality Conditioned Variational U-Net for Field-of-View Extension in Brain Diffusion MRI

• Authors: Zhiyuan Li, Tianyuan Yao, Praitayini Kanakaraj, Chenyu Gao, Shunxing Bao, Lianrui Zuo, Michael E. Kim, Nancy R. Newlin, Gaurav Rudravaram, Nazirah M. Khairi, Yuankai Huo, Kurt G. Schilling, Walter A. Kukull, Arthur W. Toga, Derek B. Archer, Timothy J. Hohman, Bennett A. Landman
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13846
• Pdf URL: https://arxiv.org/pdf/2409.13846
• Copy Paste: [[2409.13846]] Multi-Modality Conditioned Variational U-Net for Field-of-View Extension in Brain Diffusion MRI(https://arxiv.org/abs/2409.13846)
• Keywords: diffusion, generative
• Abstract: An incomplete field-of-view (FOV) in diffusion magnetic resonance imaging (dMRI) can severely hinder the volumetric and bundle analyses of whole-brain white matter connectivity. Although existing works have investigated imputing the missing regions using deep generative models, it remains unclear how to specifically utilize additional information from paired multi-modality data and whether this can enhance the imputation quality and be useful for downstream tractography. To fill this gap, we propose a novel framework for imputing dMRI scans in the incomplete part of the FOV by integrating the learned diffusion features in the acquired part of the FOV to the complete brain anatomical structure. We hypothesize that by this design the proposed framework can enhance the imputation performance of the dMRI scans and therefore be useful for repairing whole-brain tractography in corrupted dMRI scans with incomplete FOV. We tested our framework on two cohorts from different sites with a total of 96 subjects and compared it with a baseline imputation method that treats the information from T1w and dMRI scans equally. The proposed framework achieved significant improvements in imputation performance, as demonstrated by angular correlation coefficient (p < 1E-5), and in downstream tractography accuracy, as demonstrated by Dice score (p < 0.01). Results suggest that the proposed framework improved imputation performance in dMRI scans by specifically utilizing additional information from paired multi-modality data, compared with the baseline method. The imputation achieved by the proposed framework enhances whole brain tractography, and therefore reduces the uncertainty when analyzing bundles associated with neurodegenerative.

Title: Unlocking Memorization in Large Language Models with Dynamic Soft Prompting

• Authors: Zhepeng Wang, Runxue Bao, Yawen Wu, Jackson Taylor, Cao Xiao, Feng Zheng, Weiwen Jiang, Shangqian Gao, Yanfu Zhang
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13853
• Pdf URL: https://arxiv.org/pdf/2409.13853
• Copy Paste: [[2409.13853]] Unlocking Memorization in Large Language Models with Dynamic Soft Prompting(https://arxiv.org/abs/2409.13853)
• Keywords: security, privacy, extraction, transformer, large language model
• Abstract: Pretrained large language models (LLMs) have revolutionized natural language processing (NLP) tasks such as summarization, question answering, and translation. However, LLMs pose significant security risks due to their tendency to memorize training data, leading to potential privacy breaches and copyright infringement. Accurate measurement of this memorization is essential to evaluate and mitigate these potential risks. However, previous attempts to characterize memorization are constrained by either using prefixes only or by prepending a constant soft prompt to the prefixes, which cannot react to changes in input. To address this challenge, we propose a novel method for estimating LLM memorization using dynamic, prefix-dependent soft prompts. Our approach involves training a transformer-based generator to produce soft prompts that adapt to changes in input, thereby enabling more accurate extraction of memorized data. Our method not only addresses the limitations of previous methods but also demonstrates superior performance in diverse experimental settings compared to state-of-the-art techniques. In particular, our method can achieve the maximum relative improvement of 112.75% and 32.26% over the vanilla baseline in terms of discoverable memorization rate for the text generation task and code generation task respectively.

Title: Wormhole: Concept-Aware Deep Representation Learning for Co-Evolving Sequences

• Authors: Kunpeng Xu, Lifei Chen, Shengrui Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13857
• Pdf URL: https://arxiv.org/pdf/2409.13857
• Copy Paste: [[2409.13857]] Wormhole: Concept-Aware Deep Representation Learning for Co-Evolving Sequences(https://arxiv.org/abs/2409.13857)
• Keywords: robust, interpretability
• Abstract: Identifying and understanding dynamic concepts in co-evolving sequences is crucial for analyzing complex systems such as IoT applications, financial markets, and online activity logs. These concepts provide valuable insights into the underlying structures and behaviors of sequential data, enabling better decision-making and forecasting. This paper introduces Wormhole, a novel deep representation learning framework that is concept-aware and designed for co-evolving time sequences. Our model presents a self-representation layer and a temporal smoothness constraint to ensure robust identification of dynamic concepts and their transitions. Additionally, concept transitions are detected by identifying abrupt changes in the latent space, signifying a shift to new behavior - akin to passing through a wormhole. This novel mechanism accurately discerns concepts within co-evolving sequences and pinpoints the exact locations of these wormholes, enhancing the interpretability of the learned representations. Experiments demonstrate that this method can effectively segment time series data into meaningful concepts, providing a valuable tool for analyzing complex temporal patterns and advancing the detection of concept drifts.

Title: SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation

• Authors: Maying Shen, Nadine Chang, Sifei Liu, Jose M. Alvarez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.13860
• Pdf URL: https://arxiv.org/pdf/2409.13860
• Copy Paste: [[2409.13860]] SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation(https://arxiv.org/abs/2409.13860)
• Keywords: explainability
• Abstract: In recent years, the data collected for artificial intelligence has grown to an unmanageable amount. Particularly within industrial applications, such as autonomous vehicles, model training computation budgets are being exceeded while model performance is saturating -- and yet more data continues to pour in. To navigate the flood of data, we propose a framework to select the most semantically diverse and important dataset portion. Then, we further semantically enrich it by discovering meaningful new data from a massive unlabeled data pool. Importantly, we can provide explainability by leveraging foundation models to generate semantics for every data point. We quantitatively show that our Semantic Selection and Enrichment framework (SSE) can a) successfully maintain model performance with a smaller training dataset and b) improve model performance by enriching the smaller dataset without exceeding the original dataset size. Consequently, we demonstrate that semantic diversity is imperative for optimal data selection and model performance.

Title: Persistent Backdoor Attacks in Continual Learning

• Authors: Zhen Guo, Abhinav Kumar, Reza Tourani
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.13864
• Pdf URL: https://arxiv.org/pdf/2409.13864
• Copy Paste: [[2409.13864]] Persistent Backdoor Attacks in Continual Learning(https://arxiv.org/abs/2409.13864)
• Keywords: defense, attack
• Abstract: Backdoor attacks pose a significant threat to neural networks, enabling adversaries to manipulate model outputs on specific inputs, often with devastating consequences, especially in critical applications. While backdoor attacks have been studied in various contexts, little attention has been given to their practicality and persistence in continual learning, particularly in understanding how the continual updates to model parameters, as new data distributions are learned and integrated, impact the effectiveness of these attacks over time. To address this gap, we introduce two persistent backdoor attacks-Blind Task Backdoor and Latent Task Backdoor-each leveraging minimal adversarial influence. Our blind task backdoor subtly alters the loss computation without direct control over the training process, while the latent task backdoor influences only a single task's training, with all other tasks trained benignly. We evaluate these attacks under various configurations, demonstrating their efficacy with static, dynamic, physical, and semantic triggers. Our results show that both attacks consistently achieve high success rates across different continual learning algorithms, while effectively evading state-of-the-art defenses, such as SentiNet and I-BAU.

Title: Data Distribution Shifts in (Industrial) Federated Learning as a Privacy Issue

• Authors: David Brunner, Alessio Montuoro
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.13875
• Pdf URL: https://arxiv.org/pdf/2409.13875
• Copy Paste: [[2409.13875]] Data Distribution Shifts in (Industrial) Federated Learning as a Privacy Issue(https://arxiv.org/abs/2409.13875)
• Keywords: privacy, protect, attack, federate
• Abstract: We consider industrial federated learning, a collaboration between a small number of powerful, potentially competing industrial players, mediated by a third party aspiring to improve the service it provides to its customers. We argue that this configuration harbours covert privacy risks that do not arise in e.g. cross-device settings. Companies are very protective of their intellectual property and production processes. Information about changes to their production and the timing of which is to be kept private. We study a scenario in which one of the collaborators infers changes to their competitors' production by detecting potentially subtle temporal data distribution shifts. In this framing, a data distribution shift is always problematic, even if it has no negative effect on training convergence. Thus, our goal is to find means that allow the detection of distributional shifts better than customary evaluation metrics. Based on the assumption that even minor shifts translate into the collaboratively learned machine learning model, the attacker tracks the shared models' internal state with a selection of metrics from literature in order to pick up on relevant changes. In an empirical study on benchmark datasets, we show an honest-but-curious attacker to be capable of detecting subtle distributional shifts on other clients, in some cases long before they become obvious in evaluation.

Title: Achieving Predictive Precision: Leveraging LSTM and Pseudo Labeling for Volvo's Discovery Challenge at ECML-PKDD 2024

• Authors: Carlo Metta, Marco Gregnanin, Andrea Papini, Silvia Giulia Galfrè, Andrea Fois, Francesco Morandin, Marco Fantozzi, Maurizio Parton
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13877
• Pdf URL: https://arxiv.org/pdf/2409.13877
• Copy Paste: [[2409.13877]] Achieving Predictive Precision: Leveraging LSTM and Pseudo Labeling for Volvo's Discovery Challenge at ECML-PKDD 2024(https://arxiv.org/abs/2409.13877)
• Keywords: robust
• Abstract: This paper presents the second-place methodology in the Volvo Discovery Challenge at ECML-PKDD 2024, where we used Long Short-Term Memory networks and pseudo-labeling to predict maintenance needs for a component of Volvo trucks. We processed the training data to mirror the test set structure and applied a base LSTM model to label the test data iteratively. This approach refined our model's predictive capabilities and culminated in a macro-average F1-score of 0.879, demonstrating robust performance in predictive maintenance. This work provides valuable insights for applying machine learning techniques effectively in industrial settings.

Title: "I Never Said That": A dataset, taxonomy and baselines on response clarity classification

• Authors: Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, Giorgos Stamou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13879
• Pdf URL: https://arxiv.org/pdf/2409.13879
• Copy Paste: [[2409.13879]] "I Never Said That": A dataset, taxonomy and baselines on response clarity classification(https://arxiv.org/abs/2409.13879)
• Keywords: large language model
• Abstract: Equivocation and ambiguity in public speech are well-studied discourse phenomena, especially in political science and analysis of political interviews. Inspired by the well-grounded theory on equivocation, we aim to resolve the closely related problem of response clarity in questions extracted from political interviews, leveraging the capabilities of Large Language Models (LLMs) and human expertise. To this end, we introduce a novel taxonomy that frames the task of detecting and classifying response clarity and a corresponding clarity classification dataset which consists of question-answer (QA) pairs drawn from political interviews and annotated accordingly. Our proposed two-level taxonomy addresses the clarity of a response in terms of the information provided for a given question (high-level) and also provides a fine-grained taxonomy of evasion techniques that relate to unclear, ambiguous responses (lower-level). We combine ChatGPT and human annotators to collect, validate and annotate discrete QA pairs from political interviews, to be used for our newly introduced response clarity task. We provide a detailed analysis and conduct several experiments with different model architectures, sizes and adaptation methods to gain insights and establish new baselines over the proposed dataset and task.

Title: Tabular Data Generation using Binary Diffusion

• Authors: Vitaliy Kinakh, Slava Voloshynovskiy
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13882
• Pdf URL: https://arxiv.org/pdf/2409.13882
• Copy Paste: [[2409.13882]] Tabular Data Generation using Binary Diffusion(https://arxiv.org/abs/2409.13882)
• Keywords: diffusion, generative
• Abstract: Generating synthetic tabular data is critical in machine learning, especially when real data is limited or sensitive. Traditional generative models often face challenges due to the unique characteristics of tabular data, such as mixed data types and varied distributions, and require complex preprocessing or large pretrained models. In this paper, we introduce a novel, lossless binary transformation method that converts any tabular data into fixed-size binary representations, and a corresponding new generative model called Binary Diffusion, specifically designed for binary data. Binary Diffusion leverages the simplicity of XOR operations for noise addition and removal and employs binary cross-entropy loss for training. Our approach eliminates the need for extensive preprocessing, complex noise parameter tuning, and pretraining on large datasets. We evaluate our model on several popular tabular benchmark datasets, demonstrating that Binary Diffusion outperforms existing state-of-the-art models on Travel, Adult Income, and Diabetes datasets while being significantly smaller in size.

Title: A Multi-LLM Debiasing Framework

• Authors: Deonna M. Owens, Ryan A. Rossi, Sungchul Kim, Tong Yu, Franck Dernoncourt, Xiang Chen, Ruiyi Zhang, Jiuxiang Gu, Hanieh Deilamsalehy, Nedim Lipka
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13884
• Pdf URL: https://arxiv.org/pdf/2409.13884
• Copy Paste: [[2409.13884]] A Multi-LLM Debiasing Framework(https://arxiv.org/abs/2409.13884)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are powerful tools with the potential to benefit society immensely, yet, they have demonstrated biases that perpetuate societal inequalities. Despite significant advancements in bias mitigation techniques using data augmentation, zero-shot prompting, and model fine-tuning, biases continuously persist, including subtle biases that may elude human detection. Recent research has shown a growing interest in multi-LLM approaches, which have been demonstrated to be effective in improving the quality of reasoning and factuality in LLMs. Building on this approach, we propose a novel multi-LLM debiasing framework aimed at reducing bias in LLMs. Our work is the first to introduce and evaluate two distinct approaches within this framework for debiasing LLMs: a centralized method, where the conversation is facilitated by a single central LLM, and a decentralized method, where all models communicate directly. Our findings reveal that our multi-LLM framework significantly reduces bias in LLMs, outperforming the baseline method across several social groups.

Title: Causal Feature Selection Method for Contextual Multi-Armed Bandits in Recommender System

• Authors: Zhenyu Zhao, Yexi Jiang
• Subjects: cs.LG, cs.IR, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.13888
• Pdf URL: https://arxiv.org/pdf/2409.13888
• Copy Paste: [[2409.13888]] Causal Feature Selection Method for Contextual Multi-Armed Bandits in Recommender System(https://arxiv.org/abs/2409.13888)
• Keywords: interpretability
• Abstract: Features (a.k.a. context) are critical for contextual multi-armed bandits (MAB) performance. In practice of large scale online system, it is important to select and implement important features for the model: missing important features can led to sub-optimal reward outcome, and including irrelevant features can cause overfitting, poor model interpretability, and implementation cost. However, feature selection methods for conventional machine learning models fail short for contextual MAB use cases, as conventional methods select features correlated with the outcome variable, but not necessarily causing heterogeneuous treatment effect among arms which are truely important for contextual MAB. In this paper, we introduce model-free feature selection methods designed for contexutal MAB problem, based on heterogeneous causal effect contributed by the feature to the reward distribution. Empirical evaluation is conducted based on synthetic data as well as real data from an online experiment for optimizing content cover image in a recommender system. The results show this feature selection method effectively selects the important features that lead to higher contextual MAB reward than unimportant features. Compared with model embedded method, this model-free method has advantage of fast computation speed, ease of implementation, and prune of model mis-specification issues.

Title: Transfer Learning with Clinical Concept Embeddings from Large Language Models

• Authors: Yuhe Gao, Runxue Bao, Yuelyu Ji, Yiming Sun, Chenxi Song, Jeffrey P. Ferraro, Ye Ye
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13893
• Pdf URL: https://arxiv.org/pdf/2409.13893
• Copy Paste: [[2409.13893]] Transfer Learning with Clinical Concept Embeddings from Large Language Models(https://arxiv.org/abs/2409.13893)
• Keywords: large language model
• Abstract: Knowledge sharing is crucial in healthcare, especially when leveraging data from multiple clinical sites to address data scarcity, reduce costs, and enable timely interventions. Transfer learning can facilitate cross-site knowledge transfer, but a major challenge is heterogeneity in clinical concepts across different sites. Large Language Models (LLMs) show significant potential of capturing the semantic meaning of clinical concepts and reducing heterogeneity. This study analyzed electronic health records from two large healthcare systems to assess the impact of semantic embeddings from LLMs on local, shared, and transfer learning models. Results indicate that domain-specific LLMs, such as Med-BERT, consistently outperform in local and direct transfer scenarios, while generic models like OpenAI embeddings require fine-tuning for optimal performance. However, excessive tuning of models with biomedical embeddings may reduce effectiveness, emphasizing the need for balance. This study highlights the importance of domain-specific embeddings and careful model tuning for effective knowledge transfer in healthcare.

Title: LLM for Everyone: Representing the Underrepresented in Large Language Models

• Authors: Samuel Cahyawijaya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13897
• Pdf URL: https://arxiv.org/pdf/2409.13897
• Copy Paste: [[2409.13897]] LLM for Everyone: Representing the Underrepresented in Large Language Models(https://arxiv.org/abs/2409.13897)
• Keywords: large language model
• Abstract: Natural language processing (NLP) has witnessed a profound impact of large language models (LLMs) that excel in a multitude of tasks. However, the limitation of LLMs in multilingual settings, particularly in underrepresented languages, remains a significant hurdle. This thesis aims to bridge the gap in NLP research and development by focusing on underrepresented languages. A comprehensive evaluation of LLMs is conducted to assess their capabilities in these languages, revealing the challenges of multilingual and multicultural generalization. Addressing the multilingual generalization gap, this thesis proposes data-and-compute-efficient methods to mitigate the disparity in LLM ability in underrepresented languages, allowing better generalization on underrepresented languages without the loss of task generalization ability. The proposed solutions cover cross-lingual continual instruction tuning, retrieval-based cross-lingual in-context learning, and in-context query alignment. Furthermore, a novel method to measure cultural values alignment between LLMs operating in different languages is proposed, ensuring cultural sensitivity and inclusivity. These contributions aim to enhance the multilingual and multicultural alignment of LLMs in underrepresented languages, ultimately advancing the NLP field toward greater equality and inclusiveness.

Title: Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology

• Authors: Aidan Gilson, Xuguang Ai, Thilaka Arunachalam, Ziyou Chen, Ki Xiong Cheong, Amisha Dave, Cameron Duic, Mercy Kibe, Annette Kaminaka, Minali Prasad, Fares Siddig, Maxwell Singer, Wendy Wong, Qiao Jin, Tiarnan D.L. Keenan, Xia Hu, Emily Y. Chew, Zhiyong Lu, Hua Xu, Ron A. Adelman, Yih-Chung Tham, Qingyu Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13902
• Pdf URL: https://arxiv.org/pdf/2409.13902
• Copy Paste: [[2409.13902]] Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology(https://arxiv.org/abs/2409.13902)
• Keywords: large language model
• Abstract: Despite the potential of Large Language Models (LLMs) in medicine, they may generate responses lacking supporting evidence or based on hallucinated evidence. While Retrieval Augment Generation (RAG) is popular to address this issue, few studies implemented and evaluated RAG in downstream domain-specific applications. We developed a RAG pipeline with 70,000 ophthalmology-specific documents that retrieve relevant documents to augment LLMs during inference time. In a case study on long-form consumer health questions, we systematically evaluated the responses including over 500 references of LLMs with and without RAG on 100 questions with 10 healthcare professionals. The evaluation focuses on factuality of evidence, selection and ranking of evidence, attribution of evidence, and answer accuracy and completeness. LLMs without RAG provided 252 references in total. Of which, 45.3% hallucinated, 34.1% consisted of minor errors, and 20.6% were correct. In contrast, LLMs with RAG significantly improved accuracy (54.5% being correct) and reduced error rates (18.8% with minor hallucinations and 26.7% with errors). 62.5% of the top 10 documents retrieved by RAG were selected as the top references in the LLM response, with an average ranking of 4.9. The use of RAG also improved evidence attribution (increasing from 1.85 to 2.49 on a 5-point scale, P<0.001), albeit with slight decreases in accuracy (from 3.52 to 3.23, P=0.03) and completeness (from 3.47 to 3.27, P=0.17). The results demonstrate that LLMs frequently exhibited hallucinated and erroneous evidence in the responses, raising concerns for downstream applications in the medical domain. RAG substantially reduced the proportion of such evidence but encountered challenges.

Title: One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks

• Authors: Sebastian Nehrdich, Oliver Hellwig, Kurt Keutzer
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13920
• Pdf URL: https://arxiv.org/pdf/2409.13920
• Copy Paste: [[2409.13920]] One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks(https://arxiv.org/abs/2409.13920)
• Keywords: robust, segmentation
• Abstract: Morphologically rich languages are notoriously challenging to process for downstream NLP applications. This paper presents a new pretrained language model, ByT5-Sanskrit, designed for NLP applications involving the morphologically rich language Sanskrit. We evaluate ByT5-Sanskrit on established Sanskrit word segmentation tasks, where it outperforms previous data-driven approaches by a considerable margin and matches the performance of the current best lexicon-based model. It is easier to deploy and more robust to data not covered by external linguistic resources. It also achieves new state-of-the-art results in Vedic Sanskrit dependency parsing and OCR post-correction tasks. Additionally, based on the Digital Corpus of Sanskrit, we introduce a novel multitask dataset for the joint training of Sanskrit word segmentation, lemmatization, and morphosyntactic tagging tasks. We fine-tune ByT5-Sanskrit on this dataset, creating a versatile multitask model for various downstream Sanskrit applications. We have used this model in Sanskrit linguistic annotation projects, in information retrieval setups, and as a preprocessing step in a Sanskrit machine translation pipeline. We also show that our approach yields new best scores for lemmatization and dependency parsing of other morphologically rich languages. We thus demonstrate that byte-level pretrained language models can achieve excellent performance for morphologically rich languages, outperforming tokenizer-based models and presenting an important vector of exploration when constructing NLP pipelines for such languages.

Title: On-device Collaborative Language Modeling via a Mixture of Generalists and Specialists

• Authors: Dongyang Fan, Bettina Messmer, Martin Jaggi
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13931
• Pdf URL: https://arxiv.org/pdf/2409.13931
• Copy Paste: [[2409.13931]] On-device Collaborative Language Modeling via a Mixture of Generalists and Specialists(https://arxiv.org/abs/2409.13931)
• Keywords: large language model
• Abstract: We target on-device collaborative fine-tuning of Large Language Models (LLMs) by adapting a Mixture of Experts (MoE) architecture, where experts are Low-Rank Adaptation (LoRA) modules. In conventional MoE approaches, experts develop into specialists throughout training. In contrast, we propose a novel $\textbf{Co}$llaborative learning approach via a $\textbf{Mi}$xture of $\textbf{G}$eneralists and $\textbf{S}$pecialists (CoMiGS). Diversifying into the two roles is achieved by aggregating certain experts globally while keeping others localized to specialize in user-specific datasets. Central to our work is a learnable routing network that routes at a token level, balancing collaboration and personalization at the finest granularity. Our method consistently demonstrates superior performance in scenarios with high data heterogeneity across various datasets. By design, our approach accommodates varying computational resource constraints among users as shown in different numbers of LoRA experts. We further showcase that low-resourced users can benefit from high-resourced users with high data quantity.

Title: MirrorStories: Reflecting Diversity through Personalized Narrative Generation with Large Language Models

• Authors: Sarfaroz Yunusov, Hamza Sidat, Ali Emami
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.13935
• Pdf URL: https://arxiv.org/pdf/2409.13935
• Copy Paste: [[2409.13935]] MirrorStories: Reflecting Diversity through Personalized Narrative Generation with Large Language Models(https://arxiv.org/abs/2409.13935)
• Keywords: large language model
• Abstract: This study explores the effectiveness of Large Language Models (LLMs) in creating personalized "mirror stories" that reflect and resonate with individual readers' identities, addressing the significant lack of diversity in literature. We present MirrorStories, a corpus of 1,500 personalized short stories generated by integrating elements such as name, gender, age, ethnicity, reader interest, and story moral. We demonstrate that LLMs can effectively incorporate diverse identity elements into narratives, with human evaluators identifying personalized elements in the stories with high accuracy. Through a comprehensive evaluation involving 26 diverse human judges, we compare the effectiveness of MirrorStories against generic narratives. We find that personalized LLM-generated stories not only outscore generic human-written and LLM-generated ones across all metrics of engagement (with average ratings of 4.22 versus 3.37 on a 5-point scale), but also achieve higher textual diversity while preserving the intended moral. We also provide analyses that include bias assessments and a study on the potential for integrating images into personalized stories.

Title: High-Resolution Flood Probability Mapping Using Generative Machine Learning with Large-Scale Synthetic Precipitation and Inundation Data

• Authors: Lipai Huang, Federico Antolini, Ali Mostafavi, Russell Blessing, Matthew Garcia, Samuel D. Brody
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.13936
• Pdf URL: https://arxiv.org/pdf/2409.13936
• Copy Paste: [[2409.13936]] High-Resolution Flood Probability Mapping Using Generative Machine Learning with Large-Scale Synthetic Precipitation and Inundation Data(https://arxiv.org/abs/2409.13936)
• Keywords: generative
• Abstract: High-resolution flood probability maps are essential for addressing the limitations of existing flood risk assessment approaches but are often limited by the availability of historical event data. Also, producing simulated data needed for creating probabilistic flood maps using physics-based models involves significant computation and time effort inhibiting the feasibility. To address this gap, this study introduces Flood-Precip GAN (Flood-Precipitation Generative Adversarial Network), a novel methodology that leverages generative machine learning to simulate large-scale synthetic inundation data to produce probabilistic flood maps. With a focus on Harris County, Texas, Flood-Precip GAN begins with training a cell-wise depth estimator using a limited number of physics-based model-generated precipitation-flood events. This model, which emphasizes precipitation-based features, outperforms universal models. Subsequently, a Generative Adversarial Network (GAN) with constraints is employed to conditionally generate synthetic precipitation records. Strategic thresholds are established to filter these records, ensuring close alignment with true precipitation patterns. For each cell, synthetic events are smoothed using a K-nearest neighbors algorithm and processed through the depth estimator to derive synthetic depth distributions. By iterating this procedure and after generating 10,000 synthetic precipitation-flood events, we construct flood probability maps in various formats, considering different inundation depths. Validation through similarity and correlation metrics confirms the fidelity of the synthetic depth distributions relative to true data. Flood-Precip GAN provides a scalable solution for generating synthetic flood depth data needed to create high-resolution flood probability maps, significantly enhancing flood preparedness and mitigation efforts.

Title: Lightweight and Resilient Signatures for Cloud-Assisted Embedded IoT Systems

• Authors: Saif E. Nouma, Attila A. Yavuz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.13937
• Pdf URL: https://arxiv.org/pdf/2409.13937
• Copy Paste: [[2409.13937]] Lightweight and Resilient Signatures for Cloud-Assisted Embedded IoT Systems(https://arxiv.org/abs/2409.13937)
• Keywords: secure, security, attack
• Abstract: Digital signatures provide scalable authentication with non-repudiation and are vital tools for the Internet of Things (IoT). Many IoT applications harbor vast quantities of resource-limited devices often used with cloud computing. However, key compromises (e.g., physical, malware) pose a significant threat to IoTs due to increased attack vectors and open operational environments. Forward security and distributed key management are critical breach-resilient countermeasures to mitigate such threats. Yet forward-secure signatures are exorbitantly costly for low-end IoTs, while cloud-assisted approaches suffer from centrality or non-colluding semi-honest servers. In this work, we create two novel digital signatures called Lightweight and Resilient Signatures with Hardware Assistance (LRSHA) and its Forward-secure version (FLRSHA). They offer a near-optimally efficient signing with small keys and signature sizes. We synergize various design strategies, such as commitment separation to eliminate costly signing operations and hardware-assisted distributed servers to enable breach-resilient verification. Our schemes achieve magnitudes of faster forward-secure signing and compact key/signature sizes without suffering from strong security assumptions (non-colluding, central servers) or a heavy burden on the verifier (extreme storage, computation). We formally prove the security of our schemes and validate their performance with full-fledged open-source implementations on both commodity hardware and 8-bit AVR microcontrollers.

Title: TalkMosaic: Interactive PhotoMosaic with Multi-modal LLM Q&A Interactions

• Authors: Kevin Li, Fulu Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13941
• Pdf URL: https://arxiv.org/pdf/2409.13941
• Copy Paste: [[2409.13941]] TalkMosaic: Interactive PhotoMosaic with Multi-modal LLM Q&A Interactions(https://arxiv.org/abs/2409.13941)
• Keywords: protect
• Abstract: We use images of cars of a wide range of varieties to compose an image of an animal such as a bird or a lion for the theme of environmental protection to maximize the information about cars in a single composed image and to raise the awareness about environmental challenges. We present a novel way of image interaction with an artistically-composed photomosaic image, in which a simple operation of "click and display" is used to demonstrate the interactive switch between a tile image in a photomosaic image and the corresponding original car image, which will be automatically saved on the Desktop. We build a multimodal custom GPT named TalkMosaic by incorporating car images information and the related knowledge to ChatGPT. By uploading the original car image to TalkMosaic, we can ask questions about the given car image and get the corresponding answers efficiently and effectively such as where to buy the tire in the car image that satisfies high environmental standards. We give an in-depth analysis on how to speed up the inference of multimodal LLM using sparse attention and quantization techniques with presented probabilistic FlashAttention (PrFlashAttention) and Staircase Adaptive Quantization (SAQ) methods. The implemented prototype demonstrates the feasibility and effectiveness of the presented approach.

Title: Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM

• Authors: Zheng Wei Lim, Nitish Gupta, Honglin Yu, Trevor Cohn
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13949
• Pdf URL: https://arxiv.org/pdf/2409.13949
• Copy Paste: [[2409.13949]] Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM(https://arxiv.org/abs/2409.13949)
• Keywords: robust, large language model
• Abstract: Multilingual large language models (LLMs) are great translators, but this is largely limited to high-resource languages. For many LLMs, translating in and out of low-resource languages remains a challenging task. To maximize data efficiency in this low-resource setting, we introduce Mufu, which includes a selection of automatically generated multilingual candidates and an instruction to correct inaccurate translations in the prompt. Mufu prompts turn a translation task into a postediting one, and seek to harness the LLM's reasoning capability with auxiliary translation candidates, from which the model is required to assess the input quality, align the semantics cross-lingually, copy from relevant inputs and override instances that are incorrect. Our experiments on En-XX translations over the Flores-200 dataset show LLMs finetuned against Mufu-style prompts are robust to poor quality auxiliary translation candidates, achieving performance superior to NLLB 1.3B distilled model in 64% of low- and very-low-resource language pairs. We then distill these models to reduce inference cost, while maintaining on average 3.1 chrF improvement over finetune-only baseline in low-resource translations.

Title: Deep learning for fast segmentation and critical dimension metrology & characterization enabling AR/VR design and fabrication

• Authors: Kundan Chaudhary, Subhei Shaar, Raja Muthinti
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.13951
• Pdf URL: https://arxiv.org/pdf/2409.13951
• Copy Paste: [[2409.13951]] Deep learning for fast segmentation and critical dimension metrology & characterization enabling AR/VR design and fabrication(https://arxiv.org/abs/2409.13951)
• Keywords: extraction, segmentation
• Abstract: Quantitative analysis of microscopy images is essential in the design and fabrication of components used in augmented reality/virtual reality (AR/VR) modules. However, segmenting regions of interest (ROIs) from these complex images and extracting critical dimensions (CDs) requires novel techniques, such as deep learning models which are key for actionable decisions on process, material and device optimization. In this study, we report on the fine-tuning of a pre-trained Segment Anything Model (SAM) using a diverse dataset of electron microscopy images. We employed methods such as low-rank adaptation (LoRA) to reduce training time and enhance the accuracy of ROI extraction. The model's ability to generalize to unseen images facilitates zero-shot learning and supports a CD extraction model that precisely extracts CDs from the segmented ROIs. We demonstrate the accurate extraction of binary images from cross-sectional images of surface relief gratings (SRGs) and Fresnel lenses in both single and multiclass modes. Furthermore, these binary images are used to identify transition points, aiding in the extraction of relevant CDs. The combined use of the fine-tuned segmentation model and the CD extraction model offers substantial advantages to various industrial applications by enhancing analytical capabilities, time to data and insights, and optimizing manufacturing processes.

Title: Exploring Automated Keyword Mnemonics Generation with Large Language Models via Overgenerate-and-Rank

• Authors: Jaewook Lee, Hunter McNichols, Andrew Lan
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.13952
• Pdf URL: https://arxiv.org/pdf/2409.13952
• Copy Paste: [[2409.13952]] Exploring Automated Keyword Mnemonics Generation with Large Language Models via Overgenerate-and-Rank(https://arxiv.org/abs/2409.13952)
• Keywords: large language model
• Abstract: In this paper, we study an under-explored area of language and vocabulary learning: keyword mnemonics, a technique for memorizing vocabulary through memorable associations with a target word via a verbal cue. Typically, creating verbal cues requires extensive human effort and is quite time-consuming, necessitating an automated method that is more scalable. We propose a novel overgenerate-and-rank method via prompting large language models (LLMs) to generate verbal cues and then ranking them according to psycholinguistic measures and takeaways from a pilot user study. To assess cue quality, we conduct both an automated evaluation of imageability and coherence, as well as a human evaluation involving English teachers and learners. Results show that LLM-generated mnemonics are comparable to human-generated ones in terms of imageability, coherence, and perceived usefulness, but there remains plenty of room for improvement due to the diversity in background and preference among language learners.

Title: Monocular Event-Inertial Odometry with Adaptive decay-based Time Surface and Polarity-aware Tracking

• Authors: Kai Tang, Xiaolei Lang, Yukai Ma, Yuehao Huang, Laijian Li, Yong Liu, Jiajun Lv
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.13971
• Pdf URL: https://arxiv.org/pdf/2409.13971
• Copy Paste: [[2409.13971]] Monocular Event-Inertial Odometry with Adaptive decay-based Time Surface and Polarity-aware Tracking(https://arxiv.org/abs/2409.13971)
• Keywords: robust
• Abstract: Event cameras have garnered considerable attention due to their advantages over traditional cameras in low power consumption, high dynamic range, and no motion blur. This paper proposes a monocular event-inertial odometry incorporating an adaptive decay kernel-based time surface with polarity-aware tracking. We utilize an adaptive decay-based Time Surface to extract texture information from asynchronous events, which adapts to the dynamic characteristics of the event stream and enhances the representation of environmental textures. However, polarity-weighted time surfaces suffer from event polarity shifts during changes in motion direction. To mitigate its adverse effects on feature tracking, we optimize the feature tracking by incorporating an additional polarity-inverted time surface to enhance the robustness. Comparative analysis with visual-inertial and event-inertial odometry methods shows that our approach outperforms state-of-the-art techniques, with competitive results across various datasets.

Title: FracGM: A Fast Fractional Programming Technique for Geman-McClure Robust Estimator

• Authors: Bang-Shien Chen, Yu-Kai Lin, Jian-Yu Chen, Chih-Wei Huang, Jann-Long Chern, Ching-Cherng Sun
• Subjects: cs.CV, cs.RO, math.OC
• Abstract URL: https://arxiv.org/abs/2409.13978
• Pdf URL: https://arxiv.org/pdf/2409.13978
• Copy Paste: [[2409.13978]] FracGM: A Fast Fractional Programming Technique for Geman-McClure Robust Estimator(https://arxiv.org/abs/2409.13978)
• Keywords: robust
• Abstract: Robust estimation is essential in computer vision, robotics, and navigation, aiming to minimize the impact of outlier measurements for improved accuracy. We present a fast algorithm for Geman-McClure robust estimation, FracGM, leveraging fractional programming techniques. This solver reformulates the original non-convex fractional problem to a convex dual problem and a linear equation system, iteratively solving them in an alternating optimization pattern. Compared to graduated non-convexity approaches, this strategy exhibits a faster convergence rate and better outlier rejection capability. In addition, the global optimality of the proposed solver can be guaranteed under given conditions. We demonstrate the proposed FracGM solver with Wahba's rotation problem and 3-D point-cloud registration along with relaxation pre-processing and projection post-processing. Compared to state-of-the-art algorithms, when the outlier rates increase from 20\% to 80\%, FracGM shows 53\% and 88\% lower rotation and translation increases. In real-world scenarios, FracGM achieves better results in 13 out of 18 outcomes, while having a 19.43\% improvement in the computation time.

Title: Bias and Toxicity in Role-Play Reasoning

• Authors: Jinman Zhao, Zifan Qian, Linbo Cao, Yining Wang, Yitian Ding
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13979
• Pdf URL: https://arxiv.org/pdf/2409.13979
• Copy Paste: [[2409.13979]] Bias and Toxicity in Role-Play Reasoning(https://arxiv.org/abs/2409.13979)
• Keywords: large language model
• Abstract: Role-play in the Large Language Model (LLM) is a crucial technique that enables models to adopt specific perspectives, enhancing their ability to generate contextually relevant and accurate responses. By simulating different roles, theis approach improves reasoning capabilities across various NLP benchmarks, making the model's output more aligned with diverse scenarios. However, in this work, we demonstrate that role-play also carries potential risks. We systematically evaluate the impact of role-play by asking the language model to adopt different roles and testing it on multiple benchmarks that contain stereotypical and harmful questions. Despite the significant fluctuations in the benchmark results in different experiments, we find that applying role-play often increases the overall likelihood of generating stereotypical and harmful outputs.

Title: Enhancing Advanced Visual Reasoning Ability of Large Language Models

• Authors: Zhiyuan Li, Dongnan Liu, Chaoyi Zhang, Heng Wang, Tengfei Xue, Weidong Cai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13980
• Pdf URL: https://arxiv.org/pdf/2409.13980
• Copy Paste: [[2409.13980]] Enhancing Advanced Visual Reasoning Ability of Large Language Models(https://arxiv.org/abs/2409.13980)
• Keywords: robust, large language model
• Abstract: Recent advancements in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models' advanced reasoning ability. Traditional Vision-Language Models (VLMs) perform well in visual perception tasks while struggling with complex reasoning scenarios. Conversely, Large Language Models (LLMs) demonstrate robust text reasoning capabilities; however, they lack visual acuity. To bridge this gap, we propose Complex Visual Reasoning Large Language Models (CVR-LLM), capitalizing on VLMs' visual perception proficiency and LLMs' extensive reasoning capability. Unlike recent multimodal large language models (MLLMs) that require a projection layer, our approach transforms images into detailed, context-aware descriptions using an iterative self-refinement loop and leverages LLMs' text knowledge for accurate predictions without extra training. We also introduce a novel multi-modal in-context learning (ICL) methodology to enhance LLMs' contextual understanding and reasoning. Additionally, we introduce Chain-of-Comparison (CoC), a step-by-step comparison technique enabling contrasting various aspects of predictions. Our CVR-LLM presents the first comprehensive study across a wide array of complex visual reasoning tasks and achieves SOTA performance among all.

Title: CUS3D :CLIP-based Unsupervised 3D Segmentation via Object-level Denoise

• Authors: Fuyang Yu, Runze Tian, Zhen Wang, Xiaochuan Wang, Xiaohui Liang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.13982
• Pdf URL: https://arxiv.org/pdf/2409.13982
• Copy Paste: [[2409.13982]] CUS3D :CLIP-based Unsupervised 3D Segmentation via Object-level Denoise(https://arxiv.org/abs/2409.13982)
• Keywords: segmentation
• Abstract: To ease the difficulty of acquiring annotation labels in 3D data, a common method is using unsupervised and open-vocabulary semantic segmentation, which leverage 2D CLIP semantic knowledge. In this paper, unlike previous research that ignores the ``noise'' raised during feature projection from 2D to 3D, we propose a novel distillation learning framework named CUS3D. In our approach, an object-level denosing projection module is designed to screen out the ``noise'' and ensure more accurate 3D feature. Based on the obtained features, a multimodal distillation learning module is designed to align the 3D feature with CLIP semantic feature space with object-centered constrains to achieve advanced unsupervised semantic segmentation. We conduct comprehensive experiments in both unsupervised and open-vocabulary segmentation, and the results consistently showcase the superiority of our model in achieving advanced unsupervised segmentation results and its effectiveness in open-vocabulary segmentation.

Title: Enhanced Semantic Segmentation for Large-Scale and Imbalanced Point Clouds

• Authors: Haoran Gong, Haodong Wang, Di Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.13983
• Pdf URL: https://arxiv.org/pdf/2409.13983
• Copy Paste: [[2409.13983]] Enhanced Semantic Segmentation for Large-Scale and Imbalanced Point Clouds(https://arxiv.org/abs/2409.13983)
• Keywords: segmentation
• Abstract: Semantic segmentation of large-scale point clouds is of significant importance in environment perception and scene understanding. However, point clouds collected from real-world environments are usually imbalanced and small-sized objects are prone to be under-sampled or misclassified due to their low occurrence frequency, thereby reducing the overall accuracy of semantic segmentation. In this study, we propose the Multilateral Cascading Network (MCNet) for large-scale and sample-imbalanced point cloud scenes. To increase the frequency of small-sized objects, we introduce the semantic-weighted sampling module, which incorporates a probability parameter into the collected data group. To facilitate feature learning, we propose a Multilateral Cascading Attention Enhancement (MCAE) module to learn complex local features through multilateral cascading operations and attention mechanisms. To promote feature fusion, we propose a Point Cross Stage Partial (P-CSP) module to combine global and local features, optimizing the integration of valuable feature information across multiple scales. Finally, we introduce the neighborhood voting module to integrate results at the output layer. Our proposed method demonstrates either competitive or superior performance relative to state-of-the-art approaches across three widely recognized benchmark datasets: S3DIS, Toronto3D, and SensatUrban with mIoU scores of 74.0\%, 82.9\% and 64.5\%, respectively. Notably, our work yielded consistent optimal results on the under-sampled semantic categories, thereby demonstrating exceptional performance in the recognition of small-sized objects.

Title: Cycle-Consistency Uncertainty Estimation for Visual Prompting based One-Shot Defect Segmentation

• Authors: Geonuk Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.13984
• Pdf URL: https://arxiv.org/pdf/2409.13984
• Copy Paste: [[2409.13984]] Cycle-Consistency Uncertainty Estimation for Visual Prompting based One-Shot Defect Segmentation(https://arxiv.org/abs/2409.13984)
• Keywords: segmentation
• Abstract: Industrial defect detection traditionally relies on supervised learning models trained on fixed datasets of known defect types. While effective within a closed set, these models struggle with new, unseen defects, necessitating frequent re-labeling and re-training. Recent advances in visual prompting offer a solution by allowing models to adaptively infer novel categories based on provided visual cues. However, a prevalent issue in these methods is the over-confdence problem, where models can mis-classify unknown objects as known objects with high certainty. To addresssing the fundamental concerns about the adaptability, we propose a solution to estimate uncertainty of the visual prompting process by cycle-consistency. We designed to check whether it can accurately restore the original prompt from its predictions. To quantify this, we measure the mean Intersection over Union (mIoU) between the restored prompt mask and the originally provided prompt mask. Without using complex designs or ensemble methods with multiple networks, our approach achieved a yield rate of 0.9175 in the VISION24 one-shot industrial challenge.

Title: GAInS: Gradient Anomaly-aware Biomedical Instance Segmentation

• Authors: Runsheng Liu, Hao Jiang, Yanning Zhou, Huangjing Lin, Liansheng Wang, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.13988
• Pdf URL: https://arxiv.org/pdf/2409.13988
• Copy Paste: [[2409.13988]] GAInS: Gradient Anomaly-aware Biomedical Instance Segmentation(https://arxiv.org/abs/2409.13988)
• Keywords: segmentation
• Abstract: Instance segmentation plays a vital role in the morphological quantification of biomedical entities such as tissues and cells, enabling precise identification and delineation of different structures. Current methods often address the challenges of touching, overlapping or crossing instances through individual modeling, while neglecting the intrinsic interrelation between these conditions. In this work, we propose a Gradient Anomaly-aware Biomedical Instance Segmentation approach (GAInS), which leverages instance gradient information to perceive local gradient anomaly regions, thus modeling the spatial relationship between instances and refining local region segmentation. Specifically, GAInS is firstly built on a Gradient Anomaly Mapping Module (GAMM), which encodes the radial fields of instances through window sliding to obtain instance gradient anomaly maps. To efficiently refine boundaries and regions with gradient anomaly attention, we propose an Adaptive Local Refinement Module (ALRM) with a gradient anomaly-aware loss function. Extensive comparisons and ablation experiments in three biomedical scenarios demonstrate that our proposed GAInS outperforms other state-of-the-art (SOTA) instance segmentation methods. The code is available at this https URL.

Title: ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models

• Authors: Yuqing Huang, Rongyang Zhang, Xuesong He, Xuyang Zhi, Hao Wang, Xin Li, Feiyang Xu, Deguang Liu, Huadong Liang, Yi Li, Jian Cui, Zimu Liu, Shijin Wang, Guoping Hu, Guiquan Liu, Qi Liu, Defu Lian, Enhong Chen
• Subjects: cs.CL, cs.AI, cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2409.13989
• Pdf URL: https://arxiv.org/pdf/2409.13989
• Copy Paste: [[2409.13989]] ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models(https://arxiv.org/abs/2409.13989)
• Keywords: large language model
• Abstract: There is a growing interest in the role that LLMs play in chemistry which lead to an increased focus on the development of LLMs benchmarks tailored to chemical domains to assess the performance of LLMs across a spectrum of chemical tasks varying in type and complexity. However, existing benchmarks in this domain fail to adequately meet the specific requirements of chemical research professionals. To this end, we propose \textbf{\textit{ChemEval}}, which provides a comprehensive assessment of the capabilities of LLMs across a wide range of chemical domain tasks. Specifically, ChemEval identified 4 crucial progressive levels in chemistry, assessing 12 dimensions of LLMs across 42 distinct chemical tasks which are informed by open-source data and the data meticulously crafted by chemical experts, ensuring that the tasks have practical value and can effectively evaluate the capabilities of LLMs. In the experiment, we evaluate 12 mainstream LLMs on ChemEval under zero-shot and few-shot learning contexts, which included carefully selected demonstration examples and carefully designed prompts. The results show that while general LLMs like GPT-4 and Claude-3.5 excel in literature understanding and instruction following, they fall short in tasks demanding advanced chemical knowledge. Conversely, specialized LLMs exhibit enhanced chemical competencies, albeit with reduced literary comprehension. This suggests that LLMs have significant potential for enhancement when tackling sophisticated tasks in the field of chemistry. We believe our work will facilitate the exploration of their potential to drive progress in chemistry. Our benchmark and analysis will be available at {\color{blue} \url{this https URL}}.

Title: SMART-RAG: Selection using Determinantal Matrices for Augmented Retrieval

• Authors: Jiatao Li, Xinyu Hu, Xiaojun Wan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.13992
• Pdf URL: https://arxiv.org/pdf/2409.13992
• Copy Paste: [[2409.13992]] SMART-RAG: Selection using Determinantal Matrices for Augmented Retrieval(https://arxiv.org/abs/2409.13992)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) has greatly improved large language models (LLMs) by enabling them to generate accurate, contextually grounded responses through the integration of external information. However, conventional RAG approaches, which prioritize top-ranked documents based solely on query-context relevance, often introduce redundancy and conflicting information. This issue is particularly evident in unsupervised retrieval settings, where there are no mechanisms to effectively mitigate these problems, leading to suboptimal context selection. To address this, we propose Selection using Matrices for Augmented Retrieval (SMART) in question answering tasks, a fully unsupervised and training-free framework designed to optimize context selection in RAG. SMART leverages Determinantal Point Processes (DPPs) to simultaneously model relevance, diversity and conflict, ensuring the selection of potentially high-quality contexts. Experimental results across multiple datasets demonstrate that SMART significantly enhances QA performance and surpasses previous unsupervised context selection methods, showing a promising strategy for RAG.

Title: Contrastive Learning for Knowledge-Based Question Generation in Large Language Models

• Authors: Zhenhong Zhang, Jiajing Chen, Weiyan Shi, Lingjie Yi, Chihang Wang, Qian Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.13994
• Pdf URL: https://arxiv.org/pdf/2409.13994
• Copy Paste: [[2409.13994]] Contrastive Learning for Knowledge-Based Question Generation in Large Language Models(https://arxiv.org/abs/2409.13994)
• Keywords: large language model
• Abstract: With the rapid development of artificial intelligence technology, especially the increasingly widespread application of question-and-answer systems, high-quality question generation has become a key component in supporting the development of these systems. This article focuses on knowledge-based question generation technology, which aims to enable computers to simulate the human questioning process based on understanding specific texts or knowledge bases. In light of the issues of hallucination and knowledge gaps present in large-scale language models when applied to knowledge-intensive tasks, this paper proposes an enhanced question generation method that incorporates contrastive learning. This method utilizes multiple models to jointly mine domain knowledge and uses contrastive learning to guide the model in reducing noise and hallucinations in generation. Experimental results show that by designing prompts containing contrasting examples, the model's performance in question generation improves considerably, particularly when contrasting instructions and examples are used simultaneously, leading to the highest quality of generated questions and improved accuracy. These results demonstrate that the method proposed in this study, which combines contrasting context and chain-of-thought prompts, can effectively improve both the quality and the practicality of question generation.

Title: Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer

• Authors: Zheng Liu, Jinchao Zhu, Nannan Li, Gao Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.13999
• Pdf URL: https://arxiv.org/pdf/2409.13999
• Copy Paste: [[2409.13999]] Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer(https://arxiv.org/abs/2409.13999)
• Keywords: transformer
• Abstract: Parameter-efficient transfer learning (PETL) has shown great potential in adapting a vision transformer (ViT) pre-trained on large-scale datasets to various downstream tasks. Existing studies primarily focus on minimizing the number of learnable parameters. Although these methods are storage-efficient, they allocate excessive computational resources to easy samples, leading to inefficient inference. To address this issue, we introduce an inference-efficient tuning method termed multiple-exit tuning (MET). MET integrates multiple exits into the pre-trained ViT backbone. Since the predictions in ViT are made by a linear classifier, each exit is equipped with a linear prediction head. In inference stage, easy samples will exit at early exits and only hard enough samples will flow to the last exit, thus saving the computational cost for easy samples. MET consists of exit-specific adapters (E-adapters) and graph regularization. E-adapters are designed to extract suitable representations for different exits. To ensure parameter efficiency, all E-adapters share the same down-projection and up-projection matrices. As the performances of linear classifiers are influenced by the relationship among samples, we employ graph regularization to improve the representations fed into the classifiers at early exits. Finally, we conduct extensive experiments to verify the performance of MET. Experimental results show that MET has an obvious advantage over the state-of-the-art methods in terms of both accuracy and inference efficiency.

Title: Boolean Product Graph Neural Networks

• Authors: Ziyan Wang, Bin Liu, Ling Xiang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14001
• Pdf URL: https://arxiv.org/pdf/2409.14001
• Copy Paste: [[2409.14001]] Boolean Product Graph Neural Networks(https://arxiv.org/abs/2409.14001)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have recently achieved significant success, with a key operation involving the aggregation of information from neighboring nodes. Substantial researchers have focused on defining neighbors for aggregation, predominantly based on observed adjacency matrices. However, in many scenarios, the explicitly given graphs contain noise, which can be amplified during the messages-passing process. Therefore, many researchers have turned their attention to latent graph inference, specifically learning a parametric graph. To mitigate fluctuations in latent graph structure learning, this paper proposes a novel Boolean product-based graph residual connection in GNNs to link the latent graph and the original graph. It computes the Boolean product between the latent graph and the original graph at each layer to correct the learning process. The Boolean product between two adjacency matrices is equivalent to triangle detection. Accordingly, the proposed Boolean product graph neural networks can be interpreted as discovering triangular cliques from the original and the latent graph. We validate the proposed method in benchmark datasets and demonstrate its ability to enhance the performance and robustness of GNNs.

Title: Cyber-Physical Authentication Scheme for Secure V2G Transactions Using Blockchain and Smart Contracts

• Authors: Yunwang Chen, Yanmin Zhao, Siuming Yiu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14008
• Pdf URL: https://arxiv.org/pdf/2409.14008
• Copy Paste: [[2409.14008]] Cyber-Physical Authentication Scheme for Secure V2G Transactions Using Blockchain and Smart Contracts(https://arxiv.org/abs/2409.14008)
• Keywords: secure, security, privacy, attack, robust
• Abstract: The rapid adoption of electric vehicles (EVs) globally has catalyzed the need for robust cybersecurity measures within vehicle-to-grid (V2G) networks. As these networks are increasingly being integrated into smart charging infrastructures, they also introduce new vulnerabilities that threaten grid stability and user privacy This paper proposes a cyber-physical authentication protocol and trading smart contract tailored to plug and charge (PnC) operations within blockchain-based V2G systems. The protocol leverages advanced cryptographic techniques and blockchain to ensure secure, transparent, and tamper-proof energy transactions between EVs and charging stations. Key contributions include the development of a cyber-physical authentication method, the implementation of a smart contract framework for secure energy trading, and a detailed security and privacy analysis. The proposed protocol effectively mitigates risks such as distributed denial of service (DDoS) attacks, man-in-the-middle (MitM) attacks and replay attacks while preserving user anonymity and data integrity.

Title: ChronoGAN: Supervised and Embedded Generative Adversarial Networks for Time Series Generation

• Authors: MohammadReza EskandariNasab, Shah Muhammad Hamdi, Soukaina Filali Boubrahimi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14013
• Pdf URL: https://arxiv.org/pdf/2409.14013
• Copy Paste: [[2409.14013]] ChronoGAN: Supervised and Embedded Generative Adversarial Networks for Time Series Generation(https://arxiv.org/abs/2409.14013)
• Keywords: robust, generative
• Abstract: Generating time series data using Generative Adversarial Networks (GANs) presents several prevalent challenges, such as slow convergence, information loss in embedding spaces, instability, and performance variability depending on the series length. To tackle these obstacles, we introduce a robust framework aimed at addressing and mitigating these issues effectively. This advanced framework integrates the benefits of an Autoencoder-generated embedding space with the adversarial training dynamics of GANs. This framework benefits from a time series-based loss function and oversight from a supervisory network, both of which capture the stepwise conditional distributions of the data effectively. The generator functions within the latent space, while the discriminator offers essential feedback based on the feature space. Moreover, we introduce an early generation algorithm and an improved neural network architecture to enhance stability and ensure effective generalization across both short and long time series. Through joint training, our framework consistently outperforms existing benchmarks, generating high-quality time series data across a range of real and synthetic datasets with diverse characteristics.

Title: Mitigating Exposure Bias in Score-Based Generation of Molecular Conformations

• Authors: Sijia Wang, Chen Wang, Zhenhao Zhao, Jiqiang Zhang, Weiran Cai
• Subjects: cs.LG, cs.AI, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2409.14014
• Pdf URL: https://arxiv.org/pdf/2409.14014
• Copy Paste: [[2409.14014]] Mitigating Exposure Bias in Score-Based Generation of Molecular Conformations(https://arxiv.org/abs/2409.14014)
• Keywords: diffusion, generative
• Abstract: Molecular conformation generation poses a significant challenge in the field of computational chemistry. Recently, Diffusion Probabilistic Models (DPMs) and Score-Based Generative Models (SGMs) are effectively used due to their capacity for generating accurate conformations far beyond conventional physics-based approaches. However, the discrepancy between training and inference rises a critical problem known as the exposure bias. While this issue has been extensively investigated in DPMs, the existence of exposure bias in SGMs and its effective measurement remain unsolved, which hinders the use of compensation methods for SGMs, including ConfGF and Torsional Diffusion as the representatives. In this work, we first propose a method for measuring exposure bias in SGMs used for molecular conformation generation, which confirms the significant existence of exposure bias in these models and measures its value. We design a new compensation algorithm Input Perturbation (IP), which is adapted from a method originally designed for DPMs only. Experimental results show that by introducing IP, SGM-based molecular conformation models can significantly improve both the accuracy and diversity of the generated conformations. Especially by using the IP-enhanced Torsional Diffusion model, we achieve new state-of-the-art performance on the GEOM-Drugs dataset and are on par on GEOM-QM9. We provide the code publicly at this https URL.

Title: MOSE: Monocular Semantic Reconstruction Using NeRF-Lifted Noisy Priors

• Authors: Zhenhua Du, Binbin Xu, Haoyu Zhang, Kai Huo, Shuaifeng Zhi
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.14019
• Pdf URL: https://arxiv.org/pdf/2409.14019
• Copy Paste: [[2409.14019]] MOSE: Monocular Semantic Reconstruction Using NeRF-Lifted Noisy Priors(https://arxiv.org/abs/2409.14019)
• Keywords: segmentation
• Abstract: Accurately reconstructing dense and semantically annotated 3D meshes from monocular images remains a challenging task due to the lack of geometry guidance and imperfect view-dependent 2D priors. Though we have witnessed recent advancements in implicit neural scene representations enabling precise 2D rendering simply from multi-view images, there have been few works addressing 3D scene understanding with monocular priors alone. In this paper, we propose MOSE, a neural field semantic reconstruction approach to lift inferred image-level noisy priors to 3D, producing accurate semantics and geometry in both 3D and 2D space. The key motivation for our method is to leverage generic class-agnostic segment masks as guidance to promote local consistency of rendered semantics during training. With the help of semantics, we further apply a smoothness regularization to texture-less regions for better geometric quality, thus achieving mutual benefits of geometry and semantics. Experiments on the ScanNet dataset show that our MOSE outperforms relevant baselines across all metrics on tasks of 3D semantic segmentation, 2D semantic segmentation and 3D surface reconstruction.

Title: BrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidance

• Authors: Ling Wang, Chen Wu, Lin Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14021
• Pdf URL: https://arxiv.org/pdf/2409.14021
• Copy Paste: [[2409.14021]] BrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidance(https://arxiv.org/abs/2409.14021)
• Keywords: diffusion, generative
• Abstract: Can we directly visualize what we imagine in our brain together with what we describe? The inherent nature of human perception reveals that, when we think, our body can combine language description and build a vivid picture in our brain. Intuitively, generative models should also hold such versatility. In this paper, we introduce BrainDreamer, a novel end-to-end language-guided generative framework that can mimic human reasoning and generate high-quality images from electroencephalogram (EEG) brain signals. Our method is superior in its capacity to eliminate the noise introduced by non-invasive EEG data acquisition and meanwhile achieve a more precise mapping between the EEG and image modality, thus leading to significantly better-generated images. Specifically, BrainDreamer consists of two key learning stages: 1) modality alignment and 2) image generation. In the alignment stage, we propose a novel mask-based triple contrastive learning strategy to effectively align EEG, text, and image embeddings to learn a unified representation. In the generation stage, we inject the EEG embeddings into the pre-trained Stable Diffusion model by designing a learnable EEG adapter to generate high-quality reasoning-coherent images. Moreover, BrainDreamer can accept textual descriptions (e.g., color, position, etc.) to achieve controllable image generation. Extensive experiments show that our method significantly outperforms prior arts in terms of generating quality and quantitative performance.

Title: Can LLMs replace Neil deGrasse Tyson? Evaluating the Reliability of LLMs as Science Communicators

• Authors: Prasoon Bajpai, Niladri Chatterjee, Subhabrata Dutta, Tanmoy Chakraborty
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14037
• Pdf URL: https://arxiv.org/pdf/2409.14037
• Copy Paste: [[2409.14037]] Can LLMs replace Neil deGrasse Tyson? Evaluating the Reliability of LLMs as Science Communicators(https://arxiv.org/abs/2409.14037)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) and AI assistants driven by these models are experiencing exponential growth in usage among both expert and amateur users. In this work, we focus on evaluating the reliability of current LLMs as science communicators. Unlike existing benchmarks, our approach emphasizes assessing these models on scientific questionanswering tasks that require a nuanced understanding and awareness of answerability. We introduce a novel dataset, SCiPS-QA, comprising 742 Yes/No queries embedded in complex scientific concepts, along with a benchmarking suite that evaluates LLMs for correctness and consistency across various criteria. We benchmark three proprietary LLMs from the OpenAI GPT family and 13 open-access LLMs from the Meta Llama-2, Llama-3, and Mistral families. While most open-access models significantly underperform compared to GPT-4 Turbo, our experiments identify Llama-3-70B as a strong competitor, often surpassing GPT-4 Turbo in various evaluation aspects. We also find that even the GPT models exhibit a general incompetence in reliably verifying LLM responses. Moreover, we observe an alarming trend where human evaluators are deceived by incorrect responses from GPT-4 Turbo.

Title: Towards Lightweight and Privacy-preserving Data Provision in Digital Forensics for Driverless Taxi

• Authors: Yanwei Gong, Xiaolin Chang, Jelena Mišić, Vojislav B. Mišić, Junchao Fan, Kaiwen Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14039
• Pdf URL: https://arxiv.org/pdf/2409.14039
• Copy Paste: [[2409.14039]] Towards Lightweight and Privacy-preserving Data Provision in Digital Forensics for Driverless Taxi(https://arxiv.org/abs/2409.14039)
• Keywords: security, privacy
• Abstract: Data provision, referring to the data upload and data access, is one key phase in vehicular digital forensics. The unique features of Driverless Taxi (DT) bring new issues to this phase: 1) efficient verification of data integrity when diverse Data Providers (DPs) upload data; 2) DP privacy preservation during data upload; and 3) privacy preservation of both data and INvestigator (IN) under complex data ownership when accessing data. To this end, we propose a novel Lightweight and Privacy-preserving Data Provision (LPDP) approach consisting of three mechanisms: 1) the Privacy-friendly Batch Verification Mechanism (PBVm) based on elliptic curve cryptography, 2) Data Access Control Mechanism (DACm) based on ciphertext-policy attribute-based encryption, and 3) Decentralized IN Warrant Issuance Mechanism (DIWIm) based on secret sharing. Privacy preservation of data provision is achieved through: 1) ensuring the DP privacy preservation in terms of the location privacy and unlinkability of data upload requests by PBVm, 2) ensuring data privacy preservation by DACm and DIWIm, and 3) ensuring the identity privacy of IN in terms of the anonymity and unlinkability of data access requests without sacrificing the traceability. Lightweight of data provision is achieved through: 1) ensuring scalable verification of data integrity by PBVm, and 2) ensuring low-overhead warrant update with respect to DIWIm. Security analysis and performance evaluation are conducted to validate the security and performance features of LPDP.

Title: GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussion

• Authors: Tongxuan Liu, Xingyu Wang, Weizhe Huang, Wenjiang Xu, Yuting Zeng, Lei Jiang, Hailong Yang, Jing Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14051
• Pdf URL: https://arxiv.org/pdf/2409.14051
• Copy Paste: [[2409.14051]] GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussion(https://arxiv.org/abs/2409.14051)
• Keywords: large language model
• Abstract: In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse NLP tasks. Extensive research has explored how to enhance the logical reasoning abilities such as Chain-of-Thought, Chain-of-Thought with Self-Consistency, Tree-Of-Thoughts, and multi-agent debates. In the context of multi-agent debates, significant performance improvements can be achieved with an increasing number of agents and debate rounds. However, the escalation in the number of agents and debate rounds can drastically raise the tokens cost of debates, thereby limiting the scalability of the multi-agent debate technique. To better harness the advantages of multi-agent debates in logical reasoning tasks, this paper proposes a method to significantly reduce token cost in multi-agent debates. This approach involves dividing all agents into multiple debate groups, with agents engaging in debates within their respective groups and sharing interim debate results between groups. Comparative experiments across multiple datasets have demonstrated that this method can reduce the total tokens by up to 51.7% during debates and while potentially enhancing accuracy by as much as 25%. Our method significantly enhances the performance and efficiency of interactions in the multi-agent debate.

Title: Co-occurrence is not Factual Association in Language Models

• Authors: Xiao Zhang, Miao Li, Ji Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14057
• Pdf URL: https://arxiv.org/pdf/2409.14057
• Copy Paste: [[2409.14057]] Co-occurrence is not Factual Association in Language Models(https://arxiv.org/abs/2409.14057)
• Keywords: transformer
• Abstract: Pretrained language models can encode a large amount of knowledge and utilize it for various reasoning tasks, yet they can still struggle to learn novel factual knowledge effectively from finetuning on limited textual demonstrations. In this work, we show that the reason for this deficiency is that language models are biased to learn word co-occurrence statistics instead of true factual associations. We identify the differences between two forms of knowledge representation in language models: knowledge in the form of co-occurrence statistics is encoded in the middle layers of the transformer model and does not generalize well to reasoning scenarios beyond simple question answering, while true factual associations are encoded in the lower layers and can be freely utilized in various reasoning tasks. Based on these observations, we propose two strategies to improve the learning of factual associations in language models. We show that training on text with implicit rather than explicit factual associations can force the model to learn factual associations instead of co-occurrence statistics, significantly improving the generalization of newly learned knowledge. We also propose a simple training method to actively forget the learned co-occurrence statistics, which unblocks and enhances the learning of factual associations when training on plain narrative text. On both synthetic and real-world corpora, the two proposed strategies improve the generalization of the knowledge learned during finetuning to reasoning scenarios such as indirect and multi-hop question answering.

Title: Practically implementing an LLM-supported collaborative vulnerability remediation process: a team-based approach

• Authors: Xiaoqing Wang, Yuanjing Tian, Keman Huang, Bin Liang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14058
• Pdf URL: https://arxiv.org/pdf/2409.14058
• Copy Paste: [[2409.14058]] Practically implementing an LLM-supported collaborative vulnerability remediation process: a team-based approach(https://arxiv.org/abs/2409.14058)
• Keywords: security
• Abstract: Incorporating LLM into cybersecurity operations, a typical real-world high-stakes task, is critical but non-trivial in practice. Using cybersecurity as the study context, we conduct a three-step mix-method study to incorporate LLM into the vulnerability remediation process effectively. Specifically, we deconstruct the deficiencies in user satisfaction within the existing process (Study 1). This inspires us to design, implement, and empirically validate an LLM-supported collaborative vulnerability remediation process through a field study (Study 2). Given LLM's diverse contributions, we further investigate LLM's double-edge roles through the analysis of remediation reports and follow-up interviews (Study 3). In essence, our contribution lies in promoting an efficient LLM-supported collaborative vulnerability remediation process. These first-hand, real-world pieces of evidence suggest that when incorporating LLMs into practical processes, facilitating the collaborations among all associated stakeholders, reshaping LLMs' roles according to task complexity, as well as approaching the short-term side effects of improved user engagement facilitated by LLMs with a rational mindset.

Title: Soft Segmented Randomization: Enhancing Domain Generalization in SAR-ATR for Synthetic-to-Measured

• Authors: Minjun Kim, Ohtae Jang, Haekang Song, Heesub Shin, Jaewoo Ok, Minyoung Back, Jaehyuk Youn, Sungho Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14060
• Pdf URL: https://arxiv.org/pdf/2409.14060
• Copy Paste: [[2409.14060]] Soft Segmented Randomization: Enhancing Domain Generalization in SAR-ATR for Synthetic-to-Measured(https://arxiv.org/abs/2409.14060)
• Keywords: robust
• Abstract: Synthetic aperture radar technology is crucial for high-resolution imaging under various conditions; however, the acquisition of real-world synthetic aperture radar data for deep learning-based automatic target recognition remains challenging due to high costs and data availability issues. To overcome these challenges, synthetic data generated through simulations have been employed, although discrepancies between synthetic and real data can degrade model performance. In this study, we introduce a novel framework, soft segmented randomization, designed to reduce domain discrepancy and improve the generalize ability of synthetic aperture radar automatic target recognition models. The soft segmented randomization framework applies a Gaussian mixture model to segment target and clutter regions softly, introducing randomized variations that align the synthetic data's statistical properties more closely with those of real-world data. Experimental results demonstrate that the proposed soft segmented randomization framework significantly enhances model performance on measured synthetic aperture radar data, making it a promising approach for robust automatic target recognition in scenarios with limited or no access to measured data.

Title: Recovering Global Data Distribution Locally in Federated Learning

• Authors: Ziyu Yao
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14063
• Pdf URL: https://arxiv.org/pdf/2409.14063
• Copy Paste: [[2409.14063]] Recovering Global Data Distribution Locally in Federated Learning(https://arxiv.org/abs/2409.14063)
• Keywords: privacy, federate, generative
• Abstract: Federated Learning (FL) is a distributed machine learning paradigm that enables collaboration among multiple clients to train a shared model without sharing raw data. However, a major challenge in FL is the label imbalance, where clients may exclusively possess certain classes while having numerous minority and missing classes. Previous works focus on optimizing local updates or global aggregation but ignore the underlying imbalanced label distribution across clients. In this paper, we propose a novel approach ReGL to address this challenge, whose key idea is to Recover the Global data distribution Locally. Specifically, each client uses generative models to synthesize images that complement the minority and missing classes, thereby alleviating label imbalance. Moreover, we adaptively fine-tune the image generation process using local real data, which makes the synthetic images align more closely with the global distribution. Importantly, both the generation and fine-tuning processes are conducted at the client-side without leaking data privacy. Through comprehensive experiments on various image classification datasets, we demonstrate the remarkable superiority of our approach over existing state-of-the-art works in fundamentally tackling label imbalance in FL.

Title: Temporally Consistent Factuality Probing for Large Language Models

• Authors: Ashutosh Bajpai, Aaryan Goyal, Atif Anwer, Tanmoy Chakraborty
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14065
• Pdf URL: https://arxiv.org/pdf/2409.14065
• Copy Paste: [[2409.14065]] Temporally Consistent Factuality Probing for Large Language Models(https://arxiv.org/abs/2409.14065)
• Keywords: large language model
• Abstract: The prolific use of Large Language Models (LLMs) as an alternate knowledge base requires them to be factually consistent, necessitating both correctness and consistency traits for paraphrased queries. Recently, significant attempts have been made to benchmark datasets and metrics to evaluate LLMs for these traits. However, structural simplicity (subject-relation-object) and contemporary association in their query formulation limit the broader definition of factuality and consistency. In this study, we introduce TeCFaP, a novel Temporally Consistent Factuality Probe task to expand the consistent factuality probe in the temporal dimension. To this end, we propose TEMP-COFAC, a high-quality dataset of prefix-style English query paraphrases. Subsequently, we extend the definitions of existing metrics to represent consistent factuality across temporal dimension. We experiment with a diverse set of LLMs and find most of them performing poorly on TeCFaP. Next, we propose a novel solution CoTSeLF (Consistent-Time-Sensitive Learning Framework) combining multi-task instruction tuning (MT-IT) with consistent-time-sensitive reinforcement learning (CTSRL) to improve temporally consistent factuality in LLMs. Our experiments demonstrate the efficacy of CoTSeLF over several baselines.

Title: PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL

• Authors: Ruilin Luo, Liyuan Wang, Binghuai Lin, Zicheng Lin, Yujiu Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14082
• Pdf URL: https://arxiv.org/pdf/2409.14082
• Copy Paste: [[2409.14082]] PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL(https://arxiv.org/abs/2409.14082)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have emerged as powerful tools for Text-to-SQL tasks, exhibiting remarkable reasoning capabilities. Different from tasks such as math word problems and commonsense reasoning, SQL solutions have a relatively fixed pattern. This facilitates the investigation of whether LLMs can benefit from categorical thinking, mirroring how humans acquire knowledge through inductive reasoning based on comparable examples. In this study, we propose that employing query group partitioning allows LLMs to focus on learning the thought processes specific to a single problem type, consequently enhancing their reasoning abilities across diverse difficulty levels and problem categories. Our experiments reveal that multiple advanced LLMs, when equipped with PTD-SQL, can either surpass or match previous state-of-the-art (SOTA) methods on the Spider and BIRD datasets. Intriguingly, models with varying initial performances have exhibited significant improvements, mainly at the boundary of their capabilities after targeted drilling, suggesting a parallel with human progress. Code is available at this https URL.

Title: SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information

• Authors: Jiashuo Sun, Jihai Zhang, Yucheng Zhou, Zhaochen Su, Xiaoye Qu, Yu Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14083
• Pdf URL: https://arxiv.org/pdf/2409.14083
• Copy Paste: [[2409.14083]] SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information(https://arxiv.org/abs/2409.14083)
• Keywords: robust
• Abstract: Large Vision-Language Models (LVLMs) have become pivotal at the intersection of computer vision and natural language processing. However, the full potential of LVLMs Retrieval-Augmented Generation (RAG) capabilities remains underutilized. Existing works either focus solely on the text modality or are limited to specific tasks. Moreover, most LVLMs struggle to selectively utilize retrieved information and are sensitive to irrelevant or misleading references. To address these challenges, we propose a self-refinement framework designed to teach LVLMs to Selectively Utilize Retrieved Information (SURf). Specifically, when given questions that are incorrectly answered by the LVLM backbone, we obtain references that help correct the answers (positive references) and those that do not (negative references). We then fine-tune the LVLM backbone using a combination of these positive and negative references. Our experiments across three tasks and seven datasets demonstrate that our framework significantly enhances LVLMs ability to effectively utilize retrieved multimodal references and improves their robustness against irrelevant or misleading information. The source code is available at this https URL.

Title: One-shot World Models Using a Transformer Trained on a Synthetic Prior

• Authors: Fabio Ferreira, Moreno Schlageter, Raghu Rajan, Andre Biedenkapp, Frank Hutter
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14084
• Pdf URL: https://arxiv.org/pdf/2409.14084
• Copy Paste: [[2409.14084]] One-shot World Models Using a Transformer Trained on a Synthetic Prior(https://arxiv.org/abs/2409.14084)
• Keywords: transformer
• Abstract: A World Model is a compressed spatial and temporal representation of a real world environment that allows one to train an agent or execute planning methods. However, world models are typically trained on observations from the real world environment, and they usually do not enable learning policies for other real environments. We propose One-Shot World Model (OSWM), a transformer world model that is learned in an in-context learning fashion from purely synthetic data sampled from a prior distribution. Our prior is composed of multiple randomly initialized neural networks, where each network models the dynamics of each state and reward dimension of a desired target environment. We adopt the supervised learning procedure of Prior-Fitted Networks by masking next-state and reward at random context positions and query OSWM to make probabilistic predictions based on the remaining transition context. During inference time, OSWM is able to quickly adapt to the dynamics of a simple grid world, as well as the CartPole gym and a custom control environment by providing 1k transition steps as context and is then able to successfully train environment-solving agent policies. However, transferring to more complex environments remains a challenge, currently. Despite these limitations, we see this work as an important stepping-stone in the pursuit of learning world models purely from synthetic data.

Title: Encryption of Audio Signals Using the Elzaki Transformation and the Lorenz Chaotic System Lorenz Chaotic System

• Authors: Shadman R. Kareem
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14092
• Pdf URL: https://arxiv.org/pdf/2409.14092
• Copy Paste: [[2409.14092]] Encryption of Audio Signals Using the Elzaki Transformation and the Lorenz Chaotic System Lorenz Chaotic System(https://arxiv.org/abs/2409.14092)
• Keywords: security, privacy, protect, robust
• Abstract: The preservation of image privacy during storage and transmission is of paramount importance in several areas including healthcare, military, safe communication, and video conferencing. Protecting data privacy demands the use of robust image encryption techniques. Several cryptographic techniques have been particularly designed to ensure the privacy of digital images. This study presents a novel method for encrypting color images utilizing chaos theory and a special transformation. This indicated approach first employs the Lorenz chaos theory to scramble the audio files. Following that, we utilize a technique that involves using the Maclaurin series expansion of hyperbolic functions and the Elzaki transform to encrypt the audio. Subsequently, we decode it by applying the inverse Elzaki transform. The key for the coefficients obtained from the transformation is created using modular arithmetic methods. Comparisons between the techniques are conducted based on a number of performance measures, including entropy analysis, spectrogram plotting, and correlation coefficients. Theoretical analysis and simulation indicate the efficacy of the proposed approach and confirm that this method is suitable for actual audio encryption. Moreover, the security inquiry indicates that an extra layer of security is provided by the provided audio encryption approach

Title: Foundation Models for Amodal Video Instance Segmentation in Automated Driving

• Authors: Jasmin Breitenstein, Franz Jünger, Andreas Bär, Tim Fingscheidt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14095
• Pdf URL: https://arxiv.org/pdf/2409.14095
• Copy Paste: [[2409.14095]] Foundation Models for Amodal Video Instance Segmentation in Automated Driving(https://arxiv.org/abs/2409.14095)
• Keywords: segmentation
• Abstract: In this work, we study amodal video instance segmentation for automated driving. Previous works perform amodal video instance segmentation relying on methods trained on entirely labeled video data with techniques borrowed from standard video instance segmentation. Such amodally labeled video data is difficult and expensive to obtain and the resulting methods suffer from a trade-off between instance segmentation and tracking performance. To largely solve this issue, we propose to study the application of foundation models for this task. More precisely, we exploit the extensive knowledge of the Segment Anything Model (SAM), while fine-tuning it to the amodal instance segmentation task. Given an initial video instance segmentation, we sample points from the visible masks to prompt our amodal SAM. We use a point memory to store those points. If a previously observed instance is not predicted in a following frame, we retrieve its most recent points from the point memory and use a point tracking method to follow those points to the current frame, together with the corresponding last amodal instance mask. This way, while basing our method on an amodal instance segmentation, we nevertheless obtain video-level amodal instance segmentation results. Our resulting S-AModal method achieves state-of-the-art results in amodal video instance segmentation while resolving the need for amodal video-based labels. Code for S-AModal is available at this https URL.

Title: Probing Context Localization of Polysemous Words in Pre-trained Language Model Sub-Layers

• Authors: Soniya Vijayakumar, Josef van Genabith, Simon Ostermann
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14097
• Pdf URL: https://arxiv.org/pdf/2409.14097
• Copy Paste: [[2409.14097]] Probing Context Localization of Polysemous Words in Pre-trained Language Model Sub-Layers(https://arxiv.org/abs/2409.14097)
• Keywords: large language model
• Abstract: In the era of high performing Large Language Models, researchers have widely acknowledged that contextual word representations are one of the key drivers in achieving top performances in downstream tasks. In this work, we investigate the degree of contextualization encoded in the fine-grained sub-layer representations of a Pre-trained Language Model (PLM) by empirical experiments using linear probes. Unlike previous work, we are particularly interested in identifying the strength of contextualization across PLM sub-layer representations (i.e. Self-Attention, Feed-Forward Activation and Output sub-layers). To identify the main contributions of sub-layers to contextualisation, we first extract the sub-layer representations of polysemous words in minimally different sentence pairs, and compare how these representations change through the forward pass of the PLM network. Second, by probing on a sense identification classification task, we try to empirically localize the strength of contextualization information encoded in these sub-layer representations. With these probing experiments, we also try to gain a better understanding of the influence of context length and context richness on the degree of contextualization. Our main conclusion is cautionary: BERT demonstrates a high degree of contextualization in the top sub-layers if the word in question is in a specific position in the sentence with a shorter context window, but this does not systematically generalize across different word positions and context sizes.

Title: PoseAugment: Generative Human Pose Data Augmentation with Physical Plausibility for IMU-based Motion Capture

• Authors: Zhuojun Li, Chun Yu, Chen Liang, Yuanchun Shi
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.14101
• Pdf URL: https://arxiv.org/pdf/2409.14101
• Copy Paste: [[2409.14101]] PoseAugment: Generative Human Pose Data Augmentation with Physical Plausibility for IMU-based Motion Capture(https://arxiv.org/abs/2409.14101)
• Keywords: generative
• Abstract: The data scarcity problem is a crucial factor that hampers the model performance of IMU-based human motion capture. However, effective data augmentation for IMU-based motion capture is challenging, since it has to capture the physical relations and constraints of the human body, while maintaining the data distribution and quality. We propose PoseAugment, a novel pipeline incorporating VAE-based pose generation and physical optimization. Given a pose sequence, the VAE module generates infinite poses with both high fidelity and diversity, while keeping the data distribution. The physical module optimizes poses to satisfy physical constraints with minimal motion restrictions. High-quality IMU data are then synthesized from the augmented poses for training motion capture models. Experiments show that PoseAugment outperforms previous data augmentation and pose generation methods in terms of motion capture accuracy, revealing a strong potential of our method to alleviate the data collection burden for IMU-based motion capture and related tasks driven by human poses.

Title: ESDS: AI-Powered Early Stunting Detection and Monitoring System using Edited Radius-SMOTE Algorithm

• Authors: A.A. Gde Yogi Pramana, Haidar Muhammad Zidan, Muhammad Fazil Maulana, Oskar Natan
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.14105
• Pdf URL: https://arxiv.org/pdf/2409.14105
• Copy Paste: [[2409.14105]] ESDS: AI-Powered Early Stunting Detection and Monitoring System using Edited Radius-SMOTE Algorithm(https://arxiv.org/abs/2409.14105)
• Keywords: generative
• Abstract: Stunting detection is a significant issue in Indonesian healthcare, causing lower cognitive function, lower productivity, a weakened immunity, delayed neuro-development, and degenerative diseases. In regions with a high prevalence of stunting and limited welfare resources, identifying children in need of treatment is critical. The diagnostic process often raises challenges, such as the lack of experience in medical workers, incompatible anthropometric equipment, and inefficient medical bureaucracy. To counteract the issues, the use of load cell sensor and ultrasonic sensor can provide suitable anthropometric equipment and streamline the medical bureaucracy for stunting detection. This paper also employs machine learning for stunting detection based on sensor readings. The experiment results show that the sensitivity of the load cell sensor and the ultrasonic sensor is 0.9919 and 0.9986, respectively. Also, the machine learning test results have three classification classes, which are normal, stunted, and stunting with an accuracy rate of 98\%.

Title: Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm

• Authors: Jaehan Kim, Minkyoo Song, Seung Ho Na, Seungwon Shin
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14119
• Pdf URL: https://arxiv.org/pdf/2409.14119
• Copy Paste: [[2409.14119]] Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm(https://arxiv.org/abs/2409.14119)
• Keywords: security, defense, attack, robust, large language model
• Abstract: Parameter-efficient fine-tuning (PEFT) has become a key training strategy for large language models. However, its reliance on fewer trainable parameters poses security risks, such as task-agnostic backdoors. Despite their severe impact on a wide range of tasks, there is no practical defense solution available that effectively counters task-agnostic backdoors within the context of PEFT. In this study, we introduce Obliviate, a PEFT-integrable backdoor defense. We develop two techniques aimed at amplifying benign neurons within PEFT layers and penalizing the influence of trigger tokens. Our evaluations across three major PEFT architectures show that our method can significantly reduce the attack success rate of the state-of-the-art task-agnostic backdoors (83.6%$\downarrow$). Furthermore, our method exhibits robust defense capabilities against both task-specific backdoors and adaptive attacks. Source code will be obtained at this https URL.

Title: Efficient and Effective Model Extraction

• Authors: Hongyu Zhu, Wentao Hu, Sichu Liang, Fangqi Li, Wenwen Wang, Shilin Wang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14122
• Pdf URL: https://arxiv.org/pdf/2409.14122
• Copy Paste: [[2409.14122]] Efficient and Effective Model Extraction(https://arxiv.org/abs/2409.14122)
• Keywords: security, attack, extraction, data-free, generative
• Abstract: Model extraction aims to create a functionally similar copy from a machine learning as a service (MLaaS) API with minimal overhead, often for illicit purposes or as a precursor to further attacks, posing a significant threat to the MLaaS ecosystem. However, recent studies show that model extraction is inefficient, especially when the target task distribution is unavailable. In such cases, even significantly increasing the attack budget fails to yield a sufficiently similar model, reducing the adversary's incentive. In this paper, we revisit the basic design choices throughout the extraction process and propose an efficient and effective algorithm, Efficient and Effective Model Extraction (E3), which optimizes both query preparation and the training routine. E3 achieves superior generalization over state-of-the-art methods while minimizing computational costs. For example, with only 0.005 times the query budget and less than 0.2 times the runtime, E3 outperforms classical generative model-based data-free model extraction with over 50% absolute accuracy improvement on CIFAR-10. Our findings highlight the ongoing risk of model extraction and propose E3 as a useful benchmark for future security evaluations.

Title: Present and Future Generalization of Synthetic Image Detectors

• Authors: Pablo Bernabeu-Perez, Enrique Lopez-Cuena, Dario Garcia-Gasulla
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14128
• Pdf URL: https://arxiv.org/pdf/2409.14128
• Copy Paste: [[2409.14128]] Present and Future Generalization of Synthetic Image Detectors(https://arxiv.org/abs/2409.14128)
• Keywords: robust
• Abstract: The continued release of new and better image generation models increases the demand for synthetic image detectors. In such a dynamic field, detectors need to be able to generalize widely and be robust to uncontrolled alterations. The present work is motivated by this setting, when looking at the role of time, image transformations and data sources, for detector generalization. In these experiments, none of the evaluated detectors is found universal, but results indicate an ensemble could be. Experiments on data collected in the wild show this task to be more challenging than the one defined by large-scale datasets, pointing to a gap between experimentation and actual practice. Finally, we observe a race equilibrium effect, where better generators lead to better detectors, and vice versa. We hypothesize this pushes the field towards a perpetually close race between generators and detectors.

Title: Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysis

• Authors: Zeping Yu, Sophia Ananiadou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14144
• Pdf URL: https://arxiv.org/pdf/2409.14144
• Copy Paste: [[2409.14144]] Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysis(https://arxiv.org/abs/2409.14144)
• Keywords: large language model
• Abstract: We find arithmetic ability resides within a limited number of attention heads, with each head specializing in distinct operations. To delve into the reason, we introduce the Comparative Neuron Analysis (CNA) method, which identifies an internal logic chain consisting of four distinct stages from input to prediction: feature enhancing with shallow FFN neurons, feature transferring by shallow attention layers, feature predicting by arithmetic heads, and prediction enhancing among deep FFN neurons. Moreover, we identify the human-interpretable FFN neurons within both feature-enhancing and feature-predicting stages. These findings lead us to investigate the mechanism of LoRA, revealing that it enhances prediction probabilities by amplifying the coefficient scores of FFN neurons related to predictions. Finally, we apply our method in model pruning for arithmetic tasks and model editing for reducing gender bias. Code is on this https URL.

Title: JVID: Joint Video-Image Diffusion for Visual-Quality and Temporal-Consistency in Video Generation

• Authors: Hadrien Reynaud, Matthew Baugh, Mischa Dombrowski, Sarah Cechnicka, Qingjie Meng, Bernhard Kainz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14149
• Pdf URL: https://arxiv.org/pdf/2409.14149
• Copy Paste: [[2409.14149]] JVID: Joint Video-Image Diffusion for Visual-Quality and Temporal-Consistency in Video Generation(https://arxiv.org/abs/2409.14149)
• Keywords: diffusion
• Abstract: We introduce the Joint Video-Image Diffusion model (JVID), a novel approach to generating high-quality and temporally coherent videos. We achieve this by integrating two diffusion models: a Latent Image Diffusion Model (LIDM) trained on images and a Latent Video Diffusion Model (LVDM) trained on video data. Our method combines these models in the reverse diffusion process, where the LIDM enhances image quality and the LVDM ensures temporal consistency. This unique combination allows us to effectively handle the complex spatio-temporal dynamics in video generation. Our results demonstrate quantitative and qualitative improvements in producing realistic and coherent videos.

Title: When Witnesses Defend: A Witness Graph Topological Layer for Adversarial Graph Learning

• Authors: Naheed Anjum Arafat, Debabrota Basu, Yulia Gel, Yuzhou Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14161
• Pdf URL: https://arxiv.org/pdf/2409.14161
• Copy Paste: [[2409.14161]] When Witnesses Defend: A Witness Graph Topological Layer for Adversarial Graph Learning(https://arxiv.org/abs/2409.14161)
• Keywords: defense, attack, robust
• Abstract: Capitalizing on the intuitive premise that shape characteristics are more robust to perturbations, we bridge adversarial graph learning with the emerging tools from computational topology, namely, persistent homology representations of graphs. We introduce the concept of witness complex to adversarial analysis on graphs, which allows us to focus only on the salient shape characteristics of graphs, yielded by the subset of the most essential nodes (i.e., landmarks), with minimal loss of topological information on the whole graph. The remaining nodes are then used as witnesses, governing which higher-order graph substructures are incorporated into the learning process. Armed with the witness mechanism, we design Witness Graph Topological Layer (WGTL), which systematically integrates both local and global topological graph feature representations, the impact of which is, in turn, automatically controlled by the robust regularized topological loss. Given the attacker's budget, we derive the important stability guarantees of both local and global topology encodings and the associated robust topological loss. We illustrate the versatility and efficiency of WGTL by its integration with five GNNs and three existing non-topological defense mechanisms. Our extensive experiments across six datasets demonstrate that WGTL boosts the robustness of GNNs across a range of perturbations and against a range of adversarial attacks, leading to relative gains of up to 18%.

Title: On Importance of Pruning and Distillation for Efficient Low Resource NLP

• Authors: Aishwarya Mirashi, Purva Lingayat, Srushti Sonavane, Tejas Padhiyar, Raviraj Joshi, Geetanjali Kale
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14162
• Pdf URL: https://arxiv.org/pdf/2409.14162
• Copy Paste: [[2409.14162]] On Importance of Pruning and Distillation for Efficient Low Resource NLP(https://arxiv.org/abs/2409.14162)
• Keywords: transformer
• Abstract: The rise of large transformer models has revolutionized Natural Language Processing, leading to significant advances in tasks like text classification. However, this progress demands substantial computational resources, escalating training duration, and expenses with larger model sizes. Efforts have been made to downsize and accelerate English models (e.g., Distilbert, MobileBert). Yet, research in this area is scarce for low-resource languages. In this study, we explore the case of the low-resource Indic language Marathi. Leveraging the marathi-topic-all-doc-v2 model as our baseline, we implement optimization techniques to reduce computation time and memory usage. Our focus is on enhancing the efficiency of Marathi transformer models while maintaining top-tier accuracy and reducing computational demands. Using the MahaNews document classification dataset and the marathi-topic-all-doc-v2 model from L3Cube, we apply Block Movement Pruning, Knowledge Distillation, and Mixed Precision methods individually and in combination to boost efficiency. We demonstrate the importance of strategic pruning levels in achieving desired efficiency gains. Furthermore, we analyze the balance between efficiency improvements and environmental impact, highlighting how optimized model architectures can contribute to a more sustainable computational ecosystem. Implementing these techniques on a single GPU system, we determine that the optimal configuration is 25\% pruning + knowledge distillation. This approach yielded a 2.56x speedup in computation time while maintaining baseline accuracy levels.

Title: LFP: Efficient and Accurate End-to-End Lane-Level Planning via Camera-LiDAR Fusion

• Authors: Guoliang You, Xiaomeng Chu, Yifan Duan, Xingchen Li, Sha Zhang, Jianmin Ji, Yanyong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14170
• Pdf URL: https://arxiv.org/pdf/2409.14170
• Copy Paste: [[2409.14170]] LFP: Efficient and Accurate End-to-End Lane-Level Planning via Camera-LiDAR Fusion(https://arxiv.org/abs/2409.14170)
• Keywords: extraction
• Abstract: Multi-modal systems enhance performance in autonomous driving but face inefficiencies due to indiscriminate processing within each modality. Additionally, the independent feature learning of each modality lacks interaction, which results in extracted features that do not possess the complementary characteristics. These issue increases the cost of fusing redundant information across modalities. To address these challenges, we propose targeting driving-relevant elements, which reduces the volume of LiDAR features while preserving critical information. This approach enhances lane level interaction between the image and LiDAR branches, allowing for the extraction and fusion of their respective advantageous features. Building upon the camera-only framework PHP, we introduce the Lane-level camera-LiDAR Fusion Planning (LFP) method, which balances efficiency with performance by using lanes as the unit for sensor fusion. Specifically, we design three modules to enhance efficiency and performance. For efficiency, we propose an image-guided coarse lane prior generation module that forecasts the region of interest (ROI) for lanes and assigns a confidence score, guiding LiDAR processing. The LiDAR feature extraction modules leverages lane-aware priors from the image branch to guide sampling for pillar, retaining essential pillars. For performance, the lane-level cross-modal query integration and feature enhancement module uses confidence score from ROI to combine low-confidence image queries with LiDAR queries, extracting complementary depth features. These features enhance the low-confidence image features, compensating for the lack of depth. Experiments on the Carla benchmarks show that our method achieves state-of-the-art performance in both driving score and infraction score, with maximum improvement of 15% and 14% over existing algorithms, respectively, maintaining high frame rate of 19.27 FPS.

Title: QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling

• Authors: Blessed Guda, Gabrial Zencha A., Lawrence Francis, Carlee Joe-Wong
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14175
• Pdf URL: https://arxiv.org/pdf/2409.14175
• Copy Paste: [[2409.14175]] QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling(https://arxiv.org/abs/2409.14175)
• Keywords: large language model
• Abstract: Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain. Our focus was on using opensource, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of finetuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2.

Title: PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approach

• Authors: Zhihao Lin, Wei Ma, Mingyi Zhou, Yanjie Zhao, Haoyu Wang, Yang Liu, Jun Wang, Li Li
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14177
• Pdf URL: https://arxiv.org/pdf/2409.14177
• Copy Paste: [[2409.14177]] PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approach(https://arxiv.org/abs/2409.14177)
• Keywords: security, defense, attack, robust, large language model
• Abstract: In recent years, Large Language Models (LLMs) have gained widespread use, accompanied by increasing concerns over their security. Traditional jailbreak attacks rely on internal model details or have limitations when exploring the unsafe behavior of the victim model, limiting their generalizability. In this paper, we introduce PathSeeker, a novel black-box jailbreak method inspired by the concept of escaping a security maze. This work is inspired by the game of rats escaping a maze. We think that each LLM has its unique "security maze", and attackers attempt to find the exit learning from the received feedback and their accumulated experience to compromise the target LLM's security defences. Our approach leverages multi-agent reinforcement learning, where smaller models collaborate to guide the main LLM in performing mutation operations to achieve the attack objectives. By progressively modifying inputs based on the model's feedback, our system induces richer, harmful responses. During our manual attempts to perform jailbreak attacks, we found that the vocabulary of the response of the target model gradually became richer and eventually produced harmful responses. Based on the observation, we also introduce a reward mechanism that exploits the expansion of vocabulary richness in LLM responses to weaken security constraints. Our method outperforms five state-of-the-art attack techniques when tested across 13 commercial and open-source LLMs, achieving high attack success rates, especially in strongly aligned commercial models like GPT-4o-mini, Claude-3.5, and GLM-4-air with strong safety alignment. This study aims to improve the understanding of LLM security vulnerabilities and we hope that this sturdy can contribute to the development of more robust defenses.

Title: A Distribution-Aware Flow-Matching for Generating Unstructured Data for Few-Shot Reinforcement Learning

• Authors: Mohammad Pivezhandi, Abusayeed Saifullah
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14178
• Pdf URL: https://arxiv.org/pdf/2409.14178
• Copy Paste: [[2409.14178]] A Distribution-Aware Flow-Matching for Generating Unstructured Data for Few-Shot Reinforcement Learning(https://arxiv.org/abs/2409.14178)
• Keywords: robust
• Abstract: Generating realistic and diverse unstructured data is a significant challenge in reinforcement learning (RL), particularly in few-shot learning scenarios where data is scarce. Traditional RL methods often rely on extensive datasets or simulations, which are costly and time-consuming. In this paper, we introduce a distribution-aware flow matching, designed to generate synthetic unstructured data tailored specifically for an application of few-shot RL called Dynamic Voltage and Frequency Scaling (DVFS) on embedded processors. This method leverages the sample efficiency of flow matching and incorporates statistical learning techniques such as bootstrapping to improve its generalization and robustness of the latent space. Additionally, we apply feature weighting through Random Forests to prioritize critical data aspects, thereby improving the precision of the generated synthetic data. This approach not only mitigates the challenges of overfitting and data correlation in unstructured data in traditional Model-Based RL but also aligns with the Law of Large Numbers, ensuring convergence to true empirical values and optimal policy as the number of samples increases. Through extensive experimentation on an application of DVFS for low energy processing, we demonstrate that our method provides an stable convergence based on max Q-value while enhancing frame rate by 30\% in the very beginning first timestamps, making this RL model efficient in resource-constrained environments.

Title: Content-aware Tile Generation using Exterior Boundary Inpainting

• Authors: Sam Sartor, Pieter Peers
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.14184
• Pdf URL: https://arxiv.org/pdf/2409.14184
• Copy Paste: [[2409.14184]] Content-aware Tile Generation using Exterior Boundary Inpainting(https://arxiv.org/abs/2409.14184)
• Keywords: diffusion
• Abstract: We present a novel and flexible learning-based method for generating tileable image sets. Our method goes beyond simple self-tiling, supporting sets of mutually tileable images that exhibit a high degree of diversity. To promote diversity we decouple structure from content by foregoing explicit copying of patches from an exemplar image. Instead we leverage the prior knowledge of natural images and textures embedded in large-scale pretrained diffusion models to guide tile generation constrained by exterior boundary conditions and a text prompt to specify the content. By carefully designing and selecting the exterior boundary conditions, we can reformulate the tile generation process as an inpainting problem, allowing us to directly employ existing diffusion-based inpainting models without the need to retrain a model on a custom training set. We demonstrate the flexibility and efficacy of our content-aware tile generation method on different tiling schemes, such as Wang tiles, from only a text prompt. Furthermore, we introduce a novel Dual Wang tiling scheme that provides greater texture continuity and diversity than existing Wang tile variants.

Title: Knowledge in Triples for LLMs: Enhancing Table QA Accuracy with Semantic Extraction

• Authors: Hossein Sholehrasa, Sanaz Saki Norouzi, Pascal Hitzler, Majid Jaberi-Douraki
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.14192
• Pdf URL: https://arxiv.org/pdf/2409.14192
• Copy Paste: [[2409.14192]] Knowledge in Triples for LLMs: Enhancing Table QA Accuracy with Semantic Extraction(https://arxiv.org/abs/2409.14192)
• Keywords: extraction
• Abstract: Integrating structured knowledge from tabular formats poses significant challenges within natural language processing (NLP), mainly when dealing with complex, semi-structured tables like those found in the FeTaQA dataset. These tables require advanced methods to interpret and generate meaningful responses accurately. Traditional approaches, such as SQL and SPARQL, often fail to fully capture the semantics of such data, especially in the presence of irregular table structures like web tables. This paper addresses these challenges by proposing a novel approach that extracts triples straightforward from tabular data and integrates it with a retrieval-augmented generation (RAG) model to enhance the accuracy, coherence, and contextual richness of responses generated by a fine-tuned GPT-3.5-turbo-0125 model. Our approach significantly outperforms existing baselines on the FeTaQA dataset, particularly excelling in Sacre-BLEU and ROUGE metrics. It effectively generates contextually accurate and detailed long-form answers from tables, showcasing its strength in complex data interpretation.

Title: The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trends

• Authors: Xinghua Zhang, Haiyang Yu, Yongbin Li, Minzheng Wang, Longze Chen, Fei Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14195
• Pdf URL: https://arxiv.org/pdf/2409.14195
• Copy Paste: [[2409.14195]] The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trends(https://arxiv.org/abs/2409.14195)
• Keywords: large language model
• Abstract: In the era of large language models (LLMs), a vast amount of conversation logs will be accumulated thanks to the rapid development trend of language UI. Conversation Analysis (CA) strives to uncover and analyze critical information from conversation data, streamlining manual processes and supporting business insights and decision-making. The need for CA to extract actionable insights and drive empowerment is becoming increasingly prominent and attracting widespread attention. However, the lack of a clear scope for CA leads to a dispersion of various techniques, making it difficult to form a systematic technical synergy to empower business applications. In this paper, we perform a thorough review and systematize CA task to summarize the existing related work. Specifically, we formally define CA task to confront the fragmented and chaotic landscape in this field, and derive four key steps of CA from conversation scene reconstruction, to in-depth attribution analysis, and then to performing targeted training, finally generating conversations based on the targeted training for achieving the specific goals. In addition, we showcase the relevant benchmarks, discuss potential challenges and point out future directions in both industry and academia. In view of current advancements, it is evident that the majority of efforts are still concentrated on the analysis of shallow conversation elements, which presents a considerable gap between the research and business, and with the assist of LLMs, recent work has shown a trend towards research on causality and strategic tasks which are sophisticated and high-level. The analyzed experiences and insights will inevitably have broader application value in business operations that target conversation logs.

Title: Advancing Employee Behavior Analysis through Synthetic Data: Leveraging ABMs, GANs, and Statistical Models for Enhanced Organizational Efficiency

• Authors: Rakshitha Jayashankar, Mahesh Balan
• Subjects: cs.LG, cs.FL, stat.OT
• Abstract URL: https://arxiv.org/abs/2409.14197
• Pdf URL: https://arxiv.org/pdf/2409.14197
• Copy Paste: [[2409.14197]] Advancing Employee Behavior Analysis through Synthetic Data: Leveraging ABMs, GANs, and Statistical Models for Enhanced Organizational Efficiency(https://arxiv.org/abs/2409.14197)
• Keywords: privacy, protect, generative
• Abstract: Success in todays data-driven corporate climate requires a deep understanding of employee behavior. Companies aim to improve employee satisfaction, boost output, and optimize workflow. This research study delves into creating synthetic data, a powerful tool that allows us to comprehensively understand employee performance, flexibility, cooperation, and team dynamics. Synthetic data provides a detailed and accurate picture of employee activities while protecting individual privacy thanks to cutting-edge methods like agent-based models (ABMs), Generative Adversarial Networks (GANs), and statistical models. Through the creation of multiple situations, this method offers insightful viewpoints regarding increasing teamwork, improving adaptability, and accelerating overall productivity. We examine how synthetic data has evolved from a specialized field to an essential resource for researching employee behavior and enhancing management efficiency. Keywords: Agent-Based Model, Generative Adversarial Network, workflow optimization, organizational success

Title: Data-centric NLP Backdoor Defense from the Lens of Memorization

• Authors: Zhenting Wang, Zhizhi Wang, Mingyu Jin, Mengnan Du, Juan Zhai, Shiqing Ma
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14200
• Pdf URL: https://arxiv.org/pdf/2409.14200
• Copy Paste: [[2409.14200]] Data-centric NLP Backdoor Defense from the Lens of Memorization(https://arxiv.org/abs/2409.14200)
• Keywords: defense, attack
• Abstract: Backdoor attack is a severe threat to the trustworthiness of DNN-based language models. In this paper, we first extend the definition of memorization of language models from sample-wise to more fine-grained sentence element-wise (e.g., word, phrase, structure, and style), and then point out that language model backdoors are a type of element-wise memorization. Through further analysis, we find that the strength of such memorization is positively correlated to the frequency of duplicated elements in the training dataset. In conclusion, duplicated sentence elements are necessary for successful backdoor attacks. Based on this, we propose a data-centric defense. We first detect trigger candidates in training data by finding memorizable elements, i.e., duplicated elements, and then confirm real triggers by testing if the candidates can activate backdoor behaviors (i.e., malicious elements). Results show that our method outperforms state-of-the-art defenses in defending against different types of NLP backdoors.

Title: LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement

• Authors: Nan Jiang, Shanchao Liang, Chengxiao Wang, Jiannan Wang, Lin Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14201
• Pdf URL: https://arxiv.org/pdf/2409.14201
• Copy Paste: [[2409.14201]] LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement(https://arxiv.org/abs/2409.14201)
• Keywords: extraction
• Abstract: Portable Document Format (PDF) files are dominantly used for storing and disseminating scientific research, legal documents, and tax information. LaTeX is a popular application for creating PDF documents. Despite its advantages, LaTeX is not WYSWYG -- what you see is what you get, i.e., the LaTeX source and rendered PDF images look drastically different, especially for formulae and tables. This gap makes it hard to modify or export LaTeX sources for formulae and tables from PDF images, and existing work is still limited. First, prior work generates LaTeX sources in a single iteration and struggles with complex LaTeX formulae. Second, existing work mainly recognizes and extracts LaTeX sources for formulae; and is incapable or ineffective for tables. This paper proposes LATTE, the first iterative refinement framework for LaTeX recognition. Specifically, we propose delta-view as feedback, which compares and pinpoints the differences between a pair of rendered images of the extracted LaTeX source and the expected correct image. Such delta-view feedback enables our fault localization model to localize the faulty parts of the incorrect recognition more accurately and enables our LaTeX refinement model to repair the incorrect extraction more accurately. LATTE improves the LaTeX source extraction accuracy of both LaTeX formulae and tables, outperforming existing techniques as well as GPT-4V by at least 7.07% of exact match, with a success refinement rate of 46.08% (formula) and 25.51% (table).

Title: @Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology

• Authors: Xin Jiang, Junwei Zheng, Ruiping Liu, Jiahang Li, Jiaming Zhang, Sven Matthiesen, Rainer Stiefelhagen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14215
• Pdf URL: https://arxiv.org/pdf/2409.14215
• Copy Paste: [[2409.14215]] @Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology(https://arxiv.org/abs/2409.14215)
• Keywords: segmentation
• Abstract: As Vision-Language Models (VLMs) advance, human-centered Assistive Technologies (ATs) for helping People with Visual Impairments (PVIs) are evolving into generalists, capable of performing multiple tasks simultaneously. However, benchmarking VLMs for ATs remains under-explored. To bridge this gap, we first create a novel AT benchmark (@Bench). Guided by a pre-design user study with PVIs, our benchmark includes the five most crucial vision-language tasks: Panoptic Segmentation, Depth Estimation, Optical Character Recognition (OCR), Image Captioning, and Visual Question Answering (VQA). Besides, we propose a novel AT model (@Model) that addresses all tasks simultaneously and can be expanded to more assistive functions for helping PVIs. Our framework exhibits outstanding performance across tasks by integrating multi-modal information, and it offers PVIs a more comprehensive assistance. Extensive experiments prove the effectiveness and generalizability of our framework.

Title: MEGA-PT: A Meta-Game Framework for Agile Penetration Testing

• Authors: Yunfei Ge, Quanyan Zhu
• Subjects: cs.CR, cs.AI, cs.GT
• Abstract URL: https://arxiv.org/abs/2409.14219
• Pdf URL: https://arxiv.org/pdf/2409.14219
• Copy Paste: [[2409.14219]] MEGA-PT: A Meta-Game Framework for Agile Penetration Testing(https://arxiv.org/abs/2409.14219)
• Keywords: security, defense, attack
• Abstract: Penetration testing is an essential means of proactive defense in the face of escalating cybersecurity incidents. Traditional manual penetration testing methods are time-consuming, resource-intensive, and prone to human errors. Current trends in automated penetration testing are also impractical, facing significant challenges such as the curse of dimensionality, scalability issues, and lack of adaptability to network changes. To address these issues, we propose MEGA-PT, a meta-game penetration testing framework, featuring micro tactic games for node-level local interactions and a macro strategy process for network-wide attack chains. The micro- and macro-level modeling enables distributed, adaptive, collaborative, and fast penetration testing. MEGA-PT offers agile solutions for various security schemes, including optimal local penetration plans, purple teaming solutions, and risk assessment, providing fundamental principles to guide future automated penetration testing. Our experiments demonstrate the effectiveness and agility of our model by providing improved defense strategies and adaptability to changes at both local and network levels.

Title: Masks and Boxes: Combining the Best of Both Worlds for Multi-Object Tracking

• Authors: Tomasz Stanczyk, Francois Bremond
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14220
• Pdf URL: https://arxiv.org/pdf/2409.14220
• Copy Paste: [[2409.14220]] Masks and Boxes: Combining the Best of Both Worlds for Multi-Object Tracking(https://arxiv.org/abs/2409.14220)
• Keywords: robust, segmentation
• Abstract: Multi-object tracking (MOT) involves identifying and consistently tracking objects across video sequences. Traditional tracking-by-detection methods, while effective, often require extensive tuning and lack generalizability. On the other hand, segmentation mask-based methods are more generic but struggle with tracking management, making them unsuitable for MOT. We propose a novel approach, McByte, which incorporates a temporally propagated segmentation mask as a strong association cue within a tracking-by-detection framework. By combining bounding box and mask information, McByte enhances robustness and generalizability without per-sequence tuning. Evaluated on four benchmark datasets - DanceTrack, MOT17, SoccerNet-tracking 2022, and KITTI-tracking - McByte demonstrates performance gain in all cases examined. At the same time, it outperforms existing mask-based methods. Implementation code will be provided upon acceptance.

Title: Cloud Adversarial Example Generation for Remote Sensing Image Classification

• Authors: Fei Ma, Yuqiang Feng, Fan Zhang, Yongsheng Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14240
• Pdf URL: https://arxiv.org/pdf/2409.14240
• Copy Paste: [[2409.14240]] Cloud Adversarial Example Generation for Remote Sensing Image Classification(https://arxiv.org/abs/2409.14240)
• Keywords: defense, attack, robust
• Abstract: Most existing adversarial attack methods for remote sensing images merely add adversarial perturbations or patches, resulting in unnatural modifications. Clouds are common atmospheric effects in remote sensing images. Generating clouds on these images can produce adversarial examples better aligning with human perception. In this paper, we propose a Perlin noise based cloud generation attack method. Common Perlin noise based cloud generation is a random, non-optimizable process, which cannot be directly used to attack the target models. We design a Perlin Gradient Generator Network (PGGN), which takes a gradient parameter vector as input and outputs the grids of Perlin noise gradient vectors at different scales. After a series of computations based on the gradient vectors, cloud masks at corresponding scales can be produced. These cloud masks are then weighted and summed depending on a mixing coefficient vector and a scaling factor to produce the final cloud masks. The gradient vector, coefficient vector and scaling factor are collectively represented as a cloud parameter vector, transforming the cloud generation into a black-box optimization problem. The Differential Evolution (DE) algorithm is employed to solve for the optimal solution of the cloud parameter vector, achieving a query-based black-box attack. Detailed experiments confirm that this method has strong attack capabilities and achieves high query efficiency. Additionally, we analyze the transferability of the generated adversarial examples and their robustness in adversarial defense scenarios.

Title: Perfect Gradient Inversion in Federated Learning: A New Paradigm from the Hidden Subset Sum Problem

• Authors: Qiongxiu Li, Lixia Luo, Agnese Gini, Changlong Ji, Zhanhao Hu, Xiao Li, Chengfang Fang, Jie Shi, Xiaolin Hu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14260
• Pdf URL: https://arxiv.org/pdf/2409.14260
• Copy Paste: [[2409.14260]] Perfect Gradient Inversion in Federated Learning: A New Paradigm from the Hidden Subset Sum Problem(https://arxiv.org/abs/2409.14260)
• Keywords: secure, privacy, defense, attack, federate
• Abstract: Federated Learning (FL) has emerged as a popular paradigm for collaborative learning among multiple parties. It is considered privacy-friendly because local data remains on personal devices, and only intermediate parameters -- such as gradients or model updates -- are shared. Although gradient inversion is widely viewed as a common attack method in FL, analytical research on reconstructing input training samples from shared gradients remains limited and is typically confined to constrained settings like small batch sizes. In this paper, we aim to overcome these limitations by addressing the problem from a cryptographic perspective. We mathematically formulate the input reconstruction problem using the gradient information shared in FL as the Hidden Subset Sum Problem (HSSP), an extension of the well-known NP-complete Subset Sum Problem (SSP). Leveraging this formulation allows us to achieve perfect input reconstruction, thereby mitigating issues such as dependence on label diversity and underperformance with large batch sizes that hinder existing empirical gradient inversion attacks. Moreover, our analysis provides insights into why empirical input reconstruction attacks degrade with larger batch sizes. By modeling the problem as HSSP, we demonstrate that the batch size \( B \) significantly affects attack complexity, with time complexity reaching \( \mathcal{O}(B^9) \). We further show that applying secure data aggregation techniques -- such as homomorphic encryption and secure multiparty computation -- provides a strong defense by increasing the time complexity to \( \mathcal{O}(N^9 B^9) \), where \( N \) is the number of local clients in FL. To the best of our knowledge, this is the first work to rigorously analyze privacy issues in FL by modeling them as HSSP, providing a concrete analytical foundation for further exploration and development of defense strategies.

Title: Re-Evaluating Privacy in Centralized and Decentralized Learning: An Information-Theoretical and Empirical Study

• Authors: Changlong Ji, Stephane Maag, Richard Heusdens, Qiongxiu Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14261
• Pdf URL: https://arxiv.org/pdf/2409.14261
• Copy Paste: [[2409.14261]] Re-Evaluating Privacy in Centralized and Decentralized Learning: An Information-Theoretical and Empirical Study(https://arxiv.org/abs/2409.14261)
• Keywords: secure, privacy, attack, robust, federate
• Abstract: Decentralized Federated Learning (DFL) has garnered attention for its robustness and scalability compared to Centralized Federated Learning (CFL). While DFL is commonly believed to offer privacy advantages due to the decentralized control of sensitive data, recent work by Pasquini et, al. challenges this view, demonstrating that DFL does not inherently improve privacy against empirical attacks under certain assumptions. For investigating fully this issue, a formal theoretical framework is required. Our study offers a novel perspective by conducting a rigorous information-theoretical analysis of privacy leakage in FL using mutual information. We further investigate the effectiveness of privacy-enhancing techniques like Secure Aggregation (SA) in both CFL and DFL. Our simulations and real-world experiments show that DFL generally offers stronger privacy preservation than CFL in practical scenarios where a fully trusted server is not available. We address discrepancies in previous research by highlighting limitations in their assumptions about graph topology and privacy attacks, which inadequately capture information leakage in FL.

Title: Lidar Panoptic Segmentation in an Open World

• Authors: Anirudh S Chakravarthy, Meghana Reddy Ganesina, Peiyun Hu, Laura Leal-Taixe, Shu Kong, Deva Ramanan, Aljosa Osep
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14273
• Pdf URL: https://arxiv.org/pdf/2409.14273
• Copy Paste: [[2409.14273]] Lidar Panoptic Segmentation in an Open World(https://arxiv.org/abs/2409.14273)
• Keywords: segmentation
• Abstract: Addressing Lidar Panoptic Segmentation (LPS ) is crucial for safe deployment of autonomous vehicles. LPS aims to recognize and segment lidar points w.r.t. a pre-defined vocabulary of semantic classes, including thing classes of countable objects (e.g., pedestrians and vehicles) and stuff classes of amorphous regions (e.g., vegetation and road). Importantly, LPS requires segmenting individual thing instances (e.g., every single vehicle). Current LPS methods make an unrealistic assumption that the semantic class vocabulary is fixed in the real open world, but in fact, class ontologies usually evolve over time as robots encounter instances of novel classes that are considered to be unknowns w.r.t. the pre-defined class vocabulary. To address this unrealistic assumption, we study LPS in the Open World (LiPSOW): we train models on a dataset with a pre-defined semantic class vocabulary and study their generalization to a larger dataset where novel instances of thing and stuff classes can appear. This experimental setting leads to interesting conclusions. While prior art train class-specific instance segmentation methods and obtain state-of-the-art results on known classes, methods based on class-agnostic bottom-up grouping perform favorably on classes outside of the initial class vocabulary (i.e., unknown classes). Unfortunately, these methods do not perform on-par with fully data-driven methods on known classes. Our work suggests a middle ground: we perform class-agnostic point clustering and over-segment the input cloud in a hierarchical fashion, followed by binary point segment classification, akin to Region Proposal Network [1]. We obtain the final point cloud segmentation by computing a cut in the weighted hierarchical tree of point segments, independently of semantic classification. Remarkably, this unified approach leads to strong performance on both known and unknown classes.

Title: Dynamic Scattering-channel-based Approach for Multiuser Image Encryption

• Authors: Mohammadrasoul Taghavi, Edwin A. Marengo
• Subjects: cs.CR, physics.optics
• Abstract URL: https://arxiv.org/abs/2409.14275
• Pdf URL: https://arxiv.org/pdf/2409.14275
• Copy Paste: [[2409.14275]] Dynamic Scattering-channel-based Approach for Multiuser Image Encryption(https://arxiv.org/abs/2409.14275)
• Keywords: secure, attack
• Abstract: Conventional scattering-based encryption systems that operate based on a static complex medium which is used by all users are vulnerable to learning-based attacks that exploit ciphertext-plaintext pairs to model and reverse-engineer the scattering medium's response, enabling unauthorized decryption without the physical medium. In this contribution, a new dynamic scattering-channel-based technique for multiuser image encryption is developed. The established approach employs variable, dynamic scattering media which are modeled as tunable aggregates of multiple scattering nanoparticles. The proposed system supports multiple users by allowing distinct combinations of scattering matrices for different time blocks, each combined with user-specific complex-valued coefficients, enabling the creation of unique, hard-to-guess encryption keys for each user. The derived methodology enhances the practical feasibility of multiuser secure communication and storage channels employing scattering media as the encryption mechanism.

Title: ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination

• Authors: Navid Ayoobi, Lily Knab, Wen Cheng, David Pantoja, Hamidreza Alikhani, Sylvain Flamant, Jin Kim, Arjun Mukherjee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14285
• Pdf URL: https://arxiv.org/pdf/2409.14285
• Copy Paste: [[2409.14285]] ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination(https://arxiv.org/abs/2409.14285)
• Keywords: robust, large language model
• Abstract: While large language models (LLMs) exhibit significant utility across various domains, they simultaneously are susceptible to exploitation for unethical purposes, including academic misconduct and dissemination of misinformation. Consequently, AI-generated text detection systems have emerged as a countermeasure. However, these detection mechanisms demonstrate vulnerability to evasion techniques and lack robustness against textual manipulations. This paper introduces back-translation as a novel technique for evading detection, underscoring the need to enhance the robustness of current detection systems. The proposed method involves translating AI-generated text through multiple languages before back-translating to English. We present a model that combines these back-translated texts to produce a manipulated version of the original AI-generated text. Our findings demonstrate that the manipulated text retains the original semantics while significantly reducing the true positive rate (TPR) of existing detection methods. We evaluate this technique on nine AI detectors, including six open-source and three proprietary systems, revealing their susceptibility to back-translation manipulation. In response to the identified shortcomings of existing AI text detectors, we present a countermeasure to improve the robustness against this form of manipulation. Our results indicate that the TPR of the proposed method declines by only 1.85% after back-translation manipulation. Furthermore, we build a large dataset of 720k texts using eight different LLMs. Our dataset contains both human-authored and LLM-generated texts in various domains and writing styles to assess the performance of our method and existing detectors. This dataset is publicly shared for the benefit of the research community.

Title: Deep Learning Technology for Face Forgery Detection: A Survey

• Authors: Lixia Ma, Puning Yang, Yuting Xu, Ziming Yang, Peipei Li, Huaibo Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14289
• Pdf URL: https://arxiv.org/pdf/2409.14289
• Copy Paste: [[2409.14289]] Deep Learning Technology for Face Forgery Detection: A Survey(https://arxiv.org/abs/2409.14289)
• Keywords: security, privacy, generative
• Abstract: Currently, the rapid development of computer vision and deep learning has enabled the creation or manipulation of high-fidelity facial images and videos via deep generative approaches. This technology, also known as deepfake, has achieved dramatic progress and become increasingly popular in social media. However, the technology can generate threats to personal privacy and national security by spreading misinformation. To diminish the risks of deepfake, it is desirable to develop powerful forgery detection methods to distinguish fake faces from real faces. This paper presents a comprehensive survey of recent deep learning-based approaches for facial forgery detection. We attempt to provide the reader with a deeper understanding of the current advances as well as the major challenges for deepfake detection based on deep learning. We present an overview of deepfake techniques and analyse the characteristics of various deepfake datasets. We then provide a systematic review of different categories of deepfake detection and state-of-the-art deepfake detection methods. The drawbacks of existing detection methods are analyzed, and future research directions are discussed to address the challenges in improving both the performance and generalization of deepfake detection.

Title: DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation

• Authors: Xuewen Liu, Zhikai Li, Qingyi Gu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14307
• Pdf URL: https://arxiv.org/pdf/2409.14307
• Copy Paste: [[2409.14307]] DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation(https://arxiv.org/abs/2409.14307)
• Keywords: diffusion
• Abstract: Diffusion models have shown excellent performance on various image generation tasks, but the substantial computational costs and huge memory footprint hinder their low-latency applications in real-world scenarios. Quantization is a promising way to compress and accelerate models. Nevertheless, due to the wide range and time-varying activations in diffusion models, existing methods cannot maintain both accuracy and efficiency simultaneously for low-bit quantization. To tackle this issue, we propose DilateQuant, a novel quantization framework for diffusion models that offers comparable accuracy and high efficiency. Specifically, we keenly aware of numerous unsaturated in-channel weights, which can be cleverly exploited to reduce the range of activations without additional computation cost. Based on this insight, we propose Weight Dilation (WD) that maximally dilates the unsaturated in-channel weights to a constrained range through a mathematically equivalent scaling. WD costlessly absorbs the activation quantization errors into weight quantization. The range of activations decreases, which makes activations quantization easy. The range of weights remains constant, which makes model easy to converge in training stage. Considering the temporal network leads to time-varying activations, we design a Temporal Parallel Quantizer (TPQ), which sets time-step quantization parameters and supports parallel quantization for different time steps, significantly improving the performance and reducing time cost. To further enhance performance while preserving efficiency, we introduce a Block-wise Knowledge Distillation (BKD) to align the quantized models with the full-precision models at a block level. The simultaneous training of time-step quantization parameters and weights minimizes the time required, and the shorter backpropagation paths decreases the memory footprint of the quantization process.

Title: Anisotropic Diffusion Probabilistic Model for Imbalanced Image Classification

• Authors: Jingyu Kong, Yuan Guo, Yu Wang, Yuping Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14313
• Pdf URL: https://arxiv.org/pdf/2409.14313
• Copy Paste: [[2409.14313]] Anisotropic Diffusion Probabilistic Model for Imbalanced Image Classification(https://arxiv.org/abs/2409.14313)
• Keywords: robust, diffusion, generative
• Abstract: Real-world data often has a long-tailed distribution, where the scarcity of tail samples significantly limits the model's generalization ability. Denoising Diffusion Probabilistic Models (DDPM) are generative models based on stochastic differential equation theory and have demonstrated impressive performance in image classification tasks. However, existing diffusion probabilistic models do not perform satisfactorily in classifying tail classes. In this work, we propose the Anisotropic Diffusion Probabilistic Model (ADPM) for imbalanced image classification problems. We utilize the data distribution to control the diffusion speed of different class samples during the forward process, effectively improving the classification accuracy of the denoiser in the reverse process. Specifically, we provide a theoretical strategy for selecting noise levels for different categories in the diffusion process based on error analysis theory to address the imbalanced classification problem. Furthermore, we integrate global and local image prior in the forward process to enhance the model's discriminative ability in the spatial dimension, while incorporate semantic-level contextual information in the reverse process to boost the model's discriminative power and robustness. Through comparisons with state-of-the-art methods on four medical benchmark datasets, we validate the effectiveness of the proposed method in handling long-tail data. Our results confirm that the anisotropic diffusion model significantly improves the classification accuracy of rare classes while maintaining the accuracy of head classes. On the skin lesion datasets, PAD-UFES and HAM10000, the F1-scores of our method improved by 4% and 3%, respectively compared to the original diffusion probabilistic model.

Title: Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses

• Authors: Hung-Ting Su, Ya-Ching Hsu, Xudong Lin, Xiang-Qian Shi, Yulei Niu, Han-Yuan Hsu, Hung-yi Lee, Winston H. Hsu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14324
• Pdf URL: https://arxiv.org/pdf/2409.14324
• Copy Paste: [[2409.14324]] Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses(https://arxiv.org/abs/2409.14324)
• Keywords: large language model
• Abstract: Large language models (LLMs) equipped with chain-of-thoughts (CoT) prompting have shown significant multi-step reasoning capabilities in factual content like mathematics, commonsense, and logic. However, their performance in narrative reasoning, which demands greater abstraction capabilities, remains unexplored. This study utilizes tropes in movie synopses to assess the abstract reasoning abilities of state-of-the-art LLMs and uncovers their low performance. We introduce a trope-wise querying approach to address these challenges and boost the F1 score by 11.8 points. Moreover, while prior studies suggest that CoT enhances multi-step reasoning, this study shows CoT can cause hallucinations in narrative content, reducing GPT-4's performance. We also introduce an Adversarial Injection method to embed trope-related text tokens into movie synopses without explicit tropes, revealing CoT's heightened sensitivity to such injections. Our comprehensive analysis provides insights for future research directions.

Title: Data-Driven Spatiotemporal Feature Representation and Mining in Multidimensional Time Series

• Authors: Xu Yan, Yaoting Jiang, Wenyi Liu, Didi Yi, Haoyang Sang, Jianjun Wei
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14327
• Pdf URL: https://arxiv.org/pdf/2409.14327
• Copy Paste: [[2409.14327]] Data-Driven Spatiotemporal Feature Representation and Mining in Multidimensional Time Series(https://arxiv.org/abs/2409.14327)
• Keywords: interpretability
• Abstract: This paper explores a new method for time series data analysis, aiming to overcome the limitations of traditional mining techniques when dealing with multidimensional time series data. Time series data are extensively utilized in diverse fields, including backend services for monitoring and optimizing IT infrastructure, medical diagnosis through continuous patient monitoring and health trend analysis, and internet business for tracking user behavior and forecasting sales. However, since the effective information in time series data is often hidden in sequence fragments, the uncertainty of their length, quantity, and morphological variables brings challenges to mining. To this end, this paper proposes a new spatiotemporal feature representation method, which converts multidimensional time series (MTS) into one-dimensional event sequences by transforming spatially varying events, and uses a series of event symbols to represent the spatial structural information of multidimensional coupling in the sequence, which has good interpretability. Then, this paper introduces a variable-length tuple mining method to extract non-redundant key event subsequences in event sequences as spatiotemporal structural features of motion sequences. This method is an unsupervised method that does not rely on large-scale training samples and defines a new model for representing the spatiotemporal structural features of multidimensional time series. The superior performance of the STEM model is verified by pattern classification experiments on a variety of motion sequences. The research results of this paper provide an important theoretical basis and technical support for understanding and predicting human behavior patterns, and have far-reaching practical application value.

Title: PISR: Polarimetric Neural Implicit Surface Reconstruction for Textureless and Specular Objects

• Authors: Guangcheng Chen, Yicheng He, Li He, Hong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14331
• Pdf URL: https://arxiv.org/pdf/2409.14331
• Copy Paste: [[2409.14331]] PISR: Polarimetric Neural Implicit Surface Reconstruction for Textureless and Specular Objects(https://arxiv.org/abs/2409.14331)
• Keywords: robust
• Abstract: Neural implicit surface reconstruction has achieved remarkable progress recently. Despite resorting to complex radiance modeling, state-of-the-art methods still struggle with textureless and specular surfaces. Different from RGB images, polarization images can provide direct constraints on the azimuth angles of the surface normals. In this paper, we present PISR, a novel method that utilizes a geometrically accurate polarimetric loss to refine shape independently of appearance. In addition, PISR smooths surface normals in image space to eliminate severe shape distortions and leverages the hash-grid-based neural signed distance function to accelerate the reconstruction. Experimental results demonstrate that PISR achieves higher accuracy and robustness, with an L1 Chamfer distance of 0.5 mm and an F-score of 99.5% at 1 mm, while converging 4~30 times faster than previous polarimetric surface reconstruction methods.

Title: MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators

• Authors: Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, Dacheng Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14335
• Pdf URL: https://arxiv.org/pdf/2409.14335
• Copy Paste: [[2409.14335]] MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators(https://arxiv.org/abs/2409.14335)
• Keywords: interpretability, large language model
• Abstract: Large Language Models (LLMs) have shown significant potential as judges for Machine Translation (MT) quality assessment, providing both scores and fine-grained feedback. Although approaches such as GEMBA-MQM has shown SOTA performance on reference-free evaluation, the predicted errors do not align well with those annotated by human, limiting their interpretability as feedback signals. To enhance the quality of error annotations predicted by LLM evaluators, we introduce a universal and training-free framework, $\textbf{MQM-APE}$, based on the idea of filtering out non-impactful errors by Automatically Post-Editing (APE) the original translation based on each error, leaving only those errors that contribute to quality improvement. Specifically, we prompt the LLM to act as 1) $\textit{evaluator}$ to provide error annotations, 2) $\textit{post-editor}$ to determine whether errors impact quality improvement and 3) $\textit{pairwise quality verifier}$ as the error filter. Experiments show that our approach consistently improves both the reliability and quality of error spans against GEMBA-MQM, across eight LLMs in both high- and low-resource languages. Orthogonal to trained approaches, MQM-APE complements translation-specific evaluators such as Tower, highlighting its broad applicability. Further analysis confirm the effectiveness of each module and offer valuable insights into evaluator design and LLMs selection. The code will be released to facilitate the community.

Title: Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignment

• Authors: Jidong Kuang, Hongsong Wang, Chaolei Han, Jie Gui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14336
• Pdf URL: https://arxiv.org/pdf/2409.14336
• Copy Paste: [[2409.14336]] Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignment(https://arxiv.org/abs/2409.14336)
• Keywords: robust
• Abstract: Zero-shot action recognition, which addresses the issue of scalability and generalization in action recognition and allows the models to adapt to new and unseen actions dynamically, is an important research topic in computer vision communities. The key to zero-shot action recognition lies in aligning visual features with semantic vectors representing action categories. Most existing methods either directly project visual features onto the semantic space of text category or learn a shared embedding space between the two modalities. However, a direct projection cannot accurately align the two modalities, and learning robust and discriminative embedding space between visual and text representations is often difficult. To address these issues, we introduce Dual Visual-Text Alignment (DVTA) for skeleton-based zero-shot action recognition. The DVTA consists of two alignment modules-Direct Alignment (DA) and Augmented Alignment (AA)-along with a designed Semantic Description Enhancement (SDE). The DA module maps the skeleton features to the semantic space through a specially designed visual projector, followed by the SDE, which is based on cross-attention to enhance the connection between skeleton and text, thereby reducing the gap between modalities. The AA module further strengthens the learning of the embedding space by utilizing deep metric learning to learn the similarity between skeleton and text. Our approach achieves state-of-the-art performances on several popular zero-shot skeleton-based action recognition benchmarks.

Title: Self-Supervised Audio-Visual Soundscape Stylization

• Authors: Tingle Li, Renhao Wang, Po-Yao Huang, Andrew Owens, Gopala Anumanchipalli
• Subjects: cs.CV, cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.14340
• Pdf URL: https://arxiv.org/pdf/2409.14340
• Copy Paste: [[2409.14340]] Self-Supervised Audio-Visual Soundscape Stylization(https://arxiv.org/abs/2409.14340)
• Keywords: diffusion
• Abstract: Speech sounds convey a great deal of information about the scenes, resulting in a variety of effects ranging from reverberation to additional ambient sounds. In this paper, we manipulate input speech to sound as though it was recorded within a different scene, given an audio-visual conditional example recorded from that scene. Our model learns through self-supervision, taking advantage of the fact that natural video contains recurring sound events and textures. We extract an audio clip from a video and apply speech enhancement. We then train a latent diffusion model to recover the original speech, using another audio-visual clip taken from elsewhere in the video as a conditional hint. Through this process, the model learns to transfer the conditional example's sound properties to the input speech. We show that our model can be successfully trained using unlabeled, in-the-wild videos, and that an additional visual signal can improve its sound prediction abilities. Please see our project webpage for video results: https://tinglok.netlify.app/files/avsoundscape/

Title: Memory Matching is not Enough: Jointly Improving Memory Matching and Decoding for Video Object Segmentation

• Authors: Jintu Zheng, Yun Liang, Yuqing Zhang, Wanchao Su
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.14343
• Pdf URL: https://arxiv.org/pdf/2409.14343
• Copy Paste: [[2409.14343]] Memory Matching is not Enough: Jointly Improving Memory Matching and Decoding for Video Object Segmentation(https://arxiv.org/abs/2409.14343)
• Keywords: segmentation
• Abstract: Memory-based video object segmentation methods model multiple objects over long temporal-spatial spans by establishing memory bank, which achieve the remarkable performance. However, they struggle to overcome the false matching and are prone to lose critical information, resulting in confusion among different objects. In this paper, we propose an effective approach which jointly improving the matching and decoding stages to alleviate the false matching issue.For the memory matching stage, we present a cost aware mechanism that suppresses the slight errors for short-term memory and a shunted cross-scale matching for long-term memory which establish a wide filed matching spaces for various object scales. For the readout decoding stage, we implement a compensatory mechanism aims at recovering the essential information where missing at the matching stage. Our approach achieves the outstanding performance in several popular benchmarks (i.e., DAVIS 2016&2017 Val (92.4%&88.1%), and DAVIS 2017 Test (83.9%)), and achieves 84.8%&84.6% on YouTubeVOS 2018&2019 Val.

Title: Using Natural Language Processing to find Indication for Burnout with Text Classification: From Online Data to Real-World Data

• Authors: Mascha Kurpicz-Briki, Ghofrane Merhbene, Alexandre Puttick, Souhir Ben Souissi, Jannic Bieri, Thomas Jörg Müller, Christoph Golz
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14357
• Pdf URL: https://arxiv.org/pdf/2409.14357
• Copy Paste: [[2409.14357]] Using Natural Language Processing to find Indication for Burnout with Text Classification: From Online Data to Real-World Data(https://arxiv.org/abs/2409.14357)
• Keywords: interpretability
• Abstract: Burnout, classified as a syndrome in the ICD-11, arises from chronic workplace stress that has not been effectively managed. It is characterized by exhaustion, cynicism, and reduced professional efficacy, and estimates of its prevalence vary significantly due to inconsistent measurement methods. Recent advancements in Natural Language Processing (NLP) and machine learning offer promising tools for detecting burnout through textual data analysis, with studies demonstrating high predictive accuracy. This paper contributes to burnout detection in German texts by: (a) collecting an anonymous real-world dataset including free-text answers and Oldenburg Burnout Inventory (OLBI) responses; (b) demonstrating the limitations of a GermanBERT-based classifier trained on online data; (c) presenting two versions of a curated BurnoutExpressions dataset, which yielded models that perform well in real-world applications; and (d) providing qualitative insights from an interdisciplinary focus group on the interpretability of AI models used for burnout detection. Our findings emphasize the need for greater collaboration between AI researchers and clinical experts to refine burnout detection models. Additionally, more real-world data is essential to validate and enhance the effectiveness of current AI methods developed in NLP research, which are often based on data automatically scraped from online sources and not evaluated in a real-world context. This is essential for ensuring AI tools are well suited for practical applications.

Title: More Effective LLM Compressed Tokens with Uniformly Spread Position Identifiers and Compression Loss

• Authors: Runsong Zhao, Pengcheng Huang, Xinyu Liu, Chunyang Xiao, Tong Xiao, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14364
• Pdf URL: https://arxiv.org/pdf/2409.14364
• Copy Paste: [[2409.14364]] More Effective LLM Compressed Tokens with Uniformly Spread Position Identifiers and Compression Loss(https://arxiv.org/abs/2409.14364)
• Keywords: transformer
• Abstract: Compressing Transformer inputs into compressd tokens allows running LLMs with improved speed and cost efficiency. Based on the compression method ICAE, we carefully examine the position identifier choices for compressed tokens and also propose a new compression loss. We demonstrate empirically that our proposed methods achieve significantly higher compression ratios (15x compared to 4x for ICAE), while being able to attain comparable reconstruction performance.

Title: The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests

• Authors: Lior Madmoni, Amir Zait, Ilia Labzovsky, Danny Karmon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14371
• Pdf URL: https://arxiv.org/pdf/2409.14371
• Copy Paste: [[2409.14371]] The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests(https://arxiv.org/abs/2409.14371)
• Keywords: extraction, generative, large language model
• Abstract: Generative AI agents are often expected to respond to complex user requests that have No One Right Answer (NORA), e.g., "design a vegetarian meal plan below 1800 calories". Such requests may entail a set of constraints that the agent should adhere to. To successfully develop agents for NORA scenarios, an accurate automatic evaluation framework is essential, and specifically - one capable of validating the satisfaction of constraints in the agent's response. Recently, large language models (LLMs) have been adopted as versatile evaluators for many NORA tasks, but their ability to evaluate constraint-satisfaction in generated text remains unclear. To study this, we develop and release a novel Arithmetic Constraint-Satisfaction (ACS) benchmarking dataset. The dataset consists of complex user requests with corresponding constraints, agent responses and human labels indicating each constraint's satisfaction level in the response. A unique property of this dataset is that validating many of its constraints requires reviewing the response as a whole (in contrast to many other benchmarks that require the validation of a single independent item). Moreover, it assesses LLMs in performing reasoning, in-context data extraction, arithmetic calculations, and counting. We then benchmark both open and proprietary LLMs on evaluating constraint-satisfaction, and show that most models still have a significant headroom for improvement, and that errors primarily stem from reasoning issues. In addition, most models exhibit a skewed constraint-satisfaction prediction pattern, with higher accuracy where the ground-truth label is "satisfied". Lastly, few-shot prompting for our task proved to be rather challenging, since many of the studied models showed a degradation in performance when it was introduced.

Title: Sparse Low-Ranked Self-Attention Transformer for Remaining Useful Lifetime Prediction of Optical Fiber Amplifiers

• Authors: Dominic Schneider, Lutz Rapp
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.14378
• Pdf URL: https://arxiv.org/pdf/2409.14378
• Copy Paste: [[2409.14378]] Sparse Low-Ranked Self-Attention Transformer for Remaining Useful Lifetime Prediction of Optical Fiber Amplifiers(https://arxiv.org/abs/2409.14378)
• Keywords: transformer
• Abstract: Optical fiber amplifiers are key elements in present optical networks. Failures of these components result in high financial loss of income of the network operator as the communication traffic over an affected link is interrupted. Applying Remaining useful lifetime (RUL) prediction in the context of Predictive Maintenance (PdM) to optical fiber amplifiers to predict upcoming system failures at an early stage, so that network outages can be minimized through planning of targeted maintenance actions, ensures reliability and safety. Optical fiber amplifier are complex systems, that work under various operating conditions, which makes correct forecasting a difficult task. Increased monitoring capabilities of systems results in datasets that facilitate the application of data-driven RUL prediction methods. Deep learning models in particular have shown good performance, but generalization based on comparatively small datasets for RUL prediction is difficult. In this paper, we propose Sparse Low-ranked self-Attention Transformer (SLAT) as a novel RUL prediction method. SLAT is based on an encoder-decoder architecture, wherein two parallel working encoders extract features for sensors and time steps. By utilizing the self-attention mechanism, long-term dependencies can be learned from long sequences. The implementation of sparsity in the attention matrix and a low-rank parametrization reduce overfitting and increase generalization. Experimental application to optical fiber amplifiers exemplified on EDFA, as well as a reference dataset from turbofan engines, shows that SLAT outperforms the state-of-the-art methods.

Title: GroupDiff: Diffusion-based Group Portrait Editing

• Authors: Yuming Jiang, Nanxuan Zhao, Qing Liu, Krishna Kumar Singh, Shuai Yang, Chen Change Loy, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14379
• Pdf URL: https://arxiv.org/pdf/2409.14379
• Copy Paste: [[2409.14379]] GroupDiff: Diffusion-based Group Portrait Editing(https://arxiv.org/abs/2409.14379)
• Keywords: diffusion
• Abstract: Group portrait editing is highly desirable since users constantly want to add a person, delete a person, or manipulate existing persons. It is also challenging due to the intricate dynamics of human interactions and the diverse gestures. In this work, we present GroupDiff, a pioneering effort to tackle group photo editing with three dedicated contributions: 1) Data Engine: Since there is no labeled data for group photo editing, we create a data engine to generate paired data for training. The training data engine covers the diverse needs of group portrait editing. 2) Appearance Preservation: To keep the appearance consistent after editing, we inject the images of persons from the group photo into the attention modules and employ skeletons to provide intra-person guidance. 3) Control Flexibility: Bounding boxes indicating the locations of each person are used to reweight the attention matrix so that the features of each person can be injected into the correct places. This inter-person guidance provides flexible manners for manipulation. Extensive experiments demonstrate that GroupDiff exhibits state-of-the-art performance compared to existing methods. GroupDiff offers controllability for editing and maintains the fidelity of the original photos.

Title: Investigating Layer Importance in Large Language Models

• Authors: Yang Zhang, Yanfei Dong, Kenji Kawaguchi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14381
• Pdf URL: https://arxiv.org/pdf/2409.14381
• Copy Paste: [[2409.14381]] Investigating Layer Importance in Large Language Models(https://arxiv.org/abs/2409.14381)
• Keywords: large language model
• Abstract: Large language models (LLMs) have gained increasing attention due to their prominent ability to understand and process texts. Nevertheless, LLMs largely remain opaque. The lack of understanding of LLMs has obstructed the deployment in safety-critical scenarios and hindered the development of better models. In this study, we advance the understanding of LLM by investigating the significance of individual layers in LLMs. We propose an efficient sampling method to faithfully evaluate the importance of layers using Shapley values, a widely used explanation framework in feature attribution and data valuation. In addition, we conduct layer ablation experiments to assess the performance degradation resulting from the exclusion of specific layers. Our findings reveal the existence of cornerstone layers, wherein certain early layers can exhibit a dominant contribution over others. Removing one cornerstone layer leads to a drastic collapse of the model performance, often reducing it to random guessing. Conversely, removing non-cornerstone layers results in only marginal performance changes. This study identifies cornerstone layers in LLMs and underscores their critical role for future research.

Title: Prior Knowledge Distillation Network for Face Super-Resolution

• Authors: Qiu Yang, Xiao Sun, Xin-yu Li, Feng-Qi Cui, Yu-Tong Guo, Shuang-Zhen Hu, Ping Luo, Si-Ying Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14385
• Pdf URL: https://arxiv.org/pdf/2409.14385
• Copy Paste: [[2409.14385]] Prior Knowledge Distillation Network for Face Super-Resolution(https://arxiv.org/abs/2409.14385)
• Keywords: robust, extraction
• Abstract: The purpose of face super-resolution (FSR) is to reconstruct high-resolution (HR) face images from low-resolution (LR) inputs. With the continuous advancement of deep learning technologies, contemporary prior-guided FSR methods initially estimate facial priors and then use this information to assist in the super-resolution reconstruction process. However, ensuring the accuracy of prior estimation remains challenging, and straightforward cascading and convolutional operations often fail to fully leverage prior knowledge. Inaccurate or insufficiently utilized prior information inevitably degrades FSR performance. To address this issue, we propose a prior knowledge distillation network (PKDN) for FSR, which involves transferring prior information from the teacher network to the student network. This approach enables the network to learn priors during the training stage while relying solely on low-resolution facial images during the testing stage, thus mitigating the adverse effects of prior estimation inaccuracies. Additionally, we incorporate robust attention mechanisms to design a parsing map fusion block that effectively utilizes prior information. To prevent feature loss, we retain multi-scale features during the feature extraction stage and employ them in the subsequent super-resolution reconstruction process. Experimental results on benchmark datasets demonstrate that our PKDN approach surpasses existing FSR methods in generating high-quality face images.

Title: Predicting User Stances from Target-Agnostic Information using Large Language Models

• Authors: Siyuan Brandon Loh, Liang Ze Wong, Prasanta Bhattacharya, Joseph Simons, Wei Gao, Hong Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14395
• Pdf URL: https://arxiv.org/pdf/2409.14395
• Copy Paste: [[2409.14395]] Predicting User Stances from Target-Agnostic Information using Large Language Models(https://arxiv.org/abs/2409.14395)
• Keywords: large language model
• Abstract: We investigate Large Language Models' (LLMs) ability to predict a user's stance on a target given a collection of his/her target-agnostic social media posts (i.e., user-level stance prediction). While we show early evidence that LLMs are capable of this task, we highlight considerable variability in the performance of the model across (i) the type of stance target, (ii) the prediction strategy and (iii) the number of target-agnostic posts supplied. Post-hoc analyses further hint at the usefulness of target-agnostic posts in providing relevant information to LLMs through the presence of both surface-level (e.g., target-relevant keywords) and user-level features (e.g., encoding users' moral values). Overall, our findings suggest that LLMs might offer a viable method for determining public stances towards new topics based on historical and target-agnostic data. At the same time, we also call for further research to better understand LLMs' strong performance on the stance prediction task and how their effectiveness varies across task contexts.

Title: Beyond Persuasion: Towards Conversational Recommender System with Credible Explanations

• Authors: Peixin Qin, Chen Huang, Yang Deng, Wenqiang Lei, Tat-Seng Chua
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14399
• Pdf URL: https://arxiv.org/pdf/2409.14399
• Copy Paste: [[2409.14399]] Beyond Persuasion: Towards Conversational Recommender System with Credible Explanations(https://arxiv.org/abs/2409.14399)
• Keywords: large language model
• Abstract: With the aid of large language models, current conversational recommender system (CRS) has gaining strong abilities to persuade users to accept recommended items. While these CRSs are highly persuasive, they can mislead users by incorporating incredible information in their explanations, ultimately damaging the long-term trust between users and the CRS. To address this, we propose a simple yet effective method, called PC-CRS, to enhance the credibility of CRS's explanations during persuasion. It guides the explanation generation through our proposed credibility-aware persuasive strategies and then gradually refines explanations via post-hoc self-reflection. Experimental results demonstrate the efficacy of PC-CRS in promoting persuasive and credible explanations. Further analysis reveals the reason behind current methods producing incredible explanations and the potential of credible explanations to improve recommendation accuracy.

Title: COSBO: Conservative Offline Simulation-Based Policy Optimization

• Authors: Eshagh Kargar, Ville Kyrki
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.14412
• Pdf URL: https://arxiv.org/pdf/2409.14412
• Copy Paste: [[2409.14412]] COSBO: Conservative Offline Simulation-Based Policy Optimization(https://arxiv.org/abs/2409.14412)
• Keywords: robust
• Abstract: Offline reinforcement learning allows training reinforcement learning models on data from live deployments. However, it is limited to choosing the best combination of behaviors present in the training data. In contrast, simulation environments attempting to replicate the live environment can be used instead of the live data, yet this approach is limited by the simulation-to-reality gap, resulting in a bias. In an attempt to get the best of both worlds, we propose a method that combines an imperfect simulation environment with data from the target environment, to train an offline reinforcement learning policy. Our experiments demonstrate that the proposed method outperforms state-of-the-art approaches CQL, MOPO, and COMBO, especially in scenarios with diverse and challenging dynamics, and demonstrates robust behavior across a variety of experimental conditions. The results highlight that using simulator-generated data can effectively enhance offline policy learning despite the sim-to-real gap, when direct interaction with the real-world is not possible.

Title: Uncovering EDK2 Firmware Flaws: Insights from Code Audit Tools

• Authors: Mahsa Farahani, Ghazal Shenavar, Ali Hosseinghorban, Alireza Ejlali
• Subjects: cs.CR, cs.AR, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.14416
• Pdf URL: https://arxiv.org/pdf/2409.14416
• Copy Paste: [[2409.14416]] Uncovering EDK2 Firmware Flaws: Insights from Code Audit Tools(https://arxiv.org/abs/2409.14416)
• Keywords: security
• Abstract: Firmware serves as a foundational software layer in modern computers, initiating as the first code executed on platform hardware, similar in function to a minimal operating system. Defined as a software interface between an operating system and platform firmware, the Unified Extensible Firmware Interface (UEFI) standardizes system initialization and management. A prominent open-source implementation of UEFI, the EFI Development Kit II (EDK2), plays a crucial role in shaping firmware architecture. Despite its widespread adoption, the architecture faces challenges such as limited system resources at early stages and a lack of standard security features. Furthermore, the scarcity of open-source tools specifically designed for firmware analysis emphasizes the need for adaptable, innovative solutions. In this paper, we explore the application of general code audit tools to firmware, with a particular focus on EDK2. Although these tools were not originally designed for firmware analysis, they have proven effective in identifying critical areas for enhancement in firmware security. Our findings, derived from deploying key audit tools on EDK2, categorize these tools based on their methodologies and illustrate their capability to uncover unique firmware attributes, significantly contributing to the understanding and improvement of firmware security.

Title: Dormant: Defending against Pose-driven Human Image Animation

• Authors: Jiachen Zhou, Mingsi Wang, Tianlin Li, Guozhu Meng, Kai Chen
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14424
• Pdf URL: https://arxiv.org/pdf/2409.14424
• Copy Paste: [[2409.14424]] Dormant: Defending against Pose-driven Human Image Animation(https://arxiv.org/abs/2409.14424)
• Keywords: protect, attack, extraction
• Abstract: Pose-driven human image animation has achieved tremendous progress, enabling the generation of vivid and realistic human videos from just one single photo. However, it conversely exacerbates the risk of image misuse, as attackers may use one available image to create videos involving politics, violence and other illegal content. To counter this threat, we propose Dormant, a novel protection approach tailored to defend against pose-driven human image animation techniques. Dormant applies protective perturbation to one human image, preserving the visual similarity to the original but resulting in poor-quality video generation. The protective perturbation is optimized to induce misextraction of appearance features from the image and create incoherence among the generated video frames. Our extensive evaluation across 8 animation methods and 4 datasets demonstrates the superiority of Dormant over 6 baseline protection methods, leading to misaligned identities, visual distortions, noticeable artifacts, and inconsistent frames in the generated videos. Moreover, Dormant shows effectiveness on 6 real-world commercial services, even with fully black-box access.

Title: Challenging the Performance-Interpretability Trade-off: An Evaluation of Interpretable Machine Learning Models

• Authors: Sven Kruschel, Nico Hambauer, Sven Weinzierl, Sandra Zilker, Mathias Kraus, Patrick Zschech
• Subjects: cs.LG, cs.AI, cs.HC, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.14429
• Pdf URL: https://arxiv.org/pdf/2409.14429
• Copy Paste: [[2409.14429]] Challenging the Performance-Interpretability Trade-off: An Evaluation of Interpretable Machine Learning Models(https://arxiv.org/abs/2409.14429)
• Keywords: robust, fair, interpretability
• Abstract: Machine learning is permeating every conceivable domain to promote data-driven decision support. The focus is often on advanced black-box models due to their assumed performance advantages, whereas interpretable models are often associated with inferior predictive qualities. More recently, however, a new generation of generalized additive models (GAMs) has been proposed that offer promising properties for capturing complex, non-linear patterns while remaining fully interpretable. To uncover the merits and limitations of these models, this study examines the predictive performance of seven different GAMs in comparison to seven commonly used machine learning models based on a collection of twenty tabular benchmark datasets. To ensure a fair and robust model comparison, an extensive hyperparameter search combined with cross-validation was performed, resulting in 68,500 model runs. In addition, this study qualitatively examines the visual output of the models to assess their level of interpretability. Based on these results, the paper dispels the misconception that only black-box models can achieve high accuracy by demonstrating that there is no strict trade-off between predictive performance and model interpretability for tabular data. Furthermore, the paper discusses the importance of GAMs as powerful interpretable models for the field of information systems and derives implications for future work from a socio-technical perspective.

Title: EM-DARTS: Hierarchical Differentiable Architecture Search for Eye Movement Recognition

• Authors: Huafeng Qin, Hongyu Zhu, Xin Jin, Xin Yu, Mounim A. El-Yacoubi, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14432
• Pdf URL: https://arxiv.org/pdf/2409.14432
• Copy Paste: [[2409.14432]] EM-DARTS: Hierarchical Differentiable Architecture Search for Eye Movement Recognition(https://arxiv.org/abs/2409.14432)
• Keywords: secure, biometric
• Abstract: Eye movement biometrics has received increasing attention thanks to its high secure identification. Although deep learning (DL) models have been recently successfully applied for eye movement recognition, the DL architecture still is determined by human prior knowledge. Differentiable Neural Architecture Search (DARTS) automates the manual process of architecture design with high search efficiency. DARTS, however, usually stacks the same multiple learned cells to form a final neural network for evaluation, limiting therefore the diversity of the network. Incidentally, DARTS usually searches the architecture in a shallow network while evaluating it in a deeper one, which results in a large gap between the architecture depths in the search and evaluation scenarios. To address this issue, we propose EM-DARTS, a hierarchical differentiable architecture search algorithm to automatically design the DL architecture for eye movement recognition. First, we define a supernet and propose a global and local alternate Neural Architecture Search method to search the optimal architecture alternately with an differentiable neural architecture search. The local search strategy aims to find an optimal architecture for different cells while the global search strategy is responsible for optimizing the architecture of the target network. To further reduce redundancy, a transfer entropy is proposed to compute the information amount of each layer, so as to further simplify search network. Our experiments on three public databases demonstrate that the proposed EM-DARTS is capable of producing an optimal architecture that leads to state-of-the-art recognition performance.

Title: Automotive innovation landscaping using LLM

• Authors: Raju Gorain, Omkar Salunke
• Subjects: cs.CL, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.14436
• Pdf URL: https://arxiv.org/pdf/2409.14436
• Copy Paste: [[2409.14436]] Automotive innovation landscaping using LLM(https://arxiv.org/abs/2409.14436)
• Keywords: extraction, large language model
• Abstract: The process of landscaping automotive innovation through patent analysis is crucial for Research and Development teams. It aids in comprehending innovation trends, technological advancements, and the latest technologies from competitors. Traditionally, this process required intensive manual efforts. However, with the advent of Large Language Models (LLMs), it can now be automated, leading to faster and more efficient patent categorization & state-of-the-art of inventive concept extraction. This automation can assist various R\&D teams in extracting relevant information from extensive patent databases. This paper introduces a method based on prompt engineering to extract essential information for landscaping. The information includes the problem addressed by the patent, the technology utilized, and the area of innovation within the vehicle ecosystem (such as safety, Advanced Driver Assistance Systems and more).The result demonstrates the implementation of this method to create a landscape of fuel cell technology using open-source patent data. This approach provides a comprehensive overview of the current state of fuel cell technology, offering valuable insights for future research and development in this field.

Title: A Visualized Malware Detection Framework with CNN and Conditional GAN

• Authors: Fang Wang (Florence Wong), Hussam Al Hamadi, Ernesto Damiani
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14439
• Pdf URL: https://arxiv.org/pdf/2409.14439
• Copy Paste: [[2409.14439]] A Visualized Malware Detection Framework with CNN and Conditional GAN(https://arxiv.org/abs/2409.14439)
• Keywords: security, defense, generative
• Abstract: Malware visualization analysis incorporating with Machine Learning (ML) has been proven to be a promising solution for improving security defenses on different platforms. In this work, we propose an integrated framework for addressing common problems experienced by ML utilizers in developing malware detection systems. Namely, a pictorial presentation system with extensions is designed to preserve the identities of benign/malign samples by encoding each variable into binary digits and mapping them into black and white pixels. A conditional Generative Adversarial Network based model is adopted to produce synthetic images and mitigate issues of imbalance classes. Detection models architected by Convolutional Neural Networks are for validating performances while training on datasets with and without artifactual samples. Result demonstrates accuracy rates of 98.51% and 97.26% for these two training scenarios.

Title: Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysis

• Authors: Daoyang Li, Mingyu Jin, Qingcheng Zeng, Haiyan Zhao, Mengnan Du
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14459
• Pdf URL: https://arxiv.org/pdf/2409.14459
• Copy Paste: [[2409.14459]] Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysis(https://arxiv.org/abs/2409.14459)
• Keywords: large language model
• Abstract: Probing techniques for large language models (LLMs) have primarily focused on English, overlooking the vast majority of the world's languages. In this paper, we extend these probing methods to a multilingual context, investigating the behaviors of LLMs across diverse languages. We conduct experiments on several open-source LLM models, analyzing probing accuracy, trends across layers, and similarities between probing vectors for multiple languages. Our key findings reveal: (1) a consistent performance gap between high-resource and low-resource languages, with high-resource languages achieving significantly higher probing accuracy; (2) divergent layer-wise accuracy trends, where high-resource languages show substantial improvement in deeper layers similar to English; and (3) higher representational similarities among high-resource languages, with low-resource languages demonstrating lower similarities both among themselves and with high-resource languages. These results highlight significant disparities in LLMs' multilingual capabilities and emphasize the need for improved modeling of low-resource languages.

Title: Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints

• Authors: Kaikai An, Shuzheng Si, Helan Hu, Haozhe Zhao, Yuchi Wang, Qingyan Guo, Baobao Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14469
• Pdf URL: https://arxiv.org/pdf/2409.14469
• Copy Paste: [[2409.14469]] Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints(https://arxiv.org/abs/2409.14469)
• Keywords: large language model
• Abstract: Semantic Parsing aims to capture the meaning of a sentence and convert it into a logical, structured form. Previous studies show that semantic parsing enhances the performance of smaller models (e.g., BERT) on downstream tasks. However, it remains unclear whether the improvements extend similarly to LLMs. In this paper, our empirical findings reveal that, unlike smaller models, directly adding semantic parsing results into LLMs reduces their performance. To overcome this, we propose SENSE, a novel prompting approach that embeds semantic hints within the prompt. Experiments show that SENSE consistently improves LLMs' performance across various tasks, highlighting the potential of integrating semantic information to improve LLM capabilities.

Title: Blockchain Based Information Security and Privacy Protection: Challenges and Future Directions using Computational Literature Review

• Authors: Gauri Shankar, Md Raihan Uddin, Saddam Mukta, Prabhat Kumar, Shareeful Islam, A.K.M. Najmul Islam
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14472
• Pdf URL: https://arxiv.org/pdf/2409.14472
• Copy Paste: [[2409.14472]] Blockchain Based Information Security and Privacy Protection: Challenges and Future Directions using Computational Literature Review(https://arxiv.org/abs/2409.14472)
• Keywords: security, privacy, protect
• Abstract: Blockchain technology is an emerging digital innovation that has gained immense popularity in enhancing individual security and privacy within Information Systems (IS). This surge in interest is reflected in the exponential increase in research articles published on blockchain technology, highlighting its growing significance in the digital landscape. However, the rapid proliferation of published research presents significant challenges for manual analysis and synthesis due to the vast volume of information. The complexity and breadth of topics, combined with the inherent limitations of human data processing capabilities, make it difficult to comprehensively analyze and draw meaningful insights from the literature. To this end, we adopted the Computational Literature Review (CLR) to analyze pertinent literature impact and topic modelling using the Latent Dirichlet Allocation (LDA) technique. We identified 10 topics related to security and privacy and provided a detailed description of each topic. From the critical analysis, we have observed several limitations, and several future directions are provided as an outcome of this review.

Title: SynBench: A Synthetic Benchmark for Non-rigid 3D Point Cloud Registration

• Authors: Sara Monji-Azad, Marvin Kinz, Claudia Scherl, David Männle, Jürgen Hesser, Nikolas Löw
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14474
• Pdf URL: https://arxiv.org/pdf/2409.14474
• Copy Paste: [[2409.14474]] SynBench: A Synthetic Benchmark for Non-rigid 3D Point Cloud Registration(https://arxiv.org/abs/2409.14474)
• Keywords: fair
• Abstract: Non-rigid point cloud registration is a crucial task in computer vision. Evaluating a non-rigid point cloud registration method requires a dataset with challenges such as large deformation levels, noise, outliers, and incompleteness. Despite the existence of several datasets for deformable point cloud registration, the absence of a comprehensive benchmark with all challenges makes it difficult to achieve fair evaluations among different methods. This paper introduces SynBench, a new non-rigid point cloud registration dataset created using SimTool, a toolset for soft body simulation in Flex and Unreal Engine. SynBench provides the ground truth of corresponding points between two point sets and encompasses key registration challenges, including varying levels of deformation, noise, outliers, and incompleteness. To the best of the authors' knowledge, compared to existing datasets, SynBench possesses three particular characteristics: (1) it is the first benchmark that provides various challenges for non-rigid point cloud registration, (2) SynBench encompasses challenges of varying difficulty levels, and (3) it includes ground truth corresponding points both before and after deformation. The authors believe that SynBench enables future non-rigid point cloud registration methods to present a fair comparison of their achievements. SynBench is publicly available at: this https URL.

Title: Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding

• Authors: Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, Bo Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14485
• Pdf URL: https://arxiv.org/pdf/2409.14485
• Copy Paste: [[2409.14485]] Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding(https://arxiv.org/abs/2409.14485)
• Keywords: large language model
• Abstract: Although current Multi-modal Large Language Models (MLLMs) demonstrate promising results in video understanding, processing extremely long videos remains an ongoing challenge. Typically, MLLMs struggle with handling thousands of tokens that exceed the maximum context length of LLMs, and they experience reduced visual clarity due to token aggregation. Another challenge is the high computational cost stemming from the large number of video tokens. To tackle these issues, we propose Video-XL, an extra-long vision language model designed for efficient hour-scale video understanding. Specifically, we argue that LLMs can be adapted as effective visual condensers and introduce Visual Context Latent Summarization, which condenses visual contexts into highly compact forms. Extensive experiments demonstrate that our model achieves promising results on popular long video understanding benchmarks, despite being trained on limited image data. Moreover, Video-XL strikes a promising balance between efficiency and effectiveness, processing 1024 frames on a single 80GB GPU while achieving nearly 100\% accuracy in the Needle-in-a-Haystack evaluation. We envision Video-XL becoming a valuable tool for long video applications such as video summarization, surveillance anomaly detection, and Ad placement identification.

Title: Enhancing LLM-based Autonomous Driving Agents to Mitigate Perception Attacks

• Authors: Ruoyu Song, Muslum Ozgur Ozmen, Hyungsub Kim, Antonio Bianchi, Z. Berkay Celik
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14488
• Pdf URL: https://arxiv.org/pdf/2409.14488
• Copy Paste: [[2409.14488]] Enhancing LLM-based Autonomous Driving Agents to Mitigate Perception Attacks(https://arxiv.org/abs/2409.14488)
• Keywords: attack, large language model
• Abstract: There is a growing interest in integrating Large Language Models (LLMs) with autonomous driving (AD) systems. However, AD systems are vulnerable to attacks against their object detection and tracking (ODT) functions. Unfortunately, our evaluation of four recent LLM agents against ODT attacks shows that the attacks are 63.26% successful in causing them to crash or violate traffic rules due to (1) misleading memory modules that provide past experiences for decision making, (2) limitations of prompts in identifying inconsistencies, and (3) reliance on ground truth perception data. In this paper, we introduce Hudson, a driving reasoning agent that extends prior LLM-based driving systems to enable safer decision making during perception attacks while maintaining effectiveness under benign conditions. Hudson achieves this by first instrumenting the AD software to collect real-time perception results and contextual information from the driving scene. This data is then formalized into a domain-specific language (DSL). To guide the LLM in detecting and making safe control decisions during ODT attacks, Hudson translates the DSL into natural language, along with a list of custom attack detection instructions. Following query execution, Hudson analyzes the LLM's control decision to understand its causal reasoning process. We evaluate the effectiveness of Hudson using a proprietary LLM (GPT-4) and two open-source LLMs (Llama and Gemma) in various adversarial driving scenarios. GPT-4, Llama, and Gemma achieve, on average, an attack detection accuracy of 83. 3%, 63. 6%, and 73. 6%. Consequently, they make safe control decisions in 86.4%, 73.9%, and 80% of the attacks. Our results, following the growing interest in integrating LLMs into AD systems, highlight the strengths of LLMs and their potential to detect and mitigate ODT attacks.

Title: CPT-Boosted Wav2vec2.0: Towards Noise Robust Speech Recognition for Classroom Environments

• Authors: Ahmed Adel Attia, Dorottya Demszky, Tolulope Ogunremi, Jing Liu, Carol Espy-Wilson
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.14494
• Pdf URL: https://arxiv.org/pdf/2409.14494
• Copy Paste: [[2409.14494]] CPT-Boosted Wav2vec2.0: Towards Noise Robust Speech Recognition for Classroom Environments(https://arxiv.org/abs/2409.14494)
• Keywords: robust
• Abstract: Creating Automatic Speech Recognition (ASR) systems that are robust and resilient to classroom conditions is paramount to the development of AI tools to aid teachers and students. In this work, we study the efficacy of continued pretraining (CPT) in adapting Wav2vec2.0 to the classroom domain. We show that CPT is a powerful tool in that regard and reduces the Word Error Rate (WER) of Wav2vec2.0-based models by upwards of 10%. More specifically, CPT improves the model's robustness to different noises, microphones and classroom conditions.

Title: A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders

• Authors: David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Joseph Bloom
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14507
• Pdf URL: https://arxiv.org/pdf/2409.14507
• Copy Paste: [[2409.14507]] A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders(https://arxiv.org/abs/2409.14507)
• Keywords: interpretability, large language model
• Abstract: Sparse Autoencoders (SAEs) have emerged as a promising approach to decompose the activations of Large Language Models (LLMs) into human-interpretable latents. In this paper, we pose two questions. First, to what extent do SAEs extract monosemantic and interpretable latents? Second, to what extent does varying the sparsity or the size of the SAE affect monosemanticity / interpretability? By investigating these questions in the context of a simple first-letter identification task where we have complete access to ground truth labels for all tokens in the vocabulary, we are able to provide more detail than prior investigations. Critically, we identify a problematic form of feature-splitting we call feature absorption where seemingly monosemantic latents fail to fire in cases where they clearly should. Our investigation suggests that varying SAE size or sparsity is insufficient to solve this issue, and that there are deeper conceptual issues in need of resolution.

Title: Order of Magnitude Speedups for LLM Membership Inference

• Authors: Martin Bertran, Rongting Zhang, Aaron Roth
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.14513
• Pdf URL: https://arxiv.org/pdf/2409.14513
• Copy Paste: [[2409.14513]] Order of Magnitude Speedups for LLM Membership Inference(https://arxiv.org/abs/2409.14513)
• Keywords: privacy, attack, robust, membership infer, large language model
• Abstract: Large Language Models (LLMs) have the promise to revolutionize computing broadly, but their complexity and extensive training data also expose significant privacy vulnerabilities. One of the simplest privacy risks associated with LLMs is their susceptibility to membership inference attacks (MIAs), wherein an adversary aims to determine whether a specific data point was part of the model's training set. Although this is a known risk, state of the art methodologies for MIAs rely on training multiple computationally costly shadow models, making risk evaluation prohibitive for large models. Here we adapt a recent line of work which uses quantile regression to mount membership inference attacks; we extend this work by proposing a low-cost MIA that leverages an ensemble of small quantile regression models to determine if a document belongs to the model's training set or not. We demonstrate the effectiveness of this approach on fine-tuned LLMs of varying families (OPT, Pythia, Llama) and across multiple datasets. Across all scenarios we obtain comparable or improved accuracy compared to state of the art shadow model approaches, with as little as 6% of their computation budget. We demonstrate increased effectiveness across multi-epoch trained target models, and architecture miss-specification robustness, that is, we can mount an effective attack against a model using a different tokenizer and architecture, without requiring knowledge on the target model.

Title: RPKI: Not Perfect But Good Enough

• Authors: Haya Schulmann, Niklas Vogel, Michael Waidner
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14518
• Pdf URL: https://arxiv.org/pdf/2409.14518
• Copy Paste: [[2409.14518]] RPKI: Not Perfect But Good Enough(https://arxiv.org/abs/2409.14518)
• Keywords: security, protect
• Abstract: The Resource Public Key Infrastructure (RPKI) protocol was standardized to add cryptographic security to Internet routing. With over 50% of Internet resources protected with RPKI today, the protocol already impacts significant parts of Internet traffic. In addition to its growing adoption, there is also increasing political interest in RPKI. The White House indicated in its Roadmap to Enhance Internet Routing Security, on 4 September 2024, that RPKI is a mature and readily available technology for securing inter-domain routing. The Roadmap attributes the main obstacles towards wide adoption of RPKI to a lack of understanding, lack of prioritization, and administrative barriers. This work presents the first comprehensive study of the maturity of RPKI as a viable production-grade technology. We find that current RPKI implementations still lack production-grade resilience and are plagued by software vulnerabilities, inconsistent specifications, and operational challenges, raising significant security concerns. The deployments lack experience with full-fledged strict RPKI-validation in production environments and operate in fail-open test mode. We provide recommendations to improve RPKI resilience and guide stakeholders in securing their deployments against emerging threats. The numerous issues we have discovered with the current RPKI specifications and implementations inevitably lead to the question: Is RPKI sufficiently stable to align with the expectations outlined in the White House roadmap? Certainly, it is not perfect, but is it good enough? The answer, as we will explore, varies depending on one's viewpoint.

Title: An Integrated Blockchain and IPFS Solution for Secure and Efficient Source Code Repository Hosting using Middleman Approach

• Authors: Md. Rafid Haque, Sakibul Islam Munna, Sabbir Ahmed, Md. Tahmid Islam, Md Mehedi Hassan Onik, A.B.M. Ashikur Rahman
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.14530
• Pdf URL: https://arxiv.org/pdf/2409.14530
• Copy Paste: [[2409.14530]] An Integrated Blockchain and IPFS Solution for Secure and Efficient Source Code Repository Hosting using Middleman Approach(https://arxiv.org/abs/2409.14530)
• Keywords: secure, security, robust
• Abstract: Version control systems (VCS) are essential for software development, yet centralized VCS present risks such as data loss, security breaches, and ownership disputes. While blockchain-based approaches to decentralized source code repository hosting have been explored, many existing solutions struggle with challenges related to security, scalability, efficiency, and real-time collaboration. This study seeks to enhance these efforts by proposing a novel decentralized solution that leverages the Ethereum blockchain and IPFS for secure, efficient, and resilient code repository hosting and governance. Our approach introduces a hybrid architecture that combines the immutable and decentralized nature of blockchain with the efficiency of IPFS for off-chain storage. To facilitate real-time collaboration, we integrate a temporary centralized Middleman IPFS that manages transaction processing and enhances operational efficiency without compromising long-term security. This Middleman IPFS acts as an intermediary, balancing the speed of centralized systems with the resilience of decentralized architectures. Our system uses smart contracts to maintain access control and key management by dynamically verifying access rights, ensuring that only authorized users can retrieve and decrypt data stored on IPFS. This integration allows for secure, real-time collaboration in environments where multiple collaborators need concurrent access to shared resources. Our system employs a hybrid encryption scheme that combines symmetric and asymmetric cryptography. The encrypted keys are stored on the blockchain, while IPFS handles the efficient storage of the codebase itself, with a Middleman IPFS maintaining concurrent collaboration, providing a robust and scalable solution for managing large-scale, collaborative coding projects.

Title: Distributionally Robust Inverse Reinforcement Learning for Identifying Multi-Agent Coordinated Sensing

• Authors: Luke Snow, Vikram Krishnamurthy
• Subjects: cs.LG, cs.MA, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.14542
• Pdf URL: https://arxiv.org/pdf/2409.14542
• Copy Paste: [[2409.14542]] Distributionally Robust Inverse Reinforcement Learning for Identifying Multi-Agent Coordinated Sensing(https://arxiv.org/abs/2409.14542)
• Keywords: robust
• Abstract: We derive a minimax distributionally robust inverse reinforcement learning (IRL) algorithm to reconstruct the utility functions of a multi-agent sensing system. Specifically, we construct utility estimators which minimize the worst-case prediction error over a Wasserstein ambiguity set centered at noisy signal observations. We prove the equivalence between this robust estimation and a semi-infinite optimization reformulation, and we propose a consistent algorithm to compute solutions. We illustrate the efficacy of this robust IRL scheme in numerical studies to reconstruct the utility functions of a cognitive radar network from observed tracking signals.

Title: Adaptive Feedforward Gradient Estimation in Neural ODEs

• Authors: Jaouad Dabounou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14549
• Pdf URL: https://arxiv.org/pdf/2409.14549
• Copy Paste: [[2409.14549]] Adaptive Feedforward Gradient Estimation in Neural ODEs(https://arxiv.org/abs/2409.14549)
• Keywords: interpretability
• Abstract: Neural Ordinary Differential Equations (Neural ODEs) represent a significant breakthrough in deep learning, promising to bridge the gap between machine learning and the rich theoretical frameworks developed in various mathematical fields over centuries. In this work, we propose a novel approach that leverages adaptive feedforward gradient estimation to improve the efficiency, consistency, and interpretability of Neural ODEs. Our method eliminates the need for backpropagation and the adjoint method, reducing computational overhead and memory usage while maintaining accuracy. The proposed approach has been validated through practical applications, and showed good performance relative to Neural ODEs state of the art methods.

Title: Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions

• Authors: Hongchen Wang, Kangming Li, Scott Ramsay, Yao Fehlis, Edward Kim, Jason Hattrick-Simpers
• Subjects: cs.CL, cond-mat.mtrl-sci, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14572
• Pdf URL: https://arxiv.org/pdf/2409.14572
• Copy Paste: [[2409.14572]] Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions(https://arxiv.org/abs/2409.14572)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have the potential to revolutionize scientific research, yet their robustness and reliability in domain-specific applications remain insufficiently explored. This study conducts a comprehensive evaluation and robustness analysis of LLMs within the field of materials science, focusing on domain-specific question answering and materials property prediction. Three distinct datasets are used in this study: 1) a set of multiple-choice questions from undergraduate-level materials science courses, 2) a dataset including various steel compositions and yield strengths, and 3) a band gap dataset, containing textual descriptions of material crystal structures and band gap values. The performance of LLMs is assessed using various prompting strategies, including zero-shot chain-of-thought, expert prompting, and few-shot in-context learning. The robustness of these models is tested against various forms of 'noise', ranging from realistic disturbances to intentionally adversarial manipulations, to evaluate their resilience and reliability under real-world conditions. Additionally, the study uncovers unique phenomena of LLMs during predictive tasks, such as mode collapse behavior when the proximity of prompt examples is altered and performance enhancement from train/test mismatch. The findings aim to provide informed skepticism for the broad use of LLMs in materials science and to inspire advancements that enhance their robustness and reliability for practical applications.

Title: Medical Concept Normalization in a Low-Resource Setting

• Authors: Tim Patzelt
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14579
• Pdf URL: https://arxiv.org/pdf/2409.14579
• Copy Paste: [[2409.14579]] Medical Concept Normalization in a Low-Resource Setting(https://arxiv.org/abs/2409.14579)
• Keywords: transformer
• Abstract: In the field of biomedical natural language processing, medical concept normalization is a crucial task for accurately mapping mentions of concepts to a large knowledge base. However, this task becomes even more challenging in low-resource settings, where limited data and resources are available. In this thesis, I explore the challenges of medical concept normalization in a low-resource setting. Specifically, I investigate the shortcomings of current medical concept normalization methods applied to German lay texts. Since there is no suitable dataset available, a dataset consisting of posts from a German medical online forum is annotated with concepts from the Unified Medical Language System. The experiments demonstrate that multilingual Transformer-based models are able to outperform string similarity methods. The use of contextual information to improve the normalization of lay mentions is also examined, but led to inferior results. Based on the results of the best performing model, I present a systematic error analysis and lay out potential improvements to mitigate frequent errors.

Title: The X Types -- Mapping the Semantics of the Twitter Sphere

• Authors: Ogen Schlachet Drukerman, Einat Minkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14584
• Pdf URL: https://arxiv.org/pdf/2409.14584
• Copy Paste: [[2409.14584]] The X Types -- Mapping the Semantics of the Twitter Sphere(https://arxiv.org/abs/2409.14584)
• Keywords: transformer
• Abstract: Social networks form a valuable source of world knowledge, where influential entities correspond to popular accounts. Unlike factual knowledge bases (KBs), which maintain a semantic ontology, structured semantic information is not available on social media. In this work, we consider a social KB of roughly 200K popular Twitter accounts, which denotes entities of interest. We elicit semantic information about those entities. In particular, we associate them with a fine-grained set of 136 semantic types, e.g., determine whether a given entity account belongs to a politician, or a musical artist. In the lack of explicit type information in Twitter, we obtain semantic labels for a subset of the accounts via alignment with the KBs of DBpedia and Wikidata. Given the labeled dataset, we finetune a transformer-based text encoder to generate semantic embeddings of the entities based on the contents of their accounts. We then exploit this evidence alongside network-based embeddings to predict the entities semantic types. In our experiments, we show high type prediction performance on the labeled dataset. Consequently, we apply our type classification model to all of the entity accounts in the social KB. Our analysis of the results offers insights about the global semantics of the Twitter sphere. We discuss downstream applications that should benefit from semantic type information and the semantic embeddings of social entities generated in this work. In particular, we demonstrate enhanced performance on the key task of entity similarity assessment using this information.

Title: Backtracking Improves Generation Safety

• Authors: Yiming Zhang, Jianfeng Chi, Hailey Nguyen, Kartikeya Upasani, Daniel M. Bikel, Jason Weston, Eric Michael Smith
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14586
• Pdf URL: https://arxiv.org/pdf/2409.14586
• Copy Paste: [[2409.14586]] Backtracking Improves Generation Safety(https://arxiv.org/abs/2409.14586)
• Keywords: protect, attack
• Abstract: Text generation has a fundamental limitation almost by definition: there is no taking back tokens that have been generated, even when they are clearly problematic. In the context of language model safety, when a partial unsafe generation is produced, language models by their nature tend to happily keep on generating similarly unsafe additional text. This is in fact how safety alignment of frontier models gets circumvented in the wild, despite great efforts in improving their safety. Deviating from the paradigm of approaching safety alignment as prevention (decreasing the probability of harmful responses), we propose backtracking, a technique that allows language models to "undo" and recover from their own unsafe generation through the introduction of a special [RESET] token. Our method can be incorporated into either SFT or DPO training to optimize helpfulness and harmlessness. We show that models trained to backtrack are consistently safer than baseline models: backtracking Llama-3-8B is four times more safe than the baseline model (6.1\% $\to$ 1.5\%) in our evaluations without regression in helpfulness. Our method additionally provides protection against four adversarial attacks including an adaptive attack, despite not being trained to do so.

Title: Deep Learning Techniques for Atmospheric Turbulence Removal: A Review

• Authors: Paul Hill, Nantheera Anantrasirichai, Alin Achim, David Bull
• Subjects: cs.CV, astro-ph.IM
• Abstract URL: https://arxiv.org/abs/2409.14587
• Pdf URL: https://arxiv.org/pdf/2409.14587
• Copy Paste: [[2409.14587]] Deep Learning Techniques for Atmospheric Turbulence Removal: A Review(https://arxiv.org/abs/2409.14587)
• Keywords: transformer
• Abstract: The influence of atmospheric turbulence on acquired imagery makes image interpretation and scene analysis extremely difficult and reduces the effectiveness of conventional approaches for classifying and tracking objects of interest in the scene. Restoring a scene distorted by atmospheric turbulence is also a challenging problem. The effect, which is caused by random, spatially varying perturbations, makes conventional model-based approaches difficult and, in most cases, impractical due to complexity and memory requirements. Deep learning approaches offer faster operation and are capable of implementation on small devices. This paper reviews the characteristics of atmospheric turbulence and its impact on acquired imagery. It compares the performance of various state-of-the-art deep neural networks, including Transformers, SWIN and Mamba, when used to mitigate spatio-temporal image distortions.

Title: URSimulator: Human-Perception-Driven Prompt Tuning for Enhanced Virtual Urban Renewal via Diffusion Models

• Authors: Chuanbo Hu, Shan Jia, Xin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14589
• Pdf URL: https://arxiv.org/pdf/2409.14589
• Copy Paste: [[2409.14589]] URSimulator: Human-Perception-Driven Prompt Tuning for Enhanced Virtual Urban Renewal via Diffusion Models(https://arxiv.org/abs/2409.14589)
• Keywords: diffusion
• Abstract: Tackling Urban Physical Disorder (e.g., abandoned buildings, litter, messy vegetation, graffiti) is essential, as it negatively impacts the safety, well-being, and psychological state of communities. Urban Renewal is the process of revitalizing these neglected and decayed areas within a city to improve the physical environment and quality of life for residents. Effective urban renewal efforts can transform these environments, enhancing their appeal and livability. However, current research lacks simulation tools that can quantitatively assess and visualize the impacts of renewal efforts, often relying on subjective judgments. Such tools are crucial for planning and implementing effective strategies by providing a clear visualization of potential changes and their impacts. This paper presents a novel framework addressing this gap by using human perception feedback to simulate street environment enhancement. We develop a prompt tuning approach that integrates text-driven Stable Diffusion with human perception feedback, iteratively editing local areas of street view images to better align with perceptions of beauty, liveliness, and safety. Our experiments show that this framework significantly improves perceptions of urban environments, with increases of 17.60% in safety, 31.15% in beauty, and 28.82% in liveliness. In contrast, advanced methods like DiffEdit achieve only 2.31%, 11.87%, and 15.84% improvements, respectively. We applied this framework across various virtual scenarios, including neighborhood improvement, building redevelopment, green space expansion, and community garden creation. The results demonstrate its effectiveness in simulating urban renewal, offering valuable insights for urban planning and policy-making.

Title: EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Models

• Authors: Hossein Rajabzadeh, Aref Jafari, Aman Sharma, Benyamin Jami, Hyock Ju Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14595
• Pdf URL: https://arxiv.org/pdf/2409.14595
• Copy Paste: [[2409.14595]] EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Models(https://arxiv.org/abs/2409.14595)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs), with their increasing depth and number of parameters, have demonstrated outstanding performance across a variety of natural language processing tasks. However, this growth in scale leads to increased computational demands, particularly during inference and fine-tuning. To address these challenges, we introduce EchoAtt, a novel framework aimed at optimizing transformer-based models by analyzing and leveraging the similarity of attention patterns across layers. Our analysis reveals that many inner layers in LLMs, especially larger ones, exhibit highly similar attention matrices. By exploiting this similarity, EchoAtt enables the sharing of attention matrices in less critical layers, significantly reducing computational requirements without compromising performance. We incorporate this approach within a knowledge distillation setup, where a pre-trained teacher model guides the training of a smaller student model. The student model selectively shares attention matrices in layers with high similarity while inheriting key parameters from the teacher. Our best results with TinyLLaMA-1.1B demonstrate that EchoAtt improves inference speed by 15\%, training speed by 25\%, and reduces the number of parameters by approximately 4\%, all while improving zero-shot performance. These findings highlight the potential of attention matrix sharing to enhance the efficiency of LLMs, making them more practical for real-time and resource-limited applications.

Title: DarkGram: Exploring and Mitigating Cybercriminal content shared in Telegram channels

• Authors: Sayak Saha Roy, Elham Pourabbas Vafa, Kobra Khanmohammadi, Shirin Nilizadeh
• Subjects: cs.CR, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.14596
• Pdf URL: https://arxiv.org/pdf/2409.14596
• Copy Paste: [[2409.14596]] DarkGram: Exploring and Mitigating Cybercriminal content shared in Telegram channels(https://arxiv.org/abs/2409.14596)
• Keywords: attack
• Abstract: We present the first large scale analysis of 339 cybercriminal activity channels (CACs) on Telegram from February to May 2024. Collectively followed by over 23.8 million users, these channels shared a wide array of illicit content, including compromised credentials, pirated software and media, tools for blackhat hacking resources such as malware, social engineering scams, and exploit kits. We developed DarkGram, a BERT based framework that identifies malicious posts from the CACs with an accuracy of 96%, using which we conducted a quantitative analysis of 53,605 posts from these channels, revealing key characteristics of shared content. While much of this content is distributed for free, channel administrators frequently employ promotions and giveaways to engage users and boost the sales of premium cybercriminal content. These channels also pose significant risks to their own subscribers. Notably, 28.1% of shared links contained phishing attacks, and 38% of executable files were bundled with malware. Moreover, our qualitative analysis of replies in CACs shows how subscribers cultivate a dangerous sense of community through requests for illegal content, illicit knowledge sharing, and collaborative hacking efforts, while their reactions to posts, including emoji responses, further underscore their appreciation for such content. We also find that the CACs can evade scrutiny by quickly migrating to new channels with minimal subscriber loss, highlighting the resilience of this ecosystem. To counteract this, we further utilized DarkGram to detect new channels, reporting malicious content to Telegram and the affected organizations which resulted in the takedown of 196 such channels over three months. To aid further collaborative efforts in taking down these channels, we open source our dataset and the DarkGram framework.

Title: Implicit Dynamical Flow Fusion (IDFF) for Generative Modeling

• Authors: Mohammad R. Rezaei, Rahul G. Krishnan, Milos R. Popovic, Milad Lankarany
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14599
• Pdf URL: https://arxiv.org/pdf/2409.14599
• Copy Paste: [[2409.14599]] Implicit Dynamical Flow Fusion (IDFF) for Generative Modeling(https://arxiv.org/abs/2409.14599)
• Keywords: diffusion, generative
• Abstract: Conditional Flow Matching (CFM) models can generate high-quality samples from a non-informative prior, but they can be slow, often needing hundreds of network evaluations (NFE). To address this, we propose Implicit Dynamical Flow Fusion (IDFF); IDFF learns a new vector field with an additional momentum term that enables taking longer steps during sample generation while maintaining the fidelity of the generated distribution. Consequently, IDFFs reduce the NFEs by a factor of ten (relative to CFMs) without sacrificing sample quality, enabling rapid sampling and efficient handling of image and time-series data generation tasks. We evaluate IDFF on standard benchmarks such as CIFAR-10 and CelebA for image generation. We achieved likelihood and quality performance comparable to CFMs and diffusion-based models with fewer NFEs. IDFF also shows superior performance on time-series datasets modeling, including molecular simulation and sea surface temperature (SST) datasets, highlighting its versatility and effectiveness across different domains.

Title: Can pre-trained language models generate titles for research papers?

• Authors: Tohida Rehman, Debarshi Kumar Sanyal, Samiran Chattopadhyay
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14602
• Pdf URL: https://arxiv.org/pdf/2409.14602
• Copy Paste: [[2409.14602]] Can pre-trained language models generate titles for research papers?(https://arxiv.org/abs/2409.14602)
• Keywords: large language model
• Abstract: The title of a research paper communicates in a succinct style the main theme and, sometimes, the findings of the paper. Coming up with the right title is often an arduous task, and therefore, it would be beneficial to authors if title generation can be automated. In this paper, we fine-tune pre-trained and large language models to generate titles of papers from their abstracts. We also use ChatGPT in a zero-shot setting to generate paper titles. The performance of the models is measured with ROUGE, METEOR, MoverScore, BERTScore and SciBERTScore metrics.

Title: Patch Ranking: Efficient CLIP by Learning to Rank Local Patches

• Authors: Cheng-En Wu, Jinhong Lin, Yu Hen Hu, Pedro Morgado
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14607
• Pdf URL: https://arxiv.org/pdf/2409.14607
• Copy Paste: [[2409.14607]] Patch Ranking: Efficient CLIP by Learning to Rank Local Patches(https://arxiv.org/abs/2409.14607)
• Keywords: transformer
• Abstract: Contrastive image-text pre-trained models such as CLIP have shown remarkable adaptability to downstream tasks. However, they face challenges due to the high computational requirements of the Vision Transformer (ViT) backbone. Current strategies to boost ViT efficiency focus on pruning patch tokens but fall short in addressing the multimodal nature of CLIP and identifying the optimal subset of tokens for maximum performance. To address this, we propose greedy search methods to establish a "Golden Ranking" and introduce a lightweight predictor specifically trained to approximate this Ranking. To compensate for any performance degradation resulting from token pruning, we incorporate learnable visual tokens that aid in restoring and potentially enhancing the model's performance. Our work presents a comprehensive and systematic investigation of pruning tokens within the ViT backbone of CLIP models. Through our framework, we successfully reduced 40% of patch tokens in CLIP's ViT while only suffering a minimal average accuracy loss of 0.3 across seven datasets. Our study lays the groundwork for building more computationally efficient multimodal models without sacrificing their performance, addressing a key challenge in the application of advanced vision-language models.

Title: SOS: Segment Object System for Open-World Instance Segmentation With Object Priors

• Authors: Christian Wilms, Tim Rolff, Maris Hillemann, Robert Johanson, Simone Frintrop
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14627
• Pdf URL: https://arxiv.org/pdf/2409.14627
• Copy Paste: [[2409.14627]] SOS: Segment Object System for Open-World Instance Segmentation With Object Priors(https://arxiv.org/abs/2409.14627)
• Keywords: transformer, segmentation
• Abstract: We propose an approach for Open-World Instance Segmentation (OWIS), a task that aims to segment arbitrary unknown objects in images by generalizing from a limited set of annotated object classes during training. Our Segment Object System (SOS) explicitly addresses the generalization ability and the low precision of state-of-the-art systems, which often generate background detections. To this end, we generate high-quality pseudo annotations based on the foundation model SAM. We thoroughly study various object priors to generate prompts for SAM, explicitly focusing the foundation model on objects. The strongest object priors were obtained by self-attention maps from self-supervised Vision Transformers, which we utilize for prompting SAM. Finally, the post-processed segments from SAM are used as pseudo annotations to train a standard instance segmentation system. Our approach shows strong generalization capabilities on COCO, LVIS, and ADE20k datasets and improves on the precision by up to 81.6% compared to the state-of-the-art. Source code is available at: this https URL

Title: EQ-CBM: A Probabilistic Concept Bottleneck with Energy-based Models and Quantized Vectors

• Authors: Sangwon Kim, Dasom Ahn, Byoung Chul Ko, In-su Jang, Kwang-Ju Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14630
• Pdf URL: https://arxiv.org/pdf/2409.14630
• Copy Paste: [[2409.14630]] EQ-CBM: A Probabilistic Concept Bottleneck with Energy-based Models and Quantized Vectors(https://arxiv.org/abs/2409.14630)
• Keywords: interpretability
• Abstract: The demand for reliable AI systems has intensified the need for interpretable deep neural networks. Concept bottleneck models (CBMs) have gained attention as an effective approach by leveraging human-understandable concepts to enhance interpretability. However, existing CBMs face challenges due to deterministic concept encoding and reliance on inconsistent concepts, leading to inaccuracies. We propose EQ-CBM, a novel framework that enhances CBMs through probabilistic concept encoding using energy-based models (EBMs) with quantized concept activation vectors (qCAVs). EQ-CBM effectively captures uncertainties, thereby improving prediction reliability and accuracy. By employing qCAVs, our method selects homogeneous vectors during concept encoding, enabling more decisive task performance and facilitating higher levels of human intervention. Empirical results using benchmark datasets demonstrate that our approach outperforms the state-of-the-art in both concept and task accuracy.

Title: Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweighting

• Authors: Humza Wajid Hameed, Geraldin Nanfack, Eugene Belilovsky
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14637
• Pdf URL: https://arxiv.org/pdf/2409.14637
• Copy Paste: [[2409.14637]] Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweighting(https://arxiv.org/abs/2409.14637)
• Keywords: robust, fair
• Abstract: Spurious correlations are a major source of errors for machine learning models, in particular when aiming for group-level fairness. It has been recently shown that a powerful approach to combat spurious correlations is to re-train the last layer on a balanced validation dataset, isolating robust features for the predictor. However, key attributes can sometimes be discarded by neural networks towards the last layer. In this work, we thus consider retraining a classifier on a set of features derived from all layers. We utilize a recently proposed feature selection strategy to select unbiased features from all the layers. We observe this approach gives significant improvements in worst-group accuracy on several standard benchmarks.

Title: Harmonising the Clinical Melody: Tuning Large Language Models for Hospital Course Summarisation in Clinical Coding

• Authors: Bokang Bi, Leibo Liu, Oscar Perez-Concha, Sanja Lujic, Louisa Jorm
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14638
• Pdf URL: https://arxiv.org/pdf/2409.14638
• Copy Paste: [[2409.14638]] Harmonising the Clinical Melody: Tuning Large Language Models for Hospital Course Summarisation in Clinical Coding(https://arxiv.org/abs/2409.14638)
• Keywords: large language model
• Abstract: The increasing volume and complexity of clinical documentation in Electronic Medical Records systems pose significant challenges for clinical coders, who must mentally process and summarise vast amounts of clinical text to extract essential information needed for coding tasks. While large language models have been successfully applied to shorter summarisation tasks in recent years, the challenge of summarising a hospital course remains an open area for further research and development. In this study, we adapted three pre trained LLMs, Llama 3, BioMistral, Mistral Instruct v0.1 for the hospital course summarisation task, using Quantized Low Rank Adaptation fine tuning. We created a free text clinical dataset from MIMIC III data by concatenating various clinical notes as the input clinical text, paired with ground truth Brief Hospital Course sections extracted from the discharge summaries for model training. The fine tuned models were evaluated using BERTScore and ROUGE metrics to assess the effectiveness of clinical domain fine tuning. Additionally, we validated their practical utility using a novel hospital course summary assessment metric specifically tailored for clinical coding. Our findings indicate that fine tuning pre trained LLMs for the clinical domain can significantly enhance their performance in hospital course summarisation and suggest their potential as assistive tools for clinical coding. Future work should focus on refining data curation methods to create higher quality clinical datasets tailored for hospital course summary tasks and adapting more advanced open source LLMs comparable to proprietary models to further advance this research.

Title: MECURY: Practical Cross-Chain Exchange via Trusted Hardware

• Authors: Xiaoqing Wen, Quanbi Feng, Jianyu Niu, Yinqian Zhang, Chen Feng
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.14640
• Pdf URL: https://arxiv.org/pdf/2409.14640
• Copy Paste: [[2409.14640]] MECURY: Practical Cross-Chain Exchange via Trusted Hardware(https://arxiv.org/abs/2409.14640)
• Keywords: security
• Abstract: The proliferation of blockchain-backed cryptocurrencies has sparked the need for cross-chain exchanges of diverse digital assets. Unfortunately, current exchanges suffer from high on-chain verification costs, weak threat models of central trusted parties, or synchronous requirements, making them impractical for currency trading applications. In this paper, we present MERCURY, a practical cryptocurrency exchange that is trust-minimized and efficient without online-client requirements. MERCURY leverages Trusted Execution Environments (TEEs) to shield participants from malicious behaviors, eliminating the reliance on trusted participants and making on-chain verification efficient. Despite the simple idea, building a practical TEE-assisted cross-chain exchange is challenging due to the security and unavailability issues of TEEs. MERCURY tackles the unavailability problem of TEEs by implementing an efficient challenge-response mechanism executed on smart contracts. Furthermore, MERCURY utilizes a lightweight transaction verification mechanism and adopts multiple optimizations to reduce on-chain costs. Comparative evaluations with XClaim, ZK-bridge, and Tesseract demonstrate that MERCURY significantly reduces on-chain costs by approximately 67.87%, 45.01%, and 47.70%, respectively.

Title: Demystifying Trajectory Recovery From Ash: An Open-Source Evaluation and Enhancement

• Authors: Nicholas D'Silva, Toran Shahi, Øyvind Timian Dokk Husveg, Adith Sanjeeve, Erik Buchholz, Salil S. Kanhere
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14645
• Pdf URL: https://arxiv.org/pdf/2409.14645
• Copy Paste: [[2409.14645]] Demystifying Trajectory Recovery From Ash: An Open-Source Evaluation and Enhancement(https://arxiv.org/abs/2409.14645)
• Keywords: privacy, attack
• Abstract: Once analysed, location trajectories can provide valuable insights beneficial to various applications. However, such data is also highly sensitive, rendering them susceptible to privacy risks in the event of mismanagement, for example, revealing an individual's identity, home address, or political affiliations. Hence, ensuring that privacy is preserved for this data is a priority. One commonly taken measure to mitigate this concern is aggregation. Previous work by Xu et al. shows that trajectories are still recoverable from anonymised and aggregated datasets. However, the study lacks implementation details, obfuscating the mechanisms of the attack. Additionally, the attack was evaluated on commercial non-public datasets, rendering the results and subsequent claims unverifiable. This study reimplements the trajectory recovery attack from scratch and evaluates it on two open-source datasets, detailing the preprocessing steps and implementation. Results confirm that privacy leakage still exists despite common anonymisation and aggregation methods but also indicate that the initial accuracy claims may have been overly ambitious. We release all code as open-source to ensure the results are entirely reproducible and, therefore, verifiable. Moreover, we propose a stronger attack by designing a series of enhancements to the baseline attack. These enhancements yield higher accuracies by up to 16%, providing an improved benchmark for future research in trajectory recovery methods. Our improvements also enable online execution of the attack, allowing partial attacks on larger datasets previously considered unprocessable, thereby furthering the extent of privacy leakage. The findings emphasise the importance of using strong privacy-preserving mechanisms when releasing aggregated mobility data and not solely relying on aggregation as a means of anonymisation.

Title: TeeRollup: Efficient Rollup Design Using Heterogeneous TEE

• Authors: Xiaoqing Wen, Quanbi Feng, Jianyu Niu, Yinqian Zhang, Chen Feng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14647
• Pdf URL: https://arxiv.org/pdf/2409.14647
• Copy Paste: [[2409.14647]] TeeRollup: Efficient Rollup Design Using Heterogeneous TEE(https://arxiv.org/abs/2409.14647)
• Keywords: security
• Abstract: Rollups have emerged as a promising approach to improving blockchains' scalability by offloading transactions execution off-chain. Existing rollup solutions either leverage complex zero-knowledge proofs or optimistically assume execution correctness unless challenged. However, these solutions have practical issues such as high gas costs and significant withdrawal delays, hindering their adoption in decentralized applications. This paper introduces TeeRollup, an efficient rollup design with low gas costs and short withdrawal delays. TeeRollup employs Trusted Execution Environments (TEEs)-supported sequencers to execute transactions, requiring the blockchain to verify only the TEEs' signatures. TeeRollup is designed under a realistic threat model in which the integrity and availability of sequencers' TEEs may be compromised. To address these issues, we first introduce a distributed system of sequencers with heterogeneous TEEs, ensuring system security even if a minority of TEEs are compromised. Second, we propose a challenge mechanism to solve the redeemability issue caused by TEE unavailability. Furthermore, TeeRollup incorporates Data Availability Providers (DAPs) to reduce on-chain storage overhead and uses a laziness penalty game to regulate DAP behavior. We implement a prototype of TeeRollup in Golang, using the Ethereum test network, Sepolia. Our experimental results indicate that TeeRollup outperforms zero-knowledge rollups (zk-rollups), reducing on-chain verification costs by approximately 86% and withdrawal delays to a few minutes.

Title: Direct Judgement Preference Optimization

• Authors: Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, Shafiq Joty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14664
• Pdf URL: https://arxiv.org/pdf/2409.14664
• Copy Paste: [[2409.14664]] Direct Judgement Preference Optimization(https://arxiv.org/abs/2409.14664)
• Keywords: robust, generative, large language model
• Abstract: Auto-evaluation is crucial for assessing response quality and offering feedback for model development. Recent studies have explored training large language models (LLMs) as generative judges to evaluate and critique other models' outputs. In this work, we investigate the idea of learning from both positive and negative data with preference optimization to enhance the evaluation capabilities of LLM judges across an array of different use cases. We achieve this by employing three approaches to collect the preference pairs for different use cases, each aimed at improving our generative judge from a different perspective. Our comprehensive study over a wide range of benchmarks demonstrates the effectiveness of our method. In particular, our generative judge achieves the best performance on 10 out of 13 benchmarks, outperforming strong baselines like GPT-4o and specialized judge models. Further analysis show that our judge model robustly counters inherent biases such as position and length bias, flexibly adapts to any evaluation protocol specified by practitioners, and provides helpful language feedback for improving downstream generator models.

Title: Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science

• Authors: Taihang Wang, Xiaoman Xu, Yimin Wang, Ye Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14673
• Pdf URL: https://arxiv.org/pdf/2409.14673
• Copy Paste: [[2409.14673]] Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science(https://arxiv.org/abs/2409.14673)
• Keywords: large language model
• Abstract: Real-world applications of large language models (LLMs) in computational social science (CSS) tasks primarily depend on the effectiveness of instruction tuning (IT) or in-context learning (ICL). While IT has shown highly effective at fine-tuning LLMs for various tasks, ICL offers a rapid alternative for task adaptation by learning from examples without explicit gradient updates. In this paper, we evaluate the classification performance of LLMs using IT versus ICL in few-shot CSS tasks. The experimental results indicate that ICL consistently outperforms IT in most CSS tasks. Additionally, we investigate the relationship between the increasing number of training samples and LLM performance. Our findings show that simply increasing the number of samples without considering their quality does not consistently enhance the performance of LLMs with either ICL or IT and can sometimes even result in a performance decline. Finally, we compare three prompting strategies, demonstrating that ICL is more effective than zero-shot and Chain-of-Thought (CoT). Our research highlights the significant advantages of ICL in handling CSS tasks in few-shot settings and emphasizes the importance of optimizing sample quality and prompting strategies to improve LLM classification performance. The code will be made available.

Title: Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections

• Authors: Ankit Dhiman, Manan Shah, Rishubh Parihar, Yash Bhalgat, Lokesh R Boregowda, R Venkatesh Babu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14677
• Pdf URL: https://arxiv.org/pdf/2409.14677
• Copy Paste: [[2409.14677]] Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections(https://arxiv.org/abs/2409.14677)
• Keywords: diffusion, generative, segmentation
• Abstract: We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We formulate this problem as an image inpainting task, allowing for more user control over the placement of mirrors during the generation process. To enable this, we create SynMirror, a large-scale dataset of diverse synthetic scenes with objects placed in front of mirrors. SynMirror contains around 198K samples rendered from 66K unique 3D objects, along with their associated depth maps, normal maps and instance-wise segmentation masks, to capture relevant geometric properties of the scene. Using this dataset, we propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality geometrically consistent and photo-realistic mirror reflections given an input image and a mask depicting the mirror region. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis. To the best of our knowledge, we are the first to successfully tackle the challenging problem of generating controlled and faithful mirror reflections of an object in a scene using diffusion based models. SynMirror and MirrorFusion open up new avenues for image editing and augmented reality applications for practitioners and researchers alike.

Title: Adaptive and Robust Watermark for Generative Tabular Data

• Authors: Dung Daniel Ngo, Daniel Scott, Saheed Obitayo, Vamsi K. Potluru, Manuela Veloso
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14700
• Pdf URL: https://arxiv.org/pdf/2409.14700
• Copy Paste: [[2409.14700]] Adaptive and Robust Watermark for Generative Tabular Data(https://arxiv.org/abs/2409.14700)
• Keywords: attack, robust, watermark, generative
• Abstract: Recent developments in generative models have demonstrated its ability to create high-quality synthetic data. However, the pervasiveness of synthetic content online also brings forth growing concerns that it can be used for malicious purposes. To ensure the authenticity of the data, watermarking techniques have recently emerged as a promising solution due to their strong statistical guarantees. In this paper, we propose a flexible and robust watermarking mechanism for generative tabular data. Specifically, a data provider with knowledge of the downstream tasks can partition the feature space into pairs of $(key, value)$ columns. Within each pair, the data provider first uses elements in the $key$ column to generate a randomized set of ''green'' intervals, then encourages elements of the $value$ column to be in one of these ''green'' intervals. We show theoretically and empirically that the watermarked datasets (i) have negligible impact on the data quality and downstream utility, (ii) can be efficiently detected, and (iii) are robust against multiple attacks commonly observed in data science.

Title: VLEU: a Method for Automatic Evaluation for Generalizability of Text-to-Image Models

• Authors: Jingtao Cao, Zheng Zhang, Hongru Wang, Kam-Fai Wong
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14704
• Pdf URL: https://arxiv.org/pdf/2409.14704
• Copy Paste: [[2409.14704]] VLEU: a Method for Automatic Evaluation for Generalizability of Text-to-Image Models(https://arxiv.org/abs/2409.14704)
• Keywords: large language model
• Abstract: Progress in Text-to-Image (T2I) models has significantly improved the generation of images from textual descriptions. However, existing evaluation metrics do not adequately assess the models' ability to handle a diverse range of textual prompts, which is crucial for their generalizability. To address this, we introduce a new metric called Visual Language Evaluation Understudy (VLEU). VLEU uses large language models to sample from the visual text domain, the set of all possible input texts for T2I models, to generate a wide variety of prompts. The images generated from these prompts are evaluated based on their alignment with the input text using the CLIP model.VLEU quantifies a model's generalizability by computing the Kullback-Leibler divergence between the marginal distribution of the visual text and the conditional distribution of the images generated by the model. This metric provides a quantitative way to compare different T2I models and track improvements during model finetuning. Our experiments demonstrate the effectiveness of VLEU in evaluating the generalization capability of various T2I models, positioning it as an essential metric for future research in text-to-image synthesis.

Title: ERABAL: Enhancing Role-Playing Agents through Boundary-Aware Learning

• Authors: Yihong Tang, Jiao Ou, Che Liu, Fuzheng Zhang, Di Zhang, Kun Gai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14710
• Pdf URL: https://arxiv.org/pdf/2409.14710
• Copy Paste: [[2409.14710]] ERABAL: Enhancing Role-Playing Agents through Boundary-Aware Learning(https://arxiv.org/abs/2409.14710)
• Keywords: large language model
• Abstract: Role-playing is an emerging application in the field of Human-Computer Interaction (HCI), primarily implemented through the alignment training of a large language model (LLM) with assigned characters. Despite significant progress, role-playing agents (RPLAs) still struggle with maintaining role-consistency across conversations, particularly when confronted with boundary queries subtly related to character attributes. In this paper, we present ERABAL, a framework aimed at enhancing RPLAs' role-playing capabilities through boundary-aware learning. ERABAL encompasses a generation pipeline for role-specific dialogues and a concomitant methodology for alignment training. Through comprehensive evaluations, we demonstrate that ERABAL is both efficient and effective. By training with significantly fewer dialogues than those used in leading approaches, ERABAL achieves notable improvements across WikiRoleEval, CharacterEval, and the role-playing subset of MT-Bench compared to the generalist baseline models. Our code and datasets will be made publicly available to support further research.

Title: Phantom of Latent for Large Language and Vision Models

• Authors: Byung-Kwan Lee, Sangyun Chung, Chae Won Kim, Beomchan Park, Yong Man Ro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14713
• Pdf URL: https://arxiv.org/pdf/2409.14713
• Copy Paste: [[2409.14713]] Phantom of Latent for Large Language and Vision Models(https://arxiv.org/abs/2409.14713)
• Keywords: large language model
• Abstract: The success of visual instruction tuning has accelerated the development of large language and vision models (LLVMs). Following the scaling laws of instruction-tuned large language models (LLMs), LLVMs either have further increased their sizes, reaching 26B, 34B, and even 80B parameters. While this increase in model size has yielded significant performance gains, it demands substantially more hardware resources for both training and inference. Consequently, there naturally exists a strong need for efficient LLVMs that achieve the performance of larger models while being smaller in size. To achieve this need, we present a new efficient LLVM family with model sizes of 0.5B, 1.8B, 3.8B, and 7B parameters, Phantom, which significantly enhances learning capabilities within limited structures. By temporarily increasing the latent hidden dimension during multi-head self-attention (MHSA), we make LLVMs prepare to look and understand much more vision-language knowledge on the latent, without substantially increasing physical model sizes. To maximize its advantage, we introduce Phantom Optimization (PO) using both autoregressive supervised fine-tuning (SFT) and direct preference optimization (DPO)-like concept, which effectively follows correct answers while eliminating incorrect and ambiguous ones. Phantom outperforms numerous larger open- and closed-source LLVMs, positioning itself as a leading solution in the landscape of efficient LLVMs.

Title: ControlEdit: A MultiModal Local Clothing Image Editing Method

• Authors: Di Cheng, YingJie Shi, ShiXin Sun, JiaFu Zhang, WeiJing Wang, Yu Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14720
• Pdf URL: https://arxiv.org/pdf/2409.14720
• Copy Paste: [[2409.14720]] ControlEdit: A MultiModal Local Clothing Image Editing Method(https://arxiv.org/abs/2409.14720)
• Keywords: extraction, diffusion
• Abstract: Multimodal clothing image editing refers to the precise adjustment and modification of clothing images using data such as textual descriptions and visual images as control conditions, which effectively improves the work efficiency of designers and reduces the threshold for user design. In this paper, we propose a new image editing method ControlEdit, which transfers clothing image editing to multimodal-guided local inpainting of clothing images. We address the difficulty of collecting real image datasets by leveraging the self-supervised learning approach. Based on this learning approach, we extend the channels of the feature extraction network to ensure consistent clothing image style before and after editing, and we design an inverse latent loss function to achieve soft control over the content of non-edited areas. In addition, we adopt Blended Latent Diffusion as the sampling method to make the editing boundaries transition naturally and enforce consistency of non-edited area content. Extensive experiments demonstrate that ControlEdit surpasses baseline algorithms in both qualitative and quantitative evaluations.

Title: EDSNet: Efficient-DSNet for Video Summarization

• Authors: Ashish Prasad, Pranav Jeevan, Amit Sethi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14724
• Pdf URL: https://arxiv.org/pdf/2409.14724
• Copy Paste: [[2409.14724]] EDSNet: Efficient-DSNet for Video Summarization(https://arxiv.org/abs/2409.14724)
• Keywords: transformer
• Abstract: Current video summarization methods largely rely on transformer-based architectures, which, due to their quadratic complexity, require substantial computational resources. In this work, we address these inefficiencies by enhancing the Direct-to-Summarize Network (DSNet) with more resource-efficient token mixing mechanisms. We show that replacing traditional attention with alternatives like Fourier, Wavelet transforms, and Nyströmformer improves efficiency and performance. Furthermore, we explore various pooling strategies within the Regional Proposal Network, including ROI pooling, Fast Fourier Transform pooling, and flat pooling. Our experimental results on TVSum and SumMe datasets demonstrate that these modifications significantly reduce computational costs while maintaining competitive summarization performance. Thus, our work offers a more scalable solution for video summarization tasks.

Title: PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs

• Authors: Jiahao Yu, Yangguang Shao, Hanwen Miao, Junzheng Shi, Xinyu Xing
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14729
• Pdf URL: https://arxiv.org/pdf/2409.14729
• Copy Paste: [[2409.14729]] PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs(https://arxiv.org/abs/2409.14729)
• Keywords: security, defense, attack, robust, large language model
• Abstract: Large Language Models (LLMs) have gained widespread use in various applications due to their powerful capability to generate human-like text. However, prompt injection attacks, which involve overwriting a model's original instructions with malicious prompts to manipulate the generated text, have raised significant concerns about the security and reliability of LLMs. Ensuring that LLMs are robust against such attacks is crucial for their deployment in real-world applications, particularly in critical tasks. In this paper, we propose PROMPTFUZZ, a novel testing framework that leverages fuzzing techniques to systematically assess the robustness of LLMs against prompt injection attacks. Inspired by software fuzzing, PROMPTFUZZ selects promising seed prompts and generates a diverse set of prompt injections to evaluate the target LLM's resilience. PROMPTFUZZ operates in two stages: the prepare phase, which involves selecting promising initial seeds and collecting few-shot examples, and the focus phase, which uses the collected examples to generate diverse, high-quality prompt injections. Using PROMPTFUZZ, we can uncover more vulnerabilities in LLMs, even those with strong defense prompts. By deploying the generated attack prompts from PROMPTFUZZ in a real-world competition, we achieved the 7th ranking out of over 4000 participants (top 0.14%) within 2 hours. Additionally, we construct a dataset to fine-tune LLMs for enhanced robustness against prompt injection attacks. While the fine-tuned model shows improved robustness, PROMPTFUZZ continues to identify vulnerabilities, highlighting the importance of robust testing for LLMs. Our work emphasizes the critical need for effective testing tools and provides a practical framework for evaluating and improving the robustness of LLMs against prompt injection attacks.

Title: ToxiCraft: A Novel Framework for Synthetic Generation of Harmful Information

• Authors: Zheng Hui, Zhaoxiao Guo, Hang Zhao, Juanyong Duan, Congrui Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14740
• Pdf URL: https://arxiv.org/pdf/2409.14740
• Copy Paste: [[2409.14740]] ToxiCraft: A Novel Framework for Synthetic Generation of Harmful Information(https://arxiv.org/abs/2409.14740)
• Keywords: robust
• Abstract: In different NLP tasks, detecting harmful content is crucial for online environments, especially with the growing influence of social media. However, previous research has two main issues: 1) a lack of data in low-resource settings, and 2) inconsistent definitions and criteria for judging harmful content, requiring classification models to be robust to spurious features and diverse. We propose Toxicraft, a novel framework for synthesizing datasets of harmful information to address these weaknesses. With only a small amount of seed data, our framework can generate a wide variety of synthetic, yet remarkably realistic, examples of toxic information. Experimentation across various datasets showcases a notable enhancement in detection model robustness and adaptability, surpassing or close to the gold labels. We release the generated data at Github upon acceptance.

Title: Less yet robust: crucial region selection for scene recognition

• Authors: Jianqi Zhang, Mengxuan Wang, Jingyao Wang, Lingyu Si, Changwen Zheng, Fanjiang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14741
• Pdf URL: https://arxiv.org/pdf/2409.14741
• Copy Paste: [[2409.14741]] Less yet robust: crucial region selection for scene recognition(https://arxiv.org/abs/2409.14741)
• Keywords: robust
• Abstract: Scene recognition, particularly for aerial and underwater images, often suffers from various types of degradation, such as blurring or overexposure. Previous works that focus on convolutional neural networks have been shown to be able to extract panoramic semantic features and perform well on scene recognition tasks. However, low-quality images still impede model performance due to the inappropriate use of high-level semantic features. To address these To address these challenges, we propose an adaptive selection mechanism to identify the most important and robust regions with high-level features. Thus, the model can perform learning via these regions to avoid interference. implement a learnable mask in the neural network, which can filter high-level features by assigning weights to different regions of the feature matrix. We also introduce a regularization term to further enhance the significance of key high-level feature regions. Different from previous methods, our learnable matrix pays extra attention to regions that are important to multiple categories but may cause misclassification and sets constraints to reduce the influence of such regions.This is a plug-and-play architecture that can be easily extended to other methods. Additionally, we construct an Underwater Geological Scene Classification dataset to assess the effectiveness of our model. Extensive experimental results demonstrate the superiority and robustness of our proposed method over state-of-the-art techniques on two datasets.

Title: LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMs

• Authors: Sihui Yang, Keping Bi, Wanqing Cui, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14744
• Pdf URL: https://arxiv.org/pdf/2409.14744
• Copy Paste: [[2409.14744]] LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMs(https://arxiv.org/abs/2409.14744)
• Keywords: large language model
• Abstract: Non-Factoid (NF) Question Answering (QA) is challenging to evaluate due to diverse potential answers and no objective criterion. The commonly used automatic evaluation metrics like ROUGE or BERTScore cannot accurately measure semantic similarities or answers from different perspectives. Recently, Large Language Models (LLMs) have been resorted to for NFQA evaluation due to their compelling performance on various NLP tasks. Common approaches include pointwise scoring of each candidate answer and pairwise comparisons between answers. Inspired by the evolution from pointwise to pairwise to listwise in learning-to-rank methods, we propose a novel listwise NFQA evaluation approach, that utilizes LLMs to rank candidate answers in a list of reference answers sorted by descending quality. Moreover, for NF questions that do not have multi-grade or any golden answers, we leverage LLMs to generate the reference answer list of various quality to facilitate the listwise evaluation. Extensive experimental results on three NFQA datasets, i.e., ANTIQUE, the TREC-DL-NF, and WebGLM show that our method has significantly higher correlations with human annotations compared to automatic scores and common pointwise and pairwise approaches.

Title: FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension

• Authors: Junzhuo Liu, Xuzheng Yang, Weiwei Li, Peng Wang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14750
• Pdf URL: https://arxiv.org/pdf/2409.14750
• Copy Paste: [[2409.14750]] FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension(https://arxiv.org/abs/2409.14750)
• Keywords: large language model
• Abstract: Referring Expression Comprehension (REC) is a crucial cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding. Consequently, it serves as an ideal testing ground for Multi-modal Large Language Models (MLLMs). In pursuit of this goal, we have established a new REC dataset characterized by two key features: Firstly, it is designed with controllable varying levels of difficulty, necessitating multi-level fine-grained reasoning across object categories, attributes, and multi-hop relationships. Secondly, it includes negative text and images created through fine-grained editing and generation based on existing data, thereby testing the model's ability to correctly reject scenarios where the target object is not visible in the image--an essential aspect often overlooked in existing datasets and approaches. Utilizing this high-quality dataset, we conducted comprehensive evaluations of both state-of-the-art specialist models and MLLMs. Our findings indicate that there remains a significant gap in achieving satisfactory grounding performance. We anticipate that our dataset will inspire new approaches to enhance visual reasoning and develop more advanced cross-modal interaction strategies, ultimately unlocking the full potential of MLLMs. Our code and the datasets are available at this https URL.

Title: UniBEVFusion: Unified Radar-Vision BEVFusion for 3D Object Detection

• Authors: Haocheng Zhao, Runwei Guan, Taoyu Wu, Ka Lok Man, Limin Yu, Yutao Yue
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14751
• Pdf URL: https://arxiv.org/pdf/2409.14751
• Copy Paste: [[2409.14751]] UniBEVFusion: Unified Radar-Vision BEVFusion for 3D Object Detection(https://arxiv.org/abs/2409.14751)
• Keywords: robust
• Abstract: 4D millimeter-wave (MMW) radar, which provides both height information and dense point cloud data over 3D MMW radar, has become increasingly popular in 3D object detection. In recent years, radar-vision fusion models have demonstrated performance close to that of LiDAR-based models, offering advantages in terms of lower hardware costs and better resilience in extreme conditions. However, many radar-vision fusion models treat radar as a sparse LiDAR, underutilizing radar-specific information. Additionally, these multi-modal networks are often sensitive to the failure of a single modality, particularly vision. To address these challenges, we propose the Radar Depth Lift-Splat-Shoot (RDL) module, which integrates radar-specific data into the depth prediction process, enhancing the quality of visual Bird-Eye View (BEV) features. We further introduce a Unified Feature Fusion (UFF) approach that extracts BEV features across different modalities using shared module. To assess the robustness of multi-modal models, we develop a novel Failure Test (FT) ablation experiment, which simulates vision modality failure by injecting Gaussian noise. We conduct extensive experiments on the View-of-Delft (VoD) and TJ4D datasets. The results demonstrate that our proposed Unified BEVFusion (UniBEVFusion) network significantly outperforms state-of-the-art models on the TJ4D dataset, with improvements of 1.44 in 3D and 1.72 in BEV object detection accuracy.

Title: BranchPoseNet: Characterizing tree branching with a deep learning-based pose estimation approach

• Authors: Stefano Puliti, Carolin Fischer, Rasmus Astrup
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2409.14755
• Pdf URL: https://arxiv.org/pdf/2409.14755
• Copy Paste: [[2409.14755]] BranchPoseNet: Characterizing tree branching with a deep learning-based pose estimation approach(https://arxiv.org/abs/2409.14755)
• Keywords: biometric
• Abstract: This paper presents an automated pipeline for detecting tree whorls in proximally laser scanning data using a pose-estimation deep learning model. Accurate whorl detection provides valuable insights into tree growth patterns, wood quality, and offers potential for use as a biometric marker to track trees throughout the forestry value chain. The workflow processes point cloud data to create sectional images, which are subsequently used to identify keypoints representing tree whorls and branches along the stem. The method was tested on a dataset of destructively sampled individual trees, where the whorls were located along the stems of felled trees. The results demonstrated strong potential, with accurate identification of tree whorls and precise calculation of key structural metrics, unlocking new insights and deeper levels of information from individual tree point clouds.

Title: Do Large Language Models have Problem-Solving Capability under Incomplete Information Scenarios?

• Authors: Yuyan Chen, Tianhao Yu, Yueze Li, Songzhou Yan, Sijia Liu, Jiaqing Liang, Yanghua Xiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14762
• Pdf URL: https://arxiv.org/pdf/2409.14762
• Copy Paste: [[2409.14762]] Do Large Language Models have Problem-Solving Capability under Incomplete Information Scenarios?(https://arxiv.org/abs/2409.14762)
• Keywords: large language model
• Abstract: The evaluation of the problem-solving capability under incomplete information scenarios of Large Language Models (LLMs) is increasingly important, encompassing capabilities such as questioning, knowledge search, error detection, and path planning. Current research mainly focus on LLMs' problem-solving capability such as ``Twenty Questions''. However, these kinds of games do not require recognizing misleading cues which are necessary in the incomplete information scenario. Moreover, the existing game such as ``Who is undercover'' are highly subjective, making it challenging for evaluation. Therefore, in this paper, we introduce a novel game named BrainKing based on the ``Who is undercover'' and ``Twenty Questions'' for evaluating LLM capabilities under incomplete information scenarios. It requires LLMs to identify target entities with limited yes-or-no questions and potential misleading answers. By setting up easy, medium, and hard difficulty modes, we comprehensively assess the performance of LLMs across various aspects. Our results reveal the capabilities and limitations of LLMs in BrainKing, providing significant insights of LLM problem-solving levels.

Title: Robust and Flexible Omnidirectional Depth Estimation with Multiple 360{\deg} Cameras

• Authors: Ming Li, Xueqian Jin, Xuejiao Hu, Jinghao Cao, Sidan Du, Yang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14766
• Pdf URL: https://arxiv.org/pdf/2409.14766
• Copy Paste: [[2409.14766]] Robust and Flexible Omnidirectional Depth Estimation with Multiple 360{\deg} Cameras(https://arxiv.org/abs/2409.14766)
• Keywords: robust
• Abstract: Omnidirectional depth estimation has received much attention from researchers in recent years. However, challenges arise due to camera soiling and variations in camera layouts, affecting the robustness and flexibility of the algorithm. In this paper, we use the geometric constraints and redundant information of multiple 360-degree cameras to achieve robust and flexible multi-view omnidirectional depth estimation. We implement two algorithms, in which the two-stage algorithm obtains initial depth maps by pairwise stereo matching of multiple cameras and fuses the multiple depth maps to achieve the final depth estimation; the one-stage algorithm adopts spherical sweeping based on hypothetical depths to construct a uniform spherical matching cost of the multi-camera images and obtain the depth. Additionally, a generalized epipolar equirectangular projection is introduced to simplify the spherical epipolar constraints. To overcome panorama distortion, a spherical feature extractor is implemented. Furthermore, a synthetic 360-degree dataset consisting of 12K road scene panoramas and 3K ground truth depth maps is presented to train and evaluate 360-degree depth estimation algorithms. Our dataset takes soiled camera lenses and glare into consideration, which is more consistent with the real-world environment. Experiments show that our two algorithms achieve state-of-the-art performance, accurately predicting depth maps even when provided with soiled panorama inputs. The flexibility of the algorithms is experimentally validated in terms of camera layouts and numbers.

Title: OMPar: Automatic Parallelization with AI-Driven Source-to-Source Compilation

• Authors: Tal Kadosh, Niranjan Hasabnis, Prema Soundararajan, Vy A. Vo, Mihai Capota, Nesreen Ahmed, Yuval Pinter, Gal Oren
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14771
• Pdf URL: https://arxiv.org/pdf/2409.14771
• Copy Paste: [[2409.14771]] OMPar: Automatic Parallelization with AI-Driven Source-to-Source Compilation(https://arxiv.org/abs/2409.14771)
• Keywords: large language model
• Abstract: Manual parallelization of code remains a significant challenge due to the complexities of modern software systems and the widespread adoption of multi-core architectures. This paper introduces OMPar, an AI-driven tool designed to automate the parallelization of C/C++ code using OpenMP pragmas. OMPar integrates Large Language Models (LLMs) through two key components: OMPify, which assesses loop parallelization potential, and MonoCoder-OMP, a new fine-tuned model which generates precise OpenMP pragmas. The evaluation of OMPar follows the same rigorous process applied to traditional tools like source-to-source AutoPar and ICPC compilers: (1) ensuring the generated code compiles and runs correctly in serial form, (2) assessing performance with the gradual addition of threads and corresponding physical cores, and (3) verifying and validating the correctness of the code's output. Benchmarks from HeCBench and ParEval are used to evaluate accuracy and performance. Experimental results demonstrate that OMPar significantly outperforms traditional methods, achieving higher accuracy in identifying parallelizable loops and generating efficient pragmas. Beyond accuracy, OMPar offers advantages such as the ability to work on partial or incomplete codebases and the capacity to continuously learn from new code patterns, enhancing its parallelization capabilities over time. These results underscore the potential of LLMs in revolutionizing automatic parallelization techniques, paving the way for more efficient and scalable parallel computing systems.

Title: CFVNet: An End-to-End Cancelable Finger Vein Network for Recognition

• Authors: Yifan Wang, Jie Gui, Yuan Yan Tang, James Tin-Yau Kwok
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14774
• Pdf URL: https://arxiv.org/pdf/2409.14774
• Copy Paste: [[2409.14774]] CFVNet: An End-to-End Cancelable Finger Vein Network for Recognition(https://arxiv.org/abs/2409.14774)
• Keywords: secure, security, privacy, protect, biometric
• Abstract: Finger vein recognition technology has become one of the primary solutions for high-security identification systems. However, it still has information leakage problems, which seriously jeopardizes users privacy and anonymity and cause great security risks. In addition, there is no work to consider a fully integrated secure finger vein recognition system. So, different from the previous systems, we integrate preprocessing and template protection into an integrated deep learning model. We propose an end-to-end cancelable finger vein network (CFVNet), which can be used to design an secure finger vein recognition this http URL includes a plug-and-play BWR-ROIAlign unit, which consists of three sub-modules: Localization, Compression and Transformation. The localization module achieves automated localization of stable and unique finger vein ROI. The compression module losslessly removes spatial and channel redundancies. The transformation module uses the proposed BWR method to introduce unlinkability, irreversibility and revocability to the system. BWR-ROIAlign can directly plug into the model to introduce the above features for DCNN-based finger vein recognition systems. We perform extensive experiments on four public datasets to study the performance and cancelable biometric attributes of the CFVNet-based recognition system. The average accuracy, EERs and Dsys on the four datasets are 99.82%, 0.01% and 0.025, respectively, and achieves competitive performance compared with the state-of-the-arts.

Title: Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method

• Authors: Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14781
• Pdf URL: https://arxiv.org/pdf/2409.14781
• Copy Paste: [[2409.14781]] Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method(https://arxiv.org/abs/2409.14781)
• Keywords: large language model
• Abstract: As the scale of training corpora for large language models (LLMs) grows, model developers become increasingly reluctant to disclose details on their data. This lack of transparency poses challenges to scientific evaluation and ethical deployment. Recently, pretraining data detection approaches, which infer whether a given text was part of an LLM's training data through black-box access, have been explored. The Min-K% Prob method, which has achieved state-of-the-art results, assumes that a non-training example tends to contain a few outlier words with low token probabilities. However, the effectiveness may be limited as it tends to misclassify non-training texts that contain many common words with high probabilities predicted by LLMs. To address this issue, we introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection. We compute the cross-entropy (i.e., the divergence) between the token probability distribution and the token frequency distribution to derive a detection score.We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text. Experimental results on English-language benchmarks and PatentMIA demonstrate that our proposed method significantly outperforms existing methods. Our code and PatentMIA benchmark are available at this https URL

Title: Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Models

• Authors: Patrick Amadeus Irawan, Genta Indra Winata, Samuel Cahyawijaya, Ayu Purwarianti
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14785
• Pdf URL: https://arxiv.org/pdf/2409.14785
• Copy Paste: [[2409.14785]] Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Models(https://arxiv.org/abs/2409.14785)
• Keywords: robust
• Abstract: Natural Language Explanation (NLE) aims to elucidate the decision-making process by providing detailed, human-friendly explanations in natural language. It helps demystify the decision-making processes of large vision-language models (LVLMs) through the use of language models. While existing methods for creating a Vision Question-Answering with Natural Language Explanation (VQA-NLE) datasets can provide explanations, they heavily rely on human annotations that are time-consuming and costly. In this study, we propose a novel approach that leverages LVLMs to efficiently generate high-quality synthetic VQA-NLE datasets. By evaluating our synthetic data, we showcase how advanced prompting techniques can lead to the production of high-quality VQA-NLE data. Our findings indicate that this proposed method achieves up to 20x faster than human annotation, with only a minimal decrease in qualitative metrics, achieving robust quality that is nearly equivalent to human-annotated data. Furthermore, we show that incorporating visual prompts significantly enhances the relevance of text generation. Our study paves the way for a more efficient and robust automated generation of multi-modal NLE data, offering a promising solution to the problem.

Title: Advancing Depression Detection on Social Media Platforms Through Fine-Tuned Large Language Models

• Authors: Shahid Munir Shah, Syeda Anshrah Gillani, Mirza Samad Ahmed Baig, Muhammad Aamer Saleem, Muhammad Hamzah Siddiqui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14794
• Pdf URL: https://arxiv.org/pdf/2409.14794
• Copy Paste: [[2409.14794]] Advancing Depression Detection on Social Media Platforms Through Fine-Tuned Large Language Models(https://arxiv.org/abs/2409.14794)
• Keywords: robust, large language model
• Abstract: This study investigates the use of Large Language Models (LLMs) for improved depression detection from users social media data. Through the use of fine-tuned GPT 3.5 Turbo 1106 and LLaMA2-7B models and a sizable dataset from earlier studies, we were able to identify depressed content in social media posts with a high accuracy of nearly 96.0 percent. The comparative analysis of the obtained results with the relevant studies in the literature shows that the proposed fine-tuned LLMs achieved enhanced performance compared to existing state of the-art systems. This demonstrates the robustness of LLM-based fine-tuned systems to be used as potential depression detection systems. The study describes the approach in depth, including the parameters used and the fine-tuning procedure, and it addresses the important implications of our results for the early diagnosis of depression on several social media platforms.

Title: Research on Dynamic Data Flow Anomaly Detection based on Machine Learning

• Authors: Liyang Wang, Yu Cheng, Hao Gong, Jiacheng Hu, Xirui Tang, Iris Li
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14796
• Pdf URL: https://arxiv.org/pdf/2409.14796
• Copy Paste: [[2409.14796]] Research on Dynamic Data Flow Anomaly Detection based on Machine Learning(https://arxiv.org/abs/2409.14796)
• Keywords: security, attack, robust
• Abstract: The sophistication and diversity of contemporary cyberattacks have rendered the use of proxies, gateways, firewalls, and encrypted tunnels as a standalone defensive strategy inadequate. Consequently, the proactive identification of data anomalies has emerged as a prominent area of research within the field of data security. The majority of extant studies concentrate on sample equilibrium data, with the consequence that the detection effect is not optimal in the context of unbalanced data. In this study, the unsupervised learning method is employed to identify anomalies in dynamic data flows. Initially, multi-dimensional features are extracted from real-time data, and a clustering algorithm is utilised to analyse the patterns of the data. This enables the potential outliers to be automatically identified. By clustering similar data, the model is able to detect data behaviour that deviates significantly from normal traffic without the need for labelled data. The results of the experiments demonstrate that the proposed method exhibits high accuracy in the detection of anomalies across a range of scenarios. Notably, it demonstrates robust and adaptable performance, particularly in the context of unbalanced data.

Title: PrivaMatch: A Privacy-Preserving DNA Matching Scheme for Forensic Investigation

• Authors: Sankha Das
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14798
• Pdf URL: https://arxiv.org/pdf/2409.14798
• Copy Paste: [[2409.14798]] PrivaMatch: A Privacy-Preserving DNA Matching Scheme for Forensic Investigation(https://arxiv.org/abs/2409.14798)
• Keywords: security, privacy
• Abstract: DNA fingerprinting and matching for identifying suspects has been a common practice in criminal investigation. Such proceedings involve multiple parties such as investigating agencies, suspects and forensic labs. A major challenge in such settings is to carry out the matching process between the suspects' DNA samples and the samples obtained from the crime scene without compromising the privacy of the suspects' DNA profiles. Additionally, it is necessary that sensitive details pertaining to the investigation such as the identities of the suspects and evidence obtained from the crime scene must be kept private to the investigating agency. We present a novel DNA matching scheme, termed as PrivaMatch, which addresses multiple concerns about privacy of the suspects' DNA profiles and the crime scene evidence. In the proposed scheme, the investigating agencies oblivious transfer and zero-knowledge proofs to privately obtain the DNA profiles of the suspects from the forensic lab's this http URL addition, we present a clever data obfuscation technique using homomorphic encryption and modular arithmetic for the investigating agency to privately obtain the DNA profile of the crime scene's sample, keeping the profile oblivious from the forensic lab. The DNA profile of the crime scene sample is operated on using a homomorphic cryptosystem such that neither of the parties (e.g., the investigation agency, forensic labs, DNA database owners) learns about the private data of the other parties. The proposed scheme is analysed formally and the practicality of its security strengths is verified using simulations under standard assumptions.

Title: MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations

• Authors: Gia-Bao Dinh Ho, Chang Wei Tan, Zahra Zamanzadeh Darban, Mahsa Salehi, Gholamreza Haffari, Wray Buntine
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14801
• Pdf URL: https://arxiv.org/pdf/2409.14801
• Copy Paste: [[2409.14801]] MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations(https://arxiv.org/abs/2409.14801)
• Keywords: large language model
• Abstract: Detecting critical moments, such as emotional outbursts or changes in decisions during conversations, is crucial for understanding shifts in human behavior and their consequences. Our work introduces a novel problem setting focusing on these moments as turning points (TPs), accompanied by a meticulously curated, high-consensus, human-annotated multi-modal dataset. We provide precise timestamps, descriptions, and visual-textual evidence high-lighting changes in emotions, behaviors, perspectives, and decisions at these turning points. We also propose a framework, TPMaven, utilizing state-of-the-art vision-language models to construct a narrative from the videos and large language models to classify and detect turning points in our multi-modal dataset. Evaluation results show that TPMaven achieves an F1-score of 0.88 in classification and 0.61 in detection, with additional explanations aligning with human expectations.

Title: SDBA: A Stealthy and Long-Lasting Durable Backdoor Attack in Federated Learning

• Authors: Minyeong Choe, Cheolhee Park, Changho Seo, Hyunil Kim
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.14805
• Pdf URL: https://arxiv.org/pdf/2409.14805
• Copy Paste: [[2409.14805]] SDBA: A Stealthy and Long-Lasting Durable Backdoor Attack in Federated Learning(https://arxiv.org/abs/2409.14805)
• Keywords: privacy, defense, attack, robust, steal, federate
• Abstract: Federated Learning is a promising approach for training machine learning models while preserving data privacy, but its distributed nature makes it vulnerable to backdoor attacks, particularly in NLP tasks while related research remains limited. This paper introduces SDBA, a novel backdoor attack mechanism designed for NLP tasks in FL environments. Our systematic analysis across LSTM and GPT-2 models identifies the most vulnerable layers for backdoor injection and achieves both stealth and long-lasting durability through layer-wise gradient masking and top-k% gradient masking within these layers. Experiments on next token prediction and sentiment analysis tasks show that SDBA outperforms existing backdoors in durability and effectively bypasses representative defense mechanisms, with notable performance in LLM such as GPT-2. These results underscore the need for robust defense strategies in NLP-based FL systems.

Title: Past Meets Present: Creating Historical Analogy with Large Language Models

• Authors: Nianqi Li, Siyu Yuan, Jiangjie Chen, Jiaqing Liang, Feng Wei, Zujie Liang, Deqing Yang, Yanghua Xiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14820
• Pdf URL: https://arxiv.org/pdf/2409.14820
• Copy Paste: [[2409.14820]] Past Meets Present: Creating Historical Analogy with Large Language Models(https://arxiv.org/abs/2409.14820)
• Keywords: large language model
• Abstract: Historical analogies, which compare known past events with contemporary but unfamiliar events, are important abilities that help people make decisions and understand the world. However, research in applied history suggests that people have difficulty finding appropriate analogies. And previous studies in the AI community have also overlooked historical analogies. To fill this gap, in this paper, we focus on the historical analogy acquisition task, which aims to acquire analogous historical events for a given event. We explore retrieval and generation methods for acquiring historical analogies based on different large language models (LLMs). Furthermore, we propose a self-reflection method to mitigate hallucinations and stereotypes when LLMs generate historical analogies. Through human evaluations and our specially designed automatic multi-dimensional assessment, we find that LLMs generally have a good potential for historical analogies. And the performance of the models can be further improved by using our self-reflection method.

Title: Two Deep Learning Solutions for Automatic Blurring of Faces in Videos

• Authors: Roman Plaud, Jose-Luis Lisani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14828
• Pdf URL: https://arxiv.org/pdf/2409.14828
• Copy Paste: [[2409.14828]] Two Deep Learning Solutions for Automatic Blurring of Faces in Videos(https://arxiv.org/abs/2409.14828)
• Keywords: privacy, segmentation
• Abstract: The widespread use of cameras in everyday life situations generates a vast amount of data that may contain sensitive information about the people and vehicles moving in front of them (location, license plates, physical characteristics, etc). In particular, people's faces are recorded by surveillance cameras in public spaces. In order to ensure the privacy of individuals, face blurring techniques can be applied to the collected videos. In this paper we present two deep-learning based options to tackle the problem. First, a direct approach, consisting of a classical object detector (based on the YOLO architecture) trained to detect faces, which are subsequently blurred. Second, an indirect approach, in which a Unet-like segmentation network is trained to output a version of the input image in which all the faces have been blurred.

Title: Identify As A Human Does: A Pathfinder of Next-Generation Anti-Cheat Framework for First-Person Shooter Games

• Authors: Jiayi Zhang, Chenxin Sun, Yue Gu, Qingyu Zhang, Jiayi Lin, Xiaojiang Du, Chenxiong Qian
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14830
• Pdf URL: https://arxiv.org/pdf/2409.14830
• Copy Paste: [[2409.14830]] Identify As A Human Does: A Pathfinder of Next-Generation Anti-Cheat Framework for First-Person Shooter Games(https://arxiv.org/abs/2409.14830)
• Keywords: security
• Abstract: The gaming industry has experienced substantial growth, but cheating in online games poses a significant threat to the integrity of the gaming experience. Cheating, particularly in first-person shooter (FPS) games, can lead to substantial losses for the game industry. Existing anti-cheat solutions have limitations, such as client-side hardware constraints, security risks, server-side unreliable methods, and both-sides suffer from a lack of comprehensive real-world datasets. To address these limitations, the paper proposes HAWK, a server-side FPS anti-cheat framework for the popular game CS:GO. HAWK utilizes machine learning techniques to mimic human experts' identification process, leverages novel multi-view features, and it is equipped with a well-defined workflow. The authors evaluate HAWK with the first large and real-world datasets containing multiple cheat types and cheating sophistication, and it exhibits promising efficiency and acceptable overheads, shorter ban times compared to the in-use anti-cheat, a significant reduction in manual labor, and the ability to capture cheaters who evaded official inspections.

Title: GroCo: Ground Constraint for Metric Self-Supervised Monocular Depth

• Authors: Aurélien Cecille, Stefan Duffner, Franck Davoine, Thibault Neveu, Rémi Agier
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.14850
• Pdf URL: https://arxiv.org/pdf/2409.14850
• Copy Paste: [[2409.14850]] GroCo: Ground Constraint for Metric Self-Supervised Monocular Depth(https://arxiv.org/abs/2409.14850)
• Keywords: robust
• Abstract: Monocular depth estimation has greatly improved in the recent years but models predicting metric depth still struggle to generalize across diverse camera poses and datasets. While recent supervised methods mitigate this issue by leveraging ground prior information at inference, their adaptability to self-supervised settings is limited due to the additional challenge of scale recovery. Addressing this gap, we propose in this paper a novel constraint on ground areas designed specifically for the self-supervised paradigm. This mechanism not only allows to accurately recover the scale but also ensures coherence between the depth prediction and the ground prior. Experimental results show that our method surpasses existing scale recovery techniques on the KITTI benchmark and significantly enhances model generalization capabilities. This improvement can be observed by its more robust performance across diverse camera rotations and its adaptability in zero-shot conditions with previously unseen driving datasets such as DDAD.

Title: Disentanglement with Factor Quantized Variational Autoencoders

• Authors: Gulcin Baykal, Melih Kandemir, Gozde Unal
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14851
• Pdf URL: https://arxiv.org/pdf/2409.14851
• Copy Paste: [[2409.14851]] Disentanglement with Factor Quantized Variational Autoencoders(https://arxiv.org/abs/2409.14851)
• Keywords: generative
• Abstract: Disentangled representation learning aims to represent the underlying generative factors of a dataset in a latent representation independently of one another. In our work, we propose a discrete variational autoencoder (VAE) based model where the ground truth information about the generative factors are not provided to the model. We demonstrate the advantages of learning discrete representations over learning continuous representations in facilitating disentanglement. Furthermore, we propose incorporating an inductive bias into the model to further enhance disentanglement. Precisely, we propose scalar quantization of the latent variables in a latent representation with scalar values from a global codebook, and we add a total correlation term to the optimization as an inductive bias. Our method called FactorQVAE is the first method that combines optimization based disentanglement approaches with discrete representation learning, and it outperforms the former disentanglement methods in terms of two disentanglement metrics (DCI and InfoMEC) while improving the reconstruction performance. Our code can be found at \url{this https URL}.

Title: Effective and Evasive Fuzz Testing-Driven Jailbreaking Attacks against LLMs

• Authors: Xueluan Gong, Mingzhe Li, Yilin Zhang, Fengyuan Ran, Chen Chen, Yanjiao Chen, Qian Wang, Kwok-Yan Lam
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14866
• Pdf URL: https://arxiv.org/pdf/2409.14866
• Copy Paste: [[2409.14866]] Effective and Evasive Fuzz Testing-Driven Jailbreaking Attacks against LLMs(https://arxiv.org/abs/2409.14866)
• Keywords: defense, attack, robust, large language model
• Abstract: Large Language Models (LLMs) have excelled in various tasks but are still vulnerable to jailbreaking attacks, where attackers create jailbreak prompts to mislead the model to produce harmful or offensive content. Current jailbreak methods either rely heavily on manually crafted templates, which pose challenges in scalability and adaptability, or struggle to generate semantically coherent prompts, making them easy to detect. Additionally, most existing approaches involve lengthy prompts, leading to higher query this http URL this paper, to remedy these challenges, we introduce a novel jailbreaking attack framework, which is an automated, black-box jailbreaking attack framework that adapts the black-box fuzz testing approach with a series of customized designs. Instead of relying on manually crafted templates, our method starts with an empty seed pool, removing the need to search for any related jailbreaking templates. We also develop three novel question-dependent mutation strategies using an LLM helper to generate prompts that maintain semantic coherence while significantly reducing their length. Additionally, we implement a two-level judge module to accurately detect genuine successful jailbreaks. We evaluated our method on 7 representative LLMs and compared it with 5 state-of-the-art jailbreaking attack strategies. For proprietary LLM APIs, such as GPT-3.5 turbo, GPT-4, and Gemini-Pro, our method achieves attack success rates of over 90%, 80%, and 74%, respectively, exceeding existing baselines by more than 60%. Additionally, our method can maintain high semantic coherence while significantly reducing the length of jailbreak prompts. When targeting GPT-4, our method can achieve over 78\% attack success rate even with 100 tokens. Moreover, our method demonstrates transferability and is robust to state-of-the-art defenses. We will open-source our codes upon publication.

Title: Testing Dependency of Weighted Random Graphs

• Authors: Mor Oren, Vered Paslev, Wasim Huleihel
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2409.14870
• Pdf URL: https://arxiv.org/pdf/2409.14870
• Copy Paste: [[2409.14870]] Testing Dependency of Weighted Random Graphs(https://arxiv.org/abs/2409.14870)
• Keywords: generative
• Abstract: In this paper, we study the task of detecting the edge dependency between two weighted random graphs. We formulate this task as a simple hypothesis testing problem, where under the null hypothesis, the two observed graphs are statistically independent, whereas under the alternative, the edges of one graph are dependent on the edges of a randomly vertex-permuted version of the other graph. For general edge-weights distributions, we establish thresholds at which optimal testing is information-theoretically impossible and possible, as a function of the total number of nodes in the observed graphs and the generative distributions of the weights. Finally, we observe a statistical-computational gap in our problem, and we provide evidence that this is fundamental using the framework of low-degree polynomials.

Title: Mammo-Clustering:A Weakly Supervised Multi-view Global-Local Context Clustering Network for Detection and Classification in Mammography

• Authors: Shilong Yang, Chulong Zhang, Qi Zang, Juan Yu, Liang Zeng, Xiao Luo, Yexuan Xing, Xin Pan, Qi Li, Xiaokun Liang, Yaoqin Xie
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14876
• Pdf URL: https://arxiv.org/pdf/2409.14876
• Copy Paste: [[2409.14876]] Mammo-Clustering:A Weakly Supervised Multi-view Global-Local Context Clustering Network for Detection and Classification in Mammography(https://arxiv.org/abs/2409.14876)
• Keywords: extraction, transformer
• Abstract: Breast cancer has long posed a significant threat to women's health, making early screening crucial for mitigating its impact. However, mammography, the preferred method for early screening, faces limitations such as the burden of double reading by radiologists, challenges in widespread adoption in remote and underdeveloped areas, and obstacles in intelligent early screening development due to data constraints. To address these challenges, we propose a weakly supervised multi-view mammography early screening model for breast cancer based on context clustering. Context clustering, a feature extraction structure that is neither CNN nor transformer, combined with multi-view learning for information complementation, presents a promising approach. The weak supervision design specifically addresses data limitations. Our model achieves state-of-the-art performance with fewer parameters on two public datasets, with an AUC of 0.828 on the Vindr-Mammo dataset and 0.805 on the CBIS-DDSM dataset. Our model shows potential in reducing the burden on doctors and increasing the feasibility of breast cancer screening for women in underdeveloped regions.

Title: Privacy Policy Analysis through Prompt Engineering for LLMs

• Authors: Arda Goknil, Femke B. Gelderblom, Simeon Tverdal, Shukun Tokas, Hui Song
• Subjects: cs.CL, cs.CY, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.14879
• Pdf URL: https://arxiv.org/pdf/2409.14879
• Copy Paste: [[2409.14879]] Privacy Policy Analysis through Prompt Engineering for LLMs(https://arxiv.org/abs/2409.14879)
• Keywords: privacy, robust, extraction, large language model
• Abstract: Privacy policies are often obfuscated by their complexity, which impedes transparency and informed consent. Conventional machine learning approaches for automatically analyzing these policies demand significant resources and substantial domain-specific training, causing adaptability issues. Moreover, they depend on extensive datasets that may require regular maintenance due to changing privacy concerns. In this paper, we propose, apply, and assess PAPEL (Privacy Policy Analysis through Prompt Engineering for LLMs), a framework harnessing the power of Large Language Models (LLMs) through prompt engineering to automate the analysis of privacy policies. PAPEL aims to streamline the extraction, annotation, and summarization of information from these policies, enhancing their accessibility and comprehensibility without requiring additional model training. By integrating zero-shot, one-shot, and few-shot learning approaches and the chain-of-thought prompting in creating predefined prompts and prompt templates, PAPEL guides LLMs to efficiently dissect, interpret, and synthesize the critical aspects of privacy policies into user-friendly summaries. We demonstrate the effectiveness of PAPEL with two applications: (i) annotation and (ii) contradiction analysis. We assess the ability of several LLaMa and GPT models to identify and articulate data handling practices, offering insights comparable to existing automated analysis approaches while reducing training efforts and increasing the adaptability to new analytical needs. The experiments demonstrate that the LLMs PAPEL utilizes (LLaMA and Chat GPT models) achieve robust performance in privacy policy annotation, with F1 scores reaching 0.8 and above (using the OPP-115 gold standard), underscoring the effectiveness of simpler prompts across various advanced language models.

Title: End-to-End Graph Flattening Method for Large Language Models

• Authors: Bin Hong, Jinze Wu, Jiayu Liu, Liang Ding, Jing Sha, Kai Zhang, Shijin Wang, Zhenya Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14880
• Pdf URL: https://arxiv.org/pdf/2409.14880
• Copy Paste: [[2409.14880]] End-to-End Graph Flattening Method for Large Language Models(https://arxiv.org/abs/2409.14880)
• Keywords: robust, interpretability, large language model
• Abstract: In recent years, the breakthrough of Large Language Models (LLMs) offers new ideas for achieving universal methods on graph data. The common practice of converting graphs into natural language for LLMs, which refers to graph flattening, exhibits good generalizability and interpretability. However, the poor organization of the textual format results in poor performance in long-distance scenario understanding. Inspired by human cognitive reasoning habits, we propose a novel method for graph flattening to fit LLMs, termed as End-to-End DAG-Path prompting (EEDP). Experiments on real-world datasets show that EEDP enhances the reasoning performance of LLMs in long-distance scenarios while maintaining excellent performance in short-distance scenarios, demonstrating good robustness in the face of distance variations.

Title: Advancing Video Quality Assessment for AIGC

• Authors: Xinli Yue, Jianhui Sun, Han Kong, Liangchao Yao, Tianyi Wang, Lei Li, Fengyun Rao, Jing Lv, Fan Xia, Yuetang Deng, Qian Wang, Lingchen Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14888
• Pdf URL: https://arxiv.org/pdf/2409.14888
• Copy Paste: [[2409.14888]] Advancing Video Quality Assessment for AIGC(https://arxiv.org/abs/2409.14888)
• Keywords: generative
• Abstract: In recent years, AI generative models have made remarkable progress across various domains, including text generation, image generation, and video generation. However, assessing the quality of text-to-video generation is still in its infancy, and existing evaluation frameworks fall short when compared to those for natural videos. Current video quality assessment (VQA) methods primarily focus on evaluating the overall quality of natural videos and fail to adequately account for the substantial quality discrepancies between frames in generated videos. To address this issue, we propose a novel loss function that combines mean absolute error with cross-entropy loss to mitigate inter-frame quality inconsistencies. Additionally, we introduce the innovative S2CNet technique to retain critical content, while leveraging adversarial training to enhance the model's generalization capabilities. Experimental results demonstrate that our method outperforms existing VQA techniques on the AIGC Video dataset, surpassing the previous state-of-the-art by 3.1% in terms of PLCC.

Title: Kriformer: A Novel Spatiotemporal Kriging Approach Based on Graph Transformers

• Authors: Renbin Pan, Feng Xiao, Hegui Zhang, Minyu Shen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.14906
• Pdf URL: https://arxiv.org/pdf/2409.14906
• Copy Paste: [[2409.14906]] Kriformer: A Novel Spatiotemporal Kriging Approach Based on Graph Transformers(https://arxiv.org/abs/2409.14906)
• Keywords: transformer
• Abstract: Accurately estimating data in sensor-less areas is crucial for understanding system dynamics, such as traffic state estimation and environmental monitoring. This study addresses challenges posed by sparse sensor deployment and unreliable data by framing the problem as a spatiotemporal kriging task and proposing a novel graph transformer model, Kriformer. This model estimates data at locations without sensors by mining spatial and temporal correlations, even with limited resources. Kriformer utilizes transformer architecture to enhance the model's perceptual range and solve edge information aggregation challenges, capturing spatiotemporal information effectively. A carefully constructed positional encoding module embeds the spatiotemporal features of nodes, while a sophisticated spatiotemporal attention mechanism enhances estimation accuracy. The multi-head spatial interaction attention module captures subtle spatial relationships between observed and unobserved locations. During training, a random masking strategy prompts the model to learn with partial information loss, allowing the spatiotemporal embedding and multi-head attention mechanisms to synergistically capture correlations among locations. Experimental results show that Kriformer excels in representation learning for unobserved locations, validated on two real-world traffic speed datasets, demonstrating its effectiveness in spatiotemporal kriging tasks.

Title: Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization

• Authors: Aseem Srivastava, Smriti Joshi, Tanmoy Chakraborty, Md Shad Akhtar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14907
• Pdf URL: https://arxiv.org/pdf/2409.14907
• Copy Paste: [[2409.14907]] Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization(https://arxiv.org/abs/2409.14907)
• Keywords: generative, large language model
• Abstract: In mental health counseling, condensing dialogues into concise and relevant summaries (aka counseling notes) holds pivotal significance. Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks; however, their adaptation to domain-specific intricacies remains challenging, especially within mental health contexts. Unlike standard LLMs, mental health experts first plan to apply domain knowledge in writing summaries. Our work enhances LLMs' ability by introducing a novel planning engine to orchestrate structuring knowledge alignment. To achieve high-order planning, we divide knowledge encapsulation into two major phases: (i) holding dialogue structure and (ii) incorporating domain-specific knowledge. We employ a planning engine on Llama-2, resulting in a novel framework, PIECE. Our proposed system employs knowledge filtering-cum-scaffolding to encapsulate domain knowledge. Additionally, PIECE leverages sheaf convolution learning to enhance its understanding of the dialogue's structural nuances. We compare PIECE with 14 baseline methods and observe a significant improvement across ROUGE and Bleurt scores. Further, expert evaluation and analyses validate the generation quality to be effective, sometimes even surpassing the gold standard. We further benchmark PIECE with other LLMs and report improvement, including Llama-2 (+2.72%), Mistral (+2.04%), and Zephyr (+1.59%), to justify the generalizability of the planning engine.

Title: With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models

• Authors: Tyler Loakman, Yucheng Li, Chenghua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14917
• Pdf URL: https://arxiv.org/pdf/2409.14917
• Copy Paste: [[2409.14917]] With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models(https://arxiv.org/abs/2409.14917)
• Keywords: large language model
• Abstract: Recently, Large Language Models (LLMs) and Vision Language Models (VLMs) have demonstrated aptitude as potential substitutes for human participants in experiments testing psycholinguistic phenomena. However, an understudied question is to what extent models that only have access to vision and text modalities are able to implicitly understand sound-based phenomena via abstract reasoning from orthography and imagery alone. To investigate this, we analyse the ability of VLMs and LLMs to demonstrate sound symbolism (i.e., to recognise a non-arbitrary link between sounds and concepts) as well as their ability to ``hear'' via the interplay of the language and vision modules of open and closed-source multimodal models. We perform multiple experiments, including replicating the classic Kiki-Bouba and Mil-Mal shape and magnitude symbolism tasks, and comparing human judgements of linguistic iconicity with that of LLMs. Our results show that VLMs demonstrate varying levels of agreement with human labels, and more task information may be required for VLMs versus their human counterparts for in silico experimentation. We additionally see through higher maximum agreement levels that Magnitude Symbolism is an easier pattern for VLMs to identify than Shape Symbolism, and that an understanding of linguistic iconicity is highly dependent on model size.

Title: Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely

• Authors: Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K. Qiu, Lili Qiu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.14924
• Pdf URL: https://arxiv.org/pdf/2409.14924
• Copy Paste: [[2409.14924]] Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely(https://arxiv.org/abs/2409.14924)
• Keywords: large language model
• Abstract: Large language models (LLMs) augmented with external data have demonstrated remarkable capabilities in completing real-world tasks. Techniques for integrating external data into LLMs, such as Retrieval-Augmented Generation (RAG) and fine-tuning, are gaining increasing attention and widespread application. Nonetheless, the effective deployment of data-augmented LLMs across various specialized fields presents substantial challenges. These challenges encompass a wide range of issues, from retrieving relevant data and accurately interpreting user intent to fully harnessing the reasoning capabilities of LLMs for complex tasks. We believe that there is no one-size-fits-all solution for data-augmented LLM applications. In practice, underperformance often arises from a failure to correctly identify the core focus of a task or because the task inherently requires a blend of multiple capabilities that must be disentangled for better resolution. In this survey, we propose a RAG task categorization method, classifying user queries into four levels based on the type of external data required and primary focus of the task: explicit fact queries, implicit fact queries, interpretable rationale queries, and hidden rationale queries. We define these levels of queries, provide relevant datasets, and summarize the key challenges and most effective techniques for addressing these challenges. Finally, we discuss three main forms of integrating external data into LLMs: context, small model, and fine-tuning, highlighting their respective strengths, limitations, and the types of problems they are suited to solve. This work aims to help readers thoroughly understand and decompose the data requirements and key bottlenecks in building LLM applications, offering solutions to the different challenges and serving as a guide to systematically developing such applications.

Title: Improving Adversarial Robustness for 3D Point Cloud Recognition at Test-Time through Purified Self-Training

• Authors: Jinpeng Lin, Xulei Yang, Tianrui Li, Xun Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14940
• Pdf URL: https://arxiv.org/pdf/2409.14940
• Copy Paste: [[2409.14940]] Improving Adversarial Robustness for 3D Point Cloud Recognition at Test-Time through Purified Self-Training(https://arxiv.org/abs/2409.14940)
• Keywords: attack, robust, generative
• Abstract: Recognizing 3D point cloud plays a pivotal role in many real-world applications. However, deploying 3D point cloud deep learning model is vulnerable to adversarial attacks. Despite many efforts into developing robust model by adversarial training, they may become less effective against emerging attacks. This limitation motivates the development of adversarial purification which employs generative model to mitigate the impact of adversarial attacks. In this work, we highlight the remaining challenges from two perspectives. First, the purification based method requires retraining the classifier on purified samples which introduces additional computation overhead. Moreover, in a more realistic scenario, testing samples arrives in a streaming fashion and adversarial samples are not isolated from clean samples. These challenges motivates us to explore dynamically update model upon observing testing samples. We proposed a test-time purified self-training strategy to achieve this objective. Adaptive thresholding and feature distribution alignment are introduced to improve the robustness of self-training. Extensive results on different adversarial attacks suggest the proposed method is complementary to purification based method in handling continually changing adversarial attacks on the testing data stream.

Title: Adaptive Learning on User Segmentation: Universal to Specific Representation via Bipartite Neural Interaction

• Authors: Xiaoyu Tan, Yongxin Deng, Chao Qu, Siqiao Xue, Xiaoming Shi, James Zhang, Xihe Qiu
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.14945
• Pdf URL: https://arxiv.org/pdf/2409.14945
• Copy Paste: [[2409.14945]] Adaptive Learning on User Segmentation: Universal to Specific Representation via Bipartite Neural Interaction(https://arxiv.org/abs/2409.14945)
• Keywords: robust, segmentation
• Abstract: Recently, models for user representation learning have been widely applied in click-through-rate (CTR) and conversion-rate (CVR) prediction. Usually, the model learns a universal user representation as the input for subsequent scenario-specific models. However, in numerous industrial applications (e.g., recommendation and marketing), the business always operates such applications as various online activities among different user segmentation. These segmentation are always created by domain experts. Due to the difference in user distribution (i.e., user segmentation) and business objectives in subsequent tasks, learning solely on universal representation may lead to detrimental effects on both model performance and robustness. In this paper, we propose a novel learning framework that can first learn general universal user representation through information bottleneck. Then, merge and learn a segmentation-specific or a task-specific representation through neural interaction. We design the interactive learning process by leveraging a bipartite graph architecture to model the representation learning and merging between contextual clusters and each user segmentation. Our proposed method is evaluated in two open-source benchmarks, two offline business datasets, and deployed on two online marketing applications to predict users' CVR. The results demonstrate that our method can achieve superior performance and surpass the baseline methods.

Title: A new baseline for edge detection: Make Encoder-Decoder great again

• Authors: Yachuan Li, Xavier Soria Pomab, Yongke Xi, Guanlin Li, Chaozhi Yang, Qian Xiao, Yun Bai, Zongmin LI
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14976
• Pdf URL: https://arxiv.org/pdf/2409.14976
• Copy Paste: [[2409.14976]] A new baseline for edge detection: Make Encoder-Decoder great again(https://arxiv.org/abs/2409.14976)
• Keywords: extraction
• Abstract: The performance of deep learning based edge detector has far exceeded that of humans, but the huge computational cost and complex training strategy hinder its further development and application. In this paper, we eliminate these complexities with a vanilla encoder-decoder based detector. Firstly, we design a bilateral encoder to decouple the extraction process of location features and semantic features. Since the location branch no longer provides cues for the semantic branch, the richness of features can be further compressed, which is the key to make our model more compact. We propose a cascaded feature fusion decoder, where the location features are progressively refined by semantic features. The refined location features are the only basis for generating the edge map. The coarse original location features and semantic features are avoided from direct contact with the final result. So the noise in the location features and the location error in the semantic features can be suppressed in the generated edge map. The proposed New Baseline for Edge Detection (NBED) achieves superior performance consistently across multiple edge detection benchmarks, even compared with those methods with huge computational cost and complex training strategy. The ODS of NBED on BSDS500 is 0.838, achieving state-of-the-art performance. Our study shows that what really matters in the current edge detection is high-quality features, and we can make the encoder-decoder based detector great again even without complex training strategies and huge computational cost. The code is available at this https URL.

Title: Dynamic Integration of Task-Specific Adapters for Class Incremental Learning

• Authors: Jiashuo Li, Shaokun Wang, Bo Qian, Yuhang He, Xing Wei, Yihong Gong
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.14983
• Pdf URL: https://arxiv.org/pdf/2409.14983
• Copy Paste: [[2409.14983]] Dynamic Integration of Task-Specific Adapters for Class Incremental Learning(https://arxiv.org/abs/2409.14983)
• Keywords: privacy
• Abstract: Non-exemplar class Incremental Learning (NECIL) enables models to continuously acquire new classes without retraining from scratch and storing old task exemplars, addressing privacy and storage issues. However, the absence of data from earlier tasks exacerbates the challenge of catastrophic forgetting in NECIL. In this paper, we propose a novel framework called Dynamic Integration of task-specific Adapters (DIA), which comprises two key components: Task-Specific Adapter Integration (TSAI) and Patch-Level Model Alignment. TSAI boosts compositionality through a patch-level adapter integration strategy, which provides a more flexible compositional solution while maintaining low computation costs. Patch-Level Model Alignment maintains feature consistency and accurate decision boundaries via two specialized mechanisms: Patch-Level Distillation Loss (PDL) and Patch-Level Feature Reconstruction method (PFR). Specifically, the PDL preserves feature-level consistency between successive models by implementing a distillation loss based on the contributions of patch tokens to new class learning. The PFR facilitates accurate classifier alignment by reconstructing old class features from previous tasks that adapt to new task knowledge. Extensive experiments validate the effectiveness of our DIA, revealing significant improvements on benchmark datasets in the NECIL setting, maintaining an optimal balance between computational complexity and accuracy. The full code implementation will be made publicly available upon the publication of this paper.

Title: SocialCircle+: Learning the Angle-based Conditioned Interaction Representation for Pedestrian Trajectory Prediction

• Authors: Conghao Wong, Beihao Xia, Ziqian Zou, Xinge You
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.14984
• Pdf URL: https://arxiv.org/pdf/2409.14984
• Copy Paste: [[2409.14984]] SocialCircle+: Learning the Angle-based Conditioned Interaction Representation for Pedestrian Trajectory Prediction(https://arxiv.org/abs/2409.14984)
• Keywords: explainability
• Abstract: Trajectory prediction is a crucial aspect of understanding human behaviors. Researchers have made efforts to represent socially interactive behaviors among pedestrians and utilize various networks to enhance prediction capability. Unfortunately, they still face challenges not only in fully explaining and measuring how these interactive behaviors work to modify trajectories but also in modeling pedestrians' preferences to plan or participate in social interactions in response to the changeable physical environments as extra conditions. This manuscript mainly focuses on the above explainability and conditionality requirements for trajectory prediction networks. Inspired by marine animals perceiving other companions and the environment underwater by echolocation, this work constructs an angle-based conditioned social interaction representation SocialCircle+ to represent the socially interactive context and its corresponding conditions. It employs a social branch and a conditional branch to describe how pedestrians are positioned in prediction scenes socially and physically in angle-based-cyclic-sequence forms. Then, adaptive fusion is applied to fuse the above conditional clues onto the social ones to learn the final interaction representation. Experiments demonstrate the superiority of SocialCircle+ with different trajectory prediction backbones. Moreover, counterfactual interventions have been made to simultaneously verify the modeling capacity of causalities among interactive variables and the conditioning capability.

Title: Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs

• Authors: Clément Christophe, Tathagata Raha, Svetlana Maslenkova, Muhammad Umar Salman, Praveen K Kanithi, Marco AF Pimentel, Shadab Khan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14988
• Pdf URL: https://arxiv.org/pdf/2409.14988
• Copy Paste: [[2409.14988]] Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs(https://arxiv.org/abs/2409.14988)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals the impact of each technique. While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. Complex prompt engineering methods further enhance performance. These findings show the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain.

Title: Enhancing Aspect-based Sentiment Analysis in Tourism Using Large Language Models and Positional Information

• Authors: Chun Xu, Mengmeng Wang, Yan Ren, Shaolin Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.14997
• Pdf URL: https://arxiv.org/pdf/2409.14997
• Copy Paste: [[2409.14997]] Enhancing Aspect-based Sentiment Analysis in Tourism Using Large Language Models and Positional Information(https://arxiv.org/abs/2409.14997)
• Keywords: extraction, large language model
• Abstract: Aspect-Based Sentiment Analysis (ABSA) in tourism plays a significant role in understanding tourists' evaluations of specific aspects of attractions, which is crucial for driving innovation and development in the tourism industry. However, traditional pipeline models are afflicted by issues such as error propagation and incomplete extraction of sentiment elements. To alleviate this issue, this paper proposes an aspect-based sentiment analysis model, ACOS_LLM, for Aspect-Category-Opinion-Sentiment Quadruple Extraction (ACOSQE). The model comprises two key stages: auxiliary knowledge generation and ACOSQE. Firstly, Adalora is used to fine-tune large language models for generating high-quality auxiliary knowledge. To enhance model efficiency, Sparsegpt is utilized to compress the fine-tuned model to 50% sparsity. Subsequently, Positional information and sequence modeling are employed to achieve the ACOSQE task, with auxiliary knowledge and the original text as inputs. Experiments are conducted on both self-created tourism datasets and publicly available datasets, Rest15 and Rest16. Results demonstrate the model's superior performance, with an F1 improvement of 7.49% compared to other models on the tourism dataset. Additionally, there is an F1 improvement of 0.05% and 1.06% on the Rest15 and Rest16 datasets, respectively.

Title: Generalizing monocular colonoscopy image depth estimation by uncertainty-based global and local fusion network

• Authors: Sijia Du, Chengfeng Zhou, Suncheng Xiang, Jianwei Xu, Dahong Qian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15006
• Pdf URL: https://arxiv.org/pdf/2409.15006
• Copy Paste: [[2409.15006]] Generalizing monocular colonoscopy image depth estimation by uncertainty-based global and local fusion network(https://arxiv.org/abs/2409.15006)
• Keywords: robust, transformer, segmentation
• Abstract: Objective: Depth estimation is crucial for endoscopic navigation and manipulation, but obtaining ground-truth depth maps in real clinical scenarios, such as the colon, is challenging. This study aims to develop a robust framework that generalizes well to real colonoscopy images, overcoming challenges like non-Lambertian surface reflection and diverse data distributions. Methods: We propose a framework combining a convolutional neural network (CNN) for capturing local features and a Transformer for capturing global information. An uncertainty-based fusion block was designed to enhance generalization by identifying complementary contributions from the CNN and Transformer branches. The network can be trained with simulated datasets and generalize directly to unseen clinical data without any fine-tuning. Results: Our method is validated on multiple datasets and demonstrates an excellent generalization ability across various datasets and anatomical structures. Furthermore, qualitative analysis in real clinical scenarios confirmed the robustness of the proposed method. Conclusion: The integration of local and global features through the CNN-Transformer architecture, along with the uncertainty-based fusion block, improves depth estimation performance and generalization in both simulated and real-world endoscopic environments. Significance: This study offers a novel approach to estimate depth maps for endoscopy images despite the complex conditions in clinic, serving as a foundation for endoscopic automatic navigation and other clinical tasks, such as polyp detection and segmentation.

Title: DepthART: Monocular Depth Estimation as Autoregressive Refinement Task

• Authors: Bulat Gabdullin, Nina Konovalova, Nikolay Patakin, Dmitry Senushkin, Anton Konushin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15010
• Pdf URL: https://arxiv.org/pdf/2409.15010
• Copy Paste: [[2409.15010]] DepthART: Monocular Depth Estimation as Autoregressive Refinement Task(https://arxiv.org/abs/2409.15010)
• Keywords: diffusion, transformer, generative
• Abstract: Despite recent success in discriminative approaches in monocular depth estimation its quality remains limited by training datasets. Generative approaches mitigate this issue by leveraging strong priors derived from training on internet-scale datasets. Recent studies have demonstrated that large text-to-image diffusion models achieve state-of-the-art results in depth estimation when fine-tuned on small depth datasets. Concurrently, autoregressive generative approaches, such as the Visual AutoRegressive modeling~(VAR), have shown promising results in conditioned image synthesis. Following the visual autoregressive modeling paradigm, we introduce the first autoregressive depth estimation model based on the visual autoregressive transformer. Our primary contribution is DepthART -- a novel training method formulated as Depth Autoregressive Refinement Task. Unlike the original VAR training procedure, which employs static targets, our method utilizes a dynamic target formulation that enables model self-refinement and incorporates multi-modal guidance during training. Specifically, we use model predictions as inputs instead of ground truth token maps during training, framing the objective as residual minimization. Our experiments demonstrate that the proposed training approach significantly outperforms visual autoregressive modeling via next-scale prediction in the depth estimation task. The Visual Autoregressive Transformer trained with our approach on Hypersim achieves superior results on a set of unseen benchmarks compared to other generative and discriminative baselines.

Title: Evaluating Synthetic Activations composed of SAE Latents in GPT-2

• Authors: Giorgi Giglemiani, Nora Petrova, Chatrik Singh Mangat, Jett Janiak, Stefan Heimersheim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15019
• Pdf URL: https://arxiv.org/pdf/2409.15019
• Copy Paste: [[2409.15019]] Evaluating Synthetic Activations composed of SAE Latents in GPT-2(https://arxiv.org/abs/2409.15019)
• Keywords: interpretability
• Abstract: Sparse Auto-Encoders (SAEs) are commonly employed in mechanistic interpretability to decompose the residual stream into monosemantic SAE latents. Recent work demonstrates that perturbing a model's activations at an early layer results in a step-function-like change in the model's final layer activations. Furthermore, the model's sensitivity to this perturbation differs between model-generated (real) activations and random activations. In our study, we assess model sensitivity in order to compare real activations to synthetic activations composed of SAE latents. Our findings indicate that synthetic activations closely resemble real activations when we control for the sparsity and cosine similarity of the constituent SAE latents. This suggests that real activations cannot be explained by a simple "bag of SAE latents" lacking internal structure, and instead suggests that SAE latents possess significant geometric and statistical properties. Notably, we observe that our synthetic activations exhibit less pronounced activation plateaus compared to those typically surrounding real activations.

Title: Cross Branch Feature Fusion Decoder for Consistency Regularization-based Semi-Supervised Change Detection

• Authors: Yan Xing, Qi'ao Xu, Jingcheng Zeng, Rui Huang, Sihua Gao, Weifeng Xu, Yuxiang Zhang, Wei Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15021
• Pdf URL: https://arxiv.org/pdf/2409.15021
• Copy Paste: [[2409.15021]] Cross Branch Feature Fusion Decoder for Consistency Regularization-based Semi-Supervised Change Detection(https://arxiv.org/abs/2409.15021)
• Keywords: transformer
• Abstract: Semi-supervised change detection (SSCD) utilizes partially labeled data and a large amount of unlabeled data to detect changes. However, the transformer-based SSCD network does not perform as well as the convolution-based SSCD network due to the lack of labeled data. To overcome this limitation, we introduce a new decoder called Cross Branch Feature Fusion CBFF, which combines the strengths of both local convolutional branch and global transformer branch. The convolutional branch is easy to learn and can produce high-quality features with a small amount of labeled data. The transformer branch, on the other hand, can extract global context features but is hard to learn without a lot of labeled data. Using CBFF, we build our SSCD model based on a strong-to-weak consistency strategy. Through comprehensive experiments on WHU-CD and LEVIR-CD datasets, we have demonstrated the superiority of our method over seven state-of-the-art SSCD methods.

Title: Generative LLM Powered Conversational AI Application for Personalized Risk Assessment: A Case Study in COVID-19

• Authors: Mohammad Amin Roshani, Xiangyu Zhou, Yao Qiang, Srinivasan Suresh, Steve Hicks, Usha Sethuraman, Dongxiao Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15027
• Pdf URL: https://arxiv.org/pdf/2409.15027
• Copy Paste: [[2409.15027]] Generative LLM Powered Conversational AI Application for Personalized Risk Assessment: A Case Study in COVID-19(https://arxiv.org/abs/2409.15027)
• Keywords: interpretability, generative, large language model
• Abstract: Large language models (LLMs) have shown remarkable capabilities in various natural language tasks and are increasingly being applied in healthcare domains. This work demonstrates a new LLM-powered disease risk assessment approach via streaming human-AI conversation, eliminating the need for programming required by traditional machine learning approaches. In a COVID-19 severity risk assessment case study, we fine-tune pre-trained generative LLMs (e.g., Llama2-7b and Flan-t5-xl) using a few shots of natural language examples, comparing their performance with traditional classifiers (i.e., Logistic Regression, XGBoost, Random Forest) that are trained de novo using tabular data across various experimental settings. We develop a mobile application that uses these fine-tuned LLMs as its generative AI (GenAI) core to facilitate real-time interaction between clinicians and patients, providing no-code risk assessment through conversational interfaces. This integration not only allows for the use of streaming Questions and Answers (QA) as inputs but also offers personalized feature importance analysis derived from the LLM's attention layers, enhancing the interpretability of risk assessments. By achieving high Area Under the Curve (AUC) scores with a limited number of fine-tuning samples, our results demonstrate the potential of generative LLMs to outperform discriminative classification methods in low-data regimes, highlighting their real-world adaptability and effectiveness. This work aims to fill the existing gap in leveraging generative LLMs for interactive no-code risk assessment and to encourage further research in this emerging field.

Title: Anomaly Detection from a Tensor Train Perspective

• Authors: Alejandro Mata Ali, Aitor Moreno Fdez. de Leceta, Jorge López Rubio
• Subjects: cs.LG, cs.CR, cs.ET, cs.IT, quant-ph
• Abstract URL: https://arxiv.org/abs/2409.15030
• Pdf URL: https://arxiv.org/pdf/2409.15030
• Copy Paste: [[2409.15030]] Anomaly Detection from a Tensor Train Perspective(https://arxiv.org/abs/2409.15030)
• Keywords: security, attack
• Abstract: We present a series of algorithms in tensor networks for anomaly detection in datasets, by using data compression in a Tensor Train representation. These algorithms consist of preserving the structure of normal data in compression and deleting the structure of anomalous data. The algorithms can be applied to any tensor network representation. We test the effectiveness of the methods with digits and Olivetti faces datasets and a cybersecurity dataset to determine cyber-attacks.

Title: Scaling Laws of Decoder-Only Models on the Multilingual Machine Translation Task

• Authors: Gaëtan Caillaut, Raheel Qader, Mariam Nakhlé, Jingshu Liu, Jean-Gabriel Barthélemy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15051
• Pdf URL: https://arxiv.org/pdf/2409.15051
• Copy Paste: [[2409.15051]] Scaling Laws of Decoder-Only Models on the Multilingual Machine Translation Task(https://arxiv.org/abs/2409.15051)
• Keywords: transformer, large language model
• Abstract: Recent studies have showcased remarkable capabilities of decoder-only models in many NLP tasks, including translation. Yet, the machine translation field has been largely dominated by encoder-decoder models based on the Transformer architecture. As a consequence, scaling laws of encoder-decoder models for neural machine translation have already been well studied, but decoder-only models have received less attention. This work explores the scaling laws of decoder-only models on the multilingual and multidomain translation task. We trained a collection of six decoder-only models, ranging from 70M to 7B parameters, on a sentence-level, multilingual and multidomain dataset. We conducted a series of experiments showing that the loss of decoder-only models can be estimated using a scaling law similar to the one discovered for large language models, but we also show that this scaling law has difficulties to generalize to too large models or to a different data distribution. We also study different scaling methods and show that scaling the depth and the width of a model lead to similar test loss improvements, but with different impact on the model's efficiency.

Title: Brotherhood at WMT 2024: Leveraging LLM-Generated Contextual Conversations for Cross-Lingual Image Captioning

• Authors: Siddharth Betala, Ishan Chokshi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15052
• Pdf URL: https://arxiv.org/pdf/2409.15052
• Copy Paste: [[2409.15052]] Brotherhood at WMT 2024: Leveraging LLM-Generated Contextual Conversations for Cross-Lingual Image Captioning(https://arxiv.org/abs/2409.15052)
• Keywords: large language model
• Abstract: In this paper, we describe our system under the team name Brotherhood for the English-to-Lowres Multi-Modal Translation Task. We participate in the multi-modal translation tasks for English-Hindi, English-Hausa, English-Bengali, and English-Malayalam language pairs. We present a method leveraging multi-modal Large Language Models (LLMs), specifically GPT-4o and Claude 3.5 Sonnet, to enhance cross-lingual image captioning without traditional training or fine-tuning. Our approach utilizes instruction-tuned prompting to generate rich, contextual conversations about cropped images, using their English captions as additional context. These synthetic conversations are then translated into the target languages. Finally, we employ a weighted prompting strategy, balancing the original English caption with the translated conversation to generate captions in the target language. This method achieved competitive results, scoring 37.90 BLEU on the English-Hindi Challenge Set and ranking first and second for English-Hausa on the Challenge and Evaluation Leaderboards, respectively. We conduct additional experiments on a subset of 250 images, exploring the trade-offs between BLEU scores and semantic similarity across various weighting schemes.

Title: FisheyeDepth: A Real Scale Self-Supervised Depth Estimation Model for Fisheye Camera

• Authors: Guoyang Zhao, Yuxuan Liu, Weiqing Qi, Fulong Ma, Ming Liu, Jun Ma
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.15054
• Pdf URL: https://arxiv.org/pdf/2409.15054
• Copy Paste: [[2409.15054]] FisheyeDepth: A Real Scale Self-Supervised Depth Estimation Model for Fisheye Camera(https://arxiv.org/abs/2409.15054)
• Keywords: robust
• Abstract: Accurate depth estimation is crucial for 3D scene comprehension in robotics and autonomous vehicles. Fisheye cameras, known for their wide field of view, have inherent geometric benefits. However, their use in depth estimation is restricted by a scarcity of ground truth data and image distortions. We present FisheyeDepth, a self-supervised depth estimation model tailored for fisheye cameras. We incorporate a fisheye camera model into the projection and reprojection stages during training to handle image distortions, thereby improving depth estimation accuracy and training stability. Furthermore, we incorporate real-scale pose information into the geometric projection between consecutive frames, replacing the poses estimated by the conventional pose network. Essentially, this method offers the necessary physical depth for robotic tasks, and also streamlines the training and inference procedures. Additionally, we devise a multi-channel output strategy to improve robustness by adaptively fusing features at various scales, which reduces the noise from real pose data. We demonstrate the superior performance and robustness of our model in fisheye image depth estimation through evaluations on public datasets and real-world scenarios. The project website is available at: this https URL.

Title: SHFL: Secure Hierarchical Federated Learning Framework for Edge Networks

• Authors: Omid Tavallaie, Kanchana Thilakarathna, Suranga Seneviratne, Aruna Seneviratne, Albert Y. Zomaya
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.15067
• Pdf URL: https://arxiv.org/pdf/2409.15067
• Copy Paste: [[2409.15067]] SHFL: Secure Hierarchical Federated Learning Framework for Edge Networks(https://arxiv.org/abs/2409.15067)
• Keywords: secure, privacy, attack, federate
• Abstract: Federated Learning (FL) is a distributed machine learning paradigm designed for privacy-sensitive applications that run on resource-constrained devices with non-Identically and Independently Distributed (IID) data. Traditional FL frameworks adopt the client-server model with a single-level aggregation (AGR) process, where the server builds the global model by aggregating all trained local models received from client devices. However, this conventional approach encounters challenges, including susceptibility to model/data poisoning attacks. In recent years, advancements in the Internet of Things (IoT) and edge computing have enabled the development of hierarchical FL systems with a two-level AGR process running at edge and cloud servers. In this paper, we propose a Secure Hierarchical FL (SHFL) framework to address poisoning attacks in hierarchical edge networks. By aggregating trained models at the edge, SHFL employs two novel methods to address model/data poisoning attacks in the presence of client adversaries: 1) a client selection algorithm running at the edge for choosing IoT devices to participate in training, and 2) a model AGR method designed based on convex optimization theory to reduce the impact of edge models from networks with adversaries in the process of computing the global model (at the cloud level). The evaluation results reveal that compared to state-of-the-art methods, SHFL significantly increases the maximum accuracy achieved by the global model in the presence of client adversaries applying model/data poisoning attacks.

Title: Evaluating the Usability of LLMs in Threat Intelligence Enrichment

• Authors: Sanchana Srikanth, Mohammad Hasanuzzaman, Farah Tasnur Meem
• Subjects: cs.CR, cs.CL, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15072
• Pdf URL: https://arxiv.org/pdf/2409.15072
• Copy Paste: [[2409.15072]] Evaluating the Usability of LLMs in Threat Intelligence Enrichment(https://arxiv.org/abs/2409.15072)
• Keywords: security, large language model
• Abstract: Large Language Models (LLMs) have the potential to significantly enhance threat intelligence by automating the collection, preprocessing, and analysis of threat data. However, the usability of these tools is critical to ensure their effective adoption by security professionals. Despite the advanced capabilities of LLMs, concerns about their reliability, accuracy, and potential for generating inaccurate information persist. This study conducts a comprehensive usability evaluation of five LLMs ChatGPT, Gemini, Cohere, Copilot, and Meta AI focusing on their user interface design, error handling, learning curve, performance, and integration with existing tools in threat intelligence enrichment. Utilizing a heuristic walkthrough and a user study methodology, we identify key usability issues and offer actionable recommendations for improvement. Our findings aim to bridge the gap between LLM functionality and user experience, thereby promoting more efficient and accurate threat intelligence practices by ensuring these tools are user-friendly and reliable.

Title: Enhancing Scientific Reproducibility Through Automated BioCompute Object Creation Using Retrieval-Augmented Generation from Publications

• Authors: Sean Kim, Raja Mazumder
• Subjects: cs.CL, cs.AI, q-bio.OT
• Abstract URL: https://arxiv.org/abs/2409.15076
• Pdf URL: https://arxiv.org/pdf/2409.15076
• Copy Paste: [[2409.15076]] Enhancing Scientific Reproducibility Through Automated BioCompute Object Creation Using Retrieval-Augmented Generation from Publications(https://arxiv.org/abs/2409.15076)
• Keywords: extraction, large language model
• Abstract: The exponential growth in computational power and accessibility has transformed the complexity and scale of bioinformatics research, necessitating standardized documentation for transparency, reproducibility, and regulatory compliance. The IEEE BioCompute Object (BCO) standard addresses this need but faces adoption challenges due to the overhead of creating compliant documentation, especially for legacy research. This paper presents a novel approach to automate the creation of BCOs from scientific papers using Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs). We describe the development of the BCO assistant tool that leverages RAG to extract relevant information from source papers and associated code repositories, addressing key challenges such as LLM hallucination and long-context understanding. The implementation incorporates optimized retrieval processes, including a two-pass retrieval with re-ranking, and employs carefully engineered prompts for each BCO domain. We discuss the tool's architecture, extensibility, and evaluation methods, including automated and manual assessment approaches. The BCO assistant demonstrates the potential to significantly reduce the time and effort required for retroactive documentation of bioinformatics research while maintaining compliance with the standard. This approach opens avenues for AI-assisted scientific documentation and knowledge extraction from publications thereby enhancing scientific reproducibility. The BCO assistant tool and documentation is available at this https URL.

Title: TSCLIP: Robust CLIP Fine-Tuning for Worldwide Cross-Regional Traffic Sign Recognition

• Authors: Guoyang Zhao, Fulong Ma, Weiqing Qi, Chenguang Zhang, Yuxuan Liu, Ming Liu, Jun Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15077
• Pdf URL: https://arxiv.org/pdf/2409.15077
• Copy Paste: [[2409.15077]] TSCLIP: Robust CLIP Fine-Tuning for Worldwide Cross-Regional Traffic Sign Recognition(https://arxiv.org/abs/2409.15077)
• Keywords: robust
• Abstract: Traffic sign is a critical map feature for navigation and traffic control. Nevertheless, current methods for traffic sign recognition rely on traditional deep learning models, which typically suffer from significant performance degradation considering the variations in data distribution across different regions. In this paper, we propose TSCLIP, a robust fine-tuning approach with the contrastive language-image pre-training (CLIP) model for worldwide cross-regional traffic sign recognition. We first curate a cross-regional traffic sign benchmark dataset by combining data from ten different sources. Then, we propose a prompt engineering scheme tailored to the characteristics of traffic signs, which involves specific scene descriptions and corresponding rules to generate targeted text descriptions for optimizing the model training process. During the TSCLIP fine-tuning process, we implement adaptive dynamic weight ensembling (ADWE) to seamlessly incorporate outcomes from each training iteration with the zero-shot CLIP model. This approach ensures that the model retains its ability to generalize while acquiring new knowledge about traffic signs. Our method surpasses conventional classification benchmark models in cross-regional traffic sign evaluations, and it achieves state-of-the-art performance compared to existing CLIP fine-tuning techniques. To the best knowledge of authors, TSCLIP is the first contrastive language-image model used for the worldwide cross-regional traffic sign recognition task. The project website is available at: this https URL.

Title: AdapFair: Ensuring Continuous Fairness for Machine Learning Operations

• Authors: Yinghui Huang, Zihao Tang, Xiangyu Chang
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.15088
• Pdf URL: https://arxiv.org/pdf/2409.15088
• Copy Paste: [[2409.15088]] AdapFair: Ensuring Continuous Fairness for Machine Learning Operations(https://arxiv.org/abs/2409.15088)
• Keywords: fair
• Abstract: The biases and discrimination of machine learning algorithms have attracted significant attention, leading to the development of various algorithms tailored to specific contexts. However, these solutions often fall short of addressing fairness issues inherent in machine learning operations. In this paper, we present a debiasing framework designed to find an optimal fair transformation of input data that maximally preserves data predictability. A distinctive feature of our approach is its flexibility and efficiency. It can be integrated with any downstream black-box classifiers, providing continuous fairness guarantees with minimal retraining efforts, even in the face of frequent data drifts, evolving fairness requirements, and batches of similar tasks. To achieve this, we leverage the normalizing flows to enable efficient, information-preserving data transformation, ensuring that no critical information is lost during the debiasing process. Additionally, we incorporate the Wasserstein distance as the unfairness measure to guide the optimization of data transformations. Finally, we introduce an efficient optimization algorithm with closed-formed gradient computations, making our framework scalable and suitable for dynamic, real-world environments.

Title: M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Images

• Authors: Hongyi Wang, Xiuju Du, Jing Liu, Shuyi Ouyang, Yen-Wei Chen, Lanfen Lin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15092
• Pdf URL: https://arxiv.org/pdf/2409.15092
• Copy Paste: [[2409.15092]] M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Images(https://arxiv.org/abs/2409.15092)
• Keywords: transformer
• Abstract: The advancement of Spatial Transcriptomics (ST) has facilitated the spatially-aware profiling of gene expressions based on histopathology images. Although ST data offers valuable insights into the micro-environment of tumors, its acquisition cost remains expensive. Therefore, directly predicting the ST expressions from digital pathology images is desired. Current methods usually adopt existing regression backbones along with patch-sampling for this task, which ignores the inherent multi-scale information embedded in the pyramidal data structure of digital pathology images, and wastes the inter-spot visual information crucial for accurate gene expression prediction. To address these limitations, we propose M2OST, a many-to-one regression Transformer that can accommodate the hierarchical structure of the pathology images via a decoupled multi-scale feature extractor. Unlike traditional models that are trained with one-to-one image-label pairs, M2OST uses multiple images from different levels of the digital pathology image to jointly predict the gene expressions in their common corresponding spot. Built upon our many-to-one scheme, M2OST can be easily scaled to fit different numbers of inputs, and its network structure inherently incorporates nearby inter-spot features, enhancing regression performance. We have tested M2OST on three public ST datasets and the experimental results show that M2OST can achieve state-of-the-art performance with fewer parameters and floating-point operations (FLOPs). The code will be released upon acceptance.

Title: Efficiently Dispatching Flash Attention For Partially Filled Attention Masks

• Authors: Agniv Sharma, Jonas Geiping
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15097
• Pdf URL: https://arxiv.org/pdf/2409.15097
• Copy Paste: [[2409.15097]] Efficiently Dispatching Flash Attention For Partially Filled Attention Masks(https://arxiv.org/abs/2409.15097)
• Keywords: transformer
• Abstract: Transformers are widely used across various applications, many of which yield sparse or partially filled attention matrices. Examples include attention masks designed to reduce the quadratic complexity of attention, sequence packing techniques, and recent innovations like tree masking for fast validation in MEDUSA. Despite the inherent sparsity in these matrices, the state-of-the-art algorithm Flash Attention still processes them with quadratic complexity as though they were dense. In this paper, we introduce \textbf{Binary Block Masking}, a highly efficient modification that enhances Flash Attention by making it mask-aware. We further propose two optimizations: one tailored for masks with contiguous non-zero patterns and another for extremely sparse masks. Our experiments on attention masks derived from real-world scenarios demonstrate up to a 9x runtime improvement. The implementation will be publicly released to foster further research and application.

Title: Robust Federated Learning Over the Air: Combating Heavy-Tailed Noise with Median Anchored Clipping

• Authors: Jiaxing Li, Zihan Chen, Kai Fong Ernest Chong, Bikramjit Das, Tony Q. S. Quek, Howard H. Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15100
• Pdf URL: https://arxiv.org/pdf/2409.15100
• Copy Paste: [[2409.15100]] Robust Federated Learning Over the Air: Combating Heavy-Tailed Noise with Median Anchored Clipping(https://arxiv.org/abs/2409.15100)
• Keywords: privacy, robust, federate
• Abstract: Leveraging over-the-air computations for model aggregation is an effective approach to cope with the communication bottleneck in federated edge learning. By exploiting the superposition properties of multi-access channels, this approach facilitates an integrated design of communication and computation, thereby enhancing system privacy while reducing implementation costs. However, the inherent electromagnetic interference in radio channels often exhibits heavy-tailed distributions, giving rise to exceptionally strong noise in globally aggregated gradients that can significantly deteriorate the training performance. To address this issue, we propose a novel gradient clipping method, termed Median Anchored Clipping (MAC), to combat the detrimental effects of heavy-tailed noise. We also derive analytical expressions for the convergence rate of model training with analog over-the-air federated learning under MAC, which quantitatively demonstrates the effect of MAC on training performance. Extensive experimental results show that the proposed MAC algorithm effectively mitigates the impact of heavy-tailed noise, hence substantially enhancing system robustness.

Title: The BRAVO Semantic Segmentation Challenge Results in UNCV2024

• Authors: Tuan-Hung Vu, Eduardo Valle, Andrei Bursuc, Tommie Kerssies, Daan de Geus, Gijs Dubbelman, Long Qian, Bingke Zhu, Yingying Chen, Ming Tang, Jinqiao Wang, Tomáš Vojíř, Jan Šochman, Jiří Matas, Michael Smith, Frank Ferrie, Shamik Basu, Christos Sakaridis, Luc Van Gool
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15107
• Pdf URL: https://arxiv.org/pdf/2409.15107
• Copy Paste: [[2409.15107]] The BRAVO Semantic Segmentation Challenge Results in UNCV2024(https://arxiv.org/abs/2409.15107)
• Keywords: robust, segmentation
• Abstract: We propose the unified BRAVO challenge to benchmark the reliability of semantic segmentation models under realistic perturbations and unknown out-of-distribution (OOD) scenarios. We define two categories of reliability: (1) semantic reliability, which reflects the model's accuracy and calibration when exposed to various perturbations; and (2) OOD reliability, which measures the model's ability to detect object classes that are unknown during training. The challenge attracted nearly 100 submissions from international teams representing notable research institutions. The results reveal interesting insights into the importance of large-scale pre-training and minimal architectural design in developing robust and reliable semantic segmentation models.

Title: Diffusion-based RGB-D Semantic Segmentation with Deformable Attention Transformer

• Authors: Minh Bui, Kostas Alexis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15117
• Pdf URL: https://arxiv.org/pdf/2409.15117
• Copy Paste: [[2409.15117]] Diffusion-based RGB-D Semantic Segmentation with Deformable Attention Transformer(https://arxiv.org/abs/2409.15117)
• Keywords: robust, diffusion, transformer, generative, segmentation
• Abstract: Vision-based perception and reasoning is essential for scene understanding in any autonomous system. RGB and depth images are commonly used to capture both the semantic and geometric features of the environment. Developing methods to reliably interpret this data is critical for real-world applications, where noisy measurements are often unavoidable. In this work, we introduce a diffusion-based framework to address the RGB-D semantic segmentation problem. Additionally, we demonstrate that utilizing a Deformable Attention Transformer as the encoder to extract features from depth images effectively captures the characteristics of invalid regions in depth measurements. Our generative framework shows a greater capacity to model the underlying distribution of RGB-D images, achieving robust performance in challenging scenarios with significantly less training time compared to discriminative methods. Experimental results indicate that our approach achieves State-of-the-Art performance on both the NYUv2 and SUN-RGBD datasets in general and especially in the most challenging of their image data. Our project page will be available at this https URL

Title: Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation

• Authors: Manu Gaur, Darshan Singh S, Makarand Tapaswi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15125
• Pdf URL: https://arxiv.org/pdf/2409.15125
• Copy Paste: [[2409.15125]] Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation(https://arxiv.org/abs/2409.15125)
• Keywords: large language model
• Abstract: Visual Question Answering (VQA) with multiple choice questions enables a vision-centric evaluation of Multimodal Large Language Models (MLLMs). Although it reliably checks the existence of specific visual abilities, it is easier for the model to select an answer from multiple choices (VQA evaluation) than to generate the answer itself. In this work, we offer a novel perspective: we evaluate how well an MLLM understands a specific visual concept by its ability to uniquely describe two extremely similar images that differ only in the targeted visual concept. Specifically, we assess the ability of MLLMs to capture specific points of visual differences using self-retrieval, i.e., by retrieving the target image using its generated caption against the other image in the pair serving as the distractor. We curate 247 highly similar image pairs as part of the D3 benchmark. For each image pair, the model is prompted to: (1) Detect a specific visual difference, and (2) Describe the target image uniquely such that it (3) Discriminates the target image from the distractor. Self-retrieval within D3 enables whitebox evaluation across six different visual patterns, revealing that current models struggle to independently discern fine-grained visual differences, with open-source models failing to outperform random guess.

Title: UTrace: Poisoning Forensics for Private Collaborative Learning

• Authors: Evan Rose, Hidde Lycklama, Harsh Chaudhari, Anwar Hithnawi, Alina Oprea
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15126
• Pdf URL: https://arxiv.org/pdf/2409.15126
• Copy Paste: [[2409.15126]] UTrace: Poisoning Forensics for Private Collaborative Learning(https://arxiv.org/abs/2409.15126)
• Keywords: secure, privacy, defense, attack
• Abstract: Privacy-preserving machine learning (PPML) enables multiple data owners to contribute their data privately to a set of servers that run a secure multi-party computation (MPC) protocol to train a joint ML model. In these protocols, the input data remains private throughout the training process, and only the resulting model is made available. While this approach benefits privacy, it also exacerbates the risks of data poisoning, where compromised data owners induce undesirable model behavior by contributing malicious datasets. Existing MPC mechanisms can mitigate certain poisoning attacks, but these measures are not exhaustive. To complement existing poisoning defenses, we introduce UTrace: a framework for User-level Traceback of poisoning attacks in PPML. Utrace computes user responsibility scores using gradient similarity metrics aggregated across the most relevant samples in an owner's dataset. UTrace is effective at low poisoning rates and is resilient to poisoning attacks distributed across multiple data owners, unlike existing unlearning-based methods. We introduce methods for checkpointing gradients with low storage overhead, enabling traceback in the absence of data owners at deployment time. We also design several optimizations that reduce traceback time and communication in MPC. We provide a comprehensive evaluation of UTrace across four datasets from three data modalities (vision, text, and malware) and show its effectiveness against 10 poisoning attacks.

Title: Designing an Interpretable Interface for Contextual Bandits

• Authors: Andrew Maher, Matia Gobbo, Lancelot Lachartre, Subash Prabanantham, Rowan Swiers, Puli Liyanagama
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.15143
• Pdf URL: https://arxiv.org/pdf/2409.15143
• Copy Paste: [[2409.15143]] Designing an Interpretable Interface for Contextual Bandits(https://arxiv.org/abs/2409.15143)
• Keywords: interpretability
• Abstract: Contextual bandits have become an increasingly popular solution for personalized recommender systems. Despite their growing use, the interpretability of these systems remains a significant challenge, particularly for the often non-expert operators tasked with ensuring their optimal performance. In this paper, we address this challenge by designing a new interface to explain to domain experts the underlying behaviour of a bandit. Central is a metric we term "value gain", a measure derived from off-policy evaluation to quantify the real-world impact of sub-components within a bandit. We conduct a qualitative user study to evaluate the effectiveness of our interface. Our findings suggest that by carefully balancing technical rigour with accessible presentation, it is possible to empower non-experts to manage complex machine learning systems. We conclude by outlining guiding principles that other researchers should consider when building similar such interfaces in future.

Title: Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling

• Authors: Lechao Xiao
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.15156
• Pdf URL: https://arxiv.org/pdf/2409.15156
• Copy Paste: [[2409.15156]] Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling(https://arxiv.org/abs/2409.15156)
• Keywords: large language model
• Abstract: The remarkable success of large language pretraining and the discovery of scaling laws signify a paradigm shift in machine learning. Notably, the primary objective has evolved from minimizing generalization error to reducing approximation error, and the most effective strategy has transitioned from regularization (in a broad sense) to scaling up models. This raises a critical question: Do the established principles that proved successful in the generalization-centric era remain valid in this new era of scaling? This paper examines several influential regularization-based principles that may no longer hold true in the scaling-centric, large language model (LLM) era. These principles include explicit L2 regularization and implicit regularization through small batch sizes and large learning rates. Additionally, we identify a new phenomenon termed ``scaling law crossover,'' where two scaling curves intersect at a certain scale, implying that methods effective at smaller scales may not generalize to larger ones. Together, these observations highlight two fundamental questions within this new paradigm: $\bullet$ Guiding Principles for Scaling: If regularization is no longer the primary guiding principle for model design, what new principles are emerging to guide scaling? $\bullet$ Model Comparison at Scale: How to reliably and effectively compare models at the scale where only a single experiment is feasible?

Title: A Gated Residual Kolmogorov-Arnold Networks for Mixtures of Experts

• Authors: Hugo Inzirillo, Remi Genet
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.15161
• Pdf URL: https://arxiv.org/pdf/2409.15161
• Copy Paste: [[2409.15161]] A Gated Residual Kolmogorov-Arnold Networks for Mixtures of Experts(https://arxiv.org/abs/2409.15161)
• Keywords: interpretability
• Abstract: This paper introduces KAMoE, a novel Mixture of Experts (MoE) framework based on Gated Residual Kolmogorov-Arnold Networks (GRKAN). We propose GRKAN as an alternative to the traditional gating function, aiming to enhance efficiency and interpretability in MoE modeling. Through extensive experiments on digital asset markets and real estate valuation, we demonstrate that KAMoE consistently outperforms traditional MoE architectures across various tasks and model types. Our results show that GRKAN exhibits superior performance compared to standard Gating Residual Networks, particularly in LSTM-based models for sequential tasks. We also provide insights into the trade-offs between model complexity and performance gains in MoE and KAMoE architectures.

Title: Lessons Learned on Information Retrieval in Electronic Health Records: A Comparison of Embedding Models and Pooling Strategies

• Authors: Skatje Myers, Timothy A. Miller, Yanjun Gao, Matthew M. Churpek, Anoop Mayampurath, Dmitriy Dligach, Majid Afshar
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.15163
• Pdf URL: https://arxiv.org/pdf/2409.15163
• Copy Paste: [[2409.15163]] Lessons Learned on Information Retrieval in Electronic Health Records: A Comparison of Embedding Models and Pooling Strategies(https://arxiv.org/abs/2409.15163)
• Keywords: large language model
• Abstract: Objective: Applying large language models (LLMs) to the clinical domain is challenging due to the context-heavy nature of processing medical records. Retrieval-augmented generation (RAG) offers a solution by facilitating reasoning over large text sources. However, there are many parameters to optimize in just the retrieval system alone. This paper presents an ablation study exploring how different embedding models and pooling methods affect information retrieval for the clinical domain. Methods: Evaluating on three retrieval tasks on two electronic health record (EHR) data sources, we compared seven models, including medical- and general-domain models, specialized encoder embedding models, and off-the-shelf decoder LLMs. We also examine the choice of embedding pooling strategy for each model, independently on the query and the text to retrieve. Results: We found that the choice of embedding model significantly impacts retrieval performance, with BGE, a comparatively small general-domain model, consistently outperforming all others, including medical-specific models. However, our findings also revealed substantial variability across datasets and query text phrasings. We also determined the best pooling methods for each of these models to guide future design of retrieval systems. Discussion: The choice of embedding model, pooling strategy, and query formulation can significantly impact retrieval performance and the performance of these models on other public benchmarks does not necessarily transfer to new domains. Further studies such as this one are vital for guiding empirically-grounded development of retrieval frameworks, such as in the context of RAG, for the clinical domain.

Title: CamLoPA: A Hidden Wireless Camera Localization Framework via Signal Propagation Path Analysis

• Authors: Xiang Zhang, Jie Zhang, Zehua Ma, Jinyang Huang, Meng Li, Huan Yan, Peng Zhao, Zijian Zhang, Qing Guo, Tianwei Zhang, Bin Liu, Nenghai Yu
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.15169
• Pdf URL: https://arxiv.org/pdf/2409.15169
• Copy Paste: [[2409.15169]] CamLoPA: A Hidden Wireless Camera Localization Framework via Signal Propagation Path Analysis(https://arxiv.org/abs/2409.15169)
• Keywords: privacy
• Abstract: Hidden wireless cameras pose significant privacy threats, necessitating effective detection and localization methods. However, existing solutions often require spacious activity areas, expensive specialized devices, or pre-collected training data, limiting their practical deployment. To address these limitations, we introduce CamLoPA, a training-free wireless camera detection and localization framework that operates with minimal activity space constraints using low-cost commercial-off-the-shelf (COTS) devices. CamLoPA can achieve detection and localization in just 45 seconds of user activities with a Raspberry Pi board. During this short period, it analyzes the causal relationship between the wireless traffic and user movement to detect the presence of a snooping camera. Upon detection, CamLoPA employs a novel azimuth location model based on wireless signal propagation path analysis. Specifically, this model leverages the time ratio of user paths crossing the First Fresnel Zone (FFZ) to determine the azimuth angle of the camera. Then CamLoPA refines the localization by identifying the camera's quadrant. We evaluate CamLoPA across various devices and environments, demonstrating that it achieves 95.37% snooping camera detection accuracy and an average localization error of 17.23, under the significantly reduced activity space requirements. Our demo are available at this https URL.

Title: SpikeGS: Learning 3D Gaussian Fields from Continuous Spike Stream

• Authors: Jinze Yu, Xi Peng, Zhengda Lu, Laurent Kneip, Yiqun Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15176
• Pdf URL: https://arxiv.org/pdf/2409.15176
• Copy Paste: [[2409.15176]] SpikeGS: Learning 3D Gaussian Fields from Continuous Spike Stream(https://arxiv.org/abs/2409.15176)
• Keywords: robust
• Abstract: A spike camera is a specialized high-speed visual sensor that offers advantages such as high temporal resolution and high dynamic range compared to conventional frame cameras. These features provide the camera with significant advantages in many computer vision tasks. However, the tasks of 3D reconstruction and novel view synthesis based on spike cameras remain underdeveloped. Although there are existing methods for learning neural radiance fields from spike stream, they either lack robustness in extremely noisy, low-quality lighting conditions or suffer from high computational complexity due to the deep fully connected neural networks and ray marching rendering strategies used in neural radiance fields, making it difficult to recover fine texture details. In contrast, the latest advancements in 3DGS have achieved high-quality real-time rendering by optimizing the point cloud representation into Gaussian ellipsoids. Building on this, we introduce SpikeGS, the first method to learn 3D Gaussian fields solely from spike stream. We designed a differentiable spike stream rendering framework based on 3DGS, incorporating noise embedding and spiking neurons. By leveraging the multi-view consistency of 3DGS and the tile-based multi-threaded parallel rendering mechanism, we achieved high-quality real-time rendering results. Additionally, we introduced a spike rendering loss function that generalizes under varying illumination conditions. Our method can reconstruct view synthesis results with fine texture details from a continuous spike stream captured by a moving spike camera, while demonstrating high robustness in extremely noisy low-light scenarios. Experimental results on both real and synthetic datasets demonstrate that our method surpasses existing approaches in terms of rendering quality and speed. Our code will be available at this https URL.

Title: MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning

• Authors: Yue Han, Junwei Zhu, Yuxiang Feng, Xiaozhong Ji, Keke He, Xiangtai Li, zhucun xue, Yong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15179
• Pdf URL: https://arxiv.org/pdf/2409.15179
• Copy Paste: [[2409.15179]] MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning(https://arxiv.org/abs/2409.15179)
• Keywords: robust, diffusion
• Abstract: Current diffusion-based face animation methods generally adopt a ReferenceNet (a copy of U-Net) and a large amount of curated self-acquired data to learn appearance features, as robust appearance features are vital for ensuring temporal stability. However, when trained on public datasets, the results often exhibit a noticeable performance gap in image quality and temporal consistency. To address this issue, we meticulously examine the essential appearance features in the facial animation tasks, which include motion-agnostic (e.g., clothing, background) and motion-related (e.g., facial details) texture components, along with high-level discriminative identity features. Drawing from this analysis, we introduce a Motion-Identity Modulated Appearance Learning Module (MIA) that modulates CLIP features at both motion and identity levels. Additionally, to tackle the semantic/ color discontinuities between clips, we design an Inter-clip Affinity Learning Module (ICA) to model temporal relationships across clips. Our method achieves precise facial motion control (i.e., expressions and gaze), faithful identity preservation, and generates animation videos that maintain both intra/inter-clip temporal consistency. Moreover, it easily adapts to various modalities of driving sources. Extensive experiments demonstrate the superiority of our method.

Title: PALLM: Evaluating and Enhancing PALLiative Care Conversations with Large Language Models

• Authors: Zhiyuan Wang, Fangxu Yuan, Virginia LeBaron, Tabor Flickinger, Laura E. Barnes
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.15188
• Pdf URL: https://arxiv.org/pdf/2409.15188
• Copy Paste: [[2409.15188]] PALLM: Evaluating and Enhancing PALLiative Care Conversations with Large Language Models(https://arxiv.org/abs/2409.15188)
• Keywords: large language model
• Abstract: Effective patient-provider communication is crucial in clinical care, directly impacting patient outcomes and quality of life. Traditional evaluation methods, such as human ratings, patient feedback, and provider self-assessments, are often limited by high costs and scalability issues. Although existing natural language processing (NLP) techniques show promise, they struggle with the nuances of clinical communication and require sensitive clinical data for training, reducing their effectiveness in real-world applications. Emerging large language models (LLMs) offer a new approach to assessing complex communication metrics, with the potential to advance the field through integration into passive sensing and just-in-time intervention systems. This study explores LLMs as evaluators of palliative care communication quality, leveraging their linguistic, in-context learning, and reasoning capabilities. Specifically, using simulated scripts crafted and labeled by healthcare professionals, we test proprietary models (e.g., GPT-4) and fine-tune open-source LLMs (e.g., LLaMA2) with a synthetic dataset generated by GPT-4 to evaluate clinical conversations, to identify key metrics such as `understanding' and `empathy'. Our findings demonstrated LLMs' superior performance in evaluating clinical communication, providing actionable feedback with reasoning, and demonstrating the feasibility and practical viability of developing in-house LLMs. This research highlights LLMs' potential to enhance patient-provider interactions and lays the groundwork for downstream steps in developing LLM-empowered clinical health systems.

Title: Interpretability-Guided Test-Time Adversarial Defense

• Authors: Akshay Kulkarni, Tsui-Wei Weng
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15190
• Pdf URL: https://arxiv.org/pdf/2409.15190
• Copy Paste: [[2409.15190]] Interpretability-Guided Test-Time Adversarial Defense(https://arxiv.org/abs/2409.15190)
• Keywords: defense, attack, robust, interpretability
• Abstract: We propose a novel and low-cost test-time adversarial defense by devising interpretability-guided neuron importance ranking methods to identify neurons important to the output classes. Our method is a training-free approach that can significantly improve the robustness-accuracy tradeoff while incurring minimal computational overhead. While being among the most efficient test-time defenses (4x faster), our method is also robust to a wide range of black-box, white-box, and adaptive attacks that break previous test-time defenses. We demonstrate the efficacy of our method for CIFAR10, CIFAR100, and ImageNet-1k on the standard RobustBench benchmark (with average gains of 2.6%, 4.9%, and 2.8% respectively). We also show improvements (average 1.5%) over the state-of-the-art test-time defenses even under strong adaptive attacks.

Title: Enabling Tensor Decomposition for Time-Series Classification via A Simple Pseudo-Laplacian Contrast

• Authors: Man Li, Ziyue Li, Lijun Sun, Fugee Tsung
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15200
• Pdf URL: https://arxiv.org/pdf/2409.15200
• Copy Paste: [[2409.15200]] Enabling Tensor Decomposition for Time-Series Classification via A Simple Pseudo-Laplacian Contrast(https://arxiv.org/abs/2409.15200)
• Keywords: extraction
• Abstract: Tensor decomposition has emerged as a prominent technique to learn low-dimensional representation under the supervision of reconstruction error, primarily benefiting data inference tasks like completion and imputation, but not classification task. We argue that the non-uniqueness and rotation invariance of tensor decomposition allow us to identify the directions with largest class-variability and simple graph Laplacian can effectively achieve this objective. Therefore we propose a novel Pseudo Laplacian Contrast (PLC) tensor decomposition framework, which integrates the data augmentation and cross-view Laplacian to enable the extraction of class-aware representations while effectively capturing the intrinsic low-rank structure within reconstruction constraint. An unsupervised alternative optimization algorithm is further developed to iteratively estimate the pseudo graph and minimize the loss using Alternating Least Square (ALS). Extensive experimental results on various datasets demonstrate the effectiveness of our approach.

Title: ASTE Transformer Modelling Dependencies in Aspect-Sentiment Triplet Extraction

• Authors: Iwo Naglik, Mateusz Lango
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15202
• Pdf URL: https://arxiv.org/pdf/2409.15202
• Copy Paste: [[2409.15202]] ASTE Transformer Modelling Dependencies in Aspect-Sentiment Triplet Extraction(https://arxiv.org/abs/2409.15202)
• Keywords: extraction, transformer
• Abstract: Aspect-Sentiment Triplet Extraction (ASTE) is a recently proposed task of aspect-based sentiment analysis that consists in extracting (aspect phrase, opinion phrase, sentiment polarity) triples from a given sentence. Recent state-of-the-art methods approach this task by first extracting all possible text spans from a given text, then filtering the potential aspect and opinion phrases with a classifier, and finally considering all their pairs with another classifier that additionally assigns sentiment polarity to them. Although several variations of the above scheme have been proposed, the common feature is that the final result is constructed by a sequence of independent classifier decisions. This hinders the exploitation of dependencies between extracted phrases and prevents the use of knowledge about the interrelationships between classifier predictions to improve performance. In this paper, we propose a new ASTE approach consisting of three transformer-inspired layers, which enables the modelling of dependencies both between phrases and between the final classifier decisions. Experimental results show that the method achieves higher performance in terms of F1 measure than other methods studied on popular benchmarks. In addition, we show that a simple pre-training technique further improves the performance of the model.

Title: HydroVision: LiDAR-Guided Hydrometric Prediction with Vision Transformers and Hybrid Graph Learning

• Authors: Naghmeh Shafiee Roudbari, Ursula Eicker, Charalambos Poullis, Zachary Patterson
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15213
• Pdf URL: https://arxiv.org/pdf/2409.15213
• Copy Paste: [[2409.15213]] HydroVision: LiDAR-Guided Hydrometric Prediction with Vision Transformers and Hybrid Graph Learning(https://arxiv.org/abs/2409.15213)
• Keywords: protect, transformer
• Abstract: Hydrometric forecasting is crucial for managing water resources, flood prediction, and environmental protection. Water stations are interconnected, and this connectivity influences the measurements at other stations. However, the dynamic and implicit nature of water flow paths makes it challenging to extract a priori knowledge of the connectivity structure. We hypothesize that terrain elevation significantly affects flow and connectivity. To incorporate this, we use LiDAR terrain elevation data encoded through a Vision Transformer (ViT). The ViT, which has demonstrated excellent performance in image classification by directly applying transformers to sequences of image patches, efficiently captures spatial features of terrain elevation. To account for both spatial and temporal features, we employ GRU blocks enhanced with graph convolution, a method widely used in the literature. We propose a hybrid graph learning structure that combines static and dynamic graph learning. A static graph, derived from transformer-encoded LiDAR data, captures terrain elevation relationships, while a dynamic graph adapts to temporal changes, improving the overall graph representation. We apply graph convolution in two layers through these static and dynamic graphs. Our method makes daily predictions up to 12 days ahead. Empirical results from multiple water stations in Quebec demonstrate that our method significantly reduces prediction error by an average of 10\% across all days, with greater improvements for longer forecasting horizons.

Title: FLeNS: Federated Learning with Enhanced Nesterov-Newton Sketch

• Authors: Sunny Gupta, Mohit, Pankhi Kashyap, Pranav Jeevan, Amit Sethi
• Subjects: cs.LG, cs.CV, cs.DC, math.OC
• Abstract URL: https://arxiv.org/abs/2409.15216
• Pdf URL: https://arxiv.org/pdf/2409.15216
• Copy Paste: [[2409.15216]] FLeNS: Federated Learning with Enhanced Nesterov-Newton Sketch(https://arxiv.org/abs/2409.15216)
• Keywords: privacy, federate
• Abstract: Federated learning faces a critical challenge in balancing communication efficiency with rapid convergence, especially for second-order methods. While Newton-type algorithms achieve linear convergence in communication rounds, transmitting full Hessian matrices is often impractical due to quadratic complexity. We introduce Federated Learning with Enhanced Nesterov-Newton Sketch (FLeNS), a novel method that harnesses both the acceleration capabilities of Nesterov's method and the dimensionality reduction benefits of Hessian sketching. FLeNS approximates the centralized Newton's method without relying on the exact Hessian, significantly reducing communication overhead. By combining Nesterov's acceleration with adaptive Hessian sketching, FLeNS preserves crucial second-order information while preserving the rapid convergence characteristics. Our theoretical analysis, grounded in statistical learning, demonstrates that FLeNS achieves super-linear convergence rates in communication rounds - a notable advancement in federated optimization. We provide rigorous convergence guarantees and characterize tradeoffs between acceleration, sketch size, and convergence speed. Extensive empirical evaluation validates our theoretical findings, showcasing FLeNS's state-of-the-art performance with reduced communication requirements, particularly in privacy-sensitive and edge-computing scenarios. The code is available at this https URL

Title: MotifDisco: Motif Causal Discovery For Time Series Motifs

• Authors: Josephine Lamp, Mark Derdzinski, Christopher Hannemann, Sam Hatfield, Joost van der Linden
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15219
• Pdf URL: https://arxiv.org/pdf/2409.15219
• Copy Paste: [[2409.15219]] MotifDisco: Motif Causal Discovery For Time Series Motifs(https://arxiv.org/abs/2409.15219)
• Keywords: generative
• Abstract: Many time series, particularly health data streams, can be best understood as a sequence of phenomenon or events, which we call motifs. A time series motif is a short trace segment which may implicitly capture an underlying phenomenon within the time series. Specifically, we focus on glucose traces collected from continuous glucose monitors (CGMs), which inherently contain motifs representing underlying human behaviors such as eating and exercise. The ability to identify and quantify causal relationships amongst motifs can provide a mechanism to better understand and represent these patterns, useful for improving deep learning and generative models and for advanced technology development (e.g., personalized coaching and artificial insulin delivery systems). However, no previous work has developed causal discovery methods for time series motifs. Therefore, in this paper we develop MotifDisco (motif disco-very of causality), a novel causal discovery framework to learn causal relations amongst motifs from time series traces. We formalize a notion of Motif Causality (MC), inspired from Granger Causality and Transfer Entropy, and develop a Graph Neural Network-based framework that learns causality between motifs by solving an unsupervised link prediction problem. We also integrate MC with three model use cases of forecasting, anomaly detection and clustering, to showcase the use of MC as a building block for other downstream tasks. Finally, we evaluate our framework and find that Motif Causality provides a significant performance improvement in all use cases.

Title: Enhancing Pedestrian Trajectory Prediction with Crowd Trip Information

• Authors: Rei Tamaru, Pei Li, Bin Ran
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.15224
• Pdf URL: https://arxiv.org/pdf/2409.15224
• Copy Paste: [[2409.15224]] Enhancing Pedestrian Trajectory Prediction with Crowd Trip Information(https://arxiv.org/abs/2409.15224)
• Keywords: transformer
• Abstract: Pedestrian trajectory prediction is essential for various applications in active traffic management, urban planning, traffic control, crowd management, and autonomous driving, aiming to enhance traffic safety and efficiency. Accurately predicting pedestrian trajectories requires a deep understanding of individual behaviors, social interactions, and road environments. Existing studies have developed various models to capture the influence of social interactions and road conditions on pedestrian trajectories. However, these approaches are limited by the lack of a comprehensive view of social interactions and road environments. To address these limitations and enhance the accuracy of pedestrian trajectory prediction, we propose a novel approach incorporating trip information as a new modality into pedestrian trajectory models. We propose RNTransformer, a generic model that utilizes crowd trip information to capture global information on social interactions. We incorporated RNTransformer with various socially aware local pedestrian trajectory prediction models to demonstrate its performance. Specifically, by leveraging a pre-trained RNTransformer when training different pedestrian trajectory prediction models, we observed improvements in performance metrics: a 1.3/2.2% enhancement in ADE/FDE on Social-LSTM, a 6.5/28.4% improvement on Social-STGCNN, and an 8.6/4.3% improvement on S-Implicit. Evaluation results demonstrate that RNTransformer significantly enhances the accuracy of various pedestrian trajectory prediction models across multiple datasets. Further investigation reveals that the RNTransformer effectively guides local models to more accurate directions due to the consideration of global information. By exploring crowd behavior within the road network, our approach shows great promise in improving pedestrian safety through accurate trajectory predictions.

Title: Semantic Inference-Based Deep Learning and Modeling for Earth Observation: Cognitive Semantic Augmentation Satellite Networks

• Authors: Hong-fu Chou, Vu Nguyen Ha, Prabhu Thiruvasagam, Thanh-Dung Le, Geoffrey Eappen, Ti Ti Nguyen, Luis M. Garces-Socarras, Jorge L. Gonzalez-Rios, Juan Carlos Merlano-Duncan, Symeon Chatzinotas
• Subjects: cs.LG, cs.CV, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.15246
• Pdf URL: https://arxiv.org/pdf/2409.15246
• Copy Paste: [[2409.15246]] Semantic Inference-Based Deep Learning and Modeling for Earth Observation: Cognitive Semantic Augmentation Satellite Networks(https://arxiv.org/abs/2409.15246)
• Keywords: large language model
• Abstract: Earth Observation (EO) systems play a crucial role in achieving Sustainable Development Goals by collecting and analyzing vital global data through satellite networks. These systems are essential for tasks like mapping, disaster monitoring, and resource management, but they face challenges in processing and transmitting large volumes of EO data, especially in specialized fields such as agriculture and real-time disaster response. Domain-adapted Large Language Models (LLMs) provide a promising solution by facilitating data fusion between extensive EO data and semantic EO data. By improving integration and interpretation of diverse datasets, LLMs address the challenges of processing specialized information in agriculture and disaster response applications. This fusion enhances the accuracy and relevance of transmitted data. This paper presents a framework for semantic communication in EO satellite networks, aimed at improving data transmission efficiency and overall system performance through cognitive processing techniques. The proposed system employs Discrete-Task-Oriented Source-Channel Coding (DT-JSCC) and Semantic Data Augmentation (SA) to focus on relevant information while minimizing communication overhead. By integrating cognitive semantic processing and inter-satellite links, the framework enhances the analysis and transmission of multispectral satellite imagery, improving object detection, pattern recognition, and real-time decision-making. The introduction of Cognitive Semantic Augmentation (CSA) allows satellites to process and transmit semantic information, boosting adaptability to changing environments and application needs. This end-to-end architecture is tailored for next-generation satellite networks, such as those supporting 6G, and demonstrates significant improvements in efficiency and accuracy.

Title: ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models

• Authors: Sombit Dey, Jan-Nico Zaech, Nikolay Nikolov, Luc Van Gool, Danda Pani Paudel
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.15250
• Pdf URL: https://arxiv.org/pdf/2409.15250
• Copy Paste: [[2409.15250]] ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models(https://arxiv.org/abs/2409.15250)
• Keywords: robust, large language model
• Abstract: Recent progress in large language models and access to large-scale robotic datasets has sparked a paradigm shift in robotics models transforming them into generalists able to adapt to various tasks, scenes, and robot modalities. A large step for the community are open Vision Language Action models which showcase strong performance in a wide variety of tasks. In this work, we study the visual generalization capabilities of three existing robotic foundation models, and propose a corresponding evaluation framework. Our study shows that the existing models do not exhibit robustness to visual out-of-domain scenarios. This is potentially caused by limited variations in the training data and/or catastrophic forgetting, leading to domain limitations in the vision foundation models. We further explore OpenVLA, which uses two pre-trained vision foundation models and is, therefore, expected to generalize to out-of-domain experiments. However, we showcase catastrophic forgetting by DINO-v2 in OpenVLA through its failure to fulfill the task of depth regression. To overcome the aforementioned issue of visual catastrophic forgetting, we propose a gradual backbone reversal approach founded on model merging. This enables OpenVLA which requires the adaptation of the visual backbones during initial training -- to regain its visual generalization ability. Regaining this capability enables our ReVLA model to improve over OpenVLA by a factor of 77% and 66% for grasping and lifting in visual OOD tasks .

Title: Archon: An Architecture Search Framework for Inference-Time Techniques

• Authors: Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, E. Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, Azalia Mirhoseini
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.15254
• Pdf URL: https://arxiv.org/pdf/2409.15254
• Copy Paste: [[2409.15254]] Archon: An Architecture Search Framework for Inference-Time Techniques(https://arxiv.org/abs/2409.15254)
• Keywords: large language model
• Abstract: Inference-time techniques are emerging as highly effective tools to increase large language model (LLM) capabilities. However, there is still limited understanding of the best practices for developing systems that combine inference-time techniques with one or more LLMs, with challenges including: (1) effectively allocating inference compute budget, (2) understanding the interactions between different combinations of inference-time techniques and their impact on downstream performance, and 3) efficiently searching over the large space of model choices, inference-time techniques, and their compositions. To address these challenges, we introduce Archon, an automated framework for designing inference-time architectures. Archon defines an extensible design space, encompassing methods such as generation ensembling, multi-sampling, ranking, fusion, critiquing, verification, and unit testing. It then transforms the problem of selecting and combining LLMs and inference-time techniques into a hyperparameter optimization objective. To optimize this objective, we introduce automated Inference-Time Architecture Search (ITAS) algorithms. Given target benchmark(s), an inference compute budget, and available LLMs, ITAS outputs optimized architectures. We evaluate Archon architectures across a wide range of instruction-following and reasoning benchmarks, including MT-Bench, Arena-Hard-Auto, AlpacaEval 2.0, MixEval, MixEval Hard, MATH, and CodeContests. We show that automatically designed inference-time architectures by Archon outperform strong models such as GPT-4o and Claude 3.5 Sonnet on these benchmarks, achieving an average increase of 14.1 and 10.3 percentage points with all-source models and open-source models, respectively. We make our code and datasets available publicly on Github: this https URL.

Title: Behavioral Bias of Vision-Language Models: A Behavioral Finance View

• Authors: Yuhang Xiao, Yudi Lin, Ming-Chang Chiu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15256
• Pdf URL: https://arxiv.org/pdf/2409.15256
• Copy Paste: [[2409.15256]] Behavioral Bias of Vision-Language Models: A Behavioral Finance View(https://arxiv.org/abs/2409.15256)
• Keywords: large language model
• Abstract: Large Vision-Language Models (LVLMs) evolve rapidly as Large Language Models (LLMs) was equipped with vision modules to create more human-like models. However, we should carefully evaluate their applications in different domains, as they may possess undesired biases. Our work studies the potential behavioral biases of LVLMs from a behavioral finance perspective, an interdisciplinary subject that jointly considers finance and psychology. We propose an end-to-end framework, from data collection to new evaluation metrics, to assess LVLMs' reasoning capabilities and the dynamic behaviors manifested in two established human financial behavioral biases: recency bias and authority bias. Our evaluations find that recent open-source LVLMs such as LLaVA-NeXT, MobileVLM-V2, Mini-Gemini, MiniCPM-Llama3-V 2.5 and Phi-3-vision-128k suffer significantly from these two biases, while the proprietary model GPT-4o is negligibly impacted. Our observations highlight directions in which open-source models can improve. The code is available at this https URL.

Title: S$^2$AG-Vid: Enhancing Multi-Motion Alignment in Video Diffusion Models via Spatial and Syntactic Attention-Based Guidance

• Authors: Yuanhang Li, Qi Mao, Lan Chen, Zhen Fang, Lei Tian, Xinyan Xiao, Libiao Jin, Hua Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15259
• Pdf URL: https://arxiv.org/pdf/2409.15259
• Copy Paste: [[2409.15259]] S$^2$AG-Vid: Enhancing Multi-Motion Alignment in Video Diffusion Models via Spatial and Syntactic Attention-Based Guidance(https://arxiv.org/abs/2409.15259)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-video (T2V) generation using diffusion models have garnered significant attention. However, existing T2V models primarily focus on simple scenes featuring a single object performing a single motion. Challenges arise in scenarios involving multiple objects with distinct motions, often leading to incorrect video-text alignment between subjects and their corresponding motions. To address this challenge, we propose \textbf{S$^2$AG-Vid}, a training-free inference-stage optimization method that improves the alignment of multiple objects with their corresponding motions in T2V models. S$^2$AG-Vid initially applies a spatial position-based, cross-attention (CA) constraint in the early stages of the denoising process, facilitating multiple nouns distinctly attending to the correct subject regions. To enhance the motion-subject binding, we implement a syntax-guided contrastive constraint in the subsequent denoising phase, aimed at improving the correlations between the CA maps of verbs and their corresponding nouns.Both qualitative and quantitative evaluations demonstrate that the proposed framework significantly outperforms baseline approaches, producing higher-quality videos with improved subject-motion consistency.

Title: UDA-Bench: Revisiting Common Assumptions in Unsupervised Domain Adaptation Using a Standardized Framework

• Authors: Tarun Kalluri, Sreyas Ravichandran, Manmohan Chandraker
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15264
• Pdf URL: https://arxiv.org/pdf/2409.15264
• Copy Paste: [[2409.15264]] UDA-Bench: Revisiting Common Assumptions in Unsupervised Domain Adaptation Using a Standardized Framework(https://arxiv.org/abs/2409.15264)
• Keywords: fair
• Abstract: In this work, we take a deeper look into the diverse factors that influence the efficacy of modern unsupervised domain adaptation (UDA) methods using a large-scale, controlled empirical study. To facilitate our analysis, we first develop UDA-Bench, a novel PyTorch framework that standardizes training and evaluation for domain adaptation enabling fair comparisons across several UDA methods. Using UDA-Bench, our comprehensive empirical study into the impact of backbone architectures, unlabeled data quantity, and pre-training datasets reveals that: (i) the benefits of adaptation methods diminish with advanced backbones, (ii) current methods underutilize unlabeled data, and (iii) pre-training data significantly affects downstream adaptation in both supervised and self-supervised settings. In the context of unsupervised adaptation, these observations uncover several novel and surprising properties, while scientifically validating several others that were often considered empirical heuristics or practitioner intuitions in the absence of a standardized training and evaluation framework. The UDA-Bench framework and trained models are publicly available at this https URL.

Title: Peer-to-Peer Learning Dynamics of Wide Neural Networks

• Authors: Shreyas Chaudhari, Srinivasa Pranav, Emile Anand, José M. F. Moura
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2409.15267
• Pdf URL: https://arxiv.org/pdf/2409.15267
• Copy Paste: [[2409.15267]] Peer-to-Peer Learning Dynamics of Wide Neural Networks(https://arxiv.org/abs/2409.15267)
• Keywords: privacy
• Abstract: Peer-to-peer learning is an increasingly popular framework that enables beyond-5G distributed edge devices to collaboratively train deep neural networks in a privacy-preserving manner without the aid of a central server. Neural network training algorithms for emerging environments, e.g., smart cities, have many design considerations that are difficult to tune in deployment settings -- such as neural network architectures and hyperparameters. This presents a critical need for characterizing the training dynamics of distributed optimization algorithms used to train highly nonconvex neural networks in peer-to-peer learning environments. In this work, we provide an explicit, non-asymptotic characterization of the learning dynamics of wide neural networks trained using popular distributed gradient descent (DGD) algorithms. Our results leverage both recent advancements in neural tangent kernel (NTK) theory and extensive previous work on distributed learning and consensus. We validate our analytical results by accurately predicting the parameter and error dynamics of wide neural networks trained for classification tasks.

Title: OmniBench: Towards The Future of Universal Omni-Language Models

• Authors: Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, Chenghua Lin
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15272
• Pdf URL: https://arxiv.org/pdf/2409.15272
• Copy Paste: [[2409.15272]] OmniBench: Towards The Future of Universal Omni-Language Models(https://arxiv.org/abs/2409.15272)
• Keywords: robust, large language model
• Abstract: Recent advancements in multimodal large language models (MLLMs) have aimed to integrate and interpret data across diverse modalities. However, the capacity of these models to concurrently process and reason about multiple modalities remains inadequately explored, partly due to the lack of comprehensive modality-wise benchmarks. We introduce OmniBench, a novel benchmark designed to rigorously evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define models capable of such tri-modal processing as omni-language models (OLMs). OmniBench is distinguished by high-quality human annotations, ensuring that accurate responses require integrated understanding and reasoning across all three modalities. Our main findings reveal that: i) open-source OLMs exhibit critical limitations in instruction-following and reasoning capabilities within tri-modal contexts; and ii) the baseline models perform poorly (below 50% accuracy) even when provided with alternative textual representations of images and audio. These results suggest that the ability to construct a consistent context from text, image, and audio is often overlooked in existing MLLM training paradigms. We advocate for future research to focus on developing more robust tri-modal integration techniques and training strategies to enhance OLM performance across diverse modalities. The codes and live leaderboard could be found at this https URL.

Title: MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors

• Authors: Yehonathan Litman, Or Patashnik, Kangle Deng, Aviral Agrawal, Rushikesh Zawar, Fernando De la Torre, Shubham Tulsiani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15273
• Pdf URL: https://arxiv.org/pdf/2409.15273
• Copy Paste: [[2409.15273]] MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors(https://arxiv.org/abs/2409.15273)
• Keywords: diffusion
• Abstract: Recent works in inverse rendering have shown promise in using multi-view images of an object to recover shape, albedo, and materials. However, the recovered components often fail to render accurately under new lighting conditions due to the intrinsic challenge of disentangling albedo and material properties from input images. To address this challenge, we introduce MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that incorporates a 2D prior on texture and material properties. We present StableMaterial, a 2D diffusion model prior that refines multi-lit data to estimate the most likely albedo and material from given input appearances. This model is trained on albedo, material, and relit image data derived from a curated dataset of approximately ~12K artist-designed synthetic Blender objects called BlenderVault. we incorporate this diffusion prior with an inverse rendering framework where we use score distillation sampling (SDS) to guide the optimization of the albedo and materials, improving relighting performance in comparison with previous work. We validate MaterialFusion's relighting performance on 4 datasets of synthetic and real objects under diverse illumination conditions, showing our diffusion-aided approach significantly improves the appearance of reconstructed objects under novel lighting conditions. We intend to publicly release our BlenderVault dataset to support further research in this field.

Title: A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?

• Authors: Yunfei Xie, Juncheng Wu, Haoqin Tu, Siwei Yang, Bingchen Zhao, Yongshuo Zong, Qiao Jin, Cihang Xie, Yuyin Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.15277
• Pdf URL: https://arxiv.org/pdf/2409.15277
• Copy Paste: [[2409.15277]] A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?(https://arxiv.org/abs/2409.15277)
• Keywords: large language model
• Abstract: Large language models (LLMs) have exhibited remarkable capabilities across various domains and tasks, pushing the boundaries of our knowledge in learning and cognition. The latest model, OpenAI's o1, stands out as the first LLM with an internalized chain-of-thought technique using reinforcement learning strategies. While it has demonstrated surprisingly strong capabilities on various general language tasks, its performance in specialized fields such as medicine remains unknown. To this end, this report provides a comprehensive exploration of o1 on different medical scenarios, examining 3 key aspects: understanding, reasoning, and multilinguality. Specifically, our evaluation encompasses 6 tasks using data from 37 medical datasets, including two newly constructed and more challenging question-answering (QA) tasks based on professional medical quizzes from the New England Journal of Medicine (NEJM) and The Lancet. These datasets offer greater clinical relevance compared to standard medical QA benchmarks such as MedQA, translating more effectively into real-world clinical utility. Our analysis of o1 suggests that the enhanced reasoning ability of LLMs may (significantly) benefit their capability to understand various medical instructions and reason through complex clinical scenarios. Notably, o1 surpasses the previous GPT-4 in accuracy by an average of 6.2% and 6.6% across 19 datasets and two newly created complex QA scenarios. But meanwhile, we identify several weaknesses in both the model capability and the existing evaluation protocols, including hallucination, inconsistent multilingual ability, and discrepant metrics for evaluation. We release our raw data and model outputs at this https URL for future research.

Title: PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions

• Authors: Weifeng Lin, Xinyu Wei, Renrui Zhang, Le Zhuo, Shitian Zhao, Siyuan Huang, Junlin Xie, Yu Qiao, Peng Gao, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.15278
• Pdf URL: https://arxiv.org/pdf/2409.15278
• Copy Paste: [[2409.15278]] PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions(https://arxiv.org/abs/2409.15278)
• Keywords: diffusion, transformer, generative
• Abstract: This paper presents a versatile image-to-image visual assistant, PixWizard, designed for image generation, manipulation, and translation based on free-from language instructions. To this end, we tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction templates in natural language, we comprehensively include a large set of diverse vision tasks such as text-to-image generation, image restoration, image grounding, dense image prediction, image editing, controllable generation, inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with a flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes. The model also incorporates structure-aware and semantic-aware guidance to facilitate effective fusion of information from the input image. Our experiments demonstrate that PixWizard not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits promising generalization capabilities with unseen tasks and human instructions. The code and related resources are available at this https URL