2024-04-16

Title: Optimal path for Biomedical Text Summarization Using Pointer GPT

• Authors: Hyunkyung Han, Jaesik Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08654
• Pdf URL: https://arxiv.org/pdf/2404.08654
• Copy Paste: [[2404.08654]] Optimal path for Biomedical Text Summarization Using Pointer GPT(https://arxiv.org/abs/2404.08654)
• Keywords: transformer
• Abstract: Biomedical text summarization is a critical tool that enables clinicians to effectively ascertain patient status. Traditionally, text summarization has been accomplished with transformer models, which are capable of compressing long documents into brief summaries. However, transformer models are known to be among the most challenging natural language processing (NLP) tasks. Specifically, GPT models have a tendency to generate factual errors, lack context, and oversimplify words. To address these limitations, we replaced the attention mechanism in the GPT model with a pointer network. This modification was designed to preserve the core values of the original text during the summarization process. The effectiveness of the Pointer-GPT model was evaluated using the ROUGE score. The results demonstrated that Pointer-GPT outperformed the original GPT model. These findings suggest that pointer networks can be a valuable addition to EMR systems and can provide clinicians with more accurate and informative summaries of patient medical records. This research has the potential to usher in a new paradigm in EMR systems and to revolutionize the way that clinicians interact with patient medical records.

Title: Transformer-based Joint Modelling for Automatic Essay Scoring and Off-Topic Detection

• Authors: Sourya Dipta Das, Yash Vadi, Kuldeep Yadav
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08655
• Pdf URL: https://arxiv.org/pdf/2404.08655
• Copy Paste: [[2404.08655]] Transformer-based Joint Modelling for Automatic Essay Scoring and Off-Topic Detection(https://arxiv.org/abs/2404.08655)
• Keywords: robust, transformer
• Abstract: Automated Essay Scoring (AES) systems are widely popular in the market as they constitute a cost-effective and time-effective option for grading systems. Nevertheless, many studies have demonstrated that the AES system fails to assign lower grades to irrelevant responses. Thus, detecting the off-topic response in automated essay scoring is crucial in practical tasks where candidates write unrelated text responses to the given task in the question. In this paper, we are proposing an unsupervised technique that jointly scores essays and detects off-topic essays. The proposed Automated Open Essay Scoring (AOES) model uses a novel topic regularization module (TRM), which can be attached on top of a transformer model, and is trained using a proposed hybrid loss function. After training, the AOES model is further used to calculate the Mahalanobis distance score for off-topic essay detection. Our proposed method outperforms the baseline we created and earlier conventional methods on two essay-scoring datasets in off-topic detection as well as on-topic scoring. Experimental evaluation results on different adversarial strategies also show how the suggested method is robust for detecting possible human-level perturbations.

Title: Linear Cross-document Event Coreference Resolution with X-AMR

• Authors: Shafiuddin Rehan Ahmed, George Arthur Baker, Evi Judge, Michael Regan, Kristin Wright-Bettner, Martha Palmer, James H. Martin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08656
• Pdf URL: https://arxiv.org/pdf/2404.08656
• Copy Paste: [[2404.08656]] Linear Cross-document Event Coreference Resolution with X-AMR(https://arxiv.org/abs/2404.08656)
• Keywords: fair, large language model
• Abstract: Event Coreference Resolution (ECR) as a pairwise mention classification task is expensive both for automated systems and manual annotations. The task's quadratic difficulty is exacerbated when using Large Language Models (LLMs), making prompt engineering for ECR prohibitively costly. In this work, we propose a graphical representation of events, X-AMR, anchored around individual mentions using a \textbf{cross}-document version of \textbf{A}bstract \textbf{M}eaning \textbf{R}epresentation. We then linearize the ECR with a novel multi-hop coreference algorithm over the event graphs. The event graphs simplify ECR, making it a) LLM cost-effective, b) compositional and interpretable, and c) easily annotated. For a fair assessment, we first enrich an existing ECR benchmark dataset with these event graphs using an annotator-friendly tool we introduce. Then, we employ GPT-4, the newest LLM by OpenAI, for these annotations. Finally, using the ECR algorithm, we assess GPT-4 against humans and analyze its limitations. Through this research, we aim to advance the state-of-the-art for efficient ECR and shed light on the potential shortcomings of current LLMs at this task. Code and annotations: \url{https://github.com/ahmeshaf/gpt_coref}

Title: Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferences

• Authors: Jesse Atuhurra, Hidetaka Kamigaito
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08666
• Pdf URL: https://arxiv.org/pdf/2404.08666
• Copy Paste: [[2404.08666]] Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferences(https://arxiv.org/abs/2404.08666)
• Keywords: transformer, large language model
• Abstract: Natural language processing (NLP) has grown significantly since the advent of the Transformer architecture. Transformers have given birth to pre-trained large language models (PLMs). There has been tremendous improvement in the performance of NLP systems across several tasks. NLP systems are on par or, in some cases, better than humans at accomplishing specific tasks. However, it remains the norm that \emph{better quality datasets at the time of pretraining enable PLMs to achieve better performance, regardless of the task.} The need to have quality datasets has prompted NLP researchers to continue creating new datasets to satisfy particular needs. For example, the two top NLP conferences, ACL and EMNLP, accepted ninety-two papers in 2022, introducing new datasets. This work aims to uncover the trends and insights mined within these datasets. Moreover, we provide valuable suggestions to researchers interested in curating datasets in the future.

Title: Effects of Different Prompts on the Quality of GPT-4 Responses to Dementia Care Questions

• Authors: Zhuochun Li, Bo Xie, Robin Hilsabeck, Alyssa Aguirre, Ning Zou, Zhimeng Luo, Daqing He
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.08674
• Pdf URL: https://arxiv.org/pdf/2404.08674
• Copy Paste: [[2404.08674]] Effects of Different Prompts on the Quality of GPT-4 Responses to Dementia Care Questions(https://arxiv.org/abs/2404.08674)
• Keywords: large language model
• Abstract: Evidence suggests that different prompts lead large language models (LLMs) to generate responses with varying quality. Yet, little is known about prompts' effects on response quality in healthcare domains. In this exploratory study, we address this gap, focusing on a specific healthcare domain: dementia caregiving. We first developed an innovative prompt template with three components: (1) system prompts (SPs) featuring 4 different roles; (2) an initialization prompt; and (3) task prompts (TPs) specifying different levels of details, totaling 12 prompt combinations. Next, we selected 3 social media posts containing complicated, real-world questions about dementia caregivers' challenges in 3 areas: memory loss and confusion, aggression, and driving. We then entered these posts into GPT-4, with our 12 prompts, to generate 12 responses per post, totaling 36 responses. We compared the word count of the 36 responses to explore potential differences in response length. Two experienced dementia care clinicians on our team assessed the response quality using a rating scale with 5 quality indicators: factual, interpretation, application, synthesis, and comprehensiveness (scoring range: 0-5; higher scores indicate higher quality).

Title: ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming

• Authors: Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, Bo Li
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08676
• Pdf URL: https://arxiv.org/pdf/2404.08676
• Copy Paste: [[2404.08676]] ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming(https://arxiv.org/abs/2404.08676)
• Keywords: protect, large language model
• Abstract: When building Large Language Models (LLMs), it is paramount to bear safety in mind and protect them with guardrails. Indeed, LLMs should never generate content promoting or normalizing harmful, illegal, or unethical behavior that may contribute to harm to individuals or society. This principle applies to both normal and adversarial use. In response, we introduce ALERT, a large-scale benchmark to assess safety based on a novel fine-grained risk taxonomy. It is designed to evaluate the safety of LLMs through red teaming methodologies and consists of more than 45k instructions categorized using our novel taxonomy. By subjecting LLMs to adversarial testing scenarios, ALERT aims to identify vulnerabilities, inform improvements, and enhance the overall safety of the language models. Furthermore, the fine-grained taxonomy enables researchers to perform an in-depth evaluation that also helps one to assess the alignment with various policies. In our experiments, we extensively evaluate 10 popular open- and closed-source LLMs and demonstrate that many of them still struggle to attain reasonable levels of safety.

Title: Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector

• Authors: Andi Zhang, Tim Z. Xiao, Weiyang Liu, Robert Bamler, Damon Wischik
• Subjects: cs.CL, cs.AI, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.08679
• Pdf URL: https://arxiv.org/pdf/2404.08679
• Copy Paste: [[2404.08679]] Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector(https://arxiv.org/abs/2404.08679)
• Keywords: large language model
• Abstract: We revisit the likelihood ratio between a pretrained large language model (LLM) and its finetuned variant as a criterion for out-of-distribution (OOD) detection. The intuition behind such a criterion is that, the pretrained LLM has the prior knowledge about OOD data due to its large amount of training data, and once finetuned with the in-distribution data, the LLM has sufficient knowledge to distinguish their difference. Leveraging the power of LLMs, we show that, for the first time, the likelihood ratio can serve as an effective OOD detector. Moreover, we apply the proposed LLM-based likelihood ratio to detect OOD questions in question-answering (QA) systems, which can be used to improve the performance of specialized LLMs for general questions. Given that likelihood can be easily obtained by the loss functions within contemporary neural network frameworks, it is straightforward to implement this approach in practice. Since both the pretrained LLMs and its various finetuned models are available, our proposed criterion can be effortlessly incorporated for OOD detection without the need for further training. We conduct comprehensive evaluation across on multiple settings, including far OOD, near OOD, spam detection, and QA scenarios, to demonstrate the effectiveness of the method.

Title: Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning

• Authors: Teo Susnjak, Peter Hwang, Napoleon H. Reyes, Andre L. C. Barczak, Timothy R. McIntosh, Surangika Ranathunga
• Subjects: cs.CL, cs.DL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.08680
• Pdf URL: https://arxiv.org/pdf/2404.08680
• Copy Paste: [[2404.08680]] Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning(https://arxiv.org/abs/2404.08680)
• Keywords: large language model
• Abstract: This research pioneers the use of fine-tuned Large Language Models (LLMs) to automate Systematic Literature Reviews (SLRs), presenting a significant and novel contribution in integrating AI to enhance academic research methodologies. Our study employed the latest fine-tuning methodologies together with open-sourced LLMs, and demonstrated a practical and efficient approach to automating the final execution stages of an SLR process that involves knowledge synthesis. The results maintained high fidelity in factual accuracy in LLM responses, and were validated through the replication of an existing PRISMA-conforming SLR. Our research proposed solutions for mitigating LLM hallucination and proposed mechanisms for tracking LLM responses to their sources of information, thus demonstrating how this approach can meet the rigorous demands of scholarly research. The findings ultimately confirmed the potential of fine-tuned LLMs in streamlining various labor-intensive processes of conducting literature reviews. Given the potential of this approach and its applicability across all research domains, this foundational study also advocated for updating PRISMA reporting guidelines to incorporate AI-driven processes, ensuring methodological transparency and reliability in future SLRs. This study broadens the appeal of AI-enhanced tools across various academic and research fields, setting a new standard for conducting comprehensive and accurate literature reviews with more efficiency in the face of ever-increasing volumes of academic studies.

Title: EFSA: Towards Event-Level Financial Sentiment Analysis

• Authors: Tianyu Chen, Yiming Zhang, Guoxin Yu, Dapeng Zhang, Li Zeng, Qing He, Xiang Ao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08681
• Pdf URL: https://arxiv.org/pdf/2404.08681
• Copy Paste: [[2404.08681]] EFSA: Towards Event-Level Financial Sentiment Analysis(https://arxiv.org/abs/2404.08681)
• Keywords: extraction
• Abstract: In this paper, we extend financial sentiment analysis~(FSA) to event-level since events usually serve as the subject of the sentiment in financial text. Though extracting events from the financial text may be conducive to accurate sentiment predictions, it has specialized challenges due to the lengthy and discontinuity of events in a financial text. To this end, we reconceptualize the event extraction as a classification task by designing a categorization comprising coarse-grained and fine-grained event categories. Under this setting, we formulate the \textbf{E}vent-Level \textbf{F}inancial \textbf{S}entiment \textbf{A}nalysis~(\textbf{EFSA} for short) task that outputs quintuples consisting of (company, industry, coarse-grained event, fine-grained event, sentiment) from financial text. A large-scale Chinese dataset containing $12,160$ news articles and $13,725$ quintuples is publicized as a brand new testbed for our task. A four-hop Chain-of-Thought LLM-based approach is devised for this task. Systematically investigations are conducted on our dataset, and the empirical results demonstrate the benchmarking scores of existing methods and our proposed method can reach the current state-of-the-art. Our dataset and framework implementation are available at https://anonymous.4open.science/r/EFSA-645E

Title: Is English the New Programming Language? How About Pseudo-code Engineering?

• Authors: Gian Alexandre Michaelsen, Renato P. dos Santos
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.08684
• Pdf URL: https://arxiv.org/pdf/2404.08684
• Copy Paste: [[2404.08684]] Is English the New Programming Language? How About Pseudo-code Engineering?(https://arxiv.org/abs/2404.08684)
• Keywords: interpretability
• Abstract: Background: The integration of artificial intelligence (AI) into daily life, particularly through chatbots utilizing natural language processing (NLP), presents both revolutionary potential and unique challenges. This intended to investigate how different input forms impact ChatGPT, a leading language model by OpenAI, performance in understanding and executing complex, multi-intention tasks. Design: Employing a case study methodology supplemented by discourse analysis, the research analyzes ChatGPT's responses to inputs varying from natural language to pseudo-code engineering. The study specifically examines the model's proficiency across four categories: understanding of intentions, interpretability, completeness, and creativity. Setting and Participants: As a theoretical exploration of AI interaction, this study focuses on the analysis of structured and unstructured inputs processed by ChatGPT, without direct human participants. Data collection and analysis: The research utilizes synthetic case scenarios, including the organization of a "weekly meal plan" and a "shopping list," to assess ChatGPT's response to prompts in both natural language and pseudo-code engineering. The analysis is grounded in the identification of patterns, contradictions, and unique response elements across different input formats. Results: Findings reveal that pseudo-code engineering inputs significantly enhance the clarity and determinism of ChatGPT's responses, reducing ambiguity inherent in natural language. Enhanced natural language, structured through prompt engineering techniques, similarly improves the model's interpretability and creativity. Conclusions: The study underscores the potential of pseudo-code engineering in refining human-AI interaction and achieving more deterministic, concise, and direct outcomes, advocating for its broader application across disciplines requiring precise AI responses.

Title: Extractive text summarisation of Privacy Policy documents using machine learning approaches

• Authors: Chanwoo Choi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08686
• Pdf URL: https://arxiv.org/pdf/2404.08686
• Copy Paste: [[2404.08686]] Extractive text summarisation of Privacy Policy documents using machine learning approaches(https://arxiv.org/abs/2404.08686)
• Keywords: privacy, protect
• Abstract: This work demonstrates two Privacy Policy (PP) summarisation models based on two different clustering algorithms: K-means clustering and Pre-determined Centroid (PDC) clustering. K-means is decided to be used for the first model after an extensive evaluation of ten commonly used clustering algorithms. The summariser model based on the PDC-clustering algorithm summarises PP documents by segregating individual sentences by Euclidean distance from each sentence to the pre-defined cluster centres. The cluster centres are defined according to General Data Protection Regulation (GDPR)'s 14 essential topics that must be included in any privacy notices. The PDC model outperformed the K-means model for two evaluation methods, Sum of Squared Distance (SSD) and ROUGE by some margin (27% and 24% respectively). This result contrasts the K-means model's better performance in the general clustering of sentence vectors before running the task-specific evaluation. This indicates the effectiveness of operating task-specific fine-tuning measures on unsupervised machine-learning models. The summarisation mechanisms implemented in this paper demonstrates an idea of how to efficiently extract essential sentences that should be included in any PP documents. The summariser models could be further developed to an application that tests the GDPR-compliance (or any data privacy legislation) of PP documents.

Title: Towards Building a Robust Toxicity Predictor

• Authors: Dmitriy Bespalov, Sourav Bhabesh, Yi Xiang, Liutong Zhou, Yanjun Qi
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08690
• Pdf URL: https://arxiv.org/pdf/2404.08690
• Copy Paste: [[2404.08690]] Towards Building a Robust Toxicity Predictor(https://arxiv.org/abs/2404.08690)
• Keywords: attack, robust
• Abstract: Recent NLP literature pays little attention to the robustness of toxicity language predictors, while these systems are most likely to be used in adversarial contexts. This paper presents a novel adversarial attack, \texttt{ToxicTrap}, introducing small word-level perturbations to fool SOTA text classifiers to predict toxic text samples as benign. ToxicTrap exploits greedy based search strategies to enable fast and effective generation of toxic adversarial examples. Two novel goal function designs allow ToxicTrap to identify weaknesses in both multiclass and multilabel toxic language detectors. Our empirical results show that SOTA toxicity text classifiers are indeed vulnerable to the proposed attacks, attaining over 98\% attack success rates in multilabel cases. We also show how a vanilla adversarial training and its improved version can help increase robustness of a toxicity detector even against unseen attacks.

Title: Enhancing Question Answering for Enterprise Knowledge Bases using Large Language Models

• Authors: Feihu Jiang, Chuan Qin, Kaichun Yao, Chuyu Fang, Fuzhen Zhuang, Hengshu Zhu, Hui Xiong
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.08695
• Pdf URL: https://arxiv.org/pdf/2404.08695
• Copy Paste: [[2404.08695]] Enhancing Question Answering for Enterprise Knowledge Bases using Large Language Models(https://arxiv.org/abs/2404.08695)
• Keywords: security, privacy, generative, large language model
• Abstract: Efficient knowledge management plays a pivotal role in augmenting both the operational efficiency and the innovative capacity of businesses and organizations. By indexing knowledge through vectorization, a variety of knowledge retrieval methods have emerged, significantly enhancing the efficacy of knowledge management systems. Recently, the rapid advancements in generative natural language processing technologies paved the way for generating precise and coherent answers after retrieving relevant documents tailored to user queries. However, for enterprise knowledge bases, assembling extensive training data from scratch for knowledge retrieval and generation is a formidable challenge due to the privacy and security policies of private data, frequently entailing substantial costs. To address the challenge above, in this paper, we propose EKRG, a novel Retrieval-Generation framework based on large language models (LLMs), expertly designed to enable question-answering for Enterprise Knowledge bases with limited annotation costs. Specifically, for the retrieval process, we first introduce an instruction-tuning method using an LLM to generate sufficient document-question pairs for training a knowledge retriever. This method, through carefully designed instructions, efficiently generates diverse questions for enterprise knowledge bases, encompassing both fact-oriented and solution-oriented knowledge. Additionally, we develop a relevance-aware teacher-student learning strategy to further enhance the efficiency of the training process. For the generation process, we propose a novel chain of thought (CoT) based fine-tuning method to empower the LLM-based generator to adeptly respond to user questions using retrieved documents. Finally, extensive experiments on real-world datasets have demonstrated the effectiveness of our proposed framework.

Title: Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding

• Authors: Jie Ou, Yueming Chen, Wenhong Tian
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08698
• Pdf URL: https://arxiv.org/pdf/2404.08698
• Copy Paste: [[2404.08698]] Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding(https://arxiv.org/abs/2404.08698)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) have shown remarkable abilities, they are hindered by significant resource consumption and considerable latency due to autoregressive processing. In this study, we introduce Adaptive N-gram Parallel Decoding (ANPD), an innovative and lossless approach that accelerates inference by allowing the simultaneous generation of multiple tokens. ANPD incorporates a two-stage approach: it begins with a rapid drafting phase that employs an N-gram module, which adapts based on the current interactive context, followed by a verification phase, during which the original LLM assesses and confirms the proposed tokens. Consequently, ANPD preserves the integrity of the LLM's original output while enhancing processing speed. We further leverage a multi-level architecture for the N-gram module to enhance the precision of the initial draft, consequently reducing inference latency. ANPD eliminates the need for retraining or extra GPU memory, making it an efficient and plug-and-play enhancement. In our experiments, models such as LLaMA and its fine-tuned variants have shown speed improvements up to 3.67x, validating the effectiveness of our proposed ANPD.

Title: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in LLMs

• Authors: Ahmed Agiza, Mohamed Mostagir, Sherief Reda
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08699
• Pdf URL: https://arxiv.org/pdf/2404.08699
• Copy Paste: [[2404.08699]] Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in LLMs(https://arxiv.org/abs/2404.08699)
• Keywords: large language model
• Abstract: In an era where language models are increasingly integrated into decision-making and communication, understanding the biases within Large Language Models (LLMs) becomes imperative, especially when these models are applied in the economic and political domains. This work investigates the impact of fine-tuning and data selection on economic and political biases in LLM. We explore the methodological aspects of biasing LLMs towards specific ideologies, mindful of the biases that arise from their extensive training on diverse datasets. Our approach, distinct from earlier efforts that either focus on smaller models or entail resource-intensive pre-training, employs Parameter-Efficient Fine-Tuning (PEFT) techniques. These techniques allow for the alignment of LLMs with targeted ideologies by modifying a small subset of parameters. We introduce a systematic method for dataset selection, annotation, and instruction tuning, and we assess its effectiveness through both quantitative and qualitative evaluations. Our work analyzes the potential of embedding specific biases into LLMs and contributes to the dialogue on the ethical application of AI, highlighting the importance of deploying AI in a manner that aligns with societal values.

Title: Is Your LLM Outdated? Benchmarking LLMs & Alignment Algorithms for Time-Sensitive Knowledge

• Authors: Seyed Mahed Mousavi, Simone Alghisi, Giuseppe Riccardi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08700
• Pdf URL: https://arxiv.org/pdf/2404.08700
• Copy Paste: [[2404.08700]] Is Your LLM Outdated? Benchmarking LLMs & Alignment Algorithms for Time-Sensitive Knowledge(https://arxiv.org/abs/2404.08700)
• Keywords: large language model
• Abstract: We study the appropriateness of Large Language Models (LLMs) as knowledge repositories. We focus on the challenge of maintaining LLMs' factual knowledge up-to-date over time. Motivated by the lack of studies on identifying outdated knowledge within LLMs, we design and develop a dynamic benchmark with up-to-date ground truth answers for each target factual question. We evaluate eighteen open-source and closed-source state-of-the-art LLMs on time-sensitive knowledge retrieved in real-time from Wikidata. We select time-sensitive domain facts in politics, sports, and organizations, and estimate the recency of the information learned by the model during pre-training\fine-tuning. In the second contribution, we evaluate the effectiveness of knowledge editing methods for aligning LLMs with up-to-date factual knowledge and compare their performance with Retrieval Augmented Generation. The dynamic benchmark is designed to be used as-is to assess LLMs's up-to-dateness, as well as to be extended to other domains by sharing the code, the dataset, as well as evaluation and visualization scripts.

Title: MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting

• Authors: Avinash Anand, Janak Kapuriya, Apoorv Singh, Jay Saraf, Naman Lal, Astha Verma, Rushali Gupta, Rajiv Shah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08704
• Pdf URL: https://arxiv.org/pdf/2404.08704
• Copy Paste: [[2404.08704]] MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting(https://arxiv.org/abs/2404.08704)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) can achieve human-level performance in various tasks, they continue to face challenges when it comes to effectively tackling multi-step physics reasoning tasks. To identify the shortcomings of existing models and facilitate further research in this area, we curated a novel dataset, MM-PhyQA, which comprises well-constructed, high schoollevel multimodal physics problems. By evaluating the performance of contemporary LLMs that are publicly available, both with and without the incorporation of multimodal elements in these problems, we aim to shed light on their capabilities. For generating answers for questions consisting of multimodal input (in this case, images and text) we employed Zero-shot prediction using GPT-4 and utilized LLaVA (LLaVA and LLaVA-1.5), the latter of which were fine-tuned on our dataset. For evaluating the performance of LLMs consisting solely of textual input, we tested the performance of the base and fine-tuned versions of the Mistral-7B and LLaMA2-7b models. We also showcased the performance of the novel Multi-Image Chain-of-Thought (MI-CoT) Prompting technique, which when used to train LLaVA-1.5 13b yielded the best results when tested on our dataset, with superior scores in most metrics and the highest accuracy of 71.65% on the test set.

Title: Introducing L2M3, A Multilingual Medical Large Language Model to Advance Health Equity in Low-Resource Regions

• Authors: Agasthya Gangavarapu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08705
• Pdf URL: https://arxiv.org/pdf/2404.08705
• Copy Paste: [[2404.08705]] Introducing L2M3, A Multilingual Medical Large Language Model to Advance Health Equity in Low-Resource Regions(https://arxiv.org/abs/2404.08705)
• Keywords: large language model
• Abstract: Addressing the imminent shortfall of 10 million health workers by 2030, predominantly in Low- and Middle-Income Countries (LMICs), this paper introduces an innovative approach that harnesses the power of Large Language Models (LLMs) integrated with machine translation models. This solution is engineered to meet the unique needs of Community Health Workers (CHWs), overcoming language barriers, cultural sensitivities, and the limited availability of medical dialog datasets. I have crafted a model that not only boasts superior translation capabilities but also undergoes rigorous fine-tuning on open-source datasets to ensure medical accuracy and is equipped with comprehensive safety features to counteract the risks of misinformation. Featuring a modular design, this approach is specifically structured for swift adaptation across various linguistic and cultural contexts, utilizing open-source components to significantly reduce healthcare operational costs. This strategic innovation markedly improves the accessibility and quality of healthcare services by providing CHWs with contextually appropriate medical knowledge and diagnostic tools. This paper highlights the transformative impact of this context-aware LLM, underscoring its crucial role in addressing the global healthcare workforce deficit and propelling forward healthcare outcomes in LMICs.

Title: Large Language Model Can Continue Evolving From Mistakes

• Authors: Haokun Zhao, Haixia Han, Jie Shi, Chengyu Du, Jiaqing Liang, Yanghua Xiao
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08707
• Pdf URL: https://arxiv.org/pdf/2404.08707
• Copy Paste: [[2404.08707]] Large Language Model Can Continue Evolving From Mistakes(https://arxiv.org/abs/2404.08707)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) demonstrate impressive performance in various downstream tasks. However, they may still generate incorrect responses in certain scenarios due to the knowledge deficiencies and the flawed pre-training data. Continual Learning (CL) is a commonly used method to address this issue. Traditional CL is task-oriented, using novel or factually accurate data to retrain LLMs from scratch. However, this method requires more task-related training data and incurs expensive training costs. To address this challenge, we propose the Continue Evolving from Mistakes (CEM) method, inspired by the 'summarize mistakes' learning skill, to achieve iterative refinement of LLMs. Specifically, the incorrect responses of LLMs indicate knowledge deficiencies related to the questions. Therefore, we collect corpora with these knowledge from multiple data sources and follow it up with iterative supplementary training for continuous, targeted knowledge updating and supplementation. Meanwhile, we developed two strategies to construct supplementary training sets to enhance the LLM's understanding of the corpus and prevent catastrophic forgetting. We conducted extensive experiments to validate the effectiveness of this CL method. In the best case, our method resulted in a 17.00\% improvement in the accuracy of the LLM.

Title: Securing Monolithic Kernels using Compartmentalization

• Authors: Soo Yee Lim, Sidhartha Agrawal, Xueyuan Han, David Eyers, Dan O'Keeffe, Thomas Pasquier
• Subjects: cs.CR, cs.OS
• Abstract URL: https://arxiv.org/abs/2404.08716
• Pdf URL: https://arxiv.org/pdf/2404.08716
• Copy Paste: [[2404.08716]] Securing Monolithic Kernels using Compartmentalization(https://arxiv.org/abs/2404.08716)
• Keywords: security, attack
• Abstract: Monolithic operating systems, where all kernel functionality resides in a single, shared address space, are the foundation of most mainstream computer systems. However, a single flaw, even in a non-essential part of the kernel (e.g., device drivers), can cause the entire operating system to fall under an attacker's control. Kernel hardening techniques might prevent certain types of vulnerabilities, but they fail to address a fundamental weakness: the lack of intra-kernel security that safely isolates different parts of the kernel. We survey kernel compartmentalization techniques that define and enforce intra-kernel boundaries and propose a taxonomy that allows the community to compare and discuss future work. We also identify factors that complicate comparisons among compartmentalized systems, suggest new ways to compare future approaches with existing work meaningfully, and discuss emerging research directions.

Title: Exploring Contrastive Learning for Long-Tailed Multi-Label Text Classification

• Authors: Alexandre Audibert, Aurélien Gauffre, Massih-Reza Amini
• Subjects: cs.LG, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.08720
• Pdf URL: https://arxiv.org/pdf/2404.08720
• Copy Paste: [[2404.08720]] Exploring Contrastive Learning for Long-Tailed Multi-Label Text Classification(https://arxiv.org/abs/2404.08720)
• Keywords: robust
• Abstract: Learning an effective representation in multi-label text classification (MLTC) is a significant challenge in NLP. This challenge arises from the inherent complexity of the task, which is shaped by two key factors: the intricate connections between labels and the widespread long-tailed distribution of the data. To overcome this issue, one potential approach involves integrating supervised contrastive learning with classical supervised loss functions. Although contrastive learning has shown remarkable performance in multi-class classification, its impact in the multi-label framework has not been thoroughly investigated. In this paper, we conduct an in-depth study of supervised contrastive learning and its influence on representation in MLTC context. We emphasize the importance of considering long-tailed data distributions to build a robust representation space, which effectively addresses two critical challenges associated with contrastive learning that we identify: the "lack of positives" and the "attraction-repulsion imbalance". Building on this insight, we introduce a novel contrastive loss function for MLTC. It attains Micro-F1 scores that either match or surpass those obtained with other frequently employed loss functions, and demonstrates a significant improvement in Macro-F1 scores across three multi-label datasets.

Title: Beyond One-Size-Fits-All: Adapting Counterfactual Explanations to User Objectives

• Authors: Orfeas Menis Mastromichalakis, Jason Liartis, Giorgos Stamou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08721
• Pdf URL: https://arxiv.org/pdf/2404.08721
• Copy Paste: [[2404.08721]] Beyond One-Size-Fits-All: Adapting Counterfactual Explanations to User Objectives(https://arxiv.org/abs/2404.08721)
• Keywords: interpretability
• Abstract: Explainable Artificial Intelligence (XAI) has emerged as a critical area of research aimed at enhancing the transparency and interpretability of AI systems. Counterfactual Explanations (CFEs) offer valuable insights into the decision-making processes of machine learning algorithms by exploring alternative scenarios where certain factors differ. Despite the growing popularity of CFEs in the XAI community, existing literature often overlooks the diverse needs and objectives of users across different applications and domains, leading to a lack of tailored explanations that adequately address the different use cases. In this paper, we advocate for a nuanced understanding of CFEs, recognizing the variability in desired properties based on user objectives and target applications. We identify three primary user objectives and explore the desired characteristics of CFEs in each case. By addressing these differences, we aim to design more effective and tailored explanations that meet the specific needs of users, thereby enhancing collaboration with AI systems.

Title: Identification of a replicable optical security element using laser speckle

• Authors: A.M. Smolovich, A.V. Frolov, L.D. Klebanov, I.D. Laktaev, A.P. Orlov, P.A. Smolovich, O.V. Butov
• Subjects: cs.CR, physics.optics
• Abstract URL: https://arxiv.org/abs/2404.08723
• Pdf URL: https://arxiv.org/pdf/2404.08723
• Copy Paste: [[2404.08723]] Identification of a replicable optical security element using laser speckle(https://arxiv.org/abs/2404.08723)
• Keywords: security, protect
• Abstract: An optical security element containing an area of random rough relief is proposed. It combines the low cost of mass replication inherent in traditional security holograms with the impossibility of holographic copying, when the wave restored by the hologram is rewritten as a copy of this hologram. The proposed optical element is also protected from contact and photographic copying. Laboratory samples of optical elements were obtained by taking replicas of a rough surface. Identification of the authenticity of optical elements was demonstrated by calculating the cross-correlation of speckle patterns produced by coherent light scattered off different replicas. It is assumed that the proposed security elements can be mass-produced on standard equipment for embossing security holograms.

Title: Training a Vision Language Model as Smartphone Assistant

• Authors: Nicolai Dorka, Janusz Marecki, Ammar Anwar
• Subjects: cs.LG, cs.AI, cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.08755
• Pdf URL: https://arxiv.org/pdf/2404.08755
• Copy Paste: [[2404.08755]] Training a Vision Language Model as Smartphone Assistant(https://arxiv.org/abs/2404.08755)
• Keywords: large language model
• Abstract: Addressing the challenge of a digital assistant capable of executing a wide array of user tasks, our research focuses on the realm of instruction-based mobile device control. We leverage recent advancements in large language models (LLMs) and present a visual language model (VLM) that can fulfill diverse tasks on mobile devices. Our model functions by interacting solely with the user interface (UI). It uses the visual input from the device screen and mimics human-like interactions, encompassing gestures such as tapping and swiping. This generality in the input and output space allows our agent to interact with any application on the device. Unlike previous methods, our model operates not only on a single screen image but on vision-language sentences created from sequences of past screenshots along with corresponding actions. Evaluating our method on the challenging Android in the Wild benchmark demonstrates its promising efficacy and potential.

Title: The Generation Gap:Exploring Age Bias in Large Language Models

• Authors: Siyang Liu, Trish Maturi, Siqi Shen, Rada Mihalcea
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08760
• Pdf URL: https://arxiv.org/pdf/2404.08760
• Copy Paste: [[2404.08760]] The Generation Gap:Exploring Age Bias in Large Language Models(https://arxiv.org/abs/2404.08760)
• Keywords: robust, large language model
• Abstract: In this paper, we explore the alignment of values in Large Language Models (LLMs) with specific age groups, leveraging data from the World Value Survey across thirteen categories. Through a diverse set of prompts tailored to ensure response robustness, we find a general inclination of LLM values towards younger demographics. Additionally, we explore the impact of incorporating age identity information in prompts and observe challenges in mitigating value discrepancies with different age cohorts. Our findings highlight the age bias in LLMs and provide insights for future work.

Title: CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models

• Authors: Je-Yong Lee, Donghyun Lee, Genghan Zhang, Mo Tiwari, Azalia Mirhoseini
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08763
• Pdf URL: https://arxiv.org/pdf/2404.08763
• Copy Paste: [[2404.08763]] CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models(https://arxiv.org/abs/2404.08763)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have dramatically advanced AI applications, yet their deployment remains challenging due to their immense inference costs. Recent studies ameliorate the computational costs of LLMs by increasing their activation sparsity but suffer from significant performance degradation on downstream tasks. In this work, we introduce a new framework for sparsifying the activations of base LLMs and reducing inference costs, dubbed Contextually Aware Thresholding for Sparsity (CATS). CATS is relatively simple, easy to implement, and highly effective. At the heart of our framework is a new non-linear activation function. We demonstrate that CATS can be applied to various base models, including Mistral-7B and Llama2-7B, and outperforms existing sparsification techniques in downstream task performance. More precisely, CATS-based models often achieve downstream task performance within 1-2% of their base models without any fine-tuning and even at activation sparsity levels of 50%. Furthermore, CATS-based models converge faster and display better task performance than competing techniques when fine-tuning is applied. Finally, we develop a custom GPU kernel for efficient implementation of CATS that translates the activation of sparsity of CATS to real wall-clock time speedups. Our custom kernel implementation of CATS results in a ~15% improvement in wall-clock inference latency of token generation on both Llama-7B and Mistral-7B.

Title: LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning

• Authors: Junchi Wang, Lei Ke
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08767
• Pdf URL: https://arxiv.org/pdf/2404.08767
• Copy Paste: [[2404.08767]] LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning(https://arxiv.org/abs/2404.08767)
• Keywords: large language model, segmentation
• Abstract: Understanding human instructions to identify the target objects is vital for perception systems. In recent years, the advancements of Large Language Models (LLMs) have introduced new possibilities for image segmentation. In this work, we delve into reasoning segmentation, a novel task that enables segmentation system to reason and interpret implicit user intention via large language model reasoning and then segment the corresponding target. Our work on reasoning segmentation contributes on both the methodological design and dataset labeling. For the model, we propose a new framework named LLM-Seg. LLM-Seg effectively connects the current foundational Segmentation Anything Model and the LLM by mask proposals selection. For the dataset, we propose an automatic data generation pipeline and construct a new reasoning segmentation dataset named LLM-Seg40K. Experiments demonstrate that our LLM-Seg exhibits competitive performance compared with existing methods. Furthermore, our proposed pipeline can efficiently produce high-quality reasoning segmentation datasets. The LLM-Seg40K dataset, developed through this pipeline, serves as a new benchmark for training and evaluating various reasoning segmentation approaches. Our code, models and dataset are at https://github.com/wangjunchi/LLMSeg.

Title: Detecting AI-Generated Images via CLIP

• Authors: A.G. Moskowitz, T. Gaona, J. Peterson
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08788
• Pdf URL: https://arxiv.org/pdf/2404.08788
• Copy Paste: [[2404.08788]] Detecting AI-Generated Images via CLIP(https://arxiv.org/abs/2404.08788)
• Keywords: generative
• Abstract: As AI-generated image (AIGI) methods become more powerful and accessible, it has become a critical task to determine if an image is real or AI-generated. Because AIGI lack the signatures of photographs and have their own unique patterns, new models are needed to determine if an image is AI-generated. In this paper, we investigate the ability of the Contrastive Language-Image Pre-training (CLIP) architecture, pre-trained on massive internet-scale data sets, to perform this differentiation. We fine-tune CLIP on real images and AIGI from several generative models, enabling CLIP to determine if an image is AI-generated and, if so, determine what generation method was used to create it. We show that the fine-tuned CLIP architecture is able to differentiate AIGI as well or better than models whose architecture is specifically designed to detect AIGI. Our method will significantly increase access to AIGI-detecting tools and reduce the negative effects of AIGI on society, as our CLIP fine-tuning procedures require no architecture changes from publicly available model repositories and consume significantly less GPU resources than other AIGI detection models.

Title: Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes

• Authors: Ali Younis, Erik Sudderth
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.08789
• Pdf URL: https://arxiv.org/pdf/2404.08789
• Copy Paste: [[2404.08789]] Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes(https://arxiv.org/abs/2404.08789)
• Keywords: robust, generative
• Abstract: Particle filters flexibly represent multiple posterior modes nonparametrically, via a collection of weighted samples, but have classically been applied to tracking problems with known dynamics and observation likelihoods. Such generative models may be inaccurate or unavailable for high-dimensional observations like images. We instead leverage training data to discriminatively learn particle-based representations of uncertainty in latent object states, conditioned on arbitrary observations via deep neural network encoders. While prior discriminative particle filters have used heuristic relaxations of discrete particle resampling, or biased learning by truncating gradients at resampling steps, we achieve unbiased and low-variance gradient estimates by representing posteriors as continuous mixture densities. Our theory and experiments expose dramatic failures of existing reparameterization-based estimators for mixture gradients, an issue we address via an importance-sampling gradient estimator. Unlike standard recurrent neural networks, our mixture density particle filter represents multimodal uncertainty in continuous latent states, improving accuracy and robustness. On a range of challenging tracking and robot localization problems, our approach achieves dramatic improvements in accuracy, while also showing much greater stability across multiple training runs.

Title: JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models

• Authors: Yingchaojie Feng, Zhizhang Chen, Zhining Kang, Sijia Wang, Minfeng Zhu, Wei Zhang, Wei Chen
• Subjects: cs.CR, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.08793
• Pdf URL: https://arxiv.org/pdf/2404.08793
• Copy Paste: [[2404.08793]] JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models(https://arxiv.org/abs/2404.08793)
• Keywords: security, attack, large language model
• Abstract: The proliferation of large language models (LLMs) has underscored concerns regarding their security vulnerabilities, notably against jailbreak attacks, where adversaries design jailbreak prompts to circumvent safety mechanisms for potential misuse. Addressing these concerns necessitates a comprehensive analysis of jailbreak prompts to evaluate LLMs' defensive capabilities and identify potential weaknesses. However, the complexity of evaluating jailbreak performance and understanding prompt characteristics makes this analysis laborious. We collaborate with domain experts to characterize problems and propose an LLM-assisted framework to streamline the analysis process. It provides automatic jailbreak assessment to facilitate performance evaluation and support analysis of components and keywords in prompts. Based on the framework, we design JailbreakLens, a visual analysis system that enables users to explore the jailbreak performance against the target model, conduct multi-level analysis of prompt characteristics, and refine prompt instances to verify findings. Through a case study, technical evaluations, and expert interviews, we demonstrate our system's effectiveness in helping users evaluate model security and identify model weaknesses.

Title: Semantic Approach to Quantifying the Consistency of Diffusion Model Image Generation

• Authors: Brinnae Bent
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08799
• Pdf URL: https://arxiv.org/pdf/2404.08799
• Copy Paste: [[2404.08799]] Semantic Approach to Quantifying the Consistency of Diffusion Model Image Generation(https://arxiv.org/abs/2404.08799)
• Keywords: diffusion
• Abstract: In this study, we identify the need for an interpretable, quantitative score of the repeatability, or consistency, of image generation in diffusion models. We propose a semantic approach, using a pairwise mean CLIP (Contrastive Language-Image Pretraining) score as our semantic consistency score. We applied this metric to compare two state-of-the-art open-source image generation diffusion models, Stable Diffusion XL and PixArt-{\alpha}, and we found statistically significant differences between the semantic consistency scores for the models. Agreement between the Semantic Consistency Score selected model and aggregated human annotations was 94%. We also explored the consistency of SDXL and a LoRA-fine-tuned version of SDXL and found that the fine-tuned model had significantly higher semantic consistency in generated images. The Semantic Consistency Score proposed here offers a measure of image generation alignment, facilitating the evaluation of model architectures for specific tasks and aiding in informed decision-making regarding model selection.

Title: Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length

• Authors: Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, Chunting Zhou
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08801
• Pdf URL: https://arxiv.org/pdf/2404.08801
• Copy Paste: [[2404.08801]] Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length(https://arxiv.org/abs/2404.08801)
• Keywords: transformer
• Abstract: The quadratic complexity and weak length extrapolation of Transformers limits their ability to scale to long sequences, and while sub-quadratic solutions like linear attention and state space models exist, they empirically underperform Transformers in pretraining efficiency and downstream task accuracy. We introduce Megalodon, a neural architecture for efficient sequence modeling with unlimited context length. Megalodon inherits the architecture of Mega (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability and stability, including complex exponential moving average (CEMA), timestep normalization layer, normalized attention mechanism and pre-norm with two-hop residual configuration. In a controlled head-to-head comparison with Llama2, Megalodon achieves better efficiency than Transformer in the scale of 7 billion parameters and 2 trillion training tokens. Megalodon reaches a training loss of 1.70, landing mid-way between Llama2-7B (1.75) and 13B (1.67). Code: https://github.com/XuezheMax/megalodon

Title: CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation

• Authors: Matthew DeLorenzo, Vasudev Gohil, Jeyavijayan Rajendran
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08806
• Pdf URL: https://arxiv.org/pdf/2404.08806
• Copy Paste: [[2404.08806]] CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation(https://arxiv.org/abs/2404.08806)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have proved effective and efficient in generating code, leading to their utilization within the hardware design process. Prior works evaluating LLMs' abilities for register transfer level code generation solely focus on functional correctness. However, the creativity associated with these LLMs, or the ability to generate novel and unique solutions, is a metric not as well understood, in part due to the challenge of quantifying this quality. To address this research gap, we present CreativeEval, a framework for evaluating the creativity of LLMs within the context of generating hardware designs. We quantify four creative sub-components, fluency, flexibility, originality, and elaboration, through various prompting and post-processing techniques. We then evaluate multiple popular LLMs (including GPT models, CodeLlama, and VeriGen) upon this creativity metric, with results indicating GPT-3.5 as the most creative model in generating hardware designs.

Title: Enhancing IoT Malware Detection through Adaptive Model Parallelism and Resource Optimization

• Authors: Sreenitha Kasarapu, Sanket Shukla, Sai Manoj Pudukotai Dinakarrao
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.08808
• Pdf URL: https://arxiv.org/pdf/2404.08808
• Copy Paste: [[2404.08808]] Enhancing IoT Malware Detection through Adaptive Model Parallelism and Resource Optimization(https://arxiv.org/abs/2404.08808)
• Keywords: security, privacy
• Abstract: The widespread integration of IoT devices has greatly improved connectivity and computational capabilities, facilitating seamless communication across networks. Despite their global deployment, IoT devices are frequently targeted for security breaches due to inherent vulnerabilities. Among these threats, malware poses a significant risk to IoT devices. The lack of built-in security features and limited resources present challenges for implementing effective malware detection techniques on IoT devices. Moreover, existing methods assume access to all device resources for malware detection, which is often not feasible for IoT devices deployed in critical real-world scenarios. To overcome this challenge, this study introduces a novel approach to malware detection tailored for IoT devices, leveraging resource and workload awareness inspired by model parallelism. Initially, the device assesses available resources for malware detection using a lightweight regression model. Based on resource availability, ongoing workload, and communication costs, the malware detection task is dynamically allocated either on-device or offloaded to neighboring IoT nodes with sufficient resources. To uphold data integrity and user privacy, instead of transferring the entire malware detection task, the classifier is divided and distributed across multiple nodes, then integrated at the parent node for detection. Experimental results demonstrate that this proposed technique achieves a significant speedup of 9.8 x compared to on-device inference, while maintaining a high malware detection accuracy of 96.7%.

Title: Leveraging viscous Hamilton-Jacobi PDEs for uncertainty quantification in scientific machine learning

• Authors: Zongren Zou, Tingwei Meng, Paula Chen, Jérôme Darbon, George Em Karniadakis
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.08809
• Pdf URL: https://arxiv.org/pdf/2404.08809
• Copy Paste: [[2404.08809]] Leveraging viscous Hamilton-Jacobi PDEs for uncertainty quantification in scientific machine learning(https://arxiv.org/abs/2404.08809)
• Keywords: interpretability
• Abstract: Uncertainty quantification (UQ) in scientific machine learning (SciML) combines the powerful predictive power of SciML with methods for quantifying the reliability of the learned models. However, two major challenges remain: limited interpretability and expensive training procedures. We provide a new interpretation for UQ problems by establishing a new theoretical connection between some Bayesian inference problems arising in SciML and viscous Hamilton-Jacobi partial differential equations (HJ PDEs). Namely, we show that the posterior mean and covariance can be recovered from the spatial gradient and Hessian of the solution to a viscous HJ PDE. As a first exploration of this connection, we specialize to Bayesian inference problems with linear models, Gaussian likelihoods, and Gaussian priors. In this case, the associated viscous HJ PDEs can be solved using Riccati ODEs, and we develop a new Riccati-based methodology that provides computational advantages when continuously updating the model predictions. Specifically, our Riccati-based approach can efficiently add or remove data points to the training set invariant to the order of the data and continuously tune hyperparameters. Moreover, neither update requires retraining on or access to previously incorporated data. We provide several examples from SciML involving noisy data and \textit{epistemic uncertainty} to illustrate the potential advantages of our approach. In particular, this approach's amenability to data streaming applications demonstrates its potential for real-time inferences, which, in turn, allows for applications in which the predicted uncertainty is used to dynamically alter the learning process.

Title: E3: Ensemble of Expert Embedders for Adapting Synthetic Image Detectors to New Generators Using Limited Data

• Authors: Aref Azizpour, Tai D. Nguyen, Manil Shrestha, Kaidi Xu, Edward Kim, Matthew C. Stamm
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08814
• Pdf URL: https://arxiv.org/pdf/2404.08814
• Copy Paste: [[2404.08814]] E3: Ensemble of Expert Embedders for Adapting Synthetic Image Detectors to New Generators Using Limited Data(https://arxiv.org/abs/2404.08814)
• Keywords: generative
• Abstract: As generative AI progresses rapidly, new synthetic image generators continue to emerge at a swift pace. Traditional detection methods face two main challenges in adapting to these generators: the forensic traces of synthetic images from new techniques can vastly differ from those learned during training, and access to data for these new generators is often limited. To address these issues, we introduce the Ensemble of Expert Embedders (E3), a novel continual learning framework for updating synthetic image detectors. E3 enables the accurate detection of images from newly emerged generators using minimal training data. Our approach does this by first employing transfer learning to develop a suite of expert embedders, each specializing in the forensic traces of a specific generator. Then, all embeddings are jointly analyzed by an Expert Knowledge Fusion Network to produce accurate and reliable detection decisions. Our experiments demonstrate that E3 outperforms existing continual learning methods, including those developed specifically for synthetic image detection.

Title: Evaluating the Quality of Answers in Political Q&A Sessions with Large Language Models

• Authors: R. Michael Alvarez, Jacob Morrier
• Subjects: cs.CL, econ.EM
• Abstract URL: https://arxiv.org/abs/2404.08816
• Pdf URL: https://arxiv.org/pdf/2404.08816
• Copy Paste: [[2404.08816]] Evaluating the Quality of Answers in Political Q&A Sessions with Large Language Models(https://arxiv.org/abs/2404.08816)
• Keywords: large language model
• Abstract: This paper presents a new approach to evaluating the quality of answers in political question-and-answer sessions. We propose to measure an answer's quality based on the degree to which it allows us to infer the initial question accurately. This conception of answer quality inherently reflects their relevance to initial questions. Drawing parallels with semantic search, we argue that this measurement approach can be operationalized by fine-tuning a large language model on the observed corpus of questions and answers without additional labeled data. We showcase our measurement approach within the context of the Question Period in the Canadian House of Commons. Our approach yields valuable insights into the correlates of the quality of answers in the Question Period. We find that answer quality varies significantly based on the party affiliation of the members of Parliament asking the questions and uncover a meaningful correlation between answer quality and the topics of the questions.

Title: Empowering Malware Detection Efficiency within Processing-in-Memory Architecture

• Authors: Sreenitha Kasarapu, Sathwika Bavikadi, Sai Manoj Pudukotai Dinakarrao
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2404.08818
• Pdf URL: https://arxiv.org/pdf/2404.08818
• Copy Paste: [[2404.08818]] Empowering Malware Detection Efficiency within Processing-in-Memory Architecture(https://arxiv.org/abs/2404.08818)
• Keywords: security
• Abstract: The widespread integration of embedded systems across various industries has facilitated seamless connectivity among devices and bolstered computational capabilities. Despite their extensive applications, embedded systems encounter significant security threats, with one of the most critical vulnerabilities being malicious software, commonly known as malware. In recent times, malware detection techniques leveraging Machine Learning have gained popularity. Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) have proven particularly efficient in image processing tasks. However, one major drawback of neural network architectures is their substantial computational resource requirements. Continuous training of malware detection models with updated malware and benign samples demands immense computational resources, presenting a challenge for real-world applications. In response to these concerns, we propose a Processing-in-Memory (PIM)-based architecture to mitigate memory access latency, thereby reducing the resources consumed during model updates. To further enhance throughput and minimize energy consumption, we incorporate precision scaling techniques tailored for CNN models. Our proposed PIM architecture exhibits a 1.09x higher throughput compared to existing Lookup Table (LUT)-based PIM architectures. Additionally, precision scaling combined with PIM enhances energy efficiency by 1.5x compared to full-precision operations, without sacrificing performance. This innovative approach offers a promising solution to the resource-intensive nature of malware detection model updates, paving the way for more efficient and sustainable cybersecurity practices.

Title: The Illusion of State in State-Space Models

• Authors: William Merrill, Jackson Petty, Ashish Sabharwal
• Subjects: cs.LG, cs.CC, cs.CL, cs.FL
• Abstract URL: https://arxiv.org/abs/2404.08819
• Pdf URL: https://arxiv.org/pdf/2404.08819
• Copy Paste: [[2404.08819]] The Illusion of State in State-Space Models(https://arxiv.org/abs/2404.08819)
• Keywords: transformer, large language model
• Abstract: State-space models (SSMs) have emerged as a potential alternative architecture for building large language models (LLMs) compared to the previously ubiquitous transformer architecture. One theoretical weakness of transformers is that they cannot express certain kinds of sequential computation and state tracking (Merrill and Sabharwal, 2023), which SSMs are explicitly designed to address via their close architectural similarity to recurrent neural networks (RNNs). But do SSMs truly have an advantage (over transformers) in expressive power for state tracking? Surprisingly, the answer is no. Our analysis reveals that the expressive power of SSMs is limited very similarly to transformers: SSMs cannot express computation outside the complexity class $\mathsf{TC}^0$. In particular, this means they cannot solve simple state-tracking problems like permutation composition. It follows that SSMs are provably unable to accurately track chess moves with certain notation, evaluate code, or track entities in a long narrative. To supplement our formal analysis, we report experiments showing that Mamba-style SSMs indeed struggle with state tracking. Thus, despite its recurrent formulation, the "state" in an SSM is an illusion: SSMs have similar expressiveness limitations to non-recurrent models like transformers, which may fundamentally limit their ability to solve real-world state-tracking problems.

Title: Single-image driven 3d viewpoint training data augmentation for effective wine label recognition

• Authors: Yueh-Cheng Huang, Hsin-Yi Chen, Cheng-Jui Hung, Jen-Hui Chuang, Jenq-Neng Hwang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08820
• Pdf URL: https://arxiv.org/pdf/2404.08820
• Copy Paste: [[2404.08820]] Single-image driven 3d viewpoint training data augmentation for effective wine label recognition(https://arxiv.org/abs/2404.08820)
• Keywords: transformer, generative
• Abstract: Confronting the critical challenge of insufficient training data in the field of complex image recognition, this paper introduces a novel 3D viewpoint augmentation technique specifically tailored for wine label recognition. This method enhances deep learning model performance by generating visually realistic training samples from a single real-world wine label image, overcoming the challenges posed by the intricate combinations of text and logos. Classical Generative Adversarial Network (GAN) methods fall short in synthesizing such intricate content combination. Our proposed solution leverages time-tested computer vision and image processing strategies to expand our training dataset, thereby broadening the range of training samples for deep learning applications. This innovative approach to data augmentation circumvents the constraints of limited training resources. Using the augmented training images through batch-all triplet metric learning on a Vision Transformer (ViT) architecture, we can get the most discriminative embedding features for every wine label, enabling us to perform one-shot recognition of existing wine labels in the training classes or future newly collected wine labels unavailable in the training. Experimental results show a significant increase in recognition accuracy over conventional 2D data augmentation techniques.

Title: Experimental Design for Active Transductive Inference in Large Language Models

• Authors: Subhojyoti Mukherjee, Ge Liu, Aniket Deshmukh, Anusha Lalitha, Yifei Ma, Branislav Kveton
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08846
• Pdf URL: https://arxiv.org/pdf/2404.08846
• Copy Paste: [[2404.08846]] Experimental Design for Active Transductive Inference in Large Language Models(https://arxiv.org/abs/2404.08846)
• Keywords: large language model
• Abstract: Transduction, the ability to include query-specific examples in the prompt at inference time, is one of the emergent abilities of large language models (LLMs). In this work, we propose a framework for adaptive prompt design called active transductive inference (ATI). We design the LLM prompt by adaptively choosing few-shot examples for a given inference query. The examples are initially unlabeled and we query the user to label the most informative ones, which maximally reduces the uncertainty in the LLM prediction. We propose two algorithms, GO and SAL, which differ in how the few-shot examples are chosen. We analyze these algorithms in linear models: first GO and then use its equivalence with SAL. We experiment with many different tasks and show that GO and SAL outperform other methods for choosing few-shot examples in the LLM prompt at inference time.

Title: On Speculative Decoding for Multimodal Large Language Models

• Authors: Mukul Gagrani, Raghavv Goel, Wonseok Jeon, Junyoung Park, Mingu Lee, Christopher Lott
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08856
• Pdf URL: https://arxiv.org/pdf/2404.08856
• Copy Paste: [[2404.08856]] On Speculative Decoding for Multimodal Large Language Models(https://arxiv.org/abs/2404.08856)
• Keywords: large language model
• Abstract: Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing components from the draft model. Our experiments across three different tasks show that speculative decoding can achieve a memory-bound speedup of up to 2.37$\times$ using a 115M parameter language model that we trained from scratch. Additionally, we introduce a compact LLaVA draft model incorporating an image adapter, which shows marginal performance gains in image captioning while maintaining comparable results in other tasks.

Title: LLM In-Context Recall is Prompt Dependent

• Authors: Daniel Machlab, Rick Battle
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08865
• Pdf URL: https://arxiv.org/pdf/2404.08865
• Copy Paste: [[2404.08865]] LLM In-Context Recall is Prompt Dependent(https://arxiv.org/abs/2404.08865)
• Keywords: large language model
• Abstract: The proliferation of Large Language Models (LLMs) highlights the critical importance of conducting thorough evaluations to discern their comparative advantages, limitations, and optimal use cases. Particularly important is assessing their capacity to accurately retrieve information included in a given prompt. A model's ability to do this significantly influences how effectively it can utilize contextual details, thus impacting its practical efficacy and dependability in real-world applications. Our research analyzes the in-context recall performance of various LLMs using the needle-in-a-haystack method. In this approach, a factoid (the "needle") is embedded within a block of filler text (the "haystack"), which the model is asked to retrieve. We assess the recall performance of each model across various haystack lengths and with varying needle placements to identify performance patterns. This study demonstrates that an LLM's recall capability is not only contingent upon the prompt's content but also may be compromised by biases in its training data. Conversely, adjustments to model architecture, training strategy, or fine-tuning can improve performance. Our analysis provides insight into LLM behavior, offering direction for the development of more effective applications of LLMs.

Title: An evaluation framework for synthetic data generation models

• Authors: Ioannis E. Livieris, Nikos Alimpertis, George Domalis, Dimitris Tsakalidis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08866
• Pdf URL: https://arxiv.org/pdf/2404.08866
• Copy Paste: [[2404.08866]] An evaluation framework for synthetic data generation models(https://arxiv.org/abs/2404.08866)
• Keywords: privacy
• Abstract: Nowadays, the use of synthetic data has gained popularity as a cost-efficient strategy for enhancing data augmentation for improving machine learning models performance as well as addressing concerns related to sensitive data privacy. Therefore, the necessity of ensuring quality of generated synthetic data, in terms of accurate representation of real data, consists of primary importance. In this work, we present a new framework for evaluating synthetic data generation models' ability for developing high-quality synthetic data. The proposed approach is able to provide strong statistical and theoretical information about the evaluation framework and the compared models' ranking. Two use case scenarios demonstrate the applicability of the proposed framework for evaluating the ability of synthetic data generation models to generated high quality data. The implementation code can be found in https://github.com/novelcore/synthetic_data_evaluation_framework.

Title: EIVEN: Efficient Implicit Attribute Value Extraction using Multimodal LLM

• Authors: Henry Peng Zou, Gavin Heqing Yu, Ziwei Fan, Dan Bu, Han Liu, Peng Dai, Dongmei Jia, Cornelia Caragea
• Subjects: cs.CV, cs.AI, cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08886
• Pdf URL: https://arxiv.org/pdf/2404.08886
• Copy Paste: [[2404.08886]] EIVEN: Efficient Implicit Attribute Value Extraction using Multimodal LLM(https://arxiv.org/abs/2404.08886)
• Keywords: extraction, generative
• Abstract: In e-commerce, accurately extracting product attribute values from multimodal data is crucial for improving user experience and operational efficiency of retailers. However, previous approaches to multimodal attribute value extraction often struggle with implicit attribute values embedded in images or text, rely heavily on extensive labeled data, and can easily confuse similar attribute values. To address these issues, we introduce EIVEN, a data- and parameter-efficient generative framework that pioneers the use of multimodal LLM for implicit attribute value extraction. EIVEN leverages the rich inherent knowledge of a pre-trained LLM and vision encoder to reduce reliance on labeled data. We also introduce a novel Learning-by-Comparison technique to reduce model confusion by enforcing attribute value comparison and difference identification. Additionally, we construct initial open-source datasets for multimodal implicit attribute value extraction. Our extensive experiments reveal that EIVEN significantly outperforms existing methods in extracting implicit attribute values while requiring less labeled data.

Title: ChangeAnywhere: Sample Generation for Remote Sensing Change Detection via Semantic Latent Diffusion Model

• Authors: Kai Tang, Jin Chen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08892
• Pdf URL: https://arxiv.org/pdf/2404.08892
• Copy Paste: [[2404.08892]] ChangeAnywhere: Sample Generation for Remote Sensing Change Detection via Semantic Latent Diffusion Model(https://arxiv.org/abs/2404.08892)
• Keywords: diffusion
• Abstract: Remote sensing change detection (CD) is a pivotal technique that pinpoints changes on a global scale based on multi-temporal images. With the recent expansion of deep learning, supervised deep learning-based CD models have shown satisfactory performance. However, CD sample labeling is very time-consuming as it is densely labeled and requires expert knowledge. To alleviate this problem, we introduce ChangeAnywhere, a novel CD sample generation method using the semantic latent diffusion model and single-temporal images. Specifically, ChangeAnywhere leverages the relative ease of acquiring large single-temporal semantic datasets to generate large-scale, diverse, and semantically annotated bi-temporal CD datasets. ChangeAnywhere captures the two essentials of CD samples, i.e., change implies semantically different, and non-change implies reasonable change under the same semantic constraints. We generated ChangeAnywhere-100K, the largest synthesis CD dataset with 100,000 pairs of CD samples based on the proposed method. The ChangeAnywhere-100K significantly improved both zero-shot and few-shot performance on two CD benchmark datasets for various deep learning-based CD models, as demonstrated by transfer experiments. This paper delineates the enormous potential of ChangeAnywhere for CD sample generation and demonstrates the subsequent enhancement of model performance. Therefore, ChangeAnywhere offers a potent tool for remote sensing CD. All codes and pre-trained models will be available at https://github.com/tangkai-RS/ChangeAnywhere.

Title: HEAT: Head-level Parameter Efficient Adaptation of Vision Transformers with Taylor-expansion Importance Scores

• Authors: Yibo Zhong, Yao Zhou
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08894
• Pdf URL: https://arxiv.org/pdf/2404.08894
• Copy Paste: [[2404.08894]] HEAT: Head-level Parameter Efficient Adaptation of Vision Transformers with Taylor-expansion Importance Scores(https://arxiv.org/abs/2404.08894)
• Keywords: transformer
• Abstract: Prior computer vision research extensively explores adapting pre-trained vision transformers (ViT) to downstream tasks. However, the substantial number of parameters requiring adaptation has led to a focus on Parameter Efficient Transfer Learning (PETL) as an approach to efficiently adapt large pre-trained models by training only a subset of parameters, achieving both parameter and storage efficiency. Although the significantly reduced parameters have shown promising performance under transfer learning scenarios, the structural redundancy inherent in the model still leaves room for improvement, which warrants further investigation. In this paper, we propose Head-level Efficient Adaptation with Taylor-expansion importance score (HEAT): a simple method that efficiently fine-tuning ViTs at head levels. In particular, the first-order Taylor expansion is employed to calculate each head's importance score, termed Taylor-expansion Importance Score (TIS), indicating its contribution to specific tasks. Additionally, three strategies for calculating TIS have been employed to maximize the effectiveness of TIS. These strategies calculate TIS from different perspectives, reflecting varying contributions of parameters. Besides ViT, HEAT has also been applied to hierarchical transformers such as Swin Transformer, demonstrating its versatility across different transformer architectures. Through extensive experiments, HEAT has demonstrated superior performance over state-of-the-art PETL methods on the VTAB-1K benchmark.

Title: Meply: A Large-scale Dataset and Baseline Evaluations for Metastatic Perirectal Lymph Node Detection and Segmentation

• Authors: Weidong Guo, Hantao Zhang, Shouhong Wan, Bingbing Zou, Wanqin Wang, Chenyang Qiu, Jun Li, Peiquan Jin
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08916
• Pdf URL: https://arxiv.org/pdf/2404.08916
• Copy Paste: [[2404.08916]] Meply: A Large-scale Dataset and Baseline Evaluations for Metastatic Perirectal Lymph Node Detection and Segmentation(https://arxiv.org/abs/2404.08916)
• Keywords: segmentation
• Abstract: Accurate segmentation of metastatic lymph nodes in rectal cancer is crucial for the staging and treatment of rectal cancer. However, existing segmentation approaches face challenges due to the absence of pixel-level annotated datasets tailored for lymph nodes around the rectum. Additionally, metastatic lymph nodes are characterized by their relatively small size, irregular shapes, and lower contrast compared to the background, further complicating the segmentation task. To address these challenges, we present the first large-scale perirectal metastatic lymph node CT image dataset called Meply, which encompasses pixel-level annotations of 269 patients diagnosed with rectal cancer. Furthermore, we introduce a novel lymph-node segmentation model named CoSAM. The CoSAM utilizes sequence-based detection to guide the segmentation of metastatic lymph nodes in rectal cancer, contributing to improved localization performance for the segmentation model. It comprises three key components: sequence-based detection module, segmentation module, and collaborative convergence unit. To evaluate the effectiveness of CoSAM, we systematically compare its performance with several popular segmentation methods using the Meply dataset. Our code and dataset will be publicly available at: https://github.com/kanydao/CoSAM.

Title: MAProtoNet: A Multi-scale Attentive Interpretable Prototypical Part Network for 3D Magnetic Resonance Imaging Brain Tumor Classification

• Authors: Binghua Li, Jie Mao, Zhe Sun, Chao Li, Qibin Zhao, Toshihisa Tanaka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08917
• Pdf URL: https://arxiv.org/pdf/2404.08917
• Copy Paste: [[2404.08917]] MAProtoNet: A Multi-scale Attentive Interpretable Prototypical Part Network for 3D Magnetic Resonance Imaging Brain Tumor Classification(https://arxiv.org/abs/2404.08917)
• Keywords: interpretability, segmentation
• Abstract: Automated diagnosis with artificial intelligence has emerged as a promising area in the realm of medical imaging, while the interpretability of the introduced deep neural networks still remains an urgent concern. Although contemporary works, such as XProtoNet and MProtoNet, has sought to design interpretable prediction models for the issue, the localization precision of their resulting attribution maps can be further improved. To this end, we propose a Multi-scale Attentive Prototypical part Network, termed MAProtoNet, to provide more precise maps for attribution. Specifically, we introduce a concise multi-scale module to merge attentive features from quadruplet attention layers, and produces attribution maps. The proposed quadruplet attention layers can enhance the existing online class activation mapping loss via capturing interactions between the spatial and channel dimension, while the multi-scale module then fuses both fine-grained and coarse-grained information for precise maps generation. We also apply a novel multi-scale mapping loss for supervision on the proposed multi-scale module. Compared to existing interpretable prototypical part networks in medical imaging, MAProtoNet can achieve state-of-the-art performance in localization on brain tumor segmentation (BraTS) datasets, resulting in approximately 4% overall improvement on activation precision score (with a best score of 85.8%), without using additional annotated labels of segmentation. Our code will be released in https://github.com/TUAT-Novice/maprotonet.

Title: Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space

• Authors: Zhuyang Xie, Yan Yang, Jie Wang, Xiaorong Liu, Xiaofan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08923
• Pdf URL: https://arxiv.org/pdf/2404.08923
• Copy Paste: [[2404.08923]] Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space(https://arxiv.org/abs/2404.08923)
• Keywords: robust
• Abstract: Multimodal video sentiment analysis aims to integrate multiple modal information to analyze the opinions and attitudes of speakers. Most previous work focuses on exploring the semantic interactions of intra- and inter-modality. However, these works ignore the reliability of multimodality, i.e., modalities tend to contain noise, semantic ambiguity, missing modalities, etc. In addition, previous multimodal approaches treat different modalities equally, largely ignoring their different contributions. Furthermore, existing multimodal sentiment analysis methods directly regress sentiment scores without considering ordinal relationships within sentiment categories, with limited performance. To address the aforementioned problems, we propose a trustworthy multimodal sentiment ordinal network (TMSON) to improve performance in sentiment analysis. Specifically, we first devise a unimodal feature extractor for each modality to obtain modality-specific features. Then, an uncertainty distribution estimation network is customized, which estimates the unimodal uncertainty distributions. Next, Bayesian fusion is performed on the learned unimodal distributions to obtain multimodal distributions for sentiment prediction. Finally, an ordinal-aware sentiment space is constructed, where ordinal regression is used to constrain the multimodal distributions. Our proposed TMSON outperforms baselines on multimodal sentiment analysis tasks, and empirical results demonstrate that TMSON is capable of reducing uncertainty to obtain more robust predictions.

Title: Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives

• Authors: Yidan Liu, Jun Yue, Shaobo Xia, Pedram Ghamisi, Weiying Xie, Leyuan Fang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08926
• Pdf URL: https://arxiv.org/pdf/2404.08926
• Copy Paste: [[2404.08926]] Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives(https://arxiv.org/abs/2404.08926)
• Keywords: diffusion, generative
• Abstract: As a newly emerging advance in deep generative models, diffusion models have achieved state-of-the-art results in many fields, including computer vision, natural language processing, and molecule design. The remote sensing community has also noticed the powerful ability of diffusion models and quickly applied them to a variety of tasks for image processing. Given the rapid increase in research on diffusion models in the field of remote sensing, it is necessary to conduct a comprehensive review of existing diffusion model-based remote sensing papers, to help researchers recognize the potential of diffusion models and provide some directions for further exploration. Specifically, this paper first introduces the theoretical background of diffusion models, and then systematically reviews the applications of diffusion models in remote sensing, including image generation, enhancement, and interpretation. Finally, the limitations of existing remote sensing diffusion models and worthy research directions for further exploration are discussed and summarized.

Title: Shifting Spotlight for Co-supervision: A Simple yet Efficient Single-branch Network to See Through Camouflage

• Authors: Yang Hu, Jinxia Zhang, Kaihua Zhang, Yin Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08936
• Pdf URL: https://arxiv.org/pdf/2404.08936
• Copy Paste: [[2404.08936]] Shifting Spotlight for Co-supervision: A Simple yet Efficient Single-branch Network to See Through Camouflage(https://arxiv.org/abs/2404.08936)
• Keywords: extraction
• Abstract: Efficient and accurate camouflaged object detection (COD) poses a challenge in the field of computer vision. Recent approaches explored the utility of edge information for network co-supervision, achieving notable advancements. However, these approaches introduce an extra branch for complex edge extraction, complicate the model architecture and increases computational demands. Addressing this issue, our work replicates the effect that animal's camouflage can be easily revealed under a shifting spotlight, and leverages it for network co-supervision to form a compact yet efficient single-branch network, the Co-Supervised Spotlight Shifting Network (CS$^3$Net). The spotlight shifting strategy allows CS$^3$Net to learn additional prior within a single-branch framework, obviating the need for resource demanding multi-branch design. To leverage the prior of spotlight shifting co-supervision, we propose Shadow Refinement Module (SRM) and Projection Aware Attention (PAA) for feature refinement and enhancement. To ensure the continuity of multi-scale features aggregation, we utilize the Extended Neighbor Connection Decoder (ENCD) for generating the final predictions. Empirical evaluations on public datasets confirm that our CS$^3$Net offers an optimal balance between efficiency and performance: it accomplishes a 32.13% reduction in Multiply-Accumulate (MACs) operations compared to leading efficient COD models, while also delivering superior performance.

Title: Enforcing Paraphrase Generation via Controllable Latent Diffusion

• Authors: Wei Zou, Ziyuan Zhuang, Shujian Huang, Jia Liu, Jiajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08938
• Pdf URL: https://arxiv.org/pdf/2404.08938
• Copy Paste: [[2404.08938]] Enforcing Paraphrase Generation via Controllable Latent Diffusion(https://arxiv.org/abs/2404.08938)
• Keywords: diffusion
• Abstract: Paraphrase generation aims to produce high-quality and diverse utterances of a given text. Though state-of-the-art generation via the diffusion model reconciles generation quality and diversity, textual diffusion suffers from a truncation issue that hinders efficiency and quality control. In this work, we propose \textit{L}atent \textit{D}iffusion \textit{P}araphraser~(LDP), a novel paraphrase generation by modeling a controllable diffusion process given a learned latent space. LDP achieves superior generation efficiency compared to its diffusion counterparts. It facilitates only input segments to enforce paraphrase semantics, which further improves the results without external features. Experiments show that LDP achieves improved and diverse paraphrase generation compared to baselines. Further analysis shows that our method is also helpful to other similar text generations and domain adaptations. Our code and data are available at https://github.com/NIL-zhuang/ld4pg.

Title: Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles

• Authors: Abhijnan Nath, Huma Jamil, Shafiuddin Rehan Ahmed, George Baker, Rahul Ghosh, James H. Martin, Nathaniel Blanchard, Nikhil Krishnaswamy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08949
• Pdf URL: https://arxiv.org/pdf/2404.08949
• Copy Paste: [[2404.08949]] Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles(https://arxiv.org/abs/2404.08949)
• Keywords: diffusion
• Abstract: Event coreference resolution (ECR) is the task of determining whether distinct mentions of events within a multi-document corpus are actually linked to the same underlying occurrence. Images of the events can help facilitate resolution when language is ambiguous. Here, we propose a multimodal cross-document event coreference resolution method that integrates visual and textual cues with a simple linear map between vision and language models. As existing ECR benchmark datasets rarely provide images for all event mentions, we augment the popular ECB+ dataset with event-centric images scraped from the internet and generated using image diffusion models. We establish three methods that incorporate images and text for coreference: 1) a standard fused model with finetuning, 2) a novel linear mapping method without finetuning and 3) an ensembling approach based on splitting mention pairs by semantic and discourse-level difficulty. We evaluate on 2 datasets: the augmented ECB+, and AIDA Phase 1. Our ensemble systems using cross-modal linear mapping establish an upper limit (91.9 CoNLL F1) on ECB+ ECR performance given the preprocessing assumptions used, and establish a novel baseline on AIDA Phase 1. Our results demonstrate the utility of multimodal information in ECR for certain challenging coreference problems, and highlight a need for more multimodal resources in the coreference resolution space.

Title: Constructing and Exploring Intermediate Domains in Mixed Domain Semi-supervised Medical Image Segmentation

• Authors: Qinghe Ma, Jian Zhang, Lei Qi, Qian Yu, Yinghuan Shi, Yang Gao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08951
• Pdf URL: https://arxiv.org/pdf/2404.08951
• Copy Paste: [[2404.08951]] Constructing and Exploring Intermediate Domains in Mixed Domain Semi-supervised Medical Image Segmentation(https://arxiv.org/abs/2404.08951)
• Keywords: segmentation
• Abstract: Both limited annotation and domain shift are prevalent challenges in medical image segmentation. Traditional semi-supervised segmentation and unsupervised domain adaptation methods address one of these issues separately. However, the coexistence of limited annotation and domain shift is quite common, which motivates us to introduce a novel and challenging scenario: Mixed Domain Semi-supervised medical image Segmentation (MiDSS). In this scenario, we handle data from multiple medical centers, with limited annotations available for a single domain and a large amount of unlabeled data from multiple domains. We found that the key to solving the problem lies in how to generate reliable pseudo labels for the unlabeled data in the presence of domain shift with labeled data. To tackle this issue, we employ Unified Copy-Paste (UCP) between images to construct intermediate domains, facilitating the knowledge transfer from the domain of labeled data to the domains of unlabeled data. To fully utilize the information within the intermediate domain, we propose a symmetric Guidance training strategy (SymGD), which additionally offers direct guidance to unlabeled data by merging pseudo labels from intermediate samples. Subsequently, we introduce a Training Process aware Random Amplitude MixUp (TP-RAM) to progressively incorporate style-transition components into intermediate samples. Compared with existing state-of-the-art approaches, our method achieves a notable 13.57% improvement in Dice score on Prostate dataset, as demonstrated on three public datasets. Our code is available at https://github.com/MQinghe/MiDSS .

Title: Understanding Multimodal Deep Neural Networks: A Concept Selection View

• Authors: Chenming Shang, Hengyuan Zhang, Hao Wen, Yujiu Yang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08964
• Pdf URL: https://arxiv.org/pdf/2404.08964
• Copy Paste: [[2404.08964]] Understanding Multimodal Deep Neural Networks: A Concept Selection View(https://arxiv.org/abs/2404.08964)
• Keywords: extraction
• Abstract: The multimodal deep neural networks, represented by CLIP, have generated rich downstream applications owing to their excellent performance, thus making understanding the decision-making process of CLIP an essential research topic. Due to the complex structure and the massive pre-training data, it is often regarded as a black-box model that is too difficult to understand and interpret. Concept-based models map the black-box visual representations extracted by deep neural networks onto a set of human-understandable concepts and use the concepts to make predictions, enhancing the transparency of the decision-making process. However, these methods involve the datasets labeled with fine-grained attributes by expert knowledge, which incur high costs and introduce excessive human prior knowledge and bias. In this paper, we observe the long-tail distribution of concepts, based on which we propose a two-stage Concept Selection Model (CSM) to mine core concepts without introducing any human priors. The concept greedy rough selection algorithm is applied to extract head concepts, and then the concept mask fine selection method performs the extraction of core concepts. Experiments show that our approach achieves comparable performance to end-to-end black-box models, and human evaluation demonstrates that the concepts discovered by our method are interpretable and comprehensible for humans.

Title: PraFFL: A Preference-Aware Scheme in Fair Federated Learning

• Authors: Rongguang Ye, Ming Tang
• Subjects: cs.LG, cs.CY, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.08973
• Pdf URL: https://arxiv.org/pdf/2404.08973
• Copy Paste: [[2404.08973]] PraFFL: A Preference-Aware Scheme in Fair Federated Learning(https://arxiv.org/abs/2404.08973)
• Keywords: federate, fair
• Abstract: Fairness in federated learning has emerged as a critical concern, aiming to develop an unbiased model for any special group (e.g., male or female) of sensitive features. However, there is a trade-off between model performance and fairness, i.e., improving fairness will decrease model performance. Existing approaches have characterized such a trade-off by introducing hyperparameters to quantify client's preferences for fairness and model performance. Nevertheless, these methods are limited to scenarios where each client has only a single pre-defined preference. In practical systems, each client may simultaneously have multiple preferences for the model performance and fairness. The key challenge is to design a method that allows the model to adapt to diverse preferences of each client in real time. To this end, we propose a Preference-aware scheme in Fair Federated Learning paradigm (called PraFFL). PraFFL can adaptively adjust the model based on each client's preferences to meet their needs. We theoretically prove that PraFFL can provide the optimal model for client's arbitrary preferences. Experimental results show that our proposed PraFFL outperforms five existing fair federated learning algorithms in terms of the model's capability in adapting to clients' different preferences.

Title: OOVs in the Spotlight: How to Inflect them?

• Authors: Tomáš Sourada, Jana Straková, Rudolf Rosa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08974
• Pdf URL: https://arxiv.org/pdf/2404.08974
• Copy Paste: [[2404.08974]] OOVs in the Spotlight: How to Inflect them?(https://arxiv.org/abs/2404.08974)
• Keywords: transformer
• Abstract: We focus on morphological inflection in out-of-vocabulary (OOV) conditions, an under-researched subtask in which state-of-the-art systems usually are less effective. We developed three systems: a retrograde model and two sequence-to-sequence (seq2seq) models based on LSTM and Transformer. For testing in OOV conditions, we automatically extracted a large dataset of nouns in the morphologically rich Czech language, with lemma-disjoint data splits, and we further manually annotated a real-world OOV dataset of neologisms. In the standard OOV conditions, Transformer achieves the best results, with increasing performance in ensemble with LSTM, the retrograde model and SIGMORPHON baselines. On the real-world OOV dataset of neologisms, the retrograde model outperforms all neural models. Finally, our seq2seq models achieve state-of-the-art results in 9 out of 16 languages from SIGMORPHON 2022 shared task data in the OOV evaluation (feature overlap) in the large data condition. We release the Czech OOV Inflection Dataset for rigorous evaluation in OOV conditions. Further, we release the inflection system with the seq2seq models as a ready-to-use Python library.

Title: RoNID: New Intent Discovery with Generated-Reliable Labels and Cluster-friendly Representations

• Authors: Shun Zhang, Chaoran Yan, Jian Yang, Changyu Ren, Jiaqi Bai, Tongliang Li, Zhoujun Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08977
• Pdf URL: https://arxiv.org/pdf/2404.08977
• Copy Paste: [[2404.08977]] RoNID: New Intent Discovery with Generated-Reliable Labels and Cluster-friendly Representations(https://arxiv.org/abs/2404.08977)
• Keywords: robust
• Abstract: New Intent Discovery (NID) strives to identify known and reasonably deduce novel intent groups in the open-world scenario. But current methods face issues with inaccurate pseudo-labels and poor representation learning, creating a negative feedback loop that degrades overall model performance, including accuracy and the adjusted rand index. To address the aforementioned challenges, we propose a Robust New Intent Discovery (RoNID) framework optimized by an EM-style method, which focuses on constructing reliable pseudo-labels and obtaining cluster-friendly discriminative representations. RoNID comprises two main modules: reliable pseudo-label generation module and cluster-friendly representation learning module. Specifically, the pseudo-label generation module assigns reliable synthetic labels by solving an optimal transport problem in the E-step, which effectively provides high-quality supervised signals for the input of the cluster-friendly representation learning module. To learn cluster-friendly representation with strong intra-cluster compactness and large inter-cluster separation, the representation learning module combines intra-cluster and inter-cluster contrastive learning in the M-step to feed more discriminative features into the generation module. RoNID can be performed iteratively to ultimately yield a robust model with reliable pseudo-labels and cluster-friendly representations. Experimental results on multiple benchmarks demonstrate our method brings substantial improvements over previous state-of-the-art methods by a large margin of +1~+4 points.

Title: Incremental Residual Concept Bottleneck Models

• Authors: Chenming Shang, Shiji Zhou, Yujiu Yang, Hengyuan Zhang, Xinzhe Ni, Yuwang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08978
• Pdf URL: https://arxiv.org/pdf/2404.08978
• Copy Paste: [[2404.08978]] Incremental Residual Concept Bottleneck Models(https://arxiv.org/abs/2404.08978)
• Keywords: large language model
• Abstract: Concept Bottleneck Models (CBMs) map the black-box visual representations extracted by deep neural networks onto a set of interpretable concepts and use the concepts to make predictions, enhancing the transparency of the decision-making process. Multimodal pre-trained models can match visual representations with textual concept embeddings, allowing for obtaining the interpretable concept bottleneck without the expertise concept annotations. Recent research has focused on the concept bank establishment and the high-quality concept selection. However, it is challenging to construct a comprehensive concept bank through humans or large language models, which severely limits the performance of CBMs. In this work, we propose the Incremental Residual Concept Bottleneck Model (Res-CBM) to address the challenge of concept completeness. Specifically, the residual concept bottleneck model employs a set of optimizable vectors to complete missing concepts, then the incremental concept discovery module converts the complemented vectors with unclear meanings into potential concepts in the candidate concept bank. Our approach can be applied to any user-defined concept bank, as a post-hoc processing method to enhance the performance of any CBMs. Furthermore, to measure the descriptive efficiency of CBMs, the Concept Utilization Efficiency (CUE) metric is proposed. Experiments show that the Res-CBM outperforms the current state-of-the-art methods in terms of both accuracy and efficiency and achieves comparable performance to black-box models across multiple datasets.

Title: Stability and Generalization in Free Adversarial Training

• Authors: Xiwei Cheng, Kexin Fu, Farzan Farnia
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.08980
• Pdf URL: https://arxiv.org/pdf/2404.08980
• Copy Paste: [[2404.08980]] Stability and Generalization in Free Adversarial Training(https://arxiv.org/abs/2404.08980)
• Keywords: attack, robust
• Abstract: While adversarial training methods have resulted in significant improvements in the deep neural nets' robustness against norm-bounded adversarial perturbations, their generalization performance from training samples to test data has been shown to be considerably worse than standard empirical risk minimization methods. Several recent studies seek to connect the generalization behavior of adversarially trained classifiers to various gradient-based min-max optimization algorithms used for their training. In this work, we study the generalization performance of adversarial training methods using the algorithmic stability framework. Specifically, our goal is to compare the generalization performance of the vanilla adversarial training scheme fully optimizing the perturbations at every iteration vs. the free adversarial training simultaneously optimizing the norm-bounded perturbations and classifier parameters. Our proven generalization bounds indicate that the free adversarial training method could enjoy a lower generalization gap between training and test samples due to the simultaneous nature of its min-max optimization algorithm. We perform several numerical experiments to evaluate the generalization performance of vanilla, fast, and free adversarial training methods. Our empirical findings also show the improved generalization performance of the free adversarial training method and further demonstrate that the better generalization result could translate to greater robustness against black-box attack schemes. The code is available at https://github.com/Xiwei-Cheng/Stability_FreeAT.

Title: Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning

• Authors: Yijiang Liu, Rongyu Zhang, Huanrui Yang, Kurt Keutzer, Yuan Du, Li Du, Shanghang Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08985
• Pdf URL: https://arxiv.org/pdf/2404.08985
• Copy Paste: [[2404.08985]] Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning(https://arxiv.org/abs/2404.08985)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated significant potential in performing multiple tasks in multimedia applications, ranging from content generation to interactive entertainment, and artistic creation. However, the diversity of downstream tasks in multitask scenarios presents substantial adaptation challenges for LLMs. While traditional methods often succumb to knowledge confusion on their monolithic dense models, Mixture-of-Experts (MoE) has been emerged as a promising solution with its sparse architecture for effective task decoupling. Inspired by the principles of human cognitive neuroscience, we design a novel framework \texttt{Intuition-MoR1E} that leverages the inherent semantic clustering of instances to mimic the human brain to deal with multitask, offering implicit guidance to router for optimized feature allocation. Moreover, we introduce cutting-edge Rank-1 Experts formulation designed to manage a spectrum of intuitions, demonstrating enhanced parameter efficiency and effectiveness in multitask LLM finetuning. Extensive experiments demonstrate that Intuition-MoR1E achieves superior efficiency and 2.15\% overall accuracy improvement across 14 public datasets against other state-of-the-art baselines.

Title: On the critical path to implant backdoors and the effectiveness of potential mitigation techniques: Early learnings from XZ

• Authors: Mario Lins, René Mayrhofer, Michael Roland, Daniel Hofer, Martin Schwaighofer
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.08987
• Pdf URL: https://arxiv.org/pdf/2404.08987
• Copy Paste: [[2404.08987]] On the critical path to implant backdoors and the effectiveness of potential mitigation techniques: Early learnings from XZ(https://arxiv.org/abs/2404.08987)
• Keywords: attack
• Abstract: An emerging supply-chain attack due to a backdoor in XZ Utils has been identified. The backdoor allows an attacker to run commands remotely on vulnerable servers utilizing SSH without prior authentication. We have started to collect available information with regards to this attack to discuss current mitigation strategies for such kinds of supply-chain attacks. This paper introduces the critical attack path of the XZ backdoor and provides an overview about potential mitigation techniques related to relevant stages of the attack path.

Title: Labeled Morphological Segmentation with Semi-Markov Models

• Authors: Ryan Cotterell, Thomas Müller, Alexander Fraser, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08997
• Pdf URL: https://arxiv.org/pdf/2404.08997
• Copy Paste: [[2404.08997]] Labeled Morphological Segmentation with Semi-Markov Models(https://arxiv.org/abs/2404.08997)
• Keywords: segmentation
• Abstract: We present labeled morphological segmentation, an alternative view of morphological processing that unifies several tasks. From an annotation standpoint, we additionally introduce a new hierarchy of morphotactic tagsets. Finally, we develop \modelname, a discriminative morphological segmentation system that, contrary to previous work, explicitly models morphotactics. We show that \textsc{chipmunk} yields improved performance on three tasks for all six languages: (i) morphological segmentation, (ii) stemming and (iii) morphological tag classification. On morphological segmentation, our method shows absolute improvements of 2--6 points $F_1$ over the baseline.

Title: Proof-of-Learning with Incentive Security

• Authors: Zishuo Zhao, Zhixuan Fang, Xuechao Wang, Yuan Zhou
• Subjects: cs.CR, cs.AI, cs.ET, cs.GT, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09005
• Pdf URL: https://arxiv.org/pdf/2404.09005
• Copy Paste: [[2404.09005]] Proof-of-Learning with Incentive Security(https://arxiv.org/abs/2404.09005)
• Keywords: secure, security, attack
• Abstract: Most concurrent blockchain systems rely heavily on the Proof-of-Work (PoW) or Proof-of-Stake (PoS) mechanisms for decentralized consensus and security assurance. However, the substantial energy expenditure stemming from computationally intensive yet meaningless tasks has raised considerable concerns surrounding traditional PoW approaches, The PoS mechanism, while free of energy consumption, is subject to security and economic issues. Addressing these issues, the paradigm of Proof-of-Useful-Work (PoUW) seeks to employ challenges of practical significance as PoW, thereby imbuing energy consumption with tangible value. While previous efforts in Proof of Learning (PoL) explored the utilization of deep learning model training SGD tasks as PoUW challenges, recent research has revealed its vulnerabilities to adversarial attacks and the theoretical hardness in crafting a byzantine-secure PoL mechanism. In this paper, we introduce the concept of incentive-security that incentivizes rational provers to behave honestly for their best interest, bypassing the existing hardness to design a PoL mechanism with computational efficiency, a provable incentive-security guarantee and controllable difficulty. Particularly, our work is secure against two attacks to the recent work of Jia et al. [2021], and also improves the computational overhead from $\Theta(1)$ to $O(\frac{\log E}{E})$. Furthermore, while most recent research assumes trusted problem providers and verifiers, our design also guarantees frontend incentive-security even when problem providers are untrusted, and verifier incentive-security that bypasses the Verifier's Dilemma. By incorporating ML training into blockchain consensus mechanisms with provable guarantees, our research not only proposes an eco-friendly solution to blockchain systems, but also provides a proposal for a completely decentralized computing power market in the new AI age.

Title: MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild

• Authors: Kateryna Chumachenko, Alexandros Iosifidis, Moncef Gabbouj
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09010
• Pdf URL: https://arxiv.org/pdf/2404.09010
• Copy Paste: [[2404.09010]] MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild(https://arxiv.org/abs/2404.09010)
• Keywords: robust
• Abstract: Dynamic Facial Expression Recognition (DFER) has received significant interest in the recent years dictated by its pivotal role in enabling empathic and human-compatible technologies. Achieving robustness towards in-the-wild data in DFER is particularly important for real-world applications. One of the directions aimed at improving such models is multimodal emotion recognition based on audio and video data. Multimodal learning in DFER increases the model capabilities by leveraging richer, complementary data representations. Within the field of multimodal DFER, recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. Another line of research has focused on adapting pre-trained static models for DFER. In this work, we propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders. We identify main challenges associated with this task, namely, intra-modality adaptation, cross-modal alignment, and temporal adaptation, and propose solutions to each of them. As a result, we demonstrate improvement over current state-of-the-art on two popular DFER benchmarks, namely DFEW and MFAW.

Title: PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization

• Authors: Zining Chen, Weiqiu Wang, Zhicheng Zhao, Fei Su, Aidong Men, Hongying Meng
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09011
• Pdf URL: https://arxiv.org/pdf/2404.09011
• Copy Paste: [[2404.09011]] PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization(https://arxiv.org/abs/2404.09011)
• Keywords: robust
• Abstract: Domain Generalization (DG) aims to resolve distribution shifts between source and target domains, and current DG methods are default to the setting that data from source and target domains share identical categories. Nevertheless, there exists unseen classes from target domains in practical scenarios. To address this issue, Open Set Domain Generalization (OSDG) has emerged and several methods have been exclusively proposed. However, most existing methods adopt complex architectures with slight improvement compared with DG methods. Recently, vision-language models (VLMs) have been introduced in DG following the fine-tuning paradigm, but consume huge training overhead with large vision models. Therefore, in this paper, we innovate to transfer knowledge from VLMs to lightweight vision models and improve the robustness by introducing Perturbation Distillation (PD) from three perspectives, including Score, Class and Instance (SCI), named SCI-PD. Moreover, previous methods are oriented by the benchmarks with identical and fixed splits, ignoring the divergence between source domains. These methods are revealed to suffer from sharp performance decay with our proposed new benchmark Hybrid Domain Generalization (HDG) and a novel metric $H^{2}$-CV, which construct various splits to comprehensively assess the robustness of algorithms. Extensive experiments demonstrate that our method outperforms state-of-the-art algorithms on multiple datasets, especially improving the robustness when confronting data scarcity.

Title: Theoretical research on generative diffusion models: an overview

• Authors: Melike Nur Yeğin, Mehmet Fatih Amasyalı
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09016
• Pdf URL: https://arxiv.org/pdf/2404.09016
• Copy Paste: [[2404.09016]] Theoretical research on generative diffusion models: an overview(https://arxiv.org/abs/2404.09016)
• Keywords: diffusion, generative
• Abstract: Generative diffusion models showed high success in many fields with a powerful theoretical background. They convert the data distribution to noise and remove the noise back to obtain a similar distribution. Many existing reviews focused on the specific application areas without concentrating on the research about the algorithm. Unlike them we investigated the theoretical developments of the generative diffusion models. These approaches mainly divide into two: training-based and sampling-based. Awakening to this allowed us a clear and understandable categorization for the researchers who will make new developments in the future.

Title: Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies

• Authors: Benjue Weng
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09022
• Pdf URL: https://arxiv.org/pdf/2404.09022
• Copy Paste: [[2404.09022]] Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies(https://arxiv.org/abs/2404.09022)
• Keywords: large language model
• Abstract: With the surge of ChatGPT,the use of large models has significantly increased,rapidly rising to prominence across the industry and sweeping across the internet. This article is a comprehensive review of fine-tuning methods for large models. This paper investigates the latest technological advancements and the application of advanced methods in aspects such as task-adaptive fine-tuning,domain-adaptive fine-tuning,few-shot learning,knowledge distillation,multi-task learning,parameter-efficient fine-tuning,and dynamic fine-tuning.

Title: SQIAsignHD: SQIsignHD Adaptor Signature

• Authors: Farzin Renan, Péter Kutas
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09026
• Pdf URL: https://arxiv.org/pdf/2404.09026
• Copy Paste: [[2404.09026]] SQIAsignHD: SQIsignHD Adaptor Signature(https://arxiv.org/abs/2404.09026)
• Keywords: secure
• Abstract: Adaptor signatures can be viewed as a generalized form of the standard digital signature schemes where a secret randomness is hidden within a signature. Adaptor signatures are a recent cryptographic primitive and are becoming an important tool for blockchain applications such as cryptocurrencies to reduce on-chain costs, improve fungibility, and contribute to off-chain forms of payment in payment-channel networks, payment-channel hubs, and atomic swaps. However, currently used adaptor signature constructions are vulnerable to quantum adversaries due to Shor's algorithm. In this work, we introduce $\mathsf{SQIAsignHD}$, a new quantum-resistant adaptor signature scheme based on isogenies of supersingular elliptic curves, using SQIsignHD - as the underlying signature scheme - and exploiting the idea of the artificial orientation on the supersingular isogeny Diffie-Hellman key exchange protocol, SIDH, as the underlying hard relation. We, furthermore, show that our scheme is secure in the Quantum Random Oracle Model (QROM).

Title: MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts

• Authors: Yusheng Liao, Shuyang Jiang, Yu Wang, Yanfeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09027
• Pdf URL: https://arxiv.org/pdf/2404.09027
• Copy Paste: [[2404.09027]] MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts(https://arxiv.org/abs/2404.09027)
• Keywords: large language model
• Abstract: Large language models like ChatGPT have shown substantial progress in natural language understanding and generation, proving valuable across various disciplines, including the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks which often require multi-task learning capabilities. Previous approaches, although beneficial, fall short in real-world applications because they necessitate task-specific annotations at inference time, limiting broader generalization. This paper introduces MING-MOE, a novel Mixture-of-Expert~(MOE)-based medical large language model designed to manage diverse and complex medical tasks without requiring task-specific annotations, thus enhancing its usability across extensive datasets. MING-MOE employs a Mixture of Low-Rank Adaptation (MoLoRA) technique, allowing for efficient parameter usage by maintaining base model parameters static while adapting through a minimal set of trainable parameters. We demonstrate that MING-MOE achieves state-of-the-art (SOTA) performance on over 20 medical tasks, illustrating a significant improvement over existing models. This approach not only extends the capabilities of medical language models but also improves inference efficiency.

Title: Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation

• Authors: Jia Gu, Liang Pang, Huawei Shen, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09043
• Pdf URL: https://arxiv.org/pdf/2404.09043
• Copy Paste: [[2404.09043]] Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation(https://arxiv.org/abs/2404.09043)
• Keywords: large language model
• Abstract: With the rapid advancement of large language models (LLMs) and their remarkable capabilities in handling complex language tasks, an increasing number of studies are employing LLMs as agents to emulate the sequential decision-making processes of humans often represented as Markov decision-making processes (MDPs). The actions within this decision-making framework adhere to specific probability distributions and require iterative sampling. This arouses our curiosity regarding the capacity of LLM agents to comprehend probability distributions, thereby guiding the agent's behavioral decision-making through probabilistic sampling and generating behavioral sequences. To answer the above question, we divide the problem into two main aspects: simulation where the exact probability distribution is known, and generation of sequences where the probability distribution is ambiguous. In the first case, the agent is required to give the type and parameters of the probability distribution through the problem description, and then give the sampling sequence. However, our analysis shows that LLM agents perform poorly in this case, but the sampling success rate can be improved through programming tools. Real-world scenarios often entail unknown probability distributions. Thus, in the second case, we ask the agents to change the activity level in online social networks and analyze the frequency of actions. Ultimately, our analysis shows that LLM agents cannot sample probability distributions even using programming tools. Therefore, careful consideration is still required before directly applying LLM agents as agents to simulate human behavior.

Title: Adapting Mental Health Prediction Tasks for Cross-lingual Learning via Meta-Training and In-context Learning with Large Language Model

• Authors: Zita Lifelo, Huansheng Ning, Sahraoui Dhelim
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09045
• Pdf URL: https://arxiv.org/pdf/2404.09045
• Copy Paste: [[2404.09045]] Adapting Mental Health Prediction Tasks for Cross-lingual Learning via Meta-Training and In-context Learning with Large Language Model(https://arxiv.org/abs/2404.09045)
• Keywords: large language model
• Abstract: Timely identification is essential for the efficient handling of mental health illnesses such as depression. However, the current research fails to adequately address the prediction of mental health conditions from social media data in low-resource African languages like Swahili. This study introduces two distinct approaches utilising model-agnostic meta-learning and leveraging large language models (LLMs) to address this gap. Experiments are conducted on three datasets translated to low-resource language and applied to four mental health tasks, which include stress, depression, depression severity and suicidal ideation prediction. we first apply a meta-learning model with self-supervision, which results in improved model initialisation for rapid adaptation and cross-lingual transfer. The results show that our meta-trained model performs significantly better than standard fine-tuning methods, outperforming the baseline fine-tuning in macro F1 score with 18\% and 0.8\% over XLM-R and mBERT. In parallel, we use LLMs' in-context learning capabilities to assess their performance accuracy across the Swahili mental health prediction tasks by analysing different cross-lingual prompting approaches. Our analysis showed that Swahili prompts performed better than cross-lingual prompts but less than English prompts. Our findings show that in-context learning can be achieved through cross-lingual transfer through carefully crafted prompt templates with examples and instructions.

Title: Multilingual Evaluation of Semantic Textual Relatedness

• Authors: Sharvi Endait, Srushti Sonavane, Ridhima Sinare, Pritika Rohera, Advait Naik, Dipali Kadam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09047
• Pdf URL: https://arxiv.org/pdf/2404.09047
• Copy Paste: [[2404.09047]] Multilingual Evaluation of Semantic Textual Relatedness(https://arxiv.org/abs/2404.09047)
• Keywords: robust
• Abstract: The explosive growth of online content demands robust Natural Language Processing (NLP) techniques that can capture nuanced meanings and cultural context across diverse languages. Semantic Textual Relatedness (STR) goes beyond superficial word overlap, considering linguistic elements and non-linguistic factors like topic, sentiment, and perspective. Despite its pivotal role, prior NLP research has predominantly focused on English, limiting its applicability across languages. Addressing this gap, our paper dives into capturing deeper connections between sentences beyond simple word overlap. Going beyond English-centric NLP research, we explore STR in Marathi, Hindi, Spanish, and English, unlocking the potential for information retrieval, machine translation, and more. Leveraging the SemEval-2024 shared task, we explore various language models across three learning paradigms: supervised, unsupervised, and cross-lingual. Our comprehensive methodology gains promising results, demonstrating the effectiveness of our approach. This work aims to not only showcase our achievements but also inspire further research in multilingual STR, particularly for low-resourced languages.

Title: Rethinking Iterative Stereo Matching from Diffusion Bridge Model Perspective

• Authors: Yuguang Shi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09051
• Pdf URL: https://arxiv.org/pdf/2404.09051
• Copy Paste: [[2404.09051]] Rethinking Iterative Stereo Matching from Diffusion Bridge Model Perspective(https://arxiv.org/abs/2404.09051)
• Keywords: diffusion
• Abstract: Recently, iteration-based stereo matching has shown great potential. However, these models optimize the disparity map using RNN variants. The discrete optimization process poses a challenge of information loss, which restricts the level of detail that can be expressed in the generated disparity map. In order to address these issues, we propose a novel training approach that incorporates diffusion models into the iterative optimization process. We designed a Time-based Gated Recurrent Unit (T-GRU) to correlate temporal and disparity outputs. Unlike standard recurrent units, we employ Agent Attention to generate more expressive features. We also designed an attention-based context network to capture a large amount of contextual information. Experiments on several public benchmarks show that we have achieved competitive stereo matching performance. Our model ranks first in the Scene Flow dataset, achieving over a 7% improvement compared to competing methods, and requires only 8 iterations to achieve state-of-the-art results.

Title: Enhancing Security Awareness Through Gamified Approaches

• Authors: Yussuf Ahmed, Micheal Ezealor, Haitham Mahmoud, MohamedAjmal Azad, Mohamed BenFarah, Mehdi Yousefi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09052
• Pdf URL: https://arxiv.org/pdf/2404.09052
• Copy Paste: [[2404.09052]] Enhancing Security Awareness Through Gamified Approaches(https://arxiv.org/abs/2404.09052)
• Keywords: security
• Abstract: With the advent of smart grid (SG) systems, electricity networks have been able to ensure greater efficiency and utility by interconnecting their grids through cloud-based technology. As SGs become increasingly complex, a wide range of security challenges arise, threatening the grid's reliability, safety, efficiency, and stability. The security challenges include the potential exposure of personal data due to hackers intercepting the communications between the SG infrastructure and the smart meters. Security awareness plays a vital role in addressing some of these challenges. However, the traditional training programs are no longer efficient for instilling information security culture in organisations or from an individual user perspective. Gamification is a new concept in the field of information security awareness training (SAT) campaigns that can be introduced to fill in this gap by providing employees with a means of practising and learning about many security flaws and risks that exist within the organisation. Thus, this paper examines the effectiveness of gamification in promoting security awareness among smart meter components for smart grid users/operators. A gaming application is developed as part of the study with the aim of training and evaluating the results through three difficulty levels of questionnaires. Furthermore, the results are evaluated for the three difficulty levels as well as the overall flag captured. It can be demonstrated that the scores of participants in the three levels have improved by 40%, 35% and 29%, respectively. This reflects the awareness of learning within our system.

Title: ALICE: Combining Feature Selection and Inter-Rater Agreeability for Machine Learning Insights

• Authors: Bachana Anasashvili, Vahidin Jeleskovic
• Subjects: cs.LG, cs.HC, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.09053
• Pdf URL: https://arxiv.org/pdf/2404.09053
• Copy Paste: [[2404.09053]] ALICE: Combining Feature Selection and Inter-Rater Agreeability for Machine Learning Insights(https://arxiv.org/abs/2404.09053)
• Keywords: interpretability
• Abstract: This paper presents a new Python library called Automated Learning for Insightful Comparison and Evaluation (ALICE), which merges conventional feature selection and the concept of inter-rater agreeability in a simple, user-friendly manner to seek insights into black box Machine Learning models. The framework is proposed following an overview of the key concepts of interpretability in ML. The entire architecture and intuition of the main methods of the framework are also thoroughly discussed and results from initial experiments on a customer churn predictive modeling task are presented, alongside ideas for possible avenues to explore for the future. The full source code for the framework and the experiment notebooks can be found at: https://github.com/anasashb/aliceHU

Title: GView: A Versatile Assistant for Security Researchers

• Authors: Raul Zaharia, Dragoş Gavriluţ, Gheorghiţă Mutu, Dorel Lucanu
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.09058
• Pdf URL: https://arxiv.org/pdf/2404.09058
• Copy Paste: [[2404.09058]] GView: A Versatile Assistant for Security Researchers(https://arxiv.org/abs/2404.09058)
• Keywords: security, attack, extraction
• Abstract: Cyber security attacks have become increasingly complex over time, with various phases of their kill chain, involving binaries, scripts, documents, executed commands, vulnerabilities, or network traffic. We propose a tool, GView, that is designed to investigate possible attacks by providing guided analysis for various file types using automatic artifact identification, extraction, coherent correlation &,inference, and meaningful & intuitive views at different levels of granularity w.r.t. revealed information. The concept behind GView simplifies navigation through all payloads in a complex attack, streamlining the process for security researchers, and Increasing the quality of analysis. GView is generic in the sense it supports a variety of file types and has multiple visualization modes that can be automatically adjusted for each file type alone. Our evaluation shows that GView significantly improves the analysis time of an attack compared to conventional tools used in forensics.

Title: Exploring Explainability in Video Action Recognition

• Authors: Avinab Saha, Shashank Gupta, Sravan Kumar Ankireddy, Karl Chahine, Joydeep Ghosh
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09067
• Pdf URL: https://arxiv.org/pdf/2404.09067
• Copy Paste: [[2404.09067]] Exploring Explainability in Video Action Recognition(https://arxiv.org/abs/2404.09067)
• Keywords: explainability
• Abstract: Image Classification and Video Action Recognition are perhaps the two most foundational tasks in computer vision. Consequently, explaining the inner workings of trained deep neural networks is of prime importance. While numerous efforts focus on explaining the decisions of trained deep neural networks in image classification, exploration in the domain of its temporal version, video action recognition, has been scant. In this work, we take a deeper look at this problem. We begin by revisiting Grad-CAM, one of the popular feature attribution methods for Image Classification, and its extension to Video Action Recognition tasks and examine the method's limitations. To address these, we introduce Video-TCAV, by building on TCAV for Image Classification tasks, which aims to quantify the importance of specific concepts in the decision-making process of Video Action Recognition models. As the scalable generation of concepts is still an open problem, we propose a machine-assisted approach to generate spatial and spatiotemporal concepts relevant to Video Action Recognition for testing Video-TCAV. We then establish the importance of temporally-varying concepts by demonstrating the superiority of dynamic spatiotemporal concepts over trivial spatial concepts. In conclusion, we introduce a framework for investigating hypotheses in action recognition and quantitatively testing them, thus advancing research in the explainability of deep neural networks used in video action recognition.

Title: CuriousLLM: Elevating Multi-Document QA with Reasoning-Infused Knowledge Graph Prompting

• Authors: Zukang Yang, Zixuan Zhu
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09077
• Pdf URL: https://arxiv.org/pdf/2404.09077
• Copy Paste: [[2404.09077]] CuriousLLM: Elevating Multi-Document QA with Reasoning-Infused Knowledge Graph Prompting(https://arxiv.org/abs/2404.09077)
• Keywords: large language model
• Abstract: In the field of Question Answering (QA), unifying large language models (LLMs) with external databases has shown great success. However, these methods often fall short in providing the advanced reasoning needed for complex QA tasks. To address these issues, we improve over a novel approach called Knowledge Graph Prompting (KGP), which combines knowledge graphs with a LLM-based agent to improve reasoning and search accuracy. Nevertheless, the original KGP framework necessitates costly fine-tuning with large datasets yet still suffers from LLM hallucination. Therefore, we propose a reasoning-infused LLM agent to enhance this framework. This agent mimics human curiosity to ask follow-up questions to more efficiently navigate the search. This simple modification significantly boosts the LLM performance in QA tasks without the high costs and latency associated with the initial KGP framework. Our ultimate goal is to further develop this approach, leading to more accurate, faster, and cost-effective solutions in the QA domain.

Title: Probabilistic Directed Distance Fields for Ray-Based Shape Representations

• Authors: Tristan Aumentado-Armstrong, Stavros Tsogkas, Sven Dickinson, Allan Jepson
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09081
• Pdf URL: https://arxiv.org/pdf/2404.09081
• Copy Paste: [[2404.09081]] Probabilistic Directed Distance Fields for Ray-Based Shape Representations(https://arxiv.org/abs/2404.09081)
• Keywords: extraction, generative
• Abstract: In modern computer vision, the optimal representation of 3D shape continues to be task-dependent. One fundamental operation applied to such representations is differentiable rendering, as it enables inverse graphics approaches in learning frameworks. Standard explicit shape representations (voxels, point clouds, or meshes) are often easily rendered, but can suffer from limited geometric fidelity, among other issues. On the other hand, implicit representations (occupancy, distance, or radiance fields) preserve greater fidelity, but suffer from complex or inefficient rendering processes, limiting scalability. In this work, we devise Directed Distance Fields (DDFs), a novel neural shape representation that builds upon classical distance fields. The fundamental operation in a DDF maps an oriented point (position and direction) to surface visibility and depth. This enables efficient differentiable rendering, obtaining depth with a single forward pass per pixel, as well as differential geometric quantity extraction (e.g., surface normals), with only additional backward passes. Using probabilistic DDFs (PDDFs), we show how to model inherent discontinuities in the underlying field. We then apply DDFs to several applications, including single-shape fitting, generative modelling, and single-image 3D reconstruction, showcasing strong performance with simple architectural components via the versatility of our representation. Finally, since the dimensionality of DDFs permits view-dependent geometric artifacts, we conduct a theoretical investigation of the constraints necessary for view consistency. We find a small set of field properties that are sufficient to guarantee a DDF is consistent, without knowing, for instance, which shape the field is expressing.

Title: Exploring Generative AI for Sim2Real in Driving Data Synthesis

• Authors: Haonan Zhao, Yiting Wang, Thomas Bashford-Rogers, Valentina Donzella, Kurt Debattista
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09111
• Pdf URL: https://arxiv.org/pdf/2404.09111
• Copy Paste: [[2404.09111]] Exploring Generative AI for Sim2Real in Driving Data Synthesis(https://arxiv.org/abs/2404.09111)
• Keywords: diffusion, generative
• Abstract: Datasets are essential for training and testing vehicle perception algorithms. However, the collection and annotation of real-world images is time-consuming and expensive. Driving simulators offer a solution by automatically generating various driving scenarios with corresponding annotations, but the simulation-to-reality (Sim2Real) domain gap remains a challenge. While most of the Generative Artificial Intelligence (AI) follows the de facto Generative Adversarial Nets (GANs)-based methods, the recent emerging diffusion probabilistic models have not been fully explored in mitigating Sim2Real challenges for driving data synthesis. To explore the performance, this paper applied three different generative AI methods to leverage semantic label maps from a driving simulator as a bridge for the creation of realistic datasets. A comparative analysis of these methods is presented from the perspective of image quality and perception. New synthetic datasets, which include driving images and auto-generated high-quality annotations, are produced with low costs and high scene variability. The experimental results show that although GAN-based methods are adept at generating high-quality images when provided with manually annotated labels, ControlNet produces synthetic datasets with fewer artefacts and more structural fidelity when using simulator-generated labels. This suggests that the diffusion-based approach may provide improved stability and an alternative method for addressing Sim2Real challenges.

Title: GCC: Generative Calibration Clustering

• Authors: Haifeng Xia, Hai Huang, Zhengming Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09115
• Pdf URL: https://arxiv.org/pdf/2404.09115
• Copy Paste: [[2404.09115]] GCC: Generative Calibration Clustering(https://arxiv.org/abs/2404.09115)
• Keywords: diffusion, generative
• Abstract: Deep clustering as an important branch of unsupervised representation learning focuses on embedding semantically similar samples into the identical feature space. This core demand inspires the exploration of contrastive learning and subspace clustering. However, these solutions always rely on the basic assumption that there are sufficient and category-balanced samples for generating valid high-level representation. This hypothesis actually is too strict to be satisfied for real-world applications. To overcome such a challenge, the natural strategy is utilizing generative models to augment considerable instances. How to use these novel samples to effectively fulfill clustering performance improvement is still difficult and under-explored. In this paper, we propose a novel Generative Calibration Clustering (GCC) method to delicately incorporate feature learning and augmentation into clustering procedure. First, we develop a discriminative feature alignment mechanism to discover intrinsic relationship across real and generated samples. Second, we design a self-supervised metric learning to generate more reliable cluster assignment to boost the conditional diffusion generation. Extensive experimental results on three benchmarks validate the effectiveness and advantage of our proposed method over the state-of-the-art methods.

Title: Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation

• Authors: Ruixin Yang, Dheeraj Rajagopa, Shirley Anugrah Hayati, Bin Hu, Dongyeop Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09127
• Pdf URL: https://arxiv.org/pdf/2404.09127
• Copy Paste: [[2404.09127]] Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation(https://arxiv.org/abs/2404.09127)
• Keywords: generative, large language model
• Abstract: Uncertainty estimation is a significant issue for current large language models (LLMs) that are generally poorly calibrated and over-confident, especially with reinforcement learning from human feedback (RLHF). Unlike humans, whose decisions and confidences not only stem from intrinsic beliefs but can also be adjusted through daily observations, existing calibration methods for LLMs focus on estimating or eliciting individual confidence without taking full advantage of the "Collective Wisdom": the interaction among multiple LLMs that can collectively improve both accuracy and calibration. In this work, we propose Collaborative Calibration, a post-hoc training-free calibration strategy that leverages the collaborative and expressive capabilities of multiple tool-augmented LLM agents in a simulated group deliberation process. We demonstrate the effectiveness of Collaborative Calibration on generative QA tasks across various domains, showing its potential in harnessing the rationalization of collectively calibrated confidence assessments and improving the reliability of model predictions.

Title: When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models

• Authors: Yanhong Li, Chenghao Yang, Allyson Ettinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09129
• Pdf URL: https://arxiv.org/pdf/2404.09129
• Copy Paste: [[2404.09129]] When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models(https://arxiv.org/abs/2404.09129)
• Keywords: large language model
• Abstract: Recent studies suggest that self-reflective prompting can significantly enhance the reasoning capabilities of Large Language Models (LLMs). However, the use of external feedback as a stop criterion raises doubts about the true extent of LLMs' ability to emulate human-like self-reflection. In this paper, we set out to clarify these capabilities under a more stringent evaluation setting in which we disallow any kind of external feedback. Our findings under this setting show a split: while self-reflection enhances performance in TruthfulQA, it adversely affects results in HotpotQA. We conduct follow-up analyses to clarify the contributing factors in these patterns, and find that the influence of self-reflection is impacted both by reliability of accuracy in models' initial responses, and by overall question difficulty: specifically, self-reflection shows the most benefit when models are less likely to be correct initially, and when overall question difficulty is higher. We also find that self-reflection reduces tendency toward majority voting. Based on our findings, we propose guidelines for decisions on when to implement self-reflection. We release the codebase for reproducing our experiments at https://github.com/yanhong-lbh/LLM-SelfReflection-Eval.

Title: Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions

• Authors: Taojun Hu, Xiao-Hua Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09135
• Pdf URL: https://arxiv.org/pdf/2404.09135
• Copy Paste: [[2404.09135]] Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions(https://arxiv.org/abs/2404.09135)
• Keywords: large language model
• Abstract: Natural Language Processing (NLP) is witnessing a remarkable breakthrough driven by the success of Large Language Models (LLMs). LLMs have gained significant attention across academia and industry for their versatile applications in text generation, question answering, and text summarization. As the landscape of NLP evolves with an increasing number of domain-specific LLMs employing diverse techniques and trained on various corpus, evaluating performance of these models becomes paramount. To quantify the performance, it's crucial to have a comprehensive grasp of existing metrics. Among the evaluation, metrics which quantifying the performance of LLMs play a pivotal role. This paper offers a comprehensive exploration of LLM evaluation from a metrics perspective, providing insights into the selection and interpretation of metrics currently in use. Our main goal is to elucidate their mathematical formulations and statistical interpretations. We shed light on the application of these metrics using recent Biomedical LLMs. Additionally, we offer a succinct comparison of these metrics, aiding researchers in selecting appropriate metrics for diverse tasks. The overarching goal is to furnish researchers with a pragmatic guide for effective LLM evaluation and metric selection, thereby advancing the understanding and application of these large language models.

Title: From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation

• Authors: Artur Kiulian, Anton Polishko, Mykola Khandoga, Oryna Chubych, Jack Connor, Raghav Ravishankar, Adarsh Shirawalmath
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09138
• Pdf URL: https://arxiv.org/pdf/2404.09138
• Copy Paste: [[2404.09138]] From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation(https://arxiv.org/abs/2404.09138)
• Keywords: generative, large language model
• Abstract: In the rapidly advancing field of AI and NLP, generative large language models (LLMs) stand at the forefront of innovation, showcasing unparalleled abilities in text understanding and generation. However, the limited representation of low-resource languages like Ukrainian poses a notable challenge, restricting the reach and relevance of this technology. Our paper addresses this by fine-tuning the open-source Gemma and Mistral LLMs with Ukrainian datasets, aiming to improve their linguistic proficiency and benchmarking them against other existing models capable of processing Ukrainian language. This endeavor not only aims to mitigate language bias in technology but also promotes inclusivity in the digital realm. Our transparent and reproducible approach encourages further NLP research and development. Additionally, we present the Ukrainian Knowledge and Instruction Dataset (UKID) to aid future efforts in language model fine-tuning. Our research not only advances the field of NLP but also highlights the importance of linguistic diversity in AI, which is crucial for cultural preservation, education, and expanding AI's global utility. Ultimately, we advocate for a future where technology is inclusive, enabling AI to communicate effectively across all languages, especially those currently underrepresented.

Title: RF-Diffusion: Radio Signal Generation via Time-Frequency Diffusion

• Authors: Guoxuan Chi, Zheng Yang, Chenshu Wu, Jingao Xu, Yuchong Gao, Yunhao Liu, Tony Xiao Han
• Subjects: cs.LG, cs.IT, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.09140
• Pdf URL: https://arxiv.org/pdf/2404.09140
• Copy Paste: [[2404.09140]] RF-Diffusion: Radio Signal Generation via Time-Frequency Diffusion(https://arxiv.org/abs/2404.09140)
• Keywords: diffusion, transformer, generative
• Abstract: Along with AIGC shines in CV and NLP, its potential in the wireless domain has also emerged in recent years. Yet, existing RF-oriented generative solutions are ill-suited for generating high-quality, time-series RF data due to limited representation capabilities. In this work, inspired by the stellar achievements of the diffusion model in CV and NLP, we adapt it to the RF domain and propose RF-Diffusion. To accommodate the unique characteristics of RF signals, we first introduce a novel Time-Frequency Diffusion theory to enhance the original diffusion model, enabling it to tap into the information within the time, frequency, and complex-valued domains of RF signals. On this basis, we propose a Hierarchical Diffusion Transformer to translate the theory into a practical generative DNN through elaborated design spanning network architecture, functional block, and complex-valued operator, making RF-Diffusion a versatile solution to generate diverse, high-quality, and time-series RF data. Performance comparison with three prevalent generative models demonstrates the RF-Diffusion's superior performance in synthesizing Wi-Fi and FMCW signals. We also showcase the versatility of RF-Diffusion in boosting Wi-Fi sensing systems and performing channel estimation in 5G networks.

Title: ToNER: Type-oriented Named Entity Recognition with Generative Language Model

• Authors: Guochao Jiang, Ziqin Luo, Yuchen Shi, Dixuan Wang, Jiaqing Liang, Deqing Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09145
• Pdf URL: https://arxiv.org/pdf/2404.09145
• Copy Paste: [[2404.09145]] ToNER: Type-oriented Named Entity Recognition with Generative Language Model(https://arxiv.org/abs/2404.09145)
• Keywords: generative
• Abstract: In recent years, the fine-tuned generative models have been proven more powerful than the previous tagging-based or span-based models on named entity recognition (NER) task. It has also been found that the information related to entities, such as entity types, can prompt a model to achieve NER better. However, it is not easy to determine the entity types indeed existing in the given sentence in advance, and inputting too many potential entity types would distract the model inevitably. To exploit entity types' merit on promoting NER task, in this paper we propose a novel NER framework, namely ToNER based on a generative model. In ToNER, a type matching model is proposed at first to identify the entity types most likely to appear in the sentence. Then, we append a multiple binary classification task to fine-tune the generative model's encoder, so as to generate the refined representation of the input sentence. Moreover, we add an auxiliary task for the model to discover the entity types which further fine-tunes the model to output more accurate results. Our extensive experiments on some NER benchmarks verify the effectiveness of our proposed strategies in ToNER that are oriented towards entity types' exploitation.

Title: Fusion-Mamba for Cross-modality Object Detection

• Authors: Wenhao Dong, Haodong Zhu, Shaohui Lin, Xiaoyan Luo, Yunhang Shen, Xuhui Liu, Juan Zhang, Guodong Guo, Baochang Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09146
• Pdf URL: https://arxiv.org/pdf/2404.09146
• Copy Paste: [[2404.09146]] Fusion-Mamba for Cross-modality Object Detection(https://arxiv.org/abs/2404.09146)
• Keywords: robust
• Abstract: Cross-modality fusing complementary information from different modalities effectively improves object detection performance, making it more useful and robust for a wider range of applications. Existing fusion strategies combine different types of images or merge different backbone features through elaborated neural network modules. However, these methods neglect that modality disparities affect cross-modality fusion performance, as different modalities with different camera focal lengths, placements, and angles are hardly fused. In this paper, we investigate cross-modality fusion by associating cross-modal features in a hidden state space based on an improved Mamba with a gating mechanism. We design a Fusion-Mamba block (FMB) to map cross-modal features into a hidden state space for interaction, thereby reducing disparities between cross-modal features and enhancing the representation consistency of fused features. FMB contains two modules: the State Space Channel Swapping (SSCS) module facilitates shallow feature fusion, and the Dual State Space Fusion (DSSF) enables deep fusion in a hidden state space. Through extensive experiments on public datasets, our proposed approach outperforms the state-of-the-art methods on $m$AP with 5.9% on $M^3FD$ and 4.9% on FLIR-Aligned datasets, demonstrating superior object detection performance. To the best of our knowledge, this is the first work to explore the potential of Mamba for cross-modal fusion and establish a new baseline for cross-modality object detection.

Title: GeMQuAD : Generating Multilingual Question Answering Datasets from Large Language Models using Few Shot Learning

• Authors: Amani Namboori, Shivam Mangale, Andy Rosenbaum, Saleh Soltan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09163
• Pdf URL: https://arxiv.org/pdf/2404.09163
• Copy Paste: [[2404.09163]] GeMQuAD : Generating Multilingual Question Answering Datasets from Large Language Models using Few Shot Learning(https://arxiv.org/abs/2404.09163)
• Keywords: large language model
• Abstract: The emergence of Large Language Models (LLMs) with capabilities like In-Context Learning (ICL) has ushered in new possibilities for data generation across various domains while minimizing the need for extensive data collection and modeling techniques. Researchers have explored ways to use this generated synthetic data to optimize smaller student models for reduced deployment costs and lower latency in downstream tasks. However, ICL-generated data often suffers from low quality as the task specificity is limited with few examples used in ICL. In this paper, we propose GeMQuAD - a semi-supervised learning approach, extending the WeakDAP framework, applied to a dataset generated through ICL with just one example in the target language using AlexaTM 20B Seq2Seq LLM. Through our approach, we iteratively identify high-quality data to enhance model performance, especially for low-resource multilingual setting in the context of Extractive Question Answering task. Our framework outperforms the machine translation-augmented model by 0.22/1.68 F1/EM (Exact Match) points for Hindi and 0.82/1.37 F1/EM points for Spanish on the MLQA dataset, and it surpasses the performance of model trained on an English-only dataset by 5.05/6.50 F1/EM points for Hindi and 3.81/3.69 points F1/EM for Spanish on the same dataset. Notably, our approach uses a pre-trained LLM for generation with no fine-tuning (FT), utilizing just a single annotated example in ICL to generate data, providing a cost-effective development process.

Title: Post-Semantic-Thinking: A Robust Strategy to Distill Reasoning Capacity from Large Language Models

• Authors: Xiao Chen, Sihang Zhou, Ke Liang, Xinwang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09170
• Pdf URL: https://arxiv.org/pdf/2404.09170
• Copy Paste: [[2404.09170]] Post-Semantic-Thinking: A Robust Strategy to Distill Reasoning Capacity from Large Language Models(https://arxiv.org/abs/2404.09170)
• Keywords: robust, large language model
• Abstract: Chain of thought finetuning aims to endow small student models with reasoning capacity to improve their performance towards a specific task by allowing them to imitate the reasoning procedure of large language models (LLMs) beyond simply predicting the answer to the question. However, the existing methods 1) generate rationale before the answer, making their answer correctness sensitive to the hallucination in the rationale;2) force the student model to repeat the exact LLMs rationale expression word-after-word, which could have the model biased towards learning the expression in rationale but count against the model from understanding the core logic behind it. Therefore, we propose a robust Post-Semantic-Thinking (PST) strategy to generate answers before rationale. Thanks to this answer-first setting, 1) the answering procedure can escape from the adverse effects caused by hallucinations in the rationale; 2) the complex reasoning procedure is tightly bound with the relatively concise answer, making the reasoning for questions easier with the prior information in the answer; 3) the efficiency of the method can also benefit from the setting since users can stop the generation right after answers are outputted when inference is conducted. Furthermore, the PST strategy loose the constraint against the generated rationale to be close to the LLMs gold standard in the hidden semantic space instead of the vocabulary space, thus making the small student model better comprehend the semantic reasoning logic in rationale. Extensive experiments conducted across 12 reasoning tasks demonstrate the effectiveness of PST.

Title: LoopAnimate: Loopable Salient Object Animation

• Authors: Fanyi Wang, Peng Liu, Haotian Hu, Dan Meng, Jingwen Su, Jinjin Xu, Yanhao Zhang, Xiaoming Ren, Zhiwang Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09172
• Pdf URL: https://arxiv.org/pdf/2404.09172
• Copy Paste: [[2404.09172]] LoopAnimate: Loopable Salient Object Animation(https://arxiv.org/abs/2404.09172)
• Keywords: diffusion
• Abstract: Research on diffusion model-based video generation has advanced rapidly. However, limitations in object fidelity and generation length hinder its practical applications. Additionally, specific domains like animated wallpapers require seamless looping, where the first and last frames of the video match seamlessly. To address these challenges, this paper proposes LoopAnimate, a novel method for generating videos with consistent start and end frames. To enhance object fidelity, we introduce a framework that decouples multi-level image appearance and textual semantic information. Building upon an image-to-image diffusion model, our approach incorporates both pixel-level and feature-level information from the input image, injecting image appearance and textual semantic embeddings at different positions of the diffusion model. Existing UNet-based video generation models require to input the entire videos during training to encode temporal and positional information at once. However, due to limitations in GPU memory, the number of frames is typically restricted to 16. To address this, this paper proposes a three-stage training strategy with progressively increasing frame numbers and reducing fine-tuning modules. Additionally, we introduce the Temporal E nhanced Motion Module(TEMM) to extend the capacity for encoding temporal and positional information up to 36 frames. The proposed LoopAnimate, which for the first time extends the single-pass generation length of UNet-based video generation models to 35 frames while maintaining high-quality video generation. Experiments demonstrate that LoopAnimate achieves state-of-the-art performance in both objective metrics, such as fidelity and temporal consistency, and subjective evaluation results.

Title: TransformerFAM: Feedback attention is working memory

• Authors: Dongseong Hwang, Weiran Wang, Zhuoyuan Huo, Khe Chai Sim, Pedro Moreno Mengibar
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09173
• Pdf URL: https://arxiv.org/pdf/2404.09173
• Copy Paste: [[2404.09173]] TransformerFAM: Feedback attention is working memory(https://arxiv.org/abs/2404.09173)
• Keywords: transformer, large language model
• Abstract: While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences. TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models. Our experiments show that TransformerFAM significantly improves Transformer performance on long-context tasks across various model sizes (1B, 8B, and 24B). These results showcase the potential to empower Large Language Models (LLMs) to process sequences of unlimited length.

Title: HANet: A Hierarchical Attention Network for Change Detection With Bitemporal Very-High-Resolution Remote Sensing Images

• Authors: Chengxi Han, Chen Wu, Haonan Guo, Meiqi Hu, Hongruixuan Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09178
• Pdf URL: https://arxiv.org/pdf/2404.09178
• Copy Paste: [[2404.09178]] HANet: A Hierarchical Attention Network for Change Detection With Bitemporal Very-High-Resolution Remote Sensing Images(https://arxiv.org/abs/2404.09178)
• Keywords: extraction
• Abstract: Benefiting from the developments in deep learning technology, deep-learning-based algorithms employing automatic feature extraction have achieved remarkable performance on the change detection (CD) task. However, the performance of existing deep-learning-based CD methods is hindered by the imbalance between changed and unchanged pixels. To tackle this problem, a progressive foreground-balanced sampling strategy on the basis of not adding change information is proposed in this article to help the model accurately learn the features of the changed pixels during the early training process and thereby improve detection performance.Furthermore, we design a discriminative Siamese network, hierarchical attention network (HANet), which can integrate multiscale features and refine detailed features. The main part of HANet is the HAN module, which is a lightweight and effective self-attention mechanism. Extensive experiments and ablation studies on two CDdatasets with extremely unbalanced labels validate the effectiveness and efficiency of the proposed method.

Title: FaceCat: Enhancing Face Recognition Security with a Unified Generative Model Framework

• Authors: Jiawei Chen, Xiao Yang, Yinpeng Dong, Hang Su, Jianteng Peng, Zhaoxia Yin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09193
• Pdf URL: https://arxiv.org/pdf/2404.09193
• Copy Paste: [[2404.09193]] FaceCat: Enhancing Face Recognition Security with a Unified Generative Model Framework(https://arxiv.org/abs/2404.09193)
• Keywords: security, attack, robust, fair, generative
• Abstract: Face anti-spoofing (FAS) and adversarial detection (FAD) have been regarded as critical technologies to ensure the safety of face recognition systems. As a consequence of their limited practicality and generalization, some existing methods aim to devise a framework capable of concurrently detecting both threats to address the challenge. Nevertheless, these methods still encounter challenges of insufficient generalization and suboptimal robustness, potentially owing to the inherent drawback of discriminative models. Motivated by the rich structural and detailed features of face generative models, we propose FaceCat which utilizes the face generative model as a pre-trained model to improve the performance of FAS and FAD. Specifically, FaceCat elaborately designs a hierarchical fusion mechanism to capture rich face semantic features of the generative model. These features then serve as a robust foundation for a lightweight head, designed to execute FAS and FAD tasks simultaneously. As relying solely on single-modality data often leads to suboptimal performance, we further propose a novel text-guided multi-modal alignment strategy that utilizes text prompts to enrich feature representation, thereby enhancing performance. For fair evaluations, we build a comprehensive protocol with a wide range of 28 attack types to benchmark the performance. Extensive experiments validate the effectiveness of FaceCat generalizes significantly better and obtains excellent robustness against input transformations.

Title: TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models

• Authors: Ya-Qi Yu, Minghui Liao, Jihao Wu, Yongxin Liao, Xiaoyu Zheng, Wei Zeng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09204
• Pdf URL: https://arxiv.org/pdf/2404.09204
• Copy Paste: [[2404.09204]] TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models(https://arxiv.org/abs/2404.09204)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have shown impressive results on various multimodal tasks. However, most existing MLLMs are not well suited for document-oriented tasks, which require fine-grained image perception and information compression. In this paper, we present TextHawk, a MLLM that is specifically designed for document-oriented tasks, while preserving the general capabilities of MLLMs. TextHawk is aimed to explore efficient fine-grained perception by designing four dedicated components. Firstly, a ReSampling and ReArrangement (ReSA) module is proposed to reduce the redundancy in the document texts and lower the computational cost of the MLLM. We explore encoding the positions of each local feature by presenting Scalable Positional Embeddings (SPEs), which can preserve the scalability of various image sizes. A Query Proposal Network (QPN) is then adopted to initialize the queries dynamically among different sub-images. To further enhance the fine-grained visual perceptual ability of the MLLM, we design a Multi-Level Cross-Attention (MLCA) mechanism that captures the hierarchical structure and semantic relations of document images. Furthermore, we create a new instruction-tuning dataset for document-oriented tasks by enriching the multimodal document data with Gemini Pro. We conduct extensive experiments on both general and document-oriented MLLM benchmarks, and show that TextHawk outperforms the state-of-the-art methods, demonstrating its effectiveness and superiority in fine-grained document perception and general abilities.

Title: DKE-Research at SemEval-2024 Task 2: Incorporating Data Augmentation with Generative Models and Biomedical Knowledge to Enhance Inference Robustness

• Authors: Yuqi Wang, Zeqiang Wang, Wei Wang, Qi Chen, Kaizhu Huang, Anh Nguyen, Suparna De
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09206
• Pdf URL: https://arxiv.org/pdf/2404.09206
• Copy Paste: [[2404.09206]] DKE-Research at SemEval-2024 Task 2: Incorporating Data Augmentation with Generative Models and Biomedical Knowledge to Enhance Inference Robustness(https://arxiv.org/abs/2404.09206)
• Keywords: robust, generative
• Abstract: Safe and reliable natural language inference is critical for extracting insights from clinical trial reports but poses challenges due to biases in large pre-trained language models. This paper presents a novel data augmentation technique to improve model robustness for biomedical natural language inference in clinical trials. By generating synthetic examples through semantic perturbations and domain-specific vocabulary replacement and adding a new task for numerical and quantitative reasoning, we introduce greater diversity and reduce shortcut learning. Our approach, combined with multi-task learning and the DeBERTa architecture, achieved significant performance gains on the NLI4CT 2024 benchmark compared to the original language models. Ablation studies validate the contribution of each augmentation method in improving robustness. Our best-performing model ranked 12th in terms of faithfulness and 8th in terms of consistency, respectively, out of the 32 participants.

Title: DEGNN: Dual Experts Graph Neural Network Handling Both Edge and Node Feature Noise

• Authors: Tai Hasegawa, Sukwon Yun, Xin Liu, Yin Jun Phua, Tsuyoshi Murata
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09207
• Pdf URL: https://arxiv.org/pdf/2404.09207
• Copy Paste: [[2404.09207]] DEGNN: Dual Experts Graph Neural Network Handling Both Edge and Node Feature Noise(https://arxiv.org/abs/2404.09207)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have achieved notable success in various applications over graph data. However, recent research has revealed that real-world graphs often contain noise, and GNNs are susceptible to noise in the graph. To address this issue, several Graph Structure Learning (GSL) models have been introduced. While GSL models are tailored to enhance robustness against edge noise through edge reconstruction, a significant limitation surfaces: their high reliance on node features. This inherent dependence amplifies their susceptibility to noise within node features. Recognizing this vulnerability, we present DEGNN, a novel GNN model designed to adeptly mitigate noise in both edges and node features. The core idea of DEGNN is to design two separate experts: an edge expert and a node feature expert. These experts utilize self-supervised learning techniques to produce modified edges and node features. Leveraging these modified representations, DEGNN subsequently addresses downstream tasks, ensuring robustness against noise present in both edges and node features of real-world graphs. Notably, the modification process can be trained end-to-end, empowering DEGNN to adjust dynamically and achieves optimal edge and node representations for specific tasks. Comprehensive experiments demonstrate DEGNN's efficacy in managing noise, both in original real-world graphs and in graphs with synthetic noise.

Title: FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

• Authors: Changlin Song, Divya Saxena, Jiannong Cao, Yuqing Zhao
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09210
• Pdf URL: https://arxiv.org/pdf/2404.09210
• Copy Paste: [[2404.09210]] FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning(https://arxiv.org/abs/2404.09210)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) is a novel approach that allows for collaborative machine learning while preserving data privacy by leveraging models trained on decentralized devices. However, FL faces challenges due to non-uniformly distributed (non-iid) data across clients, which impacts model performance and its generalization capabilities. To tackle the non-iid issue, recent efforts have utilized the global model as a teaching mechanism for local models. However, our pilot study shows that their effectiveness is constrained by imbalanced data distribution, which induces biases in local models and leads to a 'local forgetting' phenomenon, where the ability of models to generalize degrades over time, particularly for underrepresented classes. This paper introduces FedDistill, a framework enhancing the knowledge transfer from the global model to local models, focusing on the issue of imbalanced class distribution. Specifically, FedDistill employs group distillation, segmenting classes based on their frequency in local datasets to facilitate a focused distillation process to classes with fewer samples. Additionally, FedDistill dissects the global model into a feature extractor and a classifier. This separation empowers local models with more generalized data representation capabilities and ensures more accurate classification across all classes. FedDistill mitigates the adverse effects of data imbalance, ensuring that local models do not forget underrepresented classes but instead become more adept at recognizing and classifying them accurately. Our comprehensive experiments demonstrate FedDistill's effectiveness, surpassing existing baselines in accuracy and convergence speed across several benchmark datasets.

Title: PrintListener: Uncovering the Vulnerability of Fingerprint Authentication via the Finger Friction Sound

• Authors: Man Zhou, Shuao Su, Qian Wang, Qi Li, Yuting Zhou, Xiaojing Ma, Zhengxiong Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09214
• Pdf URL: https://arxiv.org/pdf/2404.09214
• Copy Paste: [[2404.09214]] PrintListener: Uncovering the Vulnerability of Fingerprint Authentication via the Finger Friction Sound(https://arxiv.org/abs/2404.09214)
• Keywords: security, attack
• Abstract: Fingerprint authentication has been extensively employed in contemporary identity verification systems owing to its rapidity and cost-effectiveness. Due to its widespread use, fingerprint leakage may cause sensitive information theft, enormous economic and personnel losses, and even a potential compromise of national security. As a fingerprint that can coincidentally match a specific proportion of the overall fingerprint population, MasterPrint rings the alarm bells for the security of fingerprint authentication. In this paper, we propose a new side-channel attack on the minutiae-based Automatic Fingerprint Identification System (AFIS), called PrintListener, which leverages users' fingertip swiping actions on the screen to extract fingerprint pattern features (the first-level features) and synthesizes a stronger targeted PatternMasterPrint with potential second-level features. The attack scenario of PrintListener is extensive and covert. It only needs to record users' fingertip friction sound and can be launched by leveraging a large number of social media platforms. Extensive experimental results in realworld scenarios show that Printlistener can significantly improve the attack potency of MasterPrint.

Title: DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection

• Authors: Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, Dan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09216
• Pdf URL: https://arxiv.org/pdf/2404.09216
• Copy Paste: [[2404.09216]] DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection(https://arxiv.org/abs/2404.09216)
• Keywords: robust, generative, large language model
• Abstract: Existing open-vocabulary object detectors typically require a predefined set of categories from users, significantly confining their application scenarios. In this paper, we introduce DetCLIPv3, a high-performing detector that excels not only at both open-vocabulary object detection, but also generating hierarchical labels for detected objects. DetCLIPv3 is characterized by three core designs: 1. Versatile model architecture: we derive a robust open-set detection framework which is further empowered with generation ability via the integration of a caption head. 2. High information density data: we develop an auto-annotation pipeline leveraging visual large language model to refine captions for large-scale image-text pairs, providing rich, multi-granular object labels to enhance the training. 3. Efficient training strategy: we employ a pre-training stage with low-resolution inputs that enables the object captioner to efficiently learn a broad spectrum of visual concepts from extensive image-text paired data. This is followed by a fine-tuning stage that leverages a small number of high-resolution samples to further enhance detection performance. With these effective designs, DetCLIPv3 demonstrates superior open-vocabulary detection performance, \eg, our Swin-T backbone model achieves a notable 47.0 zero-shot fixed AP on the LVIS minival benchmark, outperforming GLIPv2, GroundingDINO, and DetCLIPv2 by 18.0/19.6/6.6 AP, respectively. DetCLIPv3 also achieves a state-of-the-art 19.7 AP in dense captioning task on VG dataset, showcasing its strong generative capability.

Title: Compass: Large Multilingual Language Model for South-east Asia

• Authors: Sophia Maria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09220
• Pdf URL: https://arxiv.org/pdf/2404.09220
• Copy Paste: [[2404.09220]] Compass: Large Multilingual Language Model for South-east Asia(https://arxiv.org/abs/2404.09220)
• Keywords: large language model
• Abstract: Large language models have exhibited significant proficiency in languages endowed with extensive linguistic resources, such as English and Chinese. Nevertheless, their effectiveness notably diminishes when applied to languages characterized by limited linguistic resources, particularly within the Southeast Asian linguistic landscape, such as Indonesian. The scarcity of linguistic resources for these languages presents challenges associated with inadequate training, restricted vocabulary coverage, and challenging evaluation processes. In response to these exigencies, we have introduced CompassLLM, a large multilingual model specifically tailored for Southeast Asian languages, with the primary aim of supporting the developmental requirements of Shopee. Our methodology encompasses several key strategies. To progressively enhance multilingual proficiencies, we implemented a multi-stage pre-training strategy integrated with curriculum learning, gradually intensifying the focus on low-resource languages. Concurrently, to better accommodate low-resource human instructions, we curated and generated a repository of high-quality multilingual human instructions, culminating the CompassLLM-SFT model through supervised instruction fine-tuning. Finally, to reinforce the model's alignment with human preference behaviors, we have embraced the principle of Direct Preference Optimization (DPO) to obtain CompassLLM-DPO model. Preliminary evaluation of the CompassLLM model yields promising results, with our model surpassing benchmark models like Vicuna-7b-v1.5, Sealion, Falcon and SeaLLM, across diverse evaluation tasks, as verified through both automated and human-driven assessments. Notably, our model exhibits its superior performance in South-east Asia languages, such as Indonesian language.

Title: DreamScape: 3D Scene Creation via Gaussian Splatting joint Correlation Modeling

• Authors: Xuening Yuan, Hongyu Yang, Yueming Zhao, Di Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09227
• Pdf URL: https://arxiv.org/pdf/2404.09227
• Copy Paste: [[2404.09227]] DreamScape: 3D Scene Creation via Gaussian Splatting joint Correlation Modeling(https://arxiv.org/abs/2404.09227)
• Keywords: diffusion, large language model
• Abstract: Recent progress in text-to-3D creation has been propelled by integrating the potent prior of Diffusion Models from text-to-image generation into the 3D domain. Nevertheless, generating 3D scenes characterized by multiple instances and intricate arrangements remains challenging. In this study, we present DreamScape, a method for creating highly consistent 3D scenes solely from textual descriptions, leveraging the strong 3D representation capabilities of Gaussian Splatting and the complex arrangement abilities of large language models (LLMs). Our approach involves a 3D Gaussian Guide ($3{DG^2}$) for scene representation, consisting of semantic primitives (objects) and their spatial transformations and relationships derived directly from text prompts using LLMs. This compositional representation allows for local-to-global optimization of the entire scene. A progressive scale control is tailored during local object generation, ensuring that objects of different sizes and densities adapt to the scene, which addresses training instability issue arising from simple blending in the subsequent global optimization stage. To mitigate potential biases of LLM priors, we model collision relationships between objects at the global level, enhancing physical correctness and overall realism. Additionally, to generate pervasive objects like rain and snow distributed extensively across the scene, we introduce a sparse initialization and densification strategy. Experiments demonstrate that DreamScape offers high usability and controllability, enabling the generation of high-fidelity 3D scenes from only text prompts and achieving state-of-the-art performance compared to other methods.

Title: Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms

• Authors: Diandian Guo, Manxi Lin, Jialun Pei, He Tang, Yueming Jin, Pheng-Ann Heng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09231
• Pdf URL: https://arxiv.org/pdf/2404.09231
• Copy Paste: [[2404.09231]] Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms(https://arxiv.org/abs/2404.09231)
• Keywords: large language model
• Abstract: A comprehensive understanding of surgical scenes allows for monitoring of the surgical process, reducing the occurrence of accidents and enhancing efficiency for medical professionals. Semantic modeling within operating rooms, as a scene graph generation (SGG) task, is challenging since it involves consecutive recognition of subtle surgical actions over prolonged periods. To address this challenge, we propose a Tri-modal (i.e., images, point clouds, and language) confluence with Temporal dynamics framework, termed TriTemp-OR. Diverging from previous approaches that integrated temporal information via memory graphs, our method embraces two advantages: 1) we directly exploit bi-modal temporal information from the video streaming for hierarchical feature interaction, and 2) the prior knowledge from Large Language Models (LLMs) is embedded to alleviate the class-imbalance problem in the operating theatre. Specifically, our model performs temporal interactions across 2D frames and 3D point clouds, including a scale-adaptive multi-view temporal interaction (ViewTemp) and a geometric-temporal point aggregation (PointTemp). Furthermore, we transfer knowledge from the biomedical LLM, LLaVA-Med, to deepen the comprehension of intraoperative relations. The proposed TriTemp-OR enables the aggregation of tri-modal features through relation-aware unification to predict relations so as to generate scene graphs. Experimental results on the 4D-OR benchmark demonstrate the superior performance of our model for long-term OR streaming.

Title: MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classes

• Authors: Xin-Chun Li, Shaoming Song, Yinchuan Li, Bingshuai Li, Yunfeng Shao, Yang Yang, De-Chuan Zhan
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.09232
• Pdf URL: https://arxiv.org/pdf/2404.09232
• Copy Paste: [[2404.09232]] MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classes(https://arxiv.org/abs/2404.09232)
• Keywords: federate
• Abstract: In some real-world applications, data samples are usually distributed on local devices, where federated learning (FL) techniques are proposed to coordinate decentralized clients without directly sharing users' private data. FL commonly follows the parameter server architecture and contains multiple personalization and aggregation procedures. The natural data heterogeneity across clients, i.e., Non-I.I.D. data, challenges both the aggregation and personalization goals in FL. In this paper, we focus on a special kind of Non-I.I.D. scene where clients own incomplete classes, i.e., each client can only access a partial set of the whole class set. The server aims to aggregate a complete classification model that could generalize to all classes, while the clients are inclined to improve the performance of distinguishing their observed classes. For better model aggregation, we point out that the standard softmax will encounter several problems caused by missing classes and propose "restricted softmax" as an alternative. For better model personalization, we point out that the hard-won personalized models are not well exploited and propose "inherited private model" to store the personalization experience. Our proposed algorithm named MAP could simultaneously achieve the aggregation and personalization goals in FL. Abundant experimental studies verify the superiorities of our algorithm.

Title: Fault Detection in Mobile Networks Using Diffusion Models

• Authors: Mohamad Nabeel, Doumitrou Daniil Nimara, Tahar Zanouda
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.09240
• Pdf URL: https://arxiv.org/pdf/2404.09240
• Copy Paste: [[2404.09240]] Fault Detection in Mobile Networks Using Diffusion Models(https://arxiv.org/abs/2404.09240)
• Keywords: diffusion, generative
• Abstract: In today's hyper-connected world, ensuring the reliability of telecom networks becomes increasingly crucial. Telecom networks encompass numerous underlying and intertwined software and hardware components, each providing different functionalities. To ensure the stability of telecom networks, telecom software, and hardware vendors developed several methods to detect any aberrant behavior in telecom networks and enable instant feedback and alerts. These approaches, although powerful, struggle to generalize due to the unsteady nature of the software-intensive embedded system and the complexity and diversity of multi-standard mobile networks. In this paper, we present a system to detect anomalies in telecom networks using a generative AI model. We evaluate several strategies using diffusion models to train the model for anomaly detection using multivariate time-series data. The contributions of this paper are threefold: (i) A proposal of a framework for utilizing diffusion models for time-series anomaly detection in telecom networks, (ii) A proposal of a particular Diffusion model architecture that outperforms other state-of-the-art techniques, (iii) Experiments on a real-world dataset to demonstrate that our model effectively provides explainable results, exposing some of its limitations and suggesting future research avenues to enhance its capabilities further.

Title: Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts

• Authors: Jing-Cheng Pang, Si-Hang Yang, Kaiyuan Li, Jiaji Zhang, Xiong-Hui Chen, Nan Tang, Yang Yu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09248
• Pdf URL: https://arxiv.org/pdf/2404.09248
• Copy Paste: [[2404.09248]] Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts(https://arxiv.org/abs/2404.09248)
• Keywords: large language model
• Abstract: Reinforcement learning (RL) trains agents to accomplish complex tasks through environmental interaction data, but its capacity is also limited by the scope of the available data. To obtain a knowledgeable agent, a promising approach is to leverage the knowledge from large language models (LLMs). Despite previous studies combining LLMs with RL, seamless integration of the two components remains challenging due to their semantic gap. This paper introduces a novel method, Knowledgeable Agents from Language Model Rollouts (KALM), which extracts knowledge from LLMs in the form of imaginary rollouts that can be easily learned by the agent through offline reinforcement learning methods. The primary challenge of KALM lies in LLM grounding, as LLMs are inherently limited to textual data, whereas environmental data often comprise numerical vectors unseen to LLMs. To address this, KALM fine-tunes the LLM to perform various tasks based on environmental data, including bidirectional translation between natural language descriptions of skills and their corresponding rollout data. This grounding process enhances the LLM's comprehension of environmental dynamics, enabling it to generate diverse and meaningful imaginary rollouts that reflect novel skills. Initial empirical evaluations on the CLEVR-Robot environment demonstrate that KALM enables agents to complete complex rephrasings of task goals and extend their capabilities to novel tasks requiring unprecedented optimal behaviors. KALM achieves a success rate of 46% in executing tasks with unseen goals, substantially surpassing the 26% success rate achieved by baseline methods. Furthermore, KALM effectively enables the LLM to comprehend environmental dynamics, resulting in the generation of meaningful imaginary rollouts that reflect novel skills and demonstrate the seamless integration of large language models and reinforcement learning.

Title: TEXT2TASTE: A Versatile Egocentric Vision System for Intelligent Reading Assistance Using Large Language Model

• Authors: Wiktor Mucha, Florin Cuconasu, Naome A. Etori, Valia Kalokyri, Giovanni Trappolini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09254
• Pdf URL: https://arxiv.org/pdf/2404.09254
• Copy Paste: [[2404.09254]] TEXT2TASTE: A Versatile Egocentric Vision System for Intelligent Reading Assistance Using Large Language Model(https://arxiv.org/abs/2404.09254)
• Keywords: robust, large language model
• Abstract: The ability to read, understand and find important information from written text is a critical skill in our daily lives for our independence, comfort and safety. However, a significant part of our society is affected by partial vision impairment, which leads to discomfort and dependency in daily activities. To address the limitations of this part of society, we propose an intelligent reading assistant based on smart glasses with embedded RGB cameras and a Large Language Model (LLM), whose functionality goes beyond corrective lenses. The video recorded from the egocentric perspective of a person wearing the glasses is processed to localise text information using object detection and optical character recognition methods. The LLM processes the data and allows the user to interact with the text and responds to a given query, thus extending the functionality of corrective lenses with the ability to find and summarize knowledge from the text. To evaluate our method, we create a chat-based application that allows the user to interact with the system. The evaluation is conducted in a real-world setting, such as reading menus in a restaurant, and involves four participants. The results show robust accuracy in text retrieval. The system not only provides accurate meal suggestions but also achieves high user satisfaction, highlighting the potential of smart glasses and LLMs in assisting people with special needs.

Title: Foundational GPT Model for MEG

• Authors: Richard Csaky, Mats W.J. van Es, Oiwi Parker Jones, Mark Woolrich
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.09256
• Pdf URL: https://arxiv.org/pdf/2404.09256
• Copy Paste: [[2404.09256]] Foundational GPT Model for MEG(https://arxiv.org/abs/2404.09256)
• Keywords: transformer
• Abstract: Deep learning techniques can be used to first training unsupervised models on large amounts of unlabelled data, before fine-tuning the models on specific tasks. This approach has seen massive success for various kinds of data, e.g. images, language, audio, and holds the promise of improving performance in various downstream tasks (e.g. encoding or decoding brain data). However, there has been limited progress taking this approach for modelling brain signals, such as Magneto-/electroencephalography (M/EEG). Here we propose two classes of deep learning foundational models that can be trained using forecasting of unlabelled MEG. First, we consider a modified Wavenet; and second, we consider a modified Transformer-based (GPT2) model. The modified GPT2 includes a novel application of tokenisation and embedding methods, allowing a model developed initially for the discrete domain of language to be applied to continuous multichannel time series data. We also extend the forecasting framework to include condition labels as inputs, enabling better modelling (encoding) of task data. We compare the performance of these deep learning models with standard linear autoregressive (AR) modelling on MEG data. This shows that GPT2-based models provide better modelling capabilities than Wavenet and linear AR models, by better reproducing the temporal, spatial and spectral characteristics of real data and evoked activity in task data. We show how the GPT2 model scales well to multiple subjects, while adapting its model to each subject through subject embedding. Finally, we show how such a model can be useful in downstream decoding tasks through data simulation. All code is available on GitHub (https://github.com/ricsinaruto/MEG-transfer-decoding).

Title: FedCCL: Federated Dual-Clustered Feature Contrast Under Domain Heterogeneity

• Authors: Yu Qiao, Huy Q. Le, Mengchun Zhang, Apurba Adhikary, Chaoning Zhang, Choong Seon Hong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09259
• Pdf URL: https://arxiv.org/pdf/2404.09259
• Copy Paste: [[2404.09259]] FedCCL: Federated Dual-Clustered Feature Contrast Under Domain Heterogeneity(https://arxiv.org/abs/2404.09259)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) facilitates a privacy-preserving neural network training paradigm through collaboration between edge clients and a central server. One significant challenge is that the distributed data is not independently and identically distributed (non-IID), typically including both intra-domain and inter-domain heterogeneity. However, recent research is limited to simply using averaged signals as a form of regularization and only focusing on one aspect of these non-IID challenges. Given these limitations, this paper clarifies these two non-IID challenges and attempts to introduce cluster representation to address them from both local and global perspectives. Specifically, we propose a dual-clustered feature contrast-based FL framework with dual focuses. First, we employ clustering on the local representations of each client, aiming to capture intra-class information based on these local clusters at a high level of granularity. Then, we facilitate cross-client knowledge sharing by pulling the local representation closer to clusters shared by clients with similar semantics while pushing them away from clusters with dissimilar semantics. Second, since the sizes of local clusters belonging to the same class may differ for each client, we further utilize clustering on the global side and conduct averaging to create a consistent global signal for guiding each local training in a contrastive manner. Experimental results on multiple datasets demonstrate that our proposal achieves comparable or superior performance gain under intra-domain and inter-domain heterogeneity.

Title: JaFIn: Japanese Financial Instruction Dataset

• Authors: Kota Tanabe, Masahiro Suzuki, Hiroki Sakaji, Itsuki Noda
• Subjects: cs.CL, cs.CE
• Abstract URL: https://arxiv.org/abs/2404.09260
• Pdf URL: https://arxiv.org/pdf/2404.09260
• Copy Paste: [[2404.09260]] JaFIn: Japanese Financial Instruction Dataset(https://arxiv.org/abs/2404.09260)
• Keywords: large language model
• Abstract: We construct an instruction dataset for the large language model (LLM) in the Japanese finance domain. Domain adaptation of language models, including LLMs, is receiving more attention as language models become more popular. This study demonstrates the effectiveness of domain adaptation through instruction tuning. To achieve this, we propose an instruction tuning data in Japanese called JaFIn, the Japanese Financial Instruction Dataset. JaFIn is manually constructed based on multiple data sources, including Japanese government websites, which provide extensive financial knowledge. We then utilize JaFIn to apply instruction tuning for several LLMs, demonstrating that our models specialized in finance have better domain adaptability than the original models. The financial-specialized LLMs created were evaluated using a quantitative Japanese financial benchmark and qualitative response comparisons, showing improved performance over the originals.

Title: Make Split, not Hijack: Preventing Feature-Space Hijacking Attacks in Split Learning

• Authors: Tanveer Khan, Mindaugas Budzys, Antonis Michalas
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09265
• Pdf URL: https://arxiv.org/pdf/2404.09265
• Copy Paste: [[2404.09265]] Make Split, not Hijack: Preventing Feature-Space Hijacking Attacks in Split Learning(https://arxiv.org/abs/2404.09265)
• Keywords: security, privacy, protect, attack
• Abstract: The popularity of Machine Learning (ML) makes the privacy of sensitive data more imperative than ever. Collaborative learning techniques like Split Learning (SL) aim to protect client data while enhancing ML processes. Though promising, SL has been proved to be vulnerable to a plethora of attacks, thus raising concerns about its effectiveness on data privacy. In this work, we introduce a hybrid approach combining SL and Function Secret Sharing (FSS) to ensure client data privacy. The client adds a random mask to the activation map before sending it to the servers. The servers cannot access the original function but instead work with shares generated using FSS. Consequently, during both forward and backward propagation, the servers cannot reconstruct the client's raw data from the activation map. Furthermore, through visual invertibility, we demonstrate that the server is incapable of reconstructing the raw image data from the activation map when using FSS. It enhances privacy by reducing privacy leakage compared to other SL-based approaches where the server can access client input information. Our approach also ensures security against feature space hijacking attack, protecting sensitive information from potential manipulation. Our protocols yield promising results, reducing communication overhead by over 2x and training time by over 7x compared to the same model with FSS, without any SL. Also, we show that our approach achieves >96% accuracy and remains equivalent to the plaintext models.

Title: Artificial Intelligence enhanced Security Problems in Real-Time Scenario using Blowfish Algorithm

• Authors: Yuvaraju Chinnam, Bosubabu Sambana
• Subjects: cs.CR, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.09286
• Pdf URL: https://arxiv.org/pdf/2404.09286
• Copy Paste: [[2404.09286]] Artificial Intelligence enhanced Security Problems in Real-Time Scenario using Blowfish Algorithm(https://arxiv.org/abs/2404.09286)
• Keywords: security
• Abstract: In a nutshell, "the cloud" refers to a collection of interconnected computing resources made possible by an extensive, real-time communication network like the internet. Because of its potential to reduce processing costs, the emerging paradigm of cloud computing has recently attracted a large number of academics. The exponential expansion of cloud computing has made the rapid expansion of cloud services very remarkable. Ensuring the security of personal information in today's interconnected world is no easy task. These days, security is really crucial. Models of security that are relevant to cloud computing include confidentiality, authenticity, accessibility, data integrity, and recovery. Using the Hybrid Encryption this study, we cover all the security issues and leaks in cloud infrastructure.

Title: New Class of Ciphers Using Hardware Entropy Source

• Authors: Jan J. Tatarkiewicz, Wieslaw B. Kuzmicz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09288
• Pdf URL: https://arxiv.org/pdf/2404.09288
• Copy Paste: [[2404.09288]] New Class of Ciphers Using Hardware Entropy Source(https://arxiv.org/abs/2404.09288)
• Keywords: attack
• Abstract: We present a novel, computationally simple method of hiding any message in the stream of random bits by using a secret key. The method is called Bury Among Random Numbers (BARN). A stream of random bits is produced by extracting the entropy of a physical process in a hardware-based true random number generator (TRNG). The process of placing bits of a message into the stream of random bits is governed by the number of random bits skipped between subsequent insertions. The set of numbers that correspond to the steps of BARN is derived from a random number also provided by TRNG. Hence BARN cipher does not depend on any arithmetic function. We propose an effective method of computing random keys from a given number of random bits. We estimate the number of permutations that need to be tested during a brute-force attack on the new cipher for various key lengths. Some practical applications for the new class of symmetrical ciphers are discussed.

Title: RoofDiffusion: Constructing Roofs from Severely Corrupted Point Data via Diffusion

• Authors: Kyle Shih-Huang Lo, Jörg Peters, Eric Spellman
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.09290
• Pdf URL: https://arxiv.org/pdf/2404.09290
• Copy Paste: [[2404.09290]] RoofDiffusion: Constructing Roofs from Severely Corrupted Point Data via Diffusion(https://arxiv.org/abs/2404.09290)
• Keywords: robust, diffusion
• Abstract: Accurate completion and denoising of roof height maps are crucial to reconstructing high-quality 3D buildings. Repairing sparse points can enhance low-cost sensor use and reduce UAV flight overlap. RoofDiffusion is a new end-to-end self-supervised diffusion technique for robustly completing, in particular difficult, roof height maps. RoofDiffusion leverages widely-available curated footprints and can so handle up to 99\% point sparsity and 80\% roof area occlusion (regional incompleteness). A variant, No-FP RoofDiffusion, simultaneously predicts building footprints and heights. Both quantitatively outperform state-of-the-art unguided depth completion and representative inpainting methods for Digital Elevation Models (DEM), on both a roof-specific benchmark and the BuildingNet dataset. Qualitative assessments show the effectiveness of RoofDiffusion for datasets with real-world scans including AHN3, Dales3D, and USGS 3DEP LiDAR. Tested with the leading City3D algorithm, preprocessing height maps with RoofDiffusion noticeably improves 3D building reconstruction. RoofDiffusion is complemented by a new dataset of 13k complex roof geometries, focusing on long-tail issues in remote sensing; a novel simulation of tree occlusion; and a wide variety of large-area roof cut-outs for data augmentation and benchmarking.

Title: Bridging Data Islands: Geographic Heterogeneity-Aware Federated Learning for Collaborative Remote Sensing Semantic Segmentation

• Authors: Jieyi Tan, Yansheng Li, Sergey A. Bartalev, Bo Dang, Wei Chen, Yongjun Zhang, Liangqi Yuan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09292
• Pdf URL: https://arxiv.org/pdf/2404.09292
• Copy Paste: [[2404.09292]] Bridging Data Islands: Geographic Heterogeneity-Aware Federated Learning for Collaborative Remote Sensing Semantic Segmentation(https://arxiv.org/abs/2404.09292)
• Keywords: privacy, federate, segmentation
• Abstract: Remote sensing semantic segmentation (RSS) is an essential task in Earth Observation missions. Due to data privacy concerns, high-quality remote sensing images with annotations cannot be well shared among institutions, making it difficult to fully utilize RSS data to train a generalized model. Federated Learning (FL), a privacy-preserving collaborative learning technology, is a potential solution. However, the current research on how to effectively apply FL in RSS is still scarce and requires further investigation. Remote sensing images in various institutions often exhibit strong geographical heterogeneity. More specifically, it is reflected in terms of class-distribution heterogeneity and object-appearance heterogeneity. Unfortunately, most existing FL studies show inadequate focus on geographical heterogeneity, thus leading to performance degradation in the global model. Considering the aforementioned issues, we propose a novel Geographic Heterogeneity-Aware Federated Learning (GeoFed) framework to address privacy-preserving RSS. Through Global Feature Extension and Tail Regeneration modules, class-distribution heterogeneity is alleviated. Additionally, we design an Essential Feature Mining strategy to alleviate object-appearance heterogeneity by constructing essential features. Extensive experiments on three datasets (i.e., FBP, CASID, Inria) show that our GeoFed consistently outperforms the current state-of-the-art methods. The code will be available publicly.

Title: Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A~Case~Study~at~HCMUT

• Authors: Tuan Bui, Oanh Tran, Phuong Nguyen, Bao Ho, Long Nguyen, Thang Bui, Tho Quan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09296
• Pdf URL: https://arxiv.org/pdf/2404.09296
• Copy Paste: [[2404.09296]] Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A~Case~Study~at~HCMUT(https://arxiv.org/abs/2404.09296)
• Keywords: large language model
• Abstract: In today's rapidly evolving landscape of Artificial Intelligence, large language models (LLMs) have emerged as a vibrant research topic. LLMs find applications in various fields and contribute significantly. Despite their powerful language capabilities, similar to pre-trained language models (PLMs), LLMs still face challenges in remembering events, incorporating new information, and addressing domain-specific issues or hallucinations. To overcome these limitations, researchers have proposed Retrieval-Augmented Generation (RAG) techniques, some others have proposed the integration of LLMs with Knowledge Graphs (KGs) to provide factual context, thereby improving performance and delivering more accurate feedback to user queries. Education plays a crucial role in human development and progress. With the technology transformation, traditional education is being replaced by digital or blended education. Therefore, educational data in the digital environment is increasing day by day. Data in higher education institutions are diverse, comprising various sources such as unstructured/structured text, relational databases, web/app-based API access, etc. Constructing a Knowledge Graph from these cross-data sources is not a simple task. This article proposes a method for automatically constructing a Knowledge Graph from multiple data sources and discusses some initial applications (experimental trials) of KG in conjunction with LLMs for question-answering tasks.

Title: In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognition

• Authors: Wiktor Mucha, Martin Kampel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09308
• Pdf URL: https://arxiv.org/pdf/2404.09308
• Copy Paste: [[2404.09308]] In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognition(https://arxiv.org/abs/2404.09308)
• Keywords: robust, transformer
• Abstract: Action recognition is essential for egocentric video understanding, allowing automatic and continuous monitoring of Activities of Daily Living (ADLs) without user effort. Existing literature focuses on 3D hand pose input, which requires computationally intensive depth estimation networks or wearing an uncomfortable depth sensor. In contrast, there has been insufficient research in understanding 2D hand pose for egocentric action recognition, despite the availability of user-friendly smart glasses in the market capable of capturing a single RGB image. Our study aims to fill this research gap by exploring the field of 2D hand pose estimation for egocentric action recognition, making two contributions. Firstly, we introduce two novel approaches for 2D hand pose estimation, namely EffHandNet for single-hand estimation and EffHandEgoNet, tailored for an egocentric perspective, capturing interactions between hands and objects. Both methods outperform state-of-the-art models on H2O and FPHA public benchmarks. Secondly, we present a robust action recognition architecture from 2D hand and object poses. This method incorporates EffHandEgoNet, and a transformer-based action recognition method. Evaluated on H2O and FPHA datasets, our architecture has a faster inference time and achieves an accuracy of 91.32% and 94.43%, respectively, surpassing state of the art, including 3D-based methods. Our work demonstrates that using 2D skeletal data is a robust approach for egocentric action understanding. Extensive evaluation and ablation studies show the impact of the hand pose estimation approach, and how each input affects the overall performance.

Title: Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers

• Authors: Diana-Nicoleta Grigore, Mariana-Iuliana Georgescu, Jon Alvarez Justo, Tor Johansen, Andreea Iuliana Ionescu, Radu Tudor Ionescu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09326
• Pdf URL: https://arxiv.org/pdf/2404.09326
• Copy Paste: [[2404.09326]] Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers(https://arxiv.org/abs/2404.09326)
• Keywords: transformer
• Abstract: Few-shot knowledge distillation recently emerged as a viable approach to harness the knowledge of large-scale pre-trained models, using limited data and computational resources. In this paper, we propose a novel few-shot feature distillation approach for vision transformers. Our approach is based on two key steps. Leveraging the fact that vision transformers have a consistent depth-wise structure, we first copy the weights from intermittent layers of existing pre-trained vision transformers (teachers) into shallower architectures (students), where the intermittence factor controls the complexity of the student transformer with respect to its teacher. Next, we employ an enhanced version of Low-Rank Adaptation (LoRA) to distill knowledge into the student in a few-shot scenario, aiming to recover the information processing carried out by the skipped teacher layers. We present comprehensive experiments with supervised and self-supervised transformers as teachers, on five data sets from various domains, including natural, medical and satellite images. The empirical results confirm the superiority of our approach over competitive baselines. Moreover, the ablation results demonstrate the usefulness of each component of the proposed pipeline.

Title: Large Language Models are as persuasive as humans, but why? About the cognitive effort and moral-emotional language of LLM arguments

• Authors: Carlos Carrasco-Farre
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09329
• Pdf URL: https://arxiv.org/pdf/2404.09329
• Copy Paste: [[2404.09329]] Large Language Models are as persuasive as humans, but why? About the cognitive effort and moral-emotional language of LLM arguments(https://arxiv.org/abs/2404.09329)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are already as persuasive as humans. However, we know very little about why. This paper investigates the persuasion strategies of LLMs, comparing them with human-generated arguments. Using a dataset of 1,251 participants in an experiment, we analyze the persuaion strategies of LLM-generated and human-generated arguments using measures of cognitive effort (lexical and grammatical complexity) and moral-emotional language (sentiment and moral analysis). The study reveals that LLMs produce arguments that require higher cognitive effort, exhibiting more complex grammatical and lexical structures than human counterparts. Additionally, LLMs demonstrate a significant propensity to engage more deeply with moral language, utilizing both positive and negative moral foundations more frequently than humans. In contrast with previous research, no significant difference was found in the emotional content produced by LLMs and humans. These findings contribute to the discourse on AI and persuasion, highlighting the dual potential of LLMs to both enhance and undermine informational integrity through communication strategies for digital persuasion.

Title: Self-Selected Attention Span for Accelerating Large Language Model Inference

• Authors: Tian Jin, Wanzin Yazar, Zifei Xu, Sayeh Sharify, Xin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09336
• Pdf URL: https://arxiv.org/pdf/2404.09336
• Copy Paste: [[2404.09336]] Self-Selected Attention Span for Accelerating Large Language Model Inference(https://arxiv.org/abs/2404.09336)
• Keywords: large language model
• Abstract: Large language models (LLMs) can solve challenging tasks. However, their inference computation on modern GPUs is highly inefficient due to the increasing number of tokens they must attend to as they generate new ones. To address this inefficiency, we capitalize on LLMs' problem-solving capabilities to optimize their own inference-time efficiency. We demonstrate with two specific tasks: (a) evaluating complex arithmetic expressions and (b) summarizing news articles. For both tasks, we create custom datasets to fine-tune an LLM. The goal of fine-tuning is twofold: first, to make the LLM learn to solve the evaluation or summarization task, and second, to train it to identify the minimal attention spans required for each step of the task. As a result, the fine-tuned model is able to convert these self-identified minimal attention spans into sparse attention masks on-the-fly during inference. We develop a custom CUDA kernel to take advantage of the reduced context to attend to. We demonstrate that using this custom CUDA kernel improves the throughput of LLM inference by 28%. Our work presents an end-to-end demonstration showing that training LLMs to self-select their attention spans speeds up autoregressive inference in solving real-world tasks.

Title: Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models

• Authors: Souvik Das, Lifeng Jin, Linfeng Song, Haitao Mi, Baolin Peng, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09338
• Pdf URL: https://arxiv.org/pdf/2404.09338
• Copy Paste: [[2404.09338]] Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models(https://arxiv.org/abs/2404.09338)
• Keywords: large language model
• Abstract: Large language models (LLMs) exhibit impressive natural language capabilities but suffer from hallucination -- generating content ungrounded in the realities of training data. Recent work has focused on decoding techniques to improve factuality during inference by leveraging LLMs' hierarchical representation of factual knowledge, manipulating the predicted distributions at inference time. Current state-of-the-art approaches refine decoding by contrasting early-exit distributions from a lower layer with the final layer to exploit information related to factuality within the model forward procedure. However, such methods often assume the final layer is the most reliable and the lower layer selection process depends on it. In this work, we first propose extrapolation of critical token probabilities beyond the last layer for more accurate contrasting. We additionally employ layer-wise entropy-guided lower layer selection, decoupling the selection process from the final layer. Experiments demonstrate strong performance - surpassing state-of-the-art on multiple different datasets by large margins. Analyses show different kinds of prompts respond to different selection strategies.

Title: Towards Practical Tool Usage for Continually Learning LLMs

• Authors: Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, Sarath Chandar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09339
• Pdf URL: https://arxiv.org/pdf/2404.09339
• Copy Paste: [[2404.09339]] Towards Practical Tool Usage for Continually Learning LLMs(https://arxiv.org/abs/2404.09339)
• Keywords: large language model
• Abstract: Large language models (LLMs) show an innate skill for solving language based tasks. But insights have suggested an inability to adjust for information or task-solving skills becoming outdated, as their knowledge, stored directly within their parameters, remains static in time. Tool use helps by offloading work to systems that the LLM can access through an interface, but LLMs that use them still must adapt to nonstationary environments for prolonged use, as new tools can emerge and existing tools can change. Nevertheless, tools require less specialized knowledge, therefore we hypothesize they are better suited for continual learning (CL) as they rely less on parametric memory for solving tasks and instead focus on learning when to apply pre-defined tools. To verify this, we develop a synthetic benchmark and follow this by aggregating existing NLP tasks to form a more realistic testing scenario. While we demonstrate scaling model size is not a solution, regardless of tool usage, continual learning techniques can enable tool LLMs to both adapt faster while forgetting less, highlighting their potential as continual learners.

Title: Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies

• Authors: Brian R. Bartoldson, James Diffenderfer, Konstantinos Parasyris, Bhavya Kailkhura
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09349
• Pdf URL: https://arxiv.org/pdf/2404.09349
• Copy Paste: [[2404.09349]] Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies(https://arxiv.org/abs/2404.09349)
• Keywords: attack, robust
• Abstract: This paper revisits the simple, long-studied, yet still unsolved problem of making image classifiers robust to imperceptible perturbations. Taking CIFAR10 as an example, SOTA clean accuracy is about $100$%, but SOTA robustness to $\ell_{\infty}$-norm bounded perturbations barely exceeds $70$%. To understand this gap, we analyze how model size, dataset size, and synthetic data quality affect robustness by developing the first scaling laws for adversarial training. Our scaling laws reveal inefficiencies in prior art and provide actionable feedback to advance the field. For instance, we discovered that SOTA methods diverge notably from compute-optimal setups, using excess compute for their level of robustness. Leveraging a compute-efficient setup, we surpass the prior SOTA with $20$% ($70$%) fewer training (inference) FLOPs. We trained various compute-efficient models, with our best achieving $74$% AutoAttack accuracy ($+3$% gain). However, our scaling laws also predict robustness slowly grows then plateaus at $90$%: dwarfing our new SOTA by scaling is impractical, and perfect robustness is impossible. To better understand this predicted limit, we carry out a small-scale human evaluation on the AutoAttack data that fools our top-performing model. Concerningly, we estimate that human performance also plateaus near $90$%, which we show to be attributable to $\ell_{\infty}$-constrained attacks' generation of invalid images not consistent with their original labels. Having characterized limiting roadblocks, we outline promising paths for future research.

Title: Counteracting Concept Drift by Learning with Future Malware Predictions

• Authors: Branislav Bosansky, Lada Hospodkova, Michal Najman, Maria Rigaki, Elnaz Babayeva, Viliam Lisy
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09352
• Pdf URL: https://arxiv.org/pdf/2404.09352
• Copy Paste: [[2404.09352]] Counteracting Concept Drift by Learning with Future Malware Predictions(https://arxiv.org/abs/2404.09352)
• Keywords: robust, generative
• Abstract: The accuracy of deployed malware-detection classifiers degrades over time due to changes in data distributions and increasing discrepancies between training and testing data. This phenomenon is known as the concept drift. While the concept drift can be caused by various reasons in general, new malicious files are created by malware authors with a clear intention of avoiding detection. The existence of the intention opens a possibility for predicting such future samples. Including predicted samples in training data should consequently increase the accuracy of the classifiers on new testing data. We compare two methods for predicting future samples: (1) adversarial training and (2) generative adversarial networks (GANs). The first method explicitly seeks for adversarial examples against the classifier that are then used as a part of training data. Similarly, GANs also generate synthetic training data. We use GANs to learn changes in data distributions within different time periods of training data and then apply these changes to generate samples that could be in testing data. We compare these prediction methods on two different datasets: (1) Ember public dataset and (2) the internal dataset of files incoming to Avast. We show that while adversarial training yields more robust classifiers, this method is not a good predictor of future malware in general. This is in contrast with previously reported positive results in different domains (including natural language processing and spam detection). On the other hand, we show that GANs can be successfully used as predictors of future malware. We specifically examine malware families that exhibit significant changes in their data distributions over time and the experimental results confirm that GAN-based predictions can significantly improve the accuracy of the classifier on new, previously unseen data.

Title: Hierarchical Attention Models for Multi-Relational Graphs

• Authors: Roshni G. Iyer, Wei Wang, Yizhou Sun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09365
• Pdf URL: https://arxiv.org/pdf/2404.09365
• Copy Paste: [[2404.09365]] Hierarchical Attention Models for Multi-Relational Graphs(https://arxiv.org/abs/2404.09365)
• Keywords: transformer
• Abstract: We present Bi-Level Attention-Based Relational Graph Convolutional Networks (BR-GCN), unique neural network architectures that utilize masked self-attentional layers with relational graph convolutions, to effectively operate on highly multi-relational data. BR-GCN models use bi-level attention to learn node embeddings through (1) node-level attention, and (2) relation-level attention. The node-level self-attentional layers use intra-relational graph interactions to learn relation-specific node embeddings using a weighted aggregation of neighborhood features in a sparse subgraph region. The relation-level self-attentional layers use inter-relational graph interactions to learn the final node embeddings using a weighted aggregation of relation-specific node embeddings. The BR-GCN bi-level attention mechanism extends Transformer-based multiplicative attention from the natural language processing (NLP) domain, and Graph Attention Networks (GAT)-based attention, to large-scale heterogeneous graphs (HGs). On node classification, BR-GCN outperforms baselines from 0.29% to 14.95% as a stand-alone model, and on link prediction, BR-GCN outperforms baselines from 0.02% to 7.40% as an auto-encoder model. We also conduct ablation studies to evaluate the quality of BR-GCN's relation-level attention and discuss how its learning of graph structure may be transferred to enrich other graph neural networks (GNNs). Through various experiments, we show that BR-GCN's attention mechanism is both scalable and more effective in learning compared to state-of-the-art GNNs.

Title: The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation

• Authors: Zoey Liu, Bonnie J. Dorr
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09371
• Pdf URL: https://arxiv.org/pdf/2404.09371
• Copy Paste: [[2404.09371]] The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation(https://arxiv.org/abs/2404.09371)
• Keywords: segmentation
• Abstract: Recent work to enhance data partitioning strategies for more realistic model evaluation face challenges in providing a clear optimal choice. This study addresses these challenges, focusing on morphological segmentation and synthesizing limitations related to language diversity, adoption of multiple datasets and splits, and detailed model comparisons. Our study leverages data from 19 languages, including ten indigenous or endangered languages across 10 language families with diverse morphological systems (polysynthetic, fusional, and agglutinative) and different degrees of data availability. We conduct large-scale experimentation with varying sized combinations of training and evaluation sets as well as new test data. Our results show that, when faced with new test data: (1) models trained from random splits are able to achieve higher numerical scores; (2) model rankings derived from random splits tend to generalize more consistently.

Title: \textit{sweet} -- An Open Source Modular Platform for Contactless Hand Vascular Biometric Experiments

• Authors: David Geissbühler, Sushil Bhattacharjee, Ketan Kotwal, Guillaume Clivaz, Sébastien Marcel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09376
• Pdf URL: https://arxiv.org/pdf/2404.09376
• Copy Paste: [[2404.09376]] \textit{sweet} -- An Open Source Modular Platform for Contactless Hand Vascular Biometric Experiments(https://arxiv.org/abs/2404.09376)
• Keywords: biometric
• Abstract: Current finger-vein or palm-vein recognition systems usually require direct contact of the subject with the apparatus. This can be problematic in environments where hygiene is of primary importance. In this work we present a contactless vascular biometrics sensor platform named \sweet which can be used for hand vascular biometrics studies (wrist-, palm- and finger-vein) and surface features such as palmprint. It supports several acquisition modalities such as multi-spectral Near-Infrared (NIR), RGB-color, Stereo Vision (SV) and Photometric Stereo (PS). Using this platform we collect a dataset consisting of the fingers, palm and wrist vascular data of 120 subjects and develop a powerful 3D pipeline for the pre-processing of this data. We then present biometric experimental results, focusing on Finger-Vein Recognition (FVR). Finally, we discuss fusion of multiple modalities, such palm-vein combined with palm-print biometrics. The acquisition software, parts of the hardware design, the new FV dataset, as well as source-code for our experiments are publicly available for research purposes.

Title: Orientation-conditioned Facial Texture Mapping for Video-based Facial Remote Photoplethysmography Estimation

• Authors: Sam Cantrill, David Ahmedt-Aristizabal, Lars Petersson, Hanna Suominen, Mohammad Ali Armin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09378
• Pdf URL: https://arxiv.org/pdf/2404.09378
• Copy Paste: [[2404.09378]] Orientation-conditioned Facial Texture Mapping for Video-based Facial Remote Photoplethysmography Estimation(https://arxiv.org/abs/2404.09378)
• Keywords: robust
• Abstract: Camera-based remote photoplethysmography (rPPG) enables contactless measurement of important physiological signals such as pulse rate (PR). However, dynamic and unconstrained subject motion introduces significant variability into the facial appearance in video, confounding the ability of video-based methods to accurately extract the rPPG signal. In this study, we leverage the 3D facial surface to construct a novel orientation-conditioned facial texture video representation which improves the motion robustness of existing video-based facial rPPG estimation methods. Our proposed method achieves a significant 18.2% performance improvement in cross-dataset testing on MMPD over our baseline using the PhysNet model trained on PURE, highlighting the efficacy and generalization benefits of our designed video representation. We demonstrate significant performance improvements of up to 29.6% in all tested motion scenarios in cross-dataset testing on MMPD, even in the presence of dynamic and unconstrained subject motion. Emphasizing the benefits the benefits of disentangling motion through modeling the 3D facial surface for motion robust facial rPPG estimation. We validate the efficacy of our design decisions and the impact of different video processing steps through an ablation study. Our findings illustrate the potential strengths of exploiting the 3D facial surface as a general strategy for addressing dynamic and unconstrained subject motion in videos. The code is available at https://samcantrill.github.io/orientation-uv-rppg/.

Title: Privacy at a Price: Exploring its Dual Impact on AI Fairness

• Authors: Mengmeng Yang, Ming Ding, Youyang Qu, Wei Ni, David Smith, Thierry Rakotoarivelo
• Subjects: cs.LG, cs.AI, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.09391
• Pdf URL: https://arxiv.org/pdf/2404.09391
• Copy Paste: [[2404.09391]] Privacy at a Price: Exploring its Dual Impact on AI Fairness(https://arxiv.org/abs/2404.09391)
• Keywords: privacy, protect, fair
• Abstract: The worldwide adoption of machine learning (ML) and deep learning models, particularly in critical sectors, such as healthcare and finance, presents substantial challenges in maintaining individual privacy and fairness. These two elements are vital to a trustworthy environment for learning systems. While numerous studies have concentrated on protecting individual privacy through differential privacy (DP) mechanisms, emerging research indicates that differential privacy in machine learning models can unequally impact separate demographic subgroups regarding prediction accuracy. This leads to a fairness concern, and manifests as biased performance. Although the prevailing view is that enhancing privacy intensifies fairness disparities, a smaller, yet significant, subset of research suggests the opposite view. In this article, with extensive evaluation results, we demonstrate that the impact of differential privacy on fairness is not monotonous. Instead, we observe that the accuracy disparity initially grows as more DP noise (enhanced privacy) is added to the ML process, but subsequently diminishes at higher privacy levels with even more noise. Moreover, implementing gradient clipping in the differentially private stochastic gradient descent ML method can mitigate the negative impact of DP noise on fairness. This mitigation is achieved by moderating the disparity growth through a lower clipping threshold.

Title: Data Analysis Methods Preliminaries for a Photon-based Hardware Random Number Generator

• Authors: Dmitriy Beznosko, Keith Driscoll, Fernando Guadarrama, Steven Mai, Nikolas Thornton
• Subjects: cs.CR, physics.ins-det
• Abstract URL: https://arxiv.org/abs/2404.09395
• Pdf URL: https://arxiv.org/pdf/2404.09395
• Copy Paste: [[2404.09395]] Data Analysis Methods Preliminaries for a Photon-based Hardware Random Number Generator(https://arxiv.org/abs/2404.09395)
• Keywords: security
• Abstract: High quality random numbers are necessary in the modern world. Ranging from encryption keys in cyber security to models and simulations for scientific use: it's important that these random numbers are of high quality and quickly attainable. One common solution to the generation of random numbers is that of pseudo-random number generators, or PRNGs. PRNGs generate random numbers by first quantifying some unpredictable phenomena into a number or string and feeding it into an algorithm which yields numbers randomly based on that seed. Easy places to find seeds include the user's mouse movements or the machine's uptime. These are only pseudorandom, however, as if given the same seed twice, the PRNG would generate the same 'random' output. This is great for games like Minecraft, but not so great for cybersecurity encryption key generation. By using a hardware random number generator (HRNG), random numbers that are not susceptible to the flaws found in PRNGs can be attained at a high rate.

Title: Watermark-embedded Adversarial Examples for Copyright Protection against Diffusion Models

• Authors: Peifei Zhu, Tsubasa Takahashi, Hirokatsu Kataoka
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09401
• Pdf URL: https://arxiv.org/pdf/2404.09401
• Copy Paste: [[2404.09401]] Watermark-embedded Adversarial Examples for Copyright Protection against Diffusion Models(https://arxiv.org/abs/2404.09401)
• Keywords: protect, attack, watermark, diffusion, generative
• Abstract: Diffusion Models (DMs) have shown remarkable capabilities in various image-generation tasks. However, there are growing concerns that DMs could be used to imitate unauthorized creations and thus raise copyright issues. To address this issue, we propose a novel framework that embeds personal watermarks in the generation of adversarial examples. Such examples can force DMs to generate images with visible watermarks and prevent DMs from imitating unauthorized images. We construct a generator based on conditional adversarial networks and design three losses (adversarial loss, GAN loss, and perturbation loss) to generate adversarial examples that have subtle perturbation but can effectively attack DMs to prevent copyright violations. Training a generator for a personal watermark by our method only requires 5-10 samples within 2-3 minutes, and once the generator is trained, it can generate adversarial examples with that watermark significantly fast (0.2s per image). We conduct extensive experiments in various conditional image-generation scenarios. Compared to existing methods that generate images with chaotic textures, our method adds visible watermarks on the generated images, which is a more straightforward way to indicate copyright violations. We also observe that our adversarial examples exhibit good transferability across unknown generative models. Therefore, this work provides a simple yet powerful way to protect copyright from DM-based imitation.

Title: Neural McKean-Vlasov Processes: Distributional Dependence in Diffusion Processes

• Authors: Haoming Yang, Ali Hasan, Yuting Ng, Vahid Tarokh
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.09402
• Pdf URL: https://arxiv.org/pdf/2404.09402
• Copy Paste: [[2404.09402]] Neural McKean-Vlasov Processes: Distributional Dependence in Diffusion Processes(https://arxiv.org/abs/2404.09402)
• Keywords: diffusion
• Abstract: McKean-Vlasov stochastic differential equations (MV-SDEs) provide a mathematical description of the behavior of an infinite number of interacting particles by imposing a dependence on the particle density. As such, we study the influence of explicitly including distributional information in the parameterization of the SDE. We propose a series of semi-parametric methods for representing MV-SDEs, and corresponding estimators for inferring parameters from data based on the properties of the MV-SDE. We analyze the characteristics of the different architectures and estimators, and consider their applicability in relevant machine learning problems. We empirically compare the performance of the different architectures and estimators on real and synthetic datasets for time series and probabilistic modeling. The results suggest that explicitly including distributional dependence in the parameterization of the SDE is effective in modeling temporal data with interaction under an exchangeability assumption while maintaining strong performance for standard It\^o-SDEs due to the richer class of probability flows associated with MV-SDEs.

Title: EQO: Exploring Ultra-Efficient Private Inference with Winograd-Based Protocol and Quantization Co-Optimization

• Authors: Wenxuan Zeng, Tianshi Xu, Meng Li, Runsheng Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09404
• Pdf URL: https://arxiv.org/pdf/2404.09404
• Copy Paste: [[2404.09404]] EQO: Exploring Ultra-Efficient Private Inference with Winograd-Based Protocol and Quantization Co-Optimization(https://arxiv.org/abs/2404.09404)
• Keywords: secure
• Abstract: Private convolutional neural network (CNN) inference based on secure two-party computation (2PC) suffers from high communication and latency overhead, especially from convolution layers. In this paper, we propose EQO, a quantized 2PC inference framework that jointly optimizes the CNNs and 2PC protocols. EQO features a novel 2PC protocol that combines Winograd transformation with quantization for efficient convolution computation. However, we observe naively combining quantization and Winograd convolution is sub-optimal: Winograd transformations introduce extensive local additions and weight outliers that increase the quantization bit widths and require frequent bit width conversions with non-negligible communication overhead. Therefore, at the protocol level, we propose a series of optimizations for the 2PC inference graph to minimize the communication. At the network level, We develop a sensitivity-based mixed-precision quantization algorithm to optimize network accuracy given communication constraints. We further propose a 2PC-friendly bit re-weighting algorithm to accommodate weight outliers without increasing bit widths. With extensive experiments, EQO demonstrates 11.7x, 3.6x, and 6.3x communication reduction with 1.29%, 1.16%, and 1.29% higher accuracy compared to state-of-the-art frameworks SiRNN, COINN, and CoPriv, respectively.

Title: Human-in-the-Loop Segmentation of Multi-species Coral Imagery

• Authors: Scarlett Raine, Ross Marchant, Brano Kusy, Frederic Maire, Niko Suenderhauf, Tobias Fischer
• Subjects: cs.CV, cs.HC, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.09406
• Pdf URL: https://arxiv.org/pdf/2404.09406
• Copy Paste: [[2404.09406]] Human-in-the-Loop Segmentation of Multi-species Coral Imagery(https://arxiv.org/abs/2404.09406)
• Keywords: segmentation
• Abstract: Broad-scale marine surveys performed by underwater vehicles significantly increase the availability of coral reef imagery, however it is costly and time-consuming for domain experts to label images. Point label propagation is an approach used to leverage existing image data labeled with sparse point labels. The resulting augmented ground truth generated is then used to train a semantic segmentation model. Here, we first demonstrate that recent advances in foundation models enable generation of multi-species coral augmented ground truth masks using denoised DINOv2 features and K-Nearest Neighbors (KNN), without the need for any pre-training or custom-designed algorithms. For extremely sparsely labeled images, we propose a labeling regime based on human-in-the-loop principles, resulting in significant improvement in annotation efficiency: If only 5 point labels per image are available, our proposed human-in-the-loop approach improves on the state-of-the-art by 17.3% for pixel accuracy and 22.6% for mIoU; and by 10.6% and 19.1% when 10 point labels per image are available. Even if the human-in-the-loop labeling regime is not used, the denoised DINOv2 features with a KNN outperforms the prior state-of-the-art by 3.5% for pixel accuracy and 5.7% for mIoU (5 grid points). We also provide a detailed analysis of how point labeling style and the quantity of points per image affects the point label propagation quality and provide general recommendations on maximizing point label efficiency.

Title: Wasserstein Wormhole: Scalable Optimal Transport Distance with Transformers

• Authors: Doron Haviv, Russell Zhang Kunes, Thomas Dougherty, Cassandra Burdziak, Tal Nawy, Anna Gilbert, Dana Pe'er
• Subjects: cs.LG, cs.CG, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2404.09411
• Pdf URL: https://arxiv.org/pdf/2404.09411
• Copy Paste: [[2404.09411]] Wasserstein Wormhole: Scalable Optimal Transport Distance with Transformers(https://arxiv.org/abs/2404.09411)
• Keywords: interpretability, transformer
• Abstract: Optimal transport (OT) and the related Wasserstein metric (W) are powerful and ubiquitous tools for comparing distributions. However, computing pairwise Wasserstein distances rapidly becomes intractable as cohort size grows. An attractive alternative would be to find an embedding space in which pairwise Euclidean distances map to OT distances, akin to standard multidimensional scaling (MDS). We present Wasserstein Wormhole, a transformer-based autoencoder that embeds empirical distributions into a latent space wherein Euclidean distances approximate OT distances. Extending MDS theory, we show that our objective function implies a bound on the error incurred when embedding non-Euclidean distances. Empirically, distances between Wormhole embeddings closely match Wasserstein distances, enabling linear time computation of OT distances. Along with an encoder that maps distributions to embeddings, Wasserstein Wormhole includes a decoder that maps embeddings back to distributions, allowing for operations in the embedding space to generalize to OT spaces, such as Wasserstein barycenter estimation and OT interpolation. By lending scalability and interpretability to OT approaches, Wasserstein Wormhole unlocks new avenues for data analysis in the fields of computational geometry and single-cell biology.

Title: Automatic Knowledge Graph Construction for Judicial Cases

• Authors: Jie Zhou, Xin Chen, Hang Zhang, Zhe Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09416
• Pdf URL: https://arxiv.org/pdf/2404.09416
• Copy Paste: [[2404.09416]] Automatic Knowledge Graph Construction for Judicial Cases(https://arxiv.org/abs/2404.09416)
• Keywords: robust, extraction
• Abstract: In this paper, we explore the application of cognitive intelligence in legal knowledge, focusing on the development of judicial artificial intelligence. Utilizing natural language processing (NLP) as the core technology, we propose a method for the automatic construction of case knowledge graphs for judicial cases. Our approach centers on two fundamental NLP tasks: entity recognition and relationship extraction. We compare two pre-trained models for entity recognition to establish their efficacy. Additionally, we introduce a multi-task semantic relationship extraction model that incorporates translational embedding, leading to a nuanced contextualized case knowledge representation. Specifically, in a case study involving a "Motor Vehicle Traffic Accident Liability Dispute," our approach significantly outperforms the baseline model. The entity recognition F1 score improved by 0.36, while the relationship extraction F1 score increased by 2.37. Building on these results, we detail the automatic construction process of case knowledge graphs for judicial cases, enabling the assembly of knowledge graphs for hundreds of thousands of judgments. This framework provides robust semantic support for applications of judicial AI, including the precise categorization and recommendation of related cases.

Title: ViFu: Multiple 360$^\circ$ Objects Reconstruction with Clean Background via Visible Part Fusion

• Authors: Tianhan Xu, Takuya Ikeda, Koichi Nishiwaki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09426
• Pdf URL: https://arxiv.org/pdf/2404.09426
• Copy Paste: [[2404.09426]] ViFu: Multiple 360$^\circ$ Objects Reconstruction with Clean Background via Visible Part Fusion(https://arxiv.org/abs/2404.09426)
• Keywords: segmentation
• Abstract: In this paper, we propose a method to segment and recover a static, clean background and multiple 360$^\circ$ objects from observations of scenes at different timestamps. Recent works have used neural radiance fields to model 3D scenes and improved the quality of novel view synthesis, while few studies have focused on modeling the invisible or occluded parts of the training images. These under-reconstruction parts constrain both scene editing and rendering view selection, thereby limiting their utility for synthetic data generation for downstream tasks. Our basic idea is that, by observing the same set of objects in various arrangement, so that parts that are invisible in one scene may become visible in others. By fusing the visible parts from each scene, occlusion-free rendering of both background and foreground objects can be achieved. We decompose the multi-scene fusion task into two main components: (1) objects/background segmentation and alignment, where we leverage point cloud-based methods tailored to our novel problem formulation; (2) radiance fields fusion, where we introduce visibility field to quantify the visible information of radiance fields, and propose visibility-aware rendering for the fusion of series of scenes, ultimately obtaining clean background and 360$^\circ$ object rendering. Comprehensive experiments were conducted on synthetic and real datasets, and the results demonstrate the effectiveness of our method.

Title: On the Efficiency of Privacy Attacks in Federated Learning

• Authors: Nawrin Tabassum, Ka-Ho Chow, Xuyu Wang, Wenbin Zhang, Yanzhao Wu
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09430
• Pdf URL: https://arxiv.org/pdf/2404.09430
• Copy Paste: [[2404.09430]] On the Efficiency of Privacy Attacks in Federated Learning(https://arxiv.org/abs/2404.09430)
• Keywords: privacy, attack, federate
• Abstract: Recent studies have revealed severe privacy risks in federated learning, represented by Gradient Leakage Attacks. However, existing studies mainly aim at increasing the privacy attack success rate and overlook the high computation costs for recovering private data, making the privacy attack impractical in real applications. In this study, we examine privacy attacks from the perspective of efficiency and propose a framework for improving the Efficiency of Privacy Attacks in Federated Learning (EPAFL). We make three novel contributions. First, we systematically evaluate the computational costs for representative privacy attacks in federated learning, which exhibits a high potential to optimize efficiency. Second, we propose three early-stopping techniques to effectively reduce the computational costs of these privacy attacks. Third, we perform experiments on benchmark datasets and show that our proposed method can significantly reduce computational costs and maintain comparable attack success rates for state-of-the-art privacy attacks in federated learning. We provide the codes on GitHub at https://github.com/mlsysx/EPAFL.

Title: Hybrid FedGraph: An efficient hybrid federated learning algorithm using graph convolutional neural network

• Authors: Jaeyeon Jang, Diego Klabjan, Veena Mendiratta, Fanfei Meng
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.09443
• Pdf URL: https://arxiv.org/pdf/2404.09443
• Copy Paste: [[2404.09443]] Hybrid FedGraph: An efficient hybrid federated learning algorithm using graph convolutional neural network(https://arxiv.org/abs/2404.09443)
• Keywords: privacy, federate
• Abstract: Federated learning is an emerging paradigm for decentralized training of machine learning models on distributed clients, without revealing the data to the central server. Most existing works have focused on horizontal or vertical data distributions, where each client possesses different samples with shared features, or each client fully shares only sample indices, respectively. However, the hybrid scheme is much less studied, even though it is much more common in the real world. Therefore, in this paper, we propose a generalized algorithm, FedGraph, that introduces a graph convolutional neural network to capture feature-sharing information while learning features from a subset of clients. We also develop a simple but effective clustering algorithm that aggregates features produced by the deep neural networks of each client while preserving data privacy.

Title: Exploring Text-to-Motion Generation with Human Preference

• Authors: Jenny Sheng, Matthieu Lin, Andrew Zhao, Kevin Pruvost, Yu-Hui Wen, Yangguang Li, Gao Huang, Yong-Jin Liu
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09445
• Pdf URL: https://arxiv.org/pdf/2404.09445
• Copy Paste: [[2404.09445]] Exploring Text-to-Motion Generation with Human Preference(https://arxiv.org/abs/2404.09445)
• Keywords: generative
• Abstract: This paper presents an exploration of preference learning in text-to-motion generation. We find that current improvements in text-to-motion generation still rely on datasets requiring expert labelers with motion capture systems. Instead, learning from human preference data does not require motion capture systems; a labeler with no expertise simply compares two generated motions. This is particularly efficient because evaluating the model's output is easier than gathering the motion that performs a desired task (e.g. backflip). To pioneer the exploration of this paradigm, we annotate 3,528 preference pairs generated by MotionGPT, marking the first effort to investigate various algorithms for learning from preference data. In particular, our exploration highlights important design choices when using preference data. Additionally, our experimental results show that preference learning has the potential to greatly improve current text-to-motion generative models. Our code and dataset are publicly available at https://github.com/THU-LYJ-Lab/InstructMotion}{https://github.com/THU-LYJ-Lab/InstructMotion to further facilitate research in this area.

Title: kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies

• Authors: Zhongrui Gui, Shuyang Sun, Runjia Li, Jianhao Yuan, Zhaochong An, Karsten Roth, Ameya Prabhu, Philip Torr
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09447
• Pdf URL: https://arxiv.org/pdf/2404.09447
• Copy Paste: [[2404.09447]] kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies(https://arxiv.org/abs/2404.09447)
• Keywords: segmentation
• Abstract: Rapid advancements in continual segmentation have yet to bridge the gap of scaling to large continually expanding vocabularies under compute-constrained scenarios. We discover that traditional continual training leads to catastrophic forgetting under compute constraints, unable to outperform zero-shot segmentation methods. We introduce a novel strategy for semantic and panoptic segmentation with zero forgetting, capable of adapting to continually growing vocabularies without the need for retraining or large memory costs. Our training-free approach, kNN-CLIP, leverages a database of instance embeddings to enable open-vocabulary segmentation approaches to continually expand their vocabulary on any given domain with a single-pass through data, while only storing embeddings minimizing both compute and memory costs. This method achieves state-of-the-art mIoU performance across large-vocabulary semantic and panoptic segmentation datasets. We hope kNN-CLIP represents a step forward in enabling more efficient and adaptable continual segmentation, paving the way for advances in real-world large-vocabulary continual segmentation methods.

Title: Crooked indifferentiability of the Feistel Construction

• Authors: Alexander Russell, Qiang Tang, Jiadong Zhu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09450
• Pdf URL: https://arxiv.org/pdf/2404.09450
• Copy Paste: [[2404.09450]] Crooked indifferentiability of the Feistel Construction(https://arxiv.org/abs/2404.09450)
• Keywords: security, attack
• Abstract: The Feistel construction is a fundamental technique for building pseudorandom permutations and block ciphers. This paper shows that a simple adaptation of the construction is resistant, even to algorithm substitution attacks -- that is, adversarial subversion -- of the component round functions. Specifically, we establish that a Feistel-based construction with more than $2000n/\log(1/\epsilon)$ rounds can transform a subverted random function -- which disagrees with the original one at a small fraction (denoted by $\epsilon$) of inputs -- into an object that is \emph{crooked-indifferentiable} from a random permutation, even if the adversary is aware of all the randomness used in the transformation. We also provide a lower bound showing that the construction cannot use fewer than $2n/\log(1/\epsilon)$ rounds to achieve crooked-indifferentiable security.

Title: Utility-Fairness Trade-Offs and How to Find Them

• Authors: Sepehr Dehdashtian, Bashir Sadeghi, Vishnu Naresh Boddeti
• Subjects: cs.CV, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09454
• Pdf URL: https://arxiv.org/pdf/2404.09454
• Copy Paste: [[2404.09454]] Utility-Fairness Trade-Offs and How to Find Them(https://arxiv.org/abs/2404.09454)
• Keywords: fair
• Abstract: When building classification systems with demographic fairness considerations, there are two objectives to satisfy: 1) maximizing utility for the specific task and 2) ensuring fairness w.r.t. a known demographic attribute. These objectives often compete, so optimizing both can lead to a trade-off between utility and fairness. While existing works acknowledge the trade-offs and study their limits, two questions remain unanswered: 1) What are the optimal trade-offs between utility and fairness? and 2) How can we numerically quantify these trade-offs from data for a desired prediction task and demographic attribute of interest? This paper addresses these questions. We introduce two utility-fairness trade-offs: the Data-Space and Label-Space Trade-off. The trade-offs reveal three regions within the utility-fairness plane, delineating what is fully and partially possible and impossible. We propose U-FaTE, a method to numerically quantify the trade-offs for a given prediction task and group fairness definition from data samples. Based on the trade-offs, we introduce a new scheme for evaluating representations. An extensive evaluation of fair representation learning methods and representations from over 1000 pre-trained models revealed that most current approaches are far from the estimated and achievable fairness-utility trade-offs across multiple datasets and prediction tasks.

Title: Improved Object-Based Style Transfer with Single Deep Network

• Authors: Harshmohan Kulkarni, Om Khare, Ninad Barve, Sunil Mane
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09461
• Pdf URL: https://arxiv.org/pdf/2404.09461
• Copy Paste: [[2404.09461]] Improved Object-Based Style Transfer with Single Deep Network(https://arxiv.org/abs/2404.09461)
• Keywords: segmentation
• Abstract: This research paper proposes a novel methodology for image-to-image style transfer on objects utilizing a single deep convolutional neural network. The proposed approach leverages the You Only Look Once version 8 (YOLOv8) segmentation model and the backbone neural network of YOLOv8 for style transfer. The primary objective is to enhance the visual appeal of objects in images by seamlessly transferring artistic styles while preserving the original object characteristics. The proposed approach's novelty lies in combining segmentation and style transfer in a single deep convolutional neural network. This approach omits the need for multiple stages or models, thus resulting in simpler training and deployment of the model for practical applications. The results of this approach are shown on two content images by applying different style images. The paper also demonstrates the ability to apply style transfer on multiple objects in the same image.

Title: PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI

• Authors: Yandan Yang, Baoxiong Jia, Peiyuan Zhi, Siyuan Huang
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.09465
• Pdf URL: https://arxiv.org/pdf/2404.09465
• Copy Paste: [[2404.09465]] PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI(https://arxiv.org/abs/2404.09465)
• Keywords: diffusion
• Abstract: With recent developments in Embodied Artificial Intelligence (EAI) research, there has been a growing demand for high-quality, large-scale interactive scene generation. While prior methods in scene synthesis have prioritized the naturalness and realism of the generated scenes, the physical plausibility and interactivity of scenes have been largely left unexplored. To address this disparity, we introduce PhyScene, a novel method dedicated to generating interactive 3D scenes characterized by realistic layouts, articulated objects, and rich physical interactivity tailored for embodied agents. Based on a conditional diffusion model for capturing scene layouts, we devise novel physics- and interactivity-based guidance mechanisms that integrate constraints from object collision, room layout, and object reachability. Through extensive experiments, we demonstrate that PhyScene effectively leverages these guidance functions for physically interactable scene synthesis, outperforming existing state-of-the-art scene synthesis methods by a large margin. Our findings suggest that the scenes generated by PhyScene hold considerable potential for facilitating diverse skill acquisition among agents within interactive environments, thereby catalyzing further advancements in embodied AI research. Project website: this http URL

Title: Q2A: Querying Implicit Fully Continuous Feature Pyramid to Align Features for Medical Image Segmentation

• Authors: Jiahao Yu, Li Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09472
• Pdf URL: https://arxiv.org/pdf/2404.09472
• Copy Paste: [[2404.09472]] Q2A: Querying Implicit Fully Continuous Feature Pyramid to Align Features for Medical Image Segmentation(https://arxiv.org/abs/2404.09472)
• Keywords: segmentation
• Abstract: Recent medical image segmentation methods apply implicit neural representation (INR) to the decoder for achieving a continuous coordinate decoding to tackle the drawback of conventional discrete grid-based data representations. However, the INR-based decoder cannot well handle the feature misalignment problem brought about by the naive latent code acquisition strategy in INR. Although there exist many feature alignment works, they all adopt a progressive multi-step aligning paradigm on a discrete feature pyramid, which is incompatible with the continuous one-step characteristics of INR-based decoder, and thus fails to be the solution. Therefore, we propose Q2A, a novel one-step query-based aligning paradigm, to solve the feature misalignment problem in the INR-based decoder. Specifically, for each target coordinate, Q2A first generates several queries depicting the spatial offsets and the cell resolutions of the contextual features aligned to the coordinate, then calculates the corresponding aligned features by feeding the queries into a novel implicit fully continuous feature pyramid (FCFP), finally fuses the aligned features to predict the class distribution. In FCFP, we further propose a novel universal partition-and-aggregate strategy (P&A) to replace the naive interpolation strategy for latent code acquisition in INR, which mitigates the information loss problem that occurs when the query cell resolution is relatively large and achieves an effective feature decoding at arbitrary continuous resolution. We conduct extensive experiments on two medical datasets, i.e. Glas and Synapse, and a universal dataset, i.e. Cityscapes, and they show the superiority of the proposed Q2A.

Title: TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals

• Authors: Alexander Vedernikov, Puneet Kumar, Haoyu Chen, Tapio Seppanen, Xiaobai Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09474
• Pdf URL: https://arxiv.org/pdf/2404.09474
• Copy Paste: [[2404.09474]] TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals(https://arxiv.org/abs/2404.09474)
• Keywords: transformer
• Abstract: Engagement analysis finds various applications in healthcare, education, advertisement, services. Deep Neural Networks, used for analysis, possess complex architecture and need large amounts of input data, computational power, inference time. These constraints challenge embedding systems into devices for real-time use. To address these limitations, we present a novel two-stream feature fusion "Tensor-Convolution and Convolution-Transformer Network" (TCCT-Net) architecture. To better learn the meaningful patterns in the temporal-spatial domain, we design a "CT" stream that integrates a hybrid convolutional-transformer. In parallel, to efficiently extract rich patterns from the temporal-frequency domain and boost processing speed, we introduce a "TC" stream that uses Continuous Wavelet Transform (CWT) to represent information in a 2D tensor form. Evaluated on the EngageNet dataset, the proposed method outperforms existing baselines, utilizing only two behavioral features (head pose rotations) compared to the 98 used in baseline models. Furthermore, comparative analysis shows TCCT-Net's architecture offers an order-of-magnitude improvement in inference speed compared to state-of-the-art image-based Recurrent Neural Network (RNN) methods. The code will be released at https://github.com/vedernikovphoto/TCCT_Net.

Title: SpamDam: Towards Privacy-Preserving and Adversary-Resistant SMS Spam Detection

• Authors: Yekai Li, Rufan Zhang, Wenxin Rong, Xianghang Mi
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09481
• Pdf URL: https://arxiv.org/pdf/2404.09481
• Copy Paste: [[2404.09481]] SpamDam: Towards Privacy-Preserving and Adversary-Resistant SMS Spam Detection(https://arxiv.org/abs/2404.09481)
• Keywords: privacy, attack, robust, steal, federate
• Abstract: In this study, we introduce SpamDam, a SMS spam detection framework designed to overcome key challenges in detecting and understanding SMS spam, such as the lack of public SMS spam datasets, increasing privacy concerns of collecting SMS data, and the need for adversary-resistant detection models. SpamDam comprises four innovative modules: an SMS spam radar that identifies spam messages from online social networks(OSNs); an SMS spam inspector for statistical analysis; SMS spam detectors(SSDs) that enable both central training and federated learning; and an SSD analyzer that evaluates model resistance against adversaries in realistic scenarios. Leveraging SpamDam, we have compiled over 76K SMS spam messages from Twitter and Weibo between 2018 and 2023, forming the largest dataset of its kind. This dataset has enabled new insights into recent spam campaigns and the training of high-performing binary and multi-label classifiers for spam detection. Furthermore, effectiveness of federated learning has been well demonstrated to enable privacy-preserving SMS spam detection. Additionally, we have rigorously tested the adversarial robustness of SMS spam detection models, introducing the novel reverse backdoor attack, which has shown effectiveness and stealthiness in practical tests.

Title: MMCode: Evaluating Multi-Modal Code Large Language Models with Visually Rich Programming Problems

• Authors: Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Jing Ma
• Subjects: cs.CL, cs.CV, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.09486
• Pdf URL: https://arxiv.org/pdf/2404.09486
• Copy Paste: [[2404.09486]] MMCode: Evaluating Multi-Modal Code Large Language Models with Visually Rich Programming Problems(https://arxiv.org/abs/2404.09486)
• Keywords: large language model
• Abstract: Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts. While recent developments in Large Multimodal Models have demonstrated remarkable abilities in visual reasoning and mathematical tasks, there is little work on investigating whether these models can effectively interpret visual elements for code generation. To this end, we present MMCode, the first multi-modal coding dataset for evaluating algorithmic problem-solving skills in visually rich contexts. MMCode contains 3,548 questions and 6,620 images collected from real-world programming challenges harvested from 10 code competition websites, presenting significant challenges due to the extreme demand for reasoning abilities. Our experiment results show that current state-of-the-art models struggle to solve these problems. The results highlight the lack of powerful vision-code models, and we hope MMCode can serve as an inspiration for future works in this domain. The data and code are publicly available at https://github.com/happylkx/MMCode.

Title: Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning

• Authors: Sungwon Han, Jinsung Yoon, Sercan O Arik, Tomas Pfister
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09491
• Pdf URL: https://arxiv.org/pdf/2404.09491
• Copy Paste: [[2404.09491]] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning(https://arxiv.org/abs/2404.09491)
• Keywords: large language model
• Abstract: Large Language Models (LLMs), with their remarkable ability to tackle challenging and unseen reasoning problems, hold immense potential for tabular learning, that is vital for many real-world applications. In this paper, we propose a novel in-context learning framework, FeatLLM, which employs LLMs as feature engineers to produce an input data set that is optimally suited for tabular predictions. The generated features are used to infer class likelihood with a simple downstream machine learning model, such as linear regression and yields high performance few-shot learning. The proposed FeatLLM framework only uses this simple predictive model with the discovered features at inference time. Compared to existing LLM-based approaches, FeatLLM eliminates the need to send queries to the LLM for each sample at inference time. Moreover, it merely requires API-level access to LLMs, and overcomes prompt size limitations. As demonstrated across numerous tabular datasets from a wide range of domains, FeatLLM generates high-quality rules, significantly (10% on average) outperforming alternatives such as TabLLM and STUNT.

Title: Bridging the Gap between Different Vocabularies for LLM Ensemble

• Authors: Yangyifan Xu, Jinliang Lu, Jiajun Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09492
• Pdf URL: https://arxiv.org/pdf/2404.09492
• Copy Paste: [[2404.09492]] Bridging the Gap between Different Vocabularies for LLM Ensemble(https://arxiv.org/abs/2404.09492)
• Keywords: large language model
• Abstract: Ensembling different large language models (LLMs) to unleash their complementary potential and harness their individual strengths is highly valuable. Nevertheless, vocabulary discrepancies among various LLMs have constrained previous studies to either selecting or blending completely generated outputs. This limitation hinders the dynamic correction and enhancement of outputs during the generation process, resulting in a limited capacity for effective ensemble. To address this issue, we propose a novel method to Ensemble LLMs via Vocabulary Alignment (EVA). EVA bridges the lexical gap among various LLMs, enabling meticulous ensemble at each generation step. Specifically, we first learn mappings between the vocabularies of different LLMs with the assistance of overlapping tokens. Subsequently, these mappings are employed to project output distributions of LLMs into a unified space, facilitating a fine-grained ensemble. Finally, we design a filtering strategy to exclude models that generate unfaithful tokens. Experimental results on commonsense reasoning, arithmetic reasoning, machine translation, and data-to-text generation tasks demonstrate the superiority of our approach compared with individual LLMs and previous ensemble methods conducted on complete outputs. Further analyses confirm that our approach can leverage knowledge from different language models and yield consistent improvement.

Title: On the Necessity of Collaboration in Online Model Selection with Decentralized Data

• Authors: Junfan Li, Zenglin Xu, Zheshun Wu, Irwin King
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09494
• Pdf URL: https://arxiv.org/pdf/2404.09494
• Copy Paste: [[2404.09494]] On the Necessity of Collaboration in Online Model Selection with Decentralized Data(https://arxiv.org/abs/2404.09494)
• Keywords: federate
• Abstract: We consider online model selection with decentralized data over $M$ clients, and study a fundamental problem: the necessity of collaboration. Previous work gave a negative answer from the perspective of worst-case regret minimization, while we give a different answer from the perspective of regret-computational cost trade-off. We separately propose a federated algorithm with and without communication constraint and prove regret bounds that show (i) collaboration is unnecessary if we do not limit the computational cost on each client; (ii) collaboration is necessary if we limit the computational cost on each client to $o(K)$, where $K$ is the number of candidate hypothesis spaces. As a by-product, we improve the regret bounds of algorithms for distributed online multi-kernel learning at a smaller computational and communication cost. Our algorithms rely on three new techniques, i.e., an improved Bernstein's inequality for martingale, a federated algorithmic framework, named FOMD-No-LU, and decoupling model selection and predictions, which might be of independent interest.

Title: FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba

• Authors: Xinyu Xie, Yawen Cui, Chio-In Ieong, Tao Tan, Xiaozhi Zhang, Xubin Zheng, Zitong Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09498
• Pdf URL: https://arxiv.org/pdf/2404.09498
• Copy Paste: [[2404.09498]] FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba(https://arxiv.org/abs/2404.09498)
• Keywords: transformer
• Abstract: Multi-modal image fusion aims to combine information from different modes to create a single image with comprehensive information and detailed textures. However, fusion models based on convolutional neural networks encounter limitations in capturing global image features due to their focus on local convolution operations. Transformer-based models, while excelling in global feature modeling, confront computational challenges stemming from their quadratic complexity. Recently, the Selective Structured State Space Model has exhibited significant potential for long-range dependency modeling with linear complexity, offering a promising avenue to address the aforementioned dilemma. In this paper, we propose FusionMamba, a novel dynamic feature enhancement method for multimodal image fusion with Mamba. Specifically, we devise an improved efficient Mamba model for image fusion, integrating efficient visual state space model with dynamic convolution and channel attention. This refined model not only upholds the performance of Mamba and global modeling capability but also diminishes channel redundancy while enhancing local enhancement capability. Additionally, we devise a dynamic feature fusion module (DFFM) comprising two dynamic feature enhancement modules (DFEM) and a cross modality fusion mamba module (CMFM). The former serves for dynamic texture enhancement and dynamic difference perception, whereas the latter enhances correlation features between modes and suppresses redundant intermodal information. FusionMamba has yielded state-of-the-art (SOTA) performance across various multimodal medical image fusion tasks (CT-MRI, PET-MRI, SPECT-MRI), infrared and visible image fusion task (IR-VIS) and multimodal biomedical image fusion dataset (GFP-PC), which is proved that our model has generalization ability. The code for FusionMamba is available at https://github.com/millieXie/FusionMamba.

Title: Learning Human Motion from Monocular Videos via Cross-Modal Manifold Alignment

• Authors: Shuaiying Hou, Hongyu Tao, Junheng Fang, Changqing Zou, Hujun Bao, Weiwei Xu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.09499
• Pdf URL: https://arxiv.org/pdf/2404.09499
• Copy Paste: [[2404.09499]] Learning Human Motion from Monocular Videos via Cross-Modal Manifold Alignment(https://arxiv.org/abs/2404.09499)
• Keywords: robust
• Abstract: Learning 3D human motion from 2D inputs is a fundamental task in the realms of computer vision and computer graphics. Many previous methods grapple with this inherently ambiguous task by introducing motion priors into the learning process. However, these approaches face difficulties in defining the complete configurations of such priors or training a robust model. In this paper, we present the Video-to-Motion Generator (VTM), which leverages motion priors through cross-modal latent feature space alignment between 3D human motion and 2D inputs, namely videos and 2D keypoints. To reduce the complexity of modeling motion priors, we model the motion data separately for the upper and lower body parts. Additionally, we align the motion data with a scale-invariant virtual skeleton to mitigate the interference of human skeleton variations to the motion priors. Evaluated on AIST++, the VTM showcases state-of-the-art performance in reconstructing 3D human motion from monocular videos. Notably, our VTM exhibits the capabilities for generalization to unseen view angles and in-the-wild videos.

Title: SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction

• Authors: Pin Tang, Zhongdao Wang, Guoqing Wang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, Chao Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09502
• Pdf URL: https://arxiv.org/pdf/2404.09502
• Copy Paste: [[2404.09502]] SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction(https://arxiv.org/abs/2404.09502)
• Keywords: transformer
• Abstract: Vision-based perception for autonomous driving requires an explicit modeling of a 3D space, where 2D latent representations are mapped and subsequent 3D operators are applied. However, operating on dense latent spaces introduces a cubic time and space complexity, which limits scalability in terms of perception range or spatial resolution. Existing approaches compress the dense representation using projections like Bird's Eye View (BEV) or Tri-Perspective View (TPV). Although efficient, these projections result in information loss, especially for tasks like semantic occupancy prediction. To address this, we propose SparseOcc, an efficient occupancy network inspired by sparse point cloud processing. It utilizes a lossless sparse latent representation with three key innovations. Firstly, a 3D sparse diffuser performs latent completion using spatially decomposed 3D sparse convolutional kernels. Secondly, a feature pyramid and sparse interpolation enhance scales with information from others. Finally, the transformer head is redesigned as a sparse variant. SparseOcc achieves a remarkable 74.9% reduction on FLOPs over the dense baseline. Interestingly, it also improves accuracy, from 12.8% to 14.1% mIOU, which in part can be attributed to the sparse representation's ability to avoid hallucinations on empty voxels.

Title: Learning Tracking Representations from Single Point Annotations

• Authors: Qiangqiang Wu, Antoni B. Chan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09504
• Pdf URL: https://arxiv.org/pdf/2404.09504
• Copy Paste: [[2404.09504]] Learning Tracking Representations from Single Point Annotations(https://arxiv.org/abs/2404.09504)
• Keywords: robust
• Abstract: Existing deep trackers are typically trained with largescale video frames with annotated bounding boxes. However, these bounding boxes are expensive and time-consuming to annotate, in particular for large scale datasets. In this paper, we propose to learn tracking representations from single point annotations (i.e., 4.5x faster to annotate than the traditional bounding box) in a weakly supervised manner. Specifically, we propose a soft contrastive learning (SoCL) framework that incorporates target objectness prior into end-to-end contrastive learning. Our SoCL consists of adaptive positive and negative sample generation, which is memory-efficient and effective for learning tracking representations. We apply the learned representation of SoCL to visual tracking and show that our method can 1) achieve better performance than the fully supervised baseline trained with box annotations under the same annotation time cost; 2) achieve comparable performance of the fully supervised baseline by using the same number of training frames and meanwhile reducing annotation time cost by 78% and total fees by 85%; 3) be robust to annotation noise.

Title: Magic Clothing: Controllable Garment-Driven Image Synthesis

• Authors: Weifeng Chen, Tao Gu, Yuhao Xu, Chengcai Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09512
• Pdf URL: https://arxiv.org/pdf/2404.09512
• Copy Paste: [[2404.09512]] Magic Clothing: Controllable Garment-Driven Image Synthesis(https://arxiv.org/abs/2404.09512)
• Keywords: robust, diffusion
• Abstract: We propose Magic Clothing, a latent diffusion model (LDM)-based network architecture for an unexplored garment-driven image synthesis task. Aiming at generating customized characters wearing the target garments with diverse text prompts, the image controllability is the most critical issue, i.e., to preserve the garment details and maintain faithfulness to the text prompts. To this end, we introduce a garment extractor to capture the detailed garment features, and employ self-attention fusion to incorporate them into the pretrained LDMs, ensuring that the garment details remain unchanged on the target character. Then, we leverage the joint classifier-free guidance to balance the control of garment features and text prompts over the generated results. Meanwhile, the proposed garment extractor is a plug-in module applicable to various finetuned LDMs, and it can be combined with other extensions like ControlNet and IP-Adapter to enhance the diversity and controllability of the generated characters. Furthermore, we design Matched-Points-LPIPS (MP-LPIPS), a robust metric for evaluating the consistency of the target image to the source garment. Extensive experiments demonstrate that our Magic Clothing achieves state-of-the-art results under various conditional controls for garment-driven image synthesis. Our source code is available at https://github.com/ShineChen1024/MagicClothing.

Title: Deep image learning of quantitative structure-property relationships of cooper alloys via feature augmentation on Geodesic curve in shape space

• Authors: Yuexing Han, Guanxin Wan, Bing Wang, Yi Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09515
• Pdf URL: https://arxiv.org/pdf/2404.09515
• Copy Paste: [[2404.09515]] Deep image learning of quantitative structure-property relationships of cooper alloys via feature augmentation on Geodesic curve in shape space(https://arxiv.org/abs/2404.09515)
• Keywords: generative
• Abstract: Understanding how the structure of materials affects their properties is a cornerstone of materials science and engineering. However, traditional methods have struggled to accurately describe the quantitative structure-property relationships for complex structures. In our study, we bridge this gap by leveraging machine learning to analyze images of materials' microstructures, thus offering a novel way to understand and predict the properties of materials based on their microstructures. We introduce a method known as FAGC (Feature Augmentation on Geodesic Curves), specifically demonstrated for Cu-Cr-Zr alloys. This approach utilizes machine learning to examine the shapes within images of the alloys' microstructures and predict their mechanical and electronic properties. This generative FAGC approach can effectively expand the relatively small training datasets due to the limited availability of materials images labeled with quantitative properties. The process begins with extracting features from the images using neural networks. These features are then mapped onto the Pre-shape space to construct the Geodesic curves. Along these curves, new features are generated, effectively increasing the dataset. Moreover, we design a pseudo-labeling mechanism for these newly generated features to further enhance the training dataset. Our FAGC method has shown remarkable results, significantly improving the accuracy of predicting the electronic conductivity and hardness of Cu-Cr-Zr alloys, with R-squared values of 0.978 and 0.998, respectively. These outcomes underscore the potential of FAGC to address the challenge of limited image data in materials science, providing a powerful tool for establishing detailed and quantitative relationships between complex microstructures and material properties.

Title: State Space Model for New-Generation Network Alternative to Transformers: A Survey

• Authors: Xiao Wang, Shiao Wang, Yuhe Ding, Yuehang Li, Wentao Wu, Yao Rong, Weizhe Kong, Ju Huang, Shihao Li, Haoxiang Yang, Ziwen Wang, Bo Jiang, Chenglong Li, Yaowei Wang, Yonghong Tian, Jin Tang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.09516
• Pdf URL: https://arxiv.org/pdf/2404.09516
• Copy Paste: [[2404.09516]] State Space Model for New-Generation Network Alternative to Transformers: A Survey(https://arxiv.org/abs/2404.09516)
• Keywords: transformer
• Abstract: In the post-deep learning era, the Transformer architecture has demonstrated its powerful performance across pre-trained big models and various downstream tasks. However, the enormous computational demands of this architecture have deterred many researchers. To further reduce the complexity of attention models, numerous efforts have been made to design more efficient methods. Among them, the State Space Model (SSM), as a possible replacement for the self-attention based Transformer model, has drawn more and more attention in recent years. In this paper, we give the first comprehensive review of these works and also provide experimental comparisons and analysis to better demonstrate the features and advantages of SSM. Specifically, we first give a detailed description of principles to help the readers quickly capture the key ideas of SSM. After that, we dive into the reviews of existing SSMs and their various applications, including natural language processing, computer vision, graph, multi-modal and multi-media, point cloud/event stream, time series data, and other domains. In addition, we give statistical comparisons and analysis of these models and hope it helps the readers to understand the effectiveness of different structures on various tasks. Then, we propose possible research points in this direction to better promote the development of the theoretical model and application of SSM. More related works will be continuously updated on the following GitHub: https://github.com/Event-AHU/Mamba_State_Space_Model_Paper_List.

Title: Bridging the Gap: Automated Analysis of Sancus

• Authors: Matteo Busi, Riccardo Focardi, Flaminia Luccio
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09518
• Pdf URL: https://arxiv.org/pdf/2404.09518
• Copy Paste: [[2404.09518]] Bridging the Gap: Automated Analysis of Sancus(https://arxiv.org/abs/2404.09518)
• Keywords: security, attack
• Abstract: Techniques for verifying or invalidating the security of computer systems have come a long way in recent years. Extremely sophisticated tools are available to specify and formally verify the behavior of a system and, at the same time, attack techniques have evolved to the point of questioning the possibility of obtaining adequate levels of security, especially in critical applications. In a recent paper, Bognar et al. have clearly highlighted this inconsistency between the two worlds: on one side, formal verification allows writing irrefutable proofs of the security of a system, on the other side concrete attacks make these proofs waver, exhibiting a gap between models and implementations which is very complex to bridge. In this paper, we propose a new method to reduce this gap in the Sancus embedded security architecture, by exploiting some peculiarities of both approaches. Our technique first extracts a behavioral model by directly interacting with the real Sancus system and then analyzes it to identify attacks and anomalies. Given a threat model, our method either finds attacks in the given threat model or gives probabilistic guarantees on the security of the system. We implement our method and use it to systematically rediscover known attacks and uncover new ones.

Title: Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learning

• Authors: Tidiane Camaret Ndir, André Biedenkapp, Noor Awad
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09521
• Pdf URL: https://arxiv.org/pdf/2404.09521
• Copy Paste: [[2404.09521]] Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learning(https://arxiv.org/abs/2404.09521)
• Keywords: robust
• Abstract: In this work, we address the challenge of zero-shot generalization (ZSG) in Reinforcement Learning (RL), where agents must adapt to entirely novel environments without additional training. We argue that understanding and utilizing contextual cues, such as the gravity level of the environment, is critical for robust generalization, and we propose to integrate the learning of context representations directly with policy learning. Our algorithm demonstrates improved generalization on various simulated domains, outperforming prior context-learning techniques in zero-shot settings. By jointly learning policy and context, our method acquires behavior-specific context representations, enabling adaptation to unseen environments and marks progress towards reinforcement learning systems that generalize across diverse real-world tasks. Our code and experiments are available at https://github.com/tidiane-camaret/contextual_rl_zero_shot.

Title: Dynamic fault detection and diagnosis of industrial alkaline water electrolyzer process with variational Bayesian dictionary learning

• Authors: Qi Zhang, Lei Xie, Weihua Xu, Hongye Su
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09524
• Pdf URL: https://arxiv.org/pdf/2404.09524
• Copy Paste: [[2404.09524]] Dynamic fault detection and diagnosis of industrial alkaline water electrolyzer process with variational Bayesian dictionary learning(https://arxiv.org/abs/2404.09524)
• Keywords: robust
• Abstract: Alkaline Water Electrolysis (AWE) is one of the simplest green hydrogen production method using renewable energy. AWE system typically yields process variables that are serially correlated and contaminated by measurement uncertainty. A novel robust dynamic variational Bayesian dictionary learning (RDVDL) monitoring approach is proposed to improve the reliability and safety of AWE operation. RDVDL employs a sparse Bayesian dictionary learning to preserve the dynamic mechanism information of AWE process which allows the easy interpretation of fault detection results. To improve the robustness to measurement uncertainty, a low-rank vector autoregressive (VAR) method is derived to reliably extract the serial correlation from process variables. The effectiveness of the proposed approach is demonstrated with an industrial hydrogen production process, and RDVDL can efficiently detect and diagnose critical AWE faults.

Title: Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models

• Authors: Siyan Zhao, Daniel Israel, Guy Van den Broeck, Aditya Grover
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09529
• Pdf URL: https://arxiv.org/pdf/2404.09529
• Copy Paste: [[2404.09529]] Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models(https://arxiv.org/abs/2404.09529)
• Keywords: transformer, large language model
• Abstract: During inference for transformer-based large language models (LLM), prefilling is the computation of the key-value (KV) cache for input tokens in the prompt prior to autoregressive generation. For longer input prompt lengths, prefilling will incur a significant overhead on decoding time. In this work, we highlight the following pitfall of prefilling: for batches containing high-varying prompt lengths, significant computation is wasted by the standard practice of padding sequences to the maximum length. As LLMs increasingly support longer context lengths, potentially up to 10 million tokens, variations in prompt lengths within a batch become more pronounced. To address this, we propose Prepacking, a simple yet effective method to optimize prefilling computation. To avoid redundant computation on pad tokens, prepacking combines prompts of varying lengths into a sequence and packs multiple sequences into a compact batch using a bin-packing algorithm. It then modifies the attention mask and positional encoding to compute multiple prefilled KV-caches for multiple prompts within a single sequence. On standard curated dataset containing prompts with varying lengths, we obtain a significant speed and memory efficiency improvements as compared to the default padding-based prefilling computation within Huggingface across a range of base model configurations and inference serving scenarios.

Title: RanLayNet: A Dataset for Document Layout Detection used for Domain Adaptation and Generalization

• Authors: Avinash Anand, Raj Jaiswal, Mohit Gupta, Siddhesh S Bangar, Pijush Bhuyan, Naman Lal, Rajeev Singh, Ritika Jha, Rajiv Ratn Shah, Shin'ichi Satoh
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09530
• Pdf URL: https://arxiv.org/pdf/2404.09530
• Copy Paste: [[2404.09530]] RanLayNet: A Dataset for Document Layout Detection used for Domain Adaptation and Generalization(https://arxiv.org/abs/2404.09530)
• Keywords: robust
• Abstract: Large ground-truth datasets and recent advances in deep learning techniques have been useful for layout detection. However, because of the restricted layout diversity of these datasets, training on them requires a sizable number of annotated instances, which is both expensive and time-consuming. As a result, differences between the source and target domains may significantly impact how well these models function. To solve this problem, domain adaptation approaches have been developed that use a small quantity of labeled data to adjust the model to the target domain. In this research, we introduced a synthetic document dataset called RanLayNet, enriched with automatically assigned labels denoting spatial positions, ranges, and types of layout elements. The primary aim of this endeavor is to develop a versatile dataset capable of training models with robustness and adaptability to diverse document formats. Through empirical experimentation, we demonstrate that a deep layout identification model trained on our dataset exhibits enhanced performance compared to a model trained solely on actual documents. Moreover, we conduct a comparative analysis by fine-tuning inference models using both PubLayNet and IIIT-AR-13K datasets on the Doclaynet dataset. Our findings emphasize that models enriched with our dataset are optimal for tasks such as achieving 0.398 and 0.588 mAP95 score in the scientific document domain for the TABLE class.

Title: TMPQ-DM: Joint Timestep Reduction and Quantization Precision Selection for Efficient Diffusion Models

• Authors: Haojun Sun, Chen Tang, Zhi Wang, Yuan Meng, Jingyan jiang, Xinzhu Ma, Wenwu Zhu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09532
• Pdf URL: https://arxiv.org/pdf/2404.09532
• Copy Paste: [[2404.09532]] TMPQ-DM: Joint Timestep Reduction and Quantization Precision Selection for Efficient Diffusion Models(https://arxiv.org/abs/2404.09532)
• Keywords: diffusion, generative
• Abstract: Diffusion models have emerged as preeminent contenders in the realm of generative models. Distinguished by their distinctive sequential generative processes, characterized by hundreds or even thousands of timesteps, diffusion models progressively reconstruct images from pure Gaussian noise, with each timestep necessitating full inference of the entire model. However, the substantial computational demands inherent to these models present challenges for deployment, quantization is thus widely used to lower the bit-width for reducing the storage and computing overheads. Current quantization methodologies primarily focus on model-side optimization, disregarding the temporal dimension, such as the length of the timestep sequence, thereby allowing redundant timesteps to continue consuming computational resources, leaving substantial scope for accelerating the generative process. In this paper, we introduce TMPQ-DM, which jointly optimizes timestep reduction and quantization to achieve a superior performance-efficiency trade-off, addressing both temporal and model optimization aspects. For timestep reduction, we devise a non-uniform grouping scheme tailored to the non-uniform nature of the denoising process, thereby mitigating the explosive combinations of timesteps. In terms of quantization, we adopt a fine-grained layer-wise approach to allocate varying bit-widths to different layers based on their respective contributions to the final generative performance, thus rectifying performance degradation observed in prior studies. To expedite the evaluation of fine-grained quantization, we further devise a super-network to serve as a precision solver by leveraging shared quantization results. These two design components are seamlessly integrated within our framework, enabling rapid joint exploration of the exponentially large decision space via a gradient-free evolutionary search algorithm.

Title: WiTUnet: A U-Shaped Architecture Integrating CNN and Transformer for Improved Feature Alignment and Local Information Fusion

• Authors: Bin Wang, Fei Deng, Peifan Jiang, Shuang Wang, Xiao Han, Hongjie Zheng
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09533
• Pdf URL: https://arxiv.org/pdf/2404.09533
• Copy Paste: [[2404.09533]] WiTUnet: A U-Shaped Architecture Integrating CNN and Transformer for Improved Feature Alignment and Local Information Fusion(https://arxiv.org/abs/2404.09533)
• Keywords: transformer
• Abstract: Low-dose computed tomography (LDCT) has become the technology of choice for diagnostic medical imaging, given its lower radiation dose compared to standard CT, despite increasing image noise and potentially affecting diagnostic accuracy. To address this, advanced deep learning-based LDCT denoising algorithms have been developed, primarily using Convolutional Neural Networks (CNNs) or Transformer Networks with the Unet architecture. This architecture enhances image detail by integrating feature maps from the encoder and decoder via skip connections. However, current methods often overlook enhancements to the Unet architecture itself, focusing instead on optimizing encoder and decoder structures. This approach can be problematic due to the significant differences in feature map characteristics between the encoder and decoder, where simple fusion strategies may not effectively reconstruct images.In this paper, we introduce WiTUnet, a novel LDCT image denoising method that utilizes nested, dense skip pathways instead of traditional skip connections to improve feature integration. WiTUnet also incorporates a windowed Transformer structure to process images in smaller, non-overlapping segments, reducing computational load. Additionally, the integration of a Local Image Perception Enhancement (LiPe) module in both the encoder and decoder replaces the standard multi-layer perceptron (MLP) in Transformers, enhancing local feature capture and representation. Through extensive experimental comparisons, WiTUnet has demonstrated superior performance over existing methods in key metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Root Mean Square Error (RMSE), significantly improving noise removal and image quality.

Title: Text-Driven Diverse Facial Texture Generation via Progressive Latent-Space Refinement

• Authors: Chi Wang, Junming Huang, Rong Zhang, Qi Wang, Haotian Yang, Haibin Huang, Chongyang Ma, Weiwei Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09540
• Pdf URL: https://arxiv.org/pdf/2404.09540
• Copy Paste: [[2404.09540]] Text-Driven Diverse Facial Texture Generation via Progressive Latent-Space Refinement(https://arxiv.org/abs/2404.09540)
• Keywords: generative
• Abstract: Automatic 3D facial texture generation has gained significant interest recently. Existing approaches may not support the traditional physically based rendering pipeline or rely on 3D data captured by Light Stage. Our key contribution is a progressive latent space refinement approach that can bootstrap from 3D Morphable Models (3DMMs)-based texture maps generated from facial images to generate high-quality and diverse PBR textures, including albedo, normal, and roughness. It starts with enhancing Generative Adversarial Networks (GANs) for text-guided and diverse texture generation. To this end, we design a self-supervised paradigm to overcome the reliance on ground truth 3D textures and train the generative model with only entangled texture maps. Besides, we foster mutual enhancement between GANs and Score Distillation Sampling (SDS). SDS boosts GANs with more generative modes, while GANs promote more efficient optimization of SDS. Furthermore, we introduce an edge-aware SDS for multi-view consistent facial structure. Experiments demonstrate that our method outperforms existing 3D texture generation methods regarding photo-realistic quality, diversity, and efficiency.

Title: AI-KD: Towards Alignment Invariant Face Image Quality Assessment Using Knowledge Distillation

• Authors: Žiga Babnik, Fadi Boutros, Naser Damer, Peter Peer, Vitomir Štruc
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09555
• Pdf URL: https://arxiv.org/pdf/2404.09555
• Copy Paste: [[2404.09555]] AI-KD: Towards Alignment Invariant Face Image Quality Assessment Using Knowledge Distillation(https://arxiv.org/abs/2404.09555)
• Keywords: robust
• Abstract: Face Image Quality Assessment (FIQA) techniques have seen steady improvements over recent years, but their performance still deteriorates if the input face samples are not properly aligned. This alignment sensitivity comes from the fact that most FIQA techniques are trained or designed using a specific face alignment procedure. If the alignment technique changes, the performance of most existing FIQA techniques quickly becomes suboptimal. To address this problem, we present in this paper a novel knowledge distillation approach, termed AI-KD that can extend on any existing FIQA technique, improving its robustness to alignment variations and, in turn, performance with different alignment procedures. To validate the proposed distillation approach, we conduct comprehensive experiments on 6 face datasets with 4 recent face recognition models and in comparison to 7 state-of-the-art FIQA techniques. Our results show that AI-KD consistently improves performance of the initial FIQA techniques not only with misaligned samples, but also with properly aligned facial images. Furthermore, it leads to a new state-of-the-art, when used with a competitive initial FIQA approach. The code for AI-KD is made publicly available from: https://github.com/LSIbabnikz/AI-KD.

Title: nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation

• Authors: Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, Paul F. Jaeger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09556
• Pdf URL: https://arxiv.org/pdf/2404.09556
• Copy Paste: [[2404.09556]] nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation(https://arxiv.org/abs/2404.09556)
• Keywords: transformer, segmentation
• Abstract: The release of nnU-Net marked a paradigm shift in 3D medical image segmentation, demonstrating that a properly configured U-Net architecture could still achieve state-of-the-art results. Despite this, the pursuit of novel architectures, and the respective claims of superior performance over the U-Net baseline, continued. In this study, we demonstrate that many of these recent claims fail to hold up when scrutinized for common validation shortcomings, such as the use of inadequate baselines, insufficient datasets, and neglected computational resources. By meticulously avoiding these pitfalls, we conduct a thorough and comprehensive benchmarking of current segmentation methods including CNN-based, Transformer-based, and Mamba-based approaches. In contrast to current beliefs, we find that the recipe for state-of-the-art performance is 1) employing CNN-based U-Net models, including ResNet and ConvNeXt variants, 2) using the nnU-Net framework, and 3) scaling models to modern hardware resources. These results indicate an ongoing innovation bias towards novel architectures in the field and underscore the need for more stringent validation standards in the quest for scientific progress.

Title: The revenge of BiSeNet: Efficient Multi-Task Image Segmentation

• Authors: Gabriele Rosi, Claudia Cuttano, Niccolò Cavagnero, Giuseppe Averta, Fabio Cermelli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09570
• Pdf URL: https://arxiv.org/pdf/2404.09570
• Copy Paste: [[2404.09570]] The revenge of BiSeNet: Efficient Multi-Task Image Segmentation(https://arxiv.org/abs/2404.09570)
• Keywords: segmentation
• Abstract: Recent advancements in image segmentation have focused on enhancing the efficiency of the models to meet the demands of real-time applications, especially on edge devices. However, existing research has primarily concentrated on single-task settings, especially on semantic segmentation, leading to redundant efforts and specialized architectures for different tasks. To address this limitation, we propose a novel architecture for efficient multi-task image segmentation, capable of handling various segmentation tasks without sacrificing efficiency or accuracy. We introduce BiSeNetFormer, that leverages the efficiency of two-stream semantic segmentation architectures and it extends them into a mask classification framework. Our approach maintains the efficient spatial and context paths to capture detailed and semantic information, respectively, while leveraging an efficient transformed-based segmentation head that computes the binary masks and class probabilities. By seamlessly supporting multiple tasks, namely semantic and panoptic segmentation, BiSeNetFormer offers a versatile solution for multi-task segmentation. We evaluate our approach on popular datasets, Cityscapes and ADE20K, demonstrating impressive inference speeds while maintaining competitive accuracy compared to state-of-the-art architectures. Our results indicate that BiSeNetFormer represents a significant advancement towards fast, efficient, and multi-task segmentation networks, bridging the gap between model efficiency and task adaptability.

Title: Large language models and linguistic intentionality

• Authors: Jumbly Grindrod
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09576
• Pdf URL: https://arxiv.org/pdf/2404.09576
• Copy Paste: [[2404.09576]] Large language models and linguistic intentionality(https://arxiv.org/abs/2404.09576)
• Keywords: large language model
• Abstract: Do large language models like Chat-GPT or LLaMa meaningfully use the words they produce? Or are they merely clever prediction machines, simulating language use by producing statistically plausible text? There have already been some initial attempts to answer this question by showing that these models meet the criteria for entering meaningful states according to metasemantic theories of mental content. In this paper, I will argue for a different approach - that we should instead consider whether language models meet the criteria given by our best metasemantic theories of linguistic content. In that vein, I will illustrate how this can be done by applying two such theories to the case of language models: Gareth Evans' (1982) account of naming practices and Ruth Millikan's (1984, 2004, 2005) teleosemantics. In doing so, I will argue that it is a mistake to think that the failure of LLMs to meet plausible conditions for mental intentionality thereby renders their outputs meaningless, and that a distinguishing feature of linguistic intentionality - dependency on a pre-existing linguistic system - allows for the plausible result LLM outputs are meaningful.

Title: Transformers, Contextualism, and Polysemy

• Authors: Jumbly Grindrod
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09577
• Pdf URL: https://arxiv.org/pdf/2404.09577
• Copy Paste: [[2404.09577]] Transformers, Contextualism, and Polysemy(https://arxiv.org/abs/2404.09577)
• Keywords: transformer
• Abstract: The transformer architecture, introduced by Vaswani et al. (2017), is at the heart of the remarkable recent progress in the development of language models, including famous chatbots such as Chat-gpt and Bard. In this paper, I argue that we an extract from the way the transformer architecture works a picture of the relationship between context and meaning. I call this the transformer picture, and I argue that it is a novel with regard to two related philosophical debates: the contextualism debate regarding the extent of context-sensitivity across natural language, and the polysemy debate regarding how polysemy should be captured within an account of word meaning. Although much of the paper merely tries to position the transformer picture with respect to these two debates, I will also begin to make the case for the transformer picture.

Title: Modelling Language

• Authors: Jumbly Grindrod
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09579
• Pdf URL: https://arxiv.org/pdf/2404.09579
• Copy Paste: [[2404.09579]] Modelling Language(https://arxiv.org/abs/2404.09579)
• Keywords: large language model
• Abstract: This paper argues that large language models have a valuable scientific role to play in serving as scientific models of a language. Linguistic study should not only be concerned with the cognitive processes behind linguistic competence, but also with language understood as an external, social entity. Once this is recognized, the value of large language models as scientific models becomes clear. This paper defends this position against a number of arguments to the effect that language models provide no linguistic insight. It also draws upon recent work in philosophy of science to show how large language models could serve as scientific models.

Title: Pseudo-label Learning with Calibrated Confidence Using an Energy-based Model

• Authors: Masahito Toba, Seiichi Uchida, Hideaki Hayashi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09585
• Pdf URL: https://arxiv.org/pdf/2404.09585
• Copy Paste: [[2404.09585]] Pseudo-label Learning with Calibrated Confidence Using an Energy-based Model(https://arxiv.org/abs/2404.09585)
• Keywords: extraction
• Abstract: In pseudo-labeling (PL), which is a type of semi-supervised learning, pseudo-labels are assigned based on the confidence scores provided by the classifier; therefore, accurate confidence is important for successful PL. In this study, we propose a PL algorithm based on an energy-based model (EBM), which is referred to as the energy-based PL (EBPL). In EBPL, a neural network-based classifier and an EBM are jointly trained by sharing their feature extraction parts. This approach enables the model to learn both the class decision boundary and input data distribution, enhancing confidence calibration during network training. The experimental results demonstrate that EBPL outperforms the existing PL method in semi-supervised image classification tasks, with superior confidence calibration error and recognition accuracy.

Title: Mitigating the Curse of Dimensionality for Certified Robustness via Dual Randomized Smoothing

• Authors: Song Xia, Yu Yi, Xudong Jiang, Henghui Ding
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09586
• Pdf URL: https://arxiv.org/pdf/2404.09586
• Copy Paste: [[2404.09586]] Mitigating the Curse of Dimensionality for Certified Robustness via Dual Randomized Smoothing(https://arxiv.org/abs/2404.09586)
• Keywords: robust
• Abstract: Randomized Smoothing (RS) has been proven a promising method for endowing an arbitrary image classifier with certified robustness. However, the substantial uncertainty inherent in the high-dimensional isotropic Gaussian noise imposes the curse of dimensionality on RS. Specifically, the upper bound of ${\ell_2}$ certified robustness radius provided by RS exhibits a diminishing trend with the expansion of the input dimension $d$, proportionally decreasing at a rate of $1/\sqrt{d}$. This paper explores the feasibility of providing ${\ell_2}$ certified robustness for high-dimensional input through the utilization of dual smoothing in the lower-dimensional space. The proposed Dual Randomized Smoothing (DRS) down-samples the input image into two sub-images and smooths the two sub-images in lower dimensions. Theoretically, we prove that DRS guarantees a tight ${\ell_2}$ certified robustness radius for the original input and reveal that DRS attains a superior upper bound on the ${\ell_2}$ robustness radius, which decreases proportionally at a rate of $(1/\sqrt m + 1/\sqrt n )$ with $m+n=d$. Extensive experiments demonstrate the generalizability and effectiveness of DRS, which exhibits a notable capability to integrate with established methodologies, yielding substantial improvements in both accuracy and ${\ell_2}$ certified robustness baselines of RS on the CIFAR-10 and ImageNet datasets. Code is available at https://github.com/xiasong0501/DRS.

Title: 3D Gaussian Splatting as Markov Chain Monte Carlo

• Authors: Shakiba Kheradmand, Daniel Rebain, Gopal Sharma, Weiwei Sun, Jeff Tseng, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, Kwang Moo Yi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09591
• Pdf URL: https://arxiv.org/pdf/2404.09591
• Copy Paste: [[2404.09591]] 3D Gaussian Splatting as Markov Chain Monte Carlo(https://arxiv.org/abs/2404.09591)
• Keywords: robust
• Abstract: While 3D Gaussian Splatting has recently become popular for neural rendering, current methods rely on carefully engineered cloning and splitting strategies for placing Gaussians, which does not always generalize and may lead to poor-quality renderings. In addition, for real-world scenes, they rely on a good initial point cloud to perform well. In this work, we rethink 3D Gaussians as random samples drawn from an underlying probability distribution describing the physical representation of the scene -- in other words, Markov Chain Monte Carlo (MCMC) samples. Under this view, we show that the 3D Gaussian updates are strikingly similar to a Stochastic Langevin Gradient Descent (SGLD) update. As with MCMC, samples are nothing but past visit locations, adding new Gaussians under our framework can simply be realized without heuristics as placing Gaussians at existing Gaussian locations. To encourage using fewer Gaussians for efficiency, we introduce an L1-regularizer on the Gaussians. On various standard evaluation scenes, we show that our method provides improved rendering quality, easy control over the number of Gaussians, and robustness to initialization.

Title: Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction

• Authors: Zepeng Ding, Wenhao Huang, Jiaqing Liang, Deqing Yang, Yanghua Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09593
• Pdf URL: https://arxiv.org/pdf/2404.09593
• Copy Paste: [[2404.09593]] Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction(https://arxiv.org/abs/2404.09593)
• Keywords: extraction, large language model
• Abstract: Relation triple extraction, which outputs a set of triples from long sentences, plays a vital role in knowledge acquisition. Large language models can accurately extract triples from simple sentences through few-shot learning or fine-tuning when given appropriate instructions. However, they often miss out when extracting from complex sentences. In this paper, we design an evaluation-filtering framework that integrates large language models with small models for relational triple extraction tasks. The framework includes an evaluation model that can extract related entity pairs with high precision. We propose a simple labeling principle and a deep neural network to build the model, embedding the outputs as prompts into the extraction process of the large model. We conduct extensive experiments to demonstrate that the proposed method can assist large language models in obtaining more accurate extraction results, especially from complex sentences containing multiple relational triples. Our evaluation model can also be embedded into traditional extraction models to enhance their extraction precision from complex sentences.

Title: Enhancing Code Vulnerability Detection via Vulnerability-Preserving Data Augmentation

• Authors: Shangqing Liu, Wei Ma, Jian Wang, Xiaofei Xie, Ruitao Feng, Yang Liu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09599
• Pdf URL: https://arxiv.org/pdf/2404.09599
• Copy Paste: [[2404.09599]] Enhancing Code Vulnerability Detection via Vulnerability-Preserving Data Augmentation(https://arxiv.org/abs/2404.09599)
• Keywords: attack
• Abstract: Source code vulnerability detection aims to identify inherent vulnerabilities to safeguard software systems from potential attacks. Many prior studies overlook diverse vulnerability characteristics, simplifying the problem into a binary (0-1) classification task for example determining whether it is vulnerable or not. This poses a challenge for a single deep learning-based model to effectively learn the wide array of vulnerability characteristics. Furthermore, due to the challenges associated with collecting large-scale vulnerability data, these detectors often overfit limited training datasets, resulting in lower model generalization performance. To address the aforementioned challenges, in this work, we introduce a fine-grained vulnerability detector namely FGVulDet. Unlike previous approaches, FGVulDet employs multiple classifiers to discern characteristics of various vulnerability types and combines their outputs to identify the specific type of vulnerability. Each classifier is designed to learn type-specific vulnerability semantics. Additionally, to address the scarcity of data for some vulnerability types and enhance data diversity for learning better vulnerability semantics, we propose a novel vulnerability-preserving data augmentation technique to augment the number of vulnerabilities. Taking inspiration from recent advancements in graph neural networks for learning program semantics, we incorporate a Gated Graph Neural Network (GGNN) and extend it to an edge-aware GGNN to capture edge-type information. FGVulDet is trained on a large-scale dataset from GitHub, encompassing five different types of vulnerabilities. Extensive experiments compared with static-analysis-based approaches and learning-based approaches have demonstrated the effectiveness of FGVulDet.

Title: Machine learning-based optimization workflow of the homogeneity of spunbond nonwovens with human validation

• Authors: Viny Saajan Victor, Andre Schmeißer, Heike Leitte, Simone Gramsch
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09604
• Pdf URL: https://arxiv.org/pdf/2404.09604
• Copy Paste: [[2404.09604]] Machine learning-based optimization workflow of the homogeneity of spunbond nonwovens with human validation(https://arxiv.org/abs/2404.09604)
• Keywords: protect
• Abstract: In the last ten years, the average annual growth rate of nonwoven production was 4%. In 2020 and 2021, nonwoven production has increased even further due to the huge demand for nonwoven products needed for protective clothing such as FFP2 masks to combat the COVID19 pandemic. Optimizing the production process is still a challenge due to its high nonlinearity. In this paper, we present a machine learning-based optimization workflow aimed at improving the homogeneity of spunbond nonwovens. The optimization workflow is based on a mathematical model that simulates the microstructures of nonwovens. Based on trainingy data coming from this simulator, different machine learning algorithms are trained in order to find a surrogate model for the time-consuming simulator. Human validation is employed to verify the outputs of machine learning algorithms by assessing the aesthetics of the nonwovens. We include scientific and expert knowledge into the training data to reduce the computational costs involved in the optimization process. We demonstrate the necessity and effectiveness of our workflow in optimizing the homogeneity of nonwovens.

Title: A Self-feedback Knowledge Elicitation Approach for Chemical Reaction Predictions

• Authors: Pengfei Liu, Jun Tao, Zhixiang Ren
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2404.09606
• Pdf URL: https://arxiv.org/pdf/2404.09606
• Copy Paste: [[2404.09606]] A Self-feedback Knowledge Elicitation Approach for Chemical Reaction Predictions(https://arxiv.org/abs/2404.09606)
• Keywords: extraction, large language model
• Abstract: The task of chemical reaction predictions (CRPs) plays a pivotal role in advancing drug discovery and material science. However, its effectiveness is constrained by the vast and uncertain chemical reaction space and challenges in capturing reaction selectivity, particularly due to existing methods' limitations in exploiting the data's inherent knowledge. To address these challenges, we introduce a data-curated self-feedback knowledge elicitation approach. This method starts from iterative optimization of molecular representations and facilitates the extraction of knowledge on chemical reaction types (RTs). Then, we employ adaptive prompt learning to infuse the prior knowledge into the large language model (LLM). As a result, we achieve significant enhancements: a 14.2% increase in retrosynthesis prediction accuracy, a 74.2% rise in reagent prediction accuracy, and an expansion in the model's capability for handling multi-task chemical reactions. This research offers a novel paradigm for knowledge elicitation in scientific research and showcases the untapped potential of LLMs in CRPs.

Title: UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmark

• Authors: Zhaokun Zhou, Qiulin Wang, Bin Lin, Yiwei Su, Rui Chen, Xin Tao, Amin Zheng, Li Yuan, Pengfei Wan, Di Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09619
• Pdf URL: https://arxiv.org/pdf/2404.09619
• Copy Paste: [[2404.09619]] UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmark(https://arxiv.org/abs/2404.09619)
• Keywords: large language model
• Abstract: As an alternative to expensive expert evaluation, Image Aesthetic Assessment (IAA) stands out as a crucial task in computer vision. However, traditional IAA methods are typically constrained to a single data source or task, restricting the universality and broader application. In this work, to better align with human aesthetics, we propose a Unified Multi-modal Image Aesthetic Assessment (UNIAA) framework, including a Multi-modal Large Language Model (MLLM) named UNIAA-LLaVA and a comprehensive benchmark named UNIAA-Bench. We choose MLLMs with both visual perception and language ability for IAA and establish a low-cost paradigm for transforming the existing datasets into unified and high-quality visual instruction tuning data, from which the UNIAA-LLaVA is trained. To further evaluate the IAA capability of MLLMs, we construct the UNIAA-Bench, which consists of three aesthetic levels: Perception, Description, and Assessment. Extensive experiments validate the effectiveness and rationality of UNIAA. UNIAA-LLaVA achieves competitive performance on all levels of UNIAA-Bench, compared with existing MLLMs. Specifically, our model performs better than GPT-4V in aesthetic perception and even approaches the junior-level human. We find MLLMs have great potential in IAA, yet there remains plenty of room for further improvement. The UNIAA-LLaVA and UNIAA-Bench will be released.

Title: AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception

• Authors: Yipo Huang, Xiangfei Sheng, Zhichao Yang, Quan Yuan, Zhichao Duan, Pengfei Chen, Leida Li, Weisi Lin, Guangming Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09624
• Pdf URL: https://arxiv.org/pdf/2404.09624
• Copy Paste: [[2404.09624]] AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception(https://arxiv.org/abs/2404.09624)
• Keywords: large language model
• Abstract: The highly abstract nature of image aesthetics perception (IAP) poses significant challenge for current multimodal large language models (MLLMs). The lack of human-annotated multi-modality aesthetic data further exacerbates this dilemma, resulting in MLLMs falling short of aesthetics perception capabilities. To address the above challenge, we first introduce a comprehensively annotated Aesthetic Multi-Modality Instruction Tuning (AesMMIT) dataset, which serves as the footstone for building multi-modality aesthetics foundation models. Specifically, to align MLLMs with human aesthetics perception, we construct a corpus-rich aesthetic critique database with 21,904 diverse-sourced images and 88K human natural language feedbacks, which are collected via progressive questions, ranging from coarse-grained aesthetic grades to fine-grained aesthetic descriptions. To ensure that MLLMs can handle diverse queries, we further prompt GPT to refine the aesthetic critiques and assemble the large-scale aesthetic instruction tuning dataset, i.e. AesMMIT, which consists of 409K multi-typed instructions to activate stronger aesthetic capabilities. Based on the AesMMIT database, we fine-tune the open-sourced general foundation models, achieving multi-modality Aesthetic Expert models, dubbed AesExpert. Extensive experiments demonstrate that the proposed AesExpert models deliver significantly better aesthetic perception performances than the state-of-the-art MLLMs, including the most advanced GPT-4V and Gemini-Pro-Vision. Source data will be available at https://github.com/yipoh/AesExpert.

Title: Privacy-Preserving Intrusion Detection using Convolutional Neural Networks

• Authors: Martin Kodys, Zhongmin Dai, Vrizlynn L. L. Thing
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09625
• Pdf URL: https://arxiv.org/pdf/2404.09625
• Copy Paste: [[2404.09625]] Privacy-Preserving Intrusion Detection using Convolutional Neural Networks(https://arxiv.org/abs/2404.09625)
• Keywords: privacy, protect, attack
• Abstract: Privacy-preserving analytics is designed to protect valuable assets. A common service provision involves the input data from the client and the model on the analyst's side. The importance of the privacy preservation is fuelled by legal obligations and intellectual property concerns. We explore the use case of a model owner providing an analytic service on customer's private data. No information about the data shall be revealed to the analyst and no information about the model shall be leaked to the customer. Current methods involve costs: accuracy deterioration and computational complexity. The complexity, in turn, results in a longer processing time, increased requirement on computing resources, and involves data communication between the client and the server. In order to deploy such service architecture, we need to evaluate the optimal setting that fits the constraints. And that is what this paper addresses. In this work, we enhance an attack detection system based on Convolutional Neural Networks with privacy-preserving technology based on PriMIA framework that is initially designed for medical data.

Title: Bridging Vision and Language Spaces with Assignment Prediction

• Authors: Jungin Park, Jiyoung Lee, Kwanghoon Sohn
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09632
• Pdf URL: https://arxiv.org/pdf/2404.09632
• Copy Paste: [[2404.09632]] Bridging Vision and Language Spaces with Assignment Prediction(https://arxiv.org/abs/2404.09632)
• Keywords: robust, large language model
• Abstract: This paper introduces VLAP, a novel approach that bridges pretrained vision models and large language models (LLMs) to make frozen LLMs understand the visual world. VLAP transforms the embedding space of pretrained vision models into the LLMs' word embedding space using a single linear layer for efficient and general-purpose visual and language understanding. Specifically, we harness well-established word embeddings to bridge two modality embedding spaces. The visual and text representations are simultaneously assigned to a set of word embeddings within pretrained LLMs by formulating the assigning procedure as an optimal transport problem. We predict the assignment of one modality from the representation of another modality data, enforcing consistent assignments for paired multimodal data. This allows vision and language representations to contain the same information, grounding the frozen LLMs' word embedding space in visual data. Moreover, a robust semantic taxonomy of LLMs can be preserved with visual data since the LLMs interpret and reason linguistic information from correlations between word embeddings. Experimental results show that VLAP achieves substantial improvements over the previous linear transformation-based approaches across a range of vision-language tasks, including image captioning, visual question answering, and cross-modal retrieval. We also demonstrate the learned visual representations hold a semantic taxonomy of LLMs, making visual semantic arithmetic possible.

Title: In-Context Translation: Towards Unifying Image Recognition, Processing, and Generation

• Authors: Han Xue, Qianru Sun, Li Song, Wenjun Zhang, Zhiwu Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09633
• Pdf URL: https://arxiv.org/pdf/2404.09633
• Copy Paste: [[2404.09633]] In-Context Translation: Towards Unifying Image Recognition, Processing, and Generation(https://arxiv.org/abs/2404.09633)
• Keywords: diffusion, segmentation
• Abstract: We propose In-Context Translation (ICT), a general learning framework to unify visual recognition (e.g., semantic segmentation), low-level image processing (e.g., denoising), and conditional image generation (e.g., edge-to-image synthesis). Thanks to unification, ICT significantly reduces the inherent inductive bias that comes with designing models for specific tasks, and it maximizes mutual enhancement across similar tasks. However, the unification across a large number of tasks is non-trivial due to various data formats and training pipelines. To this end, ICT introduces two designs. Firstly, it standardizes input-output data of different tasks into RGB image pairs, e.g., semantic segmentation data pairs an RGB image with its segmentation mask in the same RGB format. This turns different tasks into a general translation task between two RGB images. Secondly, it standardizes the training of different tasks into a general in-context learning, where "in-context" means the input comprises an example input-output pair of the target task and a query image. The learning objective is to generate the "missing" data paired with the query. The implicit translation process is thus between the query and the generated image. In experiments, ICT unifies ten vision tasks and showcases impressive performance on their respective benchmarks. Notably, compared to its competitors, e.g., Painter and PromptDiffusion, ICT trained on only 4 RTX 3090 GPUs is shown to be more efficient and less costly in training.

Title: All-in-one simulation-based inference

• Authors: Manuel Gloeckler, Michael Deistler, Christian Weilbach, Frank Wood, Jakob H. Macke
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.09636
• Pdf URL: https://arxiv.org/pdf/2404.09636
• Copy Paste: [[2404.09636]] All-in-one simulation-based inference(https://arxiv.org/abs/2404.09636)
• Keywords: diffusion, transformer
• Abstract: Amortized Bayesian inference trains neural networks to solve stochastic inference problems using model simulations, thereby making it possible to rapidly perform Bayesian inference for any newly observed data. However, current simulation-based amortized inference methods are simulation-hungry and inflexible: They require the specification of a fixed parametric prior, simulator, and inference tasks ahead of time. Here, we present a new amortized inference method -- the Simformer -- which overcomes these limitations. By training a probabilistic diffusion model with transformer architectures, the Simformer outperforms current state-of-the-art amortized inference approaches on benchmark tasks and is substantially more flexible: It can be applied to models with function-valued parameters, it can handle inference scenarios with missing or unstructured data, and it can sample arbitrary conditionals of the joint distribution of parameters and data, including both posterior and likelihood. We showcase the performance and flexibility of the Simformer on simulators from ecology, epidemiology, and neuroscience, and demonstrate that it opens up new possibilities and application domains for amortized Bayesian inference on simulation-based models.

Title: CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-Shot Learning

• Authors: Haojian Huang, Xiaozhen Qiao, Zhuo Chen, Haodong Chen, Bingyu Li, Zhe Sun, Mulin Chen, Xuelong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09640
• Pdf URL: https://arxiv.org/pdf/2404.09640
• Copy Paste: [[2404.09640]] CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-Shot Learning(https://arxiv.org/abs/2404.09640)
• Keywords: robust, explainability
• Abstract: Zero-shot learning (ZSL) enables the recognition of novel classes by leveraging semantic knowledge transfer from known to unknown categories. This knowledge, typically encapsulated in attribute descriptions, aids in identifying class-specific visual features, thus facilitating visual-semantic alignment and improving ZSL performance. However, real-world challenges such as distribution imbalances and attribute co-occurrence among instances often hinder the discernment of local variances in images, a problem exacerbated by the scarcity of fine-grained, region-specific attribute annotations. Moreover, the variability in visual presentation within categories can also skew attribute-category associations. In response, we propose a bidirectional cross-modal ZSL approach CREST. It begins by extracting representations for attribute and visual localization and employs Evidential Deep Learning (EDL) to measure underlying epistemic uncertainty, thereby enhancing the model's resilience against hard negatives. CREST incorporates dual learning pathways, focusing on both visual-category and attribute-category alignments, to ensure robust correlation between latent and observable spaces. Moreover, we introduce an uncertainty-informed cross-modal fusion technique to refine visual-attribute inference. Extensive experiments demonstrate our model's effectiveness and unique explainability across multiple datasets. Our code and data are available at: Comments: Ongoing work; 10 pages, 2 Tables, 9 Figures; Repo is available at https://github.com/JethroJames/CREST.

Title: Do LLMs Understand Visual Anomalies? Uncovering LLM Capabilities in Zero-shot Anomaly Detection

• Authors: Jiaqi Zhu, Shaofeng Cai, Fang Deng, Junran Wu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.09654
• Pdf URL: https://arxiv.org/pdf/2404.09654
• Copy Paste: [[2404.09654]] Do LLMs Understand Visual Anomalies? Uncovering LLM Capabilities in Zero-shot Anomaly Detection(https://arxiv.org/abs/2404.09654)
• Keywords: large language model
• Abstract: Large vision-language models (LVLMs) are markedly proficient in deriving visual representations guided by natural language. Recent explorations have utilized LVLMs to tackle zero-shot visual anomaly detection (VAD) challenges by pairing images with textual descriptions indicative of normal and abnormal conditions, referred to as anomaly prompts. However, existing approaches depend on static anomaly prompts that are prone to cross-semantic ambiguity, and prioritize global image-level representations over crucial local pixel-level image-to-text alignment that is necessary for accurate anomaly localization. In this paper, we present ALFA, a training-free approach designed to address these challenges via a unified model. We propose a run-time prompt adaptation strategy, which first generates informative anomaly prompts to leverage the capabilities of a large language model (LLM). This strategy is enhanced by a contextual scoring mechanism for per-image anomaly prompt adaptation and cross-semantic ambiguity mitigation. We further introduce a novel fine-grained aligner to fuse local pixel-level semantics for precise anomaly localization, by projecting the image-text alignment from global to local semantic spaces. Extensive evaluations on the challenging MVTec and VisA datasets confirm ALFA's effectiveness in harnessing the language potential for zero-shot VAD, achieving significant PRO improvements of 12.1% on MVTec AD and 8.9% on VisA compared to state-of-the-art zero-shot VAD approaches.

Title: Closing the Gap in the Trade-off between Fair Representations and Accuracy

• Authors: Biswajit Rout, Ananya B. Sai, Arun Rajkumar
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.09664
• Pdf URL: https://arxiv.org/pdf/2404.09664
• Copy Paste: [[2404.09664]] Closing the Gap in the Trade-off between Fair Representations and Accuracy(https://arxiv.org/abs/2404.09664)
• Keywords: fair
• Abstract: The rapid developments of various machine learning models and their deployments in several applications has led to discussions around the importance of looking beyond the accuracies of these models. Fairness of such models is one such aspect that is deservedly gaining more attention. In this work, we analyse the natural language representations of documents and sentences (i.e., encodings) for any embedding-level bias that could potentially also affect the fairness of the downstream tasks that rely on them. We identify bias in these encodings either towards or against different sub-groups based on the difference in their reconstruction errors along various subsets of principal components. We explore and recommend ways to mitigate such bias in the encodings while also maintaining a decent accuracy in classification models that use them.

Title: An Empirical Study of Open Edge Computing Platforms: Ecosystem, Usage, and Security Risks

• Authors: Yu Bi, Mingshuo Yang, Yong Fang, Xianghang Mi, Shanqing Guo, Shujun Tang, Haixin Duan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09681
• Pdf URL: https://arxiv.org/pdf/2404.09681
• Copy Paste: [[2404.09681]] An Empirical Study of Open Edge Computing Platforms: Ecosystem, Usage, and Security Risks(https://arxiv.org/abs/2404.09681)
• Keywords: security, privacy
• Abstract: Emerging in recent years, open edge computing platforms (OECPs) claim large-scale edge nodes, the extensive usage and adoption, as well as the openness to any third parties to join as edge nodes. For instance, OneThingCloud, a major OECP operated in China, advertises 5 million edge nodes, 70TB bandwidth, and 1,500PB storage. However, little information is publicly available for such OECPs with regards to their technical mechanisms and involvement in edge computing activities. Furthermore, different from known edge computing paradigms, OECPs feature an open ecosystem wherein any third party can participate as edge nodes and earn revenue for the contribution of computing and bandwidth resources, which, however, can introduce byzantine or even malicious edge nodes and thus break the traditional threat model for edge computing. In this study, we conduct the first empirical study on two representative OECPs, which is made possible through the deployment of edge nodes across locations, the efficient and semi-automatic analysis of edge traffic as well as the carefully designed security experiments. As the results, a set of novel findings and insights have been distilled with regards to their technical mechanisms, the landscape of edge nodes, the usage and adoption, and the practical security/privacy risks. Particularly, millions of daily active edge nodes have been observed, which feature a wide distribution in the network space and the extensive adoption in content delivery towards end users of 16 popular Internet services. Also, multiple practical and concerning security risks have been identified along with acknowledgements received from relevant parties, e.g., the exposure of long-term and cross-edge-node credentials, the co-location with malicious activities of diverse categories, the failures of TLS certificate verification, the extensive information leakage against end users, etc.

Title: Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation

• Authors: Juhwan Choi, Jungmin Yun, Kyohoon Jin, YoungBin Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09682
• Pdf URL: https://arxiv.org/pdf/2404.09682
• Copy Paste: [[2404.09682]] Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation(https://arxiv.org/abs/2404.09682)
• Keywords: large language model
• Abstract: The quality of the dataset is crucial for ensuring optimal performance and reliability of downstream task models. However, datasets often contain noisy data inadvertently included during the construction process. Numerous attempts have been made to correct this issue through human annotators. However, hiring and managing human annotators is expensive and time-consuming. As an alternative, recent studies are exploring the use of large language models (LLMs) for data annotation. In this study, we present a case study that extends the application of LLM-based data annotation to enhance the quality of existing datasets through a cleansing strategy. Specifically, we leverage approaches such as chain-of-thought (CoT) and majority voting to imitate human annotation and classify unrelated documents from the Multi-News dataset, which is widely used for the multi-document summarization task. Through our proposed cleansing method, we introduce an enhanced Multi-News+. By employing LLMs for data cleansing, we demonstrate an efficient and effective approach to improving dataset quality without relying on expensive human annotation efforts.

Title: Harnessing GPT-4V(ision) for Insurance: A Preliminary Exploration

• Authors: Chenwei Lin, Hanjia Lyu, Jiebo Luo, Xian Xu
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09690
• Pdf URL: https://arxiv.org/pdf/2404.09690
• Copy Paste: [[2404.09690]] Harnessing GPT-4V(ision) for Insurance: A Preliminary Exploration(https://arxiv.org/abs/2404.09690)
• Keywords: robust
• Abstract: The emergence of Large Multimodal Models (LMMs) marks a significant milestone in the development of artificial intelligence. Insurance, as a vast and complex discipline, involves a wide variety of data forms in its operational processes, including text, images, and videos, thereby giving rise to diverse multimodal tasks. Despite this, there has been limited systematic exploration of multimodal tasks specific to insurance, nor a thorough investigation into how LMMs can address these challenges. In this paper, we explore GPT-4V's capabilities in the insurance domain. We categorize multimodal tasks by focusing primarily on visual aspects based on types of insurance (e.g., auto, household/commercial property, health, and agricultural insurance) and insurance stages (e.g., risk assessment, risk monitoring, and claims processing). Our experiment reveals that GPT-4V exhibits remarkable abilities in insurance-related tasks, demonstrating not only a robust understanding of multimodal content in the insurance domain but also a comprehensive knowledge of insurance scenarios. However, there are notable shortcomings: GPT-4V struggles with detailed risk rating and loss assessment, suffers from hallucination in image understanding, and shows variable support for different languages. Through this work, we aim to bridge the insurance domain with cutting-edge LMM technology, facilitate interdisciplinary exchange and development, and provide a foundation for the continued advancement and evolution of future research endeavors.

Title: XoFTR: Cross-modal Feature Matching Transformer

• Authors: Önder Tuzcuoğlu, Aybora Köksal, Buğra Sofu, Sinan Kalkan, A. Aydın Alatan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09692
• Pdf URL: https://arxiv.org/pdf/2404.09692
• Copy Paste: [[2404.09692]] XoFTR: Cross-modal Feature Matching Transformer(https://arxiv.org/abs/2404.09692)
• Keywords: transformer
• Abstract: We introduce, XoFTR, a cross-modal cross-view method for local feature matching between thermal infrared (TIR) and visible images. Unlike visible images, TIR images are less susceptible to adverse lighting and weather conditions but present difficulties in matching due to significant texture and intensity differences. Current hand-crafted and learning-based methods for visible-TIR matching fall short in handling viewpoint, scale, and texture diversities. To address this, XoFTR incorporates masked image modeling pre-training and fine-tuning with pseudo-thermal image augmentation to handle the modality differences. Additionally, we introduce a refined matching pipeline that adjusts for scale discrepancies and enhances match reliability through sub-pixel level refinement. To validate our approach, we collect a comprehensive visible-thermal dataset, and show that our method outperforms existing methods on many benchmarks.

Title: LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models

• Authors: Guangyan Li, Yongqiang Tang, Wensheng Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09695
• Pdf URL: https://arxiv.org/pdf/2404.09695
• Copy Paste: [[2404.09695]] LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models(https://arxiv.org/abs/2404.09695)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) show excellent performance in difficult tasks, but they often require massive memories and computational resources. How to reduce the parameter scale of LLMs has become research hotspots. In this study, we make an important observation that the multi-head self-attention (MHA) sub-layer of Transformer exhibits noticeable low-rank structure, while the feed-forward network (FFN) sub-layer does not. With this regard, we design a mixed compression model, which organically combines Low-Rank matrix approximation And structured Pruning (LoRAP). For the MHA sub-layer, we propose an input activation weighted singular value decomposition method to strengthen the low-rank characteristic. Furthermore, we discover that the weight matrices in MHA sub-layer have different low-rank degrees. Thus, a novel parameter allocation scheme according to the discrepancy of low-rank degrees is devised. For the FFN sub-layer, we propose a gradient-free structured channel pruning method. During the pruning, we get an interesting finding that the least important 1% of parameter actually play a vital role in model performance. Extensive evaluations on zero-shot perplexity and zero-shot task classification indicate that our proposal is superior to previous structured compression rivals under multiple compression ratios.

Title: Are Large Language Models Reliable Argument Quality Annotators?

• Authors: Nailia Mirzakhmedova, Marcel Gohsen, Chia Hao Chang, Benno Stein
• Subjects: cs.CL, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2404.09696
• Pdf URL: https://arxiv.org/pdf/2404.09696
• Copy Paste: [[2404.09696]] Are Large Language Models Reliable Argument Quality Annotators?(https://arxiv.org/abs/2404.09696)
• Keywords: large language model
• Abstract: Evaluating the quality of arguments is a crucial aspect of any system leveraging argument mining. However, it is a challenge to obtain reliable and consistent annotations regarding argument quality, as this usually requires domain-specific expertise of the annotators. Even among experts, the assessment of argument quality is often inconsistent due to the inherent subjectivity of this task. In this paper, we study the potential of using state-of-the-art large language models (LLMs) as proxies for argument quality annotators. To assess the capability of LLMs in this regard, we analyze the agreement between model, human expert, and human novice annotators based on an established taxonomy of argument quality dimensions. Our findings highlight that LLMs can produce consistent annotations, with a moderately high agreement with human experts across most of the quality dimensions. Moreover, we show that using LLMs as additional annotators can significantly improve the agreement between annotators. These results suggest that LLMs can serve as a valuable tool for automated argument quality assessment, thus streamlining and accelerating the evaluation of large argument datasets.

Title: HSIDMamba: Exploring Bidirectional State-Space Models for Hyperspectral Denoising

• Authors: Yang Liu, Jiahua Xiao, Yu Guo, Peilin Jiang, Haiwei Yang, Fei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09697
• Pdf URL: https://arxiv.org/pdf/2404.09697
• Copy Paste: [[2404.09697]] HSIDMamba: Exploring Bidirectional State-Space Models for Hyperspectral Denoising(https://arxiv.org/abs/2404.09697)
• Keywords: transformer
• Abstract: Effectively discerning spatial-spectral dependencies in HSI denoising is crucial, but prevailing methods using convolution or transformers still face computational efficiency limitations. Recently, the emerging Selective State Space Model(Mamba) has risen with its nearly linear computational complexity in processing natural language sequences, which inspired us to explore its potential in handling long spectral sequences. In this paper, we propose HSIDMamba(HSDM), tailored to exploit the linear complexity for effectively capturing spatial-spectral dependencies in HSI denoising. In particular, HSDM comprises multiple Hyperspectral Continuous Scan Blocks, incorporating BCSM(Bidirectional Continuous Scanning Mechanism), scale residual, and spectral attention mechanisms to enhance the capture of long-range and local spatial-spectral information. BCSM strengthens spatial-spectral interactions by linking forward and backward scans and enhancing information from eight directions through SSM, significantly enhancing the perceptual capability of HSDM and improving denoising performance more effectively. Extensive evaluations against HSI denoising benchmarks validate the superior performance of HSDM, achieving state-of-the-art results in performance and surpassing the efficiency of the latest transformer architectures by $30\%$.

Title: Adaptive Patching for High-resolution Image Segmentation with Transformers

• Authors: Enzhi Zhang, Isaac Lyngaas, Peng Chen, Xiao Wang, Jun Igarashi, Yuankai Huo, Mohamed Wahib, Masaharu Munetomo
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09707
• Pdf URL: https://arxiv.org/pdf/2404.09707
• Copy Paste: [[2404.09707]] Adaptive Patching for High-resolution Image Segmentation with Transformers(https://arxiv.org/abs/2404.09707)
• Keywords: transformer, segmentation
• Abstract: Attention-based models are proliferating in the space of image analytics, including segmentation. The standard method of feeding images to transformer encoders is to divide the images into patches and then feed the patches to the model as a linear sequence of tokens. For high-resolution images, e.g. microscopic pathology images, the quadratic compute and memory cost prohibits the use of an attention-based model, if we are to use smaller patch sizes that are favorable in segmentation. The solution is to either use custom complex multi-resolution models or approximate attention schemes. We take inspiration from Adapative Mesh Refinement (AMR) methods in HPC by adaptively patching the images, as a pre-processing step, based on the image details to reduce the number of patches being fed to the model, by orders of magnitude. This method has a negligible overhead, and works seamlessly with any attention-based model, i.e. it is a pre-processing step that can be adopted by any attention-based model without friction. We demonstrate superior segmentation quality over SoTA segmentation models for real-world pathology datasets while gaining a geomean speedup of $6.9\times$ for resolutions up to $64K^2$, on up to $2,048$ GPUs.

Title: Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model

• Authors: Hyunsoo Cho
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09717
• Pdf URL: https://arxiv.org/pdf/2404.09717
• Copy Paste: [[2404.09717]] Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model(https://arxiv.org/abs/2404.09717)
• Keywords: large language model
• Abstract: Many recent studies endeavor to improve open-source language models through imitation learning, and re-training on the synthetic instruction data from state-of-the-art proprietary models like ChatGPT and GPT-4. However, the innate nature of synthetic data inherently contains noisy data, giving rise to a substantial presence of low-quality data replete with erroneous responses, and flawed reasoning. Although we intuitively grasp the potential harm of noisy data, we lack a quantitative understanding of its impact. To this end, this paper explores the correlation between the degree of noise and its impact on language models through instruction tuning. We first introduce the Falsity-Controllable (FACO) dataset, which comprises pairs of true answers with corresponding reasoning, as well as false pairs to manually control the falsity ratio of the dataset.Through our extensive experiments, we found multiple intriguing findings of the correlation between the factuality of the dataset and instruction tuning: Specifically, we verified falsity of the instruction is highly relevant to various benchmark scores. Moreover, when LLMs are trained with false instructions, they learn to lie and generate fake unfaithful answers, even though they know the correct answer for the user request. Additionally, we noted that once the language model is trained with a dataset contaminated by noise, restoring its original performance is possible, but it failed to reach full performance.

Title: VFLGAN: Vertical Federated Learning-based Generative Adversarial Network for Vertically Partitioned Data Publication

• Authors: Xun Yuan, Yang Yang, Prosanta Gope, Aryan Pasikhani, Biplab Sikdar
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09722
• Pdf URL: https://arxiv.org/pdf/2404.09722
• Copy Paste: [[2404.09722]] VFLGAN: Vertical Federated Learning-based Generative Adversarial Network for Vertically Partitioned Data Publication(https://arxiv.org/abs/2404.09722)
• Keywords: privacy, protect, attack, membership infer, federate, generative
• Abstract: In the current artificial intelligence (AI) era, the scale and quality of the dataset play a crucial role in training a high-quality AI model. However, good data is not a free lunch and is always hard to access due to privacy regulations like the General Data Protection Regulation (GDPR). A potential solution is to release a synthetic dataset with a similar distribution to that of the private dataset. Nevertheless, in some scenarios, it has been found that the attributes needed to train an AI model belong to different parties, and they cannot share the raw data for synthetic data publication due to privacy regulations. In PETS 2023, Xue et al. proposed the first generative adversary network-based model, VertiGAN, for vertically partitioned data publication. However, after thoroughly investigating, we found that VertiGAN is less effective in preserving the correlation among the attributes of different parties. This article proposes a Vertical Federated Learning-based Generative Adversarial Network, VFLGAN, for vertically partitioned data publication to address the above issues. Our experimental results show that compared with VertiGAN, VFLGAN significantly improves the quality of synthetic data. Taking the MNIST dataset as an example, the quality of the synthetic dataset generated by VFLGAN is 3.2 times better than that generated by VertiGAN w.r.t. the Fr\'echet Distance. We also designed a more efficient and effective Gaussian mechanism for the proposed VFLGAN to provide the synthetic dataset with a differential privacy guarantee. On the other hand, differential privacy only gives the upper bound of the worst-case privacy guarantee. This article also proposes a practical auditing scheme that applies membership inference attacks to estimate privacy leakage through the synthetic dataset.

Title: Privacy-Preserving Federated Unlearning with Certified Client Removal

• Authors: Ziyao Liu, Huanyi Ye, Yu Jiang, Jiyuan Shen, Jiale Guo, Ivan Tjuawinata, Kwok-Yan Lam
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09724
• Pdf URL: https://arxiv.org/pdf/2404.09724
• Copy Paste: [[2404.09724]] Privacy-Preserving Federated Unlearning with Certified Client Removal(https://arxiv.org/abs/2404.09724)
• Keywords: security, privacy, federate
• Abstract: In recent years, Federated Unlearning (FU) has gained attention for addressing the removal of a client's influence from the global model in Federated Learning (FL) systems, thereby ensuring the ``right to be forgotten" (RTBF). State-of-the-art methods for unlearning use historical data from FL clients, such as gradients or locally trained models. However, studies have revealed significant information leakage in this setting, with the possibility of reconstructing a user's local data from their uploaded information. Addressing this, we propose Starfish, a privacy-preserving federated unlearning scheme using Two-Party Computation (2PC) techniques and shared historical client data between two non-colluding servers. Starfish builds upon existing FU methods to ensure privacy in unlearning processes. To enhance the efficiency of privacy-preserving FU evaluations, we suggest 2PC-friendly alternatives for certain FU algorithm operations. We also implement strategies to reduce costs associated with 2PC operations and lessen cumulative approximation errors. Moreover, we establish a theoretical bound for the difference between the unlearned global model via Starfish and a global model retrained from scratch for certified client removal. Our theoretical and experimental analyses demonstrate that Starfish achieves effective unlearning with reasonable efficiency, maintaining privacy and security in FL systems.

Title: Convergence Analysis of Probability Flow ODE for Score-based Generative Models

• Authors: Daniel Zhengyu Huang, Jiaoyang Huang, Zhengjiang Lin
• Subjects: cs.LG, math.CA, math.NA
• Abstract URL: https://arxiv.org/abs/2404.09730
• Pdf URL: https://arxiv.org/pdf/2404.09730
• Copy Paste: [[2404.09730]] Convergence Analysis of Probability Flow ODE for Score-based Generative Models(https://arxiv.org/abs/2404.09730)
• Keywords: generative
• Abstract: Score-based generative models have emerged as a powerful approach for sampling high-dimensional probability distributions. Despite their effectiveness, their theoretical underpinnings remain relatively underdeveloped. In this work, we study the convergence properties of deterministic samplers based on probability flow ODEs from both theoretical and numerical perspectives. Assuming access to $L^2$-accurate estimates of the score function, we prove the total variation between the target and the generated data distributions can be bounded above by $\mathcal{O}(d\sqrt{\delta})$ in the continuous time level, where $d$ denotes the data dimension and $\delta$ represents the $L^2$-score matching error. For practical implementations using a $p$-th order Runge-Kutta integrator with step size $h$, we establish error bounds of $\mathcal{O}(d(\sqrt{\delta} + (dh)^p))$ at the discrete level. Finally, we present numerical studies on problems up to $128$ dimensions to verify our theory, which indicate a better score matching error and dimension dependence.

Title: Photo-Realistic Image Restoration in the Wild with Controlled Vision-Language Models

• Authors: Ziwei Luo, Fredrik K. Gustafsson, Zheng Zhao, Jens Sjölund, Thomas B. Schön
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09732
• Pdf URL: https://arxiv.org/pdf/2404.09732
• Copy Paste: [[2404.09732]] Photo-Realistic Image Restoration in the Wild with Controlled Vision-Language Models(https://arxiv.org/abs/2404.09732)
• Keywords: robust, diffusion
• Abstract: Though diffusion models have been successfully applied to various image restoration (IR) tasks, their performance is sensitive to the choice of training datasets. Typically, diffusion models trained in specific datasets fail to recover images that have out-of-distribution degradations. To address this problem, this work leverages a capable vision-language model and a synthetic degradation pipeline to learn image restoration in the wild (wild IR). More specifically, all low-quality images are simulated with a synthetic degradation pipeline that contains multiple common degradations such as blur, resize, noise, and JPEG compression. Then we introduce robust training for a degradation-aware CLIP model to extract enriched image content features to assist high-quality image restoration. Our base diffusion model is the image restoration SDE (IR-SDE). Built upon it, we further present a posterior sampling strategy for fast noise-free image generation. We evaluate our model on both synthetic and real-world degradation datasets. Moreover, experiments on the unified image restoration task illustrate that the proposed posterior sampling improves image generation quality for various degradations.

Title: Equipping Diffusion Models with Differentiable Spatial Entropy for Low-Light Image Enhancement

• Authors: Wenyi Lian, Wenjing Lian, Ziwei Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09735
• Pdf URL: https://arxiv.org/pdf/2404.09735
• Copy Paste: [[2404.09735]] Equipping Diffusion Models with Differentiable Spatial Entropy for Low-Light Image Enhancement(https://arxiv.org/abs/2404.09735)
• Keywords: diffusion
• Abstract: Image restoration, which aims to recover high-quality images from their corrupted counterparts, often faces the challenge of being an ill-posed problem that allows multiple solutions for a single input. However, most deep learning based works simply employ l1 loss to train their network in a deterministic way, resulting in over-smoothed predictions with inferior perceptual quality. In this work, we propose a novel method that shifts the focus from a deterministic pixel-by-pixel comparison to a statistical perspective, emphasizing the learning of distributions rather than individual pixel values. The core idea is to introduce spatial entropy into the loss function to measure the distribution difference between predictions and targets. To make this spatial entropy differentiable, we employ kernel density estimation (KDE) to approximate the probabilities for specific intensity values of each pixel with their neighbor areas. Specifically, we equip the entropy with diffusion models and aim for superior accuracy and enhanced perceptual quality over l1 based noise matching loss. In the experiments, we evaluate the proposed method for low light enhancement on two datasets and the NTIRE challenge 2024. All these results illustrate the effectiveness of our statistic-based entropy loss. Code is available at https://github.com/shermanlian/spatial-entropy-loss.

Title: FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features

• Authors: Andre Rochow, Max Schwarz, Sven Behnke
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09736
• Pdf URL: https://arxiv.org/pdf/2404.09736
• Copy Paste: [[2404.09736]] FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features(https://arxiv.org/abs/2404.09736)
• Keywords: transformer
• Abstract: The task of face reenactment is to transfer the head motion and facial expressions from a driving video to the appearance of a source image, which may be of a different person (cross-reenactment). Most existing methods are CNN-based and estimate optical flow from the source image to the current driving frame, which is then inpainted and refined to produce the output animation. We propose a transformer-based encoder for computing a set-latent representation of the source image(s). We then predict the output color of a query pixel using a transformer-based decoder, which is conditioned with keypoints and a facial expression vector extracted from the driving frame. Latent representations of the source person are learned in a self-supervised manner that factorize their appearance, head pose, and facial expressions. Thus, they are perfectly suited for cross-reenactment. In contrast to most related work, our method naturally extends to multiple source images and can thus adapt to person-specific facial dynamics. We also propose data augmentation and regularization schemes that are necessary to prevent overfitting and support generalizability of the learned representations. We evaluated our approach in a randomized user study. The results indicate superior performance compared to the state-of-the-art in terms of motion transfer quality and temporal consistency.

Title: Quantization of Large Language Models with an Overdetermined Basis

• Authors: Daniil Merkulov, Daria Cherniuk, Alexander Rudikov, Ivan Oseledets, Ekaterina Muravleva, Aleksandr Mikhalev, Boris Kashin
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09737
• Pdf URL: https://arxiv.org/pdf/2404.09737
• Copy Paste: [[2404.09737]] Quantization of Large Language Models with an Overdetermined Basis(https://arxiv.org/abs/2404.09737)
• Keywords: large language model
• Abstract: In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or tensor into two factors. The first factor maintains a small infinity norm, while the second exhibits a similarly constrained norm when multiplied by an orthogonal matrix. Surprisingly, the entries of factors after decomposition are well-concentrated around several peaks, which allows us to efficiently replace them with corresponding centroids for quantization purposes. We study the theoretical properties of the proposed approach and rigorously evaluate our compression algorithm in the context of next-word prediction tasks and on a set of downstream tasks for text classification. Our findings demonstrate that Kashin Quantization achieves competitive or superior quality in model performance while ensuring data compression, marking a significant advancement in the field of data quantization.

Title: Can We Break Free from Strong Data Augmentations in Self-Supervised Learning?

• Authors: Shruthi Gowda, Elahe Arani, Bahram Zonooz
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09752
• Pdf URL: https://arxiv.org/pdf/2404.09752
• Copy Paste: [[2404.09752]] Can We Break Free from Strong Data Augmentations in Self-Supervised Learning?(https://arxiv.org/abs/2404.09752)
• Keywords: robust
• Abstract: Self-supervised learning (SSL) has emerged as a promising solution for addressing the challenge of limited labeled data in deep neural networks (DNNs), offering scalability potential. However, the impact of design dependencies within the SSL framework remains insufficiently investigated. In this study, we comprehensively explore SSL behavior across a spectrum of augmentations, revealing their crucial role in shaping SSL model performance and learning mechanisms. Leveraging these insights, we propose a novel learning approach that integrates prior knowledge, with the aim of curtailing the need for extensive data augmentations and thereby amplifying the efficacy of learned representations. Notably, our findings underscore that SSL models imbued with prior knowledge exhibit reduced texture bias, diminished reliance on shortcuts and augmentations, and improved robustness against both natural and adversarial corruptions. These findings not only illuminate a new direction in SSL research, but also pave the way for enhancing DNN performance while concurrently alleviating the imperative for intensive data augmentation, thereby enhancing scalability and real-world problem-solving capabilities.

Title: Personalized Collaborative Fine-Tuning for On-Device Large Language Models

• Authors: Nicolas Wagner, Dongyang Fan, Martin Jaggi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09753
• Pdf URL: https://arxiv.org/pdf/2404.09753
• Copy Paste: [[2404.09753]] Personalized Collaborative Fine-Tuning for On-Device Large Language Models(https://arxiv.org/abs/2404.09753)
• Keywords: large language model
• Abstract: We explore on-device self-supervised collaborative fine-tuning of large language models with limited local data availability. Taking inspiration from the collaborative learning community, we introduce three distinct trust-weighted gradient aggregation schemes: weight similarity-based, prediction similarity-based and validation performance-based. To minimize communication overhead, we integrate Low-Rank Adaptation (LoRA) and only exchange LoRA weight updates. Our protocols, driven by prediction and performance metrics, surpass both FedAvg and local fine-tuning methods, which is particularly evident in realistic scenarios with more diverse local data distributions. The results underscore the effectiveness of our approach in addressing heterogeneity and scarcity within local datasets.

Title: Resilience of Large Language Models for Noisy Instructions

• Authors: Bin Wang, Chengwei Wei, Zhengyuan Liu, Geyu Lin, Nancy F. Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09754
• Pdf URL: https://arxiv.org/pdf/2404.09754
• Copy Paste: [[2404.09754]] Resilience of Large Language Models for Noisy Instructions(https://arxiv.org/abs/2404.09754)
• Keywords: large language model
• Abstract: As the rapidly advancing domain of natural language processing (NLP), large language models (LLMs) have emerged as powerful tools for interpreting human commands and generating text across various tasks. Nonetheless, the resilience of LLMs to handle text containing inherent errors, stemming from human interactions and collaborative systems, has not been thoroughly explored. Our study investigates the resilience of LLMs against five common types of disruptions including 1) ASR (Automatic Speech Recognition) errors, 2) OCR (Optical Character Recognition) errors, 3) grammatical mistakes, 4) typographical errors, and 5) distractive content. We aim to investigate how these models react by deliberately embedding these errors into instructions. Our findings reveal that while some LLMs show a degree of resistance to certain types of noise, their overall performance significantly suffers. This emphasizes the importance of further investigation into enhancing model resilience. In response to the observed decline in performance, our study also evaluates a "re-pass" strategy, designed to purify the instructions of noise before the LLMs process them. Our analysis indicates that correcting noisy instructions, particularly for open-source LLMs, presents significant challenges.

Title: KG-CTG: Citation Generation through Knowledge Graph-guided Large Language Models

• Authors: Avinash Anand, Mohit Gupta, Kritarth Prasad, Ujjwal Goel, Naman Lal, Astha Verma, Rajiv Ratn Shah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09763
• Pdf URL: https://arxiv.org/pdf/2404.09763
• Copy Paste: [[2404.09763]] KG-CTG: Citation Generation through Knowledge Graph-guided Large Language Models(https://arxiv.org/abs/2404.09763)
• Keywords: large language model
• Abstract: Citation Text Generation (CTG) is a task in natural language processing (NLP) that aims to produce text that accurately cites or references a cited document within a source document. In CTG, the generated text draws upon contextual cues from both the source document and the cited paper, ensuring accurate and relevant citation information is provided. Previous work in the field of citation generation is mainly based on the text summarization of documents. Following this, this paper presents a framework, and a comparative study to demonstrate the use of Large Language Models (LLMs) for the task of citation generation. Also, we have shown the improvement in the results of citation generation by incorporating the knowledge graph relations of the papers in the prompt for the LLM to better learn the relationship between the papers. To assess how well our model is performing, we have used a subset of standard S2ORC dataset, which only consists of computer science academic research papers in the English Language. Vicuna performs best for this task with 14.15 Meteor, 12.88 Rouge-1, 1.52 Rouge-2, and 10.94 Rouge-L. Also, Alpaca performs best, and improves the performance by 36.98% in Rouge-1, and 33.14% in Meteor by including knowledge graphs.

Title: Contrastive Pretraining for Visual Concept Explanations of Socioeconomic Outcomes

• Authors: Ivica Obadic, Alex Levering, Lars Pennig, Dario Oliveira, Diego Marcos, Xiaoxiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09768
• Pdf URL: https://arxiv.org/pdf/2404.09768
• Copy Paste: [[2404.09768]] Contrastive Pretraining for Visual Concept Explanations of Socioeconomic Outcomes(https://arxiv.org/abs/2404.09768)
• Keywords: interpretability, explainability
• Abstract: Predicting socioeconomic indicators from satellite imagery with deep learning has become an increasingly popular research direction. Post-hoc concept-based explanations can be an important step towards broader adoption of these models in policy-making as they enable the interpretation of socioeconomic outcomes based on visual concepts that are intuitive to humans. In this paper, we study the interplay between representation learning using an additional task-specific contrastive loss and post-hoc concept explainability for socioeconomic studies. Our results on two different geographical locations and tasks indicate that the task-specific pretraining imposes a continuous ordering of the latent space embeddings according to the socioeconomic outcomes. This improves the model's interpretability as it enables the latent space of the model to associate urban concepts with continuous intervals of socioeconomic outcomes. Further, we illustrate how analyzing the model's conceptual sensitivity for the intervals of socioeconomic outcomes can shed light on new insights for urban studies.

Title: The Devil is in the Few Shots: Iterative Visual Knowledge Completion for Few-shot Learning

• Authors: Yaohui Li, Qifeng Zhou, Haoxing Chen, Jianbing Zhang, Xinyu Dai, Hao Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09778
• Pdf URL: https://arxiv.org/pdf/2404.09778
• Copy Paste: [[2404.09778]] The Devil is in the Few Shots: Iterative Visual Knowledge Completion for Few-shot Learning(https://arxiv.org/abs/2404.09778)
• Keywords: robust, generative
• Abstract: Contrastive Language-Image Pre-training (CLIP) has shown powerful zero-shot learning performance. Few-shot learning aims to further enhance the transfer capability of CLIP by giving few images in each class, aka 'few shots'. Most existing methods either implicitly learn from the few shots by incorporating learnable prompts or adapters, or explicitly embed them in a cache model for inference. However, the narrow distribution of few shots often contains incomplete class information, leading to biased visual knowledge with high risk of misclassification. To tackle this problem, recent methods propose to supplement visual knowledge by generative models or extra databases, which can be costly and time-consuming. In this paper, we propose an Iterative Visual Knowledge CompLetion (KCL) method to complement visual knowledge by properly taking advantages of unlabeled samples without access to any auxiliary or synthetic data. Specifically, KCL first measures the similarities between unlabeled samples and each category. Then, the samples with top confidence to each category is selected and collected by a designed confidence criterion. Finally, the collected samples are treated as labeled ones and added to few shots to jointly re-estimate the remaining unlabeled ones. The above procedures will be repeated for a certain number of iterations with more and more samples being collected until convergence, ensuring a progressive and robust knowledge completion process. Extensive experiments on 11 benchmark datasets demonstrate the effectiveness and efficiency of KCL as a plug-and-play module under both few-shot and zero-shot learning settings. Code is available at https://github.com/Mark-Sky/KCL.

Title: Benchmarking Llama2, Mistral, Gemma and GPT for Factuality, Toxicity, Bias and Propensity for Hallucinations

• Authors: David Nadeau, Mike Kroutikov, Karen McNeil, Simon Baribeau
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09785
• Pdf URL: https://arxiv.org/pdf/2404.09785
• Copy Paste: [[2404.09785]] Benchmarking Llama2, Mistral, Gemma and GPT for Factuality, Toxicity, Bias and Propensity for Hallucinations(https://arxiv.org/abs/2404.09785)
• Keywords: large language model
• Abstract: This paper introduces fourteen novel datasets for the evaluation of Large Language Models' safety in the context of enterprise tasks. A method was devised to evaluate a model's safety, as determined by its ability to follow instructions and output factual, unbiased, grounded, and appropriate content. In this research, we used OpenAI GPT as point of comparison since it excels at all levels of safety. On the open-source side, for smaller models, Meta Llama2 performs well at factuality and toxicity but has the highest propensity for hallucination. Mistral hallucinates the least but cannot handle toxicity well. It performs well in a dataset mixing several tasks and safety vectors in a narrow vertical domain. Gemma, the newly introduced open-source model based on Google Gemini, is generally balanced but trailing behind. When engaging in back-and-forth conversation (multi-turn prompts), we find that the safety of open-source models degrades significantly. Aside from OpenAI's GPT, Mistral is the only model that still performed well in multi-turn tests.

Title: Shape Arithmetic Expressions: Advancing Scientific Discovery Beyond Closed-Form Equations

• Authors: Krzysztof Kacprzyk, Mihaela van der Schaar
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.09788
• Pdf URL: https://arxiv.org/pdf/2404.09788
• Copy Paste: [[2404.09788]] Shape Arithmetic Expressions: Advancing Scientific Discovery Beyond Closed-Form Equations(https://arxiv.org/abs/2404.09788)
• Keywords: interpretability
• Abstract: Symbolic regression has excelled in uncovering equations from physics, chemistry, biology, and related disciplines. However, its effectiveness becomes less certain when applied to experimental data lacking inherent closed-form expressions. Empirically derived relationships, such as entire stress-strain curves, may defy concise closed-form representation, compelling us to explore more adaptive modeling approaches that balance flexibility with interpretability. In our pursuit, we turn to Generalized Additive Models (GAMs), a widely used class of models known for their versatility across various domains. Although GAMs can capture non-linear relationships between variables and targets, they cannot capture intricate feature interactions. In this work, we investigate both of these challenges and propose a novel class of models, Shape Arithmetic Expressions (SHAREs), that fuses GAM's flexible shape functions with the complex feature interactions found in mathematical expressions. SHAREs also provide a unifying framework for both of these approaches. We also design a set of rules for constructing SHAREs that guarantee transparency of the found expressions beyond the standard constraints based on the model's size.

Title: TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding

• Authors: Bozhi Luan, Hao Feng, Hong Chen, Yonghui Wang, Wengang Zhou, Houqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09797
• Pdf URL: https://arxiv.org/pdf/2404.09797
• Copy Paste: [[2404.09797]] TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding(https://arxiv.org/abs/2404.09797)
• Keywords: extraction
• Abstract: The advent of Large Multimodal Models (LMMs) has sparked a surge in research aimed at harnessing their remarkable reasoning abilities. However, for understanding text-rich images, challenges persist in fully leveraging the potential of LMMs, and existing methods struggle with effectively processing high-resolution images. In this work, we propose TextCoT, a novel Chain-of-Thought framework for text-rich image understanding. TextCoT utilizes the captioning ability of LMMs to grasp the global context of the image and the grounding capability to examine local textual regions. This allows for the extraction of both global and local visual information, facilitating more accurate question-answering. Technically, TextCoT consists of three stages, including image overview, coarse localization, and fine-grained observation. The image overview stage provides a comprehensive understanding of the global scene information, and the coarse localization stage approximates the image area containing the answer based on the question asked. Then, integrating the obtained global image descriptions, the final stage further examines specific regions to provide accurate answers. Our method is free of extra training, offering immediate plug-and-play functionality. Extensive experiments are conducted on a series of text-rich image question-answering benchmark datasets based on several advanced LMMs, and the results demonstrate the effectiveness and strong generalization ability of our method. Code is available at https://github.com/bzluan/TextCoT.

Title: The Performance of Sequential Deep Learning Models in Detecting Phishing Websites Using Contextual Features of URLs

• Authors: Saroj Gopali, Akbar S. Namin, Faranak Abri, Keith S. Jones
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09802
• Pdf URL: https://arxiv.org/pdf/2404.09802
• Copy Paste: [[2404.09802]] The Performance of Sequential Deep Learning Models in Detecting Phishing Websites Using Contextual Features of URLs(https://arxiv.org/abs/2404.09802)
• Keywords: protect, attack, steal
• Abstract: Cyber attacks continue to pose significant threats to individuals and organizations, stealing sensitive data such as personally identifiable information, financial information, and login credentials. Hence, detecting malicious websites before they cause any harm is critical to preventing fraud and monetary loss. To address the increasing number of phishing attacks, protective mechanisms must be highly responsive, adaptive, and scalable. Fortunately, advances in the field of machine learning, coupled with access to vast amounts of data, have led to the adoption of various deep learning models for timely detection of these cyber crimes. This study focuses on the detection of phishing websites using deep learning models such as Multi-Head Attention, Temporal Convolutional Network (TCN), BI-LSTM, and LSTM where URLs of the phishing websites are treated as a sequence. The results demonstrate that Multi-Head Attention and BI-LSTM model outperform some other deep learning-based algorithms such as TCN and LSTM in producing better precision, recall, and F1-scores.

Title: A Universal Protocol to Benchmark Camera Calibration for Sports

• Authors: Floriane Magera, Thomas Hoyoux, Olivier Barnich, Marc Van Droogenbroeck
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09807
• Pdf URL: https://arxiv.org/pdf/2404.09807
• Copy Paste: [[2404.09807]] A Universal Protocol to Benchmark Camera Calibration for Sports(https://arxiv.org/abs/2404.09807)
• Keywords: fair
• Abstract: Camera calibration is a crucial component in the realm of sports analytics, as it serves as the foundation to extract 3D information out of the broadcast images. Despite the significance of camera calibration research in sports analytics, progress is impeded by outdated benchmarking criteria. Indeed, the annotation data and evaluation metrics provided by most currently available benchmarks strongly favor and incite the development of sports field registration methods, i.e. methods estimating homographies that map the sports field plane to the image plane. However, such homography-based methods are doomed to overlook the broader capabilities of camera calibration in bridging the 3D world to the image. In particular, real-world non-planar sports field elements (such as goals, corner flags, baskets, ...) and image distortion caused by broadcast camera lenses are out of the scope of sports field registration methods. To overcome these limitations, we designed a new benchmarking protocol, named ProCC, based on two principles: (1) the protocol should be agnostic to the camera model chosen for a camera calibration method, and (2) the protocol should fairly evaluate camera calibration methods using the reprojection of arbitrary yet accurately known 3D objects. Indirectly, we also provide insights into the metric used in SoccerNet-calibration, which solely relies on image annotation data of viewed 3D objects as ground truth, thus implementing our protocol. With experiments on the World Cup 2014, CARWC, and SoccerNet datasets, we show that our benchmarking protocol provides fairer evaluations of camera calibration methods. By defining our requirements for proper benchmarking, we hope to pave the way for a new stage in camera calibration for sports applications with high accuracy standards.

Title: FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity

• Authors: Kai Yi, Nidham Gazagnadou, Peter Richtárik, Lingjuan Lyu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09816
• Pdf URL: https://arxiv.org/pdf/2404.09816
• Copy Paste: [[2404.09816]] FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity(https://arxiv.org/abs/2404.09816)
• Keywords: secure, privacy, federate
• Abstract: The interest in federated learning has surged in recent research due to its unique ability to train a global model using privacy-secured information held locally on each client. This paper pays particular attention to the issue of client-side model heterogeneity, a pervasive challenge in the practical implementation of FL that escalates its complexity. Assuming a scenario where each client possesses varied memory storage, processing capabilities and network bandwidth - a phenomenon referred to as system heterogeneity - there is a pressing need to customize a unique model for each client. In response to this, we present an effective and adaptable federated framework FedP3, representing Federated Personalized and Privacy-friendly network Pruning, tailored for model heterogeneity scenarios. Our proposed methodology can incorporate and adapt well-established techniques to its specific instances. We offer a theoretical interpretation of FedP3 and its locally differential-private variant, DP-FedP3, and theoretically validate their efficiencies.

Title: Impact of Preference Noise on the Alignment Performance of Generative Language Models

• Authors: Yang Gao, Dana Alon, Donald Metzler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09824
• Pdf URL: https://arxiv.org/pdf/2404.09824
• Copy Paste: [[2404.09824]] Impact of Preference Noise on the Alignment Performance of Generative Language Models(https://arxiv.org/abs/2404.09824)
• Keywords: generative
• Abstract: A key requirement in developing Generative Language Models (GLMs) is to have their values aligned with human values. Preference-based alignment is a widely used paradigm for this purpose, in which preferences over generation pairs are first elicited from human annotators or AI systems, and then fed into some alignment techniques, e.g., Direct Preference Optimization. However, a substantial percent (20 - 40%) of the preference pairs used in GLM alignment are noisy, and it remains unclear how the noise affects the alignment performance and how to mitigate its negative impact. In this paper, we propose a framework to inject desirable amounts and types of noise to the preferences, and systematically study the impact of preference noise on the alignment performance in two tasks (summarization and dialogue generation). We find that the alignment performance can be highly sensitive to the noise rates in the preference data: e.g., a 10 percentage points (pp) increase of the noise rate can lead to 30 pp drop in the alignment performance (in win rate). To mitigate the impact of noise, confidence-based data filtering shows significant benefit when certain types of noise are present. We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.

Title: A Recipe for CAC: Mosaic-based Generalized Loss for Improved Class-Agnostic Counting

• Authors: Tsung-Han Chou, Brian Wang, Wei-Chen Chiu, Jun-Cheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09826
• Pdf URL: https://arxiv.org/pdf/2404.09826
• Copy Paste: [[2404.09826]] A Recipe for CAC: Mosaic-based Generalized Loss for Improved Class-Agnostic Counting(https://arxiv.org/abs/2404.09826)
• Keywords: robust, fair
• Abstract: Class agnostic counting (CAC) is a vision task that can be used to count the total occurrence number of any given reference objects in the query image. The task is usually formulated as a density map estimation problem through similarity computation among a few image samples of the reference object and the query image. In this paper, we point out a severe issue of the existing CAC framework: Given a multi-class setting, models don't consider reference images and instead blindly match all dominant objects in the query image. Moreover, the current evaluation metrics and dataset cannot be used to faithfully assess the model's generalization performance and robustness. To this end, we discover that the combination of mosaic augmentation with generalized loss is essential for addressing the aforementioned issue of CAC models to count objects of majority (i.e. dominant objects) regardless of the references. Furthermore, we introduce a new evaluation protocol and metrics for resolving the problem behind the existing CAC evaluation scheme and better benchmarking CAC models in a more fair manner. Besides, extensive evaluation results demonstrate that our proposed recipe can consistently improve the performance of different CAC models. The code will be released upon acceptance.

Title: Negation Triplet Extraction with Syntactic Dependency and Semantic Consistency

• Authors: Yuchen Shi, Deqing Yang, Jingping Liu, Yanghua Xiao, Zongyu Wang, Huimin Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09830
• Pdf URL: https://arxiv.org/pdf/2404.09830
• Copy Paste: [[2404.09830]] Negation Triplet Extraction with Syntactic Dependency and Semantic Consistency(https://arxiv.org/abs/2404.09830)
• Keywords: extraction, generative
• Abstract: Previous works of negation understanding mainly focus on negation cue detection and scope resolution, without identifying negation subject which is also significant to the downstream tasks. In this paper, we propose a new negation triplet extraction (NTE) task which aims to extract negation subject along with negation cue and scope. To achieve NTE, we devise a novel Syntax&Semantic-Enhanced Negation Extraction model, namely SSENE, which is built based on a generative pretrained language model (PLM) {of Encoder-Decoder architecture} with a multi-task learning framework. Specifically, the given sentence's syntactic dependency tree is incorporated into the PLM's encoder to discover the correlations between the negation subject, cue and scope. Moreover, the semantic consistency between the sentence and the extracted triplet is ensured by an auxiliary task learning. Furthermore, we have constructed a high-quality Chinese dataset NegComment based on the users' reviews from the real-world platform of Meituan, upon which our evaluations show that SSENE achieves the best NTE performance compared to the baselines. Our ablation and case studies also demonstrate that incorporating the syntactic information helps the PLM's recognize the distant dependency between the subject and cue, and the auxiliary task learning is helpful to extract the negation triplets with more semantic consistency.

Title: Digging into contrastive learning for robust depth estimation with diffusion models

• Authors: Jiyuan Wang, Chunyu Lin, Lang Nie, Kang Liao, Shuwei Shao, Yao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09831
• Pdf URL: https://arxiv.org/pdf/2404.09831
• Copy Paste: [[2404.09831]] Digging into contrastive learning for robust depth estimation with diffusion models(https://arxiv.org/abs/2404.09831)
• Keywords: robust, diffusion
• Abstract: Recently, diffusion-based depth estimation methods have drawn widespread attention due to their elegant denoising patterns and promising performance. However, they are typically unreliable under adverse conditions prevalent in real-world scenarios, such as rainy, snowy, etc. In this paper, we propose a novel robust depth estimation method called D4RD, featuring a custom contrastive learning mode tailored for diffusion models to mitigate performance degradation in complex environments. Concretely, we integrate the strength of knowledge distillation into contrastive learning, building the `trinity' contrastive scheme. This scheme utilizes the sampled noise of the forward diffusion process as a natural reference, guiding the predicted noise in diverse scenes toward a more stable and precise optimum. Moreover, we extend noise-level trinity to encompass more generic feature and image levels, establishing a multi-level contrast to distribute the burden of robust perception across the overall network. Before addressing complex scenarios, we enhance the stability of the baseline diffusion model with three straightforward yet effective improvements, which facilitate convergence and remove depth outliers. Extensive experiments demonstrate that D4RD surpasses existing state-of-the-art solutions on synthetic corruption datasets and real-world weather conditions. The code for D4RD will be made available for further exploration and adoption.

Title: A Diffusion-based Data Generator for Training Object Recognition Models in Ultra-Range Distance

• Authors: Eran Bamani, Eden Nissinman, Lisa Koenigsberg, Inbar Meir, Avishai Sintov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09846
• Pdf URL: https://arxiv.org/pdf/2404.09846
• Copy Paste: [[2404.09846]] A Diffusion-based Data Generator for Training Object Recognition Models in Ultra-Range Distance(https://arxiv.org/abs/2404.09846)
• Keywords: diffusion, generative
• Abstract: Object recognition, commonly performed by a camera, is a fundamental requirement for robots to complete complex tasks. Some tasks require recognizing objects far from the robot's camera. A challenging example is Ultra-Range Gesture Recognition (URGR) in human-robot interaction where the user exhibits directive gestures at a distance of up to 25~m from the robot. However, training a model to recognize hardly visible objects located in ultra-range requires an exhaustive collection of a significant amount of labeled samples. The generation of synthetic training datasets is a recent solution to the lack of real-world data, while unable to properly replicate the realistic visual characteristics of distant objects in images. In this letter, we propose the Diffusion in Ultra-Range (DUR) framework based on a Diffusion model to generate labeled images of distant objects in various scenes. The DUR generator receives a desired distance and class (e.g., gesture) and outputs a corresponding synthetic image. We apply DUR to train a URGR model with directive gestures in which fine details of the gesturing hand are challenging to distinguish. DUR is compared to other types of generative models showcasing superiority both in fidelity and in recognition success rate when training a URGR model. More importantly, training a DUR model on a limited amount of real data and then using it to generate synthetic data for training a URGR model outperforms directly training the URGR model on real data. The synthetic-based URGR model is also demonstrated in gesture-based direction of a ground robot.

Title: Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL

• Authors: Fangwei Zhong, Kui Wu, Hai Ci, Churan Wang, Hao Chen
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.09857
• Pdf URL: https://arxiv.org/pdf/2404.09857
• Copy Paste: [[2404.09857]] Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL(https://arxiv.org/abs/2404.09857)
• Keywords: robust, segmentation
• Abstract: Embodied visual tracking is to follow a target object in dynamic 3D environments using an agent's egocentric vision. This is a vital and challenging skill for embodied agents. However, existing methods suffer from inefficient training and poor generalization. In this paper, we propose a novel framework that combines visual foundation models (VFM) and offline reinforcement learning (offline RL) to empower embodied visual tracking. We use a pre-trained VFM, such as ``Tracking Anything", to extract semantic segmentation masks with text prompts. We then train a recurrent policy network with offline RL, e.g., Conservative Q-Learning, to learn from the collected demonstrations without online agent-environment interactions. To further improve the robustness and generalization of the policy network, we also introduce a mask re-targeting mechanism and a multi-level data collection strategy. In this way, we can train a robust tracker within an hour on a consumer-level GPU, e.g., Nvidia RTX 3090. Such efficiency is unprecedented for RL-based visual tracking methods. We evaluate our tracker on several high-fidelity environments with challenging situations, such as distraction and occlusion. The results show that our agent outperforms state-of-the-art methods in terms of sample efficiency, robustness to distractors, and generalization to unseen scenarios and targets. We also demonstrate the transferability of the learned tracker from the virtual world to real-world scenarios.

Title: Unsupervised Federated Optimization at the Edge: D2D-Enabled Learning without Labels

• Authors: Satyavrat Wagle, Seyyedali Hosseinalipour, Naji Khosravan, Christopher G. Brinton
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09861
• Pdf URL: https://arxiv.org/pdf/2404.09861
• Copy Paste: [[2404.09861]] Unsupervised Federated Optimization at the Edge: D2D-Enabled Learning without Labels(https://arxiv.org/abs/2404.09861)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) is a popular solution for distributed machine learning (ML). While FL has traditionally been studied for supervised ML tasks, in many applications, it is impractical to assume availability of labeled data across devices. To this end, we develop Cooperative Federated unsupervised Contrastive Learning ({\tt CF-CL)} to facilitate FL across edge devices with unlabeled datasets. {\tt CF-CL} employs local device cooperation where either explicit (i.e., raw data) or implicit (i.e., embeddings) information is exchanged through device-to-device (D2D) communications to improve local diversity. Specifically, we introduce a \textit{smart information push-pull} methodology for data/embedding exchange tailored to FL settings with either soft or strict data privacy restrictions. Information sharing is conducted through a probabilistic importance sampling technique at receivers leveraging a carefully crafted reserve dataset provided by transmitters. In the implicit case, embedding exchange is further integrated into the local ML training at the devices via a regularization term incorporated into the contrastive loss, augmented with a dynamic contrastive margin to adjust the volume of latent space explored. Numerical evaluations demonstrate that {\tt CF-CL} leads to alignment of latent spaces learned across devices, results in faster and more efficient global model training, and is effective in extreme non-i.i.d. data distribution settings across devices.

Title: Explainable Online Unsupervised Anomaly Detection for Cyber-Physical Systems via Causal Discovery from Time Series

• Authors: Daniele Meli
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2404.09871
• Pdf URL: https://arxiv.org/pdf/2404.09871
• Copy Paste: [[2404.09871]] Explainable Online Unsupervised Anomaly Detection for Cyber-Physical Systems via Causal Discovery from Time Series(https://arxiv.org/abs/2404.09871)
• Keywords: explainability
• Abstract: Online unsupervised detection of anomalies is crucial to guarantee the correct operation of cyber-physical systems and the safety of humans interacting with them. State-of-the-art approaches based on deep learning via neural networks achieve outstanding performance at anomaly recognition, evaluating the discrepancy between a normal model of the system (with no anomalies) and the real-time stream of sensor time series. However, large training data and time are typically required, and explainability is still a challenge to identify the root of the anomaly and implement predictive maintainance. In this paper, we use causal discovery to learn a normal causal graph of the system, and we evaluate the persistency of causal links during real-time acquisition of sensor data to promptly detect anomalies. On two benchmark anomaly detection datasets, we show that our method has higher training efficiency, outperforms the accuracy of state-of-the-art neural architectures and correctly identifies the sources of $>10$ different anomalies. The code for experimental replication is at this http URL

Title: Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection

• Authors: Yuxi Li, Yi Liu, Gelei Deng, Ying Zhang, Wenjia Song, Ling Shi, Kailong Wang, Yuekang Li, Yang Liu, Haoyu Wang
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.09894
• Pdf URL: https://arxiv.org/pdf/2404.09894
• Copy Paste: [[2404.09894]] Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection(https://arxiv.org/abs/2404.09894)
• Keywords: large language model
• Abstract: With the expanding application of Large Language Models (LLMs) in various domains, it becomes imperative to comprehensively investigate their unforeseen behaviors and consequent outcomes. In this study, we introduce and systematically explore the phenomenon of "glitch tokens", which are anomalous tokens produced by established tokenizers and could potentially compromise the models' quality of response. Specifically, we experiment on seven top popular LLMs utilizing three distinct tokenizers and involving a totally of 182,517 tokens. We present categorizations of the identified glitch tokens and symptoms exhibited by LLMs when interacting with glitch tokens. Based on our observation that glitch tokens tend to cluster in the embedding space, we propose GlitchHunter, a novel iterative clustering-based technique, for efficient glitch token detection. The evaluation shows that our approach notably outperforms three baseline methods on eight open-source LLMs. To the best of our knowledge, we present the first comprehensive study on glitch tokens. Our new detection further provides valuable insights into mitigating tokenization-related errors in LLMs.

Title: Larger-scale Nakamoto-style Blockchains Don't Necessarily Offer Better Security

• Authors: Jannik Albrecht, Sebastien Andreina, Frederik Armknecht, Ghassan Karame, Giorgia Marson, Julian Willingmann
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.09895
• Pdf URL: https://arxiv.org/pdf/2404.09895
• Copy Paste: [[2404.09895]] Larger-scale Nakamoto-style Blockchains Don't Necessarily Offer Better Security(https://arxiv.org/abs/2404.09895)
• Keywords: security, attack
• Abstract: Extensive research on Nakamoto-style consensus protocols has shown that network delays degrade the security of these protocols. Established results indicate that, perhaps surprisingly, maximal security is achieved when the network is as small as two nodes due to increased delays in larger networks. This contradicts the very foundation of blockchains, namely that decentralization improves security. In this paper, we take a closer look at how the network scale affects security of Nakamoto-style blockchains. We argue that a crucial aspect has been neglected in existing security models: the larger the network, the harder it is for an attacker to control a significant amount of power. To this end, we introduce a probabilistic corruption model to express the increasing difficulty for an attacker to corrupt resources in larger networks. Based on our model, we analyze the impact of the number of nodes on the (maximum) network delay and the fraction of adversarial power. In particular, we show that (1) increasing the number of nodes eventually violates security, but (2) relying on a small number of nodes does not provide decent security provisions either. We then validate our analysis by means of an empirical evaluation emulating hundreds of thousands of nodes in deployments such as Bitcoin, Monero, Cardano, and Ethereum Classic. Based on our empirical analysis, we concretely analyze the impact of various real-world parameters and configurations on the consistency bounds in existing deployments and on the adversarial power that can be tolerated while providing security. As far as we are aware, this is the first work that analytically and empirically explores the real-world tradeoffs achieved by current popular Nakamoto-style deployments.

Title: Evaluating the Explainability of Attributes and Prototypes for a Medical Classification Model

• Authors: Luisa Gallée, Catharina Silvia Lisson, Christoph Gerhard Lisson, Daniela Drees, Felix Weig, Daniel Vogele, Meinrad Beer, Michael Götz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09917
• Pdf URL: https://arxiv.org/pdf/2404.09917
• Copy Paste: [[2404.09917]] Evaluating the Explainability of Attributes and Prototypes for a Medical Classification Model(https://arxiv.org/abs/2404.09917)
• Keywords: explainability
• Abstract: Due to the sensitive nature of medicine, it is particularly important and highly demanded that AI methods are explainable. This need has been recognised and there is great research interest in xAI solutions with medical applications. However, there is a lack of user-centred evaluation regarding the actual impact of the explanations. We evaluate attribute- and prototype-based explanations with the Proto-Caps model. This xAI model reasons the target classification with human-defined visual features of the target object in the form of scores and attribute-specific prototypes. The model thus provides a multimodal explanation that is intuitively understandable to humans thanks to predefined attributes. A user study involving six radiologists shows that the explanations are subjectivly perceived as helpful, as they reflect their decision-making process. The results of the model are considered a second opinion that radiologists can discuss using the model's explanations. However, it was shown that the inclusion and increased magnitude of model explanations objectively can increase confidence in the model's predictions when the model is incorrect. We can conclude that attribute scores and visual prototypes enhance confidence in the model. However, additional development and repeated user studies are needed to tailor the explanation to the respective use case.

Title: EdgeRelight360: Text-Conditioned 360-Degree HDR Image Generation for Real-Time On-Device Video Portrait Relighting

• Authors: Min-Hui Lin, Mahesh Reddy, Guillaume Berger, Michel Sarkis, Fatih Porikli, Ning Bi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09918
• Pdf URL: https://arxiv.org/pdf/2404.09918
• Copy Paste: [[2404.09918]] EdgeRelight360: Text-Conditioned 360-Degree HDR Image Generation for Real-Time On-Device Video Portrait Relighting(https://arxiv.org/abs/2404.09918)
• Keywords: privacy, diffusion
• Abstract: In this paper, we present EdgeRelight360, an approach for real-time video portrait relighting on mobile devices, utilizing text-conditioned generation of 360-degree high dynamic range image (HDRI) maps. Our method proposes a diffusion-based text-to-360-degree image generation in the HDR domain, taking advantage of the HDR10 standard. This technique facilitates the generation of high-quality, realistic lighting conditions from textual descriptions, offering flexibility and control in portrait video relighting task. Unlike the previous relighting frameworks, our proposed system performs video relighting directly on-device, enabling real-time inference with real 360-degree HDRI maps. This on-device processing ensures both privacy and guarantees low runtime, providing an immediate response to changes in lighting conditions or user inputs. Our approach paves the way for new possibilities in real-time video applications, including video conferencing, gaming, and augmented reality, by allowing dynamic, text-based control of lighting conditions.

Title: Foundational Challenges in Assuring Alignment and Safety of Large Language Models

• Authors: Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hÉigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Yoshua Bengio, Danqi Chen, Samuel Albanie, Tegan Maharaj, Jakob Foerster, Florian Tramer, He He, Atoosa Kasirzadeh, Yejin Choi, David Krueger
• Subjects: cs.LG, cs.AI, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.09932
• Pdf URL: https://arxiv.org/pdf/2404.09932
• Copy Paste: [[2404.09932]] Foundational Challenges in Assuring Alignment and Safety of Large Language Models(https://arxiv.org/abs/2404.09932)
• Keywords: large language model
• Abstract: This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are organized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose $200+$ concrete research questions.

Title: Compression Represents Intelligence Linearly

• Authors: Yuzhen Huang, Jinghan Zhang, Zifei Shan, Junxian He
• Subjects: cs.CL, cs.AI, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09937
• Pdf URL: https://arxiv.org/pdf/2404.09937
• Copy Paste: [[2404.09937]] Compression Represents Intelligence Linearly(https://arxiv.org/abs/2404.09937)
• Keywords: large language model
• Abstract: There is a belief that learning to compress well will lead to intelligence. Recently, language modeling has been shown to be equivalent to compression, which offers a compelling rationale for the success of large language models (LLMs): the development of more advanced language models is essentially enhancing compression which facilitates intelligence. Despite such appealing discussions, little empirical evidence is present for the interplay between compression and intelligence. In this work, we examine their relationship in the context of LLMs, treating LLMs as data compressors. Given the abstract concept of "intelligence", we adopt the average downstream benchmark scores as a surrogate, specifically targeting intelligence related to knowledge and commonsense, coding, and mathematical reasoning. Across 12 benchmarks, our study brings together 30 public LLMs that originate from diverse organizations. Remarkably, we find that LLMs' intelligence -- reflected by average benchmark scores -- almost linearly correlates with their ability to compress external text corpora. These results provide concrete evidence supporting the belief that superior compression indicates greater intelligence. Furthermore, our findings suggest that compression efficiency, as an unsupervised metric derived from raw text corpora, serves as a reliable evaluation measure that is linearly associated with the model capabilities. We open-source our compression datasets as well as our data collection pipelines to facilitate future researchers to assess compression properly.

Title: Evolving Interpretable Visual Classifiers with Large Language Models

• Authors: Mia Chiquier, Utkarsh Mall, Carl Vondrick
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09941
• Pdf URL: https://arxiv.org/pdf/2404.09941
• Copy Paste: [[2404.09941]] Evolving Interpretable Visual Classifiers with Large Language Models(https://arxiv.org/abs/2404.09941)
• Keywords: interpretability, large language model
• Abstract: Multimodal pre-trained models, such as CLIP, are popular for zero-shot classification due to their open-vocabulary flexibility and high performance. However, vision-language models, which compute similarity scores between images and class labels, are largely black-box, with limited interpretability, risk for bias, and inability to discover new visual concepts not written down. Moreover, in practical settings, the vocabulary for class names and attributes of specialized concepts will not be known, preventing these methods from performing well on images uncommon in large-scale vision-language datasets. To address these limitations, we present a novel method that discovers interpretable yet discriminative sets of attributes for visual recognition. We introduce an evolutionary search algorithm that uses a large language model and its in-context learning abilities to iteratively mutate a concept bottleneck of attributes for classification. Our method produces state-of-the-art, interpretable fine-grained classifiers. We outperform the latest baselines by 18.4% on five fine-grained iNaturalist datasets and by 22.2% on two KikiBouba datasets, despite the baselines having access to privileged information about class names.

Title: Unifying Global and Local Scene Entities Modelling for Precise Action Spotting

• Authors: Kim Hoang Tran, Phuc Vuong Do, Ngoc Quoc Ly, Ngan Le
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09951
• Pdf URL: https://arxiv.org/pdf/2404.09951
• Copy Paste: [[2404.09951]] Unifying Global and Local Scene Entities Modelling for Precise Action Spotting(https://arxiv.org/abs/2404.09951)
• Keywords: interpretability
• Abstract: Sports videos pose complex challenges, including cluttered backgrounds, camera angle changes, small action-representing objects, and imbalanced action class distribution. Existing methods for detecting actions in sports videos heavily rely on global features, utilizing a backbone network as a black box that encompasses the entire spatial frame. However, these approaches tend to overlook the nuances of the scene and struggle with detecting actions that occupy a small portion of the frame. In particular, they face difficulties when dealing with action classes involving small objects, such as balls or yellow/red cards in soccer, which only occupy a fraction of the screen space. To address these challenges, we introduce a novel approach that analyzes and models scene entities using an adaptive attention mechanism. Particularly, our model disentangles the scene content into the global environment feature and local relevant scene entities feature. To efficiently extract environmental features while considering temporal information with less computational cost, we propose the use of a 2D backbone network with a time-shift mechanism. To accurately capture relevant scene entities, we employ a Vision-Language model in conjunction with the adaptive attention mechanism. Our model has demonstrated outstanding performance, securing the 1st place in the SoccerNet-v2 Action Spotting, FineDiving, and FineGym challenge with a substantial performance improvement of 1.6, 2.0, and 1.3 points in avg-mAP compared to the runner-up methods. Furthermore, our approach offers interpretability capabilities in contrast to other deep learning models, which are often designed as black boxes. Our code and models are released at: https://github.com/Fsoft-AIC/unifying-global-local-feature.

Title: How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model

• Authors: Hanxue Gu, Haoyu Dong, Jichen Yang, Maciej A. Mazurowski
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09957
• Pdf URL: https://arxiv.org/pdf/2404.09957
• Copy Paste: [[2404.09957]] How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model(https://arxiv.org/abs/2404.09957)
• Keywords: segmentation
• Abstract: Automated segmentation is a fundamental medical image analysis task, which enjoys significant advances due to the advent of deep learning. While foundation models have been useful in natural language processing and some vision tasks for some time, the foundation model developed with image segmentation in mind - Segment Anything Model (SAM) - has been developed only recently and has shown similar promise. However, there are still no systematic analyses or ``best-practice'' guidelines for optimal fine-tuning of SAM for medical image segmentation. This work summarizes existing fine-tuning strategies with various backbone architectures, model components, and fine-tuning algorithms across 18 combinations, and evaluates them on 17 datasets covering all common radiology modalities. Our study reveals that (1) fine-tuning SAM leads to slightly better performance than previous segmentation methods, (2) fine-tuning strategies that use parameter-efficient learning in both the encoder and decoder are superior to other strategies, (3) network architecture has a small impact on final performance, (4) further training SAM with self-supervised learning can improve final model performance. We also demonstrate the ineffectiveness of some methods popular in the literature and further expand our experiments into few-shot and prompt-based settings. Lastly, we released our code and MRI-specific fine-tuned weights, which consistently obtained superior performance over the original SAM, at https://github.com/mazurowski-lab/finetune-SAM.

Title: Ti-Patch: Tiled Physical Adversarial Patch for no-reference video quality metrics

• Authors: Victoria Leonenkova, Ekaterina Shumitskaya, Anastasia Antsiferova, Dmitriy Vatolin
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.09961
• Pdf URL: https://arxiv.org/pdf/2404.09961
• Copy Paste: [[2404.09961]] Ti-Patch: Tiled Physical Adversarial Patch for no-reference video quality metrics(https://arxiv.org/abs/2404.09961)
• Keywords: attack
• Abstract: Objective no-reference image- and video-quality metrics are crucial in many computer vision tasks. However, state-of-the-art no-reference metrics have become learning-based and are vulnerable to adversarial attacks. The vulnerability of quality metrics imposes restrictions on using such metrics in quality control systems and comparing objective algorithms. Also, using vulnerable metrics as a loss for deep learning model training can mislead training to worsen visual quality. Because of that, quality metrics testing for vulnerability is a task of current interest. This paper proposes a new method for testing quality metrics vulnerability in the physical space. To our knowledge, quality metrics were not previously tested for vulnerability to this attack; they were only tested in the pixel space. We applied a physical adversarial Ti-Patch (Tiled Patch) attack to quality metrics and did experiments both in pixel and physical space. We also performed experiments on the implementation of physical adversarial wallpaper. The proposed method can be used as additional quality metrics in vulnerability evaluation, complementing traditional subjective comparison and vulnerability tests in the pixel space. We made our code and adversarial videos available on GitHub: https://github.com/leonenkova/Ti-Patch.

Title: Design and Analysis of Efficient Attention in Transformers for Social Group Activity Recognition

• Authors: Masato Tamura
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09964
• Pdf URL: https://arxiv.org/pdf/2404.09964
• Copy Paste: [[2404.09964]] Design and Analysis of Efficient Attention in Transformers for Social Group Activity Recognition(https://arxiv.org/abs/2404.09964)
• Keywords: transformer
• Abstract: Social group activity recognition is a challenging task extended from group activity recognition, where social groups must be recognized with their activities and group members. Existing methods tackle this task by leveraging region features of individuals following existing group activity recognition methods. However, the effectiveness of region features is susceptible to person localization and variable semantics of individual actions. To overcome these issues, we propose leveraging attention modules in transformers to generate social group features. In this method, multiple embeddings are used to aggregate features for a social group, each of which is assigned to a group member without duplication. Due to this non-duplicated assignment, the number of embeddings must be significant to avoid missing group members and thus renders attention in transformers ineffective. To find optimal attention designs with a large number of embeddings, we explore several design choices of queries for feature aggregation and self-attention modules in transformer decoders. Extensive experimental results show that the proposed method achieves state-of-the-art performance and verify that the proposed attention designs are highly effective on social group activity recognition.

Title: Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model

• Authors: Han Lin, Jaemin Cho, Abhay Zala, Mohit Bansal
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09967
• Pdf URL: https://arxiv.org/pdf/2404.09967
• Copy Paste: [[2404.09967]] Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model(https://arxiv.org/abs/2404.09967)
• Keywords: robust, diffusion
• Abstract: ControlNets are widely used for adding spatial control in image generation with different conditions, such as depth maps, canny edges, and human poses. However, there are several challenges when leveraging the pretrained image ControlNets for controlled video generation. First, pretrained ControlNet cannot be directly plugged into new backbone models due to the mismatch of feature spaces, and the cost of training ControlNets for new backbones is a big burden. Second, ControlNet features for different frames might not effectively handle the temporal consistency. To address these challenges, we introduce Ctrl-Adapter, an efficient and versatile framework that adds diverse controls to any image/video diffusion models, by adapting pretrained ControlNets (and improving temporal alignment for videos). Ctrl-Adapter provides diverse capabilities including image control, video control, video control with sparse frames, multi-condition control, compatibility with different backbones, adaptation to unseen control conditions, and video editing. In Ctrl-Adapter, we train adapter layers that fuse pretrained ControlNet features to different image/video diffusion models, while keeping the parameters of the ControlNets and the diffusion models frozen. Ctrl-Adapter consists of temporal and spatial modules so that it can effectively handle the temporal consistency of videos. We also propose latent skipping and inverse timestep sampling for robust adaptation and sparse control. Moreover, Ctrl-Adapter enables control from multiple conditions by simply taking the (weighted) average of ControlNet outputs. With diverse image/video diffusion backbones (SDXL, Hotshot-XL, I2VGen-XL, and SVD), Ctrl-Adapter matches ControlNet for image control and outperforms all baselines for video control (achieving the SOTA accuracy on the DAVIS 2017 dataset) with significantly lower computational costs (less than 10 GPU hours).

Title: Constructing Benchmarks and Interventions for Combating Hallucinations in LLMs

• Authors: Adi Simhi, Jonathan Herzig, Idan Szpektor, Yonatan Belinkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09971
• Pdf URL: https://arxiv.org/pdf/2404.09971
• Copy Paste: [[2404.09971]] Constructing Benchmarks and Interventions for Combating Hallucinations in LLMs(https://arxiv.org/abs/2404.09971)
• Keywords: large language model
• Abstract: Large language models (LLMs) are susceptible to hallucination, which sparked a widespread effort to detect and prevent them. Recent work attempts to mitigate hallucinations by intervening in the model's computation during generation, using different setups and heuristics. Those works lack separation between different hallucination causes. In this work, we first introduce an approach for constructing datasets based on the model knowledge for detection and intervention methods in closed-book and open-book question-answering settings. We then characterize the effect of different choices for intervention, such as the intervened components (MLPs, attention block, residual stream, and specific heads), and how often and how strongly to intervene. We find that intervention success varies depending on the component, with some components being detrimental to language modeling capabilities. Finally, we find that interventions can benefit from pre-hallucination steering direction instead of post-hallucination. The code is available at https://github.com/technion-cs-nlp/hallucination-mitigation

Title: Diffscaler: Enhancing the Generative Prowess of Diffusion Transformers

• Authors: Nithin Gopalakrishnan Nair, Jeya Maria Jose Valanarasu, Vishal M. Patel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09976
• Pdf URL: https://arxiv.org/pdf/2404.09976
• Copy Paste: [[2404.09976]] Diffscaler: Enhancing the Generative Prowess of Diffusion Transformers(https://arxiv.org/abs/2404.09976)
• Keywords: diffusion, transformer, generative
• Abstract: Recently, diffusion transformers have gained wide attention with its excellent performance in text-to-image and text-to-vidoe models, emphasizing the need for transformers as backbone for diffusion models. Transformer-based models have shown better generalization capability compared to CNN-based models for general vision tasks. However, much less has been explored in the existing literature regarding the capabilities of transformer-based diffusion backbones and expanding their generative prowess to other datasets. This paper focuses on enabling a single pre-trained diffusion transformer model to scale across multiple datasets swiftly, allowing for the completion of diverse generative tasks using just one model. To this end, we propose DiffScaler, an efficient scaling strategy for diffusion models where we train a minimal amount of parameters to adapt to different tasks. In particular, we learn task-specific transformations at each layer by incorporating the ability to utilize the learned subspaces of the pre-trained model, as well as the ability to learn additional task-specific subspaces, which may be absent in the pre-training dataset. As these parameters are independent, a single diffusion model with these task-specific parameters can be used to perform multiple tasks simultaneously. Moreover, we find that transformer-based diffusion models significantly outperform CNN-based diffusion models methods while performing fine-tuning over smaller datasets. We perform experiments on four unconditional image generation datasets. We show that using our proposed method, a single pre-trained model can scale up to perform these conditional and unconditional tasks, respectively, with minimal parameter tuning while performing as close as fine-tuning an entire diffusion model for that particular task.

Title: MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models

• Authors: Nithin Gopalakrishnan Nair, Jeya Maria Jose Valanarasu, Vishal M Patel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09977
• Pdf URL: https://arxiv.org/pdf/2404.09977
• Copy Paste: [[2404.09977]] MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models(https://arxiv.org/abs/2404.09977)
• Keywords: diffusion, generative
• Abstract: Large diffusion-based Text-to-Image (T2I) models have shown impressive generative powers for text-to-image generation as well as spatially conditioned image generation. For most applications, we can train the model end-toend with paired data to obtain photorealistic generation quality. However, to add an additional task, one often needs to retrain the model from scratch using paired data across all modalities to retain good generation performance. In this paper, we tackle this issue and propose a novel strategy to scale a generative model across new tasks with minimal compute. During our experiments, we discovered that the variance maps of intermediate feature maps of diffusion models capture the intensity of conditioning. Utilizing this prior information, we propose MaxFusion, an efficient strategy to scale up text-to-image generation models to accommodate new modality conditions. Specifically, we combine aligned features of multiple models, hence bringing a compositional effect. Our fusion strategy can be integrated into off-the-shelf models to enhance their generative prowess.

Title: Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems

• Authors: Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke
• Subjects: cs.CL, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.09980
• Pdf URL: https://arxiv.org/pdf/2404.09980
• Copy Paste: [[2404.09980]] Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems(https://arxiv.org/abs/2404.09980)
• Keywords: large language model
• Abstract: Crowdsourced labels play a crucial role in evaluating task-oriented dialogue systems (TDSs). Obtaining high-quality and consistent ground-truth labels from annotators presents challenges. When evaluating a TDS, annotators must fully comprehend the dialogue before providing judgments. Previous studies suggest using only a portion of the dialogue context in the annotation process. However, the impact of this limitation on label quality remains unexplored. This study investigates the influence of dialogue context on annotation quality, considering the truncated context for relevance and usefulness labeling. We further propose to use large language models (LLMs) to summarize the dialogue context to provide a rich and short description of the dialogue context and study the impact of doing so on the annotator's performance. Reducing context leads to more positive ratings. Conversely, providing the entire dialogue context yields higher-quality relevance ratings but introduces ambiguity in usefulness ratings. Using the first user utterance as context leads to consistent ratings, akin to those obtained using the entire dialogue, with significantly reduced annotation effort. Our findings show how task design, particularly the availability of dialogue context, affects the quality and consistency of crowdsourced evaluation labels.

Title: Memory Sharing for Large Language Model based Agents

• Authors: Hang Gao, Yongfeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09982
• Pdf URL: https://arxiv.org/pdf/2404.09982
• Copy Paste: [[2404.09982]] Memory Sharing for Large Language Model based Agents(https://arxiv.org/abs/2404.09982)
• Keywords: large language model
• Abstract: In the realm of artificial intelligence, the adaptation of Large Language Model (LLM)-based agents to execute tasks via natural language prompts represents a significant advancement, notably eliminating the need for explicit retraining or fine tuning for fixed-answer tasks such as common sense questions and yes/no queries. However, the application of In-context Learning to open-ended challenges, such as poetry creation, reveals substantial limitations due to the comprehensiveness of the provided examples and agent's ability to understand the content expressed in the problem, leading to outputs that often diverge significantly from expected results. Addressing this gap, our study introduces the Memory-Sharing (MS) framework for LLM multi-agents, which utilizes a real-time memory storage and retrieval system to enhance the In-context Learning process. Each "memory" within this system captures both the posed query and the corresponding real-time response from an LLM-based agent, aggregating these memories from a broad spectrum of similar agents to enrich the memory pool shared by all agents. This framework not only aids agents in identifying the most relevant examples for specific tasks but also evaluates the potential utility of their memories for future applications by other agents. Empirical validation across three distinct domains involving specialized functions of agents demonstrates that the MS framework significantly improve the agent's performance regrading the open-ended questions. Furthermore, we also discuss what type of memory pool and what retrieval strategy in MS can better help agents, offering a future develop direction of MS. The code and data are available at: https://github.com/GHupppp/MemorySharingLLM

Title: OneChart: Purify the Chart Structural Extraction via One Auxiliary Token

• Authors: Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Han, Xiangyu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09987
• Pdf URL: https://arxiv.org/pdf/2404.09987
• Copy Paste: [[2404.09987]] OneChart: Purify the Chart Structural Extraction via One Auxiliary Token(https://arxiv.org/abs/2404.09987)
• Keywords: extraction
• Abstract: Chart parsing poses a significant challenge due to the diversity of styles, values, texts, and so forth. Even advanced large vision-language models (LVLMs) with billions of parameters struggle to handle such tasks satisfactorily. To address this, we propose OneChart: a reliable agent specifically devised for the structural extraction of chart information. Similar to popular LVLMs, OneChart incorporates an autoregressive main body. Uniquely, to enhance the reliability of the numerical parts of the output, we introduce an auxiliary token placed at the beginning of the total tokens along with an additional decoder. The numerically optimized (auxiliary) token allows subsequent tokens for chart parsing to capture enhanced numerical features through causal attention. Furthermore, with the aid of the auxiliary token, we have devised a self-evaluation mechanism that enables the model to gauge the reliability of its chart parsing results by providing confidence scores for the generated content. Compared to current state-of-the-art (SOTA) chart parsing models, e.g., DePlot, ChartVLM, ChartAst, OneChart significantly outperforms in Average Precision (AP) for chart structural extraction across multiple public benchmarks, despite enjoying only 0.2 billion parameters. Moreover, as a chart parsing agent, it also brings 10%+ accuracy gains for the popular LVLM (LLaVA-1.6) in the downstream ChartQA benchmark.

Title: in2IN: Leveraging individual Information to Generate Human INteractions

• Authors: Pablo Ruiz Ponce, German Barquero, Cristina Palmero, Sergio Escalera, Jose Garcia-Rodriguez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.09988
• Pdf URL: https://arxiv.org/pdf/2404.09988
• Copy Paste: [[2404.09988]] in2IN: Leveraging individual Information to Generate Human INteractions(https://arxiv.org/abs/2404.09988)
• Keywords: diffusion, large language model
• Abstract: Generating human-human motion interactions conditioned on textual descriptions is a very useful application in many areas such as robotics, gaming, animation, and the metaverse. Alongside this utility also comes a great difficulty in modeling the highly dimensional inter-personal dynamics. In addition, properly capturing the intra-personal diversity of interactions has a lot of challenges. Current methods generate interactions with limited diversity of intra-person dynamics due to the limitations of the available datasets and conditioning strategies. For this, we introduce in2IN, a novel diffusion model for human-human motion generation which is conditioned not only on the textual description of the overall interaction but also on the individual descriptions of the actions performed by each person involved in the interaction. To train this model, we use a large language model to extend the InterHuman dataset with individual descriptions. As a result, in2IN achieves state-of-the-art performance in the InterHuman dataset. Furthermore, in order to increase the intra-personal diversity on the existing interaction datasets, we propose DualMDM, a model composition technique that combines the motions generated with in2IN and the motions generated by a single-person motion prior pre-trained on HumanML3D. As a result, DualMDM generates motions with higher individual diversity and improves control over the intra-person dynamics while maintaining inter-personal coherence.

Title: Taming Latent Diffusion Model for Neural Radiance Field Inpainting

• Authors: Chieh Hubert Lin, Changil Kim, Jia-Bin Huang, Qinbo Li, Chih-Yao Ma, Johannes Kopf, Ming-Hsuan Yang, Hung-Yu Tseng
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09995
• Pdf URL: https://arxiv.org/pdf/2404.09995
• Copy Paste: [[2404.09995]] Taming Latent Diffusion Model for Neural Radiance Field Inpainting(https://arxiv.org/abs/2404.09995)
• Keywords: diffusion
• Abstract: Neural Radiance Field (NeRF) is a representation for 3D reconstruction from multi-view images. Despite some recent work showing preliminary success in editing a reconstructed NeRF with diffusion prior, they remain struggling to synthesize reasonable geometry in completely uncovered regions. One major reason is the high diversity of synthetic contents from the diffusion model, which hinders the radiance field from converging to a crisp and deterministic geometry. Moreover, applying latent diffusion models on real data often yields a textural shift incoherent to the image condition due to auto-encoding errors. These two problems are further reinforced with the use of pixel-distance losses. To address these issues, we propose tempering the diffusion model's stochasticity with per-scene customization and mitigating the textural shift with masked adversarial training. During the analyses, we also found the commonly used pixel and perceptual losses are harmful in the NeRF inpainting task. Through rigorous experiments, our framework yields state-of-the-art NeRF inpainting results on various real-world scenes. Project page: https://hubert0527.github.io/MALD-NeRF