2024-08-20

Title: SHARP-Net: A Refined Pyramid Network for Deficiency Segmentation in Culverts and Sewer Pipes

• Authors: Rasha Alshawi, Md Meftahul Ferdaus, Md Tamjidul Hoque, Kendall Niles, Ken Pathak, Steve Sloan, Mahdi Abdelguerfi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08879
• Pdf URL: https://arxiv.org/pdf/2408.08879
• Copy Paste: [[2408.08879]] SHARP-Net: A Refined Pyramid Network for Deficiency Segmentation in Culverts and Sewer Pipes(https://arxiv.org/abs/2408.08879)
• Keywords: segmentation
• Abstract: This paper introduces Semantic Haar-Adaptive Refined Pyramid Network (SHARP-Net), a novel architecture for semantic segmentation. SHARP-Net integrates a bottom-up pathway featuring Inception-like blocks with varying filter sizes (3x3$ and 5x5), parallel max-pooling, and additional spatial detection layers. This design captures multi-scale features and fine structural details. Throughout the network, depth-wise separable convolutions are used to reduce complexity. The top-down pathway of SHARP-Net focuses on generating high-resolution features through upsampling and information fusion using $1\times1$ and $3\times3$ depth-wise separable convolutions. We evaluated our model using our developed challenging Culvert-Sewer Defects dataset and the benchmark DeepGlobe Land Cover dataset. Our experimental evaluation demonstrated the base model's (excluding Haar-like features) effectiveness in handling irregular defect shapes, occlusions, and class imbalances. It outperformed state-of-the-art methods, including U-Net, CBAM U-Net, ASCU-Net, FPN, and SegFormer, achieving average improvements of 14.4% and 12.1% on the Culvert-Sewer Defects and DeepGlobe Land Cover datasets, respectively, with IoU scores of 77.2% and 70.6%. Additionally, the training time was reduced. Furthermore, the integration of carefully selected and fine-tuned Haar-like features enhanced the performance of deep learning models by at least 20%. The proposed SHARP-Net, incorporating Haar-like features, achieved an impressive IoU of 94.75%, representing a 22.74% improvement over the base model. These features were also applied to other deep learning models, showing a 35.0% improvement, proving their versatility and effectiveness. SHARP-Net thus provides a powerful and efficient solution for accurate semantic segmentation in challenging real-world scenarios.

Title: Kov: Transferable and Naturalistic Black-Box LLM Attacks using Markov Decision Processes and Tree Search

• Authors: Robert J. Moss
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08899
• Pdf URL: https://arxiv.org/pdf/2408.08899
• Copy Paste: [[2408.08899]] Kov: Transferable and Naturalistic Black-Box LLM Attacks using Markov Decision Processes and Tree Search(https://arxiv.org/abs/2408.08899)
• Keywords: attack, robust, interpretability, large language model
• Abstract: Eliciting harmful behavior from large language models (LLMs) is an important task to ensure the proper alignment and safety of the models. Often when training LLMs, ethical guidelines are followed yet alignment failures may still be uncovered through red teaming adversarial attacks. This work frames the red-teaming problem as a Markov decision process (MDP) and uses Monte Carlo tree search to find harmful behaviors of black-box, closed-source LLMs. We optimize token-level prompt suffixes towards targeted harmful behaviors on white-box LLMs and include a naturalistic loss term, log-perplexity, to generate more natural language attacks for better interpretability. The proposed algorithm, Kov, trains on white-box LLMs to optimize the adversarial attacks and periodically evaluates responses from the black-box LLM to guide the search towards more harmful black-box behaviors. In our preliminary study, results indicate that we can jailbreak black-box models, such as GPT-3.5, in only 10 queries, yet fail on GPT-4$-$which may indicate that newer models are more robust to token-level attacks. All work to reproduce these results is open sourced (this https URL).

Title: Audit-LLM: Multi-Agent Collaboration for Log-based Insider Threat Detection

• Authors: Chengyu Song, Linru Ma, Jianming Zheng, Jinzhi Liao, Hongyu Kuang, Lin Yang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08902
• Pdf URL: https://arxiv.org/pdf/2408.08902
• Copy Paste: [[2408.08902]] Audit-LLM: Multi-Agent Collaboration for Log-based Insider Threat Detection(https://arxiv.org/abs/2408.08902)
• Keywords: large language model
• Abstract: Log-based insider threat detection (ITD) detects malicious user activities by auditing log entries. Recently, large language models (LLMs) with strong common sense knowledge have emerged in the domain of ITD. Nevertheless, diverse activity types and overlong log files pose a significant challenge for LLMs in directly discerning malicious ones within myriads of normal activities. Furthermore, the faithfulness hallucination issue from LLMs aggravates its application difficulty in ITD, as the generated conclusion may not align with user commands and activity context. In response to these challenges, we introduce Audit-LLM, a multi-agent log-based insider threat detection framework comprising three collaborative agents: (i) the Decomposer agent, breaking down the complex ITD task into manageable sub-tasks using Chain-of-Thought (COT) reasoning;(ii) the Tool Builder agent, creating reusable tools for sub-tasks to overcome context length limitations in LLMs; and (iii) the Executor agent, generating the final detection conclusion by invoking constructed tools. To enhance conclusion accuracy, we propose a pair-wise Evidence-based Multi-agent Debate (EMAD) mechanism, where two independent Executors iteratively refine their conclusions through reasoning exchange to reach a consensus. Comprehensive experiments conducted on three publicly available ITD datasets-CERT r4.2, CERT r5.2, and PicoDomain-demonstrate the superiority of our method over existing baselines and show that the proposed EMAD significantly improves the faithfulness of explanations generated by LLMs.

Title: Privacy in Federated Learning

• Authors: Jaydip Sen, Hetvi Waghela, Sneha Rakshit
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.08904
• Pdf URL: https://arxiv.org/pdf/2408.08904
• Copy Paste: [[2408.08904]] Privacy in Federated Learning(https://arxiv.org/abs/2408.08904)
• Keywords: secure, privacy, attack, robust, membership infer, federate
• Abstract: Federated Learning (FL) represents a significant advancement in distributed machine learning, enabling multiple participants to collaboratively train models without sharing raw data. This decentralized approach enhances privacy by keeping data on local devices. However, FL introduces new privacy challenges, as model updates shared during training can inadvertently leak sensitive information. This chapter delves into the core privacy concerns within FL, including the risks of data reconstruction, model inversion attacks, and membership inference. It explores various privacy-preserving techniques, such as Differential Privacy (DP) and Secure Multi-Party Computation (SMPC), which are designed to mitigate these risks. The chapter also examines the trade-offs between model accuracy and privacy, emphasizing the importance of balancing these factors in practical implementations. Furthermore, it discusses the role of regulatory frameworks, such as GDPR, in shaping the privacy standards for FL. By providing a comprehensive overview of the current state of privacy in FL, this chapter aims to equip researchers and practitioners with the knowledge necessary to navigate the complexities of secure federated learning environments. The discussion highlights both the potential and limitations of existing privacy-enhancing techniques, offering insights into future research directions and the development of more robust solutions.

Title: An Adaptive Differential Privacy Method Based on Federated Learning

• Authors: Zhiqiang Wang, Xinyue Yu, Qianli Huang, Yongguang Gong
• Subjects: cs.CR, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.08909
• Pdf URL: https://arxiv.org/pdf/2408.08909
• Copy Paste: [[2408.08909]] An Adaptive Differential Privacy Method Based on Federated Learning(https://arxiv.org/abs/2408.08909)
• Keywords: privacy, protect, federate
• Abstract: Differential privacy is one of the methods to solve the problem of privacy protection in federated learning. Setting the same privacy budget for each round will result in reduced accuracy in training. The existing methods of the adjustment of privacy budget consider fewer influencing factors and tend to ignore the boundaries, resulting in unreasonable privacy budgets. Therefore, we proposed an adaptive differential privacy method based on federated learning. The method sets the adjustment coefficient and scoring function according to accuracy, loss, training rounds, and the number of datasets and clients. And the privacy budget is adjusted based on them. Then the local model update is processed according to the scaling factor and the noise. Fi-nally, the server aggregates the noised local model update and distributes the noised global model. The range of parameters and the privacy of the method are analyzed. Through the experimental evaluation, it can reduce the privacy budget by about 16%, while the accuracy remains roughly the same.

Title: A Survey on Blockchain-based Supply Chain Finance with Progress and Future directions

• Authors: Zhengdong Luo
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08915
• Pdf URL: https://arxiv.org/pdf/2408.08915
• Copy Paste: [[2408.08915]] A Survey on Blockchain-based Supply Chain Finance with Progress and Future directions(https://arxiv.org/abs/2408.08915)
• Keywords: privacy
• Abstract: Supply Chain Finance is very important for supply chain competition, which is an important tool to activate the capital flow in the supply chain. Supply Chain Finance-related research can support multiple applications and services, such as providing accounts receivable financing, enhancing risk management, and optimizing supply chain management. For more than a decade, the development of Blockchain has attracted widely attention in various fields, especially in finance. With the characteristics of data tamper-proof, forgery-proof, cryptography, consensus verification, and decentralization, Blockchain fits well with the realistic needs of Supply Chain Finance, which requires data integrity, authenticity, privacy, and information sharing. Therefore, it is time to summarize the applications of Blockchain technology in the field of Supply Chain Finance. What Blockchain technology brings to Supply Chain Finance is not only to alleviate the problems of information asymmetry, credit disassembly, and financing cost, but also to improve Supply Chain Finance operations through smart contracts to intelligent Supply Chain Finance and in combination with other technologies, such as artificial intelligence, cloud computing, and data mining, jointly. So there has been some work in Blockchain-based Supply Chain Finance research for different Supply Chain Finance oriented applications, but most of these work are at the management level to propose conceptual frameworks or simply use Blockchain without exploiting its deep applications. Moreover, there are few systematic reviews providing a comprehensive summary of current work in the area of Blockchain-based Supply Chain Finance. In this paper, we ...

Title: Supervised and Unsupervised Alignments for Spoofing Behavioral Biometrics

• Authors: Thomas Thebaud, Gaël Le Lan, Anthony Larcher
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08918
• Pdf URL: https://arxiv.org/pdf/2408.08918
• Copy Paste: [[2408.08918]] Supervised and Unsupervised Alignments for Spoofing Behavioral Biometrics(https://arxiv.org/abs/2408.08918)
• Keywords: security, attack, biometric
• Abstract: Biometric recognition systems are security systems based on intrinsic properties of their users, usually encoded in high dimension representations called embeddings, which potential theft would represent a greater threat than a temporary password or a replaceable key. To study the threat of embedding theft, we perform spoofing attacks on two behavioral biometric systems (an automatic speaker verification system and a handwritten digit analysis system) using a set of alignment techniques. Biometric recognition systems based on embeddings work in two phases: enrollment - where embeddings are collected and stored - then authentication - when new embeddings are compared to the stored ones -.The threat of stolen enrollment embeddings has been explored by the template reconstruction attack literature: reconstructing the original data to spoof an authentication system is doable with black-box access to their encoder. In this document, we explore the options available to perform template reconstruction attacks without any access to the encoder. To perform those attacks, we suppose general rules over the distribution of embeddings across encoders and use supervised and unsupervised algorithms to align an unlabeled set of embeddings with a set from a known encoder. The use of an alignment algorithm from the unsupervised translation literature gives promising results on spoofing two behavioral biometric systems.

Title: A Survey of Trojan Attacks and Defenses to Deep Neural Networks

• Authors: Lingxin Jin, Xianyu Wen, Wei Jiang, Jinyu Zhan
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08920
• Pdf URL: https://arxiv.org/pdf/2408.08920
• Copy Paste: [[2408.08920]] A Survey of Trojan Attacks and Defenses to Deep Neural Networks(https://arxiv.org/abs/2408.08920)
• Keywords: defense, attack
• Abstract: Deep Neural Networks (DNNs) have found extensive applications in safety-critical artificial intelligence systems, such as autonomous driving and facial recognition systems. However, recent research has revealed their susceptibility to Neural Network Trojans (NN Trojans) maliciously injected by adversaries. This vulnerability arises due to the intricate architecture and opacity of DNNs, resulting in numerous redundant neurons embedded within the models. Adversaries exploit these vulnerabilities to conceal malicious Trojans within DNNs, thereby causing erroneous outputs and posing substantial threats to the efficacy of DNN-based applications. This article presents a comprehensive survey of Trojan attacks against DNNs and the countermeasure methods employed to mitigate them. Initially, we trace the evolution of the concept from traditional Trojans to NN Trojans, highlighting the feasibility and practicality of generating NN Trojans. Subsequently, we provide an overview of notable works encompassing various attack and defense strategies, facilitating a comparative analysis of their approaches. Through these discussions, we offer constructive insights aimed at refining these techniques. In recognition of the gravity and immediacy of this subject matter, we also assess the feasibility of deploying such attacks in real-world scenarios as opposed to controlled ideal datasets. The potential real-world implications underscore the urgency of addressing this issue effectively.

Title: Prefix Guidance: A Steering Wheel for Large Language Models to Defend Against Jailbreak Attacks

• Authors: Jiawei Zhao, Kejiang Chen, Xiaojian Yuan, Weiming Zhang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08924
• Pdf URL: https://arxiv.org/pdf/2408.08924
• Copy Paste: [[2408.08924]] Prefix Guidance: A Steering Wheel for Large Language Models to Defend Against Jailbreak Attacks(https://arxiv.org/abs/2408.08924)
• Keywords: secure, security, protect, defense, attack, large language model
• Abstract: In recent years, the rapid development of large language models (LLMs) has achieved remarkable performance across various tasks. However, research indicates that LLMs are vulnerable to jailbreak attacks, where adversaries can induce the generation of harmful content through meticulously crafted prompts. This vulnerability poses significant challenges to the secure use and promotion of LLMs. Existing defense methods offer protection from different perspectives but often suffer from insufficient effectiveness or a significant impact on the model's capabilities. In this paper, we propose a plug-and-play and easy-to-deploy jailbreak defense framework, namely Prefix Guidance (PG), which guides the model to identify harmful prompts by directly setting the first few tokens of the model's output. This approach combines the model's inherent security capabilities with an external classifier to defend against jailbreak attacks. We demonstrate the effectiveness of PG across three models and five attack methods. Compared to baselines, our approach is generally more effective on average. Additionally, results on the Just-Eval benchmark further confirm PG's superiority to preserve the model's performance.

Title: Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models

• Authors: Andy K. Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W. Lin, Joey Ji, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Polycarpos Yiorkadjis, Kenny Osele, Gautham Raghupathi, Dan Boneh, Daniel E. Ho, Percy Liang
• Subjects: cs.CR, cs.AI, cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08926
• Pdf URL: https://arxiv.org/pdf/2408.08926
• Copy Paste: [[2408.08926]] Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models(https://arxiv.org/abs/2408.08926)
• Keywords: security
• Abstract: Language Model (LM) agents for cybersecurity that are capable of autonomously identifying vulnerabilities and executing exploits have the potential to cause real-world impact. Policymakers, model providers, and other researchers in the AI and cybersecurity communities are interested in quantifying the capabilities of such agents to help mitigate cyberrisk and investigate opportunities for penetration testing. Toward that end, we introduce Cybench, a framework for specifying cybersecurity tasks and evaluating agents on those tasks. We include 40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF competitions, chosen to be recent, meaningful, and spanning a wide range of difficulties. Each task includes its own description, starter files, and is initialized in an environment where an agent can execute bash commands and observe outputs. Since many tasks are beyond the capabilities of existing LM agents, we introduce subtasks, which break down a task into intermediary steps for more gradated evaluation; we add subtasks for 17 of the 40 tasks. To evaluate agent capabilities, we construct a cybersecurity agent and evaluate 7 models: GPT-4o, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct, Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct. Without guidance, we find that agents are able to solve only the easiest complete tasks that took human teams up to 11 minutes to solve, with Claude 3.5 Sonnet and GPT-4o having the highest success rates. Finally, subtasks provide more signal for measuring performance compared to unguided runs, with models achieving a 3.2\% higher success rate on complete tasks with subtask-guidance than without subtask-guidance. All code and data are publicly available at this https URL

Title: DePrompt: Desensitization and Evaluation of Personal Identifiable Information in Large Language Model Prompts

• Authors: Xiongtao Sun, Gan Liu, Zhipeng He, Hui Li, Xiaoguang Li
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08930
• Pdf URL: https://arxiv.org/pdf/2408.08930
• Copy Paste: [[2408.08930]] DePrompt: Desensitization and Evaluation of Personal Identifiable Information in Large Language Model Prompts(https://arxiv.org/abs/2408.08930)
• Keywords: privacy, protect, interpretability, generative, large language model
• Abstract: Prompt serves as a crucial link in interacting with large language models (LLMs), widely impacting the accuracy and interpretability of model outputs. However, acquiring accurate and high-quality responses necessitates precise prompts, which inevitably pose significant risks of personal identifiable information (PII) leakage. Therefore, this paper proposes DePrompt, a desensitization protection and effectiveness evaluation framework for prompt, enabling users to safely and transparently utilize LLMs. Specifically, by leveraging large model fine-tuning techniques as the underlying privacy protection method, we integrate contextual attributes to define privacy types, achieving high-precision PII entity identification. Additionally, through the analysis of key features in prompt desensitization scenarios, we devise adversarial generative desensitization methods that retain important semantic content while disrupting the link between identifiers and privacy attributes. Furthermore, we present utility evaluation metrics for prompt to better gauge and balance privacy and usability. Our framework is adaptable to prompts and can be extended to text usability-dependent scenarios. Through comparison with benchmarks and other model methods, experimental evaluations demonstrate that our desensitized prompt exhibit superior privacy protection utility and model inference results.

Title: A Factored MDP Approach To Moving Target Defense With Dynamic Threat Modeling and Cost Efficiency

• Authors: Megha Bose, Praveen Paruchuri, Akshat Kumar
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08934
• Pdf URL: https://arxiv.org/pdf/2408.08934
• Copy Paste: [[2408.08934]] A Factored MDP Approach To Moving Target Defense With Dynamic Threat Modeling and Cost Efficiency(https://arxiv.org/abs/2408.08934)
• Keywords: defense, attack, robust
• Abstract: Moving Target Defense (MTD) has emerged as a proactive and dynamic framework to counteract evolving cyber threats. Traditional MTD approaches often rely on assumptions about the attackers knowledge and behavior. However, real-world scenarios are inherently more complex, with adaptive attackers and limited prior knowledge of their payoffs and intentions. This paper introduces a novel approach to MTD using a Markov Decision Process (MDP) model that does not rely on predefined attacker payoffs. Our framework integrates the attackers real-time responses into the defenders MDP using a dynamic Bayesian Network. By employing a factored MDP model, we provide a comprehensive and realistic system representation. We also incorporate incremental updates to an attack response predictor as new data emerges. This ensures an adaptive and robust defense mechanism. Additionally, we consider the costs of switching configurations in MTD, integrating them into the reward structure to balance execution and defense costs. We first highlight the challenges of the problem through a theoretical negative result on regret. However, empirical evaluations demonstrate the frameworks effectiveness in scenarios marked by high uncertainty and dynamically changing attack landscapes.

Title: BnSentMix: A Diverse Bengali-English Code-Mixed Dataset for Sentiment Analysis

• Authors: Sadia Alam, Md Farhan Ishmam, Navid Hasin Alvee, Md Shahnewaz Siddique, Md Azam Hossain, Abu Raihan Mostofa Kamal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08964
• Pdf URL: https://arxiv.org/pdf/2408.08964
• Copy Paste: [[2408.08964]] BnSentMix: A Diverse Bengali-English Code-Mixed Dataset for Sentiment Analysis(https://arxiv.org/abs/2408.08964)
• Keywords: transformer
• Abstract: The widespread availability of code-mixed data can provide valuable insights into low-resource languages like Bengali, which have limited datasets. Sentiment analysis has been a fundamental text classification task across several languages for code-mixed data. However, there has yet to be a large-scale and diverse sentiment analysis dataset on code-mixed Bengali. We address this limitation by introducing BnSentMix, a sentiment analysis dataset on code-mixed Bengali consisting of 20,000 samples with $4$ sentiment labels from Facebook, YouTube, and e-commerce sites. We ensure diversity in data sources to replicate realistic code-mixed scenarios. Additionally, we propose $14$ baseline methods including novel transformer encoders further pre-trained on code-mixed Bengali-English, achieving an overall accuracy of $69.8\%$ and an F1 score of $69.1\%$ on sentiment classification tasks. Detailed analyses reveal variations in performance across different sentiment labels and text types, highlighting areas for future improvement.

Title: Phishing Codebook: A Structured Framework for the Characterization of Phishing Emails

• Authors: Tarini Saka, Rachiyta Jain, Kami Vaniea, Nadin Kökciyan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.08967
• Pdf URL: https://arxiv.org/pdf/2408.08967
• Copy Paste: [[2408.08967]] Phishing Codebook: A Structured Framework for the Characterization of Phishing Emails(https://arxiv.org/abs/2408.08967)
• Keywords: attack
• Abstract: Phishing is one of the most prevalent and expensive types of cybercrime faced by organizations and individuals worldwide. Most prior research has focused on various technical features and traditional representations of text to characterize phishing emails. There is a significant knowledge gap about the qualitative traits embedded in them, which could be useful in a range of phishing mitigation tasks. In this paper, we dissect the structure of phishing emails to gain a better understanding of the factors that influence human decision-making when assessing suspicious emails and identify a novel set of descriptive features. For this, we employ an iterative qualitative coding approach to identify features that are descriptive of the emails. We developed the ``Phishing Codebook'', a structured framework to systematically extract key information from phishing emails, and we apply this codebook to a publicly available dataset of 503 phishing emails collected between 2015 and 2021. We present key observations and challenges related to phishing attacks delivered indirectly through legitimate services, the challenge of recurring and long-lasting scams, and the variations within campaigns used by attackers to bypass rule-based filters. Furthermore, we provide two use cases to show how the Phishing Codebook is useful in identifying similar phishing emails and in creating well-tailored responses to end-users. We share the Phishing Codebook and the annotated benchmark dataset to help researchers have a better understanding of phishing emails.

Title: Enhancing Object Detection with Hybrid dataset in Manufacturing Environments: Comparing Federated Learning to Conventional Techniques

• Authors: Vinit Hegiste, Snehal Walunj, Jibinraj Antony, Tatjana Legler, Martin Ruskowski
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08974
• Pdf URL: https://arxiv.org/pdf/2408.08974
• Copy Paste: [[2408.08974]] Enhancing Object Detection with Hybrid dataset in Manufacturing Environments: Comparing Federated Learning to Conventional Techniques(https://arxiv.org/abs/2408.08974)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) has garnered significant attention in manufacturing for its robust model development and privacy-preserving capabilities. This paper contributes to research focused on the robustness of FL models in object detection, hereby presenting a comparative study with conventional techniques using a hybrid dataset for small object detection. Our findings demonstrate the superior performance of FL over centralized training models and different deep learning techniques when tested on test data recorded in a different environment with a variety of object viewpoints, lighting conditions, cluttered backgrounds, etc. These results highlight the potential of FL in achieving robust global models that perform efficiently even in unseen environments. The study provides valuable insights for deploying resilient object detection models in manufacturing environments.

Title: See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses

• Authors: Yulong Chen, Yang Liu, Jianhao Yan, Xuefeng Bai, Ming Zhong, Yinghao Yang, Ziyi Yang, Chenguang Zhu, Yue Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08978
• Pdf URL: https://arxiv.org/pdf/2408.08978
• Copy Paste: [[2408.08978]] See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses(https://arxiv.org/abs/2408.08978)
• Keywords: large language model
• Abstract: The impressive performance of Large Language Models (LLMs) has consistently surpassed numerous human-designed benchmarks, presenting new challenges in assessing the shortcomings of LLMs. Designing tasks and finding LLMs' limitations are becoming increasingly important. In this paper, we investigate the question of whether an LLM can discover its own limitations from the errors it makes. To this end, we propose a Self-Challenge evaluation framework with human-in-the-loop. Starting from seed instances that GPT-4 fails to answer, we prompt GPT-4 to summarize error patterns that can be used to generate new instances and incorporate human feedback on them to refine these patterns for generating more challenging data, iteratively. We end up with 8 diverse patterns, such as text manipulation and questions with assumptions. We then build a benchmark, SC-G4, consisting of 1,835 instances generated by GPT-4 using these patterns, with human-annotated gold responses. The SC-G4 serves as a challenging benchmark that allows for a detailed assessment of LLMs' abilities. Our results show that only 44.96\% of instances in SC-G4 can be answered correctly by GPT-4. Interestingly, our pilot study indicates that these error patterns also challenge other LLMs, such as Claude-3 and Llama-3, and cannot be fully resolved through fine-tuning. Our work takes the first step to demonstrate that LLMs can autonomously identify their inherent flaws and provide insights for future dynamic and automatic evaluation.

Title: Electroencephalogram Emotion Recognition via AUC Maximization

• Authors: Minheng Xiao, Shi Bo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08979
• Pdf URL: https://arxiv.org/pdf/2408.08979
• Copy Paste: [[2408.08979]] Electroencephalogram Emotion Recognition via AUC Maximization(https://arxiv.org/abs/2408.08979)
• Keywords: robust
• Abstract: Imbalanced datasets pose significant challenges in areas including neuroscience, cognitive science, and medical diagnostics, where accurately detecting minority classes is essential for robust model performance. This study addresses the issue of class imbalance, using the `Liking' label in the DEAP dataset as an example. Such imbalances are often overlooked by prior research, which typically focuses on the more balanced arousal and valence labels and predominantly uses accuracy metrics to measure model performance. To tackle this issue, we adopt numerical optimization techniques aimed at maximizing the area under the curve (AUC), thus enhancing the detection of underrepresented classes. Our approach, which begins with a linear classifier, is compared against traditional linear classifiers, including logistic regression and support vector machines (SVM). Our method significantly outperforms these models, increasing recall from 41.6\% to 79.7\% and improving the F1-score from 0.506 to 0.632. These results highlight the efficacy of AUC maximization via numerical optimization in managing imbalanced datasets, providing an effective solution for enhancing predictive accuracy in detecting minority but crucial classes in out-of-sample datasets.

Title: Deep Generative Classification of Blood Cell Morphology

• Authors: Simon Deltadahl, Julian Gilbey, Christine Van Laer, Nancy Boeckx, Mathie Leers, Tanya Freeman, Laura Aiken, Timothy Farren, Matthew Smith, Mohamad Zeina, BloodCounts! consortium, Concetta Piazzese, Joseph Taylor, Nicholas Gleadall, Carola-Bibiane Schönlieb, Suthesh Sivapalaratnam, Michael Roberts, Parashkev Nachev
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08982
• Pdf URL: https://arxiv.org/pdf/2408.08982
• Copy Paste: [[2408.08982]] Deep Generative Classification of Blood Cell Morphology(https://arxiv.org/abs/2408.08982)
• Keywords: robust, interpretability, explainability, diffusion, generative
• Abstract: Accurate classification of haematological cells is critical for diagnosing blood disorders, but presents significant challenges for machine automation owing to the complexity of cell morphology, heterogeneities of biological, pathological, and imaging characteristics, and the imbalance of cell type frequencies. We introduce CytoDiffusion, a diffusion-based classifier that effectively models blood cell morphology, combining accurate classification with robust anomaly detection, resistance to distributional shifts, interpretability, data efficiency, and superhuman uncertainty quantification. Our approach outperforms state-of-the-art discriminative models in anomaly detection (AUC 0.976 vs. 0.919), resistance to domain shifts (85.85% vs. 74.38% balanced accuracy), and performance in low-data regimes (95.88% vs. 94.95% balanced accuracy). Notably, our model generates synthetic blood cell images that are nearly indistinguishable from real images, as demonstrated by a Turing test in which expert haematologists achieved only 52.3% accuracy (95% CI: [50.5%, 54.2%]). Furthermore, we enhance model explainability through the generation of directly interpretable counterfactual heatmaps. Our comprehensive evaluation framework, encompassing these multiple performance dimensions, establishes a new benchmark for medical image analysis in haematology, ultimately enabling improved diagnostic accuracy in clinical settings. Our code is available at this https URL.

Title: Fire Dynamic Vision: Image Segmentation and Tracking for Multi-Scale Fire and Plume Behavior

• Authors: Daryn Sagel, Bryan Quaife
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08984
• Pdf URL: https://arxiv.org/pdf/2408.08984
• Copy Paste: [[2408.08984]] Fire Dynamic Vision: Image Segmentation and Tracking for Multi-Scale Fire and Plume Behavior(https://arxiv.org/abs/2408.08984)
• Keywords: segmentation
• Abstract: The increasing frequency and severity of wildfires highlight the need for accurate fire and plume spread models. We introduce an approach that effectively isolates and tracks fire and plume behavior across various spatial and temporal scales and image types, identifying physical phenomena in the system and providing insights useful for developing and validating models. Our method combines image segmentation and graph theory to delineate fire fronts and plume boundaries. We demonstrate that the method effectively distinguishes fires and plumes from visually similar objects. Results demonstrate the successful isolation and tracking of fire and plume dynamics across various image sources, ranging from synoptic-scale ($10^4$-$10^5$ m) satellite images to sub-microscale ($10^0$-$10^1$ m) images captured close to the fire environment. Furthermore, the methodology leverages image inpainting and spatio-temporal dataset generation for use in statistical and machine learning models.

Title: Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds

• Authors: Zhiyong Wang, Dongruo Zhou, John C.S. Lui, Wen Sun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08994
• Pdf URL: https://arxiv.org/pdf/2408.08994
• Copy Paste: [[2408.08994]] Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds(https://arxiv.org/abs/2408.08994)
• Keywords: fair
• Abstract: Learning a transition model via Maximum Likelihood Estimation (MLE) followed by planning inside the learned model is perhaps the most standard and simplest Model-based Reinforcement Learning (RL) framework. In this work, we show that such a simple Model-based RL scheme, when equipped with optimistic and pessimistic planning procedures, achieves strong regret and sample complexity bounds in online and offline RL settings. Particularly, we demonstrate that under the conditions where the trajectory-wise reward is normalized between zero and one and the transition is time-homogenous, it achieves horizon-free and second-order bounds. Horizon-free means that our bounds have no polynomial dependence on the horizon of the Markov Decision Process. A second-order bound is a type of instance-dependent bound that scales with respect to the variances of the returns of the policies which can be small when the system is nearly deterministic and (or) the optimal policy has small values. We highlight that our algorithms are simple, fairly standard, and indeed have been extensively studied in the RL literature: they learn a model via MLE, build a version space around the MLE solution, and perform optimistic or pessimistic planning depending on whether operating in the online or offline mode. These algorithms do not rely on additional specialized algorithmic designs such as learning variances and performing variance-weighted learning and thus can leverage rich function approximations that are significantly beyond linear or tabular structures. The simplicity of the algorithms also implies that our horizon-free and second-order regret analysis is actually standard and mainly follows the general framework of optimism/pessimism in the face of uncertainty.

Title: Classifier-Free Guidance is a Predictor-Corrector

• Authors: Arwen Bradley, Preetum Nakkiran
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09000
• Pdf URL: https://arxiv.org/pdf/2408.09000
• Copy Paste: [[2408.09000]] Classifier-Free Guidance is a Predictor-Corrector(https://arxiv.org/abs/2408.09000)
• Keywords: diffusion
• Abstract: We investigate the theoretical foundations of classifier-free guidance (CFG). CFG is the dominant method of conditional sampling for text-to-image diffusion models, yet unlike other aspects of diffusion, it remains on shaky theoretical footing. In this paper, we disprove common misconceptions, by showing that CFG interacts differently with DDPM (Ho et al., 2020) and DDIM (Song et al., 2021), and neither sampler with CFG generates the gamma-powered distribution $p(x|c)^\gamma p(x)^{1-\gamma}$. Then, we clarify the behavior of CFG by showing that it is a kind of predictor-corrector method (Song et al., 2020) that alternates between denoising and sharpening, which we call predictor-corrector guidance (PCG). We prove that in the SDE limit, CFG is actually equivalent to combining a DDIM predictor for the conditional distribution together with a Langevin dynamics corrector for a gamma-powered distribution (with a carefully chosen gamma). Our work thus provides a lens to theoretically understand CFG by embedding it in a broader design space of principled sampling methods.

Title: Comparative Performance Analysis of Transformer-Based Pre-Trained Models for Detecting Keratoconus Disease

• Authors: Nayeem Ahmed, Md Maruf Rahman, Md Fatin Ishrak, Md Imran Kabir Joy, Md Sanowar Hossain Sabuj, Md. Sadekur Rahman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09005
• Pdf URL: https://arxiv.org/pdf/2408.09005
• Copy Paste: [[2408.09005]] Comparative Performance Analysis of Transformer-Based Pre-Trained Models for Detecting Keratoconus Disease(https://arxiv.org/abs/2408.09005)
• Keywords: robust, transformer, generative
• Abstract: This study compares eight pre-trained CNNs for diagnosing keratoconus, a degenerative eye disease. A carefully selected dataset of keratoconus, normal, and suspicious cases was used. The models tested include DenseNet121, EfficientNetB0, InceptionResNetV2, InceptionV3, MobileNetV2, ResNet50, VGG16, and VGG19. To maximize model training, bad sample removal, resizing, rescaling, and augmentation were used. The models were trained with similar parameters, activation function, classification function, and optimizer to compare performance. To determine class separation effectiveness, each model was evaluated on accuracy, precision, recall, and F1-score. MobileNetV2 was the best accurate model in identifying keratoconus and normal cases with few misclassifications. InceptionV3 and DenseNet121 both performed well in keratoconus detection, but they had trouble with questionable cases. In contrast, EfficientNetB0, ResNet50, and VGG19 had more difficulty distinguishing dubious cases from regular ones, indicating the need for model refining and development. A detailed comparison of state-of-the-art CNN architectures for automated keratoconus identification reveals each model's benefits and weaknesses. This study shows that advanced deep learning models can enhance keratoconus diagnosis and treatment planning. Future research should explore hybrid models and integrate clinical parameters to improve diagnostic accuracy and robustness in real-world clinical applications, paving the way for more effective AI-driven ophthalmology tools.

Title: An optimal pairwise merge algorithm improves the quality and consistency of nonnegative matrix factorization

• Authors: Youdong Guo, Timothy E. Holy
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.09013
• Pdf URL: https://arxiv.org/pdf/2408.09013
• Copy Paste: [[2408.09013]] An optimal pairwise merge algorithm improves the quality and consistency of nonnegative matrix factorization(https://arxiv.org/abs/2408.09013)
• Keywords: extraction
• Abstract: Non-negative matrix factorization (NMF) is a key technique for feature extraction and widely used in source separation. However, existing algorithms may converge to poor local minima, or to one of several minima with similar objective value but differing feature parametrizations. Additionally, the performance of NMF greatly depends on the number of components, but choosing the optimal count remains a challenge. Here we show that some of these weaknesses may be mitigated by performing NMF in a higher-dimensional feature space and then iteratively combining components with an analytically-solvable pairwise merge strategy. Experimental results demonstrate our method helps NMF achieve better local optima and greater consistency of the solutions. Iterative merging also provides an efficient and informative framework for choosing the number of components. Surprisingly, despite these extra steps, our approach often improves computational performance by reducing the occurrence of ``convergence stalling'' near saddle points. This can be recommended as a preferred approach for most applications of NMF.

Title: A Developer-Centric Study Exploring Mobile Application Security Practices and Challenges

• Authors: Anthony Peruma, Timothy Huo, Ana Catarina Araújo, Jake Imanaka, Rick Kazman
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.09032
• Pdf URL: https://arxiv.org/pdf/2408.09032
• Copy Paste: [[2408.09032]] A Developer-Centric Study Exploring Mobile Application Security Practices and Challenges(https://arxiv.org/abs/2408.09032)
• Keywords: secure, security, privacy
• Abstract: Mobile applications (apps) have become an essential part of everyday life, offering convenient access to services such as banking, healthcare, and shopping. With these apps handling sensitive personal and financial data, ensuring their security is paramount. While previous research has explored mobile app developer practices, there is limited knowledge about the common practices and challenges that developers face in securing their apps. Our study addresses this need through a global survey of 137 experienced mobile app developers, providing a developer-centric view of mobile app security. Our findings show that developers place high importance on security, frequently implementing features such as authentication and secure storage. They face challenges with managing vulnerabilities, permissions, and privacy concerns, and often rely on resources like Stack Overflow for help. Many developers find that existing learning materials do not adequately prepare them to build secure apps and provide recommendations, such as following best practices and integrating security at the beginning of the development process. We envision our findings leading to improved security practices, better-designed tools and resources, and more effective training programs.

Title: Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition

• Authors: Muhammad Haseeb Aslam, Marco Pedersoli, Alessandro Lameiras Koerich, Eric Granger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09035
• Pdf URL: https://arxiv.org/pdf/2408.09035
• Copy Paste: [[2408.09035]] Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition(https://arxiv.org/abs/2408.09035)
• Keywords: robust
• Abstract: Human emotion is a complex phenomenon conveyed and perceived through facial expressions, vocal tones, body language, and physiological signals. Multimodal emotion recognition systems can perform well because they can learn complementary and redundant semantic information from diverse sensors. In real-world scenarios, only a subset of the modalities employed for training may be available at test time. Learning privileged information allows a model to exploit data from additional modalities that are only available during training. SOTA methods for PKD have been proposed to distill information from a teacher model (with privileged modalities) to a student model (without privileged modalities). However, such PKD methods utilize point-to-point matching and do not explicitly capture the relational information. Recently, methods have been proposed to distill the structural information. However, PKD methods based on structural similarity are primarily confined to learning from a single joint teacher representation, which limits their robustness, accuracy, and ability to learn from diverse multimodal sources. In this paper, a multi-teacher PKD (MT-PKDOT) method with self-distillation is introduced to align diverse teacher representations before distilling them to the student. MT-PKDOT employs a structural similarity KD mechanism based on a regularized optimal transport (OT) for distillation. The proposed MT-PKDOT method was validated on the Affwild2 and Biovid datasets. Results indicate that our proposed method can outperform SOTA PKD methods. It improves the visual-only baseline on Biovid data by 5.5%. On the Affwild2 dataset, the proposed method improves 3% and 5% over the visual-only baseline for valence and arousal respectively. Allowing the student to learn from multiple diverse sources is shown to increase the accuracy and implicitly avoids negative transfer to the student model.

Title: Improving VTE Identification through Language Models from Radiology Reports: A Comparative Study of Mamba, Phi-3 Mini, and BERT

• Authors: Jamie Deng, Yusen Wu, Yelena Yesha, Phuong Nguyen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09043
• Pdf URL: https://arxiv.org/pdf/2408.09043
• Copy Paste: [[2408.09043]] Improving VTE Identification through Language Models from Radiology Reports: A Comparative Study of Mamba, Phi-3 Mini, and BERT(https://arxiv.org/abs/2408.09043)
• Keywords: large language model
• Abstract: Venous thromboembolism (VTE) is a critical cardiovascular condition, encompassing deep vein thrombosis (DVT) and pulmonary embolism (PE). Accurate and timely identification of VTE is essential for effective medical care. This study builds upon our previous work, which addressed VTE detection using deep learning methods for DVT and a hybrid approach combining deep learning and rule-based classification for PE. Our earlier approaches, while effective, had two major limitations: they were complex and required expert involvement for feature engineering of the rule set. To overcome these challenges, we utilize the Mamba architecture-based classifier. This model achieves remarkable results, with a 97\% accuracy and F1 score on the DVT dataset and a 98\% accuracy and F1 score on the PE dataset. In contrast to the previous hybrid method on PE identification, the Mamba classifier eliminates the need for hand-engineered rules, significantly reducing model complexity while maintaining comparable performance. Additionally, we evaluated a lightweight Large Language Model (LLM), Phi-3 Mini, in detecting VTE. While this model delivers competitive results, outperforming the baseline BERT models, it proves to be computationally intensive due to its larger parameter set. Our evaluation shows that the Mamba-based model demonstrates superior performance and efficiency in VTE identification, offering an effective solution to the limitations of previous approaches.

Title: Language Models Show Stable Value Orientations Across Diverse Role-Plays

• Authors: Bruce W. Lee, Yeongheon Lee, Hyunsoo Cho
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.09049
• Pdf URL: https://arxiv.org/pdf/2408.09049
• Copy Paste: [[2408.09049]] Language Models Show Stable Value Orientations Across Diverse Role-Plays(https://arxiv.org/abs/2408.09049)
• Keywords: large language model
• Abstract: We demonstrate that large language models (LLMs) exhibit consistent value orientations despite adopting diverse personas, revealing a persistent inertia in their responses that remains stable across the variety of roles they are prompted to assume. To systematically explore this phenomenon, we introduce the role-play-at-scale methodology, which involves prompting LLMs with randomized, diverse personas and analyzing the macroscopic trend of their responses. Unlike previous works that simply feed these questions to LLMs as if testing human subjects, our role-play-at-scale methodology diagnoses inherent tendencies in a systematic and scalable manner by: (1) prompting the model to act in different random personas and (2) asking the same question multiple times for each random persona. This approach reveals consistent patterns in LLM responses across diverse role-play scenarios, indicating deeply encoded inherent tendencies. Our findings contribute to the discourse on value alignment in foundation models and demonstrate the efficacy of role-play-at-scale as a diagnostic tool for uncovering encoded biases in LLMs.

Title: MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality

• Authors: Zhiyi Shi, Junsik Kim, Wanhua Li, Yicong Li, Hanspeter Pfister
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09064
• Pdf URL: https://arxiv.org/pdf/2408.09064
• Copy Paste: [[2408.09064]] MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality(https://arxiv.org/abs/2408.09064)
• Keywords: robust
• Abstract: Multi-modal pre-trained models efficiently extract and fuse features from different modalities with low memory requirements for fine-tuning. Despite this efficiency, their application in disease diagnosis is under-explored. A significant challenge is the frequent occurrence of missing modalities, which impairs performance. Additionally, fine-tuning the entire pre-trained model demands substantial computational resources. To address these issues, we introduce Modality-aware Low-Rank Adaptation (MoRA), a computationally efficient method. MoRA projects each input to a low intrinsic dimension but uses different modality-aware up-projections for modality-specific adaptation in cases of missing modalities. Practically, MoRA integrates into the first block of the model, significantly improving performance when a modality is missing. It requires minimal computational resources, with less than 1.6% of the trainable parameters needed compared to training the entire model. Experimental results show that MoRA outperforms existing techniques in disease diagnosis, demonstrating superior performance, robustness, and training efficiency.

Title: Linking Robustness and Generalization: A k* Distribution Analysis of Concept Clustering in Latent Space for Vision Models

• Authors: Shashank Kotyan, Pin-Yu Chen, Danilo Vasconcellos Vargas
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09065
• Pdf URL: https://arxiv.org/pdf/2408.09065
• Copy Paste: [[2408.09065]] Linking Robustness and Generalization: A k* Distribution Analysis of Concept Clustering in Latent Space for Vision Models(https://arxiv.org/abs/2408.09065)
• Keywords: robust
• Abstract: Most evaluations of vision models use indirect methods to assess latent space quality. These methods often involve adding extra layers to project the latent space into a new one. This projection makes it difficult to analyze and compare the original latent space. This article uses the k* Distribution, a local neighborhood analysis method, to examine the learned latent space at the level of individual concepts, which can be extended to examine the entire latent space. We introduce skewness-based true and approximate metrics for interpreting individual concepts to assess the overall quality of vision models' latent space. Our findings indicate that current vision models frequently fracture the distributions of individual concepts within the latent space. Nevertheless, as these models improve in generalization across multiple datasets, the degree of fracturing diminishes. A similar trend is observed in robust vision models, where increased robustness correlates with reduced fracturing. Ultimately, this approach enables a direct interpretation and comparison of the latent spaces of different vision models and reveals a relationship between a model's generalizability and robustness. Results show that as a model becomes more general and robust, it tends to learn features that result in better clustering of concepts. Project Website is available online at this https URL

Title: CodeTaxo: Enhancing Taxonomy Expansion with Limited Examples via Code Language Prompts

• Authors: Qingkai Zeng, Yuyang Bai, Zhaoxuan Tan, Zhenyu Wu, Shangbin Feng, Meng Jiang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.09070
• Pdf URL: https://arxiv.org/pdf/2408.09070
• Copy Paste: [[2408.09070]] CodeTaxo: Enhancing Taxonomy Expansion with Limited Examples via Code Language Prompts(https://arxiv.org/abs/2408.09070)
• Keywords: large language model
• Abstract: Taxonomies play a crucial role in various applications by providing a structural representation of knowledge. The task of taxonomy expansion involves integrating emerging concepts into existing taxonomies by identifying appropriate parent concepts for these new query concepts. Previous approaches typically relied on self-supervised methods that generate annotation data from existing taxonomies. However, these methods are less effective when the existing taxonomy is small (fewer than 100 entities). In this work, we introduce \textsc{CodeTaxo}, a novel approach that leverages large language models through code language prompts to capture the taxonomic structure. Extensive experiments on five real-world benchmarks from different domains demonstrate that \textsc{CodeTaxo} consistently achieves superior performance across all evaluation metrics, significantly outperforming previous state-of-the-art methods. The code and data are available at \url{this https URL}.

Title: Gradient-Variation Online Learning under Generalized Smoothness

• Authors: Yan-Feng Xie, Peng Zhao, Zhi-Hua Zhou
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2408.09074
• Pdf URL: https://arxiv.org/pdf/2408.09074
• Copy Paste: [[2408.09074]] Gradient-Variation Online Learning under Generalized Smoothness(https://arxiv.org/abs/2408.09074)
• Keywords: robust
• Abstract: Gradient-variation online learning aims to achieve regret guarantees that scale with the variations in the gradients of online functions, which has been shown to be crucial for attaining fast convergence in games and robustness in stochastic optimization, hence receiving increased attention. Existing results often require the smoothness condition by imposing a fixed bound on the gradient Lipschitzness, but this may not hold in practice. Recent efforts in neural network optimization suggest a generalized smoothness condition, allowing smoothness to correlate with gradient norms. In this paper, we systematically study gradient-variation online learning under generalized smoothness. To this end, we extend the classic optimistic mirror descent algorithm to derive gradient-variation bounds by conducting stability analysis over the optimization trajectory and exploiting smoothness locally. Furthermore, we explore universal online learning, designing a single algorithm enjoying optimal gradient-variation regrets for convex and strongly convex functions simultaneously without knowing curvature information. The algorithm adopts a two-layer structure with a meta-algorithm running over a group of base-learners. To ensure favorable guarantees, we have designed a new meta-algorithm that is Lipschitz-adaptive to handle potentially unbounded gradients and meanwhile ensures second-order regret to cooperate with base-learners. Finally, we provide implications of our findings and obtain new results in fast-rate games and stochastic extended adversarial optimization.

Title: Twin Sorting Dynamic Programming Assisted User Association and Wireless Bandwidth Allocation for Hierarchical Federated Learning

• Authors: Rung-Hung Gau, Ting-Yu Wang, Chun-Hung Liu
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.09076
• Pdf URL: https://arxiv.org/pdf/2408.09076
• Copy Paste: [[2408.09076]] Twin Sorting Dynamic Programming Assisted User Association and Wireless Bandwidth Allocation for Hierarchical Federated Learning(https://arxiv.org/abs/2408.09076)
• Keywords: federate
• Abstract: In this paper, we study user association and wireless bandwidth allocation for a hierarchical federated learning system that consists of mobile users, edge servers, and a cloud server. To minimize the length of a global round in hierarchical federated learning with equal bandwidth allocation, we formulate a combinatorial optimization problem. We design the twin sorting dynamic programming (TSDP) algorithm that obtains a globally optimal solution in polynomial time when there are two edge servers. In addition, we put forward the TSDP-assisted algorithm for user association when there are three or more edge servers. Furthermore, given a user association matrix, we formulate and solve a convex optimization problem for optimal wireless bandwidth allocation. Simulation results show that the proposed approach outperforms a number of alternative schemes.

Title: Segment Anything with Multiple Modalities

• Authors: Aoran Xiao, Weihao Xuan, Heli Qi, Yun Xing, Naoto Yokoya, Shijian Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09085
• Pdf URL: https://arxiv.org/pdf/2408.09085
• Copy Paste: [[2408.09085]] Segment Anything with Multiple Modalities(https://arxiv.org/abs/2408.09085)
• Keywords: robust, segmentation
• Abstract: Robust and accurate segmentation of scenes has become one core functionality in various visual recognition and navigation tasks. This has inspired the recent development of Segment Anything Model (SAM), a foundation model for general mask segmentation. However, SAM is largely tailored for single-modal RGB images, limiting its applicability to multi-modal data captured with widely-adopted sensor suites, such as LiDAR plus RGB, depth plus RGB, thermal plus RGB, etc. We develop MM-SAM, an extension and expansion of SAM that supports cross-modal and multi-modal processing for robust and enhanced segmentation with different sensor suites. MM-SAM features two key designs, namely, unsupervised cross-modal transfer and weakly-supervised multi-modal fusion, enabling label-efficient and parameter-efficient adaptation toward various sensor modalities. It addresses three main challenges: 1) adaptation toward diverse non-RGB sensors for single-modal processing, 2) synergistic processing of multi-modal data via sensor fusion, and 3) mask-free training for different downstream tasks. Extensive experiments show that MM-SAM consistently outperforms SAM by large margins, demonstrating its effectiveness and robustness across various sensors and data modalities.

Title: BaThe: Defense against the Jailbreak Attack in Multimodal Large Language Models by Treating Harmful Instruction as Backdoor Trigger

• Authors: Yulin Chen, Haoran Li, Zihao Zheng, Yangqiu Song
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.09093
• Pdf URL: https://arxiv.org/pdf/2408.09093
• Copy Paste: [[2408.09093]] BaThe: Defense against the Jailbreak Attack in Multimodal Large Language Models by Treating Harmful Instruction as Backdoor Trigger(https://arxiv.org/abs/2408.09093)
• Keywords: defense, attack, generative, large language model
• Abstract: Multimodal Large Language Models (MLLMs) have showcased impressive performance in a variety of multimodal tasks. On the other hand, the integration of additional image modality may allow the malicious users to inject harmful content inside the images for jailbreaking. Unlike text-based LLMs, where adversaries need to select discrete tokens to conceal their malicious intent using specific algorithms, the continuous nature of image signals provides a direct opportunity for adversaries to inject harmful intentions. In this work, we propose $\textbf{BaThe}$ ($\textbf{Ba}$ckdoor $\textbf{T}$rigger S$\textbf{h}$i$\textbf{e}$ld), a simple yet effective jailbreak defense mechanism. Our work is motivated by recent research on jailbreak backdoor attack and virtual prompt backdoor attack in generative language models. Jailbreak backdoor attack uses harmful instructions combined with manually crafted strings as triggers to make the backdoored model generate prohibited responses. We assume that harmful instructions can function as triggers, and if we alternatively set rejection responses as the triggered response, the backdoored model then can defend against jailbreak attacks. We achieve this by utilizing virtual rejection prompt, similar to the virtual prompt backdoor attack. We embed the virtual rejection prompt into the soft text embeddings, which we call ``wedge''. Our comprehensive experiments demonstrate that BaThe effectively mitigates various types of jailbreak attacks and is adaptable to defend against unseen attacks, with minimal impact on MLLMs' performance.

Title: Depth-guided Texture Diffusion for Image Semantic Segmentation

• Authors: Wei Sun, Yuan Li, Qixiang Ye, Jianbin Jiao, Yanzhao Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09097
• Pdf URL: https://arxiv.org/pdf/2408.09097
• Copy Paste: [[2408.09097]] Depth-guided Texture Diffusion for Image Semantic Segmentation(https://arxiv.org/abs/2408.09097)
• Keywords: diffusion, segmentation
• Abstract: Depth information provides valuable insights into the 3D structure especially the outline of objects, which can be utilized to improve the semantic segmentation tasks. However, a naive fusion of depth information can disrupt feature and compromise accuracy due to the modality gap between the depth and the vision. In this work, we introduce a Depth-guided Texture Diffusion approach that effectively tackles the outlined challenge. Our method extracts low-level features from edges and textures to create a texture image. This image is then selectively diffused across the depth map, enhancing structural information vital for precisely extracting object outlines. By integrating this enriched depth map with the original RGB image into a joint feature embedding, our method effectively bridges the disparity between the depth map and the image, enabling more accurate semantic segmentation. We conduct comprehensive experiments across diverse, commonly-used datasets spanning a wide range of semantic segmentation tasks, including Camouflaged Object Detection (COD), Salient Object Detection (SOD), and indoor semantic segmentation. With source-free estimated depth or depth captured by depth cameras, our method consistently outperforms existing baselines and achieves new state-of-theart results, demonstrating the effectiveness of our Depth-guided Texture Diffusion for image semantic segmentation.

Title: HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction

• Authors: Xiao Zhao, Bo Chen, Mingyang Sun, Dingkang Yang, Youxing Wang, Xukun Zhang, Mingcheng Li, Dongliang Kou, Xiaoyi Wei, Lihua Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09104
• Pdf URL: https://arxiv.org/pdf/2408.09104
• Copy Paste: [[2408.09104]] HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction(https://arxiv.org/abs/2408.09104)
• Keywords: transformer
• Abstract: Vision-based 3D semantic scene completion (SSC) describes autonomous driving scenes through 3D volume representations. However, the occlusion of invisible voxels by scene surfaces poses challenges to current SSC methods in hallucinating refined 3D geometry. This paper proposes HybridOcc, a hybrid 3D volume query proposal method generated by Transformer framework and NeRF representation and refined in a coarse-to-fine SSC prediction framework. HybridOcc aggregates contextual features through the Transformer paradigm based on hybrid query proposals while combining it with NeRF representation to obtain depth supervision. The Transformer branch contains multiple scales and uses spatial cross-attention for 2D to 3D transformation. The newly designed NeRF branch implicitly infers scene occupancy through volume rendering, including visible and invisible voxels, and explicitly captures scene depth rather than generating RGB color. Furthermore, we present an innovative occupancy-aware ray sampling method to orient the SSC task instead of focusing on the scene surface, further improving the overall performance. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the effectiveness of our HybridOcc on the SSC task.

Title: Training Verifiably Robust Agents Using Set-Based Reinforcement Learning

• Authors: Manuel Wendl, Lukas Koller, Tobias Ladner, Matthias Althoff
• Subjects: cs.LG, cs.RO, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.09112
• Pdf URL: https://arxiv.org/pdf/2408.09112
• Copy Paste: [[2408.09112]] Training Verifiably Robust Agents Using Set-Based Reinforcement Learning(https://arxiv.org/abs/2408.09112)
• Keywords: attack, robust
• Abstract: Reinforcement learning often uses neural networks to solve complex control tasks. However, neural networks are sensitive to input perturbations, which makes their deployment in safety-critical environments challenging. This work lifts recent results from formally verifying neural networks against such disturbances to reinforcement learning in continuous state and action spaces using reachability analysis. While previous work mainly focuses on adversarial attacks for robust reinforcement learning, we train neural networks utilizing entire sets of perturbed inputs and maximize the worst-case reward. The obtained agents are verifiably more robust than agents obtained by related work, making them more applicable in safety-critical environments. This is demonstrated with an extensive empirical evaluation of four different benchmarks.

Title: GoodSAM++: Bridging Domain and Capacity Gaps via Segment Anything Model for Panoramic Semantic Segmentation

• Authors: Weiming Zhang, Yexin Liu, Xu Zheng, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09115
• Pdf URL: https://arxiv.org/pdf/2408.09115
• Copy Paste: [[2408.09115]] GoodSAM++: Bridging Domain and Capacity Gaps via Segment Anything Model for Panoramic Semantic Segmentation(https://arxiv.org/abs/2408.09115)
• Keywords: segmentation
• Abstract: This paper presents GoodSAM++, a novel framework utilizing the powerful zero-shot instance segmentation capability of SAM (i.e., teacher) to learn a compact panoramic semantic segmentation model, i.e., student, without requiring any labeled data. GoodSAM++ addresses two critical challenges: 1) SAM's inability to provide semantic labels and inherent distortion problems of panoramic images; 2) the significant capacity disparity between SAM and the student. The `out-of-the-box' insight of GoodSAM++ is to introduce a teacher assistant (TA) to provide semantic information for SAM, integrated with SAM to obtain reliable pseudo semantic maps to bridge both domain and capacity gaps. To make this possible, we first propose a Distortion-Aware Rectification (DARv2) module to address the domain gap. It effectively mitigates the object deformation and distortion problem in panoramic images to obtain pseudo semantic maps. We then introduce a Multi-level Knowledge Adaptation (MKA) module to efficiently transfer the semantic information from the TA and pseudo semantic maps to our compact student model, addressing the significant capacity gap. We conduct extensive experiments on both outdoor and indoor benchmark datasets, showing that our GoodSAM++ achieves a remarkable performance improvement over the state-of-the-art (SOTA) domain adaptation methods. Moreover, diverse open-world scenarios demonstrate the generalization capacity of our GoodSAM++. Last but not least, our most lightweight student model achieves comparable performance to the SOTA models with only 3.7 million parameters.

Title: LOID: Lane Occlusion Inpainting and Detection for Enhanced Autonomous Driving Systems

• Authors: Aayush Agrawal, Ashmitha Jaysi Sivakumar, Ibrahim Kaif, Chayan Banerjee
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.09117
• Pdf URL: https://arxiv.org/pdf/2408.09117
• Copy Paste: [[2408.09117]] LOID: Lane Occlusion Inpainting and Detection for Enhanced Autonomous Driving Systems(https://arxiv.org/abs/2408.09117)
• Keywords: robust, segmentation
• Abstract: Accurate lane detection is essential for effective path planning and lane following in autonomous driving, especially in scenarios with significant occlusion from vehicles and pedestrians. Existing models often struggle under such conditions, leading to unreliable navigation and safety risks. We propose two innovative approaches to enhance lane detection in these challenging environments, each showing notable improvements over current methods. The first approach aug-Segment improves conventional lane detection models by augmenting the training dataset of CULanes with simulated occlusions and training a segmentation model. This method achieves a 12% improvement over a number of SOTA models on the CULanes dataset, demonstrating that enriched training data can better handle occlusions, however, since this model lacked robustness to certain settings, our main contribution is the second approach, LOID Lane Occlusion Inpainting and Detection. LOID introduces an advanced lane detection network that uses an image processing pipeline to identify and mask occlusions. It then employs inpainting models to reconstruct the road environment in the occluded areas. The enhanced image is processed by a lane detection algorithm, resulting in a 20% & 24% improvement over several SOTA models on the BDDK100 and CULanes datasets respectively, highlighting the effectiveness of this novel technique.

Title: Selective Prompt Anchoring for Code Generation

• Authors: Yuan Tian, Tianyi Zhang
• Subjects: cs.LG, cs.AI, cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.09121
• Pdf URL: https://arxiv.org/pdf/2408.09121
• Copy Paste: [[2408.09121]] Selective Prompt Anchoring for Code Generation(https://arxiv.org/abs/2408.09121)
• Keywords: large language model
• Abstract: Recent advances in large language models (LLMs) such as Copilot and ChatGPT have transformed software development by automating coding tasks. Despite these advancements, challenges remain in reducing error rates and fully meeting user expectations. Our empirical study reveals LLMs tend to dilute their self-attention on the initial prompt as more code tokens are generated. We hypothesize this self-attention dilution issue is one of the root causes of inaccuracies in LLM-generated code. To mitigate this issue, we propose Selective Prompt Anchoring (SPA). SPA amplifies the influence of the selected parts in the initial prompt, which we refer to as ``anchored text'', during code generation. Specifically, SPA calculates the logit distribution difference with and without the anchored text. We prove this difference approximates the anchored text's contextual contribution to the output logits. SPA creates an augmented logit distribution by linearly combining the original logit distribution and the logit difference. We evaluate SPA with five LLMs on four benchmarks. Our results demonstrate that using SPA can consistently improve Pass@1 rates by up to 9.7% in all settings. Notably, with selective text anchoring, a small version of DeepSeek-Coder (6.7B) can achieve better performance than an original much larger version (33B). Our code is available at this https URL.

Title: MaskBEV: Towards A Unified Framework for BEV Detection and Map Segmentation

• Authors: Xiao Zhao, Xukun Zhang, Dingkang Yang, Mingyang Sun, Mingcheng Li, Shunli Wang, Lihua Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09122
• Pdf URL: https://arxiv.org/pdf/2408.09122
• Copy Paste: [[2408.09122]] MaskBEV: Towards A Unified Framework for BEV Detection and Map Segmentation(https://arxiv.org/abs/2408.09122)
• Keywords: robust, transformer, segmentation
• Abstract: Accurate and robust multimodal multi-task perception is crucial for modern autonomous driving systems. However, current multimodal perception research follows independent paradigms designed for specific perception tasks, leading to a lack of complementary learning among tasks and decreased performance in multi-task learning (MTL) due to joint training. In this paper, we propose MaskBEV, a masked attention-based MTL paradigm that unifies 3D object detection and bird's eye view (BEV) map segmentation. MaskBEV introduces a task-agnostic Transformer decoder to process these diverse tasks, enabling MTL to be completed in a unified decoder without requiring additional design of specific task heads. To fully exploit the complementary information between BEV map segmentation and 3D object detection tasks in BEV space, we propose spatial modulation and scene-level context aggregation strategies. These strategies consider the inherent dependencies between BEV segmentation and 3D detection, naturally boosting MTL performance. Extensive experiments on nuScenes dataset show that compared with previous state-of-the-art MTL methods, MaskBEV achieves 1.3 NDS improvement in 3D object detection and 2.7 mIoU improvement in BEV map segmentation, while also demonstrating slightly leading inference speed.

Title: Barbie: Text to Barbie-Style 3D Avatars

• Authors: Xiaokun Sun, Zhenyu Zhang, Ying Tai, Qian Wang, Hao Tang, Zili Yi, Jian Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09126
• Pdf URL: https://arxiv.org/pdf/2408.09126
• Copy Paste: [[2408.09126]] Barbie: Text to Barbie-Style 3D Avatars(https://arxiv.org/abs/2408.09126)
• Keywords: diffusion
• Abstract: Recent advances in text-guided 3D avatar generation have made substantial progress by distilling knowledge from diffusion models. Despite the plausible generated appearance, existing methods cannot achieve fine-grained disentanglement or high-fidelity modeling between inner body and outfit. In this paper, we propose Barbie, a novel framework for generating 3D avatars that can be dressed in diverse and high-quality Barbie-like garments and accessories. Instead of relying on a holistic model, Barbie achieves fine-grained disentanglement on avatars by semantic-aligned separated models for human body and outfits. These disentangled 3D representations are then optimized by different expert models to guarantee the domain-specific fidelity. To balance geometry diversity and reasonableness, we propose a series of losses for template-preserving and human-prior evolving. The final avatar is enhanced by unified texture refinement for superior texture consistency. Extensive experiments demonstrate that Barbie outperforms existing methods in both dressed human and outfit generation, supporting flexible apparel combination and animation. The code will be released for research purposes. Our project page is: this https URL.

Title: StylePrompter: Enhancing Domain Generalization with Test-Time Style Priors

• Authors: Jiao Zhang, Jian Xu, Xu-Yao Zhang, Cheng-Lin Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09138
• Pdf URL: https://arxiv.org/pdf/2408.09138
• Copy Paste: [[2408.09138]] StylePrompter: Enhancing Domain Generalization with Test-Time Style Priors(https://arxiv.org/abs/2408.09138)
• Keywords: robust
• Abstract: In real-world applications, the sample distribution at the inference stage often differs from the one at the training stage, causing performance degradation of trained deep models. The research on domain generalization (DG) aims to develop robust algorithms that can improve the generalized performance in unseen domains by training on a few domains. However, the domain-agnostic vision model, trained on a limited number of domains using traditional domain generalization methods, cannot guarantee its effectiveness in dealing with unseen domains. The introduction of language can break the closed cognition space of the vision model, providing additional semantic information that cannot be inferred from vision-only datasets. In this paper, we propose to overcome the challenge in previous DG methods by introducing the style prompt in the language modality to adapt the trained model dynamically. In particular, we train a style prompter to extract style information of the current image into an embedding in the token embedding space and place it in front of the candidate category words as prior knowledge to prompt the model. Our open space partition of the style token embedding space and the hand-crafted style regularization enable the trained style prompter to handle data from unknown domains effectively. Extensive experiments verify the effectiveness of our method and demonstrate state-of-the-art performances on multiple public datasets. Codes will be available after the acceptance of this paper.

Title: SSNeRF: Sparse View Semi-supervised Neural Radiance Fields with Augmentation

• Authors: Xiao Cao, Beibei Lin, Bo Wang, Zhiyong Huang, Robby T. Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09144
• Pdf URL: https://arxiv.org/pdf/2408.09144
• Copy Paste: [[2408.09144]] SSNeRF: Sparse View Semi-supervised Neural Radiance Fields with Augmentation(https://arxiv.org/abs/2408.09144)
• Keywords: robust
• Abstract: Sparse view NeRF is challenging because limited input images lead to an under constrained optimization problem for volume rendering. Existing methods address this issue by relying on supplementary information, such as depth maps. However, generating this supplementary information accurately remains problematic and often leads to NeRF producing images with undesired artifacts. To address these artifacts and enhance robustness, we propose SSNeRF, a sparse view semi supervised NeRF method based on a teacher student framework. Our key idea is to challenge the NeRF module with progressively severe sparse view degradation while providing high confidence pseudo labels. This approach helps the NeRF model become aware of noise and incomplete information associated with sparse views, thus improving its robustness. The novelty of SSNeRF lies in its sparse view specific augmentations and semi supervised learning mechanism. In this approach, the teacher NeRF generates novel views along with confidence scores, while the student NeRF, perturbed by the augmented input, learns from the high confidence pseudo labels. Our sparse view degradation augmentation progressively injects noise into volume rendering weights, perturbs feature maps in vulnerable layers, and simulates sparse view blurriness. These augmentation strategies force the student NeRF to recognize degradation and produce clearer rendered views. By transferring the student's parameters to the teacher, the teacher gains increased robustness in subsequent training iterations. Extensive experiments demonstrate the effectiveness of our SSNeRF in generating novel views with less sparse view degradation. We will release code upon acceptance.

Title: CogLM: Tracking Cognitive Development of Large Language Models

• Authors: Xinglin Wang, Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Boyuan Pan, Heda Wang, Yao Hu, Kan Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09150
• Pdf URL: https://arxiv.org/pdf/2408.09150
• Copy Paste: [[2408.09150]] CogLM: Tracking Cognitive Development of Large Language Models(https://arxiv.org/abs/2408.09150)
• Keywords: large language model
• Abstract: Piaget's Theory of Cognitive Development (PTC) posits that the development of cognitive levels forms the foundation for human learning across various abilities. As Large Language Models (LLMs) have recently shown remarkable abilities across a wide variety of tasks, we are curious about the cognitive levels of current LLMs: to what extent they have developed and how this development has been achieved. To this end, we construct a benchmark CogLM (Cognitive Ability Evaluation for Language Model) based on PTC to assess the cognitive levels of LLMs. CogLM comprises 1,220 questions spanning 10 cognitive abilities crafted by more than 20 human experts, providing a comprehensive testbed for the cognitive levels of LLMs. Through extensive experiments across multiple mainstream LLMs with CogLM, we find that: (1) Human-like cognitive abilities have emerged in advanced LLMs (GPT-4), comparable to those of a 20-year-old human. (2) The parameter size and optimization objective are two key factors affecting the cognitive levels of LLMs. (3) The performance on downstream tasks is positively correlated with the level of cognitive abilities. These findings fill the gap in research on the cognitive abilities of LLMs, tracing the development of LLMs from a cognitive perspective and guiding the future direction of their evolution.

Title: Realistic Extreme Image Rescaling via Generative Latent Space Learning

• Authors: Ce Wang, Wanjie Sun, Zhenzhong Chen
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.09151
• Pdf URL: https://arxiv.org/pdf/2408.09151
• Copy Paste: [[2408.09151]] Realistic Extreme Image Rescaling via Generative Latent Space Learning(https://arxiv.org/abs/2408.09151)
• Keywords: diffusion, generative
• Abstract: Image rescaling aims to learn the optimal downscaled low-resolution (LR) image that can be accurately reconstructed to its original high-resolution (HR) counterpart. This process is crucial for efficient image processing and storage, especially in the era of ultra-high definition media. However, extreme downscaling factors pose significant challenges due to the highly ill-posed nature of the inverse upscaling process, causing existing methods to struggle in generating semantically plausible structures and perceptually rich textures. In this work, we propose a novel framework called Latent Space Based Image Rescaling (LSBIR) for extreme image rescaling tasks. LSBIR effectively leverages powerful natural image priors learned by a pre-trained text-to-image diffusion model to generate realistic HR images. The rescaling is performed in the latent space of a pre-trained image encoder and decoder, which offers better perceptual reconstruction quality due to its stronger sparsity and richer semantics. LSBIR adopts a two-stage training strategy. In the first stage, a pseudo-invertible encoder-decoder models the bidirectional mapping between the latent features of the HR image and the target-sized LR image. In the second stage, the reconstructed features from the first stage are refined by a pre-trained diffusion model to generate more faithful and visually pleasing details. Extensive experiments demonstrate the superiority of LSBIR over previous methods in both quantitative and qualitative evaluations. The code will be available at: this https URL.

Title: Are CLIP features all you need for Universal Synthetic Image Origin Attribution?

• Authors: Dario Cioni, Christos Tzelepis, Lorenzo Seidenari, Ioannis Patras
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09153
• Pdf URL: https://arxiv.org/pdf/2408.09153
• Copy Paste: [[2408.09153]] Are CLIP features all you need for Universal Synthetic Image Origin Attribution?(https://arxiv.org/abs/2408.09153)
• Keywords: diffusion, generative
• Abstract: The steady improvement of Diffusion Models for visual synthesis has given rise to many new and interesting use cases of synthetic images but also has raised concerns about their potential abuse, which poses significant societal threats. To address this, fake images need to be detected and attributed to their source model, and given the frequent release of new generators, realistic applications need to consider an Open-Set scenario where some models are unseen at training time. Existing forensic techniques are either limited to Closed-Set settings or to GAN-generated images, relying on fragile frequency-based "fingerprint" features. By contrast, we propose a simple yet effective framework that incorporates features from large pre-trained foundation models to perform Open-Set origin attribution of synthetic images produced by various generative models, including Diffusion Models. We show that our method leads to remarkable attribution performance, even in the low-data regime, exceeding the performance of existing methods and generalizes better on images obtained from a diverse set of architectures. We make the code publicly available at: this https URL.

Title: On the KL-Divergence-based Robust Satisficing Model

• Authors: Haojie Yan, Minglong Zhou, Jiayi Guo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09157
• Pdf URL: https://arxiv.org/pdf/2408.09157
• Copy Paste: [[2408.09157]] On the KL-Divergence-based Robust Satisficing Model(https://arxiv.org/abs/2408.09157)
• Keywords: robust
• Abstract: Empirical risk minimization, a cornerstone in machine learning, is often hindered by the Optimizer's Curse stemming from discrepancies between the empirical and true data-generating this http URL address this challenge, the robust satisficing framework has emerged recently to mitigate ambiguity in the true distribution. Distinguished by its interpretable hyperparameter and enhanced performance guarantees, this approach has attracted increasing attention from academia. However, its applicability in tackling general machine learning problems, notably deep neural networks, remains largely unexplored due to the computational challenges in solving this model efficiently across general loss functions. In this study, we delve into the Kullback Leibler divergence based robust satisficing model under a general loss function, presenting analytical interpretations, diverse performance guarantees, efficient and stable numerical methods, convergence analysis, and an extension tailored for hierarchical data structures. Through extensive numerical experiments across three distinct machine learning tasks, we demonstrate the superior performance of our model compared to state-of-the-art benchmarks.

Title: Linear Attention is Enough in Spatial-Temporal Forecasting

• Authors: Xinyu Ning
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09158
• Pdf URL: https://arxiv.org/pdf/2408.09158
• Copy Paste: [[2408.09158]] Linear Attention is Enough in Spatial-Temporal Forecasting(https://arxiv.org/abs/2408.09158)
• Keywords: transformer
• Abstract: As the most representative scenario of spatial-temporal forecasting tasks, the traffic forecasting task attracted numerous attention from machine learning community due to its intricate correlation both in space and time dimension. Existing methods often treat road networks over time as spatial-temporal graphs, addressing spatial and temporal representations independently. However, these approaches struggle to capture the dynamic topology of road networks, encounter issues with message passing mechanisms and over-smoothing, and face challenges in learning spatial and temporal relationships separately. To address these limitations, we propose treating nodes in road networks at different time steps as independent spatial-temporal tokens and feeding them into a vanilla Transformer to learn complex spatial-temporal patterns, design STformer achieving SOTA. Given its quadratic complexity, we introduce a variant NSTformer based on Nystr$\ddot{o}$m method to approximate self-attention with linear complexity but even slightly better than former in a few cases astonishingly. Extensive experimental results on traffic datasets demonstrate that the proposed method achieves state-of-the-art performance at an affordable computational cost. Our code will be made available.

Title: TableBench: A Comprehensive and Complex Benchmark for Table Question Answering

• Authors: Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Guanglin Niu, Tongliang Li, Zhoujun Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09174
• Pdf URL: https://arxiv.org/pdf/2408.09174
• Copy Paste: [[2408.09174]] TableBench: A Comprehensive and Complex Benchmark for Table Question Answering(https://arxiv.org/abs/2408.09174)
• Keywords: large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have markedly enhanced the interpretation and processing of tabular data, introducing previously unimaginable capabilities. Despite these achievements, LLMs still encounter significant challenges when applied in industrial scenarios, particularly due to the increased complexity of reasoning required with real-world tabular data, underscoring a notable disparity between academic benchmarks and practical applications. To address this discrepancy, we conduct a detailed investigation into the application of tabular data in industrial scenarios and propose a comprehensive and complex benchmark TableBench, including 18 fields within four major categories of table question answering (TableQA) capabilities. Furthermore, we introduce TableLLM, trained on our meticulously constructed training set TableInstruct, achieving comparable performance with GPT-3.5. Massive experiments conducted on TableBench indicate that both open-source and proprietary LLMs still have significant room for improvement to meet real-world demands, where the most advanced model, GPT-4, achieves only a modest score compared to humans.

Title: Chinese Metaphor Recognition Using a Multi-stage Prompting Large Language Model

• Authors: Jie Wang, Jin Wang, Xuejie Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09177
• Pdf URL: https://arxiv.org/pdf/2408.09177
• Copy Paste: [[2408.09177]] Chinese Metaphor Recognition Using a Multi-stage Prompting Large Language Model(https://arxiv.org/abs/2408.09177)
• Keywords: generative, large language model
• Abstract: Metaphors are common in everyday language, and the identification and understanding of metaphors are facilitated by models to achieve a better understanding of the text. Metaphors are mainly identified and generated by pre-trained models in existing research, but situations, where tenors or vehicles are not included in the metaphor, cannot be handled. The problem can be effectively solved by using Large Language Models (LLMs), but significant room for exploration remains in this early-stage research area. A multi-stage generative heuristic-enhanced prompt framework is proposed in this study to enhance the ability of LLMs to recognize tenors, vehicles, and grounds in Chinese metaphors. In the first stage, a small model is trained to obtain the required confidence score for answer candidate generation. In the second stage, questions are clustered and sampled according to specific rules. Finally, the heuristic-enhanced prompt needed is formed by combining the generated answer candidates and demonstrations. The proposed model achieved 3rd place in Track 1 of Subtask 1, 1st place in Track 2 of Subtask 1, and 1st place in both tracks of Subtask 2 at the NLPCC-2024 Shared Task 9.

Title: PADetBench: Towards Benchmarking Physical Attacks against Object Detection

• Authors: Jiawei Lian, Jianhong Pan, Lefan Wang, Yi Wang, Lap-Pui Chau, Shaohui Mei
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09181
• Pdf URL: https://arxiv.org/pdf/2408.09181
• Copy Paste: [[2408.09181]] PADetBench: Towards Benchmarking Physical Attacks against Object Detection(https://arxiv.org/abs/2408.09181)
• Keywords: attack, robust, fair
• Abstract: Physical attacks against object detection have gained increasing attention due to their significant practical implications. However, conducting physical experiments is extremely time-consuming and labor-intensive. Moreover, physical dynamics and cross-domain transformation are challenging to strictly regulate in the real world, leading to unaligned evaluation and comparison, severely hindering the development of physically robust models. To accommodate these challenges, we explore utilizing realistic simulation to thoroughly and rigorously benchmark physical attacks with fairness under controlled physical dynamics and cross-domain transformation. This resolves the problem of capturing identical adversarial images that cannot be achieved in the real world. Our benchmark includes 20 physical attack methods, 48 object detectors, comprehensive physical dynamics, and evaluation metrics. We also provide end-to-end pipelines for dataset generation, detection, evaluation, and further analysis. In addition, we perform 8064 groups of evaluation based on our benchmark, which includes both overall evaluation and further detailed ablation studies for controlled physical dynamics. Through these experiments, we provide in-depth analyses of physical attack performance and physical adversarial robustness, draw valuable observations, and discuss potential directions for future research. Codebase: this https URL

Title: DRL-Based Resource Allocation for Motion Blur Resistant Federated Self-Supervised Learning in IoV

• Authors: Xueying Gu, Qiong Wu, Pingyi Fan, Qiang Fan, Nan Cheng, Wen Chen, Khaled B. Letaief
• Subjects: cs.CV, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.09194
• Pdf URL: https://arxiv.org/pdf/2408.09194
• Copy Paste: [[2408.09194]] DRL-Based Resource Allocation for Motion Blur Resistant Federated Self-Supervised Learning in IoV(https://arxiv.org/abs/2408.09194)
• Keywords: privacy, protect, federate
• Abstract: In the Internet of Vehicles (IoV), Federated Learning (FL) provides a privacy-preserving solution by aggregating local models without sharing data. Traditional supervised learning requires image data with labels, but data labeling involves significant manual effort. Federated Self-Supervised Learning (FSSL) utilizes Self-Supervised Learning (SSL) for local training in FL, eliminating the need for labels while protecting privacy. Compared to other SSL methods, Momentum Contrast (MoCo) reduces the demand for computing resources and storage space by creating a dictionary. However, using MoCo in FSSL requires uploading the local dictionary from vehicles to Base Station (BS), which poses a risk of privacy leakage. Simplified Contrast (SimCo) addresses the privacy leakage issue in MoCo-based FSSL by using dual temperature instead of a dictionary to control sample distribution. Additionally, considering the negative impact of motion blur on model aggregation, and based on SimCo, we propose a motion blur-resistant FSSL method, referred to as BFSSL. Furthermore, we address energy consumption and delay in the BFSSL process by proposing a Deep Reinforcement Learning (DRL)-based resource allocation scheme, called DRL-BFSSL. In this scheme, BS allocates the Central Processing Unit (CPU) frequency and transmission power of vehicles to minimize energy consumption and latency, while aggregating received models based on the motion blur level. Simulation results validate the effectiveness of our proposed aggregation and resource allocation methods.

Title: Architectural Foundations and Strategic Considerations for the Large Language Model Infrastructures

• Authors: Hongyin Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09205
• Pdf URL: https://arxiv.org/pdf/2408.09205
• Copy Paste: [[2408.09205]] Architectural Foundations and Strategic Considerations for the Large Language Model Infrastructures(https://arxiv.org/abs/2408.09205)
• Keywords: robust, large language model
• Abstract: The development of a large language model (LLM) infrastructure is a pivotal undertaking in artificial intelligence. This paper explores the intricate landscape of LLM infrastructure, software, and data management. By analyzing these core components, we emphasize the pivotal considerations and safeguards crucial for successful LLM development. This work presents a concise synthesis of the challenges and strategies inherent in constructing a robust and effective LLM infrastructure, offering valuable insights for researchers and practitioners alike.

Title: EagleEye: Attention to Unveil Malicious Event Sequences from Provenance Graphs

• Authors: Philipp Gysel, Candid Wüest, Kenneth Nwafor, Otakar Jašek, Andrey Ustyuzhanin, Dinil Mon Divakaran
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.09217
• Pdf URL: https://arxiv.org/pdf/2408.09217
• Copy Paste: [[2408.09217]] EagleEye: Attention to Unveil Malicious Event Sequences from Provenance Graphs(https://arxiv.org/abs/2408.09217)
• Keywords: attack, transformer
• Abstract: Securing endpoints is challenging due to the evolving nature of threats and attacks. With endpoint logging systems becoming mature, provenance-graph representations enable the creation of sophisticated behavior rules. However, adapting to the pace of emerging attacks is not scalable with rules. This led to the development of ML models capable of learning from endpoint logs. However, there are still open challenges: i) malicious patterns of malware are spread across long sequences of events, and ii) ML classification results are not interpretable. To address these issues, we develop and present EagleEye, a novel system that i) uses rich features from provenance graphs for behavior event representation, including command-line embeddings, ii) extracts long sequences of events and learns event embeddings, and iii) trains a lightweight Transformer model to classify behavior sequences as malicious or not. We evaluate and compare EagleEye against state-of-the-art baselines on two datasets, namely a new real-world dataset from a corporate environment, and the public DARPA dataset. On the DARPA dataset, at a false-positive rate of 1%, EagleEye detects $\approx$89% of all malicious behavior, outperforming two state-of-the-art solutions by an absolute margin of 38.5%. Furthermore, we show that the Transformer's attention mechanism can be leveraged to highlight the most suspicious events in a long sequence, thereby providing interpretation of malware alerts.

Title: Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text

• Authors: Sher Badshah, Hassan Sajjad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09235
• Pdf URL: https://arxiv.org/pdf/2408.09235
• Copy Paste: [[2408.09235]] Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text(https://arxiv.org/abs/2408.09235)
• Keywords: robust, generative, large language model
• Abstract: The rapid advancements in Large Language Models (LLMs) have highlighted the critical need for robust evaluation methods that can accurately assess the quality of generated text, particularly in free-form tasks. Traditional metrics like BLEU and ROUGE, while useful, often fail to capture the semantic richness and contextual relevance of free-form text compared to reference answers. In this study, we introduce a reference-guided verdict method that leverages multiple LLMs-as-judges to provide a more reliable and accurate evaluation of open-ended LLM generations. By integrating diverse LLMs, our approach mitigates individual model biases and significantly improves alignment with human judgments, especially in challenging tasks where traditional metrics and single-model evaluations fall short. Through experiments across multiple question-answering tasks, we show that our method closely aligns with human evaluations, establishing it as a scalable, reproducible, and effective alternative to human evaluation. Our approach not only enhances evaluation reliability but also opens new avenues for refining automated assessment in generative AI.

Title: RepControlNet: ControlNet Reparameterization

• Authors: Zhaoli Deng, Kaibin Zhou, Fanyi Wang, Zhenpeng Mi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09240
• Pdf URL: https://arxiv.org/pdf/2408.09240
• Copy Paste: [[2408.09240]] RepControlNet: ControlNet Reparameterization(https://arxiv.org/abs/2408.09240)
• Keywords: diffusion
• Abstract: With the wide application of diffusion model, the high cost of inference resources has became an important bottleneck for its universal application. Controllable generation, such as ControlNet, is one of the key research directions of diffusion model, and the research related to inference acceleration and model compression is more important. In order to solve this problem, this paper proposes a modal reparameterization method, RepControlNet, to realize the controllable generation of diffusion models without increasing computation. In the training process, RepControlNet uses the adapter to modulate the modal information into the feature space, copy the CNN and MLP learnable layers of the original diffusion model as the modal network, and initialize these weights based on the original weights and coefficients. The training process only optimizes the parameters of the modal network. In the inference process, the weights of the neutralization original diffusion model in the modal network are reparameterized, which can be compared with or even surpass the methods such as ControlNet, which use additional parameters and computational quantities, without increasing the number of parameters. We have carried out a large number of experiments on both SD1.5 and SDXL, and the experimental results show the effectiveness and efficiency of the proposed RepControlNet.

Title: Re-boosting Self-Collaboration Parallel Prompt GAN for Unsupervised Image Restoration

• Authors: Xin Lin, Yuyan Zhou, Jingtong Yue, Chao Ren, Kelvin C.K. Chan, Lu Qi, Ming-Hsuan Yang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.09241
• Pdf URL: https://arxiv.org/pdf/2408.09241
• Copy Paste: [[2408.09241]] Re-boosting Self-Collaboration Parallel Prompt GAN for Unsupervised Image Restoration(https://arxiv.org/abs/2408.09241)
• Keywords: generative
• Abstract: Unsupervised restoration approaches based on generative adversarial networks (GANs) offer a promising solution without requiring paired datasets. Yet, these GAN-based approaches struggle to surpass the performance of conventional unsupervised GAN-based frameworks without significantly modifying model structures or increasing the computational complexity. To address these issues, we propose a self-collaboration (SC) strategy for existing restoration models. This strategy utilizes information from the previous stage as feedback to guide subsequent stages, achieving significant performance improvement without increasing the framework's inference complexity. The SC strategy comprises a prompt learning (PL) module and a restorer ($Res$). It iteratively replaces the previous less powerful fixed restorer $\overline{Res}$ in the PL module with a more powerful $Res$. The enhanced PL module generates better pseudo-degraded/clean image pairs, leading to a more powerful $Res$ for the next iteration. Our SC can significantly improve the $Res$'s performance by over 1.5 dB without adding extra parameters or computational complexity during inference. Meanwhile, existing self-ensemble (SE) and our SC strategies enhance the performance of pre-trained restorers from different perspectives. As SE increases computational complexity during inference, we propose a re-boosting module to the SC (Reb-SC) to improve the SC strategy further by incorporating SE into SC without increasing inference time. This approach further enhances the restorer's performance by approximately 0.3 dB. Extensive experimental results on restoration tasks demonstrate that the proposed model performs favorably against existing state-of-the-art unsupervised restoration methods. Source code and trained models are publicly available at: \url{this https URL}.

Title: MagicID: Flexible ID Fidelity Generation System

• Authors: Zhaoli Deng, Wen Liu, Fanyi Wang, Junkang Zhang, Fan Chen, Wendong Zhang, Zhenpeng Mi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09248
• Pdf URL: https://arxiv.org/pdf/2408.09248
• Copy Paste: [[2408.09248]] MagicID: Flexible ID Fidelity Generation System(https://arxiv.org/abs/2408.09248)
• Keywords: diffusion, generative
• Abstract: Portrait Fidelity Generation is a prominent research area in generative models, with a primary focus on enhancing both controllability and fidelity. Current methods face challenges in generating high-fidelity portrait results when faces occupy a small portion of the image with a low resolution, especially in multi-person group photo settings. To tackle these issues, we propose a systematic solution called MagicID, based on a self-constructed million-level multi-modal dataset named IDZoom. MagicID consists of Multi-Mode Fusion training strategy (MMF) and DDIM Inversion based ID Restoration inference framework (DIIR). During training, MMF iteratively uses the skeleton and landmark modalities from IDZoom as conditional guidance. By introducing the Clone Face Tuning in training stage and Mask Guided Multi-ID Cross Attention (MGMICA) in inference stage, explicit constraints on face positional features are achieved for multi-ID group photo generation. The DIIR aims to address the issue of artifacts. The DDIM Inversion is used in conjunction with face landmarks, global and local face features to achieve face restoration while keeping the background unchanged. Additionally, DIIR is plug-and-play and can be applied to any diffusion-based portrait generation method. To validate the effectiveness of MagicID, we conducted extensive comparative and ablation experiments. The experimental results demonstrate that MagicID has significant advantages in both subjective and objective metrics, and achieves controllable generation in multi-person scenarios.

Title: PREMAP: A Unifying PREiMage APproximation Framework for Neural Networks

• Authors: Xiyue Zhang, Benjie Wang, Marta Kwiatkowska, Huan Zhang
• Subjects: cs.LG, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2408.09262
• Pdf URL: https://arxiv.org/pdf/2408.09262
• Copy Paste: [[2408.09262]] PREMAP: A Unifying PREiMage APproximation Framework for Neural Networks(https://arxiv.org/abs/2408.09262)
• Keywords: robust
• Abstract: Most methods for neural network verification focus on bounding the image, i.e., set of outputs for a given input set. This can be used to, for example, check the robustness of neural network predictions to bounded perturbations of an input. However, verifying properties concerning the preimage, i.e., the set of inputs satisfying an output property, requires abstractions in the input space. We present a general framework for preimage abstraction that produces under- and over-approximations of any polyhedral output set. Our framework employs cheap parameterised linear relaxations of the neural network, together with an anytime refinement procedure that iteratively partitions the input region by splitting on input features and neurons. The effectiveness of our approach relies on carefully designed heuristics and optimization objectives to achieve rapid improvements in the approximation volume. We evaluate our method on a range of tasks, demonstrating significant improvement in efficiency and scalability to high-input-dimensional image classification tasks compared to state-of-the-art techniques. Further, we showcase the application to quantitative verification and robustness analysis, presenting a sound and complete algorithm for the former and providing sound quantitative results for the latter.

Title: ByCAN: Reverse Engineering Controller Area Network (CAN) Messages from Bit to Byte Level

• Authors: Xiaojie Lin, Baihe Ma, Xu Wang, Guangsheng Yu, Ying He, Ren Ping Liu, Wei Ni
• Subjects: cs.CR, cs.LG, cs.NI, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.09265
• Pdf URL: https://arxiv.org/pdf/2408.09265
• Copy Paste: [[2408.09265]] ByCAN: Reverse Engineering Controller Area Network (CAN) Messages from Bit to Byte Level(https://arxiv.org/abs/2408.09265)
• Keywords: security
• Abstract: As the primary standard protocol for modern cars, the Controller Area Network (CAN) is a critical research target for automotive cybersecurity threats and autonomous applications. As the decoding specification of CAN is a proprietary black-box maintained by Original Equipment Manufacturers (OEMs), conducting related research and industry developments can be challenging without a comprehensive understanding of the meaning of CAN messages. In this paper, we propose a fully automated reverse-engineering system, named ByCAN, to reverse engineer CAN messages. ByCAN outperforms existing research by introducing byte-level clusters and integrating multiple features at both byte and bit levels. ByCAN employs the clustering and template matching algorithms to automatically decode the specifications of CAN frames without the need for prior knowledge. Experimental results demonstrate that ByCAN achieves high accuracy in slicing and labeling performance, i.e., the identification of CAN signal boundaries and labels. In the experiments, ByCAN achieves slicing accuracy of 80.21%, slicing coverage of 95.21%, and labeling accuracy of 68.72% for general labels when analyzing the real-world CAN frames.

Title: ConVerSum: A Contrastive Learning based Approach for Data-Scarce Solution of Cross-Lingual Summarization Beyond Direct Equivalents

• Authors: Sanzana Karim Lora, Rifat Shahriyar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09273
• Pdf URL: https://arxiv.org/pdf/2408.09273
• Copy Paste: [[2408.09273]] ConVerSum: A Contrastive Learning based Approach for Data-Scarce Solution of Cross-Lingual Summarization Beyond Direct Equivalents(https://arxiv.org/abs/2408.09273)
• Keywords: large language model
• Abstract: Cross-Lingual summarization (CLS) is a sophisticated branch in Natural Language Processing that demands models to accurately translate and summarize articles from different source languages. Despite the improvement of the subsequent studies, This area still needs data-efficient solutions along with effective training methodologies. To the best of our knowledge, there is no feasible solution for CLS when there is no available high-quality CLS data. In this paper, we propose a novel data-efficient approach, ConVerSum, for CLS leveraging the power of contrastive learning, generating versatile candidate summaries in different languages based on the given source document and contrasting these summaries with reference summaries concerning the given documents. After that, we train the model with a contrastive ranking loss. Then, we rigorously evaluate the proposed approach against current methodologies and compare it to powerful Large Language Models (LLMs)- Gemini, GPT 3.5, and GPT 4 proving our model performs better for low-resource languages' CLS. These findings represent a substantial improvement in the area, opening the door to more efficient and accurate cross-lingual summarizing techniques.

Title: Multi-Camera Multi-Person Association using Transformer-Based Dense Pixel Correspondence Estimation and Detection-Based Masking

• Authors: Daniel Kathein, Byron Hernandez, Henry Medeiros
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09295
• Pdf URL: https://arxiv.org/pdf/2408.09295
• Copy Paste: [[2408.09295]] Multi-Camera Multi-Person Association using Transformer-Based Dense Pixel Correspondence Estimation and Detection-Based Masking(https://arxiv.org/abs/2408.09295)
• Keywords: transformer
• Abstract: Multi-camera Association (MCA) is the task of identifying objects and individuals across camera views and is an active research topic, given its numerous applications across robotics, surveillance, and agriculture. We investigate a novel multi-camera multi-target association algorithm based on dense pixel correspondence estimation with a Transformer-based architecture and underlying detection-based masking. After the algorithm generates a set of corresponding keypoints and their respective confidence levels between every pair of detections in the camera views are computed, an affinity matrix is determined containing the probabilities of matches between each pair. Finally, the Hungarian algorithm is applied to generate an optimal assignment matrix with all the predicted associations between the camera views. Our method is evaluated on the WILDTRACK Seven-Camera HD Dataset, a high-resolution dataset containing footage of walking pedestrians as well as precise annotations and camera calibrations. Our results conclude that the algorithm performs exceptionally well associating pedestrians on camera pairs that are positioned close to each other and observe the scene from similar perspectives. On camera pairs with orientations that are drastically different in distance or angle, there is still significant room for improvement.

Title: CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions

• Authors: Matan Levi, Yair Alluouche, Daniel Ohayon, Anton Puzanov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09304
• Pdf URL: https://arxiv.org/pdf/2408.09304
• Copy Paste: [[2408.09304]] CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions(https://arxiv.org/abs/2408.09304)
• Keywords: security, large language model
• Abstract: Large Language Models (LLMs) have significantly advanced natural language processing (NLP), providing versatile capabilities across various applications. However, their application to complex, domain-specific tasks, such as cyber-security, often faces substantial challenges. In this study, we introduce SecKnowledge and this http URL to address these challenges and train security-expert LLMs. SecKnowledge is a domain-knowledge-driven cyber-security instruction dataset, meticulously designed using years of accumulated expert knowledge in the domain through a multi-phase generation process. this http URL refers to a family of LLMs fine-tuned using SecKnowledge, aimed at building security-specialized LLMs capable of answering and following complex security-related instructions. Additionally, we introduce SecKnowledge-Eval, a comprehensive and diverse cyber-security evaluation benchmark, composed of an extensive set of cyber-security tasks we specifically developed to assess LLMs in the field of cyber-security, along with other publicly available security benchmarks. Our results show a significant average improvement of up to 24% over the baseline models, underscoring the benefits of our expert-driven instruction dataset generation process. These findings contribute to the advancement of AI-based cyber-security applications, paving the way for security-expert LLMs that can enhance threat-hunting and investigation processes.

Title: A Benchmark Time Series Dataset for Semiconductor Fabrication Manufacturing Constructed using Component-based Discrete-Event Simulation Models

• Authors: Vamsi Krishna Pendyala, Hessam S. Sarjoughian, Bala Potineni, Edward J. Yellig
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09307
• Pdf URL: https://arxiv.org/pdf/2408.09307
• Copy Paste: [[2408.09307]] A Benchmark Time Series Dataset for Semiconductor Fabrication Manufacturing Constructed using Component-based Discrete-Event Simulation Models(https://arxiv.org/abs/2408.09307)
• Keywords: diffusion
• Abstract: Advancements in high-computing devices increase the necessity for improved and new understanding and development of smart manufacturing factories. Discrete-event models with simulators have been shown to be critical to architect, designing, building, and operating the manufacturing of semiconductor chips. The diffusion, implantation, and lithography machines have intricate processes due to their feedforward and feedback connectivity. The dataset collected from simulations of the factory models holds the promise of generating valuable machine-learning models. As surrogate data-based models, their executions are highly efficient compared to the physics-based counterpart models. For the development of surrogate models, it is beneficial to have publicly available benchmark simulation models that are grounded in factory models that have concise structures and accurate behaviors. Hence, in this research, a dataset is devised and constructed based on a benchmark model of an Intel semiconductor fabrication factory. The model is formalized using the Parallel Discrete-Event System Specification and executed using the DEVS-Suite simulator. The time series dataset is constructed using discrete-event time trajectories. This dataset is further analyzed and used to develop baseline univariate and multivariate machine learning models. The dataset can also be utilized in the machine learning community for behavioral analysis based on formalized and scalable component-based discrete-event models and simulations.

Title: Narrowing the Focus: Learned Optimizers for Pretrained Models

• Authors: Gus Kristiansen, Mark Sandler, Andrey Zhmoginov, Nolan Miller, Anirudh Goyal, Jihwan Lee, Max Vladymyrov
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09310
• Pdf URL: https://arxiv.org/pdf/2408.09310
• Copy Paste: [[2408.09310]] Narrowing the Focus: Learned Optimizers for Pretrained Models(https://arxiv.org/abs/2408.09310)
• Keywords: robust
• Abstract: In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand-designed and tuning their hyperparameters is a big part of the training process. Learned optimizers have shown some initial promise, but are generally unsuccessful as a general optimization mechanism applicable to every problem. In this work we explore a different direction: instead of learning general optimizers, we instead specialize them to a specific training environment. We propose a novel optimizer technique that learns a layer-specific linear combination of update directions provided by a set of base optimizers, effectively adapting its strategy to the specific model and dataset. When evaluated on image classification tasks, this specialized optimizer significantly outperforms both traditional off-the-shelf methods such as Adam, as well as existing general learned optimizers. Moreover, it demonstrates robust generalization with respect to model initialization, evaluating on unseen datasets, and training durations beyond its meta-training horizon.

Title: Learning Fair Invariant Representations under Covariate and Correlation Shifts Simultaneously

• Authors: Dong Li, Chen Zhao, Minglai Shao, Wenjun Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09312
• Pdf URL: https://arxiv.org/pdf/2408.09312
• Copy Paste: [[2408.09312]] Learning Fair Invariant Representations under Covariate and Correlation Shifts Simultaneously(https://arxiv.org/abs/2408.09312)
• Keywords: fair
• Abstract: Achieving the generalization of an invariant classifier from training domains to shifted test domains while simultaneously considering model fairness is a substantial and complex challenge in machine learning. Existing methods address the problem of fairness-aware domain generalization, focusing on either covariate shift or correlation shift, but rarely consider both at the same time. In this paper, we introduce a novel approach that focuses on learning a fairness-aware domain-invariant predictor within a framework addressing both covariate and correlation shifts simultaneously, ensuring its generalization to unknown test domains inaccessible during training. In our approach, data are first disentangled into content and style factors in latent spaces. Furthermore, fairness-aware domain-invariant content representations can be learned by mitigating sensitive information and retaining as much other information as possible. Extensive empirical studies on benchmark datasets demonstrate that our approach surpasses state-of-the-art methods with respect to model accuracy as well as both group and individual fairness.

Title: Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks

• Authors: Kexin Chen, Yi Liu, Dongxia Wang, Jiaying Chen, Wenhai Wang
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.09326
• Pdf URL: https://arxiv.org/pdf/2408.09326
• Copy Paste: [[2408.09326]] Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks(https://arxiv.org/abs/2408.09326)
• Keywords: security, attack, robust, large language model
• Abstract: Large Language Models (LLMs) have increasingly become pivotal in content generation with notable societal impact. These models hold the potential to generate content that could be deemed harmful.Efforts to mitigate this risk include implementing safeguards to ensure LLMs adhere to social ethics.However, despite such measures, the phenomenon of "jailbreaking" -- where carefully crafted prompts elicit harmful responses from models -- persists as a significant challenge. Recognizing the continuous threat posed by jailbreaking tactics and their repercussions for the trustworthy use of LLMs, a rigorous assessment of the models' robustness against such attacks is essential. This study introduces an comprehensive evaluation framework and conducts an large-scale empirical experiment to address this need. We concentrate on 10 cutting-edge jailbreak strategies across three categories, 1525 questions from 61 specific harmful categories, and 13 popular LLMs. We adopt multi-dimensional metrics such as Attack Success Rate (ASR), Toxicity Score, Fluency, Token Length, and Grammatical Errors to thoroughly assess the LLMs' outputs under jailbreak. By normalizing and aggregating these metrics, we present a detailed reliability score for different LLMs, coupled with strategic recommendations to reduce their susceptibility to such vulnerabilities. Additionally, we explore the relationships among the models, attack strategies, and types of harmful content, as well as the correlations between the evaluation metrics, which proves the validity of our multifaceted evaluation framework. Our extensive experimental results demonstrate a lack of resilience among all tested LLMs against certain strategies, and highlight the need to concentrate on the reliability facets of LLMs. We believe our study can provide valuable insights into enhancing the security evaluation of LLMs against jailbreak within the domain.

Title: Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs

• Authors: Jiancheng Dong, Lei Jiang, Wei Jin, Lu Cheng
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09327
• Pdf URL: https://arxiv.org/pdf/2408.09327
• Copy Paste: [[2408.09327]] Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs(https://arxiv.org/abs/2408.09327)
• Keywords: transformer
• Abstract: Packing for Supervised Fine-Tuning (SFT) in autoregressive models involves concatenating data points of varying lengths until reaching the designed maximum length to facilitate GPU processing. However, randomly concatenating data points and feeding them into an autoregressive transformer can lead to cross-contamination of sequences due to the significant difference in their subject matter. The mainstream approaches in SFT ensure that each token in the attention calculation phase only focuses on tokens within its own short sequence, without providing additional learning signals for the preceding context. To address these challenges, we introduce Threshold Filtering Packing (TFP), a method that selects samples with related context while maintaining sufficient diversity within the same pack. Our experiments show that TFP offers a simple-to-implement and scalable approach that significantly enhances SFT performance, with observed improvements of up to 7\% on GSM8K, 4\% on HumanEval, and 15\% on the adult-census-income dataset.

Title: Fostering Natural Conversation in Large Language Models with NICO: a Natural Interactive COnversation dataset

• Authors: Renliang Sun, Mengyuan Liu, Shiping Yang, Rui Wang, Junqing He, Jiaxing Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09330
• Pdf URL: https://arxiv.org/pdf/2408.09330
• Copy Paste: [[2408.09330]] Fostering Natural Conversation in Large Language Models with NICO: a Natural Interactive COnversation dataset(https://arxiv.org/abs/2408.09330)
• Keywords: large language model
• Abstract: Benefiting from diverse instruction datasets, contemporary Large Language Models (LLMs) perform effectively as AI assistants in collaborating with humans. However, LLMs still struggle to generate natural and colloquial responses in real-world applications such as chatbots and psychological counseling that require more human-like interactions. To address these limitations, we introduce NICO, a Natural Interactive COnversation dataset in Chinese. We first use GPT-4-turbo to generate dialogue drafts and make them cover 20 daily-life topics and 5 types of social interactions. Then, we hire workers to revise these dialogues to ensure that they are free of grammatical errors and unnatural utterances. We define two dialogue-level natural conversation tasks and two sentence-level tasks for identifying and rewriting unnatural sentences. Multiple open-source and closed-source LLMs are tested and analyzed in detail. The experimental results highlight the challenge of the tasks and demonstrate how NICO can help foster the natural dialogue capabilities of LLMs. The dataset will be released.

Title: SkyScript-100M: 1,000,000,000 Pairs of Scripts and Shooting Scripts for Short Drama

• Authors: Jing Tang, Quanlu Jia, Yuqiang Xie, Zeyu Gong, Xiang Wen, Jiayi Zhang, Yalong Guo, Guibin Chen, Jiangping Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09333
• Pdf URL: https://arxiv.org/pdf/2408.09333
• Copy Paste: [[2408.09333]] SkyScript-100M: 1,000,000,000 Pairs of Scripts and Shooting Scripts for Short Drama(https://arxiv.org/abs/2408.09333)
• Keywords: extraction
• Abstract: Generating high-quality shooting scripts containing information such as scene and shot language is essential for short drama script generation. We collect 6,660 popular short drama episodes from the Internet, each with an average of 100 short episodes, and the total number of short episodes is about 80,000, with a total duration of about 2,000 hours and totaling 10 terabytes (TB). We perform keyframe extraction and annotation on each episode to obtain about 10,000,000 shooting scripts. We perform 100 script restorations on the extracted shooting scripts based on our self-developed large short drama generation model SkyReels. This leads to a dataset containing 1,000,000,000 pairs of scripts and shooting scripts for short dramas, called SkyScript-100M. We compare SkyScript-100M with the existing dataset in detail and demonstrate some deeper insights that can be achieved based on SkyScript-100M. Based on SkyScript-100M, researchers can achieve several deeper and more far-reaching script optimization goals, which may drive a paradigm shift in the entire field of text-to-video and significantly advance the field of short drama video generation. The data and code are available at this https URL.

Title: Elite360M: Efficient 360 Multi-task Learning via Bi-projection Fusion and Cross-task Collaboration

• Authors: Hao Ai, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09336
• Pdf URL: https://arxiv.org/pdf/2408.09336
• Copy Paste: [[2408.09336]] Elite360M: Efficient 360 Multi-task Learning via Bi-projection Fusion and Cross-task Collaboration(https://arxiv.org/abs/2408.09336)
• Keywords: segmentation
• Abstract: 360 cameras capture the entire surrounding environment with a large FoV, exhibiting comprehensive visual information to directly infer the 3D structures, e.g., depth and surface normal, and semantic information simultaneously. Existing works predominantly specialize in a single task, leaving multi-task learning of 3D geometry and semantics largely unexplored. Achieving such an objective is, however, challenging due to: 1) inherent spherical distortion of planar equirectangular projection (ERP) and insufficient global perception induced by 360 image's ultra-wide FoV; 2) non-trivial progress in effectively merging geometry and semantics among different tasks to achieve mutual benefits. In this paper, we propose a novel end-to-end multi-task learning framework, named Elite360M, capable of inferring 3D structures via depth and surface normal estimation, and semantics via semantic segmentation simultaneously. Our key idea is to build a representation with strong global perception and less distortion while exploring the inter- and cross-task relationships between geometry and semantics. We incorporate the distortion-free and spatially continuous icosahedron projection (ICOSAP) points and combine them with ERP to enhance global perception. With a negligible cost, a Bi-projection Bi-attention Fusion module is thus designed to capture the semantic- and distance-aware dependencies between each pixel of the region-aware ERP feature and the ICOSAP point feature set. Moreover, we propose a novel Cross-task Collaboration module to explicitly extract task-specific geometric and semantic information from the learned representation to achieve preliminary predictions. It then integrates the spatial contextual information among tasks to realize cross-task fusion. Extensive experiments demonstrate the effectiveness and efficacy of Elite360M.

Title: Improvement of Bayesian PINN Training Convergence in Solving Multi-scale PDEs with Noise

• Authors: Yilong Hou, Xi'an Li, Jinran Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09340
• Pdf URL: https://arxiv.org/pdf/2408.09340
• Copy Paste: [[2408.09340]] Improvement of Bayesian PINN Training Convergence in Solving Multi-scale PDEs with Noise(https://arxiv.org/abs/2408.09340)
• Keywords: robust
• Abstract: Bayesian Physics Informed Neural Networks (BPINN) have received considerable attention for inferring differential equations' system states and physical parameters according to noisy observations. However, in practice, Hamiltonian Monte Carlo (HMC) used to estimate the internal parameters of BPINN often encounters troubles, including poor performance and awful convergence for a given step size used to adjust the momentum of those parameters. To improve the efficacy of HMC convergence for the BPINN method and extend its application scope to multi-scale partial differential equations (PDE), we developed a robust multi-scale Bayesian PINN (dubbed MBPINN) method by integrating multi-scale deep neural networks (MscaleDNN) and Bayesian inference. In this newly proposed MBPINN method, we reframe HMC with Stochastic Gradient Descent (SGD) to ensure the most ``likely'' estimation is always provided, and we configure its solver as a Fourier feature mapping-induced MscaleDNN. The MBPINN method offers several key advantages: (1) it is more robust than HMC, (2) it incurs less computational cost than HMC, and (3) it is more flexible for complex problems. We demonstrate the applicability and performance of the proposed method through general Poisson and multi-scale elliptic problems in one- to three-dimensional spaces. Our findings indicate that the proposed method can avoid HMC failures and provide valid results. Additionally, our method can handle complex PDE and produce comparable results for general PDE. These findings suggest that our proposed approach has excellent potential for physics-informed machine learning for parameter estimation and solution recovery in the case of ill-posed problems.

Title: Hyperstroke: A Novel High-quality Stroke Representation for Assistive Artistic Drawing

• Authors: Haoyun Qin, Jian Lin, Hanyuan Liu, Xueting Liu, Chengze Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09348
• Pdf URL: https://arxiv.org/pdf/2408.09348
• Copy Paste: [[2408.09348]] Hyperstroke: A Novel High-quality Stroke Representation for Assistive Artistic Drawing(https://arxiv.org/abs/2408.09348)
• Keywords: transformer
• Abstract: Assistive drawing aims to facilitate the creative process by providing intelligent guidance to artists. Existing solutions often fail to effectively model intricate stroke details or adequately address the temporal aspects of drawing. We introduce hyperstroke, a novel stroke representation designed to capture precise fine stroke details, including RGB appearance and alpha-channel opacity. Using a Vector Quantization approach, hyperstroke learns compact tokenized representations of strokes from real-life drawing videos of artistic drawing. With hyperstroke, we propose to model assistive drawing via a transformer-based architecture, to enable intuitive and user-friendly drawing applications, which are experimented in our exploratory evaluation.

Title: Panorama Tomosynthesis from Head CBCT with Simulated Projection Geometry

• Authors: Anusree P.S., Bikram Keshari Parida, Seong Yong Moon, Wonsang You
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09358
• Pdf URL: https://arxiv.org/pdf/2408.09358
• Copy Paste: [[2408.09358]] Panorama Tomosynthesis from Head CBCT with Simulated Projection Geometry(https://arxiv.org/abs/2408.09358)
• Keywords: extraction
• Abstract: Cone Beam Computed Tomography (CBCT) and Panoramic X-rays are the most commonly used imaging modalities in dental health care. CBCT can produce three-dimensional views of a patient's head, providing clinicians with better diagnostic capability, whereas Panoramic X-ray can capture the entire maxillofacial region in a single image. If the CBCT is already available, it can be beneficial to synthesize a Panoramic X-ray, thereby avoiding an immediate additional scan and extra radiation exposure. Existing methods focus on delineating an approximate dental arch and creating orthogonal projections along this arch. However, no golden standard is available for such dental arch extractions, and this choice can affect the quality of synthesized X-rays. To avoid such issues, we propose a novel method for synthesizing Panoramic X-rays from diverse head CBCTs, employing a simulated projection geometry and dynamic rotation centers. Our method effectively synthesized panoramic views from CBCT, even for patients with missing or nonexistent teeth and in the presence of severe metal implants. Our results demonstrate that this method can generate high-quality panoramic images irrespective of the CBCT scanner geometry.

Title: Angle of Arrival Estimation with Transformer: A Sparse and Gridless Method with Zero-Shot Capability

• Authors: Zhaoxuan Zhu, Chulong Chen, Bo Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09362
• Pdf URL: https://arxiv.org/pdf/2408.09362
• Copy Paste: [[2408.09362]] Angle of Arrival Estimation with Transformer: A Sparse and Gridless Method with Zero-Shot Capability(https://arxiv.org/abs/2408.09362)
• Keywords: transformer
• Abstract: Automotive Multiple-Input Multiple-Output (MIMO) radars have gained significant traction in Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles (AV) due to their cost-effectiveness, resilience to challenging operating conditions, and extended detection range. To fully leverage the advantages of MIMO radars, it is crucial to develop an Angle of Arrival (AOA) algorithm that delivers high performance with reasonable computational workload. This work introduces AAETR (Angle of Arrival Estimation with TRansformer) for high performance gridless AOA estimation. Comprehensive evaluations across various signal-to-noise ratios (SNRs) and multi-target scenarios demonstrate AAETR's superior performance compared to super resolution AOA algorithms such as Iterative Adaptive Approach (IAA). The proposed architecture features efficient, scalable, sparse and gridless angle-finding capability, overcoming the issues of high computational cost and straddling loss in SNR associated with grid-based IAA. AAETR requires fewer tunable hyper-parameters and is end-to-end trainable in a deep learning radar perception pipeline. When trained on large-scale simulated datasets then evaluated on real dataset, AAETR exhibits remarkable zero-shot sim-to-real transferability and emergent sidelobe suppression capability. This highlights the effectiveness of the proposed approach and its potential as a drop-in module in practical systems.

Title: Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communities

• Authors: Minh Duc Chu, Zihao He, Rebecca Dorn, Kristina Lerman
• Subjects: cs.CL, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2408.09366
• Pdf URL: https://arxiv.org/pdf/2408.09366
• Copy Paste: [[2408.09366]] Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communities(https://arxiv.org/abs/2408.09366)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have shown promise in representing individuals and communities, offering new ways to study complex social dynamics. However, effectively aligning LLMs with specific human groups and systematically assessing the fidelity of the alignment remains a challenge. This paper presents a robust framework for aligning LLMs with online communities via instruction-tuning and comprehensively evaluating alignment across various aspects of language, including authenticity, emotional tone, toxicity, and harm. We demonstrate the utility of our approach by applying it to online communities centered on dieting and body image. We administer an eating disorder psychometric test to the aligned LLMs to reveal unhealthy beliefs and successfully differentiate communities with varying levels of eating disorder risk. Our results highlight the potential of LLMs in automated moderation and broader applications in public health and social science research.

Title: Detecting the Undetectable: Combining Kolmogorov-Arnold Networks and MLP for AI-Generated Image Detection

• Authors: Taharim Rahman Anon, Jakaria Islam Emon
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09371
• Pdf URL: https://arxiv.org/pdf/2408.09371
• Copy Paste: [[2408.09371]] Detecting the Undetectable: Combining Kolmogorov-Arnold Networks and MLP for AI-Generated Image Detection(https://arxiv.org/abs/2408.09371)
• Keywords: robust, diffusion, generative
• Abstract: As artificial intelligence progresses, the task of distinguishing between real and AI-generated images is increasingly complicated by sophisticated generative models. This paper presents a novel detection framework adept at robustly identifying images produced by cutting-edge generative AI models, such as DALL-E 3, MidJourney, and Stable Diffusion 3. We introduce a comprehensive dataset, tailored to include images from these advanced generators, which serves as the foundation for extensive evaluation. we propose a classification system that integrates semantic image embeddings with a traditional Multilayer Perceptron (MLP). This baseline system is designed to effectively differentiate between real and AI-generated images under various challenging conditions. Enhancing this approach, we introduce a hybrid architecture that combines Kolmogorov-Arnold Networks (KAN) with the MLP. This hybrid model leverages the adaptive, high-resolution feature transformation capabilities of KAN, enabling our system to capture and analyze complex patterns in AI-generated images that are typically overlooked by conventional models. In out-of-distribution testing, our proposed model consistently outperformed the standard MLP across three out of distribution test datasets, demonstrating superior performance and robustness in classifying real images from AI-generated images with impressive F1 scores.

Title: FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model

• Authors: Ziyu Yao, Xuxin Cheng, Zhiqi Huang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.09384
• Pdf URL: https://arxiv.org/pdf/2408.09384
• Copy Paste: [[2408.09384]] FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model(https://arxiv.org/abs/2408.09384)
• Keywords: diffusion, transformer, generative
• Abstract: Talking head generation is a significant research topic that still faces numerous challenges. Previous works often adopt generative adversarial networks or regression models, which are plagued by generation quality and average facial shape problem. Although diffusion models show impressive generative ability, their exploration in talking head generation remains unsatisfactory. This is because they either solely use the diffusion model to obtain an intermediate representation and then employ another pre-trained renderer, or they overlook the feature decoupling of complex facial details, such as expressions, head poses and appearance textures. Therefore, we propose a Facial Decoupled Diffusion model for Talking head generation called FD2Talk, which fully leverages the advantages of diffusion models and decouples the complex facial details through multi-stages. Specifically, we separate facial details into motion and appearance. In the initial phase, we design the Diffusion Transformer to accurately predict motion coefficients from raw audio. These motions are highly decoupled from appearance, making them easier for the network to learn compared to high-dimensional RGB images. Subsequently, in the second phase, we encode the reference image to capture appearance textures. The predicted facial and head motions and encoded appearance then serve as the conditions for the Diffusion UNet, guiding the frame generation. Benefiting from decoupling facial details and fully leveraging diffusion models, extensive experiments substantiate that our approach excels in enhancing image quality and generating more accurate and diverse results compared to previous state-of-the-art methods.

Title: Offline RLHF Methods Need More Accurate Supervision Signals

• Authors: Shiqi Wang, Zhengze Zhang, Rui Zhao, Fei Tan, Cam Tu Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09385
• Pdf URL: https://arxiv.org/pdf/2408.09385
• Copy Paste: [[2408.09385]] Offline RLHF Methods Need More Accurate Supervision Signals(https://arxiv.org/abs/2408.09385)
• Keywords: large language model
• Abstract: With the rapid advances in Large Language Models (LLMs), aligning LLMs with human preferences become increasingly important. Although Reinforcement Learning with Human Feedback (RLHF) proves effective, it is complicated and highly resource-intensive. As such, offline RLHF has been introduced as an alternative solution, which directly optimizes LLMs with ranking losses on a fixed preference dataset. Current offline RLHF only captures the ``ordinal relationship'' between responses, overlooking the crucial aspect of ``how much'' one is preferred over the others. To address this issue, we propose a simple yet effective solution called \textbf{R}eward \textbf{D}ifference \textbf{O}ptimization, shorted as \textbf{RDO}. Specifically, we introduce {\it reward difference coefficients} to reweigh sample pairs in offline RLHF. We then develop a {\it difference model} involving rich interactions between a pair of responses for predicting these difference coefficients. Experiments with 7B LLMs on the HH and TL;DR datasets substantiate the effectiveness of our method in both automatic metrics and human evaluation, thereby highlighting its potential for aligning LLMs with human intent and values.

Title: Federated Graph Learning with Structure Proxy Alignment

• Authors: Xingbo Fu, Zihan Chen, Binchi Zhang, Chen Chen, Jundong Li
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.09393
• Pdf URL: https://arxiv.org/pdf/2408.09393
• Copy Paste: [[2408.09393]] Federated Graph Learning with Structure Proxy Alignment(https://arxiv.org/abs/2408.09393)
• Keywords: federate
• Abstract: Federated Graph Learning (FGL) aims to learn graph learning models over graph data distributed in multiple data owners, which has been applied in various applications such as social recommendation and financial fraud detection. Inherited from generic Federated Learning (FL), FGL similarly has the data heterogeneity issue where the label distribution may vary significantly for distributed graph data across clients. For instance, a client can have the majority of nodes from a class, while another client may have only a few nodes from the same class. This issue results in divergent local objectives and impairs FGL convergence for node-level tasks, especially for node classification. Moreover, FGL also encounters a unique challenge for the node classification task: the nodes from a minority class in a client are more likely to have biased neighboring information, which prevents FGL from learning expressive node embeddings with Graph Neural Networks (GNNs). To grapple with the challenge, we propose FedSpray, a novel FGL framework that learns local class-wise structure proxies in the latent space and aligns them to obtain global structure proxies in the server. Our goal is to obtain the aligned structure proxies that can serve as reliable, unbiased neighboring information for node classification. To achieve this, FedSpray trains a global feature-structure encoder and generates unbiased soft targets with structure proxies to regularize local training of GNN models in a personalized way. We conduct extensive experiments over four datasets, and experiment results validate the superiority of FedSpray compared with other baselines. Our code is available at this https URL.

Title: OU-CoViT: Copula-Enhanced Bi-Channel Multi-Task Vision Transformers with Dual Adaptation for OU-UWF Images

• Authors: Yang Li, Jianing Deng, Chong Zhong, Danjuan Yang, Meiyan Li, A.H. Welsh, Aiyi Liu, Xingtao Zhou, Catherine C. Liu, Bo Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09395
• Pdf URL: https://arxiv.org/pdf/2408.09395
• Copy Paste: [[2408.09395]] OU-CoViT: Copula-Enhanced Bi-Channel Multi-Task Vision Transformers with Dual Adaptation for OU-UWF Images(https://arxiv.org/abs/2408.09395)
• Keywords: transformer
• Abstract: Myopia screening using cutting-edge ultra-widefield (UWF) fundus imaging and joint modeling of multiple discrete and continuous clinical scores presents a promising new paradigm for multi-task problems in Ophthalmology. The bi-channel framework that arises from the Ophthalmic phenomenon of ``interocular asymmetries'' of both eyes (OU) calls for new employment on the SOTA transformer-based models. However, the application of copula models for multiple mixed discrete-continuous labels on deep learning (DL) is challenging. Moreover, the application of advanced large transformer-based models to small medical datasets is challenging due to overfitting and computational resource constraints. To resolve these challenges, we propose OU-CoViT: a novel Copula-Enhanced Bi-Channel Multi-Task Vision Transformers with Dual Adaptation for OU-UWF images, which can i) incorporate conditional correlation information across multiple discrete and continuous labels within a deep learning framework (by deriving the closed form of a novel Copula Loss); ii) take OU inputs subject to both high correlation and interocular asymmetries using a bi-channel model with dual adaptation; and iii) enable the adaptation of large vision transformer (ViT) models to small medical datasets. Solid experiments demonstrate that OU-CoViT significantly improves prediction performance compared to single-channel baseline models with empirical loss. Furthermore, the novel architecture of OU-CoViT allows generalizability and extensions of our dual adaptation and Copula Loss to various ViT variants and large DL models on small medical datasets. Our approach opens up new possibilities for joint modeling of heterogeneous multi-channel input and mixed discrete-continuous clinical scores in medical practices and has the potential to advance AI-assisted clinical decision-making in various medical domains beyond Ophthalmology.

Title: Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony

• Authors: Chao Xu, Mingze Sun, Zhi-Qi Cheng, Fei Wang, Yang Liu, Baigui Sun, Ruqi Huang, Alexander Hauptmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09397
• Pdf URL: https://arxiv.org/pdf/2408.09397
• Copy Paste: [[2408.09397]] Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony(https://arxiv.org/abs/2408.09397)
• Keywords: transformer, generative
• Abstract: In this paper, we propose a novel framework, Combo, for harmonious co-speech holistic 3D human motion generation and efficient customizable adaption. In particular, we identify that one fundamental challenge as the multiple-input-multiple-output (MIMO) nature of the generative model of interest. More concretely, on the input end, the model typically consumes both speech signals and character guidance (e.g., identity and emotion), which not only poses challenge on learning capacity but also hinders further adaptation to varying guidance; on the output end, holistic human motions mainly consist of facial expressions and body movements, which are inherently correlated but non-trivial to coordinate in current data-driven generation process. In response to the above challenge, we propose tailored designs to both ends. For the former, we propose to pre-train on data regarding a fixed identity with neutral emotion, and defer the incorporation of customizable conditions (identity and emotion) to fine-tuning stage, which is boosted by our novel X-Adapter for parameter-efficient fine-tuning. For the latter, we propose a simple yet effective transformer design, DU-Trans, which first divides into two branches to learn individual features of face expression and body movements, and then unites those to learn a joint bi-directional distribution and directly predicts combined coefficients. Evaluated on BEAT2 and SHOW datasets, Combo is highly effective in generating high-quality motions but also efficient in transferring identity and emotion. Project website: \href{this https URL}{Combo}.

Title: VrdONE: One-stage Video Visual Relation Detection

• Authors: Xinjie Jiang, Chenxi Zheng, Xuemiao Xu, Bangzhen Liu, Weiying Zheng, Huaidong Zhang, Shengfeng He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09408
• Pdf URL: https://arxiv.org/pdf/2408.09408
• Copy Paste: [[2408.09408]] VrdONE: One-stage Video Visual Relation Detection(https://arxiv.org/abs/2408.09408)
• Keywords: segmentation
• Abstract: Video Visual Relation Detection (VidVRD) focuses on understanding how entities interact over time and space in videos, a key step for gaining deeper insights into video scenes beyond basic visual tasks. Traditional methods for VidVRD, challenged by its complexity, typically split the task into two parts: one for identifying what relation categories are present and another for determining their temporal boundaries. This split overlooks the inherent connection between these elements. Addressing the need to recognize entity pairs' spatiotemporal interactions across a range of durations, we propose VrdONE, a streamlined yet efficacious one-stage model. VrdONE combines the features of subjects and objects, turning predicate detection into 1D instance segmentation on their combined representations. This setup allows for both relation category identification and binary mask generation in one go, eliminating the need for extra steps like proposal generation or post-processing. VrdONE facilitates the interaction of features across various frames, adeptly capturing both short-lived and enduring relations. Additionally, we introduce the Subject-Object Synergy (SOS) module, enhancing how subjects and objects perceive each other before combining. VrdONE achieves state-of-the-art performances on the VidOR benchmark and ImageNet-VidVRD, showcasing its superior capability in discerning relations across different temporal scales. The code is available at \textcolor[RGB]{228,58,136}{\href{this https URL}{this https URL}}.

Title: Weakly Supervised Lymph Nodes Segmentation Based on Partial Instance Annotations with Pre-trained Dual-branch Network and Pseudo Label Learning

• Authors: Litingyu Wang (1), Yijie Qu (1), Xiangde Luo (1 and 2), Wenjun Liao (1 and 3), Shichuan Zhang (1 and 3), Guotai Wang (1 and 2) ((1) University of Electronic Science and Technology of China, Chengdu, China, (2) ShangAI Laboratory, Shanghai, China, (3) Department of Radiation Oncology, Sichuan Cancer Hospital & Institute, Sichuan Cancer Center, Chengdu, China)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09411
• Pdf URL: https://arxiv.org/pdf/2408.09411
• Copy Paste: [[2408.09411]] Weakly Supervised Lymph Nodes Segmentation Based on Partial Instance Annotations with Pre-trained Dual-branch Network and Pseudo Label Learning(https://arxiv.org/abs/2408.09411)
• Keywords: extraction, segmentation
• Abstract: Assessing the presence of potentially malignant lymph nodes aids in estimating cancer progression, and identifying surrounding benign lymph nodes can assist in determining potential metastatic pathways for cancer. For quantitative analysis, automatic segmentation of lymph nodes is crucial. However, due to the labor-intensive and time-consuming manual annotation process required for a large number of lymph nodes, it is more practical to annotate only a subset of the lymph node instances to reduce annotation costs. In this study, we propose a pre-trained Dual-Branch network with Dynamically Mixed Pseudo label (DBDMP) to learn from partial instance annotations for lymph nodes segmentation. To obtain reliable pseudo labels for lymph nodes that are not annotated, we employ a dual-decoder network to generate different outputs that are then dynamically mixed. We integrate the original weak partial annotations with the mixed pseudo labels to supervise the network. To further leverage the extensive amount of unannotated voxels, we apply a self-supervised pre-training strategy to enhance the model's feature extraction capability. Experiments on the mediastinal Lymph Node Quantification (LNQ) dataset demonstrate that our method, compared to directly learning from partial instance annotations, significantly improves the Dice Similarity Coefficient (DSC) from 11.04% to 54.10% and reduces the Average Symmetric Surface Distance (ASSD) from 20.83 $mm$ to 8.72 $mm$. The code is available at this https URL

Title: Clustering and Alignment: Understanding the Training Dynamics in Modular Addition

• Authors: Tiberiu Musat
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09414
• Pdf URL: https://arxiv.org/pdf/2408.09414
• Copy Paste: [[2408.09414]] Clustering and Alignment: Understanding the Training Dynamics in Modular Addition(https://arxiv.org/abs/2408.09414)
• Keywords: transformer
• Abstract: Recent studies have revealed that neural networks learn interpretable algorithms for many simple problems. However, little is known about how these algorithms emerge during training. In this article, we study the training dynamics of a simplified transformer with 2-dimensional embeddings on the problem of modular addition. We observe that embedding vectors tend to organize into two types of structures: grids and circles. We study these structures and explain their emergence as a result of two simple tendencies exhibited by pairs of embeddings: clustering and alignment. We propose explicit formulae for these tendencies as interaction forces between different pairs of embeddings. To show that our formulae can fully account for the emergence of these structures, we construct an equivalent particle simulation where we find that identical structures emerge. We use our insights to discuss the role of weight decay and reveal a new mechanism that links regularization and training dynamics. We also release an interactive demo to support our findings: https://modular-addition.vercel.app/.

Title: Challenges and Responses in the Practice of Large Language Models

• Authors: Hongyin Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09416
• Pdf URL: https://arxiv.org/pdf/2408.09416
• Copy Paste: [[2408.09416]] Challenges and Responses in the Practice of Large Language Models(https://arxiv.org/abs/2408.09416)
• Keywords: large language model
• Abstract: This paper carefully summarizes extensive and profound questions from all walks of life, focusing on the current high-profile AI field, covering multiple dimensions such as industry trends, academic research, technological innovation and business applications. This paper meticulously curates questions that are both thought-provoking and practically relevant, providing nuanced and insightful answers to each. To facilitate readers' understanding and reference, this paper specifically classifies and organizes these questions systematically and meticulously from the five core dimensions of computing power infrastructure, software architecture, data resources, application scenarios, and brain science. This work aims to provide readers with a comprehensive, in-depth and cutting-edge AI knowledge framework to help people from all walks of life grasp the pulse of AI development, stimulate innovative thinking, and promote industrial progress.

Title: Distinguish Confusion in Legal Judgment Prediction via Revised Relation Knowledge

• Authors: Nuo Xu, Pinghui Wang, Junzhou Zhao, Feiyang Sun, Lin Lan, Jing Tao, Li Pan, Xiaohong Guan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09422
• Pdf URL: https://arxiv.org/pdf/2408.09422
• Copy Paste: [[2408.09422]] Distinguish Confusion in Legal Judgment Prediction via Revised Relation Knowledge(https://arxiv.org/abs/2408.09422)
• Keywords: robust
• Abstract: Legal Judgment Prediction (LJP) aims to automatically predict a law case's judgment results based on the text description of its facts. In practice, the confusing law articles (or charges) problem frequently occurs, reflecting that the law cases applicable to similar articles (or charges) tend to be misjudged. Although some recent works based on prior knowledge solve this issue well, they ignore that confusion also occurs between law articles with a high posterior semantic similarity due to the data imbalance problem instead of only between the prior highly similar ones, which is this work's further finding. This paper proposes an end-to-end model named \textit{D-LADAN} to solve the above challenges. On the one hand, D-LADAN constructs a graph among law articles based on their text definition and proposes a graph distillation operation (GDO) to distinguish the ones with a high prior semantic similarity. On the other hand, D-LADAN presents a novel momentum-updated memory mechanism to dynamically sense the posterior similarity between law articles (or charges) and a weighted GDO to adaptively capture the distinctions for revising the inductive bias caused by the data imbalance problem. We perform extensive experiments to demonstrate that D-LADAN significantly outperforms state-of-the-art methods in accuracy and robustness.

Title: OVOSE: Open-Vocabulary Semantic Segmentation in Event-Based Cameras

• Authors: Muhammad Rameez Ur Rahman, Jhony H. Giraldo, Indro Spinelli, Stéphane Lathuilière, Fabio Galasso
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09424
• Pdf URL: https://arxiv.org/pdf/2408.09424
• Copy Paste: [[2408.09424]] OVOSE: Open-Vocabulary Semantic Segmentation in Event-Based Cameras(https://arxiv.org/abs/2408.09424)
• Keywords: segmentation
• Abstract: Event cameras, known for low-latency operation and superior performance in challenging lighting conditions, are suitable for sensitive computer vision tasks such as semantic segmentation in autonomous driving. However, challenges arise due to limited event-based data and the absence of large-scale segmentation benchmarks. Current works are confined to closed-set semantic segmentation, limiting their adaptability to other applications. In this paper, we introduce OVOSE, the first Open-Vocabulary Semantic Segmentation algorithm for Event cameras. OVOSE leverages synthetic event data and knowledge distillation from a pre-trained image-based foundation model to an event-based counterpart, effectively preserving spatial context and transferring open-vocabulary semantic segmentation capabilities. We evaluate the performance of OVOSE on two driving semantic segmentation datasets DDD17, and DSEC-Semantic, comparing it with existing conventional image open-vocabulary models adapted for event-based data. Similarly, we compare OVOSE with state-of-the-art methods designed for closed-set settings in unsupervised domain adaptation for event-based semantic segmentation. OVOSE demonstrates superior performance, showcasing its potential for real-world applications. The code is available at this https URL.

Title: A Robust Algorithm for Contactless Fingerprint Enhancement and Matching

• Authors: Mahrukh Siddiqui, Shahzaib Iqbal, Bandar AlShammari, Bandar Alhaqbani, Tariq M. Khan, Imran Razzak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09426
• Pdf URL: https://arxiv.org/pdf/2408.09426
• Copy Paste: [[2408.09426]] A Robust Algorithm for Contactless Fingerprint Enhancement and Matching(https://arxiv.org/abs/2408.09426)
• Keywords: robust, extraction
• Abstract: Compared to contact fingerprint images, contactless fingerprint images exhibit four distinct characteristics: (1) they contain less noise; (2) they have fewer discontinuities in ridge patterns; (3) the ridge-valley pattern is less distinct; and (4) they pose an interoperability problem, as they lack the elastic deformation caused by pressing the finger against the capture device. These properties present significant challenges for the enhancement of contactless fingerprint images. In this study, we propose a novel contactless fingerprint identification solution that enhances the accuracy of minutiae detection through improved frequency estimation and a new region-quality-based minutia extraction algorithm. In addition, we introduce an efficient and highly accurate minutiae-based encoding and matching algorithm. We validate the effectiveness of our approach through extensive experimental testing. Our method achieves a minimum Equal Error Rate (EER) of 2.84\% on the PolyU contactless fingerprint dataset, demonstrating its superior performance compared to existing state-of-the-art techniques. The proposed fingerprint identification method exhibits notable precision and resilience, proving to be an effective and feasible solution for contactless fingerprint-based identification systems.

Title: Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models

• Authors: Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, Xuming Hu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09429
• Pdf URL: https://arxiv.org/pdf/2408.09429
• Copy Paste: [[2408.09429]] Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models(https://arxiv.org/abs/2408.09429)
• Keywords: large language model
• Abstract: Hallucination issues persistently plagued current multimodal large language models (MLLMs). While existing research primarily focuses on object-level or attribute-level hallucinations, sidelining the more sophisticated relation hallucinations that necessitate advanced reasoning abilities from MLLMs. Besides, recent benchmarks regarding relation hallucinations lack in-depth evaluation and effective mitigation. Moreover, their datasets are typically derived from a systematic annotation process, which could introduce inherent biases due to the predefined process. To handle the aforementioned challenges, we introduce Reefknot, a comprehensive benchmark specifically targeting relation hallucinations, consisting of over 20,000 samples derived from real-world scenarios. Specifically, we first provide a systematic definition of relation hallucinations, integrating perspectives from perceptive and cognitive domains. Furthermore, we construct the relation-based corpus utilizing the representative scene graph dataset Visual Genome (VG), from which semantic triplets follow real-world distributions. Our comparative evaluation across three distinct tasks revealed a substantial shortcoming in the capabilities of current MLLMs to mitigate relation hallucinations. Finally, we advance a novel confidence-based mitigation strategy tailored to tackle the relation hallucinations problem. Across three datasets, including Reefknot, we observed an average reduction of 9.75% in the hallucination rate. We believe our paper sheds valuable insights into achieving trustworthy multimodal intelligence. Our dataset and code will be released upon paper acceptance.

Title: FASST: Fast LLM-based Simultaneous Speech Translation

• Authors: Siqi Ouyang, Xi Xu, Chinmay Dandekar, Lei Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09430
• Pdf URL: https://arxiv.org/pdf/2408.09430
• Copy Paste: [[2408.09430]] FASST: Fast LLM-based Simultaneous Speech Translation(https://arxiv.org/abs/2408.09430)
• Keywords: large language model
• Abstract: Simultaneous speech translation (SST) takes streaming speech input and generates text translation on the fly. Existing methods either have high latency due to recomputation of input representations, or fall behind of offline ST in translation quality. In this paper, we propose FASST, a fast large language model based method for streaming speech translation. We propose blockwise-causal speech encoding and consistency mask, so that streaming speech input can be encoded incrementally without recomputation. Furthermore, we develop a two-stage training strategy to optimize FASST for simultaneous inference. We evaluate FASST and multiple strong prior models on MuST-C dataset. Experiment results show that FASST achieves the best quality-latency trade-off. It outperforms the previous best model by an average of 1.5 BLEU under the same latency for English to Spanish translation.

Title: Adversarial Attacked Teacher for Unsupervised Domain Adaptive Object Detection

• Authors: Kaiwen Wang, Yinzhe Shen, Martin Lauer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09431
• Pdf URL: https://arxiv.org/pdf/2408.09431
• Copy Paste: [[2408.09431]] Adversarial Attacked Teacher for Unsupervised Domain Adaptive Object Detection(https://arxiv.org/abs/2408.09431)
• Keywords: attack, robust
• Abstract: Object detectors encounter challenges in handling domain shifts. Cutting-edge domain adaptive object detection methods use the teacher-student framework and domain adversarial learning to generate domain-invariant pseudo-labels for self-training. However, the pseudo-labels generated by the teacher model tend to be biased towards the majority class and often mistakenly include overconfident false positives and underconfident false negatives. We reveal that pseudo-labels vulnerable to adversarial attacks are more likely to be low-quality. To address this, we propose a simple yet effective framework named Adversarial Attacked Teacher (AAT) to improve the quality of pseudo-labels. Specifically, we apply adversarial attacks to the teacher model, prompting it to generate adversarial pseudo-labels to correct bias, suppress overconfidence, and encourage underconfident proposals. An adaptive pseudo-label regularization is introduced to emphasize the influence of pseudo-labels with high certainty and reduce the negative impacts of uncertain predictions. Moreover, robust minority objects verified by pseudo-label regularization are oversampled to minimize dataset imbalance without introducing false positives. Extensive experiments conducted on various datasets demonstrate that AAT achieves superior performance, reaching 52.6 mAP on Clipart1k, surpassing the previous state-of-the-art by 6.7%.

Title: HySem: A context length optimized LLM pipeline for unstructured tabular extraction

• Authors: Narayanan PP, Anantharaman Palacode Narayana Iyer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09434
• Pdf URL: https://arxiv.org/pdf/2408.09434
• Copy Paste: [[2408.09434]] HySem: A context length optimized LLM pipeline for unstructured tabular extraction(https://arxiv.org/abs/2408.09434)
• Keywords: privacy, extraction, large language model
• Abstract: Regulatory compliance reporting in the pharmaceutical industry relies on detailed tables, but these are often under-utilized beyond compliance due to their unstructured format and arbitrary content. Extracting and semantically representing tabular data is challenging due to diverse table presentations. Large Language Models (LLMs) demonstrate substantial potential for semantic representation, yet they encounter challenges related to accuracy and context size limitations, which are crucial considerations for the industry applications. We introduce HySem, a pipeline that employs a novel context length optimization technique to generate accurate semantic JSON representations from HTML tables. This approach utilizes a custom fine-tuned model specifically designed for cost- and privacy-sensitive small and medium pharmaceutical enterprises. Running on commodity hardware and leveraging open-source models, our auto-correcting agents rectify both syntax and semantic errors in LLM-generated content. HySem surpasses its peer open-source models in accuracy and provides competitive performance when benchmarked against OpenAI GPT-4o and effectively addresses context length limitations, which is a crucial factor for supporting larger tables.

Title: Attention Is Not What You Need: Revisiting Multi-Instance Learning for Whole Slide Image Classification

• Authors: Xin Liu, Weijia Zhang, Min-Ling Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09449
• Pdf URL: https://arxiv.org/pdf/2408.09449
• Copy Paste: [[2408.09449]] Attention Is Not What You Need: Revisiting Multi-Instance Learning for Whole Slide Image Classification(https://arxiv.org/abs/2408.09449)
• Keywords: interpretability
• Abstract: Although attention-based multi-instance learning algorithms have achieved impressive performances on slide-level whole slide image (WSI) classification tasks, they are prone to mistakenly focus on irrelevant patterns such as staining conditions and tissue morphology, leading to incorrect patch-level predictions and unreliable interpretability. Moreover, these attention-based MIL algorithms tend to focus on salient instances and struggle to recognize hard-to-classify instances. In this paper, we first demonstrate that attention-based WSI classification methods do not adhere to the standard MIL assumptions. From the standard MIL assumptions, we propose a surprisingly simple yet effective instance-based MIL method for WSI classification (FocusMIL) based on max-pooling and forward amortized variational inference. We argue that synergizing the standard MIL assumption with variational inference encourages the model to focus on tumour morphology instead of spurious correlations. Our experimental evaluations show that FocusMIL significantly outperforms the baselines in patch-level classification tasks on the Camelyon16 and TCGA-NSCLC benchmarks. Visualization results show that our method also achieves better classification boundaries for identifying hard instances and mitigates the effect of spurious correlations between bags and labels.

Title: GraphSPNs: Sum-Product Networks Benefit From Canonical Orderings

• Authors: Milan Papež, Martin Rektoris, Václav Šmídl, Tomáš Pevný
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09451
• Pdf URL: https://arxiv.org/pdf/2408.09451
• Copy Paste: [[2408.09451]] GraphSPNs: Sum-Product Networks Benefit From Canonical Orderings(https://arxiv.org/abs/2408.09451)
• Keywords: generative
• Abstract: Deep generative models have recently made a remarkable progress in capturing complex probability distributions over graphs. However, they are intractable and thus unable to answer even the most basic probabilistic inference queries without resorting to approximations. Therefore, we propose graph sum-product networks (GraphSPNs), a tractable deep generative model which provides exact and efficient inference over (arbitrary parts of) graphs. We investigate different principles to make SPNs permutation invariant. We demonstrate that GraphSPNs are able to (conditionally) generate novel and chemically valid molecular graphs, being competitive to, and sometimes even better than, existing intractable models. We find out that (Graph)SPNs benefit from ensuring the permutation invariance via canonical ordering.

Title: Identifying Speakers and Addressees of Quotations in Novels with Prompt Learning

• Authors: Yuchen Yan, Hanjie Zhao, Senbin Zhu, Hongde Liu, Zhihong Zhang, Yuxiang Jia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09452
• Pdf URL: https://arxiv.org/pdf/2408.09452
• Copy Paste: [[2408.09452]] Identifying Speakers and Addressees of Quotations in Novels with Prompt Learning(https://arxiv.org/abs/2408.09452)
• Keywords: extraction, large language model
• Abstract: Quotations in literary works, especially novels, are important to create characters, reflect character relationships, and drive plot development. Current research on quotation extraction in novels primarily focuses on quotation attribution, i.e., identifying the speaker of the quotation. However, the addressee of the quotation is also important to construct the relationship between the speaker and the addressee. To tackle the problem of dataset scarcity, we annotate the first Chinese quotation corpus with elements including speaker, addressee, speaking mode and linguistic cue. We propose prompt learning-based methods for speaker and addressee identification based on fine-tuned pre-trained models. Experiments on both Chinese and English datasets show the effectiveness of the proposed methods, which outperform methods based on zero-shot and few-shot large language models.

Title: Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling

• Authors: Harry Jake Cunningham, Giorgio Giannone, Mingtian Zhang, Marc Peter Deisenroth
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09453
• Pdf URL: https://arxiv.org/pdf/2408.09453
• Copy Paste: [[2408.09453]] Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling(https://arxiv.org/abs/2408.09453)
• Keywords: transformer
• Abstract: Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ($\texttt{MRConv}$), a novel approach to parameterizing global convolutional kernels for long-sequence modelling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, $\texttt{MRConv}$ learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D $\texttt{MRConv}$ layers.

Title: Retina-inspired Object Motion Segmentation

• Authors: Victoria Clerico (1), Shay Snyder (1), Arya Lohia (1), Md Abdullah-Al Kaiser (2), Gregory Schwartz (3), Akhilesh Jaiswal (2), Maryam Parsa (1) ((1) George Mason Unviersity, (2) University of Southern, California, (3) Northwestern University)
• Subjects: cs.CV, cs.NE, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.09454
• Pdf URL: https://arxiv.org/pdf/2408.09454
• Copy Paste: [[2408.09454]] Retina-inspired Object Motion Segmentation(https://arxiv.org/abs/2408.09454)
• Keywords: robust, segmentation
• Abstract: Dynamic Vision Sensors (DVS) have emerged as a revolutionary technology with a high temporal resolution that far surpasses RGB cameras. DVS technology draws biological inspiration from photoreceptors and the initial retinal synapse. Our research showcases the potential of additional retinal functionalities to extract visual features. We provide a domain-agnostic and efficient algorithm for ego-motion compensation based on Object Motion Sensitivity (OMS), one of the multiple robust features computed within the mammalian retina. We develop a framework based on experimental neuroscience that translates OMS' biological circuitry to a low-overhead algorithm. OMS processes DVS data from dynamic scenes to perform pixel-wise object motion segmentation. Using a real and a synthetic dataset, we highlight OMS' ability to differentiate object motion from ego-motion, bypassing the need for deep networks. This paper introduces a bio-inspired computer vision method that dramatically reduces the number of parameters by a factor of 1000 compared to prior works. Our work paves the way for robust, high-speed, and low-bandwidth decision-making for in-sensor computations.

Title: G2Face: High-Fidelity Reversible Face Anonymization via Generative and Geometric Priors

• Authors: Haoxin Yang, Xuemiao Xu, Cheng Xu, Huaidong Zhang, Jing Qin, Yi Wang, Pheng-Ann Heng, Shengfeng He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09458
• Pdf URL: https://arxiv.org/pdf/2408.09458
• Copy Paste: [[2408.09458]] G2Face: High-Fidelity Reversible Face Anonymization via Generative and Geometric Priors(https://arxiv.org/abs/2408.09458)
• Keywords: privacy, generative
• Abstract: Reversible face anonymization, unlike traditional face pixelization, seeks to replace sensitive identity information in facial images with synthesized alternatives, preserving privacy without sacrificing image clarity. Traditional methods, such as encoder-decoder networks, often result in significant loss of facial details due to their limited learning capacity. Additionally, relying on latent manipulation in pre-trained GANs can lead to changes in ID-irrelevant attributes, adversely affecting data utility due to GAN inversion inaccuracies. This paper introduces G\textsuperscript{2}Face, which leverages both generative and geometric priors to enhance identity manipulation, achieving high-quality reversible face anonymization without compromising data utility. We utilize a 3D face model to extract geometric information from the input face, integrating it with a pre-trained GAN-based decoder. This synergy of generative and geometric priors allows the decoder to produce realistic anonymized faces with consistent geometry. Moreover, multi-scale facial features are extracted from the original face and combined with the decoder using our novel identity-aware feature fusion blocks (IFF). This integration enables precise blending of the generated facial patterns with the original ID-irrelevant features, resulting in accurate identity manipulation. Extensive experiments demonstrate that our method outperforms existing state-of-the-art techniques in face anonymization and recovery, while preserving high data utility. Code is available at this https URL.

Title: WPN: An Unlearning Method Based on N-pair Contrastive Learning in Language Models

• Authors: Guitao Chen, Yunshen Wang, Hongye Sun, Guang Chen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.09459
• Pdf URL: https://arxiv.org/pdf/2408.09459
• Copy Paste: [[2408.09459]] WPN: An Unlearning Method Based on N-pair Contrastive Learning in Language Models(https://arxiv.org/abs/2408.09459)
• Keywords: attack, robust, generative
• Abstract: Generative language models (LMs) offer numerous advantages but may produce inappropriate or harmful outputs due to the harmful knowledge acquired during pre-training. This knowledge often manifests as undesirable correspondences, such as "harmful prompts" leading to "harmful outputs," which our research aims to mitigate through unlearning techniques.However, existing unlearning methods based on gradient ascent can significantly impair the performance of LMs. To address this issue, we propose a novel approach called Weighted Positional N-pair (WPN) Learning, which leverages position-weighted mean pooling within an n-pair contrastive learning framework. WPN is designed to modify the output distribution of LMs by eliminating specific harmful outputs (e.g., replacing toxic responses with neutral ones), thereby transforming the model's behavior from "harmful prompt-harmful output" to "harmful prompt-harmless response".Experiments on OPT and GPT-NEO LMs show that WPN effectively reduces the proportion of harmful responses, achieving a harmless rate of up to 95.8\% while maintaining stable performance on nine common benchmarks (with less than 2\% degradation on average). Moreover, we provide empirical evidence to demonstrate WPN's ability to weaken the harmful correspondences in terms of generalizability and robustness, as evaluated on out-of-distribution test sets and under adversarial attacks.

Title: MedMAP: Promoting Incomplete Multi-modal Brain Tumor Segmentation with Alignment

• Authors: Tianyi Liu, Zhaorui Tan, Muyin Chen, Xi Yang, Haochuan Jiang, Kaizhu Huang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09465
• Pdf URL: https://arxiv.org/pdf/2408.09465
• Copy Paste: [[2408.09465]] MedMAP: Promoting Incomplete Multi-modal Brain Tumor Segmentation with Alignment(https://arxiv.org/abs/2408.09465)
• Keywords: segmentation
• Abstract: Brain tumor segmentation is often based on multiple magnetic resonance imaging (MRI). However, in clinical practice, certain modalities of MRI may be missing, which presents a more difficult scenario. To cope with this challenge, Knowledge Distillation, Domain Adaption, and Shared Latent Space have emerged as commonly promising strategies. However, recent efforts typically overlook the modality gaps and thus fail to learn important invariant feature representations across different modalities. Such drawback consequently leads to limited performance for missing modality models. To ameliorate these problems, pre-trained models are used in natural visual segmentation tasks to minimize the gaps. However, promising pre-trained models are often unavailable in medical image segmentation tasks. Along this line, in this paper, we propose a novel paradigm that aligns latent features of involved modalities to a well-defined distribution anchor as the substitution of the pre-trained model}. As a major contribution, we prove that our novel training paradigm ensures a tight evidence lower bound, thus theoretically certifying its effectiveness. Extensive experiments on different backbones validate that the proposed paradigm can enable invariant feature representations and produce models with narrowed modality gaps. Models with our alignment paradigm show their superior performance on both BraTS2018 and BraTS2020 datasets.

Title: Enhancing Adversarial Transferability with Adversarial Weight Tuning

• Authors: Jiahao Chen, Zhou Feng, Rui Zeng, Yuwen Pu, Chunyi Zhou, Yi Jiang, Yuyou Gan, Jinbao Li, Shouling Ji, Shouling_Ji
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.09469
• Pdf URL: https://arxiv.org/pdf/2408.09469
• Copy Paste: [[2408.09469]] Enhancing Adversarial Transferability with Adversarial Weight Tuning(https://arxiv.org/abs/2408.09469)
• Keywords: attack, data-free, transformer
• Abstract: Deep neural networks (DNNs) are vulnerable to adversarial examples (AEs) that mislead the model while appearing benign to human observers. A critical concern is the transferability of AEs, which enables black-box attacks without direct access to the target model. However, many previous attacks have failed to explain the intrinsic mechanism of adversarial transferability. In this paper, we rethink the property of transferable AEs and reformalize the formulation of transferability. Building on insights from this mechanism, we analyze the generalization of AEs across models with different architectures and prove that we can find a local perturbation to mitigate the gap between surrogate and target models. We further establish the inner connections between model smoothness and flat local maxima, both of which contribute to the transferability of AEs. Further, we propose a new adversarial attack algorithm, \textbf{A}dversarial \textbf{W}eight \textbf{T}uning (AWT), which adaptively adjusts the parameters of the surrogate model using generated AEs to optimize the flat local maxima and model smoothness simultaneously, without the need for extra data. AWT is a data-free tuning method that combines gradient-based and model-based attack methods to enhance the transferability of AEs. Extensive experiments on a variety of models with different architectures on ImageNet demonstrate that AWT yields superior performance over other attacks, with an average increase of nearly 5\% and 10\% attack success rates on CNN-based and Transformer-based models, respectively, compared to state-of-the-art attacks.

Title: Image-Based Geolocation Using Large Vision-Language Models

• Authors: Yi Liu, Junchen Ding, Gelei Deng, Yuekang Li, Tianwei Zhang, Weisong Sun, Yaowen Zheng, Jingquan Ge, Yang Liu
• Subjects: cs.CR, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09474
• Pdf URL: https://arxiv.org/pdf/2408.09474
• Copy Paste: [[2408.09474]] Image-Based Geolocation Using Large Vision-Language Models(https://arxiv.org/abs/2408.09474)
• Keywords: security, privacy, protect, robust
• Abstract: Geolocation is now a vital aspect of modern life, offering numerous benefits but also presenting serious privacy concerns. The advent of large vision-language models (LVLMs) with advanced image-processing capabilities introduces new risks, as these models can inadvertently reveal sensitive geolocation information. This paper presents the first in-depth study analyzing the challenges posed by traditional deep learning and LVLM-based geolocation methods. Our findings reveal that LVLMs can accurately determine geolocations from images, even without explicit geographic training. To address these challenges, we introduce \tool{}, an innovative framework that significantly enhances image-based geolocation accuracy. \tool{} employs a systematic chain-of-thought (CoT) approach, mimicking human geoguessing strategies by carefully analyzing visual and contextual cues such as vehicle types, architectural styles, natural landscapes, and cultural elements. Extensive testing on a dataset of 50,000 ground-truth data points shows that \tool{} outperforms both traditional models and human benchmarks in accuracy. It achieves an impressive average score of 4550.5 in the GeoGuessr game, with an 85.37\% win rate, and delivers highly precise geolocation predictions, with the closest distances as accurate as 0.3 km. Furthermore, our study highlights issues related to dataset integrity, leading to the creation of a more robust dataset and a refined framework that leverages LVLMs' cognitive capabilities to improve geolocation precision. These findings underscore \tool{}'s superior ability to interpret complex visual data, the urgent need to address emerging security vulnerabilities posed by LVLMs, and the importance of responsible AI development to ensure user privacy protection.

Title: Advances in Multiple Instance Learning for Whole Slide Image Analysis: Techniques, Challenges, and Future Directions

• Authors: Jun Wang, Yu Mao, Nan Guan, Chun Jason Xue
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09476
• Pdf URL: https://arxiv.org/pdf/2408.09476
• Copy Paste: [[2408.09476]] Advances in Multiple Instance Learning for Whole Slide Image Analysis: Techniques, Challenges, and Future Directions(https://arxiv.org/abs/2408.09476)
• Keywords: transformer
• Abstract: Whole slide images (WSIs) are gigapixel-scale digital images of H\&E-stained tissue samples widely used in pathology. The substantial size and complexity of WSIs pose unique analytical challenges. Multiple Instance Learning (MIL) has emerged as a powerful approach for addressing these challenges, particularly in cancer classification and detection. This survey provides a comprehensive overview of the challenges and methodologies associated with applying MIL to WSI analysis, including attention mechanisms, pseudo-labeling, transformers, pooling functions, and graph neural networks. Additionally, it explores the potential of MIL in discovering cancer cell morphology, constructing interpretable machine learning models, and quantifying cancer grading. By summarizing the current challenges, methodologies, and potential applications of MIL in WSI analysis, this survey aims to inform researchers about the state of the field and inspire future research directions.

Title: Mitigating Noise Detriment in Differentially Private Federated Learning with Model Pre-training

• Authors: Huitong Jin, Yipeng Zhou, Laizhong Cui, Quan Z. Sheng
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.09478
• Pdf URL: https://arxiv.org/pdf/2408.09478
• Copy Paste: [[2408.09478]] Mitigating Noise Detriment in Differentially Private Federated Learning with Model Pre-training(https://arxiv.org/abs/2408.09478)
• Keywords: privacy, federate
• Abstract: Pre-training exploits public datasets to pre-train an advanced machine learning model, so that the model can be easily tuned to adapt to various downstream tasks. Pre-training has been extensively explored to mitigate computation and communication resource consumption. Inspired by these advantages, we are the first to explore how model pre-training can mitigate noise detriment in differentially private federated learning (DPFL). DPFL is upgraded from federated learning (FL), the de-facto standard for privacy preservation when training the model across multiple clients owning private data. DPFL introduces differentially private (DP) noises to obfuscate model gradients exposed in FL, which however can considerably impair model accuracy. In our work, we compare head fine-tuning (HT) and full fine-tuning (FT), which are based on pre-training, with scratch training (ST) in DPFL through a comprehensive empirical study. Our experiments tune pre-trained models (obtained by pre-training on ImageNet-1K) with CIFAR-10, CHMNIST and Fashion-MNIST (FMNIST) datasets, respectively. The results demonstrate that HT and FT can significantly mitigate noise influence by diminishing gradient exposure times. In particular, HT outperforms FT when the privacy budget is tight or the model size is large. Visualization and explanation study further substantiates our findings. Our pioneering study introduces a new perspective on enhancing DPFL and expanding its practical applications.

Title: PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis

• Authors: Meng Luo, Hao Fei, Bobo Li, Shengqiong Wu, Qian Liu, Soujanya Poria, Erik Cambria, Mong-Li Lee, Wynne Hsu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09481
• Pdf URL: https://arxiv.org/pdf/2408.09481
• Copy Paste: [[2408.09481]] PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis(https://arxiv.org/abs/2408.09481)
• Keywords: extraction, large language model
• Abstract: While existing Aspect-based Sentiment Analysis (ABSA) has received extensive effort and advancement, there are still gaps in defining a more holistic research target seamlessly integrating multimodality, conversation context, fine-granularity, and also covering the changing sentiment dynamics as well as cognitive causal rationales. This paper bridges the gaps by introducing a multimodal conversational ABSA, where two novel subtasks are proposed: 1) Panoptic Sentiment Sextuple Extraction, panoramically recognizing holder, target, aspect, opinion, sentiment, rationale from multi-turn multi-party multimodal dialogue. 2) Sentiment Flipping Analysis, detecting the dynamic sentiment transformation throughout the conversation with the causal reasons. To benchmark the tasks, we construct PanoSent, a dataset annotated both manually and automatically, featuring high quality, large scale, multimodality, multilingualism, multi-scenarios, and covering both implicit and explicit sentiment elements. To effectively address the tasks, we devise a novel Chain-of-Sentiment reasoning framework, together with a novel multimodal large language model (namely Sentica) and a paraphrase-based verification mechanism. Extensive evaluations demonstrate the superiority of our methods over strong baselines, validating the efficacy of all our proposed methods. The work is expected to open up a new era for the ABSA community, and thus all our codes and data are open at this https URL

Title: Ancestral Reinforcement Learning: Unifying Zeroth-Order Optimization and Genetic Algorithms for Reinforcement Learning

• Authors: So Nakashima, Tetsuya J. Kobayashi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09493
• Pdf URL: https://arxiv.org/pdf/2408.09493
• Copy Paste: [[2408.09493]] Ancestral Reinforcement Learning: Unifying Zeroth-Order Optimization and Genetic Algorithms for Reinforcement Learning(https://arxiv.org/abs/2408.09493)
• Keywords: robust
• Abstract: Reinforcement Learning (RL) offers a fundamental framework for discovering optimal action strategies through interactions within unknown environments. Recent advancement have shown that the performance and applicability of RL can significantly be enhanced by exploiting a population of agents in various ways. Zeroth-Order Optimization (ZOO) leverages an agent population to estimate the gradient of the objective function, enabling robust policy refinement even in non-differentiable scenarios. As another application, Genetic Algorithms (GA) boosts the exploration of policy landscapes by mutational generation of policy diversity in an agent population and its refinement by selection. A natural question is whether we can have the best of two worlds that the agent population can have. In this work, we propose Ancestral Reinforcement Learning (ARL), which synergistically combines the robust gradient estimation of ZOO with the exploratory power of GA. The key idea in ARL is that each agent within a population infers gradient by exploiting the history of its ancestors, i.e., the ancestor population in the past, while maintaining the diversity of policies in the current population as in GA. We also theoretically reveal that the populational search in ARL implicitly induces the KL-regularization of the objective function, resulting in the enhanced exploration. Our results extend the applicability of populational algorithms for RL.

Title: Source-Free Test-Time Adaptation For Online Surface-Defect Detection

• Authors: Yiran Song, Qianyu Zhou, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09494
• Pdf URL: https://arxiv.org/pdf/2408.09494
• Copy Paste: [[2408.09494]] Source-Free Test-Time Adaptation For Online Surface-Defect Detection(https://arxiv.org/abs/2408.09494)
• Keywords: robust, segmentation
• Abstract: Surface defect detection is significant in industrial production. However, detecting defects with varying textures and anomaly classes during the test time is challenging. This arises due to the differences in data distributions between source and target domains. Collecting and annotating new data from the target domain and retraining the model is time-consuming and costly. In this paper, we propose a novel test-time adaptation surface-defect detection approach that adapts pre-trained models to new domains and classes during inference. Our approach involves two core ideas. Firstly, we introduce a supervisor to filter samples and select only those with high confidence to update the model. This ensures that the model is not excessively biased by incorrect data. Secondly, we propose the augmented mean prediction to generate robust pseudo labels and a dynamically-balancing loss to facilitate the model in effectively integrating classification and segmentation results to improve surface-defect detection accuracy. Our approach is real-time and does not require additional offline retraining. Experiments demonstrate it outperforms state-of-the-art techniques.

Title: StyleBrush: Style Extraction and Transfer from a Single Image

• Authors: Wancheng Feng, Wanquan Feng, Dawei Huang, Jiaming Pei, Guangliang Cheng, Lukun Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09496
• Pdf URL: https://arxiv.org/pdf/2408.09496
• Copy Paste: [[2408.09496]] StyleBrush: Style Extraction and Transfer from a Single Image(https://arxiv.org/abs/2408.09496)
• Keywords: extraction
• Abstract: Stylization for visual content aims to add specific style patterns at the pixel level while preserving the original structural features. Compared with using predefined styles, stylization guided by reference style images is more challenging, where the main difficulty is to effectively separate style from structural elements. In this paper, we propose StyleBrush, a method that accurately captures styles from a reference image and ``brushes'' the extracted style onto other input visual content. Specifically, our architecture consists of two branches: ReferenceNet, which extracts style from the reference image, and Structure Guider, which extracts structural features from the input image, thus enabling image-guided stylization. We utilize LLM and T2I models to create a dataset comprising 100K high-quality style images, encompassing a diverse range of styles and contents with high aesthetic score. To construct training pairs, we crop different regions of the same training image. Experiments show that our approach achieves state-of-the-art results through both qualitative and quantitative analyses. We will release our code and dataset upon acceptance of the paper.

Title: Out-of-distribution generalization via composition: a lens through induction heads in Transformers

• Authors: Jiajun Song, Zhuoyan Xu, Yiqiao Zhong
• Subjects: cs.CL, cs.AI, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.09503
• Pdf URL: https://arxiv.org/pdf/2408.09503
• Copy Paste: [[2408.09503]] Out-of-distribution generalization via composition: a lens through induction heads in Transformers(https://arxiv.org/abs/2408.09503)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) such as GPT-4 sometimes appear to be creative, solving novel tasks often with a few demonstrations in the prompt. These tasks require the models to generalize on distributions different from those from training data -- which is known as out-of-distribution (OOD) generalization. Despite the tremendous success of LLMs, how they approach OOD generalization remains an open and underexplored question. We examine OOD generalization in settings where instances are generated according to hidden rules, including in-context learning with symbolic reasoning. Models are required to infer the hidden rules behind input prompts without any fine-tuning. We empirically examined the training dynamics of Transformers on a synthetic example and conducted extensive experiments on a variety of pretrained LLMs, focusing on a type of components known as induction heads. We found that OOD generalization and composition are tied together -- models can learn rules by composing two self-attention layers, thereby achieving OOD generalization. Furthermore, a shared latent subspace in the embedding (or feature) space acts as a bridge for composition by aligning early layers and later layers, which we refer to as the common bridge representation hypothesis.

Title: A Unified Framework for Interpretable Transformers Using PDEs and Information Theory

• Authors: Yukun Zhang
• Subjects: cs.LG, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/2408.09523
• Pdf URL: https://arxiv.org/pdf/2408.09523
• Copy Paste: [[2408.09523]] A Unified Framework for Interpretable Transformers Using PDEs and Information Theory(https://arxiv.org/abs/2408.09523)
• Keywords: interpretability, diffusion, transformer
• Abstract: This paper presents a novel unified theoretical framework for understanding Transformer architectures by integrating Partial Differential Equations (PDEs), Neural Information Flow Theory, and Information Bottleneck Theory. We model Transformer information dynamics as a continuous PDE process, encompassing diffusion, self-attention, and nonlinear residual components. Our comprehensive experiments across image and text modalities demonstrate that the PDE model effectively captures key aspects of Transformer behavior, achieving high similarity (cosine similarity > 0.98) with Transformer attention distributions across all layers. While the model excels in replicating general information flow patterns, it shows limitations in fully capturing complex, non-linear transformations. This work provides crucial theoretical insights into Transformer mechanisms, offering a foundation for future optimizations in deep learning architectural design. We discuss the implications of our findings, potential applications in model interpretability and efficiency, and outline directions for enhancing PDE models to better mimic the intricate behaviors observed in Transformers, paving the way for more transparent and optimized AI systems.

Title: Fine-gained air quality inference based on low-quality sensing data using self-supervised learning

• Authors: Meng Xu, Ke Han, Weijian Hu, Wen Ji
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09526
• Pdf URL: https://arxiv.org/pdf/2408.09526
• Copy Paste: [[2408.09526]] Fine-gained air quality inference based on low-quality sensing data using self-supervised learning(https://arxiv.org/abs/2408.09526)
• Keywords: robust, interpretability
• Abstract: Fine-grained air quality (AQ) mapping is made possible by the proliferation of cheap AQ micro-stations (MSs). However, their measurements are often inaccurate and sensitive to local disturbances, in contrast to standardized stations (SSs) that provide accurate readings but fall short in number. To simultaneously address the issues of low data quality (MSs) and high label sparsity (SSs), a multi-task spatio-temporal network (MTSTN) is proposed, which employs self-supervised learning to utilize massive unlabeled data, aided by seasonal and trend decomposition of MS data offering reliable information as features. The MTSTN is applied to infer NO$_2$, O$_3$ and PM$_{2.5}$ concentrations in a 250 km$^2$ area in Chengdu, China, at a resolution of 500m$\times$500m$\times$1hr. Data from 55 SSs and 323 MSs were used, along with meteorological, traffic, geographic and timestamp data as features. The MTSTN excels in accuracy compared to several benchmarks, and its performance is greatly enhanced by utilizing low-quality MS data. A series of ablation and pressure tests demonstrate the results' robustness and interpretability, showcasing the MTSTN's practical value for accurate and affordable AQ inference.

Title: Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path

• Authors: Xinnan Dai, Qihao Wen, Yifei Shen, Hongzhi Wen, Dongsheng Li, Jiliang Tang, Caihua Shan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09529
• Pdf URL: https://arxiv.org/pdf/2408.09529
• Copy Paste: [[2408.09529]] Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path(https://arxiv.org/abs/2408.09529)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved great success in various reasoning tasks. In this work, we focus on the graph reasoning ability of LLMs. Although theoretical studies proved that LLMs are capable of handling graph reasoning tasks, empirical evaluations reveal numerous failures. To deepen our understanding on this discrepancy, we revisit the ability of LLMs on three fundamental graph tasks: graph description translation, graph connectivity, and the shortest-path problem. Our findings suggest that LLMs can fail to understand graph structures through text descriptions and exhibit varying performance for all these three fundamental tasks. Meanwhile, we perform a real-world investigation on knowledge graphs and make consistent observations with our findings. The codes and datasets are available.

Title: AnomalyFactory: Regard Anomaly Generation as Unsupervised Anomaly Localization

• Authors: Ying Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09533
• Pdf URL: https://arxiv.org/pdf/2408.09533
• Copy Paste: [[2408.09533]] AnomalyFactory: Regard Anomaly Generation as Unsupervised Anomaly Localization(https://arxiv.org/abs/2408.09533)
• Keywords: generative
• Abstract: Recent advances in anomaly generation approaches alleviate the effect of data insufficiency on task of anomaly localization. While effective, most of them learn multiple large generative models on different datasets and cumbersome anomaly prediction models for different classes. To address the limitations, we propose a novel scalable framework, named AnomalyFactory, that unifies unsupervised anomaly generation and localization with same network architecture. It starts with a BootGenerator that combines structure of a target edge map and appearance of a reference color image with the guidance of a learned heatmap. Then, it proceeds with a FlareGenerator that receives supervision signals from the BootGenerator and reforms the heatmap to indicate anomaly locations in the generated image. Finally, it easily transforms the same network architecture to a BlazeDetector that localizes anomaly pixels with the learned heatmap by converting the anomaly images generated by the FlareGenerator to normal images. By manipulating the target edge maps and combining them with various reference images, AnomalyFactory generates authentic and diversity samples cross domains. Comprehensive experiments carried on 5 datasets, including MVTecAD, VisA, MVTecLOCO, MADSim and RealIAD, demonstrate that our approach is superior to competitors in generation capability and scalability.

Title: Byzantine-resilient Federated Learning Employing Normalized Gradients on Non-IID Datasets

• Authors: Shiyuan Zuo, Xingrun Yan, Rongfei Fan, Li Shen, Puning Zhao, Jie Xu, Han Hu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.09539
• Pdf URL: https://arxiv.org/pdf/2408.09539
• Copy Paste: [[2408.09539]] Byzantine-resilient Federated Learning Employing Normalized Gradients on Non-IID Datasets(https://arxiv.org/abs/2408.09539)
• Keywords: attack, robust, federate
• Abstract: In practical federated learning (FL) systems, the presence of malicious Byzantine attacks and data heterogeneity often introduces biases into the learning process. However, existing Byzantine-robust methods typically only achieve a compromise between adaptability to different loss function types (including both strongly convex and non-convex) and robustness to heterogeneous datasets, but with non-zero optimality gap. Moreover, this compromise often comes at the cost of high computational complexity for aggregation, which significantly slows down the training speed. To address this challenge, we propose a federated learning approach called Federated Normalized Gradients Algorithm (Fed-NGA). Fed-NGA simply normalizes the uploaded local gradients to be unit vectors before aggregation, achieving a time complexity of $\mathcal{O}(pM)$, where $p$ represents the dimension of model parameters and $M$ is the number of participating clients. This complexity scale achieves the best level among all the existing Byzantine-robust methods. Furthermore, through rigorous proof, we demonstrate that Fed-NGA transcends the trade-off between adaptability to loss function type and data heterogeneity and the limitation of non-zero optimality gap in existing literature. Specifically, Fed-NGA can adapt to both non-convex loss functions and non-IID datasets simultaneously, with zero optimality gap at a rate of $\mathcal{O} (1/T^{\frac{1}{2} - \delta})$, where T is the iteration number and $\delta \in (0,\frac{1}{2})$. In cases where the loss function is strongly convex, the zero optimality gap achieving rate can be improved to be linear. Experimental results provide evidence of the superiority of our proposed Fed-NGA on time complexity and convergence performance over baseline methods.

Title: Using ChatGPT to Score Essays and Short-Form Constructed Responses

• Authors: Mark D. Shermis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09540
• Pdf URL: https://arxiv.org/pdf/2408.09540
• Copy Paste: [[2408.09540]] Using ChatGPT to Score Essays and Short-Form Constructed Responses(https://arxiv.org/abs/2408.09540)
• Keywords: fair, large language model
• Abstract: This study aimed to determine if ChatGPT's large language models could match the scoring accuracy of human and machine scores from the ASAP competition. The investigation focused on various prediction models, including linear regression, random forest, gradient boost, and boost. ChatGPT's performance was evaluated against human raters using quadratic weighted kappa (QWK) metrics. Results indicated that while ChatGPT's gradient boost model achieved QWKs close to human raters for some data sets, its overall performance was inconsistent and often lower than human scores. The study highlighted the need for further refinement, particularly in handling biases and ensuring scoring fairness. Despite these challenges, ChatGPT demonstrated potential for scoring efficiency, especially with domain-specific fine-tuning. The study concludes that ChatGPT can complement human scoring but requires additional development to be reliable for high-stakes assessments. Future research should improve model accuracy, address ethical considerations, and explore hybrid models combining ChatGPT with empirical methods.

Title: No Such Thing as a General Learner: Language models and their dual optimization

• Authors: Emmanuel Chemla, Ryan M. Nefdt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09544
• Pdf URL: https://arxiv.org/pdf/2408.09544
• Copy Paste: [[2408.09544]] No Such Thing as a General Learner: Language models and their dual optimization(https://arxiv.org/abs/2408.09544)
• Keywords: large language model
• Abstract: What role can the otherwise successful Large Language Models (LLMs) play in the understanding of human cognition, and in particular in terms of informing language acquisition debates? To contribute to this question, we first argue that neither humans nor LLMs are general learners, in a variety of senses. We make a novel case for how in particular LLMs follow a dual-optimization process: they are optimized during their training (which is typically compared to language acquisition), and modern LLMs have also been selected, through a process akin to natural selection in a species. From this perspective, we argue that the performance of LLMs, whether similar or dissimilar to that of humans, does not weigh easily on important debates about the importance of human cognitive biases for language.

Title: Seamless Integration: Sampling Strategies in Federated Learning Systems

• Authors: Tatjana Legler, Vinit Hegiste, Martin Ruskowski
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09545
• Pdf URL: https://arxiv.org/pdf/2408.09545
• Copy Paste: [[2408.09545]] Seamless Integration: Sampling Strategies in Federated Learning Systems(https://arxiv.org/abs/2408.09545)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) represents a paradigm shift in the field of machine learning, offering an approach for a decentralized training of models across a multitude of devices while maintaining the privacy of local data. However, the dynamic nature of FL systems, characterized by the ongoing incorporation of new clients with potentially diverse data distributions and computational capabilities, poses a significant challenge to the stability and efficiency of these distributed learning networks. The seamless integration of new clients is imperative to sustain and enhance the performance and robustness of FL systems. This paper looks into the complexities of integrating new clients into existing FL systems and explores how data heterogeneity and varying data distribution (not independent and identically distributed) among them can affect model training, system efficiency, scalability and stability. Despite these challenges, the integration of new clients into FL systems presents opportunities to enhance data diversity, improve learning performance, and leverage distributed computational power. In contrast to other fields of application such as the distributed optimization of word predictions on Gboard (where federated learning once originated), there are usually only a few clients in the production environment, which is why information from each new client becomes all the more valuable. This paper outlines strategies for effective client selection strategies and solutions for ensuring system scalability and stability. Using the example of images from optical quality inspection, it offers insights into practical approaches. In conclusion, this paper proposes that addressing the challenges presented by new client integration is crucial to the advancement and efficiency of distributed learning networks, thus paving the way for the adoption of Federated Learning in production environments.

Title: Addressing Heterogeneity in Federated Learning: Challenges and Solutions for a Shared Production Environment

• Authors: Tatjana Legler, Vinit Hegiste, Ahmed Anwar, Martin Ruskowski
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09556
• Pdf URL: https://arxiv.org/pdf/2408.09556
• Copy Paste: [[2408.09556]] Addressing Heterogeneity in Federated Learning: Challenges and Solutions for a Shared Production Environment(https://arxiv.org/abs/2408.09556)
• Keywords: privacy, robust, federate, fair
• Abstract: Federated learning (FL) has emerged as a promising approach to training machine learning models across decentralized data sources while preserving data privacy, particularly in manufacturing and shared production environments. However, the presence of data heterogeneity variations in data distribution, quality, and volume across different or clients and production sites, poses significant challenges to the effectiveness and efficiency of FL. This paper provides a comprehensive overview of heterogeneity in FL within the context of manufacturing, detailing the types and sources of heterogeneity, including non-independent and identically distributed (non-IID) data, unbalanced data, variable data quality, and statistical heterogeneity. We discuss the impact of these types of heterogeneity on model training and review current methodologies for mitigating their adverse effects. These methodologies include personalized and customized models, robust aggregation techniques, and client selection techniques. By synthesizing existing research and proposing new strategies, this paper aims to provide insight for effectively managing data heterogeneity in FL, enhancing model robustness, and ensuring fair and efficient training across diverse environments. Future research directions are also identified, highlighting the need for adaptive and scalable solutions to further improve the FL paradigm in the context of Industry 4.0.

Title: Generating Automatically Print/Scan Textures for Morphing Attack Detection Applications

• Authors: Juan E. Tapia, Maximilian Russo, Christoph Busch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09558
• Pdf URL: https://arxiv.org/pdf/2408.09558
• Copy Paste: [[2408.09558]] Generating Automatically Print/Scan Textures for Morphing Attack Detection Applications(https://arxiv.org/abs/2408.09558)
• Keywords: privacy, attack
• Abstract: Morphing Attack Detection (MAD) is a relevant topic that aims to detect attempts by unauthorised individuals to access a "valid" identity. One of the main scenarios is printing morphed images and submitting the respective print in a passport application process. Today, small datasets are available to train the MAD algorithm because of privacy concerns and the limitations resulting from the effort associated with the printing and scanning of images at large numbers. In order to improve the detection capabilities and spot such morphing attacks, it will be necessary to have a larger and more realistic dataset representing the passport application scenario with the diversity of devices and the resulting printed scanned or compressed images. Creating training data representing the diversity of attacks is a very demanding task because the training material is developed manually. This paper proposes two different methods based on transfer-transfer for automatically creating digital print/scan face images and using such images in the training of a Morphing Attack Detection algorithm. Our proposed method can reach an Equal Error Rate (EER) of 3.84% and 1.92% on the FRGC/FERET database when including our synthetic and texture-transfer print/scan with 600 dpi to handcrafted images, respectively.

Title: HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model

• Authors: Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, Ping Luo
• Subjects: cs.CL, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.09559
• Pdf URL: https://arxiv.org/pdf/2408.09559
• Copy Paste: [[2408.09559]] HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model(https://arxiv.org/abs/2408.09559)
• Keywords: robust, large language model
• Abstract: Large Language Model (LLM)-based agents exhibit significant potential across various domains, operating as interactive systems that process environmental observations to generate executable actions for target tasks. The effectiveness of these agents is significantly influenced by their memory mechanism, which records historical experiences as sequences of action-observation pairs. We categorize memory into two types: cross-trial memory, accumulated across multiple attempts, and in-trial memory (working memory), accumulated within a single attempt. While considerable research has optimized performance through cross-trial memory, the enhancement of agent performance through improved working memory utilization remains underexplored. Instead, existing approaches often involve directly inputting entire historical action-observation pairs into LLMs, leading to redundancy in long-horizon tasks. Inspired by human problem-solving strategies, this paper introduces HiAgent, a framework that leverages subgoals as memory chunks to manage the working memory of LLM-based agents hierarchically. Specifically, HiAgent prompts LLMs to formulate subgoals before generating executable actions and enables LLMs to decide proactively to replace previous subgoals with summarized observations, retaining only the action-observation pairs relevant to the current subgoal. Experimental results across five long-horizon tasks demonstrate that HiAgent achieves a twofold increase in success rate and reduces the average number of steps required by 3.8. Additionally, our analysis shows that HiAgent consistently improves performance across various steps, highlighting its robustness and generalizability. Project Page: this https URL .

Title: Grammatical Error Feedback: An Implicit Evaluation Approach

• Authors: Stefano Bannò, Kate Knill, Mark J. F. Gales
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09565
• Pdf URL: https://arxiv.org/pdf/2408.09565
• Copy Paste: [[2408.09565]] Grammatical Error Feedback: An Implicit Evaluation Approach(https://arxiv.org/abs/2408.09565)
• Keywords: large language model
• Abstract: Grammatical feedback is crucial for consolidating second language (L2) learning. Most research in computer-assisted language learning has focused on feedback through grammatical error correction (GEC) systems, rather than examining more holistic feedback that may be more useful for learners. This holistic feedback will be referred to as grammatical error feedback (GEF). In this paper, we present a novel implicit evaluation approach to GEF that eliminates the need for manual feedback annotations. Our method adopts a grammatical lineup approach where the task is to pair feedback and essay representations from a set of possible alternatives. This matching process can be performed by appropriately prompting a large language model (LLM). An important aspect of this process, explored here, is the form of the lineup, i.e., the selection of foils. This paper exploits this framework to examine the quality and need for GEC to generate feedback, as well as the system used to generate feedback, using essays from the Cambridge Learner Corpus.

Title: Enhancing ASL Recognition with GCNs and Successive Residual Connections

• Authors: Ushnish Sarkar, Archisman Chakraborti, Tapas Samanta, Sarbajit Pal, Amitabha Das
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09567
• Pdf URL: https://arxiv.org/pdf/2408.09567
• Copy Paste: [[2408.09567]] Enhancing ASL Recognition with GCNs and Successive Residual Connections(https://arxiv.org/abs/2408.09567)
• Keywords: robust
• Abstract: This study presents a novel approach for enhancing American Sign Language (ASL) recognition using Graph Convolutional Networks (GCNs) integrated with successive residual connections. The method leverages the MediaPipe framework to extract key landmarks from each hand gesture, which are then used to construct graph representations. A robust preprocessing pipeline, including translational and scale normalization techniques, ensures consistency across the dataset. The constructed graphs are fed into a GCN-based neural architecture with residual connections to improve network stability. The architecture achieves state-of-the-art results, demonstrating superior generalization capabilities with a validation accuracy of 99.14%.

Title: Say My Name: a Model's Bias Discovery Framework

• Authors: Massimiliano Ciranni, Luca Molinaro, Carlo Alberto Barbano, Attilio Fiandrotti, Vittorio Murino, Vito Paolo Pastore, Enzo Tartaglione
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.09570
• Pdf URL: https://arxiv.org/pdf/2408.09570
• Copy Paste: [[2408.09570]] Say My Name: a Model's Bias Discovery Framework(https://arxiv.org/abs/2408.09570)
• Keywords: explainability
• Abstract: In the last few years, due to the broad applicability of deep learning to downstream tasks and end-to-end training capabilities, increasingly more concerns about potential biases to specific, non-representative patterns have been raised. Many works focusing on unsupervised debiasing usually leverage the tendency of deep models to learn ``easier'' samples, for example by clustering the latent space to obtain bias pseudo-labels. However, the interpretation of such pseudo-labels is not trivial, especially for a non-expert end user, as it does not provide semantic information about the bias features. To address this issue, we introduce ``Say My Name'' (SaMyNa), the first tool to identify biases within deep models semantically. Unlike existing methods, our approach focuses on biases learned by the model. Our text-based pipeline enhances explainability and supports debiasing efforts: applicable during either training or post-hoc validation, our method can disentangle task-related information and proposes itself as a tool to analyze biases. Evaluation on traditional benchmarks demonstrates its effectiveness in detecting biases and even disclaiming them, showcasing its broad applicability for model diagnosis.

Title: Refining Packing and Shuffling Strategies for Enhanced Performance in Generative Language Models

• Authors: Yanbing Chen, Ruilin Wang, Zihao Yang, Lavender Yao Jiang, Eric Karl Oermann
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09621
• Pdf URL: https://arxiv.org/pdf/2408.09621
• Copy Paste: [[2408.09621]] Refining Packing and Shuffling Strategies for Enhanced Performance in Generative Language Models(https://arxiv.org/abs/2408.09621)
• Keywords: generative
• Abstract: Packing and shuffling tokens is a common practice in training auto-regressive language models (LMs) to prevent overfitting and improve efficiency. Typically documents are concatenated to chunks of maximum sequence length (MSL) and then shuffled. However setting the atom size, the length for each data chunk accompanied by random shuffling, to MSL may lead to contextual incoherence due to tokens from different documents being packed into the same chunk. An alternative approach is to utilize padding, another common data packing strategy, to avoid contextual incoherence by only including one document in each shuffled chunk. To optimize both packing strategies (concatenation vs padding), we investigated the optimal atom size for shuffling and compared their performance and efficiency. We found that matching atom size to MSL optimizes performance for both packing methods (concatenation and padding), and padding yields lower final perplexity (higher performance) than concatenation at the cost of more training steps and lower compute efficiency. This trade-off informs the choice of packing methods in training language models.

Title: Global BGP Attacks that Evade Route Monitoring

• Authors: Henry Birge-Lee, Maria Apostolaki, Jennifer Rexford
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.09622
• Pdf URL: https://arxiv.org/pdf/2408.09622
• Copy Paste: [[2408.09622]] Global BGP Attacks that Evade Route Monitoring(https://arxiv.org/abs/2408.09622)
• Keywords: security, protect, attack, robust
• Abstract: As the deployment of comprehensive Border Gateway Protocol (BGP) security measures is still in progress, BGP monitoring continues to play a critical role in protecting the Internet from routing attacks. Fundamentally, monitoring involves observing BGP feeds to detect suspicious announcements and taking defensive action. However, BGP monitoring relies on seeing the malicious BGP announcement in the first place! In this paper, we develop a novel attack that can hide itself from all state-of-the-art BGP monitoring systems we tested while affecting the entire Internet. The attack involves launching a sub-prefix hijack with the RFC-specified NO_EXPORT community attached to prevent networks with the malicious route installed from sending the route to BGP monitoring systems. We study the viability of this attack at four tier-1 networks and find all networks we studied were vulnerable to the attack. Finally, we propose a mitigation that significantly improves the robustness of the BGP monitoring ecosystem. Our paper aims to raise awareness of this issue and offer guidance to providers to protect against such attacks.

Title: A Strategy to Combine 1stGen Transformers and Open LLMs for Automatic Text Classification

• Authors: Claudio M. V. de Andrade, Washington Cunha, Davi Reis, Adriana Silvina Pagano, Leonardo Rocha, Marcos André Gonçalves
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09629
• Pdf URL: https://arxiv.org/pdf/2408.09629
• Copy Paste: [[2408.09629]] A Strategy to Combine 1stGen Transformers and Open LLMs for Automatic Text Classification(https://arxiv.org/abs/2408.09629)
• Keywords: transformer, large language model
• Abstract: Transformer models have achieved state-of-the-art results, with Large Language Models (LLMs), an evolution of first-generation transformers (1stTR), being considered the cutting edge in several NLP tasks. However, the literature has yet to conclusively demonstrate that LLMs consistently outperform 1stTRs across all NLP tasks. This study compares three 1stTRs (BERT, RoBERTa, and BART) with two open LLMs (Llama 2 and Bloom) across 11 sentiment analysis datasets. The results indicate that open LLMs may moderately outperform or match 1stTRs in 8 out of 11 datasets but only when fine-tuned. Given this substantial cost for only moderate gains, the practical applicability of these models in cost-sensitive scenarios is questionable. In this context, a confidence-based strategy that seamlessly integrates 1stTRs with open LLMs based on prediction certainty is proposed. High-confidence documents are classified by the more cost-effective 1stTRs, while uncertain cases are handled by LLMs in zero-shot or few-shot modes, at a much lower cost than fine-tuned versions. Experiments in sentiment analysis demonstrate that our solution not only outperforms 1stTRs, zero-shot, and few-shot LLMs but also competes closely with fine-tuned LLMs at a fraction of the cost.

Title: MoDeGPT: Modular Decomposition for Large Language Model Compression

• Authors: Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, Yen-Chang Hsu
• Subjects: cs.LG, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.09632
• Pdf URL: https://arxiv.org/pdf/2408.09632
• Copy Paste: [[2408.09632]] MoDeGPT: Modular Decomposition for Large Language Model Compression(https://arxiv.org/abs/2408.09632)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) have reshaped the landscape of artificial intelligence by demonstrating exceptional performance across various tasks. However, substantial computational requirements make their deployment challenging on devices with limited resources. Recently, compression methods using low-rank matrix techniques have shown promise, yet these often lead to degraded accuracy or introduce significant overhead in parameters and inference latency. This paper introduces \textbf{Mo}dular \textbf{De}composition (MoDeGPT), a novel structured compression framework that does not need recovery fine-tuning while resolving the above drawbacks. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions via reconstructing the module-level outputs. MoDeGPT is developed based on a theoretical framework that utilizes three well-established matrix decomposition algorithms -- Nyström approximation, CR decomposition, and SVD -- and applies them to our redefined transformer modules. Our comprehensive experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods that rely on gradient information, and saves 98% of compute costs on compressing a 13B model. On \textsc{Llama}-2/3 and OPT models, MoDeGPT maintains 90-95% zero-shot performance with 25-30% compression rates. Moreover, the compression can be done on a single GPU within a few hours and increases the inference throughput by up to 46%.

Title: Meta-Learning on Augmented Gene Expression Profiles for Enhanced Lung Cancer Detection

• Authors: Arya Hadizadeh Moghaddam, Mohsen Nayebi Kerdabadi, Cuncong Zhong, Zijun Yao
• Subjects: cs.LG, cs.AI, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2408.09635
• Pdf URL: https://arxiv.org/pdf/2408.09635
• Copy Paste: [[2408.09635]] Meta-Learning on Augmented Gene Expression Profiles for Enhanced Lung Cancer Detection(https://arxiv.org/abs/2408.09635)
• Keywords: explainability
• Abstract: Gene expression profiles obtained through DNA microarray have proven successful in providing critical information for cancer detection classifiers. However, the limited number of samples in these datasets poses a challenge to employ complex methodologies such as deep neural networks for sophisticated analysis. To address this "small data" dilemma, Meta-Learning has been introduced as a solution to enhance the optimization of machine learning models by utilizing similar datasets, thereby facilitating a quicker adaptation to target datasets without the requirement of sufficient samples. In this study, we present a meta-learning-based approach for predicting lung cancer from gene expression profiles. We apply this framework to well-established deep learning methodologies and employ four distinct datasets for the meta-learning tasks, where one as the target dataset and the rest as source datasets. Our approach is evaluated against both traditional and deep learning methodologies, and the results show the superior performance of meta-learning on augmented source data compared to the baselines trained on single datasets. Moreover, we conduct the comparative analysis between meta-learning and transfer learning methodologies to highlight the efficiency of the proposed approach in addressing the challenges associated with limited sample sizes. Finally, we incorporate the explainability study to illustrate the distinctiveness of decisions made by meta-learning.

Title: How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments

• Authors: Yusuke Ide, Yuto Nishida, Miyu Oba, Yusuke Sakai, Justin Vasselli, Hidetaka Kamigaito, Taro Watanabe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09639
• Pdf URL: https://arxiv.org/pdf/2408.09639
• Copy Paste: [[2408.09639]] How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments(https://arxiv.org/abs/2408.09639)
• Keywords: robust, large language model
• Abstract: The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where LMs are presented with a pair of acceptable and unacceptable sentences and required to judge which is acceptable. The existing dominant approach, however, naively calculates and compares the probabilities of paired sentences using LMs. Additionally, large language models (LLMs) have yet to be thoroughly examined in this field. We thus investigate how to make the most of LLMs' grammatical knowledge to comprehensively evaluate it. Through extensive experiments of nine judgment methods in English and Chinese, we demonstrate that a probability readout method, in-template LP, and a prompting-based method, Yes/No probability computing, achieve particularly high performance, surpassing the conventional approach. Our analysis reveals their different strengths, e.g., Yes/No probability computing is robust against token-length bias, suggesting that they harness different aspects of LLMs' grammatical knowledge. Consequently, we recommend using diverse judgment methods to evaluate LLMs comprehensively.

Title: ExpoMamba: Exploiting Frequency SSM Blocks for Efficient and Effective Image Enhancement

• Authors: Eashan Adhikarla, Kai Zhang, John Nicholson, Brian D. Davison
• Subjects: cs.CV, cs.AI, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.09650
• Pdf URL: https://arxiv.org/pdf/2408.09650
• Copy Paste: [[2408.09650]] ExpoMamba: Exploiting Frequency SSM Blocks for Efficient and Effective Image Enhancement(https://arxiv.org/abs/2408.09650)
• Keywords: diffusion, transformer
• Abstract: Low-light image enhancement remains a challenging task in computer vision, with existing state-of-the-art models often limited by hardware constraints and computational inefficiencies, particularly in handling high-resolution images. Recent foundation models, such as transformers and diffusion models, despite their efficacy in various domains, are limited in use on edge devices due to their computational complexity and slow inference times. We introduce ExpoMamba, a novel architecture that integrates components of the frequency state space within a modified U-Net, offering a blend of efficiency and effectiveness. This model is specifically optimized to address mixed exposure challenges, a common issue in low-light image enhancement, while ensuring computational efficiency. Our experiments demonstrate that ExpoMamba enhances low-light images up to 2-3x faster than traditional models with an inference time of 36.6 ms and achieves a PSNR improvement of approximately 15-20% over competing models, making it highly suitable for real-time image processing applications.

Title: Regularization for Adversarial Robust Learning

• Authors: Jie Wang, Rui Gao, Yao Xie
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.09672
• Pdf URL: https://arxiv.org/pdf/2408.09672
• Copy Paste: [[2408.09672]] Regularization for Adversarial Robust Learning(https://arxiv.org/abs/2408.09672)
• Keywords: attack, robust
• Abstract: Despite the growing prevalence of artificial neural networks in real-world applications, their vulnerability to adversarial attacks remains to be a significant concern, which motivates us to investigate the robustness of machine learning models. While various heuristics aim to optimize the distributionally robust risk using the $\infty$-Wasserstein metric, such a notion of robustness frequently encounters computation intractability. To tackle the computational challenge, we develop a novel approach to adversarial training that integrates $\phi$-divergence regularization into the distributionally robust risk function. This regularization brings a notable improvement in computation compared with the original formulation. We develop stochastic gradient methods with biased oracles to solve this problem efficiently, achieving the near-optimal sample complexity. Moreover, we establish its regularization effects and demonstrate it is asymptotic equivalence to a regularized empirical risk minimization (ERM) framework, by considering various scaling regimes of the regularization parameter $\eta$ and robustness level $\rho$. These regimes yield gradient norm regularization, variance regularization, or a smoothed gradient norm regularization that interpolates between these extremes. We numerically validate our proposed method in supervised learning, reinforcement learning, and contextual learning and showcase its state-of-the-art performance against various adversarial attacks.

Title: Image-based Freeform Handwriting Authentication with Energy-oriented Self-Supervised Learning

• Authors: Jingyao Wang, Luntian Mou, Changwen Zheng, Wen Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09676
• Pdf URL: https://arxiv.org/pdf/2408.09676
• Copy Paste: [[2408.09676]] Image-based Freeform Handwriting Authentication with Energy-oriented Self-Supervised Learning(https://arxiv.org/abs/2408.09676)
• Keywords: protect, robust
• Abstract: Freeform handwriting authentication verifies a person's identity from their writing style and habits in messy handwriting data. This technique has gained widespread attention in recent years as a valuable tool for various fields, e.g., fraud prevention and cultural heritage protection. However, it still remains a challenging task in reality due to three reasons: (i) severe damage, (ii) complex high-dimensional features, and (iii) lack of supervision. To address these issues, we propose SherlockNet, an energy-oriented two-branch contrastive self-supervised learning framework for robust and fast freeform handwriting authentication. It consists of four stages: (i) pre-processing: converting manuscripts into energy distributions using a novel plug-and-play energy-oriented operator to eliminate the influence of noise; (ii) generalized pre-training: learning general representation through two-branch momentum-based adaptive contrastive learning with the energy distributions, which handles the high-dimensional features and spatial dependencies of handwriting; (iii) personalized fine-tuning: calibrating the learned knowledge using a small amount of labeled data from downstream tasks; and (iv) practical application: identifying individual handwriting from scrambled, missing, or forged data efficiently and conveniently. Considering the practicality, we construct EN-HA, a novel dataset that simulates data forgery and severe damage in real applications. Finally, we conduct extensive experiments on six benchmark datasets including our EN-HA, and the results prove the robustness and efficiency of SherlockNet.

Title: MambaLoc: Efficient Camera Localisation via State Space Model

• Authors: Jialu Wang, Kaichen Zhou, Andrew Markham, Niki Trigoni
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09680
• Pdf URL: https://arxiv.org/pdf/2408.09680
• Copy Paste: [[2408.09680]] MambaLoc: Efficient Camera Localisation via State Space Model(https://arxiv.org/abs/2408.09680)
• Keywords: robust, extraction
• Abstract: Location information is pivotal for the automation and intelligence of terminal devices and edge-cloud IoT systems, such as autonomous vehicles and augmented reality. However, achieving reliable positioning across diverse IoT applications remains challenging due to significant training costs and the necessity of densely collected data. To tackle these issues, we have innovatively applied the selective state space (SSM) model to visual localization, introducing a new model named MambaLoc. The proposed model demonstrates exceptional training efficiency by capitalizing on the SSM model's strengths in efficient feature extraction, rapid computation, and memory optimization, and it further ensures robustness in sparse data environments due to its parameter sparsity. Additionally, we propose the Global Information Selector (GIS), which leverages selective SSM to implicitly achieve the efficient global feature extraction capabilities of Non-local Neural Networks. This design leverages the computational efficiency of the SSM model alongside the Non-local Neural Networks' capacity to capture long-range dependencies with minimal layers. Consequently, the GIS enables effective global information capture while significantly accelerating convergence. Our extensive experimental validation using public indoor and outdoor datasets first demonstrates our model's effectiveness, followed by evidence of its versatility with various existing localization models.

Title: Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcripts

• Authors: Jiaqing Liu, Chong Deng, Qinglin Zhang, Qian Chen, Hai Yu, Wen Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09688
• Pdf URL: https://arxiv.org/pdf/2408.09688
• Copy Paste: [[2408.09688]] Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcripts(https://arxiv.org/abs/2408.09688)
• Keywords: large language model
• Abstract: Automatic Speech Recognition (ASR) transcripts exhibit recognition errors and various spoken language phenomena such as disfluencies, ungrammatical sentences, and incomplete sentences, hence suffering from poor readability. To improve readability, we propose a Contextualized Spoken-to-Written conversion (CoS2W) task to address ASR and grammar errors and also transfer the informal text into the formal style with content preserved, utilizing contexts and auxiliary information. This task naturally matches the in-context learning capabilities of Large Language Models (LLMs). To facilitate comprehensive comparisons of various LLMs, we construct a document-level Spoken-to-Written conversion of ASR Transcripts Benchmark (SWAB) dataset. Using SWAB, we study the impact of different granularity levels on the CoS2W performance, and propose methods to exploit contexts and auxiliary information to enhance the outputs. Experimental results reveal that LLMs have the potential to excel in the CoS2W task, particularly in grammaticality and formality, our methods achieve effective understanding of contexts and auxiliary information by LLMs. We further investigate the effectiveness of using LLMs as evaluators and find that LLM evaluators show strong correlations with human evaluations on rankings of faithfulness and formality, which validates the reliability of LLM evaluators for the CoS2W task.

Title: LightWeather: Harnessing Absolute Positional Encoding to Efficient and Scalable Global Weather Forecasting

• Authors: Yisong Fu, Fei Wang, Zezhi Shao, Chengqing Yu, Yujie Li, Zhao Chen, Zhulin An, Yongjun Xu
• Subjects: cs.LG, cs.AI, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2408.09695
• Pdf URL: https://arxiv.org/pdf/2408.09695
• Copy Paste: [[2408.09695]] LightWeather: Harnessing Absolute Positional Encoding to Efficient and Scalable Global Weather Forecasting(https://arxiv.org/abs/2408.09695)
• Keywords: transformer
• Abstract: Recently, Transformers have gained traction in weather forecasting for their capability to capture long-term spatial-temporal correlations. However, their complex architectures result in large parameter counts and extended training times, limiting their practical application and scalability to global-scale forecasting. This paper aims to explore the key factor for accurate weather forecasting and design more efficient solutions. Interestingly, our empirical findings reveal that absolute positional encoding is what really works in Transformer-based weather forecasting models, which can explicitly model the spatial-temporal correlations even without attention mechanisms. We theoretically prove that its effectiveness stems from the integration of geographical coordinates and real-world time features, which are intrinsically related to the dynamics of weather. Based on this, we propose LightWeather, a lightweight and effective model for station-based global weather forecasting. We employ absolute positional encoding and a simple MLP in place of other components of Transformer. With under 30k parameters and less than one hour of training time, LightWeather achieves state-of-the-art performance on global weather datasets compared to other advanced DL methods. The results underscore the superiority of integrating spatial-temporal knowledge over complex architectures, providing novel insights for DL in weather forecasting.

Title: Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer

• Authors: Mingda Li, Abhijit Mishra, Utkarsh Mujumdar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09701
• Pdf URL: https://arxiv.org/pdf/2408.09701
• Copy Paste: [[2408.09701]] Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer(https://arxiv.org/abs/2408.09701)
• Keywords: large language model
• Abstract: The use of Large Language Models (LLMs) for program code generation has gained substantial attention, but their biases and limitations with non-English prompts challenge global inclusivity. This paper investigates the complexities of multilingual prompt-based code generation. Our evaluations of LLMs, including CodeLLaMa and CodeGemma, reveal significant disparities in code quality for non-English prompts; we also demonstrate the inadequacy of simple approaches like prompt translation, bootstrapped data augmentation, and fine-tuning. To address this, we propose a zero-shot cross-lingual approach using a neural projection technique, integrating a cross-lingual encoder like LASER artetxe2019massively to map multilingual embeddings from it into the LLM's token space. This method requires training only on English data and scales effectively to other languages. Results on a translated and quality-checked MBPP dataset show substantial improvements in code quality. This research promotes a more inclusive code generation landscape by empowering LLMs with multilingual capabilities to support the diverse linguistic spectrum in programming.

Title: Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering

• Authors: Ruofan Liang, Zan Gojcic, Merlin Nimier-David, David Acuna, Nandita Vijaykumar, Sanja Fidler, Zian Wang
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.09702
• Pdf URL: https://arxiv.org/pdf/2408.09702
• Copy Paste: [[2408.09702]] Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering(https://arxiv.org/abs/2408.09702)
• Keywords: diffusion, generative
• Abstract: The correct insertion of virtual objects in images of real-world scenes requires a deep understanding of the scene's lighting, geometry and materials, as well as the image formation process. While recent large-scale diffusion models have shown strong generative and inpainting capabilities, we find that current models do not sufficiently "understand" the scene shown in a single picture to generate consistent lighting effects (shadows, bright reflections, etc.) while preserving the identity and details of the composited object. We propose using a personalized large diffusion model as guidance to a physically based inverse rendering process. Our method recovers scene lighting and tone-mapping parameters, allowing the photorealistic composition of arbitrary virtual objects in single frames or videos of indoor or outdoor scenes. Our physically based pipeline further enables automatic materials and tone-mapping refinement.

Title: Community-Centric Graph Unlearning

• Authors: Yi Li, Shichao Zhang, Guixian Zhang, Debo Cheng
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2408.09705
• Pdf URL: https://arxiv.org/pdf/2408.09705
• Copy Paste: [[2408.09705]] Community-Centric Graph Unlearning(https://arxiv.org/abs/2408.09705)
• Keywords: security, privacy
• Abstract: Graph unlearning technology has become increasingly important since the advent of the `right to be forgotten' and the growing concerns about the privacy and security of artificial intelligence. Graph unlearning aims to quickly eliminate the effects of specific data on graph neural networks (GNNs). However, most existing deterministic graph unlearning frameworks follow a balanced partition-submodel training-aggregation paradigm, resulting in a lack of structural information between subgraph neighborhoods and redundant unlearning parameter calculations. To address this issue, we propose a novel Graph Structure Mapping Unlearning paradigm (GSMU) and a novel method based on it named Community-centric Graph Eraser (CGE). CGE maps community subgraphs to nodes, thereby enabling the reconstruction of a node-level unlearning operation within a reduced mapped graph. CGE makes the exponential reduction of both the amount of training data and the number of unlearning parameters. Extensive experiments conducted on five real-world datasets and three widely used GNN backbones have verified the high performance and efficiency of our CGE method, highlighting its potential in the field of graph unlearning.

Title: MePT: Multi-Representation Guided Prompt Tuning for Vision-Language Model

• Authors: Xinyang Wang, Yi Yang, Minfeng Zhu, Kecheng Zheng, Shi Liu, Wei Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09706
• Pdf URL: https://arxiv.org/pdf/2408.09706
• Copy Paste: [[2408.09706]] MePT: Multi-Representation Guided Prompt Tuning for Vision-Language Model(https://arxiv.org/abs/2408.09706)
• Keywords: robust
• Abstract: Recent advancements in pre-trained Vision-Language Models (VLMs) have highlighted the significant potential of prompt tuning for adapting these models to a wide range of downstream tasks. However, existing prompt tuning methods typically map an image to a single representation, limiting the model's ability to capture the diverse ways an image can be described. To address this limitation, we investigate the impact of visual prompts on the model's generalization capability and introduce a novel method termed Multi-Representation Guided Prompt Tuning (MePT). Specifically, MePT employs a three-branch framework that focuses on diverse salient regions, uncovering the inherent knowledge within images which is crucial for robust generalization. Further, we employ efficient self-ensemble techniques to integrate these versatile image representations, allowing MePT to learn all conditional, marginal, and fine-grained distributions effectively. We validate the effectiveness of MePT through extensive experiments, demonstrating significant improvements on both base-to-novel class prediction and domain generalization tasks.

Title: Pedestrian Attribute Recognition: A New Benchmark Dataset and A Large Language Model Augmented Framework

• Authors: Jiandong Jin, Xiao Wang, Qian Zhu, Haiyang Wang, Chenglong Li
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09720
• Pdf URL: https://arxiv.org/pdf/2408.09720
• Copy Paste: [[2408.09720]] Pedestrian Attribute Recognition: A New Benchmark Dataset and A Large Language Model Augmented Framework(https://arxiv.org/abs/2408.09720)
• Keywords: transformer, large language model
• Abstract: Pedestrian Attribute Recognition (PAR) is one of the indispensable tasks in human-centered research. However, existing datasets neglect different domains (e.g., environments, times, populations, and data sources), only conducting simple random splits, and the performance of these datasets has already approached saturation. In the past five years, no large-scale dataset has been opened to the public. To address this issue, this paper proposes a new large-scale, cross-domain pedestrian attribute recognition dataset to fill the data gap, termed MSP60K. It consists of 60,122 images and 57 attribute annotations across eight scenarios. Synthetic degradation is also conducted to further narrow the gap between the dataset and real-world challenging scenarios. To establish a more rigorous benchmark, we evaluate 17 representative PAR models under both random and cross-domain split protocols on our dataset. Additionally, we propose an innovative Large Language Model (LLM) augmented PAR framework, named LLM-PAR. This framework processes pedestrian images through a Vision Transformer (ViT) backbone to extract features and introduces a multi-embedding query Transformer to learn partial-aware features for attribute classification. Significantly, we enhance this framework with LLM for ensemble learning and visual feature augmentation. Comprehensive experiments across multiple PAR benchmark datasets have thoroughly validated the efficacy of our proposed framework. The dataset and source code accompanying this paper will be made publicly available at \url{this https URL}.

Title: sTransformer: A Modular Approach for Extracting Inter-Sequential and Temporal Information for Time-Series Forecasting

• Authors: Jiaheng Yin, Zhengxin Shi, Jianshen Zhang, Xiaomin Lin, Yulin Huang, Yongzhi Qi, Wei Qi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09723
• Pdf URL: https://arxiv.org/pdf/2408.09723
• Copy Paste: [[2408.09723]] sTransformer: A Modular Approach for Extracting Inter-Sequential and Temporal Information for Time-Series Forecasting(https://arxiv.org/abs/2408.09723)
• Keywords: transformer
• Abstract: In recent years, numerous Transformer-based models have been applied to long-term time-series forecasting (LTSF) tasks. However, recent studies with linear models have questioned their effectiveness, demonstrating that simple linear layers can outperform sophisticated Transformer-based models. In this work, we review and categorize existing Transformer-based models into two main types: (1) modifications to the model structure and (2) modifications to the input data. The former offers scalability but falls short in capturing inter-sequential information, while the latter preprocesses time-series data but is challenging to use as a scalable module. We propose $\textbf{sTransformer}$, which introduces the Sequence and Temporal Convolutional Network (STCN) to fully capture both sequential and temporal information. Additionally, we introduce a Sequence-guided Mask Attention mechanism to capture global feature information. Our approach ensures the capture of inter-sequential information while maintaining module scalability. We compare our model with linear models and existing forecasting models on long-term time-series forecasting, achieving new state-of-the-art results. We also conducted experiments on other time-series tasks, achieving strong performance. These demonstrate that Transformer-based structures remain effective and our model can serve as a viable baseline for time-series tasks.

Title: Mutually-Aware Feature Learning for Few-Shot Object Counting

• Authors: Yerim Jeon, Subeen Lee, Jihwan Kim, Jae-Pil Heo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09734
• Pdf URL: https://arxiv.org/pdf/2408.09734
• Copy Paste: [[2408.09734]] Mutually-Aware Feature Learning for Few-Shot Object Counting(https://arxiv.org/abs/2408.09734)
• Keywords: robust, extraction
• Abstract: Few-shot object counting has garnered significant attention for its practicality as it aims to count target objects in a query image based on given exemplars without the need for additional training. However, there is a shortcoming in the prevailing extract-and-match approach: query and exemplar features lack interaction during feature extraction since they are extracted unaware of each other and later correlated based on similarity. This can lead to insufficient target awareness of the extracted features, resulting in target confusion in precisely identifying the actual target when multiple class objects coexist. To address this limitation, we propose a novel framework, Mutually-Aware FEAture learning(MAFEA), which encodes query and exemplar features mutually aware of each other from the outset. By encouraging interaction between query and exemplar features throughout the entire pipeline, we can obtain target-aware features that are robust to a multi-category scenario. Furthermore, we introduce a background token to effectively associate the target region of query with exemplars and decouple its background region from them. Our extensive experiments demonstrate that our model reaches a new state-of-the-art performance on the two challenging benchmarks, FSCD-LVIS and FSC-147, with a remarkably reduced degree of the target confusion problem.

Title: TraDiffusion: Trajectory-Based Training-Free Image Generation

• Authors: Mingrui Wu, Oucheng Huang, Jiayi Ji, Jiale Li, Xinyue Cai, Huafeng Kuang, Jianzhuang Liu, Xiaoshuai Sun, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09739
• Pdf URL: https://arxiv.org/pdf/2408.09739
• Copy Paste: [[2408.09739]] TraDiffusion: Trajectory-Based Training-Free Image Generation(https://arxiv.org/abs/2408.09739)
• Keywords: diffusion
• Abstract: In this work, we propose a training-free, trajectory-based controllable T2I approach, termed TraDiffusion. This novel method allows users to effortlessly guide image generation via mouse trajectories. To achieve precise control, we design a distance awareness energy function to effectively guide latent variables, ensuring that the focus of generation is within the areas defined by the trajectory. The energy function encompasses a control function to draw the generation closer to the specified trajectory and a movement function to diminish activity in areas distant from the trajectory. Through extensive experiments and qualitative assessments on the COCO dataset, the results reveal that TraDiffusion facilitates simpler, more natural image control. Moreover, it showcases the ability to manipulate salient regions, attributes, and relationships within the generated images, alongside visual input based on arbitrary or enhanced trajectories.

Title: Paired Completion: Flexible Quantification of Issue-framing at Scale with LLMs

• Authors: Simon D Angus, Lachlan O'Neill
• Subjects: cs.CL, cs.AI, econ.GN
• Abstract URL: https://arxiv.org/abs/2408.09742
• Pdf URL: https://arxiv.org/pdf/2408.09742
• Copy Paste: [[2408.09742]] Paired Completion: Flexible Quantification of Issue-framing at Scale with LLMs(https://arxiv.org/abs/2408.09742)
• Keywords: generative, large language model
• Abstract: Detecting and quantifying issue framing in textual discourse - the perspective one takes to a given topic (e.g. climate science vs. denialism, misogyny vs. gender equality) - is highly valuable to a range of end-users from social and political scientists to program evaluators and policy analysts. However, conceptual framing is notoriously challenging for automated natural language processing (NLP) methods since the words and phrases used by either `side' of an issue are often held in common, with only subtle stylistic flourishes separating their use. Here we develop and rigorously evaluate new detection methods for issue framing and narrative analysis within large text datasets. By introducing a novel application of next-token log probabilities derived from generative large language models (LLMs) we show that issue framing can be reliably and efficiently detected in large corpora with only a few examples of either perspective on a given issue, a method we call `paired completion'. Through 192 independent experiments over three novel, synthetic datasets, we evaluate paired completion against prompt-based LLM methods and labelled methods using traditional NLP and recent LLM contextual embeddings. We additionally conduct a cost-based analysis to mark out the feasible set of performant methods at production-level scales, and a model bias analysis. Together, our work demonstrates a feasible path to scalable, accurate and low-bias issue-framing in large corpora.

Title: R2GenCSR: Retrieving Context Samples for Large Language Model based X-ray Medical Report Generation

• Authors: Xiao Wang, Yuehang Li, Fuling Wang, Shiao Wang, Chuanfu Li, Bo Jiang
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09743
• Pdf URL: https://arxiv.org/pdf/2408.09743
• Copy Paste: [[2408.09743]] R2GenCSR: Retrieving Context Samples for Large Language Model based X-ray Medical Report Generation(https://arxiv.org/abs/2408.09743)
• Keywords: transformer, large language model
• Abstract: Inspired by the tremendous success of Large Language Models (LLMs), existing X-ray medical report generation methods attempt to leverage large models to achieve better performance. They usually adopt a Transformer to extract the visual features of a given X-ray image, and then, feed them into the LLM for text generation. How to extract more effective information for the LLMs to help them improve final results is an urgent problem that needs to be solved. Additionally, the use of visual Transformer models also brings high computational complexity. To address these issues, this paper proposes a novel context-guided efficient X-ray medical report generation framework. Specifically, we introduce the Mamba as the vision backbone with linear complexity, and the performance obtained is comparable to that of the strong Transformer model. More importantly, we perform context retrieval from the training set for samples within each mini-batch during the training phase, utilizing both positively and negatively related samples to enhance feature representation and discriminative learning. Subsequently, we feed the vision tokens, context information, and prompt statements to invoke the LLM for generating high-quality medical reports. Extensive experiments on three X-ray report generation datasets (i.e., IU-Xray, MIMIC-CXR, CheXpert Plus) fully validated the effectiveness of our proposed model. The source code of this work will be released on \url{this https URL}.

Title: RealCustom++: Representing Images as Real-Word for Real-Time Customization

• Authors: Zhendong Mao, Mengqi Huang, Fei Ding, Mingcong Liu, Qian He, Xiaojun Chang, Yongdong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09744
• Pdf URL: https://arxiv.org/pdf/2408.09744
• Copy Paste: [[2408.09744]] RealCustom++: Representing Images as Real-Word for Real-Time Customization(https://arxiv.org/abs/2408.09744)
• Keywords: robust
• Abstract: Text-to-image customization, which takes given texts and images depicting given subjects as inputs, aims to synthesize new images that align with both text semantics and subject appearance. This task provides precise control over details that text alone cannot capture and is fundamental for various real-world applications, garnering significant interest from academia and industry. Existing works follow the pseudo-word paradigm, which involves representing given subjects as pseudo-words and combining them with given texts to collectively guide the generation. However, the inherent conflict and entanglement between the pseudo-words and texts result in a dual-optimum paradox, where subject similarity and text controllability cannot be optimal simultaneously. We propose a novel real-words paradigm termed RealCustom++ that instead represents subjects as non-conflict real words, thereby disentangling subject similarity from text controllability and allowing both to be optimized simultaneously. Specifically, RealCustom++ introduces a novel "train-inference" decoupled framework: (1) During training, RealCustom++ learns the alignment between vision conditions and all real words in the text, ensuring high subject-similarity generation in open domains. This is achieved by the cross-layer cross-scale projector to robustly and finely extract subject features, and a curriculum training recipe that adapts the generated subject to diverse poses and sizes. (2) During inference, leveraging the learned general alignment, an adaptive mask guidance is proposed to only customize the generation of the specific target real word, keeping other subject-irrelevant regions uncontaminated to ensure high text-controllability in real-time.

Title: Enhanced Cascade Prostate Cancer Classifier in mp-MRI Utilizing Recall Feedback Adaptive Loss and Prior Knowledge-Based Feature Extraction

• Authors: Kun Luo, Bowen Zheng, Shidong Lv, Jie Tao, Qiang Wei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09746
• Pdf URL: https://arxiv.org/pdf/2408.09746
• Copy Paste: [[2408.09746]] Enhanced Cascade Prostate Cancer Classifier in mp-MRI Utilizing Recall Feedback Adaptive Loss and Prior Knowledge-Based Feature Extraction(https://arxiv.org/abs/2408.09746)
• Keywords: extraction, interpretability
• Abstract: Prostate cancer is the second most common cancer in males worldwide, and mpMRI is commonly used for diagnosis. However, interpreting mpMRI is challenging and requires expertise from radiologists. This highlights the urgent need for automated grading in mpMRI. Existing studies lack integration of clinical prior information and suffer from uneven training sample distribution due to prevalence. Therefore, we propose a solution that incorporates prior knowledge, addresses the issue of uneven medical sample distribution, and maintains high interpretability in mpMRI. Firstly, we introduce Prior Knowledge-Based Feature Extraction, which mathematically models the PI-RADS criteria for prostate cancer as diagnostic information into model training. Secondly, we propose Adaptive Recall Feedback Loss to address the extremely imbalanced data problem. This method adjusts the training dynamically based on accuracy and recall in the validation set, resulting in high accuracy and recall simultaneously in the testing set.Thirdly, we design an Enhanced Cascade Prostate Cancer Classifier that classifies prostate cancer into different levels in an interpretable way, which refines the classification results and helps with clinical intervention. Our method is validated through experiments on the PI-CAI dataset and outperforms other methods with a more balanced result in both accuracy and recall rate.

Title: A Unified Framework for Iris Anti-Spoofing: Introducing IrisGeneral Dataset and Masked-MoE Method

• Authors: Hang Zou, Chenxi Du, Ajian Liu, Yuan Zhang, Jing Liu, Mingchuan Yang, Jun Wan, Hui Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09752
• Pdf URL: https://arxiv.org/pdf/2408.09752
• Copy Paste: [[2408.09752]] A Unified Framework for Iris Anti-Spoofing: Introducing IrisGeneral Dataset and Masked-MoE Method(https://arxiv.org/abs/2408.09752)
• Keywords: security
• Abstract: Iris recognition is widely used in high-security scenarios due to its stability and distinctiveness. However, the acquisition of iris images typically requires near-infrared illumination and near-infrared band filters, leading to significant and consistent differences in imaging across devices. This underscores the importance of developing cross-domain capabilities in iris anti-spoofing methods. Despite this need, there is no dataset available that comprehensively evaluates the generalization ability of the iris anti-spoofing task. To address this gap, we propose the IrisGeneral dataset, which includes 10 subsets, belonging to 7 databases, published by 4 institutions, collected with 6 types of devices. IrisGeneral is designed with three protocols, aimed at evaluating average performance, cross-racial generalization, and cross-device generalization of iris anti-spoofing models. To tackle the challenge of integrating multiple sub-datasets in IrisGeneral, we employ multiple parameter sets to learn from the various subsets. Specifically, we utilize the Mixture of Experts (MoE) to fit complex data distributions using multiple sub-neural networks. To further enhance the generalization capabilities, we introduce a novel method Masked-MoE (MMoE). It randomly masks a portion of tokens for some experts and requires their outputs to be similar to the unmasked experts, which improves the generalization ability and effectively mitigates the overfitting issue produced by MoE. We selected ResNet50, VIT-B/16, CLIP, and FLIP as representative models and benchmarked them on the IrisGeneral dataset. Experimental results demonstrate that our proposed MMoE with CLIP achieves the best performance on IrisGeneral.

Title: Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning

• Authors: Jingyu Hu, Weiru Liu, Mengnan Du
• Subjects: cs.LG, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.09757
• Pdf URL: https://arxiv.org/pdf/2408.09757
• Copy Paste: [[2408.09757]] Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning(https://arxiv.org/abs/2408.09757)
• Keywords: fair, large language model
• Abstract: Recent studies highlight the effectiveness of using in-context learning (ICL) to steer large language models (LLMs) in processing tabular data, a challenging task given the structured nature of such data. Despite advancements in performance, the fairness implications of these methods are less understood. This study investigates how varying demonstrations within ICL prompts influence the fairness outcomes of LLMs. Our findings reveal that deliberately including minority group samples in prompts significantly boosts fairness without sacrificing predictive accuracy. Further experiments demonstrate that the proportion of minority to majority samples in demonstrations affects the trade-off between fairness and prediction accuracy. Based on these insights, we introduce a mitigation technique that employs clustering and evolutionary strategies to curate a diverse and representative sample set from the training data. This approach aims to enhance both predictive performance and fairness in ICL applications. Experimental results validate that our proposed method dramatically improves fairness across various metrics, showing its efficacy in real-world scenarios.

Title: Sequential Federated Learning in Hierarchical Architecture on Non-IID Datasets

• Authors: Xingrun Yan, Shiyuan Zuo, Rongfei Fan, Han Hu, Li Shen, Puning Zhao, Yong Luo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09762
• Pdf URL: https://arxiv.org/pdf/2408.09762
• Copy Paste: [[2408.09762]] Sequential Federated Learning in Hierarchical Architecture on Non-IID Datasets(https://arxiv.org/abs/2408.09762)
• Keywords: federate
• Abstract: In a real federated learning (FL) system, communication overhead for passing model parameters between the clients and the parameter server (PS) is often a bottleneck. Hierarchical federated learning (HFL) that poses multiple edge servers (ESs) between clients and the PS can partially alleviate communication pressure but still needs the aggregation of model parameters from multiple ESs at the PS. To further reduce communication overhead, we bring sequential FL (SFL) into HFL for the first time, which removes the central PS and enables the model training to be completed only through passing the global model between two adjacent ESs for each iteration, and propose a novel algorithm adaptive to such a combinational framework, referred to as Fed-CHS. Convergence results are derived for strongly convex and non-convex loss functions under various data heterogeneity setups, which show comparable convergence performance with the algorithms for HFL or SFL solely. Experimental results provide evidence of the superiority of our proposed Fed-CHS on both communication overhead saving and test accuracy over baseline methods.

Title: Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms

• Authors: Xiao Wang, Shiao Wang, Pengpeng Shao, Bo Jiang, Lin Zhu, Yonghong Tian
• Subjects: cs.CV, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2408.09764
• Pdf URL: https://arxiv.org/pdf/2408.09764
• Copy Paste: [[2408.09764]] Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms(https://arxiv.org/abs/2408.09764)
• Keywords: privacy
• Abstract: Human Action Recognition (HAR) stands as a pivotal research domain in both computer vision and artificial intelligence, with RGB cameras dominating as the preferred tool for investigation and innovation in this field. However, in real-world applications, RGB cameras encounter numerous challenges, including light conditions, fast motion, and privacy concerns. Consequently, bio-inspired event cameras have garnered increasing attention due to their advantages of low energy consumption, high dynamic range, etc. Nevertheless, most existing event-based HAR datasets are low resolution ($346 \times 260$). In this paper, we propose a large-scale, high-definition ($1280 \times 800$) human action recognition dataset based on the CeleX-V event camera, termed CeleX-HAR. It encompasses 150 commonly occurring action categories, comprising a total of 124,625 video sequences. Various factors such as multi-view, illumination, action speed, and occlusion are considered when recording these data. To build a more comprehensive benchmark dataset, we report over 20 mainstream HAR models for future works to compare. In addition, we also propose a novel Mamba vision backbone network for event stream based HAR, termed EVMamba, which equips the spatial plane multi-directional scanning and novel voxel temporal scanning mechanism. By encoding and mining the spatio-temporal information of event streams, our EVMamba has achieved favorable results across multiple datasets. Both the dataset and source code will be released on \url{this https URL}

Title: Baby Bear: Seeking a Just Right Rating Scale for Scalar Annotations

• Authors: Xu Han, Felix Yu, Joao Sedoc, Benjamin Van Durme
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.09765
• Pdf URL: https://arxiv.org/pdf/2408.09765
• Copy Paste: [[2408.09765]] Baby Bear: Seeking a Just Right Rating Scale for Scalar Annotations(https://arxiv.org/abs/2408.09765)
• Keywords: robust
• Abstract: Our goal is a mechanism for efficiently assigning scalar ratings to each of a large set of elements. For example, "what percent positive or negative is this product review?" When sample sizes are small, prior work has advocated for methods such as Best Worst Scaling (BWS) as being more robust than direct ordinal annotation ("Likert scales"). Here we first introduce IBWS, which iteratively collects annotations through Best-Worst Scaling, resulting in robustly ranked crowd-sourced data. While effective, IBWS is too expensive for large-scale tasks. Using the results of IBWS as a best-desired outcome, we evaluate various direct assessment methods to determine what is both cost-efficient and best correlating to a large scale BWS annotation strategy. Finally, we illustrate in the domains of dialogue and sentiment how these annotations can support robust learning-to-rank models.

Title: Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?

• Authors: Shiyu Ni, Keping Bi, Lulu Yu, Jiafeng Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09773
• Pdf URL: https://arxiv.org/pdf/2408.09773
• Copy Paste: [[2408.09773]] Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?(https://arxiv.org/abs/2408.09773)
• Keywords: large language model
• Abstract: Large language models (LLMs) have been found to produce hallucinations when the question exceeds their internal knowledge boundaries. A reliable model should have a clear perception of its knowledge boundaries, providing correct answers within its scope and refusing to answer when it lacks knowledge. Existing research on LLMs' perception of their knowledge boundaries typically uses either the probability of the generated tokens or the verbalized confidence as the model's confidence in its response. However, these studies overlook the differences and connections between the two. In this paper, we conduct a comprehensive analysis and comparison of LLMs' probabilistic perception and verbalized perception of their factual knowledge boundaries. First, we investigate the pros and cons of these two perceptions. Then, we study how they change under questions of varying frequencies. Finally, we measure the correlation between LLMs' probabilistic confidence and verbalized confidence. Experimental results show that 1) LLMs' probabilistic perception is generally more accurate than verbalized perception but requires an in-domain validation set to adjust the confidence threshold. 2) Both perceptions perform better on less frequent questions. 3) It is challenging for LLMs to accurately express their internal confidence in natural language.

Title: Faster Adaptive Decentralized Learning Algorithms

• Authors: Feihu Huang, Jianyu Zhao
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2408.09775
• Pdf URL: https://arxiv.org/pdf/2408.09775
• Copy Paste: [[2408.09775]] Faster Adaptive Decentralized Learning Algorithms(https://arxiv.org/abs/2408.09775)
• Keywords: privacy, robust
• Abstract: Decentralized learning recently has received increasing attention in machine learning due to its advantages in implementation simplicity and system robustness, data privacy. Meanwhile, the adaptive gradient methods show superior performances in many machine learning tasks such as training neural networks. Although some works focus on studying decentralized optimization algorithms with adaptive learning rates, these adaptive decentralized algorithms still suffer from high sample complexity. To fill these gaps, we propose a class of faster adaptive decentralized algorithms (i.e., AdaMDOS and AdaMDOF) for distributed nonconvex stochastic and finite-sum optimization, respectively. Moreover, we provide a solid convergence analysis framework for our methods. In particular, we prove that our AdaMDOS obtains a near-optimal sample complexity of $\tilde{O}(\epsilon^{-3})$ for finding an $\epsilon$-stationary solution of nonconvex stochastic optimization. Meanwhile, our AdaMDOF obtains a near-optimal sample complexity of $O(\sqrt{n}\epsilon^{-2})$ for finding an $\epsilon$-stationary solution of nonconvex finite-sum optimization, where $n$ denotes the sample size. To the best of our knowledge, our AdaMDOF algorithm is the first adaptive decentralized algorithm for nonconvex finite-sum optimization. Some experimental results demonstrate efficiency of our algorithms.

Title: Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation

• Authors: Yunxin Li, Haoyuan Shi, Baotian Hu, Longyue Wang, Jiashun Zhu, Jinyi Xu, Zhen Zhao, Min Zhang
• Subjects: cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.09787
• Pdf URL: https://arxiv.org/pdf/2408.09787
• Copy Paste: [[2408.09787]] Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation(https://arxiv.org/abs/2408.09787)
• Keywords: generative
• Abstract: Traditional animation generation methods depend on training generative models with human-labelled data, entailing a sophisticated multi-stage pipeline that demands substantial human effort and incurs high training costs. Due to limited prompting plans, these methods typically produce brief, information-poor, and context-incoherent animations. To overcome these limitations and automate the animation process, we pioneer the introduction of large multimodal models (LMMs) as the core processor to build an autonomous animation-making agent, named Anim-Director. This agent mainly harnesses the advanced understanding and reasoning capabilities of LMMs and generative AI tools to create animated videos from concise narratives or simple instructions. Specifically, it operates in three main stages: Firstly, the Anim-Director generates a coherent storyline from user inputs, followed by a detailed director's script that encompasses settings of character profiles and interior/exterior descriptions, and context-coherent scene descriptions that include appearing characters, interiors or exteriors, and scene events. Secondly, we employ LMMs with the image generation tool to produce visual images of settings and scenes. These images are designed to maintain visual consistency across different scenes using a visual-language prompting method that combines scene descriptions and images of the appearing character and setting. Thirdly, scene images serve as the foundation for producing animated videos, with LMMs generating prompts to guide this process. The whole process is notably autonomous without manual intervention, as the LMMs interact seamlessly with generative tools to generate prompts, evaluate visual quality, and select the best one to optimize the final output.

Title: Structure-enhanced Contrastive Learning for Graph Clustering

• Authors: Xunlian Wu, Jingqi Hu, Anqi Zhang, Yining Quan, Qiguang Miao, Peng Gang Sun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09790
• Pdf URL: https://arxiv.org/pdf/2408.09790
• Copy Paste: [[2408.09790]] Structure-enhanced Contrastive Learning for Graph Clustering(https://arxiv.org/abs/2408.09790)
• Keywords: robust
• Abstract: Graph clustering is a crucial task in network analysis with widespread applications, focusing on partitioning nodes into distinct groups with stronger intra-group connections than inter-group ones. Recently, contrastive learning has achieved significant progress in graph clustering. However, most methods suffer from the following issues: 1) an over-reliance on meticulously designed data augmentation strategies, which can undermine the potential of contrastive learning. 2) overlooking cluster-oriented structural information, particularly the higher-order cluster(community) structure information, which could unveil the mesoscopic cluster structure information of the network. In this study, Structure-enhanced Contrastive Learning (SECL) is introduced to addresses these issues by leveraging inherent network structures. SECL utilizes a cross-view contrastive learning mechanism to enhance node embeddings without elaborate data augmentations, a structural contrastive learning module for ensuring structural consistency, and a modularity maximization strategy for harnessing clustering-oriented information. This comprehensive approach results in robust node representations that greatly enhance clustering performance. Extensive experiments on six datasets confirm SECL's superiority over current state-of-the-art methods, indicating a substantial improvement in the domain of graph clustering.

Title: Unsupervised Composable Representations for Audio

• Authors: Giovanni Bindi, Philippe Esling
• Subjects: cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.09792
• Pdf URL: https://arxiv.org/pdf/2408.09792
• Copy Paste: [[2408.09792]] Unsupervised Composable Representations for Audio(https://arxiv.org/abs/2408.09792)
• Keywords: diffusion, generative
• Abstract: Current generative models are able to generate high-quality artefacts but have been shown to struggle with compositional reasoning, which can be defined as the ability to generate complex structures from simpler elements. In this paper, we focus on the problem of compositional representation learning for music data, specifically targeting the fully-unsupervised setting. We propose a simple and extensible framework that leverages an explicit compositional inductive bias, defined by a flexible auto-encoding objective that can leverage any of the current state-of-art generative models. We demonstrate that our framework, used with diffusion models, naturally addresses the task of unsupervised audio source separation, showing that our model is able to perform high-quality separation. Our findings reveal that our proposal achieves comparable or superior performance with respect to other blind source separation methods and, furthermore, it even surpasses current state-of-art supervised baselines on signal-to-interference ratio metrics. Additionally, by learning an a-posteriori masking diffusion model in the space of composable representations, we achieve a system capable of seamlessly performing unsupervised source separation, unconditional generation, and variation generation. Finally, as our proposal works in the latent space of pre-trained neural audio codecs, it also provides a lower computational cost with respect to other neural baselines.

Title: Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting

• Authors: Yun-Da Tsai, Ting-Yu Yen, Keng-Te Liao, Shou-De Lin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09798
• Pdf URL: https://arxiv.org/pdf/2408.09798
• Copy Paste: [[2408.09798]] Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting(https://arxiv.org/abs/2408.09798)
• Keywords: robust, large language model
• Abstract: Converting different modalities into generalized text, which then serves as input prompts for large language models (LLMs), is a common approach for aligning multimodal models, particularly when pairwise data is limited. Text-centric alignment method leverages the unique properties of text as a modality space, transforming diverse inputs into a unified textual representation, thereby enabling downstream models to effectively interpret various modal inputs. This study evaluates the quality and robustness of multimodal representations in the face of noise imperfections, dynamic input order permutations, and missing modalities, revealing that current text-centric alignment methods can compromise downstream robustness. To address this issue, we propose a new text-centric adversarial training approach that significantly enhances robustness compared to traditional robust training methods and pre-trained multimodal foundation models. Our findings underscore the potential of this approach to improve the robustness and adaptability of multimodal representations, offering a promising solution for dynamic and real-world applications.

Title: Latent Diffusion for Guided Document Table Generation

• Authors: Syed Jawwad Haider Hamdani, Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09800
• Pdf URL: https://arxiv.org/pdf/2408.09800
• Copy Paste: [[2408.09800]] Latent Diffusion for Guided Document Table Generation(https://arxiv.org/abs/2408.09800)
• Keywords: diffusion
• Abstract: Obtaining annotated table structure data for complex tables is a challenging task due to the inherent diversity and complexity of real-world document layouts. The scarcity of publicly available datasets with comprehensive annotations for intricate table structures hinders the development and evaluation of models designed for such scenarios. This research paper introduces a novel approach for generating annotated images for table structure by leveraging conditioned mask images of rows and columns through the application of latent diffusion models. The proposed method aims to enhance the quality of synthetic data used for training object detection models. Specifically, the study employs a conditioning mechanism to guide the generation of complex document table images, ensuring a realistic representation of table layouts. To evaluate the effectiveness of the generated data, we employ the popular YOLOv5 object detection model for training. The generated table images serve as valuable training samples, enriching the dataset with diverse table structures. The model is subsequently tested on the challenging pubtables-1m testset, a benchmark for table structure recognition in complex document layouts. Experimental results demonstrate that the introduced approach significantly improves the quality of synthetic data for training, leading to YOLOv5 models with enhanced performance. The mean Average Precision (mAP) values obtained on the pubtables-1m testset showcase results closely aligned with state-of-the-art methods. Furthermore, low FID results obtained on the synthetic data further validate the efficacy of the proposed methodology in generating annotated images for table structure.

Title: A Population-to-individual Tuning Framework for Adapting Pretrained LM to On-device User Intent Prediction

• Authors: Jiahui Gong, Jingtao Ding, Fanjin Meng, Guilong Chen, Hong Chen, Shen Zhao, Haisheng Lu, Yong Li
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.09815
• Pdf URL: https://arxiv.org/pdf/2408.09815
• Copy Paste: [[2408.09815]] A Population-to-individual Tuning Framework for Adapting Pretrained LM to On-device User Intent Prediction(https://arxiv.org/abs/2408.09815)
• Keywords: extraction, generative
• Abstract: Mobile devices, especially smartphones, can support rich functions and have developed into indispensable tools in daily life. With the rise of generative AI services, smartphones can potentially transform into personalized assistants, anticipating user needs and scheduling services accordingly. Predicting user intents on smartphones, and reflecting anticipated activities based on past interactions and context, remains a pivotal step towards this vision. Existing research predominantly focuses on specific domains, neglecting the challenge of modeling diverse event sequences across dynamic contexts. Leveraging pre-trained language models (PLMs) offers a promising avenue, yet adapting PLMs to on-device user intent prediction presents significant challenges. To address these challenges, we propose PITuning, a Population-to-Individual Tuning framework. PITuning enhances common pattern extraction through dynamic event-to-intent transition modeling and addresses long-tailed preferences via adaptive unlearning strategies. Experimental results on real-world datasets demonstrate PITuning's superior intent prediction performance, highlighting its ability to capture long-tailed preferences and its practicality for on-device prediction scenarios.

Title: CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models

• Authors: Linhao Yu, Yongqi Leng, Yufei Huang, Shang Wu, Haixin Liu, Xinmeng Ji, Jiahui Zhao, Jinwang Song, Tingting Cui, Xiaoqing Cheng, Tao Liu, Deyi Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09819
• Pdf URL: https://arxiv.org/pdf/2408.09819
• Copy Paste: [[2408.09819]] CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models(https://arxiv.org/abs/2408.09819)
• Keywords: large language model
• Abstract: What a large language model (LLM) would respond in ethically relevant context? In this paper, we curate a large benchmark CMoralEval for morality evaluation of Chinese LLMs. The data sources of CMoralEval are two-fold: 1) a Chinese TV program discussing Chinese moral norms with stories from the society and 2) a collection of Chinese moral anomies from various newspapers and academic papers on morality. With these sources, we aim to create a moral evaluation dataset characterized by diversity and authenticity. We develop a morality taxonomy and a set of fundamental moral principles that are not only rooted in traditional Chinese culture but also consistent with contemporary societal norms. To facilitate efficient construction and annotation of instances in CMoralEval, we establish a platform with AI-assisted instance generation to streamline the annotation process. These help us curate CMoralEval that encompasses both explicit moral scenarios (14,964 instances) and moral dilemma scenarios (15,424 instances), each with instances from different data sources. We conduct extensive experiments with CMoralEval to examine a variety of Chinese LLMs. Experiment results demonstrate that CMoralEval is a challenging benchmark for Chinese LLMs. The dataset is publicly available at \url{this https URL}.

Title: SurgicaL-CD: Generating Surgical Images via Unpaired Image Translation with Latent Consistency Diffusion Models

• Authors: Danush Kumar Venkatesh, Dominik Rivoir, Micha Pfeiffer, Stefanie Speidel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09822
• Pdf URL: https://arxiv.org/pdf/2408.09822
• Copy Paste: [[2408.09822]] SurgicaL-CD: Generating Surgical Images via Unpaired Image Translation with Latent Consistency Diffusion Models(https://arxiv.org/abs/2408.09822)
• Keywords: privacy, diffusion, generative
• Abstract: Computer-assisted surgery (CAS) systems are designed to assist surgeons during procedures, thereby reducing complications and enhancing patient care. Training machine learning models for these systems requires a large corpus of annotated datasets, which is challenging to obtain in the surgical domain due to patient privacy concerns and the significant labeling effort required from doctors. Previous methods have explored unpaired image translation using generative models to create realistic surgical images from simulations. However, these approaches have struggled to produce high-quality, diverse surgical images. In this work, we introduce \emph{SurgicaL-CD}, a consistency-distilled diffusion method to generate realistic surgical images with only a few sampling steps without paired data. We evaluate our approach on three datasets, assessing the generated images in terms of quality and utility as downstream training datasets. Our results demonstrate that our method outperforms GANs and diffusion-based approaches. Our code is available at \url{this https URL}.

Title: Segment-Anything Models Achieve Zero-shot Robustness in Autonomous Driving

• Authors: Jun Yan, Pengyu Wang, Danni Wang, Weiquan Huang, Daniel Watzenig, Huilin Yin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09839
• Pdf URL: https://arxiv.org/pdf/2408.09839
• Copy Paste: [[2408.09839]] Segment-Anything Models Achieve Zero-shot Robustness in Autonomous Driving(https://arxiv.org/abs/2408.09839)
• Keywords: attack, robust, segmentation
• Abstract: Semantic segmentation is a significant perception task in autonomous driving. It suffers from the risks of adversarial examples. In the past few years, deep learning has gradually transitioned from convolutional neural network (CNN) models with a relatively small number of parameters to foundation models with a huge number of parameters. The segment-anything model (SAM) is a generalized image segmentation framework that is capable of handling various types of images and is able to recognize and segment arbitrary objects in an image without the need to train on a specific object. It is a unified model that can handle diverse downstream tasks, including semantic segmentation, object detection, and tracking. In the task of semantic segmentation for autonomous driving, it is significant to study the zero-shot adversarial robustness of SAM. Therefore, we deliver a systematic empirical study on the robustness of SAM without additional training. Based on the experimental results, the zero-shot adversarial robustness of the SAM under the black-box corruptions and white-box adversarial attacks is acceptable, even without the need for additional training. The finding of this study is insightful in that the gigantic model parameters and huge amounts of training data lead to the phenomenon of emergence, which builds a guarantee of adversarial robustness. SAM is a vision foundation model that can be regarded as an early prototype of an artificial general intelligence (AGI) pipeline. In such a pipeline, a unified model can handle diverse tasks. Therefore, this research not only inspects the impact of vision foundation models on safe autonomous driving but also provides a perspective on developing trustworthy AGI. The code is available at: this https URL.

Title: Continual Dialogue State Tracking via Reason-of-Select Distillation

• Authors: Yujie Feng, Bo Liu, Xiaoyu Dong, Zexin Lu, Li-Ming Zhan, Xiao-Ming Wu, Albert Y.S. Lam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09846
• Pdf URL: https://arxiv.org/pdf/2408.09846
• Copy Paste: [[2408.09846]] Continual Dialogue State Tracking via Reason-of-Select Distillation(https://arxiv.org/abs/2408.09846)
• Keywords: robust
• Abstract: An ideal dialogue system requires continuous skill acquisition and adaptation to new tasks while retaining prior knowledge. Dialogue State Tracking (DST), vital in these systems, often involves learning new services and confronting catastrophic forgetting, along with a critical capability loss termed the "Value Selection Quandary." To address these challenges, we introduce the Reason-of-Select (RoS) distillation method by enhancing smaller models with a novel 'meta-reasoning' capability. Meta-reasoning employs an enhanced multi-domain perspective, combining fragments of meta-knowledge from domain-specific dialogues during continual learning. This transcends traditional single-perspective reasoning. The domain bootstrapping process enhances the model's ability to dissect intricate dialogues from multiple possible values. Its domain-agnostic property aligns data distribution across different domains, effectively mitigating forgetting. Additionally, two novel improvements, "multi-value resolution" strategy and Semantic Contrastive Reasoning Selection method, significantly enhance RoS by generating DST-specific selection chains and mitigating hallucinations in teachers' reasoning, ensuring effective and reliable knowledge transfer. Extensive experiments validate the exceptional performance and robust generalization capabilities of our method. The source code is provided for reproducibility.

Title: Importance Weighting Can Help Large Language Models Self-Improve

• Authors: Chunyang Jiang, Chi-min Chan, Wei Xue, Qifeng Liu, Yike Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09849
• Pdf URL: https://arxiv.org/pdf/2408.09849
• Copy Paste: [[2408.09849]] Importance Weighting Can Help Large Language Models Self-Improve(https://arxiv.org/abs/2408.09849)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown remarkable capability in numerous tasks and applications. However, fine-tuning LLMs using high-quality datasets under external supervision remains prohibitively expensive. In response, LLM self-improvement approaches have been vibrantly developed recently. The typical paradigm of LLM self-improvement involves training LLM on self-generated data, part of which may be detrimental and should be filtered out due to the unstable data quality. While current works primarily employs filtering strategies based on answer correctness, in this paper, we demonstrate that filtering out correct but with high distribution shift extent (DSE) samples could also benefit the results of self-improvement. Given that the actual sample distribution is usually inaccessible, we propose a new metric called DS weight to approximate DSE, inspired by the Importance Weighting methods. Consequently, we integrate DS weight with self-consistency to comprehensively filter the self-generated samples and fine-tune the language model. Experiments show that with only a tiny valid set (up to 5\% size of the training set) to compute DS weight, our approach can notably promote the reasoning ability of current LLM self-improvement methods. The resulting performance is on par with methods that rely on external supervision from pre-trained reward models.

Title: Self-Directed Turing Test for Large Language Models

• Authors: Weiqi Wu, Hongqiu Wu, Hai Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09853
• Pdf URL: https://arxiv.org/pdf/2408.09853
• Copy Paste: [[2408.09853]] Self-Directed Turing Test for Large Language Models(https://arxiv.org/abs/2408.09853)
• Keywords: large language model
• Abstract: The Turing test examines whether AIs can exhibit human-like behaviour in natural language conversations. Traditional Turing tests adopt a rigid dialogue format where each participant sends only one message each time and require continuous human involvement to direct the entire interaction with the test subject. This fails to reflect a natural conversational style and hinders the evaluation of Large Language Models (LLMs) in complex and prolonged dialogues. This paper proposes the Self-Directed Turing Test, which extends the original test with a burst dialogue format, allowing more dynamic exchanges by multiple consecutive messages. It further efficiently reduces human workload by having the LLM self-direct the majority of the test process, iteratively generating dialogues that simulate its interaction with humans. With the pseudo-dialogue history, the model then engages in a shorter dialogue with a human, which is paired with a human-human conversation on the same topic to be judged using questionnaires. We introduce the X-Turn Pass-Rate metric to assess the human likeness of LLMs across varying durations. While LLMs like GPT-4 initially perform well, achieving pass rates of 51.9% and 38.9% during 3 turns and 10 turns of dialogues respectively, their performance drops as the dialogue progresses, which underscores the difficulty in maintaining consistency in the long term.

Title: TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidation

• Authors: Yujie Feng, Xu Chu, Yongxin Xu, Guangyuan Shi, Bo Liu, Xiao-Ming Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09857
• Pdf URL: https://arxiv.org/pdf/2408.09857
• Copy Paste: [[2408.09857]] TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidation(https://arxiv.org/abs/2408.09857)
• Keywords: protect
• Abstract: A practical dialogue system requires the capacity for ongoing skill acquisition and adaptability to new tasks while preserving prior knowledge. However, current methods for Continual Dialogue State Tracking (DST), a crucial function of dialogue systems, struggle with the catastrophic forgetting issue and knowledge transfer between tasks. We present TaSL, a novel framework for task skill localization and consolidation that enables effective knowledge transfer without relying on memory replay. TaSL uses a novel group-wise technique to pinpoint task-specific and task-shared areas. Additionally, a fine-grained skill consolidation strategy protects task-specific knowledge from being forgotten while updating shared knowledge for bi-directional knowledge transfer. As a result, TaSL strikes a balance between preserving previous knowledge and excelling at new tasks. Comprehensive experiments on various backbones highlight the significant performance improvements of TaSL over existing state-of-the-art methods. The source code is provided for reproducibility.

Title: ShortCircuit: AlphaZero-Driven Circuit Design

• Authors: Dimitrios Tsaras, Antoine Grosnit, Lei Chen, Zhiyao Xie, Haitham Bou-Ammar, Mingxuan Yuan
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2408.09858
• Pdf URL: https://arxiv.org/pdf/2408.09858
• Copy Paste: [[2408.09858]] ShortCircuit: AlphaZero-Driven Circuit Design(https://arxiv.org/abs/2408.09858)
• Keywords: transformer, generative
• Abstract: Chip design relies heavily on generating Boolean circuits, such as AND-Inverter Graphs (AIGs), from functional descriptions like truth tables. While recent advances in deep learning have aimed to accelerate circuit design, these efforts have mostly focused on tasks other than synthesis, and traditional heuristic methods have plateaued. In this paper, we introduce ShortCircuit, a novel transformer-based architecture that leverages the structural properties of AIGs and performs efficient space exploration. Contrary to prior approaches attempting end-to-end generation of logic circuits using deep networks, ShortCircuit employs a two-phase process combining supervised with reinforcement learning to enhance generalization to unseen truth tables. We also propose an AlphaZero variant to handle the double exponentially large state space and the sparsity of the rewards, enabling the discovery of near-optimal designs. To evaluate the generative performance of our trained model , we extract 500 truth tables from a benchmark set of 20 real-world circuits. ShortCircuit successfully generates AIGs for 84.6% of the 8-input test truth tables, and outperforms the state-of-the-art logic synthesis tool, ABC, by 14.61% in terms of circuits size.

Title: OccMamba: Semantic Occupancy Prediction with State Space Models

• Authors: Heng Li, Yuenan Hou, Xiaohan Xing, Xiao Sun, Yanyong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09859
• Pdf URL: https://arxiv.org/pdf/2408.09859
• Copy Paste: [[2408.09859]] OccMamba: Semantic Occupancy Prediction with State Space Models(https://arxiv.org/abs/2408.09859)
• Keywords: transformer
• Abstract: Training deep learning models for semantic occupancy prediction is challenging due to factors such as a large number of occupancy cells, severe occlusion, limited visual cues, complicated driving scenarios, etc. Recent methods often adopt transformer-based architectures given their strong capability in learning input-conditioned weights and long-range relationships. However, transformer-based networks are notorious for their quadratic computation complexity, seriously undermining their efficacy and deployment in semantic occupancy prediction. Inspired by the global modeling and linear computation complexity of the Mamba architecture, we present the first Mamba-based network for semantic occupancy prediction, termed OccMamba. However, directly applying the Mamba architecture to the occupancy prediction task yields unsatisfactory performance due to the inherent domain gap between the linguistic and 3D domains. To relieve this problem, we present a simple yet effective 3D-to-1D reordering operation, i.e., height-prioritized 2D Hilbert expansion. It can maximally retain the spatial structure of point clouds as well as facilitate the processing of Mamba blocks. Our OccMamba achieves state-of-the-art performance on three prevalent occupancy prediction benchmarks, including OpenOccupancy, SemanticKITTI and SemanticPOSS. Notably, on OpenOccupancy, our OccMamba outperforms the previous state-of-the-art Co-Occ by 3.1% IoU and 3.2% mIoU, respectively. Codes will be released upon publication.

Title: 3D-Aware Instance Segmentation and Tracking in Egocentric Videos

• Authors: Yash Bhalgat, Vadim Tschernezki, Iro Laina, João F. Henriques, Andrea Vedaldi, Andrew Zisserman
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09860
• Pdf URL: https://arxiv.org/pdf/2408.09860
• Copy Paste: [[2408.09860]] 3D-Aware Instance Segmentation and Tracking in Egocentric Videos(https://arxiv.org/abs/2408.09860)
• Keywords: robust, segmentation
• Abstract: Egocentric videos present unique challenges for 3D scene understanding due to rapid camera motion, frequent object occlusions, and limited object visibility. This paper introduces a novel approach to instance segmentation and tracking in first-person video that leverages 3D awareness to overcome these obstacles. Our method integrates scene geometry, 3D object centroid tracking, and instance segmentation to create a robust framework for analyzing dynamic egocentric scenes. By incorporating spatial and temporal cues, we achieve superior performance compared to state-of-the-art 2D approaches. Extensive evaluations on the challenging EPIC Fields dataset demonstrate significant improvements across a range of tracking and segmentation consistency metrics. Specifically, our method outperforms the next best performing approach by $7$ points in Association Accuracy (AssA) and $4.5$ points in IDF1 score, while reducing the number of ID switches by $73\%$ to $80\%$ across various object categories. Leveraging our tracked instance segmentations, we showcase downstream applications in 3D object reconstruction and amodal video object segmentation in these egocentric settings.

Title: Transferring Backdoors between Large Language Models by Knowledge Distillation

• Authors: Pengzhou Cheng, Zongru Wu, Tianjie Ju, Wei Du, Zhuosheng Zhang Gongshen Liu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.09878
• Pdf URL: https://arxiv.org/pdf/2408.09878
• Copy Paste: [[2408.09878]] Transferring Backdoors between Large Language Models by Knowledge Distillation(https://arxiv.org/abs/2408.09878)
• Keywords: security, attack, robust, steal, large language model
• Abstract: Backdoor Attacks have been a serious vulnerability against Large Language Models (LLMs). However, previous methods only reveal such risk in specific models, or present tasks transferability after attacking the pre-trained phase. So, how risky is the model transferability of a backdoor attack? In this paper, we focus on whether existing mini-LLMs may be unconsciously instructed in backdoor knowledge by poisoned teacher LLMs through knowledge distillation (KD). Specifically, we propose ATBA, an adaptive transferable backdoor attack, which can effectively distill the backdoor of teacher LLMs into small models when only executing clean-tuning. We first propose the Target Trigger Generation (TTG) module that filters out a set of indicative trigger candidates from the token list based on cosine similarity distribution. Then, we exploit a shadow model to imitate the distilling process and introduce an Adaptive Trigger Optimization (ATO) module to realize a gradient-based greedy feedback to search optimal triggers. Extensive experiments show that ATBA generates not only positive guidance for student models but also implicitly transfers backdoor knowledge. Our attack is robust and stealthy, with over 80% backdoor transferability, and hopes the attention of security.

Title: SAM-UNet:Enhancing Zero-Shot Segmentation of SAM for Universal Medical Images

• Authors: Sihan Yang, Haixia Bi, Hai Zhang, Jian Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09886
• Pdf URL: https://arxiv.org/pdf/2408.09886
• Copy Paste: [[2408.09886]] SAM-UNet:Enhancing Zero-Shot Segmentation of SAM for Universal Medical Images(https://arxiv.org/abs/2408.09886)
• Keywords: transformer, segmentation
• Abstract: Segment Anything Model (SAM) has demonstrated impressive performance on a wide range of natural image segmentation tasks. However, its performance significantly deteriorates when directly applied to medical domain, due to the remarkable differences between natural images and medical images. Some researchers have attempted to train SAM on large scale medical datasets. However, poor zero-shot performance is observed from the experimental results. In this context, inspired by the superior performance of U-Net-like models in medical image segmentation, we propose SAMUNet, a new foundation model which incorporates U-Net to the original SAM, to fully leverage the powerful contextual modeling ability of convolutions. To be specific, we parallel a convolutional branch in the image encoder, which is trained independently with the vision Transformer branch frozen. Additionally, we employ multi-scale fusion in the mask decoder, to facilitate accurate segmentation of objects with different scales. We train SAM-UNet on SA-Med2D-16M, the largest 2-dimensional medical image segmentation dataset to date, yielding a universal pretrained model for medical images. Extensive experiments are conducted to evaluate the performance of the model, and state-of-the-art result is achieved, with a dice similarity coefficient score of 0.883 on SA-Med2D-16M dataset. Specifically, in zero-shot segmentation experiments, our model not only significantly outperforms previous large medical SAM models across all modalities, but also substantially mitigates the performance degradation seen on unseen modalities. It should be highlighted that SAM-UNet is an efficient and extensible foundation model, which can be further fine-tuned for other downstream tasks in medical community. The code is available at this https URL.

Title: Forecasting Attacker Actions using Alert-driven Attack Graphs

• Authors: Ion Băbălău, Azqa Nadeem
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.09888
• Pdf URL: https://arxiv.org/pdf/2408.09888
• Copy Paste: [[2408.09888]] Forecasting Attacker Actions using Alert-driven Attack Graphs(https://arxiv.org/abs/2408.09888)
• Keywords: security, defense, attack
• Abstract: While intrusion detection systems form the first line-of-defense against cyberattacks, they often generate an overwhelming volume of alerts, leading to alert fatigue among security operations center (SOC) analysts. Alert-driven attack graphs (AGs) have been developed to reduce alert fatigue by automatically discovering attack paths in intrusion alerts. However, they only work in offline settings and cannot prioritize critical attack paths. This paper builds an action forecasting capability on top of the existing alert-driven AG framework for predicting the next likely attacker action given a sequence of observed actions, thus enabling analysts to prioritize non-trivial attack paths. We also modify the framework to build AGs in real time, as new alerts are triggered. This way, we convert alert-driven AGs into an early warning system that enables analysts to circumvent ongoing attacks and break the cyber killchain. We propose an expectation maximization approach to forecast future actions in a reversed suffix-based probabilistic deterministic finite automaton (rSPDFA). By utilizing three real-world intrusion and endpoint alert datasets, we empirically demonstrate that the best performing rSPDFA achieves an average top-3 accuracy of 67.27%, which reflects a 57.17% improvement over three baselines, on average. We also invite six SOC analysts to use the evolving AGs in two scenarios. Their responses suggest that the action forecasts help them prioritize critical incidents, while the evolving AGs enable them to choose countermeasures in real-time.

Title: Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Rates

• Authors: Puning Zhao, Jiafei Wu, Zhe Liu, Chong Wang, Rongfei Fan, Qingming Li
• Subjects: cs.LG, cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2408.09891
• Pdf URL: https://arxiv.org/pdf/2408.09891
• Copy Paste: [[2408.09891]] Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Rates(https://arxiv.org/abs/2408.09891)
• Keywords: privacy
• Abstract: We study convex optimization problems under differential privacy (DP). With heavy-tailed gradients, existing works achieve suboptimal rates. The main obstacle is that existing gradient estimators have suboptimal tail properties, resulting in a superfluous factor of $d$ in the union bound. In this paper, we explore algorithms achieving optimal rates of DP optimization with heavy-tailed gradients. Our first method is a simple clipping approach. Under bounded $p$-th order moments of gradients, with $n$ samples, it achieves $\tilde{O}(\sqrt{d/n}+\sqrt{d}(\sqrt{d}/n\epsilon)^{1-1/p})$ population risk with $\epsilon\leq 1/\sqrt{d}$. We then propose an iterative updating method, which is more complex but achieves this rate for all $\epsilon\leq 1$. The results significantly improve over existing methods. Such improvement relies on a careful treatment of the tail behavior of gradient estimators. Our results match the minimax lower bound in \cite{kamath2022improved}, indicating that the theoretical limit of stochastic convex optimization under DP is achievable.

Title: Performance Law of Large Language Models

• Authors: Chuhan Wu, Ruiming Tang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09895
• Pdf URL: https://arxiv.org/pdf/2408.09895
• Copy Paste: [[2408.09895]] Performance Law of Large Language Models(https://arxiv.org/abs/2408.09895)
• Keywords: large language model
• Abstract: Guided by the belief of the scaling law, large language models (LLMs) have achieved impressive performance in recent years. However, scaling law only gives a qualitative estimation of loss, which is influenced by various factors such as model architectures, data distributions, tokenizers, and computation precision. Thus, estimating the real performance of LLMs with different training settings rather than loss may be quite useful in practical development. In this article, we present an empirical equation named "Performance Law" to directly predict the MMLU score of an LLM, which is a widely used metric to indicate the general capability of LLMs in real-world conversations and applications. Based on only a few key hyperparameters of the LLM architecture and the size of training data, we obtain a quite accurate MMLU prediction of various LLMs with diverse sizes and architectures developed by different organizations in different years. Performance law can be used to guide the choice of LLM architecture and the effective allocation of computational resources without extensive experiments.

Title: Instruction-Based Molecular Graph Generation with Unified Text-Graph Diffusion Model

• Authors: Yuran Xiang, Haiteng Zhao, Chang Ma, Zhi-Hong Deng
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2408.09896
• Pdf URL: https://arxiv.org/pdf/2408.09896
• Copy Paste: [[2408.09896]] Instruction-Based Molecular Graph Generation with Unified Text-Graph Diffusion Model(https://arxiv.org/abs/2408.09896)
• Keywords: diffusion, transformer, large language model
• Abstract: Recent advancements in computational chemistry have increasingly focused on synthesizing molecules based on textual instructions. Integrating graph generation with these instructions is complex, leading most current methods to use molecular sequences with pre-trained large language models. In response to this challenge, we propose a novel framework, named $\textbf{UTGDiff (Unified Text-Graph Diffusion Model)}$, which utilizes language models for discrete graph diffusion to generate molecular graphs from instructions. UTGDiff features a unified text-graph transformer as the denoising network, derived from pre-trained language models and minimally modified to process graph data through attention bias. Our experimental results demonstrate that UTGDiff consistently outperforms sequence-based baselines in tasks involving instruction-based molecule generation and editing, achieving superior performance with fewer parameters given an equivalent level of pretraining corpus. Our code is availble at this https URL.

Title: Harnessing Multi-resolution and Multi-scale Attention for Underwater Image Restoration

• Authors: Alik Pramanick, Arijit Sur, V. Vijaya Saradhi
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.09912
• Pdf URL: https://arxiv.org/pdf/2408.09912
• Copy Paste: [[2408.09912]] Harnessing Multi-resolution and Multi-scale Attention for Underwater Image Restoration(https://arxiv.org/abs/2408.09912)
• Keywords: robust
• Abstract: Underwater imagery is often compromised by factors such as color distortion and low contrast, posing challenges for high-level vision tasks. Recent underwater image restoration (UIR) methods either analyze the input image at full resolution, resulting in spatial richness but contextual weakness, or progressively from high to low resolution, yielding reliable semantic information but reduced spatial accuracy. Here, we propose a lightweight multi-stage network called Lit-Net that focuses on multi-resolution and multi-scale image analysis for restoring underwater images while retaining original resolution during the first stage, refining features in the second, and focusing on reconstruction in the final stage. Our novel encoder block utilizes parallel $1\times1$ convolution layers to capture local information and speed up operations. Further, we incorporate a modified weighted color channel-specific $l_1$ loss ($cl_1$) function to recover color and detail information. Extensive experimentations on publicly available datasets suggest our model's superiority over recent state-of-the-art methods, with significant improvement in qualitative and quantitative measures, such as $29.477$ dB PSNR ($1.92\%$ improvement) and $0.851$ SSIM ($2.87\%$ improvement) on the EUVP dataset. The contributions of Lit-Net offer a more robust approach to underwater image enhancement and super-resolution, which is of considerable importance for underwater autonomous vehicles and surveillance. The code is available at: this https URL.

Title: Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit

• Authors: Qizhou Chen, Taolin Zhang, Chengyu Wang, Xiaofeng He, Dakan Wang, Tingting Liu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09916
• Pdf URL: https://arxiv.org/pdf/2408.09916
• Copy Paste: [[2408.09916]] Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit(https://arxiv.org/abs/2408.09916)
• Keywords: large language model
• Abstract: Model editing aims to correct outdated or erroneous knowledge in large models without costly retraining. Recent research discovered that the mid-layer representation of the subject's final token in a prompt has a strong influence on factual predictions, and developed Large Language Model (LLM) editing techniques based on this observation. However, for Vision-LLMs (VLLMs), how visual representations impact the predictions from a decoder-only language model remains largely unexplored. To the best of our knowledge, model editing for VLLMs has not been extensively studied in the literature. In this work, we employ the contribution allocation and noise perturbation methods to measure the contributions of visual representations for token predictions. Our attribution analysis shows that visual representations in mid-to-later layers that are highly relevant to the prompt contribute significantly to predictions. Based on these insights, we propose VisEdit, a novel model editor for VLLMs that effectively corrects knowledge by editing intermediate visual representations in regions important to the edit prompt. We evaluated VisEdit using multiple VLLM backbones and public VLLM editing benchmark datasets. The results show the superiority of VisEdit over the strong baselines adapted from existing state-of-the-art editors for LLMs.

Title: Long-Tail Temporal Action Segmentation with Group-wise Temporal Logit Adjustment

• Authors: Zhanzhong Pang, Fadime Sener, Shrinivas Ramasubramanian, Angela Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.09919
• Pdf URL: https://arxiv.org/pdf/2408.09919
• Copy Paste: [[2408.09919]] Long-Tail Temporal Action Segmentation with Group-wise Temporal Logit Adjustment(https://arxiv.org/abs/2408.09919)
• Keywords: segmentation
• Abstract: Procedural activity videos often exhibit a long-tailed action distribution due to varying action frequencies and durations. However, state-of-the-art temporal action segmentation methods overlook the long tail and fail to recognize tail actions. Existing long-tail methods make class-independent assumptions and struggle to identify tail classes when applied to temporal segmentation frameworks. This work proposes a novel group-wise temporal logit adjustment~(G-TLA) framework that combines a group-wise softmax formulation while leveraging activity information and action ordering for logit adjustment. The proposed framework significantly improves in segmenting tail actions without any performance loss on head actions.

Title: DiscoNeRF: Class-Agnostic Object Field for 3D Object Discovery

• Authors: Corentin Dumery, Aoxiang Fan, Ren Li, Nicolas Talabot, Pascal Fua
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.09928
• Pdf URL: https://arxiv.org/pdf/2408.09928
• Copy Paste: [[2408.09928]] DiscoNeRF: Class-Agnostic Object Field for 3D Object Discovery(https://arxiv.org/abs/2408.09928)
• Keywords: robust, segmentation
• Abstract: Neural Radiance Fields (NeRFs) have become a powerful tool for modeling 3D scenes from multiple images. However, NeRFs remain difficult to segment into semantically meaningful regions. Previous approaches to 3D segmentation of NeRFs either require user interaction to isolate a single object, or they rely on 2D semantic masks with a limited number of classes for supervision. As a consequence, they generalize poorly to class-agnostic masks automatically generated in real scenes. This is attributable to the ambiguity arising from zero-shot segmentation, yielding inconsistent masks across views. In contrast, we propose a method that is robust to inconsistent segmentations and successfully decomposes the scene into a set of objects of any class. By introducing a limited number of competing object slots against which masks are matched, a meaningful object representation emerges that best explains the 2D supervision and minimizes an additional regularization term. Our experiments demonstrate the ability of our method to generate 3D panoptic segmentations on complex scenes, and extract high-quality 3D assets from NeRFs that can then be used in virtual 3D environments.

Title: Privacy Technologies for Financial Intelligence

• Authors: Yang Li, Thilina Ranbaduge, Kee Siong Ng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.09935
• Pdf URL: https://arxiv.org/pdf/2408.09935
• Copy Paste: [[2408.09935]] Privacy Technologies for Financial Intelligence(https://arxiv.org/abs/2408.09935)
• Keywords: privacy
• Abstract: Financial crimes like terrorism financing and money laundering can have real impacts on society, including the abuse and mismanagement of public funds, increase in societal problems such as drug trafficking and illicit gambling with attendant economic costs, and loss of innocent lives in the case of terrorism activities. Complex financial crimes can be hard to detect primarily because data related to different pieces of the overall puzzle is usually distributed across a network of financial institutions, regulators, and law-enforcement agencies and they cannot be easily shared due to privacy constraints. Recent advances in Privacy-Preserving Data Matching and Machine Learning provide an opportunity for regulators and the financial industry to come together to solve the risk-discovery problem with technology. This paper provides a survey of the financial intelligence landscape and where opportunities lie for privacy technologies to improve the state-of-the-art in financial-crime detection.

Title: ML-CrAIST: Multi-scale Low-high Frequency Information-based Cross black Attention with Image Super-resolving Transformer

• Authors: Alik Pramanick, Utsav Bheda, Arijit Sur
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.09940
• Pdf URL: https://arxiv.org/pdf/2408.09940
• Copy Paste: [[2408.09940]] ML-CrAIST: Multi-scale Low-high Frequency Information-based Cross black Attention with Image Super-resolving Transformer(https://arxiv.org/abs/2408.09940)
• Keywords: transformer
• Abstract: Recently, transformers have captured significant interest in the area of single-image super-resolution tasks, demonstrating substantial gains in performance. Current models heavily depend on the network's extensive ability to extract high-level semantic details from images while overlooking the effective utilization of multi-scale image details and intermediate information within the network. Furthermore, it has been observed that high-frequency areas in images present significant complexity for super-resolution compared to low-frequency areas. This work proposes a transformer-based super-resolution architecture called ML-CrAIST that addresses this gap by utilizing low-high frequency information in multiple scales. Unlike most of the previous work (either spatial or channel), we operate spatial and channel self-attention, which concurrently model pixel interaction from both spatial and channel dimensions, exploiting the inherent correlations across spatial and channel axis. Further, we devise a cross-attention block for super-resolution, which explores the correlations between low and high-frequency information. Quantitative and qualitative assessments indicate that our proposed ML-CrAIST surpasses state-of-the-art super-resolution methods (e.g., 0.15 dB gain @Manga109 $\times$4). Code is available on: this https URL.

Title: Calibrating Noise for Group Privacy in Subsampled Mechanisms

• Authors: Yangfan Jiang, Xinjian Luo, Yin Yang, Xiaokui Xiao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.09943
• Pdf URL: https://arxiv.org/pdf/2408.09943
• Copy Paste: [[2408.09943]] Calibrating Noise for Group Privacy in Subsampled Mechanisms(https://arxiv.org/abs/2408.09943)
• Keywords: privacy, protect
• Abstract: Given a group size m and a sensitive dataset D, group privacy (GP) releases information about D with the guarantee that the adversary cannot infer with high confidence whether the underlying data is D or a neighboring dataset D' that differs from D by m records. GP generalizes the well-established notion of differential privacy (DP) for protecting individuals' privacy; in particular, when m=1, GP reduces to DP. Compared to DP, GP is capable of protecting the sensitive aggregate information of a group of up to m individuals, e.g., the average annual income among members of a yacht club. Despite its longstanding presence in the research literature and its promising applications, GP is often treated as an afterthought, with most approaches first developing a DP mechanism and then using a generic conversion to adapt it for GP, treating the DP solution as a black box. As we point out in the paper, this methodology is suboptimal when the underlying DP solution involves subsampling, e.g., in the classic DP-SGD method for training deep learning models. In this case, the DP-to-GP conversion is overly pessimistic in its analysis, leading to low utility in the published results under GP. Motivated by this, we propose a novel analysis framework that provides tight privacy accounting for subsampled GP mechanisms. Instead of converting a black-box DP mechanism to GP, our solution carefully analyzes and utilizes the inherent randomness in subsampled mechanisms, leading to a substantially improved bound on the privacy loss with respect to GP. The proposed solution applies to a wide variety of foundational mechanisms with subsampling. Extensive experiments with real datasets demonstrate that compared to the baseline convert-from-blackbox-DP approach, our GP mechanisms achieve noise reductions of over an order of magnitude in several practical settings, including deep neural network training.

Title: Benchmarking LLMs for Translating Classical Chinese Poetry:Evaluating Adequacy, Fluency, and Elegance

• Authors: Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.09945
• Pdf URL: https://arxiv.org/pdf/2408.09945
• Copy Paste: [[2408.09945]] Benchmarking LLMs for Translating Classical Chinese Poetry:Evaluating Adequacy, Fluency, and Elegance(https://arxiv.org/abs/2408.09945)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown remarkable performance in general translation tasks. However, the increasing demand for high-quality translations that are not only adequate but also fluent and elegant. To assess the extent to which current LLMs can meet these demands, we introduce a suitable benchmark for translating classical Chinese poetry into English. This task requires not only adequacy in translating culturally and historically significant content but also a strict adherence to linguistic fluency and poetic elegance. Our study reveals that existing LLMs fall short of this task. To address these issues, we propose RAT, a \textbf{R}etrieval-\textbf{A}ugmented machine \textbf{T}ranslation method that enhances the translation process by incorporating knowledge related to classical poetry. Additionally, we propose an automatic evaluation metric based on GPT-4, which better assesses translation quality in terms of adequacy, fluency, and elegance, overcoming the limitations of traditional metrics. Our dataset and code will be made available.

Title: C${^2}$RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval

• Authors: Zhigang Chen, Benjia Zhou, Yiqing Huang, Jun Wan, Yibo Hu, Hailin Shi, Yanyan Liang, Zhen Lei, Du Zhang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.09949
• Pdf URL: https://arxiv.org/pdf/2408.09949
• Copy Paste: [[2408.09949]] C${^2}$RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval(https://arxiv.org/abs/2408.09949)
• Keywords: robust
• Abstract: Sign Language Representation Learning (SLRL) is crucial for a range of sign language-related downstream tasks such as Sign Language Translation (SLT) and Sign Language Retrieval (SLRet). Recently, many gloss-based and gloss-free SLRL methods have been proposed, showing promising performance. Among them, the gloss-free approach shows promise for strong scalability without relying on gloss annotations. However, it currently faces suboptimal solutions due to challenges in encoding the intricate, context-sensitive characteristics of sign language videos, mainly struggling to discern essential sign features using a non-monotonic video-text alignment strategy. Therefore, we introduce an innovative pretraining paradigm for gloss-free SLRL, called C${^2}$RL, in this paper. Specifically, rather than merely incorporating a non-monotonic semantic alignment of video and text to learn language-oriented sign features, we emphasize two pivotal aspects of SLRL: Implicit Content Learning (ICL) and Explicit Context Learning (ECL). ICL delves into the content of communication, capturing the nuances, emphasis, timing, and rhythm of the signs. In contrast, ECL focuses on understanding the contextual meaning of signs and converting them into equivalent sentences. Despite its simplicity, extensive experiments confirm that the joint optimization of ICL and ECL results in robust sign language representation and significant performance gains in gloss-free SLT and SLRet tasks. Notably, C${^2}$RL improves the BLEU-4 score by +5.3 on P14T, +10.6 on CSL-daily, +6.2 on OpenASL, and +1.3 on How2Sign. It also boosts the R@1 score by +8.3 on P14T, +14.4 on CSL-daily, and +5.9 on How2Sign. Additionally, we set a new baseline for the OpenASL dataset in the SLRet task.

Title: Weakly Supervised Pretraining and Multi-Annotator Supervised Finetuning for Facial Wrinkle Detection

• Authors: Ik Jun Moon, Junho Moon, Ikbeom Jang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09952
• Pdf URL: https://arxiv.org/pdf/2408.09952
• Copy Paste: [[2408.09952]] Weakly Supervised Pretraining and Multi-Annotator Supervised Finetuning for Facial Wrinkle Detection(https://arxiv.org/abs/2408.09952)
• Keywords: segmentation
• Abstract: 1. Research question: With the growing interest in skin diseases and skin aesthetics, the ability to predict facial wrinkles is becoming increasingly important. This study aims to evaluate whether a computational model, convolutional neural networks (CNN), can be trained for automated facial wrinkle segmentation. 2. Findings: Our study presents an effective technique for integrating data from multiple annotators and illustrates that transfer learning can enhance performance, resulting in dependable segmentation of facial wrinkles. 3. Meaning: This approach automates intricate and time-consuming tasks of wrinkle analysis with a deep learning framework. It could be used to facilitate skin treatments and diagnostics.

Title: Validation of the Results of Cross-chain Smart Contract Based on Confirmation Method

• Authors: Hong Su
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.09962
• Pdf URL: https://arxiv.org/pdf/2408.09962
• Copy Paste: [[2408.09962]] Validation of the Results of Cross-chain Smart Contract Based on Confirmation Method(https://arxiv.org/abs/2408.09962)
• Keywords: secure, security
• Abstract: Smart contracts are widely utilized in cross-chain interactions, where their results are transmitted from one blockchain (the producer blockchain) to another (the consumer blockchain). Unfortunately, the consumer blockchain often accepts these results without executing the smart contracts for validation, posing potential security risks. To address this, we propose a method for validating cross-chain smart contract results. Our approach emphasizes consumer blockchain execution of cross-chain smart contracts of producer blockchain, allowing comparison of results with the transmitted ones to detect potential discrepancies and ensure data integrity during cross-chain data dissemination. Additionally, we introduce the confirmation with proof method, which involves incorporating the chain of blocks and relevant cross-chain smart contract data from the producer blockchain into the consumer blockchain as evidence (or proof), establishing a unified and secure perspective of cross-chain smart contract results. Our verification results highlight the feasibility of cross-chain validation at the smart contract level.

Title: Unsupervised Machine Learning Hybrid Approach Integrating Linear Programming in Loss Function: A Robust Optimization Technique

• Authors: Andrew Kiruluta, Andreas Lemos
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2408.09967
• Pdf URL: https://arxiv.org/pdf/2408.09967
• Copy Paste: [[2408.09967]] Unsupervised Machine Learning Hybrid Approach Integrating Linear Programming in Loss Function: A Robust Optimization Technique(https://arxiv.org/abs/2408.09967)
• Keywords: robust, interpretability
• Abstract: This paper presents a novel hybrid approach that integrates linear programming (LP) within the loss function of an unsupervised machine learning model. By leveraging the strengths of both optimization techniques and machine learning, this method introduces a robust framework for solving complex optimization problems where traditional methods may fall short. The proposed approach encapsulates the constraints and objectives of a linear programming problem directly into the loss function, guiding the learning process to adhere to these constraints while optimizing the desired outcomes. This technique not only preserves the interpretability of linear programming but also benefits from the flexibility and adaptability of machine learning, making it particularly well-suited for unsupervised or semi-supervised learning scenarios.

Title: Uniting contrastive and generative learning for event sequences models

• Authors: Aleksandr Yugay, Alexey Zaytsev
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.09995
• Pdf URL: https://arxiv.org/pdf/2408.09995
• Copy Paste: [[2408.09995]] Uniting contrastive and generative learning for event sequences models(https://arxiv.org/abs/2408.09995)
• Keywords: robust, generative
• Abstract: High-quality representation of transactional sequences is vital for modern banking applications, including risk management, churn prediction, and personalized customer offers. Different tasks require distinct representation properties: local tasks benefit from capturing the client's current state, while global tasks rely on general behavioral patterns. Previous research has demonstrated that various self-supervised approaches yield representations that better capture either global or local qualities. This study investigates the integration of two self-supervised learning techniques - instance-wise contrastive learning and a generative approach based on restoring masked events in latent space. The combined approach creates representations that balance local and global transactional data characteristics. Experiments conducted on several public datasets, focusing on sequence classification and next-event type prediction, show that the integrated method achieves superior performance compared to individual approaches and demonstrates synergistic effects. These findings suggest that the proposed approach offers a robust framework for advancing event sequences representation learning in the financial sector.

Title: The Fairness-Quality Trade-off in Clustering

• Authors: Rashida Hakim, Ana-Andreea Stoica, Christos H. Papadimitriou, Mihalis Yannakakis
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.10002
• Pdf URL: https://arxiv.org/pdf/2408.10002
• Copy Paste: [[2408.10002]] The Fairness-Quality Trade-off in Clustering(https://arxiv.org/abs/2408.10002)
• Keywords: fair
• Abstract: Fairness in clustering has been considered extensively in the past; however, the trade-off between the two objectives -- e.g., can we sacrifice just a little in the quality of the clustering to significantly increase fairness, or vice-versa? -- has rarely been addressed. We introduce novel algorithms for tracing the complete trade-off curve, or Pareto front, between quality and fairness in clustering problems; that is, computing all clusterings that are not dominated in both objectives by other clusterings. Unlike previous work that deals with specific objectives for quality and fairness, we deal with all objectives for fairness and quality in two general classes encompassing most of the special cases addressed in previous work. Our algorithm must take exponential time in the worst case as the Pareto front itself can be exponential. Even when the Pareto front is polynomial, our algorithm may take exponential time, and we prove that this is inevitable unless P = NP. However, we also present a new polynomial-time algorithm for computing the entire Pareto front when the cluster centers are fixed, and for perhaps the most natural fairness objective: minimizing the sum, over all clusters, of the imbalance between the two groups in each cluster.

Title: Detecting Adversarial Attacks in Semantic Segmentation via Uncertainty Estimation: A Deep Analysis

• Authors: Kira Maag, Roman Resner, Asja Fischer
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.10021
• Pdf URL: https://arxiv.org/pdf/2408.10021
• Copy Paste: [[2408.10021]] Detecting Adversarial Attacks in Semantic Segmentation via Uncertainty Estimation: A Deep Analysis(https://arxiv.org/abs/2408.10021)
• Keywords: defense, attack, segmentation
• Abstract: Deep neural networks have demonstrated remarkable effectiveness across a wide range of tasks such as semantic segmentation. Nevertheless, these networks are vulnerable to adversarial attacks that add imperceptible perturbations to the input image, leading to false predictions. This vulnerability is particularly dangerous in safety-critical applications like automated driving. While adversarial examples and defense strategies are well-researched in the context of image classification, there is comparatively less research focused on semantic segmentation. Recently, we have proposed an uncertainty-based method for detecting adversarial attacks on neural networks for semantic segmentation. We observed that uncertainty, as measured by the entropy of the output distribution, behaves differently on clean versus adversely perturbed images, and we utilize this property to differentiate between the two. In this extended version of our work, we conduct a detailed analysis of uncertainty-based detection of adversarial attacks including a diverse set of adversarial attacks and various state-of-the-art neural networks. Our numerical experiments show the effectiveness of the proposed uncertainty-based detection method, which is lightweight and operates as a post-processing step, i.e., no model modifications or knowledge of the adversarial example generation process are required.

Title: Towards Robust Federated Image Classification: An Empirical Study of Weight Selection Strategies in Manufacturing

• Authors: Vinit Hegiste, Tatjana Legler, Martin Ruskowski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10024
• Pdf URL: https://arxiv.org/pdf/2408.10024
• Copy Paste: [[2408.10024]] Towards Robust Federated Image Classification: An Empirical Study of Weight Selection Strategies in Manufacturing(https://arxiv.org/abs/2408.10024)
• Keywords: robust, federate
• Abstract: In the realm of Federated Learning (FL), particularly within the manufacturing sector, the strategy for selecting client weights for server aggregation is pivotal for model performance. This study investigates the comparative effectiveness of two weight selection strategies: Final Epoch Weight Selection (FEWS) and Optimal Epoch Weight Selection (OEWS). Designed for manufacturing contexts where collaboration typically involves a limited number of partners (two to four clients), our research focuses on federated image classification tasks. We employ various neural network architectures, including EfficientNet, ResNet, and VGG, to assess the impact of these weight selection strategies on model convergence and robustness. Our research aims to determine whether FEWS or OEWS enhances the global FL model's performance across communication rounds (CRs). Through empirical analysis and rigorous experimentation, we seek to provide valuable insights for optimizing FL implementations in manufacturing, ensuring that collaborative efforts yield the most effective and reliable models with a limited number of participating clients. The findings from this study are expected to refine FL practices significantly in manufacturing, thereby enhancing the efficiency and performance of collaborative machine learning endeavors in this vital sector.

Title: Dynamic Label Injection for Imbalanced Industrial Defect Segmentation

• Authors: Emanuele Caruso, Francesco Pelosin, Alessandro Simoni, Marco Boschetti
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10031
• Pdf URL: https://arxiv.org/pdf/2408.10031
• Copy Paste: [[2408.10031]] Dynamic Label Injection for Imbalanced Industrial Defect Segmentation(https://arxiv.org/abs/2408.10031)
• Keywords: segmentation
• Abstract: In this work, we propose a simple yet effective method to tackle the problem of imbalanced multi-class semantic segmentation in deep learning systems. One of the key properties for a good training set is the balancing among the classes. When the input distribution is heavily imbalanced in the number of instances, the learning process could be hindered or difficult to carry on. To this end, we propose a Dynamic Label Injection (DLI) algorithm to impose a uniform distribution in the input batch. Our algorithm computes the current batch defect distribution and re-balances it by transferring defects using a combination of Poisson-based seamless image cloning and cut-paste techniques. A thorough experimental section on the Magnetic Tiles dataset shows better results of DLI compared to other balancing loss approaches also in the challenging weakly-supervised setup. The code is available at this https URL

Title: SHARP: Segmentation of Hands and Arms by Range using Pseudo-Depth for Enhanced Egocentric 3D Hand Pose Estimation and Action Recognition

• Authors: Wiktor Mucha, Michael Wray, Martin Kampel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10037
• Pdf URL: https://arxiv.org/pdf/2408.10037
• Copy Paste: [[2408.10037]] SHARP: Segmentation of Hands and Arms by Range using Pseudo-Depth for Enhanced Egocentric 3D Hand Pose Estimation and Action Recognition(https://arxiv.org/abs/2408.10037)
• Keywords: transformer, segmentation
• Abstract: Hand pose represents key information for action recognition in the egocentric perspective, where the user is interacting with objects. We propose to improve egocentric 3D hand pose estimation based on RGB frames only by using pseudo-depth images. Incorporating state-of-the-art single RGB image depth estimation techniques, we generate pseudo-depth representations of the frames and use distance knowledge to segment irrelevant parts of the scene. The resulting depth maps are then used as segmentation masks for the RGB frames. Experimental results on H2O Dataset confirm the high accuracy of the estimated pose with our method in an action recognition task. The 3D hand pose, together with information from object detection, is processed by a transformer-based action recognition network, resulting in an accuracy of 91.73%, outperforming all state-of-the-art methods. Estimations of 3D hand pose result in competitive performance with existing methods with a mean pose error of 28.66 mm. This method opens up new possibilities for employing distance information in egocentric 3D hand pose estimation without relying on depth sensors.

Title: Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory

• Authors: Haoran Li, Wei Fan, Yulin Chen, Jiayang Cheng, Tianshu Chu, Xuebing Zhou, Peizhao Hu, Yangqiu Song
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.10053
• Pdf URL: https://arxiv.org/pdf/2408.10053
• Copy Paste: [[2408.10053]] Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory(https://arxiv.org/abs/2408.10053)
• Keywords: privacy, defense, attack, large language model
• Abstract: Privacy research has attracted wide attention as individuals worry that their private data can be easily leaked during interactions with smart devices, social platforms, and AI applications. Computer science researchers, on the other hand, commonly study privacy issues through privacy attacks and defenses on segmented fields. Privacy research is conducted on various sub-fields, including Computer Vision (CV), Natural Language Processing (NLP), and Computer Networks. Within each field, privacy has its own formulation. Though pioneering works on attacks and defenses reveal sensitive privacy issues, they are narrowly trapped and cannot fully cover people's actual privacy concerns. Consequently, the research on general and human-centric privacy research remains rather unexplored. In this paper, we formulate the privacy issue as a reasoning problem rather than simple pattern matching. We ground on the Contextual Integrity (CI) theory which posits that people's perceptions of privacy are highly correlated with the corresponding social context. Based on such an assumption, we develop the first comprehensive checklist that covers social identities, private attributes, and existing privacy regulations. Unlike prior works on CI that either cover limited expert annotated norms or model incomplete social context, our proposed privacy checklist uses the whole Health Insurance Portability and Accountability Act of 1996 (HIPAA) as an example, to show that we can resort to large language models (LLMs) to completely cover the HIPAA's regulations. Additionally, our checklist also gathers expert annotations across multiple ontologies to determine private information including but not limited to personally identifiable information (PII). We use our preliminary results on the HIPAA to shed light on future context-centric privacy research to cover more privacy regulations, social norms and standards.

Title: Facial Wrinkle Segmentation for Cosmetic Dermatology: Pretraining with Texture Map-Based Weak Supervision

• Authors: Junho Moon, Haejun Chung, Ikbeom Jang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.10060
• Pdf URL: https://arxiv.org/pdf/2408.10060
• Copy Paste: [[2408.10060]] Facial Wrinkle Segmentation for Cosmetic Dermatology: Pretraining with Texture Map-Based Weak Supervision(https://arxiv.org/abs/2408.10060)
• Keywords: segmentation
• Abstract: Facial wrinkle detection plays a crucial role in cosmetic dermatology. Precise manual segmentation of facial wrinkles is challenging and time-consuming, with inherent subjectivity leading to inconsistent results among graders. To address this issue, we propose two solutions. First, we build and release the first public facial wrinkle dataset, `FFHQ-Wrinkle', an extension of the NVIDIA FFHQ dataset. This dataset includes 1,000 images with human labels and 50,000 images with automatically generated weak labels. This dataset can foster the research community to develop advanced wrinkle detection algorithms. Second, we introduce a training strategy for U-Net-like encoder-decoder models to detect wrinkles across the face automatically. Our method employs a two-stage training strategy: texture map pretraining and finetuning on human-labeled data. Initially, we pretrain models on a large dataset with weak labels (N=50k) or masked texture maps generated through computer vision techniques, without human intervention. Subsequently, we finetune the models using human-labeled data (N=1k), which consists of manually labeled wrinkle masks. During finetuning, the network inputs a combination of RGB and masked texture maps, comprising four channels. We effectively combine labels from multiple annotators to minimize subjectivity in manual labeling. Our strategies demonstrate improved segmentation performance in facial wrinkle segmentation both quantitatively and visually compared to existing pretraining methods.

Title: LNQ 2023 challenge: Benchmark of weakly-supervised techniques for mediastinal lymph node quantification

• Authors: Reuben Dorent, Roya Khajavi, Tagwa Idris, Erik Ziegler, Bhanusupriya Somarouthu, Heather Jacene, Ann LaCasce, Jonathan Deissler, Jan Ehrhardt, Sofija Engelson, Stefan M. Fischer, Yun Gu, Heinz Handels, Satoshi Kasai, Satoshi Kondo, Klaus Maier-Hein, Julia A. Schnabel, Guotai Wang, Litingyu Wang, Tassilo Wald, Guang-Zhong Yang, Hanxiao Zhang, Minghui Zhang, Steve Pieper, Gordon Harris, Ron Kikinis, Tina Kapur
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10069
• Pdf URL: https://arxiv.org/pdf/2408.10069
• Copy Paste: [[2408.10069]] LNQ 2023 challenge: Benchmark of weakly-supervised techniques for mediastinal lymph node quantification(https://arxiv.org/abs/2408.10069)
• Keywords: robust, segmentation
• Abstract: Accurate assessment of lymph node size in 3D CT scans is crucial for cancer staging, therapeutic management, and monitoring treatment response. Existing state-of-the-art segmentation frameworks in medical imaging often rely on fully annotated datasets. However, for lymph node segmentation, these datasets are typically small due to the extensive time and expertise required to annotate the numerous lymph nodes in 3D CT scans. Weakly-supervised learning, which leverages incomplete or noisy annotations, has recently gained interest in the medical imaging community as a potential solution. Despite the variety of weakly-supervised techniques proposed, most have been validated only on private datasets or small publicly available datasets. To address this limitation, the Mediastinal Lymph Node Quantification (LNQ) challenge was organized in conjunction with the 26th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2023). This challenge aimed to advance weakly-supervised segmentation methods by providing a new, partially annotated dataset and a robust evaluation framework. A total of 16 teams from 5 countries submitted predictions to the validation leaderboard, and 6 teams from 3 countries participated in the evaluation phase. The results highlighted both the potential and the current limitations of weakly-supervised approaches. On one hand, weakly-supervised approaches obtained relatively good performance with a median Dice score of $61.0\%$. On the other hand, top-ranked teams, with a median Dice score exceeding $70\%$, boosted their performance by leveraging smaller but fully annotated datasets to combine weak supervision and full supervision. This highlights both the promise of weakly-supervised methods and the ongoing need for high-quality, fully annotated data to achieve higher segmentation performance.

Title: FFAA: Multimodal Large Language Model based Explainable Open-World Face Forgery Analysis Assistant

• Authors: Zhengchao Huang, Bin Xia, Zicheng Lin, Zhun Mou, Wenming Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.10072
• Pdf URL: https://arxiv.org/pdf/2408.10072
• Copy Paste: [[2408.10072]] FFAA: Multimodal Large Language Model based Explainable Open-World Face Forgery Analysis Assistant(https://arxiv.org/abs/2408.10072)
• Keywords: security, robust, large language model
• Abstract: The rapid advancement of deepfake technologies has sparked widespread public concern, particularly as face forgery poses a serious threat to public information security. However, the unknown and diverse forgery techniques, varied facial features and complex environmental factors pose significant challenges for face forgery analysis. Existing datasets lack descriptions of these aspects, making it difficult for models to distinguish between real and forged faces using only visual information amid various confounding factors. In addition, existing methods do not yield user-friendly and explainable results, complicating the understanding of the model's decision-making process. To address these challenges, we introduce a novel Open-World Face Forgery Analysis VQA (OW-FFA-VQA) task and the corresponding benchmark. To tackle this task, we first establish a dataset featuring a diverse collection of real and forged face images with essential descriptions and reliable forgery reasoning. Base on this dataset, we introduce FFAA: Face Forgery Analysis Assistant, consisting of a fine-tuned Multimodal Large Language Model (MLLM) and Multi-answer Intelligent Decision System (MIDS). By integrating hypothetical prompts with MIDS, the impact of fuzzy classification boundaries is effectively mitigated, enhancing the model's robustness. Extensive experiments demonstrate that our method not only provides user-friendly explainable results but also significantly boosts accuracy and robustness compared to previous methods.

Title: Federated Frank-Wolfe Algorithm

• Authors: Ali Dadras, Sourasekhar Banerjee, Karthik Prakhya, Alp Yurtsever
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.10090
• Pdf URL: https://arxiv.org/pdf/2408.10090
• Copy Paste: [[2408.10090]] Federated Frank-Wolfe Algorithm(https://arxiv.org/abs/2408.10090)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) has gained a lot of attention in recent years for building privacy-preserving collaborative learning systems. However, FL algorithms for constrained machine learning problems are still limited, particularly when the projection step is costly. To this end, we propose a Federated Frank-Wolfe Algorithm (FedFW). FedFW features data privacy, low per-iteration cost, and communication of sparse signals. In the deterministic setting, FedFW achieves an $\varepsilon$-suboptimal solution within $O(\varepsilon^{-2})$ iterations for smooth and convex objectives, and $O(\varepsilon^{-3})$ iterations for smooth but non-convex objectives. Furthermore, we present a stochastic variant of FedFW and show that it finds a solution within $O(\varepsilon^{-3})$ iterations in the convex setting. We demonstrate the empirical performance of FedFW on several machine learning tasks.

Title: GLIMMER: Incorporating Graph and Lexical Features in Unsupervised Multi-Document Summarization

• Authors: Ran Liu, Ming Liu, Min Yu, Jianguo Jiang, Gang Li, Dan Zhang, Jingyuan Li, Xiang Meng, Weiqing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.10115
• Pdf URL: https://arxiv.org/pdf/2408.10115
• Copy Paste: [[2408.10115]] GLIMMER: Incorporating Graph and Lexical Features in Unsupervised Multi-Document Summarization(https://arxiv.org/abs/2408.10115)
• Keywords: extraction
• Abstract: Pre-trained language models are increasingly being used in multi-document summarization tasks. However, these models need large-scale corpora for pre-training and are domain-dependent. Other non-neural unsupervised summarization approaches mostly rely on key sentence extraction, which can lead to information loss. To address these challenges, we propose a lightweight yet effective unsupervised approach called GLIMMER: a Graph and LexIcal features based unsupervised Multi-docuMEnt summaRization approach. It first constructs a sentence graph from the source documents, then automatically identifies semantic clusters by mining low-level features from raw texts, thereby improving intra-cluster correlation and the fluency of generated sentences. Finally, it summarizes clusters into natural sentences. Experiments conducted on Multi-News, Multi-XScience and DUC-2004 demonstrate that our approach outperforms existing unsupervised approaches. Furthermore, it surpasses state-of-the-art pre-trained multi-document summarization models (e.g. PEGASUS and PRIMERA) under zero-shot settings in terms of ROUGE scores. Additionally, human evaluations indicate that summaries generated by GLIMMER achieve high readability and informativeness scores. Our code is available at this https URL.

Title: Molecular Graph Representation Learning Integrating Large Language Models with Domain-specific Small Models

• Authors: Tianyu Zhang, Yuxiang Ren, Chengbin Hou, Hairong Lv, Xuegong Zhang
• Subjects: cs.LG, cs.AI, cs.IR, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2408.10124
• Pdf URL: https://arxiv.org/pdf/2408.10124
• Copy Paste: [[2408.10124]] Molecular Graph Representation Learning Integrating Large Language Models with Domain-specific Small Models(https://arxiv.org/abs/2408.10124)
• Keywords: large language model
• Abstract: Molecular property prediction is a crucial foundation for drug discovery. In recent years, pre-trained deep learning models have been widely applied to this task. Some approaches that incorporate prior biological domain knowledge into the pre-training framework have achieved impressive results. However, these methods heavily rely on biochemical experts, and retrieving and summarizing vast amounts of domain knowledge literature is both time-consuming and expensive. Large Language Models (LLMs) have demonstrated remarkable performance in understanding and efficiently providing general knowledge. Nevertheless, they occasionally exhibit hallucinations and lack precision in generating domain-specific knowledge. Conversely, Domain-specific Small Models (DSMs) possess rich domain knowledge and can accurately calculate molecular domain-related metrics. However, due to their limited model size and singular functionality, they lack the breadth of knowledge necessary for comprehensive representation learning. To leverage the advantages of both approaches in molecular property prediction, we propose a novel Molecular Graph representation learning framework that integrates Large language models and Domain-specific small models (MolGraph-LarDo). Technically, we design a two-stage prompt strategy where DSMs are introduced to calibrate the knowledge provided by LLMs, enhancing the accuracy of domain-specific information and thus enabling LLMs to generate more precise textual descriptions for molecular samples. Subsequently, we employ a multi-modal alignment method to coordinate various modalities, including molecular graphs and their corresponding descriptive texts, to guide the pre-training of molecular representations. Extensive experiments demonstrate the effectiveness of the proposed method.

Title: Video Object Segmentation via SAM 2: The 4th Solution for LSVOS Challenge VOS Track

• Authors: Feiyu Pan, Hao Fang, Runmin Cong, Wei Zhang, Xiankai Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10125
• Pdf URL: https://arxiv.org/pdf/2408.10125
• Copy Paste: [[2408.10125]] Video Object Segmentation via SAM 2: The 4th Solution for LSVOS Challenge VOS Track(https://arxiv.org/abs/2408.10125)
• Keywords: transformer, segmentation
• Abstract: Video Object Segmentation (VOS) task aims to segmenting a particular object instance throughout the entire video sequence given only the object mask of the first frame. Recently, Segment Anything Model 2 (SAM 2) is proposed, which is a foundation model towards solving promptable visual segmentation in images and videos. SAM 2 builds a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. SAM 2 is a simple transformer architecture with streaming memory for real-time video processing, which trained on the date provides strong performance across a wide range of tasks. In this work, we evaluate the zero-shot performance of SAM 2 on the more challenging VOS datasets MOSE and LVOS. Without fine-tuning on the training set, SAM 2 achieved 75.79 J&F on the test set and ranked 4th place for 6th LSVOS Challenge VOS Track.

Title: UNINEXT-Cutie: The 1st Solution for LSVOS Challenge RVOS Track

• Authors: Hao Fang, Feiyu Pan, Xiankai Lu, Wei Zhang, Runmin Cong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10129
• Pdf URL: https://arxiv.org/pdf/2408.10129
• Copy Paste: [[2408.10129]] UNINEXT-Cutie: The 1st Solution for LSVOS Challenge RVOS Track(https://arxiv.org/abs/2408.10129)
• Keywords: segmentation
• Abstract: Referring video object segmentation (RVOS) relies on natural language expressions to segment target objects in video. In this year, LSVOS Challenge RVOS Track replaced the origin YouTube-RVOS benchmark with MeViS. MeViS focuses on referring the target object in a video through its motion descriptions instead of static attributes, posing a greater challenge to RVOS task. In this work, we integrate strengths of that leading RVOS and VOS models to build up a simple and effective pipeline for RVOS. Firstly, We finetune the state-of-the-art RVOS model to obtain mask sequences that are correlated with language descriptions. Secondly, based on a reliable and high-quality key frames, we leverage VOS model to enhance the quality and temporal consistency of the mask results. Finally, we further improve the performance of the RVOS model using semi-supervised learning. Our solution achieved 62.57 J&F on the MeViS test set and ranked 1st place for 6th LSVOS Challenge RVOS Track.

Title: $R^2$-Mesh: Reinforcement Learning Powered Mesh Reconstruction via Geometry and Appearance Refinement

• Authors: Haoyang Wang, Liming Liu, Quanlu Jia, Jiangkai Wu, Haodan Zhang, Peiheng Wang, Xinggong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10135
• Pdf URL: https://arxiv.org/pdf/2408.10135
• Copy Paste: [[2408.10135]] $R^2$-Mesh: Reinforcement Learning Powered Mesh Reconstruction via Geometry and Appearance Refinement(https://arxiv.org/abs/2408.10135)
• Keywords: robust, extraction
• Abstract: Mesh reconstruction based on Neural Radiance Fields (NeRF) is popular in a variety of applications such as computer graphics, virtual reality, and medical imaging due to its efficiency in handling complex geometric structures and facilitating real-time rendering. However, existing works often fail to capture fine geometric details accurately and struggle with optimizing rendering quality. To address these challenges, we propose a novel algorithm that progressively generates and optimizes meshes from multi-view images. Our approach initiates with the training of a NeRF model to establish an initial Signed Distance Field (SDF) and a view-dependent appearance field. Subsequently, we iteratively refine the SDF through a differentiable mesh extraction method, continuously updating both the vertex positions and their connectivity based on the loss from mesh differentiable rasterization, while also optimizing the appearance representation. To further leverage high-fidelity and detail-rich representations from NeRF, we propose an online-learning strategy based on Upper Confidence Bound (UCB) to enhance viewpoints by adaptively incorporating images rendered by the initial NeRF model into the training dataset. Through extensive experiments, we demonstrate that our method delivers highly competitive and robust performance in both mesh rendering quality and geometric quality.

Title: Instruction Finetuning for Leaderboard Generation from Empirical AI Research

• Authors: Salomon Kabongo, Jennifer D'Souza
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.10141
• Pdf URL: https://arxiv.org/pdf/2408.10141
• Copy Paste: [[2408.10141]] Instruction Finetuning for Leaderboard Generation from Empirical AI Research(https://arxiv.org/abs/2408.10141)
• Keywords: extraction, generative, large language model
• Abstract: This study demonstrates the application of instruction finetuning of pretrained Large Language Models (LLMs) to automate the generation of AI research leaderboards, extracting (Task, Dataset, Metric, Score) quadruples from articles. It aims to streamline the dissemination of advancements in AI research by transitioning from traditional, manual community curation, or otherwise taxonomy-constrained natural language inference (NLI) models, to an automated, generative LLM-based approach. Utilizing the FLAN-T5 model, this research enhances LLMs' adaptability and reliability in information extraction, offering a novel method for structured knowledge representation.

Title: Multi-Scale Representation Learning for Image Restoration with State-Space Model

• Authors: Yuhong He, Long Peng, Qiaosi Yi, Chen Wu, Lu Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10145
• Pdf URL: https://arxiv.org/pdf/2408.10145
• Copy Paste: [[2408.10145]] Multi-Scale Representation Learning for Image Restoration with State-Space Model(https://arxiv.org/abs/2408.10145)
• Keywords: extraction, transformer
• Abstract: Image restoration endeavors to reconstruct a high-quality, detail-rich image from a degraded counterpart, which is a pivotal process in photography and various computer vision systems. In real-world scenarios, different types of degradation can cause the loss of image details at various scales and degrade image contrast. Existing methods predominantly rely on CNN and Transformer to capture multi-scale representations. However, these methods are often limited by the high computational complexity of Transformers and the constrained receptive field of CNN, which hinder them from achieving superior performance and efficiency in image restoration. To address these challenges, we propose a novel Multi-Scale State-Space Model-based (MS-Mamba) for efficient image restoration that enhances the capacity for multi-scale representation learning through our proposed global and regional SSM modules. Additionally, an Adaptive Gradient Block (AGB) and a Residual Fourier Block (RFB) are proposed to improve the network's detail extraction capabilities by capturing gradients in various directions and facilitating learning details in the frequency domain. Extensive experiments on nine public benchmarks across four classic image restoration tasks, image deraining, dehazing, denoising, and low-light enhancement, demonstrate that our proposed method achieves new state-of-the-art performance while maintaining low computational complexity. The source code will be publicly available.

Title: In-Context Learning with Representations: Contextual Generalization of Trained Transformers

• Authors: Tong Yang, Yu Huang, Yingbin Liang, Yuejie Chi
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.10147
• Pdf URL: https://arxiv.org/pdf/2408.10147
• Copy Paste: [[2408.10147]] In-Context Learning with Representations: Contextual Generalization of Trained Transformers(https://arxiv.org/abs/2408.10147)
• Keywords: transformer, large language model
• Abstract: In-context learning (ICL) refers to a remarkable capability of pretrained large language models, which can learn a new task given a few examples during inference. However, theoretical understanding of ICL is largely under-explored, particularly whether transformers can be trained to generalize to unseen examples in a prompt, which will require the model to acquire contextual knowledge of the prompt for generalization. This paper investigates the training dynamics of transformers by gradient descent through the lens of non-linear regression tasks. The contextual generalization here can be attained via learning the template function for each task in-context, where all template functions lie in a linear space with $m$ basis functions. We analyze the training dynamics of one-layer multi-head transformers to in-contextly predict unlabeled inputs given partially labeled prompts, where the labels contain Gaussian noise and the number of examples in each prompt are not sufficient to determine the template. Under mild assumptions, we show that the training loss for a one-layer multi-head transformer converges linearly to a global minimum. Moreover, the transformer effectively learns to perform ridge regression over the basis functions. To our knowledge, this study is the first provable demonstration that transformers can learn contextual (i.e., template) information to generalize to both unseen examples and tasks when prompts contain only a small number of query-answer pairs.

Title: Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models

• Authors: Amey Hengle, Prasoon Bajpai, Soham Dan, Tanmoy Chakraborty
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.10151
• Pdf URL: https://arxiv.org/pdf/2408.10151
• Copy Paste: [[2408.10151]] Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models(https://arxiv.org/abs/2408.10151)
• Keywords: large language model
• Abstract: While recent large language models (LLMs) demonstrate remarkable abilities in responding to queries in diverse languages, their ability to handle long multilingual contexts is unexplored. As such, a systematic evaluation of the long-context capabilities of LLMs in multilingual settings is crucial, specifically in the context of information retrieval. To address this gap, we introduce the MultiLingual Needle-in-a-Haystack (MLNeedle) test, designed to assess a model's ability to retrieve relevant information (the needle) from a collection of multilingual distractor texts (the haystack). This test serves as an extension of the multilingual question-answering task, encompassing both monolingual and cross-lingual retrieval. We evaluate four state-of-the-art LLMs on MLNeedle. Our findings reveal that model performance can vary significantly with language and needle position. Specifically, we observe that model performance is the lowest when the needle is (i) in a language outside the English language family and (ii) located in the middle of the input context. Furthermore, although some models claim a context size of $8k$ tokens or greater, none demonstrate satisfactory cross-lingual retrieval performance as the context length increases. Our analysis provides key insights into the long-context behavior of LLMs in multilingual settings to guide future evaluation protocols. To our knowledge, this is the first study to investigate the multilingual long-context behavior of LLMs.

Title: LoopSplat: Loop Closure by Registering 3D Gaussian Splats

• Authors: Liyuan Zhu, Yue Li, Erik Sandström, Konrad Schindler, Iro Armeni
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.10154
• Pdf URL: https://arxiv.org/pdf/2408.10154
• Copy Paste: [[2408.10154]] LoopSplat: Loop Closure by Registering 3D Gaussian Splats(https://arxiv.org/abs/2408.10154)
• Keywords: robust
• Abstract: Simultaneous Localization and Mapping (SLAM) based on 3D Gaussian Splats (3DGS) has recently shown promise towards more accurate, dense 3D scene maps. However, existing 3DGS-based methods fail to address the global consistency of the scene via loop closure and/or global bundle adjustment. To this end, we propose LoopSplat, which takes RGB-D images as input and performs dense mapping with 3DGS submaps and frame-to-model tracking. LoopSplat triggers loop closure online and computes relative loop edge constraints between submaps directly via 3DGS registration, leading to improvements in efficiency and accuracy over traditional global-to-local point cloud registration. It uses a robust pose graph optimization formulation and rigidly aligns the submaps to achieve global consistency. Evaluation on the synthetic Replica and real-world TUM-RGBD, ScanNet, and ScanNet++ datasets demonstrates competitive or superior tracking, mapping, and rendering compared to existing methods for dense RGB-D SLAM. Code is available at \href{this https URL}{this http URL}.

Title: SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models

• Authors: Anke Tang, Li Shen, Yong Luo, Shuai Xie, Han Hu, Lefei Zhang, Bo Du, Dacheng Tao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.10174
• Pdf URL: https://arxiv.org/pdf/2408.10174
• Copy Paste: [[2408.10174]] SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models(https://arxiv.org/abs/2408.10174)
• Keywords: interpretability, large language model
• Abstract: Deep model training on extensive datasets is increasingly becoming cost-prohibitive, prompting the widespread adoption of deep model fusion techniques to leverage knowledge from pre-existing models. From simple weight averaging to more sophisticated methods like AdaMerging, model fusion effectively improves model performance and accelerates the development of new models. However, potential interference between parameters of individual models and the lack of interpretability in the fusion progress remain significant challenges. Existing methods often try to resolve the parameter interference issue by evaluating attributes of parameters, such as their magnitude or sign, or by parameter pruning. In this study, we begin by examining the fine-tuning of linear layers through the lens of subspace analysis and explicitly define parameter interference as an optimization problem to shed light on this subject. Subsequently, we introduce an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction, which allows for the upscaling of source models into an MoE model without extra data or further training. Our approach relies on the observation that fine-tuning mostly keeps the important parts from the pre-training, but it uses less significant or unused areas to adapt to new tasks. Also, the issue of parameter interference, which is intrinsically intractable in the original parameter space, can be managed by expanding the dimensions. We conduct extensive experiments across diverse scenarios, such as image classification and text generalization tasks, using full fine-tuning and LoRA fine-tuning, and we apply our method to large language models (CLIP models, Flan-T5 models, and Mistral-7B models), highlighting the adaptability and scalability of SMILE. Code is available at this https URL

Title: Fairness Under Cover: Evaluating the Impact of Occlusions on Demographic Bias in Facial Recognition

• Authors: Rafael M. Mamede, Pedro C. Neto, Ana F. Sequeira
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.10175
• Pdf URL: https://arxiv.org/pdf/2408.10175
• Copy Paste: [[2408.10175]] Fairness Under Cover: Evaluating the Impact of Occlusions on Demographic Bias in Facial Recognition(https://arxiv.org/abs/2408.10175)
• Keywords: fair
• Abstract: This study investigates the effects of occlusions on the fairness of face recognition systems, particularly focusing on demographic biases. Using the Racial Faces in the Wild (RFW) dataset and synthetically added realistic occlusions, we evaluate their effect on the performance of face recognition models trained on the BUPT-Balanced and BUPT-GlobalFace datasets. We note increases in the dispersion of FMR, FNMR, and accuracy alongside decreases in fairness according to Equilized Odds, Demographic Parity, STD of Accuracy, and Fairness Discrepancy Rate. Additionally, we utilize a pixel attribution method to understand the importance of occlusions in model predictions, proposing a new metric, Face Occlusion Impact Ratio (FOIR), that quantifies the extent to which occlusions affect model performance across different demographic groups. Our results indicate that occlusions exacerbate existing demographic biases, with models placing higher importance on occlusions in an unequal fashion, particularly affecting African individuals more severely.

Title: Imbalance-Aware Culvert-Sewer Defect Segmentation Using an Enhanced Feature Pyramid Network

• Authors: Rasha Alshawi, Md Meftahul Ferdaus, Mahdi Abdelguerfi, Kendall Niles, Ken Pathak, Steve Sloan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.10181
• Pdf URL: https://arxiv.org/pdf/2408.10181
• Copy Paste: [[2408.10181]] Imbalance-Aware Culvert-Sewer Defect Segmentation Using an Enhanced Feature Pyramid Network(https://arxiv.org/abs/2408.10181)
• Keywords: extraction, segmentation
• Abstract: Imbalanced datasets are a significant challenge in real-world scenarios. They lead to models that underperform on underrepresented classes, which is a critical issue in infrastructure inspection. This paper introduces the Enhanced Feature Pyramid Network (E-FPN), a deep learning model for the semantic segmentation of culverts and sewer pipes within imbalanced datasets. The E-FPN incorporates architectural innovations like sparsely connected blocks and depth-wise separable convolutions to improve feature extraction and handle object variations. To address dataset imbalance, the model employs strategies like class decomposition and data augmentation. Experimental results on the culvert-sewer defects dataset and a benchmark aerial semantic segmentation drone dataset show that the E-FPN outperforms state-of-the-art methods, achieving an average Intersection over Union (IoU) improvement of 13.8% and 27.2%, respectively. Additionally, class decomposition and data augmentation together boost the model's performance by approximately 6.9% IoU. The proposed E-FPN presents a promising solution for enhancing object segmentation in challenging, multi-class real-world datasets, with potential applications extending beyond culvert-sewer defect detection.

Title: Assessment of Spectral based Solutions for the Detection of Floating Marine Debris

• Authors: Muhammad Alì, Francesca Razzano, Sergio Vitale, Giampaolo Ferraioli, Vito Pascazio, Gilda Schirinzi, Silvia Ullo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10187
• Pdf URL: https://arxiv.org/pdf/2408.10187
• Copy Paste: [[2408.10187]] Assessment of Spectral based Solutions for the Detection of Floating Marine Debris(https://arxiv.org/abs/2408.10187)
• Keywords: fair
• Abstract: Typically, the detection of marine debris relies on in-situ campaigns that are characterized by huge human effort and limited spatial coverage. Following the need of a rapid solution for the detection of floating plastic, methods based on remote sensing data have been proposed recently. Their main limitation is represented by the lack of a general reference for evaluating performance. Recently, the Marine Debris Archive (MARIDA) has been released as a standard dataset to develop and evaluate Machine Learning (ML) algorithms for detection of Marine Plastic Debris. The MARIDA dataset has been created for simplifying the comparison between detection solutions with the aim of stimulating the research in the field of marine environment preservation. In this work, an assessment of spectral based solutions is proposed by evaluating performance on MARIDA dataset. The outcome highlights the need of precise reference for fair evaluation.

Title: LongVILA: Scaling Long-Context Visual Language Models for Long Videos

• Authors: Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, Song Han
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.10188
• Pdf URL: https://arxiv.org/pdf/2408.10188
• Copy Paste: [[2408.10188]] LongVILA: Scaling Long-Context Visual Language Models for Long Videos(https://arxiv.org/abs/2408.10188)
• Keywords: transformer
• Abstract: Long-context capability is critical for multi-modal foundation models. We introduce LongVILA, a full-stack solution for long-context vision-language models, including system, model training, and dataset development. On the system side, we introduce the first Multi-Modal Sequence Parallelism (MM-SP) system that enables long-context training and inference, enabling 2M context length training on 256 GPUs. MM-SP is also efficient, being 2.1x - 5.7x faster than Ring-Style Sequence Parallelism and 1.1x - 1.4x faster than Megatron-LM in text-only settings. Moreover, it seamlessly integrates with Hugging Face Transformers. For model training, we propose a five-stage pipeline comprising alignment, pre-training, context extension, and long-short joint supervised fine-tuning. Regarding datasets, we meticulously construct large-scale visual language pre-training datasets and long video instruction-following datasets to support our multi-stage training process. The full-stack solution extends the feasible frame number of VILA by a factor of 128 (from 8 to 1024 frames) and improves long video captioning score from 2.00 to 3.26 (1.6x), achieving 99.5% accuracy in 1400-frames video (274k context length) needle in a haystack. LongVILA-8B also demonstrates a consistent improvement in performance on long videos within the VideoMME benchmark as the video frames increase.

Title: Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models

• Authors: Aviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter, Albert Gu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.10189
• Pdf URL: https://arxiv.org/pdf/2408.10189
• Copy Paste: [[2408.10189]] Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models(https://arxiv.org/abs/2408.10189)
• Keywords: transformer
• Abstract: Transformer architectures have become a dominant paradigm for domains like language modeling but suffer in many inference settings due to their quadratic-time self-attention. Recently proposed subquadratic architectures, such as Mamba, have shown promise, but have been pretrained with substantially less computational resources than the strongest Transformer models. In this work, we present a method that is able to distill a pretrained Transformer architecture into alternative architectures such as state space models (SSMs). The key idea to our approach is that we can view both Transformers and SSMs as applying different forms of mixing matrices over the token sequences. We can thus progressively distill the Transformer architecture by matching different degrees of granularity in the SSM: first matching the mixing matrices themselves, then the hidden units at each block, and finally the end-to-end predictions. Our method, called MOHAWK, is able to distill a Mamba-2 variant based on the Phi-1.5 architecture (Phi-Mamba) using only 3B tokens and a hybrid version (Hybrid Phi-Mamba) using 5B tokens. Despite using less than 1% of the training data typically used to train models from scratch, Phi-Mamba boasts substantially stronger performance compared to all past open-source non-Transformer models. MOHAWK allows models like SSMs to leverage computational resources invested in training Transformer-based architectures, highlighting a new avenue for building such models.

Title: SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Views

• Authors: Chao Xu, Ang Li, Linghao Chen, Yulin Liu, Ruoxi Shi, Hao Su, Minghua Liu
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.10195
• Pdf URL: https://arxiv.org/pdf/2408.10195
• Copy Paste: [[2408.10195]] SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Views(https://arxiv.org/abs/2408.10195)
• Keywords: diffusion
• Abstract: Open-world 3D generation has recently attracted considerable attention. While many single-image-to-3D methods have yielded visually appealing outcomes, they often lack sufficient controllability and tend to produce hallucinated regions that may not align with users' expectations. In this paper, we explore an important scenario in which the input consists of one or a few unposed 2D images of a single object, with little or no overlap. We propose a novel method, SpaRP, to reconstruct a 3D textured mesh and estimate the relative camera poses for these sparse-view images. SpaRP distills knowledge from 2D diffusion models and finetunes them to implicitly deduce the 3D spatial relationships between the sparse views. The diffusion model is trained to jointly predict surrogate representations for camera poses and multi-view images of the object under known poses, integrating all information from the input sparse views. These predictions are then leveraged to accomplish 3D reconstruction and pose estimation, and the reconstructed 3D model can be used to further refine the camera poses of input views. Through extensive experiments on three datasets, we demonstrate that our method not only significantly outperforms baseline methods in terms of 3D reconstruction quality and pose prediction accuracy but also exhibits strong efficiency. It requires only about 20 seconds to produce a textured mesh and camera poses for the input views. Project page: this https URL.

Title: MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model

• Authors: Minghua Liu, Chong Zeng, Xinyue Wei, Ruoxi Shi, Linghao Chen, Chao Xu, Mengqi Zhang, Zhaoning Wang, Xiaoshuai Zhang, Isabella Liu, Hongzhi Wu, Hao Su
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.10198
• Pdf URL: https://arxiv.org/pdf/2408.10198
• Copy Paste: [[2408.10198]] MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model(https://arxiv.org/abs/2408.10198)
• Keywords: diffusion, transformer
• Abstract: Open-world 3D reconstruction models have recently garnered significant attention. However, without sufficient 3D inductive bias, existing methods typically entail expensive training costs and struggle to extract high-quality 3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision. Specifically, instead of using a triplane representation, we store features in 3D sparse voxels and combine transformers with 3D convolutions to leverage an explicit 3D structure and projective bias. In addition to sparse-view RGB input, we require the network to take input and generate corresponding normal maps. The input normal maps can be predicted by 2D diffusion models, significantly aiding in the guidance and refinement of the geometry's learning. Moreover, by combining Signed Distance Function (SDF) supervision with surface rendering, we directly learn to generate high-quality meshes without the need for complex multi-stage training processes. By incorporating these explicit 3D biases, MeshFormer can be trained efficiently and deliver high-quality textured meshes with fine-grained geometric details. It can also be integrated with 2D diffusion models to enable fast single-image-to-3D and text-to-3D tasks. Project page: this https URL

Title: SoK: Runtime Integrity

• Authors: Mahmoud Ammar, Adam Caulfield, Ivan De Oliveira Nunes
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.10200
• Pdf URL: https://arxiv.org/pdf/2408.10200
• Copy Paste: [[2408.10200]] SoK: Runtime Integrity(https://arxiv.org/abs/2408.10200)
• Keywords: defense
• Abstract: This paper provides a systematic exploration of runtime integrity mechanisms, such as Control Flow Integrity (CFI) and Control Flow Attestation (CFA). It examines their differences and relationships while addressing crucial questions about the goals, assumptions, features, and design spaces. It includes examining a potential coexistence of CFI and CFA on the same platform. Through a comprehensive review of existing defenses, this paper positions CFI and CFA within the broader landscape of runtime defenses, critically evaluating their strengths, limitations, and trade-offs. The findings emphasize the importance of further research to bridge the gaps between CFI and CFA, advancing the field of runtime defenses.

Title: SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP

• Authors: Yusuke Hirota, Min-Hung Chen, Chien-Yi Wang, Yuta Nakashima, Yu-Chiang Frank Wang, Ryo Hachiuma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10202
• Pdf URL: https://arxiv.org/pdf/2408.10202
• Copy Paste: [[2408.10202]] SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP(https://arxiv.org/abs/2408.10202)
• Keywords: protect
• Abstract: Large-scale vision-language models, such as CLIP, are known to contain harmful societal bias regarding protected attributes (e.g., gender and age). In this paper, we aim to address the problems of societal bias in CLIP. Although previous studies have proposed to debias societal bias through adversarial learning or test-time projecting, our comprehensive study of these works identifies two critical limitations: 1) loss of attribute information when it is explicitly disclosed in the input and 2) use of the attribute annotations during debiasing process. To mitigate societal bias in CLIP and overcome these limitations simultaneously, we introduce a simple-yet-effective debiasing method called SANER (societal attribute neutralizer) that eliminates attribute information from CLIP text features only of attribute-neutral descriptions. Experimental results show that SANER, which does not require attribute annotations and preserves original information for attribute-specific descriptions, demonstrates superior debiasing ability than the existing methods.

Title: Criticality Leveraged Adversarial Training (CLAT) for Boosted Performance via Parameter Efficiency

• Authors: Bhavna Gopal, Huanrui Yang, Jingyang Zhang, Mark Horton, Yiran Chen
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.10204
• Pdf URL: https://arxiv.org/pdf/2408.10204
• Copy Paste: [[2408.10204]] Criticality Leveraged Adversarial Training (CLAT) for Boosted Performance via Parameter Efficiency(https://arxiv.org/abs/2408.10204)
• Keywords: robust
• Abstract: Adversarial training enhances neural network robustness but suffers from a tendency to overfit and increased generalization errors on clean data. This work introduces CLAT, an innovative approach that mitigates adversarial overfitting by introducing parameter efficiency into the adversarial training process, improving both clean accuracy and adversarial robustness. Instead of tuning the entire model, CLAT identifies and fine-tunes robustness-critical layers - those predominantly learning non-robust features - while freezing the remaining model to enhance robustness. It employs dynamic critical layer selection to adapt to changes in layer criticality throughout the fine-tuning process. Empirically, CLAT can be applied on top of existing adversarial training methods, significantly reduces the number of trainable parameters by approximately 95%, and achieves more than a 2% improvement in adversarial robustness compared to baseline methods.