2024-10-24

Title: Federated brain tumor segmentation: an extensive benchmark

• Authors: Matthis Manthe (LIRIS, CREATIS), Stefan Duffner (LIRIS), Carole Lartizien (MYRIAD)
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.17265
• Pdf URL: https://arxiv.org/pdf/2410.17265
• Copy Paste: [[2410.17265]] Federated brain tumor segmentation: an extensive benchmark(https://arxiv.org/abs/2410.17265)
• Keywords: privacy, federate, segmentation
• Abstract: Recently, federated learning has raised increasing interest in the medical image analysis field due to its ability to aggregate multi-center data with privacy-preserving properties. A large amount of federated training schemes have been published, which we categorize into global (one final model), personalized (one model per institution) or hybrid (one model per cluster of institutions) methods. However, their applicability on the recently published Federated Brain Tumor Segmentation 2022 dataset has not been explored yet. We propose an extensive benchmark of federated learning algorithms from all three classes on this task. While standard FedAvg already performs very well, we show that some methods from each category can bring a slight performance improvement and potentially limit the final model(s) bias toward the predominant data distribution of the federation. Moreover, we provide a deeper understanding of the behaviour of federated learning on this task through alternative ways of distributing the pooled dataset among institutions, namely an Independent and Identical Distributed (IID) setup, and a limited data setup.

Title: Zero-Shot Vision-and-Language Navigation with Collision Mitigation in Continuous Environment

• Authors: Seongjun Jeong, Gi-Cheon Kang, Joochan Kim, Byoung-Tak Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17267
• Pdf URL: https://arxiv.org/pdf/2410.17267
• Copy Paste: [[2410.17267]] Zero-Shot Vision-and-Language Navigation with Collision Mitigation in Continuous Environment(https://arxiv.org/abs/2410.17267)
• Keywords: large language model
• Abstract: We propose the zero-shot Vision-and-Language Navigation with Collision Mitigation (VLN-CM), which takes these considerations. VLN-CM is composed of four modules and predicts the direction and distance of the next movement at each step. We utilize large foundation models for each modules. To select the direction, we use the Attention Spot Predictor (ASP), View Selector (VS), and Progress Monitor (PM). The ASP employs a Large Language Model (e.g. ChatGPT) to split navigation instructions into attention spots, which are objects or scenes at the location to move to (e.g. a yellow door). The VS selects from panorama images provided at 30-degree intervals the one that includes the attention spot, using CLIP similarity. We then choose the angle of the selected image as the direction to move in. The PM uses a rule-based approach to decide which attention spot to focus on next, among multiple spots derived from the instructions. If the similarity between the current attention spot and the visual observations decreases consecutively at each step, the PM determines that the agent has passed the current spot and moves on to the next one. For selecting the distance to move, we employed the Open Map Predictor (OMP). The OMP uses panorama depth information to predict an occupancy mask. We then selected a collision-free distance in the predicted direction based on the occupancy mask. We evaluated our method using the validation data of VLN-CE. Our approach showed better performance than several baseline methods, and the OPM was effective in mitigating collisions for the agent.

Title: Enhancing Robustness and Efficiency of Least Square Twin SVM via Granular Computing

• Authors: M. Tanveer, R. K. Sharma, A. Quadir, M. Sajid
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17338
• Pdf URL: https://arxiv.org/pdf/2410.17338
• Copy Paste: [[2410.17338]] Enhancing Robustness and Efficiency of Least Square Twin SVM via Granular Computing(https://arxiv.org/abs/2410.17338)
• Keywords: robust
• Abstract: In the domain of machine learning, least square twin support vector machine (LSTSVM) stands out as one of the state-of-the-art models. However, LSTSVM suffers from sensitivity to noise and outliers, overlooking the SRM principle and instability in resampling. Moreover, its computational complexity and reliance on matrix inversions hinder the efficient processing of large datasets. As a remedy to the aforementioned challenges, we propose the robust granular ball LSTSVM (GBLSTSVM). GBLSTSVM is trained using granular balls instead of original data points. The core of a granular ball is found at its center, where it encapsulates all the pertinent information of the data points within the ball of specified radius. To improve scalability and efficiency, we further introduce the large-scale GBLSTSVM (LS-GBLSTSVM), which incorporates the SRM principle through regularization terms. Experiments are performed on UCI, KEEL, and NDC benchmark datasets; both the proposed GBLSTSVM and LS-GBLSTSVM models consistently outperform the baseline models.

Title: Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense

• Authors: Aditya Vikram Singh, Ethan Rathbun, Emma Graham, Lisa Oakley, Simona Boboila, Alina Oprea, Peter Chin
• Subjects: cs.LG, cs.CR, cs.MA
• Abstract URL: https://arxiv.org/abs/2410.17351
• Pdf URL: https://arxiv.org/pdf/2410.17351
• Copy Paste: [[2410.17351]] Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense(https://arxiv.org/abs/2410.17351)
• Keywords: security, defense, steal
• Abstract: Recent advances in multi-agent reinforcement learning (MARL) have created opportunities to solve complex real-world tasks. Cybersecurity is a notable application area, where defending networks against sophisticated adversaries remains a challenging task typically performed by teams of security operators. In this work, we explore novel MARL strategies for building autonomous cyber network defenses that address challenges such as large policy spaces, partial observability, and stealthy, deceptive adversarial strategies. To facilitate efficient and generalized learning, we propose a hierarchical Proximal Policy Optimization (PPO) architecture that decomposes the cyber defense task into specific sub-tasks like network investigation and host recovery. Our approach involves training sub-policies for each sub-task using PPO enhanced with domain expertise. These sub-policies are then leveraged by a master defense policy that coordinates their selection to solve complex network defense tasks. Furthermore, the sub-policies can be fine-tuned and transferred with minimal cost to defend against shifts in adversarial behavior or changes in network settings. We conduct extensive experiments using CybORG Cage 4, the state-of-the-art MARL environment for cyber defense. Comparisons with multiple baselines across different adversaries show that our hierarchical learning approach achieves top performance in terms of convergence speed, episodic return, and several interpretable metrics relevant to cybersecurity, including the fraction of clean machines on the network, precision, and false positives on recoveries.

Title: Characterizing Robocalls with Multiple Vantage Points

• Authors: Sathvik Prasad, Aleksandr Nahapetyan, Bradley Reaves
• Subjects: cs.CR, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2410.17361
• Pdf URL: https://arxiv.org/pdf/2410.17361
• Copy Paste: [[2410.17361]] Characterizing Robocalls with Multiple Vantage Points(https://arxiv.org/abs/2410.17361)
• Keywords: security
• Abstract: Telephone spam has been among the highest network security concerns for users for many years. In response, industry and government have deployed new technologies and regulations to curb the problem, and academic and industry researchers have provided methods and measurements to characterize robocalls. Have these efforts borne fruit? Are the research characterizations reliable, and have the prevention and deterrence mechanisms succeeded? In this paper, we address these questions through analysis of data from several independently-operated vantage points, ranging from industry and academic voice honeypots to public enforcement and consumer complaints, some with over 5 years of historic data. We first describe how we address the non-trivial methodological challenges of comparing disparate data sources, including comparing audio and transcripts from about 3 million voice calls. We also detail the substantial coherency of these diverse perspectives, which dramatically strengthens the evidence for the conclusions we draw about robocall characterization and mitigation while highlighting advantages of each approach. Among our many findings, we find that unsolicited calls are in slow decline, though complaints and call volumes remain high. We also find that robocallers have managed to adapt to STIR/SHAKEN, a mandatory call authentication scheme. In total, our findings highlight the most promising directions for future efforts to characterize and stop telephone spam.

Title: AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration

• Authors: Bradley McDanel
• Subjects: cs.CL, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17375
• Pdf URL: https://arxiv.org/pdf/2410.17375
• Copy Paste: [[2410.17375]] AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration(https://arxiv.org/abs/2410.17375)
• Keywords: large language model
• Abstract: Large language models typically generate tokens autoregressively, using each token as input for the next. Recent work on Speculative Decoding has sought to accelerate this process by employing a smaller, faster draft model to more quickly generate candidate tokens. These candidates are then verified in parallel by the larger (original) verify model, resulting in overall speedup compared to using the larger model by itself in an autoregressive fashion. In this work, we introduce AMUSD (Asynchronous Multi-device Speculative Decoding), a system that further accelerates generation by decoupling the draft and verify phases into a continuous, asynchronous approach. Unlike conventional speculative decoding, where only one model (draft or verify) performs token generation at a time, AMUSD enables both models to perform predictions independently on separate devices (e.g., GPUs). We evaluate our approach over multiple datasets and show that AMUSD achieves an average 29% improvement over speculative decoding and up to 1.96$\times$ speedup over conventional autoregressive decoding, while achieving identical output quality. Our system is open-source and available at this https URL.

Title: Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities

• Authors: Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, Ziqiao Ma
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17385
• Pdf URL: https://arxiv.org/pdf/2410.17385
• Copy Paste: [[2410.17385]] Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities(https://arxiv.org/abs/2410.17385)
• Keywords: robust
• Abstract: Spatial expressions in situated communication can be ambiguous, as their meanings vary depending on the frames of reference (FoR) adopted by speakers and listeners. While spatial language understanding and reasoning by vision-language models (VLMs) have gained increasing attention, potential ambiguities in these models are still under-explored. To address this issue, we present the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to systematically assess the spatial reasoning capabilities of VLMs. We evaluate nine state-of-the-art VLMs using COMFORT. Despite showing some alignment with English conventions in resolving ambiguities, our experiments reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning.

Title: AdvWeb: Controllable Black-box Attacks on VLM-powered Web Agents

• Authors: Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, Bo Li
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17401
• Pdf URL: https://arxiv.org/pdf/2410.17401
• Copy Paste: [[2410.17401]] AdvWeb: Controllable Black-box Attacks on VLM-powered Web Agents(https://arxiv.org/abs/2410.17401)
• Keywords: security, defense, attack, steal
• Abstract: Vision Language Models (VLMs) have revolutionized the creation of generalist web agents, empowering them to autonomously complete diverse tasks on real-world websites, thereby boosting human efficiency and productivity. However, despite their remarkable capabilities, the safety and security of these agents against malicious attacks remain critically underexplored, raising significant concerns about their safe deployment. To uncover and exploit such vulnerabilities in web agents, we provide AdvWeb, a novel black-box attack framework designed against web agents. AdvWeb trains an adversarial prompter model that generates and injects adversarial prompts into web pages, misleading web agents into executing targeted adversarial actions such as inappropriate stock purchases or incorrect bank transactions, actions that could lead to severe real-world consequences. With only black-box access to the web agent, we train and optimize the adversarial prompter model using DPO, leveraging both successful and failed attack strings against the target agent. Unlike prior approaches, our adversarial string injection maintains stealth and control: (1) the appearance of the website remains unchanged before and after the attack, making it nearly impossible for users to detect tampering, and (2) attackers can modify specific substrings within the generated adversarial string to seamlessly change the attack objective (e.g., purchasing stocks from a different company), enhancing attack flexibility and efficiency. We conduct extensive evaluations, demonstrating that AdvWeb achieves high success rates in attacking SOTA GPT-4V-based VLM agent across various web tasks. Our findings expose critical vulnerabilities in current LLM/VLM-based agents, emphasizing the urgent need for developing more reliable web agents and effective defenses. Our code and data are available at this https URL .

Title: ProveRAG: Provenance-Driven Vulnerability Analysis with Automated Retrieval-Augmented LLMs

• Authors: Reza Fayyazi, Stella Hoyos Trueba, Michael Zuzak, Shanchieh Jay Yang
• Subjects: cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.17406
• Pdf URL: https://arxiv.org/pdf/2410.17406
• Copy Paste: [[2410.17406]] ProveRAG: Provenance-Driven Vulnerability Analysis with Automated Retrieval-Augmented LLMs(https://arxiv.org/abs/2410.17406)
• Keywords: secure, security
• Abstract: In cybersecurity, security analysts face the challenge of mitigating newly discovered vulnerabilities in real-time, with over 300,000 Common Vulnerabilities and Exposures (CVEs) identified since 1999. The sheer volume of known vulnerabilities complicates the detection of patterns for unknown threats. While LLMs can assist, they often hallucinate and lack alignment with recent threats. Over 25,000 vulnerabilities have been identified so far in 2024, which are introduced after popular LLMs' (e.g., GPT-4) training data cutoff. This raises a major challenge of leveraging LLMs in cybersecurity, where accuracy and up-to-date information are paramount. In this work, we aim to improve the adaptation of LLMs in vulnerability analysis by mimicking how analysts perform such tasks. We propose ProveRAG, an LLM-powered system designed to assist in rapidly analyzing CVEs with automated retrieval augmentation of web data while self-evaluating its responses with verifiable evidence. ProveRAG incorporates a self-critique mechanism to help alleviate omission and hallucination common in the output of LLMs applied in cybersecurity applications. The system cross-references data from verifiable sources (NVD and CWE), giving analysts confidence in the actionable insights provided. Our results indicate that ProveRAG excels in delivering verifiable evidence to the user with over 99% and 97% accuracy in exploitation and mitigation strategies, respectively. This system outperforms direct prompting and chunking retrieval in vulnerability analysis by overcoming temporal and context-window limitations. ProveRAG guides analysts to secure their systems more effectively while documenting the process for future audits.

Title: Scalable Influence and Fact Tracing for Large Language Model Pretraining

• Authors: Tyler A. Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, Ian Tenney
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17413
• Pdf URL: https://arxiv.org/pdf/2410.17413
• Copy Paste: [[2410.17413]] Scalable Influence and Fact Tracing for Large Language Model Pretraining(https://arxiv.org/abs/2410.17413)
• Keywords: large language model
• Abstract: Training data attribution (TDA) methods aim to attribute model outputs back to specific training examples, and the application of these methods to large language model (LLM) outputs could significantly advance model transparency and data curation. However, it has been challenging to date to apply these methods to the full scale of LLM pretraining. In this paper, we refine existing gradient-based methods to work effectively at scale, allowing us to retrieve influential examples for an 8B-parameter language model from a pretraining corpus of over 160B tokens with no need for subsampling or pre-filtering. Our method combines several techniques, including optimizer state correction, a task-specific Hessian approximation, and normalized encodings, which we find to be critical for performance at scale. In quantitative evaluations on a fact tracing task, our method performs best at identifying examples that influence model predictions, but classical, model-agnostic retrieval methods such as BM25 still perform better at finding passages which explicitly contain relevant facts. These results demonstrate a misalignment between factual attribution and causal influence. With increasing model size and training tokens, we find that influence more closely aligns with attribution. Finally, we examine different types of examples identified as influential by our method, finding that while many directly entail a particular fact, others support the same output by reinforcing priors on relation types, common entities, and names.

Title: End-to-End Optimization and Learning of Fair Court Schedules

• Authors: My H Dinh, James Kotary, Lauryn P. Gouldin, William Yeoh, Ferdinando Fioretto
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.17415
• Pdf URL: https://arxiv.org/pdf/2410.17415
• Copy Paste: [[2410.17415]] End-to-End Optimization and Learning of Fair Court Schedules(https://arxiv.org/abs/2410.17415)
• Keywords: defense, fair
• Abstract: Criminal courts across the United States handle millions of cases every year, and the scheduling of those cases must accommodate a diverse set of constraints, including the preferences and availability of courts, prosecutors, and defense teams. When criminal court schedules are formed, defendants' scheduling preferences often take the least priority, although defendants may face significant consequences (including arrest or detention) for missed court dates. Additionally, studies indicate that defendants' nonappearances impose costs on the courts and other system stakeholders. To address these issues, courts and commentators have begun to recognize that pretrial outcomes for defendants and for the system would be improved with greater attention to court processes, including \emph{court scheduling practices}. There is thus a need for fair criminal court pretrial scheduling systems that account for defendants' preferences and availability, but the collection of such data poses logistical challenges. Furthermore, optimizing schedules fairly across various parties' preferences is a complex optimization problem, even when such data is available. In an effort to construct such a fair scheduling system under data uncertainty, this paper proposes a joint optimization and learning framework that combines machine learning models trained end-to-end with efficient matching algorithms. This framework aims to produce court scheduling schedules that optimize a principled measure of fairness, balancing the availability and preferences of all parties.

Title: Artificial Intelligence in Brazilian News: A Mixed-Methods Analysis

• Authors: Raphael Hernandes, Giulio Corsi
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.17423
• Pdf URL: https://arxiv.org/pdf/2410.17423
• Copy Paste: [[2410.17423]] Artificial Intelligence in Brazilian News: A Mixed-Methods Analysis(https://arxiv.org/abs/2410.17423)
• Keywords: privacy
• Abstract: The current surge in Artificial Intelligence (AI) interest, reflected in heightened media coverage since 2009, has sparked significant debate on AI's implications for privacy, social justice, workers' rights, and democracy. The media plays a crucial role in shaping public perception and acceptance of AI technologies. However, research into how AI appears in media has primarily focused on anglophone contexts, leaving a gap in understanding how AI is represented globally. This study addresses this gap by analyzing 3,560 news articles from Brazilian media published between July 1, 2023, and February 29, 2024, from 13 popular online news outlets. Using Computational Grounded Theory (CGT), the study applies Latent Dirichlet Allocation (LDA), BERTopic, and Named-Entity Recognition to investigate the main topics in AI coverage and the entities represented. The findings reveal that Brazilian news coverage of AI is dominated by topics related to applications in the workplace and product launches, with limited space for societal concerns, which mostly focus on deepfakes and electoral integrity. The analysis also highlights a significant presence of industry-related entities, indicating a strong influence of corporate agendas in the country's news. This study underscores the need for a more critical and nuanced discussion of AI's societal impacts in Brazilian media.

Title: Meta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks

• Authors: Tao Li, Henger Li, Yunian Pan, Tianyi Xu, Zizhan Zheng, Quanyan Zhu
• Subjects: cs.LG, cs.CR, cs.GT
• Abstract URL: https://arxiv.org/abs/2410.17431
• Pdf URL: https://arxiv.org/pdf/2410.17431
• Copy Paste: [[2410.17431]] Meta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks(https://arxiv.org/abs/2410.17431)
• Keywords: security, defense, attack, robust, federate
• Abstract: Federated learning (FL) is susceptible to a range of security threats. Although various defense mechanisms have been proposed, they are typically non-adaptive and tailored to specific types of attacks, leaving them insufficient in the face of multiple uncertain, unknown, and adaptive attacks employing diverse strategies. This work formulates adversarial federated learning under a mixture of various attacks as a Bayesian Stackelberg Markov game, based on which we propose the meta-Stackelberg defense composed of pre-training and online adaptation. {The gist is to simulate strong attack behavior using reinforcement learning (RL-based attacks) in pre-training and then design meta-RL-based defense to combat diverse and adaptive attacks.} We develop an efficient meta-learning approach to solve the game, leading to a robust and adaptive FL defense. Theoretically, our meta-learning algorithm, meta-Stackelberg learning, provably converges to the first-order $\varepsilon$-meta-equilibrium point in $O(\varepsilon^{-2})$ gradient iterations with $O(\varepsilon^{-4})$ samples per iteration. Experiments show that our meta-Stackelberg framework performs superbly against strong model poisoning and backdoor attacks of uncertain and unknown types.

Title: LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding

• Authors: Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, Vikas Chandra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17434
• Pdf URL: https://arxiv.org/pdf/2410.17434
• Copy Paste: [[2410.17434]] LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding(https://arxiv.org/abs/2410.17434)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To address this limitation, we propose LongVU, a spatiotemporal adaptive compression mechanism thats reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance.

Title: Interpreting Affine Recurrence Learning in GPT-style Transformers

• Authors: Samarth Bhargav, Alexander Gu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17438
• Pdf URL: https://arxiv.org/pdf/2410.17438
• Copy Paste: [[2410.17438]] Interpreting Affine Recurrence Learning in GPT-style Transformers(https://arxiv.org/abs/2410.17438)
• Keywords: interpretability, transformer
• Abstract: Understanding the internal mechanisms of GPT-style transformers, particularly their capacity to perform in-context learning (ICL), is critical for advancing AI alignment and interpretability. In-context learning allows transformers to generalize during inference without modifying their weights, yet the precise operations driving this capability remain largely opaque. This paper presents an investigation into the mechanistic interpretability of these transformers, focusing specifically on their ability to learn and predict affine recurrences as an ICL task. To address this, we trained a custom three-layer transformer to predict affine recurrences and analyzed the model's internal operations using both empirical and theoretical approaches. Our findings reveal that the model forms an initial estimate of the target sequence using a copying mechanism in the zeroth layer, which is subsequently refined through negative similarity heads in the second layer. These insights contribute to a deeper understanding of transformer behaviors in recursive tasks and offer potential avenues for improving AI alignment through mechanistic interpretability. Finally, we discuss the implications of our results for future work, including extensions to higher-dimensional recurrences and the exploration of polynomial sequences.

Title: Evaluating AI-Generated Essays with GRE Analytical Writing Assessment

• Authors: Yang Zhong, Jiangang Hao, Michael Fauss, Chen Li, Yuan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17439
• Pdf URL: https://arxiv.org/pdf/2410.17439
• Copy Paste: [[2410.17439]] Evaluating AI-Generated Essays with GRE Analytical Writing Assessment(https://arxiv.org/abs/2410.17439)
• Keywords: generative, large language model
• Abstract: The recent revolutionary advance in generative AI enables the generation of realistic and coherent texts by large language models (LLMs). Despite many existing evaluation metrics on the quality of the generated texts, there is still a lack of rigorous assessment of how well LLMs perform in complex and demanding writing assessments. This study examines essays generated by ten leading LLMs for the analytical writing assessment of the Graduate Record Exam (GRE). We assessed these essays using both human raters and the e-rater automated scoring engine as used in the GRE scoring pipeline. Notably, the top-performing GPT-4o received an average score of 4.67, falling between "generally thoughtful, well-developed analysis of the issue and conveys meaning clearly" and "presents a competent analysis of the issue and conveys meaning with acceptable clarity" according to the GRE scoring guideline. We also evaluated the detection accuracy of these essays, with detectors trained on essays generated by the same and different LLMs.

Title: Detecting Adversarial Examples

• Authors: Furkan Mumcu, Yasin Yilmaz
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17442
• Pdf URL: https://arxiv.org/pdf/2410.17442
• Copy Paste: [[2410.17442]] Detecting Adversarial Examples(https://arxiv.org/abs/2410.17442)
• Keywords: defense, attack
• Abstract: Deep Neural Networks (DNNs) have been shown to be vulnerable to adversarial examples. While numerous successful adversarial attacks have been proposed, defenses against these attacks remain relatively understudied. Existing defense approaches either focus on negating the effects of perturbations caused by the attacks to restore the DNNs' original predictions or use a secondary model to detect adversarial examples. However, these methods often become ineffective due to the continuous advancements in attack techniques. We propose a novel universal and lightweight method to detect adversarial examples by analyzing the layer outputs of DNNs. Through theoretical justification and extensive experiments, we demonstrate that our detection method is highly effective, compatible with any DNN architecture, and applicable across different domains, such as image, video, and audio.

Title: In Context Learning and Reasoning for Symbolic Regression with Large Language Models

• Authors: Samiha Sharlin, Tyler R. Josephson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17448
• Pdf URL: https://arxiv.org/pdf/2410.17448
• Copy Paste: [[2410.17448]] In Context Learning and Reasoning for Symbolic Regression with Large Language Models(https://arxiv.org/abs/2410.17448)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) are transformer-based machine learning models that have shown remarkable performance in tasks for which they were not explicitly trained. Here, we explore the potential of LLMs to perform symbolic regression -- a machine-learning method for finding simple and accurate equations from datasets. We prompt GPT-4 to suggest expressions from data, which are then optimized and evaluated using external Python tools. These results are fed back to GPT-4, which proposes improved expressions while optimizing for complexity and loss. Using chain-of-thought prompting, we instruct GPT-4 to analyze the data, prior expressions, and the scientific context (expressed in natural language) for each problem before generating new expressions. We evaluated the workflow in rediscovery of five well-known scientific equations from experimental data, and on an additional dataset without a known equation. GPT-4 successfully rediscovered all five equations, and in general, performed better when prompted to use a scratchpad and consider scientific context. We also demonstrate how strategic prompting improves the model's performance and how the natural language interface simplifies integrating theory with data. Although this approach does not outperform established SR programs where target equations are more complex, LLMs can nonetheless iterate toward improved solutions while following instructions and incorporating scientific context in natural language.

Title: Data Obfuscation through Latent Space Projection (LSP) for Privacy-Preserving AI Governance: Case Studies in Medical Diagnosis and Finance Fraud Detection

• Authors: Mahesh Vaijainthymala Krishnamoorthy
• Subjects: cs.LG, cs.AI, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.17459
• Pdf URL: https://arxiv.org/pdf/2410.17459
• Copy Paste: [[2410.17459]] Data Obfuscation through Latent Space Projection (LSP) for Privacy-Preserving AI Governance: Case Studies in Medical Diagnosis and Finance Fraud Detection(https://arxiv.org/abs/2410.17459)
• Keywords: privacy, protect, robust, federate, fair, interpretability
• Abstract: As AI systems increasingly integrate into critical societal sectors, the demand for robust privacy-preserving methods has escalated. This paper introduces Data Obfuscation through Latent Space Projection (LSP), a novel technique aimed at enhancing AI governance and ensuring Responsible AI compliance. LSP uses machine learning to project sensitive data into a latent space, effectively obfuscating it while preserving essential features for model training and inference. Unlike traditional privacy methods like differential privacy or homomorphic encryption, LSP transforms data into an abstract, lower-dimensional form, achieving a delicate balance between data utility and privacy. Leveraging autoencoders and adversarial training, LSP separates sensitive from non-sensitive information, allowing for precise control over privacy-utility trade-offs. We validate LSP's effectiveness through experiments on benchmark datasets and two real-world case studies: healthcare cancer diagnosis and financial fraud analysis. Our results show LSP achieves high performance (98.7% accuracy in image classification) while providing strong privacy (97.3% protection against sensitive attribute inference), outperforming traditional anonymization and privacy-preserving methods. The paper also examines LSP's alignment with global AI governance frameworks, such as GDPR, CCPA, and HIPAA, highlighting its contribution to fairness, transparency, and accountability. By embedding privacy within the machine learning pipeline, LSP offers a promising approach to developing AI systems that respect privacy while delivering valuable insights. We conclude by discussing future research directions, including theoretical privacy guarantees, integration with federated learning, and enhancing latent space interpretability, positioning LSP as a critical tool for ethical AI advancement.

Title: Formal Privacy Guarantees with Invariant Statistics

• Authors: Young Hyun Cho, Jordan Awan
• Subjects: cs.CR, stat.AP
• Abstract URL: https://arxiv.org/abs/2410.17468
• Pdf URL: https://arxiv.org/pdf/2410.17468
• Copy Paste: [[2410.17468]] Formal Privacy Guarantees with Invariant Statistics(https://arxiv.org/abs/2410.17468)
• Keywords: privacy
• Abstract: Motivated by the 2020 US Census products, this paper extends differential privacy (DP) to address the joint release of DP outputs and nonprivate statistics, referred to as invariant. Our framework, Semi-DP, redefines adjacency by focusing on datasets that conform to the given invariant, ensuring indistinguishability between adjacent datasets within invariant-conforming datasets. We further develop customized mechanisms that satisfy Semi-DP, including the Gaussian mechanism and the optimal $K$-norm mechanism for rank-deficient sensitivity spaces. Our framework is applied to contingency table analysis which is relevant to the 2020 US Census, illustrating how Semi-DP enables the release of private outputs given the one-way margins as the invariant. Additionally, we provide a privacy analysis of the 2020 US Decennial Census using the Semi-DP framework, revealing that the effective privacy guarantees are weaker than advertised.

Title: Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination

• Authors: Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, Boxing Chen, Sarath Chandar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17477
• Pdf URL: https://arxiv.org/pdf/2410.17477
• Copy Paste: [[2410.17477]] Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination(https://arxiv.org/abs/2410.17477)
• Keywords: generative, large language model
• Abstract: The growth in prominence of large language models (LLMs) in everyday life can be largely attributed to their generative abilities, yet some of this is also owed to the risks and costs associated with their use. On one front is their tendency to \textit{hallucinate} false or misleading information, limiting their reliability. On another is the increasing focus on the computational limitations associated with traditional self-attention based LLMs, which has brought about new alternatives, in particular recurrent models, meant to overcome them. Yet it remains uncommon to consider these two concerns simultaneously. Do changes in architecture exacerbate/alleviate existing concerns about hallucinations? Do they affect how and where they occur? Through an extensive evaluation, we study how these architecture-based inductive biases affect the propensity to hallucinate. While hallucination remains a general phenomenon not limited to specific architectures, the situations in which they occur and the ease with which specific types of hallucinations can be induced can significantly differ based on the model architecture. These findings highlight the need for better understanding both these problems in conjunction with each other, as well as consider how to design more universal techniques for handling hallucinations.

Title: Which Client is Reliable?: A Reliable and Personalized Prompt-based Federated Learning for Medical Image Question Answering

• Authors: He Zhu, Ren Togo, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17484
• Pdf URL: https://arxiv.org/pdf/2410.17484
• Copy Paste: [[2410.17484]] Which Client is Reliable?: A Reliable and Personalized Prompt-based Federated Learning for Medical Image Question Answering(https://arxiv.org/abs/2410.17484)
• Keywords: privacy, federate, transformer
• Abstract: Conventional medical artificial intelligence (AI) models face barriers in clinical application and ethical issues owing to their inability to handle the privacy-sensitive characteristics of medical data. We present a novel personalized federated learning (pFL) method for medical visual question answering (VQA) models, addressing privacy reliability challenges in the medical domain. Our method introduces learnable prompts into a Transformer architecture to efficiently train it on diverse medical datasets without massive computational costs. Then we introduce a reliable client VQA model that incorporates Dempster-Shafer evidence theory to quantify uncertainty in predictions, enhancing the model's reliability. Furthermore, we propose a novel inter-client communication mechanism that uses maximum likelihood estimation to balance accuracy and uncertainty, fostering efficient integration of insights across clients.

Title: VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning

• Authors: Yifan Peng, Krishna C. Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, Boris Ginsburg
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.17485
• Pdf URL: https://arxiv.org/pdf/2410.17485
• Copy Paste: [[2410.17485]] VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning(https://arxiv.org/abs/2410.17485)
• Keywords: large language model
• Abstract: Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models (SpeechLMs). Earlier SpeechLMs focused on single-turn speech-based question answering (QA), where user input comprised a speech context and a text question. More recent studies have extended this to multi-turn conversations, though they often require complex, multi-stage supervised fine-tuning (SFT) with diverse data. Another critical challenge with SpeechLMs is catastrophic forgetting-where models optimized for speech tasks suffer significant degradation in text-only performance. To mitigate these issues, we propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the LLM backbone. Our joint SFT combines text-only SFT data with three types of speech-related data: speech recognition and translation, speech-based QA, and mixed-modal SFT. Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks while preserving the original capabilities on text-only tasks. Furthermore, our model shows emergent abilities of effectively handling previously unseen prompts and tasks, including multi-turn, mixed-modal inputs.

Title: Unsupervised Domain Adaptation for Action Recognition via Self-Ensembling and Conditional Embedding Alignment

• Authors: Indrajeet Ghosh, Garvit Chugh, Abu Zaher Md Faridee, Nirmalya Roy
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17489
• Pdf URL: https://arxiv.org/pdf/2410.17489
• Copy Paste: [[2410.17489]] Unsupervised Domain Adaptation for Action Recognition via Self-Ensembling and Conditional Embedding Alignment(https://arxiv.org/abs/2410.17489)
• Keywords: robust
• Abstract: Recent advancements in deep learning-based wearable human action recognition (wHAR) have improved the capture and classification of complex motions, but adoption remains limited due to the lack of expert annotations and domain discrepancies from user variations. Limited annotations hinder the model's ability to generalize to out-of-distribution samples. While data augmentation can improve generalizability, unsupervised augmentation techniques must be applied carefully to avoid introducing noise. Unsupervised domain adaptation (UDA) addresses domain discrepancies by aligning conditional distributions with labeled target samples, but vanilla pseudo-labeling can lead to error propagation. To address these challenges, we propose $\mu$DAR, a novel joint optimization architecture comprised of three functions: (i) consistency regularizer between augmented samples to improve model classification generalizability, (ii) temporal ensemble for robust pseudo-label generation and (iii) conditional distribution alignment to improve domain generalizability. The temporal ensemble works by aggregating predictions from past epochs to smooth out noisy pseudo-label predictions, which are then used in the conditional distribution alignment module to minimize kernel-based class-wise conditional maximum mean discrepancy ($k$CMMD) between the source and target feature space to learn a domain invariant embedding. The consistency-regularized augmentations ensure that multiple augmentations of the same sample share the same labels; this results in (a) strong generalization with limited source domain samples and (b) consistent pseudo-label generation in target samples. The novel integration of these three modules in $\mu$DAR results in a range of $\approx$ 4-12% average macro-F1 score improvement over six state-of-the-art UDA methods in four benchmark wHAR datasets

Title: BadFair: Backdoored Fairness Attacks with Group-conditioned Triggers

• Authors: Jiaqi Xue, Qian Lou, Mengxin Zheng
• Subjects: cs.CR, cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17492
• Pdf URL: https://arxiv.org/pdf/2410.17492
• Copy Paste: [[2410.17492]] BadFair: Backdoored Fairness Attacks with Group-conditioned Triggers(https://arxiv.org/abs/2410.17492)
• Keywords: defense, attack, robust, steal, fair
• Abstract: Attacking fairness is crucial because compromised models can introduce biased outcomes, undermining trust and amplifying inequalities in sensitive applications like hiring, healthcare, and law enforcement. This highlights the urgent need to understand how fairness mechanisms can be exploited and to develop defenses that ensure both fairness and robustness. We introduce BadFair, a novel backdoored fairness attack methodology. BadFair stealthily crafts a model that operates with accuracy and fairness under regular conditions but, when activated by certain triggers, discriminates and produces incorrect results for specific groups. This type of attack is particularly stealthy and dangerous, as it circumvents existing fairness detection methods, maintaining an appearance of fairness in normal use. Our findings reveal that BadFair achieves a more than 85% attack success rate in attacks aimed at target groups on average while only incurring a minimal accuracy loss. Moreover, it consistently exhibits a significant discrimination score, distinguishing between pre-defined target and non-target attacked groups across various datasets and models.

Title: PLGS: Robust Panoptic Lifting with 3D Gaussian Splatting

• Authors: Yu Wang, Xiaobao Wei, Ming Lu, Guoliang Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17505
• Pdf URL: https://arxiv.org/pdf/2410.17505
• Copy Paste: [[2410.17505]] PLGS: Robust Panoptic Lifting with 3D Gaussian Splatting(https://arxiv.org/abs/2410.17505)
• Keywords: robust, segmentation
• Abstract: Previous methods utilize the Neural Radiance Field (NeRF) for panoptic lifting, while their training and rendering speed are unsatisfactory. In contrast, 3D Gaussian Splatting (3DGS) has emerged as a prominent technique due to its rapid training and rendering speed. However, unlike NeRF, the conventional 3DGS may not satisfy the basic smoothness assumption as it does not rely on any parameterized structures to render (e.g., MLPs). Consequently, the conventional 3DGS is, in nature, more susceptible to noisy 2D mask supervision. In this paper, we propose a new method called PLGS that enables 3DGS to generate consistent panoptic segmentation masks from noisy 2D segmentation masks while maintaining superior efficiency compared to NeRF-based methods. Specifically, we build a panoptic-aware structured 3D Gaussian model to introduce smoothness and design effective noise reduction strategies. For the semantic field, instead of initialization with structure from motion, we construct reliable semantic anchor points to initialize the 3D Gaussians. We then use these anchor points as smooth regularization during training. Additionally, we present a self-training approach using pseudo labels generated by merging the rendered masks with the noisy masks to enhance the robustness of PLGS. For the instance field, we project the 2D instance masks into 3D space and match them with oriented bounding boxes to generate cross-view consistent instance masks for supervision. Experiments on various benchmarks demonstrate that our method outperforms previous state-of-the-art methods in terms of both segmentation quality and speed.

Title: WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models

• Authors: Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, Sijia Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17509
• Pdf URL: https://arxiv.org/pdf/2410.17509
• Copy Paste: [[2410.17509]] WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models(https://arxiv.org/abs/2410.17509)
• Keywords: generative, large language model
• Abstract: The need for effective unlearning mechanisms in large language models (LLMs) is increasingly urgent, driven by the necessity to adhere to data regulations and foster ethical generative AI practices. Despite growing interest of LLM unlearning, much of the existing research has focused on varied unlearning method designs to boost effectiveness and efficiency. However, the inherent relationship between model weights and LLM unlearning has not been extensively examined. In this paper, we systematically explore how model weights interact with unlearning processes in LLMs and we design the weight attribution-guided LLM unlearning method, WAGLE, which unveils the interconnections between 'influence' of weights and 'influence' of data to forget and retain in LLM generation. By strategically guiding the LLM unlearning across different types of unlearning methods and tasks, WAGLE can erase the undesired content, while maintaining the performance of the original tasks. We refer to the weight attribution-guided LLM unlearning method as WAGLE, which unveils the interconnections between 'influence' of weights and 'influence' of data to forget and retain in LLM generation. Our extensive experiments show that WAGLE boosts unlearning performance across a range of LLM unlearning methods such as gradient difference and (negative) preference optimization, applications such as fictitious unlearning, malicious use prevention, and copyrighted information removal, and models including Zephyr-7b-beta and Llama2-7b. To the best of our knowledge, our work offers the first principled method for attributing and pinpointing the influential weights in enhancing LLM unlearning. It stands in contrast to previous methods that lack weight attribution and simpler weight attribution techniques.

Title: HCDN: A Change Detection Network for Construction Housekeeping Using Feature Fusion and Large Vision Models

• Authors: Kailai Sun, Zherui Shao, Yang Miang Goh, Jing Tian, Vincent J.L. Gan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.17513
• Pdf URL: https://arxiv.org/pdf/2410.17513
• Copy Paste: [[2410.17513]] HCDN: A Change Detection Network for Construction Housekeeping Using Feature Fusion and Large Vision Models(https://arxiv.org/abs/2410.17513)
• Keywords: segmentation
• Abstract: Workplace safety has received increasing attention as millions of workers worldwide suffer from work-related accidents. Despite poor housekeeping is a significant contributor to construction accidents, there remains a significant lack of technological research focused on improving housekeeping practices in construction sites. Recognizing and locating poor housekeeping in a dynamic construction site is an important task that can be improved through computer vision approaches. Despite advances in AI and computer vision, existing methods for detecting poor housekeeping conditions face many challenges, including limited explanations, lack of locating of poor housekeeping, and lack of annotated datasets. On the other hand, change detection which aims to detect the changed environmental conditions (e.g., changing from good to poor housekeeping) and 'where' the change has occurred (e.g., location of objects causing poor housekeeping), has not been explored to the problem of housekeeping management. To address these challenges, we propose the Housekeeping Change Detection Network (HCDN), an advanced change detection neural network that integrates a feature fusion module and a large vision model, achieving state-of-the-art performance. Additionally, we introduce the approach to establish a novel change detection dataset (named Housekeeping-CCD) focused on housekeeping in construction sites, along with a housekeeping segmentation dataset. Our contributions include significant performance improvements compared to existing methods, providing an effective tool for enhancing construction housekeeping and safety. To promote further development, we share our source code and trained models for global researchers: this https URL.

Title: Large Language Models Still Exhibit Bias in Long Text

• Authors: Wonje Jeung, Dongjae Jeon, Ashkan Yousefpour, Jonghyun Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17519
• Pdf URL: https://arxiv.org/pdf/2410.17519
• Copy Paste: [[2410.17519]] Large Language Models Still Exhibit Bias in Long Text(https://arxiv.org/abs/2410.17519)
• Keywords: protect, fair, large language model
• Abstract: Existing fairness benchmarks for large language models (LLMs) primarily focus on simple tasks, such as multiple-choice questions, overlooking biases that may arise in more complex scenarios like long-text generation. To address this gap, we introduce the Long Text Fairness Test (LTF-TEST), a framework that evaluates biases in LLMs through essay-style prompts. LTF-TEST covers 14 topics and 10 demographic axes, including gender and race, resulting in 11,948 samples. By assessing both model responses and the reasoning behind them, LTF-TEST uncovers subtle biases that are difficult to detect in simple responses. In our evaluation of five recent LLMs, including GPT-4o and LLaMa3, we identify two key patterns of bias. First, these models frequently favor certain demographic groups in their responses. Second, they show excessive sensitivity toward traditionally disadvantaged groups, often providing overly protective responses while neglecting others. To mitigate these biases, we propose FT-REGARD, a finetuning approach that pairs biased prompts with neutral responses. FT-REGARD reduces gender bias by 34.6% and improves performance by 1.4 percentage points on the BBQ benchmark, offering a promising approach to addressing biases in long-text generation tasks.

Title: MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control

• Authors: Juyong Lee, Dongyoon Hahm, June Suk Choi, W. Bradley Knox, Kimin Lee
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17520
• Pdf URL: https://arxiv.org/pdf/2410.17520
• Copy Paste: [[2410.17520]] MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control(https://arxiv.org/abs/2410.17520)
• Keywords: robust, large language model
• Abstract: Autonomous agents powered by large language models (LLMs) show promising potential in assistive tasks across various domains, including mobile device control. As these agents interact directly with personal information and device settings, ensuring their safe and reliable behavior is crucial to prevent undesirable outcomes. However, no benchmark exists for standardized evaluation of the safety of mobile device-control agents. In this work, we introduce MobileSafetyBench, a benchmark designed to evaluate the safety of device-control agents within a realistic mobile environment based on Android emulators. We develop a diverse set of tasks involving interactions with various mobile applications, including messaging and banking applications. To clearly evaluate safety apart from general capabilities, we design separate tasks measuring safety and tasks evaluating helpfulness. The safety tasks challenge agents with managing potential risks prevalent in daily life and include tests to evaluate robustness against indirect prompt injections. Our experiments demonstrate that while baseline agents, based on state-of-the-art LLMs, perform well in executing helpful tasks, they show poor performance in safety tasks. To mitigate these safety concerns, we propose a prompting method that encourages agents to prioritize safety considerations. While this method shows promise in promoting safer behaviors, there is still considerable room for improvement to fully earn user trust. This highlights the urgent need for continued research to develop more robust safety mechanisms in mobile environments. We open-source our benchmark at: this https URL.

Title: Diffusion Priors for Variational Likelihood Estimation and Image Denoising

• Authors: Jun Cheng, Shan Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17521
• Pdf URL: https://arxiv.org/pdf/2410.17521
• Copy Paste: [[2410.17521]] Diffusion Priors for Variational Likelihood Estimation and Image Denoising(https://arxiv.org/abs/2410.17521)
• Keywords: diffusion
• Abstract: Real-world noise removal is crucial in low-level computer vision. Due to the remarkable generation capabilities of diffusion models, recent attention has shifted towards leveraging diffusion priors for image restoration tasks. However, existing diffusion priors-based methods either consider simple noise types or rely on approximate posterior estimation, limiting their effectiveness in addressing structured and signal-dependent noise commonly found in real-world images. In this paper, we build upon diffusion priors and propose adaptive likelihood estimation and MAP inference during the reverse diffusion process to tackle real-world noise. We introduce an independent, non-identically distributed likelihood combined with the noise precision (inverse variance) prior and dynamically infer the precision posterior using variational Bayes during the generation process. Meanwhile, we rectify the estimated noise variance through local Gaussian convolution. The final denoised image is obtained by propagating intermediate MAP solutions that balance the updated likelihood and diffusion prior. Additionally, we explore the local diffusion prior inherent in low-resolution diffusion models, enabling direct handling of high-resolution noisy images. Extensive experiments and analyses on diverse real-world datasets demonstrate the effectiveness of our method. Code is available at this https URL.

Title: GDDA: Semantic OOD Detection on Graphs under Covariate Shift via Score-Based Diffusion Models

• Authors: Zhixia He, Chen Zhao, Minglai Shao, Yujie Lin, Dong Li, Qin Tian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17526
• Pdf URL: https://arxiv.org/pdf/2410.17526
• Copy Paste: [[2410.17526]] GDDA: Semantic OOD Detection on Graphs under Covariate Shift via Score-Based Diffusion Models(https://arxiv.org/abs/2410.17526)
• Keywords: diffusion, generative
• Abstract: Out-of-distribution (OOD) detection poses a significant challenge for Graph Neural Networks (GNNs), particularly in open-world scenarios with varying distribution shifts. Most existing OOD detection methods on graphs primarily focus on identifying instances in test data domains caused by either semantic shifts (changes in data classes) or covariate shifts (changes in data features), while leaving the simultaneous occurrence of both distribution shifts under-explored. In this work, we address both types of shifts simultaneously and introduce a novel challenge for OOD detection on graphs: graph-level semantic OOD detection under covariate shift. In this scenario, variations between the training and test domains result from the concurrent presence of both covariate and semantic shifts, where only graphs associated with unknown classes are identified as OOD samples (OODs). To tackle this challenge, we propose a novel two-phase framework called Graph Disentangled Diffusion Augmentation (GDDA). The first phase focuses on disentangling graph representations into domain-invariant semantic factors and domain-specific style factors. In the second phase, we introduce a novel distribution-shift-controlled score-based generative diffusion model that generates latent factors outside the training semantic and style spaces. Additionally, auxiliary pseudo-in-distribution (InD) and pseudo-OOD graph representations are employed to enhance the effectiveness of the energy-based semantic OOD detector. Extensive empirical studies on three benchmark datasets demonstrate that our approach outperforms state-of-the-art baselines.

Title: Navigate Complex Physical Worlds via Geometrically Constrained LLM

• Authors: Yongqiang Huang, Wentao Ye, Liyao Li, Junbo Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17529
• Pdf URL: https://arxiv.org/pdf/2410.17529
• Copy Paste: [[2410.17529]] Navigate Complex Physical Worlds via Geometrically Constrained LLM(https://arxiv.org/abs/2410.17529)
• Keywords: large language model
• Abstract: This study investigates the potential of Large Language Models (LLMs) for reconstructing and constructing the physical world solely based on textual knowledge. It explores the impact of model performance on spatial understanding abilities. To enhance the comprehension of geometric and spatial relationships in the complex physical world, the study introduces a set of geometric conventions and develops a workflow based on multi-layer graphs and multi-agent system frameworks. It examines how LLMs achieve multi-step and multi-objective geometric inference in a spatial environment using multi-layer graphs under unified geometric conventions. Additionally, the study employs a genetic algorithm, inspired by large-scale model knowledge, to solve geometric constraint problems. In summary, this work innovatively explores the feasibility of using text-based LLMs as physical world builders and designs a workflow to enhance their capabilities.

Title: Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact

• Authors: Junhua Liu, Bin Fu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17532
• Pdf URL: https://arxiv.org/pdf/2410.17532
• Copy Paste: [[2410.17532]] Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact(https://arxiv.org/abs/2410.17532)
• Keywords: large language model
• Abstract: Multilingual Large Language Models (MLLMs) represent a pivotal advancement in democratizing artificial intelligence across linguistic boundaries. While theoretical foundations are well-established, practical implementation guidelines remain scattered. This work bridges this gap by providing a comprehensive end-to-end framework for developing and deploying MLLMs in production environments. We make three distinctive contributions: First, we present an actionable pipeline from data pre-processing through deployment, integrating insights from academic research and industrial applications. Second, using Llama2 as a case study, we provide detailed optimization strategies for enhancing multilingual capabilities, including curriculum learning approaches for balancing high-resource and low-resource languages, tokenization strategies, and effective sampling methods. Third, we offer an interdisciplinary analysis that considers technical, linguistic, and cultural perspectives in MLLM development. Our findings reveal critical challenges in supporting linguistic diversity, with 88.38% of world languages categorized as low-resource, affecting over a billion speakers. We examine practical solutions through real-world applications in customer service, search engines, and machine translation. By synthesizing theoretical frameworks with production-ready implementation strategies, this survey provides essential guidance for practitioners and researchers working to develop more inclusive and effective multilingual AI systems.

Title: FedGMark: Certifiably Robust Watermarking for Federated Graph Learning

• Authors: Yuxin Yang (1 and 2), Qiang Li (1), Yuan Hong (3), Binghui Wang (2) ((1) College of Computer Science and Technology, Jilin University, (2) Department of Computer Science, Illinois Institute of Technology, (3) School of Computing, University of Connecticut)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17533
• Pdf URL: https://arxiv.org/pdf/2410.17533
• Copy Paste: [[2410.17533]] FedGMark: Certifiably Robust Watermarking for Federated Graph Learning(https://arxiv.org/abs/2410.17533)
• Keywords: protect, attack, robust, federate, watermark
• Abstract: Federated graph learning (FedGL) is an emerging learning paradigm to collaboratively train graph data from various clients. However, during the development and deployment of FedGL models, they are susceptible to illegal copying and model theft. Backdoor-based watermarking is a well-known method for mitigating these attacks, as it offers ownership verification to the model owner. We take the first step to protect the ownership of FedGL models via backdoor-based watermarking. Existing techniques have challenges in achieving the goal: 1) they either cannot be directly applied or yield unsatisfactory performance; 2) they are vulnerable to watermark removal attacks; and 3) they lack of formal guarantees. To address all the challenges, we propose FedGMark, the first certified robust backdoor-based watermarking for FedGL. FedGMark leverages the unique graph structure and client information in FedGL to learn customized and diverse watermarks. It also designs a novel GL architecture that facilitates defending against both the empirical and theoretically worst-case watermark removal attacks. Extensive experiments validate the promising empirical and provable watermarking performance of FedGMark. Source code is available at: this https URL.

Title: ProtoLens: Advancing Prototype Learning for Fine-Grained Interpretability in Text Classification

• Authors: Bowen Wei, Ziwei Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17546
• Pdf URL: https://arxiv.org/pdf/2410.17546
• Copy Paste: [[2410.17546]] ProtoLens: Advancing Prototype Learning for Fine-Grained Interpretability in Text Classification(https://arxiv.org/abs/2410.17546)
• Keywords: extraction, interpretability
• Abstract: Deep neural networks have achieved remarkable performance in various text-based tasks but often lack interpretability, making them less suitable for applications where transparency is critical. To address this, we propose ProtoLens, a novel prototype-based model that provides fine-grained, sub-sentence level interpretability for text classification. ProtoLens uses a Prototype-aware Span Extraction module to identify relevant text spans associated with learned prototypes and a Prototype Alignment mechanism to ensure prototypes are semantically meaningful throughout training. By aligning the prototype embeddings with human-understandable examples, ProtoLens provides interpretable predictions while maintaining competitive accuracy. Extensive experiments demonstrate that ProtoLens outperforms both prototype-based and non-interpretable baselines on multiple text classification benchmarks. Code and data are available at \url{this https URL}.

Title: Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors

• Authors: Bang You, Huaping Liu
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.17551
• Pdf URL: https://arxiv.org/pdf/2410.17551
• Copy Paste: [[2410.17551]] Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors(https://arxiv.org/abs/2410.17551)
• Keywords: robust
• Abstract: Reinforcement learning has achieved promising results on robotic control tasks but struggles to leverage information effectively from multiple sensory modalities that differ in many characteristics. Recent works construct auxiliary losses based on reconstruction or mutual information to extract joint representations from multiple sensory inputs to improve the sample efficiency and performance of reinforcement learning algorithms. However, the representations learned by these methods could capture information irrelevant to learning a policy and may degrade the performance. We argue that compressing information in the learned joint representations about raw multimodal observations is helpful, and propose a multimodal information bottleneck model to learn task-relevant joint representations from egocentric images and proprioception. Our model compresses and retains the predictive information in multimodal observations for learning a compressed joint representation, which fuses complementary information from visual and proprioceptive feedback and meanwhile filters out task-irrelevant information in raw multimodal observations. We propose to minimize the upper bound of our multimodal information bottleneck objective for computationally tractable optimization. Experimental evaluations on several challenging locomotion tasks with egocentric images and proprioception show that our method achieves better sample efficiency and zero-shot robustness to unseen white noise than leading baselines. We also empirically demonstrate that leveraging information from egocentric images and proprioception is more helpful for learning policies on locomotion tasks than solely using one single modality.

Title: ESpeW: Robust Copyright Protection for LLM-based EaaS via Embedding-Specific Watermark

• Authors: Zongqi Wang, Baoyuan Wu, Jingyuan Deng, Yujiu Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17552
• Pdf URL: https://arxiv.org/pdf/2410.17552
• Copy Paste: [[2410.17552]] ESpeW: Robust Copyright Protection for LLM-based EaaS via Embedding-Specific Watermark(https://arxiv.org/abs/2410.17552)
• Keywords: protect, attack, robust, extraction, watermark
• Abstract: Embeddings as a Service (EaaS) is emerging as a crucial role in AI applications. Unfortunately, EaaS is vulnerable to model extraction attacks, highlighting the urgent need for copyright this http URL some preliminary works propose applying embedding watermarks to protect EaaS, recent research reveals that these watermarks can be easily removed. Hence, it is crucial to inject robust watermarks resistant to watermark removal this http URL watermarking methods typically inject a target embedding into embeddings through linear interpolation when the text contains triggers. However, this mechanism results in each watermarked embedding having the same component, which makes the watermark easy to identify and this http URL by this, in this paper, we propose a novel embedding-specific watermarking (ESpeW) mechanism to offer robust copyright protection for EaaS. Our approach involves injecting unique, yet readily identifiable watermarks into each embedding. Watermarks inserted by ESpeW are designed to maintain a significant distance from one another and to avoid sharing common components, thus making it significantly more challenging to remove the this http URL experiments on four popular datasets demonstrate that ESpeW can even watermark successfully against a highly aggressive removal strategy without sacrificing the quality of embeddings.

Title: DisenGCD: A Meta Multigraph-assisted Disentangled Graph Learning Framework for Cognitive Diagnosis

• Authors: Shangshang Yang, Mingyang Chen, Ziwen Wang, Xiaoshan Yu, Panpan Zhang, Haiping Ma, Xingyi Zhang
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.17564
• Pdf URL: https://arxiv.org/pdf/2410.17564
• Copy Paste: [[2410.17564]] DisenGCD: A Meta Multigraph-assisted Disentangled Graph Learning Framework for Cognitive Diagnosis(https://arxiv.org/abs/2410.17564)
• Keywords: robust
• Abstract: Existing graph learning-based cognitive diagnosis (CD) methods have made relatively good results, but their student, exercise, and concept representations are learned and exchanged in an implicit unified graph, which makes the interaction-agnostic exercise and concept representations be learned poorly, failing to provide high robustness against noise in students' interactions. Besides, lower-order exercise latent representations obtained in shallow layers are not well explored when learning the student representation. To tackle the issues, this paper suggests a meta multigraph-assisted disentangled graph learning framework for CD (DisenGCD), which learns three types of representations on three disentangled graphs: student-exercise-concept interaction, exercise-concept relation, and concept dependency graphs, respectively. Specifically, the latter two graphs are first disentangled from the interaction graph. Then, the student representation is learned from the interaction graph by a devised meta multigraph learning module; multiple learnable propagation paths in this module enable current student latent representation to access lower-order exercise latent representations, which can lead to more effective nad robust student representations learned; the exercise and concept representations are learned on the relation and dependency graphs by graph attention modules. Finally, a novel diagnostic function is devised to handle three disentangled representations for prediction. Experiments show better performance and robustness of DisenGCD than state-of-the-art CD methods and demonstrate the effectiveness of the disentangled learning framework and meta multigraph module. The source code is available at \textcolor{red}{\url{this https URL}}.

Title: Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation

• Authors: Yingyu Chen, Ziyuan Yang, Ming Yan, Zhongzhou Zhang, Hui Yu, Yan Liu, Yi Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17565
• Pdf URL: https://arxiv.org/pdf/2410.17565
• Copy Paste: [[2410.17565]] Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation(https://arxiv.org/abs/2410.17565)
• Keywords: generative, segmentation
• Abstract: Multi-modality (MM) semi-supervised learning (SSL) based medical image segmentation has recently gained increasing attention for its ability to utilize MM data and reduce reliance on labeled images. However, current methods face several challenges: (1) Complex network designs hinder scalability to scenarios with more than two modalities. (2) Focusing solely on modality-invariant representation while neglecting modality-specific features, leads to incomplete MM learning. (3) Leveraging unlabeled data with generative methods can be unreliable for SSL. To address these problems, we propose Double Bank Dual Consistency (DBDC), a novel MM-SSL approach for medical image segmentation. To address challenge (1), we propose a modality all-in-one segmentation network that accommodates data from any number of modalities, removing the limitation on modality count. To address challenge (2), we design two learnable plug-in banks, Modality-Level Modulation bank (MLMB) and Modality-Level Prototype (MLPB) bank, to capture both modality-invariant and modality-specific knowledge. These banks are updated using our proposed Modality Prototype Contrastive Learning (MPCL). Additionally, we design Modality Adaptive Weighting (MAW) to dynamically adjust learning weights for each modality, ensuring balanced MM learning as different modalities learn at different rates. Finally, to address challenge (3), we introduce a Dual Consistency (DC) strategy that enforces consistency at both the image and feature levels without relying on generative methods. We evaluate our method on a 2-to-4 modality segmentation task using three open-source datasets, and extensive experiments show that our method outperforms state-of-the-art approaches.

Title: Differentially Private Learning Needs Better Model Initialization and Self-Distillation

• Authors: Ivoline C. Ngong, Joseph P. Near, Niloofar Mireshghallah
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17566
• Pdf URL: https://arxiv.org/pdf/2410.17566
• Copy Paste: [[2410.17566]] Differentially Private Learning Needs Better Model Initialization and Self-Distillation(https://arxiv.org/abs/2410.17566)
• Keywords: privacy
• Abstract: Differentially private SGD (DPSGD) enables privacy-preserving training of language models, but often reduces utility, diversity, and linguistic quality. We introduce DPRefine, a three-phase method that initializes a model using data synthesis from a small pre-trained LM with rigorous filtering, applies DP finetuning on private data, and performs self-distillation to refine outputs. This approach significantly outperforms vanilla DPSGD, with AlpacaEval preferring DPRefine's generations in 78.4% of cases across all datasets. Our analysis reveals that DPRefine reduces linguistic errors in generated text by 84.0%, mitigating grammar and spelling errors, commonly associated with DPSGD. It also reduces inconsistencies of non-private models, such as hallucinated details and misattributed quotes. We find that small models like GPT-2 can be effective for initialization and distillation, highlighting their potential in enabling scalable and efficient deployment of privacy-preserving language.

Title: Securing Federated Learning Against Novel and Classic Backdoor Threats During Foundation Model Integration

• Authors: Xiaohuan Bi, Xi Li
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17573
• Pdf URL: https://arxiv.org/pdf/2410.17573
• Copy Paste: [[2410.17573]] Securing Federated Learning Against Novel and Classic Backdoor Threats During Foundation Model Integration(https://arxiv.org/abs/2410.17573)
• Keywords: privacy, defense, attack, federate, data-free
• Abstract: Federated learning (FL) enables decentralized model training while preserving privacy. Recently, integrating Foundation Models (FMs) into FL has boosted performance but also introduced a novel backdoor attack mechanism. Attackers can exploit the FM's capabilities to embed backdoors into synthetic data generated by FMs used for model fusion, subsequently infecting all client models through knowledge sharing without involvement in the long-lasting FL process. These novel attacks render existing FL backdoor defenses ineffective, as they primarily detect anomalies among client updates, which may appear uniformly malicious under this attack. Our work proposes a novel data-free defense strategy by constraining abnormal activations in the hidden feature space during model aggregation on the server. The activation constraints, optimized using synthetic data alongside FL training, mitigate the attack while barely affecting model performance, as the parameters remain untouched. Extensive experiments demonstrate its effectiveness against both novel and classic backdoor attacks, outperforming existing defenses while maintaining model performance.

Title: MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models

• Authors: Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristià, Lucía Tormo-Bañuelos, Seungone Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17578
• Pdf URL: https://arxiv.org/pdf/2410.17578
• Copy Paste: [[2410.17578]] MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models(https://arxiv.org/abs/2410.17578)
• Keywords: large language model
• Abstract: Large language models (LLMs) are commonly used as evaluators in tasks (e.g., reward modeling, LLM-as-a-judge), where they act as proxies for human preferences or judgments. This leads to the need for meta-evaluation: evaluating the credibility of LLMs as evaluators. However, existing benchmarks primarily focus on English, offering limited insight into LLMs' effectiveness as evaluators in non-English contexts. To address this, we introduce MM-Eval, a multilingual meta-evaluation benchmark that covers 18 languages across six categories. MM-Eval evaluates various dimensions, including language-specific challenges like linguistics and language hallucinations. Evaluation results show that both proprietary and open-source language models have considerable room for improvement. Further analysis reveals a tendency for these models to assign middle-ground scores to low-resource languages. We publicly release our benchmark and code.

Title: Bonsai: Gradient-free Graph Distillation for Node Classification

• Authors: Mridul Gupta, Samyak Jain, Vansh Ramani, Hariprasad Kodamana, Sayan Ranu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17579
• Pdf URL: https://arxiv.org/pdf/2410.17579
• Copy Paste: [[2410.17579]] Bonsai: Gradient-free Graph Distillation for Node Classification(https://arxiv.org/abs/2410.17579)
• Keywords: robust
• Abstract: Graph distillation has emerged as a promising avenue to enable scalable training of GNNs by compressing the training dataset while preserving essential graph characteristics. Our study uncovers significant shortcomings in current graph distillation techniques. First, the majority of the algorithms paradoxically require training on the full dataset to perform distillation. Second, due to their gradient-emulating approach, these methods require fresh distillation for any change in hyperparameters or GNN architecture, limiting their flexibility and reusability. Finally, they fail to achieve substantial size reduction due to synthesizing fully-connected, edge-weighted graphs. To address these challenges, we present Bonsai, a novel graph distillation method empowered by the observation that \textit{computation trees} form the fundamental processing units of message-passing GNNs. Bonsai distills datasets by encoding a careful selection of \textit{exemplar} trees that maximize the representation of all computation trees in the training set. This unique approach imparts Bonsai as the first linear-time, model-agnostic graph distillation algorithm for node classification that outperforms existing baselines across $6$ real-world datasets on accuracy, while being $22$ times faster on average. Bonsai is grounded in rigorous mathematical guarantees on the adopted approximation strategies making it robust to GNN architectures, datasets, and parameters.

Title: How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?

• Authors: Jiahua Dong, Wenqi Liang, Hongliu Li, Duzhen Zhang, Meng Cao, Henghui Ding, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17594
• Pdf URL: https://arxiv.org/pdf/2410.17594
• Copy Paste: [[2410.17594]] How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?(https://arxiv.org/abs/2410.17594)
• Keywords: diffusion, generative
• Abstract: Custom diffusion models (CDMs) have attracted widespread attention due to their astonishing generative ability for personalized concepts. However, most existing CDMs unreasonably assume that personalized concepts are fixed and cannot change over time. Moreover, they heavily suffer from catastrophic forgetting and concept neglect on old personalized concepts when continually learning a series of new concepts. To address these challenges, we propose a novel Concept-Incremental text-to-image Diffusion Model (CIDM), which can resolve catastrophic forgetting and concept neglect to learn new customization tasks in a concept-incremental manner. Specifically, to surmount the catastrophic forgetting of old concepts, we develop a concept consolidation loss and an elastic weight aggregation module. They can explore task-specific and task-shared knowledge during training, and aggregate all low-rank weights of old concepts based on their contributions during inference. Moreover, in order to address concept neglect, we devise a context-controllable synthesis strategy that leverages expressive region features and noise estimation to control the contexts of generated images according to user conditions. Experiments validate that our CIDM surpasses existing custom diffusion models. The source codes are available at this https URL.

Title: Cross-model Control: Improving Multiple Large Language Models in One-time Training

• Authors: Jiayi Wu, Hao Sun, Hengyi Cai, Lixin Su, Shuaiqiang Wang, Dawei Yin, Xiang Li, Ming Gao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17599
• Pdf URL: https://arxiv.org/pdf/2410.17599
• Copy Paste: [[2410.17599]] Cross-model Control: Improving Multiple Large Language Models in One-time Training(https://arxiv.org/abs/2410.17599)
• Keywords: large language model
• Abstract: The number of large language models (LLMs) with varying parameter scales and vocabularies is increasing. While they deliver powerful performance, they also face a set of common optimization needs to meet specific requirements or standards, such as instruction following or avoiding the output of sensitive information from the real world. However, how to reuse the fine-tuning outcomes of one model to other models to reduce training costs remains a challenge. To bridge this gap, we introduce Cross-model Control (CMC), a method that improves multiple LLMs in one-time training with a portable tiny language model. Specifically, we have observed that the logit shift before and after fine-tuning is remarkably similar across different models. Based on this insight, we incorporate a tiny language model with a minimal number of parameters. By training alongside a frozen template LLM, the tiny model gains the capability to alter the logits output by the LLMs. To make this tiny language model applicable to models with different vocabularies, we propose a novel token mapping strategy named PM-MinED. We have conducted extensive experiments on instruction tuning and unlearning tasks, demonstrating the effectiveness of CMC. Our code is available at this https URL.

Title: Graphusion: A RAG Framework for Knowledge Graph Construction with a Global Perspective

• Authors: Rui Yang, Boming Yang, Aosong Feng, Sixun Ouyang, Moritz Blum, Tianwei She, Yuang Jiang, Freddy Lecue, Jinghui Lu, Irene Li
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2410.17600
• Pdf URL: https://arxiv.org/pdf/2410.17600
• Copy Paste: [[2410.17600]] Graphusion: A RAG Framework for Knowledge Graph Construction with a Global Perspective(https://arxiv.org/abs/2410.17600)
• Keywords: extraction, large language model
• Abstract: Knowledge Graphs (KGs) are crucial in the field of artificial intelligence and are widely used in downstream tasks, such as question-answering (QA). The construction of KGs typically requires significant effort from domain experts. Large Language Models (LLMs) have recently been used for Knowledge Graph Construction (KGC). However, most existing approaches focus on a local perspective, extracting knowledge triplets from individual sentences or documents, missing a fusion process to combine the knowledge in a global KG. This work introduces Graphusion, a zero-shot KGC framework from free text. It contains three steps: in Step 1, we extract a list of seed entities using topic modeling to guide the final KG includes the most relevant entities; in Step 2, we conduct candidate triplet extraction using LLMs; in Step 3, we design the novel fusion module that provides a global view of the extracted knowledge, incorporating entity merging, conflict resolution, and novel triplet discovery. Results show that Graphusion achieves scores of 2.92 and 2.37 out of 3 for entity extraction and relation recognition, respectively. Moreover, we showcase how Graphusion could be applied to the Natural Language Processing (NLP) domain and validate it in an educational scenario. Specifically, we introduce TutorQA, a new expert-verified benchmark for QA, comprising six tasks and a total of 1,200 QA pairs. Using the Graphusion-constructed KG, we achieve a significant improvement on the benchmark, for example, a 9.2% accuracy improvement on sub-graph completion.

Title: Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation

• Authors: Muquan Li, Dongyang Zhang, Tao He, Xiurui Xie, Yuan-Fang Li, Ke Qin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17606
• Pdf URL: https://arxiv.org/pdf/2410.17606
• Copy Paste: [[2410.17606]] Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation(https://arxiv.org/abs/2410.17606)
• Keywords: diffusion, data-free
• Abstract: Data-free knowledge distillation (DFKD) has emerged as a pivotal technique in the domain of model compression, substantially reducing the dependency on the original training data. Nonetheless, conventional DFKD methods that employ synthesized training data are prone to the limitations of inadequate diversity and discrepancies in distribution between the synthesized and original datasets. To address these challenges, this paper introduces an innovative approach to DFKD through diverse diffusion augmentation (DDA). Specifically, we revise the paradigm of common data synthesis in DFKD to a composite process through leveraging diffusion models subsequent to data synthesis for self-supervised augmentation, which generates a spectrum of data samples with similar distributions while retaining controlled variations. Furthermore, to mitigate excessive deviation in the embedding space, we introduce an image filtering technique grounded in cosine similarity to maintain fidelity during the knowledge distillation process. Comprehensive experiments conducted on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets showcase the superior performance of our method across various teacher-student network configurations, outperforming the contemporary state-of-the-art DFKD methods. Code will be available at:this https URL.

Title: Self-Supervised Graph Neural Networks for Enhanced Feature Extraction in Heterogeneous Information Networks

• Authors: Jianjun Wei, Yue Liu, Xin Huang, Xin Zhang, Wenyi Liu, Xu Yan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17617
• Pdf URL: https://arxiv.org/pdf/2410.17617
• Copy Paste: [[2410.17617]] Self-Supervised Graph Neural Networks for Enhanced Feature Extraction in Heterogeneous Information Networks(https://arxiv.org/abs/2410.17617)
• Keywords: extraction
• Abstract: This paper explores the applications and challenges of graph neural networks (GNNs) in processing complex graph data brought about by the rapid development of the Internet. Given the heterogeneity and redundancy problems that graph data often have, traditional GNN methods may be overly dependent on the initial structure and attribute information of the graph, which limits their ability to accurately simulate more complex relationships and patterns in the graph. Therefore, this study proposes a graph neural network model under a self-supervised learning framework, which can flexibly combine different types of additional information of the attribute graph and its nodes, so as to better mine the deep features in the graph data. By introducing a self-supervisory mechanism, it is expected to improve the adaptability of existing models to the diversity and complexity of graph data and improve the overall performance of the model.

Title: Feature Learning in Attention Mechanisms Is More Compact and Stable Than in Convolution

• Authors: Baiyuan Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17628
• Pdf URL: https://arxiv.org/pdf/2410.17628
• Copy Paste: [[2410.17628]] Feature Learning in Attention Mechanisms Is More Compact and Stable Than in Convolution(https://arxiv.org/abs/2410.17628)
• Keywords: transformer
• Abstract: Attention and convolution are fundamental techniques in machine learning. While they use different approaches to learn features - attention mechanisms capture both global and local data relathionships, while convolutional layers focus on local patterns - both methods are effective for various tasks. Although the feature learning of both models is well-studied individually, there has not been a direct comparison of their feature learning dynamics. In this paper, we compare their Lipschitz continuity with respect to the Wasserstein distance and covering numbers under similar settings. We demonstrate that attention processes data in a more compact and stable manner. Compactness refers to the lower variance and intrinsic dimensionality of the activation outputs, while stability refers to the changes between inputs and outputs. We validate our findings through experiments using topological data analysis, measuring the 1-, 2-, and infinity-Wasserstein distances between the outputs of each layer from both models. Furthermore, we extend our comparison to Vision Transformers (ViTs) and ResNets, showing that while ViTs have higher output variance, their feature learning is more stable than that of ResNets.

Title: LMLPA: Language Model Linguistic Personality Assessment

• Authors: Jingyao Zheng, Xian Wang, Simo Hosio, Xiaoxian Xu, Lik-Hang Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17632
• Pdf URL: https://arxiv.org/pdf/2410.17632
• Copy Paste: [[2410.17632]] LMLPA: Language Model Linguistic Personality Assessment(https://arxiv.org/abs/2410.17632)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) are increasingly used in everyday life and research. One of the most common use cases is conversational interactions, enabled by the language generation capabilities of LLMs. Just as between two humans, a conversation between an LLM-powered entity and a human depends on the personality of the conversants. However, measuring the personality of a given LLM is currently a challenge. This paper introduces the Language Model Linguistic Personality Assessment (LMLPA), a system designed to evaluate the linguistic personalities of LLMs. Our system helps to understand LLMs' language generation capabilities by quantitatively assessing the distinct personality traits reflected in their linguistic outputs. Unlike traditional human-centric psychometrics, the LMLPA adapts a personality assessment questionnaire, specifically the Big Five Inventory, to align with the operational capabilities of LLMs, and also incorporates the findings from previous language-based personality measurement literature. To mitigate sensitivity to the order of options, our questionnaire is designed to be open-ended, resulting in textual answers. Thus, the AI rater is needed to transform ambiguous personality information from text responses into clear numerical indicators of personality traits. Utilising Principal Component Analysis and reliability validations, our findings demonstrate that LLMs possess distinct personality traits that can be effectively quantified by the LMLPA. This research contributes to Human-Computer Interaction and Human-Centered AI, providing a robust framework for future studies to refine AI personality assessments and expand their applications in multiple areas, including education and manufacturing.

Title: Surgical Scene Segmentation by Transformer With Asymmetric Feature Enhancement

• Authors: Cheng Yuan, Yutong Ban
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17642
• Pdf URL: https://arxiv.org/pdf/2410.17642
• Copy Paste: [[2410.17642]] Surgical Scene Segmentation by Transformer With Asymmetric Feature Enhancement(https://arxiv.org/abs/2410.17642)
• Keywords: transformer, segmentation
• Abstract: Surgical scene segmentation is a fundamental task for robotic-assisted laparoscopic surgery understanding. It often contains various anatomical structures and surgical instruments, where similar local textures and fine-grained structures make the segmentation a difficult task. Vision-specific transformer method is a promising way for surgical scene understanding. However, there are still two main challenges. Firstly, the absence of inner-patch information fusion leads to poor segmentation performance. Secondly, the specific characteristics of anatomy and instruments are not specifically modeled. To tackle the above challenges, we propose a novel Transformer-based framework with an Asymmetric Feature Enhancement module (TAFE), which enhances local information and then actively fuses the improved feature pyramid into the embeddings from transformer encoders by a multi-scale interaction attention strategy. The proposed method outperforms the SOTA methods in several different surgical segmentation tasks and additionally proves its ability of fine-grained structure recognition. Code is available at this https URL.

Title: Entity-based Reinforcement Learning for Autonomous Cyber Defence

• Authors: Isaac Symes Thompson, Alberto Caron, Chris Hicks, Vasilios Mavroudis
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17647
• Pdf URL: https://arxiv.org/pdf/2410.17647
• Copy Paste: [[2410.17647]] Entity-based Reinforcement Learning for Autonomous Cyber Defence(https://arxiv.org/abs/2410.17647)
• Keywords: security, transformer
• Abstract: A significant challenge for autonomous cyber defence is ensuring a defensive agent's ability to generalise across diverse network topologies and configurations. This capability is necessary for agents to remain effective when deployed in dynamically changing environments, such as an enterprise network where devices may frequently join and leave. Standard approaches to deep reinforcement learning, where policies are parameterised using a fixed-input multi-layer perceptron (MLP) expect fixed-size observation and action spaces. In autonomous cyber defence, this makes it hard to develop agents that generalise to environments with network topologies different from those trained on, as the number of nodes affects the natural size of the observation and action spaces. To overcome this limitation, we reframe the problem of autonomous network defence using entity-based reinforcement learning, where the observation and action space of an agent are decomposed into a collection of discrete entities. This framework enables the use of policy parameterisations specialised in compositional generalisation. Namely, we train a Transformer-based policy on the Yawning Titan cyber-security simulation environment and test its generalisation capabilities across various network topologies. We demonstrate that this approach significantly outperforms an MLP-based policy on fixed networks, and has the ability for zero-shot generalisation to networks of a different size to those seen in training. These findings highlight the potential for entity-based reinforcement learning to advance the field of autonomous cyber defence by providing more generalisable policies capable of handling variations in real-world network environments.

Title: Towards Active Participant-Centric Vertical Federated Learning: Some Representations May Be All You Need

• Authors: Jon Irureta, Jon Imaz, Aizea Lojo, Marco González, Iñigo Perona
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17648
• Pdf URL: https://arxiv.org/pdf/2410.17648
• Copy Paste: [[2410.17648]] Towards Active Participant-Centric Vertical Federated Learning: Some Representations May Be All You Need(https://arxiv.org/abs/2410.17648)
• Keywords: privacy, federate
• Abstract: Vertical Federated Learning (VFL) enables collaborative model training across different participants with distinct features and common samples, while preserving data privacy. Existing VFL methodologies often struggle with realistic data partitions, typically incurring high communication costs and significant operational complexity. In this work, we introduce a novel simplified approach to VFL, Active Participant-Centric VFL (APC-VFL), that, to the best of our knowledge, is the first to require only a single communication round between participants, and allows the active participant to do inference in a non collaborative fashion. This method integrates unsupervised representation learning with knowledge distillation to achieve comparable accuracy to traditional VFL methods based on vertical split learning in classical settings, reducing required communication rounds by up to $4200\times$, while being more flexible. Our approach also shows improvements compared to non-federated local models, as well as a comparable VFL proposal, VFedTrans, offering an efficient and flexible solution for collaborative learning.

Title: ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents

• Authors: Yusheng Liao, Shuyang Jiang, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17657
• Pdf URL: https://arxiv.org/pdf/2410.17657
• Copy Paste: [[2410.17657]] ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents(https://arxiv.org/abs/2410.17657)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown promising potential in the medical domain, assisting with tasks like clinical note generation and patient communication. However, current LLMs are limited to text-based communication, hindering their ability to interact with diverse forms of information in clinical environments. Despite clinical agents succeeding in diverse signal interaction, they are oriented to a single clinical scenario and hence fail for broader applications. To evaluate clinical agents holistically, we propose ClinicalAgent Bench~(CAB), a comprehensive medical agent benchmark consisting of 18 tasks across five key realistic clinical dimensions. Building on this, we introduce ReflecTool, a novel framework that excels at utilizing domain-specific tools within two stages. The first optimization stage progressively enlarges a long-term memory by saving successful solving processes and tool-wise experience of agents in a tiny pre-defined training set. In the following inference stage, ReflecTool can search for supportive successful demonstrations from already built long-term memory to guide the tool selection strategy, and a verifier improves the tool usage according to the tool-wise experience with two verification methods--iterative refinement and candidate selection. Extensive experiments on ClinicalAgent Benchmark demonstrate that ReflecTool surpasses the pure LLMs with more than 10 points and the well-established agent-based methods with 3 points, highlighting its adaptability and effectiveness in solving complex clinical tasks.

Title: Quantifying the Risks of Tool-assisted Rephrasing to Linguistic Diversity

• Authors: Mengying Wang, Andreas Spitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17670
• Pdf URL: https://arxiv.org/pdf/2410.17670
• Copy Paste: [[2410.17670]] Quantifying the Risks of Tool-assisted Rephrasing to Linguistic Diversity(https://arxiv.org/abs/2410.17670)
• Keywords: large language model
• Abstract: Writing assistants and large language models see widespread use in the creation of text content. While their effectiveness for individual users has been evaluated in the literature, little is known about their proclivity to change language or reduce its richness when adopted by a large user base. In this paper, we take a first step towards quantifying this risk by measuring the semantic and vocabulary change enacted by the use of rephrasing tools on a multi-domain corpus of human-generated text.

Title: Optimizing Load Scheduling in Power Grids Using Reinforcement Learning and Markov Decision Processes

• Authors: Dongwen Luo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17696
• Pdf URL: https://arxiv.org/pdf/2410.17696
• Copy Paste: [[2410.17696]] Optimizing Load Scheduling in Power Grids Using Reinforcement Learning and Markov Decision Processes(https://arxiv.org/abs/2410.17696)
• Keywords: robust
• Abstract: Power grid load scheduling is a critical task that ensures the balance between electricity generation and consumption while minimizing operational costs and maintaining grid stability. Traditional optimization methods often struggle with the dynamic and stochastic nature of power systems, especially when faced with renewable energy sources and fluctuating demand. This paper proposes a reinforcement learning (RL) approach using a Markov Decision Process (MDP) framework to address the challenges of dynamic load scheduling. The MDP is defined by a state space representing grid conditions, an action space covering control operations like generator adjustments and storage management, and a reward function balancing economic efficiency and system reliability. We investigate the application of various RL algorithms, from basic Q-Learning to more advanced Deep Q-Networks (DQN) and Actor-Critic methods, to determine optimal scheduling policies. The proposed approach is evaluated through a simulated power grid environment, demonstrating its potential to improve scheduling efficiency and adapt to variable demand patterns. Our results show that the RL-based method provides a robust and scalable solution for real-time load scheduling, contributing to the efficient management of modern power grids.

Title: Beware of Calibration Data for Pruning Large Language Models

• Authors: Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, Min Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17711
• Pdf URL: https://arxiv.org/pdf/2410.17711
• Copy Paste: [[2410.17711]] Beware of Calibration Data for Pruning Large Language Models(https://arxiv.org/abs/2410.17711)
• Keywords: large language model
• Abstract: As large language models (LLMs) are widely applied across various fields, model compression has become increasingly crucial for reducing costs and improving inference efficiency. Post-training pruning is a promising method that does not require resource-intensive iterative training and only needs a small amount of calibration data to assess the importance of parameters. Previous research has primarily focused on designing advanced pruning methods, while different calibration data's impact on pruning performance still lacks systematical exploration. We fill this blank and surprisingly observe that the effects of calibration data even value more than designing advanced pruning strategies, especially for high sparsity. Our preliminary exploration also discloses that using calibration data similar to the training data can yield better performance. As pre-training data is usually inaccessible for advanced LLMs, we further provide a self-generating calibration data synthesis strategy to construct feasible calibration data. We conduct experiments on the recent strong open-source LLMs (e.g., DCLM, and LLaMA-3), and the results show that the proposed method outperforms commonly used calibration data and can effectively enhance strong pruning methods (e.g., Wanda, OWL).

Title: CogSteer: Cognition-Inspired Selective Layer Intervention for Efficient Semantic Steering in Large Language Models

• Authors: Xintong Wang, Jingheng Pan, Longqin Jiang, Liang Ding, Xingshan Li, Chris Biemann
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17714
• Pdf URL: https://arxiv.org/pdf/2410.17714
• Copy Paste: [[2410.17714]] CogSteer: Cognition-Inspired Selective Layer Intervention for Efficient Semantic Steering in Large Language Models(https://arxiv.org/abs/2410.17714)
• Keywords: interpretability, large language model
• Abstract: Despite their impressive capabilities, large language models (LLMs) often lack interpretability and can generate toxic content. While using LLMs as foundation models and applying semantic steering methods are widely practiced, we believe that efficient methods should be based on a thorough understanding of LLM behavior. To this end, we propose using eye movement measures to interpret LLM behavior across layers. We find that LLMs exhibit patterns similar to human gaze across layers and different layers function differently. Inspired by these findings, we introduce a heuristic steering layer selection and apply it to layer intervention methods via fine-tuning and inference. Using language toxification and detoxification as test beds, we demonstrate that our proposed CogSteer methods achieve better results in terms of toxicity scores while efficiently saving 97% of the computational resources and 60% of the training time. Our model-agnostic approach can be adopted into various LLMs, contributing to their interpretability and promoting trustworthiness for safe deployment.

Title: YOLOv11: An Overview of the Key Architectural Enhancements

• Authors: Rahima Khanam, Muhammad Hussain
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17725
• Pdf URL: https://arxiv.org/pdf/2410.17725
• Copy Paste: [[2410.17725]] YOLOv11: An Overview of the Key Architectural Enhancements(https://arxiv.org/abs/2410.17725)
• Keywords: extraction, segmentation
• Abstract: This study presents an architectural analysis of YOLOv11, the latest iteration in the YOLO (You Only Look Once) series of object detection models. We examine the models architectural innovations, including the introduction of the C3k2 (Cross Stage Partial with kernel size 2) block, SPPF (Spatial Pyramid Pooling - Fast), and C2PSA (Convolutional block with Parallel Spatial Attention) components, which contribute in improving the models performance in several ways such as enhanced feature extraction. The paper explores YOLOv11's expanded capabilities across various computer vision tasks, including object detection, instance segmentation, pose estimation, and oriented object detection (OBB). We review the model's performance improvements in terms of mean Average Precision (mAP) and computational efficiency compared to its predecessors, with a focus on the trade-off between parameter count and accuracy. Additionally, the study discusses YOLOv11's versatility across different model sizes, from nano to extra-large, catering to diverse application needs from edge devices to high-performance computing environments. Our research provides insights into YOLOv11's position within the broader landscape of object detection and its potential impact on real-time computer vision applications.

Title: Time-to-Lie: Identifying Industrial Control System Honeypots Using the Internet Control Message Protocol

• Authors: Jacob Williams, Matthew Edwards, Joseph Gardiner
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.17731
• Pdf URL: https://arxiv.org/pdf/2410.17731
• Copy Paste: [[2410.17731]] Time-to-Lie: Identifying Industrial Control System Honeypots Using the Internet Control Message Protocol(https://arxiv.org/abs/2410.17731)
• Keywords: security
• Abstract: The convergence of information and operational technology networks has created previously unforeseen security issues. To address these issues, both researchers and practitioners have integrated threat intelligence methods into the security operations of converged networks, with some of the most valuable tools being honeypots that imitate industrial control systems (ICS). However, the development and deployment of such honeypots is a process rich with pitfalls, which can lead to undiagnosed weaknesses in the threat intelligence being gathered. This paper presents a side-channel method of covertly identifying ICS honeypots using the time-to-live (TTL) values of target devices. We show that many ICS honeypots can be readily identified, via minimal interactions, using only basic networking tools. In a study of over 8,000 devices presenting as ICS systems, we detail how our method compares to an existing honeypot detection approach, and outline what our methodology reveals about the current population of live ICS honeypots. In demonstrating our method, this study aims to raise awareness of the viability of the TTL heuristic and the prevalence of its misconfiguration despite its presence in literature.

Title: MojoBench: Language Modeling and Benchmarks for Mojo

• Authors: Nishat Raihan, Joanna C. S. Santos, Marcos Zampieri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17736
• Pdf URL: https://arxiv.org/pdf/2410.17736
• Copy Paste: [[2410.17736]] MojoBench: Language Modeling and Benchmarks for Mojo(https://arxiv.org/abs/2410.17736)
• Keywords: robust, large language model
• Abstract: The recently introduced Mojo programming language (PL) by Modular, has received significant attention in the scientific community due to its claimed significant speed boost over Python. Despite advancements in code Large Language Models (LLMs) across various PLs, Mojo remains unexplored in this context. To address this gap, we introduce MojoBench, the first framework for Mojo code generation. MojoBench includes HumanEval-Mojo, a benchmark dataset designed for evaluating code LLMs on Mojo, and Mojo-Coder, the first LLM pretrained and finetuned for Mojo code generation, which supports instructions in 5 natural languages (NLs). Our results show that Mojo-Coder achieves a 30-35% performance improvement over leading models like GPT-4o and Claude-3.5-Sonnet. Furthermore, we provide insights into LLM behavior with underrepresented and unseen PLs, offering potential strategies for enhancing model adaptability. MojoBench contributes to our understanding of LLM capabilities and limitations in emerging programming paradigms fostering more robust code generation systems.

Title: VISAGE: Video Synthesis using Action Graphs for Surgery

• Authors: Yousef Yeganeh, Rachmadio Lazuardi, Amir Shamseddin, Emine Dari, Yash Thirani, Nassir Navab Azade Farshad
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17751
• Pdf URL: https://arxiv.org/pdf/2410.17751
• Copy Paste: [[2410.17751]] VISAGE: Video Synthesis using Action Graphs for Surgery(https://arxiv.org/abs/2410.17751)
• Keywords: diffusion
• Abstract: Surgical data science (SDS) is a field that analyzes patient data before, during, and after surgery to improve surgical outcomes and skills. However, surgical data is scarce, heterogeneous, and complex, which limits the applicability of existing machine learning methods. In this work, we introduce the novel task of future video generation in laparoscopic surgery. This task can augment and enrich the existing surgical data and enable various applications, such as simulation, analysis, and robot-aided surgery. Ultimately, it involves not only understanding the current state of the operation but also accurately predicting the dynamic and often unpredictable nature of surgical procedures. Our proposed method, VISAGE (VIdeo Synthesis using Action Graphs for Surgery), leverages the power of action scene graphs to capture the sequential nature of laparoscopic procedures and utilizes diffusion models to synthesize temporally coherent video sequences. VISAGE predicts the future frames given only a single initial frame, and the action graph triplets. By incorporating domain-specific knowledge through the action graph, VISAGE ensures the generated videos adhere to the expected visual and motion patterns observed in real laparoscopic procedures. The results of our experiments demonstrate high-fidelity video generation for laparoscopy procedures, which enables various applications in SDS.

Title: AdaDiffSR: Adaptive Region-aware Dynamic Acceleration Diffusion Model for Real-World Image Super-Resolution

• Authors: Yuanting Fan, Chengxu Liu, Nengzhong Yin, Changlong Gao, Xueming Qian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17752
• Pdf URL: https://arxiv.org/pdf/2410.17752
• Copy Paste: [[2410.17752]] AdaDiffSR: Adaptive Region-aware Dynamic Acceleration Diffusion Model for Real-World Image Super-Resolution(https://arxiv.org/abs/2410.17752)
• Keywords: diffusion
• Abstract: Diffusion models (DMs) have shown promising results on single-image super-resolution and other image-to-image translation tasks. Benefiting from more computational resources and longer inference times, they are able to yield more realistic images. Existing DMs-based super-resolution methods try to achieve an overall average recovery over all regions via iterative refinement, ignoring the consideration that different input image regions require different timesteps to reconstruct. In this work, we notice that previous DMs-based super-resolution methods suffer from wasting computational resources to reconstruct invisible details. To further improve the utilization of computational resources, we propose AdaDiffSR, a DMs-based SR pipeline with dynamic timesteps sampling strategy (DTSS). Specifically, by introducing the multi-metrics latent entropy module (MMLE), we can achieve dynamic perception of the latent spatial information gain during the denoising process, thereby guiding the dynamic selection of the timesteps. In addition, we adopt a progressive feature injection module (PFJ), which dynamically injects the original image features into the denoising process based on the current information gain, so as to generate images with both fidelity and realism. Experiments show that our AdaDiffSR achieves comparable performance over current state-of-the-art DMs-based SR methods while consuming less computational resources and inference time on both synthetic and real-world datasets.

Title: Escaping the Forest: Sparse Interpretable Neural Networks for Tabular Data

• Authors: Salvatore Raieli, Abdulrahman Altahhan, Nathalie Jeanray, Stéphane Gerart, Sebastien Vachenc
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17758
• Pdf URL: https://arxiv.org/pdf/2410.17758
• Copy Paste: [[2410.17758]] Escaping the Forest: Sparse Interpretable Neural Networks for Tabular Data(https://arxiv.org/abs/2410.17758)
• Keywords: extraction, interpretability
• Abstract: Tabular datasets are widely used in scientific disciplines such as biology. While these disciplines have already adopted AI methods to enhance their findings and analysis, they mainly use tree-based methods due to their interpretability. At the same time, artificial neural networks have been shown to offer superior flexibility and depth for rich and complex non-tabular problems, but they are falling behind tree-based models for tabular data in terms of performance and interpretability. Although sparsity has been shown to improve the interpretability and performance of ANN models for complex non-tabular datasets, enforcing sparsity structurally and formatively for tabular data before training the model, remains an open question. To address this question, we establish a method that infuses sparsity in neural networks by utilising attention mechanisms to capture the features' importance in tabular datasets. We show that our models, Sparse TABular NET or sTAB-Net with attention mechanisms, are more effective than tree-based models, reaching the state-of-the-art on biological datasets. They further permit the extraction of insights from these datasets and achieve better performance than post-hoc methods like SHAP.

Title: Anomaly Resilient Temporal QoS Prediction using Hypergraph Convoluted Transformer Network

• Authors: Suraj Kumar, Soumi Chattopadhyay, Chandranath Adak
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17762
• Pdf URL: https://arxiv.org/pdf/2410.17762
• Copy Paste: [[2410.17762]] Anomaly Resilient Temporal QoS Prediction using Hypergraph Convoluted Transformer Network(https://arxiv.org/abs/2410.17762)
• Keywords: robust, transformer
• Abstract: Quality-of-Service (QoS) prediction is a critical task in the service lifecycle, enabling precise and adaptive service recommendations by anticipating performance variations over time in response to evolving network uncertainties and user preferences. However, contemporary QoS prediction methods frequently encounter data sparsity and cold-start issues, which hinder accurate QoS predictions and limit the ability to capture diverse user preferences. Additionally, these methods often assume QoS data reliability, neglecting potential credibility issues such as outliers and the presence of greysheep users and services with atypical invocation patterns. Furthermore, traditional approaches fail to leverage diverse features, including domain-specific knowledge and complex higher-order patterns, essential for accurate QoS predictions. In this paper, we introduce a real-time, trust-aware framework for temporal QoS prediction to address the aforementioned challenges, featuring an end-to-end deep architecture called the Hypergraph Convoluted Transformer Network (HCTN). HCTN combines a hypergraph structure with graph convolution over hyper-edges to effectively address high-sparsity issues by capturing complex, high-order correlations. Complementing this, the transformer network utilizes multi-head attention along with parallel 1D convolutional layers and fully connected dense blocks to capture both fine-grained and coarse-grained dynamic patterns. Additionally, our approach includes a sparsity-resilient solution for detecting greysheep users and services, incorporating their unique characteristics to improve prediction accuracy. Trained with a robust loss function resistant to outliers, HCTN demonstrated state-of-the-art performance on the large-scale WSDREAM-2 datasets for response time and throughput.

Title: Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition

• Authors: Artem Basharin, Andrei Chertkov, Ivan Oseledets
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17765
• Pdf URL: https://arxiv.org/pdf/2410.17765
• Copy Paste: [[2410.17765]] Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition(https://arxiv.org/abs/2410.17765)
• Keywords: robust, transformer
• Abstract: We propose a new model for multi-token prediction in transformers, aiming to enhance sampling efficiency without compromising accuracy. Motivated by recent work that predicts the probabilities of subsequent tokens using multiple heads, we connect this approach to rank-$1$ canonical tensor decomposition. By generalizing it to a rank-$r$ canonical probability decomposition, we develop an improved model that predicts multiple tokens simultaneously. This model can also be interpreted as a mixture of experts, allowing us to leverage successful techniques from that domain for efficient and robust training. Importantly, the overall overhead for training and sampling remains low. Our method demonstrates significant improvements in inference speed for both text and code generation tasks, proving particularly beneficial within the self-speculative decoding paradigm. It maintains its effectiveness across various model sizes and training epochs, highlighting its robustness and scalability.

Title: Locating Information in Large Language Models via Random Matrix Theory

• Authors: Max Staats, Matthias Thamm, Bernd Rosenow
• Subjects: cs.LG, cond-mat.dis-nn
• Abstract URL: https://arxiv.org/abs/2410.17770
• Pdf URL: https://arxiv.org/pdf/2410.17770
• Copy Paste: [[2410.17770]] Locating Information in Large Language Models via Random Matrix Theory(https://arxiv.org/abs/2410.17770)
• Keywords: transformer, large language model
• Abstract: As large language models (LLMs) become central to AI applications, gaining a deeper understanding of their inner workings is increasingly important. In this work, we analyze the weight matrices of pretrained transformer models -- specifically BERT and Llama -- using random matrix theory (RMT) as a zero-information hypothesis. While randomly initialized weights perfectly agree with RMT predictions, deviations emerge after training, allowing us to locate learned structures within the models. We identify layer-type specific behaviors that are consistent across all blocks and architectures considered. By pinpointing regions that deviate from RMT predictions, we highlight areas of feature learning and confirm this through comparisons with the activation covariance matrices of the corresponding layers. Our method provides a diagnostic tool for identifying relevant regions in transformer weights using only the trained matrices. Additionally, we address the ongoing debate regarding the significance of small singular values in the context of fine-tuning and alignment in LLMs. Our findings reveal that, after fine-tuning, small singular values play a crucial role in the models' capabilities, suggesting that removing them in an already aligned transformer can be detrimental, as it may compromise model alignment.

Title: Quasi-Medial Distance Field (Q-MDF): A Robust Method for Approximating and Discretizing Neural Medial Axis

• Authors: Jiayi Kong, Chen Zong, Jun Luo, Shiqing Xin, Fei Hou, Hanqing Jiang, Chen Qian, Ying He
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.17774
• Pdf URL: https://arxiv.org/pdf/2410.17774
• Copy Paste: [[2410.17774]] Quasi-Medial Distance Field (Q-MDF): A Robust Method for Approximating and Discretizing Neural Medial Axis(https://arxiv.org/abs/2410.17774)
• Keywords: robust
• Abstract: The medial axis, a lower-dimensional shape descriptor, plays an important role in the field of digital geometry processing. Despite its importance, robust computation of the medial axis transform from diverse inputs, especially point clouds with defects, remains a significant challenge. In this paper, we tackle the challenge by proposing a new implicit method that diverges from mainstream explicit medial axis computation techniques. Our key technical insight is the difference between the signed distance field (SDF) and the medial field (MF) of a solid shape is the unsigned distance field (UDF) of the shape's medial axis. This allows for formulating medial axis computation as an implicit reconstruction problem. Utilizing a modified double covering method, we extract the medial axis as the zero level-set of the UDF. Extensive experiments show that our method has enhanced accuracy and robustness in learning compact medial axis transform from thorny meshes and point clouds compared to existing methods.

Title: ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning

• Authors: Zhiwei Hao, Jianyuan Guo, Li Shen, Yong Luo, Han Hu, Yonggang Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17779
• Pdf URL: https://arxiv.org/pdf/2410.17779
• Copy Paste: [[2410.17779]] ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning(https://arxiv.org/abs/2410.17779)
• Keywords: large language model
• Abstract: Recent advancements in multimodal fusion have witnessed the remarkable success of vision-language (VL) models, which excel in various multimodal applications such as image captioning and visual question answering. However, building VL models requires substantial hardware resources, where efficiency is restricted by two key factors: the extended input sequence of the language model with vision features demands more computational operations, and a large number of additional learnable parameters increase memory complexity. These challenges significantly restrict the broader applicability of such models. To bridge this gap, we propose ADEM-VL, an efficient vision-language method that tunes VL models based on pretrained large language models (LLMs) by adopting a parameter-free cross-attention mechanism for similarity measurements in multimodal fusion. This approach only requires embedding vision features into the language space, significantly reducing the number of trainable parameters and accelerating both training and inference speeds. To enhance representation learning in fusion module, we introduce an efficient multiscale feature generation scheme that requires only a single forward pass through the vision encoder. Moreover, we propose an adaptive fusion scheme that dynamically discards less relevant visual information for each text token based on its attention score. This ensures that the fusion process prioritizes the most pertinent visual features. With experiments on various tasks including visual question answering, image captioning, and instruction-following, we demonstrate that our framework outperforms existing approaches. Specifically, our method surpasses existing methods by an average accuracy of 0.77% on ScienceQA dataset, with reduced training and inference latency, demonstrating the superiority of our framework. The code is available at this https URL.

Title: Leveraging the Domain Adaptation of Retrieval Augmented Generation Models for Question Answering and Reducing Hallucination

• Authors: Salman Rakin, Md. A.R. Shibly, Zahin M. Hossain, Zeeshan Khan, Md. Mostofa Akbar
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.17783
• Pdf URL: https://arxiv.org/pdf/2410.17783
• Copy Paste: [[2410.17783]] Leveraging the Domain Adaptation of Retrieval Augmented Generation Models for Question Answering and Reducing Hallucination(https://arxiv.org/abs/2410.17783)
• Keywords: large language model
• Abstract: While ongoing advancements in Large Language Models have demonstrated remarkable success across various NLP tasks, Retrieval Augmented Generation Model stands out to be highly effective on downstream applications like Question Answering. Recently, RAG-end2end model further optimized the architecture and achieved notable performance improvements on domain adaptation. However, the effectiveness of these RAG-based architectures remains relatively unexplored when fine-tuned on specialized domains such as customer service for building a reliable conversational AI system. Furthermore, a critical challenge persists in reducing the occurrence of hallucinations while maintaining high domain-specific accuracy. In this paper, we investigated the performance of diverse RAG and RAG-like architectures through domain adaptation and evaluated their ability to generate accurate and relevant response grounded in the contextual knowledge base. To facilitate the evaluation of the models, we constructed a novel dataset HotelConvQA, sourced from wide range of hotel-related conversations and fine-tuned all the models on our domain specific dataset. We also addressed a critical research gap on determining the impact of domain adaptation on reducing hallucinations across different RAG architectures, an aspect that was not properly measured in prior work. Our evaluation shows positive results in all metrics by employing domain adaptation, demonstrating strong performance on QA tasks and providing insights into their efficacy in reducing hallucinations. Our findings clearly indicate that domain adaptation not only enhances the models' performance on QA tasks but also significantly reduces hallucination across all evaluated RAG architectures.

Title: TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sports

• Authors: Guillem Capellera, Luis Ferraz, Antonio Rubio, Antonio Agudo, Francesc Moreno-Noguer
• Subjects: cs.CV, cs.MA
• Abstract URL: https://arxiv.org/abs/2410.17785
• Pdf URL: https://arxiv.org/pdf/2410.17785
• Copy Paste: [[2410.17785]] TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sports(https://arxiv.org/abs/2410.17785)
• Keywords: transformer
• Abstract: Understanding trajectories in multi-agent scenarios requires addressing various tasks, including predicting future movements, imputing missing observations, inferring the status of unseen agents, and classifying different global states. Traditional data-driven approaches often handle these tasks separately with specialized models. We introduce TranSPORTmer, a unified transformer-based framework capable of addressing all these tasks, showcasing its application to the intricate dynamics of multi-agent sports scenarios like soccer and basketball. Using Set Attention Blocks, TranSPORTmer effectively captures temporal dynamics and social interactions in an equivariant manner. The model's tasks are guided by an input mask that conceals missing or yet-to-be-predicted observations. Additionally, we introduce a CLS extra agent to classify states along soccer trajectories, including passes, possessions, uncontrolled states, and out-of-play intervals, contributing to an enhancement in modeling trajectories. Evaluations on soccer and basketball datasets show that TranSPORTmer outperforms state-of-the-art task-specific models in player forecasting, player forecasting-imputation, ball inference, and ball imputation. this https URL

Title: Large Language Models Engineer Too Many Simple Features For Tabular Data

• Authors: Jaris Küken, Lennart Purucker, Frank Hutter
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17787
• Pdf URL: https://arxiv.org/pdf/2410.17787
• Copy Paste: [[2410.17787]] Large Language Models Engineer Too Many Simple Features For Tabular Data(https://arxiv.org/abs/2410.17787)
• Keywords: large language model
• Abstract: Tabular machine learning problems often require time-consuming and labor-intensive feature engineering. Recent efforts have focused on using large language models (LLMs) to capitalize on their potential domain knowledge. At the same time, researchers have observed ethically concerning negative biases in other LLM-related use cases, such as text generation. These developments motivated us to investigate whether LLMs exhibit a bias that negatively impacts the performance of feature engineering. While not ethically concerning, such a bias could hinder practitioners from fully utilizing LLMs for automated data science. Therefore, we propose a method to detect potential biases by detecting anomalies in the frequency of operators (e.g., adding two features) suggested by LLMs when engineering new features. Our experiments evaluate the bias of four LLMs, two big frontier and two small open-source models, across 27 tabular datasets. Our results indicate that LLMs are biased toward simple operators, such as addition, and can fail to utilize more complex operators, such as grouping followed by aggregations. Furthermore, the bias can negatively impact the predictive performance when using LLM-generated features. Our results call for mitigating bias when using LLMs for feature engineering.

Title: Enhancing Federated Learning Convergence with Dynamic Data Queue and Data Entropy-driven Participant Selection

• Authors: Charuka Herath, Xiaolan Liu, Sangarapillai Lambotharan, Yogachandran Rahulamathavan
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17792
• Pdf URL: https://arxiv.org/pdf/2410.17792
• Copy Paste: [[2410.17792]] Enhancing Federated Learning Convergence with Dynamic Data Queue and Data Entropy-driven Participant Selection(https://arxiv.org/abs/2410.17792)
• Keywords: security, privacy, federate
• Abstract: Federated Learning (FL) is a decentralized approach for collaborative model training on edge devices. This distributed method of model training offers advantages in privacy, security, regulatory compliance, and cost-efficiency. Our emphasis in this research lies in addressing statistical complexity in FL, especially when the data stored locally across devices is not identically and independently distributed (non-IID). We have observed an accuracy reduction of up to approximately 10\% to 30\%, particularly in skewed scenarios where each edge device trains with only 1 class of data. This reduction is attributed to weight divergence, quantified using the Euclidean distance between device-level class distributions and the population distribution, resulting in a bias term (\(\delta_k\)). As a solution, we present a method to improve convergence in FL by creating a global subset of data on the server and dynamically distributing it across devices using a Dynamic Data queue-driven Federated Learning (DDFL). Next, we leverage Data Entropy metrics to observe the process during each training round and enable reasonable device selection for aggregation. Furthermore, we provide a convergence analysis of our proposed DDFL to justify their viability in practical FL scenarios, aiming for better device selection, a non-sub-optimal global model, and faster convergence. We observe that our approach results in a substantial accuracy boost of approximately 5\% for the MNIST dataset, around 18\% for CIFAR-10, and 20\% for CIFAR-100 with a 10\% global subset of data, outperforming the state-of-the-art (SOTA) aggregation algorithms.

Title: OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation

• Authors: Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chaohong Tan
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.17799
• Pdf URL: https://arxiv.org/pdf/2410.17799
• Copy Paste: [[2410.17799]] OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation(https://arxiv.org/abs/2410.17799)
• Keywords: large language model
• Abstract: Full-duplex spoken dialogue systems significantly advance over traditional turn-based dialogue systems, as they allow simultaneous bidirectional communication, closely mirroring human-human interactions. However, achieving low latency and natural interactions in full-duplex dialogue systems remains a significant challenge, especially considering human conversation dynamics such as interruptions, backchannels, and overlapping speech. In this paper, we introduce a novel End-to-End GPT-based model OmniFlatten for full-duplex conversation, capable of effectively modeling the complex behaviors inherent to natural conversations with low latency. To achieve full-duplex communication capabilities, we propose a multi-stage post-training scheme that progressively adapts a text-based large language model (LLM) backbone into a speech-text dialogue LLM, capable of generating text and speech in real time, without modifying the architecture of the backbone LLM. The training process comprises three stages: modality alignment, half-duplex dialogue learning, and full-duplex dialogue learning. Throughout all training stages, we standardize the data using a flattening operation, which allows us to unify the training methods and the model architecture across different modalities and tasks. Our approach offers a straightforward modeling technique and a promising research direction for developing efficient and natural end-to-end full-duplex spoken dialogue systems. Audio samples of dialogues generated by OmniFlatten can be found at this web site (this https URL).

Title: GenUDC: High Quality 3D Mesh Generation with Unsigned Dual Contouring Representation

• Authors: Ruowei Wang, Jiaqi Li, Dan Zeng, Xueqi Ma, Zixiang Xu, Jianwei Zhang, Qijun Zhao
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.17802
• Pdf URL: https://arxiv.org/pdf/2410.17802
• Copy Paste: [[2410.17802]] GenUDC: High Quality 3D Mesh Generation with Unsigned Dual Contouring Representation(https://arxiv.org/abs/2410.17802)
• Keywords: diffusion, generative
• Abstract: Generating high-quality meshes with complex structures and realistic surfaces is the primary goal of 3D generative models. Existing methods typically employ sequence data or deformable tetrahedral grids for mesh generation. However, sequence-based methods have difficulty producing complex structures with many faces due to memory limits. The deformable tetrahedral grid-based method MeshDiffusion fails to recover realistic surfaces due to the inherent ambiguity in deformable grids. We propose the GenUDC framework to address these challenges by leveraging the Unsigned Dual Contouring (UDC) as the mesh representation. UDC discretizes a mesh in a regular grid and divides it into the face and vertex parts, recovering both complex structures and fine details. As a result, the one-to-one mapping between UDC and mesh resolves the ambiguity problem. In addition, GenUDC adopts a two-stage, coarse-to-fine generative process for 3D mesh generation. It first generates the face part as a rough shape and then the vertex part to craft a detailed shape. Extensive evaluations demonstrate the superiority of UDC as a mesh representation and the favorable performance of GenUDC in mesh generation. The code and trained models are available at this https URL.

Title: An Intelligent Agentic System for Complex Image Restoration Problems

• Authors: Kaiwen Zhu, Jinjin Gu, Zhiyuan You, Yu Qiao, Chao Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17809
• Pdf URL: https://arxiv.org/pdf/2410.17809
• Copy Paste: [[2410.17809]] An Intelligent Agentic System for Complex Image Restoration Problems(https://arxiv.org/abs/2410.17809)
• Keywords: large language model
• Abstract: Real-world image restoration (IR) is inherently complex and often requires combining multiple specialized models to address diverse degradations. Inspired by human problem-solving, we propose AgenticIR, an agentic system that mimics the human approach to image processing by following five key stages: Perception, Scheduling, Execution, Reflection, and Rescheduling. AgenticIR leverages large language models (LLMs) and vision-language models (VLMs) that interact via text generation to dynamically operate a toolbox of IR models. We fine-tune VLMs for image quality analysis and employ LLMs for reasoning, guiding the system step by step. To compensate for LLMs' lack of specific IR knowledge and experience, we introduce a self-exploration method, allowing the LLM to observe and summarize restoration results into referenceable documents. Experiments demonstrate AgenticIR's potential in handling complex IR tasks, representing a promising path toward achieving general intelligence in visual processing.

Title: EntityCLIP: Entity-Centric Image-Text Matching via Multimodal Attentive Contrastive Learning

• Authors: Yaxiong Wang, Yaxiong Wang, Lianwei Wu, Lechao Cheng, Zhun Zhong, Meng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17810
• Pdf URL: https://arxiv.org/pdf/2410.17810
• Copy Paste: [[2410.17810]] EntityCLIP: Entity-Centric Image-Text Matching via Multimodal Attentive Contrastive Learning(https://arxiv.org/abs/2410.17810)
• Keywords: large language model
• Abstract: Recent advancements in image-text matching have been notable, yet prevailing models predominantly cater to broad queries and struggle with accommodating fine-grained query intention. In this paper, we work towards the \textbf{E}ntity-centric \textbf{I}mage-\textbf{T}ext \textbf{M}atching (EITM), a task that the text and image involve specific entity-related information. The challenge of this task mainly lies in the larger semantic gap in entity association modeling, comparing with the general image-text matching this http URL narrow the huge semantic gap between the entity-centric text and the images, we take the fundamental CLIP as the backbone and devise a multimodal attentive contrastive learning framework to tam CLIP to adapt EITM problem, developing a model named EntityCLIP. The key of our multimodal attentive contrastive learning is to generate interpretive explanation text using Large Language Models (LLMs) as the bridge clues. In specific, we proceed by extracting explanatory text from off-the-shelf LLMs. This explanation text, coupled with the image and text, is then input into our specially crafted Multimodal Attentive Experts (MMAE) module, which effectively integrates explanation texts to narrow the gap of the entity-related text and image in a shared semantic space. Building on the enriched features derived from MMAE, we further design an effective Gated Integrative Image-text Matching (GI-ITM) strategy. The GI-ITM employs an adaptive gating mechanism to aggregate MMAE's features, subsequently applying image-text matching constraints to steer the alignment between the text and the image. Extensive experiments are conducted on three social media news benchmarks including N24News, VisualNews, and GoodNews, the results shows that our method surpasses the competition methods with a clear margin.

Title: Understanding When Tree of Thoughts Succeeds: Larger Models Excel in Generation, Not Discrimination

• Authors: Qiqi Chen, Xinpeng Wang, Philipp Mondorf, Michael A. Hedderich, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17820
• Pdf URL: https://arxiv.org/pdf/2410.17820
• Copy Paste: [[2410.17820]] Understanding When Tree of Thoughts Succeeds: Larger Models Excel in Generation, Not Discrimination(https://arxiv.org/abs/2410.17820)
• Keywords: generative, large language model
• Abstract: Tree of Thoughts (ToT) is a reasoning strategy for Large Language Models (LLMs) that employs a generator to suggest reasoning steps and a discriminator to decide which steps to implement. ToT demonstrates strong performance on reasoning tasks, often surpassing simple methods such as Input-Output (IO) prompting and Chain-of-Thought (CoT) reasoning. However, ToT does not consistently outperform such simpler methods across all models, leaving large knowledge gaps on the conditions under which ToT is most beneficial. In this paper, we analyze the roles of the generator and discriminator separately to better understand the conditions when ToT is beneficial. We find that the generator plays a more critical role than the discriminator in driving the success of ToT. While using even a smaller model as the discriminator, scaling the generator leads to notable improvements in ToT performance, whereas scaling the discriminator with a fixed generator yields only marginal gains. Our results show that models across different scales exhibit comparable discrimination capabilities, yet differ significantly in their generative performance for ToT.

Title: DREB-Net: Dual-stream Restoration Embedding Blur-feature Fusion Network for High-mobility UAV Object Detection

• Authors: Qingpeng Li, Yuxin Zhang, Leyuan Fang, Yuhan Kang, Shutao Li, Xiao Xiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17822
• Pdf URL: https://arxiv.org/pdf/2410.17822
• Copy Paste: [[2410.17822]] DREB-Net: Dual-stream Restoration Embedding Blur-feature Fusion Network for High-mobility UAV Object Detection(https://arxiv.org/abs/2410.17822)
• Keywords: extraction
• Abstract: Object detection algorithms are pivotal components of unmanned aerial vehicle (UAV) imaging systems, extensively employed in complex fields. However, images captured by high-mobility UAVs often suffer from motion blur cases, which significantly impedes the performance of advanced object detection algorithms. To address these challenges, we propose an innovative object detection algorithm specifically designed for blurry images, named DREB-Net (Dual-stream Restoration Embedding Blur-feature Fusion Network). First, DREB-Net addresses the particularities of blurry image object detection problem by incorporating a Blurry image Restoration Auxiliary Branch (BRAB) during the training phase. Second, it fuses the extracted shallow features via Multi-level Attention-Guided Feature Fusion (MAGFF) module, to extract richer features. Here, the MAGFF module comprises local attention modules and global attention modules, which assign different weights to the branches. Then, during the inference phase, the deep feature extraction of the BRAB can be removed to reduce computational complexity and improve detection speed. In loss function, a combined loss of MSE and SSIM is added to the BRAB to restore blurry images. Finally, DREB-Net introduces Fast Fourier Transform in the early stages of feature extraction, via a Learnable Frequency domain Amplitude Modulation Module (LFAMM), to adjust feature amplitude and enhance feature processing capability. Experimental results indicate that DREB-Net can still effectively perform object detection tasks under motion blur in captured images, showcasing excellent performance and broad application prospects. Our source code will be available at this https URL.

Title: Exploiting Text-Image Latent Spaces for the Description of Visual Concepts

• Authors: Laines Schmalwasser, Jakob Gawlikowski, Joachim Denzler, Julia Niebling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17832
• Pdf URL: https://arxiv.org/pdf/2410.17832
• Copy Paste: [[2410.17832]] Exploiting Text-Image Latent Spaces for the Description of Visual Concepts(https://arxiv.org/abs/2410.17832)
• Keywords: extraction
• Abstract: Concept Activation Vectors (CAVs) offer insights into neural network decision-making by linking human friendly concepts to the model's internal feature extraction process. However, when a new set of CAVs is discovered, they must still be translated into a human understandable description. For image-based neural networks, this is typically done by visualizing the most relevant images of a CAV, while the determination of the concept is left to humans. In this work, we introduce an approach to aid the interpretation of newly discovered concept sets by suggesting textual descriptions for each CAV. This is done by mapping the most relevant images representing a CAV into a text-image embedding where a joint description of these relevant images can be computed. We propose utilizing the most relevant receptive fields instead of full images encoded. We demonstrate the capabilities of this approach in multiple experiments with and without given CAV labels, showing that the proposed approach provides accurate descriptions for the CAVs and reduces the challenge of concept interpretation.

Title: Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs

• Authors: Ferdi Kossmann, Bruce Fontaine, Daya Khudia, Michael Cafarella, Samuel Madden
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17840
• Pdf URL: https://arxiv.org/pdf/2410.17840
• Copy Paste: [[2410.17840]] Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs(https://arxiv.org/abs/2410.17840)
• Keywords: large language model
• Abstract: Serving systems for Large Language Models (LLMs) improve throughput by processing several requests concurrently. However, multiplexing hardware resources between concurrent requests involves non-trivial scheduling decisions. Practical serving systems typically implement these decisions at two levels: First, a load balancer routes requests to different servers which each hold a replica of the LLM. Then, on each server, an engine-level scheduler decides when to run a request, or when to queue or preempt it. Improved scheduling policies may benefit a wide range of LLM deployments and can often be implemented as "drop-in replacements" to a system's current policy. In this work, we survey scheduling techniques from the literature and from practical serving systems. We find that schedulers from the literature often achieve good performance but introduce significant complexity. In contrast, schedulers in practical deployments often leave easy performance gains on the table but are easy to implement, deploy and configure. This finding motivates us to introduce two new scheduling techniques, which are both easy to implement, and outperform current techniques on production workload traces.

Title: The Probabilistic Tsetlin Machine: A Novel Approach to Uncertainty Quantification

• Authors: K. Darshana Abeyrathna, Sara El Mekkaoui, Andreas Hafver, Christian Agrell
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17851
• Pdf URL: https://arxiv.org/pdf/2410.17851
• Copy Paste: [[2410.17851]] The Probabilistic Tsetlin Machine: A Novel Approach to Uncertainty Quantification(https://arxiv.org/abs/2410.17851)
• Keywords: robust, interpretability
• Abstract: Tsetlin Machines (TMs) have emerged as a compelling alternative to conventional deep learning methods, offering notable advantages such as smaller memory footprint, faster inference, fault-tolerant properties, and interpretability. Although various adaptations of TMs have expanded their applicability across diverse domains, a fundamental gap remains in understanding how TMs quantify uncertainty in their predictions. In response, this paper introduces the Probabilistic Tsetlin Machine (PTM) framework, aimed at providing a robust, reliable, and interpretable approach for uncertainty quantification. Unlike the original TM, the PTM learns the probability of staying on each state of each Tsetlin Automaton (TA) across all clauses. These probabilities are updated using the feedback tables that are part of the TM framework: Type I and Type II feedback. During inference, TAs decide their actions by sampling states based on learned probability distributions, akin to Bayesian neural networks when generating weight values. In our experimental analysis, we first illustrate the spread of the probabilities across TA states for the noisy-XOR dataset. Then we evaluate the PTM alongside benchmark models using both simulated and real-world datasets. The experiments on the simulated dataset reveal the PTM's effectiveness in uncertainty quantification, particularly in delineating decision boundaries and identifying regions of high uncertainty. Moreover, when applied to multiclass classification tasks using the Iris dataset, the PTM demonstrates competitive performance in terms of predictive entropy and expected calibration error, showcasing its potential as a reliable tool for uncertainty estimation. Our findings underscore the importance of selecting appropriate models for accurate uncertainty quantification in predictive tasks, with the PTM offering a particularly interpretable and effective solution.

Title: TAGE: Trustworthy Attribute Group Editing for Stable Few-shot Image Generation

• Authors: Ruicheng Zhang, Guoheng Huang, Yejing Huo, Xiaochen Yuan, Zhizhen Zhou, Xuhang Chen, Guo Zhong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17855
• Pdf URL: https://arxiv.org/pdf/2410.17855
• Copy Paste: [[2410.17855]] TAGE: Trustworthy Attribute Group Editing for Stable Few-shot Image Generation(https://arxiv.org/abs/2410.17855)
• Keywords: transformer, generative
• Abstract: Generative Adversarial Networks (GANs) have emerged as a prominent research focus for image editing tasks, leveraging the powerful image generation capabilities of the GAN framework to produce remarkable this http URL, prevailing approaches are contingent upon extensive training datasets and explicit supervision, presenting a significant challenge in manipulating the diverse attributes of new image classes with limited sample availability. To surmount this hurdle, we introduce TAGE, an innovative image generation network comprising three integral modules: the Codebook Learning Module (CLM), the Code Prediction Module (CPM) and the Prompt-driven Semantic Module (PSM). The CPM module delves into the semantic dimensions of category-agnostic attributes, encapsulating them within a discrete codebook. This module is predicated on the concept that images are assemblages of attributes, and thus, by editing these category-independent attributes, it is theoretically possible to generate images from unseen categories. Subsequently, the CPM module facilitates naturalistic image editing by predicting indices of category-independent attribute vectors within the codebook. Additionally, the PSM module generates semantic cues that are seamlessly integrated into the Transformer architecture of the CPM, enhancing the model's comprehension of the targeted attributes for editing. With these semantic cues, the model can generate images that accentuate desired attributes more prominently while maintaining the integrity of the original category, even with a limited number of samples. We have conducted extensive experiments utilizing the Animal Faces, Flowers, and VGGFaces datasets. The results of these experiments demonstrate that our proposed method not only achieves superior performance but also exhibits a high degree of stability when compared to other few-shot image generation techniques.

Title: ROCKET-1: Master Open-World Interaction with Visual-Temporal Context Prompting

• Authors: Shaofei Cai, Zihao Wang, Kewei Lian, Zhancun Mu, Xiaojian Ma, Anji Liu, Yitao Liang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17856
• Pdf URL: https://arxiv.org/pdf/2410.17856
• Copy Paste: [[2410.17856]] ROCKET-1: Master Open-World Interaction with Visual-Temporal Context Prompting(https://arxiv.org/abs/2410.17856)
• Keywords: segmentation
• Abstract: Vision-language models (VLMs) have excelled in multimodal tasks, but adapting them to embodied decision-making in open-world environments presents challenges. A key issue is the difficulty in smoothly connecting individual entities in low-level observations with abstract concepts required for planning. A common approach to address this problem is through the use of hierarchical agents, where VLMs serve as high-level reasoners that break down tasks into executable sub-tasks, typically specified using language and imagined observations. However, language often fails to effectively convey spatial information, while generating future images with sufficient accuracy remains challenging. To address these limitations, we propose visual-temporal context prompting, a novel communication protocol between VLMs and policy models. This protocol leverages object segmentation from both past and present observations to guide policy-environment interactions. Using this approach, we train ROCKET-1, a low-level policy that predicts actions based on concatenated visual observations and segmentation masks, with real-time object tracking provided by SAM-2. Our method unlocks the full potential of VLMs visual-language reasoning abilities, enabling them to solve complex creative tasks, especially those heavily reliant on spatial understanding. Experiments in Minecraft demonstrate that our approach allows agents to accomplish previously unattainable tasks, highlighting the effectiveness of visual-temporal context prompting in embodied decision-making. Codes and demos will be available on the project page: this https URL.

Title: Understanding Layer Significance in LLM Alignment

• Authors: Guangyuan Shi, Zexin Lu, Xiaoyu Dong, Wenlong Zhang, Xuanyu Zhang, Yujie Feng, Xiao-Ming Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17875
• Pdf URL: https://arxiv.org/pdf/2410.17875
• Copy Paste: [[2410.17875]] Understanding Layer Significance in LLM Alignment(https://arxiv.org/abs/2410.17875)
• Keywords: large language model
• Abstract: Aligning large language models (LLMs) through fine-tuning is essential for tailoring them to specific applications. Therefore, understanding what LLMs learn during the alignment process is crucial. Recent studies suggest that alignment primarily adjusts a model's presentation style rather than its foundational knowledge, indicating that only certain components of the model are significantly impacted. To delve deeper into LLM alignment, we propose to identify which layers within LLMs are most critical to the alignment process, thereby uncovering how alignment influences model behavior at a granular level. We propose a novel approach to identify the important layers for LLM alignment (ILA). It involves learning a binary mask for each incremental weight matrix in the LoRA algorithm, indicating the significance of each layer. ILA consistently identifies important layers across various alignment datasets, with nearly 90% overlap even with substantial dataset differences, highlighting fundamental patterns in LLM alignment. Experimental results indicate that freezing non-essential layers improves overall model performance, while selectively tuning the most critical layers significantly enhances fine-tuning efficiency with minimal performance loss.

Title: A utility-based spatial analysis of residential street-level conditions; A case study of Rotterdam

• Authors: Sander van Cranenburgh, Francisco Garrido-Valenzuela
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17880
• Pdf URL: https://arxiv.org/pdf/2410.17880
• Copy Paste: [[2410.17880]] A utility-based spatial analysis of residential street-level conditions; A case study of Rotterdam(https://arxiv.org/abs/2410.17880)
• Keywords: explainability
• Abstract: Residential location choices are traditionally modelled using factors related to accessibility and socioeconomic environments, neglecting the importance of local street-level conditions. Arguably, this neglect is due to data practices. Today, however, street-level images -- which are highly effective at encoding street-level conditions -- are widely available. Additionally, recent advances in discrete choice models incorporating computer vision capabilities offer opportunities to integrate street-level conditions into residential location choice analysis. This study leverages these developments to investigate the spatial distribution of utility derived from street-level conditions in residential location choices on a city-wide scale. In our case study of Rotterdam, the Netherlands, we find that the utility derived from street-level conditions varies significantly on a highly localised scale, with conditions rapidly changing even within neighbourhoods. Our results also reveal that the high real-estate prices in the city centre cannot be attributed to attractive street-level conditions. Furthermore, whereas the city centre is characterised by relatively unattractive residential street-level conditions, neighbourhoods in the southern part of the city -- often perceived as problematic -- exhibit surprisingly appealing street-level environments. The methodological contribution of this paper is that it advances the discrete choice models incorporating computer vision capabilities by introducing a semantic regularisation layer to the model. Thereby, it adds explainability and eliminates the need for a separate pipeline to extract information from images, streamlining the analysis. As such, this paper's findings and methodological advancements pave the way for further studies to explore integrating street-level conditions in urban planning.

Title: AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning

• Authors: Yehonathan Refael, Jonathan Svirsky, Boris Shustin, Wasim Huleihel, Ofir Lindenbaum
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17881
• Pdf URL: https://arxiv.org/pdf/2410.17881
• Copy Paste: [[2410.17881]] AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning(https://arxiv.org/abs/2410.17881)
• Keywords: large language model
• Abstract: Training and fine-tuning large language models (LLMs) come with challenges related to memory and computational requirements due to the increasing size of the model weights and the optimizer states. Various techniques have been developed to tackle these challenges, such as low-rank adaptation (LoRA), which involves introducing a parallel trainable low-rank matrix to the fixed pre-trained weights at each layer. However, these methods often fall short compared to the full-rank weight training approach, as they restrict the parameter search to a low-rank subspace. This limitation can disrupt training dynamics and require a full-rank warm start to mitigate the impact. In this paper, we introduce a new method inspired by a phenomenon we formally prove: as training progresses, the rank of the estimated layer gradients gradually decreases, and asymptotically approaches rank one. Leveraging this, our approach involves adaptively reducing the rank of the gradients during Adam optimization steps, using an efficient online-updating low-rank projections rule. We further present a randomized SVD scheme for efficiently finding the projection matrix. Our technique enables full-parameter fine-tuning with adaptive low-rank gradient updates, significantly reducing overall memory requirements during training compared to state-of-the-art methods while improving model performance in both pretraining and fine-tuning. Finally, we provide a convergence analysis of our method and demonstrate its merits for training and fine-tuning language and biological foundation models.

Title: Scaling Diffusion Language Models via Adaptation from Autoregressive Models

• Authors: Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, Lingpeng Kong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17891
• Pdf URL: https://arxiv.org/pdf/2410.17891
• Copy Paste: [[2410.17891]] Scaling Diffusion Language Models via Adaptation from Autoregressive Models(https://arxiv.org/abs/2410.17891)
• Keywords: fair, diffusion, generative
• Abstract: Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling, potentially addressing limitations of autoregressive (AR) models. However, current DLMs have been studied at a smaller scale compared to their AR counterparts and lack fair comparison on language modeling benchmarks. Additionally, training diffusion models from scratch at scale remains challenging. Given the prevalence of open-source AR language models, we propose adapting these models to build text diffusion models. We demonstrate connections between AR and diffusion modeling objectives and introduce a simple continual pre-training approach for training diffusion models. Through systematic evaluation on language modeling, reasoning, and commonsense benchmarks, we show that we can convert AR models ranging from 127M to 7B parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training. Our experimental results reveal that these models outperform earlier DLMs and are competitive with their AR counterparts. We release a suite of DLMs (with 127M, 355M, and 7B parameters) capable of generating fluent text, performing in-context learning, filling in the middle without prompt re-ordering, and following instructions \url{this https URL}.

Title: Value Residual Learning For Alleviating Attention Concentration In Transformers

• Authors: Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Zhenzhong Lan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17897
• Pdf URL: https://arxiv.org/pdf/2410.17897
• Copy Paste: [[2410.17897]] Value Residual Learning For Alleviating Attention Concentration In Transformers(https://arxiv.org/abs/2410.17897)
• Keywords: transformer
• Abstract: Transformers can capture long-range dependencies using self-attention, allowing tokens to attend to all others directly. However, stacking multiple attention layers leads to attention concentration. One natural way to address this issue is to use cross-layer attention, allowing information from earlier layers to be directly accessible to later layers. However, this approach is computationally expensive. To address this problem, we propose Transformer with residual value (ResFormer) which approximates cross-layer attention through adding a residual connection from the values of the the first layer to all subsequent layers. Based on this method, one variant is the Transformer with single layer value (SVFormer), where all layers share the same value embedding from first layer, reducing the KV cache by nearly 50%. Comprehensive empirical evidence demonstrates that ResFormer mitigates attention concentration problem in deeper layers and enhances representation across most layers, outperforming the vanilla Transformer, DenseFormer, and NeuTRENO in training error as well as downstream tasks. SVFormer trains significantly faster than the vanilla Transformer and performs better than other methods like GQA and CLA, with performance influenced by sequence length and cumulative learning rate.

Title: Scalable Offline Reinforcement Learning for Mean Field Games

• Authors: Axel Brunnbauer, Julian Lemmel, Zahra Babaiee, Sophie Neubauer, Radu Grosu
• Subjects: cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2410.17898
• Pdf URL: https://arxiv.org/pdf/2410.17898
• Copy Paste: [[2410.17898]] Scalable Offline Reinforcement Learning for Mean Field Games(https://arxiv.org/abs/2410.17898)
• Keywords: robust
• Abstract: Reinforcement learning algorithms for mean-field games offer a scalable framework for optimizing policies in large populations of interacting agents. Existing methods often depend on online interactions or access to system dynamics, limiting their practicality in real-world scenarios where such interactions are infeasible or difficult to model. In this paper, we present Offline Munchausen Mirror Descent (Off-MMD), a novel mean-field RL algorithm that approximates equilibrium policies in mean-field games using purely offline data. By leveraging iterative mirror descent and importance sampling techniques, Off-MMD estimates the mean-field distribution from static datasets without relying on simulation or environment dynamics. Additionally, we incorporate techniques from offline reinforcement learning to address common issues like Q-value overestimation, ensuring robust policy learning even with limited data coverage. Our algorithm scales to complex environments and demonstrates strong performance on benchmark tasks like crowd exploration or navigation, highlighting its applicability to real-world multi-agent systems where online experimentation is infeasible. We empirically demonstrate the robustness of Off-MMD to low-quality datasets and conduct experiments to investigate its sensitivity to hyperparameter choices.

Title: ELAICHI: Enhancing Low-resource TTS by Addressing Infrequent and Low-frequency Character Bigrams

• Authors: Srija Anand, Praveen Srinivasa Varadhan, Mehak Singal, Mitesh M. Khapra
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.17901
• Pdf URL: https://arxiv.org/pdf/2410.17901
• Copy Paste: [[2410.17901]] ELAICHI: Enhancing Low-resource TTS by Addressing Infrequent and Low-frequency Character Bigrams(https://arxiv.org/abs/2410.17901)
• Keywords: robust
• Abstract: Recent advancements in Text-to-Speech (TTS) technology have led to natural-sounding speech for English, primarily due to the availability of large-scale, high-quality web data. However, many other languages lack access to such resources, relying instead on limited studio-quality data. This scarcity results in synthesized speech that often suffers from intelligibility issues, particularly with low-frequency character bigrams. In this paper, we propose three solutions to address this challenge. First, we leverage high-quality data from linguistically or geographically related languages to improve TTS for the target language. Second, we utilize low-quality Automatic Speech Recognition (ASR) data recorded in non-studio environments, which is refined using denoising and speech enhancement models. Third, we apply knowledge distillation from large-scale models using synthetic data to generate more robust outputs. Our experiments with Hindi demonstrate significant reductions in intelligibility issues, as validated by human evaluators. We propose this methodology as a viable alternative for languages with limited access to high-quality data, enabling them to collectively benefit from shared resources.

Title: Slot: Provenance-Driven APT Detection through Graph Reinforcement Learning

• Authors: Wei Qiao, Yebo Feng, Teng Li, Zijian Zhang, Zhengzi Xu, Zhuo Ma, Yulong Shen, JianFeng Ma, Yang Liu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17910
• Pdf URL: https://arxiv.org/pdf/2410.17910
• Copy Paste: [[2410.17910]] Slot: Provenance-Driven APT Detection through Graph Reinforcement Learning(https://arxiv.org/abs/2410.17910)
• Keywords: security, protect, defense, attack, robust
• Abstract: Advanced Persistent Threats (APTs) represent sophisticated cyberattacks characterized by their ability to remain undetected within the victim system for extended periods, aiming to exfiltrate sensitive data or disrupt operations. Existing detection approaches often struggle to effectively identify these complex threats, construct the attack chain for defense facilitation, or resist adversarial attacks. To overcome these challenges, we propose Slot, an advanced APT detection approach based on provenance graphs and graph reinforcement learning. Slot excels in uncovering multi-level hidden relationships, such as causal, contextual, and indirect connections, among system behaviors through provenance graph mining. By pioneering the integration of graph reinforcement learning, Slot dynamically adapts to new user activities and evolving attack strategies, enhancing its resilience against adversarial attacks. Additionally, Slot automatically constructs the attack chain according to detected attacks with clustering algorithms, providing precise identification of attack paths and facilitating the development of defense strategies. Evaluations with real-world datasets demonstrate Slot's outstanding accuracy, efficiency, adaptability, and robustness in APT detection, with most metrics surpassing state-of-the-art methods. Additionally, case studies conducted to assess Slot's effectiveness in supporting APT defense further establish it as a practical and reliable tool for cybersecurity protection.

Title: Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation

• Authors: Wenfang Yao, Chen Liu, Kejing Yin, William K. Cheung, Jing Qin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17918
• Pdf URL: https://arxiv.org/pdf/2410.17918
• Copy Paste: [[2410.17918]] Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation(https://arxiv.org/abs/2410.17918)
• Keywords: diffusion
• Abstract: Integrating multi-modal clinical data, such as electronic health records (EHR) and chest X-ray images (CXR), is particularly beneficial for clinical prediction tasks. However, in a temporal setting, multi-modal data are often inherently asynchronous. EHR can be continuously collected but CXR is generally taken with a much longer interval due to its high cost and radiation dose. When clinical prediction is needed, the last available CXR image might have been outdated, leading to suboptimal predictions. To address this challenge, we propose DDL-CXR, a method that dynamically generates an up-to-date latent representation of the individualized CXR images. Our approach leverages latent diffusion models for patient-specific generation strategically conditioned on a previous CXR image and EHR time series, providing information regarding anatomical structures and disease progressions, respectively. In this way, the interaction across modalities could be better captured by the latent CXR generation process, ultimately improving the prediction performance. Experiments using MIMIC datasets show that the proposed model could effectively address asynchronicity in multimodal fusion and consistently outperform existing methods.

Title: Gaze-Assisted Medical Image Segmentation

• Authors: Leila Khaertdinova, Ilya Pershin, Tatiana Shmykova, Bulat Ibragimov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17920
• Pdf URL: https://arxiv.org/pdf/2410.17920
• Copy Paste: [[2410.17920]] Gaze-Assisted Medical Image Segmentation(https://arxiv.org/abs/2410.17920)
• Keywords: segmentation
• Abstract: The annotation of patient organs is a crucial part of various diagnostic and treatment procedures, such as radiotherapy planning. Manual annotation is extremely time-consuming, while its automation using modern image analysis techniques has not yet reached levels sufficient for clinical adoption. This paper investigates the idea of semi-supervised medical image segmentation using human gaze as interactive input for segmentation correction. In particular, we fine-tuned the Segment Anything Model in Medical Images (MedSAM), a public solution that uses various prompt types as additional input for semi-automated segmentation correction. We used human gaze data from reading abdominal images as a prompt for fine-tuning MedSAM. The model was validated on a public WORD database, which consists of 120 CT scans of 16 abdominal organs. The results of the gaze-assisted MedSAM were shown to be superior to the results of the state-of-the-art segmentation models. In particular, the average Dice coefficient for 16 abdominal organs was 85.8%, 86.7%, 81.7%, and 90.5% for nnUNetV2, ResUNet, original MedSAM, and our gaze-assisted MedSAM model, respectively.

Title: Securing Stack Smashing Protection in WebAssembly Applications

• Authors: Quentin Michaud, Yohan Pipereau, Olivier Levillain, Dhouha Ayed
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17925
• Pdf URL: https://arxiv.org/pdf/2410.17925
• Copy Paste: [[2410.17925]] Securing Stack Smashing Protection in WebAssembly Applications(https://arxiv.org/abs/2410.17925)
• Keywords: secure, protect, robust
• Abstract: WebAssembly is an instruction set architecture and binary format standard, designed for secure execution by an interpreter. Previous work has shown that WebAssembly is vulnerable to buffer overflow due to the lack of effective protection mechanisms. In this paper, we evaluate the implementation of Stack Smashing Protection (SSP) in WebAssembly standalone runtimes, and uncover two weaknesses in their current implementation. The first one is the possibility to overwrite the SSP reference value because of the contiguous memory zones inside a WebAssembly process. The second comes from the reliance of WebAssembly on the runtime to provide randomness in order to initialize the SSP reference value, which impacts the robustness of the solution. We address these two flaws by hardening the SSP implementation in terms of storage and random generator failure, in a way that is generalizable to all of WebAssembly. We evaluate our new, more robust, solution to prove that the implemented improvements do not reduce the efficiency of SSP.

Title: Multi-Continental Healthcare Modelling Using Blockchain-Enabled Federated Learning

• Authors: Rui Sun, Zhipeng Wang, Hengrui Zhang, Ming Jiang, Yizhe Wen, Jiqun Zhang, Jiahao Sun, Shuoying Zhang, Erwu Liu, Kezhi Li
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17933
• Pdf URL: https://arxiv.org/pdf/2410.17933
• Copy Paste: [[2410.17933]] Multi-Continental Healthcare Modelling Using Blockchain-Enabled Federated Learning(https://arxiv.org/abs/2410.17933)
• Keywords: privacy, federate
• Abstract: One of the biggest challenges of building artificial intelligence (AI) model in healthcare area is the data sharing. Since healthcare data is private, sensitive, and heterogeneous, collecting sufficient data for modelling is exhausted, costly, and sometimes impossible. In this paper, we propose a framework for global healthcare modelling using datasets from multi-continents (Europe, North America and Asia) while without sharing the local datasets, and choose glucose management as a study model to verify its effectiveness. Technically, blockchain-enabled federated learning is implemented with adaption to make it meet with the privacy and safety requirements of healthcare data, meanwhile rewards honest participation and penalize malicious activities using its on-chain incentive mechanism. Experimental results show that the proposed framework is effective, efficient, and privacy preserved. Its prediction accuracy is much better than the models trained from limited personal data and is similar to, and even slightly better than, the results from a centralized dataset. This work paves the way for international collaborations on healthcare projects, where additional data is crucial for reducing bias and providing benefits to humanity.

Title: SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains

• Authors: Ran Xu, Hui Liu, Sreyashi Nag, Zhenwei Dai, Yaochen Xie, Xianfeng Tang, Chen Luo, Yang Li, Joyce C. Ho, Carl Yang, Qi He
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17952
• Pdf URL: https://arxiv.org/pdf/2410.17952
• Copy Paste: [[2410.17952]] SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains(https://arxiv.org/abs/2410.17952)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) enhances the question-answering (QA) abilities of large language models (LLMs) by integrating external knowledge. However, adapting general-purpose RAG systems to specialized fields such as science and medicine poses unique challenges due to distribution shifts and limited access to domain-specific data. To tackle this, we propose SimRAG, a self-training approach that equips the LLM with joint capabilities of question answering and question generation for domain adaptation. Our method first fine-tunes the LLM on instruction-following, question-answering, and search-related data. Then, it prompts the same LLM to generate diverse domain-relevant questions from unlabeled corpora, with an additional filtering strategy to retain high-quality synthetic examples. By leveraging these synthetic examples, the LLM can improve their performance on domain-specific RAG tasks. Experiments on 11 datasets, spanning two backbone sizes and three domains, demonstrate that SimRAG outperforms baselines by 1.2\%--8.6\%.

Title: Closed-form merging of parameter-efficient modules for Federated Continual Learning

• Authors: Riccardo Salami, Pietro Buzzega, Matteo Mosconi, Jacopo Bonato, Luigi Sabetta, Simone Calderara
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17961
• Pdf URL: https://arxiv.org/pdf/2410.17961
• Copy Paste: [[2410.17961]] Closed-form merging of parameter-efficient modules for Federated Continual Learning(https://arxiv.org/abs/2410.17961)
• Keywords: federate
• Abstract: Model merging has emerged as a crucial technique in Deep Learning, enabling the integration of multiple models into a unified system while preserving performance and scalability. In this respect, the compositional properties of low-rank adaptation techniques (e.g., LoRA) have proven beneficial, as simple averaging LoRA modules yields a single model that mostly integrates the capabilities of all individual modules. Building on LoRA, we take a step further by imposing that the merged model matches the responses of all learned modules. Solving this objective in closed form yields an indeterminate system with A and B as unknown variables, indicating the existence of infinitely many closed-form solutions. To address this challenge, we introduce LoRM, an alternating optimization strategy that trains one LoRA matrix at a time. This allows solving for each unknown variable individually, thus finding a unique solution. We apply our proposed methodology to Federated Class-Incremental Learning (FCIL), ensuring alignment of model responses both between clients and across tasks. Our method demonstrates state-of-the-art performance across a range of FCIL scenarios.

Title: POMDP-Driven Cognitive Massive MIMO Radar: Joint Target Detection-Tracking In Unknown Disturbances

• Authors: Imad Bouhou, Stefano Fortunati, Leila Gharsalli, Alexandre Renaux
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2410.17967
• Pdf URL: https://arxiv.org/pdf/2410.17967
• Copy Paste: [[2410.17967]] POMDP-Driven Cognitive Massive MIMO Radar: Joint Target Detection-Tracking In Unknown Disturbances(https://arxiv.org/abs/2410.17967)
• Keywords: robust
• Abstract: The joint detection and tracking of a moving target embedded in an unknown disturbance represents a key feature that motivates the development of the cognitive radar paradigm. Building upon recent advancements in robust target detection with multiple-input multiple-output (MIMO) radars, this work explores the application of a Partially Observable Markov Decision Process (POMDP) framework to enhance the tracking and detection tasks in a statistically unknown environment. In the POMDP setup, the radar system is considered as an intelligent agent that continuously senses the surrounding environment, optimizing its actions to maximize the probability of detection $(P_D)$ and improve the target position and velocity estimation, all this while keeping a constant probability of false alarm $(P_{FA})$. The proposed approach employs an online algorithm that does not require any apriori knowledge of the noise statistics, and it relies on a much more general observation model than the traditional range-azimuth-elevation model employed by conventional tracking algorithms. Simulation results clearly show substantial performance improvement of the POMDP-based algorithm compared to the State-Action-Reward-State-Action (SARSA)-based one that has been recently investigated in the context of massive MIMO (MMIMO) radar systems.

Title: Together We Can: Multilingual Automatic Post-Editing for Low-Resource Languages

• Authors: Sourabh Deoghare, Diptesh Kanojia, Pushpak Bhattacharyya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17973
• Pdf URL: https://arxiv.org/pdf/2410.17973
• Copy Paste: [[2410.17973]] Together We Can: Multilingual Automatic Post-Editing for Low-Resource Languages(https://arxiv.org/abs/2410.17973)
• Keywords: robust
• Abstract: This exploratory study investigates the potential of multilingual Automatic Post-Editing (APE) systems to enhance the quality of machine translations for low-resource Indo-Aryan languages. Focusing on two closely related language pairs, English-Marathi and English-Hindi, we exploit the linguistic similarities to develop a robust multilingual APE model. To facilitate cross-linguistic transfer, we generate synthetic Hindi-Marathi and Marathi-Hindi APE triplets. Additionally, we incorporate a Quality Estimation (QE)-APE multi-task learning framework. While the experimental results underline the complementary nature of APE and QE, we also observe that QE-APE multitask learning facilitates effective domain adaptation. Our experiments demonstrate that the multilingual APE models outperform their corresponding English-Hindi and English-Marathi single-pair models by $2.5$ and $2.39$ TER points, respectively, with further notable improvements over the multilingual APE model observed through multi-task learning ($+1.29$ and $+1.44$ TER points), data augmentation ($+0.53$ and $+0.45$ TER points) and domain adaptation ($+0.35$ and $+0.45$ TER points). We release the synthetic data, code, and models accrued during this study publicly at this https URL.

Title: Robust Two-View Geometry Estimation with Implicit Differentiation

• Authors: Vladislav Pyatov, Iaroslav Koshelev, Stamatis Lefkimmiatis
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.17983
• Pdf URL: https://arxiv.org/pdf/2410.17983
• Copy Paste: [[2410.17983]] Robust Two-View Geometry Estimation with Implicit Differentiation(https://arxiv.org/abs/2410.17983)
• Keywords: robust, extraction
• Abstract: We present a novel two-view geometry estimation framework which is based on a differentiable robust loss function fitting. We propose to treat the robust fundamental matrix estimation as an implicit layer, which allows us to avoid backpropagation through time and significantly improves the numerical stability. To take full advantage of the information from the feature matching stage we incorporate learnable weights that depend on the matching confidences. In this way our solution brings together feature extraction, matching and two-view geometry estimation in a unified end-to-end trainable pipeline. We evaluate our approach on the camera pose estimation task in both outdoor and indoor scenarios. The experiments on several datasets show that the proposed method outperforms both classic and learning-based state-of-the-art methods by a large margin. The project webpage is available at: this https URL

Title: Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data

• Authors: Zhaomin Wu, Junyi Hou, Yiqun Diao, Bingsheng He
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17986
• Pdf URL: https://arxiv.org/pdf/2410.17986
• Copy Paste: [[2410.17986]] Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data(https://arxiv.org/abs/2410.17986)
• Keywords: secure, privacy, protect, federate, transformer
• Abstract: Federated Learning (FL) is an evolving paradigm that enables multiple parties to collaboratively train models without sharing raw data. Among its variants, Vertical Federated Learning (VFL) is particularly relevant in real-world, cross-organizational collaborations, where distinct features of a shared instance group are contributed by different parties. In these scenarios, parties are often linked using fuzzy identifiers, leading to a common practice termed as multi-party fuzzy VFL. Existing models generally address either multi-party VFL or fuzzy VFL between two parties. Extending these models to practical multi-party fuzzy VFL typically results in significant performance degradation and increased costs for maintaining privacy. To overcome these limitations, we introduce the Federated Transformer (FeT), a novel framework that supports multi-party VFL with fuzzy identifiers. FeT innovatively encodes these identifiers into data representations and employs a transformer architecture distributed across different parties, incorporating three new techniques to enhance performance. Furthermore, we have developed a multi-party privacy framework for VFL that integrates differential privacy with secure multi-party computation, effectively protecting local representations while minimizing associated utility costs. Our experiments demonstrate that the FeT surpasses the baseline models by up to 46\% in terms of accuracy when scaled to 50 parties. Additionally, in two-party fuzzy VFL settings, FeT also shows improved performance and privacy over cutting-edge VFL models.

Title: Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matrices

• Authors: Chanwoo Chun, SueYeon Chung, Daniel D. Lee
• Subjects: cs.LG, math.SP, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.17998
• Pdf URL: https://arxiv.org/pdf/2410.17998
• Copy Paste: [[2410.17998]] Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matrices(https://arxiv.org/abs/2410.17998)
• Keywords: robust
• Abstract: Analyzing the structure of sampled features from an input data distribution is challenging when constrained by limited measurements in both the number of inputs and features. Traditional approaches often rely on the eigenvalue spectrum of the sample covariance matrix derived from finite measurement matrices; however, these spectra are sensitive to the size of the measurement matrix, leading to biased insights. In this paper, we introduce a novel algorithm that provides unbiased estimates of the spectral moments of the kernel integral operator in the limit of infinite inputs and features from finitely sampled measurement matrices. Our method, based upon dynamic programming, is efficient and capable of estimating the moments of the operator spectrum. We demonstrate the accuracy of our estimator on radial basis function (RBF) kernels, highlighting its consistency with the theoretical spectra. Furthermore, we showcase the practical utility and robustness of our method in understanding the geometry of learned representations in neural networks.

Title: MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

• Authors: Jingfan Zhang, Yi Zhao, Dan Chen, Xing Tian, Huanran Zheng, Wei Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18035
• Pdf URL: https://arxiv.org/pdf/2410.18035
• Copy Paste: [[2410.18035]] MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning(https://arxiv.org/abs/2410.18035)
• Keywords: transformer, large language model
• Abstract: Low-rank adaptation (LoRA) and its mixture-of-experts (MOE) variants are highly effective parameter-efficient fine-tuning (PEFT) methods. However, they introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules in the Transformer layer. To address this issue, we propose Mixture of Low-Rank Adaptation (MiLoRA), a novel and efficient LoRA variant. MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism. This mechanism calculates expert routing results once before generating the first new token and reuses these results for subsequent tokens, reducing latency. Extensive experiments and analysis on commonsense reasoning tasks, math reasoning tasks, and widely used LLM evaluation benchmarks demonstrate that MiLoRA consistently outperforms strong PEFT baselines with comparable tunable parameter budgets. Additionally, MiLoRA significantly reduces latency in multi-tenant settings compared to previous LoRA-based methods.

Title: Key Algorithms for Keyphrase Generation: Instruction-Based LLMs for Russian Scientific Keyphrases

• Authors: Anna Glazkova, Dmitry Morozov, Timur Garipov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18040
• Pdf URL: https://arxiv.org/pdf/2410.18040
• Copy Paste: [[2410.18040]] Key Algorithms for Keyphrase Generation: Instruction-Based LLMs for Russian Scientific Keyphrases(https://arxiv.org/abs/2410.18040)
• Keywords: large language model
• Abstract: Keyphrase selection is a challenging task in natural language processing that has a wide range of applications. Adapting existing supervised and unsupervised solutions for the Russian language faces several limitations due to the rich morphology of Russian and the limited number of training datasets available. Recent studies conducted on English texts show that large language models (LLMs) successfully address the task of generating keyphrases. LLMs allow achieving impressive results without task-specific fine-tuning, using text prompts instead. In this work, we access the performance of prompt-based methods for generating keyphrases for Russian scientific abstracts. First, we compare the performance of zero-shot and few-shot prompt-based methods, fine-tuned models, and unsupervised methods. Then we assess strategies for selecting keyphrase examples in a few-shot setting. We present the outcomes of human evaluation of the generated keyphrases and analyze the strengths and weaknesses of the models through expert assessment. Our results suggest that prompt-based methods can outperform common baselines even using simple text prompts.

Title: LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering

• Authors: Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, Jie Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18050
• Pdf URL: https://arxiv.org/pdf/2410.18050
• Copy Paste: [[2410.18050]] LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering(https://arxiv.org/abs/2410.18050)
• Keywords: robust, large language model
• Abstract: Long-Context Question Answering (LCQA), a challenging task, aims to reason over long-context documents to yield accurate answers to questions. Existing long-context Large Language Models (LLMs) for LCQA often struggle with the "lost in the middle" issue. Retrieval-Augmented Generation (RAG) mitigates this issue by providing external factual evidence. However, its chunking strategy disrupts the global long-context information, and its low-quality retrieval in long contexts hinders LLMs from identifying effective factual details due to substantial noise. To this end, we propose LongRAG, a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG's understanding of complex long-context knowledge (i.e., global information and factual details). We design LongRAG as a plug-and-play paradigm, facilitating adaptation to various domains and LLMs. Extensive experiments on three multi-hop datasets demonstrate that LongRAG significantly outperforms long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and Vanilla RAG (up by 17.25%). Furthermore, we conduct quantitative ablation studies and multi-dimensional analyses, highlighting the effectiveness of the system's components and fine-tuning strategies. Data and code are available at this https URL.

Title: Real time anomalies detection on video

• Authors: Fabien Poirier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18051
• Pdf URL: https://arxiv.org/pdf/2410.18051
• Copy Paste: [[2410.18051]] Real time anomalies detection on video(https://arxiv.org/abs/2410.18051)
• Keywords: security
• Abstract: Nowadays, many places use security cameras. Unfortunately, when an incident occurs, these technologies are used to show past events. So it can be considered as a deterrence tool than a detection tool. In this article, we will propose a deep learning approach trying to solve this problematic. This approach uses convolutional models (CNN) to extract relevant characteristics linked to the video images, theses characteristics will form times series to be analyzed by LSTM / GRU models.

Title: B-Side: Binary-Level Static System Call Identification

• Authors: Gaspard Thévenon, Kevin Nguetchouang, Kahina Lazri, Alain Tchana, Pierre Olivier
• Subjects: cs.CR, cs.OS
• Abstract URL: https://arxiv.org/abs/2410.18053
• Pdf URL: https://arxiv.org/pdf/2410.18053
• Copy Paste: [[2410.18053]] B-Side: Binary-Level Static System Call Identification(https://arxiv.org/abs/2410.18053)
• Keywords: secure
• Abstract: System call filtering is widely used to secure programs in multi-tenant environments, and to sandbox applications in modern desktop software deployment and package management systems. Filtering rules are hard to write and maintain manually, hence generating them automatically is essential. To that aim, analysis tools able to identify every system call that can legitimately be invoked by a program are needed. Existing static analysis works lack precision because of a high number of false positives, and/or assume the availability of program/libraries source code -- something unrealistic in many scenarios such as cloud production environments. We present B-Side, a static binary analysis tool able to identify a superset of the system calls that an x86-64 static/dynamic executable may invoke at runtime. B-Side assumes no access to program/libraries sources, and shows a good degree of precision by leveraging symbolic execution, combined with a heuristic to detect system call wrappers, which represent an important source of precision loss in existing works. B-Side also allows to statically detect phases of execution in a program in which different filtering rules can be applied. We validate B-Side and demonstrate its higher precision compared to state-of-the-art works: over a set of popular applications, B-Side's average $F_1$ score is 0.81, vs. 0.31 and 0.53 for competitors. Over 557 static and dynamically-compiled binaries taken from the Debian repositories, B-Side identifies an average of 43 system calls, vs. 271 and 95 for two state-of-the art competitors. We further evaluate the strictness of the phase-based filtering policies that can be obtained with B-Side.

Title: CLEAR: Character Unlearning in Textual and Visual Modalities

• Authors: Alexey Dontsov, Dmitrii Korzh, Alexey Zhavoronkin, Boris Mikheev, Denis Bobkov, Aibek Alanov, Oleg Y. Rogov, Ivan Oseledets, Elena Tutubalina
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18057
• Pdf URL: https://arxiv.org/pdf/2410.18057
• Copy Paste: [[2410.18057]] CLEAR: Character Unlearning in Textual and Visual Modalities(https://arxiv.org/abs/2410.18057)
• Keywords: security, privacy
• Abstract: Machine Unlearning (MU) is critical for enhancing privacy and security in deep learning models, particularly in large multimodal language models (MLLMs), by removing specific private or hazardous information. While MU has made significant progress in textual and visual modalities, multimodal unlearning (MMU) remains significantly underexplored, partially due to the absence of a suitable open-source benchmark. To address this, we introduce CLEAR, a new benchmark designed to evaluate MMU methods. CLEAR contains 200 fictitious individuals and 3,700 images linked with corresponding question-answer pairs, enabling a thorough evaluation across modalities. We assess 10 MU methods, adapting them for MMU, and highlight new challenges specific to multimodal forgetting. We also demonstrate that simple $\ell_1$ regularization on LoRA weights significantly mitigates catastrophic forgetting, preserving model performance on retained data. The dataset is available at this https URL

Title: Beyond position: how rotary embeddings shape representations and memory in autoregressive transfomers

• Authors: Valeria Ruscio, Fabrizio Silvestri
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18067
• Pdf URL: https://arxiv.org/pdf/2410.18067
• Copy Paste: [[2410.18067]] Beyond position: how rotary embeddings shape representations and memory in autoregressive transfomers(https://arxiv.org/abs/2410.18067)
• Keywords: transformer
• Abstract: Rotary Positional Embeddings (RoPE) enhance positional encoding in Transformer models, yet their full impact on model dynamics remains underexplored. This paper studies how RoPE introduces position-dependent rotations, causing phase shifts in token embeddings that influence higher-frequency components within the model's internal representations. Through spectral analysis, we demonstrate that RoPE's rotation matrices induce oscillatory behaviors in embeddings, affecting information retention across layers and shaping temporal modeling capabilities. We show that activation functions in feed-forward networks interact with RoPE-modulated embeddings to generate harmonics, leading to constructive or destructive interference based on phase alignment. Our findings reveal that phase alignment amplifies activations and sharpens attention, while misalignment weakens activations and disrupts focus on positional patterns. This study underscores the importance of frequency components as intrinsic elements of model behavior, offering new insights beyond traditional analyses.

Title: Training Free Guided Flow Matching with Optimal Control

• Authors: Luran Wang, Chaoran Cheng, Yizhen Liao, Yanru Qu, Ge Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18070
• Pdf URL: https://arxiv.org/pdf/2410.18070
• Copy Paste: [[2410.18070]] Training Free Guided Flow Matching with Optimal Control(https://arxiv.org/abs/2410.18070)
• Keywords: diffusion, generative
• Abstract: Controlled generation with pre-trained Diffusion and Flow Matching models has vast applications. One strategy for guiding ODE-based generative models is through optimizing a target loss $R(x_1)$ while staying close to the prior distribution. Along this line, some recent work showed the effectiveness of guiding flow model by differentiating through its ODE sampling process. Despite the superior performance, the theoretical understanding of this line of methods is still preliminary, leaving space for algorithm improvement. Moreover, existing methods predominately focus on Euclidean data manifold, and there is a compelling need for guided flow methods on complex geometries such as SO(3), which prevails in high-stake scientific applications like protein design. We present OC-Flow, a general and theoretically grounded training-free framework for guided flow matching using optimal control. Building upon advances in optimal control theory, we develop effective and practical algorithms for solving optimal control in guided ODE-based generation and provide a systematic theoretical analysis of the convergence guarantee in both Euclidean and SO(3). We show that existing backprop-through-ODE methods can be interpreted as special cases of Euclidean OC-Flow. OC-Flow achieved superior performance in extensive experiments on text-guided image manipulation, conditional molecule generation, and all-atom peptide design.

Title: TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing Prompts

• Authors: Yuxuan Xie, Tianhua Li, Wenqi Shao, Kaipeng Zhang
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18071
• Pdf URL: https://arxiv.org/pdf/2410.18071
• Copy Paste: [[2410.18071]] TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing Prompts(https://arxiv.org/abs/2410.18071)
• Keywords: large language model
• Abstract: Recently, multimodal large language models (MLLMs) have received much attention for their impressive capabilities. The evaluation of MLLMs is becoming critical to analyzing attributes of MLLMs and providing valuable insights. However, current benchmarks overlook the problem of prompt sensitivity - minor prompt variations may lead to significant performance fluctuations. Thus, inappropriate prompts may obscure the models' capabilities, underestimating the models' performance. Moreover, different models have different preferences for different prompts, and thus, using the same prompt for all models will cause evaluation bias. This paper analyzes this deficiency in existing benchmarks and further introduces a new evaluation framework named TP-Eval, which introduces a prompt customization method to reduce evaluation biases and tap models' potential. TP-Eval will rewrite the original prompts to different customized prompts for different models. In particular, we propose some well-designed modules for prompt customization tailored to the scenario of MLLM evaluation. Extensive experiments demonstrate the effectiveness of our approach to uncovering models' capabilities, and TP-Eval should benefit the community in developing more comprehensive and convincing MLLM evaluation benchmarks.

Title: ProFL: Performative Robust Optimal Federated Learning

• Authors: Xue Zheng, Tian Xie, Xuwei Tan, Aylin Yener, Xueru Zhang, Ali Payani, Myungjin Lee
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2410.18075
• Pdf URL: https://arxiv.org/pdf/2410.18075
• Copy Paste: [[2410.18075]] ProFL: Performative Robust Optimal Federated Learning(https://arxiv.org/abs/2410.18075)
• Keywords: robust, federate
• Abstract: Performative prediction (PP) is a framework that captures distribution shifts that occur during the training of machine learning models due to their deployment. As the trained model is used, its generated data could cause the model to evolve, leading to deviations from the original data distribution. The impact of such model-induced distribution shifts in the federated learning (FL) setup remains unexplored despite being increasingly likely to transpire in real-life use cases. Although Jin et al. (2024) recently extended PP to FL in a straightforward manner, the resulting model only converges to a performative stable point, which may be far from optimal. The methods in Izzo et al. (2021); Miller et al. (2021) can find a performative optimal point in centralized settings, but they require the performative risk to be convex and the training data to be noiseless, assumptions often violated in realistic FL systems. This paper overcomes all of these shortcomings and proposes Performative robust optimal Federated Learning (ProFL), an algorithm that finds performative optimal points in FL from noisy and contaminated data. We present the convergence analysis under the Polyak-Lojasiewicz condition, which applies to non-convex objectives. Extensive experiments on multiple datasets validate our proposed algorithms' efficiency.

Title: ALTA: Compiler-Based Analysis of Transformers

• Authors: Peter Shaw, James Cohan, Jacob Eisenstein, Kenton Lee, Jonathan Berant, Kristina Toutanova
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18077
• Pdf URL: https://arxiv.org/pdf/2410.18077
• Copy Paste: [[2410.18077]] ALTA: Compiler-Based Analysis of Transformers(https://arxiv.org/abs/2410.18077)
• Keywords: transformer
• Abstract: We propose a new programming language called ALTA and a compiler that can map ALTA programs to Transformer weights. ALTA is inspired by RASP, a language proposed by Weiss et al. (2021), and Tracr (Lindner et al., 2023), a compiler from RASP programs to Transformer weights. ALTA complements and extends this prior work, offering the ability to express loops and to compile programs to Universal Transformers, among other advantages. ALTA allows us to constructively show how Transformers can represent length-invariant algorithms for computing parity and addition, as well as a solution to the SCAN benchmark of compositional generalization tasks, without requiring intermediate scratchpad decoding steps. We also propose tools to analyze cases where the expressibility of an algorithm is established, but end-to-end training on a given training set fails to induce behavior consistent with the desired algorithm. To this end, we explore training from ALTA execution traces as a more fine-grained supervision signal. This enables additional experiments and theoretical analyses relating the learnability of various algorithms to data availability and modeling decisions, such as positional encodings. We make the ALTA framework -- language specification, symbolic interpreter, and weight compiler -- available to the community to enable further applications and insights.

Title: FreeVS: Generative View Synthesis on Free Driving Trajectory

• Authors: Qitai Wang, Lue Fan, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18079
• Pdf URL: https://arxiv.org/pdf/2410.18079
• Copy Paste: [[2410.18079]] FreeVS: Generative View Synthesis on Free Driving Trajectory(https://arxiv.org/abs/2410.18079)
• Keywords: generative
• Abstract: Existing reconstruction-based novel view synthesis methods for driving scenes focus on synthesizing camera views along the recorded trajectory of the ego vehicle. Their image rendering performance will severely degrade on viewpoints falling out of the recorded trajectory, where camera rays are untrained. We propose FreeVS, a novel fully generative approach that can synthesize camera views on free new trajectories in real driving scenes. To control the generation results to be 3D consistent with the real scenes and accurate in viewpoint pose, we propose the pseudo-image representation of view priors to control the generation process. Viewpoint transformation simulation is applied on pseudo-images to simulate camera movement in each direction. Once trained, FreeVS can be applied to any validation sequences without reconstruction process and synthesis views on novel trajectories. Moreover, we propose two new challenging benchmarks tailored to driving scenes, which are novel camera synthesis and novel trajectory synthesis, emphasizing the freedom of viewpoints. Given that no ground truth images are available on novel trajectories, we also propose to evaluate the consistency of images synthesized on novel trajectories with 3D perception models. Experiments on the Waymo Open Dataset show that FreeVS has a strong image synthesis performance on both the recorded trajectories and novel trajectories. Project Page: this https URL

Title: Prioritized Generative Replay

• Authors: Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, Alexei A. Efros
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18082
• Pdf URL: https://arxiv.org/pdf/2410.18082
• Copy Paste: [[2410.18082]] Prioritized Generative Replay(https://arxiv.org/abs/2410.18082)
• Keywords: diffusion, generative
• Abstract: Sample-efficient online reinforcement learning often uses replay buffers to store experience for reuse when updating the value function. However, uniform replay is inefficient, since certain classes of transitions can be more relevant to learning. While prioritization of more useful samples is helpful, this strategy can also lead to overfitting, as useful samples are likely to be more rare. In this work, we instead propose a prioritized, parametric version of an agent's memory, using generative models to capture online experience. This paradigm enables (1) densification of past experience, with new generations that benefit from the generative model's generalization capacity and (2) guidance via a family of "relevance functions" that push these generations towards more useful parts of an agent's acquired history. We show this recipe can be instantiated using conditional diffusion models and simple relevance functions such as curiosity- or value-based metrics. Our approach consistently improves performance and sample efficiency in both state- and pixel-based domains. We expose the mechanisms underlying these gains, showing how guidance promotes diversity in our generated transitions and reduces overfitting. We also showcase how our approach can train policies with even higher update-to-data ratios than before, opening up avenues to better scale online RL agents.

Title: DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes

• Authors: Hengwei Bian, Lingdong Kong, Haozhe Xie, Liang Pan, Yu Qiao, Ziwei Liu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.18084
• Pdf URL: https://arxiv.org/pdf/2410.18084
• Copy Paste: [[2410.18084]] DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes(https://arxiv.org/abs/2410.18084)
• Keywords: diffusion
• Abstract: LiDAR scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework capable of generating large-scale, high-quality LiDAR scenes that capture the temporal evolution of dynamic environments. DynamicCity mainly consists of two key models. 1) A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D LiDAR features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D LiDAR generation methods across multiple metrics. The code will be released to facilitate future research.