2024-09-19

Title: Optimizing Performance: How Compact Models Match or Exceed GPT's Classification Capabilities through Fine-Tuning

• Authors: Baptiste Lefort, Eric Benhamou, Jean-Jacques Ohana, David Saltiel, Beatrice Guez
• Subjects: cs.CL, q-fin.ST
• Abstract URL: https://arxiv.org/abs/2409.11408
• Pdf URL: https://arxiv.org/pdf/2409.11408
• Copy Paste: [[2409.11408]] Optimizing Performance: How Compact Models Match or Exceed GPT's Classification Capabilities through Fine-Tuning(https://arxiv.org/abs/2409.11408)
• Keywords: generative
• Abstract: In this paper, we demonstrate that non-generative, small-sized models such as FinBERT and FinDRoBERTa, when fine-tuned, can outperform GPT-3.5 and GPT-4 models in zero-shot learning settings in sentiment analysis for financial news. These fine-tuned models show comparable results to GPT-3.5 when it is fine-tuned on the task of determining market sentiment from daily financial news summaries sourced from Bloomberg. To fine-tune and compare these models, we created a novel database, which assigns a market score to each piece of news without human interpretation bias, systematically identifying the mentioned companies and analyzing whether their stocks have gone up, down, or remained neutral. Furthermore, the paper shows that the assumptions of Condorcet's Jury Theorem do not hold suggesting that fine-tuned small models are not independent of the fine-tuned GPT models, indicating behavioural similarities. Lastly, the resulted fine-tuned models are made publicly available on HuggingFace, providing a resource for further research in financial sentiment analysis and text classification.

Title: CyberNFTs: Conceptualizing a decentralized and reward-driven intrusion detection system with ML

• Authors: Synim Selimi, Blerim Rexha, Kamer Vishi
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11409
• Pdf URL: https://arxiv.org/pdf/2409.11409
• Copy Paste: [[2409.11409]] CyberNFTs: Conceptualizing a decentralized and reward-driven intrusion detection system with ML(https://arxiv.org/abs/2409.11409)
• Keywords: security
• Abstract: The rapid evolution of the Internet, particularly the emergence of Web3, has transformed the ways people interact and share data. Web3, although still not well defined, is thought to be a return to the decentralization of corporations' power over user data. Despite the obsolescence of the idea of building systems to detect and prevent cyber intrusions, this is still a topic of interest. This paper proposes a novel conceptual approach for implementing decentralized collaborative intrusion detection networks (CIDN) through a proof-of-concept. The study employs an analytical and comparative methodology, examining the synergy between cutting-edge Web3 technologies and information security. The proposed model incorporates blockchain concepts, cyber non-fungible token (cyberNFT) rewards, machine learning algorithms, and publish/subscribe architectures. Finally, the paper discusses the strengths and limitations of the proposed system, offering insights into the potential of decentralized cybersecurity models.

Title: Securing Network-Booting Linux Systems at the Example of bwLehrpool and bwForCluster NEMO

• Authors: Simon Moser
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11413
• Pdf URL: https://arxiv.org/pdf/2409.11413
• Copy Paste: [[2409.11413]] Securing Network-Booting Linux Systems at the Example of bwLehrpool and bwForCluster NEMO(https://arxiv.org/abs/2409.11413)
• Keywords: secure, security
• Abstract: The universities of Baden-Württemberg are using stateless system remote boot for services such as computer labs and data centers. It involves loading a host system over the network and allowing users to start various virtual machines. The filesystem is provided over a distributed network block device (dnbd3) mounted read-only. The process raises security concerns due to potentially untrusted networks. The aim of this work is to establish trust within this network, focusing on server-client identity, confidentiality and image authenticity. Using Secure Boot and iPXE signing, the integrity can be guaranteed for the complete boot process. The necessary effort to implement it is mainly one time at the set-up of the server, while the changes necessary once to the client could be done over the network. Afterwards, no significant delay was measured in the boot process for the main technologies, while the technique of integrating the kernel with other files resulted in a small delay measured. TPM can be used to ensure the client's identity and confidentiality. Provisioning TPM is a bigger challenge because as a trade-off has to be made between the inconvenience of using a secure medium and the ease of using an insecure channel once. Additionally, in the data center use case, hardware with TPM might have higher costs, while the additional security gained by changing from the current key storage is only little. After the provisioning is completed, the TPM can then be used to decrypt information with a securely stored key.

Title: A Comprehensive Survey of Advanced Persistent Threat Attribution: Taxonomy, Methods, Challenges and Open Research Problems

• Authors: Nanda Rani, Bikash Saha, Sandeep Kumar Shukla
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11415
• Pdf URL: https://arxiv.org/pdf/2409.11415
• Copy Paste: [[2409.11415]] A Comprehensive Survey of Advanced Persistent Threat Attribution: Taxonomy, Methods, Challenges and Open Research Problems(https://arxiv.org/abs/2409.11415)
• Keywords: security, defense, attack
• Abstract: Advanced Persistent Threat (APT) attribution is a critical challenge in cybersecurity and implies the process of accurately identifying the perpetrators behind sophisticated cyber attacks. It can significantly enhance defense mechanisms and inform strategic responses. With the growing prominence of artificial intelligence (AI) and machine learning (ML) techniques, researchers are increasingly focused on developing automated solutions to link cyber threats to responsible actors, moving away from traditional manual methods. Previous literature on automated threat attribution lacks a systematic review of automated methods and relevant artifacts that can aid in the attribution process. To address these gaps and provide context on the current state of threat attribution, we present a comprehensive survey of automated APT attribution. The presented survey starts with understanding the dispersed artifacts and provides a comprehensive taxonomy of the artifacts that aid in attribution. We comprehensively review and present the classification of the available attribution datasets and current automated APT attribution methods. Further, we raise critical comments on current literature methods, discuss challenges in automated attribution, and direct toward open research problems. This survey reveals significant opportunities for future research in APT attribution to address current gaps and challenges. By identifying strengths and limitations in current practices, this survey provides a foundation for future research and development in automated, reliable, and actionable APT attribution methods.

Title: Maritime Cybersecurity: A Comprehensive Review

• Authors: Meixuan Li, Jianying Zhou, Sudipta Chattopadhyay, Mark Goh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11417
• Pdf URL: https://arxiv.org/pdf/2409.11417
• Copy Paste: [[2409.11417]] Maritime Cybersecurity: A Comprehensive Review(https://arxiv.org/abs/2409.11417)
• Keywords: security, protect, attack, robust
• Abstract: The maritime industry stands at a critical juncture, where the imperative for technological advancement intersects with the pressing need for robust cybersecurity measures. Maritime cybersecurity refers to the protection of computer systems and digital assests within the maritime industry, as well as the broader network of interconnected components that make up the maritime ecosystem. In this survey, we aim to identify the significant domains of maritime cybersecurity and measure their effectiveness. We have provided an in-depth analysis of threats in key maritime systems, including AIS, GNSS, ECDIS, VDR, RADAR, VSAT, and GMDSS, while exploring real-world cyber incidents that have impacted the sector. A multi-dimensional taxonomy of maritime cyber attacks is presented, offering insights into threat actors, motivations, and impacts. We have also evaluated various security solutions, from integrated solutions to component specific solutions. Finally, we have shared open challenges and future solutions. In the supplementary section, we have presented definitions and vulnerabilities of vessel components that have discussed in this survey. By addressing all these critical issues with key interconnected aspects, this review aims to foster a more resilient maritime ecosystem.

Title: Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data

• Authors: Atilla Akkus, Mingjie Li, Junjie Chu, Michael Backes, Yang Zhang, Sinem Sav
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11423
• Pdf URL: https://arxiv.org/pdf/2409.11423
• Copy Paste: [[2409.11423]] Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data(https://arxiv.org/abs/2409.11423)
• Keywords: privacy, attack, extraction, membership infer, large language model
• Abstract: Large language models (LLMs) have shown considerable success in a range of domain-specific tasks, especially after fine-tuning. However, fine-tuning with real-world data usually leads to privacy risks, particularly when the fine-tuning samples exist in the pre-training data. To avoid the shortcomings of real data, developers often employ methods to automatically generate synthetic data for fine-tuning, as data generated by traditional models are often far away from the real-world pertaining data. However, given the advanced capabilities of LLMs, the distinction between real data and LLM-generated data has become negligible, which may also lead to privacy risks like real data. In this paper, we present an empirical analysis of this underexplored issue by investigating a key question: "Does fine-tuning with LLM-generated data enhance privacy, or does it pose additional privacy risks?" Based on the structure of LLM's generated data, our research focuses on two primary approaches to fine-tuning with generated data: supervised fine-tuning with unstructured generated data and self-instruct tuning. The number of successful Personal Information Identifier (PII) extractions for Pythia after fine-tuning our generated data raised over $20\%$. Furthermore, the ROC-AUC score of membership inference attacks for Pythia-6.9b after self-instruct methods also achieves more than $40\%$ improvements on ROC-AUC score than base models. The results indicate the potential privacy risks in LLMs when fine-tuning with the generated data.

Title: A Comprehensive Analysis of Machine Learning Based File Trap Selection Methods to Detect Crypto Ransomware

• Authors: Mohan Anand Putrevu, Hrushikesh Chunduri, Venkata Sai Charan Putrevu, Sandeep K Shukla
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11428
• Pdf URL: https://arxiv.org/pdf/2409.11428
• Copy Paste: [[2409.11428]] A Comprehensive Analysis of Machine Learning Based File Trap Selection Methods to Detect Crypto Ransomware(https://arxiv.org/abs/2409.11428)
• Keywords: attack
• Abstract: The use of multi-threading and file prioritization methods has accelerated the speed at which ransomware encrypts files. To minimize file loss during the ransomware attack, detecting file modifications at the earliest execution stage is considered very important. To achieve this, selecting files as traps and monitoring changes to them is a practical way to deal with modern ransomware variants. This approach minimizes overhead on the endpoint, facilitating early identification of ransomware. This paper evaluates various machine learning-based trap selection methods for reducing file loss, detection delay, and endpoint overhead. We specifically examine non-parametric clustering methods such as Affinity Propagation, Gaussian Mixture Models, Mean Shift, and Optics to assess their effectiveness in trap selection for ransomware detection. These methods select M files from a directory with N files (M

Title: Continual Learning of Conjugated Visual Representations through Higher-order Motion Flows

• Authors: Simone Marullo, Matteo Tiezzi, Marco Gori, Stefano Melacci
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11441
• Pdf URL: https://arxiv.org/pdf/2409.11441
• Copy Paste: [[2409.11441]] Continual Learning of Conjugated Visual Representations through Higher-order Motion Flows(https://arxiv.org/abs/2409.11441)
• Keywords: transformer
• Abstract: Learning with neural networks from a continuous stream of visual information presents several challenges due to the non-i.i.d. nature of the data. However, it also offers novel opportunities to develop representations that are consistent with the information flow. In this paper we investigate the case of unsupervised continual learning of pixel-wise features subject to multiple motion-induced constraints, therefore named motion-conjugated feature representations. Differently from existing approaches, motion is not a given signal (either ground-truth or estimated by external modules), but is the outcome of a progressive and autonomous learning process, occurring at various levels of the feature hierarchy. Multiple motion flows are estimated with neural networks and characterized by different levels of abstractions, spanning from traditional optical flow to other latent signals originating from higher-level features, hence called higher-order motions. Continuously learning to develop consistent multi-order flows and representations is prone to trivial solutions, which we counteract by introducing a self-supervised contrastive loss, spatially-aware and based on flow-induced similarity. We assess our model on photorealistic synthetic streams and real-world videos, comparing to pre-trained state-of-the art feature extractors (also based on Transformers) and to recent unsupervised learning models, significantly outperforming these alternatives.

Title: A Green Multi-Attribute Client Selection for Over-The-Air Federated Learning: A Grey-Wolf-Optimizer Approach

• Authors: Maryam Ben Driss, Essaid Sabir, Halima Elbiaze, Abdoulaye Baniré Diallo, Mohamed Sadik
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.11442
• Pdf URL: https://arxiv.org/pdf/2409.11442
• Copy Paste: [[2409.11442]] A Green Multi-Attribute Client Selection for Over-The-Air Federated Learning: A Grey-Wolf-Optimizer Approach(https://arxiv.org/abs/2409.11442)
• Keywords: privacy, federate, fair
• Abstract: Federated Learning (FL) has gained attention across various industries for its capability to train machine learning models without centralizing sensitive data. While this approach offers significant benefits such as privacy preservation and decreased communication overhead, it presents several challenges, including deployment complexity and interoperability issues, particularly in heterogeneous scenarios or resource-constrained environments. Over-the-air (OTA) FL was introduced to tackle these challenges by disseminating model updates without necessitating direct device-to-device connections or centralized servers. However, OTA-FL brought forth limitations associated with heightened energy consumption and network latency. In this paper, we propose a multi-attribute client selection framework employing the grey wolf optimizer (GWO) to strategically control the number of participants in each round and optimize the OTA-FL process while considering accuracy, energy, delay, reliability, and fairness constraints of participating devices. We evaluate the performance of our multi-attribute client selection approach in terms of model loss minimization, convergence time reduction, and energy efficiency. In our experimental evaluation, we assessed and compared the performance of our approach against the existing state-of-the-art methods. Our results demonstrate that the proposed GWO-based client selection outperforms these baselines across various metrics. Specifically, our approach achieves a notable reduction in model loss, accelerates convergence time, and enhances energy efficiency while maintaining high fairness and reliability indicators.

Title: Jailbreaking Large Language Models with Symbolic Mathematics

• Authors: Emet Bethany, Mazal Bethany, Juan Arturo Nolazco Flores, Sumit Kumar Jha, Peyman Najafirad
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11445
• Pdf URL: https://arxiv.org/pdf/2409.11445
• Copy Paste: [[2409.11445]] Jailbreaking Large Language Models with Symbolic Mathematics(https://arxiv.org/abs/2409.11445)
• Keywords: attack, robust, large language model
• Abstract: Recent advancements in AI safety have led to increased efforts in training and red-teaming large language models (LLMs) to mitigate unsafe content generation. However, these safety mechanisms may not be comprehensive, leaving potential vulnerabilities unexplored. This paper introduces MathPrompt, a novel jailbreaking technique that exploits LLMs' advanced capabilities in symbolic mathematics to bypass their safety mechanisms. By encoding harmful natural language prompts into mathematical problems, we demonstrate a critical vulnerability in current AI safety measures. Our experiments across 13 state-of-the-art LLMs reveal an average attack success rate of 73.6\%, highlighting the inability of existing safety training mechanisms to generalize to mathematically encoded inputs. Analysis of embedding vectors shows a substantial semantic shift between original and encoded prompts, helping explain the attack's success. This work emphasizes the importance of a holistic approach to AI safety, calling for expanded red-teaming efforts to develop robust safeguards across all potential input types and their associated risks.

Title: Evaluation of pretrained language models on music understanding

• Authors: Yannis Vasilakis, Rachel Bittner, Johan Pauwels
• Subjects: cs.LG, cs.AI, cs.IR, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.11449
• Pdf URL: https://arxiv.org/pdf/2409.11449
• Copy Paste: [[2409.11449]] Evaluation of pretrained language models on music understanding(https://arxiv.org/abs/2409.11449)
• Keywords: transformer, large language model
• Abstract: Music-text multimodal systems have enabled new approaches to Music Information Research (MIR) applications such as audio-to-text and text-to-audio retrieval, text-based song generation, and music captioning. Despite the reported success, little effort has been put into evaluating the musical knowledge of Large Language Models (LLM). In this paper, we demonstrate that LLMs suffer from 1) prompt sensitivity, 2) inability to model negation (e.g. 'rock song without guitar'), and 3) sensitivity towards the presence of specific words. We quantified these properties as a triplet-based accuracy, evaluating the ability to model the relative similarity of labels in a hierarchical ontology. We leveraged the Audioset ontology to generate triplets consisting of an anchor, a positive (relevant) label, and a negative (less relevant) label for the genre and instruments sub-tree. We evaluated the triplet-based musical knowledge for six general-purpose Transformer-based models. The triplets obtained through this methodology required filtering, as some were difficult to judge and therefore relatively uninformative for evaluation purposes. Despite the relatively high accuracy reported, inconsistencies are evident in all six models, suggesting that off-the-shelf LLMs need adaptation to music before use.

Title: Golden Ratio Search: A Low-Power Adversarial Attack for Deep Learning based Modulation Classification

• Authors: Deepsayan Sadhukhan, Nitin Priyadarshini Shankar, Sheetal Kalyani
• Subjects: cs.CR, cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.11454
• Pdf URL: https://arxiv.org/pdf/2409.11454
• Copy Paste: [[2409.11454]] Golden Ratio Search: A Low-Power Adversarial Attack for Deep Learning based Modulation Classification(https://arxiv.org/abs/2409.11454)
• Keywords: defense, attack, robust
• Abstract: We propose a minimal power white box adversarial attack for Deep Learning based Automatic Modulation Classification (AMC). The proposed attack uses the Golden Ratio Search (GRS) method to find powerful attacks with minimal power. We evaluate the efficacy of the proposed method by comparing it with existing adversarial attack approaches. Additionally, we test the robustness of the proposed attack against various state-of-the-art architectures, including defense mechanisms such as adversarial training, binarization, and ensemble methods. Experimental results demonstrate that the proposed attack is powerful, requires minimal power, and can be generated in less time, significantly challenging the resilience of current AMC methods.

Title: Two Stage Segmentation of Cervical Tumors using PocketNet

• Authors: Awj Twam, Megan Jacobsen, Rachel Glenn, Ann Klopp, Aradhana M. Venkatesan, David Fuentes
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11456
• Pdf URL: https://arxiv.org/pdf/2409.11456
• Copy Paste: [[2409.11456]] Two Stage Segmentation of Cervical Tumors using PocketNet(https://arxiv.org/abs/2409.11456)
• Keywords: robust, segmentation
• Abstract: Cervical cancer remains the fourth most common malignancy amongst women worldwide.1 Concurrent chemoradiotherapy (CRT) serves as the mainstay definitive treatment regimen for locally advanced cervical cancers and includes external beam radiation followed by brachytherapy.2 Integral to radiotherapy treatment planning is the routine contouring of both the target tumor at the level of the cervix, associated gynecologic anatomy and the adjacent organs at risk (OARs). However, manual contouring of these structures is both time and labor intensive and associated with known interobserver variability that can impact treatment outcomes. While multiple tools have been developed to automatically segment OARs and the high-risk clinical tumor volume (HR-CTV) using computed tomography (CT) images,3,4,5,6 the development of deep learning-based tumor segmentation tools using routine T2-weighted (T2w) magnetic resonance imaging (MRI) addresses an unmet clinical need to improve the routine contouring of both anatomical structures and cervical cancers, thereby increasing quality and consistency of radiotherapy planning. This work applied a novel deep-learning model (PocketNet) to segment the cervix, vagina, uterus, and tumor(s) on T2w MRI. The performance of the PocketNet architecture was evaluated, when trained on data via 5-fold cross validation. PocketNet achieved a mean Dice-Sorensen similarity coefficient (DSC) exceeding 70% for tumor segmentation and 80% for organ segmentation. These results suggest that PocketNet is robust to variations in contrast protocols, providing reliable segmentation of the ROIs.

Title: Enriching Datasets with Demographics through Large Language Models: What's in a Name?

• Authors: Khaled AlNuaimi, Gautier Marti, Mathieu Ravaut, Abdulla AlKetbi, Andreas Henschel, Raed Jaradat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11491
• Pdf URL: https://arxiv.org/pdf/2409.11491
• Copy Paste: [[2409.11491]] Enriching Datasets with Demographics through Large Language Models: What's in a Name?(https://arxiv.org/abs/2409.11491)
• Keywords: robust, large language model
• Abstract: Enriching datasets with demographic information, such as gender, race, and age from names, is a critical task in fields like healthcare, public policy, and social sciences. Such demographic insights allow for more precise and effective engagement with target populations. Despite previous efforts employing hidden Markov models and recurrent neural networks to predict demographics from names, significant limitations persist: the lack of large-scale, well-curated, unbiased, publicly available datasets, and the lack of an approach robust across datasets. This scarcity has hindered the development of traditional supervised learning approaches. In this paper, we demonstrate that the zero-shot capabilities of Large Language Models (LLMs) can perform as well as, if not better than, bespoke models trained on specialized data. We apply these LLMs to a variety of datasets, including a real-life, unlabelled dataset of licensed financial professionals in Hong Kong, and critically assess the inherent demographic biases in these models. Our work not only advances the state-of-the-art in demographic enrichment but also opens avenues for future research in mitigating biases in LLMs.

Title: Egalitarian Language Representation in Language Models: It All Begins with Tokenizers

• Authors: Menan Velayuthan, Kengatharaiyer Sarveswaran
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11501
• Pdf URL: https://arxiv.org/pdf/2409.11501
• Copy Paste: [[2409.11501]] Egalitarian Language Representation in Language Models: It All Begins with Tokenizers(https://arxiv.org/abs/2409.11501)
• Keywords: extraction, fair, large language model
• Abstract: Tokenizers act as a bridge between human language and the latent space of language models, influencing how language is represented in these models. Due to the immense popularity of English-Centric Large Language Models (LLMs), efforts are being made to adapt them for other languages. However, we demonstrate that, from a tokenization standpoint, not all tokenizers offer fair representation for complex script languages such as Tamil, Sinhala, and Hindi, primarily due to the choice of pre-tokenization methods. We go further to show that pre-tokenization plays a more critical role than the tokenization algorithm itself in achieving an egalitarian representation of these complex script languages. To address this, we introduce an improvement to the Byte Pair Encoding (BPE) algorithm by incorporating graphemes, which we term Grapheme Pair Encoding (GPE). Our experiments show that grapheme-based character extraction outperforms byte-level tokenizers for complex scripts. We validate this approach through experiments on Tamil, Sinhala, and Hindi.

Title: FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction

• Authors: Ziwei Li, Xiaoqi Wang, Hong-You Chen, Han-Wei Shen, Wei-Lun Chao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11509
• Pdf URL: https://arxiv.org/pdf/2409.11509
• Copy Paste: [[2409.11509]] FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction(https://arxiv.org/abs/2409.11509)
• Keywords: federate
• Abstract: Federated learning (FL) has rapidly evolved as a promising paradigm that enables collaborative model training across distributed participants without exchanging their local data. Despite its broad applications in fields such as computer vision, graph learning, and natural language processing, the development of a data projection model that can be effectively used to visualize data in the context of FL is crucial yet remains heavily under-explored. Neighbor embedding (NE) is an essential technique for visualizing complex high-dimensional data, but collaboratively learning a joint NE model is difficult. The key challenge lies in the objective function, as effective visualization algorithms like NE require computing loss functions among pairs of data. In this paper, we introduce \textsc{FedNE}, a novel approach that integrates the \textsc{FedAvg} framework with the contrastive NE technique, without any requirements of shareable data. To address the lack of inter-client repulsion which is crucial for the alignment in the global embedding space, we develop a surrogate loss function that each client learns and shares with each other. Additionally, we propose a data-mixing strategy to augment the local data, aiming to relax the problems of invisible neighbors and false neighbors constructed by the local $k$NN graphs. We conduct comprehensive experiments on both synthetic and real-world datasets. The results demonstrate that our \textsc{FedNE} can effectively preserve the neighborhood data structures and enhance the alignment in the global embedding space compared to several baseline methods.

Title: Mamba Fusion: Learning Actions Through Questioning

• Authors: Zhikang Dong, Apoorva Beedu, Jason Sheinkopf, Irfan Essa
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11513
• Pdf URL: https://arxiv.org/pdf/2409.11513
• Copy Paste: [[2409.11513]] Mamba Fusion: Learning Actions Through Questioning(https://arxiv.org/abs/2409.11513)
• Keywords: transformer
• Abstract: Video Language Models (VLMs) are crucial for generalizing across diverse tasks and using language cues to enhance learning. While transformer-based architectures have been the de facto in vision-language training, they face challenges like quadratic computational complexity, high GPU memory usage, and difficulty with long-term dependencies. To address these limitations, we introduce MambaVL, a novel model that leverages recent advancements in selective state space modality fusion to efficiently capture long-range dependencies and learn joint representations for vision and language data. MambaVL utilizes a shared state transition matrix across both modalities, allowing the model to capture information about actions from multiple perspectives within the scene. Furthermore, we propose a question-answering task that helps guide the model toward relevant cues. These questions provide critical information about actions, objects, and environmental context, leading to enhanced performance. As a result, MambaVL achieves state-of-the-art performance in action recognition on the Epic-Kitchens-100 dataset and outperforms baseline methods in action anticipation.

Title: Adaptive Anomaly Detection in Network Flows with Low-Rank Tensor Decompositions and Deep Unrolling

• Authors: Lukas Schynol, Marius Pesavento
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.11529
• Pdf URL: https://arxiv.org/pdf/2409.11529
• Copy Paste: [[2409.11529]] Adaptive Anomaly Detection in Network Flows with Low-Rank Tensor Decompositions and Deep Unrolling(https://arxiv.org/abs/2409.11529)
• Keywords: robust, interpretability
• Abstract: Anomaly detection (AD) is increasingly recognized as a key component for ensuring the resilience of future communication systems. While deep learning has shown state-of-the-art AD performance, its application in critical systems is hindered by concerns regarding training data efficiency, domain adaptation and interpretability. This work considers AD in network flows using incomplete measurements, leveraging a robust tensor decomposition approach and deep unrolling techniques to address these challenges. We first propose a novel block-successive convex approximation algorithm based on a regularized model-fitting objective where the normal flows are modeled as low-rank tensors and anomalies as sparse. An augmentation of the objective is introduced to decrease the computational cost. We apply deep unrolling to derive a novel deep network architecture based on our proposed algorithm, treating the regularization parameters as learnable weights. Inspired by Bayesian approaches, we extend the model architecture to perform online adaptation to per-flow and per-time-step statistics, improving AD performance while maintaining a low parameter count and preserving the problem's permutation equivariances. To optimize the deep network weights for detection performance, we employ a homotopy optimization approach based on an efficient approximation of the area under the receiver operating characteristic curve. Extensive experiments on synthetic and real-world data demonstrate that our proposed deep network architecture exhibits a high training data efficiency, outperforms reference methods, and adapts seamlessly to varying network topologies.

Title: Balancing Optimality and Diversity: Human-Centered Decision Making through Generative Curation

• Authors: Michael Lingzhi Li, Shixiang Zhu
• Subjects: cs.LG, cs.HC, math.OC
• Abstract URL: https://arxiv.org/abs/2409.11535
• Pdf URL: https://arxiv.org/pdf/2409.11535
• Copy Paste: [[2409.11535]] Balancing Optimality and Diversity: Human-Centered Decision Making through Generative Curation(https://arxiv.org/abs/2409.11535)
• Keywords: robust, generative
• Abstract: The surge in data availability has inundated decision-makers with an overwhelming array of choices. While existing approaches focus on optimizing decisions based on quantifiable metrics, practical decision-making often requires balancing measurable quantitative criteria with unmeasurable qualitative factors embedded in the broader context. In such cases, algorithms can generate high-quality recommendations, but the final decision rests with the human, who must weigh both dimensions. We define the process of selecting the optimal set of algorithmic recommendations in this context as human-centered decision making. To address this challenge, we introduce a novel framework called generative curation, which optimizes the true desirability of decision options by integrating both quantitative and qualitative aspects. Our framework uses a Gaussian process to model unknown qualitative factors and derives a diversity metric that balances quantitative optimality with qualitative diversity. This trade-off enables the generation of a manageable subset of diverse, near-optimal actions that are robust to unknown qualitative preferences. To operationalize this framework, we propose two implementation approaches: a generative neural network architecture that produces a distribution $\pi$ to efficiently sample a diverse set of near-optimal actions, and a sequential optimization method to iteratively generates solutions that can be easily incorporated into complex optimization formulations. We validate our approach with extensive datasets, demonstrating its effectiveness in enhancing decision-making processes across a range of complex environments, with significant implications for policy and management.

Title: Obfuscation Based Privacy Preserving Representations are Recoverable Using Neighborhood Information

• Authors: Kunal Chelani, Assia Benbihi, Fredrik Kahl, Torsten Sattler, Zuzana Kukelova
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11536
• Pdf URL: https://arxiv.org/pdf/2409.11536
• Copy Paste: [[2409.11536]] Obfuscation Based Privacy Preserving Representations are Recoverable Using Neighborhood Information(https://arxiv.org/abs/2409.11536)
• Keywords: privacy, attack
• Abstract: Rapid growth in the popularity of AR/VR/MR applications and cloud-based visual localization systems has given rise to an increased focus on the privacy of user content in the localization process. This privacy concern has been further escalated by the ability of deep neural networks to recover detailed images of a scene from a sparse set of 3D or 2D points and their descriptors - the so-called inversion attacks. Research on privacy-preserving localization has therefore focused on preventing these inversion attacks on both the query image keypoints and the 3D points of the scene map. To this end, several geometry obfuscation techniques that lift points to higher-dimensional spaces, i.e., lines or planes, or that swap coordinates between points % have been proposed. In this paper, we point to a common weakness of these obfuscations that allows to recover approximations of the original point positions under the assumption of known neighborhoods. We further show that these neighborhoods can be computed by learning to identify descriptors that co-occur in neighborhoods. Extensive experiments show that our approach for point recovery is practically applicable to all existing geometric obfuscation schemes. Our results show that these schemes should not be considered privacy-preserving, even though they are claimed to be privacy-preserving. Code will be available at \url{this https URL}.

Title: Chain-of-Thought Prompting for Speech Translation

• Authors: Ke Hu, Zhehuai Chen, Chao-Han Huck Yang, Piotr Żelasko, Oleksii Hrinchuk, Vitaly Lavrukhin, Jagadeesh Balam, Boris Ginsburg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11538
• Pdf URL: https://arxiv.org/pdf/2409.11538
• Copy Paste: [[2409.11538]] Chain-of-Thought Prompting for Speech Translation(https://arxiv.org/abs/2409.11538)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable advancements in language understanding and generation. Building on the success of text-based LLMs, recent research has adapted these models to use speech embeddings for prompting, resulting in Speech-LLM models that exhibit strong performance in automatic speech recognition (ASR) and automatic speech translation (AST). In this work, we propose a novel approach to leverage ASR transcripts as prompts for AST in a Speech-LLM built on an encoder-decoder text LLM. The Speech-LLM model consists of a speech encoder and an encoder-decoder structure Megatron-T5. By first decoding speech to generate ASR transcripts and subsequently using these transcripts along with encoded speech for prompting, we guide the speech translation in a two-step process like chain-of-thought (CoT) prompting. Low-rank adaptation (LoRA) is used for the T5 LLM for model adaptation and shows superior performance to full model fine-tuning. Experimental results show that the proposed CoT prompting significantly improves AST performance, achieving an average increase of 2.4 BLEU points across 6 En->X or X->En AST tasks compared to speech prompting alone. Additionally, compared to a related CoT prediction method that predicts a concatenated sequence of ASR and AST transcripts, our method performs better by an average of 2 BLEU points.

Title: Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMs

• Authors: Guillermo Marco, Luz Rello, Julio Gonzalo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11547
• Pdf URL: https://arxiv.org/pdf/2409.11547
• Copy Paste: [[2409.11547]] Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMs(https://arxiv.org/abs/2409.11547)
• Keywords: large language model
• Abstract: In this paper, we evaluate the creative fiction writing abilities of a fine-tuned small language model (SLM), BART Large, and compare its performance to humans and two large language models (LLMs): GPT-3.5 and GPT-4o. Our evaluation consists of two experiments: (i) a human evaluation where readers assess the stories generated by the SLM compared to human-written stories, and (ii) a qualitative linguistic analysis comparing the textual characteristics of the stories generated by the different models. In the first experiment, we asked 68 participants to rate short stories generated by the models and humans along dimensions such as grammaticality, relevance, creativity, and attractiveness. BART Large outperformed human writers in most aspects, except creativity, with an overall score of 2.11 compared to 1.85 for human-written texts -- a 14% improvement. In the second experiment, the qualitative analysis revealed that, while GPT-4o exhibited near-perfect internal and external coherence, it tended to produce more predictable narratives, with only 3% of its stories seen as novel. In contrast, 15% of BART's stories were considered novel, indicating a higher degree of creativity despite its smaller model size. This study provides both quantitative and qualitative insights into how model size and fine-tuning influence the balance between creativity, fluency, and coherence in creative writing tasks.

Title: A Property Encoder for Graph Neural Networks

• Authors: Anwar Said, Xenofon Koutsoukos
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.11554
• Pdf URL: https://arxiv.org/pdf/2409.11554
• Copy Paste: [[2409.11554]] A Property Encoder for Graph Neural Networks(https://arxiv.org/abs/2409.11554)
• Keywords: privacy
• Abstract: Graph machine learning, particularly using graph neural networks, fundamentally relies on node features. Nevertheless, numerous real-world systems, such as social and biological networks, often lack node features due to various reasons, including privacy concerns, incomplete or missing data, and limitations in data collection. In such scenarios, researchers typically resort to methods like structural and positional encoding to construct node features. However, the length of such features is contingent on the maximum value within the property being encoded, for example, the highest node degree, which can be exceedingly large in applications like scale-free networks. Furthermore, these encoding schemes are limited to categorical data and might not be able to encode metrics returning other type of values. In this paper, we introduce a novel, universally applicable encoder, termed PropEnc, which constructs expressive node embedding from any given graph metric. PropEnc leverages histogram construction combined with reverse index encoding, offering a flexible method for node features initialization. It supports flexible encoding in terms of both dimensionality and type of input, demonstrating its effectiveness across diverse applications. PropEnc allows encoding metrics in low-dimensional space which effectively avoids the issue of sparsity and enhances the efficiency of the models. We show that \emph{PropEnc} can construct node features that either exactly replicate one-hot encoding or closely approximate indices under various settings. Our extensive evaluations in graph classification setting across multiple social networks that lack node features support our hypothesis. The empirical results conclusively demonstrate that PropEnc is both an efficient and effective mechanism for constructing node features from diverse set of graph metrics.

Title: Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey

• Authors: Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D. Yao, Shi-Xiong Zhang, Sambit Sahu
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.11564
• Pdf URL: https://arxiv.org/pdf/2409.11564
• Copy Paste: [[2409.11564]] Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey(https://arxiv.org/abs/2409.11564)
• Keywords: generative
• Abstract: Preference tuning is a crucial process for aligning deep generative models with human preferences. This survey offers a thorough overview of recent advancements in preference tuning and the integration of human feedback. The paper is organized into three main sections: 1) introduction and preliminaries: an introduction to reinforcement learning frameworks, preference tuning tasks, models, and datasets across various modalities: language, speech, and vision, as well as different policy approaches, 2) in-depth examination of each preference tuning approach: a detailed analysis of the methods used in preference tuning, and 3) applications, discussion, and future directions: an exploration of the applications of preference tuning in downstream tasks, including evaluation methods for different modalities, and an outlook on future research directions. Our objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners. We hope to encourage further engagement and innovation in this area.

Title: HEARTS: A Holistic Framework for Explainable, Sustainable and Robust Text Stereotype Detection

• Authors: Theo King, Zekun Wu, Adriano Koshiyama, Emre Kazim, Philip Treleaven
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11579
• Pdf URL: https://arxiv.org/pdf/2409.11579
• Copy Paste: [[2409.11579]] HEARTS: A Holistic Framework for Explainable, Sustainable and Robust Text Stereotype Detection(https://arxiv.org/abs/2409.11579)
• Keywords: robust, explainability
• Abstract: Stereotypes are generalised assumptions about societal groups, and even state-of-the-art LLMs using in-context learning struggle to identify them accurately. Due to the subjective nature of stereotypes, where what constitutes a stereotype can vary widely depending on cultural, social, and individual perspectives, robust explainability is crucial. Explainable models ensure that these nuanced judgments can be understood and validated by human users, promoting trust and accountability. We address these challenges by introducing HEARTS (Holistic Framework for Explainable, Sustainable, and Robust Text Stereotype Detection), a framework that enhances model performance, minimises carbon footprint, and provides transparent, interpretable explanations. We establish the Expanded Multi-Grain Stereotype Dataset (EMGSD), comprising 57,201 labeled texts across six groups, including under-represented demographics like LGBTQ+ and regional stereotypes. Ablation studies confirm that BERT models fine-tuned on EMGSD outperform those trained on individual components. We then analyse a fine-tuned, carbon-efficient ALBERT-V2 model using SHAP to generate token-level importance values, ensuring alignment with human understanding, and calculate explainability confidence scores by comparing SHAP and LIME outputs. Finally, HEARTS is applied to assess stereotypical bias in 12 LLM outputs, revealing a gradual reduction in bias over time within model families.

Title: Advances in APPFL: A Comprehensive and Extensible Federated Learning Framework

• Authors: Zilinghan Li, Shilan He, Ze Yang, Minseok Ryu, Kibaek Kim, Ravi Madduri
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.11585
• Pdf URL: https://arxiv.org/pdf/2409.11585
• Copy Paste: [[2409.11585]] Advances in APPFL: A Comprehensive and Extensible Federated Learning Framework(https://arxiv.org/abs/2409.11585)
• Keywords: security, privacy, federate
• Abstract: Federated learning (FL) is a distributed machine learning paradigm enabling collaborative model training while preserving data privacy. In today's landscape, where most data is proprietary, confidential, and distributed, FL has become a promising approach to leverage such data effectively, particularly in sensitive domains such as medicine and the electric grid. Heterogeneity and security are the key challenges in FL, however; most existing FL frameworks either fail to address these challenges adequately or lack the flexibility to incorporate new solutions. To this end, we present the recent advances in developing APPFL, an extensible framework and benchmarking suite for federated learning, which offers comprehensive solutions for heterogeneity and security concerns, as well as user-friendly interfaces for integrating new algorithms or adapting to new applications. We demonstrate the capabilities of APPFL through extensive experiments evaluating various aspects of FL, including communication efficiency, privacy preservation, computational performance, and resource utilization. We further highlight the extensibility of APPFL through case studies in vertical, hierarchical, and decentralized FL. APPFL is open-sourced at this https URL.

Title: ProSLM : A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering

• Authors: Priyesh Vakharia, Abigail Kufeldt, Max Meyers, Ian Lane, Leilani Gilpin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11589
• Pdf URL: https://arxiv.org/pdf/2409.11589
• Copy Paste: [[2409.11589]] ProSLM : A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering(https://arxiv.org/abs/2409.11589)
• Keywords: robust, generative, large language model
• Abstract: Neurosymbolic approaches can add robustness to opaque neural systems by incorporating explainable symbolic representations. However, previous approaches have not used formal logic to contextualize queries to and validate outputs of large language models (LLMs). We propose \systemname{}, a novel neurosymbolic framework, to improve the robustness and reliability of LLMs in question-answering tasks. We provide \systemname{} with a domain-specific knowledge base, a logical reasoning system, and an integration to an existing LLM. This framework has two capabilities (1) context gathering: generating explainable and relevant context for a given query, and (2) validation: confirming and validating the factual accuracy of a statement in accordance with a knowledge base (KB). Our work opens a new area of neurosymbolic generative AI text validation and user personalization.

Title: PieClam: A Universal Graph Autoencoder Based on Overlapping Inclusive and Exclusive Communities

• Authors: Daniel Zilberg, Ron Levie
• Subjects: cs.LG, cs.SI, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.11618
• Pdf URL: https://arxiv.org/pdf/2409.11618
• Copy Paste: [[2409.11618]] PieClam: A Universal Graph Autoencoder Based on Overlapping Inclusive and Exclusive Communities(https://arxiv.org/abs/2409.11618)
• Keywords: generative
• Abstract: We propose PieClam (Prior Inclusive Exclusive Cluster Affiliation Model): a probabilistic graph model for representing any graph as overlapping generalized communities. Our method can be interpreted as a graph autoencoder: nodes are embedded into a code space by an algorithm that maximizes the log-likelihood of the decoded graph, given the input graph. PieClam is a community affiliation model that extends well-known methods like BigClam in two main manners. First, instead of the decoder being defined via pairwise interactions between the nodes in the code space, we also incorporate a learned prior on the distribution of nodes in the code space, turning our method into a graph generative model. Secondly, we generalize the notion of communities by allowing not only sets of nodes with strong connectivity, which we call inclusive communities, but also sets of nodes with strong disconnection, which we call exclusive communities. To model both types of communities, we propose a new type of decoder based the Lorentz inner product, which we prove to be much more expressive than standard decoders based on standard inner products or norm distances. By introducing a new graph similarity measure, that we call the log cut distance, we show that PieClam is a universal autoencoder, able to uniformly approximately reconstruct any graph. Our method is shown to obtain competitive performance in graph anomaly detection benchmarks.

Title: Blockchain-Enabled IoV: Secure Communication and Trustworthy Decision-Making

• Authors: Jingyi Sun, Qi Shi, Guodong Jin, Hao Xu, Erwu Liu
• Subjects: cs.CR, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.11621
• Pdf URL: https://arxiv.org/pdf/2409.11621
• Copy Paste: [[2409.11621]] Blockchain-Enabled IoV: Secure Communication and Trustworthy Decision-Making(https://arxiv.org/abs/2409.11621)
• Keywords: secure, security
• Abstract: The Internet of Vehicles (IoV), which enables interactions between vehicles, infrastructure, and the environment, faces challenges in maintaining communication security and reliable automated decisions. This paper introduces a decentralized framework comprising a primary layer for managing inter-vehicle communication and a sub-layer for securing intra-vehicle interactions. By implementing blockchain-based protocols like Blockchain-integrated Secure Authentication (BiSA) and Decentralized Blockchain Name Resolution (DBNR), the framework ensures secure, decentralized identity management and reliable data exchanges, thereby supporting safe and efficient autonomous vehicle operations.

Title: PainDiffusion: Can robot express pain?

• Authors: Quang Tien Dam, Tri Tung Nguyen Nguyen, Dinh Tuan Tran, Joo-Ho Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11635
• Pdf URL: https://arxiv.org/pdf/2409.11635
• Copy Paste: [[2409.11635]] PainDiffusion: Can robot express pain?(https://arxiv.org/abs/2409.11635)
• Keywords: diffusion
• Abstract: Pain is a more intuitive and user-friendly way of communicating problems, making it especially useful in rehabilitation nurse training robots. While most previous methods have focused on classifying or recognizing pain expressions, these approaches often result in unnatural, jiggling robot faces. We introduce PainDiffusion, a model that generates facial expressions in response to pain stimuli, with controllable pain expressiveness and emotion status. PainDiffusion leverages diffusion forcing to roll out predictions over arbitrary lengths using a conditioned temporal U-Net. It operates as a latent diffusion model within EMOCA's facial expression latent space, ensuring a compact data representation and quick rendering time. For training data, we process the BioVid Heatpain Database, extracting expression codes and subject identity configurations. We also propose a novel set of metrics to evaluate pain expressions, focusing on expressiveness, diversity, and the appropriateness of model-generated outputs. Finally, we demonstrate that PainDiffusion outperforms the autoregressive method, both qualitatively and quantitatively. Code, videos, and further analysis are available at: \href{this https URL}{this https URL}.

Title: "A Woman is More Culturally Knowledgeable than A Man?": The Effect of Personas on Cultural Norm Interpretation in LLMs

• Authors: Mahammed Kamruzzaman, Hieu Nguyen, Nazmul Hassan, Gene Louis Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11636
• Pdf URL: https://arxiv.org/pdf/2409.11636
• Copy Paste: [[2409.11636]] "A Woman is More Culturally Knowledgeable than A Man?": The Effect of Personas on Cultural Norm Interpretation in LLMs(https://arxiv.org/abs/2409.11636)
• Keywords: large language model
• Abstract: As the deployment of large language models (LLMs) expands, there is an increasing demand for personalized LLMs. One method to personalize and guide the outputs of these models is by assigning a persona -- a role that describes the expected behavior of the LLM (e.g., a man, a woman, an engineer). This study investigates whether an LLM's understanding of social norms varies across assigned personas. Ideally, the perception of a social norm should remain consistent regardless of the persona, since acceptability of a social norm should be determined by the region the norm originates from, rather than by individual characteristics such as gender, body size, or race. A norm is universal within its cultural context. In our research, we tested 36 distinct personas from 12 sociodemographic categories (e.g., age, gender, beauty) across four different LLMs. We find that LLMs' cultural norm interpretation varies based on the persona used and the norm interpretation also varies within a sociodemographic category (e.g., a fat person and a thin person as in physical appearance group) where an LLM with the more socially desirable persona (e.g., a thin person) interprets social norms more accurately than with the less socially desirable persona (e.g., a fat person). We also discuss how different types of social biases may contribute to the results that we observe.

Title: Combating Phone Scams with LLM-based Detection: Where Do We Stand?

• Authors: Zitong Shen, Kangzhong Wang, Youqian Zhang, Grace Ngai, Eugene Y. Fu
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.11643
• Pdf URL: https://arxiv.org/pdf/2409.11643
• Copy Paste: [[2409.11643]] Combating Phone Scams with LLM-based Detection: Where Do We Stand?(https://arxiv.org/abs/2409.11643)
• Keywords: protect, large language model
• Abstract: Phone scams pose a significant threat to individuals and communities, causing substantial financial losses and emotional distress. Despite ongoing efforts to combat these scams, scammers continue to adapt and refine their tactics, making it imperative to explore innovative countermeasures. This research explores the potential of large language models (LLMs) to provide detection of fraudulent phone calls. By analyzing the conversational dynamics between scammers and victims, LLM-based detectors can identify potential scams as they occur, offering immediate protection to users. While such approaches demonstrate promising results, we also acknowledge the challenges of biased datasets, relatively low recall, and hallucinations that must be addressed for further advancement in this field

Title: Hard-Label Cryptanalytic Extraction of Neural Network Models

• Authors: Yi Chen, Xiaoyang Dong, Jian Guo, Yantian Shen, Anyu Wang, Xiaoyun Wang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11646
• Pdf URL: https://arxiv.org/pdf/2409.11646
• Copy Paste: [[2409.11646]] Hard-Label Cryptanalytic Extraction of Neural Network Models(https://arxiv.org/abs/2409.11646)
• Keywords: attack, extraction
• Abstract: The machine learning problem of extracting neural network parameters has been proposed for nearly three decades. Functionally equivalent extraction is a crucial goal for research on this problem. When the adversary has access to the raw output of neural networks, various attacks, including those presented at CRYPTO 2020 and EUROCRYPT 2024, have successfully achieved this goal. However, this goal is not achieved when neural networks operate under a hard-label setting where the raw output is inaccessible. In this paper, we propose the first attack that theoretically achieves functionally equivalent extraction under the hard-label setting, which applies to ReLU neural networks. The effectiveness of our attack is validated through practical experiments on a wide range of ReLU neural networks, including neural networks trained on two real benchmarking datasets (MNIST, CIFAR10) widely used in computer vision. For a neural network consisting of $10^5$ parameters, our attack only requires several hours on a single core.

Title: Relax DARTS: Relaxing the Constraints of Differentiable Architecture Search for Eye Movement Recognition

• Authors: Hongyu Zhu, Xin Jin, Hongchao Liao, Yan Xiang, Mounim A. El-Yacoubi, Huafeng Qin
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11652
• Pdf URL: https://arxiv.org/pdf/2409.11652
• Copy Paste: [[2409.11652]] Relax DARTS: Relaxing the Constraints of Differentiable Architecture Search for Eye Movement Recognition(https://arxiv.org/abs/2409.11652)
• Keywords: secure, biometric
• Abstract: Eye movement biometrics is a secure and innovative identification method. Deep learning methods have shown good performance, but their network architecture relies on manual design and combined priori knowledge. To address these issues, we introduce automated network search (NAS) algorithms to the field of eye movement recognition and present Relax DARTS, which is an improvement of the Differentiable Architecture Search (DARTS) to realize more efficient network search and training. The key idea is to circumvent the issue of weight sharing by independently training the architecture parameters $\alpha$ to achieve a more precise target architecture. Moreover, the introduction of module input weights $\beta$ allows cells the flexibility to select inputs, to alleviate the overfitting phenomenon and improve the model performance. Results on four public databases demonstrate that the Relax DARTS achieves state-of-the-art recognition performance. Notably, Relax DARTS exhibits adaptability to other multi-feature temporal classification tasks.

Title: Few-Shot Class-Incremental Learning with Non-IID Decentralized Data

• Authors: Cuiwei Liu, Siang Xu, Huaijun Qiu, Jing Zhang, Zhi Liu, Liang Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11657
• Pdf URL: https://arxiv.org/pdf/2409.11657
• Copy Paste: [[2409.11657]] Few-Shot Class-Incremental Learning with Non-IID Decentralized Data(https://arxiv.org/abs/2409.11657)
• Keywords: security, privacy, federate, generative
• Abstract: Few-shot class-incremental learning is crucial for developing scalable and adaptive intelligent systems, as it enables models to acquire new classes with minimal annotated data while safeguarding the previously accumulated knowledge. Nonetheless, existing methods deal with continuous data streams in a centralized manner, limiting their applicability in scenarios that prioritize data privacy and security. To this end, this paper introduces federated few-shot class-incremental learning, a decentralized machine learning paradigm tailored to progressively learn new classes from scarce data distributed across multiple clients. In this learning paradigm, clients locally update their models with new classes while preserving data privacy, and then transmit the model updates to a central server where they are aggregated globally. However, this paradigm faces several issues, such as difficulties in few-shot learning, catastrophic forgetting, and data heterogeneity. To address these challenges, we present a synthetic data-driven framework that leverages replay buffer data to maintain existing knowledge and facilitate the acquisition of new knowledge. Within this framework, a noise-aware generative replay module is developed to fine-tune local models with a balance of new and replay data, while generating synthetic data of new classes to further expand the replay buffer for future tasks. Furthermore, a class-specific weighted aggregation strategy is designed to tackle data heterogeneity by adaptively aggregating class-specific parameters based on local models performance on synthetic data. This enables effective global model optimization without direct access to client data. Comprehensive experiments across three widely-used datasets underscore the effectiveness and preeminence of the introduced framework.

Title: Bridging Domain Gap for Flight-Ready Spaceborne Vision

• Authors: Tae Ha Park, Simone D'Amico
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11661
• Pdf URL: https://arxiv.org/pdf/2409.11661
• Copy Paste: [[2409.11661]] Bridging Domain Gap for Flight-Ready Spaceborne Vision(https://arxiv.org/abs/2409.11661)
• Keywords: robust, transformer
• Abstract: This work presents Spacecraft Pose Network v3 (SPNv3), a Neural Network (NN) for monocular pose estimation of a known, non-cooperative target spacecraft. As opposed to existing literature, SPNv3 is designed and trained to be computationally efficient while providing robustness to spaceborne images that have not been observed during offline training and validation on the ground. These characteristics are essential to deploying NNs on space-grade edge devices. They are achieved through careful NN design choices, and an extensive trade-off analysis reveals features such as data augmentation, transfer learning and vision transformer architecture as a few of those that contribute to simultaneously maximizing robustness and minimizing computational overhead. Experiments demonstrate that the final SPNv3 can achieve state-of-the-art pose accuracy on hardware-in-the-loop images from a robotic testbed while having trained exclusively on computer-generated synthetic images, effectively bridging the domain gap between synthetic and real imagery. At the same time, SPNv3 runs well above the update frequency of modern satellite navigation filters when tested on a representative graphical processing unit system with flight heritage. Overall, SPNv3 is an efficient, flight-ready NN model readily applicable to a wide range of close-range rendezvous and proximity operations with target resident space objects. The code implementation of SPNv3 will be made publicly available.

Title: GReDP: A More Robust Approach for Differential Privacy Training with Gradient-Preserving Noise Reduction

• Authors: Haodi Wang, Tangyu Jiang, Yu Guo, Xiaohua Jia, Chengjun Cai
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11663
• Pdf URL: https://arxiv.org/pdf/2409.11663
• Copy Paste: [[2409.11663]] GReDP: A More Robust Approach for Differential Privacy Training with Gradient-Preserving Noise Reduction(https://arxiv.org/abs/2409.11663)
• Keywords: privacy, protect, robust
• Abstract: Deep learning models have been extensively adopted in various regions due to their ability to represent hierarchical features, which highly rely on the training set and procedures. Thus, protecting the training process and deep learning algorithms is paramount in privacy preservation. Although Differential Privacy (DP) as a powerful cryptographic primitive has achieved satisfying results in deep learning training, the existing schemes still fall short in preserving model utility, i.e., they either invoke a high noise scale or inevitably harm the original gradients. To address the above issues, in this paper, we present a more robust approach for DP training called GReDP. Specifically, we compute the model gradients in the frequency domain and adopt a new approach to reduce the noise level. Unlike the previous work, our GReDP only requires half of the noise scale compared to DPSGD [1] while keeping all the gradient information intact. We present a detailed analysis of our method both theoretically and empirically. The experimental results show that our GReDP works consistently better than the baselines on all models and training settings.

Title: Agent Aggregator with Mask Denoise Mechanism for Histopathology Whole Slide Image Analysis

• Authors: Xitong Ling, Minxi Ouyang, Yizhi Wang, Xinrui Chen, Renao Yan, Hongbo Chu, Junru Cheng, Tian Guan, Sufang Tian, Xiaoping Liu, Yonghong He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11664
• Pdf URL: https://arxiv.org/pdf/2409.11664
• Copy Paste: [[2409.11664]] Agent Aggregator with Mask Denoise Mechanism for Histopathology Whole Slide Image Analysis(https://arxiv.org/abs/2409.11664)
• Keywords: interpretability
• Abstract: Histopathology analysis is the gold standard for medical diagnosis. Accurate classification of whole slide images (WSIs) and region-of-interests (ROIs) localization can assist pathologists in diagnosis. The gigapixel resolution of WSI and the absence of fine-grained annotations make direct classification and analysis challenging. In weakly supervised learning, multiple instance learning (MIL) presents a promising approach for WSI classification. The prevailing strategy is to use attention mechanisms to measure instance importance for classification. However, attention mechanisms fail to capture inter-instance information, and self-attention causes quadratic computational complexity. To address these challenges, we propose AMD-MIL, an agent aggregator with a mask denoise mechanism. The agent token acts as an intermediate variable between the query and key for computing instance importance. Mask and denoising matrices, mapped from agents-aggregated value, dynamically mask low-contribution representations and eliminate noise. AMD-MIL achieves better attention allocation by adjusting feature representations, capturing micro-metastases in cancer, and improving interpretability. Extensive experiments on CAMELYON-16, CAMELYON-17, TCGA-KIDNEY, and TCGA-LUNG show AMD-MIL's superiority over state-of-the-art methods.

Title: RUIE: Retrieval-based Unified Information Extraction using Large Language Model

• Authors: Xincheng Liao, Junwen Duan, Yixi Huang, Jianxin Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11673
• Pdf URL: https://arxiv.org/pdf/2409.11673
• Copy Paste: [[2409.11673]] RUIE: Retrieval-based Unified Information Extraction using Large Language Model(https://arxiv.org/abs/2409.11673)
• Keywords: extraction, large language model
• Abstract: Unified information extraction (UIE) aims to complete all information extraction tasks using a single model or framework. While previous work has primarily focused on instruction-tuning large language models (LLMs) with constructed datasets, these methods require significant computational resources and struggle to generalize to unseen tasks. To address these limitations, we propose RUIE (Retrieval-based Unified Information Extraction), a framework that leverages in-context learning to enable rapid generalization while reducing computational costs. The key challenge in RUIE is selecting the most beneficial demonstrations for LLMs to effectively handle diverse IE tasks. To achieve this, we integrate LLM preferences for ranking candidate demonstrations and design a keyword-enhanced reward model to capture fine-grained relationships between queries and demonstrations. We then train a bi-encoder retriever for UIE through contrastive learning and knowledge distillation. To the best of our knowledge, RUIE is the first trainable retrieval framework for UIE. Experimental results on 8 held-out datasets demonstrate RUIE's effectiveness in generalizing to unseen tasks, with average F1-score improvements of 19.22 and 3.13 compared to instruction-tuning methods and other retrievers, respectively. Further analysis confirms RUIE's adaptability to LLMs of varying sizes and the importance of its key components.

Title: Gradient-Driven 3D Segmentation and Affordance Transfer in Gaussian Splatting Using 2D Masks

• Authors: Joji Joseph, Bharadwaj Amrutur, Shalabh Bhatnagar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11681
• Pdf URL: https://arxiv.org/pdf/2409.11681
• Copy Paste: [[2409.11681]] Gradient-Driven 3D Segmentation and Affordance Transfer in Gaussian Splatting Using 2D Masks(https://arxiv.org/abs/2409.11681)
• Keywords: robust, segmentation
• Abstract: 3D Gaussian Splatting has emerged as a powerful 3D scene representation technique, capturing fine details with high efficiency. In this paper, we introduce a novel voting-based method that extends 2D segmentation models to 3D Gaussian splats. Our approach leverages masked gradients, where gradients are filtered by input 2D masks, and these gradients are used as votes to achieve accurate segmentation. As a byproduct, we discovered that inference-time gradients can also be used to prune Gaussians, resulting in up to 21% compression. Additionally, we explore few-shot affordance transfer, allowing annotations from 2D images to be effectively transferred onto 3D Gaussian splats. The robust yet straightforward mathematical formulation underlying this approach makes it a highly effective tool for numerous downstream applications, such as augmented reality (AR), object editing, and robotics. The project code and additional resources are available at this https URL.

Title: SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation

• Authors: Mingze Sun, Chen Guo, Puhua Jiang, Shiwei Mao, Yurun Chen, Ruqi Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11682
• Pdf URL: https://arxiv.org/pdf/2409.11682
• Copy Paste: [[2409.11682]] SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation(https://arxiv.org/abs/2409.11682)
• Keywords: extraction, diffusion
• Abstract: In this paper, we propose SRIF, a novel Semantic shape Registration framework based on diffusion-based Image morphing and Flow estimation. More concretely, given a pair of extrinsically aligned shapes, we first render them from multi-views, and then utilize an image interpolation framework based on diffusion models to generate sequences of intermediate images between them. The images are later fed into a dynamic 3D Gaussian splatting framework, with which we reconstruct and post-process for intermediate point clouds respecting the image morphing processing. In the end, tailored for the above, we propose a novel registration module to estimate continuous normalizing flow, which deforms source shape consistently towards the target, with intermediate point clouds as weak guidance. Our key insight is to leverage large vision models (LVMs) to associate shapes and therefore obtain much richer semantic information on the relationship between shapes than the ad-hoc feature extraction and alignment. As a consequence, SRIF achieves high-quality dense correspondences on challenging shape pairs, but also delivers smooth, semantically meaningful interpolation in between. Empirical evidence justifies the effectiveness and superiority of our method as well as specific design choices. The code is released at this https URL.

Title: Recurrent Interpolants for Probabilistic Time Series Prediction

• Authors: Yu Chen, Marin Biloš, Sarthak Mittal, Wei Deng, Kashif Rasul, Anderson Schneider
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.11684
• Pdf URL: https://arxiv.org/pdf/2409.11684
• Copy Paste: [[2409.11684]] Recurrent Interpolants for Probabilistic Time Series Prediction(https://arxiv.org/abs/2409.11684)
• Keywords: diffusion, transformer, generative
• Abstract: Sequential models such as recurrent neural networks or transformer-based models became \textit{de facto} tools for multivariate time series forecasting in a probabilistic fashion, with applications to a wide range of datasets, such as finance, biology, medicine, etc. Despite their adeptness in capturing dependencies, assessing prediction uncertainty, and efficiency in training, challenges emerge in modeling high-dimensional complex distributions and cross-feature dependencies. To tackle these issues, recent works delve into generative modeling by employing diffusion or flow-based models. Notably, the integration of stochastic differential equations or probability flow successfully extends these methods to probabilistic time series imputation and forecasting. However, scalability issues necessitate a computational-friendly framework for large-scale generative model-based predictions. This work proposes a novel approach by blending the computational efficiency of recurrent neural networks with the high-quality probabilistic modeling of the diffusion model, which addresses challenges and advances generative models' application in time series forecasting. Our method relies on the foundation of stochastic interpolants and the extension to a broader conditional generation framework with additional control features, offering insights for future developments in this dynamic field.

Title: GUNet: A Graph Convolutional Network United Diffusion Model for Stable and Diversity Pose Generation

• Authors: Shuowen Liang, Sisi Li, Qingyun Wang, Cen Zhang, Kaiquan Zhu, Tian Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11689
• Pdf URL: https://arxiv.org/pdf/2409.11689
• Copy Paste: [[2409.11689]] GUNet: A Graph Convolutional Network United Diffusion Model for Stable and Diversity Pose Generation(https://arxiv.org/abs/2409.11689)
• Keywords: diffusion, generative
• Abstract: Pose skeleton images are an important reference in pose-controllable image generation. In order to enrich the source of skeleton images, recent works have investigated the generation of pose skeletons based on natural language. These methods are based on GANs. However, it remains challenging to perform diverse, structurally correct and aesthetically pleasing human pose skeleton generation with various textual inputs. To address this problem, we propose a framework with GUNet as the main model, PoseDiffusion. It is the first generative framework based on a diffusion model and also contains a series of variants fine-tuned based on a stable diffusion model. PoseDiffusion demonstrates several desired properties that outperform existing methods. 1) Correct Skeletons. GUNet, a denoising model of PoseDiffusion, is designed to incorporate graphical convolutional neural networks. It is able to learn the spatial relationships of the human skeleton by introducing skeletal information during the training process. 2) Diversity. We decouple the key points of the skeleton and characterise them separately, and use cross-attention to introduce textual conditions. Experimental results show that PoseDiffusion outperforms existing SoTA algorithms in terms of stability and diversity of text-driven pose skeleton generation. Qualitative analyses further demonstrate its superiority for controllable generation in Stable Diffusion.

Title: ORB-SfMLearner: ORB-Guided Self-supervised Visual Odometry with Selective Online Adaptation

• Authors: Yanlin Jin, Rui-Yang Ju, Haojun Liu, Yuzhong Zhong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11692
• Pdf URL: https://arxiv.org/pdf/2409.11692
• Copy Paste: [[2409.11692]] ORB-SfMLearner: ORB-Guided Self-supervised Visual Odometry with Selective Online Adaptation(https://arxiv.org/abs/2409.11692)
• Keywords: robust, explainability
• Abstract: Deep visual odometry, despite extensive research, still faces limitations in accuracy and generalizability that prevent its broader application. To address these challenges, we propose an Oriented FAST and Rotated BRIEF (ORB)-guided visual odometry with selective online adaptation named ORB-SfMLearner. We present a novel use of ORB features for learning-based ego-motion estimation, leading to more robust and accurate results. We also introduce the cross-attention mechanism to enhance the explainability of PoseNet and have revealed that driving direction of the vehicle can be explained through attention weights, marking a novel exploration in this area. To improve generalizability, our selective online adaptation allows the network to rapidly and selectively adjust to the optimal parameters across different domains. Experimental results on KITTI and vKITTI datasets show that our method outperforms previous state-of-the-art deep visual odometry methods in terms of ego-motion accuracy and generalizability.

Title: On the second-order zero differential properties of several classes of power functions over finite fields

• Authors: Huan Zhou, Xiaoni Du, Xingbin Qiao, Wenping Yuan
• Subjects: cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2409.11693
• Pdf URL: https://arxiv.org/pdf/2409.11693
• Copy Paste: [[2409.11693]] On the second-order zero differential properties of several classes of power functions over finite fields(https://arxiv.org/abs/2409.11693)
• Keywords: attack
• Abstract: Feistel Boomerang Connectivity Table (FBCT) is an important cryptanalytic technique on analysing the resistance of the Feistel network-based ciphers to power attacks such as differential and boomerang attacks. Moreover, the coefficients of FBCT are closely related to the second-order zero differential spectra of the function $F(x)$ over the finite fields with even characteristic and the Feistel boomerang uniformity is the second-order zero differential uniformity of $F(x)$. In this paper, by computing the number of solutions of specific equations over finite fields, we determine explicitly the second-order zero differential spectra of power functions $x^{2^m+3}$ and $x^{2^m+5}$ with $m>2$ being a positive integer over finite field with even characteristic, and $x^{p^k+1}$ with integer $k\geq1$ over finite field with odd characteristic $p$. It is worth noting that $x^{2^m+3}$ is a permutation over $\mathbb{F}_{2^n}$ and only when $m$ is odd, $x^{2^m+5}$ is a permutation over $\mathbb{F}_{2^n}$, where integer $n=2m$. As a byproduct, we find $F(x)=x^4$ is a PN and second-order zero differentially $0$-uniform function over $\mathbb{F}_{3^n}$ with odd $n$. The computation of these entries and the cardinalities in each table aimed to facilitate the analysis of differential and boomerang cryptanalysis of S-boxes when studying distinguishers and trails.

Title: Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation

• Authors: Chunliang Tao, Xiaojing Fan, Yahe Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11703
• Pdf URL: https://arxiv.org/pdf/2409.11703
• Copy Paste: [[2409.11703]] Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation(https://arxiv.org/abs/2409.11703)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) advance in natural language processing, there is growing interest in leveraging their capabilities to simplify software interactions. In this paper, we propose a novel system that integrates LLMs for both classifying natural language inputs into corresponding API calls and automating the creation of sample datasets tailored to specific API functions. By classifying natural language commands, our system allows users to invoke complex software functionalities through simple inputs, improving interaction efficiency and lowering the barrier to software utilization. Our dataset generation approach also enables the efficient and systematic evaluation of different LLMs in classifying API calls, offering a practical tool for developers or business owners to assess the suitability of LLMs for customized API management. We conduct experiments on several prominent LLMs using generated sample datasets for various API functions. The results show that GPT-4 achieves a high classification accuracy of 0.996, while LLaMA-3-8B performs much worse at 0.759. These findings highlight the potential of LLMs to transform API management and validate the effectiveness of our system in guiding model testing and selection across diverse applications.

Title: From Lists to Emojis: How Format Bias Affects Model Alignment

• Authors: Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, Tong Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11704
• Pdf URL: https://arxiv.org/pdf/2409.11704
• Copy Paste: [[2409.11704]] From Lists to Emojis: How Format Bias Affects Model Alignment(https://arxiv.org/abs/2409.11704)
• Keywords: large language model
• Abstract: In this paper, we study format biases in reinforcement learning from human feedback (RLHF). We observe that many widely-used preference models, including human evaluators, GPT-4, and top-ranking models on the RewardBench benchmark, exhibit strong biases towards specific format patterns, such as lists, links, bold text, and emojis. Furthermore, large language models (LLMs) can exploit these biases to achieve higher rankings on popular benchmarks like AlpacaEval and LMSYS Chatbot Arena. One notable example of this is verbosity bias, where current preference models favor longer responses that appear more comprehensive, even when their quality is equal to or lower than shorter, competing responses. However, format biases beyond verbosity remain largely underexplored in the literature. In this work, we extend the study of biases in preference learning beyond the commonly recognized length bias, offering a comprehensive analysis of a wider range of format biases. Additionally, we show that with a small amount of biased data (less than 1%), we can inject significant bias into the reward model. Moreover, these format biases can also be easily exploited by downstream alignment algorithms, such as best-of-n sampling and online iterative DPO, as it is usually easier to manipulate the format than to improve the quality of responses. Our findings emphasize the need to disentangle format and content both for designing alignment algorithms and evaluating models.

Title: TART: An Open-Source Tool-Augmented Framework for Explainable Table-based Reasoning

• Authors: Xinyuan Lu, Liangming Pan, Yubo Ma, Preslav Nakov, Min-Yen Kan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11724
• Pdf URL: https://arxiv.org/pdf/2409.11724
• Copy Paste: [[2409.11724]] TART: An Open-Source Tool-Augmented Framework for Explainable Table-based Reasoning(https://arxiv.org/abs/2409.11724)
• Keywords: robust, explainability, large language model
• Abstract: Current Large Language Models (LLMs) exhibit limited ability to understand table structures and to apply precise numerical reasoning, which is crucial for tasks such as table question answering (TQA) and table-based fact verification (TFV). To address these challenges, we introduce our Tool-Augmented Reasoning framework for Tables (TART), which integrates LLMs with specialized tools. TART contains three key components: a table formatter to ensure accurate data representation, a tool maker to develop specific computational tools, and an explanation generator to maintain explainability. We also present the TOOLTAB dataset, a new benchmark designed specifically for training LLMs in table-tool integration. Our experiments indicate that TART achieves substantial improvements over existing methods (e.g., Chain-of-Thought) by improving both the precision of data processing and the clarity of the reasoning process. Notably, TART paired with CodeLlama achieves 90.0% of the accuracy of the closed-sourced LLM GPT-3.5-turbo, highlighting its robustness in diverse real-world scenarios. All the code and data are available at this https URL.

Title: Revealing the Challenge of Detecting Character Knowledge Errors in LLM Role-Playing

• Authors: Wenyuan Zhang, Jiawei Sheng, Shuaiyi Nie, Zefeng Zhang, Xinghua Zhang, Yongquan He, Tingwen Liu
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.11726
• Pdf URL: https://arxiv.org/pdf/2409.11726
• Copy Paste: [[2409.11726]] Revealing the Challenge of Detecting Character Knowledge Errors in LLM Role-Playing(https://arxiv.org/abs/2409.11726)
• Keywords: large language model
• Abstract: Large language model (LLM) role-playing has gained widespread attention, where the authentic character knowledge is crucial for constructing realistic LLM role-playing agents. However, existing works usually overlook the exploration of LLMs' ability to detect characters' known knowledge errors (KKE) and unknown knowledge errors (UKE) while playing roles, which would lead to low-quality automatic construction of character trainable corpus. In this paper, we propose a probing dataset to evaluate LLMs' ability to detect errors in KKE and UKE. The results indicate that even the latest LLMs struggle to effectively detect these two types of errors, especially when it comes to familiar knowledge. We experimented with various reasoning strategies and propose an agent-based reasoning method, Self-Recollection and Self-Doubt (S2RD), to further explore the potential for improving error detection capabilities. Experiments show that our method effectively improves the LLMs' ability to detect error character knowledge, but it remains an issue that requires ongoing attention.

Title: Enabling Real-Time Conversations with Minimal Training Costs

• Authors: Wang Xu, Shuo Wang, Weilin Zhao, Xu Han, Yukun Yan, Yudi Zhang, Zhe Tao, Zhiyuan Liu, Wanxiang Che
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11727
• Pdf URL: https://arxiv.org/pdf/2409.11727
• Copy Paste: [[2409.11727]] Enabling Real-Time Conversations with Minimal Training Costs(https://arxiv.org/abs/2409.11727)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated the ability to improve human efficiency through conversational interactions. Conventional LLM-powered dialogue systems, operating on a turn-based paradigm, preclude real-time interaction during response generation. To address this limitation, researchers have proposed duplex models. These models can dynamically adapt to user input, facilitating real-time interactive feedback. However, these methods typically require substantial computational resources to acquire the ability. To reduce overhead, this paper presents a new duplex decoding approach that enhances LLMs with duplex ability, requiring minimal additional training. Specifically, our method employs parallel decoding of queries and responses in conversations, effectively implementing a channel-division-multiplexing decoding strategy. Experimental results indicate that our proposed method significantly enhances the naturalness and human-likeness of user-AI interactions with minimal training costs.

Title: InverseMeetInsert: Robust Real Image Editing via Geometric Accumulation Inversion in Guided Diffusion Models

• Authors: Yan Zheng, Lemeng Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11734
• Pdf URL: https://arxiv.org/pdf/2409.11734
• Copy Paste: [[2409.11734]] InverseMeetInsert: Robust Real Image Editing via Geometric Accumulation Inversion in Guided Diffusion Models(https://arxiv.org/abs/2409.11734)
• Keywords: robust, diffusion
• Abstract: In this paper, we introduce Geometry-Inverse-Meet-Pixel-Insert, short for GEO, an exceptionally versatile image editing technique designed to cater to customized user requirements at both local and global scales. Our approach seamlessly integrates text prompts and image prompts to yield diverse and precise editing outcomes. Notably, our method operates without the need for training and is driven by two key contributions: (i) a novel geometric accumulation loss that enhances DDIM inversion to faithfully preserve pixel space geometry and layout, and (ii) an innovative boosted image prompt technique that combines pixel-level editing for text-only inversion with latent space geometry guidance for standard classifier-free reversion. Leveraging the publicly available Stable Diffusion model, our approach undergoes extensive evaluation across various image types and challenging prompt editing scenarios, consistently delivering high-fidelity editing results for real images.

Title: RockTrack: A 3D Robust Multi-Camera-Ken Multi-Object Tracking Framework

• Authors: Xiaoyu Li, Peidong Li, Lijun Zhao, Dedong Liu, Jinghan Gao, Xian Wu, Yitao Wu, Dixiao Cui
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.11749
• Pdf URL: https://arxiv.org/pdf/2409.11749
• Copy Paste: [[2409.11749]] RockTrack: A 3D Robust Multi-Camera-Ken Multi-Object Tracking Framework(https://arxiv.org/abs/2409.11749)
• Keywords: robust
• Abstract: 3D Multi-Object Tracking (MOT) obtains significant performance improvements with the rapid advancements in 3D object detection, particularly in cost-effective multi-camera setups. However, the prevalent end-to-end training approach for multi-camera trackers results in detector-specific models, limiting their versatility. Moreover, current generic trackers overlook the unique features of multi-camera detectors, i.e., the unreliability of motion observations and the feasibility of visual information. To address these challenges, we propose RockTrack, a 3D MOT method for multi-camera detectors. Following the Tracking-By-Detection framework, RockTrack is compatible with various off-the-shelf detectors. RockTrack incorporates a confidence-guided preprocessing module to extract reliable motion and image observations from distinct representation spaces from a single detector. These observations are then fused in an association module that leverages geometric and appearance cues to minimize mismatches. The resulting matches are propagated through a staged estimation process, forming the basis for heuristic noise modeling. Additionally, we introduce a novel appearance similarity metric for explicitly characterizing object affinities in multi-camera settings. RockTrack achieves state-of-the-art performance on the nuScenes vision-only tracking leaderboard with 59.1% AMOTA while demonstrating impressive computational efficiency.

Title: NPAT Null-Space Projected Adversarial Training Towards Zero Deterioration

• Authors: Hanyi Hu, Qiao Han, Kui Chen, Yao Yang
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11754
• Pdf URL: https://arxiv.org/pdf/2409.11754
• Copy Paste: [[2409.11754]] NPAT Null-Space Projected Adversarial Training Towards Zero Deterioration(https://arxiv.org/abs/2409.11754)
• Keywords: defense, attack, robust
• Abstract: To mitigate the susceptibility of neural networks to adversarial attacks, adversarial training has emerged as a prevalent and effective defense strategy. Intrinsically, this countermeasure incurs a trade-off, as it sacrifices the model's accuracy in processing normal samples. To reconcile the trade-off, we pioneer the incorporation of null-space projection into adversarial training and propose two innovative Null-space Projection based Adversarial Training(NPAT) algorithms tackling sample generation and gradient optimization, named Null-space Projected Data Augmentation (NPDA) and Null-space Projected Gradient Descent (NPGD), to search for an overarching optimal solutions, which enhance robustness with almost zero deterioration in generalization performance. Adversarial samples and perturbations are constrained within the null-space of the decision boundary utilizing a closed-form null-space projector, effectively mitigating threat of attack stemming from unreliable features. Subsequently, we conducted experiments on the CIFAR10 and SVHN datasets and reveal that our methodology can seamlessly combine with adversarial training methods and obtain comparable robustness while keeping generalization close to a high-accuracy model.

Title: Consistent Estimation of a Class of Distances Between Covariance Matrices

• Authors: Roberto Pereira, Xavier Mestre, Davig Gregoratti
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11761
• Pdf URL: https://arxiv.org/pdf/2409.11761
• Copy Paste: [[2409.11761]] Consistent Estimation of a Class of Distances Between Covariance Matrices(https://arxiv.org/abs/2409.11761)
• Keywords: robust
• Abstract: This work considers the problem of estimating the distance between two covariance matrices directly from the data. Particularly, we are interested in the family of distances that can be expressed as sums of traces of functions that are separately applied to each covariance matrix. This family of distances is particularly useful as it takes into consideration the fact that covariance matrices lie in the Riemannian manifold of positive definite matrices, thereby including a variety of commonly used metrics, such as the Euclidean distance, Jeffreys' divergence, and the log-Euclidean distance. Moreover, a statistical analysis of the asymptotic behavior of this class of distance estimators has also been conducted. Specifically, we present a central limit theorem that establishes the asymptotic Gaussianity of these estimators and provides closed form expressions for the corresponding means and variances. Empirical evaluations demonstrate the superiority of our proposed consistent estimator over conventional plug-in estimators in multivariate analytical contexts. Additionally, the central limit theorem derived in this study provides a robust statistical framework to assess of accuracy of these estimators.

Title: Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resources

• Authors: Issey Sukeda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11783
• Pdf URL: https://arxiv.org/pdf/2409.11783
• Copy Paste: [[2409.11783]] Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resources(https://arxiv.org/abs/2409.11783)
• Keywords: security, large language model
• Abstract: The recent success of large language models (LLMs) and the scaling law has led to a widespread adoption of larger models. Particularly in the healthcare industry, there is an increasing demand for locally operated LLMs due to security concerns. However, the majority of high quality open-source LLMs have a size of 70B parameters, imposing significant financial burdens on users for GPU preparation and operation. To overcome these issues, we present a medical adaptation based on the recent 7B models, which enables the operation in low computational resources. We compare the performance on medical question-answering benchmarks in two languages (Japanese and English), demonstrating that its scores reach parity with or surpass those of currently existing medical LLMs that are ten times larger. We find that fine-tuning an English-centric base model on Japanese medical dataset improves the score in both language, supporting the effect of cross-lingual knowledge transfer. We hope that this study will alleviate financial challenges, serving as a stepping stone for clinical institutions to practically utilize LLMs locally. Our evaluation code is available at this https URL.

Title: Empowering Visual Artists with Tokenized Digital Assets with NFTs

• Authors: Ruiqiang Li, Brian Yecies, Qin Wang, Shiping Chen, Jun Shen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11790
• Pdf URL: https://arxiv.org/pdf/2409.11790
• Copy Paste: [[2409.11790]] Empowering Visual Artists with Tokenized Digital Assets with NFTs(https://arxiv.org/abs/2409.11790)
• Keywords: security, privacy
• Abstract: The Non-Fungible Tokens (NFTs) has the transformative impact on the visual arts industry by examining the nexus between empowering art practices and leveraging blockchain technology. First, we establish the context for this study by introducing some basic but critical technological aspects and affordances of the blockchain domain. Second, we revisit the creative practices involved in producing traditional artwork, covering various types, production processes, trading, and monetization methods. Third, we introduce and define the key fundamentals of the blockchain ecosystem, including its structure, consensus algorithms, smart contracts, and digital wallets. Fourth, we narrow the focus to NFTs, detailing their history, mechanics, lifecycle, and standards, as well as their application in the art world. In particular, we outline the key processes for minting and trading NFTs in various marketplaces and discuss the relevant market dynamics and pricing. We also consider major security concerns, such as wash trading, to underscore some of the central cybersecurity issues facing this domain. Finally, we conclude by considering future research directions, emphasizing improvements in user experience, security, and privacy. Through this innovative research overview, which includes input from creative industry and cybersecurity sdomain expertise, we offer some new insights into how NFTs can empower visual artists and reshape the wider copyright industries.

Title: The Factuality of Large Language Models in the Legal Domain

• Authors: Rajaa El Hamdani, Thomas Bonald, Fragkiskos Malliaros, Nils Holzenberger, Fabian Suchanek
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11798
• Pdf URL: https://arxiv.org/pdf/2409.11798
• Copy Paste: [[2409.11798]] The Factuality of Large Language Models in the Legal Domain(https://arxiv.org/abs/2409.11798)
• Keywords: large language model
• Abstract: This paper investigates the factuality of large language models (LLMs) as knowledge bases in the legal domain, in a realistic usage scenario: we allow for acceptable variations in the answer, and let the model abstain from answering when uncertain. First, we design a dataset of diverse factual questions about case law and legislation. We then use the dataset to evaluate several LLMs under different evaluation methods, including exact, alias, and fuzzy matching. Our results show that the performance improves significantly under the alias and fuzzy matching methods. Further, we explore the impact of abstaining and in-context examples, finding that both strategies enhance precision. Finally, we demonstrate that additional pre-training on legal documents, as seen with SaulLM, further improves factual precision from 63% to 81%.

Title: Latent fingerprint enhancement for accurate minutiae detection

• Authors: Abdul Wahab, Tariq Mahmood Khan, Shahzaib Iqbal, Bandar AlShammari, Bandar Alhaqbani, Imran Razzak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11802
• Pdf URL: https://arxiv.org/pdf/2409.11802
• Copy Paste: [[2409.11802]] Latent fingerprint enhancement for accurate minutiae detection(https://arxiv.org/abs/2409.11802)
• Keywords: robust, generative
• Abstract: Identification of suspects based on partial and smudged fingerprints, commonly referred to as fingermarks or latent fingerprints, presents a significant challenge in the field of fingerprint recognition. Although fixed-length embeddings have shown effectiveness in recognising rolled and slap fingerprints, the methods for matching latent fingerprints have primarily centred around local minutiae-based embeddings, failing to fully exploit global representations for matching purposes. Consequently, enhancing latent fingerprints becomes critical to ensuring robust identification for forensic investigations. Current approaches often prioritise restoring ridge patterns, overlooking the fine-macroeconomic details crucial for accurate fingerprint recognition. To address this, we propose a novel approach that uses generative adversary networks (GANs) to redefine Latent Fingerprint Enhancement (LFE) through a structured approach to fingerprint generation. By directly optimising the minutiae information during the generation process, the model produces enhanced latent fingerprints that exhibit exceptional fidelity to ground-truth instances. This leads to a significant improvement in identification performance. Our framework integrates minutiae locations and orientation fields, ensuring the preservation of both local and structural fingerprint features. Extensive evaluations conducted on two publicly available datasets demonstrate our method's dominance over existing state-of-the-art techniques, highlighting its potential to significantly enhance latent fingerprint recognition accuracy in forensic applications.

Title: Model-Checking the Implementation of Consent

• Authors: Raúl Pardo, Daniel Le Métayer
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.11803
• Pdf URL: https://arxiv.org/pdf/2409.11803
• Copy Paste: [[2409.11803]] Model-Checking the Implementation of Consent(https://arxiv.org/abs/2409.11803)
• Keywords: privacy, protect
• Abstract: Privacy policies define the terms under which personal data may be collected and processed by data controllers. The General Data Protection Regulation (GDPR) imposes requirements on these policies that are often difficult to implement. Difficulties arise in particular due to the heterogeneity of existing systems (e.g., the Internet of Things (IoT), web technology, etc.). In this paper, we propose a method to refine high level GDPR privacy requirements for informed consent into low-level computational models. The method is aimed at software developers implementing systems that require consent management. We mechanize our models in TLA+ and use model-checking to prove that the low-level computational models implement the high-level privacy requirements; TLA+ has been used by software engineers in companies such as Microsoft or Amazon. We demonstrate our method in two real world scenarios: an implementation of cookie banners and a IoT system communicating via Bluetooth low energy.

Title: Constraint Guided AutoEncoders for Joint Optimization of Condition Indicator Estimation and Anomaly Detection in Machine Condition Monitoring

• Authors: Maarten Meire, Quinten Van Baelen, Ted Ooijevaar, Peter Karsmakers
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11807
• Pdf URL: https://arxiv.org/pdf/2409.11807
• Copy Paste: [[2409.11807]] Constraint Guided AutoEncoders for Joint Optimization of Condition Indicator Estimation and Anomaly Detection in Machine Condition Monitoring(https://arxiv.org/abs/2409.11807)
• Keywords: robust
• Abstract: The main goal of machine condition monitoring is, as the name implies, to monitor the condition of industrial applications. The objective of this monitoring can be mainly split into two problems. A diagnostic problem, where normal data should be distinguished from anomalous data, otherwise called Anomaly Detection (AD), or a prognostic problem, where the aim is to predict the evolution of a Condition Indicator (CI) that reflects the condition of an asset throughout its life time. When considering machine condition monitoring, it is expected that this CI shows a monotonic behavior, as the condition of a machine gradually degrades over time. This work proposes an extension to Constraint Guided AutoEncoders (CGAE), which is a robust AD method, that enables building a single model that can be used for both AD and CI estimation. For the purpose of improved CI estimation the extension incorporates a constraint that enforces the model to have monotonically increasing CI predictions over time. Experimental results indicate that the proposed algorithm performs similar, or slightly better, than CGAE, with regards to AD, while improving the monotonic behavior of the CI.

Title: EventAug: Multifaceted Spatio-Temporal Data Augmentation Methods for Event-based Learning

• Authors: Yukun Tian, Hao Chen, Yongjian Deng, Feihong Shen, Kepan Liu, Wei You, Ziyang Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11813
• Pdf URL: https://arxiv.org/pdf/2409.11813
• Copy Paste: [[2409.11813]] EventAug: Multifaceted Spatio-Temporal Data Augmentation Methods for Event-based Learning(https://arxiv.org/abs/2409.11813)
• Keywords: robust
• Abstract: The event camera has demonstrated significant success across a wide range of areas due to its low time latency and high dynamic range. However, the community faces challenges such as data deficiency and limited diversity, often resulting in over-fitting and inadequate feature learning. Notably, the exploration of data augmentation techniques in the event community remains scarce. This work aims to address this gap by introducing a systematic augmentation scheme named EventAug to enrich spatial-temporal diversity. In particular, we first propose Multi-scale Temporal Integration (MSTI) to diversify the motion speed of objects, then introduce Spatial-salient Event Mask (SSEM) and Temporal-salient Event Mask (TSEM) to enrich object variants. Our EventAug can facilitate models learning with richer motion patterns, object variants and local spatio-temporal relations, thus improving model robustness to varied moving speeds, occlusions, and action disruptions. Experiment results show that our augmentation method consistently yields significant improvements across different tasks and backbones (e.g., a 4.87% accuracy gain on DVS128 Gesture). Our code will be publicly available for this community.

Title: Graph Neural Network-State Predictive Information Bottleneck (GNN-SPIB) approach for learning molecular thermodynamics and kinetics

• Authors: Ziyue Zou, Dedi Wang, Pratyush Tiwary
• Subjects: cs.LG, cond-mat.soft, cond-mat.stat-mech
• Abstract URL: https://arxiv.org/abs/2409.11843
• Pdf URL: https://arxiv.org/pdf/2409.11843
• Copy Paste: [[2409.11843]] Graph Neural Network-State Predictive Information Bottleneck (GNN-SPIB) approach for learning molecular thermodynamics and kinetics(https://arxiv.org/abs/2409.11843)
• Keywords: robust
• Abstract: Molecular dynamics simulations offer detailed insights into atomic motions but face timescale limitations. Enhanced sampling methods have addressed these challenges but even with machine learning, they often rely on pre-selected expert-based features. In this work, we present the Graph Neural Network-State Predictive Information Bottleneck (GNN-SPIB) framework, which combines graph neural networks and the State Predictive Information Bottleneck to automatically learn low-dimensional representations directly from atomic coordinates. Tested on three benchmark systems, our approach predicts essential structural, thermodynamic and kinetic information for slow processes, demonstrating robustness across diverse systems. The method shows promise for complex systems, enabling effective enhanced sampling without requiring pre-defined reaction coordinates or input features.

Title: MEOW: MEMOry Supervised LLM Unlearning Via Inverted Facts

• Authors: Tianle Gu, Kexin Huang, Ruilin Luo, Yuanqi Yao, Yujiu Yang, Yan Teng, Yingchun Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11844
• Pdf URL: https://arxiv.org/pdf/2409.11844
• Copy Paste: [[2409.11844]] MEOW: MEMOry Supervised LLM Unlearning Via Inverted Facts(https://arxiv.org/abs/2409.11844)
• Keywords: robust, extraction, large language model
• Abstract: Large Language Models (LLMs) can memorize sensitive information, raising concerns about potential misuse. LLM Unlearning, a post-hoc approach to remove this information from trained LLMs, offers a promising solution to mitigate these risks. However, previous practices face three key challenges: 1. Utility: successful unlearning often causes catastrophic collapse on unrelated tasks. 2. Efficiency: many methods either involve adding similarly sized models, which slows down unlearning or inference, or require retain data that are difficult to obtain. 3. Robustness: even effective methods may still leak data via extraction techniques. To address these challenges, we propose MEOW, a simple yet effective gradient descent-based unlearning method. Specifically, we use an offline LLM to generate a set of inverted facts. Then, we design a new metric, MEMO, to quantify memorization in LLMs. Finally, based on the signals provided by MEMO, we select the most appropriate set of inverted facts and finetune the model based on them. We evaluate MEOW on the commonly used unlearn benchmark, ToFU, with Llama2-7B-Chat and Phi-1.5B, and test it on both NLU and NLG tasks. Results demonstrate significant improvement of MEOW in forget quality without substantial loss in model utility. Meanwhile, MEOW does not exhibit significant degradation in NLU or NLG capabilities, and there is even a slight improvement in NLU performance.

Title: Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers

• Authors: Ekaterina Grishina, Mikhail Gorbunov, Maxim Rakhuba
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11859
• Pdf URL: https://arxiv.org/pdf/2409.11859
• Copy Paste: [[2409.11859]] Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers(https://arxiv.org/abs/2409.11859)
• Keywords: robust
• Abstract: Controlling the spectral norm of the Jacobian matrix, which is related to the convolution operation, has been shown to improve generalization, training stability and robustness in CNNs. Existing methods for computing the norm either tend to overestimate it or their performance may deteriorate quickly with increasing the input and kernel sizes. In this paper, we demonstrate that the tensor version of the spectral norm of a four-dimensional convolution kernel, up to a constant factor, serves as an upper bound for the spectral norm of the Jacobian matrix associated with the convolution operation. This new upper bound is independent of the input image resolution, differentiable and can be efficiently calculated during training. Through experiments, we demonstrate how this new bound can be used to improve the performance of convolutional architectures.

Title: Distillation-free Scaling of Large SSMs for Images and Videos

• Authors: Hamid Suleman, Syed Talal Wasim, Muzammal Naseer, Juergen Gall
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11867
• Pdf URL: https://arxiv.org/pdf/2409.11867
• Copy Paste: [[2409.11867]] Distillation-free Scaling of Large SSMs for Images and Videos(https://arxiv.org/abs/2409.11867)
• Keywords: robust
• Abstract: State-space models (SSMs), exemplified by S4, have introduced a novel context modeling method by integrating state-space techniques into deep learning. However, they struggle with global context modeling due to their data-independent matrices. The Mamba model addressed this with data-dependent variants via the S6 selective-scan algorithm, enhancing context modeling, especially for long sequences. However, Mamba-based architectures are difficult to scale with respect to the number of parameters, which is a major limitation for vision applications. This paper addresses the scalability issue of large SSMs for image classification and action recognition without requiring additional techniques like knowledge distillation. We analyze the distinct characteristics of Mamba-based and Attention-based models, proposing a Mamba-Attention interleaved architecture that enhances scalability, robustness, and performance. We demonstrate that the stable and efficient interleaved architecture resolves the scalability issue of Mamba-based architectures for images and videos and increases robustness to common artifacts like JPEG compression. Our thorough evaluation on the ImageNet-1K, Kinetics-400 and Something-Something-v2 benchmarks demonstrates that our approach improves the accuracy of state-of-the-art Mamba-based architectures by up to $+1.7$.

Title: SpheriGait: Enriching Spatial Representation via Spherical Projection for LiDAR-based Gait Recognition

• Authors: Yanxi Wang, Zhigang Chang, Chen Wu, Zihao Cheng, Hongmin Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11869
• Pdf URL: https://arxiv.org/pdf/2409.11869
• Copy Paste: [[2409.11869]] SpheriGait: Enriching Spatial Representation via Spherical Projection for LiDAR-based Gait Recognition(https://arxiv.org/abs/2409.11869)
• Keywords: privacy, protect
• Abstract: Gait recognition is a rapidly progressing technique for the remote identification of individuals. Prior research predominantly employing 2D sensors to gather gait data has achieved notable advancements; nonetheless, they have unavoidably neglected the influence of 3D dynamic characteristics on recognition. Gait recognition utilizing LiDAR 3D point clouds not only directly captures 3D spatial features but also diminishes the impact of lighting conditions while ensuring privacy protection.The essence of the problem lies in how to effectively extract discriminative 3D dynamic representation from point this http URL this paper, we proposes a method named SpheriGait for extracting and enhancing dynamic features from point clouds for Lidar-based gait recognition. Specifically, it substitutes the conventional point cloud plane projection method with spherical projection to augment the perception of dynamic feature.Additionally, a network block named DAM-L is proposed to extract gait cues from the projected point cloud data. We conducted extensive experiments and the results demonstrated the SpheriGait achieved state-of-the-art performance on the SUSTech1K dataset, and verified that the spherical projection method can serve as a universal data preprocessing technique to enhance the performance of other LiDAR-based gait recognition methods, exhibiting exceptional flexibility and practicality.

Title: ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Images

• Authors: Abhinaw Jagtap, Nachiket Tapas, R. G. Brajesh
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.11874
• Pdf URL: https://arxiv.org/pdf/2409.11874
• Copy Paste: [[2409.11874]] ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Images(https://arxiv.org/abs/2409.11874)
• Keywords: robust, diffusion, generative
• Abstract: In the fast-evolving field of Generative AI, platforms like MidJourney, DALL-E, and Stable Diffusion have transformed Text-to-Image (T2I) Generation. However, despite their impressive ability to create high-quality images, they often struggle to generate accurate text within these images. Theoretically, if we could achieve accurate text generation in AI images in a ``zero-shot'' manner, it would not only make AI-generated images more meaningful but also democratize the graphic design industry. The first step towards this goal is to create a robust scoring matrix for evaluating text accuracy in AI-generated images. Although there are existing bench-marking methods like CLIP SCORE and T2I-CompBench++, there's still a gap in systematically evaluating text and typography in AI-generated images, especially with diffusion-based methods. In this paper, we introduce a novel evaluation matrix designed explicitly for quantifying the performance of text and typography generation within AI-generated images. We have used letter by letter matching strategy to compute the exact matching scores from the reference text to the AI generated text. Our novel approach to calculate the score takes care of multiple redundancies such as repetition of words, case sensitivity, mixing of words, irregular incorporation of letters etc. Moreover, we have developed a Novel method named as brevity adjustment to handle excess text. In addition we have also done a quantitative analysis of frequent errors arise due to frequently used words and less frequently used words. Project page is available at: this https URL.

Title: DocMamba: Efficient Document Pre-training with State Space Model

• Authors: Pengfei Hu, Zhenrong Zhang, Jiefeng Ma, Shuhang Liu, Jun Du, Jianshu Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11887
• Pdf URL: https://arxiv.org/pdf/2409.11887
• Copy Paste: [[2409.11887]] DocMamba: Efficient Document Pre-training with State Space Model(https://arxiv.org/abs/2409.11887)
• Keywords: transformer
• Abstract: In recent years, visually-rich document understanding has attracted increasing attention. Transformer-based pre-trained models have become the mainstream approach, yielding significant performance gains in this field. However, the self-attention mechanism's quadratic computational complexity hinders their efficiency and ability to process long documents. In this paper, we present DocMamba, a novel framework based on the state space model. It is designed to reduce computational complexity to linear while preserving global modeling capabilities. To further enhance its effectiveness in document processing, we introduce the Segment-First Bidirectional Scan (SFBS) to capture contiguous semantic information. Experimental results demonstrate that DocMamba achieves new state-of-the-art results on downstream datasets such as FUNSD, CORD, and SORIE, while significantly improving speed and reducing memory usage. Notably, experiments on the HRDoc confirm DocMamba's potential for length extrapolation. The code will be available online.

Title: Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction

• Authors: Caihong Wang, Du Xu, Zonghang Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11890
• Pdf URL: https://arxiv.org/pdf/2409.11890
• Copy Paste: [[2409.11890]] Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction(https://arxiv.org/abs/2409.11890)
• Keywords: robust, extraction
• Abstract: In the era of rapid Internet development, log data has become indispensable for recording the operations of computer devices and software. These data provide valuable insights into system behavior and necessitate thorough analysis. Recent advances in text analysis have enabled deep learning to achieve significant breakthroughs in log anomaly detection. However, the high cost of manual annotation and the dynamic nature of usage scenarios present major challenges to effective log analysis. This study proposes a novel log feature extraction model called DualGCN-LogAE, designed to adapt to various scenarios. It leverages the expressive power of large models for log content analysis and the capability of graph structures to encapsulate correlations between logs. It retains key log information while integrating the causal relationships between logs to achieve effective feature extraction. Additionally, we introduce Log2graphs, an unsupervised log anomaly detection method based on the feature extractor. By employing graph clustering algorithms for log anomaly detection, Log2graphs enables the identification of abnormal logs without the need for labeled data. We comprehensively evaluate the feature extraction capability of DualGCN-LogAE and the anomaly detection performance of Log2graphs using public log datasets across five different scenarios. Our evaluation metrics include detection accuracy and graph clustering quality scores. Experimental results demonstrate that the log features extracted by DualGCN-LogAE outperform those obtained by other methods on classic classifiers. Moreover, Log2graphs surpasses existing unsupervised log detection methods, providing a robust tool for advancing log anomaly detection research.

Title: Multi-Grid Graph Neural Networks with Self-Attention for Computational Mechanics

• Authors: Paul Garnier, Jonathan Viquerat, Elie Hachem
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2409.11899
• Pdf URL: https://arxiv.org/pdf/2409.11899
• Copy Paste: [[2409.11899]] Multi-Grid Graph Neural Networks with Self-Attention for Computational Mechanics(https://arxiv.org/abs/2409.11899)
• Keywords: robust
• Abstract: Advancement in finite element methods have become essential in various disciplines, and in particular for Computational Fluid Dynamics (CFD), driving research efforts for improved precision and efficiency. While Convolutional Neural Networks (CNNs) have found success in CFD by mapping meshes into images, recent attention has turned to leveraging Graph Neural Networks (GNNs) for direct mesh processing. This paper introduces a novel model merging Self-Attention with Message Passing in GNNs, achieving a 15\% reduction in RMSE on the well known flow past a cylinder benchmark. Furthermore, a dynamic mesh pruning technique based on Self-Attention is proposed, that leads to a robust GNN-based multigrid approach, also reducing RMSE by 15\%. Additionally, a new self-supervised training method based on BERT is presented, resulting in a 25\% RMSE reduction. The paper includes an ablation study and outperforms state-of-the-art models on several challenging datasets, promising advancements similar to those recently achieved in natural language and image processing. Finally, the paper introduces a dataset with meshes larger than existing ones by at least an order of magnitude. Code and Datasets will be released at this https URL.

Title: LLMs + Persona-Plug = Personalized LLMs

• Authors: Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, Zhicheng Dou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11901
• Pdf URL: https://arxiv.org/pdf/2409.11901
• Copy Paste: [[2409.11901]] LLMs + Persona-Plug = Personalized LLMs(https://arxiv.org/abs/2409.11901)
• Keywords: large language model
• Abstract: Personalization plays a critical role in numerous language tasks and applications, since users with the same requirements may prefer diverse outputs based on their individual interests. This has led to the development of various personalized approaches aimed at adapting large language models (LLMs) to generate customized outputs aligned with user preferences. Some of them involve fine-tuning a unique personalized LLM for each user, which is too expensive for widespread application. Alternative approaches introduce personalization information in a plug-and-play manner by retrieving the user's relevant historical texts as demonstrations. However, this retrieval-based strategy may break the continuity of the user history and fail to capture the user's overall styles and patterns, hence leading to sub-optimal performance. To address these challenges, we propose a novel personalized LLM model, \ours{}. It constructs a user-specific embedding for each individual by modeling all her historical contexts through a lightweight plug-in user embedder module. By attaching this embedding to the task input, LLMs can better understand and capture user habits and preferences, thereby producing more personalized outputs without tuning their own parameters. Extensive experiments on various tasks in the language model personalization (LaMP) benchmark demonstrate that the proposed model significantly outperforms existing personalized LLM approaches.

Title: Less Memory Means smaller GPUs: Backpropagation with Compressed Activations

• Authors: Daniel Barley, Holger Fröning
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11902
• Pdf URL: https://arxiv.org/pdf/2409.11902
• Copy Paste: [[2409.11902]] Less Memory Means smaller GPUs: Backpropagation with Compressed Activations(https://arxiv.org/abs/2409.11902)
• Keywords: large language model
• Abstract: The ever-growing scale of deep neural networks (DNNs) has lead to an equally rapid growth in computational resource requirements. Many recent architectures, most prominently Large Language Models, have to be trained using supercomputers with thousands of accelerators, such as GPUs or TPUs. Next to the vast number of floating point operations the memory footprint of DNNs is also exploding. In contrast, GPU architectures are notoriously short on memory. Even comparatively small architectures like some EfficientNet variants cannot be trained on a single consumer-grade GPU at reasonable mini-batch sizes. During training, intermediate input activations have to be stored until backpropagation for gradient calculation. These make up the vast majority of the memory footprint. In this work we therefore consider compressing activation maps for the backward pass using pooling, which can reduce both the memory footprint and amount of data movement. The forward computation remains uncompressed. We empirically show convergence and study effects on feature detection at the example of the common vision architecture ResNet. With this approach we are able to reduce the peak memory consumption by 29% at the cost of a longer training schedule, while maintaining prediction accuracy compared to an uncompressed baseline.

Title: Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation

• Authors: Dimitrios Christodoulou, Mads Kuhlmann-Jørgensen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11904
• Pdf URL: https://arxiv.org/pdf/2409.11904
• Copy Paste: [[2409.11904]] Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation(https://arxiv.org/abs/2409.11904)
• Keywords: diffusion
• Abstract: Efficiently evaluating the performance of text-to-image models is difficult as it inherently requires subjective judgment and human preference, making it hard to compare different models and quantify the state of the art. Leveraging Rapidata's technology, we present an efficient annotation framework that sources human feedback from a diverse, global pool of annotators. Our study collected over 2 million annotations across 4,512 images, evaluating four prominent models (DALL-E 3, Flux.1, MidJourney, and Stable Diffusion) on style preference, coherence, and text-to-image alignment. We demonstrate that our approach makes it feasible to comprehensively rank image generation models based on a vast pool of annotators and show that the diverse annotator demographics reflect the world population, significantly decreasing the risk of biases.

Title: LLMs in Education: Novel Perspectives, Challenges, and Opportunities

• Authors: Bashar Alhafni, Sowmya Vajjala, Stefano Bannò, Kaushal Kumar Maurya, Ekaterina Kochmar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11917
• Pdf URL: https://arxiv.org/pdf/2409.11917
• Copy Paste: [[2409.11917]] LLMs in Education: Novel Perspectives, Challenges, and Opportunities(https://arxiv.org/abs/2409.11917)
• Keywords: large language model
• Abstract: The role of large language models (LLMs) in education is an increasing area of interest today, considering the new opportunities they offer for teaching, learning, and assessment. This cutting-edge tutorial provides an overview of the educational applications of NLP and the impact that the recent advances in LLMs have had on this field. We will discuss the key challenges and opportunities presented by LLMs, grounding them in the context of four major educational applications: reading, writing, and speaking skills, and intelligent tutoring systems (ITS). This COLING 2025 tutorial is designed for researchers and practitioners interested in the educational applications of NLP and the role LLMs have to play in this area. It is the first of its kind to address this timely topic.

Title: LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Foundation Models

• Authors: Amaia Cardiel, Eloi Zablocki, Oriane Siméoni, Elias Ramzi, Matthieu Cord
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11919
• Pdf URL: https://arxiv.org/pdf/2409.11919
• Copy Paste: [[2409.11919]] LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Foundation Models(https://arxiv.org/abs/2409.11919)
• Keywords: large language model
• Abstract: Vision Language Models (VLMs) have shown impressive performances on numerous tasks but their zero-shot capabilities can be limited compared to dedicated or fine-tuned models. Yet, fine-tuning VLMs comes with limitations as it requires `white-box' access to the model's architecture and weights as well as expertise to design the fine-tuning objectives and optimize the hyper-parameters, which are specific to each VLM and downstream task. In this work, we propose LLM-wrapper, a novel approach to adapt VLMs in a `black-box' manner by leveraging large language models (LLMs) so as to reason on their outputs. We demonstrate the effectiveness of LLM-wrapper on Referring Expression Comprehension (REC), a challenging open-vocabulary task that requires spatial and semantic reasoning. Our approach significantly boosts the performance of off-the-shelf models, resulting in competitive results when compared with classic fine-tuning.

Title: Generation of Complex 3D Human Motion by Temporal and Spatial Composition of Diffusion Models

• Authors: Lorenzo Mandelli, Stefano Berretti
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11920
• Pdf URL: https://arxiv.org/pdf/2409.11920
• Copy Paste: [[2409.11920]] Generation of Complex 3D Human Motion by Temporal and Spatial Composition of Diffusion Models(https://arxiv.org/abs/2409.11920)
• Keywords: diffusion
• Abstract: In this paper, we address the challenge of generating realistic 3D human motions for action classes that were never seen during the training phase. Our approach involves decomposing complex actions into simpler movements, specifically those observed during training, by leveraging the knowledge of human motion contained in GPTs models. These simpler movements are then combined into a single, realistic animation using the properties of diffusion models. Our claim is that this decomposition and subsequent recombination of simple movements can synthesize an animation that accurately represents the complex input action. This method operates during the inference phase and can be integrated with any pre-trained diffusion model, enabling the synthesis of motion classes not present in the training data. We evaluate our method by dividing two benchmark human motion datasets into basic and complex actions, and then compare its performance against the state-of-the-art.

Title: Agglomerative Token Clustering

• Authors: Joakim Bruslund Haurum, Sergio Escalera, Graham W. Taylor, Thomas B. Moeslund
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11923
• Pdf URL: https://arxiv.org/pdf/2409.11923
• Copy Paste: [[2409.11923]] Agglomerative Token Clustering(https://arxiv.org/abs/2409.11923)
• Keywords: segmentation
• Abstract: We present Agglomerative Token Clustering (ATC), a novel token merging method that consistently outperforms previous token merging and pruning methods across image classification, image synthesis, and object detection & segmentation tasks. ATC merges clusters through bottom-up hierarchical clustering, without the introduction of extra learnable parameters. We find that ATC achieves state-of-the-art performance across all tasks, and can even perform on par with prior state-of-the-art when applied off-the-shelf, i.e. without fine-tuning. ATC is particularly effective when applied with low keep rates, where only a small fraction of tokens are kept and retaining task performance is especially difficult.

Title: An Explainable Machine Learning Approach to Traffic Accident Fatality Prediction

• Authors: Md. Asif Khan Rifat, Ahmedul Kabir, Armana Sabiha Huq
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11929
• Pdf URL: https://arxiv.org/pdf/2409.11929
• Copy Paste: [[2409.11929]] An Explainable Machine Learning Approach to Traffic Accident Fatality Prediction(https://arxiv.org/abs/2409.11929)
• Keywords: interpretability
• Abstract: Road traffic accidents (RTA) pose a significant public health threat worldwide, leading to considerable loss of life and economic burdens. This is particularly acute in developing countries like Bangladesh. Building reliable models to forecast crash outcomes is crucial for implementing effective preventive measures. To aid in developing targeted safety interventions, this study presents a machine learning-based approach for classifying fatal and non-fatal road accident outcomes using data from the Dhaka metropolitan traffic crash database from 2017 to 2022. Our framework utilizes a range of machine learning classification algorithms, comprising Logistic Regression, Support Vector Machines, Naive Bayes, Random Forest, Decision Tree, Gradient Boosting, LightGBM, and Artificial Neural Network. We prioritize model interpretability by employing the SHAP (SHapley Additive exPlanations) method, which elucidates the key factors influencing accident fatality. Our results demonstrate that LightGBM outperforms other models, achieving a ROC-AUC score of 0.72. The global, local, and feature dependency analyses are conducted to acquire deeper insights into the behavior of the model. SHAP analysis reveals that casualty class, time of accident, location, vehicle type, and road type play pivotal roles in determining fatality risk. These findings offer valuable insights for policymakers and road safety practitioners in developing countries, enabling the implementation of evidence-based strategies to reduce traffic crash fatalities.

Title: Reinforcement Learning as an Improvement Heuristic for Real-World Production Scheduling

• Authors: Arthur Müller, Lukas Vollenkemper
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11933
• Pdf URL: https://arxiv.org/pdf/2409.11933
• Copy Paste: [[2409.11933]] Reinforcement Learning as an Improvement Heuristic for Real-World Production Scheduling(https://arxiv.org/abs/2409.11933)
• Keywords: transformer
• Abstract: The integration of Reinforcement Learning (RL) with heuristic methods is an emerging trend for solving optimization problems, which leverages RL's ability to learn from the data generated during the search process. One promising approach is to train an RL agent as an improvement heuristic, starting with a suboptimal solution that is iteratively improved by applying small changes. We apply this approach to a real-world multiobjective production scheduling problem. Our approach utilizes a network architecture that includes Transformer encoding to learn the relationships between jobs. Afterwards, a probability matrix is generated from which pairs of jobs are sampled and then swapped to improve the solution. We benchmarked our approach against other heuristics using real data from our industry partner, demonstrating its superior performance.

Title: Tracking Any Point with Frame-Event Fusion Network at High Frame Rate

• Authors: Jiaxiong Liu, Bo Wang, Zhen Tan, Jinpu Zhang, Hui Shen, Dewen Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11953
• Pdf URL: https://arxiv.org/pdf/2409.11953
• Copy Paste: [[2409.11953]] Tracking Any Point with Frame-Event Fusion Network at High Frame Rate(https://arxiv.org/abs/2409.11953)
• Keywords: robust, transformer
• Abstract: Tracking any point based on image frames is constrained by frame rates, leading to instability in high-speed scenarios and limited generalization in real-world applications. To overcome these limitations, we propose an image-event fusion point tracker, FE-TAP, which combines the contextual information from image frames with the high temporal resolution of events, achieving high frame rate and robust point tracking under various challenging conditions. Specifically, we designed an Evolution Fusion module (EvoFusion) to model the image generation process guided by events. This module can effectively integrate valuable information from both modalities operating at different frequencies. To achieve smoother point trajectories, we employed a transformer-based refinement strategy that updates the point's trajectories and features iteratively. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, particularly improving expected feature age by 24$\%$ on EDS datasets. Finally, we qualitatively validated the robustness of our algorithm in real driving scenarios using our custom-designed high-resolution image-event synchronization device. Our source code will be released at this https URL.

Title: Efficacy of Synthetic Data as a Benchmark

• Authors: Gaurav Maheshwari, Dmitry Ivanov, Kevin El Haddad
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11968
• Pdf URL: https://arxiv.org/pdf/2409.11968
• Copy Paste: [[2409.11968]] Efficacy of Synthetic Data as a Benchmark(https://arxiv.org/abs/2409.11968)
• Keywords: large language model
• Abstract: Large language models (LLMs) have enabled a range of applications in zero-shot and few-shot learning settings, including the generation of synthetic datasets for training and testing. However, to reliably use these synthetic datasets, it is essential to understand how representative they are of real-world data. We investigate this by assessing the effectiveness of generating synthetic data through LLM and using it as a benchmark for various NLP tasks. Our experiments across six datasets, and three different tasks, show that while synthetic data can effectively capture performance of various methods for simpler tasks, such as intent classification, it falls short for more complex tasks like named entity recognition. Additionally, we propose a new metric called the bias factor, which evaluates the biases introduced when the same LLM is used to both generate benchmarking data and to perform the tasks. We find that smaller LLMs exhibit biases towards their own generated data, whereas larger models do not. Overall, our findings suggest that the effectiveness of synthetic data as a benchmark varies depending on the task, and that practitioners should rely on data generated from multiple larger models whenever possible.

Title: Unveiling the Black Box: Independent Functional Module Evaluation for Bird's-Eye-View Perception Model

• Authors: Ludan Zhang, Xiaokang Ding, Yuqi Dai, Lei He, Keqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11969
• Pdf URL: https://arxiv.org/pdf/2409.11969
• Copy Paste: [[2409.11969]] Unveiling the Black Box: Independent Functional Module Evaluation for Bird's-Eye-View Perception Model(https://arxiv.org/abs/2409.11969)
• Keywords: robust
• Abstract: End-to-end models are emerging as the mainstream in autonomous driving perception. However, the inability to meticulously deconstruct their internal mechanisms results in diminished development efficacy and impedes the establishment of trust. Pioneering in the issue, we present the Independent Functional Module Evaluation for Bird's-Eye-View Perception Model (BEV-IFME), a novel framework that juxtaposes the module's feature maps against Ground Truth within a unified semantic Representation Space to quantify their similarity, thereby assessing the training maturity of individual functional modules. The core of the framework lies in the process of feature map encoding and representation aligning, facilitated by our proposed two-stage Alignment AutoEncoder, which ensures the preservation of salient information and the consistency of feature structure. The metric for evaluating the training maturity of functional modules, Similarity Score, demonstrates a robust positive correlation with BEV metrics, with an average correlation coefficient of 0.9387, attesting to the framework's reliability for assessment purposes.

Title: Sampling Latent Material-Property Information From LLM-Derived Embedding Representations

• Authors: Luke P. J. Gilligan, Matteo Cobelli, Hasan M. Sayeed, Taylor D. Sparks, Stefano Sanvito
• Subjects: cs.CL, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2409.11971
• Pdf URL: https://arxiv.org/pdf/2409.11971
• Copy Paste: [[2409.11971]] Sampling Latent Material-Property Information From LLM-Derived Embedding Representations(https://arxiv.org/abs/2409.11971)
• Keywords: large language model
• Abstract: Vector embeddings derived from large language models (LLMs) show promise in capturing latent information from the literature. Interestingly, these can be integrated into material embeddings, potentially useful for data-driven predictions of materials properties. We investigate the extent to which LLM-derived vectors capture the desired information and their potential to provide insights into material properties without additional training. Our findings indicate that, although LLMs can be used to generate representations reflecting certain property information, extracting the embeddings requires identifying the optimal contextual clues and appropriate comparators. Despite this restriction, it appears that LLMs still have the potential to be useful in generating meaningful materials-science representations.

Title: MitoSeg: Mitochondria Segmentation Tool

• Authors: Faris Serdar Taşel, Efe Çiftci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11974
• Pdf URL: https://arxiv.org/pdf/2409.11974
• Copy Paste: [[2409.11974]] MitoSeg: Mitochondria Segmentation Tool(https://arxiv.org/abs/2409.11974)
• Keywords: generative, segmentation
• Abstract: Recent studies suggest a potential link between the physical structure of mitochondria and neurodegenerative diseases. With advances in Electron Microscopy techniques, it has become possible to visualize the boundary and internal membrane structures of mitochondria in detail. It is crucial to automatically segment mitochondria from these images to investigate the relationship between mitochondria and diseases. In this paper, we present a software solution for mitochondrial segmentation, highlighting mitochondria boundaries in electron microscopy tomography images and generating corresponding 3D meshes.

Title: Panoptic-Depth Forecasting

• Authors: Juana Valeria Hurtado, Riya Mohan, Abhinav Valada
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.12008
• Pdf URL: https://arxiv.org/pdf/2409.12008
• Copy Paste: [[2409.12008]] Panoptic-Depth Forecasting(https://arxiv.org/abs/2409.12008)
• Keywords: transformer, segmentation
• Abstract: Forecasting the semantics and 3D structure of scenes is essential for robots to navigate and plan actions safely. Recent methods have explored semantic and panoptic scene forecasting; however, they do not consider the geometry of the scene. In this work, we propose the panoptic-depth forecasting task for jointly predicting the panoptic segmentation and depth maps of unobserved future frames, from monocular camera images. To facilitate this work, we extend the popular KITTI-360 and Cityscapes benchmarks by computing depth maps from LiDAR point clouds and leveraging sequential labeled data. We also introduce a suitable evaluation metric that quantifies both the panoptic quality and depth estimation accuracy of forecasts in a coherent manner. Furthermore, we present two baselines and propose the novel PDcast architecture that learns rich spatio-temporal representations by incorporating a transformer-based encoder, a forecasting module, and task-specific decoders to predict future panoptic-depth outputs. Extensive evaluations demonstrate the effectiveness of PDcast across two datasets and three forecasting tasks, consistently addressing the primary challenges. We make the code publicly available at this https URL.

Title: ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation

• Authors: Peiyu Li, Xiaobao Huang, Yijun Tian, Nitesh V. Chawla
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12010
• Pdf URL: https://arxiv.org/pdf/2409.12010
• Copy Paste: [[2409.12010]] ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation(https://arxiv.org/abs/2409.12010)
• Keywords: large language model
• Abstract: Significant work has been conducted in the domain of food computing, yet these studies typically focus on single tasks such as t2t (instruction generation from food titles and ingredients), i2t (recipe generation from food images), or t2i (food image generation from recipes). None of these approaches integrate all modalities simultaneously. To address this gap, we introduce a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, it2t, and t2ti. By leveraging large language models (LLMs) and pre-trained image encoder and decoder models, our model can perform a diverse array of food computing-related tasks, including food understanding, food recognition, recipe generation, and food image generation. Compared to previous models, our foundation model demonstrates a significantly broader range of capabilities and exhibits superior performance, particularly in food image generation and recipe generation tasks. We open-sourced ChefFusion at GitHub.

Title: LEMON: Localized Editing with Mesh Optimization and Neural Shaders

• Authors: Furkan Mert Algan, Umut Yazgan, Driton Salihu, Cem Eteke, Eckehard Steinbach
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12024
• Pdf URL: https://arxiv.org/pdf/2409.12024
• Copy Paste: [[2409.12024]] LEMON: Localized Editing with Mesh Optimization and Neural Shaders(https://arxiv.org/abs/2409.12024)
• Keywords: diffusion, segmentation
• Abstract: In practical use cases, polygonal mesh editing can be faster than generating new ones, but it can still be challenging and time-consuming for users. Existing solutions for this problem tend to focus on a single task, either geometry or novel view synthesis, which often leads to disjointed results between the mesh and view. In this work, we propose LEMON, a mesh editing pipeline that combines neural deferred shading with localized mesh optimization. Our approach begins by identifying the most important vertices in the mesh for editing, utilizing a segmentation model to focus on these key regions. Given multi-view images of an object, we optimize a neural shader and a polygonal mesh while extracting the normal map and the rendered image from each view. By using these outputs as conditioning data, we edit the input images with a text-to-image diffusion model and iteratively update our dataset while deforming the mesh. This process results in a polygonal mesh that is edited according to the given text instruction, preserving the geometric characteristics of the initial mesh while focusing on the most significant areas. We evaluate our pipeline using the DTU dataset, demonstrating that it generates finely-edited meshes more rapidly than the current state-of-the-art methods. We include our code and additional results in the supplementary material.

Title: On Vision Transformers for Classification Tasks in Side-Scan Sonar Imagery

• Authors: BW Sheffield, Jeffrey Ellen, Ben Whitmore
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12026
• Pdf URL: https://arxiv.org/pdf/2409.12026
• Copy Paste: [[2409.12026]] On Vision Transformers for Classification Tasks in Side-Scan Sonar Imagery(https://arxiv.org/abs/2409.12026)
• Keywords: transformer
• Abstract: Side-scan sonar (SSS) imagery presents unique challenges in the classification of man-made objects on the seafloor due to the complex and varied underwater environments. Historically, experts have manually interpreted SSS images, relying on conventional machine learning techniques with hand-crafted features. While Convolutional Neural Networks (CNNs) significantly advanced automated classification in this domain, they often fall short when dealing with diverse seafloor textures, such as rocky or ripple sand bottoms, where false positive rates may increase. Recently, Vision Transformers (ViTs) have shown potential in addressing these limitations by utilizing a self-attention mechanism to capture global information in image patches, offering more flexibility in processing spatial hierarchies. This paper rigorously compares the performance of ViT models alongside commonly used CNN architectures, such as ResNet and ConvNext, for binary classification tasks in SSS imagery. The dataset encompasses diverse geographical seafloor types and is balanced between the presence and absence of man-made objects. ViT-based models exhibit superior classification performance across f1-score, precision, recall, and accuracy metrics, although at the cost of greater computational resources. CNNs, with their inductive biases, demonstrate better computational efficiency, making them suitable for deployment in resource-constrained environments like underwater vehicles. Future research directions include exploring self-supervised learning for ViTs and multi-modal fusion to further enhance performance in challenging underwater environments.

Title: PhysMamba: Efficient Remote Physiological Measurement with SlowFast Temporal Difference Mamba

• Authors: Chaoqi Luo, Yiping Xie, Zitong Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12031
• Pdf URL: https://arxiv.org/pdf/2409.12031
• Copy Paste: [[2409.12031]] PhysMamba: Efficient Remote Physiological Measurement with SlowFast Temporal Difference Mamba(https://arxiv.org/abs/2409.12031)
• Keywords: transformer
• Abstract: Facial-video based Remote photoplethysmography (rPPG) aims at measuring physiological signals and monitoring heart activity without any contact, showing significant potential in various applications. Previous deep learning based rPPG measurement are primarily based on CNNs and Transformers. However, the limited receptive fields of CNNs restrict their ability to capture long-range spatio-temporal dependencies, while Transformers also struggle with modeling long video sequences with high complexity. Recently, the state space models (SSMs) represented by Mamba are known for their impressive performance on capturing long-range dependencies from long sequences. In this paper, we propose the PhysMamba, a Mamba-based framework, to efficiently represent long-range physiological dependencies from facial videos. Specifically, we introduce the Temporal Difference Mamba block to first enhance local dynamic differences and further model the long-range spatio-temporal context. Moreover, a dual-stream SlowFast architecture is utilized to fuse the multi-scale temporal features. Extensive experiments are conducted on three benchmark datasets to demonstrate the superiority and efficiency of PhysMamba. The codes are available at this https URL

Title: SFDA-rPPG: Source-Free Domain Adaptive Remote Physiological Measurement with Spatio-Temporal Consistency

• Authors: Yiping Xie, Zitong Yu, Bingjie Wu, Weicheng Xie, Linlin Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12040
• Pdf URL: https://arxiv.org/pdf/2409.12040
• Copy Paste: [[2409.12040]] SFDA-rPPG: Source-Free Domain Adaptive Remote Physiological Measurement with Spatio-Temporal Consistency(https://arxiv.org/abs/2409.12040)
• Keywords: privacy
• Abstract: Remote Photoplethysmography (rPPG) is a non-contact method that uses facial video to predict changes in blood volume, enabling physiological metrics measurement. Traditional rPPG models often struggle with poor generalization capacity in unseen domains. Current solutions to this problem is to improve its generalization in the target domain through Domain Generalization (DG) or Domain Adaptation (DA). However, both traditional methods require access to both source domain data and target domain data, which cannot be implemented in scenarios with limited access to source data, and another issue is the privacy of accessing source domain data. In this paper, we propose the first Source-free Domain Adaptation benchmark for rPPG measurement (SFDA-rPPG), which overcomes these limitations by enabling effective domain adaptation without access to source domain data. Our framework incorporates a Three-Branch Spatio-Temporal Consistency Network (TSTC-Net) to enhance feature consistency across domains. Furthermore, we propose a new rPPG distribution alignment loss based on the Frequency-domain Wasserstein Distance (FWD), which leverages optimal transport to align power spectrum distributions across domains effectively and further enforces the alignment of the three branches. Extensive cross-domain experiments and ablation studies demonstrate the effectiveness of our proposed method in source-free domain adaptation settings. Our findings highlight the significant contribution of the proposed FWD loss for distributional alignment, providing a valuable reference for future research and applications. The source code is available at this https URL

Title: Using Large Language Models to Generate Clinical Trial Tables and Figures

• Authors: Yumeng Yang, Peter Krusche, Kristyn Pantoja, Cheng Shi, Ethan Ludmir, Kirk Roberts, Gen Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12046
• Pdf URL: https://arxiv.org/pdf/2409.12046
• Copy Paste: [[2409.12046]] Using Large Language Models to Generate Clinical Trial Tables and Figures(https://arxiv.org/abs/2409.12046)
• Keywords: large language model
• Abstract: Tables, figures, and listings (TFLs) are essential tools for summarizing clinical trial data. Creation of TFLs for reporting activities is often a time-consuming task encountered routinely during the execution of clinical trials. This study explored the use of large language models (LLMs) to automate the generation of TFLs through prompt engineering and few-shot transfer learning. Using public clinical trial data in ADaM format, our results demonstrated that LLMs can efficiently generate TFLs with prompt instructions, showcasing their potential in this domain. Furthermore, we developed a conservational agent named Clinical Trial TFL Generation Agent: An app that matches user queries to predefined prompts that produce customized programs to generate specific predefined TFLs.

Title: A Survey-Based Quantitative Analysis of Stress Factors and Their Impacts Among Cybersecurity Professionals

• Authors: Sunil Arora, John D. Hastings
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.12047
• Pdf URL: https://arxiv.org/pdf/2409.12047
• Copy Paste: [[2409.12047]] A Survey-Based Quantitative Analysis of Stress Factors and Their Impacts Among Cybersecurity Professionals(https://arxiv.org/abs/2409.12047)
• Keywords: security, protect
• Abstract: This study investigates the prevalence and underlying causes of work-related stress and burnout among cybersecurity professionals using a quantitative survey approach guided by the Job Demands-Resources model. Analysis of responses from 50 cybersecurity practitioners reveals an alarming reality: 44% report experiencing severe work-related stress and burnout, while an additional 28% are uncertain about their condition. The demanding nature of cybersecurity roles, unrealistic expectations, and unsupportive organizational cultures emerge as primary factors fueling this crisis. Notably, 66% of respondents perceive cybersecurity jobs as more stressful than other IT positions, with 84% facing additional challenges due to the pandemic and recent high-profile breaches. The study finds that most cybersecurity experts are reluctant to report their struggles to management, perpetuating a cycle of silence and neglect. To address this critical issue, the paper recommends that organizations foster supportive work environments, implement mindfulness programs, and address systemic challenges. By prioritizing the mental health of cybersecurity professionals, organizations can cultivate a more resilient and effective workforce to protect against an ever-evolving threat landscape.

Title: Artemis: Efficient Commit-and-Prove SNARKs for zkML

• Authors: Hidde Lycklama, Alexander Viand, Nikolay Avramov, Nicolas Küchler, Anwar Hithnawi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.12055
• Pdf URL: https://arxiv.org/pdf/2409.12055
• Copy Paste: [[2409.12055]] Artemis: Efficient Commit-and-Prove SNARKs for zkML(https://arxiv.org/abs/2409.12055)
• Keywords: privacy
• Abstract: The widespread adoption of machine learning (ML) in various critical applications, from healthcare to autonomous systems, has raised significant concerns about privacy, accountability, and trustworthiness. To address these concerns, recent research has focused on developing zero-knowledge machine learning (zkML) techniques that enable the verification of various aspects of ML models without revealing sensitive information. Recent advances in zkML have substantially improved efficiency; however, these efforts have primarily optimized the process of proving ML computations correct, often overlooking the substantial overhead associated with verifying the necessary commitments to the model and data. To address this gap, this paper introduces two new Commit-and-Prove SNARK (CP-SNARK) constructions (Apollo and Artemis) that effectively address the emerging challenge of commitment verification in zkML pipelines. Apollo operates on KZG commitments and requires white-box use of the underlying proof system, whereas Artemis is compatible with any homomorphic polynomial commitment and only makes black-box use of the proof system. As a result, Artemis is compatible with state-of-the-art proof systems without trusted setup. We present the first implementation of these CP-SNARKs, evaluate their performance on a diverse set of ML models, and show substantial improvements over existing methods, achieving significant reductions in prover costs and maintaining efficiency even for large-scale models. For example, for the VGG model, we reduce the overhead associated with commitment checks from 11.5x to 1.2x. Our results suggest that these contributions can move zkML towards practical deployment, particularly in scenarios involving large and complex ML models.

Title: Dual-Layer Training and Decoding of Large Language Model with Simultaneously Thinking and Speaking

• Authors: Ningyuan Xi, Xiaoyu Wang, Yetao Wu, Teng Chen, Qingqing Gu, Jinxian Qu, Zhonglin Jiang, Yong Chen, Luo Ji
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12059
• Pdf URL: https://arxiv.org/pdf/2409.12059
• Copy Paste: [[2409.12059]] Dual-Layer Training and Decoding of Large Language Model with Simultaneously Thinking and Speaking(https://arxiv.org/abs/2409.12059)
• Keywords: large language model
• Abstract: Large Language Model can reasonably understand and generate human expressions but may lack of thorough thinking and reasoning mechanisms. Recently there have been several studies which enhance the thinking ability of language models but most of them are not data-driven or training-based. In this paper, we are motivated by the cognitive mechanism in the natural world, and design a novel model architecture called TaS which allows it to first consider the thoughts and then express the response based upon the query. We design several pipelines to annotate or generate the thought contents from prompt-response samples, then add language heads in a middle layer which behaves as the thinking layer. We train the language model by the thoughts-augmented data and successfully let the thinking layer automatically generate reasonable thoughts and finally output more reasonable responses. Both qualitative examples and quantitative results validate the effectiveness and performance of TaS. Our code is available at https://anonymous.4open.science/r/TadE.

Title: PAD-FT: A Lightweight Defense for Backdoor Attacks via Data Purification and Fine-Tuning

• Authors: Yukai Xu, Yujie Gu, Kouichi Sakurai
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12072
• Pdf URL: https://arxiv.org/pdf/2409.12072
• Copy Paste: [[2409.12072]] PAD-FT: A Lightweight Defense for Backdoor Attacks via Data Purification and Fine-Tuning(https://arxiv.org/abs/2409.12072)
• Keywords: defense, attack
• Abstract: Backdoor attacks pose a significant threat to deep neural networks, particularly as recent advancements have led to increasingly subtle implantation, making the defense more challenging. Existing defense mechanisms typically rely on an additional clean dataset as a standard reference and involve retraining an auxiliary model or fine-tuning the entire victim model. However, these approaches are often computationally expensive and not always feasible in practical applications. In this paper, we propose a novel and lightweight defense mechanism, termed PAD-FT, that does not require an additional clean dataset and fine-tunes only a very small part of the model to disinfect the victim model. To achieve this, our approach first introduces a simple data purification process to identify and select the most-likely clean data from the poisoned training dataset. The self-purified clean dataset is then used for activation clipping and fine-tuning only the last classification layer of the victim model. By integrating data purification, activation clipping, and classifier fine-tuning, our mechanism PAD-FT demonstrates superior effectiveness across multiple backdoor attack methods and datasets, as confirmed through extensive experimental evaluation.

Title: Unsupervised Domain Adaptation Via Data Pruning

• Authors: Andrea Napoli, Paul White
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12076
• Pdf URL: https://arxiv.org/pdf/2409.12076
• Copy Paste: [[2409.12076]] Unsupervised Domain Adaptation Via Data Pruning(https://arxiv.org/abs/2409.12076)
• Keywords: robust
• Abstract: The removal of carefully-selected examples from training data has recently emerged as an effective way of improving the robustness of machine learning models. However, the best way to select these examples remains an open question. In this paper, we consider the problem from the perspective of unsupervised domain adaptation (UDA). We propose AdaPrune, a method for UDA whereby training examples are removed to attempt to align the training distribution to that of the target data. By adopting the maximum mean discrepancy (MMD) as the criterion for alignment, the problem can be neatly formulated and solved as an integer quadratic program. We evaluate our approach on a real-world domain shift task of bioacoustic event detection. As a method for UDA, we show that AdaPrune outperforms related techniques, and is complementary to other UDA algorithms such as CORAL. Our analysis of the relationship between the MMD and model accuracy, along with t-SNE plots, validate the proposed method as a principled and well-founded way of performing data pruning.

Title: Towards Interpretable End-Stage Renal Disease (ESRD) Prediction: Utilizing Administrative Claims Data with Explainable AI Techniques

• Authors: Yubo Li, Saba Al-Sayouri, Rema Padman
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12087
• Pdf URL: https://arxiv.org/pdf/2409.12087
• Copy Paste: [[2409.12087]] Towards Interpretable End-Stage Renal Disease (ESRD) Prediction: Utilizing Administrative Claims Data with Explainable AI Techniques(https://arxiv.org/abs/2409.12087)
• Keywords: interpretability
• Abstract: This study explores the potential of utilizing administrative claims data, combined with advanced machine learning and deep learning techniques, to predict the progression of Chronic Kidney Disease (CKD) to End-Stage Renal Disease (ESRD). We analyze a comprehensive, 10-year dataset provided by a major health insurance organization to develop prediction models for multiple observation windows using traditional machine learning methods such as Random Forest and XGBoost as well as deep learning approaches such as Long Short-Term Memory (LSTM) networks. Our findings demonstrate that the LSTM model, particularly with a 24-month observation window, exhibits superior performance in predicting ESRD progression, outperforming existing models in the literature. We further apply SHapley Additive exPlanations (SHAP) analysis to enhance interpretability, providing insights into the impact of individual features on predictions at the individual patient level. This study underscores the value of leveraging administrative claims data for CKD management and predicting ESRD progression.

Title: Skill matching at scale: freelancer-project alignment for efficient multilingual candidate retrieval

• Authors: Warren Jouanneau, Marc Palyart, Emma Jouffroy
• Subjects: cs.CL, cs.IR, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.12097
• Pdf URL: https://arxiv.org/pdf/2409.12097
• Copy Paste: [[2409.12097]] Skill matching at scale: freelancer-project alignment for efficient multilingual candidate retrieval(https://arxiv.org/abs/2409.12097)
• Keywords: transformer
• Abstract: Finding the perfect match between a job proposal and a set of freelancers is not an easy task to perform at scale, especially in multiple languages. In this paper, we propose a novel neural retriever architecture that tackles this problem in a multilingual setting. Our method encodes project descriptions and freelancer profiles by leveraging pre-trained multilingual language models. The latter are used as backbone for a custom transformer architecture that aims to keep the structure of the profiles and project. This model is trained with a contrastive loss on historical data. Thanks to several experiments, we show that this approach effectively captures skill matching similarity and facilitates efficient matching, outperforming traditional methods.

Title: Brain-Streams: fMRI-to-Image Reconstruction with Multi-modal Guidance

• Authors: Jaehoon Joo, Taejin Jeong, Seongjae Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12099
• Pdf URL: https://arxiv.org/pdf/2409.12099
• Copy Paste: [[2409.12099]] Brain-Streams: fMRI-to-Image Reconstruction with Multi-modal Guidance(https://arxiv.org/abs/2409.12099)
• Keywords: diffusion, generative
• Abstract: Understanding how humans process visual information is one of the crucial steps for unraveling the underlying mechanism of brain activity. Recently, this curiosity has motivated the fMRI-to-image reconstruction task; given the fMRI data from visual stimuli, it aims to reconstruct the corresponding visual stimuli. Surprisingly, leveraging powerful generative models such as the Latent Diffusion Model (LDM) has shown promising results in reconstructing complex visual stimuli such as high-resolution natural images from vision datasets. Despite the impressive structural fidelity of these reconstructions, they often lack details of small objects, ambiguous shapes, and semantic nuances. Consequently, the incorporation of additional semantic knowledge, beyond mere visuals, becomes imperative. In light of this, we exploit how modern LDMs effectively incorporate multi-modal guidance (text guidance, visual guidance, and image layout) for structurally and semantically plausible image generations. Specifically, inspired by the two-streams hypothesis suggesting that perceptual and semantic information are processed in different brain regions, our framework, Brain-Streams, maps fMRI signals from these brain regions to appropriate embeddings. That is, by extracting textual guidance from semantic information regions and visual guidance from perceptual information regions, Brain-Streams provides accurate multi-modal guidance to LDMs. We validate the reconstruction ability of Brain-Streams both quantitatively and qualitatively on a real fMRI dataset comprising natural image stimuli and fMRI data.

Title: Symmetry-Enriched Learning: A Category-Theoretic Framework for Robust Machine Learning Models

• Authors: Ronald Katende
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12100
• Pdf URL: https://arxiv.org/pdf/2409.12100
• Copy Paste: [[2409.12100]] Symmetry-Enriched Learning: A Category-Theoretic Framework for Robust Machine Learning Models(https://arxiv.org/abs/2409.12100)
• Keywords: robust
• Abstract: This manuscript presents a novel framework that integrates higher-order symmetries and category theory into machine learning. We introduce new mathematical constructs, including hyper-symmetry categories and functorial representations, to model complex transformations within learning algorithms. Our contributions include the design of symmetry-enriched learning models, the development of advanced optimization techniques leveraging categorical symmetries, and the theoretical analysis of their implications for model robustness, generalization, and convergence. Through rigorous proofs and practical applications, we demonstrate that incorporating higher-dimensional categorical structures enhances both the theoretical foundations and practical capabilities of modern machine learning algorithms, opening new directions for research and innovation.

Title: FedLF: Adaptive Logit Adjustment and Feature Optimization in Federated Long-Tailed Learning

• Authors: Xiuhua Lu, Peng Li, Xuefeng Jiang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12105
• Pdf URL: https://arxiv.org/pdf/2409.12105
• Copy Paste: [[2409.12105]] FedLF: Adaptive Logit Adjustment and Feature Optimization in Federated Long-Tailed Learning(https://arxiv.org/abs/2409.12105)
• Keywords: privacy, federate
• Abstract: Federated learning offers a paradigm to the challenge of preserving privacy in distributed machine learning. However, datasets distributed across each client in the real world are inevitably heterogeneous, and if the datasets can be globally aggregated, they tend to be long-tailed distributed, which greatly affects the performance of the model. The traditional approach to federated learning primarily addresses the heterogeneity of data among clients, yet it fails to address the phenomenon of class-wise bias in global long-tailed data. This results in the trained model focusing on the head classes while neglecting the equally important tail classes. Consequently, it is essential to develop a methodology that considers classes holistically. To address the above problems, we propose a new method FedLF, which introduces three modifications in the local training phase: adaptive logit adjustment, continuous class centred optimization, and feature decorrelation. We compare seven state-of-the-art methods with varying degrees of data heterogeneity and long-tailed distribution. Extensive experiments on benchmark datasets CIFAR-10-LT and CIFAR-100-LT demonstrate that our approach effectively mitigates the problem of model performance degradation due to data heterogeneity and long-tailed distribution. our code is available at this https URL.

Title: Measuring Human and AI Values based on Generative Psychometrics with Large Language Models

• Authors: Haoran Ye, Yuhang Xie, Yuanyi Ren, Hanjun Fang, Xin Zhang, Guojie Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12106
• Pdf URL: https://arxiv.org/pdf/2409.12106
• Copy Paste: [[2409.12106]] Measuring Human and AI Values based on Generative Psychometrics with Large Language Models(https://arxiv.org/abs/2409.12106)
• Keywords: generative, large language model
• Abstract: Human values and their measurement are long-standing interdisciplinary inquiry. Recent advances in AI have sparked renewed interest in this area, with large language models (LLMs) emerging as both tools and subjects of value measurement. This work introduces Generative Psychometrics for Values (GPV), an LLM-based, data-driven value measurement paradigm, theoretically grounded in text-revealed selective perceptions. We begin by fine-tuning an LLM for accurate perception-level value measurement and verifying the capability of LLMs to parse texts into perceptions, forming the core of the GPV pipeline. Applying GPV to human-authored blogs, we demonstrate its stability, validity, and superiority over prior psychological tools. Then, extending GPV to LLM value measurement, we advance the current art with 1) a psychometric methodology that measures LLM values based on their scalable and free-form outputs, enabling context-specific measurement; 2) a comparative analysis of measurement paradigms, indicating response biases of prior methods; and 3) an attempt to bridge LLM values and their safety, revealing the predictive power of different value systems and the impacts of various values on LLM safety. Through interdisciplinary efforts, we aim to leverage AI for next-generation psychometrics and psychometrics for value-aligned AI.

Title: SPRMamba: Surgical Phase Recognition for Endoscopic Submucosal Dissection with Mamba

• Authors: Xiangning Zhang, Jinnan Chen, Qingwei Zhang, Chengfeng Zhou, Zhengjie Zhang, Xiaobo Li, Dahong Qian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12108
• Pdf URL: https://arxiv.org/pdf/2409.12108
• Copy Paste: [[2409.12108]] SPRMamba: Surgical Phase Recognition for Endoscopic Submucosal Dissection with Mamba(https://arxiv.org/abs/2409.12108)
• Keywords: robust, transformer
• Abstract: Endoscopic Submucosal Dissection (ESD) is a minimally invasive procedure initially designed for the treatment of early gastric cancer but is now widely used for various gastrointestinal lesions. Computer-assisted Surgery systems have played a crucial role in improving the precision and safety of ESD procedures, however, their effectiveness is limited by the accurate recognition of surgical phases. The intricate nature of ESD, with different lesion characteristics and tissue structures, presents challenges for real-time surgical phase recognition algorithms. Existing surgical phase recognition algorithms struggle to efficiently capture temporal contexts in video-based scenarios, leading to insufficient performance. To address these issues, we propose SPRMamba, a novel Mamba-based framework for ESD surgical phase recognition. SPRMamba leverages the strengths of Mamba for long-term temporal modeling while introducing the Scaled Residual TranMamba block to enhance the capture of fine-grained details, overcoming the limitations of traditional temporal models like Temporal Convolutional Networks and Transformers. Moreover, a Temporal Sample Strategy is introduced to accelerate the processing, which is essential for real-time phase recognition in clinical settings. Extensive testing on the ESD385 dataset and the cholecystectomy Cholec80 dataset demonstrates that SPRMamba surpasses existing state-of-the-art methods and exhibits greater robustness across various surgical phase recognition tasks.

Title: Applications of Knowledge Distillation in Remote Sensing: A Survey

• Authors: Yassine Himeur, Nour Aburaed, Omar Elharrouss, Iraklis Varlamis, Shadi Atalla, Wathiq Mansoor, Hussain Al Ahmad
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12111
• Pdf URL: https://arxiv.org/pdf/2409.12111
• Copy Paste: [[2409.12111]] Applications of Knowledge Distillation in Remote Sensing: A Survey(https://arxiv.org/abs/2409.12111)
• Keywords: segmentation
• Abstract: With the ever-growing complexity of models in the field of remote sensing (RS), there is an increasing demand for solutions that balance model accuracy with computational efficiency. Knowledge distillation (KD) has emerged as a powerful tool to meet this need, enabling the transfer of knowledge from large, complex models to smaller, more efficient ones without significant loss in performance. This review article provides an extensive examination of KD and its innovative applications in RS. KD, a technique developed to transfer knowledge from a complex, often cumbersome model (teacher) to a more compact and efficient model (student), has seen significant evolution and application across various domains. Initially, we introduce the fundamental concepts and historical progression of KD methods. The advantages of employing KD are highlighted, particularly in terms of model compression, enhanced computational efficiency, and improved performance, which are pivotal for practical deployments in RS scenarios. The article provides a comprehensive taxonomy of KD techniques, where each category is critically analyzed to demonstrate the breadth and depth of the alternative options, and illustrates specific case studies that showcase the practical implementation of KD methods in RS tasks, such as instance segmentation and object detection. Further, the review discusses the challenges and limitations of KD in RS, including practical constraints and prospective future directions, providing a comprehensive overview for researchers and practitioners in the field of RS. Through this organization, the paper not only elucidates the current state of research in KD but also sets the stage for future research opportunities, thereby contributing significantly to both academic research and real-world applications.

Title: Pareto Data Framework: Steps Towards Resource-Efficient Decision Making Using Minimum Viable Data (MVD)

• Authors: Tashfain Ahmed, Josh Siegel
• Subjects: cs.LG, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.12112
• Pdf URL: https://arxiv.org/pdf/2409.12112
• Copy Paste: [[2409.12112]] Pareto Data Framework: Steps Towards Resource-Efficient Decision Making Using Minimum Viable Data (MVD)(https://arxiv.org/abs/2409.12112)
• Keywords: extraction
• Abstract: This paper introduces the Pareto Data Framework, an approach for identifying and selecting the Minimum Viable Data (MVD) required for enabling machine learning applications on constrained platforms such as embedded systems, mobile devices, and Internet of Things (IoT) devices. We demonstrate that strategic data reduction can maintain high performance while significantly reducing bandwidth, energy, computation, and storage costs. The framework identifies Minimum Viable Data (MVD) to optimize efficiency across resource-constrained environments without sacrificing performance. It addresses common inefficient practices in an IoT application such as overprovisioning of sensors and overprecision, and oversampling of signals, proposing scalable solutions for optimal sensor selection, signal extraction and transmission, and data representation. An experimental methodology demonstrates effective acoustic data characterization after downsampling, quantization, and truncation to simulate reduced-fidelity sensors and network and storage constraints; results shows that performance can be maintained up to 95\% with sample rates reduced by 75\% and bit depths and clip length reduced by 50\% which translates into substantial cost and resource reduction. These findings have implications on the design and development of constrained systems. The paper also discusses broader implications of the framework, including the potential to democratize advanced AI technologies across IoT applications and sectors such as agriculture, transportation, and manufacturing to improve access and multiply the benefits of data-driven insights.

Title: Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement

• Authors: An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, Zhenru Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12122
• Pdf URL: https://arxiv.org/pdf/2409.12122
• Copy Paste: [[2409.12122]] Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement(https://arxiv.org/abs/2409.12122)
• Keywords: large language model
• Abstract: In this report, we present a series of math-specific large language models: Qwen2.5-Math and Qwen2.5-Math-Instruct-1.5B/7B/72B. The core innovation of the Qwen2.5 series lies in integrating the philosophy of self-improvement throughout the entire pipeline, from pre-training and post-training to inference: (1) During the pre-training phase, Qwen2-Math-Instruct is utilized to generate large-scale, high-quality mathematical data. (2) In the post-training phase, we develop a reward model (RM) by conducting massive sampling from Qwen2-Math-Instruct. This RM is then applied to the iterative evolution of data in supervised fine-tuning (SFT). With a stronger SFT model, it's possible to iteratively train and update the RM, which in turn guides the next round of SFT data iteration. On the final SFT model, we employ the ultimate RM for reinforcement learning, resulting in the Qwen2.5-Math-Instruct. (3) Furthermore, during the inference stage, the RM is used to guide sampling, optimizing the model's performance. Qwen2.5-Math-Instruct supports both Chinese and English, and possess advanced mathematical reasoning capabilities, including Chain-of-Thought (CoT) and Tool-Integrated Reasoning (TIR). We evaluate our models on 10 mathematics datasets in both English and Chinese, such as GSM8K, MATH, GaoKao, AMC23, and AIME24, covering a range of difficulties from grade school level to math competition problems.

Title: MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion

• Authors: Kalakonda Sai Shashank, Shubh Maheshwari, Ravi Kiran Sarvadevabhatla
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.12140
• Pdf URL: https://arxiv.org/pdf/2409.12140
• Copy Paste: [[2409.12140]] MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion(https://arxiv.org/abs/2409.12140)
• Keywords: diffusion, large language model
• Abstract: We introduce MoRAG, a novel multi-part fusion based retrieval-augmented generation strategy for text-based human motion generation. The method enhances motion diffusion models by leveraging additional knowledge obtained through an improved motion retrieval process. By effectively prompting large language models (LLMs), we address spelling errors and rephrasing issues in motion retrieval. Our approach utilizes a multi-part retrieval strategy to improve the generalizability of motion retrieval across the language space. We create diverse samples through the spatial composition of the retrieved motions. Furthermore, by utilizing low-level, part-specific motion information, we can construct motion samples for unseen text descriptions. Our experiments demonstrate that our framework can serve as a plug-and-play module, improving the performance of motion diffusion models. Code, pretrained models and sample videos will be made available at: this https URL

Title: MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning

• Authors: Justin Chih-Yao Chen, Archiki Prasad, Swarnadeep Saha, Elias Stengel-Eskin, Mohit Bansal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.12147
• Pdf URL: https://arxiv.org/pdf/2409.12147
• Copy Paste: [[2409.12147]] MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning(https://arxiv.org/abs/2409.12147)
• Keywords: large language model
• Abstract: Large Language Models' (LLM) reasoning can be improved using test-time aggregation strategies, i.e., generating multiple samples and voting among generated samples. While these improve performance, they often reach a saturation point. Refinement offers an alternative by using LLM-generated feedback to improve solution quality. However, refinement introduces 3 key challenges: (1) Excessive refinement: Uniformly refining all instances can over-correct and reduce the overall performance. (2) Inability to localize and address errors: LLMs have a limited ability to self-correct and struggle to identify and correct their own mistakes. (3) Insufficient refinement: Deciding how many iterations of refinement are needed is non-trivial, and stopping too soon could leave errors unaddressed. To tackle these issues, we propose MAgICoRe, which avoids excessive refinement by categorizing problem difficulty as easy or hard, solving easy problems with coarse-grained aggregation and hard ones with fine-grained and iterative multi-agent refinement. To improve error localization, we incorporate external step-wise reward model (RM) scores. Moreover, to ensure effective refinement, we employ a multi-agent loop with three agents: Solver, Reviewer (which generates targeted feedback based on step-wise RM scores), and the Refiner (which incorporates feedback). To ensure sufficient refinement, we re-evaluate updated solutions, iteratively initiating further rounds of refinement. We evaluate MAgICoRe on Llama-3-8B and GPT-3.5 and show its effectiveness across 5 math datasets. Even one iteration of MAgICoRe beats Self-Consistency by 3.4%, Best-of-k by 3.2%, and Self-Refine by 4.0% while using less than half the samples. Unlike iterative refinement with baselines, MAgICoRe continues to improve with more iterations. Finally, our ablations highlight the importance of MAgICoRe's RMs and multi-agent communication.

Title: JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation

• Authors: Sai Tanmay Reddy Chakkera, Aggelina Chatziagapi, Dimitris Samaras
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12156
• Pdf URL: https://arxiv.org/pdf/2409.12156
• Copy Paste: [[2409.12156]] JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation(https://arxiv.org/abs/2409.12156)
• Keywords: transformer
• Abstract: We introduce a novel method for joint expression and audio-guided talking face generation. Recent approaches either struggle to preserve the speaker identity or fail to produce faithful facial expressions. To address these challenges, we propose a NeRF-based network. Since we train our network on monocular videos without any ground truth, it is essential to learn disentangled representations for audio and expression. We first learn audio features in a self-supervised manner, given utterances from multiple subjects. By incorporating a contrastive learning technique, we ensure that the learned audio features are aligned to the lip motion and disentangled from the muscle motion of the rest of the face. We then devise a transformer-based architecture that learns expression features, capturing long-range facial expressions and disentangling them from the speech-specific mouth movements. Through quantitative and qualitative evaluation, we demonstrate that our method can synthesize high-fidelity talking face videos, achieving state-of-the-art facial expression transfer along with lip synchronization to unseen audio.

Title: LogoRA: Local-Global Representation Alignment for Robust Time Series Classification

• Authors: Huanyu Zhang, Yi-Fan Zhang, Zhang Zhang, Qingsong Wen, Liang Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12169
• Pdf URL: https://arxiv.org/pdf/2409.12169
• Copy Paste: [[2409.12169]] LogoRA: Local-Global Representation Alignment for Robust Time Series Classification(https://arxiv.org/abs/2409.12169)
• Keywords: robust, extraction, transformer
• Abstract: Unsupervised domain adaptation (UDA) of time series aims to teach models to identify consistent patterns across various temporal scenarios, disregarding domain-specific differences, which can maintain their predictive accuracy and effectively adapt to new domains. However, existing UDA methods struggle to adequately extract and align both global and local features in time series data. To address this issue, we propose the Local-Global Representation Alignment framework (LogoRA), which employs a two-branch encoder, comprising a multi-scale convolutional branch and a patching transformer branch. The encoder enables the extraction of both local and global representations from time series. A fusion module is then introduced to integrate these representations, enhancing domain-invariant feature alignment from multi-scale perspectives. To achieve effective alignment, LogoRA employs strategies like invariant feature learning on the source domain, utilizing triplet loss for fine alignment and dynamic time warping-based feature alignment. Additionally, it reduces source-target domain gaps through adversarial training and per-class prototype alignment. Our evaluations on four time-series datasets demonstrate that LogoRA outperforms strong baselines by up to $12.52\%$, showcasing its superiority in time series UDA tasks.

Title: Semantic Interoperability on Blockchain by Generating Smart Contracts Based on Knowledge Graphs

• Authors: William Van Woensel, Oshani Seneviratne
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12171
• Pdf URL: https://arxiv.org/pdf/2409.12171
• Copy Paste: [[2409.12171]] Semantic Interoperability on Blockchain by Generating Smart Contracts Based on Knowledge Graphs(https://arxiv.org/abs/2409.12171)
• Keywords: large language model
• Abstract: Background: Health 3.0 allows decision making to be based on longitudinal data from multiple institutions, from across the patient's healthcare journey. In such a distributed setting, blockchain smart contracts can act as neutral intermediaries to implement trustworthy decision making. Objective: In a distributed setting, transmitted data will be structured using standards (such as HL7 FHIR) for semantic interoperability. In turn, the smart contract will require interoperability with this standard, implement a complex communication setup (e.g., using oracles), and be developed using blockchain languages (e.g., Solidity). We propose the encoding of smart contract logic using a high-level semantic Knowledge Graph, using concepts from the domain standard. We then deploy this semantic KG on blockchain. Methods: Off-chain, a code generation pipeline compiles the KG into a concrete smart contract, which is then deployed on-chain. Our pipeline targets an intermediary bridge representation, which can be transpiled into a specific blockchain language. Our choice avoids on-chain rule engines, with unpredictable and likely higher computational cost; it is thus in line with the economic rules of blockchain. Results: We applied our code generation approach to generate smart contracts for 3 health insurance cases from Medicare. We discuss the suitability of our approach - the need for a neutral intermediary - for a number of healthcare use cases. Our evaluation finds that the generated contracts perform well in terms of correctness and execution cost ("gas") on blockchain. Conclusions: We showed that it is feasible to automatically generate smart contract code based on a semantic KG, in a way that respects the economic rules of blockchain. Future work includes studying the use of Large Language Models (LLM) in our approach, and evaluations on other blockchains.

Title: Expanding Expressivity in Transformer Models with M\"obiusAttention

• Authors: Anna-Maria Halacheva, Mojtaba Nayyeri, Steffen Staab
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12175
• Pdf URL: https://arxiv.org/pdf/2409.12175
• Copy Paste: [[2409.12175]] Expanding Expressivity in Transformer Models with M\"obiusAttention(https://arxiv.org/abs/2409.12175)
• Keywords: transformer
• Abstract: Attention mechanisms and Transformer architectures have revolutionized Natural Language Processing (NLP) by enabling exceptional modeling of long-range dependencies and capturing intricate linguistic patterns. However, their inherent reliance on linear operations in the form of matrix multiplications limits their ability to fully capture inter-token relationships on their own. We propose MöbiusAttention, a novel approach that integrates Möbius transformations within the attention mechanism of Transformer-based models. Möbius transformations are non-linear operations in spaces over complex numbers with the ability to map between various geometries. By incorporating these properties, MöbiusAttention empowers models to learn more intricate geometric relationships between tokens and capture a wider range of information through complex-valued weight vectors. We build and pre-train a BERT and a RoFormer version enhanced with MöbiusAttention, which we then finetune on the GLUE benchmark. We evaluate empirically our approach against the baseline BERT and RoFormer models on a range of downstream tasks. Our approach compares favorably against the baseline models, even with smaller number of parameters suggesting the enhanced expressivity of MöbiusAttention. This research paves the way for exploring the potential of Möbius transformations in the complex projective space to enhance the expressivity and performance of foundation models.

Title: Finetuning Language Models to Emit Linguistic Expressions of Uncertainty

• Authors: Arslan Chaudhry, Sridhar Thiagarajan, Dilan Gorur
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12180
• Pdf URL: https://arxiv.org/pdf/2409.12180
• Copy Paste: [[2409.12180]] Finetuning Language Models to Emit Linguistic Expressions of Uncertainty(https://arxiv.org/abs/2409.12180)
• Keywords: large language model
• Abstract: Large language models (LLMs) are increasingly employed in information-seeking and decision-making tasks. Despite their broad utility, LLMs tend to generate information that conflicts with real-world facts, and their persuasive style can make these inaccuracies appear confident and convincing. As a result, end-users struggle to consistently align the confidence expressed by LLMs with the accuracy of their predictions, often leading to either blind trust in all outputs or a complete disregard for their reliability. In this work, we explore supervised finetuning on uncertainty-augmented predictions as a method to develop models that produce linguistic expressions of uncertainty. Specifically, we measure the calibration of pre-trained models and then fine-tune language models to generate calibrated linguistic expressions of uncertainty. Through experiments on various question-answering datasets, we demonstrate that LLMs are well-calibrated in assessing their predictions, and supervised finetuning based on the model's own confidence leads to well-calibrated expressions of uncertainty, particularly for single-claim answers.

Title: To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning

• Authors: Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, Greg Durrett
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12183
• Pdf URL: https://arxiv.org/pdf/2409.12183
• Copy Paste: [[2409.12183]] To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning(https://arxiv.org/abs/2409.12183)
• Keywords: large language model
• Abstract: Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thinking'' really helpful? To analyze this, we conducted a quantitative meta-analysis covering over 100 papers using CoT and ran our own evaluations of 20 datasets across 14 models. Our results show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks. On MMLU, directly generating the answer without CoT leads to almost identical accuracy as CoT unless the question or model's response contains an equals sign, indicating symbolic operations and reasoning. Following this finding, we analyze the behavior of CoT on these problems by separating planning and execution and comparing against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic execution, but it underperforms relative to using a symbolic solver. Our results indicate that CoT can be applied selectively, maintaining performance while saving inference costs. Furthermore, they suggest a need to move beyond prompt-based CoT to new paradigms that better leverage intermediate computation across the whole range of LLM applications.

Title: Democratizing MLLMs in Healthcare: TinyLLaVA-Med for Efficient Healthcare Diagnostics in Resource-Constrained Settings

• Authors: Aya El Mir, Lukelo Thadei Luoga, Boyuan Chen, Muhammad Abdullah Hanif, Muhammad Shafique
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12184
• Pdf URL: https://arxiv.org/pdf/2409.12184
• Copy Paste: [[2409.12184]] Democratizing MLLMs in Healthcare: TinyLLaVA-Med for Efficient Healthcare Diagnostics in Resource-Constrained Settings(https://arxiv.org/abs/2409.12184)
• Keywords: large language model
• Abstract: Deploying Multi-Modal Large Language Models (MLLMs) in healthcare is hindered by their high computational demands and significant memory requirements, which are particularly challenging for resource-constrained devices like the Nvidia Jetson Xavier. This problem is particularly evident in remote medical settings where advanced diagnostics are needed but resources are limited. In this paper, we introduce an optimization method for the general-purpose MLLM, TinyLLaVA, which we have adapted and renamed TinyLLaVA-Med. This adaptation involves instruction-tuning and fine-tuning TinyLLaVA on a medical dataset by drawing inspiration from the LLaVA-Med training pipeline. Our approach successfully minimizes computational complexity and power consumption, with TinyLLaVA-Med operating at 18.9W and using 11.9GB of memory, while achieving accuracies of 64.54% on VQA-RAD and 70.70% on SLAKE for closed-ended questions. Therefore, TinyLLaVA-Med achieves deployment viability in hardware-constrained environments with low computational resources, maintaining essential functionalities and delivering accuracies close to state-of-the-art models.

Title: Massively Multi-Person 3D Human Motion Forecasting with Scene Context

• Authors: Felix B Mueller, Julian Tanke, Juergen Gall
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12189
• Pdf URL: https://arxiv.org/pdf/2409.12189
• Copy Paste: [[2409.12189]] Massively Multi-Person 3D Human Motion Forecasting with Scene Context(https://arxiv.org/abs/2409.12189)
• Keywords: diffusion, transformer
• Abstract: Forecasting long-term 3D human motion is challenging: the stochasticity of human behavior makes it hard to generate realistic human motion from the input sequence alone. Information on the scene environment and the motion of nearby people can greatly aid the generation process. We propose a scene-aware social transformer model (SAST) to forecast long-term (10s) human motion motion. Unlike previous models, our approach can model interactions between both widely varying numbers of people and objects in a scene. We combine a temporal convolutional encoder-decoder architecture with a Transformer-based bottleneck that allows us to efficiently combine motion and scene information. We model the conditional motion distribution using denoising diffusion models. We benchmark our approach on the Humans in Kitchens dataset, which contains 1 to 16 persons and 29 to 50 objects that are visible simultaneously. Our model outperforms other approaches in terms of realism and diversity on different metrics and in a user study. Code is available at this https URL.

Title: Vista3D: Unravel the 3D Darkside of a Single Image

• Authors: Qiuhong Shen, Xingyi Yang, Michael Bi Mi, Xinchao Wang
• Subjects: cs.CV, cs.AI, cs.GT, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.12193
• Pdf URL: https://arxiv.org/pdf/2409.12193
• Copy Paste: [[2409.12193]] Vista3D: Unravel the 3D Darkside of a Single Image(https://arxiv.org/abs/2409.12193)
• Keywords: diffusion
• Abstract: We embark on the age-old quest: unveiling the hidden dimensions of objects from mere glimpses of their visible parts. To address this, we present Vista3D, a framework that realizes swift and consistent 3D generation within a mere 5 minutes. At the heart of Vista3D lies a two-phase approach: the coarse phase and the fine phase. In the coarse phase, we rapidly generate initial geometry with Gaussian Splatting from a single image. In the fine phase, we extract a Signed Distance Function (SDF) directly from learned Gaussian Splatting, optimizing it with a differentiable isosurface representation. Furthermore, it elevates the quality of generation by using a disentangled representation with two independent implicit functions to capture both visible and obscured aspects of objects. Additionally, it harmonizes gradients from 2D diffusion prior with 3D-aware diffusion priors by angular diffusion prior composition. Through extensive evaluation, we demonstrate that Vista3D effectively sustains a balance between the consistency and diversity of the generated 3D objects. Demos and code will be available at this https URL.

Title: Gender Representation and Bias in Indian Civil Service Mock Interviews

• Authors: Somonnoy Banerjee, Sujan Dutta, Soumyajit Datta, Ashiqur R. KhudaBukhsh
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.12194
• Pdf URL: https://arxiv.org/pdf/2409.12194
• Copy Paste: [[2409.12194]] Gender Representation and Bias in Indian Civil Service Mock Interviews(https://arxiv.org/abs/2409.12194)
• Keywords: large language model
• Abstract: This paper makes three key contributions. First, via a substantial corpus of 51,278 interview questions sourced from 888 YouTube videos of mock interviews of Indian civil service candidates, we demonstrate stark gender bias in the broad nature of questions asked to male and female candidates. Second, our experiments with large language models show a strong presence of gender bias in explanations provided by the LLMs on the gender inference task. Finally, we present a novel dataset of 51,278 interview questions that can inform future social science studies.