2024-09-16

Title: Gaussian Differentially Private Human Faces Under a Face Radial Curve Representation

• Authors: Carlos Soto, Matthew Reimherr, Aleksandra Slavkovic, Mark Shriver
• Subjects: cs.CR, cs.CV, cs.LG, math.FA, math.ST
• Abstract URL: https://arxiv.org/abs/2409.08301
• Pdf URL: https://arxiv.org/pdf/2409.08301
• Copy Paste: [[2409.08301]] Gaussian Differentially Private Human Faces Under a Face Radial Curve Representation(https://arxiv.org/abs/2409.08301)
• Keywords: privacy, protect
• Abstract: In this paper we consider the problem of releasing a Gaussian Differentially Private (GDP) 3D human face. The human face is a complex structure with many features and inherently tied to one's identity. Protecting this data, in a formally private way, is important yet challenging given the dimensionality of the problem. We extend approximate DP techniques for functional data to the GDP framework. We further propose a novel representation, face radial curves, of a 3D face as a set of functions and then utilize our proposed GDP functional data mechanism. To preserve the shape of the face while injecting noise we rely on tools from shape analysis for our novel representation of the face. We show that our method preserves the shape of the average face and injects less noise than traditional methods for the same privacy budget. Our mechanism consists of two primary components, the first is generally applicable to function value summaries (as are commonly found in nonparametric statistics or functional data analysis) while the second is general to disk-like surfaces and hence more applicable than just to human faces.

Title: DiReDi: Distillation and Reverse Distillation for AIoT Applications

• Authors: Chen Sun, Qing Tong, Wenshuang Yang, Wenqi Zhang
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.08308
• Pdf URL: https://arxiv.org/pdf/2409.08308
• Copy Paste: [[2409.08308]] DiReDi: Distillation and Reverse Distillation for AIoT Applications(https://arxiv.org/abs/2409.08308)
• Keywords: privacy, protect
• Abstract: Typically, the significant efficiency can be achieved by deploying different edge AI models in various real world scenarios while a few large models manage those edge AI models remotely from cloud servers. However, customizing edge AI models for each user's specific application or extending current models to new application scenarios remains a challenge. Inappropriate local training or fine tuning of edge AI models by users can lead to model malfunction, potentially resulting in legal issues for the manufacturer. To address aforementioned issues, this paper proposes an innovative framework called "DiReD", which involves knowledge DIstillation & REverse DIstillation. In the initial step, an edge AI model is trained with presumed data and a KD process using the cloud AI model in the upper management cloud server. This edge AI model is then dispatched to edge AI devices solely for inference in the user's application scenario. When the user needs to update the edge AI model to better fit the actual scenario, the reverse distillation (RD) process is employed to extract the knowledge: the difference between user preferences and the manufacturer's presumptions from the edge AI model using the user's exclusive data. Only the extracted knowledge is reported back to the upper management cloud server to update the cloud AI model, thus protecting user privacy by not using any exclusive data. The updated cloud AI can then update the edge AI model with the extended knowledge. Simulation results demonstrate that the proposed "DiReDi" framework allows the manufacturer to update the user model by learning new knowledge from the user's actual scenario with private data. The initial redundant knowledge is reduced since the retraining emphasizes user private data.

Title: Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue

• Authors: Johnathan Ivey, Shivani Kumar, Jiayu Liu, Hua Shen, Sushrita Rakshit, Rohan Raju, Haotian Zhang, Aparna Ananthasubramaniam, Junghwan Kim, Bowen Yi, Dustin Wright, Abraham Israeli, Anders Giovanni Møller, Lechen Zhang, David Jurgens
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.08330
• Pdf URL: https://arxiv.org/pdf/2409.08330
• Copy Paste: [[2409.08330]] Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue(https://arxiv.org/abs/2409.08330)
• Keywords: large language model
• Abstract: Studying and building datasets for dialogue tasks is both expensive and time-consuming due to the need to recruit, train, and collect data from study participants. In response, much recent work has sought to use large language models (LLMs) to simulate both human-human and human-LLM interactions, as they have been shown to generate convincingly human-like text in many settings. However, to what extent do LLM-based simulations \textit{actually} reflect human dialogues? In this work, we answer this question by generating a large-scale dataset of 100,000 paired LLM-LLM and human-LLM dialogues from the WildChat dataset and quantifying how well the LLM simulations align with their human counterparts. Overall, we find relatively low alignment between simulations and human interactions, demonstrating a systematic divergence along the multiple textual properties, including style and content. Further, in comparisons of English, Chinese, and Russian dialogues, we find that models perform similarly. Our results suggest that LLMs generally perform better when the human themself writes in a way that is more similar to the LLM's own style.

Title: SIG: A Synthetic Identity Generation Pipeline for Generating Evaluation Datasets for Face Recognition

• Authors: Kassi Nzalasse, Rishav Raj, Eli Laird, Corey Clark
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08345
• Pdf URL: https://arxiv.org/pdf/2409.08345
• Copy Paste: [[2409.08345]] SIG: A Synthetic Identity Generation Pipeline for Generating Evaluation Datasets for Face Recognition(https://arxiv.org/abs/2409.08345)
• Keywords: privacy, fair
• Abstract: As Artificial Intelligence applications expand, the evaluation of models faces heightened scrutiny. Ensuring public readiness requires evaluation datasets, which differ from training data by being disjoint and ethically sourced in compliance with privacy regulations. The performance and fairness of face recognition systems depend significantly on the quality and representativeness of these evaluation datasets. This data is sometimes scraped from the internet without user's consent, causing ethical concerns that can prohibit its use without proper releases. In rare cases, data is collected in a controlled environment with consent, however, this process is time-consuming, expensive, and logistically difficult to execute. This creates a barrier for those unable to conjure the immense resources required to gather ethically sourced evaluation datasets. To address these challenges, we introduce the Synthetic Identity Generation pipeline, or SIG, that allows for the targeted creation of ethical, balanced datasets for face recognition evaluation. Our proposed and demonstrated pipeline generates high-quality images of synthetic identities with controllable pose, facial features, and demographic attributes, such as race, gender, and age. We also release an open-source evaluation dataset named ControlFace10k, consisting of 10,008 face images of 3,336 unique synthetic identities balanced across race, gender, and age, generated using the proposed SIG pipeline. We analyze ControlFace10k along with a non-synthetic BUPT dataset using state-of-the-art face recognition algorithms to demonstrate its effectiveness as an evaluation tool. This analysis highlights the dataset's characteristics and its utility in assessing algorithmic bias across different demographic groups.

Title: FedProphet: Memory-Efficient Federated Adversarial Training via Theoretic-Robustness and Low-Inconsistency Cascade Learning

• Authors: Minxue Tang, Yitu Wang, Jingyang Zhang, Louis DiValentin, Aolin Ding, Amin Hass, Yiran Chen, Hai "Helen" Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08372
• Pdf URL: https://arxiv.org/pdf/2409.08372
• Copy Paste: [[2409.08372]] FedProphet: Memory-Efficient Federated Adversarial Training via Theoretic-Robustness and Low-Inconsistency Cascade Learning(https://arxiv.org/abs/2409.08372)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) provides a strong privacy guarantee by enabling local training across edge devices without training data sharing, and Federated Adversarial Training (FAT) further enhances the robustness against adversarial examples, promoting a step toward trustworthy artificial intelligence. However, FAT requires a large model to preserve high accuracy while achieving strong robustness, and it is impractically slow when directly training with memory-constrained edge devices due to the memory-swapping latency. Moreover, existing memory-efficient FL methods suffer from poor accuracy and weak robustness in FAT because of inconsistent local and global models, i.e., objective inconsistency. In this paper, we propose FedProphet, a novel FAT framework that can achieve memory efficiency, adversarial robustness, and objective consistency simultaneously. FedProphet partitions the large model into small cascaded modules such that the memory-constrained devices can conduct adversarial training module-by-module. A strong convexity regularization is derived to theoretically guarantee the robustness of the whole model, and we show that the strong robustness implies low objective inconsistency in FedProphet. We also develop a training coordinator on the server of FL, with Adaptive Perturbation Adjustment for utility-robustness balance and Differentiated Module Assignment for objective inconsistency mitigation. FedProphet empirically shows a significant improvement in both accuracy and robustness compared to previous memory-efficient methods, achieving almost the same performance of end-to-end FAT with 80% memory reduction and up to 10.8x speedup in training time.

Title: Automated Cybersecurity Compliance and Threat Response Using AI, Blockchain & Smart Contracts

• Authors: Lampis Alevizos, Vinh Thong Ta
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08390
• Pdf URL: https://arxiv.org/pdf/2409.08390
• Copy Paste: [[2409.08390]] Automated Cybersecurity Compliance and Threat Response Using AI, Blockchain & Smart Contracts(https://arxiv.org/abs/2409.08390)
• Keywords: security
• Abstract: To address the challenges of internal security policy compliance and dynamic threat response in organizations, we present a novel framework that integrates artificial intelligence (AI), blockchain, and smart contracts. We propose a system that automates the enforcement of security policies, reducing manual effort and potential human error. Utilizing AI, we can analyse cyber threat intelligence rapidly, identify non-compliances and automatically adjust cyber defence mechanisms. Blockchain technology provides an immutable ledger for transparent logging of compliance actions, while smart contracts ensure uniform application of security measures. The framework's effectiveness is demonstrated through simulations, showing improvements in compliance enforcement rates and response times compared to traditional methods. Ultimately, our approach provides for a scalable solution for managing complex security policies, reducing costs and enhancing the efficiency while achieving compliance. Finally, we discuss practical implications and propose future research directions to further refine the system and address implementation challenges.

Title: Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning

• Authors: Hanyang Zhao, Haoxian Chen, Ji Zhang, David D. Yao, Wenpin Tang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08400
• Pdf URL: https://arxiv.org/pdf/2409.08400
• Copy Paste: [[2409.08400]] Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning(https://arxiv.org/abs/2409.08400)
• Keywords: diffusion, generative
• Abstract: Reinforcement Learning from human feedback (RLHF) has been shown a promising direction for aligning generative models with human intent and has also been explored in recent works for alignment of diffusion generative models. In this work, we provide a rigorous treatment by formulating the task of fine-tuning diffusion models, with reward functions learned from human feedback, as an exploratory continuous-time stochastic control problem. Our key idea lies in treating the score-matching functions as controls/actions, and upon this, we develop a unified framework from a continuous-time perspective, to employ reinforcement learning (RL) algorithms in terms of improving the generation quality of diffusion models. We also develop the corresponding continuous-time RL theory for policy optimization and regularization under assumptions of stochastic different equations driven environment. Experiments on the text-to-image (T2I) generation will be reported in the accompanied paper.

Title: Knowledge Tagging with Large Language Model based Multi-Agent System

• Authors: Hang Li, Tianlong Xu, Ethan Chang, Qingsong Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08406
• Pdf URL: https://arxiv.org/pdf/2409.08406
• Copy Paste: [[2409.08406]] Knowledge Tagging with Large Language Model based Multi-Agent System(https://arxiv.org/abs/2409.08406)
• Keywords: large language model
• Abstract: Knowledge tagging for questions is vital in modern intelligent educational applications, including learning progress diagnosis, practice question recommendations, and course content organization. Traditionally, these annotations have been performed by pedagogical experts, as the task demands not only a deep semantic understanding of question stems and knowledge definitions but also a strong ability to link problem-solving logic with relevant knowledge concepts. With the advent of advanced natural language processing (NLP) algorithms, such as pre-trained language models and large language models (LLMs), pioneering studies have explored automating the knowledge tagging process using various machine learning models. In this paper, we investigate the use of a multi-agent system to address the limitations of previous algorithms, particularly in handling complex cases involving intricate knowledge definitions and strict numerical constraints. By demonstrating its superior performance on the publicly available math question knowledge tagging dataset, MathKnowCT, we highlight the significant potential of an LLM-based multi-agent system in overcoming the challenges that previous methods have encountered. Finally, through an in-depth discussion of the implications of automating knowledge tagging, we underscore the promising results of deploying LLM-based algorithms in educational contexts.

Title: Wasserstein Distributionally Robust Multiclass Support Vector Machine

• Authors: Michael Ibrahim, Heraldo Rozas, Nagi Gebraeel
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.08409
• Pdf URL: https://arxiv.org/pdf/2409.08409
• Copy Paste: [[2409.08409]] Wasserstein Distributionally Robust Multiclass Support Vector Machine(https://arxiv.org/abs/2409.08409)
• Keywords: robust
• Abstract: We study the problem of multiclass classification for settings where data features $\mathbf{x}$ and their labels $\mathbf{y}$ are uncertain. We identify that distributionally robust one-vs-all (OVA) classifiers often struggle in settings with imbalanced data. To address this issue, we use Wasserstein distributionally robust optimization to develop a robust version of the multiclass support vector machine (SVM) characterized by the Crammer-Singer (CS) loss. First, we prove that the CS loss is bounded from above by a Lipschitz continuous function for all $\mathbf{x} \in \mathcal{X}$ and $\mathbf{y} \in \mathcal{Y}$, then we exploit strong duality results to express the dual of the worst-case risk problem, and we show that the worst-case risk minimization problem admits a tractable convex reformulation due to the regularity of the CS loss. Moreover, we develop a kernel version of our proposed model to account for nonlinear class separation, and we show that it admits a tractable convex upper bound. We also propose a projected subgradient method algorithm for a special case of our proposed linear model to improve scalability. Our numerical experiments demonstrate that our model outperforms state-of-the art OVA models in settings where the training data is highly imbalanced. We also show through experiments on popular real-world datasets that our proposed model often outperforms its regularized counterpart as the first accounts for uncertain labels unlike the latter.

Title: Introducing CausalBench: A Flexible Benchmark Framework for Causal Analysis and Machine Learning

• Authors: Ahmet Kapkiç, Pratanu Mandal, Shu Wan, Paras Sheth, Abhinav Gorantla, Yoonhyuk Choi, Huan Liu, K. Selçuk Candan
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.08419
• Pdf URL: https://arxiv.org/pdf/2409.08419
• Copy Paste: [[2409.08419]] Introducing CausalBench: A Flexible Benchmark Framework for Causal Analysis and Machine Learning(https://arxiv.org/abs/2409.08419)
• Keywords: fair
• Abstract: While witnessing the exceptional success of machine learning (ML) technologies in many applications, users are starting to notice a critical shortcoming of ML: correlation is a poor substitute for causation. The conventional way to discover causal relationships is to use randomized controlled experiments (RCT); in many situations, however, these are impractical or sometimes unethical. Causal learning from observational data offers a promising alternative. While being relatively recent, causal learning aims to go far beyond conventional machine learning, yet several major challenges remain. Unfortunately, advances are hampered due to the lack of unified benchmark datasets, algorithms, metrics, and evaluation service interfaces for causal learning. In this paper, we introduce {\em CausalBench}, a transparent, fair, and easy-to-use evaluation platform, aiming to (a) enable the advancement of research in causal learning by facilitating scientific collaboration in novel algorithms, datasets, and metrics and (b) promote scientific objectivity, reproducibility, fairness, and awareness of bias in causal learning research. CausalBench provides services for benchmarking data, algorithms, models, and metrics, impacting the needs of a broad of scientific and engineering disciplines.

Title: When Context Leads but Parametric Memory Follows in Large Language Models

• Authors: Yufei Tao, Adam Hiatt, Erik Haake, Antonie J. Jetter, Ameeta Agrawal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08435
• Pdf URL: https://arxiv.org/pdf/2409.08435
• Copy Paste: [[2409.08435]] When Context Leads but Parametric Memory Follows in Large Language Models(https://arxiv.org/abs/2409.08435)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable progress in leveraging diverse knowledge sources. This study investigates how nine widely used LLMs allocate knowledge between local context and global parameters when answering open-ended questions in knowledge-consistent scenarios. We introduce a novel dataset, WikiAtomic, and systematically vary context sizes to analyze how LLMs prioritize and utilize the provided information and their parametric knowledge in knowledge-consistent scenarios. Additionally, we also study their tendency to hallucinate under varying context sizes. Our findings reveal consistent patterns across models, including a consistent reliance on both contextual (around 70%) and parametric (around 30%) knowledge, and a decrease in hallucinations with increasing context. These insights highlight the importance of more effective context organization and developing models that use input more deterministically for robust performance.

Title: Towards Unified Facial Action Unit Recognition Framework by Large Language Models

• Authors: Guohong Hu, Xing Lan, Hanyu Jiang, Jiayi Lyu, Jian Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08444
• Pdf URL: https://arxiv.org/pdf/2409.08444
• Copy Paste: [[2409.08444]] Towards Unified Facial Action Unit Recognition Framework by Large Language Models(https://arxiv.org/abs/2409.08444)
• Keywords: large language model
• Abstract: Facial Action Units (AUs) are of great significance in the realm of affective computing. In this paper, we propose AU-LLaVA, the first unified AU recognition framework based on the Large Language Model (LLM). AU-LLaVA consists of a visual encoder, a linear projector layer, and a pre-trained LLM. We meticulously craft the text descriptions and fine-tune the model on various AU datasets, allowing it to generate different formats of AU recognition results for the same input image. On the BP4D and DISFA datasets, AU-LLaVA delivers the most accurate recognition results for nearly half of the AUs. Our model achieves improvements of F1-score up to 11.4% in specific AU recognition compared to previous benchmark results. On the FEAFA dataset, our method achieves significant improvements over all 24 AUs compared to previous benchmark results. AU-LLaVA demonstrates exceptional performance and versatility in AU recognition.

Title: VistaFormer: Scalable Vision Transformers for Satellite Image Time Series Segmentation

• Authors: Ezra MacDonald, Derek Jacoby, Yvonne Coady
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08461
• Pdf URL: https://arxiv.org/pdf/2409.08461
• Copy Paste: [[2409.08461]] VistaFormer: Scalable Vision Transformers for Satellite Image Time Series Segmentation(https://arxiv.org/abs/2409.08461)
• Keywords: transformer, segmentation
• Abstract: We introduce VistaFormer, a lightweight Transformer-based model architecture for the semantic segmentation of remote-sensing images. This model uses a multi-scale Transformer-based encoder with a lightweight decoder that aggregates global and local attention captured in the encoder blocks. VistaFormer uses position-free self-attention layers which simplifies the model architecture and removes the need to interpolate temporal and spatial codes, which can reduce model performance when training and testing image resolutions differ. We investigate simple techniques for filtering noisy input signals like clouds and demonstrate that improved model scalability can be achieved by substituting Multi-Head Self-Attention (MHSA) with Neighbourhood Attention (NA). Experiments on the PASTIS and MTLCC crop-type segmentation benchmarks show that VistaFormer achieves better performance than comparable models and requires only 8% of the floating point operations using MHSA and 11% using NA while also using fewer trainable parameters. VistaFormer with MHSA improves on state-of-the-art mIoU scores by 0.1% on the PASTIS benchmark and 3% on the MTLCC benchmark while VistaFormer with NA improves on the MTLCC benchmark by 3.7%.

Title: VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation

• Authors: Hanning Chen, Yang Ni, Wenjun Huang, Yezi Liu, SungHeon Jeong, Fei Wen, Nathaniel Bastian, Hugo Latapie, Mohsen Imani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08464
• Pdf URL: https://arxiv.org/pdf/2409.08464
• Copy Paste: [[2409.08464]] VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation(https://arxiv.org/abs/2409.08464)
• Keywords: transformer, large language model, segmentation
• Abstract: Vision Transformers (ViTs) have emerged as the backbone of many segmentation models, consistently achieving state-of-the-art (SOTA) performance. However, their success comes at a significant computational cost. Image token pruning is one of the most effective strategies to address this complexity. However, previous approaches fall short when applied to more complex task-oriented segmentation (TOS), where the class of each image patch is not predefined but dependent on the specific input task. This work introduces the Vision Language Guided Token Pruning (VLTP), a novel token pruning mechanism that can accelerate ViTbased segmentation models, particularly for TOS guided by multi-modal large language model (MLLM). We argue that ViT does not need to process every image token through all of its layers only the tokens related to reasoning tasks are necessary. We design a new pruning decoder to take both image tokens and vision-language guidance as input to predict the relevance of each image token to the task. Only image tokens with high relevance are passed to deeper layers of the ViT. Experiments show that the VLTP framework reduces the computational costs of ViT by approximately 25% without performance degradation and by around 40% with only a 1% performance drop.

Title: Generalization Boosted Adapter for Open-Vocabulary Segmentation

• Authors: Wenhao Xu, Changwei Wang, Xuxiang Feng, Rongtao Xu, Longzhao Huang, Zherui Zhang, Li Guo, Shibiao Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08468
• Pdf URL: https://arxiv.org/pdf/2409.08468
• Copy Paste: [[2409.08468]] Generalization Boosted Adapter for Open-Vocabulary Segmentation(https://arxiv.org/abs/2409.08468)
• Keywords: robust, segmentation
• Abstract: Vision-language models (VLMs) have demonstrated remarkable open-vocabulary object recognition capabilities, motivating their adaptation for dense prediction tasks like segmentation. However, directly applying VLMs to such tasks remains challenging due to their lack of pixel-level granularity and the limited data available for fine-tuning, leading to overfitting and poor generalization. To address these limitations, we propose Generalization Boosted Adapter (GBA), a novel adapter strategy that enhances the generalization and robustness of VLMs for open-vocabulary segmentation. GBA comprises two core components: (1) a Style Diversification Adapter (SDA) that decouples features into amplitude and phase components, operating solely on the amplitude to enrich the feature space representation while preserving semantic consistency; and (2) a Correlation Constraint Adapter (CCA) that employs cross-attention to establish tighter semantic associations between text categories and target regions, suppressing irrelevant low-frequency ``noise'' information and avoiding erroneous associations. Through the synergistic effect of the shallow SDA and the deep CCA, GBA effectively alleviates overfitting issues and enhances the semantic relevance of feature representations. As a simple, efficient, and plug-and-play component, GBA can be flexibly integrated into various CLIP-based methods, demonstrating broad applicability and achieving state-of-the-art performance on multiple open-vocabulary segmentation benchmarks.

Title: RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision

• Authors: Shuo Wang, Chunlong Xia, Feng Lv, Yifeng Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08475
• Pdf URL: https://arxiv.org/pdf/2409.08475
• Copy Paste: [[2409.08475]] RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision(https://arxiv.org/abs/2409.08475)
• Keywords: transformer
• Abstract: RT-DETR is the first real-time end-to-end transformer-based object detector. Its efficiency comes from the framework design and the Hungarian matching. However, compared to dense supervision detectors like the YOLO series, the Hungarian matching provides much sparser supervision, leading to insufficient model training and difficult to achieve optimal results. To address these issues, we proposed a hierarchical dense positive supervision method based on RT-DETR, named RT-DETRv3. Firstly, we introduce a CNN-based auxiliary branch that provides dense supervision that collaborates with the original decoder to enhance the encoder feature representation. Secondly, to address insufficient decoder training, we propose a novel learning strategy involving self-attention perturbation. This strategy diversifies label assignment for positive samples across multiple query groups, thereby enriching positive supervisions. Additionally, we introduce a shared-weight decoder branch for dense positive supervision to ensure more high-quality queries matching each ground truth. Notably, all aforementioned modules are training-only. We conduct extensive experiments to demonstrate the effectiveness of our approach on COCO val2017. RT-DETRv3 significantly outperforms existing real-time detectors, including the RT-DETR series and the YOLO series. For example, RT-DETRv3-R18 achieves 48.1% AP (+1.6%/+1.4%) compared to RT-DETR-R18/RT-DETRv2-R18 while maintaining the same latency. Meanwhile, it requires only half of epochs to attain a comparable performance. Furthermore, RT-DETRv3-R101 can attain an impressive 54.6% AP outperforming YOLOv10-X. Code will be released soon.

Title: Research on Data Right Confirmation Mechanism of Federated Learning based on Blockchain

• Authors: Xiaogang Cheng, Ren Guo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08476
• Pdf URL: https://arxiv.org/pdf/2409.08476
• Copy Paste: [[2409.08476]] Research on Data Right Confirmation Mechanism of Federated Learning based on Blockchain(https://arxiv.org/abs/2409.08476)
• Keywords: privacy, protect, federate
• Abstract: Federated learning can solve the privacy protection problem in distributed data mining and machine learning, and how to protect the ownership, use and income rights of all parties involved in federated learning is an important issue. This paper proposes a federated learning data ownership confirmation mechanism based on blockchain and smart contract, which uses decentralized blockchain technology to save the contribution of each participant on the blockchain, and distributes the benefits of federated learning results through the blockchain. In the local simulation environment of the blockchain, the relevant smart contracts and data structures are simulated and implemented, and the feasibility of the scheme is preliminarily demonstrated.

Title: Integrating Neural Operators with Diffusion Models Improves Spectral Representation in Turbulence Modeling

• Authors: Vivek Oommen, Aniruddha Bora, Zhen Zhang, George Em Karniadakis
• Subjects: cs.LG, cs.AI, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2409.08477
• Pdf URL: https://arxiv.org/pdf/2409.08477
• Copy Paste: [[2409.08477]] Integrating Neural Operators with Diffusion Models Improves Spectral Representation in Turbulence Modeling(https://arxiv.org/abs/2409.08477)
• Keywords: diffusion, generative
• Abstract: We integrate neural operators with diffusion models to address the spectral limitations of neural operators in surrogate modeling of turbulent flows. While neural operators offer computational efficiency, they exhibit deficiencies in capturing high-frequency flow dynamics, resulting in overly smooth approximations. To overcome this, we condition diffusion models on neural operators to enhance the resolution of turbulent structures. Our approach is validated for different neural operators on diverse datasets, including a high Reynolds number jet flow simulation and experimental Schlieren velocimetry. The proposed method significantly improves the alignment of predicted energy spectra with true distributions compared to neural operators alone. Additionally, proper orthogonal decomposition analysis demonstrates enhanced spectral fidelity in space-time. This work establishes a new paradigm for combining generative models with neural operators to advance surrogate modeling of turbulent systems, and it can be used in other scientific applications that involve microstructure and high-frequency content. See our project page: this http URL

Title: Risks When Sharing LoRA Fine-Tuned Diffusion Model Weights

• Authors: Dixi Yao
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08482
• Pdf URL: https://arxiv.org/pdf/2409.08482
• Copy Paste: [[2409.08482]] Risks When Sharing LoRA Fine-Tuned Diffusion Model Weights(https://arxiv.org/abs/2409.08482)
• Keywords: privacy, defense, diffusion, generative
• Abstract: With the emerging trend in generative models and convenient public access to diffusion models pre-trained on large datasets, users can fine-tune these models to generate images of personal faces or items in new contexts described by natural language. Parameter efficient fine-tuning (PEFT) such as Low Rank Adaptation (LoRA) has become the most common way to save memory and computation usage on the user end during fine-tuning. However, a natural question is whether the private images used for fine-tuning will be leaked to adversaries when sharing model weights. In this paper, we study the issue of privacy leakage of a fine-tuned diffusion model in a practical setting, where adversaries only have access to model weights, rather than prompts or images used for fine-tuning. We design and build a variational network autoencoder that takes model weights as input and outputs the reconstruction of private images. To improve the efficiency of training such an autoencoder, we propose a training paradigm with the help of timestep embedding. The results give a surprising answer to this research question: an adversary can generate images containing the same identities as the private images. Furthermore, we demonstrate that no existing defense method, including differential privacy-based methods, can preserve the privacy of private data used for fine-tuning a diffusion model without compromising the utility of a fine-tuned model.

Title: A BERT-Based Summarization approach for depression detection

• Authors: Hossein Salahshoor Gavalan, Mohmmad Naim Rastgoo, Bahareh Nakisa
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08483
• Pdf URL: https://arxiv.org/pdf/2409.08483
• Copy Paste: [[2409.08483]] A BERT-Based Summarization approach for depression detection(https://arxiv.org/abs/2409.08483)
• Keywords: robust, extraction, large language model
• Abstract: Depression is a globally prevalent mental disorder with potentially severe repercussions if not addressed, especially in individuals with recurrent episodes. Prior research has shown that early intervention has the potential to mitigate or alleviate symptoms of depression. However, implementing such interventions in a real-world setting may pose considerable challenges. A promising strategy involves leveraging machine learning and artificial intelligence to autonomously detect depression indicators from diverse data sources. One of the most widely available and informative data sources is text, which can reveal a person's mood, thoughts, and feelings. In this context, virtual agents programmed to conduct interviews using clinically validated questionnaires, such as those found in the DAIC-WOZ dataset, offer a robust means for depression detection through linguistic analysis. Utilizing BERT-based models, which are powerful and versatile yet use fewer resources than contemporary large language models, to convert text into numerical representations significantly enhances the precision of depression diagnosis. These models adeptly capture complex semantic and syntactic nuances, improving the detection accuracy of depressive symptoms. Given the inherent limitations of these models concerning text length, our study proposes text summarization as a preprocessing technique to diminish the length and intricacies of input texts. Implementing this method within our uniquely developed framework for feature extraction and classification yielded an F1-score of 0.67 on the test set surpassing all prior benchmarks and 0.81 on the validation set exceeding most previous results on the DAIC-WOZ dataset. Furthermore, we have devised a depression lexicon to assess summary quality and relevance. This lexicon constitutes a valuable asset for ongoing research in depression detection.

Title: Sub-graph Based Diffusion Model for Link Prediction

• Authors: Hang Li, Wei Jin, Geri Skenderi, Harry Shomer, Wenzhuo Tang, Wenqi Fan, Jiliang Tang
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.08487
• Pdf URL: https://arxiv.org/pdf/2409.08487
• Copy Paste: [[2409.08487]] Sub-graph Based Diffusion Model for Link Prediction(https://arxiv.org/abs/2409.08487)
• Keywords: attack, robust, diffusion, generative
• Abstract: Denoising Diffusion Probabilistic Models (DDPMs) represent a contemporary class of generative models with exceptional qualities in both synthesis and maximizing the data likelihood. These models work by traversing a forward Markov Chain where data is perturbed, followed by a reverse process where a neural network learns to undo the perturbations and recover the original data. There have been increasing efforts exploring the applications of DDPMs in the graph domain. However, most of them have focused on the generative perspective. In this paper, we aim to build a novel generative model for link prediction. In particular, we treat link prediction between a pair of nodes as a conditional likelihood estimation of its enclosing sub-graph. With a dedicated design to decompose the likelihood estimation process via the Bayesian formula, we are able to separate the estimation of sub-graph structure and its node features. Such designs allow our model to simultaneously enjoy the advantages of inductive learning and the strong generalization capability. Remarkably, comprehensive experiments across various datasets validate that our proposed method presents numerous advantages: (1) transferability across datasets without retraining, (2) promising generalization on limited training data, and (3) robustness against graph adversarial attacks.

Title: PSTNet: Enhanced Polyp Segmentation with Multi-scale Alignment and Frequency Domain Integration

• Authors: Wenhao Xu, Rongtao Xu, Changwei Wang, Xiuli Li, Shibiao Xu, Li Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08501
• Pdf URL: https://arxiv.org/pdf/2409.08501
• Copy Paste: [[2409.08501]] PSTNet: Enhanced Polyp Segmentation with Multi-scale Alignment and Frequency Domain Integration(https://arxiv.org/abs/2409.08501)
• Keywords: transformer, segmentation
• Abstract: Accurate segmentation of colorectal polyps in colonoscopy images is crucial for effective diagnosis and management of colorectal cancer (CRC). However, current deep learning-based methods primarily rely on fusing RGB information across multiple scales, leading to limitations in accurately identifying polyps due to restricted RGB domain information and challenges in feature misalignment during multi-scale aggregation. To address these limitations, we propose the Polyp Segmentation Network with Shunted Transformer (PSTNet), a novel approach that integrates both RGB and frequency domain cues present in the images. PSTNet comprises three key modules: the Frequency Characterization Attention Module (FCAM) for extracting frequency cues and capturing polyp characteristics, the Feature Supplementary Alignment Module (FSAM) for aligning semantic information and reducing misalignment noise, and the Cross Perception localization Module (CPM) for synergizing frequency cues with high-level semantics to achieve efficient polyp segmentation. Extensive experiments on challenging datasets demonstrate PSTNet's significant improvement in polyp segmentation accuracy across various metrics, consistently outperforming state-of-the-art methods. The integration of frequency domain cues and the novel architectural design of PSTNet contribute to advancing computer-assisted polyp segmentation, facilitating more accurate diagnosis and management of CRC.

Title: Enhancing Privacy in ControlNet and Stable Diffusion via Split Learning

• Authors: Dixi Yao
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08503
• Pdf URL: https://arxiv.org/pdf/2409.08503
• Copy Paste: [[2409.08503]] Enhancing Privacy in ControlNet and Stable Diffusion via Split Learning(https://arxiv.org/abs/2409.08503)
• Keywords: privacy, attack, federate, diffusion, generative
• Abstract: With the emerging trend of large generative models, ControlNet is introduced to enable users to fine-tune pre-trained models with their own data for various use cases. A natural question arises: how can we train ControlNet models while ensuring users' data privacy across distributed devices? Exploring different distributed training schemes, we find conventional federated learning and split learning unsuitable. Instead, we propose a new distributed learning structure that eliminates the need for the server to send gradients back. Through a comprehensive evaluation of existing threats, we discover that in the context of training ControlNet with split learning, most existing attacks are ineffective, except for two mentioned in previous literature. To counter these threats, we leverage the properties of diffusion models and design a new timestep sampling policy during forward processes. We further propose a privacy-preserving activation function and a method to prevent private text prompts from leaving clients, tailored for image generation with diffusion models. Our experimental results demonstrate that our algorithms and systems greatly enhance the efficiency of distributed training for ControlNet while ensuring users' data privacy without compromising image generation quality.

Title: Identifying Human Indoor Daily Life Behavior employing Thermal Sensor Arrays (TSAs)

• Authors: Dina E. Abdelaleem, Hassan M. Ahmed, M. Sami Soliman, Tarek M. Said
• Subjects: cs.CV, eess.SP, physics.med-ph
• Abstract URL: https://arxiv.org/abs/2409.08508
• Pdf URL: https://arxiv.org/pdf/2409.08508
• Copy Paste: [[2409.08508]] Identifying Human Indoor Daily Life Behavior employing Thermal Sensor Arrays (TSAs)(https://arxiv.org/abs/2409.08508)
• Keywords: privacy
• Abstract: Daily activity monitoring systems used in households provide vital information for health status, particularly with aging residents. Multiple approaches have been introduced to achieve such goals, typically obtrusive and non-obtrusive. Amongst the obtrusive approaches are the wearable devices, and among the non-obtrusive approaches are the movement detection systems, including motion sensors and thermal sensor arrays (TSAs). TSA systems are advantageous when preserving a person's privacy and picking his precise spatial location. In this study, human daily living activities were monitored day and night, constructing the corresponding activity time series and spatial probability distribution and employing a TSA system. The monitored activities are classified into two categories: sleeping and daily activity. Results showed the possibility of distinguishing between classes regardless of day and night. The obtained sleep activity duration was compared with previous research using the same raw data. Results showed that the duration of sleep activity, on average, was 9 hours/day, and daily life activity was 7 hours/day. The person's spatial probability distribution was determined using the bivariate distribution for the monitored location. In conclusion, the results showed that sleeping activity was dominant. Our study showed that TSAs were the optimum choice when monitoring human activity. Our proposed approach tackled limitations encountered by previous human activity monitoring systems, such as preserving human privacy while knowing his precise spatial location.

Title: Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defense

• Authors: Jeremy Styborski, Mingzhi Lyu, Yi Huang, Adams Kong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08509
• Pdf URL: https://arxiv.org/pdf/2409.08509
• Copy Paste: [[2409.08509]] Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defense(https://arxiv.org/abs/2409.08509)
• Keywords: defense, attack, robust
• Abstract: Availability poisons exploit supervised learning (SL) algorithms by introducing class-related shortcut features in images such that models trained on poisoned data are useless for real-world datasets. Self-supervised learning (SSL), which utilizes augmentations to learn instance discrimination, is regarded as a strong defense against poisoned data. However, by extending the study of SSL across multiple poisons on the CIFAR-10 and ImageNet-100 datasets, we demonstrate that it often performs poorly, far below that of training on clean data. Leveraging the vulnerability of SL to poison attacks, we introduce adversarial training (AT) on SL to obfuscate poison features and guide robust feature learning for SSL. Our proposed defense, designated VESPR (Vulnerability Exploitation of Supervised Poisoning for Robust SSL), surpasses the performance of six previous defenses across seven popular availability poisons. VESPR displays superior performance over all previous defenses, boosting the minimum and average ImageNet-100 test accuracies of poisoned models by 16% and 9%, respectively. Through analysis and ablation studies, we elucidate the mechanisms by which VESPR learns robust class features.

Title: CasDyF-Net: Image Dehazing via Cascaded Dynamic Filters

• Authors: Wang Yinglong, He Bin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08510
• Pdf URL: https://arxiv.org/pdf/2409.08510
• Copy Paste: [[2409.08510]] CasDyF-Net: Image Dehazing via Cascaded Dynamic Filters(https://arxiv.org/abs/2409.08510)
• Keywords: segmentation
• Abstract: Image dehazing aims to restore image clarity and visual quality by reducing atmospheric scattering and absorption effects. While deep learning has made significant strides in this area, more and more methods are constrained by network depth. Consequently, lots of approaches have adopted parallel branching strategies. however, they often prioritize aspects such as resolution, receptive field, or frequency domain segmentation without dynamically partitioning branches based on the distribution of input features. Inspired by dynamic filtering, we propose using cascaded dynamic filters to create a multi-branch network by dynamically generating filter kernels based on feature map distribution. To better handle branch features, we propose a residual multiscale block (RMB), combining different receptive fields. Furthermore, we also introduce a dynamic convolution-based local fusion method to merge features from adjacent branches. Experiments on RESIDE, Haze4K, and O-Haze datasets validate our method's effectiveness, with our model achieving a PSNR of 43.21dB on the RESIDE-Indoor dataset. The code is available at this https URL.

Title: AWF: Adaptive Weight Fusion for Enhanced Class Incremental Semantic Segmentation

• Authors: Zechao Sun, Haolin Jin, Weitong Chen, Luping Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08516
• Pdf URL: https://arxiv.org/pdf/2409.08516
• Copy Paste: [[2409.08516]] AWF: Adaptive Weight Fusion for Enhanced Class Incremental Semantic Segmentation(https://arxiv.org/abs/2409.08516)
• Keywords: segmentation
• Abstract: Class Incremental Semantic Segmentation (CISS) aims to mitigate catastrophic forgetting by maintaining a balance between previously learned and newly introduced knowledge. Existing methods, primarily based on regularization techniques like knowledge distillation, help preserve old knowledge but often face challenges in effectively integrating new knowledge, resulting in limited overall improvement. Endpoints Weight Fusion (EWF) method, while simple, effectively addresses some of these limitations by dynamically fusing the model weights from previous steps with those from the current step, using a fusion parameter alpha determined by the relative number of previously known classes and newly introduced classes. However, the simplicity of the alpha calculation may limit its ability to fully capture the complexities of different task scenarios, potentially leading to suboptimal fusion outcomes. In this paper, we propose an enhanced approach called Adaptive Weight Fusion (AWF), which introduces an alternating training strategy for the fusion parameter, allowing for more flexible and adaptive weight integration. AWF achieves superior performance by better balancing the retention of old knowledge with the learning of new classes, significantly improving results on benchmark CISS tasks compared to the original EWF. And our experiment code will be released on Github.

Title: Eir: Thai Medical Large Language Models

• Authors: Yutthakorn Thiprak, Rungtam Ngodngamthaweesuk, Songtam Ngodngamtaweesuk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08523
• Pdf URL: https://arxiv.org/pdf/2409.08523
• Copy Paste: [[2409.08523]] Eir: Thai Medical Large Language Models(https://arxiv.org/abs/2409.08523)
• Keywords: secure, security, large language model
• Abstract: We present Eir Thai Medical LLM, a large language model with 8 billion parameters, specifically designed to enhance the accuracy of handling medical tasks in the Thai language. This model focuses on providing clear and easy-to-understand answers for both healthcare professionals and patients, thereby improving the efficiency of diagnosis and treatment processes. Human evaluation was conducted to ensure that the model adheres to care standards and provides unbiased answers. To prioritize data security, the model is deployed within the hospital's internal network, ensuring both high security and faster processing speeds. The internal API connection is secured with encryption and strict authentication measures to prevent data leaks and unauthorized access. We evaluated several open-source large language models with 8 billion parameters on four medical benchmarks: MedQA, MedMCQA, PubMedQA, and the medical subset of MMLU. The best-performing baselines were used to develop Eir Thai Medical LLM. Our evaluation employed multiple questioning strategies, including zero-shot, few-shot, chain-of-thought reasoning, and ensemble/self-consistency voting methods. Our model outperformed commercially available Thai-language large language models by more than 10%. In addition, we developed enhanced model testing tailored for clinical use in Thai across 18 clinical tasks, where our model exceeded GPT-4o performance by more than 11%

Title: 1D-CNN-IDS: 1D CNN-based Intrusion Detection System for IIoT

• Authors: Muhammad Arslan, Muhammad Mubeen, Muhammad Bilal, Saadullah Farooq Abbasi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08529
• Pdf URL: https://arxiv.org/pdf/2409.08529
• Copy Paste: [[2409.08529]] 1D-CNN-IDS: 1D CNN-based Intrusion Detection System for IIoT(https://arxiv.org/abs/2409.08529)
• Keywords: secure, security, privacy, attack
• Abstract: The demand of the Internet of Things (IoT) has witnessed exponential growth. These progresses are made possible by the technological advancements in artificial intelligence, cloud computing, and edge computing. However, these advancements exhibit multiple challenges, including cyber threats, security and privacy concerns, and the risk of potential financial losses. For this reason, this study developed a computationally inexpensive one-dimensional convolutional neural network (1DCNN) algorithm for cyber-attack classification. The proposed study achieved an accuracy of 99.90% to classify nine cyber-attacks. Multiple other performance metrices have been evaluated to validate the efficacy of the proposed scheme. In addition, comparison has been done with existing state-of-the-art schemes. The findings of the proposed study can significantly contribute to the development of secure intrusion detection for IIoT systems.

Title: Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamics

• Authors: Wenqing Zhang, Junming Huang, Ruotong Wang, Changsong Wei, Wenqian Huang, Yuxin Qiao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08530
• Pdf URL: https://arxiv.org/pdf/2409.08530
• Copy Paste: [[2409.08530]] Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamics(https://arxiv.org/abs/2409.08530)
• Keywords: transformer
• Abstract: Long-short range time series forecasting is essential for predicting future trends and patterns over extended periods. While deep learning models such as Transformers have made significant strides in advancing time series forecasting, they often encounter difficulties in capturing long-term dependencies and effectively managing sparse semantic features. The state-space model, Mamba, addresses these issues through its adept handling of selective input and parallel computing, striking a balance between computational efficiency and prediction accuracy. This article examines the advantages and disadvantages of both Mamba and Transformer models, and introduces a combined approach, MAT, which leverages the strengths of each model to capture unique long-short range dependencies and inherent evolutionary patterns in multivariate time series. Specifically, MAT harnesses the long-range dependency capabilities of Mamba and the short-range characteristics of Transformers. Experimental results on benchmark weather datasets demonstrate that MAT outperforms existing comparable methods in terms of prediction accuracy, scalability, and memory efficiency.

Title: An Efficient Privacy-aware Split Learning Framework for Satellite Communications

• Authors: Jianfei Sun, Cong Wu, Shahid Mumtaz, Junyi Tao, Mingsheng Cao, Mei Wang, Valerio Frascolla
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08538
• Pdf URL: https://arxiv.org/pdf/2409.08538
• Copy Paste: [[2409.08538]] An Efficient Privacy-aware Split Learning Framework for Satellite Communications(https://arxiv.org/abs/2409.08538)
• Keywords: privacy
• Abstract: In the rapidly evolving domain of satellite communications, integrating advanced machine learning techniques, particularly split learning, is crucial for enhancing data processing and model training efficiency across satellites, space stations, and ground stations. Traditional ML approaches often face significant challenges within satellite networks due to constraints such as limited bandwidth and computational resources. To address this gap, we propose a novel framework for more efficient SL in satellite communications. Our approach, Dynamic Topology Informed Pruning, namely DTIP, combines differential privacy with graph and model pruning to optimize graph neural networks for distributed learning. DTIP strategically applies differential privacy to raw graph data and prunes GNNs, thereby optimizing both model size and communication load across network tiers. Extensive experiments across diverse datasets demonstrate DTIP's efficacy in enhancing privacy, accuracy, and computational efficiency. Specifically, on Amazon2M dataset, DTIP maintains an accuracy of 0.82 while achieving a 50% reduction in floating-point operations per second. Similarly, on ArXiv dataset, DTIP achieves an accuracy of 0.85 under comparable conditions. Our framework not only significantly improves the operational efficiency of satellite communications but also establishes a new benchmark in privacy-aware distributed learning, potentially revolutionizing data handling in space-based networks.

Title: Causal GNNs: A GNN-Driven Instrumental Variable Approach for Causal Inference in Networks

• Authors: Xiaojing Du, Feiyu Yang, Wentao Gao, Xiongren Chen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.08544
• Pdf URL: https://arxiv.org/pdf/2409.08544
• Copy Paste: [[2409.08544]] Causal GNNs: A GNN-Driven Instrumental Variable Approach for Causal Inference in Networks(https://arxiv.org/abs/2409.08544)
• Keywords: robust
• Abstract: As network data applications continue to expand, causal inference within networks has garnered increasing attention. However, hidden confounders complicate the estimation of causal effects. Most methods rely on the strong ignorability assumption, which presumes the absence of hidden confounders-an assumption that is both difficult to validate and often unrealistic in practice. To address this issue, we propose CgNN, a novel approach that leverages network structure as instrumental variables (IVs), combined with graph neural networks (GNNs) and attention mechanisms, to mitigate hidden confounder bias and improve causal effect estimation. By utilizing network structure as IVs, we reduce confounder bias while preserving the correlation with treatment. Our integration of attention mechanisms enhances robustness and improves the identification of important nodes. Validated on two real-world datasets, our results demonstrate that CgNN effectively mitigates hidden confounder bias and offers a robust GNN-driven IV framework for causal inference in complex network data.

Title: LLM-Powered Grapheme-to-Phoneme Conversion: Benchmark and Case Study

• Authors: Mahta Fetrat Qharabagh, Zahra Dehghanian, Hamid R. Rabiee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08554
• Pdf URL: https://arxiv.org/pdf/2409.08554
• Copy Paste: [[2409.08554]] LLM-Powered Grapheme-to-Phoneme Conversion: Benchmark and Case Study(https://arxiv.org/abs/2409.08554)
• Keywords: large language model
• Abstract: Grapheme-to-phoneme (G2P) conversion is critical in speech processing, particularly for applications like speech synthesis. G2P systems must possess linguistic understanding and contextual awareness of languages with polyphone words and context-dependent phonemes. Large language models (LLMs) have recently demonstrated significant potential in various language tasks, suggesting that their phonetic knowledge could be leveraged for G2P. In this paper, we evaluate the performance of LLMs in G2P conversion and introduce prompting and post-processing methods that enhance LLM outputs without additional training or labeled data. We also present a benchmarking dataset designed to assess G2P performance on sentence-level phonetic challenges of the Persian language. Our results show that by applying the proposed methods, LLMs can outperform traditional G2P tools, even in an underrepresented language like Persian, highlighting the potential of developing LLM-aided G2P systems.

Title: Fair CoVariance Neural Networks

• Authors: Andrea Cavallo, Madeline Navarro, Santiago Segarra, Elvin Isufi
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.08558
• Pdf URL: https://arxiv.org/pdf/2409.08558
• Copy Paste: [[2409.08558]] Fair CoVariance Neural Networks(https://arxiv.org/abs/2409.08558)
• Keywords: robust, fair
• Abstract: Covariance-based data processing is widespread across signal processing and machine learning applications due to its ability to model data interconnectivities and dependencies. However, harmful biases in the data may become encoded in the sample covariance matrix and cause data-driven methods to treat different subpopulations unfairly. Existing works such as fair principal component analysis (PCA) mitigate these effects, but remain unstable in low sample regimes, which in turn may jeopardize the fairness goal. To address both biases and instability, we propose Fair coVariance Neural Networks (FVNNs), which perform graph convolutions on the covariance matrix for both fair and accurate predictions. Our FVNNs provide a flexible model compatible with several existing bias mitigation techniques. In particular, FVNNs allow for mitigating the bias in two ways: first, they operate on fair covariance estimates that remove biases from their principal components; second, they are trained in an end-to-end fashion via a fairness regularizer in the loss function so that the model parameters are tailored to solve the task directly in a fair manner. We prove that FVNNs are intrinsically fairer than analogous PCA approaches thanks to their stability in low sample regimes. We validate the robustness and fairness of our model on synthetic and real-world data, showcasing the flexibility of FVNNs along with the tradeoff between fair and accurate performance.

Title: Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding

• Authors: Tianqiao Liu, Zui Chen, Zitao Liu, Mi Tian, Weiqi Luo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08561
• Pdf URL: https://arxiv.org/pdf/2409.08561
• Copy Paste: [[2409.08561]] Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding(https://arxiv.org/abs/2409.08561)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in tasks requiring reasoning and multi-step problem-solving through the use of chain-of-thought (CoT) prompting. However, generating the full CoT process results in significantly longer output sequences, leading to increased computational costs and latency during inference. To address this challenge, we propose a novel approach to compress the CoT process through semantic alignment, enabling more efficient decoding while preserving the benefits of CoT reasoning. Our method introduces an auxiliary CoT model that learns to generate and compress the full thought process into a compact special token representation semantically aligned with the original CoT output. This compressed representation is then integrated into the input of the Hidden Chain-of-Thought (HCoT) model. The training process follows a two-stage procedure: First, the CoT model is optimized to generate the compressed token representations aligned with the ground-truth CoT outputs using a contrastive loss. Subsequently, with the CoT model parameters frozen, the HCoT model is fine-tuned to generate accurate subsequent predictions conditioned on the prefix instruction and the compressed CoT representations from the CoT model. Extensive experiments across three challenging domains - mathematical reasoning, agent invocation, and question answering - demonstrate that our semantic compression approach achieves competitive or improved performance compared to the full CoT baseline, while providing significant speedups of at least 1.5x in decoding time. Moreover, incorporating contrastive learning objectives further enhances the quality of the compressed representations, leading to better CoT prompting and improved task accuracy. Our work paves the way for more efficient exploitation of multi-step reasoning capabilities in LLMs across a wide range of applications.

Title: CSS: Overcoming Pose and Scene Challenges in Crowd-Sourced 3D Gaussian Splatting

• Authors: Runze Chen, Mingyu Xiao, Haiyong Luo, Fang Zhao, Fan Wu, Hao Xiong, Qi Liu, Meng Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08562
• Pdf URL: https://arxiv.org/pdf/2409.08562
• Copy Paste: [[2409.08562]] CSS: Overcoming Pose and Scene Challenges in Crowd-Sourced 3D Gaussian Splatting(https://arxiv.org/abs/2409.08562)
• Keywords: robust
• Abstract: We introduce Crowd-Sourced Splatting (CSS), a novel 3D Gaussian Splatting (3DGS) pipeline designed to overcome the challenges of pose-free scene reconstruction using crowd-sourced imagery. The dream of reconstructing historically significant but inaccessible scenes from collections of photographs has long captivated researchers. However, traditional 3D techniques struggle with missing camera poses, limited viewpoints, and inconsistent lighting. CSS addresses these challenges through robust geometric priors and advanced illumination modeling, enabling high-quality novel view synthesis under complex, real-world conditions. Our method demonstrates clear improvements over existing approaches, paving the way for more accurate and flexible applications in AR, VR, and large-scale 3D reconstruction.

Title: Second-order difference subspace

• Authors: Kazuhiro Fukui, Pedro H.V. Valois, Lincon Souza, Takumi Kobayashi
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08563
• Pdf URL: https://arxiv.org/pdf/2409.08563
• Copy Paste: [[2409.08563]] Second-order difference subspace(https://arxiv.org/abs/2409.08563)
• Keywords: biometric
• Abstract: Subspace representation is a fundamental technique in various fields of machine learning. Analyzing a geometrical relationship among multiple subspaces is essential for understanding subspace series' temporal and/or spatial dynamics. This paper proposes the second-order difference subspace, a higher-order extension of the first-order difference subspace between two subspaces that can analyze the geometrical difference between them. As a preliminary for that, we extend the definition of the first-order difference subspace to the more general setting that two subspaces with different dimensions have an intersection. We then define the second-order difference subspace by combining the concept of first-order difference subspace and principal component subspace (Karcher mean) between two subspaces, motivated by the second-order central difference method. We can understand that the first/second-order difference subspaces correspond to the velocity and acceleration of subspace dynamics from the viewpoint of a geodesic on a Grassmann manifold. We demonstrate the validity and naturalness of our second-order difference subspace by showing numerical results on two applications: temporal shape analysis of a 3D object and time series analysis of a biometric signal.

Title: Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia

• Authors: Fajri Koto
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08564
• Pdf URL: https://arxiv.org/pdf/2409.08564
• Copy Paste: [[2409.08564]] Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia(https://arxiv.org/abs/2409.08564)
• Keywords: large language model
• Abstract: While knowledge evaluation in large language models has predominantly focused on academic subjects like math and physics, these assessments often fail to capture the practical demands of real-world professions. In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Our comprehensive evaluation of 27 large language models shows that these models struggle particularly in fields with strong local contexts, such as insurance and finance. Additionally, while using the entire dataset, shuffling answer options generally maintains consistent evaluation results across models, but it introduces instability specifically in the insurance and finance sectors.

Title: Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detection

• Authors: Hyewon Park, Hyejin Park, Jueun Ko, Dongbo Min
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08566
• Pdf URL: https://arxiv.org/pdf/2409.08566
• Copy Paste: [[2409.08566]] Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detection(https://arxiv.org/abs/2409.08566)
• Keywords: robust
• Abstract: Continual Test Time Adaptation (CTTA) has emerged as a critical approach for bridging the domain gap between the controlled training environments and the real-world scenarios, enhancing model adaptability and robustness. Existing CTTA methods, typically categorized into Full-Tuning (FT) and Efficient-Tuning (ET), struggle with effectively addressing domain shifts. To overcome these challenges, we propose Hybrid-TTA, a holistic approach that dynamically selects instance-wise tuning method for optimal adaptation. Our approach introduces the Dynamic Domain Shift Detection (DDSD) strategy, which identifies domain shifts by leveraging temporal correlations in input sequences and dynamically switches between FT and ET to adapt to varying domain shifts effectively. Additionally, the Masked Image Modeling based Adaptation (MIMA) framework is integrated to ensure domain-agnostic robustness with minimal computational overhead. Our Hybrid-TTA achieves a notable 1.6%p improvement in mIoU on the Cityscapes-to-ACDC benchmark dataset, surpassing previous state-of-the-art methods and offering a robust solution for real-world continual adaptation challenges.

Title: DiffFAS: Face Anti-Spoofing via Generative Diffusion Models

• Authors: Xinxu Ge, Xin Liu, Zitong Yu, Jingang Shi, Chun Qi, Jie Li, Heikki Kälviäinen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08572
• Pdf URL: https://arxiv.org/pdf/2409.08572
• Copy Paste: [[2409.08572]] DiffFAS: Face Anti-Spoofing via Generative Diffusion Models(https://arxiv.org/abs/2409.08572)
• Keywords: attack, diffusion, generative
• Abstract: Face anti-spoofing (FAS) plays a vital role in preventing face recognition (FR) systems from presentation attacks. Nowadays, FAS systems face the challenge of domain shift, impacting the generalization performance of existing FAS methods. In this paper, we rethink about the inherence of domain shift and deconstruct it into two factors: image style and image quality. Quality influences the purity of the presentation of spoof information, while style affects the manner in which spoof information is presented. Based on our analysis, we propose DiffFAS framework, which quantifies quality as prior information input into the network to counter image quality shift, and performs diffusion-based high-fidelity cross-domain and cross-attack types generation to counter image style shift. DiffFAS transforms easily collectible live faces into high-fidelity attack faces with precise labels while maintaining consistency between live and spoof face identities, which can also alleviate the scarcity of labeled data with novel type attacks faced by nowadays FAS system. We demonstrate the effectiveness of our framework on challenging cross-domain and cross-attack FAS datasets, achieving the state-of-the-art performance. Available at this https URL.

Title: HTR-VT: Handwritten Text Recognition with Vision Transformer

• Authors: Yuting Li, Dexiong Chen, Tinglong Tang, Xi Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08573
• Pdf URL: https://arxiv.org/pdf/2409.08573
• Copy Paste: [[2409.08573]] HTR-VT: Handwritten Text Recognition with Vision Transformer(https://arxiv.org/abs/2409.08573)
• Keywords: extraction, transformer
• Abstract: We explore the application of Vision Transformer (ViT) for handwritten text recognition. The limited availability of labeled data in this domain poses challenges for achieving high performance solely relying on ViT. Previous transformer-based models required external data or extensive pre-training on large datasets to excel. To address this limitation, we introduce a data-efficient ViT method that uses only the encoder of the standard transformer. We find that incorporating a Convolutional Neural Network (CNN) for feature extraction instead of the original patch embedding and employ Sharpness-Aware Minimization (SAM) optimizer to ensure that the model can converge towards flatter minima and yield notable enhancements. Furthermore, our introduction of the span mask technique, which masks interconnected features in the feature map, acts as an effective regularizer. Empirically, our approach competes favorably with traditional CNN-based models on small datasets like IAM and READ2016. Additionally, it establishes a new benchmark on the LAM dataset, currently the largest dataset with 19,830 training text lines. The code is publicly available at: this https URL.

Title: Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions

• Authors: Lingwei Meng, Shujie Hu, Jiawen Kang, Zhaoqing Li, Yuejiao Wang, Wenxuan Wu, Xixin Wu, Xunying Liu, Helen Meng
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.08596
• Pdf URL: https://arxiv.org/pdf/2409.08596
• Copy Paste: [[2409.08596]] Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions(https://arxiv.org/abs/2409.08596)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have revolutionized various domains, bringing significant progress and new opportunities. Despite progress in speech-related tasks, LLMs have not been sufficiently explored in multi-talker scenarios. In this work, we present a pioneering effort to investigate the capability of LLMs in transcribing speech in multi-talker environments, following versatile instructions related to multi-talker automatic speech recognition (ASR), target talker ASR, and ASR based on specific talker attributes such as sex, occurrence order, language, and keyword spoken. Our approach utilizes WavLM and Whisper encoder to extract multi-faceted speech representations that are sensitive to speaker characteristics and semantic context. These representations are then fed into an LLM fine-tuned using LoRA, enabling the capabilities for speech comprehension and transcription. Comprehensive experiments reveal the promising performance of our proposed system, MT-LLM, in cocktail party scenarios, highlighting the potential of LLM to handle speech-related tasks based on user instructions in such complex settings.

Title: Improving Analog Neural Network Robustness: A Noise-Agnostic Approach with Explainable Regularizations

• Authors: Alice Duque, Pedro Freire, Egor Manuylovich, Dmitrii Stoliarov, Jaroslaw Prilepsky, Sergei Turitsyn
• Subjects: cs.LG, cs.AI, physics.optics
• Abstract URL: https://arxiv.org/abs/2409.08633
• Pdf URL: https://arxiv.org/pdf/2409.08633
• Copy Paste: [[2409.08633]] Improving Analog Neural Network Robustness: A Noise-Agnostic Approach with Explainable Regularizations(https://arxiv.org/abs/2409.08633)
• Keywords: robust
• Abstract: This work tackles the critical challenge of mitigating "hardware noise" in deep analog neural networks, a major obstacle in advancing analog signal processing devices. We propose a comprehensive, hardware-agnostic solution to address both correlated and uncorrelated noise affecting the activation layers of deep neural models. The novelty of our approach lies in its ability to demystify the "black box" nature of noise-resilient networks by revealing the underlying mechanisms that reduce sensitivity to noise. In doing so, we introduce a new explainable regularization framework that harnesses these mechanisms to significantly enhance noise robustness in deep neural architectures.

Title: Utilizing Data Fingerprints for Privacy-Preserving Algorithm Selection in Time Series Classification: Performance and Uncertainty Estimation on Unseen Datasets

• Authors: Lars Böcking, Leopold Müller, Niklas Kühl
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08636
• Pdf URL: https://arxiv.org/pdf/2409.08636
• Copy Paste: [[2409.08636]] Utilizing Data Fingerprints for Privacy-Preserving Algorithm Selection in Time Series Classification: Performance and Uncertainty Estimation on Unseen Datasets(https://arxiv.org/abs/2409.08636)
• Keywords: privacy
• Abstract: The selection of algorithms is a crucial step in designing AI services for real-world time series classification use cases. Traditional methods such as neural architecture search, automated machine learning, combined algorithm selection, and hyperparameter optimizations are effective but require considerable computational resources and necessitate access to all data points to run their optimizations. In this work, we introduce a novel data fingerprint that describes any time series classification dataset in a privacy-preserving manner and provides insight into the algorithm selection problem without requiring training on the (unseen) dataset. By decomposing the multi-target regression problem, only our data fingerprints are used to estimate algorithm performance and uncertainty in a scalable and adaptable manner. Our approach is evaluated on the 112 University of California riverside benchmark datasets, demonstrating its effectiveness in predicting the performance of 35 state-of-the-art algorithms and providing valuable insights for effective algorithm selection in time series classification service systems, improving a naive baseline by 7.32% on average in estimating the mean performance and 15.81% in estimating the uncertainty.

Title: Byzantine-Robust and Communication-Efficient Distributed Learning via Compressed Momentum Filtering

• Authors: Changxin Liu, Yanghao Li, Yuhao Yi, Karl H. Johansson
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.08640
• Pdf URL: https://arxiv.org/pdf/2409.08640
• Copy Paste: [[2409.08640]] Byzantine-Robust and Communication-Efficient Distributed Learning via Compressed Momentum Filtering(https://arxiv.org/abs/2409.08640)
• Keywords: privacy, robust
• Abstract: Distributed learning has become the standard approach for training large-scale machine learning models across private data silos. While distributed learning enhances privacy preservation and training efficiency, it faces critical challenges related to Byzantine robustness and communication reduction. Existing Byzantine-robust and communication-efficient methods rely on full gradient information either at every iteration or at certain iterations with a probability, and they only converge to an unnecessarily large neighborhood around the solution. Motivated by these issues, we propose a novel Byzantine-robust and communication-efficient stochastic distributed learning method that imposes no requirements on batch size and converges to a smaller neighborhood around the optimal solution than all existing methods, aligning with the theoretical lower bound. Our key innovation is leveraging Polyak Momentum to mitigate the noise caused by both biased compressors and stochastic gradients, thus defending against Byzantine workers under information compression. We provide proof of tight complexity bounds for our algorithm in the context of non-convex smooth loss functions, demonstrating that these bounds match the lower bounds in Byzantine-free scenarios. Finally, we validate the practical significance of our algorithm through an extensive series of experiments, benchmarking its performance on both binary classification and image classification tasks.

Title: Promoting Fairness in Link Prediction with Graph Enhancement

• Authors: Yezi Liu, Hanning Chen, Mohsen Imani
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08658
• Pdf URL: https://arxiv.org/pdf/2409.08658
• Copy Paste: [[2409.08658]] Promoting Fairness in Link Prediction with Graph Enhancement(https://arxiv.org/abs/2409.08658)
• Keywords: fair
• Abstract: Link prediction is a crucial task in network analysis, but it has been shown to be prone to biased predictions, particularly when links are unfairly predicted between nodes from different sensitive groups. In this paper, we study the fair link prediction problem, which aims to ensure that the predicted link probability is independent of the sensitive attributes of the connected nodes. Existing methods typically incorporate debiasing techniques within graph embeddings to mitigate this issue. However, training on large real-world graphs is already challenging, and adding fairness constraints can further complicate the process. To overcome this challenge, we propose FairLink, a method that learns a fairness-enhanced graph to bypass the need for debiasing during the link predictor's training. FairLink maintains link prediction accuracy by ensuring that the enhanced graph follows a training trajectory similar to that of the original input graph. Meanwhile, it enhances fairness by minimizing the absolute difference in link probabilities between node pairs within the same sensitive group and those between node pairs from different sensitive groups. Our extensive experiments on multiple large-scale graphs demonstrate that FairLink not only promotes fairness but also often achieves link prediction accuracy comparable to baseline methods. Most importantly, the enhanced graph exhibits strong generalizability across different GNN architectures.

Title: Towards certifiable AI in aviation: landscape, challenges, and opportunities

• Authors: Hymalai Bello, Daniel Geißler, Lala Ray, Stefan Müller-Divéky, Peter Müller, Shannon Kittrell, Mengxi Liu, Bo Zhou, Paul Lukowicz
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08666
• Pdf URL: https://arxiv.org/pdf/2409.08666
• Copy Paste: [[2409.08666]] Towards certifiable AI in aviation: landscape, challenges, and opportunities(https://arxiv.org/abs/2409.08666)
• Keywords: attack, robust
• Abstract: Artificial Intelligence (AI) methods are powerful tools for various domains, including critical fields such as avionics, where certification is required to achieve and maintain an acceptable level of safety. General solutions for safety-critical systems must address three main questions: Is it suitable? What drives the system's decisions? Is it robust to errors/attacks? This is more complex in AI than in traditional methods. In this context, this paper presents a comprehensive mind map of formal AI certification in avionics. It highlights the challenges of certifying AI development with an example to emphasize the need for qualification beyond performance metrics.

Title: GenMapping: Unleashing the Potential of Inverse Perspective Mapping for Robust Online HD Map Construction

• Authors: Siyu Li, Kailun Yang, Hao Shi, Song Wang, You Yao, Zhiyong Li
• Subjects: cs.CV, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.08688
• Pdf URL: https://arxiv.org/pdf/2409.08688
• Copy Paste: [[2409.08688]] GenMapping: Unleashing the Potential of Inverse Perspective Mapping for Robust Online HD Map Construction(https://arxiv.org/abs/2409.08688)
• Keywords: robust
• Abstract: Online High-Definition (HD) maps have emerged as the preferred option for autonomous driving, overshadowing the counterpart offline HD maps due to flexible update capability and lower maintenance costs. However, contemporary online HD map models embed parameters of visual sensors into training, resulting in a significant decrease in generalization performance when applied to visual sensors with different parameters. Inspired by the inherent potential of Inverse Perspective Mapping (IPM), where camera parameters are decoupled from the training process, we have designed a universal map generation framework, GenMapping. The framework is established with a triadic synergy architecture, including principal and dual auxiliary branches. When faced with a coarse road image with local distortion translated via IPM, the principal branch learns robust global features under the state space models. The two auxiliary branches are a dense perspective branch and a sparse prior branch. The former exploits the correlation information between static and moving objects, whereas the latter introduces the prior knowledge of OpenStreetMap (OSM). The triple-enhanced merging module is crafted to synergistically integrate the unique spatial features from all three branches. To further improve generalization capabilities, a Cross-View Map Learning (CVML) scheme is leveraged to realize joint learning within the common space. Additionally, a Bidirectional Data Augmentation (BiDA) module is introduced to mitigate reliance on datasets concurrently. A thorough array of experimental results shows that the proposed model surpasses current state-of-the-art methods in both semantic mapping and vectorized mapping, while also maintaining a rapid inference speed. The source code will be publicly available at this https URL.

Title: Autoregressive Sequence Modeling for 3D Medical Image Representation

• Authors: Siwen Wang, Churan Wang, Fei Gao, Lixian Su, Fandong Zhang, Yizhou Wang, Yizhou Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08691
• Pdf URL: https://arxiv.org/pdf/2409.08691
• Copy Paste: [[2409.08691]] Autoregressive Sequence Modeling for 3D Medical Image Representation(https://arxiv.org/abs/2409.08691)
• Keywords: robust
• Abstract: Three-dimensional (3D) medical images, such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), are essential for clinical applications. However, the need for diverse and comprehensive representations is particularly pronounced when considering the variability across different organs, diagnostic tasks, and imaging modalities. How to effectively interpret the intricate contextual information and extract meaningful insights from these images remains an open challenge to the community. While current self-supervised learning methods have shown potential, they often consider an image as a whole thereby overlooking the extensive, complex relationships among local regions from one or multiple images. In this work, we introduce a pioneering method for learning 3D medical image representations through an autoregressive pre-training framework. Our approach sequences various 3D medical images based on spatial, contrast, and semantic correlations, treating them as interconnected visual tokens within a token sequence. By employing an autoregressive sequence modeling task, we predict the next visual token in the sequence, which allows our model to deeply understand and integrate the contextual information inherent in 3D medical images. Additionally, we implement a random startup strategy to avoid overestimating token relationships and to enhance the robustness of learning. The effectiveness of our approach is demonstrated by the superior performance over others on nine downstream tasks in public datasets.

Title: L3Cube-IndicQuest: A Benchmark Questing Answering Dataset for Evaluating Knowledge of LLMs in Indic Context

• Authors: Pritika Rohera, Chaitrali Ginimav, Akanksha Salunke, Gayatri Sawant, Raviraj Joshi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08706
• Pdf URL: https://arxiv.org/pdf/2409.08706
• Copy Paste: [[2409.08706]] L3Cube-IndicQuest: A Benchmark Questing Answering Dataset for Evaluating Knowledge of LLMs in Indic Context(https://arxiv.org/abs/2409.08706)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have made significant progress in incorporating Indic languages within multilingual models. However, it is crucial to quantitatively assess whether these languages perform comparably to globally dominant ones, such as English. Currently, there is a lack of benchmark datasets specifically designed to evaluate the regional knowledge of LLMs in various Indic languages. In this paper, we present the L3Cube-IndicQuest, a gold-standard question-answering benchmark dataset designed to evaluate how well multilingual LLMs capture regional knowledge across various Indic languages. The dataset contains 200 question-answer pairs, each for English and 19 Indic languages, covering five domains specific to the Indic region. We aim for this dataset to serve as a benchmark, providing ground truth for evaluating the performance of LLMs in understanding and representing knowledge relevant to the Indian context. The IndicQuest can be used for both reference-based evaluation and LLM-as-a-judge evaluation. The dataset is shared publicly at this https URL .

Title: Bridging Dynamic Factor Models and Neural Controlled Differential Equations for Nowcasting GDP

• Authors: Seonkyu Lim, Jeongwhan Choi, Noseong Park, Sang-Ha Yoon, ShinHyuck Kang, Young-Min Kim, Hyunjoong Kang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08732
• Pdf URL: https://arxiv.org/pdf/2409.08732
• Copy Paste: [[2409.08732]] Bridging Dynamic Factor Models and Neural Controlled Differential Equations for Nowcasting GDP(https://arxiv.org/abs/2409.08732)
• Keywords: extraction, interpretability
• Abstract: Gross domestic product (GDP) nowcasting is crucial for policy-making as GDP growth is a key indicator of economic conditions. Dynamic factor models (DFMs) have been widely adopted by government agencies for GDP nowcasting due to their ability to handle irregular or missing macroeconomic indicators and their interpretability. However, DFMs face two main challenges: i) the lack of capturing economic uncertainties such as sudden recessions or booms, and ii) the limitation of capturing irregular dynamics from mixed-frequency data. To address these challenges, we introduce NCDENow, a novel GDP nowcasting framework that integrates neural controlled differential equations (NCDEs) with DFMs. This integration effectively handles the dynamics of irregular time series. NCDENow consists of 3 main modules: i) factor extraction leveraging DFM, ii) dynamic modeling using NCDE, and iii) GDP growth prediction through regression. We evaluate NCDENow against 6 baselines on 2 real-world GDP datasets from South Korea and the United Kingdom, demonstrating its enhanced predictive capability. Our empirical results favor our method, highlighting the significant potential of integrating NCDE into nowcasting models. Our code and dataset are available at this https URL.

Title: SAUC: Sparsity-Aware Uncertainty Calibration for Spatiotemporal Prediction with Graph Neural Networks

• Authors: Dingyi Zhuang, Yuheng Bu, Guang Wang, Shenhao Wang, Jinhua Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08766
• Pdf URL: https://arxiv.org/pdf/2409.08766
• Copy Paste: [[2409.08766]] SAUC: Sparsity-Aware Uncertainty Calibration for Spatiotemporal Prediction with Graph Neural Networks(https://arxiv.org/abs/2409.08766)
• Keywords: robust
• Abstract: Quantifying uncertainty is crucial for robust and reliable predictions. However, existing spatiotemporal deep learning mostly focuses on deterministic prediction, overlooking the inherent uncertainty in such prediction. Particularly, highly-granular spatiotemporal datasets are often sparse, posing extra challenges in prediction and uncertainty quantification. To address these issues, this paper introduces a novel post-hoc Sparsity-awar Uncertainty Calibration (SAUC) framework, which calibrates uncertainty in both zero and non-zero values. To develop SAUC, we firstly modify the state-of-the-art deterministic spatiotemporal Graph Neural Networks (ST-GNNs) to probabilistic ones in the pre-calibration phase. Then we calibrate the probabilistic ST-GNNs for zero and non-zero values using quantile approaches.Through extensive experiments, we demonstrate that SAUC can effectively fit the variance of sparse data and generalize across two real-world spatiotemporal datasets at various granularities. Specifically, our empirical experiments show a 20\% reduction in calibration errors in zero entries on the sparse traffic accident and urban crime prediction. Overall, this work demonstrates the theoretical and empirical values of the SAUC framework, thus bridging a significant gap between uncertainty quantification and spatiotemporal prediction.

Title: Causal Transformer for Fusion and Pose Estimation in Deep Visual Inertial Odometry

• Authors: Yunus Bilge Kurt, Ahmet Akman, A. Aydın Alatan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08769
• Pdf URL: https://arxiv.org/pdf/2409.08769
• Copy Paste: [[2409.08769]] Causal Transformer for Fusion and Pose Estimation in Deep Visual Inertial Odometry(https://arxiv.org/abs/2409.08769)
• Keywords: transformer
• Abstract: In recent years, transformer-based architectures become the de facto standard for sequence modeling in deep learning frameworks. Inspired by the successful examples, we propose a causal visual-inertial fusion transformer (VIFT) for pose estimation in deep visual-inertial odometry. This study aims to improve pose estimation accuracy by leveraging the attention mechanisms in transformers, which better utilize historical data compared to the recurrent neural network (RNN) based methods seen in recent methods. Transformers typically require large-scale data for training. To address this issue, we utilize inductive biases for deep VIO networks. Since latent visual-inertial feature vectors encompass essential information for pose estimation, we employ transformers to refine pose estimates by updating latent vectors temporally. Our study also examines the impact of data imbalance and rotation learning methods in supervised end-to-end learning of visual inertial odometry by utilizing specialized gradients in backpropagation for the elements of SE$(3)$ group. The proposed method is end-to-end trainable and requires only a monocular camera and IMU during inference. Experimental results demonstrate that VIFT increases the accuracy of monocular VIO networks, achieving state-of-the-art results when compared to previous methods on the KITTI dataset. The code will be made available at this https URL.

Title: In-depth Analysis of Low-rank Matrix Factorisation in a Federated Setting

• Authors: Constantin Philippenko, Kevin Scaman, Laurent Massoulié
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2409.08771
• Pdf URL: https://arxiv.org/pdf/2409.08771
• Copy Paste: [[2409.08771]] In-depth Analysis of Low-rank Matrix Factorisation in a Federated Setting(https://arxiv.org/abs/2409.08771)
• Keywords: federate
• Abstract: We analyze a distributed algorithm to compute a low-rank matrix factorization on $N$ clients, each holding a local dataset $\mathbf{S}^i \in \mathbb{R}^{n_i \times d}$, mathematically, we seek to solve $min_{\mathbf{U}^i \in \mathbb{R}^{n_i\times r}, \mathbf{V}\in \mathbb{R}^{d \times r} } \frac{1}{2} \sum_{i=1}^N \|\mathbf{S}^i - \mathbf{U}^i \mathbf{V}^\top\|^2_{\text{F}}$. Considering a power initialization of $\mathbf{V}$, we rewrite the previous smooth non-convex problem into a smooth strongly-convex problem that we solve using a parallel Nesterov gradient descent potentially requiring a single step of communication at the initialization step. For any client $i$ in $\{1, \dots, N\}$, we obtain a global $\mathbf{V}$ in $\mathbb{R}^{d \times r}$ common to all clients and a local variable $\mathbf{U}^i$ in $\mathbb{R}^{n_i \times r}$. We provide a linear rate of convergence of the excess loss which depends on $\sigma_{\max} / \sigma_{r}$, where $\sigma_{r}$ is the $r^{\mathrm{th}}$ singular value of the concatenation $\mathbf{S}$ of the matrices $(\mathbf{S}^i)_{i=1}^N$. This result improves the rates of convergence given in the literature, which depend on $\sigma_{\max}^2 / \sigma_{\min}^2$. We provide an upper bound on the Frobenius-norm error of reconstruction under the power initialization strategy. We complete our analysis with experiments on both synthetic and real data.

Title: An Attack on $p$-adic Lattice Public-key Cryptosystems and Signature Schemes

• Authors: Chi Zhang
• Subjects: cs.CR, math.NT
• Abstract URL: https://arxiv.org/abs/2409.08774
• Pdf URL: https://arxiv.org/pdf/2409.08774
• Copy Paste: [[2409.08774]] An Attack on $p$-adic Lattice Public-key Cryptosystems and Signature Schemes(https://arxiv.org/abs/2409.08774)
• Keywords: attack
• Abstract: Lattices have many significant applications in cryptography. In 2021, the $p$-adic signature scheme and public-key encryption cryptosystem were introduced. They are based on the Longest Vector Problem (LVP) and the Closest Vector Problem (CVP) in $p$-adic lattices. These problems are considered to be challenging and there are no known deterministic polynomial time algorithms to solve them. In this paper, we improve the LVP algorithm in local fields. The modified LVP algorithm is a deterministic polynomial time algorithm when the field is totally ramified and $p$ is a polynomial in the rank of the input lattice. We utilize this algorithm to attack the above schemes so that we are able to forge a valid signature of any message and decrypt any ciphertext. Although these schemes are broken, this work does not mean that $p$-adic lattices are not suitable in constructing cryptographic primitives. We propose some possible modifications to avoid our attack at the end of this paper.

Title: Sign Language Sense Disambiguation

• Authors: Jana Grimm, Miriam Winkler, Oliver Kraus, Tanalp Agustoslu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08780
• Pdf URL: https://arxiv.org/pdf/2409.08780
• Copy Paste: [[2409.08780]] Sign Language Sense Disambiguation(https://arxiv.org/abs/2409.08780)
• Keywords: transformer
• Abstract: This project explores methods to enhance sign language translation of German sign language, specifically focusing on disambiguation of homonyms. Sign language is ambiguous and understudied which is the basis for our experiments. We approach the improvement by training transformer-based models on various bodypart representations to shift the focus on said bodypart. To determine the impact of, e.g., the hand or mouth representations, we experiment with different combinations. The results show that focusing on the mouth increases the performance in small dataset settings while shifting the focus on the hands retrieves better results in larger dataset settings. Our results contribute to better accessibility for non-hearing persons by improving the systems powering digital assistants, enabling a more accurate interaction. The code for this project can be found on GitHub.

Title: Contactless Fingerprint Recognition Using 3D Graph Matching

• Authors: Zhe Cui, Yuwei Jia, Siyang Zheng, Fei Su
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08782
• Pdf URL: https://arxiv.org/pdf/2409.08782
• Copy Paste: [[2409.08782]] Contactless Fingerprint Recognition Using 3D Graph Matching(https://arxiv.org/abs/2409.08782)
• Keywords: extraction
• Abstract: Contactless fingerprint is a newly developed type of fingerprint, and has gained lots of attention in recent fingerprint studies. However, most existing contactless fingerprint algorithms treat contactless fingerprints as 2D plain fingerprints, and utilize similar recognition methods as traditional contact-based 2D fingerprints. This recognition approach does not consider the modality difference between contactless and contact fingerprints, especially the intrinsic 3D characteristic of contactless fingerprints. This paper proposes a novel contactless fingerprint recognition algorithm that captures the revealed 3D feature of contactless fingerprints rather than the plain 2D feature. The proposed method first recovers 3D features from the input contactless fingerprint, including the 3D shape model and 3D fingerprint feature (minutiae, orientation, etc.). Then, a novel 3D graph matching is conducted in 3D space according to the extracted 3D feature. Our method captures the real 3D nature of contactless fingerprints as the whole feature extraction and matching algorithms are completed in real 3D space. Experiments results on contactless fingerprint databases show that the proposed method successfully improves the matching accuracy of contactless fingerprints. Exceptionally, our method performs stably across multiple poses of contactless fingerprints due to 3D graph matching, which is a great advantage compared to previous contactless fingerprint recognition algorithms.

Title: Double Index Calculus Algorithm: Faster Solving Discrete Logarithm Problem in Finite Prime Field

• Authors: Wen Huang, Zhishuo Zhang, Weixin Zhao, Jian Peng, Yongjian Liao, Yuyu Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08784
• Pdf URL: https://arxiv.org/pdf/2409.08784
• Copy Paste: [[2409.08784]] Double Index Calculus Algorithm: Faster Solving Discrete Logarithm Problem in Finite Prime Field(https://arxiv.org/abs/2409.08784)
• Keywords: security
• Abstract: Solving the discrete logarithm problem in a finite prime field is an extremely important computing problem in modern cryptography. The hardness of solving the discrete logarithm problem in a finite prime field is the security foundation of numerous cryptography schemes. In this paper, we propose the double index calculus algorithm to solve the discrete logarithm problem in a finite prime field. Our algorithm is faster than the index calculus algorithm, which is the state-of-the-art algorithm for solving the discrete logarithm problem in a finite prime field. Empirical experiment results indicate that our algorithm could be more than a 30-fold increase in computing speed than the index calculus algorithm when the bit length of the order of prime field is 70 bits. In addition, our algorithm is more general than the index calculus algorithm. Specifically, when the base of the target discrete logarithm problem is not the multiplication generator, the index calculus algorithm may fail to solve the discrete logarithm problem while our algorithm still can work.

Title: Electrocardiogram Report Generation and Question Answering via Retrieval-Augmented Self-Supervised Modeling

• Authors: Jialu Tang, Tong Xia, Yuan Lu, Cecilia Mascolo, Aaqib Saeed
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08788
• Pdf URL: https://arxiv.org/pdf/2409.08788
• Copy Paste: [[2409.08788]] Electrocardiogram Report Generation and Question Answering via Retrieval-Augmented Self-Supervised Modeling(https://arxiv.org/abs/2409.08788)
• Keywords: large language model
• Abstract: Interpreting electrocardiograms (ECGs) and generating comprehensive reports remain challenging tasks in cardiology, often requiring specialized expertise and significant time investment. To address these critical issues, we propose ECG-ReGen, a retrieval-based approach for ECG-to-text report generation and question answering. Our method leverages a self-supervised learning for the ECG encoder, enabling efficient similarity searches and report retrieval. By combining pre-training with dynamic retrieval and Large Language Model (LLM)-based refinement, ECG-ReGen effectively analyzes ECG data and answers related queries, with the potential of improving patient care. Experiments conducted on the PTB-XL and MIMIC-IV-ECG datasets demonstrate superior performance in both in-domain and cross-domain scenarios for report generation. Furthermore, our approach exhibits competitive performance on ECG-QA dataset compared to fully supervised methods when utilizing off-the-shelf LLMs for zero-shot question answering. This approach, effectively combining self-supervised encoder and LLMs, offers a scalable and efficient solution for accurate ECG interpretation, holding significant potential to enhance clinical decision-making.

Title: Optimizing Ingredient Substitution Using Large Language Models to Enhance Phytochemical Content in Recipes

• Authors: Luis Rita, Josh Southern, Ivan Laponogov, Kyle Higgins, Kirill Veselkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08792
• Pdf URL: https://arxiv.org/pdf/2409.08792
• Copy Paste: [[2409.08792]] Optimizing Ingredient Substitution Using Large Language Models to Enhance Phytochemical Content in Recipes(https://arxiv.org/abs/2409.08792)
• Keywords: large language model
• Abstract: In the emerging field of computational gastronomy, aligning culinary practices with scientifically supported nutritional goals is increasingly important. This study explores how large language models (LLMs) can be applied to optimize ingredient substitutions in recipes, specifically to enhance the phytochemical content of meals. Phytochemicals are bioactive compounds found in plants, which, based on preclinical studies, may offer potential health benefits. We fine-tuned models, including OpenAI's GPT-3.5, DaVinci, and Meta's TinyLlama, using an ingredient substitution dataset. These models were used to predict substitutions that enhance phytochemical content and create a corresponding enriched recipe dataset. Our approach improved Hit@1 accuracy on ingredient substitution tasks, from the baseline 34.53 plus-minus 0.10% to 38.03 plus-minus 0.28% on the original GISMo dataset, and from 40.24 plus-minus 0.36% to 54.46 plus-minus 0.29% on a refined version of the same dataset. These substitutions led to the creation of 1,951 phytochemically enriched ingredient pairings and 1,639 unique recipes. While this approach demonstrates potential in optimizing ingredient substitutions, caution must be taken when drawing conclusions about health benefits, as the claims are based on preclinical evidence. Future work should include clinical validation and broader datasets to further evaluate the nutritional impact of these substitutions. This research represents a step forward in using AI to promote healthier eating practices, providing potential pathways for integrating computational methods with nutritional science.

Title: TabKANet: Tabular Data Modelling with Kolmogorov-Arnold Network and Transformer

• Authors: Weihao Gao, Zheng Gong, Zhuo Deng, Fuju Rong, Chucheng Chen, Lan Ma
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08806
• Pdf URL: https://arxiv.org/pdf/2409.08806
• Copy Paste: [[2409.08806]] TabKANet: Tabular Data Modelling with Kolmogorov-Arnold Network and Transformer(https://arxiv.org/abs/2409.08806)
• Keywords: transformer
• Abstract: Tabular data is the most common type of data in real-life scenarios. In this study, we propose a method based on the TabKANet architecture, which utilizes the Kolmogorov-Arnold network to encode numerical features and merge them with categorical features, enabling unified modeling of tabular data on the Transformer architecture. This model demonstrates outstanding performance in six widely used binary classification tasks, suggesting that TabKANet has the potential to become a standard approach for tabular modeling, surpassing traditional neural networks. Furthermore, this research reveals the significant advantages of the Kolmogorov-Arnold network in encoding numerical features. The code of our work is available at this https URL.

Title: Your Weak LLM is Secretly a Strong Teacher for Alignment

• Authors: Leitian Tao, Yixuan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08813
• Pdf URL: https://arxiv.org/pdf/2409.08813
• Copy Paste: [[2409.08813]] Your Weak LLM is Secretly a Strong Teacher for Alignment(https://arxiv.org/abs/2409.08813)
• Keywords: large language model
• Abstract: The burgeoning capabilities of large language models (LLMs) have underscored the need for alignment to ensure these models act in accordance with human values and intentions. Existing alignment frameworks present constraints either in the form of expensive human effort or high computational costs. This paper explores a promising middle ground, where we employ a weak LLM that is significantly less resource-intensive than top-tier models, yet offers more automation than purely human feedback. We present a systematic study to evaluate and understand weak LLM's ability to generate feedback for alignment. Our empirical findings demonstrate that weak LLMs can provide feedback that rivals or even exceeds that of fully human-annotated data. Our study indicates a minimized impact of model size on feedback efficacy, shedding light on a scalable and sustainable alignment strategy. To deepen our understanding of alignment under weak LLM feedback, we conduct a series of qualitative and quantitative analyses, offering novel insights into the quality discrepancies between human feedback vs. weak LLM feedback.

Title: Pathfinder for Low-altitude Aircraft with Binary Neural Network

• Authors: Kaijie Yin, Tian Gao, Hui Kong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08824
• Pdf URL: https://arxiv.org/pdf/2409.08824
• Copy Paste: [[2409.08824]] Pathfinder for Low-altitude Aircraft with Binary Neural Network(https://arxiv.org/abs/2409.08824)
• Keywords: extraction, transformer, segmentation
• Abstract: A prior global topological map (e.g., the OpenStreetMap, OSM) can boost the performance of autonomous mapping by a ground mobile robot. However, the prior map is usually incomplete due to lacking labeling in partial paths. To solve this problem, this paper proposes an OSM maker using airborne sensors carried by low-altitude aircraft, where the core of the OSM maker is a novel efficient pathfinder approach based on LiDAR and camera data, i.e., a binary dual-stream road segmentation model. Specifically, a multi-scale feature extraction based on the UNet architecture is implemented for images and point clouds. To reduce the effect caused by the sparsity of point cloud, an attention-guided gated block is designed to integrate image and point-cloud features. For enhancing the efficiency of the model, we propose a binarization streamline to each model component, including a variant of vision transformer (ViT) architecture as the encoder of the image branch, and new focal and perception losses to optimize the model training. The experimental results on two datasets demonstrate that our pathfinder method achieves SOTA accuracy with high efficiency in finding paths from the low-level airborne sensors, and we can create complete OSM prior maps based on the segmented road skeletons. Code and data are available at:this https URL}{this https URL.

Title: Breaking reCAPTCHAv2

• Authors: Andreas Plesner, Tobias Vontobel, Roger Wattenhofer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08831
• Pdf URL: https://arxiv.org/pdf/2409.08831
• Copy Paste: [[2409.08831]] Breaking reCAPTCHAv2(https://arxiv.org/abs/2409.08831)
• Keywords: segmentation
• Abstract: Our work examines the efficacy of employing advanced machine learning methods to solve captchas from Google's reCAPTCHAv2 system. We evaluate the effectiveness of automated systems in solving captchas by utilizing advanced YOLO models for image segmentation and classification. Our main result is that we can solve 100% of the captchas, while previous work only solved 68-71%. Furthermore, our findings suggest that there is no significant difference in the number of challenges humans and bots must solve to pass the captchas in reCAPTCHAv2. This implies that current AI technologies can exploit advanced image-based captchas. We also look under the hood of reCAPTCHAv2, and find evidence that reCAPTCHAv2 is heavily based on cookie and browser history data when evaluating whether a user is human or not. The code is provided alongside this paper.

Title: Can Kans (re)discover predictive models for Direct-Drive Laser Fusion?

• Authors: Rahman Ejaz, Varchas Gopalaswamy, Riccardo Betti, Aarne Lees, Christopher Kanan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08832
• Pdf URL: https://arxiv.org/pdf/2409.08832
• Copy Paste: [[2409.08832]] Can Kans (re)discover predictive models for Direct-Drive Laser Fusion?(https://arxiv.org/abs/2409.08832)
• Keywords: interpretability
• Abstract: The domain of laser fusion presents a unique and challenging predictive modeling application landscape for machine learning methods due to high problem complexity and limited training data. Data-driven approaches utilizing prescribed functional forms, inductive biases and physics-informed learning (PIL) schemes have been successful in the past for achieving desired generalization ability and model interpretation that aligns with physics expectations. In complex multi-physics application domains, however, it is not always obvious how architectural biases or discriminative penalties can be formulated. In this work, focusing on nuclear fusion energy using high powered lasers, we present the use of Kolmogorov-Arnold Networks (KANs) as an alternative to PIL for developing a new type of data-driven predictive model which is able to achieve high prediction accuracy and physics interpretability. A KAN based model, a MLP with PIL, and a baseline MLP model are compared in generalization ability and interpretation with a domain expert-derived symbolic regression model. Through empirical studies in this high physics complexity domain, we show that KANs can potentially provide benefits when developing predictive models for data-starved physics applications.

Title: AIPO: Improving Training Objective for Iterative Preference Optimization

• Authors: Yaojie Shen, Xinyao Wang, Yulei Niu, Ying Zhou, Lexin Tang, Libo Zhang, Fan Chen, Longyin Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08845
• Pdf URL: https://arxiv.org/pdf/2409.08845
• Copy Paste: [[2409.08845]] AIPO: Improving Training Objective for Iterative Preference Optimization(https://arxiv.org/abs/2409.08845)
• Keywords: large language model
• Abstract: Preference Optimization (PO), is gaining popularity as an alternative choice of Proximal Policy Optimization (PPO) for aligning Large Language Models (LLMs). Recent research on aligning LLMs iteratively with synthetic or partially synthetic data shows promising results in scaling up PO training for both academic settings and proprietary trained models such as Llama3. Despite its success, our study shows that the length exploitation issue present in PO is even more severe in Iterative Preference Optimization (IPO) due to the iterative nature of the process. In this work, we study iterative preference optimization with synthetic data. We share the findings and analysis along the way of building the iterative preference optimization pipeline. More specifically, we discuss the length exploitation issue during iterative preference optimization and propose our training objective for iterative preference optimization, namely Agreement-aware Iterative Preference Optimization (AIPO). To demonstrate the effectiveness of our method, we conduct comprehensive experiments and achieve state-of-the-art performance on MT-Bench, AlpacaEval 2.0, and Arena-Hard. Our implementation and model checkpoints will be made available at this https URL.

Title: FP-VEC: Fingerprinting Large Language Models via Efficient Vector Addition

• Authors: Zhenhua Xu, Wenpeng Xing, Zhebo Wang, Chang Hu, Chen Jie, Meng Han
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08846
• Pdf URL: https://arxiv.org/pdf/2409.08846
• Copy Paste: [[2409.08846]] FP-VEC: Fingerprinting Large Language Models via Efficient Vector Addition(https://arxiv.org/abs/2409.08846)
• Keywords: protect, large language model
• Abstract: Training Large Language Models (LLMs) requires immense computational power and vast amounts of data. As a result, protecting the intellectual property of these models through fingerprinting is essential for ownership authentication. While adding fingerprints to LLMs through fine-tuning has been attempted, it remains costly and unscalable. In this paper, we introduce FP-VEC, a pilot study on using fingerprint vectors as an efficient fingerprinting method for LLMs. Our approach generates a fingerprint vector that represents a confidential signature embedded in the model, allowing the same fingerprint to be seamlessly incorporated into an unlimited number of LLMs via vector addition. Results on several LLMs show that FP-VEC is lightweight by running on CPU-only devices for fingerprinting, scalable with a single training and unlimited fingerprinting process, and preserves the model's normal behavior. The project page is available at this https URL .

Title: Kinect Calibration and Data Optimization For Anthropometric Parameters

• Authors: M.S. Gokmen, M. Akbaba, O. Findik
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08847
• Pdf URL: https://arxiv.org/pdf/2409.08847
• Copy Paste: [[2409.08847]] Kinect Calibration and Data Optimization For Anthropometric Parameters(https://arxiv.org/abs/2409.08847)
• Keywords: biometric
• Abstract: Recently, through development of several 3d vision systems, widely used in various applications, medical and biometric fields. Microsoft kinect sensor have been most of used camera among 3d vision systems. Microsoft kinect sensor can obtain depth images of a scene and 3d coordinates of human joints. Thus, anthropometric features can extractable easily. Anthropometric feature and 3d joint coordinate raw datas which captured from kinect sensor is unstable. The strongest reason for this, datas vary by distance between joints of individual and location of kinect sensor. Consequently, usage of this datas without kinect calibration and data optimization does not result in sufficient and healthy. In this study, proposed a novel method to calibrating kinect sensor and optimizing skeleton features. Results indicate that the proposed method is quite effective and worthy of further study in more general scenarios.

Title: DeCLIP: Decoding CLIP representations for deepfake localization

• Authors: Stefan Smeu, Elisabeta Oneata, Dan Oneata
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08849
• Pdf URL: https://arxiv.org/pdf/2409.08849
• Copy Paste: [[2409.08849]] DeCLIP: Decoding CLIP representations for deepfake localization(https://arxiv.org/abs/2409.08849)
• Keywords: robust, diffusion, generative
• Abstract: Generative models can create entirely new images, but they can also partially modify real images in ways that are undetectable to the human eye. In this paper, we address the challenge of automatically detecting such local manipulations. One of the most pressing problems in deepfake detection remains the ability of models to generalize to different classes of generators. In the case of fully manipulated images, representations extracted from large self-supervised models (such as CLIP) provide a promising direction towards more robust detectors. Here, we introduce DeCLIP, a first attempt to leverage such large pretrained features for detecting local manipulations. We show that, when combined with a reasonably large convolutional decoder, pretrained self-supervised representations are able to perform localization and improve generalization capabilities over existing methods. Unlike previous work, our approach is able to perform localization on the challenging case of latent diffusion models, where the entire image is affected by the fingerprint of the generator. Moreover, we observe that this type of data, which combines local semantic information with a global fingerprint, provides more stable generalization than other categories of generative methods.

Title: InstantDrag: Improving Interactivity in Drag-based Image Editing

• Authors: Joonghyuk Shin, Daehyeon Choi, Jaesik Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08857
• Pdf URL: https://arxiv.org/pdf/2409.08857
• Copy Paste: [[2409.08857]] InstantDrag: Improving Interactivity in Drag-based Image Editing(https://arxiv.org/abs/2409.08857)
• Keywords: diffusion
• Abstract: Drag-based image editing has recently gained popularity for its interactivity and precision. However, despite the ability of text-to-image models to generate samples within a second, drag editing still lags behind due to the challenge of accurately reflecting user interaction while maintaining image content. Some existing approaches rely on computationally intensive per-image optimization or intricate guidance-based methods, requiring additional inputs such as masks for movable regions and text prompts, thereby compromising the interactivity of the editing process. We introduce InstantDrag, an optimization-free pipeline that enhances interactivity and speed, requiring only an image and a drag instruction as input. InstantDrag consists of two carefully designed networks: a drag-conditioned optical flow generator (FlowGen) and an optical flow-conditioned diffusion model (FlowDiffusion). InstantDrag learns motion dynamics for drag-based image editing in real-world video datasets by decomposing the task into motion generation and motion-conditioned image generation. We demonstrate InstantDrag's capability to perform fast, photo-realistic edits without masks or text prompts through experiments on facial video datasets and general scenes. These results highlight the efficiency of our approach in handling drag-based image editing, making it a promising solution for interactive, real-time applications.

Title: Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control

• Authors: Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, Ricky T. Q. Chen
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.08861
• Pdf URL: https://arxiv.org/pdf/2409.08861
• Copy Paste: [[2409.08861]] Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control(https://arxiv.org/abs/2409.08861)
• Keywords: diffusion, generative
• Abstract: Dynamical generative models that produce samples through an iterative process, such as Flow Matching and denoising diffusion models, have seen widespread use, but there has not been many theoretically-sound methods for improving these models with reward fine-tuning. In this work, we cast reward fine-tuning as stochastic optimal control (SOC). Critically, we prove that a very specific memoryless noise schedule must be enforced during fine-tuning, in order to account for the dependency between the noise variable and the generated samples. We also propose a new algorithm named Adjoint Matching which outperforms existing SOC algorithms, by casting SOC problems as a regression problem. We find that our approach significantly improves over existing methods for reward fine-tuning, achieving better consistency, realism, and generalization to unseen human preference reward models, while retaining sample diversity.

Title: Exploring Graph Structure Comprehension Ability of Multimodal Large Language Models: Case Studies

• Authors: Zhiqiang Zhong, Davide Mottin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08864
• Pdf URL: https://arxiv.org/pdf/2409.08864
• Copy Paste: [[2409.08864]] Exploring Graph Structure Comprehension Ability of Multimodal Large Language Models: Case Studies(https://arxiv.org/abs/2409.08864)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities in processing various data structures, including graphs. While previous research has focused on developing textual encoding methods for graph representation, the emergence of multimodal LLMs presents a new frontier for graph comprehension. These advanced models, capable of processing both text and images, offer potential improvements in graph understanding by incorporating visual representations alongside traditional textual data. This study investigates the impact of graph visualisations on LLM performance across a range of benchmark tasks at node, edge, and graph levels. Our experiments compare the effectiveness of multimodal approaches against purely textual graph representations. The results provide valuable insights into both the potential and limitations of leveraging visual graph modalities to enhance LLMs' graph structure comprehension abilities.

Title: Detect Fake with Fake: Leveraging Synthetic Data-driven Representation for Synthetic Image Detection

• Authors: Hina Otake, Yoshihiro Fukuhara, Yoshiki Kubotani, Shigeo Morishima
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08884
• Pdf URL: https://arxiv.org/pdf/2409.08884
• Copy Paste: [[2409.08884]] Detect Fake with Fake: Leveraging Synthetic Data-driven Representation for Synthetic Image Detection(https://arxiv.org/abs/2409.08884)
• Keywords: transformer
• Abstract: Are general-purpose visual representations acquired solely from synthetic data useful for detecting fake images? In this work, we show the effectiveness of synthetic data-driven representations for synthetic image detection. Upon analysis, we find that vision transformers trained by the latest visual representation learners with synthetic data can effectively distinguish fake from real images without seeing any real images during pre-training. Notably, using SynCLR as the backbone in a state-of-the-art detection method demonstrates a performance improvement of +10.32 mAP and +4.73% accuracy over the widely used CLIP, when tested on previously unseen GAN models. Code is available at this https URL.

Title: Visual Language Tracking with Multi-modal Interaction: A Robust Benchmark

• Authors: Xuchen Li, Shiyu Hu, Xiaokun Feng, Dailing Zhang, Meiqi Wu, Jing Zhang, Kaiqi Huang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08887
• Pdf URL: https://arxiv.org/pdf/2409.08887
• Copy Paste: [[2409.08887]] Visual Language Tracking with Multi-modal Interaction: A Robust Benchmark(https://arxiv.org/abs/2409.08887)
• Keywords: robust
• Abstract: Visual Language Tracking (VLT) enhances tracking by mitigating the limitations of relying solely on the visual modality, utilizing high-level semantic information through language. This integration of the language enables more advanced human-machine interaction. The essence of interaction is cognitive alignment, which typically requires multiple information exchanges, especially in the sequential decision-making process of VLT. However, current VLT benchmarks do not account for multi-round interactions during tracking. They provide only an initial text and bounding box (bbox) in the first frame, with no further interaction as tracking progresses, deviating from the original motivation of the VLT task. To address these limitations, we propose a novel and robust benchmark, VLT-MI (Visual Language Tracking with Multi-modal Interaction), which introduces multi-round interaction into the VLT task for the first time. (1) We generate diverse, multi-granularity texts for multi-round, multi-modal interaction based on existing mainstream VLT benchmarks using DTLLM-VLT, leveraging the world knowledge of LLMs. (2) We propose a new VLT interaction paradigm that achieves multi-round interaction through text updates and object recovery. When multiple tracking failures occur, we provide the tracker with more aligned texts and corrected bboxes through interaction, thereby expanding the scope of VLT downstream tasks. (3) We conduct comparative experiments on both traditional VLT benchmarks and VLT-MI, evaluating and analyzing the accuracy and robustness of trackers under the interactive paradigm. This work offers new insights and paradigms for the VLT task, enabling a fine-grained evaluation of multi-modal trackers. We believe this approach can be extended to additional datasets in the future, supporting broader evaluations and comparisons of video-language model capabilities.

Title: Latent Space Score-based Diffusion Model for Probabilistic Multivariate Time Series Imputation

• Authors: Guojun Liang, Najmeh Abiri, Atiye Sadat Hashemi, Jens Lundström, Stefan Byttner, Prayag Tiwari
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.08917
• Pdf URL: https://arxiv.org/pdf/2409.08917
• Copy Paste: [[2409.08917]] Latent Space Score-based Diffusion Model for Probabilistic Multivariate Time Series Imputation(https://arxiv.org/abs/2409.08917)
• Keywords: diffusion, generative
• Abstract: Accurate imputation is essential for the reliability and success of downstream tasks. Recently, diffusion models have attracted great attention in this field. However, these models neglect the latent distribution in a lower-dimensional space derived from the observed data, which limits the generative capacity of the diffusion model. Additionally, dealing with the original missing data without labels becomes particularly problematic. To address these issues, we propose the Latent Space Score-Based Diffusion Model (LSSDM) for probabilistic multivariate time series imputation. Observed values are projected onto low-dimensional latent space and coarse values of the missing data are reconstructed without knowing their ground truth values by this unsupervised learning approach. Finally, the reconstructed values are fed into a conditional diffusion model to obtain the precise imputed values of the time series. In this way, LSSDM not only possesses the power to identify the latent distribution but also seamlessly integrates the diffusion model to obtain the high-fidelity imputed values and assess the uncertainty of the dataset. Experimental results demonstrate that LSSDM achieves superior imputation performance while also providing a better explanation and uncertainty analysis of the imputation mechanism. The website of the code is \textit{this https URL\_imputation}.

Title: XSub: Explanation-Driven Adversarial Attack against Blackbox Classifiers via Feature Substitution

• Authors: Kiana Vu, Phung Lai, Truc Nguyen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08919
• Pdf URL: https://arxiv.org/pdf/2409.08919
• Copy Paste: [[2409.08919]] XSub: Explanation-Driven Adversarial Attack against Blackbox Classifiers via Feature Substitution(https://arxiv.org/abs/2409.08919)
• Keywords: attack, steal
• Abstract: Despite its significant benefits in enhancing the transparency and trustworthiness of artificial intelligence (AI) systems, explainable AI (XAI) has yet to reach its full potential in real-world applications. One key challenge is that XAI can unintentionally provide adversaries with insights into black-box models, inevitably increasing their vulnerability to various attacks. In this paper, we develop a novel explanation-driven adversarial attack against black-box classifiers based on feature substitution, called XSub. The key idea of XSub is to strategically replace important features (identified via XAI) in the original sample with corresponding important features from a "golden sample" of a different label, thereby increasing the likelihood of the model misclassifying the perturbed sample. The degree of feature substitution is adjustable, allowing us to control how much of the original samples information is replaced. This flexibility effectively balances a trade-off between the attacks effectiveness and its stealthiness. XSub is also highly cost-effective in that the number of required queries to the prediction model and the explanation model in conducting the attack is in O(1). In addition, XSub can be easily extended to launch backdoor attacks in case the attacker has access to the models training data. Our evaluation demonstrates that XSub is not only effective and stealthy but also cost-effective, enabling its application across a wide range of AI models.

Title: Pushing Joint Image Denoising and Classification to the Edge

• Authors: Thomas C Markhorst, Jan C van Gemert, Osman S Kayhan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.08943
• Pdf URL: https://arxiv.org/pdf/2409.08943
• Copy Paste: [[2409.08943]] Pushing Joint Image Denoising and Classification to the Edge(https://arxiv.org/abs/2409.08943)
• Keywords: security
• Abstract: In this paper, we jointly combine image classification and image denoising, aiming to enhance human perception of noisy images captured by edge devices, like low-light security cameras. In such settings, it is important to retain the ability of humans to verify the automatic classification decision and thus jointly denoise the image to enhance human perception. Since edge devices have little computational power, we explicitly optimize for efficiency by proposing a novel architecture that integrates the two tasks. Additionally, we alter a Neural Architecture Search (NAS) method, which searches for classifiers to search for the integrated model while optimizing for a target latency, classification accuracy, and denoising performance. The NAS architectures outperform our manually designed alternatives in both denoising and classification, offering a significant improvement to human perception. Our approach empowers users to construct architectures tailored to domains like medical imaging, surveillance systems, and industrial inspections.

Title: A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

• Authors: Yohan Poirier-Ginter, Alban Gauthier, Julien Phillip, Jean-Francois Lalonde, George Drettakis
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.08947
• Pdf URL: https://arxiv.org/pdf/2409.08947
• Copy Paste: [[2409.08947]] A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis(https://arxiv.org/abs/2409.08947)
• Keywords: diffusion
• Abstract: Relighting radiance fields is severely underconstrained for multi-view data, which is most often captured under a single illumination condition; It is especially hard for full scenes containing multiple objects. We introduce a method to create relightable radiance fields using such single-illumination data by exploiting priors extracted from 2D image diffusion models. We first fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by light direction, allowing us to augment a single-illumination capture into a realistic -- but possibly inconsistent -- multi-illumination dataset from directly defined light directions. We use this augmented data to create a relightable radiance field represented by 3D Gaussian splats. To allow direct control of light direction for low-frequency lighting, we represent appearance with a multi-layer perceptron parameterized on light direction. To enforce multi-view consistency and overcome inaccuracies we optimize a per-image auxiliary feature vector. We show results on synthetic and real multi-view data under single illumination, demonstrating that our method successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes. Project site this https URL

Title: National Treasure: The Call for e-Democracy and US Election Security

• Authors: Adam Dorian Wong
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.08952
• Pdf URL: https://arxiv.org/pdf/2409.08952
• Copy Paste: [[2409.08952]] National Treasure: The Call for e-Democracy and US Election Security(https://arxiv.org/abs/2409.08952)
• Keywords: security, protect, defense, attack
• Abstract: Faith in the US electoral system is at risk. This issue stems from trust or lack thereof. Poor leaders ranted and attempted to sew discord in the democratic process and even tried to influence election results. Historically, the US has relied on paper ballots to cast private votes. Votes are watered down by the Electoral College. Elections are contested due to voter IDs and proof of citizenship. Methods of voting are nonsensically complex. In the technology age, this can be solved with a Smartcard National ID backed by Public-Key Infrastructure (PKI). This could be a method to restore hope in democracy and move the country back towards elections under a Popular Vote. Numbers are empirical and immutable and can solve the issue of Election Security in a bipartisan way. NATO allies like Estonia have already broken ground in using technology for eDemocracy or (Internet-based) iVoting. Acknowledging cyber attacks will happen, this is an opportunity for DHS and DOD (CYBERCOM) to collaborate on domestic operations and protect critical election infrastructure. This idea will not fix malicious information operations or civil stupidity. However, this is the way forward to securing elections now and forever. The views expressed by this whitepaper are those of the author and do not reflect the official policy or position of Dakota State University, the N.H. Army National Guard, the U.S. Army, the Department of Defense, or the U.S. Government. Cleared for release by DOPSR on 13 SEP 2024.

Title: PINNfluence: Influence Functions for Physics-Informed Neural Networks

• Authors: Jonas R. Naujoks, Aleksander Krasowski, Moritz Weckbecker, Thomas Wiegand, Sebastian Lapuschkin, Wojciech Samek, René P. Klausen
• Subjects: cs.LG, cs.AI, physics.comp-ph, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2409.08958
• Pdf URL: https://arxiv.org/pdf/2409.08958
• Copy Paste: [[2409.08958]] PINNfluence: Influence Functions for Physics-Informed Neural Networks(https://arxiv.org/abs/2409.08958)
• Keywords: interpretability
• Abstract: Recently, physics-informed neural networks (PINNs) have emerged as a flexible and promising application of deep learning to partial differential equations in the physical sciences. While offering strong performance and competitive inference speeds on forward and inverse problems, their black-box nature limits interpretability, particularly regarding alignment with expected physical behavior. In the present work, we explore the application of influence functions (IFs) to validate and debug PINNs post-hoc. Specifically, we apply variations of IF-based indicators to gauge the influence of different types of collocation points on the prediction of PINNs applied to a 2D Navier-Stokes fluid flow problem. Our results demonstrate how IFs can be adapted to PINNs to reveal the potential for further studies.

Title: Clean Label Attacks against SLU Systems

• Authors: Henry Li Xinyuan, Sonal Joshi, Thomas Thebaud, Jesus Villalba, Najim Dehak, Sanjeev Khudanpur
• Subjects: cs.CR, cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.08985
• Pdf URL: https://arxiv.org/pdf/2409.08985
• Copy Paste: [[2409.08985]] Clean Label Attacks against SLU Systems(https://arxiv.org/abs/2409.08985)
• Keywords: defense, attack
• Abstract: Poisoning backdoor attacks involve an adversary manipulating the training data to induce certain behaviors in the victim model by inserting a trigger in the signal at inference time. We adapted clean label backdoor (CLBD)-data poisoning attacks, which do not modify the training labels, on state-of-the-art speech recognition models that support/perform a Spoken Language Understanding task, achieving 99.8% attack success rate by poisoning 10% of the training data. We analyzed how varying the signal-strength of the poison, percent of samples poisoned, and choice of trigger impact the attack. We also found that CLBD attacks are most successful when applied to training samples that are inherently hard for a proxy model. Using this strategy, we achieved an attack success rate of 99.3% by poisoning a meager 1.5% of the training data. Finally, we applied two previously developed defenses against gradient-based attacks, and found that they attain mixed success against poisoning.

Title: E2MoCase: A Dataset for Emotional, Event and Moral Observations in News Articles on High-impact Legal Cases

• Authors: Candida M. Greco, Lorenzo Zangari, Davide Picca, Andrea Tagarelli
• Subjects: cs.CL, cs.AI, cs.CY, cs.DL, physics.soc-ph
• Abstract URL: https://arxiv.org/abs/2409.09001
• Pdf URL: https://arxiv.org/pdf/2409.09001
• Copy Paste: [[2409.09001]] E2MoCase: A Dataset for Emotional, Event and Moral Observations in News Articles on High-impact Legal Cases(https://arxiv.org/abs/2409.09001)
• Keywords: extraction
• Abstract: The way media reports on legal cases can significantly shape public opinion, often embedding subtle biases that influence societal views on justice and morality. Analyzing these biases requires a holistic approach that captures the emotional tone, moral framing, and specific events within the narratives. In this work we introduce E2MoCase, a novel dataset designed to facilitate the integrated analysis of emotions, moral values, and events within legal narratives and media coverage. By leveraging advanced models for emotion detection, moral value identification, and event extraction, E2MoCase offers a multi-dimensional perspective on how legal cases are portrayed in news articles.

Title: SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity

• Authors: Qitian Wu, Kai Yang, Hengrui Zhang, David Wipf, Junchi Yan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09007
• Pdf URL: https://arxiv.org/pdf/2409.09007
• Copy Paste: [[2409.09007]] SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity(https://arxiv.org/abs/2409.09007)
• Keywords: transformer
• Abstract: Learning representations on large graphs is a long-standing challenge due to the inter-dependence nature. Transformers recently have shown promising performance on small graphs thanks to its global attention for capturing all-pair interactions beyond observed structures. Existing approaches tend to inherit the spirit of Transformers in language and vision tasks, and embrace complicated architectures by stacking deep attention-based propagation layers. In this paper, we attempt to evaluate the necessity of adopting multi-layer attentions in Transformers on graphs, which considerably restricts the efficiency. Specifically, we analyze a generic hybrid propagation layer, comprised of all-pair attention and graph-based propagation, and show that multi-layer propagation can be reduced to one-layer propagation, with the same capability for representation learning. It suggests a new technical path for building powerful and efficient Transformers on graphs, particularly through simplifying model architectures without sacrificing expressiveness. As exemplified by this work, we propose a Simplified Single-layer Graph Transformers (SGFormer), whose main component is a single-layer global attention that scales linearly w.r.t. graph sizes and requires none of any approximation for accommodating all-pair interactions. Empirically, SGFormer successfully scales to the web-scale graph ogbn-papers100M, yielding orders-of-magnitude inference acceleration over peer Transformers on medium-sized graphs, and demonstrates competitiveness with limited labeled data.

Title: Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach

• Authors: Siqi Li, Danni Liu, Jan Niehues
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.09009
• Pdf URL: https://arxiv.org/pdf/2409.09009
• Copy Paste: [[2409.09009]] Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach(https://arxiv.org/abs/2409.09009)
• Keywords: robust
• Abstract: Direct speech translation (ST) models often struggle with rare words. Incorrect translation of these words can have severe consequences, impacting translation quality and user trust. While rare word translation is inherently challenging for neural models due to sparse learning signals, real-world scenarios often allow access to translations of past recordings on similar topics. To leverage these valuable resources, we propose a retrieval-and-demonstration approach to enhance rare word translation accuracy in direct ST models. First, we adapt existing ST models to incorporate retrieved examples for rare word translation, which allows the model to benefit from prepended examples, similar to in-context learning. We then develop a cross-modal (speech-to-speech, speech-to-text, text-to-text) retriever to locate suitable examples. We demonstrate that standard ST models can be effectively adapted to leverage examples for rare word translation, improving rare word translation accuracy over the baseline by 17.6% with gold examples and 8.5% with retrieved examples. Moreover, our speech-to-speech retrieval approach outperforms other modalities and exhibits higher robustness to unseen speakers. Our code is publicly available (this https URL).

Title: An Efficient and Streaming Audio Visual Active Speaker Detection System

• Authors: Arnav Kundu, Yanzi Jin, Mohammad Sekhavat, Max Horton, Danny Tormoen, Devang Naik
• Subjects: cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.09018
• Pdf URL: https://arxiv.org/pdf/2409.09018
• Copy Paste: [[2409.09018]] An Efficient and Streaming Audio Visual Active Speaker Detection System(https://arxiv.org/abs/2409.09018)
• Keywords: transformer
• Abstract: This paper delves into the challenging task of Active Speaker Detection (ASD), where the system needs to determine in real-time whether a person is speaking or not in a series of video frames. While previous works have made significant strides in improving network architectures and learning effective representations for ASD, a critical gap exists in the exploration of real-time system deployment. Existing models often suffer from high latency and memory usage, rendering them impractical for immediate applications. To bridge this gap, we present two scenarios that address the key challenges posed by real-time constraints. First, we introduce a method to limit the number of future context frames utilized by the ASD model. By doing so, we alleviate the need for processing the entire sequence of future frames before a decision is made, significantly reducing latency. Second, we propose a more stringent constraint that limits the total number of past frames the model can access during inference. This tackles the persistent memory issues associated with running streaming ASD systems. Beyond these theoretical frameworks, we conduct extensive experiments to validate our approach. Our results demonstrate that constrained transformer models can achieve performance comparable to or even better than state-of-the-art recurrent models, such as uni-directional GRUs, with a significantly reduced number of context frames. Moreover, we shed light on the temporal memory requirements of ASD systems, revealing that larger past context has a more profound impact on accuracy than future context. When profiling on a CPU we find that our efficient architecture is memory bound by the amount of past context it can use and that the compute cost is negligible as compared to the memory cost.