2024-11-14

Title: Intelligent Green Efficiency for Intrusion Detection

• Authors: Pedro Pereira, Paulo Mendes, João Vitorino, Eva Maia, Isabel Praça
• Subjects: cs.CR, cs.LG, cs.PF
• Abstract URL: https://arxiv.org/abs/2411.08069
• Pdf URL: https://arxiv.org/pdf/2411.08069
• Copy Paste: [[2411.08069]] Intelligent Green Efficiency for Intrusion Detection(https://arxiv.org/abs/2411.08069)
• Keywords: attack
• Abstract: Artificial Intelligence (AI) has emerged in popularity recently, recording great progress in various industries. However, the environmental impact of AI is a growing concern, in terms of the energy consumption and carbon footprint of Machine Learning (ML) and Deep Learning (DL) models, making essential investigate Green AI, an attempt to reduce the climate impact of AI systems. This paper presents an assessment of different programming languages and Feature Selection (FS) methods to improve computation performance of AI focusing on Network Intrusion Detection (NID) and cyber-attack classification tasks. Experiments were conducted using five ML models - Random Forest, XGBoost, LightGBM, Multi-Layer Perceptron, and Long Short-Term Memory - implemented in four programming languages - Python, Java, R, and Rust - along with three FS methods - Information Gain, Recursive Feature Elimination, and Chi-Square. The obtained results demonstrated that FS plays an important role enhancing the computational efficiency of AI models without compromising detection accuracy, highlighting languages like Python and R, that benefit from a rich AI libraries environment. These conclusions can be useful to design efficient and sustainable AI systems that still provide a good generalization and a reliable detection.

Title: TIPO: Text to Image with Text Presampling for Prompt Optimization

• Authors: Shih-Ying Yeh, Sang-Hyun Park, Giyeong Oh, Min Song, Youngjae Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08127
• Pdf URL: https://arxiv.org/pdf/2411.08127
• Copy Paste: [[2411.08127]] TIPO: Text to Image with Text Presampling for Prompt Optimization(https://arxiv.org/abs/2411.08127)
• Keywords: large language model
• Abstract: TIPO (Text to Image with text pre-sampling for Prompt Optimization) is an innovative framework designed to enhance text-to-image (T2I) generation by language model (LM) for automatic prompt engineering. By refining and extending user-provided prompts, TIPO bridges the gap between simple inputs and the detailed prompts required for high-quality image generation. Unlike previous approaches that rely on Large Language Models (LLMs) or reinforcement learning (RL), TIPO adjusts user input prompts with the distribution of a trained prompt dataset, eliminating the need for complex runtime cost via lightweight model. This pre-sampling approach enables efficient and scalable prompt optimization, grounded in the model's training distribution. Experimental results demonstrate TIPO's effectiveness in improving aesthetic scores, reducing image corruption, and better aligning generated images with dataset distributions. These findings highlight the critical role of prompt engineering in T2I systems and open avenues for broader applications of automatic prompt refinement.

Title: CameraHMR: Aligning People with Perspective

• Authors: Priyanka Patel, Michael J. Black
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08128
• Pdf URL: https://arxiv.org/pdf/2411.08128
• Copy Paste: [[2411.08128]] CameraHMR: Aligning People with Perspective(https://arxiv.org/abs/2411.08128)
• Keywords: robust
• Abstract: We address the challenge of accurate 3D human pose and shape estimation from monocular images. The key to accuracy and robustness lies in high-quality training data. Existing training datasets containing real images with pseudo ground truth (pGT) use SMPLify to fit SMPL to sparse 2D joint locations, assuming a simplified camera with default intrinsics. We make two contributions that improve pGT accuracy. First, to estimate camera intrinsics, we develop a field-of-view prediction model (HumanFoV) trained on a dataset of images containing people. We use the estimated intrinsics to enhance the 4D-Humans dataset by incorporating a full perspective camera model during SMPLify fitting. Second, 2D joints provide limited constraints on 3D body shape, resulting in average-looking bodies. To address this, we use the BEDLAM dataset to train a dense surface keypoint detector. We apply this detector to the 4D-Humans dataset and modify SMPLify to fit the detected keypoints, resulting in significantly more realistic body shapes. Finally, we upgrade the HMR2.0 architecture to include the estimated camera parameters. We iterate model training and SMPLify fitting initialized with the previously trained model. This leads to more accurate pGT and a new model, CameraHMR, with state-of-the-art accuracy. Code and pGT are available for research purposes.

Title: Impactful Bit-Flip Search on Full-precision Models

• Authors: Nadav Benedek, Matan Levy, Mahmood Sharif
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.08133
• Pdf URL: https://arxiv.org/pdf/2411.08133
• Copy Paste: [[2411.08133]] Impactful Bit-Flip Search on Full-precision Models(https://arxiv.org/abs/2411.08133)
• Keywords: attack, steal
• Abstract: Neural networks have shown remarkable performance in various tasks, yet they remain susceptible to subtle changes in their input or model parameters. One particularly impactful vulnerability arises through the Bit-Flip Attack (BFA), where flipping a small number of critical bits in a model's parameters can severely degrade its performance. A common technique for inducing bit flips in DRAM is the Row-Hammer attack, which exploits frequent uncached memory accesses to alter data. Identifying susceptible bits can be achieved through exhaustive search or progressive layer-by-layer analysis, especially in quantized networks. In this work, we introduce Impactful Bit-Flip Search (IBS), a novel method for efficiently pinpointing and flipping critical bits in full-precision networks. Additionally, we propose a Weight-Stealth technique that strategically modifies the model's parameters in a way that maintains the float values within the original distribution, thereby bypassing simple range checks often used in tamper detection.

Title: Large Language Models Can Self-Improve in Long-context Reasoning

• Authors: Siheng Li, Cheng Yang, Zesen Cheng, Lemao Liu, Mo Yu, Yujiu Yang, Wai Lam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08147
• Pdf URL: https://arxiv.org/pdf/2411.08147
• Copy Paste: [[2411.08147]] Large Language Models Can Self-Improve in Long-context Reasoning(https://arxiv.org/abs/2411.08147)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved substantial progress in processing long contexts but still struggle with long-context reasoning. Existing approaches typically involve fine-tuning LLMs with synthetic data, which depends on annotations from human experts or advanced models like GPT-4, thus restricting further advancements. To address this issue, we investigate the potential for LLMs to self-improve in long-context reasoning and propose \ours, an approach specifically designed for this purpose. This approach is straightforward: we sample multiple outputs for each question, score them with Minimum Bayes Risk, and then apply supervised fine-tuning or preference optimization based on these outputs. Extensive experiments on several leading LLMs demonstrate the effectiveness of \ours, with an absolute improvement of $4.2$ points for Llama-3.1-8B-Instruct. Furthermore, \ours achieves superior performance compared to prior approaches that depend on data produced by human experts or advanced models. We anticipate that this work will open new avenues for self-improvement techniques in long-context scenarios, which are essential for the continual advancement of LLMs.

Title: EAPCR: A Universal Feature Extractor for Scientific Data without Explicit Feature Relation Patterns

• Authors: Zhuohang Yu, Ling An, Yansong Li, Yu Wu, Zeyu Dong, Zhangdi Liu, Le Gao, Zhenyu Zhang, Chichun Zhou
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08164
• Pdf URL: https://arxiv.org/pdf/2411.08164
• Copy Paste: [[2411.08164]] EAPCR: A Universal Feature Extractor for Scientific Data without Explicit Feature Relation Patterns(https://arxiv.org/abs/2411.08164)
• Keywords: robust, transformer
• Abstract: Conventional methods, including Decision Tree (DT)-based methods, have been effective in scientific tasks, such as non-image medical diagnostics, system anomaly detection, and inorganic catalysis efficiency prediction. However, most deep-learning techniques have struggled to surpass or even match this level of success as traditional machine-learning methods. The primary reason is that these applications involve multi-source, heterogeneous data where features lack explicit relationships. This contrasts with image data, where pixels exhibit spatial relationships; textual data, where words have sequential dependencies; and graph data, where nodes are connected through established associations. The absence of explicit Feature Relation Patterns (FRPs) presents a significant challenge for deep learning techniques in scientific applications that are not image, text, and graph-based. In this paper, we introduce EAPCR, a universal feature extractor designed for data without explicit FRPs. Tested across various scientific tasks, EAPCR consistently outperforms traditional methods and bridges the gap where deep learning models fall short. To further demonstrate its robustness, we synthesize a dataset without explicit FRPs. While Kolmogorov-Arnold Network (KAN) and feature extractors like Convolutional Neural Networks (CNNs), Graph Convolutional Networks (GCNs), and Transformers struggle, EAPCR excels, demonstrating its robustness and superior performance in scientific tasks without FRPs.

Title: Multi-Agent Stochastic Bandits Robust to Adversarial Corruptions

• Authors: Fatemeh Ghaffari, Xuchuang Wang, Jinhang Zuo, Mohammad Hajiesmaili
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.08167
• Pdf URL: https://arxiv.org/pdf/2411.08167
• Copy Paste: [[2411.08167]] Multi-Agent Stochastic Bandits Robust to Adversarial Corruptions(https://arxiv.org/abs/2411.08167)
• Keywords: robust
• Abstract: We study the problem of multi-agent multi-armed bandits with adversarial corruption in a heterogeneous setting, where each agent accesses a subset of arms. The adversary can corrupt the reward observations for all agents. Agents share these corrupted rewards with each other, and the objective is to maximize the cumulative total reward of all agents (and not be misled by the adversary). We propose a multi-agent cooperative learning algorithm that is robust to adversarial corruptions. For this newly devised algorithm, we demonstrate that an adversary with an unknown corruption budget $C$ only incurs an additive $O((L / L_{\min}) C)$ term to the standard regret of the model in non-corruption settings, where $L$ is the total number of agents, and $L_{\min}$ is the minimum number of agents with mutual access to an arm. As a side-product, our algorithm also improves the state-of-the-art regret bounds when reducing to both the single-agent and homogeneous multi-agent scenarios, tightening multiplicative $K$ (the number of arms) and $L$ (the number of agents) factors, respectively.

Title: SCORE: Syntactic Code Representations for Static Script Malware Detection

• Authors: Ecenaz Erdemir, Kyuhong Park, Michael J. Morais, Vianne R. Gao, Marion Marschalek, Yi Fan
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08182
• Pdf URL: https://arxiv.org/pdf/2411.08182
• Copy Paste: [[2411.08182]] SCORE: Syntactic Code Representations for Static Script Malware Detection(https://arxiv.org/abs/2411.08182)
• Keywords: security, protect, attack, steal, extraction
• Abstract: As businesses increasingly adopt cloud technologies, they also need to be aware of new security challenges, such as server-side script attacks, to ensure the integrity of their systems and data. These scripts can steal data, compromise credentials, and disrupt operations. Unlike executables with standardized formats (e.g., ELF, PE), scripts are plaintext files with diverse syntax, making them harder to detect using traditional methods. As a result, more sophisticated approaches are needed to protect cloud infrastructures from these evolving threats. In this paper, we propose novel feature extraction and deep learning (DL)-based approaches for static script malware detection, targeting server-side threats. We extract features from plain-text code using two techniques: syntactic code highlighting (SCH) and abstract syntax tree (AST) construction. SCH leverages complex regexes to parse syntactic elements of code, such as keywords, variable names, etc. ASTs generate a hierarchical representation of a program's syntactic structure. We then propose a sequential and a graph-based model that exploits these feature representations to detect script malware. We evaluate our approach on more than 400K server-side scripts in Bash, Python and Perl. We use a balanced dataset of 90K scripts for training, validation, and testing, with the remaining from 400K reserved for further analysis. Experiments show that our method achieves a true positive rate (TPR) up to 81% higher than leading signature-based antivirus solutions, while maintaining a low false positive rate (FPR) of 0.17%. Moreover, our approach outperforms various neural network-based detectors, demonstrating its effectiveness in learning code maliciousness for accurate detection of script malware.

Title: TractoEmbed: Modular Multi-level Embedding framework for white matter tract segmentation

• Authors: Anoushkrit Goel, Bipanjit Singh, Ankita Joshi, Ranjeet Ranjan Jha, Chirag Ahuja, Aditya Nigam, Arnav Bhavsar
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08187
• Pdf URL: https://arxiv.org/pdf/2411.08187
• Copy Paste: [[2411.08187]] TractoEmbed: Modular Multi-level Embedding framework for white matter tract segmentation(https://arxiv.org/abs/2411.08187)
• Keywords: segmentation
• Abstract: White matter tract segmentation is crucial for studying brain structural connectivity and neurosurgical planning. However, segmentation remains challenging due to issues like class imbalance between major and minor tracts, structural similarity, subject variability, symmetric streamlines between hemispheres etc. To address these challenges, we propose TractoEmbed, a modular multi-level embedding framework, that encodes localized representations through learning tasks in respective encoders. In this paper, TractoEmbed introduces a novel hierarchical streamline data representation that captures maximum spatial information at each level i.e. individual streamlines, clusters, and patches. Experiments show that TractoEmbed outperforms state-of-the-art methods in white matter tract segmentation across different datasets, and spanning various age groups. The modular framework directly allows the integration of additional embeddings in future works.

Title: An Explainable Machine Learning Approach for Age and Gender Estimation in Living Individuals Using Dental Biometrics

• Authors: Mohsin Ali, Haider Raza, John Q Gan, Ariel Pokhojaev, Matanel Katz, Esra Kosan, Dian Agustin Wahjuningrum, Omnina Saleh, Rachel Sarig, Akhilanada Chaurasia
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08195
• Pdf URL: https://arxiv.org/pdf/2411.08195
• Copy Paste: [[2411.08195]] An Explainable Machine Learning Approach for Age and Gender Estimation in Living Individuals Using Dental Biometrics(https://arxiv.org/abs/2411.08195)
• Keywords: biometric
• Abstract: Objectives: Age and gender estimation is crucial for various applications, including forensic investigations and anthropological studies. This research aims to develop a predictive system for age and gender estimation in living individuals, leveraging dental measurements such as Coronal Height (CH), Coronal Pulp Cavity Height (CPCH), and Tooth Coronal Index (TCI). Methods: Machine learning models were employed in our study, including Cat Boost Classifier (Catboost), Gradient Boosting Machine (GBM), Ada Boost Classifier (AdaBoost), Random Forest (RF), eXtreme Gradient Boosting (XGB), Light Gradient Boosting Machine (LGB), and Extra Trees Classifier (ETC), to analyze dental data from 862 living individuals (459 males and 403 females). Specifically, periapical radiographs from six teeth per individual were utilized, including premolars and molars from both maxillary and mandibular. A novel ensemble learning technique was developed, which uses multiple models each tailored to distinct dental metrics, to estimate age and gender accurately. Furthermore, an explainable AI model has been created utilizing SHAP, enabling dental experts to make judicious decisions based on comprehensible insight. Results: The RF and XGB models were particularly effective, yielding the highest F1 score for age and gender estimation. Notably, the XGB model showed a slightly better performance in age estimation, achieving an F1 score of 73.26%. A similar trend for the RF model was also observed in gender estimation, achieving a F1 score of 77.53%. Conclusions: This study marks a significant advancement in dental forensic methods, showcasing the potential of machine learning to automate age and gender estimation processes with improved accuracy.

Title: Latent Space Disentanglement in Diffusion Transformers Enables Precise Zero-shot Semantic Editing

• Authors: Zitao Shuai, Chenwei Wu, Zhengxu Tang, Bowen Song, Liyue Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08196
• Pdf URL: https://arxiv.org/pdf/2411.08196
• Copy Paste: [[2411.08196]] Latent Space Disentanglement in Diffusion Transformers Enables Precise Zero-shot Semantic Editing(https://arxiv.org/abs/2411.08196)
• Keywords: diffusion, transformer
• Abstract: Diffusion Transformers (DiTs) have recently achieved remarkable success in text-guided image generation. In image editing, DiTs project text and image inputs to a joint latent space, from which they decode and synthesize new images. However, it remains largely unexplored how multimodal information collectively forms this joint space and how they guide the semantics of the synthesized images. In this paper, we investigate the latent space of DiT models and uncover two key properties: First, DiT's latent space is inherently semantically disentangled, where different semantic attributes can be controlled by specific editing directions. Second, consistent semantic editing requires utilizing the entire joint latent space, as neither encoded image nor text alone contains enough semantic information. We show that these editing directions can be obtained directly from text prompts, enabling precise semantic control without additional training or mask annotations. Based on these insights, we propose a simple yet effective Encode-Identify-Manipulate (EIM) framework for zero-shot fine-grained image editing. Specifically, we first encode both the given source image and the text prompt that describes the image, to obtain the joint latent embedding. Then, using our proposed Hessian Score Distillation Sampling (HSDS) method, we identify editing directions that control specific target attributes while preserving other image features. These directions are guided by text prompts and used to manipulate the latent embeddings. Moreover, we propose a new metric to quantify the disentanglement degree of the latent space of diffusion models. Extensive experiment results on our new curated benchmark dataset and analysis demonstrate DiT's disentanglement properties and effectiveness of the EIM framework.

Title: PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model

• Authors: Yilun Liu, Yunpu Ma, Shuo Chen, Zifeng Ding, Bailan He, Zhen Han, Volker Tresp
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08212
• Pdf URL: https://arxiv.org/pdf/2411.08212
• Copy Paste: [[2411.08212]] PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model(https://arxiv.org/abs/2411.08212)
• Keywords: transformer
• Abstract: The Mixture-of-Experts (MoE) paradigm has emerged as a powerful approach for scaling transformers with improved resource utilization. However, efficiently fine-tuning MoE models remains largely underexplored. Inspired by recent works on Parameter-Efficient Fine-Tuning (PEFT), we present a unified framework for integrating PEFT modules directly into the MoE mechanism. Aligning with the core principles and architecture of MoE, our framework encompasses a set of design dimensions including various functional and composition strategies. By combining design choices within our framework, we introduce Parameter-Efficient Routed Fine-Tuning (PERFT) as a flexible and scalable family of PEFT strategies tailored for MoE models. Extensive experiments on adapting OLMoE-1B-7B and Mixtral-8$\times$7B for commonsense and arithmetic reasoning tasks demonstrate the effectiveness, scalability, and intriguing dynamics of PERFT. Additionally, we provide empirical findings for each specific design choice to facilitate better application of MoE and PEFT.

Title: Joint Diffusion models in Continual Learning

• Authors: Paweł Skierś, Kamil Deja
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08224
• Pdf URL: https://arxiv.org/pdf/2411.08224
• Copy Paste: [[2411.08224]] Joint Diffusion models in Continual Learning(https://arxiv.org/abs/2411.08224)
• Keywords: diffusion, generative
• Abstract: In this work, we introduce JDCL - a new method for continual learning with generative rehearsal based on joint diffusion models. Neural networks suffer from catastrophic forgetting defined as abrupt loss in the model's performance when retrained with additional data coming from a different distribution. Generative-replay-based continual learning methods try to mitigate this issue by retraining a model with a combination of new and rehearsal data sampled from a generative model. In this work, we propose to extend this idea by combining a continually trained classifier with a diffusion-based generative model into a single - jointly optimized neural network. We show that such shared parametrization, combined with the knowledge distillation technique allows for stable adaptation to new tasks without catastrophic forgetting. We evaluate our approach on several benchmarks, where it outperforms recent state-of-the-art generative replay techniques. Additionally, we extend our method to the semi-supervised continual learning setup, where it outperforms competing buffer-based replay techniques, and evaluate, in a self-supervised manner, the quality of trained representations.

Title: DPU: Dynamic Prototype Updating for Multimodal Out-of-Distribution Detection

• Authors: Shawn Li, Huixian Gong, Hao Dong, Tiankai Yang, Zhengzhong Tu, Yue Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08227
• Pdf URL: https://arxiv.org/pdf/2411.08227
• Copy Paste: [[2411.08227]] DPU: Dynamic Prototype Updating for Multimodal Out-of-Distribution Detection(https://arxiv.org/abs/2411.08227)
• Keywords: robust
• Abstract: Out-of-distribution (OOD) detection is essential for ensuring the robustness of machine learning models by identifying samples that deviate from the training distribution. While traditional OOD detection has primarily focused on single-modality inputs, such as images, recent advances in multimodal models have demonstrated the potential of leveraging multiple modalities (e.g., video, optical flow, audio) to enhance detection performance. However, existing methods often overlook intra-class variability within in-distribution (ID) data, assuming that samples of the same class are perfectly cohesive and consistent. This assumption can lead to performance degradation, especially when prediction discrepancies are uniformly amplified across all samples. To address this issue, we propose Dynamic Prototype Updating (DPU), a novel plug-and-play framework for multimodal OOD detection that accounts for intra-class variations. Our method dynamically updates class center representations for each class by measuring the variance of similar samples within each batch, enabling adaptive adjustments. This approach allows us to amplify prediction discrepancies based on the updated class centers, thereby improving the model's robustness and generalization across different modalities. Extensive experiments on two tasks, five datasets, and nine base OOD algorithms demonstrate that DPU significantly improves OOD detection performance, setting a new state-of-the-art in multimodal OOD detection, with improvements of up to 80 percent in Far-OOD detection. To facilitate accessibility and reproducibility, our code is publicly available on GitHub.

Title: Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset

• Authors: Khaoula Chehbouni, Jonathan Colaço-Carr, Yash More, Jackie CK Cheung, Golnoosh Farnadi
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.08243
• Pdf URL: https://arxiv.org/pdf/2411.08243
• Copy Paste: [[2411.08243]] Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset(https://arxiv.org/abs/2411.08243)
• Keywords: large language model
• Abstract: In an effort to mitigate the harms of large language models (LLMs), learning from human feedback (LHF) has been used to steer LLMs towards outputs that are intended to be both less harmful and more helpful. Despite the widespread adoption of LHF in practice, the quality of this feedback and its effectiveness as a safety mitigation technique remain unclear. This study addresses these issues by auditing the widely-used Helpful and Harmless (HH) dataset by Anthropic. Our work includes: (1) a thorough investigation of the dataset's content through both manual and automated evaluation; (2) experiments demonstrating the dataset's impact on models' safety; and (3) an analysis of the 100 most influential papers citing this dataset. Through our audit, we showcase how conceptualization failures and quality issues identified in the HH dataset can create additional harms by leading to disparate safety behaviors across demographic groups. Our findings highlight the need for more nuanced, context-sensitive approaches to safety mitigation in LLMs.

Title: NVCiM-PT: An NVCiM-assisted Prompt Tuning Framework for Edge LLMs

• Authors: Ruiyang Qin, Pengyu Ren, Zheyu Yan, Liu Liu, Dancheng Liu, Amir Nassereldine, Jinjun Xiong, Kai Ni, Sharon Hu, Yiyu Shi
• Subjects: cs.LG, cs.ET
• Abstract URL: https://arxiv.org/abs/2411.08244
• Pdf URL: https://arxiv.org/pdf/2411.08244
• Copy Paste: [[2411.08244]] NVCiM-PT: An NVCiM-assisted Prompt Tuning Framework for Edge LLMs(https://arxiv.org/abs/2411.08244)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) deployed on edge devices, known as edge LLMs, need to continuously fine-tune their model parameters from user-generated data under limited resource constraints. However, most existing learning methods are not applicable for edge LLMs because of their reliance on high resources and low learning capacity. Prompt tuning (PT) has recently emerged as an effective fine-tuning method for edge LLMs by only modifying a small portion of LLM parameters, but it suffers from user domain shifts, resulting in repetitive training and losing resource efficiency. Conventional techniques to address domain shift issues often involve complex neural networks and sophisticated training, which are incompatible for PT for edge LLMs. Therefore, an open research question is how to address domain shift issues for edge LLMs with limited resources. In this paper, we propose a prompt tuning framework for edge LLMs, exploiting the benefits offered by non-volatile computing-in-memory (NVCiM) architectures. We introduce a novel NVCiM-assisted PT framework, where we narrow down the core operations to matrix-matrix multiplication, which can then be accelerated by performing in-situ computation on NVCiM. To the best of our knowledge, this is the first work employing NVCiM to improve the edge LLM PT performance.

Title: Deceiving Question-Answering Models: A Hybrid Word-Level Adversarial Approach

• Authors: Jiyao Li, Mingze Ni, Yongshun Gong, Wei Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08248
• Pdf URL: https://arxiv.org/pdf/2411.08248
• Copy Paste: [[2411.08248]] Deceiving Question-Answering Models: A Hybrid Word-Level Adversarial Approach(https://arxiv.org/abs/2411.08248)
• Keywords: attack, robust
• Abstract: Deep learning underpins most of the currently advanced natural language processing (NLP) tasks such as textual classification, neural machine translation (NMT), abstractive summarization and question-answering (QA). However, the robustness of the models, particularly QA models, against adversarial attacks is a critical concern that remains insufficiently explored. This paper introduces QA-Attack (Question Answering Attack), a novel word-level adversarial strategy that fools QA models. Our attention-based attack exploits the customized attention mechanism and deletion ranking strategy to identify and target specific words within contextual passages. It creates deceptive inputs by carefully choosing and substituting synonyms, preserving grammatical integrity while misleading the model to produce incorrect responses. Our approach demonstrates versatility across various question types, particularly when dealing with extensive long textual inputs. Extensive experiments on multiple benchmark datasets demonstrate that QA-Attack successfully deceives baseline QA models and surpasses existing adversarial techniques regarding success rate, semantics changes, BLEU score, fluency and grammar error rate.

Title: GPTree: Towards Explainable Decision-Making via LLM-powered Decision Trees

• Authors: Sichao Xiong, Yigit Ihlamur, Fuat Alican, Aaron Ontoyin Yin
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2411.08257
• Pdf URL: https://arxiv.org/pdf/2411.08257
• Copy Paste: [[2411.08257]] GPTree: Towards Explainable Decision-Making via LLM-powered Decision Trees(https://arxiv.org/abs/2411.08257)
• Keywords: explainability
• Abstract: Traditional decision tree algorithms are explainable but struggle with non-linear, high-dimensional data, limiting its applicability in complex decision-making. Neural networks excel at capturing complex patterns but sacrifice explainability in the process. In this work, we present GPTree, a novel framework combining explainability of decision trees with the advanced reasoning capabilities of LLMs. GPTree eliminates the need for feature engineering and prompt chaining, requiring only a task-specific prompt and leveraging a tree-based structure to dynamically split samples. We also introduce an expert-in-the-loop feedback mechanism to further enhance performance by enabling human intervention to refine and rebuild decision paths, emphasizing the harmony between human expertise and machine intelligence. Our decision tree achieved a 7.8% precision rate for identifying "unicorn" startups at the inception stage of a startup, surpassing gpt-4o with few-shot learning as well as the best human decision-makers (3.1% to 5.6%).

Title: LBONet: Supervised Spectral Descriptors for Shape Analysis

• Authors: Oguzhan Yigit, Richard C. Wilson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08272
• Pdf URL: https://arxiv.org/pdf/2411.08272
• Copy Paste: [[2411.08272]] LBONet: Supervised Spectral Descriptors for Shape Analysis(https://arxiv.org/abs/2411.08272)
• Keywords: segmentation
• Abstract: The Laplace-Beltrami operator has established itself in the field of non-rigid shape analysis due to its many useful properties such as being invariant under isometric transformation, having a countable eigensystem forming an orthonormal basis, and fully characterizing geodesic distances of the manifold. However, this invariancy only applies under isometric deformations, which leads to a performance breakdown in many real-world applications. In recent years emphasis has been placed upon extracting optimal features using deep learning methods, however spectral signatures play a crucial role and still add value. In this paper we take a step back, revisiting the LBO and proposing a supervised way to learn several operators on a manifold. Depending on the task, by applying these functions, we can train the LBO eigenbasis to be more task-specific. The optimization of the LBO leads to enormous improvements to established descriptors such as the heat kernel signature in various tasks such as retrieval, classification, segmentation, and correspondence, proving the adaption of the LBO eigenbasis to both global and highly local learning settings.

Title: Knowledge Bases in Support of Large Language Models for Processing Web News

• Authors: Yihe Zhang, Nabin Pakka, Nian-feng Tzeng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08278
• Pdf URL: https://arxiv.org/pdf/2411.08278
• Copy Paste: [[2411.08278]] Knowledge Bases in Support of Large Language Models for Processing Web News(https://arxiv.org/abs/2411.08278)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have received considerable interest in wide applications lately. During pre-training via massive datasets, such a model implicitly memorizes the factual knowledge of trained datasets in its hidden parameters. However, knowledge held implicitly in parameters often makes its use by downstream applications ineffective due to the lack of common-sense reasoning. In this article, we introduce a general framework that permits to build knowledge bases with an aid of LLMs, tailored for processing Web news. The framework applies a rule-based News Information Extractor (NewsIE) to news items for extracting their relational tuples, referred to as knowledge bases, which are then graph-convoluted with the implicit knowledge facts of news items obtained by LLMs, for their classification. It involves two lightweight components: 1) NewsIE: for extracting the structural information of every news item, in the form of relational tuples; 2) BERTGraph: for graph convoluting the implicit knowledge facts with relational tuples extracted by NewsIE. We have evaluated our framework under different news-related datasets for news category classification, with promising experimental results.

Title: MBA-SLAM: Motion Blur Aware Dense Visual SLAM with Radiance Fields Representation

• Authors: Peng Wang, Lingzhe Zhao, Yin Zhang, Shiyu Zhao, Peidong Liu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.08279
• Pdf URL: https://arxiv.org/pdf/2411.08279
• Copy Paste: [[2411.08279]] MBA-SLAM: Motion Blur Aware Dense Visual SLAM with Radiance Fields Representation(https://arxiv.org/abs/2411.08279)
• Keywords: robust
• Abstract: Emerging 3D scene representations, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have demonstrated their effectiveness in Simultaneous Localization and Mapping (SLAM) for photo-realistic rendering, particularly when using high-quality video sequences as input. However, existing methods struggle with motion-blurred frames, which are common in real-world scenarios like low-light or long-exposure conditions. This often results in a significant reduction in both camera localization accuracy and map reconstruction quality. To address this challenge, we propose a dense visual SLAM pipeline (i.e. MBA-SLAM) to handle severe motion-blurred inputs. Our approach integrates an efficient motion blur-aware tracker with either neural radiance fields or Gaussian Splatting based mapper. By accurately modeling the physical image formation process of motion-blurred images, our method simultaneously learns 3D scene representation and estimates the cameras' local trajectory during exposure time, enabling proactive compensation for motion blur caused by camera movement. In our experiments, we demonstrate that MBA-SLAM surpasses previous state-of-the-art methods in both camera localization and map reconstruction, showcasing superior performance across a range of datasets, including synthetic and real datasets featuring sharp images as well as those affected by motion blur, highlighting the versatility and robustness of our approach. Code is available at this https URL.

Title: TowerDebias: A Novel Debiasing Method based on the Tower Property

• Authors: Norman Matloff, Aditya Mittal
• Subjects: cs.LG, cs.AI, math.PR, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.08297
• Pdf URL: https://arxiv.org/pdf/2411.08297
• Copy Paste: [[2411.08297]] TowerDebias: A Novel Debiasing Method based on the Tower Property(https://arxiv.org/abs/2411.08297)
• Keywords: fair
• Abstract: Decision-making processes have increasingly come to rely on sophisticated machine learning tools, raising concerns about the fairness of their predictions with respect to any sensitive groups. The widespread use of commercial black-box machine learning models necessitates careful consideration of their legal and ethical implications on consumers. In situations where users have access to these "black-box" models, a key question emerges: how can we mitigate or eliminate the influence of sensitive attributes, such as race or gender? We propose towerDebias (tDB), a novel approach designed to reduce the influence of sensitive variables in predictions made by black-box models. Using the Tower Property from probability theory, tDB aims to improve prediction fairness during the post-processing stage in a manner amenable to the Fairness-Utility Tradeoff. This method is highly flexible, requiring no prior knowledge of the original model's internal structure, and can be extended to a range of different applications. We provide a formal improvement theorem for tDB and demonstrate its effectiveness in both regression and classification tasks, underscoring its impact on the fairness-utility tradeoff.

Title: R3HF: Reward Redistribution for Enhancing Reinforcement Learning from Human Feedback

• Authors: Jiahui Li, Tai-wei Chang, Fengda Zhang, Kun Kuang, Long Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08302
• Pdf URL: https://arxiv.org/pdf/2411.08302
• Copy Paste: [[2411.08302]] R3HF: Reward Redistribution for Enhancing Reinforcement Learning from Human Feedback(https://arxiv.org/abs/2411.08302)
• Keywords: large language model
• Abstract: Reinforcement learning from human feedback (RLHF) provides a paradigm for aligning large language models (LLMs) with human preferences. This involves the initial training of a reward model based on pairwise human feedback. The reward model is subsequently utilized in reinforcement learning to assess the scores of each generated sentence as a whole, further guiding the optimization of LLMs. However, current approaches have a significant shortcoming: \emph{They allocate a single, sparse, and delayed reward to an entire sequence of output}. This may overlook some significant individual contributions of each token towards the desired outcome. To overcome this limitation, our paper proposes a novel reward redistribution method called R3HF, which facilitates a more fine-grained, token-level reward allocation. Specifically, our method treats the reward prediction task of the reward model as a regression problem. As a result, the redistributed rewards are computed by evaluating the specific contribution of each token to the reward model's output. This detailed approach improves the model's understanding of language nuances, leading to more precise enhancements in its performance. Our method is crafted to integrate seamlessly with most current techniques while incurring minimal computational costs. Through comprehensive experiments across diverse datasets and tasks, we have verified the effectiveness and superiority of our approach.

Title: SDDBench: A Benchmark for Synthesizable Drug Design

• Authors: Songtao Liu, Zhengkai Tu, Hanjun Dai, Peng Liu
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.08306
• Pdf URL: https://arxiv.org/pdf/2411.08306
• Copy Paste: [[2411.08306]] SDDBench: A Benchmark for Synthesizable Drug Design(https://arxiv.org/abs/2411.08306)
• Keywords: generative
• Abstract: A significant challenge in wet lab experiments with current drug design generative models is the trade-off between pharmacological properties and synthesizability. Molecules predicted to have highly desirable properties are often difficult to synthesize, while those that are easily synthesizable tend to exhibit less favorable properties. As a result, evaluating the synthesizability of molecules in general drug design scenarios remains a significant challenge in the field of drug discovery. The commonly used synthetic accessibility (SA) score aims to evaluate the ease of synthesizing generated molecules, but it falls short of guaranteeing that synthetic routes can actually be found. Inspired by recent advances in top-down synthetic route generation, we propose a new, data-driven metric to evaluate molecule synthesizability. Our approach directly assesses the feasibility of synthetic routes for a given molecule through our proposed round-trip score. This novel metric leverages the synergistic duality between retrosynthetic planners and reaction predictors, both of which are trained on extensive reaction datasets. To demonstrate the efficacy of our method, we conduct a comprehensive evaluation of round-trip scores alongside search success rate across a range of representative molecule generative models. Code is available at this https URL.

Title: Evaluating Synthetic Command Attacks on Smart Voice Assistants

• Authors: Zhengxian He, Ashish Kundu, Mustaque Ahamad
• Subjects: cs.CR, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.08316
• Pdf URL: https://arxiv.org/pdf/2411.08316
• Copy Paste: [[2411.08316]] Evaluating Synthetic Command Attacks on Smart Voice Assistants(https://arxiv.org/abs/2411.08316)
• Keywords: defense, attack
• Abstract: Recent advances in voice synthesis, coupled with the ease with which speech can be harvested for millions of people, introduce new threats to applications that are enabled by devices such as voice assistants (e.g., Amazon Alexa, Google Home etc.). We explore if unrelated and limited amount of speech from a target can be used to synthesize commands for a voice assistant like Amazon Alexa. More specifically, we investigate attacks on voice assistants with synthetic commands when they match command sources to authorized users, and applications (e.g., Alexa Skills) process commands only when their source is an authorized user with a chosen confidence level. We demonstrate that even simple concatenative speech synthesis can be used by an attacker to command voice assistants to perform sensitive operations. We also show that such attacks, when launched by exploiting compromised devices in the vicinity of voice assistants, can have relatively small host and network footprint. Our results demonstrate the need for better defenses against synthetic malicious commands that could target voice assistants.

Title: Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle

• Authors: Hui Dai, Ryan Teehan, Mengye Ren
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08324
• Pdf URL: https://arxiv.org/pdf/2411.08324
• Copy Paste: [[2411.08324]] Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle(https://arxiv.org/abs/2411.08324)
• Keywords: large language model
• Abstract: Many existing evaluation benchmarks for Large Language Models (LLMs) quickly become outdated due to the emergence of new models and training data. These benchmarks also fall short in assessing how LLM performance changes over time, as they consist of static questions without a temporal dimension. To address these limitations, we propose using future event prediction as a continuous evaluation method to assess LLMs' temporal generalization and forecasting abilities. Our benchmark, Daily Oracle, automatically generates question-answer (QA) pairs from daily news, challenging LLMs to predict "future" event outcomes. Our findings reveal that as pre-training data becomes outdated, LLM performance degrades over time. While Retrieval Augmented Generation (RAG) has the potential to enhance prediction accuracy, the performance degradation pattern persists, highlighting the need for continuous model updates.

Title: Motion Control for Enhanced Complex Action Video Generation

• Authors: Qiang Zhou, Shaofeng Zhang, Nianzu Yang, Ye Qian, Hao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08328
• Pdf URL: https://arxiv.org/pdf/2411.08328
• Copy Paste: [[2411.08328]] Motion Control for Enhanced Complex Action Video Generation(https://arxiv.org/abs/2411.08328)
• Keywords: robust, diffusion
• Abstract: Existing text-to-video (T2V) models often struggle with generating videos with sufficiently pronounced or complex actions. A key limitation lies in the text prompt's inability to precisely convey intricate motion details. To address this, we propose a novel framework, MVideo, designed to produce long-duration videos with precise, fluid actions. MVideo overcomes the limitations of text prompts by incorporating mask sequences as an additional motion condition input, providing a clearer, more accurate representation of intended actions. Leveraging foundational vision models such as GroundingDINO and SAM2, MVideo automatically generates mask sequences, enhancing both efficiency and robustness. Our results demonstrate that, after training, MVideo effectively aligns text prompts with motion conditions to produce videos that simultaneously meet both criteria. This dual control mechanism allows for more dynamic video generation by enabling alterations to either the text prompt or motion condition independently, or both in tandem. Furthermore, MVideo supports motion condition editing and composition, facilitating the generation of videos with more complex actions. MVideo thus advances T2V motion generation, setting a strong benchmark for improved action depiction in current video diffusion models. Our project page is available at this https URL.

Title: DyConfidMatch: Dynamic Thresholding and Re-sampling for 3D Semi-supervised Learning

• Authors: Zhimin Chen, Bing Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08340
• Pdf URL: https://arxiv.org/pdf/2411.08340
• Copy Paste: [[2411.08340]] DyConfidMatch: Dynamic Thresholding and Re-sampling for 3D Semi-supervised Learning(https://arxiv.org/abs/2411.08340)
• Keywords: robust
• Abstract: Semi-supervised learning (SSL) leverages limited labeled and abundant unlabeled data but often faces challenges with data imbalance, especially in 3D contexts. This study investigates class-level confidence as an indicator of learning status in 3D SSL, proposing a novel method that utilizes dynamic thresholding to better use unlabeled data, particularly from underrepresented classes. A re-sampling strategy is also introduced to mitigate bias towards well-represented classes, ensuring equitable class representation. Through extensive experiments in 3D SSL, our method surpasses state-of-the-art counterparts in classification and detection tasks, highlighting its effectiveness in tackling data imbalance. This approach presents a significant advancement in SSL for 3D datasets, providing a robust solution for data imbalance issues.

Title: Bangla Grammatical Error Detection Leveraging Transformer-based Token Classification

• Authors: Shayekh Bin Islam, Ridwanul Hasan Tanvir, Sihat Afnan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08344
• Pdf URL: https://arxiv.org/pdf/2411.08344
• Copy Paste: [[2411.08344]] Bangla Grammatical Error Detection Leveraging Transformer-based Token Classification(https://arxiv.org/abs/2411.08344)
• Keywords: transformer
• Abstract: Bangla is the seventh most spoken language by a total number of speakers in the world, and yet the development of an automated grammar checker in this language is an understudied problem. Bangla grammatical error detection is a task of detecting sub-strings of a Bangla text that contain grammatical, punctuation, or spelling errors, which is crucial for developing an automated Bangla typing assistant. Our approach involves breaking down the task as a token classification problem and utilizing state-of-the-art transformer-based models. Finally, we combine the output of these models and apply rule-based post-processing to generate a more reliable and comprehensive result. Our system is evaluated on a dataset consisting of over 25,000 texts from various sources. Our best model achieves a Levenshtein distance score of 1.04. Finally, we provide a detailed analysis of different components of our system.

Title: Refining Translations with LLMs: A Constraint-Aware Iterative Prompting Approach

• Authors: Shangfeng Chen, Xiayang Shi, Pu Li, Yinlin Li, Jingjing Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.08348
• Pdf URL: https://arxiv.org/pdf/2411.08348
• Copy Paste: [[2411.08348]] Refining Translations with LLMs: A Constraint-Aware Iterative Prompting Approach(https://arxiv.org/abs/2411.08348)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable proficiency in machine translation (MT), even without specific training on the languages in question. However, translating rare words in low-resource or domain-specific contexts remains challenging for LLMs. To address this issue, we propose a multi-step prompt chain that enhances translation faithfulness by prioritizing key terms crucial for semantic accuracy. Our method first identifies these keywords and retrieves their translations from a bilingual dictionary, integrating them into the LLM's context using Retrieval-Augmented Generation (RAG). We further mitigate potential output hallucinations caused by long prompts through an iterative self-checking mechanism, where the LLM refines its translations based on lexical and semantic constraints. Experiments using Llama and Qwen as base models on the FLORES-200 and WMT datasets demonstrate significant improvements over baselines, highlighting the effectiveness of our approach in enhancing translation faithfulness and robustness, particularly in low-resource scenarios.

Title: MultiKG: Multi-Source Threat Intelligence Aggregation for High-Quality Knowledge Graph Representation of Attack Techniques

• Authors: Jian Wang, Tiantian Zhu, Chunlin Xiong, Yan Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.08359
• Pdf URL: https://arxiv.org/pdf/2411.08359
• Copy Paste: [[2411.08359]] MultiKG: Multi-Source Threat Intelligence Aggregation for High-Quality Knowledge Graph Representation of Attack Techniques(https://arxiv.org/abs/2411.08359)
• Keywords: security, attack, large language model
• Abstract: The construction of attack technique knowledge graphs aims to transform various types of attack knowledge into structured representations for more effective attack procedure modeling. Existing methods typically rely on textual data, such as Cyber Threat Intelligence (CTI) reports, which are often coarse-grained and unstructured, resulting in incomplete and inaccurate knowledge graphs. To address these issues, we expand attack knowledge sources by incorporating audit logs and static code analysis alongside CTI reports, providing finer-grained data for constructing attack technique knowledge graphs. We propose MultiKG, a fully automated framework that integrates multiple threat knowledge sources. MultiKG processes data from CTI reports, dynamic logs, and static code separately, then merges them into a unified attack knowledge graph. Through system design and the utilization of the Large Language Model (LLM), MultiKG automates the analysis, construction, and merging of attack graphs across these sources, producing a fine-grained, multi-source attack knowledge graph. We implemented MultiKG and evaluated it using 1,015 real attack techniques and 9,006 attack intelligence entries from CTI reports. Results show that MultiKG effectively extracts attack knowledge graphs from diverse sources and aggregates them into accurate, comprehensive representations. Through case studies, we demonstrate that our approach directly benefits security tasks such as attack reconstruction and detection.

Title: Coverage Analysis for Digital Cousin Selection -- Improving Multi-Environment Q-Learning

• Authors: Talha Bozkus, Tara Javidi, Urbashi Mitra
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.08360
• Pdf URL: https://arxiv.org/pdf/2411.08360
• Copy Paste: [[2411.08360]] Coverage Analysis for Digital Cousin Selection -- Improving Multi-Environment Q-Learning(https://arxiv.org/abs/2411.08360)
• Keywords: robust
• Abstract: Q-learning is widely employed for optimizing various large-dimensional networks with unknown system dynamics. Recent advancements include multi-environment mixed Q-learning (MEMQ) algorithms, which utilize multiple independent Q-learning algorithms across multiple, structurally related but distinct environments and outperform several state-of-the-art Q-learning algorithms in terms of accuracy, complexity, and robustness. We herein conduct a comprehensive probabilistic coverage analysis to ensure optimal data coverage conditions for MEMQ algorithms. First, we derive upper and lower bounds on the expectation and variance of different coverage coefficients (CC) for MEMQ algorithms. Leveraging these bounds, we develop a simple way of comparing the utilities of multiple environments in MEMQ algorithms. This approach appears to be near optimal versus our previously proposed partial ordering approach. We also present a novel CC-based MEMQ algorithm to improve the accuracy and complexity of existing MEMQ algorithms. Numerical experiments are conducted using random network graphs with four different graph properties. Our algorithm can reduce the average policy error (APE) by 65% compared to partial ordering and is 95% faster than the exhaustive search. It also achieves 60% less APE than several state-of-the-art reinforcement learning and prior MEMQ algorithms. Additionally, we numerically verify the theoretical results and show their scalability with the action-space size.

Title: Multiscale Graph Construction Using Non-local Cluster Features

• Authors: Reina Kaneko, Hayate Kojima, Kenta Yanagiya, Junya Hara, Hiroshi Higashi, Yuichi Tanaka
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.08371
• Pdf URL: https://arxiv.org/pdf/2411.08371
• Copy Paste: [[2411.08371]] Multiscale Graph Construction Using Non-local Cluster Features(https://arxiv.org/abs/2411.08371)
• Keywords: segmentation
• Abstract: This paper presents a multiscale graph construction method using both graph and signal features. Multiscale graph is a hierarchical representation of the graph, where a node at each level indicates a cluster in a finer resolution. To obtain the hierarchical clusters, existing methods often use graph clustering; however, they may ignore signal variations. As a result, these methods could fail to detect the clusters having similar features on nodes. In this paper, we consider graph and node-wise features simultaneously for multiscale clustering of a graph. With given clusters of the graph, the clusters are merged hierarchically in three steps: 1) Feature vectors in the clusters are extracted. 2) Similarities among cluster features are calculated using optimal transport. 3) A variable $k$-nearest neighbor graph (V$k$NNG) is constructed and graph spectral clustering is applied to the V$k$NNG to obtain clusters at a coarser scale. Additionally, the multiscale graph in this paper has \textit{non-local} characteristics: Nodes with similar features are merged even if they are spatially separated. In experiments on multiscale image and point cloud segmentation, we demonstrate the effectiveness of the proposed method.

Title: Federated Graph Learning with Graphless Clients

• Authors: Xingbo Fu, Song Wang, Yushun Dong, Binchi Zhang, Chen Chen, Jundong Li
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.08374
• Pdf URL: https://arxiv.org/pdf/2411.08374
• Copy Paste: [[2411.08374]] Federated Graph Learning with Graphless Clients(https://arxiv.org/abs/2411.08374)
• Keywords: federate
• Abstract: Federated Graph Learning (FGL) is tasked with training machine learning models, such as Graph Neural Networks (GNNs), for multiple clients, each with its own graph data. Existing methods usually assume that each client has both node features and graph structure of its graph data. In real-world scenarios, however, there exist federated systems where only a part of the clients have such data while other clients (i.e. graphless clients) may only have node features. This naturally leads to a novel problem in FGL: how to jointly train a model over distributed graph data with graphless clients? In this paper, we propose a novel framework FedGLS to tackle the problem in FGL with graphless clients. In FedGLS, we devise a local graph learner on each graphless client which learns the local graph structure with the structure knowledge transferred from other clients. To enable structure knowledge transfer, we design a GNN model and a feature encoder on each client. During local training, the feature encoder retains the local graph structure knowledge together with the GNN model via knowledge distillation, and the structure knowledge is transferred among clients in global update. Our extensive experiments demonstrate the superiority of the proposed FedGLS over five baselines.

Title: Physics Informed Distillation for Diffusion Models

• Authors: Joshua Tian Jin Tee, Kang Zhang, Hee Suk Yoon, Dhananjaya Nagaraja Gowda, Chanwoo Kim, Chang D. Yoo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08378
• Pdf URL: https://arxiv.org/pdf/2411.08378
• Copy Paste: [[2411.08378]] Physics Informed Distillation for Diffusion Models(https://arxiv.org/abs/2411.08378)
• Keywords: diffusion, generative
• Abstract: Diffusion models have recently emerged as a potent tool in generative modeling. However, their inherent iterative nature often results in sluggish image generation due to the requirement for multiple model evaluations. Recent progress has unveiled the intrinsic link between diffusion models and Probability Flow Ordinary Differential Equations (ODEs), thus enabling us to conceptualize diffusion models as ODE systems. Simultaneously, Physics Informed Neural Networks (PINNs) have substantiated their effectiveness in solving intricate differential equations through implicit modeling of their solutions. Building upon these foundational insights, we introduce Physics Informed Distillation (PID), which employs a student model to represent the solution of the ODE system corresponding to the teacher diffusion model, akin to the principles employed in PINNs. Through experiments on CIFAR 10 and ImageNet 64x64, we observe that PID achieves performance comparable to recent distillation methods. Notably, it demonstrates predictable trends concerning method-specific hyperparameters and eliminates the need for synthetic dataset generation during the distillation process. Both of which contribute to its easy-to-use nature as a distillation approach for Diffusion Models. Our code and pre-trained checkpoint are publicly available at: this https URL.

Title: MambaXCTrack: Mamba-based Tracker with SSM Cross-correlation and Motion Prompt for Ultrasound Needle Tracking

• Authors: Yuelin Zhang, Qingpeng Ding, Long Lei, Jiwei Shan, Wenxuan Xie, Tianyi Zhang, Wanquan Yan, Raymond Shing-Yan Tang, Shing Shin Cheng
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.08395
• Pdf URL: https://arxiv.org/pdf/2411.08395
• Copy Paste: [[2411.08395]] MambaXCTrack: Mamba-based Tracker with SSM Cross-correlation and Motion Prompt for Ultrasound Needle Tracking(https://arxiv.org/abs/2411.08395)
• Keywords: robust
• Abstract: Ultrasound (US)-guided needle insertion is widely employed in percutaneous interventions. However, providing feedback on the needle tip position via US image presents challenges due to noise, artifacts, and the thin imaging plane of US, which degrades needle features and leads to intermittent tip visibility. In this paper, a Mamba-based US needle tracker MambaXCTrack utilizing structured state space models cross-correlation (SSMX-Corr) and implicit motion prompt is proposed, which is the first application of Mamba in US needle tracking. The SSMX-Corr enhances cross-correlation by long-range modeling and global searching of distant semantic features between template and search maps, benefiting the tracking under noise and artifacts by implicitly learning potential distant semantic cues. By combining with cross-map interleaved scan (CIS), local pixel-wise interaction with positional inductive bias can also be introduced to SSMX-Corr. The implicit low-level motion descriptor is proposed as a non-visual prompt to enhance tracking robustness, addressing the intermittent tip visibility problem. Extensive experiments on a dataset with motorized needle insertion in both phantom and tissue samples demonstrate that the proposed tracker outperforms other state-of-the-art trackers while ablation studies further highlight the effectiveness of each proposed tracking module.

Title: V2X-R: Cooperative LiDAR-4D Radar Fusion for 3D Object Detection with Denoising Diffusion

• Authors: Xun Huang, Jinlong Wang, Qiming Xia, Siheng Chen, Bisheng Yang, Cheng Wang, Chenglu Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08402
• Pdf URL: https://arxiv.org/pdf/2411.08402
• Copy Paste: [[2411.08402]] V2X-R: Cooperative LiDAR-4D Radar Fusion for 3D Object Detection with Denoising Diffusion(https://arxiv.org/abs/2411.08402)
• Keywords: robust, diffusion
• Abstract: Current Vehicle-to-Everything (V2X) systems have significantly enhanced 3D object detection using LiDAR and camera data. However, these methods suffer from performance degradation in adverse weather conditions. The weatherrobust 4D radar provides Doppler and additional geometric information, raising the possibility of addressing this challenge. To this end, we present V2X-R, the first simulated V2X dataset incorporating LiDAR, camera, and 4D radar. V2X-R contains 12,079 scenarios with 37,727 frames of LiDAR and 4D radar point clouds, 150,908 images, and 170,859 annotated 3D vehicle bounding boxes. Subsequently, we propose a novel cooperative LiDAR-4D radar fusion pipeline for 3D object detection and implement it with various fusion strategies. To achieve weather-robust detection, we additionally propose a Multi-modal Denoising Diffusion (MDD) module in our fusion pipeline. MDD utilizes weather-robust 4D radar feature as a condition to prompt the diffusion model to denoise noisy LiDAR features. Experiments show that our LiDAR-4D radar fusion pipeline demonstrates superior performance in the V2X-R dataset. Over and above this, our MDD module further improved the performance of basic fusion model by up to 5.73%/6.70% in foggy/snowy conditions with barely disrupting normal performance. The dataset and code will be publicly available at: this https URL.

Title: The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense

• Authors: Yangyang Guo, Fangkai Jiao, Liqiang Nie, Mohan Kankanhalli
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08410
• Pdf URL: https://arxiv.org/pdf/2411.08410
• Copy Paste: [[2411.08410]] The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense(https://arxiv.org/abs/2411.08410)
• Keywords: defense, attack, large language model
• Abstract: The vulnerability of Vision Large Language Models (VLLMs) to jailbreak attacks appears as no surprise. However, recent defense mechanisms against these attacks have reached near-saturation performance on benchmarks, often with minimal effort. This simultaneous high performance in both attack and defense presents a perplexing paradox. Resolving it is critical for advancing the development of trustworthy models. To address this research gap, we first investigate why VLLMs are prone to these attacks. We then make a key observation: existing defense mechanisms suffer from an \textbf{over-prudence} problem, resulting in unexpected abstention even in the presence of benign inputs. Additionally, we find that the two representative evaluation methods for jailbreak often exhibit chance agreement. This limitation makes it potentially misleading when evaluating attack strategies or defense mechanisms. Beyond these empirical observations, our another contribution in this work is to repurpose the guardrails of LLMs on the shelf, as an effective alternative detector prior to VLLM response. We believe these findings offer useful insights to rethink the foundational development of VLLM safety with respect to benchmark datasets, evaluation methods, and defense strategies.

Title: A Heterogeneous Graph Neural Network Fusing Functional and Structural Connectivity for MCI Diagnosis

• Authors: Feiyu Yin, Yu Lei, Siyuan Dai, Wenwen Zeng, Guoqing Wu, Liang Zhan, Jinhua Yu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08424
• Pdf URL: https://arxiv.org/pdf/2411.08424
• Copy Paste: [[2411.08424]] A Heterogeneous Graph Neural Network Fusing Functional and Structural Connectivity for MCI Diagnosis(https://arxiv.org/abs/2411.08424)
• Keywords: diffusion
• Abstract: Brain connectivity alternations associated with brain disorders have been widely reported in resting-state functional imaging (rs-fMRI) and diffusion tensor imaging (DTI). While many dual-modal fusion methods based on graph neural networks (GNNs) have been proposed, they generally follow homogenous fusion ways ignoring rich heterogeneity of dual-modal information. To address this issue, we propose a novel method that integrates functional and structural connectivity based on heterogeneous graph neural networks (HGNNs) to better leverage the rich heterogeneity in dual-modal images. We firstly use blood oxygen level dependency and whiter matter structure information provided by rs-fMRI and DTI to establish homo-meta-path, capturing node relationships within the same modality. At the same time, we propose to establish hetero-meta-path based on structure-function coupling and brain community searching to capture relations among cross-modal nodes. Secondly, we further introduce a heterogeneous graph pooling strategy that automatically balances homo- and hetero-meta-path, effectively leveraging heterogeneous information and preventing feature confusion after pooling. Thirdly, based on the flexibility of heterogeneous graphs, we propose a heterogeneous graph data augmentation approach that can conveniently address the sample imbalance issue commonly seen in clinical diagnosis. We evaluate our method on ADNI-3 dataset for mild cognitive impairment (MCI) diagnosis. Experimental results indicate the proposed method is effective and superior to other algorithms, with a mean classification accuracy of 93.3%.

Title: Properties of fairness measures in the context of varying class imbalance and protected group ratios

• Authors: Dariusz Brzezinski, Julia Stachowiak, Jerzy Stefanowski, Izabela Szczech, Robert Susmaga, Sofya Aksenyuk, Uladzimir Ivashka, Oleksandr Yasinskyi
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.08425
• Pdf URL: https://arxiv.org/pdf/2411.08425
• Copy Paste: [[2411.08425]] Properties of fairness measures in the context of varying class imbalance and protected group ratios(https://arxiv.org/abs/2411.08425)
• Keywords: protect, fair
• Abstract: Society is increasingly relying on predictive models in fields like criminal justice, credit risk management, or hiring. To prevent such automated systems from discriminating against people belonging to certain groups, fairness measures have become a crucial component in socially relevant applications of machine learning. However, existing fairness measures have been designed to assess the bias between predictions for protected groups without considering the imbalance in the classes of the target variable. Current research on the potential effect of class imbalance on fairness focuses on practical applications rather than dataset-independent measure properties. In this paper, we study the general properties of fairness measures for changing class and protected group proportions. For this purpose, we analyze the probability mass functions of six of the most popular group fairness measures. We also measure how the probability of achieving perfect fairness changes for varying class imbalance ratios. Moreover, we relate the dataset-independent properties of fairness measures described in this paper to classifier fairness in real-life tasks. Our results show that measures such as Equal Opportunity and Positive Predictive Parity are more sensitive to changes in class imbalance than Accuracy Equality. These findings can help guide researchers and practitioners in choosing the most appropriate fairness measures for their classification problems.

Title: One STEP at a time: Language Agents are Stepwise Planners

• Authors: Minh Nguyen, Ehsan Shareghi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08432
• Pdf URL: https://arxiv.org/pdf/2411.08432
• Copy Paste: [[2411.08432]] One STEP at a time: Language Agents are Stepwise Planners(https://arxiv.org/abs/2411.08432)
• Keywords: large language model
• Abstract: Language agents have shown promising adaptability in dynamic environments to perform complex tasks. However, despite the versatile knowledge embedded in large language models, these agents still fall short when it comes to tasks that require planning. We introduce STEP, a novel framework designed to efficiently learn from previous experiences to enhance the planning capabilities of language agents in future steps. Concretely, STEP functions through four interconnected components. First, the Planner takes on the task, breaks it down into subtasks and provides relevant insights. Then the Executor generates action candidates, while the Evaluator ensures the actions align with learned rules from previous experiences. Lastly, Memory stores experiences to inform future decisions. In the ScienceWorld benchmark, our results show that STEP consistently outperforms state-of-the-art models, achieving an overall score of 67.4 and successfully completing 12 out of 18 tasks. These findings highlight STEP's potential as a framework for enhancing planning capabilities in language agents, paving the way for more sophisticated task-solving in dynamic environments.

Title: A Fully Local Last-Generated Rule in a Blockchain

• Authors: Akira Sakurai, Kazuyuki Shudo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.08439
• Pdf URL: https://arxiv.org/pdf/2411.08439
• Copy Paste: [[2411.08439]] A Fully Local Last-Generated Rule in a Blockchain(https://arxiv.org/abs/2411.08439)
• Keywords: attack
• Abstract: An effective method for suppressing intentional forks in a blockchain is the last-generated rule, which selects the most recent chain as the main chain in the event of a chain tie. This rule helps invalidate blocks that are withheld by adversaries for a certain period. However, existing last-generated rules face an issue in that their applications to the system are not fully localized. In conservative cryptocurrency systems such as Bitcoin, it is desirable for methods to be applied in a fully local manner. In this paper, we propose a locally applicable last-generated rule. Our method is straightforward and is based on a relative time reference. By conservatively setting the upper bound for the clock skews $\Delta_{O_i}$ to 200 s, our proposed method reduces the proportion $\gamma$ of honest miners following the attacker during chain ties by more than 40% compared to existing local methods.

Title: Machine Unlearning on Pre-trained Models by Residual Feature Alignment Using LoRA

• Authors: Laiqiao Qin, Tianqing Zhu, Linlin Wang, Wanlei Zhou
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08443
• Pdf URL: https://arxiv.org/pdf/2411.08443
• Copy Paste: [[2411.08443]] Machine Unlearning on Pre-trained Models by Residual Feature Alignment Using LoRA(https://arxiv.org/abs/2411.08443)
• Keywords: privacy, protect
• Abstract: Machine unlearning is new emerged technology that removes a subset of the training data from a trained model without affecting the model performance on the remaining data. This topic is becoming increasingly important in protecting user privacy and eliminating harmful or outdated data. The key challenge lies in effectively and efficiently unlearning specific information without compromising the model's utility on the retained data. For the pre-trained models, fine-tuning is an important way to achieve the unlearning target. Previous work typically fine-tuned the entire model's parameters, which incurs significant computation costs. In addition, the fine-tuning process may cause shifts in the intermediate layer features, affecting the model's overall utility. In this work, we propose a novel and efficient machine unlearning method on pre-trained models. We term the method as Residual Feature Alignment Unlearning. Specifically, we leverage LoRA (Low-Rank Adaptation) to decompose the model's intermediate features into pre-trained features and residual features. By adjusting the residual features, we align the unlearned model with the pre-trained model at the intermediate feature level to achieve both unlearning and remaining targets. The method aims to learn the zero residuals on the retained set and shifted residuals on the unlearning set. Extensive experiments on numerous datasets validate the effectiveness of our approach.

Title: Biomass phenotyping of oilseed rape through UAV multi-view oblique imaging with 3DGS and SAM model

• Authors: Yutao Shen (1 and 2), Hongyu Zhou (3), Xin Yang (1 and 2), Xuqi Lu (1 and 2), Ziyue Guo (1 and 2), Lixi Jiang (3), Yong He (1 and 2), Haiyan Cen (1 and 2) ((1) College of Biosystems Engineering and Food Science, Zhejiang University, Hangzhou, P.R. China (2) Key Laboratory of Spectroscopy Sensing, Ministry of Agriculture and Rural Affairs, Hangzhou, P.R. China (3) College of Agriculture and Biotechnology, Zhejiang University, Hangzhou, P.R. China)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08453
• Pdf URL: https://arxiv.org/pdf/2411.08453
• Copy Paste: [[2411.08453]] Biomass phenotyping of oilseed rape through UAV multi-view oblique imaging with 3DGS and SAM model(https://arxiv.org/abs/2411.08453)
• Keywords: extraction, segmentation
• Abstract: Biomass estimation of oilseed rape is crucial for optimizing crop productivity and breeding strategies. While UAV-based imaging has advanced high-throughput phenotyping, current methods often rely on orthophoto images, which struggle with overlapping leaves and incomplete structural information in complex field environments. This study integrates 3D Gaussian Splatting (3DGS) with the Segment Anything Model (SAM) for precise 3D reconstruction and biomass estimation of oilseed rape. UAV multi-view oblique images from 36 angles were used to perform 3D reconstruction, with the SAM module enhancing point cloud segmentation. The segmented point clouds were then converted into point cloud volumes, which were fitted to ground-measured biomass using linear regression. The results showed that 3DGS (7k and 30k iterations) provided high accuracy, with peak signal-to-noise ratios (PSNR) of 27.43 and 29.53 and training times of 7 and 49 minutes, respectively. This performance exceeded that of structure from motion (SfM) and mipmap Neural Radiance Fields (Mip-NeRF), demonstrating superior efficiency. The SAM module achieved high segmentation accuracy, with a mean intersection over union (mIoU) of 0.961 and an F1-score of 0.980. Additionally, a comparison of biomass extraction models found the point cloud volume model to be the most accurate, with an determination coefficient (R2) of 0.976, root mean square error (RMSE) of 2.92 g/plant, and mean absolute percentage error (MAPE) of 6.81%, outperforming both the plot crop volume and individual crop volume models. This study highlights the potential of combining 3DGS with multi-view UAV imaging for improved biomass phenotyping.

Title: Trap-MID: Trapdoor-based Defense against Model Inversion Attacks

• Authors: Zhen-Ting Liu, Shang-Tse Chen
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08460
• Pdf URL: https://arxiv.org/pdf/2411.08460
• Copy Paste: [[2411.08460]] Trap-MID: Trapdoor-based Defense against Model Inversion Attacks(https://arxiv.org/abs/2411.08460)
• Keywords: privacy, defense, attack
• Abstract: Model Inversion (MI) attacks pose a significant threat to the privacy of Deep Neural Networks by recovering training data distribution from well-trained models. While existing defenses often rely on regularization techniques to reduce information leakage, they remain vulnerable to recent attacks. In this paper, we propose the Trapdoor-based Model Inversion Defense (Trap-MID) to mislead MI attacks. A trapdoor is integrated into the model to predict a specific label when the input is injected with the corresponding trigger. Consequently, this trapdoor information serves as the "shortcut" for MI attacks, leading them to extract trapdoor triggers rather than private data. We provide theoretical insights into the impacts of trapdoor's effectiveness and naturalness on deceiving MI attacks. In addition, empirical experiments demonstrate the state-of-the-art defense performance of Trap-MID against various MI attacks without the requirements for extra data or large computational overhead. Our source code is publicly available at this https URL.

Title: Can MLLMs Guide Weakly-Supervised Temporal Action Localization Tasks?

• Authors: Quan Zhang, Yuxin Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08466
• Pdf URL: https://arxiv.org/pdf/2411.08466
• Copy Paste: [[2411.08466]] Can MLLMs Guide Weakly-Supervised Temporal Action Localization Tasks?(https://arxiv.org/abs/2411.08466)
• Keywords: robust, large language model
• Abstract: Recent breakthroughs in Multimodal Large Language Models (MLLMs) have gained significant recognition within the deep learning community, where the fusion of the Video Foundation Models (VFMs) and Large Language Models(LLMs) has proven instrumental in constructing robust video understanding systems, effectively surmounting constraints associated with predefined visual tasks. These sophisticated MLLMs exhibit remarkable proficiency in comprehending videos, swiftly attaining unprecedented performance levels across diverse benchmarks. However, their operation demands substantial memory and computational resources, underscoring the continued importance of traditional models in video comprehension tasks. In this paper, we introduce a novel learning paradigm termed MLLM4WTAL. This paradigm harnesses the potential of MLLM to offer temporal action key semantics and complete semantic priors for conventional Weakly-supervised Temporal Action Localization (WTAL) methods. MLLM4WTAL facilitates the enhancement of WTAL by leveraging MLLM guidance. It achieves this by integrating two distinct modules: Key Semantic Matching (KSM) and Complete Semantic Reconstruction (CSR). These modules work in tandem to effectively address prevalent issues like incomplete and over-complete outcomes common in WTAL methods. Rigorous experiments are conducted to validate the efficacy of our proposed approach in augmenting the performance of various heterogeneous WTAL models.

Title: HyperFace: Generating Synthetic Face Recognition Datasets by Exploring Face Embedding Hypersphere

• Authors: Hatef Otroshi Shahreza, Sébastien Marcel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08470
• Pdf URL: https://arxiv.org/pdf/2411.08470
• Copy Paste: [[2411.08470]] HyperFace: Generating Synthetic Face Recognition Datasets by Exploring Face Embedding Hypersphere(https://arxiv.org/abs/2411.08470)
• Keywords: privacy, generative
• Abstract: Face recognition datasets are often collected by crawling Internet and without individuals' consents, raising ethical and privacy concerns. Generating synthetic datasets for training face recognition models has emerged as a promising alternative. However, the generation of synthetic datasets remains challenging as it entails adequate inter-class and intra-class variations. While advances in generative models have made it easier to increase intra-class variations in face datasets (such as pose, illumination, etc.), generating sufficient inter-class variation is still a difficult task. In this paper, we formulate the dataset generation as a packing problem on the embedding space (represented on a hypersphere) of a face recognition model and propose a new synthetic dataset generation approach, called HyperFace. We formalize our packing problem as an optimization problem and solve it with a gradient descent-based approach. Then, we use a conditional face generator model to synthesize face images from the optimized embeddings. We use our generated datasets to train face recognition models and evaluate the trained models on several benchmarking real datasets. Our experimental results show that models trained with HyperFace achieve state-of-the-art performance in training face recognition using synthetic datasets.

Title: A survey on Graph Deep Representation Learning for Facial Expression Recognition

• Authors: Théo Gueuret, Akrem Sellami, Chaabane Djeraba
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08472
• Pdf URL: https://arxiv.org/pdf/2411.08472
• Copy Paste: [[2411.08472]] A survey on Graph Deep Representation Learning for Facial Expression Recognition(https://arxiv.org/abs/2411.08472)
• Keywords: diffusion
• Abstract: This comprehensive review delves deeply into the various methodologies applied to facial expression recognition (FER) through the lens of graph representation learning (GRL). Initially, we introduce the task of FER and the concepts of graph representation and GRL. Afterward, we discuss some of the most prevalent and valuable databases for this task. We explore promising approaches for graph representation in FER, including graph diffusion, spatio-temporal graphs, and multi-stream architectures. Finally, we identify future research opportunities and provide concluding remarks.

Title: Methodology for a Statistical Analysis of Influencing Factors on 3D Object Detection Performance

• Authors: Anton Kuznietsov, Dirk Schweickard, Steven Peters
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08482
• Pdf URL: https://arxiv.org/pdf/2411.08482
• Copy Paste: [[2411.08482]] Methodology for a Statistical Analysis of Influencing Factors on 3D Object Detection Performance(https://arxiv.org/abs/2411.08482)
• Keywords: robust
• Abstract: In autonomous driving, object detection is an essential task to perceive the environment by localizing and classifying objects. Most object detection algorithms rely on deep learning for their superior performance. However, their black box nature makes it challenging to ensure safety. In this paper, we propose a first-of-its-kind methodology for statistical analysis of the influence of various factors related to the objects to detect or the environment on the detection performance of both LiDAR- and camera-based 3D object detectors. We perform a univariate analysis between each of the factors and the detection error in order to compare the strength of influence. To better identify potential sources of detection errors, we also analyze the performance in dependency of the influencing factors and examine the interdependencies between the different influencing factors. Recognizing the factors that influence detection performance helps identify robustness issues in the trained object detector and supports the safety approval of object detection systems.

Title: Impact of Iris Pigmentation on Performance Bias in Visible Iris Verification Systems: A Comparative Study

• Authors: Geetanjali Sharma, Abhishek Tandon, Gaurav Jaswal, Aditya Nigam, Raghavendra Ramachandra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08490
• Pdf URL: https://arxiv.org/pdf/2411.08490
• Copy Paste: [[2411.08490]] Impact of Iris Pigmentation on Performance Bias in Visible Iris Verification Systems: A Comparative Study(https://arxiv.org/abs/2411.08490)
• Keywords: robust, biometric
• Abstract: Iris recognition technology plays a critical role in biometric identification systems, but their performance can be affected by variations in iris pigmentation. In this work, we investigate the impact of iris pigmentation on the efficacy of biometric recognition systems, focusing on a comparative analysis of blue and dark irises. Data sets were collected using multiple devices, including P1, P2, and P3 smartphones [4], to assess the robustness of the systems in different capture environments [19]. Both traditional machine learning techniques and deep learning models were used, namely Open-Iris, ViT-b, and ResNet50, to evaluate performance metrics such as Equal Error Rate (EER) and True Match Rate (TMR). Our results indicate that iris recognition systems generally exhibit higher accuracy for blue irises compared to dark irises. Furthermore, we examined the generalization capabilities of these systems across different iris colors and devices, finding that while training on diverse datasets enhances recognition performance, the degree of improvement is contingent on the specific model and device used. Our analysis also identifies inherent biases in recognition performance related to iris color and cross-device variability. These findings underscore the need for more inclusive dataset collection and model refinement to reduce bias and promote equitable biometric recognition across varying iris pigmentation and device configurations.

Title: An Information Theoretic Approach to Operationalize Right to Data Protection

• Authors: Abhinav Java, Simra Shahid, Chirag Agarwal
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.08506
• Pdf URL: https://arxiv.org/pdf/2411.08506
• Copy Paste: [[2411.08506]] An Information Theoretic Approach to Operationalize Right to Data Protection(https://arxiv.org/abs/2411.08506)
• Keywords: protect
• Abstract: The widespread practice of indiscriminate data scraping to fine-tune language models (LMs) raises significant legal and ethical concerns, particularly regarding compliance with data protection laws such as the General Data Protection Regulation (GDPR). This practice often results in the unauthorized use of personal information, prompting growing debate within the academic and regulatory communities. Recent works have introduced the concept of generating unlearnable datasets (by adding imperceptible noise to the clean data), such that the underlying model achieves lower loss during training but fails to generalize to the unseen test setting. Though somewhat effective, these approaches are predominantly designed for images and are limited by several practical constraints like requiring knowledge of the target model. To this end, we introduce RegText, a framework that injects imperceptible spurious correlations into natural language datasets, effectively rendering them unlearnable without affecting semantic content. We demonstrate RegText's utility through rigorous empirical analysis of small and large LMs. Notably, RegText can restrict newer models like GPT-4o and Llama from learning on our generated data, resulting in a drop in their test accuracy compared to their zero-shot performance and paving the way for generating unlearnable text to protect public data.

Title: Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understanding

• Authors: Deyi Ji, Lanyun Zhu, Siqi Gao, Peng Xu, Hongtao Lu, Jieping Ye, Feng Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.08516
• Pdf URL: https://arxiv.org/pdf/2411.08516
• Copy Paste: [[2411.08516]] Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understanding(https://arxiv.org/abs/2411.08516)
• Keywords: large language model
• Abstract: The ubiquity and value of tables as semi-structured data across various domains necessitate advanced methods for understanding their complexity and vast amounts of information. Despite the impressive capabilities of large language models (LLMs) in advancing the natural language understanding frontier, their application to large-scale tabular data presents significant challenges, specifically regarding table size and complex intricate relationships. Existing works have shown promise with small-scale tables but often flounder when tasked with the complex reasoning required by larger, interconnected tables found in real-world scenarios. To address this gap, we introduce "Tree-of-Table", a novel approach designed to enhance LLMs' reasoning capabilities over large and complex tables. Our method employs Table Condensation and Decomposition to distill and reorganize relevant data into a manageable format, followed by the construction of a hierarchical Table-Tree that facilitates tree-structured reasoning. Through a meticulous Table-Tree Execution process, we systematically unravel the tree-structured reasoning chain to derive the solutions. Experiments across diverse datasets, including WikiTQ, TableFact, FeTaQA, and BIRD, demonstrate that Tree-of-Table sets a new benchmark with superior performance, showcasing remarkable efficiency and generalization capabilities in large-scale table reasoning.

Title: SAD-TIME: a Spatiotemporal-fused network for depression detection with Automated multi-scale Depth-wise and TIME-interval-related common feature extractor

• Authors: Han-Guang Wang, Hui-Rang Hou, Li-Cheng Jin, Chen-Yang Xu, Zhong-Yi Zhang, Qing-Hao Meng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08521
• Pdf URL: https://arxiv.org/pdf/2411.08521
• Copy Paste: [[2411.08521]] SAD-TIME: a Spatiotemporal-fused network for depression detection with Automated multi-scale Depth-wise and TIME-interval-related common feature extractor(https://arxiv.org/abs/2411.08521)
• Keywords: robust, transformer
• Abstract: Background and Objective: Depression is a severe mental disorder, and accurate diagnosis is pivotal to the cure and rehabilitation of people with depression. However, the current questionnaire-based diagnostic methods could bring subjective biases and may be denied by subjects. In search of a more objective means of diagnosis, researchers have begun to experiment with deep learning-based methods for identifying depressive disorders in recent years. Methods: In this study, a novel Spatiotemporal-fused network with Automated multi-scale Depth-wise and TIME-interval-related common feature extractor (SAD-TIME) is proposed. SAD-TIME incorporates an automated nodes' common features extractor (CFE), a spatial sector (SpS), a modified temporal sector (TeS), and a domain adversarial learner (DAL). The CFE includes a multi-scale depth-wise 1D-convolutional neural network and a time-interval embedding generator, where the unique information of each channel is preserved. The SpS fuses the functional connectivity with the distance-based connectivity containing spatial position of EEG electrodes. A multi-head-attention graph convolutional network is also applied in the SpS to fuse the features from different EEG channels. The TeS is based on long short-term memory and graph transformer networks, where the temporal information of different time-windows is fused. Moreover, the DAL is used after the SpS to obtain the domain-invariant feature. Results: Experimental results under tenfold cross-validation show that the proposed SAD-TIME method achieves 92.00% and 94.00% depression classification accuracies on two datasets, respectively, in cross-subject mode. Conclusion: SAD-TIME is a robust depression detection model, where the automatedly-generated features, the SpS and the TeS assist the classification performance with the fusion of the innate spatiotemporal information in the EEG signals.

Title: Efficient Whole Slide Image Classification through Fisher Vector Representation

• Authors: Ravi Kant Gupta, Dadi Dharani, Shambhavi Shanker, Amit Sethi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08530
• Pdf URL: https://arxiv.org/pdf/2411.08530
• Copy Paste: [[2411.08530]] Efficient Whole Slide Image Classification through Fisher Vector Representation(https://arxiv.org/abs/2411.08530)
• Keywords: robust
• Abstract: The advancement of digital pathology, particularly through computational analysis of whole slide images (WSI), is poised to significantly enhance diagnostic precision and efficiency. However, the large size and complexity of WSIs make it difficult to analyze and classify them using computers. This study introduces a novel method for WSI classification by automating the identification and examination of the most informative patches, thus eliminating the need to process the entire slide. Our method involves two-stages: firstly, it extracts only a few patches from the WSIs based on their pathological significance; and secondly, it employs Fisher vectors (FVs) for representing features extracted from these patches, which is known for its robustness in capturing fine-grained details. This approach not only accentuates key pathological features within the WSI representation but also significantly reduces computational overhead, thus making the process more efficient and scalable. We have rigorously evaluated the proposed method across multiple datasets to benchmark its performance against comprehensive WSI analysis and contemporary weakly-supervised learning methodologies. The empirical results indicate that our focused analysis of select patches, combined with Fisher vector representation, not only aligns with, but at times surpasses, the classification accuracy of standard practices. Moreover, this strategy notably diminishes computational load and resource expenditure, thereby establishing an efficient and precise framework for WSI analysis in the realm of digital pathology.

Title: Classification and Morphological Analysis of DLBCL Subtypes in H\&E-Stained Slides

• Authors: Ravi Kant Gupta, Mohit Jindal, Garima Jain, Epari Sridhar, Subhash Yadav, Hasmukh Jain, Tanuja Shet, Uma Sakhdeo, Manju Sengar, Lingaraj Nayak, Bhausaheb Bagal, Umesh Apkare, Amit Sethi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08531
• Pdf URL: https://arxiv.org/pdf/2411.08531
• Copy Paste: [[2411.08531]] Classification and Morphological Analysis of DLBCL Subtypes in H\&E-Stained Slides(https://arxiv.org/abs/2411.08531)
• Keywords: robust
• Abstract: We address the challenge of automated classification of diffuse large B-cell lymphoma (DLBCL) into its two primary subtypes: activated B-cell-like (ABC) and germinal center B-cell-like (GCB). Accurate classification between these subtypes is essential for determining the appropriate therapeutic strategy, given their distinct molecular profiles and treatment responses. Our proposed deep learning model demonstrates robust performance, achieving an average area under the curve (AUC) of (87.4 pm 5.7)\% during cross-validation. It shows a high positive predictive value (PPV), highlighting its potential for clinical application, such as triaging for molecular testing. To gain biological insights, we performed an analysis of morphological features of ABC and GCB subtypes. We segmented cell nuclei using a pre-trained deep neural network and compared the statistics of geometric and color features for ABC and GCB. We found that the distributions of these features were not very different for the two subtypes, which suggests that the visual differences between them are more subtle. These results underscore the potential of our method to assist in more precise subtype classification and can contribute to improved treatment management and outcomes for patients of DLBCL.

Title: Neural Topic Modeling with Large Language Models in the Loop

• Authors: Xiaohao Yang, He Zhao, Weijie Xu, Yuanyuan Qi, Jueqing Lu, Dinh Phung, Lan Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.08534
• Pdf URL: https://arxiv.org/pdf/2411.08534
• Copy Paste: [[2411.08534]] Neural Topic Modeling with Large Language Models in the Loop(https://arxiv.org/abs/2411.08534)
• Keywords: interpretability, large language model
• Abstract: Topic modeling is a fundamental task in natural language processing, allowing the discovery of latent thematic structures in text corpora. While Large Language Models (LLMs) have demonstrated promising capabilities in topic discovery, their direct application to topic modeling suffers from issues such as incomplete topic coverage, misalignment of topics, and inefficiency. To address these limitations, we propose LLM-ITL, a novel LLM-in-the-loop framework that integrates LLMs with many existing Neural Topic Models (NTMs). In LLM-ITL, global topics and document representations are learned through the NTM, while an LLM refines the topics via a confidence-weighted Optimal Transport (OT)-based alignment objective. This process enhances the interpretability and coherence of the learned topics, while maintaining the efficiency of NTMs. Extensive experiments demonstrate that LLM-ITL can help NTMs significantly improve their topic interpretability while maintaining the quality of document representation.

Title: MLV$^2$-Net: Rater-Based Majority-Label Voting for Consistent Meningeal Lymphatic Vessel Segmentation

• Authors: Fabian Bongratz, Markus Karmann, Adrian Holz, Moritz Bonhoeffer, Viktor Neumaier, Sarah Deli, Benita Schmitz-Koep, Claus Zimmer, Christian Sorg, Melissa Thalhammer, Dennis M Hedderich, Christian Wachinger
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08537
• Pdf URL: https://arxiv.org/pdf/2411.08537
• Copy Paste: [[2411.08537]] MLV$^2$-Net: Rater-Based Majority-Label Voting for Consistent Meningeal Lymphatic Vessel Segmentation(https://arxiv.org/abs/2411.08537)
• Keywords: segmentation
• Abstract: Meningeal lymphatic vessels (MLVs) are responsible for the drainage of waste products from the human brain. An impairment in their functionality has been associated with aging as well as brain disorders like multiple sclerosis and Alzheimer's disease. However, MLVs have only recently been described for the first time in magnetic resonance imaging (MRI), and their ramified structure renders manual segmentation particularly difficult. Further, as there is no consistent notion of their appearance, human-annotated MLV structures contain a high inter-rater variability that most automatic segmentation methods cannot take into account. In this work, we propose a new rater-aware training scheme for the popular nnU-Net model, and we explore rater-based ensembling strategies for accurate and consistent segmentation of MLVs. This enables us to boost nnU-Net's performance while obtaining explicit predictions in different annotation styles and a rater-based uncertainty estimation. Our final model, MLV$^2$-Net, achieves a Dice similarity coefficient of 0.806 with respect to the human reference standard. The model further matches the human inter-rater reliability and replicates age-related associations with MLV volume.

Title: CorrSynth -- A Correlated Sampling Method for Diverse Dataset Generation from LLMs

• Authors: Suhas S Kowshik, Abhishek Divekar, Vijit Malik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.08553
• Pdf URL: https://arxiv.org/pdf/2411.08553
• Copy Paste: [[2411.08553]] CorrSynth -- A Correlated Sampling Method for Diverse Dataset Generation from LLMs(https://arxiv.org/abs/2411.08553)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable performance in diverse tasks using zero-shot and few-shot prompting. Even though their capabilities of data synthesis have been studied well in recent years, the generated data suffers from a lack of diversity, less adherence to the prompt, and potential biases that creep into the data from the generator model. In this work, we tackle the challenge of generating datasets with high diversity, upon which a student model is trained for downstream tasks. Taking the route of decoding-time guidance-based approaches, we propose CorrSynth, which generates data that is more diverse and faithful to the input prompt using a correlated sampling strategy. Further, our method overcomes the complexity drawbacks of some other guidance-based techniques like classifier-based guidance. With extensive experiments, we show the effectiveness of our approach and substantiate our claims. In particular, we perform intrinsic evaluation to show the improvements in diversity. Our experiments show that CorrSynth improves both student metrics and intrinsic metrics upon competitive baselines across four datasets, showing the innate advantage of our method.

Title: Saliency Map-based Image Retrieval using Invariant Krawtchouk Moments

• Authors: Ashkan Nejad, Mohammad Reza Faraji, Xiaojun Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08567
• Pdf URL: https://arxiv.org/pdf/2411.08567
• Copy Paste: [[2411.08567]] Saliency Map-based Image Retrieval using Invariant Krawtchouk Moments(https://arxiv.org/abs/2411.08567)
• Keywords: extraction
• Abstract: With the widespread adoption of digital devices equipped with cameras and the rapid development of Internet technology, numerous content-based image retrieval systems and novel image feature extraction techniques have emerged in recent years. This paper introduces a saliency map-based image retrieval approach using invariant Krawtchouk moments (SM-IKM) to enhance retrieval speed and accuracy. The proposed method applies a global contrast-based salient region detection algorithm to create a saliency map that effectively isolates the foreground from the background. It then combines multiple orders of invariant Krawtchouk moments (IKM) with local binary patterns (LBPs) and color histograms to comprehensively represent the foreground and background. Additionally, it incorporates LBPs derived from the saliency map to improve discriminative power, facilitating more precise image differentiation. A bag-of-visual-words (BoVW) model is employed to generate a codebook for classification and discrimination. By using compact IKMs in the BoVW framework and integrating a range of region-based feature-including color histograms, LBPs, and saliency map-enhanced LBPs, our proposed SM-IKM achieves efficient and accurate image retrieval. xtensive experiments on publicly available datasets, such as Caltech 101 and Wang, demonstrate that SM-IKM outperforms recent state-of-the-art retrieval methods. The source code for SM-IKM is available at this http URL.

Title: UIFormer: A Unified Transformer-based Framework for Incremental Few-Shot Object Detection and Instance Segmentation

• Authors: Chengyuan Zhang, Yilin Zhang, Lei Zhu, Deyin Liu, Lin Wu, Bo Li, Shichao Zhang, Mohammed Bennamoun, Farid Boussaid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08569
• Pdf URL: https://arxiv.org/pdf/2411.08569
• Copy Paste: [[2411.08569]] UIFormer: A Unified Transformer-based Framework for Incremental Few-Shot Object Detection and Instance Segmentation(https://arxiv.org/abs/2411.08569)
• Keywords: transformer, segmentation
• Abstract: This paper introduces a novel framework for unified incremental few-shot object detection (iFSOD) and instance segmentation (iFSIS) using the Transformer architecture. Our goal is to create an optimal solution for situations where only a few examples of novel object classes are available, with no access to training data for base or old classes, while maintaining high performance across both base and novel classes. To achieve this, We extend Mask-DINO into a two-stage incremental learning framework. Stage 1 focuses on optimizing the model using the base dataset, while Stage 2 involves fine-tuning the model on novel classes. Besides, we incorporate a classifier selection strategy that assigns appropriate classifiers to the encoder and decoder according to their distinct functions. Empirical evidence indicates that this approach effectively mitigates the over-fitting on novel classes learning. Furthermore, we implement knowledge distillation to prevent catastrophic forgetting of base classes. Comprehensive evaluations on the COCO and LVIS datasets for both iFSIS and iFSOD tasks demonstrate that our method significantly outperforms state-of-the-art approaches.

Title: Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval

• Authors: Saul Santos, Vlad Niculae, Daniel McNamee, André F. T. Martins
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08590
• Pdf URL: https://arxiv.org/pdf/2411.08590
• Copy Paste: [[2411.08590]] Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval(https://arxiv.org/abs/2411.08590)
• Keywords: transformer
• Abstract: Associative memory models, such as Hopfield networks and their modern variants, have garnered renewed interest due to advancements in memory capacity and connections with self-attention in transformers. In this work, we introduce a unified framework-Hopfield-Fenchel-Young networks-which generalizes these models to a broader family of energy functions. Our energies are formulated as the difference between two Fenchel-Young losses: one, parameterized by a generalized entropy, defines the Hopfield scoring mechanism, while the other applies a post-transformation to the Hopfield output. By utilizing Tsallis and norm entropies, we derive end-to-end differentiable update rules that enable sparse transformations, uncovering new connections between loss margins, sparsity, and exact retrieval of single memory patterns. We further extend this framework to structured Hopfield networks using the SparseMAP transformation, allowing the retrieval of pattern associations rather than a single pattern. Our framework unifies and extends traditional and modern Hopfield networks and provides an energy minimization perspective for widely used post-transformations like $\ell_2$-normalization and layer normalization-all through suitable choices of Fenchel-Young losses and by using convex analysis as a building block. Finally, we validate our Hopfield-Fenchel-Young networks on diverse memory recall tasks, including free and sequential recall. Experiments on simulated data, image retrieval, multiple instance learning, and text rationalization demonstrate the effectiveness of our approach.

Title: Slender Object Scene Segmentation in Remote Sensing Image Based on Learnable Morphological Skeleton with Segment Anything Model

• Authors: Jun Xie, Wenxiao Li, Faqiang Wang, Liqiang Zhang, Zhengyang Hou, Jun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08592
• Pdf URL: https://arxiv.org/pdf/2411.08592
• Copy Paste: [[2411.08592]] Slender Object Scene Segmentation in Remote Sensing Image Based on Learnable Morphological Skeleton with Segment Anything Model(https://arxiv.org/abs/2411.08592)
• Keywords: segmentation
• Abstract: Morphological methods play a crucial role in remote sensing image processing, due to their ability to capture and preserve small structural details. However, most of the existing deep learning models for semantic segmentation are based on the encoder-decoder architecture including U-net and Segment Anything Model (SAM), where the downsampling process tends to discard fine details. In this paper, we propose a new approach that integrates learnable morphological skeleton prior into deep neural networks using the variational method. To address the difficulty in backpropagation in neural networks caused by the non-differentiability presented in classical morphological operations, we provide a smooth representation of the morphological skeleton and design a variational segmentation model integrating morphological skeleton prior by employing operator splitting and dual methods. Then, we integrate this model into the network architecture of SAM, which is achieved by adding a token to mask decoder and modifying the final sigmoid layer, ensuring the final segmentation results preserve the skeleton structure as much as possible. Experimental results on remote sensing datasets, including buildings and roads, demonstrate that our method outperforms the original SAM on slender object segmentation and exhibits better generalization capability.

Title: Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Models

• Authors: Felix Stahlberg, Jared Lichtarge, Shankar Kumar
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08610
• Pdf URL: https://arxiv.org/pdf/2411.08610
• Copy Paste: [[2411.08610]] Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Models(https://arxiv.org/abs/2411.08610)
• Keywords: large language model
• Abstract: We propose a novel parameter-efficient training (PET) method for large language models that adapts models to downstream tasks by optimizing a small subset of the existing model parameters. Unlike prior methods, this subset is not fixed in location but rather which parameters are modified evolves over the course of training. This dynamic parameter selection can yield good performance with many fewer parameters than extant methods. Our method enables a seamless scaling of the subset size across an arbitrary proportion of the total model size, while popular PET approaches like prompt tuning and LoRA cover only a small part of this spectrum. We match or outperform prompt tuning and LoRA in most cases on a variety of NLP tasks (MT, QA, GSM8K, SuperGLUE) for a given parameter budget across different model families and sizes.

Title: Zero-shot capability of SAM-family models for bone segmentation in CT scans

• Authors: Caroline Magg, Hoel Kervadec, Clara I. Sánchez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08629
• Pdf URL: https://arxiv.org/pdf/2411.08629
• Copy Paste: [[2411.08629]] Zero-shot capability of SAM-family models for bone segmentation in CT scans(https://arxiv.org/abs/2411.08629)
• Keywords: segmentation
• Abstract: The Segment Anything Model (SAM) and similar models build a family of promptable foundation models (FMs) for image and video segmentation. The object of interest is identified using prompts, such as bounding boxes or points. With these FMs becoming part of medical image segmentation, extensive evaluation studies are required to assess their strengths and weaknesses in clinical setting. Since the performance is highly dependent on the chosen prompting strategy, it is important to investigate different prompting techniques to define optimal guidelines that ensure effective use in medical image segmentation. Currently, no dedicated evaluation studies exist specifically for bone segmentation in CT scans, leaving a gap in understanding the performance for this task. Thus, we use non-iterative, ``optimal'' prompting strategies composed of bounding box, points and combinations to test the zero-shot capability of SAM-family models for bone CT segmentation on three different skeletal regions. Our results show that the best settings depend on the model type and size, dataset characteristics and objective to optimize. Overall, SAM and SAM2 prompted with a bounding box in combination with the center point for all the components of an object yield the best results across all tested settings. As the results depend on multiple factors, we provide a guideline for informed decision-making in 2D prompting with non-interactive, ''optimal'' prompts.

Title: Robot See, Robot Do: Imitation Reward for Noisy Financial Environments

• Authors: Sven Goluža, Tomislav Kovačević, Stjepan Begušić, Zvonko Kostanjčar
• Subjects: cs.LG, cs.RO, q-fin.TR
• Abstract URL: https://arxiv.org/abs/2411.08637
• Pdf URL: https://arxiv.org/pdf/2411.08637
• Copy Paste: [[2411.08637]] Robot See, Robot Do: Imitation Reward for Noisy Financial Environments(https://arxiv.org/abs/2411.08637)
• Keywords: robust
• Abstract: The sequential nature of decision-making in financial asset trading aligns naturally with the reinforcement learning (RL) framework, making RL a common approach in this domain. However, the low signal-to-noise ratio in financial markets results in noisy estimates of environment components, including the reward function, which hinders effective policy learning by RL agents. Given the critical importance of reward function design in RL problems, this paper introduces a novel and more robust reward function by leveraging imitation learning, where a trend labeling algorithm acts as an expert. We integrate imitation (expert's) feedback with reinforcement (agent's) feedback in a model-free RL algorithm, effectively embedding the imitation learning problem within the RL paradigm to handle the stochasticity of reward signals. Empirical results demonstrate that this novel approach improves financial performance metrics compared to traditional benchmarks and RL agents trained solely using reinforcement feedback.

Title: Towards Secure Intelligent O-RAN Architecture: Vulnerabilities, Threats and Promising Technical Solutions using LLMs

• Authors: Mojdeh Karbalaee Motalleb, Chafika Benzaid, Tarik Taleb, Marcos Katz, Vahid Shah-Mansouri, JaeSeung Song
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08640
• Pdf URL: https://arxiv.org/pdf/2411.08640
• Copy Paste: [[2411.08640]] Towards Secure Intelligent O-RAN Architecture: Vulnerabilities, Threats and Promising Technical Solutions using LLMs(https://arxiv.org/abs/2411.08640)
• Keywords: secure, security, defense, robust, large language model
• Abstract: The evolution of wireless communication systems will be fundamentally impacted by an open radio access network (O-RAN), a new concept defining an intelligent architecture with enhanced flexibility, openness, and the ability to slice services more efficiently. For all its promises, and like any technological advancement, O-RAN is not without risks that need to be carefully assessed and properly addressed to accelerate its wide adoption in future mobile networks. In this paper, we present an in-depth security analysis of the O-RAN architecture, discussing the potential threats that may arise in the different O-RAN architecture layers and their impact on the Confidentiality, Integrity, and Availability (CIA) triad. We also promote the potential of zero trust, Moving Target Defense (MTD), blockchain, and large language models(LLM) technologies in fortifying O-RAN's security posture. Furthermore, we numerically demonstrate the effectiveness of MTD in empowering robust deep reinforcement learning methods for dynamic network slice admission control in the O-RAN architecture. Moreover, we examine the effect of explainable AI (XAI) based on LLMs in securing the system.

Title: Towards More Accurate Fake Detection on Images Generated from Advanced Generative and Neural Rendering Models

• Authors: Chengdong Dong, Vijayakumar Bhagavatula, Zhenyu Zhou, Ajay Kumar
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08642
• Pdf URL: https://arxiv.org/pdf/2411.08642
• Copy Paste: [[2411.08642]] Towards More Accurate Fake Detection on Images Generated from Advanced Generative and Neural Rendering Models(https://arxiv.org/abs/2411.08642)
• Keywords: robust, diffusion, generative
• Abstract: The remarkable progress in neural-network-driven visual data generation, especially with neural rendering techniques like Neural Radiance Fields and 3D Gaussian splatting, offers a powerful alternative to GANs and diffusion models. These methods can produce high-fidelity images and lifelike avatars, highlighting the need for robust detection methods. In response, an unsupervised training technique is proposed that enables the model to extract comprehensive features from the Fourier spectrum magnitude, thereby overcoming the challenges of reconstructing the spectrum due to its centrosymmetric properties. By leveraging the spectral domain and dynamically combining it with spatial domain information, we create a robust multimodal detector that demonstrates superior generalization capabilities in identifying challenging synthetic images generated by the latest image synthesis techniques. To address the absence of a 3D neural rendering-based fake image database, we develop a comprehensive database that includes images generated by diverse neural rendering techniques, providing a robust foundation for evaluating and advancing detection methods.

Title: MikuDance: Animating Character Art with Mixed Motion Dynamics

• Authors: Jiaxu Zhang, Xianfang Zeng, Xin Chen, Wei Zuo, Gang Yu, Zhigang Tu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08656
• Pdf URL: https://arxiv.org/pdf/2411.08656
• Copy Paste: [[2411.08656]] MikuDance: Animating Character Art with Mixed Motion Dynamics(https://arxiv.org/abs/2411.08656)
• Keywords: diffusion
• Abstract: We propose MikuDance, a diffusion-based pipeline incorporating mixed motion dynamics to animate stylized character art. MikuDance consists of two key techniques: Mixed Motion Modeling and Mixed-Control Diffusion, to address the challenges of high-dynamic motion and reference-guidance misalignment in character art animation. Specifically, a Scene Motion Tracking strategy is presented to explicitly model the dynamic camera in pixel-wise space, enabling unified character-scene motion modeling. Building on this, the Mixed-Control Diffusion implicitly aligns the scale and body shape of diverse characters with motion guidance, allowing flexible control of local character motion. Subsequently, a Motion-Adaptive Normalization module is incorporated to effectively inject global scene motion, paving the way for comprehensive character art animation. Through extensive experiments, we demonstrate the effectiveness and generalizability of MikuDance across various character art and motion guidance, consistently producing high-quality animations with remarkable motion dynamics.

Title: Toward Human Understanding with Controllable Synthesis

• Authors: Hanz Cuevas-Velasquez, Priyanka Patel, Haiwen Feng, Michael Black
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08663
• Pdf URL: https://arxiv.org/pdf/2411.08663
• Copy Paste: [[2411.08663]] Toward Human Understanding with Controllable Synthesis(https://arxiv.org/abs/2411.08663)
• Keywords: robust, generative
• Abstract: Training methods to perform robust 3D human pose and shape (HPS) estimation requires diverse training images with accurate ground truth. While BEDLAM demonstrates the potential of traditional procedural graphics to generate such data, the training images are clearly synthetic. In contrast, generative image models produce highly realistic images but without ground truth. Putting these methods together seems straightforward: use a generative model with the body ground truth as controlling signal. However, we find that, the more realistic the generated images, the more they deviate from the ground truth, making them inappropriate for training and evaluation. Enhancements of realistic details, such as clothing and facial expressions, can lead to subtle yet significant deviations from the ground truth, potentially misleading training models. We empirically verify that this misalignment causes the accuracy of HPS networks to decline when trained with generated images. To address this, we design a controllable synthesis method that effectively balances image realism with precise ground truth. We use this to create the Generative BEDLAM (Gen-B) dataset, which improves the realism of the existing synthetic BEDLAM dataset while preserving ground truth accuracy. We perform extensive experiments, with various noise-conditioning strategies, to evaluate the tradeoff between visual realism and HPS accuracy. We show, for the first time, that generative image models can be controlled by traditional graphics methods to produce training data that increases the accuracy of HPS methods.

Title: UniMat: Unifying Materials Embeddings through Multi-modal Learning

• Authors: Janghoon Ock, Joseph Montoya, Daniel Schweigert, Linda Hung, Santosh K. Suram, Weike Ye
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2411.08664
• Pdf URL: https://arxiv.org/pdf/2411.08664
• Copy Paste: [[2411.08664]] UniMat: Unifying Materials Embeddings through Multi-modal Learning(https://arxiv.org/abs/2411.08664)
• Keywords: robust
• Abstract: Materials science datasets are inherently heterogeneous and are available in different modalities such as characterization spectra, atomic structures, microscopic images, and text-based synthesis conditions. The advancements in multi-modal learning, particularly in vision and language models, have opened new avenues for integrating data in different forms. In this work, we evaluate common techniques in multi-modal learning (alignment and fusion) in unifying some of the most important modalities in materials science: atomic structure, X-ray diffraction patterns (XRD), and composition. We show that structure graph modality can be enhanced by aligning with XRD patterns. Additionally, we show that aligning and fusing more experimentally accessible data formats, such as XRD patterns and compositions, can create more robust joint embeddings than individual modalities across various tasks. This lays the groundwork for future studies aiming to exploit the full potential of multi-modal data in materials science, facilitating more informed decision-making in materials design and discovery.

Title: OSMLoc: Single Image-Based Visual Localization in OpenStreetMap with Geometric and Semantic Guidances

• Authors: Youqi Liao, Xieyuanli Chen, Shuhao Kang, Jianping Li, Zhen Dong, Hongchao Fan, Bisheng Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08665
• Pdf URL: https://arxiv.org/pdf/2411.08665
• Copy Paste: [[2411.08665]] OSMLoc: Single Image-Based Visual Localization in OpenStreetMap with Geometric and Semantic Guidances(https://arxiv.org/abs/2411.08665)
• Keywords: robust
• Abstract: OpenStreetMap (OSM), an online and versatile source of volunteered geographic information (VGI), is widely used for human self-localization by matching nearby visual observations with vectorized map data. However, due to the divergence in modalities and views, image-to-OSM (I2O) matching and localization remain challenging for robots, preventing the full utilization of VGI data in the unmanned ground vehicles and logistic industry. Inspired by the fact that the human brain relies on geometric and semantic understanding of sensory information for spatial localization tasks, we propose the OSMLoc in this paper. OSMLoc is a brain-inspired single-image visual localization method with semantic and geometric guidance to improve accuracy, robustness, and generalization ability. First, we equip the OSMLoc with the visual foundational model to extract powerful image features. Second, a geometry-guided depth distribution adapter is proposed to bridge the monocular depth estimation and camera-to-BEV transform. Thirdly, the semantic embeddings from the OSM data are utilized as auxiliary guidance for image-to-OSM feature matching. To validate the proposed OSMLoc, we collect a worldwide cross-area and cross-condition (CC) benchmark for extensive evaluation. Experiments on the MGL dataset, CC validation benchmark, and KITTI dataset have demonstrated the superiority of our method. Code, pre-trained models, CC validation benchmark, and additional results are available on: this https URL

Title: FedSub: Introducing class-aware Subnetworks Fusion to Enhance Personalized Federated Learning in Ubiquitous Systems

• Authors: Mattia Giovanni Campana, Franca Delmastro
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08699
• Pdf URL: https://arxiv.org/pdf/2411.08699
• Copy Paste: [[2411.08699]] FedSub: Introducing class-aware Subnetworks Fusion to Enhance Personalized Federated Learning in Ubiquitous Systems(https://arxiv.org/abs/2411.08699)
• Keywords: privacy, federate
• Abstract: Personalized Federated Learning is essential in AI-driven ubiquitous systems, supporting the distributed development of models able to adapt to diverse and evolving user behaviors while safeguarding privacy. Despite addressing heterogeneous user data distributions in collaborative model training, existing methods often face limitations balancing personalization and generalization, oversimplifying user similarities, or relying heavily on global models. In this paper, we propose FedSub, a novel federated approach designed to enhance personalization through the use of class-aware prototypes and model subnetworks. Prototypes serve as compact representations of user data, clustered on the server to identify similarities based on specific label patterns. Concurrently, subnetworks -- model components necessary to process each class -- are extracted locally and fused by the server according to these clusters, producing highly tailored model updates for each user. This fine-grained, class-specific aggregation of clients' models allows FedSub to capture the unique characteristics of individual user data patterns. The effectiveness of FedSub is validated in three real-world scenarios characterized by high data heterogeneity, derived from human activity recognition and mobile health applications. Experimental evaluations demonstrate FedSub's performance improvements with respect to the state-of-the-art and significant advancements in personalization for ubiquitous systems based on personal mobile and wearable devices.

Title: TRACE: Transformer-based Risk Assessment for Clinical Evaluation

• Authors: Dionysis Christopoulos, Sotiris Spanos, Valsamis Ntouskos, Konstantinos Karantzalos
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08701
• Pdf URL: https://arxiv.org/pdf/2411.08701
• Copy Paste: [[2411.08701]] TRACE: Transformer-based Risk Assessment for Clinical Evaluation(https://arxiv.org/abs/2411.08701)
• Keywords: explainability, transformer
• Abstract: We present TRACE (Transformer-based Risk Assessment for Clinical Evaluation), a novel method for clinical risk assessment based on clinical data, leveraging the self-attention mechanism for enhanced feature interaction and result interpretation. Our approach is able to handle different data modalities, including continuous, categorical and multiple-choice (checkbox) attributes. The proposed architecture features a shared representation of the clinical data obtained by integrating specialized embeddings of each data modality, enabling the detection of high-risk individuals using Transformer encoder layers. To assess the effectiveness of the proposed method, a strong baseline based on non-negative multi-layer perceptrons (MLPs) is introduced. The proposed method outperforms various baselines widely used in the domain of clinical risk assessment, while effectively handling missing values. In terms of explainability, our Transformer-based method offers easily interpretable results via attention weights, further enhancing the clinicians' decision-making process.

Title: MVKTrans: Multi-View Knowledge Transfer for Robust Multiomics Classification

• Authors: Shan Cong, Zhiling Sang, Hongwei Liu, Haoran Luo, Xin Wang, Hong Liang, Jie Hao, Xiaohui Yao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08703
• Pdf URL: https://arxiv.org/pdf/2411.08703
• Copy Paste: [[2411.08703]] MVKTrans: Multi-View Knowledge Transfer for Robust Multiomics Classification(https://arxiv.org/abs/2411.08703)
• Keywords: robust
• Abstract: The distinct characteristics of multiomics data, including complex interactions within and across biological layers and disease heterogeneity (e.g., heterogeneity in etiology and clinical symptoms), drive us to develop novel designs to address unique challenges in multiomics prediction. In this paper, we propose the multi-view knowledge transfer learning (MVKTrans) framework, which transfers intra- and inter-omics knowledge in an adaptive manner by reviewing data heterogeneity and suppressing bias transfer, thereby enhancing classification performance. Specifically, we design a graph contrastive module that is trained on unlabeled data to effectively learn and transfer the underlying intra-omics patterns to the supervised task. This unsupervised pretraining promotes learning general and unbiased representations for each modality, regardless of the downstream tasks. In light of the varying discriminative capacities of modalities across different diseases and/or samples, we introduce an adaptive and bi-directional cross-omics distillation module. This module automatically identifies richer modalities and facilitates dynamic knowledge transfer from more informative to less informative omics, thereby enabling a more robust and generalized integration. Extensive experiments on four real biomedical datasets demonstrate the superior performance and robustness of MVKTrans compared to the state-of-the-art. Code and data are available at this https URL.

Title: Are Triggers Needed for Document-Level Event Extraction?

• Authors: Shaden Shaar, Wayne Chen, Maitreyi Chatterjee, Barry Wang, Wenting Zhao, Claire Cardie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.08708
• Pdf URL: https://arxiv.org/pdf/2411.08708
• Copy Paste: [[2411.08708]] Are Triggers Needed for Document-Level Event Extraction?(https://arxiv.org/abs/2411.08708)
• Keywords: robust, extraction
• Abstract: Most existing work on event extraction has focused on sentence-level texts and presumes the identification of a trigger-span -- a word or phrase in the input that evokes the occurrence of an event of interest. Event arguments are then extracted with respect to the trigger. Indeed, triggers are treated as integral to, and trigger detection as an essential component of, event extraction. In this paper, we provide the first investigation of the role of triggers for the more difficult and much less studied task of document-level event extraction. We analyze their usefulness in multiple end-to-end and pipelined neural event extraction models for three document-level event extraction datasets, measuring performance using triggers of varying quality (human-annotated, LLM-generated, keyword-based, and random). Our research shows that trigger effectiveness varies based on the extraction task's characteristics and data quality, with basic, automatically-generated triggers serving as a viable alternative to human-annotated ones. Furthermore, providing detailed event descriptions to the extraction model helps maintain robust performance even when trigger quality degrades. Perhaps surprisingly, we also find that the mere existence of trigger input, even random ones, is important for prompt-based LLM approaches to the task.

Title: High-resolution optical and acoustic remote sensing datasets of the Puck Lagoon, Southern Baltic

• Authors: Łukasz Janowski, Dimitrios Skarlatos, Panagiotis Agrafiotis, Paweł Tysiąc, Andrzej Pydyn, Mateusz Popek, Anna M. Kotarba-Morley, Gottfried Mandlburger, Łukasz Gajewski, Mateusz Kołakowski, Alexandra Papadaki, Juliusz Gajewski
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.08712
• Pdf URL: https://arxiv.org/pdf/2411.08712
• Copy Paste: [[2411.08712]] High-resolution optical and acoustic remote sensing datasets of the Puck Lagoon, Southern Baltic(https://arxiv.org/abs/2411.08712)
• Keywords: protect
• Abstract: The very shallow marine basin of Puck Lagoon in the southern Baltic Sea, on the Northern coast of Poland, hosts valuable benthic habitats and cultural heritage sites. These include, among others, protected Zostera marina meadows, one of the Baltic's major medieval harbours, a ship graveyard, and likely other submerged features that are yet to be discovered. Prior to this project, no comprehensive high-resolution remote sensing data were available for this area. This article describes the first Digital Elevation Models (DEMs) derived from a combination of airborne bathymetric LiDAR, multibeam echosounder, airborne photogrammetry and satellite imagery. These datasets also include multibeam echosounder backscatter and LiDAR intensity, allowing determination of the character and properties of the seafloor. Combined, these datasets are a vital resource for assessing and understanding seafloor morphology, benthic habitats, cultural heritage, and submerged landscapes. Given the significance of Puck Lagoon's hydrographical, ecological, geological, and archaeological environs, the high-resolution bathymetry, acquired by our project, can provide the foundation for sustainable management and informed decision-making for this area of interest.

Title: Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs

• Authors: Kazuki Fujii, Taishi Nakamura, Rio Yokota
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08719
• Pdf URL: https://arxiv.org/pdf/2411.08719
• Copy Paste: [[2411.08719]] Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs(https://arxiv.org/abs/2411.08719)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have attracted significant attention due to their human-like language understanding and generation capabilities, as well as their applicability across various domains. These models, characterized by their massive scale and extensive training data, continue to push the boundaries of what is possible in natural language processing. The Llama 3 series, for instance, exemplifies this trend with its flagship model boasting 405 billion parameters trained on 15.6 trillion tokens. The immense computational demands associated with training such models have spurred ongoing research into optimizing the efficiency of the training process, particularly through the use of lower-precision formats. NVIDIA's H100 GPU, which introduces support for FP8 in addition to the more conventional FP16 and BF16 formats, has emerged as a focal point in this optimization effort. Preliminary studies suggest that FP8 could offer substantial reductions in training time without sacrificing model performance when compared to BF16, making it a promising candidate for large-scale model training. However, the broader implications of adopting FP8, particularly in terms of training stability and downstream task performance, have yet to be fully understood. In this study, we delve into the practical trade-offs involved in adopting FP8 over BF16 for training LLMs.

Title: QCG-Rerank: Chunks Graph Rerank with Query Expansion in Retrieval-Augmented LLMs for Tourism Domain

• Authors: Qikai Wei, Mingzhi Yang, Chunlong Han, Jingfu Wei, Minghao Zhang, Feifei Shi, Huansheng Ning
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08724
• Pdf URL: https://arxiv.org/pdf/2411.08724
• Copy Paste: [[2411.08724]] QCG-Rerank: Chunks Graph Rerank with Query Expansion in Retrieval-Augmented LLMs for Tourism Domain(https://arxiv.org/abs/2411.08724)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) mitigates the issue of hallucination in Large Language Models (LLMs) by integrating information retrieval techniques. However, in the tourism domain, since the query is usually brief and the content in the database is diverse, existing RAG may contain a significant amount of irrelevant or contradictory information contents after retrieval. To address this challenge, we propose the QCG-Rerank model. This model first performs an initial retrieval to obtain candidate chunks and then enhances semantics by extracting critical information to expand the original query. Next, we utilize the expanded query and candidate chunks to calculate similarity scores as the initial transition probability and construct the chunks graph. Subsequently, We iteratively compute the transition probabilities based on an initial estimate until convergence. The chunks with the highest score are selected and input into the LLMs to generate responses. We evaluate the model on Cultour, IIRC, StrategyQA, HotpotQA, SQuAD, and MuSiQue datasets. The experimental results demonstrate the effectiveness and superiority of the QCG-Rerank method.

Title: Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models

• Authors: Somanshu Singla, Zhen Wang, Tianyang Liu, Abdullah Ashfaq, Zhiting Hu, Eric P. Xing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.08733
• Pdf URL: https://arxiv.org/pdf/2411.08733
• Copy Paste: [[2411.08733]] Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models(https://arxiv.org/abs/2411.08733)
• Keywords: large language model
• Abstract: Aligning Large Language Models (LLMs) traditionally relies on costly training and human preference annotations. Self-alignment seeks to reduce these expenses by enabling models to align themselves. To further lower costs and achieve alignment without any expensive tuning or annotations, we introduce a new tuning-free approach for self-alignment, Dynamic Rewarding with Prompt Optimization (\ours). Our approach leverages a search-based optimization framework that allows LLMs to iteratively self-improve and craft the optimal alignment instructions, all without additional training or human intervention. The core of \ours is a dynamic rewarding mechanism, which identifies and rectifies model-specific alignment weaknesses, allowing LLMs to adapt efficiently to diverse alignment challenges. Empirical evaluations on eight recent LLMs, both open- and closed-sourced, demonstrate that \ours significantly enhances alignment performance, with base models outperforming their SFT/RLHF-tuned counterparts. Moreover, the prompts automatically optimized by \ours surpass those curated by human experts, further validating the effectiveness of our approach. Our findings highlight the great potential of current LLMs to achieve adaptive self-alignment through inference-time optimization, complementing tuning-based alignment methods.

Title: A Comparative Study of Discrete Speech Tokens for Semantic-Related Tasks with Large Language Models

• Authors: Dingdong Wang, Mingyu Cui, Dongchao Yang, Xueyuan Chen, Helen Meng
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.08742
• Pdf URL: https://arxiv.org/pdf/2411.08742
• Copy Paste: [[2411.08742]] A Comparative Study of Discrete Speech Tokens for Semantic-Related Tasks with Large Language Models(https://arxiv.org/abs/2411.08742)
• Keywords: fair, large language model
• Abstract: With the rise of Speech Large Language Models (Speech LLMs), there has been growing interest in discrete speech tokens for their ability to integrate with text-based tokens seamlessly. Compared to most studies that focus on continuous speech features, although discrete-token based LLMs have shown promising results on certain tasks, the performance gap between these two paradigms is rarely explored. In this paper, we present a fair and thorough comparison between discrete and continuous features across a variety of semantic-related tasks using a light-weight LLM (Qwen1.5-0.5B). Our findings reveal that continuous features generally outperform discrete tokens, particularly in tasks requiring fine-grained semantic understanding. Moreover, this study goes beyond surface-level comparison by identifying key factors behind the under-performance of discrete tokens, such as limited token granularity and inefficient information retention. To enhance the performance of discrete tokens, we explore potential aspects based on our analysis. We hope our results can offer new insights into the opportunities for advancing discrete speech tokens in Speech LLMs.

Title: Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers

• Authors: Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, Robert West
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08745
• Pdf URL: https://arxiv.org/pdf/2411.08745
• Copy Paste: [[2411.08745]] Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers(https://arxiv.org/abs/2411.08745)
• Keywords: transformer, large language model
• Abstract: A central question in multilingual language modeling is whether large language models (LLMs) develop a universal concept representation, disentangled from specific languages. In this paper, we address this question by analyzing latent representations (latents) during a word translation task in transformer-based LLMs. We strategically extract latents from a source translation prompt and insert them into the forward pass on a target translation prompt. By doing so, we find that the output language is encoded in the latent at an earlier layer than the concept to be translated. Building on this insight, we conduct two key experiments. First, we demonstrate that we can change the concept without changing the language and vice versa through activation patching alone. Second, we show that patching with the mean over latents across different languages does not impair and instead improves the models' performance in translating the concept. Our results provide evidence for the existence of language-agnostic concept representations within the investigated models.

Title: Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation

• Authors: Yangyang Li, Xuanting Hao, Ronghua Shang, Licheng Jiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08756
• Pdf URL: https://arxiv.org/pdf/2411.08756
• Copy Paste: [[2411.08756]] Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation(https://arxiv.org/abs/2411.08756)
• Keywords: generative, segmentation
• Abstract: In view of the fact that semi- and self-supervised learning share a fundamental principle, effectively modeling knowledge from unlabeled data, various semi-supervised semantic segmentation methods have integrated representative self-supervised learning paradigms for further regularization. However, the potential of the state-of-the-art generative self-supervised paradigm, masked image modeling, has been scarcely studied. This paradigm learns the knowledge through establishing connections between the masked and visible parts of masked image, during the pixel reconstruction process. By inheriting and extending this insight, we successfully leverage masked image modeling to boost semi-supervised semantic segmentation. Specifically, we introduce a novel class-wise masked image modeling that independently reconstructs different image regions according to their respective classes. In this way, the mask-induced connections are established within each class, mitigating the semantic confusion that arises from plainly reconstructing images in basic masked image modeling. To strengthen these intra-class connections, we further develop a feature aggregation strategy that minimizes the distances between features corresponding to the masked and visible parts within the same class. Additionally, in semantic space, we explore the application of masked image modeling to enhance regularization. Extensive experiments conducted on well-known benchmarks demonstrate that our approach achieves state-of-the-art performance. The code will be available at this https URL.

Title: Flow reconstruction in time-varying geometries using graph neural networks

• Authors: Bogdan A. Danciu, Vito A. Pagone, Benjamin Böhm, Marius Schmidt, Christos E. Frouzakis
• Subjects: cs.LG, cs.AI, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2411.08764
• Pdf URL: https://arxiv.org/pdf/2411.08764
• Copy Paste: [[2411.08764]] Flow reconstruction in time-varying geometries using graph neural networks(https://arxiv.org/abs/2411.08764)
• Keywords: robust
• Abstract: The paper presents a Graph Attention Convolutional Network (GACN) for flow reconstruction from very sparse data in time-varying geometries. The model incorporates a feature propagation algorithm as a preprocessing step to handle extremely sparse inputs, leveraging information from neighboring nodes to initialize missing features. In addition, a binary indicator is introduced as a validity mask to distinguish between the original and propagated data points, enabling more effective learning from sparse inputs. Trained on a unique data set of Direct Numerical Simulations (DNS) of a motored engine at a technically relevant operating condition, the GACN shows robust performance across different resolutions and domain sizes and can effectively handle unstructured data and variable input sizes. The model is tested on previously unseen DNS data as well as on an experimental data set from Particle Image Velocimetry (PIV) measurements that were not considered during training. A comparative analysis shows that the GACN consistently outperforms both a conventional Convolutional Neural Network (CNN) and cubic interpolation methods on the DNS and PIV test sets by achieving lower reconstruction errors and better capturing fine-scale turbulent structures. In particular, the GACN effectively reconstructs flow fields from domains up to 14 times larger than those observed during training, with the performance advantage increasing for larger domains.

Title: Sharingan: Extract User Action Sequence from Desktop Recordings

• Authors: Yanting Chen, Yi Ren, Xiaoting Qin, Jue Zhang, Kehong Yuan, Lu Han, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08768
• Pdf URL: https://arxiv.org/pdf/2411.08768
• Copy Paste: [[2411.08768]] Sharingan: Extract User Action Sequence from Desktop Recordings(https://arxiv.org/abs/2411.08768)
• Keywords: extraction
• Abstract: Video recordings of user activities, particularly desktop recordings, offer a rich source of data for understanding user behaviors and automating processes. However, despite advancements in Vision-Language Models (VLMs) and their increasing use in video analysis, extracting user actions from desktop recordings remains an underexplored area. This paper addresses this gap by proposing two novel VLM-based methods for user action extraction: the Direct Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and the Differential Frame-Based Approach (DiffF), which incorporates explicit frame differences detected via computer vision techniques. We evaluate these methods using a basic self-curated dataset and an advanced benchmark adapted from prior work. Our results show that the DF approach achieves an accuracy of 70% to 80% in identifying user actions, with the extracted action sequences being re-playable though Robotic Process Automation. We find that while VLMs show potential, incorporating explicit UI changes can degrade performance, making the DF approach more reliable. This work represents the first application of VLMs for extracting user action sequences from desktop recordings, contributing new methods, benchmarks, and insights for future research.

Title: SoK: Towards a Common Understanding of Cryptographic Agility

• Authors: Christian Näther, Daniel Herzinger, Jan-Philipp Steghöfer, Stefan-Lukas Gazdag, Eduard Hirsch, Daniel Loebenberger
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.08781
• Pdf URL: https://arxiv.org/pdf/2411.08781
• Copy Paste: [[2411.08781]] SoK: Towards a Common Understanding of Cryptographic Agility(https://arxiv.org/abs/2411.08781)
• Keywords: security
• Abstract: Cryptographic agility is gaining attention due to its crucial role in maintaining cryptographic security in a rapidly evolving technological landscape. However, despite its increasing importance, the term cryptographic agility remains vaguely defined and there is no clear consensus on its exact meaning. This lack of clarity poses a challenge since the need for agility becomes more urgent as new cryptographic vulnerabilities and advanced computing threats emerge, emphasizing the need for a systematic approach to clarify and refine the notion on cryptographic agility. In this paper, we systematize the concept of cryptographic agility by providing three research contributions. First, we review current definitions across academic and gray literature, identifying six distinct categories to differentiate every aspect within the definitions. Second, we synthesize these insights to establish a comprehensive, canonical definition of cryptographic agility. Third, we explore the relationship between cryptographic agility and the related concepts cryptographic versatility and interoperability. In our discussion, we examine the relevance of cryptographic agility, highlight its trade-offs with complexity, assess its individual applicability, and illustrate its various contexts by offering an additional application-specific definition. Our work provides a new perspective on cryptographic agility and related concepts, based on systematical research to clarify and enhance its future use.

Title: Zero-shot Cross-lingual Transfer Learning with Multiple Source and Target Languages for Information Extraction: Language Selection and Adversarial Training

• Authors: Nghia Trung Ngo, Thien Huu Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08785
• Pdf URL: https://arxiv.org/pdf/2411.08785
• Copy Paste: [[2411.08785]] Zero-shot Cross-lingual Transfer Learning with Multiple Source and Target Languages for Information Extraction: Language Selection and Adversarial Training(https://arxiv.org/abs/2411.08785)
• Keywords: robust, extraction
• Abstract: The majority of previous researches addressing multi-lingual IE are limited to zero-shot cross-lingual single-transfer (one-to-one) setting, with high-resource languages predominantly as source training data. As a result, these works provide little understanding and benefit for the realistic goal of developing a multi-lingual IE system that can generalize to as many languages as possible. Our study aims to fill this gap by providing a detailed analysis on Cross-Lingual Multi-Transferability (many-to-many transfer learning), for the recent IE corpora that cover a diverse set of languages. Specifically, we first determine the correlation between single-transfer performance and a wide range of linguistic-based distances. From the obtained insights, a combined language distance metric can be developed that is not only highly correlated but also robust across different tasks and model scales. Next, we investigate the more general zero-shot multi-lingual transfer settings where multiple languages are involved in the training and evaluation processes. Language clustering based on the newly defined distance can provide directions for achieving the optimal cost-performance trade-off in data (languages) selection problem. Finally, a relational-transfer setting is proposed to further incorporate multi-lingual unlabeled data based on adversarial training using the relation induced from the above linguistic distance.

Title: Can sparse autoencoders be used to decompose and interpret steering vectors?

• Authors: Harry Mayne, Yushi Yang, Adam Mahdi
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.08790
• Pdf URL: https://arxiv.org/pdf/2411.08790
• Copy Paste: [[2411.08790]] Can sparse autoencoders be used to decompose and interpret steering vectors?(https://arxiv.org/abs/2411.08790)
• Keywords: large language model
• Abstract: Steering vectors are a promising approach to control the behaviour of large language models. However, their underlying mechanisms remain poorly understood. While sparse autoencoders (SAEs) may offer a potential method to interpret steering vectors, recent findings show that SAE-reconstructed vectors often lack the steering properties of the original vectors. This paper investigates why directly applying SAEs to steering vectors yields misleading decompositions, identifying two reasons: (1) steering vectors fall outside the input distribution for which SAEs are designed, and (2) steering vectors can have meaningful negative projections in feature directions, which SAEs are not designed to accommodate. These limitations hinder the direct use of SAEs for interpreting steering vectors.

Title: Locally Private Sampling with Public Data

• Authors: Behnoosh Zamanlooy, Mario Diaz, Shahab Asoodeh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08791
• Pdf URL: https://arxiv.org/pdf/2411.08791
• Copy Paste: [[2411.08791]] Locally Private Sampling with Public Data(https://arxiv.org/abs/2411.08791)
• Keywords: privacy, protect
• Abstract: Local differential privacy (LDP) is increasingly employed in privacy-preserving machine learning to protect user data before sharing it with an untrusted aggregator. Most LDP methods assume that users possess only a single data record, which is a significant limitation since users often gather extensive datasets (e.g., images, text, time-series data) and frequently have access to public datasets. To address this limitation, we propose a locally private sampling framework that leverages both the private and public datasets of each user. Specifically, we assume each user has two distributions: $p$ and $q$ that represent their private dataset and the public dataset, respectively. The objective is to design a mechanism that generates a private sample approximating $p$ while simultaneously preserving $q$. We frame this objective as a minimax optimization problem using $f$-divergence as the utility measure. We fully characterize the minimax optimal mechanisms for general $f$-divergences provided that $p$ and $q$ are discrete distributions. Remarkably, we demonstrate that this optimal mechanism is universal across all $f$-divergences. Experiments validate the effectiveness of our minimax optimal sampler compared to the state-of-the-art locally private sampler.

Title: Multimodal Instruction Tuning with Hybrid State Space Models

• Authors: Jianing Zhou, Han Li, Shuai Zhang, Ning Xie, Ruijie Wang, Xiaohan Nie, Sheng Liu, Lingyun Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08840
• Pdf URL: https://arxiv.org/pdf/2411.08840
• Copy Paste: [[2411.08840]] Multimodal Instruction Tuning with Hybrid State Space Models(https://arxiv.org/abs/2411.08840)
• Keywords: transformer, large language model
• Abstract: Handling lengthy context is crucial for enhancing the recognition and understanding capabilities of multimodal large language models (MLLMs) in applications such as processing high-resolution images or high frame rate videos. The rise in image resolution and frame rate substantially increases computational demands due to the increased number of input tokens. This challenge is further exacerbated by the quadratic complexity with respect to sequence length of the self-attention mechanism. Most prior works either pre-train models with long contexts, overlooking the efficiency problem, or attempt to reduce the context length via downsampling (e.g., identify the key image patches or frames) to decrease the context length, which may result in information loss. To circumvent this issue while keeping the remarkable effectiveness of MLLMs, we propose a novel approach using a hybrid transformer-MAMBA model to efficiently handle long contexts in multimodal applications. Our multimodal model can effectively process long context input exceeding 100k tokens, outperforming existing models across various benchmarks. Remarkably, our model enhances inference efficiency for high-resolution images and high-frame-rate videos by about 4 times compared to current models, with efficiency gains increasing as image resolution or video frames rise. Furthermore, our model is the first to be trained on low-resolution images or low-frame-rate videos while being capable of inference on high-resolution images and high-frame-rate videos, offering flexibility for inference in diverse scenarios.

Title: LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMs

• Authors: Piyush Jha, Arnav Arora, Vijay Ganesh
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.08862
• Pdf URL: https://arxiv.org/pdf/2411.08862
• Copy Paste: [[2411.08862]] LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMs(https://arxiv.org/abs/2411.08862)
• Keywords: attack, robust, large language model
• Abstract: We introduce LLMStinger, a novel approach that leverages Large Language Models (LLMs) to automatically generate adversarial suffixes for jailbreak attacks. Unlike traditional methods, which require complex prompt engineering or white-box access, LLMStinger uses a reinforcement learning (RL) loop to fine-tune an attacker LLM, generating new suffixes based on existing attacks for harmful questions from the HarmBench benchmark. Our method significantly outperforms existing red-teaming approaches (we compared against 15 of the latest methods), achieving a +57.2% improvement in Attack Success Rate (ASR) on LLaMA2-7B-chat and a +50.3% ASR increase on Claude 2, both models known for their extensive safety measures. Additionally, we achieved a 94.97% ASR on GPT-3.5 and 99.4% on Gemma-2B-it, demonstrating the robustness and adaptability of LLMStinger across open and closed-source models.

Title: The Limited Impact of Medical Adaptation of Large Language and Vision-Language Models

• Authors: Daniel P. Jeong, Pranav Mani, Saurabh Garg, Zachary C. Lipton, Michael Oberst
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08870
• Pdf URL: https://arxiv.org/pdf/2411.08870
• Copy Paste: [[2411.08870]] The Limited Impact of Medical Adaptation of Large Language and Vision-Language Models(https://arxiv.org/abs/2411.08870)
• Keywords: large language model
• Abstract: Several recent works seek to develop foundation models specifically for medical applications, adapting general-purpose large language models (LLMs) and vision-language models (VLMs) via continued pretraining on publicly available biomedical corpora. These works typically claim that such domain-adaptive pretraining (DAPT) improves performance on downstream medical tasks, such as answering medical licensing exam questions. In this paper, we compare ten public "medical" LLMs and two VLMs against their corresponding base models, arriving at a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models in the zero-/few-shot prompting and supervised fine-tuning regimes for medical question-answering (QA). For instance, across all tasks and model pairs we consider in the 3-shot setting, medical LLMs only outperform their base models in 22.7% of cases, reach a (statistical) tie in 36.8% of cases, and are significantly worse than their base models in the remaining 40.5% of cases. Our conclusions are based on (i) comparing each medical model head-to-head, directly against the corresponding base model; (ii) optimizing the prompts for each model separately in zero-/few-shot prompting; and (iii) accounting for statistical uncertainty in comparisons. While these basic practices are not consistently adopted in the literature, our ablations show that they substantially impact conclusions. Meanwhile, we find that after fine-tuning on specific QA tasks, medical LLMs can show performance improvements, but the benefits do not carry over to tasks based on clinical notes. Our findings suggest that state-of-the-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities, and offer recommendations to strengthen the conclusions of future studies.

Title: 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization

• Authors: Mijeong Kim, Jongwoo Lim, Bohyung Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08879
• Pdf URL: https://arxiv.org/pdf/2411.08879
• Copy Paste: [[2411.08879]] 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization(https://arxiv.org/abs/2411.08879)
• Keywords: diffusion
• Abstract: Novel view synthesis of dynamic scenes is becoming important in various applications, including augmented and virtual reality. We propose a novel 4D Gaussian Splatting (4DGS) algorithm for dynamic scenes from casually recorded monocular videos. To overcome the overfitting problem of existing work for these real-world videos, we introduce an uncertainty-aware regularization that identifies uncertain regions with few observations and selectively imposes additional priors based on diffusion models and depth smoothness on such regions. This approach improves both the performance of novel view synthesis and the quality of training image reconstruction. We also identify the initialization problem of 4DGS in fast-moving dynamic regions, where the Structure from Motion (SfM) algorithm fails to provide reliable 3D landmarks. To initialize Gaussian primitives in such regions, we present a dynamic region densification method using the estimated depth maps and scene flow. Our experiments show that the proposed method improves the performance of 4DGS reconstruction from a video captured by a handheld monocular camera and also exhibits promising results in few-shot static scene reconstruction.