2025-01-06

Title: Fundamental Risks in the Current Deployment of General-Purpose AI Models: What Have We (Not) Learnt From Cybersecurity?

• Authors: Mario Fritz
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01435
• Pdf URL: https://arxiv.org/pdf/2501.01435
• Copy Paste: [[2501.01435]] Fundamental Risks in the Current Deployment of General-Purpose AI Models: What Have We (Not) Learnt From Cybersecurity?(https://arxiv.org/abs/2501.01435)
• Keywords: security, large language model
• Abstract: General Purpose AI - such as Large Language Models (LLMs) - have seen rapid deployment in a wide range of use cases. Most surprisingly, they have have made their way from plain language models, to chat-bots, all the way to an almost ``operating system''-like status that can control decisions and logic of an application. Tool-use, Microsoft co-pilot/office integration, and OpenAIs Altera are just a few examples of increased autonomy, data access, and execution capabilities. These methods come with a range of cybersecurity challenges. We highlight some of the work we have done in terms of evaluation as well as outline future opportunities and challenges.

Title: LS-GAN: Human Motion Synthesis with Latent-space GANs

• Authors: Avinash Amballa, Gayathri Akkinapalli, Vinitra Muralikrishnan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01449
• Pdf URL: https://arxiv.org/pdf/2501.01449
• Copy Paste: [[2501.01449]] LS-GAN: Human Motion Synthesis with Latent-space GANs(https://arxiv.org/abs/2501.01449)
• Keywords: diffusion, generative
• Abstract: Human motion synthesis conditioned on textual input has gained significant attention in recent years due to its potential applications in various domains such as gaming, film production, and virtual reality. Conditioned Motion synthesis takes a text input and outputs a 3D motion corresponding to the text. While previous works have explored motion synthesis using raw motion data and latent space representations with diffusion models, these approaches often suffer from high training and inference times. In this paper, we introduce a novel framework that utilizes Generative Adversarial Networks (GANs) in the latent space to enable faster training and inference while achieving results comparable to those of the state-of-the-art diffusion methods. We perform experiments on the HumanML3D, HumanAct12 benchmarks and demonstrate that a remarkably simple GAN in the latent space achieves a FID of 0.482 with more than 91% in FLOPs reduction compared to latent diffusion model. Our work opens up new possibilities for efficient and high-quality motion synthesis using latent space GANs.

Title: Geometry Matters: Benchmarking Scientific ML Approaches for Flow Prediction around Complex Geometries

• Authors: Ali Rabeh, Ethan Herron, Aditya Balu, Soumik Sarkar, Chinmay Hegde, Adarsh Krishnamurthy, Baskar Ganapathysubramanian
• Subjects: cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2501.01453
• Pdf URL: https://arxiv.org/pdf/2501.01453
• Copy Paste: [[2501.01453]] Geometry Matters: Benchmarking Scientific ML Approaches for Flow Prediction around Complex Geometries(https://arxiv.org/abs/2501.01453)
• Keywords: robust, transformer
• Abstract: Rapid yet accurate simulations of fluid dynamics around complex geometries is critical in a variety of engineering and scientific applications, including aerodynamics and biomedical flows. However, while scientific machine learning (SciML) has shown promise, most studies are constrained to simple geometries, leaving complex, real-world scenarios underexplored. This study addresses this gap by benchmarking diverse SciML models, including neural operators and vision transformer-based foundation models, for fluid flow prediction over intricate geometries. Using a high-fidelity dataset of steady-state flows across various geometries, we evaluate the impact of geometric representations -- Signed Distance Fields (SDF) and binary masks -- on model accuracy, scalability, and generalization. Central to this effort is the introduction of a novel, unified scoring framework that integrates metrics for global accuracy, boundary layer fidelity, and physical consistency to enable a robust, comparative evaluation of model performance. Our findings demonstrate that foundation models significantly outperform neural operators, particularly in data-limited scenarios, and that SDF representations yield superior results with sufficient training data. Despite these advancements, all models struggle with out-of-distribution generalization, highlighting a critical challenge for future SciML applications. By advancing both evaluation methodologies and modeling capabilities, this work paves the way for robust and scalable ML solutions for fluid dynamics across complex geometries.

Title: Reinforcing Thinking through Reasoning-Enhanced Reward Models

• Authors: Diji Yang, Linda Zeng, Kezhen Chen, Yi Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01457
• Pdf URL: https://arxiv.org/pdf/2501.01457
• Copy Paste: [[2501.01457]] Reinforcing Thinking through Reasoning-Enhanced Reward Models(https://arxiv.org/abs/2501.01457)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit great potential in complex multi-step reasoning through inference-time thinking but still struggle with deciding when to stop thinking due to limited self-awareness about their knowledge boundaries. While human preference alignment has shown extraordinary opportunities, expensive labeling challenges adherence to scaling law. Language model self-critique, as an alternative to using human-labeled reasoning data, is questioned with its inherited biases. This work addresses these challenges by distilling the LLM's own reasoning processes into synthetic behavioral data, eliminating the need for manual labeling of intermediate steps. Building on this concept, we propose Distillation-Reinforcement-Reasoning (DRR), a three-step framework that leverages the LLM's inherent behaviors as external feedback by first generating behavioral data using the Reasoner (LLM) to reflect its reasoning capabilities, then training a lightweight discriminative reward model (DM) on behavioral data, and finally deploying the DM at inference time to assist the Reasoner's decision-making. Experiments on multiple benchmarks show that the DRR framework outperforms self-critique approaches without relying on additional complex data annotation. Benefiting from lightweight design, ease of replication, and adaptability, DRR is applicable to a wide range of LLM-centric tasks.

Title: Goal Recognition using Actor-Critic Optimization

• Authors: Ben Nageris, Felipe Meneguzzi, Reuth Mirsky
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.01463
• Pdf URL: https://arxiv.org/pdf/2501.01463
• Copy Paste: [[2501.01463]] Goal Recognition using Actor-Critic Optimization(https://arxiv.org/abs/2501.01463)
• Keywords: robust
• Abstract: Goal Recognition aims to infer an agent's goal from a sequence of observations. Existing approaches often rely on manually engineered domains and discrete representations. Deep Recognition using Actor-Critic Optimization (DRACO) is a novel approach based on deep reinforcement learning that overcomes these limitations by providing two key contributions. First, it is the first goal recognition algorithm that learns a set of policy networks from unstructured data and uses them for inference. Second, DRACO introduces new metrics for assessing goal hypotheses through continuous policy representations. DRACO achieves state-of-the-art performance for goal recognition in discrete settings while not using the structured inputs used by existing approaches. Moreover, it outperforms these approaches in more challenging, continuous settings at substantially reduced costs in both computing and memory. Together, these results showcase the robustness of the new algorithm, bridging traditional goal recognition and deep reinforcement learning.

Title: Balance-aware Sequence Sampling Makes Multi-modal Learning Better

• Authors: Zhi-Hao Guan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01470
• Pdf URL: https://arxiv.org/pdf/2501.01470
• Copy Paste: [[2501.01470]] Balance-aware Sequence Sampling Makes Multi-modal Learning Better(https://arxiv.org/abs/2501.01470)
• Keywords: robust
• Abstract: To address the modality imbalance caused by data heterogeneity, existing multi-modal learning (MML) approaches primarily focus on balancing this difference from the perspective of optimization objectives. However, almost all existing methods ignore the impact of sample sequences, i.e., an inappropriate training order tends to trigger learning bias in the model, further exacerbating modality imbalance. In this paper, we propose Balance-aware Sequence Sampling (BSS) to enhance the robustness of MML. Specifically, we first define a multi-perspective measurer to evaluate the balance degree of each sample. Via the evaluation, we employ a heuristic scheduler based on curriculum learning (CL) that incrementally provides training subsets, progressing from balanced to imbalanced samples to rebalance MML. Moreover, considering that sample balance may evolve as the model capability increases, we propose a learning-based probabilistic sampling method to dynamically update the training sequences at the epoch level, further improving MML performance. Extensive experiments on widely used datasets demonstrate the superiority of our method compared with state-of-the-art (SOTA) MML approaches.

Title: Unraveling Indirect In-Context Learning Using Influence Functions

• Authors: Hadi Askari, Shivanshu Gupta, Terry Tong, Fei Wang, Anshuman Chhabra, Muhao Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01473
• Pdf URL: https://arxiv.org/pdf/2501.01473
• Copy Paste: [[2501.01473]] Unraveling Indirect In-Context Learning Using Influence Functions(https://arxiv.org/abs/2501.01473)
• Keywords: robust
• Abstract: This work introduces a novel paradigm for generalized In-Context Learning (ICL), termed Indirect In-Context Learning. In Indirect ICL, we explore demonstration selection strategies tailored for two distinct real-world scenarios: Mixture of Tasks and Noisy Demonstrations. We systematically evaluate the effectiveness of Influence Functions (IFs) as a selection tool for these settings, highlighting the potential for IFs to better capture the informativeness of examples within the demonstration pool. For the Mixture of Tasks setting, demonstrations are drawn from 28 diverse tasks, including MMLU, BigBench, StrategyQA, and CommonsenseQA. We demonstrate that combining BertScore-Recall (BSR) with an IF surrogate model can significantly improve performance, leading to average absolute accuracy gains of 0.37\% and 1.45\% for 3-shot and 5-shot setups when compared to traditional ICL metrics. In the Noisy Demonstrations setting, we examine scenarios where demonstrations might be mislabeled. Our experiments show that reweighting traditional ICL selectors (BSR and Cosine Similarity) with IF-based selectors boosts accuracy by an average of 2.90\% for Cosine Similarity and 2.94\% for BSR on noisy GLUE benchmarks. In sum, we propose a robust framework for demonstration selection that generalizes beyond traditional ICL, offering valuable insights into the role of IFs for Indirect ICL.

Title: Explainable Brain Age Gap Prediction in Neurodegenerative Conditions using coVariance Neural Networks

• Authors: Saurabh Sihag, Gonzalo Mateos, Alejandro Ribeiro
• Subjects: cs.LG, eess.SP, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2501.01510
• Pdf URL: https://arxiv.org/pdf/2501.01510
• Copy Paste: [[2501.01510]] Explainable Brain Age Gap Prediction in Neurodegenerative Conditions using coVariance Neural Networks(https://arxiv.org/abs/2501.01510)
• Keywords: explainability, generative
• Abstract: Brain age is the estimate of biological age derived from neuroimaging datasets using machine learning algorithms. Increasing \textit{brain age gap} characterized by an elevated brain age relative to the chronological age can reflect increased vulnerability to neurodegeneration and cognitive decline. Hence, brain age gap is a promising biomarker for monitoring brain health. However, black-box machine learning approaches to brain age gap prediction have limited practical utility. Recent studies on coVariance neural networks (VNN) have proposed a relatively transparent deep learning pipeline for neuroimaging data analyses, which possesses two key features: (i) inherent \textit{anatomically interpretablity} of derived biomarkers; and (ii) a methodologically interpretable perspective based on \textit{linkage with eigenvectors of anatomic covariance matrix}. In this paper, we apply the VNN-based approach to study brain age gap using cortical thickness features for various prevalent neurodegenerative conditions. Our results reveal distinct anatomic patterns for brain age gap in Alzheimer's disease, frontotemporal dementia, and atypical Parkinsonian disorders. Furthermore, we demonstrate that the distinct anatomic patterns of brain age gap are linked with the differences in how VNN leverages the eigenspectrum of the anatomic covariance matrix, thus lending explainability to the reported results.

Title: Improving Robustness Estimates in Natural Language Explainable AI though Synonymity Weighted Similarity Measures

• Authors: Christopher Burger
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01516
• Pdf URL: https://arxiv.org/pdf/2501.01516
• Copy Paste: [[2501.01516]] Improving Robustness Estimates in Natural Language Explainable AI though Synonymity Weighted Similarity Measures(https://arxiv.org/abs/2501.01516)
• Keywords: robust
• Abstract: Explainable AI (XAI) has seen a surge in recent interest with the proliferation of powerful but intractable black-box models. Moreover, XAI has come under fire for techniques that may not offer reliable explanations. As many of the methods in XAI are themselves models, adversarial examples have been prominent in the literature surrounding the effectiveness of XAI, with the objective of these examples being to alter the explanation while maintaining the output of the original model. For explanations in natural language, it is natural to use measures found in the domain of information retrieval for use with ranked lists to guide the adversarial XAI process. We show that the standard implementation of these measures are poorly suited for the comparison of explanations in adversarial XAI and amend them by using information that is discarded, the synonymity of perturbed words. This synonymity weighting produces more accurate estimates of the actual weakness of XAI methods to adversarial examples.

Title: Securing Wi-Fi 6 Connection Establishment Against Relay and Spoofing Threats

• Authors: Naureen Hoque, Hanif Rahbari
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.01517
• Pdf URL: https://arxiv.org/pdf/2501.01517
• Copy Paste: [[2501.01517]] Securing Wi-Fi 6 Connection Establishment Against Relay and Spoofing Threats(https://arxiv.org/abs/2501.01517)
• Keywords: security, defense, attack
• Abstract: Wireless local area networks remain vulnerable to attacks initiated during the connection establishment (CE) phase. Current Wi-Fi security protocols fail to fully mitigate attacks like man-in-the-middle, preamble spoofing, and relaying. To fortify the CE phase, in this paper we design a backward-compatible scheme using a digital signature interwoven into the preambles at the physical (PHY) layer with time constraints to effectively counter those attacks. This approach slices a MAC-layer signature and embeds the slices within CE frame preambles without extending frame size, allowing one or multiple stations to concurrently verify their respective APs' transmissions. The concurrent CEs are supported by enabling the stations to analyze the consistent patterns of PHY-layer headers and identify whether the received frames are the anticipated ones from the expected APs, achieving 100% accuracy without needing to examine their MAC-layer headers. Additionally, we design and implement a fast relay attack to challenge our proposed defense and determine its effectiveness. We extend existing open-source tools to support IEEE 802.11ax to evaluate the effectiveness and practicality of our proposed scheme in a testbed consisting of USRPs, commercial APs, and Wi-Fi devices, and we show that our relay attack detection achieves 96-100% true positive rates. Finally, end-to-end formal security analyses confirm the security and correctness of the proposed solution.

Title: SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers

• Authors: Bhavna Gopal, Huanrui Yang, Mark Horton, Yiran Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01529
• Pdf URL: https://arxiv.org/pdf/2501.01529
• Copy Paste: [[2501.01529]] SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers(https://arxiv.org/abs/2501.01529)
• Keywords: robust, transformer
• Abstract: Vision transformers (ViTs) have become essential backbones in advanced computer vision applications and multi-modal foundation models. Despite their strengths, ViTs remain vulnerable to adversarial perturbations, comparable to or even exceeding the vulnerability of convolutional neural networks (CNNs). Furthermore, the large parameter count and complex architecture of ViTs make them particularly prone to adversarial overfitting, often compromising both clean and adversarial accuracy. This paper mitigates adversarial overfitting in ViTs through a novel, layer-selective fine-tuning approach: SAFER. Instead of optimizing the entire model, we identify and selectively fine-tune a small subset of layers most susceptible to overfitting, applying sharpness-aware minimization to these layers while freezing the rest of the model. Our method consistently enhances both clean and adversarial accuracy over baseline approaches. Typical improvements are around 5%, with some cases achieving gains as high as 20% across various ViT architectures and datasets.

Title: BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery

• Authors: Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Louise Li, Aditi Bhaskar, Mohammed Zaman, Noah D. Goodman
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01540
• Pdf URL: https://arxiv.org/pdf/2501.01540
• Copy Paste: [[2501.01540]] BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery(https://arxiv.org/abs/2501.01540)
• Keywords: generative
• Abstract: Understanding the world and explaining it with scientific theories is a central aspiration of artificial intelligence research. Proposing theories, designing experiments to test them, and then revising them based on data are fundamental to scientific discovery. Despite the significant promise of LLM-based scientific agents, no benchmarks systematically test LLM's ability to propose scientific models, collect experimental data, and revise them in light of new data. We introduce BoxingGym, a benchmark with 10 environments for systematically evaluating both experimental design (e.g. collecting data to test a scientific theory) and model discovery (e.g. proposing and revising scientific theories). To enable tractable and quantitative evaluation, we implement each environment as a generative probabilistic model with which a scientific agent can run interactive experiments. These probabilistic models are drawn from various real-world scientific domains ranging from psychology to ecology. To quantitatively evaluate a scientific agent's ability to collect informative experimental data, we compute the expected information gain (EIG), an information-theoretic quantity which measures how much an experiment reduces uncertainty about the parameters of a generative model. A good scientific theory is a concise and predictive explanation. Therefore, to quantitatively evaluate model discovery, we ask a scientific agent to explain their model and then assess whether this explanation enables another scientific agent to make reliable predictions about this environment. In addition to this explanation-based evaluation, we compute standard model evaluation metrics such as prediction errors. We find that current LLMs, such as GPT-4o, struggle with both experimental design and model discovery. We find that augmenting the LLM-based agent with an explicit statistical model does not reliably improve these results.

Title: Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information

• Authors: Rasul Tutnov, Antoine Grosnit, Haitham Bou-Ammar
• Subjects: cs.LG, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.01544
• Pdf URL: https://arxiv.org/pdf/2501.01544
• Copy Paste: [[2501.01544]] Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information(https://arxiv.org/abs/2501.01544)
• Keywords: robust, large language model
• Abstract: Post-alignment of large language models (LLMs) is critical in improving their utility, safety, and alignment with human intentions. Direct preference optimisation (DPO) has become one of the most widely used algorithms for achieving this alignment, given its ability to optimise models based on human feedback directly. However, the vast number of DPO variants in the literature has made it increasingly difficult for researchers to navigate and fully grasp the connections between these approaches. This paper introduces a unifying framework inspired by mutual information, which proposes a new loss function with flexible priors. By carefully specifying these priors, we demonstrate that many existing algorithms, such as SimPO, TDPO, SparsePO, and others, can be derived from our framework. This unification offers a clearer and more structured approach, allowing researchers to understand the relationships between different DPO variants better. We aim to simplify the landscape of DPO algorithms, making it easier for the research community to gain insights and foster further advancements in LLM alignment. Ultimately, we hope our framework can be a foundation for developing more robust and interpretable alignment techniques.

Title: Click-Calib: A Robust Extrinsic Calibration Method for Surround-View Systems

• Authors: Lihao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01557
• Pdf URL: https://arxiv.org/pdf/2501.01557
• Copy Paste: [[2501.01557]] Click-Calib: A Robust Extrinsic Calibration Method for Surround-View Systems(https://arxiv.org/abs/2501.01557)
• Keywords: robust
• Abstract: Surround-View System (SVS) is an essential component in Advanced Driver Assistance System (ADAS) and requires precise calibrations. However, conventional offline extrinsic calibration methods are cumbersome and time-consuming as they rely heavily on physical patterns. Additionally, these methods primarily focus on short-range areas surrounding the vehicle, resulting in lower calibration quality in more distant zones. To address these limitations, we propose Click-Calib, a pattern-free approach for offline SVS extrinsic calibration. Without requiring any special setup, the user only needs to click a few keypoints on the ground in natural scenes. Unlike other offline calibration approaches, Click-Calib optimizes camera poses over a wide range by minimizing reprojection distance errors of keypoints, thereby achieving accurate calibrations at both short and long distances. Furthermore, Click-Calib supports both single-frame and multiple-frame modes, with the latter offering even better results. Evaluations on our in-house dataset and the public WoodScape dataset demonstrate its superior accuracy and robustness compared to baseline methods. Code is avalaible at this https URL.

Title: Predicting the Performance of Black-box LLMs through Self-Queries

• Authors: Dylan Sam, Marc Finzi, J. Zico Kolter
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01558
• Pdf URL: https://arxiv.org/pdf/2501.01558
• Copy Paste: [[2501.01558]] Predicting the Performance of Black-box LLMs through Self-Queries(https://arxiv.org/abs/2501.01558)
• Keywords: large language model
• Abstract: As large language models (LLMs) are increasingly relied on in AI systems, predicting when they make mistakes is crucial. While a great deal of work in the field uses internal representations to interpret model behavior, these representations are inaccessible when given solely black-box access through an API. In this paper, we extract features of LLMs in a black-box manner by using follow-up prompts and taking the probabilities of different responses as representations to train reliable predictors of model behavior. We demonstrate that training a linear model on these low-dimensional representations produces reliable and generalizable predictors of model performance at the instance level (e.g., if a particular generation correctly answers a question). Remarkably, these can often outperform white-box linear predictors that operate over a model's hidden state or the full distribution over its vocabulary. In addition, we demonstrate that these extracted features can be used to evaluate more nuanced aspects of a language model's state. For instance, they can be used to distinguish between a clean version of GPT-4o-mini and a version that has been influenced via an adversarial system prompt that answers question-answering tasks incorrectly or introduces bugs into generated code. Furthermore, they can reliably distinguish between different model architectures and sizes, enabling the detection of misrepresented models provided through an API (e.g., identifying if GPT-3.5 is supplied instead of GPT-4o-mini).

Title: Stackelberg Game Based Performance Optimization in Digital Twin Assisted Federated Learning over NOMA Networks

• Authors: Bibo Wu, Fang Fang, Xianbin Wang
• Subjects: cs.LG, cs.CR, cs.GT, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.01584
• Pdf URL: https://arxiv.org/pdf/2501.01584
• Copy Paste: [[2501.01584]] Stackelberg Game Based Performance Optimization in Digital Twin Assisted Federated Learning over NOMA Networks(https://arxiv.org/abs/2501.01584)
• Keywords: privacy, attack, federate
• Abstract: Despite the advantage of preserving data privacy, federated learning (FL) still suffers from the straggler issue due to the limited computing resources of distributed clients and the unreliable wireless communication environment. By effectively imitating the distributed resources, digital twin (DT) shows great potential in alleviating this issue. In this paper, we leverage DT in the FL framework over non-orthogonal multiple access (NOMA) network to assist FL training process, considering malicious attacks on model updates from clients. A reputationbased client selection scheme is proposed, which accounts for client heterogeneity in multiple aspects and effectively mitigates the risks of poisoning attacks in FL systems. To minimize the total latency and energy consumption in the proposed system, we then formulate a Stackelberg game by considering clients and the server as the leader and the follower, respectively. Specifically, the leader aims to minimize the energy consumption while the objective of the follower is to minimize the total latency during FL training. The Stackelberg equilibrium is achieved to obtain the optimal solutions. We first derive the strategies for the followerlevel problem and include them in the leader-level problem which is then solved via problem decomposition. Simulation results verify the superior performance of the proposed scheme.

Title: (WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges

• Authors: Mohamed Hisham Abdellatif
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01588
• Pdf URL: https://arxiv.org/pdf/2501.01588
• Copy Paste: [[2501.01588]] (WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges(https://arxiv.org/abs/2501.01588)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have become essential tools across various domains due to their impressive capabilities in understanding and generating human-like text. The ability to accurately answer multiple-choice questions (MCQs) holds significant value in education, particularly in automated tutoring systems and assessment platforms. However, adapting LLMs to handle MCQ tasks effectively remains challenging due to the hallucinations and unclear prompts. This work explores the potential of Microsoft's PHI-3\cite{Abdin2024}, a compact yet efficient LLM, for MCQ answering. Our contributions include fine-tuning the model on the TruthfulQA dataset, designing optimized prompts to enhance model performance, and evaluating using perplexity and traditional metrics like accuracy and F1 score. Results show a remarkable improvement in PHI-3.5's MCQ handling post-fine-tuning, with perplexity decreasing from 4.68 to 2.27, and accuracy rising from 62\% to 90.8\%. This research underlines the importance of efficient models in adaptive learning systems and educational assessments, paving the way for broader integration into the classroom, particularly in fields like test preparation, student feedback, and personalized learning.

Title: D$^3$-Human: Dynamic Disentangled Digital Human from Monocular Video

• Authors: Honghu Chen, Bo Peng, Yunfan Tao, Juyong Zhang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.01589
• Pdf URL: https://arxiv.org/pdf/2501.01589
• Copy Paste: [[2501.01589]] D$^3$-Human: Dynamic Disentangled Digital Human from Monocular Video(https://arxiv.org/abs/2501.01589)
• Keywords: robust
• Abstract: We introduce D$^3$-Human, a method for reconstructing Dynamic Disentangled Digital Human geometry from monocular videos. Past monocular video human reconstruction primarily focuses on reconstructing undecoupled clothed human bodies or only reconstructing clothing, making it difficult to apply directly in applications such as animation production. The challenge in reconstructing decoupled clothing and body lies in the occlusion caused by clothing over the body. To this end, the details of the visible area and the plausibility of the invisible area must be ensured during the reconstruction process. Our proposed method combines explicit and implicit representations to model the decoupled clothed human body, leveraging the robustness of explicit representations and the flexibility of implicit representations. Specifically, we reconstruct the visible region as SDF and propose a novel human manifold signed distance field (hmSDF) to segment the visible clothing and visible body, and then merge the visible and invisible body. Extensive experimental results demonstrate that, compared with existing reconstruction schemes, D$^3$-Human can achieve high-quality decoupled reconstruction of the human body wearing different clothing, and can be directly applied to clothing transfer and animation.

Title: Multivariate Time Series Anomaly Detection using DiffGAN Model

• Authors: Guangqiang Wu, Fu Zhang
• Subjects: cs.LG, math.ST
• Abstract URL: https://arxiv.org/abs/2501.01591
• Pdf URL: https://arxiv.org/pdf/2501.01591
• Copy Paste: [[2501.01591]] Multivariate Time Series Anomaly Detection using DiffGAN Model(https://arxiv.org/abs/2501.01591)
• Keywords: diffusion, generative
• Abstract: In recent years, some researchers have applied diffusion models to multivariate time series anomaly detection. The partial diffusion strategy, which depends on the diffusion steps, is commonly used for anomaly detection in these models. However, different diffusion steps have an impact on the reconstruction of the original data, thereby impacting the effectiveness of anomaly detection. To address this issue, we propose a novel method named DiffGAN, which adds a generative adversarial network component to the denoiser of diffusion model. This addition allows for the simultaneous generation of noisy data and prediction of diffusion steps. Compared to multiple state-of-the-art reconstruction models, experimental results demonstrate that DiffGAN achieves superior performance in anomaly detection.

Title: PSYCHE: A Multi-faceted Patient Simulation Framework for Evaluation of Psychiatric Assessment Conversational Agents

• Authors: Jingoo Lee, Kyungho Lim, Young-Chul Jung, Byung-Hoon Kim
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01594
• Pdf URL: https://arxiv.org/pdf/2501.01594
• Copy Paste: [[2501.01594]] PSYCHE: A Multi-faceted Patient Simulation Framework for Evaluation of Psychiatric Assessment Conversational Agents(https://arxiv.org/abs/2501.01594)
• Keywords: large language model
• Abstract: Recent advances in large language models (LLMs) have accelerated the development of conversational agents capable of generating human-like responses. Since psychiatric assessments typically involve complex conversational interactions between psychiatrists and patients, there is growing interest in developing LLM-based psychiatric assessment conversational agents (PACAs) that aim to simulate the role of psychiatrists in clinical evaluations. However, standardized methods for benchmarking the clinical appropriateness of PACAs' interaction with patients still remain underexplored. Here, we propose PSYCHE, a novel framework designed to enable the 1) clinically relevant, 2) ethically safe, 3) cost-efficient, and 4) quantitative evaluation of PACAs. This is achieved by simulating psychiatric patients based on a multi-faceted psychiatric construct that defines the simulated patients' profiles, histories, and behaviors, which PACAs are expected to assess. We validate the effectiveness of PSYCHE through a study with 10 board-certified psychiatrists, supported by an in-depth analysis of the simulated patient utterances.

Title: Adaptive Homophily Clustering: A Structure Homophily Graph Learning with Adaptive Filter for Hyperspectral Image

• Authors: Yao Ding, Weijie Kang, Aitao Yang, Zhili Zhang, Junyang Zhao, Jie Feng, Danfeng Hong, Qinhe Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01595
• Pdf URL: https://arxiv.org/pdf/2501.01595
• Copy Paste: [[2501.01595]] Adaptive Homophily Clustering: A Structure Homophily Graph Learning with Adaptive Filter for Hyperspectral Image(https://arxiv.org/abs/2501.01595)
• Keywords: robust
• Abstract: Hyperspectral image (HSI) clustering has been a fundamental but challenging task with zero training labels. Currently, some deep graph clustering methods have been successfully explored for HSI due to their outstanding performance in effective spatial structural information encoding. Nevertheless, insufficient structural information utilization, poor feature presentation ability, and weak graph update capability limit their performance. Thus, in this paper, a homophily structure graph learning with an adaptive filter clustering method (AHSGC) for HSI is proposed. Specifically, homogeneous region generation is first developed for HSI processing and constructing the original graph. Afterward, an adaptive filter graph encoder is designed to adaptively capture the high and low frequency features on the graph for subsequence processing. Then, a graph embedding clustering self-training decoder is developed with KL Divergence, with which the pseudo-label is generated for network training. Meanwhile, homophily-enhanced structure learning is introduced to update the graph according to the clustering task, in which the orient correlation estimation is adopted to estimate the node connection, and graph edge sparsification is designed to adjust the edges in the graph dynamically. Finally, a joint network optimization is introduced to achieve network self-training and update the graph. The K-means is adopted to express the latent features. Extensive experiments and repeated comparative analysis have verified that our AHSGC contains high clustering accuracy, low computational complexity, and strong robustness. The code source will be available at this https URL.

Title: Few-shot Implicit Function Generation via Equivariance

• Authors: Suizhi Huang, Xingyi Yang, Hongtao Lu, Xinchao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01601
• Pdf URL: https://arxiv.org/pdf/2501.01601
• Copy Paste: [[2501.01601]] Few-shot Implicit Function Generation via Equivariance(https://arxiv.org/abs/2501.01601)
• Keywords: diffusion
• Abstract: Implicit Neural Representations (INRs) have emerged as a powerful framework for representing continuous signals. However, generating diverse INR weights remains challenging due to limited training data. We introduce Few-shot Implicit Function Generation, a new problem setup that aims to generate diverse yet functionally consistent INR weights from only a few examples. This is challenging because even for the same signal, the optimal INRs can vary significantly depending on their initializations. To tackle this, we propose EquiGen, a framework that can generate new INRs from limited data. The core idea is that functionally similar networks can be transformed into one another through weight permutations, forming an equivariance group. By projecting these weights into an equivariant latent space, we enable diverse generation within these groups, even with few examples. EquiGen implements this through an equivariant encoder trained via contrastive learning and smooth augmentation, an equivariance-guided diffusion process, and controlled perturbations in the equivariant subspace. Experiments on 2D image and 3D shape INR datasets demonstrate that our approach effectively generates diverse INR weights while preserving their functional properties in few-shot scenarios.

Title: Google is all you need: Semi-Supervised Transfer Learning Strategy For Light Multimodal Multi-Task Classification Model

• Authors: Haixu Liu, Penghao Jiang, Zerui Tao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01611
• Pdf URL: https://arxiv.org/pdf/2501.01611
• Copy Paste: [[2501.01611]] Google is all you need: Semi-Supervised Transfer Learning Strategy For Light Multimodal Multi-Task Classification Model(https://arxiv.org/abs/2501.01611)
• Keywords: robust
• Abstract: As the volume of digital image data increases, the effectiveness of image classification intensifies. This study introduces a robust multi-label classification system designed to assign multiple labels to a single image, addressing the complexity of images that may be associated with multiple categories (ranging from 1 to 19, excluding 12). We propose a multi-modal classifier that merges advanced image recognition algorithms with Natural Language Processing (NLP) models, incorporating a fusion module to integrate these distinct modalities. The purpose of integrating textual data is to enhance the accuracy of label prediction by providing contextual understanding that visual analysis alone cannot fully capture. Our proposed classification model combines Convolutional Neural Networks (CNN) for image processing with NLP techniques for analyzing textual description (i.e., captions). This approach includes rigorous training and validation phases, with each model component verified and analyzed through ablation experiments. Preliminary results demonstrate the classifier's accuracy and efficiency, highlighting its potential as an automatic image-labeling system.

Title: Merging Context Clustering with Visual State Space Models for Medical Image Segmentation

• Authors: Yun Zhu, Dong Zhang, Yi Lin, Yifei Feng, Jinhui Tang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01618
• Pdf URL: https://arxiv.org/pdf/2501.01618
• Copy Paste: [[2501.01618]] Merging Context Clustering with Visual State Space Models for Medical Image Segmentation(https://arxiv.org/abs/2501.01618)
• Keywords: segmentation
• Abstract: Medical image segmentation demands the aggregation of global and local feature representations, posing a challenge for current methodologies in handling both long-range and short-range feature interactions. Recently, vision mamba (ViM) models have emerged as promising solutions for addressing model complexities by excelling in long-range feature iterations with linear complexity. However, existing ViM approaches overlook the importance of preserving short-range local dependencies by directly flattening spatial tokens and are constrained by fixed scanning patterns that limit the capture of dynamic spatial context information. To address these challenges, we introduce a simple yet effective method named context clustering ViM (CCViM), which incorporates a context clustering module within the existing ViM models to segment image tokens into distinct windows for adaptable local clustering. Our method effectively combines long-range and short-range feature interactions, thereby enhancing spatial contextual representations for medical image segmentation tasks. Extensive experimental evaluations on diverse public datasets, i.e., Kumar, CPM17, ISIC17, ISIC18, and Synapse demonstrate the superior performance of our method compared to current state-of-the-art methods. Our code can be found at this https URL.

Title: Adaptive Meta-learning-based Adversarial Training for Robust Automatic Modulation Classification

• Authors: Amirmohammad Bamdad, Ali Owfi, Fatemeh Afghah
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.01620
• Pdf URL: https://arxiv.org/pdf/2501.01620
• Copy Paste: [[2501.01620]] Adaptive Meta-learning-based Adversarial Training for Robust Automatic Modulation Classification(https://arxiv.org/abs/2501.01620)
• Keywords: defense, attack, robust
• Abstract: DL-based automatic modulation classification (AMC) models are highly susceptible to adversarial attacks, where even minimal input perturbations can cause severe misclassifications. While adversarially training an AMC model based on an adversarial attack significantly increases its robustness against that attack, the AMC model will still be defenseless against other adversarial attacks. The theoretically infinite possibilities for adversarial perturbations mean that an AMC model will inevitably encounter new unseen adversarial attacks if it is ever to be deployed to a real-world communication system. Moreover, the computational limitations and challenges of obtaining new data in real-time will not allow a full training process for the AMC model to adapt to the new attack when it is online. To this end, we propose a meta-learning-based adversarial training framework for AMC models that substantially enhances robustness against unseen adversarial attacks and enables fast adaptation to these attacks using just a few new training samples, if any are available. Our results demonstrate that this training framework provides superior robustness and accuracy with much less online training time than conventional adversarial training of AMC models, making it highly efficient for real-world deployment.

Title: ICPC: In-context Prompt Compression with Faster Inference

• Authors: Ziyang Yu, Yuyu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01625
• Pdf URL: https://arxiv.org/pdf/2501.01625
• Copy Paste: [[2501.01625]] ICPC: In-context Prompt Compression with Faster Inference(https://arxiv.org/abs/2501.01625)
• Keywords: large language model
• Abstract: Despite the recent success of Large Language Models (LLMs), it remains challenging to feed LLMs with long prompts due to the fixed size of LLM inputs. As a remedy, prompt compression becomes a promising solution by removing redundant tokens in the prompt. However, using LLM in the existing works requires additional computation resources and leads to memory overheads. To address it, we propose ICPC (In-context Prompt Compression), a novel and scalable prompt compression method that adaptively reduces the prompt length. The key idea of ICPC is to calculate the probability of each word appearing in the prompt using encoders and calculate information carried by each word through the information function, which effectively reduces the information loss during prompt compression and increases the speed of compression. Empirically, we demonstrate that ICPC can effectively compress long texts of different categories and thus achieve better performance and speed on different types of NLP tasks.

Title: A Probabilistic Model for Node Classification in Directed Graphs

• Authors: Diego Huerta, Gerardo Arizmendi
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.01630
• Pdf URL: https://arxiv.org/pdf/2501.01630
• Copy Paste: [[2501.01630]] A Probabilistic Model for Node Classification in Directed Graphs(https://arxiv.org/abs/2501.01630)
• Keywords: generative
• Abstract: In this work, we present a probabilistic model for directed graphs where nodes have attributes and labels. This model serves as a generative classifier capable of predicting the labels of unseen nodes using either maximum likelihood or maximum a posteriori estimations. The predictions made by this model are highly interpretable, contrasting with some common methods for node classification, such as graph neural networks. We applied the model to two datasets, demonstrating predictive performance that is competitive with, and even superior to, state-of-the-art methods. One of the datasets considered is adapted from the Math Genealogy Project, which has not previously been utilized for this purpose. Consequently, we evaluated several classification algorithms on this dataset to compare the performance of our model and provide benchmarks for this new resource.

Title: ACE: Anti-Editing Concept Erasure in Text-to-Image Models

• Authors: Zihao Wang, Yuxiang Wei, Fan Li, Renjing Pei, Hang Xu, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01633
• Pdf URL: https://arxiv.org/pdf/2501.01633
• Copy Paste: [[2501.01633]] ACE: Anti-Editing Concept Erasure in Text-to-Image Models(https://arxiv.org/abs/2501.01633)
• Keywords: diffusion
• Abstract: Recent advance in text-to-image diffusion models have significantly facilitated the generation of high-quality images, but also raising concerns about the illegal creation of harmful content, such as copyrighted images. Existing concept erasure methods achieve superior results in preventing the production of erased concept from prompts, but typically perform poorly in preventing undesired editing. To address this issue, we propose an Anti-Editing Concept Erasure (ACE) method, which not only erases the target concept during generation but also filters out it during editing. Specifically, we propose to inject the erasure guidance into both conditional and the unconditional noise prediction, enabling the model to effectively prevent the creation of erasure concepts during both editing and generation. Furthermore, a stochastic correction guidance is introduced during training to address the erosion of unrelated concepts. We conducted erasure editing experiments with representative editing methods (i.e., LEDITS++ and MasaCtrl) to erase IP characters, and the results indicate that our ACE effectively filters out target concepts in both types of edits. Additional experiments on erasing explicit concepts and artistic styles further demonstrate that our ACE performs favorably against state-of-the-art methods. Our code will be publicly available at this https URL.

Title: A non-ergodic framework for understanding emergent capabilities in Large Language Models

• Authors: Javier Marin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01638
• Pdf URL: https://arxiv.org/pdf/2501.01638
• Copy Paste: [[2501.01638]] A non-ergodic framework for understanding emergent capabilities in Large Language Models(https://arxiv.org/abs/2501.01638)
• Keywords: large language model
• Abstract: Large language models have emergent capabilities that come unexpectedly at scale, but we need a theoretical framework to explain why and how they emerge. We prove that language models are actually non-ergodic systems while providing a mathematical framework based on Stuart Kauffman's theory of the adjacent possible (TAP) to explain capability emergence. Our resource-constrained TAP equation demonstrates how architectural, training, and contextual constraints interact to shape model capabilities through phase transitions in semantic space. We prove through experiments with three different language models that capacities emerge through discrete transitions guided by constraint interactions and path-dependent exploration. This framework provides a theoretical basis for understanding emergence in language models and guides the development of architectures that can guide capability emergence.

Title: Uncertainty and Energy based Loss Guided Semi-Supervised Semantic Segmentation

• Authors: Rini Smita Thakur, Vinod K. Kurmi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01640
• Pdf URL: https://arxiv.org/pdf/2501.01640
• Copy Paste: [[2501.01640]] Uncertainty and Energy based Loss Guided Semi-Supervised Semantic Segmentation(https://arxiv.org/abs/2501.01640)
• Keywords: generative, segmentation
• Abstract: Semi-supervised (SS) semantic segmentation exploits both labeled and unlabeled images to overcome tedious and costly pixel-level annotation problems. Pseudolabel supervision is one of the core approaches of training networks with both pseudo labels and ground-truth labels. This work uses aleatoric or data uncertainty and energy based modeling in intersection-union pseudo supervised this http URL aleatoric uncertainty is modeling the inherent noise variations of the data in a network with two predictive branches. The per-pixel variance parameter obtained from the network gives a quantitative idea about the data uncertainty. Moreover, energy-based loss realizes the potential of generative modeling on the downstream SS segmentation task. The aleatoric and energy loss are applied in conjunction with pseudo-intersection labels, pseudo-union labels, and ground-truth on the respective network branch. The comparative analysis with state-of-the-art methods has shown improvement in performance metrics.

Title: iCBIR-Sli: Interpretable Content-Based Image Retrieval with 2D Slice Embeddings

• Authors: Shuhei Tomoshige, Hayato Muraki, Kenichi Oishi, Hitoshi Iyatomi
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.01642
• Pdf URL: https://arxiv.org/pdf/2501.01642
• Copy Paste: [[2501.01642]] iCBIR-Sli: Interpretable Content-Based Image Retrieval with 2D Slice Embeddings(https://arxiv.org/abs/2501.01642)
• Keywords: robust, interpretability, segmentation
• Abstract: Current methods for searching brain MR images rely on text-based approaches, highlighting a significant need for content-based image retrieval (CBIR) systems. Directly applying 3D brain MR images to machine learning models offers the benefit of effectively learning the brain's structure; however, building the generalized model necessitates a large amount of training data. While models that consider depth direction and utilize continuous 2D slices have demonstrated success in segmentation and classification tasks involving 3D data, concerns remain. Specifically, using general 2D slices may lead to the oversight of pathological features and discontinuities in depth direction information. Furthermore, to the best of the authors' knowledge, there have been no attempts to develop a practical CBIR system that preserves the entire brain's structural information. In this study, we propose an interpretable CBIR method for brain MR images, named iCBIR-Sli (Interpretable CBIR with 2D Slice Embedding), which, for the first time globally, utilizes a series of 2D slices. iCBIR-Sli addresses the challenges associated with using 2D slices by effectively aggregating slice information, thereby achieving low-dimensional representations with high completeness, usability, robustness, and interoperability, which are qualities essential for effective CBIR. In retrieval evaluation experiments utilizing five publicly available brain MR datasets (ADNI2/3, OASIS3/4, AIBL) for Alzheimer's disease and cognitively normal, iCBIR-Sli demonstrated top-1 retrieval performance (macro F1 = 0.859), comparable to existing deep learning models explicitly designed for classification, without the need for an external classifier. Additionally, the method provided high interpretability by clearly identifying the brain regions indicative of the searched-for disease.

Title: Multimodal Contrastive Representation Learning in Augmented Biomedical Knowledge Graphs

• Authors: Tien Dang, Viet Thanh Duy Nguyen, Minh Tuan Le, Truong-Son Hy
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01644
• Pdf URL: https://arxiv.org/pdf/2501.01644
• Copy Paste: [[2501.01644]] Multimodal Contrastive Representation Learning in Augmented Biomedical Knowledge Graphs(https://arxiv.org/abs/2501.01644)
• Keywords: robust
• Abstract: Biomedical Knowledge Graphs (BKGs) integrate diverse datasets to elucidate complex relationships within the biomedical field. Effective link prediction on these graphs can uncover valuable connections, such as potential novel drug-disease relations. We introduce a novel multimodal approach that unifies embeddings from specialized Language Models (LMs) with Graph Contrastive Learning (GCL) to enhance intra-entity relationships while employing a Knowledge Graph Embedding (KGE) model to capture inter-entity relationships for effective link prediction. To address limitations in existing BKGs, we present PrimeKG++, an enriched knowledge graph incorporating multimodal data, including biological sequences and textual descriptions for each entity type. By combining semantic and relational information in a unified representation, our approach demonstrates strong generalizability, enabling accurate link predictions even for unseen nodes. Experimental results on PrimeKG++ and the DrugBank drug-target interaction dataset demonstrate the effectiveness and robustness of our method across diverse biomedical datasets. Our source code, pre-trained models, and data are publicly available at this https URL

Title: HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding

• Authors: Heqing Zou, Tianze Luo, Guiyang Xie, Victor (Xiao Jie)Zhang, Fengmao Lv, Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01645
• Pdf URL: https://arxiv.org/pdf/2501.01645
• Copy Paste: [[2501.01645]] HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding(https://arxiv.org/abs/2501.01645)
• Keywords: large language model
• Abstract: Multimodal large language models have become a popular topic in deep visual understanding due to many promising real-world applications. However, hour-long video understanding, spanning over one hour and containing tens of thousands of visual frames, remains under-explored because of 1) challenging long-term video analyses, 2) inefficient large-model approaches, and 3) lack of large-scale benchmark datasets. Among them, in this paper, we focus on building a large-scale hour-long long video benchmark, HLV-1K, designed to evaluate long video understanding models. HLV-1K comprises 1009 hour-long videos with 14,847 high-quality question answering (QA) and multi-choice question asnwering (MCQA) pairs with time-aware query and diverse annotations, covering frame-level, within-event-level, cross-event-level, and long-term reasoning tasks. We evaluate our benchmark using existing state-of-the-art methods and demonstrate its value for testing deep long video understanding capabilities at different levels and for various tasks. This includes promoting future long video understanding tasks at a granular level, such as deep understanding of long live videos, meeting recordings, and movies.

Title: Dual Mutual Learning Network with Global-local Awareness for RGB-D Salient Object Detection

• Authors: Kang Yi, Haoran Tang, Yumeng Li, Jing Xu, Jun Zhang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2501.01648
• Pdf URL: https://arxiv.org/pdf/2501.01648
• Copy Paste: [[2501.01648]] Dual Mutual Learning Network with Global-local Awareness for RGB-D Salient Object Detection(https://arxiv.org/abs/2501.01648)
• Keywords: transformer
• Abstract: RGB-D salient object detection (SOD), aiming to highlight prominent regions of a given scene by jointly modeling RGB and depth information, is one of the challenging pixel-level prediction tasks. Recently, the dual-attention mechanism has been devoted to this area due to its ability to strengthen the detection process. However, most existing methods directly fuse attentional cross-modality features under a manual-mandatory fusion paradigm without considering the inherent discrepancy between the RGB and depth, which may lead to a reduction in performance. Moreover, the long-range dependencies derived from global and local information make it difficult to leverage a unified efficient fusion strategy. Hence, in this paper, we propose the GL-DMNet, a novel dual mutual learning network with global-local awareness. Specifically, we present a position mutual fusion module and a channel mutual fusion module to exploit the interdependencies among different modalities in spatial and channel dimensions. Besides, we adopt an efficient decoder based on cascade transformer-infused reconstruction to integrate multi-level fusion features jointly. Extensive experiments on six benchmark datasets demonstrate that our proposed GL-DMNet performs better than 24 RGB-D SOD methods, achieving an average improvement of ~3% across four evaluation metrics compared to the second-best model (S3Net). Codes and results are available at this https URL.

Title: MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments

• Authors: Cai Yin, Gu Zhouhong, Du Zhaohan, Ye Zheyu, Cao Shaosheng, Xu Yiqian, Feng Hongwei, Chen Ping
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01652
• Pdf URL: https://arxiv.org/pdf/2501.01652
• Copy Paste: [[2501.01652]] MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments(https://arxiv.org/abs/2501.01652)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities in environmental perception, reasoning-based decision-making, and simulating complex human behaviors, particularly in interactive role-playing contexts. This paper introduces the Multiverse Interactive Role-play Ability General Evaluation (MIRAGE), a comprehensive framework designed to assess LLMs' proficiency in portraying advanced human behaviors through murder mystery games. MIRAGE features eight intricately crafted scripts encompassing diverse themes and styles, providing a rich simulation. To evaluate LLMs' performance, MIRAGE employs four distinct methods: the Trust Inclination Index (TII) to measure dynamics of trust and suspicion, the Clue Investigation Capability (CIC) to measure LLMs' capability of conducting information, the Interactivity Capability Index (ICI) to assess role-playing capabilities and the Script Compliance Index (SCI) to assess LLMs' capability of understanding and following instructions. Our experiments indicate that even popular models like GPT-4 face significant challenges in navigating the complexities presented by the MIRAGE. The datasets and simulation codes are available in \href{this https URL}{github}.

Title: Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning

• Authors: Danni Peng, Yuan Wang, Huazhu Fu, Jinpeng Jiang, Yong Liu, Rick Siow Mong Goh, Qingsong Wei
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.01653
• Pdf URL: https://arxiv.org/pdf/2501.01653
• Copy Paste: [[2501.01653]] Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning(https://arxiv.org/abs/2501.01653)
• Keywords: federate
• Abstract: Personalized federated learning (PFL) studies effective model personalization to address the data heterogeneity issue among clients in traditional federated learning (FL). Existing PFL approaches mainly generate personalized models by relying solely on the clients' latest updated models while ignoring their previous updates, which may result in suboptimal personalized model learning. To bridge this gap, we propose a novel framework termed pFedSeq, designed for personalizing adapters to fine-tune a foundation model in FL. In pFedSeq, the server maintains and trains a sequential learner, which processes a sequence of past adapter updates from clients and generates calibrations for personalized adapters. To effectively capture the cross-client and cross-step relations hidden in previous updates and generate high-performing personalized adapters, pFedSeq adopts the powerful selective state space model (SSM) as the architecture of sequential learner. Through extensive experiments on four public benchmark datasets, we demonstrate the superiority of pFedSeq over state-of-the-art PFL methods.

Title: EAUWSeg: Eliminating annotation uncertainty in weakly-supervised medical image segmentation

• Authors: Wang Lituan, Zhang Lei, Wang Yan, Wang Zhenbin, Zhang Zhenwei, Zhang Yi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01658
• Pdf URL: https://arxiv.org/pdf/2501.01658
• Copy Paste: [[2501.01658]] EAUWSeg: Eliminating annotation uncertainty in weakly-supervised medical image segmentation(https://arxiv.org/abs/2501.01658)
• Keywords: segmentation
• Abstract: Weakly-supervised medical image segmentation is gaining traction as it requires only rough annotations rather than accurate pixel-to-pixel labels, thereby reducing the workload for specialists. Although some progress has been made, there is still a considerable performance gap between the label-efficient methods and fully-supervised one, which can be attributed to the uncertainty nature of these weak labels. To address this issue, we propose a novel weak annotation method coupled with its learning framework EAUWSeg to eliminate the annotation uncertainty. Specifically, we first propose the Bounded Polygon Annotation (BPAnno) by simply labeling two polygons for a lesion. Then, the tailored learning mechanism that explicitly treat bounded polygons as two separated annotations is proposed to learn invariant feature by providing adversarial supervision signal for model training. Subsequently, a confidence-auxiliary consistency learner incorporates with a classification-guided confidence generator is designed to provide reliable supervision signal for pixels in uncertain region by leveraging the feature presentation consistency across pixels within the same category as well as class-specific information encapsulated in bounded polygons annotation. Experimental results demonstrate that EAUWSeg outperforms existing weakly-supervised segmentation methods. Furthermore, compared to fully-supervised counterparts, the proposed method not only delivers superior performance but also costs much less annotation workload. This underscores the superiority and effectiveness of our approach.

Title: BARTPredict: Empowering IoT Security with LLM-Driven Cyber Threat Prediction

• Authors: Alaeddine Diaf, Abdelaziz Amara Korba, Nour Elislem Karabadji, Yacine Ghamri-Doudane
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01664
• Pdf URL: https://arxiv.org/pdf/2501.01664
• Copy Paste: [[2501.01664]] BARTPredict: Empowering IoT Security with LLM-Driven Cyber Threat Prediction(https://arxiv.org/abs/2501.01664)
• Keywords: security, attack, transformer, large language model
• Abstract: The integration of Internet of Things (IoT) technology in various domains has led to operational advancements, but it has also introduced new vulnerabilities to cybersecurity threats, as evidenced by recent widespread cyberattacks on IoT devices. Intrusion detection systems are often reactive, triggered by specific patterns or anomalies observed within the network. To address this challenge, this work proposes a proactive approach to anticipate and preemptively mitigate malicious activities, aiming to prevent potential damage before it occurs. This paper proposes an innovative intrusion prediction framework empowered by Pre-trained Large Language Models (LLMs). The framework incorporates two LLMs: a fine-tuned Bidirectional and AutoRegressive Transformers (BART) model for predicting network traffic and a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model for evaluating the predicted traffic. By harnessing the bidirectional capabilities of BART the framework then identifies malicious packets among these predictions. Evaluated using the CICIoT2023 IoT attack dataset, our framework showcases a notable enhancement in predictive performance, attaining an impressive 98% overall accuracy, providing a powerful response to the cybersecurity challenges that confront IoT networks.

Title: FairSense: Long-Term Fairness Analysis of ML-Enabled Systems

• Authors: Yining She, Sumon Biswas, Christian Kästner, Eunsuk Kang
• Subjects: cs.LG, cs.CY, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.01665
• Pdf URL: https://arxiv.org/pdf/2501.01665
• Copy Paste: [[2501.01665]] FairSense: Long-Term Fairness Analysis of ML-Enabled Systems(https://arxiv.org/abs/2501.01665)
• Keywords: fair
• Abstract: Algorithmic fairness of machine learning (ML) models has raised significant concern in the recent years. Many testing, verification, and bias mitigation techniques have been proposed to identify and reduce fairness issues in ML models. The existing methods are model-centric and designed to detect fairness issues under static settings. However, many ML-enabled systems operate in a dynamic environment where the predictive decisions made by the system impact the environment, which in turn affects future decision-making. Such a self-reinforcing feedback loop can cause fairness violations in the long term, even if the immediate outcomes are fair. In this paper, we propose a simulation-based framework called FairSense to detect and analyze long-term unfairness in ML-enabled systems. Given a fairness requirement, FairSense performs Monte-Carlo simulation to enumerate evolution traces for each system configuration. Then, FairSense performs sensitivity analysis on the space of possible configurations to understand the impact of design options and environmental factors on the long-term fairness of the system. We demonstrate FairSense's potential utility through three real-world case studies: Loan lending, opioids risk scoring, and predictive policing.

Title: Practical Secure Inference Algorithm for Fine-tuned Large Language Model Based on Fully Homomorphic Encryption

• Authors: Zhang Ruoyan, Zheng Zhongxiang, Bao Wankang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.01672
• Pdf URL: https://arxiv.org/pdf/2501.01672
• Copy Paste: [[2501.01672]] Practical Secure Inference Algorithm for Fine-tuned Large Language Model Based on Fully Homomorphic Encryption(https://arxiv.org/abs/2501.01672)
• Keywords: secure, security, privacy, protect, attack, extraction, large language model
• Abstract: Large language models(LLMs) are currently at the forefront of the machine learning field, which show a broad application prospect but at the same time expose some risks of privacy leakage. We combined Fully Homomorphic Encryption(FHE) and provable security theory with Parameter-Efficient Fine-Tuning(PEFT) to propose an efficient and secure inference scheme for LLMs. More specially, we focus on pre-trained LLMs who rely on open-sourced base model and then fine-tuned with the private datasets by LoRA. This is a popular road-map for Vertical Domain Models such as LawGPT and BenTsao. We use two key technologies below. Firstly, we divide the whole model into the public part and the private part. The weights of public part are publicly accessible(e.g. the open-sourced base model) while the private part needs to be protected(e.g. the LoRA matrices). In this way, the overhead brought by computing on private data can be greatly reduced. Secondly, we propose a general method to transform a linear layer into another one which provides security against model extraction attacks and preserves its original functionality, which denotes as Private Linear Layer(PLL). Then we use this method on the LoRA matrices to make sure that the server protects their private weights without restricting the user's input. We also show that the difficulty of performing model extraction attacks for PLL can be generalized to the well-known hard problem Learning with Errors(LWE). Combing this method with FHE, we can protect user's input at the same time. This transform method can be applied to any linear layer to gain an extra protection against model extraction attacks. In this paper, we use the open-source model ChatGLM2-6B as the base model which is fine-tuned by LoRA. Experimental results show the inference efficiency of our scheme reaches 1.61s/token which shows that the scheme has good practicality.

Title: Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models

• Authors: Lei Tang, Jinghui Qin, Wenxuan Ye, Hao Tan, Zhijing Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01679
• Pdf URL: https://arxiv.org/pdf/2501.01679
• Copy Paste: [[2501.01679]] Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models(https://arxiv.org/abs/2501.01679)
• Keywords: large language model
• Abstract: Recently, Large language models (LLMs) with in-context learning have demonstrated remarkable potential in handling neural machine translation. However, existing evidence shows that LLMs are prompt-sensitive and it is sub-optimal to apply the fixed prompt to any input for downstream machine translation tasks. To address this issue, we propose an adaptive few-shot prompting (AFSP) framework to automatically select suitable translation demonstrations for various source input sentences to further elicit the translation capability of an LLM for better machine translation. First, we build a translation demonstration retrieval module based on LLM's embedding to retrieve top-k semantic-similar translation demonstrations from aligned parallel translation corpus. Rather than using other embedding models for semantic demonstration retrieval, we build a hybrid demonstration retrieval module based on the embedding layer of the deployed LLM to build better input representation for retrieving more semantic-related translation demonstrations. Then, to ensure better semantic consistency between source inputs and target outputs, we force the deployed LLM itself to generate multiple output candidates in the target language with the help of translation demonstrations and rerank these candidates. Besides, to better evaluate the effectiveness of our AFSP framework on the latest language and extend the research boundary of neural machine translation, we construct a high-quality diplomatic Chinese-English parallel dataset that consists of 5,528 parallel Chinese-English sentences. Finally, extensive experiments on the proposed diplomatic Chinese-English parallel dataset and the United Nations Parallel Corpus (Chinese-English part) show the effectiveness and superiority of our proposed AFSP.

Title: IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks

• Authors: Aecheon Jung, Soyun Choi, Junhong Min, Sungeun Hong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01685
• Pdf URL: https://arxiv.org/pdf/2501.01685
• Copy Paste: [[2501.01685]] IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks(https://arxiv.org/abs/2501.01685)
• Keywords: robust, segmentation
• Abstract: Image segmentation is a vital task for providing human assistance and enhancing autonomy in our daily lives. In particular, RGB-D segmentation-leveraging both visual and depth cues-has attracted increasing attention as it promises richer scene understanding than RGB-only methods. However, most existing efforts have primarily focused on semantic segmentation and thus leave a critical gap. There is a relative scarcity of instance-level RGB-D segmentation datasets, which restricts current methods to broad category distinctions rather than fully capturing the fine-grained details required for recognizing individual objects. To bridge this gap, we introduce three RGB-D instance segmentation benchmarks, distinguished at the instance level. These datasets are versatile, supporting a wide range of applications from indoor navigation to robotic manipulation. In addition, we present an extensive evaluation of various baseline models on these benchmarks. This comprehensive analysis identifies both their strengths and shortcomings, guiding future work toward more robust, generalizable solutions. Finally, we propose a simple yet effective method for RGB-D data integration. Extensive evaluations affirm the effectiveness of our approach, offering a robust framework for advancing toward more nuanced scene understanding.

Title: Quantitative Gait Analysis from Single RGB Videos Using a Dual-Input Transformer-Based Network

• Authors: Hiep Dinh, Son Le, My Than, Minh Ho, Nicolas Vuillerme, Hieu Pham
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01689
• Pdf URL: https://arxiv.org/pdf/2501.01689
• Copy Paste: [[2501.01689]] Quantitative Gait Analysis from Single RGB Videos Using a Dual-Input Transformer-Based Network(https://arxiv.org/abs/2501.01689)
• Keywords: transformer
• Abstract: Gait and movement analysis have become a well-established clinical tool for diagnosing health conditions, monitoring disease progression for a wide spectrum of diseases, and to implement and assess treatment, surgery and or rehabilitation interventions. However, quantitative motion assessment remains limited to costly motion capture systems and specialized personnel, restricting its accessibility and broader application. Recent advancements in deep neural networks have enabled quantitative movement analysis using single-camera videos, offering an accessible alternative to conventional motion capture systems. In this paper, we present an efficient approach for clinical gait analysis through a dual-pattern input convolutional Transformer network. The proposed system leverages a dual-input Transformer model to estimate essential gait parameters from single RGB videos captured by a single-view camera. The system demonstrates high accuracy in estimating critical metrics such as the gait deviation index (GDI), knee flexion angle, step length, and walking cadence, validated on a dataset of individuals with movement disorders. Notably, our approach surpasses state-of-the-art methods in various scenarios, using fewer resources and proving highly suitable for clinical application, particularly in resource-constrained environments.

Title: VidFormer: A novel end-to-end framework fused by 3DCNN and Transformer for Video-based Remote Physiological Measurement

• Authors: Jiachen Li, Shisheng Guo, Longzhen Tang, Cuolong Cui, Lingjiang Kong, Xiaobo Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01691
• Pdf URL: https://arxiv.org/pdf/2501.01691
• Copy Paste: [[2501.01691]] VidFormer: A novel end-to-end framework fused by 3DCNN and Transformer for Video-based Remote Physiological Measurement(https://arxiv.org/abs/2501.01691)
• Keywords: extraction, transformer
• Abstract: Remote physiological signal measurement based on facial videos, also known as remote photoplethysmography (rPPG), involves predicting changes in facial vascular blood flow from facial videos. While most deep learning-based methods have achieved good results, they often struggle to balance performance across small and large-scale datasets due to the inherent limitations of convolutional neural networks (CNNs) and Transformer. In this paper, we introduce VidFormer, a novel end-to-end framework that integrates 3-Dimension Convolutional Neural Network (3DCNN) and Transformer models for rPPG tasks. Initially, we conduct an analysis of the traditional skin reflection model and subsequently introduce an enhanced model for the reconstruction of rPPG signals. Based on this improved model, VidFormer utilizes 3DCNN and Transformer to extract local and global features from input data, respectively. To enhance the spatiotemporal feature extraction capabilities of VidFormer, we incorporate temporal-spatial attention mechanisms tailored for both 3DCNN and Transformer. Additionally, we design a module to facilitate information exchange and fusion between the 3DCNN and Transformer. Our evaluation on five publicly available datasets demonstrates that VidFormer outperforms current state-of-the-art (SOTA) methods. Finally, we discuss the essential roles of each VidFormer module and examine the effects of ethnicity, makeup, and exercise on its performance.

Title: Denoising and Adaptive Online Vertical Federated Learning for Sequential Multi-Sensor Data in Industrial Internet of Things

• Authors: Heqiang Wang, Xiaoxiong Zhong, Kang Liu, Fangming Liu, Weizhe Zhang
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.01693
• Pdf URL: https://arxiv.org/pdf/2501.01693
• Copy Paste: [[2501.01693]] Denoising and Adaptive Online Vertical Federated Learning for Sequential Multi-Sensor Data in Industrial Internet of Things(https://arxiv.org/abs/2501.01693)
• Keywords: privacy, federate
• Abstract: With the continuous improvement in the computational capabilities of edge devices such as intelligent sensors in the Industrial Internet of Things, these sensors are no longer limited to mere data collection but are increasingly capable of performing complex computational tasks. This advancement provides both the motivation and the foundation for adopting distributed learning approaches. This study focuses on an industrial assembly line scenario where multiple sensors, distributed across various locations, sequentially collect real-time data characterized by distinct feature spaces. To leverage the computational potential of these sensors while addressing the challenges of communication overhead and privacy concerns inherent in centralized learning, we propose the Denoising and Adaptive Online Vertical Federated Learning (DAO-VFL) algorithm. Tailored to the industrial assembly line scenario, DAO-VFL effectively manages continuous data streams and adapts to shifting learning objectives. Furthermore, it can address critical challenges prevalent in industrial environment, such as communication noise and heterogeneity of sensor capabilities. To support the proposed algorithm, we provide a comprehensive theoretical analysis, highlighting the effects of noise reduction and adaptive local iteration decisions on the regret bound. Experimental results on two real-world datasets further demonstrate the superior performance of DAO-VFL compared to benchmarks algorithms.

Title: Robust Self-Paced Hashing for Cross-Modal Retrieval with Noisy Labels

• Authors: Ruitao Pu, Yuan Sun, Yang Qin, Zhenwen Ren, Xiaomin Song, Huiming Zheng, Dezhong Peng
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2501.01699
• Pdf URL: https://arxiv.org/pdf/2501.01699
• Copy Paste: [[2501.01699]] Robust Self-Paced Hashing for Cross-Modal Retrieval with Noisy Labels(https://arxiv.org/abs/2501.01699)
• Keywords: robust
• Abstract: Cross-modal hashing (CMH) has appeared as a popular technique for cross-modal retrieval due to its low storage cost and high computational efficiency in large-scale data. Most existing methods implicitly assume that multi-modal data is correctly labeled, which is expensive and even unattainable due to the inevitable imperfect annotations (i.e., noisy labels) in real-world scenarios. Inspired by human cognitive learning, a few methods introduce self-paced learning (SPL) to gradually train the model from easy to hard samples, which is often used to mitigate the effects of feature noise or outliers. It is a less-touched problem that how to utilize SPL to alleviate the misleading of noisy labels on the hash model. To tackle this problem, we propose a new cognitive cross-modal retrieval method called Robust Self-paced Hashing with Noisy Labels (RSHNL), which can mimic the human cognitive process to identify the noise while embracing robustness against noisy labels. Specifically, we first propose a contrastive hashing learning (CHL) scheme to improve multi-modal consistency, thereby reducing the inherent semantic gap. Afterward, we propose center aggregation learning (CAL) to mitigate the intra-class variations. Finally, we propose Noise-tolerance Self-paced Hashing (NSH) that dynamically estimates the learning difficulty for each instance and distinguishes noisy labels through the difficulty level. For all estimated clean pairs, we further adopt a self-paced regularizer to gradually learn hash codes from easy to hard. Extensive experiments demonstrate that the proposed RSHNL performs remarkably well over the state-of-the-art CMH methods.

Title: Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models

• Authors: Guosheng Zhang, Keyao Wang, Haixiao Yue, Ajian Liu, Gang Zhang, Kun Yao, Errui Ding, Jingdong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01720
• Pdf URL: https://arxiv.org/pdf/2501.01720
• Copy Paste: [[2501.01720]] Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models(https://arxiv.org/abs/2501.01720)
• Keywords: security, large language model
• Abstract: Face Anti-Spoofing (FAS) is essential for ensuring the security and reliability of facial recognition systems. Most existing FAS methods are formulated as binary classification tasks, providing confidence scores without interpretation. They exhibit limited generalization in out-of-domain scenarios, such as new environments or unseen spoofing types. In this work, we introduce a multimodal large language model (MLLM) framework for FAS, termed Interpretable Face Anti-Spoofing (I-FAS), which transforms the FAS task into an interpretable visual question answering (VQA) paradigm. Specifically, we propose a Spoof-aware Captioning and Filtering (SCF) strategy to generate high-quality captions for FAS images, enriching the model's supervision with natural language interpretations. To mitigate the impact of noisy captions during training, we develop a Lopsided Language Model (L-LM) loss function that separates loss calculations for judgment and interpretation, prioritizing the optimization of the former. Furthermore, to enhance the model's perception of global visual features, we design a Globally Aware Connector (GAC) to align multi-level visual representations with the language model. Extensive experiments on standard and newly devised One to Eleven cross-domain benchmarks, comprising 12 public datasets, demonstrate that our method significantly outperforms state-of-the-art methods.

Title: AR4D: Autoregressive 4D Generation from Monocular Videos

• Authors: Hanxin Zhu, Tianyu He, Xiqian Yu, Junliang Guo, Zhibo Chen, Jiang Bian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01722
• Pdf URL: https://arxiv.org/pdf/2501.01722
• Copy Paste: [[2501.01722]] AR4D: Autoregressive 4D Generation from Monocular Videos(https://arxiv.org/abs/2501.01722)
• Keywords: generative
• Abstract: Recent advancements in generative models have ignited substantial interest in dynamic 3D content creation (\ie, 4D generation). Existing approaches primarily rely on Score Distillation Sampling (SDS) to infer novel-view videos, typically leading to issues such as limited diversity, spatial-temporal inconsistency and poor prompt alignment, due to the inherent randomness of SDS. To tackle these problems, we propose AR4D, a novel paradigm for SDS-free 4D generation. Specifically, our paradigm consists of three stages. To begin with, for a monocular video that is either generated or captured, we first utilize pre-trained expert models to create a 3D representation of the first frame, which is further fine-tuned to serve as the canonical space. Subsequently, motivated by the fact that videos happen naturally in an autoregressive manner, we propose to generate each frame's 3D representation based on its previous frame's representation, as this autoregressive generation manner can facilitate more accurate geometry and motion estimation. Meanwhile, to prevent overfitting during this process, we introduce a progressive view sampling strategy, utilizing priors from pre-trained large-scale 3D reconstruction models. To avoid appearance drift introduced by autoregressive generation, we further incorporate a refinement stage based on a global deformation field and the geometry of each frame's 3D representation. Extensive experiments have demonstrated that AR4D can achieve state-of-the-art 4D generation without SDS, delivering greater diversity, improved spatial-temporal consistency, and better alignment with input prompts.

Title: IGAF: Incremental Guided Attention Fusion for Depth Super-Resolution

• Authors: Athanasios Tragakis, Chaitanya Kaul, Kevin J. Mitchell, Hang Dai, Roderick Murray-Smith, Daniele Faccio
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01723
• Pdf URL: https://arxiv.org/pdf/2501.01723
• Copy Paste: [[2501.01723]] IGAF: Incremental Guided Attention Fusion for Depth Super-Resolution(https://arxiv.org/abs/2501.01723)
• Keywords: robust
• Abstract: Accurate depth estimation is crucial for many fields, including robotics, navigation, and medical imaging. However, conventional depth sensors often produce low-resolution (LR) depth maps, making detailed scene perception challenging. To address this, enhancing LR depth maps to high-resolution (HR) ones has become essential, guided by HR-structured inputs like RGB or grayscale images. We propose a novel sensor fusion methodology for guided depth super-resolution (GDSR), a technique that combines LR depth maps with HR images to estimate detailed HR depth maps. Our key contribution is the Incremental guided attention fusion (IGAF) module, which effectively learns to fuse features from RGB images and LR depth maps, producing accurate HR depth maps. Using IGAF, we build a robust super-resolution model and evaluate it on multiple benchmark datasets. Our model achieves state-of-the-art results compared to all baseline models on the NYU v2 dataset for $\times 4$, $\times 8$, and $\times 16$ upsampling. It also outperforms all baselines in a zero-shot setting on the Middlebury, Lu, and RGB-D-D datasets. Code, environments, and models are available on GitHub.

Title: Combined Hyper-Extensible Extremely-Secured Zero-Trust CIAM-PAM architecture

• Authors: Shivom Aggarwal, Shourya Mehra, Safeer Sathar
• Subjects: cs.CR, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.01732
• Pdf URL: https://arxiv.org/pdf/2501.01732
• Copy Paste: [[2501.01732]] Combined Hyper-Extensible Extremely-Secured Zero-Trust CIAM-PAM architecture(https://arxiv.org/abs/2501.01732)
• Keywords: secure, security, federate
• Abstract: Customer Identity and Access Management (CIAM) systems play a pivotal role in securing enterprise infrastructures. However, the complexity of implementing these systems requires careful architectural planning to ensure positive Return on Investment (RoI) and avoid costly delays. The proliferation of Active Persistent cyber threats, coupled with advancements in AI, cloud computing, and geographically distributed customer populations, necessitates a paradigm shift towards adaptive and zero-trust security frameworks. This paper introduces the Combined Hyper-Extensible Extremely-Secured Zero-Trust (CHEZ) CIAM-PAM architecture, designed specifically for large-scale enterprises. The CHEZ PL CIAM-PAM framework addresses critical security gaps by integrating federated identity management (private and public identities), password-less authentication, adaptive multi-factor authentication (MFA), microservice-based PEP (Policy Entitlement Point), multi-layer RBAC (Role Based Access Control) and multi-level trust systems. This future-proof design also includes end-to-end data encryption, and seamless integration with state-of-the-art AI-based threat detection systems, while ensuring compliance with stringent regulatory standards.

Title: Augmentation Matters: A Mix-Paste Method for X-Ray Prohibited Item Detection under Noisy Annotations

• Authors: Ruikang Chen, Yan Yan, Jing-Hao Xue, Yang Lu, Hanzi Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01733
• Pdf URL: https://arxiv.org/pdf/2501.01733
• Copy Paste: [[2501.01733]] Augmentation Matters: A Mix-Paste Method for X-Ray Prohibited Item Detection under Noisy Annotations(https://arxiv.org/abs/2501.01733)
• Keywords: robust
• Abstract: Automatic X-ray prohibited item detection is vital for public safety. Existing deep learning-based methods all assume that the annotations of training X-ray images are correct. However, obtaining correct annotations is extremely hard if not impossible for large-scale X-ray images, where item overlapping is this http URL a result, X-ray images are easily contaminated with noisy annotations, leading to performance deterioration of existing this http URL this paper, we address the challenging problem of training a robust prohibited item detector under noisy annotations (including both category noise and bounding box noise) from a novel perspective of data augmentation, and propose an effective label-aware mixed patch paste augmentation method (Mix-Paste). Specifically, for each item patch, we mix several item patches with the same category label from different images and replace the original patch in the image with the mixed patch. In this way, the probability of containing the correct prohibited item within the generated image is increased. Meanwhile, the mixing process mimics item overlapping, enabling the model to learn the characteristics of X-ray images. Moreover, we design an item-based large-loss suppression (LLS) strategy to suppress the large losses corresponding to potentially positive predictions of additional items due to the mixing operation. We show the superiority of our method on X-ray datasets under noisy annotations. In addition, we evaluate our method on the noisy MS-COCO dataset to showcase its generalization ability. These results clearly indicate the great potential of data augmentation to handle noise annotations. The source code is released at this https URL.

Title: Automating Legal Concept Interpretation with LLMs: Retrieval, Generation, and Evaluation

• Authors: Kangcheng Luo, Quzhe Huang, Cong Jiang, Yansong Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01743
• Pdf URL: https://arxiv.org/pdf/2501.01743
• Copy Paste: [[2501.01743]] Automating Legal Concept Interpretation with LLMs: Retrieval, Generation, and Evaluation(https://arxiv.org/abs/2501.01743)
• Keywords: large language model
• Abstract: Legal articles often include vague concepts to adapt to the ever-changing society. Providing detailed interpretations of these concepts is a critical task for legal practitioners, which requires meticulous and professional annotations by legal experts, admittedly time-consuming and expensive to collect at scale. In this paper, we introduce a novel retrieval-augmented generation framework, ATRI, for AuTomatically Retrieving relevant information from past judicial precedents and Interpreting vague legal concepts. We further propose a new benchmark, Legal Concept Entailment, to automate the evaluation of generated concept interpretations without expert involvement. Automatic evaluations indicate that our generated interpretations can effectively assist large language models (LLMs) in understanding vague legal concepts. Multi-faceted evaluations by legal experts indicate that the quality of our concept interpretations is comparable to those written by human experts. Our work has strong implications for leveraging LLMs to support legal practitioners in interpreting vague legal concepts and beyond.

Title: From Age Estimation to Age-Invariant Face Recognition: Generalized Age Feature Extraction Using Order-Enhanced Contrastive Learning

• Authors: Haoyi Wang, Victor Sanchez, Chang-Tsun Li, Nathan Clarke
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01760
• Pdf URL: https://arxiv.org/pdf/2501.01760
• Copy Paste: [[2501.01760]] From Age Estimation to Age-Invariant Face Recognition: Generalized Age Feature Extraction Using Order-Enhanced Contrastive Learning(https://arxiv.org/abs/2501.01760)
• Keywords: extraction
• Abstract: Generalized age feature extraction is crucial for age-related facial analysis tasks, such as age estimation and age-invariant face recognition (AIFR). Despite the recent successes of models in homogeneous-dataset experiments, their performance drops significantly in cross-dataset evaluations. Most of these models fail to extract generalized age features as they only attempt to map extracted features with training age labels directly without explicitly modeling the natural progression of aging. In this paper, we propose Order-Enhanced Contrastive Learning (OrdCon), which aims to extract generalized age features to minimize the domain gap across different datasets and scenarios. OrdCon aligns the direction vector of two features with either the natural aging direction or its reverse to effectively model the aging process. The method also leverages metric learning which is incorporated with a novel soft proxy matching loss to ensure that features are positioned around the center of each age cluster with minimum intra-class variance. We demonstrate that our proposed method achieves comparable results to state-of-the-art methods on various benchmark datasets in homogeneous-dataset evaluations for both age estimation and AIFR. In cross-dataset experiments, our method reduces the mean absolute error by about 1.38 in average for age estimation task and boosts the average accuracy for AIFR by 1.87%.

Title: Adverse Weather Conditions Augmentation of LiDAR Scenes with Latent Diffusion Models

• Authors: Andrea Matteazzi, Pascal Colling, Michael Arnold, Dietmar Tutsch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01761
• Pdf URL: https://arxiv.org/pdf/2501.01761
• Copy Paste: [[2501.01761]] Adverse Weather Conditions Augmentation of LiDAR Scenes with Latent Diffusion Models(https://arxiv.org/abs/2501.01761)
• Keywords: robust, diffusion, generative
• Abstract: LiDAR scenes constitute a fundamental source for several autonomous driving applications. Despite the existence of several datasets, scenes from adverse weather conditions are rarely available. This limits the robustness of downstream machine learning models, and restrains the reliability of autonomous driving systems in particular locations and seasons. Collecting feature-diverse scenes under adverse weather conditions is challenging due to seasonal limitations. Generative models are therefore essentials, especially for generating adverse weather conditions for specific driving scenarios. In our work, we propose a latent diffusion process constituted by autoencoder and latent diffusion models. Moreover, we leverage the clear condition LiDAR scenes with a postprocessing step to improve the realism of the generated adverse weather condition scenes.

Title: SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation

• Authors: Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, Yisen Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01765
• Pdf URL: https://arxiv.org/pdf/2501.01765
• Copy Paste: [[2501.01765]] SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation(https://arxiv.org/abs/2501.01765)
• Keywords: large language model
• Abstract: As advancements in large language models (LLMs) continue and the demand for personalized models increases, parameter-efficient fine-tuning (PEFT) methods (e.g., LoRA) will become essential due to their efficiency in reducing computation costs. However, recent studies have raised alarming concerns that LoRA fine-tuning could potentially compromise the safety alignment in LLMs, posing significant risks for the model owner. In this paper, we first investigate the underlying mechanism by analyzing the changes in safety alignment related features before and after fine-tuning. Then, we propose a fixed safety module calculated by safety data and a task-specific initialization for trainable parameters in low-rank adaptations, termed Safety-alignment preserved Low-Rank Adaptation (SaLoRA). Unlike previous LoRA methods and their variants, SaLoRA enables targeted modifications to LLMs without disrupting their original alignments. Our experiments show that SaLoRA outperforms various adapters-based approaches across various evaluation metrics in different fine-tuning tasks.

Title: LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extraction

• Authors: Er Jin, Qihui Feng, Yongli Mou, Stefan Decker, Gerhard Lakemeyer, Oliver Simons, Johannes Stegmaier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01767
• Pdf URL: https://arxiv.org/pdf/2501.01767
• Copy Paste: [[2501.01767]] LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extraction(https://arxiv.org/abs/2501.01767)
• Keywords: extraction
• Abstract: Logical image understanding involves interpreting and reasoning about the relationships and consistency within an image's visual content. This capability is essential in applications such as industrial inspection, where logical anomaly detection is critical for maintaining high-quality standards and minimizing costly recalls. Previous research in anomaly detection (AD) has relied on prior knowledge for designing algorithms, which often requires extensive manual annotations, significant computing power, and large amounts of data for training. Autoregressive, multimodal Vision Language Models (AVLMs) offer a promising alternative due to their exceptional performance in visual reasoning across various domains. Despite this, their application to logical AD remains unexplored. In this work, we investigate using AVLMs for logical AD and demonstrate that they are well-suited to the task. Combining AVLMs with format embedding and a logic reasoner, we achieve SOTA performance on public benchmarks, MVTec LOCO AD, with an AUROC of 86.0% and F1-max of 83.7%, along with explanations of anomalies. This significantly outperforms the existing SOTA method by a large margin.

Title: Can Synthetic Data be Fair and Private? A Comparative Study of Synthetic Data Generation and Fairness Algorithms

• Authors: Qinyi Liu, Oscar Deho, Farhad Vadiee, Mohammad Khalil, Srecko Joksimovic, George Siemens
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.01785
• Pdf URL: https://arxiv.org/pdf/2501.01785
• Copy Paste: [[2501.01785]] Can Synthetic Data be Fair and Private? A Comparative Study of Synthetic Data Generation and Fairness Algorithms(https://arxiv.org/abs/2501.01785)
• Keywords: privacy, fair
• Abstract: The increasing use of machine learning in learning analytics (LA) has raised significant concerns around algorithmic fairness and privacy. Synthetic data has emerged as a dual-purpose tool, enhancing privacy and improving fairness in LA models. However, prior research suggests an inverse relationship between fairness and privacy, making it challenging to optimize both. This study investigates which synthetic data generators can best balance privacy and fairness, and whether pre-processing fairness algorithms, typically applied to real datasets, are effective on synthetic data. Our results highlight that the DEbiasing CAusal Fairness (DECAF) algorithm achieves the best balance between privacy and fairness. However, DECAF suffers in utility, as reflected in its predictive accuracy. Notably, we found that applying pre-processing fairness algorithms to synthetic data improves fairness even more than when applied to real data. These findings suggest that combining synthetic data generation with fairness pre-processing offers a promising approach to creating fairer LA models.

Title: Advancing privacy in learning analytics using differential privacy

• Authors: Qinyi Liu, Ronas Shakya, Mohammad Khalil, Jelena Jovanovic
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.01786
• Pdf URL: https://arxiv.org/pdf/2501.01786
• Copy Paste: [[2501.01786]] Advancing privacy in learning analytics using differential privacy(https://arxiv.org/abs/2501.01786)
• Keywords: privacy, protect, attack, robust
• Abstract: This paper addresses the challenge of balancing learner data privacy with the use of data in learning analytics (LA) by proposing a novel framework by applying Differential Privacy (DP). The need for more robust privacy protection keeps increasing, driven by evolving legal regulations and heightened privacy concerns, as well as traditional anonymization methods being insufficient for the complexities of educational data. To address this, we introduce the first DP framework specifically designed for LA and provide practical guidance for its implementation. We demonstrate the use of this framework through a LA usage scenario and validate DP in safeguarding data privacy against potential attacks through an experiment on a well-known LA dataset. Additionally, we explore the trade-offs between data privacy and utility across various DP settings. Our work contributes to the field of LA by offering a practical DP framework that can support researchers and practitioners in adopting DP in their works.

Title: Ingredients: Blending Custom Photos with Video Diffusion Transformers

• Authors: Zhengcong Fei, Debang Li, Di Qiu, Changqian Yu, Mingyuan Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01790
• Pdf URL: https://arxiv.org/pdf/2501.01790
• Copy Paste: [[2501.01790]] Ingredients: Blending Custom Photos with Video Diffusion Transformers(https://arxiv.org/abs/2501.01790)
• Keywords: diffusion, transformer, generative
• Abstract: This paper presents a powerful framework to customize video creations by incorporating multiple specific identity (ID) photos, with video diffusion Transformers, referred to as \texttt{Ingredients}. Generally, our method consists of three primary modules: (\textbf{i}) a facial extractor that captures versatile and precise facial features for each human ID from both global and local perspectives; (\textbf{ii}) a multi-scale projector that maps face embeddings into the contextual space of image query in video diffusion transformers; (\textbf{iii}) an ID router that dynamically combines and allocates multiple ID embedding to the corresponding space-time regions. Leveraging a meticulously curated text-video dataset and a multi-stage training protocol, \texttt{Ingredients} demonstrates superior performance in turning custom photos into dynamic and personalized video content. Qualitative evaluations highlight the advantages of proposed method, positioning it as a significant advancement toward more effective generative video control tools in Transformer-based architecture, compared to existing methods. The data, code, and model weights are publicly available at: \url{this https URL}.

Title: Creating Artificial Students that Never Existed: Leveraging Large Language Models and CTGANs for Synthetic Data Generation

• Authors: Mohammad Khalil, Farhad Vadiee, Ronas Shakya, Qinyi Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01793
• Pdf URL: https://arxiv.org/pdf/2501.01793
• Copy Paste: [[2501.01793]] Creating Artificial Students that Never Existed: Leveraging Large Language Models and CTGANs for Synthetic Data Generation(https://arxiv.org/abs/2501.01793)
• Keywords: privacy, protect, generative, large language model
• Abstract: In this study, we explore the growing potential of AI and deep learning technologies, particularly Generative Adversarial Networks (GANs) and Large Language Models (LLMs), for generating synthetic tabular data. Access to quality students data is critical for advancing learning analytics, but privacy concerns and stricter data protection regulations worldwide limit their availability and usage. Synthetic data offers a promising alternative. We investigate whether synthetic data can be leveraged to create artificial students for serving learning analytics models. Using the popular GAN model CTGAN and three LLMs- GPT2, DistilGPT2, and DialoGPT, we generate synthetic tabular student data. Our results demonstrate the strong potential of these methods to produce high-quality synthetic datasets that resemble real students data. To validate our findings, we apply a comprehensive set of utility evaluation metrics to assess the statistical and predictive performance of the synthetic data and compare the different generator models used, specially the performance of LLMs. Our study aims to provide the learning analytics community with valuable insights into the use of synthetic data, laying the groundwork for expanding the field methodological toolbox with new innovative approaches for learning analytics data generation.

Title: Reading Between the Lines: A dataset and a study on why some texts are tougher than others

• Authors: Nouran Khallaf, Carlo Eugeni, Serge Sharoff
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01796
• Pdf URL: https://arxiv.org/pdf/2501.01796
• Copy Paste: [[2501.01796]] Reading Between the Lines: A dataset and a study on why some texts are tougher than others(https://arxiv.org/abs/2501.01796)
• Keywords: transformer
• Abstract: Our research aims at better understanding what makes a text difficult to read for specific audiences with intellectual disabilities, more specifically, people who have limitations in cognitive functioning, such as reading and understanding skills, an IQ below 70, and challenges in conceptual domains. We introduce a scheme for the annotation of difficulties which is based on empirical research in psychology as well as on research in translation studies. The paper describes the annotated dataset, primarily derived from the parallel texts (standard English and Easy to Read English translations) made available online. we fine-tuned four different pre-trained transformer models to perform the task of multiclass classification to predict the strategies required for simplification. We also investigate the possibility to interpret the decisions of this language model when it is aimed at predicting the difficulty of sentences. The resources are available from this https URL

Title: End-to-End Long Document Summarization using Gradient Caching

• Authors: Rohit Saxena, Hao Tang, Frank Keller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01805
• Pdf URL: https://arxiv.org/pdf/2501.01805
• Copy Paste: [[2501.01805]] End-to-End Long Document Summarization using Gradient Caching(https://arxiv.org/abs/2501.01805)
• Keywords: transformer
• Abstract: Training transformer-based encoder-decoder models for long document summarization poses a significant challenge due to the quadratic memory consumption during training. Several approaches have been proposed to extend the input length at test time, but training with these approaches is still difficult, requiring truncation of input documents and causing a mismatch between training and test conditions. In this work, we propose CachED (Gradient $\textbf{Cach}$ing for $\textbf{E}$ncoder-$\textbf{D}$ecoder models), an approach that enables end-to-end training of existing transformer-based encoder-decoder models, using the entire document without truncation. Specifically, we apply non-overlapping sliding windows to input documents, followed by fusion in decoder. During backpropagation, the gradients are cached at the decoder and are passed through the encoder in chunks by re-computing the hidden vectors, similar to gradient checkpointing. In the experiments on long document summarization, we extend BART to CachED BART, processing more than 500K tokens during training and achieving superior performance without using any additional parameters.

Title: Uncertainty-Aware Label Refinement on Hypergraphs for Personalized Federated Facial Expression Recognition

• Authors: Hu Ding, Yan Yan, Yang Lu, Jing-Hao Xue, Hanzi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01816
• Pdf URL: https://arxiv.org/pdf/2501.01816
• Copy Paste: [[2501.01816]] Uncertainty-Aware Label Refinement on Hypergraphs for Personalized Federated Facial Expression Recognition(https://arxiv.org/abs/2501.01816)
• Keywords: privacy, robust, federate
• Abstract: Most facial expression recognition (FER) models are trained on large-scale expression data with centralized learning. Unfortunately, collecting a large amount of centralized expression data is difficult in practice due to privacy concerns of facial images. In this paper, we investigate FER under the framework of personalized federated learning, which is a valuable and practical decentralized setting for real-world applications. To this end, we develop a novel uncertainty-Aware label refineMent on hYpergraphs (AMY) method. For local training, each local model consists of a backbone, an uncertainty estimation (UE) block, and an expression classification (EC) block. In the UE block, we leverage a hypergraph to model complex high-order relationships between expression samples and incorporate these relationships into uncertainty features. A personalized uncertainty estimator is then introduced to estimate reliable uncertainty weights of samples in the local client. In the EC block, we perform label propagation on the hypergraph, obtaining high-quality refined labels for retraining an expression classifier. Based on the above, we effectively alleviate heterogeneous sample uncertainty across clients and learn a robust personalized FER model in each client. Experimental results on two challenging real-world facial expression databases show that our proposed method consistently outperforms several state-of-the-art methods. This indicates the superiority of hypergraph modeling for uncertainty estimation and label refinement on the personalized federated FER task. The source code will be released at this https URL.

Title: Rerouting LLM Routers

• Authors: Avital Shafran, Roei Schuster, Thomas Ristenpart, Vitaly Shmatikov
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01818
• Pdf URL: https://arxiv.org/pdf/2501.01818
• Copy Paste: [[2501.01818]] Rerouting LLM Routers(https://arxiv.org/abs/2501.01818)
• Keywords: defense, attack, robust
• Abstract: LLM routers aim to balance quality and cost of generation by classifying queries and routing them to a cheaper or more expensive LLM depending on their complexity. Routers represent one type of what we call LLM control planes: systems that orchestrate use of one or more LLMs. In this paper, we investigate routers' adversarial robustness. We first define LLM control plane integrity, i.e., robustness of LLM orchestration to adversarial inputs, as a distinct problem in AI safety. Next, we demonstrate that an adversary can generate query-independent token sequences we call ``confounder gadgets'' that, when added to any query, cause LLM routers to send the query to a strong LLM. Our quantitative evaluation shows that this attack is successful both in white-box and black-box settings against a variety of open-source and commercial routers, and that confounding queries do not affect the quality of LLM responses. Finally, we demonstrate that gadgets can be effective while maintaining low perplexity, thus perplexity-based filtering is not an effective defense. We finish by investigating alternative defenses.

Title: Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models

• Authors: Yanjiang Liu, Shuhen Zhou, Yaojie Lu, Huijia Zhu, Weiqiang Wang, Hongyu Lin, Ben He, Xianpei Han, Le Sun
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01830
• Pdf URL: https://arxiv.org/pdf/2501.01830
• Copy Paste: [[2501.01830]] Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models(https://arxiv.org/abs/2501.01830)
• Keywords: security, defense, attack, large language model
• Abstract: Automated red-teaming has become a crucial approach for uncovering vulnerabilities in large language models (LLMs). However, most existing methods focus on isolated safety flaws, limiting their ability to adapt to dynamic defenses and uncover complex vulnerabilities efficiently. To address this challenge, we propose Auto-RT, a reinforcement learning framework that automatically explores and optimizes complex attack strategies to effectively uncover security vulnerabilities through malicious queries. Specifically, we introduce two key mechanisms to reduce exploration complexity and improve strategy optimization: 1) Early-terminated Exploration, which accelerate exploration by focusing on high-potential attack strategies; and 2) Progressive Reward Tracking algorithm with intermediate downgrade models, which dynamically refine the search trajectory toward successful vulnerability exploitation. Extensive experiments across diverse LLMs demonstrate that, by significantly improving exploration efficiency and automatically optimizing attack strategies, Auto-RT detects a boarder range of vulnerabilities, achieving a faster detection speed and 16.63\% higher success rates compared to existing methods.

Title: Time Series Language Model for Descriptive Caption Generation

• Authors: Mohamed Trabelsi, Aidan Boyd, Jin Cao, Huseyin Uzunalioglu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01832
• Pdf URL: https://arxiv.org/pdf/2501.01832
• Copy Paste: [[2501.01832]] Time Series Language Model for Descriptive Caption Generation(https://arxiv.org/abs/2501.01832)
• Keywords: interpretability, large language model
• Abstract: The automatic generation of representative natural language descriptions for observable patterns in time series data enhances interpretability, simplifies analysis and increases cross-domain utility of temporal data. While pre-trained foundation models have made considerable progress in natural language processing (NLP) and computer vision (CV), their application to time series analysis has been hindered by data scarcity. Although several large language model (LLM)-based methods have been proposed for time series forecasting, time series captioning is under-explored in the context of LLMs. In this paper, we introduce TSLM, a novel time series language model designed specifically for time series captioning. TSLM operates as an encoder-decoder model, leveraging both text prompts and time series data representations to capture subtle temporal patterns across multiple phases and generate precise textual descriptions of time series inputs. TSLM addresses the data scarcity problem in time series captioning by first leveraging an in-context prompting synthetic data generation, and second denoising the generated data via a novel cross-modal dense retrieval scoring applied to time series-caption pairs. Experimental findings on various time series captioning datasets demonstrate that TSLM outperforms existing state-of-the-art approaches from multiple data modalities by a significant margin.

Title: MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning

• Authors: Pu Yang, Bin Dong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01834
• Pdf URL: https://arxiv.org/pdf/2501.01834
• Copy Paste: [[2501.01834]] MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning(https://arxiv.org/abs/2501.01834)
• Keywords: large language model
• Abstract: Image captioning is a critical task at the intersection of computer vision and natural language processing, with wide-ranging applications across various domains. For complex tasks such as diagnostic report generation, deep learning models require not only domain-specific image-caption datasets but also the incorporation of relevant general knowledge to provide contextual accuracy. Existing approaches exhibit inherent limitations: specialized models excel in capturing domain-specific details but lack generalization, while vision-language models (VLMs) built on large language models (LLMs) leverage general knowledge but struggle with domain-specific adaptation. To address these limitations, this paper proposes a novel agent-enhanced model collaboration framework, which we called \textbf{MoColl}, designed to effectively integrate domain-specific and general knowledge. Specifically, our approach is to decompose complex image captioning tasks into a series of interconnected question-answer subtasks. A trainable visual question answering (VQA) model is employed as a specialized tool to focus on domain-specific visual analysis, answering task-specific questions based on image content. Concurrently, an LLM-based agent with general knowledge formulates these questions and synthesizes the resulting question-answer pairs into coherent captions. Beyond its role in leveraging the VQA model, the agent further guides its training to enhance its domain-specific capabilities. Experimental results on radiology report generation validate the effectiveness of the proposed framework, demonstrating significant improvements in the quality of generated reports.

Title: Dedicated Inference Engine and Binary-Weight Neural Networks for Lightweight Instance Segmentation

• Authors: Tse-Wei Chen, Wei Tao, Dongyue Zhao, Kazuhiro Mima, Tadayuki Ito, Kinya Osa, Masami Kato
• Subjects: cs.CV, cs.AR
• Abstract URL: https://arxiv.org/abs/2501.01841
• Pdf URL: https://arxiv.org/pdf/2501.01841
• Copy Paste: [[2501.01841]] Dedicated Inference Engine and Binary-Weight Neural Networks for Lightweight Instance Segmentation(https://arxiv.org/abs/2501.01841)
• Keywords: segmentation
• Abstract: Reducing computational costs is an important issue for development of embedded systems. Binary-weight Neural Networks (BNNs), in which weights are binarized and activations are quantized, are employed to reduce computational costs of various kinds of applications. In this paper, a design methodology of hardware architecture for inference engines is proposed to handle modern BNNs with two operation modes. Multiply-Accumulate (MAC) operations can be simplified by replacing multiply operations with bitwise operations. The proposed method can effectively reduce the gate count of inference engines by removing a part of computational costs from the hardware system. The architecture of MAC operations can calculate the inference results of BNNs efficiently with only 52% of hardware costs compared with the related works. To show that the inference engine can handle practical applications, two lightweight networks which combine the backbones of SegNeXt and the decoder of SparseInst for instance segmentation are also proposed. The output results of the lightweight networks are computed using only bitwise operations and add operations. The proposed inference engine has lower hardware costs than related works. The experimental results show that the proposed inference engine can handle the proposed instance-segmentation networks and achieves higher accuracy than YOLACT on the "Person" category although the model size is 77.7$\times$ smaller compared with YOLACT.

Title: Semantic Segmentation for Sequential Historical Maps by Learning from Only One Map

• Authors: Yunshuang Yuan, Frank Thiemann, Monika Sester
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01845
• Pdf URL: https://arxiv.org/pdf/2501.01845
• Copy Paste: [[2501.01845]] Semantic Segmentation for Sequential Historical Maps by Learning from Only One Map(https://arxiv.org/abs/2501.01845)
• Keywords: segmentation
• Abstract: Historical maps are valuable resources that capture detailed geographical information from the past. However, these maps are typically available in printed formats, which are not conducive to modern computer-based analyses. Digitizing these maps into a machine-readable format enables efficient computational analysis. In this paper, we propose an automated approach to digitization using deep-learning-based semantic segmentation, which assigns a semantic label to each pixel in scanned historical maps. A key challenge in this process is the lack of ground-truth annotations required for training deep neural networks, as manual labeling is time-consuming and labor-intensive. To address this issue, we introduce a weakly-supervised age-tracing strategy for model fine-tuning. This approach exploits the similarity in appearance and land-use patterns between historical maps from neighboring time periods to guide the training process. Specifically, model predictions for one map are utilized as pseudo-labels for training on maps from adjacent time periods. Experiments conducted on our newly curated \textit{Hameln} dataset demonstrate that the proposed age-tracing strategy significantly enhances segmentation performance compared to baseline models. In the best-case scenario, the mean Intersection over Union (mIoU) achieved 77.3\%, reflecting an improvement of approximately 20\% over baseline methods. Additionally, the fine-tuned model achieved an average overall accuracy of 97\%, highlighting the effectiveness of our approach for digitizing historical maps.

Title: LCFed: An Efficient Clustered Federated Learning Framework for Heterogeneous Data

• Authors: Yuxin Zhang, Haoyu Chen, Zheng Lin, Zhe Chen, Jin Zhao
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.01850
• Pdf URL: https://arxiv.org/pdf/2501.01850
• Copy Paste: [[2501.01850]] LCFed: An Efficient Clustered Federated Learning Framework for Heterogeneous Data(https://arxiv.org/abs/2501.01850)
• Keywords: federate
• Abstract: Clustered federated learning (CFL) addresses the performance challenges posed by data heterogeneity in federated learning (FL) by organizing edge devices with similar data distributions into clusters, enabling collaborative model training tailored to each group. However, existing CFL approaches strictly limit knowledge sharing to within clusters, lacking the integration of global knowledge with intra-cluster training, which leads to suboptimal performance. Moreover, traditional clustering methods incur significant computational overhead, especially as the number of edge devices increases. In this paper, we propose LCFed, an efficient CFL framework to combat these challenges. By leveraging model partitioning and adopting distinct aggregation strategies for each sub-model, LCFed effectively incorporates global knowledge into intra-cluster co-training, achieving optimal training performance. Additionally, LCFed customizes a computationally efficient model similarity measurement method based on low-rank models, enabling real-time cluster updates with minimal computational overhead. Extensive experiments show that LCFed outperforms state-of-the-art benchmarks in both test accuracy and clustering computational efficiency.

Title: UAV-DETR: Efficient End-to-End Object Detection for Unmanned Aerial Vehicle Imagery

• Authors: Huaxiang Zhang, Kai Liu, Zhongxue Gan, Guo-Niu Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01855
• Pdf URL: https://arxiv.org/pdf/2501.01855
• Copy Paste: [[2501.01855]] UAV-DETR: Efficient End-to-End Object Detection for Unmanned Aerial Vehicle Imagery(https://arxiv.org/abs/2501.01855)
• Keywords: transformer
• Abstract: Unmanned aerial vehicle object detection (UAV-OD) has been widely used in various scenarios. However, most existing UAV-OD algorithms rely on manually designed components, which require extensive tuning. End-to-end models that do not depend on such manually designed components are mainly designed for natural images, which are less effective for UAV imagery. To address such challenges, this paper proposes an efficient detection transformer (DETR) framework tailored for UAV imagery, i.e., UAV-DETR. The framework includes a multi-scale feature fusion with frequency enhancement module, which captures both spatial and frequency information at different scales. In addition, a frequency-focused down-sampling module is presented to retain critical spatial details during down-sampling. A semantic alignment and calibration module is developed to align and fuse features from different fusion paths. Experimental results demonstrate the effectiveness and generalization of our approach across various UAV imagery datasets. On the VisDrone dataset, our method improves AP by 3.1\% and $\text{AP}_{50}$ by 4.2\% over the baseline. Similar enhancements are observed on the UAVVaste dataset. The project page: this https URL

Title: Towards Hard and Soft Shadow Removal via Dual-Branch Separation Network and Vision Transformer

• Authors: Jiajia Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01864
• Pdf URL: https://arxiv.org/pdf/2501.01864
• Copy Paste: [[2501.01864]] Towards Hard and Soft Shadow Removal via Dual-Branch Separation Network and Vision Transformer(https://arxiv.org/abs/2501.01864)
• Keywords: transformer
• Abstract: Image shadow removal is a crucial task in computer vision. In real-world scenes, shadows alter image color and brightness, posing challenges for perception and texture recognition. Traditional and deep learning methods often overlook the distinct needs for handling hard and soft shadows, thereby lacking detailed processing to specifically address each type of shadow in this http URL propose a dual-path model that processes these shadows separately using specially designed loss functions to accomplish the hard and soft shadow removal. The model classifies shadow types and processes them through appropriate paths to produce shadow-free outputs, integrating a Vision Transformer with UNet++ for enhanced edge detail and feature fusion. Our model outperforms state-of-the-art methods and achieves 2.905 RMSE value on the ISTD dataset, which demonstrates greater effectiveness than typical single-path approaches.

Title: Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions

• Authors: Rachneet Sachdeva, Rima Hazra, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01872
• Pdf URL: https://arxiv.org/pdf/2501.01872
• Copy Paste: [[2501.01872]] Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions(https://arxiv.org/abs/2501.01872)
• Keywords: defense, attack, robust, large language model
• Abstract: Despite significant efforts to align large language models with human values and ethical guidelines, these models remain susceptible to sophisticated jailbreak attacks that exploit their reasoning capabilities. Traditional safety mechanisms often focus on detecting explicit malicious intent, leaving deeper vulnerabilities unaddressed. In this work, we introduce a jailbreak technique, POATE (Polar Opposite query generation, Adversarial Template construction, and Elaboration), which leverages contrastive reasoning to elicit unethical responses. POATE generates prompts with semantically opposite intents and combines them with adversarial templates to subtly direct models toward producing harmful responses. We conduct extensive evaluations across six diverse language model families of varying parameter sizes, including LLaMA3, Gemma2, Phi3, and GPT-4, to demonstrate the robustness of the attack, achieving significantly higher attack success rates (~44%) compared to existing methods. We evaluate our proposed attack against seven safety defenses, revealing their limitations in addressing reasoning-based vulnerabilities. To counteract this, we propose a defense strategy that improves reasoning robustness through chain-of-thought prompting and reverse thinking, mitigating reasoning-driven adversarial exploits.

Title: DFF: Decision-Focused Fine-tuning for Smarter Predict-then-Optimize with Limited Data

• Authors: Jiaqi Yang, Enming Liang, Zicheng Su, Zhichao Zou, Peng Zhen, Jiecheng Guo, Wanjing Ma, Kun An
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01874
• Pdf URL: https://arxiv.org/pdf/2501.01874
• Copy Paste: [[2501.01874]] DFF: Decision-Focused Fine-tuning for Smarter Predict-then-Optimize with Limited Data(https://arxiv.org/abs/2501.01874)
• Keywords: robust
• Abstract: Decision-focused learning (DFL) offers an end-to-end approach to the predict-then-optimize (PO) framework by training predictive models directly on decision loss (DL), enhancing decision-making performance within PO contexts. However, the implementation of DFL poses distinct challenges. Primarily, DL can result in deviation from the physical significance of the predictions under limited data. Additionally, some predictive models are non-differentiable or black-box, which cannot be adjusted using gradient-based methods. To tackle the above challenges, we propose a novel framework, Decision-Focused Fine-tuning (DFF), which embeds the DFL module into the PO pipeline via a novel bias correction module. DFF is formulated as a constrained optimization problem that maintains the proximity of the DL-enhanced model to the original predictive model within a defined trust region. We theoretically prove that DFF strictly confines prediction bias within a predetermined upper bound, even with limited datasets, thereby substantially reducing prediction shifts caused by DL under limited data. Furthermore, the bias correction module can be integrated into diverse predictive models, enhancing adaptability to a broad range of PO tasks. Extensive evaluations on synthetic and real-world datasets, including network flow, portfolio optimization, and resource allocation problems with different predictive models, demonstrate that DFF not only improves decision performance but also adheres to fine-tuning constraints, showcasing robust adaptability across various scenarios.

Title: Exploring Equality: An Investigation into Custom Loss Functions for Fairness Definitions

• Authors: Gordon Lee, Simeon Sayer
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.01889
• Pdf URL: https://arxiv.org/pdf/2501.01889
• Copy Paste: [[2501.01889]] Exploring Equality: An Investigation into Custom Loss Functions for Fairness Definitions(https://arxiv.org/abs/2501.01889)
• Keywords: fair
• Abstract: This paper explores the complex tradeoffs between various fairness metrics such as equalized odds, disparate impact, and equal opportunity and predictive accuracy within COMPAS by building neural networks trained with custom loss functions optimized to specific fairness criteria. This paper creates the first fairness-driven implementation of the novel Group Accuracy Parity (GAP) framework, as theoretically proposed by Gupta et al. (2024), and applies it to COMPAS. To operationalize and accurately compare the fairness of COMPAS models optimized to differing fairness ideals, this paper develops and proposes a combinatory analytical procedure that incorporates Pareto front and multivariate analysis, leveraging data visualizations such as violin graphs. This paper concludes that GAP achieves an enhanced equilibrium between fairness and accuracy compared to COMPAS's current nationwide implementation and alternative implementations of COMPAS optimized to more traditional fairness definitions. While this paper's algorithmic improvements of COMPAS significantly augment its fairness, external biases undermine the fairness of its implementation. Practices such as predictive policing and issues such as the lack of transparency regarding COMPAS's internal workings have contributed to the algorithm's historical injustice. In conjunction with developments regarding COMPAS's predictive methodology, legal and institutional changes must happen for COMPAS's just deployment.

Title: Virgo: A Preliminary Exploration on Reproducing o1-like MLLM

• Authors: Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, Ji-Rong Wen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01904
• Pdf URL: https://arxiv.org/pdf/2501.01904
• Copy Paste: [[2501.01904]] Virgo: A Preliminary Exploration on Reproducing o1-like MLLM(https://arxiv.org/abs/2501.01904)
• Keywords: large language model
• Abstract: Recently, slow-thinking reasoning systems, built upon large language models (LLMs), have garnered widespread attention by scaling the thinking time during inference. There is also growing interest in adapting this capability to multimodal large language models (MLLMs). Given that MLLMs handle more complex data semantics across different modalities, it is intuitively more challenging to implement multimodal slow-thinking systems. To address this issue, in this paper, we explore a straightforward approach by fine-tuning a capable MLLM with a small amount of textual long-form thought data, resulting in a multimodal slow-thinking system, Virgo (Visual reasoning with long thought). We find that these long-form reasoning processes, expressed in natural language, can be effectively transferred to MLLMs. Moreover, it seems that such textual reasoning data can be even more effective than visual reasoning data in eliciting the slow-thinking capacities of MLLMs. While this work is preliminary, it demonstrates that slow-thinking capacities are fundamentally associated with the language model component, which can be transferred across modalities or domains. This finding can be leveraged to guide the development of more powerful slow-thinking reasoning systems. We release our resources at this https URL.

Title: Detecting and Mitigating Adversarial Attacks on Deep Learning-Based MRI Reconstruction Without Any Retraining

• Authors: Mahdi Saberi, Chi Zhang, Mehmet Akcakaya
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01908
• Pdf URL: https://arxiv.org/pdf/2501.01908
• Copy Paste: [[2501.01908]] Detecting and Mitigating Adversarial Attacks on Deep Learning-Based MRI Reconstruction Without Any Retraining(https://arxiv.org/abs/2501.01908)
• Keywords: attack
• Abstract: Deep learning (DL) methods, especially those based on physics-driven DL, have become the state-of-the-art for reconstructing sub-sampled magnetic resonance imaging (MRI) data. However, studies have shown that these methods are susceptible to small adversarial input perturbations, or attacks, resulting in major distortions in the output images. Various strategies have been proposed to reduce the effects of these attacks, but they require retraining and may lower reconstruction quality for non-perturbed/clean inputs. In this work, we propose a novel approach for detecting and mitigating adversarial attacks on MRI reconstruction models without any retraining. Our detection strategy is based on the idea of cyclic measurement consistency. The output of the model is mapped to another set of MRI measurements for a different sub-sampling pattern, and this synthesized data is reconstructed with the same model. Intuitively, without an attack, the second reconstruction is expected to be consistent with the first, while with an attack, disruptions are present. Subsequently, this idea is extended to devise a novel objective function, which is minimized within a small ball around the attack input for mitigation. Experimental results show that our method substantially reduces the impact of adversarial perturbations across different datasets, attack types/strengths and PD-DL networks, and qualitatively and quantitatively outperforms conventional mitigation methods that involve retraining.

Title: Mingling with the Good to Backdoor Federated Learning

• Authors: Nuno Neves
• Subjects: cs.CR, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.01913
• Pdf URL: https://arxiv.org/pdf/2501.01913
• Copy Paste: [[2501.01913]] Mingling with the Good to Backdoor Federated Learning(https://arxiv.org/abs/2501.01913)
• Keywords: security, privacy, protect, defense, attack, federate
• Abstract: Federated learning (FL) is a decentralized machine learning technique that allows multiple entities to jointly train a model while preserving dataset privacy. However, its distributed nature has raised various security concerns, which have been addressed by increasingly sophisticated defenses. These protections utilize a range of data sources and metrics to, for example, filter out malicious model updates, ensuring that the impact of attacks is minimized or eliminated. This paper explores the feasibility of designing a generic attack method capable of installing backdoors in FL while evading a diverse array of defenses. Specifically, we focus on an attacker strategy called MIGO, which aims to produce model updates that subtly blend with legitimate ones. The resulting effect is a gradual integration of a backdoor into the global model, often ensuring its persistence long after the attack concludes, while generating enough ambiguity to hinder the effectiveness of defenses. MIGO was employed to implant three types of backdoors across five datasets and different model architectures. The results demonstrate the significant threat posed by these backdoors, as MIGO consistently achieved exceptionally high backdoor accuracy (exceeding 90%) while maintaining the utility of the main task. Moreover, MIGO exhibited strong evasion capabilities against ten defenses, including several state-of-the-art methods. When compared to four other attack strategies, MIGO consistently outperformed them across most configurations. Notably, even in extreme scenarios where the attacker controls just 0.1% of the clients, the results indicate that successful backdoor insertion is possible if the attacker can persist for a sufficient number of rounds.

Title: Transformer-Driven Inverse Problem Transform for Fast Blind Hyperspectral Image Dehazing

• Authors: Po-Wei Tang, Chia-Hsiang Lin, Yangrui Liu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.01924
• Pdf URL: https://arxiv.org/pdf/2501.01924
• Copy Paste: [[2501.01924]] Transformer-Driven Inverse Problem Transform for Fast Blind Hyperspectral Image Dehazing(https://arxiv.org/abs/2501.01924)
• Keywords: transformer
• Abstract: Hyperspectral dehazing (HyDHZ) has become a crucial signal processing technology to facilitate the subsequent identification and classification tasks, as the airborne visible/infrared imaging spectrometer (AVIRIS) data portal reports a massive portion of haze-corrupted areas in typical hyperspectral remote sensing images. The idea of inverse problem transform (IPT) has been proposed in recent remote sensing literature in order to reformulate a hardly tractable inverse problem (e.g., HyDHZ) into a relatively simple one. Considering the emerging spectral super-resolution (SSR) technique, which spectrally upsamples multispectral data to hyperspectral data, we aim to solve the challenging HyDHZ problem by reformulating it as an SSR problem. Roughly speaking, the proposed algorithm first automatically selects some uncorrupted/informative spectral bands, from which SSR is applied to spectrally upsample the selected bands in the feature space, thereby obtaining a clean hyperspectral image (HSI). The clean HSI is then further refined by a deep transformer network to obtain the final dehazed HSI, where a global attention mechanism is designed to capture nonlocal information. There are very few HyDHZ works in existing literature, and this article introduces the powerful spatial-spectral transformer into HyDHZ for the first time. Remarkably, the proposed transformer-driven IPT-based HyDHZ (T2HyDHZ) is a blind algorithm without requiring the user to manually select the corrupted region. Extensive experiments demonstrate the superiority of T2HyDHZ with less color distortion.

Title: Bridging Classification and Segmentation in Osteosarcoma Assessment via Foundation and Discrete Diffusion Models

• Authors: Manh Duong Nguyen, Dac Thai Nguyen, Trung Viet Nguyen, Homi Yamada, Huy Hieu Pham, Phi Le Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01932
• Pdf URL: https://arxiv.org/pdf/2501.01932
• Copy Paste: [[2501.01932]] Bridging Classification and Segmentation in Osteosarcoma Assessment via Foundation and Discrete Diffusion Models(https://arxiv.org/abs/2501.01932)
• Keywords: diffusion, segmentation
• Abstract: Osteosarcoma, the most common primary bone cancer, often requires accurate necrosis assessment from whole slide images (WSIs) for effective treatment planning and prognosis. However, manual assessments are subjective and prone to variability. In response, we introduce FDDM, a novel framework bridging the gap between patch classification and region-based segmentation. FDDM operates in two stages: patch-based classification, followed by region-based refinement, enabling cross-patch information intergation. Leveraging a newly curated dataset of osteosarcoma images, FDDM demonstrates superior segmentation performance, achieving up to a 10% improvement mIOU and a 32.12% enhancement in necrosis rate estimation over state-of-the-art methods. This framework sets a new benchmark in osteosarcoma assessment, highlighting the potential of foundation models and diffusion-based refinements in complex medical imaging tasks.

Title: Fusion DeepONet: A Data-Efficient Neural Operator for Geometry-Dependent Hypersonic Flows on Arbitrary Grids

• Authors: Ahmad Peyvan, Varun Kumar
• Subjects: cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2501.01934
• Pdf URL: https://arxiv.org/pdf/2501.01934
• Copy Paste: [[2501.01934]] Fusion DeepONet: A Data-Efficient Neural Operator for Geometry-Dependent Hypersonic Flows on Arbitrary Grids(https://arxiv.org/abs/2501.01934)
• Keywords: robust
• Abstract: Designing re-entry vehicles requires accurate predictions of hypersonic flow around their geometry. Rapid prediction of such flows can revolutionize vehicle design, particularly for morphing geometries. We evaluate advanced neural operator models such as Deep Operator Networks (DeepONet), parameter-conditioned U-Net, Fourier Neural Operator (FNO), and MeshGraphNet, with the objective of addressing the challenge of learning geometry-dependent hypersonic flow fields with limited data. Specifically, we compare the performance of these models for two grid types: uniform Cartesian and irregular grids. To train these models, we use 36 unique elliptic geometries for generating high-fidelity simulations with a high-order entropy-stable DGSEM solver, emphasizing the challenge of working with a scarce dataset. We evaluate and compare the four operator-based models for their efficacy in predicting hypersonic flow field around the elliptic body. Moreover, we develop a novel framework, called Fusion DeepONet, which leverages neural field concepts and generalizes effectively across varying geometries. Despite the scarcity of training data, Fusion DeepONet achieves performance comparable to parameter-conditioned U-Net on uniform grids while it outperforms MeshGraphNet and vanilla DeepONet on irregular, arbitrary grids. Fusion DeepONet requires significantly fewer trainable parameters as compared to U-Net, MeshGraphNet, and FNO, making it computationally efficient. We also analyze the basis functions of the Fusion DeepONet model using Singular Value Decomposition. This analysis reveals that Fusion DeepONet generalizes effectively to unseen solutions and adapts to varying geometries and grid points, demonstrating its robustness in scenarios with limited training data.

Title: VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment

• Authors: Wenyan Cong, Kevin Wang, Jiahui Lei, Colton Stearns, Yuanhao Cai, Dilin Wang, Rakesh Ranjan, Matt Feiszli, Leonidas Guibas, Zhangyang Wang, Weiyao Wang, Zhiwen Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01949
• Pdf URL: https://arxiv.org/pdf/2501.01949
• Copy Paste: [[2501.01949]] VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment(https://arxiv.org/abs/2501.01949)
• Keywords: robust
• Abstract: Efficiently reconstructing accurate 3D models from monocular video is a key challenge in computer vision, critical for advancing applications in virtual reality, robotics, and scene understanding. Existing approaches typically require pre-computed camera parameters and frame-by-frame reconstruction pipelines, which are prone to error accumulation and entail significant computational overhead. To address these limitations, we introduce VideoLifter, a novel framework that leverages geometric priors from a learnable model to incrementally optimize a globally sparse to dense 3D representation directly from video sequences. VideoLifter segments the video sequence into local windows, where it matches and registers frames, constructs consistent fragments, and aligns them hierarchically to produce a unified 3D model. By tracking and propagating sparse point correspondences across frames and fragments, VideoLifter incrementally refines camera poses and 3D structure, minimizing reprojection error for improved accuracy and robustness. This approach significantly accelerates the reconstruction process, reducing training time by over 82% while surpassing current state-of-the-art methods in visual fidelity and computational efficiency.

Title: MADGEN -- Mass-Spec attends to De Novo Molecular generation

• Authors: Yinkai Wang, Xiaohui Chen, Liping Liu, Soha Hassoun
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01950
• Pdf URL: https://arxiv.org/pdf/2501.01950
• Copy Paste: [[2501.01950]] MADGEN -- Mass-Spec attends to De Novo Molecular generation(https://arxiv.org/abs/2501.01950)
• Keywords: generative
• Abstract: The annotation (assigning structural chemical identities) of MS/MS spectra remains a significant challenge due to the enormous molecular diversity in biological samples and the limited scope of reference databases. Currently, the vast majority of spectral measurements remain in the "dark chemical space" without structural annotations. To improve annotation, we propose MADGEN (Mass-spec Attends to De Novo Molecular GENeration), a scaffold-based method for de novo molecular structure generation guided by mass spectrometry data. MADGEN operates in two stages: scaffold retrieval and spectra-conditioned molecular generation starting with the scaffold. In the first stage, given an MS/MS spectrum, we formulate scaffold retrieval as a ranking problem and employ contrastive learning to align mass spectra with candidate molecular scaffolds. In the second stage, starting from the retrieved scaffold, we employ the MS/MS spectrum to guide an attention-based generative model to generate the final molecule. Our approach constrains the molecular generation search space, reducing its complexity and improving generation accuracy. We evaluate MADGEN on three datasets (NIST23, CANOPUS, and MassSpecGym) and evaluate MADGEN's performance with a predictive scaffold retriever and with an oracle retriever. We demonstrate the effectiveness of using attention to integrate spectral information throughout the generation process to achieve strong results with the oracle retriever.

Title: VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

• Authors: Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, Ran He
• Subjects: cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.01957
• Pdf URL: https://arxiv.org/pdf/2501.01957
• Copy Paste: [[2501.01957]] VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction(https://arxiv.org/abs/2501.01957)
• Keywords: large language model
• Abstract: Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction. However, speech plays a crucial role in multimodal dialogue systems, and implementing high-performance in both vision and speech tasks remains a significant challenge due to the fundamental modality differences. In this paper, we propose a carefully designed multi-stage training methodology that progressively trains LLM to understand both visual and speech information, ultimately enabling fluent vision and speech interaction. Our approach not only preserves strong vision-language capacity, but also enables efficient speech-to-speech dialogue capabilities without separate ASR and TTS modules, significantly accelerating multimodal end-to-end response speed. By comparing our method against state-of-the-art counterparts across benchmarks for image, video, and speech tasks, we demonstrate that our model is equipped with both strong visual and speech capabilities, making near real-time vision and speech interaction.