2024-11-18

Title: Early-Scheduled Handover Preparation in 5G NR Millimeter-Wave Systems

• Authors: Dino Pjanić, Alexandros Sopasakis, Andres Reial, Fredrik Tufvesson
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2411.09720
• Pdf URL: https://arxiv.org/pdf/2411.09720
• Copy Paste: [[2411.09720]] Early-Scheduled Handover Preparation in 5G NR Millimeter-Wave Systems(https://arxiv.org/abs/2411.09720)
• Keywords: robust
• Abstract: The handover (HO) procedure is one of the most critical functions in a cellular network driven by measurements of the user channel of the serving and neighboring cells. The success rate of the entire HO procedure is significantly affected by the preparation stage. As massive Multiple-Input Multiple-Output (MIMO) systems with large antenna arrays allow resolving finer details of channel behavior, we investigate how machine learning can be applied to time series data of beam measurements in the Fifth Generation (5G) New Radio (NR) system to improve the HO procedure. This paper introduces the Early-Scheduled Handover Preparation scheme designed to enhance the robustness and efficiency of the HO procedure, particularly in scenarios involving high mobility and dense small cell deployments. Early-Scheduled Handover Preparation focuses on optimizing the timing of the HO preparation phase by leveraging machine learning techniques to predict the earliest possible trigger points for HO events. We identify a new early trigger for HO preparation and demonstrate how it can beneficially reduce the required time for HO execution reducing channel quality degradation. These insights enable a new HO preparation scheme that offers a novel, user-aware, and proactive HO decision making in MIMO scenarios incorporating mobility.

Title: SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation

• Authors: Mikhail Khodak, Lester Mackey, Alexandra Chouldechova, Miroslav Dudík
• Subjects: cs.LG, cs.AI, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.09730
• Pdf URL: https://arxiv.org/pdf/2411.09730
• Copy Paste: [[2411.09730]] SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation(https://arxiv.org/abs/2411.09730)
• Keywords: fair
• Abstract: Disaggregated evaluation -- estimation of performance of a machine learning model on different subpopulations -- is a core task when assessing performance and group-fairness of AI systems. A key challenge is that evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny. Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually. This gives rise to what we call the multi-task disaggregated evaluation problem, wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task). In this work we develop a disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models. SureMap's efficiency gains come from (1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the AI system creator or from their other clients. Our method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE). We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.

Title: Adversarial Attacks Using Differentiable Rendering: A Survey

• Authors: Matthew Hull, Chao Zhang, Zsolt Kira, Duen Horng Chau
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09749
• Pdf URL: https://arxiv.org/pdf/2411.09749
• Copy Paste: [[2411.09749]] Adversarial Attacks Using Differentiable Rendering: A Survey(https://arxiv.org/abs/2411.09749)
• Keywords: attack
• Abstract: Differentiable rendering methods have emerged as a promising means for generating photo-realistic and physically plausible adversarial attacks by manipulating 3D objects and scenes that can deceive deep neural networks (DNNs). Recently, differentiable rendering capabilities have evolved significantly into a diverse landscape of libraries, such as Mitsuba, PyTorch3D, and methods like Neural Radiance Fields and 3D Gaussian Splatting for solving inverse rendering problems that share conceptually similar properties commonly used to attack DNNs, such as back-propagation and optimization. However, the adversarial machine learning research community has not yet fully explored or understood such capabilities for generating attacks. Some key reasons are that researchers often have different attack goals, such as misclassification or misdetection, and use different tasks to accomplish these goals by manipulating different representation in a scene, such as the mesh or texture of an object. This survey adopts a task-oriented unifying framework that systematically summarizes common tasks, such as manipulating textures, altering illumination, and modifying 3D meshes to exploit vulnerabilities in DNNs. Our framework enables easy comparison of existing works, reveals research gaps and spotlights exciting future research directions in this rapidly evolving field. Through focusing on how these tasks enable attacks on various DNNs such as image classification, facial recognition, object detection, optical flow and depth estimation, our survey helps researchers and practitioners better understand the vulnerabilities of computer vision systems against photorealistic adversarial attacks that could threaten real-world applications.

Title: Partial Multi-View Clustering via Meta-Learning and Contrastive Feature Alignment

• Authors: BoHao Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09758
• Pdf URL: https://arxiv.org/pdf/2411.09758
• Copy Paste: [[2411.09758]] Partial Multi-View Clustering via Meta-Learning and Contrastive Feature Alignment(https://arxiv.org/abs/2411.09758)
• Keywords: transformer
• Abstract: Partial multi-view clustering (PVC) presents significant challenges practical research problem for data analysis in real-world applications, especially when some views of the data are partially missing. Existing clustering methods struggle to handle incomplete views effectively, leading to suboptimal clustering performance. In this paper, we propose a novel dual optimization framework based on contrastive learning, which aims to maximize the consistency of latent features in incomplete multi-view data and improve clustering performance through deep learning models. By combining a fine-tuned Vision Transformer and k-nearest neighbors (KNN), we fill in missing views and dynamically adjust view weights using self-supervised learning and meta-learning. Experimental results demonstrate that our framework outperforms state-of-the-art clustering models on the BDGP and HW datasets, particularly in handling complex and incomplete multi-view data.

Title: NACNet: A Histology Context-aware Transformer Graph Convolution Network for Predicting Treatment Response to Neoadjuvant Chemotherapy in Triple Negative Breast Cancer

• Authors: Qiang Li, George Teodoro, Yi Jiang, Jun Kong
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.09766
• Pdf URL: https://arxiv.org/pdf/2411.09766
• Copy Paste: [[2411.09766]] NACNet: A Histology Context-aware Transformer Graph Convolution Network for Predicting Treatment Response to Neoadjuvant Chemotherapy in Triple Negative Breast Cancer(https://arxiv.org/abs/2411.09766)
• Keywords: transformer
• Abstract: Neoadjuvant chemotherapy (NAC) response prediction for triple negative breast cancer (TNBC) patients is a challenging task clinically as it requires understanding complex histology interactions within the tumor microenvironment (TME). Digital whole slide images (WSIs) capture detailed tissue information, but their giga-pixel size necessitates computational methods based on multiple instance learning, which typically analyze small, isolated image tiles without the spatial context of the TME. To address this limitation and incorporate TME spatial histology interactions in predicting NAC response for TNBC patients, we developed a histology context-aware transformer graph convolution network (NACNet). Our deep learning method identifies the histopathological labels on individual image tiles from WSIs, constructs a spatial TME graph, and represents each node with features derived from tissue texture and social network analysis. It predicts NAC response using a transformer graph convolution network model enhanced with graph isomorphism network layers. We evaluate our method with WSIs of a cohort of TNBC patient (N=105) and compared its performance with multiple state-of-the-art machine learning and deep learning models, including both graph and non-graph approaches. Our NACNet achieves 90.0% accuracy, 96.0% sensitivity, 88.0% specificity, and an AUC of 0.82, through eight-fold cross-validation, outperforming baseline models. These comprehensive experimental results suggest that NACNet holds strong potential for stratifying TNBC patients by NAC response, thereby helping to prevent overtreatment, improve patient quality of life, reduce treatment cost, and enhance clinical outcomes, marking an important advancement toward personalized breast cancer treatment.

Title: Misbinding Raw Public Keys to Identities in TLS

• Authors: Mariam Moustafa, Mohit Sethi, Tuomas Aura
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09770
• Pdf URL: https://arxiv.org/pdf/2411.09770
• Copy Paste: [[2411.09770]] Misbinding Raw Public Keys to Identities in TLS(https://arxiv.org/abs/2411.09770)
• Keywords: security, protect, attack
• Abstract: The adoption of security protocols such as Transport Layer Security (TLS) has significantly improved the state of traffic encryption and integrity protection on the Internet. Despite rigorous analysis, vulnerabilities continue to emerge, sometimes due to fundamental flaws in the protocol specification. This paper examines the security of TLS when using Raw Public Key (RPK) authentication. This mode has not been as extensively studied as X.509 certificates and Pre-Shared Keys (PSK). We develop a formal model of TLS RPK using applied pi calculus and the ProVerif verification tool, revealing that the RPK mode is susceptible to identity misbinding attacks. Our contributions include formal models of TLS RPK with several mechanisms for binding the endpoint identity to its public key, verification results, practical scenarios demonstrating the misbinding attack, and recommendations for mitigating such vulnerabilities. These findings highlight the need for improved security measures in TLS RPK.

Title: Beyond Static Tools: Evaluating Large Language Models for Cryptographic Misuse Detection

• Authors: Zohaib Masood (1), Miguel Vargas Martin (1) ((1) Ontario Tech University)
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09772
• Pdf URL: https://arxiv.org/pdf/2411.09772
• Copy Paste: [[2411.09772]] Beyond Static Tools: Evaluating Large Language Models for Cryptographic Misuse Detection(https://arxiv.org/abs/2411.09772)
• Keywords: secure, security, large language model
• Abstract: The use of Large Language Models (LLMs) in software development is rapidly growing, with developers increasingly relying on these models for coding assistance, including security-critical tasks. Our work presents a comprehensive comparison between traditional static analysis tools for cryptographic API misuse detection-CryptoGuard, CogniCrypt, and Snyk Code-and the LLMs-GPT and Gemini. Using benchmark datasets (OWASP, CryptoAPI, and MASC), we evaluate the effectiveness of each tool in identifying cryptographic misuses. Our findings show that GPT 4-o-mini surpasses current state-of-the-art static analysis tools on the CryptoAPI and MASC datasets, though it lags on the OWASP dataset. Additionally, we assess the quality of LLM responses to determine which models provide actionable and accurate advice, giving developers insights into their practical utility for secure coding. This study highlights the comparative strengths and limitations of static analysis versus LLM-driven approaches, offering valuable insights into the evolving role of AI in advancing software security practices.

Title: Combining Machine Learning Defenses without Conflicts

• Authors: Vasisht Duddu, Rui Zhang, N. Asokan
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09776
• Pdf URL: https://arxiv.org/pdf/2411.09776
• Copy Paste: [[2411.09776]] Combining Machine Learning Defenses without Conflicts(https://arxiv.org/abs/2411.09776)
• Keywords: security, privacy, protect, defense, fair
• Abstract: Machine learning (ML) defenses protect against various risks to security, privacy, and fairness. Real-life models need simultaneous protection against multiple different risks which necessitates combining multiple defenses. But combining defenses with conflicting interactions in an ML model can be ineffective, incurring a significant drop in the effectiveness of one or more defenses being combined. Practitioners need a way to determine if a given combination can be effective. Experimentally identifying effective combinations can be time-consuming and expensive, particularly when multiple defenses need to be combined. We need an inexpensive, easy-to-use combination technique to identify effective combinations. Ideally, a combination technique should be (a) accurate (correctly identifies whether a combination is effective or not), (b) scalable (allows combining multiple defenses), (c) non-invasive (requires no change to the defenses being combined), and (d) general (is applicable to different types of defenses). Prior works have identified several ad-hoc techniques but none satisfy all the requirements above. We propose a principled combination technique, Def\Con, to identify effective defense combinations. Def\Con meets all requirements, achieving 90% accuracy on eight combinations explored in prior work and 81% in 30 previously unexplored combinations that we empirically evaluate in this paper.

Title: Modeling human decomposition: a Bayesian approach

• Authors: D. Hudson Smith, Noah Nisbet, Carl Ehrett, Cristina I. Tica, Madeline M. Atwell, Katherine E. Weisensee
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09802
• Pdf URL: https://arxiv.org/pdf/2411.09802
• Copy Paste: [[2411.09802]] Modeling human decomposition: a Bayesian approach(https://arxiv.org/abs/2411.09802)
• Keywords: generative
• Abstract: Environmental and individualistic variables affect the rate of human decomposition in complex ways. These effects complicate the estimation of the postmortem interval (PMI) based on observed decomposition characteristics. In this work, we develop a generative probabilistic model for decomposing human remains based on PMI and a wide range of environmental and individualistic variables. This model explicitly represents the effect of each variable, including PMI, on the appearance of each decomposition characteristic, allowing for direct interpretation of model effects and enabling the use of the model for PMI inference and optimal experimental design. In addition, the probabilistic nature of the model allows for the integration of expert knowledge in the form of prior distributions. We fit this model to a diverse set of 2,529 cases from the GeoFOR dataset. We demonstrate that the model accurately predicts 24 decomposition characteristics with an ROC AUC score of 0.85. Using Bayesian inference techniques, we invert the decomposition model to predict PMI as a function of the observed decomposition characteristics and environmental and individualistic variables, producing an R-squared measure of 71%. Finally, we demonstrate how to use the fitted model to design future experiments that maximize the expected amount of new information about the mechanisms of decomposition using the Expected Information Gain formalism.

Title: Fair Resource Allocation in Weakly Coupled Markov Decision Processes

• Authors: Xiaohui Tu, Yossiri Adulyasak, Nima Akbarzadeh, Erick Delage
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09804
• Pdf URL: https://arxiv.org/pdf/2411.09804
• Copy Paste: [[2411.09804]] Fair Resource Allocation in Weakly Coupled Markov Decision Processes(https://arxiv.org/abs/2411.09804)
• Keywords: fair
• Abstract: We consider fair resource allocation in sequential decision-making environments modeled as weakly coupled Markov decision processes, where resource constraints couple the action spaces of $N$ sub-Markov decision processes (sub-MDPs) that would otherwise operate independently. We adopt a fairness definition using the generalized Gini function instead of the traditional utilitarian (total-sum) objective. After introducing a general but computationally prohibitive solution scheme based on linear programming, we focus on the homogeneous case where all sub-MDPs are identical. For this case, we show for the first time that the problem reduces to optimizing the utilitarian objective over the class of "permutation invariant" policies. This result is particularly useful as we can exploit Whittle index policies in the restless bandits setting while, for the more general setting, we introduce a count-proportion-based deep reinforcement learning approach. Finally, we validate our theoretical findings with comprehensive experiments, confirming the effectiveness of our proposed method in achieving fairness.

Title: Can Features for Phishing URL Detection Be Trusted Across Diverse Datasets? A Case Study with Explainable AI

• Authors: Maraz Mia, Darius Derakhshan, Mir Mehedi A. Pritom
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09813
• Pdf URL: https://arxiv.org/pdf/2411.09813
• Copy Paste: [[2411.09813]] Can Features for Phishing URL Detection Be Trusted Across Diverse Datasets? A Case Study with Explainable AI(https://arxiv.org/abs/2411.09813)
• Keywords: defense
• Abstract: Phishing has been a prevalent cyber threat that manipulates users into revealing sensitive private information through deceptive tactics, designed to masquerade as trustworthy entities. Over the years, proactively detection of phishing URLs (or websites) has been established as an widely-accepted defense approach. In literature, we often find supervised Machine Learning (ML) models with highly competitive performance for detecting phishing websites based on the extracted features from both phishing and benign (i.e., legitimate) websites. However, it is still unclear if these features or indicators are dependent on a particular dataset or they are generalized for overall phishing detection. In this paper, we delve deeper into this issue by analyzing two publicly available phishing URL datasets, where each dataset has its own set of unique and overlapping features related to URL string and website contents. We want to investigate if overlapping features are similar in nature across datasets and how does the model perform when trained on one dataset and tested on the other. We conduct practical experiments and leverage explainable AI (XAI) methods such as SHAP plots to provide insights into different features' contributions in case of phishing detection to answer our primary question, ``Can features for phishing URL detection be trusted across diverse dataset?''. Our case study experiment results show that features for phishing URL detection can often be dataset-dependent and thus may not be trusted across different datasets even though they share same set of feature behaviors.

Title: Learning Parameter Sharing with Tensor Decompositions and Sparsity

• Authors: Cem Üyük, Mike Lasby, Mohamed Yassin, Utku Evci, Yani Ioannou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09816
• Pdf URL: https://arxiv.org/pdf/2411.09816
• Copy Paste: [[2411.09816]] Learning Parameter Sharing with Tensor Decompositions and Sparsity(https://arxiv.org/abs/2411.09816)
• Keywords: transformer
• Abstract: Large neural networks achieve remarkable performance, but their size hinders deployment on resource-constrained devices. While various compression techniques exist, parameter sharing remains relatively unexplored. This paper introduces Fine-grained Parameter Sharing (FiPS), a novel algorithm that leverages the relationship between parameter sharing, tensor decomposition, and sparsity to efficiently compress large vision transformer models. FiPS employs a shared base and sparse factors to represent shared neurons across multi-layer perception (MLP) modules. Shared parameterization is initialized via Singular Value Decomposition (SVD) and optimized by minimizing block-wise reconstruction error. Experiments demonstrate that FiPS compresses DeiT-B and Swin-L MLPs to 25-40% of their original parameter count while maintaining accuracy within 1 percentage point of the original models.

Title: A Self-Supervised Model for Multi-modal Stroke Risk Prediction

• Authors: Camille Delgrange, Olga Demler, Samia Mora, Bjoern Menze, Ezequiel de la Rosa, Neda Davoudi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09822
• Pdf URL: https://arxiv.org/pdf/2411.09822
• Copy Paste: [[2411.09822]] A Self-Supervised Model for Multi-modal Stroke Risk Prediction(https://arxiv.org/abs/2411.09822)
• Keywords: robust
• Abstract: Predicting stroke risk is a complex challenge that can be enhanced by integrating diverse clinically available data modalities. This study introduces a self-supervised multimodal framework that combines 3D brain imaging, clinical data, and image-derived features to improve stroke risk prediction prior to onset. By leveraging large unannotated clinical datasets, the framework captures complementary and synergistic information across image and tabular data modalities. Our approach is based on a contrastive learning framework that couples contrastive language-image pretraining with an image-tabular matching module, to better align multimodal data representations in a shared latent space. The model is trained on the UK Biobank, which includes structural brain MRI and clinical data. We benchmark its performance against state-of-the-art unimodal and multimodal methods using tabular, image, and image-tabular combinations under diverse frozen and trainable model settings. The proposed model outperformed self-supervised tabular (image) methods by 2.6% (2.6%) in ROC-AUC and by 3.3% (5.6%) in balanced accuracy. Additionally, it showed a 7.6% increase in balanced accuracy compared to the best multimodal supervised model. Through interpretable tools, our approach demonstrated better integration of tabular and image data, providing richer and more aligned embeddings. Gradient-weighted Class Activation Mapping heatmaps further revealed activated brain regions commonly associated in the literature with brain aging, stroke risk, and clinical outcomes. This robust self-supervised multimodal framework surpasses state-of-the-art methods for stroke risk prediction and offers a strong foundation for future studies integrating diverse data modalities to advance clinical predictive modelling.

Title: Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting

• Authors: Yian Wang, Xiaowen Qiu, Jiageng Liu, Zhehuan Chen, Jiting Cai, Yufei Wang, Tsun-Hsuan Wang, Zhou Xian, Chuang Gan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09823
• Pdf URL: https://arxiv.org/pdf/2411.09823
• Copy Paste: [[2411.09823]] Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting(https://arxiv.org/abs/2411.09823)
• Keywords: diffusion, generative, large language model
• Abstract: Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, are hindered by limitations such as excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object configuration than LLMs, we address these challenges by introducing Architect, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate placement of large furniture and small objects to enrich the scene. This iterative structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments.

Title: Evaluating Gender Bias in Large Language Models

• Authors: Michael Döll, Markus Döhring, Andreas Müller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09826
• Pdf URL: https://arxiv.org/pdf/2411.09826
• Copy Paste: [[2411.09826]] Evaluating Gender Bias in Large Language Models(https://arxiv.org/abs/2411.09826)
• Keywords: large language model
• Abstract: Gender bias in artificial intelligence has become an important issue, particularly in the context of language models used in communication-oriented applications. This study examines the extent to which Large Language Models (LLMs) exhibit gender bias in pronoun selection in occupational contexts. The analysis evaluates the models GPT-4, GPT-4o, PaLM 2 Text Bison and Gemini 1.0 Pro using a self-generated dataset. The jobs considered include a range of occupations, from those with a significant male presence to those with a notable female concentration, as well as jobs with a relatively equal gender distribution. Three different sentence processing methods were used to assess potential gender bias: masked tokens, unmasked sentences, and sentence completion. In addition, the LLMs suggested names of individuals in specific occupations, which were then examined for gender distribution. The results show a positive correlation between the models' pronoun choices and the gender distribution present in U.S. labor force data. Female pronouns were more often associated with female-dominated occupations, while male pronouns were more often associated with male-dominated occupations. Sentence completion showed the strongest correlation with actual gender distribution, while name generation resulted in a more balanced 'politically correct' gender distribution, albeit with notable variations in predominantly male or female occupations. Overall, the prompting method had a greater impact on gender distribution than the model selection itself, highlighting the complexity of addressing gender bias in LLMs. The findings highlight the importance of prompting in gender mapping.

Title: A Benchmark for Long-Form Medical Question Answering

• Authors: Pedram Hosseini, Jessica M. Sin, Bing Ren, Bryceton G. Thomas, Elnaz Nouri, Ali Farahanchi, Saeed Hassanpour
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09834
• Pdf URL: https://arxiv.org/pdf/2411.09834
• Copy Paste: [[2411.09834]] A Benchmark for Long-Form Medical Question Answering(https://arxiv.org/abs/2411.09834)
• Keywords: large language model
• Abstract: There is a lack of benchmarks for evaluating large language models (LLMs) in long-form medical question answering (QA). Most existing medical QA evaluation benchmarks focus on automatic metrics and multiple-choice questions. While valuable, these benchmarks fail to fully capture or assess the complexities of real-world clinical applications where LLMs are being deployed. Furthermore, existing studies on evaluating long-form answer generation in medical QA are primarily closed-source, lacking access to human medical expert annotations, which makes it difficult to reproduce results and enhance existing baselines. In this work, we introduce a new publicly available benchmark featuring real-world consumer medical questions with long-form answer evaluations annotated by medical doctors. We performed pairwise comparisons of responses from various open and closed-source medical and general-purpose LLMs based on criteria such as correctness, helpfulness, harmfulness, and bias. Additionally, we performed a comprehensive LLM-as-a-judge analysis to study the alignment between human judgments and LLMs. Our preliminary results highlight the strong potential of open LLMs in medical QA compared to leading closed models. Code & Data: this https URL

Title: Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation Models

• Authors: Kirill Vasilevski, Dayi Lin, Ahmed Hassan
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2411.09837
• Pdf URL: https://arxiv.org/pdf/2411.09837
• Copy Paste: [[2411.09837]] Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation Models(https://arxiv.org/abs/2411.09837)
• Keywords: large language model
• Abstract: To balance the quality and inference cost of a Foundation Model (FM, such as large language models (LLMs)) powered software, people often opt to train a routing model that routes requests to FMs with different sizes and capabilities. Existing routing models rely on learning the optimal routing decision from carefully curated data, require complex computations to be updated, and do not consider the potential evolution of weaker FMs. In this paper, we propose Real-time Adaptive Routing (RAR), an approach to continuously adapt FM routing decisions while using guided in-context learning to enhance the capabilities of weaker FM. The goal is to reduce reliance on stronger, more expensive FMs. We evaluate our approach on different subsets of the popular MMLU benchmark. Over time, our approach routes 50.2% fewer requests to computationally expensive models while maintaining around 90.5% of the general response quality. In addition, the guides generated from stronger models have shown intra-domain generalization and led to a better quality of responses compared to an equivalent approach with a standalone weaker FM.

Title: FedRewind: Rewinding Continual Model Exchange for Decentralized Federated Learning

• Authors: Luca Palazzo, Matteo Pennisi, Federica Proietto Salanitri, Giovanni Bellitto, Simone Palazzo, Concetto Spampinato
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09842
• Pdf URL: https://arxiv.org/pdf/2411.09842
• Copy Paste: [[2411.09842]] FedRewind: Rewinding Continual Model Exchange for Decentralized Federated Learning(https://arxiv.org/abs/2411.09842)
• Keywords: federate
• Abstract: In this paper, we present FedRewind, a novel approach to decentralized federated learning that leverages model exchange among nodes to address the issue of data distribution shift. Drawing inspiration from continual learning (CL) principles and cognitive neuroscience theories for memory retention, FedRewind implements a decentralized routing mechanism where nodes send/receive models to/from other nodes in the federation to address spatial distribution challenges inherent in distributed learning (FL). During local training, federation nodes periodically send their models back (i.e., rewind) to the nodes they received them from for a limited number of iterations. This strategy reduces the distribution shift between nodes' data, leading to enhanced learning and generalization performance. We evaluate our method on multiple benchmarks, demonstrating its superiority over standard decentralized federated learning methods and those enforcing specific routing schemes within the federation. Furthermore, the combination of federated and continual learning concepts enables our method to tackle the more challenging federated continual learning task, with data shifts over both space and time, surpassing existing baselines.

Title: Towards a Fairer Non-negative Matrix Factorization

• Authors: Lara Kassab, Erin George, Deanna Needell, Haowen Geng, Nika Jafar Nia, Aoxi Li
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.09847
• Pdf URL: https://arxiv.org/pdf/2411.09847
• Copy Paste: [[2411.09847]] Towards a Fairer Non-negative Matrix Factorization(https://arxiv.org/abs/2411.09847)
• Keywords: protect, fair
• Abstract: Topic modeling, or more broadly, dimensionality reduction, techniques provide powerful tools for uncovering patterns in large datasets and are widely applied across various domains. We investigate how Non-negative Matrix Factorization (NMF) can introduce bias in the representation of data groups, such as those defined by demographics or protected attributes. We present an approach, called Fairer-NMF, that seeks to minimize the maximum reconstruction loss for different groups relative to their size and intrinsic complexity. Further, we present two algorithms for solving this problem. The first is an alternating minimization (AM) scheme and the second is a multiplicative updates (MU) scheme which demonstrates a reduced computational time compared to AM while still achieving similar performance. Lastly, we present numerical experiments on synthetic and real datasets to evaluate the overall performance and trade-offs of Fairer-NMF

Title: Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements

• Authors: Shijie Zhou, Huaisheng Zhu, Rohan Sharma, Ruiyi Zhang, Kaiyi Ji, Changyou Chen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09850
• Pdf URL: https://arxiv.org/pdf/2411.09850
• Copy Paste: [[2411.09850]] Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements(https://arxiv.org/abs/2411.09850)
• Keywords: diffusion, generative
• Abstract: Diffusion models have emerged as a powerful foundation model for visual generation. With an appropriate sampling process, it can effectively serve as a generative prior to solve general inverse problems. Current posterior sampling based methods take the measurement (i.e., degraded image sample) into the posterior sampling to infer the distribution of the target data (i.e., clean image sample). However, in this manner, we show that high-frequency information can be prematurely introduced during the early stages, which could induce larger posterior estimate errors during the restoration sampling. To address this issue, we first reveal that forming the log posterior gradient with the noisy measurement ( i.e., samples from a diffusion forward process) instead of the clean one can benefit the reverse process. Consequently, we propose a novel diffusion posterior sampling method DPS-CM, which incorporates a Crafted Measurement (i.e., samples generated by a reverse denoising process, compared to random sampling with noise in standard methods) to form the posterior estimate. This integration aims to mitigate the misalignment with the diffusion prior caused by cumulative posterior estimate errors. Experimental results demonstrate that our approach significantly improves the overall capacity to solve general and noisy inverse problems, such as Gaussian deblurring, super-resolution, inpainting, nonlinear deblurring, and tasks with Poisson noise, relative to existing approaches.

Title: Fair Secretaries with Unfair Predictions

• Authors: Eric Balkanski, Will Ma, Andreas Maggiori
• Subjects: cs.LG, cs.DS
• Abstract URL: https://arxiv.org/abs/2411.09854
• Pdf URL: https://arxiv.org/pdf/2411.09854
• Copy Paste: [[2411.09854]] Fair Secretaries with Unfair Predictions(https://arxiv.org/abs/2411.09854)
• Keywords: fair
• Abstract: Algorithms with predictions is a recent framework for decision-making under uncertainty that leverages the power of machine-learned predictions without making any assumption about their quality. The goal in this framework is for algorithms to achieve an improved performance when the predictions are accurate while maintaining acceptable guarantees when the predictions are erroneous. A serious concern with algorithms that use predictions is that these predictions can be biased and, as a result, cause the algorithm to make decisions that are deemed unfair. We show that this concern manifests itself in the classical secretary problem in the learning-augmented setting -- the state-of-the-art algorithm can have zero probability of accepting the best candidate, which we deem unfair, despite promising to accept a candidate whose expected value is at least $\max\{\Omega (1) , 1 - O(\epsilon)\}$ times the optimal value, where $\epsilon$ is the prediction error. We show how to preserve this promise while also guaranteeing to accept the best candidate with probability $\Omega(1)$. Our algorithm and analysis are based on a new "pegging" idea that diverges from existing works and simplifies/unifies some of their results. Finally, we extend to the $k$-secretary problem and complement our theoretical analysis with experiments.

Title: Masked Image Contrastive Learning for Efficient Visual Conceptual Pre-training

• Authors: Xiaoyu Yang, Lijian Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09858
• Pdf URL: https://arxiv.org/pdf/2411.09858
• Copy Paste: [[2411.09858]] Masked Image Contrastive Learning for Efficient Visual Conceptual Pre-training(https://arxiv.org/abs/2411.09858)
• Keywords: transformer
• Abstract: This paper proposes a scalable and straightforward pre-training paradigm for efficient visual conceptual representation called masked image contrastive learning (MiCL). Our MiCL approach is simple: we randomly mask patches to generate different views within an image and contrast them among a mini-batch of images. The core idea behind MiCL consists of two designs. First, masked tokens have the potential to significantly diminish the conceptual redundancy inherent in images, and create distinct views with substantial fine-grained differences on the semantic concept level instead of the instance level. Second, contrastive learning is adept at extracting high-level semantic conceptual features during the pre-training, circumventing the high-frequency interference and additional costs associated with image reconstruction. Importantly, MiCL learns highly semantic conceptual representations efficiently without relying on hand-crafted data augmentations or additional auxiliary modules. Empirically, MiCL demonstrates high scalability with Vision Transformers, as the ViT-L/16 can complete pre-training in 133 hours using only 4 A100 GPUs, achieving 85.8% accuracy in downstream fine-tuning tasks.

Title: Face De-identification: State-of-the-art Methods and Comparative Studies

• Authors: Jingyi Cao, Xiangyi Chen, Bo Liu, Ming Ding, Rong Xie, Li Song, Zhu Li, Wenjun Zhang
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09863
• Pdf URL: https://arxiv.org/pdf/2411.09863
• Copy Paste: [[2411.09863]] Face De-identification: State-of-the-art Methods and Comparative Studies(https://arxiv.org/abs/2411.09863)
• Keywords: privacy, protect, diffusion, generative
• Abstract: The widespread use of image acquisition technologies, along with advances in facial recognition, has raised serious privacy concerns. Face de-identification usually refers to the process of concealing or replacing personal identifiers, which is regarded as an effective means to protect the privacy of facial images. A significant number of methods for face de-identification have been proposed in recent years. In this survey, we provide a comprehensive review of state-of-the-art face de-identification methods, categorized into three levels: pixel-level, representation-level, and semantic-level techniques. We systematically evaluate these methods based on two key criteria, the effectiveness of privacy protection and preservation of image utility, highlighting their advantages and limitations. Our analysis includes qualitative and quantitative comparisons of the main algorithms, demonstrating that deep learning-based approaches, particularly those using Generative Adversarial Networks (GANs) and diffusion models, have achieved significant advancements in balancing privacy and utility. Experimental results reveal that while recent methods demonstrate strong privacy protection, trade-offs remain in visual fidelity and computational complexity. This survey not only summarizes the current landscape but also identifies key challenges and future research directions in face de-identification.

Title: Content-Aware Preserving Image Generation

• Authors: Giang H. Le, Anh Q. Nguyen, Byeongkeun Kang, Yeejin Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09871
• Pdf URL: https://arxiv.org/pdf/2411.09871
• Copy Paste: [[2411.09871]] Content-Aware Preserving Image Generation(https://arxiv.org/abs/2411.09871)
• Keywords: generative
• Abstract: Remarkable progress has been achieved in image generation with the introduction of generative models. However, precisely controlling the content in generated images remains a challenging task due to their fundamental training objective. This paper addresses this challenge by proposing a novel image generation framework explicitly designed to incorporate desired content in output images. The framework utilizes advanced encoding techniques, integrating subnetworks called content fusion and frequency encoding modules. The frequency encoding module first captures features and structures of reference images by exclusively focusing on selected frequency components. Subsequently, the content fusion module generates a content-guiding vector that encapsulates desired content features. During the image generation process, content-guiding vectors from real images are fused with projected noise vectors. This ensures the production of generated images that not only maintain consistent content from guiding images but also exhibit diverse stylistic variations. To validate the effectiveness of the proposed framework in preserving content attributes, extensive experiments are conducted on widely used benchmark datasets, including Flickr-Faces-High Quality, Animal Faces High Quality, and Large-scale Scene Understanding datasets.

Title: Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation

• Authors: Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, Anqi Liu
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.09891
• Pdf URL: https://arxiv.org/pdf/2411.09891
• Copy Paste: [[2411.09891]] Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation(https://arxiv.org/abs/2411.09891)
• Keywords: generative
• Abstract: Training a policy in a source domain for deployment in the target domain under a dynamics shift can be challenging, often resulting in performance degradation. Previous work tackles this challenge by training on the source domain with modified rewards derived by matching distributions between the source and the target optimal trajectories. However, pure modified rewards only ensure the behavior of the learned policy in the source domain resembles trajectories produced by the target optimal policies, which does not guarantee optimal performance when the learned policy is actually deployed to the target domain. In this work, we propose to utilize imitation learning to transfer the policy learned from the reward modification to the target domain so that the new policy can generate the same trajectories in the target domain. Our approach, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL), utilizes the reward modification for domain adaptation and follows the general framework of generative adversarial imitation learning from observation (GAIfO) by applying a reward augmented estimator for the policy optimization step. Theoretically, we present an error bound for our method under a mild assumption regarding the dynamics shift to justify the motivation of our method. Empirically, our method outperforms the pure modified reward method without imitation learning and also outperforms other baselines in benchmark off-dynamics environments.

Title: Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement

• Authors: Yanyan Huang, Weiqin Zhao, Yihang Chen, Yu Fu, Lequan Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09894
• Pdf URL: https://arxiv.org/pdf/2411.09894
• Copy Paste: [[2411.09894]] Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement(https://arxiv.org/abs/2411.09894)
• Keywords: interpretability
• Abstract: Whole slide image (WSI) analysis is gaining prominence within the medical imaging field. Recent advances in pathology foundation models have shown the potential to extract powerful feature representations from WSIs for downstream tasks. However, these foundation models are usually designed for general-purpose pathology image analysis and may not be optimal for specific downstream tasks or cancer types. In this work, we present Concept Anchor-guided Task-specific Feature Enhancement (CATE), an adaptable paradigm that can boost the expressivity and discriminativeness of pathology foundation models for specific downstream tasks. Based on a set of task-specific concepts derived from the pathology vision-language model with expert-designed prompts, we introduce two interconnected modules to dynamically calibrate the generic image features extracted by foundation models for certain tasks or cancer types. Specifically, we design a Concept-guided Information Bottleneck module to enhance task-relevant characteristics by maximizing the mutual information between image features and concept anchors while suppressing superfluous information. Moreover, a Concept-Feature Interference module is proposed to utilize the similarity between calibrated features and concept anchors to further generate discriminative task-specific features. The extensive experiments on public WSI datasets demonstrate that CATE significantly enhances the performance and generalizability of MIL models. Additionally, heatmap and umap visualization results also reveal the effectiveness and interpretability of CATE. The source code is available at this https URL.

Title: Exploiting Cross-Layer Vulnerabilities: Off-Path Attacks on the TCP/IP Protocol Suite

• Authors: Xuewei Feng, Qi Li, Kun Sun, Ke Xu, Jianping Wu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.09895
• Pdf URL: https://arxiv.org/pdf/2411.09895
• Copy Paste: [[2411.09895]] Exploiting Cross-Layer Vulnerabilities: Off-Path Attacks on the TCP/IP Protocol Suite(https://arxiv.org/abs/2411.09895)
• Keywords: security, attack, robust, steal
• Abstract: After more than 40 years of development, the fundamental TCP/IP protocol suite, serving as the backbone of the Internet, is widely recognized for having achieved an elevated level of robustness and security. Distinctively, we take a new perspective to investigate the security implications of cross-layer interactions within the TCP/IP protocol suite caused by ICMP error messages. Through a comprehensive analysis of interactions among Wi-Fi, IP, ICMP, UDP, and TCP due to ICMP errors, we uncover several significant vulnerabilities, including information leakage, desynchronization, semantic gaps, and identity spoofing. These vulnerabilities can be exploited by off-path attackers to manipulate network traffic stealthily, affecting over 20% of popular websites and more than 89% of public Wi-Fi networks, thus posing risks to the Internet. By responsibly disclosing these vulnerabilities to affected vendors and proposing effective countermeasures, we enhance the robustness of the TCP/IP protocol suite, receiving acknowledgments from well-known organizations such as the Linux community, the OpenWrt community, the FreeBSD community, Wi-Fi Alliance, Qualcomm, HUAWEI, China Telecom, Alibaba, and H3C.

Title: A Survey of Machine Learning-based Physical-Layer Authentication in Wireless Communications

• Authors: Rui Meng, Bingxuan Xu, Xiaodong Xu, Mengying Sun, Bizhu Wanga, Shujun Han, Suyu Lv, Ping Zhang
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.09906
• Pdf URL: https://arxiv.org/pdf/2411.09906
• Copy Paste: [[2411.09906]] A Survey of Machine Learning-based Physical-Layer Authentication in Wireless Communications(https://arxiv.org/abs/2411.09906)
• Keywords: secure, attack
• Abstract: To ensure secure and reliable communication in wireless systems, authenticating the identities of numerous nodes is imperative. Traditional cryptography-based authentication methods suffer from issues such as low compatibility, reliability, and high complexity. Physical-Layer Authentication (PLA) is emerging as a promising complement due to its exploitation of unique properties in wireless environments. Recently, Machine Learning (ML)-based PLA has gained attention for its intelligence, adaptability, universality, and scalability compared to non-ML approaches. However, a comprehensive overview of state-of-the-art ML-based PLA and its foundational aspects is lacking. This paper presents a comprehensive survey of characteristics and technologies that can be used in the ML-based PLA. We categorize existing ML-based PLA schemes into two main types: multi-device identification and attack detection schemes. In deep learning-based multi-device identification schemes, Deep Neural Networks are employed to train models, avoiding complex processing and expert feature transformation. Deep learning-based multi-device identification schemes are further subdivided, with schemes based on Convolutional Neural Networks being extensively researched. In ML-based attack detection schemes, receivers utilize intelligent ML techniques to set detection thresholds automatically, eliminating the need for manual calculation or knowledge of channel models. ML-based attack detection schemes are categorized into three sub-types: Supervised Learning, Unsupervised Learning, and Reinforcement Learning. Additionally, we summarize open-source datasets used for PLA, encompassing Radio Frequency fingerprints and channel fingerprints. Finally, this paper outlines future research directions to guide researchers in related fields.

Title: DiffFNO: Diffusion Fourier Neural Operator

• Authors: Xiaoyi Liu, Hao Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09911
• Pdf URL: https://arxiv.org/pdf/2411.09911
• Copy Paste: [[2411.09911]] DiffFNO: Diffusion Fourier Neural Operator(https://arxiv.org/abs/2411.09911)
• Keywords: diffusion
• Abstract: We introduce DiffFNO, a novel diffusion framework for arbitrary-scale super-resolution strengthened by a Weighted Fourier Neural Operator (WFNO). Mode Re-balancing in WFNO effectively captures critical frequency components, significantly improving the reconstruction of high-frequency image details that are crucial for super-resolution tasks. Gated Fusion Mechanism (GFM) adaptively complements WFNO's spectral features with spatial features from an Attention-based Neural Operator (AttnNO). This enhances the network's capability to capture both global structures and local details. Adaptive Time-Step (ATS) ODE solver, a deterministic sampling strategy, accelerates inference without sacrificing output quality by dynamically adjusting integration step sizes ATS. Extensive experiments demonstrate that DiffFNO achieves state-of-the-art (SOTA) results, outperforming existing methods across various scaling factors by a margin of 2 to 4 dB in PSNR, including those beyond the training distribution. It also achieves this at lower inference time. Our approach sets a new standard in super-resolution, delivering both superior accuracy and computational efficiency.

Title: mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual Reality

• Authors: Luoyu Mei, Ruofeng Liu, Zhimeng Yin, Qingchuan Zhao, Wenchao Jiang, Shuai Wang, Kangjie Lu, Tian He
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09914
• Pdf URL: https://arxiv.org/pdf/2411.09914
• Copy Paste: [[2411.09914]] mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual Reality(https://arxiv.org/abs/2411.09914)
• Keywords: security, privacy, protect, attack, extraction
• Abstract: Virtual reality (VR), while enhancing user experiences, introduces significant privacy risks. This paper reveals a novel vulnerability in VR systems that allows attackers to capture VR privacy through obstacles utilizing millimeter-wave (mmWave) signals without physical intrusion and virtual connection with the VR devices. We propose mmSpyVR, a novel attack on VR user's privacy via mmWave radar. The mmSpyVR framework encompasses two main parts: (i) A transfer learning-based feature extraction model to achieve VR feature extraction from mmWave signal. (ii) An attention-based VR privacy spying module to spy VR privacy information from the extracted feature. The mmSpyVR demonstrates the capability to extract critical VR privacy from the mmWave signals that have penetrated through obstacles. We evaluate mmSpyVR through IRB-approved user studies. Across 22 participants engaged in four experimental scenes utilizing VR devices from three different manufacturers, our system achieves an application recognition accuracy of 98.5\% and keystroke recognition accuracy of 92.6\%. This newly discovered vulnerability has implications across various domains, such as cybersecurity, privacy protection, and VR technology development. We also engage with VR manufacturer Meta to discuss and explore potential mitigation strategies. Data and code are publicly available for scrutiny and research at this https URL

Title: Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level

• Authors: Andong Deng, Tongjia Chen, Shoubin Yu, Taojiannan Yang, Lincoln Spencer, Yapeng Tian, Ajmal Saeed Mian, Mohit Bansal, Chen Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09921
• Pdf URL: https://arxiv.org/pdf/2411.09921
• Copy Paste: [[2411.09921]] Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level(https://arxiv.org/abs/2411.09921)
• Keywords: robust, segmentation
• Abstract: In this paper, we introduce Motion-Grounded Video Reasoning, a new motion understanding task that requires generating visual answers (video segmentation masks) according to the input question, and hence needs implicit spatiotemporal reasoning and grounding. This task extends existing spatiotemporal grounding work focusing on explicit action/motion grounding, to a more general format by enabling implicit reasoning via questions. To facilitate the development of the new task, we collect a large-scale dataset called GROUNDMORE, which comprises 1,715 video clips, 249K object masks that are deliberately designed with 4 question types (Causal, Sequential, Counterfactual, and Descriptive) for benchmarking deep and comprehensive motion reasoning abilities. GROUNDMORE uniquely requires models to generate visual answers, providing a more concrete and visually interpretable response than plain texts. It evaluates models on both spatiotemporal grounding and reasoning, fostering to address complex challenges in motion-related video reasoning, temporal perception, and pixel-level understanding. Furthermore, we introduce a novel baseline model named Motion-Grounded Video Reasoning Assistant (MORA). MORA incorporates the multimodal reasoning ability from the Multimodal LLM, the pixel-level perception capability from the grounding model (SAM), and the temporal perception ability from a lightweight localization head. MORA achieves respectable performance on GROUNDMORE outperforming the best existing visual grounding baseline model by an average of 21.5% relatively. We hope this novel and challenging task will pave the way for future advancements in robust and general motion understanding via video reasoning segmentation

Title: A Polarization Image Dehazing Method Based on the Principle of Physical Diffusion

• Authors: Zhenjun Zhang, Lijun Tang, Hongjin Wang, Lilian Zhang, Yunze He, Yaonan Wang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.09924
• Pdf URL: https://arxiv.org/pdf/2411.09924
• Copy Paste: [[2411.09924]] A Polarization Image Dehazing Method Based on the Principle of Physical Diffusion(https://arxiv.org/abs/2411.09924)
• Keywords: diffusion
• Abstract: Computer vision is increasingly used in areas such as unmanned vehicles, surveillance systems and remote sensing. However, in foggy scenarios, image degradation leads to loss of target details, which seriously affects the accuracy and effectiveness of these vision tasks. Polarized light, due to the fact that its electromagnetic waves vibrate in a specific direction, is able to resist scattering and refraction effects in complex media more effectively compared to unpolarized light. As a result, polarized light has a greater ability to maintain its polarization characteristics in complex transmission media and under long-distance imaging conditions. This property makes polarized imaging especially suitable for complex scenes such as outdoor and underwater, especially in foggy environments, where higher quality images can be obtained. Based on this advantage, we propose an innovative semi-physical polarization dehazing method that does not rely on an external light source. The method simulates the diffusion process of fog and designs a diffusion kernel that corresponds to the image blurriness caused by this diffusion. By employing spatiotemporal Fourier transforms and deconvolution operations, the method recovers the state of fog droplets prior to diffusion and the light inversion distribution of objects. This approach effectively achieves dehazing and detail enhancement of the scene.

Title: JRadiEvo: A Japanese Radiology Report Generation Model Enhanced by Evolutionary Optimization of Model Merging

• Authors: Kaito Baba, Ryota Yagi, Junichiro Takahashi, Risa Kishikawa, Satoshi Kodera
• Subjects: cs.CV, cs.AI, cs.CL, cs.NE
• Abstract URL: https://arxiv.org/abs/2411.09933
• Pdf URL: https://arxiv.org/pdf/2411.09933
• Copy Paste: [[2411.09933]] JRadiEvo: A Japanese Radiology Report Generation Model Enhanced by Evolutionary Optimization of Model Merging(https://arxiv.org/abs/2411.09933)
• Keywords: security, privacy, large language model
• Abstract: With the rapid advancement of large language models (LLMs), foundational models (FMs) have seen significant advancements. Healthcare is one of the most crucial application areas for these FMs, given the significant time and effort required for physicians to analyze large volumes of patient data. Recent efforts have focused on adapting multimodal FMs to the medical domain through techniques like instruction-tuning, leading to the development of medical foundation models (MFMs). However, these approaches typically require large amounts of training data to effectively adapt models to the medical field. Moreover, most existing models are trained on English datasets, limiting their practicality in non-English-speaking regions where healthcare professionals and patients are not always fluent in English. The need for translation introduces additional costs and inefficiencies. To address these challenges, we propose a \textbf{J}apanese \textbf{Radi}ology report generation model enhanced by \textbf{Evo}lutionary optimization of model merging (JRadiEvo). This is the first attempt to extend a non-medical vision-language foundation model to the medical domain through evolutionary optimization of model merging. We successfully created a model that generates accurate Japanese reports from X-ray images using only 50 translated samples from publicly available data. This model, developed with highly efficient use of limited data, outperformed leading models from recent research trained on much larger datasets. Additionally, with only 8 billion parameters, this relatively compact foundation model can be deployed locally within hospitals, making it a practical solution for environments where APIs and other external services cannot be used due to strict privacy and security requirements.

Title: Refined and Segmented Price Sentiment Indices from Survey Comments

• Authors: Masahiro Suzuki, Hiroki Sakaji
• Subjects: cs.CL, q-fin.CP
• Abstract URL: https://arxiv.org/abs/2411.09937
• Pdf URL: https://arxiv.org/pdf/2411.09937
• Copy Paste: [[2411.09937]] Refined and Segmented Price Sentiment Indices from Survey Comments(https://arxiv.org/abs/2411.09937)
• Keywords: large language model
• Abstract: We aim to enhance a price sentiment index and to more precisely understand price trends from the perspective of not only consumers but also businesses. We extract comments related to prices from the Economy Watchers Survey conducted by the Cabinet Office of Japan and classify price trends using a large language model (LLM). We classify whether the survey sample reflects the perspective of consumers or businesses, and whether the comments pertain to goods or services by utilizing information on the fields of comments and the industries of respondents included in the Economy Watchers Survey. From these classified price-related comments, we construct price sentiment indices not only for a general purpose but also for more specific objectives by combining perspectives on consumers and prices, as well as goods and services. It becomes possible to achieve a more accurate classification of price directions by employing a LLM for classification. Furthermore, integrating the outputs of multiple LLMs suggests the potential for the better performance of the classification. The use of more accurately classified comments allows for the construction of an index with a higher correlation to existing indices than previous studies. We demonstrate that the correlation of the price index for consumers, which has a larger sample size, is further enhanced by selecting comments for aggregation based on the industry of the survey respondents.

Title: SlimLM: An Efficient Small Language Model for On-Device Document Assistance

• Authors: Thang M. Pham, Phat T. Nguyen, Seunghyun Yoon, Viet Dac Lai, Franck Dernoncourt, Trung Bui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09944
• Pdf URL: https://arxiv.org/pdf/2411.09944
• Copy Paste: [[2411.09944]] SlimLM: An Efficient Small Language Model for On-Device Document Assistance(https://arxiv.org/abs/2411.09944)
• Keywords: privacy
• Abstract: While small language models (SLMs) show promises for mobile deployment, their real-world performance and applications on smartphones remains underexplored. We present SlimLM, a series of SLMs optimized for document assistance tasks on mobile devices. Through extensive experiments on a Samsung Galaxy S24, we identify the optimal trade-offs between model size (ranging from 125M to 7B parameters), context length, and inference time for efficient on-device processing. SlimLM is pre-trained on SlimPajama-627B and fine-tuned on DocAssist, our constructed dataset for summarization, question answering and suggestion tasks. Our smallest model demonstrates efficient performance on S24, while larger variants offer enhanced capabilities within mobile constraints. We evaluate SlimLM against existing SLMs, showing comparable or superior performance and offering a benchmark for future research in on-device language models. We also provide an Android application, offering practical insights into SLM deployment. Our findings provide valuable insights and illuminate the capabilities of running advanced language models on high-end smartphones, potentially reducing server costs and enhancing privacy through on-device processing.

Title: TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers have Pre-Trained Models

• Authors: Ding Li, Ziqi Zhang, Mengyu Yao, Yifeng Cai, Yao Guo, Xiangqun Chen
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09945
• Pdf URL: https://arxiv.org/pdf/2411.09945
• Copy Paste: [[2411.09945]] TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers have Pre-Trained Models(https://arxiv.org/abs/2411.09945)
• Keywords: secure, security, privacy, protect, attack, large language model
• Abstract: Trusted Execution Environments (TEE) are used to safeguard on-device models. However, directly employing TEEs to secure the entire DNN model is challenging due to the limited computational speed. Utilizing GPU can accelerate DNN's computation speed but commercial widely-available GPUs usually lack security protection. To this end, scholars introduce TSDP, a method that protects privacy-sensitive weights within TEEs and offloads insensitive weights to GPUs. Nevertheless, current methods do not consider the presence of a knowledgeable adversary who can access abundant publicly available pre-trained models and datasets. This paper investigates the security of existing methods against such a knowledgeable adversary and reveals their inability to fulfill their security promises. Consequently, we introduce a novel partition before training strategy, which effectively separates privacy-sensitive weights from other components of the model. Our evaluation demonstrates that our approach can offer full model protection with a computational cost reduced by a factor of 10. In addition to traditional CNN models, we also demonstrate the scalability to large language models. Our approach can compress the private functionalities of the large language model to lightweight slices and achieve the same level of protection as the shielding-whole-model baseline.

Title: Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era

• Authors: Thanh Tam Nguyen, Zhao Ren, Trinh Pham, Phi Le Nguyen, Hongzhi Yin, Quoc Viet Hung Nguyen
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.09955
• Pdf URL: https://arxiv.org/pdf/2411.09955
• Copy Paste: [[2411.09955]] Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era(https://arxiv.org/abs/2411.09955)
• Keywords: diffusion, generative, large language model
• Abstract: The rapid advancement of large language models (LLMs) and multimodal learning has transformed digital content creation and manipulation. Traditional visual editing tools require significant expertise, limiting accessibility. Recent strides in instruction-based editing have enabled intuitive interaction with visual content, using natural language as a bridge between user intent and complex editing operations. This survey provides an overview of these techniques, focusing on how LLMs and multimodal models empower users to achieve precise visual modifications without deep technical knowledge. By synthesizing over 100 publications, we explore methods from generative adversarial networks to diffusion models, examining multimodal integration for fine-grained content control. We discuss practical applications across domains such as fashion, 3D scene manipulation, and video synthesis, highlighting increased accessibility and alignment with human intuition. Our survey compares existing literature, emphasizing LLM-empowered editing, and identifies key challenges to stimulate further research. We aim to democratize powerful visual editing across various industries, from entertainment to education. Interested readers are encouraged to access our repository at this https URL.

Title: Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs

• Authors: Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, Jieping Ye
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09968
• Pdf URL: https://arxiv.org/pdf/2411.09968
• Copy Paste: [[2411.09968]] Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs(https://arxiv.org/abs/2411.09968)
• Keywords: large language model
• Abstract: The hallucination problem in multimodal large language models (MLLMs) remains a common issue. Although image tokens occupy a majority of the input sequence of MLLMs, there is limited research to explore the relationship between image tokens and hallucinations. In this paper, we analyze the distribution of attention scores for image tokens across each layer and head of the model, revealing an intriguing and common phenomenon: most hallucinations are closely linked to the pattern of attention sinks in the self-attention matrix of image tokens, where shallow layers exhibit dense attention sinks and deeper layers show sparse attention sinks. We further analyze the attention heads of different layers and find that heads with high-density attention sink in the image part play a positive role in alleviating hallucinations. In this paper, we propose a training-free method named \textcolor{red}{\textbf{E}}nhancing \textcolor{red}{\textbf{A}}ttention \textcolor{red}{\textbf{H}}eads (EAH), an approach designed to enhance the convergence of image tokens attention sinks in the shallow layers. EAH identifies the attention head that shows the vision sink in a shallow layer and extracts its attention matrix. This attention map is then broadcast to other heads in the layer, thereby strengthening the layer to pay more attention to the image itself. With extensive experiments, EAH shows significant hallucination-mitigating performance on different MLLMs and metrics, proving its effectiveness and generality.

Title: Explanation for Trajectory Planning using Multi-modal Large Language Model for Autonomous Driving

• Authors: Shota Yamazaki, Chenyu Zhang, Takuya Nanri, Akio Shigekane, Siyuan Wang, Jo Nishiyama, Tao Chu, Kohei Yokosawa
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.09971
• Pdf URL: https://arxiv.org/pdf/2411.09971
• Copy Paste: [[2411.09971]] Explanation for Trajectory Planning using Multi-modal Large Language Model for Autonomous Driving(https://arxiv.org/abs/2411.09971)
• Keywords: interpretability, large language model
• Abstract: End-to-end style autonomous driving models have been developed recently. These models lack interpretability of decision-making process from perception to control of the ego vehicle, resulting in anxiety for passengers. To alleviate it, it is effective to build a model which outputs captions describing future behaviors of the ego vehicle and their reason. However, the existing approaches generate reasoning text that inadequately reflects the future plans of the ego vehicle, because they train models to output captions using momentary control signals as inputs. In this study, we propose a reasoning model that takes future planning trajectories of the ego vehicle as inputs to solve this limitation with the dataset newly collected.

Title: Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems

• Authors: Taaha Kazi, Ruiliang Lyu, Sizhe Zhou, Dilek Hakkani-Tur, Gokhan Tur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09972
• Pdf URL: https://arxiv.org/pdf/2411.09972
• Copy Paste: [[2411.09972]] Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems(https://arxiv.org/abs/2411.09972)
• Keywords: large language model
• Abstract: Traditionally, offline datasets have been used to evaluate task-oriented dialogue (TOD) models. These datasets lack context awareness, making them suboptimal benchmarks for conversational systems. In contrast, user-agents, which are context-aware, can simulate the variability and unpredictability of human conversations, making them better alternatives as evaluators. Prior research has utilized large language models (LLMs) to develop user-agents. Our work builds upon this by using LLMs to create user-agents for the evaluation of TOD systems. This involves prompting an LLM, using in-context examples as guidance, and tracking the user-goal state. Our evaluation of diversity and task completion metrics for the user-agents shows improved performance with the use of better prompts. Additionally, we propose methodologies for the automatic evaluation of TOD models within this dynamic framework.

Title: Establishing and Evaluating Trustworthy AI: Overview and Research Challenges

• Authors: Dominik Kowald, Sebastian Scher, Viktoria Pammer-Schindler, Peter Müllner, Kerstin Waxnegger, Lea Demelius, Angela Fessl, Maximilian Toller, Inti Gabriel Mendoza Estrada, Ilija Simic, Vedran Sabol, Andreas Truegler, Eduardo Veas, Roman Kern, Tomislav Nad, Simone Kopeinik
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.09973
• Pdf URL: https://arxiv.org/pdf/2411.09973
• Copy Paste: [[2411.09973]] Establishing and Evaluating Trustworthy AI: Overview and Research Challenges(https://arxiv.org/abs/2411.09973)
• Keywords: security, privacy, robust, fair, explainability
• Abstract: Artificial intelligence (AI) technologies (re-)shape modern life, driving innovation in a wide range of sectors. However, some AI systems have yielded unexpected or undesirable outcomes or have been used in questionable manners. As a result, there has been a surge in public and academic discussions about aspects that AI systems must fulfill to be considered trustworthy. In this paper, we synthesize existing conceptualizations of trustworthy AI along six requirements: 1) human agency and oversight, 2) fairness and non-discrimination, 3) transparency and explainability, 4) robustness and accuracy, 5) privacy and security, and 6) accountability. For each one, we provide a definition, describe how it can be established and evaluated, and discuss requirement-specific research challenges. Finally, we conclude this analysis by identifying overarching research challenges across the requirements with respect to 1) interdisciplinary research, 2) conceptual clarity, 3) context-dependency, 4) dynamics in evolving systems, and 5) investigations in real-world contexts. Thus, this paper synthesizes and consolidates a wide-ranging and active discussion currently taking place in various academic sub-communities and public forums. It aims to serve as a reference for a broad audience and as a basis for future research directions.

Title: HistoLens: An LLM-Powered Framework for Multi-Layered Analysis of Historical Texts -- A Case Application of Yantie Lun

• Authors: Yifan Zeng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09978
• Pdf URL: https://arxiv.org/pdf/2411.09978
• Copy Paste: [[2411.09978]] HistoLens: An LLM-Powered Framework for Multi-Layered Analysis of Historical Texts -- A Case Application of Yantie Lun(https://arxiv.org/abs/2411.09978)
• Keywords: large language model
• Abstract: This paper proposes HistoLens, a multi-layered analysis framework for historical texts based on Large Language Models (LLMs). Using the important Western Han dynasty text "Yantie Lun" as a case study, we demonstrate the framework's potential applications in historical research and education. HistoLens integrates NLP technology (especially LLMs), including named entity recognition, knowledge graph construction, and geographic information visualization. The paper showcases how HistoLens explores Western Han culture in "Yantie Lun" through multi-dimensional, visual, and quantitative methods, focusing particularly on the influence of Confucian and Legalist thoughts on political, economic, military, and ethnic. We also demonstrate how HistoLens constructs a machine teaching scenario using LLMs for explainable analysis, based on a dataset of Confucian and Legalist ideas extracted with LLM assistance. This approach offers novel and diverse perspectives for studying historical texts like "Yantie Lun" and provides new auxiliary tools for history education. The framework aims to equip historians and learners with LLM-assisted tools to facilitate in-depth, multi-layered analysis of historical texts and foster innovation in historical education.

Title: Strategic Roadmap for Quantum- Resistant Security: A Framework for Preparing Industries for the Quantum Threat

• Authors: Arit Kumar Bishwas, Mousumi Sen
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2411.09995
• Pdf URL: https://arxiv.org/pdf/2411.09995
• Copy Paste: [[2411.09995]] Strategic Roadmap for Quantum- Resistant Security: A Framework for Preparing Industries for the Quantum Threat(https://arxiv.org/abs/2411.09995)
• Keywords: security, protect, attack
• Abstract: As quantum computing continues to advance, its ability to compromise widely used cryptographic systems projects a significant challenge to modern cybersecurity. This paper outlines a strategic roadmap for industries to anticipate and mitigate the risks posed by quantum attacks. Our study explores the development of a quantum-resistant cryptographic solutioning framework for the industry, offering a practical and strategic approach to mitigating quantum attacks. We, here, propose a novel strategic framework, coined name STL-QCRYPTO, outlines tailored, industry-specific methodologies to implement quantum-safe security systems, ensuring long-term protection against the disruptive potential of quantum computing. The following fourteen high-risk sectors: Financial Services, Banking, Healthcare, Critical Infrastructure, Government & Defence, E-commerce, Energy & Utilities, Automotive & Transportation, Cloud Computing & Data Storage, Insurance, Internet & Telecommunications, Blockchain Applications, Metaverse Applications, and Multiagent AI Systems - are critically assessed for their vulnerability to quantum threats. The evaluation emphasizes practical approaches for the deployment of quantum-safe security systems to safeguard these industries against emerging quantum-enabled cyber risks. Additionally, the paper addresses the technical, operational, and regulatory hurdles associated with adopting quantum-resistant technologies. By presenting a structured timeline and actionable recommendations, this roadmap with proposed framework prepares industries with the essential strategy to safeguard their potential security threats in the quantum computing era.

Title: Adaptive Non-Uniform Timestep Sampling for Diffusion Model Training

• Authors: Myunsoo Kim, Donghyeon Ki, Seong-Woong Shim, Byung-Jun Lee
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09998
• Pdf URL: https://arxiv.org/pdf/2411.09998
• Copy Paste: [[2411.09998]] Adaptive Non-Uniform Timestep Sampling for Diffusion Model Training(https://arxiv.org/abs/2411.09998)
• Keywords: robust, diffusion, generative
• Abstract: As a highly expressive generative model, diffusion models have demonstrated exceptional success across various domains, including image generation, natural language processing, and combinatorial optimization. However, as data distributions grow more complex, training these models to convergence becomes increasingly computationally intensive. While diffusion models are typically trained using uniform timestep sampling, our research shows that the variance in stochastic gradients varies significantly across timesteps, with high-variance timesteps becoming bottlenecks that hinder faster convergence. To address this issue, we introduce a non-uniform timestep sampling method that prioritizes these more critical timesteps. Our method tracks the impact of gradient updates on the objective for each timestep, adaptively selecting those most likely to minimize the objective effectively. Experimental results demonstrate that this approach not only accelerates the training process, but also leads to improved performance at convergence. Furthermore, our method shows robust performance across various datasets, scheduling strategies, and diffusion architectures, outperforming previously proposed timestep sampling and weighting heuristics that lack this degree of robustness.

Title: Orca: Enhancing Role-Playing Abilities of Large Language Models by Integrating Personality Traits

• Authors: Yuxuan Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10006
• Pdf URL: https://arxiv.org/pdf/2411.10006
• Copy Paste: [[2411.10006]] Orca: Enhancing Role-Playing Abilities of Large Language Models by Integrating Personality Traits(https://arxiv.org/abs/2411.10006)
• Keywords: large language model
• Abstract: Large language models has catalyzed the development of personalized dialogue systems, numerous role-playing conversational agents have emerged. While previous research predominantly focused on enhancing the model's capability to follow instructions by designing character profiles, neglecting the psychological factors that drive human conversations. In this paper, we propose Orca, a framework for data processing and training LLMs of custom characters by integrating personality traits. Orca comprises four stages: (1) Personality traits inferring, leverage LLMs to infer user's BigFive personality trait reports and scores. (2) Data Augment, simulate user's profile, background story, and psychological activities. (3) Dataset construction, personality-conditioned instruction prompting (PCIP) to stimulate LLMs. (4) Modeling and Training, personality-conditioned instruction tuning (PTIT and PSIT), using the generated data to enhance existing open-source LLMs. We introduce OrcaBench, the first benchmark for evaluating the quality of content generated by LLMs on social platforms across multiple scales. Our experiments demonstrate that our proposed model achieves superior performance on this benchmark, demonstrating its excellence and effectiveness in perceiving personality traits that significantly improve role-playing abilities. Our Code is available at this https URL.

Title: Efficient Depth Estimation for Unstable Stereo Camera Systems on AR Glasses

• Authors: Yongfan Liu, Hyoukjun Kwon
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10013
• Pdf URL: https://arxiv.org/pdf/2411.10013
• Copy Paste: [[2411.10013]] Efficient Depth Estimation for Unstable Stereo Camera Systems on AR Glasses(https://arxiv.org/abs/2411.10013)
• Keywords: robust
• Abstract: Stereo depth estimation is a fundamental component in augmented reality (AR) applications. Although AR applications require very low latency for their real-time applications, traditional depth estimation models often rely on time-consuming preprocessing steps such as rectification to achieve high accuracy. Also, non standard ML operator based algorithms such as cost volume also require significant latency, which is aggravated on compute resource-constrained mobile platforms. Therefore, we develop hardware-friendly alternatives to the costly cost volume and preprocessing and design two new models based on them, MultiHeadDepth and HomoDepth. Our approaches for cost volume is replacing it with a new group-pointwise convolution-based operator and approximation of consine similarity based on layernorm and dot product. For online stereo rectification (preprocessing), we introduce homograhy matrix prediction network with a rectification positional encoding (RPE), which delivers both low latency and robustness to unrectified images, which eliminates the needs for preprocessing. Our MultiHeadDepth, which includes optimized cost volume, provides 11.8-30.3% improvements in accuracy and 22.9-25.2% reduction in latency compared to a state-of-the-art depth estimation model for AR glasses from industry. Our HomoDepth, which includes optimized preprocessing (Homograhpy + RPE) upon MultiHeadDepth, can process unrectified images and reduce the end-to-end latency by 44.5%. We adopt a multi-task learning framework to handle misaligned stereo inputs on HomoDepth, which reduces theAbsRel error by 10.0-24.3%. The results demonstrate the efficacy of our approaches in achieving both high model performance with low latency, which makes a step forward toward practical depth estimation on future AR devices.

Title: Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models?

• Authors: Yan Hu, Xu Zuo, Yujia Zhou, Xueqing Peng, Jimin Huang, Vipina K. Keloth, Vincent J. Zhang, Ruey-Ling Weng, Qingyu Chen, Xiaoqian Jiang, Kirk E. Roberts, Hua Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10020
• Pdf URL: https://arxiv.org/pdf/2411.10020
• Copy Paste: [[2411.10020]] Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models?(https://arxiv.org/abs/2411.10020)
• Keywords: extraction, generative, large language model
• Abstract: Backgrounds: Information extraction (IE) is critical in clinical natural language processing (NLP). While large language models (LLMs) excel on generative tasks, their performance on extractive tasks remains debated. Methods: We investigated Named Entity Recognition (NER) and Relation Extraction (RE) using 1,588 clinical notes from four sources (UT Physicians, MTSamples, MIMIC-III, and i2b2). We developed an annotated corpus covering 4 clinical entities and 16 modifiers, and compared instruction-tuned LLaMA-2 and LLaMA-3 against BiomedBERT in terms of performance, generalizability, computational resources, and throughput to BiomedBERT. Results: LLaMA models outperformed BiomedBERT across datasets. With sufficient training data, LLaMA showed modest improvements (1% on NER, 1.5-3.7% on RE); improvements were larger with limited training data. On unseen i2b2 data, LLaMA-3-70B outperformed BiomedBERT by 7% (F1) on NER and 4% on RE. However, LLaMA models required more computing resources and ran up to 28 times slower. We implemented "Kiwi," a clinical IE package featuring both models, available at this https URL. Conclusion: This study is among the first to develop and evaluate a comprehensive clinical IE system using open-source LLMs. Results indicate that LLaMA models outperform BiomedBERT for clinical NER and RE but with higher computational costs and lower throughputs. These findings highlight that choosing between LLMs and traditional deep learning methods for clinical IE applications should remain task-specific, taking into account both performance metrics and practical considerations such as available computing resources and the intended use case scenarios.

Title: Model Inversion Attacks: A Survey of Approaches and Countermeasures

• Authors: Zhanke Zhou, Jianing Zhu, Fengfei Yu, Xuan Li, Xiong Peng, Tongliang Liu, Bo Han
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10023
• Pdf URL: https://arxiv.org/pdf/2411.10023
• Copy Paste: [[2411.10023]] Model Inversion Attacks: A Survey of Approaches and Countermeasures(https://arxiv.org/abs/2411.10023)
• Keywords: privacy, defense, attack
• Abstract: The success of deep neural networks has driven numerous research studies and applications from Euclidean to non-Euclidean data. However, there are increasing concerns about privacy leakage, as these networks rely on processing private data. Recently, a new type of privacy attack, the model inversion attacks (MIAs), aims to extract sensitive features of private data for training by abusing access to a well-trained model. The effectiveness of MIAs has been demonstrated in various domains, including images, texts, and graphs. These attacks highlight the vulnerability of neural networks and raise awareness about the risk of privacy leakage within the research community. Despite the significance, there is a lack of systematic studies that provide a comprehensive overview and deeper insights into MIAs across different domains. This survey aims to summarize up-to-date MIA methods in both attacks and defenses, highlighting their contributions and limitations, underlying modeling principles, optimization challenges, and future directions. We hope this survey bridges the gap in the literature and facilitates future research in this critical area. Besides, we are maintaining a repository to keep track of relevant research at this https URL.

Title: MOT\_FCG++: Enhanced Representation of Motion and Appearance Features

• Authors: Yanzhao Fang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10028
• Pdf URL: https://arxiv.org/pdf/2411.10028
• Copy Paste: [[2411.10028]] MOT\_FCG++: Enhanced Representation of Motion and Appearance Features(https://arxiv.org/abs/2411.10028)
• Keywords: robust
• Abstract: The goal of multi-object tracking (MOT) is to detect and track all objects in a scene across frames, while maintaining a unique identity for each object. Most existing methods rely on the spatial motion features and appearance embedding features of the detected objects in consecutive frames. Effectively and robustly representing the spatial and appearance features of long trajectories has become a critical factor affecting the performance of MOT. We propose a novel approach for appearance and spatial feature representation, improving upon the clustering association method MOT\_FCG. For spatial motion features, we propose Diagonal Modulated GIoU, which more accurately represents the relationship between the position and shape of the objects. For appearance features, we utilize a dynamic appearance representation that incorporates confidence information, enabling the trajectory appearance features to be more robust and global. Based on the baseline model MOT\_FCG, we achieved 76.1 HOTA, 80.4 MOTA and 81.3 IDF1 on the MOT17 validation set, and also achieved competitive performance on the MOT20 and DanceTrack validation sets.

Title: Toward Robust and Accurate Adversarial Camouflage Generation against Vehicle Detectors

• Authors: Jiawei Zhou, Linye Lyu, Daojing He, Yu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10029
• Pdf URL: https://arxiv.org/pdf/2411.10029
• Copy Paste: [[2411.10029]] Toward Robust and Accurate Adversarial Camouflage Generation against Vehicle Detectors(https://arxiv.org/abs/2411.10029)
• Keywords: attack, robust
• Abstract: Adversarial camouflage is a widely used physical attack against vehicle detectors for its superiority in multi-view attack performance. One promising approach involves using differentiable neural renderers to facilitate adversarial camouflage optimization through gradient back-propagation. However, existing methods often struggle to capture environmental characteristics during the rendering process or produce adversarial textures that can precisely map to the target vehicle. Moreover, these approaches neglect diverse weather conditions, reducing the efficacy of generated camouflage across varying weather scenarios. To tackle these challenges, we propose a robust and accurate camouflage generation method, namely RAUCA. The core of RAUCA is a novel neural rendering component, End-to-End Neural Renderer Plus (E2E-NRP), which can accurately optimize and project vehicle textures and render images with environmental characteristics such as lighting and weather. In addition, we integrate a multi-weather dataset for camouflage generation, leveraging the E2E-NRP to enhance the attack robustness. Experimental results on six popular object detectors show that RAUCA-final outperforms existing methods in both simulation and real-world settings.

Title: VMID: A Multimodal Fusion LLM Framework for Detecting and Identifying Misinformation of Short Videos

• Authors: Weihao Zhong, Yinhao Xiao, Minghui Xu, Xiuzhen Cheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10032
• Pdf URL: https://arxiv.org/pdf/2411.10032
• Copy Paste: [[2411.10032]] VMID: A Multimodal Fusion LLM Framework for Detecting and Identifying Misinformation of Short Videos(https://arxiv.org/abs/2411.10032)
• Keywords: robust, large language model
• Abstract: Short video platforms have become important channels for news dissemination, offering a highly engaging and immediate way for users to access current events and share information. However, these platforms have also emerged as significant conduits for the rapid spread of misinformation, as fake news and rumors can leverage the visual appeal and wide reach of short videos to circulate extensively among audiences. Existing fake news detection methods mainly rely on single-modal information, such as text or images, or apply only basic fusion techniques, limiting their ability to handle the complex, multi-layered information inherent in short videos. To address these limitations, this paper presents a novel fake news detection method based on multimodal information, designed to identify misinformation through a multi-level analysis of video content. This approach effectively utilizes different modal representations to generate a unified textual description, which is then fed into a large language model for comprehensive evaluation. The proposed framework successfully integrates multimodal features within videos, significantly enhancing the accuracy and reliability of fake news detection. Experimental results demonstrate that the proposed approach outperforms existing models in terms of accuracy, robustness, and utilization of multimodal information, achieving an accuracy of 90.93%, which is significantly higher than the best baseline model (SV-FEND) at 81.05%. Furthermore, case studies provide additional evidence of the effectiveness of the approach in accurately distinguishing between fake news, debunking content, and real incidents, highlighting its reliability and robustness in real-world applications.

Title: GSEditPro: 3D Gaussian Splatting Editing with Attention-based Progressive Localization

• Authors: Yanhao Sun, RunZe Tian, Xiao Han, XinYao Liu, Yan Zhang, Kai Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10033
• Pdf URL: https://arxiv.org/pdf/2411.10033
• Copy Paste: [[2411.10033]] GSEditPro: 3D Gaussian Splatting Editing with Attention-based Progressive Localization(https://arxiv.org/abs/2411.10033)
• Keywords: generative
• Abstract: With the emergence of large-scale Text-to-Image(T2I) models and implicit 3D representations like Neural Radiance Fields (NeRF), many text-driven generative editing methods based on NeRF have appeared. However, the implicit encoding of geometric and textural information poses challenges in accurately locating and controlling objects during editing. Recently, significant advancements have been made in the editing methods of 3D Gaussian Splatting, a real-time rendering technology that relies on explicit representation. However, these methods still suffer from issues including inaccurate localization and limited manipulation over editing. To tackle these challenges, we propose GSEditPro, a novel 3D scene editing framework which allows users to perform various creative and precise editing using text prompts only. Leveraging the explicit nature of the 3D Gaussian distribution, we introduce an attention-based progressive localization module to add semantic labels to each Gaussian during rendering. This enables precise localization on editing areas by classifying Gaussians based on their relevance to the editing prompts derived from cross-attention layers of the T2I model. Furthermore, we present an innovative editing optimization method based on 3D Gaussian Splatting, obtaining stable and refined editing results through the guidance of Score Distillation Sampling and pseudo ground truth. We prove the efficacy of our method through extensive experiments.

Title: EveGuard: Defeating Vibration-based Side-Channel Eavesdropping with Audio Adversarial Perturbations

• Authors: Jung-Woo Chang, Ke Sun, David Xia, Xinyu Zhang, Farinaz Koushanfar
• Subjects: cs.CR, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.10034
• Pdf URL: https://arxiv.org/pdf/2411.10034
• Copy Paste: [[2411.10034]] EveGuard: Defeating Vibration-based Side-Channel Eavesdropping with Audio Adversarial Perturbations(https://arxiv.org/abs/2411.10034)
• Keywords: privacy, protect, defense, attack
• Abstract: Vibrometry-based side channels pose a significant privacy risk, exploiting sensors like mmWave radars, light sensors, and accelerometers to detect vibrations from sound sources or proximate objects, enabling speech eavesdropping. Despite various proposed defenses, these involve costly hardware solutions with inherent physical limitations. This paper presents EveGuard, a software-driven defense framework that creates adversarial audio, protecting voice privacy from side channels without compromising human perception. We leverage the distinct sensing capabilities of side channels and traditional microphones where side channels capture vibrations and microphones record changes in air pressure, resulting in different frequency responses. EveGuard first proposes a perturbation generator model (PGM) that effectively suppresses sensor-based eavesdropping while maintaining high audio quality. Second, to enable end-to-end training of PGM, we introduce a new domain translation task called Eve-GAN for inferring an eavesdropped signal from a given audio. We further apply few-shot learning to mitigate the data collection overhead for Eve-GAN training. Our extensive experiments show that EveGuard achieves a protection rate of more than 97 percent from audio classifiers and significantly hinders eavesdropped audio reconstruction. We further validate the performance of EveGuard across three adaptive attack mechanisms. We have conducted a user study to verify the perceptual quality of our perturbed audio.

Title: Physics-informed neural networks need a physicist to be accurate: the case of mass and heat transport in Fischer-Tropsch catalyst particles

• Authors: Tymofii Nikolaienko, Harshil Patel, Aniruddha Panda, Subodh Madhav Joshi, Stanislav Jaso, Kaushic Kalyanaraman
• Subjects: cs.LG, cs.AI, physics.chem-ph, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2411.10048
• Pdf URL: https://arxiv.org/pdf/2411.10048
• Copy Paste: [[2411.10048]] Physics-informed neural networks need a physicist to be accurate: the case of mass and heat transport in Fischer-Tropsch catalyst particles(https://arxiv.org/abs/2411.10048)
• Keywords: diffusion
• Abstract: Physics-Informed Neural Networks (PINNs) have emerged as an influential technology, merging the swift and automated capabilities of machine learning with the precision and dependability of simulations grounded in theoretical physics. PINNs are often employed to solve algebraic or differential equations to replace some or even all steps of multi-stage computational workflows, leading to their significant speed-up. However, wide adoption of PINNs is still hindered by reliability issues, particularly at extreme ends of the input parameter ranges. In this study, we demonstrate this in the context of a system of coupled non-linear differential reaction-diffusion and heat transfer equations related to Fischer-Tropsch synthesis, which are solved by a finite-difference method with a PINN used in evaluating their source terms. It is shown that the testing strategies traditionally used to assess the accuracy of neural networks as function approximators can overlook the peculiarities which ultimately cause instabilities of the finite-difference solver. We propose a domain knowledge-based modifications to the PINN architecture ensuring its correct asymptotic behavior. When combined with an improved numerical scheme employed as an initial guess generator, the proposed modifications are shown to recover the overall stability of the simulations, while preserving the speed-up brought by PINN as the workflow component. We discuss the possible applications of the proposed hybrid transport equation solver in context of chemical reactors simulations.

Title: Jal Anveshak: Prediction of fishing zones using fine-tuned LlaMa 2

• Authors: Arnav Mejari, Maitreya Vaghulade, Paarshva Chitaliya, Arya Telang, Lynette D'mello
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10050
• Pdf URL: https://arxiv.org/pdf/2411.10050
• Copy Paste: [[2411.10050]] Jal Anveshak: Prediction of fishing zones using fine-tuned LlaMa 2(https://arxiv.org/abs/2411.10050)
• Keywords: large language model
• Abstract: In recent years, the global and Indian government efforts in monitoring and collecting data related to the fisheries industry have witnessed significant advancements. Despite this wealth of data, there exists an untapped potential for leveraging artificial intelligence based technological systems to benefit Indian fishermen in coastal areas. To fill this void in the Indian technology ecosystem, the authors introduce Jal Anveshak. This is an application framework written in Dart and Flutter that uses a Llama 2 based Large Language Model fine-tuned on pre-processed and augmented government data related to fishing yield and availability. Its main purpose is to help Indian fishermen safely get the maximum yield of fish from coastal areas and to resolve their fishing related queries in multilingual and multimodal ways.

Title: Self-Defense: Optimal QIF Solutions and Application to Website Fingerprinting

• Authors: Andreas Athanasiou (COMETE, LIX), Konstantinos Chatzikokolakis (NKUA), Catuscia Palamidessi (COMETE, LIX)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.10059
• Pdf URL: https://arxiv.org/pdf/2411.10059
• Copy Paste: [[2411.10059]] Self-Defense: Optimal QIF Solutions and Application to Website Fingerprinting(https://arxiv.org/abs/2411.10059)
• Keywords: secure, defense, attack, robust
• Abstract: Quantitative Information Flow (QIF) provides a robust information-theoretical framework for designing secure systems with minimal information leakage. While previous research has addressed the design of such systems under hard constraints (e.g. application limitations) and soft constraints (e.g. utility), scenarios often arise where the core system's behavior is considered fixed. In such cases, the challenge is to design a new component for the existing system that minimizes leakage without altering the original system. In this work we address this problem by proposing optimal solutions for constructing a new row, in a known and unmodifiable information-theoretic channel, aiming at minimizing the leakage. We first model two types of adversaries: an exact-guessing adversary, aiming to guess the secret in one try, and a s-distinguishing one, which tries to distinguish the secret s from all the other this http URL, we discuss design strategies for both fixed and unknown priors by offering, for each adversary, an optimal solution under linear constraints, using Linear this http URL apply our approach to the problem of website fingerprinting defense, considering a scenario where a site administrator can modify their own site but not others. We experimentally evaluate our proposed solutions against other natural approaches. First, we sample real-world news websites and then, for both adversaries, we demonstrate that the proposed solutions are effective in achieving the least leakage. Finally, we simulate an actual attack by training an ML classifier for the s-distinguishing adversary and show that our approach decreases the accuracy of the attacker.

Title: Layer Importance and Hallucination Analysis in Large Language Models via Enhanced Activation Variance-Sparsity

• Authors: Zichen Song, Sitan Huang, Yuxin Wu, Zhongfeng Kang
• Subjects: cs.CL, cs.PF
• Abstract URL: https://arxiv.org/abs/2411.10069
• Pdf URL: https://arxiv.org/pdf/2411.10069
• Copy Paste: [[2411.10069]] Layer Importance and Hallucination Analysis in Large Language Models via Enhanced Activation Variance-Sparsity(https://arxiv.org/abs/2411.10069)
• Keywords: robust, interpretability, large language model
• Abstract: Evaluating the importance of different layers in large language models (LLMs) is crucial for optimizing model performance and interpretability. This paper first explores layer importance using the Activation Variance-Sparsity Score (AVSS), which combines normalized activation variance and sparsity to quantify each layer's contribution to overall model performance. By ranking layers based on AVSS and pruning the least impactful 25\%, our experiments on tasks such as question answering, language modeling, and sentiment classification show that over 90\% of the original performance is retained, highlighting potential redundancies in LLM architectures. Building on AVSS, we propose an enhanced version tailored to assess hallucination propensity across layers (EAVSS). This improved approach introduces Hallucination-Specific Activation Variance (HSAV) and Hallucination-Specific Sparsity (HSS) metrics, allowing precise identification of hallucination-prone layers. By incorporating contrastive learning on these layers, we effectively mitigate hallucination generation, contributing to more robust and efficient LLMs(The maximum performance improvement is 12\%). Our results on the NQ, SciQ, TriviaQA, TruthfulQA, and WikiQA datasets demonstrate the efficacy of this method, offering a comprehensive framework for both layer importance evaluation and hallucination mitigation in LLMs.

Title: Evidential Federated Learning for Skin Lesion Image Classification

• Authors: Rutger Hendrix, Federica Proietto Salanitri, Concetto Spampinato, Simone Palazzo, Ulas Bagci
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10071
• Pdf URL: https://arxiv.org/pdf/2411.10071
• Copy Paste: [[2411.10071]] Evidential Federated Learning for Skin Lesion Image Classification(https://arxiv.org/abs/2411.10071)
• Keywords: privacy, federate, transformer
• Abstract: We introduce FedEvPrompt, a federated learning approach that integrates principles of evidential deep learning, prompt tuning, and knowledge distillation for distributed skin lesion classification. FedEvPrompt leverages two sets of prompts: b-prompts (for low-level basic visual knowledge) and t-prompts (for task-specific knowledge) prepended to frozen pre-trained Vision Transformer (ViT) models trained in an evidential learning framework to maximize class evidences. Crucially, knowledge sharing across federation clients is achieved only through knowledge distillation on attention maps generated by the local ViT models, ensuring enhanced privacy preservation compared to traditional parameter or synthetic image sharing methodologies. FedEvPrompt is optimized within a round-based learning paradigm, where each round involves training local models followed by attention maps sharing with all federation clients. Experimental validation conducted in a real distributed setting, on the ISIC2019 dataset, demonstrates the superior performance of FedEvPrompt against baseline federated learning algorithms and knowledge distillation methods, without sharing model parameters. In conclusion, FedEvPrompt offers a promising approach for federated learning, effectively addressing challenges such as data heterogeneity, imbalance, privacy preservation, and knowledge sharing.

Title: Uncertainty-Weighted Mutual Distillation for Multi-View Fusion

• Authors: Jiwoong Yang, Haejun Chung, Ikbeom Jang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10077
• Pdf URL: https://arxiv.org/pdf/2411.10077
• Copy Paste: [[2411.10077]] Uncertainty-Weighted Mutual Distillation for Multi-View Fusion(https://arxiv.org/abs/2411.10077)
• Keywords: robust, transformer
• Abstract: Multi-view learning often faces challenges in effectively leveraging images captured from different angles and locations. This challenge is particularly pronounced when addressing inconsistencies and uncertainties between views. In this paper, we propose a novel Multi-View Uncertainty-Weighted Mutual Distillation (MV-UWMD) method. Our method enhances prediction consistency by performing hierarchical mutual distillation across all possible view combinations, including single-view, partial multi-view, and full multi-view predictions. This introduces an uncertainty-based weighting mechanism through mutual distillation, allowing effective exploitation of unique information from each view while mitigating the impact of uncertain predictions. We extend a CNN-Transformer hybrid architecture to facilitate robust feature learning and integration across multiple view combinations. We conducted extensive experiments using a large, unstructured dataset captured from diverse, non-fixed viewpoints. The results demonstrate that MV-UWMD improves prediction accuracy and consistency compared to existing multi-view learning approaches.

Title: CorrCLIP: Reconstructing Correlations in CLIP with Off-the-Shelf Foundation Models for Open-Vocabulary Semantic Segmentation

• Authors: Dengke Zhang, Fagui Liu, Quan Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10086
• Pdf URL: https://arxiv.org/pdf/2411.10086
• Copy Paste: [[2411.10086]] CorrCLIP: Reconstructing Correlations in CLIP with Off-the-Shelf Foundation Models for Open-Vocabulary Semantic Segmentation(https://arxiv.org/abs/2411.10086)
• Keywords: segmentation
• Abstract: Open-vocabulary semantic segmentation aims to assign semantic labels to each pixel without relying on a predefined set of categories. Contrastive Language-Image Pre-training (CLIP) demonstrates outstanding zero-shot classification capabilities but struggles with the pixel-wise segmentation task as the captured inter-patch correlations correspond to no specific visual concepts. Despite previous CLIP-based works improving inter-patch correlations by self-self attention, they still face the inherent limitation that image patches tend to have high similarity to outlier ones. In this work, we introduce CorrCLIP, a training-free approach for open-vocabulary semantic segmentation, which reconstructs significantly coherent inter-patch correlations utilizing foundation models. Specifically, it employs the Segment Anything Model (SAM) to define the scope of patch interactions, ensuring that patches interact only with semantically similar ones. Furthermore, CorrCLIP obtains an understanding of an image's semantic layout via self-supervised models to determine concrete similarity values between image patches, which addresses the similarity irregularity problem caused by the aforementioned restricted patch interaction regime. Finally, CorrCLIP reuses the region masks produced by SAM to update the segmentation map. As a training-free method, CorrCLIP achieves a notable improvement across eight challenging benchmarks regarding the averaged mean Intersection over Union, boosting it from 44.4% to 51.0%.

Title: Towards Multi-View Consistent Style Transfer with One-Step Diffusion via Vision Conditioning

• Authors: Yushen Zuo, Jun Xiao, Kin-Chung Chan, Rongkang Dong, Cuixin Yang, Zongqi He, Hao Xie, Kin-Man Lam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10130
• Pdf URL: https://arxiv.org/pdf/2411.10130
• Copy Paste: [[2411.10130]] Towards Multi-View Consistent Style Transfer with One-Step Diffusion via Vision Conditioning(https://arxiv.org/abs/2411.10130)
• Keywords: diffusion, generative
• Abstract: The stylization of 3D scenes is an increasingly attractive topic in 3D vision. Although image style transfer has been extensively researched with promising results, directly applying 2D style transfer methods to 3D scenes often fails to preserve the structural and multi-view properties of 3D environments, resulting in unpleasant distortions in images from different viewpoints. To address these issues, we leverage the remarkable generative prior of diffusion-based models and propose a novel style transfer method, OSDiffST, based on a pre-trained one-step diffusion model (i.e., SD-Turbo) for rendering diverse styles in multi-view images of 3D scenes. To efficiently adapt the pre-trained model for multi-view style transfer on small datasets, we introduce a vision condition module to extract style information from the reference style image to serve as conditional input for the diffusion model and employ LoRA in diffusion model for adaptation. Additionally, we consider color distribution alignment and structural similarity between the stylized and content images using two specific loss functions. As a result, our method effectively preserves the structural information and multi-view consistency in stylized images without any 3D information. Experiments show that our method surpasses other promising style transfer methods in synthesizing various styles for multi-view images of 3D scenes. Stylized images from different viewpoints generated by our method achieve superior visual quality, with better structural integrity and less distortion. The source code is available at this https URL.

Title: Omnichain Web: The Universal Framework for Streamlined Chain Abstraction and Cross-Layer Interaction

• Authors: Hardik Gajera, Akhil Reddy, Bhagath Reddy
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.10132
• Pdf URL: https://arxiv.org/pdf/2411.10132
• Copy Paste: [[2411.10132]] Omnichain Web: The Universal Framework for Streamlined Chain Abstraction and Cross-Layer Interaction(https://arxiv.org/abs/2411.10132)
• Keywords: secure
• Abstract: The evolution of the Web3 ecosystem has been hindered by fragmented liquidity and limited interoperability across Layer 1 (L1) and Layer 2 (L2) blockchains, which leads to inefficiencies and elevated costs. Omnichain Web addresses these challenges by introducing a comprehensive framework to unify decentralized networks through its core components: OmniRollups, Proof Network, Ragno Network, and Builder Marketplace. This ecosystem enables seamless cross-chain asset settlement, interoperability, and user-friendly decentralized application (dApp) development, driven by innovative technologies such as modular proof networks and trusted execution environments (TEEs). By integrating advanced zero-knowledge proof systems and compatibility with AI agents, Omnichain Web empowers intent-driven and autonomous functionalities, streamlining liquidity management and user interactions across blockchains. Furthermore, its decentralized marketplace for L1 infrastructure reduces operational overhead and promotes scalable, secure, and efficient cross-chain protocols. As a pioneering solution, Omnichain Web seamlessly connects Web2 and Web3, enabling a holistic and interconnected digital economy.

Title: CoSAM: Self-Correcting SAM for Domain Generalization in 2D Medical Image Segmentation

• Authors: Yihang Fu, Ziyang Chen, Yiwen Ye, Xingliang Lei, Zhisong Wang, Yong Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10136
• Pdf URL: https://arxiv.org/pdf/2411.10136
• Copy Paste: [[2411.10136]] CoSAM: Self-Correcting SAM for Domain Generalization in 2D Medical Image Segmentation(https://arxiv.org/abs/2411.10136)
• Keywords: segmentation
• Abstract: Medical images often exhibit distribution shifts due to variations in imaging protocols and scanners across different medical centers. Domain Generalization (DG) methods aim to train models on source domains that can generalize to unseen target domains. Recently, the segment anything model (SAM) has demonstrated strong generalization capabilities due to its prompt-based design, and has gained significant attention in image segmentation tasks. Existing SAM-based approaches attempt to address the need for manual prompts by introducing prompt generators that automatically generate these prompts. However, we argue that auto-generated prompts may not be sufficiently accurate under distribution shifts, potentially leading to incorrect predictions that still require manual verification and correction by clinicians. To address this challenge, we propose a method for 2D medical image segmentation called Self-Correcting SAM (CoSAM). Our approach begins by generating coarse masks using SAM in a prompt-free manner, providing prior prompts for the subsequent stages, and eliminating the need for prompt generators. To automatically refine these coarse masks, we introduce a generalized error decoder that simulates the correction process typically performed by clinicians. Furthermore, we generate diverse prompts as feedback based on the corrected masks, which are used to iteratively refine the predictions within a self-correcting loop, enhancing the generalization performance of our model. Extensive experiments on two medical image segmentation benchmarks across multiple scenarios demonstrate the superiority of CoSAM over state-of-the-art SAM-based methods.

Title: Legal Evalutions and Challenges of Large Language Models

• Authors: Jiaqi Wang, Huan Zhao, Zhenyuan Yang, Peng Shu, Junhao Chen, Haobo Sun, Ruixi Liang, Shixin Li, Pengcheng Shi, Longjun Ma, Zongjia Liu, Zhengliang Liu, Tianyang Zhong, Yutong Zhang, Chong Ma, Xin Zhang, Tuo Zhang, Tianli Ding, Yudan Ren, Tianming Liu, Xi Jiang, Shu Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10137
• Pdf URL: https://arxiv.org/pdf/2411.10137
• Copy Paste: [[2411.10137]] Legal Evalutions and Challenges of Large Language Models(https://arxiv.org/abs/2411.10137)
• Keywords: large language model
• Abstract: In this paper, we review legal testing methods based on Large Language Models (LLMs), using the OPENAI o1 model as a case study to evaluate the performance of large models in applying legal provisions. We compare current state-of-the-art LLMs, including open-source, closed-source, and legal-specific models trained specifically for the legal domain. Systematic tests are conducted on English and Chinese legal cases, and the results are analyzed in depth. Through systematic testing of legal cases from common law systems and China, this paper explores the strengths and weaknesses of LLMs in understanding and applying legal texts, reasoning through legal issues, and predicting judgments. The experimental results highlight both the potential and limitations of LLMs in legal applications, particularly in terms of challenges related to the interpretation of legal language and the accuracy of legal reasoning. Finally, the paper provides a comprehensive analysis of the advantages and disadvantages of various types of models, offering valuable insights and references for the future application of AI in the legal field.

Title: An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks

• Authors: Yijiong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10145
• Pdf URL: https://arxiv.org/pdf/2411.10145
• Copy Paste: [[2411.10145]] An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks(https://arxiv.org/abs/2411.10145)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long texts and have almost perfect performance in traditional retrieval tasks. However, their performance significantly degrades when it comes to numerical calculations in the long-context. Numeric-involved long-context tasks typically cannot be addressed by current LLMs in normal settings due to their inherent limitations in simultaneously handling complex and massive information. Some CoT like prompting methods can improve accuracy but demands massive output tokens, which is costly and slow. To address this issue, we propose a workflow, which decompose a numeric-involved long-context task into 4 low-level subtasks: judging, extracting and processing with code and conclusion. The former 2 subtasks is relatively simple, which allows us to use smaller models for efficiently processing long context. When numerical calculations are required, we use code generated by LLMs to avoid the disadvantage of LLM not being good at calculations. The results in 2 numeric-involved long-context benchmarks demonstrate our workflow can not only improve accuracy, but also significantly reduce the cost of API calls.

Title: Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions

• Authors: Yutao Hou, Yajing Luo, Zhiwen Ruan, Hongru Wang, Weifeng Ge, Yun Chen, Guanhua Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10163
• Pdf URL: https://arxiv.org/pdf/2411.10163
• Copy Paste: [[2411.10163]] Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions(https://arxiv.org/abs/2411.10163)
• Keywords: large language model
• Abstract: Large language models (LLMs) demonstrate remarkable performance across various tasks, prompting researchers to develop diverse evaluation benchmarks. However, existing benchmarks typically measure the ability of LLMs to respond to individual questions, neglecting the complex interactions in real-world applications. In this paper, we introduce Compound Question Synthesis (CQ-Syn) to create the Compound-QA benchmark, focusing on compound questions with multiple sub-questions. This benchmark is derived from existing QA datasets, annotated with proprietary LLMs and verified by humans for accuracy. It encompasses five categories: Factual-Statement, Cause-and-Effect, Hypothetical-Analysis, Comparison-and-Selection, and Evaluation-and-Suggestion. It evaluates the LLM capability in terms of three dimensions including understanding, reasoning, and knowledge. Our assessment of eight open-source LLMs using Compound-QA reveals distinct patterns in their responses to compound questions, which are significantly poorer than those to non-compound questions. Additionally, we investigate various methods to enhance LLMs performance on compound questions. The results indicate that these approaches significantly improve the models' comprehension and reasoning abilities on compound questions.

Title: Increasing the Accessibility of Causal Domain Knowledge via Causal Information Extraction Methods: A Case Study in the Semiconductor Manufacturing Industry

• Authors: Houssam Razouk, Leonie Benischke, Daniel Garber, Roman Kern
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10172
• Pdf URL: https://arxiv.org/pdf/2411.10172
• Copy Paste: [[2411.10172]] Increasing the Accessibility of Causal Domain Knowledge via Causal Information Extraction Methods: A Case Study in the Semiconductor Manufacturing Industry(https://arxiv.org/abs/2411.10172)
• Keywords: extraction
• Abstract: The extraction of causal information from textual data is crucial in the industry for identifying and mitigating potential failures, enhancing process efficiency, prompting quality improvements, and addressing various operational challenges. This paper presents a study on the development of automated methods for causal information extraction from actual industrial documents in the semiconductor manufacturing industry. The study proposes two types of causal information extraction methods, single-stage sequence tagging (SST) and multi-stage sequence tagging (MST), and evaluates their performance using existing documents from a semiconductor manufacturing company, including presentation slides and FMEA (Failure Mode and Effects Analysis) documents. The study also investigates the effect of representation learning on downstream tasks. The presented case study showcases that the proposed MST methods for extracting causal information from industrial documents are suitable for practical applications, especially for semi structured documents such as FMEAs, with a 93\% F1 score. Additionally, MST achieves a 73\% F1 score on texts extracted from presentation slides. Finally, the study highlights the importance of choosing a language model that is more aligned with the domain and in-domain fine-tuning.

Title: A Hard-Label Cryptanalytic Extraction of Non-Fully Connected Deep Neural Networks using Side-Channel Attacks

• Authors: Benoit Coqueret, Mathieu Carbone, Olivier Sentieys, Gabriel Zaid
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10174
• Pdf URL: https://arxiv.org/pdf/2411.10174
• Copy Paste: [[2411.10174]] A Hard-Label Cryptanalytic Extraction of Non-Fully Connected Deep Neural Networks using Side-Channel Attacks(https://arxiv.org/abs/2411.10174)
• Keywords: protect, attack, extraction
• Abstract: During the past decade, Deep Neural Networks (DNNs) proved their value on a large variety of subjects. However despite their high value and public accessibility, the protection of the intellectual property of DNNs is still an issue and an emerging research field. Recent works have successfully extracted fully-connected DNNs using cryptanalytic methods in hard-label settings, proving that it was possible to copy a DNN with high fidelity, i.e., high similitude in the output predictions. However, the current cryptanalytic attacks cannot target complex, i.e., not fully connected, DNNs and are limited to special cases of neurons present in deep networks. In this work, we introduce a new end-to-end attack framework designed for model extraction of embedded DNNs with high fidelity. We describe a new black-box side-channel attack which splits the DNN in several linear parts for which we can perform cryptanalytic extraction and retrieve the weights in hard-label settings. With this method, we are able to adapt cryptanalytic extraction, for the first time, to non-fully connected DNNs, while maintaining a high fidelity. We validate our contributions by targeting several architectures implemented on a microcontroller unit, including a Multi-Layer Perceptron (MLP) of 1.7 million parameters and a shortened MobileNetv1. Our framework successfully extracts all of these DNNs with high fidelity (88.4% for the MobileNetv1 and 93.2% for the MLP). Furthermore, we use the stolen model to generate adversarial examples and achieve close to white-box performance on the victim's model (95.8% and 96.7% transfer rate).

Title: CART: Compositional Auto-Regressive Transformer for Image Generation

• Authors: Siddharth Roheda
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10180
• Pdf URL: https://arxiv.org/pdf/2411.10180
• Copy Paste: [[2411.10180]] CART: Compositional Auto-Regressive Transformer for Image Generation(https://arxiv.org/abs/2411.10180)
• Keywords: transformer
• Abstract: In recent years, image synthesis has achieved remarkable advancements, enabling diverse applications in content creation, virtual reality, and beyond. We introduce a novel approach to image generation using Auto-Regressive (AR) modeling, which leverages a next-detail prediction strategy for enhanced fidelity and scalability. While AR models have achieved transformative success in language modeling, replicating this success in vision tasks has presented unique challenges due to the inherent spatial dependencies in images. Our proposed method addresses these challenges by iteratively adding finer details to an image compositionally, constructing it as a hierarchical combination of base and detail image factors. This strategy is shown to be more effective than the conventional next-token prediction and even surpasses the state-of-the-art next-scale prediction approaches. A key advantage of this method is its scalability to higher resolutions without requiring full model retraining, making it a versatile solution for high-resolution image generation.

Title: Reachability Analysis of the Domain Name System

• Authors: Dhruv Nevatia, Si Liu, David Basin
• Subjects: cs.CR, cs.FL
• Abstract URL: https://arxiv.org/abs/2411.10188
• Pdf URL: https://arxiv.org/pdf/2411.10188
• Copy Paste: [[2411.10188]] Reachability Analysis of the Domain Name System(https://arxiv.org/abs/2411.10188)
• Keywords: security, attack
• Abstract: The high complexity of DNS poses unique challenges for ensuring its security and reliability. Despite continuous advances in DNS testing, monitoring, and verification, protocol-level defects still give rise to numerous bugs and attacks. In this paper, we provide the first decision procedure for the DNS verification problem, establishing its complexity as $\mathsf{2ExpTime}$, which was previously unknown. We begin by formalizing the semantics of DNS as a system of recursive communicating processes extended with timers and an infinite message alphabet. We provide an algebraic abstraction of the alphabet with finitely many equivalence classes, using the subclass of semigroups that recognize positive prefix-testable languages. We then introduce a novel generalization of bisimulation for labelled transition systems, weaker than strong bisimulation, to show that our abstraction is sound and complete. Finally, using this abstraction, we reduce the DNS verification problem to the verification problem for pushdown systems. To show the expressiveness of our framework, we model two of the most prominent attack vectors on DNS, namely amplification attacks and rewrite blackholing.

Title: NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics

• Authors: Chenhao Li, Taishi Ono, Takeshi Uemori, Sho Nitta, Hajime Mihara, Alexander Gatto, Hajime Nagahara, Yusuke Moriuchi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10189
• Pdf URL: https://arxiv.org/pdf/2411.10189
• Copy Paste: [[2411.10189]] NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics(https://arxiv.org/abs/2411.10189)
• Keywords: robust
• Abstract: Recent inverse rendering methods have greatly improved shape, material, and illumination reconstruction by utilizing polarization cues. However, existing methods only support dielectrics, ignoring conductors that are found everywhere in life. Since conductors and dielectrics have different reflection properties, using previous conductor methods will lead to obvious errors. In addition, conductors are glossy, which may cause strong specular reflection and is hard to reconstruct. To solve the above issues, we propose NeISF++, an inverse rendering pipeline that supports conductors and dielectrics. The key ingredient for our proposal is a general pBRDF that describes both conductors and dielectrics. As for the strong specular reflection problem, we propose a novel geometry initialization method using DoLP images. This physical cue is invariant to intensities and thus robust to strong specular reflections. Experimental results on our synthetic and real datasets show that our method surpasses the existing polarized inverse rendering methods for geometry and material decomposition as well as downstream tasks like relighting.

Title: DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization

• Authors: Christos Koutlis, Symeon Papadopoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10193
• Pdf URL: https://arxiv.org/pdf/2411.10193
• Copy Paste: [[2411.10193]] DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization(https://arxiv.org/abs/2411.10193)
• Keywords: transformer
• Abstract: Deepfake technology has rapidly advanced, posing significant threats to information integrity and societal trust. While significant progress has been made in detecting deepfakes, the simultaneous manipulation of audio and visual modalities, sometimes at small parts but still altering the meaning, presents a more challenging detection scenario. We present a novel audio-visual deepfake detection framework that leverages the inter-modality differences in machine perception of speech, based on the assumption that in real samples - in contrast to deepfakes - visual and audio signals coincide in terms of information. Our framework leverages features from deep networks that specialize in video and audio speech recognition to spot frame-level cross-modal incongruities, and in that way to temporally localize the deepfake forgery. To this end, DiMoDif employs a Transformer encoder-based architecture with a feature pyramid scheme and local attention, and optimizes the detection model through a composite loss function accounting for frame-level detections and fake intervals localization. DiMoDif outperforms the state-of-the-art on the Temporal Forgery Localization task by +47.88% AP@0.75 on AV-Deepfake1M, and performs on-par on LAV-DF. On the Deepfake Detection task, it outperforms the state-of-the-art by +30.5% AUC on AV-Deepfake1M, +2.8% AUC on FakeAVCeleb, and performs on-par on LAV-DF. Code available at this https URL.

Title: Learning Generalizable 3D Manipulation With 10 Demonstrations

• Authors: Yu Ren, Yang Cong, Ronghan Chen, Jiahao Long
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.10203
• Pdf URL: https://arxiv.org/pdf/2411.10203
• Copy Paste: [[2411.10203]] Learning Generalizable 3D Manipulation With 10 Demonstrations(https://arxiv.org/abs/2411.10203)
• Keywords: robust, diffusion
• Abstract: Learning robust and generalizable manipulation skills from demonstrations remains a key challenge in robotics, with broad applications in industrial automation and service robotics. While recent imitation learning methods have achieved impressive results, they often require large amounts of demonstration data and struggle to generalize across different spatial variants. In this work, we present a novel framework that learns manipulation skills from as few as 10 demonstrations, yet still generalizes to spatial variants such as different initial object positions and camera viewpoints. Our framework consists of two key modules: Semantic Guided Perception (SGP), which constructs task-focused, spatially aware 3D point cloud representations from RGB-D inputs; and Spatial Generalized Decision (SGD), an efficient diffusion-based decision-making module that generates actions via denoising. To effectively learn generalization ability from limited data, we introduce a critical spatially equivariant training strategy that captures the spatial knowledge embedded in expert demonstrations. We validate our framework through extensive experiments on both simulation benchmarks and real-world robotic systems. Our method demonstrates a 60 percent improvement in success rates over state-of-the-art approaches on a series of challenging tasks, even with substantial variations in object poses and camera viewpoints. This work shows significant potential for advancing efficient, generalizable manipulation skill learning in real-world applications.

Title: Embedding Byzantine Fault Tolerance into Federated Learning via Virtual Data-Driven Consistency Scoring Plugin

• Authors: Youngjoon Lee, Jinu Gong, Joonhyuk Kang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10212
• Pdf URL: https://arxiv.org/pdf/2411.10212
• Copy Paste: [[2411.10212]] Embedding Byzantine Fault Tolerance into Federated Learning via Virtual Data-Driven Consistency Scoring Plugin(https://arxiv.org/abs/2411.10212)
• Keywords: attack, robust, federate
• Abstract: Given sufficient data from multiple edge devices, federated learning (FL) enables training a shared model without transmitting private data to a central server. However, FL is generally vulnerable to Byzantine attacks from compromised edge devices, which can significantly degrade the model performance. In this paper, we propose a intuitive plugin that can be integrated into existing FL techniques to achieve Byzantine-Resilience. Key idea is to generate virtual data samples and evaluate model consistency scores across local updates to effectively filter out compromised edge devices. By utilizing this scoring mechanism before the aggregation phase, the proposed plugin enables existing FL techniques to become robust against Byzantine attacks while maintaining their original benefits. Numerical results on medical image classification task validate that plugging the proposed approach into representative FL algorithms, effectively achieves Byzantine resilience. Furthermore, the proposed plugin maintains the original convergence properties of the base FL algorithms when no Byzantine attacks are present.

Title: A Low-Resolution Image is Worth 1x1 Words: Enabling Fine Image Super-Resolution with Transformers and TaylorShift

• Authors: Sanath Budakegowdanadoddi Nagaraju, Brian Bernhard Moser, Tobias Christian Nauen, Stanislav Frolov, Federico Raue, Andreas Dengel
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.10231
• Pdf URL: https://arxiv.org/pdf/2411.10231
• Copy Paste: [[2411.10231]] A Low-Resolution Image is Worth 1x1 Words: Enabling Fine Image Super-Resolution with Transformers and TaylorShift(https://arxiv.org/abs/2411.10231)
• Keywords: transformer
• Abstract: Transformer-based Super-Resolution (SR) models have recently advanced image reconstruction quality, yet challenges remain due to computational complexity and an over-reliance on large patch sizes, which constrain fine-grained detail enhancement. In this work, we propose TaylorIR to address these limitations by utilizing a patch size of 1x1, enabling pixel-level processing in any transformer-based SR model. To address the significant computational demands under the traditional self-attention mechanism, we employ the TaylorShift attention mechanism, a memory-efficient alternative based on Taylor series expansion, achieving full token-to-token interactions with linear complexity. Experimental results demonstrate that our approach achieves new state-of-the-art SR performance while reducing memory consumption by up to 60% compared to traditional self-attention-based transformers.

Title: ColorEdit: Training-free Image-Guided Color editing with diffusion model

• Authors: Xingxi Yin, Zhi Li, Jingfeng Zhang, Chenglin Li, Yin Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10232
• Pdf URL: https://arxiv.org/pdf/2411.10232
• Copy Paste: [[2411.10232]] ColorEdit: Training-free Image-Guided Color editing with diffusion model(https://arxiv.org/abs/2411.10232)
• Keywords: diffusion, generative
• Abstract: Text-to-image (T2I) diffusion models, with their impressive generative capabilities, have been adopted for image editing tasks, demonstrating remarkable efficacy. However, due to attention leakage and collision between the cross-attention map of the object and the new color attribute from the text prompt, text-guided image editing methods may fail to change the color of an object, resulting in a misalignment between the resulting image and the text prompt. In this paper, we conduct an in-depth analysis on the process of text-guided image synthesizing and what semantic information different cross-attention blocks have learned. We observe that the visual representation of an object is determined in the up-block of the diffusion model in the early stage of the denoising process, and color adjustment can be achieved through value matrices alignment in the cross-attention layer. Based on our findings, we propose a straightforward, yet stable, and effective image-guided method to modify the color of an object without requiring any additional fine-tuning or training. Lastly, we present a benchmark dataset called COLORBENCH, the first benchmark to evaluate the performance of color change methods. Extensive experiments validate the effectiveness of our method in object-level color editing and surpass the performance of popular text-guided image editing approaches in both synthesized and real images.

Title: ScribbleVS: Scribble-Supervised Medical Image Segmentation via Dynamic Competitive Pseudo Label Selection

• Authors: Tao Wang, Xinlin Zhang, Yuanbin Chen, Yuanbo Zhou, Longxuan Zhao, Tao Tan, Tong Tong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10237
• Pdf URL: https://arxiv.org/pdf/2411.10237
• Copy Paste: [[2411.10237]] ScribbleVS: Scribble-Supervised Medical Image Segmentation via Dynamic Competitive Pseudo Label Selection(https://arxiv.org/abs/2411.10237)
• Keywords: diffusion, segmentation
• Abstract: In clinical medicine, precise image segmentation can provide substantial support to clinicians. However, achieving such precision often requires a large amount of finely annotated data, which can be costly. Scribble annotation presents a more efficient alternative, boosting labeling efficiency. However, utilizing such minimal supervision for medical image segmentation training, especially with scribble annotations, poses significant challenges. To address these challenges, we introduce ScribbleVS, a novel framework that leverages scribble annotations. We introduce a Regional Pseudo Labels Diffusion Module to expand the scope of supervision and reduce the impact of noise present in pseudo labels. Additionally, we propose a Dynamic Competitive Selection module for enhanced refinement in selecting pseudo labels. Experiments conducted on the ACDC and MSCMRseg datasets have demonstrated promising results, achieving performance levels that even exceed those of fully supervised methodologies. The codes of this study are available at this https URL.

Title: Measuring Non-Adversarial Reproduction of Training Data in Large Language Models

• Authors: Michael Aerni, Javier Rando, Edoardo Debenedetti, Nicholas Carlini, Daphne Ippolito, Florian Tramèr
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10242
• Pdf URL: https://arxiv.org/pdf/2411.10242
• Copy Paste: [[2411.10242]] Measuring Non-Adversarial Reproduction of Training Data in Large Language Models(https://arxiv.org/abs/2411.10242)
• Keywords: defense, large language model
• Abstract: Large language models memorize parts of their training data. Memorizing short snippets and facts is required to answer questions about the world and to be fluent in any language. But models have also been shown to reproduce long verbatim sequences of memorized text when prompted by a motivated adversary. In this work, we investigate an intermediate regime of memorization that we call non-adversarial reproduction, where we quantify the overlap between model responses and pretraining data when responding to natural and benign prompts. For a variety of innocuous prompt categories (e.g., writing a letter or a tutorial), we show that up to 15% of the text output by popular conversational language models overlaps with snippets from the Internet. In worst cases, we find generations where 100% of the content can be found exactly online. For the same tasks, we find that human-written text has far less overlap with Internet data. We further study whether prompting strategies can close this reproduction gap between models and humans. While appropriate prompting can reduce non-adversarial reproduction on average, we find that mitigating worst-case reproduction of training data requires stronger defenses -- even for benign interactions.

Title: Morpho-Aware Global Attention for Image Matting

• Authors: Jingru Yang, Chengzhi Cao, Chentianye Xu, Zhongwei Xie, Kaixiang Huang, Yang Zhou, Shengfeng He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10251
• Pdf URL: https://arxiv.org/pdf/2411.10251
• Copy Paste: [[2411.10251]] Morpho-Aware Global Attention for Image Matting(https://arxiv.org/abs/2411.10251)
• Keywords: transformer
• Abstract: Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) face inherent challenges in image matting, particularly in preserving fine structural details. ViTs, with their global receptive field enabled by the self-attention mechanism, often lose local details such as hair strands. Conversely, CNNs, constrained by their local receptive field, rely on deeper layers to approximate global context but struggle to retain fine structures at greater depths. To overcome these limitations, we propose a novel Morpho-Aware Global Attention (MAGA) mechanism, designed to effectively capture the morphology of fine structures. MAGA employs Tetris-like convolutional patterns to align the local shapes of fine structures, ensuring optimal local correspondence while maintaining sensitivity to morphological details. The extracted local morphology information is used as query embeddings, which are projected onto global key embeddings to emphasize local details in a broader context. Subsequently, by projecting onto value embeddings, MAGA seamlessly integrates these emphasized morphological details into a unified global structure. This approach enables MAGA to simultaneously focus on local morphology and unify these details into a coherent whole, effectively preserving fine structures. Extensive experiments show that our MAGA-based ViT achieves significant performance gains, outperforming state-of-the-art methods across two benchmarks with average improvements of 4.3% in SAD and 39.5% in MSE.

Title: Visual-Linguistic Agent: Towards Collaborative Contextual Object Reasoning

• Authors: Jingru Yang, Huan Yu, Yang Jingxin, Chentianye Xu, Yin Biao, Yu Sun, Shengfeng He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10252
• Pdf URL: https://arxiv.org/pdf/2411.10252
• Copy Paste: [[2411.10252]] Visual-Linguistic Agent: Towards Collaborative Contextual Object Reasoning(https://arxiv.org/abs/2411.10252)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) excel at descriptive tasks within images but often struggle with precise object localization, a critical element for reliable visual interpretation. In contrast, traditional object detection models provide high localization accuracy but frequently generate detections lacking contextual coherence due to limited modeling of inter-object relationships. To address this fundamental limitation, we introduce the \textbf{Visual-Linguistic Agent (VLA), a collaborative framework that combines the relational reasoning strengths of MLLMs with the precise localization capabilities of traditional object detectors. In the VLA paradigm, the MLLM serves as a central Linguistic Agent, working collaboratively with specialized Vision Agents for object detection and classification. The Linguistic Agent evaluates and refines detections by reasoning over spatial and contextual relationships among objects, while the classification Vision Agent offers corrective feedback to improve classification accuracy. This collaborative approach enables VLA to significantly enhance both spatial reasoning and object localization, addressing key challenges in multimodal understanding. Extensive evaluations on the COCO dataset demonstrate substantial performance improvements across multiple detection models, highlighting VLA's potential to set a new benchmark in accurate and contextually coherent object detection.

Title: The Unreasonable Effectiveness of Guidance for Diffusion Models

• Authors: Tim Kaiser, Nikolas Adaloglou, Markus Kollmann
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10257
• Pdf URL: https://arxiv.org/pdf/2411.10257
• Copy Paste: [[2411.10257]] The Unreasonable Effectiveness of Guidance for Diffusion Models(https://arxiv.org/abs/2411.10257)
• Keywords: diffusion, generative
• Abstract: Guidance is an error-correcting technique used to improve the perceptual quality of images generated by diffusion models. Typically, the correction is achieved by linear extrapolation, using an auxiliary diffusion model that has lower performance than the primary model. Using a 2D toy example, we show that it is highly beneficial when the auxiliary model exhibits similar errors as the primary one but stronger. We verify this finding in higher dimensions, where we show that competitive generative performance to state-of-the-art guidance methods can be achieved when the auxiliary model differs from the primary one only by having stronger weight regularization. As an independent contribution, we investigate whether upweighting long-range spatial dependencies improves visual fidelity. The result is a novel guidance method, which we call sliding window guidance (SWG), that guides the primary model with itself by constraining its receptive field. Intriguingly, SWG aligns better with human preferences than state-of-the-art guidance methods while requiring neither training, architectural modifications, nor class conditioning. The code will be released.

Title: MDHP-Net: Detecting Injection Attacks on In-vehicle Network using Multi-Dimensional Hawkes Process and Temporal Model

• Authors: Qi Liu, Yanchen Liu, Ruifeng Li, Chenhong Cao, Yufeng Li, Xingyu Li, Peng Wang, Runhan Feng
• Subjects: cs.CR, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.10258
• Pdf URL: https://arxiv.org/pdf/2411.10258
• Copy Paste: [[2411.10258]] MDHP-Net: Detecting Injection Attacks on In-vehicle Network using Multi-Dimensional Hawkes Process and Temporal Model(https://arxiv.org/abs/2411.10258)
• Keywords: security, attack, extraction
• Abstract: The integration of intelligent and connected technologies in modern vehicles, while offering enhanced functionalities through Electronic Control Unit and interfaces like OBD-II and telematics, also exposes the vehicle's in-vehicle network (IVN) to potential cyberattacks. In this paper, we consider a specific type of cyberattack known as the injection attack. As demonstrated by empirical data from real-world cybersecurity adversarial competitions(available at this https URL ), these injection attacks have excitation effect over time, gradually manipulating network traffic and disrupting the vehicle's normal functioning, ultimately compromising both its stability and safety. To profile the abnormal behavior of attackers, we propose a novel injection attack detector to extract long-term features of attack behavior. Specifically, we first provide a theoretical analysis of modeling the time-excitation effects of the attack using Multi-Dimensional Hawkes Process (MDHP). A gradient descent solver specifically tailored for MDHP, MDHP-GDS, is developed to accurately estimate optimal MDHP parameters. We then propose an injection attack detector, MDHP-Net, which integrates optimal MDHP parameters with MDHP-LSTM blocks to enhance temporal feature extraction. By introducing MDHP parameters, MDHP-Net captures complex temporal features that standard Long Short-Term Memory (LSTM) cannot, enriching temporal dependencies within our customized structure. Extensive evaluations demonstrate the effectiveness of our proposed detection approach.

Title: Fill in the blanks: Rethinking Interpretability in vision

• Authors: Pathirage N. Deelaka, Tharindu Wickremasinghe, Devin Y. De Silva, Lisara N. Gajaweera
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10273
• Pdf URL: https://arxiv.org/pdf/2411.10273
• Copy Paste: [[2411.10273]] Fill in the blanks: Rethinking Interpretability in vision(https://arxiv.org/abs/2411.10273)
• Keywords: interpretability, explainability
• Abstract: Model interpretability is a key challenge that has yet to align with the advancements observed in contemporary state-of-the-art deep learning models. In particular, deep learning aided vision tasks require interpretability, in order for their adoption in more specialized domains such as medical imaging. Although the field of explainable AI (XAI) developed methods for interpreting vision models along with early convolutional neural networks, recent XAI research has mainly focused on assigning attributes via saliency maps. As such, these methods are restricted to providing explanations at a sample level, and many explainability methods suffer from low adaptability across a wide range of vision models. In our work, we re-think vision-model explainability from a novel perspective, to probe the general input structure that a model has learnt during its training. To this end, we ask the question: "How would a vision model fill-in a masked-image". Experiments on standard vision datasets and pre-trained models reveal consistent patterns, and could be intergrated as an additional model-agnostic explainability tool in modern machine-learning platforms. The code will be available at \url{this https URL}

Title: Lateral Movement Detection via Time-aware Subgraph Classification on Authentication Logs

• Authors: Jiajun Zhou, Jiacheng Yao, Xuanze Chen, Shanqing Yu, Qi Xuan, Xiaoniu Yang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10279
• Pdf URL: https://arxiv.org/pdf/2411.10279
• Copy Paste: [[2411.10279]] Lateral Movement Detection via Time-aware Subgraph Classification on Authentication Logs(https://arxiv.org/abs/2411.10279)
• Keywords: security, attack, steal
• Abstract: Lateral movement is a crucial component of advanced persistent threat (APT) attacks in networks. Attackers exploit security vulnerabilities in internal networks or IoT devices, expanding their control after initial infiltration to steal sensitive data or carry out other malicious activities, posing a serious threat to system security. Existing research suggests that attackers generally employ seemingly unrelated operations to mask their malicious intentions, thereby evading existing lateral movement detection methods and hiding their intrusion traces. In this regard, we analyze host authentication log data from a graph perspective and propose a multi-scale lateral movement detection framework called LMDetect. The main workflow of this framework proceeds as follows: 1) Construct a heterogeneous multigraph from host authentication log data to strengthen the correlations among internal system entities; 2) Design a time-aware subgraph generator to extract subgraphs centered on authentication events from the heterogeneous authentication multigraph; 3) Design a multi-scale attention encoder that leverages both local and global attention to capture hidden anomalous behavior patterns in the authentication subgraphs, thereby achieving lateral movement detection. Extensive experiments on two real-world authentication log datasets demonstrate the effectiveness and superiority of our framework in detecting lateral movement behaviors.

Title: Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation

• Authors: Tim Elsner, Paula Usinger, Julius Nehring-Wirxel, Gregor Kobsik, Victor Czech, Yanjiang He, Isaak Lim, Leif Kobbelt
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10281
• Pdf URL: https://arxiv.org/pdf/2411.10281
• Copy Paste: [[2411.10281]] Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation(https://arxiv.org/abs/2411.10281)
• Keywords: transformer
• Abstract: In language processing, transformers benefit greatly from text being condensed. This is achieved through a larger vocabulary that captures word fragments instead of plain characters. This is often done with Byte Pair Encoding. In the context of images, tokenisation of visual data is usually limited to regular grids obtained from quantisation methods, without global content awareness. Our work improves tokenisation of visual data by bringing Byte Pair Encoding from 1D to multiple dimensions, as a complementary add-on to existing compression. We achieve this through counting constellations of token pairs and replacing the most frequent token pair with a newly introduced token. The multidimensionality only increases the computation time by a factor of 2 for images, making it applicable even to large datasets like ImageNet within minutes on consumer hardware. This is a lossless preprocessing step. Our evaluation shows improved training and inference performance of transformers on visual data achieved by compressing frequent constellations of tokens: The resulting sequences are shorter, with more uniformly distributed information content, e.g. condensing empty regions in an image into single tokens. As our experiments show, these condensed sequences are easier to process. We additionally introduce a strategy to amplify this compression further by clustering the vocabulary.

Title: Transformers -- Messages in Disguise

• Authors: Joshua H. Tyler, Mohamed K.M. Fadul, Donald R. Reising
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.10287
• Pdf URL: https://arxiv.org/pdf/2411.10287
• Copy Paste: [[2411.10287]] Transformers -- Messages in Disguise(https://arxiv.org/abs/2411.10287)
• Keywords: secure, transformer
• Abstract: Modern cryptography, such as Rivest Shamir Adleman (RSA) and Secure Hash Algorithm (SHA), has been designed by humans based on our understanding of cryptographic methods. Neural Network (NN) based cryptography is being investigated due to its ability to learn and implement random cryptographic schemes that may be harder to decipher than human-designed algorithms. NN based cryptography may create a new cryptographic scheme that is NN specific and that changes every time the NN is (re)trained. This is attractive since it would require an adversary to restart its process(es) to learn or break the cryptographic scheme every time the NN is (re)trained. Current challenges facing NN-based encryption include additional communication overhead due to encoding to correct bit errors, quantizing the continuous-valued output of the NN, and enabling One-Time-Pad encryption. With this in mind, the Random Adversarial Data Obfuscation Model (RANDOM) Adversarial Neural Cryptography (ANC) network is introduced. RANDOM is comprised of three new NN layers: the (i) projection layer, (ii) inverse projection layer, and (iii) dot-product layer. This results in an ANC network that (i) is computationally efficient, (ii) ensures the encrypted message is unique to the encryption key, and (iii) does not induce any communication overhead. RANDOM only requires around 100 KB to store and can provide up to 2.5 megabytes per second of end-to-end encrypted communication.

Title: RETR: Multi-View Radar Detection Transformer for Indoor Perception

• Authors: Ryoma Yataka, Adriano Cardace, Pu Perry Wang, Petros Boufounos, Ryuhei Takahashi
• Subjects: cs.CV, cs.AI, cs.LG, math.DG
• Abstract URL: https://arxiv.org/abs/2411.10293
• Pdf URL: https://arxiv.org/pdf/2411.10293
• Copy Paste: [[2411.10293]] RETR: Multi-View Radar Detection Transformer for Indoor Perception(https://arxiv.org/abs/2411.10293)
• Keywords: privacy, transformer, segmentation
• Abstract: Indoor radar perception has seen rising interest due to affordable costs driven by emerging automotive imaging radar developments and the benefits of reduced privacy concerns and reliability under hazardous conditions (e.g., fire and smoke). However, existing radar perception pipelines fail to account for distinctive characteristics of the multi-view radar setting. In this paper, we propose Radar dEtection TRansformer (RETR), an extension of the popular DETR architecture, tailored for multi-view radar perception. RETR inherits the advantages of DETR, eliminating the need for hand-crafted components for object detection and segmentation in the image plane. More importantly, RETR incorporates carefully designed modifications such as 1) depth-prioritized feature similarity via a tunable positional encoding (TPE); 2) a tri-plane loss from both radar and camera coordinates; and 3) a learnable radar-to-camera transformation via reparameterization, to account for the unique multi-view radar setting. Evaluated on two indoor radar perception datasets, our approach outperforms existing state-of-the-art methods by a margin of 15.38+ AP for object detection and 11.77+ IoU for instance segmentation, respectively.

Title: Modification Takes Courage: Seamless Image Stitching via Reference-Driven Inpainting

• Authors: Ziqi Xie, Xiao Lai, Weidong Zhao, Xianhui Liu, Wenlong Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10309
• Pdf URL: https://arxiv.org/pdf/2411.10309
• Copy Paste: [[2411.10309]] Modification Takes Courage: Seamless Image Stitching via Reference-Driven Inpainting(https://arxiv.org/abs/2411.10309)
• Keywords: diffusion, large language model
• Abstract: Current image stitching methods often produce noticeable seams in challenging scenarios such as uneven hue and large parallax. To tackle this problem, we propose the Reference-Driven Inpainting Stitcher (RDIStitcher), which reformulates the image fusion and rectangling as a reference-based inpainting model, incorporating a larger modification fusion area and stronger modification intensity than previous methods. Furthermore, we introduce a self-supervised model training method, which enables the implementation of RDIStitcher without requiring labeled data by fine-tuning a Text-to-Image (T2I) diffusion model. Recognizing difficulties in assessing the quality of stitched images, we present the Multimodal Large Language Models (MLLMs)-based metrics, offering a new perspective on evaluating stitched image quality. Compared to the state-of-the-art (SOTA) method, extensive experiments demonstrate that our method significantly enhances content coherence and seamless transitions in the stitched images. Especially in the zero-shot experiments, our method exhibits strong generalization capabilities. Code: this https URL

Title: M3TR: Generalist HD Map Construction with Variable Map Priors

• Authors: Fabian Immel, Richard Fehler, Frank Bieder, Jan-Hendrik Pauls, Christoph Stiller
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.10316
• Pdf URL: https://arxiv.org/pdf/2411.10316
• Copy Paste: [[2411.10316]] M3TR: Generalist HD Map Construction with Variable Map Priors(https://arxiv.org/abs/2411.10316)
• Keywords: transformer
• Abstract: Autonomous vehicles require road information for their operation, usually in form of HD maps. Since offline maps eventually become outdated or may only be partially available, online HD map construction methods have been proposed to infer map information from live sensor data. A key issue remains how to exploit such partial or outdated map information as a prior. We introduce M3TR (Multi-Masking Map Transformer), a generalist approach for HD map construction both with and without map priors. We address shortcomings in ground truth generation for Argoverse 2 and nuScenes and propose the first realistic scenarios with semantically diverse map priors. Examining various query designs, we use an improved method for integrating prior map elements into a HD map construction model, increasing performance by +4.3 mAP. Finally, we show that training across all prior scenarios yields a single Generalist model, whose performance is on par with previous Expert models that can handle only one specific type of map prior. M3TR thus is the first model capable of leveraging variable map priors, making it suitable for real-world deployment. Code is available at this https URL

Title: Probabilistic Prior Driven Attention Mechanism Based on Diffusion Model for Imaging Through Atmospheric Turbulence

• Authors: Guodong Sun, Qixiang Ma, Liqiang Zhang, Hongwei Wang, Zixuan Gao, Haotian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10321
• Pdf URL: https://arxiv.org/pdf/2411.10321
• Copy Paste: [[2411.10321]] Probabilistic Prior Driven Attention Mechanism Based on Diffusion Model for Imaging Through Atmospheric Turbulence(https://arxiv.org/abs/2411.10321)
• Keywords: robust, extraction, diffusion, transformer
• Abstract: Atmospheric turbulence introduces severe spatial and geometric distortions, challenging traditional image restoration methods. We propose the Probabilistic Prior Turbulence Removal Network (PPTRN), which combines probabilistic diffusion-based prior modeling with Transformer-driven feature extraction to address this issue. PPTRN employs a two-stage approach: first, a latent encoder and Transformer are jointly trained on clear images to establish robust feature representations. Then, a Denoising Diffusion Probabilistic Model (DDPM) models prior distributions over latent vectors, guiding the Transformer in capturing diverse feature variations essential for restoration. A key innovation in PPTRN is the Probabilistic Prior Driven Cross Attention mechanism, which integrates the DDPM-generated prior with feature embeddings to reduce artifacts and enhance spatial coherence. Extensive experiments validate that PPTRN significantly improves restoration quality on turbulence-degraded images, setting a new benchmark in clarity and structural fidelity.

Title: Safe Text-to-Image Generation: Simply Sanitize the Prompt Embedding

• Authors: Huming Qiu, Guanxu Chen, Mi Zhang, Min Yang
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10329
• Pdf URL: https://arxiv.org/pdf/2411.10329
• Copy Paste: [[2411.10329]] Safe Text-to-Image Generation: Simply Sanitize the Prompt Embedding(https://arxiv.org/abs/2411.10329)
• Keywords: attack, robust, interpretability
• Abstract: In recent years, text-to-image (T2I) generation models have made significant progress in generating high-quality images that align with text descriptions. However, these models also face the risk of unsafe generation, potentially producing harmful content that violates usage policies, such as explicit material. Existing safe generation methods typically focus on suppressing inappropriate content by erasing undesired concepts from visual representations, while neglecting to sanitize the textual representation. Although these methods help mitigate the risk of misuse to certain extent, their robustness remains insufficient when dealing with adversarial attacks. Given that semantic consistency between input text and output image is a fundamental requirement for T2I models, we identify that textual representations (i.e., prompt embeddings) are likely the primary source of unsafe generation. To this end, we propose a vision-agnostic safe generation framework, Embedding Sanitizer (ES), which focuses on erasing inappropriate concepts from prompt embeddings and uses the sanitized embeddings to guide the model for safe generation. ES is applied to the output of the text encoder as a plug-and-play module, enabling seamless integration with different T2I models as well as other safeguards. In addition, ES's unique scoring mechanism assigns a score to each token in the prompt to indicate its potential harmfulness, and dynamically adjusts the sanitization intensity to balance defensive performance and generation quality. Through extensive evaluation on five prompt benchmarks, our approach achieves state-of-the-art robustness by sanitizing the source (prompt embedding) of unsafe generation compared to nine baseline methods. It significantly outperforms existing safeguards in terms of interpretability and controllability while maintaining generation quality.

Title: Number it: Temporal Grounding Videos like Flipping Manga

• Authors: Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, Xu Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10332
• Pdf URL: https://arxiv.org/pdf/2411.10332
• Copy Paste: [[2411.10332]] Number it: Temporal Grounding Videos like Flipping Manga(https://arxiv.org/abs/2411.10332)
• Keywords: large language model
• Abstract: Video Large Language Models (Vid-LLMs) have made remarkable advancements in comprehending video content for QA dialogue. However, they struggle to extend this visual understanding to tasks requiring precise temporal localization, known as Video Temporal Grounding (VTG). To address this gap, we introduce Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual comprehension with temporal grounding by adding unique numerical identifiers to each video frame. Treating a video as a sequence of numbered frame images, NumPro transforms VTG into an intuitive process: flipping through manga panels in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking visual content with corresponding temporal information. Our experiments demonstrate that NumPro significantly boosts VTG performance of top-tier Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and 8.5\% in mAP for highlight detection. The code will be available at this https URL.

Title: Y-MAP-Net: Real-time depth, normals, segmentation, multi-label captioning and 2D human pose in RGB images

• Authors: Ammar Qammaz, Nikolaos Vasilikopoulos, Iason Oikonomidis, Antonis A. Argyros
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10334
• Pdf URL: https://arxiv.org/pdf/2411.10334
• Copy Paste: [[2411.10334]] Y-MAP-Net: Real-time depth, normals, segmentation, multi-label captioning and 2D human pose in RGB images(https://arxiv.org/abs/2411.10334)
• Keywords: segmentation
• Abstract: We present Y-MAP-Net, a Y-shaped neural network architecture designed for real-time multi-task learning on RGB images. Y-MAP-Net, simultaneously predicts depth, surface normals, human pose, semantic segmentation and generates multi-label captions, all from a single network evaluation. To achieve this, we adopt a multi-teacher, single-student training paradigm, where task-specific foundation models supervise the network's learning, enabling it to distill their capabilities into a lightweight architecture suitable for real-time applications. Y-MAP-Net, exhibits strong generalization, simplicity and computational efficiency, making it ideal for robotics and other practical scenarios. To support future research, we will release our code publicly.

Title: Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability

• Authors: Pooja Aslami, Kejun Chen, Timothy M. Hansen, Malik Hassanaly
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10367
• Pdf URL: https://arxiv.org/pdf/2411.10367
• Copy Paste: [[2411.10367]] Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability(https://arxiv.org/abs/2411.10367)
• Keywords: attack, steal, explainability
• Abstract: False data injection attacks (FDIAs) on smart inverters are a growing concern linked to increased renewable energy production. While data-based FDIA detection methods are also actively developed, we show that they remain vulnerable to impactful and stealthy adversarial examples that can be crafted using Reinforcement Learning (RL). We propose to include such adversarial examples in data-based detection training procedure via a continual adversarial RL (CARL) approach. This way, one can pinpoint the deficiencies of data-based detection, thereby offering explainability during their incremental improvement. We show that a continual learning implementation is subject to catastrophic forgetting, and additionally show that forgetting can be addressed by employing a joint training strategy on all generated FDIA scenarios.

Title: Mechanisms of Generative Image-to-Image Translation Networks

• Authors: Guangzong Chen, Mingui Sun, Zhi-Hong Mao, Kangni Liu, Wenyan Jia
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.10368
• Pdf URL: https://arxiv.org/pdf/2411.10368
• Copy Paste: [[2411.10368]] Mechanisms of Generative Image-to-Image Translation Networks(https://arxiv.org/abs/2411.10368)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) are a class of neural networks that have been widely used in the field of image-to-image translation. In this paper, we propose a streamlined image-to-image translation network with a simpler architecture compared to existing models. We investigate the relationship between GANs and autoencoders and provide an explanation for the efficacy of employing only the GAN component for tasks involving image translation. We show that adversarial for GAN models yields results comparable to those of existing methods without additional complex loss penalties. Subsequently, we elucidate the rationale behind this phenomenon. We also incorporate experimental results to demonstrate the validity of our findings.

Title: Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusion

• Authors: Haoran Wei, Wencheng Han, Xingping Dong, Jianbing Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10369
• Pdf URL: https://arxiv.org/pdf/2411.10369
• Copy Paste: [[2411.10369]] Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusion(https://arxiv.org/abs/2411.10369)
• Keywords: diffusion
• Abstract: Recent diffusion-based Single-image 3D portrait generation methods typically employ 2D diffusion models to provide multi-view knowledge, which is then distilled into 3D representations. However, these methods usually struggle to produce high-fidelity 3D models, frequently yielding excessively blurred textures. We attribute this issue to the insufficient consideration of cross-view consistency during the diffusion process, resulting in significant disparities between different views and ultimately leading to blurred 3D representations. In this paper, we address this issue by comprehensively exploiting multi-view priors in both the conditioning and diffusion procedures to produce consistent, detail-rich portraits. From the conditioning standpoint, we propose a Hybrid Priors Diffsion model, which explicitly and implicitly incorporates multi-view priors as conditions to enhance the status consistency of the generated multi-view portraits. From the diffusion perspective, considering the significant impact of the diffusion noise distribution on detailed texture generation, we propose a Multi-View Noise Resamplig Strategy integrated within the optimization process leveraging cross-view priors to enhance representation consistency. Extensive experiments demonstrate that our method can produce 3D portraits with accurate geometry and rich details from a single image. The project page is at \url{this https URL}.

Title: Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare

• Authors: Suraj Racha, Shubh Gupta, Humaira Firdowse, Aastik Solanki, Ganesh Ramakrishnan, Kshitij S. Jadhav
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10383
• Pdf URL: https://arxiv.org/pdf/2411.10383
• Copy Paste: [[2411.10383]] Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare(https://arxiv.org/abs/2411.10383)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) is a pioneering approach in distributed machine learning, enabling collaborative model training across multiple clients while retaining data privacy. However, the inherent heterogeneity due to imbalanced resource representations across multiple clients poses significant challenges, often introducing bias towards the majority class. This issue is particularly prevalent in healthcare settings, where hospitals acting as clients share medical images. To address class imbalance and reduce bias, we propose a co-distillation driven framework in a federated healthcare setting. Unlike traditional federated setups with a designated server client, our framework promotes knowledge sharing among clients to collectively improve learning outcomes. Our experiments demonstrate that in a federated healthcare setting, co-distillation outperforms other federated methods in handling class imbalance. Additionally, we demonstrate that our framework has the least standard deviation with increasing imbalance while outperforming other baselines, signifying the robustness of our framework for FL in healthcare.

Title: Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation

• Authors: Markus Karmann, Onay Urfalioglu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.10411
• Pdf URL: https://arxiv.org/pdf/2411.10411
• Copy Paste: [[2411.10411]] Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation(https://arxiv.org/abs/2411.10411)
• Keywords: diffusion, segmentation
• Abstract: Recent progress in interactive point prompt based Image Segmentation allows to significantly reduce the manual effort to obtain high quality semantic labels. State-of-the-art unsupervised methods use self-supervised pre-trained models to obtain pseudo-labels which are used in training a prompt-based segmentation model. In this paper, we propose a novel unsupervised and training-free approach based solely on the self-attention of Stable Diffusion. We interpret the self-attention tensor as a Markov transition operator, which enables us to iteratively construct a Markov chain. Pixel-wise counting of the required number of iterations along the Markov-chain to reach a relative probability threshold yields a Markov-iteration-map, which we simply call a Markov-map. Compared to the raw attention maps, we show that our proposed Markov-map has less noise, sharper semantic boundaries and more uniform values within semantically similar regions. We integrate the Markov-map in a simple yet effective truncated nearest neighbor framework to obtain interactive point prompt based segmentation. Despite being training-free, we experimentally show that our approach yields excellent results in terms of Number of Clicks (NoC), even outperforming state-of-the-art training based unsupervised methods in most of the datasets.

Title: Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations

• Authors: Jianfeng Chi, Ujjwal Karn, Hongyuan Zhan, Eric Smith, Javier Rando, Yiming Zhang, Kate Plawiak, Zacharie Delpierre Coudert, Kartikeya Upasani, Mahesh Pasupuleti
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.10414
• Pdf URL: https://arxiv.org/pdf/2411.10414
• Copy Paste: [[2411.10414]] Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations(https://arxiv.org/abs/2411.10414)
• Keywords: attack, robust
• Abstract: We introduce Llama Guard 3 Vision, a multimodal LLM-based safeguard for human-AI conversations that involves image understanding: it can be used to safeguard content for both multimodal LLM inputs (prompt classification) and outputs (response classification). Unlike the previous text-only Llama Guard versions (Inan et al., 2023; Llama Team, 2024b,a), it is specifically designed to support image reasoning use cases and is optimized to detect harmful multimodal (text and image) prompts and text responses to these prompts. Llama Guard 3 Vision is fine-tuned on Llama 3.2-Vision and demonstrates strong performance on the internal benchmarks using the MLCommons taxonomy. We also test its robustness against adversarial attacks. We believe that Llama Guard 3 Vision serves as a good starting point to build more capable and robust content moderation tools for human-AI conversation with multimodal capabilities.

Title: Back to Supervision: Boosting Word Boundary Detection through Frame Classification

• Authors: Simone Carnemolla, Salvatore Calcagno, Simone Palazzo, Daniela Giordano
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.10423
• Pdf URL: https://arxiv.org/pdf/2411.10423
• Copy Paste: [[2411.10423]] Back to Supervision: Boosting Word Boundary Detection through Frame Classification(https://arxiv.org/abs/2411.10423)
• Keywords: robust, segmentation
• Abstract: Speech segmentation at both word and phoneme levels is crucial for various speech processing tasks. It significantly aids in extracting meaningful units from an utterance, thus enabling the generation of discrete elements. In this work we propose a model-agnostic framework to perform word boundary detection in a supervised manner also employing a labels augmentation technique and an output-frame selection strategy. We trained and tested on the Buckeye dataset and only tested on TIMIT one, using state-of-the-art encoder models, including pre-trained solutions (Wav2Vec 2.0 and HuBERT), as well as convolutional and convolutional recurrent networks. Our method, with the HuBERT encoder, surpasses the performance of other state-of-the-art architectures, whether trained in supervised or self-supervised settings on the same datasets. Specifically, we achieved F-values of 0.8427 on the Buckeye dataset and 0.7436 on the TIMIT dataset, along with R-values of 0.8489 and 0.7807, respectively. These results establish a new state-of-the-art for both datasets. Beyond the immediate task, our approach offers a robust and efficient preprocessing method for future research in audio tokenization.

Title: M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation

• Authors: Sucheng Ren, Yaodong Yu, Nataniel Ruiz, Feng Wang, Alan Yuille, Cihang Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10433
• Pdf URL: https://arxiv.org/pdf/2411.10433
• Copy Paste: [[2411.10433]] M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation(https://arxiv.org/abs/2411.10433)
• Keywords: diffusion
• Abstract: There exists recent work in computer vision, named VAR, that proposes a new autoregressive paradigm for image generation. Diverging from the vanilla next-token prediction, VAR structurally reformulates the image generation into a coarse to fine next-scale prediction. In this paper, we show that this scale-wise autoregressive framework can be effectively decoupled into \textit{intra-scale modeling}, which captures local spatial dependencies within each scale, and \textit{inter-scale modeling}, which models cross-scale relationships progressively from coarse-to-fine scales. This decoupling structure allows to rebuild VAR in a more computationally efficient manner. Specifically, for intra-scale modeling -- crucial for generating high-fidelity images -- we retain the original bidirectional self-attention design to ensure comprehensive modeling; for inter-scale modeling, which semantically connects different scales but is computationally intensive, we apply linear-complexity mechanisms like Mamba to substantially reduce computational overhead. We term this new framework M-VAR. Extensive experiments demonstrate that our method outperforms existing models in both image quality and generation speed. For example, our 1.5B model, with fewer parameters and faster inference speed, outperforms the largest VAR-d30-2B. Moreover, our largest model M-VAR-d32 impressively registers 1.78 FID on ImageNet 256$\times$256 and outperforms the prior-art autoregressive models LlamaGen/VAR by 0.4/0.19 and popular diffusion models LDM/DiT by 1.82/0.49, respectively. Code is avaiable at \url{this https URL}.

Title: Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization

• Authors: Yuhan Fu, Ruobing Xie, Xingwu Sun, Zhanhui Kang, Xirong Li
• Subjects: cs.CL, cs.AI, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.10436
• Pdf URL: https://arxiv.org/pdf/2411.10436
• Copy Paste: [[2411.10436]] Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization(https://arxiv.org/abs/2411.10436)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) are known to hallucinate, which limits their practical applications. Recent works have attempted to apply Direct Preference Optimization (DPO) to enhance the performance of MLLMs, but have shown inconsistent improvements in mitigating hallucinations. To address this issue more effectively, we introduce Hallucination-targeted Direct Preference Optimization (HDPO) to reduce hallucinations in MLLMs. Unlike previous approaches, our method tackles hallucinations from their diverse forms and causes. Specifically, we develop three types of preference pair data targeting the following causes of MLLM hallucinations: (1) insufficient visual capabilities, (2) long context generation, and (3) multimodal conflicts. Experimental results demonstrate that our method achieves superior performance across multiple hallucination evaluation datasets, surpassing most state-of-the-art (SOTA) methods and highlighting the potential of our approach. Ablation studies and in-depth analyses further confirm the effectiveness of our method and suggest the potential for further improvements through scaling up.

Title: MARS: Unleashing the Power of Variance Reduction for Training Large Models

• Authors: Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, Quanquan Gu
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.10438
• Pdf URL: https://arxiv.org/pdf/2411.10438
• Copy Paste: [[2411.10438]] MARS: Unleashing the Power of Variance Reduction for Training Large Models(https://arxiv.org/abs/2411.10438)
• Keywords: large language model
• Abstract: Training deep neural networks--and more recently, large models--demands efficient and scalable optimizers. Adaptive gradient algorithms like Adam, AdamW, and their variants have been central to this task. Despite the development of numerous variance reduction algorithms in the past decade aimed at accelerating stochastic optimization in both convex and nonconvex settings, variance reduction has not found widespread success in training deep neural networks or large language models. Consequently, it has remained a less favored approach in modern AI. In this paper, to unleash the power of variance reduction for efficient training of large models, we propose a unified optimization framework, MARS (Make vAriance Reduction Shine), which reconciles preconditioned gradient methods with variance reduction via a scaled stochastic recursive momentum technique. Within our framework, we introduce three instances of MARS that leverage preconditioned gradient updates based on AdamW, Lion, and Shampoo, respectively. We also draw a connection between our algorithms and existing optimizers. Experimental results on training GPT-2 models indicate that MARS consistently outperforms AdamW by a large margin.

Title: LLaVA-o1: Let Vision Language Models Reason Step-by-Step

• Authors: Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, Li Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10440
• Pdf URL: https://arxiv.org/pdf/2411.10440
• Copy Paste: [[2411.10440]] LLaVA-o1: Let Vision Language Models Reason Step-by-Step(https://arxiv.org/abs/2411.10440)
• Keywords: large language model
• Abstract: Large language models have demonstrated substantial advancements in reasoning capabilities, particularly through inference-time scaling, as illustrated by models such as OpenAI's o1. However, current Vision-Language Models (VLMs) often struggle to perform systematic and structured reasoning, especially when handling complex visual question-answering tasks. In this work, we introduce LLaVA-o1, a novel VLM designed to conduct autonomous multistage reasoning. Unlike chain-of-thought prompting, LLaVA-o1 independently engages in sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation. This structured approach enables LLaVA-o1 to achieve marked improvements in precision on reasoning-intensive tasks. To accomplish this, we compile the LLaVA-o1-100k dataset, integrating samples from various visual question answering sources and providing structured reasoning annotations. Besides, we propose an inference-time stage-level beam search method, which enables effective inference-time scaling. Remarkably, with only 100k training samples and a simple yet effective inference time scaling method, LLaVA-o1 not only outperforms its base model by 8.9% on a wide range of multimodal reasoning benchmarks, but also surpasses the performance of larger and even closed-source models, such as Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct.

Title: Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

• Authors: Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, Jifeng Dai
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.10442
• Pdf URL: https://arxiv.org/pdf/2411.10442
• Copy Paste: [[2411.10442]] Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization(https://arxiv.org/abs/2411.10442)
• Keywords: large language model
• Abstract: Existing open-source multimodal large language models (MLLMs) generally follow a training process involving pre-training and supervised fine-tuning. However, these models suffer from distribution shifts, which limit their multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance. To address this, we introduce a preference optimization (PO) process to enhance the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data side, we design an automated preference data construction pipeline to create MMPR, a high-quality, large-scale multimodal reasoning preference dataset. and (2) on the model side, we explore integrating PO with MLLMs, developing a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance. Our approach demonstrates improved performance across multiple benchmarks, particularly in multimodal reasoning tasks. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance comparable to the 10x larger InternVL2-76B. We hope this study could inspire further advancements in MLLMs. Code, data, and model shall be publicly released.