2025-01-09

Title: A Survey on Large Language Models with some Insights on their Capabilities and Limitations

• Authors: Andrea Matarazzo, Riccardo Torlone
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2501.04040
• Pdf URL: https://arxiv.org/pdf/2501.04040
• Copy Paste: [[2501.04040]] A Survey on Large Language Models with some Insights on their Capabilities and Limitations(https://arxiv.org/abs/2501.04040)
• Keywords: transformer, large language model
• Abstract: The rapid advancement of artificial intelligence, particularly with the development of Large Language Models (LLMs) built on the transformer architecture, has redefined the capabilities of natural language processing. These models now exhibit remarkable performance across various language-related tasks, such as text generation, question answering, translation, and summarization, often rivaling human-like comprehension. More intriguingly, LLMs have demonstrated emergent abilities extending beyond their core functions, showing proficiency in tasks like commonsense reasoning, code generation, and arithmetic. This survey paper explores the foundational components, scaling mechanisms, and architectural strategies that drive these capabilities. Emphasizing models like GPT and LLaMA, we analyze the impact of exponential data and computational growth on LLM performance, while also addressing the trade-offs associated with scaling. We also examine LLM applications across sectors, such as healthcare, finance, education, and law, highlighting their adaptability and potential to solve domain-specific challenges. Central to this work are the questions of how LLMs generalize across diverse tasks, exhibit planning, and reasoning abilities, and whether these emergent abilities can be systematically elicited or enhanced. In particular, we provide some insights into the CoT (Chain of Thought) and PoT (Plan of Thought) abilities within LLMs, focusing on how pre-training data influences their emergence. Additionally, we investigate LLM-modulo frameworks that integrate external systems, allowing LLMs to handle complex, dynamic tasks. By analyzing these factors, this paper aims to foster the ongoing discussion on the capabilities and limits of LLMs, promoting their responsible development and application in novel and increasingly complex environments.

Title: The Power of Negative Zero: Datatype Customization for Quantized Large Language Models

• Authors: Yuzong Chen, Xilai Dai, Chi-chih Chang, Yash Akhauri, Mohamed S. Abdelfattah
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04052
• Pdf URL: https://arxiv.org/pdf/2501.04052
• Copy Paste: [[2501.04052]] The Power of Negative Zero: Datatype Customization for Quantized Large Language Models(https://arxiv.org/abs/2501.04052)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across various machine learning tasks, quickly becoming one of the most prevalent AI workloads. Yet the substantial memory requirement of LLMs significantly hinders their deployment for end users. Post-training quantization (PTQ) serves as one of the most hardware-efficient methods to mitigate the memory and computational demands of LLMs. Although the traditional integer (INT) datatype has received widespread adoption in PTQ methods, floating-point (FP) quantization has emerged as a viable alternative thanks to its effectiveness in fitting LLM numerical distributions. However, the FP datatype in sign-magnitude binary representation contains both positive and negative zero, which constrains its representation capability, particularly under low precision (3 and 4 bits). In this paper, we extend the basic FP datatype to perform Redundant Zero Remapping (RaZeR), which remaps the negative zero FP encoding to a set of pre-defined special values to maximally utilize FP quantization encodings and to better fit LLM numerical distributions. Through careful selection of special values, RaZeR outperforms conventional asymmetric INT quantization while achieving high computational efficiency. We demonstrate that RaZeR can be seamlessly integrated with quantization algorithms for both weights and KV-cache, including advanced methods with clipping and transformations, and consistently achieve better model accuracy. Additionally, we implement a fast GEMV kernel with fused dequantization that efficiently converts the 4-bit RaZeR value to FP16 through novel bit-level manipulation. On modern GPUs, our evaluation shows that RaZeR improves the GEMV speed by up to 7.56$\times$ compared to the FP16 implementation, while achieving up to 2.72$\times$ speedup in the LLM decoding throughput.

Title: Causal Machine Learning Methods for Estimating Personalised Treatment Effects -- Insights on validity from two large trials

• Authors: Hongruyu Chen, Helena Aebersold, Milo Alan Puhan, Miquel Serra-Burriel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04061
• Pdf URL: https://arxiv.org/pdf/2501.04061
• Copy Paste: [[2501.04061]] Causal Machine Learning Methods for Estimating Personalised Treatment Effects -- Insights on validity from two large trials(https://arxiv.org/abs/2501.04061)
• Keywords: robust
• Abstract: Causal machine learning (ML) methods hold great promise for advancing precision medicine by estimating personalized treatment effects. However, their reliability remains largely unvalidated in empirical settings. In this study, we assessed the internal and external validity of 17 mainstream causal heterogeneity ML methods -- including metalearners, tree-based methods, and deep learning methods -- using data from two large randomized controlled trials: the International Stroke Trial (N=19,435) and the Chinese Acute Stroke Trial (N=21,106). Our findings reveal that none of the ML methods reliably validated their performance, neither internal nor external, showing significant discrepancies between training and test data on the proposed evaluation metrics. The individualized treatment effects estimated from training data failed to generalize to the test data, even in the absence of distribution shifts. These results raise concerns about the current applicability of causal ML models in precision medicine, and highlight the need for more robust validation techniques to ensure generalizability.

Title: FedKD-hybrid: Federated Hybrid Knowledge Distillation for Lithography Hotspot Detection

• Authors: Yuqi Li, Xingyou Lin, Kai Zhang, Chuanguang Yang, Zhongliang Guo, Jianping Gou, Yanli Li
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2501.04066
• Pdf URL: https://arxiv.org/pdf/2501.04066
• Copy Paste: [[2501.04066]] FedKD-hybrid: Federated Hybrid Knowledge Distillation for Lithography Hotspot Detection(https://arxiv.org/abs/2501.04066)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) provides novel solutions for machine learning (ML)-based lithography hotspot detection (LHD) under distributed privacy-preserving settings. Currently, two research pipelines have been investigated to aggregate local models and achieve global consensus, including parameter/nonparameter based (also known as knowledge distillation, namely KD). While these two kinds of methods show effectiveness in specific scenarios, we note they have not fully utilized and transferred the information learned, leaving the potential of FL-based LDH remains unexplored. Thus, we propose FedKDhybrid in this study to mitigate the research gap. Specifically, FedKD-hybrid clients agree on several identical layers across all participants and a public dataset for achieving global consensus. During training, the trained local model will be evaluated on the public dataset, and the generated logits will be uploaded along with the identical layer parameters. The aggregated information is consequently used to update local models via the public dataset as a medium. We compare our proposed FedKD-hybrid with several state-of-the-art (SOTA) FL methods under ICCAD-2012 and FAB (real-world collected) datasets with different settings; the experimental results demonstrate the superior performance of the FedKD-hybrid algorithm. Our code is available at this https URL

Title: More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives

• Authors: Xiaoqing Zhang, Ang Lv, Yuhan Liu, Flood Sung, Wei Liu, Shuo Shang, Xiuying Chen, Rui Yan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04070
• Pdf URL: https://arxiv.org/pdf/2501.04070
• Copy Paste: [[2501.04070]] More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives(https://arxiv.org/abs/2501.04070)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel at few-shot in-context learning (ICL) without requiring parameter updates. However, as the number of ICL demonstrations increases from a few to many, performance tends to plateau and eventually decline. We identify two primary causes for this trend: the suboptimal negative log-likelihood (NLL) optimization objective and the incremental data noise. To address these issues, we introduce DR-ICL, a novel optimization method that enhances model performance through Differentiated Learning and advantage-based Reweighting objectives. Globally, DR-ICL utilizes differentiated learning to optimize the NLL objective, ensuring that many-shot performance surpasses zero-shot levels. Locally, it dynamically adjusts the weighting of many-shot demonstrations by leveraging cumulative advantages inspired by reinforcement learning, thereby improving generalization. This approach allows the model to handle varying numbers of shots effectively, mitigating the impact of noisy data. Recognizing the lack of multi-task datasets with diverse many-shot distributions, we develop the Many-Shot ICL Benchmark (MICLB)-a large-scale benchmark covering shot numbers from 1 to 350 within sequences of up to 8,000 tokens-for fine-tuning purposes. MICLB facilitates the evaluation of many-shot ICL strategies across seven prominent NLP tasks and 50 distinct datasets. Experimental results demonstrate that LLMs enhanced with DR-ICL achieve significant improvements in many-shot setups across various tasks, including both in-domain and out-of-domain scenarios. We release the code and benchmark dataset hoping to facilitate further research in many-shot ICL.

Title: TrojanDec: Data-free Detection of Trojan Inputs in Self-supervised Learning

• Authors: Yupei Liu, Yanting Wang, Jinyuan Jia
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04108
• Pdf URL: https://arxiv.org/pdf/2501.04108
• Copy Paste: [[2501.04108]] TrojanDec: Data-free Detection of Trojan Inputs in Self-supervised Learning(https://arxiv.org/abs/2501.04108)
• Keywords: defense, attack, data-free
• Abstract: An image encoder pre-trained by self-supervised learning can be used as a general-purpose feature extractor to build downstream classifiers for various downstream tasks. However, many studies showed that an attacker can embed a trojan into an encoder such that multiple downstream classifiers built based on the trojaned encoder simultaneously inherit the trojan behavior. In this work, we propose TrojanDec, the first data-free method to identify and recover a test input embedded with a trigger. Given a (trojaned or clean) encoder and a test input, TrojanDec first predicts whether the test input is trojaned. If not, the test input is processed in a normal way to maintain the utility. Otherwise, the test input will be further restored to remove the trigger. Our extensive evaluation shows that TrojanDec can effectively identify the trojan (if any) from a given test input and recover it under state-of-the-art trojan attacks. We further demonstrate by experiments that our TrojanDec outperforms the state-of-the-art defenses.

Title: BiasGuard: Guardrailing Fairness in Machine Learning Production Systems

• Authors: Nurit Cohen-Inger, Seffi Cohen, Neomi Rabaev, Lior Rokach, Bracha Shapira
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.04142
• Pdf URL: https://arxiv.org/pdf/2501.04142
• Copy Paste: [[2501.04142]] BiasGuard: Guardrailing Fairness in Machine Learning Production Systems(https://arxiv.org/abs/2501.04142)
• Keywords: protect, fair, generative
• Abstract: As machine learning (ML) systems increasingly impact critical sectors such as hiring, financial risk assessments, and criminal justice, the imperative to ensure fairness has intensified due to potential negative implications. While much ML fairness research has focused on enhancing training data and processes, addressing the outputs of already deployed systems has received less attention. This paper introduces 'BiasGuard', a novel approach designed to act as a fairness guardrail in production ML systems. BiasGuard leverages Test-Time Augmentation (TTA) powered by Conditional Generative Adversarial Network (CTGAN), a cutting-edge generative AI model, to synthesize data samples conditioned on inverted protected attribute values, thereby promoting equitable outcomes across diverse groups. This method aims to provide equal opportunities for both privileged and unprivileged groups while significantly enhancing the fairness metrics of deployed systems without the need for retraining. Our comprehensive experimental analysis across diverse datasets reveals that BiasGuard enhances fairness by 31% while only reducing accuracy by 0.09% compared to non-mitigated benchmarks. Additionally, BiasGuard outperforms existing post-processing methods in improving fairness, positioning it as an effective tool to safeguard against biases when retraining the model is impractical.

Title: Chirpy3D: Continuous Part Latents for Creative 3D Bird Generation

• Authors: Kam Woh Ng, Jing Yang, Jia Wei Sii, Jiankang Deng, Chee Seng Chan, Yi-Zhe Song, Tao Xiang, Xiatian Zhu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.04144
• Pdf URL: https://arxiv.org/pdf/2501.04144
• Copy Paste: [[2501.04144]] Chirpy3D: Continuous Part Latents for Creative 3D Bird Generation(https://arxiv.org/abs/2501.04144)
• Keywords: diffusion
• Abstract: In this paper, we push the boundaries of fine-grained 3D generation into truly creative territory. Current methods either lack intricate details or simply mimic existing objects -- we enable both. By lifting 2D fine-grained understanding into 3D through multi-view diffusion and modeling part latents as continuous distributions, we unlock the ability to generate entirely new, yet plausible parts through interpolation and sampling. A self-supervised feature consistency loss further ensures stable generation of these unseen parts. The result is the first system capable of creating novel 3D objects with species-specific details that transcend existing examples. While we demonstrate our approach on birds, the underlying framework extends beyond things that can chirp! Code will be released at this https URL.

Title: KGIF: Optimizing Relation-Aware Recommendations with Knowledge Graph Information Fusion

• Authors: Dong Hyun Jeon, Wenbo Sun, Houbing Herbert Song, Dongfang Liu, Velasquez Alvaro, Yixin Chloe Xie, Shuteng Niu
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.04161
• Pdf URL: https://arxiv.org/pdf/2501.04161
• Copy Paste: [[2501.04161]] KGIF: Optimizing Relation-Aware Recommendations with Knowledge Graph Information Fusion(https://arxiv.org/abs/2501.04161)
• Keywords: robust
• Abstract: While deep-learning-enabled recommender systems demonstrate strong performance benchmarks, many struggle to adapt effectively in real-world environments due to limited use of user-item relationship data and insufficient transparency in recommendation generation. Traditional collaborative filtering approaches fail to integrate multifaceted item attributes, and although Factorization Machines account for item-specific details, they overlook broader relational patterns. Collaborative knowledge graph-based models have progressed by embedding user-item interactions with item-attribute relationships, offering a holistic perspective on interconnected entities. However, these models frequently aggregate attribute and interaction data in an implicit manner, leaving valuable relational nuances underutilized. This study introduces the Knowledge Graph Attention Network with Information Fusion (KGIF), a specialized framework designed to merge entity and relation embeddings explicitly through a tailored self-attention mechanism. The KGIF framework integrates reparameterization via dynamic projection vectors, enabling embeddings to adaptively represent intricate relationships within knowledge graphs. This explicit fusion enhances the interplay between user-item interactions and item-attribute relationships, providing a nuanced balance between user-centric and item-centric representations. An attentive propagation mechanism further optimizes knowledge graph embeddings, capturing multi-layered interaction patterns. The contributions of this work include an innovative method for explicit information fusion, improved robustness for sparse knowledge graphs, and the ability to generate explainable recommendations through interpretable path visualization.

Title: Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation

• Authors: Alireza Salemi, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, Tao Chen, Zhuowan Li, Michael Bendersky, Hamed Zamani
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.04167
• Pdf URL: https://arxiv.org/pdf/2501.04167
• Copy Paste: [[2501.04167]] Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation(https://arxiv.org/abs/2501.04167)
• Keywords: large language model
• Abstract: Personalized text generation requires a unique ability of large language models (LLMs) to learn from context that they often do not encounter during their standard training. One way to encourage LLMs to better use personalized context for generating outputs that better align with the user's expectations is to instruct them to reason over the user's past preferences, background knowledge, or writing style. To achieve this, we propose Reasoning-Enhanced Self-Training for Personalized Text Generation (REST-PG), a framework that trains LLMs to reason over personal data during response generation. REST-PG first generates reasoning paths to train the LLM's reasoning abilities and then employs Expectation-Maximization Reinforced Self-Training to iteratively train the LLM based on its own high-reward outputs. We evaluate REST-PG on the LongLaMP benchmark, consisting of four diverse personalized long-form text generation tasks. Our experiments demonstrate that REST-PG achieves significant improvements over state-of-the-art baselines, with an average relative performance gain of 14.5% on the benchmark.

Title: Multimodal Multihop Source Retrieval for Web Question Answering

• Authors: Navya Yarrabelly, Saloni Mittal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04173
• Pdf URL: https://arxiv.org/pdf/2501.04173
• Copy Paste: [[2501.04173]] Multimodal Multihop Source Retrieval for Web Question Answering(https://arxiv.org/abs/2501.04173)
• Keywords: transformer
• Abstract: This work deals with the challenge of learning and reasoning over multi-modal multi-hop question answering (QA). We propose a graph reasoning network based on the semantic structure of the sentences to learn multi-source reasoning paths and find the supporting facts across both image and text modalities for answering the question. In this paper, we investigate the importance of graph structure for multi-modal multi-hop question answering. Our analysis is centered on WebQA. We construct a strong baseline model, that finds relevant sources using a pairwise classification task. We establish that, with the proper use of feature representations from pre-trained models, graph structure helps in improving multi-modal multi-hop question answering. We point out that both graph structure and adjacency matrix are task-related prior knowledge, and graph structure can be leveraged to improve the retrieval performance for the task. Experiments and visualized analysis demonstrate that message propagation over graph networks or the entire graph structure can replace massive multimodal transformers with token-wise cross-attention. We demonstrated the applicability of our method and show a performance gain of \textbf{4.6$\%$} retrieval F1score over the transformer baselines, despite being a very light model. We further demonstrated the applicability of our model to a large scale retrieval setting.

Title: Generative Dataset Distillation Based on Self-knowledge Distillation

• Authors: Longzhen Li, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04202
• Pdf URL: https://arxiv.org/pdf/2501.04202
• Copy Paste: [[2501.04202]] Generative Dataset Distillation Based on Self-knowledge Distillation(https://arxiv.org/abs/2501.04202)
• Keywords: generative
• Abstract: Dataset distillation is an effective technique for reducing the cost and complexity of model training while maintaining performance by compressing large datasets into smaller, more efficient versions. In this paper, we present a novel generative dataset distillation method that can improve the accuracy of aligning prediction logits. Our approach integrates self-knowledge distillation to achieve more precise distribution matching between the synthetic and original data, thereby capturing the overall structure and relationships within the data. To further improve the accuracy of alignment, we introduce a standardization step on the logits before performing distribution matching, ensuring consistency in the range of logits. Through extensive experiments, we demonstrate that our method outperforms existing state-of-the-art methods, resulting in superior distillation performance.

Title: LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition

• Authors: Bowen Hao, Dongliang Zhou, Xiaojie Li, Xingyu Zhang, Liang Xie, Jianlong Wu, Erwei Yin
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2501.04204
• Pdf URL: https://arxiv.org/pdf/2501.04204
• Copy Paste: [[2501.04204]] LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition(https://arxiv.org/abs/2501.04204)
• Keywords: robust
• Abstract: Visual speech recognition (VSR), commonly known as lip reading, has garnered significant attention due to its wide-ranging practical applications. The advent of deep learning techniques and advancements in hardware capabilities have significantly enhanced the performance of lip reading models. Despite these advancements, existing datasets predominantly feature stable video recordings with limited variability in lip movements. This limitation results in models that are highly sensitive to variations encountered in real-world scenarios. To address this issue, we propose a novel framework, LipGen, which aims to improve model robustness by leveraging speech-driven synthetic visual data, thereby mitigating the constraints of current datasets. Additionally, we introduce an auxiliary task that incorporates viseme classification alongside attention mechanisms. This approach facilitates the efficient integration of temporal information, directing the model's focus toward the relevant segments of speech, thereby enhancing discriminative capabilities. Our method demonstrates superior performance compared to the current state-of-the-art on the lip reading in the wild (LRW) dataset and exhibits even more pronounced advantages under challenging conditions.

Title: CURing Large Models: Compression via CUR Decomposition

• Authors: Sanghyeon Park, Soo-Mook Moon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04211
• Pdf URL: https://arxiv.org/pdf/2501.04211
• Copy Paste: [[2501.04211]] CURing Large Models: Compression via CUR Decomposition(https://arxiv.org/abs/2501.04211)
• Keywords: interpretability
• Abstract: Large deep learning models have achieved remarkable success but are resource-intensive, posing challenges in computational cost and memory usage. We introduce CURing, a novel model compression method based on CUR matrix decomposition, which approximates weight matrices as the product of selected columns (C) and rows (R), and a small linking matrix (U). We apply this decomposition to weights chosen based on the combined influence of their magnitudes and activations. By identifying and retaining informative rows and columns, CURing significantly reduces model size with minimal performance loss. It preserves the original network's input/output structures, retains important features such as non-negativity, and the compressed model's activation patterns align with the original, thereby enhancing interpretability.

Title: Continual Self-supervised Learning Considering Medical Domain Knowledge in Chest CT Images

• Authors: Ren Tasai, Guang Li, Ren Togo, Minghui Tang, Takaaki Yoshimura, Hiroyuki Sugimori, Kenji Hirata, Takahiro Ogawa, Kohsuke Kudo, Miki Haseyama
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04217
• Pdf URL: https://arxiv.org/pdf/2501.04217
• Copy Paste: [[2501.04217]] Continual Self-supervised Learning Considering Medical Domain Knowledge in Chest CT Images(https://arxiv.org/abs/2501.04217)
• Keywords: robust
• Abstract: We propose a novel continual self-supervised learning method (CSSL) considering medical domain knowledge in chest CT images. Our approach addresses the challenge of sequential learning by effectively capturing the relationship between previously learned knowledge and new information at different stages. By incorporating an enhanced DER into CSSL and maintaining both diversity and representativeness within the rehearsal buffer of DER, the risk of data interference during pretraining is reduced, enabling the model to learn more richer and robust feature representations. In addition, we incorporate a mixup strategy and feature distillation to further enhance the model's ability to learn meaningful representations. We validate our method using chest CT images obtained under two different imaging conditions, demonstrating superior performance compared to state-of-the-art methods.

Title: IOLBENCH: Benchmarking LLMs on Linguistic Reasoning

• Authors: Satyam Goyal, Soham Dan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04249
• Pdf URL: https://arxiv.org/pdf/2501.04249
• Copy Paste: [[2501.04249]] IOLBENCH: Benchmarking LLMs on Linguistic Reasoning(https://arxiv.org/abs/2501.04249)
• Keywords: large language model
• Abstract: Despite the remarkable advancements and widespread applications of deep neural networks, their ability to perform reasoning tasks remains limited, particularly in domains requiring structured, abstract thought. In this paper, we investigate the linguistic reasoning capabilities of state-of-the-art large language models (LLMs) by introducing IOLBENCH, a novel benchmark derived from International Linguistics Olympiad (IOL) problems. This dataset encompasses diverse problems testing syntax, morphology, phonology, and semantics, all carefully designed to be self-contained and independent of external knowledge. These tasks challenge models to engage in metacognitive linguistic reasoning, requiring the deduction of linguistic rules and patterns from minimal examples. Through extensive benchmarking of leading LLMs, we find that even the most advanced models struggle to handle the intricacies of linguistic complexity, particularly in areas demanding compositional generalization and rule abstraction. Our analysis highlights both the strengths and persistent limitations of current models in linguistic problem-solving, offering valuable insights into their reasoning capabilities. By introducing IOLBENCH, we aim to foster further research into developing models capable of human-like reasoning, with broader implications for the fields of computational linguistics and artificial intelligence.

Title: Open set label noise learning with robust sample selection and margin-guided module

• Authors: Yuandi Zhao, Qianxi Xia, Yang Sun, Zhijie Wen, Liyan Ma, Shihui Ying
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04269
• Pdf URL: https://arxiv.org/pdf/2501.04269
• Copy Paste: [[2501.04269]] Open set label noise learning with robust sample selection and margin-guided module(https://arxiv.org/abs/2501.04269)
• Keywords: robust, noise learning
• Abstract: In recent years, the remarkable success of deep neural networks (DNNs) in computer vision is largely due to large-scale, high-quality labeled datasets. Training directly on real-world datasets with label noise may result in overfitting. The traditional method is limited to deal with closed set label noise, where noisy training data has true class labels within the known label space. However, there are some real-world datasets containing open set label noise, which means that some samples belong to an unknown class outside the known label space. To address the open set label noise problem, we introduce a method based on Robust Sample Selection and Margin-Guided Module (RSS-MGM). Firstly, unlike the prior clean sample selection approach, which only select a limited number of clean samples, a robust sample selection module combines small loss selection or high-confidence sample selection to obtain more clean samples. Secondly, to efficiently distinguish open set label noise and closed set ones, margin functions are designed to filter open-set data and closed set data. Thirdly, different processing methods are selected for different types of samples in order to fully utilize the data's prior information and optimize the whole model. Furthermore, extensive experimental results with noisy labeled data from benchmark datasets and real-world datasets, such as CIFAR-100N-C, CIFAR80N-O, WebFG-469, and Food101N, indicate that our approach outperforms many state-of-the-art label noise learning methods. Especially, it can more accurately divide open set label noise samples and closed set ones.

Title: ContextMRI: Enhancing Compressed Sensing MRI through Metadata Conditioning

• Authors: Hyungjin Chung, Dohun Lee, Zihui Wu, Byung-Hoon Kim, Katherine L. Bouman, Jong Chul Ye
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04284
• Pdf URL: https://arxiv.org/pdf/2501.04284
• Copy Paste: [[2501.04284]] ContextMRI: Enhancing Compressed Sensing MRI through Metadata Conditioning(https://arxiv.org/abs/2501.04284)
• Keywords: diffusion
• Abstract: Compressed sensing MRI seeks to accelerate MRI acquisition processes by sampling fewer k-space measurements and then reconstructing the missing data algorithmically. The success of these approaches often relies on strong priors or learned statistical models. While recent diffusion model-based priors have shown great potential, previous methods typically ignore clinically available metadata (e.g. patient demographics, imaging parameters, slice-specific information). In practice, metadata contains meaningful cues about the anatomy and acquisition protocol, suggesting it could further constrain the reconstruction problem. In this work, we propose ContextMRI, a text-conditioned diffusion model for MRI that integrates granular metadata into the reconstruction process. We train a pixel-space diffusion model directly on minimally processed, complex-valued MRI images. During inference, metadata is converted into a structured text prompt and fed to the model via CLIP text embeddings. By conditioning the prior on metadata, we unlock more accurate reconstructions and show consistent gains across multiple datasets, acceleration factors, and undersampling patterns. Our experiments demonstrate that increasing the fidelity of metadata, ranging from slice location and contrast to patient age, sex, and pathology, systematically boosts reconstruction performance. This work highlights the untapped potential of leveraging clinical context for inverse problems and opens a new direction for metadata-driven MRI reconstruction.

Title: Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models

• Authors: Bahman Torkamandi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04286
• Pdf URL: https://arxiv.org/pdf/2501.04286
• Copy Paste: [[2501.04286]] Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models(https://arxiv.org/abs/2501.04286)
• Keywords: transformer, large language model
• Abstract: In the realm of fractal geometry, intricate structures emerge from simple iterative processes that partition parameter spaces into regions of stability and instability. Likewise, training large language models involves iteratively applying update functions, such as Adam, where even slight hyperparameter adjustments can shift the training process from convergence to divergence. Recent evidence from miniature neural networks suggests that the boundary separating these outcomes displays fractal characteristics [1]. Building on these insights, this study extends them to medium-sized, decoder-only transformer architectures by employing a more consistent convergence measure and examining the learning rate hyperparameter landscape for attention and fully connected layers. The results show that the trainability frontier is not a simple threshold; rather, it forms a self-similar yet seemingly random structure at multiple scales, with statistically consistent and repeating patterns. Within this landscape, a region of stable convergence is surrounded by a complex chaotic border, illustrating the sensitive nature of the underlying training dynamics.

Title: An Analysis of Model Robustness across Concurrent Distribution Shifts

• Authors: Myeongho Jeon, Suhwan Choi, Hyoje Lee, Teresa Yeo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04288
• Pdf URL: https://arxiv.org/pdf/2501.04288
• Copy Paste: [[2501.04288]] An Analysis of Model Robustness across Concurrent Distribution Shifts(https://arxiv.org/abs/2501.04288)
• Keywords: robust
• Abstract: Machine learning models, meticulously optimized for source data, often fail to predict target data when faced with distribution shifts (DSs). Previous benchmarking studies, though extensive, have mainly focused on simple DSs. Recognizing that DSs often occur in more complex forms in real-world scenarios, we broadened our study to include multiple concurrent shifts, such as unseen domain shifts combined with spurious correlations. We evaluated 26 algorithms that range from simple heuristic augmentations to zero-shot inference using foundation models, across 168 source-target pairs from eight datasets. Our analysis of over 100K models reveals that (i) concurrent DSs typically worsen performance compared to a single shift, with certain exceptions, (ii) if a model improves generalization for one distribution shift, it tends to be effective for others, and (iii) heuristic data augmentations achieve the best overall performance on both synthetic and real-world datasets.

Title: TADFormer : Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learning

• Authors: Seungmin Baek, Soyul Lee, Hayeon Jo, Hyesong Choi, Dongbo Min
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04293
• Pdf URL: https://arxiv.org/pdf/2501.04293
• Copy Paste: [[2501.04293]] TADFormer : Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learning(https://arxiv.org/abs/2501.04293)
• Keywords: transformer
• Abstract: Transfer learning paradigm has driven substantial advancements in various vision tasks. However, as state-of-the-art models continue to grow, classical full fine-tuning often becomes computationally impractical, particularly in multi-task learning (MTL) setup where training complexity increases proportional to the number of tasks. Consequently, recent studies have explored Parameter-Efficient Fine-Tuning (PEFT) for MTL architectures. Despite some progress, these approaches still exhibit limitations in capturing fine-grained, task-specific features that are crucial to MTL. In this paper, we introduce Task-Adaptive Dynamic transFormer, termed TADFormer, a novel PEFT framework that performs task-aware feature adaptation in the fine-grained manner by dynamically considering task-specific input contexts. TADFormer proposes the parameter-efficient prompting for task adaptation and the Dynamic Task Filter (DTF) to capture task information conditioned on input contexts. Experiments on the PASCAL-Context benchmark demonstrate that the proposed method achieves higher accuracy in dense scene understanding tasks, while reducing the number of trainable parameters by up to 8.4 times when compared to full fine-tuning of MTL models. TADFormer also demonstrates superior parameter efficiency and accuracy compared to recent PEFT methods.

Title: Handling Incomplete Heterogeneous Data using a Data-Dependent Kernel

• Authors: Youran Zhou, Mohamed Reda Bouadjenek, Jonathan Wells, Sunil Aryal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04300
• Pdf URL: https://arxiv.org/pdf/2501.04300
• Copy Paste: [[2501.04300]] Handling Incomplete Heterogeneous Data using a Data-Dependent Kernel(https://arxiv.org/abs/2501.04300)
• Keywords: robust
• Abstract: Handling incomplete data in real-world applications is a critical challenge due to two key limitations of existing methods: (i) they are primarily designed for numeric data and struggle with categorical or heterogeneous/mixed datasets; (ii) they assume that data is missing completely at random, which is often not the case in practice -- in reality, data is missing in patterns, leading to biased results if these patterns are not accounted for. To address these two limitations, this paper presents a novel approach to handling missing values using the Probability Mass Similarity Kernel (PMK), a data-dependent kernel, which does not make any assumptions about data types and missing mechanisms. It eliminates the need for prior knowledge or extensive pre-processing steps and instead leverages the distribution of observed data. Our method unifies the representation of diverse data types by capturing more meaningful pairwise similarities and enhancing downstream performance. We evaluated our approach across over 10 datasets with numerical-only, categorical-only, and mixed features under different missing mechanisms and rates. Across both classification and clustering tasks, our approach consistently outperformed existing techniques, demonstrating its robustness and effectiveness in managing incomplete heterogeneous data.

Title: H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving

• Authors: Siran Chen, Yuxiao Luo, Yue Ma, Yu Qiao, Yali Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04302
• Pdf URL: https://arxiv.org/pdf/2501.04302
• Copy Paste: [[2501.04302]] H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving(https://arxiv.org/abs/2501.04302)
• Keywords: large language model
• Abstract: With the prevalence of Multimodal Large Language Models(MLLMs), autonomous driving has encountered new opportunities and challenges. In particular, multi-modal video understanding is critical to interactively analyze what will happen in the procedure of autonomous driving. However, videos in such a dynamical scene that often contains complex spatial-temporal movements, which restricts the generalization capacity of the existing MLLMs in this field. To bridge the gap, we propose a novel Hierarchical Mamba Adaptation (H-MBA) framework to fit the complicated motion changes in autonomous driving videos. Specifically, our H-MBA consists of two distinct modules, including Context Mamba (C-Mamba) and Query Mamba (Q-Mamba). First, C-Mamba contains various types of structure state space models, which can effectively capture multi-granularity video context for different temporal resolutions. Second, Q-Mamba flexibly transforms the current frame as the learnable query, and attentively selects multi-granularity video context into query. Consequently, it can adaptively integrate all the video contexts of multi-scale temporal resolutions to enhance video understanding. Via a plug-and-play paradigm in MLLMs, our H-MBA shows the remarkable performance on multi-modal video tasks in autonomous driving, e.g., for risk object detection, it outperforms the previous SOTA method with 5.5% mIoU improvement.

Title: Multimodal Graph Constrastive Learning and Prompt for ChartQA

• Authors: Yue Dai, Soyeon Caren Han, Wei Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04303
• Pdf URL: https://arxiv.org/pdf/2501.04303
• Copy Paste: [[2501.04303]] Multimodal Graph Constrastive Learning and Prompt for ChartQA(https://arxiv.org/abs/2501.04303)
• Keywords: transformer, large language model
• Abstract: ChartQA presents significant challenges due to the complex distribution of chart elements and the implicit patterns embedded within the underlying data. In this chapter, we have developed a joint multimodal scene graph for charts, explicitly representing the relationships between chart elements and their associated patterns. Our proposed multimodal scene graph consists of two components: a visual graph and a textual graph, each designed to capture the structural and semantic information within the chart. To unify representations across these different modalities, we introduce a multimodal graph contrastive learning approach that learns unified representations by maximizing similarity between nodes representing the same object across multimodal graphs. The learned graph representations can be seamlessly incorporated into a transformer decoder as a soft prompt. Additionally, given the growing need for Multimodal Large Language Models (MLLMs) in zero-shot scenarios, we have designed Chain-of-Thought (CoT) prompts for MLLMs to reduce hallucinations. We tested both methods on public benchmarks such as ChartQA, OpenCQA, and ChartX, demonstrating improved performance and validating the effectiveness of our proposed methods.

Title: DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models

• Authors: Hyogon Ryu, NaHyeon Park, Hyunjung Shim
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04304
• Pdf URL: https://arxiv.org/pdf/2501.04304
• Copy Paste: [[2501.04304]] DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models(https://arxiv.org/abs/2501.04304)
• Keywords: diffusion
• Abstract: Despite the widespread use of text-to-image diffusion models across various tasks, their computational and memory demands limit practical applications. To mitigate this issue, quantization of diffusion models has been explored. It reduces memory usage and computational costs by compressing weights and activations into lower-bit formats. However, existing methods often struggle to preserve both image quality and text-image alignment, particularly in lower-bit($<$ 8bits) quantization. In this paper, we analyze the challenges associated with quantizing text-to-image diffusion models from a distributional perspective. Our analysis reveals that activation outliers play a crucial role in determining image quality. Additionally, we identify distinctive patterns in cross-attention scores, which significantly affects text-image alignment. To address these challenges, we propose Distribution-aware Group Quantization (DGQ), a method that identifies and adaptively handles pixel-wise and channel-wise outliers to preserve image quality. Furthermore, DGQ applies prompt-specific logarithmic quantization scales to maintain text-image alignment. Our method demonstrates remarkable performance on datasets such as MS-COCO and PartiPrompts. We are the first to successfully achieve low-bit quantization of text-to-image diffusion models without requiring additional fine-tuning of weight quantization parameters.

Title: Physics-Informed Super-Resolution Diffusion for 6D Phase Space Diagnostics

• Authors: Alexander Scheinker
• Subjects: cs.LG, math.DS, physics.acc-ph
• Abstract URL: https://arxiv.org/abs/2501.04305
• Pdf URL: https://arxiv.org/pdf/2501.04305
• Copy Paste: [[2501.04305]] Physics-Informed Super-Resolution Diffusion for 6D Phase Space Diagnostics(https://arxiv.org/abs/2501.04305)
• Keywords: robust, diffusion
• Abstract: Adaptive physics-informed super-resolution diffusion is developed for non-invasive virtual diagnostics of the 6D phase space density of charged particle beams. An adaptive variational autoencoder (VAE) embeds initial beam condition images and scalar measurements to a low-dimensional latent space from which a 326 pixel 6D tensor representation of the beam's 6D phase space density is generated. Projecting from a 6D tensor generates physically consistent 2D projections. Physics-guided super-resolution diffusion transforms low-resolution images of the 6D density to high resolution 256x256 pixel images. Un-supervised adaptive latent space tuning enables tracking of time-varying beams without knowledge of time-varying initial conditions. The method is demonstrated with experimental data and multi-particle simulations at the HiRES UED. The general approach is applicable to a wide range of complex dynamic systems evolving in high-dimensional phase space. The method is shown to be robust to distribution shift without re-training.

Title: LLM4SR: A Survey on Large Language Models for Scientific Research

• Authors: Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, Xinya Du
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2501.04306
• Pdf URL: https://arxiv.org/pdf/2501.04306
• Copy Paste: [[2501.04306]] LLM4SR: A Survey on Large Language Models for Scientific Research(https://arxiv.org/abs/2501.04306)
• Keywords: large language model
• Abstract: In recent years, the rapid advancement of Large Language Models (LLMs) has transformed the landscape of scientific research, offering unprecedented support across various stages of the research cycle. This paper presents the first systematic survey dedicated to exploring how LLMs are revolutionizing the scientific research process. We analyze the unique roles LLMs play across four critical stages of research: hypothesis discovery, experiment planning and implementation, scientific writing, and peer reviewing. Our review comprehensively showcases the task-specific methodologies and evaluation benchmarks. By identifying current challenges and proposing future research directions, this survey not only highlights the transformative potential of LLMs, but also aims to inspire and guide researchers and practitioners in leveraging LLMs to advance scientific inquiry. Resources are available at the following repository: this https URL

Title: RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation

• Authors: Jun Liu, Zhenglun Kong, Peiyan Dong, Xuan Shen, Pu Zhao, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, Dong Huang, Yanzhi Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04315
• Pdf URL: https://arxiv.org/pdf/2501.04315
• Copy Paste: [[2501.04315]] RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation(https://arxiv.org/abs/2501.04315)
• Keywords: robust, large language model
• Abstract: Fine-tuning helps large language models (LLM) recover degraded information and enhance task this http URL Low-Rank Adaptation (LoRA) is widely used and effective for fine-tuning, we have observed that its scaling factor can limit or even reduce performance as the rank size increases. To address this issue, we propose RoRA (Rank-adaptive Reliability Optimization), a simple yet effective method for optimizing LoRA's scaling factor. By replacing $\alpha/r$ with $\alpha/\sqrt{r}$, RoRA ensures improved performance as rank size increases. Moreover, RoRA enhances low-rank adaptation in fine-tuning uncompressed models and excels in the more challenging task of accuracy recovery when fine-tuning pruned models. Extensive experiments demonstrate the effectiveness of RoRA in fine-tuning both uncompressed and pruned models. RoRA surpasses the state-of-the-art (SOTA) in average accuracy and robustness on LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B, specifically outperforming LoRA and DoRA by 6.5% and 2.9% on LLaMA-7B, respectively. In pruned model fine-tuning, RoRA shows significant advantages; for SHEARED-LLAMA-1.3, a LLaMA-7B with 81.4% pruning, RoRA achieves 5.7% higher average accuracy than LoRA and 3.9% higher than DoRA.

Title: Who Does the Giant Number Pile Like Best: Analyzing Fairness in Hiring Contexts

• Authors: Preethi Seshadri, Seraphina Goldfarb-Tarrant
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04316
• Pdf URL: https://arxiv.org/pdf/2501.04316
• Copy Paste: [[2501.04316]] Who Does the Giant Number Pile Like Best: Analyzing Fairness in Hiring Contexts(https://arxiv.org/abs/2501.04316)
• Keywords: fair, generative, large language model
• Abstract: Large language models (LLMs) are increasingly being deployed in high-stakes applications like hiring, yet their potential for unfair decision-making and outcomes remains understudied, particularly in generative settings. In this work, we examine the fairness of LLM-based hiring systems through two real-world tasks: resume summarization and retrieval. By constructing a synthetic resume dataset and curating job postings, we investigate whether model behavior differs across demographic groups and is sensitive to demographic perturbations. Our findings reveal that race-based differences appear in approximately 10% of generated summaries, while gender-based differences occur in only 1%. In the retrieval setting, all evaluated models display non-uniform selection patterns across demographic groups and exhibit high sensitivity to both gender and race-based perturbations. Surprisingly, retrieval models demonstrate comparable sensitivity to non-demographic changes, suggesting that fairness issues may stem, in part, from general brittleness issues. Overall, our results indicate that LLM-based hiring systems, especially at the retrieval stage, can exhibit notable biases that lead to discriminatory outcomes in real-world contexts.

Title: VerifBFL: Leveraging zk-SNARKs for A Verifiable Blockchained Federated Learning

• Authors: Ahmed Ayoub Bellachia, Mouhamed Amine Bouchiha, Yacine Ghamri-Doudane, Mourad Rabah
• Subjects: cs.CR, cs.DC, cs.ET, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04319
• Pdf URL: https://arxiv.org/pdf/2501.04319
• Copy Paste: [[2501.04319]] VerifBFL: Leveraging zk-SNARKs for A Verifiable Blockchained Federated Learning(https://arxiv.org/abs/2501.04319)
• Keywords: security, privacy, protect, attack, federate
• Abstract: Blockchain-based Federated Learning (FL) is an emerging decentralized machine learning paradigm that enables model training without relying on a central server. Although some BFL frameworks are considered privacy-preserving, they are still vulnerable to various attacks, including inference and model poisoning. Additionally, most of these solutions employ strong trust assumptions among all participating entities or introduce incentive mechanisms to encourage collaboration, making them susceptible to multiple security flaws. This work presents VerifBFL, a trustless, privacy-preserving, and verifiable federated learning framework that integrates blockchain technology and cryptographic protocols. By employing zero-knowledge Succinct Non-Interactive Argument of Knowledge (zk-SNARKs) and incrementally verifiable computation (IVC), VerifBFL ensures the verifiability of both local training and aggregation processes. The proofs of training and aggregation are verified on-chain, guaranteeing the integrity and auditability of each participant's contributions. To protect training data from inference attacks, VerifBFL leverages differential privacy. Finally, to demonstrate the efficiency of the proposed protocols, we built a proof of concept using emerging tools. The results show that generating proofs for local training and aggregation in VerifBFL takes less than 81s and 2s, respectively, while verifying them on-chain takes less than 0.6s.

Title: Navigating the Designs of Privacy-Preserving Fine-tuning for Large Language Models

• Authors: Shi Haonan, Ouyang Tu, Wang An
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.04323
• Pdf URL: https://arxiv.org/pdf/2501.04323
• Copy Paste: [[2501.04323]] Navigating the Designs of Privacy-Preserving Fine-tuning for Large Language Models(https://arxiv.org/abs/2501.04323)
• Keywords: privacy, protect, attack, large language model
• Abstract: Instruction tuning has proven effective in enhancing Large Language Models' (LLMs) performance on downstream tasks. However, real-world fine-tuning faces inherent conflicts between model providers' intellectual property protection, clients' data privacy requirements, and tuning costs. While recent approaches like split learning and offsite tuning demonstrate promising architectures for privacy-preserving fine-tuning, there is a gap in systematically addressing the multidimensional trade-offs required for diverse real-world deployments. We propose several indicative evaluation metrics to guide design trade-offs for privacy-preserving fine-tuning and a series of example designs, collectively named GuardedTuning; they result from novel combinations of system architectures with adapted privacy-enhancement methods and emerging computation techniques. Each design represents distinct trade-offs across model utility, privacy guarantees, and costs. Experimental results demonstrate that these designs protect against data reconstruction attacks while maintaining competitive fine-tuning performance.

Title: Edit as You See: Image-guided Video Editing via Masked Motion Modeling

• Authors: Zhi-Lin Huang, Yixuan Liu, Chujun Qin, Zhongdao Wang, Dong Zhou, Dong Li, Emad Barsoum
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04325
• Pdf URL: https://arxiv.org/pdf/2501.04325
• Copy Paste: [[2501.04325]] Edit as You See: Image-guided Video Editing via Masked Motion Modeling(https://arxiv.org/abs/2501.04325)
• Keywords: robust, diffusion
• Abstract: Recent advancements in diffusion models have significantly facilitated text-guided video editing. However, there is a relative scarcity of research on image-guided video editing, a method that empowers users to edit videos by merely indicating a target object in the initial frame and providing an RGB image as reference, without relying on the text prompts. In this paper, we propose a novel Image-guided Video Editing Diffusion model, termed IVEDiff for the image-guided video editing. IVEDiff is built on top of image editing models, and is equipped with learnable motion modules to maintain the temporal consistency of edited video. Inspired by self-supervised learning concepts, we introduce a masked motion modeling fine-tuning strategy that empowers the motion module's capabilities for capturing inter-frame motion dynamics, while preserving the capabilities for intra-frame semantic correlations modeling of the base image editing model. Moreover, an optical-flow-guided motion reference network is proposed to ensure the accurate propagation of information between edited video frames, alleviating the misleading effects of invalid information. We also construct a benchmark to facilitate further research. The comprehensive experiments demonstrate that our method is able to generate temporally smooth edited videos while robustly dealing with various editing objects with high quality.

Title: An Efficient Adaptive Compression Method for Human Perception and Machine Vision Tasks

• Authors: Lei Liu, Zhenghao Chen, Zhihao Hu, Dong Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04329
• Pdf URL: https://arxiv.org/pdf/2501.04329
• Copy Paste: [[2501.04329]] An Efficient Adaptive Compression Method for Human Perception and Machine Vision Tasks(https://arxiv.org/abs/2501.04329)
• Keywords: segmentation
• Abstract: While most existing neural image compression (NIC) and neural video compression (NVC) methodologies have achieved remarkable success, their optimization is primarily focused on human visual perception. However, with the rapid development of artificial intelligence, many images and videos will be used for various machine vision tasks. Consequently, such existing compression methodologies cannot achieve competitive performance in machine vision. In this work, we introduce an efficient adaptive compression (EAC) method tailored for both human perception and multiple machine vision tasks. Our method involves two key modules: 1), an adaptive compression mechanism, that adaptively selects several subsets from latent features to balance the optimizations for multiple machine vision tasks (e.g., segmentation, and detection) and human vision. 2), a task-specific adapter, that uses the parameter-efficient delta-tuning strategy to stimulate the comprehensive downstream analytical networks for specific machine vision tasks. By using the above two modules, we can optimize the bit-rate costs and improve machine vision performance. In general, our proposed EAC can seamlessly integrate with existing NIC (i.e., Ballé2018, and Cheng2020) and NVC (i.e., DVC, and FVC) methods. Extensive evaluation on various benchmark datasets (i.e., VOC2007, ILSVRC2012, VOC2012, COCO, UCF101, and DAVIS) shows that our method enhances performance for multiple machine vision tasks while maintaining the quality of human vision.

Title: Understanding Before Reasoning: Enhancing Chain-of-Thought with Iterative Summarization Pre-Prompting

• Authors: Dong-Hai Zhu, Yu-Jie Xiong, Jia-Chen Zhang, Xi-Jiong Xie, Chun-Ming Xia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04341
• Pdf URL: https://arxiv.org/pdf/2501.04341
• Copy Paste: [[2501.04341]] Understanding Before Reasoning: Enhancing Chain-of-Thought with Iterative Summarization Pre-Prompting(https://arxiv.org/abs/2501.04341)
• Keywords: extraction, large language model
• Abstract: Chain-of-Thought (CoT) Prompting is a dominant paradigm in Large Language Models (LLMs) to enhance complex reasoning. It guides LLMs to present multi-step reasoning, rather than generating the final answer directly. However, CoT encounters difficulties when key information required for reasoning is implicit or missing. This occurs because CoT emphasizes the sequence of reasoning steps while overlooking the early extraction of essential information. We propose a pre-prompting method called Iterative Summarization Pre-Prompting (ISP^2) to refine LLM reasoning when key information is not explicitly provided. First, entities and their corresponding descriptions are extracted to form potential key information pairs. Next, we use a reliability rating to assess these pairs, then merge the two lowest-ranked pairs into a new entity description. This process is repeated until a unique key information pair is obtained. Finally, that pair, along with the original question, is fed into LLMs to produce the answer. Extensive experiments demonstrate a 7.1% improvement compared to existing methods. Unlike traditional prompting, ISP^2 adopts an inductive approach with pre-prompting, offering flexible integration into diverse reasoning frameworks. The code is available at this https URL.

Title: DeFusion: An Effective Decoupling Fusion Network for Multi-Modal Pregnancy Prediction

• Authors: Xueqiang Ouyang, Jia Wei, Wenjie Huo, Xiaocong Wang, Rui Li, Jianlong Zhou
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04353
• Pdf URL: https://arxiv.org/pdf/2501.04353
• Copy Paste: [[2501.04353]] DeFusion: An Effective Decoupling Fusion Network for Multi-Modal Pregnancy Prediction(https://arxiv.org/abs/2501.04353)
• Keywords: transformer
• Abstract: Temporal embryo images and parental fertility table indicators are both valuable for pregnancy prediction in \textbf{in vitro fertilization embryo transfer} (IVF-ET). However, current machine learning models cannot make full use of the complementary information between the two modalities to improve pregnancy prediction performance. In this paper, we propose a Decoupling Fusion Network called DeFusion to effectively integrate the multi-modal information for IVF-ET pregnancy prediction. Specifically, we propose a decoupling fusion module that decouples the information from the different modalities into related and unrelated information, thereby achieving a more delicate fusion. And we fuse temporal embryo images with a spatial-temporal position encoding, and extract fertility table indicator information with a table transformer. To evaluate the effectiveness of our model, we use a new dataset including 4046 cases collected from Southern Medical University. The experiments show that our model outperforms state-of-the-art methods. Meanwhile, the performance on the eye disease prediction dataset reflects the model's good generalization. Our code and dataset are available at this https URL.

Title: Real-world actor-based image steganalysis via classifier inconsistency detection

• Authors: Daniel Lerch-Hostalot, David Megías
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.04362
• Pdf URL: https://arxiv.org/pdf/2501.04362
• Copy Paste: [[2501.04362]] Real-world actor-based image steganalysis via classifier inconsistency detection(https://arxiv.org/abs/2501.04362)
• Keywords: robust, extraction
• Abstract: In this paper, we propose a robust method for detecting guilty actors in image steganography while effectively addressing the Cover Source Mismatch (CSM) problem, which arises when classifying images from one source using a classifier trained on images from another source. Designed for an actor-based scenario, our method combines the use of Detection of Classifier Inconsistencies (DCI) prediction with EfficientNet neural networks for feature extraction, and a Gradient Boosting Machine for the final classification. The proposed approach successfully determines whether an actor is innocent or guilty, or if they should be discarded due to excessive CSM. We show that the method remains reliable even in scenarios with high CSM, consistently achieving accuracy above 80% and outperforming the baseline method. This novel approach contributes to the field of steganalysis by offering a practical and efficient solution for handling CSM and detecting guilty actors in real-world applications.

Title: Instructive3D: Editing Large Reconstruction Models with Text Instructions

• Authors: Kunal Kathare, Ankit Dhiman, K Vikas Gowda, Siddharth Aravindan, Shubham Monga, Basavaraja Shanthappa Vandrotti, Lokesh R Boregowda
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04374
• Pdf URL: https://arxiv.org/pdf/2501.04374
• Copy Paste: [[2501.04374]] Instructive3D: Editing Large Reconstruction Models with Text Instructions(https://arxiv.org/abs/2501.04374)
• Keywords: diffusion, transformer
• Abstract: Transformer based methods have enabled users to create, modify, and comprehend text and image data. Recently proposed Large Reconstruction Models (LRMs) further extend this by providing the ability to generate high-quality 3D models with the help of a single object image. These models, however, lack the ability to manipulate or edit the finer details, such as adding standard design patterns or changing the color and reflectance of the generated objects, thus lacking fine-grained control that may be very helpful in domains such as augmented reality, animation and gaming. Naively training LRMs for this purpose would require generating precisely edited images and 3D object pairs, which is computationally expensive. In this paper, we propose Instructive3D, a novel LRM based model that integrates generation and fine-grained editing, through user text prompts, of 3D objects into a single model. We accomplish this by adding an adapter that performs a diffusion process conditioned on a text prompt specifying edits in the triplane latent space representation of 3D object models. Our method does not require the generation of edited 3D objects. Additionally, Instructive3D allows us to perform geometrically consistent modifications, as the edits done through user-defined text prompts are applied to the triplane latent representation thus enhancing the versatility and precision of 3D objects generated. We compare the objects generated by Instructive3D and a baseline that first generates the 3D object meshes using a standard LRM model and then edits these 3D objects using text prompts when images are provided from the Objaverse LVIS dataset. We find that Instructive3D produces qualitatively superior 3D objects with the properties specified by the edit prompts.

Title: Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixing

• Authors: Xinghe Fu, Zhiyuan Yan, Taiping Yao, Shen Chen, Xi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04376
• Pdf URL: https://arxiv.org/pdf/2501.04376
• Copy Paste: [[2501.04376]] Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixing(https://arxiv.org/abs/2501.04376)
• Keywords: transformer
• Abstract: The generalization problem is broadly recognized as a critical challenge in detecting deepfakes. Most previous work believes that the generalization gap is caused by the differences among various forgery methods. However, our investigation reveals that the generalization issue can still occur when forgery-irrelevant factors shift. In this work, we identify two biases that detectors may also be prone to overfitting: position bias and content bias, as depicted in Fig. 1. For the position bias, we observe that detectors are prone to lazily depending on the specific positions within an image (e.g., central regions even no forgery). As for content bias, we argue that detectors may potentially and mistakenly utilize forgery-unrelated information for detection (e.g., background, and hair). To intervene these biases, we propose two branches for shuffling and mixing with tokens in the latent space of transformers. For the shuffling branch, we rearrange the tokens and corresponding position embedding for each image while maintaining the local correlation. For the mixing branch, we randomly select and mix the tokens in the latent space between two images with the same label within the mini-batch to recombine the content information. During the learning process, we align the outputs of detectors from different branches in both feature space and logit space. Contrastive losses for features and divergence losses for logits are applied to obtain unbiased feature representation and classifiers. We demonstrate and verify the effectiveness of our method through extensive experiments on widely used evaluation datasets.

Title: iFADIT: Invertible Face Anonymization via Disentangled Identity Transform

• Authors: Lin Yuan, Kai Liang, Xiong Li, Tao Wu, Nannan Wang, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04390
• Pdf URL: https://arxiv.org/pdf/2501.04390
• Copy Paste: [[2501.04390]] iFADIT: Invertible Face Anonymization via Disentangled Identity Transform(https://arxiv.org/abs/2501.04390)
• Keywords: secure, security, privacy, attack, interpretability, generative
• Abstract: Face anonymization aims to conceal the visual identity of a face to safeguard the individual's privacy. Traditional methods like blurring and pixelation can largely remove identifying features, but these techniques significantly degrade image quality and are vulnerable to deep reconstruction attacks. Generative models have emerged as a promising solution for anonymizing faces while preserving a natural this http URL, many still face limitations in visual quality and often overlook the potential to recover the original face from the anonymized version, which can be valuable in specific contexts such as image forensics. This paper proposes a novel framework named iFADIT, an acronym for Invertible Face Anonymization via Disentangled Identity this http URL framework features a disentanglement architecture coupled with a secure flow-based model: the former decouples identity information from non-identifying attributes, while the latter transforms the decoupled identity into an anonymized version in an invertible manner controlled by a secret key. The anonymized face can then be reconstructed based on a pre-trained StyleGAN that ensures high image quality and realistic facial details. Recovery of the original face (aka de-anonymization) is possible upon the availability of the matching secret, by inverting the anonymization process based on the same set of model parameters. Furthermore, a dedicated secret-key mechanism along with a dual-phase training strategy is devised to ensure the desired properties of face anonymization. Qualitative and quantitative experiments demonstrate the superiority of the proposed approach in anonymity, reversibility, security, diversity, and interpretability over competing methods.

Title: SEO: Stochastic Experience Optimization for Large Language Models

• Authors: Jitao Xu, Hongyun Zhou, Lei Shen, Conghui Zhu, Jin Huang, Yitao Duan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04393
• Pdf URL: https://arxiv.org/pdf/2501.04393
• Copy Paste: [[2501.04393]] SEO: Stochastic Experience Optimization for Large Language Models(https://arxiv.org/abs/2501.04393)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) can benefit from useful experiences to improve their performance on specific tasks. However, finding helpful experiences for different LLMs is not obvious, since it is unclear what experiences suit specific LLMs. Previous studies intended to automatically find useful experiences using LLMs, while it is difficult to ensure the effectiveness of the obtained experience. In this paper, we propose Stochastic Experience Optimization (SEO), an iterative approach that finds optimized model-specific experience without modifying model parameters through experience update in natural language. In SEO, we propose a stochastic validation method to ensure the update direction of experience, avoiding unavailing updates. Experimental results on three tasks for three LLMs demonstrate that experiences optimized by SEO can achieve consistently improved performance. Further analysis indicates that SEO-optimized experience can generalize to out-of-distribution data, boosting the performance of LLMs on similar tasks.

Title: Modern Hardware Security: A Review of Attacks and Countermeasures

• Authors: Jyotiprakash Mishra, Sanjay K. Sahay
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2501.04394
• Pdf URL: https://arxiv.org/pdf/2501.04394
• Copy Paste: [[2501.04394]] Modern Hardware Security: A Review of Attacks and Countermeasures(https://arxiv.org/abs/2501.04394)
• Keywords: secure, security, protect, defense, attack, robust
• Abstract: With the exponential rise in the use of cloud services, smart devices, and IoT devices, advanced cyber attacks have become increasingly sophisticated and ubiquitous. Furthermore, the rapid evolution of computing architectures and memory technologies has created an urgent need to understand and address hardware security vulnerabilities. In this paper, we review the current state of vulnerabilities and mitigation strategies in contemporary computing systems. We discuss cache side-channel attacks (including Spectre and Meltdown), power side-channel attacks (such as Simple Power Analysis, Differential Power Analysis, Correlation Power Analysis, and Template Attacks), and advanced techniques like Voltage Glitching and Electromagnetic Analysis to help understand and build robust cybersecurity defense systems and guide further research. We also examine memory encryption, focusing on confidentiality, granularity, key management, masking, and re-keying strategies. Additionally, we cover Cryptographic Instruction Set Architectures, Secure Boot, Root of Trust mechanisms, Physical Unclonable Functions, and hardware fault injection techniques. The paper concludes with an analysis of the RISC-V architecture's unique security challenges. The comprehensive analysis presented in this paper is essential for building resilient hardware security solutions that can protect against both current and emerging threats in an increasingly challenging security landscape.

Title: Tracking UWB Devices Through Radio Frequency Fingerprinting Is Possible

• Authors: Thibaud Ardoin, Niklas Pauli, Benedikt Groß, Mahsa Kholghi, Khan Reaz, Gerhard Wunder
• Subjects: cs.LG, cs.IT, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.04401
• Pdf URL: https://arxiv.org/pdf/2501.04401
• Copy Paste: [[2501.04401]] Tracking UWB Devices Through Radio Frequency Fingerprinting Is Possible(https://arxiv.org/abs/2501.04401)
• Keywords: security, privacy
• Abstract: Ultra-wideband (UWB) is a state-of-the-art technology designed for applications requiring centimeter-level localization. Its widespread adoption by smartphone manufacturer naturally raises security and privacy concerns. Successfully implementing Radio Frequency Fingerprinting (RFF) to UWB could enable physical layer security, but might also allow undesired tracking of the devices. The scope of this paper is to explore the feasibility of applying RFF to UWB and investigates how well this technique generalizes across different environments. We collected a realistic dataset using off-the-shelf UWB devices with controlled variation in device positioning. Moreover, we developed an improved deep learning pipeline to extract the hardware signature from the signal data. In stable conditions, the extracted RFF achieves over 99% accuracy. While the accuracy decreases in more changing environments, we still obtain up to 76% accuracy in untrained locations.

Title: Lossless Privacy-Preserving Aggregation for Decentralized Federated Learning

• Authors: Xiaoye Miao, Bin Li, Yangyang Wu, Meng Xi, Xinkui Zhao, Jianwei Yin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04409
• Pdf URL: https://arxiv.org/pdf/2501.04409
• Copy Paste: [[2501.04409]] Lossless Privacy-Preserving Aggregation for Decentralized Federated Learning(https://arxiv.org/abs/2501.04409)
• Keywords: privacy, protect, federate
• Abstract: Privacy concerns arise as sensitive data proliferate. Despite decentralized federated learning (DFL) aggregating gradients from neighbors to avoid direct data transmission, it still poses indirect data leaks from the transmitted gradients. Existing privacy-preserving methods for DFL add noise to gradients. They either diminish the model predictive accuracy or suffer from ineffective gradient protection. In this paper, we propose a novel lossless privacy-preserving aggregation rule named LPPA to enhance gradient protection as much as possible but without loss of DFL model predictive accuracy. LPPA subtly injects the noise difference between the sent and received noise into transmitted gradients for gradient protection. The noise difference incorporates neighbors' randomness for each client, effectively safeguarding against data leaks. LPPA employs the noise flow conservation theory to ensure that the noise impact can be globally eliminated. The global sum of all noise differences remains zero, ensuring that accurate gradient aggregation is unaffected and the model accuracy remains intact. We theoretically prove that the privacy-preserving capacity of LPPA is \sqrt{2} times greater than that of noise addition, while maintaining comparable model accuracy to the standard DFL aggregation without noise injection. Experimental results verify the theoretical findings and show that LPPA achieves a 13% mean improvement in accuracy over noise addition. We also demonstrate the effectiveness of LPPA in protecting raw data and guaranteeing lossless model accuracy.

Title: End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated Approach

• Authors: H.M. Shadman Tabib, Jaber Ahmed Deedar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04425
• Pdf URL: https://arxiv.org/pdf/2501.04425
• Copy Paste: [[2501.04425]] End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated Approach(https://arxiv.org/abs/2501.04425)
• Keywords: large language model
• Abstract: This work introduces systematic approach for enhancing large language models (LLMs) to address Bangla AI mathematical challenges. Through the assessment of diverse LLM configurations, fine-tuning with specific datasets, and the implementation of Retrieval-Augmented Generation (RAG), we enhanced the model's reasoning precision in a multilingual setting. Crucial discoveries indicate that customized prompting, dataset augmentation, and iterative reasoning improve the model's efficiency regarding Olympiad-level mathematical challenges.

Title: Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions

• Authors: Na Yan, Yang Su, Yansha Deng, Robert Schober
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04436
• Pdf URL: https://arxiv.org/pdf/2501.04436
• Copy Paste: [[2501.04436]] Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions(https://arxiv.org/abs/2501.04436)
• Keywords: privacy, federate, large language model
• Abstract: Federated learning (FL) provides a privacy-preserving solution for fine-tuning pre-trained large language models (LLMs) using distributed private datasets, enabling task-specific adaptation while preserving data privacy. However, fine-tuning the extensive parameters in LLMs is particularly challenging in resource-constrained federated scenarios due to the significant communication and computational costs. To gain a deeper understanding of how these challenges can be addressed, this article conducts a comparative analysis three advanced federated LLM (FedLLM) frameworks that integrate knowledge distillation (KD) and split learning (SL) to mitigate these issues: 1) FedLLMs, where clients upload model parameters or gradients to enable straightforward and effective fine-tuning; 2) KD-FedLLMs, which leverage KD for efficient knowledge sharing via logits; and 3) Split-FedLLMs, which split the LLMs into two parts, with one part executed on the client and the other one on the server, to balance the computational load. Each framework is evaluated based on key performance metrics, including model accuracy, communication overhead, and client-side computational load, offering insights into their effectiveness for various federated fine-tuning scenarios. Through this analysis, we identify framework-specific optimization opportunities to enhance the efficiency of FedLLMs and discuss broader research directions, highlighting open opportunities to better adapt FedLLMs for real-world applications. A use case is presented to demonstrate the performance comparison of these three frameworks under varying configurations and settings.

Title: A novel Facial Recognition technique with Focusing on Masked Faces

• Authors: Dana A Abdullah, Dana Rasul Hamad, Hakem Beitollahi, Ismail Y Maolood, Abdulhady Abas Abdullah, Aso Khaleel Ameen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04444
• Pdf URL: https://arxiv.org/pdf/2501.04444
• Copy Paste: [[2501.04444]] A novel Facial Recognition technique with Focusing on Masked Faces(https://arxiv.org/abs/2501.04444)
• Keywords: security
• Abstract: Recognizing the same faces with and without masks is important for ensuring consistent identification in security, access control, and public safety. This capability is crucial in scenarios like law enforcement, healthcare, and surveillance, where accurate recognition must be maintained despite facial occlusion. This research focuses on the challenge of recognizing the same faces with and without masks by employing cosine similarity as the primary technique. With the increased use of masks, traditional facial recognition systems face significant accuracy issues, making it crucial to develop methods that can reliably identify individuals in masked conditions. For that reason, this study proposed Masked-Unmasked Face Matching Model (MUFM). This model employs transfer learning using the Visual Geometry Group (VGG16) model to extract significant facial features, which are subsequently classified utilizing the K-Nearest Neighbors (K-NN) algorithm. The cosine similarity metric is employed to compare masked and unmasked faces of the same individuals. This approach represents a novel contribution, as the task of recognizing the same individual with and without a mask using cosine similarity has not been previously addressed. By integrating these advanced methodologies, the research demonstrates effective identification of individuals despite the presence of masks, addressing a significant limitation in traditional systems. Using data is another essential part of this work, by collecting and preparing an image dataset from three different sources especially some of those data are real provided a comprehensive power of this research. The image dataset used were already collected in three different datasets of masked and unmasked for the same faces.

Title: Gradient Purification: Defense Against Poisoning Attack in Decentralized Federated Learning

• Authors: Bin Li, Xiaoye Miao, Yongheng Shang, Xinkui Zhao, Shuiguang Deng, Jianwei Yin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04453
• Pdf URL: https://arxiv.org/pdf/2501.04453
• Copy Paste: [[2501.04453]] Gradient Purification: Defense Against Poisoning Attack in Decentralized Federated Learning(https://arxiv.org/abs/2501.04453)
• Keywords: defense, attack, federate
• Abstract: Decentralized federated learning (DFL) is inherently vulnerable to poisoning attacks, as malicious clients can transmit manipulated model gradients to neighboring clients. Existing defense methods either reject suspicious gradients per iteration or restart DFL aggregation after detecting all malicious clients. They overlook the potential accuracy benefit from the discarded malicious gradients. In this paper, we propose a novel gradient purification defense, named GPD, that integrates seamlessly with existing DFL aggregation to defend against poisoning attacks. It aims to mitigate the harm in model gradients while retaining the benefit in model weights for enhancing accuracy. For each benign client in GPD, a recording variable is designed to track the historically aggregated gradients from one of its neighbors. It allows benign clients to precisely detect malicious neighbors and swiftly mitigate aggregated malicious gradients via historical consistency checks. Upon mitigation, GPD optimizes model weights via aggregating gradients solely from benign clients. This retains the previously beneficial portions from malicious clients and exploits the contributions from benign clients, thereby significantly enhancing the model accuracy. We analyze the convergence of GPD, as well as its ability to harvest high accuracy. Extensive experiments over three datasets demonstrate that, GPD is capable of mitigating poisoning attacks under both iid and non-iid data distributions. It significantly outperforms state-of-the-art defenses in terms of accuracy against various poisoning attacks.

Title: A Taxonomy of Functional Security Features and How They Can Be Located

• Authors: Kevin Hermann, Simon Schneider, Catherine Tony, Asli Yardim, Sven Peldszus, Thorsten Berger, Riccardo Scandariato, M. Angela Sasse, Alena Naiakshina
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.04454
• Pdf URL: https://arxiv.org/pdf/2501.04454
• Copy Paste: [[2501.04454]] A Taxonomy of Functional Security Features and How They Can Be Located(https://arxiv.org/abs/2501.04454)
• Keywords: secure, security
• Abstract: Security must be considered in almost every software system. Unfortunately, selecting and implementing security features remains challenging due to the variety of security threats and possible countermeasures. While security standards are intended to help developers, they are usually too abstract and vague to help implement security features, or they merely help configure such. A resource that describes security features at an abstraction level between high-level (i.e., rather too general) and low-level (i.e., rather too specific) security standards could facilitate secure systems development. To realize security features, developers typically use external security frameworks, to minimize implementation mistakes. Even then, developers still make mistakes, often resulting in security vulnerabilities. When security incidents occur or the system needs to be audited or maintained, it is essential to know the implemented security features and, more importantly, where they are located. This task, commonly referred to as feature location, is often tedious and error-prone. Therefore, we have to support long-term tracking of implemented security features. We present a study of security features in the literature and their coverage in popular security frameworks. We contribute (1) a taxonomy of 68 functional implementation-level security features including a mapping to widely used security standards, (2) an examination of 21 popular security frameworks concerning which of these security features they provide, and (3) a discussion on the representation of security features in source code. Our taxonomy aims to aid developers in selecting appropriate security features and frameworks and relating them to security standards when they need to choose and implement security features for a software system.

Title: Hidden Entity Detection from GitHub Leveraging Large Language Models

• Authors: Lu Gan, Martin Blum, Danilo Dessi, Brigitte Mathiak, Ralf Schenkel, Stefan Dietze
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2501.04455
• Pdf URL: https://arxiv.org/pdf/2501.04455
• Copy Paste: [[2501.04455]] Hidden Entity Detection from GitHub Leveraging Large Language Models(https://arxiv.org/abs/2501.04455)
• Keywords: large language model
• Abstract: Named entity recognition is an important task when constructing knowledge bases from unstructured data sources. Whereas entity detection methods mostly rely on extensive training data, Large Language Models (LLMs) have paved the way towards approaches that rely on zero-shot learning (ZSL) or few-shot learning (FSL) by taking advantage of the capabilities LLMs acquired during pretraining. Specifically, in very specialized scenarios where large-scale training data is not available, ZSL / FSL opens new opportunities. This paper follows this recent trend and investigates the potential of leveraging Large Language Models (LLMs) in such scenarios to automatically detect datasets and software within textual content from GitHub repositories. While existing methods focused solely on named entities, this study aims to broaden the scope by incorporating resources such as repositories and online hubs where entities are also represented by URLs. The study explores different FSL prompt learning approaches to enhance the LLMs' ability to identify dataset and software mentions within repository texts. Through analyses of LLM effectiveness and learning strategies, this paper offers insights into the potential of advanced language models for automated entity detection.

Title: When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages

• Authors: Archchana Sindhujan, Diptesh Kanojia, Constantin Orasan, Shenbin Qian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04473
• Pdf URL: https://arxiv.org/pdf/2501.04473
• Copy Paste: [[2501.04473]] When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages(https://arxiv.org/abs/2501.04473)
• Keywords: large language model
• Abstract: This paper investigates the reference-less evaluation of machine translation for low-resource language pairs, known as quality estimation (QE). Segment-level QE is a challenging cross-lingual language understanding task that provides a quality score (0-100) to the translated output. We comprehensively evaluate large language models (LLMs) in zero/few-shot scenarios and perform instruction fine-tuning using a novel prompt based on annotation guidelines. Our results indicate that prompt-based approaches are outperformed by the encoder-based fine-tuned QE models. Our error analysis reveals tokenization issues, along with errors due to transliteration and named entities, and argues for refinement in LLM pre-training for cross-lingual tasks. We release the data, and models trained publicly for further research.

Title: Rethinking High-speed Image Reconstruction Framework with Spike Camera

• Authors: Kang Chen, Yajing Zheng, Tiejun Huang, Zhaofei Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04477
• Pdf URL: https://arxiv.org/pdf/2501.04477
• Copy Paste: [[2501.04477]] Rethinking High-speed Image Reconstruction Framework with Spike Camera(https://arxiv.org/abs/2501.04477)
• Keywords: robust
• Abstract: Spike cameras, as innovative neuromorphic devices, generate continuous spike streams to capture high-speed scenes with lower bandwidth and higher dynamic range than traditional RGB cameras. However, reconstructing high-quality images from the spike input under low-light conditions remains challenging. Conventional learning-based methods often rely on the synthetic dataset as the supervision for training. Still, these approaches falter when dealing with noisy spikes fired under the low-light environment, leading to further performance degradation in the real-world dataset. This phenomenon is primarily due to inadequate noise modelling and the domain gap between synthetic and real datasets, resulting in recovered images with unclear textures, excessive noise, and diminished brightness. To address these challenges, we introduce a novel spike-to-image reconstruction framework SpikeCLIP that goes beyond traditional training paradigms. Leveraging the CLIP model's powerful capability to align text and images, we incorporate the textual description of the captured scene and unpaired high-quality datasets as the supervision. Our experiments on real-world low-light datasets U-CALTECH and U-CIFAR demonstrate that SpikeCLIP significantly enhances texture details and the luminance balance of recovered images. Furthermore, the reconstructed images are well-aligned with the broader visual features needed for downstream tasks, ensuring more robust and versatile performance in challenging environments.

Title: MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration

• Authors: Zhi Jin, Yuwei Qiu, Kaihao Zhang, Hongdong Li, Wenhan Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04486
• Pdf URL: https://arxiv.org/pdf/2501.04486
• Copy Paste: [[2501.04486]] MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration(https://arxiv.org/abs/2501.04486)
• Keywords: transformer
• Abstract: Recently, Transformer networks have demonstrated outstanding performance in the field of image restoration due to the global receptive field and adaptability to input. However, the quadratic computational complexity of Softmax-attention poses a significant limitation on its extensive application in image restoration tasks, particularly for high-resolution images. To tackle this challenge, we propose a novel variant of the Transformer. This variant leverages the Taylor expansion to approximate the Softmax-attention and utilizes the concept of norm-preserving mapping to approximate the remainder of the first-order Taylor expansion, resulting in a linear computational complexity. Moreover, we introduce a multi-branch architecture featuring multi-scale patch embedding into the proposed Transformer, which has four distinct advantages: 1) various sizes of the receptive field; 2) multi-level semantic information; 3) flexible shapes of the receptive field; 4) accelerated training and inference speed. Hence, the proposed model, named the second version of Taylor formula expansion-based Transformer (for short MB-TaylorFormer V2) has the capability to concurrently process coarse-to-fine features, capture long-distance pixel interactions with limited computational cost, and improve the approximation of the Taylor expansion remainder. Experimental results across diverse image restoration benchmarks demonstrate that MB-TaylorFormer V2 achieves state-of-the-art performance in multiple image restoration tasks, such as image dehazing, deraining, desnowing, motion deblurring, and denoising, with very little computational overhead. The source code is available at this https URL.

Title: Integrating remote sensing data assimilation, deep learning and large language model for interactive wheat breeding yield prediction

• Authors: Guofeng Yang, Nanfei Jin, Wenjie Ai, Zhonghua Zheng, Yuhong He, Yong He
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04487
• Pdf URL: https://arxiv.org/pdf/2501.04487
• Copy Paste: [[2501.04487]] Integrating remote sensing data assimilation, deep learning and large language model for interactive wheat breeding yield prediction(https://arxiv.org/abs/2501.04487)
• Keywords: transformer, large language model
• Abstract: Yield is one of the core goals of crop breeding. By predicting the potential yield of different breeding materials, breeders can screen these materials at various growth stages to select the best performing. Based on unmanned aerial vehicle remote sensing technology, high-throughput crop phenotyping data in breeding areas is collected to provide data support for the breeding decisions of breeders. However, the accuracy of current yield predictions still requires improvement, and the usability and user-friendliness of yield forecasting tools remain suboptimal. To address these challenges, this study introduces a hybrid method and tool for crop yield prediction, designed to allow breeders to interactively and accurately predict wheat yield by chatting with a large language model (LLM). First, the newly designed data assimilation algorithm is used to assimilate the leaf area index into the WOFOST model. Then, selected outputs from the assimilation process, along with remote sensing inversion results, are used to drive the time-series temporal fusion transformer model for wheat yield prediction. Finally, based on this hybrid method and leveraging an LLM with retrieval augmented generation technology, we developed an interactive yield prediction Web tool that is user-friendly and supports sustainable data updates. This tool integrates multi-source data to assist breeding decision-making. This study aims to accelerate the identification of high-yield materials in the breeding process, enhance breeding efficiency, and enable more scientific and smart breeding decisions.

Title: Multichannel Steganography: A Provably Secure Hybrid Steganographic Model for Secure Communication

• Authors: Obinna Omego, Michal Bosy
• Subjects: cs.CR, cs.MM
• Abstract URL: https://arxiv.org/abs/2501.04511
• Pdf URL: https://arxiv.org/pdf/2501.04511
• Copy Paste: [[2501.04511]] Multichannel Steganography: A Provably Secure Hybrid Steganographic Model for Secure Communication(https://arxiv.org/abs/2501.04511)
• Keywords: secure, security, attack, robust
• Abstract: This study introduces a novel steganographic model that synthesizes Steganography by Cover Modification (CMO) and Steganography by Cover Synthesis (CSY), enhancing both security and undetectability by generating cover messages or parameters while retaining the original cover's form, thus minimizing detection risks and overcoming the limitations of single-method techniques. Building upon this model, a refined Steganographic Communication Protocol is proposed, enhancing resilience against sophisticated threats such as Multichannel Replay Attacks and Multichannel Man-in-the-Middle Attacks, fortifying the protocol against potential tampering and improving upon prior works. To evaluate the security of the proposed protocol, a novel adversarial model is developed simulating a probabilistic polynomial time (PPT) adversary capable of intercepting communications across multiple channels. This model assesses the adversary's ability to compromise the protocol, providing a comprehensive security analysis. Finally, this study explores the practicality and adaptability of the model to both constrained environments like SMS banking and resource-rich settings such as blockchain transactions, demonstrating their potential to enhance financial services and security. These contributions present a robust and adaptable framework for secure steganographic communication, offering practical solutions for secure communications across diverse environments.

Title: Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-time

• Authors: Uri Berger, Omri Abend, Lea Frermann, Gabriel Stanovsky
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04513
• Pdf URL: https://arxiv.org/pdf/2501.04513
• Copy Paste: [[2501.04513]] Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-time(https://arxiv.org/abs/2501.04513)
• Keywords: robust, generative
• Abstract: Incorporating automatically predicted human feedback into the process of training generative models has attracted substantial recent interest, while feedback at inference time has received less attention. The typical feedback at training time, i.e., preferences of choice given two samples, does not naturally transfer to the inference phase. We introduce a novel type of feedback -- caption reformulations -- and train models to mimic reformulation feedback based on human annotations. Our method does not require training the image captioning model itself, thereby demanding substantially less computational effort. We experiment with two types of reformulation feedback: first, we collect a dataset of human reformulations that correct errors in the generated captions. We find that incorporating reformulation models trained on this data into the inference phase of existing image captioning models results in improved captions, especially when the original captions are of low quality. We apply our method to non-English image captioning, a domain where robust models are less prevalent, and gain substantial improvement. Second, we apply reformulations to style transfer. Quantitative evaluations reveal state-of-the-art performance on German image captioning and English style transfer, while human validation with a detailed comparative framework exposes the specific axes of improvement.

Title: Towards Fair Class-wise Robustness: Class Optimal Distribution Adversarial Training

• Authors: Hongxin Zhi, Hongtao Yu, Shaome Li, Xiuming Zhao, Yiteng Wu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04527
• Pdf URL: https://arxiv.org/pdf/2501.04527
• Copy Paste: [[2501.04527]] Towards Fair Class-wise Robustness: Class Optimal Distribution Adversarial Training(https://arxiv.org/abs/2501.04527)
• Keywords: attack, robust, fair
• Abstract: Adversarial training has proven to be a highly effective method for improving the robustness of deep neural networks against adversarial attacks. Nonetheless, it has been observed to exhibit a limitation in terms of robust fairness, characterized by a significant disparity in robustness across different classes. Recent efforts to mitigate this problem have turned to class-wise reweighted methods. However, these methods suffer from a lack of rigorous theoretical analysis and are limited in their exploration of the weight space, as they mainly rely on existing heuristic algorithms or intuition to compute weights. In addition, these methods fail to guarantee the consistency of the optimization direction due to the decoupled optimization of weights and the model parameters. They potentially lead to suboptimal weight assignments and consequently, a suboptimal model. To address these problems, this paper proposes a novel min-max training framework, Class Optimal Distribution Adversarial Training (CODAT), which employs distributionally robust optimization to fully explore the class-wise weight space, thus enabling the identification of the optimal weight with theoretical guarantees. Furthermore, we derive a closed-form optimal solution to the internal maximization and then get a deterministic equivalent objective function, which provides a theoretical basis for the joint optimization of weights and model parameters. Meanwhile, we propose a fairness elasticity coefficient for the evaluation of the algorithm with regard to both robustness and robust fairness. Experimental results on various datasets show that the proposed method can effectively improve the robust fairness of the model and outperform the state-of-the-art approaches.

Title: OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis

• Authors: Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, Yangyi Chen, Hamid Alinejad-Rokny, Fei Huang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04561
• Pdf URL: https://arxiv.org/pdf/2501.04561
• Copy Paste: [[2501.04561]] OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis(https://arxiv.org/abs/2501.04561)
• Keywords: large language model
• Abstract: Recent advancements in omnimodal learning have been achieved in understanding and generation across images, text, and speech, though mainly within proprietary models. Limited omnimodal datasets and the inherent challenges associated with real-time emotional speech generation have hindered open-source progress. To address these issues, we propose openomni, a two-stage training method combining omnimodal alignment and speech generation to develop a state-of-the-art omnimodal large language model. In the alignment phase, a pre-trained speech model is further trained on text-image tasks to generalize from vision to speech in a (near) zero-shot manner, outperforming models trained on tri-modal datasets. In the speech generation phase, a lightweight decoder facilitates real-time emotional speech through training on speech tasks and preference learning. Experiments demonstrate that openomni consistently improves across omnimodal, vision-language, and speech-language evaluations, enabling natural, emotion-rich dialogues and real-time emotional speech generation.

Title: Learnable Scaled Gradient Descent for Guaranteed Robust Tensor PCA

• Authors: Lanlan Feng, Ce Zhu, Yipeng Liu, Saiprasad Ravishankar, Longxiu Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04565
• Pdf URL: https://arxiv.org/pdf/2501.04565
• Copy Paste: [[2501.04565]] Learnable Scaled Gradient Descent for Guaranteed Robust Tensor PCA(https://arxiv.org/abs/2501.04565)
• Keywords: robust
• Abstract: Robust tensor principal component analysis (RTPCA) aims to separate the low-rank and sparse components from multi-dimensional data, making it an essential technique in the signal processing and computer vision fields. Recently emerging tensor singular value decomposition (t-SVD) has gained considerable attention for its ability to better capture the low-rank structure of tensors compared to traditional matrix SVD. However, existing methods often rely on the computationally expensive tensor nuclear norm (TNN), which limits their scalability for real-world tensors. To address this issue, we explore an efficient scaled gradient descent (SGD) approach within the t-SVD framework for the first time, and propose the RTPCA-SGD method. Theoretically, we rigorously establish the recovery guarantees of RTPCA-SGD under mild assumptions, demonstrating that with appropriate parameter selection, it achieves linear convergence to the true low-rank tensor at a constant rate, independent of the condition number. To enhance its practical applicability, we further propose a learnable self-supervised deep unfolding model, which enables effective parameter learning. Numerical experiments on both synthetic and real-world datasets demonstrate the superior performance of the proposed methods while maintaining competitive computational efficiency, especially consuming less time than RTPCA-TNN.

Title: Scalable Data Notarization Leveraging Hybrid DLTs

• Authors: Domenico Tortola, Claudio Felicioli, Andrea Canciani, Fabio Severino
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.04571
• Pdf URL: https://arxiv.org/pdf/2501.04571
• Copy Paste: [[2501.04571]] Scalable Data Notarization Leveraging Hybrid DLTs(https://arxiv.org/abs/2501.04571)
• Keywords: secure
• Abstract: Notarization is a procedure that enhance data management by ensuring the authentication of data during audits, thereby increasing trust in the audited data. Blockchain is frequently used as a secure, immutable, and transparent storage, contributing to make data notarization procedures more effective and trustable. Several blockchain-based data notarization protocols have been proposed in literature and commercial solutions. However, these implementations, whether on public or private blockchains, face inherent challenges: high fees on public blockchains and trust issues on private platforms, limiting the adoption of blockchains for data notarization or forcing several trade-offs. In this paper, we explore the use of hybrid blockchain architectures for data notarization, with a focus on scalability issues. Through the analysis of a real-world use case, the data notarization of product passports in supply chains, we propose a novel approach utilizing a data structure designed to efficiently manage the trade-offs in terms of storage occupation and costs involved in notarizing a large collection of data.

Title: Goldilocks Isolation: High Performance VMs with Edera

• Authors: Marina Moore, Alex Zenla
• Subjects: cs.CR, cs.OS
• Abstract URL: https://arxiv.org/abs/2501.04580
• Pdf URL: https://arxiv.org/pdf/2501.04580
• Copy Paste: [[2501.04580]] Goldilocks Isolation: High Performance VMs with Edera(https://arxiv.org/abs/2501.04580)
• Keywords: protect, attack
• Abstract: Organizations run applications on cloud infrastructure shared between multiple users and organizations. Popular tooling for this shared infrastructure, including Docker and Kubernetes, supports such multi-tenancy through the use of operating system virtualization. With operating system virtualization (known as containerization), multiple applications share the same kernel, reducing the runtime overhead. However, this shared kernel presents a large attack surface and has led to a proliferation of container escape attacks in which a kernel exploit lets an attacker escape the isolation of operating system virtualization to access other applications or the operating system itself. To address this, some systems have proposed a return to hypervisor virtualization for stronger isolation between applications. However, no existing system has achieved both the isolation of hypervisor virtualization and the performance and usability of operating system virtualization. We present Edera, an optimized type 1 hypervisor that uses paravirtualization to improve the runtime of hypervisor virtualization. We illustrate Edera's usability and performance through two use cases. First, we create a container runtime compatible with Kubernetes that runs on the Edera hypervisor. This implementation can be used as a drop-in replacement for the Kubernetes runtime and is compatible with all the tooling in the Kubernetes ecosystem. Second, we use Edera to provide driver isolation for hardware drivers, including those for networking, storage, and GPUs. This use of isolation protects the hypervisor and other applications from driver vulnerabilities. We find that Edera has runtime comparable to Docker with .9% slower cpu speeds, an average of 3% faster system call performance, and memory performance 0-7% faster. It achieves this with a 648 millisecond increase in startup time from Docker's 177.4 milliseconds.

Title: Identity-Preserving Video Dubbing Using Motion Warping

• Authors: Runzhen Liu, Qinjie Lin, Yunfei Liu, Lijian Lin, Ye Zhu, Yu Li, Chuhua Xian, Fa-Ting Hong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04586
• Pdf URL: https://arxiv.org/pdf/2501.04586
• Copy Paste: [[2501.04586]] Identity-Preserving Video Dubbing Using Motion Warping(https://arxiv.org/abs/2501.04586)
• Keywords: robust, transformer
• Abstract: Video dubbing aims to synthesize realistic, lip-synced videos from a reference video and a driving audio signal. Although existing methods can accurately generate mouth shapes driven by audio, they often fail to preserve identity-specific features, largely because they do not effectively capture the nuanced interplay between audio cues and the visual attributes of reference identity . As a result, the generated outputs frequently lack fidelity in reproducing the unique textural and structural details of the reference identity. To address these limitations, we propose IPTalker, a novel and robust framework for video dubbing that achieves seamless alignment between driving audio and reference identity while ensuring both lip-sync accuracy and high-fidelity identity preservation. At the core of IPTalker is a transformer-based alignment mechanism designed to dynamically capture and model the correspondence between audio features and reference images, thereby enabling precise, identity-aware audio-visual integration. Building on this alignment, a motion warping strategy further refines the results by spatially deforming reference images to match the target audio-driven configuration. A dedicated refinement process then mitigates occlusion artifacts and enhances the preservation of fine-grained textures, such as mouth details and skin features. Extensive qualitative and quantitative evaluations demonstrate that IPTalker consistently outperforms existing approaches in terms of realism, lip synchronization, and identity retention, establishing a new state of the art for high-quality, identity-consistent video dubbing.

Title: Federated-Continual Dynamic Segmentation of Histopathology guided by Barlow Continuity

• Authors: Niklas Babendererde, Haozhe Zhu, Moritz Fuchs, Jonathan Stieber, Anirban Mukhopadhyay
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04588
• Pdf URL: https://arxiv.org/pdf/2501.04588
• Copy Paste: [[2501.04588]] Federated-Continual Dynamic Segmentation of Histopathology guided by Barlow Continuity(https://arxiv.org/abs/2501.04588)
• Keywords: privacy, federate, segmentation
• Abstract: Federated- and Continual Learning have been established as approaches to enable privacy-aware learning on continuously changing data, as required for deploying AI systems in histopathology images. However, data shifts can occur in a dynamic world, spatially between institutions and temporally, due to changing data over time. This leads to two issues: Client Drift, where the central model degrades from aggregating data from clients trained on shifted data, and Catastrophic Forgetting, from temporal shifts such as changes in patient populations. Both tend to degrade the model's performance of previously seen data or spatially distributed training. Despite both problems arising from the same underlying problem of data shifts, existing research addresses them only individually. In this work, we introduce a method that can jointly alleviate Client Drift and Catastrophic Forgetting by using our proposed Dynamic Barlow Continuity that evaluates client updates on a public reference dataset and uses this to guide the training process to a spatially and temporally shift-invariant model. We evaluate our approach on the histopathology datasets BCSS and Semicol and prove our method to be highly effective by jointly improving the dice score as much as from 15.8% to 71.6% in Client Drift and from 42.5% to 62.8% in Catastrophic Forgetting. This enables Dynamic Learning by establishing spatio-temporal shift-invariance.

Title: Quantum-inspired Embeddings Projection and Similarity Metrics for Representation Learning

• Authors: Ivan Kankeu, Stefan Gerd Fritsch, Gunnar Schönhoff, Elie Mounzer, Paul Lukowicz, Maximilian Kiefer-Emmanouilidis
• Subjects: cs.CL, cond-mat.dis-nn, quant-ph
• Abstract URL: https://arxiv.org/abs/2501.04591
• Pdf URL: https://arxiv.org/pdf/2501.04591
• Copy Paste: [[2501.04591]] Quantum-inspired Embeddings Projection and Similarity Metrics for Representation Learning(https://arxiv.org/abs/2501.04591)
• Keywords: large language model
• Abstract: Over the last decade, representation learning, which embeds complex information extracted from large amounts of data into dense vector spaces, has emerged as a key technique in machine learning. Among other applications, it has been a key building block for large language models and advanced computer vision systems based on contrastive learning. A core component of representation learning systems is the projection head, which maps the original embeddings into different, often compressed spaces, while preserving the similarity relationship between vectors. In this paper, we propose a quantum-inspired projection head that includes a corresponding quantum-inspired similarity metric. Specifically, we map classical embeddings onto quantum states in Hilbert space and introduce a quantum circuit-based projection head to reduce embedding dimensionality. To evaluate the effectiveness of this approach, we extended the BERT language model by integrating our projection head for embedding compression. We compared the performance of embeddings, which were compressed using our quantum-inspired projection head, with those compressed using a classical projection head on information retrieval tasks using the TREC 2019 and TREC 2020 Deep Learning benchmarks. The results demonstrate that our quantum-inspired method achieves competitive performance relative to the classical method while utilizing 32 times fewer parameters. Furthermore, when trained from scratch, it notably excels, particularly on smaller datasets. This work not only highlights the effectiveness of the quantum-inspired approach but also emphasizes the utility of efficient, ad hoc low-entanglement circuit simulations within neural networks as a powerful quantum-inspired technique.

Title: Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversion

• Authors: Yangfan He, Sida Li, Kun Li, Jianhui Wang, Binxu Li, Tianyu Shi, Jun Yin, Miao Zhang, Xueqian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04606
• Pdf URL: https://arxiv.org/pdf/2501.04606
• Copy Paste: [[2501.04606]] Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversion(https://arxiv.org/abs/2501.04606)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-image (T2I) generation using diffusion models have enabled cost-effective video-editing applications by leveraging pre-trained models, eliminating the need for resource-intensive training. However, the frame-independence of T2I generation often results in poor temporal consistency. Existing methods address this issue through temporal layer fine-tuning or inference-based temporal propagation, but these approaches suffer from high training costs or limited temporal coherence. To address these challenges, we propose a General and Efficient Adapter (GE-Adapter) that integrates temporal-spatial and semantic consistency with Baliteral DDIM inversion. This framework introduces three key components: (1) Frame-based Temporal Consistency Blocks (FTC Blocks) to capture frame-specific features and enforce smooth inter-frame transitions via temporally-aware loss functions; (2) Channel-dependent Spatial Consistency Blocks (SCD Blocks) employing bilateral filters to enhance spatial coherence by reducing noise and artifacts; and (3) Token-based Semantic Consistency Module (TSC Module) to maintain semantic alignment using shared prompt tokens and frame-specific tokens. Our method significantly improves perceptual quality, text-image alignment, and temporal coherence, as demonstrated on the MSR-VTT dataset. Additionally, it achieves enhanced fidelity and frame-to-frame coherence, offering a practical solution for T2V editing.

Title: Resilient Peer-to-peer Learning based on Adaptive Aggregation

• Authors: Chandreyee Bhowmick, Xenofon Koutsoukos
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04610
• Pdf URL: https://arxiv.org/pdf/2501.04610
• Copy Paste: [[2501.04610]] Resilient Peer-to-peer Learning based on Adaptive Aggregation(https://arxiv.org/abs/2501.04610)
• Keywords: privacy, attack
• Abstract: Collaborative learning in peer-to-peer networks offers the benefits of distributed learning while mitigating the risks associated with single points of failure inherent in centralized servers. However, adversarial workers pose potential threats by attempting to inject malicious information into the network. Thus, ensuring the resilience of peer-to-peer learning emerges as a pivotal research objective. The challenge is exacerbated in the presence of non-convex loss functions and non-iid data distributions. This paper introduces a resilient aggregation technique tailored for such scenarios, aimed at fostering similarity among peers' learning processes. The aggregation weights are determined through an optimization procedure, and use the loss function computed using the neighbor's models and individual private data, thereby addressing concerns regarding data privacy in distributed machine learning. Theoretical analysis demonstrates convergence of parameters with non-convex loss functions and non-iid data distributions. Empirical evaluations across three distinct machine learning tasks support the claims. The empirical findings, which encompass a range of diverse attack models, also demonstrate improved accuracy when compared to existing methodologies.

Title: Disentangled Clothed Avatar Generation with Layered Representation

• Authors: Weitian Zhang, Sijing Wu, Manwen Liao, Yichao Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04631
• Pdf URL: https://arxiv.org/pdf/2501.04631
• Copy Paste: [[2501.04631]] Disentangled Clothed Avatar Generation with Layered Representation(https://arxiv.org/abs/2501.04631)
• Keywords: diffusion
• Abstract: Clothed avatar generation has wide applications in virtual and augmented reality, filmmaking, and more. Previous methods have achieved success in generating diverse digital avatars, however, generating avatars with disentangled components (\eg, body, hair, and clothes) has long been a challenge. In this paper, we propose LayerAvatar, the first feed-forward diffusion-based method for generating component-disentangled clothed avatars. To achieve this, we first propose a layered UV feature plane representation, where components are distributed in different layers of the Gaussian-based UV feature plane with corresponding semantic labels. This representation supports high-resolution and real-time rendering, as well as expressive animation including controllable gestures and facial expressions. Based on the well-designed representation, we train a single-stage diffusion model and introduce constrain terms to address the severe occlusion problem of the innermost human body layer. Extensive experiments demonstrate the impressive performances of our method in generating disentangled clothed avatars, and we further explore its applications in component transfer. The project page is available at: this https URL

Title: A Statistical Theory of Contrastive Pre-training and Multimodal Generative AI

• Authors: Kazusato Oko, Licong Lin, Yuhang Cai, Song Mei
• Subjects: cs.LG, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.04641
• Pdf URL: https://arxiv.org/pdf/2501.04641
• Copy Paste: [[2501.04641]] A Statistical Theory of Contrastive Pre-training and Multimodal Generative AI(https://arxiv.org/abs/2501.04641)
• Keywords: diffusion, transformer, generative
• Abstract: Multi-modal generative AI systems, such as those combining vision and language, rely on contrastive pre-training to learn representations across different modalities. While their practical benefits are widely acknowledged, a rigorous theoretical understanding of the contrastive pre-training framework remains limited. This paper develops a theoretical framework to explain the success of contrastive pre-training in downstream tasks, such as zero-shot classification, conditional diffusion models, and vision-language models. We introduce the concept of approximate sufficient statistics, a generalization of the classical sufficient statistics, and show that near-minimizers of the contrastive pre-training loss are approximately sufficient, making them adaptable to diverse downstream tasks. We further propose the Joint Generative Hierarchical Model for the joint distribution of images and text, showing that transformers can efficiently approximate relevant functions within this model via belief propagation. Building on this framework, we derive sample complexity guarantees for multi-modal learning based on contrastive pre-trained representations. Numerical simulations validate these theoretical findings, demonstrating the strong generalization performance of contrastively pre-trained transformers in various multi-modal tasks.

Title: Multi-task retriever fine-tuning for domain-specific and efficient RAG

• Authors: Patrice Béchard, Orlando Marquez Ayala
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04652
• Pdf URL: https://arxiv.org/pdf/2501.04652
• Copy Paste: [[2501.04652]] Multi-task retriever fine-tuning for domain-specific and efficient RAG(https://arxiv.org/abs/2501.04652)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) has become ubiquitous when deploying Large Language Models (LLMs), as it can address typical limitations such as generating hallucinated or outdated information. However, when building real-world RAG applications, practical issues arise. First, the retrieved information is generally domain-specific. Since it is computationally expensive to fine-tune LLMs, it is more feasible to fine-tune the retriever to improve the quality of the data included in the LLM input. Second, as more applications are deployed in the same real-world system, one cannot afford to deploy separate retrievers. Moreover, these RAG applications normally retrieve different kinds of data. Our solution is to instruction fine-tune a small retriever encoder on a variety of domain-specific tasks to allow us to deploy one encoder that can serve many use cases, thereby achieving low-cost, scalability, and speed. We show how this encoder generalizes to out-of-domain settings as well as to an unseen retrieval task on real-world enterprise use cases.

Title: Assessing Language Comprehension in Large Language Models Using Construction Grammar

• Authors: Wesley Scivetti, Melissa Torgbi, Austin Blodgett, Mollie Shichman, Taylor Hudson, Claire Bonial, Harish Tayyar Madabushi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04661
• Pdf URL: https://arxiv.org/pdf/2501.04661
• Copy Paste: [[2501.04661]] Assessing Language Comprehension in Large Language Models Using Construction Grammar(https://arxiv.org/abs/2501.04661)
• Keywords: large language model
• Abstract: Large Language Models, despite their significant capabilities, are known to fail in surprising and unpredictable ways. Evaluating their true `understanding' of language is particularly challenging due to the extensive web-scale data they are trained on. Therefore, we construct an evaluation to systematically assess natural language understanding (NLU) in LLMs by leveraging Construction Grammar (CxG), which provides insights into the meaning captured by linguistic elements known as constructions (Cxns). CxG is well-suited for this purpose because provides a theoretical basis to construct targeted evaluation sets. These datasets are carefully constructed to include examples which are unlikely to appear in pre-training data, yet intuitive and easy for humans to understand, enabling a more targeted and reliable assessment. Our experiments focus on downstream natural language inference and reasoning tasks by comparing LLMs' understanding of the underlying meanings communicated through 8 unique Cxns with that of humans. The results show that while LLMs demonstrate some knowledge of constructional information, even the latest models including GPT-o1 struggle with abstract meanings conveyed by these Cxns, as demonstrated in cases where test sentences are dissimilar to their pre-training data. We argue that such cases provide a more accurate test of true language understanding, highlighting key limitations in LLMs' semantic capabilities. We make our novel dataset and associated experimental data including prompts and model responses publicly available.

Title: Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling

• Authors: Nannan Li, Kevin J. Shih, Bryan A. Plummer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04666
• Pdf URL: https://arxiv.org/pdf/2501.04666
• Copy Paste: [[2501.04666]] Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling(https://arxiv.org/abs/2501.04666)
• Keywords: extraction
• Abstract: Given an isolated garment image in a canonical product view and a separate image of a person, the virtual try-on task aims to generate a new image of the person wearing the target garment. Prior virtual try-on works face two major challenges in achieving this goal: a) the paired (human, garment) training data has limited availability; b) generating textures on the human that perfectly match that of the prompted garment is difficult, often resulting in distorted text and faded textures. Our work explores ways to tackle these issues through both synthetic data as well as model refinement. We introduce a garment extraction model that generates (human, synthetic garment) pairs from a single image of a clothed individual. The synthetic pairs can then be used to augment the training of virtual try-on. We also propose an Error-Aware Refinement-based Schrödinger Bridge (EARSB) that surgically targets localized generation errors for correcting the output of a base virtual try-on model. To identify likely errors, we propose a weakly-supervised error classifier that localizes regions for refinement, subsequently augmenting the Schrödinger Bridge's noise schedule with its confidence heatmap. Experiments on VITON-HD and DressCode-Upper demonstrate that our synthetic data augmentation enhances the performance of prior work, while EARSB improves the overall image quality. In user studies, our model is preferred by the users in an average of 59% of cases.

Title: Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs

• Authors: Yikang Zhou, Tao Zhang, Shilin Xu, Shihao Chen, Qianyu Zhou, Yunhai Tong, Shunping Ji, Jiangning Zhang, Xiangtai Li, Lu Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04670
• Pdf URL: https://arxiv.org/pdf/2501.04670
• Copy Paste: [[2501.04670]] Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs(https://arxiv.org/abs/2501.04670)
• Keywords: fair
• Abstract: Recent advancements in multimodal models have shown a strong ability in visual perception, reasoning abilities, and vision-language understanding. However, studies on visual matching ability are missing, where finding the visual correspondence of objects is essential in vision research. Our research reveals that the matching capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings, even with current strong MLLMs models, GPT-4o. In particular, we construct a Multimodal Visual Matching (MMVM) benchmark to fairly benchmark over 30 different MLLMs. The MMVM benchmark is built from 15 open-source datasets and Internet videos with manual annotation. We categorize the data samples of MMVM benchmark into eight aspects based on the required cues and capabilities to more comprehensively evaluate and analyze current MLLMs. In addition, we have designed an automatic annotation pipeline to generate the MMVM SFT dataset, including 220K visual matching data with reasoning annotation. Finally, we present CoLVA, a novel contrastive MLLM with two novel technical designs: fine-grained vision expert with object-level contrastive learning and instruction augmentation strategy. CoLVA achieves 51.06\% overall accuracy (OA) on the MMVM benchmark, surpassing GPT-4o and baseline by 8.41\% and 23.58\% OA, respectively. The results show the effectiveness of our MMVM SFT dataset and our novel technical designs. Code, benchmark, dataset, and models are available at this https URL.

Title: Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations

• Authors: Archita Srivastava, Abhas Kumar, Rajesh Kumar, Prabhakar Srinivasan
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04675
• Pdf URL: https://arxiv.org/pdf/2501.04675
• Copy Paste: [[2501.04675]] Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations(https://arxiv.org/abs/2501.04675)
• Keywords: large language model
• Abstract: Chart interpretation is crucial for visual data analysis, but accurately extracting information from charts poses significant challenges for automated models. This study investigates the fine-tuning of DEPLOT, a modality conversion module that translates the image of a plot or chart to a linearized table, on a custom dataset of 50,000 bar charts. The dataset comprises simple, stacked, and grouped bar charts, targeting the unique structural features of these visualizations. The finetuned DEPLOT model is evaluated against its base version using a test set of 1,000 images and two metrics: Relative Mapping Similarity (RMS), which measures categorical mapping accuracy, and Relative Number Set Similarity (RNSS), which evaluates numerical interpretation accuracy. To further explore the reasoning capabilities of large language models (LLMs), we curate an additional set of 100 bar chart images paired with question answer sets. Our findings demonstrate that providing a structured intermediate table alongside the image significantly enhances LLM reasoning performance compared to direct image queries.

Title: URSA: Understanding and Verifying Chain-of-thought Reasoning in Multimodal Mathematics

• Authors: Ruilin Luo, Zhuofan Zheng, Yifan Wang, Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, Yujiu Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04686
• Pdf URL: https://arxiv.org/pdf/2501.04686
• Copy Paste: [[2501.04686]] URSA: Understanding and Verifying Chain-of-thought Reasoning in Multimodal Mathematics(https://arxiv.org/abs/2501.04686)
• Keywords: robust, large language model
• Abstract: Chain-of-thought (CoT) reasoning has been widely applied in the mathematical reasoning of Large Language Models (LLMs). Recently, the introduction of derivative process supervision on CoT trajectories has sparked discussions on enhancing scaling capabilities during test time, thereby boosting the potential of these models. However, in multimodal mathematical reasoning, the scarcity of high-quality CoT training data has hindered existing models from achieving high-precision CoT reasoning and has limited the realization of reasoning potential during test time. In this work, we propose a three-module synthesis strategy that integrates CoT distillation, trajectory-format rewriting, and format unification. It results in a high-quality CoT reasoning instruction fine-tuning dataset in multimodal mathematics, MMathCoT-1M. We comprehensively validate the state-of-the-art (SOTA) performance of the trained URSA-7B model on multiple multimodal mathematical benchmarks. For test-time scaling, we introduce a data synthesis strategy that automatically generates process annotation datasets, known as DualMath-1.1M, focusing on both interpretation and logic. By further training URSA-7B on DualMath-1.1M, we transition from CoT reasoning capabilities to robust supervision abilities. The trained URSA-RM-7B acts as a verifier, effectively enhancing the performance of URSA-7B at test time. URSA-RM-7B also demonstrates excellent out-of-distribution (OOD) verifying capabilities, showcasing its generalization. Model weights, training data and code will be open-sourced.

Title: SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images

• Authors: Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M. Rehg, Varun Jampani
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.04689
• Pdf URL: https://arxiv.org/pdf/2501.04689
• Copy Paste: [[2501.04689]] SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images(https://arxiv.org/abs/2501.04689)
• Keywords: diffusion, generative
• Abstract: We study the problem of single-image 3D object reconstruction. Recent works have diverged into two directions: regression-based modeling and generative modeling. Regression methods efficiently infer visible surfaces, but struggle with occluded regions. Generative methods handle uncertain regions better by modeling distributions, but are computationally expensive and the generation is often misaligned with visible surfaces. In this paper, we present SPAR3D, a novel two-stage approach aiming to take the best of both directions. The first stage of SPAR3D generates sparse 3D point clouds using a lightweight point diffusion model, which has a fast sampling speed. The second stage uses both the sampled point cloud and the input image to create highly detailed meshes. Our two-stage design enables probabilistic modeling of the ill-posed single-image 3D task while maintaining high computational efficiency and great output fidelity. Using point clouds as an intermediate representation further allows for interactive user edits. Evaluated on diverse datasets, SPAR3D demonstrates superior performance over previous state-of-the-art methods, at an inference speed of 0.7 seconds. Project page with code and model: this https URL

Title: Re-ranking the Context for Multimodal Retrieval Augmented Generation

• Authors: Matin Mortaheb, Mohammad A. Amir Khojastepour, Srimat T. Chakradhar, Sennur Ulukus
• Subjects: cs.LG, cs.CV, cs.IR, cs.IT
• Abstract URL: https://arxiv.org/abs/2501.04695
• Pdf URL: https://arxiv.org/pdf/2501.04695
• Copy Paste: [[2501.04695]] Re-ranking the Context for Multimodal Retrieval Augmented Generation(https://arxiv.org/abs/2501.04695)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating external knowledge to generate a response within a context with improved accuracy and reduced hallucinations. However, multi-modal RAG systems face unique challenges: (i) the retrieval process may select irrelevant entries to user query (e.g., images, documents), and (ii) vision-language models or multi-modal language models like GPT-4o may hallucinate when processing these entries to generate RAG output. In this paper, we aim to address the first challenge, i.e, improving the selection of relevant context from the knowledge-base in retrieval phase of the multi-modal RAG. Specifically, we leverage the relevancy score (RS) measure designed in our previous work for evaluating the RAG performance to select more relevant entries in retrieval process. The retrieval based on embeddings, say CLIP-based embedding, and cosine similarity usually perform poorly particularly for multi-modal data. We show that by using a more advanced relevancy measure, one can enhance the retrieval process by selecting more relevant pieces from the knowledge-base and eliminate the irrelevant pieces from the context by adaptively selecting up-to-$k$ entries instead of fixed number of entries. Our evaluation using COCO dataset demonstrates significant enhancement in selecting relevant context and accuracy of the generated response.

Title: Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation

• Authors: Ulindu De Silva, Didula Samaraweera, Sasini Wanigathunga, Kavindu Kariyawasam, Kanchana Ranasinghe, Muzammal Naseer, Ranga Rodrigo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04696
• Pdf URL: https://arxiv.org/pdf/2501.04696
• Copy Paste: [[2501.04696]] Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation(https://arxiv.org/abs/2501.04696)
• Keywords: segmentation
• Abstract: We present Seg-TTO, a novel framework for zero-shot, open-vocabulary semantic segmentation (OVSS), designed to excel in specialized domain tasks. While current open vocabulary approaches show impressive performance on standard segmentation benchmarks under zero-shot settings, they fall short of supervised counterparts on highly domain-specific datasets. We focus on segmentation-specific test-time optimization to address this gap. Segmentation requires an understanding of multiple concepts within a single image while retaining the locality and spatial structure of representations. We propose a novel self-supervised objective adhering to these requirements and use it to align the model parameters with input images at test time. In the textual modality, we learn multiple embeddings for each category to capture diverse concepts within an image, while in the visual modality, we calculate pixel-level losses followed by embedding aggregation operations specific to preserving spatial structure. Our resulting framework termed Seg-TTO is a plug-in-play module. We integrate Seg-TTO with three state-of-the-art OVSS approaches and evaluate across 22 challenging OVSS tasks covering a range of specialized domains. Our Seg-TTO demonstrates clear performance improvements across these establishing new state-of-the-art. Code: this https URL.

Title: ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning

• Authors: Yuzhou Huang, Ziyang Yuan, Quande Liu, Qiulin Wang, Xintao Wang, Ruimao Zhang, Pengfei Wan, Di Zhang, Kun Gai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04698
• Pdf URL: https://arxiv.org/pdf/2501.04698
• Copy Paste: [[2501.04698]] ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning(https://arxiv.org/abs/2501.04698)
• Keywords: diffusion, transformer
• Abstract: Text-to-video generation has made remarkable advancements through diffusion models. However, Multi-Concept Video Customization (MCVC) remains a significant challenge. We identify two key challenges in this task: 1) the identity decoupling problem, where directly adopting existing customization methods inevitably mix attributes when handling multiple concepts simultaneously, and 2) the scarcity of high-quality video-entity pairs, which is crucial for training such a model that represents and decouples various concepts well. To address these challenges, we introduce ConceptMaster, an innovative framework that effectively tackles the critical issues of identity decoupling while maintaining concept fidelity in customized videos. Specifically, we introduce a novel strategy of learning decoupled multi-concept embeddings that are injected into the diffusion models in a standalone manner, which effectively guarantees the quality of customized videos with multiple identities, even for highly similar visual concepts. To further overcome the scarcity of high-quality MCVC data, we carefully establish a data construction pipeline, which enables systematic collection of precise multi-concept video-entity data across diverse concepts. A comprehensive benchmark is designed to validate the effectiveness of our model from three critical dimensions: concept fidelity, identity decoupling ability, and video generation quality across six different concept composition scenarios. Extensive experiments demonstrate that our ConceptMaster significantly outperforms previous approaches for this task, paving the way for generating personalized and semantically accurate videos across multiple concepts.

Title: EditAR: Unified Conditional Generation with Autoregressive Models

• Authors: Jiteng Mu, Nuno Vasconcelos, Xiaolong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04699
• Pdf URL: https://arxiv.org/pdf/2501.04699
• Copy Paste: [[2501.04699]] EditAR: Unified Conditional Generation with Autoregressive Models(https://arxiv.org/abs/2501.04699)
• Keywords: diffusion, segmentation
• Abstract: Recent progress in controllable image generation and editing is largely driven by diffusion-based methods. Although diffusion models perform exceptionally well in specific tasks with tailored designs, establishing a unified model is still challenging. In contrast, autoregressive models inherently feature a unified tokenized representation, which simplifies the creation of a single foundational model for various tasks. In this work, we propose EditAR, a single unified autoregressive framework for a variety of conditional image generation tasks, e.g., image editing, depth-to-image, edge-to-image, segmentation-to-image. The model takes both images and instructions as inputs, and predicts the edited images tokens in a vanilla next-token paradigm. To enhance the text-to-image alignment, we further propose to distill the knowledge from foundation models into the autoregressive modeling process. We evaluate its effectiveness across diverse tasks on established benchmarks, showing competitive performance to various state-of-the-art task-specific methods. Project page: this https URL