2025-01-27

Title: Fanar: An Arabic-Centric Multimodal Generative AI Platform

• Authors: Fanar Team: Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus'ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, Chaoyi Ruan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13944
• Pdf URL: https://arxiv.org/pdf/2501.13944
• Copy Paste: [[2501.13944]] Fanar: An Arabic-Centric Multimodal Generative AI Platform(https://arxiv.org/abs/2501.13944)
• Keywords: generative
• Abstract: We present Fanar, a platform for Arabic-centric multimodal generative AI systems, that supports language, speech and image generation tasks. At the heart of Fanar are Fanar Star and Fanar Prime, two highly capable Arabic Large Language Models (LLMs) that are best in the class on well established benchmarks for similar sized models. Fanar Star is a 7B (billion) parameter model that was trained from scratch on nearly 1 trillion clean and deduplicated Arabic, English and Code tokens. Fanar Prime is a 9B parameter model continually trained on the Gemma-2 9B base model on the same 1 trillion token set. Both models are concurrently deployed and designed to address different types of prompts transparently routed through a custom-built orchestrator. The Fanar platform provides many other capabilities including a customized Islamic Retrieval Augmented Generation (RAG) system for handling religious prompts, a Recency RAG for summarizing information about current or recent events that have occurred after the pre-training data cut-off date. The platform provides additional cognitive capabilities including in-house bilingual speech recognition that supports multiple Arabic dialects, voice and image generation that is fine-tuned to better reflect regional characteristics. Finally, Fanar provides an attribution service that can be used to verify the authenticity of fact based generated content. The design, development, and implementation of Fanar was entirely undertaken at Hamad Bin Khalifa University's Qatar Computing Research Institute (QCRI) and was sponsored by Qatar's Ministry of Communications and Information Technology to enable sovereign AI technology development.

Title: Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks

• Authors: Diego Gosmar, Deborah A. Dahl
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.13946
• Pdf URL: https://arxiv.org/pdf/2501.13946
• Copy Paste: [[2501.13946]] Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks(https://arxiv.org/abs/2501.13946)
• Keywords: generative
• Abstract: Hallucinations remain a significant challenge in current Generative AI models, undermining trust in AI systems and their reliability. This study investigates how orchestrating multiple specialized Artificial Intelligent Agents can help mitigate such hallucinations, with a focus on systems leveraging Natural Language Processing (NLP) to facilitate seamless agent interactions. To achieve this, we design a pipeline that introduces over three hundred prompts, purposefully crafted to induce hallucinations, into a front-end agent. The outputs are then systematically reviewed and refined by second- and third-level agents, each employing distinct large language models and tailored strategies to detect unverified claims, incorporate explicit disclaimers, and clarify speculative content. Additionally, we introduce a set of novel Key Performance Indicators (KPIs) specifically designed to evaluate hallucination score levels. A dedicated fourth-level AI agent is employed to evaluate these KPIs, providing detailed assessments and ensuring accurate quantification of shifts in hallucination-related behaviors. A core component of this investigation is the use of the OVON (Open Voice Network) framework, which relies on universal NLP-based interfaces to transfer contextual information among agents. Through structured JSON messages, each agent communicates its assessment of the hallucination likelihood and the reasons underlying questionable content, thereby enabling the subsequent stage to refine the text without losing context. The results demonstrate that employing multiple specialized agents capable of interoperating with each other through NLP-based agentic frameworks can yield promising outcomes in hallucination mitigation, ultimately bolstering trust within the AI community.

Title: A Comprehensive Survey on Integrating Large Language Models with Knowledge-Based Methods

• Authors: Lilian Some, Wenli Yang, Michael Bain, Byeong Kang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13947
• Pdf URL: https://arxiv.org/pdf/2501.13947
• Copy Paste: [[2501.13947]] A Comprehensive Survey on Integrating Large Language Models with Knowledge-Based Methods(https://arxiv.org/abs/2501.13947)
• Keywords: generative
• Abstract: The rapid development of artificial intelligence has brought about substantial advancements in the field. One promising direction is the integration of Large Language Models (LLMs) with structured knowledge-based systems. This approach aims to enhance AI capabilities by combining the generative language understanding of LLMs with the precise knowledge representation of structured systems. This survey explores the synergy between LLMs and knowledge bases, focusing on real-world applications and addressing associated technical, operational, and ethical challenges. Through a comprehensive literature review, the study identifies critical issues and evaluates existing solutions. The paper highlights the benefits of integrating generative AI with knowledge bases, including improved data contextualization, enhanced model accuracy, and better utilization of knowledge resources. The findings provide a detailed overview of the current state of research, identify key gaps, and offer actionable recommendations. These insights contribute to advancing AI technologies and support their practical deployment across various sectors.

Title: DEFEND: A Large-scale 1M Dataset and Foundation Model for Tobacco Addiction Prevention

• Authors: Naga VS Raviteja Chappa, Matthew Shepard, Connor McCurtain, Charlotte McCormick, Page Daniel Dobbs, Khoa Luu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13950
• Pdf URL: https://arxiv.org/pdf/2501.13950
• Copy Paste: [[2501.13950]] DEFEND: A Large-scale 1M Dataset and Foundation Model for Tobacco Addiction Prevention(https://arxiv.org/abs/2501.13950)
• Keywords: foundation model
• Abstract: While tobacco advertising innovates at unprecedented speed, traditional surveillance methods remain frozen in time, especially in the context of social media. The lack of large-scale, comprehensive datasets and sophisticated monitoring systems has created a widening gap between industry advancement and public health oversight. This paper addresses this critical challenge by introducing Tobacco-1M, a comprehensive dataset of one million tobacco product images with hierarchical labels spanning 75 product categories, and DEFEND, a novel foundation model for tobacco product understanding. Our approach integrates a Feature Enhancement Module for rich multimodal representation learning, a Local-Global Visual Coherence mechanism for detailed feature discrimination, and an Enhanced Image-Text Alignment strategy for precise product characterization. Experimental results demonstrate DEFEND's superior performance, achieving 83.1% accuracy in product classification and 73.8% in visual question-answering tasks, outperforming existing methods by significant margins. Moreover, the model exhibits robust zero-shot learning capabilities with 45.6% accuracy on novel product categories. This work provides regulatory bodies and public health researchers with powerful tools for monitoring emerging tobacco products and marketing strategies, potentially revolutionizing approaches to tobacco control and public health surveillance.

Title: Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)

• Authors: Jadon Geathers, Yann Hicke, Colleen Chan, Niroop Rajashekar, Justin Sewell, Susannah Cornes, Rene Kizilcec, Dennis Shung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13957
• Pdf URL: https://arxiv.org/pdf/2501.13957
• Copy Paste: [[2501.13957]] Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)(https://arxiv.org/abs/2501.13957)
• Keywords: generative
• Abstract: Introduction. Objective Structured Clinical Examinations (OSCEs) are widely used to assess medical students' communication skills, but scoring interview-based assessments is time-consuming and potentially subject to human bias. This study explored the potential of large language models (LLMs) to automate OSCE evaluations using the Master Interview Rating Scale (MIRS). Methods. We compared the performance of four state-of-the-art LLMs (GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro) in evaluating OSCE transcripts across all 28 items of the MIRS under the conditions of zero-shot, chain-of-thought (CoT), few-shot, and multi-step prompting. The models were benchmarked against a dataset of 10 OSCE cases with 174 expert consensus scores available. Model performance was measured using three accuracy metrics (exact, off-by-one, thresholded). Results. Averaging across all MIRS items and OSCE cases, LLMs performed with low exact accuracy (0.27 to 0.44), and moderate to high off-by-one accuracy (0.67 to 0.87) and thresholded accuracy (0.75 to 0.88). A zero temperature parameter ensured high intra-rater reliability ($\alpha = 0.98$ for GPT-4o). CoT, few-shot, and multi-step techniques proved valuable when tailored to specific assessment items. The performance was consistent across MIRS items independent of encounter phases and communication domains. Conclusion. We demonstrated the feasibility of AI-assisted OSCE evaluation and provided benchmarking of multiple LLMs across multiple prompt techniques. Our work provides a baseline performance assessment for LLMs that lays a foundation for future research in automated assessment of clinical communication skills.

Title: InsTex: Indoor Scenes Stylized Texture Synthesis

• Authors: Yunfan Zhang, Zhiwei Xiong, Zhiqi Shen, Guosheng Lin, Hao Wang, Nicolas Vun
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13969
• Pdf URL: https://arxiv.org/pdf/2501.13969
• Copy Paste: [[2501.13969]] InsTex: Indoor Scenes Stylized Texture Synthesis(https://arxiv.org/abs/2501.13969)
• Keywords: diffusion, generative
• Abstract: Generating high-quality textures for 3D scenes is crucial for applications in interior design, gaming, and augmented/virtual reality (AR/VR). Although recent advancements in 3D generative models have enhanced content creation, significant challenges remain in achieving broad generalization and maintaining style consistency across multiple viewpoints. Current methods, such as 2D diffusion models adapted for 3D texturing, suffer from lengthy processing times and visual artifacts, while approaches driven by 3D data often fail to generalize effectively. To overcome these challenges, we introduce InsTex, a two-stage architecture designed to generate high-quality, style-consistent textures for 3D indoor scenes. InsTex utilizes depth-to-image diffusion priors in a coarse-to-fine pipeline, first generating multi-view images with a pre-trained 2D diffusion model and subsequently refining the textures for consistency. Our method supports both textual and visual prompts, achieving state-of-the-art results in visual quality and quantitative metrics, and demonstrates its effectiveness across various 3D texturing applications.

Title: Towards Safer Social Media Platforms: Scalable and Performant Few-Shot Harmful Content Moderation Using Large Language Models

• Authors: Akash Bonagiri, Lucen Li, Rajvardhan Oak, Zeerak Babar, Magdalena Wojcieszak, Anshuman Chhabra
• Subjects: cs.CL, cs.AI, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.13976
• Pdf URL: https://arxiv.org/pdf/2501.13976
• Copy Paste: [[2501.13976]] Towards Safer Social Media Platforms: Scalable and Performant Few-Shot Harmful Content Moderation Using Large Language Models(https://arxiv.org/abs/2501.13976)
• Keywords: in-context
• Abstract: The prevalence of harmful content on social media platforms poses significant risks to users and society, necessitating more effective and scalable content moderation strategies. Current approaches rely on human moderators, supervised classifiers, and large volumes of training data, and often struggle with scalability, subjectivity, and the dynamic nature of harmful content (e.g., violent content, dangerous challenge trends, etc.). To bridge these gaps, we utilize Large Language Models (LLMs) to undertake few-shot dynamic content moderation via in-context learning. Through extensive experiments on multiple LLMs, we demonstrate that our few-shot approaches can outperform existing proprietary baselines (Perspective and OpenAI Moderation) as well as prior state-of-the-art few-shot learning methods, in identifying harm. We also incorporate visual information (video thumbnails) and assess if different multimodal techniques improve model performance. Our results underscore the significant benefits of employing LLM based methods for scalable and dynamic harmful content moderation online.

Title: An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks

• Authors: Vivek Bharadwaj, Austin Scott Glover, Aydin Buluc, James Demmel
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13986
• Pdf URL: https://arxiv.org/pdf/2501.13986
• Copy Paste: [[2501.13986]] An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks(https://arxiv.org/abs/2501.13986)
• Keywords: foundation model
• Abstract: Rotation equivariant graph neural networks, i.e., networks designed to guarantee certain geometric relations between their inputs and outputs, yield state-of-the-art performance on spatial deep learning tasks. They exhibit high data efficiency during training and significantly reduced inference time for interatomic potential calculations compared to classical approaches. Key to these models is the Clebsch-Gordon (CG) tensor product, a kernel that contracts two dense feature vectors with a highly structured sparse tensor to produce a dense output vector. The operation, which may be repeated millions of times for typical equivariant models, is a costly and inefficient bottleneck. We introduce a GPU sparse kernel generator for the CG tensor product that provides significant speedup over the best existing open and closed-source implementations. Our implementation achieves high performance by carefully managing GPU shared memory through static analysis at model compile-time, minimizing reads and writes to global memory. We break the tensor product into a series of kernels with operands that fit entirely into registers, enabling us to emit long arithmetic instruction streams that maximize instruction-level parallelism. By fusing the CG tensor product with a subsequent graph convolution, we reduce both intermediate storage and global memory traffic over naive approaches that duplicate input data. We also provide optimized kernels for the gradient of the CG tensor product and a novel identity for the higher partial derivatives required to predict interatomic forces. Our fused kernels offer up to 4.5x speedup for the forward pass and 3x for the backward pass over NVIDIA cuEquivariance, as well as >10x speedup over the widely-used e3nn package. We offer up to 5.3x inference-time speedup for the MACE chemistry foundation model over the original unoptimized version.

Title: CGI: Identifying Conditional Generative Models with Example Images

• Authors: Zhi Zhou, Hao-Zhe Tan, Peng-Xiao Song, Lan-Zhe Guo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13991
• Pdf URL: https://arxiv.org/pdf/2501.13991
• Copy Paste: [[2501.13991]] CGI: Identifying Conditional Generative Models with Example Images(https://arxiv.org/abs/2501.13991)
• Keywords: generative
• Abstract: Generative models have achieved remarkable performance recently, and thus model hubs have emerged. Existing model hubs typically assume basic text matching is sufficient to search for models. However, in reality, due to different abstractions and the large number of models in model hubs, it is not easy for users to review model descriptions and example images, choosing which model best meets their needs. Therefore, it is necessary to describe model functionality wisely so that future users can efficiently search for the most suitable model for their needs. Efforts to address this issue remain limited. In this paper, we propose Conditional Generative Model Identification (CGI), which aims to provide an effective way to identify the most suitable model using user-provided example images rather than requiring users to manually review a large number of models with example images. To address this problem, we propose the PromptBased Model Identification (PMI) , which can adequately describe model functionality and precisely match requirements with specifications. To evaluate PMI approach and promote related research, we provide a benchmark comprising 65 models and 9100 identification tasks. Extensive experimental and human evaluation results demonstrate that PMI is effective. For instance, 92% of models are correctly identified with significantly better FID scores when four example images are provided.

Title: INDIGO+: A Unified INN-Guided Probabilistic Diffusion Algorithm for Blind and Non-Blind Image Restoration

• Authors: Di You, Pier Luigi Dragotti
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.14014
• Pdf URL: https://arxiv.org/pdf/2501.14014
• Copy Paste: [[2501.14014]] INDIGO+: A Unified INN-Guided Probabilistic Diffusion Algorithm for Blind and Non-Blind Image Restoration(https://arxiv.org/abs/2501.14014)
• Keywords: diffusion, generative
• Abstract: Generative diffusion models are becoming one of the most popular prior in image restoration (IR) tasks due to their remarkable ability to generate realistic natural images. Despite achieving satisfactory results, IR methods based on diffusion models present several limitations. First of all, most non-blind approaches require an analytical expression of the degradation model to guide the sampling process. Secondly, most existing blind approaches rely on families of pre-defined degradation models for training their deep networks. The above issues limit the flexibility of these approaches and so their ability to handle real-world degradation tasks. In this paper, we propose a novel INN-guided probabilistic diffusion algorithm for non-blind and blind image restoration, namely INDIGO and BlindINDIGO, which combines the merits of the perfect reconstruction property of invertible neural networks (INN) with the strong generative capabilities of pre-trained diffusion models. Specifically, we train the forward process of the INN to simulate an arbitrary degradation process and use the inverse to obtain an intermediate image that we use to guide the reverse diffusion sampling process through a gradient step. We also introduce an initialization strategy, to further improve the performance and inference speed of our algorithm. Experiments demonstrate that our algorithm obtains competitive results compared with recently leading methods both quantitatively and visually on synthetic and real-world low-quality images.

Title: LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps

• Authors: Andrey Palaev, Adil Khan, Syed M. Ahsan Kazmi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14046
• Pdf URL: https://arxiv.org/pdf/2501.14046
• Copy Paste: [[2501.14046]] LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps(https://arxiv.org/abs/2501.14046)
• Keywords: diffusion, generative
• Abstract: The advancement of text-to-image synthesis has introduced powerful generative models capable of creating realistic images from textual prompts. However, precise control over image attributes remains challenging, especially at the instance level. While existing methods offer some control through fine-tuning or auxiliary information, they often face limitations in flexibility and accuracy. To address these challenges, we propose a pipeline leveraging Large Language Models (LLMs), open-vocabulary detectors, cross-attention maps and intermediate activations of diffusion U-Net for instance-level image manipulation. Our method detects objects mentioned in the prompt and present in the generated image, enabling precise manipulation without extensive training or input masks. By incorporating cross-attention maps, our approach ensures coherence in manipulated images while controlling object positions. Our method enables precise manipulations at the instance level without fine-tuning or auxiliary information such as masks or bounding boxes. Code is available at this https URL

Title: Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models

• Authors: Jakob Krogh Petersen, Valdemar Licht, Mads Nielsen, Asbjørn Munk
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14051
• Pdf URL: https://arxiv.org/pdf/2501.14051
• Copy Paste: [[2501.14051]] Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models(https://arxiv.org/abs/2501.14051)
• Keywords: foundation model
• Abstract: Multi-modal models require aligned, shared embedding spaces. However, common CLIP-based approaches need large amounts of samples and do not natively support 3D or tabular data, both of which are crucial in the medical domain. To address these issues, we revisit CLIP-style alignment by training a domain-specific 3D foundation model as an image encoder and demonstrate that modality alignment is feasible with only 62 MRI scans. Our approach is enabled by a simple embedding accumulation strategy required for training in 3D, which scales the amount of negative pairs across batches in order to stabilize training. We perform a thorough evaluation of various design choices, including the choice of backbone and loss functions, and evaluate the proposed methodology on zero-shot classification and image-retrieval tasks. While zero-shot image-retrieval remains challenging, zero-shot classification results demonstrate that the proposed approach can meaningfully align the representations of 3D MRI with tabular data.

Title: Test-Time Code-Switching for Cross-lingual Aspect Sentiment Triplet Extraction

• Authors: Dongming Sheng, Kexin Han, Hao Li, Yan Zhang, Yucheng Huang, Jun Lang, Wenqiang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14144
• Pdf URL: https://arxiv.org/pdf/2501.14144
• Copy Paste: [[2501.14144]] Test-Time Code-Switching for Cross-lingual Aspect Sentiment Triplet Extraction(https://arxiv.org/abs/2501.14144)
• Keywords: generative
• Abstract: Aspect Sentiment Triplet Extraction (ASTE) is a thriving research area with impressive outcomes being achieved on high-resource languages. However, the application of cross-lingual transfer to the ASTE task has been relatively unexplored, and current code-switching methods still suffer from term boundary detection issues and out-of-dictionary problems. In this study, we introduce a novel Test-Time Code-SWitching (TT-CSW) framework, which bridges the gap between the bilingual training phase and the monolingual test-time prediction. During training, a generative model is developed based on bilingual code-switched training data and can produce bilingual ASTE triplets for bilingual inputs. In the testing stage, we employ an alignment-based code-switching technique for test-time augmentation. Extensive experiments on cross-lingual ASTE datasets validate the effectiveness of our proposed method. We achieve an average improvement of 3.7% in terms of weighted-averaged F1 in four datasets with different languages. Additionally, we set a benchmark using ChatGPT and GPT-4, and demonstrate that even smaller generative models fine-tuned with our proposed TT-CSW framework surpass ChatGPT and GPT-4 by 14.2% and 5.0% respectively.

Title: Argos: Agentic Time-Series Anomaly Detection with Autonomous Rule Generation via Large Language Models

• Authors: Yile Gu, Yifan Xiong, Jonathan Mace, Yuting Jiang, Yigong Hu, Baris Kasikci, Peng Cheng
• Subjects: cs.LG, cs.DC, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.14170
• Pdf URL: https://arxiv.org/pdf/2501.14170
• Copy Paste: [[2501.14170]] Argos: Agentic Time-Series Anomaly Detection with Autonomous Rule Generation via Large Language Models(https://arxiv.org/abs/2501.14170)
• Keywords: anomaly
• Abstract: Observability in cloud infrastructure is critical for service providers, driving the widespread adoption of anomaly detection systems for monitoring metrics. However, existing systems often struggle to simultaneously achieve explainability, reproducibility, and autonomy, which are three indispensable properties for production use. We introduce Argos, an agentic system for detecting time-series anomalies in cloud infrastructure by leveraging large language models (LLMs). Argos proposes to use explainable and reproducible anomaly rules as intermediate representation and employs LLMs to autonomously generate such rules. The system will efficiently train error-free and accuracy-guaranteed anomaly rules through multiple collaborative agents and deploy the trained rules for low-cost online anomaly detection. Through evaluation results, we demonstrate that Argos outperforms state-of-the-art methods, increasing $F_1$ scores by up to $9.5\%$ and $28.3\%$ on public anomaly detection datasets and an internal dataset collected from Microsoft, respectively.

Title: RL + Transformer = A General-Purpose Problem Solver

• Authors: Micah Rentschler, Jesse Roberts
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14176
• Pdf URL: https://arxiv.org/pdf/2501.14176
• Copy Paste: [[2501.14176]] RL + Transformer = A General-Purpose Problem Solver(https://arxiv.org/abs/2501.14176)
• Keywords: in-context
• Abstract: What if artificial intelligence could not only solve problems for which it was trained but also learn to teach itself to solve new problems (i.e., meta-learn)? In this study, we demonstrate that a pre-trained transformer fine-tuned with reinforcement learning over multiple episodes develops the ability to solve problems that it has never encountered before - an emergent ability called In-Context Reinforcement Learning (ICRL). This powerful meta-learner not only excels in solving unseen in-distribution environments with remarkable sample efficiency, but also shows strong performance in out-of-distribution environments. In addition, we show that it exhibits robustness to the quality of its training data, seamlessly stitches together behaviors from its context, and adapts to non-stationary environments. These behaviors demonstrate that an RL-trained transformer can iteratively improve upon its own solutions, making it an excellent general-purpose problem solver.

Title: VideoShield: Regulating Diffusion-based Video Generation Models via Watermarking

• Authors: Runyi Hu, Jie Zhang, Yiming Li, Jiwei Li, Qing Guo, Han Qiu, Tianwei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14195
• Pdf URL: https://arxiv.org/pdf/2501.14195
• Copy Paste: [[2501.14195]] VideoShield: Regulating Diffusion-based Video Generation Models via Watermarking(https://arxiv.org/abs/2501.14195)
• Keywords: diffusion
• Abstract: Artificial Intelligence Generated Content (AIGC) has advanced significantly, particularly with the development of video generation models such as text-to-video (T2V) models and image-to-video (I2V) models. However, like other AIGC types, video generation requires robust content control. A common approach is to embed watermarks, but most research has focused on images, with limited attention given to videos. Traditional methods, which embed watermarks frame-by-frame in a post-processing manner, often degrade video quality. In this paper, we propose VideoShield, a novel watermarking framework specifically designed for popular diffusion-based video generation models. Unlike post-processing methods, VideoShield embeds watermarks directly during video generation, eliminating the need for additional training. To ensure video integrity, we introduce a tamper localization feature that can detect changes both temporally (across frames) and spatially (within individual frames). Our method maps watermark bits to template bits, which are then used to generate watermarked noise during the denoising process. Using DDIM Inversion, we can reverse the video to its original watermarked noise, enabling straightforward watermark extraction. Additionally, template bits allow precise detection for potential temporal and spatial modification. Extensive experiments across various video models (both T2V and I2V models) demonstrate that our method effectively extracts watermarks and detects tamper without compromising video quality. Furthermore, we show that this approach is applicable to image generation models, enabling tamper detection in generated images as well. Codes and models are available at \href{this https URL}{this https URL}.

Title: Bi-directional Curriculum Learning for Graph Anomaly Detection: Dual Focus on Homogeneity and Heterogeneity

• Authors: Yitong Hao, Enbo He, Yue Zhang, Guisheng Yin
• Subjects: cs.LG, cs.SI, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.14197
• Pdf URL: https://arxiv.org/pdf/2501.14197
• Copy Paste: [[2501.14197]] Bi-directional Curriculum Learning for Graph Anomaly Detection: Dual Focus on Homogeneity and Heterogeneity(https://arxiv.org/abs/2501.14197)
• Keywords: anomaly
• Abstract: Graph anomaly detection (GAD) aims to identify nodes from a graph that are significantly different from normal patterns. Most previous studies are model-driven, focusing on enhancing the detection effect by improving the model structure. However, these approaches often treat all nodes equally, neglecting the different contributions of various nodes to the training. Therefore, we introduce graph curriculum learning as a simple and effective plug-and-play module to optimize GAD methods. The existing graph curriculum learning mainly focuses on the homogeneity of graphs and treats nodes with high homogeneity as easy nodes. In fact, GAD models can handle not only graph homogeneity but also heterogeneity, which leads to the unsuitability of these existing methods. To address this problem, we propose an innovative Bi-directional Curriculum Learning strategy (BCL), which considers nodes with higher and lower similarity to neighbor nodes as simple nodes in the direction of focusing on homogeneity and focusing on heterogeneity, respectively, and prioritizes their training. Extensive experiments show that BCL can be quickly integrated into existing detection processes and significantly improves the performance of ten GAD anomaly detection models on seven commonly used datasets.

Title: TFG-Flow: Training-free Guidance in Multimodal Generative Flow

• Authors: Haowei Lin, Shanda Li, Haotian Ye, Yiming Yang, Stefano Ermon, Yitao Liang, Jianzhu Ma
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2501.14216
• Pdf URL: https://arxiv.org/pdf/2501.14216
• Copy Paste: [[2501.14216]] TFG-Flow: Training-free Guidance in Multimodal Generative Flow(https://arxiv.org/abs/2501.14216)
• Keywords: diffusion, foundation model, generative
• Abstract: Given an unconditional generative model and a predictor for a target property (e.g., a classifier), the goal of training-free guidance is to generate samples with desirable target properties without additional training. As a highly efficient technique for steering generative models toward flexible outcomes, training-free guidance has gained increasing attention in diffusion models. However, existing methods only handle data in continuous spaces, while many scientific applications involve both continuous and discrete data (referred to as multimodality). Another emerging trend is the growing use of the simple and general flow matching framework in building generative foundation models, where guided generation remains under-explored. To address this, we introduce TFG-Flow, a novel training-free guidance method for multimodal generative flow. TFG-Flow addresses the curse-of-dimensionality while maintaining the property of unbiased sampling in guiding discrete variables. We validate TFG-Flow on four molecular design tasks and show that TFG-Flow has great potential in drug design by generating molecules with desired properties.

Title: Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Game

• Authors: Rong Ye, Yongxin Zhang, Yikai Zhang, Haoyu Kuang, Zhongyu Wei, Peng Sun
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.14225
• Pdf URL: https://arxiv.org/pdf/2501.14225
• Copy Paste: [[2501.14225]] Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Game(https://arxiv.org/abs/2501.14225)
• Keywords: in-context
• Abstract: Achieving Artificial General Intelligence (AGI) requires AI agents that can not only make stratigic decisions but also engage in flexible and meaningful communication. Inspired by Wittgenstein's language game theory in Philosophical Investigations, we propose that language agents can learn through in-context interaction rather than traditional multi-stage frameworks that separate decision-making from language expression. Using Werewolf, a social deduction game that tests language understanding, strategic interaction, and adaptability, we develop the Multi-agent Kahneman & Tversky's Optimization (MaKTO). MaKTO engages diverse models in extensive gameplay to generate unpaired desirable and unacceptable responses, then employs KTO to refine the model's decision-making process. In 9-player Werewolf games, MaKTO achieves a 61% average win rate across various models, outperforming GPT-4o and two-stage RL agents by relative improvements of 23.0% and 10.9%, respectively. Notably, MaKTO also demonstrates human-like performance, winning 60% against expert players and showing only 49% detectability in Turing-style blind tests. These results showcase MaKTO's superior decision-making, strategic adaptation, and natural language generation in complex social deduction games.

Title: TrajFlow: A Generative Framework for Occupancy Density Estimation Using Normalizing Flows

• Authors: Mitch Kosieradzki, Seongjin Choi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14266
• Pdf URL: https://arxiv.org/pdf/2501.14266
• Copy Paste: [[2501.14266]] TrajFlow: A Generative Framework for Occupancy Density Estimation Using Normalizing Flows(https://arxiv.org/abs/2501.14266)
• Keywords: generative
• Abstract: In transportation systems and autonomous vehicles, intelligent agents must understand the future motion of traffic participants to effectively plan motion trajectories. At the same time, the motion of traffic participants is inherently uncertain. In this paper, we propose TrajFlow, a generative framework for estimating the occupancy density of traffic participants. Our framework utilizes a causal encoder to extract semantically meaningful embeddings of the observed trajectory, as well as a normalizing flow to decode these embeddings and determine the most likely future location of traffic participants at some time point in the future. Our formulation differs from existing approaches because we model the marginal distribution of spatial locations instead of the joint distribution of unobserved trajectories. The advantages of a marginal formulation are numerous. First, we demonstrate that the marginal formulation produces higher accuracy on challenging trajectory forecasting benchmarks. Second, the marginal formulation allows for a fully continuous sampling of future locations. Finally, marginal densities are better suited for downstream tasks as they allow for the computation of per-agent motion trajectories and occupancy grids, the two most commonly used representations for motion forecasting. We present a novel architecture based entirely on neural differential equations as an implementation of this framework and provide ablations to demonstrate the advantages of a continuous implementation over a more traditional discrete neural network based approach. The code is available at this https URL .

Title: Distinguishing Parkinson's Patients Using Voice-Based Feature Extraction and Classification

• Authors: Burak Çelik, Ayhan Akbal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14390
• Pdf URL: https://arxiv.org/pdf/2501.14390
• Copy Paste: [[2501.14390]] Distinguishing Parkinson's Patients Using Voice-Based Feature Extraction and Classification(https://arxiv.org/abs/2501.14390)
• Keywords: generative
• Abstract: Parkinson's disease (PD) is a progressive neurodegenerative disorder that impacts motor functions and speech characteristics This study focuses on differentiating individuals with Parkinson's disease from healthy controls through the extraction and classification of speech features. Patients were further divided into 2 groups. Med On represents the patient with medication, while Med Off represents the patient without medication. The dataset consisted of patients and healthy individuals who read a predefined text using the H1N Zoom microphone in a suitable recording environment at Fırat University Neurology Department. Speech recordings from PD patients and healthy controls were analyzed, and 19 key features were extracted, including jitter, luminance, zero-crossing rate (ZCR), root mean square (RMS) energy, entropy, skewness, and this http URL features were visualized in graphs and statistically evaluated to identify distinctive patterns in PD patients. Using MATLAB's Classification Learner toolbox, several machine learning classification algorithm models were applied to classify groups and significant accuracy rates were achieved. The accuracy of our 3-layer artificial neural network architecture was also compared with classical machine learning algorithms. This study highlights the potential of noninvasive voice analysis combined with machine learning for early detection and monitoring of PD patients. Future research can improve diagnostic accuracy by optimizing feature selection and exploring advanced classification techniques.

Title: CENTS: Generating synthetic electricity consumption time series for rare and unseen scenarios

• Authors: Michael Fuest, Alfredo Cuesta, Kalyan Veeramachaneni
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14426
• Pdf URL: https://arxiv.org/pdf/2501.14426
• Copy Paste: [[2501.14426]] CENTS: Generating synthetic electricity consumption time series for rare and unseen scenarios(https://arxiv.org/abs/2501.14426)
• Keywords: foundation model, generative
• Abstract: Recent breakthroughs in large-scale generative modeling have demonstrated the potential of foundation models in domains such as natural language, computer vision, and protein structure prediction. However, their application in the energy and smart grid sector remains limited due to the scarcity and heterogeneity of high-quality data. In this work, we propose a method for creating high-fidelity electricity consumption time series data for rare and unseen context variables (e.g. location, building type, photovoltaics). Our approach, Context Encoding and Normalizing Time Series Generation, or CENTS, includes three key innovations: (i) A context normalization approach that enables inverse transformation for time series context variables unseen during training, (ii) a novel context encoder to condition any state-of-the-art time-series generator on arbitrary numbers and combinations of context variables, (iii) a framework for training this context encoder jointly with a time-series generator using an auxiliary context classification loss designed to increase expressivity of context embeddings and improve model performance. We further provide a comprehensive overview of different evaluation metrics for generative time series models. Our results highlight the efficacy of the proposed method in generating realistic household-level electricity consumption data, paving the way for training larger foundation models in the energy domain on synthetic as well as real-world data.

Title: Pesti-Gen: Unleashing a Generative Molecule Approach for Toxicity Aware Pesticide Design

• Authors: Taehan Kim, Wonduk Seo
• Subjects: cs.LG, cs.AI, q-bio.BM, q-bio.MN
• Abstract URL: https://arxiv.org/abs/2501.14469
• Pdf URL: https://arxiv.org/pdf/2501.14469
• Copy Paste: [[2501.14469]] Pesti-Gen: Unleashing a Generative Molecule Approach for Toxicity Aware Pesticide Design(https://arxiv.org/abs/2501.14469)
• Keywords: generative
• Abstract: Global climate change has reduced crop resilience and pesticide efficacy, making reliance on synthetic pesticides inevitable, even though their widespread use poses significant health and environmental risks. While these pesticides remain a key tool in pest management, previous machine-learning applications in pesticide and agriculture have focused on classification or regression, leaving the fundamental challenge of generating new molecular structures or designing novel candidates unaddressed. In this paper, we propose Pesti-Gen, a novel generative model based on variational auto-encoders, designed to create pesticide candidates with optimized properties for the first time. Specifically, Pesti-Gen leverages a two-stage learning process: an initial pre-training phase that captures a generalized chemical structure representation, followed by a fine-tuning stage that incorporates toxicity-specific information. The model simultaneously optimizes over multiple toxicity metrics, such as (1) livestock toxicity and (2) aqua toxicity to generate environmentally friendly pesticide candidates. Notably, Pesti-Gen achieves approximately 68\% structural validity in generating new molecular structures, demonstrating the model's effectiveness in producing optimized and feasible pesticide candidates, thereby providing a new way for safer and more sustainable pest management solutions.

Title: Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion 2.*

• Authors: Ludovica Schaerf, Andrea Alfarano, Fabrizio Silvestri, Leonardo Impett
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14524
• Pdf URL: https://arxiv.org/pdf/2501.14524
• Copy Paste: [[2501.14524]] Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion 2.*(https://arxiv.org/abs/2501.14524)
• Keywords: diffusion
• Abstract: Despite significant recent advances in image generation with diffusion models, their internal latent representations remain poorly understood. Existing works focus on the bottleneck layer (h-space) of Stable Diffusion's U-Net or leverage the cross-attention, self-attention, or decoding layers. Our model, SkipInject takes advantage of U-Net's skip connections. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our methods state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff.

Title: Exploring Answer Set Programming for Provenance Graph-Based Cyber Threat Detection: A Novel Approach

• Authors: Fang Li, Fei Zuo, Gopal Gupta
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2501.14555
• Pdf URL: https://arxiv.org/pdf/2501.14555
• Copy Paste: [[2501.14555]] Exploring Answer Set Programming for Provenance Graph-Based Cyber Threat Detection: A Novel Approach(https://arxiv.org/abs/2501.14555)
• Keywords: anomaly
• Abstract: Provenance graphs are useful and powerful tools for representing system-level activities in cybersecurity; however, existing approaches often struggle with complex queries and flexible reasoning. This paper presents a novel approach using Answer Set Programming (ASP) to model and analyze provenance graphs. We introduce an ASP-based representation that captures intricate relationships between system entities, including temporal and causal dependencies. Our model enables sophisticated analysis capabilities such as attack path tracing, data exfiltration detection, and anomaly identification. The declarative nature of ASP allows for concise expression of complex security patterns and policies, facilitating both real-time threat detection and forensic analysis. We demonstrate our approach's effectiveness through case studies showcasing its threat detection capabilities. Experimental results illustrate the model's ability to handle large-scale provenance graphs while providing expressive querying. The model's extensibility allows for incorporation of new system behaviors and security rules, adapting to evolving cyber threats. This work contributes a powerful, flexible, and explainable framework for reasoning about system behaviors and security incidents, advancing the development of effective threat detection and forensic investigation tools.

Title: ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations

• Authors: Tianming Liang, Kun-Yu Lin, Chaolei Tan, Jianguo Zhang, Wei-Shi Zheng, Jian-Fang Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.14607
• Pdf URL: https://arxiv.org/pdf/2501.14607
• Copy Paste: [[2501.14607]] ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations(https://arxiv.org/abs/2501.14607)
• Keywords: foundation model
• Abstract: Referring video object segmentation (RVOS) aims to segment target objects throughout a video based on a text description. Despite notable progress in recent years, current RVOS models remain struggle to handle complicated object descriptions due to their limited video-language understanding. To address this limitation, we present \textbf{ReferDINO}, an end-to-end RVOS model that inherits strong vision-language understanding from the pretrained visual grounding foundation models, and is further endowed with effective temporal understanding and object segmentation capabilities. In ReferDINO, we contribute three technical innovations for effectively adapting the foundation models to RVOS: 1) an object-consistent temporal enhancer that capitalizes on the pretrained object-text representations to enhance temporal understanding and object consistency; 2) a grounding-guided deformable mask decoder that integrates text and grounding conditions to generate accurate object masks; 3) a confidence-aware query pruning strategy that significantly improves the object decoding efficiency without compromising performance. We conduct extensive experiments on five public RVOS benchmarks to demonstrate that our proposed ReferDINO outperforms state-of-the-art methods significantly. Project page: \url{this https URL}

Title: Funzac at CoMeDi Shared Task: Modeling Annotator Disagreement from Word-In-Context Perspectives

• Authors: Olufunke O. Sarumi, Charles Welch, Lucie Flek, Jörg Schlötterer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14617
• Pdf URL: https://arxiv.org/pdf/2501.14617
• Copy Paste: [[2501.14617]] Funzac at CoMeDi Shared Task: Modeling Annotator Disagreement from Word-In-Context Perspectives(https://arxiv.org/abs/2501.14617)
• Keywords: in-context
• Abstract: In this work, we evaluate annotator disagreement in Word-in-Context (WiC) tasks exploring the relationship between contextual meaning and disagreement as part of the CoMeDi shared task competition. While prior studies have modeled disagreement by analyzing annotator attributes with single-sentence inputs, this shared task incorporates WiC to bridge the gap between sentence-level semantic representation and annotator judgment variability. We describe three different methods that we developed for the shared task, including a feature enrichment approach that combines concatenation, element-wise differences, products, and cosine similarity, Euclidean and Manhattan distances to extend contextual embedding representations, a transformation by Adapter blocks to obtain task-specific representations of contextual embeddings, and classifiers of varying complexities, including ensembles. The comparison of our methods demonstrates improved performance for methods that include enriched and task-specfic features. While the performance of our method falls short in comparison to the best system in subtask 1 (OGWiC), it is competitive to the official evaluation results in subtask 2 (DisWiC).

Title: ACT-JEPA: Joint-Embedding Predictive Architecture Improves Policy Representation Learning

• Authors: Aleksandar Vujinovic, Aleksandar Kovacevic
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14622
• Pdf URL: https://arxiv.org/pdf/2501.14622
• Copy Paste: [[2501.14622]] ACT-JEPA: Joint-Embedding Predictive Architecture Improves Policy Representation Learning(https://arxiv.org/abs/2501.14622)
• Keywords: self-supervised
• Abstract: Learning efficient representations for decision-making policies is a challenge in imitation learning (IL). Current IL methods require expert demonstrations, which are expensive to collect. Consequently, they often have underdeveloped world models. Self-supervised learning (SSL) offers an alternative by allowing models to learn from diverse, unlabeled data, including failures. However, SSL methods often operate in raw input space, making them inefficient. In this work, we propose ACT-JEPA, a novel architecture that integrates IL and SSL to enhance policy representations. We train a policy to predict (1) action sequences and (2) abstract observation sequences. The first objective uses action chunking to improve action prediction and reduce compounding errors. The second objective extends this idea of chunking by predicting abstract observation sequences. We utilize Joint-Embedding Predictive Architecture to predict in abstract representation space, allowing the model to filter out irrelevant details, improve efficiency, and develop a robust world model. Our experiments show that ACT-JEPA improves the quality of representations by learning temporal environment dynamics. Additionally, the model's ability to predict abstract observation sequences results in representations that effectively generalize to action sequence prediction. ACT-JEPA performs on par with established baselines across a range of decision-making tasks.

Title: Towards Scalable Topological Regularizers

• Authors: Hiu-Tung Wong, Darrick Lee, Hong Yan
• Subjects: cs.LG, math.AT
• Abstract URL: https://arxiv.org/abs/2501.14641
• Pdf URL: https://arxiv.org/pdf/2501.14641
• Copy Paste: [[2501.14641]] Towards Scalable Topological Regularizers(https://arxiv.org/abs/2501.14641)
• Keywords: generative
• Abstract: Latent space matching, which consists of matching distributions of features in latent space, is a crucial component for tasks such as adversarial attacks and defenses, domain adaptation, and generative modelling. Metrics for probability measures, such as Wasserstein and maximum mean discrepancy, are commonly used to quantify the differences between such distributions. However, these are often costly to compute, or do not appropriately take the geometric and topological features of the distributions into consideration. Persistent homology is a tool from topological data analysis which quantifies the multi-scale topological structure of point clouds, and has recently been used as a topological regularizer in learning tasks. However, computation costs preclude larger scale computations, and discontinuities in the gradient lead to unstable training behavior such as in adversarial tasks. We propose the use of principal persistence measures, based on computing the persistent homology of a large number of small subsamples, as a topological regularizer. We provide a parallelized GPU implementation of this regularizer, and prove that gradients are continuous for smooth densities. Furthermore, we demonstrate the efficacy of this regularizer on shape matching, image generation, and semi-supervised learning tasks, opening the door towards a scalable regularizer for topological features.

Title: Towards Automated Self-Supervised Learning for Truly Unsupervised Graph Anomaly Detection

• Authors: Zhong Li, Yuhang Wang, Matthijs van Leeuwen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14694
• Pdf URL: https://arxiv.org/pdf/2501.14694
• Copy Paste: [[2501.14694]] Towards Automated Self-Supervised Learning for Truly Unsupervised Graph Anomaly Detection(https://arxiv.org/abs/2501.14694)
• Keywords: self-supervised, anomaly
• Abstract: Self-supervised learning (SSL) is an emerging paradigm that exploits supervisory signals generated from the data itself, and many recent studies have leveraged SSL to conduct graph anomaly detection. However, we empirically found that three important factors can substantially impact detection performance across datasets: 1) the specific SSL strategy employed; 2) the tuning of the strategy's hyperparameters; and 3) the allocation of combination weights when using multiple strategies. Most SSL-based graph anomaly detection methods circumvent these issues by arbitrarily or selectively (i.e., guided by label information) choosing SSL strategies, hyperparameter settings, and combination weights. While an arbitrary choice may lead to subpar performance, using label information in an unsupervised setting is label information leakage and leads to severe overestimation of a method's performance. Leakage has been criticized as "one of the top ten data mining mistakes", yet many recent studies on SSL-based graph anomaly detection have been using label information to select hyperparameters. To mitigate this issue, we propose to use an internal evaluation strategy (with theoretical analysis) to select hyperparameters in SSL for unsupervised anomaly detection. We perform extensive experiments using 10 recent SSL-based graph anomaly detection algorithms on various benchmark datasets, demonstrating both the prior issues with hyperparameter selection and the effectiveness of our proposed strategy.