2024-10-25

Title: Gesture2Text: A Generalizable Decoder for Word-Gesture Keyboards in XR Through Trajectory Coarse Discretization and Pre-training

• Authors: Junxiao Shen, Khadija Khaldi, Enmin Zhou, Hemant Bhaskar Surale, Amy Karlson
• Subjects: cs.CV, cs.AI, cs.CL, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18099
• Pdf URL: https://arxiv.org/pdf/2410.18099
• Copy Paste: [[2410.18099]] Gesture2Text: A Generalizable Decoder for Word-Gesture Keyboards in XR Through Trajectory Coarse Discretization and Pre-training(https://arxiv.org/abs/2410.18099)
• Keywords: robust
• Abstract: Text entry with word-gesture keyboards (WGK) is emerging as a popular method and becoming a key interaction for Extended Reality (XR). However, the diversity of interaction modes, keyboard sizes, and visual feedback in these environments introduces divergent word-gesture trajectory data patterns, thus leading to complexity in decoding trajectories into text. Template-matching decoding methods, such as SHARK^2, are commonly used for these WGK systems because they are easy to implement and configure. However, these methods are susceptible to decoding inaccuracies for noisy trajectories. While conventional neural-network-based decoders (neural decoders) trained on word-gesture trajectory data have been proposed to improve accuracy, they have their own limitations: they require extensive data for training and deep-learning expertise for implementation. To address these challenges, we propose a novel solution that combines ease of implementation with high decoding accuracy: a generalizable neural decoder enabled by pre-training on large-scale coarsely discretized word-gesture trajectories. This approach produces a ready-to-use WGK decoder that is generalizable across mid-air and on-surface WGK systems in augmented reality (AR) and virtual reality (VR), which is evident by a robust average Top-4 accuracy of 90.4% on four diverse datasets. It significantly outperforms SHARK^2 with a 37.2% enhancement and surpasses the conventional neural decoder by 7.4%. Moreover, the Pre-trained Neural Decoder's size is only 4 MB after quantization, without sacrificing accuracy, and it can operate in real-time, executing in just 97 milliseconds on Quest 3.

Title: NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired People

• Authors: Jun Yu, Yifan Zhang, Badrinadh Aila, Vinod Namboodiri
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18109
• Pdf URL: https://arxiv.org/pdf/2410.18109
• Copy Paste: [[2410.18109]] NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired People(https://arxiv.org/abs/2410.18109)
• Keywords: fair
• Abstract: Indoor navigation is challenging due to the absence of satellite positioning. This challenge is manifold greater for Visually Impaired People (VIPs) who lack the ability to get information from wayfinding signage. Other sensor signals (e.g., Bluetooth and LiDAR) can be used to create turn-by-turn navigation solutions with position updates for users. Unfortunately, these solutions require tags to be installed all around the environment or the use of fairly expensive hardware. Moreover, these solutions require a high degree of manual involvement that raises costs, thus hampering scalability. We propose an image dataset and associated image-centric solution called NaVIP towards visual intelligence that is infrastructure-free and task-scalable, and can assist VIPs in understanding their surroundings. Specifically, we start by curating large-scale phone camera data in a four-floor research building, with 300K images, to lay the foundation for creating an image-centric indoor navigation and exploration solution for inclusiveness. Every image is labelled with precise 6DoF camera poses, details of indoor PoIs, and descriptive captions to assist VIPs. We benchmark on two main aspects: 1) positioning system and 2) exploration support, prioritizing training scalability and real-time inference, to validate the prospect of image-based solution towards indoor navigation. The dataset, code, and model checkpoints are made publicly available at this https URL.

Title: Efficient Adaptive Federated Optimization

• Authors: Su Hyeong Lee, Sidharth Sharma, Manzil Zaheer, Tian Li
• Subjects: cs.LG, cs.DC, math.OC
• Abstract URL: https://arxiv.org/abs/2410.18117
• Pdf URL: https://arxiv.org/pdf/2410.18117
• Copy Paste: [[2410.18117]] Efficient Adaptive Federated Optimization(https://arxiv.org/abs/2410.18117)
• Keywords: federate
• Abstract: Adaptive optimization plays a pivotal role in federated learning, where simultaneous server and client-side adaptivity have been shown to be essential for maximizing its performance. However, the scalability of jointly adaptive systems is often constrained by limited resources in communication and memory. In this paper, we introduce a class of efficient adaptive algorithms, named $FedAda^2$, designed specifically for large-scale, cross-device federated environments. $FedAda^2$ optimizes communication efficiency by avoiding the transfer of preconditioners between the server and clients. At the same time, it leverages memory-efficient adaptive optimizers on the client-side to reduce on-device memory consumption. Theoretically, we demonstrate that $FedAda^2$ achieves the same convergence rates for general, non-convex objectives as its more resource-intensive counterparts that directly integrate joint adaptivity. Empirically, we showcase the benefits of joint adaptivity and the effectiveness of $FedAda^2$ on both image and text datasets.

Title: R2Gen-Mamba: A Selective State Space Model for Radiology Report Generation

• Authors: Yongheng Sun, Yueh Z. Lee, Genevieve A. Woodard, Hongtu Zhu, Chunfeng Lian, Mingxia Liu
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18135
• Pdf URL: https://arxiv.org/pdf/2410.18135
• Copy Paste: [[2410.18135]] R2Gen-Mamba: A Selective State Space Model for Radiology Report Generation(https://arxiv.org/abs/2410.18135)
• Keywords: transformer
• Abstract: Radiology report generation is crucial in medical imaging,but the manual annotation process by physicians is time-consuming and labor-intensive, necessitating the develop-ment of automatic report generation methods. Existingresearch predominantly utilizes Transformers to generateradiology reports, which can be computationally intensive,limiting their use in real applications. In this work, we presentR2Gen-Mamba, a novel automatic radiology report genera-tion method that leverages the efficient sequence processingof the Mamba with the contextual benefits of Transformerarchitectures. Due to lower computational complexity ofMamba, R2Gen-Mamba not only enhances training and in-ference efficiency but also produces high-quality this http URL results on two benchmark datasets with morethan 210,000 X-ray image-report pairs demonstrate the ef-fectiveness of R2Gen-Mamba regarding report quality andcomputational efficiency compared with several state-of-the-art methods. The source code can be accessed online.

Title: Advancing Super-Resolution in Neural Radiance Fields via Variational Diffusion Strategies

• Authors: Shrey Vishen, Jatin Sarabu, Chinmay Bharathulwar, Rithwick Lakshmanan, Vishnu Srinivas
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18137
• Pdf URL: https://arxiv.org/pdf/2410.18137
• Copy Paste: [[2410.18137]] Advancing Super-Resolution in Neural Radiance Fields via Variational Diffusion Strategies(https://arxiv.org/abs/2410.18137)
• Keywords: diffusion
• Abstract: We present a novel method for diffusion-guided frameworks for view-consistent super-resolution (SR) in neural rendering. Our approach leverages existing 2D SR models in conjunction with advanced techniques such as Variational Score Distilling (VSD) and a LoRA fine-tuning helper, with spatial training to significantly boost the quality and consistency of upscaled 2D images compared to the previous methods in the literature, such as Renoised Score Distillation (RSD) proposed in DiSR-NeRF (1), or SDS proposed in DreamFusion. The VSD score facilitates precise fine-tuning of SR models, resulting in high-quality, view-consistent images. To address the common challenge of inconsistencies among independent SR 2D images, we integrate Iterative 3D Synchronization (I3DS) from the DiSR-NeRF framework. Our quantitative benchmarks and qualitative results on the LLFF dataset demonstrate the superior performance of our system compared to existing methods such as DiSR-NeRF.

Title: Analyzing Nobel Prize Literature with Large Language Models

• Authors: Yang Zhenyuan, Liu Zhengliang, Zhang Jing, Lu Cen, Tai Jiaxin, Zhong Tianyang, Li Yiwei, Zhao Siyan, Yao Teng, Liu Qing, Yang Jinlin, Liu Qixin, Li Zhaowei, Wang Kexin, Ma Longjun, Zhu Dajiang, Ren Yudan, Ge Bao, Zhang Wei, Qiang Ning, Zhang Tuo, Liu Tianming
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18142
• Pdf URL: https://arxiv.org/pdf/2410.18142
• Copy Paste: [[2410.18142]] Analyzing Nobel Prize Literature with Large Language Models(https://arxiv.org/abs/2410.18142)
• Keywords: large language model
• Abstract: This study examines the capabilities of advanced Large Language Models (LLMs), particularly the o1 model, in the context of literary analysis. The outputs of these models are compared directly to those produced by graduate-level human participants. By focusing on two Nobel Prize-winning short stories, 'Nine Chapters' by Han Kang, the 2024 laureate, and 'Friendship' by Jon Fosse, the 2023 laureate, the research explores the extent to which AI can engage with complex literary elements such as thematic analysis, intertextuality, cultural and historical contexts, linguistic and structural innovations, and character development. Given the Nobel Prize's prestige and its emphasis on cultural, historical, and linguistic richness, applying LLMs to these works provides a deeper understanding of both human and AI approaches to interpretation. The study uses qualitative and quantitative evaluations of coherence, creativity, and fidelity to the text, revealing the strengths and limitations of AI in tasks typically reserved for human expertise. While LLMs demonstrate strong analytical capabilities, particularly in structured tasks, they often fall short in emotional nuance and coherence, areas where human interpretation excels. This research underscores the potential for human-AI collaboration in the humanities, opening new opportunities in literary studies and beyond.

Title: Meaning Typed Prompting: A Technique for Efficient, Reliable Structured Output Generation

• Authors: Chandra Irugalbandara
• Subjects: cs.CL, cs.AI, cs.PL
• Abstract URL: https://arxiv.org/abs/2410.18146
• Pdf URL: https://arxiv.org/pdf/2410.18146
• Copy Paste: [[2410.18146]] Meaning Typed Prompting: A Technique for Efficient, Reliable Structured Output Generation(https://arxiv.org/abs/2410.18146)
• Keywords: large language model
• Abstract: Extending Large Language Models (LLMs) to advanced applications requires reliable structured output generation. Existing methods which often rely on rigid JSON schemas, can lead to unreliable outputs, diminished reasoning capabilities, and increased computational overhead, limiting LLMs' adaptability for complex tasks. We introduce Meaning Typed Prompting (MTP), a technique for efficient structured output generation that integrates types, meanings, and abstractions, such as variables and classes, into the prompting process. By utilizing expressive type definitions, MTP enhances output clarity and reduces dependence on complex abstractions, simplifying development, and improving implementation efficiency. This enables LLMs to understand relationships and generate structured data more effectively. Empirical evaluations on multiple benchmarks demonstrate that MTP outperforms existing frameworks in accuracy, reliability, consistency, and token efficiency. We present Semantix, a framework that implements MTP, providing practical insights into its application.

Title: Deep Autoencoder with SVD-Like Convergence and Flat Minima

• Authors: Nithin Somasekharan, Shaowu Pan
• Subjects: cs.LG, cs.AI, physics.comp-ph, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.18148
• Pdf URL: https://arxiv.org/pdf/2410.18148
• Copy Paste: [[2410.18148]] Deep Autoencoder with SVD-Like Convergence and Flat Minima(https://arxiv.org/abs/2410.18148)
• Keywords: robust
• Abstract: Representation learning for high-dimensional, complex physical systems aims to identify a low-dimensional intrinsic latent space, which is crucial for reduced-order modeling and modal analysis. To overcome the well-known Kolmogorov barrier, deep autoencoders (AEs) have been introduced in recent years, but they often suffer from poor convergence behavior as the rank of the latent space increases. To address this issue, we propose the learnable weighted hybrid autoencoder, a hybrid approach that combines the strengths of singular value decomposition (SVD) with deep autoencoders through a learnable weighted framework. We find that the introduction of learnable weighting parameters is essential - without them, the resulting model would either collapse into a standard POD or fail to exhibit the desired convergence behavior. Additionally, we empirically find that our trained model has a sharpness thousands of times smaller compared to other models. Our experiments on classical chaotic PDE systems, including the 1D Kuramoto-Sivashinsky and forced isotropic turbulence datasets, demonstrate that our approach significantly improves generalization performance compared to several competing methods, paving the way for robust representation learning of high-dimensional, complex physical systems.

Title: A Type System to Ensure Non-Interference in ReScript

• Authors: Benjamin Bennetzen, Daniel Vang Kleist, Emilie Sonne Steinmann, Loke Walsted, Nikolaj Rossander Kristensen, Peter Buus Steffensen
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2410.18157
• Pdf URL: https://arxiv.org/pdf/2410.18157
• Copy Paste: [[2410.18157]] A Type System to Ensure Non-Interference in ReScript(https://arxiv.org/abs/2410.18157)
• Keywords: security, protect, robust
• Abstract: Protecting confidential data from leaking is a critical challenge in computer systems, particularly given the growing number of observers on the internet. Therefore, limiting information flow using robust security policies becomes increasingly vital. We focus on the non-interference policy, where the goal is to ensure that confidential data can not impact public data. This paper presents a type system, for a subset of the ReScript syntax, designed to enforce non-interference. We conclude with a proof of soundness for the type system, demonstrating that if an expression is type-able, it is inherently non-interferent. In addition, we provide a brief overview of a type checker that implements the previously mentioned type system.

Title: Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction

• Authors: Nicholas Walker
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18160
• Pdf URL: https://arxiv.org/pdf/2410.18160
• Copy Paste: [[2410.18160]] Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction(https://arxiv.org/abs/2410.18160)
• Keywords: transformer, generative
• Abstract: Causal decoder-only transformer models used for generative language modelling, such as Generative Pre-trained Transformers (GPT), are trained to predict the next token in a sequence based only on its previous tokens. Despite this simple training objective, they have proved to be powerful AI tools. However, only predicting the next token results in top layer embedding vectors that are highly token-focused. There may be benefits in generating embedding vectors at each token position that better capture the overall meaning of longer sequences of future text. Recent studies matching brain scans with deep language models suggest that humans also predict upcoming words when listening or reading but consider multiple future tokens rather than just one. This research investigates a new pretraining method called Future Token Prediction (FTP). In FTP, a large transformer encoder generates top layer embedding vectors for each token position, which, instead of being passed to a language head, are linearly and expansively projected to a pseudo-sequence, which is cross attended to by a small transformer decoder to predict the next N tokens forward from that position in the sequence. The top layer embedding vectors from FTP models exhibit distinct properties compared to those from standard GPT models, varying smoothly along a text sequence as measured by cosine similarity between adjacent tokens. Text generated by FTP models show improved topic coherence compared to standard GPT-like models trained with the same prediction perplexity for the next single token. The vectors are shown to better represent the topic of text based on the results of text classification examples. On a toy, but complex, coding problem, FTP networks produce significantly better results than GPT networks.

Title: Gazelle: An Instruction Dataset for Arabic Writing Assistance

• Authors: Samar M. Magdy, Fakhraddin Alwajih, Sang Yun Kwon, Reem Abdel-Salam, Muhammad Abdul-Mageed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18163
• Pdf URL: https://arxiv.org/pdf/2410.18163
• Copy Paste: [[2410.18163]] Gazelle: An Instruction Dataset for Arabic Writing Assistance(https://arxiv.org/abs/2410.18163)
• Keywords: generative, large language model
• Abstract: Writing has long been considered a hallmark of human intelligence and remains a pinnacle task for artificial intelligence (AI) due to the intricate cognitive processes involved. Recently, rapid advancements in generative AI, particularly through the development of Large Language Models (LLMs), have significantly transformed the landscape of writing assistance. However, underrepresented languages like Arabic encounter significant challenges in the development of advanced AI writing tools, largely due to the limited availability of data. This scarcity constrains the training of effective models, impeding the creation of sophisticated writing assistance technologies. To address these issues, we present Gazelle, a comprehensive dataset for Arabic writing assistance. In addition, we offer an evaluation framework designed to enhance Arabic writing assistance tools. Our human evaluation of leading LLMs, including GPT-4, GPT-4o, Cohere Command R+, and Gemini 1.5 Pro, highlights their respective strengths and limitations in addressing the challenges of Arabic writing. Our findings underscore the need for continuous model training and dataset enrichment to manage the complexities of Arabic language processing, paving the way for more effective AI-powered Arabic writing tools.

Title: TabDPT: Scaling Tabular Foundation Models

• Authors: Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, Anthony L. Caterini
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.18164
• Pdf URL: https://arxiv.org/pdf/2410.18164
• Copy Paste: [[2410.18164]] TabDPT: Scaling Tabular Foundation Models(https://arxiv.org/abs/2410.18164)
• Keywords: transformer, large language model
• Abstract: The challenges faced by neural networks on tabular data are well-documented and have hampered the progress of tabular foundation models. Techniques leveraging in-context learning (ICL) have shown promise here, allowing for dynamic adaptation to unseen data. ICL can provide predictions for entirely new datasets without further training or hyperparameter tuning, therefore providing very fast inference when encountering a novel task. However, scaling ICL for tabular data remains an issue: approaches based on large language models cannot efficiently process numeric tables, and tabular-specific techniques have not been able to effectively harness the power of real data to improve performance and generalization. We are able to overcome these challenges by training tabular-specific ICL-based architectures on real data with self-supervised learning and retrieval, combining the best of both worlds. Our resulting model -- the Tabular Discriminative Pre-trained Transformer (TabDPT) -- achieves state-of-the-art performance on the CC18 (classification) and CTR23 (regression) benchmarks with no task-specific fine-tuning, demonstrating the adapatability and speed of ICL once the model is pre-trained. TabDPT also demonstrates strong scaling as both model size and amount of available data increase, pointing towards future improvements simply through the curation of larger tabular pre-training datasets and training larger models.

Title: Automated Defect Detection and Grading of Piarom Dates Using Deep Learning

• Authors: Nasrin Azimi, Danial Mohammad Rezaei
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18208
• Pdf URL: https://arxiv.org/pdf/2410.18208
• Copy Paste: [[2410.18208]] Automated Defect Detection and Grading of Piarom Dates Using Deep Learning(https://arxiv.org/abs/2410.18208)
• Keywords: robust, segmentation
• Abstract: Grading and quality control of Piarom dates, a premium and high-value variety cultivated predominantly in Iran, present significant challenges due to the complexity and variability of defects, as well as the absence of specialized automated systems tailored to this fruit. Traditional manual inspection methods are labor intensive, time consuming, and prone to human error, while existing AI-based sorting solutions are insufficient for addressing the nuanced characteristics of Piarom dates. In this study, we propose an innovative deep learning framework designed specifically for the real-time detection, classification, and grading of Piarom dates. Leveraging a custom dataset comprising over 9,900 high-resolution images annotated across 11 distinct defect categories, our framework integrates state-of-the-art object detection algorithms and Convolutional Neural Networks (CNNs) to achieve high precision in defect identification. Furthermore, we employ advanced segmentation techniques to estimate the area and weight of each date, thereby optimizing the grading process according to industry standards. Experimental results demonstrate that our system significantly outperforms existing methods in terms of accuracy and computational efficiency, making it highly suitable for industrial applications requiring real-time processing. This work not only provides a robust and scalable solution for automating quality control in the Piarom date industry but also contributes to the broader field of AI-driven food inspection technologies, with potential applications across various agricultural products.

Title: CorrectionLM: Self-Corrections with SLM for Dialogue State Tracking

• Authors: Chia-Hsuan Lee, Hao Cheng, Mari Ostendorf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18209
• Pdf URL: https://arxiv.org/pdf/2410.18209
• Copy Paste: [[2410.18209]] CorrectionLM: Self-Corrections with SLM for Dialogue State Tracking(https://arxiv.org/abs/2410.18209)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated self-improvement capabilities via feedback and refinement, but current small language models (SLMs) have had limited success in this area. Existing correction approaches often rely on distilling knowledge from LLMs, which imposes significant computation demands. In this work, we introduce CORRECTIONLM, a novel correction framework that enables SLMs to self-correct using in-context exemplars without LLM involvement. Applied to two dialogue state tracking (DST) tasks in low-resource settings, CORRECTIONLM achieves results similar to a state-of-the-art LLM at a small fraction of the computation costs.

Title: Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks

• Authors: Samuele Poppi, Zheng-Xin Yong, Yifei He, Bobbie Chern, Han Zhao, Aobo Yang, Jianfeng Chi
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18210
• Pdf URL: https://arxiv.org/pdf/2410.18210
• Copy Paste: [[2410.18210]] Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks(https://arxiv.org/abs/2410.18210)
• Keywords: attack, large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have sparked widespread concerns about their safety. Recent work demonstrates that safety alignment of LLMs can be easily removed by fine-tuning with a few adversarially chosen instruction-following examples, i.e., fine-tuning attacks. We take a further step to understand fine-tuning attacks in multilingual LLMs. We first discover cross-lingual generalization of fine-tuning attacks: using a few adversarially chosen instruction-following examples in one language, multilingual LLMs can also be easily compromised (e.g., multilingual LLMs fail to refuse harmful prompts in other languages). Motivated by this finding, we hypothesize that safety-related information is language-agnostic and propose a new method termed Safety Information Localization (SIL) to identify the safety-related information in the model parameter space. Through SIL, we validate this hypothesis and find that only changing 20% of weight parameters in fine-tuning attacks can break safety alignment across all languages. Furthermore, we provide evidence to the alternative pathways hypothesis for why freezing safety-related parameters does not prevent fine-tuning attacks, and we demonstrate that our attack vector can still jailbreak LLMs adapted to new languages.

Title: MsMorph: An Unsupervised pyramid learning network for brain image registration

• Authors: Jiaofen Nan, Gaodeng Fan, Kaifan Zhang, Chen Zhao, Fubao Zhu, Weihua Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18228
• Pdf URL: https://arxiv.org/pdf/2410.18228
• Copy Paste: [[2410.18228]] MsMorph: An Unsupervised pyramid learning network for brain image registration(https://arxiv.org/abs/2410.18228)
• Keywords: interpretability
• Abstract: In the field of medical image analysis, image registration is a crucial technique. Despite the numerous registration models that have been proposed, existing methods still fall short in terms of accuracy and interpretability. In this paper, we present MsMorph, a deep learning-based image registration framework aimed at mimicking the manual process of registering image pairs to achieve more similar deformations, where the registered image pairs exhibit consistency or similarity in features. By extracting the feature differences between image pairs across various as-pects using gradients, the framework decodes semantic information at different scales and continuously compen-sates for the predicted deformation field, driving the optimization of parameters to significantly improve registration accuracy. The proposed method simulates the manual approach to registration, focusing on different regions of the image pairs and their neighborhoods to predict the deformation field between the two images, which provides strong interpretability. We compared several existing registration methods on two public brain MRI datasets, including LPBA and Mindboggle. The experimental results show that our method consistently outperforms state of the art in terms of metrics such as Dice score, Hausdorff distance, average symmetric surface distance, and non-Jacobian. The source code is publicly available at this https URL

Title: DMTG: A Human-Like Mouse Trajectory Generation Bot Based on Entropy-Controlled Diffusion Networks

• Authors: Jiahua Liu, Zeyuan Cui, Wenhan Ge, Pengxiang Zhan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18233
• Pdf URL: https://arxiv.org/pdf/2410.18233
• Copy Paste: [[2410.18233]] DMTG: A Human-Like Mouse Trajectory Generation Bot Based on Entropy-Controlled Diffusion Networks(https://arxiv.org/abs/2410.18233)
• Keywords: protect, attack, diffusion
• Abstract: CAPTCHAs protect against resource misuse and data theft by distinguishing human activity from automated bots. Advances in machine learning have made traditional image and text-based CAPTCHAs vulnerable to attacks, leading modern CAPTCHAs, such as GeeTest and Akamai, to incorporate behavioral analysis like mouse trajectory detection. Existing bypass techniques struggle to fully mimic human behavior, making it difficult to evaluate the effectiveness of anti-bot measures. To address this, we propose a diffusion model-based mouse trajectory generation framework (DMTG), which controls trajectory complexity and produces realistic human-like mouse movements. DMTG also provides white-box and black-box testing methods to assess its ability to bypass CAPTCHA systems. In experiments, DMTG reduces bot detection accuracy by 4.75%-9.73% compared to other models. Additionally, it mimics physical human behaviors, such as slow initiation and directional force differences, demonstrating improved performance in both simulation and real-world CAPTCHA scenarios.

Title: CARLA2Real: a tool for reducing the sim2real gap in CARLA simulator

• Authors: Stefanos Pasios, Nikos Nikolaidis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18238
• Pdf URL: https://arxiv.org/pdf/2410.18238
• Copy Paste: [[2410.18238]] CARLA2Real: a tool for reducing the sim2real gap in CARLA simulator(https://arxiv.org/abs/2410.18238)
• Keywords: extraction, segmentation
• Abstract: Simulators are indispensable for research in autonomous systems such as self-driving cars, autonomous robots and drones. Despite significant progress in various simulation aspects, such as graphical realism, an evident gap persists between the virtual and real-world environments. Since the ultimate goal is to deploy the autonomous systems in the real world, closing the sim2real gap is of utmost importance. In this paper, we employ a state-ofthe-art approach to enhance the photorealism of simulated data, aligning them with the visual characteristics of real-world datasets. Based on this, we developed CARLA2Real, an easy-to-use, publicly available tool (plug-in) for the widely used and open-source CARLA simulator. This tool enhances the output of CARLA in near realtime, achieving a frame rate of 13 FPS, translating it to the visual style and realism of real-world datasets such as Cityscapes, KITTI, and Mapillary Vistas. By employing the proposed tool, we generated synthetic datasets from both the simulator and the enhancement model outputs, including their corresponding ground truth annotations for tasks related to autonomous driving. Then, we performed a number of experiments to evaluate the impact of the proposed approach on feature extraction and semantic segmentation methods when trained on the enhanced synthetic data. The results demonstrate that the sim2real gap is significant and can indeed be reduced by the introduced approach.

Title: Efficient Inference for Augmented Large Language Models

• Authors: Rana Shahout, Cong Liang, Shiji Xin, Qianru Lao, Yong Cui, Minlan Yu, Michael Mitzenmacher
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18248
• Pdf URL: https://arxiv.org/pdf/2410.18248
• Copy Paste: [[2410.18248]] Efficient Inference for Augmented Large Language Models(https://arxiv.org/abs/2410.18248)
• Keywords: large language model
• Abstract: Augmented Large Language Models (LLMs) enhance the capabilities of standalone LLMs by integrating external data sources through API calls. In interactive LLM applications, efficient scheduling is crucial for maintaining low request completion times, directly impacting user engagement. However, these augmentations introduce scheduling challenges due to the need to manage limited memory for cached information (KV caches). As a result, traditional size-based scheduling algorithms, such as Shortest Job First (SJF), become less effective at minimizing completion times. Existing work focuses only on handling requests during API calls by preserving, discarding, or swapping memory without considering how to schedule requests with API calls. In this paper, we propose LAMPS, a novel LLM inference framework for augmented LLMs. LAMPS minimizes request completion time through a unified scheduling approach that considers the total length of requests and their handling strategies during API calls. Recognizing that LLM inference is memory-bound, our approach ranks requests based on their consumption of memory over time, which depends on both the output sizes and how a request is managed during its API calls. To implement our scheduling, LAMPS predicts the strategy that minimizes memory waste of a request during its API calls, aligning with but improving upon existing approaches. We also propose starvation prevention techniques and optimizations to mitigate the overhead of our scheduling. We implement LAMPS on top of vLLM and evaluate its performance against baseline LLM inference systems, demonstrating improvements in end-to-end latency by 27%-85% and reductions in TTFT by 4%-96% compared to the existing augmented-LLM system, with even greater gains over vLLM.

Title: Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models

• Authors: Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, Aaron Courville
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18252
• Pdf URL: https://arxiv.org/pdf/2410.18252
• Copy Paste: [[2410.18252]] Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models(https://arxiv.org/abs/2410.18252)
• Keywords: robust, large language model
• Abstract: The dominant paradigm for RLHF is online and on-policy RL: synchronously generating from the large language model (LLM) policy, labelling with a reward model, and learning using feedback on the LLM's own outputs. While performant, this paradigm is computationally inefficient. Inspired by classical deep RL literature, we propose separating generation and learning in RLHF. This enables asynchronous generation of new samples while simultaneously training on old samples, leading to faster training and more compute-optimal scaling. However, asynchronous training relies on an underexplored regime, online but off-policy RLHF: learning on samples from previous iterations of our model. To understand the challenges in this regime, we investigate a fundamental question: how much off-policyness can we tolerate for asynchronous training to speed up learning but maintain performance? Among several RLHF algorithms we tested, we find that online DPO is most robust to off-policy data, and robustness increases with the scale of the policy model. We study further compute optimizations for asynchronous RLHF but find that they come at a performance cost, giving rise to a trade-off. Finally, we verify the scalability of asynchronous RLHF by training LLaMA 3.1 8B on an instruction-following task 40% faster than a synchronous run while matching final performance.

Title: Multilingual Hallucination Gaps in Large Language Models

• Authors: Cléa Chataigner, Afaf Taïk, Golnoosh Farnadi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18270
• Pdf URL: https://arxiv.org/pdf/2410.18270
• Copy Paste: [[2410.18270]] Multilingual Hallucination Gaps in Large Language Models(https://arxiv.org/abs/2410.18270)
• Keywords: large language model
• Abstract: Large language models (LLMs) are increasingly used as alternatives to traditional search engines given their capacity to generate text that resembles human language. However, this shift is concerning, as LLMs often generate hallucinations, misleading or false information that appears highly credible. In this study, we explore the phenomenon of hallucinations across multiple languages in freeform text generation, focusing on what we call multilingual hallucination gaps. These gaps reflect differences in the frequency of hallucinated answers depending on the prompt and language used. To quantify such hallucinations, we used the FactScore metric and extended its framework to a multilingual setting. We conducted experiments using LLMs from the LLaMA, Qwen, and Aya families, generating biographies in 19 languages and comparing the results to Wikipedia pages. Our results reveal variations in hallucination rates, especially between high and low resource languages, raising important questions about LLM multilingual performance and the challenges in evaluating hallucinations in multilingual freeform text generation.

Title: KhmerST: A Low-Resource Khmer Scene Text Detection and Recognition Benchmark

• Authors: Vannkinh Nom, Souhail Bakkali, Muhammad Muzzamil Luqman, Mickaël Coustaty, Jean-Marc Ogier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18277
• Pdf URL: https://arxiv.org/pdf/2410.18277
• Copy Paste: [[2410.18277]] KhmerST: A Low-Resource Khmer Scene Text Detection and Recognition Benchmark(https://arxiv.org/abs/2410.18277)
• Keywords: robust
• Abstract: Developing effective scene text detection and recognition models hinges on extensive training data, which can be both laborious and costly to obtain, especially for low-resourced languages. Conventional methods tailored for Latin characters often falter with non-Latin scripts due to challenges like character stacking, diacritics, and variable character widths without clear word boundaries. In this paper, we introduce the first Khmer scene-text dataset, featuring 1,544 expert-annotated images, including 997 indoor and 547 outdoor scenes. This diverse dataset includes flat text, raised text, poorly illuminated text, distant and partially obscured text. Annotations provide line-level text and polygonal bounding box coordinates for each scene. The benchmark includes baseline models for scene-text detection and recognition tasks, providing a robust starting point for future research endeavors. The KhmerST dataset is publicly accessible at this https URL.

Title: Augmenting Training Data with Vector-Quantized Variational Autoencoder for Classifying RF Signals

• Authors: Srihari Kamesh Kompella, Kemal Davaslioglu, Yalin E. Sagduyu, Sastry Kompella
• Subjects: cs.LG, cs.NI, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.18283
• Pdf URL: https://arxiv.org/pdf/2410.18283
• Copy Paste: [[2410.18283]] Augmenting Training Data with Vector-Quantized Variational Autoencoder for Classifying RF Signals(https://arxiv.org/abs/2410.18283)
• Keywords: secure, robust
• Abstract: Radio frequency (RF) communication has been an important part of civil and military communication for decades. With the increasing complexity of wireless environments and the growing number of devices sharing the spectrum, it has become critical to efficiently manage and classify the signals that populate these frequencies. In such scenarios, the accurate classification of wireless signals is essential for effective spectrum management, signal interception, and interference mitigation. However, the classification of wireless RF signals often faces challenges due to the limited availability of labeled training data, especially under low signal-to-noise ratio (SNR) conditions. To address these challenges, this paper proposes the use of a Vector-Quantized Variational Autoencoder (VQ-VAE) to augment training data, thereby enhancing the performance of a baseline wireless classifier. The VQ-VAE model generates high-fidelity synthetic RF signals, increasing the diversity and fidelity of the training dataset by capturing the complex variations inherent in RF communication signals. Our experimental results show that incorporating VQ-VAE-generated data significantly improves the classification accuracy of the baseline model, particularly in low SNR conditions. This augmentation leads to better generalization and robustness of the classifier, overcoming the constraints imposed by limited real-world data. By improving RF signal classification, the proposed approach enhances the efficacy of wireless communication in both civil and tactical settings, ensuring reliable and secure operations. This advancement supports critical decision-making and operational readiness in environments where communication fidelity is essential.

Title: LEGO: Language Model Building Blocks

• Authors: Shrenik Bhansali, Alwin Jin, Tyler Lizzo, Larry Heck
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18287
• Pdf URL: https://arxiv.org/pdf/2410.18287
• Copy Paste: [[2410.18287]] LEGO: Language Model Building Blocks(https://arxiv.org/abs/2410.18287)
• Keywords: privacy, robust, federate, large language model
• Abstract: Large language models (LLMs) are essential in natural language processing (NLP) but are costly in data collection, pre-training, fine-tuning, and inference. Task-specific small language models (SLMs) offer a cheaper alternative but lack robustness and generalization. This paper proposes LEGO, a novel technique to extract SLMs from an LLM and recombine them. Using state-of-the-art LLM pruning strategies, we can create task- and user-specific SLM building blocks that are efficient for fine-tuning and inference while also preserving user data privacy. LEGO utilizes Federated Learning and a novel aggregation scheme for the LLM reconstruction, maintaining robustness without high costs and preserving user data privacy. We experimentally demonstrate the versatility of LEGO, showing its ability to enable model heterogeneity and mitigate the effects of data heterogeneity while maintaining LLM robustness.

Title: Enhancing Enterprise Security with Zero Trust Architecture

• Authors: Mahmud Hasan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18291
• Pdf URL: https://arxiv.org/pdf/2410.18291
• Copy Paste: [[2410.18291]] Enhancing Enterprise Security with Zero Trust Architecture(https://arxiv.org/abs/2410.18291)
• Keywords: security, defense, attack, segmentation
• Abstract: Zero Trust Architecture (ZTA) represents a transformative approach to modern cybersecurity, directly addressing the shortcomings of traditional perimeter-based security models. With the rise of cloud computing, remote work, and increasingly sophisticated cyber threats, perimeter defenses have proven ineffective at mitigating risks, particularly those involving insider threats and lateral movement within networks. ZTA shifts the security paradigm by assuming that no user, device, or system can be trusted by default, requiring continuous verification and the enforcement of least privilege access for all entities. This paper explores the key components of ZTA, such as identity and access management (IAM), micro-segmentation, continuous monitoring, and behavioral analytics, and evaluates their effectiveness in reducing vulnerabilities across diverse sectors, including finance, healthcare, and technology. Through case studies and industry reports, the advantages of ZTA in mitigating insider threats and minimizing attack surfaces are discussed. Additionally, the paper addresses the challenges faced during ZTA implementation, such as scalability, integration complexity, and costs, while providing best practices for overcoming these obstacles. Lastly, future research directions focusing on emerging technologies like AI, machine learning, blockchain, and their integration into ZTA are examined to enhance its capabilities further.

Title: Robust and Explainable Depression Identification from Speech Using Vowel-Based Ensemble Learning Approaches

• Authors: Kexin Feng, Theodora Chaspari
• Subjects: cs.LG, cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.18298
• Pdf URL: https://arxiv.org/pdf/2410.18298
• Copy Paste: [[2410.18298]] Robust and Explainable Depression Identification from Speech Using Vowel-Based Ensemble Learning Approaches(https://arxiv.org/abs/2410.18298)
• Keywords: robust, explainability
• Abstract: This study investigates explainable machine learning algorithms for identifying depression from speech. Grounded in evidence from speech production that depression affects motor control and vowel generation, pre-trained vowel-based embeddings, that integrate semantically meaningful linguistic units, are used. Following that, an ensemble learning approach decomposes the problem into constituent parts characterized by specific depression symptoms and severity levels. Two methods are explored: a "bottom-up" approach with 8 models predicting individual Patient Health Questionnaire-8 (PHQ-8) item scores, and a "top-down" approach using a Mixture of Experts (MoE) with a router module for assessing depression severity. Both methods depict performance comparable to state-of-the-art baselines, demonstrating robustness and reduced susceptibility to dataset mean/median values. System explainability benefits are discussed highlighting their potential to assist clinicians in depression diagnosis and screening.

Title: CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation

• Authors: Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, Yiran Chen
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18311
• Pdf URL: https://arxiv.org/pdf/2410.18311
• Copy Paste: [[2410.18311]] CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation(https://arxiv.org/abs/2410.18311)
• Keywords: large language model
• Abstract: Large language models (LLMs) with billions of parameters have sparked a new wave of exciting AI applications. However, their high computational costs and memory demands during inference pose significant challenges. Adaptive sparse activation inference, which activates only a small number of neurons for each token, offers a novel way to accelerate model inference without degrading performance, showing great potential for resource-constrained hardware devices. Nevertheless, existing methods predict activated neurons based on individual tokens with additional MLP, which involve frequent changes in activation maps and resource calls, limiting the acceleration benefits of sparse activation. In this paper, we introduce CoreInfer, an MLP-free adaptive sparse activation inference method based on sentence-level prediction. Specifically, we propose the concept of sentence-wise core neurons, which refers to the subset of neurons most critical for a given sentence, and empirically demonstrate its effectiveness. To determine the core neurons, we explore the correlation between core neurons and the sentence's semantics. Remarkably, we discovered that core neurons exhibit both stability and similarity in relation to the sentence's semantics -- an insight overlooked by previous studies. Building on this finding, we further design two semantic-based methods for predicting core neurons to fit different input scenarios. In CoreInfer, the core neurons are determined during the pre-filling stage and fixed during the encoding stage, enabling zero-cost sparse inference. We evaluated the model generalization and task generalization of CoreInfer across various models and tasks. Notably, on an NVIDIA TITAN XP GPU, CoreInfer achieved a 10.33 times and 2.72 times speedup compared to the Huggingface implementation and PowerInfer, respectively.

Title: Countering Autonomous Cyber Threats

• Authors: Kade M. Heckel, Adrian Weller
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.18312
• Pdf URL: https://arxiv.org/pdf/2410.18312
• Copy Paste: [[2410.18312]] Countering Autonomous Cyber Threats(https://arxiv.org/abs/2410.18312)
• Keywords: security, defense, attack, generative
• Abstract: With the capability to write convincing and fluent natural language and generate code, Foundation Models present dual-use concerns broadly and within the cyber domain specifically. Generative AI has already begun to impact cyberspace through a broad illicit marketplace for assisting malware development and social engineering attacks through hundreds of malicious-AI-as-a-services tools. More alarming is that recent research has shown the potential for these advanced models to inform or independently execute offensive cyberspace operations. However, these previous investigations primarily focused on the threats posed by proprietary models due to the until recent lack of strong open-weight model and additionally leave the impacts of network defenses or potential countermeasures unexplored. Critically, understanding the aptitude of downloadable models to function as offensive cyber agents is vital given that they are far more difficult to govern and prevent their misuse. As such, this work evaluates several state-of-the-art FMs on their ability to compromise machines in an isolated network and investigates defensive mechanisms to defeat such AI-powered attacks. Using target machines from a commercial provider, the most recently released downloadable models are found to be on par with a leading proprietary model at conducting simple cyber attacks with common hacking tools against known vulnerabilities. To mitigate such LLM-powered threats, defensive prompt injection (DPI) payloads for disrupting the malicious cyber agent's workflow are demonstrated to be effective. From these results, the implications for AI safety and governance with respect to cybersecurity is analyzed.

Title: AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models

• Authors: Kim Sung-Bin, Oh Hyun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, Tae-Hyun Oh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18325
• Pdf URL: https://arxiv.org/pdf/2410.18325
• Copy Paste: [[2410.18325]] AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models(https://arxiv.org/abs/2410.18325)
• Keywords: robust, large language model
• Abstract: Following the success of Large Language Models (LLMs), expanding their boundaries to new modalities represents a significant paradigm shift in multimodal understanding. Human perception is inherently multimodal, relying not only on text but also on auditory and visual cues for a complete understanding of the world. In recognition of this fact, audio-visual LLMs have recently emerged. Despite promising developments, the lack of dedicated benchmarks poses challenges for understanding and evaluating models. In this work, we show that audio-visual LLMs struggle to discern subtle relationships between audio and visual signals, leading to hallucinations, underscoring the need for reliable benchmarks. To address this, we introduce AVHBench, the first comprehensive benchmark specifically designed to evaluate the perception and comprehension capabilities of audio-visual LLMs. Our benchmark includes tests for assessing hallucinations, as well as the cross-modal matching and reasoning abilities of these models. Our results reveal that most existing audio-visual LLMs struggle with hallucinations caused by cross-interactions between modalities, due to their limited capacity to perceive complex multimodal signals and their relationships. Additionally, we demonstrate that simple training with our AVHBench improves robustness of audio-visual LLMs against hallucinations.

Title: Advancing Network Security: A Comprehensive Testbed and Dataset for Machine Learning-Based Intrusion Detection

• Authors: Talaya Farasat, JongWon Kim, Joachim Posegga
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2410.18332
• Pdf URL: https://arxiv.org/pdf/2410.18332
• Copy Paste: [[2410.18332]] Advancing Network Security: A Comprehensive Testbed and Dataset for Machine Learning-Based Intrusion Detection(https://arxiv.org/abs/2410.18332)
• Keywords: security
• Abstract: This paper introduces a Testbed designed for generating network traffic, leveraging the capabilities of containers, Kubernetes, and eBPF/XDP technologies. Our Testbed serves as an advanced platform for producing network traffic for machine learning based network experiments. By utilizing this Testbed, we offer small malicious network traffic dataset publically that satisfy ground truth property completely.

Title: Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems

• Authors: Junyi Ye, Jingyi Gu, Xinyun Zhao, Wenpeng Yin, Guiling Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18336
• Pdf URL: https://arxiv.org/pdf/2410.18336
• Copy Paste: [[2410.18336]] Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems(https://arxiv.org/abs/2410.18336)
• Keywords: large language model
• Abstract: The mathematical capabilities of AI systems are complex and multifaceted. Most existing research has predominantly focused on the correctness of AI-generated solutions to mathematical problems. In this work, we argue that beyond producing correct answers, AI systems should also be capable of, or assist humans in, developing novel solutions to mathematical challenges. This study explores the creative potential of Large Language Models (LLMs) in mathematical reasoning, an aspect that has received limited attention in prior research. We introduce a novel framework and benchmark, CreativeMath, which encompasses problems ranging from middle school curricula to Olympic-level competitions, designed to assess LLMs' ability to propose innovative solutions after some known solutions have been provided. Our experiments demonstrate that, while LLMs perform well on standard mathematical tasks, their capacity for creative problem-solving varies considerably. Notably, the Gemini-1.5-Pro model outperformed other LLMs in generating novel solutions. This research opens a new frontier in evaluating AI creativity, shedding light on both the strengths and limitations of LLMs in fostering mathematical innovation, and setting the stage for future developments in AI-assisted mathematical discovery.

Title: Aggregated Knowledge Model: Enhancing Domain-Specific QA with Fine-Tuned and Retrieval-Augmented Generation Models

• Authors: Fengchen Liu, Jordan Jung, Wei Feinstein, Jeff DAmbrogia, Gary Jung
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18344
• Pdf URL: https://arxiv.org/pdf/2410.18344
• Copy Paste: [[2410.18344]] Aggregated Knowledge Model: Enhancing Domain-Specific QA with Fine-Tuned and Retrieval-Augmented Generation Models(https://arxiv.org/abs/2410.18344)
• Keywords: large language model
• Abstract: This paper introduces a novel approach to enhancing closed-domain Question Answering (QA) systems, focusing on the specific needs of the Lawrence Berkeley National Laboratory (LBL) Science Information Technology (ScienceIT) domain. Utilizing a rich dataset derived from the ScienceIT documentation, our study embarks on a detailed comparison of two fine-tuned large language models and five retrieval-augmented generation (RAG) models. Through data processing techniques, we transform the documentation into structured context-question-answer triples, leveraging the latest Large Language Models (AWS Bedrock, GCP PaLM2, Meta LLaMA2, OpenAI GPT-4, Google Gemini-Pro) for data-driven insights. Additionally, we introduce the Aggregated Knowledge Model (AKM), which synthesizes responses from the seven models mentioned above using K-means clustering to select the most representative answers. The evaluation of these models across multiple metrics offers a comprehensive look into their effectiveness and suitability for the LBL ScienceIT environment. The results demonstrate the potential benefits of integrating fine-tuning and retrieval-augmented strategies, highlighting significant performance improvements achieved with the AKM. The insights gained from this study can be applied to develop specialized QA systems tailored to specific domains.

Title: AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability

• Authors: Sudhanshu Agrawal, Wonseok Jeon, Mingu Lee
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18351
• Pdf URL: https://arxiv.org/pdf/2410.18351
• Copy Paste: [[2410.18351]] AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability(https://arxiv.org/abs/2410.18351)
• Keywords: robust, large language model
• Abstract: Speculative decoding is a powerful technique that attempts to circumvent the autoregressive constraint of modern Large Language Models (LLMs). The aim of speculative decoding techniques is to improve the average inference time of a large, target model without sacrificing its accuracy, by using a more efficient draft model to propose draft tokens which are then verified in parallel. The number of draft tokens produced in each drafting round is referred to as the draft length and is often a static hyperparameter chosen based on the acceptance rate statistics of the draft tokens. However, setting a static draft length can negatively impact performance, especially in scenarios where drafting is expensive and there is a high variance in the number of tokens accepted. Adaptive Entropy-based Draft Length (AdaEDL) is a simple, training and parameter-free criteria which allows for early stopping of the token drafting process by approximating a lower bound on the expected acceptance probability of the drafted token based on the currently observed entropy of the drafted logits. We show that AdaEDL consistently outperforms static draft-length speculative decoding by 10%-57% as well as other training-free draft-stopping techniques by upto 10% in a variety of settings and datasets. At the same time, we show that AdaEDL is more robust than these techniques and preserves performance in high-sampling-temperature scenarios. Since it is training-free, in contrast to techniques that rely on the training of dataset-specific draft-stopping predictors, AdaEDL can seamlessly be integrated into a variety of pre-existing LLM systems.

Title: FedBaF: Federated Learning Aggregation Biased by a Foundation Model

• Authors: Jong-Ik Park, Srinivasa Pranav, José M. F. Moura, Carlee Joe-Wong
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.18352
• Pdf URL: https://arxiv.org/pdf/2410.18352
• Copy Paste: [[2410.18352]] FedBaF: Federated Learning Aggregation Biased by a Foundation Model(https://arxiv.org/abs/2410.18352)
• Keywords: security, privacy, federate, transformer
• Abstract: Foundation models are now a major focus of leading technology organizations due to their ability to generalize across diverse tasks. Existing approaches for adapting foundation models to new applications often rely on Federated Learning (FL) and disclose the foundation model weights to clients when using it to initialize the global model. While these methods ensure client data privacy, they compromise model and information security. In this paper, we introduce Federated Learning Aggregation Biased by a Foundation Model (FedBaF), a novel method for dynamically integrating pre-trained foundation model weights during the FL aggregation phase. Unlike conventional methods, FedBaF preserves the confidentiality of the foundation model while still leveraging its power to train more accurate models, especially in non-IID and adversarial scenarios. Our comprehensive experiments use Pre-ResNet and foundation models like Vision Transformer to demonstrate that FedBaF not only matches, but often surpasses the test accuracy of traditional weight initialization methods by up to 11.4\% in IID and up to 15.8\% in non-IID settings. Additionally, FedBaF applied to a Transformer-based language model significantly reduced perplexity by up to 39.2\%.

Title: Precision Soil Quality Analysis Using Transformer-based Data Fusion Strategies: A Systematic Review

• Authors: Mahdi Saki, Rasool Keshavarz, Daniel Franklin, Mehran Abolhasan, Justin Lipman, Negin Shariati
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18353
• Pdf URL: https://arxiv.org/pdf/2410.18353
• Copy Paste: [[2410.18353]] Precision Soil Quality Analysis Using Transformer-based Data Fusion Strategies: A Systematic Review(https://arxiv.org/abs/2410.18353)
• Keywords: transformer
• Abstract: This review explores the most recent advancements in transformer-based data fusion techniques in agricultural remote sensing (RS), with a particular focus on soil analysis. Utilizing a systematic, data-driven approach, we demonstrate that transformers have significantly outperformed conventional deep learning and machine learning methods since 2022, achieving prediction performance between 92% and 97%. The review is specifically focused on soil analysis, due to the importance of soil condition in optimizing crop productivity and ensuring sustainable farming practices. Transformer-based models have shown remarkable capabilities in handling complex multivariate soil data, improving the accuracy of soil moisture prediction, soil element analysis, and other soil-related applications. This systematic review primarily focuses on 1) analysing research trends and patterns in the literature, both chronologically and technically, and 2) conducting a comparative analysis of data fusion approaches, considering factors such as data types, techniques, and RS applications. Finally, we propose a roadmap for implementing data fusion methods in agricultural RS.

Title: Multi-objective Optimization in CPU Design Space Exploration: Attention is All You Need

• Authors: Runzhen Xue, Hao Wu, Mingyu Yan, Ziheng Xiao, Xiaochun Ye, Dongrui Fan
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2410.18368
• Pdf URL: https://arxiv.org/pdf/2410.18368
• Copy Paste: [[2410.18368]] Multi-objective Optimization in CPU Design Space Exploration: Attention is All You Need(https://arxiv.org/abs/2410.18368)
• Keywords: interpretability
• Abstract: Design space exploration (DSE) enables architects to systematically evaluate various design options, guiding decisions on the most suitable configurations to meet specific objectives such as optimizing performance, power, and area. However, the growing complexity of modern CPUs has dramatically increased the number of micro-architectural parameters and expanded the overall design space, making DSE more challenging and time-consuming. Existing DSE frameworks struggle in large-scale design spaces due to inaccurate models and limited insights into parameter impact, hindering efficient identification of optimal micro-architectures within tight timeframes. In this work, we introduce AttentionDSE. Its key idea is to use the attention mechanism to establish a direct mapping of micro-architectural parameters to their contributions to predicted performance. This approach enhances both the prediction accuracy and interpretability of the performance model. Furthermore, the weights are dynamically adjusted, enabling the model to respond to design changes and effectively pinpoint the key micro-architectural parameters/components responsible for performance bottlenecks. Thus, AttentionDSE accurately, purposefully, and rapidly discovers optimal designs. Experiments on SPEC 2017 demonstrate that AttentionDSE significantly reduces exploration time by over 80\% and achieves 3.9\% improvement in Pareto Hypervolume compared to state-of-the-art DSE frameworks while maintaining superior prediction accuracy and efficiency with an increasing number of parameters.

Title: Delta: A Cloud-assisted Data Enrichment Framework for On-Device Continual Learning

• Authors: Chen Gong, Zhenzhe Zheng, Fan Wu, Xiaofeng Jia, Guihai Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18378
• Pdf URL: https://arxiv.org/pdf/2410.18378
• Copy Paste: [[2410.18378]] Delta: A Cloud-assisted Data Enrichment Framework for On-Device Continual Learning(https://arxiv.org/abs/2410.18378)
• Keywords: federate
• Abstract: In modern mobile applications, users frequently encounter various new contexts, necessitating on-device continual learning (CL) to ensure consistent model performance. While existing research predominantly focused on developing lightweight CL frameworks, we identify that data scarcity is a critical bottleneck for on-device CL. In this work, we explore the potential of leveraging abundant cloud-side data to enrich scarce on-device data, and propose a private, efficient and effective data enrichment framework Delta. Specifically, Delta first introduces a directory dataset to decompose the data enrichment problem into device-side and cloud-side sub-problems without sharing sensitive data. Next, Delta proposes a soft data matching strategy to effectively solve the device-side sub-problem with sparse user data, and an optimal data sampling scheme for cloud server to retrieve the most suitable dataset for enrichment with low computational complexity. Further, Delta refines the data sampling scheme by jointly considering the impact of enriched data on both new and past contexts, mitigating the catastrophic forgetting issue from a new aspect. Comprehensive experiments across four typical mobile computing tasks with varied data modalities demonstrate that Delta could enhance the overall model accuracy by an average of 15.1%, 12.4%, 1.1% and 5.6% for visual, IMU, audio and textual tasks compared with few-shot CL, and consistently reduce the communication costs by over 90% compared to federated CL.

Title: Harnessing PU Learning for Enhanced Cloud-based DDoS Detection: A Comparative Analysis

• Authors: Robert Dilworth, Charan Gudla
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18380
• Pdf URL: https://arxiv.org/pdf/2410.18380
• Copy Paste: [[2410.18380]] Harnessing PU Learning for Enhanced Cloud-based DDoS Detection: A Comparative Analysis(https://arxiv.org/abs/2410.18380)
• Keywords: security, robust
• Abstract: This paper explores the application of Positive-Unlabeled (PU) learning for enhanced Distributed Denial-of-Service (DDoS) detection in cloud environments. Utilizing the $\texttt{BCCC-cPacket-Cloud-DDoS-2024}$ dataset, we implement PU learning with four machine learning algorithms: XGBoost, Random Forest, Support Vector Machine, and Naïve Bayes. Our results demonstrate the superior performance of ensemble methods, with XGBoost and Random Forest achieving $F_{1}$ scores exceeding 98%. We quantify the efficacy of each approach using metrics including $F_{1}$ score, ROC AUC, Recall, and Precision. This study bridges the gap between PU learning and cloud-based anomaly detection, providing a foundation for addressing Context-Aware DDoS Detection in multi-cloud environments. Our findings highlight the potential of PU learning in scenarios with limited labeled data, offering valuable insights for developing more robust and adaptive cloud security mechanisms.

Title: Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

• Authors: Lehan Wang, Haonan Wang, Honglong Yang, Jiaji Mao, Zehong Yang, Jun Shen, Xiaomeng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18387
• Pdf URL: https://arxiv.org/pdf/2410.18387
• Copy Paste: [[2410.18387]] Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks(https://arxiv.org/abs/2410.18387)
• Keywords: interpretability, large language model
• Abstract: Several medical Multimodal Large Languange Models (MLLMs) have been developed to address tasks involving visual images with textual instructions across various medical modalities, achieving impressive results. Most current medical generalist models are region-agnostic, treating the entire image as a holistic representation. However, they struggle to identify which specific regions they are focusing on when generating a this http URL mimic the behavior of doctors, who typically begin by reviewing the entire image before concentrating on specific regions for a thorough evaluation, we aim to enhance the capability of medical MLLMs in understanding anatomical regions within entire medical scans. To achieve it, we first formulate Region-Centric tasks and construct a large-scale dataset, MedRegInstruct, to incorporate regional information into training. Combining our collected dataset with other medical multimodal corpora for training, we propose a Region-Aware medical MLLM, MedRegA, which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level medical vision-language tasks across a broad range of modalities. Our MedRegA not only enables three region-centric tasks, but also achieves the best performance for visual question answering, report generation and medical image classification over 8 modalities, showcasing significant versatility. Experiments demonstrate that our model can not only accomplish powerful performance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans, boosting the interpretability and user interactivity of medical MLLMs. Our project page is this https URL.

Title: Monolingual and Multilingual Misinformation Detection for Low-Resource Languages: A Comprehensive Survey

• Authors: Xinyu Wang, Wenbo Zhang, Sarah Rajtmajer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18390
• Pdf URL: https://arxiv.org/pdf/2410.18390
• Copy Paste: [[2410.18390]] Monolingual and Multilingual Misinformation Detection for Low-Resource Languages: A Comprehensive Survey(https://arxiv.org/abs/2410.18390)
• Keywords: robust
• Abstract: In today's global digital landscape, misinformation transcends linguistic boundaries, posing a significant challenge for moderation systems. While significant advances have been made in misinformation detection, the focus remains largely on monolingual high-resource contexts, with low-resource languages often overlooked. This survey aims to bridge that gap by providing a comprehensive overview of the current research on low-resource language misinformation detection in both monolingual and multilingual settings. We review the existing datasets, methodologies, and tools used in these domains, identifying key challenges related to: data resources, model development, cultural and linguistic context, real-world applications, and research efforts. We also examine emerging approaches, such as language-agnostic models and multi-modal techniques, while emphasizing the need for improved data collection practices, interdisciplinary collaboration, and stronger incentives for socially responsible AI research. Our findings underscore the need for robust, inclusive systems capable of addressing misinformation across diverse linguistic and cultural contexts.

Title: Faster Algorithms for User-Level Private Stochastic Convex Optimization

• Authors: Andrew Lowy, Daogao Liu, Hilal Asi
• Subjects: cs.LG, cs.CR, math.OC
• Abstract URL: https://arxiv.org/abs/2410.18391
• Pdf URL: https://arxiv.org/pdf/2410.18391
• Copy Paste: [[2410.18391]] Faster Algorithms for User-Level Private Stochastic Convex Optimization(https://arxiv.org/abs/2410.18391)
• Keywords: privacy, protect
• Abstract: We study private stochastic convex optimization (SCO) under user-level differential privacy (DP) constraints. In this setting, there are $n$ users (e.g., cell phones), each possessing $m$ data items (e.g., text messages), and we need to protect the privacy of each user's entire collection of data items. Existing algorithms for user-level DP SCO are impractical in many large-scale machine learning scenarios because: (i) they make restrictive assumptions on the smoothness parameter of the loss function and require the number of users to grow polynomially with the dimension of the parameter space; or (ii) they are prohibitively slow, requiring at least $(mn)^{3/2}$ gradient computations for smooth losses and $(mn)^3$ computations for non-smooth losses. To address these limitations, we provide novel user-level DP algorithms with state-of-the-art excess risk and runtime guarantees, without stringent assumptions. First, we develop a linear-time algorithm with state-of-the-art excess risk (for a non-trivial linear-time algorithm) under a mild smoothness assumption. Our second algorithm applies to arbitrary smooth losses and achieves optimal excess risk in $\approx (mn)^{9/8}$ gradient computations. Third, for non-smooth loss functions, we obtain optimal excess risk in $n^{11/8} m^{5/4}$ gradient computations. Moreover, our algorithms do not require the number of users to grow polynomially with the dimension.

Title: SPEED++: A Multilingual Event Extraction Framework for Epidemic Prediction and Preparedness

• Authors: Tanmay Parekh, Jeffrey Kwan, Jiarui Yu, Sparsh Johri, Hyosang Ahn, Sreya Muppalla, Kai-Wei Chang, Wei Wang, Nanyun Peng
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2410.18393
• Pdf URL: https://arxiv.org/pdf/2410.18393
• Copy Paste: [[2410.18393]] SPEED++: A Multilingual Event Extraction Framework for Epidemic Prediction and Preparedness(https://arxiv.org/abs/2410.18393)
• Keywords: extraction
• Abstract: Social media is often the first place where communities discuss the latest societal trends. Prior works have utilized this platform to extract epidemic-related information (e.g. infections, preventive measures) to provide early warnings for epidemic prediction. However, these works only focused on English posts, while epidemics can occur anywhere in the world, and early discussions are often in the local, non-English languages. In this work, we introduce the first multilingual Event Extraction (EE) framework SPEED++ for extracting epidemic event information for a wide range of diseases and languages. To this end, we extend a previous epidemic ontology with 20 argument roles; and curate our multilingual EE dataset SPEED++ comprising 5.1K tweets in four languages for four diseases. Annotating data in every language is infeasible; thus we develop zero-shot cross-lingual cross-disease models (i.e., training only on English COVID data) utilizing multilingual pre-training and show their efficacy in extracting epidemic-related events for 65 diverse languages across different diseases. Experiments demonstrate that our framework can provide epidemic warnings for COVID-19 in its earliest stages in Dec 2019 (3 weeks before global discussions) from Chinese Weibo posts without any training in Chinese. Furthermore, we exploit our framework's argument extraction capabilities to aggregate community epidemic discussions like symptoms and cure measures, aiding misinformation detection and public attention monitoring. Overall, we lay a strong foundation for multilingual epidemic preparedness.

Title: A contrastive-learning approach for auditory attention detection

• Authors: Seyed Ali Alavi Bajestan, Mark Pitt, Donald S. Williamson
• Subjects: cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.18395
• Pdf URL: https://arxiv.org/pdf/2410.18395
• Copy Paste: [[2410.18395]] A contrastive-learning approach for auditory attention detection(https://arxiv.org/abs/2410.18395)
• Keywords: robust
• Abstract: Carrying conversations in multi-sound environments is one of the more challenging tasks, since the sounds overlap across time and frequency making it difficult to understand a single sound source. One proposed approach to help isolate an attended speech source is through decoding the electroencephalogram (EEG) and identifying the attended audio source using statistical or machine learning techniques. However, the limited amount of data in comparison to other machine learning problems and the distributional shift between different EEG recordings emphasizes the need for a self supervised approach that works with limited data to achieve a more robust solution. In this paper, we propose a method based on self supervised learning to minimize the difference between the latent representations of an attended speech signal and the corresponding EEG signal. This network is further finetuned for the auditory attention classification task. We compare our results with previously published methods and achieve state-of-the-art performance on the validation set.

Title: You Only Look Around: Learning Illumination Invariant Feature for Low-light Object Detection

• Authors: Mingbo Hong, Shen Cheng, Haibin Huang, Haoqiang Fan, Shuaicheng Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18398
• Pdf URL: https://arxiv.org/pdf/2410.18398
• Copy Paste: [[2410.18398]] You Only Look Around: Learning Illumination Invariant Feature for Low-light Object Detection(https://arxiv.org/abs/2410.18398)
• Keywords: extraction
• Abstract: In this paper, we introduce YOLA, a novel framework for object detection in low-light scenarios. Unlike previous works, we propose to tackle this challenging problem from the perspective of feature learning. Specifically, we propose to learn illumination-invariant features through the Lambertian image formation model. We observe that, under the Lambertian assumption, it is feasible to approximate illumination-invariant feature maps by exploiting the interrelationships between neighboring color channels and spatially adjacent pixels. By incorporating additional constraints, these relationships can be characterized in the form of convolutional kernels, which can be trained in a detection-driven manner within a network. Towards this end, we introduce a novel module dedicated to the extraction of illumination-invariant features from low-light images, which can be easily integrated into existing object detection frameworks. Our empirical findings reveal significant improvements in low-light object detection tasks, as well as promising results in both well-lit and over-lit scenarios. Code is available at \url{this https URL}.

Title: Enhancing Feature-Specific Data Protection via Bayesian Coordinate Differential Privacy

• Authors: Maryam Aliakbarpour, Syomantak Chaudhuri, Thomas A. Courtade, Alireza Fallah, Michael I. Jordan
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.18404
• Pdf URL: https://arxiv.org/pdf/2410.18404
• Copy Paste: [[2410.18404]] Enhancing Feature-Specific Data Protection via Bayesian Coordinate Differential Privacy(https://arxiv.org/abs/2410.18404)
• Keywords: privacy, protect
• Abstract: Local Differential Privacy (LDP) offers strong privacy guarantees without requiring users to trust external parties. However, LDP applies uniform protection to all data features, including less sensitive ones, which degrades performance of downstream tasks. To overcome this limitation, we propose a Bayesian framework, Bayesian Coordinate Differential Privacy (BCDP), that enables feature-specific privacy quantification. This more nuanced approach complements LDP by adjusting privacy protection according to the sensitivity of each feature, enabling improved performance of downstream tasks without compromising privacy. We characterize the properties of BCDP and articulate its connections with standard non-Bayesian privacy frameworks. We further apply our BCDP framework to the problems of private mean estimation and ordinary least-squares regression. The BCDP-based approach obtains improved accuracy compared to a purely LDP-based approach, without compromising on privacy.

Title: MoMQ: Mixture-of-Experts Enhances Multi-Dialect Query Generation across Relational and Non-Relational Databases

• Authors: Zhisheng Lin, Yifu Liu, Zhiling Luo, Jinyang Gao, Yu Li
• Subjects: cs.CL, cs.AI, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18406
• Pdf URL: https://arxiv.org/pdf/2410.18406
• Copy Paste: [[2410.18406]] MoMQ: Mixture-of-Experts Enhances Multi-Dialect Query Generation across Relational and Non-Relational Databases(https://arxiv.org/abs/2410.18406)
• Keywords: robust, large language model
• Abstract: The improvement in translating natural language to structured query language (SQL) can be attributed to the advancements in large language models (LLMs). Open-source LLMs, tailored for specific database dialects such as MySQL, have shown great performance. However, cloud service providers are looking for a unified database manager service (e.g., Cosmos DB from Azure, Amazon Aurora from AWS, Lindorm from AlibabaCloud) that can support multiple dialects. This requirement has led to the concept of multi-dialect query generation, which presents challenges to LLMs. These challenges include syntactic differences among dialects and imbalanced data distribution across multiple dialects. To tackle these challenges, we propose MoMQ, a novel Mixture-of-Experts-based multi-dialect query generation framework across both relational and non-relational databases. MoMQ employs a dialect expert group for each dialect and a multi-level routing strategy to handle dialect-specific knowledge, reducing interference during query generation. Additionally, a shared expert group is introduced to address data imbalance, facilitating the transfer of common knowledge from high-resource dialects to low-resource ones. Furthermore, we have developed a high-quality multi-dialect query generation benchmark that covers relational and non-relational databases such as MySQL, PostgreSQL, Cypher for Neo4j, and nGQL for NebulaGraph. Extensive experiments have shown that MoMQ performs effectively and robustly even in resource-imbalanced scenarios.

Title: Scale Propagation Network for Generalizable Depth Completion

• Authors: Haotian Wang, Meng Yang, Xinhu Zheng, Gang Hua
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18408
• Pdf URL: https://arxiv.org/pdf/2410.18408
• Copy Paste: [[2410.18408]] Scale Propagation Network for Generalizable Depth Completion(https://arxiv.org/abs/2410.18408)
• Keywords: robust
• Abstract: Depth completion, inferring dense depth maps from sparse measurements, is crucial for robust 3D perception. Although deep learning based methods have made tremendous progress in this problem, these models cannot generalize well across different scenes that are unobserved in training, posing a fundamental limitation that yet to be overcome. A careful analysis of existing deep neural network architectures for depth completion, which are largely borrowing from successful backbones for image analysis tasks, reveals that a key design bottleneck actually resides in the conventional normalization layers. These normalization layers are designed, on one hand, to make training more stable, on the other hand, to build more visual invariance across scene scales. However, in depth completion, the scale is actually what we want to robustly estimate in order to better generalize to unseen scenes. To mitigate, we propose a novel scale propagation normalization (SP-Norm) method to propagate scales from input to output, and simultaneously preserve the normalization operator for easy convergence. More specifically, we rescale the input using learned features of a single-layer perceptron from the normalized input, rather than directly normalizing the input as conventional normalization layers. We then develop a new network architecture based on SP-Norm and the ConvNeXt V2 backbone. We explore the composition of various basic blocks and architectures to achieve superior performance and efficient inference for generalizable depth completion. Extensive experiments are conducted on six unseen datasets with various types of sparse depth maps, i.e., randomly sampled 0.1\%/1\%/10\% valid pixels, 4/8/16/32/64-line LiDAR points, and holes from Structured-Light. Our model consistently achieves the best accuracy with faster speed and lower memory when compared to state-of-the-art methods.

Title: FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling

• Authors: Zhengqiang Zhang, Ruihuang Li, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18410
• Pdf URL: https://arxiv.org/pdf/2410.18410
• Copy Paste: [[2410.18410]] FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling(https://arxiv.org/abs/2410.18410)
• Keywords: diffusion
• Abstract: While image generation with diffusion models has achieved a great success, generating images of higher resolution than the training size remains a challenging task due to the high computational cost. Current methods typically perform the entire sampling process at full resolution and process all frequency components simultaneously, contradicting with the inherent coarse-to-fine nature of latent diffusion models and wasting computations on processing premature high-frequency details at early diffusion stages. To address this issue, we introduce an efficient $\textbf{Fre}$quency-aware $\textbf{Ca}$scaded $\textbf{S}$ampling framework, $\textbf{FreCaS}$ in short, for higher-resolution image generation. FreCaS decomposes the sampling process into cascaded stages with gradually increased resolutions, progressively expanding frequency bands and refining the corresponding details. We propose an innovative frequency-aware classifier-free guidance (FA-CFG) strategy to assign different guidance strengths for different frequency components, directing the diffusion model to add new details in the expanded frequency domain of each stage. Additionally, we fuse the cross-attention maps of previous and current stages to avoid synthesizing unfaithful layouts. Experiments demonstrate that FreCaS significantly outperforms state-of-the-art methods in image quality and generation speed. In particular, FreCaS is about 2.86$\times$ and 6.07$\times$ faster than ScaleCrafter and DemoFusion in generating a 2048$\times$2048 image using a pre-trained SDXL model and achieves an FID$_b$ improvement of 11.6 and 3.7, respectively. FreCaS can be easily extended to more complex models such as SD3. The source code of FreCaS can be found at $\href{\text{this https URL}}{this https URL}$.

Title: Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains

• Authors: Kun Li, Tianhua Zhang, Xixin Wu, Hongyin Luo, James Glass, Helen Meng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18415
• Pdf URL: https://arxiv.org/pdf/2410.18415
• Copy Paste: [[2410.18415]] Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains(https://arxiv.org/abs/2410.18415)
• Keywords: robust, large language model
• Abstract: Knowledge Graphs (KGs) can serve as reliable knowledge sources for question answering (QA) due to their structured representation of knowledge. Existing research on the utilization of KG for large language models (LLMs) prevalently relies on subgraph retriever or iterative prompting, overlooking the potential synergy of LLMs' step-wise reasoning capabilities and KGs' structural nature. In this paper, we present DoG (Decoding on Graphs), a novel framework that facilitates a deep synergy between LLMs and KGs. We first define a concept, well-formed chain, which consists of a sequence of interrelated fact triplets on the KGs, starting from question entities and leading to answers. We argue that this concept can serve as a principle for making faithful and sound reasoning for KGQA. To enable LLMs to generate well-formed chains, we propose graph-aware constrained decoding, in which a constraint derived from the topology of the KG regulates the decoding process of the LLMs. This constrained decoding method ensures the generation of well-formed chains while making full use of the step-wise reasoning capabilities of LLMs. Based on the above, DoG, a training-free approach, is able to provide faithful and sound reasoning trajectories grounded on the KGs. Experiments across various KGQA tasks with different background KGs demonstrate that DoG achieves superior and robust performance. DoG also shows general applicability with various open-source LLMs.

Title: Large Language Models Reflect the Ideology of their Creators

• Authors: Maarten Buyl, Alexander Rogiers, Sander Noels, Iris Dominguez-Catena, Edith Heiter, Raphael Romero, Iman Johary, Alexandru-Cristian Mara, Jefrey Lijffijt, Tijl De Bie
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18417
• Pdf URL: https://arxiv.org/pdf/2410.18417
• Copy Paste: [[2410.18417]] Large Language Models Reflect the Ideology of their Creators(https://arxiv.org/abs/2410.18417)
• Keywords: large language model
• Abstract: Large language models (LLMs) are trained on vast amounts of data to generate natural language, enabling them to perform tasks like text summarization and question answering. These models have become popular in artificial intelligence (AI) assistants like ChatGPT and already play an influential role in how humans access information. However, the behavior of LLMs varies depending on their design, training, and use. In this paper, we uncover notable diversity in the ideological stance exhibited across different LLMs and languages in which they are accessed. We do this by prompting a diverse panel of popular LLMs to describe a large number of prominent and controversial personalities from recent world history, both in English and in Chinese. By identifying and analyzing moral assessments reflected in the generated descriptions, we find consistent normative differences between how the same LLM responds in Chinese compared to English. Similarly, we identify normative disagreements between Western and non-Western LLMs about prominent actors in geopolitical conflicts. Furthermore, popularly hypothesized disparities in political goals among Western models are reflected in significant normative differences related to inclusion, social inequality, and political scandals. Our results show that the ideological stance of an LLM often reflects the worldview of its creators. This raises important concerns around technological and regulatory efforts with the stated aim of making LLMs ideologically `unbiased', and it poses risks for political instrumentalization.

Title: Knowledge-Assisted Privacy Preserving in Semantic Communication

• Authors: Xuesong Liu, Yao Sun, Runze Cheng, Le Xia, Hanaa Abumarshoud, Lei Zhang, Muhammad Ali Imran
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18418
• Pdf URL: https://arxiv.org/pdf/2410.18418
• Copy Paste: [[2410.18418]] Knowledge-Assisted Privacy Preserving in Semantic Communication(https://arxiv.org/abs/2410.18418)
• Keywords: privacy, attack
• Abstract: Semantic communication (SC) offers promising advancements in data transmission efficiency and reliability by focusing on delivering true meaning rather than solely binary bits of messages. However, privacy concerns in SC might become outstanding. Eavesdroppers equipped with advanced semantic coding models and extensive knowledge could be capable of correctly decoding and reasoning sensitive semantics from just a few stolen bits. To this end, this article explores utilizing knowledge to enhance data privacy in SC networks. Specifically, we first identify the potential attacks in SC based on the analysis of knowledge. Then, we propose a knowledge-assisted privacy preserving SC framework, which consists of a data transmission layer for precisely encoding and decoding source messages, and a knowledge management layer responsible for injecting appropriate knowledge into the transmission pair. Moreover, we elaborate on the transceiver design in the proposed SC framework to explain how knowledge should be utilized properly. Finally, some challenges of the proposed SC framework are discussed to expedite the practical implementation.

Title: A Causal Graph-Enhanced Gaussian Process Regression for Modeling Engine-out NOx

• Authors: Shrenik Zinage, Ilias Bilionis, Peter Meckl
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18424
• Pdf URL: https://arxiv.org/pdf/2410.18424
• Copy Paste: [[2410.18424]] A Causal Graph-Enhanced Gaussian Process Regression for Modeling Engine-out NOx(https://arxiv.org/abs/2410.18424)
• Keywords: robust
• Abstract: The stringent regulatory requirements on nitrogen oxides (NOx) emissions from diesel compression ignition engines require accurate and reliable models for real-time monitoring and diagnostics. Although traditional methods such as physical sensors and virtual engine control module (ECM) sensors provide essential data, they are only used for estimation. Ubiquitous literature primarily focuses on deterministic models with little emphasis on capturing the uncertainties due to sensors. The lack of probabilistic frameworks restricts the applicability of these models for robust diagnostics. The objective of this paper is to develop and validate a probabilistic model to predict engine-out NOx emissions using Gaussian process regression. Our approach is as follows. We employ three variants of Gaussian process models: the first with a standard radial basis function kernel with input window, the second incorporating a deep kernel using convolutional neural networks to capture temporal dependencies, and the third enriching the deep kernel with a causal graph derived via graph convolutional networks. The causal graph embeds physics knowledge into the learning process. All models are compared against a virtual ECM sensor using both quantitative and qualitative metrics. We conclude that our model provides an improvement in predictive performance when using an input window and a deep kernel structure. Even more compelling is the further enhancement achieved by the incorporation of a causal graph into the deep kernel. These findings are corroborated across different validation datasets.

Title: Segmentation-aware Prior Assisted Joint Global Information Aggregated 3D Building Reconstruction

• Authors: Hongxin Peng, Yongjian Liao, Weijun Li, Chuanyu Fu, Guoxin Zhang, Ziquan Ding, Zijie Huang, Qiku Cao, Shuting Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18433
• Pdf URL: https://arxiv.org/pdf/2410.18433
• Copy Paste: [[2410.18433]] Segmentation-aware Prior Assisted Joint Global Information Aggregated 3D Building Reconstruction(https://arxiv.org/abs/2410.18433)
• Keywords: segmentation
• Abstract: Multi-View Stereo plays a pivotal role in civil engineering by facilitating 3D modeling, precise engineering surveying, quantitative analysis, as well as monitoring and maintenance. It serves as a valuable tool, offering high-precision and real-time spatial information crucial for various engineering projects. However, Multi-View Stereo algorithms encounter challenges in reconstructing weakly-textured regions within large-scale building scenes. In these areas, the stereo matching of pixels often fails, leading to inaccurate depth estimations. Based on the Segment Anything Model and RANSAC algorithm, we propose an algorithm that accurately segments weakly-textured regions and constructs their plane priors. These plane priors, combined with triangulation priors, form a reliable prior candidate set. Additionally, we introduce a novel global information aggregation cost function. This function selects optimal plane prior information based on global information in the prior candidate set, constrained by geometric consistency during the depth estimation update process. Experimental results on both the ETH3D benchmark dataset, aerial dataset, building dataset and real scenarios substantiate the superior performance of our method in producing 3D building models compared to other state-of-the-art methods. In summary, our work aims to enhance the completeness and density of 3D building reconstruction, carrying implications for broader applications in urban planning and virtual reality.

Title: Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching

• Authors: Seoyeon Kim, Huiseo Kim, Chanjun Park, Jinyoung Yeo, Dongha Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18436
• Pdf URL: https://arxiv.org/pdf/2410.18436
• Copy Paste: [[2410.18436]] Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching(https://arxiv.org/abs/2410.18436)
• Keywords: large language model
• Abstract: Code-switching (CS), a phenomenon where multilingual speakers alternate between languages in a discourse, can convey subtle cultural and linguistic nuances that can be otherwise lost in translation. Recent state-of-the-art multilingual large language models (LLMs) demonstrate excellent multilingual abilities in various aspects including understanding CS, but the power of CS in eliciting language-specific knowledge is yet to be discovered. Therefore, we investigate the effectiveness of code-switching on a wide range of multilingual LLMs in terms of knowledge activation, or the act of identifying and leveraging knowledge for reasoning. To facilitate the research, we first present EnKoQA, a synthetic English-Korean CS question-answering dataset. We provide a comprehensive analysis on a variety of multilingual LLMs by subdividing activation process into knowledge identification and knowledge leveraging. Our experiments demonstrate that compared to English text, CS can faithfully activate knowledge inside LLMs, especially on language-specific domains. In addition, the performance gap between CS and English is larger in models that show excellent monolingual abilities, suggesting that there exists a correlation with CS and Korean proficiency.

Title: The Nature of Mathematical Modeling and Probabilistic Optimization Engineering in Generative AI

• Authors: Fulu Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18441
• Pdf URL: https://arxiv.org/pdf/2410.18441
• Copy Paste: [[2410.18441]] The Nature of Mathematical Modeling and Probabilistic Optimization Engineering in Generative AI(https://arxiv.org/abs/2410.18441)
• Keywords: transformer, generative
• Abstract: In this paper, we give an in-depth analysis on the mathematical problem formulations and the probabilistic optimization explorations for some of the key components in Transformer model [33] in the field of generative AI. We explore and discuss some potential further enhancement for current state of the art methods for some key underlying technologies of generative AI models from algorithmic and probabilistic optimization perspective. In particular, we present an optimal solution for sub-word encoding (SWE) based on similar initial settings as that of byte-pair encoding (BPE) algorithm in [9] with similar objectives as that of WordPiece approach in [28, 31] to maximize the likelihood of the training data. We also present cross entropy optimization method to optimize hyperparameters for word2vec model [17]. In addition, we propose a factored combination of rotary positional encoding (RoPE) [32] and attention with linear biases (ALiBi) [23] with a harmonic series. We also present a probabilistic FlashAttention [6, 7] (PrFlashAttention) method with a probability distribution over block distances in the matrix to decide which block is likely to participate in a given round of attention computation while maintaining the lower triangle shape of the tensor for autoregressive language models by re-shaping the tensors. Finally, we present staircase adaptive quantization (SAQ) of key-value (KV) cache for multi-query attention (MQA) based on the framework presented in [16] to have gradual quantization degradation while achieving reasonable model quality and cost savings.

Title: ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesis

• Authors: Zezhong Wang, Xingshan Zeng, Weiwen Liu, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18447
• Pdf URL: https://arxiv.org/pdf/2410.18447
• Copy Paste: [[2410.18447]] ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesis(https://arxiv.org/abs/2410.18447)
• Keywords: large language model
• Abstract: Supervised fine-tuning (SFT) is a common method to enhance the tool calling capabilities of Large Language Models (LLMs), with the training data often being synthesized. The current data synthesis process generally involves sampling a set of tools, formulating a requirement based on these tools, and generating the call statements. However, tools sampled randomly lack relevance, making them difficult to combine and thus reducing the diversity of the data. Additionally, current work overlooks the coherence between turns of dialogues, leading to a gap between the synthesized data and real-world scenarios. To address these issues, we propose a Graph-based Sampling strategy to sample more relevant tool combinations, and a Planned-generation strategy to create plans that guide the synthesis of coherent dialogues. We integrate these two strategies and enable multiple agents to synthesize the dialogue data interactively, resulting in our tool-calling data synthesis pipeline ToolFlow. Data quality assessments demonstrate improvements in the naturalness and coherence of our synthesized dialogues. Finally, we apply SFT on LLaMA-3.1-8B using 8,000 synthetic dialogues generated with ToolFlow. Results show that the model achieves tool-calling performance comparable to or even surpassing GPT-4, while maintaining strong general capabilities.

Title: Integrating Deep Feature Extraction and Hybrid ResNet-DenseNet Model for Multi-Class Abnormality Detection in Endoscopic Images

• Authors: Aman Sagar, Preeti Mehta, Monika Shrivastva, Suchi Kumari
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18457
• Pdf URL: https://arxiv.org/pdf/2410.18457
• Copy Paste: [[2410.18457]] Integrating Deep Feature Extraction and Hybrid ResNet-DenseNet Model for Multi-Class Abnormality Detection in Endoscopic Images(https://arxiv.org/abs/2410.18457)
• Keywords: robust, extraction
• Abstract: This paper presents a deep learning framework for the multi-class classification of gastrointestinal abnormalities in Video Capsule Endoscopy (VCE) frames. The aim is to automate the identification of ten GI abnormality classes, including angioectasia, bleeding, and ulcers, thereby reducing the diagnostic burden on gastroenterologists. Utilizing an ensemble of DenseNet and ResNet architectures, the proposed model achieves an overall accuracy of 94\% across a well-structured dataset. Precision scores range from 0.56 for erythema to 1.00 for worms, with recall rates peaking at 98% for normal findings. This study emphasizes the importance of robust data preprocessing techniques, including normalization and augmentation, in enhancing model performance. The contributions of this work lie in developing an effective AI-driven tool that streamlines the diagnostic process in gastroenterology, ultimately improving patient care and clinical outcomes.

Title: Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities

• Authors: Chung-En Sun, Xiaodong Liu, Weiwei Yang, Tsui-Wei Weng, Hao Cheng, Aidan San, Michel Galley, Jianfeng Gao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18469
• Pdf URL: https://arxiv.org/pdf/2410.18469
• Copy Paste: [[2410.18469]] Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities(https://arxiv.org/abs/2410.18469)
• Keywords: attack, large language model
• Abstract: Recent research has shown that Large Language Models (LLMs) are vulnerable to automated jailbreak attacks, where adversarial suffixes crafted by algorithms appended to harmful queries bypass safety alignment and trigger unintended responses. Current methods for generating these suffixes are computationally expensive and have low Attack Success Rates (ASR), especially against well-aligned models like Llama2 and Llama3. To overcome these limitations, we introduce ADV-LLM, an iterative self-tuning process that crafts adversarial LLMs with enhanced jailbreak ability. Our framework significantly reduces the computational cost of generating adversarial suffixes while achieving nearly 100\% ASR on various open-source LLMs. Moreover, it exhibits strong attack transferability to closed-source models, achieving 99% ASR on GPT-3.5 and 49% ASR on GPT-4, despite being optimized solely on Llama3. Beyond improving jailbreak ability, ADV-LLM provides valuable insights for future safety alignment research through its ability to generate large datasets for studying LLM safety. Our code is available at: this https URL

Title: Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift

• Authors: Junbao Chen, Jingfeng Xue, Yong Wang, Zhenyan Liu, Lu Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18478
• Pdf URL: https://arxiv.org/pdf/2410.18478
• Copy Paste: [[2410.18478]] Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift(https://arxiv.org/abs/2410.18478)
• Keywords: federate
• Abstract: Data heterogeneity is one of the key challenges in federated learning, and many efforts have been devoted to tackling this problem. However, distributed concept drift with data heterogeneity, where clients may additionally experience different concept drifts, is a largely unexplored area. In this work, we focus on real drift, where the conditional distribution $P(Y|X)$ changes. We first study how distributed concept drift affects the model training and find that local classifier plays a critical role in drift adaptation. Moreover, to address data heterogeneity, we study the feature alignment under distributed concept drift, and find two factors that are crucial for feature alignment: the conditional distribution $P(Y|X)$ and the degree of data heterogeneity. Motivated by the above findings, we propose FedCCFA, a federated learning framework with classifier clustering and feature alignment. To enhance collaboration under distributed concept drift, FedCCFA clusters local classifiers at class-level and generates clustered feature anchors according to the clustering results. Assisted by these anchors, FedCCFA adaptively aligns clients' feature spaces based on the entropy of label distribution $P(Y)$, alleviating the inconsistency in feature space. Our results demonstrate that FedCCFA significantly outperforms existing methods under various concept drift settings. Code is available at this https URL.

Title: Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction

• Authors: Sergio Burdisso, Srikanth Madikeri, Petr Motlicek
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18481
• Pdf URL: https://arxiv.org/pdf/2410.18481
• Copy Paste: [[2410.18481]] Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction(https://arxiv.org/abs/2410.18481)
• Keywords: extraction, large language model
• Abstract: Efficiently deriving structured workflows from unannotated dialogs remains an underexplored and formidable challenge in computational linguistics. Automating this process could significantly accelerate the manual design of workflows in new domains and enable the grounding of large language models in domain-specific flowcharts, enhancing transparency and controllability. In this paper, we introduce Dialog2Flow (D2F) embeddings, which differ from conventional sentence embeddings by mapping utterances to a latent space where they are grouped according to their communicative and informative functions (i.e., the actions they represent). D2F allows for modeling dialogs as continuous trajectories in a latent space with distinct action-related regions. By clustering D2F embeddings, the latent space is quantized, and dialogs can be converted into sequences of region/action IDs, facilitating the extraction of the underlying workflow. To pre-train D2F, we build a comprehensive dataset by unifying twenty task-oriented dialog datasets with normalized per-turn action annotations. We also introduce a novel soft contrastive loss that leverages the semantic information of these actions to guide the representation learning process, showing superior performance compared to standard supervised contrastive loss. Evaluation against various sentence embeddings, including dialog-specific ones, demonstrates that D2F yields superior qualitative and quantitative results across diverse domains.

Title: FirmRCA: Towards Post-Fuzzing Analysis on ARM Embedded Firmware with Efficient Event-based Fault Localization

• Authors: Boyu Chang, Binbin Zhao, Qiao Zhang, Peiyu Liu, Yuan Tian, Raheem Beyah, Shouling Ji
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18483
• Pdf URL: https://arxiv.org/pdf/2410.18483
• Copy Paste: [[2410.18483]] FirmRCA: Towards Post-Fuzzing Analysis on ARM Embedded Firmware with Efficient Event-based Fault Localization(https://arxiv.org/abs/2410.18483)
• Keywords: security
• Abstract: While fuzzing has demonstrated its effectiveness in exposing vulnerabilities within embedded firmware, the discovery of crashing test cases is only the first step in improving the security of these critical systems. The subsequent fault localization process, which aims to precisely identify the root causes of observed crashes, is a crucial yet time-consuming post-fuzzing work. Unfortunately, the automated root cause analysis on embedded firmware crashes remains an underexplored area, which is challenging from several perspectives: (1) the fuzzing campaign towards the embedded firmware lacks adequate debugging mechanisms, making it hard to automatically extract essential runtime information for analysis; (2) the inherent raw binary nature of embedded firmware often leads to over-tainted and noisy suspicious instructions, which provides limited guidance for analysts in manually investigating the root cause and remediating the underlying vulnerability. To address these challenges, we design and implement FirmRCA, a practical fault localization framework tailored specifically for embedded firmware. FirmRCA introduces an event-based footprint collection approach to aid and significantly expedite reverse execution. Next, to solve the complicated memory alias problem, FirmRCA proposes a history-driven method by tracking data propagation through the execution trace, enabling precise identification of deep crash origins. Finally, FirmRCA proposes a novel strategy to highlight key instructions related to the root cause, providing practical guidance in the final investigation. We evaluate FirmRCA with both synthetic and real-world targets, including 41 crashing test cases across 17 firmware images. The results show that FirmRCA can effectively (92.7% success rate) identify the root cause of crashing test cases within the top 10 instructions.

Title: Synth4Seg -- Learning Defect Data Synthesis for Defect Segmentation using Bi-level Optimization

• Authors: Shancong Mou, Raviteja Vemulapalli, Shiyu Li, Yuxuan Liu, C Thomas, Meng Cao, Haoping Bai, Oncel Tuzel, Ping Huang, Jiulong Shan, Jianjun Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18490
• Pdf URL: https://arxiv.org/pdf/2410.18490
• Copy Paste: [[2410.18490]] Synth4Seg -- Learning Defect Data Synthesis for Defect Segmentation using Bi-level Optimization(https://arxiv.org/abs/2410.18490)
• Keywords: segmentation
• Abstract: Defect segmentation is crucial for quality control in advanced manufacturing, yet data scarcity poses challenges for state-of-the-art supervised deep learning. Synthetic defect data generation is a popular approach for mitigating data challenges. However, many current methods simply generate defects following a fixed set of rules, which may not directly relate to downstream task performance. This can lead to suboptimal performance and may even hinder the downstream task. To solve this problem, we leverage a novel bi-level optimization-based synthetic defect data generation framework. We use an online synthetic defect generation module grounded in the commonly-used Cut\&Paste framework, and adopt an efficient gradient-based optimization algorithm to solve the bi-level optimization problem. We achieve simultaneous training of the defect segmentation network, and learn various parameters of the data synthesis module by maximizing the validation performance of the trained defect segmentation network. Our experimental results on benchmark datasets under limited data settings show that the proposed bi-level optimization method can be used for learning the most effective locations for pasting synthetic defects thereby improving the segmentation performance by up to 18.3\% when compared to pasting defects at random locations. We also demonstrate up to 2.6\% performance gain by learning the importance weights for different augmentation-specific defect data sources when compared to giving equal importance to all the data sources.

Title: ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models

• Authors: Hengxiang Zhang, Hongfu Gao, Qiang Hu, Guanhua Chen, Lili Yang, Bingyi Jing, Hongxin Wei, Bing Wang, Haifeng Bai, Lei Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18491
• Pdf URL: https://arxiv.org/pdf/2410.18491
• Copy Paste: [[2410.18491]] ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models(https://arxiv.org/abs/2410.18491)
• Keywords: large language model
• Abstract: With the rapid development of Large language models (LLMs), understanding the capabilities of LLMs in identifying unsafe content has become increasingly important. While previous works have introduced several benchmarks to evaluate the safety risk of LLMs, the community still has a limited understanding of current LLMs' capability to recognize illegal and unsafe content in Chinese contexts. In this work, we present a Chinese safety benchmark (ChineseSafe) to facilitate research on the content safety of large language models. To align with the regulations for Chinese Internet content moderation, our ChineseSafe contains 205,034 examples across 4 classes and 10 sub-classes of safety issues. For Chinese contexts, we add several special types of illegal content: political sensitivity, pornography, and variant/homophonic words. Moreover, we employ two methods to evaluate the legal risks of popular LLMs, including open-sourced models and APIs. The results reveal that many LLMs exhibit vulnerability to certain types of safety issues, leading to legal risks in China. Our work provides a guideline for developers and researchers to facilitate the safety of LLMs. Our results are also available at this https URL.

Title: CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models

• Authors: Liangdong Wang, Bo-Wen Zhang, Chengwei Wu, Hanyu Zhao, Xiaofeng Shi, Shuhao Gu, Jijie Li, Quanyue Ma, TengFei Pan, Guang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18505
• Pdf URL: https://arxiv.org/pdf/2410.18505
• Copy Paste: [[2410.18505]] CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models(https://arxiv.org/abs/2410.18505)
• Keywords: large language model
• Abstract: We present CCI3.0-HQ (this https URL), a high-quality 500GB subset of the Chinese Corpora Internet 3.0 (CCI3.0)(this https URL), developed using a novel two-stage hybrid filtering pipeline that significantly enhances data quality. To evaluate its effectiveness, we trained a 0.5B parameter model from scratch on 100B tokens across various datasets, achieving superior performance on 10 benchmarks in a zero-shot setting compared to CCI3.0, SkyPile, and WanjuanV1. The high-quality filtering process effectively distills the capabilities of the Qwen2-72B-instruct model into a compact 0.5B model, attaining optimal F1 scores for Chinese web data classification. We believe this open-access dataset will facilitate broader access to high-quality language models.

Title: KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing

• Authors: Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, Zhi Chen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18517
• Pdf URL: https://arxiv.org/pdf/2410.18517
• Copy Paste: [[2410.18517]] KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing(https://arxiv.org/abs/2410.18517)
• Keywords: transformer, large language model
• Abstract: The development of large language models (LLMs) has significantly expanded model sizes, resulting in substantial GPU memory requirements during inference. The key and value storage of the attention map in the KV (key-value) cache accounts for more than 80\% of this memory consumption. Nowadays, most existing KV cache compression methods focus on intra-layer compression within a single Transformer layer but few works consider layer-wise compression. In this paper, we propose a plug-and-play method called \textit{KVSharer}, which shares the KV cache between layers to achieve layer-wise compression. Rather than intuitively sharing based on higher similarity, we discover a counterintuitive phenomenon: sharing dissimilar KV caches better preserves the model performance. Experiments show that \textit{KVSharer} can reduce KV cache computation by 30\%, thereby lowering memory consumption without significantly impacting model performance and it can also achieve at least 1.3 times generation acceleration. Additionally, we verify that \textit{KVSharer} is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory.

Title: Unsupervised semantic segmentation of urban high-density multispectral point clouds

• Authors: Oona Oinonen, Lassi Ruoppa, Josef Taher, Matti Lehtomäki, Leena Matikainen, Kirsi Karila, Teemu Hakala, Antero Kukko, Harri Kaartinen, Juha Hyyppä
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18520
• Pdf URL: https://arxiv.org/pdf/2410.18520
• Copy Paste: [[2410.18520]] Unsupervised semantic segmentation of urban high-density multispectral point clouds(https://arxiv.org/abs/2410.18520)
• Keywords: segmentation
• Abstract: The availability of highly accurate urban airborne laser scanning (ALS) data will increase rapidly in the future, especially as acquisition costs decrease, for example through the use of drones. Current challenges in data processing are related to the limited spectral information and low point density of most ALS datasets. Another challenge will be the growing need for annotated training data, frequently produced by manual processes, to enable semantic interpretation of point clouds. This study proposes to semantically segment new high-density (1200 points per square metre on average) multispectral ALS data with an unsupervised ground-aware deep clustering method GroupSP inspired by the unsupervised GrowSP algorithm. GroupSP divides the scene into superpoints as a preprocessing step. The neural network is trained iteratively by grouping the superpoints and using the grouping assignments as pseudo-labels. The predictions for the unseen data are given by over-segmenting the test set and mapping the predicted classes into ground truth classes manually or with automated majority voting. GroupSP obtained an overall accuracy (oAcc) of 97% and a mean intersection over union (mIoU) of 80%. When compared to other unsupervised semantic segmentation methods, GroupSP outperformed GrowSP and non-deep K-means. However, a supervised random forest classifier outperformed GroupSP. The labelling efforts in GroupSP can be minimal; it was shown, that the GroupSP can semantically segment seven urban classes (building, high vegetation, low vegetation, asphalt, rock, football field, and gravel) with oAcc of 95% and mIoU of 75% using only 0.004% of the available annotated points in the mapping assignment. Finally, the multispectral information was examined; adding each new spectral channel improved the mIoU. Additionally, echo deviation was valuable, especially when distinguishing ground-level classes.

Title: A Systematic Survey on Instructional Text: From Representation and Downstream NLP Tasks

• Authors: Abdulfattah Safa, Tamta Kapanadze, Arda Uzunoğlu, Gözde Gül Şahin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18529
• Pdf URL: https://arxiv.org/pdf/2410.18529
• Copy Paste: [[2410.18529]] A Systematic Survey on Instructional Text: From Representation and Downstream NLP Tasks(https://arxiv.org/abs/2410.18529)
• Keywords: large language model
• Abstract: Recent advances in large language models have demonstrated promising capabilities in following simple instructions through instruction tuning. However, real-world tasks often involve complex, multi-step instructions that remain challenging for current NLP systems. Despite growing interest in this area, there lacks a comprehensive survey that systematically analyzes the landscape of complex instruction understanding and processing. Through a systematic review of the literature, we analyze available resources, representation schemes, and downstream tasks related to instructional text. Our study examines 177 papers, identifying trends, challenges, and opportunities in this emerging field. We provide AI/NLP researchers with essential background knowledge and a unified view of various approaches to complex instruction understanding, bridging gaps between different research directions and highlighting future research opportunities.

Title: Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics

• Authors: Jinghao Hu, Yuhe Zhang, GuoHua Geng, Liuyuxin Yang, JiaRui Yan, Jingtao Cheng, YaDong Zhang, Kang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18537
• Pdf URL: https://arxiv.org/pdf/2410.18537
• Copy Paste: [[2410.18537]] Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics(https://arxiv.org/abs/2410.18537)
• Keywords: diffusion
• Abstract: Traditionally, style has been primarily considered in terms of artistic elements such as colors, brushstrokes, and lighting. However, identical semantic subjects, like people, boats, and houses, can vary significantly across different artistic traditions, indicating that style also encompasses the underlying semantics. Therefore, in this study, we propose a zero-shot scheme for image variation with coordinated semantics. Specifically, our scheme transforms the image-to-image problem into an image-to-text-to-image problem. The image-to-text operation employs vision-language models e.g., BLIP) to generate text describing the content of the input image, including the objects and their positions. Subsequently, the input style keyword is elaborated into a detailed description of this style and then merged with the content text using the reasoning capabilities of ChatGPT. Finally, the text-to-image operation utilizes a Diffusion model to generate images based on the text prompt. To enable the Diffusion model to accommodate more styles, we propose a fine-tuning strategy that injects text and style constraints into cross-attention. This ensures that the output image exhibits similar semantics in the desired style. To validate the performance of the proposed scheme, we constructed a benchmark comprising images of various styles and scenes and introduced two novel metrics. Despite its simplicity, our scheme yields highly plausible results in a zero-shot manner, particularly for generating stylized images with high-fidelity semantics.

Title: SMITE: Segment Me In TimE

• Authors: Amirhossein Alimohammadi, Sauradip Nag, Saeid Asgari Taghanaki, Andrea Tagliasacchi, Ghassan Hamarneh, Ali Mahdavi Amiri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18538
• Pdf URL: https://arxiv.org/pdf/2410.18538
• Copy Paste: [[2410.18538]] SMITE: Segment Me In TimE(https://arxiv.org/abs/2410.18538)
• Keywords: diffusion, segmentation
• Abstract: Segmenting an object in a video presents significant challenges. Each pixel must be accurately labelled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or a few sample images. In this paper, we address this issue by employing a pre-trained text to image diffusion model supplemented with an additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of-the-art alternatives.

Title: On Explaining with Attention Matrices

• Authors: Omar Naim, Nicholas Asher
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18541
• Pdf URL: https://arxiv.org/pdf/2410.18541
• Copy Paste: [[2410.18541]] On Explaining with Attention Matrices(https://arxiv.org/abs/2410.18541)
• Keywords: transformer
• Abstract: This paper explores the much discussed, possible explanatory link between attention weights (AW) in transformer models and predicted output. Contrary to intuition and early research on attention, more recent prior research has provided formal arguments and empirical evidence that AW are not explanatorily relevant. We show that the formal arguments are incorrect. We introduce and effectively compute efficient attention, which isolates the effective components of attention matrices in tasks and models in which AW play an explanatory role. We show that efficient attention has a causal role (provides minimally necessary and sufficient conditions) for predicting model output in NLP tasks requiring contextual information, and we show, contrary to [7], that efficient attention matrices are probability distributions and are effectively calculable. Thus, they should play an important part in the explanation of attention based model behavior. We offer empirical experiments in support of our method illustrating various properties of efficient attention with various metrics on four datasets.

Title: IMAN: An Adaptive Network for Robust NPC Mortality Prediction with Missing Modalities

• Authors: Yejing Huo, Guoheng Huang, Lianglun Cheng, Jianbin He, Xuhang Chen, Xiaochen Yuan, Guo Zhong, Chi-Man Pun
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18551
• Pdf URL: https://arxiv.org/pdf/2410.18551
• Copy Paste: [[2410.18551]] IMAN: An Adaptive Network for Robust NPC Mortality Prediction with Missing Modalities(https://arxiv.org/abs/2410.18551)
• Keywords: robust, transformer
• Abstract: Accurate prediction of mortality in nasopharyngeal carcinoma (NPC), a complex malignancy particularly challenging in advanced stages, is crucial for optimizing treatment strategies and improving patient outcomes. However, this predictive process is often compromised by the high-dimensional and heterogeneous nature of NPC-related data, coupled with the pervasive issue of incomplete multi-modal data, manifesting as missing radiological images or incomplete diagnostic reports. Traditional machine learning approaches suffer significant performance degradation when faced with such incomplete data, as they fail to effectively handle the high-dimensionality and intricate correlations across modalities. Even advanced multi-modal learning techniques like Transformers struggle to maintain robust performance in the presence of missing modalities, as they lack specialized mechanisms to adaptively integrate and align the diverse data types, while also capturing nuanced patterns and contextual relationships within the complex NPC data. To address these problem, we introduce IMAN: an adaptive network for robust NPC mortality prediction with missing modalities.

Title: Complexity Matters: Effective Dimensionality as a Measure for Adversarial Robustness

• Authors: David Khachaturov, Robert Mullins
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18556
• Pdf URL: https://arxiv.org/pdf/2410.18556
• Copy Paste: [[2410.18556]] Complexity Matters: Effective Dimensionality as a Measure for Adversarial Robustness(https://arxiv.org/abs/2410.18556)
• Keywords: robust
• Abstract: Quantifying robustness in a single measure for the purposes of model selection, development of adversarial training methods, and anticipating trends has so far been elusive. The simplest metric to consider is the number of trainable parameters in a model but this has previously been shown to be insufficient at explaining robustness properties. A variety of other metrics, such as ones based on boundary thickness and gradient flatness have been proposed but have been shown to be inadequate proxies for robustness. In this work, we investigate the relationship between a model's effective dimensionality, which can be thought of as model complexity, and its robustness properties. We run experiments on commercial-scale models that are often used in real-world environments such as YOLO and ResNet. We reveal a near-linear inverse relationship between effective dimensionality and adversarial robustness, that is models with a lower dimensionality exhibit better robustness. We investigate the effect of a variety of adversarial training methods on effective dimensionality and find the same inverse linear relationship present, suggesting that effective dimensionality can serve as a useful criterion for model selection and robustness evaluation, providing a more nuanced and effective metric than parameter count or previously-tested measures.

Title: Research on gesture recognition method based on SEDCNN-SVM

• Authors: Mingjin Zhang, Jiahao Wang, Jianming Wang, Qi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18557
• Pdf URL: https://arxiv.org/pdf/2410.18557
• Copy Paste: [[2410.18557]] Research on gesture recognition method based on SEDCNN-SVM(https://arxiv.org/abs/2410.18557)
• Keywords: extraction
• Abstract: Gesture recognition based on surface electromyographic signal (sEMG) is one of the most used methods. The traditional manual feature extraction can only extract some low-level signal features, this causes poor classifier performance and low recognition accuracy when dealing with some complex signals. A recognition method, namely SEDCNN-SVM, is proposed to recognize sEMG of different gestures. SEDCNN-SVM consists of an improved deep convolutional neural network (DCNN) and a support vector machine (SVM). The DCNN can automatically extract and learn the feature information of sEMG through the convolution operation of the convolutional layer, so that it can capture the complex and high-level features in the data. The Squeeze and Excitation Networks (SE-Net) and the residual module were added to the model, so that the feature representation of each channel could be improved, the loss of feature information in convolutional operations was reduced, useful feature information was captured, and the problem of network gradient vanishing was eased. The SVM can improve the generalization ability and classification accuracy of the model by constructing an optimal hyperplane of the feature space. Hence, the SVM was used to replace the full connection layer and the Softmax function layer of the DCNN, the use of a suitable kernel function in SVM can improve the model's generalization ability and classification accuracy. To verify the effectiveness of the proposed classification algorithm, this method is analyzed and compared with other comparative classification methods. The recognition accuracy of SEDCNN-SVM can reach 0.955, it is significantly improved compared with other classification methods, the SEDCNN-SVM model is recognized online in real time.

Title: Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation

• Authors: Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, Remigiusz Kinas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18565
• Pdf URL: https://arxiv.org/pdf/2410.18565
• Copy Paste: [[2410.18565]] Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation(https://arxiv.org/abs/2410.18565)
• Keywords: generative
• Abstract: We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish language processing. Trained on curated Polish corpora, this model addresses key challenges in language model development through innovative techniques. These include Weighted Instruction Cross-Entropy Loss, which balances the learning of different instruction types, and Adaptive Learning Rate, which dynamically adjusts the learning rate based on training progress. To evaluate performance, we created the Open PL LLM Leaderboard and Polish MT-Bench, novel frameworks assessing various NLP tasks and conversational abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9 percentage point increase in average score compared to Mistral-7B-v0.1 on the RAG Reader task. It also excels in the Polish MT-Bench, particularly in Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model represents a substantial advancement in Polish language AI, offering a powerful tool for diverse linguistic applications and setting new benchmarks in the field.

Title: Taipan: Efficient and Expressive State Space Language Models with Selective Attention

• Authors: Chien Van Nguyen, Huy Huu Nguyen, Thang M. Pham, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Ryan A. Rossi, Trung Bui, Viet Dac Lai, Franck Dernoncourt, Thien Huu Nguyen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18572
• Pdf URL: https://arxiv.org/pdf/2410.18572
• Copy Paste: [[2410.18572]] Taipan: Efficient and Expressive State Space Language Models with Selective Attention(https://arxiv.org/abs/2410.18572)
• Keywords: transformer
• Abstract: Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long sequences due to quadratic computational complexity in training and linearly scaling memory costs during inference. Recent State Space Models (SSMs) such as Mamba offer alternatives with constant memory usage, but they underperform in tasks requiring extensive in-context retrieval. We introduce Taipan, a novel hybrid architecture that combines Mamba-2 with Selective Attention Layers (SALs). These SALs identify tokens requiring long-range interactions, remove less important features, and then augment their representations using the attention module. This approach balances Mamba's efficiency with Transformer-like performance in memory-intensive tasks. By constraining the attention budget, Taipan extends accurate predictions to context lengths of up to 1 million tokens while preserving computational efficiency. Our experiments demonstrate Taipan's superior performance across various scales and tasks, offering a promising solution for efficient long-context language modeling.

Title: On Model-Free Re-ranking for Visual Place Recognition with Deep Learned Local Features

• Authors: Tomáš Pivoňka, Libor Přeučil
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.18573
• Pdf URL: https://arxiv.org/pdf/2410.18573
• Copy Paste: [[2410.18573]] On Model-Free Re-ranking for Visual Place Recognition with Deep Learned Local Features(https://arxiv.org/abs/2410.18573)
• Keywords: robust
• Abstract: Re-ranking is the second stage of a visual place recognition task, in which the system chooses the best-matching images from a pre-selected subset of candidates. Model-free approaches compute the image pair similarity based on a spatial comparison of corresponding local visual features, eliminating the need for computationally expensive estimation of a model describing transformation between images. The article focuses on model-free re-ranking based on standard local visual features and their applicability in long-term autonomy systems. It introduces three new model-free re-ranking methods that were designed primarily for deep-learned local visual features. These features evince high robustness to various appearance changes, which stands as a crucial property for use with long-term autonomy systems. All the introduced methods were employed in a new visual place recognition system together with the D2-net feature detector (Dusmanu, 2019) and experimentally tested with diverse, challenging public datasets. The obtained results are on par with current state-of-the-art methods, affirming that model-free approaches are a viable and worthwhile path for long-term visual place recognition.

Title: Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data

• Authors: Anup Shirgaonkar, Nikhil Pandey, Nazmiye Ceren Abay, Tolga Aktas, Vijay Aski
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18588
• Pdf URL: https://arxiv.org/pdf/2410.18588
• Copy Paste: [[2410.18588]] Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data(https://arxiv.org/abs/2410.18588)
• Keywords: large language model
• Abstract: Leading open-source large language models (LLMs) such as Llama-3.1-Instruct-405B are extremely capable at generating text, answering questions, and solving a variety of natural language understanding tasks. However, they incur higher inference cost and latency compared to smaller LLMs. Knowledge distillation provides a way to use outputs from these large, capable teacher models to train smaller student models which can be used for inference at lower cost and latency, while retaining comparable accuracy. We investigate the efficacy of distillation using the Llama-3.1-405B-Instruct teacher and the smaller Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct student models. Contributions of this work include (a) We evaluate the generalizability of distillation with the above Llama-3.1 teacher-student pairs across different tasks and datasets (b) We show that using synthetic data during distillation significantly improves the accuracy of 8B and 70B models, and when used with reasoning chains, even matches or surpasses the zero-shot accuracy of 405B model on some datasets (c) We empirically show that distillation enables 8B and 70B models to internalize 405B's reasoning ability by using only standard fine-tuning (without customizing any loss function). This allows cost and latency-efficient student model inference. (d) We show pitfalls in evaluation of distillation, and present task-specific evaluation, including both human and LLM-grading, and ground-truth based traditional accuracy benchmarks. This methodical study brings out the fundamental importance of synthetic data quality in knowledge distillation, and of combining multiple, task-specific ways of accuracy and quality evaluation in assessing the effectiveness of distillation.

Title: TripCast: Pre-training of Masked 2D Transformers for Trip Time Series Forecasting

• Authors: Yuhua Liao, Zetian Wang, Peng Wei, Qiangqiang Nie, Zhenhua Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18612
• Pdf URL: https://arxiv.org/pdf/2410.18612
• Copy Paste: [[2410.18612]] TripCast: Pre-training of Masked 2D Transformers for Trip Time Series Forecasting(https://arxiv.org/abs/2410.18612)
• Keywords: transformer
• Abstract: Deep learning and pre-trained models have shown great success in time series forecasting. However, in the tourism industry, time series data often exhibit a leading time property, presenting a 2D structure. This introduces unique challenges for forecasting in this sector. In this study, we propose a novel modelling paradigm, TripCast, which treats trip time series as 2D data and learns representations through masking and reconstruction processes. Pre-trained on large-scale real-world data, TripCast notably outperforms other state-of-the-art baselines in in-domain forecasting scenarios and demonstrates strong scalability and transferability in out-domain forecasting scenarios.

Title: Rethinking Softmax: Self-Attention with Polynomial Activations

• Authors: Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang, Simon Lucey
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.18613
• Pdf URL: https://arxiv.org/pdf/2410.18613
• Copy Paste: [[2410.18613]] Rethinking Softmax: Self-Attention with Polynomial Activations(https://arxiv.org/abs/2410.18613)
• Keywords: transformer
• Abstract: This paper challenges the conventional belief that softmax attention in transformers is effective primarily because it generates a probability distribution for attention allocation. Instead, we theoretically show that its success lies in its ability to implicitly regularize the Frobenius norm of the attention matrix during training. We then explore alternative activations that regularize the Frobenius norm of the attention matrix, demonstrating that certain polynomial activations can achieve this effect, making them suitable for attention-based architectures. Empirical results indicate these activations perform comparably or better than softmax across various computer vision and language tasks, suggesting new possibilities for attention mechanisms beyond softmax.

Title: FairQueue: Rethinking Prompt Learning for Fair Text-to-Image Generation

• Authors: Christopher T.H Teo, Milad Abdollahzadeh, Xinda Ma, Ngai-man Cheung
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18615
• Pdf URL: https://arxiv.org/pdf/2410.18615
• Copy Paste: [[2410.18615]] FairQueue: Rethinking Prompt Learning for Fair Text-to-Image Generation(https://arxiv.org/abs/2410.18615)
• Keywords: fair
• Abstract: Recently, prompt learning has emerged as the state-of-the-art (SOTA) for fair text-to-image (T2I) generation. Specifically, this approach leverages readily available reference images to learn inclusive prompts for each target Sensitive Attribute (tSA), allowing for fair image generation. In this work, we first reveal that this prompt learning-based approach results in degraded sample quality. Our analysis shows that the approach's training objective -- which aims to align the embedding differences of learned prompts and reference images -- could be sub-optimal, resulting in distortion of the learned prompts and degraded generated images. To further substantiate this claim, as our major contribution, we deep dive into the denoising subnetwork of the T2I model to track down the effect of these learned prompts by analyzing the cross-attention maps. In our analysis, we propose a novel prompt switching analysis: I2H and H2I. Furthermore, we propose new quantitative characterization of cross-attention maps. Our analysis reveals abnormalities in the early denoising steps, perpetuating improper global structure that results in degradation in the generated samples. Building on insights from our analysis, we propose two ideas: (i) Prompt Queuing and (ii) Attention Amplification to address the quality issue. Extensive experimental results on a wide range of tSAs show that our proposed method outperforms SOTA approach's image generation quality, while achieving competitive fairness. More resources at FairQueue Project site: this https URL

Title: Environment Maps Editing using Inverse Rendering and Adversarial Implicit Functions

• Authors: Antonio D'Orazio, Davide Sforza, Fabio Pellacini, Iacopo Masi
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.18622
• Pdf URL: https://arxiv.org/pdf/2410.18622
• Copy Paste: [[2410.18622]] Environment Maps Editing using Inverse Rendering and Adversarial Implicit Functions(https://arxiv.org/abs/2410.18622)
• Keywords: generative
• Abstract: Editing High Dynamic Range (HDR) environment maps using an inverse differentiable rendering architecture is a complex inverse problem due to the sparsity of relevant pixels and the challenges in balancing light sources and background. The pixels illuminating the objects are a small fraction of the total image, leading to noise and convergence issues when the optimization directly involves pixel values. HDR images, with pixel values beyond the typical Standard Dynamic Range (SDR), pose additional challenges. Higher learning rates corrupt the background during optimization, while lower learning rates fail to manipulate light sources. Our work introduces a novel method for editing HDR environment maps using a differentiable rendering, addressing sparsity and variance between values. Instead of introducing strong priors that extract the relevant HDR pixels and separate the light sources, or using tricks such as optimizing the HDR image in the log space, we propose to model the optimized environment map with a new variant of implicit neural representations able to handle HDR images. The neural representation is trained with adversarial perturbations over the weights to ensure smooth changes in the output when it receives gradients from the inverse rendering. In this way, we obtain novel and cheap environment maps without relying on latent spaces of expensive generative models, maintaining the original visual consistency. Experimental results demonstrate the method's effectiveness in reconstructing the desired lighting effects while preserving the fidelity of the map and reflections on objects in the scene. Our approach can pave the way to interesting tasks, such as estimating a new environment map given a rendering with novel light sources, maintaining the initial perceptual features, and enabling brush stroke-based editing of existing environment maps.

Title: Prompting and Fine-Tuning of Small LLMs for Length-Controllable Telephone Call Summarization

• Authors: David Thulke, Yingbo Gao, Rricha Jalota, Christian Dugast, Hermann Ney
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18624
• Pdf URL: https://arxiv.org/pdf/2410.18624
• Copy Paste: [[2410.18624]] Prompting and Fine-Tuning of Small LLMs for Length-Controllable Telephone Call Summarization(https://arxiv.org/abs/2410.18624)
• Keywords: large language model
• Abstract: This paper explores the rapid development of a telephone call summarization system utilizing large language models (LLMs). Our approach involves initial experiments with prompting existing LLMs to generate summaries of telephone conversations, followed by the creation of a tailored synthetic training dataset utilizing stronger frontier models. We place special focus on the diversity of the generated data and on the ability to control the length of the generated summaries to meet various use-case specific requirements. The effectiveness of our method is evaluated using two state-of-the-art LLM-as-a-judge-based evaluation techniques to ensure the quality and relevance of the summaries. Our results show that fine-tuned Llama-2-7B-based summarization model performs on-par with GPT-4 in terms of factual accuracy, completeness and conciseness. Our findings demonstrate the potential for quickly bootstrapping a practical and efficient call summarization system.

Title: Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Model

• Authors: Jinxu Lin, Linwei Tao, Minjing Dong, Chang Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18639
• Pdf URL: https://arxiv.org/pdf/2410.18639
• Copy Paste: [[2410.18639]] Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Model(https://arxiv.org/abs/2410.18639)
• Keywords: diffusion, generative
• Abstract: As diffusion models become increasingly popular, the misuse of copyrighted and private images has emerged as a major concern. One promising solution to mitigate this issue is identifying the contribution of specific training samples in generative models, a process known as data attribution. Existing data attribution methods for diffusion models typically quantify the contribution of a training sample by evaluating the change in diffusion loss when the sample is included or excluded from the training process. However, we argue that the direct usage of diffusion loss cannot represent such a contribution accurately due to the calculation of diffusion loss. Specifically, these approaches measure the divergence between predicted and ground truth distributions, which leads to an indirect comparison between the predicted distributions and cannot represent the variances between model behaviors. To address these issues, we aim to measure the direct comparison between predicted distributions with an attribution score to analyse the training sample importance, which is achieved by Diffusion Attribution Score (DAS). Underpinned by rigorous theoretical analysis, we elucidate the effectiveness of DAS. Additionally, we explore strategies to accelerate DAS calculations, facilitating its application to large-scale diffusion models. Our extensive experiments across various datasets and diffusion models demonstrate that DAS significantly surpasses previous benchmarks in terms of the linear data-modelling score, establishing new state-of-the-art performance.

Title: Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

• Authors: Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, Rui Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18640
• Pdf URL: https://arxiv.org/pdf/2410.18640
• Copy Paste: [[2410.18640]] Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model(https://arxiv.org/abs/2410.18640)
• Keywords: steal
• Abstract: Aligning language models (LMs) with human preferences has become a key area of research, enabling these models to meet diverse user needs better. Inspired by weak-to-strong generalization, where a strong LM fine-tuned on labels generated by a weaker model can consistently outperform its weak supervisor, we extend this idea to model alignment. In this work, we observe that the alignment behavior in weaker models can be effectively transferred to stronger models and even exhibit an amplification effect. Based on this insight, we propose a method called Weak-to-Strong Preference Optimization (WSPO), which achieves strong model alignment by learning the distribution differences before and after the alignment of the weak model. Experiments demonstrate that WSPO delivers outstanding performance, improving the win rate of Qwen2-7B-Instruct on Arena-Hard from 39.70 to 49.60, achieving a remarkable 47.04 length-controlled win rate on AlpacaEval 2, and scoring 7.33 on MT-bench. Our results suggest that using the weak model to elicit a strong model with a high alignment ability is feasible.

Title: $C^2$: Scalable Auto-Feedback for LLM-based Chart Generation

• Authors: Woosung Koh, Jang Han Yoon, MinHyung Lee, Youngjin Song, Jaegwan Cho, Jaehyun Kang, Taehyeon Kim, Se-young Yun, Youngjae Yu, Bongshin Lee
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18652
• Pdf URL: https://arxiv.org/pdf/2410.18652
• Copy Paste: [[2410.18652]] $C^2$: Scalable Auto-Feedback for LLM-based Chart Generation(https://arxiv.org/abs/2410.18652)
• Keywords: large language model
• Abstract: Generating high-quality charts with Large Language Models presents significant challenges due to limited data and the high cost of scaling through human curation. Instruction, data, and code triplets are scarce and expensive to manually curate as their creation demands technical expertise. To address this scalability issue, we introduce a reference-free automatic feedback generator, which eliminates the need for costly human intervention. Our novel framework, $C^2$, consists of (1) an automatic feedback provider (ChartAF) and (2) a diverse, reference-free dataset (ChartUIE-8K). Quantitative results are compelling: in our first experiment, 74% of respondents strongly preferred, and 10% preferred, the results after feedback. The second post-feedback experiment demonstrates that ChartAF outperforms nine baselines. Moreover, ChartUIE-8K significantly improves data diversity by increasing queries, datasets, and chart types by 5982%, 1936%, and 91%, respectively, over benchmarks. Finally, an LLM user study revealed that 94% of participants preferred ChartUIE-8K's queries, with 93% deeming them aligned with real-world use cases. Core contributions are available as open-source at an anonymized project site, with ample qualitative examples.

Title: Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework

• Authors: Esteban Garces Arias, Hannah Blocher, Julian Rodemann, Meimingwei Li, Christian Heumann, Matthias Aßenmacher
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18653
• Pdf URL: https://arxiv.org/pdf/2410.18653
• Copy Paste: [[2410.18653]] Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework(https://arxiv.org/abs/2410.18653)
• Keywords: robust
• Abstract: Open-ended text generation has become a prominent task in natural language processing due to the rise of powerful (large) language models. However, evaluating the quality of these models and the employed decoding strategies remains challenging because of trade-offs among widely used metrics such as coherence, diversity, and perplexity. Decoding methods often excel in some metrics while underperforming in others, complicating the establishment of a clear ranking. In this paper, we present novel ranking strategies within this multicriteria framework. Specifically, we employ benchmarking approaches based on partial orderings and present a new summary metric designed to balance existing automatic indicators, providing a more holistic evaluation of text generation quality. Furthermore, we discuss the alignment of these approaches with human judgments. Our experiments demonstrate that the proposed methods offer a robust way to compare decoding strategies, exhibit similarities with human preferences, and serve as valuable tools in guiding model selection for open-ended text generation tasks. Finally, we suggest future directions for improving evaluation methodologies in text generation. Our codebase, datasets, and models are publicly available.

Title: DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation

• Authors: Yuang Ai, Xiaoqiang Zhou, Huaibo Huang, Xiaotian Han, Zhengyu Chen, Quanzeng You, Hongxia Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18666
• Pdf URL: https://arxiv.org/pdf/2410.18666
• Copy Paste: [[2410.18666]] DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation(https://arxiv.org/abs/2410.18666)
• Keywords: privacy, robust, diffusion, transformer, generative, large language model
• Abstract: Image restoration (IR) in real-world scenarios presents significant challenges due to the lack of high-capacity models and comprehensive datasets. To tackle these issues, we present a dual strategy: GenIR, an innovative data curation pipeline, and DreamClear, a cutting-edge Diffusion Transformer (DiT)-based image restoration model. GenIR, our pioneering contribution, is a dual-prompt learning pipeline that overcomes the limitations of existing datasets, which typically comprise only a few thousand images and thus offer limited generalizability for larger models. GenIR streamlines the process into three stages: image-text pair construction, dual-prompt based fine-tuning, and data generation & filtering. This approach circumvents the laborious data crawling process, ensuring copyright compliance and providing a cost-effective, privacy-safe solution for IR dataset construction. The result is a large-scale dataset of one million high-quality images. Our second contribution, DreamClear, is a DiT-based image restoration model. It utilizes the generative priors of text-to-image (T2I) diffusion models and the robust perceptual capabilities of multi-modal large language models (MLLMs) to achieve photorealistic restoration. To boost the model's adaptability to diverse real-world degradations, we introduce the Mixture of Adaptive Modulator (MoAM). It employs token-wise degradation priors to dynamically integrate various restoration experts, thereby expanding the range of degradations the model can address. Our exhaustive experiments confirm DreamClear's superior performance, underlining the efficacy of our dual strategy for real-world image restoration. Code and pre-trained models will be available at: this https URL.

Title: Homomorphism Counts as Structural Encodings for Graph Learning

• Authors: Linus Bao, Emily Jin, Michael Bronstein, İsmail İlkan Ceylan, Matthias Lanzinger
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18676
• Pdf URL: https://arxiv.org/pdf/2410.18676
• Copy Paste: [[2410.18676]] Homomorphism Counts as Structural Encodings for Graph Learning(https://arxiv.org/abs/2410.18676)
• Keywords: transformer
• Abstract: Graph Transformers are popular neural networks that extend the well-known Transformer architecture to the graph domain. These architectures operate by applying self-attention on graph nodes and incorporating graph structure through the use of positional encodings (e.g., Laplacian positional encoding) or structural encodings (e.g., random-walk structural encoding). The quality of such encodings is critical, since they provide the necessary $\textit{graph inductive biases}$ to condition the model on graph structure. In this work, we propose $\textit{motif structural encoding}$ (MoSE) as a flexible and powerful structural encoding framework based on counting graph homomorphisms. Theoretically, we compare the expressive power of MoSE to random-walk structural encoding and relate both encodings to the expressive power of standard message passing neural networks. Empirically, we observe that MoSE outperforms other well-known positional and structural encodings across a range of architectures, and it achieves state-of-the-art performance on widely studied molecular property prediction datasets.

Title: Enhancing pretraining efficiency for medical image segmentation via transferability metrics

• Authors: Gábor Hidy, Bence Bakos, András Lukács
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.18677
• Pdf URL: https://arxiv.org/pdf/2410.18677
• Copy Paste: [[2410.18677]] Enhancing pretraining efficiency for medical image segmentation via transferability metrics(https://arxiv.org/abs/2410.18677)
• Keywords: robust, segmentation
• Abstract: In medical image segmentation tasks, the scarcity of labeled training data poses a significant challenge when training deep neural networks. When using U-Net-style architectures, it is common practice to address this problem by pretraining the encoder part on a large general-purpose dataset like ImageNet. However, these methods are resource-intensive and do not guarantee improved performance on the downstream task. In this paper we investigate a variety of training setups on medical image segmentation datasets, using ImageNet-pretrained models. By examining over 300 combinations of models, datasets, and training methods, we find that shorter pretraining often leads to better results on the downstream task, providing additional proof to the well-known fact that the accuracy of the model on ImageNet is a poor indicator for downstream performance. As our main contribution, we introduce a novel transferability metric, based on contrastive learning, that measures how robustly a pretrained model is able to represent the target data. In contrast to other transferability scores, our method is applicable to the case of transferring from ImageNet classification to medical image segmentation. We apply our robustness score by measuring it throughout the pretraining phase to indicate when the model weights are optimal for downstream transfer. This reduces pretraining time and improves results on the target task.

Title: Ali-AUG: Innovative Approaches to Labeled Data Augmentation using One-Step Diffusion Model

• Authors: Ali Hamza, Aizea Lojo, Adrian Núñez-Marcos, Aitziber Atutxa
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18678
• Pdf URL: https://arxiv.org/pdf/2410.18678
• Copy Paste: [[2410.18678]] Ali-AUG: Innovative Approaches to Labeled Data Augmentation using One-Step Diffusion Model(https://arxiv.org/abs/2410.18678)
• Keywords: diffusion
• Abstract: This paper introduces Ali-AUG, a novel single-step diffusion model for efficient labeled data augmentation in industrial applications. Our method addresses the challenge of limited labeled data by generating synthetic, labeled images with precise feature insertion. Ali-AUG utilizes a stable diffusion architecture enhanced with skip connections and LoRA modules to efficiently integrate masks and images, ensuring accurate feature placement without affecting unrelated image content. Experimental validation across various industrial datasets demonstrates Ali-AUG's superiority in generating high-quality, defect-enhanced images while maintaining rapid single-step inference. By offering precise control over feature insertion and minimizing required training steps, our technique significantly enhances data augmentation capabilities, providing a powerful tool for improving the performance of deep learning models in scenarios with limited labeled data. Ali-AUG is especially useful for use cases like defective product image generation to train AI-based models to improve their ability to detect defects in manufacturing processes. Using different data preparation strategies, including Classification Accuracy Score (CAS) and Naive Augmentation Score (NAS), we show that Ali-AUG improves model performance by 31% compared to other augmentation methods and by 45% compared to models without data augmentation. Notably, Ali-AUG reduces training time by 32% and supports both paired and unpaired datasets, enhancing flexibility in data preparation.

Title: Rigid Single-Slice-in-Volume registration via rotation-equivariant 2D/3D feature matching

• Authors: Stefan Brandstätter, Philipp Seeböck, Christoph Fürböck, Svitlana Pochepnia, Helmut Prosch, Georg Langs
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18683
• Pdf URL: https://arxiv.org/pdf/2410.18683
• Copy Paste: [[2410.18683]] Rigid Single-Slice-in-Volume registration via rotation-equivariant 2D/3D feature matching(https://arxiv.org/abs/2410.18683)
• Keywords: robust
• Abstract: 2D to 3D registration is essential in tasks such as diagnosis, surgical navigation, environmental understanding, navigation in robotics, autonomous systems, or augmented reality. In medical imaging, the aim is often to place a 2D image in a 3D volumetric observation to w. Current approaches for rigid single slice in volume registration are limited by requirements such as pose initialization, stacks of adjacent slices, or reliable anatomical landmarks. Here, we propose a self-supervised 2D/3D registration approach to match a single 2D slice to the corresponding 3D volume. The method works in data without anatomical priors such as images of tumors. It addresses the dimensionality disparity and establishes correspondences between 2D in-plane and 3D out-of-plane rotation-equivariant features by using group equivariant CNNs. These rotation-equivariant features are extracted from the 2D query slice and aligned with their 3D counterparts. Results demonstrate the robustness of the proposed slice-in-volume registration on the NSCLC-Radiomics CT and KIRBY21 MRI datasets, attaining an absolute median angle error of less than 2 degrees and a mean-matching feature accuracy of 89% at a tolerance of 3 pixels.

Title: Every Component Counts: Rethinking the Measure of Success for Medical Semantic Segmentation in Multi-Instance Segmentation Tasks

• Authors: Alexander Jaus, Constantin Seibold, Simon Reiß, Zdravko Marinov, Keyi Li, Zeling Ye, Stefan Krieg, Jens Kleesiek, Rainer Stiefelhagen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18684
• Pdf URL: https://arxiv.org/pdf/2410.18684
• Copy Paste: [[2410.18684]] Every Component Counts: Rethinking the Measure of Success for Medical Semantic Segmentation in Multi-Instance Segmentation Tasks(https://arxiv.org/abs/2410.18684)
• Keywords: segmentation
• Abstract: We present Connected-Component~(CC)-Metrics, a novel semantic segmentation evaluation protocol, targeted to align existing semantic segmentation metrics to a multi-instance detection scenario in which each connected component matters. We motivate this setup in the common medical scenario of semantic metastases segmentation in a full-body PET/CT. We show how existing semantic segmentation metrics suffer from a bias towards larger connected components contradicting the clinical assessment of scans in which tumor size and clinical relevance are uncorrelated. To rebalance existing segmentation metrics, we propose to evaluate them on a per-component basis thus giving each tumor the same weight irrespective of its size. To match predictions to ground-truth segments, we employ a proximity-based matching criterion, evaluating common metrics locally at the component of interest. Using this approach, we break free of biases introduced by large metastasis for overlap-based metrics such as Dice or Surface Dice. CC-Metrics also improves distance-based metrics such as Hausdorff Distances which are uninformative for small changes that do not influence the maximum or 95th percentile, and avoids pitfalls introduced by directly combining counting-based metrics with overlap-based metrics as it is done in Panoptic Quality.

Title: Hierarchical Multimodal LLMs with Semantic Space Alignment for Enhanced Time Series Classification

• Authors: Xiaoyu Tao, Tingyue Pan, Mingyue Cheng, Yucong Luo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18686
• Pdf URL: https://arxiv.org/pdf/2410.18686
• Copy Paste: [[2410.18686]] Hierarchical Multimodal LLMs with Semantic Space Alignment for Enhanced Time Series Classification(https://arxiv.org/abs/2410.18686)
• Keywords: large language model
• Abstract: Leveraging large language models (LLMs) has garnered increasing attention and introduced novel perspectives in time series classification. However, existing approaches often overlook the crucial dynamic temporal information inherent in time series data and face challenges in aligning this data with textual semantics. To address these limitations, we propose HiTime, a hierarchical multi-modal model that seamlessly integrates temporal information into LLMs for multivariate time series classification (MTSC). Our model employs a hierarchical feature encoder to capture diverse aspects of time series data through both data-specific and task-specific embeddings. To facilitate semantic space alignment between time series and text, we introduce a dual-view contrastive alignment module that bridges the gap between modalities. Additionally, we adopt a hybrid prompting strategy to fine-tune the pre-trained LLM in a parameter-efficient manner. By effectively incorporating dynamic temporal features and ensuring semantic alignment, HiTime enables LLMs to process continuous time series data and achieves state-of-the-art classification performance through text generation. Extensive experiments on benchmark datasets demonstrate that HiTime significantly enhances time series classification accuracy compared to most competitive baseline methods. Our findings highlight the potential of integrating temporal features into LLMs, paving the way for advanced time series analysis. The code is publicly available for further research and validation. Our codes are publicly available1.

Title: PESFormer: Boosting Macro- and Micro-expression Spotting with Direct Timestamp Encoding

• Authors: Wang-Wang Yu, Kai-Fu Yang, Xiangrui Hu, Jingwen Jiang, Hong-Mei Yan, Yong-Jie Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18695
• Pdf URL: https://arxiv.org/pdf/2410.18695
• Copy Paste: [[2410.18695]] PESFormer: Boosting Macro- and Micro-expression Spotting with Direct Timestamp Encoding(https://arxiv.org/abs/2410.18695)
• Keywords: transformer
• Abstract: The task of macro- and micro-expression spotting aims to precisely localize and categorize temporal expression instances within untrimmed videos. Given the sparse distribution and varying durations of expressions, existing anchor-based methods often represent instances by encoding their deviations from predefined anchors. Additionally, these methods typically slice the untrimmed videos into fixed-length sliding windows. However, anchor-based encoding often fails to capture all training intervals, and slicing the original video as sliding windows can result in valuable training intervals being discarded. To overcome these limitations, we introduce PESFormer, a simple yet effective model based on the vision transformer architecture to achieve point-to-interval expression spotting. PESFormer employs a direct timestamp encoding (DTE) approach to replace anchors, enabling binary classification of each timestamp instead of optimizing entire ground truths. Thus, all training intervals are retained in the form of discrete timestamps. To maximize the utilization of training intervals, we enhance the preprocessing process by replacing the short videos produced through the sliding window this http URL, we implement a strategy that involves zero-padding the untrimmed training videos to create uniform, longer videos of a predetermined duration. This operation efficiently preserves the original training intervals and eliminates video slice this http URL qualitative and quantitative evaluations on three datasets -- CAS(ME)^2, CAS(ME)^3 and SAMM-LV -- demonstrate that our PESFormer outperforms existing techniques, achieving the best performance.

Title: BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching

• Authors: Peizhuang Cong, Qizhi Chen, Haochen Zhao, Tong Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18701
• Pdf URL: https://arxiv.org/pdf/2410.18701
• Copy Paste: [[2410.18701]] BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching(https://arxiv.org/abs/2410.18701)
• Keywords: large language model
• Abstract: The advanced capabilities of Large Language Models (LLMs) have inspired the development of various interactive web services or applications, such as ChatGPT, which offer query inference services for users. Unlike traditional DNN model, the inference of LLM entails different iterations of forward computation for different queries, which result in efficiency challenges for existing run-to-completion batch-wise inference. Hence, some methods refine batch-wise inference to iteration-level by duplicating all nonlinear layers of LLM. However, this approach not only increases resource usage but also introduces idle computations to the batch due to the prefilling of newly added queries. Therefore, we propose BATON, an efficient batch-wise LLM inference scheme by dynamically adjusting processing batch, which can achieve near-zero idle computations without incurring additional resource consumption. To do so, BATON 1) shapes the vectors involved in the inference of the newly inserted query and processing batch to align dimensions and generates a new attention mask based on vector shaping to ensure inference correctness, which enables query inserting without consuming additional resource; 2) embeds prefilled Keys and Values of the new query into the KV_Cache of the processing batch by leveraging the prefilling and decoding separation mechanism, eliminating idle computations to the batch introduced by the prefilling process of the new query. Experimental results show that compared to the state-of-the-art solution Orca, BATON improves query processing by up to 1.75 times.

Title: Exploiting Interpretable Capabilities with Concept-Enhanced Diffusion and Prototype Networks

• Authors: Alba Carballo-Castro, Sonia Laguna, Moritz Vandenhirtz, Julia E. Vogt
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18705
• Pdf URL: https://arxiv.org/pdf/2410.18705
• Copy Paste: [[2410.18705]] Exploiting Interpretable Capabilities with Concept-Enhanced Diffusion and Prototype Networks(https://arxiv.org/abs/2410.18705)
• Keywords: diffusion
• Abstract: Concept-based machine learning methods have increasingly gained importance due to the growing interest in making neural networks interpretable. However, concept annotations are generally challenging to obtain, making it crucial to leverage all their prior knowledge. By creating concept-enriched models that incorporate concept information into existing architectures, we exploit their interpretable capabilities to the fullest extent. In particular, we propose Concept-Guided Conditional Diffusion, which can generate visual representations of concepts, and Concept-Guided Prototype Networks, which can create a concept prototype dataset and leverage it to perform interpretable concept prediction. These results open up new lines of research by exploiting pre-existing information in the quest for rendering machine learning more human-understandable.

Title: Retrieval-Augmented Diffusion Models for Time Series Forecasting

• Authors: Jingwei Liu, Ling Yang, Hongyan Li, Shenda Hong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18712
• Pdf URL: https://arxiv.org/pdf/2410.18712
• Copy Paste: [[2410.18712]] Retrieval-Augmented Diffusion Models for Time Series Forecasting(https://arxiv.org/abs/2410.18712)
• Keywords: diffusion
• Abstract: While time series diffusion models have received considerable focus from many recent works, the performance of existing models remains highly unstable. Factors limiting time series diffusion models include insufficient time series datasets and the absence of guidance. To address these limitations, we propose a Retrieval- Augmented Time series Diffusion model (RATD). The framework of RATD consists of two parts: an embedding-based retrieval process and a reference-guided diffusion model. In the first part, RATD retrieves the time series that are most relevant to historical time series from the database as references. The references are utilized to guide the denoising process in the second part. Our approach allows leveraging meaningful samples within the database to aid in sampling, thus maximizing the utilization of datasets. Meanwhile, this reference-guided mechanism also compensates for the deficiencies of existing time series diffusion models in terms of guidance. Experiments and visualizations on multiple datasets demonstrate the effectiveness of our approach, particularly in complicated prediction tasks.

Title: ChatSearch: a Dataset and a Generative Retrieval Model for General Conversational Image Retrieval

• Authors: Zijia Zhao, Longteng Guo, Tongtian Yue, Erdong Hu, Shuai Shao, Zehuan Yuan, Hua Huang, Jing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18715
• Pdf URL: https://arxiv.org/pdf/2410.18715
• Copy Paste: [[2410.18715]] ChatSearch: a Dataset and a Generative Retrieval Model for General Conversational Image Retrieval(https://arxiv.org/abs/2410.18715)
• Keywords: generative
• Abstract: In this paper, we investigate the task of general conversational image retrieval on open-domain images. The objective is to search for images based on interactive conversations between humans and computers. To advance this task, we curate a dataset called ChatSearch. This dataset includes a multi-round multimodal conversational context query for each target image, thereby requiring the retrieval system to find the accurate image from database. Simultaneously, we propose a generative retrieval model named ChatSearcher, which is trained end-to-end to accept/produce interleaved image-text inputs/outputs. ChatSearcher exhibits strong capability in reasoning with multimodal context and can leverage world knowledge to yield visual retrieval results. It demonstrates superior performance on the ChatSearch dataset and also achieves competitive results on other image retrieval tasks and visual conversation tasks. We anticipate that this work will inspire further research on interactive multimodal retrieval systems. Our dataset will be available at this https URL.

Title: Low-Latency Video Anonymization for Crowd Anomaly Detection: Privacy vs. Performance

• Authors: Mulugeta Weldezgina Asres, Lei Jiao, Christian Walter Omlin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18717
• Pdf URL: https://arxiv.org/pdf/2410.18717
• Copy Paste: [[2410.18717]] Low-Latency Video Anonymization for Crowd Anomaly Detection: Privacy vs. Performance(https://arxiv.org/abs/2410.18717)
• Keywords: privacy, protect
• Abstract: Recent advancements in artificial intelligence promise ample potential in monitoring applications with surveillance cameras. However, concerns about privacy and model bias have made it challenging to utilize them in public. Although de-identification approaches have been proposed in the literature, aiming to achieve a certain level of anonymization, most of them employ deep learning models that are computationally demanding for real-time edge deployment. In this study, we revisit conventional anonymization solutions for privacy protection and real-time video anomaly detection (VAD) applications. We propose a novel lightweight adaptive anonymization for VAD (LA3D) that employs dynamic adjustment to enhance privacy protection. We evaluated the approaches on publicly available privacy and VAD data sets to examine the strengths and weaknesses of the different anonymization techniques and highlight the promising efficacy of our approach. Our experiment demonstrates that LA3D enables substantial improvement in the privacy anonymization capability without majorly degrading VAD efficacy.

Title: GeoLoRA: Geometric integration for parameter efficient fine-tuning

• Authors: Steffen Schotthöfer, Emanuele Zangrando, Gianluca Ceruti, Francesco Tudisco, Jonas Kusch
• Subjects: cs.LG, cs.AI, math.NA
• Abstract URL: https://arxiv.org/abs/2410.18720
• Pdf URL: https://arxiv.org/pdf/2410.18720
• Copy Paste: [[2410.18720]] GeoLoRA: Geometric integration for parameter efficient fine-tuning(https://arxiv.org/abs/2410.18720)
• Keywords: robust
• Abstract: Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We introduce GeoLoRA, a novel approach that addresses these limitations by leveraging dynamical low-rank approximation theory. GeoLoRA requires only a single backpropagation pass over the small-rank adapters, significantly reducing computational cost as compared to similar dynamical low-rank training methods and making it faster than popular baselines such as AdaLoRA. This allows GeoLoRA to efficiently adapt the allocated parameter budget across the model, achieving smaller low-rank adapters compared to heuristic methods like AdaLoRA and LoRA, while maintaining critical convergence, descent, and error-bound theoretical guarantees. The resulting method is not only more efficient but also more robust to varying hyperparameter settings. We demonstrate the effectiveness of GeoLoRA on several state-of-the-art benchmarks, showing that it outperforms existing methods in both accuracy and computational efficiency.

Title: Rectified Diffusion Guidance for Conditional Generation

• Authors: Mengfei Xia, Nan Xue, Yujun Shen, Ran Yi, Tieliang Gong, Yong-Jin Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18737
• Pdf URL: https://arxiv.org/pdf/2410.18737
• Copy Paste: [[2410.18737]] Rectified Diffusion Guidance for Conditional Generation(https://arxiv.org/abs/2410.18737)
• Keywords: diffusion, generative
• Abstract: Classifier-Free Guidance (CFG), which combines the conditional and unconditional score functions with two coefficients summing to one, serves as a practical technique for diffusion model sampling. Theoretically, however, denoising with CFG cannot be expressed as a reciprocal diffusion process, which may consequently leave some hidden risks during use. In this work, we revisit the theory behind CFG and rigorously confirm that the improper configuration of the combination coefficients (i.e., the widely used summing-to-one version) brings about expectation shift of the generative distribution. To rectify this issue, we propose ReCFG with a relaxation on the guidance coefficients such that denoising with ReCFG strictly aligns with the diffusion theory. We further show that our approach enjoys a closed-form solution given the guidance strength. That way, the rectified coefficients can be readily pre-computed via traversing the observed data, leaving the sampling speed barely affected. Empirical evidence on real-world data demonstrate the compatibility of our post-hoc design with existing state-of-the-art diffusion models, including both class-conditioned ones (e.g., EDM2 on ImageNet) and text-conditioned ones (e.g., SD3 on CC12M), without any retraining. We will open-source the code to facilitate further research.

Title: Cellpose+, a morphological analysis tool for feature extraction of stained cell images

• Authors: Israel A. Huaman, Fares D.E. Ghorabe, Sofya S. Chumakova, Alexandra A. Pisarenko, Alexey E. Dudaev, Tatiana G. Volova, Galina A. Ryltseva, Sviatlana A. Ulasevich, Ekaterina I. Shishatskaya, Ekaterina V. Skorb, Pavel S. Zun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18738
• Pdf URL: https://arxiv.org/pdf/2410.18738
• Copy Paste: [[2410.18738]] Cellpose+, a morphological analysis tool for feature extraction of stained cell images(https://arxiv.org/abs/2410.18738)
• Keywords: extraction, segmentation
• Abstract: Advanced image segmentation and processing tools present an opportunity to study cell processes and their dynamics. However, image analysis is often routine and time-consuming. Nowadays, alternative data-driven approaches using deep learning are potentially offering automatized, accurate, and fast image analysis. In this paper, we extend the applications of Cellpose, a state-of-the-art cell segmentation framework, with feature extraction capabilities to assess morphological characteristics. We also introduce a dataset of DAPI and FITC stained cells to which our new method is applied.

Title: Why Does the Effective Context Length of LLMs Fall Short?

• Authors: Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, Lingpeng Kong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18745
• Pdf URL: https://arxiv.org/pdf/2410.18745
• Copy Paste: [[2410.18745]] Why Does the Effective Context Length of LLMs Fall Short?(https://arxiv.org/abs/2410.18745)
• Keywords: large language model
• Abstract: Advancements in distributed training and efficient attention mechanisms have significantly expanded the context window sizes of large language models (LLMs). However, recent work reveals that the effective context lengths of open-source LLMs often fall short, typically not exceeding half of their training lengths. In this work, we attribute this limitation to the left-skewed frequency distribution of relative positions formed in LLMs pretraining and post-training stages, which impedes their ability to effectively gather distant information. To address this challenge, we introduce ShifTed Rotray position embeddING (STRING). STRING shifts well-trained positions to overwrite the original ineffective positions during inference, enhancing performance within their existing training lengths. Experimental results show that without additional training, STRING dramatically improves the performance of the latest large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on popular long-context benchmarks RULER and InfiniteBench, establishing new state-of-the-art results for open-source LLMs. Compared to commercial models, Llama 3.1 70B with \method even achieves better performance than GPT-4-128K and clearly surpasses Claude 2 and Kimi-chat.

Title: Does Differential Privacy Impact Bias in Pretrained NLP Models?

• Authors: Md. Khairul Islam, Andrew Wang, Tianhao Wang, Yangfeng Ji, Judy Fox, Jieyu Zhao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18749
• Pdf URL: https://arxiv.org/pdf/2410.18749
• Copy Paste: [[2410.18749]] Does Differential Privacy Impact Bias in Pretrained NLP Models?(https://arxiv.org/abs/2410.18749)
• Keywords: privacy, protect, fair, large language model
• Abstract: Differential privacy (DP) is applied when fine-tuning pre-trained large language models (LLMs) to limit leakage of training examples. While most DP research has focused on improving a model's privacy-utility tradeoff, some find that DP can be unfair to or biased against underrepresented groups. In this work, we show the impact of DP on bias in LLMs through empirical analysis. Differentially private training can increase the model bias against protected groups w.r.t AUC-based bias metrics. DP makes it more difficult for the model to differentiate between the positive and negative examples from the protected groups and other groups in the rest of the population. Our results also show that the impact of DP on bias is not only affected by the privacy protection level but also the underlying distribution of the dataset.

Title: Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing

• Authors: Haonan Lin, Mengmeng Wang, Jiahao Wang, Wenbin An, Yan Chen, Yong Liu, Feng Tian, Guang Dai, Jingdong Wang, Qianying Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18756
• Pdf URL: https://arxiv.org/pdf/2410.18756
• Copy Paste: [[2410.18756]] Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing(https://arxiv.org/abs/2410.18756)
• Keywords: diffusion
• Abstract: Text-guided diffusion models have significantly advanced image editing, enabling high-quality and diverse modifications driven by text prompts. However, effective editing requires inverting the source image into a latent space, a process often hindered by prediction errors inherent in DDIM inversion. These errors accumulate during the diffusion process, resulting in inferior content preservation and edit fidelity, especially with conditional inputs. We address these challenges by investigating the primary contributors to error accumulation in DDIM inversion and identify the singularity problem in traditional noise schedules as a key issue. To resolve this, we introduce the Logistic Schedule, a novel noise schedule designed to eliminate singularities, improve inversion stability, and provide a better noise space for image editing. This schedule reduces noise prediction errors, enabling more faithful editing that preserves the original content of the source image. Our approach requires no additional retraining and is compatible with various existing editing methods. Experiments across eight editing tasks demonstrate the Logistic Schedule's superior performance in content preservation and edit fidelity compared to traditional noise schedules, highlighting its adaptability and effectiveness.

Title: Task Calibration: Calibrating Large Language Models on Inference Tasks

• Authors: Yingjie Li, Yun Luo, Xiaotian Xie, Yue Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18764
• Pdf URL: https://arxiv.org/pdf/2410.18764
• Copy Paste: [[2410.18764]] Task Calibration: Calibrating Large Language Models on Inference Tasks(https://arxiv.org/abs/2410.18764)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have exhibited impressive zero-shot performance on inference tasks. However, LLMs may suffer from spurious correlations between input texts and output labels, which limits LLMs' ability to reason based purely on general language understanding. In other words, LLMs may make predictions primarily based on premise or hypothesis, rather than both components. To address this problem that may lead to unexpected performance degradation, we propose task calibration (TC), a zero-shot and inference-only calibration method inspired by mutual information which recovers LLM performance through task reformulation. TC encourages LLMs to reason based on both premise and hypothesis, while mitigating the models' over-reliance on individual premise or hypothesis for inference. Experimental results show that TC achieves a substantial improvement on 13 inference tasks in the zero-shot setup. We further validate the effectiveness of TC in few-shot setups and various natural language understanding tasks. Further analysis indicates that TC is also robust to prompt templates and has the potential to be integrated with other calibration methods.

Title: Attention-based Citywide Electric Vehicle Charging Demand Prediction Approach Considering Urban Region and Dynamic Influences

• Authors: Haoxuan Kuang, Kunxiang Deng, Linlin You, Jun Li
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.18766
• Pdf URL: https://arxiv.org/pdf/2410.18766
• Copy Paste: [[2410.18766]] Attention-based Citywide Electric Vehicle Charging Demand Prediction Approach Considering Urban Region and Dynamic Influences(https://arxiv.org/abs/2410.18766)
• Keywords: transformer
• Abstract: Electric vehicle charging demand prediction is important for vacant charging pile recommendation and charging infrastructure planning, thus facilitating vehicle electrification and green energy development. The performance of previous spatio-temporal studies is still far from satisfactory because the traditional graphs are difficult to model non-pairwise spatial relationships and multivariate temporal features are not adequately taken into account. To tackle these issues, we propose an attention-based heterogeneous multivariate data fusion approach (AHMDF) for citywide electric vehicle charging demand prediction, which incorporates geo-based clustered hypergraph and multivariate gated Transformer to considers both static and dynamic influences. To learn non-pairwise relationships, we cluster service areas by the types and numbers of points of interest in the areas and develop attentive hypergraph networks accordingly. Graph attention mechanisms are used for information propagation between neighboring areas. Additionally, we improve the Transformer encoder utilizing gated mechanisms so that it can selectively learn dynamic auxiliary information and temporal features. Experiments on an electric vehicle charging benchmark dataset demonstrate the effectiveness of our proposed approach compared with a broad range of competing baselines. Furthermore, we demonstrate the impact of dynamic influences on prediction results in different areas of the city and the effectiveness of our clustering method.

Title: Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances

• Authors: Shilin Lu, Zihan Zhou, Jiayou Lu, Yuanzhi Zhu, Adams Wai-Kin Kong
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18775
• Pdf URL: https://arxiv.org/pdf/2410.18775
• Copy Paste: [[2410.18775]] Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances(https://arxiv.org/abs/2410.18775)
• Keywords: protect, attack, robust, watermark, diffusion, generative
• Abstract: Current image watermarking methods are vulnerable to advanced image editing techniques enabled by large-scale text-to-image models. These models can distort embedded watermarks during editing, posing significant challenges to copyright protection. In this work, we introduce W-Bench, the first comprehensive benchmark designed to evaluate the robustness of watermarking methods against a wide range of image editing techniques, including image regeneration, global editing, local editing, and image-to-video generation. Through extensive evaluations of eleven representative watermarking methods against prevalent editing techniques, we demonstrate that most methods fail to detect watermarks after such edits. To address this limitation, we propose VINE, a watermarking method that significantly enhances robustness against various image editing techniques while maintaining high image quality. Our approach involves two key innovations: (1) we analyze the frequency characteristics of image editing and identify that blurring distortions exhibit similar frequency properties, which allows us to use them as surrogate attacks during training to bolster watermark robustness; (2) we leverage a large-scale pretrained diffusion model SDXL-Turbo, adapting it for the watermarking task to achieve more imperceptible and robust watermark embedding. Experimental results show that our method achieves outstanding watermarking performance under various image editing techniques, outperforming existing methods in both image quality and robustness. Code is available at this https URL.

Title: A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs

• Authors: Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh, Ayan Chakrabarti, Wittawat Jitkrittum, Vladimir Feinberg, Seungyeon Kim, Hrayr Harutyunyan, Nikunj Saunshi, Zachary Nado, Rakesh Shivanna, Sashank J. Reddi, Aditya Krishna Menon, Rohan Anil, Sanjiv Kumar
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18779
• Pdf URL: https://arxiv.org/pdf/2410.18779
• Copy Paste: [[2410.18779]] A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs(https://arxiv.org/abs/2410.18779)
• Keywords: large language model
• Abstract: A primary challenge in large language model (LLM) development is their onerous pre-training cost. Typically, such pre-training involves optimizing a self-supervised objective (such as next-token prediction) over a large corpus. This paper explores a promising paradigm to improve LLM pre-training efficiency and quality by suitably leveraging a small language model (SLM). In particular, this paradigm relies on an SLM to both (1) provide soft labels as additional training supervision, and (2) select a small subset of valuable ("informative" and "hard") training examples. Put together, this enables an effective transfer of the SLM's predictive distribution to the LLM, while prioritizing specific regions of the training data distribution. Empirically, this leads to reduced LLM training time compared to standard training, while improving the overall quality. Theoretically, we develop a statistical framework to systematically study the utility of SLMs in enabling efficient training of high-quality LLMs. In particular, our framework characterizes how the SLM's seemingly low-quality supervision can enhance the training of a much more capable LLM. Furthermore, it also highlights the need for an adaptive utilization of such supervision, by striking a balance between the bias and variance introduced by the SLM-provided soft labels. We corroborate our theoretical framework by improving the pre-training of an LLM with 2.8B parameters by utilizing a smaller LM with 1.5B parameters on the Pile dataset.

Title: Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality

• Authors: Zhihan Huang, Yuting Wei, Yuxin Chen
• Subjects: cs.LG, eess.SP, math.NA, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.18784
• Pdf URL: https://arxiv.org/pdf/2410.18784
• Copy Paste: [[2410.18784]] Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality(https://arxiv.org/abs/2410.18784)
• Keywords: diffusion, generative
• Abstract: The denoising diffusion probabilistic model (DDPM) has emerged as a mainstream generative model in generative AI. While sharp convergence guarantees have been established for the DDPM, the iteration complexity is, in general, proportional to the ambient data dimension, resulting in overly conservative theory that fails to explain its practical efficiency. This has motivated the recent work Li and Yan (2024a) to investigate how the DDPM can achieve sampling speed-ups through automatic exploitation of intrinsic low dimensionality of data. We strengthen this prior work by demonstrating, in some sense, optimal adaptivity to unknown low dimensionality. For a broad class of data distributions with intrinsic dimension $k$, we prove that the iteration complexity of the DDPM scales nearly linearly with $k$, which is optimal when using KL divergence to measure distributional discrepancy. Our theory is established based on a key observation: the DDPM update rule is equivalent to running a suitably parameterized SDE upon discretization, where the nonlinear component of the drift term is intrinsically low-dimensional.

Title: WARP-LCA: Efficient Convolutional Sparse Coding with Locally Competitive Algorithm

• Authors: Geoffrey Kasenbacher, Felix Ehret, Gerrit Ecke, Sebastian Otte
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.18794
• Pdf URL: https://arxiv.org/pdf/2410.18794
• Copy Paste: [[2410.18794]] WARP-LCA: Efficient Convolutional Sparse Coding with Locally Competitive Algorithm(https://arxiv.org/abs/2410.18794)
• Keywords: robust
• Abstract: The locally competitive algorithm (LCA) can solve sparse coding problems across a wide range of use cases. Recently, convolution-based LCA approaches have been shown to be highly effective for enhancing robustness for image recognition tasks in vision pipelines. To additionally maximize representational sparsity, LCA with hard-thresholding can be applied. While this combination often yields very good solutions satisfying an $\ell_0$ sparsity criterion, it comes with significant drawbacks for practical application: (i) LCA is very inefficient, typically requiring hundreds of optimization cycles for convergence; (ii) the use of hard-thresholding results in a non-convex loss function, which might lead to suboptimal minima. To address these issues, we propose the Locally Competitive Algorithm with State Warm-up via Predictive Priming (WARP-LCA), which leverages a predictor network to provide a suitable initial guess of the LCA state based on the current input. Our approach significantly improves both convergence speed and the quality of solutions, while maintaining and even enhancing the overall strengths of LCA. We demonstrate that WARP-LCA converges faster by orders of magnitude and reaches better minima compared to conventional LCA. Moreover, the learned representations are more sparse and exhibit superior properties in terms of reconstruction and denoising quality as well as robustness when applied in deep recognition pipelines. Furthermore, we apply WARP-LCA to image denoising tasks, showcasing its robustness and practical effectiveness. Our findings confirm that the naive use of LCA with hard-thresholding results in suboptimal minima, whereas initializing LCA with a predictive guess results in better outcomes. This research advances the field of biologically inspired deep learning by providing a novel approach to convolutional sparse coding.

Title: Distill Visual Chart Reasoning Ability from LLMs to MLLMs

• Authors: Wei He, Zhiheng Xi, Wanxu Zhao, Xiaoran Fan, Yiwen Ding, Zifei Shan, Tao Gui, Qi Zhang, Xuanjing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18798
• Pdf URL: https://arxiv.org/pdf/2410.18798
• Copy Paste: [[2410.18798]] Distill Visual Chart Reasoning Ability from LLMs to MLLMs(https://arxiv.org/abs/2410.18798)
• Keywords: large language model
• Abstract: Solving complex chart Q&A tasks requires advanced visual reasoning abilities in multimodal large language models (MLLMs). Recent studies highlight that these abilities consist of two main parts: recognizing key information from visual inputs and conducting reasoning over it. Thus, a promising approach to enhance MLLMs is to construct relevant training data focusing on the two aspects. However, collecting and annotating complex charts and questions is costly and time-consuming, and ensuring the quality of annotated answers remains a challenge. In this paper, we propose Code-as-Intermediary Translation (CIT), a cost-effective, efficient and easily scalable data synthesis method for distilling visual reasoning abilities from LLMs to MLLMs. The code serves as an intermediary that translates visual chart representations into textual representations, enabling LLMs to understand cross-modal information. Specifically, we employ text-based synthesizing techniques to construct chart-plotting code and produce ReachQA, a dataset containing 3k reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and reasoning abilities. Experiments show that when fine-tuned with our data, models not only perform well on chart-related benchmarks, but also demonstrate improved multimodal reasoning abilities on general mathematical benchmarks like MathVista. The code and dataset are publicly available at this https URL.

Title: PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Clouds

• Authors: Balázs Gyenes, Nikolai Franke, Philipp Becker, Gerhard Neumann
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.18800
• Pdf URL: https://arxiv.org/pdf/2410.18800
• Copy Paste: [[2410.18800]] PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Clouds(https://arxiv.org/abs/2410.18800)
• Keywords: transformer
• Abstract: Perceiving the environment via cameras is crucial for Reinforcement Learning (RL) in robotics. While images are a convenient form of representation, they often complicate extracting important geometric details, especially with varying geometries or deformable objects. In contrast, point clouds naturally represent this geometry and easily integrate color and positional data from multiple camera views. However, while deep learning on point clouds has seen many recent successes, RL on point clouds is under-researched, with only the simplest encoder architecture considered in the literature. We introduce PointPatchRL (PPRL), a method for RL on point clouds that builds on the common paradigm of dividing point clouds into overlapping patches, tokenizing them, and processing the tokens with transformers. PPRL provides significant improvements compared with other point-cloud processing architectures previously used for RL. We then complement PPRL with masked reconstruction for representation learning and show that our method outperforms strong model-free and model-based baselines on image observations in complex manipulation tasks containing deformable objects and variations in target object geometry. Videos and code are available at this https URL

Title: Fast constrained sampling in pre-trained diffusion models

• Authors: Alexandros Graikos, Nebojsa Jojic, Dimitris Samaras
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18804
• Pdf URL: https://arxiv.org/pdf/2410.18804
• Copy Paste: [[2410.18804]] Fast constrained sampling in pre-trained diffusion models(https://arxiv.org/abs/2410.18804)
• Keywords: diffusion, generative
• Abstract: Diffusion models have dominated the field of large, generative image models, with the prime examples of Stable Diffusion and DALL-E 3 being widely adopted. These models have been trained to perform text-conditioned generation on vast numbers of image-caption pairs and as a byproduct, have acquired general knowledge about natural image statistics. However, when confronted with the task of constrained sampling, e.g. generating the right half of an image conditioned on the known left half, applying these models is a delicate and slow process, with previously proposed algorithms relying on expensive iterative operations that are usually orders of magnitude slower than text-based inference. This is counter-intuitive, as image-conditioned generation should rely less on the difficult-to-learn semantic knowledge that links captions and imagery, and should instead be achievable by lower-level correlations among image pixels. In practice, inverse models are trained or tuned separately for each inverse problem, e.g. by providing parts of images during training as an additional condition, to allow their application in realistic settings. However, we argue that this is not necessary and propose an algorithm for fast-constrained sampling in large pre-trained diffusion models (Stable Diffusion) that requires no expensive backpropagation operations through the model and produces results comparable even to the state-of-the-art \emph{tuned} models. Our method is based on a novel optimization perspective to sampling under constraints and employs a numerical approximation to the expensive gradients, previously computed using backpropagation, incurring significant speed-ups.

Title: Delving into the Reversal Curse: How Far Can Large Language Models Generalize?

• Authors: Zhengkai Lin, Zhihang Fu, Kai Liu, Liang Xie, Binbin Lin, Wenxiao Wang, Deng Cai, Yue Wu, Jieping Ye
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18808
• Pdf URL: https://arxiv.org/pdf/2410.18808
• Copy Paste: [[2410.18808]] Delving into the Reversal Curse: How Far Can Large Language Models Generalize?(https://arxiv.org/abs/2410.18808)
• Keywords: large language model
• Abstract: While large language models (LLMs) showcase unprecedented capabilities, they also exhibit certain inherent limitations when facing seemingly trivial tasks. A prime example is the recently debated "reversal curse", which surfaces when models, having been trained on the fact "A is B", struggle to generalize this knowledge to infer that "B is A". In this paper, we examine the manifestation of the reversal curse across various tasks and delve into both the generalization abilities and the problem-solving mechanisms of LLMs. This investigation leads to a series of significant insights: (1) LLMs are able to generalize to "B is A" when both A and B are presented in the context as in the case of a multiple-choice question. (2) This generalization ability is highly correlated to the structure of the fact "A is B" in the training documents. For example, this generalization only applies to biographies structured in "[Name] is [Description]" but not to "[Description] is [Name]". (3) We propose and verify the hypothesis that LLMs possess an inherent bias in fact recalling during knowledge application, which explains and underscores the importance of the document structure to successful learning. (4) The negative impact of this bias on the downstream performance of LLMs can hardly be mitigated through training alone. Based on these intriguing findings, our work not only presents a novel perspective for interpreting LLMs' generalization abilities from their intrinsic working mechanism but also provides new insights for the development of more effective learning methods for LLMs.

Title: Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis

• Authors: Liang Han, Junsheng Zhou, Yu-Shen Liu, Zhizhong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18822
• Pdf URL: https://arxiv.org/pdf/2410.18822
• Copy Paste: [[2410.18822]] Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis(https://arxiv.org/abs/2410.18822)
• Keywords: robust
• Abstract: Novel view synthesis from sparse inputs is a vital yet challenging task in 3D computer vision. Previous methods explore 3D Gaussian Splatting with neural priors (e.g. depth priors) as an additional supervision, demonstrating promising quality and efficiency compared to the NeRF based methods. However, the neural priors from 2D pretrained models are often noisy and blurry, which struggle to precisely guide the learning of radiance fields. In this paper, We propose a novel method for synthesizing novel views from sparse views with Gaussian Splatting that does not require external prior as supervision. Our key idea lies in exploring the self-supervisions inherent in the binocular stereo consistency between each pair of binocular images constructed with disparity-guided image warping. To this end, we additionally introduce a Gaussian opacity constraint which regularizes the Gaussian locations and avoids Gaussian redundancy for improving the robustness and efficiency of inferring 3D Gaussians from sparse views. Extensive experiments on the LLFF, DTU, and Blender datasets demonstrate that our method significantly outperforms the state-of-the-art methods.

Title: PSY: Posterior Sampling Based Privacy Enhancer in Large Language Models

• Authors: Yulian Sun, Li Duan, Yong Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18824
• Pdf URL: https://arxiv.org/pdf/2410.18824
• Copy Paste: [[2410.18824]] PSY: Posterior Sampling Based Privacy Enhancer in Large Language Models(https://arxiv.org/abs/2410.18824)
• Keywords: privacy, attack, extraction, membership infer, large language model
• Abstract: Privacy vulnerabilities in LLMs, such as leakage from memorization, have been constantly identified, and various mitigation proposals have been proposed. LoRA is usually used in fine-tuning LLMs and a good entry point to insert privacy-enhancing modules. In this ongoing research, we introduce PSY, a Posterior Sampling based PrivacY enhancer that can be used in LoRA. We propose a simple yet effective realization of PSY using posterior sampling, which effectively prevents privacy leakage from intermediate information and, in turn, preserves the privacy of data owners. We evaluate LoRA extended with PSY against state-of-the-art membership inference and data extraction attacks. The experiments are executed on three different LLM architectures fine-tuned on three datasets with LoRA. In contrast to the commonly used differential privacy method, we find that our proposed modification consistently reduces the attack success rate. Meanwhile, our method has almost no negative impact on model fine-tuning or final performance. Most importantly, PSY reveals a promising path toward privacy enhancement with latent space extensions.

Title: Multi-Scale Diffusion: Enhancing Spatial Layout in High-Resolution Panoramic Image Generation

• Authors: Xiaoyu Zhang, Teng Zhou, Xinlong Zhang, Jia Wei, Yongchuan Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18830
• Pdf URL: https://arxiv.org/pdf/2410.18830
• Copy Paste: [[2410.18830]] Multi-Scale Diffusion: Enhancing Spatial Layout in High-Resolution Panoramic Image Generation(https://arxiv.org/abs/2410.18830)
• Keywords: diffusion
• Abstract: Diffusion models have recently gained recognition for generating diverse and high-quality content, especially in the domain of image synthesis. These models excel not only in creating fixed-size images but also in producing panoramic images. However, existing methods often struggle with spatial layout consistency when producing high-resolution panoramas, due to the lack of guidance of the global image layout. In this paper, we introduce the Multi-Scale Diffusion (MSD) framework, a plug-and-play module that extends the existing panoramic image generation framework to multiple resolution levels. By utilizing gradient descent techniques, our method effectively incorporates structural information from low-resolution images into high-resolution outputs. A comprehensive evaluation of the proposed method was conducted, comparing it with the prior works in qualitative and quantitative dimensions. The evaluation results demonstrate that our method significantly outperforms others in generating coherent high-resolution panoramas.

Title: From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages

• Authors: Artur Kiulian, Anton Polishko, Mykola Khandoga, Yevhen Kostiuk, Guillermo Gabrielli, Łukasz Gagała, Fadi Zaraket, Qusai Abu Obaida, Hrishikesh Garud, Wendy Wing Yee Mak, Dmytro Chaplynskyi, Selma Belhadj Amor, Grigol Peradze
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18836
• Pdf URL: https://arxiv.org/pdf/2410.18836
• Copy Paste: [[2410.18836]] From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages(https://arxiv.org/abs/2410.18836)
• Keywords: fair, large language model
• Abstract: In this paper, we propose a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language. The method includes vocabulary expansion, initialization of new embeddings, model training and evaluation. We performed our experiments with three languages, each using a non-Latin script - Ukrainian, Arabic, and Georgian. Our approach demonstrates improved language performance while reducing computational costs. It mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar. Additionally, we introduce new metrics to evaluate language quality, revealing that vocabulary size significantly impacts the quality of generated text.

Title: From Efficiency to Equity: Measuring Fairness in Preference Learning

• Authors: Shreeyash Gowaikar, Hugo Berard, Rashid Mushkani, Shin Koseki
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18841
• Pdf URL: https://arxiv.org/pdf/2410.18841
• Copy Paste: [[2410.18841]] From Efficiency to Equity: Measuring Fairness in Preference Learning(https://arxiv.org/abs/2410.18841)
• Keywords: fair, generative
• Abstract: As AI systems, particularly generative models, increasingly influence decision-making, ensuring that they are able to fairly represent diverse human preferences becomes crucial. This paper introduces a novel framework for evaluating epistemic fairness in preference learning models inspired by economic theories of inequality and Rawlsian justice. We propose metrics adapted from the Gini Coefficient, Atkinson Index, and Kuznets Ratio to quantify fairness in these models. We validate our approach using two datasets: a custom visual preference dataset (AI-EDI-Space) and the Jester Jokes dataset. Our analysis reveals variations in model performance across users, highlighting potential epistemic injustices. We explore pre-processing and in-processing techniques to mitigate these inequalities, demonstrating a complex relationship between model efficiency and fairness. This work contributes to AI ethics by providing a framework for evaluating and improving epistemic fairness in preference learning models, offering insights for developing more inclusive AI systems in contexts where diverse human preferences are crucial.

Title: Learning to Explore with Lagrangians for Bandits under Unknown Linear Constraints

• Authors: Udvas Das, Debabrota Basu
• Subjects: cs.LG, cs.AI, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.18844
• Pdf URL: https://arxiv.org/pdf/2410.18844
• Copy Paste: [[2410.18844]] Learning to Explore with Lagrangians for Bandits under Unknown Linear Constraints(https://arxiv.org/abs/2410.18844)
• Keywords: fair
• Abstract: Pure exploration in bandits models multiple real-world problems, such as tuning hyper-parameters or conducting user studies, where different safety, resource, and fairness constraints on the decision space naturally appear. We study these problems as pure exploration in multi-armed bandits with unknown linear constraints, where the aim is to identify an $r$$\textit{-good feasible policy}$. First, we propose a Lagrangian relaxation of the sample complexity lower bound for pure exploration under constraints. We show how this lower bound evolves with the sequential estimation of constraints. Second, we leverage the Lagrangian lower bound and the properties of convex optimisation to propose two computationally efficient extensions of Track-and-Stop and Gamified Explorer, namely LATS and LAGEX. To this end, we propose a constraint-adaptive stopping rule, and while tracking the lower bound, use pessimistic estimate of the feasible set at each step. We show that these algorithms achieve asymptotically optimal sample complexity upper bounds up to constraint-dependent constants. Finally, we conduct numerical experiments with different reward distributions and constraints that validate efficient performance of LAGEX and LATS with respect to baselines.

Title: We Augmented Whisper With kNN and You Won't Believe What Came Next

• Authors: Maya K. Nachesa, Vlad Niculae
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.18850
• Pdf URL: https://arxiv.org/pdf/2410.18850
• Copy Paste: [[2410.18850]] We Augmented Whisper With kNN and You Won't Believe What Came Next(https://arxiv.org/abs/2410.18850)
• Keywords: transformer
• Abstract: Speech recognition performance varies by language, domain, and speaker characteristics such as accent, and fine-tuning a model on any of these categories may lead to catastrophic forgetting. $k$ nearest neighbor search ($k$NN), first proposed for neural sequence decoders for natural language generation (NLG) and machine translation (MT), is a non-parametric method that can instead adapt by building an external datastore that can then be searched during inference time, without training the underlying model. We show that Whisper, a transformer end-to-end speech model, benefits from $k$NN. We investigate the differences between the speech and text setups. We discuss implications for speaker adaptation, and analyze improvements by gender, accent, and age.

Title: DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations

• Authors: Aryo Pradipta Gema, Chen Jin, Ahmed Abdulaal, Tom Diethe, Philip Teare, Beatrice Alex, Pasquale Minervini, Amrutha Saseendran
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18860
• Pdf URL: https://arxiv.org/pdf/2410.18860
• Copy Paste: [[2410.18860]] DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations(https://arxiv.org/abs/2410.18860)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) often hallucinate, producing unfaithful or factually incorrect outputs by misrepresenting the provided context or incorrectly recalling internal knowledge. Recent studies have identified specific attention heads within the Transformer architecture, known as retrieval heads, responsible for extracting relevant contextual information. We hypothesise that masking these retrieval heads can induce hallucinations and that contrasting the outputs of the base LLM and the masked LLM can reduce hallucinations. To this end, we propose Decoding by Contrasting Retrieval Heads (DeCoRe), a novel training-free decoding strategy that amplifies information found in the context and model parameters. DeCoRe mitigates potentially hallucinated responses by dynamically contrasting the outputs of the base LLM and the masked LLM, using conditional entropy as a guide. Our extensive experiments confirm that DeCoRe significantly improves performance on tasks requiring high contextual faithfulness, such as summarisation (XSum by 18.6%), instruction following (MemoTrap by 10.9%), and open-book question answering (NQ-Open by 2.4% and NQ-Swap by 5.5%).

Title: Provably Robust Watermarks for Open-Source Language Models

• Authors: Miranda Christ, Sam Gunn, Tal Malkin, Mariana Raykova
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18861
• Pdf URL: https://arxiv.org/pdf/2410.18861
• Copy Paste: [[2410.18861]] Provably Robust Watermarks for Open-Source Language Models(https://arxiv.org/abs/2410.18861)
• Keywords: attack, robust, watermark, generative, large language model
• Abstract: The recent explosion of high-quality language models has necessitated new methods for identifying AI-generated text. Watermarking is a leading solution and could prove to be an essential tool in the age of generative AI. Existing approaches embed watermarks at inference and crucially rely on the large language model (LLM) specification and parameters being secret, which makes them inapplicable to the open-source setting. In this work, we introduce the first watermarking scheme for open-source LLMs. Our scheme works by modifying the parameters of the model, but the watermark can be detected from just the outputs of the model. Perhaps surprisingly, we prove that our watermarks are unremovable under certain assumptions about the adversary's knowledge. To demonstrate the behavior of our construction under concrete parameter instantiations, we present experimental results with OPT-6.7B and OPT-1.3B. We demonstrate robustness to both token substitution and perturbation of the model parameters. We find that the stronger of these attacks, the model-perturbation attack, requires deteriorating the quality score to 0 out of 100 in order to bring the detection rate down to 50%.

Title: FedSPD: A Soft-clustering Approach for Personalized Decentralized Federated Learning

• Authors: I-Cheng Lin, Osman Yagan, Carlee Joe-Wong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18862
• Pdf URL: https://arxiv.org/pdf/2410.18862
• Copy Paste: [[2410.18862]] FedSPD: A Soft-clustering Approach for Personalized Decentralized Federated Learning(https://arxiv.org/abs/2410.18862)
• Keywords: federate
• Abstract: Federated learning has recently gained popularity as a framework for distributed clients to collaboratively train a machine learning model using local data. While traditional federated learning relies on a central server for model aggregation, recent advancements adopt a decentralized framework, enabling direct model exchange between clients and eliminating the single point of failure. However, existing decentralized frameworks often assume all clients train a shared model. Personalizing each client's model can enhance performance, especially with heterogeneous client data distributions. We propose FedSPD, an efficient personalized federated learning algorithm for the decentralized setting, and show that it learns accurate models even in low-connectivity networks. To provide theoretical guarantees on convergence, we introduce a clustering-based framework that enables consensus on models for distinct data clusters while personalizing to unique mixtures of these clusters at different clients. This flexibility, allowing selective model updates based on data distribution, substantially reduces communication costs compared to prior work on personalized federated learning in decentralized settings. Experimental results on real-world datasets show that FedSPD outperforms multiple decentralized variants of personalized federated learning algorithms, especially in scenarios with low-connectivity networks.

Title: Multi-Class Abnormality Classification in Video Capsule Endoscopy Using Deep Learning

• Authors: Arnav Samal, Ranya
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18879
• Pdf URL: https://arxiv.org/pdf/2410.18879
• Copy Paste: [[2410.18879]] Multi-Class Abnormality Classification in Video Capsule Endoscopy Using Deep Learning(https://arxiv.org/abs/2410.18879)
• Keywords: transformer
• Abstract: This report outlines Team Seq2Cure's deep learning approach for the Capsule Vision 2024 Challenge, leveraging an ensemble of convolutional neural networks (CNNs) and transformer-based architectures for multi-class abnormality classification in video capsule endoscopy frames. The dataset comprised over 50,000 frames from three public sources and one private dataset, labeled across 10 abnormality classes. To overcome the limitations of traditional CNNs in capturing global context, we integrated CNN and transformer models within a multi-model ensemble. Our approach achieved a balanced accuracy of 86.34 percent and a mean AUC-ROC score of 0.9908 on the validation set, with significant improvements in classifying complex abnormalities. Code is available at this http URL .

Title: Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences

• Authors: Weijian Luo
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18881
• Pdf URL: https://arxiv.org/pdf/2410.18881
• Copy Paste: [[2410.18881]] Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences(https://arxiv.org/abs/2410.18881)
• Keywords: diffusion, data-free
• Abstract: One-step text-to-image generator models offer advantages such as swift inference efficiency, flexible architectures, and state-of-the-art generation performance. In this paper, we study the problem of aligning one-step generator models with human preferences for the first time. Inspired by the success of reinforcement learning using human feedback (RLHF), we formulate the alignment problem as maximizing expected human reward functions while adding an Integral Kullback-Leibler divergence term to prevent the generator from diverging. By overcoming technical challenges, we introduce Diff-Instruct++ (DI++), the first, fast-converging and image data-free human preference alignment method for one-step text-to-image generators. We also introduce novel theoretical insights, showing that using CFG for diffusion distillation is secretly doing RLHF with DI++. Such an interesting finding brings understanding and potential contributions to future research involving CFG. In the experiment sections, we align both UNet-based and DiT-based one-step generators using DI++, which use the Stable Diffusion 1.5 and the PixelArt-$\alpha$ as the reference diffusion processes. The resulting DiT-based one-step text-to-image model achieves a strong Aesthetic Score of 6.19 and an Image Reward of 1.24 on the COCO validation prompt dataset. It also achieves a leading Human preference Score (HPSv2.0) of 28.48, outperforming other open-sourced models such as Stable Diffusion XL, DMD2, SD-Turbo, as well as PixelArt-$\alpha$. Both theoretical contributions and empirical evidence indicate that DI++ is a strong human-preference alignment approach for one-step text-to-image models.

Title: Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance

• Authors: Omer Nahum, Nitay Calderon, Orgad Keller, Idan Szpektor, Roi Reichart
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18889
• Pdf URL: https://arxiv.org/pdf/2410.18889
• Copy Paste: [[2410.18889]] Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance(https://arxiv.org/abs/2410.18889)
• Keywords: large language model
• Abstract: NLP benchmarks rely on standardized datasets for training and evaluating models and are crucial for advancing the field. Traditionally, expert annotations ensure high-quality labels; however, the cost of expert annotation does not scale well with the growing demand for larger datasets required by modern models. While crowd-sourcing provides a more scalable solution, it often comes at the expense of annotation precision and consistency. Recent advancements in large language models (LLMs) offer new opportunities to enhance the annotation process, particularly for detecting label errors in existing datasets. In this work, we consider the recent approach of LLM-as-a-judge, leveraging an ensemble of LLMs to flag potentially mislabeled examples. Through a case study of four datasets from the TRUE benchmark, covering different tasks and domains, we empirically analyze the labeling quality of existing datasets, and compare expert, crowd-sourced, and our LLM-based annotations in terms of agreement, label quality, and efficiency, demonstrating the strengths and limitations of each annotation method. Our findings reveal a substantial number of label errors, which, when corrected, induce a significant upward shift in reported model performance. This suggests that many of the LLMs so-called mistakes are due to label errors rather than genuine model failures. Additionally, we discuss the implications of mislabeled data and propose methods to mitigate them in training to improve model performance.

Title: Meta-Learning with Heterogeneous Tasks

• Authors: Zhaofeng Si, Shu Hu, Kaiyi Ji, Siwei Lyu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18894
• Pdf URL: https://arxiv.org/pdf/2410.18894
• Copy Paste: [[2410.18894]] Meta-Learning with Heterogeneous Tasks(https://arxiv.org/abs/2410.18894)
• Keywords: robust
• Abstract: Meta-learning is a general approach to equip machine learning models with the ability to handle few-shot scenarios when dealing with many tasks. Most existing meta-learning methods work based on the assumption that all tasks are of equal importance. However, real-world applications often present heterogeneous tasks characterized by varying difficulty levels, noise in training samples, or being distinctively different from most other tasks. In this paper, we introduce a novel meta-learning method designed to effectively manage such heterogeneous tasks by employing rank-based task-level learning objectives, Heterogeneous Tasks Robust Meta-learning (HeTRoM). HeTRoM is proficient in handling heterogeneous tasks, and it prevents easy tasks from overwhelming the meta-learner. The approach allows for an efficient iterative optimization algorithm based on bi-level optimization, which is then improved by integrating statistical guidance. Our experimental results demonstrate that our method provides flexibility, enabling users to adapt to diverse task settings and enhancing the meta-learner's overall performance.

Title: ArterialNet: Reconstructing Arterial Blood Pressure Waveform with Wearable Pulsatile Signals, a Cohort-Aware Approach

• Authors: Sicong Huang, Roozbeh Jafari, Bobak J. Mortazavi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18895
• Pdf URL: https://arxiv.org/pdf/2410.18895
• Copy Paste: [[2410.18895]] ArterialNet: Reconstructing Arterial Blood Pressure Waveform with Wearable Pulsatile Signals, a Cohort-Aware Approach(https://arxiv.org/abs/2410.18895)
• Keywords: robust, extraction
• Abstract: Continuous arterial blood pressure (ABP) monitoring is invasive but essential for hemodynamic monitoring. Recent techniques have reconstructed ABP non-invasively using pulsatile signals but produced inaccurate systolic and diastolic blood pressure (SBP and DBP) values and were sensitive to individual variability. ArterialNet integrates generalized pulsatile-to-ABP signal translation and personalized feature extraction using hybrid loss functions and regularization. We validated ArterialNet using the MIMIC-III dataset and achieved a root mean square error (RMSE) of 5.41 mmHg, with at least a 58% lower standard deviation. ArterialNet reconstructed ABP with an RMSE of 7.99 mmHg in remote health scenarios. ArterialNet achieved superior performance in ABP reconstruction and SBP and DBP estimations, with significantly reduced subject variance, demonstrating its potential in remote health settings. We also ablated ArterialNet architecture to investigate the contributions of each component and evaluated its translational impact and robustness by conducting a series of ablations on data quality and availability.

Title: LLMs for Extremely Low-Resource Finno-Ugric Languages

• Authors: Taido Purason, Hele-Andra Kuulmets, Mark Fishel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18902
• Pdf URL: https://arxiv.org/pdf/2410.18902
• Copy Paste: [[2410.18902]] LLMs for Extremely Low-Resource Finno-Ugric Languages(https://arxiv.org/abs/2410.18902)
• Keywords: large language model
• Abstract: The advancement of large language models (LLMs) has predominantly focused on high-resource languages, leaving low-resource languages, such as those in the Finno-Ugric family, significantly underrepresented. This paper addresses this gap by focusing on Võro, Livonian, and Komi. We cover almost the entire cycle of LLM creation, from data collection to instruction tuning and evaluation. Our contributions include developing multilingual base and instruction-tuned models; creating evaluation benchmarks, including the smugri-MT-bench multi-turn conversational benchmark; and conducting human evaluation. We intend for this work to promote linguistic diversity, ensuring that lesser-resourced languages can benefit from advancements in NLP.

Title: PRISM: A Methodology for Auditing Biases in Large Language Models

• Authors: Leif Azzopardi, Yashar Moshfeghi
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.18906
• Pdf URL: https://arxiv.org/pdf/2410.18906
• Copy Paste: [[2410.18906]] PRISM: A Methodology for Auditing Biases in Large Language Models(https://arxiv.org/abs/2410.18906)
• Keywords: large language model
• Abstract: Auditing Large Language Models (LLMs) to discover their biases and preferences is an emerging challenge in creating Responsible Artificial Intelligence (AI). While various methods have been proposed to elicit the preferences of such models, countermeasures have been taken by LLM trainers, such that LLMs hide, obfuscate or point blank refuse to disclosure their positions on certain subjects. This paper presents PRISM, a flexible, inquiry-based methodology for auditing LLMs - that seeks to illicit such positions indirectly through task-based inquiry prompting rather than direct inquiry of said preferences. To demonstrate the utility of the methodology, we applied PRISM on the Political Compass Test, where we assessed the political leanings of twenty-one LLMs from seven providers. We show LLMs, by default, espouse positions that are economically left and socially liberal (consistent with prior work). We also show the space of positions that these models are willing to espouse - where some models are more constrained and less compliant than others - while others are more neutral and objective. In sum, PRISM can more reliably probe and audit LLMs to understand their preferences, biases and constraints.

Title: From Blind Solvers to Logical Thinkers: Benchmarking LLMs' Logical Integrity on Faulty Mathematical Problems

• Authors: A M Muntasir Rahman, Junyi Ye, Wei Yao, Wenpeng Yin, Guiling Wang
• Subjects: cs.CL, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2410.18921
• Pdf URL: https://arxiv.org/pdf/2410.18921
• Copy Paste: [[2410.18921]] From Blind Solvers to Logical Thinkers: Benchmarking LLMs' Logical Integrity on Faulty Mathematical Problems(https://arxiv.org/abs/2410.18921)
• Keywords: large language model
• Abstract: Consider the math problem: "Lily received 3 cookies from her best friend yesterday and ate 5 for breakfast. Today, her friend gave her 3 more cookies. How many cookies does Lily have now?" Many large language models (LLMs) in previous research approach this problem by calculating the answer "1" using the equation "3 - 5 + 3." However, from a human perspective, we recognize the inherent flaw in this problem: Lily cannot eat 5 cookies if she initially only had 3. This discrepancy prompts a key question: Are current LLMs merely Blind Solver that apply mathematical operations without deeper reasoning, or can they function as Logical Thinker capable of identifying logical inconsistencies? To explore this question, we propose a benchmark dataset, FaultyMath, which includes faulty math problems of rich diversity: i) multiple mathematical categories, e.g., algebra, geometry, number theory, etc., ii) varying levels of difficulty, and iii) different origins of faultiness -- ranging from violations of common sense and ambiguous statements to mathematical contradictions and more. We evaluate a broad spectrum of LLMs, including open-source, closed-source, and math-specialized models, using FaultyMath across three dimensions: (i) How accurately can the models detect faulty math problems without being explicitly prompted to do so? (ii) When provided with hints -- either correct or misleading -- about the validity of the problems, to what extent do LLMs adapt to become reliable Logical Thinker? (iii) How trustworthy are the explanations generated by LLMs when they recognize a math problem as flawed? Through extensive experimentation and detailed analysis, our results demonstrate that existing LLMs largely function as Blind Solver and fall short of the reasoning capabilities required to perform as Logical Thinker.

Title: SegLLM: Multi-round Reasoning Segmentation

• Authors: XuDong Wang, Shaolun Zhang, Shufan Li, Konstantinos Kallidromitis, Kehan Li, Yusuke Kato, Kazuki Kozuka, Trevor Darrell
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18923
• Pdf URL: https://arxiv.org/pdf/2410.18923
• Copy Paste: [[2410.18923]] SegLLM: Multi-round Reasoning Segmentation(https://arxiv.org/abs/2410.18923)
• Keywords: segmentation
• Abstract: We present SegLLM, a novel multi-round interactive reasoning segmentation model that enhances LLM-based segmentation by exploiting conversational memory of both visual and textual outputs. By leveraging a mask-aware multimodal LLM, SegLLM re-integrates previous segmentation results into its input stream, enabling it to reason about complex user intentions and segment objects in relation to previously identified entities, including positional, interactional, and hierarchical relationships, across multiple interactions. This capability allows SegLLM to respond to visual and text queries in a chat-like manner. Evaluated on the newly curated MRSeg benchmark, SegLLM outperforms existing methods in multi-round interactive reasoning segmentation by over 20%. Additionally, we observed that training on multi-round reasoning segmentation data enhances performance on standard single-round referring segmentation and localization tasks, resulting in a 5.5% increase in cIoU for referring expression segmentation and a 4.5% improvement in Acc@0.5 for referring expression localization.

Title: SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models

• Authors: Zonghao Ying, Aishan Liu, Siyuan Liang, Lei Huang, Jinyang Guo, Wenbo Zhou, Xianglong Liu, Dacheng Tao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.18927
• Pdf URL: https://arxiv.org/pdf/2410.18927
• Copy Paste: [[2410.18927]] SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models(https://arxiv.org/abs/2410.18927)
• Keywords: security, large language model
• Abstract: Multimodal Large Language Models (MLLMs) are showing strong safety concerns (e.g., generating harmful outputs for users), which motivates the development of safety evaluation benchmarks. However, we observe that existing safety benchmarks for MLLMs show limitations in query quality and evaluation reliability limiting the detection of model safety implications as MLLMs continue to evolve. In this paper, we propose \toolns, a comprehensive framework designed for conducting safety evaluations of MLLMs. Our framework consists of a comprehensive harmful query dataset and an automated evaluation protocol that aims to address the above limitations, respectively. We first design an automatic safety dataset generation pipeline, where we employ a set of LLM judges to recognize and categorize the risk scenarios that are most harmful and diverse for MLLMs; based on the taxonomy, we further ask these judges to generate high-quality harmful queries accordingly resulting in 23 risk scenarios with 2,300 multi-modal harmful query pairs. During safety evaluation, we draw inspiration from the jury system in judicial proceedings and pioneer the jury deliberation evaluation protocol that adopts collaborative LLMs to evaluate whether target models exhibit specific harmful behaviors, providing a reliable and unbiased assessment of content security risks. In addition, our benchmark can also be extended to the audio modality showing high scalability and potential. Based on our framework, we conducted large-scale experiments on 15 widely-used open-source MLLMs and 6 commercial MLLMs (e.g., GPT-4o, Gemini), where we revealed widespread safety issues in existing MLLMs and instantiated several insights on MLLM safety performance such as image quality and parameter size.

Title: Dynamic Vocabulary Pruning in Early-Exit LLMs

• Authors: Jort Vincenti, Karim Abdel Sadek, Joan Velja, Matteo Nulli, Metod Jazbec
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18952
• Pdf URL: https://arxiv.org/pdf/2410.18952
• Copy Paste: [[2410.18952]] Dynamic Vocabulary Pruning in Early-Exit LLMs(https://arxiv.org/abs/2410.18952)
• Keywords: large language model
• Abstract: Increasing the size of large language models (LLMs) has been shown to lead to better performance. However, this comes at the cost of slower and more expensive inference. Early-exiting is a promising approach for improving the efficiency of LLM inference by enabling next token prediction at intermediate layers. Yet, the large vocabulary size in modern LLMs makes the confidence estimation required for exit decisions computationally expensive, diminishing the efficiency gains. To address this, we propose dynamically pruning the vocabulary at test time for each token. Specifically, the vocabulary is pruned at one of the initial layers, and the smaller vocabulary is then used throughout the rest of the forward pass. Our experiments demonstrate that such post-hoc dynamic vocabulary pruning improves the efficiency of confidence estimation in early-exit LLMs while maintaining competitive performance.

Title: BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuning

• Authors: Yujuan Velvin Fu, Giridhar Kaushik Ramachandran, Namu Park, Kevin Lybarger, Fei Xia, Ozlem Uzuner, Meliha Yetisgen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18955
• Pdf URL: https://arxiv.org/pdf/2410.18955
• Copy Paste: [[2410.18955]] BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuning(https://arxiv.org/abs/2410.18955)
• Keywords: extraction, large language model
• Abstract: Large language models (LLMs) such as ChatGPT are fine-tuned on large and diverse instruction-following corpora, and can generalize to new tasks. However, those instruction-tuned LLMs often perform poorly in specialized medical natural language understanding (NLU) tasks that require domain knowledge, granular text comprehension, and structured data extraction. To bridge the gap, we: (1) propose a unified prompting format for 7 important NLU tasks, % through span extraction and multi-choice question-answering (QA), (2) curate an instruction-tuning dataset, MNLU-Instruct, utilizing diverse existing open-source medical NLU corpora, and (3) develop BioMistral-NLU, a generalizable medical NLU model, through fine-tuning BioMistral on MNLU-Instruct. We evaluate BioMistral-NLU in a zero-shot setting, across 6 important NLU tasks, from two widely adopted medical NLU benchmarks: Biomedical Language Understanding Evaluation (BLUE) and Biomedical Language Understanding and Reasoning Benchmark (BLURB). Our experiments show that our BioMistral-NLU outperforms the original BioMistral, as well as the proprietary LLMs - ChatGPT and GPT-4. Our dataset-agnostic prompting strategy and instruction tuning step over diverse NLU tasks enhance LLMs' generalizability across diverse medical NLU tasks. Our ablation experiments show that instruction-tuning on a wider variety of tasks, even when the total number of training instances remains constant, enhances downstream zero-shot generalization.

Title: Large Spatial Model: End-to-end Unposed Images to Semantic 3D

• Authors: Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjie Li, Kairun Wen, Shijie Zhou, Achuta Kadambi, Zhangyang Wang, Danfei Xu, Boris Ivanovic, Marco Pavone, Yue Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18956
• Pdf URL: https://arxiv.org/pdf/2410.18956
• Copy Paste: [[2410.18956]] Large Spatial Model: End-to-end Unposed Images to Semantic 3D(https://arxiv.org/abs/2410.18956)
• Keywords: transformer, segmentation
• Abstract: Reconstructing and understanding 3D structures from a limited number of images is a well-established problem in computer vision. Traditional methods usually break this task into multiple subtasks, each requiring complex transformations between different data representations. For instance, dense reconstruction through Structure-from-Motion (SfM) involves converting images into key points, optimizing camera parameters, and estimating structures. Afterward, accurate sparse reconstructions are required for further dense modeling, which is subsequently fed into task-specific neural networks. This multi-step process results in considerable processing time and increased engineering complexity. In this work, we present the Large Spatial Model (LSM), which processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation, and it can generate versatile label maps by interacting with language at novel viewpoints. Leveraging a Transformer-based architecture, LSM integrates global geometry through pixel-aligned point maps. To enhance spatial attribute regression, we incorporate local context aggregation with multi-scale fusion, improving the accuracy of fine local details. To tackle the scarcity of labeled 3D semantic data and enable natural language-driven scene manipulation, we incorporate a pre-trained 2D language-based segmentation model into a 3D-consistent semantic feature field. An efficient decoder then parameterizes a set of semantic anisotropic Gaussians, facilitating supervised end-to-end learning. Extensive experiments across various tasks show that LSM unifies multiple 3D vision tasks directly from unposed images, achieving real-time semantic 3D reconstruction for the first time.

Title: Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code

• Authors: Jipeng Zhang, Jianshu Zhang, Yuanzhe Li, Renjie Pi, Rui Pan, Runtao Liu, Ziqiang Zheng, Tong Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18957
• Pdf URL: https://arxiv.org/pdf/2410.18957
• Copy Paste: [[2410.18957]] Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code(https://arxiv.org/abs/2410.18957)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) demonstrate strong proficiency in generating code for high-resource programming languages (HRPLs) like Python but struggle significantly with low-resource programming languages (LRPLs) such as Racket or D. This performance gap deepens the digital divide, preventing developers using LRPLs from benefiting equally from LLM advancements and reinforcing disparities in innovation within underrepresented programming communities. While generating additional training data for LRPLs is promising, it faces two key challenges: manual annotation is labor-intensive and costly, and LLM-generated LRPL code is often of subpar quality. The underlying cause of this issue is the gap between natural language to programming language gap (NL-PL Gap), which is especially pronounced in LRPLs due to limited aligned data. In this work, we introduce a novel approach called Bridge-Coder, which leverages LLMs' intrinsic capabilities to enhance the performance on LRPLs. Our method consists of two key stages. Bridge Generation, where we create high-quality dataset by utilizing LLMs' general knowledge understanding, proficiency in HRPLs, and in-context learning abilities. Then, we apply the Bridged Alignment, which progressively improves the alignment between NL instructions and LRPLs. Experimental results across multiple LRPLs show that Bridge-Coder significantly enhances model performance, demonstrating the effectiveness and generalization of our approach. Furthermore, we offer a detailed analysis of the key components of our method, providing valuable insights for future work aimed at addressing the challenges associated with LRPLs.

Title: Stable Consistency Tuning: Understanding and Improving Consistency Models

• Authors: Fu-Yun Wang, Zhengyang Geng, Hongsheng Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18958
• Pdf URL: https://arxiv.org/pdf/2410.18958
• Copy Paste: [[2410.18958]] Stable Consistency Tuning: Understanding and Improving Consistency Models(https://arxiv.org/abs/2410.18958)
• Keywords: diffusion, generative
• Abstract: Diffusion models achieve superior generation quality but suffer from slow generation speed due to the iterative nature of denoising. In contrast, consistency models, a new generative family, achieve competitive performance with significantly faster sampling. These models are trained either through consistency distillation, which leverages pretrained diffusion models, or consistency training/tuning directly from raw data. In this work, we propose a novel framework for understanding consistency models by modeling the denoising process of the diffusion model as a Markov Decision Process (MDP) and framing consistency model training as the value estimation through Temporal Difference~(TD) Learning. More importantly, this framework allows us to analyze the limitations of current consistency training/tuning strategies. Built upon Easy Consistency Tuning (ECT), we propose Stable Consistency Tuning (SCT), which incorporates variance-reduced learning using the score identity. SCT leads to significant performance improvements on benchmarks such as CIFAR-10 and ImageNet-64. On ImageNet-64, SCT achieves 1-step FID 2.42 and 2-step FID 1.55, a new SoTA for consistency models.

Title: Where Am I and What Will I See: An Auto-Regressive Model for Spatial Localization and View Prediction

• Authors: Junyi Chen, Di Huang, Weicai Ye, Wanli Ouyang, Tong He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18962
• Pdf URL: https://arxiv.org/pdf/2410.18962
• Copy Paste: [[2410.18962]] Where Am I and What Will I See: An Auto-Regressive Model for Spatial Localization and View Prediction(https://arxiv.org/abs/2410.18962)
• Keywords: transformer, generative
• Abstract: Spatial intelligence is the ability of a machine to perceive, reason, and act in three dimensions within space and time. Recent advancements in large-scale auto-regressive models have demonstrated remarkable capabilities across various reasoning tasks. However, these models often struggle with fundamental aspects of spatial reasoning, particularly in answering questions like "Where am I?" and "What will I see?". While some attempts have been done, existing approaches typically treat them as separate tasks, failing to capture their interconnected nature. In this paper, we present Generative Spatial Transformer (GST), a novel auto-regressive framework that jointly addresses spatial localization and view prediction. Our model simultaneously estimates the camera pose from a single image and predicts the view from a new camera pose, effectively bridging the gap between spatial awareness and visual prediction. The proposed innovative camera tokenization method enables the model to learn the joint distribution of 2D projections and their corresponding spatial perspectives in an auto-regressive manner. This unified training paradigm demonstrates that joint optimization of pose estimation and novel view synthesis leads to improved performance in both tasks, for the first time, highlighting the inherent relationship between spatial awareness and visual prediction.

Title: On the Crucial Role of Initialization for Matrix Factorization

• Authors: Bingcong Li, Liang Zhang, Aryan Mokhtari, Niao He
• Subjects: cs.LG, eess.SP, math.OC
• Abstract URL: https://arxiv.org/abs/2410.18965
• Pdf URL: https://arxiv.org/pdf/2410.18965
• Copy Paste: [[2410.18965]] On the Crucial Role of Initialization for Matrix Factorization(https://arxiv.org/abs/2410.18965)
• Keywords: diffusion
• Abstract: This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth optimization. We introduce Nystrom initialization, which significantly improves the global convergence of Scaled Gradient Descent (ScaledGD) in both symmetric and asymmetric matrix factorization tasks. Specifically, we prove that ScaledGD with Nystrom initialization achieves quadratic convergence in cases where only linear rates were previously known. Furthermore, we extend this initialization to low-rank adapters (LoRA) commonly used for finetuning foundation models. Our approach, NoRA, i.e., LoRA with Nystrom initialization, demonstrates superior performance across various downstream tasks and model scales, from 1B to 7B parameters, in large language and diffusion models.

Title: Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions

• Authors: Yujuan Fu, Ozlem Uzuner, Meliha Yetisgen, Fei Xia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18966
• Pdf URL: https://arxiv.org/pdf/2410.18966
• Copy Paste: [[2410.18966]] Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions(https://arxiv.org/abs/2410.18966)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated great performance across various benchmarks, showing potential as general-purpose task solvers. However, as LLMs are typically trained on vast amounts of data, a significant concern in their evaluation is data contamination, where overlap between training data and evaluation datasets inflates performance assessments. While multiple approaches have been developed to identify data contamination, these approaches rely on specific assumptions that may not hold universally across different settings. To bridge this gap, we systematically review 47 papers on data contamination detection, categorize the underlying assumptions, and assess whether they have been rigorously validated. We identify and analyze eight categories of assumptions and test three of them as case studies. Our analysis reveals that when classifying instances used for pretraining LLMs, detection approaches based on these three assumptions perform close to random guessing, suggesting that current LLMs learn data distributions rather than memorizing individual instances. Overall, this work underscores the importance of approaches clearly stating their underlying assumptions and testing their validity across various scenarios.

Title: Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms

• Authors: Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, Zhe Gan
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18967
• Pdf URL: https://arxiv.org/pdf/2410.18967
• Copy Paste: [[2410.18967]] Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms(https://arxiv.org/abs/2410.18967)
• Keywords: large language model
• Abstract: Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities.

Title: Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques

• Authors: David Ortiz-Perez, Manuel Benavent-Lledo, Jose Garcia-Rodriguez, David Tomás, M. Flores Vizcaya-Moreno
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18972
• Pdf URL: https://arxiv.org/pdf/2410.18972
• Copy Paste: [[2410.18972]] Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques(https://arxiv.org/abs/2410.18972)
• Keywords: transformer
• Abstract: Cognitive decline is a natural part of aging, often resulting in reduced cognitive abilities. In some cases, however, this decline is more pronounced, typically due to disorders such as Alzheimer's disease. Early detection of anomalous cognitive decline is crucial, as it can facilitate timely professional intervention. While medical data can help in this detection, it often involves invasive procedures. An alternative approach is to employ non-intrusive techniques such as speech or handwriting analysis, which do not necessarily affect daily activities. This survey reviews the most relevant methodologies that use deep learning techniques to automate the cognitive decline estimation task, including audio, text, and visual processing. We discuss the key features and advantages of each modality and methodology, including state-of-the-art approaches like Transformer architecture and foundation models. In addition, we present works that integrate different modalities to develop multimodal models. We also highlight the most significant datasets and the quantitative results from studies using these resources. From this review, several conclusions emerge. In most cases, the textual modality achieves the best results and is the most relevant for detecting cognitive decline. Moreover, combining various approaches from individual modalities into a multimodal model consistently enhances performance across nearly all scenarios.

Title: 3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation

• Authors: Hansheng Chen, Bokui Shen, Yulin Liu, Ruoxi Shi, Linqi Zhou, Connor Z. Lin, Jiayuan Gu, Hao Su, Gordon Wetzstein, Leonidas Guibas
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18974
• Pdf URL: https://arxiv.org/pdf/2410.18974
• Copy Paste: [[2410.18974]] 3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation(https://arxiv.org/abs/2410.18974)
• Keywords: diffusion
• Abstract: Multi-view image diffusion models have significantly advanced open-domain 3D object generation. However, most existing models rely on 2D network architectures that lack inherent 3D biases, resulting in compromised geometric consistency. To address this challenge, we introduce 3D-Adapter, a plug-in module designed to infuse 3D geometry awareness into pretrained image diffusion models. Central to our approach is the idea of 3D feedback augmentation: for each denoising step in the sampling loop, 3D-Adapter decodes intermediate multi-view features into a coherent 3D representation, then re-encodes the rendered RGBD views to augment the pretrained base model through feature addition. We study two variants of 3D-Adapter: a fast feed-forward version based on Gaussian splatting and a versatile training-free version utilizing neural fields and meshes. Our extensive experiments demonstrate that 3D-Adapter not only greatly enhances the geometry quality of text-to-multi-view models such as Instant3D and Zero123++, but also enables high-quality 3D generation using the plain text-to-image Stable Diffusion. Furthermore, we showcase the broad application potential of 3D-Adapter by presenting high quality results in text-to-3D, image-to-3D, text-to-texture, and text-to-avatar tasks.

Title: Unbounded: A Generative Infinite Game of Character Life Simulation

• Authors: Jialu Li, Yuanzhen Li, Neal Wadhwa, Yael Pritch, David E. Jacobs, Michael Rubinstein, Mohit Bansal, Nataniel Ruiz
• Subjects: cs.CV, cs.AI, cs.CL, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18975
• Pdf URL: https://arxiv.org/pdf/2410.18975
• Copy Paste: [[2410.18975]] Unbounded: A Generative Infinite Game of Character Life Simulation(https://arxiv.org/abs/2410.18975)
• Keywords: generative, large language model
• Abstract: We introduce the concept of a generative infinite game, a video game that transcends the traditional boundaries of finite, hard-coded systems by using generative models. Inspired by James P. Carse's distinction between finite and infinite games, we leverage recent advances in generative AI to create Unbounded: a game of character life simulation that is fully encapsulated in generative models. Specifically, Unbounded draws inspiration from sandbox life simulations and allows you to interact with your autonomous virtual character in a virtual world by feeding, playing with and guiding it - with open-ended mechanics generated by an LLM, some of which can be emergent. In order to develop Unbounded, we propose technical innovations in both the LLM and visual generation domains. Specifically, we present: (1) a specialized, distilled large language model (LLM) that dynamically generates game mechanics, narratives, and character interactions in real-time, and (2) a new dynamic regional image prompt Adapter (IP-Adapter) for vision models that ensures consistent yet flexible visual generation of a character across multiple environments. We evaluate our system through both qualitative and quantitative analysis, showing significant improvements in character life simulation, user instruction following, narrative coherence, and visual consistency for both characters and the environments compared to traditional related approaches.

Title: MotionCLR: Motion Generation and Training-free Editing via Understanding Attention Mechanisms

• Authors: Ling-Hao Chen, Wenxun Dai, Xuan Ju, Shunlin Lu, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.18977
• Pdf URL: https://arxiv.org/pdf/2410.18977
• Copy Paste: [[2410.18977]] MotionCLR: Motion Generation and Training-free Editing via Understanding Attention Mechanisms(https://arxiv.org/abs/2410.18977)
• Keywords: explainability, diffusion
• Abstract: This research delves into the problem of interactive editing of human motion generation. Previous motion diffusion models lack explicit modeling of the word-level text-motion correspondence and good explainability, hence restricting their fine-grained editing ability. To address this issue, we propose an attention-based motion diffusion model, namely MotionCLR, with CLeaR modeling of attention mechanisms. Technically, MotionCLR models the in-modality and cross-modality interactions with self-attention and cross-attention, respectively. More specifically, the self-attention mechanism aims to measure the sequential similarity between frames and impacts the order of motion features. By contrast, the cross-attention mechanism works to find the fine-grained word-sequence correspondence and activate the corresponding timesteps in the motion sequence. Based on these key properties, we develop a versatile set of simple yet effective motion editing methods via manipulating attention maps, such as motion (de-)emphasizing, in-place motion replacement, and example-based motion generation, etc. For further verification of the explainability of the attention mechanism, we additionally explore the potential of action-counting and grounded motion generation ability via attention maps. Our experimental results show that our method enjoys good generation and editing ability with good explainability.

Title: PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views

• Authors: Xin Fei, Wenzhao Zheng, Yueqi Duan, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, Jiwen Lu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18979
• Pdf URL: https://arxiv.org/pdf/2410.18979
• Copy Paste: [[2410.18979]] PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views(https://arxiv.org/abs/2410.18979)
• Keywords: transformer
• Abstract: We propose PixelGaussian, an efficient feed-forward framework for learning generalizable 3D Gaussian reconstruction from arbitrary views. Most existing methods rely on uniform pixel-wise Gaussian representations, which learn a fixed number of 3D Gaussians for each view and cannot generalize well to more input views. Differently, our PixelGaussian dynamically adapts both the Gaussian distribution and quantity based on geometric complexity, leading to more efficient representations and significant improvements in reconstruction quality. Specifically, we introduce a Cascade Gaussian Adapter to adjust Gaussian distribution according to local geometry complexity identified by a keypoint scorer. CGA leverages deformable attention in context-aware hypernetworks to guide Gaussian pruning and splitting, ensuring accurate representation in complex regions while reducing redundancy. Furthermore, we design a transformer-based Iterative Gaussian Refiner module that refines Gaussian representations through direct image-Gaussian interactions. Our PixelGaussian can effectively reduce Gaussian redundancy as input views increase. We conduct extensive experiments on the large-scale ACID and RealEstate10K datasets, where our method achieves state-of-the-art performance with good generalization to various numbers of views. Code: this https URL.