2024-09-11

Title: Transformer-Enhanced Iterative Feedback Mechanism for Polyp Segmentation

• Authors: Nikhil Kumar Tomar, Debesh Jha, Koushik Biswas, Tyler M. Berzin, Rajesh Keswani, Michael Wallace, Ulas Bagci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05875
• Pdf URL: https://arxiv.org/pdf/2409.05875
• Copy Paste: [[2409.05875]] Transformer-Enhanced Iterative Feedback Mechanism for Polyp Segmentation(https://arxiv.org/abs/2409.05875)
• Keywords: transformer, segmentation
• Abstract: Colorectal cancer (CRC) is the third most common cause of cancer diagnosed in the United States and the second leading cause of cancer-related death among both genders. Notably, CRC is the leading cause of cancer in younger men less than 50 years old. Colonoscopy is considered the gold standard for the early diagnosis of CRC. Skills vary significantly among endoscopists, and a high miss rate is reported. Automated polyp segmentation can reduce the missed rates, and timely treatment is possible in the early stage. To address this challenge, we introduce \textit{\textbf{\ac{FANetv2}}}, an advanced encoder-decoder network designed to accurately segment polyps from colonoscopy images. Leveraging an initial input mask generated by Otsu thresholding, FANetv2 iteratively refines its binary segmentation masks through a novel feedback attention mechanism informed by the mask predictions of previous epochs. Additionally, it employs a text-guided approach that integrates essential information about the number (one or many) and size (small, medium, large) of polyps to further enhance its feature representation capabilities. This dual-task approach facilitates accurate polyp segmentation and aids in the auxiliary classification of polyp attributes, significantly boosting the model's performance. Our comprehensive evaluations on the publicly available BKAI-IGH and CVC-ClinicDB datasets demonstrate the superior performance of FANetv2, evidenced by high dice similarity coefficients (DSC) of 0.9186 and 0.9481, along with low Hausdorff distances of 2.83 and 3.19, respectively. The source code for FANetv2 is available at this https URL.

Title: A Dual-Path neural network model to construct the flame nonlinear thermoacoustic response in the time domain

• Authors: Jiawei Wu, Teng Wang, Jiaqi Nan, Lijun Yang, Jingxuan Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05885
• Pdf URL: https://arxiv.org/pdf/2409.05885
• Copy Paste: [[2409.05885]] A Dual-Path neural network model to construct the flame nonlinear thermoacoustic response in the time domain(https://arxiv.org/abs/2409.05885)
• Keywords: robust
• Abstract: Traditional numerical simulation methods require substantial computational resources to accurately determine the complete nonlinear thermoacoustic response of flames to various perturbation frequencies and amplitudes. In this paper, we have developed deep learning algorithms that can construct a comprehensive flame nonlinear response from limited numerical simulation data. To achieve this, we propose using a frequency-sweeping data type as the training dataset, which incorporates a rich array of learnable information within a constrained dataset. To enhance the precision in learning flame nonlinear response patterns from the training data, we introduce a Dual-Path neural network. This network consists of a Chronological Feature Path and a Temporal Detail Feature Path. The Dual-Path network is specifically designed to focus intensively on the temporal characteristics of velocity perturbation sequences, yielding more accurate flame response patterns and enhanced generalization capabilities. Validations confirm that our approach can accurately model flame nonlinear responses, even under conditions of significant nonlinearity, and exhibits robust generalization capabilities across various test scenarios.

Title: OPAL: Outlier-Preserved Microscaling Quantization A ccelerator for Generative Large Language Models

• Authors: Jahyun Koo, Dahoon Park, Sangwoo Jung, Jaeha Kung
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2409.05902
• Pdf URL: https://arxiv.org/pdf/2409.05902
• Copy Paste: [[2409.05902]] OPAL: Outlier-Preserved Microscaling Quantization A ccelerator for Generative Large Language Models(https://arxiv.org/abs/2409.05902)
• Keywords: generative, large language model
• Abstract: To overcome the burden on the memory size and bandwidth due to ever-increasing size of large language models (LLMs), aggressive weight quantization has been recently studied, while lacking research on quantizing activations. In this paper, we present a hardware-software co-design method that results in an energy-efficient LLM accelerator, named OPAL, for generation tasks. First of all, a novel activation quantization method that leverages the microscaling data format while preserving several outliers per sub-tensor block (e.g., four out of 128 elements) is proposed. Second, on top of preserving outliers, mixed precision is utilized that sets 5-bit for inputs to sensitive layers in the decoder block of an LLM, while keeping inputs to less sensitive layers to 3-bit. Finally, we present the OPAL hardware architecture that consists of FP units for handling outliers and vectorized INT multipliers for dominant non-outlier related operations. In addition, OPAL uses log2-based approximation on softmax operations that only requires shift and subtraction to maximize power efficiency. As a result, we are able to improve the energy efficiency by 1.6~2.2x, and reduce the area by 2.4~3.1x with negligible accuracy loss, i.e., <1 perplexity increase.

Title: STLLM-DF: A Spatial-Temporal Large Language Model with Diffusion for Enhanced Multi-Mode Traffic System Forecasting

• Authors: Zhiqi Shao, Haoning Xi, Haohui Lu, Ze Wang, Michael G.H. Bell, Junbin Gao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05921
• Pdf URL: https://arxiv.org/pdf/2409.05921
• Copy Paste: [[2409.05921]] STLLM-DF: A Spatial-Temporal Large Language Model with Diffusion for Enhanced Multi-Mode Traffic System Forecasting(https://arxiv.org/abs/2409.05921)
• Keywords: robust, diffusion, transformer, large language model
• Abstract: The rapid advancement of Intelligent Transportation Systems (ITS) presents challenges, particularly with missing data in multi-modal transportation and the complexity of handling diverse sequential tasks within a centralized framework. To address these issues, we propose the Spatial-Temporal Large Language Model Diffusion (STLLM-DF), an innovative model that leverages Denoising Diffusion Probabilistic Models (DDPMs) and Large Language Models (LLMs) to improve multi-task transportation prediction. The DDPM's robust denoising capabilities enable it to recover underlying data patterns from noisy inputs, making it particularly effective in complex transportation systems. Meanwhile, the non-pretrained LLM dynamically adapts to spatial-temporal relationships within multi-modal networks, allowing the system to efficiently manage diverse transportation tasks in both long-term and short-term predictions. Extensive experiments demonstrate that STLLM-DF consistently outperforms existing models, achieving an average reduction of 2.40\% in MAE, 4.50\% in RMSE, and 1.51\% in MAPE. This model significantly advances centralized ITS by enhancing predictive accuracy, robustness, and overall system performance across multiple tasks, thus paving the way for more effective spatio-temporal traffic forecasting through the integration of frozen transformer language models and diffusion techniques.

Title: A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets

• Authors: Mariana Yukari Noguti, Edduardo Vellasques, Luiz Eduardo Soares Oliveira
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05972
• Pdf URL: https://arxiv.org/pdf/2409.05972
• Copy Paste: [[2409.05972]] A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets(https://arxiv.org/abs/2409.05972)
• Keywords: transformer
• Abstract: Recent advances in language modelling has significantly decreased the need of labelled data in text classification tasks. Transformer-based models, pre-trained on unlabeled data, can outmatch the performance of models trained from scratch for each task. However, the amount of labelled data need to fine-tune such type of model is still considerably high for domains requiring expert-level annotators, like the legal domain. This paper investigates the best strategies for optimizing the use of a small labeled dataset and large amounts of unlabeled data and perform a classification task in the legal area with 50 predefined topics. More specifically, we use the records of demands to a Brazilian Public Prosecutor's Office aiming to assign the descriptions in one of the subjects, which currently demands deep legal knowledge for manual filling. The task of optimizing the performance of classifiers in this scenario is especially challenging, given the low amount of resources available regarding the Portuguese language, especially in the legal domain. Our results demonstrate that classic supervised models such as logistic regression and SVM and the ensembles random forest and gradient boosting achieve better performance along with embeddings extracted with word2vec when compared to BERT language model. The latter demonstrates superior performance in association with the architecture of the model itself as a classifier, having surpassed all previous models in that regard. The best result was obtained with Unsupervised Data Augmentation (UDA), which jointly uses BERT, data augmentation, and strategies of semi-supervised learning, with an accuracy of 80.7% in the aforementioned task.

Title: CoDiCast: Conditional Diffusion Model for Weather Prediction with Uncertainty Quantification

• Authors: Jimeng Shi, Bowen Jin, Jiawei Han, Giri Narasimhan
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2409.05975
• Pdf URL: https://arxiv.org/pdf/2409.05975
• Copy Paste: [[2409.05975]] CoDiCast: Conditional Diffusion Model for Weather Prediction with Uncertainty Quantification(https://arxiv.org/abs/2409.05975)
• Keywords: diffusion
• Abstract: Accurate weather forecasting is critical for science and society. Yet, existing methods have not managed to simultaneously have the properties of high accuracy, low uncertainty, and high computational efficiency. On one hand, to quantify the uncertainty in weather predictions, the strategy of ensemble forecast (i.e., generating a set of diverse predictions) is often employed. However, traditional ensemble numerical weather prediction (NWP) is computationally intensive. On the other hand, most existing machine learning-based weather prediction (MLWP) approaches are efficient and accurate. Nevertheless, they are deterministic and cannot capture the uncertainty of weather forecasting. In this work, we propose CoDiCast, a conditional diffusion model to generate accurate global weather prediction, while achieving uncertainty quantification with ensemble forecasts and modest computational cost. The key idea is to simulate a conditional version of the reverse denoising process in diffusion models, which starts from pure Gaussian noise to generate realistic weather scenarios for a future time point. Each denoising step is conditioned on observations from the recent past. Ensemble forecasts are achieved by repeatedly sampling from stochastic Gaussian noise to represent uncertainty quantification. CoDiCast is trained on a decade of ERA5 reanalysis data from the European Centre for Medium-Range Weather Forecasts (ECMWF). Experimental results demonstrate that our approach outperforms several existing data-driven methods in accuracy. Our conditional diffusion model, CoDiCast, can generate 3-day global weather forecasts, at 6-hour steps and $5.625^\circ$ latitude-longitude resolution, for over 5 variables, in about 12 minutes on a commodity A100 GPU machine with 80GB memory. The open-souced code is provided at \url{this https URL}.

Title: FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

• Authors: Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, Ang Li
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.05976
• Pdf URL: https://arxiv.org/pdf/2409.05976
• Copy Paste: [[2409.05976]] FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations(https://arxiv.org/abs/2409.05976)
• Keywords: privacy, federate, large language model
• Abstract: The rapid development of Large Language Models (LLMs) has been pivotal in advancing AI, with pre-trained LLMs being adaptable to diverse downstream tasks through fine-tuning. Federated learning (FL) further enhances fine-tuning in a privacy-aware manner by utilizing clients' local data through in-situ computation, eliminating the need for data movement. However, fine-tuning LLMs, given their massive scale of parameters, poses challenges for clients with constrained and heterogeneous resources in FL. Previous methods employed low-rank adaptation (LoRA) for efficient federated fine-tuning but utilized traditional FL aggregation strategies on LoRA adapters. These approaches led to mathematically inaccurate aggregation noise, reducing fine-tuning effectiveness and failing to address heterogeneous LoRAs. In this work, we first highlight the mathematical incorrectness of LoRA aggregation in existing federated fine-tuning methods. We introduce a new approach called FLORA that enables federated fine-tuning on heterogeneous LoRA adapters across clients through a novel stacking-based aggregation method. Our approach is noise-free and seamlessly supports heterogeneous LoRA adapters. Extensive experiments demonstrate FLORA' s superior performance in both homogeneous and heterogeneous settings, surpassing state-of-the-art methods. We envision this work as a milestone for efficient, privacy-preserving, and accurate federated fine-tuning of LLMs. Our code is available at this https URL.

Title: AI for Mathematics Mathematical Formalized Problem Solving and Theorem Proving in Different Fields in Lean4

• Authors: Xichen Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05977
• Pdf URL: https://arxiv.org/pdf/2409.05977
• Copy Paste: [[2409.05977]] AI for Mathematics Mathematical Formalized Problem Solving and Theorem Proving in Different Fields in Lean4(https://arxiv.org/abs/2409.05977)
• Keywords: large language model
• Abstract: Using computerized verifiable formal languages like Lean 4 to prove mathematical theorems has a significant impact on mathematical formalization. Lean 4 offers prominent potential for advancing mathematical reasoning. However, existing efforts are limited to mathematical formalization languages in substantial online corpora and are dedicated to keeping pace with rapidly evolving languages. To bridge the gap between the traditional and computerized proof, my approach to formalizing theorem proving involves generating formal steps and complete proofs using Large Language Models (LLMs) based on Natural Language (NL) proofs. The method is to introduce the basic structure and tactics in general, determine how AI can assist the mathematical formalization process to improve its performance, and give examples of solving problems in Lean 4 comparing to NL, mainly in IMO, and a sample theorem proving in abstract algebra.

Title: FairHome: A Fair Housing and Fair Lending Dataset

• Authors: Anusha Bagalkotkar (1), Aveek Karmakar (1), Gabriel Arnson (1), Ondrej Linda (1) ((1) Zillow Group)
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05990
• Pdf URL: https://arxiv.org/pdf/2409.05990
• Copy Paste: [[2409.05990]] FairHome: A Fair Housing and Fair Lending Dataset(https://arxiv.org/abs/2409.05990)
• Keywords: protect, fair, large language model
• Abstract: We present a Fair Housing and Fair Lending dataset (FairHome): A dataset with around 75,000 examples across 9 protected categories. To the best of our knowledge, FairHome is the first publicly available dataset labeled with binary labels for compliance risk in the housing domain. We demonstrate the usefulness and effectiveness of such a dataset by training a classifier and using it to detect potential violations when using a large language model (LLM) in the context of real-estate transactions. We benchmark the trained classifier against state-of-the-art LLMs including GPT-3.5, GPT-4, LLaMA-3, and Mistral Large in both zero-shot and few-shot contexts. Our classifier outperformed with an F1-score of 0.91, underscoring the effectiveness of our dataset.

Title: TransformerRanker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks

• Authors: Lukas Garbas, Max Ploner, Alan Akbik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05997
• Pdf URL: https://arxiv.org/pdf/2409.05997
• Copy Paste: [[2409.05997]] TransformerRanker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks(https://arxiv.org/abs/2409.05997)
• Keywords: transformer
• Abstract: Classification tasks in NLP are typically addressed by selecting a pre-trained language model (PLM) from a model hub, and fine-tuning it for the task at hand. However, given the very large number of PLMs that are currently available, a practical challenge is to determine which of them will perform best for a specific downstream task. With this paper, we introduce TransformerRanker, a lightweight library that efficiently ranks PLMs for classification tasks without the need for computationally costly fine-tuning. Our library implements current approaches for transferability estimation (LogME, H-Score, kNN), in combination with layer aggregation options, which we empirically showed to yield state-of-the-art rankings of PLMs (Garbas et al., 2024). We designed the interface to be lightweight and easy to use, allowing users to directly connect to the HuggingFace Transformers and Dataset libraries. Users need only select a downstream classification task and a list of PLMs to create a ranking of likely best-suited PLMs for their task. We make TransformerRanker available as a pip-installable open-source library this https URL.

Title: Enhanced Generative Data Augmentation for Semantic Segmentation via Stronger Guidance

• Authors: Quang-Huy Che, Duc-Tri Le, Vinh-Tiep Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06002
• Pdf URL: https://arxiv.org/pdf/2409.06002
• Copy Paste: [[2409.06002]] Enhanced Generative Data Augmentation for Semantic Segmentation via Stronger Guidance(https://arxiv.org/abs/2409.06002)
• Keywords: diffusion, generative, segmentation
• Abstract: Data augmentation is a widely used technique for creating training data for tasks that require labeled data, such as semantic segmentation. This method benefits pixel-wise annotation tasks requiring much effort and intensive labor. Traditional data augmentation methods involve simple transformations like rotations and flips to create new images from existing ones. However, these new images may lack diversity along the main semantic axes in the data and not change high-level semantic properties. To address this issue, generative models have emerged as an effective solution for augmenting data by generating synthetic images. Controllable generative models offer a way to augment data for semantic segmentation tasks using a prompt and visual reference from the original image. However, using these models directly presents challenges, such as creating an effective prompt and visual reference to generate a synthetic image that accurately reflects the content and structure of the original. In this work, we introduce an effective data augmentation method for semantic segmentation using the Controllable Diffusion Model. Our proposed method includes efficient prompt generation using Class-Prompt Appending and Visual Prior Combination to enhance attention to labeled classes in real images. These techniques allow us to generate images that accurately depict segmented classes in the real image. In addition, we employ the class balancing algorithm to ensure efficiency when merging the synthetic and original images to generate balanced data for the training dataset. We evaluated our method on the PASCAL VOC datasets and found it highly effective for synthesizing images in semantic segmentation.

Title: Identifying the sources of ideological bias in GPT models through linguistic variation in output

• Authors: Christina Walker, Joan C. Timoneda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06043
• Pdf URL: https://arxiv.org/pdf/2409.06043
• Copy Paste: [[2409.06043]] Identifying the sources of ideological bias in GPT models through linguistic variation in output(https://arxiv.org/abs/2409.06043)
• Keywords: generative
• Abstract: Extant work shows that generative AI models such as GPT-3.5 and 4 perpetuate social stereotypes and biases. One concerning but less explored source of bias is ideology. Do GPT models take ideological stances on politically sensitive topics? In this article, we provide an original approach to identifying ideological bias in generative models, showing that bias can stem from both the training data and the filtering algorithm. We leverage linguistic variation in countries with contrasting political attitudes to evaluate bias in average GPT responses to sensitive political topics in those languages. First, we find that GPT output is more conservative in languages that map well onto conservative societies (i.e., Polish), and more liberal in languages used uniquely in liberal societies (i.e., Swedish). This result provides strong evidence of training data bias in GPT models. Second, differences across languages observed in GPT-3.5 persist in GPT-4, even though GPT-4 is significantly more liberal due to OpenAI's filtering policy. Our main takeaway is that generative model training must focus on high-quality, curated datasets to reduce bias, even if it entails a compromise in training data size. Filtering responses after training only introduces new biases and does not remove the underlying training biases.

Title: Statistical Mechanics of Min-Max Problems

• Authors: Yuma Ichikawa, Koji Hukushima
• Subjects: cs.LG, cond-mat.dis-nn, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.06053
• Pdf URL: https://arxiv.org/pdf/2409.06053
• Copy Paste: [[2409.06053]] Statistical Mechanics of Min-Max Problems(https://arxiv.org/abs/2409.06053)
• Keywords: fair, generative
• Abstract: Min-max optimization problems, also known as saddle point problems, have attracted significant attention due to their applications in various fields, such as fair beamforming, generative adversarial networks (GANs), and adversarial learning. However, understanding the properties of these min-max problems has remained a substantial challenge. This study introduces a statistical mechanical formalism for analyzing the equilibrium values of min-max problems in the high-dimensional limit, while appropriately addressing the order of operations for min and max. As a first step, we apply this formalism to bilinear min-max games and simple GANs, deriving the relationship between the amount of training data and generalization error and indicating the optimal ratio of fake to real data for effective learning. This formalism provides a groundwork for a deeper theoretical analysis of the equilibrium properties in various machine learning methods based on min-max problems and encourages the development of new algorithms and architectures.

Title: DiffusionPen: Towards Controlling the Style of Handwritten Text Generation

• Authors: Konstantina Nikolaidou, George Retsinas, Giorgos Sfikas, Marcus Liwicki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06065
• Pdf URL: https://arxiv.org/pdf/2409.06065
• Copy Paste: [[2409.06065]] DiffusionPen: Towards Controlling the Style of Handwritten Text Generation(https://arxiv.org/abs/2409.06065)
• Keywords: robust, diffusion
• Abstract: Handwritten Text Generation (HTG) conditioned on text and style is a challenging task due to the variability of inter-user characteristics and the unlimited combinations of characters that form new words unseen during training. Diffusion Models have recently shown promising results in HTG but still remain under-explored. We present DiffusionPen (DiffPen), a 5-shot style handwritten text generation approach based on Latent Diffusion Models. By utilizing a hybrid style extractor that combines metric learning and classification, our approach manages to capture both textual and stylistic characteristics of seen and unseen words and styles, generating realistic handwritten samples. Moreover, we explore several variation strategies of the data with multi-style mixtures and noisy embeddings, enhancing the robustness and diversity of the generated data. Extensive experiments using IAM offline handwriting database show that our method outperforms existing methods qualitatively and quantitatively, and its additional generated data can improve the performance of Handwriting Text Recognition (HTR) systems. The code is available at: this https URL.

Title: Privacy-Preserving Data Linkage Across Private and Public Datasets for Collaborative Agriculture Research

• Authors: Osama Zafar, Rosemarie Santa Gonzalez, Gabriel Wilkins, Alfonso Morales, Erman Ayday
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.06069
• Pdf URL: https://arxiv.org/pdf/2409.06069
• Copy Paste: [[2409.06069]] Privacy-Preserving Data Linkage Across Private and Public Datasets for Collaborative Agriculture Research(https://arxiv.org/abs/2409.06069)
• Keywords: secure, security, privacy, protect
• Abstract: Digital agriculture leverages technology to enhance crop yield, disease resilience, and soil health, playing a critical role in agricultural research. However, it raises privacy concerns such as adverse pricing, price discrimination, higher insurance costs, and manipulation of resources, deterring farm operators from sharing data due to potential misuse. This study introduces a privacy-preserving framework that addresses these risks while allowing secure data sharing for digital agriculture. Our framework enables comprehensive data analysis while protecting privacy. It allows stakeholders to harness research-driven policies that link public and private datasets. The proposed algorithm achieves this by: (1) identifying similar farmers based on private datasets, (2) providing aggregate information like time and location, (3) determining trends in price and product availability, and (4) correlating trends with public policy data, such as food insecurity statistics. We validate the framework with real-world Farmer's Market datasets, demonstrating its efficacy through machine learning models trained on linked privacy-preserved data. The results support policymakers and researchers in addressing food insecurity and pricing issues. This work significantly contributes to digital agriculture by providing a secure method for integrating and analyzing data, driving advancements in agricultural technology and development.

Title: DetoxBench: Benchmarking Large Language Models for Multitask Fraud & Abuse Detection

• Authors: Joymallya Chakraborty, Wei Xia, Anirban Majumder, Dan Ma, Walid Chaabene, Naveed Janvekar
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06072
• Pdf URL: https://arxiv.org/pdf/2409.06072
• Copy Paste: [[2409.06072]] DetoxBench: Benchmarking Large Language Models for Multitask Fraud & Abuse Detection(https://arxiv.org/abs/2409.06072)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks. However, their practical application in high-stake domains, such as fraud and abuse detection, remains an area that requires further exploration. The existing applications often narrowly focus on specific tasks like toxicity or hate speech detection. In this paper, we present a comprehensive benchmark suite designed to assess the performance of LLMs in identifying and mitigating fraudulent and abusive language across various real-world scenarios. Our benchmark encompasses a diverse set of tasks, including detecting spam emails, hate speech, misogynistic language, and more. We evaluated several state-of-the-art LLMs, including models from Anthropic, Mistral AI, and the AI21 family, to provide a comprehensive assessment of their capabilities in this critical domain. The results indicate that while LLMs exhibit proficient baseline performance in individual fraud and abuse detection tasks, their performance varies considerably across tasks, particularly struggling with tasks that demand nuanced pragmatic reasoning, such as identifying diverse forms of misogynistic language. These findings have important implications for the responsible development and deployment of LLMs in high-risk applications. Our benchmark suite can serve as a tool for researchers and practitioners to systematically evaluate LLMs for multi-task fraud detection and drive the creation of more robust, trustworthy, and ethically-aligned systems for fraud and abuse detection.

Title: SVS-GAN: Leveraging GANs for Semantic Video Synthesis

• Authors: Khaled M. Seyam, Julian Wiederer, Markus Braun, Bin Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06074
• Pdf URL: https://arxiv.org/pdf/2409.06074
• Copy Paste: [[2409.06074]] SVS-GAN: Leveraging GANs for Semantic Video Synthesis(https://arxiv.org/abs/2409.06074)
• Keywords: diffusion, generative, segmentation
• Abstract: In recent years, there has been a growing interest in Semantic Image Synthesis (SIS) through the use of Generative Adversarial Networks (GANs) and diffusion models. This field has seen innovations such as the implementation of specialized loss functions tailored for this task, diverging from the more general approaches in Image-to-Image (I2I) translation. While the concept of Semantic Video Synthesis (SVS)$\unicode{x2013}$the generation of temporally coherent, realistic sequences of images from semantic maps$\unicode{x2013}$is newly formalized in this paper, some existing methods have already explored aspects of this field. Most of these approaches rely on generic loss functions designed for video-to-video translation or require additional data to achieve temporal coherence. In this paper, we introduce the SVS-GAN, a framework specifically designed for SVS, featuring a custom architecture and loss functions. Our approach includes a triple-pyramid generator that utilizes SPADE blocks. Additionally, we employ a U-Net-based network for the image discriminator, which performs semantic segmentation for the OASIS loss. Through this combination of tailored architecture and objective engineering, our framework aims to bridge the existing gap between SIS and SVS, outperforming current state-of-the-art models on datasets like Cityscapes and KITTI-360.

Title: SGC-VQGAN: Towards Complex Scene Representation via Semantic Guided Clustering Codebook

• Authors: Chenjing Ding, Chiyu Wang, Boshi Liu, Xi Guo, Weixuan Tang, Wei Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06105
• Pdf URL: https://arxiv.org/pdf/2409.06105
• Copy Paste: [[2409.06105]] SGC-VQGAN: Towards Complex Scene Representation via Semantic Guided Clustering Codebook(https://arxiv.org/abs/2409.06105)
• Keywords: segmentation
• Abstract: Vector quantization (VQ) is a method for deterministically learning features through discrete codebook representations. Recent works have utilized visual tokenizers to discretize visual regions for self-supervised representation learning. However, a notable limitation of these tokenizers is lack of semantics, as they are derived solely from the pretext task of reconstructing raw image pixels in an auto-encoder paradigm. Additionally, issues like imbalanced codebook distribution and codebook collapse can adversely impact performance due to inefficient codebook utilization. To address these challenges, We introduce SGC-VQGAN through Semantic Online Clustering method to enhance token semantics through Consistent Semantic Learning. Utilizing inference results from segmentation model , our approach constructs a temporospatially consistent semantic codebook, addressing issues of codebook collapse and imbalanced token semantics. Our proposed Pyramid Feature Learning pipeline integrates multi-level features to capture both image details and semantics simultaneously. As a result, SGC-VQGAN achieves SOTA performance in both reconstruction quality and various downstream tasks. Its simplicity, requiring no additional parameter learning, enables its direct application in downstream tasks, presenting significant potential.

Title: Doppelg\"anger's Watch: A Split Objective Approach to Large Language Models

• Authors: Shervin Ghasemlou, Ashish Katiyar, Aparajita Saraf, Seungwhan Moon, Mangesh Pujari, Pinar Donmez, Babak Damavandi, Anuj Kumar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06107
• Pdf URL: https://arxiv.org/pdf/2409.06107
• Copy Paste: [[2409.06107]] Doppelg\"anger's Watch: A Split Objective Approach to Large Language Models(https://arxiv.org/abs/2409.06107)
• Keywords: large language model
• Abstract: In this paper, we investigate the problem of "generation supervision" in large language models, and present a novel bicameral architecture to separate supervision signals from their core capability, helpfulness. Doppelgänger, a new module parallel to the underlying language model, supervises the generation of each token, and learns to concurrently predict the supervision score(s) of the sequences up to and including each token. In this work, we present the theoretical findings, and leave the report on experimental results to a forthcoming publication.

Title: Contrastive Federated Learning with Tabular Data Silos

• Authors: Achmad Ginanjar, Xue Li, Wen Hua
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06123
• Pdf URL: https://arxiv.org/pdf/2409.06123
• Copy Paste: [[2409.06123]] Contrastive Federated Learning with Tabular Data Silos(https://arxiv.org/abs/2409.06123)
• Keywords: privacy, federate
• Abstract: Learning from data silos is a difficult task for organizations that need to obtain knowledge of objects that appeared in multiple independent data silos. Objects in multi-organizations, such as government agents, are referred by different identifiers, such as driver license, passport number, and tax file number. The data distributions in data silos are mostly non-IID (Independently and Identically Distributed), labelless, and vertically partitioned (i.e., having different attributes). Privacy concerns harden the above issues. Conditions inhibit enthusiasm for collaborative work. While Federated Learning (FL) has been proposed to address these issues, the difficulty of labeling, namely, label costliness, often hinders optimal model performance. A potential solution lies in contrastive learning, an unsupervised self-learning technique to represent semantic data by contrasting similar data pairs. However, contrastive learning is currently not designed to handle tabular data silos that existed within multiple organizations where data linkage by quasi identifiers are needed. To address these challenges, we propose using semi-supervised contrastive federated learning, which we refer to as Contrastive Federated Learning with Data Silos (CFL). Our approach tackles the aforementioned issues with an integrated solution. Our experimental results demonstrate that CFL outperforms current methods in addressing these challenges and providing improvements in accuracy. Additionally, we present positive results that showcase the advantages of our contrastive federated learning approach in complex client environments.

Title: Conditional Encryption with Applications to Secure Personalized Password Typo Correction

• Authors: Mohammad Hassan Ameri, Jeremiah Blocki
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.06128
• Pdf URL: https://arxiv.org/pdf/2409.06128
• Copy Paste: [[2409.06128]] Conditional Encryption with Applications to Secure Personalized Password Typo Correction(https://arxiv.org/abs/2409.06128)
• Keywords: secure, security, attack
• Abstract: We introduce the notion of a conditional encryption scheme as an extension of public key encryption. In addition to the standard public key algorithms ($\mathsf{KG}$, $\mathsf{Enc}$, $\mathsf{Dec}$) for key generation, encryption and decryption, a conditional encryption scheme for a binary predicate $P$ adds a new conditional encryption algorithm $\mathsf{CEnc}$. The conditional encryption algorithm $c=\mathsf{CEnc}_{pk}(c_1,m_2,m_3)$ takes as input the public encryption key $pk$, a ciphertext $c_1 = \mathsf{Enc}_{pk}(m_1)$ for an unknown message $m_1$, a control message $m_2$ and a payload message $m_3$ and outputs a conditional ciphertext $c$. Intuitively, if $P(m_1,m_2)=1$ then the conditional ciphertext $c$ should decrypt to the payload message $m_3$. On the other hand if $P(m_1,m_2) = 0$ then the ciphertext should not leak any information about the control message $m_2$ or the payload message $m_3$ even if the attacker already has the secret decryption key $sk$. We formalize the notion of conditional encryption secrecy and provide concretely efficient constructions for a set of predicates relevant to password typo correction. Our practical constructions utilize the Paillier partially homomorphic encryption scheme as well as Shamir Secret Sharing. We prove that our constructions are secure and demonstrate how to use conditional encryption to improve the security of personalized password typo correction systems such as TypTop. We implement a C++ library for our practically efficient conditional encryption schemes and evaluate the performance empirically. We also update the implementation of TypTop to utilize conditional encryption for enhanced security guarantees and evaluate the performance of the updated implementation.

Title: DECOLLAGE: 3D Detailization by Controllable, Localized, and Learned Geometry Enhancement

• Authors: Qimin Chen, Zhiqin Chen, Vladimir G. Kim, Noam Aigerman, Hao Zhang, Siddhartha Chaudhuri
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06129
• Pdf URL: https://arxiv.org/pdf/2409.06129
• Copy Paste: [[2409.06129]] DECOLLAGE: 3D Detailization by Controllable, Localized, and Learned Geometry Enhancement(https://arxiv.org/abs/2409.06129)
• Keywords: generative
• Abstract: We present a 3D modeling method which enables end-users to refine or detailize 3D shapes using machine learning, expanding the capabilities of AI-assisted 3D content creation. Given a coarse voxel shape (e.g., one produced with a simple box extrusion tool or via generative modeling), a user can directly "paint" desired target styles representing compelling geometric details, from input exemplar shapes, over different regions of the coarse shape. These regions are then up-sampled into high-resolution geometries which adhere with the painted styles. To achieve such controllable and localized 3D detailization, we build on top of a Pyramid GAN by making it masking-aware. We devise novel structural losses and priors to ensure that our method preserves both desired coarse structures and fine-grained features even if the painted styles are borrowed from diverse sources, e.g., different semantic parts and even different shape categories. Through extensive experiments, we show that our ability to localize details enables novel interactive creative workflows and applications. Our experiments further demonstrate that in comparison to prior techniques built on global detailization, our method generates structure-preserving, high-resolution stylized geometries with more coherent shape details and style transitions.

Title: On the Weaknesses of Backdoor-based Model Watermarking: An Information-theoretic Perspective

• Authors: Aoting Hu, Yanzhi Chen, Renjie Xie, Adrian Weller
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06130
• Pdf URL: https://arxiv.org/pdf/2409.06130
• Copy Paste: [[2409.06130]] On the Weaknesses of Backdoor-based Model Watermarking: An Information-theoretic Perspective(https://arxiv.org/abs/2409.06130)
• Keywords: security, protect, attack, robust, watermark
• Abstract: Safeguarding the intellectual property of machine learning models has emerged as a pressing concern in AI security. Model watermarking is a powerful technique for protecting ownership of machine learning models, yet its reliability has been recently challenged by recent watermark removal attacks. In this work, we investigate why existing watermark embedding techniques particularly those based on backdooring are vulnerable. Through an information-theoretic analysis, we show that the resilience of watermarking against erasure attacks hinges on the choice of trigger-set samples, where current uses of out-distribution trigger-set are inherently vulnerable to white-box adversaries. Based on this discovery, we propose a novel model watermarking scheme, In-distribution Watermark Embedding (IWE), to overcome the limitations of existing method. To further minimise the gap to clean models, we analyze the role of logits as watermark information carriers and propose a new approach to better conceal watermark information within the logits. Experiments on real-world datasets including CIFAR-100 and Caltech-101 demonstrate that our method robustly defends against various adversaries with negligible accuracy loss (< 0.1%).

Title: Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review

• Authors: Neha Prakriya, Jui-Nan Yen, Cho-Jui Hsieh, Jason Cong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06131
• Pdf URL: https://arxiv.org/pdf/2409.06131
• Copy Paste: [[2409.06131]] Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review(https://arxiv.org/abs/2409.06131)
• Keywords: large language model
• Abstract: Large Language Model (LLM) pretraining traditionally relies on autoregressive language modeling on randomly sampled data blocks from web-scale datasets. We take inspiration from human learning techniques like spaced repetition to hypothesize that random data sampling for LLMs leads to high training cost and low quality models which tend to forget data. In order to effectively commit web-scale information to long-term memory, we propose the LFR (Learn, Focus, and Review) pedagogy, a new dynamic training paradigm which focuses and repeatedly reviews complex data blocks at systematic intervals based on the model's learning pace and progress. LFR records the model perplexities for different data blocks and frequently revisits blocks with higher perplexity which are more likely to be forgotten. We pretrain the GPT-2 models (124M - 1.5B) from scratch on the OpenWebText dataset using LFR. We test on downstream tasks from the language modeling, question answering, translation, and problem solving domains to achieve consistently lower perplexity and higher accuracy than the baseline OpenAI models, while obtaining a 20x pretraining speed-up.

Title: UniLearn: Enhancing Dynamic Facial Expression Recognition through Unified Pre-Training and Fine-Tuning on Images and Videos

• Authors: Yin Chen, Jia Li, Yu Zhang, Zhenzhen Hu, Shiguang Shan, Meng Wang, Richang Hong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06154
• Pdf URL: https://arxiv.org/pdf/2409.06154
• Copy Paste: [[2409.06154]] UniLearn: Enhancing Dynamic Facial Expression Recognition through Unified Pre-Training and Fine-Tuning on Images and Videos(https://arxiv.org/abs/2409.06154)
• Keywords: robust
• Abstract: Dynamic facial expression recognition (DFER) is essential for understanding human emotions and behavior. However, conventional DFER methods, which primarily use dynamic facial data, often underutilize static expression images and their labels, limiting their performance and robustness. To overcome this, we introduce UniLearn, a novel unified learning paradigm that integrates static facial expression recognition (SFER) data to enhance DFER task. UniLearn employs a dual-modal self-supervised pre-training method, leveraging both facial expression images and videos to enhance a ViT model's spatiotemporal representation capability. Then, the pre-trained model is fine-tuned on both static and dynamic expression datasets using a joint fine-tuning strategy. To prevent negative transfer during joint fine-tuning, we introduce an innovative Mixture of Adapter Experts (MoAE) module that enables task-specific knowledge acquisition and effectively integrates information from both static and dynamic expression data. Extensive experiments demonstrate UniLearn's effectiveness in leveraging complementary information from static and dynamic facial data, leading to more accurate and robust DFER. UniLearn consistently achieves state-of-the-art performance on FERV39K, MAFW, and DFEW benchmarks, with weighted average recall (WAR) of 53.65\%, 58.44\%, and 76.68\%, respectively. The source code and model weights will be publicly available at \url{this https URL}.

Title: MCDGLN: Masked Connection-based Dynamic Graph Learning Network for Autism Spectrum Disorder

• Authors: Peng Wang, Xin Wen, Ruochen Cao, Chengxin Gao, Yanrong Hao, Rui Cao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06163
• Pdf URL: https://arxiv.org/pdf/2409.06163
• Copy Paste: [[2409.06163]] MCDGLN: Masked Connection-based Dynamic Graph Learning Network for Autism Spectrum Disorder(https://arxiv.org/abs/2409.06163)
• Keywords: extraction
• Abstract: Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by complex physiological processes. Previous research has predominantly focused on static cerebral interactions, often neglecting the brain's dynamic nature and the challenges posed by network noise. To address these gaps, we introduce the Masked Connection-based Dynamic Graph Learning Network (MCDGLN). Our approach first segments BOLD signals using sliding temporal windows to capture dynamic brain characteristics. We then employ a specialized weighted edge aggregation (WEA) module, which uses the cross convolution with channel-wise element-wise convolutional kernel, to integrate dynamic functional connectivity and to isolating task-relevant connections. This is followed by topological feature extraction via a hierarchical graph convolutional network (HGCN), with key attributes highlighted by a self-attention module. Crucially, we refine static functional connections using a customized task-specific mask, reducing noise and pruning irrelevant links. The attention-based connection encoder (ACE) then enhances critical connections and compresses static features. The combined features are subsequently used for classification. Applied to the Autism Brain Imaging Data Exchange I (ABIDE I) dataset, our framework achieves a 73.3\% classification accuracy between ASD and Typical Control (TC) groups among 1,035 subjects. The pivotal roles of WEA and ACE in refining connectivity and enhancing classification accuracy underscore their importance in capturing ASD-specific features, offering new insights into the disorder.

Title: Deep Learning and Large Language Models for Audio and Text Analysis in Predicting Suicidal Acts in Chinese Psychological Support Hotlines

• Authors: Yining Chen, Jianqiang Li, Changwei Song, Qing Zhao, Yongsheng Tong, Guanghui Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06164
• Pdf URL: https://arxiv.org/pdf/2409.06164
• Copy Paste: [[2409.06164]] Deep Learning and Large Language Models for Audio and Text Analysis in Predicting Suicidal Acts in Chinese Psychological Support Hotlines(https://arxiv.org/abs/2409.06164)
• Keywords: large language model
• Abstract: Suicide is a pressing global issue, demanding urgent and effective preventive interventions. Among the various strategies in place, psychological support hotlines had proved as a potent intervention method. Approximately two million people in China attempt suicide annually, with many individuals making multiple attempts. Prompt identification and intervention for high-risk individuals are crucial to preventing tragedies. With the rapid advancement of artificial intelligence (AI), especially the development of large-scale language models (LLMs), new technological tools have been introduced to the field of mental health. This study included 1284 subjects, and was designed to validate whether deep learning models and LLMs, using audio and transcribed text from support hotlines, can effectively predict suicide risk. We proposed a simple LLM-based pipeline that first summarizes transcribed text from approximately one hour of speech to extract key features, and then predict suicidial bahaviours in the future. We compared our LLM-based method with the traditional manual scale approach in a clinical setting and with five advanced deep learning models. Surprisingly, the proposed simple LLM pipeline achieved strong performance on a test set of 46 subjects, with an F1 score of 76\% when combined with manual scale rating. This is 7\% higher than the best speech-based deep learning models and represents a 27.82\% point improvement in F1 score compared to using the manual scale apporach alone. Our study explores new applications of LLMs and demonstrates their potential for future use in suicide prevention efforts.

Title: Revisiting Prompt Pretraining of Vision-Language Models

• Authors: Zhenyuan Chen, Lingfeng Yang, Shuo Chen, Zhaowei Chen, Jiajun Liang, Xiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06166
• Pdf URL: https://arxiv.org/pdf/2409.06166
• Copy Paste: [[2409.06166]] Revisiting Prompt Pretraining of Vision-Language Models(https://arxiv.org/abs/2409.06166)
• Keywords: robust
• Abstract: Prompt learning is an effective method to customize Vision-Language Models (VLMs) for various downstream tasks, involving tuning very few parameters of input prompt tokens. Recently, prompt pretraining in large-scale dataset (e.g., ImageNet-21K) has played a crucial role in prompt learning for universal visual discrimination. However, we revisit and observe that the limited learnable prompts could face underfitting risks given the extensive images during prompt pretraining, simultaneously leading to poor generalization. To address the above issues, in this paper, we propose a general framework termed Revisiting Prompt Pretraining (RPP), which targets at improving the fitting and generalization ability from two aspects: prompt structure and prompt supervision. For prompt structure, we break the restriction in common practice where query, key, and value vectors are derived from the shared learnable prompt token. Instead, we introduce unshared individual query, key, and value learnable prompts, thereby enhancing the model's fitting capacity through increased parameter diversity. For prompt supervision, we additionally utilize soft labels derived from zero-shot probability predictions provided by a pretrained Contrastive Language Image Pretraining (CLIP) teacher model. These soft labels yield more nuanced and general insights into the inter-class relationships, thereby endowing the pretraining process with better generalization ability. RPP produces a more resilient prompt initialization, enhancing its robust transferability across diverse visual recognition tasks. Experiments across various benchmarks consistently confirm the state-of-the-art (SOTA) performance of our pretrained prompts. Codes and models will be made available soon.

Title: Larger Language Models Don't Care How You Think: Why Chain-of-Thought Prompting Fails in Subjective Tasks

• Authors: Georgios Chochlakis, Niyantha Maruthu Pandiyan, Kristina Lerman, Shrikanth Narayanan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06173
• Pdf URL: https://arxiv.org/pdf/2409.06173
• Copy Paste: [[2409.06173]] Larger Language Models Don't Care How You Think: Why Chain-of-Thought Prompting Fails in Subjective Tasks(https://arxiv.org/abs/2409.06173)
• Keywords: large language model
• Abstract: In-Context Learning (ICL) in Large Language Models (LLM) has emerged as the dominant technique for performing natural language tasks, as it does not require updating the model parameters with gradient-based methods. ICL promises to "adapt" the LLM to perform the present task at a competitive or state-of-the-art level at a fraction of the computational cost. ICL can be augmented by incorporating the reasoning process to arrive at the final label explicitly in the prompt, a technique called Chain-of-Thought (CoT) prompting. However, recent work has found that ICL relies mostly on the retrieval of task priors and less so on "learning" to perform tasks, especially for complex subjective domains like emotion and morality, where priors ossify posterior predictions. In this work, we examine whether "enabling" reasoning also creates the same behavior in LLMs, wherein the format of CoT retrieves reasoning priors that remain relatively unchanged despite the evidence in the prompt. We find that, surprisingly, CoT indeed suffers from the same posterior collapse as ICL for larger language models. Code is avalaible at this https URL.

Title: EDADepth: Enhanced Data Augmentation for Monocular Depth Estimation

• Authors: Nischal Khanal, Shivanand Venkanna Sheshappanavar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06183
• Pdf URL: https://arxiv.org/pdf/2409.06183
• Copy Paste: [[2409.06183]] EDADepth: Enhanced Data Augmentation for Monocular Depth Estimation(https://arxiv.org/abs/2409.06183)
• Keywords: extraction, diffusion, segmentation
• Abstract: Due to their text-to-image synthesis feature, diffusion models have recently seen a rise in visual perception tasks, such as depth estimation. The lack of good-quality datasets makes the extraction of a fine-grain semantic context challenging for the diffusion models. The semantic context with fewer details further worsens the process of creating effective text embeddings that will be used as input for diffusion models. In this paper, we propose a novel EDADepth, an enhanced data augmentation method to estimate monocular depth without using additional training data. We use Swin2SR, a super-resolution model, to enhance the quality of input images. We employ the BEiT pre-trained semantic segmentation model for better extraction of text embeddings. We introduce BLIP-2 tokenizer to generate tokens from these text embeddings. The novelty of our approach is the introduction of Swin2SR, the BEiT model, and the BLIP-2 tokenizer in the diffusion-based pipeline for the monocular depth estimation. Our model achieves state-of-the-art results (SOTA) on the {\delta}3 metric on NYUv2 and KITTI datasets. It also achieves results comparable to those of the SOTA models in the RMSE and REL metrics. Finally, we also show improvements in the visualization of the estimated depth compared to the SOTA diffusion-based monocular depth estimation models. Code: this https URL.

Title: Can Large Language Models Unlock Novel Scientific Research Ideas?

• Authors: Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, Asif Ekbal
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06185
• Pdf URL: https://arxiv.org/pdf/2409.06185
• Copy Paste: [[2409.06185]] Can Large Language Models Unlock Novel Scientific Research Ideas?(https://arxiv.org/abs/2409.06185)
• Keywords: large language model
• Abstract: "An idea is nothing more nor less than a new combination of old elements" (Young, J.W.). The widespread adoption of Large Language Models (LLMs) and publicly available ChatGPT have marked a significant turning point in the integration of Artificial Intelligence (AI) into people's everyday lives. This study explores the capability of LLMs in generating novel research ideas based on information from research papers. We conduct a thorough examination of 4 LLMs in five domains (e.g., Chemistry, Computer, Economics, Medical, and Physics). We found that the future research ideas generated by Claude-2 and GPT-4 are more aligned with the author's perspective than GPT-3.5 and Gemini. We also found that Claude-2 generates more diverse future research ideas than GPT-4, GPT-3.5, and Gemini 1.0. We further performed a human evaluation of the novelty, relevancy, and feasibility of the generated future research ideas. This investigation offers insights into the evolving role of LLMs in idea generation, highlighting both its capability and limitations. Our work contributes to the ongoing efforts in evaluating and utilizing language models for generating future research ideas. We make our datasets and codes publicly available.

Title: Bottleneck-based Encoder-decoder ARchitecture (BEAR) for Learning Unbiased Consumer-to-Consumer Image Representations

• Authors: Pablo Rivas, Gisela Bichler, Tomas Cerny, Laurie Giddens, Stacie Petter
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06187
• Pdf URL: https://arxiv.org/pdf/2409.06187
• Copy Paste: [[2409.06187]] Bottleneck-based Encoder-decoder ARchitecture (BEAR) for Learning Unbiased Consumer-to-Consumer Image Representations(https://arxiv.org/abs/2409.06187)
• Keywords: extraction
• Abstract: Unbiased representation learning is still an object of study under specific applications and contexts. Novel architectures are usually crafted to resolve particular problems using mixtures of fundamental pieces. This paper presents different image feature extraction mechanisms that work together with residual connections to encode perceptual image information in an autoencoder configuration. We use image data that aims to support a larger research agenda dealing with issues regarding criminal activity in consumer-to-consumer online platforms. Preliminary results suggest that the proposed architecture can learn rich spaces using ours and other image datasets resolving important challenges that are identified.

Title: MyGo: Consistent and Controllable Multi-View Driving Video Generation with Camera Control

• Authors: Yining Yao, Xi Guo, Chenjing Ding, Wei Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06189
• Pdf URL: https://arxiv.org/pdf/2409.06189
• Copy Paste: [[2409.06189]] MyGo: Consistent and Controllable Multi-View Driving Video Generation with Camera Control(https://arxiv.org/abs/2409.06189)
• Keywords: diffusion, generative
• Abstract: High-quality driving video generation is crucial for providing training data for autonomous driving models. However, current generative models rarely focus on enhancing camera motion control under multi-view tasks, which is essential for driving video generation. Therefore, we propose MyGo, an end-to-end framework for video generation, introducing motion of onboard cameras as conditions to make progress in camera controllability and multi-view consistency. MyGo employs additional plug-in modules to inject camera parameters into the pre-trained video diffusion model, which retains the extensive knowledge of the pre-trained model as much as possible. Furthermore, we use epipolar constraints and neighbor view information during the generation process of each view to enhance spatial-temporal consistency. Experimental results show that MyGo has achieved state-of-the-art results in both general camera-controlled video generation and multi-view driving video generation tasks, which lays the foundation for more accurate environment simulation in autonomous driving. Project page: \href{this https URL}{this http URL\_project/MyGo/page.html}

Title: UdeerLID+: Integrating LiDAR, Image, and Relative Depth with Semi-Supervised

• Authors: Tao Ni, Xin Zhan, Tao Luo, Wenbin Liu, Zhan Shi, JunBo Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06197
• Pdf URL: https://arxiv.org/pdf/2409.06197
• Copy Paste: [[2409.06197]] UdeerLID+: Integrating LiDAR, Image, and Relative Depth with Semi-Supervised(https://arxiv.org/abs/2409.06197)
• Keywords: robust, segmentation
• Abstract: Road segmentation is a critical task for autonomous driving systems, requiring accurate and robust methods to classify road surfaces from various environmental data. Our work introduces an innovative approach that integrates LiDAR point cloud data, visual image, and relative depth maps derived from images. The integration of multiple data sources in road segmentation presents both opportunities and challenges. One of the primary challenges is the scarcity of large-scale, accurately labeled datasets that are necessary for training robust deep learning models. To address this, we have developed the [UdeerLID+] framework under a semi-supervised learning paradigm. Experiments results on KITTI datasets validate the superior performance.

Title: Deep kernel representations of latent space features for low-dose PET-MR imaging robust to variable dose reduction

• Authors: Cameron Dennis Pain, Yasmeen George, Alex Fornito, Gary Egan, Zhaolin Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06198
• Pdf URL: https://arxiv.org/pdf/2409.06198
• Copy Paste: [[2409.06198]] Deep kernel representations of latent space features for low-dose PET-MR imaging robust to variable dose reduction(https://arxiv.org/abs/2409.06198)
• Keywords: robust
• Abstract: Low-dose positron emission tomography (PET) image reconstruction methods have potential to significantly improve PET as an imaging modality. Deep learning provides a promising means of incorporating prior information into the image reconstruction problem to produce quantitatively accurate images from compromised signal. Deep learning-based methods for low-dose PET are generally poorly conditioned and perform unreliably on images with features not present in the training distribution. We present a method which explicitly models deep latent space features using a robust kernel representation, providing robust performance on previously unseen dose reduction factors. Additional constraints on the information content of deep latent features allow for tuning in-distribution accuracy and generalisability. Tests with out-of-distribution dose reduction factors ranging from $\times 10$ to $\times 1000$ and with both paired and unpaired MR, demonstrate significantly improved performance relative to conventional deep-learning methods trained using the same data. Code:this https URL

Title: RealisDance: Equip controllable character animation with realistic hands

• Authors: Jingkai Zhou, Benzhi Wang, Weihua Chen, Jingqi Bai, Dongyang Li, Aixi Zhang, Hao Xu, Mingyang Yang, Fan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06202
• Pdf URL: https://arxiv.org/pdf/2409.06202
• Copy Paste: [[2409.06202]] RealisDance: Equip controllable character animation with realistic hands(https://arxiv.org/abs/2409.06202)
• Keywords: robust
• Abstract: Controllable character animation is an emerging task that generates character videos controlled by pose sequences from given character images. Although character consistency has made significant progress via reference UNet, another crucial factor, pose control, has not been well studied by existing methods yet, resulting in several issues: 1) The generation may fail when the input pose sequence is corrupted. 2) The hands generated using the DWPose sequence are blurry and unrealistic. 3) The generated video will be shaky if the pose sequence is not smooth enough. In this paper, we present RealisDance to handle all the above issues. RealisDance adaptively leverages three types of poses, avoiding failed generation caused by corrupted pose sequences. Among these pose types, HaMeR provides accurate 3D and depth information of hands, enabling RealisDance to generate realistic hands even for complex gestures. Besides using temporal attention in the main UNet, RealisDance also inserts temporal attention into the pose guidance network, smoothing the video from the pose condition aspect. Moreover, we introduce pose shuffle augmentation during training to further improve generation robustness and video smoothness. Qualitative experiments demonstrate the superiority of RealisDance over other existing methods, especially in hand quality.

Title: Watching TV with the Second-Party: A First Look at Automatic Content Recognition Tracking in Smart TVs

• Authors: Gianluca Anselmi, Yash Vekaria, Alexander D'Souza, Patricia Callejo, Anna Maria Mandalari, Zubair Shafiq
• Subjects: cs.CR, cs.CY, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.06203
• Pdf URL: https://arxiv.org/pdf/2409.06203
• Copy Paste: [[2409.06203]] Watching TV with the Second-Party: A First Look at Automatic Content Recognition Tracking in Smart TVs(https://arxiv.org/abs/2409.06203)
• Keywords: privacy
• Abstract: Smart TVs implement a unique tracking approach called Automatic Content Recognition (ACR) to profile viewing activity of their users. ACR is a Shazam-like technology that works by periodically capturing the content displayed on a TV's screen and matching it against a content library to detect what content is being displayed at any given point in time. While prior research has investigated third-party tracking in the smart TV ecosystem, it has not looked into second-party ACR tracking that is directly conducted by the smart TV platform. In this work, we conduct a black-box audit of ACR network traffic between ACR clients on the smart TV and ACR servers. We use our auditing approach to systematically investigate whether (1) ACR tracking is agnostic to how a user watches TV (e.g., linear vs. streaming vs. HDMI), (2) privacy controls offered by smart TVs have an impact on ACR tracking, and (3) there are any differences in ACR tracking between the UK and the US. We perform a series of experiments on two major smart TV platforms: Samsung and LG. Our results show that ACR works even when the smart TV is used as a "dumb" external display, opting-out stops network traffic to ACR servers, and there are differences in how ACR works across the UK and the US.

Title: AgileIR: Memory-Efficient Group Shifted Windows Attention for Agile Image Restoration

• Authors: Hongyi Cai, Mohammad Mahdinur Rahman, Mohammad Shahid Akhtar, Jie Li, Jingyu Wu, Zhili Fang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06206
• Pdf URL: https://arxiv.org/pdf/2409.06206
• Copy Paste: [[2409.06206]] AgileIR: Memory-Efficient Group Shifted Windows Attention for Agile Image Restoration(https://arxiv.org/abs/2409.06206)
• Keywords: transformer
• Abstract: Image Transformers show a magnificent success in Image Restoration tasks. Nevertheless, most of transformer-based models are strictly bounded by exorbitant memory occupancy. Our goal is to reduce the memory consumption of Swin Transformer and at the same time speed up the model during training process. Thus, we introduce AgileIR, group shifted attention mechanism along with window attention, which sparsely simplifies the model in architecture. We propose Group Shifted Window Attention (GSWA) to decompose Shift Window Multi-head Self Attention (SW-MSA) and Window Multi-head Self Attention (W-MSA) into groups across their attention heads, contributing to shrinking memory usage in back propagation. In addition to that, we keep shifted window masking and its shifted learnable biases during training, in order to induce the model interacting across windows within the channel. We also re-allocate projection parameters to accelerate attention matrix calculation, which we found a negligible decrease in performance. As a result of experiment, compared with our baseline SwinIR and other efficient quantization models, AgileIR keeps the performance still at 32.20 dB on Set5 evaluation dataset, exceeding other methods with tailor-made efficient methods and saves over 50% memory while a large batch size is employed.

Title: Adaptive Transformer Modelling of Density Function for Nonparametric Survival Analysis

• Authors: Xin Zhang, Deval Mehta, Yanan Hu, Chao Zhu, David Darby, Zhen Yu, Daniel Merlo, Melissa Gresle, Anneke Van Der Walt, Helmut Butzkueven, Zongyuan Ge
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06209
• Pdf URL: https://arxiv.org/pdf/2409.06209
• Copy Paste: [[2409.06209]] Adaptive Transformer Modelling of Density Function for Nonparametric Survival Analysis(https://arxiv.org/abs/2409.06209)
• Keywords: transformer
• Abstract: Survival analysis holds a crucial role across diverse disciplines, such as economics, engineering and healthcare. It empowers researchers to analyze both time-invariant and time-varying data, encompassing phenomena like customer churn, material degradation and various medical outcomes. Given the complexity and heterogeneity of such data, recent endeavors have demonstrated successful integration of deep learning methodologies to address limitations in conventional statistical approaches. However, current methods typically involve cluttered probability distribution function (PDF), have lower sensitivity in censoring prediction, only model static datasets, or only rely on recurrent neural networks for dynamic modelling. In this paper, we propose a novel survival regression method capable of producing high-quality unimodal PDFs without any prior distribution assumption, by optimizing novel Margin-Mean-Variance loss and leveraging the flexibility of Transformer to handle both temporal and non-temporal data, coined UniSurv. Extensive experiments on several datasets demonstrate that UniSurv places a significantly higher emphasis on censoring compared to other methods.

Title: INTRA: Interaction Relationship-aware Weakly Supervised Affordance Grounding

• Authors: Ji Ha Jang, Hoigi Seo, Se Young Chun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06210
• Pdf URL: https://arxiv.org/pdf/2409.06210
• Copy Paste: [[2409.06210]] INTRA: Interaction Relationship-aware Weakly Supervised Affordance Grounding(https://arxiv.org/abs/2409.06210)
• Keywords: robust
• Abstract: Affordance denotes the potential interactions inherent in objects. The perception of affordance can enable intelligent agents to navigate and interact with new environments efficiently. Weakly supervised affordance grounding teaches agents the concept of affordance without costly pixel-level annotations, but with exocentric images. Although recent advances in weakly supervised affordance grounding yielded promising results, there remain challenges including the requirement for paired exocentric and egocentric image dataset, and the complexity in grounding diverse affordances for a single object. To address them, we propose INTeraction Relationship-aware weakly supervised Affordance grounding (INTRA). Unlike prior arts, INTRA recasts this problem as representation learning to identify unique features of interactions through contrastive learning with exocentric images only, eliminating the need for paired datasets. Moreover, we leverage vision-language model embeddings for performing affordance grounding flexibly with any text, designing text-conditioned affordance map generation to reflect interaction relationship for contrastive learning and enhancing robustness with our text synonym augmentation. Our method outperformed prior arts on diverse datasets such as AGD20K, IIT-AFF, CAD and UMD. Additionally, experimental results demonstrate that our method has remarkable domain scalability for synthesized images / illustrations and is capable of performing affordance grounding for novel interactions and objects.

Title: STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning

• Authors: Jaeseong Lee, seung-won hwang, Aurick Qiao, Daniel F Campos, Zhewei Yao, Yuxiong He
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06211
• Pdf URL: https://arxiv.org/pdf/2409.06211
• Copy Paste: [[2409.06211]] STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning(https://arxiv.org/abs/2409.06211)
• Keywords: generative, large language model
• Abstract: Mixture-of-experts (MoEs) have been adopted for reducing inference costs by sparsely activating experts in Large language models (LLMs). Despite this reduction, the massive number of experts in MoEs still makes them expensive to serve. In this paper, we study how to address this, by pruning MoEs. Among pruning methodologies, unstructured pruning has been known to achieve the highest performance for a given pruning ratio, compared to structured pruning, since the latter imposes constraints on the sparsification structure. This is intuitive, as the solution space of unstructured pruning subsumes that of structured pruning. However, our counterintuitive finding reveals that expert pruning, a form of structured pruning, can actually precede unstructured pruning to outperform unstructured-only pruning. As existing expert pruning, requiring $O(\frac{k^n}{\sqrt{n}})$ forward passes for $n$ experts, cannot scale for recent MoEs, we propose a scalable alternative with $O(1)$ complexity, yet outperforming the more expensive methods. The key idea is leveraging a latent structure between experts, based on behavior similarity, such that the greedy decision of whether to prune closely captures the joint pruning effect. Ours is highly effective -- for Snowflake Arctic, a 480B-sized MoE with 128 experts, our method needs only one H100 and two hours to achieve nearly no loss in performance with 40% sparsity, even in generative tasks such as GSM8K, where state-of-the-art unstructured pruning fails to. The code will be made publicly available.

Title: BACKRUNNER: Mitigating Smart Contract Attacks in the Real World

• Authors: Chaofan Shou, Yuanyu Ke, Yupeng Yang, Qi Su, Or Dadosh, Assaf Eli, David Benchimol, Doudou Lu, Daniel Tong, Dex Chen, Zoey Tan, Jacob Chia, Koushik Sen, Wenke Lee
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.06213
• Pdf URL: https://arxiv.org/pdf/2409.06213
• Copy Paste: [[2409.06213]] BACKRUNNER: Mitigating Smart Contract Attacks in the Real World(https://arxiv.org/abs/2409.06213)
• Keywords: protect, attack
• Abstract: Billions of dollars have been lost due to vulnerabilities in smart contracts. To counteract this, researchers have proposed attack frontrunning protections designed to preempt malicious transactions by inserting "whitehat" transactions ahead of them to protect the assets. In this paper, we demonstrate that existing frontrunning protections have become ineffective in real-world scenarios. Specifically, we collected 158 recent real-world attack transactions and discovered that 141 of them can bypass state-of-the-art frontrunning protections. We systematically analyze these attacks and show how inherent limitations of existing frontrunning techniques hinder them from protecting valuable assets in the real world. We then propose a new approach involving 1) preemptive hijack, and 2) attack backrunning, which circumvent the existing limitations and can help protect assets before and after an attack. Our approach adapts the exploit used in the attack to the same or similar contracts before and after the attack to safeguard the assets. We conceptualize adapting exploits as a program repair problem and apply established techniques to implement our approach into a full-fledged framework, BACKRUNNER. Running on previous attacks in 2023, BACKRUNNER can successfully rescue more than \$410M. In the real world, it has helped rescue over \$11.2M worth of assets in 28 separate incidents within two months.

Title: DACAT: Dual-stream Adaptive Clip-aware Time Modeling for Robust Online Surgical Phase Recognition

• Authors: Kaixiang Yang, Qiang Li, Zhiwei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06217
• Pdf URL: https://arxiv.org/pdf/2409.06217
• Copy Paste: [[2409.06217]] DACAT: Dual-stream Adaptive Clip-aware Time Modeling for Robust Online Surgical Phase Recognition(https://arxiv.org/abs/2409.06217)
• Keywords: robust
• Abstract: Surgical phase recognition has become a crucial requirement in laparoscopic surgery, enabling various clinical applications like surgical risk forecasting. Current methods typically identify the surgical phase using individual frame-wise embeddings as the fundamental unit for time modeling. However, this approach is overly sensitive to current observations, often resulting in discontinuous and erroneous predictions within a complete surgical phase. In this paper, we propose DACAT, a novel dual-stream model that adaptively learns clip-aware context information to enhance the temporal relationship. In one stream, DACAT pretrains a frame encoder, caching all historical frame-wise features. In the other stream, DACAT fine-tunes a new frame encoder to extract the frame-wise feature at the current moment. Additionally, a max clip-response read-out (Max-R) module is introduced to bridge the two streams by using the current frame-wise feature to adaptively fetch the most relevant past clip from the feature cache. The clip-aware context feature is then encoded via cross-attention between the current frame and its fetched adaptive clip, and further utilized to enhance the time modeling for accurate online surgical phase recognition. The benchmark results on three public datasets, i.e., Cholec80, M2CAI16, and AutoLaparo, demonstrate the superiority of our proposed DACAT over existing state-of-the-art methods, with improvements in Jaccard scores of at least 4.5%, 4.6%, and 2.7%, respectively. Our code and models have been released at this https URL.

Title: Advancing Topic Segmentation of Broadcasted Speech with Multilingual Semantic Embeddings

• Authors: Sakshi Deo Shukla, Pavel Denisov, Tugtekin Turan
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.06222
• Pdf URL: https://arxiv.org/pdf/2409.06222
• Copy Paste: [[2409.06222]] Advancing Topic Segmentation of Broadcasted Speech with Multilingual Semantic Embeddings(https://arxiv.org/abs/2409.06222)
• Keywords: segmentation
• Abstract: Recent advancements in speech-based topic segmentation have highlighted the potential of pretrained speech encoders to capture semantic representations directly from speech. Traditionally, topic segmentation has relied on a pipeline approach in which transcripts of the automatic speech recognition systems are generated, followed by text-based segmentation algorithms. In this paper, we introduce an end-to-end scheme that bypasses this conventional two-step process by directly employing semantic speech encoders for segmentation. Focused on the broadcasted news domain, which poses unique challenges due to the diversity of speakers and topics within single recordings, we address the challenge of accessing topic change points efficiently in an end-to-end manner. Furthermore, we propose a new benchmark for spoken news topic segmentation by utilizing a dataset featuring approximately 1000 hours of publicly available recordings across six European languages and including an evaluation set in Hindi to test the model's cross-domain performance in a cross-lingual, zero-shot scenario. This setup reflects real-world diversity and the need for models adapting to various linguistic settings. Our results demonstrate that while the traditional pipeline approach achieves a state-of-the-art $P_k$ score of 0.2431 for English, our end-to-end model delivers a competitive $P_k$ score of 0.2564. When trained multilingually, these scores further improve to 0.1988 and 0.2370, respectively. To support further research, we release our model along with data preparation scripts, facilitating open research on multilingual spoken news topic segmentation.

Title: MIP-GAF: A MLLM-annotated Benchmark for Most Important Person Localization and Group Context Understanding

• Authors: Surbhi Madan, Shreya Ghosh, Lownish Rai Sookha, M.A. Ganaie, Ramanathan Subramanian, Abhinav Dhall, Tom Gedeon
• Subjects: cs.CV, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.06224
• Pdf URL: https://arxiv.org/pdf/2409.06224
• Copy Paste: [[2409.06224]] MIP-GAF: A MLLM-annotated Benchmark for Most Important Person Localization and Group Context Understanding(https://arxiv.org/abs/2409.06224)
• Keywords: robust, large language model
• Abstract: Estimating the Most Important Person (MIP) in any social event setup is a challenging problem mainly due to contextual complexity and scarcity of labeled data. Moreover, the causality aspects of MIP estimation are quite subjective and diverse. To this end, we aim to address the problem by annotating a large-scale `in-the-wild' dataset for identifying human perceptions about the `Most Important Person (MIP)' in an image. The paper provides a thorough description of our proposed Multimodal Large Language Model (MLLM) based data annotation strategy, and a thorough data quality analysis. Further, we perform a comprehensive benchmarking of the proposed dataset utilizing state-of-the-art MIP localization methods, indicating a significant drop in performance compared to existing datasets. The performance drop shows that the existing MIP localization algorithms must be more robust with respect to `in-the-wild' situations. We believe the proposed dataset will play a vital role in building the next-generation social situation understanding methods. The code and data is available at this https URL.

Title: Recurrent Neural Networks for Still Images

• Authors: Dmitri (Dima)Lvov, Yair Smadar, Ran Bezen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06235
• Pdf URL: https://arxiv.org/pdf/2409.06235
• Copy Paste: [[2409.06235]] Recurrent Neural Networks for Still Images(https://arxiv.org/abs/2409.06235)
• Keywords: transformer
• Abstract: In this paper, we explore the application of Recurrent Neural Network (RNN) for still images. Typically, Convolutional Neural Networks (CNNs) are the prevalent method applied for this type of data, and more recently, transformers have gained popularity, although they often require large models. Unlike these methods, RNNs are generally associated with processing sequences over time rather than single images. We argue that RNNs can effectively handle still images by interpreting the pixels as a sequence. This approach could be particularly advantageous for compact models designed for embedded systems, where resources are limited. Additionally, we introduce a novel RNN design tailored for two-dimensional inputs, such as images, and a custom version of BiDirectional RNN (BiRNN) that is more memory-efficient than traditional implementations. In our research, we have tested these layers in Convolutional Recurrent Neural Networks (CRNNs), predominantly composed of Conv2D layers, with RNN layers at or close to the end. Experiments on the COCO and CIFAR100 datasets show better results, particularly for small networks.

Title: Test-Time Certifiable Self-Supervision to Bridge the Sim2Real Gap in Event-Based Satellite Pose Estimation

• Authors: Mohsi Jawaid, Rajat Talak, Yasir Latif, Luca Carlone, Tat-Jun Chin
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.06240
• Pdf URL: https://arxiv.org/pdf/2409.06240
• Copy Paste: [[2409.06240]] Test-Time Certifiable Self-Supervision to Bridge the Sim2Real Gap in Event-Based Satellite Pose Estimation(https://arxiv.org/abs/2409.06240)
• Keywords: robust
• Abstract: Deep learning plays a critical role in vision-based satellite pose estimation. However, the scarcity of real data from the space environment means that deep models need to be trained using synthetic data, which raises the Sim2Real domain gap problem. A major cause of the Sim2Real gap are novel lighting conditions encountered during test time. Event sensors have been shown to provide some robustness against lighting variations in vision-based pose estimation. However, challenging lighting conditions due to strong directional light can still cause undesirable effects in the output of commercial off-the-shelf event sensors, such as noisy/spurious events and inhomogeneous event densities on the object. Such effects are non-trivial to simulate in software, thus leading to Sim2Real gap in the event domain. To close the Sim2Real gap in event-based satellite pose estimation, the paper proposes a test-time self-supervision scheme with a certifier module. Self-supervision is enabled by an optimisation routine that aligns a dense point cloud of the predicted satellite pose with the event data to attempt to rectify the inaccurately estimated pose. The certifier attempts to verify the corrected pose, and only certified test-time inputs are backpropagated via implicit differentiation to refine the predicted landmarks, thus improving the pose estimates and closing the Sim2Real gap. Results show that the our method outperforms established test-time adaptation schemes.

Title: Differential Degradation Vulnerabilities in Censorship Circumvention Systems

• Authors: Zhen Sun, Vitaly Shmatikov
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.06247
• Pdf URL: https://arxiv.org/pdf/2409.06247
• Copy Paste: [[2409.06247]] Differential Degradation Vulnerabilities in Censorship Circumvention Systems(https://arxiv.org/abs/2409.06247)
• Keywords: attack
• Abstract: Several recently proposed censorship circumvention systems use encrypted network channels of popular applications to hide their communications. For example, a Tor pluggable transport called Snowflake uses the WebRTC data channel, while a system called Protozoa substitutes content in a WebRTC video-call application. By using the same channel as the cover application and (in the case of Protozoa) matching its observable traffic characteristics, these systems aim to resist powerful network-based censors capable of large-scale traffic analysis. Protozoa, in particular, achieves a strong indistinguishability property known as behavioral independence. We demonstrate that this class of systems is generically vulnerable to a new type of active attacks we call "differential degradation." These attacks do not require multi-flow measurements or traffic classification and are thus available to all real-world censors. They exploit the discrepancies between the respective network requirements of the circumvention system and its cover application. We show how a censor can use the minimal application-level information exposed by WebRTC to create network conditions that cause the circumvention system to suffer a much bigger degradation in performance than the cover application. Even when the attack causes no observable differences in network traffic and behavioral independence still holds, the censor can block circumvention at a low cost, without resorting to traffic analysis, and with minimal collateral damage to non-circumvention users. We present effective differential degradation attacks against Snowflake and Protozoa. We explain the root cause of these vulnerabilities, analyze the tradeoffs faced by the designers of circumvention systems, and propose a modified version of Protozoa that resists differential degradation attacks.

Title: ALSS-YOLO: An Adaptive Lightweight Channel Split and Shuffling Network for TIR Wildlife Detection in UAV Imagery

• Authors: Ang He, Xiaobo Li, Ximei Wu, Chengyue Su, Jing Chen, Sheng Xu, Xiaobin Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06259
• Pdf URL: https://arxiv.org/pdf/2409.06259
• Copy Paste: [[2409.06259]] ALSS-YOLO: An Adaptive Lightweight Channel Split and Shuffling Network for TIR Wildlife Detection in UAV Imagery(https://arxiv.org/abs/2409.06259)
• Keywords: robust, extraction
• Abstract: Unmanned aerial vehicles (UAVs) equipped with thermal infrared (TIR) cameras play a crucial role in combating nocturnal wildlife poaching. However, TIR images often face challenges such as jitter, and wildlife overlap, necessitating UAVs to possess the capability to identify blurred and overlapping small targets. Current traditional lightweight networks deployed on UAVs struggle to extract features from blurry small targets. To address this issue, we developed ALSS-YOLO, an efficient and lightweight detector optimized for TIR aerial images. Firstly, we propose a novel Adaptive Lightweight Channel Split and Shuffling (ALSS) module. This module employs an adaptive channel split strategy to optimize feature extraction and integrates a channel shuffling mechanism to enhance information exchange between channels. This improves the extraction of blurry features, crucial for handling jitter-induced blur and overlapping targets. Secondly, we developed a Lightweight Coordinate Attention (LCA) module that employs adaptive pooling and grouped convolution to integrate feature information across dimensions. This module ensures lightweight operation while maintaining high detection precision and robustness against jitter and target overlap. Additionally, we developed a single-channel focus module to aggregate the width and height information of each channel into four-dimensional channel fusion, which improves the feature representation efficiency of infrared images. Finally, we modify the localization loss function to emphasize the loss value associated with small objects to improve localization accuracy. Extensive experiments on the BIRDSAI and ISOD TIR UAV wildlife datasets show that ALSS-YOLO achieves state-of-the-art performance, Our code is openly available at this https URL.

Title: Keyword-Aware ASR Error Augmentation for Robust Dialogue State Tracking

• Authors: Jihyun Lee, Solee Im, Wonjun Lee, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06263
• Pdf URL: https://arxiv.org/pdf/2409.06263
• Copy Paste: [[2409.06263]] Keyword-Aware ASR Error Augmentation for Robust Dialogue State Tracking(https://arxiv.org/abs/2409.06263)
• Keywords: robust
• Abstract: Dialogue State Tracking (DST) is a key part of task-oriented dialogue systems, identifying important information in conversations. However, its accuracy drops significantly in spoken dialogue environments due to named entity errors from Automatic Speech Recognition (ASR) systems. We introduce a simple yet effective data augmentation method that targets those entities to improve the robustness of DST model. Our novel method can control the placement of errors using keyword-highlighted prompts while introducing phonetically similar errors. As a result, our method generated sufficient error patterns on keywords, leading to improved accuracy in noised and low-accuracy ASR environments.

Title: Mahalanobis k-NN: A Statistical Lens for Robust Point-Cloud Registrations

• Authors: Tejas Anvekar, Shivanand Venkanna Sheshappanavar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06267
• Pdf URL: https://arxiv.org/pdf/2409.06267
• Copy Paste: [[2409.06267]] Mahalanobis k-NN: A Statistical Lens for Robust Point-Cloud Registrations(https://arxiv.org/abs/2409.06267)
• Keywords: robust
• Abstract: In this paper, we discuss Mahalanobis k-NN: a statistical lens designed to address the challenges of feature matching in learning-based point cloud registration when confronted with an arbitrary density of point clouds, either in the source or target point cloud. We tackle this by adopting Mahalanobis k-NN's inherent property to capture the distribution of the local neighborhood and surficial geometry. Our method can be seamlessly integrated into any local-graph-based point cloud analysis method. In this paper, we focus on two distinct methodologies: Deep Closest Point (DCP) and Deep Universal Manifold Embedding (DeepUME). Our extensive benchmarking on the ModelNet40 and Faust datasets highlights the efficacy of the proposed method in point cloud registration tasks. Moreover, we establish for the first time that the features acquired through point cloud registration inherently can possess discriminative capabilities. This is evident by a substantial improvement of about 20\% in the average accuracy observed in the point cloud few-shot classification task benchmarked on ModelNet40 and ScanObjectNN. The code is publicly available at this https URL

Title: Towards Robust Uncertainty-Aware Incomplete Multi-View Classification

• Authors: Mulin Chen, Haojian Huang, Qiang Li
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06270
• Pdf URL: https://arxiv.org/pdf/2409.06270
• Copy Paste: [[2409.06270]] Towards Robust Uncertainty-Aware Incomplete Multi-View Classification(https://arxiv.org/abs/2409.06270)
• Keywords: robust
• Abstract: Handling incomplete data in multi-view classification is challenging, especially when traditional imputation methods introduce biases that compromise uncertainty estimation. Existing Evidential Deep Learning (EDL) based approaches attempt to address these issues, but they often struggle with conflicting evidence due to the limitations of the Dempster-Shafer combination rule, leading to unreliable decisions. To address these challenges, we propose the Alternating Progressive Learning Network (APLN), specifically designed to enhance EDL-based methods in incomplete MVC scenarios. Our approach mitigates bias from corrupted observed data by first applying coarse imputation, followed by mapping the data to a latent space. In this latent space, we progressively learn an evidence distribution aligned with the target domain, incorporating uncertainty considerations through EDL. Additionally, we introduce a conflict-aware Dempster-Shafer combination rule (DSCR) to better handle conflicting evidence. By sampling from the learned distribution, we optimize the latent representations of missing views, reducing bias and enhancing decision-making robustness. Extensive experiments demonstrate that APLN, combined with DSCR, significantly outperforms traditional methods, particularly in environments characterized by high uncertainty and conflicting evidence, establishing it as a promising solution for incomplete multi-view classification.

Title: Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models

• Authors: Yao Shu, Wenyang Hu, See-Kiong Ng, Bryan Kian Hsiang Low, Fei Richard Yu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06277
• Pdf URL: https://arxiv.org/pdf/2409.06277
• Copy Paste: [[2409.06277]] Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models(https://arxiv.org/abs/2409.06277)
• Keywords: privacy, federate, large language model
• Abstract: Large Language Models (LLMs) have become indispensable in numerous real-world applications. Unfortunately, fine-tuning these models at scale, especially in federated settings where data privacy and communication efficiency are critical, presents significant challenges. Existing methods often resort to parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but this typically comes at the cost of model accuracy. To address these limitations, we propose federated full-parameter tuning at scale for LLMs (Ferret), the first first-order method with shared randomness to enable scalable full-parameter tuning of LLMs across decentralized data sources while maintaining competitive model accuracy. Ferret accomplishes this through three aspects: (1) it employs widely applied first-order methods for efficient local updates; (2) it projects these updates into a low-dimensional space to considerably reduce communication overhead; and (3) it reconstructs local updates from this low-dimensional space with shared randomness to facilitate effective full-parameter global aggregation, ensuring fast convergence and competitive final performance. Our rigorous theoretical analyses and insights along with extensive experiments, show that Ferret significantly enhances the scalability of existing federated full-parameter tuning approaches by achieving high computational efficiency, reduced communication overhead, and fast convergence, all while maintaining competitive model accuracy. Our implementation is available at this https URL.

Title: Catch Me if You Can: Detecting Unauthorized Data Use in Deep Learning Models

• Authors: Zitao Chen, Karthik Pattabiraman
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06280
• Pdf URL: https://arxiv.org/pdf/2409.06280
• Copy Paste: [[2409.06280]] Catch Me if You Can: Detecting Unauthorized Data Use in Deep Learning Models(https://arxiv.org/abs/2409.06280)
• Keywords: membership infer
• Abstract: The rise of deep learning (DL) has led to a surging demand for training data, which incentivizes the creators of DL models to trawl through the Internet for training materials. Meanwhile, users often have limited control over whether their data (e.g., facial images) are used to train DL models without their consent, which has engendered pressing concerns. This work proposes MembershipTracker, a practical data provenance tool that can empower ordinary users to take agency in detecting the unauthorized use of their data in training DL models. We view tracing data provenance through the lens of membership inference (MI). MembershipTracker consists of a lightweight data marking component to mark the target data with small and targeted changes, which can be strongly memorized by the model trained on them; and a specialized MI-based verification process to audit whether the model exhibits strong memorization on the target samples. Overall, MembershipTracker only requires the users to mark a small fraction of data (0.005% to 0.1% in proportion to the training set), and it enables the users to reliably detect the unauthorized use of their data (average 0% FPR@100% TPR). We show that MembershipTracker is highly effective across various settings, including industry-scale training on the full-size ImageNet-1k dataset. We finally evaluate MembershipTracker under multiple classes of countermeasures.

Title: Learning Augmentation Policies from A Model Zoo for Time Series Forecasting

• Authors: Haochen Yuan, Xuelin Li, Yunbo Wang, Xiaokang Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06282
• Pdf URL: https://arxiv.org/pdf/2409.06282
• Copy Paste: [[2409.06282]] Learning Augmentation Policies from A Model Zoo for Time Series Forecasting(https://arxiv.org/abs/2409.06282)
• Keywords: generative
• Abstract: Time series forecasting models typically rely on a fixed-size training set and treat all data uniformly, which may not effectively capture the specific patterns present in more challenging training samples. To address this issue, we introduce AutoTSAug, a learnable data augmentation method based on reinforcement learning. Our approach begins with an empirical analysis to determine which parts of the training data should be augmented. Specifically, we identify the so-called marginal samples by considering the prediction diversity across a set of pretrained forecasting models. Next, we propose using variational masked autoencoders as the augmentation model and applying the REINFORCE algorithm to transform the marginal samples into new data. The goal of this generative model is not only to mimic the distribution of real data but also to reduce the variance of prediction errors across the model zoo. By augmenting the marginal samples with a learnable policy, AutoTSAug substantially improves forecasting performance, advancing the prior art in this field with minimal additional computational cost.

Title: Context Enhancement with Reconstruction as Sequence for Unified Unsupervised Anomaly Detection

• Authors: Hui-Yue Yang, Hui Chen, Lihao Liu, Zijia Lin, Kai Chen, Liejun Wang, Jungong Han, Guiguang Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06285
• Pdf URL: https://arxiv.org/pdf/2409.06285
• Copy Paste: [[2409.06285]] Context Enhancement with Reconstruction as Sequence for Unified Unsupervised Anomaly Detection(https://arxiv.org/abs/2409.06285)
• Keywords: robust, transformer
• Abstract: Unsupervised anomaly detection (AD) aims to train robust detection models using only normal samples, while can generalize well to unseen anomalies. Recent research focuses on a unified unsupervised AD setting in which only one model is trained for all classes, i.e., n-class-one-model paradigm. Feature-reconstruction-based methods achieve state-of-the-art performance in this scenario. However, existing methods often suffer from a lack of sufficient contextual awareness, thereby compromising the quality of the reconstruction. To address this issue, we introduce a novel Reconstruction as Sequence (RAS) method, which enhances the contextual correspondence during feature reconstruction from a sequence modeling perspective. In particular, based on the transformer technique, we integrate a specialized RASFormer block into RAS. This block enables the capture of spatial relationships among different image regions and enhances sequential dependencies throughout the reconstruction process. By incorporating the RASFormer block, our RAS method achieves superior contextual awareness capabilities, leading to remarkable performance. Experimental results show that our RAS significantly outperforms competing methods, well demonstrating the effectiveness and superiority of our method. Our code is available at this https URL.

Title: Enhancing Long Video Understanding via Hierarchical Event-Based Memory

• Authors: Dingxin Cheng, Mingda Li, Jingyu Liu, Yongxin Guo, Bin Jiang, Qingbin Liu, Xi Chen, Bo Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06299
• Pdf URL: https://arxiv.org/pdf/2409.06299
• Copy Paste: [[2409.06299]] Enhancing Long Video Understanding via Hierarchical Event-Based Memory(https://arxiv.org/abs/2409.06299)
• Keywords: large language model, segmentation
• Abstract: Recently, integrating visual foundation models into large language models (LLMs) to form video understanding systems has attracted widespread attention. Most of the existing models compress diverse semantic information within the whole video and feed it into LLMs for content comprehension. While this method excels in short video understanding, it may result in a blend of multiple event information in long videos due to coarse compression, which causes information redundancy. Consequently, the semantics of key events might be obscured within the vast information that hinders the model's understanding capabilities. To address this issue, we propose a Hierarchical Event-based Memory-enhanced LLM (HEM-LLM) for better understanding of long videos. Firstly, we design a novel adaptive sequence segmentation scheme to divide multiple events within long videos. In this way, we can perform individual memory modeling for each event to establish intra-event contextual connections, thereby reducing information redundancy. Secondly, while modeling current event, we compress and inject the information of the previous event to enhance the long-term inter-event dependencies in videos. Finally, we perform extensive experiments on various video understanding tasks and the results show that our model achieves state-of-the-art performances.

Title: High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study

• Authors: Shijie Chang, Lihe Zhang, Huchuan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06305
• Pdf URL: https://arxiv.org/pdf/2409.06305
• Copy Paste: [[2409.06305]] High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study(https://arxiv.org/abs/2409.06305)
• Keywords: segmentation
• Abstract: Existing few-shot segmentation (FSS) methods mainly focus on designing novel support-query matching and self-matching mechanisms to exploit implicit knowledge in pre-trained backbones. However, the performance of these methods is often constrained by models pre-trained on classification tasks. The exploration of what types of pre-trained models can provide more beneficial implicit knowledge for FSS remains limited. In this paper, inspired by the representation consistency of foundational computer vision models, we develop a FSS framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence and introduce a lightweight decoder to refine coarse correspondence for fine-grained segmentation. We systematically summarize the performance of various foundation models on FSS and discover that the implicit knowledge within some of these models is more beneficial for FSS than models pre-trained on classification tasks. Extensive experiments on two widely used datasets demonstrate the effectiveness of our approach in leveraging the implicit knowledge of foundation models. Notably, the combination of DINOv2 and DFN exceeds previous state-of-the-art methods by 17.5% on COCO-20i. Code is available at this https URL.

Title: PPMamba: A Pyramid Pooling Local Auxiliary SSM-Based Model for Remote Sensing Image Semantic Segmentation

• Authors: Yin Hu, Xianping Ma, Jialu Sui, Man-On Pun
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.06309
• Pdf URL: https://arxiv.org/pdf/2409.06309
• Copy Paste: [[2409.06309]] PPMamba: A Pyramid Pooling Local Auxiliary SSM-Based Model for Remote Sensing Image Semantic Segmentation(https://arxiv.org/abs/2409.06309)
• Keywords: transformer, segmentation
• Abstract: Semantic segmentation is a vital task in the field of remote sensing (RS). However, conventional convolutional neural network (CNN) and transformer-based models face limitations in capturing long-range dependencies or are often computationally intensive. Recently, an advanced state space model (SSM), namely Mamba, was introduced, offering linear computational complexity while effectively establishing long-distance dependencies. Despite their advantages, Mamba-based methods encounter challenges in preserving local semantic information. To cope with these challenges, this paper proposes a novel network called Pyramid Pooling Mamba (PPMamba), which integrates CNN and Mamba for RS semantic segmentation tasks. The core structure of PPMamba, the Pyramid Pooling-State Space Model (PP-SSM) block, combines a local auxiliary mechanism with an omnidirectional state space model (OSS) that selectively scans feature maps from eight directions, capturing comprehensive feature information. Additionally, the auxiliary mechanism includes pyramid-shaped convolutional branches designed to extract features at multiple scales. Extensive experiments on two widely-used datasets, ISPRS Vaihingen and LoveDA Urban, demonstrate that PPMamba achieves competitive performance compared to state-of-the-art models.

Title: Rate-Constrained Quantization for Communication-Efficient Federated Learning

• Authors: Shayan Mohajer Hamidi, Ali Bereyhi
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.06319
• Pdf URL: https://arxiv.org/pdf/2409.06319
• Copy Paste: [[2409.06319]] Rate-Constrained Quantization for Communication-Efficient Federated Learning(https://arxiv.org/abs/2409.06319)
• Keywords: federate
• Abstract: Quantization is a common approach to mitigate the communication cost of federated learning (FL). In practice, the quantized local parameters are further encoded via an entropy coding technique, such as Huffman coding, for efficient data compression. In this case, the exact communication overhead is determined by the bit rate of the encoded gradients. Recognizing this fact, this work deviates from the existing approaches in the literature and develops a novel quantized FL framework, called \textbf{r}ate-\textbf{c}onstrained \textbf{fed}erated learning (RC-FED), in which the gradients are quantized subject to both fidelity and data rate constraints. We formulate this scheme, as a joint optimization in which the quantization distortion is minimized while the rate of encoded gradients is kept below a target threshold. This enables for a tunable trade-off between quantization distortion and communication cost. We analyze the convergence behavior of RC-FED, and show its superior performance against baseline quantized FL schemes on several datasets.

Title: G3PT: Unleash the power of Autoregressive Modeling in 3D Generation via Cross-scale Querying Transformer

• Authors: Jinzhi Zhang, Feng Xiong, Mu Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06322
• Pdf URL: https://arxiv.org/pdf/2409.06322
• Copy Paste: [[2409.06322]] G3PT: Unleash the power of Autoregressive Modeling in 3D Generation via Cross-scale Querying Transformer(https://arxiv.org/abs/2409.06322)
• Keywords: transformer, generative
• Abstract: Autoregressive transformers have revolutionized generative models in language processing and shown substantial promise in image and video generation. However, these models face significant challenges when extended to 3D generation tasks due to their reliance on next-token prediction to learn token sequences, which is incompatible with the unordered nature of 3D data. Instead of imposing an artificial order on 3D data, in this paper, we introduce G3PT, a scalable coarse-to-fine 3D generative model utilizing a cross-scale querying transformer. The key is to map point-based 3D data into discrete tokens with different levels of detail, naturally establishing a sequential relationship between different levels suitable for autoregressive modeling. Additionally, the cross-scale querying transformer connects tokens globally across different levels of detail without requiring an ordered sequence. Benefiting from this approach, G3PT features a versatile 3D generation pipeline that effortlessly supports diverse conditional structures, enabling the generation of 3D shapes from various types of conditions. Extensive experiments demonstrate that G3PT achieves superior generation quality and generalization ability compared to previous 3D generation methods. Most importantly, for the first time in 3D generation, scaling up G3PT reveals distinct power-law scaling behaviors.

Title: LAMP: Learnable Meta-Path Guided Adversarial Contrastive Learning for Heterogeneous Graphs

• Authors: Siqing Li, Jin-Duk Park, Wei Huang, Xin Cao, Won-Yong Shin, Zhiqiang Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06323
• Pdf URL: https://arxiv.org/pdf/2409.06323
• Copy Paste: [[2409.06323]] LAMP: Learnable Meta-Path Guided Adversarial Contrastive Learning for Heterogeneous Graphs(https://arxiv.org/abs/2409.06323)
• Keywords: robust
• Abstract: Heterogeneous graph neural networks (HGNNs) have significantly propelled the information retrieval (IR) field. Still, the effectiveness of HGNNs heavily relies on high-quality labels, which are often expensive to acquire. This challenge has shifted attention towards Heterogeneous Graph Contrastive Learning (HGCL), which usually requires pre-defined meta-paths. However, our findings reveal that meta-path combinations significantly affect performance in unsupervised settings, an aspect often overlooked in current literature. Existing HGCL methods have considerable variability in outcomes across different meta-path combinations, thereby challenging the optimization process to achieve consistent and high performance. In response, we introduce \textsf{LAMP} (\underline{\textbf{L}}earn\underline{\textbf{A}}ble \underline{\textbf{M}}eta-\underline{\textbf{P}}ath), a novel adversarial contrastive learning approach that integrates various meta-path sub-graphs into a unified and stable structure, leveraging the overlap among these sub-graphs. To address the denseness of this integrated sub-graph, we propose an adversarial training strategy for edge pruning, maintaining sparsity to enhance model performance and robustness. \textsf{LAMP} aims to maximize the difference between meta-path and network schema views for guiding contrastive learning to capture the most meaningful information. Our extensive experimental study conducted on four diverse datasets from the Heterogeneous Graph Benchmark (HGB) demonstrates that \textsf{LAMP} significantly outperforms existing state-of-the-art unsupervised models in terms of accuracy and robustness.

Title: SDF-Net: A Hybrid Detection Network for Mediastinal Lymph Node Detection on Contrast CT Images

• Authors: Jiuli Xiong, Lanzhuju Mei, Jiameng Liu, Dinggang Shen, Zhong Xue, Xiaohuan Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06324
• Pdf URL: https://arxiv.org/pdf/2409.06324
• Copy Paste: [[2409.06324]] SDF-Net: A Hybrid Detection Network for Mediastinal Lymph Node Detection on Contrast CT Images(https://arxiv.org/abs/2409.06324)
• Keywords: segmentation
• Abstract: Accurate lymph node detection and quantification are crucial for cancer diagnosis and staging on contrast-enhanced CT images, as they impact treatment planning and prognosis. However, detecting lymph nodes in the mediastinal area poses challenges due to their low contrast, irregular shapes and dispersed distribution. In this paper, we propose a Swin-Det Fusion Network (SDF-Net) to effectively detect lymph nodes. SDF-Net integrates features from both segmentation and detection to enhance the detection capability of lymph nodes with various shapes and sizes. Specifically, an auto-fusion module is designed to merge the feature maps of segmentation and detection networks at different levels. To facilitate effective learning without mask annotations, we introduce a shape-adaptive Gaussian kernel to represent lymph node in the training stage and provide more anatomical information for effective learning. Comparative results demonstrate promising performance in addressing the complex lymph node detection problem.

Title: Extracting Paragraphs from LLM Token Activations

• Authors: Nicholas Pochinkov, Angelo Benoit, Lovkush Agarwal, Zainab Ali Majid, Lucile Ter-Minassian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06328
• Pdf URL: https://arxiv.org/pdf/2409.06328
• Copy Paste: [[2409.06328]] Extracting Paragraphs from LLM Token Activations(https://arxiv.org/abs/2409.06328)
• Keywords: generative, large language model
• Abstract: Generative large language models (LLMs) excel in natural language processing tasks, yet their inner workings remain underexplored beyond token-level predictions. This study investigates the degree to which these models decide the content of a paragraph at its onset, shedding light on their contextual understanding. By examining the information encoded in single-token activations, specifically the "\textbackslash n\textbackslash n" double newline token, we demonstrate that patching these activations can transfer significant information about the context of the following paragraph, providing further insights into the model's capacity to plan ahead.

Title: Multi-Weather Image Restoration via Histogram-Based Transformer Feature Enhancement

• Authors: Yang Wen, Anyu Lai, Bo Qian, Hao Wang, Wuzhen Shi, Wenming Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06334
• Pdf URL: https://arxiv.org/pdf/2409.06334
• Copy Paste: [[2409.06334]] Multi-Weather Image Restoration via Histogram-Based Transformer Feature Enhancement(https://arxiv.org/abs/2409.06334)
• Keywords: transformer
• Abstract: Currently, the mainstream restoration tasks under adverse weather conditions have predominantly focused on single-weather scenarios. However, in reality, multiple weather conditions always coexist and their degree of mixing is usually unknown. Under such complex and diverse weather conditions, single-weather restoration models struggle to meet practical demands. This is particularly critical in fields such as autonomous driving, where there is an urgent need for a model capable of effectively handling mixed weather conditions and enhancing image quality in an automated manner. In this paper, we propose a Task Sequence Generator module that, in conjunction with the Task Intra-patch Block, effectively extracts task-specific features embedded in degraded images. The Task Intra-patch Block introduces an external learnable sequence that aids the network in capturing task-specific information. Additionally, we employ a histogram-based transformer module as the backbone of our network, enabling the capture of both global and local dynamic range features. Our proposed model achieves state-of-the-art performance on public datasets.

Title: Improving Conditional Level Generation using Automated Validation in Match-3 Games

• Authors: Monica Villanueva Aylagas, Joakim Bergdahl, Jonas Gillberg, Alessandro Sestini, Theodor Tolstoy, Linus Gisslén
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06349
• Pdf URL: https://arxiv.org/pdf/2409.06349
• Copy Paste: [[2409.06349]] Improving Conditional Level Generation using Automated Validation in Match-3 Games(https://arxiv.org/abs/2409.06349)
• Keywords: generative
• Abstract: Generative models for level generation have shown great potential in game production. However, they often provide limited control over the generation, and the validity of the generated levels is unreliable. Despite this fact, only a few approaches that learn from existing data provide the users with ways of controlling the generation, simultaneously addressing the generation of unsolvable levels. %One of the main challenges it faces is that levels generated through automation may not be solvable thus requiring validation. are not always engaging, challenging, or even solvable. This paper proposes Avalon, a novel method to improve models that learn from existing level designs using difficulty statistics extracted from gameplay. In particular, we use a conditional variational autoencoder to generate layouts for match-3 levels, conditioning the model on pre-collected statistics such as game mechanics like difficulty and relevant visual features like size and symmetry. Our method is general enough that multiple approaches could potentially be used to generate these statistics. We quantitatively evaluate our approach by comparing it to an ablated model without difficulty conditioning. Additionally, we analyze both quantitatively and qualitatively whether the style of the dataset is preserved in the generated levels. Our approach generates more valid levels than the same method without difficulty conditioning.

Title: DiffQRCoder: Diffusion-based Aesthetic QR Code Generation with Scanning Robustness Guided Iterative Refinement

• Authors: Jia-Wei Liao, Winston Wang, Tzu-Sian Wang, Li-Xuan Peng, Ju-Hsuan Weng, Cheng-Fu Chou, Jun-Cheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06355
• Pdf URL: https://arxiv.org/pdf/2409.06355
• Copy Paste: [[2409.06355]] DiffQRCoder: Diffusion-based Aesthetic QR Code Generation with Scanning Robustness Guided Iterative Refinement(https://arxiv.org/abs/2409.06355)
• Keywords: robust, diffusion
• Abstract: With the success of Diffusion Models for image generation, the technologies also have revolutionized the aesthetic Quick Response (QR) code generation. Despite significant improvements in visual attractiveness for the beautified codes, their scannabilities are usually sacrificed and thus hinder their practical uses in real-world scenarios. To address this issue, we propose a novel Diffusion-based QR Code generator (DiffQRCoder) to effectively craft both scannable and visually pleasing QR codes. The proposed approach introduces Scanning-Robust Perceptual Guidance (SRPG), a new diffusion guidance for Diffusion Models to guarantee the generated aesthetic codes to obey the ground-truth QR codes while maintaining their attractiveness during the denoising process. Additionally, we present another post-processing technique, Scanning Robust Manifold Projected Gradient Descent (SR-MPGD), to further enhance their scanning robustness through iterative latent space optimization. With extensive experiments, the results demonstrate that our approach not only outperforms other compared methods in Scanning Success Rate (SSR) with better or comparable CLIP aesthetic score (CLIP-aes.) but also significantly improves the SSR of the ControlNet-only approach from 60% to 99%. The subjective evaluation indicates that our approach achieves promising visual attractiveness to users as well. Finally, even with different scanning angles and the most rigorous error tolerance settings, our approach robustly achieves over 95% SSR, demonstrating its capability for real-world applications.

Title: SoK: Evaluating 5G Protocols Against Legacy and Emerging Privacy and Security Attacks

• Authors: Stavros Eleftherakis, Domenico Giustiniano, Nicolas Kourtellis
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.06360
• Pdf URL: https://arxiv.org/pdf/2409.06360
• Copy Paste: [[2409.06360]] SoK: Evaluating 5G Protocols Against Legacy and Emerging Privacy and Security Attacks(https://arxiv.org/abs/2409.06360)
• Keywords: security, privacy, attack
• Abstract: Ensuring user privacy remains a critical concern within mobile cellular networks, particularly given the proliferation of interconnected devices and services. In fact, a lot of user privacy issues have been raised in 2G, 3G, 4G/LTE networks. Recognizing this general concern, 3GPP has prioritized addressing these issues in the development of 5G, implementing numerous modifications to enhance user privacy since 5G Release 15. In this systematization of knowledge paper, we first provide a framework for studying privacy and security related attacks in cellular networks, setting as privacy objective the User Identity Confidentiality defined in 3GPP standards. Using this framework, we discuss existing privacy and security attacks in pre-5G networks, analyzing the weaknesses that lead to these attacks. Furthermore, we thoroughly study the security characteristics of 5G up to the new Release 19, and examine mitigation mechanisms of 5G to the identified pre-5G attacks. Afterwards, we analyze how recent 5G attacks try to overcome these mitigation mechanisms. Finally, we identify current limitations and open problems in security of 5G, and propose directions for future work.

Title: Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks

• Authors: Teresa Dorszewski, Lenka Tětková, Lorenz Linhardt, Lars Kai Hansen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06362
• Pdf URL: https://arxiv.org/pdf/2409.06362
• Copy Paste: [[2409.06362]] Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks(https://arxiv.org/abs/2409.06362)
• Keywords: transformer
• Abstract: Understanding how neural networks align with human cognitive processes is a crucial step toward developing more interpretable and reliable AI systems. Motivated by theories of human cognition, this study examines the relationship between \emph{convexity} in neural network representations and \emph{human-machine alignment} based on behavioral data. We identify a correlation between these two dimensions in pretrained and fine-tuned vision transformer models. Our findings suggest that the convex regions formed in latent spaces of neural networks to some extent align with human-defined categories and reflect the similarity relations humans use in cognitive tasks. While optimizing for alignment generally enhances convexity, increasing convexity through fine-tuning yields inconsistent effects on alignment, which suggests a complex relationship between the two. This study presents a first step toward understanding the relationship between the convexity of latent representations and human-machine alignment.

Title: What happens to diffusion model likelihood when your model is conditional?

• Authors: Mattias Cross, Anton Ragni
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06364
• Pdf URL: https://arxiv.org/pdf/2409.06364
• Copy Paste: [[2409.06364]] What happens to diffusion model likelihood when your model is conditional?(https://arxiv.org/abs/2409.06364)
• Keywords: diffusion
• Abstract: Diffusion Models (DMs) iteratively denoise random samples to produce high-quality data. The iterative sampling process is derived from Stochastic Differential Equations (SDEs), allowing a speed-quality trade-off chosen at inference. Another advantage of sampling with differential equations is exact likelihood computation. These likelihoods have been used to rank unconditional DMs and for out-of-domain classification. Despite the many existing and possible uses of DM likelihoods, the distinct properties captured are unknown, especially in conditional contexts such as Text-To-Image (TTI) or Text-To-Speech synthesis (TTS). Surprisingly, we find that TTS DM likelihoods are agnostic to the text input. TTI likelihood is more expressive but cannot discern confounding prompts. Our results show that applying DMs to conditional tasks reveals inconsistencies and strengthens claims that the properties of DM likelihood are unknown. This impact sheds light on the previously unknown nature of DM likelihoods. Although conditional DMs maximise likelihood, the likelihood in question is not as sensitive to the conditioning input as one expects. This investigation provides a new point-of-view on diffusion likelihoods.

Title: Distilling Generative-Discriminative Representations for Very Low-Resolution Face Recognition

• Authors: Junzheng Zhang, Weijia Guo, Bochao Liu, Ruixin Shi, Yong Li, Shiming Ge
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.06371
• Pdf URL: https://arxiv.org/pdf/2409.06371
• Copy Paste: [[2409.06371]] Distilling Generative-Discriminative Representations for Very Low-Resolution Face Recognition(https://arxiv.org/abs/2409.06371)
• Keywords: robust, diffusion, generative
• Abstract: Very low-resolution face recognition is challenging due to the serious loss of informative facial details in resolution degradation. In this paper, we propose a generative-discriminative representation distillation approach that combines generative representation with cross-resolution aligned knowledge distillation. This approach facilitates very low-resolution face recognition by jointly distilling generative and discriminative models via two distillation modules. Firstly, the generative representation distillation takes the encoder of a diffusion model pretrained for face super-resolution as the generative teacher to supervise the learning of the student backbone via feature regression, and then freezes the student backbone. After that, the discriminative representation distillation further considers a pretrained face recognizer as the discriminative teacher to supervise the learning of the student head via cross-resolution relational contrastive distillation. In this way, the general backbone representation can be transformed into discriminative head representation, leading to a robust and discriminative student model for very low-resolution face recognition. Our approach improves the recovery of the missing details in very low-resolution faces and achieves better knowledge transfer. Extensive experiments on face datasets demonstrate that our approach enhances the recognition accuracy of very low-resolution faces, showcasing its effectiveness and adaptability.

Title: SpeechTaxi: On Multilingual Semantic Speech Classification

• Authors: Lennart Keller, Goran Glavaš
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.06372
• Pdf URL: https://arxiv.org/pdf/2409.06372
• Copy Paste: [[2409.06372]] SpeechTaxi: On Multilingual Semantic Speech Classification(https://arxiv.org/abs/2409.06372)
• Keywords: robust
• Abstract: Recent advancements in multilingual speech encoding as well as transcription raise the question of the most effective approach to semantic speech classification. Concretely, can (1) end-to-end (E2E) classifiers obtained by fine-tuning state-of-the-art multilingual speech encoders (MSEs) match or surpass the performance of (2) cascading (CA), where speech is first transcribed into text and classification is delegated to a text-based classifier. To answer this, we first construct SpeechTaxi, an 80-hour multilingual dataset for semantic speech classification of Bible verses, covering 28 diverse languages. We then leverage SpeechTaxi to conduct a wide range of experiments comparing E2E and CA in monolingual semantic speech classification as well as in cross-lingual transfer. We find that E2E based on MSEs outperforms CA in monolingual setups, i.e., when trained on in-language data. However, MSEs seem to have poor cross-lingual transfer abilities, with E2E substantially lagging CA both in (1) zero-shot transfer to languages unseen in training and (2) multilingual training, i.e., joint training on multiple languages. Finally, we devise a novel CA approach based on transcription to Romanized text as a language-agnostic intermediate representation and show that it represents a robust solution for languages without native ASR support. Our SpeechTaxi dataset is publicly available at: this https URL datasets/LennartKeller/SpeechTaxi/.

Title: The Impact of SBOM Generators on Vulnerability Assessment in Python: A Comparison and a Novel Approach

• Authors: Giacomo Benedetti, Serena Cofano, Alessandro Brighente, Mauro Conti
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.06390
• Pdf URL: https://arxiv.org/pdf/2409.06390
• Copy Paste: [[2409.06390]] The Impact of SBOM Generators on Vulnerability Assessment in Python: A Comparison and a Novel Approach(https://arxiv.org/abs/2409.06390)
• Keywords: security
• Abstract: The Software Supply Chain (SSC) security is a critical concern for both users and developers. Recent incidents, like the SolarWinds Orion compromise, proved the widespread impact resulting from the distribution of compromised software. The reliance on open-source components, which constitute a significant portion of modern software, further exacerbates this risk. To enhance SSC security, the Software Bill of Materials (SBOM) has been promoted as a tool to increase transparency and verifiability in software composition. However, despite its promise, SBOMs are not without limitations. Current SBOM generation tools often suffer from inaccuracies in identifying components and dependencies, leading to the creation of erroneous or incomplete representations of the SSC. Despite existing studies exposing these limitations, their impact on the vulnerability detection capabilities of security tools is still unknown. In this paper, we perform the first security analysis on the vulnerability detection capabilities of tools receiving SBOMs as input. We comprehensively evaluate SBOM generation tools by providing their outputs to vulnerability identification software. Based on our results, we identify the root causes of these tools' ineffectiveness and propose PIP-sbom, a novel pip-inspired solution that addresses their shortcomings. PIP-sbom provides improved accuracy in component identification and dependency resolution. Compared to best-performing state-of-the-art tools, PIP-sbom increases the average precision and recall by 60%, and reduces by ten times the number of false positives.

Title: Length Desensitization in Directed Preference Optimization

• Authors: Wei Liu, Yang Bai, Chengcheng Han, Rongxiang Weng, Jun Xu, Xuezhi Cao, Jingang Wang, Xunliang Cai
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06411
• Pdf URL: https://arxiv.org/pdf/2409.06411
• Copy Paste: [[2409.06411]] Length Desensitization in Directed Preference Optimization(https://arxiv.org/abs/2409.06411)
• Keywords: large language model
• Abstract: Direct Preference Optimization (DPO) is widely utilized in the Reinforcement Learning from Human Feedback (RLHF) phase to align Large Language Models (LLMs) with human preferences, thereby enhancing both their harmlessness and efficacy. However, it has been observed that DPO tends to over-optimize for verbosity, which can detrimentally affect both performance and user experience. In this paper, we conduct an in-depth theoretical analysis of DPO's optimization objective and reveal a strong correlation between its implicit reward and data length. This correlation misguides the optimization direction, resulting in length sensitivity during the DPO training and leading to verbosity. To address this issue, we propose a length-desensitization improvement method for DPO, termed LD-DPO. The proposed method aims to desensitize DPO to data length by decoupling explicit length preference, which is relatively insignificant, from the other implicit preferences, thereby enabling more effective learning of the intrinsic preferences. We utilized two settings (Base and Instruct) of Llama2-13B, Llama3-8B, and Qwen2-7B for experimental validation on various benchmarks including MT-Bench and AlpacaEval 2. The experimental results indicate that LD-DPO consistently outperforms DPO and other baseline methods, achieving more concise responses with a 10-40\% reduction in length compared to DPO. We conducted in-depth experimental analyses to demonstrate that LD-DPO can indeed achieve length desensitization and align the model more closely with human-real preferences.

Title: A Pervasive, Efficient and Private Future: Realizing Privacy-Preserving Machine Learning Through Hybrid Homomorphic Encryption

• Authors: Khoa Nguyen, Mindaugas Budzys, Eugene Frimpong, Tanveer Khan, Antonis Michalas
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.06422
• Pdf URL: https://arxiv.org/pdf/2409.06422
• Copy Paste: [[2409.06422]] A Pervasive, Efficient and Private Future: Realizing Privacy-Preserving Machine Learning Through Hybrid Homomorphic Encryption(https://arxiv.org/abs/2409.06422)
• Keywords: security, privacy, attack
• Abstract: Machine Learning (ML) has become one of the most impactful fields of data science in recent years. However, a significant concern with ML is its privacy risks due to rising attacks against ML models. Privacy-Preserving Machine Learning (PPML) methods have been proposed to mitigate the privacy and security risks of ML models. A popular approach to achieving PPML uses Homomorphic Encryption (HE). However, the highly publicized inefficiencies of HE make it unsuitable for highly scalable scenarios with resource-constrained devices. Hence, Hybrid Homomorphic Encryption (HHE) -- a modern encryption scheme that combines symmetric cryptography with HE -- has recently been introduced to overcome these challenges. HHE potentially provides a foundation to build new efficient and privacy-preserving services that transfer expensive HE operations to the cloud. This work introduces HHE to the ML field by proposing resource-friendly PPML protocols for edge devices. More precisely, we utilize HHE as the primary building block of our PPML protocols. We assess the performance of our protocols by first extensively evaluating each party's communication and computational cost on a dummy dataset and show the efficiency of our protocols by comparing them with similar protocols implemented using plain BFV. Subsequently, we demonstrate the real-world applicability of our construction by building an actual PPML application that uses HHE as its foundation to classify heart disease based on sensitive ECG data.

Title: A Likelihood Ratio-Based Approach to Segmenting Unknown Objects

• Authors: Nazir Nayal, Youssef Shoeb, Fatma Güney
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06424
• Pdf URL: https://arxiv.org/pdf/2409.06424
• Copy Paste: [[2409.06424]] A Likelihood Ratio-Based Approach to Segmenting Unknown Objects(https://arxiv.org/abs/2409.06424)
• Keywords: robust, segmentation
• Abstract: Addressing the Out-of-Distribution (OoD) segmentation task is a prerequisite for perception systems operating in an open-world environment. Large foundational models are frequently used in downstream tasks, however, their potential for OoD remains mostly unexplored. We seek to leverage a large foundational model to achieve robust representation. Outlier supervision is a widely used strategy for improving OoD detection of the existing segmentation networks. However, current approaches for outlier supervision involve retraining parts of the original network, which is typically disruptive to the model's learned feature representation. Furthermore, retraining becomes infeasible in the case of large foundational models. Our goal is to retrain for outlier segmentation without compromising the strong representation space of the foundational model. To this end, we propose an adaptive, lightweight unknown estimation module (UEM) for outlier supervision that significantly enhances the OoD segmentation performance without affecting the learned feature representation of the original network. UEM learns a distribution for outliers and a generic distribution for known classes. Using the learned distributions, we propose a likelihood-ratio-based outlier scoring function that fuses the confidence of UEM with that of the pixel-wise segmentation inlier network to detect unknown objects. We also propose an objective to optimize this score directly. Our approach achieves a new state-of-the-art across multiple datasets, outperforming the previous best method by 5.74% average precision points while having a lower false-positive rate. Importantly, strong inlier performance remains unaffected.

Title: Prompt2Fashion: An automatically generated fashion dataset

• Authors: Georgia Argyro, Angeliki Dimitriou, Maria Lymperaiou, Giorgos Filandrianos, Giorgos Stamou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06442
• Pdf URL: https://arxiv.org/pdf/2409.06442
• Copy Paste: [[2409.06442]] Prompt2Fashion: An automatically generated fashion dataset(https://arxiv.org/abs/2409.06442)
• Keywords: generative, large language model
• Abstract: Despite the rapid evolution and increasing efficacy of language and vision generative models, there remains a lack of comprehensive datasets that bridge the gap between personalized fashion needs and AI-driven design, limiting the potential for truly inclusive and customized fashion solutions. In this work, we leverage generative models to automatically construct a fashion image dataset tailored to various occasions, styles, and body types as instructed by users. We use different Large Language Models (LLMs) and prompting strategies to offer personalized outfits of high aesthetic quality, detail, and relevance to both expert and non-expert users' requirements, as demonstrated by qualitative analysis. Up until now the evaluation of the generated outfits has been conducted by non-expert human subjects. Despite the provided fine-grained insights on the quality and relevance of generation, we extend the discussion on the importance of expert knowledge for the evaluation of artistic AI-generated datasets such as this one. Our dataset is publicly available on GitHub at this https URL.

Title: Knowledge Distillation via Query Selection for Detection Transformer

• Authors: Yi Liu, Luting Wang, Zongheng Tang, Yue Liao, Yifan Sun, Lijun Zhang, Si Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06443
• Pdf URL: https://arxiv.org/pdf/2409.06443
• Copy Paste: [[2409.06443]] Knowledge Distillation via Query Selection for Detection Transformer(https://arxiv.org/abs/2409.06443)
• Keywords: transformer
• Abstract: Transformers have revolutionized the object detection landscape by introducing DETRs, acclaimed for their simplicity and efficacy. Despite their advantages, the substantial size of these models poses significant challenges for practical deployment, particularly in resource-constrained environments. This paper addresses the challenge of compressing DETR by leveraging knowledge distillation, a technique that holds promise for maintaining model performance while reducing size. A critical aspect of DETRs' performance is their reliance on queries to interpret object representations accurately. Traditional distillation methods often focus exclusively on positive queries, identified through bipartite matching, neglecting the rich information present in hard-negative queries. Our visual analysis indicates that hard-negative queries, focusing on foreground elements, are crucial for enhancing distillation outcomes. To this end, we introduce a novel Group Query Selection strategy, which diverges from traditional query selection in DETR distillation by segmenting queries based on their Generalized Intersection over Union (GIoU) with ground truth objects, thereby uncovering valuable hard-negative queries for distillation. Furthermore, we present the Knowledge Distillation via Query Selection for DETR (QSKD) framework, which incorporates Attention-Guided Feature Distillation (AGFD) and Local Alignment Prediction Distillation (LAPD). These components optimize the distillation process by focusing on the most informative aspects of the teacher model's intermediate features and output. Our comprehensive experimental evaluation of the MS-COCO dataset demonstrates the effectiveness of our approach, significantly improving average precision (AP) across various DETR architectures without incurring substantial computational costs. Specifically, the AP of Conditional DETR ResNet-18 increased from 35.8 to 39.9.

Title: Learning Generative Interactive Environments By Trained Agent Exploration

• Authors: Naser Kazemi, Nedko Savov, Danda Paudel, Luc Van Gool
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06445
• Pdf URL: https://arxiv.org/pdf/2409.06445
• Copy Paste: [[2409.06445]] Learning Generative Interactive Environments By Trained Agent Exploration(https://arxiv.org/abs/2409.06445)
• Keywords: generative
• Abstract: World models are increasingly pivotal in interpreting and simulating the rules and actions of complex environments. Genie, a recent model, excels at learning from visually diverse environments but relies on costly human-collected data. We observe that their alternative method of using random agents is too limited to explore the environment. We propose to improve the model by employing reinforcement learning based agents for data generation. This approach produces diverse datasets that enhance the model's ability to adapt and perform well across various scenarios and realistic actions within the environment. In this paper, we first release the model GenieRedux - an implementation based on Genie. Additionally, we introduce GenieRedux-G, a variant that uses the agent's readily available actions to factor out action prediction uncertainty during validation. Our evaluation, including a replication of the Coinrun case study, shows that GenieRedux-G achieves superior visual fidelity and controllability using the trained agent exploration. The proposed approach is reproducable, scalable and adaptable to new types of environments. Our codebase is available at this https URL .

Title: HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Data

• Authors: Hossein Hajipour, Lea Schönherr, Thorsten Holz, Mario Fritz
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.06446
• Pdf URL: https://arxiv.org/pdf/2409.06446
• Copy Paste: [[2409.06446]] HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Data(https://arxiv.org/abs/2409.06446)
• Keywords: secure, security, large language model
• Abstract: Large language models (LLMs) have shown great potential for automatic code generation and form the basis for various tools such as GitHub Copilot. However, recent studies highlight that many LLM-generated code contains serious security vulnerabilities. While previous work tries to address this by training models that generate secure code, these attempts remain constrained by limited access to training data and labor-intensive data preparation. In this paper, we introduce HexaCoder, a novel approach to enhance the ability of LLMs to generate secure codes by automatically synthesizing secure codes, which reduces the effort of finding suitable training data. HexaCoder comprises two key components: an oracle-guided data synthesis pipeline and a two-step process for secure code generation. The data synthesis pipeline generates pairs of vulnerable and fixed codes for specific Common Weakness Enumeration (CWE) types by utilizing a state-of-the-art LLM for repairing vulnerable code. A security oracle identifies vulnerabilities, and a state-of-the-art LLM repairs them by extending and/or editing the codes, creating data pairs for fine-tuning using the Low-Rank Adaptation (LoRA) method. Each example of our fine-tuning dataset includes the necessary security-related libraries and code that form the basis of our novel two-step generation approach. This allows the model to integrate security-relevant libraries before generating the main code, significantly reducing the number of generated vulnerable codes by up to 85% compared to the baseline methods. We perform extensive evaluations on three different benchmarks for four LLMs, demonstrating that HexaCoder not only improves the security of the generated code but also maintains a high level of functional correctness.

Title: Ransomware Detection Using Machine Learning in the Linux Kernel

• Authors: Adrian Brodzik, Tomasz Malec-Kruszyński, Wojciech Niewolski, Mikołaj Tkaczyk, Krzysztof Bocianiak, Sok-Yen Loui
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06452
• Pdf URL: https://arxiv.org/pdf/2409.06452
• Copy Paste: [[2409.06452]] Ransomware Detection Using Machine Learning in the Linux Kernel(https://arxiv.org/abs/2409.06452)
• Keywords: protect, attack
• Abstract: Linux-based cloud environments have become lucrative targets for ransomware attacks, employing various encryption schemes at unprecedented speeds. Addressing the urgency for real-time ransomware protection, we propose leveraging the extended Berkeley Packet Filter (eBPF) to collect system call information regarding active processes and infer about the data directly at the kernel level. In this study, we implement two Machine Learning (ML) models in eBPF - a decision tree and a multilayer perceptron. Benchmarking latency and accuracy against their user space counterparts, our findings underscore the efficacy of this approach.

Title: A Machine Learning Based Approach for Statistical Analysis of Detonation Cells from Soot Foils

• Authors: Vansh Sharma, Michael Ullman, Venkat Raman
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06466
• Pdf URL: https://arxiv.org/pdf/2409.06466
• Copy Paste: [[2409.06466]] A Machine Learning Based Approach for Statistical Analysis of Detonation Cells from Soot Foils(https://arxiv.org/abs/2409.06466)
• Keywords: robust, segmentation
• Abstract: This study presents a novel algorithm based on machine learning (ML) for the precise segmentation and measurement of detonation cells from soot foil images, addressing the limitations of manual and primitive edge detection methods prevalent in the field. Using advances in cellular biology segmentation models, the proposed algorithm is designed to accurately extract cellular patterns without a training procedure or dataset, which is a significant challenge in detonation research. The algorithm's performance was validated using a series of test cases that mimic experimental and numerical detonation studies. The results demonstrated consistent accuracy, with errors remaining within 10%, even in complex cases. The algorithm effectively captured key cell metrics such as cell area and span, revealing trends across different soot foil samples with uniform to highly irregular cellular structures. Although the model proved robust, challenges remain in segmenting and analyzing highly complex or irregular cellular patterns. This work highlights the broad applicability and potential of the algorithm to advance the understanding of detonation wave dynamics.

Title: UAVDB: Trajectory-Guided Adaptable Bounding Boxes for UAV Detection

• Authors: Yu-Hsi Chen
• Subjects: cs.CV, stat.AP
• Abstract URL: https://arxiv.org/abs/2409.06490
• Pdf URL: https://arxiv.org/pdf/2409.06490
• Copy Paste: [[2409.06490]] UAVDB: Trajectory-Guided Adaptable Bounding Boxes for UAV Detection(https://arxiv.org/abs/2409.06490)
• Keywords: security
• Abstract: With the rapid development of drone technology, accurate detection of Unmanned Aerial Vehicles (UAVs) has become essential for applications such as surveillance, security, and airspace management. In this paper, we propose a novel trajectory-guided method, the Patch Intensity Convergence (PIC) technique, which generates high-fidelity bounding boxes for UAV detection tasks and no need for the effort required for labeling. The PIC technique forms the foundation for developing UAVDB, a database explicitly created for UAV detection. Unlike existing datasets, which often use low-resolution footage or focus on UAVs in simple backgrounds, UAVDB employs high-resolution video to capture UAVs at various scales, ranging from hundreds of pixels to nearly single-digit sizes. This broad-scale variation enables comprehensive evaluation of detection algorithms across different UAV sizes and distances. Applying the PIC technique, we can also efficiently generate detection datasets from trajectory or positional data, even without size information. We extensively benchmark UAVDB using YOLOv8 series detectors, offering a detailed performance analysis. Our findings highlight UAVDB's potential as a vital database for advancing UAV detection, particularly in high-resolution and long-distance tracking scenarios.

Title: Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models

• Authors: Rohit Jena, Ali Taghibakhshi, Sahil Jain, Gerald Shen, Nima Tajbakhsh, Arash Vahdat
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06493
• Pdf URL: https://arxiv.org/pdf/2409.06493
• Copy Paste: [[2409.06493]] Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models(https://arxiv.org/abs/2409.06493)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) diffusion models have become prominent tools for generating high-fidelity images from text prompts. However, when trained on unfiltered internet data, these models can produce unsafe, incorrect, or stylistically undesirable images that are not aligned with human preferences. To address this, recent approaches have incorporated human preference datasets to fine-tune T2I models or to optimize reward functions that capture these preferences. Although effective, these methods are vulnerable to reward hacking, where the model overfits to the reward function, leading to a loss of diversity in the generated images. In this paper, we prove the inevitability of reward hacking and study natural regularization techniques like KL divergence and LoRA scaling, and their limitations for diffusion models. We also introduce Annealed Importance Guidance (AIG), an inference-time regularization inspired by Annealed Importance Sampling, which retains the diversity of the base model while achieving Pareto-Optimal reward-diversity tradeoffs. Our experiments demonstrate the benefits of AIG for Stable Diffusion models, striking the optimal balance between reward optimization and image diversity. Furthermore, a user study confirms that AIG improves diversity and quality of generated images across different model architectures and reward functions.

Title: Neural Laplacian Operator for 3D Point Clouds

• Authors: Bo Pang, Zhongtian Zheng, Yilong Li, Guoping Wang, Peng-Shuai Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.06506
• Pdf URL: https://arxiv.org/pdf/2409.06506
• Copy Paste: [[2409.06506]] Neural Laplacian Operator for 3D Point Clouds(https://arxiv.org/abs/2409.06506)
• Keywords: robust
• Abstract: The discrete Laplacian operator holds a crucial role in 3D geometry processing, yet it is still challenging to define it on point clouds. Previous works mainly focused on constructing a local triangulation around each point to approximate the underlying manifold for defining the Laplacian operator, which may not be robust or accurate. In contrast, we simply use the K-nearest neighbors (KNN) graph constructed from the input point cloud and learn the Laplacian operator on the KNN graph with graph neural networks (GNNs). However, the ground-truth Laplacian operator is defined on a manifold mesh with a different connectivity from the KNN graph and thus cannot be directly used for training. To train the GNN, we propose a novel training scheme by imitating the behavior of the ground-truth Laplacian operator on a set of probe functions so that the learned Laplacian operator behaves similarly to the ground-truth Laplacian operator. We train our network on a subset of ShapeNet and evaluate it across a variety of point clouds. Compared with previous methods, our method reduces the error by an order of magnitude and excels in handling sparse point clouds with thin structures or sharp features. Our method also demonstrates a strong generalization ability to unseen shapes. With our learned Laplacian operator, we further apply a series of Laplacian-based geometry processing algorithms directly to point clouds and achieve accurate results, enabling many exciting possibilities for geometry processing on point clouds. The code and trained models are available at this https URL.

Title: DroneXNFT: An NFT-Driven Framework for Secure Autonomous UAV Operations and Flight Data Management

• Authors: Khaoula Hidawi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.06507
• Pdf URL: https://arxiv.org/pdf/2409.06507
• Copy Paste: [[2409.06507]] DroneXNFT: An NFT-Driven Framework for Secure Autonomous UAV Operations and Flight Data Management(https://arxiv.org/abs/2409.06507)
• Keywords: secure, privacy
• Abstract: Non-Fungible Tokens (NFTs) have emerged as a revolutionary method for managing digital assets, providing transparency and secure ownership records on a blockchain. In this paper, we present a theoretical framework for leveraging NFTs to manage UAV (Unmanned Aerial Vehicle) flight data. Our approach focuses on ensuring data integrity, ownership transfer, and secure data sharing among stakeholders. This framework utilizes cryptographic methods, smart contracts, and access control mechanisms to enable a tamper-proof and privacy-preserving management system for UAV flight data.

Title: Aligning Machine and Human Visual Representations across Abstraction Levels

• Authors: Lukas Muttenthaler, Klaus Greff, Frieda Born, Bernhard Spitzer, Simon Kornblith, Michael C. Mozer, Klaus-Robert Müller, Thomas Unterthiner, Andrew K. Lampinen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06509
• Pdf URL: https://arxiv.org/pdf/2409.06509
• Copy Paste: [[2409.06509]] Aligning Machine and Human Visual Representations across Abstraction Levels(https://arxiv.org/abs/2409.06509)
• Keywords: robust
• Abstract: Deep neural networks have achieved success across a wide range of applications, including as models of human behavior in vision tasks. However, neural network training and human learning differ in fundamental ways, and neural networks often fail to generalize as robustly as humans do, raising questions regarding the similarity of their underlying representations. What is missing for modern learning systems to exhibit more human-like behavior? We highlight a key misalignment between vision models and humans: whereas human conceptual knowledge is hierarchically organized from fine- to coarse-scale distinctions, model representations do not accurately capture all these levels of abstraction. To address this misalignment, we first train a teacher model to imitate human judgments, then transfer human-like structure from its representations into pretrained state-of-the-art vision foundation models. These human-aligned models more accurately approximate human behavior and uncertainty across a wide range of similarity tasks, including a new dataset of human judgments spanning multiple levels of semantic abstractions. They also perform better on a diverse set of machine learning tasks, increasing generalization and out-of-distribution robustness. Thus, infusing neural networks with additional human knowledge yields a best-of-both-worlds representation that is both more consistent with human cognition and more practically useful, thus paving the way toward more robust, interpretable, and human-like artificial intelligence systems.

Title: Questioning Internal Knowledge Structure of Large Language Models Through the Lens of the Olympic Games

• Authors: Juhwan Choi, YoungBin Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06518
• Pdf URL: https://arxiv.org/pdf/2409.06518
• Copy Paste: [[2409.06518]] Questioning Internal Knowledge Structure of Large Language Models Through the Lens of the Olympic Games(https://arxiv.org/abs/2409.06518)
• Keywords: large language model
• Abstract: Large language models (LLMs) have become a dominant approach in natural language processing, yet their internal knowledge structures remain largely unexplored. In this paper, we analyze the internal knowledge structures of LLMs using historical medal tallies from the Olympic Games. We task the models with providing the medal counts for each team and identifying which teams achieved specific rankings. Our results reveal that while state-of-the-art LLMs perform remarkably well in reporting medal counts for individual teams, they struggle significantly with questions about specific rankings. This suggests that the internal knowledge structures of LLMs are fundamentally different from those of humans, who can easily infer rankings from known medal counts. To support further research, we publicly release our code, dataset, and model outputs.

Title: In Flight Boresight Rectification for Lightweight Airborne Pushbroom Imaging Spectrometry

• Authors: Julien Yuuki Burkhard, Jesse Ray Murray Lahaye, Laurent Valentin Jospin, Jan Skaloud
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06520
• Pdf URL: https://arxiv.org/pdf/2409.06520
• Copy Paste: [[2409.06520]] In Flight Boresight Rectification for Lightweight Airborne Pushbroom Imaging Spectrometry(https://arxiv.org/abs/2409.06520)
• Keywords: extraction
• Abstract: Hyperspectral cameras have recently been miniaturized for operation on lightweight airborne platforms such as UAV or small aircraft. Unlike frame cameras (RGB or Multispectral), many hyperspectral sensors use a linear array or 'push-broom' scanning design. This design presents significant challenges for image rectification and the calibration of the intrinsic and extrinsic camera parameters. Typically, methods employed to address such tasks rely on a precise GPS/INS estimate of the airborne platform trajectory and a detailed terrain model. However, inaccuracies in the trajectory or surface model information can introduce systematic errors and complicate geometric modeling which ultimately degrade the quality of the rectification. To overcome these challenges, we propose a method for tie point extraction and camera calibration for 'push-broom' hyperspectral sensors using only the raw spectral imagery and raw, possibly low quality, GPS/INS trajectory. We demonstrate that our approach allows for the automatic calibration of airborne systems with hyperspectral cameras, outperforms other state-of-the-art automatic rectification methods and reaches an accuracy on par with manual calibration methods.

Title: Deep Learning for Koopman Operator Estimation in Idealized Atmospheric Dynamics

• Authors: David Millard, Arielle Carr, Stéphane Gaudreault
• Subjects: cs.LG, math.DS, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2409.06522
• Pdf URL: https://arxiv.org/pdf/2409.06522
• Copy Paste: [[2409.06522]] Deep Learning for Koopman Operator Estimation in Idealized Atmospheric Dynamics(https://arxiv.org/abs/2409.06522)
• Keywords: interpretability
• Abstract: Deep learning is revolutionizing weather forecasting, with new data-driven models achieving accuracy on par with operational physical models for medium-term predictions. However, these models often lack interpretability, making their underlying dynamics difficult to understand and explain. This paper proposes methodologies to estimate the Koopman operator, providing a linear representation of complex nonlinear dynamics to enhance the transparency of data-driven models. Despite its potential, applying the Koopman operator to large-scale problems, such as atmospheric modeling, remains challenging. This study aims to identify the limitations of existing methods, refine these models to overcome various bottlenecks, and introduce novel convolutional neural network architectures that capture simplified dynamics.

Title: PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation

• Authors: Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, Grégory Rogez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06535
• Pdf URL: https://arxiv.org/pdf/2409.06535
• Copy Paste: [[2409.06535]] PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation(https://arxiv.org/abs/2409.06535)
• Keywords: transformer
• Abstract: Aligning multiple modalities in a latent space, such as images and texts, has shown to produce powerful semantic visual representations, fueling tasks like image captioning, text-to-image generation, or image grounding. In the context of human-centric vision, albeit CLIP-like representations encode most standard human poses relatively well (such as standing or sitting), they lack sufficient acuteness to discern detailed or uncommon ones. Actually, while 3D human poses have been often associated with images (e.g. to perform pose estimation or pose-conditioned image generation), or more recently with text (e.g. for text-to-pose generation), they have seldom been paired with both. In this work, we combine 3D poses, person's pictures and textual pose descriptions to produce an enhanced 3D-, visual- and semantic-aware human pose representation. We introduce a new transformer-based model, trained in a retrieval fashion, which can take as input any combination of the aforementioned modalities. When composing modalities, it outperforms a standard multi-modal alignment retrieval model, making it possible to sort out partial information (e.g. image with the lower body occluded). We showcase the potential of such an embroidered pose representation for (1) SMPL regression from image with optional text cue; and (2) on the task of fine-grained instruction generation, which consists in generating a text that describes how to move from one 3D pose to another (as a fitness coach). Unlike prior works, our model can take any kind of input (image and/or pose) without retraining.

Title: Adversary Resilient Learned Bloom Filters

• Authors: Allison Bishop, Hayder Tirmazi
• Subjects: cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2409.06556
• Pdf URL: https://arxiv.org/pdf/2409.06556
• Copy Paste: [[2409.06556]] Adversary Resilient Learned Bloom Filters(https://arxiv.org/abs/2409.06556)
• Keywords: secure
• Abstract: Creating an adversary resilient Learned Bloom Filter \cite{learnedindexstructures} with provable guarantees is an open problem \cite{reviriego1}. We define a strong adversarial model for the Learned Bloom Filter. We also construct two adversary resilient variants of the Learned Bloom Filter called the Uptown Bodega Filter and the Downtown Bodega Filter. Our adversarial model extends an existing adversarial model designed for the Classical (i.e not ``Learned'') Bloom Filter by Naor Yogev~\cite{moni1} and considers computationally bounded adversaries that run in probabilistic polynomial time (PPT). We show that if pseudo-random permutations exist, then a secure Learned Bloom Filter may be constructed with $\lambda$ extra bits of memory and at most one extra pseudo-random permutation in the critical path. We further show that, if pseudo-random permutations exist, then a \textit{high utility} Learned Bloom Filter may be constructed with $2\lambda$ extra bits of memory and at most one extra pseudo-random permutation in the critical path. Finally, we construct a hybrid adversarial model for the case where a fraction of the workload is chosen by an adversary. We show realistic scenarios where using the Downtown Bodega Filter gives better performance guarantees compared to alternative approaches in this hybrid model.

Title: Advancing Android Privacy Assessments with Automation

• Authors: Mugdha Khedkar, Michael Schlichtig, Eric Bodden
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.06564
• Pdf URL: https://arxiv.org/pdf/2409.06564
• Copy Paste: [[2409.06564]] Advancing Android Privacy Assessments with Automation(https://arxiv.org/abs/2409.06564)
• Keywords: security, privacy, protect
• Abstract: Android apps collecting data from users must comply with legal frameworks to ensure data protection. This requirement has become even more important since the implementation of the General Data Protection Regulation (GDPR) by the European Union in 2018. Moreover, with the proposed Cyber Resilience Act on the horizon, stakeholders will soon need to assess software against even more stringent security and privacy standards. Effective privacy assessments require collaboration among groups with diverse expertise to function effectively as a cohesive unit. This paper motivates the need for an automated approach that enhances understanding of data protection in Android apps and improves communication between the various parties involved in privacy assessments. We propose the Assessor View, a tool designed to bridge the knowledge gap between these parties, facilitating more effective privacy assessments of Android applications.

Title: Quantifying and Enabling the Interpretability of CLIP-like Models

• Authors: Avinash Madasu, Yossi Gandelsman, Vasudev Lal, Phillip Howard
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06579
• Pdf URL: https://arxiv.org/pdf/2409.06579
• Copy Paste: [[2409.06579]] Quantifying and Enabling the Interpretability of CLIP-like Models(https://arxiv.org/abs/2409.06579)
• Keywords: interpretability, segmentation
• Abstract: CLIP is one of the most popular foundational models and is heavily used for many vision-language tasks. However, little is known about the inner workings of CLIP. To bridge this gap we propose a study to quantify the interpretability in CLIP like models. We conduct this study on six different CLIP models from OpenAI and OpenCLIP which vary by size, type of pre-training data and patch size. Our approach begins with using the TEXTSPAN algorithm and in-context learning to break down individual attention heads into specific properties. We then evaluate how easily these heads can be interpreted using new metrics which measure property consistency within heads and property disentanglement across heads. Our findings reveal that larger CLIP models are generally more interpretable than their smaller counterparts. To further assist users in understanding the inner workings of CLIP models, we introduce CLIP-InterpreT, a tool designed for interpretability analysis. CLIP-InterpreT offers five types of analyses: property-based nearest neighbor search, per-head topic segmentation, contrastive segmentation, per-head nearest neighbors of an image, and per-head nearest neighbors of text.

Title: Semi-Supervised 3D Object Detection with Chanel Augmentation using Transformation Equivariance

• Authors: Minju Kang, Taehun Kong, Tae-Kyun Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06583
• Pdf URL: https://arxiv.org/pdf/2409.06583
• Copy Paste: [[2409.06583]] Semi-Supervised 3D Object Detection with Chanel Augmentation using Transformation Equivariance(https://arxiv.org/abs/2409.06583)
• Keywords: robust
• Abstract: Accurate 3D object detection is crucial for autonomous vehicles and robots to navigate and interact with the environment safely and effectively. Meanwhile, the performance of 3D detector relies on the data size and annotation which is expensive. Consequently, the demand of training with limited labeled data is growing. We explore a novel teacher-student framework employing channel augmentation for 3D semi-supervised object detection. The teacher-student SSL typically adopts a weak augmentation and strong augmentation to teacher and student, respectively. In this work, we apply multiple channel augmentations to both networks using the transformation equivariance detector (TED). The TED allows us to explore different combinations of augmentation on point clouds and efficiently aggregates multi-channel transformation equivariance features. In principle, by adopting fixed channel augmentations for the teacher network, the student can train stably on reliable pseudo-labels. Adopting strong channel augmentations can enrich the diversity of data, fostering robustness to transformations and enhancing generalization performance of the student network. We use SOTA hierarchical supervision as a baseline and adapt its dual-threshold to TED, which is called channel IoU consistency. We evaluate our method with KITTI dataset, and achieved a significant performance leap, surpassing SOTA 3D semi-supervised object detection models.

Title: Transtreaming: Adaptive Delay-aware Transformer for Real-time Streaming Perception

• Authors: Xiang Zhang, Yufei Cui, Chenchen Fu, Weiwei Wu, Zihao Wang, Yuyang Sun, Xue Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06584
• Pdf URL: https://arxiv.org/pdf/2409.06584
• Copy Paste: [[2409.06584]] Transtreaming: Adaptive Delay-aware Transformer for Real-time Streaming Perception(https://arxiv.org/abs/2409.06584)
• Keywords: robust, transformer
• Abstract: Real-time object detection is critical for the decision-making process for many real-world applications, such as collision avoidance and path planning in autonomous driving. This work presents an innovative real-time streaming perception method, Transtreaming, which addresses the challenge of real-time object detection with dynamic computational delay. The core innovation of Transtreaming lies in its adaptive delay-aware transformer, which can concurrently predict multiple future frames and select the output that best matches the real-world present time, compensating for any system-induced computation delays. The proposed model outperforms the existing state-of-the-art methods, even in single-frame detection scenarios, by leveraging a transformer-based methodology. It demonstrates robust performance across a range of devices, from powerful V100 to modest 2080Ti, achieving the highest level of perceptual accuracy on all platforms. Unlike most state-of-the-art methods that struggle to complete computation within a single frame on less powerful devices, Transtreaming meets the stringent real-time processing requirements on all kinds of devices. The experimental results emphasize the system's adaptability and its potential to significantly improve the safety and reliability for many real-world systems, such as autonomous driving.

Title: Seg-HGNN: Unsupervised and Light-Weight Image Segmentation with Hyperbolic Graph Neural Networks

• Authors: Debjyoti Mondal, Rahul Mishra, Chandan Pandey
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06589
• Pdf URL: https://arxiv.org/pdf/2409.06589
• Copy Paste: [[2409.06589]] Seg-HGNN: Unsupervised and Light-Weight Image Segmentation with Hyperbolic Graph Neural Networks(https://arxiv.org/abs/2409.06589)
• Keywords: segmentation
• Abstract: Image analysis in the euclidean space through linear hyperspaces is well studied. However, in the quest for more effective image representations, we turn to hyperbolic manifolds. They provide a compelling alternative to capture complex hierarchical relationships in images with remarkably small dimensionality. To demonstrate hyperbolic embeddings' competence, we introduce a light-weight hyperbolic graph neural network for image segmentation, encompassing patch-level features in a very small embedding size. Our solution, Seg-HGNN, surpasses the current best unsupervised method by 2.5\%, 4\% on VOC-07, VOC-12 for localization, and by 0.8\%, 1.3\% on CUB-200, ECSSD for segmentation, respectively. With less than 7.5k trainable parameters, Seg-HGNN delivers effective and fast ($\approx 2$ images/second) results on very standard GPUs like the GTX1650. This empirical evaluation presents compelling evidence of the efficacy and potential of hyperbolic representations for vision tasks.

Title: Lightweight Multiscale Feature Fusion Super-Resolution Network Based on Two-branch Convolution and Transformer

• Authors: Li Ke, Liu Yukai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06590
• Pdf URL: https://arxiv.org/pdf/2409.06590
• Copy Paste: [[2409.06590]] Lightweight Multiscale Feature Fusion Super-Resolution Network Based on Two-branch Convolution and Transformer(https://arxiv.org/abs/2409.06590)
• Keywords: transformer
• Abstract: The single image super-resolution(SISR) algorithms under deep learning currently have two main models, one based on convolutional neural networks and the other based on Transformer. The former uses the stacking of convolutional layers with different convolutional kernel sizes to design the model, which enables the model to better extract the local features of the image; the latter uses the self-attention mechanism to design the model, which allows the model to establish long-distance dependencies between image pixel points through the self-attention mechanism and then better extract the global features of the image. However, both of the above methods face their problems. Based on this, this paper proposes a new lightweight multi-scale feature fusion network model based on two-way complementary convolutional and Transformer, which integrates the respective features of Transformer and convolutional neural networks through a two-branch network architecture, to realize the mutual fusion of global and local information. Meanwhile, considering the partial loss of information caused by the low-pixel images trained by the deep neural network, this paper designs a modular connection method of multi-stage feature supplementation to fuse the feature maps extracted from the shallow stage of the model with those extracted from the deep stage of the model, to minimize the loss of the information in the feature images that is beneficial to the image restoration as much as possible, to facilitate the obtaining of a higher-quality restored image. The practical results finally show that the model proposed in this paper is optimal in image recovery performance when compared with other lightweight models with the same amount of parameters.

Title: GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering

• Authors: Sacha Muller, António Loison, Bilel Omrani, Gautier Viaud
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06595
• Pdf URL: https://arxiv.org/pdf/2409.06595
• Copy Paste: [[2409.06595]] GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering(https://arxiv.org/abs/2409.06595)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) has emerged as a common paradigm to use Large Language Models (LLMs) alongside private and up-to-date knowledge bases. In this work, we address the challenges of using LLM-as-a-Judge when evaluating grounded answers generated by RAG systems. To assess the calibration and discrimination capabilities of judge models, we identify 7 generator failure modes and introduce GroUSE (Grounded QA Unitary Scoring of Evaluators), a meta-evaluation benchmark of 144 unit tests. This benchmark reveals that existing automated RAG evaluation frameworks often overlook important failure modes, even when using GPT-4 as a judge. To improve on the current design of automated RAG evaluation frameworks, we propose a novel pipeline and find that while closed models perform well on GroUSE, state-of-the-art open-source judges do not generalize to our proposed criteria, despite strong correlation with GPT-4's judgement. Our findings suggest that correlation with GPT-4 is an incomplete proxy for the practical performance of judge models and should be supplemented with evaluations on unit tests for precise failure mode detection. We further show that finetuning Llama-3 on GPT-4's reasoning traces significantly boosts its evaluation capabilities, improving upon both correlation with GPT-4's evaluations and calibration on reference situations.

Title: Alleviating Hallucinations in Large Language Models with Scepticism Modeling

• Authors: Yetao Wu, Yihong Wang, Teng Chen, Chenxi Liu, Ningyuan Xi, Qingqing Gu, Hongyang Lei, Zhonglin Jiang, Yong Chen, Luo Ji
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06601
• Pdf URL: https://arxiv.org/pdf/2409.06601
• Copy Paste: [[2409.06601]] Alleviating Hallucinations in Large Language Models with Scepticism Modeling(https://arxiv.org/abs/2409.06601)
• Keywords: large language model
• Abstract: Hallucinations is a major challenge for large language models (LLMs), prevents adoption in diverse fields. Uncertainty estimation could be used for alleviating the damages of hallucinations. The skeptical emotion of human could be useful for enhancing the ability of self estimation. Inspirited by this observation, we proposed a new approach called Skepticism Modeling (SM). This approach is formalized by combining the information of token and logits for self estimation. We construct the doubt emotion aware data, perform continual pre-training, and then fine-tune the LLMs, improve their ability of self estimation. Experimental results demonstrate this new approach effectively enhances a model's ability to estimate their uncertainty, and validate its generalization ability of other tasks by out-of-domain experiments.

Title: A Practical Gated Recurrent Transformer Network Incorporating Multiple Fusions for Video Denoising

• Authors: Kai Guo, Seungwon Choi, Jongseong Choi, Lae-Hoon Kim
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.06603
• Pdf URL: https://arxiv.org/pdf/2409.06603
• Copy Paste: [[2409.06603]] A Practical Gated Recurrent Transformer Network Incorporating Multiple Fusions for Video Denoising(https://arxiv.org/abs/2409.06603)
• Keywords: robust, transformer
• Abstract: State-of-the-art (SOTA) video denoising methods employ multi-frame simultaneous denoising mechanisms, resulting in significant delays (e.g., 16 frames), making them impractical for real-time cameras. To overcome this limitation, we propose a multi-fusion gated recurrent Transformer network (GRTN) that achieves SOTA denoising performance with only a single-frame delay. Specifically, the spatial denoising module extracts features from the current frame, while the reset gate selects relevant information from the previous frame and fuses it with current frame features via the temporal denoising module. The update gate then further blends this result with the previous frame features, and the reconstruction module integrates it with the current frame. To robustly compute attention for noisy features, we propose a residual simplified Swin Transformer with Euclidean distance (RSSTE) in the spatial and temporal denoising modules. Comparative objective and subjective results show that our GRTN achieves denoising performance comparable to SOTA multi-frame delay networks, with only a single-frame delay.

Title: When to Extract ReID Features: A Selective Approach for Improved Multiple Object Tracking

• Authors: Emirhan Bayar, Cemal Aker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06617
• Pdf URL: https://arxiv.org/pdf/2409.06617
• Copy Paste: [[2409.06617]] When to Extract ReID Features: A Selective Approach for Improved Multiple Object Tracking(https://arxiv.org/abs/2409.06617)
• Keywords: extraction
• Abstract: Extracting and matching Re-Identification (ReID) features is used by many state-of-the-art (SOTA) Multiple Object Tracking (MOT) methods, particularly effective against frequent and long-term occlusions. While end-to-end object detection and tracking have been the main focus of recent research, they have yet to outperform traditional methods in benchmarks like MOT17 and MOT20. Thus, from an application standpoint, methods with separate detection and embedding remain the best option for accuracy, modularity, and ease of implementation, though they are impractical for edge devices due to the overhead involved. In this paper, we investigate a selective approach to minimize the overhead of feature extraction while preserving accuracy, modularity, and ease of implementation. This approach can be integrated into various SOTA methods. We demonstrate its effectiveness by applying it to StrongSORT and Deep OC-SORT. Experiments on MOT17, MOT20, and DanceTrack datasets show that our mechanism retains the advantages of feature extraction during occlusions while significantly reducing runtime. Additionally, it improves accuracy by preventing confusion in the feature-matching stage, particularly in cases of deformation and appearance similarity, which are common in DanceTrack. this https URL, this https URL

Title: A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Ratio

• Authors: Ningyuan Xi, Yetao Wu, Kun Fan, Teng Chen, Qingqing Gu, Peng Yu, Jinxian Qu, Chenxi Liu, Zhonglin Jiang, Yong Chen, Luo Ji
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06624
• Pdf URL: https://arxiv.org/pdf/2409.06624
• Copy Paste: [[2409.06624]] A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Ratio(https://arxiv.org/abs/2409.06624)
• Keywords: large language model
• Abstract: Large Language Models (LLM) often needs to be Continual Pre-Trained (CPT) to obtain the unfamiliar language skill or adapt into new domains. The huge training cost of CPT often asks for cautious choice of key hyper-parameters such as the mixture ratio of extra language or domain corpus. However, there is no systematic study which bridge the gap between the optimal mixture ratio and the actual model performance, and the gap between experimental scaling law and the actual deployment in the full model size. In this paper, we perform CPT on Llama-3 8B and 70B to enhance its Chinese ability. We study the optimal correlation between the Additional Language Mixture Ratio (ALMR) and the Learning Rate (LR) on the 8B size which directly indicate the optimal experimental set up. By thorough choice of hyper-parameter, and subsequent fine-tuning, the model capability is improved not only on the Chinese-related benchmark, but also some specific domains including math, coding and emotional intelligence. We deploy the final 70B version of LLM on an real-life chat system which obtain satisfying performance.

Title: Towards Localizing Structural Elements: Merging Geometrical Detection with Semantic Verification in RGB-D Data

• Authors: Ali Tourani, Saad Ejaz, Hriday Bavle, Jose Luis Sanchez-Lopez, Holger Voos
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.06625
• Pdf URL: https://arxiv.org/pdf/2409.06625
• Copy Paste: [[2409.06625]] Towards Localizing Structural Elements: Merging Geometrical Detection with Semantic Verification in RGB-D Data(https://arxiv.org/abs/2409.06625)
• Keywords: segmentation
• Abstract: RGB-D cameras supply rich and dense visual and spatial information for various robotics tasks such as scene understanding, map reconstruction, and localization. Integrating depth and visual information can aid robots in localization and element mapping, advancing applications like 3D scene graph generation and Visual Simultaneous Localization and Mapping (VSLAM). While point cloud data containing such information is primarily used for enhanced scene understanding, exploiting their potential to capture and represent rich semantic information has yet to be adequately targeted. This paper presents a real-time pipeline for localizing building components, including wall and ground surfaces, by integrating geometric calculations for pure 3D plane detection followed by validating their semantic category using point cloud data from RGB-D cameras. It has a parallel multi-thread architecture to precisely estimate poses and equations of all the planes detected in the environment, filters the ones forming the map structure using a panoptic segmentation validation, and keeps only the validated building components. Incorporating the proposed method into a VSLAM framework confirmed that constraining the map with the detected environment-driven semantic elements can improve scene understanding and map reconstruction accuracy. It can also ensure (re-)association of these detected components into a unified 3D scene graph, bridging the gap between geometric accuracy and semantic understanding. Additionally, the pipeline allows for the detection of potential higher-level structural entities, such as rooms, by identifying the relationships between building components based on their layout.

Title: SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation

• Authors: Teng Hu, Jiangning Zhang, Ran Yi, Hongrui Huang, Yabiao Wang, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06633
• Pdf URL: https://arxiv.org/pdf/2409.06633
• Copy Paste: [[2409.06633]] SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation(https://arxiv.org/abs/2409.06633)
• Keywords: diffusion, generative
• Abstract: In recent years, the development of diffusion models has led to significant progress in image and video generation tasks, with pre-trained models like the Stable Diffusion series playing a crucial role. Inspired by model pruning which lightens large pre-trained models by removing unimportant parameters, we propose a novel model fine-tuning method to make full use of these ineffective parameters and enable the pre-trained model with new task-specified capabilities. In this work, we first investigate the importance of parameters in pre-trained diffusion models, and discover that the smallest 10% to 20% of parameters by absolute values do not contribute to the generation process. Based on this observation, we propose a method termed SaRA that re-utilizes these temporarily ineffective parameters, equating to optimizing a sparse weight matrix to learn the task-specific knowledge. To mitigate overfitting, we propose a nuclear-norm-based low-rank sparse training scheme for efficient fine-tuning. Furthermore, we design a new progressive parameter adjustment strategy to make full use of the re-trained/finetuned parameters. Finally, we propose a novel unstructural backpropagation strategy, which significantly reduces memory costs during fine-tuning. Our method enhances the generative capabilities of pre-trained models in downstream applications and outperforms traditional fine-tuning methods like LoRA in maintaining model's generalization ability. We validate our approach through fine-tuning experiments on SD models, demonstrating significant improvements. SaRA also offers a practical advantage that requires only a single line of code modification for efficient implementation and is seamlessly compatible with existing methods.

Title: Data Collection-free Masked Video Modeling

• Authors: Yuchi Ishikawa, Masayoshi Kondo, Yoshimitsu Aoki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06665
• Pdf URL: https://arxiv.org/pdf/2409.06665
• Copy Paste: [[2409.06665]] Data Collection-free Masked Video Modeling(https://arxiv.org/abs/2409.06665)
• Keywords: privacy, transformer
• Abstract: Pre-training video transformers generally requires a large amount of data, presenting significant challenges in terms of data collection costs and concerns related to privacy, licensing, and inherent biases. Synthesizing data is one of the promising ways to solve these issues, yet pre-training solely on synthetic data has its own challenges. In this paper, we introduce an effective self-supervised learning framework for videos that leverages readily available and less costly static images. Specifically, we define the Pseudo Motion Generator (PMG) module that recursively applies image transformations to generate pseudo-motion videos from images. These pseudo-motion videos are then leveraged in masked video modeling. Our approach is applicable to synthetic images as well, thus entirely freeing video pre-training from data collection costs and other concerns in real data. Through experiments in action recognition tasks, we demonstrate that this framework allows effective learning of spatio-temporal features through pseudo-motion videos, significantly improving over existing methods which also use static images and partially outperforming those using both real and synthetic videos. These results uncover fragments of what video transformers learn through masked video modeling.

Title: LLaMA-Omni: Seamless Speech Interaction with Large Language Models

• Authors: Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, Yang Feng
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.06666
• Pdf URL: https://arxiv.org/pdf/2409.06666
• Copy Paste: [[2409.06666]] LLaMA-Omni: Seamless Speech Interaction with Large Language Models(https://arxiv.org/abs/2409.06666)
• Keywords: large language model
• Abstract: Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.

Title: DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models

• Authors: Maryam Akhavan Aghdam, Hongpeng Jin, Yanzhao Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06669
• Pdf URL: https://arxiv.org/pdf/2409.06669
• Copy Paste: [[2409.06669]] DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models(https://arxiv.org/abs/2409.06669)
• Keywords: transformer
• Abstract: Transformer-based Mixture-of-Experts (MoE) models have been driving several recent technological advancements in Natural Language Processing (NLP). These MoE models adopt a router mechanism to determine which experts to activate for routing input tokens. However, existing router mechanisms allocate a fixed number of experts to each token, which neglects the varying importance of different input tokens. In this study, we propose a novel dynamic router mechanism that Dynamically Allocates a variable number of experts for Mixture-of-Experts (DA-MoE) models based on an effective token importance measure. First, we show that the Transformer attention mechanism provides a natural and effective way of calculating token importance. Second, we propose a dynamic router mechanism that effectively decides the optimal number of experts (K) and allocates the top-K experts for each input token. Third, comprehensive experiments on several benchmark datasets demonstrate that our DA-MoE approach consistently outperforms the state-of-the-art Transformer based MoE model on the popular GLUE benchmark.

Title: A Semantic Segmentation Approach on Sweet Orange Leaf Diseases Detection Utilizing YOLO

• Authors: Sabit Ahamed Preanto (4IR Research Cell Daffodil International University, Dhaka, Bangladesh), Md. Taimur Ahad (4IR Research Cell Daffodil International University, Dhaka, Bangladesh), Yousuf Rayhan Emon (4IR Research Cell Daffodil International University, Dhaka, Bangladesh), Sumaya Mustofa (4IR Research Cell Daffodil International University, Dhaka, Bangladesh), Md Alamin (4IR Research Cell Daffodil International University, Dhaka, Bangladesh)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06671
• Pdf URL: https://arxiv.org/pdf/2409.06671
• Copy Paste: [[2409.06671]] A Semantic Segmentation Approach on Sweet Orange Leaf Diseases Detection Utilizing YOLO(https://arxiv.org/abs/2409.06671)
• Keywords: extraction, segmentation
• Abstract: This research introduces an advanced method for diagnosing diseases in sweet orange leaves by utilising advanced artificial intelligence models like YOLOv8 . Due to their significance as a vital agricultural product, sweet oranges encounter significant threats from a variety of diseases that harmfully affect both their yield and quality. Conventional methods for disease detection primarily depend on manual inspection which is ineffective and frequently leads to errors, resulting in delayed treatment and increased financial losses. In response to this challenge, the research utilized YOLOv8 , harnessing their proficiencies in detecting objects and analyzing images. YOLOv8 is recognized for its rapid and precise performance, while VIT is acknowledged for its detailed feature extraction abilities. Impressively, during both the training and validation stages, YOLOv8 exhibited a perfect accuracy of 80.4%, while VIT achieved an accuracy of 99.12%, showcasing their potential to transform disease detection in agriculture. The study comprehensively examined the practical challenges related to the implementation of AI technologies in agriculture, encompassing the computational demands and user accessibility, and offering viable solutions for broader usage. Moreover, it underscores the environmental considerations, particularly the potential for reduced pesticide usage, thereby promoting sustainable farming and environmental conservation. These findings provide encouraging insights into the application of AI in agriculture, suggesting a transition towards more effective, sustainable, and technologically advanced farming methods. This research not only highlights the efficacy of YOLOv8 within a specific agricultural domain but also lays the foundation for further studies that encompass a broader application in crop management and sustainable agricultural practices.

Title: E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

• Authors: Zihan Liao, Jun Wang, Hang Yu, Lingxiao Wei, Jianguo Li, Jun Wang, Wei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06679
• Pdf URL: https://arxiv.org/pdf/2409.06679
• Copy Paste: [[2409.06679]] E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning(https://arxiv.org/abs/2409.06679)
• Keywords: large language model
• Abstract: In the realm of Large Language Models (LLMs), the ability to process long contexts is increasingly crucial for tasks such as multi-round dialogues, code generation, and document summarization. This paper addresses the challenges of enhancing the long-context performance, reducing computational complexity, and leveraging pretrained models collectively termed the "impossible triangle." We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. The method involves splitting long contexts into chunks, compressing each into embedding vectors via a pretrained text encoder, and utilizing an adapter to align these representations with a decoder-only LLM. Two training objectives, focusing on reconstruction of the encoder output and long-context instruction fine-tuning, are employed to facilitate the understanding of soft prompts by the LLM. Experimental results demonstrate that E2LLM achieves superior performance in long-context scenarios while balancing efficiency, performance, and compatibility with pretrained models. Our framework thus represents a significant advancement in the field, contributing to effective long-text modeling.

Title: Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving

• Authors: Kairui Ding, Boyuan Chen, Yuchen Su, Huan-ang Gao, Bu Jin, Chonghao Sima, Wuqiang Zhang, Xiaohui Li, Paul Barsch, Hongyang Li, Hao Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06702
• Pdf URL: https://arxiv.org/pdf/2409.06702
• Copy Paste: [[2409.06702]] Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving(https://arxiv.org/abs/2409.06702)
• Keywords: interpretability
• Abstract: End-to-end architectures in autonomous driving (AD) face a significant challenge in interpretability, impeding human-AI trust. Human-friendly natural language has been explored for tasks such as driving explanation and 3D captioning. However, previous works primarily focused on the paradigm of declarative interpretability, where the natural language interpretations are not grounded in the intermediate outputs of AD systems, making the interpretations only declarative. In contrast, aligned interpretability establishes a connection between language and the intermediate outputs of AD systems. Here we introduce Hint-AD, an integrated AD-language system that generates language aligned with the holistic perception-prediction-planning outputs of the AD model. By incorporating the intermediate outputs and a holistic token mixer sub-network for effective feature adaptation, Hint-AD achieves desirable accuracy, achieving state-of-the-art results in driving language tasks including driving explanation, 3D dense captioning, and command prediction. To facilitate further study on driving explanation task on nuScenes, we also introduce a human-labeled dataset, Nu-X. Codes, dataset, and models will be publicly available.

Title: GeoCalib: Learning Single-image Calibration with Geometric Optimization

• Authors: Alexander Veicht, Paul-Edouard Sarlin, Philipp Lindenberger, Marc Pollefeys
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06704
• Pdf URL: https://arxiv.org/pdf/2409.06704
• Copy Paste: [[2409.06704]] GeoCalib: Learning Single-image Calibration with Geometric Optimization(https://arxiv.org/abs/2409.06704)
• Keywords: robust
• Abstract: From a single image, visual cues can help deduce intrinsic and extrinsic camera parameters like the focal length and the gravity direction. This single-image calibration can benefit various downstream applications like image editing and 3D mapping. Current approaches to this problem are based on either classical geometry with lines and vanishing points or on deep neural networks trained end-to-end. The learned approaches are more robust but struggle to generalize to new environments and are less accurate than their classical counterparts. We hypothesize that they lack the constraints that 3D geometry provides. In this work, we introduce GeoCalib, a deep neural network that leverages universal rules of 3D geometry through an optimization process. GeoCalib is trained end-to-end to estimate camera parameters and learns to find useful visual cues from the data. Experiments on various benchmarks show that GeoCalib is more robust and more accurate than existing classical and learned approaches. Its internal optimization estimates uncertainties, which help flag failure cases and benefit downstream applications like visual localization. The code and trained models are publicly available at this https URL.