2024-04-02

Title: Fingerprinting web servers through Transformer-encoded HTTP response headers

• Authors: Patrick Darwinkel
• Subjects: cs.CR, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.00056
• Pdf URL: https://arxiv.org/pdf/2404.00056
• Copy Paste: [[2404.00056]] Fingerprinting web servers through Transformer-encoded HTTP response headers(https://arxiv.org/abs/2404.00056)
• Keywords: transformer
• Abstract: We explored leveraging state-of-the-art deep learning, big data, and natural language processing to enhance the detection of vulnerable web server versions. Focusing on improving accuracy and specificity over rule-based systems, we conducted experiments by sending various ambiguous and non-standard HTTP requests to 4.77 million domains and capturing HTTP response status lines. We represented these status lines through training a BPE tokenizer and RoBERTa encoder for unsupervised masked language modeling. We then dimensionality reduced and concatenated encoded response lines to represent each domain's web server. A Random Forest and multilayer perceptron (MLP) classified these web servers, and achieved 0.94 and 0.96 macro F1-score, respectively, on detecting the five most popular origin web servers. The MLP achieved a weighted F1-score of 0.55 on classifying 347 major type and minor version pairs. Analysis indicates that our test cases are meaningful discriminants of web server types. Our approach demonstrates promise as a powerful and flexible alternative to rule-based systems.

Title: Modelling the Impact of Quantum Circuit Imperfections on Networks and Computer Applications

• Authors: Savo Glisic
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2404.00062
• Pdf URL: https://arxiv.org/pdf/2404.00062
• Copy Paste: [[2404.00062]] Modelling the Impact of Quantum Circuit Imperfections on Networks and Computer Applications(https://arxiv.org/abs/2404.00062)
• Keywords: fair
• Abstract: Post Quantum and Quantum Cryptography schemes are feasible quantum computer applications for 7G networks. These schemes could possibly replace existing schemes. These algorithms have been compromised by advances in quantum search algorithms run on quantum computers like Shor algorithm. Shor algorithm is a quantum algorithm for finding the prime factors of an integer which is the basis of existing algorithm. This has become an available quantum computer application putting the use of ESA algorithm at risk. Our recent paper provides a detailed survey of the work on post quantum and quantum cryptography algorithms with focus on their applicability in 7G networks. Since the paper focuses on the cryptography algorithms as a follow up, in this paper, we provide a new framework for quantum network optimization and survey in detail the work on enabling technologies (quantum hardware) for the practical implementation of these algorithms including the most important segments of quantum hardware in 7G. As always in engineering practice practical solutions are a compromise between the performance and complexity of the implementation. For this reason, as the main contribution, the paper presents a network and computer applications optimization framework that includes implementation imperfections. The tools should be useful in optimizing future generation practical computer system design. After that a comprehensive survey of the existing work on quantum hardware is presented pointing out the sources of these imperfections. This enables us to make a fair assessment of how much investment into quantum hardware improvements contributes to the performance enhancement of the overall system. In this way a decision can be made on proper partitioning between the investment in hardware and system level complexity.

Title: A Data-Driven Predictive Analysis on Cyber Security Threats with Key Risk Factors

• Authors: Fatama Tuz Johora (1), Md Shahedul Islam Khan (2), Esrath Kanon (1), Mohammad Abu Tareq Rony (3), Md Zubair (4), (5)Iqbal H. Sarker ((1) Department of Computer Science and Engineering, University of Chittagong, Chattogram, Bangladesh, (2) Department of School of Electronics and Information, Northwestern Polytechnical University, Xi'an, Shaanxi, China (3) Department of Statistics, Noakhali Science and Technology University, Noakhali, Bangladesh (4) Department of Computer Science and Engineering, Chittagong University of Engineering & Technology, Chattogram, Bangladesh (5) Centre for Securing Digital Futures, Edith Cowan University, Perth, WA, Australia)
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00068
• Pdf URL: https://arxiv.org/pdf/2404.00068
• Copy Paste: [[2404.00068]] A Data-Driven Predictive Analysis on Cyber Security Threats with Key Risk Factors(https://arxiv.org/abs/2404.00068)
• Keywords: security, attack
• Abstract: Cyber risk refers to the risk of defacing reputation, monetary losses, or disruption of an organization or individuals, and this situation usually occurs by the unconscious use of cyber systems. The cyber risk is unhurriedly increasing day by day and it is right now a global threat. Developing countries like Bangladesh face major cyber risk challenges. The growing cyber threat worldwide focuses on the need for effective modeling to predict and manage the associated risk. This paper exhibits a Machine Learning(ML) based model for predicting individuals who may be victims of cyber attacks by analyzing socioeconomic factors. We collected the dataset from victims and non-victims of cyberattacks based on socio-demographic features. The study involved the development of a questionnaire to gather data, which was then used to measure the significance of features. Through data augmentation, the dataset was expanded to encompass 3286 entries, setting the stage for our investigation and modeling. Among several ML models with 19, 20, 21, and 26 features, we proposed a novel Pertinent Features Random Forest (RF) model, which achieved maximum accuracy with 20 features (95.95\%) and also demonstrated the association among the selected features using the Apriori algorithm with Confidence (above 80\%) according to the victim. We generated 10 important association rules and presented the framework that is rigorously evaluated on real-world datasets, demonstrating its potential to predict cyberattacks and associated risk factors effectively. Looking ahead, future efforts will be directed toward refining the predictive model's precision and delving into additional risk factors, to fortify the proposed framework's efficacy in navigating the complex terrain of cybersecurity threats.

Title: A Two-Phase Recall-and-Select Framework for Fast Model Selection

• Authors: Jianwei Cui, Wenhang Shi, Honglin Tao, Wei Lu, Xiaoyong Du
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00069
• Pdf URL: https://arxiv.org/pdf/2404.00069
• Copy Paste: [[2404.00069]] A Two-Phase Recall-and-Select Framework for Fast Model Selection(https://arxiv.org/abs/2404.00069)
• Keywords: robust
• Abstract: As the ubiquity of deep learning in various machine learning applications has amplified, a proliferation of neural network models has been trained and shared on public model repositories. In the context of a targeted machine learning assignment, utilizing an apt source model as a starting point typically outperforms the strategy of training from scratch, particularly with limited training data. Despite the investigation and development of numerous model selection strategies in prior work, the process remains time-consuming, especially given the ever-increasing scale of model repositories. In this paper, we propose a two-phase (coarse-recall and fine-selection) model selection framework, aiming to enhance the efficiency of selecting a robust model by leveraging the models' training performances on benchmark datasets. Specifically, the coarse-recall phase clusters models showcasing similar training performances on benchmark datasets in an offline manner. A light-weight proxy score is subsequently computed between this model cluster and the target dataset, which serves to recall a significantly smaller subset of potential candidate models in a swift manner. In the following fine-selection phase, the final model is chosen by fine-tuning the recalled models on the target dataset with successive halving. To accelerate the process, the final fine-tuning performance of each potential model is predicted by mining the model's convergence trend on the benchmark datasets, which aids in filtering lower performance models more earlier during fine-tuning. Through extensive experimentation on tasks covering natural language processing and computer vision, it has been demonstrated that the proposed methodology facilitates the selection of a high-performing model at a rate about 3x times faster than conventional baseline methods. Our code is available at https://github.com/plasware/two-phase-selection.

Title: BEACON: Bayesian Experimental design Acceleration with Conditional Normalizing flows $-$ a case study in optimal monitor well placement for CO$_2$ sequestration

• Authors: Rafael Orozco, Abhinav Gahlot, Felix J. Herrmann
• Subjects: cs.LG, math-ph
• Abstract URL: https://arxiv.org/abs/2404.00075
• Pdf URL: https://arxiv.org/pdf/2404.00075
• Copy Paste: [[2404.00075]] BEACON: Bayesian Experimental design Acceleration with Conditional Normalizing flows $-$ a case study in optimal monitor well placement for CO$_2$ sequestration(https://arxiv.org/abs/2404.00075)
• Keywords: generative
• Abstract: CO$_2$ sequestration is a crucial engineering solution for mitigating climate change. However, the uncertain nature of reservoir properties, necessitates rigorous monitoring of CO$_2$ plumes to prevent risks such as leakage, induced seismicity, or breaching licensed boundaries. To address this, project managers use borehole wells for direct CO$_2$ and pressure monitoring at specific locations. Given the high costs associated with drilling, it is crucial to strategically place a limited number of wells to ensure maximally effective monitoring within budgetary constraints. Our approach for selecting well locations integrates fluid-flow solvers for forecasting plume trajectories with generative neural networks for plume inference uncertainty. Our methodology is extensible to three-dimensional domains and is developed within a Bayesian framework for optimal experimental design, ensuring scalability and mathematical optimality. We use a realistic case study to verify these claims by demonstrating our method's application in a large scale domain and optimal performance as compared to baseline well placement.

Title: A Backdoor Approach with Inverted Labels Using Dirty Label-Flipping Attacks

• Authors: Orson Mengara
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.00076
• Pdf URL: https://arxiv.org/pdf/2404.00076
• Copy Paste: [[2404.00076]] A Backdoor Approach with Inverted Labels Using Dirty Label-Flipping Attacks(https://arxiv.org/abs/2404.00076)
• Keywords: attack, steal
• Abstract: Audio-based machine learning systems frequently use public or third-party data, which might be inaccurate. This exposes deep neural network (DNN) models trained on such data to potential data poisoning attacks. In this type of assault, attackers can train the DNN model using poisoned data, potentially degrading its performance. Another type of data poisoning attack that is extremely relevant to our investigation is label flipping, in which the attacker manipulates the labels for a subset of data. It has been demonstrated that these assaults may drastically reduce system performance, even for attackers with minimal abilities. In this study, we propose a backdoor attack named 'DirtyFlipping', which uses dirty label techniques, "label-on-label", to input triggers (clapping) in the selected data patterns associated with the target class, thereby enabling a stealthy backdoor.

Title: DVIS-DAQ: Improving Video Segmentation via Dynamic Anchor Queries

• Authors: Yikang Zhou, Tao Zhang, Shunping JI, Shuicheng Yan, Xiangtai Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00086
• Pdf URL: https://arxiv.org/pdf/2404.00086
• Copy Paste: [[2404.00086]] DVIS-DAQ: Improving Video Segmentation via Dynamic Anchor Queries(https://arxiv.org/abs/2404.00086)
• Keywords: segmentation
• Abstract: Modern video segmentation methods adopt object queries to perform inter-frame association and demonstrate satisfactory performance in tracking continuously appearing objects despite large-scale motion and transient occlusion. However, they all underperform on newly emerging and disappearing objects that are common in the real world because they attempt to model object emergence and disappearance through feature transitions between background and foreground queries that have significant feature gaps. We introduce Dynamic Anchor Queries (DAQ) to shorten the transition gap between the anchor and target queries by dynamically generating anchor queries based on the features of potential candidates. Furthermore, we introduce a query-level object Emergence and Disappearance Simulation (EDS) strategy, which unleashes DAQ's potential without any additional cost. Finally, we combine our proposed DAQ and EDS with DVIS~\cite{zhang2023dvis} to obtain DVIS-DAQ. Extensive experiments demonstrate that DVIS-DAQ achieves a new state-of-the-art (SOTA) performance on five mainstream video segmentation benchmarks. Code and models are available at \url{https://github.com/SkyworkAI/DAQ-VS}.

Title: GDA: Generalized Diffusion for Robust Test-time Adaptation

• Authors: Yun-Yun Tsai, Fu-Chen Chen, Albert Y. C. Chen, Junfeng Yang, Che-Chun Su, Min Sun, Cheng-Hao Kuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00095
• Pdf URL: https://arxiv.org/pdf/2404.00095
• Copy Paste: [[2404.00095]] GDA: Generalized Diffusion for Robust Test-time Adaptation(https://arxiv.org/abs/2404.00095)
• Keywords: robust, diffusion
• Abstract: Machine learning models struggle with generalization when encountering out-of-distribution (OOD) samples with unexpected distribution shifts. For vision tasks, recent studies have shown that test-time adaptation employing diffusion models can achieve state-of-the-art accuracy improvements on OOD samples by generating new samples that align with the model's domain without the need to modify the model's weights. Unfortunately, those studies have primarily focused on pixel-level corruptions, thereby lacking the generalization to adapt to a broader range of OOD types. We introduce Generalized Diffusion Adaptation (GDA), a novel diffusion-based test-time adaptation method robust against diverse OOD types. Specifically, GDA iteratively guides the diffusion by applying a marginal entropy loss derived from the model, in conjunction with style and content preservation losses during the reverse sampling process. In other words, GDA considers the model's output behavior with the semantic information of the samples as a whole, which can reduce ambiguity in downstream tasks during the generation process. Evaluation across various popular model architectures and OOD benchmarks shows that GDA consistently outperforms prior work on diffusion-driven adaptation. Notably, it achieves the highest classification accuracy improvements, ranging from 4.4\% to 5.02\% on ImageNet-C and 2.5\% to 7.4\% on Rendition, Sketch, and Stylized benchmarks. This performance highlights GDA's generalization to a broader range of OOD benchmarks.

Title: Sparse Views, Near Light: A Practical Paradigm for Uncalibrated Point-light Photometric Stereo

• Authors: Mohammed Brahimi, Bjoern Haefner, Zhenzhang Ye, Bastian Goldluecke, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00098
• Pdf URL: https://arxiv.org/pdf/2404.00098
• Copy Paste: [[2404.00098]] Sparse Views, Near Light: A Practical Paradigm for Uncalibrated Point-light Photometric Stereo(https://arxiv.org/abs/2404.00098)
• Keywords: fair
• Abstract: Neural approaches have shown a significant progress on camera-based reconstruction. But they require either a fairly dense sampling of the viewing sphere, or pre-training on an existing dataset, thereby limiting their generalizability. In contrast, photometric stereo (PS) approaches have shown great potential for achieving high-quality reconstruction under sparse viewpoints. Yet, they are impractical because they typically require tedious laboratory conditions, are restricted to dark rooms, and often multi-staged, making them subject to accumulated errors. To address these shortcomings, we propose an end-to-end uncalibrated multi-view PS framework for reconstructing high-resolution shapes acquired from sparse viewpoints in a real-world environment. We relax the dark room assumption, and allow a combination of static ambient lighting and dynamic near LED lighting, thereby enabling easy data capture outside the lab. Experimental validation confirms that it outperforms existing baseline approaches in the regime of sparse viewpoints by a large margin. This allows to bring high-accuracy 3D reconstruction from the dark room to the real world, while maintaining a reasonable data capture complexity.

Title: Robust Ensemble Person Re-Identification via Orthogonal Fusion with Occlusion Handling

• Authors: Syeda Nyma Ferdous, Xin Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00107
• Pdf URL: https://arxiv.org/pdf/2404.00107
• Copy Paste: [[2404.00107]] Robust Ensemble Person Re-Identification via Orthogonal Fusion with Occlusion Handling(https://arxiv.org/abs/2404.00107)
• Keywords: robust, transformer
• Abstract: Occlusion remains one of the major challenges in person reidentification (ReID) as a result of the diversity of poses and the variation of appearances. Developing novel architectures to improve the robustness of occlusion-aware person Re-ID requires new insights, especially on low-resolution edge cameras. We propose a deep ensemble model that harnesses both CNN and Transformer architectures to generate robust feature representations. To achieve robust Re-ID without the need to manually label occluded regions, we propose to take an ensemble learning-based approach derived from the analogy between arbitrarily shaped occluded regions and robust feature representation. Using the orthogonality principle, our developed deep CNN model makes use of masked autoencoder (MAE) and global-local feature fusion for robust person identification. Furthermore, we present a part occlusion-aware transformer capable of learning feature space that is robust to occluded regions. Experimental results are reported on several Re-ID datasets to show the effectiveness of our developed ensemble model named orthogonal fusion with occlusion handling (OFOH). Compared to competing methods, the proposed OFOH approach has achieved competent rank-1 and mAP performance.

Title: Efficient Data-Free Model Stealing with Label Diversity

• Authors: Yiyong Liu, Rui Wen, Michael Backes, Yang Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.00108
• Pdf URL: https://arxiv.org/pdf/2404.00108
• Copy Paste: [[2404.00108]] Efficient Data-Free Model Stealing with Label Diversity(https://arxiv.org/abs/2404.00108)
• Keywords: attack, steal, data-free
• Abstract: Machine learning as a Service (MLaaS) allows users to query the machine learning model in an API manner, which provides an opportunity for users to enjoy the benefits brought by the high-performance model trained on valuable data. This interface boosts the proliferation of machine learning based applications, while on the other hand, it introduces the attack surface for model stealing attacks. Existing model stealing attacks have relaxed their attack assumptions to the data-free setting, while keeping the effectiveness. However, these methods are complex and consist of several components, which obscure the core on which the attack really depends. In this paper, we revisit the model stealing problem from a diversity perspective and demonstrate that keeping the generated data samples more diverse across all the classes is the critical point for improving the attack performance. Based on this conjecture, we provide a simplified attack framework. We empirically signify our conjecture by evaluating the effectiveness of our attack, and experimental results show that our approach is able to achieve comparable or even better performance compared with the state-of-the-art method. Furthermore, benefiting from the absence of redundant components, our method demonstrates its advantages in attack efficiency and query budget.

Title: Deepfake Sentry: Harnessing Ensemble Intelligence for Resilient Detection and Generalisation

• Authors: Liviu-Daniel Ştefan (1), Dan-Cristian Stanciu (1), Mihai Dogariu (1), Mihai Gabriel Constantin (1), Andrei Cosmin Jitaru (1), Bogdan Ionescu (1) ((1) University "Politehnica" of Bucharest, Romania)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00114
• Pdf URL: https://arxiv.org/pdf/2404.00114
• Copy Paste: [[2404.00114]] Deepfake Sentry: Harnessing Ensemble Intelligence for Resilient Detection and Generalisation(https://arxiv.org/abs/2404.00114)
• Keywords: attack, generative
• Abstract: Recent advancements in Generative Adversarial Networks (GANs) have enabled photorealistic image generation with high quality. However, the malicious use of such generated media has raised concerns regarding visual misinformation. Although deepfake detection research has demonstrated high accuracy, it is vulnerable to advances in generation techniques and adversarial iterations on detection countermeasures. To address this, we propose a proactive and sustainable deepfake training augmentation solution that introduces artificial fingerprints into models. We achieve this by employing an ensemble learning approach that incorporates a pool of autoencoders that mimic the effect of the artefacts introduced by the deepfake generator models. Experiments on three datasets reveal that our proposed ensemble autoencoder-based data augmentation learning approach offers improvements in terms of generalisation, resistance against basic data perturbations such as noise, blurring, sharpness enhancement, and affine transforms, resilience to commonly used lossy compression algorithms such as JPEG, and enhanced resistance against adversarial attacks.

Title: AgileFormer: Spatially Agile Transformer UNet for Medical Image Segmentation

• Authors: Peijie Qiu, Jin Yang, Sayantan Kumar, Soumyendu Sekhar Ghosh, Aristeidis Sotiras
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.00122
• Pdf URL: https://arxiv.org/pdf/2404.00122
• Copy Paste: [[2404.00122]] AgileFormer: Spatially Agile Transformer UNet for Medical Image Segmentation(https://arxiv.org/abs/2404.00122)
• Keywords: transformer, segmentation
• Abstract: In the past decades, deep neural networks, particularly convolutional neural networks, have achieved state-of-the-art performance in a variety of medical image segmentation tasks. Recently, the introduction of the vision transformer (ViT) has significantly altered the landscape of deep segmentation models. There has been a growing focus on ViTs, driven by their excellent performance and scalability. However, we argue that the current design of the vision transformer-based UNet (ViT-UNet) segmentation models may not effectively handle the heterogeneous appearance (e.g., varying shapes and sizes) of objects of interest in medical image segmentation tasks. To tackle this challenge, we present a structured approach to introduce spatially dynamic components to the ViT-UNet. This adaptation enables the model to effectively capture features of target objects with diverse appearances. This is achieved by three main components: \textbf{(i)} deformable patch embedding; \textbf{(ii)} spatially dynamic multi-head attention; \textbf{(iii)} deformable positional encoding. These components were integrated into a novel architecture, termed AgileFormer. AgileFormer is a spatially agile ViT-UNet designed for medical image segmentation. Experiments in three segmentation tasks using publicly available datasets demonstrated the effectiveness of the proposed method. The code is available at \href{https://github.com/sotiraslab/AgileFormer}{https://github.com/sotiraslab/AgileFormer}.

Title: Memristor-Based Lightweight Encryption

• Authors: Muhammad Ali Siddiqi, Jan Andrés Galvan Hernández, Anteneh Gebregiorgis, Rajendra Bishnoi, Christos Strydis, Said Hamdioui, Mottaqiallah Taouil
• Subjects: cs.CR, cs.AR, cs.ET
• Abstract URL: https://arxiv.org/abs/2404.00125
• Pdf URL: https://arxiv.org/pdf/2404.00125
• Copy Paste: [[2404.00125]] Memristor-Based Lightweight Encryption(https://arxiv.org/abs/2404.00125)
• Keywords: protect, attack
• Abstract: Next-generation personalized healthcare devices are undergoing extreme miniaturization in order to improve user acceptability. However, such developments make it difficult to incorporate cryptographic primitives using available target technologies since these algorithms are notorious for their energy consumption. Besides, strengthening these schemes against side-channel attacks further adds to the device overheads. Therefore, viable alternatives among emerging technologies are being sought. In this work, we investigate the possibility of using memristors for implementing lightweight encryption. We propose a 40-nm RRAM-based GIFT-cipher implementation using a 1T1R configuration with promising results; it exhibits roughly half the energy consumption of a CMOS-only implementation. More importantly, its non-volatile and reconfigurable substitution boxes offer an energy-efficient protection mechanism against side-channel attacks. The complete cipher takes 0.0034 mm$^2$ of area, and encrypting a 128-bit block consumes a mere 242 pJ.

Title: FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures

• Authors: Lisa Mais, Peter Hirsch, Claire Managan, Ramya Kandarpa, Josef Lorenz Rumberger, Annika Reinke, Lena Maier-Hein, Gudrun Ihrke, Dagmar Kainmueller
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00130
• Pdf URL: https://arxiv.org/pdf/2404.00130
• Copy Paste: [[2404.00130]] FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures(https://arxiv.org/abs/2404.00130)
• Keywords: segmentation
• Abstract: Instance segmentation of neurons in volumetric light microscopy images of nervous systems enables groundbreaking research in neuroscience by facilitating joint functional and morphological analyses of neural circuits at cellular resolution. Yet said multi-neuron light microscopy data exhibits extremely challenging properties for the task of instance segmentation: Individual neurons have long-ranging, thin filamentous and widely branching morphologies, multiple neurons are tightly inter-weaved, and partial volume effects, uneven illumination and noise inherent to light microscopy severely impede local disentangling as well as long-range tracing of individual neurons. These properties reflect a current key challenge in machine learning research, namely to effectively capture long-range dependencies in the data. While respective methodological research is buzzing, to date methods are typically benchmarked on synthetic datasets. To address this gap, we release the FlyLight Instance Segmentation Benchmark (FISBe) dataset, the first publicly available multi-neuron light microscopy dataset with pixel-wise annotations. In addition, we define a set of instance segmentation metrics for benchmarking that we designed to be meaningful with regard to downstream analyses. Lastly, we provide three baselines to kick off a competition that we envision to both advance the field of machine learning regarding methodology for capturing long-range data dependencies, and facilitate scientific discovery in basic neuroscience.

Title: Security Risks Concerns of Generative AI in the IoT

• Authors: Honghui Xu, Yingshu Li, Olusesi Balogun, Shaoen Wu, Yue Wang, Zhipeng Cai
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00139
• Pdf URL: https://arxiv.org/pdf/2404.00139
• Copy Paste: [[2404.00139]] Security Risks Concerns of Generative AI in the IoT(https://arxiv.org/abs/2404.00139)
• Keywords: security, privacy, robust, generative
• Abstract: In an era where the Internet of Things (IoT) intersects increasingly with generative Artificial Intelligence (AI), this article scrutinizes the emergent security risks inherent in this integration. We explore how generative AI drives innovation in IoT and we analyze the potential for data breaches when using generative AI and the misuse of generative AI technologies in IoT ecosystems. These risks not only threaten the privacy and efficiency of IoT systems but also pose broader implications for trust and safety in AI-driven environments. The discussion in this article extends to strategic approaches for mitigating these risks, including the development of robust security protocols, the multi-layered security approaches, and the adoption of AI technological solutions. Through a comprehensive analysis, this article aims to shed light on the critical balance between embracing AI advancements and ensuring stringent security in IoT, providing insights into the future direction of these intertwined technologies.

Title: Classifying Conspiratorial Narratives At Scale: False Alarms and Erroneous Connections

• Authors: Ahmad Diab, Rr. Nefriana, Yu-Ru Lin
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.00141
• Pdf URL: https://arxiv.org/pdf/2404.00141
• Copy Paste: [[2404.00141]] Classifying Conspiratorial Narratives At Scale: False Alarms and Erroneous Connections(https://arxiv.org/abs/2404.00141)
• Keywords: transformer, generative, large language model
• Abstract: Online discussions frequently involve conspiracy theories, which can contribute to the proliferation of belief in them. However, not all discussions surrounding conspiracy theories promote them, as some are intended to debunk them. Existing research has relied on simple proxies or focused on a constrained set of signals to identify conspiracy theories, which limits our understanding of conspiratorial discussions across different topics and online communities. This work establishes a general scheme for classifying discussions related to conspiracy theories based on authors' perspectives on the conspiracy belief, which can be expressed explicitly through narrative elements, such as the agent, action, or objective, or implicitly through references to known theories, such as chemtrails or the New World Order. We leverage human-labeled ground truth to train a BERT-based model for classifying online CTs, which we then compared to the Generative Pre-trained Transformer machine (GPT) for detecting online conspiratorial content. Despite GPT's known strengths in its expressiveness and contextual understanding, our study revealed significant flaws in its logical reasoning, while also demonstrating comparable strengths from our classifiers. We present the first large-scale classification study using posts from the most active conspiracy-related Reddit forums and find that only one-third of the posts are classified as positive. This research sheds light on the potential applications of large language models in tasks demanding nuanced contextual comprehension.

Title: Uncovering Bias in Large Vision-Language Models with Counterfactuals

• Authors: Phillip Howard, Anahita Bhiwandiwalla, Kathleen C. Fraser, Svetlana Kiritchenko
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00166
• Pdf URL: https://arxiv.org/pdf/2404.00166
• Copy Paste: [[2404.00166]] Uncovering Bias in Large Vision-Language Models with Counterfactuals(https://arxiv.org/abs/2404.00166)
• Keywords: large language model
• Abstract: With the advent of Large Language Models (LLMs) possessing increasingly impressive capabilities, a number of Large Vision-Language Models (LVLMs) have been proposed to augment LLMs with visual inputs. Such models condition generated text on both an input image and a text prompt, enabling a variety of use cases such as visual question answering and multimodal chat. While prior studies have examined the social biases contained in text generated by LLMs, this topic has been relatively unexplored in LVLMs. Examining social biases in LVLMs is particularly challenging due to the confounding contributions of bias induced by information contained across the text and visual modalities. To address this challenging problem, we conduct a large-scale study of text generated by different LVLMs under counterfactual changes to input images. Specifically, we present LVLMs with identical open-ended text prompts while conditioning on images from different counterfactual sets, where each set contains images which are largely identical in their depiction of a common subject (e.g., a doctor), but vary only in terms of intersectional social attributes (e.g., race and gender). We comprehensively evaluate the text produced by different LVLMs under this counterfactual generation setting and find that social attributes such as race, gender, and physical characteristics depicted in input images can significantly influence toxicity and the generation of competency-associated words.

Title: Multi-Region Transfer Learning for Segmentation of Crop Field Boundaries in Satellite Images with Limited Labels

• Authors: Hannah Kerner, Saketh Sundar, Mathan Satish
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00179
• Pdf URL: https://arxiv.org/pdf/2404.00179
• Copy Paste: [[2404.00179]] Multi-Region Transfer Learning for Segmentation of Crop Field Boundaries in Satellite Images with Limited Labels(https://arxiv.org/abs/2404.00179)
• Keywords: segmentation
• Abstract: The goal of field boundary delineation is to predict the polygonal boundaries and interiors of individual crop fields in overhead remotely sensed images (e.g., from satellites or drones). Automatic delineation of field boundaries is a necessary task for many real-world use cases in agriculture, such as estimating cultivated area in a region or predicting end-of-season yield in a field. Field boundary delineation can be framed as an instance segmentation problem, but presents unique research challenges compared to traditional computer vision datasets used for instance segmentation. The practical applicability of previous work is also limited by the assumption that a sufficiently-large labeled dataset is available where field boundary delineation models will be applied, which is not the reality for most regions (especially under-resourced regions such as Sub-Saharan Africa). We present an approach for segmentation of crop field boundaries in satellite images in regions lacking labeled data that uses multi-region transfer learning to adapt model weights for the target region. We show that our approach outperforms existing methods and that multi-region transfer learning substantially boosts performance for multiple model architectures. Our implementation and datasets are publicly available to enable use of the approach by end-users and serve as a benchmark for future work.

Title: On Inherent Adversarial Robustness of Active Vision Systems

• Authors: Amitangshu Mukherjee, Timur Ibrayev, Kaushik Roy
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00185
• Pdf URL: https://arxiv.org/pdf/2404.00185
• Copy Paste: [[2404.00185]] On Inherent Adversarial Robustness of Active Vision Systems(https://arxiv.org/abs/2404.00185)
• Keywords: attack, robust
• Abstract: Current Deep Neural Networks are vulnerable to adversarial examples, which alter their predictions by adding carefully crafted noise. Since human eyes are robust to such inputs, it is possible that the vulnerability stems from the standard way of processing inputs in one shot by processing every pixel with the same importance. In contrast, neuroscience suggests that the human vision system can differentiate salient features by (1) switching between multiple fixation points (saccades) and (2) processing the surrounding with a non-uniform external resolution (foveation). In this work, we advocate that the integration of such active vision mechanisms into current deep learning systems can offer robustness benefits. Specifically, we empirically demonstrate the inherent robustness of two active vision methods - GFNet and FALcon - under a black box threat model. By learning and inferencing based on downsampled glimpses obtained from multiple distinct fixation points within an input, we show that these active methods achieve (2-3) times greater robustness compared to a standard passive convolutional network under state-of-the-art adversarial attacks. More importantly, we provide illustrative and interpretable visualization analysis that demonstrates how performing inference from distinct fixation points makes active vision methods less vulnerable to malicious inputs.

Title: DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries

• Authors: Manit Mishra, Abderrahman Braham, Charles Marsom, Bryan Chung, Gavin Griffin, Dakshesh Sidnerlikar, Chatanya Sarin, Arjun Rajaram
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00188
• Pdf URL: https://arxiv.org/pdf/2404.00188
• Copy Paste: [[2404.00188]] DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries(https://arxiv.org/abs/2404.00188)
• Keywords: large language model
• Abstract: Conventional processes for analyzing datasets and extracting meaningful information are often time-consuming and laborious. Previous work has identified manual, repetitive coding and data collection as major obstacles that hinder data scientists from undertaking more nuanced labor and high-level projects. To combat this, we evaluated OpenAI's GPT-3.5 as a "Language Data Scientist" (LDS) that can extrapolate key findings, including correlations and basic information, from a given dataset. The model was tested on a diverse set of benchmark datasets to evaluate its performance across multiple standards, including data science code-generation based tasks involving libraries such as NumPy, Pandas, Scikit-Learn, and TensorFlow, and was broadly successful in correctly answering a given data science query related to the benchmark dataset. The LDS used various novel prompt engineering techniques to effectively answer a given question, including Chain-of-Thought reinforcement and SayCan prompt engineering. Our findings demonstrate great potential for leveraging Large Language Models for low-level, zero-shot data analysis.

Title: GPTA: Generative Prompt Tuning Assistant for Synergistic Downstream Neural Network Enhancement with LLMs

• Authors: Xiao Liu, Jiawei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00189
• Pdf URL: https://arxiv.org/pdf/2404.00189
• Copy Paste: [[2404.00189]] GPTA: Generative Prompt Tuning Assistant for Synergistic Downstream Neural Network Enhancement with LLMs(https://arxiv.org/abs/2404.00189)
• Keywords: security, generative, large language model
• Abstract: This study introduces GPTA, a Large Language Model assistance training framework, that enhances the training of downstream task models via prefix prompt. By minimizing data exposure to LLM, the framework addresses the security and legal challenges of applying LLM in downstream task model training. GPTA utilizes a new synergistic training approach, optimizing the downstream models with parameter gradients and LLMs with the novel ``dialogue gradient''. The framework not only demonstrates significant improvements in model performance across six NLP benchmark datasets, but also reduces overfitting in low-resource scenarios effectively. The detailed analyses further validate that our pioneer framework provides a cost-efficient and adaptive method for downstream task model training with LLM support.

Title: GuaranTEE: Towards Attestable and Private ML with CCA

• Authors: Sandra Siby, Sina Abdollahi, Mohammad Maheri, Marios Kogias, Hamed Haddadi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.00190
• Pdf URL: https://arxiv.org/pdf/2404.00190
• Copy Paste: [[2404.00190]] GuaranTEE: Towards Attestable and Private ML with CCA(https://arxiv.org/abs/2404.00190)
• Keywords: secure, privacy, protect
• Abstract: Machine-learning (ML) models are increasingly being deployed on edge devices to provide a variety of services. However, their deployment is accompanied by challenges in model privacy and auditability. Model providers want to ensure that (i) their proprietary models are not exposed to third parties; and (ii) be able to get attestations that their genuine models are operating on edge devices in accordance with the service agreement with the user. Existing measures to address these challenges have been hindered by issues such as high overheads and limited capability (processing/secure memory) on edge devices. In this work, we propose GuaranTEE, a framework to provide attestable private machine learning on the edge. GuaranTEE uses Confidential Computing Architecture (CCA), Arm's latest architectural extension that allows for the creation and deployment of dynamic Trusted Execution Environments (TEEs) within which models can be executed. We evaluate CCA's feasibility to deploy ML models by developing, evaluating, and openly releasing a prototype. We also suggest improvements to CCA to facilitate its use in protecting the entire ML deployment pipeline on edge devices.

Title: Optimal Blackjack Strategy Recommender: A Comprehensive Study on Computer Vision Integration for Enhanced Gameplay

• Authors: Krishnanshu Gupta, Devon Bolt, Ben Hinchliff
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00191
• Pdf URL: https://arxiv.org/pdf/2404.00191
• Copy Paste: [[2404.00191]] Optimal Blackjack Strategy Recommender: A Comprehensive Study on Computer Vision Integration for Enhanced Gameplay(https://arxiv.org/abs/2404.00191)
• Keywords: robust, extraction, segmentation
• Abstract: This research project investigates the application of several computer vision techniques for playing card detection and recognition in the context of the popular casino game, blackjack. The primary objective is to develop a robust system that is capable of detecting and accurately classifying playing cards in real-time, and displaying the optimal move recommendation based on the given image of the current game. The proposed methodology involves using K-Means for image segmentation, card reprojection and feature extraction, training of the KNN classifier using a labeled dataset, and integration of the detection system into a Blackjack Basic Strategy recommendation algorithm. Further, the study aims to observe the effectiveness of this approach in detecting various card designs under different lighting conditions and occlusions. Overall, the project examines the potential benefits of incorporating computer vision techniques, with a specific focus on card detection, into commonly played games aiming to enhance player decision-making and optimize strategic outcomes. The results obtained from our experimental evaluations with models developed under considerable time constraints, highlight the potential for practical implementation in real-world casino environments and across other similarly structured games.

Title: Combined Static Analysis and Machine Learning Prediction for Application Debloating

• Authors: Chris Porter, Sharjeel Khan, Kangqi Ni, Santosh Pande
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.00196
• Pdf URL: https://arxiv.org/pdf/2404.00196
• Copy Paste: [[2404.00196]] Combined Static Analysis and Machine Learning Prediction for Application Debloating(https://arxiv.org/abs/2404.00196)
• Keywords: attack
• Abstract: Software debloating can effectively thwart certain code reuse attacks by reducing attack surfaces to break gadget chains. Approaches based on static analysis enable a reduced set of functions reachable at a callsite for execution by leveraging static properties of the callgraph. This achieves low runtime overhead, but the function set is conservatively computed, negatively affecting reduction. In contrast, approaches based on machine learning (ML) have much better precision and can sharply reduce function sets, leading to significant improvement in attack surface. Nevertheless, mispredictions occur in ML-based approaches. These cause overheads, and worse, there is no clear way to distinguish between mispredictions and actual attacks. In this work, we contend that a software debloating approach that incorporates ML-based predictions at runtime is realistic in a whole application setting, and that it can achieve significant attack surface reductions beyond the state of the art. We develop a framework, Predictive Debloat with Static Guarantees (PDSG). PDSG is fully sound and works on application source code. At runtime it predicts the dynamic callee set emanating from a callsite, and to resolve mispredictions, it employs a lightweight audit based on static invariants of call chains. We deduce the invariants offline and assert that they hold at runtime when there is a misprediction. To the best of our knowledge, it achieves the highest gadget reductions among similar techniques on SPEC CPU 2017, reducing 82.5% of the total gadgets on average. It triggers misprediction checks on only 3.8% of the total predictions invoked at runtime, and it leverages Datalog to verify dynamic call sequences conform to the static call relations. It has an overhead of 8.9%, which makes the scheme attractive for practical deployments.

Title: Conceptual and Unbiased Reasoning in Language Models

• Authors: Ben Zhou, Hongming Zhang, Sihao Chen, Dian Yu, Hongwei Wang, Baolin Peng, Dan Roth, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00205
• Pdf URL: https://arxiv.org/pdf/2404.00205
• Copy Paste: [[2404.00205]] Conceptual and Unbiased Reasoning in Language Models(https://arxiv.org/abs/2404.00205)
• Keywords: robust, large language model
• Abstract: Conceptual reasoning, the ability to reason in abstract and high-level perspectives, is key to generalization in human cognition. However, limited study has been done on large language models' capability to perform conceptual reasoning. In this work, we bridge this gap and propose a novel conceptualization framework that forces models to perform conceptual reasoning on abstract questions and generate solutions in a verifiable symbolic space. Using this framework as an analytical tool, we show that existing large language models fall short on conceptual reasoning, dropping 9% to 28% on various benchmarks compared to direct inference methods. We then discuss how models can improve since high-level abstract reasoning is key to unbiased and generalizable decision-making. We propose two techniques to add trustworthy induction signals by generating familiar questions with similar underlying reasoning paths and asking models to perform self-refinement. Experiments show that our proposed techniques improve models' conceptual reasoning performance by 8% to 11%, achieving a more robust reasoning system that relies less on inductive biases.

Title: EventGround: Narrative Reasoning by Grounding to Eventuality-centric Knowledge Graphs

• Authors: Cheng Jiayang, Lin Qiu, Chunkit Chan, Xin Liu, Yangqiu Song, Zheng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00209
• Pdf URL: https://arxiv.org/pdf/2404.00209
• Copy Paste: [[2404.00209]] EventGround: Narrative Reasoning by Grounding to Eventuality-centric Knowledge Graphs(https://arxiv.org/abs/2404.00209)
• Keywords: extraction, interpretability, large language model
• Abstract: Narrative reasoning relies on the understanding of eventualities in story contexts, which requires a wealth of background world knowledge. To help machines leverage such knowledge, existing solutions can be categorized into two groups. Some focus on implicitly modeling eventuality knowledge by pretraining language models (LMs) with eventuality-aware objectives. However, this approach breaks down knowledge structures and lacks interpretability. Others explicitly collect world knowledge of eventualities into structured eventuality-centric knowledge graphs (KGs). However, existing research on leveraging these knowledge sources for free-texts is limited. In this work, we propose an initial comprehensive framework called EventGround, which aims to tackle the problem of grounding free-texts to eventuality-centric KGs for contextualized narrative reasoning. We identify two critical problems in this direction: the event representation and sparsity problems. We provide simple yet effective parsing and partial information extraction methods to tackle these problems. Experimental results demonstrate that our approach consistently outperforms baseline models when combined with graph neural network (GNN) or large language model (LLM) based graph reasoning models. Our framework, incorporating grounded knowledge, achieves state-of-the-art performance while providing interpretable evidence.

Title: Multi-Conditional Ranking with Large Language Models

• Authors: Pouya Pezeshkpour, Estevam Hruschka
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00211
• Pdf URL: https://arxiv.org/pdf/2404.00211
• Copy Paste: [[2404.00211]] Multi-Conditional Ranking with Large Language Models(https://arxiv.org/abs/2404.00211)
• Keywords: large language model
• Abstract: Utilizing large language models (LLMs) to rank a set of items has become a common approach in recommendation and retrieval systems. Typically, these systems focus on ordering a substantial number of documents in a monotonic order based on a given query. However, real-world scenarios often present a different challenge: ranking a comparatively smaller set of items, but according to a variety of diverse and occasionally conflicting conditions. In this paper, we define and explore the task of multi-conditional ranking by introducing MCRank, a benchmark tailored for assessing multi-conditional ranking across various item types and conditions. Our analysis of LLMs using MCRank indicates a significant decrease in performance as the number and complexity of items and conditions grow. To overcome this limitation, we propose a novel decomposed reasoning method, consisting of EXtracting and Sorting the conditions, and then Iterativly Ranking the items (EXSIR). Our extensive experiments show that this decomposed reasoning method enhances LLMs' performance significantly, achieving up to a 12% improvement over existing LLMs. We also provide a detailed analysis of LLMs performance across various condition categories, and examine the effectiveness of decomposition step. Furthermore, we compare our method with existing approaches such as Chain-of-Thought and an encoder-type ranking model, demonstrating the superiority of our approach and complexity of MCR task. We released our dataset and code.

Title: Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning

• Authors: Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, Tolga Aktas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00213
• Pdf URL: https://arxiv.org/pdf/2404.00213
• Copy Paste: [[2404.00213]] Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning(https://arxiv.org/abs/2404.00213)
• Keywords: large language model
• Abstract: In recent years, Large Language Models (LLMs) have shown remarkable performance in generating human-like text, proving to be a valuable asset across various applications. However, adapting these models to incorporate new, out-of-domain knowledge remains a challenge, particularly for facts and events that occur after the model's knowledge cutoff date. This paper investigates the effectiveness of Supervised Fine-Tuning (SFT) as a method for knowledge injection in LLMs, specifically focusing on the domain of recent sporting events. We compare different dataset generation strategies -- token-based and fact-based scaling -- to create training data that helps the model learn new information. Our experiments on GPT-4 demonstrate that while token-based scaling can lead to improvements in Q&A accuracy, it may not provide uniform coverage of new knowledge. Fact-based scaling, on the other hand, offers a more systematic approach to ensure even coverage across all facts. We present a novel dataset generation process that leads to more effective knowledge ingestion through SFT, and our results show considerable performance improvements in Q&A tasks related to out-of-domain knowledge. This study contributes to the understanding of domain adaptation for LLMs and highlights the potential of SFT in enhancing the factuality of LLM responses in specific knowledge domains.

Title: Is Factuality Decoding a Free Lunch for LLMs? Evaluation on Knowledge Editing Benchmark

• Authors: Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei, Xueqi Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00216
• Pdf URL: https://arxiv.org/pdf/2404.00216
• Copy Paste: [[2404.00216]] Is Factuality Decoding a Free Lunch for LLMs? Evaluation on Knowledge Editing Benchmark(https://arxiv.org/abs/2404.00216)
• Keywords: large language model
• Abstract: The rapid development of large language models (LLMs) enables them to convey factual knowledge in a more human-like fashion. Extensive efforts have been made to reduce factual hallucinations by modifying LLMs with factuality decoding. However, they also pose risks of hindering knowledge updates, as they make models overly confident in known facts. In this work, we first revisite the current factuality decoding methods and verified their effectiveness in enhancing factual accuracy. Subsequently, we conduct further evaluation of several strong factuality decoding methods on the knowledge editing benchmark. All these decoding methods significantly diminish the performance of llama2 models compared to their original decoding, with the largest decrease being a staggering 81.3\%. This further indicates that the current existing decoding methods still cannot perfectly address the factual hallucinations, as they overlook the importance of preserving the flexibility for knowledge editing. Therefore, our work suggests that research into factual alignment should simultaneously focus on the effectiveness of knowledge editing.

Title: Classification and Clustering of Sentence-Level Embeddings of Scientific Articles Generated by Contrastive Learning

• Authors: Gustavo Bartz Guedes, Ana Estela Antunes da Silva
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00224
• Pdf URL: https://arxiv.org/pdf/2404.00224
• Copy Paste: [[2404.00224]] Classification and Clustering of Sentence-Level Embeddings of Scientific Articles Generated by Contrastive Learning(https://arxiv.org/abs/2404.00224)
• Keywords: transformer
• Abstract: Scientific articles are long text documents organized into sections, each describing aspects of the research. Analyzing scientific production has become progressively challenging due to the increase in the number of available articles. Within this scenario, our approach consisted of fine-tuning transformer language models to generate sentence-level embeddings from scientific articles, considering the following labels: background, objective, methods, results, and conclusion. We trained our models on three datasets with contrastive learning. Two datasets are from the article's abstracts in the computer science and medical domains. Also, we introduce PMC-Sents-FULL, a novel dataset of sentences extracted from the full texts of medical articles. We compare the fine-tuned and baseline models in clustering and classification tasks to evaluate our approach. On average, clustering agreement measures values were five times higher. For the classification measures, in the best-case scenario, we had an average improvement in F1-micro of 30.73\%. Results show that fine-tuning sentence transformers with contrastive learning and using the generated embeddings in downstream tasks is a feasible approach to sentence classification in scientific articles. Our experiment codes are available on GitHub.

Title: Design as Desired: Utilizing Visual Question Answering for Multimodal Pre-training

• Authors: Tongkun Su, Jun Li, Xi Zhang, Haibo Jin, Hao Chen, Qiong Wang, Faqin Lv, Baoliang Zhao, Yin Hu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00226
• Pdf URL: https://arxiv.org/pdf/2404.00226
• Copy Paste: [[2404.00226]] Design as Desired: Utilizing Visual Question Answering for Multimodal Pre-training(https://arxiv.org/abs/2404.00226)
• Keywords: transformer, segmentation
• Abstract: Multimodal pre-training demonstrates its potential in the medical domain, which learns medical visual representations from paired medical reports. However, many pre-training tasks require extra annotations from clinicians, and most of them fail to explicitly guide the model to learn the desired features of different pathologies. To the best of our knowledge, we are the first to utilize Visual Question Answering (VQA) for multimodal pre-training to guide the framework focusing on targeted pathological features. In this work, we leverage descriptions in medical reports to design multi-granular question-answer pairs associated with different diseases, which assist the framework in pre-training without requiring extra annotations from experts. We also propose a novel pre-training framework with a quasi-textual feature transformer, a module designed to transform visual features into a quasi-textual space closer to the textual domain via a contrastive learning strategy. This narrows the vision-language gap and facilitates modality alignment. Our framework is applied to four downstream tasks: report generation, classification, segmentation, and detection across five datasets. Extensive experiments demonstrate the superiority of our framework compared to other state-of-the-art methods. Our code will be released upon acceptance.

Title: Latent Watermark: Inject and Detect Watermarks in Latent Diffusion Space

• Authors: Zheling Meng, Bo Peng, Jing Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00230
• Pdf URL: https://arxiv.org/pdf/2404.00230
• Copy Paste: [[2404.00230]] Latent Watermark: Inject and Detect Watermarks in Latent Diffusion Space(https://arxiv.org/abs/2404.00230)
• Keywords: attack, robust, steal, watermark, diffusion
• Abstract: Watermarking is a tool for actively identifying and attributing the images generated by latent diffusion models. Existing methods face the dilemma of watermark robustness and image quality. The reason for this dilemma is that watermark detection is performed in pixel space, implying an intrinsic link between image quality and watermark robustness. In this paper, we highlight that an effective solution to the problem is to both inject and detect watermarks in latent space, and propose Latent Watermark (LW) with a progressive training strategy. Experiments show that compared to the recently proposed methods such as StegaStamp, StableSignature, RoSteALS and TreeRing, LW not only surpasses them in terms of robustness but also offers superior image quality. When we inject 64-bit messages, LW can achieve an identification performance close to 100% and an attribution performance above 97% under 9 single-attack scenarios and one all-attack scenario. Our code will be available on GitHub.

Title: Attention-based Shape-Deformation Networks for Artifact-Free Geometry Reconstruction of Lumbar Spine from MR Images

• Authors: Linchen Qian, Jiasong Chen, Linhai Ma, Timur Urakov, Weiyong Gu, Liang Liang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00231
• Pdf URL: https://arxiv.org/pdf/2404.00231
• Copy Paste: [[2404.00231]] Attention-based Shape-Deformation Networks for Artifact-Free Geometry Reconstruction of Lumbar Spine from MR Images(https://arxiv.org/abs/2404.00231)
• Keywords: robust, segmentation
• Abstract: Lumbar disc degeneration, a progressive structural wear and tear of lumbar intervertebral disc, is regarded as an essential role on low back pain, a significant global health concern. Automated lumbar spine geometry reconstruction from MR images will enable fast measurement of medical parameters to evaluate the lumbar status, in order to determine a suitable treatment. Existing image segmentation-based techniques often generate erroneous segments or unstructured point clouds, unsuitable for medical parameter measurement. In this work, we present TransDeformer: a novel attention-based deep learning approach that reconstructs the contours of the lumbar spine with high spatial accuracy and mesh correspondence across patients, and we also present a variant of TransDeformer for error estimation. Specially, we devise new attention modules with a new attention formula, which integrates image features and tokenized contour features to predict the displacements of the points on a shape template without the need for image segmentation. The deformed template reveals the lumbar spine geometry in the input image. We develop a multi-stage training strategy to enhance model robustness with respect to template initialization. Experiment results show that our TransDeformer generates artifact-free geometry outputs, and its variant predicts the error of a reconstructed geometry. Our code is available at https://github.com/linchenq/TransDeformer-Mesh.

Title: Grid Diffusion Models for Text-to-Video Generation

• Authors: Taegyeong Lee, Soyeong Kwon, Taehwan Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00234
• Pdf URL: https://arxiv.org/pdf/2404.00234
• Copy Paste: [[2404.00234]] Grid Diffusion Models for Text-to-Video Generation(https://arxiv.org/abs/2404.00234)
• Keywords: diffusion
• Abstract: Recent advances in the diffusion models have significantly improved text-to-image generation. However, generating videos from text is a more challenging task than generating images from text, due to the much larger dataset and higher computational cost required. Most existing video generation methods use either a 3D U-Net architecture that considers the temporal dimension or autoregressive generation. These methods require large datasets and are limited in terms of computational costs compared to text-to-image generation. To tackle these challenges, we propose a simple but effective novel grid diffusion for text-to-video generation without temporal dimension in architecture and a large text-video paired dataset. We can generate a high-quality video using a fixed amount of GPU memory regardless of the number of frames by representing the video as a grid image. Additionally, since our method reduces the dimensions of the video to the dimensions of the image, various image-based methods can be applied to videos, such as text-guided video manipulation from image manipulation. Our proposed method outperforms the existing methods in both quantitative and qualitative evaluations, demonstrating the suitability of our model for real-world video generation.

Title: Information Security and Privacy in the Digital World: Some Selected Topics

• Authors: Jaydip Sen, Joceli Mayer, Subhasis Dasgupta, Subrata Nandi, Srinivasan Krishnaswamy, Pinaki Mitra, Mahendra Pratap Singh, Naga Prasanthi Kundeti, Chandra Sekhara Rao MVP, Sudha Sree Chekuri, Seshu Babu Pallapothu, Preethi Nanjundan, Jossy P. George, Abdelhadi El Allahi, Ilham Morino, Salma AIT Oussous, Siham Beloualid, Ahmed Tamtaoui, Abderrahim Bajit
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00235
• Pdf URL: https://arxiv.org/pdf/2404.00235
• Copy Paste: [[2404.00235]] Information Security and Privacy in the Digital World: Some Selected Topics(https://arxiv.org/abs/2404.00235)
• Keywords: security, privacy, protect, robust, generative
• Abstract: In the era of generative artificial intelligence and the Internet of Things, while there is explosive growth in the volume of data and the associated need for processing, analysis, and storage, several new challenges are faced in identifying spurious and fake information and protecting the privacy of sensitive data. This has led to an increasing demand for more robust and resilient schemes for authentication, integrity protection, encryption, non-repudiation, and privacy-preservation of data. The chapters in this book present some of the state-of-the-art research works in the field of cryptography and security in computing and communications.

Title: DeFT: Flash Tree-attention with IO-Awareness for Efficient Tree-search-based LLM Inference

• Authors: Jinwei Yao, Kaiqi Chen, Kexun Zhang, Jiaxuan You, Binhang Yuan, Zeke Wang, Tao Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00242
• Pdf URL: https://arxiv.org/pdf/2404.00242
• Copy Paste: [[2404.00242]] DeFT: Flash Tree-attention with IO-Awareness for Efficient Tree-search-based LLM Inference(https://arxiv.org/abs/2404.00242)
• Keywords: transformer, large language model
• Abstract: Decoding using tree search can greatly enhance the inference quality for transformer-based Large Language Models (LLMs). Depending on the guidance signal, it searches for the best path from root to leaf in the tree by forming LLM outputs to improve controllability, reasoning ability, alignment, et cetera. However, current tree decoding strategies and their inference systems do not suit each other well due to redundancy in computation, memory footprints, and memory access, resulting in inefficient inference. To address this issue, we propose DeFT, an IO-aware tree attention algorithm that maintains memory-efficient attention calculation with low memory footprints in two stages: (1) QKV Preparation: we propose a KV-Guided Tree Split strategy to group QKV wisely for high utilization of GPUs and reduction of memory reads/writes for the KV cache between GPU global memory and on-chip shared memory as much as possible; (2) Attention Calculation: we calculate partial attention of each QKV groups in a fused kernel then apply a Tree-topology-aware Global Reduction strategy to get final attention. Thanks to a reduction in KV cache IO by 3.6-4.5$\times$, along with an additional reduction in IO for $\mathbf{Q} \mathbf{K}^\top$ and Softmax equivalent to 25% of the total KV cache IO, DeFT can achieve a speedup of 1.7-2.4$\times$ in end-to-end latency across two practical reasoning tasks over the SOTA attention algorithms.

Title: Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World

• Authors: Guande Wu, Chen Zhao, Claudio Silva, He He
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.00246
• Pdf URL: https://arxiv.org/pdf/2404.00246
• Copy Paste: [[2404.00246]] Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World(https://arxiv.org/abs/2404.00246)
• Keywords: large language model
• Abstract: Language agents that interact with the world on their own have great potential for automating digital tasks. While large language model (LLM) agents have made progress in understanding and executing tasks such as textual games and webpage control, many real-world tasks also require collaboration with humans or other LLMs in equal roles, which involves intent understanding, task coordination, and communication. To test LLM's ability to collaborate, we design a blocks-world environment, where two agents, each having unique goals and skills, build a target structure together. To complete the goals, they can act in the world and communicate in natural language. Under this environment, we design increasingly challenging settings to evaluate different collaboration perspectives, from independent to more complex, dependent tasks. We further adopt chain-of-thought prompts that include intermediate reasoning steps to model the partner's state and identify and correct execution errors. Both human-machine and machine-machine experiments show that LLM agents have strong grounding capacities, and our approach significantly improves the evaluation metric.

Title: Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation

• Authors: Yuan Wang, Rui Sun, Naisong Luo, Yuwen Pan, Tianzhu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00262
• Pdf URL: https://arxiv.org/pdf/2404.00262
• Copy Paste: [[2404.00262]] Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation(https://arxiv.org/abs/2404.00262)
• Keywords: robust, segmentation
• Abstract: Open-vocabulary semantic segmentation (OVS) aims to segment images of arbitrary categories specified by class labels or captions. However, most previous best-performing methods, whether pixel grouping methods or region recognition methods, suffer from false matches between image features and category labels. We attribute this to the natural gap between the textual features and visual features. In this work, we rethink how to mitigate false matches from the perspective of image-to-image matching and propose a novel relation-aware intra-modal matching (RIM) framework for OVS based on visual foundation models. RIM achieves robust region classification by firstly constructing diverse image-modal reference features and then matching them with region features based on relation-aware ranking distribution. The proposed RIM enjoys several merits. First, the intra-modal reference features are better aligned, circumventing potential ambiguities that may arise in cross-modal matching. Second, the ranking-based matching process harnesses the structure information implicit in the inter-class relationships, making it more robust than comparing individually. Extensive experiments on three benchmarks demonstrate that RIM outperforms previous state-of-the-art methods by large margins, obtaining a lead of more than 10% in mIoU on PASCAL VOC benchmark.

Title: DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation

• Authors: Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi, Manabu Okumura
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00264
• Pdf URL: https://arxiv.org/pdf/2404.00264
• Copy Paste: [[2404.00264]] DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation(https://arxiv.org/abs/2404.00264)
• Keywords: large language model
• Abstract: Dataset distillation aims to compress a training dataset by creating a small number of informative synthetic samples such that neural networks trained on them perform as well as those trained on the original training dataset. Current text dataset distillation methods create each synthetic sample as a sequence of word embeddings instead of a text to apply gradient-based optimization; however, such embedding-level distilled datasets cannot be used for training other models whose word embedding weights are different from the model used for distillation. To address this issue, we propose a novel text dataset distillation approach, called Distilling dataset into Language Model (DiLM), which trains a language model to generate informative synthetic training samples as text data, instead of directly optimizing synthetic samples. We evaluated DiLM on various text classification datasets and showed that distilled synthetic datasets from DiLM outperform those from current coreset selection methods. DiLM achieved remarkable generalization performance in training different types of models and in-context learning of large language models. Our code will be available at https://github.com/arumaekawa/DiLM.

Title: Secret Keepers: The Impact of LLMs on Linguistic Markers of Personal Traits

• Authors: Zhivar Sourati, Meltem Ozcan, Colin McDaniel, Alireza Ziabari, Nuan Wen, Ala Tak, Fred Morstatter, Morteza Dehghani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00267
• Pdf URL: https://arxiv.org/pdf/2404.00267
• Copy Paste: [[2404.00267]] Secret Keepers: The Impact of LLMs on Linguistic Markers of Personal Traits(https://arxiv.org/abs/2404.00267)
• Keywords: large language model
• Abstract: Prior research has established associations between individuals' language usage and their personal traits; our linguistic patterns reveal information about our personalities, emotional states, and beliefs. However, with the increasing adoption of Large Language Models (LLMs) as writing assistants in everyday writing, a critical question emerges: are authors' linguistic patterns still predictive of their personal traits when LLMs are involved in the writing process? We investigate the impact of LLMs on the linguistic markers of demographic and psychological traits, specifically examining three LLMs - GPT3.5, Llama 2, and Gemini - across six different traits: gender, age, political affiliation, personality, empathy, and morality. Our findings indicate that although the use of LLMs slightly reduces the predictive power of linguistic patterns over authors' personal traits, the significant changes are infrequent, and the use of LLMs does not fully diminish the predictive power of authors' linguistic patterns over their personal traits. We also note that some theoretically established lexical-based linguistic markers lose their reliability as predictors when LLMs are used in the writing process. Our findings have important implications for the study of linguistic markers of personal traits in the age of LLMs.

Title: IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Images

• Authors: Yushuang Wu, Luyue Shi, Junhao Cai, Weihao Yuan, Lingteng Qiu, Zilong Dong, Liefeng Bo, Shuguang Cui, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00269
• Pdf URL: https://arxiv.org/pdf/2404.00269
• Copy Paste: [[2404.00269]] IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Images(https://arxiv.org/abs/2404.00269)
• Keywords: diffusion, transformer
• Abstract: Generalizable 3D object reconstruction from single-view RGB-D images remains a challenging task, particularly with real-world data. Current state-of-the-art methods develop Transformer-based implicit field learning, necessitating an intensive learning paradigm that requires dense query-supervision uniformly sampled throughout the entire space. We propose a novel approach, IPoD, which harmonizes implicit field learning with point diffusion. This approach treats the query points for implicit field learning as a noisy point cloud for iterative denoising, allowing for their dynamic adaptation to the target object shape. Such adaptive query points harness diffusion learning's capability for coarse shape recovery and also enhances the implicit representation's ability to delineate finer details. Besides, an additional self-conditioning mechanism is designed to use implicit predictions as the guidance of diffusion learning, leading to a cooperative system. Experiments conducted on the CO3D-v2 dataset affirm the superiority of IPoD, achieving 7.8% improvement in F-score and 28.6% in Chamfer distance over existing methods. The generalizability of IPoD is also demonstrated on the MVImgNet dataset. Our project page is at https://yushuang-wu.github.io/IPoD.

Title: TG-NAS: Leveraging Zero-Cost Proxies with Transformer and Graph Convolution Networks for Efficient Neural Architecture Search

• Authors: Ye Qiao, Haocheng Xu, Sitao Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00271
• Pdf URL: https://arxiv.org/pdf/2404.00271
• Copy Paste: [[2404.00271]] TG-NAS: Leveraging Zero-Cost Proxies with Transformer and Graph Convolution Networks for Efficient Neural Architecture Search(https://arxiv.org/abs/2404.00271)
• Keywords: transformer
• Abstract: Neural architecture search (NAS) is an effective method for discovering new convolutional neural network (CNN) architectures. However, existing approaches often require time-consuming training or intensive sampling and evaluations. Zero-shot NAS aims to create training-free proxies for architecture performance prediction. However, existing proxies have suboptimal performance, and are often outperformed by simple metrics such as model parameter counts or the number of floating-point operations. Besides, existing model-based proxies cannot be generalized to new search spaces with unseen new types of operators without golden accuracy truth. A universally optimal proxy remains elusive. We introduce TG-NAS, a novel model-based universal proxy that leverages a transformer-based operator embedding generator and a graph convolution network (GCN) to predict architecture performance. This approach guides neural architecture search across any given search space without the need of retraining. Distinct from other model-based predictor subroutines, TG-NAS itself acts as a zero-cost (ZC) proxy, guiding architecture search with advantages in terms of data independence, cost-effectiveness, and consistency across diverse search spaces. Our experiments showcase its advantages over existing proxies across various NAS benchmarks, suggesting its potential as a foundational element for efficient architecture search. TG-NAS achieves up to 300X improvements in search efficiency compared to previous SOTA ZC proxy methods. Notably, it discovers competitive models with 93.75% CIFAR-10 accuracy on the NAS-Bench-201 space and 74.5% ImageNet top-1 accuracy on the DARTS space.

Title: HSIMamba: Hyperpsectral Imaging Efficient Feature Learning with Bidirectional State Space for Classification

• Authors: Judy X Yang, Jun Zhou, Jing Wang, Hui Tian, Alan Wee Chung Liew
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00272
• Pdf URL: https://arxiv.org/pdf/2404.00272
• Copy Paste: [[2404.00272]] HSIMamba: Hyperpsectral Imaging Efficient Feature Learning with Bidirectional State Space for Classification(https://arxiv.org/abs/2404.00272)
• Keywords: extraction, transformer
• Abstract: Classifying hyperspectral images is a difficult task in remote sensing, due to their complex high-dimensional data. To address this challenge, we propose HSIMamba, a novel framework that uses bidirectional reversed convolutional neural network pathways to extract spectral features more efficiently. Additionally, it incorporates a specialized block for spatial analysis. Our approach combines the operational efficiency of CNNs with the dynamic feature extraction capability of attention mechanisms found in Transformers. However, it avoids the associated high computational demands. HSIMamba is designed to process data bidirectionally, significantly enhancing the extraction of spectral features and integrating them with spatial information for comprehensive analysis. This approach improves classification accuracy beyond current benchmarks and addresses computational inefficiencies encountered with advanced models like Transformers. HSIMamba were tested against three widely recognized datasets Houston 2013, Indian Pines, and Pavia University and demonstrated exceptional performance, surpassing existing state-of-the-art models in HSI classification. This method highlights the methodological innovation of HSIMamba and its practical implications, which are particularly valuable in contexts where computational resources are limited. HSIMamba redefines the standards of efficiency and accuracy in HSI classification, thereby enhancing the capabilities of remote sensing applications. Hyperspectral imaging has become a crucial tool for environmental surveillance, agriculture, and other critical areas that require detailed analysis of the Earth surface. Please see our code in HSIMamba for more details.

Title: Look-Around Before You Leap: High-Frequency Injected Transformer for Image Restoration

• Authors: Shihao Zhou, Duosheng Chen, Jinshan Pan, Jufeng Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00279
• Pdf URL: https://arxiv.org/pdf/2404.00279
• Copy Paste: [[2404.00279]] Look-Around Before You Leap: High-Frequency Injected Transformer for Image Restoration(https://arxiv.org/abs/2404.00279)
• Keywords: transformer
• Abstract: Transformer-based approaches have achieved superior performance in image restoration, since they can model long-term dependencies well. However, the limitation in capturing local information restricts their capacity to remove degradations. While existing approaches attempt to mitigate this issue by incorporating convolutional operations, the core component in Transformer, i.e., self-attention, which serves as a low-pass filter, could unintentionally dilute or even eliminate the acquired local patterns. In this paper, we propose HIT, a simple yet effective High-frequency Injected Transformer for image restoration. Specifically, we design a window-wise injection module (WIM), which incorporates abundant high-frequency details into the feature map, to provide reliable references for restoring high-quality images. We also develop a bidirectional interaction module (BIM) to aggregate features at different scales using a mutually reinforced paradigm, resulting in spatially and contextually improved representations. In addition, we introduce a spatial enhancement unit (SEU) to preserve essential spatial relationships that may be lost due to the computations carried out across channel dimensions in the BIM. Extensive experiments on 9 tasks (real noise, real rain streak, raindrop, motion blur, moir\'e, shadow, snow, haze, and low-light condition) demonstrate that HIT with linear computational complexity performs favorably against the state-of-the-art methods. The source code and pre-trained models will be available at https://github.com/joshyZhou/HIT.

Title: Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods

• Authors: Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Guolong Liu, Gaoqi Liang, Junhua Zhao, Yun Li
• Subjects: cs.LG, cs.AI, cs.CL, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.00282
• Pdf URL: https://arxiv.org/pdf/2404.00282
• Copy Paste: [[2404.00282]] Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods(https://arxiv.org/abs/2404.00282)
• Keywords: large language model
• Abstract: With extensive pre-trained knowledge and high-level general capabilities, large language models (LLMs) emerge as a promising avenue to augment reinforcement learning (RL) in aspects such as multi-task learning, sample efficiency, and task planning. In this survey, we provide a comprehensive review of the existing literature in $\textit{LLM-enhanced RL}$ and summarize its characteristics compared to conventional RL methods, aiming to clarify the research scope and directions for future studies. Utilizing the classical agent-environment interaction paradigm, we propose a structured taxonomy to systematically categorize LLMs' functionalities in RL, including four roles: information processor, reward designer, decision-maker, and generator. Additionally, for each role, we summarize the methodologies, analyze the specific RL challenges that are mitigated, and provide insights into future directions. Lastly, potential applications, prospective opportunities and challenges of the $\textit{LLM-enhanced RL}$ are discussed.

Title: Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration

• Authors: Shihao Zhou, Jinshan Pan, Jinglei Shi, Duosheng Chen, Lishen Qu, Jufeng Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00288
• Pdf URL: https://arxiv.org/pdf/2404.00288
• Copy Paste: [[2404.00288]] Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration(https://arxiv.org/abs/2404.00288)
• Keywords: transformer
• Abstract: How to explore useful features from images as prompts to guide the deep image restoration models is an effective way to solve image restoration. In contrast to mining spatial relations within images as prompt, which leads to characteristics of different frequencies being neglected and further remaining subtle or undetectable artifacts in the restored image, we develop a Frequency Prompting image restoration method, dubbed FPro, which can effectively provide prompt components from a frequency perspective to guild the restoration model address these differences. Specifically, we first decompose input features into separate frequency parts via dynamically learned filters, where we introduce a gating mechanism for suppressing the less informative elements within the kernels. To propagate useful frequency information as prompt, we then propose a dual prompt block, consisting of a low-frequency prompt modulator (LPM) and a high-frequency prompt modulator (HPM), to handle signals from different bands respectively. Each modulator contains a generation process to incorporate prompting components into the extracted frequency maps, and a modulation part that modifies the prompt feature with the guidance of the decoder features. Experimental results on commonly used benchmarks have demonstrated the favorable performance of our pipeline against SOTA methods on 5 image restoration tasks, including deraining, deraindrop, demoir\'eing, deblurring, and dehazing. The source code and pre-trained models will be available at https://github.com/joshyZhou/FPro.

Title: LAKE-RED: Camouflaged Images Generation by Latent Background Knowledge Retrieval-Augmented Diffusion

• Authors: Pancheng Zhao, Peng Xu, Pengda Qin, Deng-Ping Fan, Zhicheng Zhang, Guoli Jia, Bowen Zhou, Jufeng Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00292
• Pdf URL: https://arxiv.org/pdf/2404.00292
• Copy Paste: [[2404.00292]] LAKE-RED: Camouflaged Images Generation by Latent Background Knowledge Retrieval-Augmented Diffusion(https://arxiv.org/abs/2404.00292)
• Keywords: interpretability, diffusion
• Abstract: Camouflaged vision perception is an important vision task with numerous practical applications. Due to the expensive collection and labeling costs, this community struggles with a major bottleneck that the species category of its datasets is limited to a small number of object species. However, the existing camouflaged generation methods require specifying the background manually, thus failing to extend the camouflaged sample diversity in a low-cost manner. In this paper, we propose a Latent Background Knowledge Retrieval-Augmented Diffusion (LAKE-RED) for camouflaged image generation. To our knowledge, our contributions mainly include: (1) For the first time, we propose a camouflaged generation paradigm that does not need to receive any background inputs. (2) Our LAKE-RED is the first knowledge retrieval-augmented method with interpretability for camouflaged generation, in which we propose an idea that knowledge retrieval and reasoning enhancement are separated explicitly, to alleviate the task-specific challenges. Moreover, our method is not restricted to specific foreground targets or backgrounds, offering a potential for extending camouflaged vision perception to more diverse domains. (3) Experimental results demonstrate that our method outperforms the existing approaches, generating more realistic camouflage images.

Title: TRABSA: Interpretable Sentiment Analysis of Tweets using Attention-based BiLSTM and Twitter-RoBERTa

• Authors: Md Abrar Jahin, Md Sakib Hossain Shovon, M. F. Mridha
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00297
• Pdf URL: https://arxiv.org/pdf/2404.00297
• Copy Paste: [[2404.00297]] TRABSA: Interpretable Sentiment Analysis of Tweets using Attention-based BiLSTM and Twitter-RoBERTa(https://arxiv.org/abs/2404.00297)
• Keywords: interpretability, explainability, transformer
• Abstract: Sentiment analysis is crucial for understanding public opinion and consumer behavior. Existing models face challenges with linguistic diversity, generalizability, and explainability. We propose TRABSA, a hybrid framework integrating transformer-based architectures, attention mechanisms, and BiLSTM networks to address this. Leveraging RoBERTa-trained on 124M tweets, we bridge gaps in sentiment analysis benchmarks, ensuring state-of-the-art accuracy. Augmenting datasets with tweets from 32 countries and US states, we compare six word-embedding techniques and three lexicon-based labeling techniques, selecting the best for optimal sentiment analysis. TRABSA outperforms traditional ML and deep learning models with 94% accuracy and significant precision, recall, and F1-score gains. Evaluation across diverse datasets demonstrates consistent superiority and generalizability. SHAP and LIME analyses enhance interpretability, improving confidence in predictions. Our study facilitates pandemic resource management, aiding resource planning, policy formation, and vaccination tactics.

Title: A Comprehensive Study on NLP Data Augmentation for Hate Speech Detection: Legacy Methods, BERT, and LLMs

• Authors: Md Saroar Jahan, Mourad Oussalah, Djamila Romaissa Beddia, Jhuma kabir Mim, Nabil Arhab
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00303
• Pdf URL: https://arxiv.org/pdf/2404.00303
• Copy Paste: [[2404.00303]] A Comprehensive Study on NLP Data Augmentation for Hate Speech Detection: Legacy Methods, BERT, and LLMs(https://arxiv.org/abs/2404.00303)
• Keywords: large language model
• Abstract: The surge of interest in data augmentation within the realm of NLP has been driven by the need to address challenges posed by hate speech domains, the dynamic nature of social media vocabulary, and the demands for large-scale neural networks requiring extensive training data. However, the prevalent use of lexical substitution in data augmentation has raised concerns, as it may inadvertently alter the intended meaning, thereby impacting the efficacy of supervised machine learning models. In pursuit of suitable data augmentation methods, this study explores both established legacy approaches and contemporary practices such as Large Language Models (LLM), including GPT in Hate Speech detection. Additionally, we propose an optimized utilization of BERT-based encoder models with contextual cosine similarity filtration, exposing significant limitations in prior synonym substitution methods. Our comparative analysis encompasses five popular augmentation techniques: WordNet and Fast-Text synonym replacement, Back-translation, BERT-mask contextual augmentation, and LLM. Our analysis across five benchmarked datasets revealed that while traditional methods like back-translation show low label alteration rates (0.3-1.5%), and BERT-based contextual synonym replacement offers sentence diversity but at the cost of higher label alteration rates (over 6%). Our proposed BERT-based contextual cosine similarity filtration markedly reduced label alteration to just 0.05%, demonstrating its efficacy in 0.7% higher F1 performance. However, augmenting data with GPT-3 not only avoided overfitting with up to sevenfold data increase but also improved embedding space coverage by 15% and classification F1 score by 1.4% over traditional methods, and by 0.8% over our method.

Title: ST-LLM: Large Language Models Are Effective Temporal Learners

• Authors: Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, Ge Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00308
• Pdf URL: https://arxiv.org/pdf/2404.00308
• Copy Paste: [[2404.00308]] ST-LLM: Large Language Models Are Effective Temporal Learners(https://arxiv.org/abs/2404.00308)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have showcased impressive capabilities in text comprehension and generation, prompting research efforts towards video LLMs to facilitate human-AI interaction at the video level. However, how to effectively encode and understand videos in video-based dialogue systems remains to be solved. In this paper, we investigate a straightforward yet unexplored question: Can we feed all spatial-temporal tokens into the LLM, thus delegating the task of video sequence modeling to the LLMs? Surprisingly, this simple approach yields significant improvements in video understanding. Based upon this, we propose ST-LLM, an effective video-LLM baseline with Spatial-Temporal sequence modeling inside LLM. Furthermore, to address the overhead and stability issues introduced by uncompressed video tokens within LLMs, we develop a dynamic masking strategy with tailor-made training objectives. For particularly long videos, we have also designed a global-local input module to balance efficiency and effectiveness. Consequently, we harness LLM for proficient spatial-temporal modeling, while upholding efficiency and stability. Extensive experimental results attest to the effectiveness of our method. Through a more concise model and training pipeline, ST-LLM establishes a new state-of-the-art result on VideoChatGPT-Bench and MVBench. Codes have been available at https://github.com/TencentARC/ST-LLM.

Title: Bayesian Exploration of Pre-trained Models for Low-shot Image Classification

• Authors: Yibo Miao, Yu Lei, Feng Zhou, Zhijie Deng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00312
• Pdf URL: https://arxiv.org/pdf/2404.00312
• Copy Paste: [[2404.00312]] Bayesian Exploration of Pre-trained Models for Low-shot Image Classification(https://arxiv.org/abs/2404.00312)
• Keywords: robust
• Abstract: Low-shot image classification is a fundamental task in computer vision, and the emergence of large-scale vision-language models such as CLIP has greatly advanced the forefront of research in this field. However, most existing CLIP-based methods lack the flexibility to effectively incorporate other pre-trained models that encompass knowledge distinct from CLIP. To bridge the gap, this work proposes a simple and effective probabilistic model ensemble framework based on Gaussian processes, which have previously demonstrated remarkable efficacy in processing small data. We achieve the integration of prior knowledge by specifying the mean function with CLIP and the kernel function with an ensemble of deep kernels built upon various pre-trained models. By regressing the classification label directly, our framework enables analytical inference, straightforward uncertainty quantification, and principled hyper-parameter tuning. Through extensive experiments on standard benchmarks, we demonstrate that our method consistently outperforms competitive ensemble baselines regarding predictive performance. Additionally, we assess the robustness of our method and the quality of the yielded uncertainty estimates on out-of-distribution datasets. We also illustrate that our method, despite relying on label regression, still enjoys superior model calibration compared to most deterministic baselines.

Title: Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange

• Authors: Ankit Satpute, Noah Giessing, Andre Greiner-Petter, Moritz Schubotz, Olaf Teschke, Akiko Aizawa, Bela Gipp
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.00344
• Pdf URL: https://arxiv.org/pdf/2404.00344
• Copy Paste: [[2404.00344]] Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange(https://arxiv.org/abs/2404.00344)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities in various natural language tasks, often achieving performances that surpass those of humans. Despite these advancements, the domain of mathematics presents a distinctive challenge, primarily due to its specialized structure and the precision it demands. In this study, we adopted a two-step approach for investigating the proficiency of LLMs in answering mathematical questions. First, we employ the most effective LLMs, as identified by their performance on math question-answer benchmarks, to generate answers to 78 questions from the Math Stack Exchange (MSE). Second, a case analysis is conducted on the LLM that showed the highest performance, focusing on the quality and accuracy of its answers through manual evaluation. We found that GPT-4 performs best (nDCG of 0.48 and P@10 of 0.37) amongst existing LLMs fine-tuned for answering mathematics questions and outperforms the current best approach on ArqMATH3 Task1, considering P@10. Our Case analysis indicates that while the GPT-4 can generate relevant responses in certain instances, it does not consistently answer all questions accurately. This paper explores the current limitations of LLMs in navigating complex mathematical problem-solving. Through case analysis, we shed light on the gaps in LLM capabilities within mathematics, thereby setting the stage for future research and advancements in AI-driven mathematical reasoning. We make our code and findings publicly available for research: \url{https://github.com/gipplab/LLM-Investig-MathStackExchange}

Title: MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Text

• Authors: Takayuki Hara, Tatsuya Harada
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00345
• Pdf URL: https://arxiv.org/pdf/2404.00345
• Copy Paste: [[2404.00345]] MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Text(https://arxiv.org/abs/2404.00345)
• Keywords: generative
• Abstract: The generation of 3D scenes from user-specified conditions offers a promising avenue for alleviating the production burden in 3D applications. Previous studies required significant effort to realize the desired scene, owing to limited control conditions. We propose a method for controlling and generating 3D scenes under multimodal conditions using partial images, layout information represented in the top view, and text prompts. Combining these conditions to generate a 3D scene involves the following significant difficulties: (1) the creation of large datasets, (2) reflection on the interaction of multimodal conditions, and (3) domain dependence of the layout conditions. We decompose the process of 3D scene generation into 2D image generation from the given conditions and 3D scene generation from 2D images. 2D image generation is achieved by fine-tuning a pretrained text-to-image model with a small artificial dataset of partial images and layouts, and 3D scene generation is achieved by layout-conditioned depth estimation and neural radiance fields (NeRF), thereby avoiding the creation of large datasets. The use of a common representation of spatial information using 360-degree images allows for the consideration of multimodal condition interactions and reduces the domain dependence of the layout control. The experimental results qualitatively and quantitatively demonstrated that the proposed method can generate 3D scenes in diverse domains, from indoor to outdoor, according to multimodal conditions.

Title: SGDFormer: One-stage Transformer-based Architecture for Cross-Spectral Stereo Image Guided Denoising

• Authors: Runmin Zhang, Zhu Yu, Zehua Sheng, Jiacheng Ying, Si-Yuan Cao, Shu-Jie Chen, Bailin Yang, Junwei Li, Hui-Liang Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00349
• Pdf URL: https://arxiv.org/pdf/2404.00349
• Copy Paste: [[2404.00349]] SGDFormer: One-stage Transformer-based Architecture for Cross-Spectral Stereo Image Guided Denoising(https://arxiv.org/abs/2404.00349)
• Keywords: robust, transformer
• Abstract: Cross-spectral image guided denoising has shown its great potential in recovering clean images with rich details, such as using the near-infrared image to guide the denoising process of the visible one. To obtain such image pairs, a feasible and economical way is to employ a stereo system, which is widely used on mobile devices. Current works attempt to generate an aligned guidance image to handle the disparity between two images. However, due to occlusion, spectral differences and noise degradation, the aligned guidance image generally exists ghosting and artifacts, leading to an unsatisfactory denoised result. To address this issue, we propose a one-stage transformer-based architecture, named SGDFormer, for cross-spectral Stereo image Guided Denoising. The architecture integrates the correspondence modeling and feature fusion of stereo images into a unified network. Our transformer block contains a noise-robust cross-attention (NRCA) module and a spatially variant feature fusion (SVFF) module. The NRCA module captures the long-range correspondence of two images in a coarse-to-fine manner to alleviate the interference of noise. The SVFF module further enhances salient structures and suppresses harmful artifacts through dynamically selecting useful information. Thanks to the above design, our SGDFormer can restore artifact-free images with fine structures, and achieves state-of-the-art performance on various datasets. Additionally, our SGDFormer can be extended to handle other unaligned cross-model guided restoration tasks such as guided depth super-resolution.

Title: Rethinking Attention-Based Multiple Instance Learning for Whole-Slide Pathological Image Classification: An Instance Attribute Viewpoint

• Authors: Linghan Cai, Shenjin Huang, Ye Zhang, Jinpeng Lu, Yongbing Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00351
• Pdf URL: https://arxiv.org/pdf/2404.00351
• Copy Paste: [[2404.00351]] Rethinking Attention-Based Multiple Instance Learning for Whole-Slide Pathological Image Classification: An Instance Attribute Viewpoint(https://arxiv.org/abs/2404.00351)
• Keywords: robust, extraction
• Abstract: Multiple instance learning (MIL) is a robust paradigm for whole-slide pathological image (WSI) analysis, processing gigapixel-resolution images with slide-level labels. As pioneering efforts, attention-based MIL (ABMIL) and its variants are increasingly becoming popular due to the characteristics of simultaneously handling clinical diagnosis and tumor localization. However, the attention mechanism exhibits limitations in discriminating between instances, which often misclassifies tissues and potentially impairs MIL performance. This paper proposes an Attribute-Driven MIL (AttriMIL) framework to address these issues. Concretely, we dissect the calculation process of ABMIL and present an attribute scoring mechanism that measures the contribution of each instance to bag prediction effectively, quantifying instance attributes. Based on attribute quantification, we develop a spatial attribute constraint and an attribute ranking constraint to model instance correlations within and across slides, respectively. These constraints encourage the network to capture the spatial correlation and semantic similarity of instances, improving the ability of AttriMIL to distinguish tissue types and identify challenging instances. Additionally, AttriMIL employs a histopathology adaptive backbone that maximizes the pre-trained model's feature extraction capability for collecting pathological features. Extensive experiments on three public benchmarks demonstrate that our AttriMIL outperforms existing state-of-the-art frameworks across multiple evaluation metrics. The implementation code is available at https://github.com/MedCAI/AttriMIL.

Title: Spread Your Wings: A Radial Strip Transformer for Image Deblurring

• Authors: Duosheng Chen, Shihao Zhou, Jinshan Pan, Jinglei Shi, Lishen Qu, Jufeng Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00358
• Pdf URL: https://arxiv.org/pdf/2404.00358
• Copy Paste: [[2404.00358]] Spread Your Wings: A Radial Strip Transformer for Image Deblurring(https://arxiv.org/abs/2404.00358)
• Keywords: extraction, transformer
• Abstract: Exploring motion information is important for the motion deblurring task. Recent the window-based transformer approaches have achieved decent performance in image deblurring. Note that the motion causing blurry results is usually composed of translation and rotation movements and the window-shift operation in the Cartesian coordinate system by the window-based transformer approaches only directly explores translation motion in orthogonal directions. Thus, these methods have the limitation of modeling the rotation part. To alleviate this problem, we introduce the polar coordinate-based transformer, which has the angles and distance to explore rotation motion and translation information together. In this paper, we propose a Radial Strip Transformer (RST), which is a transformer-based architecture that restores the blur images in a polar coordinate system instead of a Cartesian one. RST contains a dynamic radial embedding module (DRE) to extract the shallow feature by a radial deformable convolution. We design a polar mask layer to generate the offsets for the deformable convolution, which can reshape the convolution kernel along the radius to better capture the rotation motion information. Furthermore, we proposed a radial strip attention solver (RSAS) as deep feature extraction, where the relationship of windows is organized by azimuth and radius. This attention module contains radial strip windows to reweight image features in the polar coordinate, which preserves more useful information in rotation and translation motion together for better recovering the sharp images. Experimental results on six synthesis and real-world datasets prove that our method performs favorably against other SOTA methods for the image deblurring task.

Title: Controllable and Diverse Data Augmentation with Large Language Model for Low-Resource Open-Domain Dialogue Generation

• Authors: Zhenhua Liu, Tong Zhu, Jianxiang Xiang, Wenliang Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00361
• Pdf URL: https://arxiv.org/pdf/2404.00361
• Copy Paste: [[2404.00361]] Controllable and Diverse Data Augmentation with Large Language Model for Low-Resource Open-Domain Dialogue Generation(https://arxiv.org/abs/2404.00361)
• Keywords: large language model
• Abstract: Data augmentation (DA) is crucial to mitigate model training instability and over-fitting problems in low-resource open-domain dialogue generation. However, traditional DA methods often neglect semantic data diversity, restricting the overall quality. Recently, large language models (LLM) have been used for DA to generate diversified dialogues. However, they have limited controllability and tend to generate dialogues with a distribution shift compared to the seed dialogues. To maximize the augmentation diversity and address the controllability problem, we propose \textbf{S}ummary-based \textbf{D}ialogue \textbf{A}ugmentation with LLM (SDA). Our approach enhances the controllability of LLM by using dialogue summaries as a planning tool. Based on summaries, SDA can generate high-quality and diverse dialogue data even with a small seed dataset. To evaluate the efficacy of data augmentation methods for open-domain dialogue, we designed a clustering-based metric to characterize the semantic diversity of the augmented dialogue data. The experimental results show that SDA can augment high-quality and semantically diverse dialogues given a small seed dataset and an LLM, and the augmented data can boost the performance of open-domain dialogue models.

Title: STBA: Towards Evaluating the Robustness of DNNs for Query-Limited Black-box Scenario

• Authors: Renyang Liu, Kwok-Yan Lam, Wei Zhou, Sixing Wu, Jun Zhao, Dongting Hu, Mingming Gong
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.00362
• Pdf URL: https://arxiv.org/pdf/2404.00362
• Copy Paste: [[2404.00362]] STBA: Towards Evaluating the Robustness of DNNs for Query-Limited Black-box Scenario(https://arxiv.org/abs/2404.00362)
• Keywords: attack, robust
• Abstract: Many attack techniques have been proposed to explore the vulnerability of DNNs and further help to improve their robustness. Despite the significant progress made recently, existing black-box attack methods still suffer from unsatisfactory performance due to the vast number of queries needed to optimize desired perturbations. Besides, the other critical challenge is that adversarial examples built in a noise-adding manner are abnormal and struggle to successfully attack robust models, whose robustness is enhanced by adversarial training against small perturbations. There is no doubt that these two issues mentioned above will significantly increase the risk of exposure and result in a failure to dig deeply into the vulnerability of DNNs. Hence, it is necessary to evaluate DNNs' fragility sufficiently under query-limited settings in a non-additional way. In this paper, we propose the Spatial Transform Black-box Attack (STBA), a novel framework to craft formidable adversarial examples in the query-limited scenario. Specifically, STBA introduces a flow field to the high-frequency part of clean images to generate adversarial examples and adopts the following two processes to enhance their naturalness and significantly improve the query efficiency: a) we apply an estimated flow field to the high-frequency part of clean images to generate adversarial examples instead of introducing external noise to the benign image, and b) we leverage an efficient gradient estimation method based on a batch of samples to optimize such an ideal flow field under query-limited settings. Compared to existing score-based black-box baselines, extensive experiments indicated that STBA could effectively improve the imperceptibility of the adversarial examples and remarkably boost the attack success rate under query-limited settings.

Title: Efficient Multi-branch Segmentation Network for Situation Awareness in Autonomous Navigation

• Authors: Guan-Cheng Zhou, Chen Chengb, Yan-zhou Chena
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00366
• Pdf URL: https://arxiv.org/pdf/2404.00366
• Copy Paste: [[2404.00366]] Efficient Multi-branch Segmentation Network for Situation Awareness in Autonomous Navigation(https://arxiv.org/abs/2404.00366)
• Keywords: robust, extraction, segmentation
• Abstract: Real-time and high-precision situational awareness technology is critical for autonomous navigation of unmanned surface vehicles (USVs). In particular, robust and fast obstacle semantic segmentation methods are essential. However, distinguishing between the sea and the sky is challenging due to the differences between port and maritime environments. In this study, we built a dataset that captured perspectives from USVs and unmanned aerial vehicles in a maritime port environment and analysed the data features. Statistical analysis revealed a high correlation between the distribution of the sea and sky and row positional information. Based on this finding, a three-branch semantic segmentation network with a row position encoding module (RPEM) was proposed to improve the prediction accuracy between the sea and the sky. The proposed RPEM highlights the effect of row coordinates on feature extraction. Compared to the baseline, the three-branch network with RPEM significantly improved the ability to distinguish between the sea and the sky without significantly reducing the computational speed.

Title: From Learning to Analytics: Improving Model Efficacy with Goal-Directed Client Selection

• Authors: Jingwen Tong, Zhenzhen Chen, Liqun Fu, Jun Zhang, Zhu Han
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.00371
• Pdf URL: https://arxiv.org/pdf/2404.00371
• Copy Paste: [[2404.00371]] From Learning to Analytics: Improving Model Efficacy with Goal-Directed Client Selection(https://arxiv.org/abs/2404.00371)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) is an appealing paradigm for learning a global model among distributed clients while preserving data privacy. Driven by the demand for high-quality user experiences, evaluating the well-trained global model after the FL process is crucial. In this paper, we propose a closed-loop model analytics framework that allows for effective evaluation of the trained global model using clients' local data. To address the challenges posed by system and data heterogeneities in the FL process, we study a goal-directed client selection problem based on the model analytics framework by selecting a subset of clients for the model training. This problem is formulated as a stochastic multi-armed bandit (SMAB) problem. We first put forth a quick initial upper confidence bound (Quick-Init UCB) algorithm to solve this SMAB problem under the federated analytics (FA) framework. Then, we further propose a belief propagation-based UCB (BP-UCB) algorithm under the democratized analytics (DA) framework. Moreover, we derive two regret upper bounds for the proposed algorithms, which increase logarithmically over the time horizon. The numerical results demonstrate that the proposed algorithms achieve nearly optimal performance, with a gap of less than 1.44% and 3.12% under the FA and DA frameworks, respectively.

Title: Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks

• Authors: Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, Jaewoo Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00376
• Pdf URL: https://arxiv.org/pdf/2404.00376
• Copy Paste: [[2404.00376]] Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks(https://arxiv.org/abs/2404.00376)
• Keywords: security, privacy, large language model
• Abstract: While recent advancements in commercial large language models (LM) have shown promising results in medical tasks, their closed-source nature poses significant privacy and security concerns, hindering their widespread use in the medical field. Despite efforts to create open-source models, their limited parameters often result in insufficient multi-step reasoning capabilities required for solving complex medical problems. To address this, we introduce Meerkat-7B, a novel medical AI system with 7 billion parameters. Meerkat-7B was trained using our new synthetic dataset consisting of high-quality chain-of-thought reasoning paths sourced from 18 medical textbooks, along with diverse instruction-following datasets. Our system achieved remarkable accuracy across seven medical benchmarks, surpassing GPT-3.5 by 13.1%, as well as outperforming the previous best 7B models such as MediTron-7B and BioMistral-7B by 13.4% and 9.8%, respectively. Notably, it surpassed the passing threshold of the United States Medical Licensing Examination (USMLE) for the first time for a 7B-parameter model. Additionally, our system offered more detailed free-form responses to clinical queries compared to existing 7B and 13B models, approaching the performance level of GPT-3.5. This significantly narrows the performance gap with large LMs, showcasing its effectiveness in addressing complex medical challenges.

Title: DHR: Dual Features-Driven Hierarchical Rebalancing in Inter- and Intra-Class Regions for Weakly-Supervised Semantic Segmentation

• Authors: Sanghyun Jo, Fei Pan, In-Jae Yu, Kyungsu Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00380
• Pdf URL: https://arxiv.org/pdf/2404.00380
• Copy Paste: [[2404.00380]] DHR: Dual Features-Driven Hierarchical Rebalancing in Inter- and Intra-Class Regions for Weakly-Supervised Semantic Segmentation(https://arxiv.org/abs/2404.00380)
• Keywords: segmentation
• Abstract: Weakly-supervised semantic segmentation (WSS) ensures high-quality segmentation with limited data and excels when employed as input seed masks for large-scale vision models such as Segment Anything. However, WSS faces challenges related to minor classes since those are overlooked in images with adjacent multiple classes, a limitation originating from the overfitting of traditional expansion methods like Random Walk. We first address this by employing unsupervised and weakly-supervised feature maps instead of conventional methodologies, allowing for hierarchical mask enhancement. This method distinctly categorizes higher-level classes and subsequently separates their associated lower-level classes, ensuring all classes are correctly restored in the mask without losing minor ones. Our approach, validated through extensive experimentation, significantly improves WSS across five benchmarks (VOC: 79.8\%, COCO: 53.9\%, Context: 49.0\%, ADE: 32.9\%, Stuff: 37.4\%), reducing the gap with fully supervised methods by over 84\% on the VOC validation set. Code is available at https://github.com/shjo-april/DHR.

Title: TTD: Text-Tag Self-Distillation Enhancing Image-Text Alignment in CLIP to Alleviate Single Tag Bias

• Authors: Sanghyun Jo, Soohyun Ryu, Sungyub Kim, Eunho Yang, Kyungsu Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00384
• Pdf URL: https://arxiv.org/pdf/2404.00384
• Copy Paste: [[2404.00384]] TTD: Text-Tag Self-Distillation Enhancing Image-Text Alignment in CLIP to Alleviate Single Tag Bias(https://arxiv.org/abs/2404.00384)
• Keywords: extraction, segmentation
• Abstract: We identify a critical bias in contemporary CLIP-based models, which we denote as \textit{single tag bias}. This bias manifests as a disproportionate focus on a singular tag (word) while neglecting other pertinent tags, stemming from CLIP's text embeddings that prioritize one specific tag in image-text relationships. When deconstructing text into individual tags, only one tag tends to have high relevancy with CLIP's image embedding, leading to an imbalanced tag relevancy. This results in an uneven alignment among multiple tags present in the text. To tackle this challenge, we introduce a novel two-step fine-tuning approach. First, our method leverages the similarity between tags and their nearest pixels for scoring, enabling the extraction of image-relevant tags from the text. Second, we present a self-distillation strategy aimed at aligning the combined masks from extracted tags with the text-derived mask. This approach mitigates the single tag bias, thereby significantly improving the alignment of CLIP's model without necessitating additional data or supervision. Our technique demonstrates model-agnostic improvements in multi-tag classification and segmentation tasks, surpassing competing methods that rely on external resources. Code is available at https://github.com/shjo-april/TTD.

Title: Jetsons at FinNLP 2024: Towards Understanding the ESG Impact of a News Article using Transformer-based Models

• Authors: Parag Pravin Dakle, Alolika Gon, Sihan Zha, Liang Wang, SaiKrishna Rallabandi, Preethi Raghavan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00386
• Pdf URL: https://arxiv.org/pdf/2404.00386
• Copy Paste: [[2404.00386]] Jetsons at FinNLP 2024: Towards Understanding the ESG Impact of a News Article using Transformer-based Models(https://arxiv.org/abs/2404.00386)
• Keywords: transformer
• Abstract: In this paper, we describe the different approaches explored by the Jetsons team for the Multi-Lingual ESG Impact Duration Inference (ML-ESG-3) shared task. The shared task focuses on predicting the duration and type of the ESG impact of a news article. The shared task dataset consists of 2,059 news titles and articles in English, French, Korean, and Japanese languages. For the impact duration classification task, we fine-tuned XLM-RoBERTa with a custom fine-tuning strategy and using self-training and DeBERTa-v3 using only English translations. These models individually ranked first on the leaderboard for Korean and Japanese and in an ensemble for the English language, respectively. For the impact type classification task, our XLM-RoBERTa model fine-tuned using a custom fine-tuning strategy ranked first for the English language.

Title: An Analysis of BPE Vocabulary Trimming in Neural Machine Translation

• Authors: Marco Cognetta, Tatsuya Hiraoka, Naoaki Okazaki, Rico Sennrich, Yuval Pinter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00397
• Pdf URL: https://arxiv.org/pdf/2404.00397
• Copy Paste: [[2404.00397]] An Analysis of BPE Vocabulary Trimming in Neural Machine Translation(https://arxiv.org/abs/2404.00397)
• Keywords: robust
• Abstract: We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a postprocessing step that replaces rare subwords with their component subwords. The technique is available in popular tokenization libraries but has not been subjected to rigorous scientific scrutiny. While the removal of rare subwords is suggested as best practice in machine translation implementations, both as a means to reduce model size and for improving model performance through robustness, our experiments indicate that, across a large space of hyperparameter settings, vocabulary trimming fails to improve performance, and is even prone to incurring heavy degradation.

Title: Aurora-M: The First Open Source Multilingual Language Model Red-teamed according to the U.S. Executive Order

• Authors: Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, Diganta Misra, Ben Bogin, Xuan-Son Vu, Marzena Karpinska, Arnav Varma Dantuluri, Wojciech Kusa, Tommaso Furlanello, Rio Yokota, Niklas Muennighoff, Suhas Pai, Tosin Adewumi, Veronika Laippala, Xiaozhe Yao, Adalberto Junior, Alpay Ariyak, Aleksandr Drozd, Jordan Clive, Kshitij Gupta, Liangyu Chen, Qi Sun, Ken Tsui, Noah Persaud, Nour Fahmy, Tianlong Chen, Mohit Bansal, Nicolo Monti, Tai Dang, Ziyang Luo, Tien-Tung Bui, Roberto Navigli, Virendra Mehta, Matthew Blumberg, Victor May, Huu Nguyen, Sampo Pyysalo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00399
• Pdf URL: https://arxiv.org/pdf/2404.00399
• Copy Paste: [[2404.00399]] Aurora-M: The First Open Source Multilingual Language Model Red-teamed according to the U.S. Executive Order(https://arxiv.org/abs/2404.00399)
• Keywords: secure, robust
• Abstract: Pretrained language models underpin several AI applications, but their high computational cost for training limits accessibility. Initiatives such as BLOOM and StarCoder aim to democratize access to pretrained models for collaborative community development. However, such existing models face challenges: limited multilingual capabilities, continual pretraining causing catastrophic forgetting, whereas pretraining from scratch is computationally expensive, and compliance with AI safety and development laws. This paper presents Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. Continually pretrained from StarCoderPlus on 435 billion additional tokens, Aurora-M surpasses 2 trillion tokens in total training token count. It is the first open-source multilingual model fine-tuned on human-reviewed safety instructions, thus aligning its development not only with conventional red-teaming considerations, but also with the specific concerns articulated in the Biden-Harris Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Aurora-M is rigorously evaluated across various tasks and languages, demonstrating robustness against catastrophic forgetting and outperforming alternatives in multilingual settings, particularly in safety evaluations. To promote responsible open-source LLM development, Aurora-M and its variants are released at https://huggingface.co/collections/aurora-m/aurora-m-models-65fdfdff62471e09812f5407 .

Title: How Robust are the Tabular QA Models for Scientific Tables? A Study using Customized Dataset

• Authors: Akash Ghosh, B Venkata Sahith, Niloy Ganguly, Pawan Goyal, Mayank Singh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00401
• Pdf URL: https://arxiv.org/pdf/2404.00401
• Copy Paste: [[2404.00401]] How Robust are the Tabular QA Models for Scientific Tables? A Study using Customized Dataset(https://arxiv.org/abs/2404.00401)
• Keywords: robust
• Abstract: Question-answering (QA) on hybrid scientific tabular and textual data deals with scientific information, and relies on complex numerical reasoning. In recent years, while tabular QA has seen rapid progress, understanding their robustness on scientific information is lacking due to absence of any benchmark dataset. To investigate the robustness of the existing state-of-the-art QA models on scientific hybrid tabular data, we propose a new dataset, "SciTabQA", consisting of 822 question-answer pairs from scientific tables and their descriptions. With the help of this dataset, we assess the state-of-the-art Tabular QA models based on their ability (i) to use heterogeneous information requiring both structured data (table) and unstructured data (text) and (ii) to perform complex scientific reasoning tasks. In essence, we check the capability of the models to interpret scientific tables and text. Our experiments show that "SciTabQA" is an innovative dataset to study question-answering over scientific heterogeneous data. We benchmark three state-of-the-art Tabular QA models, and find that the best F1 score is only 0.462.

Title: UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause

• Authors: Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Hasti Seifi, Jiayuan Xie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00403
• Pdf URL: https://arxiv.org/pdf/2404.00403
• Copy Paste: [[2404.00403]] UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause(https://arxiv.org/abs/2404.00403)
• Keywords: extraction
• Abstract: Multimodal emotion recognition in conversation (MERC) and multimodal emotion-cause pair extraction (MECPE) has recently garnered significant attention. Emotions are the expression of affect or feelings; responses to specific events, thoughts, or situations are known as emotion causes. Both are like two sides of a coin, collectively describing human behaviors and intents. However, most existing works treat MERC and MECPE as separate tasks, which may result in potential challenges in integrating emotion and cause in real-world applications. In this paper, we propose a Unified Multimodal Emotion recognition and Emotion-Cause analysis framework (UniMEEC) to explore the causality and complementarity between emotion and emotion cause. Concretely, UniMEEC reformulates the MERC and MECPE tasks as two mask prediction problems, enhancing the interaction between emotion and cause. Meanwhile, UniMEEC shares the prompt learning among modalities for probing modality-specific knowledge from the Pre-trained model. Furthermore, we propose a task-specific hierarchical context aggregation to control the information flow to the task. Experiment results on four public benchmark datasets verify the model performance on MERC and MECPE tasks and achieve consistent improvements compared with state-of-the-art methods.

Title: TACO -- Twitter Arguments from COnversations

• Authors: Marc Feger, Stefan Dietze
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00406
• Pdf URL: https://arxiv.org/pdf/2404.00406
• Copy Paste: [[2404.00406]] TACO -- Twitter Arguments from COnversations(https://arxiv.org/abs/2404.00406)
• Keywords: transformer
• Abstract: Twitter has emerged as a global hub for engaging in online conversations and as a research corpus for various disciplines that have recognized the significance of its user-generated content. Argument mining is an important analytical task for processing and understanding online discourse. Specifically, it aims to identify the structural elements of arguments, denoted as information and inference. These elements, however, are not static and may require context within the conversation they are in, yet there is a lack of data and annotation frameworks addressing this dynamic aspect on Twitter. We contribute TACO, the first dataset of Twitter Arguments utilizing 1,814 tweets covering 200 entire conversations spanning six heterogeneous topics annotated with an agreement of 0.718 Krippendorff's alpha among six experts. Second, we provide our annotation framework, incorporating definitions from the Cambridge Dictionary, to define and identify argument components on Twitter. Our transformer-based classifier achieves an 85.06\% macro F1 baseline score in detecting arguments. Moreover, our data reveals that Twitter users tend to engage in discussions involving informed inferences and information. TACO serves multiple purposes, such as training tweet classifiers to manage tweets based on inference and information elements, while also providing valuable insights into the conversational reply patterns of tweets.

Title: SVGCraft: Beyond Single Object Text-to-SVG Synthesis with Comprehensive Canvas Layout

• Authors: Ayan Banerjee, Nityanand Mathur, Josep Lladós, Umapada Pal, Anjan Dutta
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00412
• Pdf URL: https://arxiv.org/pdf/2404.00412
• Copy Paste: [[2404.00412]] SVGCraft: Beyond Single Object Text-to-SVG Synthesis with Comprehensive Canvas Layout(https://arxiv.org/abs/2404.00412)
• Keywords: diffusion
• Abstract: Generating VectorArt from text prompts is a challenging vision task, requiring diverse yet realistic depictions of the seen as well as unseen entities. However, existing research has been mostly limited to the generation of single objects, rather than comprehensive scenes comprising multiple elements. In response, this work introduces SVGCraft, a novel end-to-end framework for the creation of vector graphics depicting entire scenes from textual descriptions. Utilizing a pre-trained LLM for layout generation from text prompts, this framework introduces a technique for producing masked latents in specified bounding boxes for accurate object placement. It introduces a fusion mechanism for integrating attention maps and employs a diffusion U-Net for coherent composition, speeding up the drawing process. The resulting SVG is optimized using a pre-trained encoder and LPIPS loss with opacity modulation to maximize similarity. Additionally, this work explores the potential of primitive shapes in facilitating canvas completion in constrained environments. Through both qualitative and quantitative assessments, SVGCraft is demonstrated to surpass prior works in abstraction, recognizability, and detail, as evidenced by its performance metrics (CLIP-T: 0.4563, Cosine Similarity: 0.6342, Confusion: 0.66, Aesthetic: 6.7832). The code will be available at https://github.com/ayanban011/SVGCraft.

Title: CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP

• Authors: Chandra Kiran Reddy Evuru, Sreyan Ghosh, Sonal Kumar, Ramaneswaran S, Utkarsh Tyagi, Dinesh Manocha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00415
• Pdf URL: https://arxiv.org/pdf/2404.00415
• Copy Paste: [[2404.00415]] CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP(https://arxiv.org/abs/2404.00415)
• Keywords: large language model
• Abstract: We present CoDa (Constrained Generation based Data Augmentation), a controllable, effective, and training-free data augmentation technique for low-resource (data-scarce) NLP. Our approach is based on prompting off-the-shelf instruction-following Large Language Models (LLMs) for generating text that satisfies a set of constraints. Precisely, we extract a set of simple constraints from every instance in the low-resource dataset and verbalize them to prompt an LLM to generate novel and diverse training instances. Our findings reveal that synthetic data that follows simple constraints in the downstream dataset act as highly effective augmentations, and CoDa can achieve this without intricate decoding-time constrained generation techniques or fine-tuning with complex algorithms that eventually make the model biased toward the small number of training instances. Additionally, CoDa is the first framework that provides users explicit control over the augmentation generation process, thereby also allowing easy adaptation to several domains. We demonstrate the effectiveness of CoDa across 11 datasets spanning 3 tasks and 3 low-resource settings. CoDa outperforms all our baselines, qualitatively and quantitatively, with improvements of 0.12%-7.19%. Code is available here: https://github.com/Sreyan88/CoDa

Title: Do Vision-Language Models Understand Compound Nouns?

• Authors: Sonal Kumar, Sreyan Ghosh, S Sakshi, Utkarsh Tyagi, Dinesh Manocha
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00419
• Pdf URL: https://arxiv.org/pdf/2404.00419
• Copy Paste: [[2404.00419]] Do Vision-Language Models Understand Compound Nouns?(https://arxiv.org/abs/2404.00419)
• Keywords: large language model
• Abstract: Open-vocabulary vision-language models (VLMs) like CLIP, trained using contrastive loss, have emerged as a promising new paradigm for text-to-image retrieval. However, do VLMs understand compound nouns (CNs) (e.g., lab coat) as well as they understand nouns (e.g., lab)? We curate Compun, a novel benchmark with 400 unique and commonly used CNs, to evaluate the effectiveness of VLMs in interpreting CNs. The Compun benchmark challenges a VLM for text-to-image retrieval where, given a text prompt with a CN, the task is to select the correct image that shows the CN among a pair of distractor images that show the constituent nouns that make up the CN. Next, we perform an in-depth analysis to highlight CLIPs' limited understanding of certain types of CNs. Finally, we present an alternative framework that moves beyond hand-written templates for text prompts widely used by CLIP-like models. We employ a Large Language Model to generate multiple diverse captions that include the CN as an object in the scene described by the caption. Our proposed method improves CN understanding of CLIP by 8.25% on Compun. Code and benchmark are available at: https://github.com/sonalkum/Compun

Title: Keep your memory dump shut: Unveiling data leaks in password managers

• Authors: Efstratios Chatzoglou, Vyron Kampourakis, Zisis Tsiatsikas, Georgios Karopoulos, Georgios Kambourakis
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.00423
• Pdf URL: https://arxiv.org/pdf/2404.00423
• Copy Paste: [[2404.00423]] Keep your memory dump shut: Unveiling data leaks in password managers(https://arxiv.org/abs/2404.00423)
• Keywords: protect
• Abstract: Password management has long been a persistently challenging task. This led to the introduction of password management software, which has been around for at least 25 years in various forms, including desktop and browser-based applications. This work assesses the ability of two dozen password managers, 12 desktop applications, and 12 browser-plugins, to effectively protect the confidentiality of secret credentials in six representative scenarios. Our analysis focuses on the period during which a Password Manager (PM) resides in the RAM. Despite the sensitive nature of these applications, our results show that across all scenarios, only three desktop PM applications and two browser plugins do not store plaintext passwords in the system memory. Oddly enough, at the time of writing, only two vendors recognized the exploit as a vulnerability, reserving CVE-2023-23349, while the rest chose to disregard or underrate the issue.

Title: Multiway Point Cloud Mosaicking with Diffusion and Global Optimization

• Authors: Shengze Jin (Department of Computer Science, ETH Zurich, Switzerland), Iro Armeni (Department of Civil and Environmental Engineering, Stanford University), Marc Pollefeys (Department of Computer Science, ETH Zurich, Switzerland and Microsoft Mixed Reality & AI Lab, Zurich, Switzerland), Daniel Barath (Department of Computer Science, ETH Zurich, Switzerland)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00429
• Pdf URL: https://arxiv.org/pdf/2404.00429
• Copy Paste: [[2404.00429]] Multiway Point Cloud Mosaicking with Diffusion and Global Optimization(https://arxiv.org/abs/2404.00429)
• Keywords: robust, diffusion
• Abstract: We introduce a novel framework for multiway point cloud mosaicking (named Wednesday), designed to co-align sets of partially overlapping point clouds -- typically obtained from 3D scanners or moving RGB-D cameras -- into a unified coordinate system. At the core of our approach is ODIN, a learned pairwise registration algorithm that iteratively identifies overlaps and refines attention scores, employing a diffusion-based process for denoising pairwise correlation matrices to enhance matching accuracy. Further steps include constructing a pose graph from all point clouds, performing rotation averaging, a novel robust algorithm for re-estimating translations optimally in terms of consensus maximization and translation optimization. Finally, the point cloud rotations and positions are optimized jointly by a diffusion-based approach. Tested on four diverse, large-scale datasets, our method achieves state-of-the-art pairwise and multiway registration results by a large margin on all benchmarks. Our code and models are available at https://github.com/jinsz/Multiway-Point-Cloud-Mosaicking-with-Diffusion-and-Global-Optimization.

Title: Automatic explanation of the classification of Spanish legal judgments in jurisdiction-dependent law categories with tree estimators

• Authors: Jaime González-González, Francisco de Arriba-Pérez, Silvia García-Méndez, Andrea Busto-Castiñeira, Francisco J. González-Castaño
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00437
• Pdf URL: https://arxiv.org/pdf/2404.00437
• Copy Paste: [[2404.00437]] Automatic explanation of the classification of Spanish legal judgments in jurisdiction-dependent law categories with tree estimators(https://arxiv.org/abs/2404.00437)
• Keywords: extraction
• Abstract: Automatic legal text classification systems have been proposed in the literature to address knowledge extraction from judgments and detect their aspects. However, most of these systems are black boxes even when their models are interpretable. This may raise concerns about their trustworthiness. Accordingly, this work contributes with a system combining Natural Language Processing (NLP) with Machine Learning (ML) to classify legal texts in an explainable manner. We analyze the features involved in the decision and the threshold bifurcation values of the decision paths of tree structures and present this information to the users in natural language. This is the first work on automatic analysis of legal texts combining NLP and ML along with Explainable Artificial Intelligence techniques to automatically make the models' decisions understandable to end users. Furthermore, legal experts have validated our solution, and this knowledge has also been incorporated into the explanation process as "expert-in-the-loop" dictionaries. Experimental results on an annotated data set in law categories by jurisdiction demonstrate that our system yields competitive classification performance, with accuracy values well above 90%, and that its automatic explanations are easily understandable even to non-expert users.

Title: DOCMASTER: A Unified Platform for Annotation, Training, & Inference in Document Question-Answering

• Authors: Alex Nguyen, Zilong Wang, Jingbo Shang, Dheeraj Mekala
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00439
• Pdf URL: https://arxiv.org/pdf/2404.00439
• Copy Paste: [[2404.00439]] DOCMASTER: A Unified Platform for Annotation, Training, & Inference in Document Question-Answering(https://arxiv.org/abs/2404.00439)
• Keywords: privacy
• Abstract: The application of natural language processing models to PDF documents is pivotal for various business applications yet the challenge of training models for this purpose persists in businesses due to specific hurdles. These include the complexity of working with PDF formats that necessitate parsing text and layout information for curating training data and the lack of privacy-preserving annotation tools. This paper introduces DOCMASTER, a unified platform designed for annotating PDF documents, model training, and inference, tailored to document question-answering. The annotation interface enables users to input questions and highlight text spans within the PDF file as answers, saving layout information and text spans accordingly. Furthermore, DOCMASTER supports both state-of-the-art layout-aware and text models for comprehensive training purposes. Importantly, as annotations, training, and inference occur on-device, it also safeguards privacy. The platform has been instrumental in driving several research prototypes concerning document analysis such as the AI assistant utilized by University of California San Diego's (UCSD) International Services and Engagement Office (ISEO) for processing a substantial volume of PDF documents.

Title: Planning and Editing What You Retrieve for Enhanced Tool Learning

• Authors: Tenghao Huang, Dongwon Jung, Muhao Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00450
• Pdf URL: https://arxiv.org/pdf/2404.00450
• Copy Paste: [[2404.00450]] Planning and Editing What You Retrieve for Enhanced Tool Learning(https://arxiv.org/abs/2404.00450)
• Keywords: large language model
• Abstract: Recent advancements in integrating external tools with Large Language Models (LLMs) have opened new frontiers, with applications in mathematical reasoning, code generators, and smart assistants. However, existing methods, relying on simple one-time retrieval strategies, fall short on effectively and accurately shortlisting relevant tools. This paper introduces a novel \modelname (\modelmeaning) approach, encompassing ``Plan-and-Retrieve (P\&R)'' and ``Edit-and-Ground (E\&G)'' paradigms. The P\&R paradigm consists of a neural retrieval module for shortlisting relevant tools and an LLM-based query planner that decomposes complex queries into actionable tasks, enhancing the effectiveness of tool utilization. The E\&G paradigm utilizes LLMs to enrich tool descriptions based on user scenarios, bridging the gap between user queries and tool functionalities. Experiment results demonstrate that these paradigms significantly improve the recall and NDCG in tool retrieval tasks, significantly surpassing current state-of-the-art models.

Title: MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks

• Authors: Letian Peng, Zilong Wang, Feng Yao, Zihan Wang, Jingbo Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00457
• Pdf URL: https://arxiv.org/pdf/2404.00457
• Copy Paste: [[2404.00457]] MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks(https://arxiv.org/abs/2404.00457)
• Keywords: extraction, large language model
• Abstract: Information extraction (IE) is a fundamental area in natural language processing where prompting large language models (LLMs), even with in-context examples, cannot defeat small LMs tuned on very small IE datasets. We observe that IE tasks, such as named entity recognition and relation extraction, all focus on extracting important information, which can be formalized as a label-to-span matching. In this paper, we propose a novel framework MetaIE to build a small LM as meta-model by learning to extract "important information", i.e., the meta-understanding of IE, so that this meta-model can be adapted to all kind of IE tasks effectively and efficiently. Specifically, MetaIE obtains the small LM via a symbolic distillation from an LLM following the label-to-span scheme. We construct the distillation dataset via sampling sentences from language model pre-training datasets (e.g., OpenWebText in our implementation) and prompting an LLM to identify the typed spans of "important information". We evaluate the meta-model under the few-shot adaptation setting. Extensive results on 13 datasets from 6 IE tasks confirm that MetaIE can offer a better starting point for few-shot tuning on IE datasets and outperform other meta-models from (1) vanilla language model pre-training, (2) multi-IE-task pre-training with human annotations, and (3) single-IE-task symbolic distillation from LLM. Moreover, we provide comprehensive analyses of MetaIE, such as the size of the distillation dataset, the meta-model architecture, and the size of the meta-model.

Title: Shortcuts Arising from Contrast: Effective and Covert Clean-Label Attacks in Prompt-Based Learning

• Authors: Xiaopeng Xie, Ming Yan, Xiwen Zhou, Chenlong Zhao, Suli Wang, Yong Zhang, Joey Tianyi Zhou
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.00461
• Pdf URL: https://arxiv.org/pdf/2404.00461
• Copy Paste: [[2404.00461]] Shortcuts Arising from Contrast: Effective and Covert Clean-Label Attacks in Prompt-Based Learning(https://arxiv.org/abs/2404.00461)
• Keywords: attack, steal
• Abstract: Prompt-based learning paradigm has demonstrated remarkable efficacy in enhancing the adaptability of pretrained language models (PLMs), particularly in few-shot scenarios. However, this learning paradigm has been shown to be vulnerable to backdoor attacks. The current clean-label attack, employing a specific prompt as a trigger, can achieve success without the need for external triggers and ensure correct labeling of poisoned samples, which is more stealthy compared to the poisoned-label attack, but on the other hand, it faces significant issues with false activations and poses greater challenges, necessitating a higher rate of poisoning. Using conventional negative data augmentation methods, we discovered that it is challenging to trade off between effectiveness and stealthiness in a clean-label setting. In addressing this issue, we are inspired by the notion that a backdoor acts as a shortcut and posit that this shortcut stems from the contrast between the trigger and the data utilized for poisoning. In this study, we propose a method named Contrastive Shortcut Injection (CSI), by leveraging activation values, integrates trigger design and data selection strategies to craft stronger shortcut features. With extensive experiments on full-shot and few-shot text classification tasks, we empirically validate CSI's high effectiveness and high stealthiness at low poisoning rates. Notably, we found that the two approaches play leading roles in full-shot and few-shot settings, respectively.

Title: Zero-shot Safety Prediction for Autonomous Robots with Foundation World Models

• Authors: Zhenjiang Mao, Siqi Dai, Yuang Geng, Ivan Ruchkin
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.00462
• Pdf URL: https://arxiv.org/pdf/2404.00462
• Copy Paste: [[2404.00462]] Zero-shot Safety Prediction for Autonomous Robots with Foundation World Models(https://arxiv.org/abs/2404.00462)
• Keywords: large language model
• Abstract: A world model creates a surrogate world to train a controller and predict safety violations by learning the internal dynamic model of systems. However, the existing world models rely solely on statistical learning of how observations change in response to actions, lacking precise quantification of how accurate the surrogate dynamics are, which poses a significant challenge in safety-critical systems. To address this challenge, we propose foundation world models that embed observations into meaningful and causally latent representations. This enables the surrogate dynamics to directly predict causal future states by leveraging a training-free large language model. In two common benchmarks, this novel model outperforms standard world models in the safety prediction task and has a performance comparable to supervised learning despite not using any data. We evaluate its performance with a more specialized and system-relevant metric by comparing estimated states instead of aggregating observation-wide error.

Title: Addressing Both Statistical and Causal Gender Fairness in NLP Models

• Authors: Hannah Chen, Yangfeng Ji, David Evans
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00463
• Pdf URL: https://arxiv.org/pdf/2404.00463
• Copy Paste: [[2404.00463]] Addressing Both Statistical and Causal Gender Fairness in NLP Models(https://arxiv.org/abs/2404.00463)
• Keywords: protect, fair
• Abstract: Statistical fairness stipulates equivalent outcomes for every protected group, whereas causal fairness prescribes that a model makes the same prediction for an individual regardless of their protected characteristics. Counterfactual data augmentation (CDA) is effective for reducing bias in NLP models, yet models trained with CDA are often evaluated only on metrics that are closely tied to the causal fairness notion; similarly, sampling-based methods designed to promote statistical fairness are rarely evaluated for causal fairness. In this work, we evaluate both statistical and causal debiasing methods for gender bias in NLP models, and find that while such methods are effective at reducing bias as measured by the targeted metric, they do not necessarily improve results on other bias metrics. We demonstrate that combinations of statistical and causal debiasing techniques are able to reduce bias measured through both types of metrics.

Title: Leveraging Pre-trained and Transformer-derived Embeddings from EHRs to Characterize Heterogeneity Across Alzheimer's Disease and Related Dementias

• Authors: Matthew West, Colin Magdamo, Lily Cheng, Yingnan He, Sudeshna Das
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00464
• Pdf URL: https://arxiv.org/pdf/2404.00464
• Copy Paste: [[2404.00464]] Leveraging Pre-trained and Transformer-derived Embeddings from EHRs to Characterize Heterogeneity Across Alzheimer's Disease and Related Dementias(https://arxiv.org/abs/2404.00464)
• Keywords: transformer, generative
• Abstract: Alzheimer's disease is a progressive, debilitating neurodegenerative disease that affects 50 million people globally. Despite this substantial health burden, available treatments for the disease are limited and its fundamental causes remain poorly understood. Previous work has suggested the existence of clinically-meaningful sub-types, which it is suggested may correspond to distinct etiologies, disease courses, and ultimately appropriate treatments. Here, we use unsupervised learning techniques on electronic health records (EHRs) from a cohort of memory disorder patients to characterise heterogeneity in this disease population. Pre-trained embeddings for medical codes as well as transformer-derived Clinical BERT embeddings of free text are used to encode patient EHRs. We identify the existence of sub-populations on the basis of comorbidities and shared textual features, and discuss their clinical significance.

Title: Computation and Communication Efficient Lightweighting Vertical Federated Learning

• Authors: Heqiang Wang, Jieming Bian, Lei Wang
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.00466
• Pdf URL: https://arxiv.org/pdf/2404.00466
• Copy Paste: [[2404.00466]] Computation and Communication Efficient Lightweighting Vertical Federated Learning(https://arxiv.org/abs/2404.00466)
• Keywords: robust, federate
• Abstract: The exploration of computational and communication efficiency within Federated Learning (FL) has emerged as a prominent and crucial field of study. While most existing efforts to enhance these efficiencies have focused on Horizontal FL, the distinct processes and model structures of Vertical FL preclude the direct application of Horizontal FL-based techniques. In response, we introduce the concept of Lightweight Vertical Federated Learning (LVFL), targeting both computational and communication efficiencies. This approach involves separate lightweighting strategies for the feature model, to improve computational efficiency, and for feature embedding, to enhance communication efficiency. Moreover, we establish a convergence bound for our LVFL algorithm, which accounts for both communication and computational lightweighting ratios. Our evaluation of the algorithm on a image classification dataset reveals that LVFL significantly alleviates computational and communication demands while preserving robust learning performance. This work effectively addresses the gaps in communication and computational efficiency within Vertical FL.

Title: Privacy Backdoors: Stealing Data with Corrupted Pretrained Models

• Authors: Shanglun Feng, Florian Tramèr
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00473
• Pdf URL: https://arxiv.org/pdf/2404.00473
• Copy Paste: [[2404.00473]] Privacy Backdoors: Stealing Data with Corrupted Pretrained Models(https://arxiv.org/abs/2404.00473)
• Keywords: secure, privacy, attack, steal, transformer
• Abstract: Practitioners commonly download pretrained machine learning models from open repositories and finetune them to fit specific applications. We show that this practice introduces a new risk of privacy backdoors. By tampering with a pretrained model's weights, an attacker can fully compromise the privacy of the finetuning data. We show how to build privacy backdoors for a variety of models, including transformers, which enable an attacker to reconstruct individual finetuning samples, with a guaranteed success! We further show that backdoored models allow for tight privacy attacks on models trained with differential privacy (DP). The common optimistic practice of training DP models with loose privacy guarantees is thus insecure if the model is not trusted. Overall, our work highlights a crucial and overlooked supply chain attack on machine learning privacy.

Title: Cross-lingual Named Entity Corpus for Slavic Languages

• Authors: Jakub Piskorski, Michał Marcińczuk, Roman Yangarber
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00482
• Pdf URL: https://arxiv.org/pdf/2404.00482
• Copy Paste: [[2404.00482]] Cross-lingual Named Entity Corpus for Slavic Languages(https://arxiv.org/abs/2404.00482)
• Keywords: transformer
• Abstract: This paper presents a corpus manually annotated with named entities for six Slavic languages - Bulgarian, Czech, Polish, Slovenian, Russian, and Ukrainian. This work is the result of a series of shared tasks, conducted in 2017-2023 as a part of the Workshops on Slavic Natural Language Processing. The corpus consists of 5 017 documents on seven topics. The documents are annotated with five classes of named entities. Each entity is described by a category, a lemma, and a unique cross-lingual identifier. We provide two train-tune dataset splits - single topic out and cross topics. For each split, we set benchmarks using a transformer-based neural network architecture with the pre-trained multilingual models - XLM-RoBERTa-large for named entity mention recognition and categorization, and mT5-large for named entity lemmatization and linking.

Title: Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4

• Authors: Aryo Pradipta Gema, Giwon Hong, Pasquale Minervini, Luke Daines, Beatrice Alex
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00484
• Pdf URL: https://arxiv.org/pdf/2404.00484
• Copy Paste: [[2404.00484]] Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4(https://arxiv.org/abs/2404.00484)
• Keywords: large language model
• Abstract: The NLI4CT task assesses Natural Language Inference systems in predicting whether hypotheses entail or contradict evidence from Clinical Trial Reports. In this study, we evaluate various Large Language Models (LLMs) with multiple strategies, including Chain-of-Thought, In-Context Learning, and Parameter-Efficient Fine-Tuning (PEFT). We propose a PEFT method to improve the consistency of LLMs by merging adapters that were fine-tuned separately using triplet and language modelling objectives. We found that merging the two PEFT adapters improves the F1 score (+0.0346) and consistency (+0.152) of the LLMs. However, our novel methods did not produce more accurate results than GPT-4 in terms of faithfulness and consistency. Averaging the three metrics, GPT-4 ranks joint-first in the competition with 0.8328. Finally, our contamination analysis with GPT-4 indicates that there was no test data leakage.

Title: DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans

• Authors: Akash Sengupta, Thiemo Alldieck, Nikos Kolotouros, Enric Corona, Andrei Zanfir, Cristian Sminchisescu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00485
• Pdf URL: https://arxiv.org/pdf/2404.00485
• Copy Paste: [[2404.00485]] DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans(https://arxiv.org/abs/2404.00485)
• Keywords: diffusion
• Abstract: We present DiffHuman, a probabilistic method for photorealistic 3D human reconstruction from a single RGB image. Despite the ill-posed nature of this problem, most methods are deterministic and output a single solution, often resulting in a lack of geometric detail and blurriness in unseen or uncertain regions. In contrast, DiffHuman predicts a probability distribution over 3D reconstructions conditioned on an input 2D image, which allows us to sample multiple detailed 3D avatars that are consistent with the image. DiffHuman is implemented as a conditional diffusion model that denoises pixel-aligned 2D observations of an underlying 3D shape representation. During inference, we may sample 3D avatars by iteratively denoising 2D renders of the predicted 3D representation. Furthermore, we introduce a generator neural network that approximates rendering with considerably reduced runtime (55x speed up), resulting in a novel dual-branch diffusion framework. Our experiments show that DiffHuman can produce diverse and detailed reconstructions for the parts of the person that are unseen or uncertain in the input image, while remaining competitive with the state-of-the-art when reconstructing visible surfaces.

Title: Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs

• Authors: Shu Yang, Jiayuan Su, Han Jiang, Mengdi Li, Keyuan Cheng, Muhammad Asif Ali, Lijie Hu, Di Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00486
• Pdf URL: https://arxiv.org/pdf/2404.00486
• Copy Paste: [[2404.00486]] Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs(https://arxiv.org/abs/2404.00486)
• Keywords: security, defense, attack, large language model
• Abstract: With the rise of large language models (LLMs), ensuring they embody the principles of being helpful, honest, and harmless (3H), known as Human Alignment, becomes crucial. While existing alignment methods like RLHF, DPO, etc., effectively fine-tune LLMs to match preferences in the preference dataset, they often lead LLMs to highly receptive human input and external evidence, even when this information is poisoned. This leads to a tendency for LLMs to be Adaptive Chameleons when external evidence conflicts with their parametric memory. This exacerbates the risk of LLM being attacked by external poisoned data, which poses a significant security risk to LLM system applications such as Retrieval-augmented generation (RAG). To address the challenge, we propose a novel framework: Dialectical Alignment (DA), which (1) utilizes AI feedback to identify optimal strategies for LLMs to navigate inter-context conflicts and context-memory conflicts with different external evidence in context window (i.e., different ratios of poisoned factual contexts); (2) constructs the SFT dataset as well as the preference dataset based on the AI feedback and strategies above; (3) uses the above datasets for LLM alignment to defense poisoned context attack while preserving the effectiveness of in-context knowledge editing. Our experiments show that the dialectical alignment model improves poisoned data attack defense by 20 and does not require any additional prompt engineering or prior declaration of ``you may be attacked`` to the LLMs' context window.

Title: Noise-Aware Training of Layout-Aware Language Models

• Authors: Ritesh Sarkhel, Xiaoqi Ren, Lauro Beltrao Costa, Guolong Su, Vincent Perot, Yanan Xie, Emmanouil Koukoumidis, Arnab Nandi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00488
• Pdf URL: https://arxiv.org/pdf/2404.00488
• Copy Paste: [[2404.00488]] Noise-Aware Training of Layout-Aware Language Models(https://arxiv.org/abs/2404.00488)
• Keywords: robust
• Abstract: A visually rich document (VRD) utilizes visual features along with linguistic cues to disseminate information. Training a custom extractor that identifies named entities from a document requires a large number of instances of the target document type annotated at textual and visual modalities. This is an expensive bottleneck in enterprise scenarios, where we want to train custom extractors for thousands of different document types in a scalable way. Pre-training an extractor model on unlabeled instances of the target document type, followed by a fine-tuning step on human-labeled instances does not work in these scenarios, as it surpasses the maximum allowable training time allocated for the extractor. We address this scenario by proposing a Noise-Aware Training method or NAT in this paper. Instead of acquiring expensive human-labeled documents, NAT utilizes weakly labeled documents to train an extractor in a scalable way. To avoid degradation in the model's quality due to noisy, weakly labeled samples, NAT estimates the confidence of each training sample and incorporates it as uncertainty measure during training. We train multiple state-of-the-art extractor models using NAT. Experiments on a number of publicly available and in-house datasets show that NAT-trained models are not only robust in performance -- it outperforms a transfer-learning baseline by up to 6% in terms of macro-F1 score, but it is also more label-efficient -- it reduces the amount of human-effort required to obtain comparable performance by up to 73%.

Title: PROMPT-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression

• Authors: Muhammad Asif Ali, Zhengping Li, Shu Yang, Keyuan Cheng, Yang Cao, Tianhao Huang, Lijie Hu, Lu Yu, Di Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00489
• Pdf URL: https://arxiv.org/pdf/2404.00489
• Copy Paste: [[2404.00489]] PROMPT-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression(https://arxiv.org/abs/2404.00489)
• Keywords: interpretability, large language model
• Abstract: Large language models (LLMs) have shown exceptional abilities for multiple different natural language processing tasks. While prompting is a crucial tool for LLM inference, we observe that there is a significant cost associated with exceedingly lengthy prompts. Existing attempts to compress lengthy prompts lead to sub-standard results in terms of readability and interpretability of the compressed prompt, with a detrimental impact on prompt utility. To address this, we propose PROMPT-SAW: Prompt compresSion via Relation AWare graphs, an effective strategy for prompt compression over task-agnostic and task-aware prompts. PROMPT-SAW uses the prompt's textual information to build a graph, later extracts key information elements in the graph to come up with the compressed prompt. We also propose GSM8K-AUG, i.e., an extended version of the existing GSM8k benchmark for task-agnostic prompts in order to provide a comprehensive evaluation platform. Experimental evaluation using benchmark datasets shows that prompts compressed by PROMPT-SAW are not only better in terms of readability, but they also outperform the best-performing baseline models by up to 14.3 and 13.7 respectively for task-aware and task-agnostic settings while compressing the original prompt text by 33.0 and 56.7.

Title: Denoising Monte Carlo Renders With Diffusion Models

• Authors: Vaibhav Vavilala, Rahul Vasanth, David Forsyth
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00491
• Pdf URL: https://arxiv.org/pdf/2404.00491
• Copy Paste: [[2404.00491]] Denoising Monte Carlo Renders With Diffusion Models(https://arxiv.org/abs/2404.00491)
• Keywords: diffusion
• Abstract: Physically-based renderings contain Monte-Carlo noise, with variance that increases as the number of rays per pixel decreases. This noise, while zero-mean for good modern renderers, can have heavy tails (most notably, for scenes containing specular or refractive objects). Learned methods for restoring low fidelity renders are highly developed, because suppressing render noise means one can save compute and use fast renders with few rays per pixel. We demonstrate that a diffusion model can denoise low fidelity renders successfully. Furthermore, our method can be conditioned on a variety of natural render information, and this conditioning helps performance. Quantitative experiments show that our method is competitive with SOTA across a range of sampling rates, but current metrics slightly favor competitor methods. Qualitative examination of the reconstructions suggests that the metrics themselves may not be reliable. The image prior applied by a diffusion method strongly favors reconstructions that are "like" real images -- so have straight shadow boundaries, curved specularities, no "fireflies" and the like -- and metrics do not account for this. We show numerous examples where methods preferred by current metrics produce qualitatively weaker reconstructions than ours.

Title: Multi-hop Question Answering under Temporal Knowledge Editing

• Authors: Keyuan Cheng, Gang Lin, Haoyang Fei, Yuxuan zhai, Lu Yu, Muhammad Asif Ali, Lijie Hu, Di Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00492
• Pdf URL: https://arxiv.org/pdf/2404.00492
• Copy Paste: [[2404.00492]] Multi-hop Question Answering under Temporal Knowledge Editing(https://arxiv.org/abs/2404.00492)
• Keywords: large language model
• Abstract: Multi-hop question answering (MQA) under knowledge editing (KE) has garnered significant attention in the era of large language models. However, existing models for MQA under KE exhibit poor performance when dealing with questions containing explicit temporal contexts. To address this limitation, we propose a novel framework, namely TEMPoral knowLEdge augmented Multi-hop Question Answering (TEMPLE-MQA). Unlike previous methods, TEMPLE-MQA first constructs a time-aware graph (TAG) to store edit knowledge in a structured manner. Then, through our proposed inference path, structural retrieval, and joint reasoning stages, TEMPLE-MQA effectively discerns temporal contexts within the question query. Experiments on benchmark datasets demonstrate that TEMPLE-MQA significantly outperforms baseline models. Additionally, we contribute a new dataset, namely TKEMQA, which serves as the inaugural benchmark tailored specifically for MQA with temporal scopes.

Title: Configurable Safety Tuning of Language Models with Synthetic Preference Data

• Authors: Victor Gallego
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00495
• Pdf URL: https://arxiv.org/pdf/2404.00495
• Copy Paste: [[2404.00495]] Configurable Safety Tuning of Language Models with Synthetic Preference Data(https://arxiv.org/abs/2404.00495)
• Keywords: robust
• Abstract: State-of-the-art language model fine-tuning techniques, such as Direct Preference Optimization (DPO), restrict user control by hard-coding predefined behaviors into the model. To address this, we propose a novel method, Configurable Safety Tuning (CST), that augments DPO using synthetic preference data to facilitate flexible safety configuration of LLMs at inference time. CST overcomes the constraints of vanilla DPO by introducing a system prompt specifying safety configurations, enabling LLM deployers to disable/enable safety preferences based on their need, just changing the system prompt. Our experimental evaluations indicate that CST successfully manages different safety configurations and retains the original functionality of LLMs, showing it is a robust method for configurable deployment. Data and models available at https://github.com/vicgalle/configurable-safety-tuning

Title: Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models

• Authors: Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Chen, Miao Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00506
• Pdf URL: https://arxiv.org/pdf/2404.00506
• Copy Paste: [[2404.00506]] Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models(https://arxiv.org/abs/2404.00506)
• Keywords: privacy
• Abstract: Machine unlearning aims to remove information derived from forgotten data while preserving that of the remaining dataset in a well-trained model. With the increasing emphasis on data privacy, several approaches to machine unlearning have emerged. However, these methods typically rely on complete supervision throughout the unlearning process. Unfortunately, obtaining such supervision, whether for the forgetting or remaining data, can be impractical due to the substantial cost associated with annotating real-world datasets. This challenge prompts us to propose a supervision-free unlearning approach that operates without the need for labels during the unlearning process. Specifically, we introduce a variational approach to approximate the distribution of representations for the remaining data. Leveraging this approximation, we adapt the original model to eliminate information from the forgotten data at the representation level. To further address the issue of lacking supervision information, which hinders alignment with ground truth, we introduce a contrastive loss to facilitate the matching of representations between the remaining data and those of the original model, thus preserving predictive performance. Experimental results across various unlearning tasks demonstrate the effectiveness of our proposed method, Label-Agnostic Forgetting (LAF) without using any labels, which achieves comparable performance to state-of-the-art methods that rely on full supervision information. Furthermore, our approach excels in semi-supervised scenarios, leveraging limited supervision information to outperform fully supervised baselines. This work not only showcases the viability of supervision-free unlearning in deep models but also opens up a new possibility for future research in unlearning at the representation level.

Title: MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Models

• Authors: Zebang Cheng, Fuqiang Niu, Yuxiang Lin, Zhi-Qi Cheng, Bowen Zhang, Xiaojiang Peng
• Subjects: cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.00511
• Pdf URL: https://arxiv.org/pdf/2404.00511
• Copy Paste: [[2404.00511]] MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Models(https://arxiv.org/abs/2404.00511)
• Keywords: extraction
• Abstract: This paper presents our winning submission to Subtask 2 of SemEval 2024 Task 3 on multimodal emotion cause analysis in conversations. We propose a novel Multimodal Emotion Recognition and Multimodal Emotion Cause Extraction (MER-MCE) framework that integrates text, audio, and visual modalities using specialized emotion encoders. Our approach sets itself apart from top-performing teams by leveraging modality-specific features for enhanced emotion understanding and causality inference. Experimental evaluation demonstrates the advantages of our multimodal approach, with our submission achieving a competitive weighted F1 score of 0.3435, ranking third with a margin of only 0.0339 behind the 1st team and 0.0025 behind the 2nd team. Project: https://github.com/MIPS-COLT/MER-MCE.git

Title: Transformer based Pluralistic Image Completion with Reduced Information Loss

• Authors: Qiankun Liu, Yuqi Jiang, Zhentao Tan, Dongdong Chen, Ying Fu, Qi Chu, Gang Hua, Nenghai Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00513
• Pdf URL: https://arxiv.org/pdf/2404.00513
• Copy Paste: [[2404.00513]] Transformer based Pluralistic Image Completion with Reduced Information Loss(https://arxiv.org/abs/2404.00513)
• Keywords: transformer
• Abstract: Transformer based methods have achieved great success in image inpainting recently. However, we find that these solutions regard each pixel as a token, thus suffering from an information loss issue from two aspects: 1) They downsample the input image into much lower resolutions for efficiency consideration. 2) They quantize $256^3$ RGB values to a small number (such as 512) of quantized color values. The indices of quantized pixels are used as tokens for the inputs and prediction targets of the transformer. To mitigate these issues, we propose a new transformer based framework called "PUT". Specifically, to avoid input downsampling while maintaining computation efficiency, we design a patch-based auto-encoder P-VQVAE. The encoder converts the masked image into non-overlapped patch tokens and the decoder recovers the masked regions from the inpainted tokens while keeping the unmasked regions unchanged. To eliminate the information loss caused by input quantization, an Un-quantized Transformer is applied. It directly takes features from the P-VQVAE encoder as input without any quantization and only regards the quantized tokens as prediction targets. Furthermore, to make the inpainting process more controllable, we introduce semantic and structural conditions as extra guidance. Extensive experiments show that our method greatly outperforms existing transformer based methods on image fidelity and achieves much higher diversity and better fidelity than state-of-the-art pluralistic inpainting methods on complex large-scale datasets (e.g., ImageNet). Codes are available at https://github.com/liuqk3/PUT.

Title: CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalization

• Authors: Yao Ni, Piotr Koniusz
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00521
• Pdf URL: https://arxiv.org/pdf/2404.00521
• Copy Paste: [[2404.00521]] CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalization(https://arxiv.org/abs/2404.00521)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) significantly advanced image generation but their performance heavily depends on abundant training data. In scenarios with limited data, GANs often struggle with discriminator overfitting and unstable training. Batch Normalization (BN), despite being known for enhancing generalization and training stability, has rarely been used in the discriminator of Data-Efficient GANs. Our work addresses this gap by identifying a critical flaw in BN: the tendency for gradient explosion during the centering and scaling steps. To tackle this issue, we present CHAIN (lipsCHitz continuity constrAIned Normalization), which replaces the conventional centering step with zero-mean regularization and integrates a Lipschitz continuity constraint in the scaling step. CHAIN further enhances GAN training by adaptively interpolating the normalized and unnormalized features, effectively avoiding discriminator overfitting. Our theoretical analyses firmly establishes CHAIN's effectiveness in reducing gradients in latent features and weights, improving stability and generalization in GAN training. Empirical evidence supports our theory. CHAIN achieves state-of-the-art results in data-limited scenarios on CIFAR-10/100, ImageNet, five low-shot and seven high-resolution few-shot image datasets.

Title: Creating synthetic energy meter data using conditional diffusion and building metadata

• Authors: Chun Fu, Hussain Kazmi, Matias Quintana, Clayton Miller
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2404.00525
• Pdf URL: https://arxiv.org/pdf/2404.00525
• Copy Paste: [[2404.00525]] Creating synthetic energy meter data using conditional diffusion and building metadata(https://arxiv.org/abs/2404.00525)
• Keywords: diffusion, generative
• Abstract: Advances in machine learning and increased computational power have driven progress in energy-related research. However, limited access to private energy data from buildings hinders traditional regression models relying on historical data. While generative models offer a solution, previous studies have primarily focused on short-term generation periods (e.g., daily profiles) and a limited number of meters. Thus, the study proposes a conditional diffusion model for generating high-quality synthetic energy data using relevant metadata. Using a dataset comprising 1,828 power meters from various buildings and countries, this model is compared with traditional methods like Conditional Generative Adversarial Networks (CGAN) and Conditional Variational Auto-Encoders (CVAE). It explicitly handles long-term annual consumption profiles, harnessing metadata such as location, weather, building, and meter type to produce coherent synthetic data that closely resembles real-world energy consumption patterns. The results demonstrate the proposed diffusion model's superior performance, with a 36% reduction in Frechet Inception Distance (FID) score and a 13% decrease in Kullback-Leibler divergence (KL divergence) compared to the following best method. The proposed method successfully generates high-quality energy data through metadata, and its code will be open-sourced, establishing a foundation for a broader array of energy data generation models in the future.

Title: Generative weather for improved crop model simulations

• Authors: Yuji Saikai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00528
• Pdf URL: https://arxiv.org/pdf/2404.00528
• Copy Paste: [[2404.00528]] Generative weather for improved crop model simulations(https://arxiv.org/abs/2404.00528)
• Keywords: generative
• Abstract: Accurate and precise crop yield prediction is invaluable for decision making at both farm levels and regional levels. To make yield prediction, crop models are widely used for their capability to simulate hypothetical scenarios. While accuracy and precision of yield prediction critically depend on weather inputs to simulations, surprisingly little attention has been paid to preparing weather inputs. We propose a new method to construct generative models for long-term weather forecasts and ultimately improve crop yield prediction. We demonstrate use of the method in two representative scenarios -- single-year production of wheat, barley and canola and three-year production using rotations of these crops. Results show significant improvement from the conventional method, measured in terms of mean and standard deviation of prediction errors. Our method outperformed the conventional method in every one of 18 metrics for the first scenario and in 29 out of 36 metrics for the second scenario. For individual crop modellers to start applying the method to their problems, technical details are carefully explained, and all the code, trained PyTorch models, APSIM simulation files and result data are made available.

Title: Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

• Authors: Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, Aditya Grover
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00530
• Pdf URL: https://arxiv.org/pdf/2404.00530
• Copy Paste: [[2404.00530]] Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization(https://arxiv.org/abs/2404.00530)
• Keywords: large language model
• Abstract: A common technique for aligning large language models (LLMs) relies on acquiring human preferences by comparing multiple generations conditioned on a fixed context. This only leverages the pairwise comparisons when the generations are placed in an identical context. However, such conditional rankings often fail to capture the complex and multidimensional aspects of human preferences. In this work, we revisit the traditional paradigm of preference acquisition and propose a new axis that is based on eliciting preferences jointly over the instruction-response pairs. While prior preference optimizations are designed for conditional ranking protocols (e.g., DPO), our proposed preference acquisition protocol introduces DOVE, a new preference optimization objective that upweights the joint probability of the chosen instruction-response pair over the rejected instruction-response pair. Interestingly, we find that the LLM trained with joint instruction-response preference data using DOVE outperforms the LLM trained with DPO by 5.2% and 3.3% win-rate for the summarization and open-ended dialogue datasets, respectively. Our findings reveal that joint preferences over instruction and response pairs can significantly enhance the alignment of LLMs by tapping into a broader spectrum of human preference elicitation. The data and code is available at https://github.com/Hritikbansal/dove.

Title: LLMs are Good Action Recognizers

• Authors: Haoxuan Qu, Yujun Cai, Jun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00532
• Pdf URL: https://arxiv.org/pdf/2404.00532
• Copy Paste: [[2404.00532]] LLMs are Good Action Recognizers(https://arxiv.org/abs/2404.00532)
• Keywords: large language model
• Abstract: Skeleton-based action recognition has attracted lots of research attention. Recently, to build an accurate skeleton-based action recognizer, a variety of works have been proposed. Among them, some works use large model architectures as backbones of their recognizers to boost the skeleton data representation capability, while some other works pre-train their recognizers on external data to enrich the knowledge. In this work, we observe that large language models which have been extensively used in various natural language processing tasks generally hold both large model architectures and rich implicit knowledge. Motivated by this, we propose a novel LLM-AR framework, in which we investigate treating the Large Language Model as an Action Recognizer. In our framework, we propose a linguistic projection process to project each input action signal (i.e., each skeleton sequence) into its ``sentence format'' (i.e., an ``action sentence''). Moreover, we also incorporate our framework with several designs to further facilitate this linguistic projection process. Extensive experiments demonstrate the efficacy of our proposed framework.

Title: Eclipse Attack Detection on a Blockchain Network as a Non-Parametric Change Detection Problem

• Authors: Anurag Gupta, Brian Sadler
• Subjects: cs.CR, stat.AP
• Abstract URL: https://arxiv.org/abs/2404.00538
• Pdf URL: https://arxiv.org/pdf/2404.00538
• Copy Paste: [[2404.00538]] Eclipse Attack Detection on a Blockchain Network as a Non-Parametric Change Detection Problem(https://arxiv.org/abs/2404.00538)
• Keywords: attack
• Abstract: This paper introduces a novel non-parametric change detection algorithm to identify eclipse attacks on a blockchain network; the non-parametric algorithm relies only on the empirical mean and variance of the dataset, making it highly adaptable. An eclipse attack occurs when malicious actors isolate blockchain users, disrupting their ability to reach consensus with the broader network, thereby distorting their local copy of the ledger. To detect an eclipse attack, we monitor changes in the Fr\'echet mean and variance of the evolving blockchain communication network connecting blockchain users. First, we leverage the Johnson-Lindenstrauss lemma to project large-dimensional networks into a lower-dimensional space, preserving essential statistical properties. Subsequently, we employ a non-parametric change detection procedure, leading to a test statistic that converges weakly to a Brownian bridge process in the absence of an eclipse attack. This enables us to quantify the false alarm rate of the detector. Our detector can be implemented as a smart contract on the blockchain, offering a tamper-proof and reliable solution. Finally, we use numerical examples to compare the proposed eclipse attack detector with a detector based on the random forest model.

Title: Embodied Active Defense: Leveraging Recurrent Feedback to Counter Adversarial Patches

• Authors: Lingxuan Wu, Xiao Yang, Yinpeng Dong, Liuwei Xie, Hang Su, Jun Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00540
• Pdf URL: https://arxiv.org/pdf/2404.00540
• Copy Paste: [[2404.00540]] Embodied Active Defense: Leveraging Recurrent Feedback to Counter Adversarial Patches(https://arxiv.org/abs/2404.00540)
• Keywords: defense, attack, robust
• Abstract: The vulnerability of deep neural networks to adversarial patches has motivated numerous defense strategies for boosting model robustness. However, the prevailing defenses depend on single observation or pre-established adversary information to counter adversarial patches, often failing to be confronted with unseen or adaptive adversarial attacks and easily exhibiting unsatisfying performance in dynamic 3D environments. Inspired by active human perception and recurrent feedback mechanisms, we develop Embodied Active Defense (EAD), a proactive defensive strategy that actively contextualizes environmental information to address misaligned adversarial patches in 3D real-world settings. To achieve this, EAD develops two central recurrent sub-modules, i.e., a perception module and a policy module, to implement two critical functions of active vision. These models recurrently process a series of beliefs and observations, facilitating progressive refinement of their comprehension of the target object and enabling the development of strategic actions to counter adversarial patches in 3D environments. To optimize learning efficiency, we incorporate a differentiable approximation of environmental dynamics and deploy patches that are agnostic to the adversary strategies. Extensive experiments demonstrate that EAD substantially enhances robustness against a variety of patches within just a few steps through its action policy in safety-critical tasks (e.g., face recognition and object detection), without compromising standard accuracy. Furthermore, due to the attack-agnostic characteristic, EAD facilitates excellent generalization to unseen attacks, diminishing the averaged attack success rate by 95 percent across a range of unseen adversarial attacks.

Title: Denoising Distillation Makes Event-Frame Transformers as Accurate Gaze Trackers

• Authors: Jiading Li, Zhiyu Zhu, Jinhui Hou, Junhui Hou, Jinjian Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00548
• Pdf URL: https://arxiv.org/pdf/2404.00548
• Copy Paste: [[2404.00548]] Denoising Distillation Makes Event-Frame Transformers as Accurate Gaze Trackers(https://arxiv.org/abs/2404.00548)
• Keywords: diffusion, transformer
• Abstract: This paper tackles the problem of passive gaze estimation using both event and frame data. Considering inherently different physiological structures, it's intractable to accurately estimate purely based on a given state. Thus, we reformulate the gaze estimation as the quantification of state transitions from the current state to several prior registered anchor states. Technically, we propose a two-stage learning-based gaze estimation framework to divide the whole gaze estimation process into a coarse-to-fine process of anchor state selection and final gaze location. Moreover, to improve generalization ability, we align a group of local experts with a student network, where a novel denoising distillation algorithm is introduced to utilize denoising diffusion technique to iteratively remove inherent noise of event data. Extensive experiments demonstrate the effectiveness of the proposed method, which greatly surpasses state-of-the-art methods by a large extent of 15$\%$. The code will be publicly available at https://github.com/jdjdli/Denoise_distill_EF_gazetracker.

Title: Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction

• Authors: Junuk Cha, Jihyeon Kim, Jae Shin Yoon, Seungryul Baek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00562
• Pdf URL: https://arxiv.org/pdf/2404.00562
• Copy Paste: [[2404.00562]] Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction(https://arxiv.org/abs/2404.00562)
• Keywords: diffusion, transformer
• Abstract: This paper introduces the first text-guided work for generating the sequence of hand-object interaction in 3D. The main challenge arises from the lack of labeled data where existing ground-truth datasets are nowhere near generalizable in interaction type and object category, which inhibits the modeling of diverse 3D hand-object interaction with the correct physical implication (e.g., contacts and semantics) from text prompts. To address this challenge, we propose to decompose the interaction generation task into two subtasks: hand-object contact generation; and hand-object motion generation. For contact generation, a VAE-based network takes as input a text and an object mesh, and generates the probability of contacts between the surfaces of hands and the object during the interaction. The network learns a variety of local geometry structure of diverse objects that is independent of the objects' category, and thus, it is applicable to general objects. For motion generation, a Transformer-based diffusion model utilizes this 3D contact map as a strong prior for generating physically plausible hand-object motion as a function of text prompts by learning from the augmented labeled dataset; where we annotate text labels from many existing 3D hand and object motion data. Finally, we further introduce a hand refiner module that minimizes the distance between the object surface and hand joints to improve the temporal stability of the object-hand contacts and to suppress the penetration artifacts. In the experiments, we demonstrate that our method can generate more realistic and diverse interactions compared to other baseline methods. We also show that our method is applicable to unseen objects. We will release our model and newly labeled data as a strong foundation for future research. Codes and data are available in: https://github.com/JunukCha/Text2HOI.

Title: Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation

• Authors: Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, Yang Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00563
• Pdf URL: https://arxiv.org/pdf/2404.00563
• Copy Paste: [[2404.00563]] Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation(https://arxiv.org/abs/2404.00563)
• Keywords: robust
• Abstract: Dataset distillation has emerged as a promising approach in deep learning, enabling efficient training with small synthetic datasets derived from larger real ones. Particularly, distribution matching-based distillation methods attract attention thanks to its effectiveness and low computational cost. However, these methods face two primary limitations: the dispersed feature distribution within the same class in synthetic datasets, reducing class discrimination, and an exclusive focus on mean feature consistency, lacking precision and comprehensiveness. To address these challenges, we introduce two novel constraints: a class centralization constraint and a covariance matching constraint. The class centralization constraint aims to enhance class discrimination by more closely clustering samples within classes. The covariance matching constraint seeks to achieve more accurate feature distribution matching between real and synthetic datasets through local feature covariance matrices, particularly beneficial when sample sizes are much smaller than the number of features. Experiments demonstrate notable improvements with these constraints, yielding performance boosts of up to 6.6% on CIFAR10, 2.9% on SVHN, 2.5% on CIFAR100, and 2.5% on TinyImageNet, compared to the state-of-the-art relevant methods. In addition, our method maintains robust performance in cross-architecture settings, with a maximum performance drop of 1.7% on four architectures. Code is available at https://github.com/VincenDen/IID.

Title: ParaICL: Towards Robust Parallel In-Context Learning

• Authors: Xingxuan Li, Xuan-Phi Nguyen, Shafiq Joty, Lidong Bing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00570
• Pdf URL: https://arxiv.org/pdf/2404.00570
• Copy Paste: [[2404.00570]] ParaICL: Towards Robust Parallel In-Context Learning(https://arxiv.org/abs/2404.00570)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have become the norm in natural language processing (NLP), excelling in few-shot in-context learning (ICL) with their remarkable abilities. Nonetheless, the success of ICL largely hinges on the choice of few-shot demonstration examples, making the selection process increasingly crucial. Existing methods have delved into optimizing the quantity and semantic similarity of these examples to improve ICL performances. However, our preliminary experiments indicate that the effectiveness of ICL is limited by the length of the input context. Moreover, varying combinations of few-shot demonstration examples can significantly boost accuracy across different test samples. To address this, we propose a novel method named parallel in-context learning (ParaICL) that effectively utilizes all demonstration examples without exceeding the manageable input context length. ParaICL employs parallel batching to distribute demonstration examples into different batches according to the semantic similarities of the questions in the demonstrations to the test question. It then computes normalized batch semantic scores for each batch. A weighted average semantic objective, constrained by adaptive plausibility, is applied to select the most appropriate tokens. Through extensive experiments, we validate the effectiveness of ParaICL and conduct ablation studies to underscore its design rationale. We further demonstrate that ParaICL can seamlessly integrate with existing methods.

Title: M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models

• Authors: Fan Bai, Yuxin Du, Tiejun Huang, Max Q.-H. Meng, Bo Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00578
• Pdf URL: https://arxiv.org/pdf/2404.00578
• Copy Paste: [[2404.00578]] M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models(https://arxiv.org/abs/2404.00578)
• Keywords: robust, large language model, segmentation
• Abstract: Medical image analysis is essential to clinical diagnosis and treatment, which is increasingly supported by multi-modal large language models (MLLMs). However, previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information. This paper aims to advance 3D medical image analysis with MLLMs. To this end, we present a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs specifically tailored for various 3D medical tasks, such as image-text retrieval, report generation, visual question answering, positioning, and segmentation. Additionally, we propose M3D-LaMed, a versatile multi-modal large language model for 3D medical image analysis. Furthermore, we introduce a new 3D multi-modal medical benchmark, M3D-Bench, which facilitates automatic evaluation across eight tasks. Through comprehensive evaluation, our method proves to be a robust model for 3D medical image analysis, outperforming existing solutions. All code, data, and models are publicly available at: https://github.com/BAAI-DCAI/M3D.

Title: Harnessing the Power of Large Language Model for Uncertainty Aware Graph Processing

• Authors: Zhenyu Qian, Yiming Qian, Yuting Song, Fei Gao, Hai Jin, Chen Yu, Xia Xie
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00589
• Pdf URL: https://arxiv.org/pdf/2404.00589
• Copy Paste: [[2404.00589]] Harnessing the Power of Large Language Model for Uncertainty Aware Graph Processing(https://arxiv.org/abs/2404.00589)
• Keywords: explainability, large language model
• Abstract: Handling graph data is one of the most difficult tasks. Traditional techniques, such as those based on geometry and matrix factorization, rely on assumptions about the data relations that become inadequate when handling large and complex graph data. On the other hand, deep learning approaches demonstrate promising results in handling large graph data, but they often fall short of providing interpretable explanations. To equip the graph processing with both high accuracy and explainability, we introduce a novel approach that harnesses the power of a large language model (LLM), enhanced by an uncertainty-aware module to provide a confidence score on the generated answer. We experiment with our approach on two graph processing tasks: few-shot knowledge graph completion and graph classification. Our results demonstrate that through parameter efficient fine-tuning, the LLM surpasses state-of-the-art algorithms by a substantial margin across ten diverse benchmark datasets. Moreover, to address the challenge of explainability, we propose an uncertainty estimation based on perturbation, along with a calibration scheme to quantify the confidence scores of the generated answers. Our confidence measure achieves an AUC of 0.8 or higher on seven out of the ten datasets in predicting the correctness of the answer generated by LLM.

Title: LAESI: Leaf Area Estimation with Synthetic Imagery

• Authors: Jacek Kałużny, Yannik Schreckenberg, Karol Cyganik, Peter Annighöfer, Sören Pirk, Dominik L. Michels, Mikolaj Cieslak, Farhah Assaad-Gerbert, Bedrich Benes, Wojciech Pałubicki
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00593
• Pdf URL: https://arxiv.org/pdf/2404.00593
• Copy Paste: [[2404.00593]] LAESI: Leaf Area Estimation with Synthetic Imagery(https://arxiv.org/abs/2404.00593)
• Keywords: generative, segmentation
• Abstract: We introduce LAESI, a Synthetic Leaf Dataset of 100,000 synthetic leaf images on millimeter paper, each with semantic masks and surface area labels. This dataset provides a resource for leaf morphology analysis primarily aimed at beech and oak leaves. We evaluate the applicability of the dataset by training machine learning models for leaf surface area prediction and semantic segmentation, using real images for validation. Our validation shows that these models can be trained to predict leaf surface area with a relative error not greater than an average human annotator. LAESI also provides an efficient framework based on 3D procedural models and generative AI for the large-scale, controllable generation of data with potential further applications in agriculture and biology. We evaluate the inclusion of generative AI in our procedural data generation pipeline and show how data filtering based on annotation consistency results in datasets which allow training the highest performing vision models.

Title: Query-driven Relevant Paragraph Extraction from Legal Judgments

• Authors: T.Y.S.S Santosh, Elvin Quero Hernandez, Matthias Grabmair
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.00595
• Pdf URL: https://arxiv.org/pdf/2404.00595
• Copy Paste: [[2404.00595]] Query-driven Relevant Paragraph Extraction from Legal Judgments(https://arxiv.org/abs/2404.00595)
• Keywords: extraction
• Abstract: Legal professionals often grapple with navigating lengthy legal judgements to pinpoint information that directly address their queries. This paper focus on this task of extracting relevant paragraphs from legal judgements based on the query. We construct a specialized dataset for this task from the European Court of Human Rights (ECtHR) using the case law guides. We assess the performance of current retrieval models in a zero-shot way and also establish fine-tuning benchmarks using various models. The results highlight the significant gap between fine-tuned and zero-shot performance, emphasizing the challenge of handling distribution shift in the legal domain. We notice that the legal pre-training handles distribution shift on the corpus side but still struggles on query side distribution shift, with unseen legal queries. We also explore various Parameter Efficient Fine-Tuning (PEFT) methods to evaluate their practicality within the context of information retrieval, shedding light on the effectiveness of different PEFT methods across diverse configurations with pre-training and model architectures influencing the choice of PEFT method.

Title: EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories

• Authors: Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, Zhi Jin
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.00599
• Pdf URL: https://arxiv.org/pdf/2404.00599
• Copy Paste: [[2404.00599]] EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories(https://arxiv.org/abs/2404.00599)
• Keywords: robust, large language model
• Abstract: How to evaluate Large Language Models (LLMs) in code generation is an open question. Existing benchmarks demonstrate poor alignment with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. This paper proposes a new benchmark - EvoCodeBench to address the preceding problems, which has three primary advances. (1) EvoCodeBench aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) EvoCodeBench offers comprehensive annotations (e.g., requirements, reference code, and reference dependencies), and robust evaluation metrics (e.g., Pass@k and Recall@k). (3) EvoCodeBench is an evolving benchmark to avoid data leakage. We build an automatic pipeline to update EvoCodeBench from the latest repositories. We release the first version - EvoCodeBench-2403, containing 275 samples from 25 real-world repositories. Based on EvoCodeBench, we propose repository-level code generation and evaluate 10 popular LLMs (e.g., gpt-4, gpt-3.5, DeepSeek Coder, StarCoder 2, CodeLLaMa, Gemma, and Qwen 1.5). Our experiments reveal the coding abilities of these LLMs in real-world repositories. For example, the highest Pass@1 of gpt-4 only is 20.73% in our experiments. We also analyze failed cases and summarize the shortcomings of existing LLMs in EvoCodeBench. We release EvoCodeBench, all prompts, and LLMs' completions for further community analysis.

Title: 1-out-of-n Oblivious Signatures: Security Revisited and a Generic Construction with an Efficient Communication Cost

• Authors: Masayuki Tezuka, Keisuke Tanaka
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.00602
• Pdf URL: https://arxiv.org/pdf/2404.00602
• Copy Paste: [[2404.00602]] 1-out-of-n Oblivious Signatures: Security Revisited and a Generic Construction with an Efficient Communication Cost(https://arxiv.org/abs/2404.00602)
• Keywords: security
• Abstract: 1-out-of-n oblivious signature by Chen (ESORIC 1994) is a protocol between the user and the signer. In this scheme, the user makes a list of n messages and chooses the message that the user wants to obtain a signature from the list. The user interacts with the signer by providing this message list and obtains the signature for only the chosen message without letting the signer identify which messages the user chooses. Tso et al. (ISPEC 2008) presented a formal treatment of 1-out-of-n oblivious signatures. They defined unforgeability and ambiguity for 1-out-of-n oblivious signatures as a security requirement. In this work, first, we revisit the unforgeability security definition by Tso et al. and point out that their security definition has problems. We address these problems by modifying their security model and redefining unforgeable security. Second, we improve the generic construction of a 1-out-of-n oblivious signature scheme by Zhou et al. (IEICE Trans 2022). We reduce the communication cost by modifying their scheme with a Merkle tree. Then we prove the security of our modified scheme.

Title: Extensive Self-Contrast Enables Feedback-Free Language Model Alignment

• Authors: Xiao Liu, Xixuan Song, Yuxiao Dong, Jie Tang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00604
• Pdf URL: https://arxiv.org/pdf/2404.00604
• Copy Paste: [[2404.00604]] Extensive Self-Contrast Enables Feedback-Free Language Model Alignment(https://arxiv.org/abs/2404.00604)
• Keywords: large language model
• Abstract: Reinforcement learning from human feedback (RLHF) has been a central technique for recent large language model (LLM) alignment. However, its heavy dependence on costly human or LLM-as-Judge preference feedback could stymie its wider applications. In this work, we introduce Self-Contrast, a feedback-free large language model alignment method via exploiting extensive self-generated negatives. With only supervised fine-tuning (SFT) targets, Self-Contrast leverages the LLM itself to generate massive diverse candidates, and harnesses a pre-trained embedding model to filter multiple negatives according to text similarity. Theoretically, we illustrate that in this setting, merely scaling negative responses can still effectively approximate situations with more balanced positive and negative preference annotations. Our experiments with direct preference optimization (DPO) on three datasets show that, Self-Contrast could consistently outperform SFT and standard DPO training by large margins. And as the number of self-generated negatives increases, the performance of Self-Contrast continues to grow. Code and data are available at https://github.com/THUDM/Self-Contrast.

Title: RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation

• Authors: Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, Jie Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00610
• Pdf URL: https://arxiv.org/pdf/2404.00610
• Copy Paste: [[2404.00610]] RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation(https://arxiv.org/abs/2404.00610)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit remarkable capabilities but are prone to generating inaccurate or hallucinatory responses. This limitation stems from their reliance on vast pretraining datasets, making them susceptible to errors in unseen scenarios. To tackle these challenges, Retrieval-Augmented Generation (RAG) addresses this by incorporating external, relevant documents into the response generation process, thus leveraging non-parametric knowledge alongside LLMs' in-context learning abilities. However, existing RAG implementations primarily focus on initial input for context retrieval, overlooking the nuances of ambiguous or complex queries that necessitate further clarification or decomposition for accurate responses. To this end, we propose learning to Refine Query for Retrieval Augmented Generation (RQ-RAG) in this paper, endeavoring to enhance the model by equipping it with capabilities for explicit rewriting, decomposition, and disambiguation. Our experimental results indicate that our method, when applied to a 7B Llama2 model, surpasses the previous state-of-the-art (SOTA) by an average of 1.9\% across three single-hop QA datasets, and also demonstrates enhanced performance in handling complex, multi-hop QA datasets. Our code is available at https://github.com/chanchimin/RQ-RAG.

Title: Object-level Copy-Move Forgery Image Detection based on Inconsistency Mining

• Authors: Jingyu Wang, Niantai Jing, Ziyao Liu, Jie Nie, Yuxin Qi, Chi-Hung Chi, Kwok-Yan Lam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00611
• Pdf URL: https://arxiv.org/pdf/2404.00611
• Copy Paste: [[2404.00611]] Object-level Copy-Move Forgery Image Detection based on Inconsistency Mining(https://arxiv.org/abs/2404.00611)
• Keywords: robust
• Abstract: In copy-move tampering operations, perpetrators often employ techniques, such as blurring, to conceal tampering traces, posing significant challenges to the detection of object-level targets with intact structures. Focus on these challenges, this paper proposes an Object-level Copy-Move Forgery Image Detection based on Inconsistency Mining (IMNet). To obtain complete object-level targets, we customize prototypes for both the source and tampered regions and dynamically update them. Additionally, we extract inconsistent regions between coarse similar regions obtained through self-correlation calculations and regions composed of prototypes. The detected inconsistent regions are used as supplements to coarse similar regions to refine pixel-level detection. We operate experiments on three public datasets which validate the effectiveness and the robustness of the proposed IMNet.

Title: Learning to Plan for Language Modeling from Unlabeled Data

• Authors: Nathan Cornille, Marie-Francine Moens, Florian Mai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00614
• Pdf URL: https://arxiv.org/pdf/2404.00614
• Copy Paste: [[2404.00614]] Learning to Plan for Language Modeling from Unlabeled Data(https://arxiv.org/abs/2404.00614)
• Keywords: large language model
• Abstract: By training to predict the next token in an unlabeled corpus, large language models learn to perform many tasks without any labeled data. However, their next-token-prediction objective arguably limits their performance in scenarios that require planning, such as writing a coherent article. In this paper, we train a module for planning the future writing process via a self-supervised learning objective. By conditioning on generated latent plans, our model extends the successful language model formula to more abstract planning in an unsupervised way. Empirically, we demonstrate that our method improves language modeling performance in general, particularly with respect to the text structure. Because our framework uses a planner module that is unsupervised and external to the language model, new planner modules can be trained at large scale and easily be shared with the community.

Title: Domain Generalizable Person Search Using Unreal Dataset

• Authors: Minyoung Oh, Duhyun Kim, Jae-Young Sim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00626
• Pdf URL: https://arxiv.org/pdf/2404.00626
• Copy Paste: [[2404.00626]] Domain Generalizable Person Search Using Unreal Dataset(https://arxiv.org/abs/2404.00626)
• Keywords: privacy
• Abstract: Collecting and labeling real datasets to train the person search networks not only requires a lot of time and effort, but also accompanies privacy issues. The weakly-supervised and unsupervised domain adaptation methods have been proposed to alleviate the labeling burden for target datasets, however, their generalization capability is limited. We introduce a novel person search method based on the domain generalization framework, that uses an automatically labeled unreal dataset only for training but is applicable to arbitrary unseen real datasets. To alleviate the domain gaps when transferring the knowledge from the unreal source dataset to the real target datasets, we estimate the fidelity of person instances which is then used to train the end-to-end network adaptively. Moreover, we devise a domain-invariant feature learning scheme to encourage the network to suppress the domain-related features. Experimental results demonstrate that the proposed method provides the competitive performance to existing person search methods even though it is applicable to arbitrary unseen datasets without any prior knowledge and re-training burdens.

Title: Against The Achilles' Heel: A Survey on Red Teaming for Generative Models

• Authors: Lizhi Lin, Honglin Mu, Zenan Zhai, Minghan Wang, Yuxia Wang, Renxi Wang, Junjie Gao, Yixuan Zhang, Wanxiang Che, Timothy Baldwin, Xudong Han, Haonan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00629
• Pdf URL: https://arxiv.org/pdf/2404.00629
• Copy Paste: [[2404.00629]] Against The Achilles' Heel: A Survey on Red Teaming for Generative Models(https://arxiv.org/abs/2404.00629)
• Keywords: defense, attack, generative
• Abstract: Generative models are rapidly gaining popularity and being integrated into everyday applications, raising concerns over their safety issues as various vulnerabilities are exposed. Faced with the problem, the field of red teaming is experiencing fast-paced growth, which highlights the need for a comprehensive organization covering the entire pipeline and addressing emerging topics for the community. Our extensive survey, which examines over 120 papers, introduces a taxonomy of fine-grained attack strategies grounded in the inherent capabilities of language models. Additionally, we have developed the searcher framework that unifies various automatic red teaming approaches. Moreover, our survey covers novel areas including multimodal attacks and defenses, risks around multilingual models, overkill of harmless queries, and safety of downstream applications. We hope this survey can provide a systematic perspective on the field and unlock new areas of research.

Title: IPT-V2: Efficient Image Processing Transformer using Hierarchical Attentions

• Authors: Zhijun Tu, Kunpeng Du, Hanting Chen, Hailing Wang, Wei Li, Jie Hu, Yunhe Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00633
• Pdf URL: https://arxiv.org/pdf/2404.00633
• Copy Paste: [[2404.00633]] IPT-V2: Efficient Image Processing Transformer using Hierarchical Attentions(https://arxiv.org/abs/2404.00633)
• Keywords: diffusion, transformer
• Abstract: Recent advances have demonstrated the powerful capability of transformer architecture in image restoration. However, our analysis indicates that existing transformerbased methods can not establish both exact global and local dependencies simultaneously, which are much critical to restore the details and missing content of degraded images. To this end, we present an efficient image processing transformer architecture with hierarchical attentions, called IPTV2, adopting a focal context self-attention (FCSA) and a global grid self-attention (GGSA) to obtain adequate token interactions in local and global receptive fields. Specifically, FCSA applies the shifted window mechanism into the channel self-attention, helps capture the local context and mutual interaction across channels. And GGSA constructs long-range dependencies in the cross-window grid, aggregates global information in spatial dimension. Moreover, we introduce structural re-parameterization technique to feed-forward network to further improve the model capability. Extensive experiments demonstrate that our proposed IPT-V2 achieves state-of-the-art results on various image processing tasks, covering denoising, deblurring, deraining and obtains much better trade-off for performance and computational complexity than previous methods. Besides, we extend our method to image generation as latent diffusion backbone, and significantly outperforms DiTs.

Title: HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphs

• Authors: Sunwoo Kim, Shinhwan Kang, Fanchen Bu, Soo Yong Lee, Jaemin Yoo, Kijung Shin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00638
• Pdf URL: https://arxiv.org/pdf/2404.00638
• Copy Paste: [[2404.00638]] HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphs(https://arxiv.org/abs/2404.00638)
• Keywords: generative
• Abstract: Hypergraphs are marked by complex topology, expressing higher-order interactions among multiple nodes with hyperedges, and better capturing the topology is essential for effective representation learning. Recent advances in generative self-supervised learning (SSL) suggest that hypergraph neural networks learned from generative self supervision have the potential to effectively encode the complex hypergraph topology. Designing a generative SSL strategy for hypergraphs, however, is not straightforward. Questions remain with regard to its generative SSL task, connection to downstream tasks, and empirical properties of learned representations. In light of the promises and challenges, we propose a novel generative SSL strategy for hypergraphs. We first formulate a generative SSL task on hypergraphs, hyperedge filling, and highlight its theoretical connection to node classification. Based on the generative SSL task, we propose a hypergraph SSL method, HypeBoy. HypeBoy learns effective general-purpose hypergraph representations, outperforming 16 baseline methods across 11 benchmark datasets.

Title: SoK: Liquid Staking Tokens (LSTs)

• Authors: Krzysztof Gogol, Yaron Velner, Benjamin Kraner, Claudio Tessone
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.00644
• Pdf URL: https://arxiv.org/pdf/2404.00644
• Copy Paste: [[2404.00644]] SoK: Liquid Staking Tokens (LSTs)(https://arxiv.org/abs/2404.00644)
• Keywords: security
• Abstract: Liquid Staking Tokens (LSTs) function as tokenized representations of staked native assets while also accruing staking rewards. They emerged as a preferred method of staking within Proof of Stake (PoS) blockchains, owing to their ease of use and tradability. In this Systematization of Knowledge (SoK), we establish a general framework describing the design choices and protocols underlying liquid staking. We then employ the framework to systematically compare the top LST implementations, examining their node operator selection, validator operations, and staking rewards distribution models. We further discuss security concerns associated with liquid staking, its implications for PoS blockchain security, and Distributed Validator technology (DVT) as a potential solution. Finally, we empirically analyze LSTs' performance and find that the design choices and market events affect peg stability; particularly, LSTs with centralized governance and operations are more efficient in tracking staking rewards.

Title: Attire-Based Anomaly Detection in Restricted Areas Using YOLOv8 for Enhanced CCTV Security

• Authors: Abdul Aziz A.B, Aindri Bajpai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00645
• Pdf URL: https://arxiv.org/pdf/2404.00645
• Copy Paste: [[2404.00645]] Attire-Based Anomaly Detection in Restricted Areas Using YOLOv8 for Enhanced CCTV Security(https://arxiv.org/abs/2404.00645)
• Keywords: security, robust
• Abstract: This research introduces an innovative security enhancement approach, employing advanced image analysis and soft computing. The focus is on an intelligent surveillance system that detects unauthorized individuals in restricted areas by analyzing attire. Traditional security measures face challenges in monitoring unauthorized access. Leveraging YOLOv8, an advanced object detection algorithm, our system identifies authorized personnel based on their attire in CCTV footage. The methodology involves training the YOLOv8 model on a comprehensive dataset of uniform patterns, ensuring precise recognition in specific regions. Soft computing techniques enhance adaptability to dynamic environments and varying lighting conditions. This research contributes to image analysis and soft computing, providing a sophisticated security solution. Emphasizing uniform-based anomaly detection, it establishes a foundation for robust security systems in restricted areas. The outcomes highlight the potential of YOLOv8-based surveillance in ensuring safety in sensitive locations.

Title: SpiralMLP: A Lightweight Vision MLP Architecture

• Authors: Haojie Mu, Burhan Ul Tayyab, Nicholas Chua
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00648
• Pdf URL: https://arxiv.org/pdf/2404.00648
• Copy Paste: [[2404.00648]] SpiralMLP: A Lightweight Vision MLP Architecture(https://arxiv.org/abs/2404.00648)
• Keywords: transformer
• Abstract: We present SpiralMLP, a novel architecture that introduces a Spiral FC layer as a replacement for the conventional Token Mixing approach. Differing from several existing MLP-based models that primarily emphasize axes, our Spiral FC layer is designed as a deformable convolution layer with spiral-like offsets. We further adapt Spiral FC into two variants: Self-Spiral FC and Cross-Spiral FC, which enable both local and global feature integration seamlessly, eliminating the need for additional processing steps. To thoroughly investigate the effectiveness of the spiral-like offsets and validate our design, we conduct ablation studies and explore optimal configurations. In empirical tests, SpiralMLP reaches state-of-the-art performance, similar to Transformers, CNNs, and other MLPs, benchmarking on ImageNet-1k, COCO and ADE20K. SpiralMLP still maintains linear computational complexity O(HW) and is compatible with varying input image resolutions. Our study reveals that targeting the full receptive field is not essential for achieving high performance, instead, adopting a refined approach offers better results.

Title: Deep Instruction Tuning for Segment Anything Model

• Authors: Xiaorui Huang, Gen Luo, Chaoyang Zhu, Bo Tong, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00650
• Pdf URL: https://arxiv.org/pdf/2404.00650
• Copy Paste: [[2404.00650]] Deep Instruction Tuning for Segment Anything Model(https://arxiv.org/abs/2404.00650)
• Keywords: segmentation
• Abstract: Segment Anything Model (SAM) exhibits powerful yet versatile capabilities on (un) conditional image segmentation tasks recently. Although SAM can support various segmentation prompts, we note that, compared to point- and box-guided segmentation, it performs much worse on text-instructed tasks. We argue that deep text instruction tuning is key to mitigate such shortcoming caused by the shallow fusion scheme in its default light-weight mask decoder. In this paper, two \emph{deep instruction tuning} (DIT) methods are proposed, one is end-to-end and the other is layer-wise. With these tuning methods, we can regard the image encoder of SAM as a stand-alone vision-language learner in contrast to building another deep fusion branch. Extensive experiments on three highly competitive benchmark datasets of referring image segmentation show that a simple end-to-end DIT improves SAM by a large margin, with layer-wise DIT further boosts the performance to state-of-the-art. Our code is anonymously released at: https://github.com/wysnzzzz/DIT.

Title: WavLLM: Towards Robust and Adaptive Speech Large Language Model

• Authors: Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.00656
• Pdf URL: https://arxiv.org/pdf/2404.00656
• Copy Paste: [[2404.00656]] WavLLM: Towards Robust and Adaptive Speech Large Language Model(https://arxiv.org/abs/2404.00656)
• Keywords: robust, large language model
• Abstract: The recent advancements in large language models (LLMs) have revolutionized the field of natural language processing, progressively broadening their scope to multimodal perception and generation. However, effectively integrating listening capabilities into LLMs poses significant challenges, particularly with respect to generalizing across varied contexts and executing complex auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach. Leveraging dual encoders, we decouple different types of speech information, utilizing a Whisper encoder to process the semantic content of speech, and a WavLM encoder to capture the unique characteristics of the speaker's identity. Within the curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks such as combinations of the elementary tasks. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. Furthermore, our model successfully completes Gaokao tasks without specialized training. The codes, models, audio, and Gaokao evaluation set can be accessed at \url{aka.ms/wavllm}.

Title: KTPFormer: Kinematics and Trajectory Prior Knowledge-Enhanced Transformer for 3D Human Pose Estimation

• Authors: Jihua Peng, Yanghong Zhou, P.Y. Mok
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00658
• Pdf URL: https://arxiv.org/pdf/2404.00658
• Copy Paste: [[2404.00658]] KTPFormer: Kinematics and Trajectory Prior Knowledge-Enhanced Transformer for 3D Human Pose Estimation(https://arxiv.org/abs/2404.00658)
• Keywords: diffusion, transformer
• Abstract: This paper presents a novel Kinematics and Trajectory Prior Knowledge-Enhanced Transformer (KTPFormer), which overcomes the weakness in existing transformer-based methods for 3D human pose estimation that the derivation of Q, K, V vectors in their self-attention mechanisms are all based on simple linear mapping. We propose two prior attention modules, namely Kinematics Prior Attention (KPA) and Trajectory Prior Attention (TPA) to take advantage of the known anatomical structure of the human body and motion trajectory information, to facilitate effective learning of global dependencies and features in the multi-head self-attention. KPA models kinematic relationships in the human body by constructing a topology of kinematics, while TPA builds a trajectory topology to learn the information of joint motion trajectory across frames. Yielding Q, K, V vectors with prior knowledge, the two modules enable KTPFormer to model both spatial and temporal correlations simultaneously. Extensive experiments on three benchmarks (Human3.6M, MPI-INF-3DHP and HumanEva) show that KTPFormer achieves superior performance in comparison to state-of-the-art methods. More importantly, our KPA and TPA modules have lightweight plug-and-play designs and can be integrated into various transformer-based networks (i.e., diffusion-based) to improve the performance with only a very small increase in the computational overhead. The code is available at: https://github.com/JihuaPeng/KTPFormer.

Title: DeeDSR: Towards Real-World Image Super-Resolution via Degradation-Aware Stable Diffusion

• Authors: Chunyang Bi, Xin Luo, Sheng Shen, Mengxi Zhang, Huanjing Yue, Jingyu Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00661
• Pdf URL: https://arxiv.org/pdf/2404.00661
• Copy Paste: [[2404.00661]] DeeDSR: Towards Real-World Image Super-Resolution via Degradation-Aware Stable Diffusion(https://arxiv.org/abs/2404.00661)
• Keywords: diffusion, generative
• Abstract: Diffusion models, known for their powerful generative capabilities, play a crucial role in addressing real-world super-resolution challenges. However, these models often focus on improving local textures while neglecting the impacts of global degradation, which can significantly reduce semantic fidelity and lead to inaccurate reconstructions and suboptimal super-resolution performance. To address this issue, we introduce a novel two-stage, degradation-aware framework that enhances the diffusion model's ability to recognize content and degradation in low-resolution images. In the first stage, we employ unsupervised contrastive learning to obtain representations of image degradations. In the second stage, we integrate a degradation-aware module into a simplified ControlNet, enabling flexible adaptation to various degradations based on the learned representations. Furthermore, we decompose the degradation-aware features into global semantics and local details branches, which are then injected into the diffusion denoising module to modulate the target generation. Our method effectively recovers semantically precise and photorealistic details, particularly under significant degradation conditions, demonstrating state-of-the-art performance across various benchmarks. Codes will be released at https://github.com/bichunyang419/DeeDSR.

Title: Weakly-Supervised Cross-Domain Segmentation of Electron Microscopy with Sparse Point Annotation

• Authors: Dafei Qiu, Shan Xiong, Jiajin Yi, Jialin Peng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00667
• Pdf URL: https://arxiv.org/pdf/2404.00667
• Copy Paste: [[2404.00667]] Weakly-Supervised Cross-Domain Segmentation of Electron Microscopy with Sparse Point Annotation(https://arxiv.org/abs/2404.00667)
• Keywords: robust, segmentation
• Abstract: Accurate segmentation of organelle instances from electron microscopy (EM) images plays an essential role in many neuroscience researches. However, practical scenarios usually suffer from high annotation costs, label scarcity, and large domain diversity. While unsupervised domain adaptation (UDA) that assumes no annotation effort on the target data is promising to alleviate these challenges, its performance on complicated segmentation tasks is still far from practical usage. To address these issues, we investigate a highly annotation-efficient weak supervision, which assumes only sparse center-points on a small subset of object instances in the target training images. To achieve accurate segmentation with partial point annotations, we introduce instance counting and center detection as auxiliary tasks and design a multitask learning framework to leverage correlations among the counting, detection, and segmentation, which are all tasks with partial or no supervision. Building upon the different domain-invariances of the three tasks, we enforce counting estimation with a novel soft consistency loss as a global prior for center detection, which further guides the per-pixel segmentation. To further compensate for annotation sparsity, we develop a cross-position cut-and-paste for label augmentation and an entropy-based pseudo-label selection. The experimental results highlight that, by simply using extremely weak annotation, e.g., 15\% sparse points, for model training, the proposed model is capable of significantly outperforming UDA methods and produces comparable performance as the supervised counterpart. The high robustness of our model shown in the validations and the low requirement of expert knowledge for sparse point annotation further improve the potential application value of our model.

Title: Statistical Analysis by Semiparametric Additive Regression and LSTM-FCN Based Hierarchical Classification for Computer Vision Quantification of Parkinsonian Bradykinesia

• Authors: Youngseo Cho, In Hee Kwak, Dohyeon Kim, Jinhee Na, Hanjoo Sung, Jeongjae Lee, Young Eun Kim, Hyeo-il Ma
• Subjects: cs.CV, q-bio.QM, stat.AP
• Abstract URL: https://arxiv.org/abs/2404.00670
• Pdf URL: https://arxiv.org/pdf/2404.00670
• Copy Paste: [[2404.00670]] Statistical Analysis by Semiparametric Additive Regression and LSTM-FCN Based Hierarchical Classification for Computer Vision Quantification of Parkinsonian Bradykinesia(https://arxiv.org/abs/2404.00670)
• Keywords: robust
• Abstract: Bradykinesia, characterized by involuntary slowing or decrement of movement, is a fundamental symptom of Parkinson's Disease (PD) and is vital for its clinical diagnosis. Despite various methodologies explored to quantify bradykinesia, computer vision-based approaches have shown promising results. However, these methods often fall short in adequately addressing key bradykinesia characteristics in repetitive limb movements: "occasional arrest" and "decrement in amplitude." This research advances vision-based quantification of bradykinesia by introducing nuanced numerical analysis to capture decrement in amplitudes and employing a simple deep learning technique, LSTM-FCN, for precise classification of occasional arrests. Our approach structures the classification process hierarchically, tailoring it to the unique dynamics of bradykinesia in PD. Statistical analysis of the extracted features, including those representing arrest and fatigue, has demonstrated their statistical significance in most cases. This finding underscores the importance of considering "occasional arrest" and "decrement in amplitude" in bradykinesia quantification of limb movement. Our enhanced diagnostic tool has been rigorously tested on an extensive dataset comprising 1396 motion videos from 310 PD patients, achieving an accuracy of 80.3%. The results confirm the robustness and reliability of our method.

Title: A General and Efficient Training for Transformer via Token Expansion

• Authors: Wenxuan Huang, Yunhang Shen, Jiao Xie, Baochang Zhang, Gaoqi He, Ke Li, Xing Sun, Shaohui Lin
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00672
• Pdf URL: https://arxiv.org/pdf/2404.00672
• Copy Paste: [[2404.00672]] A General and Efficient Training for Transformer via Token Expansion(https://arxiv.org/abs/2404.00672)
• Keywords: transformer
• Abstract: The remarkable performance of Vision Transformers (ViTs) typically requires an extremely large training cost. Existing methods have attempted to accelerate the training of ViTs, yet typically disregard method universality with accuracy dropping. Meanwhile, they break the training consistency of the original transformers, including the consistency of hyper-parameters, architecture, and strategy, which prevents them from being widely applied to different Transformer networks. In this paper, we propose a novel token growth scheme Token Expansion (termed ToE) to achieve consistent training acceleration for ViTs. We introduce an "initialization-expansion-merging" pipeline to maintain the integrity of the intermediate feature distribution of original transformers, preventing the loss of crucial learnable information in the training process. ToE can not only be seamlessly integrated into the training and fine-tuning process of transformers (e.g., DeiT and LV-ViT), but also effective for efficient training frameworks (e.g., EfficientTrain), without twisting the original training hyper-parameters, architecture, and introducing additional training strategies. Extensive experiments demonstrate that ToE achieves about 1.3x faster for the training of ViTs in a lossless manner, or even with performance gains over the full-token training baselines. Code is available at https://github.com/Osilly/TokenExpansion .

Title: A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasures

• Authors: Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Thanh Toan Nguyen, Phi Le Nguyen, Hongzhi Yin, Quoc Viet Hung Nguyen
• Subjects: cs.CR, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00673
• Pdf URL: https://arxiv.org/pdf/2404.00673
• Copy Paste: [[2404.00673]] A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasures(https://arxiv.org/abs/2404.00673)
• Keywords: privacy, attack, explainability
• Abstract: As the adoption of explainable AI (XAI) continues to expand, the urgency to address its privacy implications intensifies. Despite a growing corpus of research in AI privacy and explainability, there is little attention on privacy-preserving model explanations. This article presents the first thorough survey about privacy attacks on model explanations and their countermeasures. Our contribution to this field comprises a thorough analysis of research papers with a connected taxonomy that facilitates the categorisation of privacy attacks and countermeasures based on the targeted explanations. This work also includes an initial investigation into the causes of privacy leaks. Finally, we discuss unresolved issues and prospective research directions uncovered in our analysis. This survey aims to be a valuable resource for the research community and offers clear insights for those new to this domain. To support ongoing research, we have established an online resource repository, which will be continuously updated with new and relevant findings. Interested readers are encouraged to access our repository at https://github.com/tamlhp/awesome-privex.

Title: LLM meets Vision-Language Models for Zero-Shot One-Class Classification

• Authors: Yassir Bendou, Giulia Lioi, Bastien Pasdeloup, Lukas Mauch, Ghouthi Boukli Hacene, Fabien Cardinaux, Vincent Gripon
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00675
• Pdf URL: https://arxiv.org/pdf/2404.00675
• Copy Paste: [[2404.00675]] LLM meets Vision-Language Models for Zero-Shot One-Class Classification(https://arxiv.org/abs/2404.00675)
• Keywords: large language model
• Abstract: We consider the problem of zero-shot one-class visual classification. In this setting, only the label of the target class is available, and the goal is to discriminate between positive and negative query samples without requiring any validation example from the target task. We propose a two-step solution that first queries large language models for visually confusing objects and then relies on vision-language pre-trained models (e.g., CLIP) to perform classification. By adapting large-scale vision benchmarks, we demonstrate the ability of the proposed method to outperform adapted off-the-shelf alternatives in this setting. Namely, we propose a realistic benchmark where negative query samples are drawn from the same original dataset as positive ones, including a granularity-controlled version of iNaturalist, where negative samples are at a fixed distance in the taxonomy tree from the positive ones. Our work shows that it is possible to discriminate between a single category and other semantically related ones using only its label

Title: OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos

• Authors: Dongyoung Choi, Hyeonjoong Jang, Min H. Kim
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.00676
• Pdf URL: https://arxiv.org/pdf/2404.00676
• Copy Paste: [[2404.00676]] OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos(https://arxiv.org/abs/2404.00676)
• Keywords: segmentation
• Abstract: Omnidirectional cameras are extensively used in various applications to provide a wide field of vision. However, they face a challenge in synthesizing novel views due to the inevitable presence of dynamic objects, including the photographer, in their wide field of view. In this paper, we introduce a new approach called Omnidirectional Local Radiance Fields (OmniLocalRF) that can render static-only scene views, removing and inpainting dynamic objects simultaneously. Our approach combines the principles of local radiance fields with the bidirectional optimization of omnidirectional rays. Our input is an omnidirectional video, and we evaluate the mutual observations of the entire angle between the previous and current frames. To reduce ghosting artifacts of dynamic objects and inpaint occlusions, we devise a multi-resolution motion mask prediction module. Unlike existing methods that primarily separate dynamic components through the temporal domain, our method uses multi-resolution neural feature planes for precise segmentation, which is more suitable for long 360-degree videos. Our experiments validate that OmniLocalRF outperforms existing methods in both qualitative and quantitative metrics, especially in scenarios with complex real-world scenes. In particular, our approach eliminates the need for manual interaction, such as drawing motion masks by hand and additional pose estimation, making it a highly effective and efficient solution.

Title: Learning to Rank Patches for Unbiased Image Redundancy Reduction

• Authors: Yang Luo, Zhineng Chen, Peng Zhou, Zuxuan Wu, Xieping Gao, Yu-Gang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00680
• Pdf URL: https://arxiv.org/pdf/2404.00680
• Copy Paste: [[2404.00680]] Learning to Rank Patches for Unbiased Image Redundancy Reduction(https://arxiv.org/abs/2404.00680)
• Keywords: fair
• Abstract: Images suffer from heavy spatial redundancy because pixels in neighboring regions are spatially correlated. Existing approaches strive to overcome this limitation by reducing less meaningful image regions. However, current leading methods rely on supervisory signals. They may compel models to preserve content that aligns with labeled categories and discard content belonging to unlabeled categories. This categorical inductive bias makes these methods less effective in real-world scenarios. To address this issue, we propose a self-supervised framework for image redundancy reduction called Learning to Rank Patches (LTRP). We observe that image reconstruction of masked image modeling models is sensitive to the removal of visible patches when the masking ratio is high (e.g., 90\%). Building upon it, we implement LTRP via two steps: inferring the semantic density score of each patch by quantifying variation between reconstructions with and without this patch, and learning to rank the patches with the pseudo score. The entire process is self-supervised, thus getting out of the dilemma of categorical inductive bias. We design extensive experiments on different datasets and tasks. The results demonstrate that LTRP outperforms both supervised and other self-supervised methods due to the fair assessment of image content.

Title: CoUDA: Coherence Evaluation via Unified Data Augmentation

• Authors: Dawei Zhu, Wenhao Wu, Yifan Song, Fangwei Zhu, Ziqiang Cao, Sujian Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00681
• Pdf URL: https://arxiv.org/pdf/2404.00681
• Copy Paste: [[2404.00681]] CoUDA: Coherence Evaluation via Unified Data Augmentation(https://arxiv.org/abs/2404.00681)
• Keywords: generative, large language model
• Abstract: Coherence evaluation aims to assess the organization and structure of a discourse, which remains challenging even in the era of large language models. Due to the scarcity of annotated data, data augmentation is commonly used for training coherence evaluation models. However, previous augmentations for this task primarily rely on heuristic rules, lacking designing criteria as guidance. In this paper, we take inspiration from linguistic theory of discourse structure, and propose a data augmentation framework named CoUDA. CoUDA breaks down discourse coherence into global and local aspects, and designs augmentation strategies for both aspects, respectively. Especially for local coherence, we propose a novel generative strategy for constructing augmentation samples, which involves post-pretraining a generative model and applying two controlling mechanisms to control the difficulty of generated samples. During inference, CoUDA also jointly evaluates both global and local aspects to comprehensively assess the overall coherence of a discourse. Extensive experiments in coherence evaluation show that, with only 233M parameters, CoUDA achieves state-of-the-art performance in both pointwise scoring and pairwise ranking tasks, even surpassing recent GPT-3.5 and GPT-4 based metrics.

Title: DMSSN: Distilled Mixed Spectral-Spatial Network for Hyperspectral Salient Object Detection

• Authors: Haolin Qin, Tingfa Xu, Peifu Liu, Jingxuan Xu, Jianan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00694
• Pdf URL: https://arxiv.org/pdf/2404.00694
• Copy Paste: [[2404.00694]] DMSSN: Distilled Mixed Spectral-Spatial Network for Hyperspectral Salient Object Detection(https://arxiv.org/abs/2404.00694)
• Keywords: robust, extraction, transformer
• Abstract: Hyperspectral salient object detection (HSOD) has exhibited remarkable promise across various applications, particularly in intricate scenarios where conventional RGB-based approaches fall short. Despite the considerable progress in HSOD method advancements, two critical challenges require immediate attention. Firstly, existing hyperspectral data dimension reduction techniques incur a loss of spectral information, which adversely affects detection accuracy. Secondly, previous methods insufficiently harness the inherent distinctive attributes of hyperspectral images (HSIs) during the feature extraction process. To address these challenges, we propose a novel approach termed the Distilled Mixed Spectral-Spatial Network (DMSSN), comprising a Distilled Spectral Encoding process and a Mixed Spectral-Spatial Transformer (MSST) feature extraction network. The encoding process utilizes knowledge distillation to construct a lightweight autoencoder for dimension reduction, striking a balance between robust encoding capabilities and low computational costs. The MSST extracts spectral-spatial features through multiple attention head groups, collaboratively enhancing its resistance to intricate scenarios. Moreover, we have created a large-scale HSOD dataset, HSOD-BIT, to tackle the issue of data scarcity in this field and meet the fundamental data requirements of deep network training. Extensive experiments demonstrate that our proposed DMSSN achieves state-of-the-art performance on multiple datasets. We will soon make the code and dataset publicly available on https://github.com/anonymous0519/HSOD-BIT.

Title: Privacy Re-identification Attacks on Tabular GANs

• Authors: Abdallah Alshantti, Adil Rasheed, Frank Westad
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00696
• Pdf URL: https://arxiv.org/pdf/2404.00696
• Copy Paste: [[2404.00696]] Privacy Re-identification Attacks on Tabular GANs(https://arxiv.org/abs/2404.00696)
• Keywords: privacy, attack, generative
• Abstract: Generative models are subject to overfitting and thus may potentially leak sensitive information from the training data. In this work. we investigate the privacy risks that can potentially arise from the use of generative adversarial networks (GANs) for creating tabular synthetic datasets. For the purpose, we analyse the effects of re-identification attacks on synthetic data, i.e., attacks which aim at selecting samples that are predicted to correspond to memorised training samples based on their proximity to the nearest synthetic records. We thus consider multiple settings where different attackers might have different access levels or knowledge of the generative model and predictive, and assess which information is potentially most useful for launching more successful re-identification attacks. In doing so we also consider the situation for which re-identification attacks are formulated as reconstruction attacks, i.e., the situation where an attacker uses evolutionary multi-objective optimisation for perturbing synthetic samples closer to the training space. The results indicate that attackers can indeed pose major privacy risks by selecting synthetic samples that are likely representative of memorised training samples. In addition, we notice that privacy threats considerably increase when the attacker either has knowledge or has black-box access to the generative models. We also find that reconstruction attacks through multi-objective optimisation even increase the risk of identifying confidential samples.

Title: How Much are LLMs Contaminated? A Comprehensive Survey and the LLMSanitize Library

• Authors: Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, Shafiq Joty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00699
• Pdf URL: https://arxiv.org/pdf/2404.00699
• Copy Paste: [[2404.00699]] How Much are LLMs Contaminated? A Comprehensive Survey and the LLMSanitize Library(https://arxiv.org/abs/2404.00699)
• Keywords: large language model
• Abstract: With the rise of Large Language Models (LLMs) in recent years, new opportunities are emerging, but also new challenges, and contamination is quickly becoming critical. Business applications and fundraising in AI have reached a scale at which a few percentage points gained on popular question-answering benchmarks could translate into dozens of millions of dollars, placing high pressure on model integrity. At the same time, it is becoming harder and harder to keep track of the data that LLMs have seen; if not impossible with closed-source models like GPT-4 and Claude-3 not divulging any information on the training set. As a result, contamination becomes a critical issue: LLMs' performance may not be reliable anymore, as the high performance may be at least partly due to their previous exposure to the data. This limitation jeopardizes the entire progress in the field of NLP, yet, there remains a lack of methods on how to efficiently address contamination, or a clear consensus on prevention, mitigation and classification of contamination. In this paper, we survey all recent work on contamination with LLMs, and help the community track contamination levels of LLMs by releasing an open-source Python library named LLMSanitize implementing major contamination detection algorithms, which link is: https://github.com/ntunlp/LLMSanitize.

Title: Training-Free Semantic Segmentation via LLM-Supervision

• Authors: Wenfang Sun, Yingjun Du, Gaowen Liu, Ramana Kompella, Cees G.M. Snoek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00701
• Pdf URL: https://arxiv.org/pdf/2404.00701
• Copy Paste: [[2404.00701]] Training-Free Semantic Segmentation via LLM-Supervision(https://arxiv.org/abs/2404.00701)
• Keywords: large language model, segmentation
• Abstract: Recent advancements in open vocabulary models, like CLIP, have notably advanced zero-shot classification and segmentation by utilizing natural language for class-specific embeddings. However, most research has focused on improving model accuracy through prompt engineering, prompt learning, or fine-tuning with limited labeled data, thereby overlooking the importance of refining the class descriptors. This paper introduces a new approach to text-supervised semantic segmentation using supervision by a large language model (LLM) that does not require extra training. Our method starts from an LLM, like GPT-3, to generate a detailed set of subclasses for more accurate class representation. We then employ an advanced text-supervised semantic segmentation model to apply the generated subclasses as target labels, resulting in diverse segmentation results tailored to each subclass's unique characteristics. Additionally, we propose an assembly that merges the segmentation maps from the various subclass descriptors to ensure a more comprehensive representation of the different aspects in the test images. Through comprehensive experiments on three standard benchmarks, our method outperforms traditional text-supervised semantic segmentation methods by a marked margin.

Title: Unknown Prompt, the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization

• Authors: Mainak Singha, Ankit Jha, Shirsha Bose, Ashwin Nair, Moloud Abdar, Biplab Banerjee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00710
• Pdf URL: https://arxiv.org/pdf/2404.00710
• Copy Paste: [[2404.00710]] Unknown Prompt, the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization(https://arxiv.org/abs/2404.00710)
• Keywords: diffusion
• Abstract: We delve into Open Domain Generalization (ODG), marked by domain and category shifts between training's labeled source and testing's unlabeled target domains. Existing solutions to ODG face limitations due to constrained generalizations of traditional CNN backbones and errors in detecting target open samples in the absence of prior knowledge. Addressing these pitfalls, we introduce ODG-CLIP, harnessing the semantic prowess of the vision-language model, CLIP. Our framework brings forth three primary innovations: Firstly, distinct from prevailing paradigms, we conceptualize ODG as a multi-class classification challenge encompassing both known and novel categories. Central to our approach is modeling a unique prompt tailored for detecting unknown class samples, and to train this, we employ a readily accessible stable diffusion model, elegantly generating proxy images for the open class. Secondly, aiming for domain-tailored classification (prompt) weights while ensuring a balance of precision and simplicity, we devise a novel visual stylecentric prompt learning mechanism. Finally, we infuse images with class-discriminative knowledge derived from the prompt space to augment the fidelity of CLIP's visual embeddings. We introduce a novel objective to safeguard the continuity of this infused semantic intel across domains, especially for the shared classes. Through rigorous testing on diverse datasets, covering closed and open-set DG contexts, ODG-CLIP demonstrates clear supremacy, consistently outpacing peers with performance boosts between 8%-16%. Code will be available at https://github.com/mainaksingha01/ODG-CLIP.

Title: Survey of Computerized Adaptive Testing: A Machine Learning Perspective

• Authors: Qi Liu, Yan Zhuang, Haoyang Bi, Zhenya Huang, Weizhe Huang, Jiatong Li, Junhao Yu, Zirui Liu, Zirui Hu, Yuting Hong, Zachary A. Pardos, Haiping Ma, Mengxiao Zhu, Shijin Wang, Enhong Chen
• Subjects: cs.LG, cs.AI, cs.CY, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.00712
• Pdf URL: https://arxiv.org/pdf/2404.00712
• Copy Paste: [[2404.00712]] Survey of Computerized Adaptive Testing: A Machine Learning Perspective(https://arxiv.org/abs/2404.00712)
• Keywords: robust, fair
• Abstract: Computerized Adaptive Testing (CAT) provides an efficient and tailored method for assessing the proficiency of examinees, by dynamically adjusting test questions based on their performance. Widely adopted across diverse fields like education, healthcare, sports, and sociology, CAT has revolutionized testing practices. While traditional methods rely on psychometrics and statistics, the increasing complexity of large-scale testing has spurred the integration of machine learning techniques. This paper aims to provide a machine learning-focused survey on CAT, presenting a fresh perspective on this adaptive testing method. By examining the test question selection algorithm at the heart of CAT's adaptivity, we shed light on its functionality. Furthermore, we delve into cognitive diagnosis models, question bank construction, and test control within CAT, exploring how machine learning can optimize these components. Through an analysis of current methods, strengths, limitations, and challenges, we strive to develop robust, fair, and efficient CAT systems. By bridging psychometric-driven CAT research with machine learning, this survey advocates for a more inclusive and interdisciplinary approach to the future of adaptive testing.

Title: DRCT: Saving Image Super-resolution away from Information Bottleneck

• Authors: Chih-Chung Hsu, Chia-Ming Lee, Yi-Shiuan Chou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00722
• Pdf URL: https://arxiv.org/pdf/2404.00722
• Copy Paste: [[2404.00722]] DRCT: Saving Image Super-resolution away from Information Bottleneck(https://arxiv.org/abs/2404.00722)
• Keywords: transformer
• Abstract: In recent years, Vision Transformer-based applications to low-level vision tasks have achieved widespread success. Unlike CNN-based models, Transformers are more adept at capturing long-range dependencies, enabling the reconstruction of images utilizing information from non-local areas. In the domain of super-resolution, Swin-transformer-based approaches have become mainstream due to their capacity to capture global spatial information and their shifting-window attention mechanism that facilitates the interchange of information between different windows. Many researchers have enhanced image quality and network efficiency by expanding the receptive field or designing complex networks, yielding commendable results. However, we observed that spatial information tends to diminish during the forward propagation process due to increased depth, leading to a loss of spatial information and, consequently, limiting the model's potential. To address this, we propose the Dense-residual-connected Transformer (DRCT), aimed at mitigating the loss of spatial information through dense-residual connections between layers, thereby unleashing the model's potential and enhancing performance. Experiment results indicate that our approach is not only straightforward but also achieves remarkable efficiency, surpassing state-of-the-art methods and performing commendably at NTIRE2024.

Title: Opera Graeca Adnotata: Building a 34M+ Token Multilayer Corpus for Ancient Greek

• Authors: Giuseppe G. A. Celano
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00739
• Pdf URL: https://arxiv.org/pdf/2404.00739
• Copy Paste: [[2404.00739]] Opera Graeca Adnotata: Building a 34M+ Token Multilayer Corpus for Ancient Greek(https://arxiv.org/abs/2404.00739)
• Keywords: segmentation
• Abstract: In this article, the beta version 0.1.0 of Opera Graeca Adnotata (OGA), the largest open-access multilayer corpus for Ancient Greek (AG) is presented. OGA consists of 1,687 literary works and 34M+ tokens coming from the PerseusDL and OpenGreekAndLatin GitHub repositories, which host AG texts ranging from about 800 BCE to about 250 CE. The texts have been enriched with seven annotation layers: (i) tokenization layer; (ii) sentence segmentation layer; (iii) lemmatization layer; (iv) morphological layer; (v) dependency layer; (vi) dependency function layer; (vii) Canonical Text Services (CTS) citation layer. The creation of each layer is described by highlighting the main technical and annotation-related issues encountered. Tokenization, sentence segmentation, and CTS citation are performed by rule-based algorithms, while morphosyntactic annotation is the output of the COMBO parser trained on the data of the Ancient Greek Dependency Treebank. For the sake of scalability and reusability, the corpus is released in the standoff formats PAULA XML and its offspring LAULA XML.

Title: Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts

• Authors: Qin Liu, Jaemin Cho, Mohit Bansal, Marc Niethammer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00741
• Pdf URL: https://arxiv.org/pdf/2404.00741
• Copy Paste: [[2404.00741]] Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts(https://arxiv.org/abs/2404.00741)
• Keywords: segmentation
• Abstract: The goal of interactive image segmentation is to delineate specific regions within an image via visual or language prompts. Low-latency and high-quality interactive segmentation with diverse prompts remain challenging for existing specialist and generalist models. Specialist models, with their limited prompts and task-specific designs, experience high latency because the image must be recomputed every time the prompt is updated, due to the joint encoding of image and visual prompts. Generalist models, exemplified by the Segment Anything Model (SAM), have recently excelled in prompt diversity and efficiency, lifting image segmentation to the foundation model era. However, for high-quality segmentations, SAM still lags behind state-of-the-art specialist models despite SAM being trained with x100 more segmentation masks. In this work, we delve deep into the architectural differences between the two types of models. We observe that dense representation and fusion of visual prompts are the key design choices contributing to the high segmentation quality of specialist models. In light of this, we reintroduce this dense design into the generalist models, to facilitate the development of generalist models with high segmentation quality. To densely represent diverse visual prompts, we propose to use a dense map to capture five types: clicks, boxes, polygons, scribbles, and masks. Thus, we propose SegNext, a next-generation interactive segmentation approach offering low latency, high quality, and diverse prompt support. Our method outperforms current state-of-the-art methods on HQSeg-44K and DAVIS, both quantitatively and qualitatively.

Title: Adapting to Length Shift: FlexiLength Network for Trajectory Prediction

• Authors: Yi Xu, Yun Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00742
• Pdf URL: https://arxiv.org/pdf/2404.00742
• Copy Paste: [[2404.00742]] Adapting to Length Shift: FlexiLength Network for Trajectory Prediction(https://arxiv.org/abs/2404.00742)
• Keywords: robust
• Abstract: Trajectory prediction plays an important role in various applications, including autonomous driving, robotics, and scene understanding. Existing approaches mainly focus on developing compact neural networks to increase prediction precision on public datasets, typically employing a standardized input duration. However, a notable issue arises when these models are evaluated with varying observation lengths, leading to a significant performance drop, a phenomenon we term the Observation Length Shift. To address this issue, we introduce a general and effective framework, the FlexiLength Network (FLN), to enhance the robustness of existing trajectory prediction techniques against varying observation periods. Specifically, FLN integrates trajectory data with diverse observation lengths, incorporates FlexiLength Calibration (FLC) to acquire temporal invariant representations, and employs FlexiLength Adaptation (FLA) to further refine these representations for more accurate future trajectory predictions. Comprehensive experiments on multiple datasets, ie, ETH/UCY, nuScenes, and Argoverse 1, demonstrate the effectiveness and flexibility of our proposed FLN framework.

Title: Can Language Models Recognize Convincing Arguments?

• Authors: Paula Rescala, Manoel Horta Ribeiro, Tiancheng Hu, Robert West
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.00750
• Pdf URL: https://arxiv.org/pdf/2404.00750
• Copy Paste: [[2404.00750]] Can Language Models Recognize Convincing Arguments?(https://arxiv.org/abs/2404.00750)
• Keywords: large language model
• Abstract: The remarkable and ever-increasing capabilities of Large Language Models (LLMs) have raised concerns about their potential misuse for creating personalized, convincing misinformation and propaganda. To gain insights into LLMs' persuasive capabilities without directly engaging in experimentation with humans, we propose studying their performance on the related task of detecting convincing arguments. We extend a dataset by Durmus & Cardie (2018) with debates, votes, and user traits and propose tasks measuring LLMs' ability to (1) distinguish between strong and weak arguments, (2) predict stances based on beliefs and demographic characteristics, and (3) determine the appeal of an argument to an individual based on their traits. We show that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, even surpassing human performance. The data and code released with this paper contribute to the crucial ongoing effort of continuously evaluating and monitoring the rapidly evolving capabilities and potential impact of LLMs.

Title: From Robustness to Improved Generalization and Calibration in Pre-trained Language Models

• Authors: Josip Jukić, Jan Šnajder
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00758
• Pdf URL: https://arxiv.org/pdf/2404.00758
• Copy Paste: [[2404.00758]] From Robustness to Improved Generalization and Calibration in Pre-trained Language Models(https://arxiv.org/abs/2404.00758)
• Keywords: robust
• Abstract: Enhancing generalization and uncertainty quantification in pre-trained language models (PLMs) is crucial for their effectiveness and reliability. Building on machine learning research that established the importance of robustness for improving generalization, we investigate the role of representation smoothness, achieved via Jacobian and Hessian regularization, in enhancing PLM performance. Although such regularization methods have proven effective in computer vision, their application in natural language processing (NLP), where PLM inputs are derived from a discrete domain, poses unique challenges. We introduce a novel two-phase regularization approach, JacHess, which minimizes the norms of the Jacobian and Hessian matrices within PLM intermediate representations relative to their inputs. Our evaluation using the GLUE benchmark demonstrates that JacHess significantly improves in-domain generalization and calibration in PLMs, outperforming unregularized fine-tuning and other similar regularization methods.

Title: Privacy-preserving Optics for Enhancing Protection in Face De-identification

• Authors: Jhon Lopez, Carlos Hinojosa, Henry Arguello, Bernard Ghanem
• Subjects: cs.CV, cs.AI, cs.CR, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.00777
• Pdf URL: https://arxiv.org/pdf/2404.00777
• Copy Paste: [[2404.00777]] Privacy-preserving Optics for Enhancing Protection in Face De-identification(https://arxiv.org/abs/2404.00777)
• Keywords: security, privacy, protect, attack
• Abstract: The modern surge in camera usage alongside widespread computer vision technology applications poses significant privacy and security concerns. Current artificial intelligence (AI) technologies aid in recognizing relevant events and assisting in daily tasks in homes, offices, hospitals, etc. The need to access or process personal information for these purposes raises privacy concerns. While software-level solutions like face de-identification provide a good privacy/utility trade-off, they present vulnerabilities to sniffing attacks. In this paper, we propose a hardware-level face de-identification method to solve this vulnerability. Specifically, our approach first learns an optical encoder along with a regression model to obtain a face heatmap while hiding the face identity from the source image. We also propose an anonymization framework that generates a new face using the privacy-preserving image, face heatmap, and a reference face image from a public dataset as input. We validate our approach with extensive simulations and hardware experiments.

Title: Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning

• Authors: Mohamed Elsayed, A. Rupam Mahmood
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00781
• Pdf URL: https://arxiv.org/pdf/2404.00781
• Copy Paste: [[2404.00781]] Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning(https://arxiv.org/abs/2404.00781)
• Keywords: protect
• Abstract: Deep representation learning methods struggle with continual learning, suffering from both catastrophic forgetting of useful units and loss of plasticity, often due to rigid and unuseful units. While many methods address these two issues separately, only a few currently deal with both simultaneously. In this paper, we introduce Utility-based Perturbed Gradient Descent (UPGD) as a novel approach for the continual learning of representations. UPGD combines gradient updates with perturbations, where it applies smaller modifications to more useful units, protecting them from forgetting, and larger modifications to less useful units, rejuvenating their plasticity. We use a challenging streaming learning setup where continual learning problems have hundreds of non-stationarities and unknown task boundaries. We show that many existing methods suffer from at least one of the issues, predominantly manifested by their decreasing accuracy over tasks. On the other hand, UPGD continues to improve performance and surpasses or is competitive with all methods in all problems. Finally, in extended reinforcement learning experiments with PPO, we show that while Adam exhibits a performance drop after initial learning, UPGD avoids it by addressing both continual learning issues.

Title: Disentangling Hippocampal Shape Variations: A Study of Neurological Disorders Using Graph Variational Autoencoder with Contrastive Learning

• Authors: Jakaria Rabbi, Johannes Kiechle, Christian Beaulieu, Nilanjan Ray, Dana Cobzas
• Subjects: cs.CV, cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2404.00785
• Pdf URL: https://arxiv.org/pdf/2404.00785
• Copy Paste: [[2404.00785]] Disentangling Hippocampal Shape Variations: A Study of Neurological Disorders Using Graph Variational Autoencoder with Contrastive Learning(https://arxiv.org/abs/2404.00785)
• Keywords: interpretability, diffusion
• Abstract: This paper presents a comprehensive study focused on disentangling hippocampal shape variations from diffusion tensor imaging (DTI) datasets within the context of neurological disorders. Leveraging a Graph Variational Autoencoder (VAE) enhanced with Supervised Contrastive Learning, our approach aims to improve interpretability by disentangling two distinct latent variables corresponding to age and the presence of diseases. In our ablation study, we investigate a range of VAE architectures and contrastive loss functions, showcasing the enhanced disentanglement capabilities of our approach. This evaluation uses synthetic 3D torus mesh data and real 3D hippocampal mesh datasets derived from the DTI hippocampal dataset. Our supervised disentanglement model outperforms several state-of-the-art (SOTA) methods like attribute and guided VAEs in terms of disentanglement scores. Our model distinguishes between age groups and disease status in patients with Multiple Sclerosis (MS) using the hippocampus data. Our Graph VAE with Supervised Contrastive Learning shows the volume changes of the hippocampus of MS populations at different ages, and the result is consistent with the current neuroimaging literature. This research provides valuable insights into the relationship between neurological disorder and hippocampal shape changes in different age groups of MS populations using a Graph VAE with Supervised Contrastive loss.

Title: Rehearsal-Free Modular and Compositional Continual Learning for Language Models

• Authors: Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schütze
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00790
• Pdf URL: https://arxiv.org/pdf/2404.00790
• Copy Paste: [[2404.00790]] Rehearsal-Free Modular and Compositional Continual Learning for Language Models(https://arxiv.org/abs/2404.00790)
• Keywords: privacy
• Abstract: Continual learning aims at incrementally acquiring new knowledge while not forgetting existing knowledge. To overcome catastrophic forgetting, methods are either rehearsal-based, i.e., store data examples from previous tasks for data replay, or isolate parameters dedicated to each task. However, rehearsal-based methods raise privacy and memory issues, and parameter-isolation continual learning does not consider interaction between tasks, thus hindering knowledge transfer. In this work, we propose MoCL, a rehearsal-free Modular and Compositional Continual Learning framework which continually adds new modules to language models and composes them with existing modules. Experiments on various benchmarks show that MoCL outperforms state of the art and effectively facilitates knowledge transfer.

Title: On Difficulties of Attention Factorization through Shared Memory

• Authors: Uladzislau Yorsh, Martin Holeňa, Ondřej Bojar, David Herel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00798
• Pdf URL: https://arxiv.org/pdf/2404.00798
• Copy Paste: [[2404.00798]] On Difficulties of Attention Factorization through Shared Memory(https://arxiv.org/abs/2404.00798)
• Keywords: transformer
• Abstract: Transformers have revolutionized deep learning in numerous fields, including natural language processing, computer vision, and audio processing. Their strength lies in their attention mechanism, which allows for the discovering of complex input relationships. However, this mechanism's quadratic time and memory complexity pose challenges for larger inputs. Researchers are now investigating models like Linear Unified Nested Attention (Luna) or Memory Augmented Transformer, which leverage external learnable memory to either reduce the attention computation complexity down to linear, or to propagate information between chunks in chunk-wise processing. Our findings challenge the conventional thinking on these models, revealing that interfacing with the memory directly through an attention operation is suboptimal, and that the performance may be considerably improved by filtering the input signal before communicating with memory.

Title: Towards Realistic Scene Generation with LiDAR Diffusion Models

• Authors: Haoxi Ran, Vitor Guizilini, Yue Wang
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.00815
• Pdf URL: https://arxiv.org/pdf/2404.00815
• Copy Paste: [[2404.00815]] Towards Realistic Scene Generation with LiDAR Diffusion Models(https://arxiv.org/abs/2404.00815)
• Keywords: diffusion
• Abstract: Diffusion models (DMs) excel in photo-realistic image synthesis, but their adaptation to LiDAR scene generation poses a substantial hurdle. This is primarily because DMs operating in the point space struggle to preserve the curve-like patterns and 3D geometry of LiDAR scenes, which consumes much of their representation power. In this paper, we propose LiDAR Diffusion Models (LiDMs) to generate LiDAR-realistic scenes from a latent space tailored to capture the realism of LiDAR scenes by incorporating geometric priors into the learning pipeline. Our method targets three major desiderata: pattern realism, geometry realism, and object realism. Specifically, we introduce curve-wise compression to simulate real-world LiDAR patterns, point-wise coordinate supervision to learn scene geometry, and patch-wise encoding for a full 3D object context. With these three core designs, our method achieves competitive performance on unconditional LiDAR generation in 64-beam scenario and state of the art on conditional LiDAR generation, while maintaining high efficiency compared to point-based DMs (up to 107$\times$ faster). Furthermore, by compressing LiDAR scenes into a latent space, we enable the controllability of DMs with various conditions such as semantic maps, camera views, and text prompts. Our code and pretrained weights are available at https://github.com/hancyran/LiDAR-Diffusion.

Title: Extracting Social Determinants of Health from Pediatric Patient Notes Using Large Language Models: Novel Corpus and Methods

• Authors: Yujuan Fu, Giridhar Kaushik Ramachandran, Nicholas J Dobbins, Namu Park, Michael Leu, Abby R. Rosenberg, Kevin Lybarger, Fei Xia, Ozlem Uzuner, Meliha Yetisgen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00826
• Pdf URL: https://arxiv.org/pdf/2404.00826
• Copy Paste: [[2404.00826]] Extracting Social Determinants of Health from Pediatric Patient Notes Using Large Language Models: Novel Corpus and Methods(https://arxiv.org/abs/2404.00826)
• Keywords: extraction, large language model
• Abstract: Social determinants of health (SDoH) play a critical role in shaping health outcomes, particularly in pediatric populations where interventions can have long-term implications. SDoH are frequently studied in the Electronic Health Record (EHR), which provides a rich repository for diverse patient data. In this work, we present a novel annotated corpus, the Pediatric Social History Annotation Corpus (PedSHAC), and evaluate the automatic extraction of detailed SDoH representations using fine-tuned and in-context learning methods with Large Language Models (LLMs). PedSHAC comprises annotated social history sections from 1,260 clinical notes obtained from pediatric patients within the University of Washington (UW) hospital system. Employing an event-based annotation scheme, PedSHAC captures ten distinct health determinants to encompass living and economic stability, prior trauma, education access, substance use history, and mental health with an overall annotator agreement of 81.9 F1. Our proposed fine-tuning LLM-based extractors achieve high performance at 78.4 F1 for event arguments. In-context learning approaches with GPT-4 demonstrate promise for reliable SDoH extraction with limited annotated examples, with extraction performance at 82.3 F1 for event triggers.

Title: PID Control-Based Self-Healing to Improve the Robustness of Large Language Models

• Authors: Zhuotong Chen, Zihu Wang, Yifan Yang, Qianxiao Li, Zheng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00828
• Pdf URL: https://arxiv.org/pdf/2404.00828
• Copy Paste: [[2404.00828]] PID Control-Based Self-Healing to Improve the Robustness of Large Language Models(https://arxiv.org/abs/2404.00828)
• Keywords: robust, large language model
• Abstract: Despite the effectiveness of deep neural networks in numerous natural language processing applications, recent findings have exposed the vulnerability of these language models when minor perturbations are introduced. While appearing semantically indistinguishable to humans, these perturbations can significantly reduce the performance of well-trained language models, raising concerns about the reliability of deploying them in safe-critical situations. In this work, we construct a computationally efficient self-healing process to correct undesired model behavior during online inference when perturbations are applied to input data. This is formulated as a trajectory optimization problem in which the internal states of the neural network layers are automatically corrected using a PID (Proportional-Integral-Derivative) control mechanism. The P controller targets immediate state adjustments, while the I and D controllers consider past states and future dynamical trends, respectively. We leverage the geometrical properties of the training data to design effective linear PID controllers. This approach reduces the computational cost to that of using just the P controller, instead of the full PID control. Further, we introduce an analytical method for approximating the optimal control solutions, enhancing the real-time inference capabilities of this controlled system. Moreover, we conduct a theoretical error analysis of the analytic solution in a simplified setting. The proposed PID control-based self-healing is a low cost framework that improves the robustness of pre-trained large language models, whether standard or robustly trained, against a wide range of perturbations. A detailed implementation can be found in:https://github.com/zhuotongchen/PID-Control-Based-Self-Healing-to-Improve-the-Robustness-of-Large-Language-Models.

Title: Towards Robust Event-guided Low-Light Image Enhancement: A Large-Scale Real-World Event-Image Dataset and Novel Approach

• Authors: Guoqiang Liang, Kanghao Chen, Hangyu Li, Yunfan Lu, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00834
• Pdf URL: https://arxiv.org/pdf/2404.00834
• Copy Paste: [[2404.00834]] Towards Robust Event-guided Low-Light Image Enhancement: A Large-Scale Real-World Event-Image Dataset and Novel Approach(https://arxiv.org/abs/2404.00834)
• Keywords: robust
• Abstract: Event camera has recently received much attention for low-light image enhancement (LIE) thanks to their distinct advantages, such as high dynamic range. However, current research is prohibitively restricted by the lack of large-scale, real-world, and spatial-temporally aligned event-image datasets. To this end, we propose a real-world (indoor and outdoor) dataset comprising over 30K pairs of images and events under both low and normal illumination conditions. To achieve this, we utilize a robotic arm that traces a consistent non-linear trajectory to curate the dataset with spatial alignment precision under 0.03mm. We then introduce a matching alignment strategy, rendering 90% of our dataset with errors less than 0.01s. Based on the dataset, we propose a novel event-guided LIE approach, called EvLight, towards robust performance in real-world low-light scenes. Specifically, we first design the multi-scale holistic fusion branch to extract holistic structural and textural information from both events and images. To ensure robustness against variations in the regional illumination and noise, we then introduce a Signal-to-Noise-Ratio (SNR)-guided regional feature selection to selectively fuse features of images from regions with high SNR and enhance those with low SNR by extracting regional structure information from events. Extensive experiments on our dataset and the synthetic SDSD dataset demonstrate our EvLight significantly surpasses the frame-based methods. Code and datasets are available at https://vlislab22.github.io/eg-lowlight/.

Title: Transfer Learning with Point Transformers

• Authors: Kartik Gupta, Rahul Vippala, Sahima Srivastava
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00846
• Pdf URL: https://arxiv.org/pdf/2404.00846
• Copy Paste: [[2404.00846]] Transfer Learning with Point Transformers(https://arxiv.org/abs/2404.00846)
• Keywords: transformer, segmentation
• Abstract: Point Transformers are near state-of-the-art models for classification, segmentation, and detection tasks on Point Cloud data. They utilize a self attention based mechanism to model large range spatial dependencies between multiple point sets. In this project we explore two things: classification performance of these attention based networks on ModelNet10 dataset and then, we use the trained model to classify 3D MNIST dataset after finetuning. We also train the model from scratch on 3D MNIST dataset to compare the performance of finetuned and from-scratch model on the MNIST dataset. We observe that since the two datasets have a large difference in the degree of the distributions, transfer learned models do not outperform the from-scratch models in this case. Although we do expect transfer learned models to converge faster since they already know the lower level edges, corners, etc features from the ModelNet10 dataset.

Title: Collaborative Learning of Anomalies with Privacy (CLAP) for Unsupervised Video Anomaly Detection: A New Baseline

• Authors: Anas Al-lahham, Muhammad Zaigham Zaheer, Nurbek Tastan, Karthik Nandakumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00847
• Pdf URL: https://arxiv.org/pdf/2404.00847
• Copy Paste: [[2404.00847]] Collaborative Learning of Anomalies with Privacy (CLAP) for Unsupervised Video Anomaly Detection: A New Baseline(https://arxiv.org/abs/2404.00847)
• Keywords: privacy
• Abstract: Unsupervised (US) video anomaly detection (VAD) in surveillance applications is gaining more popularity recently due to its practical real-world applications. As surveillance videos are privacy sensitive and the availability of large-scale video data may enable better US-VAD systems, collaborative learning can be highly rewarding in this setting. However, due to the extremely challenging nature of the US-VAD task, where learning is carried out without any annotations, privacy-preserving collaborative learning of US-VAD systems has not been studied yet. In this paper, we propose a new baseline for anomaly detection capable of localizing anomalous events in complex surveillance videos in a fully unsupervised fashion without any labels on a privacy-preserving participant-based distributed training configuration. Additionally, we propose three new evaluation protocols to benchmark anomaly detection approaches on various scenarios of collaborations and data availability. Based on these protocols, we modify existing VAD datasets to extensively evaluate our approach as well as existing US SOTA methods on two large-scale datasets including UCF-Crime and XD-Violence. All proposed evaluation protocols, dataset splits, and codes are available here: https://github.com/AnasEmad11/CLAP

Title: Generating Content for HDR Deghosting from Frequency View

• Authors: Tao Hu, Qingsen Yan, Yuankai Qi, Yanning Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00849
• Pdf URL: https://arxiv.org/pdf/2404.00849
• Copy Paste: [[2404.00849]] Generating Content for HDR Deghosting from Frequency View(https://arxiv.org/abs/2404.00849)
• Keywords: diffusion
• Abstract: Recovering ghost-free High Dynamic Range (HDR) images from multiple Low Dynamic Range (LDR) images becomes challenging when the LDR images exhibit saturation and significant motion. Recent Diffusion Models (DMs) have been introduced in HDR imaging field, demonstrating promising performance, particularly in achieving visually perceptible results compared to previous DNN-based methods. However, DMs require extensive iterations with large models to estimate entire images, resulting in inefficiency that hinders their practical application. To address this challenge, we propose the Low-Frequency aware Diffusion (LF-Diff) model for ghost-free HDR imaging. The key idea of LF-Diff is implementing the DMs in a highly compacted latent space and integrating it into a regression-based model to enhance the details of reconstructed images. Specifically, as low-frequency information is closely related to human visual perception we propose to utilize DMs to create compact low-frequency priors for the reconstruction process. In addition, to take full advantage of the above low-frequency priors, the Dynamic HDR Reconstruction Network (DHRNet) is carried out in a regression-based manner to obtain final HDR images. Extensive experiments conducted on synthetic and real-world benchmark datasets demonstrate that our LF-Diff performs favorably against several state-of-the-art methods and is 10$\times$ faster than previous DM-based methods.

Title: Do language models plan ahead for future tokens?

• Authors: Wilson Wu, John X. Morris, Lionel Levine
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00859
• Pdf URL: https://arxiv.org/pdf/2404.00859
• Copy Paste: [[2404.00859]] Do language models plan ahead for future tokens?(https://arxiv.org/abs/2404.00859)
• Keywords: transformer
• Abstract: Do transformers "think ahead" during inference at a given position? It is known transformers prepare information in the hidden states of the forward pass at $t$ that is then used in future forward passes $t+\tau$. We posit two explanations for this phenomenon: pre-caching, in which off-diagonal gradient terms present in training result in the model computing features at $t$ irrelevant to the present inference task but useful for the future, and breadcrumbs, in which features most relevant to time step $t$ are already the same as those that would most benefit inference at time $t+\tau$. We test these hypotheses by training language models without propagating gradients to past timesteps, a scheme we formalize as myopic training. In a synthetic data setting, we find clear evidence for pre-caching. In the autoregressive language modeling setting, our experiments are more suggestive of the breadcrumbs hypothesis.

Title: Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance

• Authors: Giung Nam, Byeongho Heo, Juho Lee
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00860
• Pdf URL: https://arxiv.org/pdf/2404.00860
• Copy Paste: [[2404.00860]] Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance(https://arxiv.org/abs/2404.00860)
• Keywords: robust
• Abstract: Large-scale contrastive vision-language pre-trained models provide the zero-shot model achieving competitive performance across a range of image classification tasks without requiring training on downstream data. Recent works have confirmed that while additional fine-tuning of the zero-shot model on the reference data results in enhanced downstream performance, it compromises the model's robustness against distribution shifts. Our investigation begins by examining the conditions required to achieve the goals of robust fine-tuning, employing descriptions based on feature distortion theory and joint energy-based models. Subsequently, we propose a novel robust fine-tuning algorithm, Lipsum-FT, that effectively utilizes the language modeling aspect of the vision-language pre-trained models. Extensive experiments conducted on distribution shift scenarios in DomainNet and ImageNet confirm the superiority of our proposed Lipsum-FT approach over existing robust fine-tuning methods.

Title: Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding

• Authors: Lung-Chuan Chen, Zong-Ru Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00862
• Pdf URL: https://arxiv.org/pdf/2404.00862
• Copy Paste: [[2404.00862]] Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding(https://arxiv.org/abs/2404.00862)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated exceptional performance in various NLP applications. However, the majority of existing open-source LLMs are pre-trained primarily on English data and little part of other languages. This deficiency in multilingual training data results in suboptimal performance when applied to languages with fewer available resources. Furthermore, enhancing the performance of LLMs on low-resource languages by full-parameter fine-tuning with additional data requires substantial computational resources, posing computational barriers for research organizations and individual researchers. Consequently, several techniques such as parameter-efficient tuning and advanced embedding initialization have been proposed to address these challenges. In this work, we combine them to facilitate cross-lingual transfer on English-dominated open-source LLM. To effectively enhance the model's proficiency in Traditional Chinese, we conduct secondary pre-training on Llama 2 7B with Traditional Chinese data by leveraging QLoRA and our proposed zip-tie embedding initialization. The resulting model called Bailong, which stands for Bilingual trAnsfer learnIng based on qLOra and zip-tie embeddiNG. We present Bailong-instruct 7B, a fine-tuned version of Bailong 7B optimized for multi-turn dialogue scenarios. Recognizing the inadequacy of benchmark datasets in Traditional Chinese, we further introduce Bailong-bench to assess the alignment of models with human preferences and the capability to follow instructions in both Traditional Chinese and English tasks. In our evaluation, Bailong-instruct 7B exhibits competitive performance on Bailong-bench and other benchmark datasets when compared to other open-source models of similar or even larger parameter sizes. Bailong-instruct 7B and Bailong-bench are publicly available with the aim of empowering the community to build upon our efforts.

Title: Towards Automated Generation of Smart Grid Cyber Range for Cybersecurity Experiments and Training

• Authors: Daisuke Mashima, Muhammad M. Roomi, Bennet Ng, Zbigniew Kalbarczyk, S.M. Suhail Hussain, Ee-chien Chang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.00869
• Pdf URL: https://arxiv.org/pdf/2404.00869
• Copy Paste: [[2404.00869]] Towards Automated Generation of Smart Grid Cyber Range for Cybersecurity Experiments and Training(https://arxiv.org/abs/2404.00869)
• Keywords: security, attack
• Abstract: Assurance of cybersecurity is crucial to ensure dependability and resilience of smart power grid systems. In order to evaluate the impact of potential cyber attacks, to assess deployability and effectiveness of cybersecurity measures, and to enable hands-on exercise and training of personals, an interactive, virtual environment that emulates the behaviour of a smart grid system, namely smart grid cyber range, has been demanded by industry players as well as academia. A smart grid cyber range is typically implemented as a combination of cyber system emulation, which allows interactivity, and physical system (i.e., power grid) simulation that are tightly coupled for consistent cyber and physical behaviours. However, its design and implementation require intensive expertise and efforts in cyber and physical aspects of smart power systems as well as software/system engineering. While many industry players, including power grid operators, device vendors, research and education sectors are interested, availability of the smart grid cyber range is limited to a small number of research labs. To address this challenge, we have developed a framework for modelling a smart grid cyber range using an XML-based language, called SG-ML, and for "compiling" the model into an operational cyber range with minimal engineering efforts. The modelling language includes standardized schema from IEC 61850 and IEC 61131, which allows industry players to utilize their existing configurations. The SG-ML framework aims at making a smart grid cyber range available to broader user bases to facilitate cybersecurity R\&D and hands-on exercises.

Title: DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF

• Authors: Jie Long Lee, Chen Li, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00874
• Pdf URL: https://arxiv.org/pdf/2404.00874
• Copy Paste: [[2404.00874]] DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF(https://arxiv.org/abs/2404.00874)
• Keywords: diffusion
• Abstract: We present DiSR-NeRF, a diffusion-guided framework for view-consistent super-resolution (SR) NeRF. Unlike prior works, we circumvent the requirement for high-resolution (HR) reference images by leveraging existing powerful 2D super-resolution models. Nonetheless, independent SR 2D images are often inconsistent across different views. We thus propose Iterative 3D Synchronization (I3DS) to mitigate the inconsistency problem via the inherent multi-view consistency property of NeRF. Specifically, our I3DS alternates between upscaling low-resolution (LR) rendered images with diffusion models, and updating the underlying 3D representation with standard NeRF training. We further introduce Renoised Score Distillation (RSD), a novel score-distillation objective for 2D image resolution. Our RSD combines features from ancestral sampling and Score Distillation Sampling (SDS) to generate sharp images that are also LR-consistent. Qualitative and quantitative results on both synthetic and real-world datasets demonstrate that our DiSR-NeRF can achieve better results on NeRF super-resolution compared with existing works. Code and video results available at the project website.

Title: MGMap: Mask-Guided Learning for Online Vectorized HD Map Construction

• Authors: Xiaolu Liu, Song Wang, Wentong Li, Ruizi Yang, Junbo Chen, Jianke Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00876
• Pdf URL: https://arxiv.org/pdf/2404.00876
• Copy Paste: [[2404.00876]] MGMap: Mask-Guided Learning for Online Vectorized HD Map Construction(https://arxiv.org/abs/2404.00876)
• Keywords: robust, extraction
• Abstract: Currently, high-definition (HD) map construction leans towards a lightweight online generation tendency, which aims to preserve timely and reliable road scene information. However, map elements contain strong shape priors. Subtle and sparse annotations make current detection-based frameworks ambiguous in locating relevant feature scopes and cause the loss of detailed structures in prediction. To alleviate these problems, we propose MGMap, a mask-guided approach that effectively highlights the informative regions and achieves precise map element localization by introducing the learned masks. Specifically, MGMap employs learned masks based on the enhanced multi-scale BEV features from two perspectives. At the instance level, we propose the Mask-activated instance (MAI) decoder, which incorporates global instance and structural information into instance queries by the activation of instance masks. At the point level, a novel position-guided mask patch refinement (PG-MPR) module is designed to refine point locations from a finer-grained perspective, enabling the extraction of point-specific patch information. Compared to the baselines, our proposed MGMap achieves a notable improvement of around 10 mAP for different input modalities. Extensive experiments also demonstrate that our approach showcases strong robustness and generalization capabilities. Our code can be found at https://github.com/xiaolul2/MGMap.

Title: TryOn-Adapter: Efficient Fine-Grained Clothing Identity Adaptation for High-Fidelity Virtual Try-On

• Authors: Jiazheng Xing, Chao Xu, Yijie Qian, Yang Liu, Guang Dai, Baigui Sun, Yong Liu, Jingdong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00878
• Pdf URL: https://arxiv.org/pdf/2404.00878
• Copy Paste: [[2404.00878]] TryOn-Adapter: Efficient Fine-Grained Clothing Identity Adaptation for High-Fidelity Virtual Try-On(https://arxiv.org/abs/2404.00878)
• Keywords: diffusion
• Abstract: Virtual try-on focuses on adjusting the given clothes to fit a specific person seamlessly while avoiding any distortion of the patterns and textures of the garment. However, the clothing identity uncontrollability and training inefficiency of existing diffusion-based methods, which struggle to maintain the identity even with full parameter training, are significant limitations that hinder the widespread applications. In this work, we propose an effective and efficient framework, termed TryOn-Adapter. Specifically, we first decouple clothing identity into fine-grained factors: style for color and category information, texture for high-frequency details, and structure for smooth spatial adaptive transformation. Our approach utilizes a pre-trained exemplar-based diffusion model as the fundamental network, whose parameters are frozen except for the attention layers. We then customize three lightweight modules (Style Preserving, Texture Highlighting, and Structure Adapting) incorporated with fine-tuning techniques to enable precise and efficient identity control. Meanwhile, we introduce the training-free T-RePaint strategy to further enhance clothing identity preservation while maintaining the realistic try-on effect during the inference. Our experiments demonstrate that our approach achieves state-of-the-art performance on two widely-used benchmarks. Additionally, compared with recent full-tuning diffusion-based methods, we only use about half of their tunable parameters during training. The code will be made publicly available at https://github.com/jiazheng-xing/TryOn-Adapter.

Title: Model-Agnostic Human Preference Inversion in Diffusion Models

• Authors: Jeeyung Kim, Ze Wang, Qiang Qiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00879
• Pdf URL: https://arxiv.org/pdf/2404.00879
• Copy Paste: [[2404.00879]] Model-Agnostic Human Preference Inversion in Diffusion Models(https://arxiv.org/abs/2404.00879)
• Keywords: diffusion
• Abstract: Efficient text-to-image generation remains a challenging task due to the high computational costs associated with the multi-step sampling in diffusion models. Although distillation of pre-trained diffusion models has been successful in reducing sampling steps, low-step image generation often falls short in terms of quality. In this study, we propose a novel sampling design to achieve high-quality one-step image generation aligning with human preferences, particularly focusing on exploring the impact of the prior noise distribution. Our approach, Prompt Adaptive Human Preference Inversion (PAHI), optimizes the noise distributions for each prompt based on human preferences without the need for fine-tuning diffusion models. Our experiments showcase that the tailored noise distributions significantly improve image quality with only a marginal increase in computational cost. Our findings underscore the importance of noise optimization and pave the way for efficient and high-quality text-to-image synthesis.

Title: Rethinking the Relationship between Recurrent and Non-Recurrent Neural Networks: A Study in Sparsity

• Authors: Quincy Hershey, Randy Paffenroth, Harsh Pathak, Simon Tavener
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00880
• Pdf URL: https://arxiv.org/pdf/2404.00880
• Copy Paste: [[2404.00880]] Rethinking the Relationship between Recurrent and Non-Recurrent Neural Networks: A Study in Sparsity(https://arxiv.org/abs/2404.00880)
• Keywords: transformer
• Abstract: Neural networks (NN) can be divided into two broad categories, recurrent and non-recurrent. Both types of neural networks are popular and extensively studied, but they are often treated as distinct families of machine learning algorithms. In this position paper, we argue that there is a closer relationship between these two types of neural networks than is normally appreciated. We show that many common neural network models, such as Recurrent Neural Networks (RNN), Multi-Layer Perceptrons (MLP), and even deep multi-layer transformers, can all be represented as iterative maps. The close relationship between RNNs and other types of NNs should not be surprising. In particular, RNNs are known to be Turing complete, and therefore capable of representing any computable function (such as any other types of NNs), but herein we argue that the relationship runs deeper and is more practical than this. For example, RNNs are often thought to be more difficult to train than other types of NNs, with RNNs being plagued by issues such as vanishing or exploding gradients. However, as we demonstrate in this paper, MLPs, RNNs, and many other NNs lie on a continuum, and this perspective leads to several insights that illuminate both theoretical and practical aspects of NNs.

Title: Interpretable Multi-View Clustering Based on Anchor Graph Tensor Factorization

• Authors: Jing Li, Quanxue Gao, Cheng Deng, Qianqian Wang, Ming Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00883
• Pdf URL: https://arxiv.org/pdf/2404.00883
• Copy Paste: [[2404.00883]] Interpretable Multi-View Clustering Based on Anchor Graph Tensor Factorization(https://arxiv.org/abs/2404.00883)
• Keywords: interpretability
• Abstract: The clustering method based on the anchor graph has gained significant attention due to its exceptional clustering performance and ability to process large-scale data. One common approach is to learn bipartite graphs with K-connected components, helping avoid the need for post-processing. However, this method has strict parameter requirements and may not always get K-connected components. To address this issue, an alternative approach is to directly obtain the cluster label matrix by performing non-negative matrix factorization (NMF) on the anchor graph. Nevertheless, existing multi-view clustering methods based on anchor graph factorization lack adequate cluster interpretability for the decomposed matrix and often overlook the inter-view information. We address this limitation by using non-negative tensor factorization to decompose an anchor graph tensor that combines anchor graphs from multiple views. This approach allows us to consider inter-view information comprehensively. The decomposed tensors, namely the sample indicator tensor and the anchor indicator tensor, enhance the interpretability of the factorization. Extensive experiments validate the effectiveness of this method.

Title: Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models

• Authors: Wei He, Shichun Liu, Jun Zhao, Yiwen Ding, Yi Lu, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00884
• Pdf URL: https://arxiv.org/pdf/2404.00884
• Copy Paste: [[2404.00884]] Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models(https://arxiv.org/abs/2404.00884)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown promising abilities of in-context learning (ICL), adapting swiftly to new tasks with only few-shot demonstrations. However, current few-shot methods heavily depend on high-quality, query-specific demos, which are often lacking. When faced with out-of-demonstration (OOD) queries, methods that rely on hand-crafted demos or external retrievers might fail. To bridge the gap between limited demos and OOD queries, we propose Self-Demos, a novel prompting method that elicits the inherent generalizability in LLMs by query-aware demo generation. The generated demos strategically interpolate between existing demos and the given query, transforming the query from OOD to ID. To evaluate the effectiveness of our approach, we manually constructed OOD-Toolset, a dataset in the tool-using scenario with over 300 real-world APIs and 1000 instances, each consisting of three tool-use cases as demos and an OOD query. Thorough experiments on our dataset and two public math benchmarks have shown that our method can outperform state-of-the-art baselines in the OOD setting. Moreover, we conduct a range of analyses to validate Self-Demos's generalization and provide more insights.

Title: Machine Learning Robustness: A Primer

• Authors: Houssem Ben Braiek, Foutse Khomh
• Subjects: cs.LG, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.00897
• Pdf URL: https://arxiv.org/pdf/2404.00897
• Copy Paste: [[2404.00897]] Machine Learning Robustness: A Primer(https://arxiv.org/abs/2404.00897)
• Keywords: attack, robust, explainability
• Abstract: This chapter explores the foundational concept of robustness in Machine Learning (ML) and its integral role in establishing trustworthiness in Artificial Intelligence (AI) systems. The discussion begins with a detailed definition of robustness, portraying it as the ability of ML models to maintain stable performance across varied and unexpected environmental conditions. ML robustness is dissected through several lenses: its complementarity with generalizability; its status as a requirement for trustworthy AI; its adversarial vs non-adversarial aspects; its quantitative metrics; and its indicators such as reproducibility and explainability. The chapter delves into the factors that impede robustness, such as data bias, model complexity, and the pitfalls of underspecified ML pipelines. It surveys key techniques for robustness assessment from a broad perspective, including adversarial attacks, encompassing both digital and physical realms. It covers non-adversarial data shifts and nuances of Deep Learning (DL) software testing methodologies. The discussion progresses to explore amelioration strategies for bolstering robustness, starting with data-centric approaches like debiasing and augmentation. Further examination includes a variety of model-centric methods such as transfer learning, adversarial training, and randomized smoothing. Lastly, post-training methods are discussed, including ensemble techniques, pruning, and model repairs, emerging as cost-effective strategies to make models more resilient against the unpredictable. This chapter underscores the ongoing challenges and limitations in estimating and achieving ML robustness by existing approaches. It offers insights and directions for future research on this crucial concept, as a prerequisite for trustworthy AI systems.

Title: CAAP: Class-Dependent Automatic Data Augmentation Based On Adaptive Policies For Time Series

• Authors: Tien-Yu Chang, Hao Dai, Vincent S. Tseng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00898
• Pdf URL: https://arxiv.org/pdf/2404.00898
• Copy Paste: [[2404.00898]] CAAP: Class-Dependent Automatic Data Augmentation Based On Adaptive Policies For Time Series(https://arxiv.org/abs/2404.00898)
• Keywords: extraction
• Abstract: Data Augmentation is a common technique used to enhance the performance of deep learning models by expanding the training dataset. Automatic Data Augmentation (ADA) methods are getting popular because of their capacity to generate policies for various datasets. However, existing ADA methods primarily focused on overall performance improvement, neglecting the problem of class-dependent bias that leads to performance reduction in specific classes. This bias poses significant challenges when deploying models in real-world applications. Furthermore, ADA for time series remains an underexplored domain, highlighting the need for advancements in this field. In particular, applying ADA techniques to vital signals like an electrocardiogram (ECG) is a compelling example due to its potential in medical domains such as heart disease diagnostics. We propose a novel deep learning-based approach called Class-dependent Automatic Adaptive Policies (CAAP) framework to overcome the notable class-dependent bias problem while maintaining the overall improvement in time-series data augmentation. Specifically, we utilize the policy network to generate effective sample-wise policies with balanced difficulty through class and feature information extraction. Second, we design the augmentation probability regulation method to minimize class-dependent bias. Third, we introduce the information region concepts into the ADA framework to preserve essential regions in the sample. Through a series of experiments on real-world ECG datasets, we demonstrate that CAAP outperforms representative methods in achieving lower class-dependent bias combined with superior overall performance. These results highlight the reliability of CAAP as a promising ADA method for time series modeling that fits for the demands of real-world applications.

Title: TM-TREK at SemEval-2024 Task 8: Towards LLM-Based Automatic Boundary Detection for Human-Machine Mixed Text

• Authors: Xiaoyan Qu, Xiangfeng Meng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00899
• Pdf URL: https://arxiv.org/pdf/2404.00899
• Copy Paste: [[2404.00899]] TM-TREK at SemEval-2024 Task 8: Towards LLM-Based Automatic Boundary Detection for Human-Machine Mixed Text(https://arxiv.org/abs/2404.00899)
• Keywords: large language model, segmentation
• Abstract: With the increasing prevalence of text generated by large language models (LLMs), there is a growing concern about distinguishing between LLM-generated and human-written texts in order to prevent the misuse of LLMs, such as the dissemination of misleading information and academic dishonesty. Previous research has primarily focused on classifying text as either entirely human-written or LLM-generated, neglecting the detection of mixed texts that contain both types of content. This paper explores LLMs' ability to identify boundaries in human-written and machine-generated mixed texts. We approach this task by transforming it into a token classification problem and regard the label turning point as the boundary. Notably, our ensemble model of LLMs achieved first place in the 'Human-Machine Mixed Text Detection' sub-task of the SemEval'24 Competition Task 8. Additionally, we investigate factors that influence the capability of LLMs in detecting boundaries within mixed texts, including the incorporation of extra layers on top of LLMs, combination of segmentation loss, and the impact of pretraining. Our findings aim to provide valuable insights for future research in this area.

Title: Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning

• Authors: Rongjie Li, Yu Wu, Xuming He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00909
• Pdf URL: https://arxiv.org/pdf/2404.00909
• Copy Paste: [[2404.00909]] Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning(https://arxiv.org/abs/2404.00909)
• Keywords: generative, large language model
• Abstract: Generative vision-language models (VLMs) have shown impressive performance in zero-shot vision-language tasks like image captioning and visual question answering. However, improving their zero-shot reasoning typically requires second-stage instruction tuning, which relies heavily on human-labeled or large language model-generated annotation, incurring high labeling costs. To tackle this challenge, we introduce Image-Conditioned Caption Correction (ICCC), a novel pre-training task designed to enhance VLMs' zero-shot performance without the need for labeled task-aware data. The ICCC task compels VLMs to rectify mismatches between visual and language concepts, thereby enhancing instruction following and text generation conditioned on visual inputs. Leveraging language structure and a lightweight dependency parser, we construct data samples of ICCC task from image-text datasets with low labeling and computation costs. Experimental results on BLIP-2 and InstructBLIP demonstrate significant improvements in zero-shot image-text generation-based VL tasks through ICCC instruction tuning.

Title: LLaMA-Excitor: General Instruction Tuning via Indirect Feature Interaction

• Authors: Bo Zou, Chao Yang, Yu Qiao, Chengbin Quan, Youjian Zhao
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00913
• Pdf URL: https://arxiv.org/pdf/2404.00913
• Copy Paste: [[2404.00913]] LLaMA-Excitor: General Instruction Tuning via Indirect Feature Interaction(https://arxiv.org/abs/2404.00913)
• Keywords: transformer
• Abstract: Existing methods to fine-tune LLMs, like Adapter, Prefix-tuning, and LoRA, which introduce extra modules or additional input sequences to inject new skills or knowledge, may compromise the innate abilities of LLMs. In this paper, we propose LLaMA-Excitor, a lightweight method that stimulates the LLMs' potential to better follow instructions by gradually paying more attention to worthwhile information. Specifically, the LLaMA-Excitor does not directly change the intermediate hidden state during the self-attention calculation of the transformer structure. We designed the Excitor block as a bypass module for the similarity score computation in LLMs' self-attention to reconstruct keys and change the importance of values by learnable prompts. LLaMA-Excitor ensures a self-adaptive allocation of additional attention to input instructions, thus effectively preserving LLMs' pre-trained knowledge when fine-tuning LLMs on low-quality instruction-following datasets. Furthermore, we unify the modeling of multi-modal tuning and language-only tuning, extending LLaMA-Excitor to a powerful visual instruction follower without the need for complex multi-modal alignment. Our proposed approach is evaluated in language-only and multi-modal tuning experimental scenarios. Notably, LLaMA-Excitor is the only method that maintains basic capabilities while achieving a significant improvement (+6%) on the MMLU benchmark. In the visual instruction tuning, we achieve a new state-of-the-art image captioning performance of 157.5 CIDEr on MSCOCO, and a comparable performance (88.39%) on ScienceQA to cutting-edge models with more parameters and extensive vision-language pertaining.

Title: Token-Efficient Leverage Learning in Large Language Models

• Authors: Yuanhao Zeng, Min Wang, Yihang Wang, Yingxia Shao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00914
• Pdf URL: https://arxiv.org/pdf/2404.00914
• Copy Paste: [[2404.00914]] Token-Efficient Leverage Learning in Large Language Models(https://arxiv.org/abs/2404.00914)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have excelled in various tasks but perform better in high-resource scenarios, which presents challenges in low-resource scenarios. Data scarcity and the inherent difficulty of adapting LLMs to specific tasks compound the challenge. To address the twin hurdles, we introduce \textbf{Leverage Learning}. We present a streamlined implement of this methodology called Token-Efficient Leverage Learning (TELL). TELL showcases the potential of Leverage Learning, demonstrating effectiveness across various LLMs and low-resource tasks, ranging from $10^4$ to $10^6$ tokens. It reduces task data requirements by up to nearly an order of magnitude compared to conventional Supervised Fine-Tuning (SFT) while delivering competitive performance. With the same amount of task data, TELL leads in improving task performance compared to SFT. We discuss the mechanism of Leverage Learning, suggesting it aligns with quantization hypothesis and explore its promising potential through empirical testing.

Title: Scalable 3D Registration via Truncated Entry-wise Absolute Residuals

• Authors: Tianyu Huang, Liangzu Peng, René Vidal, Yun-Hui Liu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.00915
• Pdf URL: https://arxiv.org/pdf/2404.00915
• Copy Paste: [[2404.00915]] Scalable 3D Registration via Truncated Entry-wise Absolute Residuals(https://arxiv.org/abs/2404.00915)
• Keywords: robust
• Abstract: Given an input set of $3$D point pairs, the goal of outlier-robust $3$D registration is to compute some rotation and translation that align as many point pairs as possible. This is an important problem in computer vision, for which many highly accurate approaches have been recently proposed. Despite their impressive performance, these approaches lack scalability, often overflowing the $16$GB of memory of a standard laptop to handle roughly $30,000$ point pairs. In this paper, we propose a $3$D registration approach that can process more than ten million ($10^7$) point pairs with over $99\%$ random outliers. Moreover, our method is efficient, entails low memory costs, and maintains high accuracy at the same time. We call our method TEAR, as it involves minimizing an outlier-robust loss that computes Truncated Entry-wise Absolute Residuals. To minimize this loss, we decompose the original $6$-dimensional problem into two subproblems of dimensions $3$ and $2$, respectively, solved in succession to global optimality via a customized branch-and-bound method. While branch-and-bound is often slow and unscalable, this does not apply to TEAR as we propose novel bounding functions that are tight and computationally efficient. Experiments on various datasets are conducted to validate the scalability and efficiency of our method.

Title: Rethinking Saliency-Guided Weakly-Supervised Semantic Segmentation

• Authors: Beomyoung Kim, Donghyeon Kim, Sung Ju Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00918
• Pdf URL: https://arxiv.org/pdf/2404.00918
• Copy Paste: [[2404.00918]] Rethinking Saliency-Guided Weakly-Supervised Semantic Segmentation(https://arxiv.org/abs/2404.00918)
• Keywords: segmentation
• Abstract: This paper presents a fresh perspective on the role of saliency maps in weakly-supervised semantic segmentation (WSSS) and offers new insights and research directions based on our empirical findings. We conduct comprehensive experiments and observe that the quality of the saliency map is a critical factor in saliency-guided WSSS approaches. Nonetheless, we find that the saliency maps used in previous works are often arbitrarily chosen, despite their significant impact on WSSS. Additionally, we observe that the choice of the threshold, which has received less attention before, is non-trivial in WSSS. To facilitate more meaningful and rigorous research for saliency-guided WSSS, we introduce \texttt{WSSS-BED}, a standardized framework for conducting research under unified conditions. \texttt{WSSS-BED} provides various saliency maps and activation maps for seven WSSS methods, as well as saliency maps from unsupervised salient object detection models.

Title: Towards Label-Efficient Human Matting: A Simple Baseline for Weakly Semi-Supervised Trimap-Free Human Matting

• Authors: Beomyoung Kim, Myeong Yeon Yi, Joonsang Yu, Young Joon Yoo, Sung Ju Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00921
• Pdf URL: https://arxiv.org/pdf/2404.00921
• Copy Paste: [[2404.00921]] Towards Label-Efficient Human Matting: A Simple Baseline for Weakly Semi-Supervised Trimap-Free Human Matting(https://arxiv.org/abs/2404.00921)
• Keywords: robust, segmentation
• Abstract: This paper presents a new practical training method for human matting, which demands delicate pixel-level human region identification and significantly laborious annotations. To reduce the annotation cost, most existing matting approaches often rely on image synthesis to augment the dataset. However, the unnaturalness of synthesized training images brings in a new domain generalization challenge for natural images. To address this challenge, we introduce a new learning paradigm, weakly semi-supervised human matting (WSSHM), which leverages a small amount of expensive matte labels and a large amount of budget-friendly segmentation labels, to save the annotation cost and resolve the domain generalization problem. To achieve the goal of WSSHM, we propose a simple and effective training method, named Matte Label Blending (MLB), that selectively guides only the beneficial knowledge of the segmentation and matte data to the matting model. Extensive experiments with our detailed analysis demonstrate our method can substantially improve the robustness of the matting model using a few matte data and numerous segmentation data. Our training method is also easily applicable to real-time models, achieving competitive accuracy with breakneck inference speed (328 FPS on NVIDIA V100 GPU). The implementation code is available at \url{https://github.com/clovaai/WSSHM}.

Title: Towards Memorization-Free Diffusion Models

• Authors: Chen Chen, Daochang Liu, Chang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00922
• Pdf URL: https://arxiv.org/pdf/2404.00922
• Copy Paste: [[2404.00922]] Towards Memorization-Free Diffusion Models(https://arxiv.org/abs/2404.00922)
• Keywords: diffusion
• Abstract: Pretrained diffusion models and their outputs are widely accessible due to their exceptional capacity for synthesizing high-quality images and their open-source nature. The users, however, may face litigation risks owing to the models' tendency to memorize and regurgitate training data during inference. To address this, we introduce Anti-Memorization Guidance (AMG), a novel framework employing three targeted guidance strategies for the main causes of memorization: image and caption duplication, and highly specific user prompts. Consequently, AMG ensures memorization-free outputs while maintaining high image quality and text alignment, leveraging the synergy of its guidance methods, each indispensable in its own right. AMG also features an innovative automatic detection system for potential memorization during each step of inference process, allows selective application of guidance strategies, minimally interfering with the original sampling process to preserve output utility. We applied AMG to pretrained Denoising Diffusion Probabilistic Models (DDPM) and Stable Diffusion across various generation tasks. The results demonstrate that AMG is the first approach to successfully eradicates all instances of memorization with no or marginal impacts on image quality and text-alignment, as evidenced by FID and CLIP scores.

Title: BadPart: Unified Black-box Adversarial Patch Attacks against Pixel-wise Regression Tasks

• Authors: Zhiyuan Cheng, Zhaoyi Liu, Tengda Guo, Shiwei Feng, Dongfang Liu, Mingjie Tang, Xiangyu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00924
• Pdf URL: https://arxiv.org/pdf/2404.00924
• Copy Paste: [[2404.00924]] BadPart: Unified Black-box Adversarial Patch Attacks against Pixel-wise Regression Tasks(https://arxiv.org/abs/2404.00924)
• Keywords: security, attack, robust
• Abstract: Pixel-wise regression tasks (e.g., monocular depth estimation (MDE) and optical flow estimation (OFE)) have been widely involved in our daily life in applications like autonomous driving, augmented reality and video composition. Although certain applications are security-critical or bear societal significance, the adversarial robustness of such models are not sufficiently studied, especially in the black-box scenario. In this work, we introduce the first unified black-box adversarial patch attack framework against pixel-wise regression tasks, aiming to identify the vulnerabilities of these models under query-based black-box attacks. We propose a novel square-based adversarial patch optimization framework and employ probabilistic square sampling and score-based gradient estimation techniques to generate the patch effectively and efficiently, overcoming the scalability problem of previous black-box patch attacks. Our attack prototype, named BadPart, is evaluated on both MDE and OFE tasks, utilizing a total of 7 models. BadPart surpasses 3 baseline methods in terms of both attack performance and efficiency. We also apply BadPart on the Google online service for portrait depth estimation, causing 43.5% relative distance error with 50K queries. State-of-the-art (SOTA) countermeasures cannot defend our attack effectively.

Title: LLMs are Good Sign Language Translators

• Authors: Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, Jun Liu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00925
• Pdf URL: https://arxiv.org/pdf/2404.00925
• Copy Paste: [[2404.00925]] LLMs are Good Sign Language Translators(https://arxiv.org/abs/2404.00925)
• Keywords: large language model
• Abstract: Sign Language Translation (SLT) is a challenging task that aims to translate sign videos into spoken language. Inspired by the strong translation capabilities of large language models (LLMs) that are trained on extensive multilingual text corpora, we aim to harness off-the-shelf LLMs to handle SLT. In this paper, we regularize the sign videos to embody linguistic characteristics of spoken language, and propose a novel SignLLM framework to transform sign videos into a language-like representation for improved readability by off-the-shelf LLMs. SignLLM comprises two key modules: (1) The Vector-Quantized Visual Sign module converts sign videos into a sequence of discrete character-level sign tokens, and (2) the Codebook Reconstruction and Alignment module converts these character-level tokens into word-level sign representations using an optimal transport formulation. A sign-text alignment loss further bridges the gap between sign and text tokens, enhancing semantic compatibility. We achieve state-of-the-art gloss-free results on two widely-used SLT benchmarks.

Title: Instance-Aware Group Quantization for Vision Transformers

• Authors: Jaehyeon Moon, Dohyung Kim, Junyong Cheon, Bumsub Ham
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00928
• Pdf URL: https://arxiv.org/pdf/2404.00928
• Copy Paste: [[2404.00928]] Instance-Aware Group Quantization for Vision Transformers(https://arxiv.org/abs/2404.00928)
• Keywords: transformer, segmentation
• Abstract: Post-training quantization (PTQ) is an efficient model compression technique that quantizes a pretrained full-precision model using only a small calibration set of unlabeled samples without retraining. PTQ methods for convolutional neural networks (CNNs) provide quantization results comparable to full-precision counterparts. Directly applying them to vision transformers (ViTs), however, incurs severe performance degradation, mainly due to the differences in architectures between CNNs and ViTs. In particular, the distribution of activations for each channel vary drastically according to input instances, making PTQ methods for CNNs inappropriate for ViTs. To address this, we introduce instance-aware group quantization for ViTs (IGQ-ViT). To this end, we propose to split the channels of activation maps into multiple groups dynamically for each input instance, such that activations within each group share similar statistical properties. We also extend our scheme to quantize softmax attentions across tokens. In addition, the number of groups for each layer is adjusted to minimize the discrepancies between predictions from quantized and full-precision models, under a bit-operation (BOP) constraint. We show extensive experimental results on image classification, object detection, and instance segmentation, with various transformer architectures, demonstrating the effectiveness of our approach.

Title: A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias

• Authors: Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Yuqi Ye, Hanwen Gu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00929
• Pdf URL: https://arxiv.org/pdf/2404.00929
• Copy Paste: [[2404.00929]] A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias(https://arxiv.org/abs/2404.00929)
• Keywords: large language model
• Abstract: Based on the foundation of Large Language Models (LLMs), Multilingual Large Language Models (MLLMs) have been developed to address the challenges of multilingual natural language processing tasks, hoping to achieve knowledge transfer from high-resource to low-resource languages. However, significant limitations and challenges still exist, such as language imbalance, multilingual alignment, and inherent bias. In this paper, we aim to provide a comprehensive analysis of MLLMs, delving deeply into discussions surrounding these critical issues. First of all, we start by presenting an overview of MLLMs, covering their evolution, key techniques, and multilingual capacities. Secondly, we explore widely utilized multilingual corpora for MLLMs' training and multilingual datasets oriented for downstream tasks that are crucial for enhancing the cross-lingual capability of MLLMs. Thirdly, we survey the existing studies on multilingual representations and investigate whether the current MLLMs can learn a universal language representation. Fourthly, we discuss bias on MLLMs including its category and evaluation metrics, and summarize the existing debiasing techniques. Finally, we discuss existing challenges and point out promising research directions. By demonstrating these aspects, this paper aims to facilitate a deeper understanding of MLLMs and their potentiality in various domains.

Title: PSYDIAL: Personality-based Synthetic Dialogue Generation using Large Language Models

• Authors: Ji-Eun Han, Jun-Seok Koh, Hyeon-Tae Seo, Du-Seong Chang, Kyung-Ah Sohn
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00930
• Pdf URL: https://arxiv.org/pdf/2404.00930
• Copy Paste: [[2404.00930]] PSYDIAL: Personality-based Synthetic Dialogue Generation using Large Language Models(https://arxiv.org/abs/2404.00930)
• Keywords: large language model
• Abstract: We present a novel end-to-end personality-based synthetic dialogue data generation pipeline, specifically designed to elicit responses from large language models via prompting. We design the prompts to generate more human-like dialogues considering real-world scenarios when users engage with chatbots. We introduce PSYDIAL, the first Korean dialogue dataset focused on personality-based dialogues, curated using our proposed pipeline. Notably, we focus on the Extraversion dimension of the Big Five personality model in our research. Experimental results indicate that while pre-trained models and those fine-tuned with a chit-chat dataset struggle to generate responses reflecting personality, models trained with PSYDIAL show significant improvements. The versatility of our pipeline extends beyond dialogue tasks, offering potential for other non-dialogue related applications. This research opens doors for more nuanced, personality-driven conversational AI in Korean and potentially other languages. Our code is publicly available at https://github.com/jiSilverH/psydial.

Title: GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields

• Authors: Yunsong Wang, Hanlin Chen, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00931
• Pdf URL: https://arxiv.org/pdf/2404.00931
• Copy Paste: [[2404.00931]] GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields(https://arxiv.org/abs/2404.00931)
• Keywords: segmentation
• Abstract: Recent advancements in vision-language foundation models have significantly enhanced open-vocabulary 3D scene understanding. However, the generalizability of existing methods is constrained due to their framework designs and their reliance on 3D data. We address this limitation by introducing Generalizable Open-Vocabulary Neural Semantic Fields (GOV-NeSF), a novel approach offering a generalizable implicit representation of 3D scenes with open-vocabulary semantics. We aggregate the geometry-aware features using a cost volume, and propose a Multi-view Joint Fusion module to aggregate multi-view features through a cross-view attention mechanism, which effectively predicts view-specific blending weights for both colors and open-vocabulary features. Remarkably, our GOV-NeSF exhibits state-of-the-art performance in both 2D and 3D open-vocabulary semantic segmentation, eliminating the need for ground truth semantic labels or depth priors, and effectively generalize across scenes and datasets without fine-tuning.

Title: ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback

• Authors: Zhenyu Hou, Yiin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, Yuxiao Dong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00934
• Pdf URL: https://arxiv.org/pdf/2404.00934
• Copy Paste: [[2404.00934]] ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback(https://arxiv.org/abs/2404.00934)
• Keywords: large language model
• Abstract: ChatGLM is a free-to-use AI service powered by the ChatGLM family of large language models (LLMs). In this paper, we present the ChatGLM-RLHF pipeline -- a reinforcement learning from human feedback (RLHF) system -- designed to enhance ChatGLM's alignment with human preferences. ChatGLM-RLHF encompasses three major components: the collection of human preference data, the training of the reward model, and the optimization of policies. Throughout the process of integrating ChatGLM-RLHF into production, we encountered and addressed several unprecedented challenges. We introduce the strategies to mitigate reward variance for stabilized large-scale training, implement model parallelism with fused gradient-descent, and design regularization constraints to avoid catastrophic forgetting in LLMs. Experiments show that ChatGLM-RLHF brings significant improvements in alignment tasks compared to the supervised fine-tuned (SFT) version of ChatGLM. For instance, it achieves on average 15\% more wins against ChatGLM-SFT in Chinese alignment tasks. The work presents our practices of aligning LLMs with human preferences, offering insights into the challenges and solutions in RLHF implementations.

Title: Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs

• Authors: Xiaoze Liu, Feijie Wu, Tianyang Xu, Zhuo Chen, Yichi Zhang, Xiaoqian Wang, Jing Gao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.00942
• Pdf URL: https://arxiv.org/pdf/2404.00942
• Copy Paste: [[2404.00942]] Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs(https://arxiv.org/abs/2404.00942)
• Keywords: large language model
• Abstract: The advent of Large Language Models (LLMs) has significantly transformed the AI landscape, enhancing machine learning and AI capabilities. Factuality issue is a critical concern for LLMs, as they may generate factually incorrect responses. In this paper, we propose GraphEval to evaluate an LLM's performance using a substantially large test dataset. Specifically, the test dataset is retrieved from a large knowledge graph with more than 10 million facts without expensive human efforts. Unlike conventional methods that evaluate LLMs based on generated responses, GraphEval streamlines the evaluation process by creating a judge model to estimate the correctness of the answers given by the LLM. Our experiments demonstrate that the judge model's factuality assessment aligns closely with the correctness of the LLM's generated outputs, while also substantially reducing evaluation costs. Besides, our findings offer valuable insights into LLM performance across different metrics and highlight the potential for future improvements in ensuring the factual integrity of LLM outputs. The code is publicly available at https://github.com/xz-liu/GraphEval.

Title: Evalverse: Unified and Accessible Library for Large Language Model Evaluation

• Authors: Jihoo Kim, Wonho Song, Dahyun Kim, Yunsu Kim, Yungi Kim, Chanjun Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00943
• Pdf URL: https://arxiv.org/pdf/2404.00943
• Copy Paste: [[2404.00943]] Evalverse: Unified and Accessible Library for Large Language Model Evaluation(https://arxiv.org/abs/2404.00943)
• Keywords: large language model
• Abstract: This paper introduces Evalverse, a novel library that streamlines the evaluation of Large Language Models (LLMs) by unifying disparate evaluation tools into a single, user-friendly framework. Evalverse enables individuals with limited knowledge of artificial intelligence to easily request LLM evaluations and receive detailed reports, facilitated by an integration with communication platforms like Slack. Thus, Evalverse serves as a powerful tool for the comprehensive assessment of LLMs, offering both researchers and practitioners a centralized and easily accessible evaluation framework. Finally, we also provide a demo video for Evalverse, showcasing its capabilities and implementation in a two-minute format.

Title: Exploring the Efficacy of Group-Normalization in Deep Learning Models for Alzheimer's Disease Classification

• Authors: Gousia Habib, Ishfaq Ahmed Malik, Jameel Ahmad, Imtiaz Ahmed, Shaima Qureshi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00946
• Pdf URL: https://arxiv.org/pdf/2404.00946
• Copy Paste: [[2404.00946]] Exploring the Efficacy of Group-Normalization in Deep Learning Models for Alzheimer's Disease Classification(https://arxiv.org/abs/2404.00946)
• Keywords: segmentation
• Abstract: Batch Normalization is an important approach to advancing deep learning since it allows multiple networks to train simultaneously. A problem arises when normalizing along the batch dimension because B.N.'s error increases significantly as batch size shrinks because batch statistics estimates are inaccurate. As a result, computer vision tasks like detection, segmentation, and video, which require tiny batches based on memory consumption, aren't suitable for using Batch Normalization for larger model training and feature transfer. Here, we explore Group Normalization as an easy alternative to using Batch Normalization A Group Normalization is a channel normalization method in which each group is divided into different channels, and the corresponding mean and variance are calculated for each group. Group Normalization computations are accurate across a wide range of batch sizes and are independent of batch size. When trained using a large ImageNet database on ResNet-50, GN achieves a very low error rate of 10.6% compared to Batch Normalization. when a smaller batch size of only 2 is used. For usual batch sizes, the performance of G.N. is comparable to that of Batch Normalization, but at the same time, it outperforms other normalization techniques. Implementing Group Normalization as a direct alternative to B.N to combat the serious challenges faced by the Batch Normalization in deep learning models with comparable or improved classification accuracy. Additionally, Group Normalization can be naturally transferred from the pre-training to the fine-tuning phase. .

Title: Harnessing The Power of Attention For Patch-Based Biomedical Image Classification

• Authors: Gousia Habib, Shaima Qureshi, Malik ishfaq
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00949
• Pdf URL: https://arxiv.org/pdf/2404.00949
• Copy Paste: [[2404.00949]] Harnessing The Power of Attention For Patch-Based Biomedical Image Classification(https://arxiv.org/abs/2404.00949)
• Keywords: robust
• Abstract: Biomedical image analysis can be facilitated by an innovative architecture rooted in self-attention mechanisms. The traditional convolutional neural network (CNN), characterized by fixed-sized windows, needs help capturing intricate spatial and temporal relations at the pixel level. The immutability of CNN filter weights post-training further restricts input fluctuations. Recognizing these limitations, we propose a new paradigm of attention-based models instead of convolutions. As an alternative to traditional CNNs, these models demonstrate robust modelling capabilities and the ability to grasp comprehensive long-range contextual information efficiently. Providing a solution to critical challenges faced by attention-based vision models such as inductive bias, weight sharing, receptive field limitations, and data handling in high resolution, our work combines non-overlapping (vanilla patching) with novel overlapped Shifted Patching Techniques (S.P.T.s) to induce local context that enhances model generalization. Moreover, we examine the novel Lancoz5 interpolation technique, which adapts variable image sizes to higher resolutions. Experimental evidence validates our model's generalization effectiveness, comparing favourably with existing approaches. Attention-based methods are particularly effective with ample data, especially when advanced data augmentation methodologies are integrated to strengthen their robustness.

Title: AISPACE at SemEval-2024 task 8: A Class-balanced Soft-voting System for Detecting Multi-generator Machine-generated Text

• Authors: Renhua Gu, Xiangfeng Meng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00950
• Pdf URL: https://arxiv.org/pdf/2404.00950
• Copy Paste: [[2404.00950]] AISPACE at SemEval-2024 task 8: A Class-balanced Soft-voting System for Detecting Multi-generator Machine-generated Text(https://arxiv.org/abs/2404.00950)
• Keywords: transformer, large language model
• Abstract: SemEval-2024 Task 8 provides a challenge to detect human-written and machine-generated text. There are 3 subtasks for different detection scenarios. This paper proposes a system that mainly deals with Subtask B. It aims to detect if given full text is written by human or is generated by a specific Large Language Model (LLM), which is actually a multi-class text classification task. Our team AISPACE conducted a systematic study of fine-tuning transformer-based models, including encoderonly, decoder-only and encoder-decoder models. We compared their performance on this task and identified that encoder-only models performed exceptionally well. We also applied a weighted Cross Entropy loss function to address the issue of data imbalance of different class samples. Additionally, we employed softvoting strategy over multi-models ensemble to enhance the reliability of our predictions. Our system ranked top 1 in Subtask B, which sets a state-of-the-art benchmark for this new challenge.

Title: Diffusion-Driven Domain Adaptation for Generating 3D Molecules

• Authors: Haokai Hong, Wanyu Lin, Kay Chen Tan
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2404.00962
• Pdf URL: https://arxiv.org/pdf/2404.00962
• Copy Paste: [[2404.00962]] Diffusion-Driven Domain Adaptation for Generating 3D Molecules(https://arxiv.org/abs/2404.00962)
• Keywords: diffusion, generative
• Abstract: Can we train a molecule generator that can generate 3D molecules from a new domain, circumventing the need to collect data? This problem can be cast as the problem of domain adaptive molecule generation. This work presents a novel and principled diffusion-based approach, called GADM, that allows shifting a generative model to desired new domains without the need to collect even a single molecule. As the domain shift is typically caused by the structure variations of molecules, e.g., scaffold variations, we leverage a designated equivariant masked autoencoder (MAE) along with various masking strategies to capture the structural-grained representations of the in-domain varieties. In particular, with an asymmetric encoder-decoder module, the MAE can generalize to unseen structure variations from the target domains. These structure variations are encoded with an equivariant encoder and treated as domain supervisors to control denoising. We show that, with these encoded structural-grained domain supervisors, GADM can generate effective molecules within the desired new domains. We conduct extensive experiments across various domain adaptation tasks over benchmarking datasets. We show that our approach can improve up to 65.6% in terms of success rate defined based on molecular validity, uniqueness, and novelty compared to alternative baselines.

Title: S2RC-GCN: A Spatial-Spectral Reliable Contrastive Graph Convolutional Network for Complex Land Cover Classification Using Hyperspectral Images

• Authors: Renxiang Guan, Zihao Li, Chujia Song, Guo Yu, Xianju Li, Ruyi Feng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00964
• Pdf URL: https://arxiv.org/pdf/2404.00964
• Copy Paste: [[2404.00964]] S2RC-GCN: A Spatial-Spectral Reliable Contrastive Graph Convolutional Network for Complex Land Cover Classification Using Hyperspectral Images(https://arxiv.org/abs/2404.00964)
• Keywords: robust
• Abstract: Spatial correlations between different ground objects are an important feature of mining land cover research. Graph Convolutional Networks (GCNs) can effectively capture such spatial feature representations and have demonstrated promising results in performing hyperspectral imagery (HSI) classification tasks of complex land. However, the existing GCN-based HSI classification methods are prone to interference from redundant information when extracting complex features. To classify complex scenes more effectively, this study proposes a novel spatial-spectral reliable contrastive graph convolutional classification framework named S2RC-GCN. Specifically, we fused the spectral and spatial features extracted by the 1D- and 2D-encoder, and the 2D-encoder includes an attention model to automatically extract important information. We then leveraged the fused high-level features to construct graphs and fed the resulting graphs into the GCNs to determine more effective graph representations. Furthermore, a novel reliable contrastive graph convolution was proposed for reliable contrastive learning to learn and fuse robust features. Finally, to test the performance of the model on complex object classification, we used imagery taken by Gaofen-5 in the Jiang Xia area to construct complex land cover datasets. The test results show that compared with other models, our model achieved the best results and effectively improved the classification performance of complex remote sensing imagery.

Title: VideoDistill: Language-aware Vision Distillation for Video Question Answering

• Authors: Bo Zou, Chao Yang, Yu Qiao, Chengbin Quan, Youjian Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00973
• Pdf URL: https://arxiv.org/pdf/2404.00973
• Copy Paste: [[2404.00973]] VideoDistill: Language-aware Vision Distillation for Video Question Answering(https://arxiv.org/abs/2404.00973)
• Keywords: extraction
• Abstract: Significant advancements in video question answering (VideoQA) have been made thanks to thriving large image-language pretraining frameworks. Although these image-language models can efficiently represent both video and language branches, they typically employ a goal-free vision perception process and do not interact vision with language well during the answer generation, thus omitting crucial visual cues. In this paper, we are inspired by the human recognition and learning pattern and propose VideoDistill, a framework with language-aware (i.e., goal-driven) behavior in both vision perception and answer generation process. VideoDistill generates answers only from question-related visual embeddings and follows a thinking-observing-answering approach that closely resembles human behavior, distinguishing it from previous research. Specifically, we develop a language-aware gating mechanism to replace the standard cross-attention, avoiding language's direct fusion into visual representations. We incorporate this mechanism into two key components of the entire framework. The first component is a differentiable sparse sampling module, which selects frames containing the necessary dynamics and semantics relevant to the questions. The second component is a vision refinement module that merges existing spatial-temporal attention layers to ensure the extraction of multi-grained visual semantics associated with the questions. We conduct experimental evaluations on various challenging video question-answering benchmarks, and VideoDistill achieves state-of-the-art performance in both general and long-form VideoQA datasets. In Addition, we verify that VideoDistill can effectively alleviate the utilization of language shortcut solutions in the EgoTaskQA dataset.

Title: Prior Constraints-based Reward Model Training for Aligning Large Language Models

• Authors: Hang Zhou, Chenglong Wang, Yimin Hu, Tong Xiao, Chunliang Zhang, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00978
• Pdf URL: https://arxiv.org/pdf/2404.00978
• Copy Paste: [[2404.00978]] Prior Constraints-based Reward Model Training for Aligning Large Language Models(https://arxiv.org/abs/2404.00978)
• Keywords: large language model
• Abstract: Reinforcement learning with human feedback for aligning large language models (LLMs) trains a reward model typically using ranking loss with comparison pairs.However, the training procedure suffers from an inherent problem: the uncontrolled scaling of reward scores during reinforcement learning due to the lack of constraints while training the reward model.This paper proposes a Prior Constraints-based Reward Model (namely PCRM) training method to mitigate this problem. PCRM incorporates prior constraints, specifically, length ratio and cosine similarity between outputs of each comparison pair, during reward model training to regulate optimization magnitude and control score margins. We comprehensively evaluate PCRM by examining its rank correlation with human preferences and its effectiveness in aligning LLMs via RL. Experimental results demonstrate that PCRM significantly improves alignment performance by effectively constraining reward score scaling. As another bonus, our method is easily integrated into arbitrary rank-based alignment methods, such as direct preference optimization, and can yield consistent improvement.

Title: PDF: A Probability-Driven Framework for Open World 3D Point Cloud Semantic Segmentation

• Authors: Jinfeng Xu, Siyuan Yang, Xianzhi Li, Yuan Tang, Yixue Hao, Long Hu, Min Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00979
• Pdf URL: https://arxiv.org/pdf/2404.00979
• Copy Paste: [[2404.00979]] PDF: A Probability-Driven Framework for Open World 3D Point Cloud Semantic Segmentation(https://arxiv.org/abs/2404.00979)
• Keywords: segmentation
• Abstract: Existing point cloud semantic segmentation networks cannot identify unknown classes and update their knowledge, due to a closed-set and static perspective of the real world, which would induce the intelligent agent to make bad decisions. To address this problem, we propose a Probability-Driven Framework (PDF) for open world semantic segmentation that includes (i) a lightweight U-decoder branch to identify unknown classes by estimating the uncertainties, (ii) a flexible pseudo-labeling scheme to supply geometry features along with probability distribution features of unknown classes by generating pseudo labels, and (iii) an incremental knowledge distillation strategy to incorporate novel classes into the existing knowledge base gradually. Our framework enables the model to behave like human beings, which could recognize unknown objects and incrementally learn them with the corresponding knowledge. Experimental results on the S3DIS and ScanNetv2 datasets demonstrate that the proposed PDF outperforms other methods by a large margin in both important tasks of open world semantic segmentation.

Title: Continual Learning for Smart City: A Survey

• Authors: Li Yang, Zhipeng Luo, Shiming Zhang, Fei Teng, Tianrui Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00983
• Pdf URL: https://arxiv.org/pdf/2404.00983
• Copy Paste: [[2404.00983]] Continual Learning for Smart City: A Survey(https://arxiv.org/abs/2404.00983)
• Keywords: federate
• Abstract: With the digitization of modern cities, large data volumes and powerful computational resources facilitate the rapid update of intelligent models deployed in smart cities. Continual learning (CL) is a novel machine learning paradigm that constantly updates models to adapt to changing environments, where the learning tasks, data, and distributions can vary over time. Our survey provides a comprehensive review of continual learning methods that are widely used in smart city development. The content consists of three parts: 1) Methodology-wise. We categorize a large number of basic CL methods and advanced CL frameworks in combination with other learning paradigms including graph learning, spatial-temporal learning, multi-modal learning, and federated learning. 2) Application-wise. We present numerous CL applications covering transportation, environment, public health, safety, networks, and associated datasets related to urban computing. 3) Challenges. We discuss current problems and challenges and envision several promising research directions. We believe this survey can help relevant researchers quickly familiarize themselves with the current state of continual learning research used in smart city development and direct them to future research trends.

Title: FlexiDreamer: Single Image-to-3D Generation with FlexiCubes

• Authors: Ruowen Zhao, Zhengyi Wang, Yikai Wang, Zihan Zhou, Jun Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00987
• Pdf URL: https://arxiv.org/pdf/2404.00987
• Copy Paste: [[2404.00987]] FlexiDreamer: Single Image-to-3D Generation with FlexiCubes(https://arxiv.org/abs/2404.00987)
• Keywords: extraction
• Abstract: 3D content generation from text prompts or single images has made remarkable progress in quality and speed recently. One of its dominant paradigms involves generating consistent multi-view images followed by a sparse-view reconstruction. However, due to the challenge of directly deforming the mesh representation to approach the target topology, most methodologies learn an implicit representation (such as NeRF) during the sparse-view reconstruction and acquire the target mesh by a post-processing extraction. Although the implicit representation can effectively model rich 3D information, its training typically entails a long convergence time. In addition, the post-extraction operation from the implicit field also leads to undesirable visual artifacts. In this paper, we propose FlexiDreamer, a novel single image-to-3d generation framework that reconstructs the target mesh in an end-to-end manner. By leveraging a flexible gradient-based extraction known as FlexiCubes, our method circumvents the defects brought by the post-processing and facilitates a direct acquisition of the target mesh. Furthermore, we incorporate a multi-resolution hash grid encoding scheme that progressively activates the encoding levels into the implicit field in FlexiCubes to help capture geometric details for per-step optimization. Notably, FlexiDreamer recovers a dense 3D structure from a single-view image in approximately 1 minute on a single NVIDIA A100 GPU, outperforming previous methodologies by a large margin.

Title: Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey

• Authors: Weicong Qin, Zhongxiang Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.00990
• Pdf URL: https://arxiv.org/pdf/2404.00990
• Copy Paste: [[2404.00990]] Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey(https://arxiv.org/abs/2404.00990)
• Keywords: interpretability, large language model
• Abstract: With the advancement of Artificial Intelligence (AI) and Large Language Models (LLMs), there is a profound transformation occurring in the realm of natural language processing tasks within the legal domain. The capabilities of LLMs are increasingly demonstrating unique roles in the legal sector, bringing both distinctive benefits and various challenges. This survey delves into the synergy between LLMs and the legal system, such as their applications in tasks like legal text comprehension, case retrieval, and analysis. Furthermore, this survey highlights key challenges faced by LLMs in the legal domain, including bias, interpretability, and ethical considerations, as well as how researchers are addressing these issues. The survey showcases the latest advancements in fine-tuned legal LLMs tailored for various legal systems, along with legal datasets available for fine-tuning LLMs in various languages. Additionally, it proposes directions for future research and development.

Title: AMOR: Ambiguous Authorship Order

• Authors: Maximilian Weiherer, Andreea Dogaru, Shreya Kapoor, Hannah Schieber, Bernhard Egger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00994
• Pdf URL: https://arxiv.org/pdf/2404.00994
• Copy Paste: [[2404.00994]] AMOR: Ambiguous Authorship Order(https://arxiv.org/abs/2404.00994)
• Keywords: fair
• Abstract: As we all know, writing scientific papers together with our beloved colleagues is a truly remarkable experience (partially): endless discussions about the same useless paragraph over and over again, followed by long days and long nights -- both at the same time. What a wonderful ride it is! What a beautiful life we have. But wait, there's one tiny little problem that utterly shatters the peace, turning even renowned scientists into bloodthirsty monsters: author order. The reason is that, contrary to widespread opinion, it's not the font size that matters, but the way things are ordered. Of course, this is a fairly well-known fact among scientists all across the planet (and beyond) and explains clearly why we regularly have to read about yet another escalated paper submission in local police reports. In this paper, we take an important step backwards to tackle this issue by solving the so-called author ordering problem (AOP) once and for all. Specifically, we propose AMOR, a system that replaces silly constructs like co-first or co-middle authorship with a simple yet easy probabilistic approach based on random shuffling of the author list at viewing time. In addition to AOP, we also solve the ambiguous author ordering citation problem} (AAOCP) on the fly. Stop author violence, be human.

Title: PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation

• Authors: Jaejung Seol, Seojun Kim, Jaejun Yoo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.00995
• Pdf URL: https://arxiv.org/pdf/2404.00995
• Copy Paste: [[2404.00995]] PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation(https://arxiv.org/abs/2404.00995)
• Keywords: robust, generative
• Abstract: Visual layout plays a critical role in graphic design fields such as advertising, posters, and web UI design. The recent trend towards content-aware layout generation through generative models has shown promise, yet it often overlooks the semantic intricacies of layout design by treating it as a simple numerical optimization. To bridge this gap, we introduce PosterLlama, a network designed for generating visually and textually coherent layouts by reformatting layout elements into HTML code and leveraging the rich design knowledge embedded within language models. Furthermore, we enhance the robustness of our model with a unique depth-based poster augmentation strategy. This ensures our generated layouts remain semantically rich but also visually appealing, even with limited data. Our extensive evaluations across several benchmarks demonstrate that PosterLlama outperforms existing methods in producing authentic and content-aware layouts. It supports an unparalleled range of conditions, including but not limited to unconditional layout generation, element conditional layout generation, layout completion, among others, serving as a highly versatile user manipulation tool.

Title: LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation

• Authors: Zilong Wang, Xufang Luo, Xinyang Jiang, Dongsheng Li, Lili Qiu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.00998
• Pdf URL: https://arxiv.org/pdf/2404.00998
• Copy Paste: [[2404.00998]] LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation(https://arxiv.org/abs/2404.00998)
• Keywords: large language model
• Abstract: Evaluating generated radiology reports is crucial for the development of radiology AI, but existing metrics fail to reflect the task's clinical requirements. This study proposes a novel evaluation framework using large language models (LLMs) to compare radiology reports for assessment. We compare the performance of various LLMs and demonstrate that, when using GPT-4, our proposed metric achieves evaluation consistency close to that of radiologists. Furthermore, to reduce costs and improve accessibility, making this method practical, we construct a dataset using LLM evaluation results and perform knowledge distillation to train a smaller model. The distilled model achieves evaluation capabilities comparable to GPT-4. Our framework and distilled model offer an accessible and efficient evaluation method for radiology report generation, facilitating the development of more clinically relevant models. The model will be further open-sourced and accessible.

Title: Teeth-SEG: An Efficient Instance Segmentation Framework for Orthodontic Treatment based on Anthropic Prior Knowledge

• Authors: Bo Zou, Shaofeng Wang, Hao Liu, Gaoyue Sun, Yajie Wang, FeiFei Zuo, Chengbin Quan, Youjian Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01013
• Pdf URL: https://arxiv.org/pdf/2404.01013
• Copy Paste: [[2404.01013]] Teeth-SEG: An Efficient Instance Segmentation Framework for Orthodontic Treatment based on Anthropic Prior Knowledge(https://arxiv.org/abs/2404.01013)
• Keywords: segmentation
• Abstract: Teeth localization, segmentation, and labeling in 2D images have great potential in modern dentistry to enhance dental diagnostics, treatment planning, and population-based studies on oral health. However, general instance segmentation frameworks are incompetent due to 1) the subtle differences between some teeth' shapes (e.g., maxillary first premolar and second premolar), 2) the teeth's position and shape variation across subjects, and 3) the presence of abnormalities in the dentition (e.g., caries and edentulism). To address these problems, we propose a ViT-based framework named TeethSEG, which consists of stacked Multi-Scale Aggregation (MSA) blocks and an Anthropic Prior Knowledge (APK) layer. Specifically, to compose the two modules, we design 1) a unique permutation-based upscaler to ensure high efficiency while establishing clear segmentation boundaries with 2) multi-head self/cross-gating layers to emphasize particular semantics meanwhile maintaining the divergence between token embeddings. Besides, we collect 3) the first open-sourced intraoral image dataset IO150K, which comprises over 150k intraoral photos, and all photos are annotated by orthodontists using a human-machine hybrid algorithm. Experiments on IO150K demonstrate that our TeethSEG outperforms the state-of-the-art segmentation models on dental image segmentation.

Title: Harnessing Large Language Models for Training-free Video Anomaly Detection

• Authors: Luca Zanella, Willi Menapace, Massimiliano Mancini, Yiming Wang, Elisa Ricci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01014
• Pdf URL: https://arxiv.org/pdf/2404.01014
• Copy Paste: [[2404.01014]] Harnessing Large Language Models for Training-free Video Anomaly Detection(https://arxiv.org/abs/2404.01014)
• Keywords: large language model
• Abstract: Video anomaly detection (VAD) aims to temporally locate abnormal events in a video. Existing works mostly rely on training deep models to learn the distribution of normality with either video-level supervision, one-class supervision, or in an unsupervised setting. Training-based methods are prone to be domain-specific, thus being costly for practical deployment as any domain change will involve data collection and model training. In this paper, we radically depart from previous efforts and propose LAnguage-based VAD (LAVAD), a method tackling VAD in a novel, training-free paradigm, exploiting the capabilities of pre-trained large language models (LLMs) and existing vision-language models (VLMs). We leverage VLM-based captioning models to generate textual descriptions for each frame of any test video. With the textual scene description, we then devise a prompting mechanism to unlock the capability of LLMs in terms of temporal aggregation and anomaly score estimation, turning LLMs into an effective video anomaly detector. We further leverage modality-aligned VLMs and propose effective techniques based on cross-modal similarity for cleaning noisy captions and refining the LLM-based anomaly scores. We evaluate LAVAD on two large datasets featuring real-world surveillance scenarios (UCF-Crime and XD-Violence), showing that it outperforms both unsupervised and one-class methods without requiring any training or data collection.

Title: PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison

• Authors: ChaeHun Park, Minseok Choi, Dohyun Lee, Jaegul Choo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01015
• Pdf URL: https://arxiv.org/pdf/2404.01015
• Copy Paste: [[2404.01015]] PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison(https://arxiv.org/abs/2404.01015)
• Keywords: robust
• Abstract: Building a reliable and automated evaluation metric is a necessary but challenging problem for open-domain dialogue systems. Recent studies proposed evaluation metrics that assess generated responses by considering their relevance to previous dialogue histories. Although effective, these metrics evaluate individual responses directly rather than considering their relative quality compared to other responses. To handle this, we propose PairEval, a novel dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations. PairEval is built on top of open-sourced and moderate-size language models, and we make them specialized in pairwise comparison between dialogue responses. Extensive experiments on multiple benchmarks demonstrate that our metric exhibits a higher correlation with human judgments than baseline metrics. We also find that the proposed comparative metric is more robust in detecting common failures from open-domain dialogue systems, including repetition and speaker insensitivity.

Title: Source-Aware Training Enables Knowledge Attribution in Language Models

• Authors: Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, Hao Peng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01019
• Pdf URL: https://arxiv.org/pdf/2404.01019
• Copy Paste: [[2404.01019]] Source-Aware Training Enables Knowledge Attribution in Language Models(https://arxiv.org/abs/2404.01019)
• Keywords: interpretability, large language model
• Abstract: Large language models (LLMs) learn a vast amount of knowledge during pretraining, but they are often oblivious to the source(s) of such knowledge. We investigate the problem of intrinsic source citation, where LLMs are required to cite the pretraining source supporting a generated response. Intrinsic source citation can enhance LLM transparency, interpretability, and verifiability. To give LLMs such ability, we explore source-aware training -- a post pretraining recipe that involves (i) training the LLM to associate unique source document identifiers with the knowledge in each document, followed by (ii) an instruction-tuning to teach the LLM to cite a supporting pretraining source when prompted. Source-aware training can easily be applied to pretrained LLMs off the shelf, and diverges minimally from existing pretraining/fine-tuning frameworks. Through experiments on carefully curated data, we demonstrate that our training recipe can enable faithful attribution to the pretraining data without a substantial impact on the model's quality compared to standard pretraining. Our results also highlight the importance of data augmentation in achieving attribution.

Title: Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation

• Authors: Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Rebecca Pattichis, Kai-Wei Chang
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.01030
• Pdf URL: https://arxiv.org/pdf/2404.01030
• Copy Paste: [[2404.01030]] Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation(https://arxiv.org/abs/2404.01030)
• Keywords: fair, generative
• Abstract: The recent advancement of large and powerful models with Text-to-Image (T2I) generation abilities -- such as OpenAI's DALLE-3 and Google's Gemini -- enables users to generate high-quality images from textual prompts. However, it has become increasingly evident that even simple prompts could cause T2I models to exhibit conspicuous social bias in generated images. Such bias might lead to both allocational and representational harms in society, further marginalizing minority groups. Noting this problem, a large body of recent works has been dedicated to investigating different dimensions of bias in T2I systems. However, an extensive review of these studies is lacking, hindering a systematic understanding of current progress and research gaps. We present the first extensive survey on bias in T2I generative models. In this survey, we review prior studies on dimensions of bias: Gender, Skintone, and Geo-Culture. Specifically, we discuss how these works define, evaluate, and mitigate different aspects of bias. We found that: (1) while gender and skintone biases are widely studied, geo-cultural bias remains under-explored; (2) most works on gender and skintone bias investigated occupational association, while other aspects are less frequently studied; (3) almost all gender bias works overlook non-binary identities in their studies; (4) evaluation datasets and metrics are scattered, with no unified framework for measuring biases; and (5) current mitigation methods fail to resolve biases comprehensively. Based on current limitations, we point out future research directions that contribute to human-centric definitions, evaluations, and mitigation of biases. We hope to highlight the importance of studying biases in T2I systems, as well as encourage future efforts to holistically understand and tackle biases, building fair and trustworthy T2I technologies for everyone.

Title: ARAGOG: Advanced RAG Output Grading

• Authors: Matouš Eibich, Shivay Nagpal, Alexander Fred-Ojala
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.01037
• Pdf URL: https://arxiv.org/pdf/2404.01037
• Copy Paste: [[2404.01037]] ARAGOG: Advanced RAG Output Grading(https://arxiv.org/abs/2404.01037)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) is essential for integrating external knowledge into Large Language Model (LLM) outputs. While the literature on RAG is growing, it primarily focuses on systematic reviews and comparisons of new state-of-the-art (SoTA) techniques against their predecessors, with a gap in extensive experimental comparisons. This study begins to address this gap by assessing various RAG methods' impacts on retrieval precision and answer similarity. We found that Hypothetical Document Embedding (HyDE) and LLM reranking significantly enhance retrieval precision. However, Maximal Marginal Relevance (MMR) and Cohere rerank did not exhibit notable advantages over a baseline Naive RAG system, and Multi-query approaches underperformed. Sentence Window Retrieval emerged as the most effective for retrieval precision, despite its variable performance on answer similarity. The study confirms the potential of the Document Summary Index as a competent retrieval approach. All resources related to this research are publicly accessible for further investigation through our GitHub repository ARAGOG (https://github.com/predlico/ARAGOG). We welcome the community to further this exploratory study in RAG systems.

Title: Can LLMs get help from other LLMs without revealing private information?

• Authors: Florian Hartmann, Duc-Hieu Tran, Peter Kairouz, Victor Cărbune, Blaise Aguera y Arcas
• Subjects: cs.LG, cs.AI, cs.CR, cs.MA
• Abstract URL: https://arxiv.org/abs/2404.01041
• Pdf URL: https://arxiv.org/pdf/2404.01041
• Copy Paste: [[2404.01041]] Can LLMs get help from other LLMs without revealing private information?(https://arxiv.org/abs/2404.01041)
• Keywords: privacy, large language model
• Abstract: Cascades are a common type of machine learning systems in which a large, remote model can be queried if a local model is not able to accurately label a user's data by itself. Serving stacks for large language models (LLMs) increasingly use cascades due to their ability to preserve task performance while dramatically reducing inference costs. However, applying cascade systems in situations where the local model has access to sensitive data constitutes a significant privacy risk for users since such data could be forwarded to the remote model. In this work, we show the feasibility of applying cascade systems in such setups by equipping the local model with privacy-preserving techniques that reduce the risk of leaking private information when querying the remote model. To quantify information leakage in such setups, we introduce two privacy measures. We then propose a system that leverages the recently introduced social learning paradigm in which LLMs collaboratively learn from each other by exchanging natural language. Using this paradigm, we demonstrate on several datasets that our methods minimize the privacy loss while at the same time improving task performance compared to a non-cascade baseline.

Title: Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation

• Authors: Haofeng Liu, Chenshu Xu, Yifei Yang, Lihua Zeng, Shengfeng He
• Subjects: cs.CV, cs.GR, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01050
• Pdf URL: https://arxiv.org/pdf/2404.01050
• Copy Paste: [[2404.01050]] Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation(https://arxiv.org/abs/2404.01050)
• Keywords: robust, diffusion, generative
• Abstract: Point-based interactive editing serves as an essential tool to complement the controllability of existing generative models. A concurrent work, DragDiffusion, updates the diffusion latent map in response to user inputs, causing global latent map alterations. This results in imprecise preservation of the original content and unsuccessful editing due to gradient vanishing. In contrast, we present DragNoise, offering robust and accelerated editing without retracing the latent map. The core rationale of DragNoise lies in utilizing the predicted noise output of each U-Net as a semantic editor. This approach is grounded in two critical observations: firstly, the bottleneck features of U-Net inherently possess semantically rich features ideal for interactive editing; secondly, high-level semantics, established early in the denoising process, show minimal variation in subsequent stages. Leveraging these insights, DragNoise edits diffusion semantics in a single denoising step and efficiently propagates these changes, ensuring stability and efficiency in diffusion editing. Comparative experiments reveal that DragNoise achieves superior control and semantic retention, reducing the optimization time by over 50% compared to DragDiffusion. Our codes are available at https://github.com/haofengl/DragNoise.

Title: Action Detection via an Image Diffusion Process

• Authors: Lin Geng Foo, Tianjiao Li, Hossein Rahmani, Jun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01051
• Pdf URL: https://arxiv.org/pdf/2404.01051
• Copy Paste: [[2404.01051]] Action Detection via an Image Diffusion Process(https://arxiv.org/abs/2404.01051)
• Keywords: diffusion, transformer
• Abstract: Action detection aims to localize the starting and ending points of action instances in untrimmed videos, and predict the classes of those instances. In this paper, we make the observation that the outputs of the action detection task can be formulated as images. Thus, from a novel perspective, we tackle action detection via a three-image generation process to generate starting point, ending point and action-class predictions as images via our proposed Action Detection Image Diffusion (ADI-Diff) framework. Furthermore, since our images differ from natural images and exhibit special properties, we further explore a Discrete Action-Detection Diffusion Process and a Row-Column Transformer design to better handle their processing. Our ADI-Diff framework achieves state-of-the-art results on two widely-used datasets.

Title: Regularized Best-of-N Sampling to Mitigate Reward Hacking for Language Model Alignment

• Authors: Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01054
• Pdf URL: https://arxiv.org/pdf/2404.01054
• Copy Paste: [[2404.01054]] Regularized Best-of-N Sampling to Mitigate Reward Hacking for Language Model Alignment(https://arxiv.org/abs/2404.01054)
• Keywords: large language model
• Abstract: Best-of-N (BoN) sampling with a reward model has been shown to be an effective strategy for aligning Large Language Models (LLMs) to human preferences at the time of decoding. BoN sampling is susceptible to a problem known as reward hacking. Because the reward model is an imperfect proxy for the true objective, over-optimizing its value can compromise its performance on the true objective. A common solution to prevent reward hacking in preference learning techniques is to optimize a reward using proximity regularization (e.g., KL regularization), which ensures that the language model remains close to the reference model. In this research, we propose Regularized Best-of-N (RBoN), a variant of BoN that aims to mitigate reward hacking by incorporating a proximity term in response selection, similar to preference learning techniques. We evaluate two variants of RBoN on the AlpacaFarm dataset and find that they outperform BoN, especially when the proxy reward model has a low correlation with the true objective.

Title: A comparison of Single- and Double-generator formalisms for Thermodynamics-Informed Neural Networks

• Authors: Pau Urdeitx, Icíar Alfaro, David González, Francisco Chinesta, Elías Cueto
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01060
• Pdf URL: https://arxiv.org/pdf/2404.01060
• Copy Paste: [[2404.01060]] A comparison of Single- and Double-generator formalisms for Thermodynamics-Informed Neural Networks(https://arxiv.org/abs/2404.01060)
• Keywords: robust
• Abstract: The development of inductive biases has been shown to be a very effective way to increase the accuracy and robustness of neural networks, particularly when they are used to predict physical phenomena. These biases significantly increase the certainty of predictions, decrease the error made and allow considerably smaller datasets to be used. There are a multitude of methods in the literature to develop these biases. One of the most effective ways, when dealing with physical phenomena, is to introduce physical principles of recognised validity into the network architecture. The problem becomes more complex without knowledge of the physical principles governing the phenomena under study. A very interesting possibility then is to turn to the principles of thermodynamics, which are universally valid, regardless of the level of abstraction of the description sought for the phenomenon under study. To ensure compliance with the principles of thermodynamics, there are formulations that have a long tradition in many branches of science. In the field of rheology, for example, two main types of formalisms are used to ensure compliance with these principles: one-generator and two-generator formalisms. In this paper we study the advantages and disadvantages of each, using classical problems with known solutions and synthetic data.

Title: T-Mamba: Frequency-Enhanced Gated Long-Range Dependency for Tooth 3D CBCT Segmentation

• Authors: Jing Hao, Lei He, Kuo Feng Hung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01065
• Pdf URL: https://arxiv.org/pdf/2404.01065
• Copy Paste: [[2404.01065]] T-Mamba: Frequency-Enhanced Gated Long-Range Dependency for Tooth 3D CBCT Segmentation(https://arxiv.org/abs/2404.01065)
• Keywords: transformer, segmentation
• Abstract: Efficient tooth segmentation in three-dimensional (3D) imaging, critical for orthodontic diagnosis, remains challenging due to noise, low contrast, and artifacts in CBCT images. Both convolutional Neural Networks (CNNs) and transformers have emerged as popular architectures for image segmentation. However, their efficacy in handling long-range dependencies is limited due to inherent locality or computational complexity. To address this issue, we propose T-Mamba, integrating shared positional encoding and frequency-based features into vision mamba, to address limitations in spatial position preservation and feature enhancement in frequency domain. Besides, we also design a gate selection unit to integrate two features in spatial domain and one feature in frequency domain adaptively. T-Mamba is the first work to introduce frequency-based features into vision mamba. Extensive experiments demonstrate that T-Mamba achieves new SOTA results on the public Tooth CBCT dataset and outperforms previous SOTA methods by a large margin, i.e., IoU + 3.63%, SO + 2.43%, DSC +2.30%, HD -4.39mm, and ASSD -0.37mm. The code and models are publicly available at https://github.com/isbrycee/T-Mamba.

Title: Advancing AI with Integrity: Ethical Challenges and Solutions in Neural Machine Translation

• Authors: Richard Kimera, Yun-Seon Kim, Heeyoul Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01070
• Pdf URL: https://arxiv.org/pdf/2404.01070
• Copy Paste: [[2404.01070]] Advancing AI with Integrity: Ethical Challenges and Solutions in Neural Machine Translation(https://arxiv.org/abs/2404.01070)
• Keywords: privacy, fair, transformer
• Abstract: This paper addresses the ethical challenges of Artificial Intelligence in Neural Machine Translation (NMT) systems, emphasizing the imperative for developers to ensure fairness and cultural sensitivity. We investigate the ethical competence of AI models in NMT, examining the Ethical considerations at each stage of NMT development, including data handling, privacy, data ownership, and consent. We identify and address ethical issues through empirical studies. These include employing Transformer models for Luganda-English translations and enhancing efficiency with sentence mini-batching. And complementary studies that refine data labeling techniques and fine-tune BERT and Longformer models for analyzing Luganda and English social media content. Our second approach is a literature review from databases such as Google Scholar and platforms like GitHub. Additionally, the paper probes the distribution of responsibility between AI systems and humans, underscoring the essential role of human oversight in upholding NMT ethical standards. Incorporating a biblical perspective, we discuss the societal impact of NMT and the broader ethical responsibilities of developers, positing them as stewards accountable for the societal repercussions of their creations.

Title: Prompt Learning for Oriented Power Transmission Tower Detection in High-Resolution SAR Images

• Authors: Tianyang Li, Chao Wang, Hong Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01074
• Pdf URL: https://arxiv.org/pdf/2404.01074
• Copy Paste: [[2404.01074]] Prompt Learning for Oriented Power Transmission Tower Detection in High-Resolution SAR Images(https://arxiv.org/abs/2404.01074)
• Keywords: transformer
• Abstract: Detecting transmission towers from synthetic aperture radar (SAR) images remains a challenging task due to the comparatively small size and side-looking geometry, with background clutter interference frequently hindering tower identification. A large number of interfering signals superimposes the return signal from the tower. We found that localizing or prompting positions of power transmission towers is beneficial to address this obstacle. Based on this revelation, this paper introduces prompt learning into the oriented object detector (P2Det) for multimodal information learning. P2Det contains the sparse prompt coding and cross-attention between the multimodal data. Specifically, the sparse prompt encoder (SPE) is proposed to represent point locations, converting prompts into sparse embeddings. The image embeddings are generated through the Transformer layers. Then a two-way fusion module (TWFM) is proposed to calculate the cross-attention of the two different embeddings. The interaction of image-level and prompt-level features is utilized to address the clutter interference. A shape-adaptive refinement module (SARM) is proposed to reduce the effect of aspect ratio. Extensive experiments demonstrated the effectiveness of the proposed model on high-resolution SAR images. P2Det provides a novel insight for multimodal object detection due to its competitive performance.

Title: Efficient Prompting Methods for Large Language Models: A Survey

• Authors: Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Tong Xiao, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01077
• Pdf URL: https://arxiv.org/pdf/2404.01077
• Copy Paste: [[2404.01077]] Efficient Prompting Methods for Large Language Models: A Survey(https://arxiv.org/abs/2404.01077)
• Keywords: large language model
• Abstract: Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks. While this approach opens the door to in-context learning of LLMs, it brings the additional computational burden of model inference and human effort of manual-designed prompts, particularly when using lengthy and complex prompts to guide and control the behavior of LLMs. As a result, the LLM field has seen a remarkable surge in efficient prompting methods. In this paper, we present a comprehensive overview of these methods. At a high level, efficient prompting methods can broadly be categorized into two approaches: prompting with efficient computation and prompting with efficient design. The former involves various ways of compressing prompts, and the latter employs techniques for automatic prompt optimization. We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.

Title: Stale Diffusion: Hyper-realistic 5D Movie Generation Using Old-school Methods

• Authors: Joao F. Henriques, Dylan Campbell, Tengda Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01079
• Pdf URL: https://arxiv.org/pdf/2404.01079
• Copy Paste: [[2404.01079]] Stale Diffusion: Hyper-realistic 5D Movie Generation Using Old-school Methods(https://arxiv.org/abs/2404.01079)
• Keywords: diffusion
• Abstract: Two years ago, Stable Diffusion achieved super-human performance at generating images with super-human numbers of fingers. Following the steady decline of its technical novelty, we propose Stale Diffusion, a method that solidifies and ossifies Stable Diffusion in a maximum-entropy state. Stable Diffusion works analogously to a barn (the Stable) from which an infinite set of horses have escaped (the Diffusion). As the horses have long left the barn, our proposal may be seen as antiquated and irrelevant. Nevertheless, we vigorously defend our claim of novelty by identifying as early adopters of the Slow Science Movement, which will produce extremely important pearls of wisdom in the future. Our speed of contributions can also be seen as a quasi-static implementation of the recent call to pause AI experiments, which we wholeheartedly support. As a result of a careful archaeological expedition to 18-months-old Git commit histories, we found that naturally-accumulating errors have produced a novel entropy-maximising Stale Diffusion method, that can produce sleep-inducing hyper-realistic 5D video that is as good as one's imagination.

Title: AILS-NTUA at SemEval-2024 Task 9: Cracking Brain Teasers: Transformer Models for Lateral Thinking Puzzles

• Authors: Ioannis Panagiotopoulos, Giorgos Filandrianos, Maria Lymperaiou, Giorgos Stamou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01084
• Pdf URL: https://arxiv.org/pdf/2404.01084
• Copy Paste: [[2404.01084]] AILS-NTUA at SemEval-2024 Task 9: Cracking Brain Teasers: Transformer Models for Lateral Thinking Puzzles(https://arxiv.org/abs/2404.01084)
• Keywords: secure, transformer
• Abstract: In this paper, we outline our submission for the SemEval-2024 Task 9 competition: 'BRAINTEASER: A Novel Task Defying Common Sense'. We engage in both sub-tasks: Sub-task A-Sentence Puzzle and Sub-task B-Word Puzzle. We evaluate a plethora of pre-trained transformer-based language models of different sizes through fine-tuning. Subsequently, we undertake an analysis of their scores and responses to aid future researchers in understanding and utilizing these models effectively. Our top-performing approaches secured competitive positions on the competition leaderboard across both sub-tasks. In the evaluation phase, our best submission attained an average accuracy score of 81.7% in the Sentence Puzzle, and 85.4% in the Word Puzzle, significantly outperforming the best neural baseline (ChatGPT) by more than 20% and 30% respectively.

Title: Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On

• Authors: Xu Yang, Changxing Ding, Zhibin Hong, Junhao Huang, Jin Tao, Xiangmin Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01089
• Pdf URL: https://arxiv.org/pdf/2404.01089
• Copy Paste: [[2404.01089]] Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On(https://arxiv.org/abs/2404.01089)
• Keywords: diffusion
• Abstract: Image-based virtual try-on is an increasingly important task for online shopping. It aims to synthesize images of a specific person wearing a specified garment. Diffusion model-based approaches have recently become popular, as they are excellent at image synthesis tasks. However, these approaches usually employ additional image encoders and rely on the cross-attention mechanism for texture transfer from the garment to the person image, which affects the try-on's efficiency and fidelity. To address these issues, we propose an Texture-Preserving Diffusion (TPD) model for virtual try-on, which enhances the fidelity of the results and introduces no additional image encoders. Accordingly, we make contributions from two aspects. First, we propose to concatenate the masked person and reference garment images along the spatial dimension and utilize the resulting image as the input for the diffusion model's denoising UNet. This enables the original self-attention layers contained in the diffusion model to achieve efficient and accurate texture transfer. Second, we propose a novel diffusion-based method that predicts a precise inpainting mask based on the person and reference garment images, further enhancing the reliability of the try-on results. In addition, we integrate mask prediction and image synthesis into a single compact model. The experimental results show that our approach can be applied to various try-on tasks, e.g., garment-to-person and person-to-person try-ons, and significantly outperforms state-of-the-art methods on popular VITON, VITON-HD databases.

Title: HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach

• Authors: Maxim Nikolaev, Mikhail Kuznetsov, Dmitry Vetrov, Aibek Alanov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01094
• Pdf URL: https://arxiv.org/pdf/2404.01094
• Copy Paste: [[2404.01094]] HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach(https://arxiv.org/abs/2404.01094)
• Keywords: robust
• Abstract: Our paper addresses the complex task of transferring a hairstyle from a reference image to an input photo for virtual hair try-on. This task is challenging due to the need to adapt to various photo poses, the sensitivity of hairstyles, and the lack of objective metrics. The current state of the art hairstyle transfer methods use an optimization process for different parts of the approach, making them inexcusably slow. At the same time, faster encoder-based models are of very low quality because they either operate in StyleGAN's W+ space or use other low-dimensional image generators. Additionally, both approaches have a problem with hairstyle transfer when the source pose is very different from the target pose, because they either don't consider the pose at all or deal with it inefficiently. In our paper, we present the HairFast model, which uniquely solves these problems and achieves high resolution, near real-time performance, and superior reconstruction compared to optimization problem-based methods. Our solution includes a new architecture operating in the FS latent space of StyleGAN, an enhanced inpainting approach, and improved encoders for better alignment, color transfer, and a new encoder for post-processing. The effectiveness of our approach is demonstrated on realism metrics after random hairstyle transfer and reconstruction when the original hairstyle is transferred. In the most difficult scenario of transferring both shape and color of a hairstyle from different images, our method performs in less than a second on the Nvidia V100. Our code is available at https://github.com/AIRI-Institute/HairFastGAN.

Title: What's in Your "Safe" Data?: Identifying Benign Data that Breaks Safety

• Authors: Luxi He, Mengzhou Xia, Peter Henderson
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.01099
• Pdf URL: https://arxiv.org/pdf/2404.01099
• Copy Paste: [[2404.01099]] What's in Your "Safe" Data?: Identifying Benign Data that Breaks Safety(https://arxiv.org/abs/2404.01099)
• Keywords: large language model
• Abstract: Current Large Language Models (LLMs), even those tuned for safety and alignment, are susceptible to jailbreaking. Some have found that just further fine-tuning an aligned model with benign data (i.e., data without harmful content) surprisingly leads to substantial degradation in safety. We delve into the data-centric aspects of why benign fine-tuning inadvertently contributes to jailbreaking. First, we represent fine-tuning data through two lenses: representation and gradient spaces. Furthermore, we propose a bi-directional anchoring method that prioritizes data points that are close to harmful examples and distant from benign ones. By doing so, our approach effectively identifies subsets of benign data that are more likely to degrade the model's safety after fine-tuning. Training on just 100 of these seemingly benign datapoints can lead to the fine-tuned model affirmatively responding to > 70% of tested harmful requests, compared to < 20% after fine-tuning on randomly selected data. We further find that selected data are often in the form of lists and bullet points, or math questions.

Title: UFID: A Unified Framework for Input-level Backdoor Detection on Diffusion Models

• Authors: Zihan Guan, Mengxuan Hu, Sheng Li, Anil Vullikanti
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01101
• Pdf URL: https://arxiv.org/pdf/2404.01101
• Copy Paste: [[2404.01101]] UFID: A Unified Framework for Input-level Backdoor Detection on Diffusion Models(https://arxiv.org/abs/2404.01101)
• Keywords: attack, diffusion, generative
• Abstract: Diffusion Models are vulnerable to backdoor attacks, where malicious attackers inject backdoors by poisoning some parts of the training samples during the training stage. This poses a serious threat to the downstream users, who query the diffusion models through the API or directly download them from the internet. To mitigate the threat of backdoor attacks, there have been a plethora of investigations on backdoor detections. However, none of them designed a specialized backdoor detection method for diffusion models, rendering the area much under-explored. Moreover, these prior methods mainly focus on the traditional neural networks in the classification task, which cannot be adapted to the backdoor detections on the generative task easily. Additionally, most of the prior methods require white-box access to model weights and architectures, or the probability logits as additional information, which are not always practical. In this paper, we propose a Unified Framework for Input-level backdoor Detection (UFID) on the diffusion models, which is motivated by observations in the diffusion models and further validated with a theoretical causality analysis. Extensive experiments across different datasets on both conditional and unconditional diffusion models show that our method achieves a superb performance on detection effectiveness and run-time efficiency. The code is available at https://github.com/GuanZihan/official_UFID.

Title: MagLive: Near-Field Magnetic Sensing-Based Voice Liveness Detection on Smartphones

• Authors: Xiping Sun, Jing Chen, Cong Wu, Kun He, Haozhe Xu, Yebo Feng, Ruiying Du, Xianhao Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.01106
• Pdf URL: https://arxiv.org/pdf/2404.01106
• Copy Paste: [[2404.01106]] MagLive: Near-Field Magnetic Sensing-Based Voice Liveness Detection on Smartphones(https://arxiv.org/abs/2404.01106)
• Keywords: security, attack, robust
• Abstract: Voice authentication has been widely used on smartphones. However, it remains vulnerable to spoofing attacks, where the attacker replays recorded voice samples from authentic humans using loudspeakers to bypass the voice authentication system. In this paper, we present MagLive, a robust voice liveness detection scheme designed for smartphones to mitigate such spoofing attacks. MagLive leverages differences in magnetic field patterns generated by different speakers (i.e., humans or loudspeakers) when speaking for liveness detection. It uses the built-in magnetometer on smartphones to capture these magnetic field changes. Specifically, MagLive utilizes two CNN-based submodels and a self-attention-based feature fusion model to extract effective and robust features. Supervised contrastive learning is then employed to achieve user-irrelevance, device-irrelevance, and content-irrelevance. MagLive imposes no additional burdens on users and does not rely on active sensing or extra devices. We conducted comprehensive experiments with various settings to evaluate the security and robustness of MagLive. Our results demonstrate that MagLive effectively distinguishes between humans and attackers (i.e., loudspeakers), achieving a balanced accuracy of 99.01% and an equal error rate of 0.77%.

Title: An incremental hybrid adaptive network-based IDS in Software Defined Networks to detect stealth attacks

• Authors: Abdullah H Alqahtani
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01109
• Pdf URL: https://arxiv.org/pdf/2404.01109
• Copy Paste: [[2404.01109]] An incremental hybrid adaptive network-based IDS in Software Defined Networks to detect stealth attacks(https://arxiv.org/abs/2404.01109)
• Keywords: security, attack, steal
• Abstract: Network attacks have became increasingly more sophisticated and stealthy due to the advances in technologies and the growing sophistication of attackers. Advanced Persistent Threats (APTs) are a type of attack that implement a wide range of strategies to evade detection and be under the defence radar. Software Defined Network (SDN) is a network paradigm that implements dynamic configuration by separating the control plane from the network plane. This approach improves security aspects by facilitating the employment of network intrusion detection systems. Implementing Machine Learning (ML) techniques in Intrusion Detection Systems (IDSs) is widely used to detect such attacks but has a challenge when the data distribution changes. Concept drift is a term that describes the change in the relationship between the input data and the target value (label or class). The model is expected to degrade as certain forms of change occur. In this paper, the primary form of change will be in user behaviour (particularly changes in attacker behaviour). It is essential for a model to adapt itself to deviations in data distribution. SDN can help in monitoring changes in data distribution. This paper discusses changes in stealth attacker behaviour. The work described here investigates various concept drift detection algorithms. An incremental hybrid adaptive Network Intrusion Detection System (NIDS) is proposed to tackle the issue of concept drift in SDN. It can detect known and unknown attacks. The model is evaluated over different datasets showing promising results.

Title: Few shot point cloud reconstruction and denoising via learned Guassian splats renderings and fine-tuned diffusion features

• Authors: Pietro Bonazzi, Marie-Julie Rakatosaona, Marco Cannici, Federico Tombari, Davide Scaramuzza
• Subjects: cs.CV, cs.CG
• Abstract URL: https://arxiv.org/abs/2404.01112
• Pdf URL: https://arxiv.org/pdf/2404.01112
• Copy Paste: [[2404.01112]] Few shot point cloud reconstruction and denoising via learned Guassian splats renderings and fine-tuned diffusion features(https://arxiv.org/abs/2404.01112)
• Keywords: diffusion
• Abstract: Existing deep learning methods for the reconstruction and denoising of point clouds rely on small datasets of 3D shapes. We circumvent the problem by leveraging deep learning methods trained on billions of images. We propose a method to reconstruct point clouds from few images and to denoise point clouds from their rendering by exploiting prior knowledge distilled from image-based deep learning models. To improve reconstruction in constraint settings, we regularize the training of a differentiable renderer with hybrid surface and appearance by introducing semantic consistency supervision. In addition, we propose a pipeline to finetune Stable Diffusion to denoise renderings of noisy point clouds and we demonstrate how these learned filters can be used to remove point cloud noise coming without 3D supervision. We compare our method with DSS and PointRadiance and achieved higher quality 3D reconstruction on the Sketchfab Testset and SCUT Dataset.

Title: Motion Blur Decomposition with Cross-shutter Guidance

• Authors: Xiang Ji, Haiyang Jiang, Yinqiang Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01120
• Pdf URL: https://arxiv.org/pdf/2404.01120
• Copy Paste: [[2404.01120]] Motion Blur Decomposition with Cross-shutter Guidance(https://arxiv.org/abs/2404.01120)
• Keywords: robust
• Abstract: Motion blur is a frequently observed image artifact, especially under insufficient illumination where exposure time has to be prolonged so as to collect more photons for a bright enough image. Rather than simply removing such blurring effects, recent researches have aimed at decomposing a blurry image into multiple sharp images with spatial and temporal coherence. Since motion blur decomposition itself is highly ambiguous, priors from neighbouring frames or human annotation are usually needed for motion disambiguation. In this paper, inspired by the complementary exposure characteristics of a global shutter (GS) camera and a rolling shutter (RS) camera, we propose to utilize the ordered scanline-wise delay in a rolling shutter image to robustify motion decomposition of a single blurry image. To evaluate this novel dual imaging setting, we construct a triaxial system to collect realistic data, as well as a deep network architecture that explicitly addresses temporal and contextual information through reciprocal branches for cross-shutter motion blur decomposition. Experiment results have verified the effectiveness of our proposed algorithm, as well as the validity of our dual imaging setting.

Title: CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening

• Authors: Wen-Jie Shu, Hong-Xia Dou, Rui Wen, Xiao Wu, Liang-Jian Deng
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.01121
• Pdf URL: https://arxiv.org/pdf/2404.01121
• Copy Paste: [[2404.01121]] CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening(https://arxiv.org/abs/2404.01121)
• Keywords: robust, transformer
• Abstract: Pansharpening aims to enhance remote sensing image (RSI) quality by merging high-resolution panchromatic (PAN) with multispectral (MS) images. However, prior techniques struggled to optimally fuse PAN and MS images for enhanced spatial and spectral information, due to a lack of a systematic framework capable of effectively coordinating their individual strengths. In response, we present the Cross Modulation Transformer (CMT), a pioneering method that modifies the attention mechanism. This approach utilizes a robust modulation technique from signal processing, integrating it into the attention mechanism's calculations. It dynamically tunes the weights of the carrier's value (V) matrix according to the modulator's features, thus resolving historical challenges and achieving a seamless integration of spatial and spectral attributes. Furthermore, considering that RSI exhibits large-scale features and edge details along with local textures, we crafted a hybrid loss function that combines Fourier and wavelet transforms to effectively capture these characteristics, thereby enhancing both spatial and spectral accuracy in pansharpening. Extensive experiments demonstrate our framework's superior performance over existing state-of-the-art methods. The code will be publicly available to encourage further research.

Title: Medical Visual Prompting (MVP): A Unified Framework for Versatile and High-Quality Medical Image Segmentation

• Authors: Yulin Chen, Guoheng Huang, Kai Huang, Zijin Lin, Guo Zhong, Shenghong Luo, Jie Deng, Jian Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01127
• Pdf URL: https://arxiv.org/pdf/2404.01127
• Copy Paste: [[2404.01127]] Medical Visual Prompting (MVP): A Unified Framework for Versatile and High-Quality Medical Image Segmentation(https://arxiv.org/abs/2404.01127)
• Keywords: segmentation
• Abstract: Accurate segmentation of lesion regions is crucial for clinical diagnosis and treatment across various diseases. While deep convolutional networks have achieved satisfactory results in medical image segmentation, they face challenges such as loss of lesion shape information due to continuous convolution and downsampling, as well as the high cost of manually labeling lesions with varying shapes and sizes. To address these issues, we propose a novel medical visual prompting (MVP) framework that leverages pre-training and prompting concepts from natural language processing (NLP). The framework utilizes three key components: Super-Pixel Guided Prompting (SPGP) for superpixelating the input image, Image Embedding Guided Prompting (IEGP) for freezing patch embedding and merging with superpixels to provide visual prompts, and Adaptive Attention Mechanism Guided Prompting (AAGP) for pinpointing prompt content and efficiently adapting all layers. By integrating SPGP, IEGP, and AAGP, the MVP enables the segmentation network to better learn shape prompting information and facilitates mutual learning across different tasks. Extensive experiments conducted on five datasets demonstrate superior performance of this method in various challenging medical image tasks, while simplifying single-task medical segmentation models. This novel framework offers improved performance with fewer parameters and holds significant potential for accurate segmentation of lesion regions in various medical tasks, making it clinically valuable.

Title: Structured Information Matters: Incorporating Abstract Meaning Representation into LLMs for Improved Open-Domain Dialogue Evaluation

• Authors: Bohao Yang, Kun Zhao, Chen Tang, Liang Zhan, Chenghua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01129
• Pdf URL: https://arxiv.org/pdf/2404.01129
• Copy Paste: [[2404.01129]] Structured Information Matters: Incorporating Abstract Meaning Representation into LLMs for Improved Open-Domain Dialogue Evaluation(https://arxiv.org/abs/2404.01129)
• Keywords: robust, large language model
• Abstract: Automatic open-domain dialogue evaluation has attracted increasing attention. Trainable evaluation metrics are commonly trained with true positive and randomly selected negative responses, resulting in a tendency for them to assign a higher score to the responses that share higher content similarity with a given context. However, adversarial negative responses possess high content similarity with the contexts whilst being semantically different. Therefore, existing evaluation metrics are not robust enough to evaluate such responses, resulting in low correlations with human judgments. While recent studies have shown some efficacy in utilizing Large Language Models (LLMs) for open-domain dialogue evaluation, they still encounter challenges in effectively handling adversarial negative examples. In this paper, we propose a simple yet effective framework for open-domain dialogue evaluation, which combines domain-specific language models (SLMs) with LLMs. The SLMs can explicitly incorporate Abstract Meaning Representation (AMR) graph information of the dialogue through a gating mechanism for enhanced semantic representation learning. The evaluation result of SLMs and AMR graph information are plugged into the prompt of LLM, for the enhanced in-context learning performance. Experimental results on open-domain dialogue evaluation tasks demonstrate the superiority of our method compared to a wide range of state-of-the-art baselines, especially in discriminating adversarial negative responses. Our code is available at https://github.com/Bernard-Yang/SIMAMR.

Title: Enhancing Reasoning Capacity of SLM using Cognitive Enhancement

• Authors: Jonathan Pan, Swee Liang Wong, Xin Wei Chia, Yidi Yuan
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01135
• Pdf URL: https://arxiv.org/pdf/2404.01135
• Copy Paste: [[2404.01135]] Enhancing Reasoning Capacity of SLM using Cognitive Enhancement(https://arxiv.org/abs/2404.01135)
• Keywords: security, privacy, large language model
• Abstract: Large Language Models (LLMs) have been applied to automate cyber security activities and processes including cyber investigation and digital forensics. However, the use of such models for cyber investigation and digital forensics should address accountability and security considerations. Accountability ensures models have the means to provide explainable reasonings and outcomes. This information can be extracted through explicit prompt requests. For security considerations, it is crucial to address privacy and confidentiality of the involved data during data processing as well. One approach to deal with this consideration is to have the data processed locally using a local instance of the model. Due to limitations of locally available resources, namely memory and GPU capacities, a Smaller Large Language Model (SLM) will typically be used. These SLMs have significantly fewer parameters compared to the LLMs. However, such size reductions have notable performance reduction, especially when tasked to provide reasoning explanations. In this paper, we aim to mitigate performance reduction through the integration of cognitive strategies that humans use for problem-solving. We term this as cognitive enhancement through prompts. Our experiments showed significant improvement gains of the SLMs' performances when such enhancements were applied. We believe that our exploration study paves the way for further investigation into the use of cognitive enhancement to optimize SLM for cyber security applications.

Title: Structured Initialization for Attention in Vision Transformers

• Authors: Jianqiao Zheng, Xueqian Li, Simon Lucey
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01139
• Pdf URL: https://arxiv.org/pdf/2404.01139
• Copy Paste: [[2404.01139]] Structured Initialization for Attention in Vision Transformers(https://arxiv.org/abs/2404.01139)
• Keywords: transformer
• Abstract: The training of vision transformer (ViT) networks on small-scale datasets poses a significant challenge. By contrast, convolutional neural networks (CNNs) have an architectural inductive bias enabling them to perform well on such problems. In this paper, we argue that the architectural bias inherent to CNNs can be reinterpreted as an initialization bias within ViT. This insight is significant as it empowers ViTs to perform equally well on small-scale problems while maintaining their flexibility for large-scale applications. Our inspiration for this ``structured'' initialization stems from our empirical observation that random impulse filters can achieve comparable performance to learned filters within CNNs. Our approach achieves state-of-the-art performance for data-efficient ViT learning across numerous benchmarks including CIFAR-10, CIFAR-100, and SVHN.

Title: SoK: A Review of Differentially Private Linear Models For High-Dimensional Data

• Authors: Amol Khanna, Edward Raff, Nathan Inkawhich
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.01141
• Pdf URL: https://arxiv.org/pdf/2404.01141
• Copy Paste: [[2404.01141]] SoK: A Review of Differentially Private Linear Models For High-Dimensional Data(https://arxiv.org/abs/2404.01141)
• Keywords: privacy, robust
• Abstract: Linear models are ubiquitous in data science, but are particularly prone to overfitting and data memorization in high dimensions. To guarantee the privacy of training data, differential privacy can be used. Many papers have proposed optimization techniques for high-dimensional differentially private linear models, but a systematic comparison between these methods does not exist. We close this gap by providing a comprehensive review of optimization methods for private high-dimensional linear models. Empirical tests on all methods demonstrate robust and coordinate-optimized algorithms perform best, which can inform future research. Code for implementing all methods is released online.

Title: Condition-Aware Neural Network for Controlled Image Generation

• Authors: Han Cai, Muyang Li, Zhuoyang Zhang, Qinsheng Zhang, Ming-Yu Liu, Song Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01143
• Pdf URL: https://arxiv.org/pdf/2404.01143
• Copy Paste: [[2404.01143]] Condition-Aware Neural Network for Controlled Image Generation(https://arxiv.org/abs/2404.01143)
• Keywords: diffusion, transformer, generative
• Abstract: We present Condition-Aware Neural Network (CAN), a new method for adding control to image generative models. In parallel to prior conditional control methods, CAN controls the image generation process by dynamically manipulating the weight of the neural network. This is achieved by introducing a condition-aware weight generation module that generates conditional weight for convolution/linear layers based on the input condition. We test CAN on class-conditional image generation on ImageNet and text-to-image generation on COCO. CAN consistently delivers significant improvements for diffusion transformer models, including DiT and UViT. In particular, CAN combined with EfficientViT (CaT) achieves 2.78 FID on ImageNet 512x512, surpassing DiT-XL/2 while requiring 52x fewer MACs per sampling step.

Title: Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit

• Authors: Parker Seegmiller, Joseph Gatto, Omar Sharif, Madhusudan Basak, Sarah Masud Preum
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01147
• Pdf URL: https://arxiv.org/pdf/2404.01147
• Copy Paste: [[2404.01147]] Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit(https://arxiv.org/abs/2404.01147)
• Keywords: large language model
• Abstract: Large language models (LLMs) have been shown to be proficient in correctly answering questions in the context of online discourse. However, the study of using LLMs to model human-like answers to fact-driven social media questions is still under-explored. In this work, we investigate how LLMs model the wide variety of human answers to fact-driven questions posed on several topic-specific Reddit communities, or subreddits. We collect and release a dataset of 409 fact-driven questions and 7,534 diverse, human-rated answers from 15 r/Ask{Topic} communities across 3 categories: profession, social identity, and geographic location. We find that LLMs are considerably better at modeling highly-rated human answers to such questions, as opposed to poorly-rated human answers. We present several directions for future research based on our initial findings.

Title: Uncovering the Text Embedding in Text-to-Image Diffusion Models

• Authors: Hu Yu, Hao Luo, Fan Wang, Feng Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01154
• Pdf URL: https://arxiv.org/pdf/2404.01154
• Copy Paste: [[2404.01154]] Uncovering the Text Embedding in Text-to-Image Diffusion Models(https://arxiv.org/abs/2404.01154)
• Keywords: diffusion
• Abstract: The correspondence between input text and the generated image exhibits opacity, wherein minor textual modifications can induce substantial deviations in the generated image. While, text embedding, as the pivotal intermediary between text and images, remains relatively underexplored. In this paper, we address this research gap by delving into the text embedding space, unleashing its capacity for controllable image editing and explicable semantic direction attributes within a learning-free framework. Specifically, we identify two critical insights regarding the importance of per-word embedding and their contextual correlations within text embedding, providing instructive principles for learning-free image editing. Additionally, we find that text embedding inherently possesses diverse semantic potentials, and further reveal this property through the lens of singular value decomposition (SVD). These uncovered properties offer practical utility for image editing and semantic discovery. More importantly, we expect the in-depth analyses and findings of the text embedding can enhance the understanding of text-to-image diffusion models.

Title: Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Training

• Authors: Vivian Liu, Yiqiao Yin
• Subjects: cs.CL, cs.PF
• Abstract URL: https://arxiv.org/abs/2404.01157
• Pdf URL: https://arxiv.org/pdf/2404.01157
• Copy Paste: [[2404.01157]] Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Training(https://arxiv.org/abs/2404.01157)
• Keywords: robust, large language model
• Abstract: Prominent works in the field of Natural Language Processing have long attempted to create new innovative models by improving upon previous model training approaches, altering model architecture, and developing more in-depth datasets to better their performance. However, with the quickly advancing field of NLP comes increased greenhouse gas emissions, posing concerns over the environmental damage caused by training LLMs. Gaining a comprehensive understanding of the various costs, particularly those pertaining to environmental aspects, that are associated with artificial intelligence serves as the foundational basis for ensuring safe AI models. Currently, investigations into the CO2 emissions of AI models remain an emerging area of research, and as such, in this paper, we evaluate the CO2 emissions of well-known large language models, which have an especially high carbon footprint due to their significant amount of model parameters. We argue for the training of LLMs in a way that is responsible and sustainable by suggesting measures for reducing carbon emissions. Furthermore, we discuss how the choice of hardware affects CO2 emissions by contrasting the CO2 emissions during model training for two widely used GPUs. Based on our results, we present the benefits and drawbacks of our proposed solutions and make the argument for the possibility of training more environmentally safe AI models without sacrificing their robustness and performance.

Title: LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models

• Authors: Haoran Li, Junqi Liu, Zexian Wang, Shiyuan Luo, Xiaowei Jia, Huaxiu Yao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01165
• Pdf URL: https://arxiv.org/pdf/2404.01165
• Copy Paste: [[2404.01165]] LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models(https://arxiv.org/abs/2404.01165)
• Keywords: large language model
• Abstract: The modeling of environmental ecosystems plays a pivotal role in the sustainable management of our planet. Accurate prediction of key environmental variables over space and time can aid in informed policy and decision-making, thus improving people's livelihood. Recently, deep learning-based methods have shown promise in modeling the spatial-temporal relationships for predicting environmental variables. However, these approaches often fall short in handling incomplete features and distribution shifts, which are commonly observed in environmental data due to the substantial cost of data collection and malfunctions in measuring instruments. To address these issues, we propose LITE -- a multimodal large language model for environmental ecosystems modeling. Specifically, LITE unifies different environmental variables by transforming them into natural language descriptions and line graph images. Then, LITE utilizes unified encoders to capture spatial-temporal dynamics and correlations in different modalities. During this step, the incomplete features are imputed by a sparse Mixture-of-Experts framework, and the distribution shift is handled by incorporating multi-granularity information from past observations. Finally, guided by domain instructions, a language model is employed to fuse the multimodal representations for the prediction. Our experiments demonstrate that LITE significantly enhances performance in environmental spatial-temporal prediction across different domains compared to the best baseline, with a 41.25% reduction in prediction error. This justifies its effectiveness. Our data and code are available at https://github.com/hrlics/LITE.

Title: Poisoning Decentralized Collaborative Recommender System and Its Countermeasures

• Authors: Ruiqi Zheng, Liang Qu, Tong Chen, Kai Zheng, Yuhui Shi, Hongzhi Yin
• Subjects: cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.01177
• Pdf URL: https://arxiv.org/pdf/2404.01177
• Copy Paste: [[2404.01177]] Poisoning Decentralized Collaborative Recommender System and Its Countermeasures(https://arxiv.org/abs/2404.01177)
• Keywords: security, privacy, attack, robust, federate
• Abstract: To make room for privacy and efficiency, the deployment of many recommender systems is experiencing a shift from central servers to personal devices, where the federated recommender systems (FedRecs) and decentralized collaborative recommender systems (DecRecs) are arguably the two most representative paradigms. While both leverage knowledge (e.g., gradients) sharing to facilitate learning local models, FedRecs rely on a central server to coordinate the optimization process, yet in DecRecs, the knowledge sharing directly happens between clients. Knowledge sharing also opens a backdoor for model poisoning attacks, where adversaries disguise themselves as benign clients and disseminate polluted knowledge to achieve malicious goals like promoting an item's exposure rate. Although research on such poisoning attacks provides valuable insights into finding security loopholes and corresponding countermeasures, existing attacks mostly focus on FedRecs, and are either inapplicable or ineffective for DecRecs. Compared with FedRecs where the tampered information can be universally distributed to all clients once uploaded to the cloud, each adversary in DecRecs can only communicate with neighbor clients of a small size, confining its impact to a limited range. To fill the gap, we present a novel attack method named Poisoning with Adaptive Malicious Neighbors (PAMN). With item promotion in top-K recommendation as the attack objective, PAMN effectively boosts target items' ranks with several adversaries that emulate benign clients and transfers adaptively crafted gradients conditioned on each adversary's neighbors. Moreover, with the vulnerabilities of DecRecs uncovered, a dedicated defensive mechanism based on user-level gradient clipping with sparsified updating is proposed. Extensive experiments demonstrate the effectiveness of the poisoning attack and the robustness of our defensive mechanism.

Title: A Neuro-Symbolic Approach to Monitoring Salt Content in Food

• Authors: Anuja Tayal, Barbara Di Eugenio, Devika Salunke, Andrew D. Boyd, Carolyn A Dickens, Eulalia P Abril, Olga Garcia-Bedoya, Paula G Allen-Meares
• Subjects: cs.CL, cs.SC
• Abstract URL: https://arxiv.org/abs/2404.01182
• Pdf URL: https://arxiv.org/pdf/2404.01182
• Copy Paste: [[2404.01182]] A Neuro-Symbolic Approach to Monitoring Salt Content in Food(https://arxiv.org/abs/2404.01182)
• Keywords: transformer
• Abstract: We propose a dialogue system that enables heart failure patients to inquire about salt content in foods and help them monitor and reduce salt intake. Addressing the lack of specific datasets for food-based salt content inquiries, we develop a template-based conversational dataset. The dataset is structured to ask clarification questions to identify food items and their salt content. Our findings indicate that while fine-tuning transformer-based models on the dataset yields limited performance, the integration of Neuro-Symbolic Rules significantly enhances the system's performance. Our experiments show that by integrating neuro-symbolic rules, our system achieves an improvement in joint goal accuracy of over 20% across different data sizes compared to naively fine-tuning transformer-based models.

Title: MonoBox: Tightness-free Box-supervised Polyp 001 001 Segmentation using Monotonicity Constraint

• Authors: Qiang Hu, Zhenyu Yi, Ying Zhou, Ting Li, Fan Huang, Mei Liu, Zhiwei Wang, Qiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01188
• Pdf URL: https://arxiv.org/pdf/2404.01188
• Copy Paste: [[2404.01188]] MonoBox: Tightness-free Box-supervised Polyp 001 001 Segmentation using Monotonicity Constraint(https://arxiv.org/abs/2404.01188)
• Keywords: robust, segmentation
• Abstract: We propose MonoBox, an innovative box-supervised segmentation method constrained by monotonicity to liberate its training from the user-unfriendly box-tightness assumption. In contrast to conventional box-supervised segmentation, where the box edges must precisely touch the target boundaries, MonoBox leverages imprecisely-annotated boxes to achieve robust pixel-wise segmentation. The 'linchpin' is that, within the noisy zones around box edges, MonoBox discards the traditional misguiding multiple-instance learning loss, and instead optimizes a carefully-designed objective, termed monotonicity constraint. Along directions transitioning from the foreground to background, this new constraint steers responses to adhere to a trend of monotonically decreasing values. Consequently, the originally unreliable learning within the noisy zones is transformed into a correct and effective monotonicity optimization. Moreover, an adaptive label correction is introduced, enabling MonoBox to enhance the tightness of box annotations using predicted masks from the previous epoch and dynamically shrink the noisy zones as training progresses. We verify MonoBox in the box-supervised segmentation task of polyps, where satisfying box-tightness is challenging due to the vague boundaries between the polyp and normal tissues. Experiments on both public synthetic and in-house real noisy datasets demonstrate that MonoBox exceeds other anti-noise state-of-the-arts by improving Dice by at least 5.5% and 3.3%, respectively. Codes are at https://github.com/Huster-Hq/MonoBox.

Title: Adaptive Query Prompting for Multi-Domain Landmark Detection

• Authors: Qiusen Wei, Guoheng Huang, Xiaochen Yuan, Xuhang Chen, Guo Zhong, Jianwen Huang, Jiajie Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01194
• Pdf URL: https://arxiv.org/pdf/2404.01194
• Copy Paste: [[2404.01194]] Adaptive Query Prompting for Multi-Domain Landmark Detection(https://arxiv.org/abs/2404.01194)
• Keywords: transformer
• Abstract: Medical landmark detection is crucial in various medical imaging modalities and procedures. Although deep learning-based methods have achieve promising performance, they are mostly designed for specific anatomical regions or tasks. In this work, we propose a universal model for multi-domain landmark detection by leveraging transformer architecture and developing a prompting component, named as Adaptive Query Prompting (AQP). Instead of embedding additional modules in the backbone network, we design a separate module to generate prompts that can be effectively extended to any other transformer network. In our proposed AQP, prompts are learnable parameters maintained in a memory space called prompt pool. The central idea is to keep the backbone frozen and then optimize prompts to instruct the model inference process. Furthermore, we employ a lightweight decoder to decode landmarks from the extracted features, namely Light-MLD. Thanks to the lightweight nature of the decoder and AQP, we can handle multiple datasets by sharing the backbone encoder and then only perform partial parameter tuning without incurring much additional cost. It has the potential to be extended to more landmark detection tasks. We conduct experiments on three widely used X-ray datasets for different medical landmark detection tasks. Our proposed Light-MLD coupled with AQP achieves SOTA performance on many metrics even without the use of elaborate structural designs or complex frameworks.

Title: Video Interpolation with Diffusion Models

• Authors: Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Hołyński, Ben Poole, Janne Kontkanen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01203
• Pdf URL: https://arxiv.org/pdf/2404.01203
• Copy Paste: [[2404.01203]] Video Interpolation with Diffusion Models(https://arxiv.org/abs/2404.01203)
• Keywords: diffusion, generative
• Abstract: We present VIDIM, a generative model for video interpolation, which creates short videos given a start and end frame. In order to achieve high fidelity and generate motions unseen in the input data, VIDIM uses cascaded diffusion models to first generate the target video at low resolution, and then generate the high-resolution video conditioned on the low-resolution generated video. We compare VIDIM to previous state-of-the-art methods on video interpolation, and demonstrate how such works fail in most settings where the underlying motion is complex, nonlinear, or ambiguous while VIDIM can easily handle such cases. We additionally demonstrate how classifier-free guidance on the start and end frame and conditioning the super-resolution model on the original high-resolution frames without additional parameters unlocks high-fidelity results. VIDIM is fast to sample from as it jointly denoises all the frames to be generated, requires less than a billion parameters per diffusion model to produce compelling results, and still enjoys scalability and improved quality at larger parameter counts.

Title: The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis

• Authors: Chen Yang, Junzhuo Li, Xinyao Niu, Xinrun Du, Songyang Gao, Haoran Zhang, Zhaoliang Chen, Xingwei Qu, Ruibin Yuan, Yizhi Li, Jiaheng Liu, Stephen W. Huang, Shawn Yue, Wenhu Chen, Jie Fu, Ge Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01204
• Pdf URL: https://arxiv.org/pdf/2404.01204
• Copy Paste: [[2404.01204]] The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis(https://arxiv.org/abs/2404.01204)
• Keywords: large language model
• Abstract: Uncovering early-stage metrics that reflect final model performance is one core principle for large-scale pretraining. The existing scaling law demonstrates the power-law correlation between pretraining loss and training flops, which serves as an important indicator of the current training state for large language models. However, this principle only focuses on the model's compression properties on the training data, resulting in an inconsistency with the ability improvements on the downstream tasks. Some follow-up works attempted to extend the scaling-law to more complex metrics (such as hyperparameters), but still lacked a comprehensive analysis of the dynamic differences among various capabilities during pretraining. To address the aforementioned limitations, this paper undertakes a comprehensive comparison of model capabilities at various pretraining intermediate checkpoints. Through this analysis, we confirm that specific downstream metrics exhibit similar training dynamics across models of different sizes, up to 67 billion parameters. In addition to our core findings, we've reproduced Amber and OpenLLaMA, releasing their intermediate checkpoints. This initiative offers valuable resources to the research community and facilitates the verification and exploration of LLM pretraining by open-source researchers. Besides, we provide empirical summaries, including performance comparisons of different models and capabilities, and tuition of key metrics for different training phases. Based on these findings, we provide a more user-friendly strategy for evaluating the optimization state, offering guidance for establishing a stable pretraining process.

Title: Machine Unlearning for Traditional Models and Large Language Models: A Short Survey

• Authors: Yi Xu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.01206
• Pdf URL: https://arxiv.org/pdf/2404.01206
• Copy Paste: [[2404.01206]] Machine Unlearning for Traditional Models and Large Language Models: A Short Survey(https://arxiv.org/abs/2404.01206)
• Keywords: privacy, large language model
• Abstract: With the implementation of personal data privacy regulations, the field of machine learning (ML) faces the challenge of the "right to be forgotten". Machine unlearning has emerged to address this issue, aiming to delete data and reduce its impact on models according to user requests. Despite the widespread interest in machine unlearning, comprehensive surveys on its latest advancements, especially in the field of Large Language Models (LLMs) is lacking. This survey aims to fill this gap by providing an in-depth exploration of machine unlearning, including the definition, classification and evaluation criteria, as well as challenges in different environments and their solutions. Specifically, this paper categorizes and investigates unlearning on both traditional models and LLMs, and proposes methods for evaluating the effectiveness and efficiency of unlearning, and standards for performance measurement. This paper reveals the limitations of current unlearning techniques and emphasizes the importance of a comprehensive unlearning evaluation to avoid arbitrary forgetting. This survey not only summarizes the key concepts of unlearning technology but also points out its prominent issues and feasible directions for future research, providing valuable guidance for scholars in the field.

Title: Vision-language models for decoding provider attention during neonatal resuscitation

• Authors: Felipe Parodi, Jordan Matelsky, Alejandra Regla-Vargas, Elizabeth Foglia, Charis Lim, Danielle Weinberg, Konrad Kording, Heidi Herrick, Michael Platt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01207
• Pdf URL: https://arxiv.org/pdf/2404.01207
• Copy Paste: [[2404.01207]] Vision-language models for decoding provider attention during neonatal resuscitation(https://arxiv.org/abs/2404.01207)
• Keywords: transformer, segmentation
• Abstract: Neonatal resuscitations demand an exceptional level of attentiveness from providers, who must process multiple streams of information simultaneously. Gaze strongly influences decision making; thus, understanding where a provider is looking during neonatal resuscitations could inform provider training, enhance real-time decision support, and improve the design of delivery rooms and neonatal intensive care units (NICUs). Current approaches to quantifying neonatal providers' gaze rely on manual coding or simulations, which limit scalability and utility. Here, we introduce an automated, real-time, deep learning approach capable of decoding provider gaze into semantic classes directly from first-person point-of-view videos recorded during live resuscitations. Combining state-of-the-art, real-time segmentation with vision-language models (CLIP), our low-shot pipeline attains 91\% classification accuracy in identifying gaze targets without training. Upon fine-tuning, the performance of our gaze-guided vision transformer exceeds 98\% accuracy in gaze classification, approaching human-level precision. This system, capable of real-time inference, enables objective quantification of provider attention dynamics during live neonatal resuscitation. Our approach offers a scalable solution that seamlessly integrates with existing infrastructure for data-scarce gaze analysis, thereby offering new opportunities for understanding and refining clinical decision making.

Title: Incorporating Domain Differential Equations into Graph Convolutional Networks to Lower Generalization Discrepancy

• Authors: Yue Sun, Chao Chen, Yuesheng Xu, Sihong Xie, Rick S. Blum, Parv Venkitasubramaniam
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01217
• Pdf URL: https://arxiv.org/pdf/2404.01217
• Copy Paste: [[2404.01217]] Incorporating Domain Differential Equations into Graph Convolutional Networks to Lower Generalization Discrepancy(https://arxiv.org/abs/2404.01217)
• Keywords: robust, diffusion
• Abstract: Ensuring both accuracy and robustness in time series prediction is critical to many applications, ranging from urban planning to pandemic management. With sufficient training data where all spatiotemporal patterns are well-represented, existing deep-learning models can make reasonably accurate predictions. However, existing methods fail when the training data are drawn from different circumstances (e.g., traffic patterns on regular days) compared to test data (e.g., traffic patterns after a natural disaster). Such challenges are usually classified under domain generalization. In this work, we show that one way to address this challenge in the context of spatiotemporal prediction is by incorporating domain differential equations into Graph Convolutional Networks (GCNs). We theoretically derive conditions where GCNs incorporating such domain differential equations are robust to mismatched training and testing data compared to baseline domain agnostic models. To support our theory, we propose two domain-differential-equation-informed networks called Reaction-Diffusion Graph Convolutional Network (RDGCN), which incorporates differential equations for traffic speed evolution, and Susceptible-Infectious-Recovered Graph Convolutional Network (SIRGCN), which incorporates a disease propagation model. Both RDGCN and SIRGCN are based on reliable and interpretable domain differential equations that allow the models to generalize to unseen patterns. We experimentally show that RDGCN and SIRGCN are more robust with mismatched testing data than the state-of-the-art deep learning methods.

Title: Towards System Modelling to Support Diseases Data Extraction from the Electronic Health Records for Physicians Research Activities

• Authors: Bushra F. Alsaqer, Alaa F. Alsaqer, Amna Asif
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.01218
• Pdf URL: https://arxiv.org/pdf/2404.01218
• Copy Paste: [[2404.01218]] Towards System Modelling to Support Diseases Data Extraction from the Electronic Health Records for Physicians Research Activities(https://arxiv.org/abs/2404.01218)
• Keywords: extraction
• Abstract: The use of Electronic Health Records (EHRs) has increased dramatically in the past 15 years, as, it is considered an important source of managing data od patients. The EHRs are primary sources of disease diagnosis and demographic data of patients worldwide. Therefore, the data can be utilized for secondary tasks such as research. This paper aims to make such data usable for research activities such as monitoring disease statistics for a specific population. As a result, the researchers can detect the disease causes for the behavior and lifestyle of the target group. One of the limitations of EHRs systems is that the data is not available in the standard format but in various forms. Therefore, it is required to first convert the names of the diseases and demographics data into one standardized form to make it usable for research activities. There is a large amount of EHRs available, and solving the standardizing issues requires some optimized techniques. We used a first-hand EHR dataset extracted from EHR systems. Our application uploads the dataset from the EHRs and converts it to the ICD-10 coding system to solve the standardization problem. So, we first apply the steps of pre-processing, annotation, and transforming the data to convert it into the standard form. The data pre-processing is applied to normalize demographic formats. In the annotation step, a machine learning model is used to recognize the diseases from the text. Furthermore, the transforming step converts the disease name to the ICD-10 coding format. The model was evaluated manually by comparing its performance in terms of disease recognition with an available dictionary-based system (MetaMap). The accuracy of the proposed machine learning model is 81%, that outperformed MetaMap accuracy of 67%. This paper contributed to system modelling for EHR data extraction to support research activities.

Title: Collaborative Pareto Set Learning in Multiple Multi-Objective Optimization Problems

• Authors: Chikai Shang, Rongguang Ye, Jiaqi Jiang, Fangqing Gu
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2404.01224
• Pdf URL: https://arxiv.org/pdf/2404.01224
• Copy Paste: [[2404.01224]] Collaborative Pareto Set Learning in Multiple Multi-Objective Optimization Problems(https://arxiv.org/abs/2404.01224)
• Keywords: robust
• Abstract: Pareto Set Learning (PSL) is an emerging research area in multi-objective optimization, focusing on training neural networks to learn the mapping from preference vectors to Pareto optimal solutions. However, existing PSL methods are limited to addressing a single Multi-objective Optimization Problem (MOP) at a time. When faced with multiple MOPs, this limitation not only leads to significant inefficiencies but also fails to exploit the potential synergies across varying MOPs. In this paper, we propose a Collaborative Pareto Set Learning (CoPSL) framework, which simultaneously learns the Pareto sets of multiple MOPs in a collaborative manner. CoPSL employs an architecture consisting of shared and MOP-specific layers, where shared layers aim to capture common relationships among MOPs collaboratively, and MOP-specific layers process these relationships to generate solution sets for each MOP. This collaborative approach enables CoPSL to efficiently learn the Pareto sets of multiple MOPs in a single run while leveraging the relationships among various MOPs. To further understand these relationships, we experimentally demonstrate that there exist shareable representations among MOPs. Leveraging these collaboratively shared representations can effectively improve the capability to approximate Pareto sets. Extensive experiments underscore the superior efficiency and robustness of CoPSL in approximating Pareto sets compared to state-of-the-art approaches on a variety of synthetic and real-world MOPs. Code is available at https://github.com/ckshang/CoPSL.

Title: LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models

• Authors: Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, Furu Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01230
• Pdf URL: https://arxiv.org/pdf/2404.01230
• Copy Paste: [[2404.01230]] LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models(https://arxiv.org/abs/2404.01230)
• Keywords: large language model
• Abstract: This paper presents a comprehensive survey of the current status and opportunities for Large Language Models (LLMs) in strategic reasoning, a sophisticated form of reasoning that necessitates understanding and predicting adversary actions in multi-agent settings while adjusting strategies accordingly. Strategic reasoning is distinguished by its focus on the dynamic and uncertain nature of interactions among multi-agents, where comprehending the environment and anticipating the behavior of others is crucial. We explore the scopes, applications, methodologies, and evaluation metrics related to strategic reasoning with LLMs, highlighting the burgeoning development in this area and the interdisciplinary approaches enhancing their decision-making performance. It aims to systematize and clarify the scattered literature on this subject, providing a systematic review that underscores the importance of strategic reasoning as a critical cognitive capability and offers insights into future research directions and potential improvements.

Title: Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models

• Authors: Yuxin Wen, Leo Marchyok, Sanghyun Hong, Jonas Geiping, Tom Goldstein, Nicholas Carlini
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01231
• Pdf URL: https://arxiv.org/pdf/2404.01231
• Copy Paste: [[2404.01231]] Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models(https://arxiv.org/abs/2404.01231)
• Keywords: privacy, attack, membership infer, large language model
• Abstract: It is commonplace to produce application-specific models by fine-tuning large pre-trained models using a small bespoke dataset. The widespread availability of foundation model checkpoints on the web poses considerable risks, including the vulnerability to backdoor attacks. In this paper, we unveil a new vulnerability: the privacy backdoor attack. This black-box privacy attack aims to amplify the privacy leakage that arises when fine-tuning a model: when a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. We conduct extensive experiments on various datasets and models, including both vision-language models (CLIP) and large language models, demonstrating the broad applicability and effectiveness of such an attack. Additionally, we carry out multiple ablation studies with different fine-tuning methods and inference strategies to thoroughly analyze this new threat. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.

Title: Open-Vocabulary Federated Learning with Multimodal Prototyping

• Authors: Huimin Zeng, Zhenrui Yue, Dong Wang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01232
• Pdf URL: https://arxiv.org/pdf/2404.01232
• Copy Paste: [[2404.01232]] Open-Vocabulary Federated Learning with Multimodal Prototyping(https://arxiv.org/abs/2404.01232)
• Keywords: robust, federate
• Abstract: Existing federated learning (FL) studies usually assume the training label space and test label space are identical. However, in real-world applications, this assumption is too ideal to be true. A new user could come up with queries that involve data from unseen classes, and such open-vocabulary queries would directly defect such FL systems. Therefore, in this work, we explicitly focus on the under-explored open-vocabulary challenge in FL. That is, for a new user, the global server shall understand her/his query that involves arbitrary unknown classes. To address this problem, we leverage the pre-trained vision-language models (VLMs). In particular, we present a novel adaptation framework tailored for VLMs in the context of FL, named as Federated Multimodal Prototyping (Fed-MP). Fed-MP adaptively aggregates the local model weights based on light-weight client residuals, and makes predictions based on a novel multimodal prototyping mechanism. Fed-MP exploits the knowledge learned from the seen classes, and robustifies the adapted VLM to unseen categories. Our empirical evaluation on various datasets validates the effectiveness of Fed-MP.

Title: GFLean: An Autoformalisation Framework for Lean via GF

• Authors: Shashank Pathak
• Subjects: cs.CL, math.LO
• Abstract URL: https://arxiv.org/abs/2404.01234
• Pdf URL: https://arxiv.org/pdf/2404.01234
• Copy Paste: [[2404.01234]] GFLean: An Autoformalisation Framework for Lean via GF(https://arxiv.org/abs/2404.01234)
• Keywords: robust
• Abstract: We present an autoformalisation framework for the Lean theorem prover, called GFLean. GFLean uses a high-level grammar writing tool called Grammatical Framework (GF) for parsing and linearisation. GFLean is implemented in Haskell. We explain the functionalities of GFLean, its inner working and discuss its limitations. We also discuss how we can use neural network based translation programs and rule based translation programs together complimenting each other to build robust autoformalisation frameworks.

Title: StructLDM: Structured Latent Diffusion for 3D Human Generation

• Authors: Tao Hu, Fangzhou Hong, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01241
• Pdf URL: https://arxiv.org/pdf/2404.01241
• Copy Paste: [[2404.01241]] StructLDM: Structured Latent Diffusion for 3D Human Generation(https://arxiv.org/abs/2404.01241)
• Keywords: diffusion, generative
• Abstract: Recent 3D human generative models have achieved remarkable progress by learning 3D-aware GANs from 2D images. However, existing 3D human generative methods model humans in a compact 1D latent space, ignoring the articulated structure and semantics of human body topology. In this paper, we explore more expressive and higher-dimensional latent space for 3D human modeling and propose StructLDM, a diffusion-based unconditional 3D human generative model, which is learned from 2D images. StructLDM solves the challenges imposed due to the high-dimensional growth of latent space with three key designs: 1) A semantic structured latent space defined on the dense surface manifold of a statistical human body template. 2) A structured 3D-aware auto-decoder that factorizes the global latent space into several semantic body parts parameterized by a set of conditional structured local NeRFs anchored to the body template, which embeds the properties learned from the 2D training data and can be decoded to render view-consistent humans under different poses and clothing styles. 3) A structured latent diffusion model for generative human appearance sampling. Extensive experiments validate StructLDM's state-of-the-art generation performance and illustrate the expressiveness of the structured latent space over the well-adopted 1D latent space. Notably, StructLDM enables different levels of controllable 3D human generation and editing, including pose/view/shape control, and high-level tasks including compositional generations, part-aware clothing editing, 3D virtual try-on, etc. Our project page is at: https://taohuumd.github.io/projects/StructLDM/.

Title: A Unified and Interpretable Emotion Representation and Expression Generation

• Authors: Reni Paskaleva, Mykyta Holubakha, Andela Ilic, Saman Motamed, Luc Van Gool, Danda Paudel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01243
• Pdf URL: https://arxiv.org/pdf/2404.01243
• Copy Paste: [[2404.01243]] A Unified and Interpretable Emotion Representation and Expression Generation(https://arxiv.org/abs/2404.01243)
• Keywords: diffusion
• Abstract: Canonical emotions, such as happy, sad, and fearful, are easy to understand and annotate. However, emotions are often compound, e.g. happily surprised, and can be mapped to the action units (AUs) used for expressing emotions, and trivially to the canonical ones. Intuitively, emotions are continuous as represented by the arousal-valence (AV) model. An interpretable unification of these four modalities - namely, Canonical, Compound, AUs, and AV - is highly desirable, for a better representation and understanding of emotions. However, such unification remains to be unknown in the current literature. In this work, we propose an interpretable and unified emotion model, referred as C2A2. We also develop a method that leverages labels of the non-unified models to annotate the novel unified one. Finally, we modify the text-conditional diffusion models to understand continuous numbers, which are then used to generate continuous expressions using our unified emotion model. Through quantitative and qualitative experiments, we show that our generated images are rich and capture subtle expressions. Our work allows a fine-grained generation of expressions in conjunction with other textual inputs and offers a new label space for emotions at the same time.

Title: An image speaks a thousand words, but can everyone listen? On translating images for cultural relevance

• Authors: Simran Khanuja, Sathyanarayanan Ramamoorthy, Yueqi Song, Graham Neubig
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01247
• Pdf URL: https://arxiv.org/pdf/2404.01247
• Copy Paste: [[2404.01247]] An image speaks a thousand words, but can everyone listen? On translating images for cultural relevance(https://arxiv.org/abs/2404.01247)
• Keywords: generative
• Abstract: Given the rise of multimedia content, human translators increasingly focus on culturally adapting not only words but also other modalities such as images to convey the same meaning. While several applications stand to benefit from this, machine translation systems remain confined to dealing with language in speech and text. In this work, we take a first step towards translating images to make them culturally relevant. First, we build three pipelines comprising state-of-the-art generative models to do the task. Next, we build a two-part evaluation dataset: i) concept: comprising 600 images that are cross-culturally coherent, focusing on a single concept per image, and ii) application: comprising 100 images curated from real-world applications. We conduct a multi-faceted human evaluation of translated images to assess for cultural relevance and meaning preservation. We find that as of today, image-editing models fail at this task, but can be improved by leveraging LLMs and retrievers in the loop. Best pipelines can only translate 5% of images for some countries in the easier concept dataset and no translation is successful for some countries in the application dataset, highlighting the challenging nature of the task. Our code and data is released here: https://github.com/simran-khanuja/image-transcreation.

Title: Scalable Scene Modeling from Perspective Imaging: Physics-based Appearance and Geometry Inference

• Authors: Shuang Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01248
• Pdf URL: https://arxiv.org/pdf/2404.01248
• Copy Paste: [[2404.01248]] Scalable Scene Modeling from Perspective Imaging: Physics-based Appearance and Geometry Inference(https://arxiv.org/abs/2404.01248)
• Keywords: robust
• Abstract: 3D scene modeling techniques serve as the bedrocks in the geospatial engineering and computer science, which drives many applications ranging from automated driving, terrain mapping, navigation, virtual, augmented, mixed, and extended reality (for gaming and movie industry etc.). This dissertation presents a fraction of contributions that advances 3D scene modeling to its state of the art, in the aspects of both appearance and geometry modeling. In contrast to the prevailing deep learning methods, as a core contribution, this thesis aims to develop algorithms that follow first principles, where sophisticated physic-based models are introduced alongside with simpler learning and inference tasks. The outcomes of these algorithms yield processes that can consume much larger volume of data for highly accurate reconstructing 3D scenes at a scale without losing methodological generality, which are not possible by contemporary complex-model based deep learning methods. Specifically, the dissertation introduces three novel methodologies that address the challenges of inferring appearance and geometry through physics-based modeling. Overall, the research encapsulated in this dissertation marks a series of methodological triumphs in the processing of complex datasets. By navigating the confluence of deep learning, computational geometry, and photogrammetry, this work lays down a robust framework for future exploration and practical application in the rapidly evolving field of 3D scene reconstruction. The outcomes of these studies are evidenced through rigorous experiments and comparisons with existing state-of-the-art methods, demonstrating the efficacy and scalability of the proposed approaches.

Title: FireANTs: Adaptive Riemannian Optimization for Multi-Scale Diffeomorphic Registration

• Authors: Rohit Jena, Pratik Chaudhari, James C. Gee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01249
• Pdf URL: https://arxiv.org/pdf/2404.01249
• Copy Paste: [[2404.01249]] FireANTs: Adaptive Riemannian Optimization for Multi-Scale Diffeomorphic Registration(https://arxiv.org/abs/2404.01249)
• Keywords: robust, segmentation
• Abstract: Diffeomorphic Image Registration is a critical part of the analysis in various imaging modalities and downstream tasks like image translation, segmentation, and atlas building. Registration algorithms based on optimization have stood the test of time in terms of accuracy, reliability, and robustness across a wide spectrum of modalities and acquisition settings. However, these algorithms converge slowly, are prohibitively expensive to run, and their usage requires a steep learning curve, limiting their scalability to larger clinical and scientific studies. In this paper, we develop multi-scale Adaptive Riemannian Optimization algorithms for diffeomorphic image registration. We demonstrate compelling improvements on image registration across a spectrum of modalities and anatomies by measuring structural and landmark overlap of the registered image volumes. Our proposed framework leads to a consistent improvement in performance, and from 300x up to 2000x speedup over existing algorithms. Our modular library design makes it easy to use and allows customization via user-defined cost functions.

Title: UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing

• Authors: Yijun Yang, Jie He, Pinzhen Chen, Víctor Gutiérrez-Basulto, Jeff Z. Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01253
• Pdf URL: https://arxiv.org/pdf/2404.01253
• Copy Paste: [[2404.01253]] UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing(https://arxiv.org/abs/2404.01253)
• Keywords: extraction, large language model
• Abstract: Several recent papers have investigated the potential of language models as knowledge bases as well as the existence of severe biases when extracting factual knowledge. In this work, we focus on the factual probing performance over unseen prompts from tuning, and using a probabilistic view we show the inherent misalignment between pre-training and downstream tuning objectives in language models for probing knowledge. We hypothesize that simultaneously debiasing these objectives can be the key to generalisation over unseen prompts. We propose an adapter-based framework, UniArk, for generalised and consistent factual knowledge extraction through simple methods without introducing extra parameters. Extensive experiments show that UniArk can significantly improve the model's out-of-domain generalisation as well as consistency under various prompts. Additionally, we construct ParaTrex, a large-scale and diverse dataset for measuring the inconsistency and out-of-domain generation of models. Further, ParaTrex offers a reference method for constructing paraphrased datasets using large language models.

Title: Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward

• Authors: Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, Yiming Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01258
• Pdf URL: https://arxiv.org/pdf/2404.01258
• Copy Paste: [[2404.01258]] Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward(https://arxiv.org/abs/2404.01258)
• Keywords: robust, large language model
• Abstract: Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for detecting hallucinations in generated responses, remains a significant challenge. Previous studies have explored using large large multimodal models (LMMs) as reward models to guide preference modeling, but their ability to accurately assess the factuality of generated responses compared to corresponding videos has not been conclusively established. This paper introduces a novel framework that utilizes detailed video captions as a proxy of video content, enabling language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. Our approach demonstrates robust alignment with OpenAI GPT-4V model's reward mechanism, which directly takes video frames as input. Furthermore, we show that applying this tailored reward through DPO significantly improves the performance of video LMMs on video QA tasks.

Title: Bridging Remote Sensors with Multisensor Geospatial Foundation Models

• Authors: Boran Han, Shuai Zhang, Xingjian Shi, Markus Reichstein
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01260
• Pdf URL: https://arxiv.org/pdf/2404.01260
• Copy Paste: [[2404.01260]] Bridging Remote Sensors with Multisensor Geospatial Foundation Models(https://arxiv.org/abs/2404.01260)
• Keywords: segmentation
• Abstract: In the realm of geospatial analysis, the diversity of remote sensors, encompassing both optical and microwave technologies, offers a wealth of distinct observational capabilities. Recognizing this, we present msGFM, a multisensor geospatial foundation model that effectively unifies data from four key sensor modalities. This integration spans an expansive dataset of two million multisensor images. msGFM is uniquely adept at handling both paired and unpaired sensor data. For data originating from identical geolocations, our model employs an innovative cross-sensor pretraining approach in masked image modeling, enabling the synthesis of joint representations from diverse sensors. msGFM, incorporating four remote sensors, upholds strong performance, forming a comprehensive model adaptable to various sensor types. msGFM has demonstrated enhanced proficiency in a range of both single-sensor and multisensor downstream tasks. These include scene classification, segmentation, cloud removal, and pan-sharpening. A key discovery of our research is that representations derived from natural images are not always compatible with the distinct characteristics of geospatial remote sensors, underscoring the limitations of existing representations in this field. Our work can serve as a guide for developing multisensor geospatial pretraining models, paving the way for more advanced geospatial capabilities.

Title: FABLES: Evaluating faithfulness and content selection in book-length summarization

• Authors: Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, Mohit Iyyer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01261
• Pdf URL: https://arxiv.org/pdf/2404.01261
• Copy Paste: [[2404.01261]] FABLES: Evaluating faithfulness and content selection in book-length summarization(https://arxiv.org/abs/2404.01261)
• Keywords: large language model
• Abstract: While long-context large language models (LLMs) can technically summarize book-length documents (>100K tokens), the length and complexity of the documents have so far prohibited evaluations of input-dependent aspects like faithfulness. In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books. Our study mitigates the issue of data contamination by focusing on summaries of books published in 2023 or 2024, and we hire annotators who have fully read each book prior to the annotation task to minimize cost and cognitive burden. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD, which allows us to rank LLM summarizers based on faithfulness: Claude-3-Opus significantly outperforms all closed-source LLMs, while the open-source Mixtral is on par with GPT-3.5-Turbo. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate. While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially with regard to detecting unfaithful claims. Our experiments suggest that detecting unfaithful claims is an important future direction not only for summarization evaluation but also as a testbed for long-context understanding. Finally, we move beyond faithfulness by exploring content selection errors in book-length summarization: we develop a typology of omission errors related to crucial narrative elements and also identify a systematic over-emphasis on events occurring towards the end of the book.

Title: Mapping the Increasing Use of LLMs in Scientific Papers

• Authors: Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D Manning, James Y. Zou
• Subjects: cs.CL, cs.AI, cs.DL, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.01268
• Pdf URL: https://arxiv.org/pdf/2404.01268
• Copy Paste: [[2404.01268]] Mapping the Increasing Use of LLMs in Scientific Papers(https://arxiv.org/abs/2404.01268)
• Keywords: robust, large language model
• Abstract: Scientific publishing lays the foundation of science by disseminating research findings, fostering collaboration, encouraging reproducibility, and ensuring that scientific knowledge is accessible, verifiable, and built upon over time. Recently, there has been immense speculation about how many people are using large language models (LLMs) like ChatGPT in their academic writing, and to what extent this tool might have an effect on global scientific practices. However, we lack a precise measure of the proportion of academic writing substantially modified or produced by LLMs. To address this gap, we conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals, using a population-level statistical framework to measure the prevalence of LLM-modified content over time. Our statistical estimation operates on the corpus level and is more robust than inference on individual instances. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers (up to 17.5%). In comparison, Mathematics papers and the Nature portfolio showed the least LLM modification (up to 6.3%). Moreover, at an aggregate level, our analysis reveals that higher levels of LLM-modification are associated with papers whose first authors post preprints more frequently, papers in more crowded research areas, and papers of shorter lengths. Our findings suggests that LLMs are being broadly used in scientific writings.

Title: Decentralized Collaborative Learning Framework with External Privacy Leakage Analysis

• Authors: Tsuyoshi Idé, Dzung T. Phan, Rudy Raymond
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.01270
• Pdf URL: https://arxiv.org/pdf/2404.01270
• Copy Paste: [[2404.01270]] Decentralized Collaborative Learning Framework with External Privacy Leakage Analysis(https://arxiv.org/abs/2404.01270)
• Keywords: privacy
• Abstract: This paper presents two methodological advancements in decentralized multi-task learning under privacy constraints, aiming to pave the way for future developments in next-generation Blockchain platforms. First, we expand the existing framework for collaborative dictionary learning (CollabDict), which has previously been limited to Gaussian mixture models, by incorporating deep variational autoencoders (VAEs) into the framework, with a particular focus on anomaly detection. We demonstrate that the VAE-based anomaly score function shares the same mathematical structure as the non-deep model, and provide comprehensive qualitative comparison. Second, considering the widespread use of "pre-trained models," we provide a mathematical analysis on data privacy leakage when models trained with CollabDict are shared externally. We show that the CollabDict approach, when applied to Gaussian mixtures, adheres to a Renyi differential privacy criterion. Additionally, we propose a practical metric for monitoring internal privacy breaches during the learning process.

Title: Language Guided Domain Generalized Medical Image Segmentation

• Authors: Shahina Kunhimon, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01272
• Pdf URL: https://arxiv.org/pdf/2404.01272
• Copy Paste: [[2404.01272]] Language Guided Domain Generalized Medical Image Segmentation(https://arxiv.org/abs/2404.01272)
• Keywords: privacy, robust, segmentation
• Abstract: Single source domain generalization (SDG) holds promise for more reliable and consistent image segmentation across real-world clinical settings particularly in the medical domain, where data privacy and acquisition cost constraints often limit the availability of diverse datasets. Depending solely on visual features hampers the model's capacity to adapt effectively to various domains, primarily because of the presence of spurious correlations and domain-specific characteristics embedded within the image features. Incorporating text features alongside visual features is a potential solution to enhance the model's understanding of the data, as it goes beyond pixel-level information to provide valuable context. Textual cues describing the anatomical structures, their appearances, and variations across various imaging modalities can guide the model in domain adaptation, ultimately contributing to more robust and consistent segmentation. In this paper, we propose an approach that explicitly leverages textual information by incorporating a contrastive learning mechanism guided by the text encoder features to learn a more robust feature representation. We assess the effectiveness of our text-guided contrastive feature alignment technique in various scenarios, including cross-modality, cross-sequence, and cross-site settings for different segmentation tasks. Our approach achieves favorable performance against existing methods in literature. Our code and model weights are available at https://github.com/ShahinaKK/LG_SDG.git.

Title: TWIN-GPT: Digital Twins for Clinical Trials via Large Language Model

• Authors: Yue Wang, Yingzhou Lu, Yinlong Xu, Zihan Ma, Hongxia Xu, Bang Du, Honghao Gao, Jian Wu
• Subjects: cs.LG, cs.CL, stat.ME
• Abstract URL: https://arxiv.org/abs/2404.01273
• Pdf URL: https://arxiv.org/pdf/2404.01273
• Copy Paste: [[2404.01273]] TWIN-GPT: Digital Twins for Clinical Trials via Large Language Model(https://arxiv.org/abs/2404.01273)
• Keywords: large language model
• Abstract: Recently, there has been a burgeoning interest in virtual clinical trials, which simulate real-world scenarios and hold the potential to significantly enhance patient safety, expedite development, reduce costs, and contribute to the broader scientific knowledge in healthcare. Existing research often focuses on leveraging electronic health records (EHRs) to support clinical trial outcome prediction. Yet, trained with limited clinical trial outcome data, existing approaches frequently struggle to perform accurate predictions. Some research has attempted to generate EHRs to augment model development but has fallen short in personalizing the generation for individual patient profiles. Recently, the emergence of large language models has illuminated new possibilities, as their embedded comprehensive clinical knowledge has proven beneficial in addressing medical issues. In this paper, we propose a large language model-based digital twin creation approach, called TWIN-GPT. TWIN-GPT can establish cross-dataset associations of medical information given limited data, generating unique personalized digital twins for different patients, thereby preserving individual patient characteristics. Comprehensive experiments show that using digital twins created by TWIN-GPT can boost clinical trial outcome prediction, exceeding various previous prediction approaches. Besides, we also demonstrate that TWIN-GPT can generate high-fidelity trial data that closely approximate specific patients, aiding in more accurate result predictions in data-scarce situations. Moreover, our study provides practical evidence for the application of digital twins in healthcare, highlighting its potential significance.

Title: Large Motion Model for Unified Multi-Modal Motion Generation

• Authors: Mingyuan Zhang, Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01284
• Pdf URL: https://arxiv.org/pdf/2404.01284
• Copy Paste: [[2404.01284]] Large Motion Model for Unified Multi-Modal Motion Generation(https://arxiv.org/abs/2404.01284)
• Keywords: diffusion, transformer
• Abstract: Human motion generation, a cornerstone technique in animation and video production, has widespread applications in various tasks like text-to-motion and music-to-dance. Previous works focus on developing specialist models tailored for each task without scalability. In this work, we present Large Motion Model (LMM), a motion-centric, multi-modal framework that unifies mainstream motion generation tasks into a generalist model. A unified motion model is appealing since it can leverage a wide range of motion data to achieve broad generalization beyond a single task. However, it is also challenging due to the heterogeneous nature of substantially different motion data and tasks. LMM tackles these challenges from three principled aspects: 1) Data: We consolidate datasets with different modalities, formats and tasks into a comprehensive yet unified motion generation dataset, MotionVerse, comprising 10 tasks, 16 datasets, a total of 320k sequences, and 100 million frames. 2) Architecture: We design an articulated attention mechanism ArtAttention that incorporates body part-aware modeling into Diffusion Transformer backbone. 3) Pre-Training: We propose a novel pre-training strategy for LMM, which employs variable frame rates and masking forms, to better exploit knowledge from diverse training data. Extensive experiments demonstrate that our generalist LMM achieves competitive performance across various standard motion generation tasks over state-of-the-art specialist models. Notably, LMM exhibits strong generalization capabilities and emerging properties across many unseen tasks. Additionally, our ablation studies reveal valuable insights about training and scaling up large motion models for future research.

Title: Large Language Models are Capable of Offering Cognitive Reappraisal, if Guided

• Authors: Hongli Zhan, Allen Zheng, Yoon Kyung Lee, Jina Suh, Junyi Jessy Li, Desmond C. Ong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.01288
• Pdf URL: https://arxiv.org/pdf/2404.01288
• Copy Paste: [[2404.01288]] Large Language Models are Capable of Offering Cognitive Reappraisal, if Guided(https://arxiv.org/abs/2404.01288)
• Keywords: large language model
• Abstract: Large language models (LLMs) have offered new opportunities for emotional support, and recent work has shown that they can produce empathic responses to people in distress. However, long-term mental well-being requires emotional self-regulation, where a one-time empathic response falls short. This work takes a first step by engaging with cognitive reappraisals, a strategy from psychology practitioners that uses language to targetedly change negative appraisals that an individual makes of the situation; such appraisals is known to sit at the root of human emotional experience. We hypothesize that psychologically grounded principles could enable such advanced psychology capabilities in LLMs, and design RESORT which consists of a series of reappraisal constitutions across multiple dimensions that can be used as LLM instructions. We conduct a first-of-its-kind expert evaluation (by clinical psychologists with M.S. or Ph.D. degrees) of an LLM's zero-shot ability to generate cognitive reappraisal responses to medium-length social media messages asking for support. This fine-grained evaluation showed that even LLMs at the 7B scale guided by RESORT are capable of generating empathic responses that can help users reappraise their situations.

Title: Evaluating Text-to-Visual Generation with Image-to-Text Generation

• Authors: Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, Deva Ramanan
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.01291
• Pdf URL: https://arxiv.org/pdf/2404.01291
• Copy Paste: [[2404.01291]] Evaluating Text-to-Visual Generation with Image-to-Text Generation(https://arxiv.org/abs/2404.01291)
• Keywords: diffusion, generative
• Abstract: Despite significant progress in generative AI, comprehensive evaluation remains challenging because of the lack of effective metrics and standardized benchmarks. For instance, the widely-used CLIPScore measures the alignment between a (generated) image and text prompt, but it fails to produce reliable scores for complex prompts involving compositions of objects, attributes, and relations. One reason is that text encoders of CLIP can notoriously act as a "bag of words", conflating prompts such as "the horse is eating the grass" with "the grass is eating the horse". To address this, we introduce the VQAScore, which uses a visual-question-answering (VQA) model to produce an alignment score by computing the probability of a "Yes" answer to a simple "Does this figure show '{text}'?" question. Though simpler than prior art, VQAScore computed with off-the-shelf models produces state-of-the-art results across many (8) image-text alignment benchmarks. We also compute VQAScore with an in-house model that follows best practices in the literature. For example, we use a bidirectional image-question encoder that allows image embeddings to depend on the question being asked (and vice versa). Our in-house model, CLIP-FlanT5, outperforms even the strongest baselines that make use of the proprietary GPT-4V. Interestingly, although we train with only images, VQAScore can also align text with video and 3D models. VQAScore allows researchers to benchmark text-to-visual generation using complex texts that capture the compositional structure of real-world prompts. We introduce GenAI-Bench, a more challenging benchmark with 1,600 compositional text prompts that require parsing scenes, objects, attributes, relationships, and high-order reasoning like comparison and logic. GenAI-Bench also offers over 15,000 human ratings for leading image and video generation models such as Stable Diffusion, DALL-E 3, and Gen2.

Title: Measuring Style Similarity in Diffusion Models

• Authors: Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geiping, Abhinav Shrivastava, Tom Goldstein
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01292
• Pdf URL: https://arxiv.org/pdf/2404.01292
• Copy Paste: [[2404.01292]] Measuring Style Similarity in Diffusion Models(https://arxiv.org/abs/2404.01292)
• Keywords: diffusion, generative
• Abstract: Generative models are now widely used by graphic designers and artists. Prior works have shown that these models remember and often replicate content from their training data during generation. Hence as their proliferation increases, it has become important to perform a database search to determine whether the properties of the image are attributable to specific training data, every time before a generated image is used for professional purposes. Existing tools for this purpose focus on retrieving images of similar semantic content. Meanwhile, many artists are concerned with style replication in text-to-image models. We present a framework for understanding and extracting style descriptors from images. Our framework comprises a new dataset curated using the insight that style is a subjective property of an image that captures complex yet meaningful interactions of factors including but not limited to colors, textures, shapes, etc. We also propose a method to extract style descriptors that can be used to attribute style of a generated image to the images used in the training dataset of a text-to-image model. We showcase promising results in various style retrieval tasks. We also quantitatively and qualitatively analyze style attribution and matching in the Stable Diffusion model. Code and artifacts are available at https://github.com/learn2phoenix/CSD.

Title: CosmicMan: A Text-to-Image Foundation Model for Humans

• Authors: Shikai Li, Jianglin Fu, Kaiyuan Liu, Wentao Wang, Kwan-Yee Lin, Wayne Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01294
• Pdf URL: https://arxiv.org/pdf/2404.01294
• Copy Paste: [[2404.01294]] CosmicMan: A Text-to-Image Foundation Model for Humans(https://arxiv.org/abs/2404.01294)
• Keywords: diffusion
• Abstract: We present CosmicMan, a text-to-image foundation model specialized for generating high-fidelity human images. Unlike current general-purpose foundation models that are stuck in the dilemma of inferior quality and text-image misalignment for humans, CosmicMan enables generating photo-realistic human images with meticulous appearance, reasonable structure, and precise text-image alignment with detailed dense descriptions. At the heart of CosmicMan's success are the new reflections and perspectives on data and models: (1) We found that data quality and a scalable data production flow are essential for the final results from trained models. Hence, we propose a new data production paradigm, Annotate Anyone, which serves as a perpetual data flywheel to produce high-quality data with accurate yet cost-effective annotations over time. Based on this, we constructed a large-scale dataset, CosmicMan-HQ 1.0, with 6 Million high-quality real-world human images in a mean resolution of 1488x1255, and attached with precise text annotations deriving from 115 Million attributes in diverse granularities. (2) We argue that a text-to-image foundation model specialized for humans must be pragmatic -- easy to integrate into down-streaming tasks while effective in producing high-quality human images. Hence, we propose to model the relationship between dense text descriptions and image pixels in a decomposed manner, and present Decomposed-Attention-Refocusing (Daring) training framework. It seamlessly decomposes the cross-attention features in existing text-to-image diffusion model, and enforces attention refocusing without adding extra modules. Through Daring, we show that explicitly discretizing continuous text space into several basic groups that align with human body structure is the key to tackling the misalignment problem in a breeze.

Title: Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models

• Authors: Yi-Lin Tuan, Xilun Chen, Eric Michael Smith, Louis Martin, Soumya Batra, Asli Celikyilmaz, William Yang Wang, Daniel M. Bikel
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.01295
• Pdf URL: https://arxiv.org/pdf/2404.01295
• Copy Paste: [[2404.01295]] Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models(https://arxiv.org/abs/2404.01295)
• Keywords: large language model
• Abstract: As large language models (LLMs) become easily accessible nowadays, the trade-off between safety and helpfulness can significantly impact user experience. A model that prioritizes safety will cause users to feel less engaged and assisted while prioritizing helpfulness will potentially cause harm. Possible harms include teaching people how to build a bomb, exposing youth to inappropriate content, and hurting users' mental health. In this work, we propose to balance safety and helpfulness in diverse use cases by controlling both attributes in LLM. We explore training-free and fine-tuning methods that do not require extra human annotations and analyze the challenges of controlling safety and helpfulness in LLMs. Our experiments demonstrate that our method can rewind a learned model and unlock its controllability.

Title: MagicMirror: Fast and High-Quality Avatar Generation with a Constrained Search Space

• Authors: Armand Comas-Massagué, Di Qiu, Menglei Chai, Marcel Bühler, Amit Raj, Ruiqi Gao, Qiangeng Xu, Mark Matthews, Paulo Gotardo, Octavia Camps, Sergio Orts-Escolano, Thabo Beeler
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.01296
• Pdf URL: https://arxiv.org/pdf/2404.01296
• Copy Paste: [[2404.01296]] MagicMirror: Fast and High-Quality Avatar Generation with a Constrained Search Space(https://arxiv.org/abs/2404.01296)
• Keywords: diffusion
• Abstract: We introduce a novel framework for 3D human avatar generation and personalization, leveraging text prompts to enhance user engagement and customization. Central to our approach are key innovations aimed at overcoming the challenges in photo-realistic avatar synthesis. Firstly, we utilize a conditional Neural Radiance Fields (NeRF) model, trained on a large-scale unannotated multi-view dataset, to create a versatile initial solution space that accelerates and diversifies avatar generation. Secondly, we develop a geometric prior, leveraging the capabilities of Text-to-Image Diffusion Models, to ensure superior view invariance and enable direct optimization of avatar geometry. These foundational ideas are complemented by our optimization pipeline built on Variational Score Distillation (VSD), which mitigates texture loss and over-saturation issues. As supported by our extensive experiments, these strategies collectively enable the creation of custom avatars with unparalleled visual quality and better adherence to input text prompts. You can find more results and videos in our website: https://syntec-research.github.io/MagicMirror

Title: Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras

• Authors: Ruiming Cao, Dekel Galor, Amit Kohli, Jacob L Yates, Laura Waller
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.01298
• Pdf URL: https://arxiv.org/pdf/2404.01298
• Copy Paste: [[2404.01298]] Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras(https://arxiv.org/abs/2404.01298)
• Keywords: robust
• Abstract: Event cameras capture changes of intensity over time as a stream of 'events' and generally cannot measure intensity itself; hence, they are only used for imaging dynamic scenes. However, fluctuations due to random photon arrival inevitably trigger noise events, even for static scenes. While previous efforts have been focused on filtering out these undesirable noise events to improve signal quality, we find that, in the photon-noise regime, these noise events are correlated with the static scene intensity. We analyze the noise event generation and model its relationship to illuminance. Based on this understanding, we propose a method, called Noise2Image, to leverage the illuminance-dependent noise characteristics to recover the static parts of a scene, which are otherwise invisible to event cameras. We experimentally collect a dataset of noise events on static scenes to train and validate Noise2Image. Our results show that Noise2Image can robustly recover intensity images solely from noise events, providing a novel approach for capturing static scenes in event cameras, without additional hardware.

Title: NeRF-MAE : Masked AutoEncoders for Self Supervised 3D representation Learning for Neural Radiance Fields

• Authors: Muhammad Zubair Irshad, Sergey Zakahrov, Vitor Guizilini, Adrien Gaidon, Zsolt Kira, Rares Ambrus
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.01300
• Pdf URL: https://arxiv.org/pdf/2404.01300
• Copy Paste: [[2404.01300]] NeRF-MAE : Masked AutoEncoders for Self Supervised 3D representation Learning for Neural Radiance Fields(https://arxiv.org/abs/2404.01300)
• Keywords: transformer
• Abstract: Neural fields excel in computer vision and robotics due to their ability to understand the 3D visual world such as inferring semantics, geometry, and dynamics. Given the capabilities of neural fields in densely representing a 3D scene from 2D images, we ask the question: Can we scale their self-supervised pretraining, specifically using masked autoencoders, to generate effective 3D representations from posed RGB images. Owing to the astounding success of extending transformers to novel data modalities, we employ standard 3D Vision Transformers to suit the unique formulation of NeRFs. We leverage NeRF's volumetric grid as a dense input to the transformer, contrasting it with other 3D representations such as pointclouds where the information density can be uneven, and the representation is irregular. Due to the difficulty of applying masked autoencoders to an implicit representation, such as NeRF, we opt for extracting an explicit representation that canonicalizes scenes across domains by employing the camera trajectory for sampling. Our goal is made possible by masking random patches from NeRF's radiance and density grid and employing a standard 3D Swin Transformer to reconstruct the masked patches. In doing so, the model can learn the semantic and spatial structure of complete scenes. We pretrain this representation at scale on our proposed curated posed-RGB data, totaling over 1.6 million images. Once pretrained, the encoder is used for effective 3D transfer learning. Our novel self-supervised pretraining for NeRFs, NeRF-MAE, scales remarkably well and improves performance on various challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining, NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF scene understanding baselines on Front3D and ScanNet datasets with an absolute performance improvement of over 20% AP50 and 8% AP25 for 3D object detection.