2024-03-20

Title: Mobile Application for Oral Disease Detection using Federated Learning

• Authors: Shankara Narayanan V, Sneha Varsha M, Syed Ashfaq Ahmed, Guruprakash J
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12044
• Pdf URL: https://arxiv.org/pdf/2403.12044
• Copy Paste: [[2403.12044]] Mobile Application for Oral Disease Detection using Federated Learning(https://arxiv.org/abs/2403.12044)
• Keywords: privacy, federate, segmentation
• Abstract: The mouth, often regarded as a window to the internal state of the body, plays an important role in reflecting one's overall health. Poor oral hygiene has far-reaching consequences, contributing to severe conditions like heart disease, cancer, and diabetes, while inadequate care leads to discomfort, pain, and costly treatments. Federated Learning (FL) for object detection can be utilized for this use case due to the sensitivity of the oral image data of the patients. FL ensures data privacy by storing the images used for object detection on the local device and trains the model on the edge. The updated weights are federated to a central server where all the collected weights are updated via The Federated Averaging algorithm. Finally, we have developed a mobile app named OralH which provides user-friendly solutions, allowing people to conduct self-assessments through mouth scans and providing quick oral health insights. Upon detection of the issues, the application alerts the user about potential oral health concerns or diseases and provides details about dental clinics in the user's locality. Designed as a Progressive Web Application (PWA), the platform ensures ubiquitous access, catering to users across devices for a seamless experience. The application aims to provide state-of-the-art segmentation and detection techniques, leveraging the YOLOv8 object detection model to identify oral hygiene issues and diseases. This study deals with the benefits of leveraging FL in healthcare with promising real-world results.

Title: GPT-4V(ision) Unsuitable for Clinical Care and Education: A Clinician-Evaluated Assessment

• Authors: Senthujan Senkaiahliyan, Augustin Toma, Jun Ma, An-Wen Chan, Andrew Ha, Kevin R. An, Hrishikesh Suresh, Barry Rubin, Bo Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12046
• Pdf URL: https://arxiv.org/pdf/2403.12046
• Copy Paste: [[2403.12046]] GPT-4V(ision) Unsuitable for Clinical Care and Education: A Clinician-Evaluated Assessment(https://arxiv.org/abs/2403.12046)
• Keywords: large language model
• Abstract: OpenAI's large multimodal model, GPT-4V(ision), was recently developed for general image interpretation. However, less is known about its capabilities with medical image interpretation and diagnosis. Board-certified physicians and senior residents assessed GPT-4V's proficiency across a range of medical conditions using imaging modalities such as CT scans, MRIs, ECGs, and clinical photographs. Although GPT-4V is able to identify and explain medical images, its diagnostic accuracy and clinical decision-making abilities are poor, posing risks to patient safety. Despite the potential that large language models may have in enhancing medical education and delivery, the current limitations of GPT-4V in interpreting medical images reinforces the importance of appropriate caution when using it for clinical decision-making.

Title: Alpha-wolves and Alpha-mammals: Exploring Dictionary Attacks on Iris Recognition Systems

• Authors: Sudipta Banerjee, Anubhav Jain, Zehua Jiang, Nasir Memon, Julian Togelius, Arun Ross
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12047
• Pdf URL: https://arxiv.org/pdf/2403.12047
• Copy Paste: [[2403.12047]] Alpha-wolves and Alpha-mammals: Exploring Dictionary Attacks on Iris Recognition Systems(https://arxiv.org/abs/2403.12047)
• Keywords: security, attack, biometric
• Abstract: A dictionary attack in a biometric system entails the use of a small number of strategically generated images or templates to successfully match with a large number of identities, thereby compromising security. We focus on dictionary attacks at the template level, specifically the IrisCodes used in iris recognition systems. We present an hitherto unknown vulnerability wherein we mix IrisCodes using simple bitwise operators to generate alpha-mixtures - alpha-wolves (combining a set of "wolf" samples) and alpha-mammals (combining a set of users selected via search optimization) that increase false matches. We evaluate this vulnerability using the IITD, CASIA-IrisV4-Thousand and Synthetic datasets, and observe that an alpha-wolf (from two wolves) can match upto 71 identities @FMR=0.001%, while an alpha-mammal (from two identities) can match upto 133 other identities @FMR=0.01% on the IITD dataset.

Title: Toward Improving Robustness of Object Detectors Against Domain Shift

• Authors: Le-Anh Tran, Chung Nguyen Tran, Dong-Chul Park, Jordi Carrabina, David Castells-Rufas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12049
• Pdf URL: https://arxiv.org/pdf/2403.12049
• Copy Paste: [[2403.12049]] Toward Improving Robustness of Object Detectors Against Domain Shift(https://arxiv.org/abs/2403.12049)
• Keywords: robust
• Abstract: This paper proposes a data augmentation method for improving the robustness of driving object detectors against domain shift. Domain shift problem arises when there is a significant change between the distribution of the source data domain used in the training phase and that of the target data domain in the deployment phase. Domain shift is known as one of the most popular reasons resulting in the considerable drop in the performance of deep neural network models. In order to address this problem, one effective approach is to increase the diversity of training data. To this end, we propose a data synthesis module that can be utilized to train more robust and effective object detectors. By adopting YOLOv4 as a base object detector, we have witnessed a remarkable improvement in performance on both the source and target domain data. The code of this work is publicly available at https://github.com/tranleanh/haze-synthesis.

Title: A Dataset and Benchmark for Copyright Protection from Text-to-Image Diffusion Models

• Authors: Rui Ma, Qiang Zhou, Bangjun Xiao, Yizhu Jin, Daquan Zhou, Xiuyu Li, Aishani Singh, Yi Qu, Kurt Keutzer, Xiaodong Xie, Jingtong Hu, Zhen Dong, Shanghang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12052
• Pdf URL: https://arxiv.org/pdf/2403.12052
• Copy Paste: [[2403.12052]] A Dataset and Benchmark for Copyright Protection from Text-to-Image Diffusion Models(https://arxiv.org/abs/2403.12052)
• Keywords: protect, diffusion
• Abstract: Copyright is a legal right that grants creators the exclusive authority to reproduce, distribute, and profit from their creative works. However, the recent advancements in text-to-image generation techniques have posed significant challenges to copyright protection, as these methods have facilitated the learning of unauthorized content, artistic creations, and portraits, which are subsequently utilized to generate and disseminate uncontrolled content. Especially, the use of stable diffusion, an emerging model for text-to-image generation, poses an increased risk of unauthorized copyright infringement and distribution. Currently, there is a lack of systematic studies evaluating the potential correlation between content generated by stable diffusion and those under copyright protection. Conducting such studies faces several challenges, including i) the intrinsic ambiguity related to copyright infringement in text-to-image models, ii) the absence of a comprehensive large-scale dataset, and iii) the lack of standardized metrics for defining copyright infringement. This work provides the first large-scale standardized dataset and benchmark on copyright protection. Specifically, we propose a pipeline to coordinate CLIP, ChatGPT, and diffusion models to generate a dataset that contains anchor images, corresponding prompts, and images generated by text-to-image models, reflecting the potential abuses of copyright. Furthermore, we explore a suite of evaluation metrics to judge the effectiveness of copyright protection methods. The proposed dataset, benchmark library, and evaluation metrics will be open-sourced to facilitate future research and application. The website and dataset can be accessed website dataset.

Title: Haze Removal via Regional Saturation-Value Translation and Soft Segmentation

• Authors: Le-Anh Tran, Dong-Chul Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12054
• Pdf URL: https://arxiv.org/pdf/2403.12054
• Copy Paste: [[2403.12054]] Haze Removal via Regional Saturation-Value Translation and Soft Segmentation(https://arxiv.org/abs/2403.12054)
• Keywords: segmentation
• Abstract: This paper proposes a single image dehazing prior, called Regional Saturation-Value Translation (RSVT), to tackle the color distortion problems caused by conventional dehazing approaches in bright regions. The RSVT prior is developed based on two key observations regarding the relationship between hazy and haze-free points in the HSV color space. First, the hue component shows marginal variation between corresponding hazy and haze-free points, consolidating a hypothesis that the pixel value variability induced by haze primarily occurs in the saturation and value spaces. Second, in the 2D saturation-value coordinate system, most lines passing through hazy-clean point pairs are likely to intersect near the atmospheric light coordinates. Accordingly, haze removal for the bright regions can be performed by properly translating saturation-value coordinates. In addition, an effective soft segmentation method based on a morphological min-max channel is introduced. By combining the soft segmentation mask with the RSVT prior, a comprehensive single image dehazing framework is devised. Experimental results on various synthetic and realistic hazy image datasets demonstrate that the proposed scheme successfully addresses color distortion issues and restores visually appealing images. The code of this work is available at https://github.com/tranleanh/rsvt.

Title: Deep learning based detection of collateral circulation in coronary angiographies

• Authors: Cosmin-Andrei Hatfaludi, Daniel Bunescu, Costin Florian Ciusdel, Alex Serban, Karl Bose, Marc Oppel, Stephanie Schroder, Christopher Seehase, Harald F. Langer, Jeanette Erdmann, Henry Nording, Lucian Mihai Itu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12055
• Pdf URL: https://arxiv.org/pdf/2403.12055
• Copy Paste: [[2403.12055]] Deep learning based detection of collateral circulation in coronary angiographies(https://arxiv.org/abs/2403.12055)
• Keywords: segmentation
• Abstract: Coronary artery disease (CAD) is the dominant cause of death and hospitalization across the globe. Atherosclerosis, an inflammatory condition that gradually narrows arteries and has potentially fatal effects, is the most frequent cause of CAD. Nonetheless, the circulation regularly adapts in the presence of atherosclerosis, through the formation of collateral arteries, resulting in significant long-term health benefits. Therefore, timely detection of coronary collateral circulation (CCC) is crucial for CAD personalized medicine. We propose a novel deep learning based method to detect CCC in angiographic images. Our method relies on a convolutional backbone to extract spatial features from each frame of an angiography sequence. The features are then concatenated, and subsequently processed by another convolutional layer that processes embeddings temporally. Due to scarcity of data, we also experiment with pretraining the backbone on coronary artery segmentation, which improves the results consistently. Moreover, we experiment with few-shot learning to further improve performance, given our low data regime. We present our results together with subgroup analyses based on Rentrop grading, collateral flow, and collateral grading, which provide valuable insights into model performance. Overall, the proposed method shows promising results in detecting CCC, and can be further extended to perform landmark based CCC detection and CCC quantification.

Title: Enhancing Digital Hologram Reconstruction Using Reverse-Attention Loss for Untrained Physics-Driven Deep Learning Models with Uncertain Distance

• Authors: Xiwen Chen, Hao Wang, Zhao Zhang, Zhenmin Li, Huayu Li, Tong Ye, Abolfazl Razi
• Subjects: cs.CV, physics.optics
• Abstract URL: https://arxiv.org/abs/2403.12056
• Pdf URL: https://arxiv.org/pdf/2403.12056
• Copy Paste: [[2403.12056]] Enhancing Digital Hologram Reconstruction Using Reverse-Attention Loss for Untrained Physics-Driven Deep Learning Models with Uncertain Distance(https://arxiv.org/abs/2403.12056)
• Keywords: interpretability
• Abstract: Untrained Physics-based Deep Learning (DL) methods for digital holography have gained significant attention due to their benefits, such as not requiring an annotated training dataset, and providing interpretability since utilizing the governing laws of hologram formation. However, they are sensitive to the hard-to-obtain precise object distance from the imaging plane, posing the $\textit{Autofocusing}$ challenge. Conventional solutions involve reconstructing image stacks for different potential distances and applying focus metrics to select the best results, which apparently is computationally inefficient. In contrast, recently developed DL-based methods treat it as a supervised task, which again needs annotated data and lacks generalizability. To address this issue, we propose $\textit{reverse-attention loss}$, a weighted sum of losses for all possible candidates with learnable weights. This is a pioneering approach to addressing the Autofocusing challenge in untrained deep-learning methods. Both theoretical analysis and experiments demonstrate its superiority in efficiency and accuracy. Interestingly, our method presents a significant reconstruction performance over rival methods (i.e. alternating descent-like optimization, non-weighted loss integration, and random distance assignment) and even is almost equal to that achieved with a precisely known object distance. For example, the difference is less than 1dB in PSNR and 0.002 in SSIM for the target sample in our experiment.

Title: Discriminative Consensus Mining with A Thousand Groups for More Accurate Co-Salient Object Detection

• Authors: Peng Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12057
• Pdf URL: https://arxiv.org/pdf/2403.12057
• Copy Paste: [[2403.12057]] Discriminative Consensus Mining with A Thousand Groups for More Accurate Co-Salient Object Detection(https://arxiv.org/abs/2403.12057)
• Keywords: segmentation
• Abstract: Co-Salient Object Detection (CoSOD) is a rapidly growing task, extended from Salient Object Detection (SOD) and Common Object Segmentation (Co-Segmentation). It is aimed at detecting the co-occurring salient object in the given image group. Many effective approaches have been proposed on the basis of existing datasets. However, there is still no standard and efficient training set in CoSOD, which makes it chaotic to choose training sets in the recently proposed CoSOD methods. First, the drawbacks of existing training sets in CoSOD are analyzed in a comprehensive way, and potential improvements are provided to solve existing problems to some extent. In particular, in this thesis, a new CoSOD training set is introduced, named Co-Saliency of ImageNet (CoSINe) dataset. The proposed CoSINe is the largest number of groups among all existing CoSOD datasets. The images obtained here span a wide variety in terms of categories, object sizes, etc. In experiments, models trained on CoSINe can achieve significantly better performance with fewer images compared to all existing datasets. Second, to make the most of the proposed CoSINe, a novel CoSOD approach named Hierarchical Instance-aware COnsensus MinEr (HICOME) is proposed, which efficiently mines the consensus feature from different feature levels and discriminates objects of different classes in an object-aware contrastive way. As extensive experiments show, the proposed HICOME achieves SoTA performance on all the existing CoSOD test sets. Several useful training tricks suitable for training CoSOD models are also provided. Third, practical applications are given using the CoSOD technique to show the effectiveness. Finally, the remaining challenges and potential improvements of CoSOD are discussed to inspire related work in the future. The source code, the dataset, and the online demo will be publicly available at github.com/ZhengPeng7/CoSINe.

Title: Consistency Models Improve Diffusion Inverse Solvers

• Authors: Tongda Xu, Ziran Zhu, Dailan He, Yuanyuan Wang, Ming Sun, Ning Li, Hongwei Qin, Yan Wang, Jingjing Liu, Ya-Qin Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12063
• Pdf URL: https://arxiv.org/pdf/2403.12063
• Copy Paste: [[2403.12063]] Consistency Models Improve Diffusion Inverse Solvers(https://arxiv.org/abs/2403.12063)
• Keywords: diffusion, segmentation
• Abstract: Diffusion inverse solvers (DIS) aim to find an image $x$ that lives on the diffusion prior while satisfying the constraint $f(x) = y$, given an operator $f(.)$ and measurement $y$. Most non-linear DIS use posterior mean $\hat{x}_{0|t}=\mathbb{E}[x_0|x_t]$ to evaluate $f(.)$ and minimize the distance $||f(\hat{x}_{0|t})-y||^2$. Previous works show that posterior mean-based distance is biased; instead, posterior sample $x_{0|t}\sim p_{\theta}(x_0|x_t)$ promises a better candidate. In this paper, we first clarify when is posterior sample better: $1)$ When $f(.)$ is linear, the distance with posterior mean is as good as single posterior sample, thus preferable as it does not require Monte Carlo; $2)$ When $f(.)$ is non-linear, the distance using posterior sample is better. As previous approximations to posterior sample do not look like a real image, we propose to use consistency model (CM) as a high quality approximation. In addition, we propose a new family of DIS using pure CM. Empirically, we show that replacing posterior mean by CM improves DIS performance on non-linear $f(.)$ (e.g. semantic segmentation, image captioning). Further, our pure CM inversion works well for both linear and non-linear $f(.)$.

Title: Adapting SAM for Volumetric X-Ray Data-sets of Arbitrary Sizes

• Authors: Roland Gruber, Steffen Rüger, Thomas Wittenberg
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12066
• Pdf URL: https://arxiv.org/pdf/2403.12066
• Copy Paste: [[2403.12066]] Adapting SAM for Volumetric X-Ray Data-sets of Arbitrary Sizes(https://arxiv.org/abs/2403.12066)
• Keywords: segmentation
• Abstract: Objective: We propose a new approach for volumetric instance segmentation in X-ray Computed Tomography (CT) data for Non-Destructive Testing (NDT) by combining the Segment Anything Model (SAM) with tile-based Flood Filling Networks (FFN). Our work evaluates the performance of SAM on volumetric NDT data-sets and demonstrates its effectiveness to segment instances in challenging imaging scenarios. Methods: We implemented and evaluated techniques to extend the image-based SAM algorithm fo the use with volumetric data-sets, enabling the segmentation of three-dimensional objects using FFN's spatially adaptability. The tile-based approach for SAM leverages FFN's capabilities to segment objects of any size. We also explore the use of dense prompts to guide SAM in combining segmented tiles for improved segmentation accuracy. Results: Our research indicates the potential of combining SAM with FFN for volumetric instance segmentation tasks, particularly in NDT scenarios and segmenting large entities and objects. Conclusion: While acknowledging remaining limitations, our study provides insights and establishes a foundation for advancements in instance segmentation in NDT scenarios.

Title: Evaluating Robustness of Generative Search Engine on Adversarial Factual Questions

• Authors: Xuming Hu, Xiaochuan Li, Junzhe Chen, Yinghui Li, Yangning Li, Xiaoguang Li, Yasheng Wang, Qun Liu, Lijie Wen, Philip S. Yu, Zhijiang Guo
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.12077
• Pdf URL: https://arxiv.org/pdf/2403.12077
• Copy Paste: [[2403.12077]] Evaluating Robustness of Generative Search Engine on Adversarial Factual Questions(https://arxiv.org/abs/2403.12077)
• Keywords: security, robust, generative, large language model
• Abstract: Generative search engines have the potential to transform how people seek information online, but generated responses from existing large language models (LLMs)-backed generative search engines may not always be accurate. Nonetheless, retrieval-augmented generation exacerbates safety concerns, since adversaries may successfully evade the entire system by subtly manipulating the most vulnerable part of a claim. To this end, we propose evaluating the robustness of generative search engines in the realistic and high-risk setting, where adversaries have only black-box system access and seek to deceive the model into returning incorrect responses. Through a comprehensive human evaluation of various generative search engines, such as Bing Chat, PerplexityAI, and YouChat across diverse queries, we demonstrate the effectiveness of adversarial factual questions in inducing incorrect responses. Moreover, retrieval-augmented generation exhibits a higher susceptibility to factual errors compared to LLMs without retrieval. These findings highlight the potential security risks of these systems and emphasize the need for rigorous evaluation before deployment.

Title: Deep Generative Design for Mass Production

• Authors: Jihoon Kim, Yongmin Kwon, Namwoo Kang
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.12098
• Pdf URL: https://arxiv.org/pdf/2403.12098
• Copy Paste: [[2403.12098]] Deep Generative Design for Mass Production(https://arxiv.org/abs/2403.12098)
• Keywords: generative
• Abstract: Generative Design (GD) has evolved as a transformative design approach, employing advanced algorithms and AI to create diverse and innovative solutions beyond traditional constraints. Despite its success, GD faces significant challenges regarding the manufacturability of complex designs, often necessitating extensive manual modifications due to limitations in standard manufacturing processes and the reliance on additive manufacturing, which is not ideal for mass production. Our research introduces an innovative framework addressing these manufacturability concerns by integrating constraints pertinent to die casting and injection molding into GD, through the utilization of 2D depth images. This method simplifies intricate 3D geometries into manufacturable profiles, removing unfeasible features such as non-manufacturable overhangs and allowing for the direct consideration of essential manufacturing aspects like thickness and rib design. Consequently, designs previously unsuitable for mass production are transformed into viable solutions. We further enhance this approach by adopting an advanced 2D generative model, which offer a more efficient alternative to traditional 3D shape generation methods. Our results substantiate the efficacy of this framework, demonstrating the production of innovative, and, importantly, manufacturable designs. This shift towards integrating practical manufacturing considerations into GD represents a pivotal advancement, transitioning from purely inspirational concepts to actionable, production-ready solutions. Our findings underscore usefulness and potential of GD for broader industry adoption, marking a significant step forward in aligning GD with the demands of manufacturing challenges.

Title: GCAM: Gaussian and causal-attention model of food fine-grained recognition

• Authors: Guohang Zhuang, Yue Hu, Tianxing Yan, JiaZhan Gao
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12109
• Pdf URL: https://arxiv.org/pdf/2403.12109
• Copy Paste: [[2403.12109]] GCAM: Gaussian and causal-attention model of food fine-grained recognition(https://arxiv.org/abs/2403.12109)
• Keywords: extraction
• Abstract: Currently, most food recognition relies on deep learning for category classification. However, these approaches struggle to effectively distinguish between visually similar food samples, highlighting the pressing need to address fine-grained issues in food recognition. To mitigate these challenges, we propose the adoption of a Gaussian and causal-attention model for fine-grained object recognition.In particular, we train to obtain Gaussian features over target regions, followed by the extraction of fine-grained features from the objects, thereby enhancing the feature mapping capabilities of the target regions. To counteract data drift resulting from uneven data distributions, we employ a counterfactual reasoning approach. By using counterfactual interventions, we analyze the impact of the learned image attention mechanism on network predictions, enabling the network to acquire more useful attention weights for fine-grained image recognition. Finally, we design a learnable loss strategy to balance training stability across various modules, ultimately improving the accuracy of the final target recognition. We validate our approach on four relevant datasets, demonstrating its excellent performance across these four datasets.We experimentally show that GCAM surpasses state-of-the-art methods on the ETH-FOOD101, UECFOOD256, and Vireo-FOOD172 datasets. Furthermore, our approach also achieves state-of-the-art performance on the CUB-200 dataset.

Title: Graph Neural Networks for Learning Equivariant Representations of Neural Networks

• Authors: Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J. Burghouts, Efstratios Gavves, Cees G. M. Snoek, David W. Zhang
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12143
• Pdf URL: https://arxiv.org/pdf/2403.12143
• Copy Paste: [[2403.12143]] Graph Neural Networks for Learning Equivariant Representations of Neural Networks(https://arxiv.org/abs/2403.12143)
• Keywords: transformer
• Abstract: Neural networks that process the parameters of other neural networks find applications in domains as diverse as classifying implicit neural representations, generating neural network weights, and predicting generalization errors. However, existing approaches either overlook the inherent permutation symmetry in the neural network or rely on intricate weight-sharing patterns to achieve equivariance, while ignoring the impact of the network architecture itself. In this work, we propose to represent neural networks as computational graphs of parameters, which allows us to harness powerful graph neural networks and transformers that preserve permutation symmetry. Consequently, our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations, predicting generalization performance, and learning to optimize, while consistently outperforming state-of-the-art methods. The source code is open-sourced at https://github.com/mkofinas/neural-graphs.

Title: Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasets

• Authors: Ashwin Daswani, Rohan Sawant, Najoung Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12145
• Pdf URL: https://arxiv.org/pdf/2403.12145
• Copy Paste: [[2403.12145]] Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasets(https://arxiv.org/abs/2403.12145)
• Keywords: robust, generative, large language model
• Abstract: Sensitivity to false assumptions (or false premises) in information-seeking questions is critical for robust question-answering (QA) systems. Recent work has shown that false assumptions in naturally occurring questions pose challenges to current models, with low performance on both generative QA and simple detection tasks (Kim et al. 2023). However, the focus of existing work on naturally occurring questions leads to a gap in the analysis of model behavior on the long tail of the distribution of possible questions. To this end, we introduce Syn-(QA)$^2$, a set of two synthetically generated QA datasets: one generated using perturbed relations from Wikidata, and the other by perturbing HotpotQA (Yang et al. 2018). Our findings from evaluating a range of large language models are threefold: (1) false assumptions in QA are challenging, echoing the findings of prior work, (2) the binary detection task is challenging even compared to the difficulty of generative QA itself, possibly due to the linguistic structure of the problem, and (3) the detection task is more challenging with long-tail questions compared to naturally occurring questions, highlighting the utility of our synthetic datasets and generation method.

Title: Development of Automated Neural Network Prediction for Echocardiographic Left ventricular Ejection Fraction

• Authors: Yuting Zhang, Boyang Liu, Karina V. Bunting, David Brind, Alexander Thorley, Andreas Karwath, Wenqi Lu, Diwei Zhou, Xiaoxia Wang, Alastair R. Mobley, Otilia Tica, Georgios Gkoutos, Dipak Kotecha, Jinming Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12152
• Pdf URL: https://arxiv.org/pdf/2403.12152
• Copy Paste: [[2403.12152]] Development of Automated Neural Network Prediction for Echocardiographic Left ventricular Ejection Fraction(https://arxiv.org/abs/2403.12152)
• Keywords: segmentation
• Abstract: The echocardiographic measurement of left ventricular ejection fraction (LVEF) is fundamental to the diagnosis and classification of patients with heart failure (HF). In order to quantify LVEF automatically and accurately, this paper proposes a new pipeline method based on deep neural networks and ensemble learning. Within the pipeline, an Atrous Convolutional Neural Network (ACNN) was first trained to segment the left ventricle (LV), before employing the area-length formulation based on the ellipsoid single-plane model to calculate LVEF values. This formulation required inputs of LV area, derived from segmentation using an improved Jeffrey's method, as well as LV length, derived from a novel ensemble learning model. To further improve the pipeline's accuracy, an automated peak detection algorithm was used to identify end-diastolic and end-systolic frames, avoiding issues with human error. Subsequently, single-beat LVEF values were averaged across all cardiac cycles to obtain the final LVEF. This method was developed and internally validated in an open-source dataset containing 10,030 echocardiograms. The Pearson's correlation coefficient was 0.83 for LVEF prediction compared to expert human analysis (p<0.001), with a subsequent area under the receiver operator curve (AUROC) of 0.98 (95% confidence interval 0.97 to 0.99) for categorisation of HF with reduced ejection (HFrEF; LVEF<40%). In an external dataset with 200 echocardiograms, this method achieved an AUC of 0.90 (95% confidence interval 0.88 to 0.91) for HFrEF assessment. This study demonstrates that an automated neural network-based calculation of LVEF is comparable to expert clinicians performing time-consuming, frame-by-frame manual evaluation of cardiac systolic function.

Title: The Power of Few: Accelerating and Enhancing Data Reweighting with Coreset Selection

• Authors: Mohammad Jafari, Yimeng Zhang, Yihua Zhang, Sijia Liu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12166
• Pdf URL: https://arxiv.org/pdf/2403.12166
• Copy Paste: [[2403.12166]] The Power of Few: Accelerating and Enhancing Data Reweighting with Coreset Selection(https://arxiv.org/abs/2403.12166)
• Keywords: robust
• Abstract: As machine learning tasks continue to evolve, the trend has been to gather larger datasets and train increasingly larger models. While this has led to advancements in accuracy, it has also escalated computational costs to unsustainable levels. Addressing this, our work aims to strike a delicate balance between computational efficiency and model accuracy, a persisting challenge in the field. We introduce a novel method that employs core subset selection for reweighting, effectively optimizing both computational time and model performance. By focusing on a strategically selected coreset, our approach offers a robust representation, as it efficiently minimizes the influence of outliers. The re-calibrated weights are then mapped back to and propagated across the entire dataset. Our experimental results substantiate the effectiveness of this approach, underscoring its potential as a scalable and precise solution for model training.

Title: EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models

• Authors: Weikang Zhou, Xiao Wang, Limao Xiong, Han Xia, Yingshuang Gu, Mingxu Chai, Fukang Zhu, Caishuang Huang, Shihan Dou, Zhiheng Xi, Rui Zheng, Songyang Gao, Yicheng Zou, Hang Yan, Yifan Le, Ruohui Wang, Lijun Li, Jing Shao, Tao Gui, Qi Zhang, Xuanjing Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12171
• Pdf URL: https://arxiv.org/pdf/2403.12171
• Copy Paste: [[2403.12171]] EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models(https://arxiv.org/abs/2403.12171)
• Keywords: security, attack, large language model
• Abstract: Jailbreak attacks are crucial for identifying and mitigating the security vulnerabilities of Large Language Models (LLMs). They are designed to bypass safeguards and elicit prohibited outputs. However, due to significant differences among various jailbreak methods, there is no standard implementation framework available for the community, which limits comprehensive security evaluations. This paper introduces EasyJailbreak, a unified framework simplifying the construction and evaluation of jailbreak attacks against LLMs. It builds jailbreak attacks using four components: Selector, Mutator, Constraint, and Evaluator. This modular framework enables researchers to easily construct attacks from combinations of novel and existing components. So far, EasyJailbreak supports 11 distinct jailbreak methods and facilitates the security validation of a broad spectrum of LLMs. Our validation across 10 distinct LLMs reveals a significant vulnerability, with an average breach probability of 60% under various jailbreaking attacks. Notably, even advanced models like GPT-3.5-Turbo and GPT-4 exhibit average Attack Success Rates (ASR) of 57% and 33%, respectively. We have released a wealth of resources for researchers, including a web platform, PyPI published package, screencast video, and experimental outputs.

Title: Graph-Jigsaw Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection

• Authors: Ali Karami, Thi Kieu Khanh Ho, Narges Armanfard
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12172
• Pdf URL: https://arxiv.org/pdf/2403.12172
• Copy Paste: [[2403.12172]] Graph-Jigsaw Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection(https://arxiv.org/abs/2403.12172)
• Keywords: diffusion
• Abstract: Skeleton-based video anomaly detection (SVAD) is a crucial task in computer vision. Accurately identifying abnormal patterns or events enables operators to promptly detect suspicious activities, thereby enhancing safety. Achieving this demands a comprehensive understanding of human motions, both at body and region levels, while also accounting for the wide variations of performing a single action. However, existing studies fail to simultaneously address these crucial properties. This paper introduces a novel, practical and lightweight framework, namely Graph-Jigsaw Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection (GiCiSAD) to overcome the challenges associated with SVAD. GiCiSAD consists of three novel modules: the Graph Attention-based Forecasting module to capture the spatio-temporal dependencies inherent in the data, the Graph-level Jigsaw Puzzle Maker module to distinguish subtle region-level discrepancies between normal and abnormal motions, and the Graph-based Conditional Diffusion model to generate a wide spectrum of human motions. Extensive experiments on four widely used skeleton-based video datasets show that GiCiSAD outperforms existing methods with significantly fewer training parameters, establishing it as the new state-of-the-art.

Title: TnT-LLM: Text Mining at Scale with Large Language Models

• Authors: Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, Chirag Shah, Ryen W White, Longqi Yang, Reid Andersen, Georg Buscher, Dhruv Joshi, Nagu Rangan
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.12173
• Pdf URL: https://arxiv.org/pdf/2403.12173
• Copy Paste: [[2403.12173]] TnT-LLM: Text Mining at Scale with Large Language Models(https://arxiv.org/abs/2403.12173)
• Keywords: large language model
• Abstract: Transforming unstructured text into structured and meaningful forms, organized by useful category labels, is a fundamental step in text mining for downstream analysis and application. However, most existing methods for producing label taxonomies and building text-based label classifiers still rely heavily on domain expertise and manual curation, making the process expensive and time-consuming. This is particularly challenging when the label space is under-specified and large-scale data annotations are unavailable. In this paper, we address these challenges with Large Language Models (LLMs), whose prompt-based interface facilitates the induction and use of large-scale pseudo labels. We propose TnT-LLM, a two-phase framework that employs LLMs to automate the process of end-to-end label generation and assignment with minimal human effort for any given use-case. In the first phase, we introduce a zero-shot, multi-stage reasoning approach which enables LLMs to produce and refine a label taxonomy iteratively. In the second phase, LLMs are used as data labelers that yield training samples so that lightweight supervised classifiers can be reliably built, deployed, and served at scale. We apply TnT-LLM to the analysis of user intent and conversational domain for Bing Copilot (formerly Bing Chat), an open-domain chat-based search engine. Extensive experiments using both human and automatic evaluation metrics demonstrate that TnT-LLM generates more accurate and relevant label taxonomies when compared against state-of-the-art baselines, and achieves a favorable balance between accuracy and efficiency for classification at scale. We also share our practical experiences and insights on the challenges and opportunities of using LLMs for large-scale text mining in real-world applications.

Title: Shifting the Lens: Detecting Malware in npm Ecosystem with Large Language Models

• Authors: Nusrat Zahan, Philipp Burckhardt, Mikola Lysenko, Feross Aboukhadijeh, Laurie Williams
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12196
• Pdf URL: https://arxiv.org/pdf/2403.12196
• Copy Paste: [[2403.12196]] Shifting the Lens: Detecting Malware in npm Ecosystem with Large Language Models(https://arxiv.org/abs/2403.12196)
• Keywords: security, attack, large language model
• Abstract: The Gartner 2022 report predicts that 45% of organizations worldwide will encounter software supply chain attacks by 2025, highlighting the urgency to improve software supply chain security for community and national interests. Current malware detection techniques aid in the manual review process by filtering benign and malware packages, yet such techniques have high false-positive rates and limited automation support. Therefore, malware detection techniques could benefit from advanced, more automated approaches for accurate and minimally false-positive results. The goal of this study is to assist security analysts in identifying malicious packages through the empirical study of large language models (LLMs) to detect potential malware in the npm ecosystem. We present SocketAI Scanner, a multi-stage decision-maker malware detection workflow using iterative self-refinement and zero-shot-role-play-Chain of Thought (CoT) prompting techniques for ChatGPT. We studied 5,115 npm packages (of which 2,180 are malicious) and performed a baseline comparison of the GPT-3 and GPT-4 models with a static analysis tool. Our findings showed promising results for GPT models with low misclassification alert rates. Our baseline comparison demonstrates a notable improvement over static analysis in precision scores above 25% and F1 scores above 15%. We attained precision and F1 scores of 91% and 94%, respectively, for the GPT-3 model. Overall, GPT-4 demonstrates superior performance in precision (99%) and F1 (97%) scores, while GPT-3 presents a cost-effective balance between performance and expenditure.

Title: E2F-Net: Eyes-to-Face Inpainting via StyleGAN Latent Space

• Authors: Ahmad Hassanpour, Fatemeh Jamalbafrani, Bian Yang, Kiran Raja, Raymond Veldhuis, Julian Fierrez
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12197
• Pdf URL: https://arxiv.org/pdf/2403.12197
• Copy Paste: [[2403.12197]] E2F-Net: Eyes-to-Face Inpainting via StyleGAN Latent Space(https://arxiv.org/abs/2403.12197)
• Keywords: generative
• Abstract: Face inpainting, the technique of restoring missing or damaged regions in facial images, is pivotal for applications like face recognition in occluded scenarios and image analysis with poor-quality captures. This process not only needs to produce realistic visuals but also preserve individual identity characteristics. The aim of this paper is to inpaint a face given periocular region (eyes-to-face) through a proposed new Generative Adversarial Network (GAN)-based model called Eyes-to-Face Network (E2F-Net). The proposed approach extracts identity and non-identity features from the periocular region using two dedicated encoders have been used. The extracted features are then mapped to the latent space of a pre-trained StyleGAN generator to benefit from its state-of-the-art performance and its rich, diverse and expressive latent space without any additional training. We further improve the StyleGAN output to find the optimal code in the latent space using a new optimization for GAN inversion technique. Our E2F-Net requires a minimum training process reducing the computational complexity as a secondary benefit. Through extensive experiments, we show that our method successfully reconstructs the whole face with high quality, surpassing current techniques, despite significantly less training and supervision efforts. We have generated seven eyes-to-face datasets based on well-known public face datasets for training and verifying our proposed methods. The code and datasets are publicly available.

Title: DeCoTR: Enhancing Depth Completion with 2D and 3D Attentions

• Authors: Yunxiao Shi, Manish Kumar Singh, Hong Cai, Fatih Porikli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12202
• Pdf URL: https://arxiv.org/pdf/2403.12202
• Copy Paste: [[2403.12202]] DeCoTR: Enhancing Depth Completion with 2D and 3D Attentions(https://arxiv.org/abs/2403.12202)
• Keywords: transformer
• Abstract: In this paper, we introduce a novel approach that harnesses both 2D and 3D attentions to enable highly accurate depth completion without requiring iterative spatial propagations. Specifically, we first enhance a baseline convolutional depth completion model by applying attention to 2D features in the bottleneck and skip connections. This effectively improves the performance of this simple network and sets it on par with the latest, complex transformer-based models. Leveraging the initial depths and features from this network, we uplift the 2D features to form a 3D point cloud and construct a 3D point transformer to process it, allowing the model to explicitly learn and exploit 3D geometric features. In addition, we propose normalization techniques to process the point cloud, which improves learning and leads to better accuracy than directly using point transformers off the shelf. Furthermore, we incorporate global attention on downsampled point cloud features, which enables long-range context while still being computationally feasible. We evaluate our method, DeCoTR, on established depth completion benchmarks, including NYU Depth V2 and KITTI, showcasing that it sets new state-of-the-art performance. We further conduct zero-shot evaluations on ScanNet and DDAD benchmarks and demonstrate that DeCoTR has superior generalizability compared to existing approaches.

Title: Evaluating Named Entity Recognition: Comparative Analysis of Mono- and Multilingual Transformer Models on Brazilian Corporate Earnings Call Transcriptions

• Authors: Ramon Abilio, Guilherme Palermo Coelho, Ana Estela Antunes da Silva
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12212
• Pdf URL: https://arxiv.org/pdf/2403.12212
• Copy Paste: [[2403.12212]] Evaluating Named Entity Recognition: Comparative Analysis of Mono- and Multilingual Transformer Models on Brazilian Corporate Earnings Call Transcriptions(https://arxiv.org/abs/2403.12212)
• Keywords: transformer
• Abstract: Named Entity Recognition (NER) is a Natural Language Processing technique for extracting information from textual documents. However, much of the existing research on NER has been centered around English-language documents, leaving a gap in the availability of datasets tailored to the financial domain in Portuguese. This study addresses the need for NER within the financial domain, focusing on Portuguese-language texts extracted from earnings call transcriptions of Brazilian banks. By curating a comprehensive dataset comprising 384 transcriptions and leveraging weak supervision techniques for annotation, we evaluate the performance of monolingual models trained on Portuguese (BERTimbau and PTT5) and multilingual models (mBERT and mT5). Notably, we introduce a novel approach that reframes the token classification task as a text generation problem, enabling fine-tuning and evaluation of T5 models. Following the fine-tuning of the models, we conduct an evaluation on the test dataset, employing performance and error metrics. Our findings reveal that BERT-based models consistently outperform T5-based models. Furthermore, while the multilingual models exhibit comparable macro F1-scores, BERTimbau demonstrates superior performance over PTT5. A manual analysis of sentences generated by PTT5 and mT5 unveils a degree of similarity ranging from 0.89 to 1.0, between the original and generated sentences. However, critical errors emerge as both models exhibit discrepancies, such as alterations to monetary and percentage values, underscoring the importance of accuracy and consistency in the financial domain. Despite these challenges, PTT5 and mT5 achieve impressive macro F1-scores of 98.52% and 98.85%, respectively, with our proposed approach. Furthermore, our study sheds light on notable disparities in memory and time consumption for inference across the models.

Title: Fusion Transformer with Object Mask Guidance for Image Forgery Analysis

• Authors: Dimitrios Karageorgiou, Giorgos Kordopatis-Zilos, Symeon Papadopoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12229
• Pdf URL: https://arxiv.org/pdf/2403.12229
• Copy Paste: [[2403.12229]] Fusion Transformer with Object Mask Guidance for Image Forgery Analysis(https://arxiv.org/abs/2403.12229)
• Keywords: attack, robust, transformer
• Abstract: In this work, we introduce OMG-Fuser, a fusion transformer-based network designed to extract information from various forensic signals to enable robust image forgery detection and localization. Our approach can operate with an arbitrary number of forensic signals and leverages object information for their analysis -- unlike previous methods that rely on fusion schemes with few signals and often disregard image semantics. To this end, we design a forensic signal stream composed of a transformer guided by an object attention mechanism, associating patches that depict the same objects. In that way, we incorporate object-level information from the image. Each forensic signal is processed by a different stream that adapts to its peculiarities. Subsequently, a token fusion transformer efficiently aggregates the outputs of an arbitrary number of network streams and generates a fused representation for each image patch. These representations are finally processed by a long-range dependencies transformer that captures the intrinsic relations between the image patches. We assess two fusion variants on top of the proposed approach: (i) score-level fusion that fuses the outputs of multiple image forensics algorithms and (ii) feature-level fusion that fuses low-level forensic traces directly. Both variants exceed state-of-the-art performance on seven datasets for image forgery detection and localization, with a relative average improvement of 12.1% and 20.4% in terms of F1. Our network demonstrates robustness against traditional and novel forgery attacks and can be expanded with new signals without training from scratch.

Title: Efficient Transformer-based Hyper-parameter Optimization for Resource-constrained IoT Environments

• Authors: Ibrahim Shaer, Soodeh Nikan, Abdallah Shami
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12237
• Pdf URL: https://arxiv.org/pdf/2403.12237
• Copy Paste: [[2403.12237]] Efficient Transformer-based Hyper-parameter Optimization for Resource-constrained IoT Environments(https://arxiv.org/abs/2403.12237)
• Keywords: transformer
• Abstract: The hyper-parameter optimization (HPO) process is imperative for finding the best-performing Convolutional Neural Networks (CNNs). The automation process of HPO is characterized by its sizable computational footprint and its lack of transparency; both important factors in a resource-constrained Internet of Things (IoT) environment. In this paper, we address these problems by proposing a novel approach that combines transformer architecture and actor-critic Reinforcement Learning (RL) model, TRL-HPO, equipped with multi-headed attention that enables parallelization and progressive generation of layers. These assumptions are founded empirically by evaluating TRL-HPO on the MNIST dataset and comparing it with state-of-the-art approaches that build CNN models from scratch. The results show that TRL-HPO outperforms the classification results of these approaches by 6.8% within the same time frame, demonstrating the efficiency of TRL-HPO for the HPO process. The analysis of the results identifies the main culprit for performance degradation attributed to stacking fully connected layers. This paper identifies new avenues for improving RL-based HPO processes in resource-constrained environments.

Title: Large language models in 6G security: challenges and opportunities

• Authors: Tri Nguyen, Huong Nguyen, Ahmad Ijaz, Saeid Sheikhi, Athanasios V. Vasilakos, Panos Kostakos
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.12239
• Pdf URL: https://arxiv.org/pdf/2403.12239
• Copy Paste: [[2403.12239]] Large language models in 6G security: challenges and opportunities(https://arxiv.org/abs/2403.12239)
• Keywords: security, defense, attack, generative, large language model
• Abstract: The rapid integration of Generative AI (GenAI) and Large Language Models (LLMs) in sectors such as education and healthcare have marked a significant advancement in technology. However, this growth has also led to a largely unexplored aspect: their security vulnerabilities. As the ecosystem that includes both offline and online models, various tools, browser plugins, and third-party applications continues to expand, it significantly widens the attack surface, thereby escalating the potential for security breaches. These expansions in the 6G and beyond landscape provide new avenues for adversaries to manipulate LLMs for malicious purposes. We focus on the security aspects of LLMs from the viewpoint of potential adversaries. We aim to dissect their objectives and methodologies, providing an in-depth analysis of known security weaknesses. This will include the development of a comprehensive threat taxonomy, categorizing various adversary behaviors. Also, our research will concentrate on how LLMs can be integrated into cybersecurity efforts by defense teams, also known as blue teams. We will explore the potential synergy between LLMs and blockchain technology, and how this combination could lead to the development of next-generation, fully autonomous security solutions. This approach aims to establish a unified cybersecurity strategy across the entire computing continuum, enhancing overall digital security infrastructure.

Title: Reference-based Metrics Disprove Themselves in Question Generation

• Authors: Bang Nguyen, Mengxia Yu, Yun Huang, Meng Jiang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12242
• Pdf URL: https://arxiv.org/pdf/2403.12242
• Copy Paste: [[2403.12242]] Reference-based Metrics Disprove Themselves in Question Generation(https://arxiv.org/abs/2403.12242)
• Keywords: large language model
• Abstract: Reference-based metrics such as BLEU and BERTScore are widely used to evaluate question generation (QG). In this study, on QG benchmarks such as SQuAD and HotpotQA, we find that using human-written references cannot guarantee the effectiveness of the reference-based metrics. Most QG benchmarks have only one reference; we replicated the annotation process and collect another reference. A good metric was expected to grade a human-validated question no worse than generated questions. However, the results of reference-based metrics on our newly collected reference disproved the metrics themselves. We propose a reference-free metric consisted of multi-dimensional criteria such as naturalness, answerability, and complexity, utilizing large language models. These criteria are not constrained to the syntactic or semantic of a single reference question, and the metric does not require a diverse set of references. Experiments reveal that our metric accurately distinguishes between high-quality questions and flawed ones, and achieves state-of-the-art alignment with human judgment.

Title: Zero-Shot Multi-task Hallucination Detection

• Authors: Patanjali Bhamidipati, Advaith Malladi, Manish Shrivastava, Radhika Mamidi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12244
• Pdf URL: https://arxiv.org/pdf/2403.12244
• Copy Paste: [[2403.12244]] Zero-Shot Multi-task Hallucination Detection(https://arxiv.org/abs/2403.12244)
• Keywords: robust, large language model
• Abstract: In recent studies, the extensive utilization of large language models has underscored the importance of robust evaluation methodologies for assessing text generation quality and relevance to specific tasks. This has revealed a prevalent issue known as hallucination, an emergent condition in the model where generated text lacks faithfulness to the source and deviates from the evaluation criteria. In this study, we formally define hallucination and propose a framework for its quantitative detection in a zero-shot setting, leveraging our definition and the assumption that model outputs entail task and sample specific inputs. In detecting hallucinations, our solution achieves an accuracy of 0.78 in a model-aware setting and 0.61 in a model-agnostic setting. Notably, our solution maintains computational efficiency, requiring far less computational resources than other SOTA approaches, aligning with the trend towards lightweight and compressed models.

Title: Parasitic Circus:On the Feasibility of Golden Free PCB Verification

• Authors: Maryam Saadat Safa, Patrick Schaumont, Shahin Tajik
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.12252
• Pdf URL: https://arxiv.org/pdf/2403.12252
• Copy Paste: [[2403.12252]] Parasitic Circus:On the Feasibility of Golden Free PCB Verification(https://arxiv.org/abs/2403.12252)
• Keywords: attack, extraction
• Abstract: Printed circuit boards (PCBs) are an integral part of electronic systems. Hence, verifying their physical integrity in the presence of supply chain attacks (e.g., tampering and counterfeiting) is of utmost importance. Recently, tamper detection techniques grounded in impedance characterization of PCB's Power Delivery Network (PDN) have gained prominence due to their global detection coverage, non-invasive, and low-cost nature. Similar to other physical verification methods, these techniques rely on the existence of a physical golden sample for signature comparisons. However, having access to a physical golden sample for golden signature extraction is not feasible in many real-world scenarios. In this work, we assess the feasibility of eliminating a physical golden sample and replacing it with a simulated golden signature obtained by the PCB design files. By performing extensive simulation and measurements on an in-house designed PCB, we demonstrate how the parasitic impedance of the PCB components plays a major role in reaching a successful verification. Based on the obtained results and using statistical metrics, we show that we can mitigate the discrepancy between collected signatures from simulation and measurements.

Title: FinLlama: Financial Sentiment Classification for Algorithmic Trading Applications

• Authors: Thanos Konstantinidis, Giorgos Iacovides, Mingxue Xu, Tony G. Constantinides, Danilo Mandic
• Subjects: cs.CL, cs.LG, q-fin.ST, q-fin.TR
• Abstract URL: https://arxiv.org/abs/2403.12285
• Pdf URL: https://arxiv.org/pdf/2403.12285
• Copy Paste: [[2403.12285]] FinLlama: Financial Sentiment Classification for Algorithmic Trading Applications(https://arxiv.org/abs/2403.12285)
• Keywords: generative, large language model
• Abstract: There are multiple sources of financial news online which influence market movements and trader's decisions. This highlights the need for accurate sentiment analysis, in addition to having appropriate algorithmic trading techniques, to arrive at better informed trading decisions. Standard lexicon based sentiment approaches have demonstrated their power in aiding financial decisions. However, they are known to suffer from issues related to context sensitivity and word ordering. Large Language Models (LLMs) can also be used in this context, but they are not finance-specific and tend to require significant computational resources. To facilitate a finance specific LLM framework, we introduce a novel approach based on the Llama 2 7B foundational model, in order to benefit from its generative nature and comprehensive language manipulation. This is achieved by fine-tuning the Llama2 7B model on a small portion of supervised financial sentiment analysis data, so as to jointly handle the complexities of financial lexicon and context, and further equipping it with a neural network based decision mechanism. Such a generator-classifier scheme, referred to as FinLlama, is trained not only to classify the sentiment valence but also quantify its strength, thus offering traders a nuanced insight into financial news articles. Complementing this, the implementation of parameter-efficient fine-tuning through LoRA optimises trainable parameters, thus minimising computational and memory requirements, without sacrificing accuracy. Simulation results demonstrate the ability of the proposed FinLlama to provide a framework for enhanced portfolio management decisions and increased market returns. These results underpin the ability of FinLlama to construct high-return portfolios which exhibit enhanced resilience, even during volatile periods and unpredictable market events.

Title: Leveraging Large Language Models to Extract Information on Substance Use Disorder Severity from Clinical Notes: A Zero-shot Learning Approach

• Authors: Maria Mahbub, Gregory M. Dams, Sudarshan Srinivasan, Caitlin Rizy, Ioana Danciu, Jodie Trafton, Kathryn Knight
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12297
• Pdf URL: https://arxiv.org/pdf/2403.12297
• Copy Paste: [[2403.12297]] Leveraging Large Language Models to Extract Information on Substance Use Disorder Severity from Clinical Notes: A Zero-shot Learning Approach(https://arxiv.org/abs/2403.12297)
• Keywords: large language model
• Abstract: Substance use disorder (SUD) poses a major concern due to its detrimental effects on health and society. SUD identification and treatment depend on a variety of factors such as severity, co-determinants (e.g., withdrawal symptoms), and social determinants of health. Existing diagnostic coding systems used by American insurance providers, like the International Classification of Diseases (ICD-10), lack granularity for certain diagnoses, but clinicians will add this granularity (as that found within the Diagnostic and Statistical Manual of Mental Disorders classification or DSM-5) as supplemental unstructured text in clinical notes. Traditional natural language processing (NLP) methods face limitations in accurately parsing such diverse clinical language. Large Language Models (LLMs) offer promise in overcoming these challenges by adapting to diverse language patterns. This study investigates the application of LLMs for extracting severity-related information for various SUD diagnoses from clinical notes. We propose a workflow employing zero-shot learning of LLMs with carefully crafted prompts and post-processing techniques. Through experimentation with Flan-T5, an open-source LLM, we demonstrate its superior recall compared to the rule-based approach. Focusing on 11 categories of SUD diagnoses, we show the effectiveness of LLMs in extracting severity information, contributing to improved risk assessment and treatment planning for SUD patients.

Title: Prototipo de un Contador Bidireccional Automático de Personas basado en sensores de visión 3D

• Authors: Benjamín Ojeda-Magaña, Rubén Ruelas, José Guadalupe Robledo-Hernández, Víctor Manuel Rangel-Cobián, Fernando López Aguilar-Hernández
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.12310
• Pdf URL: https://arxiv.org/pdf/2403.12310
• Copy Paste: [[2403.12310]] Prototipo de un Contador Bidireccional Automático de Personas basado en sensores de visión 3D(https://arxiv.org/abs/2403.12310)
• Keywords: security
• Abstract: 3D sensors, also known as RGB-D sensors, utilize depth images where each pixel measures the distance from the camera to objects, using principles like structured light or time-of-flight. Advances in artificial vision have led to affordable 3D cameras capable of real-time object detection without object movement, surpassing 2D cameras in information depth. These cameras can identify objects of varying colors and reflectivities and are less affected by lighting changes. The described prototype uses RGB-D sensors for bidirectional people counting in venues, aiding security and surveillance in spaces like stadiums or airports. It determines real-time occupancy and checks against maximum capacity, crucial during emergencies. The system includes a RealSense D415 depth camera and a mini-computer running object detection algorithms to count people and a 2D camera for identity verification. The system supports statistical analysis and uses C++, Python, and PHP with OpenCV for image processing, demonstrating a comprehensive approach to monitoring venue occupancy.

Title: Improving LoRA in Privacy-preserving Federated Learning

• Authors: Youbang Sun, Zitao Li, Yaliang Li, Bolin Ding
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.12313
• Pdf URL: https://arxiv.org/pdf/2403.12313
• Copy Paste: [[2403.12313]] Improving LoRA in Privacy-preserving Federated Learning(https://arxiv.org/abs/2403.12313)
• Keywords: privacy, federate
• Abstract: Low-rank adaptation (LoRA) is one of the most popular task-specific parameter-efficient fine-tuning (PEFT) methods on pre-trained language models for its good performance and computational efficiency. LoRA injects a product of two trainable rank decomposition matrices over the top of each frozen pre-trained model module. However, when applied in the setting of privacy-preserving federated learning (FL), LoRA may become unstable due to the following facts: 1) the effects of data heterogeneity and multi-step local updates are non-negligible, 2) additive noise enforced on updating gradients to guarantee differential privacy (DP) can be amplified and 3) the final performance is susceptible to hyper-parameters. A key factor leading to these phenomena is the discordance between jointly optimizing the two low-rank matrices by local clients and separately aggregating them by the central server. Thus, this paper proposes an efficient and effective version of LoRA, Federated Freeze A LoRA (FFA-LoRA), to alleviate these challenges and further halve the communication cost of federated fine-tuning LLMs. The core idea of FFA-LoRA is to fix the randomly initialized non-zero matrices and only fine-tune the zero-initialized matrices. Compared to LoRA, FFA-LoRA is motivated by practical and theoretical benefits in privacy-preserved FL. Our experiments demonstrate that FFA-LoRA provides more consistent performance with better computational efficiency over vanilla LoRA in various FL tasks.

Title: OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety

• Authors: Chuang Liu, Linhao Yu, Jiaxuan Li, Renren Jin, Yufei Huang, Ling Shi, Junhui Zhang, Xinmeng Ji, Tingting Cui, Tao Liu, Jinwang Song, Hongying Zan, Sun Li, Deyi Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12316
• Pdf URL: https://arxiv.org/pdf/2403.12316
• Copy Paste: [[2403.12316]] OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety(https://arxiv.org/abs/2403.12316)
• Keywords: large language model
• Abstract: The rapid development of Chinese large language models (LLMs) poses big challenges for efficient LLM evaluation. While current initiatives have introduced new benchmarks or evaluation platforms for assessing Chinese LLMs, many of these focus primarily on capabilities, usually overlooking potential alignment and safety issues. To address this gap, we introduce OpenEval, an evaluation testbed that benchmarks Chinese LLMs across capability, alignment and safety. For capability assessment, we include 12 benchmark datasets to evaluate Chinese LLMs from 4 sub-dimensions: NLP tasks, disciplinary knowledge, commonsense reasoning and mathematical reasoning. For alignment assessment, OpenEval contains 7 datasets that examines the bias, offensiveness and illegalness in the outputs yielded by Chinese LLMs. To evaluate safety, especially anticipated risks (e.g., power-seeking, self-awareness) of advanced LLMs, we include 6 datasets. In addition to these benchmarks, we have implemented a phased public evaluation and benchmark update strategy to ensure that OpenEval is in line with the development of Chinese LLMs or even able to provide cutting-edge benchmark datasets to guide the development of Chinese LLMs. In our first public evaluation, we have tested a range of Chinese LLMs, spanning from 7B to 72B parameters, including both open-source and proprietary models. Evaluation results indicate that while Chinese LLMs have shown impressive performance in certain tasks, more attention should be directed towards broader aspects such as commonsense reasoning, alignment, and safety.

Title: EffiPerception: an Efficient Framework for Various Perception Tasks

• Authors: Xinhao Xiang, Simon Dräger, Jiawei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12317
• Pdf URL: https://arxiv.org/pdf/2403.12317
• Copy Paste: [[2403.12317]] EffiPerception: an Efficient Framework for Various Perception Tasks(https://arxiv.org/abs/2403.12317)
• Keywords: robust, segmentation
• Abstract: The accuracy-speed-memory trade-off is always the priority to consider for several computer vision perception tasks. Previous methods mainly focus on a single or small couple of these tasks, such as creating effective data augmentation, feature extractor, learning strategies, etc. These approaches, however, could be inherently task-specific: their proposed model's performance may depend on a specific perception task or a dataset. Targeting to explore common learning patterns and increasing the module robustness, we propose the EffiPerception framework. It could achieve great accuracy-speed performance with relatively low memory cost under several perception tasks: 2D Object Detection, 3D Object Detection, 2D Instance Segmentation, and 3D Point Cloud Segmentation. Overall, the framework consists of three parts: (1) Efficient Feature Extractors, which extract the input features for each modality. (2) Efficient Layers, plug-in plug-out layers that further process the feature representation, aggregating core learned information while pruning noisy proposals. (3) The EffiOptim, an 8-bit optimizer to further cut down the computational cost and facilitate performance stability. Extensive experiments on the KITTI, semantic-KITTI, and COCO datasets revealed that EffiPerception could show great accuracy-speed-memory overall performance increase within the four detection and segmentation tasks, in comparison to earlier, well-respected methods.

Title: Removing Undesirable Concepts in Text-to-Image Generative Models with Learnable Prompts

• Authors: Anh Bui, Khanh Doan, Trung Le, Paul Montague, Tamas Abraham, Dinh Phung
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12326
• Pdf URL: https://arxiv.org/pdf/2403.12326
• Copy Paste: [[2403.12326]] Removing Undesirable Concepts in Text-to-Image Generative Models with Learnable Prompts(https://arxiv.org/abs/2403.12326)
• Keywords: diffusion, generative
• Abstract: Generative models have demonstrated remarkable potential in generating visually impressive content from textual descriptions. However, training these models on unfiltered internet data poses the risk of learning and subsequently propagating undesirable concepts, such as copyrighted or unethical content. In this paper, we propose a novel method to remove undesirable concepts from text-to-image generative models by incorporating a learnable prompt into the cross-attention module. This learnable prompt acts as additional memory to transfer the knowledge of undesirable concepts into it and reduce the dependency of these concepts on the model parameters and corresponding textual inputs. Because of this knowledge transfer into the prompt, erasing these undesirable concepts is more stable and has minimal negative impact on other concepts. We demonstrate the effectiveness of our method on the Stable Diffusion model, showcasing its superiority over state-of-the-art erasure methods in terms of removing undesirable content while preserving other unrelated elements.

Title: FedFisher: Leveraging Fisher Information for One-Shot Federated Learning

• Authors: Divyansh Jhunjhunwala, Shiqiang Wang, Gauri Joshi
• Subjects: cs.LG, cs.DC, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12329
• Pdf URL: https://arxiv.org/pdf/2403.12329
• Copy Paste: [[2403.12329]] FedFisher: Leveraging Fisher Information for One-Shot Federated Learning(https://arxiv.org/abs/2403.12329)
• Keywords: privacy, attack, federate
• Abstract: Standard federated learning (FL) algorithms typically require multiple rounds of communication between the server and the clients, which has several drawbacks, including requiring constant network connectivity, repeated investment of computational resources, and susceptibility to privacy attacks. One-Shot FL is a new paradigm that aims to address this challenge by enabling the server to train a global model in a single round of communication. In this work, we present FedFisher, a novel algorithm for one-shot FL that makes use of Fisher information matrices computed on local client models, motivated by a Bayesian perspective of FL. First, we theoretically analyze FedFisher for two-layer over-parameterized ReLU neural networks and show that the error of our one-shot FedFisher global model becomes vanishingly small as the width of the neural networks and amount of local training at clients increases. Next, we propose practical variants of FedFisher using the diagonal Fisher and K-FAC approximation for the full Fisher and highlight their communication and compute efficiency for FL. Finally, we conduct extensive experiments on various datasets, which show that these variants of FedFisher consistently improve over competing baselines.

Title: Temporally-Consistent Koopman Autoencoders for Forecasting Dynamical Systems

• Authors: Indranil Nayak, Debdipta Goswami, Mrinal Kumar, Fernando Teixeira
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12335
• Pdf URL: https://arxiv.org/pdf/2403.12335
• Copy Paste: [[2403.12335]] Temporally-Consistent Koopman Autoencoders for Forecasting Dynamical Systems(https://arxiv.org/abs/2403.12335)
• Keywords: robust
• Abstract: Absence of sufficiently high-quality data often poses a key challenge in data-driven modeling of high-dimensional spatio-temporal dynamical systems. Koopman Autoencoders (KAEs) harness the expressivity of deep neural networks (DNNs), the dimension reduction capabilities of autoencoders, and the spectral properties of the Koopman operator to learn a reduced-order feature space with simpler, linear dynamics. However, the effectiveness of KAEs is hindered by limited and noisy training datasets, leading to poor generalizability. To address this, we introduce the Temporally-Consistent Koopman Autoencoder (tcKAE), designed to generate accurate long-term predictions even with constrained and noisy training data. This is achieved through a consistency regularization term that enforces prediction coherence across different time steps, thus enhancing the robustness and generalizability of tcKAE over existing models. We provide analytical justification for this approach based on Koopman spectral theory and empirically demonstrate tcKAE's superior performance over state-of-the-art KAE models across a variety of test cases, including simple pendulum oscillations, kinetic plasmas, fluid flows, and sea surface temperature data.

Title: Friendly Sharpness-Aware Minimization

• Authors: Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, Xiaolin Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12350
• Pdf URL: https://arxiv.org/pdf/2403.12350
• Copy Paste: [[2403.12350]] Friendly Sharpness-Aware Minimization(https://arxiv.org/abs/2403.12350)
• Keywords: robust
• Abstract: Sharpness-Aware Minimization (SAM) has been instrumental in improving deep neural network training by minimizing both training loss and loss sharpness. Despite the practical success, the mechanisms behind SAM's generalization enhancements remain elusive, limiting its progress in deep learning optimization. In this work, we investigate SAM's core components for generalization improvement and introduce "Friendly-SAM" (F-SAM) to further enhance SAM's generalization. Our investigation reveals the key role of batch-specific stochastic gradient noise within the adversarial perturbation, i.e., the current minibatch gradient, which significantly influences SAM's generalization performance. By decomposing the adversarial perturbation in SAM into full gradient and stochastic gradient noise components, we discover that relying solely on the full gradient component degrades generalization while excluding it leads to improved performance. The possible reason lies in the full gradient component's increase in sharpness loss for the entire dataset, creating inconsistencies with the subsequent sharpness minimization step solely on the current minibatch data. Inspired by these insights, F-SAM aims to mitigate the negative effects of the full gradient component. It removes the full gradient estimated by an exponentially moving average (EMA) of historical stochastic gradients, and then leverages stochastic gradient noise for improved generalization. Moreover, we provide theoretical validation for the EMA approximation and prove the convergence of F-SAM on non-convex problems. Extensive experiments demonstrate the superior generalization performance and robustness of F-SAM over vanilla SAM. Code is available at https://github.com/nblt/F-SAM.

Title: E-DoH: Elegantly Detecting the Depths of Open DoH Service on the Internet

• Authors: Cong Dong, Jiahai Yang, Yun Li, Yue Wu, Yufan Chen, Chenglong Li, Haoran Jiao, Xia Yin, Yuling Liu
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.12363
• Pdf URL: https://arxiv.org/pdf/2403.12363
• Copy Paste: [[2403.12363]] E-DoH: Elegantly Detecting the Depths of Open DoH Service on the Internet(https://arxiv.org/abs/2403.12363)
• Keywords: protect
• Abstract: In recent years, DNS over Encrypted (DoE) methods have been regarded as a novel trend within the realm of the DNS ecosystem. In these DoE methods, DNS over HTTPS (DoH) provides encryption to protect data confidentiality while providing better obfuscation to avoid censorship by multiplexing port 443 with web services. This development introduced certain inconveniences in discovering publicly available DoH services. In this paper, we propose the E-DoH method for elegant and efficient DoH service detection. First, we optimized the probing mechanism to enable a single DoH connection to accomplish multiple tasks including service discovery, correctness validation and dependency construction. Second, we propose an efficient DoH detection tool. This tool can enhance probing efficiency while significantly reduce the required traffic volume. Third, based on the above optimization methods, we conducted an exploration of the IPv4 space and performed an in-depth analysis of DoH based on the collected information. Through experiments, our approach demonstrates a remarkable 80% improvement in time efficiency, and only requires 4%-20% traffic volume to complete the detection task. In wild detection, our approach discovered 46k DoH services, which nearly doubles the number discovered by the state-of-the-art. Based on the collected data, we present several intriguing conclusions about the current DoH service ecosystem.

Title: Class and Region-Adaptive Constraints for Network Calibration

• Authors: Balamurali Murugesan, Julio Silva-Rodriguez, Ismail Ben Ayed, Jose Dolz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12364
• Pdf URL: https://arxiv.org/pdf/2403.12364
• Copy Paste: [[2403.12364]] Class and Region-Adaptive Constraints for Network Calibration(https://arxiv.org/abs/2403.12364)
• Keywords: segmentation
• Abstract: In this work, we present a novel approach to calibrate segmentation networks that considers the inherent challenges posed by different categories and object regions. In particular, we present a formulation that integrates class and region-wise constraints into the learning objective, with multiple penalty weights to account for class and region differences. Finding the optimal penalty weights manually, however, might be unfeasible, and potentially hinder the optimization process. To overcome this limitation, we propose an approach based on Class and Region-Adaptive constraints (CRaC), which allows to learn the class and region-wise penalty weights during training. CRaC is based on a general Augmented Lagrangian method, a well-established technique in constrained optimization. Experimental results on two popular segmentation benchmarks, and two well-known segmentation networks, demonstrate the superiority of CRaC compared to existing approaches. The code is available at: https://github.com/Bala93/CRac/

Title: GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation

• Authors: Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, Ulrich Neumann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12365
• Pdf URL: https://arxiv.org/pdf/2403.12365
• Copy Paste: [[2403.12365]] GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation(https://arxiv.org/abs/2403.12365)
• Keywords: generative
• Abstract: Creating 4D fields of Gaussian Splatting from images or videos is a challenging task due to its under-constrained nature. While the optimization can draw photometric reference from the input videos or be regulated by generative models, directly supervising Gaussian motions remains underexplored. In this paper, we introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. The Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into the image space. This differentiable process enables direct dynamic supervision from optical flow. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting, especially for contents with rich motions that are hard to be handled by existing methods. The common color drifting issue that happens in 4D generation is also resolved with improved Guassian dynamics. Superior visual quality on extensive experiments demonstrates our method's effectiveness. Quantitative and qualitative evaluations show that our method achieves state-of-the-art results on both tasks of 4D generation and 4D novel view synthesis. Project page: https://zerg-overmind.github.io/GaussianFlow.github.io/

Title: Characteristic AI Agents via Large Language Models

• Authors: Xi Wang, Hongliang Dai, Shen Gao, Piji Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12368
• Pdf URL: https://arxiv.org/pdf/2403.12368
• Copy Paste: [[2403.12368]] Characteristic AI Agents via Large Language Models(https://arxiv.org/abs/2403.12368)
• Keywords: large language model
• Abstract: The advancement of Large Language Models (LLMs) has led to significant enhancements in the performance of chatbot systems. Many researchers have dedicated their efforts to the development of bringing characteristics to chatbots. While there have been commercial products for developing role-driven chatbots using LLMs, it is worth noting that academic research in this area remains relatively scarce. Our research focuses on investigating the performance of LLMs in constructing Characteristic AI Agents by simulating real-life individuals across different settings. Current investigations have primarily focused on act on roles with simple profiles. In response to this research gap, we create a benchmark for the characteristic AI agents task, including dataset, techniques, and evaluation metrics. A dataset called ``Character100'' is built for this benchmark, comprising the most-visited people on Wikipedia for language models to role-play. With the constructed dataset, we conduct comprehensive assessment of LLMs across various settings. In addition, we devise a set of automatic metrics for quantitative performance evaluation. The experimental results underscore the potential directions for further improvement in the capabilities of LLMs in constructing characteristic AI agents. The benchmark is available at https://github.com/nuaa-nlp/Character100.

Title: XPose: eXplainable Human Pose Estimation

• Authors: Luyu Qiu, Jianing Li, Lei Wen, Chi Su, Fei Hao, Chen Jason Zhang, Lei Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12370
• Pdf URL: https://arxiv.org/pdf/2403.12370
• Copy Paste: [[2403.12370]] XPose: eXplainable Human Pose Estimation(https://arxiv.org/abs/2403.12370)
• Keywords: interpretability
• Abstract: Current approaches in pose estimation primarily concentrate on enhancing model architectures, often overlooking the importance of comprehensively understanding the rationale behind model decisions. In this paper, we propose XPose, a novel framework that incorporates Explainable AI (XAI) principles into pose estimation. This integration aims to elucidate the individual contribution of each keypoint to final prediction, thereby elevating the model's transparency and interpretability. Conventional XAI techniques have predominantly addressed tasks with single-target tasks like classification. Additionally, the application of Shapley value, a common measure in XAI, to pose estimation has been hindered by prohibitive computational demands. To address these challenges, this work introduces an innovative concept called Group Shapley Value (GSV). This approach strategically organizes keypoints into clusters based on their interdependencies. Within these clusters, GSV meticulously calculates Shapley value for keypoints, while for inter-cluster keypoints, it opts for a more holistic group-level valuation. This dual-level computation framework meticulously assesses keypoint contributions to the final outcome, optimizing computational efficiency. Building on the insights into keypoint interactions, we devise a novel data augmentation technique known as Group-based Keypoint Removal (GKR). This method ingeniously removes individual keypoints during training phases, deliberately preserving those with strong mutual connections, thereby refining the model's predictive prowess for non-visible keypoints. The empirical validation of GKR across a spectrum of standard approaches attests to its efficacy. GKR's success demonstrates how using Explainable AI (XAI) can directly enhance pose estimation models.

Title: Advancing Time Series Classification with Multimodal Language Modeling

• Authors: Mingyue Cheng, Yiheng Chen, Qi Liu, Zhiding Liu, Yucong Luo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12371
• Pdf URL: https://arxiv.org/pdf/2403.12371
• Copy Paste: [[2403.12371]] Advancing Time Series Classification with Multimodal Language Modeling(https://arxiv.org/abs/2403.12371)
• Keywords: generative
• Abstract: For the advancements of time series classification, scrutinizing previous studies, most existing methods adopt a common learning-to-classify paradigm - a time series classifier model tries to learn the relation between sequence inputs and target label encoded by one-hot distribution. Although effective, this paradigm conceals two inherent limitations: (1) encoding target categories with one-hot distribution fails to reflect the comparability and similarity between labels, and (2) it is very difficult to learn transferable model across domains, which greatly hinder the development of universal serving paradigm. In this work, we propose InstructTime, a novel attempt to reshape time series classification as a learning-to-generate paradigm. Relying on the powerful generative capacity of the pre-trained language model, the core idea is to formulate the classification of time series as a multimodal understanding task, in which both task-specific instructions and raw time series are treated as multimodal inputs while the label information is represented by texts. To accomplish this goal, three distinct designs are developed in the InstructTime. Firstly, a time series discretization module is designed to convert continuous time series into a sequence of hard tokens to solve the inconsistency issue across modal inputs. To solve the modality representation gap issue, for one thing, we introduce an alignment projected layer before feeding the transformed token of time series into language models. For another, we highlight the necessity of auto-regressive pre-training across domains, which can facilitate the transferability of the language model and boost the generalization performance. Extensive experiments are conducted over benchmark datasets, whose results uncover the superior performance of InstructTime and the potential for a universal foundation model in time series classification.

Title: RankPrompt: Step-by-Step Comparisons Make Language Models Better Reasoners

• Authors: Chi Hu, Yuan Ge, Xiangnan Ma, Hang Cao, Qiang Li, Yonghua Yang, Tong Xiao, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12373
• Pdf URL: https://arxiv.org/pdf/2403.12373
• Copy Paste: [[2403.12373]] RankPrompt: Step-by-Step Comparisons Make Language Models Better Reasoners(https://arxiv.org/abs/2403.12373)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have achieved impressive performance across various reasoning tasks. However, even state-of-the-art LLMs such as ChatGPT are prone to logical errors during their reasoning processes. Existing solutions, which include deploying task-specific verifiers or voting over multiple reasoning paths, either require extensive human annotations or fail in scenarios with inconsistent responses. To address these challenges, we introduce RankPrompt, a new prompting method that enables LLMs to self-rank their responses without additional resources. RankPrompt breaks down the ranking problem into a series of comparisons among diverse responses, leveraging the inherent capabilities of LLMs to generate chains of comparison as contextual exemplars. Our experiments across 11 arithmetic and commonsense reasoning tasks show that RankPrompt significantly enhances the reasoning performance of ChatGPT and GPT-4, with improvements of up to 13\%. RankPrompt also excels in LLM-based automatic evaluations for open-ended generation, aligning with human preferences 74\% of the time in the AlpacaEval set. Moreover, RankPrompt demonstrates robustness against variations in the orderings and consistencies of responses.

Title: Improving Generalizability of Extracting Social Determinants of Health Using Large Language Models through Prompt-tuning

• Authors: Cheng Peng, Zehao Yu, Kaleb E Smith, Wei-Hsuan Lo-Ciganic, Jiang Bian, Yonghui Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12374
• Pdf URL: https://arxiv.org/pdf/2403.12374
• Copy Paste: [[2403.12374]] Improving Generalizability of Extracting Social Determinants of Health Using Large Language Models through Prompt-tuning(https://arxiv.org/abs/2403.12374)
• Keywords: extraction, large language model
• Abstract: The progress in natural language processing (NLP) using large language models (LLMs) has greatly improved patient information extraction from clinical narratives. However, most methods based on the fine-tuning strategy have limited transfer learning ability for cross-domain applications. This study proposed a novel approach that employs a soft prompt-based learning architecture, which introduces trainable prompts to guide LLMs toward desired outputs. We examined two types of LLM architectures, including encoder-only GatorTron and decoder-only GatorTronGPT, and evaluated their performance for the extraction of social determinants of health (SDoH) using a cross-institution dataset from the 2022 n2c2 challenge and a cross-disease dataset from the University of Florida (UF) Health. The results show that decoder-only LLMs with prompt tuning achieved better performance in cross-domain applications. GatorTronGPT achieved the best F1 scores for both datasets, outperforming traditional fine-tuned GatorTron by 8.9% and 21.8% in a cross-institution setting, and 5.5% and 14.5% in a cross-disease setting.

Title: VideoBadminton: A Video Dataset for Badminton Action Recognition

• Authors: Qi Li, Tzu-Chen Chiu, Hsiang-Wei Huang, Min-Te Sun, Wei-Shinn Ku
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12385
• Pdf URL: https://arxiv.org/pdf/2403.12385
• Copy Paste: [[2403.12385]] VideoBadminton: A Video Dataset for Badminton Action Recognition(https://arxiv.org/abs/2403.12385)
• Keywords: transformer
• Abstract: In the dynamic and evolving field of computer vision, action recognition has become a key focus, especially with the advent of sophisticated methodologies like Convolutional Neural Networks (CNNs), Convolutional 3D, Transformer, and spatial-temporal feature fusion. These technologies have shown promising results on well-established benchmarks but face unique challenges in real-world applications, particularly in sports analysis, where the precise decomposition of activities and the distinction of subtly different actions are crucial. Existing datasets like UCF101, HMDB51, and Kinetics have offered a diverse range of video data for various scenarios. However, there's an increasing need for fine-grained video datasets that capture detailed categorizations and nuances within broader action categories. In this paper, we introduce the VideoBadminton dataset derived from high-quality badminton footage. Through an exhaustive evaluation of leading methodologies on this dataset, this study aims to advance the field of action recognition, particularly in badminton sports. The introduction of VideoBadminton could not only serve for badminton action recognition but also provide a dataset for recognizing fine-grained actions. The insights gained from these evaluations are expected to catalyze further research in action comprehension, especially within sports contexts.

Title: Pipelined Biomedical Event Extraction Rivaling Joint Learning

• Authors: Pengchao Wu, Xuefeng Li, Jinghang Gu, Longhua Qian, Guodong Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12386
• Pdf URL: https://arxiv.org/pdf/2403.12386
• Copy Paste: [[2403.12386]] Pipelined Biomedical Event Extraction Rivaling Joint Learning(https://arxiv.org/abs/2403.12386)
• Keywords: extraction
• Abstract: Biomedical event extraction is an information extraction task to obtain events from biomedical text, whose targets include the type, the trigger, and the respective arguments involved in an event. Traditional biomedical event extraction usually adopts a pipelined approach, which contains trigger identification, argument role recognition, and finally event construction either using specific rules or by machine learning. In this paper, we propose an n-ary relation extraction method based on the BERT pre-training model to construct Binding events, in order to capture the semantic information about an event's context and its participants. The experimental results show that our method achieves promising results on the GE11 and GE13 corpora of the BioNLP shared task with F1 scores of 63.14% and 59.40%, respectively. It demonstrates that by significantly improving theperformance of Binding events, the overall performance of the pipelined event extraction approach or even exceeds those of current joint learning methods.

Title: FairSTG: Countering performance heterogeneity via collaborative sample-level optimization

• Authors: Gengyu Lin, Zhengyang Zhou, Qihe Huang, Kuo Yang, Shifen Cheng, Yang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12391
• Pdf URL: https://arxiv.org/pdf/2403.12391
• Copy Paste: [[2403.12391]] FairSTG: Countering performance heterogeneity via collaborative sample-level optimization(https://arxiv.org/abs/2403.12391)
• Keywords: fair
• Abstract: Spatiotemporal learning plays a crucial role in mobile computing techniques to empower smart cites. While existing research has made great efforts to achieve accurate predictions on the overall dataset, they still neglect the significant performance heterogeneity across samples. In this work, we designate the performance heterogeneity as the reason for unfair spatiotemporal learning, which not only degrades the practical functions of models, but also brings serious potential risks to real-world urban applications. To fix this gap, we propose a model-independent Fairness-aware framework for SpatioTemporal Graph learning (FairSTG), which inherits the idea of exploiting advantages of well-learned samples to challenging ones with collaborative mix-up. Specifically, FairSTG consists of a spatiotemporal feature extractor for model initialization, a collaborative representation enhancement for knowledge transfer between well-learned samples and challenging ones, and fairness objectives for immediately suppressing sample-level performance heterogeneity. Experiments on four spatiotemporal datasets demonstrate that our FairSTG significantly improves the fairness quality while maintaining comparable forecasting accuracy. Case studies show FairSTG can counter both spatial and temporal performance heterogeneity by our sample-level retrieval and compensation, and our work can potentially alleviate the risks on spatiotemporal resource allocation for underrepresented urban regions.

Title: Dr3: Ask Large Language Models Not to Give Off-Topic Answers in Open Domain Multi-Hop Question Answering

• Authors: Yuan Gao, Yiheng Zhu, Yuanbin Cao, Yinzhi Zhou, Zhen Wu, Yujie Chen, Shenglan Wu, Haoyuan Hu, Xinyu Dai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12393
• Pdf URL: https://arxiv.org/pdf/2403.12393
• Copy Paste: [[2403.12393]] Dr3: Ask Large Language Models Not to Give Off-Topic Answers in Open Domain Multi-Hop Question Answering(https://arxiv.org/abs/2403.12393)
• Keywords: large language model
• Abstract: Open Domain Multi-Hop Question Answering (ODMHQA) plays a crucial role in Natural Language Processing (NLP) by aiming to answer complex questions through multi-step reasoning over retrieved information from external knowledge sources. Recently, Large Language Models (LLMs) have demonstrated remarkable performance in solving ODMHQA owing to their capabilities including planning, reasoning, and utilizing tools. However, LLMs may generate off-topic answers when attempting to solve ODMHQA, namely the generated answers are irrelevant to the original questions. This issue of off-topic answers accounts for approximately one-third of incorrect answers, yet remains underexplored despite its significance. To alleviate this issue, we propose the Discriminate->Re-Compose->Re- Solve->Re-Decompose (Dr3) mechanism. Specifically, the Discriminator leverages the intrinsic capabilities of LLMs to judge whether the generated answers are off-topic. In cases where an off-topic answer is detected, the Corrector performs step-wise revisions along the reversed reasoning chain (Re-Compose->Re-Solve->Re-Decompose) until the final answer becomes on-topic. Experimental results on the HotpotQA and 2WikiMultiHopQA datasets demonstrate that our Dr3 mechanism considerably reduces the occurrence of off-topic answers in ODMHQA by nearly 13%, improving the performance in Exact Match (EM) by nearly 3% compared to the baseline method without the Dr3 mechanism.

Title: OV9D: Open-Vocabulary Category-Level 9D Object Pose and Size Estimation

• Authors: Junhao Cai, Yisheng He, Weihao Yuan, Siyu Zhu, Zilong Dong, Liefeng Bo, Qifeng Chen
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.12396
• Pdf URL: https://arxiv.org/pdf/2403.12396
• Copy Paste: [[2403.12396]] OV9D: Open-Vocabulary Category-Level 9D Object Pose and Size Estimation(https://arxiv.org/abs/2403.12396)
• Keywords: diffusion
• Abstract: This paper studies a new open-set problem, the open-vocabulary category-level object pose and size estimation. Given human text descriptions of arbitrary novel object categories, the robot agent seeks to predict the position, orientation, and size of the target object in the observed scene image. To enable such generalizability, we first introduce OO3D-9D, a large-scale photorealistic dataset for this task. Derived from OmniObject3D, OO3D-9D is the largest and most diverse dataset in the field of category-level object pose and size estimation. It includes additional annotations for the symmetry axis of each category, which help resolve symmetric ambiguity. Apart from the large-scale dataset, we find another key to enabling such generalizability is leveraging the strong prior knowledge in pre-trained visual-language foundation models. We then propose a framework built on pre-trained DinoV2 and text-to-image stable diffusion models to infer the normalized object coordinate space (NOCS) maps of the target instances. This framework fully leverages the visual semantic prior from DinoV2 and the aligned visual and language knowledge within the text-to-image diffusion model, which enables generalization to various text descriptions of novel categories. Comprehensive quantitative and qualitative experiments demonstrate that the proposed open-vocabulary method, trained on our large-scale synthesized data, significantly outperforms the baseline and can effectively generalize to real-world images of unseen categories. The project page is at https://ov9d.github.io.

Title: Electioneering the Network: Dynamic Multi-Step Adversarial Attacks for Community Canvassing

• Authors: Saurabh Sharma, Ambuj SIngh
• Subjects: cs.LG, cs.CR, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.12399
• Pdf URL: https://arxiv.org/pdf/2403.12399
• Copy Paste: [[2403.12399]] Electioneering the Network: Dynamic Multi-Step Adversarial Attacks for Community Canvassing(https://arxiv.org/abs/2403.12399)
• Keywords: attack, diffusion
• Abstract: The problem of online social network manipulation for community canvassing is of real concern in today's world. Motivated by the study of voter models, opinion and polarization dynamics on networks, we model community canvassing as a dynamic process over a network enabled via gradient-based attacks on GNNs. Existing attacks on GNNs are all single-step and do not account for the dynamic cascading nature of information diffusion in networks. We consider the realistic scenario where an adversary uses a GNN as a proxy to predict and manipulate voter preferences, especially uncertain voters. Gradient-based attacks on the GNN inform the adversary of strategic manipulations that can be made to proselytize targeted voters. In particular, we explore $\textit{minimum budget attacks for community canvassing}$ (MBACC). We show that the MBACC problem is NP-Hard and propose Dynamic Multi-Step Adversarial Community Canvassing (MAC) to address it. MAC makes dynamic local decisions based on the heuristic of low budget and high second-order influence to convert and perturb target voters. MAC is a dynamic multi-step attack that discovers low-budget and high-influence targets from which efficient cascading attacks can happen. We evaluate MAC against single-step baselines on the MBACC problem with multiple underlying networks and GNN models. Our experiments show the superiority of MAC which is able to discover efficient multi-hop attacks for adversarial community canvassing. Our code implementation and data is available at https://github.com/saurabhsharma1993/mac.

Title: Finding the Missing Data: A BERT-inspired Approach Against Package Loss in Wireless Sensing

• Authors: Zijian Zhao, Tingwei Chen, Fanyi Meng, Hang Li, Xiaoyang Li, Guangxu Zhu
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.12400
• Pdf URL: https://arxiv.org/pdf/2403.12400
• Copy Paste: [[2403.12400]] Finding the Missing Data: A BERT-inspired Approach Against Package Loss in Wireless Sensing(https://arxiv.org/abs/2403.12400)
• Keywords: transformer
• Abstract: Despite the development of various deep learning methods for Wi-Fi sensing, package loss often results in noncontinuous estimation of the Channel State Information (CSI), which negatively impacts the performance of the learning models. To overcome this challenge, we propose a deep learning model based on Bidirectional Encoder Representations from Transformers (BERT) for CSI recovery, named CSI-BERT. CSI-BERT can be trained in an self-supervised manner on the target dataset without the need for additional data. Furthermore, unlike traditional interpolation methods that focus on one subcarrier at a time, CSI-BERT captures the sequential relationships across different subcarriers. Experimental results demonstrate that CSI-BERT achieves lower error rates and faster speed compared to traditional interpolation methods, even when facing with high loss rates. Moreover, by harnessing the recovered CSI obtained from CSI-BERT, other deep learning models like Residual Network and Recurrent Neural Network can achieve an average increase in accuracy of approximately 15\% in Wi-Fi sensing tasks. The collected dataset WiGesture and code for our model are publicly available at https://github.com/RS2002/CSI-BERT.

Title: Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationales

• Authors: Ayushi Nirmal, Amrita Bhattacharjee, Paras Sheth, Huan Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12403
• Pdf URL: https://arxiv.org/pdf/2403.12403
• Copy Paste: [[2403.12403]] Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationales(https://arxiv.org/abs/2403.12403)
• Keywords: interpretability, large language model
• Abstract: Although social media platforms are a prominent arena for users to engage in interpersonal discussions and express opinions, the facade and anonymity offered by social media may allow users to spew hate speech and offensive content. Given the massive scale of such platforms, there arises a need to automatically identify and flag instances of hate speech. Although several hate speech detection methods exist, most of these black-box methods are not interpretable or explainable by design. To address the lack of interpretability, in this paper, we propose to use state-of-the-art Large Language Models (LLMs) to extract features in the form of rationales from the input text, to train a base hate speech classifier, thereby enabling faithful interpretability by design. Our framework effectively combines the textual understanding capabilities of LLMs and the discriminative power of state-of-the-art hate speech classifiers to make these classifiers faithfully interpretable. Our comprehensive evaluation on a variety of social media hate speech datasets demonstrate: (1) the goodness of the LLM-extracted rationales, and (2) the surprising retention of detector performance even after training to ensure interpretability.

Title: Understanding Training-free Diffusion Guidance: Mechanisms and Limitations

• Authors: Yifei Shen, Xinyang Jiang, Yezhen Wang, Yifan Yang, Dongqi Han, Dongsheng Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12404
• Pdf URL: https://arxiv.org/pdf/2403.12404
• Copy Paste: [[2403.12404]] Understanding Training-free Diffusion Guidance: Mechanisms and Limitations(https://arxiv.org/abs/2403.12404)
• Keywords: diffusion
• Abstract: Adding additional control to pretrained diffusion models has become an increasingly popular research area, with extensive applications in computer vision, reinforcement learning, and AI for science. Recently, several studies have proposed training-free diffusion guidance by using off-the-shelf networks pretrained on clean images. This approach enables zero-shot conditional generation for universal control formats, which appears to offer a free lunch in diffusion guidance. In this paper, we aim to develop a deeper understanding of the operational mechanisms and fundamental limitations of training-free guidance. We offer a theoretical analysis that supports training-free guidance from the perspective of optimization, distinguishing it from classifier-based (or classifier-free) guidance. To elucidate their drawbacks, we theoretically demonstrate that training-free methods are more susceptible to adversarial gradients and exhibit slower convergence rates compared to classifier guidance. We then introduce a collection of techniques designed to overcome the limitations, accompanied by theoretical rationale and empirical evidence. Our experiments in image and motion generation confirm the efficacy of these techniques.

Title: ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance

• Authors: Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12409
• Pdf URL: https://arxiv.org/pdf/2403.12409
• Copy Paste: [[2403.12409]] ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance(https://arxiv.org/abs/2403.12409)
• Keywords: diffusion
• Abstract: Generating high-quality 3D assets from a given image is highly desirable in various applications such as AR/VR. Recent advances in single-image 3D generation explore feed-forward models that learn to infer the 3D model of an object without optimization. Though promising results have been achieved in single object generation, these methods often struggle to model complex 3D assets that inherently contain multiple objects. In this work, we present ComboVerse, a 3D generation framework that produces high-quality 3D assets with complex compositions by learning to combine multiple models. 1) We first perform an in-depth analysis of this ``multi-object gap'' from both model and data perspectives. 2) Next, with reconstructed 3D models of different objects, we seek to adjust their sizes, rotation angles, and locations to create a 3D asset that matches the given image. 3) To automate this process, we apply spatially-aware score distillation sampling (SSDS) from pretrained diffusion models to guide the positioning of objects. Our proposed framework emphasizes spatial alignment of objects, compared with standard score distillation sampling, and thus achieves more accurate results. Extensive experiments validate ComboVerse achieves clear improvements over existing methods in generating compositional 3D assets.

Title: VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation

• Authors: Hao Wang, Jiayou Qin, Ashish Bastola, Xiwen Chen, John Suchanek, Zihao Gong, Abolfazl Razi
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.12415
• Pdf URL: https://arxiv.org/pdf/2403.12415
• Copy Paste: [[2403.12415]] VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation(https://arxiv.org/abs/2403.12415)
• Keywords: large language model
• Abstract: This paper explores the potential of Large Language Models(LLMs) in zero-shot anomaly detection for safe visual navigation. With the assistance of the state-of-the-art real-time open-world object detection model Yolo-World and specialized prompts, the proposed framework can identify anomalies within camera-captured frames that include any possible obstacles, then generate concise, audio-delivered descriptions emphasizing abnormalities, assist in safe visual navigation in complex circumstances. Moreover, our proposed framework leverages the advantages of LLMs and the open-vocabulary object detection model to achieve the dynamic scenario switch, which allows users to transition smoothly from scene to scene, which addresses the limitation of traditional visual navigation. Furthermore, this paper explored the performance contribution of different prompt components, provided the vision for future improvement in visual accessibility, and paved the way for LLMs in video anomaly detection and vision-language understanding.

Title: Eye-gaze Guided Multi-modal Alignment Framework for Radiology

• Authors: Chong Ma, Hanqi Jiang, Wenting Chen, Zihao Wu, Xiaowei Yu, Fang Zeng, Lei Guo, Dajiang Zhu, Tuo Zhang, Dinggang Shen, Tianming Liu, Xiang Li
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12416
• Pdf URL: https://arxiv.org/pdf/2403.12416
• Copy Paste: [[2403.12416]] Eye-gaze Guided Multi-modal Alignment Framework for Radiology(https://arxiv.org/abs/2403.12416)
• Keywords: robust, interpretability
• Abstract: In multi-modal frameworks, the alignment of cross-modal features presents a significant challenge. The predominant approach in multi-modal pre-training emphasizes either global or local alignment between modalities, utilizing extensive datasets. This bottom-up driven method often suffers from a lack of interpretability, a critical concern in radiology. Previous studies have integrated high-level labels in medical images or text, but these still rely on manual annotation, a costly and labor-intensive process. Our work introduces a novel approach by using eye-gaze data, collected synchronously by radiologists during diagnostic evaluations. This data, indicating radiologists' focus areas, naturally links chest X-rays to diagnostic texts. We propose the Eye-gaze Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for better alignment of image and text features, aiming to reduce reliance on manual annotations and thus cut training costs. Our model demonstrates robust performance, outperforming other state-of-the-art methods in zero-shot classification and retrieval tasks. The incorporation of easily-obtained eye-gaze data during routine radiological diagnoses signifies a step towards minimizing manual annotation dependency. Additionally, we explore the impact of varying amounts of eye-gaze data on model performance, highlighting the feasibility and utility of integrating this auxiliary data into multi-modal pre-training.

Title: Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization

• Authors: Haocheng Xi, Yuxiang Chen, Kang Zhao, Kaijun Zheng, Jianfei Chen, Jun Zhu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12422
• Pdf URL: https://arxiv.org/pdf/2403.12422
• Copy Paste: [[2403.12422]] Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization(https://arxiv.org/abs/2403.12422)
• Keywords: transformer
• Abstract: Pretraining transformers are generally time-consuming. Fully quantized training (FQT) is a promising approach to speed up pretraining. However, most FQT methods adopt a quantize-compute-dequantize procedure, which often leads to suboptimal speedup and significant performance degradation when used in transformers due to the high memory access overheads and low-precision computations. In this work, we propose Jetfire, an efficient and accurate INT8 training method specific to transformers. Our method features an INT8 data flow to optimize memory access and a per-block quantization method to maintain the accuracy of pretrained transformers. Extensive experiments demonstrate that our INT8 FQT method achieves comparable accuracy to the FP16 training baseline and outperforms the existing INT8 training works for transformers. Moreover, for a standard transformer block, our method offers an end-to-end training speedup of 1.42x and a 1.49x memory reduction compared to the FP16 baseline.

Title: Multimodal Fusion Method with Spatiotemporal Sequences and Relationship Learning for Valence-Arousal Estimation

• Authors: Jun Yu, Gongpeng Zhao, Yongqi Wan, Zhihong Wei, Yang Zheng, Zerui Zhang, Zhongpeng Cai, Guochen Xie, Jichao Zhu, Wangyuan Zhu
• Subjects: cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.12425
• Pdf URL: https://arxiv.org/pdf/2403.12425
• Copy Paste: [[2403.12425]] Multimodal Fusion Method with Spatiotemporal Sequences and Relationship Learning for Valence-Arousal Estimation(https://arxiv.org/abs/2403.12425)
• Keywords: extraction, transformer
• Abstract: This paper presents our approach for the VA (Valence-Arousal) estimation task in the ABAW6 competition. We devised a comprehensive model by preprocessing video frames and audio segments to extract visual and audio features. Through the utilization of Temporal Convolutional Network (TCN) modules, we effectively captured the temporal and spatial correlations between these features. Subsequently, we employed a Transformer encoder structure to learn long-range dependencies, thereby enhancing the model's performance and generalization ability. Our method leverages a multimodal data fusion approach, integrating pre-trained audio and video backbones for feature extraction, followed by TCN-based spatiotemporal encoding and Transformer-based temporal information capture. Experimental results demonstrate the effectiveness of our approach, achieving competitive performance in VA estimation on the AffWild2 dataset.

Title: Human Mesh Recovery from Arbitrary Multi-view Images

• Authors: Xiaoben Li, Mancheng Meng, Ziyan Wu, Terrence Chen, Fan Yang, Dinggang Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12434
• Pdf URL: https://arxiv.org/pdf/2403.12434
• Copy Paste: [[2403.12434]] Human Mesh Recovery from Arbitrary Multi-view Images(https://arxiv.org/abs/2403.12434)
• Keywords: transformer
• Abstract: Human mesh recovery from arbitrary multi-view images involves two characteristics: the arbitrary camera poses and arbitrary number of camera views. Because of the variability, designing a unified framework to tackle this task is challenging. The challenges can be summarized as the dilemma of being able to simultaneously estimate arbitrary camera poses and recover human mesh from arbitrary multi-view images while maintaining flexibility. To solve this dilemma, we propose a divide and conquer framework for Unified Human Mesh Recovery (U-HMR) from arbitrary multi-view images. In particular, U-HMR consists of a decoupled structure and two main components: camera and body decoupling (CBD), camera pose estimation (CPE), and arbitrary view fusion (AVF). As camera poses and human body mesh are independent of each other, CBD splits the estimation of them into two sub-tasks for two individual sub-networks (\ie, CPE and AVF) to handle respectively, thus the two sub-tasks are disentangled. In CPE, since each camera pose is unrelated to the others, we adopt a shared MLP to process all views in a parallel way. In AVF, in order to fuse multi-view information and make the fusion operation independent of the number of views, we introduce a transformer decoder with a SMPL parameters query token to extract cross-view features for mesh recovery. To demonstrate the efficacy and flexibility of the proposed framework and effect of each component, we conduct extensive experiments on three public datasets: Human3.6M, MPI-INF-3DHP, and TotalCapture.

Title: Precise-Physics Driven Text-to-3D Generation

• Authors: Qingshan Xu, Jiao Liu, Melvin Wong, Caishun Chen, Yew-Soon Ong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12438
• Pdf URL: https://arxiv.org/pdf/2403.12438
• Copy Paste: [[2403.12438]] Precise-Physics Driven Text-to-3D Generation(https://arxiv.org/abs/2403.12438)
• Keywords: diffusion, generative
• Abstract: Text-to-3D generation has shown great promise in generating novel 3D content based on given text prompts. However, existing generative methods mostly focus on geometric or visual plausibility while ignoring precise physics perception for the generated 3D shapes. This greatly hinders the practicality of generated 3D shapes in real-world applications. In this work, we propose Phy3DGen, a precise-physics-driven text-to-3D generation method. By analyzing the solid mechanics of generated 3D shapes, we reveal that the 3D shapes generated by existing text-to-3D generation methods are impractical for real-world applications as the generated 3D shapes do not conform to the laws of physics. To this end, we leverage 3D diffusion models to provide 3D shape priors and design a data-driven differentiable physics layer to optimize 3D shape priors with solid mechanics. This allows us to optimize geometry efficiently and learn precise physics information about 3D shapes at the same time. Experimental results demonstrate that our method can consider both geometric plausibility and precise physics perception, further bridging 3D virtual modeling and precise physical worlds.

Title: Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory

• Authors: Sensen Gao, Xiaojun Jia, Xuhong Ren, Ivor Tsang, Qing Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12445
• Pdf URL: https://arxiv.org/pdf/2403.12445
• Copy Paste: [[2403.12445]] Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory(https://arxiv.org/abs/2403.12445)
• Keywords: attack
• Abstract: Vision-language pre-training (VLP) models exhibit remarkable capabilities in comprehending both images and text, yet they remain susceptible to multimodal adversarial examples (AEs). Strengthening adversarial attacks and uncovering vulnerabilities, especially common issues in VLP models (e.g., high transferable AEs), can stimulate further research on constructing reliable and practical VLP models. A recent work (i.e., Set-level guidance attack) indicates that augmenting image-text pairs to increase AE diversity along the optimization path enhances the transferability of adversarial examples significantly. However, this approach predominantly emphasizes diversity around the online adversarial examples (i.e., AEs in the optimization period), leading to the risk of overfitting the victim model and affecting the transferability. In this study, we posit that the diversity of adversarial examples towards the clean input and online AEs are both pivotal for enhancing transferability across VLP models. Consequently, we propose using diversification along the intersection region of adversarial trajectory to expand the diversity of AEs. To fully leverage the interaction between modalities, we introduce text-guided adversarial example selection during optimization. Furthermore, to further mitigate the potential overfitting, we direct the adversarial text deviating from the last intersection region along the optimization path, rather than adversarial images as in existing methods. Extensive experiments affirm the effectiveness of our method in improving transferability across various VLP models and downstream vision-and-language tasks (e.g., Image-Text Retrieval(ITR), Visual Grounding(VG), Image Captioning(IC)).

Title: Do Generated Data Always Help Contrastive Learning?

• Authors: Yifei Wang, Jizhe Zhang, Yisen Wang
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12448
• Pdf URL: https://arxiv.org/pdf/2403.12448
• Copy Paste: [[2403.12448]] Do Generated Data Always Help Contrastive Learning?(https://arxiv.org/abs/2403.12448)
• Keywords: diffusion, generative
• Abstract: Contrastive Learning (CL) has emerged as one of the most successful paradigms for unsupervised visual representation learning, yet it often depends on intensive manual data augmentations. With the rise of generative models, especially diffusion models, the ability to generate realistic images close to the real data distribution has been well recognized. These generated high-equality images have been successfully applied to enhance contrastive representation learning, a technique termed ``data inflation''. However, we find that the generated data (even from a good diffusion model like DDPM) may sometimes even harm contrastive learning. We investigate the causes behind this failure from the perspective of both data inflation and data augmentation. For the first time, we reveal the complementary roles that stronger data inflation should be accompanied by weaker augmentations, and vice versa. We also provide rigorous theoretical explanations for these phenomena via deriving its generalization bounds under data inflation. Drawing from these insights, we propose Adaptive Inflation (AdaInf), a purely data-centric strategy without introducing any extra computation cost. On benchmark datasets, AdaInf can bring significant improvements for various contrastive learning methods. Notably, without using external data, AdaInf obtains 94.70% linear accuracy on CIFAR-10 with SimCLR, setting a new record that surpasses many sophisticated methods. Code is available at https://github.com/PKU-ML/adainf.

Title: CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation

• Authors: Wenqi Zhu, Jiale Cao, Jin Xie, Shuangming Yang, Yanwei Pang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12455
• Pdf URL: https://arxiv.org/pdf/2403.12455
• Copy Paste: [[2403.12455]] CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation(https://arxiv.org/abs/2403.12455)
• Keywords: transformer, segmentation
• Abstract: Open-vocabulary video instance segmentation strives to segment and track instances belonging to an open set of categories in a video. The vision-language model Contrastive Language-Image Pre-training (CLIP) has shown strong zero-shot classification ability in image-level open-vocabulary task. In this paper, we propose a simple encoder-decoder network, called CLIP-VIS, to adapt CLIP for open-vocabulary video instance segmentation. Our CLIP-VIS adopts frozen CLIP image encoder and introduces three modules, including class-agnostic mask generation, temporal topK-enhanced matching, and weighted open-vocabulary classification. Given a set of initial queries, class-agnostic mask generation employs a transformer decoder to predict query masks and corresponding object scores and mask IoU scores. Then, temporal topK-enhanced matching performs query matching across frames by using K mostly matched frames. Finally, weighted open-vocabulary classification first generates query visual features with mask pooling, and second performs weighted classification using object scores and mask IoU scores. Our CLIP-VIS does not require the annotations of instance categories and identities. The experiments are performed on various video instance segmentation datasets, which demonstrate the effectiveness of our proposed method, especially on novel categories. When using ConvNeXt-B as backbone, our CLIP-VIS achieves the AP and APn scores of 32.1% and 40.3% on validation set of LV-VIS dataset, which outperforms OV2Seg by 11.0% and 24.0% respectively. We will release the source code and models at https://github.com/zwq456/CLIP-VIS.git.

Title: Privacy-Preserving Face Recognition Using Trainable Feature Subtraction

• Authors: Yuxi Mi, Zhizhou Zhong, Yuge Huang, Jiazhen Ji, Jianqing Xu, Jun Wang, Shaoming Wang, Shouhong Ding, Shuigeng Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12457
• Pdf URL: https://arxiv.org/pdf/2403.12457
• Copy Paste: [[2403.12457]] Privacy-Preserving Face Recognition Using Trainable Feature Subtraction(https://arxiv.org/abs/2403.12457)
• Keywords: privacy, protect, attack
• Abstract: The widespread adoption of face recognition has led to increasing privacy concerns, as unauthorized access to face images can expose sensitive personal information. This paper explores face image protection against viewing and recovery attacks. Inspired by image compression, we propose creating a visually uninformative face image through feature subtraction between an original face and its model-produced regeneration. Recognizable identity features within the image are encouraged by co-training a recognition model on its high-dimensional feature representation. To enhance privacy, the high-dimensional representation is crafted through random channel shuffling, resulting in randomized recognizable images devoid of attacker-leverageable texture details. We distill our methodologies into a novel privacy-preserving face recognition method, MinusFace. Experiments demonstrate its high recognition accuracy and effective privacy protection. Its code is available at https://github.com/Tencent/TFace.

Title: Non-negative Contrastive Learning

• Authors: Yifei Wang, Qi Zhang, Yaoyu Guo, Yisen Wang
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12459
• Pdf URL: https://arxiv.org/pdf/2403.12459
• Copy Paste: [[2403.12459]] Non-negative Contrastive Learning(https://arxiv.org/abs/2403.12459)
• Keywords: interpretability
• Abstract: Deep representations have shown promising performance when transferred to downstream tasks in a black-box manner. Yet, their inherent lack of interpretability remains a significant challenge, as these features are often opaque to human understanding. In this paper, we propose Non-negative Contrastive Learning (NCL), a renaissance of Non-negative Matrix Factorization (NMF) aimed at deriving interpretable features. The power of NCL lies in its enforcement of non-negativity constraints on features, reminiscent of NMF's capability to extract features that align closely with sample clusters. NCL not only aligns mathematically well with an NMF objective but also preserves NMF's interpretability attributes, resulting in a more sparse and disentangled representation compared to standard contrastive learning (CL). Theoretically, we establish guarantees on the identifiability and downstream generalization of NCL. Empirically, we show that these advantages enable NCL to outperform CL significantly on feature disentanglement, feature selection, as well as downstream classification tasks. At last, we show that NCL can be easily extended to other learning scenarios and benefit supervised learning as well. Code is available at https://github.com/PKU-ML/non_neg.

Title: SC-Diff: 3D Shape Completion with Latent Diffusion Models

• Authors: Juan D. Galvis, Xingxing Zuo, Simon Schaefer, Stefan Leutengger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12470
• Pdf URL: https://arxiv.org/pdf/2403.12470
• Copy Paste: [[2403.12470]] SC-Diff: 3D Shape Completion with Latent Diffusion Models(https://arxiv.org/abs/2403.12470)
• Keywords: diffusion
• Abstract: This paper introduces a 3D shape completion approach using a 3D latent diffusion model optimized for completing shapes, represented as Truncated Signed Distance Functions (TSDFs), from partial 3D scans. Our method combines image-based conditioning through cross-attention and spatial conditioning through the integration of 3D features from captured partial scans. This dual guidance enables high-fidelity, realistic shape completions at superior resolutions. At the core of our approach is the compression of 3D data into a low-dimensional latent space using an auto-encoder inspired by 2D latent diffusion models. This compression facilitates the processing of higher-resolution shapes and allows us to apply our model across multiple object classes, a significant improvement over other existing diffusion-based shape completion methods, which often require a separate diffusion model for each class. We validated our approach against two common benchmarks in the field of shape completion, demonstrating competitive performance in terms of accuracy and realism and performing on par with state-of-the-art methods despite operating at a higher resolution with a single model for all object classes. We present a comprehensive evaluation of our model, showcasing its efficacy in handling diverse shape completion challenges, even on unseen object classes. The code will be released upon acceptance.

Title: PostoMETRO: Pose Token Enhanced Mesh Transformer for Robust 3D Human Mesh Recovery

• Authors: Wendi Yang, Zihang Jiang, Shang Zhao, S. Kevin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12473
• Pdf URL: https://arxiv.org/pdf/2403.12473
• Copy Paste: [[2403.12473]] PostoMETRO: Pose Token Enhanced Mesh Transformer for Robust 3D Human Mesh Recovery(https://arxiv.org/abs/2403.12473)
• Keywords: robust, transformer
• Abstract: With the recent advancements in single-image-based human mesh recovery, there is a growing interest in enhancing its performance in certain extreme scenarios, such as occlusion, while maintaining overall model accuracy. Although obtaining accurately annotated 3D human poses under occlusion is challenging, there is still a wealth of rich and precise 2D pose annotations that can be leveraged. However, existing works mostly focus on directly leveraging 2D pose coordinates to estimate 3D pose and mesh. In this paper, we present PostoMETRO($\textbf{Pos}$e $\textbf{to}$ken enhanced $\textbf{ME}$sh $\textbf{TR}$ansf$\textbf{O}$rmer), which integrates occlusion-resilient 2D pose representation into transformers in a token-wise manner. Utilizing a specialized pose tokenizer, we efficiently condense 2D pose data to a compact sequence of pose tokens and feed them to the transformer together with the image tokens. This process not only ensures a rich depiction of texture from the image but also fosters a robust integration of pose and image information. Subsequently, these combined tokens are queried by vertex and joint tokens to decode 3D coordinates of mesh vertices and human joints. Facilitated by the robust pose token representation and the effective combination, we are able to produce more precise 3D coordinates, even under extreme scenarios like occlusion. Experiments on both standard and occlusion-specific benchmarks demonstrate the effectiveness of PostoMETRO. Qualitative results further illustrate the clarity of how 2D pose can help 3D reconstruction. Code will be made available.

Title: FairSIN: Achieving Fairness in Graph Neural Networks through Sensitive Information Neutralization

• Authors: Cheng Yang, Jixi Liu, Yunhe Yan, Chuan Shi
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.12474
• Pdf URL: https://arxiv.org/pdf/2403.12474
• Copy Paste: [[2403.12474]] FairSIN: Achieving Fairness in Graph Neural Networks through Sensitive Information Neutralization(https://arxiv.org/abs/2403.12474)
• Keywords: fair
• Abstract: Despite the remarkable success of graph neural networks (GNNs) in modeling graph-structured data, like other machine learning models, GNNs are also susceptible to making biased predictions based on sensitive attributes, such as race and gender. For fairness consideration, recent state-of-the-art (SOTA) methods propose to filter out sensitive information from inputs or representations, e.g., edge dropping or feature masking. However, we argue that such filtering-based strategies may also filter out some non-sensitive feature information, leading to a sub-optimal trade-off between predictive performance and fairness. To address this issue, we unveil an innovative neutralization-based paradigm, where additional Fairness-facilitating Features (F3) are incorporated into node features or representations before message passing. The F3 are expected to statistically neutralize the sensitive bias in node representations and provide additional nonsensitive information. We also provide theoretical explanations for our rationale, concluding that F3 can be realized by emphasizing the features of each node's heterogeneous neighbors (neighbors with different sensitive attributes). We name our method as FairSIN, and present three implementation variants from both data-centric and model-centric perspectives. Experimental results on five benchmark datasets with three different GNN backbones show that FairSIN significantly improves fairness metrics while maintaining high prediction accuracies.

Title: TT-BLIP: Enhancing Fake News Detection Using BLIP and Tri-Transformer

• Authors: Eunjee Choi, Jong-Kook Kim
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12481
• Pdf URL: https://arxiv.org/pdf/2403.12481
• Copy Paste: [[2403.12481]] TT-BLIP: Enhancing Fake News Detection Using BLIP and Tri-Transformer(https://arxiv.org/abs/2403.12481)
• Keywords: extraction, transformer
• Abstract: Detecting fake news has received a lot of attention. Many previous methods concatenate independently encoded unimodal data, ignoring the benefits of integrated multimodal information. Also, the absence of specialized feature extraction for text and images further limits these methods. This paper introduces an end-to-end model called TT-BLIP that applies the bootstrapping language-image pretraining for unified vision-language understanding and generation (BLIP) for three types of information: BERT and BLIP\textsubscript{Txt} for text, ResNet and BLIP\textsubscript{Img} for images, and bidirectional BLIP encoders for multimodal information. The Multimodal Tri-Transformer fuses tri-modal features using three types of multi-head attention mechanisms, ensuring integrated modalities for enhanced representations and improved multimodal data analysis. The experiments are performed using two fake news datasets, Weibo and Gossipcop. The results indicate TT-BLIP outperforms the state-of-the-art models.

Title: A Hybrid Transformer-Sequencer approach for Age and Gender classification from in-wild facial images

• Authors: Aakash Singh, Vivek Kumar Singh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12483
• Pdf URL: https://arxiv.org/pdf/2403.12483
• Copy Paste: [[2403.12483]] A Hybrid Transformer-Sequencer approach for Age and Gender classification from in-wild facial images(https://arxiv.org/abs/2403.12483)
• Keywords: transformer
• Abstract: The advancements in computer vision and image processing techniques have led to emergence of new application in the domain of visual surveillance, targeted advertisement, content-based searching, and human-computer interaction etc. Out of the various techniques in computer vision, face analysis, in particular, has gained much attention. Several previous studies have tried to explore different applications of facial feature processing for a variety of tasks, including age and gender classification. However, despite several previous studies having explored the problem, the age and gender classification of in-wild human faces is still far from the achieving the desired levels of accuracy required for real-world applications. This paper, therefore, attempts to bridge this gap by proposing a hybrid model that combines self-attention and BiLSTM approaches for age and gender classification problems. The proposed models performance is compared with several state-of-the-art model proposed so far. An improvement of approximately 10percent and 6percent over the state-of-the-art implementations for age and gender classification, respectively, are noted for the proposed model. The proposed model is thus found to achieve superior performance and is found to provide a more generalized learning. The model can, therefore, be applied as a core classification component in various image processing and computer vision problems.

Title: NTK-Guided Few-Shot Class Incremental Learning

• Authors: Jingren Liu, Zhong Ji, Yanwei Pang, YunLong Yu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12486
• Pdf URL: https://arxiv.org/pdf/2403.12486
• Copy Paste: [[2403.12486]] NTK-Guided Few-Shot Class Incremental Learning(https://arxiv.org/abs/2403.12486)
• Keywords: robust
• Abstract: While anti-amnesia FSCIL learners often excel in incremental sessions, they tend to prioritize mitigating knowledge attrition over harnessing the model's potential for knowledge acquisition. In this paper, we delve into the foundations of model generalization in FSCIL through the lens of the Neural Tangent Kernel (NTK). Our primary design focus revolves around ensuring optimal NTK convergence and NTK-related generalization error, serving as the theoretical bedrock for exceptional generalization. To attain globally optimal NTK convergence, we employ a meta-learning mechanism grounded in mathematical principles to guide the optimization process within an expanded network. Furthermore, to reduce the NTK-related generalization error, we commence from the foundational level, optimizing the relevant factors constituting its generalization loss. Specifically, we initiate self-supervised pre-training on the base session to shape the initial network weights. Then they are carefully refined through curricular alignment, followed by the application of dual NTK regularization tailored specifically for both convolutional and linear layers. Through the combined effects of these measures, our network acquires robust NTK properties, significantly enhancing its foundational generalization. On popular FSCIL benchmark datasets, our NTK-FSCIL surpasses contemporary state-of-the-art approaches, elevating end-session accuracy by 2.9% to 8.7%.

Title: DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM

• Authors: Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Jian Wu, Philip Torr
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12488
• Pdf URL: https://arxiv.org/pdf/2403.12488
• Copy Paste: [[2403.12488]] DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM(https://arxiv.org/abs/2403.12488)
• Keywords: large language model
• Abstract: We present DetToolChain, a novel prompting paradigm, to unleash the zero-shot object detection ability of multimodal large language models (MLLMs), such as GPT-4V and Gemini. Our approach consists of a detection prompting toolkit inspired by high-precision detection priors and a new Chain-of-Thought to implement these prompts. Specifically, the prompts in the toolkit are designed to guide the MLLM to focus on regional information (e.g., zooming in), read coordinates according to measure standards (e.g., overlaying rulers and compasses), and infer from the contextual information (e.g., overlaying scene graphs). Building upon these tools, the new detection chain-of-thought can automatically decompose the task into simple subtasks, diagnose the predictions, and plan for progressive box refinements. The effectiveness of our framework is demonstrated across a spectrum of detection tasks, especially hard cases. Compared to existing state-of-the-art methods, GPT-4V with our DetToolChain improves state-of-the-art object detectors by +21.5% AP50 on MS COCO Novel class set for open-vocabulary detection, +24.23% Acc on RefCOCO val set for zero-shot referring expression comprehension, +14.5% AP on D-cube describe object detection FULL setting.

Title: A Trainable Feature Extractor Module for Deep Neural Networks and Scanpath Classification

• Authors: Wolfgang Fuhl
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12493
• Pdf URL: https://arxiv.org/pdf/2403.12493
• Copy Paste: [[2403.12493]] A Trainable Feature Extractor Module for Deep Neural Networks and Scanpath Classification(https://arxiv.org/abs/2403.12493)
• Keywords: extraction
• Abstract: Scanpath classification is an area in eye tracking research with possible applications in medicine, manufacturing as well as training systems for students in various domains. In this paper we propose a trainable feature extraction module for deep neural networks. The purpose of this module is to transform a scanpath into a feature vector which is directly useable for the deep neural network architecture. Based on the backpropagated error of the deep neural network, the feature extraction module adapts its parameters to improve the classification performance. Therefore, our feature extraction module is jointly trainable with the deep neural network. The motivation to this feature extraction module is based on classical histogram-based approaches which usually compute distributions over a scanpath. We evaluated our module on three public datasets and compared it to the state of the art approaches.

Title: Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices

• Authors: Sara Abdali, Richard Anarfi, CJ Barberan, Jia He
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12503
• Pdf URL: https://arxiv.org/pdf/2403.12503
• Copy Paste: [[2403.12503]] Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices(https://arxiv.org/abs/2403.12503)
• Keywords: security, privacy, attack, large language model
• Abstract: Large language models (LLMs) have significantly transformed the landscape of Natural Language Processing (NLP). Their impact extends across a diverse spectrum of tasks, revolutionizing how we approach language understanding and generations. Nevertheless, alongside their remarkable utility, LLMs introduce critical security and risk considerations. These challenges warrant careful examination to ensure responsible deployment and safeguard against potential vulnerabilities. This research paper thoroughly investigates security and privacy concerns related to LLMs from five thematic perspectives: security and privacy concerns, vulnerabilities against adversarial attacks, potential harms caused by misuses of LLMs, mitigation strategies to address these challenges while identifying limitations of current strategies. Lastly, the paper recommends promising avenues for future research to enhance the security and risk management of LLMs.

Title: Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation

• Authors: Xu Zheng, Pengyuan Zhou, Athanasios Vasilakos, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12505
• Pdf URL: https://arxiv.org/pdf/2403.12505
• Copy Paste: [[2403.12505]] Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation(https://arxiv.org/abs/2403.12505)
• Keywords: extraction, segmentation
• Abstract: This paper addresses an interesting yet challenging problem -- source-free unsupervised domain adaptation (SFUDA) for pinhole-to-panoramic semantic segmentation -- given only a pinhole image-trained model (i.e., source) and unlabeled panoramic images (i.e., target). Tackling this problem is nontrivial due to the semantic mismatches, style discrepancies, and inevitable distortion of panoramic images. To this end, we propose a novel method that utilizes Tangent Projection (TP) as it has less distortion and meanwhile slits the equirectangular projection (ERP) with a fixed FoV to mimic the pinhole images. Both projections are shown effective in extracting knowledge from the source model. However, the distinct projection discrepancies between source and target domains impede the direct knowledge transfer; thus, we propose a panoramic prototype adaptation module (PPAM) to integrate panoramic prototypes from the extracted knowledge for adaptation. We then impose the loss constraints on both predictions and prototypes and propose a cross-dual attention module (CDAM) at the feature level to better align the spatial and channel characteristics across the domains and projections. Both knowledge extraction and transfer processes are synchronously updated to reach the best performance. Extensive experiments on the synthetic and real-world benchmarks, including outdoor and indoor scenarios, demonstrate that our method achieves significantly better performance than prior SFUDA methods for pinhole-to-panoramic adaptation.

Title: Generalized Consistency Trajectory Models for Image Manipulation

• Authors: Beomsu Kim, Jaemin Kim, Jeongsol Kim, Jong Chul Ye
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12510
• Pdf URL: https://arxiv.org/pdf/2403.12510
• Copy Paste: [[2403.12510]] Generalized Consistency Trajectory Models for Image Manipulation(https://arxiv.org/abs/2403.12510)
• Keywords: diffusion, generative
• Abstract: Diffusion-based generative models excel in unconditional generation, as well as on applied tasks such as image editing and restoration. The success of diffusion models lies in the iterative nature of diffusion: diffusion breaks down the complex process of mapping noise to data into a sequence of simple denoising tasks. Moreover, we are able to exert fine-grained control over the generation process by injecting guidance terms into each denoising step. However, the iterative process is also computationally intensive, often taking from tens up to thousands of function evaluations. Although consistency trajectory models (CTMs) enable traversal between any time points along the probability flow ODE (PFODE) and score inference with a single function evaluation, CTMs only allow translation from Gaussian noise to data. Thus, this work aims to unlock the full potential of CTMs by proposing generalized CTMs (GCTMs), which translate between arbitrary distributions via ODEs. We discuss the design space of GCTMs and demonstrate their efficacy in various image manipulation tasks such as image-to-image translation, restoration, and editing. Code: \url{https://github.com/1202kbs/GCTM}

Title: GraphERE: Jointly Multiple Event-Event Relation Extraction via Graph-Enhanced Event Embeddings

• Authors: Haochen Li, Di Geng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12523
• Pdf URL: https://arxiv.org/pdf/2403.12523
• Copy Paste: [[2403.12523]] GraphERE: Jointly Multiple Event-Event Relation Extraction via Graph-Enhanced Event Embeddings(https://arxiv.org/abs/2403.12523)
• Keywords: extraction, transformer
• Abstract: Events describe the state changes of entities. In a document, multiple events are connected by various relations (e.g., Coreference, Temporal, Causal, and Subevent). Therefore, obtaining the connections between events through Event-Event Relation Extraction (ERE) is critical to understand natural language. There are two main problems in the current ERE works: a. Only embeddings of the event triggers are used for event feature representation, ignoring event arguments (e.g., time, place, person, etc.) and their structure within the event. b. The interconnection between relations (e.g., temporal and causal relations usually interact with each other ) is ignored. To solve the above problems, this paper proposes a jointly multiple ERE framework called GraphERE based on Graph-enhanced Event Embeddings. First, we enrich the event embeddings with event argument and structure features by using static AMR graphs and IE graphs; Then, to jointly extract multiple event relations, we use Node Transformer and construct Task-specific Dynamic Event Graphs for each type of relation. Finally, we used a multi-task learning strategy to train the whole framework. Experimental results on the latest MAVEN-ERE dataset validate that GraphERE significantly outperforms existing methods. Further analyses indicate the effectiveness of the graph-enhanced event embeddings and the joint extraction strategy.

Title: Prompt-based Graph Model for Joint Liberal Event Extraction and Event Schema Induction

• Authors: Haochen Li, Di Geng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12526
• Pdf URL: https://arxiv.org/pdf/2403.12526
• Copy Paste: [[2403.12526]] Prompt-based Graph Model for Joint Liberal Event Extraction and Event Schema Induction(https://arxiv.org/abs/2403.12526)
• Keywords: extraction
• Abstract: Events are essential components of speech and texts, describing the changes in the state of entities. The event extraction task aims to identify and classify events and find their participants according to event schemas. Manually predefined event schemas have limited coverage and are hard to migrate across domains. Therefore, the researchers propose Liberal Event Extraction (LEE), which aims to extract events and discover event schemas simultaneously. However, existing LEE models rely heavily on external language knowledge bases and require the manual development of numerous rules for noise removal and knowledge alignment, which is complex and laborious. To this end, we propose a Prompt-based Graph Model for Liberal Event Extraction (PGLEE). Specifically, we use a prompt-based model to obtain candidate triggers and arguments, and then build heterogeneous event graphs to encode the structures within and between events. Experimental results prove that our approach achieves excellent performance with or without predefined event schemas, while the automatically detected event schemas are proven high quality.

Title: PCT: Perspective Cue Training Framework for Multi-Camera BEV Segmentation

• Authors: Haruya Ishikawa, Takumi Iida, Yoshinori Konishi, Yoshimitsu Aoki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12530
• Pdf URL: https://arxiv.org/pdf/2403.12530
• Copy Paste: [[2403.12530]] PCT: Perspective Cue Training Framework for Multi-Camera BEV Segmentation(https://arxiv.org/abs/2403.12530)
• Keywords: segmentation
• Abstract: Generating annotations for bird's-eye-view (BEV) segmentation presents significant challenges due to the scenes' complexity and the high manual annotation cost. In this work, we address these challenges by leveraging the abundance of unlabeled data available. We propose the Perspective Cue Training (PCT) framework, a novel training framework that utilizes pseudo-labels generated from unlabeled perspective images using publicly available semantic segmentation models trained on large street-view datasets. PCT applies a perspective view task head to the image encoder shared with the BEV segmentation head, effectively utilizing the unlabeled data to be trained with the generated pseudo-labels. Since image encoders are present in nearly all camera-based BEV segmentation architectures, PCT is flexible and applicable to various existing BEV architectures. PCT can be applied to various settings where unlabeled data is available. In this paper, we applied PCT for semi-supervised learning (SSL) and unsupervised domain adaptation (UDA). Additionally, we introduce strong input perturbation through Camera Dropout (CamDrop) and feature perturbation via BEV Feature Dropout (BFD), which are crucial for enhancing SSL capabilities using our teacher-student framework. Our comprehensive approach is simple and flexible but yields significant improvements over various baselines for SSL and UDA, achieving competitive performances even against the current state-of-the-art.

Title: UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All

• Authors: Yuanhuiyi Lyu, Xu Zheng, Jiazhou Zhou, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12532
• Pdf URL: https://arxiv.org/pdf/2403.12532
• Copy Paste: [[2403.12532]] UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All(https://arxiv.org/abs/2403.12532)
• Keywords: large language model
• Abstract: We present UniBind, a flexible and efficient approach that learns a unified representation space for seven diverse modalities -- images, text, audio, point cloud, thermal, video, and event data. Existing works, eg., ImageBind, treat the image as the central modality and build an image-centered representation space; however, the space may be sub-optimal as it leads to an unbalanced representation space among all modalities. Moreover, the category names are directly used to extract text embeddings for the downstream tasks, making it hardly possible to represent the semantics of multi-modal data. The 'out-of-the-box' insight of our UniBind is to make the alignment center modality-agnostic and further learn a unified and balanced representation space, empowered by the large language models (LLMs). UniBind is superior in its flexible application to all CLIP-style models and delivers remarkable performance boosts. To make this possible, we 1) construct a knowledge base of text embeddings with the help of LLMs and multi-modal LLMs; 2) adaptively build LLM-augmented class-wise embedding center on top of the knowledge base and encoded visual embeddings; 3) align all the embeddings to the LLM-augmented embedding center via contrastive learning to achieve a unified and balanced representation space. UniBind shows strong zero-shot recognition performance gains over prior arts by an average of 6.36%. Finally, we achieve new state-of-the-art performance, eg., a 6.75% gain on ImageNet, on the multi-modal fine-tuning setting while reducing 90% of the learnable parameters.

Title: ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More

• Authors: Jiazhou Zhou, Xu Zheng, Yuanhuiyi Lyu, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12534
• Pdf URL: https://arxiv.org/pdf/2403.12534
• Copy Paste: [[2403.12534]] ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More(https://arxiv.org/abs/2403.12534)
• Keywords: privacy
• Abstract: Event cameras have recently been shown beneficial for practical vision tasks, such as action recognition, thanks to their high temporal resolution, power efficiency, and reduced privacy concerns. However, current research is hindered by 1) the difficulty in processing events because of their prolonged duration and dynamic actions with complex and ambiguous semantics and 2) the redundant action depiction of the event frame representation with fixed stacks. We find language naturally conveys abundant semantic information, rendering it stunningly superior in reducing semantic uncertainty. In light of this, we propose ExACT, a novel approach that, for the first time, tackles event-based action recognition from a cross-modal conceptualizing perspective. Our ExACT brings two technical contributions. Firstly, we propose an adaptive fine-grained event (AFE) representation to adaptively filter out the repeated events for the stationary objects while preserving dynamic ones. This subtly enhances the performance of ExACT without extra computational cost. Then, we propose a conceptual reasoning-based uncertainty estimation module, which simulates the recognition process to enrich the semantic representation. In particular, conceptual reasoning builds the temporal relation based on the action semantics, and uncertainty estimation tackles the semantic uncertainty of actions based on the distributional representation. Experiments show that our ExACT achieves superior recognition accuracy of 94.83%(+2.23%), 90.10%(+37.47%) and 67.24% on PAF, HARDVS and our SeAct datasets respectively.

Title: Vox-Fusion++: Voxel-based Neural Implicit Dense Tracking and Mapping with Multi-maps

• Authors: Hongjia Zhai, Hai Li, Xingrui Yang, Gan Huang, Yuhang Ming, Hujun Bao, Guofeng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12536
• Pdf URL: https://arxiv.org/pdf/2403.12536
• Copy Paste: [[2403.12536]] Vox-Fusion++: Voxel-based Neural Implicit Dense Tracking and Mapping with Multi-maps(https://arxiv.org/abs/2403.12536)
• Keywords: robust
• Abstract: In this paper, we introduce Vox-Fusion++, a multi-maps-based robust dense tracking and mapping system that seamlessly fuses neural implicit representations with traditional volumetric fusion techniques. Building upon the concept of implicit mapping and positioning systems, our approach extends its applicability to real-world scenarios. Our system employs a voxel-based neural implicit surface representation, enabling efficient encoding and optimization of the scene within each voxel. To handle diverse environments without prior knowledge, we incorporate an octree-based structure for scene division and dynamic expansion. To achieve real-time performance, we propose a high-performance multi-process framework. This ensures the system's suitability for applications with stringent time constraints. Additionally, we adopt the idea of multi-maps to handle large-scale scenes, and leverage loop detection and hierarchical pose optimization strategies to reduce long-term pose drift and remove duplicate geometry. Through comprehensive evaluations, we demonstrate that our method outperforms previous methods in terms of reconstruction quality and accuracy across various scenarios. We also show that our Vox-Fusion++ can be used in augmented reality and collaborative mapping applications. Our source code will be publicly available at \url{https://github.com/zju3dv/Vox-Fusion_Plus_Plus}

Title: Prompt-Guided Adaptive Model Transformation for Whole Slide Image Classification

• Authors: Yi Lin, Zhengjie Zhu, Kwang-Ting Cheng, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12537
• Pdf URL: https://arxiv.org/pdf/2403.12537
• Copy Paste: [[2403.12537]] Prompt-Guided Adaptive Model Transformation for Whole Slide Image Classification(https://arxiv.org/abs/2403.12537)
• Keywords: extraction
• Abstract: Multiple instance learning (MIL) has emerged as a popular method for classifying histopathology whole slide images (WSIs). Existing approaches typically rely on frozen pre-trained models to extract instance features, neglecting the substantial domain shift between pre-training natural and histopathological images. To address this issue, we propose PAMT, a novel Prompt-guided Adaptive Model Transformation framework that enhances MIL classification performance by seamlessly adapting pre-trained models to the specific characteristics of histopathology data. To capture the intricate histopathology distribution, we introduce Representative Patch Sampling (RPS) and Prototypical Visual Prompt (PVP) to reform the input data, building a compact while informative representation. Furthermore, to narrow the domain gap, we introduce Adaptive Model Transformation (AMT) that integrates adapter blocks within the feature extraction pipeline, enabling the pre-trained models to learn domain-specific features. We rigorously evaluate our approach on two publicly available datasets, Camelyon16 and TCGA-NSCLC, showcasing substantial improvements across various MIL models. Our findings affirm the potential of PAMT to set a new benchmark in WSI classification, underscoring the value of a targeted reprogramming approach.

Title: TAGS: Real-time Intrusion Detection with Tag-Propagation-based Provenance Graph Alignment on Streaming Events

• Authors: Zhenyuan Li, Yangyang Wei, Xiangmin Shen, Lingzhi Wang, Yan Chen, Haitao Xu, Shouling Ji, Fan Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.12541
• Pdf URL: https://arxiv.org/pdf/2403.12541
• Copy Paste: [[2403.12541]] TAGS: Real-time Intrusion Detection with Tag-Propagation-based Provenance Graph Alignment on Streaming Events(https://arxiv.org/abs/2403.12541)
• Keywords: security, attack, interpretability
• Abstract: The evolution and advancement of cyberattacks pose challenges to existing security products. Recent concentrated research on provenance graph-based detection has proved its effectiveness in attack detection and investigation. However, implementing these approaches in practice encounters challenges such as high overhead, slow responsiveness, and low interpretability and extensibility. Towards practical attack detection and investigation with provenance graphs, we propose TAGS, a tag-propagation-based streaming provenance graph alignment system. Utilizing the tag-based intermediate result caching mechanism alongside carefully crafted propagation rules, we eliminate the need to store and duplicate raw data processing. This approach effectively mitigates in-memory storage requirements and minimizes data processing overhead, facilitating rapid on-stream graph alignment while significantly conserving CPU and memory resources. As a result, TAGS can detect and investigate various cyber-attacks in real-time. Moreover, TAGS allows analysts to customize attack query graphs flexibly to identify extended attacks in data streams. We conduct experimental evaluations on two large-scale public datasets containing 257.42 GB of audit logs with 12 relevant query graphs of varying sizes, covering multiple attack techniques and scenarios. The results demonstrate that TAGS is sufficiently efficient to process 176K events per second while sufficiently accurately identifying all 29 alignments in massive data. Moreover, it can effectively handle the dependency explosion problem with steady, low-level memory consumption (less than 300MB), producing only 3 false positives. Overall, the performance of TAGS significantly outperforms the state-of-the-art methods.

Title: M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving

• Authors: Dongyang Xu, Haokun Li, Qingfan Wang, Ziying Song, Lei Chen, Hanming Deng
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.12552
• Pdf URL: https://arxiv.org/pdf/2403.12552
• Copy Paste: [[2403.12552]] M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving(https://arxiv.org/abs/2403.12552)
• Keywords: transformer
• Abstract: End-to-end autonomous driving has witnessed remarkable progress. However, the extensive deployment of autonomous vehicles has yet to be realized, primarily due to 1) inefficient multi-modal environment perception: how to integrate data from multi-modal sensors more efficiently; 2) non-human-like scene understanding: how to effectively locate and predict critical risky agents in traffic scenarios like an experienced driver. To overcome these challenges, in this paper, we propose a Multi-Modal fusion transformer incorporating Driver Attention (M2DA) for autonomous driving. To better fuse multi-modal data and achieve higher alignment between different modalities, a novel Lidar-Vision-Attention-based Fusion (LVAFusion) module is proposed. By incorporating driver attention, we empower the human-like scene understanding ability to autonomous vehicles to identify crucial areas within complex scenarios precisely and ensure safety. We conduct experiments on the CARLA simulator and achieve state-of-the-art performance with less data in closed-loop benchmarks. Source codes are available at https://anonymous.4open.science/r/M2DA-4772.

Title: Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation

• Authors: Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, Guoqing Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12556
• Pdf URL: https://arxiv.org/pdf/2403.12556
• Copy Paste: [[2403.12556]] Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation(https://arxiv.org/abs/2403.12556)
• Keywords: large language model
• Abstract: Previous Sign Language Translation (SLT) methods achieve superior performance by relying on gloss annotations. However, labeling high-quality glosses is a labor-intensive task, which limits the further development of SLT. Although some approaches work towards gloss-free SLT through jointly training the visual encoder and translation network, these efforts still suffer from poor performance and inefficient use of the powerful Large Language Model (LLM). Most seriously, we find that directly introducing LLM into SLT will lead to insufficient learning of visual representations as LLM dominates the learning curve. To address these problems, we propose Factorized Learning assisted with Large Language Model (FLa-LLM) for gloss-free SLT. Concretely, we factorize the training process into two stages. In the visual initialing stage, we employ a lightweight translation model after the visual encoder to pre-train the visual encoder. In the LLM fine-tuning stage, we freeze the acquired knowledge in the visual encoder and integrate it with a pre-trained LLM to inspire the LLM's translation potential. This factorized training strategy proves to be highly effective as evidenced by significant improvements achieved across three SLT datasets which are all conducted under the gloss-free setting.

Title: Simple Hack for Transformers against Heavy Long-Text Classification on a Time- and Memory-Limited GPU Service

• Authors: Mirza Alim Mutasodirin, Radityo Eko Prasojo, Achmad F. Abka, Hanif Rasyidi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12563
• Pdf URL: https://arxiv.org/pdf/2403.12563
• Copy Paste: [[2403.12563]] Simple Hack for Transformers against Heavy Long-Text Classification on a Time- and Memory-Limited GPU Service(https://arxiv.org/abs/2403.12563)
• Keywords: fair, transformer
• Abstract: Many NLP researchers rely on free computational services, such as Google Colab, to fine-tune their Transformer models, causing a limitation for hyperparameter optimization (HPO) in long-text classification due to the method having quadratic complexity and needing a bigger resource. In Indonesian, only a few works were found on long-text classification using Transformers. Most only use a small amount of data and do not report any HPO. In this study, using 18k news articles, we investigate which pretrained models are recommended to use based on the output length of the tokenizer. We then compare some hacks to shorten and enrich the sequences, which are the removals of stopwords, punctuation, low-frequency words, and recurring words. To get a fair comparison, we propose and run an efficient and dynamic HPO procedure that can be done gradually on a limited resource and does not require a long-running optimization library. Using the best hack found, we then compare 512, 256, and 128 tokens length. We find that removing stopwords while keeping punctuation and low-frequency words is the best hack. Some of our setups manage to outperform taking 512 first tokens using a smaller 128 or 256 first tokens which manage to represent the same information while requiring less computational resources. The findings could help developers to efficiently pursue optimal performance of the models using limited resources.

Title: Memory-Efficient and Secure DNN Inference on TrustZone-enabled Consumer IoT Devices

• Authors: Xueshuo Xie, Haoxu Wang, Zhaolong Jian, Tao Li, Wei Wang, Zhiwei Xu, Guiling Wang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12568
• Pdf URL: https://arxiv.org/pdf/2403.12568
• Copy Paste: [[2403.12568]] Memory-Efficient and Secure DNN Inference on TrustZone-enabled Consumer IoT Devices(https://arxiv.org/abs/2403.12568)
• Keywords: secure, security, privacy
• Abstract: Edge intelligence enables resource-demanding Deep Neural Network (DNN) inference without transferring original data, addressing concerns about data privacy in consumer Internet of Things (IoT) devices. For privacy-sensitive applications, deploying models in hardware-isolated trusted execution environments (TEEs) becomes essential. However, the limited secure memory in TEEs poses challenges for deploying DNN inference, and alternative techniques like model partitioning and offloading introduce performance degradation and security issues. In this paper, we present a novel approach for advanced model deployment in TrustZone that ensures comprehensive privacy preservation during model inference. We design a memory-efficient management method to support memory-demanding inference in TEEs. By adjusting the memory priority, we effectively mitigate memory leakage risks and memory overlap conflicts, resulting in 32 lines of code alterations in the trusted operating system. Additionally, we leverage two tiny libraries: S-Tinylib (2,538 LoCs), a tiny deep learning library, and Tinylibm (827 LoCs), a tiny math library, to support efficient inference in TEEs. We implemented a prototype on Raspberry Pi 3B+ and evaluated it using three well-known lightweight DNN models. The experimental results demonstrate that our design significantly improves inference speed by 3.13 times and reduces power consumption by over 66.5% compared to non-memory optimization method in TEEs.

Title: Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images

• Authors: Chaoqin Huang, Aofan Jiang, Jinghao Feng, Ya Zhang, Xinchao Wang, Yanfeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12570
• Pdf URL: https://arxiv.org/pdf/2403.12570
• Copy Paste: [[2403.12570]] Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images(https://arxiv.org/abs/2403.12570)
• Keywords: segmentation
• Abstract: Recent advancements in large-scale visual-language pre-trained models have led to significant progress in zero-/few-shot anomaly detection within natural image domains. However, the substantial domain divergence between natural and medical images limits the effectiveness of these methodologies in medical anomaly detection. This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. This multi-level adaptation is guided by multi-level, pixel-wise visual-language feature alignment loss functions, which recalibrate the model's focus from object semantics in natural imagery to anomaly identification in medical images. The adapted features exhibit improved generalization across various medical data types, even in zero-shot scenarios where the model encounters unseen medical modalities and anatomical regions during training. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models, with an average AUC improvement of 6.24% and 7.33% for anomaly classification, 2.03% and 2.37% for anomaly segmentation, under the zero-shot and few-shot settings, respectively. Source code is available at: https://github.com/MediaBrain-SJTU/MVFA-AD

Title: Compound Expression Recognition via Multi Model Ensemble

• Authors: Jun Yu, Jichao Zhu, Wangyuan Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12572
• Pdf URL: https://arxiv.org/pdf/2403.12572
• Copy Paste: [[2403.12572]] Compound Expression Recognition via Multi Model Ensemble(https://arxiv.org/abs/2403.12572)
• Keywords: transformer
• Abstract: Compound Expression Recognition (CER) plays a crucial role in interpersonal interactions. Due to the existence of Compound Expressions , human emotional expressions are complex, requiring consideration of both local and global facial expressions to make judgments. In this paper, to address this issue, we propose a solution based on ensemble learning methods for Compound Expression Recognition. Specifically, our task is classification, where we train three expression classification models based on convolutional networks, Vision Transformers, and multi-scale local attention networks. Then, through model ensemble using late fusion, we merge the outputs of multiple models to predict the final result. Our method achieves high accuracy on RAF-DB and is able to recognize expressions through zero-shot on certain portions of C-EXPR-DB.

Title: Lifting Multi-View Detection and Tracking to the Bird's Eye View

• Authors: Torben Teepe, Philipp Wolters, Johannes Gilg, Fabian Herzog, Gerhard Rigoll
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12573
• Pdf URL: https://arxiv.org/pdf/2403.12573
• Copy Paste: [[2403.12573]] Lifting Multi-View Detection and Tracking to the Bird's Eye View(https://arxiv.org/abs/2403.12573)
• Keywords: robust
• Abstract: Taking advantage of multi-view aggregation presents a promising solution to tackle challenges such as occlusion and missed detection in multi-object tracking and detection. Recent advancements in multi-view detection and 3D object recognition have significantly improved performance by strategically projecting all views onto the ground plane and conducting detection analysis from a Bird's Eye View. In this paper, we compare modern lifting methods, both parameter-free and parameterized, to multi-view aggregation. Additionally, we present an architecture that aggregates the features of multiple times steps to learn robust detection and combines appearance- and motion-based cues for tracking. Most current tracking approaches either focus on pedestrians or vehicles. In our work, we combine both branches and add new challenges to multi-view detection with cross-scene setups. Our method generalizes to three public datasets across two domains: (1) pedestrian: Wildtrack and MultiviewX, and (2) roadside perception: Synthehicle, achieving state-of-the-art performance in detection and tracking. https://github.com/tteepe/TrackTacular

Title: AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework

• Authors: Xiang Li, Zhenyu Li, Chen Shi, Yong Xu, Qing Du, Mingkui Tan, Jun Huang, Wei Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12582
• Pdf URL: https://arxiv.org/pdf/2403.12582
• Copy Paste: [[2403.12582]] AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework(https://arxiv.org/abs/2403.12582)
• Keywords: interpretability, large language model
• Abstract: The task of financial analysis primarily encompasses two key areas: stock trend prediction and the corresponding financial question answering. Currently, machine learning and deep learning algorithms (ML&DL) have been widely applied for stock trend predictions, leading to significant progress. However, these methods fail to provide reasons for predictions, lacking interpretability and reasoning processes. Also, they can not integrate textual information such as financial news or reports. Meanwhile, large language models (LLMs) have remarkable textual understanding and generation ability. But due to the scarcity of financial training datasets and limited integration with real-time knowledge, LLMs still suffer from hallucinations and are unable to keep up with the latest information. To tackle these challenges, we first release AlphaFin datasets, combining traditional research datasets, real-time financial data, and handwritten chain-of-thought (CoT) data. It has a positive impact on training LLMs for completing financial analysis. We then use AlphaFin datasets to benchmark a state-of-the-art method, called Stock-Chain, for effectively tackling the financial analysis task, which integrates retrieval-augmented generation (RAG) techniques. Extensive experiments are conducted to demonstrate the effectiveness of our framework on financial analysis.

Title: LASPA: Latent Spatial Alignment for Fast Training-free Single Image Editing

• Authors: Yazeed Alharbi, Peter Wonka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12585
• Pdf URL: https://arxiv.org/pdf/2403.12585
• Copy Paste: [[2403.12585]] LASPA: Latent Spatial Alignment for Fast Training-free Single Image Editing(https://arxiv.org/abs/2403.12585)
• Keywords: diffusion
• Abstract: We present a novel, training-free approach for textual editing of real images using diffusion models. Unlike prior methods that rely on computationally expensive finetuning, our approach leverages LAtent SPatial Alignment (LASPA) to efficiently preserve image details. We demonstrate how the diffusion process is amenable to spatial guidance using a reference image, leading to semantically coherent edits. This eliminates the need for complex optimization and costly model finetuning, resulting in significantly faster editing compared to previous methods. Additionally, our method avoids the storage requirements associated with large finetuned models. These advantages make our approach particularly well-suited for editing on mobile devices and applications demanding rapid response times. While simple and fast, our method achieves 62-71\% preference in a user-study and significantly better model-based editing strength and image preservation scores.

Title: LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models

• Authors: Chuang Liu, Renren Jin, Yuqi Ren, Deyi Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12601
• Pdf URL: https://arxiv.org/pdf/2403.12601
• Copy Paste: [[2403.12601]] LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models(https://arxiv.org/abs/2403.12601)
• Keywords: large language model
• Abstract: Chinese Large Language Models (LLMs) have recently demonstrated impressive capabilities across various NLP benchmarks and real-world applications. However, the existing benchmarks for comprehensively evaluating these LLMs are still insufficient, particularly in terms of measuring knowledge that LLMs capture. Current datasets collect questions from Chinese examinations across different subjects and educational levels to address this issue. Yet, these benchmarks primarily focus on objective questions such as multiple-choice questions, leading to a lack of diversity in question types. To tackle this problem, we propose LHMKE, a Large-scale, Holistic, and Multi-subject Knowledge Evaluation benchmark in this paper. LHMKE is designed to provide a comprehensive evaluation of the knowledge acquisition capabilities of Chinese LLMs. It encompasses 10,465 questions across 75 tasks covering 30 subjects, ranging from primary school to professional certification exams. Notably, LHMKE includes both objective and subjective questions, offering a more holistic evaluation of the knowledge level of LLMs. We have assessed 11 Chinese LLMs under the zero-shot setting, which aligns with real examinations, and compared their performance across different subjects. We also conduct an in-depth analysis to check whether GPT-4 can automatically score subjective predictions. Our findings suggest that LHMKE is a challenging and advanced testbed for Chinese LLMs.

Title: On the Effectiveness of Heterogeneous Ensemble Methods for Re-identification

• Authors: Simon Klüttermann, Jérôme Rutinowski, Anh Nguyen, Britta Grimme, Moritz Roidl, Emmanuel Müller
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12606
• Pdf URL: https://arxiv.org/pdf/2403.12606
• Copy Paste: [[2403.12606]] On the Effectiveness of Heterogeneous Ensemble Methods for Re-identification(https://arxiv.org/abs/2403.12606)
• Keywords: extraction
• Abstract: In this contribution, we introduce a novel ensemble method for the re-identification of industrial entities, using images of chipwood pallets and galvanized metal plates as dataset examples. Our algorithms replace commonly used, complex siamese neural networks with an ensemble of simplified, rudimentary models, providing wider applicability, especially in hardware-restricted scenarios. Each ensemble sub-model uses different types of extracted features of the given data as its input, allowing for the creation of effective ensembles in a fraction of the training duration needed for more complex state-of-the-art models. We reach state-of-the-art performance at our task, with a Rank-1 accuracy of over 77% and a Rank-10 accuracy of over 99%, and introduce five distinct feature extraction approaches, and study their combination using different ensemble methods.

Title: A Practical Guide to Statistical Distances for Evaluating Generative Models in Science

• Authors: Sebastian Bischoff, Alana Darcher, Michael Deistler, Richard Gao, Franziska Gerken, Manuel Gloeckler, Lisa Haxel, Jaivardhan Kapoor, Janne K Lappalainen, Jakob H Macke, Guy Moss, Matthijs Pals, Felix Pei, Rachel Rapp, A Erdem Sağtekin, Cornelius Schröder, Auguste Schulz, Zinovia Stefanidi, Shoji Toyota, Linda Ulmer, Julius Vetter
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12636
• Pdf URL: https://arxiv.org/pdf/2403.12636
• Copy Paste: [[2403.12636]] A Practical Guide to Statistical Distances for Evaluating Generative Models in Science(https://arxiv.org/abs/2403.12636)
• Keywords: generative
• Abstract: Generative models are invaluable in many fields of science because of their ability to capture high-dimensional and complicated distributions, such as photo-realistic images, protein structures, and connectomes. How do we evaluate the samples these models generate? This work aims to provide an accessible entry point to understanding popular notions of statistical distances, requiring only foundational knowledge in mathematics and statistics. We focus on four commonly used notions of statistical distances representing different methodologies: Using low-dimensional projections (Sliced-Wasserstein; SW), obtaining a distance using classifiers (Classifier Two-Sample Tests; C2ST), using embeddings through kernels (Maximum Mean Discrepancy; MMD), or neural networks (Fr\'echet Inception Distance; FID). We highlight the intuition behind each distance and explain their merits, scalability, complexity, and pitfalls. To demonstrate how these distances are used in practice, we evaluate generative models from different scientific domains, namely a model of decision making and a model generating medical images. We showcase that distinct distances can give different results on similar data. Through this guide, we aim to help researchers to use, interpret, and evaluate statistical distances for generative models in science.

Title: When Does Your Brain Know You? Segment Length and Its Impact on EEG-based Biometric Authentication Accuracy

• Authors: Nibras Abo Alzahab, Lorenzo Scalise, Marco Baldi
• Subjects: cs.CR, eess.SP, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2403.12644
• Pdf URL: https://arxiv.org/pdf/2403.12644
• Copy Paste: [[2403.12644]] When Does Your Brain Know You? Segment Length and Its Impact on EEG-based Biometric Authentication Accuracy(https://arxiv.org/abs/2403.12644)
• Keywords: secure, biometric
• Abstract: In the quest for optimal EEG-based biometric authentication, this study investigates the pivotal balance for accurate identification without sacrificing performance or adding unnecessary computational complexity. Through a methodical exploration of segment durations, and employing a variety of sophisticated machine learning models, the research seeks to pinpoint a threshold where EEG data provides maximum informational yield for authentication purposes. The findings are set to advance the field of non-invasive biometric technologies, proposing a practical approach to secure and user-friendly identity verification systems while also raising considerations for the real-world application of EEG-based biometric authentication beyond controlled environments.

Title: Tuning-Free Image Customization with Image and Text Guidance

• Authors: Pengzhi Li, Qiang Nie, Ying Chen, Xi Jiang, Kai Wu, Yuhuan Lin, Yong Liu, Jinlong Peng, Chengjie Wang, Feng Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12658
• Pdf URL: https://arxiv.org/pdf/2403.12658
• Copy Paste: [[2403.12658]] Tuning-Free Image Customization with Image and Text Guidance(https://arxiv.org/abs/2403.12658)
• Keywords: diffusion
• Abstract: Despite significant advancements in image customization with diffusion models, current methods still have several limitations: 1) unintended changes in non-target areas when regenerating the entire image; 2) guidance solely by a reference image or text descriptions; and 3) time-consuming fine-tuning, which limits their practical application. In response, we introduce a tuning-free framework for simultaneous text-image-guided image customization, enabling precise editing of specific image regions within seconds. Our approach preserves the semantic features of the reference image subject while allowing modification of detailed attributes based on text descriptions. To achieve this, we propose an innovative attention blending strategy that blends self-attention features in the UNet decoder during the denoising process. To our knowledge, this is the first tuning-free method that concurrently utilizes text and image guidance for image customization in specific regions. Our approach outperforms previous methods in both human and quantitative evaluations, providing an efficient solution for various practical applications, such as image synthesis, design, and creative photography.

Title: Deciphering AutoML Ensembles: cattleia's Assistance in Decision-Making

• Authors: Anna Kozak, Dominik Kędzierski, Jakub Piwko, Malwina Wojewoda, Katarzyna Woźnica
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12664
• Pdf URL: https://arxiv.org/pdf/2403.12664
• Copy Paste: [[2403.12664]] Deciphering AutoML Ensembles: cattleia's Assistance in Decision-Making(https://arxiv.org/abs/2403.12664)
• Keywords: interpretability
• Abstract: In many applications, model ensembling proves to be better than a single predictive model. Hence, it is the most common post-processing technique in Automated Machine Learning (AutoML). The most popular frameworks use ensembles at the expense of reducing the interpretability of the final models. In our work, we propose cattleia - an application that deciphers the ensembles for regression, multiclass, and binary classification tasks. This tool works with models built by three AutoML packages: auto-sklearn, AutoGluon, and FLAML. The given ensemble is analyzed from different perspectives. We conduct a predictive performance investigation through evaluation metrics of the ensemble and its component models. We extend the validation perspective by introducing new measures to assess the diversity and complementarity of the model predictions. Moreover, we apply explainable artificial intelligence (XAI) techniques to examine the importance of variables. Summarizing obtained insights, we can investigate and adjust the weights with a modification tool to tune the ensemble in the desired way. The application provides the aforementioned aspects through dedicated interactive visualizations, making it accessible to a diverse audience. We believe the cattleia can support users in decision-making and deepen the comprehension of AutoML frameworks.

Title: Enhancing Security of AI-Based Code Synthesis with GitHub Copilot via Cheap and Efficient Prompt-Engineering

• Authors: Jakub Res, Ivan Homoliak, Martin Perešíni, Aleš Smrčka, Kamil Malinka, Petr Hanacek
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12671
• Pdf URL: https://arxiv.org/pdf/2403.12671
• Copy Paste: [[2403.12671]] Enhancing Security of AI-Based Code Synthesis with GitHub Copilot via Cheap and Efficient Prompt-Engineering(https://arxiv.org/abs/2403.12671)
• Keywords: secure, security
• Abstract: AI assistants for coding are on the rise. However one of the reasons developers and companies avoid harnessing their full potential is the questionable security of the generated code. This paper first reviews the current state-of-the-art and identifies areas for improvement on this issue. Then, we propose a systematic approach based on prompt-altering methods to achieve better code security of (even proprietary black-box) AI-based code generators such as GitHub Copilot, while minimizing the complexity of the application from the user point-of-view, the computational resources, and operational costs. In sum, we propose and evaluate three prompt altering methods: (1) scenario-specific, (2) iterative, and (3) general clause, while we discuss their combination. Contrary to the audit of code security, the latter two of the proposed methods require no expert knowledge from the user. We assess the effectiveness of the proposed methods on the GitHub Copilot using the OpenVPN project in realistic scenarios, and we demonstrate that the proposed methods reduce the number of insecure generated code samples by up to 16\% and increase the number of secure code by up to 8\%. Since our approach does not require access to the internals of the AI models, it can be in general applied to any AI-based code synthesizer, not only GitHub Copilot.

Title: Improving Interpretability of Scores in Anomaly Detection Based on Gaussian-Bernoulli Restricted Boltzmann Machine

• Authors: Kaiji Sekimoto, Muneki Yasuda
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12672
• Pdf URL: https://arxiv.org/pdf/2403.12672
• Copy Paste: [[2403.12672]] Improving Interpretability of Scores in Anomaly Detection Based on Gaussian-Bernoulli Restricted Boltzmann Machine(https://arxiv.org/abs/2403.12672)
• Keywords: interpretability
• Abstract: Gaussian-Bernoulli restricted Boltzmann machines (GBRBMs) are often used for semi-supervised anomaly detection, where they are trained using only normal data points. In GBRBM-based anomaly detection, normal and anomalous data are classified based on a score that is identical to an energy function of the marginal GBRBM. However, the classification threshold is difficult to set to an appropriate value, as this score cannot be interpreted. In this study, we propose a measure that improves score's interpretability based on its cumulative distribution, and establish a guideline for setting the threshold using the interpretable measure. The results of numerical experiments show that the guideline is reasonable when setting the threshold solely using normal data points. Moreover, because identifying the measure involves computationally infeasible evaluation of the minimum score value, we also propose an evaluation method for the minimum score based on simulated annealing, which is widely used for optimization problems. The proposed evaluation method was also validated using numerical experiments.

Title: Pragmatic Competence Evaluation of Large Language Models for Korean

• Authors: Dojun Park, Jiwoo Lee, Hyeyun Jeong, Seohyun Park, Sungeun Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12675
• Pdf URL: https://arxiv.org/pdf/2403.12675
• Copy Paste: [[2403.12675]] Pragmatic Competence Evaluation of Large Language Models for Korean(https://arxiv.org/abs/2403.12675)
• Keywords: large language model
• Abstract: The current evaluation of Large Language Models (LLMs) predominantly relies on benchmarks focusing on their embedded knowledge by testing through multiple-choice questions (MCQs), a format inherently suited for automated evaluation. Our study extends this evaluation to explore LLMs' pragmatic competence--a facet previously underexamined before the advent of sophisticated LLMs, specifically in the context of Korean. We employ two distinct evaluation setups: the conventional MCQ format, adapted for automatic evaluation, and Open-Ended Questions (OEQs), assessed by human experts, to examine LLMs' narrative response capabilities without predefined options. Our findings reveal that GPT-4 excels, scoring 81.11 and 85.69 in the MCQ and OEQ setups, respectively, with HyperCLOVA X, an LLM optimized for Korean, closely following, especially in the OEQ setup, demonstrating a score of 81.56 with a marginal difference of 4.13 points compared to GPT-4. Furthermore, while few-shot learning strategies generally enhance LLM performance, Chain-of-Thought (CoT) prompting introduces a bias toward literal interpretations, hindering accurate pragmatic inference. Considering the growing expectation for LLMs to understand and produce language that aligns with human communicative norms, our findings emphasize the importance for advancing LLMs' abilities to grasp and convey sophisticated meanings beyond mere literal interpretations.

Title: WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar

• Authors: Runwei Guan, Liye Jia, Fengyufan Yang, Shanliang Yao, Erick Purwanto, Xiaohui Zhu, Eng Gee Lim, Jeremy Smith, Ka Lok Man, Yutao Yue
• Subjects: cs.CV, cs.MM, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.12686
• Pdf URL: https://arxiv.org/pdf/2403.12686
• Copy Paste: [[2403.12686]] WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar(https://arxiv.org/abs/2403.12686)
• Keywords: segmentation
• Abstract: The perception of waterways based on human intent holds significant importance for autonomous navigation and operations of Unmanned Surface Vehicles (USVs) in water environments. Inspired by visual grounding, in this paper, we introduce WaterVG, the first visual grounding dataset designed for USV-based waterway perception based on human intention prompts. WaterVG encompasses prompts describing multiple targets, with annotations at the instance level including bounding boxes and masks. Notably, WaterVG includes 11,568 samples with 34,950 referred targets, which integrates both visual and radar characteristics captured by monocular camera and millimeter-wave (mmWave) radar, enabling a finer granularity of text prompts. Furthermore, we propose a novel multi-modal visual grounding model, Potamoi, which is a multi-modal and multi-task model based on the one-stage paradigm with a designed Phased Heterogeneous Modality Fusion (PHMF) structure, including Adaptive Radar Weighting (ARW) and Multi-Head Slim Cross Attention (MHSCA). In specific, MHSCA is a low-cost and efficient fusion module with a remarkably small parameter count and FLOPs, elegantly aligning and fusing scenario context information captured by two sensors with linguistic features, which can effectively address tasks of referring expression comprehension and segmentation based on fine-grained prompts. Comprehensive experiments and evaluations have been conducted on WaterVG, where our Potamoi archives state-of-the-art performances compared with counterparts.

Title: SEVEN: Pruning Transformer Model by Reserving Sentinels

• Authors: Jinying Xiao, Ping Li, Jie Nie, Zhe Tang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12688
• Pdf URL: https://arxiv.org/pdf/2403.12688
• Copy Paste: [[2403.12688]] SEVEN: Pruning Transformer Model by Reserving Sentinels(https://arxiv.org/abs/2403.12688)
• Keywords: robust, transformer
• Abstract: Large-scale Transformer models (TM) have demonstrated outstanding performance across various tasks. However, their considerable parameter size restricts their applicability, particularly on mobile devices. Due to the dynamic and intricate nature of gradients on TM compared to Convolutional Neural Networks, commonly used pruning methods tend to retain weights with larger gradient noise. This results in pruned models that are sensitive to sparsity and datasets, exhibiting suboptimal performance. Symbolic Descent (SD) is a general approach for training and fine-tuning TM. In this paper, we attempt to describe the noisy batch gradient sequences on TM through the cumulative process of SD. We utilize this design to dynamically assess the importance scores of weights.SEVEN is introduced by us, which particularly favors weights with consistently high sensitivity, i.e., weights with small gradient noise. These weights are tended to be preserved by SEVEN. Extensive experiments on various TM in natural language, question-answering, and image classification domains are conducted to validate the effectiveness of SEVEN. The results demonstrate significant improvements of SEVEN in multiple pruning scenarios and across different sparsity levels. Additionally, SEVEN exhibits robust performance under various fine-tuning strategies. The code is publicly available at https://github.com/xiaojinying/SEVEN.

Title: As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?

• Authors: Anjun Hu, Jindong Gu, Francesco Pinto, Konstantinos Kamnitsas, Philip Torr
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12693
• Pdf URL: https://arxiv.org/pdf/2403.12693
• Copy Paste: [[2403.12693]] As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?(https://arxiv.org/abs/2403.12693)
• Keywords: attack, segmentation
• Abstract: Foundation models pre-trained on web-scale vision-language data, such as CLIP, are widely used as cornerstones of powerful machine learning systems. While pre-training offers clear advantages for downstream learning, it also endows downstream models with shared adversarial vulnerabilities that can be easily identified through the open-sourced foundation model. In this work, we expose such vulnerabilities in CLIP's downstream models and show that foundation models can serve as a basis for attacking their downstream systems. In particular, we propose a simple yet effective adversarial attack strategy termed Patch Representation Misalignment (PRM). Solely based on open-sourced CLIP vision encoders, this method produces adversaries that simultaneously fool more than 20 downstream models spanning 4 common vision-language tasks (semantic segmentation, object detection, image captioning and visual question-answering). Our findings highlight the concerning safety risks introduced by the extensive usage of public foundational models in the development of downstream systems, calling for extra caution in these scenarios.

Title: Learning Cross-view Visual Geo-localization without Ground Truth

• Authors: Haoyuan Li, Chang Xu, Wen Yang, Huai Yu, Gui-Song Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12702
• Pdf URL: https://arxiv.org/pdf/2403.12702
• Copy Paste: [[2403.12702]] Learning Cross-view Visual Geo-localization without Ground Truth(https://arxiv.org/abs/2403.12702)
• Keywords: robust
• Abstract: Cross-View Geo-Localization (CVGL) involves determining the geographical location of a query image by matching it with a corresponding GPS-tagged reference image. Current state-of-the-art methods predominantly rely on training models with labeled paired images, incurring substantial annotation costs and training burdens. In this study, we investigate the adaptation of frozen models for CVGL without requiring ground truth pair labels. We observe that training on unlabeled cross-view images presents significant challenges, including the need to establish relationships within unlabeled data and reconcile view discrepancies between uncertain queries and references. To address these challenges, we propose a self-supervised learning framework to train a learnable adapter for a frozen Foundation Model (FM). This adapter is designed to map feature distributions from diverse views into a uniform space using unlabeled data exclusively. To establish relationships within unlabeled data, we introduce an Expectation-Maximization-based Pseudo-labeling module, which iteratively estimates associations between cross-view features and optimizes the adapter. To maintain the robustness of the FM's representation, we incorporate an information consistency module with a reconstruction loss, ensuring that adapted features retain strong discriminative ability across views. Experimental results demonstrate that our proposed method achieves significant improvements over vanilla FMs and competitive accuracy compared to supervised methods, while necessitating fewer training parameters and relying solely on unlabeled data. Evaluation of our adaptation for task-specific models further highlights its broad applicability.

Title: AnimateDiff-Lightning: Cross-Model Diffusion Distillation

• Authors: Shanchuan Lin, Xiao Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12706
• Pdf URL: https://arxiv.org/pdf/2403.12706
• Copy Paste: [[2403.12706]] AnimateDiff-Lightning: Cross-Model Diffusion Distillation(https://arxiv.org/abs/2403.12706)
• Keywords: diffusion
• Abstract: We present AnimateDiff-Lightning for lightning-fast video generation. Our model uses progressive adversarial diffusion distillation to achieve new state-of-the-art in few-step video generation. We discuss our modifications to adapt it for the video modality. Furthermore, we propose to simultaneously distill the probability flow of multiple base diffusion models, resulting in a single distilled motion module with broader style compatibility. We are pleased to release our distilled AnimateDiff-Lightning model for the community's use.

Title: Selective Domain-Invariant Feature for Generalizable Deepfake Detection

• Authors: Yingxin Lai, Guoqing Yang Yifan He, Zhiming Luo, Shaozi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12707
• Pdf URL: https://arxiv.org/pdf/2403.12707
• Copy Paste: [[2403.12707]] Selective Domain-Invariant Feature for Generalizable Deepfake Detection(https://arxiv.org/abs/2403.12707)
• Keywords: extraction
• Abstract: With diverse presentation forgery methods emerging continually, detecting the authenticity of images has drawn growing attention. Although existing methods have achieved impressive accuracy in training dataset detection, they still perform poorly in the unseen domain and suffer from forgery of irrelevant information such as background and identity, affecting generalizability. To solve this problem, we proposed a novel framework Selective Domain-Invariant Feature (SDIF), which reduces the sensitivity to face forgery by fusing content features and styles. Specifically, we first use a Farthest-Point Sampling (FPS) training strategy to construct a task-relevant style sample representation space for fusing with content features. Then, we propose a dynamic feature extraction module to generate features with diverse styles to improve the performance and effectiveness of the feature extractor. Finally, a domain separation strategy is used to retain domain-related features to help distinguish between real and fake faces. Both qualitative and quantitative results in existing benchmarks and proposals demonstrate the effectiveness of our approach.

Title: Selective, Interpretable, and Motion Consistent Privacy Attribute Obfuscation for Action Recognition

• Authors: Filip Ilic, He Zhao, Thomas Pock, Richard P. Wildes
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12710
• Pdf URL: https://arxiv.org/pdf/2403.12710
• Copy Paste: [[2403.12710]] Selective, Interpretable, and Motion Consistent Privacy Attribute Obfuscation for Action Recognition(https://arxiv.org/abs/2403.12710)
• Keywords: privacy, interpretability
• Abstract: Concerns for the privacy of individuals captured in public imagery have led to privacy-preserving action recognition. Existing approaches often suffer from issues arising through obfuscation being applied globally and a lack of interpretability. Global obfuscation hides privacy sensitive regions, but also contextual regions important for action recognition. Lack of interpretability erodes trust in these new technologies. We highlight the limitations of current paradigms and propose a solution: Human selected privacy templates that yield interpretability by design, an obfuscation scheme that selectively hides attributes and also induces temporal consistency, which is important in action recognition. Our approach is architecture agnostic and directly modifies input imagery, while existing approaches generally require architecture training. Our approach offers more flexibility, as no retraining is required, and outperforms alternatives on three widely used datasets.

Title: Bilevel Hypergraph Networks for Multi-Modal Alzheimer's Diagnosis

• Authors: Angelica I. Aviles-Rivero, Chun-Wun Cheng, Zhongying Deng, Zoe Kourtzi, Carola-Bibiane Schönlieb
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12719
• Pdf URL: https://arxiv.org/pdf/2403.12719
• Copy Paste: [[2403.12719]] Bilevel Hypergraph Networks for Multi-Modal Alzheimer's Diagnosis(https://arxiv.org/abs/2403.12719)
• Keywords: robust
• Abstract: Early detection of Alzheimer's disease's precursor stages is imperative for significantly enhancing patient outcomes and quality of life. This challenge is tackled through a semi-supervised multi-modal diagnosis framework. In particular, we introduce a new hypergraph framework that enables higher-order relations between multi-modal data, while utilising minimal labels. We first introduce a bilevel hypergraph optimisation framework that jointly learns a graph augmentation policy and a semi-supervised classifier. This dual learning strategy is hypothesised to enhance the robustness and generalisation capabilities of the model by fostering new pathways for information propagation. Secondly, we introduce a novel strategy for generating pseudo-labels more effectively via a gradient-driven flow. Our experimental results demonstrate the superior performance of our framework over current techniques in diagnosing Alzheimer's disease.

Title: CLASSLA-web: Comparable Web Corpora of South Slavic Languages Enriched with Linguistic and Genre Annotation

• Authors: Nikola Ljubešić, Taja Kuzman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12721
• Pdf URL: https://arxiv.org/pdf/2403.12721
• Copy Paste: [[2403.12721]] CLASSLA-web: Comparable Web Corpora of South Slavic Languages Enriched with Linguistic and Genre Annotation(https://arxiv.org/abs/2403.12721)
• Keywords: transformer
• Abstract: This paper presents a collection of highly comparable web corpora of Slovenian, Croatian, Bosnian, Montenegrin, Serbian, Macedonian, and Bulgarian, covering thereby the whole spectrum of official languages in the South Slavic language space. The collection of these corpora comprises a total of 13 billion tokens of texts from 26 million documents. The comparability of the corpora is ensured by a comparable crawling setup and the usage of identical crawling and post-processing technology. All the corpora were linguistically annotated with the state-of-the-art CLASSLA-Stanza linguistic processing pipeline, and enriched with document-level genre information via the Transformer-based multilingual X-GENRE classifier, which further enhances comparability at the level of linguistic annotation and metadata enrichment. The genre-focused analysis of the resulting corpora shows a rather consistent distribution of genres throughout the seven corpora, with variations in the most prominent genre categories being well-explained by the economic strength of each language community. A comparison of the distribution of genre categories across the corpora indicates that web corpora from less developed countries primarily consist of news articles. Conversely, web corpora from economically more developed countries exhibit a smaller proportion of news content, with a greater presence of promotional and opinionated texts.

Title: Python Fuzzing for Trustworthy Machine Learning Frameworks

• Authors: Ilya Yegorov, Eli Kobrin, Darya Parygina, Alexey Vishnyakov, Andrey Fedotov
• Subjects: cs.CR, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.12723
• Pdf URL: https://arxiv.org/pdf/2403.12723
• Copy Paste: [[2403.12723]] Python Fuzzing for Trustworthy Machine Learning Frameworks(https://arxiv.org/abs/2403.12723)
• Keywords: secure, security, attack, robust
• Abstract: Ensuring the security and reliability of machine learning frameworks is crucial for building trustworthy AI-based systems. Fuzzing, a popular technique in secure software development lifecycle (SSDLC), can be used to develop secure and robust software. Popular machine learning frameworks such as PyTorch and TensorFlow are complex and written in multiple programming languages including C/C++ and Python. We propose a dynamic analysis pipeline for Python projects using the Sydr-Fuzz toolset. Our pipeline includes fuzzing, corpus minimization, crash triaging, and coverage collection. Crash triaging and severity estimation are important steps to ensure that the most critical vulnerabilities are addressed promptly. Furthermore, the proposed pipeline is integrated in GitLab CI. To identify the most vulnerable parts of the machine learning frameworks, we analyze their potential attack surfaces and develop fuzz targets for PyTorch, TensorFlow, and related projects such as h5py. Applying our dynamic analysis pipeline to these targets, we were able to discover 3 new bugs and propose fixes for them.

Title: Diffusion-Driven Self-Supervised Learning for Shape Reconstruction and Pose Estimation

• Authors: Jingtao Sun, Yaonan Wang, Mingtao Feng, Chao Ding, Mike Zheng Shou, Ajmal Saeed Mian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12728
• Pdf URL: https://arxiv.org/pdf/2403.12728
• Copy Paste: [[2403.12728]] Diffusion-Driven Self-Supervised Learning for Shape Reconstruction and Pose Estimation(https://arxiv.org/abs/2403.12728)
• Keywords: diffusion, transformer
• Abstract: Fully-supervised category-level pose estimation aims to determine the 6-DoF poses of unseen instances from known categories, requiring expensive mannual labeling costs. Recently, various self-supervised category-level pose estimation methods have been proposed to reduce the requirement of the annotated datasets. However, most methods rely on synthetic data or 3D CAD model for self-supervised training, and they are typically limited to addressing single-object pose problems without considering multi-objective tasks or shape reconstruction. To overcome these challenges and limitations, we introduce a diffusion-driven self-supervised network for multi-object shape reconstruction and categorical pose estimation, only leveraging the shape priors. Specifically, to capture the SE(3)-equivariant pose features and 3D scale-invariant shape information, we present a Prior-Aware Pyramid 3D Point Transformer in our network. This module adopts a point convolutional layer with radial-kernels for pose-aware learning and a 3D scale-invariant graph convolution layer for object-level shape representation, respectively. Furthermore, we introduce a pretrain-to-refine self-supervised training paradigm to train our network. It enables proposed network to capture the associations between shape priors and observations, addressing the challenge of intra-class shape variations by utilising the diffusion mechanism. Extensive experiments conducted on four public datasets and a self-built dataset demonstrate that our method significantly outperforms state-of-the-art self-supervised category-level baselines and even surpasses some fully-supervised instance-level and category-level methods.

Title: Towards Multimodal In-Context Learning for Vision & Language Models

• Authors: Sivan Doveh, Shaked Perek, M. Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, Leonid Karlinsky
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12736
• Pdf URL: https://arxiv.org/pdf/2403.12736
• Copy Paste: [[2403.12736]] Towards Multimodal In-Context Learning for Vision & Language Models(https://arxiv.org/abs/2403.12736)
• Keywords: large language model
• Abstract: Inspired by the emergence of Large Language Models (LLMs) that can truly understand human language, significant progress has been made in aligning other, non-language, modalities to be `understandable' by an LLM, primarily via converting their samples into a sequence of embedded language-like tokens directly fed into the LLM (decoder) input stream. However, so far limited attention has been given to transferring (and evaluating) one of the core LLM capabilities to the emerging VLMs, namely the In-Context Learning (ICL) ability, or in other words to guide VLMs to desired target downstream tasks or output structure using in-context image+text demonstrations. In this work, we dive deeper into analyzing the capabilities of some of the state-of-the-art VLMs to follow ICL instructions, discovering them to be somewhat lacking. We discover that even models that underwent large-scale mixed modality pre-training and were implicitly guided to make use of interleaved image and text information (intended to consume helpful context from multiple images) under-perform when prompted with few-shot (ICL) demonstrations, likely due to their lack of `direct' ICL instruction tuning. To test this conjecture, we propose a simple, yet surprisingly effective, strategy of extending a common VLM alignment framework with ICL support, methodology, and curriculum. We explore, analyze, and provide insights into effective data mixes, leading up to a significant 21.03% (and 11.3% on average) ICL performance boost over the strongest VLM baselines and a variety of ICL benchmarks. We also contribute new benchmarks for ICL evaluation in VLMs and discuss their advantages over the prior art.

Title: Towards Controllable Face Generation with Semantic Latent Diffusion Models

• Authors: Alex Ergasti, Claudio Ferrari, Tomaso Fontanini, Massimo Bertozzi, Andrea Prati
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12743
• Pdf URL: https://arxiv.org/pdf/2403.12743
• Copy Paste: [[2403.12743]] Towards Controllable Face Generation with Semantic Latent Diffusion Models(https://arxiv.org/abs/2403.12743)
• Keywords: diffusion
• Abstract: Semantic Image Synthesis (SIS) is among the most popular and effective techniques in the field of face generation and editing, thanks to its good generation quality and the versatility is brings along. Recent works attempted to go beyond the standard GAN-based framework, and started to explore Diffusion Models (DMs) for this task as these stand out with respect to GANs in terms of both quality and diversity. On the other hand, DMs lack in fine-grained controllability and reproducibility. To address that, in this paper we propose a SIS framework based on a novel Latent Diffusion Model architecture for human face generation and editing that is both able to reproduce and manipulate a real reference image and generate diversity-driven results. The proposed system utilizes both SPADE normalization and cross-attention layers to merge shape and style information and, by doing so, allows for a precise control over each of the semantic parts of the human face. This was not possible with previous methods in the state of the art. Finally, we performed an extensive set of experiments to prove that our model surpasses current state of the art, both qualitatively and quantitatively.

Title: Instructing Large Language Models to Identify and Ignore Irrelevant Conditions

• Authors: Zhenyu Wu, Chao Shen, Meng Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12744
• Pdf URL: https://arxiv.org/pdf/2403.12744
• Copy Paste: [[2403.12744]] Instructing Large Language Models to Identify and Ignore Irrelevant Conditions(https://arxiv.org/abs/2403.12744)
• Keywords: large language model
• Abstract: Math word problem (MWP) solving requires generating a reasoning path based on a given problem description that often contains irrelevant conditions. Existing chain-of-thought (CoT) prompting methods elicited multi-step reasoning abilities of large language models (LLMs) to solve MWPs. However, they were seriously confused by the irrelevant conditions, resulting in low accuracy. In this paper, we propose a novel approach named I$^3$C that instructs LLMs to identify and ignore irrelevant conditions. It identifies a set of irrelevant condition candidates that have a weak semantic relevance with the question. Then it prompts LLMs to verify the irrelevant conditions. Lastly it instructs the LLMs with the verification on relevant and irrelevant conditions to avoid confusion and improve reasoning paths. Moreover, we propose to select (problem, reasoning paths) pairs as demonstrations to enhance I$^3$C with few-shot reasoning. We develop I$^3$C-Select that selects the most confusing problems based on the semantic relevance measurement. We conduct extensive experiments on eight MWP datasets. I$^3$C can be combined with any CoT prompting methods to improve the performance of solving MWPs. Notably, with GPT-3.5-Turbo and I$^3$C-Select, we achieve an accuracy of 96.0 and 94.1 on GSM-IC2-1K and GSM-ICM-1K, respectively, significantly outperforming the state-of-the-art few-shot prompting method Complex-CoT by +11.7 and +11.1. Our implementation is made publicly available at https://wzy6642.github.io/I3C.github.io/.

Title: Building Brain Tumor Segmentation Networks with User-Assisted Filter Estimation and Selection

• Authors: Matheus A. Cerqueira, Flávia Sprenger, Bernardo C. A. Teixeira, Alexandre X. Falcão
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12748
• Pdf URL: https://arxiv.org/pdf/2403.12748
• Copy Paste: [[2403.12748]] Building Brain Tumor Segmentation Networks with User-Assisted Filter Estimation and Selection(https://arxiv.org/abs/2403.12748)
• Keywords: segmentation
• Abstract: Brain tumor image segmentation is a challenging research topic in which deep-learning models have presented the best results. However, the traditional way of training those models from many pre-annotated images leaves several unanswered questions. Hence methodologies, such as Feature Learning from Image Markers (FLIM), have involved an expert in the learning loop to reduce human effort in data annotation and build models sufficiently deep for a given problem. FLIM has been successfully used to create encoders, estimating the filters of all convolutional layers from patches centered at marker voxels. In this work, we present Multi-Step (MS) FLIM - a user-assisted approach to estimating and selecting the most relevant filters from multiple FLIM executions. MS-FLIM is used only for the first convolutional layer, and the results already indicate improvement over FLIM. For evaluation, we build a simple U-shaped encoder-decoder network, named sU-Net, for glioblastoma segmentation using T1Gd and FLAIR MRI scans, varying the encoder's training method, using FLIM, MS-FLIM, and backpropagation algorithm. Also, we compared these sU-Nets with two State-Of-The-Art (SOTA) deep-learning models using two datasets. The results show that the sU-Net based on MS-FLIM outperforms the other training methods and achieves effectiveness within the standard deviations of the SOTA models.

Title: WaveFace: Authentic Face Restoration with Efficient Frequency Recovery

• Authors: Yunqi Miao, Jiankang Deng, Jungong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12760
• Pdf URL: https://arxiv.org/pdf/2403.12760
• Copy Paste: [[2403.12760]] WaveFace: Authentic Face Restoration with Efficient Frequency Recovery(https://arxiv.org/abs/2403.12760)
• Keywords: diffusion
• Abstract: Although diffusion models are rising as a powerful solution for blind face restoration, they are criticized for two problems: 1) slow training and inference speed, and 2) failure in preserving identity and recovering fine-grained facial details. In this work, we propose WaveFace to solve the problems in the frequency domain, where low- and high-frequency components decomposed by wavelet transformation are considered individually to maximize authenticity as well as efficiency. The diffusion model is applied to recover the low-frequency component only, which presents general information of the original image but 1/16 in size. To preserve the original identity, the generation is conditioned on the low-frequency component of low-quality images at each denoising step. Meanwhile, high-frequency components at multiple decomposition levels are handled by a unified network, which recovers complex facial details in a single step. Evaluations on four benchmark datasets show that: 1) WaveFace outperforms state-of-the-art methods in authenticity, especially in terms of identity preservation, and 2) authentic images are restored with the efficiency 10x faster than existing diffusion model-based BFR methods.

Title: NovelQA: A Benchmark for Long-Range Novel Question Answering

• Authors: Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, Yue Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12766
• Pdf URL: https://arxiv.org/pdf/2403.12766
• Copy Paste: [[2403.12766]] NovelQA: A Benchmark for Long-Range Novel Question Answering(https://arxiv.org/abs/2403.12766)
• Keywords: large language model
• Abstract: The rapid advancement of Large Language Models (LLMs) has introduced a new frontier in natural language processing, particularly in understanding and processing long-context information. However, the evaluation of these models' long-context abilities remains a challenge due to the limitations of current benchmarks. To address this gap, we introduce NovelQA, a benchmark specifically designed to test the capabilities of LLMs with extended texts. Constructed from English novels, NovelQA offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding in LLMs. This paper presents the design and construction of NovelQA, highlighting its manual annotation, and diverse question types. Our evaluation of Long-context LLMs on NovelQA reveals significant insights into the models' performance, particularly emphasizing the challenges they face with multi-hop reasoning, detail-oriented questions, and extremely long input with more than 100,000 tokens. The results underscore the necessity for further advancements in LLMs to improve their long-context comprehension and computational literary studies.

Title: Inter- and intra-uncertainty based feature aggregation model for semi-supervised histopathology image segmentation

• Authors: Qiangguo Jin, Hui Cui, Changming Sun, Yang Song, Jiangbin Zheng, Leilei Cao, Leyi Wei, Ran Su
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12767
• Pdf URL: https://arxiv.org/pdf/2403.12767
• Copy Paste: [[2403.12767]] Inter- and intra-uncertainty based feature aggregation model for semi-supervised histopathology image segmentation(https://arxiv.org/abs/2403.12767)
• Keywords: segmentation
• Abstract: Acquiring pixel-level annotations is often limited in applications such as histology studies that require domain expertise. Various semi-supervised learning approaches have been developed to work with limited ground truth annotations, such as the popular teacher-student models. However, hierarchical prediction uncertainty within the student model (intra-uncertainty) and image prediction uncertainty (inter-uncertainty) have not been fully utilized by existing methods. To address these issues, we first propose a novel inter- and intra-uncertainty regularization method to measure and constrain both inter- and intra-inconsistencies in the teacher-student architecture. We also propose a new two-stage network with pseudo-mask guided feature aggregation (PG-FANet) as the segmentation model. The two-stage structure complements with the uncertainty regularization strategy to avoid introducing extra modules in solving uncertainties and the aggregation mechanisms enable multi-scale and multi-stage feature integration. Comprehensive experimental results over the MoNuSeg and CRAG datasets show that our PG-FANet outperforms other state-of-the-art methods and our semi-supervised learning framework yields competitive performance with a limited amount of labeled data.

Title: Automated Data Curation for Robust Language Model Fine-Tuning

• Authors: Jiuhai Chen, Jonas Mueller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12776
• Pdf URL: https://arxiv.org/pdf/2403.12776
• Copy Paste: [[2403.12776]] Automated Data Curation for Robust Language Model Fine-Tuning(https://arxiv.org/abs/2403.12776)
• Keywords: robust, large language model
• Abstract: Large Language Models have become the de facto approach to sequence-to-sequence text generation tasks, but for specialized tasks/domains, a pretrained LLM lacks specific capabilities to produce accurate or well-formatted responses. Supervised fine-tuning specializes a LLM by training it on dataset of example prompts with target responses, but real-world data tends to be noisy. While many fine-tuning algorithms exist, here we consider a \emph{data-centric AI} perspective on LLM fine-tuning, studying how to \emph{systematically} curate the training dataset to improve the LLM produced via \emph{any} fine-tuning algorithm. We introduce an automated data curation pipeline CLEAR (Confidence-based LLM Evaluation And Rectification) for instruction tuning datasets, that can be used with any LLM and fine-tuning procedure. CLEAR estimates which training data is low-quality and either filters or corrects it. Automatically identifying which data to filter or correct is done via LLM-derived confidence estimates, to ensure only confident modifications to the dataset. Unlike existing data curation techniques, CLEAR is a comprehensive framework that can improve a dataset (and trained model outputs) without additional fine-tuning computations. We don't assume access to a stronger LLM than the model being fine-tuned (e.g.\ relying on GPT-4 when fine-tuning GPT-3.5), to see whether CLEAR can meaningfully improve the capabilities of any LLM. Experiments reveal that CLEAR consistently improves the performance of fine-tuned models across many datasets and models (like GPT-3.5 and Llama2).

Title: Discover and Mitigate Multiple Biased Subgroups in Image Classifiers

• Authors: Zeliang Zhang, Mingqian Feng, Zhiheng Li, Chenliang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12777
• Pdf URL: https://arxiv.org/pdf/2403.12777
• Copy Paste: [[2403.12777]] Discover and Mitigate Multiple Biased Subgroups in Image Classifiers(https://arxiv.org/abs/2403.12777)
• Keywords: robust
• Abstract: Machine learning models can perform well on in-distribution data but often fail on biased subgroups that are underrepresented in the training data, hindering the robustness of models for reliable applications. Such subgroups are typically unknown due to the absence of subgroup labels. Discovering biased subgroups is the key to understanding models' failure modes and further improving models' robustness. Most previous works of subgroup discovery make an implicit assumption that models only underperform on a single biased subgroup, which does not hold on in-the-wild data where multiple biased subgroups exist. In this work, we propose Decomposition, Interpretation, and Mitigation (DIM), a novel method to address a more challenging but also more practical problem of discovering multiple biased subgroups in image classifiers. Our approach decomposes the image features into multiple components that represent multiple subgroups. This decomposition is achieved via a bilinear dimension reduction method, Partial Least Square (PLS), guided by useful supervision from the image classifier. We further interpret the semantic meaning of each subgroup component by generating natural language descriptions using vision-language foundation models. Finally, DIM mitigates multiple biased subgroups simultaneously via two strategies, including the data- and model-centric strategies. Extensive experiments on CIFAR-100 and Breeds datasets demonstrate the effectiveness of DIM in discovering and mitigating multiple biased subgroups. Furthermore, DIM uncovers the failure modes of the classifier on Hard ImageNet, showcasing its broader applicability to understanding model bias in image classifiers. The code is available at https://github.com/ZhangAIPI/DIM.

Title: ViTGaze: Gaze Following with Interaction Features in Vision Transformers

• Authors: Yuehao Song, Xinggang Wang, Jingfeng Yao, Wenyu Liu, Jinglin Zhang, Xiangmin Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12778
• Pdf URL: https://arxiv.org/pdf/2403.12778
• Copy Paste: [[2403.12778]] ViTGaze: Gaze Following with Interaction Features in Vision Transformers(https://arxiv.org/abs/2403.12778)
• Keywords: transformer
• Abstract: Gaze following aims to interpret human-scene interactions by predicting the person's focal point of gaze. Prevailing approaches often use multi-modality inputs, most of which adopt a two-stage framework. Hence their performance highly depends on the previous prediction accuracy. Others use a single-modality approach with complex decoders, increasing network computational load. Inspired by the remarkable success of pre-trained plain Vision Transformers (ViTs), we introduce a novel single-modality gaze following framework, ViTGaze. In contrast to previous methods, ViTGaze creates a brand new gaze following framework based mainly on powerful encoders (dec. param. less than 1%). Our principal insight lies in that the inter-token interactions within self-attention can be transferred to interactions between humans and scenes. Leveraging this presumption, we formulate a framework consisting of a 4D interaction encoder and a 2D spatial guidance module to extract human-scene interaction information from self-attention maps. Furthermore, our investigation reveals that ViT with self-supervised pre-training exhibits an enhanced ability to extract correlated information. A large number of experiments have been conducted to demonstrate the performance of the proposed method. Our method achieves state-of-the-art (SOTA) performance among all single-modality methods (3.4% improvement on AUC, 5.1% improvement on AP) and very comparable performance against multi-modality methods with 59% number of parameters less.

Title: DDSB: An Unsupervised and Training-free Method for Phase Detection in Echocardiography

• Authors: Zhenyu Bu, Yang Liu, Jiayu Huo, Jingjing Peng, Kaini Wang, Guangquan Zhou, Rachel Sparks, Prokar Dasgupta, Alejandro Granados, Sebastien Ourselin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12787
• Pdf URL: https://arxiv.org/pdf/2403.12787
• Copy Paste: [[2403.12787]] DDSB: An Unsupervised and Training-free Method for Phase Detection in Echocardiography(https://arxiv.org/abs/2403.12787)
• Keywords: robust, segmentation
• Abstract: Accurate identification of End-Diastolic (ED) and End-Systolic (ES) frames is key for cardiac function assessment through echocardiography. However, traditional methods face several limitations: they require extensive amounts of data, extensive annotations by medical experts, significant training resources, and often lack robustness. Addressing these challenges, we proposed an unsupervised and training-free method, our novel approach leverages unsupervised segmentation to enhance fault tolerance against segmentation inaccuracies. By identifying anchor points and analyzing directional deformation, we effectively reduce dependence on the accuracy of initial segmentation images and enhance fault tolerance, all while improving robustness. Tested on Echo-dynamic and CAMUS datasets, our method achieves comparable accuracy to learning-based models without their associated drawbacks. The code is available at https://github.com/MRUIL/DDSB

Title: Investigating Text Shortening Strategy in BERT: Truncation vs Summarization

• Authors: Mirza Alim Mutasodirin, Radityo Eko Prasojo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12799
• Pdf URL: https://arxiv.org/pdf/2403.12799
• Copy Paste: [[2403.12799]] Investigating Text Shortening Strategy in BERT: Truncation vs Summarization(https://arxiv.org/abs/2403.12799)
• Keywords: transformer
• Abstract: The parallelism of Transformer-based models comes at the cost of their input max-length. Some studies proposed methods to overcome this limitation, but none of them reported the effectiveness of summarization as an alternative. In this study, we investigate the performance of document truncation and summarization in text classification tasks. Each of the two was investigated with several variations. This study also investigated how close their performances are to the performance of full-text. We used a dataset of summarization tasks based on Indonesian news articles (IndoSum) to do classification tests. This study shows how the summaries outperform the majority of truncation method variations and lose to only one. The best strategy obtained in this study is taking the head of the document. The second is extractive summarization. This study explains what happened to the result, leading to further research in order to exploit the potential of document summarization as a shortening alternative. The code and data used in this work are publicly available in https://github.com/mirzaalimm/TruncationVsSummarization.

Title: RelationVLM: Making Large Vision-Language Models Understand Visual Relations

• Authors: Zhipeng Huang, Zhizheng Zhang, Zheng-Jun Zha, Yan Lu, Baining Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12801
• Pdf URL: https://arxiv.org/pdf/2403.12801
• Copy Paste: [[2403.12801]] RelationVLM: Making Large Vision-Language Models Understand Visual Relations(https://arxiv.org/abs/2403.12801)
• Keywords: large language model
• Abstract: The development of Large Vision-Language Models (LVLMs) is striving to catch up with the success of Large Language Models (LLMs), yet it faces more challenges to be resolved. Very recent works enable LVLMs to localize object-level visual contents and ground text to them. Nonetheless, current LVLMs still struggle to precisely understand visual relations due to the lack of relevant data. In this work, we present RelationVLM, a large vision-language model capable of comprehending various levels and types of relations whether across multiple images or within a video. Specifically, we devise a multi-stage relation-aware training scheme and a series of corresponding data configuration strategies to bestow RelationVLM with the capabilities of understanding semantic relations, temporal associations and geometric transforms. Extensive case studies and quantitative evaluations show RelationVLM has strong capability in understanding such relations and emerges impressive in-context capability of reasoning from few-shot examples by comparison. This work fosters the advancements of LVLMs by enabling them to support a wider range of downstream applications toward artificial general intelligence.

Title: DreamDA: Generative Data Augmentation with Diffusion Models

• Authors: Yunxiang Fu, Chaoqi Chen, Yu Qiao, Yizhou Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12803
• Pdf URL: https://arxiv.org/pdf/2403.12803
• Copy Paste: [[2403.12803]] DreamDA: Generative Data Augmentation with Diffusion Models(https://arxiv.org/abs/2403.12803)
• Keywords: diffusion, generative
• Abstract: The acquisition of large-scale, high-quality data is a resource-intensive and time-consuming endeavor. Compared to conventional Data Augmentation (DA) techniques (e.g. cropping and rotation), exploiting prevailing diffusion models for data generation has received scant attention in classification tasks. Existing generative DA methods either inadequately bridge the domain gap between real-world and synthesized images, or inherently suffer from a lack of diversity. To solve these issues, this paper proposes a new classification-oriented framework DreamDA, which enables data synthesis and label generation by way of diffusion models. DreamDA generates diverse samples that adhere to the original data distribution by considering training images in the original data as seeds and perturbing their reverse diffusion process. In addition, since the labels of the generated data may not align with the labels of their corresponding seed images, we introduce a self-training paradigm for generating pseudo labels and training classifiers using the synthesized data. Extensive experiments across four tasks and five datasets demonstrate consistent improvements over strong baselines, revealing the efficacy of DreamDA in synthesizing high-quality and diverse images with accurate labels. Our code will be available at https://github.com/yunxiangfu2001/DreamDA.

Title: The Emergence of Hardware Fuzzing: A Critical Review of its Significance

• Authors: Raghul Saravanan, Sai Manoj Pudukotai Dinakarrao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.12812
• Pdf URL: https://arxiv.org/pdf/2403.12812
• Copy Paste: [[2403.12812]] The Emergence of Hardware Fuzzing: A Critical Review of its Significance(https://arxiv.org/abs/2403.12812)
• Keywords: security
• Abstract: In recent years, there has been a notable surge in attention towards hardware security, driven by the increasing complexity and integration of processors, SoCs, and third-party IPs aimed at delivering advanced solutions. However, this complexity also introduces vulnerabilities and bugs into hardware systems, necessitating early detection during the IC design cycle to uphold system integrity and mitigate re-engineering costs. While the Design Verification (DV) community employs dynamic and formal verification strategies, they encounter challenges such as scalability for intricate designs and significant human intervention, leading to prolonged verification durations. As an alternative approach, hardware fuzzing, inspired by software testing methodologies, has gained prominence for its efficacy in identifying bugs within complex hardware designs. Despite the introduction of various hardware fuzzing techniques, obstacles such as inefficient conversion of hardware modules into software models impede their effectiveness. This Systematization of Knowledge (SoK) initiative delves into the fundamental principles of existing hardware fuzzing, methodologies, and their applicability across diverse hardware designs. Additionally, it evaluates factors such as the utilization of golden reference models (GRMs), coverage metrics, and toolchains to gauge their potential for broader adoption, akin to traditional formal verification methods. Furthermore, this work examines the reliability of existing hardware fuzzing techniques in identifying vulnerabilities and identifies research gaps for future advancements in design verification techniques.

Title: Re-identification from histopathology images

• Authors: Jonathan Ganz, Jonas Ammeling, Samir Jabari, Katharina Breininger, Marc Aubreville
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12816
• Pdf URL: https://arxiv.org/pdf/2403.12816
• Copy Paste: [[2403.12816]] Re-identification from histopathology images(https://arxiv.org/abs/2403.12816)
• Keywords: privacy
• Abstract: In numerous studies, deep learning algorithms have proven their potential for the analysis of histopathology images, for example, for revealing the subtypes of tumors or the primary origin of metastases. These models require large datasets for training, which must be anonymized to prevent possible patient identity leaks. This study demonstrates that even relatively simple deep learning algorithms can re-identify patients in large histopathology datasets with substantial accuracy. We evaluated our algorithms on two TCIA datasets including lung squamous cell carcinoma (LSCC) and lung adenocarcinoma (LUAD). We also demonstrate the algorithm's performance on an in-house dataset of meningioma tissue. We predicted the source patient of a slide with F1 scores of 50.16 % and 52.30 % on the LSCC and LUAD datasets, respectively, and with 62.31 % on our meningioma dataset. Based on our findings, we formulated a risk assessment scheme to estimate the risk to the patient's privacy prior to publication.

Title: FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer

• Authors: Dongyeong Hwang, Hyunju Kim, Sunwoo Kim, Kijung Shin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12821
• Pdf URL: https://arxiv.org/pdf/2403.12821
• Copy Paste: [[2403.12821]] FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer(https://arxiv.org/abs/2403.12821)
• Keywords: transformer
• Abstract: The success of a specific neural network architecture is closely tied to the dataset and task it tackles; there is no one-size-fits-all solution. Thus, considerable efforts have been made to quickly and accurately estimate the performances of neural architectures, without full training or evaluation, for given tasks and datasets. Neural architecture encoding has played a crucial role in the estimation, and graphbased methods, which treat an architecture as a graph, have shown prominent performance. For enhanced representation learning of neural architectures, we introduce FlowerFormer, a powerful graph transformer that incorporates the information flows within a neural architecture. FlowerFormer consists of two key components: (a) bidirectional asynchronous message passing, inspired by the flows; (b) global attention built on flow-based masking. Our extensive experiments demonstrate the superiority of FlowerFormer over existing neural encoding methods, and its effectiveness extends beyond computer vision models to include graph neural networks and auto speech recognition models. Our code is available at this http URL

Title: Has Approximate Machine Unlearning been evaluated properly? From Auditing to Side Effects

• Authors: Cheng-Long Wang, Qi Li, Zihang Xiang, Di Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.12830
• Pdf URL: https://arxiv.org/pdf/2403.12830
• Copy Paste: [[2403.12830]] Has Approximate Machine Unlearning been evaluated properly? From Auditing to Side Effects(https://arxiv.org/abs/2403.12830)
• Keywords: security, privacy, membership infer
• Abstract: The growing concerns surrounding data privacy and security have underscored the critical necessity for machine unlearning--aimed at fully removing data lineage from machine learning models. MLaaS providers expect this to be their ultimate safeguard for regulatory compliance. Despite its critical importance, the pace at which privacy communities have been developing and implementing strong methods to verify the effectiveness of machine unlearning has been disappointingly slow, with this vital area often receiving insufficient focus. This paper seeks to address this shortfall by introducing well-defined and effective metrics for black-box unlearning auditing tasks. We transform the auditing challenge into a question of non-membership inference and develop efficient metrics for auditing. By relying exclusively on the original and unlearned models--eliminating the need to train additional shadow models--our approach simplifies the evaluation of unlearning at the individual data point level. Utilizing these metrics, we conduct an in-depth analysis of current approximate machine unlearning algorithms, identifying three key directions where these approaches fall short: utility, resilience, and equity. Our aim is that this work will greatly improve our understanding of approximate machine unlearning methods, taking a significant stride towards converting the theoretical right to data erasure into a auditable reality.

Title: Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation

• Authors: Karol Gotkowski, Carsten Lüth, Paul F. Jäger, Sebastian Ziegler, Lars Krämer, Stefan Denner, Shuhan Xiao, Nico Disch, Klaus H. Maier-Hein, Fabian Isensee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12834
• Pdf URL: https://arxiv.org/pdf/2403.12834
• Copy Paste: [[2403.12834]] Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation(https://arxiv.org/abs/2403.12834)
• Keywords: segmentation
• Abstract: Traditionally, segmentation algorithms require dense annotations for training, demanding significant annotation efforts, particularly within the 3D medical imaging field. Scribble-supervised learning emerges as a possible solution to this challenge, promising a reduction in annotation efforts when creating large-scale datasets. Recently, a plethora of methods for optimized learning from scribbles have been proposed, but have so far failed to position scribble annotation as a beneficial alternative. We relate this shortcoming to two major issues: 1) the complex nature of many methods which deeply ties them to the underlying segmentation model, thus preventing a migration to more powerful state-of-the-art models as the field progresses and 2) the lack of a systematic evaluation to validate consistent performance across the broader medical domain, resulting in a lack of trust when applying these methods to new segmentation problems. To address these issues, we propose a comprehensive scribble supervision benchmark consisting of seven datasets covering a diverse set of anatomies and pathologies imaged with varying modalities. We furthermore propose the systematic use of partial losses, i.e. losses that are only computed on annotated voxels. Contrary to most existing methods, these losses can be seamlessly integrated into state-of-the-art segmentation methods, enabling them to learn from scribble annotations while preserving their original loss formulations. Our evaluation using nnU-Net reveals that while most existing methods suffer from a lack of generalization, the proposed approach consistently delivers state-of-the-art performance. Thanks to its simplicity, our approach presents an embarrassingly simple yet effective solution to the challenges of scribble supervision. Source code as well as our extensive scribble benchmarking suite will be made publicly available upon publication.

Title: MELTing point: Mobile Evaluation of Language Transformers

• Authors: Stefanos Laskaridis, Kleomenis Kateveas, Lorenzo Minto, Hamed Haddadi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12844
• Pdf URL: https://arxiv.org/pdf/2403.12844
• Copy Paste: [[2403.12844]] MELTing point: Mobile Evaluation of Language Transformers(https://arxiv.org/abs/2403.12844)
• Keywords: privacy, transformer, large language model
• Abstract: Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with ``sparks of intelligence''. However, their runtime requirements have prevented them from being broadly deployed on mobile. As personal devices become increasingly powerful and prompt privacy becomes an ever more pressing issue, we explore the current state of mobile execution of Large Language Models (LLMs). To achieve this, we have created our own automation infrastructure, MELT, which supports the headless execution and benchmarking of LLMs on device, supporting different models, devices and frameworks, including Android, iOS and Nvidia Jetson devices. We evaluate popular instruction fine-tuned LLMs and leverage different frameworks to measure their end-to-end and granular performance, tracing their memory and energy requirements along the way. Our analysis is the first systematic study of on-device LLM execution, quantifying performance, energy efficiency and accuracy across various state-of-the-art models and showcases the state of on-device intelligence in the era of hyperscale models. Results highlight the performance heterogeneity across targets and corroborates that LLM inference is largely memory-bound. Quantization drastically reduces memory requirements and renders execution viable, but at a non-negligible accuracy cost. Drawing from its energy footprint and thermal behavior, the continuous execution of LLMs remains elusive, as both factors negatively affect user experience. Last, our experience shows that the ecosystem is still in its infancy, and algorithmic as well as hardware breakthroughs can significantly shift the execution cost. We expect NPU acceleration, and framework-hardware co-design to be the biggest bet towards efficient standalone execution, with the alternative of offloading tailored towards edge deployments.

Title: Compositional 3D Scene Synthesis with Scene Graph Guided Layout-Shape Generation

• Authors: Yao Wei, Martin Renqiang Min, George Vosselman, Li Erran Li, Michael Ying Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12848
• Pdf URL: https://arxiv.org/pdf/2403.12848
• Copy Paste: [[2403.12848]] Compositional 3D Scene Synthesis with Scene Graph Guided Layout-Shape Generation(https://arxiv.org/abs/2403.12848)
• Keywords: diffusion, generative, large language model
• Abstract: Compositional 3D scene synthesis has diverse applications across a spectrum of industries such as robotics, films, and video games, as it closely mirrors the complexity of real-world multi-object environments. Early works typically employ shape retrieval based frameworks which naturally suffer from limited shape diversity. Recent progresses have been made in shape generation with powerful generative models, such as diffusion models, which increases the shape fidelity. However, these approaches separately treat 3D shape generation and layout generation. The synthesized scenes are usually hampered by layout collision, which implies that the scene-level fidelity is still under-explored. In this paper, we aim at generating realistic and reasonable 3D scenes from scene graph. To enrich the representation capability of the given scene graph inputs, large language model is utilized to explicitly aggregate the global graph features with local relationship features. With a unified graph convolution network (GCN), graph features are extracted from scene graphs updated via joint layout-shape distribution. During scene generation, an IoU-based regularization loss is introduced to constrain the predicted 3D layouts. Benchmarked on the SG-FRONT dataset, our method achieves better 3D scene synthesis, especially in terms of scene-level fidelity. The source code will be released after publication.

Title: Equivariant Ensembles and Regularization for Reinforcement Learning in Map-based Path Planning

• Authors: Mirco Theile, Hongpeng Cao, Marco Caccamo, Alberto L. Sangiovanni-Vincentelli
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.12856
• Pdf URL: https://arxiv.org/pdf/2403.12856
• Copy Paste: [[2403.12856]] Equivariant Ensembles and Regularization for Reinforcement Learning in Map-based Path Planning(https://arxiv.org/abs/2403.12856)
• Keywords: robust
• Abstract: In reinforcement learning (RL), exploiting environmental symmetries can significantly enhance efficiency, robustness, and performance. However, ensuring that the deep RL policy and value networks are respectively equivariant and invariant to exploit these symmetries is a substantial challenge. Related works try to design networks that are equivariant and invariant by construction, limiting them to a very restricted library of components, which in turn hampers the expressiveness of the networks. This paper proposes a method to construct equivariant policies and invariant value functions without specialized neural network components, which we term equivariant ensembles. We further add a regularization term for adding inductive bias during training. In a map-based path planning case study, we show how equivariant ensembles and regularization benefit sample efficiency and performance.

Title: A Comparison of Deep Learning Architectures for Spacecraft Anomaly Detection

• Authors: Daniel Lakey, Tim Schlippe
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12864
• Pdf URL: https://arxiv.org/pdf/2403.12864
• Copy Paste: [[2403.12864]] A Comparison of Deep Learning Architectures for Spacecraft Anomaly Detection(https://arxiv.org/abs/2403.12864)
• Keywords: transformer
• Abstract: Spacecraft operations are highly critical, demanding impeccable reliability and safety. Ensuring the optimal performance of a spacecraft requires the early detection and mitigation of anomalies, which could otherwise result in unit or mission failures. With the advent of deep learning, a surge of interest has been seen in leveraging these sophisticated algorithms for anomaly detection in space operations. This study aims to compare the efficacy of various deep learning architectures in detecting anomalies in spacecraft data. The deep learning models under investigation include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Transformer-based architectures. Each of these models was trained and validated using a comprehensive dataset sourced from multiple spacecraft missions, encompassing diverse operational scenarios and anomaly types. Initial results indicate that while CNNs excel in identifying spatial patterns and may be effective for some classes of spacecraft data, LSTMs and RNNs show a marked proficiency in capturing temporal anomalies seen in time-series spacecraft telemetry. The Transformer-based architectures, given their ability to focus on both local and global contexts, have showcased promising results, especially in scenarios where anomalies are subtle and span over longer durations. Additionally, considerations such as computational efficiency, ease of deployment, and real-time processing capabilities were evaluated. While CNNs and LSTMs demonstrated a balance between accuracy and computational demands, Transformer architectures, though highly accurate, require significant computational resources. In conclusion, the choice of deep learning architecture for spacecraft anomaly detection is highly contingent on the nature of the data, the type of anomalies, and operational constraints.

Title: Wildfire danger prediction optimization with transfer learning

• Authors: Spiros Maggioros, Nikos Tsalkitzis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12871
• Pdf URL: https://arxiv.org/pdf/2403.12871
• Copy Paste: [[2403.12871]] Wildfire danger prediction optimization with transfer learning(https://arxiv.org/abs/2403.12871)
• Keywords: robust
• Abstract: Convolutional Neural Networks (CNNs) have proven instrumental across various computer science domains, enabling advancements in object detection, classification, and anomaly detection. This paper explores the application of CNNs to analyze geospatial data specifically for identifying wildfire-affected areas. Leveraging transfer learning techniques, we fine-tuned CNN hyperparameters and integrated the Canadian Fire Weather Index (FWI) to assess moisture conditions. The study establishes a methodology for computing wildfire risk levels on a scale of 0 to 5, dynamically linked to weather patterns. Notably, through the integration of transfer learning, the CNN model achieved an impressive accuracy of 95\% in identifying burnt areas. This research sheds light on the inner workings of CNNs and their practical, real-time utility in predicting and mitigating wildfires. By combining transfer learning and CNNs, this study contributes a robust approach to assess burnt areas, facilitating timely interventions and preventative measures against conflagrations.

Title: Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models

• Authors: Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, Feng Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12881
• Pdf URL: https://arxiv.org/pdf/2403.12881
• Copy Paste: [[2403.12881]] Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models(https://arxiv.org/abs/2403.12881)
• Keywords: large language model
• Abstract: Open-sourced Large Language Models (LLMs) have achieved great success in various NLP tasks, however, they are still far inferior to API-based models when acting as agents. How to integrate agent ability into general LLMs becomes a crucial and urgent problem. This paper first delivers three key observations: (1) the current agent training corpus is entangled with both formats following and agent reasoning, which significantly shifts from the distribution of its pre-training data; (2) LLMs exhibit different learning speeds on the capabilities required by agent tasks; and (3) current approaches have side-effects when improving agent abilities by introducing hallucinations. Based on the above findings, we propose Agent-FLAN to effectively Fine-tune LANguage models for Agents. Through careful decomposition and redesign of the training corpus, Agent-FLAN enables Llama2-7B to outperform prior best works by 3.5\% across various agent evaluation datasets. With comprehensively constructed negative samples, Agent-FLAN greatly alleviates the hallucination issues based on our established evaluation benchmark. Besides, it consistently improves the agent capability of LLMs when scaling model sizes while slightly enhancing the general capability of LLMs. The code will be available at https://github.com/InternLM/Agent-FLAN.

Title: Confusing Pair Correction Based on Category Prototype for Domain Adaptation under Noisy Environments

• Authors: Churan Zhi, Junbao Zhuo, Shuhui Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12883
• Pdf URL: https://arxiv.org/pdf/2403.12883
• Copy Paste: [[2403.12883]] Confusing Pair Correction Based on Category Prototype for Domain Adaptation under Noisy Environments(https://arxiv.org/abs/2403.12883)
• Keywords: robust
• Abstract: In this paper, we address unsupervised domain adaptation under noisy environments, which is more challenging and practical than traditional domain adaptation. In this scenario, the model is prone to overfitting noisy labels, resulting in a more pronounced domain shift and a notable decline in the overall model performance. Previous methods employed prototype methods for domain adaptation on robust feature spaces. However, these approaches struggle to effectively classify classes with similar features under noisy environments. To address this issue, we propose a new method to detect and correct confusing class pair. We first divide classes into easy and hard classes based on the small loss criterion. We then leverage the top-2 predictions for each sample after aligning the source and target domain to find the confusing pair in the hard classes. We apply label correction to the noisy samples within the confusing pair. With the proposed label correction method, we can train our model with more accurate labels. Extensive experiments confirm the effectiveness of our method and demonstrate its favorable performance compared with existing state-of-the-art methods. Our codes are publicly available at https://github.com/Hehxcf/CPC/.

Title: HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning

• Authors: Fucai Ke, Zhixi Cai, Simindokht Jahangard, Weiqing Wang, Pari Delir Haghighi, Hamid Rezatofighi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12884
• Pdf URL: https://arxiv.org/pdf/2403.12884
• Copy Paste: [[2403.12884]] HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning(https://arxiv.org/abs/2403.12884)
• Keywords: large language model
• Abstract: Recent advances in visual reasoning (VR), particularly with the aid of Large Vision-Language Models (VLMs), show promise but require access to large-scale datasets and face challenges such as high computational costs and limited generalization capabilities. Compositional visual reasoning approaches have emerged as effective strategies; however, they heavily rely on the commonsense knowledge encoded in Large Language Models (LLMs) to perform planning, reasoning, or both, without considering the effect of their decisions on the visual reasoning process, which can lead to errors or failed procedures. To address these challenges, we introduce HYDRA, a multi-stage dynamic compositional visual reasoning framework designed for reliable and incrementally progressive general reasoning. HYDRA integrates three essential modules: a planner, a Reinforcement Learning (RL) agent serving as a cognitive controller, and a reasoner. The planner and reasoner modules utilize an LLM to generate instruction samples and executable code from the selected instruction, respectively, while the RL agent dynamically interacts with these modules, making high-level decisions on selection of the best instruction sample given information from the historical state stored through a feedback loop. This adaptable design enables HYDRA to adjust its actions based on previous feedback received during the reasoning process, leading to more reliable reasoning outputs and ultimately enhancing its overall effectiveness. Our framework demonstrates state-of-the-art performance in various VR tasks on four different widely-used datasets.

Title: mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding

• Authors: Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, Jingren Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12895
• Pdf URL: https://arxiv.org/pdf/2403.12895
• Copy Paste: [[2403.12895]] mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding(https://arxiv.org/abs/2403.12895)
• Keywords: large language model
• Abstract: Structure information is critical for understanding the semantics of text-rich images, such as documents, tables, and charts. Existing Multimodal Large Language Models (MLLMs) for Visual Document Understanding are equipped with text recognition ability but lack general structure understanding abilities for text-rich document images. In this work, we emphasize the importance of structure information in Visual Document Understanding and propose the Unified Structure Learning to boost the performance of MLLMs. Our Unified Structure Learning comprises structure-aware parsing tasks and multi-grained text localization tasks across 5 domains: document, webpage, table, chart, and natural image. To better encode structure information, we design a simple and effective vision-to-text module H-Reducer, which can not only maintain the layout information but also reduce the length of visual features by merging horizontal adjacent patches through convolution, enabling the LLM to understand high-resolution images more efficiently. Furthermore, by constructing structure-aware text sequences and multi-grained pairs of texts and bounding boxes for publicly available text-rich images, we build a comprehensive training set DocStruct4M to support structure learning. Finally, we construct a small but high-quality reasoning tuning dataset DocReason25K to trigger the detailed explanation ability in the document domain. Our model DocOwl 1.5 achieves state-of-the-art performance on 10 visual document understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.

Title: Ultra-High-Resolution Image Synthesis with Pyramid Diffusion Model

• Authors: Jiajie Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12915
• Pdf URL: https://arxiv.org/pdf/2403.12915
• Copy Paste: [[2403.12915]] Ultra-High-Resolution Image Synthesis with Pyramid Diffusion Model(https://arxiv.org/abs/2403.12915)
• Keywords: diffusion, generative
• Abstract: We introduce the Pyramid Diffusion Model (PDM), a novel architecture designed for ultra-high-resolution image synthesis. PDM utilizes a pyramid latent representation, providing a broader design space that enables more flexible, structured, and efficient perceptual compression which enable AutoEncoder and Network of Diffusion to equip branches and deeper layers. To enhance PDM's capabilities for generative tasks, we propose the integration of Spatial-Channel Attention and Res-Skip Connection, along with the utilization of Spectral Norm and Decreasing Dropout Strategy for the Diffusion Network and AutoEncoder. In summary, PDM achieves the synthesis of images with a 2K resolution for the first time, demonstrated on two new datasets comprising images of sizes 2048x2048 pixels and 2048x1024 pixels respectively. We believe that this work offers an alternative approach to designing scalable image generative models, while also providing incremental reinforcement for existing frameworks.

Title: Supporting Energy Policy Research with Large Language Models

• Authors: Grant Buster, Pavlo Pinchuk, Jacob Barrons, Ryan McKeever, Aaron Levine, Anthony Lopez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12924
• Pdf URL: https://arxiv.org/pdf/2403.12924
• Copy Paste: [[2403.12924]] Supporting Energy Policy Research with Large Language Models(https://arxiv.org/abs/2403.12924)
• Keywords: extraction, large language model
• Abstract: The recent growth in renewable energy development in the United States has been accompanied by a simultaneous surge in renewable energy siting ordinances. These zoning laws play a critical role in dictating the placement of wind and solar resources that are critical for achieving low-carbon energy futures. In this context, efficient access to and management of siting ordinance data becomes imperative. The National Renewable Energy Laboratory (NREL) recently introduced a public wind and solar siting database to fill this need. This paper presents a method for harnessing Large Language Models (LLMs) to automate the extraction of these siting ordinances from legal documents, enabling this database to maintain accurate up-to-date information in the rapidly changing energy policy landscape. A novel contribution of this research is the integration of a decision tree framework with LLMs. Our results show that this approach is 85 to 90% accurate with outputs that can be used directly in downstream quantitative modeling. We discuss opportunities to use this work to support similar large-scale policy research in the energy sector. By unlocking new efficiencies in the extraction and analysis of legal documents using LLMs, this study enables a path forward for automated large-scale energy policy research.

Title: You Only Sample Once: Taming One-Step Text-To-Image Synthesis by Self-Cooperative Diffusion GANs

• Authors: Yihong Luo, Xiaolong Chen, Jing Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12931
• Pdf URL: https://arxiv.org/pdf/2403.12931
• Copy Paste: [[2403.12931]] You Only Sample Once: Taming One-Step Text-To-Image Synthesis by Self-Cooperative Diffusion GANs(https://arxiv.org/abs/2403.12931)
• Keywords: diffusion, transformer, generative
• Abstract: We introduce YOSO, a novel generative model designed for rapid, scalable, and high-fidelity one-step image synthesis. This is achieved by integrating the diffusion process with GANs. Specifically, we smooth the distribution by the denoising generator itself, performing self-cooperative learning. We show that our method can serve as a one-step generation model training from scratch with competitive performance. Moreover, we show that our method can be extended to finetune pre-trained text-to-image diffusion for high-quality one-step text-to-image synthesis even with LoRA fine-tuning. In particular, we provide the first diffusion transformer that can generate images in one step trained on 512 resolution, with the capability of adapting to 1024 resolution without explicit training. Our code is provided at https://github.com/Luo-Yihong/YOSO.

Title: Zero-Reference Low-Light Enhancement via Physical Quadruple Priors

• Authors: Wenjing Wang, Huan Yang, Jianlong Fu, Jiaying Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12933
• Pdf URL: https://arxiv.org/pdf/2403.12933
• Copy Paste: [[2403.12933]] Zero-Reference Low-Light Enhancement via Physical Quadruple Priors(https://arxiv.org/abs/2403.12933)
• Keywords: robust, interpretability, diffusion, generative
• Abstract: Understanding illumination and reducing the need for supervision pose a significant challenge in low-light enhancement. Current approaches are highly sensitive to data usage during training and illumination-specific hyper-parameters, limiting their ability to handle unseen scenarios. In this paper, we propose a new zero-reference low-light enhancement framework trainable solely with normal light images. To accomplish this, we devise an illumination-invariant prior inspired by the theory of physical light transfer. This prior serves as the bridge between normal and low-light images. Then, we develop a prior-to-image framework trained without low-light data. During testing, this framework is able to restore our illumination-invariant prior back to images, automatically achieving low-light enhancement. Within this framework, we leverage a pretrained generative diffusion model for model ability, introduce a bypass decoder to handle detail distortion, as well as offer a lightweight version for practicality. Extensive experiments demonstrate our framework's superiority in various scenarios as well as good interpretability, robustness, and efficiency. Code is available on our project homepage: this http URL

Title: Segment Anything for comprehensive analysis of grapevine cluster architecture and berry properties

• Authors: Efrain Torres-Lomas, Jimena Lado-Jimena, Guillermo Garcia-Zamora, Luis Diaz-Garcia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12935
• Pdf URL: https://arxiv.org/pdf/2403.12935
• Copy Paste: [[2403.12935]] Segment Anything for comprehensive analysis of grapevine cluster architecture and berry properties(https://arxiv.org/abs/2403.12935)
• Keywords: segmentation
• Abstract: Grape cluster architecture and compactness are complex traits influencing disease susceptibility, fruit quality, and yield. Evaluation methods for these traits include visual scoring, manual methodologies, and computer vision, with the latter being the most scalable approach. Most of the existing computer vision approaches for processing cluster images often rely on conventional segmentation or machine learning with extensive training and limited generalization. The Segment Anything Model (SAM), a novel foundation model trained on a massive image dataset, enables automated object segmentation without additional training. This study demonstrates out-of-the-box SAM's high accuracy in identifying individual berries in 2D cluster images. Using this model, we managed to segment approximately 3,500 cluster images, generating over 150,000 berry masks, each linked with spatial coordinates within their clusters. The correlation between human-identified berries and SAM predictions was very strong (Pearson r2=0.96). Although the visible berry count in images typically underestimates the actual cluster berry count due to visibility issues, we demonstrated that this discrepancy could be adjusted using a linear regression model (adjusted R2=0.87). We emphasized the critical importance of the angle at which the cluster is imaged, noting its substantial effect on berry counts and architecture. We proposed different approaches in which berry location information facilitated the calculation of complex features related to cluster architecture and compactness. Finally, we discussed SAM's potential integration into currently available pipelines for image generation and processing in vineyard conditions.

Title: Automatic Information Extraction From Employment Tribunal Judgements Using Large Language Models

• Authors: Joana Ribeiro de Faria, Huiyuan Xie, Felix Steffek
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12936
• Pdf URL: https://arxiv.org/pdf/2403.12936
• Copy Paste: [[2403.12936]] Automatic Information Extraction From Employment Tribunal Judgements Using Large Language Models(https://arxiv.org/abs/2403.12936)
• Keywords: extraction, large language model
• Abstract: Court transcripts and judgments are rich repositories of legal knowledge, detailing the intricacies of cases and the rationale behind judicial decisions. The extraction of key information from these documents provides a concise overview of a case, crucial for both legal experts and the public. With the advent of large language models (LLMs), automatic information extraction has become increasingly feasible and efficient. This paper presents a comprehensive study on the application of GPT-4, a large language model, for automatic information extraction from UK Employment Tribunal (UKET) cases. We meticulously evaluated GPT-4's performance in extracting critical information with a manual verification process to ensure the accuracy and relevance of the extracted data. Our research is structured around two primary extraction tasks: the first involves a general extraction of eight key aspects that hold significance for both legal specialists and the general public, including the facts of the case, the claims made, references to legal statutes, references to precedents, general case outcomes and corresponding labels, detailed order and remedies and reasons for the decision. The second task is more focused, aimed at analysing three of those extracted features, namely facts, claims and outcomes, in order to facilitate the development of a tool capable of predicting the outcome of employment law disputes. Through our analysis, we demonstrate that LLMs like GPT-4 can obtain high accuracy in legal information extraction, highlighting the potential of LLMs in revolutionising the way legal information is processed and utilised, offering significant implications for legal research and practice.

Title: Neural Differential Algebraic Equations

• Authors: James Koch, Madelyn Shapiro, Himanshu Sharma, Draguna Vrabie, Jan Drgona
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12938
• Pdf URL: https://arxiv.org/pdf/2403.12938
• Copy Paste: [[2403.12938]] Neural Differential Algebraic Equations(https://arxiv.org/abs/2403.12938)
• Keywords: robust
• Abstract: Differential-Algebraic Equations (DAEs) describe the temporal evolution of systems that obey both differential and algebraic constraints. Of particular interest are systems that contain implicit relationships between their components, such as conservation relationships. Here, we present Neural Differential-Algebraic Equations (NDAEs) suitable for data-driven modeling of DAEs. This methodology is built upon the concept of the Universal Differential Equation; that is, a model constructed as a system of Neural Ordinary Differential Equations informed by theory from particular science domains. In this work, we show that the proposed NDAEs abstraction is suitable for relevant system-theoretic data-driven modeling tasks. Presented examples include (i) the inverse problem of tank-manifold dynamics and (ii) discrepancy modeling of a network of pumps, tanks, and pipes. Our experiments demonstrate the proposed method's robustness to noise and extrapolation ability to (i) learn the behaviors of the system components and their interaction physics and (ii) disambiguate between data trends and mechanistic relationships contained in the system.

Title: Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes

• Authors: He Wang, Laixi Shi, Yuejie Chi
• Subjects: cs.LG, math.ST
• Abstract URL: https://arxiv.org/abs/2403.12946
• Pdf URL: https://arxiv.org/pdf/2403.12946
• Copy Paste: [[2403.12946]] Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes(https://arxiv.org/abs/2403.12946)
• Keywords: robust
• Abstract: In offline reinforcement learning (RL), the absence of active exploration calls for attention on the model robustness to tackle the sim-to-real gap, where the discrepancy between the simulated and deployed environments can significantly undermine the performance of the learned policy. To endow the learned policy with robustness in a sample-efficient manner in the presence of high-dimensional state-action space, this paper considers the sample complexity of distributionally robust linear Markov decision processes (MDPs) with an uncertainty set characterized by the total variation distance using offline data. We develop a pessimistic model-based algorithm and establish its sample complexity bound under minimal data coverage assumptions, which outperforms prior art by at least $\tilde{O}(d)$, where $d$ is the feature dimension. We further improve the performance guarantee of the proposed algorithm by incorporating a carefully-designed variance estimator.

Title: GVGEN: Text-to-3D Generation with Volumetric Representation

• Authors: Xianglong He, Junyi Chen, Sida Peng, Di Huang, Yangguang Li, Xiaoshui Huang, Chun Yuan, Wanli Ouyang, Tong He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12957
• Pdf URL: https://arxiv.org/pdf/2403.12957
• Copy Paste: [[2403.12957]] GVGEN: Text-to-3D Generation with Volumetric Representation(https://arxiv.org/abs/2403.12957)
• Keywords: diffusion
• Abstract: In recent years, 3D Gaussian splatting has emerged as a powerful technique for 3D reconstruction and generation, known for its fast and high-quality rendering capabilities. To address these shortcomings, this paper introduces a novel diffusion-based framework, GVGEN, designed to efficiently generate 3D Gaussian representations from text input. We propose two innovative techniques:(1) Structured Volumetric Representation. We first arrange disorganized 3D Gaussian points as a structured form GaussianVolume. This transformation allows the capture of intricate texture details within a volume composed of a fixed number of Gaussians. To better optimize the representation of these details, we propose a unique pruning and densifying method named the Candidate Pool Strategy, enhancing detail fidelity through selective optimization. (2) Coarse-to-fine Generation Pipeline. To simplify the generation of GaussianVolume and empower the model to generate instances with detailed 3D geometry, we propose a coarse-to-fine pipeline. It initially constructs a basic geometric structure, followed by the prediction of complete Gaussian attributes. Our framework, GVGEN, demonstrates superior performance in qualitative and quantitative assessments compared to existing 3D generation methods. Simultaneously, it maintains a fast generation speed ($\sim$7 seconds), effectively striking a balance between quality and efficiency.

Title: Dated Data: Tracing Knowledge Cutoffs in Large Language Models

• Authors: Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12958
• Pdf URL: https://arxiv.org/pdf/2403.12958
• Copy Paste: [[2403.12958]] Dated Data: Tracing Knowledge Cutoffs in Large Language Models(https://arxiv.org/abs/2403.12958)
• Keywords: large language model
• Abstract: Released Large Language Models (LLMs) are often paired with a claimed knowledge cutoff date, or the dates at which training data was gathered. Such information is crucial for applications where the LLM must provide up to date information. However, this statement only scratches the surface: do all resources in the training data share the same knowledge cutoff date? Does the model's demonstrated knowledge for these subsets closely align to their cutoff dates? In this work, we define the notion of an effective cutoff. This is distinct from the LLM designer reported cutoff and applies separately to sub-resources and topics. We propose a simple approach to estimate effective cutoffs on the resource-level temporal alignment of an LLM by probing across versions of the data. Using this analysis, we find that effective cutoffs often differ from reported cutoffs. To understand the root cause of this observation, we conduct a direct large-scale analysis on open pre-training datasets. Our analysis reveals two reasons for these inconsistencies: (1) temporal biases of CommonCrawl data due to non-trivial amounts of old data in new dumps and (2) complications in LLM deduplication schemes involving semantic duplicates and lexical near-duplicates. Overall, our results show that knowledge cutoffs are not as simple as they have seemed and that care must be taken both by LLM dataset curators as well as practitioners who seek to use information from these models.

Title: FaceXFormer: A Unified Transformer for Facial Analysis

• Authors: Kartik Narayan, Vibashan VS, Rama Chellappa, Vishal M. Patel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12960
• Pdf URL: https://arxiv.org/pdf/2403.12960
• Copy Paste: [[2403.12960]] FaceXFormer: A Unified Transformer for Facial Analysis(https://arxiv.org/abs/2403.12960)
• Keywords: robust, transformer
• Abstract: In this work, we introduce FaceXformer, an end-to-end unified transformer model for a comprehensive range of facial analysis tasks such as face parsing, landmark detection, head pose estimation, attributes recognition, and estimation of age, gender, race, and landmarks visibility. Conventional methods in face analysis have often relied on task-specific designs and preprocessing techniques, which limit their approach to a unified architecture. Unlike these conventional methods, our FaceXformer leverages a transformer-based encoder-decoder architecture where each task is treated as a learnable token, enabling the integration of multiple tasks within a single framework. Moreover, we propose a parameter-efficient decoder, FaceX, which jointly processes face and task tokens, thereby learning generalized and robust face representations across different tasks. To the best of our knowledge, this is the first work to propose a single model capable of handling all these facial analysis tasks using transformers. We conducted a comprehensive analysis of effective backbones for unified face task processing and evaluated different task queries and the synergy between them. We conduct experiments against state-of-the-art specialized models and previous multi-task models in both intra-dataset and cross-dataset evaluations across multiple benchmarks. Additionally, our model effectively handles images "in-the-wild," demonstrating its robustness and generalizability across eight different tasks, all while maintaining the real-time performance of 37 FPS.

Title: TexTile: A Differentiable Metric for Texture Tileability

• Authors: Carlos Rodriguez-Pardo, Dan Casas, Elena Garces, Jorge Lopez-Moreno
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12961
• Pdf URL: https://arxiv.org/pdf/2403.12961
• Copy Paste: [[2403.12961]] TexTile: A Differentiable Metric for Texture Tileability(https://arxiv.org/abs/2403.12961)
• Keywords: robust, diffusion
• Abstract: We introduce TexTile, a novel differentiable metric to quantify the degree upon which a texture image can be concatenated with itself without introducing repeating artifacts (i.e., the tileability). Existing methods for tileable texture synthesis focus on general texture quality, but lack explicit analysis of the intrinsic repeatability properties of a texture. In contrast, our TexTile metric effectively evaluates the tileable properties of a texture, opening the door to more informed synthesis and analysis of tileable textures. Under the hood, TexTile is formulated as a binary classifier carefully built from a large dataset of textures of different styles, semantics, regularities, and human annotations.Key to our method is a set of architectural modifications to baseline pre-train image classifiers to overcome their shortcomings at measuring tileability, along with a custom data augmentation and training regime aimed at increasing robustness and accuracy. We demonstrate that TexTile can be plugged into different state-of-the-art texture synthesis methods, including diffusion-based strategies, and generate tileable textures while keeping or even improving the overall texture quality. Furthermore, we show that TexTile can objectively evaluate any tileable texture synthesis method, whereas the current mix of existing metrics produces uncorrelated scores which heavily hinders progress in the field.

Title: FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation

• Authors: Shuai Yang, Yifan Zhou, Ziwei Liu, Chen Change Loy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12962
• Pdf URL: https://arxiv.org/pdf/2403.12962
• Copy Paste: [[2403.12962]] FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation(https://arxiv.org/abs/2403.12962)
• Keywords: robust, diffusion
• Abstract: The remarkable efficacy of text-to-image diffusion models has motivated extensive exploration of their potential application in video domains. Zero-shot methods seek to extend image diffusion models to videos without necessitating model training. Recent methods mainly focus on incorporating inter-frame correspondence into attention mechanisms. However, the soft constraint imposed on determining where to attend to valid features can sometimes be insufficient, resulting in temporal inconsistency. In this paper, we introduce FRESCO, intra-frame correspondence alongside inter-frame correspondence to establish a more robust spatial-temporal constraint. This enhancement ensures a more consistent transformation of semantically similar content across frames. Beyond mere attention guidance, our approach involves an explicit update of features to achieve high spatial-temporal consistency with the input video, significantly improving the visual coherence of the resulting translated videos. Extensive experiments demonstrate the effectiveness of our proposed framework in producing high-quality, coherent videos, marking a notable improvement over existing zero-shot methods.

Title: FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis

• Authors: Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12963
• Pdf URL: https://arxiv.org/pdf/2403.12963
• Copy Paste: [[2403.12963]] FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis(https://arxiv.org/abs/2403.12963)
• Keywords: diffusion
• Abstract: In this study, we delve into the generation of high-resolution images from pre-trained diffusion models, addressing persistent challenges, such as repetitive patterns and structural distortions, that emerge when models are applied beyond their trained resolutions. To address this issue, we introduce an innovative, training-free approach FouriScale from the perspective of frequency domain analysis. We replace the original convolutional layers in pre-trained diffusion models by incorporating a dilation technique along with a low-pass operation, intending to achieve structural consistency and scale consistency across resolutions, respectively. Further enhanced by a padding-then-crop strategy, our method can flexibly handle text-to-image generation of various aspect ratios. By using the FouriScale as guidance, our method successfully balances the structural integrity and fidelity of generated images, achieving an astonishing capacity of arbitrary-size, high-resolution, and high-quality generation. With its simplicity and compatibility, our method can provide valuable insights for future explorations into the synthesis of ultra-high-resolution images. The code will be released at https://github.com/LeonHLJ/FouriScale.

Title: Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models

• Authors: Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, Jiwen Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12966
• Pdf URL: https://arxiv.org/pdf/2403.12966
• Copy Paste: [[2403.12966]] Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models(https://arxiv.org/abs/2403.12966)
• Keywords: extraction
• Abstract: In the realm of vision-language understanding, the proficiency of models in interpreting and reasoning over visual content has become a cornerstone for numerous applications. However, it is challenging for the visual encoder in Large Vision-Language Models (LVLMs) to extract useful features tailored to questions that aid the language model's response. Furthermore, a common practice among existing LVLMs is to utilize lower-resolution images, which restricts the ability for visual recognition. Our work introduces the Chain-of-Spot (CoS) method, which we describe as Interactive Reasoning, a novel approach that enhances feature extraction by focusing on key regions of interest (ROI) within the image, corresponding to the posed questions or instructions. This technique allows LVLMs to access more detailed visual information without altering the original image resolution, thereby offering multi-granularity image features. By integrating Chain-of-Spot with instruct-following LLaVA-1.5 models, the process of image reasoning consistently improves performance across a wide range of multimodal datasets and benchmarks without bells and whistles and achieves new state-of-the-art results. Our empirical findings demonstrate a significant improvement in LVLMs' ability to understand and reason about visual content, paving the way for more sophisticated visual instruction-following applications. Code and models are available at https://github.com/dongyh20/Chain-of-Spot

Title: LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression

• Authors: Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12968
• Pdf URL: https://arxiv.org/pdf/2403.12968
• Copy Paste: [[2403.12968]] LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression(https://arxiv.org/abs/2403.12968)
• Keywords: robust, transformer
• Abstract: This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units according to their information entropy obtained from a causal language model such as LLaMa-7B. The challenge is that information entropy may be a suboptimal compression metric: (i) it only leverages unidirectional context and may fail to capture all essential information needed for prompt compression; (ii) it is not aligned with the prompt compression objective. To address these issues, we propose a data distillation procedure to derive knowledge from an LLM to compress prompts without losing crucial information, and meantime, introduce an extractive text compression dataset. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one, and use a Transformer encoder as the base architecture to capture all essential information for prompt compression from the full bidirectional context. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTa-large and mBERT. We evaluate our method on both in-domain and out-of-domain datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its small size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability across different LLMs. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x.