2024-04-25

Title: ControlTraj: Controllable Trajectory Generation with Topology-Constrained Diffusion Model

• Authors: Yuanshao Zhu, James Jianqiao Yu, Xiangyu Zhao, Qidong Liu, Yongchao Ye, Wei Chen, Zijian Zhang, Xuetao Wei, Yuxuan Liang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15380
• Pdf URL: https://arxiv.org/pdf/2404.15380
• Copy Paste: [[2404.15380]] ControlTraj: Controllable Trajectory Generation with Topology-Constrained Diffusion Model(https://arxiv.org/abs/2404.15380)
• Keywords: privacy, diffusion
• Abstract: Generating trajectory data is among promising solutions to addressing privacy concerns, collection costs, and proprietary restrictions usually associated with human mobility analyses. However, existing trajectory generation methods are still in their infancy due to the inherent diversity and unpredictability of human activities, grappling with issues such as fidelity, flexibility, and generalizability. To overcome these obstacles, we propose ControlTraj, a Controllable Trajectory generation framework with the topology-constrained diffusion model. Distinct from prior approaches, ControlTraj utilizes a diffusion model to generate high-fidelity trajectories while integrating the structural constraints of road network topology to guide the geographical outcomes. Specifically, we develop a novel road segment autoencoder to extract fine-grained road segment embedding. The encoded features, along with trip attributes, are subsequently merged into the proposed geographic denoising UNet architecture, named GeoUNet, to synthesize geographic trajectories from white noise. Through experimentation across three real-world data settings, ControlTraj demonstrates its ability to produce human-directed, high-fidelity trajectory generation with adaptability to unexplored geographical contexts.

Title: Advances and Open Challenges in Federated Learning with Foundation Models

• Authors: Chao Ren, Han Yu, Hongyi Peng, Xiaoli Tang, Anran Li, Yulan Gao, Alysa Ziying Tan, Bo Zhao, Xiaoxiao Li, Zengxiang Li, Qiang Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15381
• Pdf URL: https://arxiv.org/pdf/2404.15381
• Copy Paste: [[2404.15381]] Advances and Open Challenges in Federated Learning with Foundation Models(https://arxiv.org/abs/2404.15381)
• Keywords: security, privacy, federate
• Abstract: The integration of Foundation Models (FMs) with Federated Learning (FL) presents a transformative paradigm in Artificial Intelligence (AI), offering enhanced capabilities while addressing concerns of privacy, data decentralization, and computational efficiency. This paper provides a comprehensive survey of the emerging field of Federated Foundation Models (FedFM), elucidating their synergistic relationship and exploring novel methodologies, challenges, and future directions that the FL research field needs to focus on in order to thrive in the age of foundation models. A systematic multi-tiered taxonomy is proposed, categorizing existing FedFM approaches for model training, aggregation, trustworthiness, and incentivization. Key challenges, including how to enable FL to deal with high complexity of computational demands, privacy considerations, contribution evaluation, and communication efficiency, are thoroughly discussed. Moreover, the paper explores the intricate challenges of communication, scalability and security inherent in training/fine-tuning FMs via FL, highlighting the potential of quantum computing to revolutionize the training, inference, optimization and data encryption processes. This survey underscores the importance of further research to propel innovation in FedFM, emphasizing the need for developing trustworthy solutions. It serves as a foundational guide for researchers and practitioners interested in contributing to this interdisciplinary and rapidly advancing field.

Title: Feature Distribution Shift Mitigation with Contrastive Pretraining for Intrusion Detection

• Authors: Weixing Wang, Haojin Yang, Christoph Meinel, Hasan Yagiz Özkan, Cristian Bermudez Serna, Carmen Mas-Machuca
• Subjects: cs.LG, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.15382
• Pdf URL: https://arxiv.org/pdf/2404.15382
• Copy Paste: [[2404.15382]] Feature Distribution Shift Mitigation with Contrastive Pretraining for Intrusion Detection(https://arxiv.org/abs/2404.15382)
• Keywords: robust
• Abstract: In recent years, there has been a growing interest in using Machine Learning (ML), especially Deep Learning (DL) to solve Network Intrusion Detection (NID) problems. However, the feature distribution shift problem remains a difficulty, because the change in features' distributions over time negatively impacts the model's performance. As one promising solution, model pretraining has emerged as a novel training paradigm, which brings robustness against feature distribution shift and has proven to be successful in Computer Vision (CV) and Natural Language Processing (NLP). To verify whether this paradigm is beneficial for NID problem, we propose SwapCon, a ML model in the context of NID, which compresses shift-invariant feature information during the pretraining stage and refines during the finetuning stage. We exemplify the evidence of feature distribution shift using the Kyoto2006+ dataset. We demonstrate how pretraining a model with the proper size can increase robustness against feature distribution shifts by over 8%. Moreover, we show how an adequate numerical embedding strategy also enhances the performance of pretrained models. Further experiments show that the proposed SwapCon model also outperforms eXtreme Gradient Boosting (XGBoost) and K-Nearest Neighbor (KNN) based models by a large margin.

Title: FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering

• Authors: Siqi Ping, Yuzhu Mao, Yang Liu, Xiao-Ping Zhang, Wenbo Ding
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15384
• Pdf URL: https://arxiv.org/pdf/2404.15384
• Copy Paste: [[2404.15384]] FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering(https://arxiv.org/abs/2404.15384)
• Keywords: federate
• Abstract: Although large-scale pre-trained models hold great potential for adapting to downstream tasks through fine-tuning, the performance of such fine-tuned models is often limited by the difficulty of collecting sufficient high-quality, task-specific data. Federated Learning (FL) offers a promising solution by enabling fine-tuning across large-scale clients with a variety of task data, but it is bottlenecked by significant communication overhead due to the pre-trained models' extensive size. This paper addresses the high communication cost for fine-tuning large pre-trained models within FL frameworks through low-rank fine-tuning. Specifically, we train a low-rank adapter for each individual task on the client side, followed by server-side clustering for similar group of adapters to achieve task-specific aggregation. Extensive experiments on various language and vision tasks, such as GLUE and CIFAR-10/100, reveal the evolution of task-specific adapters throughout the FL training process and verify the effectiveness of the proposed low-rank task-specific adapter clustering (TAC) method.

Title: Sum of Group Error Differences: A Critical Examination of Bias Evaluation in Biometric Verification and a Dual-Metric Measure

• Authors: Alaa Elobaid, Nathan Ramoly, Lara Younes, Symeon Papadopoulos, Eirini Ntoutsi, Ioannis Kompatsiaris
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.15385
• Pdf URL: https://arxiv.org/pdf/2404.15385
• Copy Paste: [[2404.15385]] Sum of Group Error Differences: A Critical Examination of Bias Evaluation in Biometric Verification and a Dual-Metric Measure(https://arxiv.org/abs/2404.15385)
• Keywords: biometric, fair
• Abstract: Biometric Verification (BV) systems often exhibit accuracy disparities across different demographic groups, leading to biases in BV applications. Assessing and quantifying these biases is essential for ensuring the fairness of BV systems. However, existing bias evaluation metrics in BV have limitations, such as focusing exclusively on match or non-match error rates, overlooking bias on demographic groups with performance levels falling between the best and worst performance levels, and neglecting the magnitude of the bias present. This paper presents an in-depth analysis of the limitations of current bias evaluation metrics in BV and, through experimental analysis, demonstrates their contextual suitability, merits, and limitations. Additionally, it introduces a novel general-purpose bias evaluation measure for BV, the ``Sum of Group Error Differences (SEDG)''. Our experimental results on controlled synthetic datasets demonstrate the effectiveness of demographic bias quantification when using existing metrics and our own proposed measure. We discuss the applicability of the bias evaluation metrics in a set of simulated demographic bias scenarios and provide scenario-based metric recommendations. Our code is publicly available under \url{https://github.com/alaaobeid/SEDG}.

Title: Machine Learning Techniques with Fairness for Prediction of Completion of Drug and Alcohol Rehabilitation

• Authors: Karen Roberts-Licklider, Theodore Trafalis
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.15418
• Pdf URL: https://arxiv.org/pdf/2404.15418
• Copy Paste: [[2404.15418]] Machine Learning Techniques with Fairness for Prediction of Completion of Drug and Alcohol Rehabilitation(https://arxiv.org/abs/2404.15418)
• Keywords: fair
• Abstract: The aim of this study is to look at predicting whether a person will complete a drug and alcohol rehabilitation program and the number of times a person attends. The study is based on demographic data obtained from Substance Abuse and Mental Health Services Administration (SAMHSA) from both admissions and discharge data from drug and alcohol rehabilitation centers in Oklahoma. Demographic data is highly categorical which led to binary encoding being used and various fairness measures being utilized to mitigate bias of nine demographic variables. Kernel methods such as linear, polynomial, sigmoid, and radial basis functions were compared using support vector machines at various parameter ranges to find the optimal values. These were then compared to methods such as decision trees, random forests, and neural networks. Synthetic Minority Oversampling Technique Nominal (SMOTEN) for categorical data was used to balance the data with imputation for missing data. The nine bias variables were then intersectionalized to mitigate bias and the dual and triple interactions were integrated to use the probabilities to look at worst case ratio fairness mitigation. Disparate Impact, Statistical Parity difference, Conditional Statistical Parity Ratio, Demographic Parity, Demographic Parity Ratio, Equalized Odds, Equalized Odds Ratio, Equal Opportunity, and Equalized Opportunity Ratio were all explored at both the binary and multiclass scenarios.

Title: XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference

• Authors: João Monteiro, Étienne Marcotte, Pierre-André Noël, Valentina Zantedeschi, David Vázquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15420
• Pdf URL: https://arxiv.org/pdf/2404.15420
• Copy Paste: [[2404.15420]] XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference(https://arxiv.org/abs/2404.15420)
• Keywords: transformer
• Abstract: In-context learning (ICL) approaches typically leverage prompting to condition decoder-only language model generation on reference information. Just-in-time processing of a context is inefficient due to the quadratic cost of self-attention operations, and caching is desirable. However, caching transformer states can easily require almost as much space as the model parameters. When the right context isn't known in advance, caching ICL can be challenging. This work addresses these limitations by introducing models that, inspired by the encoder-decoder architecture, use cross-attention to condition generation on reference text without the prompt. More precisely, we leverage pre-trained decoder-only models and only train a small number of added layers. We use Question-Answering (QA) as a testbed to evaluate the ability of our models to perform conditional generation and observe that they outperform ICL, are comparable to fine-tuned prompted LLMs, and drastically reduce the space footprint relative to standard KV caching by two orders of magnitude.

Title: Iterative Cluster Harvesting for Wafer Map Defect Patterns

• Authors: Alina Pleli, Simon Baeuerle, Michel Janus, Jonas Barth, Ralf Mikut, Hendrik P. A. Lensch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15436
• Pdf URL: https://arxiv.org/pdf/2404.15436
• Copy Paste: [[2404.15436]] Iterative Cluster Harvesting for Wafer Map Defect Patterns(https://arxiv.org/abs/2404.15436)
• Keywords: extraction
• Abstract: Unsupervised clustering of wafer map defect patterns is challenging because the appearance of certain defect patterns varies significantly. This includes changing shape, location, density, and rotation of the defect area on the wafer. We present a harvesting approach, which can cluster even challenging defect patterns of wafer maps well. Our approach makes use of a well-known, three-step procedure: feature extraction, dimension reduction, and clustering. The novelty in our approach lies in repeating dimensionality reduction and clustering iteratively while filtering out one cluster per iteration according to its silhouette score. This method leads to an improvement of clustering performance in general and is especially useful for difficult defect patterns. The low computational effort allows for a quick assessment of large datasets and can be used to support manual labeling efforts. We benchmark against related approaches from the literature and show improved results on a real-world industrial dataset.

Title: OffRAMPS: An FPGA-based Intermediary for Analysis and Modification of Additive Manufacturing Control Systems

• Authors: Jason Blocklove, Md Raz, Prithwish Basu Roy, Hammond Pearce, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2404.15446
• Pdf URL: https://arxiv.org/pdf/2404.15446
• Copy Paste: [[2404.15446]] OffRAMPS: An FPGA-based Intermediary for Analysis and Modification of Additive Manufacturing Control Systems(https://arxiv.org/abs/2404.15446)
• Keywords: security, defense, attack
• Abstract: Cybersecurity threats in Additive Manufacturing (AM) are an increasing concern as AM adoption continues to grow. AM is now being used for parts in the aerospace, transportation, and medical domains. Threat vectors which allow for part compromise are particularly concerning, as any failure in these domains would have life-threatening consequences. A major challenge to investigation of AM part-compromises comes from the difficulty in evaluating and benchmarking both identified threat vectors as well as methods for detecting adversarial actions. In this work, we introduce a generalized platform for systematic analysis of attacks against and defenses for 3D printers. Our "OFFRAMPS" platform is based on the open-source 3D printer control board "RAMPS." OFFRAMPS allows analysis, recording, and modification of all control signals and I/O for a 3D printer. We show the efficacy of OFFRAMPS by presenting a series of case studies based on several Trojans, including ones identified in the literature, and show that OFFRAMPS can both emulate and detect these attacks, i.e., it can both change and detect arbitrary changes to the g-code print commands.

Title: GLoD: Composing Global Contexts and Local Details in Image Generation

• Authors: Moyuru Yamada
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15447
• Pdf URL: https://arxiv.org/pdf/2404.15447
• Copy Paste: [[2404.15447]] GLoD: Composing Global Contexts and Local Details in Image Generation(https://arxiv.org/abs/2404.15447)
• Keywords: diffusion
• Abstract: Diffusion models have demonstrated their capability to synthesize high-quality and diverse images from textual prompts. However, simultaneous control over both global contexts (e.g., object layouts and interactions) and local details (e.g., colors and emotions) still remains a significant challenge. The models often fail to understand complex descriptions involving multiple objects and reflect specified visual attributes to wrong targets or ignore them. This paper presents Global-Local Diffusion (\textit{GLoD}), a novel framework which allows simultaneous control over the global contexts and the local details in text-to-image generation without requiring training or fine-tuning. It assigns multiple global and local prompts to corresponding layers and composes their noises to guide a denoising process using pre-trained diffusion models. Our framework enables complex global-local compositions, conditioning objects in the global prompt with the local prompts while preserving other unspecified identities. Our quantitative and qualitative evaluations demonstrate that GLoD effectively generates complex images that adhere to both user-provided object interactions and object details.

Title: ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with Reward Feedback Learning

• Authors: Weifeng Chen, Jiacheng Zhang, Jie Wu, Hefeng Wu, Xuefeng Xiao, Liang Lin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15449
• Pdf URL: https://arxiv.org/pdf/2404.15449
• Copy Paste: [[2404.15449]] ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with Reward Feedback Learning(https://arxiv.org/abs/2404.15449)
• Keywords: diffusion
• Abstract: The rapid development of diffusion models has triggered diverse applications. Identity-preserving text-to-image generation (ID-T2I) particularly has received significant attention due to its wide range of application scenarios like AI portrait and advertising. While existing ID-T2I methods have demonstrated impressive results, several key challenges remain: (1) It is hard to maintain the identity characteristics of reference portraits accurately, (2) The generated images lack aesthetic appeal especially while enforcing identity retention, and (3) There is a limitation that cannot be compatible with LoRA-based and Adapter-based methods simultaneously. To address these issues, we present \textbf{ID-Aligner}, a general feedback learning framework to enhance ID-T2I performance. To resolve identity features lost, we introduce identity consistency reward fine-tuning to utilize the feedback from face detection and recognition models to improve generated identity preservation. Furthermore, we propose identity aesthetic reward fine-tuning leveraging rewards from human-annotated preference data and automatically constructed feedback on character structure generation to provide aesthetic tuning signals. Thanks to its universal feedback fine-tuning framework, our method can be readily applied to both LoRA and Adapter models, achieving consistent performance gains. Extensive experiments on SD1.5 and SDXL diffusion models validate the effectiveness of our approach. \textbf{Project Page: \url{https://idaligner.github.io/}}

Title: CFPFormer: Feature-pyramid like Transformer Decoder for Segmentation and Detection

• Authors: Hongyi Cai, Mohammad Mahdinur Rahman, Jingyu Wu, Yulun Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15451
• Pdf URL: https://arxiv.org/pdf/2404.15451
• Copy Paste: [[2404.15451]] CFPFormer: Feature-pyramid like Transformer Decoder for Segmentation and Detection(https://arxiv.org/abs/2404.15451)
• Keywords: extraction, transformer, segmentation
• Abstract: Feature pyramids have been widely adopted in convolutional neural networks (CNNs) and transformers for tasks like medical image segmentation and object detection. However, the currently existing models generally focus on the Encoder-side Transformer to extract features, from which decoder improvement can bring further potential with well-designed architecture. We propose CFPFormer, a novel decoder block that integrates feature pyramids and transformers. Specifically, by leveraging patch embedding, cross-layer feature concatenation, and Gaussian attention mechanisms, CFPFormer enhances feature extraction capabilities while promoting generalization across diverse tasks. Benefiting from Transformer structure and U-shaped Connections, our introduced model gains the ability to capture long-range dependencies and effectively up-sample feature maps. Our model achieves superior performance in detecting small objects compared to existing methods. We evaluate CFPFormer on medical image segmentation datasets and object detection benchmarks (VOC 2007, VOC2012, MS-COCO), demonstrating its effectiveness and versatility. On the ACDC Post-2017-MICCAI-Challenge online test set, our model reaches exceptionally impressive accuracy, and performed well compared with the original decoder setting in Synapse multi-organ segmentation dataset.

Title: Large Language Models Spot Phishing Emails with Surprising Accuracy: A Comparative Analysis of Performance

• Authors: Het Patel, Umair Rehman, Farkhund Iqbal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15485
• Pdf URL: https://arxiv.org/pdf/2404.15485
• Copy Paste: [[2404.15485]] Large Language Models Spot Phishing Emails with Surprising Accuracy: A Comparative Analysis of Performance(https://arxiv.org/abs/2404.15485)
• Keywords: security, attack, large language model
• Abstract: Phishing, a prevalent cybercrime tactic for decades, remains a significant threat in today's digital world. By leveraging clever social engineering elements and modern technology, cybercrime targets many individuals, businesses, and organizations to exploit trust and security. These cyber-attackers are often disguised in many trustworthy forms to appear as legitimate sources. By cleverly using psychological elements like urgency, fear, social proof, and other manipulative strategies, phishers can lure individuals into revealing sensitive and personalized information. Building on this pervasive issue within modern technology, this paper aims to analyze the effectiveness of 15 Large Language Models (LLMs) in detecting phishing attempts, specifically focusing on a randomized set of "419 Scam" emails. The objective is to determine which LLMs can accurately detect phishing emails by analyzing a text file containing email metadata based on predefined criteria. The experiment concluded that the following models, ChatGPT 3.5, GPT-3.5-Turbo-Instruct, and ChatGPT, were the most effective in detecting phishing emails.

Title: IryoNLP at MEDIQA-CORR 2024: Tackling the Medical Error Detection & Correction Task On the Shoulders of Medical Agents

• Authors: Jean-Philippe Corbeil
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2404.15488
• Pdf URL: https://arxiv.org/pdf/2404.15488
• Copy Paste: [[2404.15488]] IryoNLP at MEDIQA-CORR 2024: Tackling the Medical Error Detection & Correction Task On the Shoulders of Medical Agents(https://arxiv.org/abs/2404.15488)
• Keywords: large language model
• Abstract: In natural language processing applied to the clinical domain, utilizing large language models has emerged as a promising avenue for error detection and correction on clinical notes, a knowledge-intensive task for which annotated data is scarce. This paper presents MedReAct'N'MedReFlex, which leverages a suite of four LLM-based medical agents. The MedReAct agent initiates the process by observing, analyzing, and taking action, generating trajectories to guide the search to target a potential error in the clinical notes. Subsequently, the MedEval agent employs five evaluators to assess the targeted error and the proposed correction. In cases where MedReAct's actions prove insufficient, the MedReFlex agent intervenes, engaging in reflective analysis and proposing alternative strategies. Finally, the MedFinalParser agent formats the final output, preserving the original style while ensuring the integrity of the error correction process. One core component of our method is our RAG pipeline based on our ClinicalCorp corpora. Among other well-known sources containing clinical guidelines and information, we preprocess and release the open-source MedWiki dataset for clinical RAG application. Our results demonstrate the central role of our RAG approach with ClinicalCorp leveraged through the MedReAct'N'MedReFlex framework. It achieved the ninth rank on the MEDIQA-CORR 2024 final leaderboard.

Title: FedGreen: Carbon-aware Federated Learning with Model Size Adaptation

• Authors: Ali Abbasi, Fan Dong, Xin Wang, Henry Leung, Jiayu Zhou, Steve Drew
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.15503
• Pdf URL: https://arxiv.org/pdf/2404.15503
• Copy Paste: [[2404.15503]] FedGreen: Carbon-aware Federated Learning with Model Size Adaptation(https://arxiv.org/abs/2404.15503)
• Keywords: federate
• Abstract: Federated learning (FL) provides a promising collaborative framework to build a model from distributed clients, and this work investigates the carbon emission of the FL process. Cloud and edge servers hosting FL clients may exhibit diverse carbon footprints influenced by their geographical locations with varying power sources, offering opportunities to reduce carbon emissions by training local models with adaptive computations and communications. In this paper, we propose FedGreen, a carbon-aware FL approach to efficiently train models by adopting adaptive model sizes shared with clients based on their carbon profiles and locations using ordered dropout as a model compression technique. We theoretically analyze the trade-offs between the produced carbon emissions and the convergence accuracy, considering the carbon intensity discrepancy across countries to choose the parameters optimally. Empirical studies show that FedGreen can substantially reduce the carbon footprints of FL compared to the state-of-the-art while maintaining competitive model accuracy.

Title: ToM-LM: Delegating Theory Of Mind Reasoning to External Symbolic Executors in Large Language Models

• Authors: Weizhi Tang, Vaishak Belle
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15515
• Pdf URL: https://arxiv.org/pdf/2404.15515
• Copy Paste: [[2404.15515]] ToM-LM: Delegating Theory Of Mind Reasoning to External Symbolic Executors in Large Language Models(https://arxiv.org/abs/2404.15515)
• Keywords: large language model
• Abstract: Theory of Mind (ToM) refers to the ability of individuals to attribute mental states to others. While Large Language Models (LLMs) have shown some promise with ToM ability, they still struggle with complex ToM reasoning. Our approach leverages an external symbolic executor, specifically the SMCDEL model checker, and fine-tuning to improve the ToM reasoning ability of LLMs. In our approach, an LLM is first fine-tuned through pairs of natural language and symbolic formulation representation of ToM problems and is then instructed to generate the symbolic formulation with a one-shot in-context example. The generated symbolic formulation is then executed by the SMCDEL model checker to perform transparent and verifiable ToM reasoning and give the final result. We demonstrate that our approach, ToM-LM, shows a significant improvement over all the constructed baselines. Our study proposes a novel view about externalizing a particular component of ToM reasoning, mainly reasoning about beliefs, and suggests generalizing it to other aspects of ToM reasoning.

Title: Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval

• Authors: Young Kyun Jang, Donghyun Kim, Zihang Meng, Dat Huynh, Ser-Nam Lim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15516
• Pdf URL: https://arxiv.org/pdf/2404.15516
• Copy Paste: [[2404.15516]] Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval(https://arxiv.org/abs/2404.15516)
• Keywords: large language model
• Abstract: Composed Image Retrieval (CIR) is a task that retrieves images similar to a query, based on a provided textual modification. Current techniques rely on supervised learning for CIR models using labeled triplets of the reference image, text, target image. These specific triplets are not as commonly available as simple image-text pairs, limiting the widespread use of CIR and its scalability. On the other hand, zero-shot CIR can be relatively easily trained with image-caption pairs without considering the image-to-image relation, but this approach tends to yield lower accuracy. We propose a new semi-supervised CIR approach where we search for a reference and its related target images in auxiliary data and learn our large language model-based Visual Delta Generator (VDG) to generate text describing the visual difference (i.e., visual delta) between the two. VDG, equipped with fluent language knowledge and being model agnostic, can generate pseudo triplets to boost the performance of CIR models. Our approach significantly improves the existing supervised learning approaches and achieves state-of-the-art results on the CIR benchmarks.

Title: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models

• Authors: Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15522
• Pdf URL: https://arxiv.org/pdf/2404.15522
• Copy Paste: [[2404.15522]] Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models(https://arxiv.org/abs/2404.15522)
• Keywords: large language model
• Abstract: Recently developed large language models (LLMs) have been shown to perform remarkably well on a wide range of language understanding tasks. But, can they really "reason" over the natural language? This question has been receiving significant research attention and many reasoning skills such as commonsense, numerical, and qualitative have been studied. However, the crucial skill pertaining to 'logical reasoning' has remained underexplored. Existing work investigating this reasoning ability of LLMs has focused only on a couple of inference rules (such as modus ponens and modus tollens) of propositional and first-order logic. Addressing the above limitation, we comprehensively evaluate the logical reasoning ability of LLMs on 25 different reasoning patterns spanning over propositional, first-order, and non-monotonic logics. To enable systematic evaluation, we introduce LogicBench, a natural language question-answering dataset focusing on the use of a single inference rule. We conduct detailed analysis with a range of LLMs such as GPT-4, ChatGPT, Gemini, Llama-2, and Mistral using chain-of-thought prompting. Experimental results show that existing LLMs do not fare well on LogicBench; especially, they struggle with instances involving complex reasoning and negations. Furthermore, they sometimes overlook contextual information necessary for reasoning to arrive at the correct conclusion. We believe that our work and findings facilitate future research for evaluating and enhancing the logical reasoning ability of LLMs. Data and code are available at https://github.com/Mihir3009/LogicBench.

Title: Understanding Hyperbolic Metric Learning through Hard Negative Sampling

• Authors: Yun Yue, Fangzhou Lin, Guanyi Mou, Ziming Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15523
• Pdf URL: https://arxiv.org/pdf/2404.15523
• Copy Paste: [[2404.15523]] Understanding Hyperbolic Metric Learning through Hard Negative Sampling(https://arxiv.org/abs/2404.15523)
• Keywords: transformer
• Abstract: In recent years, there has been a growing trend of incorporating hyperbolic geometry methods into computer vision. While these methods have achieved state-of-the-art performance on various metric learning tasks using hyperbolic distance measurements, the underlying theoretical analysis supporting this superior performance remains under-exploited. In this study, we investigate the effects of integrating hyperbolic space into metric learning, particularly when training with contrastive loss. We identify a need for a comprehensive comparison between Euclidean and hyperbolic spaces regarding the temperature effect in the contrastive loss within the existing literature. To address this gap, we conduct an extensive investigation to benchmark the results of Vision Transformers (ViTs) using a hybrid objective function that combines loss from Euclidean and hyperbolic spaces. Additionally, we provide a theoretical analysis of the observed performance improvement. We also reveal that hyperbolic metric learning is highly related to hard negative sampling, providing insights for future work. This work will provide valuable data points and experience in understanding hyperbolic image embeddings. To shed more light on problem-solving and encourage further investigation into our approach, our code is available online (https://github.com/YunYunY/HypMix).

Title: PRISM: Patient Records Interpretation for Semantic Clinical Trial Matching using Large Language Models

• Authors: Shashi Kant Gupta, Aditya Basu, Mauro Nievas, Jerrin Thomas, Nathan Wolfrath, Adhitya Ramamurthi, Bradley Taylor, Anai N. Kothari, Therica M. Miller, Sorena Nadaf-Rahrov, Yanshan Wang, Hrituraj Singh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15549
• Pdf URL: https://arxiv.org/pdf/2404.15549
• Copy Paste: [[2404.15549]] PRISM: Patient Records Interpretation for Semantic Clinical Trial Matching using Large Language Models(https://arxiv.org/abs/2404.15549)
• Keywords: large language model
• Abstract: Clinical trial matching is the task of identifying trials for which patients may be potentially eligible. Typically, this task is labor-intensive and requires detailed verification of patient electronic health records (EHRs) against the stringent inclusion and exclusion criteria of clinical trials. This process is manual, time-intensive, and challenging to scale up, resulting in many patients missing out on potential therapeutic options. Recent advancements in Large Language Models (LLMs) have made automating patient-trial matching possible, as shown in multiple concurrent research studies. However, the current approaches are confined to constrained, often synthetic datasets that do not adequately mirror the complexities encountered in real-world medical data. In this study, we present the first, end-to-end large-scale empirical evaluation of clinical trial matching using real-world EHRs. Our study showcases the capability of LLMs to accurately match patients with appropriate clinical trials. We perform experiments with proprietary LLMs, including GPT-4 and GPT-3.5, as well as our custom fine-tuned model called OncoLLM and show that OncoLLM, despite its significantly smaller size, not only outperforms GPT-3.5 but also matches the performance of qualified medical doctors. All experiments were carried out on real-world EHRs that include clinical notes and available clinical trials from a single cancer center in the United States.

Title: Cross-Temporal Spectrogram Autoencoder (CTSAE): Unsupervised Dimensionality Reduction for Clustering Gravitational Wave Glitches

• Authors: Yi Li, Yunan Wu, Aggelos K. Katsaggelos
• Subjects: cs.CV, astro-ph.IM, cs.LG, gr-qc
• Abstract URL: https://arxiv.org/abs/2404.15552
• Pdf URL: https://arxiv.org/pdf/2404.15552
• Copy Paste: [[2404.15552]] Cross-Temporal Spectrogram Autoencoder (CTSAE): Unsupervised Dimensionality Reduction for Clustering Gravitational Wave Glitches(https://arxiv.org/abs/2404.15552)
• Keywords: transformer
• Abstract: The advancement of The Laser Interferometer Gravitational-Wave Observatory (LIGO) has significantly enhanced the feasibility and reliability of gravitational wave detection. However, LIGO's high sensitivity makes it susceptible to transient noises known as glitches, which necessitate effective differentiation from real gravitational wave signals. Traditional approaches predominantly employ fully supervised or semi-supervised algorithms for the task of glitch classification and clustering. In the future task of identifying and classifying glitches across main and auxiliary channels, it is impractical to build a dataset with manually labeled ground-truth. In addition, the patterns of glitches can vary with time, generating new glitches without manual labels. In response to this challenge, we introduce the Cross-Temporal Spectrogram Autoencoder (CTSAE), a pioneering unsupervised method for the dimensionality reduction and clustering of gravitational wave glitches. CTSAE integrates a novel four-branch autoencoder with a hybrid of Convolutional Neural Networks (CNN) and Vision Transformers (ViT). To further extract features across multi-branches, we introduce a novel multi-branch fusion method using the CLS (Class) token. Our model, trained and evaluated on the GravitySpy O3 dataset on the main channel, demonstrates superior performance in clustering tasks when compared to state-of-the-art semi-supervised learning methods. To the best of our knowledge, CTSAE represents the first unsupervised approach tailored specifically for clustering LIGO data, marking a significant step forward in the field of gravitational wave research. The code of this paper is available at https://github.com/Zod-L/CTSAE

Title: Retrieval Head Mechanistically Explains Long-Context Factuality

• Authors: Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, Yao Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15574
• Pdf URL: https://arxiv.org/pdf/2404.15574
• Copy Paste: [[2404.15574]] Retrieval Head Mechanistically Explains Long-Context Factuality(https://arxiv.org/abs/2404.15574)
• Keywords: transformer
• Abstract: Despite the recent progress in long-context language models, it remains elusive how transformer-based models exhibit the capability to retrieve relevant information from arbitrary locations within the long context. This paper aims to address this question. Our systematic investigation across a wide spectrum of models reveals that a special type of attention heads are largely responsible for retrieving information, which we dub retrieval heads. We identify intriguing properties of retrieval heads:(1) universal: all the explored models with long-context capability have a set of retrieval heads; (2) sparse: only a small portion (less than 5\%) of the attention heads are retrieval. (3) intrinsic: retrieval heads already exist in models pretrained with short context. When extending the context length by continual pretraining, it is still the same set of heads that perform information retrieval. (4) dynamically activated: take Llama-2 7B for example, 12 retrieval heads always attend to the required information no matter how the context is changed. The rest of the retrieval heads are activated in different contexts. (5) causal: completely pruning retrieval heads leads to failure in retrieving relevant information and results in hallucination, while pruning random non-retrieval heads does not affect the model's retrieval ability. We further show that retrieval heads strongly influence chain-of-thought (CoT) reasoning, where the model needs to frequently refer back the question and previously-generated context. Conversely, tasks where the model directly generates the answer using its intrinsic knowledge are less impacted by masking out retrieval heads. These observations collectively explain which internal part of the model seeks information from the input tokens. We believe our insights will foster future research on reducing hallucination, improving reasoning, and compressing the KV cache.

Title: Can Foundational Large Language Models Assist with Conducting Pharmaceuticals Manufacturing Investigations?

• Authors: Hossein Salami (1), Brandye Smith-Goettler (2), Vijay Yadav (2) ((1) Digital Services, MMD, Merck & Co., Inc., Rahway, NJ, USA, (2) Digital Services, MMD, Merck & Co., Inc., West Point, PA, USA)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15578
• Pdf URL: https://arxiv.org/pdf/2404.15578
• Copy Paste: [[2404.15578]] Can Foundational Large Language Models Assist with Conducting Pharmaceuticals Manufacturing Investigations?(https://arxiv.org/abs/2404.15578)
• Keywords: extraction, transformer, generative, large language model
• Abstract: General purpose Large Language Models (LLM) such as the Generative Pretrained Transformer (GPT) and Large Language Model Meta AI (LLaMA) have attracted much attention in recent years. There is strong evidence that these models can perform remarkably well in various natural language processing tasks. However, how to leverage them to approach domain-specific use cases and drive value remains an open question. In this work, we focus on a specific use case, pharmaceutical manufacturing investigations, and propose that leveraging historical records of manufacturing incidents and deviations in an organization can be beneficial for addressing and closing new cases, or de-risking new manufacturing campaigns. Using a small but diverse dataset of real manufacturing deviations selected from different product lines, we evaluate and quantify the power of three general purpose LLMs (GPT-3.5, GPT-4, and Claude-2) in performing tasks related to the above goal. In particular, (1) the ability of LLMs in automating the process of extracting specific information such as root cause of a case from unstructured data, as well as (2) the possibility of identifying similar or related deviations by performing semantic search on the database of historical records are examined. While our results point to the high accuracy of GPT-4 and Claude-2 in the information extraction task, we discuss cases of complex interplay between the apparent reasoning and hallucination behavior of LLMs as a risk factor. Furthermore, we show that semantic search on vector embedding of deviation descriptions can be used to identify similar records, such as those with a similar type of defect, with a high level of accuracy. We discuss further improvements to enhance the accuracy of similar record identification.

Title: MiM: Mask in Mask Self-Supervised Pre-Training for 3D Medical Image Analysis

• Authors: Jiaxin Zhuang, Linshan Wu, Qiong Wang, Varut Vardhanabhuti, Lin Luo, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15580
• Pdf URL: https://arxiv.org/pdf/2404.15580
• Copy Paste: [[2404.15580]] MiM: Mask in Mask Self-Supervised Pre-Training for 3D Medical Image Analysis(https://arxiv.org/abs/2404.15580)
• Keywords: transformer, segmentation
• Abstract: The Vision Transformer (ViT) has demonstrated remarkable performance in Self-Supervised Learning (SSL) for 3D medical image analysis. Mask AutoEncoder (MAE) for feature pre-training can further unleash the potential of ViT on various medical vision tasks. However, due to large spatial sizes with much higher dimensions of 3D medical images, the lack of hierarchical design for MAE may hinder the performance of downstream tasks. In this paper, we propose a novel \textit{Mask in Mask (MiM)} pre-training framework for 3D medical images, which aims to advance MAE by learning discriminative representation from hierarchical visual tokens across varying scales. We introduce multiple levels of granularity for masked inputs from the volume, which are then reconstructed simultaneously ranging at both fine and coarse levels. Additionally, a cross-level alignment mechanism is applied to adjacent level volumes to enforce anatomical similarity hierarchically. Furthermore, we adopt a hybrid backbone to enhance the hierarchical representation learning efficiently during the pre-training. MiM was pre-trained on a large scale of available 3D volumetric images, \textit{i.e.,} Computed Tomography (CT) images containing various body parts. Extensive experiments on thirteen public datasets demonstrate the superiority of MiM over other SSL methods in organ/lesion/tumor segmentation and disease classification. We further scale up the MiM to large pre-training datasets with more than 10k volumes, showing that large-scale pre-training can further enhance the performance of downstream tasks. The improvement also concluded that the research community should pay more attention to the scale of the pre-training dataset towards the healthcare foundation model for 3D medical images.

Title: Armored Core of PKI: Remove Signing Keys for CA via Physically Unclonable Function

• Authors: Xiaolin Zhang, Chenghao Chen, Kailun Qin, Chi Zhang, Shipei Qu, Tengfei Wang, Yuxuan Wang, Dawu Gu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15582
• Pdf URL: https://arxiv.org/pdf/2404.15582
• Copy Paste: [[2404.15582]] Armored Core of PKI: Remove Signing Keys for CA via Physically Unclonable Function(https://arxiv.org/abs/2404.15582)
• Keywords: security, protect, attack
• Abstract: The protection of CA's signing keys is one of the most crucial security concerns in PKI. However, these keys can still be exposed today by human errors or various carefully designed attacks. Traditional protections like TEE and HSM fail to eliminate this risk since they can be bypassed by skilled attackers. This dilemma motivates us to consider removing CA' signing keys and propose Armored Core, a PKI security extension applying the physically trusted binding provided by Physically Unclonable Function (PUF) for CA. CAs in Armored Core issue PUF-based X509v3 TLS certificates, where they use PUF instead of signing algorithms to generate endorsements for domain public keys. The new transparency logging mechanism, built upon CT, will record the PUF calling behaviors of CA, ensuring the monitoring of PUF usage. We provide a formal cryptographic proof of Armored Core's main functions. We also implement it on the real-world PKI codebase. The results show that the incorporation of Armored Core into original systems do not cause any extra overhead, but instead improves computing efficiency by >4.9% and saves >20% of certificate storage.

Title: Brain Storm Optimization Based Swarm Learning for Diabetic Retinopathy Image Classification

• Authors: Liang Qu, Cunze Wang, Yuhui Shi
• Subjects: cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.15585
• Pdf URL: https://arxiv.org/pdf/2404.15585
• Copy Paste: [[2404.15585]] Brain Storm Optimization Based Swarm Learning for Diabetic Retinopathy Image Classification(https://arxiv.org/abs/2404.15585)
• Keywords: privacy, federate
• Abstract: The application of deep learning techniques to medical problems has garnered widespread research interest in recent years, such as applying convolutional neural networks to medical image classification tasks. However, data in the medical field is often highly private, preventing different hospitals from sharing data to train an accurate model. Federated learning, as a privacy-preserving machine learning architecture, has shown promising performance in balancing data privacy and model utility by keeping private data on the client's side and using a central server to coordinate a set of clients for model training through aggregating their uploaded model parameters. Yet, this architecture heavily relies on a trusted third-party server, which is challenging to achieve in real life. Swarm learning, as a specialized decentralized federated learning architecture that does not require a central server, utilizes blockchain technology to enable direct parameter exchanges between clients. However, the mining of blocks requires significant computational resources, limiting its scalability. To address this issue, this paper integrates the brain storm optimization algorithm into the swarm learning framework, named BSO-SL. This approach clusters similar clients into different groups based on their model distributions. Additionally, leveraging the architecture of BSO, clients are given the probability to engage in collaborative learning both within their cluster and with clients outside their cluster, preventing the model from converging to local optima. The proposed method has been validated on a real-world diabetic retinopathy image classification dataset, and the experimental results demonstrate the effectiveness of the proposed approach.

Title: Security Analysis of WiFi-based Sensing Systems: Threats from Perturbation Attacks

• Authors: Hangcheng Cao, Wenbin Huang, Guowen Xu, Xianhao Chen, Ziyang He, Jingyang Hu, Hongbo Jiang, Yuguang Fang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15587
• Pdf URL: https://arxiv.org/pdf/2404.15587
• Copy Paste: [[2404.15587]] Security Analysis of WiFi-based Sensing Systems: Threats from Perturbation Attacks(https://arxiv.org/abs/2404.15587)
• Keywords: security, attack, robust, steal, generative
• Abstract: Deep learning technologies are pivotal in enhancing the performance of WiFi-based wireless sensing systems. However, they are inherently vulnerable to adversarial perturbation attacks, and regrettably, there is lacking serious attention to this security issue within the WiFi sensing community. In this paper, we elaborate such an attack, called WiIntruder, distinguishing itself with universality, robustness, and stealthiness, which serves as a catalyst to assess the security of existing WiFi-based sensing systems. This attack encompasses the following salient features: (1) Maximizing transferability by differentiating user-state-specific feature spaces across sensing models, leading to a universally effective perturbation attack applicable to common applications; (2) Addressing perturbation signal distortion caused by device synchronization and wireless propagation when critical parameters are optimized through a heuristic particle swarm-driven perturbation generation algorithm; and (3) Enhancing attack pattern diversity and stealthiness through random switching of perturbation surrogates generated by a generative adversarial network. Extensive experimental results confirm the practical threats of perturbation attacks to common WiFi-based services, including user authentication and respiratory monitoring.

Title: ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction

• Authors: Henry Peng Zou, Vinay Samuel, Yue Zhou, Weizhi Zhang, Liancheng Fang, Zihe Song, Philip S. Yu, Cornelia Caragea
• Subjects: cs.CV, cs.AI, cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15592
• Pdf URL: https://arxiv.org/pdf/2404.15592
• Copy Paste: [[2404.15592]] ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction(https://arxiv.org/abs/2404.15592)
• Keywords: extraction, large language model
• Abstract: Existing datasets for attribute value extraction (AVE) predominantly focus on explicit attribute values while neglecting the implicit ones, lack product images, are often not publicly available, and lack an in-depth human inspection across diverse domains. To address these limitations, we present ImplicitAVE, the first, publicly available multimodal dataset for implicit attribute value extraction. ImplicitAVE, sourced from the MAVE dataset, is carefully curated and expanded to include implicit AVE and multimodality, resulting in a refined dataset of 68k training and 1.6k testing data across five domains. We also explore the application of multimodal large language models (MLLMs) to implicit AVE, establishing a comprehensive benchmark for MLLMs on the ImplicitAVE dataset. Six recent MLLMs with eleven variants are evaluated across diverse settings, revealing that implicit value extraction remains a challenging task for MLLMs. The contributions of this work include the development and release of ImplicitAVE, and the exploration and benchmarking of various MLLMs for implicit AVE, providing valuable insights and potential future research directions. Dataset and code are available at https://github.com/HenryPengZou/ImplicitAVE

Title: Federated Learning with Only Positive Labels by Exploring Label Correlations

• Authors: Xuming An, Dui Wang, Li Shen, Yong Luo, Han Hu, Bo Du, Yonggang Wen, Dacheng Tao
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15598
• Pdf URL: https://arxiv.org/pdf/2404.15598
• Copy Paste: [[2404.15598]] Federated Learning with Only Positive Labels by Exploring Label Correlations(https://arxiv.org/abs/2404.15598)
• Keywords: privacy, federate
• Abstract: Federated learning aims to collaboratively learn a model by using the data from multiple users under privacy constraints. In this paper, we study the multi-label classification problem under the federated learning setting, where trivial solution and extremely poor performance may be obtained, especially when only positive data w.r.t. a single class label are provided for each client. This issue can be addressed by adding a specially designed regularizer on the server-side. Although effective sometimes, the label correlations are simply ignored and thus sub-optimal performance may be obtained. Besides, it is expensive and unsafe to exchange user's private embeddings between server and clients frequently, especially when training model in the contrastive way. To remedy these drawbacks, we propose a novel and generic method termed Federated Averaging by exploring Label Correlations (FedALC). Specifically, FedALC estimates the label correlations in the class embedding learning for different label pairs and utilizes it to improve the model training. To further improve the safety and also reduce the communication overhead, we propose a variant to learn fixed class embedding for each client, so that the server and clients only need to exchange class embeddings once. Extensive experiments on multiple popular datasets demonstrate that our FedALC can significantly outperform existing counterparts.

Title: Hybrid LLM/Rule-based Approaches to Business Insights Generation from Structured Data

• Authors: Aliaksei Vertsel, Mikhail Rumiantsau
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15604
• Pdf URL: https://arxiv.org/pdf/2404.15604
• Copy Paste: [[2404.15604]] Hybrid LLM/Rule-based Approaches to Business Insights Generation from Structured Data(https://arxiv.org/abs/2404.15604)
• Keywords: robust, large language model
• Abstract: In the field of business data analysis, the ability to extract actionable insights from vast and varied datasets is essential for informed decision-making and maintaining a competitive edge. Traditional rule-based systems, while reliable, often fall short when faced with the complexity and dynamism of modern business data. Conversely, Artificial Intelligence (AI) models, particularly Large Language Models (LLMs), offer significant potential in pattern recognition and predictive analytics but can lack the precision necessary for specific business applications. This paper explores the efficacy of hybrid approaches that integrate the robustness of rule-based systems with the adaptive power of LLMs in generating actionable business insights.

Title: Understanding and Improving CNNs with Complex Structure Tensor: A Biometrics Study

• Authors: Kevin Hernandez-Diaz, Josef Bigun, Fernando Alonso-Fernandez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15608
• Pdf URL: https://arxiv.org/pdf/2404.15608
• Copy Paste: [[2404.15608]] Understanding and Improving CNNs with Complex Structure Tensor: A Biometrics Study(https://arxiv.org/abs/2404.15608)
• Keywords: biometric, explainability
• Abstract: Our study provides evidence that CNNs struggle to effectively extract orientation features. We show that the use of Complex Structure Tensor, which contains compact orientation features with certainties, as input to CNNs consistently improves identification accuracy compared to using grayscale inputs alone. Experiments also demonstrated that our inputs, which were provided by mini complex conv-nets, combined with reduced CNN sizes, outperformed full-fledged, prevailing CNN architectures. This suggests that the upfront use of orientation features in CNNs, a strategy seen in mammalian vision, not only mitigates their limitations but also enhances their explainability and relevance to thin-clients. Experiments were done on publicly available data sets comprising periocular images for biometric identification and verification (Close and Open World) using 6 State of the Art CNN architectures. We reduced SOA Equal Error Rate (EER) on the PolyU dataset by 5-26% depending on data and scenario.

Title: PoisonedFL: Model Poisoning Attacks to Federated Learning via Multi-Round Consistency

• Authors: Yueqi Xie, Minghong Fang, Neil Zhenqiang Gong
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15611
• Pdf URL: https://arxiv.org/pdf/2404.15611
• Copy Paste: [[2404.15611]] PoisonedFL: Model Poisoning Attacks to Federated Learning via Multi-Round Consistency(https://arxiv.org/abs/2404.15611)
• Keywords: security, defense, attack, robust, federate
• Abstract: Model poisoning attacks are critical security threats to Federated Learning (FL). Existing model poisoning attacks suffer from two key limitations: 1) they achieve suboptimal effectiveness when defenses are deployed, and/or 2) they require knowledge of the model updates or local training data on genuine clients. In this work, we make a key observation that their suboptimal effectiveness arises from only leveraging model-update consistency among malicious clients within individual training rounds, making the attack effect self-cancel across training rounds. In light of this observation, we propose PoisonedFL, which enforces multi-round consistency among the malicious clients' model updates while not requiring any knowledge about the genuine clients. Our empirical evaluation on five benchmark datasets shows that PoisonedFL breaks eight state-of-the-art defenses and outperforms seven existing model poisoning attacks. Moreover, we also explore new defenses that are tailored to PoisonedFL, but our results show that we can still adapt PoisonedFL to break them. Our study shows that FL systems are considerably less robust than previously thought, underlining the urgency for the development of new defense mechanisms.

Title: Optimizing OOD Detection in Molecular Graphs: A Novel Approach with Diffusion Models

• Authors: Xu Shen, Yili Wang, Kaixiong Zhou, Shirui Pan, Xin Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15625
• Pdf URL: https://arxiv.org/pdf/2404.15625
• Copy Paste: [[2404.15625]] Optimizing OOD Detection in Molecular Graphs: A Novel Approach with Diffusion Models(https://arxiv.org/abs/2404.15625)
• Keywords: diffusion, generative
• Abstract: The open-world test dataset is often mixed with out-of-distribution (OOD) samples, where the deployed models will struggle to make accurate predictions. Traditional detection methods need to trade off OOD detection and in-distribution (ID) classification performance since they share the same representation learning model. In this work, we propose to detect OOD molecules by adopting an auxiliary diffusion model-based framework, which compares similarities between input molecules and reconstructed graphs. Due to the generative bias towards reconstructing ID training samples, the similarity scores of OOD molecules will be much lower to facilitate detection. Although it is conceptually simple, extending this vanilla framework to practical detection applications is still limited by two significant challenges. First, the popular similarity metrics based on Euclidian distance fail to consider the complex graph structure. Second, the generative model involving iterative denoising steps is time-consuming especially when it runs on the enormous pool of drugs. To address these challenges, our research pioneers an approach of Prototypical Graph Reconstruction for Molecular OOD Detection, dubbed as PGR-MOOD and hinges on three innovations: i) An effective metric to comprehensively quantify the matching degree of input and reconstructed molecules; ii) A creative graph generator to construct prototypical graphs that are in line with ID but away from OOD; iii) An efficient and scalable OOD detector to compare the similarity between test samples and pre-constructed prototypical graphs and omit the generative process on every new molecule. Extensive experiments on ten benchmark datasets and six baselines are conducted to demonstrate our superiority.

Title: A Real-time Evaluation Framework for Pedestrian's Potential Risk at Non-Signalized Intersections Based on Predicted Post-Encroachment Time

• Authors: Tengfeng Lin, Zhixiong Jin, Seongjin Choi, Hwasoo Yeo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15635
• Pdf URL: https://arxiv.org/pdf/2404.15635
• Copy Paste: [[2404.15635]] A Real-time Evaluation Framework for Pedestrian's Potential Risk at Non-Signalized Intersections Based on Predicted Post-Encroachment Time(https://arxiv.org/abs/2404.15635)
• Keywords: protect
• Abstract: Addressing pedestrian safety at intersections is one of the paramount concerns in the field of transportation research, driven by the urgency of reducing traffic-related injuries and fatalities. With advances in computer vision technologies and predictive models, the pursuit of developing real-time proactive protection systems is increasingly recognized as vital to improving pedestrian safety at intersections. The core of these protection systems lies in the prediction-based evaluation of pedestrian's potential risks, which plays a significant role in preventing the occurrence of accidents. The major challenges in the current prediction-based potential risk evaluation research can be summarized into three aspects: the inadequate progress in creating a real-time framework for the evaluation of pedestrian's potential risks, the absence of accurate and explainable safety indicators that can represent the potential risk, and the lack of tailor-made evaluation criteria specifically for each category of pedestrians. To address these research challenges, in this study, a framework with computer vision technologies and predictive models is developed to evaluate the potential risk of pedestrians in real time. Integral to this framework is a novel surrogate safety measure, the Predicted Post-Encroachment Time (P-PET), derived from deep learning models capable to predict the arrival time of pedestrians and vehicles at intersections. To further improve the effectiveness and reliability of pedestrian risk evaluation, we classify pedestrians into distinct categories and apply specific evaluation criteria for each group. The results demonstrate the framework's ability to effectively identify potential risks through the use of P-PET, indicating its feasibility for real-time applications and its improved performance in risk evaluation across different categories of pedestrians.

Title: PriorNet: A Novel Lightweight Network with Multidimensional Interactive Attention for Efficient Image Dehazing

• Authors: Yutong Chen, Zhang Wen, Chao Wang, Lei Gong, Zhongchao Yi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15638
• Pdf URL: https://arxiv.org/pdf/2404.15638
• Copy Paste: [[2404.15638]] PriorNet: A Novel Lightweight Network with Multidimensional Interactive Attention for Efficient Image Dehazing(https://arxiv.org/abs/2404.15638)
• Keywords: extraction
• Abstract: Hazy images degrade visual quality, and dehazing is a crucial prerequisite for subsequent processing tasks. Most current dehazing methods rely on neural networks and face challenges such as high computational parameter pressure and weak generalization capabilities. This paper introduces PriorNet--a novel, lightweight, and highly applicable dehazing network designed to significantly improve the clarity and visual quality of hazy images while avoiding excessive detail extraction issues. The core of PriorNet is the original Multi-Dimensional Interactive Attention (MIA) mechanism, which effectively captures a wide range of haze characteristics, substantially reducing the computational load and generalization difficulties associated with complex systems. By utilizing a uniform convolutional kernel size and incorporating skip connections, we have streamlined the feature extraction process. Simplifying the number of layers and architecture not only enhances dehazing efficiency but also facilitates easier deployment on edge devices. Extensive testing across multiple datasets has demonstrated PriorNet's exceptional performance in dehazing and clarity restoration, maintaining image detail and color fidelity in single-image dehazing tasks. Notably, with a model size of just 18Kb, PriorNet showcases superior dehazing generalization capabilities compared to other methods. Our research makes a significant contribution to advancing image dehazing technology, providing new perspectives and tools for the field and related domains, particularly emphasizing the importance of improving universality and deployability.

Title: CodeIP: A Grammar-Guided Multi-Bit Watermark for Large Language Models of Code

• Authors: Batu Guan, Yao Wan, Zhangqian Bi, Zheng Wang, Hongyu Zhang, Yulei Sui, Pan Zhou, Lichao Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15639
• Pdf URL: https://arxiv.org/pdf/2404.15639
• Copy Paste: [[2404.15639]] CodeIP: A Grammar-Guided Multi-Bit Watermark for Large Language Models of Code(https://arxiv.org/abs/2404.15639)
• Keywords: protect, watermark, large language model
• Abstract: As Large Language Models (LLMs) are increasingly used to automate code generation, it is often desired to know if the code is AI-generated and by which model, especially for purposes like protecting intellectual property (IP) in industry and preventing academic misconduct in education. Incorporating watermarks into machine-generated content is one way to provide code provenance, but existing solutions are restricted to a single bit or lack flexibility. We present CodeIP, a new watermarking technique for LLM-based code generation. CodeIP enables the insertion of multi-bit information while preserving the semantics of the generated code, improving the strength and diversity of the inerseted watermark. This is achieved by training a type predictor to predict the subsequent grammar type of the next token to enhance the syntactical and semantic correctness of the generated code. Experiments on a real-world dataset across five programming languages showcase the effectiveness of CodeIP.

Title: Return of EM: Entity-driven Answer Set Expansion for QA Evaluation

• Authors: Dongryeol Lee, Minwoo Lee, Kyungmin Min, Joonsuk Park, Kyomin Jung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15650
• Pdf URL: https://arxiv.org/pdf/2404.15650
• Copy Paste: [[2404.15650]] Return of EM: Entity-driven Answer Set Expansion for QA Evaluation(https://arxiv.org/abs/2404.15650)
• Keywords: interpretability, large language model
• Abstract: Recently, directly using large language models (LLMs) has been shown to be the most reliable method to evaluate QA models. However, it suffers from limited interpretability, high cost, and environmental harm. To address these, we propose to use soft EM with entity-driven answer set expansion. Our approach expands the gold answer set to include diverse surface forms, based on the observation that the surface forms often follow particular patterns depending on the entity type. The experimental results show that our method outperforms traditional evaluation methods by a large margin. Moreover, the reliability of our evaluation method is comparable to that of LLM-based ones, while offering the benefits of high interpretability and reduced environmental harm.

Title: CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data

• Authors: Sachin Mehta, Maxwell Horton, Fartash Faghri, Mohammad Hossein Sekhavat, Mahyar Najibi, Mehrdad Farajtabar, Oncel Tuzel, Mohammad Rastegari
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15653
• Pdf URL: https://arxiv.org/pdf/2404.15653
• Copy Paste: [[2404.15653]] CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data(https://arxiv.org/abs/2404.15653)
• Keywords: segmentation
• Abstract: Contrastive learning has emerged as a transformative method for learning effective visual representations through the alignment of image and text embeddings. However, pairwise similarity computation in contrastive loss between image and text pairs poses computational challenges. This paper presents a novel weakly supervised pre-training of vision models on web-scale image-text data. The proposed method reframes pre-training on image-text data as a classification task. Consequently, it eliminates the need for pairwise similarity computations in contrastive loss, achieving a remarkable $2.7\times$ acceleration in training speed compared to contrastive learning on web-scale data. Through extensive experiments spanning diverse vision tasks, including detection and segmentation, we demonstrate that the proposed method maintains high representation quality. Our source code along with pre-trained model weights and training recipes is available at \url{https://github.com/apple/corenet}.

Title: Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering

• Authors: Jiawei Yao, Qi Qian, Juhua Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15655
• Pdf URL: https://arxiv.org/pdf/2404.15655
• Copy Paste: [[2404.15655]] Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering(https://arxiv.org/abs/2404.15655)
• Keywords: large language model
• Abstract: Multiple clustering has gained significant attention in recent years due to its potential to reveal multiple hidden structures of data from different perspectives. The advent of deep multiple clustering techniques has notably advanced the performance by uncovering complex patterns and relationships within large datasets. However, a major challenge arises as users often do not need all the clusterings that algorithms generate, and figuring out the one needed requires a substantial understanding of each clustering result. Traditionally, aligning a user's brief keyword of interest with the corresponding vision components was challenging, but the emergence of multi-modal and large language models (LLMs) has begun to bridge this gap. In response, given unlabeled target visual data, we propose Multi-MaP, a novel method employing a multi-modal proxy learning process. It leverages CLIP encoders to extract coherent text and image embeddings, with GPT-4 integrating users' interests to formulate effective textual contexts. Moreover, reference word constraint and concept-level constraint are designed to learn the optimal text proxy according to the user's interest. Multi-MaP not only adeptly captures a user's interest via a keyword but also facilitates identifying relevant clusterings. Our extensive experiments show that Multi-MaP consistently outperforms state-of-the-art methods in all benchmark multi-clustering vision tasks. Our code is available at https://github.com/Alexander-Yao/Multi-MaP.

Title: MISLEAD: Manipulating Importance of Selected features for Learning Epsilon in Evasion Attack Deception

• Authors: Vidit Khazanchi, Pavan Kulkarni, Yuvaraj Govindarajulu, Manojkumar Parmar
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15656
• Pdf URL: https://arxiv.org/pdf/2404.15656
• Copy Paste: [[2404.15656]] MISLEAD: Manipulating Importance of Selected features for Learning Epsilon in Evasion Attack Deception(https://arxiv.org/abs/2404.15656)
• Keywords: security, attack
• Abstract: Emerging vulnerabilities in machine learning (ML) models due to adversarial attacks raise concerns about their reliability. Specifically, evasion attacks manipulate models by introducing precise perturbations to input data, causing erroneous predictions. To address this, we propose a methodology combining SHapley Additive exPlanations (SHAP) for feature importance analysis with an innovative Optimal Epsilon technique for conducting evasion attacks. Our approach begins with SHAP-based analysis to understand model vulnerabilities, crucial for devising targeted evasion strategies. The Optimal Epsilon technique, employing a Binary Search algorithm, efficiently determines the minimum epsilon needed for successful evasion. Evaluation across diverse machine learning architectures demonstrates the technique's precision in generating adversarial samples, underscoring its efficacy in manipulating model outcomes. This study emphasizes the critical importance of continuous assessment and monitoring to identify and mitigate potential security risks in machine learning systems.

Title: FedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantification

• Authors: Hui Chen, Hengyu Liu, Zhangkai Wu, Xuhui Fan, Longbing Cao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15657
• Pdf URL: https://arxiv.org/pdf/2404.15657
• Copy Paste: [[2404.15657]] FedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantification(https://arxiv.org/abs/2404.15657)
• Keywords: federate
• Abstract: While deep neural networks (DNNs) based personalized federated learning (PFL) is demanding for addressing data heterogeneity and shows promising performance, existing methods for federated learning (FL) suffer from efficient systematic uncertainty quantification. The Bayesian DNNs-based PFL is usually questioned of either over-simplified model structures or high computational and memory costs. In this paper, we introduce FedSI, a novel Bayesian DNNs-based subnetwork inference PFL framework. FedSI is simple and scalable by leveraging Bayesian methods to incorporate systematic uncertainties effectively. It implements a client-specific subnetwork inference mechanism, selects network parameters with large variance to be inferred through posterior distributions, and fixes the rest as deterministic ones. FedSI achieves fast and scalable inference while preserving the systematic uncertainties to the fullest extent. Extensive experiments on three different benchmark datasets demonstrate that FedSI outperforms existing Bayesian and non-Bayesian FL baselines in heterogeneous FL scenarios.

Title: KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering

• Authors: Xinxin Zheng, Feihu Che, Jinyang Wu, Shuai Zhang, Shuai Nie, Kang Liu, Jianhua Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15660
• Pdf URL: https://arxiv.org/pdf/2404.15660
• Copy Paste: [[2404.15660]] KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering(https://arxiv.org/abs/2404.15660)
• Keywords: large language model
• Abstract: Large language models (LLMs) suffer from the hallucination problem and face significant challenges when applied to knowledge-intensive tasks. A promising approach is to leverage evidence documents as extra supporting knowledge, which can be obtained through retrieval or generation. However, existing methods directly leverage the entire contents of the evidence document, which may introduce noise information and impair the performance of large language models. To tackle this problem, we propose a novel Knowledge Selection of Large Language Models (KS-LLM) method, aiming to identify valuable information from evidence documents. The KS-LLM approach utilizes triples to effectively select knowledge snippets from evidence documents that are beneficial to answering questions. Specifically, we first generate triples based on the input question, then select the evidence sentences most similar to triples from the evidence document, and finally combine the evidence sentences and triples to assist large language models in generating answers. Experimental comparisons on several question answering datasets, such as TriviaQA, WebQ, and NQ, demonstrate that the proposed method surpasses the baselines and achieves the best results.

Title: The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews

• Authors: Aleksi Huotala, Miikka Kuutila, Paul Ralph, Mika Mäntylä
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15667
• Pdf URL: https://arxiv.org/pdf/2404.15667
• Copy Paste: [[2404.15667]] The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews(https://arxiv.org/abs/2404.15667)
• Keywords: large language model
• Abstract: Systematic review (SR) is a popular research method in software engineering (SE). However, conducting an SR takes an average of 67 weeks. Thus, automating any step of the SR process could reduce the effort associated with SRs. Our objective is to investigate if Large Language Models (LLMs) can accelerate title-abstract screening by simplifying abstracts for human screeners, and automating title-abstract screening. We performed an experiment where humans screened titles and abstracts for 20 papers with both original and simplified abstracts from a prior SR. The experiment with human screeners was reproduced with GPT-3.5 and GPT-4 LLMs to perform the same screening tasks. We also studied if different prompting techniques (Zero-shot (ZS), One-shot (OS), Few-shot (FS), and Few-shot with Chain-of-Thought (FS-CoT)) improve the screening performance of LLMs. Lastly, we studied if redesigning the prompt used in the LLM reproduction of screening leads to improved performance. Text simplification did not increase the screeners' screening performance, but reduced the time used in screening. Screeners' scientific literacy skills and researcher status predict screening performance. Some LLM and prompt combinations perform as well as human screeners in the screening tasks. Our results indicate that the GPT-4 LLM is better than its predecessor, GPT-3.5. Additionally, Few-shot and One-shot prompting outperforms Zero-shot prompting. Using LLMs for text simplification in the screening process does not significantly improve human performance. Using LLMs to automate title-abstract screening seems promising, but current LLMs are not significantly more accurate than human screeners. To recommend the use of LLMs in the screening process of SRs, more research is needed. We recommend future SR studies publish replication packages with screening data to enable more conclusive experimenting with LLM screening.

Title: Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision

• Authors: Mohammad Reza Hosseinzadeh Taher, Michael B. Gotway, Jianming Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15672
• Pdf URL: https://arxiv.org/pdf/2404.15672
• Copy Paste: [[2404.15672]] Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision(https://arxiv.org/abs/2404.15672)
• Keywords: robust
• Abstract: Humans effortlessly interpret images by parsing them into part-whole hierarchies; deep learning excels in learning multi-level feature spaces, but they often lack explicit coding of part-whole relations, a prominent property of medical imaging. To overcome this limitation, we introduce Adam-v2, a new self-supervised learning framework extending Adam [79] by explicitly incorporating part-whole hierarchies into its learning objectives through three key branches: (1) Localizability, acquiring discriminative representations to distinguish different anatomical patterns; (2) Composability, learning each anatomical structure in a parts-to-whole manner; and (3) Decomposability, comprehending each anatomical structure in a whole-to-parts manner. Experimental results across 10 tasks, compared to 11 baselines in zero-shot, few-shot transfer, and full fine-tuning settings, showcase Adam-v2's superior performance over large-scale medical models and existing SSL methods across diverse downstream tasks. The higher generality and robustness of Adam-v2's representations originate from its explicit construction of hierarchies for distinct anatomical structures from unlabeled medical images. Adam-v2 preserves a semantic balance of anatomical diversity and harmony in its embedding, yielding representations that are both generic and semantically meaningful, yet overlooked in existing SSL methods. All code and pretrained models are available at https://github.com/JLiangLab/Eden.

Title: Augmented CARDS: A machine learning approach to identifying triggers of climate change misinformation on Twitter

• Authors: Cristian Rojas, Frank Algra-Maschio, Mark Andrejevic, Travis Coan, John Cook, Yuan-Fang Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15673
• Pdf URL: https://arxiv.org/pdf/2404.15673
• Copy Paste: [[2404.15673]] Augmented CARDS: A machine learning approach to identifying triggers of climate change misinformation on Twitter(https://arxiv.org/abs/2404.15673)
• Keywords: attack
• Abstract: Misinformation about climate change poses a significant threat to societal well-being, prompting the urgent need for effective mitigation strategies. However, the rapid proliferation of online misinformation on social media platforms outpaces the ability of fact-checkers to debunk false claims. Automated detection of climate change misinformation offers a promising solution. In this study, we address this gap by developing a two-step hierarchical model, the Augmented CARDS model, specifically designed for detecting contrarian climate claims on Twitter. Furthermore, we apply the Augmented CARDS model to five million climate-themed tweets over a six-month period in 2022. We find that over half of contrarian climate claims on Twitter involve attacks on climate actors or conspiracy theories. Spikes in climate contrarianism coincide with one of four stimuli: political events, natural events, contrarian influencers, or convinced influencers. Implications for automated responses to climate misinformation are discussed.

Title: Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs

• Authors: Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, Shuai Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15676
• Pdf URL: https://arxiv.org/pdf/2404.15676
• Copy Paste: [[2404.15676]] Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs(https://arxiv.org/abs/2404.15676)
• Keywords: large language model
• Abstract: Chain-of-Thought (CoT) has been a widely adopted prompting method, eliciting impressive reasoning abilities of Large Language Models (LLMs). Inspired by the sequential thought structure of CoT, a number of Chain-of-X (CoX) methods have been developed to address various challenges across diverse domains and tasks involving LLMs. In this paper, we provide a comprehensive survey of Chain-of-X methods for LLMs in different contexts. Specifically, we categorize them by taxonomies of nodes, i.e., the X in CoX, and application tasks. We also discuss the findings and implications of existing CoX methods, as well as potential future directions. Our survey aims to serve as a detailed and up-to-date resource for researchers seeking to apply the idea of CoT to broader scenarios.

Title: CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models

• Authors: Qinghe Wang, Baolu Li, Xiaomin Li, Bing Cao, Liqian Ma, Huchuan Lu, Xu Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15677
• Pdf URL: https://arxiv.org/pdf/2404.15677
• Copy Paste: [[2404.15677]] CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models(https://arxiv.org/abs/2404.15677)
• Keywords: diffusion
• Abstract: Recent advances in text-to-image models have opened new frontiers in human-centric generation. However, these models cannot be directly employed to generate images with consistent newly coined identities. In this work, we propose CharacterFactory, a framework that allows sampling new characters with consistent identities in the latent space of GANs for diffusion models. More specifically, we consider the word embeddings of celeb names as ground truths for the identity-consistent generation task and train a GAN model to learn the mapping from a latent space to the celeb embedding space. In addition, we design a context-consistent loss to ensure that the generated identity embeddings can produce identity-consistent images in various contexts. Remarkably, the whole model only takes 10 minutes for training, and can sample infinite characters end-to-end during inference. Extensive experiments demonstrate excellent performance of the proposed CharacterFactory on character creation in terms of identity consistency and editability. Furthermore, the generated characters can be seamlessly combined with the off-the-shelf image/video/3D diffusion models. We believe that the proposed CharacterFactory is an important step for identity-consistent character generation. Project page is available at: https://qinghew.github.io/CharacterFactory/.

Title: Automated Creation of Source Code Variants of a Cryptographic Hash Function Implementation Using Generative Pre-Trained Transformer Models

• Authors: Elijah Pelofske, Vincent Urias, Lorie M. Liebrock
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15681
• Pdf URL: https://arxiv.org/pdf/2404.15681
• Copy Paste: [[2404.15681]] Automated Creation of Source Code Variants of a Cryptographic Hash Function Implementation Using Generative Pre-Trained Transformer Models(https://arxiv.org/abs/2404.15681)
• Keywords: secure, security, transformer, generative
• Abstract: Generative pre-trained transformers (GPT's) are a type of large language machine learning model that are unusually adept at producing novel, and coherent, natural language. In this study the ability of GPT models to generate novel and correct versions, and notably very insecure versions, of implementations of the cryptographic hash function SHA-1 is examined. The GPT models Llama-2-70b-chat-h, Mistral-7B-Instruct-v0.1, and zephyr-7b-alpha are used. The GPT models are prompted to re-write each function using a modified version of the localGPT framework and langchain to provide word embedding context of the full source code and header files to the model, resulting in over 130,000 function re-write GPT output text blocks, approximately 40,000 of which were able to be parsed as C code and subsequently compiled. The generated code is analyzed for being compilable, correctness of the algorithm, memory leaks, compiler optimization stability, and character distance to the reference implementation. Remarkably, several generated function variants have a high implementation security risk of being correct for some test vectors, but incorrect for other test vectors. Additionally, many function implementations were not correct to the reference algorithm of SHA-1, but produced hashes that have some of the basic characteristics of hash functions. Many of the function re-writes contained serious flaws such as memory leaks, integer overflows, out of bounds accesses, use of uninitialised values, and compiler optimization instability. Compiler optimization settings and SHA-256 hash checksums of the compiled binaries are used to cluster implementations that are equivalent but may not have identical syntax - using this clustering over 100,000 novel and correct versions of the SHA-1 codebase were generated where each component C function of the reference implementation is different from the original code.

Title: AnoFPDM: Anomaly Segmentation with Forward Process of Diffusion Models for Brain MRI

• Authors: Yiming Che, Fazle Rafsani, Jay Shah, Md Mahfuzur Rahman Siddiquee, Teresa Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15683
• Pdf URL: https://arxiv.org/pdf/2404.15683
• Copy Paste: [[2404.15683]] AnoFPDM: Anomaly Segmentation with Forward Process of Diffusion Models for Brain MRI(https://arxiv.org/abs/2404.15683)
• Keywords: diffusion, segmentation
• Abstract: Weakly-supervised diffusion models (DM) in anomaly segmentation, leveraging image-level labels, have attracted significant attention for their superior performance compared to unsupervised methods. It eliminates the need for pixel-level labels in training, offering a more cost-effective alternative to supervised methods. However, existing methods are not fully weakly-supervised because they heavily rely on costly pixel-level labels for hyperparameter tuning in inference. To tackle this challenge, we introduce Anomaly Segmentation with Forward Process of Diffusion Models (AnoFPDM), a fully weakly-supervised framework that operates without the need for pixel-level labels. Leveraging the unguided forward process as a reference, we identify suitable hyperparameters, i.e., noise scale and threshold, for each input image. We aggregate anomaly maps from each step in the forward process, enhancing the signal strength of anomalous regions. Remarkably, our proposed method outperforms recent state-of-the-art weakly-supervised approaches, even without utilizing pixel-level labels.

Title: Noise Variance Optimization in Differential Privacy: A Game-Theoretic Approach Through Per-Instance Differential Privacy

• Authors: Sehyun Ryu, Jonggyu Jang, Hyun Jong Yang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15686
• Pdf URL: https://arxiv.org/pdf/2404.15686
• Copy Paste: [[2404.15686]] Noise Variance Optimization in Differential Privacy: A Game-Theoretic Approach Through Per-Instance Differential Privacy(https://arxiv.org/abs/2404.15686)
• Keywords: privacy, defense, attack, membership infer
• Abstract: The concept of differential privacy (DP) can quantitatively measure privacy loss by observing the changes in the distribution caused by the inclusion of individuals in the target dataset. The DP, which is generally used as a constraint, has been prominent in safeguarding datasets in machine learning in industry giants like Apple and Google. A common methodology for guaranteeing DP is incorporating appropriate noise into query outputs, thereby establishing statistical defense systems against privacy attacks such as membership inference and linkage attacks. However, especially for small datasets, existing DP mechanisms occasionally add excessive amount of noise to query output, thereby discarding data utility. This is because the traditional DP computes privacy loss based on the worst-case scenario, i.e., statistical outliers. In this work, to tackle this challenge, we utilize per-instance DP (pDP) as a constraint, measuring privacy loss for each data instance and optimizing noise tailored to individual instances. In a nutshell, we propose a per-instance noise variance optimization (NVO) game, framed as a common interest sequential game, and show that the Nash equilibrium (NE) points of it inherently guarantee pDP for all data instances. Through extensive experiments, our proposed pDP algorithm demonstrated an average performance improvement of up to 99.53% compared to the conventional DP algorithm in terms of KL divergence.

Title: Neural Proto-Language Reconstruction

• Authors: Chenxuan Cui, Ying Chen, Qinxin Wang, David R. Mortensen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15690
• Pdf URL: https://arxiv.org/pdf/2404.15690
• Copy Paste: [[2404.15690]] Neural Proto-Language Reconstruction(https://arxiv.org/abs/2404.15690)
• Keywords: transformer
• Abstract: Proto-form reconstruction has been a painstaking process for linguists. Recently, computational models such as RNN and Transformers have been proposed to automate this process. We take three different approaches to improve upon previous methods, including data augmentation to recover missing reflexes, adding a VAE structure to the Transformer model for proto-to-language prediction, and using a neural machine translation model for the reconstruction task. We find that with the additional VAE structure, the Transformer model has a better performance on the WikiHan dataset, and the data augmentation step stabilizes the training.

Title: Deep Learning for Accelerated and Robust MRI Reconstruction: a Review

• Authors: Reinhard Heckel, Mathews Jacob, Akshay Chaudhari, Or Perlman, Efrat Shimron
• Subjects: cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.15692
• Pdf URL: https://arxiv.org/pdf/2404.15692
• Copy Paste: [[2404.15692]] Deep Learning for Accelerated and Robust MRI Reconstruction: a Review(https://arxiv.org/abs/2404.15692)
• Keywords: robust, generative
• Abstract: Deep learning (DL) has recently emerged as a pivotal technology for enhancing magnetic resonance imaging (MRI), a critical tool in diagnostic radiology. This review paper provides a comprehensive overview of recent advances in DL for MRI reconstruction. It focuses on DL approaches and architectures designed to improve image quality, accelerate scans, and address data-related challenges. These include end-to-end neural networks, pre-trained networks, generative models, and self-supervised methods. The paper also discusses the role of DL in optimizing acquisition protocols, enhancing robustness against distribution shifts, and tackling subtle bias. Drawing on the extensive literature and practical insights, it outlines current successes, limitations, and future directions for leveraging DL in MRI reconstruction, while emphasizing the potential of DL to significantly impact clinical imaging practices.

Title: DeepFeatureX Net: Deep Features eXtractors based Network for discriminating synthetic from real images

• Authors: Orazio Pontorno (1), Luca Guarnera (1), Sebastiano Battiato (1) ((1) University of Catania)
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15697
• Pdf URL: https://arxiv.org/pdf/2404.15697
• Copy Paste: [[2404.15697]] DeepFeatureX Net: Deep Features eXtractors based Network for discriminating synthetic from real images(https://arxiv.org/abs/2404.15697)
• Keywords: robust, diffusion
• Abstract: Deepfakes, synthetic images generated by deep learning algorithms, represent one of the biggest challenges in the field of Digital Forensics. The scientific community is working to develop approaches that can discriminate the origin of digital images (real or AI-generated). However, these methodologies face the challenge of generalization, that is, the ability to discern the nature of an image even if it is generated by an architecture not seen during training. This usually leads to a drop in performance. In this context, we propose a novel approach based on three blocks called Base Models, each of which is responsible for extracting the discriminative features of a specific image class (Diffusion Model-generated, GAN-generated, or real) as it is trained by exploiting deliberately unbalanced datasets. The features extracted from each block are then concatenated and processed to discriminate the origin of the input image. Experimental results showed that this approach not only demonstrates good robust capabilities to JPEG compression but also outperforms state-of-the-art methods in several generalization tests. Code, models and dataset are available at https://github.com/opontorno/block-based_deepfake-detection.

Title: MAS-SAM: Segment Any Marine Animal with Aggregated Features

• Authors: Tianyu Yan, Zifu Wan, Xinhao Deng, Pingping Zhang, Yang Liu, Huchuan Lu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.15700
• Pdf URL: https://arxiv.org/pdf/2404.15700
• Copy Paste: [[2404.15700]] MAS-SAM: Segment Any Marine Animal with Aggregated Features(https://arxiv.org/abs/2404.15700)
• Keywords: extraction, segmentation
• Abstract: Recently, Segment Anything Model (SAM) shows exceptional performance in generating high-quality object masks and achieving zero-shot image segmentation. However, as a versatile vision model, SAM is primarily trained with large-scale natural light images. In underwater scenes, it exhibits substantial performance degradation due to the light scattering and absorption. Meanwhile, the simplicity of the SAM's decoder might lead to the loss of fine-grained object details. To address the above issues, we propose a novel feature learning framework named MAS-SAM for marine animal segmentation, which involves integrating effective adapters into the SAM's encoder and constructing a pyramidal decoder. More specifically, we first build a new SAM's encoder with effective adapters for underwater scenes. Then, we introduce a Hypermap Extraction Module (HEM) to generate multi-scale features for a comprehensive guidance. Finally, we propose a Progressive Prediction Decoder (PPD) to aggregate the multi-scale features and predict the final segmentation results. When grafting with the Fusion Attention Module (FAM), our method enables to extract richer marine information from global contextual cues to fine-grained local details. Extensive experiments on four public MAS datasets demonstrate that our MAS-SAM can obtain better results than other typical segmentation methods. The source code is available at https://github.com/Drchip61/MAS-SAM.

Title: Nyonic Technical Report

• Authors: Junfeng Tian, Rui Wang, Cong Li, Yudong Zhou, Jun Liu, Jun Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15702
• Pdf URL: https://arxiv.org/pdf/2404.15702
• Copy Paste: [[2404.15702]] Nyonic Technical Report(https://arxiv.org/abs/2404.15702)
• Keywords: robust, large language model
• Abstract: This report details the development and key achievements of our latest language model designed for custom large language models. The advancements introduced include a novel Online Data Scheduler that supports flexible training data adjustments and curriculum learning. The model's architecture is fortified with state-of-the-art techniques such as Rotary Positional Embeddings, QK-LayerNorm, and a specially crafted multilingual tokenizer to enhance stability and performance. Moreover, our robust training framework incorporates advanced monitoring and rapid recovery features to ensure optimal efficiency. Our Wonton 7B model has demonstrated competitive performance on a range of multilingual and English benchmarks. Future developments will prioritize narrowing the performance gap with more extensively trained models, thereby enhancing the model's real-world efficacy and adaptability.GitHub: \url{https://github.com/nyonicai/nyonic-public}

Title: Efficient Multi-Model Fusion with Adversarial Complementary Representation Learning

• Authors: Zuheng Kang, Yayun He, Jianzong Wang, Junqing Peng, Jing Xiao
• Subjects: cs.LG, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.15704
• Pdf URL: https://arxiv.org/pdf/2404.15704
• Copy Paste: [[2404.15704]] Efficient Multi-Model Fusion with Adversarial Complementary Representation Learning(https://arxiv.org/abs/2404.15704)
• Keywords: robust
• Abstract: Single-model systems often suffer from deficiencies in tasks such as speaker verification (SV) and image classification, relying heavily on partial prior knowledge during decision-making, resulting in suboptimal performance. Although multi-model fusion (MMF) can mitigate some of these issues, redundancy in learned representations may limits improvements. To this end, we propose an adversarial complementary representation learning (ACoRL) framework that enables newly trained models to avoid previously acquired knowledge, allowing each individual component model to learn maximally distinct, complementary representations. We make three detailed explanations of why this works and experimental results demonstrate that our method more efficiently improves performance compared to traditional MMF. Furthermore, attribution analysis validates the model trained under ACoRL acquires more complementary knowledge, highlighting the efficacy of our approach in enhancing efficiency and robustness across tasks.

Title: HDBN: A Novel Hybrid Dual-branch Network for Robust Skeleton-based Action Recognition

• Authors: Jinfu Liu, Baiqiao Yin, Jiaying Lin, Jiajun Wen, Yue Li, Mengyuan Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15719
• Pdf URL: https://arxiv.org/pdf/2404.15719
• Copy Paste: [[2404.15719]] HDBN: A Novel Hybrid Dual-branch Network for Robust Skeleton-based Action Recognition(https://arxiv.org/abs/2404.15719)
• Keywords: robust, transformer
• Abstract: Skeleton-based action recognition has gained considerable traction thanks to its utilization of succinct and robust skeletal representations. Nonetheless, current methodologies often lean towards utilizing a solitary backbone to model skeleton modality, which can be limited by inherent flaws in the network backbone. To address this and fully leverage the complementary characteristics of various network architectures, we propose a novel Hybrid Dual-Branch Network (HDBN) for robust skeleton-based action recognition, which benefits from the graph convolutional network's proficiency in handling graph-structured data and the powerful modeling capabilities of Transformers for global information. In detail, our proposed HDBN is divided into two trunk branches: MixGCN and MixFormer. The two branches utilize GCNs and Transformers to model both 2D and 3D skeletal modalities respectively. Our proposed HDBN emerged as one of the top solutions in the Multi-Modal Video Reasoning and Analyzing Competition (MMVRAC) of 2024 ICME Grand Challenge, achieving accuracies of 47.95% and 75.36% on two benchmarks of the UAV-Human dataset by outperforming most existing methods. Our code will be publicly available at: https://github.com/liujf69/ICMEW2024-Track10.

Title: SPARO: Selective Attention for Robust and Compositional Transformer Encodings for Vision

• Authors: Ankit Vani, Bac Nguyen, Samuel Lavoie, Ranjay Krishna, Aaron Courville
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15721
• Pdf URL: https://arxiv.org/pdf/2404.15721
• Copy Paste: [[2404.15721]] SPARO: Selective Attention for Robust and Compositional Transformer Encodings for Vision(https://arxiv.org/abs/2404.15721)
• Keywords: robust, transformer
• Abstract: Selective attention helps us focus on task-relevant aspects in the constant flood of our sensory input. This constraint in our perception allows us to robustly generalize under distractions and to new compositions of perceivable concepts. Transformers employ a similar notion of attention in their architecture, but representation learning models with transformer backbones like CLIP and DINO often fail to demonstrate robustness and compositionality. We highlight a missing architectural prior: unlike human perception, transformer encodings do not separately attend over individual concepts. In response, we propose SPARO, a read-out mechanism that partitions encodings into separately-attended slots, each produced by a single attention head. Using SPARO with CLIP imparts an inductive bias that the vision and text modalities are different views of a shared compositional world with the same corresponding concepts. Using SPARO, we demonstrate improvements on downstream recognition, robustness, retrieval, and compositionality benchmarks with CLIP (up to +14% for ImageNet, +4% for SugarCrepe), and on nearest neighbors and linear probe for ImageNet with DINO (+3% each). We also showcase a powerful ability to intervene and select individual SPARO concepts to further improve downstream task performance (up from +4% to +9% for SugarCrepe) and use this ability to study the robustness of SPARO's representation structure. Finally, we provide insights through ablation experiments and visualization of learned concepts.

Title: Gradformer: Graph Transformer with Exponential Decay

• Authors: Chuang Liu, Zelin Yao, Yibing Zhan, Xueqi Ma, Shirui Pan, Wenbin Hu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15729
• Pdf URL: https://arxiv.org/pdf/2404.15729
• Copy Paste: [[2404.15729]] Gradformer: Graph Transformer with Exponential Decay(https://arxiv.org/abs/2404.15729)
• Keywords: transformer
• Abstract: Graph Transformers (GTs) have demonstrated their advantages across a wide range of tasks. However, the self-attention mechanism in GTs overlooks the graph's inductive biases, particularly biases related to structure, which are crucial for the graph tasks. Although some methods utilize positional encoding and attention bias to model inductive biases, their effectiveness is still suboptimal analytically. Therefore, this paper presents Gradformer, a method innovatively integrating GT with the intrinsic inductive bias by applying an exponential decay mask to the attention matrix. Specifically, the values in the decay mask matrix diminish exponentially, correlating with the decreasing node proximities within the graph structure. This design enables Gradformer to retain its ability to capture information from distant nodes while focusing on the graph's local details. Furthermore, Gradformer introduces a learnable constraint into the decay mask, allowing different attention heads to learn distinct decay masks. Such an design diversifies the attention heads, enabling a more effective assimilation of diverse structural information within the graph. Extensive experiments on various benchmarks demonstrate that Gradformer consistently outperforms the Graph Neural Network and GT baseline models in various graph classification and regression tasks. Additionally, Gradformer has proven to be an effective method for training deep GT models, maintaining or even enhancing accuracy compared to shallow models as the network deepens, in contrast to the significant accuracy drop observed in other GT models.Codes are available at \url{https://github.com/LiuChuang0059/Gradformer}.

Title: Replacing Cryptopuzzles with Useful Computation in Blockchain Proof-of-Work Protocols

• Authors: Andrea Merlina, Thiago Garrett, Roman Vitenberg
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.15735
• Pdf URL: https://arxiv.org/pdf/2404.15735
• Copy Paste: [[2404.15735]] Replacing Cryptopuzzles with Useful Computation in Blockchain Proof-of-Work Protocols(https://arxiv.org/abs/2404.15735)
• Keywords: robust
• Abstract: Proof-of-Work (PoW) blockchains have emerged as a robust and effective consensus mechanism in open environments like the Internet, leading to widespread deployment with numerous cryptocurrency platforms and substantial investments. However, the current PoW implementation primarily focuses on validating the discovery of a winning nonce. Exploring the notion of replacing cryptographic puzzles with useful computing tasks becomes compelling, given the substantial computational capacity of blockchain networks and the global pursuit of a more sustainable IT infrastructure. In this study, we conduct a comprehensive analysis of the prerequisites for alternative classes of tasks, examining proposed designs from existing literature in light of these requirements. We distill pertinent techniques and address gaps in the current state-of-the-art, providing valuable insights into the evolution of consensus mechanisms beyond traditional PoW.

Title: What Makes Multimodal In-Context Learning Work?

• Authors: Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, Benjamin Piwowarski
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15736
• Pdf URL: https://arxiv.org/pdf/2404.15736
• Copy Paste: [[2404.15736]] What Makes Multimodal In-Context Learning Work?(https://arxiv.org/abs/2404.15736)
• Keywords: large language model
• Abstract: Large Language Models have demonstrated remarkable performance across various tasks, exhibiting the capacity to swiftly acquire new skills, such as through In-Context Learning (ICL) with minimal demonstration examples. In this work, we present a comprehensive framework for investigating Multimodal ICL (M-ICL) in the context of Large Multimodal Models. We consider the best open-source multimodal models (e.g., IDEFICS, OpenFlamingo) and a wide range of multimodal tasks. Our study unveils several noteworthy findings: (1) M-ICL primarily relies on text-driven mechanisms, showing little to no influence from the image modality. (2) When used with advanced-ICL strategy (like RICES), M-ICL is not better than a simple strategy based on majority voting over context examples. Moreover, we identify several biases and limitations of M-ICL that warrant consideration prior to deployment. Code available at https://gitlab.com/folbaeni/multimodal-icl}{gitlab.com/folbaeni/multimodal-icl

Title: SRAGAN: Saliency Regularized and Attended Generative Adversarial Network for Chinese Ink-wash Painting Generation

• Authors: Xiang Gao, Yuqi Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15743
• Pdf URL: https://arxiv.org/pdf/2404.15743
• Copy Paste: [[2404.15743]] SRAGAN: Saliency Regularized and Attended Generative Adversarial Network for Chinese Ink-wash Painting Generation(https://arxiv.org/abs/2404.15743)
• Keywords: generative
• Abstract: This paper handles the problem of converting real pictures into traditional Chinese ink-wash paintings, i.e., Chinese ink-wash painting style transfer. Though this problem could be realized by a wide range of image-to-image translation models, a notable issue with all these methods is that the original image content details could be easily erased or corrupted due to transfer of ink-wash style elements. To solve or ameliorate this issue, we propose to incorporate saliency detection into the unpaired image-to-image translation framework to regularize content information of the generated paintings. The saliency map is utilized for content regularization from two aspects, both explicitly and implicitly: (\romannumeral1) we propose saliency IOU (SIOU) loss to explicitly regularize saliency consistency before and after stylization; (\romannumeral2) we propose saliency adaptive normalization (SANorm) which implicitly enhances content integrity of the generated paintings by injecting saliency information to the generator network to guide painting generation. Besides, we also propose saliency attended discriminator network which harnesses saliency mask to focus generative adversarial attention onto salient image regions, it contributes to producing finer ink-wash stylization effect for salient objects of images. Qualitative and quantitative experiments consistently demonstrate superiority of our model over related advanced methods for Chinese ink-wash painting style transfer.

Title: A General Black-box Adversarial Attack on Graph-based Fake News Detectors

• Authors: Peican Zhu, Zechen Pan, Yang Liu, Jiwei Tian, Keke Tang, Zhen Wang
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15744
• Pdf URL: https://arxiv.org/pdf/2404.15744
• Copy Paste: [[2404.15744]] A General Black-box Adversarial Attack on Graph-based Fake News Detectors(https://arxiv.org/abs/2404.15744)
• Keywords: attack
• Abstract: Graph Neural Network (GNN)-based fake news detectors apply various methods to construct graphs, aiming to learn distinctive news embeddings for classification. Since the construction details are unknown for attackers in a black-box scenario, it is unrealistic to conduct the classical adversarial attacks that require a specific adjacency matrix. In this paper, we propose the first general black-box adversarial attack framework, i.e., General Attack via Fake Social Interaction (GAFSI), against detectors based on different graph structures. Specifically, as sharing is an important social interaction for GNN-based fake news detectors to construct the graph, we simulate sharing behaviors to fool the detectors. Firstly, we propose a fraudster selection module to select engaged users leveraging local and global information. In addition, a post injection module guides the selected users to create shared relations by sending posts. The sharing records will be added to the social context, leading to a general attack against different detectors. Experimental results on empirical datasets demonstrate the effectiveness of GAFSI.

Title: Let's Think Dot by Dot: Hidden Computation in Transformer Language Models

• Authors: Jacob Pfau, William Merrill, Samuel R. Bowman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15758
• Pdf URL: https://arxiv.org/pdf/2404.15758
• Copy Paste: [[2404.15758]] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models(https://arxiv.org/abs/2404.15758)
• Keywords: transformer, large language model
• Abstract: Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens. However, we find empirically that learning to use filler tokens is difficult and requires specific, dense supervision to converge. We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula. For problems satisfying this characterization, chain-of-thought tokens need not provide information about the intermediate computational steps involved in multi-token computations. In summary, our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.

Title: 3D Face Morphing Attack Generation using Non-Rigid Registration

• Authors: Jag Mohan Singh, Raghavendra Ramachandra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15765
• Pdf URL: https://arxiv.org/pdf/2404.15765
• Copy Paste: [[2404.15765]] 3D Face Morphing Attack Generation using Non-Rigid Registration(https://arxiv.org/abs/2404.15765)
• Keywords: attack
• Abstract: Face Recognition Systems (FRS) are widely used in commercial environments, such as e-commerce and e-banking, owing to their high accuracy in real-world conditions. However, these systems are vulnerable to facial morphing attacks, which are generated by blending face color images of different subjects. This paper presents a new method for generating 3D face morphs from two bona fide point clouds. The proposed method first selects bona fide point clouds with neutral expressions. The two input point clouds were then registered using a Bayesian Coherent Point Drift (BCPD) without optimization, and the geometry and color of the registered point clouds were averaged to generate a face morphing point cloud. The proposed method generates 388 face-morphing point clouds from 200 bona fide subjects. The effectiveness of the method was demonstrated through extensive vulnerability experiments, achieving a Generalized Morphing Attack Potential (G-MAP) of 97.93%, which is superior to the existing state-of-the-art (SOTA) with a G-MAP of 81.61%.

Title: Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equations

• Authors: Kaiwen Xue, Yuhao Zhou, Shen Nie, Xu Min, Xiaolu Zhang, Jun Zhou, Chongxuan Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15766
• Pdf URL: https://arxiv.org/pdf/2404.15766
• Copy Paste: [[2404.15766]] Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equations(https://arxiv.org/abs/2404.15766)
• Keywords: diffusion
• Abstract: Bayesian flow networks (BFNs) iteratively refine the parameters, instead of the samples in diffusion models (DMs), of distributions at various noise levels through Bayesian inference. Owing to its differentiable nature, BFNs are promising in modeling both continuous and discrete data, while simultaneously maintaining fast sampling capabilities. This paper aims to understand and enhance BFNs by connecting them with DMs through stochastic differential equations (SDEs). We identify the linear SDEs corresponding to the noise-addition processes in BFNs, demonstrate that BFN's regression losses are aligned with denoise score matching, and validate the sampler in BFN as a first-order solver for the respective reverse-time SDE. Based on these findings and existing recipes of fast sampling in DMs, we propose specialized solvers for BFNs that markedly surpass the original BFN sampler in terms of sample quality with a limited number of function evaluations (e.g., 10) on both image and text datasets. Notably, our best sampler achieves an increase in speed of 5~20 times for free. Our code is available at https://github.com/ML-GSAI/BFN-Solver.

Title: ChEX: Interactive Localization and Region Description in Chest X-rays

• Authors: Philip Müller, Georgios Kaissis, Daniel Rueckert
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15770
• Pdf URL: https://arxiv.org/pdf/2404.15770
• Copy Paste: [[2404.15770]] ChEX: Interactive Localization and Region Description in Chest X-rays(https://arxiv.org/abs/2404.15770)
• Keywords: interpretability
• Abstract: Report generation models offer fine-grained textual interpretations of medical images like chest X-rays, yet they often lack interactivity (i.e. the ability to steer the generation process through user queries) and localized interpretability (i.e. visually grounding their predictions), which we deem essential for future adoption in clinical practice. While there have been efforts to tackle these issues, they are either limited in their interactivity by not supporting textual queries or fail to also offer localized interpretability. Therefore, we propose a novel multitask architecture and training paradigm integrating textual prompts and bounding boxes for diverse aspects like anatomical regions and pathologies. We call this approach the Chest X-Ray Explainer (ChEX). Evaluations across a heterogeneous set of 9 chest X-ray tasks, including localized image interpretation and report generation, showcase its competitiveness with SOTA models while additional analysis demonstrates ChEX's interactive capabilities.

Title: DVF: Advancing Robust and Accurate Fine-Grained Image Retrieval with Retrieval Guidelines

• Authors: Xin Jiang, Hao Tang, Rui Yan, Jinhui Tang, Zechao Li
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.15771
• Pdf URL: https://arxiv.org/pdf/2404.15771
• Copy Paste: [[2404.15771]] DVF: Advancing Robust and Accurate Fine-Grained Image Retrieval with Retrieval Guidelines(https://arxiv.org/abs/2404.15771)
• Keywords: robust, transformer
• Abstract: Fine-grained image retrieval (FGIR) is to learn visual representations that distinguish visually similar objects while maintaining generalization. Existing methods propose to generate discriminative features, but rarely consider the particularity of the FGIR task itself. This paper presents a meticulous analysis leading to the proposal of practical guidelines to identify subcategory-specific discrepancies and generate discriminative features to design effective FGIR models. These guidelines include emphasizing the object (G1), highlighting subcategory-specific discrepancies (G2), and employing effective training strategy (G3). Following G1 and G2, we design a novel Dual Visual Filtering mechanism for the plain visual transformer, denoted as DVF, to capture subcategory-specific discrepancies. Specifically, the dual visual filtering mechanism comprises an object-oriented module and a semantic-oriented module. These components serve to magnify objects and identify discriminative regions, respectively. Following G3, we implement a discriminative model training strategy to improve the discriminability and generalization ability of DVF. Extensive analysis and ablation studies confirm the efficacy of our proposed guidelines. Without bells and whistles, the proposed DVF achieves state-of-the-art performance on three widely-used fine-grained datasets in closed-set and open-set settings.

Title: Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting

• Authors: Aobo Liang, Xingguo Jiang, Yan Sun, Chang Lu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15772
• Pdf URL: https://arxiv.org/pdf/2404.15772
• Copy Paste: [[2404.15772]] Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting(https://arxiv.org/abs/2404.15772)
• Keywords: transformer
• Abstract: Long-term time series forecasting (LTSF) provides longer insights into future trends and patterns. In recent years, deep learning models especially Transformers have achieved advanced performance in LTSF tasks. However, the quadratic complexity of Transformers rises the challenge of balancing computaional efficiency and predicting performance. Recently, a new state space model (SSM) named Mamba is proposed. With the selective capability on input data and the hardware-aware parallel computing algorithm, Mamba can well capture long-term dependencies while maintaining linear computational complexity. Mamba has shown great ability for long sequence modeling and is a potential competitor to Transformer-based models in LTSF. In this paper, we propose Bi-Mamba4TS, a bidirectional Mamba for time series forecasting. To address the sparsity of time series semantics, we adopt the patching technique to enrich the local information while capturing the evolutionary patterns of time series in a finer granularity. To select more appropriate modeling method based on the characteristics of the dataset, our model unifies the channel-independent and channel-mixing tokenization strategies and uses a series-relation-aware decider to control the strategy choosing process. Extensive experiments on seven real-world datasets show that our model achieves more accurate predictions compared with state-of-the-art methods.

Title: Toward Physics-Aware Deep Learning Architectures for LiDAR Intensity Simulation

• Authors: Vivek Anand, Bharat Lohani, Gaurav Pandey, Rakesh Mishra
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.15774
• Pdf URL: https://arxiv.org/pdf/2404.15774
• Copy Paste: [[2404.15774]] Toward Physics-Aware Deep Learning Architectures for LiDAR Intensity Simulation(https://arxiv.org/abs/2404.15774)
• Keywords: generative
• Abstract: Autonomous vehicles (AVs) heavily rely on LiDAR perception for environment understanding and navigation. LiDAR intensity provides valuable information about the reflected laser signals and plays a crucial role in enhancing the perception capabilities of AVs. However, accurately simulating LiDAR intensity remains a challenge due to the unavailability of material properties of the objects in the environment, and complex interactions between the laser beam and the environment. The proposed method aims to improve the accuracy of intensity simulation by incorporating physics-based modalities within the deep learning framework. One of the key entities that captures the interaction between the laser beam and the objects is the angle of incidence. In this work we demonstrate that the addition of the LiDAR incidence angle as a separate input to the deep neural networks significantly enhances the results. We present a comparative study between two prominent deep learning architectures: U-NET a Convolutional Neural Network (CNN), and Pix2Pix a Generative Adversarial Network (GAN). We implemented these two architectures for the intensity prediction task and used SemanticKITTI and VoxelScape datasets for experiments. The comparative analysis reveals that both architectures benefit from the incidence angle as an additional input. Moreover, the Pix2Pix architecture outperforms U-NET, especially when the incidence angle is incorporated.

Title: A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry

• Authors: Yining Huang, Keke Tang, Meilian Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15777
• Pdf URL: https://arxiv.org/pdf/2404.15777
• Copy Paste: [[2404.15777]] A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry(https://arxiv.org/abs/2404.15777)
• Keywords: transformer, large language model
• Abstract: Since the inception of the Transformer architecture in 2017, Large Language Models (LLMs) such as GPT and BERT have evolved significantly, impacting various industries with their advanced capabilities in language understanding and generation. These models have shown potential to transform the medical field, highlighting the necessity for specialized evaluation frameworks to ensure their effective and ethical deployment. This comprehensive survey delineates the extensive application and requisite evaluation of LLMs within healthcare, emphasizing the critical need for empirical validation to fully exploit their capabilities in enhancing healthcare outcomes. Our survey is structured to provide an in-depth analysis of LLM applications across clinical settings, medical text data processing, research, education, and public health awareness. We begin by exploring the roles of LLMs in different medical applications, detailing how they are evaluated based on their performance in tasks such as clinical application, medical text data processing, information retrieval, data analysis, medical scientific writing, educational content generation etc. The subsequent sections delve into the methodologies employed in these evaluations, discussing the benchmarks and metrics used to assess the models' effectiveness, accuracy, and ethical alignment. Through this survey, we aim to equip healthcare professionals, researchers, and policymakers with a comprehensive understanding of the potential strengths and limitations of LLMs in medical applications. By providing detailed insights into the evaluation processes and the challenges faced in integrating LLMs into healthcare, this survey seeks to guide the responsible development and deployment of these powerful models, ensuring they are harnessed to their full potential while maintaining stringent ethical standards.

Title: BASS: Batched Attention-optimized Speculative Sampling

• Authors: Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Xiaofei Ma, Anoop Deoras
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15778
• Pdf URL: https://arxiv.org/pdf/2404.15778
• Copy Paste: [[2404.15778]] BASS: Batched Attention-optimized Speculative Sampling(https://arxiv.org/abs/2404.15778)
• Keywords: generative, large language model
• Abstract: Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models. However, most existing implementations focus on generating a single sequence. Real-world generative AI applications often require multiple responses and how to perform speculative decoding in a batched setting while preserving its latency benefits poses non-trivial challenges. This paper describes a system of batched speculative decoding that sets a new state of the art in multi-sequence generation latency and that demonstrates superior GPU utilization as well as quality of generations within a time budget. For example, for a 7.8B-size model on a single A100 GPU and with a batch size of 8, each sequence is generated at an average speed of 5.8ms per token, the overall throughput being 1.1K tokens per second. These results represent state-of-the-art latency and a 2.15X speed-up over optimized regular decoding. Within a time budget that regular decoding does not finish, our system is able to generate sequences with HumanEval Pass@First of 43% and Pass@All of 61%, far exceeding what's feasible with single-sequence speculative decoding. Our peak GPU utilization during decoding reaches as high as 15.8%, more than 3X the highest of that of regular decoding and around 10X of single-sequence speculative decoding.

Title: Real-Time Compressed Sensing for Joint Hyperspectral Image Transmission and Restoration for CubeSat

• Authors: Chih-Chung Hsu, Chih-Yu Jian, Eng-Shen Tu, Chia-Ming Lee, Guan-Lin Chen
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.15781
• Pdf URL: https://arxiv.org/pdf/2404.15781
• Copy Paste: [[2404.15781]] Real-Time Compressed Sensing for Joint Hyperspectral Image Transmission and Restoration for CubeSat(https://arxiv.org/abs/2404.15781)
• Keywords: robust
• Abstract: This paper addresses the challenges associated with hyperspectral image (HSI) reconstruction from miniaturized satellites, which often suffer from stripe effects and are computationally resource-limited. We propose a Real-Time Compressed Sensing (RTCS) network designed to be lightweight and require only relatively few training samples for efficient and robust HSI reconstruction in the presence of the stripe effect and under noisy transmission conditions. The RTCS network features a simplified architecture that reduces the required training samples and allows for easy implementation on integer-8-based encoders, facilitating rapid compressed sensing for stripe-like HSI, which exactly matches the moderate design of miniaturized satellites on push broom scanning mechanism. This contrasts optimization-based models that demand high-precision floating-point operations, making them difficult to deploy on edge devices. Our encoder employs an integer-8-compatible linear projection for stripe-like HSI data transmission, ensuring real-time compressed sensing. Furthermore, based on the novel two-streamed architecture, an efficient HSI restoration decoder is proposed for the receiver side, allowing for edge-device reconstruction without needing a sophisticated central server. This is particularly crucial as an increasing number of miniaturized satellites necessitates significant computing resources on the ground station. Extensive experiments validate the superior performance of our approach, offering new and vital capabilities for existing miniaturized satellite systems.

Title: An Empirical Study of Aegis

• Authors: Daniel Saragih, Paridhi Goel, Tejas Balaji, Alyssa Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15784
• Pdf URL: https://arxiv.org/pdf/2404.15784
• Copy Paste: [[2404.15784]] An Empirical Study of Aegis(https://arxiv.org/abs/2404.15784)
• Keywords: defense, attack, robust
• Abstract: Bit flipping attacks are one class of attacks on neural networks with numerous defense mechanisms invented to mitigate its potency. Due to the importance of ensuring the robustness of these defense mechanisms, we perform an empirical study on the Aegis framework. We evaluate the baseline mechanisms of Aegis on low-entropy data (MNIST), and we evaluate a pre-trained model with the mechanisms fine-tuned on MNIST. We also compare the use of data augmentation to the robustness training of Aegis, and how Aegis performs under other adversarial attacks, such as the generation of adversarial examples. We find that both the dynamic-exit strategy and robustness training of Aegis has some drawbacks. In particular, we see drops in accuracy when testing on perturbed data, and on adversarial examples, as compared to baselines. Moreover, we found that the dynamic exit-strategy loses its uniformity when tested on simpler datasets. The code for this project is available on GitHub.

Title: MotionMaster: Training-free Camera Motion Transfer For Video Generation

• Authors: Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15789
• Pdf URL: https://arxiv.org/pdf/2404.15789
• Copy Paste: [[2404.15789]] MotionMaster: Training-free Camera Motion Transfer For Video Generation(https://arxiv.org/abs/2404.15789)
• Keywords: diffusion
• Abstract: The emergence of diffusion models has greatly propelled the progress in image and video generation. Recently, some efforts have been made in controllable video generation, including text-to-video generation and video motion control, among which camera motion control is an important topic. However, existing camera motion control methods rely on training a temporal camera module, and necessitate substantial computation resources due to the large amount of parameters in video generation models. Moreover, existing methods pre-define camera motion types during training, which limits their flexibility in camera control. Therefore, to reduce training costs and achieve flexible camera control, we propose COMD, a novel training-free video motion transfer model, which disentangles camera motions and object motions in source videos and transfers the extracted camera motions to new videos. We first propose a one-shot camera motion disentanglement method to extract camera motion from a single source video, which separates the moving objects from the background and estimates the camera motion in the moving objects region based on the motion in the background by solving a Poisson equation. Furthermore, we propose a few-shot camera motion disentanglement method to extract the common camera motion from multiple videos with similar camera motions, which employs a window-based clustering technique to extract the common features in temporal attention maps of multiple videos. Finally, we propose a motion combination method to combine different types of camera motions together, enabling our model a more controllable and flexible camera control. Extensive experiments demonstrate that our training-free approach can effectively decouple camera-object motion and apply the decoupled camera motion to a wide range of controllable video generation tasks, achieving flexible and diverse camera motion control.

Title: Leveraging Large Language Models for Multimodal Search

• Authors: Oriol Barbany, Michael Huang, Xinliang Zhu, Arnab Dhua
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15790
• Pdf URL: https://arxiv.org/pdf/2404.15790
• Copy Paste: [[2404.15790]] Leveraging Large Language Models for Multimodal Search(https://arxiv.org/abs/2404.15790)
• Keywords: large language model
• Abstract: Multimodal search has become increasingly important in providing users with a natural and effective way to ex-press their search intentions. Images offer fine-grained details of the desired products, while text allows for easily incorporating search modifications. However, some existing multimodal search systems are unreliable and fail to address simple queries. The problem becomes harder with the large variability of natural language text queries, which may contain ambiguous, implicit, and irrelevant in-formation. Addressing these issues may require systems with enhanced matching capabilities, reasoning abilities, and context-aware query parsing and rewriting. This paper introduces a novel multimodal search model that achieves a new performance milestone on the Fashion200K dataset. Additionally, we propose a novel search interface integrating Large Language Models (LLMs) to facilitate natural language interaction. This interface routes queries to search systems while conversationally engaging with users and considering previous searches. When coupled with our multimodal search model, it heralds a new era of shopping assistants capable of offering human-like interaction and enhancing the overall search experience.

Title: Raformer: Redundancy-Aware Transformer for Video Wire Inpainting

• Authors: Zhong Ji, Yimu Su, Yan Zhang, Jiacheng Hou, Yanwei Pang, Jungong Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15802
• Pdf URL: https://arxiv.org/pdf/2404.15802
• Copy Paste: [[2404.15802]] Raformer: Redundancy-Aware Transformer for Video Wire Inpainting(https://arxiv.org/abs/2404.15802)
• Keywords: transformer
• Abstract: Video Wire Inpainting (VWI) is a prominent application in video inpainting, aimed at flawlessly removing wires in films or TV series, offering significant time and labor savings compared to manual frame-by-frame removal. However, wire removal poses greater challenges due to the wires being longer and slimmer than objects typically targeted in general video inpainting tasks, and often intersecting with people and background objects irregularly, which adds complexity to the inpainting process. Recognizing the limitations posed by existing video wire datasets, which are characterized by their small size, poor quality, and limited variety of scenes, we introduce a new VWI dataset with a novel mask generation strategy, namely Wire Removal Video Dataset 2 (WRV2) and Pseudo Wire-Shaped (PWS) Masks. WRV2 dataset comprises over 4,000 videos with an average length of 80 frames, designed to facilitate the development and efficacy of inpainting models. Building upon this, our research proposes the Redundancy-Aware Transformer (Raformer) method that addresses the unique challenges of wire removal in video inpainting. Unlike conventional approaches that indiscriminately process all frame patches, Raformer employs a novel strategy to selectively bypass redundant parts, such as static background segments devoid of valuable information for inpainting. At the core of Raformer is the Redundancy-Aware Attention (RAA) module, which isolates and accentuates essential content through a coarse-grained, window-based attention mechanism. This is complemented by a Soft Feature Alignment (SFA) module, which refines these features and achieves end-to-end feature alignment. Extensive experiments on both the traditional video inpainting datasets and our proposed WRV2 dataset demonstrate that Raformer outperforms other state-of-the-art methods.

Title: GeckOpt: LLM System Efficiency via Intent-Based Tool Selection

• Authors: Michael Fore, Simranjit Singh, Dimitrios Stamoulis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15804
• Pdf URL: https://arxiv.org/pdf/2404.15804
• Copy Paste: [[2404.15804]] GeckOpt: LLM System Efficiency via Intent-Based Tool Selection(https://arxiv.org/abs/2404.15804)
• Keywords: large language model
• Abstract: In this preliminary study, we investigate a GPT-driven intent-based reasoning approach to streamline tool selection for large language models (LLMs) aimed at system efficiency. By identifying the intent behind user prompts at runtime, we narrow down the API toolset required for task execution, reducing token consumption by up to 24.6\%. Early results on a real-world, massively parallel Copilot platform with over 100 GPT-4-Turbo nodes show cost reductions and potential towards improving LLM-based system efficiency.

Title: One Subgraph for All: Efficient Reasoning on Opening Subgraphs for Inductive Knowledge Graph Completion

• Authors: Zhiwen Xie, Yi Zhang, Guangyou Zhou, Jin Liu, Xinhui Tu, Jimmy Xiangji Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15807
• Pdf URL: https://arxiv.org/pdf/2404.15807
• Copy Paste: [[2404.15807]] One Subgraph for All: Efficient Reasoning on Opening Subgraphs for Inductive Knowledge Graph Completion(https://arxiv.org/abs/2404.15807)
• Keywords: extraction
• Abstract: Knowledge Graph Completion (KGC) has garnered massive research interest recently, and most existing methods are designed following a transductive setting where all entities are observed during training. Despite the great progress on the transductive KGC, these methods struggle to conduct reasoning on emerging KGs involving unseen entities. Thus, inductive KGC, which aims to deduce missing links among unseen entities, has become a new trend. Many existing studies transform inductive KGC as a graph classification problem by extracting enclosing subgraphs surrounding each candidate triple. Unfortunately, they still face certain challenges, such as the expensive time consumption caused by the repeat extraction of enclosing subgraphs, and the deficiency of entity-independent feature learning. To address these issues, we propose a global-local anchor representation (GLAR) learning method for inductive KGC. Unlike previous methods that utilize enclosing subgraphs, we extract a shared opening subgraph for all candidates and perform reasoning on it, enabling the model to perform reasoning more efficiently. Moreover, we design some transferable global and local anchors to learn rich entity-independent features for emerging entities. Finally, a global-local graph reasoning model is applied on the opening subgraph to rank all candidates. Extensive experiments show that our GLAR outperforms most existing state-of-the-art methods.

Title: Facilitating Advanced Sentinel-2 Analysis Through a Simplified Computation of Nadir BRDF Adjusted Reflectance

• Authors: David Montero, Miguel D. Mahecha, César Aybar, Clemens Mosig, Sebastian Wieneke
• Subjects: cs.CV, astro-ph.IM
• Abstract URL: https://arxiv.org/abs/2404.15812
• Pdf URL: https://arxiv.org/pdf/2404.15812
• Copy Paste: [[2404.15812]] Facilitating Advanced Sentinel-2 Analysis Through a Simplified Computation of Nadir BRDF Adjusted Reflectance(https://arxiv.org/abs/2404.15812)
• Keywords: robust
• Abstract: The Sentinel-2 (S2) mission from the European Space Agency's Copernicus program provides essential data for Earth surface analysis. Its Level-2A products deliver high-to-medium resolution (10-60 m) surface reflectance (SR) data through the MultiSpectral Instrument (MSI). To enhance the accuracy and comparability of SR data, adjustments simulating a nadir viewing perspective are essential. These corrections address the anisotropic nature of SR and the variability in sun and observation angles, ensuring consistent image comparisons over time and under different conditions. The $c$-factor method, a simple yet effective algorithm, adjusts observed S2 SR by using the MODIS BRDF model to achieve Nadir BRDF Adjusted Reflectance (NBAR). Despite the straightforward application of the $c$-factor to individual images, a cohesive Python framework for its application across multiple S2 images and Earth System Data Cubes (ESDCs) from cloud-stored data has been lacking. Here we introduce sen2nbar, a Python package crafted to convert S2 SR data to NBAR, supporting both individual images and ESDCs derived from cloud-stored data. This package simplifies the conversion of S2 SR data to NBAR via a single function, organized into modules for efficient process management. By facilitating NBAR conversion for both SAFE files and ESDCs from SpatioTemporal Asset Catalogs (STAC), sen2nbar is developed as a flexible tool that can handle diverse data format requirements. We anticipate that sen2nbar will considerably contribute to the standardization and harmonization of S2 data, offering a robust solution for a diverse range of users across various applications. sen2nbar is an open-source tool available at https://github.com/ESDS-Leipzig/sen2nbar.

Title: Fast Ensembling with Diffusion Schrödinger Bridge

• Authors: Hyunsu Kim, Jongmin Yoon, Juho Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15814
• Pdf URL: https://arxiv.org/pdf/2404.15814
• Copy Paste: [[2404.15814]] Fast Ensembling with Diffusion Schrödinger Bridge(https://arxiv.org/abs/2404.15814)
• Keywords: diffusion
• Abstract: Deep Ensemble (DE) approach is a straightforward technique used to enhance the performance of deep neural networks by training them from different initial points, converging towards various local optima. However, a limitation of this methodology lies in its high computational overhead for inference, arising from the necessity to store numerous learned parameters and execute individual forward passes for each parameter during the inference stage. We propose a novel approach called Diffusion Bridge Network (DBN) to address this challenge. Based on the theory of the Schr\"odinger bridge, this method directly learns to simulate an Stochastic Differential Equation (SDE) that connects the output distribution of a single ensemble member to the output distribution of the ensembled model, allowing us to obtain ensemble prediction without having to invoke forward pass through all the ensemble models. By substituting the heavy ensembles with this lightweight neural network constructing DBN, we achieved inference with reduced computational cost while maintaining accuracy and uncertainty scores on benchmark datasets such as CIFAR-10, CIFAR-100, and TinyImageNet. Our implementation is available at https://github.com/kim-hyunsu/dbn.

Title: Vision Transformer-based Adversarial Domain Adaptation

• Authors: Yahan Li, Yuan Wu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15817
• Pdf URL: https://arxiv.org/pdf/2404.15817
• Copy Paste: [[2404.15817]] Vision Transformer-based Adversarial Domain Adaptation(https://arxiv.org/abs/2404.15817)
• Keywords: transformer, segmentation
• Abstract: Unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. The most recent UDA methods always resort to adversarial training to yield state-of-the-art results and a dominant number of existing UDA methods employ convolutional neural networks (CNNs) as feature extractors to learn domain invariant features. Vision transformer (ViT) has attracted tremendous attention since its emergence and has been widely used in various computer vision tasks, such as image classification, object detection, and semantic segmentation, yet its potential in adversarial domain adaptation has never been investigated. In this paper, we fill this gap by employing the ViT as the feature extractor in adversarial domain adaptation. Moreover, we empirically demonstrate that ViT can be a plug-and-play component in adversarial domain adaptation, which means directly replacing the CNN-based feature extractor in existing UDA methods with the ViT-based feature extractor can easily obtain performance improvement. The code is available at https://github.com/LluckyYH/VT-ADA.

Title: SynthEval: A Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Data

• Authors: Anton Danholt Lautrup, Tobias Hyrup, Arthur Zimek, Peter Schneider-Kamp
• Subjects: cs.LG, cs.PF
• Abstract URL: https://arxiv.org/abs/2404.15821
• Pdf URL: https://arxiv.org/pdf/2404.15821
• Copy Paste: [[2404.15821]] SynthEval: A Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Data(https://arxiv.org/abs/2404.15821)
• Keywords: privacy, robust, fair
• Abstract: With the growing demand for synthetic data to address contemporary issues in machine learning, such as data scarcity, data fairness, and data privacy, having robust tools for assessing the utility and potential privacy risks of such data becomes crucial. SynthEval, a novel open-source evaluation framework distinguishes itself from existing tools by treating categorical and numerical attributes with equal care, without assuming any special kind of preprocessing steps. This~makes it applicable to virtually any synthetic dataset of tabular records. Our tool leverages statistical and machine learning techniques to comprehensively evaluate synthetic data fidelity and privacy-preserving integrity. SynthEval integrates a wide selection of metrics that can be used independently or in highly customisable benchmark configurations, and can easily be extended with additional metrics. In this paper, we describe SynthEval and illustrate its versatility with examples. The framework facilitates better benchmarking and more consistent comparisons of model capabilities.

Title: Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation

• Authors: Maja Stahl, Leon Biermann, Andreas Nehring, Henning Wachsmuth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15845
• Pdf URL: https://arxiv.org/pdf/2404.15845
• Copy Paste: [[2404.15845]] Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation(https://arxiv.org/abs/2404.15845)
• Keywords: large language model
• Abstract: Individual feedback can help students improve their essay writing skills. However, the manual effort required to provide such feedback limits individualization in practice. Automatically-generated essay feedback may serve as an alternative to guide students at their own pace, convenience, and desired frequency. Large language models (LLMs) have demonstrated strong performance in generating coherent and contextually relevant text. Yet, their ability to provide helpful essay feedback is unclear. This work explores several prompting strategies for LLM-based zero-shot and few-shot generation of essay feedback. Inspired by Chain-of-Thought prompting, we study how and to what extent automated essay scoring (AES) can benefit the quality of generated feedback. We evaluate both the AES performance that LLMs can achieve with prompting only and the helpfulness of the generated essay feedback. Our results suggest that tackling AES and feedback generation jointly improves AES performance. However, while our manual evaluation emphasizes the quality of the generated essay feedback, the impact of essay scoring on the generated feedback remains low ultimately.

Title: From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models

• Authors: Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, Yanghua Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15846
• Pdf URL: https://arxiv.org/pdf/2404.15846
• Copy Paste: [[2404.15846]] From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models(https://arxiv.org/abs/2404.15846)
• Keywords: large language model
• Abstract: It is imperative for Large language models (LLMs) to follow instructions with elaborate requirements (i.e. Complex Instructions Following). Yet, it remains under-explored how to enhance the ability of LLMs to follow complex instructions with multiple constraints. To bridge the gap, we initially study what training data is effective in enhancing complex constraints following abilities. We found that training LLMs with instructions containing multiple constraints enhances their understanding of complex instructions, especially those with lower complexity levels. The improvement can even generalize to compositions of out-of-domain constraints. Additionally, we further propose methods addressing how to obtain and utilize the effective training data. Finally, we conduct extensive experiments to prove the effectiveness of our methods in terms of overall performance, training efficiency, and generalization abilities under four settings.

Title: Detecting Conceptual Abstraction in LLMs

• Authors: Michaela Regneri, Alhassan Abdelhalim, Sören Laue
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15848
• Pdf URL: https://arxiv.org/pdf/2404.15848
• Copy Paste: [[2404.15848]] Detecting Conceptual Abstraction in LLMs(https://arxiv.org/abs/2404.15848)
• Keywords: explainability, large language model
• Abstract: We present a novel approach to detecting noun abstraction within a large language model (LLM). Starting from a psychologically motivated set of noun pairs in taxonomic relationships, we instantiate surface patterns indicating hypernymy and analyze the attention matrices produced by BERT. We compare the results to two sets of counterfactuals and show that we can detect hypernymy in the abstraction mechanism, which cannot solely be related to the distributional similarity of noun pairs. Our findings are a first step towards the explainability of conceptual abstraction in LLMs.

Title: Porting Large Language Models to Mobile Devices for Question Answering

• Authors: Hannes Fassold
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15851
• Pdf URL: https://arxiv.org/pdf/2404.15851
• Copy Paste: [[2404.15851]] Porting Large Language Models to Mobile Devices for Question Answering(https://arxiv.org/abs/2404.15851)
• Keywords: large language model
• Abstract: Deploying Large Language Models (LLMs) on mobile devices makes all the capabilities of natural language processing available on the device. An important use case of LLMs is question answering, which can provide accurate and contextually relevant answers to a wide array of user queries. We describe how we managed to port state of the art LLMs to mobile devices, enabling them to operate natively on the device. We employ the llama.cpp framework, a flexible and self-contained C++ framework for LLM inference. We selected a 6-bit quantized version of the Orca-Mini-3B model with 3 billion parameters and present the correct prompt format for this model. Experimental results show that LLM inference runs in interactive speed on a Galaxy S21 smartphone and that the model delivers high-quality answers to user queries related to questions from different subjects like politics, geography or history.

Title: CLAD: Robust Audio Deepfake Detection Against Manipulation Attacks with Contrastive Learning

• Authors: Haolin Wu, Jing Chen, Ruiying Du, Cong Wu, Kun He, Xingcan Shang, Hao Ren, Guowen Xu
• Subjects: cs.CR, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.15854
• Pdf URL: https://arxiv.org/pdf/2404.15854
• Copy Paste: [[2404.15854]] CLAD: Robust Audio Deepfake Detection Against Manipulation Attacks with Contrastive Learning(https://arxiv.org/abs/2404.15854)
• Keywords: security, attack, robust
• Abstract: The increasing prevalence of audio deepfakes poses significant security threats, necessitating robust detection methods. While existing detection systems exhibit promise, their robustness against malicious audio manipulations remains underexplored. To bridge the gap, we undertake the first comprehensive study of the susceptibility of the most widely adopted audio deepfake detectors to manipulation attacks. Surprisingly, even manipulations like volume control can significantly bypass detection without affecting human perception. To address this, we propose CLAD (Contrastive Learning-based Audio deepfake Detector) to enhance the robustness against manipulation attacks. The key idea is to incorporate contrastive learning to minimize the variations introduced by manipulations, therefore enhancing detection robustness. Additionally, we incorporate a length loss, aiming to improve the detection accuracy by clustering real audios more closely in the feature space. We comprehensively evaluated the most widely adopted audio deepfake detection models and our proposed CLAD against various manipulation attacks. The detection models exhibited vulnerabilities, with FAR rising to 36.69%, 31.23%, and 51.28% under volume control, fading, and noise injection, respectively. CLAD enhanced robustness, reducing the FAR to 0.81% under noise injection and consistently maintaining an FAR below 1.63% across all tests. Our source code and documentation are available in the artifact repository (https://github.com/CLAD23/CLAD).

Title: CONNECTION: COvert chaNnel NEtwork attaCk Through bIt-rate mOdulatioN

• Authors: Simone Soderi, Rocco De Nicola
• Subjects: cs.CR, cs.IT, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.15858
• Pdf URL: https://arxiv.org/pdf/2404.15858
• Copy Paste: [[2404.15858]] CONNECTION: COvert chaNnel NEtwork attaCk Through bIt-rate mOdulatioN(https://arxiv.org/abs/2404.15858)
• Keywords: security, protect, attack, robust, steal
• Abstract: Covert channel networks are a well-known method for circumventing the security measures organizations put in place to protect their networks from adversarial attacks. This paper introduces a novel method based on bit-rate modulation for implementing covert channels between devices connected over a wide area network. This attack can be exploited to exfiltrate sensitive information from a machine (i.e., covert sender) and stealthily transfer it to a covert receiver while evading network security measures and detection systems. We explain how to implement this threat, focusing specifically on covert channel networks and their potential security risks to network information transmission. The proposed method leverages bit-rate modulation, where a high bit rate represents a '1' and a low bit rate represents a '0', enabling covert communication. We analyze the key metrics associated with covert channels, including robustness in the presence of legitimate traffic and other interference, bit-rate capacity, and bit error rate. Experiments demonstrate the good performance of this attack, which achieved 5 bps with excellent robustness and a channel capacity of up to 0.9239 bps/Hz under different noise sources. Therefore, we show that bit-rate modulation effectively violates network security and compromises sensitive data.

Title: Secure and Privacy-Preserving Authentication for Data Subject Rights Enforcement

• Authors: Malte Hansen, Andre Büttner
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15859
• Pdf URL: https://arxiv.org/pdf/2404.15859
• Copy Paste: [[2404.15859]] Secure and Privacy-Preserving Authentication for Data Subject Rights Enforcement(https://arxiv.org/abs/2404.15859)
• Keywords: secure, security, privacy
• Abstract: In light of the GDPR, data controllers (DC) need to allow data subjects (DS) to exercise certain data subject rights. A key requirement here is that DCs can reliably authenticate a DS. Due to a lack of clear technical specifications, this has been realized in different ways, such as by requesting copies of ID documents or by email address verification. However, previous research has shown that this is associated with various security and privacy risks and that identifying DSs can be a non-trivial task. In this paper, we review different authentication schemes and propose an architecture that enables DCs to authenticate DSs with the help of independent Identity Providers in a secure and privacy-preserving manner by utilizing attribute-based credentials and eIDs. Our work contributes to a more standardized and privacy-preserving way of authenticating DSs, which will benefit both DCs and DSs.

Title: Steal Now and Attack Later: Evaluating Robustness of Object Detection against Black-box Adversarial Attacks

• Authors: Erh-Chung Chen, Pin-Yu Chen, I-Hsin Chung, Che-Rung Lee
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15881
• Pdf URL: https://arxiv.org/pdf/2404.15881
• Copy Paste: [[2404.15881]] Steal Now and Attack Later: Evaluating Robustness of Object Detection against Black-box Adversarial Attacks(https://arxiv.org/abs/2404.15881)
• Keywords: security, attack, robust, steal
• Abstract: Latency attacks against object detection represent a variant of adversarial attacks that aim to inflate the inference time by generating additional ghost objects in a target image. However, generating ghost objects in the black-box scenario remains a challenge since information about these unqualified objects remains opaque. In this study, we demonstrate the feasibility of generating ghost objects in adversarial examples by extending the concept of "steal now, decrypt later" attacks. These adversarial examples, once produced, can be employed to exploit potential vulnerabilities in the AI service, giving rise to significant security concerns. The experimental results demonstrate that the proposed attack achieves successful attacks across various commonly used models and Google Vision API without any prior knowledge about the target model. Additionally, the average cost of each attack is less than \$ 1 dollars, posing a significant threat to AI security.

Title: Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains

• Authors: Eunsu Baek, Keondo Park, Jiyoon Kim, Hyung-Sin Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15882
• Pdf URL: https://arxiv.org/pdf/2404.15882
• Copy Paste: [[2404.15882]] Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains(https://arxiv.org/abs/2404.15882)
• Keywords: robust
• Abstract: Computer vision applications predict on digital images acquired by a camera from physical scenes through light. However, conventional robustness benchmarks rely on perturbations in digitized images, diverging from distribution shifts occurring in the image acquisition process. To bridge this gap, we introduce a new distribution shift dataset, ImageNet-ES, comprising variations in environmental and camera sensor factors by directly capturing 202k images with a real camera in a controllable testbed. With the new dataset, we evaluate out-of-distribution (OOD) detection and model robustness. We find that existing OOD detection methods do not cope with the covariate shifts in ImageNet-ES, implying that the definition and detection of OOD should be revisited to embrace real-world distribution shifts. We also observe that the model becomes more robust in both ImageNet-C and -ES by learning environment and sensor variations in addition to existing digital augmentations. Lastly, our results suggest that effective shift mitigation via camera sensor control can significantly improve performance without increasing model size. With these findings, our benchmark may aid future research on robustness, OOD, and camera sensor control for computer vision. Our code and dataset are available at https://github.com/Edw2n/ImageNet-ES.

Title: Privacy-Preserving Billing for Local Energy Markets (Long Version)

• Authors: Eman Alqahtani, Mustafa A. Mustafa
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.15886
• Pdf URL: https://arxiv.org/pdf/2404.15886
• Copy Paste: [[2404.15886]] Privacy-Preserving Billing for Local Energy Markets (Long Version)(https://arxiv.org/abs/2404.15886)
• Keywords: security, privacy
• Abstract: We propose a privacy-preserving billing protocol for local energy markets (PBP-LEMs) that takes into account market participants' energy volume deviations from their bids. PBP-LEMs enables a group of market entities to jointly compute participants' bills in a decentralized and privacy-preserving manner without sacrificing correctness. It also mitigates risks on individuals' privacy arising from any potential internal collusion. We first propose a novel, efficient, and privacy-preserving individual billing scheme, achieving information-theoretic security, which serves as a building block. PBP-LEMs utilizes this scheme, along with other techniques such as multiparty computation, Pedersen commitments and inner product functional encryption, to ensure data confidentiality and accuracy. Additionally, we present three approaches, resulting in different levels of privacy and performance. We prove that the protocol meets its security and privacy requirements and is feasible for deployment in real LEMs. Our analysis also shows variations in overall performance and identifies areas where overhead is concentrated based on the applied approach.

Title: Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Control

• Authors: Linzi Qu, Jiaxiang Shang, Hui Ye, Xiaoguang Han, Hongbo Fu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.15889
• Pdf URL: https://arxiv.org/pdf/2404.15889
• Copy Paste: [[2404.15889]] Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Control(https://arxiv.org/abs/2404.15889)
• Keywords: diffusion
• Abstract: Geometry- and appearance-controlled full-body human image generation is an interesting but challenging task. Existing solutions are either unconditional or dependent on coarse conditions (e.g., pose, text), thus lacking explicit geometry and appearance control of body and garment. Sketching offers such editing ability and has been adopted in various sketch-based face generation and editing solutions. However, directly adapting sketch-based face generation to full-body generation often fails to produce high-fidelity and diverse results due to the high complexity and diversity in the pose, body shape, and garment shape and texture. Recent geometrically controllable diffusion-based methods mainly rely on prompts to generate appearance and it is hard to balance the realism and the faithfulness of their results to the sketch when the input is coarse. This work presents Sketch2Human, the first system for controllable full-body human image generation guided by a semantic sketch (for geometry control) and a reference image (for appearance control). Our solution is based on the latent space of StyleGAN-Human with inverted geometry and appearance latent codes as input. Specifically, we present a sketch encoder trained with a large synthetic dataset sampled from StyleGAN-Human's latent space and directly supervised by sketches rather than real images. Considering the entangled information of partial geometry and texture in StyleGAN-Human and the absence of disentangled datasets, we design a novel training scheme that creates geometry-preserved and appearance-transferred training data to tune a generator to achieve disentangled geometry and appearance control. Although our method is trained with synthetic data, it can handle hand-drawn sketches as well. Qualitative and quantitative evaluations demonstrate the superior performance of our method to state-of-the-art methods.

Title: OMEGAS: Object Mesh Extraction from Large Scenes Guided by Gaussian Segmentation

• Authors: Lizhi Wang, Feng Zhou, Jianqin Yin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15891
• Pdf URL: https://arxiv.org/pdf/2404.15891
• Copy Paste: [[2404.15891]] OMEGAS: Object Mesh Extraction from Large Scenes Guided by Gaussian Segmentation(https://arxiv.org/abs/2404.15891)
• Keywords: extraction, diffusion, segmentation
• Abstract: Recent advancements in 3D reconstruction technologies have paved the way for high-quality and real-time rendering of complex 3D scenes. Despite these achievements, a notable challenge persists: it is difficult to precisely reconstruct specific objects from large scenes. Current scene reconstruction techniques frequently result in the loss of object detail textures and are unable to reconstruct object portions that are occluded or unseen in views. To address this challenge, we delve into the meticulous 3D reconstruction of specific objects within large scenes and propose a framework termed OMEGAS: Object Mesh Extraction from Large Scenes Guided by GAussian Segmentation. OMEGAS employs a multi-step approach, grounded in several excellent off-the-shelf methodologies. Specifically, initially, we utilize the Segment Anything Model (SAM) to guide the segmentation of 3D Gaussian Splatting (3DGS), thereby creating a basic 3DGS model of the target object. Then, we leverage large-scale diffusion priors to further refine the details of the 3DGS model, especially aimed at addressing invisible or occluded object portions from the original scene views. Subsequently, by re-rendering the 3DGS model onto the scene views, we achieve accurate object segmentation and effectively remove the background. Finally, these target-only images are used to improve the 3DGS model further and extract the definitive 3D object mesh by the SuGaR model. In various scenarios, our experiments demonstrate that OMEGAS significantly surpasses existing scene reconstruction methods. Our project page is at: https://github.com/CrystalWlz/OMEGAS

Title: Assessing The Potential Of Mid-Sized Language Models For Clinical QA

• Authors: Elliot Bolton, Betty Xiong, Vijaytha Muralidharan, Joel Schamroth, Vivek Muralidharan, Christopher D. Manning, Roxana Daneshjou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15894
• Pdf URL: https://arxiv.org/pdf/2404.15894
• Copy Paste: [[2404.15894]] Assessing The Potential Of Mid-Sized Language Models For Clinical QA(https://arxiv.org/abs/2404.15894)
• Keywords: large language model
• Abstract: Large language models, such as GPT-4 and Med-PaLM, have shown impressive performance on clinical tasks; however, they require access to compute, are closed-source, and cannot be deployed on device. Mid-size models such as BioGPT-large, BioMedLM, LLaMA 2, and Mistral 7B avoid these drawbacks, but their capacity for clinical tasks has been understudied. To help assess their potential for clinical use and help researchers decide which model they should use, we compare their performance on two clinical question-answering (QA) tasks: MedQA and consumer query answering. We find that Mistral 7B is the best performing model, winning on all benchmarks and outperforming models trained specifically for the biomedical domain. While Mistral 7B's MedQA score of 63.0% approaches the original Med-PaLM, and it often can produce plausible responses to consumer health queries, room for improvement still exists. This study provides the first head-to-head assessment of open source mid-sized models on clinical tasks.

Title: ST-MambaSync: The Confluence of Mamba Structure and Spatio-Temporal Transformers for Precipitous Traffic Prediction

• Authors: Zhiqi Shao, Xusheng Yao, Ze Wang, Junbin Gao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15899
• Pdf URL: https://arxiv.org/pdf/2404.15899
• Copy Paste: [[2404.15899]] ST-MambaSync: The Confluence of Mamba Structure and Spatio-Temporal Transformers for Precipitous Traffic Prediction(https://arxiv.org/abs/2404.15899)
• Keywords: transformer
• Abstract: Balancing accuracy with computational efficiency is paramount in machine learning, particularly when dealing with high-dimensional data, such as spatial-temporal datasets. This study introduces ST-MambaSync, an innovative framework that integrates a streamlined attention layer with a simplified state-space layer. The model achieves competitive accuracy in spatial-temporal prediction tasks. We delve into the relationship between attention mechanisms and the Mamba component, revealing that Mamba functions akin to attention within a residual network structure. This comparative analysis underpins the efficiency of state-space models, elucidating their capability to deliver superior performance at reduced computational costs.

Title: Drawing the Line: Deep Segmentation for Extracting Art from Ancient Etruscan Mirrors

• Authors: Rafael Sterzinger, Simon Brenner, Robert Sablatnig
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15903
• Pdf URL: https://arxiv.org/pdf/2404.15903
• Copy Paste: [[2404.15903]] Drawing the Line: Deep Segmentation for Extracting Art from Ancient Etruscan Mirrors(https://arxiv.org/abs/2404.15903)
• Keywords: segmentation
• Abstract: Etruscan mirrors constitute a significant category within Etruscan art and, therefore, undergo systematic examinations to obtain insights into ancient times. A crucial aspect of their analysis involves the labor-intensive task of manually tracing engravings from the backside. Additionally, this task is inherently challenging due to the damage these mirrors have sustained, introducing subjectivity into the process. We address these challenges by automating the process through photometric-stereo scanning in conjunction with deep segmentation networks which, however, requires effective usage of the limited data at hand. We accomplish this by incorporating predictions on a per-patch level, and various data augmentations, as well as exploring self-supervised learning. Compared to our baseline, we improve predictive performance w.r.t. the pseudo-F-Measure by around 16%. When assessing performance on complete mirrors against a human baseline, our approach yields quantitative similar performance to a human annotator and significantly outperforms existing binarization methods. With our proposed methodology, we streamline the annotation process, enhance its objectivity, and reduce overall workload, offering a valuable contribution to the examination of these historical artifacts and other non-traditional documents.

Title: Learning Long-form Video Prior via Generative Pre-Training

• Authors: Jinheng Xie, Jiajun Feng, Zhaoxu Tian, Kevin Qinghong Lin, Yawen Huang, Xi Xia, Nanxu Gong, Xu Zuo, Jiaqi Yang, Yefeng Zheng, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15909
• Pdf URL: https://arxiv.org/pdf/2404.15909
• Copy Paste: [[2404.15909]] Learning Long-form Video Prior via Generative Pre-Training(https://arxiv.org/abs/2404.15909)
• Keywords: generative
• Abstract: Concepts involved in long-form videos such as people, objects, and their interactions, can be viewed as following an implicit prior. They are notably complex and continue to pose challenges to be comprehensively learned. In recent years, generative pre-training (GPT) has exhibited versatile capacities in modeling any kind of text content even visual locations. Can this manner work for learning long-form video prior? Instead of operating on pixel space, it is efficient to employ visual locations like bounding boxes and keypoints to represent key information in videos, which can be simply discretized and then tokenized for consumption by GPT. Due to the scarcity of suitable data, we create a new dataset called \textbf{Storyboard20K} from movies to serve as a representative. It includes synopses, shot-by-shot keyframes, and fine-grained annotations of film sets and characters with consistent IDs, bounding boxes, and whole body keypoints. In this way, long-form videos can be represented by a set of tokens and be learned via generative pre-training. Experimental results validate that our approach has great potential for learning long-form video prior. Code and data will be released at \url{https://github.com/showlab/Long-form-Video-Prior}.

Title: An Element-Wise Weights Aggregation Method for Federated Learning

• Authors: Yi Hu, Hanchi Ren, Chen Hu, Jingjing Deng, Xianghua Xie
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15919
• Pdf URL: https://arxiv.org/pdf/2404.15919
• Copy Paste: [[2404.15919]] An Element-Wise Weights Aggregation Method for Federated Learning(https://arxiv.org/abs/2404.15919)
• Keywords: privacy, robust, federate
• Abstract: Federated learning (FL) is a powerful Machine Learning (ML) paradigm that enables distributed clients to collaboratively learn a shared global model while keeping the data on the original device, thereby preserving privacy. A central challenge in FL is the effective aggregation of local model weights from disparate and potentially unbalanced participating clients. Existing methods often treat each client indiscriminately, applying a single proportion to the entire local model. However, it is empirically advantageous for each weight to be assigned a specific proportion. This paper introduces an innovative Element-Wise Weights Aggregation Method for Federated Learning (EWWA-FL) aimed at optimizing learning performance and accelerating convergence speed. Unlike traditional FL approaches, EWWA-FL aggregates local weights to the global model at the level of individual elements, thereby allowing each participating client to make element-wise contributions to the learning process. By taking into account the unique dataset characteristics of each client, EWWA-FL enhances the robustness of the global model to different datasets while also achieving rapid convergence. The method is flexible enough to employ various weighting strategies. Through comprehensive experiments, we demonstrate the advanced capabilities of EWWA-FL, showing significant improvements in both accuracy and convergence speed across a range of backbones and benchmarks.

Title: Generalization Measures for Zero-Shot Cross-Lingual Transfer

• Authors: Saksham Bassi, Duygu Ataman, Kyunghyun Cho
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15928
• Pdf URL: https://arxiv.org/pdf/2404.15928
• Copy Paste: [[2404.15928]] Generalization Measures for Zero-Shot Cross-Lingual Transfer(https://arxiv.org/abs/2404.15928)
• Keywords: robust
• Abstract: A model's capacity to generalize its knowledge to interpret unseen inputs with different characteristics is crucial to build robust and reliable machine learning systems. Language model evaluation tasks lack information metrics about model generalization and their applicability in a new setting is measured using task and language-specific downstream performance, which is often lacking in many languages and tasks. In this paper, we explore a set of efficient and reliable measures that could aid in computing more information related to the generalization capability of language models in cross-lingual zero-shot settings. In addition to traditional measures such as variance in parameters after training and distance from initialization, we also measure the effectiveness of sharpness in loss landscape in capturing the success in cross-lingual transfer and propose a novel and stable algorithm to reliably compute the sharpness of a model optimum that correlates to generalization.

Title: Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Scheme

• Authors: Qianyu Long, Qiyuan Wang, Christos Anagnostopoulos, Daning Bi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15943
• Pdf URL: https://arxiv.org/pdf/2404.15943
• Copy Paste: [[2404.15943]] Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Scheme(https://arxiv.org/abs/2404.15943)
• Keywords: robust, federate
• Abstract: Decentralized Federated Learning (DFL) has become popular due to its robustness and avoidance of centralized coordination. In this paradigm, clients actively engage in training by exchanging models with their networked neighbors. However, DFL introduces increased costs in terms of training and communication. Existing methods focus on minimizing communication often overlooking training efficiency and data heterogeneity. To address this gap, we propose a novel \textit{sparse-to-sparser} training scheme: DA-DPFL. DA-DPFL initializes with a subset of model parameters, which progressively reduces during training via \textit{dynamic aggregation} and leads to substantial energy savings while retaining adequate information during critical learning periods. Our experiments showcase that DA-DPFL substantially outperforms DFL baselines in test accuracy, while achieving up to $5$ times reduction in energy costs. We provide a theoretical analysis of DA-DPFL's convergence by solidifying its applicability in decentralized and personalized learning. The code is available at:https://github.com/EricLoong/da-dpfl

Title: Mammo-CLIP: Leveraging Contrastive Language-Image Pre-training (CLIP) for Enhanced Breast Cancer Diagnosis with Multi-view Mammography

• Authors: Xuxin Chen, Yuheng Li, Mingzhe Hu, Ella Salari, Xiaoqian Chen, Richard L.J. Qiu, Bin Zheng, Xiaofeng Yang
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.15946
• Pdf URL: https://arxiv.org/pdf/2404.15946
• Copy Paste: [[2404.15946]] Mammo-CLIP: Leveraging Contrastive Language-Image Pre-training (CLIP) for Enhanced Breast Cancer Diagnosis with Multi-view Mammography(https://arxiv.org/abs/2404.15946)
• Keywords: transformer
• Abstract: Although fusion of information from multiple views of mammograms plays an important role to increase accuracy of breast cancer detection, developing multi-view mammograms-based computer-aided diagnosis (CAD) schemes still faces challenges and no such CAD schemes have been used in clinical practice. To overcome the challenges, we investigate a new approach based on Contrastive Language-Image Pre-training (CLIP), which has sparked interest across various medical imaging tasks. By solving the challenges in (1) effectively adapting the single-view CLIP for multi-view feature fusion and (2) efficiently fine-tuning this parameter-dense model with limited samples and computational resources, we introduce Mammo-CLIP, the first multi-modal framework to process multi-view mammograms and corresponding simple texts. Mammo-CLIP uses an early feature fusion strategy to learn multi-view relationships in four mammograms acquired from the CC and MLO views of the left and right breasts. To enhance learning efficiency, plug-and-play adapters are added into CLIP image and text encoders for fine-tuning parameters and limiting updates to about 1% of the parameters. For framework evaluation, we assembled two datasets retrospectively. The first dataset, comprising 470 malignant and 479 benign cases, was used for few-shot fine-tuning and internal evaluation of the proposed Mammo-CLIP via 5-fold cross-validation. The second dataset, including 60 malignant and 294 benign cases, was used to test generalizability of Mammo-CLIP. Study results show that Mammo-CLIP outperforms the state-of-art cross-view transformer in AUC (0.841 vs. 0.817, 0.837 vs. 0.807) on both datasets. It also surpasses previous two CLIP-based methods by 20.3% and 14.3%. This study highlights the potential of applying the finetuned vision-language models for developing next-generation, image-text-based CAD schemes of breast cancer.

Title: Sequence can Secretly Tell You What to Discard

• Authors: Jincheng Dai, Zhuowei Huang, Haiyun Jiang, Chen Chen, Deng Cai, Wei Bi, Shuming Shi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15949
• Pdf URL: https://arxiv.org/pdf/2404.15949
• Copy Paste: [[2404.15949]] Sequence can Secretly Tell You What to Discard(https://arxiv.org/abs/2404.15949)
• Keywords: large language model
• Abstract: Large Language Models (LLMs), despite their impressive performance on a wide range of tasks, require significant GPU memory and consume substantial computational resources. In addition to model weights, the memory occupied by KV cache increases linearly with sequence length, becoming a main bottleneck for inference. In this paper, we introduce a novel approach for optimizing the KV cache which significantly reduces its memory footprint. Through a comprehensive investigation, we find that on LLaMA2 series models, (i) the similarity between adjacent tokens' query vectors is remarkably high, and (ii) current query's attention calculation can rely solely on the attention information of a small portion of the preceding queries. Based on these observations, we propose CORM, a KV cache eviction policy that dynamically retains important key-value pairs for inference without finetuning the model. We validate that CORM reduces the inference memory usage of KV cache by up to 70% without noticeable performance degradation across six tasks in LongBench.

Title: Beyond Deepfake Images: Detecting AI-Generated Videos

• Authors: Danial Samadi Vahdati, Tai D. Nguyen, Aref Azizpour, Matthew C. Stamm
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15955
• Pdf URL: https://arxiv.org/pdf/2404.15955
• Copy Paste: [[2404.15955]] Beyond Deepfake Images: Detecting AI-Generated Videos(https://arxiv.org/abs/2404.15955)
• Keywords: generative
• Abstract: Recent advances in generative AI have led to the development of techniques to generate visually realistic synthetic video. While a number of techniques have been developed to detect AI-generated synthetic images, in this paper we show that synthetic image detectors are unable to detect synthetic videos. We demonstrate that this is because synthetic video generators introduce substantially different traces than those left by image generators. Despite this, we show that synthetic video traces can be learned, and used to perform reliable synthetic video detection or generator source attribution even after H.264 re-compression. Furthermore, we demonstrate that while detecting videos from new generators through zero-shot transferability is challenging, accurate detection of videos from a new generator can be achieved through few-shot learning.

Title: A Survey on Visual Mamba

• Authors: Hanwei Zhang, Ying Zhu, Dan Wang, Lijun Zhang, Tianxiang Chen, Zi Ye
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15956
• Pdf URL: https://arxiv.org/pdf/2404.15956
• Copy Paste: [[2404.15956]] A Survey on Visual Mamba(https://arxiv.org/abs/2404.15956)
• Keywords: transformer, segmentation
• Abstract: State space models (SSMs) with selection mechanisms and hardware-aware architectures, namely Mamba, have recently demonstrated significant promise in long-sequence modeling. Since the self-attention mechanism in transformers has quadratic complexity with image size and increasing computational demands, the researchers are now exploring how to adapt Mamba for computer vision tasks. This paper is the first comprehensive survey aiming to provide an in-depth analysis of Mamba models in the field of computer vision. It begins by exploring the foundational concepts contributing to Mamba's success, including the state space model framework, selection mechanisms, and hardware-aware design. Next, we review these vision mamba models by categorizing them into foundational ones and enhancing them with techniques such as convolution, recurrence, and attention to improve their sophistication. We further delve into the widespread applications of Mamba in vision tasks, which include their use as a backbone in various levels of vision processing. This encompasses general visual tasks, Medical visual tasks (e.g., 2D / 3D segmentation, classification, and image registration, etc.), and Remote Sensing visual tasks. We specially introduce general visual tasks from two levels: High/Mid-level vision (e.g., Object detection, Segmentation, Video classification, etc.) and Low-level vision (e.g., Image super-resolution, Image restoration, Visual generation, etc.). We hope this endeavor will spark additional interest within the community to address current challenges and further apply Mamba models in computer vision.

Title: HDDGAN: A Heterogeneous Dual-Discriminator Generative Adversarial Network for Infrared and Visible Image Fusion

• Authors: Guosheng Lu, Zile Fang, Chunming He, Zhigang Zhao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.15992
• Pdf URL: https://arxiv.org/pdf/2404.15992
• Copy Paste: [[2404.15992]] HDDGAN: A Heterogeneous Dual-Discriminator Generative Adversarial Network for Infrared and Visible Image Fusion(https://arxiv.org/abs/2404.15992)
• Keywords: extraction, generative
• Abstract: Infrared and visible image fusion (IVIF) aims to preserve thermal radiation information from infrared images while integrating texture details from visible images, enabling the capture of important features and hidden details of subjects in complex scenes and disturbed environments. Consequently, IVIF offers distinct advantages in practical applications such as video surveillance, night navigation, and target recognition. However, prevailing methods often face challenges in simultaneously capturing thermal region features and detailed information due to the disparate characteristics of infrared and visible images. Consequently, fusion outcomes frequently entail a compromise between thermal target area information and texture details. In this study, we introduce a novel heterogeneous dual-discriminator generative adversarial network (HDDGAN) to address this issue. Specifically, the generator is structured as a multi-scale skip-connected structure, facilitating the extraction of essential features from different source images. To enhance the information representation ability of the fusion result, an attention mechanism is employed to construct the information fusion layer within the generator, leveraging the disparities between the source images. Moreover, recognizing the distinct learning requirements of information in infrared and visible images, we design two discriminators with differing structures. This approach aims to guide the model to learn salient information from infrared images while simultaneously capturing detailed information from visible images. Extensive experiments conducted on various public datasets demonstrate the superiority of our proposed HDDGAN over other state-of-the-art (SOTA) algorithms, highlighting its enhanced potential for practical applications.

Title: Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach

• Authors: Linyu Liu, Yu Pan, Xiaocheng Li, Guanting Chen
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15993
• Pdf URL: https://arxiv.org/pdf/2404.15993
• Copy Paste: [[2404.15993]] Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach(https://arxiv.org/abs/2404.15993)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) are highly capable of many tasks but they can sometimes generate unreliable or inaccurate outputs. To tackle this issue, this paper studies the problem of uncertainty estimation and calibration for LLMs. We begin by formulating the uncertainty estimation problem for LLMs and then propose a supervised approach that takes advantage of the labeled datasets and estimates the uncertainty of the LLMs' responses. Based on the formulation, we illustrate the difference between the uncertainty estimation for LLMs and that for standard ML models and explain why the hidden activations of the LLMs contain uncertainty information. Our designed approach effectively demonstrates the benefits of utilizing hidden activations for enhanced uncertainty estimation across various tasks and shows robust transferability in out-of-distribution settings. Moreover, we distinguish the uncertainty estimation task from the uncertainty calibration task and show that a better uncertainty estimation mode leads to a better calibration performance. In practice, our method is easy to implement and is adaptable to different levels of model transparency including black box, grey box, and white box, each demonstrating strong performance based on the accessibility of the LLM's internal mechanisms.

Title: BeSound: Bluetooth-Based Position Estimation Enhancing with Cross-Modality Distillation

• Authors: Hymalai Bello, Sungho Suh, Bo Zhou, Paul Lukowicz
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.15999
• Pdf URL: https://arxiv.org/pdf/2404.15999
• Copy Paste: [[2404.15999]] BeSound: Bluetooth-Based Position Estimation Enhancing with Cross-Modality Distillation(https://arxiv.org/abs/2404.15999)
• Keywords: privacy, protect
• Abstract: Smart factories leverage advanced technologies to optimize manufacturing processes and enhance efficiency. Implementing worker tracking systems, primarily through camera-based methods, ensures accurate monitoring. However, concerns about worker privacy and technology protection make it necessary to explore alternative approaches. We propose a non-visual, scalable solution using Bluetooth Low Energy (BLE) and ultrasound coordinates. BLE position estimation offers a very low-power and cost-effective solution, as the technology is available on smartphones and is scalable due to the large number of smartphone users, facilitating worker localization and safety protocol transmission. Ultrasound signals provide faster response times and higher accuracy but require custom hardware, increasing costs. To combine the benefits of both modalities, we employ knowledge distillation (KD) from ultrasound signals to BLE RSSI data. Once the student model is trained, the model only takes as inputs the BLE-RSSI data for inference, retaining the advantages of ubiquity and low cost of BLE RSSI. We tested our approach using data from an experiment with twelve participants in a smart factory test bed environment. We obtained an increase of 11.79% in the F1-score compared to the baseline (target model without KD and trained with BLE-RSSI data only).

Title: RetinaRegNet: A Versatile Approach for Retinal Image Registration

• Authors: Vishal Balaji Sivaraman, Muhammad Imran, Qingyue Wei, Preethika Muralidharan, Michelle R. Tamplin, Isabella M . Grumbach, Randy H. Kardon, Jui-Kai Wang, Yuyin Zhou, Wei Shao
• Subjects: cs.CV, cs.AI, cs.GT, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.16017
• Pdf URL: https://arxiv.org/pdf/2404.16017
• Copy Paste: [[2404.16017]] RetinaRegNet: A Versatile Approach for Retinal Image Registration(https://arxiv.org/abs/2404.16017)
• Keywords: diffusion
• Abstract: We introduce the RetinaRegNet model, which can achieve state-of-the-art performance across various retinal image registration tasks. RetinaRegNet does not require training on any retinal images. It begins by establishing point correspondences between two retinal images using image features derived from diffusion models. This process involves the selection of feature points from the moving image using the SIFT algorithm alongside random point sampling. For each selected feature point, a 2D correlation map is computed by assessing the similarity between the feature vector at that point and the feature vectors of all pixels in the fixed image. The pixel with the highest similarity score in the correlation map corresponds to the feature point in the moving image. To remove outliers in the estimated point correspondences, we first applied an inverse consistency constraint, followed by a transformation-based outlier detector. This method proved to outperform the widely used random sample consensus (RANSAC) outlier detector by a significant margin. To handle large deformations, we utilized a two-stage image registration framework. A homography transformation was used in the first stage and a more accurate third-order polynomial transformation was used in the second stage. The model's effectiveness was demonstrated across three retinal image datasets: color fundus images, fluorescein angiography images, and laser speckle flowgraphy images. RetinaRegNet outperformed current state-of-the-art methods in all three datasets. It was especially effective for registering image pairs with large displacement and scaling deformations. This innovation holds promise for various applications in retinal image analysis. Our code is publicly available at https://github.com/mirthAI/RetinaRegNet.

Title: The PRISM Alignment Project: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models

• Authors: Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, Scott A. Hale
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16019
• Pdf URL: https://arxiv.org/pdf/2404.16019
• Copy Paste: [[2404.16019]] The PRISM Alignment Project: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models(https://arxiv.org/abs/2404.16019)
• Keywords: large language model
• Abstract: Human feedback plays a central role in the alignment of Large Language Models (LLMs). However, open questions remain about the methods (how), domains (where), people (who) and objectives (to what end) of human feedback collection. To navigate these questions, we introduce PRISM, a new dataset which maps the sociodemographics and stated preferences of 1,500 diverse participants from 75 countries, to their contextual preferences and fine-grained feedback in 8,011 live conversations with 21 LLMs. PRISM contributes (i) wide geographic and demographic participation in human feedback data; (ii) two census-representative samples for understanding collective welfare (UK and US); and (iii) individualised feedback where every rating is linked to a detailed participant profile, thus permitting exploration of personalisation and attribution of sample artefacts. We focus on collecting conversations that centre subjective and multicultural perspectives on value-laden and controversial topics, where we expect the most interpersonal and cross-cultural disagreement. We demonstrate the usefulness of PRISM via three case studies of dialogue diversity, preference diversity, and welfare outcomes, showing that it matters which humans set alignment norms. As well as offering a rich community resource, we advocate for broader participation in AI development and a more inclusive approach to technology design.

Title: Universal Adversarial Triggers Are Not Universal

• Authors: Nicholas Meade, Arkil Patel, Siva Reddy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16020
• Pdf URL: https://arxiv.org/pdf/2404.16020
• Copy Paste: [[2404.16020]] Universal Adversarial Triggers Are Not Universal(https://arxiv.org/abs/2404.16020)
• Keywords: robust
• Abstract: Recent work has developed optimization procedures to find token sequences, called adversarial triggers, which can elicit unsafe responses from aligned language models. These triggers are believed to be universally transferable, i.e., a trigger optimized on one model can jailbreak other models. In this paper, we concretely show that such adversarial triggers are not universal. We extensively investigate trigger transfer amongst 13 open models and observe inconsistent transfer. Our experiments further reveal a significant difference in robustness to adversarial triggers between models Aligned by Preference Optimization (APO) and models Aligned by Fine-Tuning (AFT). We find that APO models are extremely hard to jailbreak even when the trigger is optimized directly on the model. On the other hand, while AFT models may appear safe on the surface, exhibiting refusals to a range of unsafe instructions, we show that they are highly susceptible to adversarial triggers. Lastly, we observe that most triggers optimized on AFT models also generalize to new unsafe instructions from five diverse domains, further emphasizing their vulnerability. Overall, our work highlights the need for more comprehensive safety evaluations for aligned language models.

Title: PuLID: Pure and Lightning ID Customization via Contrastive Alignment

• Authors: Zinan Guo, Yanze Wu, Zhuowei Chen, Lang Chen, Qian He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.16022
• Pdf URL: https://arxiv.org/pdf/2404.16022
• Copy Paste: [[2404.16022]] PuLID: Pure and Lightning ID Customization via Contrastive Alignment(https://arxiv.org/abs/2404.16022)
• Keywords: diffusion
• Abstract: We propose Pure and Lightning ID customization (PuLID), a novel tuning-free ID customization method for text-to-image generation. By incorporating a Lightning T2I branch with a standard diffusion one, PuLID introduces both contrastive alignment loss and accurate ID loss, minimizing disruption to the original model and ensuring high ID fidelity. Experiments show that PuLID achieves superior performance in both ID fidelity and editability. Another attractive property of PuLID is that the image elements (e.g., background, lighting, composition, and style) before and after the ID insertion are kept as consistent as possible. Codes and models will be available at https://github.com/ToTheBeginning/PuLID

Title: Editable Image Elements for Controllable Synthesis

• Authors: Jiteng Mu, Michaël Gharbi, Richard Zhang, Eli Shechtman, Nuno Vasconcelos, Xiaolong Wang, Taesung Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.16029
• Pdf URL: https://arxiv.org/pdf/2404.16029
• Copy Paste: [[2404.16029]] Editable Image Elements for Controllable Synthesis(https://arxiv.org/abs/2404.16029)
• Keywords: diffusion
• Abstract: Diffusion models have made significant advances in text-guided synthesis tasks. However, editing user-provided images remains challenging, as the high dimensional noise input space of diffusion models is not naturally suited for image inversion or spatial editing. In this work, we propose an image representation that promotes spatial editing of input images using a diffusion model. Concretely, we learn to encode an input into "image elements" that can faithfully reconstruct an input image. These elements can be intuitively edited by a user, and are decoded by a diffusion model into realistic images. We show the effectiveness of our representation on various image editing tasks, such as object resizing, rearrangement, dragging, de-occlusion, removal, variation, and image composition. Project page: https://jitengmu.github.io/Editable_Image_Elements/

Title: Studying Large Language Model Behaviors Under Realistic Knowledge Conflicts

• Authors: Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen, Seong Joon Oh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.16032
• Pdf URL: https://arxiv.org/pdf/2404.16032
• Copy Paste: [[2404.16032]] Studying Large Language Model Behaviors Under Realistic Knowledge Conflicts(https://arxiv.org/abs/2404.16032)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) mitigates many problems of fully parametric language models, such as temporal degradation, hallucinations, and lack of grounding. In RAG, the model's knowledge can be updated from documents provided in context. This leads to cases of conflict between the model's parametric knowledge and the contextual information, where the model may not always update its knowledge. Previous work studied knowledge conflicts by creating synthetic documents that contradict the model's correct parametric answers. We present a framework for studying knowledge conflicts in a realistic setup. We update incorrect parametric knowledge using real conflicting documents. This reflects how knowledge conflicts arise in practice. In this realistic scenario, we find that knowledge updates fail less often than previously reported. In cases where the models still fail to update their answers, we find a parametric bias: the incorrect parametric answer appearing in context makes the knowledge update likelier to fail. These results suggest that the factual parametric knowledge of LLMs can negatively influence their reading abilities and behaviors. Our code is available at https://github.com/kortukov/realistic_knowledge_conflicts/.

Title: Cantor: Inspiring Multimodal Chain-of-Thought of MLLM

• Authors: Timin Gao, Peixian Chen, Mengdan Zhang, Chaoyou Fu, Yunhang Shen, Yan Zhang, Shengchuan Zhang, Xiawu Zheng, Xing Sun, Liujuan Cao, Rongrong Ji
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16033
• Pdf URL: https://arxiv.org/pdf/2404.16033
• Copy Paste: [[2404.16033]] Cantor: Inspiring Multimodal Chain-of-Thought of MLLM(https://arxiv.org/abs/2404.16033)
• Keywords: large language model
• Abstract: With the advent of large language models(LLMs) enhanced by the chain-of-thought(CoT) methodology, visual reasoning problem is usually decomposed into manageable sub-tasks and tackled sequentially with various external tools. However, such a paradigm faces the challenge of the potential "determining hallucinations" in decision-making due to insufficient visual information and the limitation of low-level perception tools that fail to provide abstract summaries necessary for comprehensive reasoning. We argue that converging visual context acquisition and logical reasoning is pivotal for tackling visual reasoning tasks. This paper delves into the realm of multimodal CoT to solve intricate visual reasoning tasks with multimodal large language models(MLLMs) and their cognitive capability. To this end, we propose an innovative multimodal CoT framework, termed Cantor, characterized by a perception-decision architecture. Cantor first acts as a decision generator and integrates visual inputs to analyze the image and problem, ensuring a closer alignment with the actual context. Furthermore, Cantor leverages the advanced cognitive functions of MLLMs to perform as multifaceted experts for deriving higher-level information, enhancing the CoT generation process. Our extensive experiments demonstrate the efficacy of the proposed framework, showing significant improvements in multimodal CoT performance across two complex visual reasoning datasets, without necessitating fine-tuning or ground-truth rationales. Project Page: https://ggg0919.github.io/cantor/ .

Title: MaGGIe: Masked Guided Gradual Human Instance Matting

• Authors: Chuong Huynh, Seoung Wug Oh, Abhinav Shrivastava, Joon-Young Lee
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16035
• Pdf URL: https://arxiv.org/pdf/2404.16035
• Copy Paste: [[2404.16035]] MaGGIe: Masked Guided Gradual Human Instance Matting(https://arxiv.org/abs/2404.16035)
• Keywords: robust, transformer
• Abstract: Human matting is a foundation task in image and video processing, where human foreground pixels are extracted from the input. Prior works either improve the accuracy by additional guidance or improve the temporal consistency of a single instance across frames. We propose a new framework MaGGIe, Masked Guided Gradual Human Instance Matting, which predicts alpha mattes progressively for each human instances while maintaining the computational cost, precision, and consistency. Our method leverages modern architectures, including transformer attention and sparse convolution, to output all instance mattes simultaneously without exploding memory and latency. Although keeping constant inference costs in the multiple-instance scenario, our framework achieves robust and versatile performance on our proposed synthesized benchmarks. With the higher quality image and video matting benchmarks, the novel multi-instance synthesis approach from publicly available sources is introduced to increase the generalization of models in real-world scenarios.