2024-03-08

Title: Effect of Ambient-Intrinsic Dimension Gap on Adversarial Vulnerability

• Authors: Rajdeep Haldar, Yue Xing, Qifan Song
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.03967
• Pdf URL: https://arxiv.org/pdf/2403.03967
• Copy Paste: [[2403.03967]] Effect of Ambient-Intrinsic Dimension Gap on Adversarial Vulnerability(https://arxiv.org/abs/2403.03967)
• Keywords: attack
• Abstract: The existence of adversarial attacks on machine learning models imperceptible to a human is still quite a mystery from a theoretical perspective. In this work, we introduce two notions of adversarial attacks: natural or on-manifold attacks, which are perceptible by a human/oracle, and unnatural or off-manifold attacks, which are not. We argue that the existence of the off-manifold attacks is a natural consequence of the dimension gap between the intrinsic and ambient dimensions of the data. For 2-layer ReLU networks, we prove that even though the dimension gap does not affect generalization performance on samples drawn from the observed data space, it makes the clean-trained model more vulnerable to adversarial perturbations in the off-manifold direction of the data space. Our main results provide an explicit relationship between the $\ell_2,\ell_{\infty}$ attack strength of the on/off-manifold attack and the dimension gap.

Title: OpenVPN is Open to VPN Fingerprinting

• Authors: Diwen Xue, Reethika Ramesh, Arham Jain, Michalis Kallitsis, J. Alex Halderman, Jedidiah R. Crandall, Roya Ensafi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.03998
• Pdf URL: https://arxiv.org/pdf/2403.03998
• Copy Paste: [[2403.03998]] OpenVPN is Open to VPN Fingerprinting(https://arxiv.org/abs/2403.03998)
• Keywords: privacy, defense, attack
• Abstract: VPN adoption has seen steady growth over the past decade due to increased public awareness of privacy and surveillance threats. In response, certain governments are attempting to restrict VPN access by identifying connections using "dual use" DPI technology. To investigate the potential for VPN blocking, we develop mechanisms for accurately fingerprinting connections using OpenVPN, the most popular protocol for commercial VPN services. We identify three fingerprints based on protocol features such as byte pattern, packet size, and server response. Playing the role of an attacker who controls the network, we design a two-phase framework that performs passive fingerprinting and active probing in sequence. We evaluate our framework in partnership with a million-user ISP and find that we identify over 85% of OpenVPN flows with only negligible false positives, suggesting that OpenVPN-based services can be effectively blocked with little collateral damage. Although some commercial VPNs implement countermeasures to avoid detection, our framework successfully identified connections to 34 out of 41 "obfuscated" VPN configurations. We discuss the implications of the VPN fingerprintability for different threat models and propose short-term defenses. In the longer term, we urge commercial VPN providers to be more transparent about their obfuscation approaches and to adopt more principled detection countermeasures, such as those developed in censorship circumvention research.

Title: Temporal Cross-Attention for Dynamic Embedding and Tokenization of Multimodal Electronic Health Records

• Authors: Yingbo Ma, Suraj Kolla, Dhruv Kaliraman, Victoria Nolan, Zhenhong Hu, Ziyuan Guan, Yuanfang Ren, Brooke Armfield, Tezcan Ozrazgat-Baslanti, Tyler J. Loftus, Parisa Rashidi, Azra Bihorac, Benjamin Shickel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04012
• Pdf URL: https://arxiv.org/pdf/2403.04012
• Copy Paste: [[2403.04012]] Temporal Cross-Attention for Dynamic Embedding and Tokenization of Multimodal Electronic Health Records(https://arxiv.org/abs/2403.04012)
• Keywords: transformer
• Abstract: The breadth, scale, and temporal granularity of modern electronic health records (EHR) systems offers great potential for estimating personalized and contextual patient health trajectories using sequential deep learning. However, learning useful representations of EHR data is challenging due to its high dimensionality, sparsity, multimodality, irregular and variable-specific recording frequency, and timestamp duplication when multiple measurements are recorded simultaneously. Although recent efforts to fuse structured EHR and unstructured clinical notes suggest the potential for more accurate prediction of clinical outcomes, less focus has been placed on EHR embedding approaches that directly address temporal EHR challenges by learning time-aware representations from multimodal patient time series. In this paper, we introduce a dynamic embedding and tokenization framework for precise representation of multimodal clinical time series that combines novel methods for encoding time and sequential position with temporal cross-attention. Our embedding and tokenization framework, when integrated into a multitask transformer classifier with sliding window attention, outperformed baseline approaches on the exemplar task of predicting the occurrence of nine postoperative complications of more than 120,000 major inpatient surgeries using multimodal data from three hospitals and two academic health centers in the United States.

Title: Can Large Language Models do Analytical Reasoning?

• Authors: Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, Dong Yu, Fei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04031
• Pdf URL: https://arxiv.org/pdf/2403.04031
• Copy Paste: [[2403.04031]] Can Large Language Models do Analytical Reasoning?(https://arxiv.org/abs/2403.04031)
• Keywords: large language model
• Abstract: This paper explores the cutting-edge Large Language Model with analytical reasoning on sports. Our analytical reasoning embodies the tasks of letting large language models count how many points each team scores in a quarter in the NBA and NFL games. Our major discoveries are in two folds. Firstly, we find among all the models we employed, GPT-4 stands out in effectiveness, followed by Claude-2.1, with GPT-3.5, Gemini-Pro, and Llama-2-70b lagging behind. Specifically, we compare three different prompting techniques and a divide-and-conquer approach, we find that the latter was the most effective. Our divide-and-conquer approach breaks down play-by-play data into smaller, more manageable segments, solves each piece individually, and then aggregates them together. Besides the divide-and-conquer approach, we also explore the Chain of Thought (CoT) strategy, which markedly improves outcomes for certain models, notably GPT-4 and Claude-2.1, with their accuracy rates increasing significantly. However, the CoT strategy has negligible or even detrimental effects on the performance of other models like GPT-3.5 and Gemini-Pro. Secondly, to our surprise, we observe that most models, including GPT-4, struggle to accurately count the total scores for NBA quarters despite showing strong performance in counting NFL quarter scores. This leads us to further investigate the factors that impact the complexity of analytical reasoning tasks with extensive experiments, through which we conclude that task complexity depends on the length of context, the information density, and the presence of related information. Our research provides valuable insights into the complexity of analytical reasoning tasks and potential directions for developing future large language models.

Title: Unsupervised Contrastive Learning for Robust RF Device Fingerprinting Under Time-Domain Shift

• Authors: Jun Chen, Weng-Keen Wong, Bechir Hamdaoui
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.04036
• Pdf URL: https://arxiv.org/pdf/2403.04036
• Copy Paste: [[2403.04036]] Unsupervised Contrastive Learning for Robust RF Device Fingerprinting Under Time-Domain Shift(https://arxiv.org/abs/2403.04036)
• Keywords: robust
• Abstract: Radio Frequency (RF) device fingerprinting has been recognized as a potential technology for enabling automated wireless device identification and classification. However, it faces a key challenge due to the domain shift that could arise from variations in the channel conditions and environmental settings, potentially degrading the accuracy of RF-based device classification when testing and training data is collected in different domains. This paper introduces a novel solution that leverages contrastive learning to mitigate this domain shift problem. Contrastive learning, a state-of-the-art self-supervised learning approach from deep learning, learns a distance metric such that positive pairs are closer (i.e. more similar) in the learned metric space than negative pairs. When applied to RF fingerprinting, our model treats RF signals from the same transmission as positive pairs and those from different transmissions as negative pairs. Through experiments on wireless and wired RF datasets collected over several days, we demonstrate that our contrastive learning approach captures domain-invariant features, diminishing the effects of domain-specific variations. Our results show large and consistent improvements in accuracy (10.8\% to 27.8\%) over baseline models, thus underscoring the effectiveness of contrastive learning in improving device classification under domain shift.

Title: OCD-FL: A Novel Communication-Efficient Peer Selection-based Decentralized Federated Learning

• Authors: Nizar Masmoudi, Wael Jaafar
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.04037
• Pdf URL: https://arxiv.org/pdf/2403.04037
• Copy Paste: [[2403.04037]] OCD-FL: A Novel Communication-Efficient Peer Selection-based Decentralized Federated Learning(https://arxiv.org/abs/2403.04037)
• Keywords: federate
• Abstract: The conjunction of edge intelligence and the ever-growing Internet-of-Things (IoT) network heralds a new era of collaborative machine learning, with federated learning (FL) emerging as the most prominent paradigm. With the growing interest in these learning schemes, researchers started addressing some of their most fundamental limitations. Indeed, conventional FL with a central aggregator presents a single point of failure and a network bottleneck. To bypass this issue, decentralized FL where nodes collaborate in a peer-to-peer network has been proposed. Despite the latter's efficiency, communication costs and data heterogeneity remain key challenges in decentralized FL. In this context, we propose a novel scheme, called opportunistic communication-efficient decentralized federated learning, a.k.a., OCD-FL, consisting of a systematic FL peer selection for collaboration, aiming to achieve maximum FL knowledge gain while reducing energy consumption. Experimental results demonstrate the capability of OCD-FL to achieve similar or better performances than the fully collaborative FL, while significantly reducing consumed energy by at least 30% and up to 80%.

Title: Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations

• Authors: Xiaolin Sun, Zizhan Zheng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04050
• Pdf URL: https://arxiv.org/pdf/2403.04050
• Copy Paste: [[2403.04050]] Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations(https://arxiv.org/abs/2403.04050)
• Keywords: protect, attack, robust, diffusion
• Abstract: Reinforcement learning (RL) has achieved phenomenal success in various domains. However, its data-driven nature also introduces new vulnerabilities that can be exploited by malicious opponents. Recent work shows that a well-trained RL agent can be easily manipulated by strategically perturbing its state observations at the test stage. Existing solutions either introduce a regularization term to improve the smoothness of the trained policy against perturbations or alternatively train the agent's policy and the attacker's policy. However, the former does not provide sufficient protection against strong attacks, while the latter is computationally prohibitive for large environments. In this work, we propose a new robust RL algorithm for deriving a pessimistic policy to safeguard against an agent's uncertainty about true states. This approach is further enhanced with belief state inference and diffusion-based state purification to reduce uncertainty. Empirical results show that our approach obtains superb performance under strong attacks and has a comparable training overhead with regularization-based methods. Our code is available at https://github.com/SliencerX/Belief-enriched-robust-Q-learning.

Title: Improving Adversarial Training using Vulnerability-Aware Perturbation Budget

• Authors: Olukorede Fakorede, Modeste Atsague, Jin Tian
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04070
• Pdf URL: https://arxiv.org/pdf/2403.04070
• Copy Paste: [[2403.04070]] Improving Adversarial Training using Vulnerability-Aware Perturbation Budget(https://arxiv.org/abs/2403.04070)
• Keywords: attack, robust
• Abstract: Adversarial Training (AT) effectively improves the robustness of Deep Neural Networks (DNNs) to adversarial attacks. Generally, AT involves training DNN models with adversarial examples obtained within a pre-defined, fixed perturbation bound. Notably, individual natural examples from which these adversarial examples are crafted exhibit varying degrees of intrinsic vulnerabilities, and as such, crafting adversarial examples with fixed perturbation radius for all instances may not sufficiently unleash the potency of AT. Motivated by this observation, we propose two simple, computationally cheap vulnerability-aware reweighting functions for assigning perturbation bounds to adversarial examples used for AT, named Margin-Weighted Perturbation Budget (MWPB) and Standard-Deviation-Weighted Perturbation Budget (SDWPB). The proposed methods assign perturbation radii to individual adversarial samples based on the vulnerability of their corresponding natural examples. Experimental results show that the proposed methods yield genuine improvements in the robustness of AT algorithms against various adversarial attacks.

Title: Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection

• Authors: Jianfeng He, Hang Su, Jason Cai, Igor Shalyminov, Hwanjun Song, Saab Mansour
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04073
• Pdf URL: https://arxiv.org/pdf/2403.04073
• Copy Paste: [[2403.04073]] Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection(https://arxiv.org/abs/2403.04073)
• Keywords: generative
• Abstract: Semi-supervised dialogue summarization (SSDS) leverages model-generated summaries to reduce reliance on human-labeled data and improve the performance of summarization models. While addressing label noise, previous works on semi-supervised learning primarily focus on natural language understanding tasks, assuming each sample has a unique label. However, these methods are not directly applicable to SSDS, as it is a generative task, and each dialogue can be summarized in different ways. In this work, we propose a novel scoring approach, SiCF, which encapsulates three primary dimensions of summarization model quality: Semantic invariance (indicative of model confidence), Coverage (factual recall), and Faithfulness (factual precision). Using the SiCF score, we select unlabeled dialogues with high-quality generated summaries to train summarization models. Comprehensive experiments on three public datasets demonstrate the effectiveness of SiCF scores in uncertainty estimation and semi-supervised learning for dialogue summarization tasks. Our code is available at \url{https://github.com/amazon-science/summarization-sicf-score}.

Title: Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysis

• Authors: Abdelrahman Abdallah, Daniel Eberharter, Zoe Pfister, Adam Jatowt
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04080
• Pdf URL: https://arxiv.org/pdf/2403.04080
• Copy Paste: [[2403.04080]] Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysis(https://arxiv.org/abs/2403.04080)
• Keywords: transformer
• Abstract: This paper presents a comprehensive survey of research works on the topic of form understanding in the context of scanned documents. We delve into recent advancements and breakthroughs in the field, highlighting the significance of language models and transformers in solving this challenging task. Our research methodology involves an in-depth analysis of popular documents and forms of understanding of trends over the last decade, enabling us to offer valuable insights into the evolution of this domain. Focusing on cutting-edge models, we showcase how transformers have propelled the field forward, revolutionizing form-understanding techniques. Our exploration includes an extensive examination of state-of-the-art language models designed to effectively tackle the complexities of noisy scanned documents. Furthermore, we present an overview of the latest and most relevant datasets, which serve as essential benchmarks for evaluating the performance of selected models. By comparing and contrasting the capabilities of these models, we aim to provide researchers and practitioners with useful guidance in choosing the most suitable solutions for their specific form understanding tasks.

Title: Many-Objective Multi-Solution Transport

• Authors: Ziyue Li, Tian Li, Virginia Smith, Jeff Bilmes, Tianyi Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04099
• Pdf URL: https://arxiv.org/pdf/2403.04099
• Copy Paste: [[2403.04099]] Many-Objective Multi-Solution Transport(https://arxiv.org/abs/2403.04099)
• Keywords: federate
• Abstract: Optimizing the performance of many objectives (instantiated by tasks or clients) jointly with a few Pareto stationary solutions (models) is critical in machine learning. However, previous multi-objective optimization methods often focus on a few number of objectives and cannot scale to many objectives that outnumber the solutions, leading to either subpar performance or ignored objectives. We introduce Many-objective multi-solution Transport (MosT), a framework that finds multiple diverse solutions in the Pareto front of many objectives. Our insight is to seek multiple solutions, each performing as a domain expert and focusing on a specific subset of objectives while collectively covering all of them. MosT formulates the problem as a bi-level optimization of weighted objectives for each solution, where the weights are defined by an optimal transport between the objectives and solutions. Our algorithm ensures convergence to Pareto stationary solutions for complementary subsets of objectives. On a range of applications in federated learning, multi-task learning, and mixture-of-prompt learning for LLMs, MosT distinctly outperforms strong baselines, delivering high-quality, diverse solutions that profile the entire Pareto frontier, thus ensuring balanced trade-offs across many objectives.

Title: ZTRAN: Prototyping Zero Trust Security xApps for Open Radio Access Network Deployments

• Authors: Aly S. Abdalla, Joshua Moore, Nisha Adhikari, Vuk Marojevic
• Subjects: cs.CR, cs.ET, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.04113
• Pdf URL: https://arxiv.org/pdf/2403.04113
• Copy Paste: [[2403.04113]] ZTRAN: Prototyping Zero Trust Security xApps for Open Radio Access Network Deployments(https://arxiv.org/abs/2403.04113)
• Keywords: secure, security
• Abstract: The open radio access network (O-RAN) offers new degrees of freedom for building and operating advanced cellular networks. Emphasizing on RAN disaggregation, open interfaces, multi-vendor support, and RAN intelligent controllers (RICs), O-RAN facilitates adaptation to new applications and technology trends. Yet, this architecture introduces new security challenges. This paper proposes leveraging zero trust principles for O-RAN security. We introduce zero trust RAN (ZTRAN), which embeds service authentication, intrusion detection, and secure slicing subsystems that are encapsulated as xApps. We implement ZTRAN on the open artificial intelligence cellular (OAIC) research platform and demonstrate its feasibility and effectiveness in terms of legitimate user throughput and latency figures. Our experimental analysis illustrates how ZTRAN's intrusion detection and secure slicing microservices operate effectively and in concert as part of O-RAN Alliance's containerized near-real time RIC. Research directions include exploring machine learning and additional threat intelligence feeds for improving the performance and extending the scope of ZTRAN.

Title: A data-centric approach to class-specific bias in image data augmentation

• Authors: Athanasios Angelakis, Andrey Rass
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04120
• Pdf URL: https://arxiv.org/pdf/2403.04120
• Copy Paste: [[2403.04120]] A data-centric approach to class-specific bias in image data augmentation(https://arxiv.org/abs/2403.04120)
• Keywords: robust, transformer
• Abstract: Data augmentation (DA) enhances model generalization in computer vision but may introduce biases, impacting class accuracy unevenly. Our study extends this inquiry, examining DA's class-specific bias across various datasets, including those distinct from ImageNet, through random cropping. We evaluated this phenomenon with ResNet50, EfficientNetV2S, and SWIN ViT, discovering that while residual models showed similar bias effects, Vision Transformers exhibited greater robustness or altered dynamics. This suggests a nuanced approach to model selection, emphasizing bias mitigation. We also refined a "data augmentation robustness scouting" method to manage DA-induced biases more efficiently, reducing computational demands significantly (training 112 models instead of 1860; a reduction of factor 16.2) while still capturing essential bias trends.

Title: Scalable and Robust Transformer Decoders for Interpretable Image Classification with Foundation Models

• Authors: Evelyn Mannix, Howard Bondell
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04125
• Pdf URL: https://arxiv.org/pdf/2403.04125
• Copy Paste: [[2403.04125]] Scalable and Robust Transformer Decoders for Interpretable Image Classification with Foundation Models(https://arxiv.org/abs/2403.04125)
• Keywords: robust, transformer, segmentation
• Abstract: Interpretable computer vision models can produce transparent predictions, where the features of an image are compared with prototypes from a training dataset and the similarity between them forms a basis for classification. Nevertheless these methods are computationally expensive to train, introduce additional complexity and may require domain knowledge to adapt hyper-parameters to a new dataset. Inspired by developments in object detection, segmentation and large-scale self-supervised foundation vision models, we introduce Component Features (ComFe), a novel explainable-by-design image classification approach using a transformer-decoder head and hierarchical mixture-modelling. With only global image labels and no segmentation or part annotations, ComFe can identify consistent image components, such as the head, body, wings and tail of a bird, and the image background, and determine which of these features are informative in making a prediction. We demonstrate that ComFe obtains higher accuracy compared to previous interpretable models across a range of fine-grained vision benchmarks, without the need to individually tune hyper-parameters for each dataset. We also show that ComFe outperforms a non-interpretable linear head across a range of datasets, including ImageNet, and improves performance on generalisation and robustness benchmarks.

Title: An Explainable AI Framework for Artificial Intelligence of Medical Things

• Authors: Al Amin, Kamrul Hasan, Saleh Zein-Sabatto, Deo Chimba, Imtiaz Ahmed, Tariqul Islam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04130
• Pdf URL: https://arxiv.org/pdf/2403.04130
• Copy Paste: [[2403.04130]] An Explainable AI Framework for Artificial Intelligence of Medical Things(https://arxiv.org/abs/2403.04130)
• Keywords: robust
• Abstract: The healthcare industry has been revolutionized by the convergence of Artificial Intelligence of Medical Things (AIoMT), allowing advanced data-driven solutions to improve healthcare systems. With the increasing complexity of Artificial Intelligence (AI) models, the need for Explainable Artificial Intelligence (XAI) techniques become paramount, particularly in the medical domain, where transparent and interpretable decision-making becomes crucial. Therefore, in this work, we leverage a custom XAI framework, incorporating techniques such as Local Interpretable Model-Agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), and Gradient-weighted Class Activation Mapping (Grad-Cam), explicitly designed for the domain of AIoMT. The proposed framework enhances the effectiveness of strategic healthcare methods and aims to instill trust and promote understanding in AI-driven medical applications. Moreover, we utilize a majority voting technique that aggregates predictions from multiple convolutional neural networks (CNNs) and leverages their collective intelligence to make robust and accurate decisions in the healthcare system. Building upon this decision-making process, we apply the XAI framework to brain tumor detection as a use case demonstrating accurate and transparent diagnosis. Evaluation results underscore the exceptional performance of the XAI framework, achieving high precision, recall, and F1 scores with a training accuracy of 99% and a validation accuracy of 98%. Combining advanced XAI techniques with ensemble-based deep-learning (DL) methodologies allows for precise and reliable brain tumor diagnoses as an application of AIoMT.

Title: FL-GUARD: A Holistic Framework for Run-Time Detection and Recovery of Negative Federated Learning

• Authors: Hong Lin, Lidan Shou, Ke Chen, Gang Chen, Sai Wu
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.04146
• Pdf URL: https://arxiv.org/pdf/2403.04146
• Copy Paste: [[2403.04146]] FL-GUARD: A Holistic Framework for Run-Time Detection and Recovery of Negative Federated Learning(https://arxiv.org/abs/2403.04146)
• Keywords: privacy, robust, federate
• Abstract: Federated learning (FL) is a promising approach for learning a model from data distributed on massive clients without exposing data privacy. It works effectively in the ideal federation where clients share homogeneous data distribution and learning behavior. However, FL may fail to function appropriately when the federation is not ideal, amid an unhealthy state called Negative Federated Learning (NFL), in which most clients gain no benefit from participating in FL. Many studies have tried to address NFL. However, their solutions either (1) predetermine to prevent NFL in the entire learning life-cycle or (2) tackle NFL in the aftermath of numerous learning rounds. Thus, they either (1) indiscriminately incur extra costs even if FL can perform well without such costs or (2) waste numerous learning rounds. Additionally, none of the previous work takes into account the clients who may be unwilling/unable to follow the proposed NFL solutions when using those solutions to upgrade an FL system in use. This paper introduces FL-GUARD, a holistic framework that can be employed on any FL system for tackling NFL in a run-time paradigm. That is, to dynamically detect NFL at the early stage (tens of rounds) of learning and then to activate recovery measures when necessary. Specifically, we devise a cost-effective NFL detection mechanism, which relies on an estimation of performance gain on clients. Only when NFL is detected, we activate the NFL recovery process, in which each client learns in parallel an adapted model when training the global model. Extensive experiment results confirm the effectiveness of FL-GUARD in detecting NFL and recovering from NFL to a healthy learning state. We also show that FL-GUARD is compatible with previous NFL solutions and robust against clients unwilling/unable to take any recovery measures.

Title: MAP: MAsk-Pruning for Source-Free Model Intellectual Property Protection

• Authors: Boyang Peng, Sanqing Qu, Yong Wu, Tianpei Zou, Lianghua He, Alois Knoll, Guang Chen, changjun jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04149
• Pdf URL: https://arxiv.org/pdf/2403.04149
• Copy Paste: [[2403.04149]] MAP: MAsk-Pruning for Source-Free Model Intellectual Property Protection(https://arxiv.org/abs/2403.04149)
• Keywords: protect, data-free
• Abstract: Deep learning has achieved remarkable progress in various applications, heightening the importance of safeguarding the intellectual property (IP) of well-trained models. It entails not only authorizing usage but also ensuring the deployment of models in authorized data domains, i.e., making models exclusive to certain target domains. Previous methods necessitate concurrent access to source training data and target unauthorized data when performing IP protection, making them risky and inefficient for decentralized private data. In this paper, we target a practical setting where only a well-trained source model is available and investigate how we can realize IP protection. To achieve this, we propose a novel MAsk Pruning (MAP) framework. MAP stems from an intuitive hypothesis, i.e., there are target-related parameters in a well-trained model, locating and pruning them is the key to IP protection. Technically, MAP freezes the source model and learns a target-specific binary mask to prevent unauthorized data usage while minimizing performance degradation on authorized data. Moreover, we introduce a new metric aimed at achieving a better balance between source and target performance degradation. To verify the effectiveness and versatility, we have evaluated MAP in a variety of scenarios, including vanilla source-available, practical source-free, and challenging data-free. Extensive experiments indicate that MAP yields new state-of-the-art performance.

Title: Stabilizing Policy Gradients for Stochastic Differential Equations via Consistency with Perturbation Process

• Authors: Xiangxin Zhou, Liang Wang, Yichi Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04154
• Pdf URL: https://arxiv.org/pdf/2403.04154
• Copy Paste: [[2403.04154]] Stabilizing Policy Gradients for Stochastic Differential Equations via Consistency with Perturbation Process(https://arxiv.org/abs/2403.04154)
• Keywords: generative
• Abstract: Considering generating samples with high rewards, we focus on optimizing deep neural networks parameterized stochastic differential equations (SDEs), the advanced generative models with high expressiveness, with policy gradient, the leading algorithm in reinforcement learning. Nevertheless, when applying policy gradients to SDEs, since the policy gradient is estimated on a finite set of trajectories, it can be ill-defined, and the policy behavior in data-scarce regions may be uncontrolled. This challenge compromises the stability of policy gradients and negatively impacts sample complexity. To address these issues, we propose constraining the SDE to be consistent with its associated perturbation process. Since the perturbation process covers the entire space and is easy to sample, we can mitigate the aforementioned problems. Our framework offers a general approach allowing for a versatile selection of policy gradient methods to effectively and efficiently train SDEs. We evaluate our algorithm on the task of structure-based drug design and optimize the binding affinity of generated ligand molecules. Our method achieves the best Vina score -9.07 on the CrossDocked2020 dataset.

Title: Noisy Spiking Actor Network for Exploration

• Authors: Ding Chen, Peixi Peng, Tiejun Huang, Yonghong Tian
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2403.04162
• Pdf URL: https://arxiv.org/pdf/2403.04162
• Copy Paste: [[2403.04162]] Noisy Spiking Actor Network for Exploration(https://arxiv.org/abs/2403.04162)
• Keywords: robust
• Abstract: As a general method for exploration in deep reinforcement learning (RL), NoisyNet can produce problem-specific exploration strategies. Spiking neural networks (SNNs), due to their binary firing mechanism, have strong robustness to noise, making it difficult to realize efficient exploration with local disturbances. To solve this exploration problem, we propose a noisy spiking actor network (NoisySAN) that introduces time-correlated noise during charging and transmission. Moreover, a noise reduction method is proposed to find a stable policy for the agent. Extensive experimental results demonstrate that our method outperforms the state-of-the-art performance on a wide range of continuous control tasks from OpenAI gym.

Title: ProMISe: Promptable Medical Image Segmentation using SAM

• Authors: Jinfeng Wang, Sifan Song, Xinkun Wang, Yiyi Wang, Yiyi Miao, Jionglong Su, S. Kevin Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04164
• Pdf URL: https://arxiv.org/pdf/2403.04164
• Copy Paste: [[2403.04164]] ProMISe: Promptable Medical Image Segmentation using SAM(https://arxiv.org/abs/2403.04164)
• Keywords: segmentation
• Abstract: With the proposal of the Segment Anything Model (SAM), fine-tuning SAM for medical image segmentation (MIS) has become popular. However, due to the large size of the SAM model and the significant domain gap between natural and medical images, fine-tuning-based strategies are costly with potential risk of instability, feature damage and catastrophic forgetting. Furthermore, some methods of transferring SAM to a domain-specific MIS through fine-tuning strategies disable the model's prompting capability, severely limiting its utilization scenarios. In this paper, we propose an Auto-Prompting Module (APM), which provides SAM-based foundation model with Euclidean adaptive prompts in the target domain. Our experiments demonstrate that such adaptive prompts significantly improve SAM's non-fine-tuned performance in MIS. In addition, we propose a novel non-invasive method called Incremental Pattern Shifting (IPS) to adapt SAM to specific medical domains. Experimental results show that the IPS enables SAM to achieve state-of-the-art or competitive performance in MIS without the need for fine-tuning. By coupling these two methods, we propose ProMISe, an end-to-end non-fine-tuned framework for Promptable Medical Image Segmentation. Our experiments demonstrate that both using our methods individually or in combination achieves satisfactory performance in low-cost pattern shifting, with all of SAM's parameters frozen.

Title: SDPL: Shifting-Dense Partition Learning for UAV-View Geo-Localization

• Authors: Quan Chen, Tingyu Wang, Zihao Yang, Haoran Li, Rongfeng Lu, Yaoqi Sun, Bolun Zheng, Chenggang Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04172
• Pdf URL: https://arxiv.org/pdf/2403.04172
• Copy Paste: [[2403.04172]] SDPL: Shifting-Dense Partition Learning for UAV-View Geo-Localization(https://arxiv.org/abs/2403.04172)
• Keywords: robust, segmentation
• Abstract: Cross-view geo-localization aims to match images of the same target from different platforms, e.g., drone and satellite. It is a challenging task due to the changing both appearance of targets and environmental content from different views. Existing methods mainly focus on digging more comprehensive information through feature maps segmentation, while inevitably destroy the image structure and are sensitive to the shifting and scale of the target in the query. To address the above issues, we introduce a simple yet effective part-based representation learning, called shifting-dense partition learning (SDPL). Specifically, we propose the dense partition strategy (DPS), which divides the image into multiple parts to explore contextual-information while explicitly maintain the global structure. To handle scenarios with non-centered targets, we further propose the shifting-fusion strategy, which generates multiple sets of parts in parallel based on various segmentation centers and then adaptively fuses all features to select the best partitions. Extensive experiments show that our SDPL is robust to position shifting and scale variations, and achieves competitive performance on two prevailing benchmarks, i.e., University-1652 and SUES-200.

Title: Image Coding for Machines with Edge Information Learning Using Segment Anything

• Authors: Takahiro Shindo, Kein Yamada, Taiju Watanabe, Hiroshi Watanabe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04173
• Pdf URL: https://arxiv.org/pdf/2403.04173
• Copy Paste: [[2403.04173]] Image Coding for Machines with Edge Information Learning Using Segment Anything(https://arxiv.org/abs/2403.04173)
• Keywords: privacy, protect, robust
• Abstract: Image Coding for Machines (ICM) is an image compression technique for image recognition. This technique is essential due to the growing demand for image recognition AI. In this paper, we propose a method for ICM that focuses on encoding and decoding only the edge information of object parts in an image, which we call SA-ICM. This is an Learned Image Compression (LIC) model trained using edge information created by Segment Anything. Our method can be used for image recognition models with various tasks. SA-ICM is also robust to changes in input data, making it effective for a variety of use cases. Additionally, our method provides benefits from a privacy point of view, as it removes human facial information on the encoder's side, thus protecting one's privacy. Furthermore, this LIC model training method can be used to train Neural Representations for Videos (NeRV), which is a video compression model. By training NeRV using edge information created by Segment Anything, it is possible to create a NeRV that is effective for image recognition (SA-NeRV). Experimental results confirm the advantages of SA-ICM, presenting the best performance in image compression for image recognition. We also show that SA-NeRV is superior to ordinary NeRV in video compression for machines.

Title: RATSF: Empowering Customer Service Volume Management through Retrieval-Augmented Time-Series Forecasting

• Authors: Tianfeng Wang, Gaojie Cui
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.04180
• Pdf URL: https://arxiv.org/pdf/2403.04180
• Copy Paste: [[2403.04180]] RATSF: Empowering Customer Service Volume Management through Retrieval-Augmented Time-Series Forecasting(https://arxiv.org/abs/2403.04180)
• Keywords: transformer
• Abstract: An efficient customer service management system hinges on precise forecasting of service volume. In this scenario, where data non-stationarity is pronounced, successful forecasting heavily relies on identifying and leveraging similar historical data rather than merely summarizing periodic patterns. Existing models based on RNN or Transformer architectures often struggle with this flexible and effective utilization. To address this challenge, we propose an efficient and adaptable cross-attention module termed RACA, which effectively leverages historical segments in forecasting task, and we devised a precise representation scheme for querying historical sequences, coupled with the design of a knowledge repository. These critical components collectively form our Retrieval-Augmented Temporal Sequence Forecasting framework (RATSF). RATSF not only significantly enhances performance in the context of Fliggy hotel service volume forecasting but, more crucially, can be seamlessly integrated into other Transformer-based time-series forecasting models across various application scenarios. Extensive experimentation has validated the effectiveness and generalizability of this system design across multiple diverse contexts.

Title: Metric-aware LLM inference

• Authors: Michal Lukasik, Harikrishna Narasimhan, Aditya Krishna Menon, Felix Yu, Sanjiv Kumar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04182
• Pdf URL: https://arxiv.org/pdf/2403.04182
• Copy Paste: [[2403.04182]] Metric-aware LLM inference(https://arxiv.org/abs/2403.04182)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated strong results on a range of NLP tasks. Typically, outputs are obtained via autoregressive sampling from the LLM's underlying distribution. We show that this inference strategy can be suboptimal for a range of tasks and associated evaluation metrics. As a remedy, we propose metric aware LLM inference: a decision theoretic approach optimizing for custom metrics at inference time. We report improvements over baselines on academic benchmarks and publicly available models.

Title: Generative AI for Synthetic Data Generation: Methods, Challenges and the Future

• Authors: Xu Guo, Yiqiang Chen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04190
• Pdf URL: https://arxiv.org/pdf/2403.04190
• Copy Paste: [[2403.04190]] Generative AI for Synthetic Data Generation: Methods, Challenges and the Future(https://arxiv.org/abs/2403.04190)
• Keywords: generative, large language model
• Abstract: The recent surge in research focused on generating synthetic data from large language models (LLMs), especially for scenarios with limited data availability, marks a notable shift in Generative Artificial Intelligence (AI). Their ability to perform comparably to real-world data positions this approach as a compelling solution to low-resource challenges. This paper delves into advanced technologies that leverage these gigantic LLMs for the generation of task-specific training data. We outline methodologies, evaluation techniques, and practical applications, discuss the current limitations, and suggest potential pathways for future research.

Title: VAEMax: Open-Set Intrusion Detection based on OpenMax and Variational Autoencoder

• Authors: Zhiyin Qiu, Ding Zhou, Yahui Zhai, Bo Liu, Lei He, Jiuxin Cao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.04193
• Pdf URL: https://arxiv.org/pdf/2403.04193
• Copy Paste: [[2403.04193]] VAEMax: Open-Set Intrusion Detection based on OpenMax and Variational Autoencoder(https://arxiv.org/abs/2403.04193)
• Keywords: attack
• Abstract: Promptly discovering unknown network attacks is critical for reducing the risk of major loss imposed on system or equipment. This paper aims to develop an open-set intrusion detection model to classify known attacks as well as inferring unknown ones. To achieve this, we employ OpenMax and variational autoencoder to propose a dual detection model, VAEMax. First, we extract flow payload feature based on one-dimensional convolutional neural network. Then, the OpenMax is used to classify flows, during which some unknown attacks can be detected, while the rest are misclassified into a certain class of known flows. Finally, use VAE to perform secondary detection on each class of flows, and determine whether the flow is an unknown attack based on the reconstruction loss. Experiments performed on dataset CIC-IDS2017 and CSE-CIC-IDS2018 show our approach is better than baseline models and can be effectively applied to realistic network environments.

Title: SAM-PD: How Far Can SAM Take Us in Tracking and Segmenting Anything in Videos by Prompt Denoising

• Authors: Tao Zhou, Wenhan Luo, Qi Ye, Zhiguo Shi, Jiming Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04194
• Pdf URL: https://arxiv.org/pdf/2403.04194
• Copy Paste: [[2403.04194]] SAM-PD: How Far Can SAM Take Us in Tracking and Segmenting Anything in Videos by Prompt Denoising(https://arxiv.org/abs/2403.04194)
• Keywords: robust, segmentation
• Abstract: Recently, promptable segmentation models, such as the Segment Anything Model (SAM), have demonstrated robust zero-shot generalization capabilities on static images. These promptable models exhibit denoising abilities for imprecise prompt inputs, such as imprecise bounding boxes. In this paper, we explore the potential of applying SAM to track and segment objects in videos where we recognize the tracking task as a prompt denoising task. Specifically, we iteratively propagate the bounding box of each object's mask in the preceding frame as the prompt for the next frame. Furthermore, to enhance SAM's denoising capability against position and size variations, we propose a multi-prompt strategy where we provide multiple jittered and scaled box prompts for each object and preserve the mask prediction with the highest semantic similarity to the template mask. We also introduce a point-based refinement stage to handle occlusions and reduce cumulative errors. Without involving tracking modules, our approach demonstrates comparable performance in video object/instance segmentation tasks on three datasets: DAVIS2017, YouTubeVOS2018, and UVO, serving as a concise baseline and endowing SAM-based downstream applications with tracking capabilities.

Title: Fill-and-Spill: Deep Reinforcement Learning Policy Gradient Methods for Reservoir Operation Decision and Control

• Authors: Sadegh Sadeghi Tabas, Vidya Samadi
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2403.04195
• Pdf URL: https://arxiv.org/pdf/2403.04195
• Copy Paste: [[2403.04195]] Fill-and-Spill: Deep Reinforcement Learning Policy Gradient Methods for Reservoir Operation Decision and Control(https://arxiv.org/abs/2403.04195)
• Keywords: robust
• Abstract: Changes in demand, various hydrological inputs, and environmental stressors are among the issues that water managers and policymakers face on a regular basis. These concerns have sparked interest in applying different techniques to determine reservoir operation policy decisions. As the resolution of the analysis increases, it becomes more difficult to effectively represent a real-world system using traditional methods such as Dynamic Programming (DP) and Stochastic Dynamic Programming (SDP) for determining the best reservoir operation policy. One of the challenges is the "curse of dimensionality," which means the number of samples needed to estimate an arbitrary function with a given level of accuracy grows exponentially with respect to the number of input variables (i.e., dimensionality) of the function. Deep Reinforcement Learning (DRL) is an intelligent approach to overcome the curses of stochastic optimization problems for reservoir operation policy decisions. To our knowledge, this study is the first attempt that examine various novel DRL continuous-action policy gradient methods (PGMs), including Deep Deterministic Policy Gradients (DDPG), Twin Delayed DDPG (TD3), and two different versions of Soft Actor-Critic (SAC18 and SAC19) for optimizing reservoir operation policy. In this study, multiple DRL techniques were implemented in order to find the optimal operation policy of Folsom Reservoir in California, USA. The reservoir system supplies agricultural, municipal, hydropower, and environmental flow demands and flood control operations to the City of Sacramento. Analysis suggests that the TD3 and SAC are robust to meet the Folsom Reservoir's demands and optimize reservoir operation policies.

Title: Large Language Models are In-Context Molecule Learners

• Authors: Jiatong Li, Wei Liu, Zhihao Ding, Wenqi Fan, Yuqiang Li, Qing Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04197
• Pdf URL: https://arxiv.org/pdf/2403.04197
• Copy Paste: [[2403.04197]] Large Language Models are In-Context Molecule Learners(https://arxiv.org/abs/2403.04197)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional performance in biochemical tasks, especially the molecule caption translation task, which aims to bridge the gap between molecules and natural language texts. However, previous methods in adapting LLMs to the molecule-caption translation task required extra domain-specific pre-training stages, suffered weak alignment between molecular and textual spaces, or imposed stringent demands on the scale of LLMs. To resolve the challenges, we propose In-Context Molecule Adaptation (ICMA), as a new paradigm allowing LLMs to learn the molecule-text alignment from context examples via In-Context Molecule Tuning. Specifically, ICMA incorporates the following three stages: Cross-modal Retrieval, Post-retrieval Re-ranking, and In-context Molecule Tuning. Initially, Cross-modal Retrieval utilizes BM25 Caption Retrieval and Molecule Graph Retrieval to retrieve informative context examples. Additionally, we also propose Post-retrieval Re-ranking with Sequence Reversal and Random Walk to further improve the quality of retrieval results. Finally, In-Context Molecule Tuning unlocks the in-context molecule learning capability of LLMs with retrieved examples and adapts the parameters of LLMs for the molecule-caption translation task. Experimental results demonstrate that ICMT can empower LLMs to achieve state-of-the-art or comparable performance without extra training corpora and intricate structures, showing that LLMs are inherently in-context molecule learners.

Title: ACC-ViT : Atrous Convolution's Comeback in Vision Transformers

• Authors: Nabil Ibtehaz, Ning Yan, Masood Mortazavi, Daisuke Kihara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04200
• Pdf URL: https://arxiv.org/pdf/2403.04200
• Copy Paste: [[2403.04200]] ACC-ViT : Atrous Convolution's Comeback in Vision Transformers(https://arxiv.org/abs/2403.04200)
• Keywords: transformer
• Abstract: Transformers have elevated to the state-of-the-art vision architectures through innovations in attention mechanism inspired from visual perception. At present two classes of attentions prevail in vision transformers, regional and sparse attention. The former bounds the pixel interactions within a region; the latter spreads them across sparse grids. The opposing natures of them have resulted in a dilemma between either preserving hierarchical relation or attaining a global context. In this work, taking inspiration from atrous convolution, we introduce Atrous Attention, a fusion of regional and sparse attention, which can adaptively consolidate both local and global information, while maintaining hierarchical relations. As a further tribute to atrous convolution, we redesign the ubiquitous inverted residual convolution blocks with atrous convolution. Finally, we propose a generalized, hybrid vision transformer backbone, named ACC-ViT, following conventional practices for standard vision tasks. Our tiny version model achieves $\sim 84 \%$ accuracy on ImageNet-1K, with less than $28.5$ million parameters, which is $0.42\%$ improvement over state-of-the-art MaxViT while having $8.4\%$ less parameters. In addition, we have investigated the efficacy of ACC-ViT backbone under different evaluation settings, such as finetuning, linear probing, and zero-shot learning on tasks involving medical image analysis, object detection, and language-image contrastive learning. ACC-ViT is therefore a strong vision backbone, which is also competitive in mobile-scale versions, ideal for niche applications with small datasets.

Title: HeteroSwitch: Characterizing and Taming System-Induced Data Heterogeneity in Federated Learning

• Authors: Gyudong Kim, Mehdi Ghasemi, Soroush Heidari, Seungryong Kim, Young Geun Kim, Sarma Vrudhula, Carole-Jean Wu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.04207
• Pdf URL: https://arxiv.org/pdf/2403.04207
• Copy Paste: [[2403.04207]] HeteroSwitch: Characterizing and Taming System-Induced Data Heterogeneity in Federated Learning(https://arxiv.org/abs/2403.04207)
• Keywords: privacy, protect, federate, fair
• Abstract: Federated Learning (FL) is a practical approach to train deep learning models collaboratively across user-end devices, protecting user privacy by retaining raw data on-device. In FL, participating user-end devices are highly fragmented in terms of hardware and software configurations. Such fragmentation introduces a new type of data heterogeneity in FL, namely \textit{system-induced data heterogeneity}, as each device generates distinct data depending on its hardware and software configurations. In this paper, we first characterize the impact of system-induced data heterogeneity on FL model performance. We collect a dataset using heterogeneous devices with variations across vendors and performance tiers. By using this dataset, we demonstrate that \textit{system-induced data heterogeneity} negatively impacts accuracy, and deteriorates fairness and domain generalization problems in FL. To address these challenges, we propose HeteroSwitch, which adaptively adopts generalization techniques (i.e., ISP transformation and SWAD) depending on the level of bias caused by varying HW and SW configurations. In our evaluation with a realistic FL dataset (FLAIR), HeteroSwitch reduces the variance of averaged precision by 6.3\% across device types.

Title: Persona Extraction Through Semantic Similarity for Emotional Support Conversation Generation

• Authors: Seunghee Han, Se Jin Park, Chae Won Kim, Yong Man Ro
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04212
• Pdf URL: https://arxiv.org/pdf/2403.04212
• Copy Paste: [[2403.04212]] Persona Extraction Through Semantic Similarity for Emotional Support Conversation Generation(https://arxiv.org/abs/2403.04212)
• Keywords: extraction
• Abstract: Providing emotional support through dialogue systems is becoming increasingly important in today's world, as it can support both mental health and social interactions in many conversation scenarios. Previous works have shown that using persona is effective for generating empathetic and supportive responses. They have often relied on pre-provided persona rather than inferring them during conversations. However, it is not always possible to obtain a user persona before the conversation begins. To address this challenge, we propose PESS (Persona Extraction through Semantic Similarity), a novel framework that can automatically infer informative and consistent persona from dialogues. We devise completeness loss and consistency loss based on semantic similarity scores. The completeness loss encourages the model to generate missing persona information, and the consistency loss guides the model to distinguish between consistent and inconsistent persona. Our experimental results demonstrate that high-quality persona information inferred by PESS is effective in generating emotionally supportive responses.

Title: Self-Evaluation of Large Language Model based on Glass-box Features

• Authors: Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, Tiejun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04222
• Pdf URL: https://arxiv.org/pdf/2403.04222
• Copy Paste: [[2403.04222]] Self-Evaluation of Large Language Model based on Glass-box Features(https://arxiv.org/abs/2403.04222)
• Keywords: large language model
• Abstract: The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect - model-aware glass-box features - is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable indicator for quality evaluation. Furthermore, we propose two strategies to enhance the evaluation by incorporating features derived from references. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features.

Title: Aligners: Decoupling LLMs and Alignment

• Authors: Lilian Ngweta, Mayank Agarwal, Subha Maity, Alex Gittens, Yuekai Sun, Mikhail Yurochkin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04224
• Pdf URL: https://arxiv.org/pdf/2403.04224
• Copy Paste: [[2403.04224]] Aligners: Decoupling LLMs and Alignment(https://arxiv.org/abs/2403.04224)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) need to be aligned with human expectations to ensure their safety and utility in most applications. Alignment is challenging, costly, and needs to be repeated for every LLM and alignment criterion. We propose to decouple LLMs and alignment by training aligner models that can be used to align any LLM for a given criteria on an as-needed basis, thus also reducing the potential negative impacts of alignment on performance. Our recipe for training the aligner models solely relies on synthetic data generated with a (prompted) LLM and can be easily adjusted for a variety of alignment criteria. We illustrate our method by training an "ethical" aligner and verify its efficacy empirically.

Title: 3DTextureTransformer: Geometry Aware Texture Generation for Arbitrary Mesh Topology

• Authors: Dharma KC, Clayton T. Morrison
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04225
• Pdf URL: https://arxiv.org/pdf/2403.04225
• Copy Paste: [[2403.04225]] 3DTextureTransformer: Geometry Aware Texture Generation for Arbitrary Mesh Topology(https://arxiv.org/abs/2403.04225)
• Keywords: transformer
• Abstract: Learning to generate textures for a novel 3D mesh given a collection of 3D meshes and real-world 2D images is an important problem with applications in various domains such as 3D simulation, augmented and virtual reality, gaming, architecture, and design. Existing solutions either do not produce high-quality textures or deform the original high-resolution input mesh topology into a regular grid to make this generation easier but also lose the original mesh topology. In this paper, we present a novel framework called the 3DTextureTransformer that enables us to generate high-quality textures without deforming the original, high-resolution input mesh. Our solution, a hybrid of geometric deep learning and StyleGAN-like architecture, is flexible enough to work on arbitrary mesh topologies and also easily extensible to texture generation for point cloud representations. Our solution employs a message-passing framework in 3D in conjunction with a StyleGAN-like architecture for 3D texture generation. The architecture achieves state-of-the-art performance among a class of solutions that can learn from a collection of 3D geometry and real-world 2D images while working with any arbitrary mesh topology.

Title: DEEP-ICL: Definition-Enriched Experts for Language Model In-Context Learning

• Authors: Xingwei Qu, Yiming Liang, Yucheng Wang, Tianyu Zheng, Tommy Yue, Lei Ma, Stephen W. Huang, Jiajun Zhang, Wenhu Chen, Chenghua Lin, Jie Fu, Ge Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04233
• Pdf URL: https://arxiv.org/pdf/2403.04233
• Copy Paste: [[2403.04233]] DEEP-ICL: Definition-Enriched Experts for Language Model In-Context Learning(https://arxiv.org/abs/2403.04233)
• Keywords: large language model
• Abstract: It has long been assumed that the sheer number of parameters in large language models (LLMs) drives in-context learning (ICL) capabilities, enabling remarkable performance improvements by leveraging task-specific demonstrations. Challenging this hypothesis, we introduce DEEP-ICL, a novel task Definition Enriched ExPert Ensembling methodology for ICL. DEEP-ICL explicitly extracts task definitions from given demonstrations and generates responses through learning task-specific examples. We argue that improvement from ICL does not directly rely on model size, but essentially stems from understanding task definitions and task-guided learning. Inspired by this, DEEP-ICL combines two 3B models with distinct roles (one for concluding task definitions and the other for learning task demonstrations) and achieves comparable performance to LLaMA2-13B. Furthermore, our framework outperforms conventional ICL by overcoming pretraining sequence length limitations, by supporting unlimited demonstrations. We contend that DEEP-ICL presents a novel alternative for achieving efficient few-shot learning, extending beyond the conventional ICL.

Title: UltraWiki: Ultra-fine-grained Entity Set Expansion with Negative Seed Entities

• Authors: Yangning Li, Qingsong Lv, Tianyu Yu, Yinghui Li, Shulin Huang, Tingwei Lu, Xuming Hu, Wenhao JIang, Hai-Tao Zheng, Hui Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04247
• Pdf URL: https://arxiv.org/pdf/2403.04247
• Copy Paste: [[2403.04247]] UltraWiki: Ultra-fine-grained Entity Set Expansion with Negative Seed Entities(https://arxiv.org/abs/2403.04247)
• Keywords: large language model
• Abstract: Entity Set Expansion (ESE) aims to identify new entities belonging to the same semantic class as a given set of seed entities. Traditional methods primarily relied on positive seed entities to represent a target semantic class, which poses challenge for the representation of ultra-fine-grained semantic classes. Ultra-fine-grained semantic classes are defined based on fine-grained semantic classes with more specific attribute constraints. Describing it with positive seed entities alone cause two issues: (i) Ambiguity among ultra-fine-grained semantic classes. (ii) Inability to define "unwanted" semantic. Due to these inherent shortcomings, previous methods struggle to address the ultra-fine-grained ESE (Ultra-ESE). To solve this issue, we first introduce negative seed entities in the inputs, which belong to the same fine-grained semantic class as the positive seed entities but differ in certain attributes. Negative seed entities eliminate the semantic ambiguity by contrast between positive and negative attributes. Meanwhile, it provide a straightforward way to express "unwanted". To assess model performance in Ultra-ESE, we constructed UltraWiki, the first large-scale dataset tailored for Ultra-ESE. UltraWiki encompasses 236 ultra-fine-grained semantic classes, where each query of them is represented with 3-5 positive and negative seed entities. A retrieval-based framework RetExpan and a generation-based framework GenExpan are proposed to comprehensively assess the efficacy of large language models from two different paradigms in Ultra-ESE. Moreover, we devised three strategies to enhance models' comprehension of ultra-fine-grained entities semantics: contrastive learning, retrieval augmentation, and chain-of-thought reasoning. Extensive experiments confirm the effectiveness of our proposed strategies and also reveal that there remains a large space for improvement in Ultra-ESE.

Title: Depth-aware Test-Time Training for Zero-shot Video Object Segmentation

• Authors: Weihuang Liu, Xi Shen, Haolun Li, Xiuli Bi, Bo Liu, Chi-Man Pun, Xiaodong Cun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04258
• Pdf URL: https://arxiv.org/pdf/2403.04258
• Copy Paste: [[2403.04258]] Depth-aware Test-Time Training for Zero-shot Video Object Segmentation(https://arxiv.org/abs/2403.04258)
• Keywords: segmentation
• Abstract: Zero-shot Video Object Segmentation (ZSVOS) aims at segmenting the primary moving object without any human annotations. Mainstream solutions mainly focus on learning a single model on large-scale video datasets, which struggle to generalize to unseen videos. In this work, we introduce a test-time training (TTT) strategy to address the problem. Our key insight is to enforce the model to predict consistent depth during the TTT process. In detail, we first train a single network to perform both segmentation and depth prediction tasks. This can be effectively learned with our specifically designed depth modulation layer. Then, for the TTT process, the model is updated by predicting consistent depth maps for the same frame under different data augmentations. In addition, we explore different TTT weight updating strategies. Our empirical results suggest that the momentum-based weight initialization and looping-based training scheme lead to more stable improvements. Experiments show that the proposed method achieves clear improvements on ZSVOS. Our proposed video TTT strategy provides significant superiority over state-of-the-art TTT methods. Our code is available at: https://nifangbaage.github.io/DATTT.

Title: Controllable Generation with Text-to-Image Diffusion Models: A Survey

• Authors: Pu Cao, Feng Zhou, Qing Song, Lu Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04279
• Pdf URL: https://arxiv.org/pdf/2403.04279
• Copy Paste: [[2403.04279]] Controllable Generation with Text-to-Image Diffusion Models: A Survey(https://arxiv.org/abs/2403.04279)
• Keywords: diffusion, generative
• Abstract: In the rapidly advancing realm of visual generation, diffusion models have revolutionized the landscape, marking a significant shift in capabilities with their impressive text-guided generative functions. However, relying solely on text for conditioning these models does not fully cater to the varied and complex requirements of different applications and scenarios. Acknowledging this shortfall, a variety of studies aim to control pre-trained text-to-image (T2I) models to support novel conditions. In this survey, we undertake a thorough review of the literature on controllable generation with T2I diffusion models, covering both the theoretical foundations and practical advancements in this domain. Our review begins with a brief introduction to the basics of denoising diffusion probabilistic models (DDPMs) and widely used T2I diffusion models. We then reveal the controlling mechanisms of diffusion models, theoretically analyzing how novel conditions are introduced into the denoising process for conditional generation. Additionally, we offer a detailed overview of research in this area, organizing it into distinct categories from the condition perspective: generation with specific conditions, generation with multiple conditions, and universal controllable generation. For an exhaustive list of the controllable generation literature surveyed, please refer to our curated repository at \url{https://github.com/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models}.

Title: Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy

• Authors: Yu Zhu, Chuxiong Sun, Wenfei Yang, Wenqiang Wei, Bo Tang, Tianzhu Zhang, Zhiyu Li, Shifeng Zhang, Feiyu Xiong, Jie Hu, Mingchuan yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04283
• Pdf URL: https://arxiv.org/pdf/2403.04283
• Copy Paste: [[2403.04283]] Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy(https://arxiv.org/abs/2403.04283)
• Keywords: large language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is the prevailing approach to ensure Large Language Models (LLMs) align with human values. However, existing RLHF methods require a high computational cost, one main reason being that RLHF assigns both the generation and alignment tasks to the LLM simultaneously. In this paper, we introduce Proxy-RLHF, which decouples the generation and alignment processes of LLMs, achieving alignment with human values at a much lower computational cost. We start with a novel Markov Decision Process (MDP) designed for the alignment process and employ Reinforcement Learning (RL) to train a streamlined proxy model that oversees the token generation of the LLM, without altering the LLM itself. Experiments show that our method achieves a comparable level of alignment with only 1\% of the training parameters of other methods.

Title: A$^{3}$lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP

• Authors: Zeng Tao, Yan Wang, Junxiong Lin, Haoran Wang, Xinji Mai, Jiawen Yu, Xuan Tong, Ziheng Zhou, Shaoqi Yan, Qing Zhao, Liyuan Han, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04294
• Pdf URL: https://arxiv.org/pdf/2403.04294
• Copy Paste: [[2403.04294]] A$^{3}$lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP(https://arxiv.org/abs/2403.04294)
• Keywords: extraction
• Abstract: The performance of CLIP in dynamic facial expression recognition (DFER) task doesn't yield exceptional results as observed in other CLIP-based classification tasks. While CLIP's primary objective is to achieve alignment between images and text in the feature space, DFER poses challenges due to the abstract nature of text and the dynamic nature of video, making label representation limited and perfect alignment difficult. To address this issue, we have designed A$^{3}$lign-DFER, which introduces a new DFER labeling paradigm to comprehensively achieve alignment, thus enhancing CLIP's suitability for the DFER task. Specifically, our A$^{3}$lign-DFER method is designed with multiple modules that work together to obtain the most suitable expanded-dimensional embeddings for classification and to achieve alignment in three key aspects: affective, dynamic, and bidirectional. We replace the input label text with a learnable Multi-Dimensional Alignment Token (MAT), enabling alignment of text to facial expression video samples in both affective and dynamic dimensions. After CLIP feature extraction, we introduce the Joint Dynamic Alignment Synchronizer (JAS), further facilitating synchronization and alignment in the temporal dimension. Additionally, we implement a Bidirectional Alignment Training Paradigm (BAP) to ensure gradual and steady training of parameters for both modalities. Our insightful and concise A$^{3}$lign-DFER method achieves state-of-the-art results on multiple DFER datasets, including DFEW, FERV39k, and MAFW. Extensive ablation experiments and visualization studies demonstrate the effectiveness of A$^{3}$lign-DFER. The code will be available in the future.

Title: LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking

• Authors: Jialin Li, Qiang Nie, Weifu Fu, Yuhuan Lin, Guangpin Tao, Yong Liu, Chengjie Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04303
• Pdf URL: https://arxiv.org/pdf/2403.04303
• Copy Paste: [[2403.04303]] LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking(https://arxiv.org/abs/2403.04303)
• Keywords: transformer
• Abstract: Deep learning models, particularly those based on transformers, often employ numerous stacked structures, which possess identical architectures and perform similar functions. While effective, this stacking paradigm leads to a substantial increase in the number of parameters, posing challenges for practical applications. In today's landscape of increasingly large models, stacking depth can even reach dozens, further exacerbating this issue. To mitigate this problem, we introduce LORS (LOw-rank Residual Structure). LORS allows stacked modules to share the majority of parameters, requiring a much smaller number of unique ones per module to match or even surpass the performance of using entirely distinct ones, thereby significantly reducing parameter usage. We validate our method by applying it to the stacked decoders of a query-based object detector, and conduct extensive experiments on the widely used MS COCO dataset. Experimental results demonstrate the effectiveness of our method, as even with a 70\% reduction in the parameters of the decoder, our method still enables the model to achieve comparable or

Title: Effectiveness Assessment of Recent Large Vision-Language Models

• Authors: Yao Jiang, Xinyu Yan, Ge-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng-Ping Fan, Fahad Shahbaz Khan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04306
• Pdf URL: https://arxiv.org/pdf/2403.04306
• Copy Paste: [[2403.04306]] Effectiveness Assessment of Recent Large Vision-Language Models(https://arxiv.org/abs/2403.04306)
• Keywords: robust
• Abstract: The advent of large vision-language models (LVLMs) represents a noteworthy advancement towards the pursuit of artificial general intelligence. However, the extent of their efficacy across both specialized and general tasks warrants further investigation. This article endeavors to evaluate the competency of popular LVLMs in specialized and general tasks, respectively, aiming to offer a comprehensive comprehension of these innovative methodologies. To gauge their efficacy in specialized tasks, we tailor a comprehensive testbed comprising three distinct scenarios: natural, healthcare, and industrial, encompassing six challenging tasks. These tasks include salient, camouflaged, and transparent object detection, as well as polyp and skin lesion detection, alongside industrial anomaly detection. We examine the performance of three recent open-source LVLMs -- MiniGPT-v2, LLaVA-1.5, and Shikra -- in the realm of visual recognition and localization. Moreover, we conduct empirical investigations utilizing the aforementioned models alongside GPT-4V, assessing their multi-modal understanding capacities in general tasks such as object counting, absurd question answering, affordance reasoning, attribute recognition, and spatial relation reasoning. Our investigations reveal that these models demonstrate limited proficiency not only in specialized tasks but also in general tasks. We delve deeper into this inadequacy and suggest several potential factors, including limited cognition in specialized tasks, object hallucination, text-to-image interference, and decreased robustness in complex problems. We hope this study would provide valuable insights for the future development of LVLMs, augmenting their power in coping with both general and specialized applications.

Title: HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild

• Authors: Zhiying Zhu, Zhiqing Sun, Yiming Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04307
• Pdf URL: https://arxiv.org/pdf/2403.04307
• Copy Paste: [[2403.04307]] HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild(https://arxiv.org/abs/2403.04307)
• Keywords: large language model
• Abstract: Hallucinations pose a significant challenge to the reliability of large language models (LLMs) in critical domains. Recent benchmarks designed to assess LLM hallucinations within conventional NLP tasks, such as knowledge-intensive question answering (QA) and summarization, are insufficient for capturing the complexities of user-LLM interactions in dynamic, real-world settings. To address this gap, we introduce HaluEval-Wild, the first benchmark specifically designed to evaluate LLM hallucinations in the wild. We meticulously collect challenging (adversarially filtered by Alpaca) user queries from existing real-world user-LLM interaction datasets, including ShareGPT, to evaluate the hallucination rates of various LLMs. Upon analyzing the collected queries, we categorize them into five distinct types, which enables a fine-grained analysis of the types of hallucinations LLMs exhibit, and synthesize the reference answers with the powerful GPT-4 model and retrieval-augmented generation (RAG). Our benchmark offers a novel approach towards enhancing our comprehension and improvement of LLM reliability in scenarios reflective of real-world interactions.

Title: AO-DETR: Anti-Overlapping DETR for X-Ray Prohibited Items Detection

• Authors: Mingyuan Li, Tong Jia, Hao Wang, Bowen Ma, Shuyang Lin, Da Cai, Dongyue Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04309
• Pdf URL: https://arxiv.org/pdf/2403.04309
• Copy Paste: [[2403.04309]] AO-DETR: Anti-Overlapping DETR for X-Ray Prohibited Items Detection(https://arxiv.org/abs/2403.04309)
• Keywords: security
• Abstract: Prohibited item detection in X-ray images is one of the most essential and highly effective methods widely employed in various security inspection scenarios. Considering the significant overlapping phenomenon in X-ray prohibited item images, we propose an Anti-Overlapping DETR (AO-DETR) based on one of the state-of-the-art general object detectors, DINO. Specifically, to address the feature coupling issue caused by overlapping phenomena, we introduce the Category-Specific One-to-One Assignment (CSA) strategy to constrain category-specific object queries in predicting prohibited items of fixed categories, which can enhance their ability to extract features specific to prohibited items of a particular category from the overlapping foreground-background features. To address the edge blurring problem caused by overlapping phenomena, we propose the Look Forward Densely (LFD) scheme, which improves the localization accuracy of reference boxes in mid-to-high-level decoder layers and enhances the ability to locate blurry edges of the final layer. Similar to DINO, our AO-DETR provides two different versions with distinct backbones, tailored to meet diverse application requirements. Extensive experiments on the PIXray and OPIXray datasets demonstrate that the proposed method surpasses the state-of-the-art object detectors, indicating its potential applications in the field of prohibited item detection. The source code will be released at https://github.com/Limingyuan001/AO-DETR-test.

Title: Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders

• Authors: Yuwei Zhang, Siffi Singh, Sailik Sengupta, Igor Shalyminov, Hang Su, Hwanjun Song, Saab Mansour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04314
• Pdf URL: https://arxiv.org/pdf/2403.04314
• Copy Paste: [[2403.04314]] Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders(https://arxiv.org/abs/2403.04314)
• Keywords: large language model
• Abstract: Conversational systems often rely on embedding models for intent classification and intent clustering tasks. The advent of Large Language Models (LLMs), which enable instructional embeddings allowing one to adjust semantics over the embedding space using prompts, are being viewed as a panacea for these downstream conversational tasks. However, traditional evaluation benchmarks rely solely on task metrics that don't particularly measure gaps related to semantic understanding. Thus, we propose an intent semantic toolkit that gives a more holistic view of intent embedding models by considering three tasks-- (1) intent classification, (2) intent clustering, and (3) a novel triplet task. The triplet task gauges the model's understanding of two semantic concepts paramount in real-world conversational systems-- negation and implicature. We observe that current embedding models fare poorly in semantic understanding of these concepts. To address this, we propose a pre-training approach to improve the embedding model by leveraging augmentation with data generated by an auto-regressive model and a contrastive loss term. Our approach improves the semantic understanding of the intent embedding model on the aforementioned linguistic dimensions while slightly effecting their performance on downstream task metrics.

Title: Online Adaptation of Language Models with a Memory of Amortized Contexts

• Authors: Jihoon Tack, Jaehyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Teh, Jonathan Richard Schwarz
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04317
• Pdf URL: https://arxiv.org/pdf/2403.04317
• Copy Paste: [[2403.04317]] Online Adaptation of Language Models with a Memory of Amortized Contexts(https://arxiv.org/abs/2403.04317)
• Keywords: extraction, large language model
• Abstract: Due to the rapid generation and dissemination of information, large language models (LLMs) quickly run out of date despite enormous development costs. Due to this crucial need to keep models updated, online learning has emerged as a critical necessity when utilizing LLMs for real-world applications. However, given the ever-expanding corpus of unseen documents and the large parameter space of modern LLMs, efficient adaptation is essential. To address these challenges, we propose Memory of Amortized Contexts (MAC), an efficient and effective online adaptation framework for LLMs with strong knowledge retention. We propose an amortized feature extraction and memory-augmentation approach to compress and extract information from new documents into compact modulations stored in a memory bank. When answering questions, our model attends to and extracts relevant knowledge from this memory bank. To learn informative modulations in an efficient manner, we utilize amortization-based meta-learning, which substitutes the optimization process with a single forward pass of the encoder. Subsequently, we learn to choose from and aggregate selected documents into a single modulation by conditioning on the question, allowing us to adapt a frozen language model during test time without requiring further gradient updates. Our experiment demonstrates the superiority of MAC in multiple aspects, including online adaptation performance, time, and memory efficiency. Code is available at: https://github.com/jihoontack/MAC.

Title: Discriminative Probing and Tuning for Text-to-Image Generation

• Authors: Leigang Qu, Wenjie Wang, Yongqi Li, Hanwang Zhang, Liqiang Nie, Tat-Seng Chua
• Subjects: cs.CV, cs.AI, cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.04321
• Pdf URL: https://arxiv.org/pdf/2403.04321
• Copy Paste: [[2403.04321]] Discriminative Probing and Tuning for Text-to-Image Generation(https://arxiv.org/abs/2403.04321)
• Keywords: generative, large language model
• Abstract: Despite advancements in text-to-image generation (T2I), prior methods often face text-image misalignment problems such as relation confusion in generated images. Existing solutions involve cross-attention manipulation for better compositional understanding or integrating large language models for improved layout planning. However, the inherent alignment capabilities of T2I models are still inadequate. By reviewing the link between generative and discriminative modeling, we posit that T2I models' discriminative abilities may reflect their text-image alignment proficiency during generation. In this light, we advocate bolstering the discriminative abilities of T2I models to achieve more precise text-to-image alignment for generation. We present a discriminative adapter built on T2I models to probe their discriminative abilities on two representative tasks and leverage discriminative fine-tuning to improve their text-image alignment. As a bonus of the discriminative adapter, a self-correction mechanism can leverage discriminative gradients to better align generated images to text prompts during inference. Comprehensive evaluations across three benchmark datasets, including both in-distribution and out-of-distribution scenarios, demonstrate our method's superior generation performance. Meanwhile, it achieves state-of-the-art discriminative performance on the two discriminative tasks compared to other generative models.

Title: Measuring Meaning Composition in the Human Brain with Composition Scores from Large Language Models

• Authors: Changjiang Gao, Jixing Li, Jiajun Chen, Shujian Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04325
• Pdf URL: https://arxiv.org/pdf/2403.04325
• Copy Paste: [[2403.04325]] Measuring Meaning Composition in the Human Brain with Composition Scores from Large Language Models(https://arxiv.org/abs/2403.04325)
• Keywords: transformer, large language model
• Abstract: The process of meaning composition, wherein smaller units like morphemes or words combine to form the meaning of phrases and sentences, is essential for human sentence comprehension. Despite extensive neurolinguistic research into the brain regions involved in meaning composition, a computational metric to quantify the extent of composition is still lacking. Drawing on the key-value memory interpretation of transformer feed-forward network blocks, we introduce the Composition Score, a novel model-based metric designed to quantify the degree of meaning composition during sentence comprehension. Experimental findings show that this metric correlates with brain clusters associated with word frequency, structural processing, and general sensitivity to words, suggesting the multifaceted nature of meaning composition during human sentence comprehension.

Title: Spatiotemporal Pooling on Appropriate Topological Maps Represented as Two-Dimensional Images for EEG Classification

• Authors: Takuto Fukushima, Ryusuke Miyamoto
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04353
• Pdf URL: https://arxiv.org/pdf/2403.04353
• Copy Paste: [[2403.04353]] Spatiotemporal Pooling on Appropriate Topological Maps Represented as Two-Dimensional Images for EEG Classification(https://arxiv.org/abs/2403.04353)
• Keywords: transformer
• Abstract: Motor imagery classification based on electroencephalography (EEG) signals is one of the most important brain-computer interface applications, although it needs further improvement. Several methods have attempted to obtain useful information from EEG signals by using recent deep learning techniques such as transformers. To improve the classification accuracy, this study proposes a novel EEG-based motor imagery classification method with three key features: generation of a topological map represented as a two-dimensional image from EEG signals with coordinate transformation based on t-SNE, use of the InternImage to extract spatial features, and use of spatiotemporal pooling inspired by PoolFormer to exploit spatiotemporal information concealed in a sequence of EEG images. Experimental results using the PhysioNet EEG Motor Movement/Imagery dataset showed that the proposed method achieved the best classification accuracy of 88.57%, 80.65%, and 70.17% on two-, three-, and four-class motor imagery tasks in cross-individual validation.

Title: Multi-step Temporal Modeling for UAV Tracking

• Authors: Xiaoying Yuan, Tingfa Xu, Xincong Liu, Ying Wang, Haolin Qin, Yuqiang Fang, Jianan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04363
• Pdf URL: https://arxiv.org/pdf/2403.04363
• Copy Paste: [[2403.04363]] Multi-step Temporal Modeling for UAV Tracking(https://arxiv.org/abs/2403.04363)
• Keywords: transformer
• Abstract: In the realm of unmanned aerial vehicle (UAV) tracking, Siamese-based approaches have gained traction due to their optimal balance between efficiency and precision. However, UAV scenarios often present challenges such as insufficient sampling resolution, fast motion and small objects with limited feature information. As a result, temporal context in UAV tracking tasks plays a pivotal role in target location, overshadowing the target's precise features. In this paper, we introduce MT-Track, a streamlined and efficient multi-step temporal modeling framework designed to harness the temporal context from historical frames for enhanced UAV tracking. This temporal integration occurs in two steps: correlation map generation and correlation map refinement. Specifically, we unveil a unique temporal correlation module that dynamically assesses the interplay between the template and search region features. This module leverages temporal information to refresh the template feature, yielding a more precise correlation map. Subsequently, we propose a mutual transformer module to refine the correlation maps of historical and current frames by modeling the temporal knowledge in the tracking sequence. This method significantly trims computational demands compared to the raw transformer. The compact yet potent nature of our tracking framework ensures commendable tracking outcomes, particularly in extended tracking scenarios.

Title: Video-Driven Animation of Neural Head Avatars

• Authors: Wolfgang Paier, Paul Hinzer, Anna Hilsmann, Peter Eisert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04380
• Pdf URL: https://arxiv.org/pdf/2403.04380
• Copy Paste: [[2403.04380]] Video-Driven Animation of Neural Head Avatars(https://arxiv.org/abs/2403.04380)
• Keywords: generative
• Abstract: We present a new approach for video-driven animation of high-quality neural 3D head models, addressing the challenge of person-independent animation from video input. Typically, high-quality generative models are learned for specific individuals from multi-view video footage, resulting in person-specific latent representations that drive the generation process. In order to achieve person-independent animation from video input, we introduce an LSTM-based animation network capable of translating person-independent expression features into personalized animation parameters of person-specific 3D head models. Our approach combines the advantages of personalized head models (high quality and realism) with the convenience of video-driven animation employing multi-person facial performance capture. We demonstrate the effectiveness of our approach on synthesized animations with high quality based on different source videos as well as an ablation study.

Title: Acceleron: A Tool to Accelerate Research Ideation

• Authors: Harshit Nigam, Manasi Patwardhan, Lovekesh Vig, Gautam Shroff
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04382
• Pdf URL: https://arxiv.org/pdf/2403.04382
• Copy Paste: [[2403.04382]] Acceleron: A Tool to Accelerate Research Ideation(https://arxiv.org/abs/2403.04382)
• Keywords: large language model
• Abstract: Several tools have recently been proposed for assisting researchers during various stages of the research life-cycle. However, these primarily concentrate on tasks such as retrieving and recommending relevant literature, reviewing and critiquing the draft, and writing of research manuscripts. Our investigation reveals a significant gap in availability of tools specifically designed to assist researchers during the challenging ideation phase of the research life-cycle. To aid with research ideation, we propose `Acceleron', a research accelerator for different phases of the research life cycle, and which is specially designed to aid the ideation process. Acceleron guides researchers through the formulation of a comprehensive research proposal, encompassing a novel research problem. The proposals motivation is validated for novelty by identifying gaps in the existing literature and suggesting a plausible list of techniques to solve the proposed problem. We leverage the reasoning and domain-specific skills of Large Language Models (LLMs) to create an agent-based architecture incorporating colleague and mentor personas for LLMs. The LLM agents emulate the ideation process undertaken by researchers, engaging researchers in an interactive fashion to aid in the development of the research proposal. Notably, our tool addresses challenges inherent in LLMs, such as hallucinations, implements a two-stage aspect-based retrieval to manage precision-recall trade-offs, and tackles issues of unanswerability. As evaluation, we illustrate the execution of our motivation validation and method synthesis workflows on proposals from the ML and NLP domain, given by 3 distinct researchers. Our observations and evaluations provided by the researchers illustrate the efficacy of the tool in terms of assisting researchers with appropriate inputs at distinct stages and thus leading to improved time efficiency.

Title: Impacts of Color and Texture Distortions on Earth Observation Data in Deep Learning

• Authors: Martin Willbo, Aleksis Pirinen, John Martinsson, Edvin Listo Zec, Olof Mogren, Mikael Nilsson
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04385
• Pdf URL: https://arxiv.org/pdf/2403.04385
• Copy Paste: [[2403.04385]] Impacts of Color and Texture Distortions on Earth Observation Data in Deep Learning(https://arxiv.org/abs/2403.04385)
• Keywords: robust, transformer, segmentation
• Abstract: Land cover classification and change detection are two important applications of remote sensing and Earth observation (EO) that have benefited greatly from the advances of deep learning. Convolutional and transformer-based U-net models are the state-of-the-art architectures for these tasks, and their performances have been boosted by an increased availability of large-scale annotated EO datasets. However, the influence of different visual characteristics of the input EO data on a model's predictions is not well understood. In this work we systematically examine model sensitivities with respect to several color- and texture-based distortions on the input EO data during inference, given models that have been trained without such distortions. We conduct experiments with multiple state-of-the-art segmentation networks for land cover classification and show that they are in general more sensitive to texture than to color distortions. Beyond revealing intriguing characteristics of widely used land cover classification models, our results can also be used to guide the development of more robust models within the EO domain.

Title: Collaborative Cybersecurity Using Blockchain: A Survey

• Authors: Loïc Miller, Marc-Oliver Pahl
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.04410
• Pdf URL: https://arxiv.org/pdf/2403.04410
• Copy Paste: [[2403.04410]] Collaborative Cybersecurity Using Blockchain: A Survey(https://arxiv.org/abs/2403.04410)
• Keywords: security
• Abstract: Collaborative cybersecurity relies on organizations sharing information to boost security, but trust management is a key concern. Decentralized solutions like distributed ledgers, particularly blockchain, are crucial for eliminating single points of failure. However, the existing literature on blockchain-based collaborative cybersecurity is limited, lacking comprehensive insights. This paper addresses this gap by surveying blockchain's role in collaborative cybersecurity from 2016 to 2023. It explores various applications, trends, and the evolution of blockchain technology, focusing on access control, data validation policies, underlying tech, and consensus mechanisms. A key finding is the fragmentation of the field with no dominant research group or venue. Many recent projects poorly select consensus protocols for their blockchain. To aid researchers and practitioners, this paper offers guidelines for choosing the right blockchain for specific purposes and highlights open research areas and lessons learned from past blockchain applications in collaborative cybersecurity, encouraging further exploration in this field.

Title: Exploring the Influence of Dimensionality Reduction on Anomaly Detection Performance in Multivariate Time Series

• Authors: Mahsun Altin, Altan Cakir
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04429
• Pdf URL: https://arxiv.org/pdf/2403.04429
• Copy Paste: [[2403.04429]] Exploring the Influence of Dimensionality Reduction on Anomaly Detection Performance in Multivariate Time Series(https://arxiv.org/abs/2403.04429)
• Keywords: robust, transformer
• Abstract: This paper presents an extensive empirical study on the integration of dimensionality reduction techniques with advanced unsupervised time series anomaly detection models, focusing on the MUTANT and Anomaly-Transformer models. The study involves a comprehensive evaluation across three different datasets: MSL, SMAP, and SWaT. Each dataset poses unique challenges, allowing for a robust assessment of the models' capabilities in varied contexts. The dimensionality reduction techniques examined include PCA, UMAP, Random Projection, and t-SNE, each offering distinct advantages in simplifying high-dimensional data. Our findings reveal that dimensionality reduction not only aids in reducing computational complexity but also significantly enhances anomaly detection performance in certain scenarios. Moreover, a remarkable reduction in training times was observed, with reductions by approximately 300\% and 650\% when dimensionality was halved and minimized to the lowest dimensions, respectively. This efficiency gain underscores the dual benefit of dimensionality reduction in both performance enhancement and operational efficiency. The MUTANT model exhibits notable adaptability, especially with UMAP reduction, while the Anomaly-Transformer demonstrates versatility across various reduction techniques. These insights provide a deeper understanding of the synergistic effects of dimensionality reduction and anomaly detection, contributing valuable perspectives to the field of time series analysis. The study underscores the importance of selecting appropriate dimensionality reduction strategies based on specific model requirements and dataset characteristics, paving the way for more efficient, accurate, and scalable solutions in anomaly detection.

Title: On-demand Quantization for Green Federated Generative Diffusion in Mobile Edge Networks

• Authors: Bingkun Lai, Jiayi He, Jiawen Kang, Gaolei Li, Minrui Xu, Tao zhang, Shengli Xie
• Subjects: cs.LG, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.04430
• Pdf URL: https://arxiv.org/pdf/2403.04430
• Copy Paste: [[2403.04430]] On-demand Quantization for Green Federated Generative Diffusion in Mobile Edge Networks(https://arxiv.org/abs/2403.04430)
• Keywords: federate, diffusion, generative
• Abstract: Generative Artificial Intelligence (GAI) shows remarkable productivity and creativity in Mobile Edge Networks, such as the metaverse and the Industrial Internet of Things. Federated learning is a promising technique for effectively training GAI models in mobile edge networks due to its data distribution. However, there is a notable issue with communication consumption when training large GAI models like generative diffusion models in mobile edge networks. Additionally, the substantial energy consumption associated with training diffusion-based models, along with the limited resources of edge devices and complexities of network environments, pose challenges for improving the training efficiency of GAI models. To address this challenge, we propose an on-demand quantized energy-efficient federated diffusion approach for mobile edge networks. Specifically, we first design a dynamic quantized federated diffusion training scheme considering various demands from the edge devices. Then, we study an energy efficiency problem based on specific quantization requirements. Numerical results show that our proposed method significantly reduces system energy consumption and transmitted model size compared to both baseline federated diffusion and fixed quantized federated diffusion methods while effectively maintaining reasonable quality and diversity of generated data.

Title: Boosting Fairness and Robustness in Over-the-Air Federated Learning

• Authors: Halil Yigit Oksuz, Fabio Molinari, Henning Sprekeler, Joerg Raisch
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.04431
• Pdf URL: https://arxiv.org/pdf/2403.04431
• Copy Paste: [[2403.04431]] Boosting Fairness and Robustness in Over-the-Air Federated Learning(https://arxiv.org/abs/2403.04431)
• Keywords: privacy, robust, federate, fair
• Abstract: Over-the-Air Computation is a beyond-5G communication strategy that has recently been shown to be useful for the decentralized training of machine learning models due to its efficiency. In this paper, we propose an Over-the-Air federated learning algorithm that aims to provide fairness and robustness through minmax optimization. By using the epigraph form of the problem at hand, we show that the proposed algorithm converges to the optimal solution of the minmax problem. Moreover, the proposed approach does not require reconstructing channel coefficients by complex encoding-decoding schemes as opposed to state-of-the-art approaches. This improves both efficiency and privacy.

Title: StableDrag: Stable Dragging for Point-based Image Editing

• Authors: Yutao Cui, Xiaotong Zhao, Guozhen Zhang, Shengming Cao, Kai Ma, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04437
• Pdf URL: https://arxiv.org/pdf/2403.04437
• Copy Paste: [[2403.04437]] StableDrag: Stable Dragging for Point-based Image Editing(https://arxiv.org/abs/2403.04437)
• Keywords: diffusion, generative
• Abstract: Point-based image editing has attracted remarkable attention since the emergence of DragGAN. Recently, DragDiffusion further pushes forward the generative quality via adapting this dragging technique to diffusion models. Despite these great success, this dragging scheme exhibits two major drawbacks, namely inaccurate point tracking and incomplete motion supervision, which may result in unsatisfactory dragging outcomes. To tackle these issues, we build a stable and precise drag-based editing framework, coined as StableDrag, by designing a discirminative point tracking method and a confidence-based latent enhancement strategy for motion supervision. The former allows us to precisely locate the updated handle points, thereby boosting the stability of long-range manipulation, while the latter is responsible for guaranteeing the optimized latent as high-quality as possible across all the manipulation steps. Thanks to these unique designs, we instantiate two types of image editing models including StableDrag-GAN and StableDrag-Diff, which attains more stable dragging performance, through extensive qualitative experiments and quantitative assessment on DragBench.

Title: FriendNet: Detection-Friendly Dehazing Network

• Authors: Yihua Fan, Yongzhen Wang, Mingqiang Wei, Fu Lee Wang, Haoran Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04443
• Pdf URL: https://arxiv.org/pdf/2403.04443
• Copy Paste: [[2403.04443]] FriendNet: Detection-Friendly Dehazing Network(https://arxiv.org/abs/2403.04443)
• Keywords: extraction
• Abstract: Adverse weather conditions often impair the quality of captured images, inevitably inducing cutting-edge object detection models for advanced driver assistance systems (ADAS) and autonomous driving. In this paper, we raise an intriguing question: can the combination of image restoration and object detection enhance detection performance in adverse weather conditions? To answer it, we propose an effective architecture that bridges image dehazing and object detection together via guidance information and task-driven learning to achieve detection-friendly dehazing, termed FriendNet. FriendNet aims to deliver both high-quality perception and high detection capacity. Different from existing efforts that intuitively treat image dehazing as pre-processing, FriendNet establishes a positive correlation between these two tasks. Clean features generated by the dehazing network potentially contribute to improvements in object detection performance. Conversely, object detection crucially guides the learning process of the image dehazing network under the task-driven learning scheme. We shed light on how downstream tasks can guide upstream dehazing processes, considering both network architecture and learning objectives. We design Guidance Fusion Block (GFB) and Guidance Attention Block (GAB) to facilitate the integration of detection information into the network. Furthermore, the incorporation of the detection task loss aids in refining the optimization process. Additionally, we introduce a new Physics-aware Feature Enhancement Block (PFEB), which integrates physics-based priors to enhance the feature extraction and representation capabilities. Extensive experiments on synthetic and real-world datasets demonstrate the superiority of our method over state-of-the-art methods on both image quality and detection precision. Our source code is available at https://github.com/fanyihua0309/FriendNet.

Title: Disentangled Diffusion-Based 3D Human Pose Estimation with Hierarchical Spatial and Temporal Denoiser

• Authors: Qingyuan Cai, Xuecai Hu, Saihui Hou, Li Yao, Yongzhen Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04444
• Pdf URL: https://arxiv.org/pdf/2403.04444
• Copy Paste: [[2403.04444]] Disentangled Diffusion-Based 3D Human Pose Estimation with Hierarchical Spatial and Temporal Denoiser(https://arxiv.org/abs/2403.04444)
• Keywords: diffusion, transformer
• Abstract: Recently, diffusion-based methods for monocular 3D human pose estimation have achieved state-of-the-art (SOTA) performance by directly regressing the 3D joint coordinates from the 2D pose sequence. Although some methods decompose the task into bone length and bone direction prediction based on the human anatomical skeleton to explicitly incorporate more human body prior constraints, the performance of these methods is significantly lower than that of the SOTA diffusion-based methods. This can be attributed to the tree structure of the human skeleton. Direct application of the disentangled method could amplify the accumulation of hierarchical errors, propagating through each hierarchy. Meanwhile, the hierarchical information has not been fully explored by the previous methods. To address these problems, a Disentangled Diffusion-based 3D Human Pose Estimation method with Hierarchical Spatial and Temporal Denoiser is proposed, termed DDHPose. In our approach: (1) We disentangle the 3D pose and diffuse the bone length and bone direction during the forward process of the diffusion model to effectively model the human pose prior. A disentanglement loss is proposed to supervise diffusion model learning. (2) For the reverse process, we propose Hierarchical Spatial and Temporal Denoiser (HSTDenoiser) to improve the hierarchical modeling of each joint. Our HSTDenoiser comprises two components: the Hierarchical-Related Spatial Transformer (HRST) and the Hierarchical-Related Temporal Transformer (HRTT). HRST exploits joint spatial information and the influence of the parent joint on each joint for spatial modeling, while HRTT utilizes information from both the joint and its hierarchical adjacent joints to explore the hierarchical temporal correlations among joints.

Title: FRRI: a novel algorithm for fuzzy-rough rule induction

• Authors: Henri Bollaert, Marko Palangetić, Chris Cornelis, Salvatore Greco, Roman Słowiński
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04447
• Pdf URL: https://arxiv.org/pdf/2403.04447
• Copy Paste: [[2403.04447]] FRRI: a novel algorithm for fuzzy-rough rule induction(https://arxiv.org/abs/2403.04447)
• Keywords: interpretability
• Abstract: Interpretability is the next frontier in machine learning research. In the search for white box models - as opposed to black box models, like random forests or neural networks - rule induction algorithms are a logical and promising option, since the rules can easily be understood by humans. Fuzzy and rough set theory have been successfully applied to this archetype, almost always separately. As both approaches to rule induction involve granular computing based on the concept of equivalence classes, it is natural to combine them. The QuickRules\cite{JensenCornelis2009} algorithm was a first attempt at using fuzzy rough set theory for rule induction. It is based on QuickReduct, a greedy algorithm for building decision reducts. QuickRules already showed an improvement over other rule induction methods. However, to evaluate the full potential of a fuzzy rough rule induction algorithm, one needs to start from the foundations. In this paper, we introduce a novel rule induction algorithm called Fuzzy Rough Rule Induction (FRRI). We provide background and explain the workings of our algorithm. Furthermore, we perform a computational experiment to evaluate the performance of our algorithm and compare it to other state-of-the-art rule induction approaches. We find that our algorithm is more accurate while creating small rulesets consisting of relatively short rules. We end the paper by outlining some directions for future work.

Title: Membership Inference Attacks and Privacy in Topic Modeling

• Authors: Nico Manzonelli, Wanrong Zhang, Salil Vadhan
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04451
• Pdf URL: https://arxiv.org/pdf/2403.04451
• Copy Paste: [[2403.04451]] Membership Inference Attacks and Privacy in Topic Modeling(https://arxiv.org/abs/2403.04451)
• Keywords: privacy, attack, membership infer, generative, large language model
• Abstract: Recent research shows that large language models are susceptible to privacy attacks that infer aspects of the training data. However, it is unclear if simpler generative models, like topic models, share similar vulnerabilities. In this work, we propose an attack against topic models that can confidently identify members of the training data in Latent Dirichlet Allocation. Our results suggest that the privacy risks associated with generative modeling are not restricted to large neural models. Additionally, to mitigate these vulnerabilities, we explore differentially private (DP) topic modeling. We propose a framework for private topic modeling that incorporates DP vocabulary selection as a pre-processing step, and show that it improves privacy while having limited effects on practical utility.

Title: Vlearn: Off-Policy Learning with Efficient State-Value Function Estimation

• Authors: Fabian Otto, Philipp Becker, Vien Ang Ngo, Gerhard Neumann
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04453
• Pdf URL: https://arxiv.org/pdf/2403.04453
• Copy Paste: [[2403.04453]] Vlearn: Off-Policy Learning with Efficient State-Value Function Estimation(https://arxiv.org/abs/2403.04453)
• Keywords: robust
• Abstract: Existing off-policy reinforcement learning algorithms typically necessitate an explicit state-action-value function representation, which becomes problematic in high-dimensional action spaces. These algorithms often encounter challenges where they struggle with the curse of dimensionality, as maintaining a state-action-value function in such spaces becomes data-inefficient. In this work, we propose a novel off-policy trust region optimization approach, called Vlearn, that eliminates the requirement for an explicit state-action-value function. Instead, we demonstrate how to efficiently leverage just a state-value function as the critic, thus overcoming several limitations of existing methods. By doing so, Vlearn addresses the computational challenges posed by high-dimensional action spaces. Furthermore, Vlearn introduces an efficient approach to address the challenges associated with pure state-value function learning in the off-policy setting. This approach not only simplifies the implementation of off-policy policy gradient algorithms but also leads to consistent and robust performance across various benchmark tasks. Specifically, by removing the need for a state-action-value function Vlearn simplifies the learning process and allows for more efficient exploration and exploitation in complex environments

Title: Low-Resource Court Judgment Summarization for Common Law Systems

• Authors: Shuaiqi Liu, Jiannong Cao, Yicong Li, Ruosong Yang, Zhiyuan Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04454
• Pdf URL: https://arxiv.org/pdf/2403.04454
• Copy Paste: [[2403.04454]] Low-Resource Court Judgment Summarization for Common Law Systems(https://arxiv.org/abs/2403.04454)
• Keywords: large language model
• Abstract: Common law courts need to refer to similar precedents' judgments to inform their current decisions. Generating high-quality summaries of court judgment documents can facilitate legal practitioners to efficiently review previous cases and assist the general public in accessing how the courts operate and how the law is applied. Previous court judgment summarization research focuses on civil law or a particular jurisdiction's judgments. However, judges can refer to the judgments from all common law jurisdictions. Current summarization datasets are insufficient to satisfy the demands of summarizing precedents across multiple jurisdictions, especially when labeled data are scarce for many jurisdictions. To address the lack of datasets, we present CLSum, the first dataset for summarizing multi-jurisdictional common law court judgment documents. Besides, this is the first court judgment summarization work adopting large language models (LLMs) in data augmentation, summary generation, and evaluation. Specifically, we design an LLM-based data augmentation method incorporating legal knowledge. We also propose a legal knowledge enhanced evaluation metric based on LLM to assess the quality of generated judgment summaries. Our experimental results verify that the LLM-based summarization methods can perform well in the few-shot and zero-shot settings. Our LLM-based data augmentation method can mitigate the impact of low data resources. Furthermore, we carry out comprehensive comparative experiments to find essential model components and settings that are capable of enhancing summarization performance.

Title: Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset

• Authors: Minjin Kim, Minju Kim, Hana Kim, Beong-woo Kwak, Soyeon Chun, Hyunseo Kim, SeongKu Kang, Youngjae Yu, Jinyoung Yeo, Dongha Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04460
• Pdf URL: https://arxiv.org/pdf/2403.04460
• Copy Paste: [[2403.04460]] Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset(https://arxiv.org/abs/2403.04460)
• Keywords: large language model
• Abstract: Conversational recommender system is an emerging area that has garnered an increasing interest in the community, especially with the advancements in large language models (LLMs) that enable diverse reasoning over conversational input. Despite the progress, the field has many aspects left to explore. The currently available public datasets for conversational recommendation lack specific user preferences and explanations for recommendations, hindering high-quality recommendations. To address such challenges, we present a novel conversational recommendation dataset named PEARL, synthesized with persona- and knowledge-augmented LLM simulators. We obtain detailed persona and knowledge from real-world reviews and construct a large-scale dataset with over 57k dialogues. Our experimental results demonstrate that utterances in PEARL include more specific user preferences, show expertise in the target domain, and provide recommendations more relevant to the dialogue context than those in prior datasets.

Title: A Survey of Graph Neural Networks in Real world: Imbalance, Noise, Privacy and OOD Challenges

• Authors: Wei Ju, Siyu Yi, Yifan Wang, Zhiping Xiao, Zhengyang Mao, Hourun Li, Yiyang Gu, Yifang Qin, Nan Yin, Senzhang Wang, Xinwang Liu, Xiao Luo, Philip S. Yu, Ming Zhang
• Subjects: cs.LG, cs.AI, cs.IR, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.04468
• Pdf URL: https://arxiv.org/pdf/2403.04468
• Copy Paste: [[2403.04468]] A Survey of Graph Neural Networks in Real world: Imbalance, Noise, Privacy and OOD Challenges(https://arxiv.org/abs/2403.04468)
• Keywords: security, privacy, protect, robust
• Abstract: Graph-structured data exhibits universality and widespread applicability across diverse domains, such as social network analysis, biochemistry, financial fraud detection, and network security. Significant strides have been made in leveraging Graph Neural Networks (GNNs) to achieve remarkable success in these areas. However, in real-world scenarios, the training environment for models is often far from ideal, leading to substantial performance degradation of GNN models due to various unfavorable factors, including imbalance in data distribution, the presence of noise in erroneous data, privacy protection of sensitive information, and generalization capability for out-of-distribution (OOD) scenarios. To tackle these issues, substantial efforts have been devoted to improving the performance of GNN models in practical real-world scenarios, as well as enhancing their reliability and robustness. In this paper, we present a comprehensive survey that systematically reviews existing GNN models, focusing on solutions to the four mentioned real-world challenges including imbalance, noise, privacy, and OOD in practical scenarios that many existing reviews have not considered. Specifically, we first highlight the four key challenges faced by existing GNNs, paving the way for our exploration of real-world GNN models. Subsequently, we provide detailed discussions on these four aspects, dissecting how these solutions contribute to enhancing the reliability and robustness of GNN models. Last but not least, we outline promising directions and offer future perspectives in the field.

Title: TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document

• Authors: Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, Xiang Bai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04473
• Pdf URL: https://arxiv.org/pdf/2403.04473
• Copy Paste: [[2403.04473]] TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document(https://arxiv.org/abs/2403.04473)
• Keywords: interpretability
• Abstract: We present TextMonkey, a large multimodal model (LMM) tailored for text-centric tasks, including document question answering (DocVQA) and scene text analysis. Our approach introduces enhancement across several dimensions: by adopting Shifted Window Attention with zero-initialization, we achieve cross-window connectivity at higher input resolutions and stabilize early training; We hypothesize that images may contain redundant tokens, and by using similarity to filter out significant tokens, we can not only streamline the token length but also enhance the model's performance. Moreover, by expanding our model's capabilities to encompass text spotting and grounding, and incorporating positional information into responses, we enhance interpretability and minimize hallucinations. Additionally, TextMonkey can be finetuned to gain the ability to comprehend commands for clicking screenshots. Overall, our method notably boosts performance across various benchmark datasets, achieving increases of 5.2%, 6.9%, and 2.8% in Scene Text-Centric VQA, Document Oriented VQA, and KIE, respectively, especially with a score of 561 on OCRBench, surpassing prior open-sourced large multimodal models for document understanding. Code will be released at https://github.com/Yuliang-Liu/Monkey.

Title: Do Large Language Model Understand Multi-Intent Spoken Language ?

• Authors: Shangjian Yin, Peijie Huang, Yuhong Xu, Haojing Huang, Jiatian Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04481
• Pdf URL: https://arxiv.org/pdf/2403.04481
• Copy Paste: [[2403.04481]] Do Large Language Model Understand Multi-Intent Spoken Language ?(https://arxiv.org/abs/2403.04481)
• Keywords: generative, large language model
• Abstract: This study marks a significant advancement by harnessing Large Language Models (LLMs) for multi-intent spoken language understanding (SLU), proposing a unique methodology that capitalizes on the generative power of LLMs within an SLU context. Our innovative technique reconfigures entity slots specifically for LLM application in multi-intent SLU environments and introduces the concept of Sub-Intent Instruction (SII), enhancing the dissection and interpretation of intricate, multi-intent communication within varied domains. The resultant datasets, dubbed LM-MixATIS and LM-MixSNIPS, are crafted from pre-existing benchmarks. Our research illustrates that LLMs can match and potentially excel beyond the capabilities of current state-of-the-art multi-intent SLU models. It further explores LLM efficacy across various intent configurations and dataset proportions. Moreover, we introduce two pioneering metrics, Entity Slot Accuracy (ESA) and Combined Semantic Accuracy (CSA), to provide an in-depth analysis of LLM proficiency in this complex field.

Title: On the Topology Awareness and Generalization Performance of Graph Neural Networks

• Authors: Junwei Su, Chuan Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04482
• Pdf URL: https://arxiv.org/pdf/2403.04482
• Copy Paste: [[2403.04482]] On the Topology Awareness and Generalization Performance of Graph Neural Networks(https://arxiv.org/abs/2403.04482)
• Keywords: fair
• Abstract: Many computer vision and machine learning problems are modelled as learning tasks on graphs, where graph neural networks (GNNs) have emerged as a dominant tool for learning representations of graph-structured data. A key feature of GNNs is their use of graph structures as input, enabling them to exploit the graphs' inherent topological properties-known as the topology awareness of GNNs. Despite the empirical successes of GNNs, the influence of topology awareness on generalization performance remains unexplored, particularly for node-level tasks that diverge from the assumption of data being independent and identically distributed (I.I.D.). The precise definition and characterization of the topology awareness of GNNs, especially concerning different topological features, are still unclear. This paper introduces a comprehensive framework to characterize the topology awareness of GNNs across any topological feature. Using this framework, we investigate the effects of topology awareness on GNN generalization performance. Contrary to the prevailing belief that enhancing the topology awareness of GNNs is always advantageous, our analysis reveals a critical insight: improving the topology awareness of GNNs may inadvertently lead to unfair generalization across structural groups, which might not be desired in some scenarios. Additionally, we conduct a case study using the intrinsic graph metric, the shortest path distance, on various benchmark datasets. The empirical results of this case study confirm our theoretical insights. Moreover, we demonstrate the practical applicability of our framework by using it to tackle the cold start problem in graph active learning.

Title: Source Matters: Source Dataset Impact on Model Robustness in Medical Imaging

• Authors: Dovile Juodelyte, Yucheng Lu, Amelia Jiménez-Sánchez, Sabrina Bottazzi, Enzo Ferrante, Veronika Cheplygina
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04484
• Pdf URL: https://arxiv.org/pdf/2403.04484
• Copy Paste: [[2403.04484]] Source Matters: Source Dataset Impact on Model Robustness in Medical Imaging(https://arxiv.org/abs/2403.04484)
• Keywords: robust
• Abstract: Transfer learning has become an essential part of medical imaging classification algorithms, often leveraging ImageNet weights. However, the domain shift from natural to medical images has prompted alternatives such as RadImageNet, often demonstrating comparable classification performance. However, it remains unclear whether the performance gains from transfer learning stem from improved generalization or shortcut learning. To address this, we investigate potential confounders -- whether synthetic or sampled from the data -- across two publicly available chest X-ray and CT datasets. We show that ImageNet and RadImageNet achieve comparable classification performance, yet ImageNet is much more prone to overfitting to confounders. We recommend that researchers using ImageNet-pretrained models reexamine their model robustness by conducting similar experiments. Our code and experiments are available at https://github.com/DovileDo/source-matters.

Title: Privacy in Cloud Computing through Immersion-based Coding

• Authors: Haleh Hayati, Nathan van de Wouw, Carlos Murguia
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.04485
• Pdf URL: https://arxiv.org/pdf/2403.04485
• Copy Paste: [[2403.04485]] Privacy in Cloud Computing through Immersion-based Coding(https://arxiv.org/abs/2403.04485)
• Keywords: privacy
• Abstract: Cloud computing enables users to process and store data remotely on high-performance computers and servers by sharing data over the Internet. However, transferring data to clouds causes unavoidable privacy concerns. Here, we present a synthesis framework to design coding mechanisms that allow sharing and processing data in a privacy-preserving manner without sacrificing data utility and algorithmic performance. We consider the setup where the user aims to run an algorithm in the cloud using private data. The cloud then returns some data utility back to the user (utility refers to the service that the algorithm provides, e.g., classification, prediction, AI models, etc.). To avoid privacy concerns, the proposed scheme provides tools to co-design: 1) coding mechanisms to distort the original data and guarantee a prescribed differential privacy level; 2) an equivalent-but-different algorithm (referred here to as the target algorithm) that runs on distorted data and produces distorted utility; and 3) a decoding function that extracts the true utility from the distorted one with a negligible error. Then, instead of sharing the original data and algorithm with the cloud, only the distorted data and target algorithm are disclosed, thereby avoiding privacy concerns. The proposed scheme is built on the synergy of differential privacy and system immersion tools from control theory. The key underlying idea is to design a higher-dimensional target algorithm that embeds all trajectories of the original algorithm and works on randomly encoded data to produce randomly encoded utility. We show that the proposed scheme can be designed to offer any level of differential privacy without degrading the algorithm's utility. We present two use cases to illustrate the performance of the developed tools: privacy in optimization/learning algorithms and a nonlinear networked control system.

Title: What makes an image realistic?

• Authors: Lucas Theis
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.04493
• Pdf URL: https://arxiv.org/pdf/2403.04493
• Copy Paste: [[2403.04493]] What makes an image realistic?(https://arxiv.org/abs/2403.04493)
• Keywords: generative
• Abstract: The last decade has seen tremendous progress in our ability to generate realistic-looking data, be it images, text, audio, or video. Here, we discuss the closely related problem of quantifying realism, that is, designing functions that can reliably tell realistic data from unrealistic data. This problem turns out to be significantly harder to solve and remains poorly understood, despite its prevalence in machine learning and recent breakthroughs in generative AI. Drawing on insights from algorithmic information theory, we discuss why this problem is challenging, why a good generative model alone is insufficient to solve it, and what a good solution would look like. In particular, we introduce the notion of a universal critic, which unlike adversarial critics does not require adversarial training. While universal critics are not immediately practical, they can serve both as a North Star for guiding practical implementations and as a tool for analyzing existing attempts to capture realism.

Title: NLPre: a revised approach towards language-centric benchmarking of Natural Language Preprocessing systems

• Authors: Martyna Wiącek, Piotr Rybak, Łukasz Pszenny, Alina Wróblewska
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04507
• Pdf URL: https://arxiv.org/pdf/2403.04507
• Copy Paste: [[2403.04507]] NLPre: a revised approach towards language-centric benchmarking of Natural Language Preprocessing systems(https://arxiv.org/abs/2403.04507)
• Keywords: fair, transformer
• Abstract: With the advancements of transformer-based architectures, we observe the rise of natural language preprocessing (NLPre) tools capable of solving preliminary NLP tasks (e.g. tokenisation, part-of-speech tagging, dependency parsing, or morphological analysis) without any external linguistic guidance. It is arduous to compare novel solutions to well-entrenched preprocessing toolkits, relying on rule-based morphological analysers or dictionaries. Aware of the shortcomings of existing NLPre evaluation approaches, we investigate a novel method of reliable and fair evaluation and performance reporting. Inspired by the GLUE benchmark, the proposed language-centric benchmarking system enables comprehensive ongoing evaluation of multiple NLPre tools, while credibly tracking their performance. The prototype application is configured for Polish and integrated with the thoroughly assembled NLPre-PL benchmark. Based on this benchmark, we conduct an extensive evaluation of a variety of Polish NLPre systems. To facilitate the construction of benchmarking environments for other languages, e.g. NLPre-GA for Irish or NLPre-ZH for Chinese, we ensure full customization of the publicly released source code of the benchmarking system. The links to all the resources (deployed platforms, source code, trained models, datasets etc.) can be found on the project website: https://sites.google.com/view/nlpre-benchmark.

Title: Where does In-context Translation Happen in Large Language Models

• Authors: Suzanna Sia, David Mueller, Kevin Duh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04510
• Pdf URL: https://arxiv.org/pdf/2403.04510
• Copy Paste: [[2403.04510]] Where does In-context Translation Happen in Large Language Models(https://arxiv.org/abs/2403.04510)
• Keywords: large language model
• Abstract: Self-supervised large language models have demonstrated the ability to perform Machine Translation (MT) via in-context learning, but little is known about where the model performs the task with respect to prompt instructions and demonstration examples. In this work, we attempt to characterize the region where large language models transition from in-context learners to translation models. Through a series of layer-wise context-masking experiments on \textsc{GPTNeo2.7B}, \textsc{Bloom3B}, \textsc{Llama7b} and \textsc{Llama7b-chat}, we demonstrate evidence of a "task recognition" point where the translation task is encoded into the input representations and attention to context is no longer necessary. We further observe correspondence between the low performance when masking out entire layers, and the task recognition layers. Taking advantage of this redundancy results in 45\% computational savings when prompting with 5 examples, and task recognition achieved at layer 14 / 32. Our layer-wise fine-tuning experiments indicate that the most effective layers for MT fine-tuning are the layers critical to task recognition.

Title: Uncertainty-Aware Relational Graph Neural Network for Few-Shot Knowledge Graph Completion

• Authors: Qian Li, Shu Guo, Yingjia Chen, Cheng Ji, Jiawei Sheng, Jianxin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04521
• Pdf URL: https://arxiv.org/pdf/2403.04521
• Copy Paste: [[2403.04521]] Uncertainty-Aware Relational Graph Neural Network for Few-Shot Knowledge Graph Completion(https://arxiv.org/abs/2403.04521)
• Keywords: robust
• Abstract: Few-shot knowledge graph completion (FKGC) aims to query the unseen facts of a relation given its few-shot reference entity pairs. The side effect of noises due to the uncertainty of entities and triples may limit the few-shot learning, but existing FKGC works neglect such uncertainty, which leads them more susceptible to limited reference samples with noises. In this paper, we propose a novel uncertainty-aware few-shot KG completion framework (UFKGC) to model uncertainty for a better understanding of the limited data by learning representations under Gaussian distribution. Uncertainty representation is first designed for estimating the uncertainty scope of the entity pairs after transferring feature representations into a Gaussian distribution. Further, to better integrate the neighbors with uncertainty characteristics for entity features, we design an uncertainty-aware relational graph neural network (UR-GNN) to conduct convolution operations between the Gaussian distributions. Then, multiple random samplings are conducted for reference triples within the Gaussian distribution to generate smooth reference representations during the optimization. The final completion score for each query instance is measured by the designed uncertainty optimization to make our approach more robust to the noises in few-shot scenarios. Experimental results show that our approach achieves excellent performance on two benchmark datasets compared to its competitors.

Title: T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers

• Authors: Mariano V. Ntrougkas, Nikolaos Gkalelis, Vasileios Mezaris
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.04523
• Pdf URL: https://arxiv.org/pdf/2403.04523
• Copy Paste: [[2403.04523]] T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers(https://arxiv.org/abs/2403.04523)
• Keywords: explainability, transformer
• Abstract: The development and adoption of Vision Transformers and other deep-learning architectures for image classification tasks has been rapid. However, the "black box" nature of neural networks is a barrier to adoption in applications where explainability is essential. While some techniques for generating explanations have been proposed, primarily for Convolutional Neural Networks, adapting such techniques to the new paradigm of Vision Transformers is non-trivial. This paper presents T-TAME, Transformer-compatible Trainable Attention Mechanism for Explanations, a general methodology for explaining deep neural networks used in image classification tasks. The proposed architecture and training technique can be easily applied to any convolutional or Vision Transformer-like neural network, using a streamlined training approach. After training, explanation maps can be computed in a single forward pass; these explanation maps are comparable to or outperform the outputs of computationally expensive perturbation-based explainability techniques, achieving SOTA performance. We apply T-TAME to three popular deep learning classifier architectures, VGG-16, ResNet-50, and ViT-B-16, trained on the ImageNet dataset, and we demonstrate improvements over existing state-of-the-art explainability methods. A detailed analysis of the results and an ablation study provide insights into how the T-TAME design choices affect the quality of the generated explanation maps.

Title: Hyperspectral unmixing for Raman spectroscopy via physics-constrained autoencoders

• Authors: Dimitar Georgiev, Álvaro Fernández-Galiana, Simon Vilms Pedersen, Georgios Papadopoulos, Ruoxiao Xie, Molly M. Stevens, Mauricio Barahona
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04526
• Pdf URL: https://arxiv.org/pdf/2403.04526
• Copy Paste: [[2403.04526]] Hyperspectral unmixing for Raman spectroscopy via physics-constrained autoencoders(https://arxiv.org/abs/2403.04526)
• Keywords: robust
• Abstract: Raman spectroscopy is widely used across scientific domains to characterize the chemical composition of samples in a non-destructive, label-free manner. Many applications entail the unmixing of signals from mixtures of molecular species to identify the individual components present and their proportions, yet conventional methods for chemometrics often struggle with complex mixture scenarios encountered in practice. Here, we develop hyperspectral unmixing algorithms based on autoencoder neural networks, and we systematically validate them using both synthetic and experimental benchmark datasets created in-house. Our results demonstrate that unmixing autoencoders provide improved accuracy, robustness and efficiency compared to standard unmixing methods. We also showcase the applicability of autoencoders to complex biological settings by showing improved biochemical characterization of volumetric Raman imaging data from a monocytic cell.

Title: Enhancing Data Quality in Federated Fine-Tuning of Foundation Models

• Authors: Wanru Zhao, Yaxin Du, Nicholas Donald Lane, Siheng Chen, Yanfeng Wang
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.04529
• Pdf URL: https://arxiv.org/pdf/2403.04529
• Copy Paste: [[2403.04529]] Enhancing Data Quality in Federated Fine-Tuning of Foundation Models(https://arxiv.org/abs/2403.04529)
• Keywords: federate
• Abstract: In the current landscape of foundation model training, there is a significant reliance on public domain data, which is nearing exhaustion according to recent research. To further scale up, it is crucial to incorporate collaboration among multiple specialized and high-quality private domain data sources. However, the challenge of training models locally without sharing private data presents numerous obstacles in data quality control. To tackle this issue, we propose a data quality control pipeline for federated fine-tuning of foundation models. This pipeline computes scores reflecting the quality of training data and determines a global threshold for a unified standard, aiming for improved global performance. Our experiments show that the proposed quality control pipeline facilitates the effectiveness and reliability of the model training, leading to better performance.

Title: Architectural Blueprint For Heterogeneity-Resilient Federated Learning

• Authors: Satwat Bashir, Tasos Dagiuklas, Kasra Kassai, Muddesar Iqbal
• Subjects: cs.LG, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.04546
• Pdf URL: https://arxiv.org/pdf/2403.04546
• Copy Paste: [[2403.04546]] Architectural Blueprint For Heterogeneity-Resilient Federated Learning(https://arxiv.org/abs/2403.04546)
• Keywords: privacy, federate
• Abstract: This paper proposes a novel three tier architecture for federated learning to optimize edge computing environments. The proposed architecture addresses the challenges associated with client data heterogeneity and computational constraints. It introduces a scalable, privacy preserving framework that enhances the efficiency of distributed machine learning. Through experimentation, the paper demonstrates the architecture capability to manage non IID data sets more effectively than traditional federated learning models. Additionally, the paper highlights the potential of this innovative approach to significantly improve model accuracy, reduce communication overhead, and facilitate broader adoption of federated learning technologies.

Title: Explainable Face Verification via Feature-Guided Gradient Backpropagation

• Authors: Yuhang Lu, Zewei Xu, Touradj Ebrahimi
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.04549
• Pdf URL: https://arxiv.org/pdf/2403.04549
• Copy Paste: [[2403.04549]] Explainable Face Verification via Feature-Guided Gradient Backpropagation(https://arxiv.org/abs/2403.04549)
• Keywords: security
• Abstract: Recent years have witnessed significant advancement in face recognition (FR) techniques, with their applications widely spread in people's lives and security-sensitive areas. There is a growing need for reliable interpretations of decisions of such systems. Existing studies relying on various mechanisms have investigated the usage of saliency maps as an explanation approach, but suffer from different limitations. This paper first explores the spatial relationship between face image and its deep representation via gradient backpropagation. Then a new explanation approach FGGB has been conceived, which provides precise and insightful similarity and dissimilarity saliency maps to explain the "Accept" and "Reject" decision of an FR system. Extensive visual presentation and quantitative measurement have shown that FGGB achieves superior performance in both similarity and dissimilarity maps when compared to current state-of-the-art explainable face verification approaches.

Title: Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes

• Authors: Stamatios Georgoulis, Weining Ren, Alfredo Bochicchio, Daniel Eckert, Yuanyou Li, Abel Gawel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04562
• Pdf URL: https://arxiv.org/pdf/2403.04562
• Copy Paste: [[2403.04562]] Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes(https://arxiv.org/abs/2403.04562)
• Keywords: transformer, segmentation
• Abstract: Rapid and reliable identification of dynamic scene parts, also known as motion segmentation, is a key challenge for mobile sensors. Contemporary RGB camera-based methods rely on modeling camera and scene properties however, are often under-constrained and fall short in unknown categories. Event cameras have the potential to overcome these limitations, but corresponding methods have only been demonstrated in smaller-scale indoor environments with simplified dynamic objects. This work presents an event-based method for class-agnostic motion segmentation that can successfully be deployed across complex large-scale outdoor environments too. To this end, we introduce a novel divide-and-conquer pipeline that combines: (a) ego-motion compensated events, computed via a scene understanding module that predicts monocular depth and camera pose as auxiliary tasks, and (b) optical flow from a dedicated optical flow module. These intermediate representations are then fed into a segmentation module that predicts motion segmentation masks. A novel transformer-based temporal attention module in the segmentation module builds correlations across adjacent 'frames' to get temporally consistent segmentation masks. Our method sets the new state-of-the-art on the classic EV-IMO benchmark (indoors), where we achieve improvements of 2.19 moving object IoU (2.22 mIoU) and 4.52 point IoU respectively, as well as on a newly-generated motion segmentation and tracking benchmark (outdoors) based on the DSEC event dataset, termed DSEC-MOTS, where we show improvement of 12.91 moving object IoU.

Title: Embodied Understanding of Driving Scenarios

• Authors: Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, Hongyang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04593
• Pdf URL: https://arxiv.org/pdf/2403.04593
• Copy Paste: [[2403.04593]] Embodied Understanding of Driving Scenarios(https://arxiv.org/abs/2403.04593)
• Keywords: robust
• Abstract: Embodied scene understanding serves as the cornerstone for autonomous agents to perceive, interpret, and respond to open driving scenarios. Such understanding is typically founded upon Vision-Language Models (VLMs). Nevertheless, existing VLMs are restricted to the 2D domain, devoid of spatial awareness and long-horizon extrapolation proficiencies. We revisit the key aspects of autonomous driving and formulate appropriate rubrics. Hereby, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents' understanding of driving scenes with large spatial and temporal spans. ELM incorporates space-aware pre-training to endow the agent with robust spatial localization capabilities. Besides, the model employs time-aware token selection to accurately inquire about temporal cues. We instantiate ELM on the reformulated multi-faced benchmark, and it surpasses previous state-of-the-art approaches in all aspects. All code, data, and models will be publicly shared.

Title: Pix2Gif: Motion-Guided Diffusion for GIF Generation

• Authors: Hitesh Kandala, Jianfeng Gao, Jianwei Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04634
• Pdf URL: https://arxiv.org/pdf/2403.04634
• Copy Paste: [[2403.04634]] Pix2Gif: Motion-Guided Diffusion for GIF Generation(https://arxiv.org/abs/2403.04634)
• Keywords: diffusion
• Abstract: We present Pix2Gif, a motion-guided diffusion model for image-to-GIF (video) generation. We tackle this problem differently by formulating the task as an image translation problem steered by text and motion magnitude prompts, as shown in teaser fig. To ensure that the model adheres to motion guidance, we propose a new motion-guided warping module to spatially transform the features of the source image conditioned on the two types of prompts. Furthermore, we introduce a perceptual loss to ensure the transformed feature map remains within the same space as the target image, ensuring content consistency and coherence. In preparation for the model training, we meticulously curated data by extracting coherent image frames from the TGIF video-caption dataset, which provides rich information about the temporal changes of subjects. After pretraining, we apply our model in a zero-shot manner to a number of video datasets. Extensive qualitative and quantitative experiments demonstrate the effectiveness of our model -- it not only captures the semantic prompt from text but also the spatial ones from motion guidance. We train all our models using a single node of 16xV100 GPUs. Code, dataset and models are made public at: https://hiteshk03.github.io/Pix2Gif/.

Title: CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios

• Authors: Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, Xiaochun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04640
• Pdf URL: https://arxiv.org/pdf/2403.04640
• Copy Paste: [[2403.04640]] CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios(https://arxiv.org/abs/2403.04640)
• Keywords: large language model
• Abstract: This paper focuses on the challenge of answering questions in scenarios that are composed of rich and complex dynamic audio-visual components. Although existing Multimodal Large Language Models (MLLMs) can respond to audio-visual content, these responses are sometimes ambiguous and fail to describe specific audio-visual events. To overcome this limitation, we introduce the CAT, which enhances MLLM in three ways: 1) besides straightforwardly bridging audio and video, we design a clue aggregator that aggregates question-related clues in dynamic audio-visual scenarios to enrich the detailed knowledge required for large language models. 2) CAT is trained on a mixed multimodal dataset, allowing direct application in audio-visual scenarios. Notably, we collect an audio-visual joint instruction dataset named AVinstruct, to further enhance the capacity of CAT to model cross-semantic correlations. 3) we propose AI-assisted ambiguity-aware direct preference optimization, a strategy specialized in retraining the model to favor the non-ambiguity response and improve the ability to localize specific audio-visual objects. Extensive experimental results demonstrate that CAT outperforms existing methods on multimodal tasks, especially in Audio-Visual Question Answering (AVQA) tasks. The codes and the collected instructions are released at https://github.com/rikeilong/Bay-CAT.

Title: Teaching Large Language Models to Reason with Reinforcement Learning

• Authors: Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, Roberta Raileanu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04642
• Pdf URL: https://arxiv.org/pdf/2403.04642
• Copy Paste: [[2403.04642]] Teaching Large Language Models to Reason with Reinforcement Learning(https://arxiv.org/abs/2403.04642)
• Keywords: large language model
• Abstract: Reinforcement Learning from Human Feedback (\textbf{RLHF}) has emerged as a dominant approach for aligning LLM outputs with human preferences. Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback (Expert Iteration, Proximal Policy Optimization (\textbf{PPO}), Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate both sparse and dense rewards provided to the LLM both heuristically and via a learned reward model. We additionally start from multiple model sizes and initializations both with and without supervised fine-tuning (\textbf{SFT}) data. Overall, we find all algorithms perform comparably, with Expert Iteration performing best in most cases. Surprisingly, we find the sample complexity of Expert Iteration is similar to that of PPO, requiring at most on the order of $10^6$ samples to converge from a pretrained checkpoint. We investigate why this is the case, concluding that during RL training models fail to explore significantly beyond solutions already produced by SFT models. Additionally, we discuss a trade off between maj@1 and pass@96 metric performance during SFT training and how conversely RL training improves both simultaneously. We then conclude by discussing the implications of our findings for RLHF and the future role of RL in LLM fine-tuning.

Title: Context-Based Multimodal Fusion

• Authors: Bilal Faye, Hanane Azzag, Mustapha Lebbah, Djamel Bouchaffra
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04650
• Pdf URL: https://arxiv.org/pdf/2403.04650
• Copy Paste: [[2403.04650]] Context-Based Multimodal Fusion(https://arxiv.org/abs/2403.04650)
• Keywords: robust
• Abstract: The fusion models, which effectively combine information from different sources, are widely used in solving multimodal tasks. However, they have significant limitations related to aligning data distributions across different modalities. This challenge can lead to inconsistencies and difficulties in learning robust representations. Alignment models, while specifically addressing this issue, often require training "from scratch" with large datasets to achieve optimal results, which can be costly in terms of resources and time. To overcome these limitations, we propose an innovative model called Context-Based Multimodal Fusion (CBMF), which combines both modality fusion and data distribution alignment. In CBMF, each modality is represented by a specific context vector, fused with the embedding of each modality. This enables the use of large pre-trained models that can be frozen, reducing the computational and training data requirements. Additionally, the network learns to differentiate embeddings of different modalities through fusion with context and aligns data distributions using a contrastive approach for self-supervised learning. Thus, CBMF offers an effective and economical solution for solving complex multimodal tasks.

Title: Yi: Open Foundation Models by 01.AI

• Authors: 01.AI: Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, Zonghong Dai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04652
• Pdf URL: https://arxiv.org/pdf/2403.04652
• Copy Paste: [[2403.04652]] Yi: Open Foundation Models by 01.AI(https://arxiv.org/abs/2403.04652)
• Keywords: transformer
• Abstract: We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.

Title: Chain of Thought Explanation for Dialogue State Tracking

• Authors: Lin Xu, Ningxin Peng, Daquan Zhou, See-Kiong Ng, Jinlan Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04656
• Pdf URL: https://arxiv.org/pdf/2403.04656
• Copy Paste: [[2403.04656]] Chain of Thought Explanation for Dialogue State Tracking(https://arxiv.org/abs/2403.04656)
• Keywords: generative
• Abstract: Dialogue state tracking (DST) aims to record user queries and goals during a conversational interaction achieved by maintaining a prede- fined set of slots and their corresponding values. Current approaches decide slot values opaquely, while humans usually adopt a more deliberate approach by collecting information from relevant dialogue turns and then reasoning the appropriate values. In this work, we focus on the steps needed to figure out slot values by proposing a model named Chain-of-Thought-Explanation (CoTE) for the DST task. CoTE, which is built on the generative DST framework, is designed to create detailed explanations step by step after determining the slot values. This process leads to more accurate and reliable slot values. More-over, to improve the reasoning ability of the CoTE, we further construct more fluent and high-quality explanations with automatic paraphrasing, leading the method CoTE-refined. Experimental results on three widely recognized DST benchmarks-MultiWOZ 2.2, WoZ 2.0, and M2M-demonstrate the remarkable effectiveness of the CoTE. Furthermore, through a meticulous fine-grained analysis, we observe significant benefits of our CoTE on samples characterized by longer dialogue turns, user responses, and reasoning steps.

Title: "Did They Consent to That?": Safer Digital Intimacy via Proactive Protection Against Image-Based Sexual Abuse

• Authors: Lucy Qin, Vaughn Hamilton, Sharon Wang, Yigit Aydinalp, Marin Scarlett, Elissa M. Redmiles
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.04659
• Pdf URL: https://arxiv.org/pdf/2403.04659
• Copy Paste: [[2403.04659]] "Did They Consent to That?": Safer Digital Intimacy via Proactive Protection Against Image-Based Sexual Abuse(https://arxiv.org/abs/2403.04659)
• Keywords: security, privacy, protect
• Abstract: As many as 8 in 10 adults share intimate content such as nude or lewd images. Sharing such content has significant benefits for relationship intimacy and body image, and can offer employment. However, stigmatizing attitudes and a lack of technological mitigations put those sharing such content at risk of sexual violence. An estimated 1 in 3 people have been subjected to image-based sexual abuse (IBSA), a spectrum of violence that includes the nonconsensual distribution or threat of distribution of consensually-created intimate content (also called NDII). In this work, we conducted a rigorous empirical interview study of 52 European creators of intimate content to examine the threats they face and how they defend against them, situated in the context of their different use cases for intimate content sharing and their choice of technologies for storing and sharing such content. Synthesizing our results with the limited body of prior work on technological prevention of NDII, we offer concrete next steps for both platforms and security & privacy researchers to work toward safer intimate content sharing through proactive protection.

Title: Dynamic Cross Attention for Audio-Visual Person Verification

• Authors: R. Gnana Praveen, Jahangir Alam
• Subjects: cs.CV, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.04661
• Pdf URL: https://arxiv.org/pdf/2403.04661
• Copy Paste: [[2403.04661]] Dynamic Cross Attention for Audio-Visual Person Verification(https://arxiv.org/abs/2403.04661)
• Keywords: robust
• Abstract: Although person or identity verification has been predominantly explored using individual modalities such as face and voice, audio-visual fusion has recently shown immense potential to outperform unimodal approaches. Audio and visual modalities are often expected to pose strong complementary relationships, which plays a crucial role in effective audio-visual fusion. However, they may not always strongly complement each other, they may also exhibit weak complementary relationships, resulting in poor audio-visual feature representations. In this paper, we propose a Dynamic Cross-Attention (DCA) model that can dynamically select the cross-attended or unattended features on the fly based on the strong or weak complementary relationships, respectively, across audio and visual modalities. In particular, a conditional gating layer is designed to evaluate the contribution of the cross-attention mechanism and choose cross-attended features only when they exhibit strong complementary relationships, otherwise unattended features. Extensive experiments are conducted on the Voxceleb1 dataset to demonstrate the robustness of the proposed model. Results indicate that the proposed model consistently improves the performance on multiple variants of cross-attention while outperforming the state-of-the-art methods.

Title: Telecom Language Models: Must They Be Large?

• Authors: Nicola Piovesan, Antonio De Domenico, Fadhel Ayed
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04666
• Pdf URL: https://arxiv.org/pdf/2403.04666
• Copy Paste: [[2403.04666]] Telecom Language Models: Must They Be Large?(https://arxiv.org/abs/2403.04666)
• Keywords: large language model
• Abstract: The increasing interest in Large Language Models (LLMs) within the telecommunications sector underscores their potential to revolutionize operational efficiency. However, the deployment of these sophisticated models is often hampered by their substantial size and computational demands, raising concerns about their viability in resource-constrained environments. Addressing this challenge, recent advancements have seen the emergence of small language models that surprisingly exhibit performance comparable to their larger counterparts in many tasks, such as coding and common-sense reasoning. Phi-2, a compact yet powerful model, exemplifies this new wave of efficient small language models. This paper conducts a comprehensive evaluation of Phi-2's intrinsic understanding of the telecommunications domain. Recognizing the scale-related limitations, we enhance Phi-2's capabilities through a Retrieval-Augmented Generation approach, meticulously integrating an extensive knowledge base specifically curated with telecom standard specifications. The enhanced Phi-2 model demonstrates a profound improvement in accuracy, answering questions about telecom standards with a precision that closely rivals the more resource-intensive GPT-3.5. The paper further explores the refined capabilities of Phi-2 in addressing problem-solving scenarios within the telecom sector, highlighting its potential and limitations.

Title: End-to-end Conditional Robust Optimization

• Authors: Abhilash Chenreddy, Erick Delage
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04670
• Pdf URL: https://arxiv.org/pdf/2403.04670
• Copy Paste: [[2403.04670]] End-to-end Conditional Robust Optimization(https://arxiv.org/abs/2403.04670)
• Keywords: robust
• Abstract: The field of Contextual Optimization (CO) integrates machine learning and optimization to solve decision making problems under uncertainty. Recently, a risk sensitive variant of CO, known as Conditional Robust Optimization (CRO), combines uncertainty quantification with robust optimization in order to promote safety and reliability in high stake applications. Exploiting modern differentiable optimization methods, we propose a novel end-to-end approach to train a CRO model in a way that accounts for both the empirical risk of the prescribed decisions and the quality of conditional coverage of the contextual uncertainty set that supports them. While guarantees of success for the latter objective are impossible to obtain from the point of view of conformal prediction theory, high quality conditional coverage is achieved empirically by ingeniously employing a logistic regression differentiable layer within the calculation of coverage quality in our training loss. We show that the proposed training algorithms produce decisions that outperform the traditional estimate then optimize approaches.

Title: PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation

• Authors: Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, Zhenguo Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04692
• Pdf URL: https://arxiv.org/pdf/2403.04692
• Copy Paste: [[2403.04692]] PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation(https://arxiv.org/abs/2403.04692)
• Keywords: diffusion, transformer
• Abstract: In this paper, we introduce PixArt-\Sigma, a Diffusion Transformer model~(DiT) capable of directly generating images at 4K resolution. PixArt-\Sigma represents a significant advancement over its predecessor, PixArt-\alpha, offering images of markedly higher fidelity and improved alignment with text prompts. A key feature of PixArt-\Sigma is its training efficiency. Leveraging the foundational pre-training of PixArt-\alpha, it evolves from the `weaker' baseline to a `stronger' model via incorporating higher quality data, a process we term "weak-to-strong training". The advancements in PixArt-\Sigma are twofold: (1) High-Quality Training Data: PixArt-\Sigma incorporates superior-quality image data, paired with more precise and detailed image captions. (2) Efficient Token Compression: we propose a novel attention module within the DiT framework that compresses both keys and values, significantly improving efficiency and facilitating ultra-high-resolution image generation. Thanks to these improvements, PixArt-\Sigma achieves superior image quality and user prompt adherence capabilities with significantly smaller model size (0.6B parameters) than existing text-to-image diffusion models, such as SDXL (2.6B parameters) and SD Cascade (5.1B parameters). Moreover, PixArt-\Sigma's capability to generate 4K images supports the creation of high-resolution posters and wallpapers, efficiently bolstering the production of high-quality visual content in industries such as film and gaming.

Title: Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification

• Authors: Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04696
• Pdf URL: https://arxiv.org/pdf/2403.04696
• Copy Paste: [[2403.04696]] Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification(https://arxiv.org/abs/2403.04696)
• Keywords: large language model
• Abstract: Large language models (LLMs) are notorious for hallucinating, i.e., producing erroneous claims in their output. Such hallucinations can be dangerous, as occasional factual inaccuracies in the generated text might be obscured by the rest of the output being generally factual, making it extremely hard for the users to spot them. Current services that leverage LLMs usually do not provide any means for detecting unreliable generations. Here, we aim to bridge this gap. In particular, we propose a novel fact-checking and hallucination detection pipeline based on token-level uncertainty quantification. Uncertainty scores leverage information encapsulated in the output of a neural network or its layers to detect unreliable predictions, and we show that they can be used to fact-check the atomic claims in the LLM output. Moreover, we present a novel token-level uncertainty quantification method that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. Our method Claim Conditioned Probability (CCP) measures only the uncertainty of particular claim value expressed by the model. Experiments on the task of biography generation demonstrate strong improvements for CCP compared to the baselines for six different LLMs and three languages. Human evaluation reveals that the fact-checking pipeline based on uncertainty quantification is competitive with a fact-checking tool that leverages external knowledge.

Title: AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectors

• Authors: Kaishen Yuan, Zitong Yu, Xin Liu, Weicheng Xie, Huanjing Yue, Jingyu Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04697
• Pdf URL: https://arxiv.org/pdf/2403.04697
• Copy Paste: [[2403.04697]] AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectors(https://arxiv.org/abs/2403.04697)
• Keywords: robust, transformer
• Abstract: Facial Action Units (AU) is a vital concept in the realm of affective computing, and AU detection has always been a hot research topic. Existing methods suffer from overfitting issues due to the utilization of a large number of learnable parameters on scarce AU-annotated datasets or heavy reliance on substantial additional relevant data. Parameter-Efficient Transfer Learning (PETL) provides a promising paradigm to address these challenges, whereas its existing methods lack design for AU characteristics. Therefore, we innovatively investigate PETL paradigm to AU detection, introducing AUFormer and proposing a novel Mixture-of-Knowledge Expert (MoKE) collaboration mechanism. An individual MoKE specific to a certain AU with minimal learnable parameters first integrates personalized multi-scale and correlation knowledge. Then the MoKE collaborates with other MoKEs in the expert group to obtain aggregated information and inject it into the frozen Vision Transformer (ViT) to achieve parameter-efficient AU detection. Additionally, we design a Margin-truncated Difficulty-aware Weighted Asymmetric Loss (MDWA-Loss), which can encourage the model to focus more on activated AUs, differentiate the difficulty of unactivated AUs, and discard potential mislabeled samples. Extensive experiments from various perspectives, including within-domain, cross-domain, data efficiency, and micro-expression domain, demonstrate AUFormer's state-of-the-art performance and robust generalization abilities without relying on additional relevant data. The code for AUFormer is available at https://github.com/yuankaishen2001/AUFormer.

Title: Delving into the Trajectory Long-tail Distribution for Muti-object Tracking

• Authors: Sijia Chen, En Yu, Jinyang Li, Wenbing Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04700
• Pdf URL: https://arxiv.org/pdf/2403.04700
• Copy Paste: [[2403.04700]] Delving into the Trajectory Long-tail Distribution for Muti-object Tracking(https://arxiv.org/abs/2403.04700)
• Keywords: diffusion
• Abstract: Multiple Object Tracking (MOT) is a critical area within computer vision, with a broad spectrum of practical implementations. Current research has primarily focused on the development of tracking algorithms and enhancement of post-processing techniques. Yet, there has been a lack of thorough examination concerning the nature of tracking data it self. In this study, we pioneer an exploration into the distribution patterns of tracking data and identify a pronounced long-tail distribution issue within existing MOT datasets. We note a significant imbalance in the distribution of trajectory lengths across different pedestrians, a phenomenon we refer to as "pedestrians trajectory long-tail distribution". Addressing this challenge, we introduce a bespoke strategy designed to mitigate the effects of this skewed distribution. Specifically, we propose two data augmentation strategies, including Stationary Camera View Data Augmentation (SVA) and Dynamic Camera View Data Augmentation (DVA) , designed for viewpoint states and the Group Softmax (GS) module for Re-ID. SVA is to backtrack and predict the pedestrian trajectory of tail classes, and DVA is to use diffusion model to change the background of the scene. GS divides the pedestrians into unrelated groups and performs softmax operation on each group individually. Our proposed strategies can be integrated into numerous existing tracking systems, and extensive experimentation validates the efficacy of our method in reducing the influence of long-tail distribution on multi-object tracking performance. The code is available at https://github.com/chen-si-jia/Trajectory-Long-tail-Distribution-for-MOT.

Title: ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes

• Authors: Hashmat Shadab Malik, Muhammad Huzaifa, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04701
• Pdf URL: https://arxiv.org/pdf/2403.04701
• Copy Paste: [[2403.04701]] ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes(https://arxiv.org/abs/2403.04701)
• Keywords: robust, diffusion, generative, large language model
• Abstract: Given the large-scale multi-modal training of recent vision-based models and their generalization capabilities, understanding the extent of their robustness is critical for their real-world deployment. In this work, we evaluate the resilience of current vision-based models against diverse object-to-background context variations. The majority of robustness evaluation methods have introduced synthetic datasets to induce changes to object characteristics (viewpoints, scale, color) or utilized image transformation techniques (adversarial changes, common corruptions) on real images to simulate shifts in distributions. Recent works have explored leveraging large language models and diffusion models to generate changes in the background. However, these methods either lack in offering control over the changes to be made or distort the object semantics, making them unsuitable for the task. Our method, on the other hand, can induce diverse object-to-background changes while preserving the original semantics and appearance of the object. To achieve this goal, we harness the generative capabilities of text-to-image, image-to-text, and image-to-segment models to automatically generate a broad spectrum of object-to-background changes. We induce both natural and adversarial background changes by either modifying the textual prompts or optimizing the latents and textual embedding of text-to-image models. This allows us to quantify the role of background context in understanding the robustness and generalization of deep neural networks. We produce various versions of standard vision datasets (ImageNet, COCO), incorporating either diverse and realistic backgrounds into the images or introducing color, texture, and adversarial changes in the background. We conduct extensive experiment to analyze the robustness of vision-based models against object-to-background context variations across diverse tasks.

Title: Rethinking of Encoder-based Warm-start Methods in Hyperparameter Optimization

• Authors: Dawid Płudowski, Antoni Zajko, Anna Kozak, Katarzyna Woźnica
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04720
• Pdf URL: https://arxiv.org/pdf/2403.04720
• Copy Paste: [[2403.04720]] Rethinking of Encoder-based Warm-start Methods in Hyperparameter Optimization(https://arxiv.org/abs/2403.04720)
• Keywords: extraction
• Abstract: Effectively representing heterogeneous tabular datasets for meta-learning remains an open problem. Previous approaches rely on predefined meta-features, for example, statistical measures or landmarkers. Encoder-based models, such as Dataset2Vec, allow us to extract significant meta-features automatically without human intervention. This research introduces a novel encoder-based representation of tabular datasets implemented within the liltab package available on GitHub https://github.com/azoz01/liltab. Our package is based on an established model for heterogeneous tabular data proposed in [Iwata and Kumagai, 2020]. The proposed approach employs a different model for encoding feature relationships, generating alternative representations compared to existing methods like Dataset2Vec. Both of them leverage the fundamental assumption of dataset similarity learning. In this work, we evaluate Dataset2Vec and liltab on two common meta-tasks - representing entire datasets and hyperparameter optimization warm-start. However, validation on an independent metaMIMIC dataset highlights the nuanced challenges in representation learning. We show that general representations may not suffice for some meta-tasks where requirements are not explicitly considered during extraction. [Iwata and Kumagai, 2020] Tomoharu Iwata and Atsutoshi Kumagai. Meta-learning from Tasks with Heterogeneous Attribute Spaces. In Advances in Neural Information Processing Systems, 2020.

Title: Masked Capsule Autoencoders

• Authors: Miles Everett, Mingjun Zhong, Georgios Leontidis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04724
• Pdf URL: https://arxiv.org/pdf/2403.04724
• Copy Paste: [[2403.04724]] Masked Capsule Autoencoders(https://arxiv.org/abs/2403.04724)
• Keywords: transformer
• Abstract: We propose Masked Capsule Autoencoders (MCAE), the first Capsule Network that utilises pretraining in a self-supervised manner. Capsule Networks have emerged as a powerful alternative to Convolutional Neural Networks (CNNs), and have shown favourable properties when compared to Vision Transformers (ViT), but have struggled to effectively learn when presented with more complex data, leading to Capsule Network models that do not scale to modern tasks. Our proposed MCAE model alleviates this issue by reformulating the Capsule Network to use masked image modelling as a pretraining stage before finetuning in a supervised manner. Across several experiments and ablations studies we demonstrate that similarly to CNNs and ViTs, Capsule Networks can also benefit from self-supervised pretraining, paving the way for further advancements in this neural network domain. For instance, pretraining on the Imagenette dataset, a dataset of 10 classes of Imagenet-sized images, we achieve not only state-of-the-art results for Capsule Networks but also a 9% improvement compared to purely supervised training. Thus we propose that Capsule Networks benefit from and should be trained within a masked image modelling framework, with a novel capsule decoder, to improve a Capsule Network's performance on realistic-sized images.

Title: LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

• Authors: Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04746
• Pdf URL: https://arxiv.org/pdf/2403.04746
• Copy Paste: [[2403.04746]] LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error(https://arxiv.org/abs/2403.04746)
• Keywords: large language model
• Abstract: Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM's 'imagination' to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.

Title: Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed

• Authors: Yifan Wang, Xingyi He, Sida Peng, Dongli Tan, Xiaowei Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04765
• Pdf URL: https://arxiv.org/pdf/2403.04765
• Copy Paste: [[2403.04765]] Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed(https://arxiv.org/abs/2403.04765)
• Keywords: transformer
• Abstract: We present a novel method for efficiently producing semi-dense matches across images. Previous detector-free matcher LoFTR has shown remarkable matching capability in handling large-viewpoint change and texture-poor scenarios but suffers from low efficiency. We revisit its design choices and derive multiple improvements for both efficiency and accuracy. One key observation is that performing the transformer over the entire feature map is redundant due to shared local information, therefore we propose an aggregated attention mechanism with adaptive token selection for efficiency. Furthermore, we find spatial variance exists in LoFTR's fine correlation module, which is adverse to matching accuracy. A novel two-stage correlation layer is proposed to achieve accurate subpixel correspondences for accuracy improvement. Our efficiency optimized model is $\sim 2.5\times$ faster than LoFTR which can even surpass state-of-the-art efficient sparse matching pipeline SuperPoint + LightGlue. Moreover, extensive experiments show that our method can achieve higher accuracy compared with competitive semi-dense matchers, with considerable efficiency benefits. This opens up exciting prospects for large-scale or latency-sensitive applications such as image retrieval and 3D reconstruction. Project page: https://zju3dv.github.io/efficientloftr.